We’re not participating in ForecastBench at the moment. We wanted to explain why, and how we think the benchmark could be improved.
ForecastBench, run by the Forecasting Research Institute (FRI), tests AI models’ forecasting accuracy against live outcomes. Running a live benchmark takes sustained work, and we appreciate FRI making its questions, forecasts, and methodology public.
Benchmarks are essential for measuring and supporting AI progress, but leading benchmarks can often be gamed (“benchmaxxing”) or too easily saturated.
Forecasting benchmarks have an unusual advantage: future outcomes cannot be memorised in training. And since predicting the future is difficult, there is ample headroom for models to improve. Distinguishing forecasting skill from luck requires careful evaluation.
Our analysis suggests that strong rankings can be achieved on ForecastBench without LLMs by reusing public forecasts, relying on historical base rates, and submitting multiple entries.
Problems with market questions
Half the ForecastBench questions are directly borrowed from forecasting platforms and prediction markets like Kalshi. These are often interesting questions but they have one major problem:
(1) The forecasters can defer heavily to the market prices
AI forecasting systems can find the market predictions, e.g. by visiting Kalshi. These provide a strong baseline that entrants can copy.
Across 14 ForecastBench rounds1, copying existing public forecasts would have ranked tied 76th of 464 on the market leaderboard - in the top 20%.2 Its approximate 95% confidence interval overlaps with the 2nd ranked entry.3
We worry that the easy availability of public forecasts makes distinguishing between AI systems difficult.
How can forecasting benchmarks address this problem?
Focus on less liquid markets, where the price is a weaker baseline. Current selection rules require revealed community forecasts or minimum liquidity and trading-activity thresholds.4
Use questions where the human community forecast is not available before the submission deadline, as Metaculus has in its Q1 2025 bot tournament.
Ask questions where AI systems are more likely to disagree, which is what our AI forecasting benchmark Crucible is designed around, with the aim of finding opportunities for further forecasting progress.
Problems with dataset questions
The other half of ForecastBench questions are “dataset” questions: programmatically generated questions about future observations in a regularly updated data source, such as the number of US job postings on Indeed.
(2) The questions are not sufficiently interesting or useful
We would like more dataset questions where research and judgement can lead to useful new predictions. ForecastBench’s current dataset questions include Amazon and Walmart product prices. For example:
Will the listed price in USD for a new ‘Hamilton Beach 25361MN Indoor Searing Grill with Viewing Window’, sold by Amazon.com, be higher on 2027-04-23 than on 2026-10-25?
It also asks whether a particular flight’s departure delay on a given date will exceed its recent median. We would expect limited gains from research and judgement on these questions. Good Judgment raises a similar concern: performance on these questions may not transfer to the novel risks and decisions that forecasting is meant to inform.
We welcome the retirement of weather and vaccine questions from new rounds, although they still contribute to the leaderboard.
(3) Simple historical baselines
As with market questions, the benchmark should measure what agents add to a strong baseline. Weather models and climatology provide one for temperature questions. For vaccine development, even a simple base rate can be a strong forecast: of 3,930 resolved forecasts, 99.95% resolved as “No”.5
Motivated by similar patterns for other question types, we tested a naive base rate heuristic which doesn’t use LLMs. For each question type and forecast horizon, we estimated the share of earlier questions that resolved Yes, using only outcomes known before each round.
Across the same 14 question sets, one continuing entry scored equivalent to tied 266th of 464 on the dataset leaderboard. Using the submission techniques discussed in section 6 gives a best score equivalent to tied 65th, still without using a single token.6
We would favour questions with more scope for research and judgement to improve beyond base rates. We’ve found that disagreement between models can reveal useful failures. ForecastBench could test this approach by running an initial set of forecasting agents and examining where they disagree.
(4) The outcomes are correlated
As others have noted, correlated outcomes provide less independent evidence of forecasting skill than the question count suggests. For example, a heatwave can make temperature questions at many nearby weather stations resolve the same way. Across 1,246 resolved seven-day temperature forecasts at 36 stations in mainland France, we measured an outcome correlation of 0.49.7
ForecastBench’s analysis does not account for these correlations, so its confidence intervals may be too narrow when forecast errors for related questions and events move together. We would reduce the weight of highly correlated question families, as per Metaculus’s scoring policy, and reduce the number of highly correlated questions in the benchmark.
Problems across all questions
(5) Binary questions are less informative
ForecastBench currently includes only binary questions, whose outcomes carry at most one bit of information. We welcome FRI’s announced plans to add quantile forecasts, which would capture more detail about numerical outcomes. We would also welcome categorical and date questions.
Metaculus and Crucible already support numerical forecasts, while Good Judgment Open and Polymarket split questions into multiple outcome ranges.
(6) Selecting the best entry can overstate performance
Organisations can enter up to 3 new models per round (rounds take place every 2 weeks). The October 25 leaderboard contains 18 entries from Torchcast and 22 from Google DeepMind.
Ranking entries separately can artificially boost a team’s leaderboard position. Using a new submission identity each round for the copied public forecasts, instead of one continuing identity, raises the highest rank from the equivalent of tied 76th to tied 23rd on the market leaderboard.
Submitting three fresh entries per round—one copying the public forecasts and two applying fixed shifts towards Yes and No—raises that to tied 12th, all without using an LLM.8
FutureSearch describes a similar approach for the 25 October round: with so many market questions about the midterms, it submitted two variants shifting forecasts towards Democrats or Republicans, arguing that one is likely to score disproportionately well.
We suggest limiting each team to one primary submission per round and restricting how often organisations can change the identity of their primary system.
Wrapping up
Writing good forecasting questions is hard. We appreciate ForecastBench’s open infrastructure and welcome the improvements already being made. We would like to see the further changes outlined here, and would participate again if it better measured the ability to make useful new predictions.
We started with all rounds with scoring data available, and retained those with the resolved outcomes available to estimate base rates for every dataset question type and horizon up to 180 days. This left 14 rounds, March 1–August 30, 2026. The base rate heuristic uses only outcomes publicly available before that round.
We use ForecastBench's published question effects for difficulty adjustment and compare each entry separately with 463 published category scores.
All 1,690 resolved market forecasts are included. We use historical quotes up to 23:55 UTC on submission day; INFER uses earlier quotes confirmed available before submission. The approximate 95% interval is [70.48, 72.27]. It ignores correlations between questions; overlapping intervals do not imply equal performance.
Counts cover resolved forecasts from July 2024 - September 2026 sets, as of October 25, 2026. Each round and horizon counts separately.
The heuristic covers all 27,959 issued dataset forecasts, including 10,504 resolved. 50% is used where no history is available.
The correlation uses 1,246 seven-day outcomes across 36 mainland stations and 37 target dates. We weigh each date equally, then each station pair within it. This measures correlation between outcomes, not forecast errors.
Each round uses unchanged forecasts and fixed log-odds shifts of −0.5 and +0.5, turning 50% into 37.8% and 62.2%. Each variant receives a fresh identity each round.