Backtesting against Polymarket

Mantic is better calibrated and diversifying to the market

Prediction markets are among the most powerful forecasting mechanisms available today.¹ In the 2024 US presidential election, for example, Polymarket, one of the world’s largest prediction markets, outperformed pollsters and pundits.

The majority of trading volume on Polymarket is concentrated in sports and crypto, but there are also markets on more sober themes such as geopolitics, national and regional elections, central bank decisions, technological developments, and armed conflict. One example is when traffic through the Strait of Hormuz returns to normal.

We’re often asked how Mantic’s accuracy compares to Polymarket. To investigate this, we don’t need Mantic to make trades — instead, we can treat Polymarket’s midpoint price (i.e. what you see on their website) as the Polymarket prediction and compare against Mantic on standard forecasting metrics like Brier score.

The experiment

We have built our prediction engine to be backtestable, meaning it can be run today as though it were some date in the past. For example, we can run our system as though it were April and ask ‘When will Keir Starmer stop being Prime Minister?’

We would have said: “An elevated risk (43%) that Keir Starmer will cease to be Prime Minister by June 30, 2026, with heavy Labour losses expected at the May 7 local and devolved elections and a worsening Mandelson vetting scandal as the likeliest triggers.”

We take Polymarket prices at 10,073 observations from 4,139 markets over 817 resolved events between February and August 2026, sampling at volume snapshots set at open, $1k, $10k, $100k, $1M, and $10M volume, and ran the Mantic system to compare forecasts.²

This post includes five summary findings. Reach out for the full report.

Mantic is significantly more accurate than Polymarket up to $100k of traded volume

We are significantly more accurate³ than Polymarket until a question reaches $100k of traded volume across its markets. At open, we’re closer to the outcome on 84% of markets; our win rate is still 72% at $1k and 62% at $10k. There is not a significant difference in accuracy from $100k but our win rate stays above 50%.

Mantic is better calibrated

Calibration measures how well a forecaster’s odds actually match up with how often things happen.⁴ We’re significantly better calibrated than Polymarket over questions with at least $1k traded across their markets, with an expected calibration error⁵ (ECE) of 0.022 against Polymarket’s 0.058, where lower is better.

Mantic’s edge is largest where price discovery is limited, and grows when it disagrees with the market

Our advantage is strongest for markets with lower volume (top left), lower liquidity (bottom left), and higher bid-ask spread (top right), i.e. where price discovery is limited. We also find the more we disagree, the bigger our advantage (bottom right).

Taking positions in the direction of Mantic’s disagreement yields positive payoffs

Taking a one-share position in the direction of our disagreement and holding until resolution yields +15 percentage points (pp) on average at market prices, or +3.8 pp when limited only to disagreements outside the spread and entering at the bid or ask. Ordered by market volume (left), the cumulative payoff rises at every volume level. By snapshot (right), the payoff is significantly positive through the $100k snapshot.⁶

Mantic’s forecasts are diversifying to market prices even when its accuracy advantage is no longer significant

What happens if we combine the two? We can mix our forecasts in an ensemble that weights the Mantic prediction with the Polymarket price, choosing the weight that gives the best accuracy.⁷ At open and $1k, the best ensemble ignores the Polymarket price entirely, and at $10k it puts 96% of the weight on Mantic. At $100k, where the two are equally accurate on their own, the weight splits roughly evenly and the ensemble beats both on held-out events.

The scale of AI forecasting

It’s very hard to durably out-perform highly liquid markets over the long-term (but we’ll try!). Either way, what’s most exciting is the scale of AI forecasting. We’re extending PM-level accuracy to all important issues, free from constraints. Come join the mission!

1. A prediction market is an exchange where traders buy and sell contracts that pay out depending on whether real-world events occur. Consider a contract that pays out $1 if some event happens this weekend. How much would you pay for it? If you think the event has a 30% chance of happening, you’d be willing to pay up to 30¢. The market price, set by many traders, can therefore be read as a collective probability estimate.

2. Our forecaster sees the Polymarket price as part of its research, along with market data such as traded volume and liquidity, just as it would see any other source. As with everything in a backtest, it only sees them as they stood at the moment of the forecast. It treats the price as evidence and it’s free to disagree. The results show that when Mantic disagrees, it is often right.

3. Accuracy is measured using the Brier score. For binary predictions, like those on Polymarket, the Brier score is equivalent to the mean squared error. This is a proper scoring rule, meaning that you expect to maximise your score by faithfully reporting your probability estimate i.e. if you think something has a 10% chance of happening, you expect to score highest by saying 10% and not 5% or 20% etc.

4. The likelihood a forecaster gives to some event is their forecast probability. The rate at which things actually happen is the empirical probability. For a well-calibrated forecaster, over many events we expect these numbers to be the same; in the aggregate, things happen as often as they think they will. So, if they gave a forecast of 30% to 100 independent events, we’d expect to see about 30 of those events actually happen.

5. The expected calibration error is calculated by taking many resolved forecasts, binning them into groups along the number line, finding the difference between the empirical probability and the forecast probability for the group, and averaging these across the bins. In terms of the plots above, this is how much the points stray from the identity at 45 degrees. A lower score is better and perfect calibration produces a score of 0.

6. This is a scoring exercise, not a trading simulation: it doesn't account for position sizing, market impact or fees.

7. The ensemble forecast is w × Mantic + (1 − w) × Polymarket, with w between 0 and 1 chosen to minimise the Brier score. We score it on held-out events: each event is scored using a weight fitted on all the other events.