Details
Loading…
How the run worked
-
Gather the questions dataset
100 of the top markets by Open Interest in Polymarket over the next 2 years, balanced for different midpoint prices. We form pairs using the sorted dataset by midpoint price descending, with the goal of having question pairs of comparable odds/difficulty.
-
Snapshot the price, then research
Take a snapshot of the market's current midpoint price and generate a research with a custom deep research agent that avoids including explicit prediction market odds.
-
Run inference and log it
Run inference over the entire dataset using OpenRouter and log the results both on-chain and on a local database.
Methodology
- Prompt iterations
- How many times the exact same individual forecast is requested. Multiple forecasts with the same exact inputs allow us to measure odds spread.
- Negated questions
- We ask each model the same number of times for the probability the market resolves to "Yes" as we ask for "No". This lets us observe the impact of 1 aspect of the question's framing that has 0 logical effect and should produce the exact same but inverted results, eg Yes=40% should always lead the model to forecast No=60%.
- Pairwise judgments
- Given two options, A or B, which is likelier? We present each model with 4 variations of each pair of markets: (A, B), (¬A, B), (A, ¬B), (¬A, ¬B). Each pair is asked in four ways for each model so we can observe the model's logical consistency when changing how the question is framed, both in terms of negating the question and when asking for a standalone forecast of each question.
The five measures
Odds and negated spread are absolute distances in probability: a 3.0% odds spread means a model's repeated answers to one question sat 3 points of probability apart on average. The other tree measures are shares of judgments: a 12.0% pairwise disagreement means 12 of every 100 comparable judgments came back different.
| Measure | What it asks, and how | Field | Best |
|---|
Pair combinations
The pairs (A vs B) and (not A vs not B) are directly usable to test logical consistency with standalone forecasts. If model says p(A) > p(B) on standalone forecasts, then it must also choose A over B and not-A over not-B. But (A vs not-B) and (not-A vs B) are special: p(not_B) can be higher than p(A) but p(A) > p(B). Example: p(A) = 0.2, p(B) = 0.1, then p(not_B) = 0.9, which is higher than p(A) = 0.2, but p(A) > p(B). That's why for "Negation disagreement" we need to compare these two sets of pairs separately against each other.
| Asked | Correct choice assuming higher odds for A |
|---|---|
| A vs B | A |
| not-A vs not-B | B |
| A vs not-B | Depends on the odds for A and B |
| not-A vs B | Depends on the odds for A and B |
Questions that went unanswered
Not every request came back with a usable number. Those attempts are absent from the analysis. A repeat set holding fewer than two surviving forecasts is skipped by odds spread instead of scoring zero, and a question answered in only one wording is skipped by negated spread.
Unanswerable requests, by model
Transparency
The entire code for both the benchmark and this site is available at https://github.com/fpluis/noisebench.
The code for the onchain contract where forecasts are logged is available at https://github.com/fpluis/forecast-registry.
The onchain forecasts for this benchmark run are publicly logged and can be seen at https://basescan.org/address/0x0189d7d85bc94421e1a4e51dfb40ffcea4f8d9af. Note that due to a misconfiguration error, all the negated forecasts had to be repeated so they will appear twice, here we report only the second, correct attempt.