WorksComputer Science
ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents
Abstract
Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
Introduction
Forecasting tournaments have shown that trained forecasters and aggregated crowds can predict world events well [21, 31], and online prediction markets such as Polymarket1 and Kalshi2 let participants trade contracts on such events, with prices that move as news appears on the Web. Large language model (LLM) agents now take on a similar role [12, 39], searching the Web, reasoning over what they find, and revising their probability as new information arrives. Augmented with retrieval [18], such systems approach the accuracy of human forecasting crowds [11, 27], which calls for evaluating them under comparable conditions. Crucially, a forecast depends on both the evidence behind it and the model that reads it. Existing benchmarks follow two routes: ❶ live benchmarks collect forecasts on open questions and score them after resolution [15, 36, 40], and ❷ retrospective benchmarks replay resolved questions with historical evidence [10, 34, 38, 41], summarized in Table 1.
Despite these advances, evaluating forecasting agents still faces four challenges. (i) Leaky reconstruction: retrospective evaluation must recover the evidence available at time . Date-filtered retrieval can admit content updated after [8, 22], and even historical page snapshots can be selected or ranked using today’s search index [24, 34]. Preserving old content alone therefore does not preserve what a search service would have returned then. (ii) Entangled retrieval and reasoning: when benchmarks let each system gather its own evidence, a score mixes what it found with how it reasoned [5, 30]. Shared contexts can isolate model differences [36], but reproducible comparisons also require keeping the evidence fixed across later runs. (iii) Uncontrolled evidence dynamics: evidence evolves as an event approaches, and how agents revise their beliefs as it does [39] can be compared only if they experience the same evidence trajectory. Repeated live forecasts alone do not supply a reusable trajectory on which to vary memory while holding incoming evidence fixed. (iv) Search-source variation: the search provider changes what an agent sees, so its effect must be separated from that of the model. Provider comparisons on static factual tasks [28, 29] and time-gated search tests on resolved questions [24] motivate a further requirement: comparing sources alongside models and memory on the same prospectively recorded forecasting trajectories.
To address these challenges, we introduce ForecastGym, a benchmark that treats evidence as a controlled experimental variable, as illustrated in Figure 1. Its key insight is simple: what a search API returns while a question is still open was knowable at that time by construction. ForecastGym enables retrospective experimentation under prospectively recorded information conditions, preserving both the evidence content and the search provider’s selection before resolution. Figure 2 shows its two phases. In Phase 1, Live Capture, ForecastGym queries four commercial web search APIs every two days for unresolved prediction-market questions and stores each provider’s results as a time-stamped evidence state, as detailed in § 3.3. Providers share an initial query plan and follow-up policy, and their results are stored separately. In Phase 2, Controlled Replay, agents follow the protocol in § 3.4 to read the frozen states offline in chronological order once outcomes are known. Replay isolates forecasting and belief revision: for a given provider, every run follows the same evidence trajectory. The model, memory, and search source can then be varied one at a time while the other settings remain fixed, allowing score differences to be attributed to the component changed.
Figure 1. Design principles of ForecastGym.
The first release contains 150 questions, 614 evidence states, each holding one question’s evidence at one collection time, and 131k search results. The same queries rerun after settlement reveal the outcome for up to 92% of the questions despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnosis of high-gap states attributes nearly three times as much of their gap to missed public information as to misreading. Switching the search API moves the Brier score by up to 0.043, more than the spread across models, while memory mainly steadies the forecasts. The evidence a forecaster sees can matter as much as the model that reads it.
Our contributions are: (1) Live: a reusable collection protocol and pipeline, which records search results while questions are unresolved and demonstrates leakage under the tested post-hoc date filters; (2) Controlled: an evaluation protocol on fixed evidence, which separates model comparison from evidence acquisition and establishes model and market baselines; (3) Replayable: recorded evidence trajectories and sequential replay, which support controlled memory experiments and show that memory mainly steadies forecasts; and (4) Source-Aware: a controlled comparison of four commercial search APIs, showing that source differences can exceed model differences in accuracy.
Related Work
Forecasting benchmarks. Early datasets framed event forecasting over time-restricted text: ForecastQA [14] casts it as question answering, and Autocast [41] pairs forecasting-tournament questions with a date-indexed news corpus. Subsequent systems retrieve news, reason with tools, or aggregate several models to produce probabilistic forecasts [11, 12, 27]. Live benchmarks, including ForecastBench [15], FutureX [40], and Prophet Arena [36], evaluate predictions submitted before outcomes are known, and refresh their questions to avoid contamination, as LiveBench [33] does for general tasks. Along the same line, Daily Oracle [6] generates questions continuously from daily news, and MIRAI [37] evaluates agents that query event databases. Other work scores forecasters before resolution by testing whether their predictions on related questions agree [23], or scores financial forecasts under trading costs [13]. Retrospective benchmarks instead provide reusable historical environments: Bench to the Future [34] supplies offline web corpora, and Hindcast [38] replays prediction-market questions against a frozen Reddit archive. Such evaluations must account for both model knowledge of resolved outcomes and the temporal validity of retrieved evidence [8, 22]. Live evaluation does not by itself provide retrieved evidence for later controlled comparisons, while retrospective evaluation must preserve both historical content and the search service’s selection of it. For example, BTF freezes page content but uses live Google search over its stored pages, leaving a possible change in ranking [34]. ForecastGym combines prospective collection with offline replay, preserving the results exactly as each provider returned them before resolution.
Search evidence for agents. Because the knowledge of language models goes stale [17, 20], questions about the current world are answered better with search, as RealTime QA [16] and FreshLLMs [32] show, which makes the retrieved evidence part of what is evaluated. Search benchmarks therefore fix it: BrowseComp-Plus [4] replaces live search with a curated corpus to separate retrieval from reasoning, and Deep Research Bench [1] freezes scraped pages so that agents can be compared offline. Within forecasting, Dr-CiK [30] tests retrieval of supporting context for time-series predictions, and WorldReasoner [5] scores the evidence an event forecast cites. The search provider adds its own variation: OpenHands-Versa [29] succeeds on GAIA at different rates with different providers, and Selvam and Ghosh [28] find similar accuracy but unequal evidence across providers on SealQA-Hard. Prophet Arena audits time-gated search with prospective captures, but acknowledges ranking leakage and fallible leakage detection [24]. It exports omit page text, precluding exact offline replay from those exports. Among the benchmarks in Table 1, ForecastGym alone combines prospectively captured, provider-specific evidence trajectories with controlled offline comparisons of models and memory.
Table 1. Forecasting benchmarks by the principles of Figure 1. Live: open questions at collection. Controlled: reusable fixed evidence. Replayable: multiple states and controlled memory studies. Source-aware: controlled provider comparisons. ◦: partial.
Belief revision and memory. EvolveCast [39] compares forecasts revised after news updates with human revisions, for both individual and accumulated news, although its question–news associations are reconstructed retrospectively. FutureSim [10] replays a chronological news stream to study long-horizon adaptation, search, and memory. ForecastCompass [3] instead transfers experience across forecasting tasks through adaptive factor and reasoning memories, while LongMemEval [35] examines retention and knowledge updates in extended conversations. Yet the forecasting studies among them either rebuild the evidence after the fact or transfer memory across tasks, so none isolates memory on evidence captured before resolution. ForecastGym addresses this gap by holding each provider’s evidence trajectory for the same question fixed and varying only what the agent retains, so that differences in forecasts can be attributed to memory alone.
The ForecastGym Benchmark
ForecastGym is built to answer one question: how does a forecast change when a single component of the forecaster changes? Section 3.1 formalizes this question and the requirements it places on the evidence, and the remaining subsections show how the two phases of Figure 2 meet them.
Figure 2. The ForecastGym lifecycle described in Section 3. Phase 1 records what four search APIs return every two days for open questions, with each result stored with its source, query, and retrieval time. Phase 2 replays the frozen states to a forecasting agent behind three safeguards and changes one argument of Eq. (1) at a time. Orange cards follow one question of the first batch.
Problem Formulation
A forecasting agent that searches the Web combines three components: a search source that acquires evidence, a memory that carries it across time, and a model that reasons over it. At time , an agent with search provider forecasts a binary question as
where is the evidence that provider returns at time , what the agent carries from earlier times, and the model.
Definition 3.1 (Evidence state). Let be the set of search providers. The pair , with in the set of times at which evidence for is collected, is a state, to which each provider contributes the evidence . A benchmark instance is the tuple
in which the outcome is added only after resolution.
ForecastGym evaluates forecasters through controlled comparisons, each of which changes a single argument of Eq. (1). Let be the score of an agent on a batch, such as its mean Brier score over questions as defined in Section 5. To compare models and while holding the search source and memory strategy fixed, we measure
and the effects of the search source and of the memory are defined in the same way.
Estimating Eq. (3) places three requirements on the evidence. R1. Time validity: every item in was knowable at time , since otherwise the effect is measured on leaked answers. R2. Source isolation: provider comparisons fix collection times, initial queries, and the follow-up rule and budget. R3. Invariance across runs: both terms of Eq. (3) read the same evidence. Live evaluation does not guarantee R3, while retrospective search must guard R1 against later content and ranking signals. Sections 3.2 and 3.3 show how Live Capture meets R1 and R2; Controlled Replay meets R3 by fixing the evidence across runs, as described in Section 3.4.
Temporal Validity by Construction
Writing for the content available on the Web up to time , requirement R1 asks that . Retrospective benchmarks search after resolution, at some time , and keep the results dated before ,
where denotes the results of queries on the Web . This set errs in both directions. A page without a date is dropped even if it was knowable at , and a page whose content was replaced after , such as a game preview later rewritten into a box score, keeps its early date and passes the filter with the outcome inside. ForecastGym instead issues the queries at time itself and records
so R1 holds by construction, whatever dates the pages carry.
Retrieval time thus replaces publication date as the bound on what was knowable, and Sections 4 and 6.1 measure how far Eq. (4) and Eq. (5) diverge in practice. Live capture also fixes the provider’s selection at : even a search over historical page snapshots can otherwise use a later index or ranking [24, 34]. Validity concerns only when an item became knowable, not whether it is safe to show, so Controlled Replay adds safeguards.
Live Capture
We use Polymarket and Kalshi to ground evaluation in real prediction markets, with tradable questions, observed prices as an external baseline, and platform settlements as outcomes.
Rolling collection protocol. Answers that already exist can be memorized, and a fixed question set falls behind every new model. ForecastGym supplies a common protocol for users to collect fresh binary questions closing within 14 days. Batch size and domain quotas are configurable; this paper evaluates one budget-limited batch of 150 questions. A model is evaluated only on batches opened after its release. To keep questions that are still uncertain, settle unambiguously, and leave time to revise a forecast, we drop markets priced below 10% or above 90% or closing within 48 hours, and an LLM reviewer removes questions with vague rules or hard-to-verify outcomes. Selection then balances horizon, domain, and trading activity, and keeps at most one question per real-world event, following the rules in Appendix § 9.
Evidence trajectories. ForecastGym collects each question’s evidence every two days until the forecast cutoff, the earliest of its deadline, the start of the event, and its settlement, so the states follow the evidence as the event approaches but never report the event itself. Every result is stored with its source, query, and retrieval time, and a late collection is marked as missed rather than reconstructed, so Eq. (5) holds for every stored item.
Source comparison under a shared query policy. At each collection time ForecastGym queries four web search APIs that stand for different ways of searching. Serper returns Google’s ranked results, Tavily and Parallel are built for LLM agents and return page content or excerpts chosen for the query, and Exa searches its own index by meaning rather than by keywords.3 To meet R2, an LLM writes a shared initial plan of up to four queries per collection time. A resolution query identifies the rule and source that will settle the question; a current-evidence query retrieves the latest news. A base-rate query asks how often similar events occur, and a counterevidence query seeks reasons the likely outcome may fail. Where a provider’s results miss one of these needs, the LLM sends that provider up to two follow-up queries. At each state, queries are planned during capture, independently of the forecasting models, so one recording serves every model.
Freezing the record. Outcomes are read from the platform that posed the question, and disputed or ambiguous ones are excluded. A completed batch is frozen as a versioned snapshot, and unsettled questions are scored later without changing their evidence.
Controlled Replay
Controlled Replay evaluates forecasters on the frozen record offline in Phase 2 of Figure 2, much as a trader backtests a strategy on recorded market data rather than on prices reconstructed afterwards. The agent sees one provider’s states of a question in order, as if it were forecasting live, and forecasts each state once without calling any live service, so every run reads the same evidence and R3 holds.
Replay safeguards. A valid state can still give a model a shortcut, so three safeguards filter it before it is shown. Items dated later than the state, such as previews dated by an upcoming event, are hidden, and so are items that show the question’s own market price, which the model could otherwise copy. Near-duplicates are merged, so that repeated coverage does not count as more evidence, and everything else is shown regardless of search rank.
Memory as an explicit state. Following retrieval-augmented forecasters [11], a lightweight agent reads the question, its resolution criteria, the current time, and the evidence , and returns a probability , the items it relied on, and a short rationale . Since a state shows only what was retrieved at that time, the agent needs memory, and keeping it in a growing context would tie it to the model’s input. ForecastGym instead passes memory between states as an explicit object that can be replaced like the other arguments of Eq. (1). In the main condition it holds the previous forecast and rationale and notes that the agent rewrites at every state, covering the key evidence with its dates, the factors that matter, and how it has been updating, in the spirit of ForecastCompass [3]:
At the first state, the previous forecast is absent and there are no notes from earlier states.
Single-factor interventions. Every comparison in Section 6 starts from this main condition and changes one argument of Eq. (1), the model, the provider, the evidence, or the memory, as Table 2 lists. Previous only retains the preceding forecast and rationale but drops the notes; No memory carries none of these forward. Two memory variants replace memory with all evidence so far, , read in full or searched with the keyword-based ranking function BM25 [26]. Beyond these interventions, a trading mode shows the agent the market’s quotes and its own position and lets it buy, hold, or close at each state.
| Argument | Variant | Evidence | Memory |
| Main | Main | ||
| Model | Four LLMs | as Main | as Main |
| Source | Four providers | as Main | |
| Evidence | (i) Closed-book | none, last state | none |
| Evidence | (ii) Final only | , last state | none |
| Memory | (iii) Previous only | ||
| Memory | (iv) No memory | none | |
| Memory | (v) All so far | none | |
| Memory | (vi) BM25 search | BM25 over | none |
Table 2. Single-factor interventions on Eq. (1). Each row changes one argument of the main condition and keeps the others as in Main. Evidence: what the agent reads at state . Memory: what it carries from the state before.
The First ForecastGym Release
The first batch opened on 14 September 2026 and was collected for 14 days. It contains 150 questions, 614 evidence states and 131k search results, with two to seven states per question. The batch targets forecasting on real prediction-market questions, with domain quotas broadening coverage across six domains within the platforms’ supply. One question was cancelled and five await settlement, leaving 144 scored questions. Their initial market prices score 0.215 Brier, against 0.250 for a constant forecast of 0.5, consistent with uncertainty at collection. Appendix A reports the full statistics and scope.
Figure 3. One question of the first batch, replayed by Gemini 3.8 Flash on Tavily’s evidence. The market price is never shown to the model.
Before evaluating any forecaster, we use the batch to test the three premises behind the design of § 3: that the search source shapes the evidence, that the evidence evolves, and that publication dates cannot bound what was knowable.
Search providers largely disagree. As Figure 4 shows, for the same question and time any two providers’ URL sets overlap little, and four in five pooled URLs come from exactly one provider. Even under this shared policy, provider choice changes most of the returned URLs.
Figure 4. The four providers return largely different pages under the shared query policy, averaged over all states. Jaccard overlap: shared URLs over all URLs of the pair.
Evidence keeps changing. At a question’s second state, 71% of the pooled URLs are new, and at every later state more than 40% still are. Over its lifetime a question accumulates 374 distinct URLs on average, about three times as many as a single state holds, so a single snapshot would miss much of what became knowable.
Page dates are unreliable. The date filter of Eq. (4) errs in two directions, and the batch shows that both are common. Most pages carry no date and would be dropped: only 36.8% of our documents have one. Some dates are also wrong: 6.7% of dated documents are dated after they were retrieved, typically previews and box-score pages dated by the upcoming event, whose content is later replaced by the outcome. This matches the leakage reported for date-filtered retrieval [8], whereas Eq. (5) bounds every document by its retrieval time.
Experimental Setup
Following Figure 1, RQ1 tests Live capture; RQ2 uses Controlled evidence to compare models and diagnose errors; RQ3 studies memory and belief revision through Replayable trajectories; and RQ4 tests Source-Aware provider comparisons.
RQ1: Can post-hoc search replace live capture? We sample 60 settled questions in proportion to the batch’s domains. For each question’s last replay state , we rerun its two initial queries for current evidence and counterevidence after settlement, once raw and once with each provider’s filter for pages published before . An outcome-aware LLM judge marks an item as leaking if its content carries information from after , whatever the page shows. We report Leaking, the share of leaking items, and Revealed, the share of questions whose leaking items all but state the outcome. Gemini 3.8 Flash forecasts from the filtered and live results of the same queries; Appendix B.3 gives the filter settings.
RQ2: How do models compare with the market, and why? All four models forecast every state in the main condition with Tavily’s evidence and once more closed-book. We compare their scores with the market price at the same states and when collection began. Under the blind rule, we buy $1 of the side a model favors whenever its forecast differs from the quote by more than 0.05; in the trading mode, Gemini 3.8 Flash makes its own trades with the quotes in view. Trades fill at the reduced price plus or minus half the bid–ask spread, pay Kalshi’s fee, and are scored by profit per dollar staked, with details in Appendix B.4. We use outcome-aware Gemini 3.8 Flash for two distinct diagnostic labeling tasks. Evidence-direction labeling classifies each state’s evidence as pointing toward the outcome, against it, or both ways. For states trailing the market by more than 0.1 Brier, gap attribution identifies the fact that would have improved the forecast and checks the displayed evidence, items removed by the safeguards, and other providers’ results, as detailed in Appendix § 10.§ 5.
RQ3: Do forecasts sharpen, and does memory help? We track scores by days until the event and ablate memory for Gemini 3.8 Flash with Tavily’s evidence. For each pair of consecutive states, we compare the forecast’s absolute probability change and direction with the market’s move over the same interval, as detailed in Appendix § 12.§ 5.
RQ4: How much does the search source matter? Gemini 3.8 Flash replays every state with each provider’s evidence. To see whether the provider also reorders the models, all four forecast the last state without memory from Tavily’s evidence and from Exa’s, whose results overlap least with Tavily’s.
Models. We evaluate four LLMs from four developers, GPT-5.6 Terra from OpenAI, Claude Sonnet 5 from Anthropic, Gemini 3.8 Flash from Google, and the open-weight DeepSeek-V4.1-Flash from DeepSeek-AI [7]. All were released before the batch opened, as listed in Appendix § 10.§ 1, so none can have seen its outcomes. All receive the same prompt and settings given in Appendix § 11. Gemini 3.8 Flash, chosen for its speed and cost, runs the variants beyond the main comparison. GPT-6.1 Sol, released after the batch resolved, is not ranked but checks whether a newer model closes the gap and whether knowledge of the outcomes shows in its closed-book forecasts.
Choice of providers. Tavily is the main provider, chosen before any forecast from properties measured without outcomes. At collection time, the query-planning LLM rated its results highest for covering the query purposes, and they held the largest share of items relevant to the question, although it reports no publication dates, as Appendix § 10.§ 2 details.
Metrics. Our primary metric is the Brier score , the squared error of a forecast probability. It is strictly proper, so a forecaster scores best in expectation by reporting its true belief [2, 9], and a constant forecast of 0.5 scores 0.25. To score a whole trajectory, the time-averaged Brier score (TBS) averages it over the scored states of a settled question ,
and the mean TBS or the mean last-state Brier score over questions serves as the score of Eq. (3). The last state is the latest before the forecast cutoff. Accuracy is the share of last-state forecasts on the correct side of 0.5, counting a forecast of exactly 0.5 as half. The market price at each state, the platform’s latest price at or before the replay time, serves as an external baseline, since traders are not limited to the recorded evidence. Differences are paired by question, averaging over a question’s states first, and come with 95% bootstrap intervals obtained by resampling questions. We call a difference significant when its interval excludes zero, and compare model rankings with Kendall’s .
Results and Analysis
Sections 6.1, 6.2, 6.3, and 6.4 address RQ1–RQ4, respectively.
The live snapshot is not at zero because its flagged cases are mostly results already public at , such as a review score already above the threshold, or LLM-judge false alarms, such as an article about an earlier game.
Parallel has no filtered run: its filter only keeps pages published after a date.
Post-hoc Search Leaks the Outcome
Date filters reduce leakage but do not stop it. As Table 3 shows, the date-filtered evidence of Eq. (4) still leaks for every provider, and each fails in its own way. Tavily attaches no dates to its results, so its filter cannot be audited at all. Exa’s filter removes every result it dates after but passes the 61% of results it leaves undated, and Serper’s still returns 12% of results dated after . The leaking items are mostly final scores, earnings releases, and product changelogs, while the judge flags few live results, providing a check on false alarms.
| Results | Leaking raw | Leaking filtered | Revealed raw | Revealed filtered | Brier filtered |
| Live snapshot, Tavily* | 3% | 10% | 0.224 | ||
| Tavily | 42% | 18% | 87% | 60% | 0.137* |
| Exa | 31% | 27% | 90% | 92% | 0.080* |
| Serper | 32% | 17% | 73% | 43% | 0.194 |
| Parallel | 37% | n/a | 90% | n/a | – |
Table 3. Post-hoc search on 60 settled questions, with raw and date-filtered results. *Significantly below the live snapshot.
The leaks inflate the scores. Forecasting from Tavily’s or Exa’s filtered results, Gemini scores 0.137 and 0.080 Brier, far better than any model or even the market does on live evidence in Table 4. The tested date filters thus leave post-hoc evidence temporally unreliable, motivating capture before resolution, as in the remaining experiments.
| Forecaster | Last-state Brier↓ | Accuracy↑ | Closed-book↓ | Gap |
| GPT-5.6 Terra | 0.217 | 0.642 | 0.260 | +0.022 |
| Claude Sonnet 5 | 0.244 | 0.597 | 0.279 | +0.046 |
| Gemini 3.8 Flash | 0.218 | 0.628 | 0.300 | +0.028 |
| DeepSeek-V4.1-Flash | 0.220 | 0.660 | 0.258 | +0.028 |
| GPT-6.1 Sol† | 0.208 | 0.653 | 0.266 | +0.018 |
| Market | 0.176 | 0.719 |
Table 4. Main condition with Tavily’s evidence, at each question’s last state. Gap: to the market over all states, significant for all. †Released after the batch resolved.
Models Trail the Market
Model Differences Are Small
Table 4 gives the results, and the closed-book scores show what the evidence adds, which can be negative [30, 38].
On the same evidence, the models are hard to tell apart. Score differences among GPT-5.6 Terra, Gemini 3.8 Flash, and DeepSeek-V4.1-Flash are not significant, and accuracy orders them differently than the Brier score does. Only Claude Sonnet 5 falls clearly behind, through overconfidence at the low end: its forecasts near 0.05 came true 38% of the time, and in fig:10c it took undated pages about a 2018 win for the upcoming match. All models lean on the newest evidence, as 92–95% of the items they cite were first retrieved at the latest state. GPT-6.1 Sol, released after every question had settled, is within noise of the top three, and its closed-book score shows no sign of knowing the outcomes.
The evidence is what makes these forecasts useful. Without it, no model beats a constant forecast of 0.5, so background knowledge does not suffice, and with it every model improves, significantly so for GPT-5.6 Terra and Gemini.
Yet no model beats the market. Every model trails the market at the last state and over the full trajectory. Excluding the 216 states priced at a bid–ask midpoint leaves the gap unchanged. Nor does combining models close it: their mean is no better than the best single model, and even a blend weighted toward the market falls short of the market alone, as shown in Appendix D.1.
The tested trading strategies yield no significant profit. A forecaster that trails the market could still profit if it knew when the market was wrong, as tested in Appendix D.2. Under the blind rule, every model loses money when it sells at the next state, significantly for three of the four, since prices tend to move away from a model that disagrees with them, and holding to settlement only shrinks the loss. In the trading mode, Gemini keeps about half of its disagreement with the market, which improves its Brier score by 0.018 but leaves it 0.010 behind, and its trades make no significant profit. These comparisons favor reporting intervals and the market price beside model scores.
The Gap Diagnosis Points Mainly to Missing Information
The gap is concentrated outside standard sports contracts. Sports match winners, spreads, and totals make up more than half the states, and the models come within 0.004 Brier of the market on these questions. fig:5a shows that almost all of the gap lies in other domains and culture questions, such as album sales, review scores, and awards.
The models trail in resolution, not in calibration. A Murphy decomposition separates calibration, whether events forecast at 70% occur 70% of the time, from resolution, the ability to give different probabilities to questions that resolve differently. The mean of the four models is as well calibrated as the market and spreads its forecasts as widely, but its resolution is only about half the market’s. This shortfall in resolution is consistent with the missing-information diagnosis below.
The gap arises where the evidence is mixed or misleading. Evidence-direction labeling characterizes each state’s evidence, as detailed in Appendix D.3. Where the evidence points toward the outcome, the models slightly beat the market, and only 12–17% of each model’s forecasts on the wrong side of 0.5 fall on such evidence, against 22% for the market. The whole gap thus arises where the evidence is mixed or points the wrong way, and it is concentrated: the 19% of states in which the models trail the market by more than 0.1 Brier account for more than all of it.
Missing information costs more than misreading. For these states across all four question types, gap attribution assigns each state one of four causes in fig:5b. Misread: the models saw the fact but misweighted it, a small share. Every model read, for instance, that the Panama Canal’s caps allowed at most 226 transits in a week that needed more than 225, yet three put 0.43 to 0.60 on it, against the market’s 0.06. Not retrieved: the fact was public but missing from the evidence, mostly because another provider returned it or the query-planning LLM never searched for it. This is the largest cause. Outside indexed text: the signal lived in sales tracking, streaming counts, or data dashboards. For one album, every provider returned projections of at most 46,000 first-week units, while the market priced more than 50,000 at 0.99, and it opened with 61,000. Unknowable: nothing public pointed to the outcome, and the market itself was unsure. The judge identified no indexing-delay cases.
The judge attributes nearly three times as much of the gap in these states to missed public information as to misreading. These diagnoses can guide query design for future live batches; RQ3 and RQ4 examine memory and the captured sources.
Forecasts Do Not Sharpen, and Memory Steadies Them
Forecasts do not sharpen as the event approaches. As Figure 6 shows, the evidence keeps each model well below its closed-book score at every horizon, but the scores stay roughly flat as the event approaches, and the market leads every model throughout, consistent with the missing signals identified in § 7.1.2. The models instead converge on one another, disagreeing less in the last days than a week before, as shown in Appendix § 12.§ 4. Every model also scores better on 10–14-day than on 2–5-day questions, by 0.057 for Gemini, though with only 21 short-horizon questions this may reflect their composition.
Figure 6. Brier score by days until the event in the main condition with Tavily’s evidence. Dashed: closed-book. {sup}†Released after the batch resolved.
Memory steadies forecasts more than it improves them. Table 5 compares the main condition with the four memory variants of Table 2, each of which changes only in Eq. (1). Persistent notes give the best scores by small margins; only dropping all memory or using BM25 significantly worsens TBS, as shown in Appendix § 12.§ 5. Without memory, however, the forecasts jump from state to state. The mean update nearly doubles, exceeds the market’s own moves, and goes in the market’s direction only half the time. With notes, updates are smaller than the market’s and more often agree with it in direction, a mild under-reaction in line with earlier findings [39].
| Variant | TBS↓ | Last↓ | Update | Agree | Ratio |
| Main | 0.228 | 0.218 | 0.045 | 0.59 | 0.76 |
| Previous only | 0.232 | 0.221 | 0.049 | 0.59 | 0.83 |
| No memory | 0.237 | 0.230 | 0.084 | 0.50 | 1.24 |
| All so far | 0.232 | 0.227 | 0.059 | 0.49 | 0.91 |
| BM25 search | 0.241 | 0.249 | 0.075 | 0.46 | 1.10 |
Table 5. Memory variants of Gemini 3.8 Flash. Last: last-state Brier. Update: mean change between states. Agree: share in the market’s direction. Ratio: size relative to the market’s.
Short notes reduce the cost of rereading. Rereading all evidence scores about as well as keeping notes but reads twice the input, 19.5M tokens per replay against 9.5M. Searching the same pool with BM25 is the only alternative significantly worse in both metrics, and its forecasts differ from rereading by 0.057 on average, in line with the weakness of BM25 in controlled search benchmarks [4]. Steadiness has a cost when the notes carry a wrong reading forward: for Norges Bank’s September decision in Figure 10a, all four models drifted down as their notes stressed softer inflation, while the market rose and the bank hiked. Self-written notes offer a compact way to steady forecasts, but errors can persist, and none of the tested memory variants closes the gap to the market.
Figure 10. Three case questions, all resolved Yes. Forecasts at each state (main condition, Tavily) and the market in red on (a) Norges Bank’s hike, (b) a Metascore above 90, and (c) the Eagles beating the Titans.
Search-Source Differences Can Exceed Model Differences
The observed source spread exceeds the model spread. With the model fixed, switching the search API, the effect of Eq. (3), moves Gemini’s last-state Brier score by up to 0.043, more than the 0.027 that separates the four models in Table 4, as Table 6 shows. Most of this spread comes from Serper, significantly worse than Tavily, with the lowest share of relevant items in Appendix § 10.§ 2. The effect includes retrieval, adaptive follow-ups, and evidence formatting under the shared collection policy. Exa and Parallel stay close to Tavily, and none of the three leads at every horizon, consistent with the diagnosis in § 7.1.2, where another provider often held the missing fact.
| Provider | Last-state↓ | Accuracy↑ | TBS↓ | 0–1 | 4–7 |
| Tavily | 0.218 | 0.628 | 0.228 | 0.220 | 0.232 |
| Exa | 0.213 | 0.646 | 0.232 | 0.211 | 0.243 |
| Serper | 0.255* | 0.594 | 0.252* | 0.246 | 0.250 |
| Parallel | 0.212 | 0.639 | 0.229 | 0.209 | 0.246 |
Table 6. Gemini 3.8 Flash with each provider’s evidence. The last two columns give the Brier score 0–1 and 4–7 days before the event. *Worse than Tavily.
The observed model order changes with the source. Figure 7 shows different point-estimate rankings on Tavily’s and Exa’s evidence; with only four models, evidence for a systematic ranking effect is limited. All four score better with Exa, but only Claude Sonnet 5 improves significantly, bringing its score close to the others. Benchmarks should therefore fix or report each model’s search source; pooling providers may help, as diverse evidence does in multi-agent forecasting [19].
Figure 7. Last-state Brier scores and model ranks without memory. Hollow markers: Tavily; filled markers: Exa. Kendall’s across four models. ∗Only significant change. †Not ranked.
Conclusion
FORECASTGYM provides a reusable protocol for prospectively recording search evidence and replaying it to compare forecasting models, memory strategies, and search sources. The first batch shows that the tested date filters leave substantial leakage and that source differences can exceed model differences in accuracy. All tested models trail the market, and our diagnosis attributes more of the gap in high-gap states to missed public information than to misreading. Persistent notes steady forecasts with less input than rereading, but errors can carry forward as evidence changes.
These findings motivate improving both evidence acquisition and belief revision. Frozen batches support controlled comparisons on recorded evidence, while new query policies require fresh live collection. The shared collection protocol makes these studies repeatable on fresh questions as models and search policies evolve.
Batch Details
Table 7 summarizes the first batch. Selection balances horizon, domain, and activity within the two platforms’ supply. Candidate pools use bounded Polymarket sampling and exhaustive discovery within Kalshi’s two-week window; their sports-heavy mix is consistent with real platform activity [25]. The quotas expand domain coverage without imposing equal domain shares, so the selected batch is grounded in the platforms’ available questions. Our 14-day batch contains 150 questions, compared with 80 in FutureX’s contemporaneous weekly release for 16–22 September.4 Quotas cover six domains, namely sports and culture, economics, politics, international affairs, business, and science and technology, and three levels of trading activity, the lower, middle, and upper thirds of 24-hour volume within each platform and domain. The 576 states of the 144 settled questions are scored, and the 56 states with a failed search request are flagged. The cancelled question was settled at 50/50, five questions are still open, and one question that its platform had not settled a week after the event was settled from the official result.
| Questions | |
| Candidates screened / eligible | 6,754 / 279 |
| Candidates, Polymarket / Kalshi | 5,099 / 1,655 |
| Sports share, Polymarket / Kalshi | 95.5% / 92.2% |
| Selected (Kalshi / Polymarket) | 150 (104 / 46) |
| Horizon 2–5 / 6–9 / 10–14 days | 23 / 85 / 42 |
| Activity tier low / medium / high | 47 / 51 / 52 |
| Sports and culture / other domains | 109 / 41 |
| Settled to date (% resolved Yes) | 144 (46.5%) |
| Attribution questions / states | 49 / 111 |
| Collection | |
| Evidence states (per question) | 614 (2–7) |
| Search queries (shared first round) | 14,377 (9,816) |
| Search results | 131,542 |
| Unique URLs / documents | 47,581 / 102,419 |
| URLs per state: one provider / pooled | 39–45 / 133 |
| URLs per question, all states | 374 |
| Documents with page text | 68.8% |
| Documents with publication date | 36.8% |
Table 7. Statistics of the first batch. Document: a unique URL and content pair. Per-state and per-question counts are means.
Implementation Details
Models
Table 8 lists each model’s release date and reported training cutoff. The four main models were released before the batch opened on 14 September 2026, and GPT-6.1 Sol only after it resolved.
| Model | Released | Cutoff |
| GPT-5.6 Terra | 9 Jul 2026 | Feb 2026 |
| Claude Sonnet 5 | 30 Jun 2026 | Jan 2026 |
| Gemini 3.8 Flash | 2 Sep 2026 | Mar 2026 |
| DeepSeek-V4.1-Flash | 10 Sep 2026 | not disclosed |
| GPT-6.1 Sol† | 29 Sep 2026 | Apr 2026 |
Table 8. Evaluated models. †Released after the batch resolved, with cutoff from third parties.
Search Providers
Table 9 gives the outcome-free properties behind the choice of Tavily as the main provider. Coverage is the query-planning LLM’s rating, after the follow-up queries, of how well a provider’s results cover the current-evidence, base-rate, and counterevidence purposes. It leaves out resolution, whose evidence rarely exists before a question settles. Relevance is the share of items whose title and snippet contain at least a third of the question’s content words. Models read text evidence from the search APIs without a separate full-page reading stage. The market-price guard withholds 6.5% of all results. Collecting the batch’s 14,377 requests cost about $130 at these prices.
| Provider | Coverage | Relevant | Dated | Price |
| Tavily | 0.68 | 66% | 0% | $16 |
| Exa | 0.64 | 51% | 39% | $15 |
| Serper | 0.53 | 42% | 55% | $1 |
| Parallel | 0.64 | 56% | 56% | $5 |
Table 9. Search providers at the last state. Coverage rating, relevant and dated items, and list price per 1,000 requests.
Post-hoc Search Probe
The probe draws its questions in proportion to the batch, 44 from sports and culture and 16 from other domains, and uses each provider’s own date filter: Tavily’s end_date, Exa’s endPublishedDate, and Serper’s custom date range. After the safeguards and deduplication, the post-hoc results give 12–16 items per question, against 15 in the live snapshot.
Trading Simulation
Both trading experiments fill orders at half the question’s quoted bid–ask spread, taken from its order book at collection and capped at 0.10, and pay Kalshi’s taker fee of per contract priced c. Contracts priced below 0.05 are not bought, since a single 1-cent contract would otherwise decide the total. In the trading mode, the agent may stake up to $1 per state, and a position still open after the last state is held to settlement. The simulation allows fractional contracts and does not round Kalshi’s fees up to the cent, so if anything it favors the trader.
Diagnostic Labeling Tasks
Gemini 3.8 Flash performs both diagnostic tasks of § 7.1.2, with a separate prompt for each; the labels enter no score. For evidence-direction labeling, it reads each state’s evidence with the question, criteria, forecast time, and outcome, and rates how strongly the evidence points. For gap attribution, it seeks a signal available beforehand rather than the outcome itself, and checks four pools in order: the evidence the model saw, Tavily items the market-price guard removed, the other providers’ items, and later Tavily items dated at or before the state. The first pool holding the fact sets the label, and otherwise the LLM judge picks one of the remaining causes of Figure 5.
Figure 5. Where the models’ Brier gap to the market lies. Panel a shows shares of the gap by question type against each type’s share of states; panel b shows causes across the 111 states trailing the market by more than 0.1, drawn from 49 questions.
Prompts
Each forecast is one request consisting of instructions and a plain-text input, with a JSON schema requiring probability, cited_ evidence_ids, and rationale. The main condition also includes the note lists evidence_notes, factors, and reasoning_notes, capped at 8, 6, and 4 entries, respectively. In the main condition the instructions are:
You are forecasting the outcome of a binary question, which resolves according to the resolution criteria given with it. Give the probability that the first option is the outcome, as of the current time stated in the input. Use only the evidence provided. Each item shows its title, the website that published it, its publication date when known, and the date it was retrieved, followed by a snippet. Do not use any knowledge of events after the current time, including the outcome. In cited_evidence_ids, list the ids (such as E3) of the items you relied on. You also see your forecast and rationale from the previous state. Weigh them as one input among others, and do not anchor on them if the evidence now points elsewhere. You keep short notes on this question across states: the key evidence so far with its dates, the factors that have mattered, and how you have been updating. Revise them and return the updated lists in evidence_notes, factors, and reasoning_notes. The notes must stand on their own: do not refer to evidence ids, since items are renumbered at every state. Base any revision only on information available by the current time.
The input gives the current time, the question, its options and resolution criteria, and the evidence items rendered as in § 5 under the heading “Evidence retrieved at the current state ( items; earlier evidence appears only if it was retrieved again)”. It ends with the previous forecast, its rationale, and the notes from earlier states.
At a question’s first state the previous forecast is absent and the notes read “none yet”. Evidence identifiers that a model still writes into its notes are removed before the notes are carried forward. The other conditions change the request as follows. Previous only: the notes, the note-keeping sentences, and the note lists in the schema are dropped. No memory: in addition, the previous forecast, its rationale, and the sentences about them are dropped. All so far: as no memory, with the evidence section headed “Evidence ( items, in the order they were first retrieved)”. Closed-book: no evidence or memory, and the sentences about evidence and citations are replaced by “No evidence is provided: rely only on the question and your general background knowledge.” BM25 search: as no memory, but the model first receives only the current time, the question, and its resolution criteria, and is asked to “propose up to 3 short, neutral web-search-style queries that would help resolve this forecasting question as of the current time, without presuming an outcome”. The documents that BM25 retrieves for these queries from the evidence so far form the evidence section.
Additional Results
Ensembles and Blends
Since LLM ensembles can rival human crowds [27] and a weighted blend of a model and the crowd can beat both [11], we score the mean and median of the four models’ forecasts and blends (1−) with the market price , for and for of Halawi et al. [11]. As Table 10 shows, the mean and median of the four models only match the best single model, at the last state and over all states. Blending in the market helps, the more so the larger its weight, but even the blend that gives the market a weight of 0.8 falls just short of the market alone.
Table 10: Brier score of model ensembles and model–market blends, where is the models’ weight.
| Forecaster | Last state↓ | All states↓ |
| Best single model | 0.217 | 0.222 |
| Mean of four | 0.219 | 0.224 |
| Median of four | 0.215 | 0.222 |
| Blend, | 0.189 | 0.205 |
| Blend, | 0.179 | 0.200 |
| Market | 0.176 | 0.199 |
Table 10. Brier score of model ensembles and model–market blends, where is the models’ weight.
Trading
Table 11 gives the profit of both trading experiments, under the rules of Appendix B.4.
| Trader | Trades | Sold at next state | Held to settlement |
| Blind rule | |||
| GPT-5.6-Terra | 231 | −13.8 [−23.4, −2.5] | −5.7 [−28.7, 19.1] |
| Claude Sonnet 5 | 278 | −12.4 [−20.7, −3.6] | −3.7 [−23.7, 18.8] |
| Gemini 3.8 Flash | 214 | −12.7 [−23.2, −0.4] | −3.8 [−28.4, 25.2] |
| DeepSeek-V4.1-Flash | 224 | −10.9 [−20.8, 0.8] | −5.4 [−28.1, 20.4] |
| Trading mode, own exits | |||
| Gemini 3.8 Flash | 177 | −5.4 [−26.4, 19.1] |
Table 11. Profit in cents per dollar staked, with 95% intervals over questions.
Table 11: Profit in cents per dollar staked, with 95% intervals over questions.
Figure 8 shows how Gemini uses the price in the trading mode. It keeps about half of its disagreement with the market and seldom copies the price outright. Where its blind forecast differs clearly from the market, it almost always stays on the same side and only narrows the gap, so it hedges toward the market rather than defers to it.
Figure 8. Gemini hedges toward the market when shown its price. Forecast minus market price at the 575 scored states that have a market price, with the price hidden and shown. The dashed line marks ignoring the price, and the horizontal axis marks copying it.
Evidence Direction and Properties
Table 12 groups the states using the evidence-direction labeling task of § 7.1.2. Where the evidence points toward the outcome, the models score about as well as the market or better, and few of their wrong-side forecasts fall there. The gap opens where the evidence is mixed or points the other way.
| Forecaster | Brier↓ | Wrong side of 0.5 | |||||
| Toward | Mixed | Against | Toward | Mixed | Against | ||
| GPT-5.6 Terra | 0.133 | 0.266 | 0.387 | 216 | 17% | 32% | 50% |
| Claude Sonnet 5 | 0.140 | 0.301 | 0.449 | 223 | 13% | 36% | 50% |
| Gemini 3.8 Flash | 0.126 | 0.277 | 0.405 | 219 | 13% | 34% | 52% |
| DeepSeek-V4.1-Flash | 0.120 | 0.273 | 0.429 | 206 | 12% | 33% | 54% |
| Market | 0.137 | 0.222 | 0.313 | 185 | 22% | 29% | 49% |
| States | 310 | 142 | 122 |
Table 12. Forecasts by the direction of a state’s evidence (main condition, Tavily). The Brier columns use the LLM judge’s labels. The wrong-side split counts weak support as mixed and requires at least moderate strength for toward. : number of wrong-side forecasts.
Simple evidence measures, such as the number of items, their lexical relevance, the share of new URLs, and the mix of query purposes, are at most weakly related to the Brier score, with no Spearman correlation above 0.11 in magnitude. The size of the models’ updates follows the market’s moves, with a correlation of 0.26, but not the arrival of new URLs, with 0.02.
Model Disagreement over Time
Figure 9 shows that the models disagree more a week or more before the event than in its last days, while the Brier score of their mean forecast barely changes. The buckets shrink from about 140 states in each two-day bucket of the last week to 18 two weeks out, so the earliest points are noisier.
Figure 9. Model disagreement and accuracy by days until the event. (a) Disagreement among the four models, as mean pairwise difference and standard deviation, and (b) Brier of their mean forecast and of the market.
Memory Ablations
Table 13 gives the paired differences of each RQ5 condition from the main condition and the per-state forecast difference . Searching the accumulated evidence with BM25 also scores worse at the last state than rereading the same pool, though not significantly.
| Condition | Last [95% CI] | TBS [95% CI] | |||
| Previous only | +0.003 [−0.004, 0.012] | +0.004 [−0.002, 0.009] | +0.003 | 0.025 | 13% |
| No memory | +0.013 [−0.003, 0.032] | +0.010 [0.002, 0.018] | +0.008 | 0.042 | 21% |
| All so far | +0.009 [−0.001, 0.022] | +0.004 [−0.003, 0.012] | +0.000 | 0.029 | 15% |
| BM25 search | +0.031 [0.006, 0.058] | +0.013 [0.004, 0.023] | +0.012 | 0.059 | 28% |
Table 13. Paired differences from the main condition (positive: worse). Last: last-state Brier. : forecast minus the main condition’s at the same state, with mean and mean absolute value .
Case Studies
Figure 10 traces three questions chosen by fixed rules: (a) the largest miss shared by all models while the market was right, (b) the largest
update toward the outcome, and (c) the largest gap between two models’ updates. Together they mark the boundaries of what live evidence can do. In (a), the notes kept stressing softer inflation, so memory held the models on a wrong reading while the market rose, as Section 6 discusses. In (b), Metascores above the threshold appeared the day after release, and the models rose sharply within one state while the market stayed below 0.5. The decisive signal was published on the Web as soon as it existed, so retrieval put every model ahead of the market, the reverse of the usual gap. In (c), three models read the same evidence correctly, so the error lies in one model’s reading of undated pages rather than in retrieval, and it vanished at the next state.
References
- [1]Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. 2025. Deep Research Bench: Evaluating AI Web Research Agents. arXiv:2506.06287 [cs.AI]arxiv.org/abs/2506.06287
- [2]Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78, 1 (1950), 1–3. doi:10.1175/1520-0493(1950)078<0001:VOFET>2.0.CO;2doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
- [3]Yurui Chang, Yongkang Du, Yuanyou Cao, Jinghui Chen, and Lu Lin. 2026. ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory. arXiv:2605.30858 [cs.LG]arxiv.org/abs/2605.30858
- [4]Zijian Chen, Xuexuang Zhuang, Shengyao Zhuang, Ping Nie, Kai Zou, Sahel Sharify-moghadam, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Hosna Oyarhoseini, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. 2026. BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 22349–22370. doi:10.18653/v1/2026.acl-long.1023DOI
- [5]Yizhou Chi, Eric Chamoun, Zifeng Ding, and Andreas Vlachos. 2026. World-Reasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning. arXiv:2606.11816 [cs.CL]arxiv.org/abs/2606.11816
- [6]Hui Dai, Ryan Teehan, and Mengye Ren. 2025. Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle. In Proceedings of the 42nd International Conference on Machine Learning.arxiv.org/abs/2411.08324
- [7]DeepSeek-AI. 2026. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. arXiv:2609.19969 [cs.CL]
- [8]Ali El Lahib, Ying-Jieh Xia, Zehan Li, Yuxuan Wang, and Xinyu Pi. 2026. Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 787–795. doi:10.18653/v1/2026.acl-short.65DOI
- [9]Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359–378. doi:10.1198/016214506000001437DOI
- [10]Yisheng Goel, Nikhil Chandarkar, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, and Jonas Geiping. 2026. FutureSim: Replaying World Events to Evaluate Adaptive Agents. arXiv:2605.15188 [cs.LG]
- [11]Danny Halawi, Fred Zhang, Yueh-Han Chen, and Jacob Steinhardt. 2024. Approaching Human-Level Forecasting with Language Models. In Advances in Neural Information Processing Systems, Vol. 37. doi:10.52202/079017-1598DOI
- [12]Elvis Hsieh, Preston Fu, and Jonathan Chen. 2024. Reasoning and Tools for Human-Level Forecasting. In Proceedings of the Workshop on the Future of Event Detection (FutureED). Association for Computational Linguistics, 48–57. https://aclanthology.org/2024.futured-1.5/
- [13]Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-Tao Xia, Dawei Cheng, and Changjun Jiang. 2026. FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting. Frontiers of Computer Science (2026). doi:10.1007/s11704-026-51064-5DOI
- [14]Wojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. 2021. ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4636–4650. doi:10.18653/v1/2021.acl-long.357
- [15]Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. 2025. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. In International Conference on Learning Representations.arxiv.org/abs/2409.19839
- [16]Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dagmar Radov, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. RealTime QA: What’s the Answer Right Now? In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), Vol. 36.arxiv.org/abs/2207.13332
- [17]Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenes, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the Gap: Assessing Temporal Generalization in Neural Language Models. In Advances in Neural Information Processing Systems, Vol. 34.
- [18]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, Vol. 33.arxiv.org/abs/2005.11401
- [19]Yuante Li, Yicheng Tao, Kate Zhang, Taozhi Wang, Gefei Gu, and Yaxin Zhou. 2026. Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry. arXiv:2607.01661 [cs.AI]arxiv.org/abs/2607.01661
- [20]Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezenler, Devang Agrawal, Cyprien De Masson D’Autume, Tim Schotles, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162). PMLR, 13604–13622. https://proceedings.mlr.press/v162/liska22a.htmlarxiv.org/abs/2205.11388
- [21]Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fischer, Sydney E. Scott, Don Moore, Pavel Atanasov, Samuel A. Swift, Terry Murray, Eric Stone, and Philip E. Tetlock. 2014. Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science 25, 5 (2014), 1106–1115.DOI
- [22]Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr. 2025. Pitfalls in Evaluating Language Model Forecasters. arXiv:2506.00723 [cs.LG]arxiv.org/abs/2506.00723
- [23]Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tramèr. 2025. Consistency Checks for Language Model Forecasters. In International Conference on Learning Representations.arxiv.org/abs/2412.18544
- [24]Prophet Arena. 2026. Time-Gated Search Benchmark. https://www.prophetarena.com/docs/search-bench. Accessed October 25, 2026.
- [25]Kaitlyn Radde. 2026. Prediction Markets’ Trading Volume Doubled between May and July, Largely Driven by Sports. Pew Research Center. https://www.pewresearch.org/short-reads/2026/09/23/prediction-markets-trading-volume-doubled-between-may-and-july-largely-driven-by-sports/ Accessed October 26, 2026.
- [26]Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3, 4 (2009), 333–389.
- [27]Philipp Schoenegger, Indre Tuminauskaitė, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. 2024. Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy. Science Advances 10, 45 (2024), eadp1528. doi:10.1126/sciadv.adp1528DOI
- [28]Sriram Selvam and Anmesha Ghosh. 2026. Similar Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents. arXiv:2607.10198 [cs.CL]arxiv.org/abs/2607.10198
- [29]Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. 2025. Coding Agents with Multimodal Browsing are Generalist Problem Solvers. arXiv:2506.03011 [cs.CL]arxiv.org/abs/2506.03011
- [30]Yihong Tang, Andrew Robert Williams, Arjun Ashok, Vincent Zhihao Zheng, Shubham Gupta, Lijun Sun, Alexandre Drouin, Issam H. Laradji, Étienne Marcotte, and Valentina Zantedeschi. 2026. Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting. arXiv:2605.27904 [cs.AI]arxiv.org/abs/2605.27904
- [31]Philip E. Tetlock and Dan Gardner. 2015. Superforecasting: The Art and Science of Prediction. Crown.
- [32]Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 13697–13720. doi:10.18653/v1/2024.findings-acl.813
- [33]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. In International Conference on Learning Representations.
- [34]Jack Wildman, Nikos I. Bosse, Daniel Hylyk, Peter Mühlbacher, Finn Hambly, Jon Evans, Dan Schwarz, and Lawrence Phillips. 2025. Bench to the Future: A Pastcasting Benchmark for Forecasting Agents. arXiv:2506.21558 [cs.CL]arxiv.org/abs/2506.21558
- [35]Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2025. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. In International Conference on Learning Representations.arxiv.org/abs/2410.10813
- [36]Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. 2025. LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena. arXiv:2510.17638 [cs.AI]arxiv.org/abs/2510.17638
- [37]Chencheng Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. 2024. MIRAI: Evaluating LLM Agents for Event Forecasting. arXiv:2407.01231 [cs.CL]
- [38]Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, and Ben Zhou. 2026. Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters. arXiv:2607.14051 [cs.CL]arxiv.org/abs/2607.14051
- [39]Zhiquan Zhang, Zifeng Ding, and Andreas Vlachos. 2025. Do Language Models Update their Forecasts with New Information? arXiv:2509.23936 [cs.CL]
- [40]Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianyi He, Yiliao Tian, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianlei Cai, Zehui Chen, Jieca Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Jiangsheng Li, Jingkai Liu, Shuang Ni, Zhoufu Wen, Ge Zhang, Kaiyuan Zhang, Xin Zhou, Jose Blanchet, Xipeng Qiu, Mengdi Wang, and Wenhao Huang. 2025. FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction. arXiv:2508.11987 [cs.AI]arxiv.org/abs/2508.11987
- [41]Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecasting Future World Events with Neural Networks. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), Vol. 35.









