WorksComputer Science
FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs
Abstract
Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
Introduction
Retrieval-augmented generation (RAG) asks models to use external evidence while resisting it when it is wrong. When a passage conflicts with a model’s own answer, which source should take precedence? Asking the model to check the passage seems like a natural safeguard, yet the same prompt can also instruct it to treat that passage as primary. Such prompts bundle a verification step with a source-priority rule, assigning passage priority or knowledge priority when sources conflict.
Checking and assigning priority are different decisions. Chain-of-thought prompting and self-correction motivate additional checking [22, 13]. Intrinsic
self-correction can fail without external feedback [6]; a check for relevance, sufficiency, or coherence can leave a plausible false claim intact. Figure 1 shows the conflict on a pre-cutoff question about Biden’s December 2020 HHS nominee. o4-mini answers correctly without a passage and under Baseline with a false passage, yet adopts the planted name under Verify. The bars summarize the full pre-cutoff evaluation; Appendix G gives further cases.
Figure 1. Same passage, different answers (o4-mini; pre-cutoff).
Prior work shows that coherent counter-evidence can persuade models [24] and that false context can override correct answers [23, 7]. These findings motivate a further question: how much of a verification prompt’s behavior comes from asking for a check, and how much from specifying which source wins? Answering it requires a no-context reference, controlled evidence, and prompt comparisons on the same questions and passages.
We introduce FutureBench, a renewable framework that links these controls. Building on timestamped TiEBe questions (Almeida et al., [1]), FutureBench pairs each question with three passages supporting the gold answer, a planted false answer, or insufficient evidence. A no-context probe separates items each model answers correctly on its own from those it does not. Paired prompt comparisons then measure how checking and source-priority instructions reshape evidence use. Fixed cohorts permit reassessment as models change; new event cohorts extend the evaluation beyond earlier knowledge cutoffs.
Our evaluation of 3,629 questions across six models supports three findings:
Verification with passage priority can undermine correct answers. Verify raises false-answer adoption across all five models evaluated on pre-cutoff events, with larger increases on known items. Post-cutoff responses differ, with only o4-mini showing an increase (§ 5.1–§ 5.4).
Passage priority matters even without a check. On the 902-item subset, Passage-first raises adoption over Baseline; the four-prompt comparison separates its increase from the offset under Verify (§ 5.5).
Displaced answers can be recovered. Adversarial-Aware restores most of the same pre-cutoff answers that Verify displaced, with little change in true-passage accuracy. On o4-mini’s 902-item subset, it lowers adoption relative to Verify at the same true-passage accuracy cost (§ 5.6).
FutureBench supplies the probes, passage triplets, and paired comparisons needed to test how models resist false passages while using true ones as models and events change. Data and code will be released at https://github.com/pa0lai/FutureBench.
Related Work
Knowledge–context conflict. Entity substitutions can expose disagreement between model answers and context (Longpre et al., [12]), while coherent counter-evidence can make models highly receptive to conflicting claims (Xie et al., [24]). The balance also depends on factual familiarity (Mallen et al., [14]). ClashEval (Wu et al., [23]) tests both erroneous evidence overriding correct priors and correct evidence repairing wrong priors, relating context preference to perturbation size and prior confidence. To Trust or Not to Trust? (Huang et al., [7]) evaluates factual correctness under true and false contexts and develops confidence-based conflict-resolution methods. These works establish the need to assess both resisting misinformation and accepting useful evidence. Our focus is repeated diagnosis of source-priority prompts over event cohorts.
Context faithfulness and factual correctness. Context-faithful prompting (Zhou et al., [26]) encourages models to follow supplied information, including counterfactual facts. FaithEval (Ming et al., [16]) evaluates contextual faithfulness under unanswerable, inconsistent, and counterfactual contexts. Its counterfactual task rewards adherence to stipulated context even when that context disagrees with world knowledge. FutureBench retains the factual gold answer and measures adoption of a planted false account, alongside correct-evidence use. Li et al. [10] quantify memory strength through consistency across question paraphrases and examine evidence styles. FutureBench instead uses one no-context answer per item, reducing the cost of repeating the probe across models and event cohorts.
Verification and evidence use. Retrieval augmentation supplies information beyond model parameters (Lewis et al., [9]), but document position and unsupported generation can still affect evidence use (Liu et al., [11]; Niu et al., [17]). Multi-step retrieval benchmarks (Krishna et al., [8]) and taxonomies of knowledge conflicts (Xu et al., [25]) motivate separating evidence availability from its use. Self-Refine studies iterative feedback (Madaan et al., [13]), while intrinsic self-correction can fail without external feedback (Huang et al., [6]). Generated explanations also need not faithfully reflect the underlying computation (Turpin et al., [21]). We measure verification through final answers under fixed passages and prompt templates.
Renewable temporal evaluation. Temporal QA benchmarks assess changing facts and model knowledge (Vu et al., 2024; Chen et al., [3]). TiEBe provides timestamped event questions and a pipeline for adding new events (Almeida et al., [1]). FutureBench builds on those questions by attaching passage triplets, model-specific no-context probes, and paired prompt comparisons. Event dates define the temporal cohort, while the probe tests each model’s recall on an item. Together, they support reassessment of a fixed conflict as models change and evaluation on newly added events. The framework in § 3 specifies these controls; § 4 describes the present experimental instantiation.
The FutureBench Framework
FutureBench separates what a model can answer from how it uses a passage (Figure 2). The no-context probe establishes a reference answer; the passage triplet varies evidence quality; paired prompts vary the source-priority strategy. Each prompt comparison holds the model configuration, question, and passage fixed. The English event-question instantiation below supplies these controls, and § 3.5 explains how to repeat them on fixed and newly added cohorts.
Figure 2. FutureBench evaluation framework.
Questions and the No-Context Probe
TiEBe provides multilingual factoid questions about timestamped notable events [1]. We use its English question–answer pairs as seeds. Before measuring how a passage changes an answer, we ask each model to answer the question using only its own knowledge. The model must abstain if it cannot answer; no passage is shown.
For each model, an item is known if the no-context probe answers correctly and unknown otherwise. Incorrect answers and abstentions retain separate response labels. This per-model, per-item grouping supplies the reference for measuring how passages change answers.
Three Passages per Question
Each question receives three length-matched passages generated using Gemini-3-Flash [5] via OpenRouter:
True passage (). Supports the gold answer, testing use of correct evidence.
False passage (). Supports a planted incorrect answer, testing misinformation adoption.
Insufficient passage (). Lacks the answer, testing abstention when passage evidence is missing.
Appendix § 13 describes a passage-review sample and criteria.
For false passages, 96.8% are generated directly as fluent accounts asserting a plausible false answer. Numerical perturbations account for 2.3%, and entity-pool substitutions for 0.9%. Automated checks require the true passage to contain the gold answer, the false passage to contain the planted answer but not the gold answer, and the insufficient passage to contain neither. Items failing these checks are regenerated.
Each prompt uses the same question and fixed passage triplet. Once constructed, a false passage supplies the same competing answer to every instruction condition. This prevents a prompt difference from being confounded with a different false claim. Length matching and a common news style also reduce obvious surface cues. Answer-string checks filter construction errors; semantic validation separately assesses whether each passage supports its intended answer or leaves the question unresolved. A passage can contain an answer string without supporting its claim, or imply an answer without naming it.
Instructions for Choosing a Source
Table 1 collects all eight prompt names used below. Baseline and Verify are the main comparison; No context and Context only provide diagnostic references. Passage-first and Soft enter the four-prompt comparison in § 5.5; the two mitigation prompts are tested in § 5.6. Full templates are in Appendix § 9.
| Prompt | Source priority | Check | Instruction |
| Core prompts and diagnostic references | |||
| Baseline | Knowledge | No | Use both sources; resolve conflicts with the answer believed correct. |
| No context | Knowledge | No | Answer without a passage. |
| Context only | Passage | No | Ignore outside knowledge. |
| Verify | Passage | Yes | Check passage reliability; answer from it or abstain. |
| Four-prompt comparison (§ 5.5) | |||
| Passage-first | Passage | No | Prioritize the passage without checking. |
| Soft | Knowledge | Weigh | Weigh both sources with model knowledge as primary. |
| Knowledge-priority mitigations (§ 5.6) | |||
| Adversarial-Aware | Knowledge | No (warning) | Warn of misleading evidence and prioritize model knowledge. |
| Knowledge-Anchor | Knowledge | Compare | Answer from knowledge, then compare with the passage. |
Table 1. Prompt guide.
Moving from Baseline to Verify adds a checking step and assigns passage priority. The four-prompt comparison in § 5.5 examines these two features separately. The no-context probe is run without a passage; the other strategies are evaluated with each passage type.
Outcome Measures
Each response receives one of four labels (Appendix § 13): Follow (adopts the planted false claim), Stick (matches the gold answer), Reject (abstains), or Wrong (other incorrect answer). In the main text and figures, these behaviors are called false adoption, correct, abstention, and other error. We report false-answer adoption as the Follow rate on false passages, true-passage accuracy as the Stick rate on true passages, and abstention as the Reject rate. Errors unrelated to the planted claim do not count as false-answer adoption.
The factual gold answer stays fixed across conditions, including when the prompt gives the passage priority.
A positive change from Baseline to Verify means more false-answer adoption. Tables and figures report this change in percentage points (pp). For items with complete paired labels, let be 1 if item is labeled Follow under strategy , and 0 otherwise. Here denotes Baseline and denotes Verify; other prompt pairs use the same calculation. Then
where counts non-Follow-to-Follow transitions and counts the reverse. For known and unknown items, is the paired size of the group. The pre-cutoff group comparisons and mitigation analyses contain 1,825 complete pairs from the 1,827-item cohort.
Renewing the Benchmark
Renewability concerns the event questions and their attached evidence. As a reported knowledge cutoff advances, an existing post-cutoff cohort can become a pre-cutoff cohort for later models. Retaining its passages allows the same conflict to be tested again. A new cohort can then be constructed from more recent TiEBe events using the same true,
false, and insufficient passage requirements. The no-context probe is repeated for each evaluated model configuration, assigning each question to known or unknown items for that model and run.
For repeated evaluation, we recommend retaining two comparisons. First, rerun a fixed cohort to measure changes on the same inputs. Second, evaluate a newly added cohort to measure responses to more recent evidence. Report these cohorts separately, since replacing old questions changes both time coverage and item difficulty. Each release should preserve question identifiers, event dates, gold and planted answers, passages, prompt templates, model configurations, predictions, and judge labels. Recording access dates and immutable model versions, where available, is also necessary to interpret later changes.
Experimental Setup
The experiment combines temporal coverage with per-item recall. Event dates define two cohorts; the no-context probe determines what each model actually answers correctly within them. Prompt comparisons then reuse the same questions and passages in each cohort. This design connects event recency to evidence use without assigning knowledge status from dates alone.
Temporal Coverage and Models
The current benchmark contains 3,629 items in two cohorts:
Pre-cutoff: 1,827 items, covering January 2020–April 2024. These events predate the reported knowledge cutoffs of all five models evaluated on this cohort. The no-context probe identifies which items each model can actually answer.
Post-cutoff: 1,802 items, covering November 2024–April 2026. These events follow the knowledge-cutoff dates used for the six models (Appendix § 13 for Claude-3.7’s cutoff convention). No-context accuracy is low on this cohort, providing a complementary setting with few correct no-context answers.
The post-cutoff cohort was constructed in two chronological batches: 900 items from November 2024–November 2025, followed by 902 items from December 2025–April 2026. The four-prompt comparison (§ 5.5) and post-cutoff mitigation test (§ 5.6) use the latter, termed the 902-item subset. Together the batches form the full 1,802-item post-cutoff cohort.
All models are accessed through OpenRouter at temperature 0 (Appendix § 13). Claude-3.7 was unavailable during the pre-cutoff evaluation and is included only in the post-cutoff cohort. Unless stated otherwise, o4-mini denotes the model configuration openai/o4-mini-high. We use Claude-3.7 and Llama-4-M as short names for Claude 3.7 Sonnet and Llama 4 Maverick, respectively. Cohort membership follows TiEBe event timestamps; the no-context probe establishes correct no-context answers for the question itself.
Response Judging
The passage generator, Gemini-3-Flash, is also the primary judge. An independent GPT-5.1 judge agrees on 92.0% of a label-stratified 200-response sample (). Expanded evaluation finds 99.4% agreement on the 6,314 responses added for the four-prompt comparison across four models, excluding o4-mini; Appendix § 13 gives the cohort details and label-level results.
Comparisons between prompts use McNemar tests. Bootstrap confidence intervals and paired transition counts are in Appendices § 15 and § 16.
Results
We begin with the Baseline–Verify comparison on false passages, then ask whether the model could answer each question without a passage. The later experiments separate passage priority from checking and test whether changing that priority helps. True-passage results track the cost of each change. Rates are percentages; changes are percentage points.
False-Answer Adoption Rises on Pre-cutoff Events
Verify turns many Baseline non-adoptions into false-answer adoption on pre-cutoff events. False-answer adoption increases across all five pre-cutoff models (Table 2; all ), with o4-mini showing the largest increase. Its paired transitions reveal the imbalance: 976 items move into adoption, while only six move out (Appendix § 16). Across models, the largest increases occur where Baseline had the most non-adopting responses.
| Model | False-answer adoption | True-passage accuracy | |||
| Baseline | Verify | Change | Baseline | Verify | |
| Pre-cutoff | |||||
| DeepSeek-V3 | 79.9 | 86.8 | +6.9{sup}*** | 77.4 | 55.9 |
| Llama-4-M | 80.8 | 86.8 | +6.0{sup}*** | 82.9 | 65.0 |
| GPT-4.1-Mini | 82.1 | 91.6 | +9.5{sup}*** | 84.2 | 65.8 |
| GPT-5.1 | 42.7 | 74.8 | +32.1{sup}*** | 92.3 | 62.5 |
| o4-mini | 33.9 | 87.0 | +53.2{sup}*** | 85.8 | 64.2 |
| Post-cutoff | |||||
| DeepSeek-V3 | 93.4 | 89.0 | −4.4{sup}*** | 82.7 | 63.4 |
| Llama-4-M | 88.8 | 76.9 | −12.0{sup}*** | 82.0 | 62.5 |
| GPT-4.1-Mini | 94.1 | 94.0 | −0.1 | 86.6 | 82.4 |
| GPT-5.1 | 92.0 | 91.2 | −0.7 | 86.9 | 78.0 |
| o4-mini | 80.5 | 94.2 | +13.7{sup}*** | 86.2 | 83.7 |
| Claude-3.7 | 71.6 | 60.9 | −10.7{sup}*** | 71.6 | 56.0 |
Table 2. Baseline versus Verify by temporal cohort. Rates are percentages; changes are percentage points. All pre-cutoff changes have . Post-cutoff: {sup}***; unmarked changes are not significant. McNemar tests are paired by item.
Models such as DeepSeek-V3 already adopt most planted answers under Baseline; Verify pushes adoption higher. GPT-5.1 and o4-mini start lower but show much larger increases in false-answer adoption.
Baseline adoption is lower on pre-cutoff than post-cutoff events for every model with both cohorts (Appendix § 19, Table 14). Verify weakens this pre-cutoff advantage, particularly for GPT-5.1 and o4-mini.
| Model | Baseline adoption (%) | ||
| Pre-cutoff | Post-cutoff | Change () | |
| DeepSeek-V3 | 79.9 | 93.4 | −13.5*** |
| Llama-4-M | 80.8 | 88.8 | −8.0*** |
| GPT-4.1-Mini | 82.1 | 94.1 | −12.0*** |
| GPT-5.1 | 42.7 | 92.0 | −49.3*** |
| o4-mini | 33.9 | 80.5 | −46.7*** |
| Claude-3.7 | — | 71.6 | — |
Table 14.
The increase in false adoption accompanies a loss of correct answers even when the passage is true (Table 2, right columns). For GPT-5.1, true-passage accuracy falls from 92.3% to 62.5%. Verify therefore worsens both measured outcomes on pre-cutoff events. The no-context probe tests whether the adoption increase is concentrated on answers the model could already supply.
The Increase Is Larger on Known Items
Verify raises false-answer adoption more on known than unknown items in all five pre-cutoff models. Table 3 reports the change in false-answer adoption within each group, using all paired items in that group as the denominator. Baseline adoption in the unknown group is already high for DeepSeek-V3 (91.6%) and Llama-4-M (92.8%); Appendix § 19, Table 17). These high starting rates leave little room for further increases. For DeepSeek-V3, Llama-4-M, and GPT-4.1-Mini, the increase is concentrated on known items; GPT-5.1 and o4-mini show increases in both groups. On known items, the rise is consistent with knowledge override: the model supplies the correct answer in the probe, yet adopts the false account more often under Verify. Unknown items can instead move from an unrelated error or abstention to the planted claim, with no correct probe answer demonstrated.
| Model | Known | Known (pp) | Unknown | Unknown (pp) |
| DeepSeek-V3 | 749 | +16.4 | 1076 | +0.3 |
| Llama-4-M | 959 | +12.3 | 866 | −0.9 |
| GPT-4.1-Mini | 1091 | +14.9 | 734 | +1.5 |
| GPT-5.1 | 1422 | +37.3 | 403 | +13.9 |
| o4-mini | 1139 | +60.4 | 686 | +41.1 |
Table 3. Adoption changes on known and unknown items.
Verify Produces Opposite Failures on False and True Passages
On false passages, correct Baseline answers account for the largest share of new adoptions under Verify in every model: 72.4% for o4-mini and 92.2% for GPT-5.1 (Appendix § 16). On true passages, the dominant loss is abstention: it accounts for 97.5% of GPT-5.1’s lost correct answers and 93.6% of o4-mini’s. Verify can therefore replace a correct answer with a planted claim when evidence is false, yet withhold the answer when evidence supports it.
The two failures can even occur on the same question. Among items answered correctly under Baseline with both passages, 195 of 877 for o4-mini and 186 of 958 for GPT-5.1 switch to false adoption with the false passage and abstention with the true passage. The example in Figure 3 shows this contrast for o4-mini.
Figure 3. One question, opposite failures (o4-mini; shortened).
These distinct failure paths connect the pre- and post-cutoff results. On pre-cutoff events, models supply many correct answers that Verify can override; on post-cutoff events, where correct no-context answers are scarce, we test whether Verify still increases false adoption or instead shifts responses toward abstention.
Post-cutoff Events Show a Different Pattern
The uniform increase on pre-cutoff events disappears on post-cutoff events, where no-context accuracy is just 1.3–10.0% (Appendix § 18). Models fall into three patterns (Figure 4): Claude-3.7, DeepSeek-V3, and Llama-4-M reduce false adoption; GPT-5.1 and GPT-4.1-Mini change little and continue adopting more than 91% of planted answers; only o4-mini increases adoption in both cohorts.
Figure 4. Verify adoption changes by temporal cohort (pp).
For o4-mini, Verify raises post-cutoff adoption by 13.7 pp (), nearly matching its Context only reference. The paired responses show an asymmetric shift into adoption (Appendix § 16).
Where adoption falls on post-cutoff events, responses shift mostly to abstention. Of Claude-3.7’s 247 responses that stop adopting the false answer, 243 become abstentions and only four become correct answers. Llama-4-M shows the same pattern (Appendix § 16). This withdrawal also affects useful evidence: Claude-3.7’s true-passage accuracy falls from 71.6% to 56.0%, with 290 initially correct responses becoming abstentions. Verify curbs false adoption here mainly by withholding answers, while also losing correct answers on true passages.
Why does Verify reduce adoption in some models while pushing o4-mini toward the Context only reference? Verify combines a checking request with passage priority. Comparing prompts that change these features provides a way to examine the divergence.
Separating Source Priority from Verification
Does adoption rise because the model is asked to check, or because it is told to prioritize the passage? We return to the post-cutoff questions and compare these instructions on the 902-item subset. Baseline and Verify remain the reference prompts. Passage-first adds passage priority without checking; Soft instead asks the model to weigh the passage against its own knowledge. All four prompts are compared in five models; Claude-3.7 lacks Passage-first and Soft runs (templates in Appendix § 9).
Passage priority raises adoption over Baseline without a check. Passage-first alone raises adoption over Baseline in all five models (all ). DeepSeek-V3 and GPT-4.1-Mini rise from an already high Baseline adoption rate of 95.1%. Figure 5 shows opposing shifts across the prompt sequence: adoption rises from Baseline to Passage-first, then falls from Passage-first to Verify. Soft also lowers adoption relative to Passage-first in all five models (all ; Appendix § 14).
Figure 5. Prompt contrasts (902-item subset; pp).
The balance of the two shifts determines the net result. For Llama-4-M, the reduction from Passage-first to Verify exceeds the increase after assigning passage priority, so Verify ends below Baseline. For o4-mini, passage priority adds 15.1 pp of adoption, while moving to Verify reduces it by only 1.8 pp, leaving Verify 13.3 pp above Baseline (diamonds in Figure 5). The same sequence of instructions produces opposite net outcomes because the relative sizes of these shifts differ.
Brief and CoT-Extended, two alternative checking prompts, also yield more false-answer adoption than Baseline for o4-mini (Appendix § 19, Table 15).
| Prompt | Adoption (%) |
| Baseline | 82.3 |
| Context only | 96.6 |
| Verify | 95.6 |
| Brief | 89.5 |
| CoT-Extended | 93.0 |
| Soft | 69.0 |
Table 15. Prompt controls (o4-mini; 902-item subset).
Knowledge-Priority Mitigation
To test whether a change in source priority reduces adoption without losing correct answers, we evaluate Adversarial-Aware, which warns that the passage may be misleading and tells the model to give its own knowledge precedence. We return first to the pre-cutoff events and test the same five models as in § 5.1. We then test o4-mini on the same 902-item subset used in the preceding prompt comparison.
Pre-cutoff events: lower adoption, stable accuracy. Adversarial-Aware lowers false-answer adoption in every pre-cutoff model (Figure 6, left; all ). The reduction appears on both known and unknown items (Appendix § 19). Meanwhile, true-passage accuracy changes by less than one percentage point in each model, with no significant difference from Baseline. The left panel of Figure 6 shows the reduction alongside nearly unchanged true-passage accuracy.
Figure 6. Mitigation outcomes. Left: Baseline (open) to Adversarial-Aware (filled). Right: 902-item subset.
The same displaced answers can be recovered. We track the items answered correctly under Baseline but changed to the planted answer under Verify. Figure 7 traces all initially correct answers through Verify and Adversarial-Aware; band widths count matched items. Adversarial-Aware restores the correct answer on 546 of GPT-5.1’s 552 such items (98.9%); only two become abstentions and four become other errors. For o4-mini, it restores 611 of 707 answers (86.4%). Recovery also dominates in DeepSeek-V3, Llama-4-M, and GPT-4.1-Mini, covering 82.5–94.7% of their displaced answers (Appendix § 16).
Figure 7. Answer paths from Baseline through Verify to Adversarial-Aware (pre-cutoff).
The paired true/false failures in § 5.3 are similarly recoverable. Of GPT-5.1’s 186 items where Verify adopts the false passage but abstains on the true passage, Adversarial-Aware answers both versions correctly on 184. For o4-mini, it does so on 169 of 195 items.
Post-cutoff events: lower adoption than Verify at the same accuracy cost. Figure 6 (right) compares prompts on o4-mini’s 902-item subset. Adversarial-Aware lowers false-answer adoption to 75.2%, below both Baseline (82.3%) and Verify (95.6%). Verify and Adversarial-Aware both reduce true-passage accuracy by the same 2.5 pp relative to Baseline (Appendix § 19, Table 16). Relative to Verify, this switches from passage priority to knowledge priority without adding to the accuracy cost.
| Prompt | Adoption (%) | Accuracy change (pp) |
| Baseline | 82.3 | — |
| Verify | 95.6 | −2.5** |
| Knowledge-Anchor | 84.1 | +1.1 (ns) |
| Adversarial-Aware | 75.2 | −2.5** |
Table 16. Mitigation results (o4-mini; 902-item subset).
** denotes ; ns denotes not significant.
We also test Knowledge-Anchor on this subset. It asks the model for an answer from its own knowledge before comparing that answer with the passage and instructs it to retain that answer when the two conflict. Its adoption remains above Baseline (84.1% versus 82.3%; Appendix § 19). Adversarial-Aware performs better on the same questions. Both mitigation prompts favor model knowledge, as does Baseline; Adversarial-Aware additionally warns of deliberately misleading evidence and makes knowledge priority explicit.
Discussion
Treat passage priority as a design decision. A RAG prompt can ask for careful checking while already specifying the winner of a knowledge–context conflict. On the 902-item subset, Passage-first raises false-answer adoption over Baseline without any verification step. A practical prompt review should therefore examine what “use the supplied passage as primary” authorizes the model to do when that passage is wrong. Checking instructions should be evaluated alongside this rule, using both false and true evidence.
Persuasiveness depends on the instructions around the evidence. Prior work finds that coherent counter-evidence can persuade models ([24]), and ClashEval shows that false context can override correct answers ([23]). Our prompt comparisons add an instruction-level perspective: holding the passage fixed still leaves room for substantial changes in adoption. The model-dependent prompt contrasts in § 5.5 show that the surrounding instructions change how the same evidence is used.
Protect knowledge without shutting out useful evidence. The practical goal is to resist false passages while using true ones. Adversarial-Aware lowers false adoption with nearly unchanged true-passage accuracy on pre-cutoff events. On the o4-mini mitigation test, it achieves lower adoption than Verify at the same accuracy cost. These results favor reporting both outcomes when selecting a RAG prompt.
Conclusion
FutureBench combines no-context probes, passage triplets, and paired prompts to evaluate knowledge–context conflict as models and events change. Across all five pre-cutoff models, verification with passage priority increases false-answer adoption, primarily by displacing correct Baseline answers; most lost correct answers on true passages become abstentions. On the 902-item post-cutoff subset,
passage priority alone raises adoption, while moving to Verify lowers it relative to that priority reference. Adversarial-Aware recovers 82.5–98.9% of displaced pre-cutoff answers across the five models, with less than one percentage point of change in true-passage accuracy. On o4-mini’s post-cutoff subset, it lowers adoption relative to Verify at the same accuracy cost. We will release the benchmark, predictions, and analysis code to support repeated evaluation as models and events change. Verification should be evaluated by what survives the check: correct answers the model can already give, and useful evidence it should accept.
Limitations
Scope and prompt comparisons. FutureBench tests English event factoids with single, source-free passages; retrieval, document ranking, source metadata, and multi-document conflicts are outside its scope. The four-prompt comparison covers five models on 902 post-cutoff items. Templates vary in wording as well as priority and checking, so their contrasts do not isolate individual instruction effects. Answer transitions describe behavior; cross-model differences and the selected explanation cases do not establish internal reasoning mechanisms.
Knowledge and reproducibility. Known/unknown groups reflect a single no-context response, and event dates can differ from the dates of the facts being tested. Temporal cohorts also vary in topic and difficulty. Results concern the model configurations: immutable routed snapshots and exact original access dates were not retained, and later prompt runs may include provider-side changes.
Validation and analysis coverage. Gemini generates passages and judges answers. Independent automated checks provide agreement estimates, with sampling and model coverage detailed in Appendix § 13; no human adjudication was collected. Passage and label errors may therefore affect the results. Soft lacks a true-passage evaluation. Only the o4-mini post-cutoff Baseline–Verify comparison is the primary test; other analyses are exploratory, with no correction across the full comparison family.
Release and Licensing
The planned release at https://github.com/paolai/FutureBench includes TiEBe-derived questions, generated passage triplets, raw model responses, judge labels, and aggregation scripts. Generated artifacts and code are distributed under the MIT License unless otherwise noted. TiEBe-derived source materials remain subject to upstream terms.
Prompt Templates
All prompts are rendered with {question} and, where applicable, {context} substituted per item. Every prompt ends with FinalAnswer: as the output prefix. These are the task templates used in the runs; answer-generation temperature is fixed at 0.
The placeholder {context} denotes the supplied passage.
Main Experiment Prompts
Baseline.
You are a helpful assistant. Answer the question concisely, using BOTH the provided context and your own knowledge. The provided context may be complete, incomplete, irrelevant, or even incorrect. Rules: If the context and your knowledge agree → give the correct answer. If the context contradicts your knowledge → provide the answer you believe is correct. If you truly do not know the answer, or if neither the context nor your knowledge provides enough information → reply exactly with “NOT_SUPPORTED”. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
No context.
You are a helpful assistant. Answer the question using ONLY your own knowledge. Rules: If you truly do not know the answer, or if your knowledge does not provide enough information → reply exactly with “NOT_SUPPORTED”. Do not explain or add extra words. Output only the final answer.
Question: {question} FinalAnswer:
Context only.
You are a careful factual question-answering system. Answer the question using ONLY the provided context. Rules: Use only the provided context as evidence. Do not rely on outside knowledge. If the context directly supports an answer, provide that answer concisely. If the context is insufficient, irrelevant, or internally conflicting, reply exactly with “NOT_SUPPORTED”. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Verify.
You are a careful factual question-answering system. First verify whether the provided context is sufficient and trustworthy for answering the question, then provide a final answer.
Think step by step internally before answering: 1. Identify what the question asks. 2. Check whether the context contains enough evidence to answer it. 3. Check whether the context is internally consistent and relevant. 4. If the context is insufficient or unreliable, abstain.
Rules for your FINAL answer: If the context provides sufficient and coherent evidence, answer the question concisely. If the context is insufficient, irrelevant, or self-contradictory, reply exactly with “NOT_SUPPORTED”. Use the context as the primary basis for your decision, but you may rely on your own knowledge to detect obvious contradictions or unsupported claims. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Additional Source-Priority Condition
Passage-first.
You are a helpful assistant. Answer the question concisely. The provided context is your primary source of information.
Rules: Prefer the context over your own knowledge when they conflict. If the context provides an answer, give it. If the context is insufficient or irrelevant, use your own knowledge if you have it. If neither the context nor your knowledge is sufficient → reply exactly with “NOT_SUPPORTED”. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Verification and Weighing Variants
Brief.
You are a careful assistant. Before answering, check whether the provided context is reliable and sufficient. If reliable and sufficient: answer based on the context. If unreliable or insufficient: answer from your own knowledge, or say NOT_SUPPORTED. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
CoT-Extended.
You are a careful factual assistant. Work through these steps internally before giving your final answer.
[Evaluation] Is the context relevant to this specific question? Is it internally consistent and credible? [Decision] Should you: (a) trust the context as the main source, (b) rely on your own knowledge, or (c) say NOT_SUPPORTED? [Answer] Provide your final answer based on the decision above.
Rules: If using context: answer concisely from the context evidence. If using own knowledge: answer from what you know. If neither is sufficient: reply exactly “NOT_SUPPORTED”. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Soft Knowledge-Priority Weighing.
You are a helpful assistant with broad knowledge. The provided context may or may not be accurate.
Carefully weigh the context against your own knowledge: If the context seems accurate and relevant, you may rely on it. If the context seems incomplete, irrelevant, or possibly incorrect, prefer your own knowledge. If neither the context nor your knowledge is sufficient, reply exactly “NOT_SUPPORTED”.
Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Mitigation Prompts
Knowledge-Anchor.
You are a fact-checking assistant.
Step 1: Based on your own knowledge, determine the answer to this question first. Step 2: Now read the provided context. Does it support or contradict your knowledge-based answer? Step 3: If the context supports your answer or adds useful detail → answer using that. If the context contradicts what you know → the context is likely wrong; stick with your knowledge. If you have no prior knowledge → use the context carefully; say NOT_SUPPORTED if it is insufficient.
Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Adversarial-Aware.
You are a resilient question-answering assistant. The provided context may be accurate, partially accurate, or deliberately misleading.
Priority rule: Your own knowledge takes precedence over the provided context. If the context matches your knowledge: answer using either source. If the context contradicts your knowledge: trust your knowledge; the context is likely wrong. If you have no prior knowledge on this topic: use the context cautiously. If neither the context nor your knowledge is sufficient: reply exactly “NOT_SUPPORTED”.
Do not be misled by a context that sounds authoritative but contradicts facts you know. Do not explain or add extra words. Output only the final answer.
Question: {question} Context: {context} FinalAnswer:
Experimental Configuration
Model configurations and knowledge cutoffs. All models were accessed via OpenRouter at temperature 0. Table 5 lists the evaluated models, providers, reported knowledge cutoffs, and API identifiers. We refer to the API configuration openai/o4-mini-high simply as o4-mini.
| Model / role | Provider | Cutoff | API identifier |
| Generator / primary judge | Google DeepMind | — | google/gemini-3-flash |
| DeepSeek-V3 ([4]) | DeepSeek | Jul 2024 | deepseek/deepseek-chat |
| Llama-4-M ([15]) | Meta | Aug 2024 | meta-llama/llama-4-maverick |
| GPT-4.1-Mini ([19]) | OpenAI | Jun 2024 | openai/gpt-4.1-mini |
| GPT-5.1 ([18]) | OpenAI | Sep 2024 | openai/gpt-5.1 |
| o4-mini ([20]) | OpenAI | Jun 2024 | openai/o4-mini-high |
| Claude-3.7 ([2]) | Anthropic | Oct 2024 | anthropic/claude-3-7-sonnet |
| Independent judge | OpenAI | — | openai/gpt-5.1 |
Table 5. Model configurations and API identifiers.
Exact routed snapshots and original access dates were not retained (see Limitations).
Judge rubric. The following compact rubric defines the four labels. It is a summary of the label rules, rather than a verbatim API message payload. The additional-run judging scripts pass the question, prediction, gold answer, and full passage to a shared context-aware rubric, with Reject checked before Stick, Follow, and Wrong.
Given a question and a model response, classify the response into exactly one of the following labels: Stick: the response is semantically equivalent to the gold answer. Follow: the response is semantically equivalent to the false answer. Reject: the model abstains or responds with NOT_SUPPORTED. Wrong: the response is incorrect and matches neither the gold nor the false answer.
answer. Reject: the model abstains or responds with NOT_SUPPORTED. Wrong: the response is incorrect and matches neither the gold nor the false answer.
Question: {question} Gold answer: {gold} False answer: {false} Model response: {response} Output only the label.
We release a stratified 100-item passage-review sample. Its review criteria cover support for the gold answer in true passages, plausibility and gold-answer leakage in false passages, and answer availability in insufficient passages.
Independent second-judge validation. Because the primary judge (Gemini-3-Flash) also generated the false contexts, we validate its labels with an independent judge, GPT-5.1, on a sample of 200 model responses stratified by primary-judge label (50 each of Stick/Follow/Reject/Wrong), using the identical judge prompt. The two judges agree on 92.0% of items (Cohen’s ). Agreement by primary-judge label is Stick 100%, Reject 98%, Follow 90%, and Wrong 80% (Table 6). Most disagreements involve responses originally labeled Wrong, with further disagreements among Follow responses. Both validation cohorts measure agreement between LLM judges; neither includes human adjudication.
| Primary \ 2nd | Stick | Follow | Reject | Wrong |
| Stick | 50 | 0 | 0 | 0 |
| Follow | 4 | 45 | 0 | 1 |
| Reject | 0 | 0 | 49 | 1 |
| Wrong | 7 | 3 | 0 | 40 |
Table 6. Judge agreement (200 responses).
Four-prompt comparison validation. The four-prompt comparison adds seven model–prompt cells of 902 responses each, totaling 6,314 outputs: Passage-first and Soft runs for DeepSeek-V3, Llama-4-M, and GPT-4.1-Mini, plus Soft for GPT-5.1. The corresponding cells for o4-mini and GPT-5.1’s Passage-first cell were already available and are not included in this new validation cohort. These outputs use the same 902 questions and false passages; the subset covers December 2025–April 2026.
GPT-5.1 re-evaluates every new output. The independent-judge script does not explicitly set sampling temperature; its output labels are retained for reanalysis. The judges agree on 6,278/6,314 labels (99.4%; ). Among responses originally labeled Follow, agreement is 5,969/5,997 (99.5%). Adoption rates in the paper use the primary judge’s labels; independent labels are retained for the agreement analysis.
The two validation cohorts cover different response sets and sampling designs: the original sample balances primary labels, while the expanded check covers every output in the seven new cells
Claude-3.7 uses the October 2024 knowledge cutoff in its system card, which separately states that public training data extend through November 2024 ([2]).
listed above. Table 7 uses Gemini primary labels as rows and independent GPT-5.1 labels as columns. In Table 8, each agreement denominator is the number of primary Follow labels for that model.
| Primary \ second | Stick | Follow | Reject | Wrong |
| Stick | 174 | 4 | 0 | 1 |
| Follow | 19 | 5969 | 0 | 9 |
| Reject | 1 | 0 | 128 | 0 |
| Wrong | 1 | 1 | 0 | 7 |
Table 7. Judge agreement (6,314 responses).
| Model | All outputs | Follow agreement | % |
| DeepSeek-V3 | 1804 | 1720/1728 | 99.5 |
| Llama-4-M | 1804 | 1698/1702 | 99.8 |
| GPT-4.1-Mini | 1804 | 1719/1731 | 99.3 |
| GPT-5.1 | 902 | 832/836 | 99.5 |
Table 8. Follow-label agreement by model.
Release contents and licensing are described in Appendix § 8.
Four-Prompt Comparison: Paired Results
Table 4 gives the four-condition adoption rates. Table 9 gives the discordant counts used in exact two-sided McNemar tests. The discordant cells use the definitions in Appendix § 15; the paired rate difference is percentage points. All comparisons here are exploratory.
| Model | Baseline | Passage-first | Verify | Soft |
| DeepSeek-V3 | 95.1 | 97.0 | 92.2 | 94.6 |
| Llama-4-M | 90.7 | 96.7 | 79.2 | 92.0 |
| GPT-4.1-Mini | 95.1 | 97.5 | 95.0 | 94.5 |
| GPT-5.1 | 93.6 | 97.0 | 92.8 | 92.7 |
| o4-mini | 82.3 | 97.3 | 95.6 | 69.0 |
| Claude-3.7 | 72.6 | — | 61.1 | — |
Table 4. Four-prompt comparison (902-item subset).
| Model | Change (pp) | Exact | ||
| Passage-first minus Baseline | ||||
| DeepSeek-V3 | 22 | 5 | +1.9 | 0.002 |
| Llama-4-M | 54 | 0 | +6.0 | < 0.001 |
| GPT-4.1-Mini | 22 | 1 | +2.3 | < 0.001 |
| GPT-5.1 | 33 | 2 | +3.4 | < 0.001 |
| o4-mini | 136 | 0 | +15.1 | < 0.001 |
| Verify minus Passage-first | ||||
| DeepSeek-V3 | 2 | 45 | −4.8 | < 0.001 |
| Llama-4-M | 1 | 159 | −17.5 | < 0.001 |
| GPT-4.1-Mini | 0 | 22 | −2.4 | < 0.001 |
| GPT-5.1 | 2 | 40 | −4.2 | < 0.001 |
| o4-mini | 2 | 18 | −1.8 | < 0.001 |
| Soft minus Passage-first | ||||
| DeepSeek-V3 | 9 | 31 | −2.4 | < 0.001 |
| Llama-4-M | 1 | 43 | −4.7 | < 0.001 |
| GPT-4.1-Mini | 2 | 29 | −3.0 | < 0.001 |
| GPT-5.1 | 4 | 43 | −4.3 | < 0.001 |
| o4-mini | 0 | 256 | −28.4 | < 0.001 |
| Verify minus Baseline | ||||
| DeepSeek-V3 | 10 | 36 | −2.9 | < 0.001 |
| Llama-4-M | 12 | 116 | −11.5 | < 0.001 |
| GPT-4.1-Mini | 9 | 10 | −0.1 | 1.000 |
| GPT-5.1 | 15 | 22 | −0.8 | 0.324 |
| o4-mini | 123 | 3 | +13.3 | < 0.001 |
| Claude-3.7 | 29 | 133 | −11.5 | < 0.001 |
Table 9. Paired prompt contrasts (902-item subset).
Moving from Passage-first to Verify reduces adoption in all five models (exact paired McNemar tests, all ).
Statistical Testing Details
Rates and changes are rounded independently from their underlying values. All prompt comparisons on the same item set are paired by item. For false-answer adoption rates, significance is assessed with McNemar tests on the two discordant transition cells: for Baseline non-adoption to Verify adoption, and for Baseline adoption to Verify non-adoption. Confidence intervals for are non-parametric bootstrap intervals over items with 5,000 resamples. The cross-cohort Baseline comparison in Table 14 uses two-proportion -tests across temporal cohorts. As stated in the main text, the o4-mini post-cutoff McNemar comparison is the primary test; all other significance markers are exploratory and are not corrected across the full comparison family.
Paired Transition Counts
Pre-cutoff answer displacement. The pre-cutoff runs cover January 2020–April 2024 and contain 1,825 matched Baseline–Verify items per model and passage condition. We verify identical questions, contexts, gold answers, and planted answers within pairs. False-passage Stick-to-Follow counts are 171 for DeepSeek-V3, 152 for Llama-4-M, 176 for GPT-4.1-Mini, 552 for GPT-5.1, and 707 for o4-mini. Total transitions entering Follow are respectively 188, 166, 203, 599, and 976. On true passages, Stick-to-Reject accounts for 385 of 403, 321 of 348, 321 of 351, 536 of 550, and 398 of 425 transitions leaving Stick, in the same model order. These directional counts describe displaced answers; net accuracy changes also include responses entering Stick.
Recovery of the same displaced answers. Restricting to Baseline Stick and Verify Follow on pre-cutoff false passages, Adversarial-Aware restores Stick on 141/171 items for DeepSeek-V3, 144/152 for Llama-4-M, 163/176 for GPT-4.1-Mini, 546/552 for GPT-5.1, and 611/707 for o4-mini. Among items where Baseline answers correctly with both passage types but Verify follows the false passage and rejects the true passage, Adversarial-Aware restores Stick on both versions in respectively 36/42, 33/37, 34/41, 184/186, and 169/195 cases. These comparisons track identical item identifiers across the three prompts and both passage types.
Where reduced post-cutoff adoption goes. We merge the 900-item November 2024–November 2025 batch and the 902-item December 2025–April 2026 batch by item identifier. All six models have 1,802 matched Baseline–Verify responses for both false and true passages; questions, passages, gold answers, and planted answers match within each pair. Among false-passage responses leaving Follow, the destinations (Stick / Reject / Wrong) are 4 / 243 / 0 for Claude-3.7, 3 / 95 / 0 for DeepSeek-V3, 3 / 239 / 0 for Llama-4-M, 3 / 18 / 0 for GPT-4.1-Mini, 6 / 43 / 0 for GPT-5.1, and 0 / 5 / 0 for o4-mini. On true passages, Stick-to-Reject counts are respectively 290, 351, 339, 73, 162, and 74. These are directional transition counts; responses moving in the opposite direction also contribute to net rates.
For each model, the McNemar test compares two discordant cells on false passages, using the discordant-cell definitions in Appendix E. Table 10 shows the full paired matrix for o4-mini on both temporal cohorts, and Table 11 summarizes and for all models. The o4-mini discordant-cell ratios are 251:5 on post-cutoff items and 976:6 on pre-cutoff items; both differences are by McNemar test. The pre-cutoff matrix has 1,825 paired items because two items lack paired Baseline and Verify labels.
| Cohort | Baseline | Verify: no adopt. | Verify: adopt. |
| Pre-cutoff () | No adoption | 231 | 976 |
| Pre-cutoff () | Adoption | 6 | 612 |
| Post-cutoff () | No adoption | 100 | 251 |
| Post-cutoff () | Adoption | 5 | 1446 |
Table 10. Baseline–Verify transitions (o4-mini).
| Model | Pre-cutoff | Post-cutoff | ||||
| ratio | ratio | |||||
| DeepSeek-V3 | 188 | 62 | 3.0 | 19 | 98 | 0.2 |
| Llama-4-M | 166 | 56 | 3.0 | 26 | 242 | 0.1 |
| GPT-4.1-Mini | 203 | 29 | 7.0 | 19 | 21 | 0.9 |
| GPT-5.1 | 599 | 13 | 46.1 | 36 | 49 | 0.7 |
| o4-mini | 976 | 6 | 162.7 | 251 | 5 | 50.2 |
| Claude-3.7 | — | — | — | 55 | 247 | 0.2 |
Table 11. Ratio = . Claude-3.7 was unavailable for pre-cutoff.
Cells show item counts. Full cohort .
Case Studies: Override of Correct No-Context Answers
The pre-cutoff example in Figure 1 concerns Biden’s HHS nominee (item US:2020-12:0080). The gold answer is Xavier Becerra and the planted answer is Julian Castro; o4-mini answers correctly under No context and Baseline, then gives Castro under Verify.
Table 12 shows three illustrative items from the post-cutoff cohort where o4-mini replaces a correct no-context answer. Correct no-context answers can reflect earlier background facts or information inferable from the question; these cases establish answer displacement, independently of when the fact was learned. The Windows 10 example is item World:2025-10:0327; dates are shortened in the table. The Fairfax item belongs to an April 2026 TiEBe entry but asks about his 2018–2022 office. Cohort membership follows the source-event timestamp, while the question can concern an earlier background fact. In each case the model answers correctly under No context and Baseline, but switches to the false answer under Verify. Gold is the correct answer, False is the answer planted in , and No context/Baseline/Verify are the model responses under each prompt.
| Question | Gold | False (in context) | No context | Baseline | Verify |
| How long had the insurgency waged by Boko Haram in Borno State lasted by April 2026? | 17 years | 15 years | 17 years | 17 years | 15 years |
| What political office did Justin Fairfax hold in Virginia between 2018 and 2022? | Lieutenant Governor | Attorney General | Lt. Gov. of Virginia | Lt. Gov. of Virginia | Attorney General |
| On what date did Microsoft end technical support and security updates for Windows 10? | Oct. 14, 2025 | Oct. 10, 2028 | Oct. 14, 2025 | Oct. 14, 2025 | Oct. 10, 2028 |
Table 12. Verify responses show false-answer adoption.
Follow-up explanation probe. For the 31 selected post-cutoff cases with a correct no-context answer, a correct Baseline answer, and a false Verify answer, we re-queried o4-mini with an explanation prompt. It asked the model to state what it knew, assess agreement with the passage, and then answer. The model adopted the false answer in 27 of 31 cases. Its responses did not mention the prior answer or flag a conflict; longer responses sometimes argued for the false claim.
Additional Results by Passage Type
Table 13 reports the reference outcomes: Baseline abstention on insufficient passages in both cohorts, context-only false-answer adoption, and no-context accuracy on post-cutoff items. Core Baseline/Verify rates are in Table 2. Positive values indicate increased adoption under Verify. Bootstrap 95% intervals for the post-cutoff values are: DeepSeek-V3 [−5.3, −3.5], Llama-4-M [−13.3, −10.7], GPT-4.1-Mini [−0.6, +0.4], GPT-5.1 [−1.5, +0.1], o4-mini [ +12.4, +14.9], and Claude-3.7 [−12.0, −9.2].
| Model | Baseline abstention on insufficient passages | Post-cutoff references | ||
| Pre-cutoff | Post-cutoff | Context only false adoption | No-context accuracy | |
| DeepSeek-V3 | 31.6 | 62.4 | 92.6 | 1.3 |
| Llama-4-M | 14.7 | 64.4 | 91.0 | 1.8 |
| GPT-4.1-Mini | 6.1 | 27.0 | 95.0 | 4.8 |
| GPT-5.1 | 5.3 | 59.7 | 92.8 | 4.6 |
| o4-mini | 7.8 | 47.0 | 94.8 | 10.0 |
| Claude-3.7 | — | 80.9 | 88.5 | 1.8 |
Table 13. Rates are percentages. Claude-3.7 was unavailable for the pre-cutoff evaluation.
Supplementary Result Tables
These tables support four analyses summarized in the main text: the cross-cohort Baseline contrast (Table 14), the Verify prompt-wording ablation (Table 15), the o4-mini mitigation on the 902-item subset (Table 16), and the Baseline/mitigation group rates (Table 17). is the pre-cutoff minus post-cutoff Baseline adoption rate, tested with a two-proportion -test. In Table 15, all variants differ from Verify by McNemar test (). In Table 16, the true-passage accuracy change is relative to Baseline. Rates are percentages and changes are percentage points.
| Model | Known: Baseline | Known: Adv.-Aware | Known: Change (pp) | Unknown: Baseline | Unknown: Adv.-Aware | Unknown: Change (pp) |
| DeepSeek-V3 | 63.2 | 29.8 | −33.4 | 91.6 | 68.6 | −23.0 |
| Llama-4-M | 70.0 | 47.8 | −22.2 | 92.8 | 82.1 | −10.7 |
| GPT-4.1-Mini | 74.1 | 53.5 | −20.5 | 94.0 | 84.5 | −9.5 |
| GPT-5.1 | 33.1 | 12.9 | −20.3 | 76.4 | 54.1 | −22.3 |
| o4-mini | 21.9 | 13.8 | −8.1 | 53.8 | 44.0 | −9.8 |
Table 17. Pre-cutoff mitigation by known/unknown items.
Rates are percentages; negative changes indicate reduced adoption.
References
- [1]Thales Sales Almeida, Giovana Kerche Bonás, João Guilherme Alves Santos, Hugo Abonizio, and Rodrigo Nogueira. 2025. TiEBe: Tracking language model recall of notable worldwide events through time. Preprint, arXiv:2501.07482.arxiv.org/abs/2501.07482
- [2]Anthropic. 2025. Claude 3.7 Sonnet system card. Technical report, Anthropic. Accessed November 17, 2026.
- [3]Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).arxiv.org/abs/2108.06314
- [4]DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. DeepSeek-V3 technical report. Preprint, arXiv:2412.19437.arxiv.org/abs/2412.19437
- [5]Google DeepMind. 2025. Gemini 3 Flash: Model card. Technical report, Google DeepMind. Accessed November 17, 2026.
- [6]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaxiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.01798
- [7]Yukun Huang, Sanxing Chen, Hongyi Cai, and Bhuwan Dhingra. 2025. To trust or not to trust? Enhancing large language models’ situated faithfulness to external contexts. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.14675
- [8]Satyapriya Krishna, Kalpesh Krishna, Anhad Mohanеy, Steven Schwarz, Adam Stambler, Shyam Upadhyay, and Manaan Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4745–4759, Albuquerque, New Mexico. Association for Computational Linguistics.
- [9]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.arxiv.org/abs/2005.11401
- [10]Yuepei Li, Kang Zhou, Qiao Qiao, Bach Nguyen, Qing Wang, and Qi Li. 2025. Investigating context faithfulness in large language models: The roles of memory strength and evidence style. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4789–4807, Vienna, Association for Computational Linguistics. Association for Computational Linguistics.
- [11]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
- [12]Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- [13]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.arxiv.org/abs/2303.17651
- [14]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannah Hajishirzi. 2023. When not to trust language models: Investigating the effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
- [15]Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. Published April 5, 2025; Accessed November 17, 2026.
- [16]Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. FaithEval: Can your language model stay faithful to context, even if “the moon is made of marshmallows”. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.03727
- [17]Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10862–10878, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2401.00396
- [18]OpenAI. 2025a. GPT-5.1 instant and GPT-5.1 thinking system card addendum. Technical report, OpenAI. Published November 12, 2025; Accessed November 17, 2026.
- [19]OpenAI. 2025b. Introducing GPT-4.1 in the API. Published April 14, 2025; Accessed November 17, 2026.
- [20]OpenAI. 2025c. o3 and o4-mini system card. Technical report, OpenAI. Accessed May 26, 2026.
- [21]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.arxiv.org/abs/2305.04388
- [22]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.arxiv.org/abs/2201.11903
- [23]Kevin Wu, Eric Wu, and James Zou. 2024. ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2404.10198
- [24]Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2305.13300
- [25]Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8541–8565, Miami, Florida, USA. Association for Computational Linguistics.
- [26]Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544–14556, Singapore. Association for Computational Linguistics.arxiv.org/abs/2303.11315






