betaFeedback

WorksComputer Science

A Reality Check of Inference-Time Context Augmentation Methods

Abstract

Large language models (LLMs) often fail to utilise the context given in the input, resulting in hallucinations or a fallback to parametric knowledge. Context augmentation methods aim to improve model performance by modifying the input, from inserting filler tokens that add computational time to adding relevant content such as elaboration of a context during inference. However, these methods have been evaluated in isolation and only by their performance gains, leaving it unclear how they compare and whether they reliably improve performance. In this paper, we present an evaluation framework spanning five methods that differ in how informative the inserted content is and apply them at the sentence level across seven models spanning multiple sizes on three datasets. We find that accuracy gains are not always genuine improvements but undirected perturbations: the methods correct some predictions while breaking others, particularly when the inserted text allows only for more computational time. Our analysis reveals that methods that bring no additional content primarily change model predictions near the decision boundary. Our findings highlight the need to evaluate the reliability of the accuracy gains before adopting them in practice.

Introduction

Providing contextual knowledge during inference time allows large language models (LLMs) to better ground their answers in information, especially recent, domain-specific knowledge, without additional training. Despite the contextual knowledge being explicitly provided in the input, LLMs often fail to utilise it, resulting in hallucinations or defaulting to parametric knowledge [23, 24, 19]. To address1

this, a growing body of work proposes methods that insert additional content into the context at inference time, ranging from inserting filler tokens to augmenting the input with additional knowledge [25, 15, 29, 1]. We refer to these as context augmentation methods, a subset of the broader family of context engineering methods.

However, these methods have been developed and evaluated in isolation and only by their performance gains. Performance gains alone do not explain whether the methods affect the model predictions in a reliable way. Furthermore, existing methods insert content at a limited, single, predetermined location, which differs across methods. To compare these methods systematically and understand their effects, we propose an evaluation framework that investigates the reliability of the accuracy gains across seven models and three datasets covering multiple-choice question answering (MCQA) and fact-checking.

Our evaluation framework covers five methods, each representing a family of context augmentation methods from prior work, as shown in Figure 1. We choose them to vary in how much information the inserted content carries, which allows us to compare how different amounts of information affect the model’s predictions. We further investigate how the intervention’s position within the context affects its effectiveness. We therefore apply each method after every sentence of the context, one position at a time (Table 1). This provides a consistent set of intervention positions across methods and enables a systematic analysis of position-dependent effects.

Evaluation framework for context augmentation methods

Figure 1. Our evaluation framework covers five methods, which can be split into two groups: one that only adds computation and the other that adds both computation and new content. While a method can change the model prediction, we further test whether the change is a directed improvement or undirected perturbation through the sign-randomisation test.

MMLU – Professional Law #753
ContextOne day, while riding his motorcycle through a residential area, a man hit and injured a child who darted into his path. The child’s mother files suit against the man. At trial, the mother is then called to testify as to her conversation with the man at the hospital where her son was rushed immediately following the accident. [Intervention text is inserted in this position.] According to the mother, the man told her, “Don’t worry, my insurance company will pay for all of your son’s medical expenses.” If offered into evidence, the court will most likely rule that the mother’s statement is
OptionsA. admissible, on the issue of the defendant’s negligence.<br>B. admissible, to show that the defendant is able to pay for the medical expenses.<br>C. inadmissible, because the testimony was highly prejudicial.<br>D. admissible, only to prove ownership or control of the motorcycle.
Intervention text
ParaphraseAt trial, the mother is then asked to testify about her conversation with the man at the hospital where her son was taken immediately after the accident.
Self-explanationThe mother’s testimony is crucial in establishing the man’s admission of liability and potential negligence in the accident.
Teacher-explanationThis means that during the court proceeding, the plaintiff (the mother) takes the witness stand and is asked to describe what the man said to her when they spoke at the hospital shortly after her son was taken there because of the accident.

Table 1. Example of intervention texts for an MMLU question. Each intervention is inserted at the marked position; the correct option is in bold.

Our framework evaluates each method in two ways: by its accuracy, and by a sign-randomisation test that assesses how reliably an accuracy gain reflects improvement. The test examines whether a gain stems from moving the model toward the correct answer given the context, or from perturbing its predictions without a consistent direction (§ 3.2.2). This distinction is necessary because such undirected perturbation can itself raise accuracy, reproducibly. We further analyse individual sentence-level interventions to understand which predictions each method changes (§6.1), at how many positions in a context (§6.2), and whether successful positions can be identified in advance (§6.3).

Our findings are:

  • Perturbations are undirected. Accuracy gains from context augmentation methods do not always reflect a directed improvement (§ 5). Particularly for methods with less information, they often arise from random changes, which we refer to as undirected perturbation. The effect of a method also varies considerably across models and datasets.

  • Interventions act near the decision boundary. Context augmentation methods with less information mainly change predictions that lie close to the decision boundary (§6.1). Whether they flip a prediction toward or away from the correct answer, they shift the prediction by similar amounts.

  • Intervention position is important. Incorrect and correct predictions can be flipped from half of the positions in context (§6.2). Successful positions, however, cannot be predicted in advance, neither from lexical characteristics of a sentence nor by using the last position, where content is commonly added (§6.3). Identifying successful positions remains an open problem for future work.

Related Work

Context augmentation methods modify the input to enhance model capability. For example, inserting filler tokens expands the computational space before answer generation, providing the model with additional computation to better process the input text ([15, 4, 9]). Input repetition, on the other hand, enriches the model’s internal representations by enabling bidirectional-like computation ([17]), and has been shown to improve performance on downstream language understanding tasks ([25, 11]).

Another line of work augments the context with additional knowledge rather than restructuring it. Retrieval-augmented generation (RAG) prepends retrieved passages to the input and has become a dominant paradigm for knowledge-intensive tasks ([12, 27]). Beyond retrieval from external corpora, several studies leverage the model itself or a better-performing model as a knowledge source, generating explanations or elaborations first and then incorporating them back into the input text ([14, 1, 13]). These methods also differ in where they intervene in the input: over the whole input ([25, 11]), between the context and the question ([14, 13]), or at the end of the input ([15, 6]).

Zhang et al. [28] examine how robust models are to task-irrelevant context by prepending it to benchmark questions, including semantically meaningless pseudo-words. They find that aggregate accuracy remains nearly unchanged while individual predictions flip in both directions, and call for per-example evaluation. Their study covers only the least informative kind of intervention; we compare it with methods that add increasing amounts of information and examine how their effects differ.

Overall, existing methods have been developed and evaluated in isolation and only by their performance gains. Because they intervene at different positions, practitioners have little guidance on when to apply which method and where. We address this gap with an evaluation framework that compares methods under the same conditions and analyses.

Benchmarking Context Augmentation Methods

Context augmentation methods intervene on the input text at inference time. We focus on methods that insert additional content into the input to improve context utilisation. These methods differ in how informative the inserted content is. Some add no new information and merely increase the amount of computation the model performs before answering, for example by inserting semantically irrelevant filler tokens [15, 6]. Others supply knowledge relevant to the question.

Existing methods have been proposed and evaluated in isolation, without systematic comparison. To compare them under the same conditions, we apply each method at the sentence level, standardising the intervention position across methods and allowing us to examine whether position matters. Our evaluation framework covers five methods, chosen to vary in how much information the inserted content carries, allowing us to compare how different amounts of information affect the model.

Context Augmentation Methods

For a context c=[s1,…,sn,…,sNc]c = [s_1, \ldots, s_n, \ldots, s_{N_c}] containing NcN_c sentences, an intervention function I\mathcal{I} targets a sentence sns_n by appending an intervention ini_n after it. Formally, I(c,sn)=[s1,…,sn,in,…,sNc]\mathcal{I}(c, s_n) = [s_1, \ldots, s_n, i_n, \ldots, s_{N_c}]. Each context is followed by a question, and the model outputs a token distribution over the vocabulary. Specifically, we compute the log probability of all possible answers and select the one with the highest log probability as the model answer. Our intervention approaches include five types: Blank, Repeat, Paraphrase, Self-explanation, and Teacher-explanation, which are described below.

Blank We append semantically irrelevant filler tokens after sns_n, increasing the amount of computation the model performs over the context without adding any information.

Repeat A verbatim copy of sns_n is added. Beyond adding computation, repetition reinforces the original contextual information without introducing additional knowledge. This follows work showing that re-reading or repeating the input improves performance [25, 11].

Paraphrase We insert a paraphrase of sns_n that preserves its meaning but uses different wording. Compared with Repeat, the content is the same, but the surface form differs, allowing us to test whether restating information in different words affects the model beyond repetition.

Self-explanation We insert an explanation of sns_n generated by the evaluated model itself. Since the relevant knowledge the model holds in its parameters is only implicit, generating it as text enables us to place it in the context, where the model can attend to it when answering [26, 18, 13]. Unlike Paraphrase, this adds information beyond the context, but only information the model already has.

Teacher-explanation We insert an explanation of sns_n generated by a larger teacher model. It adds computation and information in the same way as Self-explanation, but from a more capable model, and can therefore supply knowledge the evaluated model lacks.

Evaluation Metrics

Each intervened input contains a single intervention applied after one of the original sentences. A context cc with NcN_c sentences thus yields NcN_c separate intervened inputs per method, all paired with the same question. We look at individual intervened inputs to understand when and how each method affects the model prediction.

Sentence-Averaged Accuracy

The sentence-averaged accuracy of a method is the fraction of its NcN_c intervened inputs that are answered correctly, averaged over contexts:

Accsent=1∣D∣∑c∈D1Nc∑n=1Ncy(I(c,sn)),(1)\mathrm{Acc}_{\mathrm{sent}} = \frac{1}{|\mathcal{D}|} \sum_{c \in\mathcal{D}} \frac{1}{N_c} \sum_{n=1}^{N_c} y(\mathcal{I}(c,s_n)), \tag*{(1)}

where D\mathcal{D} is the set of contexts and y(x)∈{0,1}y(x) \in\{0,1\} indicates whether the model answers the question correctly given input xx. Without intervention, the accuracy is

Accorig=1∣D∣∑c∈Dy(c).(2)\mathrm{Acc}_{\mathrm{orig}} = \frac{1}{|\mathcal{D}|} \sum_{c \in\mathcal{D}} y(c). \tag*{(2)}

The sentence-averaged accuracy decomposes as Accsent=Accorig+Corr−Brk\mathrm{Acc}_{\mathrm{sent}} = \mathrm{Acc}_{\mathrm{orig}} + \mathrm{Corr} - \mathrm{Brk}, where Corr is the fraction of intervention positions at which the prediction flips from incorrect to correct, and Brk is the fraction at which it flips from correct to incorrect, both averaged over contexts in the same way as Accsent\mathrm{Acc}_{\mathrm{sent}}.

Sign-Randomisation Test

The sign-randomisation test examines how reliably an accuracy gain reflects improvement. An accuracy gain can also arise from perturbing the model’s predictions without a consistent direction. Because such a gain is deterministic and reproducible, neither standard significance tests nor variance across random seeds can separate it from a genuine improvement. We therefore test whether a method moves the model toward the correct answer more consistently than undirected perturbations of the same size.

For each input, we compute the margin mm between the log-probability of the correct option o∗o^* and that of the highest-scoring incorrect option:

m=log⁡p(o∗)−max⁡o≠o∗log⁡p(o),(3)m = \log p(o^*) - \max_{o \ne o^*} \log p(o), \tag*{(3)}

The margin is positive when the model answers correctly and negative when the answer is incorrect. For a context cc, let mcm_c denote the margin without intervention and mc,nm_{c,n} the margin for I(c,sn)\mathcal{I}(c,s_n), i.e., with the intervention applied after sentence sns_n. The change in margin caused by the intervention is Δc,n=mc,n−mc\Delta_{c,n} = m_{c,n} - m_c.

We set a null hypothesis that a method changes the margin without a consistent direction: it corrects some predictions and breaks others, so that the sign of each context’s changes is arbitrary. We refer to such changes as undirected perturbations. Rejecting the null hypothesis indicates that the method moves the model toward the correct answer more than expected by chance.

For each intervention position, we draw a random sign σc,n∈{−1,+1}\sigma_{c,n} \in\{-1,+1\} with equal probability and apply it to its change, giving sign-randomised margins m~c,n=mc+σc,nΔc,n\widetilde{m}_{c,n} = m_c + \sigma_{c,n}\Delta_{c,n}. This preserves the magnitude of each change and removes only its direction. We recompute Corr and Brk on the sign-randomised margins and repeat this procedure B=1000B = 1000 times to obtain a random distribution for each.

A method rejects the null hypothesis when its observed Corr exceeds the 95% interval of its random distribution while its observed Brk does not, i.e., it corrects incorrect predictions more often than undirected perturbations of the same size, without breaking correct predictions more often. An undirected perturbation that pushes predictions away from the current answer raises both Corr and Brk, and therefore does not reject the null hypothesis.

Experimental Setup

We apply our evaluation framework for context augmentation methods to seven open-weight models across three different datasets.

Datasets

As our evaluation framework examines where in the context an intervention should be applied, we select datasets where gold answers are not explicit in contexts. If a single sentence contained the answer, it would trivially be the most effective position to intervene. To this end, we study our framework with MedQA ([10]), an MCQA dataset for the medical domain, MMLU ([8]), an MCQA dataset for world knowledge, and DRUID ([7]), a fact-checking dataset. For those datasets, the context provides background that is necessary to understand the question but does not reveal the answer.

Most MMLU questions consist of a single short sentence without an accompanying context, which leaves no sentence positions at which to intervene. We therefore restrict MMLU to five subjects whose questions include a multi-sentence passage before the question itself. For DRUID, we exclude claims with insufficient evidence, since a method can fail because of context quality, not the method itself. We provide details for each dataset and a subject selection for MMLU in Appendix A.1.

Implementation of Interventions

For the Blank intervention, we set the intervention text ini_n to a sequence of whitespace characters matching the character length of sns_n. For Repeat, we set in=sni_n = s_n, repeating the sentence. We obtain paraphrased sentences for Paraphrase by strictly prompting GPT-5.4-Mini2, without introducing additional information. For Self-explanation, ini_n is an elaboration of sns_n that is generated by the same model being evaluated. For generating an intervention text for Teacher-explanation, we query the teacher model to generate an elaboration of sns_n. We use the same GPT-5.4-Mini model as above. To prevent explanations from offering a shortcut to the answer (e.g. simply stating the correct answer), the generating model sees only the context, without the question or answer options. We use the same input prompt for querying self-explanation. We provide the prompts used to generate intervention texts in Appendix A.2. Table 1 shows an example of intervention texts with original context and question. For all methods, we utilise the existing evaluation scheme, which standardises evaluation across different datasets ([2]).

Models

We evaluate seven open-weight LLMs from four model families for reproducibility: OLMo-2-0425-1B-Instruct (OLMo-1b, (Team OLMo et al., 2025)), OLMo-3-7B-Instruct (OLMo-7b, (Team Olmo et al., 2025)), Llama-3.2-3B-Instruct (Llama-3b, [5]), Meta-Llama-3.1-8B-Instruct (Llama-8b, [5]), Gemma-3-4b-it (Gemma-4b, [3]), Gemma-3-27b-it (Gemma-27b, [3]), and Qwen2.5-72B-Instruct-AWQ (Qwen-72b, [16]).

Results

Sentence-Averaged Accuracy Results

Table 2 reports sentence-averaged accuracy of the context augmentation methods (§ 3.1) on MedQA.

ModelOriginalBlankRepeatParaphraseSelf-expl.Teacher-expl.
OLMo-1b21.224.7 +18.8/−15.3+18.8/-15.324.4 +19.9/−16.7+19.9/-16.724.4 +19.9/−16.7+19.9/-16.724.9 +18.4/−14.7+18.4/-14.725.3 +24.1/−20.0+24.1/-20.0
Llama-3b52.947.6 +5.4/−10.7+5.4/-10.747.6 +5.9/−11.1+5.9/-11.148.2 +6.2/−11.0+6.2/-11.049.6 +10.1/−13.4+10.1/-13.454.9 +12.4/−10.4+12.4/-10.4
Gemma-4b45.746.6 +2.1/−1.1+2.1/-1.146.8 +2.8/−1.7+2.8/-1.746.9 +3.0/−1.8+3.0/-1.848.1 +8.7/−6.2+8.7/-6.255.0 +14.1/−4.7+14.1/-4.7
OLMo-7b40.640.8 +3.2/−3.0+3.2/-3.040.8 +3.5/−3.2+3.5/-3.240.7 +3.5/−3.3+3.5/-3.343.0 +9.4/−7.0+9.4/-7.047.6 +15.5/−8.5+15.5/-8.5
Llama-8b53.646.3 +5.2/−12.5+5.2/-12.543.2 +3.9/−14.4+3.9/-14.443.6 +4.2/−14.2+4.2/-14.249.3 +9.4/−13.7+9.4/-13.752.2 +10.7/−12.1+10.7/-12.1
Gemma-27b67.067.7 +1.3/−0.7+1.3/-0.767.7 +2.0/−1.3+2.0/-1.367.5 +1.9/−1.4+1.9/-1.469.5 +5.9/−3.4+5.9/-3.473.9 +9.7/−2.8+9.7/-2.8
Qwen-72b73.473.4 +1.4/−1.4+1.4/-1.472.7 +1.2/−1.9+1.2/-1.972.7 +1.4/−2.0+1.4/-2.073.7 +3.4/−3.1+3.4/-3.176.8 +6.3/−2.9+6.3/-2.9

Table 2. Sentence-averaged accuracy (%) of each method on MedQA. Original reports an accuracy without intervention. Bold numbers mark accuracies higher than the Original. Small numbers with + are the fraction of intervention positions at which the prediction flips from incorrect to correct (Corr), and those with − are the fraction of the positions at which the prediction flips from correct to incorrect (Brk). We colour cells where the method corrects wrong predictions more often than expected under undirected perturbation, according to the sign-randomisation test (§ 5.2).

OLMo-1bLlama-3bGemma-4bOLMo-7bLlama-8bGemma-27bQwen-72b
Blank-■•, ■-▲--
Repeat■■---•■
Paraphrase-----•■
Self-explanation■--▲▲, ■-•, ■
Teacher-explanation-•, ▲, ■--▲, ■••, ■

Table 3. Sign-randomisation test results across all context augmentation methods and models. We mark the datasets on which the observed corrected proportion exceeds the 95% interval of the null distribution while the broken proportion does not. • denotes the MedQA dataset, ▲ MMLU, ■ DRUID. - indicates that no dataset passes the test.

Tables 8 and 9 (Appendix § 10) report sentence-averaged accuracies on MMLU and DRUID. Methods that add new information yield the largest gains on MedQA. Teacher-explanation improves sentence-averaged accuracy over Original for six of seven models, by 2.0 to 9.3 points, and Self-explanation by 0.3 to 3.7 points for five models. However, the uncoloured cells show that these gains are not necessarily directed. The largest gains, Teacher-explanation with Gemma-4b (+9.3) and OLMo-7b (+7.0), do not reject the null hypothesis. Although Corr exceeds Brk, because Brk significantly exceeds what undirected perturbations of the same size would produce, the gain does not reflect a consistent move toward the correct answer.

Methods that add no content (Blank, Repeat, and Paraphrase) barely change sentence-averaged accuracy for most models: Corr and Brk are of similar size and cancel each other out (e.g., +3.2/−3.0 for Blank with OLMo-7b). Accuracy alone cannot distinguish such cancelling from a method having no effect: for example, Teacher-explanation with OLMo-7b on MMLU changes sentence-averaged accuracy by only 0.1 points, while changing the prediction for 19.1% of intervention positions.

Overall, applying context augmentation at the sentence level shows that accuracy gains are inconsistent across models and datasets, and that many of them arise from undirected perturbation rather than from moving the model toward the correct answer. This highlights the need to complement accuracy-based evaluation with tests of direction, such as our sign-randomisation test.

Sign-Randomisation Test Results

Table 3 marks the datasets on which a method corrects wrong predictions more often than expected under undirected perturbation. Methods that add new information reject the null hypothesis more often, indicating they move the models toward the correct answer more than expected by chance. Teacher-explanation does so in 8 of 21 model–dataset pairs and Self-explanation in 6, compared with 4 for Repeat and 4 for Blank. Results also vary across models: while undirected perturbation prevails for OLMo models, Qwen-72b and Llama models reject the null hypothesis most often.

This is consistent with prior observations that context interventions can change predictions in both directions [13, 28]. Appendix § 10 reports random distributions for both Corr and Brk, including the difference between observed and null accuracy.

Analysis

We analyse individual interventions on each sentence to understand for which instances (§6.1), how consistently (§6.2), and at which positions in the context (§6.3) each method changes the model’s prediction.

Which Instances are Affected?

We examine how the margin (Equation (3)) changes from Original when a context augmentation method is applied at a single sentence position to understand which instances the methods affect. We classify each intervention as Rescue, Break, or Unchanged. Rescue is when an intervention turns an incorrect prediction into a correct one. Break is when an intervention turns a correct prediction into an incorrect one. Unchanged is when an intervention does not change the model’s prediction, including both correct and incorrect ones.

Figure 2 presents results for Blank and Teacher-explanation on MedQA. Results for other methods and datasets are in Appendix § 11.§ 1. For most models on MedQA, Rescue and Break involve instances near the decision boundary (dashed line, margin = 0). We also note that both categories change the margin by similar amounts in opposite directions. This trend holds across all datasets. This implies that methods change predictions mainly where the model is already uncertain. In contrast, Unchanged groups lie far from the boundary, and their margins shift only slightly on average.

Margin changes before and after intervention for Blank and Teacher-explanation across seven models

Figure 2. We show how margin changes before (Original, hollow circles) and after (filled circles) intervention, averaged within each category on MedQA across seven models with Blank and Teacher-explanation. For each model, the first row and the second row are Rescue and Break and the last two rows are Unchanged.

Comparing methods shows that margin-size changes depend on the method. Knowledge-oriented methods shift the margin more strongly than computation-oriented methods, in both directions.

How Many Positions are Affected?

Sentence-averaged accuracy shows how often a single intervention leads to the correct answer, but not how many opportunities exist within a context. This determines how much a method’s advantage depends on choosing the position. If most positions succeed, any choice is likely to work, whereas if only one works, identification of the position becomes important.

We measure multiplicity of methods along two axes, Rescue and Break. In Table 4, the Rescue column shows the fraction of positions at which an intervention turns an incorrect prediction into a correct one, among contexts rescued at least once. The Break column shows the fraction of positions at which it turns a correct prediction into an incorrect one, among contexts broken at least once. We average across all methods and models; the detailed results are presented in Appendix § 11.§ 2.

DatasetRescue rateBreak rate
MedQA0.553±0.180.553 \pm0.180.569±0.230.569 \pm0.23
MMLU0.422±0.090.422 \pm0.090.422±0.090.422 \pm0.09
DRUID0.459±0.110.459 \pm0.110.411±0.110.411 \pm0.11

Table 4. Multiplicity analysis. Rescue rate: fraction of sentence positions at which an intervention rescues the prediction, among contexts rescued at least once. Break rate: the same for breaking, among contexts broken at least once. Values are means ±\pm standard deviations over all methods and models.

The Rescue fraction is similar to the Break fraction across datasets and models. The high standard deviation on MedQA is mainly due to OLMo-1b (Table 10). Once a context can be affected, the methods change the prediction at about half of its sentence positions, regardless of whether the original prediction was correct. The chosen position, therefore, determines whether a method helps or harms, so knowing which positions are likely to succeed is important to maximise its benefit. This motivates the question of whether successful positions can be identified in advance (§6.3).

BlankRepeatParaphraseSelf-explanationTeacher-explanation
ModelRescueBreakRescueBreakRescueBreakRescueBreakRescueBreak
OLMo-1b0.9220.9220.9160.9150.8970.9030.8160.8230.9580.965
Llama-3b0.7470.8830.6460.8130.6280.7870.5600.6240.5210.555
Gemma-4b0.4490.3730.3900.3930.3800.3390.4710.4090.4570.341
OLMo-7b0.5490.5210.4750.4640.4190.4630.4210.4120.4610.451
Llama-8b0.7610.8950.7010.8440.6790.8250.5500.6360.5040.637
Gemma-27b0.3780.3300.3770.3300.3370.3450.4060.3000.4650.279
Qwen-72b0.5350.4980.3840.4930.3760.4440.3670.3690.4420.333

Table 10. Multiplicity analysis on MedQA.

Can Positions be Identified?

We examine if a successful position can be identified from two perspectives: lexical characteristics of the target sentence and a fixed position.

Lexical characteristics. We first look at lexical characteristics of a sentence, which are cheap to obtain, as they require no model access. We follow the taxonomy of context characteristics of Hagström et al. [7], apply it at the sentence level, and add coreference characteristics, as context augmentation methods may help the model resolve references across sentences. Table 6 lists ten characteristics used in the analysis. We extract coreference chains with the coreferee library.3 For each characteristic, we compute AUROC across all interventions: the probability that a sentence where the intervention leads to Rescue (or Break) has a higher value than another sentence across all contexts where it does not. All AUROCs are near chance (0.48–0.54; Table 16), indicating that lexical characteristics alone cannot identify successful intervention positions. We describe results in detail in Appendix § 11.§ 3.

CharacteristicDescription
LengthNumber of characters in the sentence.
OverlapJaccard similarity between the content words of the sentence and of the question (MedQA, MMLU) or claim (DRUID).
ReadabilityFlesch reading ease score.
Named entitiesNumber of named entities in the sentence.
NegationWhether the sentence contains a negation (a syntactic negation or a negation cue such as no, not, without).
PronounsNumber of pronouns in the sentence.
Pronoun ratioShare of tokens in the sentence that are pronouns.
Coref. mentionsNumber of mentions in the sentence that belong to a coreference chain in the context.
Coref. chainsNumber of distinct coreference chains with a mention in the sentence.
Backward refs.Number of mentions in the sentence whose chain was first mentioned in an earlier sentence.

Table 6. Sentence-level lexical characteristics. The first five follow Hagström et al. [7]; the last five capture references across sentences. Coreference is resolved over the whole context with coreferee.

CharacteristicBlank RBlank BRepeat RRepeat BParaphrase RParaphrase BSelf-explanation RSelf-explanation BTeacher-explanation RTeacher-explanation B
Length0.510.510.540.520.530.520.500.490.510.49
Overlap0.510.510.520.520.520.520.500.500.510.51
Readability0.490.500.490.500.480.500.500.510.500.51
Named entities0.510.500.530.520.530.520.510.500.510.50
Negation0.480.500.490.500.490.490.480.490.480.49
Pronouns0.490.500.500.490.500.490.490.480.490.49
Pronoun ratio0.490.490.490.490.490.490.490.480.480.49
Coref. mentions0.510.520.520.520.510.520.500.510.510.51
Coref. chains0.510.520.520.520.510.520.500.510.510.51
Backward refs.0.510.510.510.520.510.520.500.510.500.51

Table 16. AUROC (mean over seven models and three datasets) of each lexical characteristic for identifying intervention positions that lead to a Rescue (R) or a Break (B). AUROCs are computed over all intervention positions pooled across contexts; 0.5 indicates no relation.

Fixed position. We compare accuracy when intervening at the last sentence, directly before the question, with accuracy at a randomly selected position among the rest of the positions except the last position (Random non-last position). We randomly select a position three times and average the accuracy. For every method, model, and dataset, performance at the last position is mostly on par with the other positions, with all points lying on the diagonal except for some Teacher-explanation results (Figure 3). Although intervening at the end of the input is common practice (Liu et al. [14]), our results suggest that other positions perform as well on average.

Accuracy when intervening after the last sentence versus after a randomly selected non-last sentence

Figure 3. Accuracy when intervening after the last sentence (y-axis) versus after a randomly selected non-last sentence (x-axis; mean over three random draws). Points close to the dashed diagonal indicate that the last position is no better than other positions.

Together with the multiplicity analysis (§6.2), these results show that successful positions are plentiful but cannot be identified in advance, neither from a sentence’s content nor from its location.

Discussion & Conclusion

Prior work reports that context augmentation methods, from inserting filler tokens to adding knowledge, improve performance ([14, 15, 13]). Our sentence-averaged accuracy results (Table 2, 8, and Table 9) also show accuracy gains, though not in all settings. This is consistent with Li et al. [13], who find that the benefit of self-generated documents, a variant of

ModelOriginalBlankRepeatParaphraseSelf-expl.Teacher-expl.
OLMo-1b82.482.2 +1.3/−1.582.5 +2.5/−2.482.5 +2.4/−2.383.1 +4.6/−4.080.8 +2.5/−4.2
Llama-3b71.171.4 +1.4/−1.172.0 +2.3/−1.471.6 +2.1/−1.771.1 +1.8/−1.972.1 +2.6/−1.6
Gemma-4b87.187.1 +1.0/−1.086.9 +1.3/−1.586.6 +1.2/−1.787.0 +1.5/−1.686.9 +1.9/−2.1
OLMo-7b85.081.7 +2.1/−5.382.3 +2.0/−4.782.0 +2.1/−5.181.2 +2.1/−6.077.0 +2.6/−10.6
Llama-8b79.479.2 +1.1/−1.379.2 +1.5/−1.779.3 +1.6/−1.780.3 +2.3/−1.480.3 +2.4/−1.5
Gemma-27b89.789.5 +0.3/−0.589.6 +0.6/−0.789.4 +0.5/−0.889.4 +0.7/−1.089.2 +0.7/−1.2
Qwen-72b84.884.5 +0.2/−0.585.2 +0.8/−0.485.1 +0.8/−0.485.5 +1.1/−0.485.3 +1.0/−0.5

Table 9. Sentence-averaged accuracy (%) of each method on DRUID.

Self-explanation, depends on the document format and dataset. Our results extend this to other methods and also show that a method’s effect is inconsistent across datasets and depends on the model.

However, accuracy gains alone cannot distinguish a method that guides the model toward the correct answer from one that merely perturbs its prediction. Zhang et al. [28] make a related point, showing that predictions resulting from task-irrelevant interventions flip in both directions even when aggregate accuracy is stable. Our sign-randomisation test shows that such undirected perturbation is not limited to methods that only add computation, but also occurs for methods that add content. Furthermore, our multiplicity analysis offers a finer-grained view of the findings of Zhang et al. [28]: once a prediction can be changed, it is rescued or broken at similar fractions of sentence positions (§6.2).

Context augmentation methods should therefore not be applied blindly. Our framework allows researchers and practitioners to verify whether a method’s gains are directed and how often it breaks correct predictions. Our results favour methods that add new information, validated for each model family and dataset. Since the chosen position determines whether an intervention corrects or breaks a prediction, identifying successful positions in advance, and thereby maximising the benefits of context augmentation methods, remains an open direction for future work.

Limitations

Our datasets are multiple-choice and binary classification tasks, and we score answer options by log-likelihood rather than generating free-form text. Our findings may therefore not transfer to open-ended generation. We also apply one intervention at a time, after a single sentence, to isolate the effect of each position. Prior work typically applies these methods once to the whole context, and gains may differ in that setting or when several interventions are combined. Our evaluation framework, however, can be extended to any intervention that changes the model’s margin.

We evaluate seven instruction-tuned open-weight models from 1B to 72B parameters. To avoid confounding the interventions with model-generated reasoning, we do not include reasoning models, whose own chain of thought may interact with the inserted text. All datasets are in English.

We used AI assistants for grammar checking and stylistic suggestions, and for help with writing analysis and plotting code. No generated text was used without revision, and all code and results were reviewed and checked by the authors.

Acknowledgements

This research was funded by the European Union (ERC, ExplainYourself, 101077481) and supported by the Pioneer Centre for AI, DNRF grant number P1.

Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

Experimental Details

Dataset Details

MedQA is a multiple-choice Question Answering (MCQA) dataset in which a model is required to select the correct option given a patient description as context and a question. MMLU is an MCQA dataset, which examines models’ world knowledge and problem-solving ability. Finally, DRUID is a fact-checking dataset containing claims and evidence as contexts, retrieved from the real world. We only take instances with support and refute labels, making it a binary classification.

DatasetNumber of QuestionsMean ± SDTotal sentences
MedQA1,2707.09±3.287.09 \pm3.288,998
MMLU2,3106.52±3.846.52 \pm3.8415,063
DRUID2,6695.72±3.255.72 \pm3.2515,269

Table 5. Dataset statistics. Sentences per context are given as mean ± standard deviation; the total number of sentences equals the number of intervention positions per model and method.

For MMLU, we select the five subjects whose questions exceed 500 characters on average: high school European, US and world history, professional law, and professional medicine (654–1,391 characters). All remaining subjects average below 330 characters, so the threshold clearly separates the two groups.

Implementation of Interventions

Table Table 7 presents prompts used to generate intervention texts for Paraphrase, Self-explanation, and Teacher-explanation. Sentences are segmented with spaCy (en_core_web_sm).4 For our experiments, each run (one model, method, and dataset) took one to two hours on an AMD Instinct MI250X GPU. For Qwen-72b, we used an NVIDIA B200 GPU, which took 0.5 hours per run.

ModelOriginalBlankRepeatParaphraseSelf-expl.Teacher-expl.
OLMo-1b26.827.1 +2.2/−1.827.4 +2.7/−2.127.3 +3.0/−2.427.6 +4.0/−3.125.9 +9.8/−10.6
Llama-3b54.053.6 +1.2/−1.653.4 +2.0/−2.654.0 +3.4/−3.353.3 +4.0/−4.755.3 +5.7/−4.3
Gemma-4b52.251.4 +1.2/−2.051.7 +1.7/−2.351.4 +1.7/−2.551.7 +3.5/−4.053.8 +6.1/−4.5
OLMo-7b49.349.1 +1.4/−1.649.2 +2.0/−2.149.4 +2.7/−2.650.2 +5.4/−4.549.4 +9.6/−9.5
Llama-8b52.653.1 +1.4/−1.052.2 +1.6/−2.052.9 +1.9/−1.753.8 +4.3/−3.254.5 +4.8/−2.9
Gemma-27b67.467.1 +1.0/−1.366.9 +1.1/−1.667.1 +1.5/−1.967.2 +3.4/−3.768.2 +4.7/−3.9
Qwen-72b74.674.5 +0.7/−0.874.5 +1.1/−1.274.3 +1.2/−1.573.5 +1.8/−3.074.4 +2.9/−3.1

Table 7. We present prompts used for generating intervention texts for Paraphrase, Self-explanation, and Teacher-explanation. Sentence-averaged accuracy (%) of each method on MMLU. Original reports an accuracy without intervention. Small numbers with + are the fraction of intervention positions at which the prediction flips from incorrect to correct (Corr), and those with − are the fraction of the positions at which the prediction flips from correct to incorrect (Brk). We colour cells where the method corrects wrong predictions more often than the 95% interval of the random distribution, without breaking correct predictions more often than the 95% interval of the random distribution.

Results

Tables 8 and 9 show sentence-averaged accuracy results on MMLU and DRUID, respectively. On MMLU, context augmentation methods reject the null hypothesis 5 out of 35 times, while DRUID shows the most rejections: 12 out of 35. Teacher-explanation shows the most frequent accuracy gains on both datasets. On MMLU, methods that add only computation do not consistently improve performance.

Tables 13, 14 and 15 show detailed sign-randomisation test results.

Analysis

Margin Analysis Results

Figures 4, 5, 6, 7, and 8 show margin analysis results that are not presented in the main paper. Methods that add both computation and new content (Self-explanation and Teacher-explanation) yield bigger changes in margins than the methods that add only computation.

Multiplicity Analysis

Tables 10, 11 and 12 show detailed multiplicity results. We observe similar Rescue and Break rates across models and methods.

Predictability with Lexical Characteristics

We compute all characteristics on exactly the sentences after which the interventions are inserted, so that each value corresponds to one intervention position.

Computation of Characteristics. Length is the number of characters in the sentence. Overlap is the Jaccard similarity between the sets of lowercased word tokens of the sentence and of the question (MedQA, MMLU) or claim (DRUID), after removing stopwords. Readability is the Flesch reading ease score, computed with the textstat library.5 Named entities is the number of entities Input prompt for Paraphrase

Paraphrase the following sentence. Only output the paraphrased sentence. \n\n Sentence: %s

Input prompt for Self-explanation and Teacher-explanation

Generate additional knowledge to understand the sentence marked with << >> in the [Context]. Only generate a sentence. \n\n Context: %s

recognised by spaCy (en_core_web_lg).6 Negation indicates whether the sentence contains a syntactic negation (a neg dependency) or a lexical negation cue (e.g., no, not, without, denies). Pronouns is the number of pronoun tokens in the sentence, and pronoun ratio their share among all tokens. For the coreference characteristics, we run coreferee over the full context and assign each mention to the sentence containing it. Coref. mentions is the number of mentions in the sentence that belong to a coreference chain, coref. chains the number of distinct chains with a mention in the sentence, and backward refs. the number of mentions whose chain was first mentioned in an earlier sentence.

AUROC. For each characteristic, model, method, and dataset, we label each intervention position by whether the intervention leads to a Rescue (or a Break) and compute the AUROC of the characteristic for separating these positions from all others, pooled across contexts (ties count as 0.5). 95% confidence intervals are obtained by bootstrapping over contexts (2,000 resamples), keeping all positions of a context together.

Table 16 shows AUROC results, where most values are near 0.5, indicating no relation.

Margin analysis results on MedQA with Repeat, Paraphrase, and Self-explanation

Figure 4. Margin analysis results on MedQA with Repeat, Paraphrase, and Self-explanation.

Margin analysis results on MMLU with Blank, Repeat, and Paraphrase

Figure 5. Margin analysis results on MMLU with Blank, Repeat, and Paraphrase.

Margin analysis results on MMLU with Self-explanation and Teacher-explanation

Figure 6. Margin analysis results on MMLU with Self-explanation and Teacher-explanation.

Margin analysis results on DRUID with Blank, Repeat, and Paraphrase

Figure 7. Margin analysis results on DRUID with Blank, Repeat, and Paraphrase.

BlankRepeatParaphraseSelf-explanationTeacher-explanation
ModelRescueBreakRescueBreakRescueBreakRescueBreakRescueBreak
OLMo-1b0.5010.5260.4750.4280.4700.4520.4630.4250.8890.902
Llama-3b0.3940.4470.4340.4390.4580.4630.4120.4390.4190.412
Gemma-4b0.3920.4190.3820.3810.3850.3690.3750.3790.3960.359
OLMo-7b0.3360.4360.4170.4270.3970.4030.4140.4200.4740.489
Llama-8b0.4300.3950.4330.4180.4000.3770.4330.3840.3740.348
Gemma-27b0.4480.4310.3620.4430.3820.4250.3660.3650.3880.364
Qwen-72b0.3730.3370.3860.4000.3610.3650.3750.3940.3650.324

Table 11. Multiplicity analysis on MMLU.

BlankRepeatParaphraseSelf-explanationTeacher-explanation
ModelRescueBreakRescueBreakRescueBreakRescueBreakRescueBreak
OLMo-1b0.3740.3920.4210.4520.3840.3830.4640.3990.4450.505
Llama-3b0.4370.4240.4370.3920.4360.4200.4210.4250.4430.386
Gemma-4b0.3770.3360.4410.3900.3960.3720.4130.3490.4560.358
OLMo-7b0.7440.7200.6480.6370.7050.6200.6340.6260.7500.678
Llama-8b0.4370.3530.4360.3950.4170.3670.4710.3600.4610.347
Gemma-27b0.3050.2860.4100.3530.3800.3150.3810.3370.4090.340
Qwen-72b0.4560.3920.4010.3440.3930.3490.4210.2840.4430.297

Table 12. Multiplicity analysis on DRUID.

MethodModelCorr Obs.Corr NullCorr 95% intervalCorr Δ\DeltaBrk Obs.Brk NullBrk 95% intervalBrk Δ\DeltaPass
BlankOLMo-1b18.89.0[8.5, 9.5]+9.815.37.3[6.9, 7.7]+8.0
Llama-3b5.47.8[7.4, 8.3]−2.410.76.9[6.4, 7.4]+3.8
Gemma-4b2.11.7[1.5, 1.9]+0.41.11.1[0.9, 1.2]+0.1✓
OLMo-7b3.22.7[2.4, 3.0]+0.53.02.5[2.3, 2.8]+0.4
Llama-8b5.26.3[5.9, 6.7]−1.212.59.1[8.6, 9.6]+3.4
Gemma-27b1.31.2[1.0, 1.4]+0.10.70.9[0.8, 1.0]−0.2
Qwen-72b1.41.2[1.0, 1.3]+0.21.41.1[0.9, 1.3]+0.3
RepeatOLMo-1b19.99.6[9.0, 10.1]+10.316.78.0[7.6, 8.5]+8.7
Llama-3b5.98.2[7.8, 8.7]−2.411.17.4[7.0, 7.9]+3.7
Gemma-4b2.82.3[2.0, 2.5]+0.61.71.3[1.1, 1.5]+0.4
OLMo-7b3.53.0[2.7, 3.3]+0.53.22.6[2.3, 2.9]+0.6
Llama-8b3.95.9[5.5, 6.3]−2.014.49.9[9.3, 10.4]+4.5
Gemma-27b2.01.6[1.4, 1.9]+0.31.31.2[1.1, 1.4]+0.0✓
Qwen-72b1.21.3[1.1, 1.5]−0.11.91.4[1.2, 1.6]+0.5
ParaphraseOLMo-1b19.99.6[9.1, 10.1]+10.316.78.1[7.7, 8.6]+8.6
Llama-3b6.29.1[8.7, 9.6]−2.911.08.3[7.8, 8.8]+2.7
Gemma-4b3.02.4[2.1, 2.7]+0.61.81.4[1.2, 1.5]+0.4
OLMo-7b3.53.1[2.8, 3.4]+0.43.32.7[2.4, 3.0]+0.6
Llama-8b4.25.9[5.5, 6.3]−1.714.29.7[9.2, 10.3]+4.4
Gemma-27b1.91.6[1.4, 1.8]+0.31.41.2[1.1, 1.4]+0.1✓
Qwen-72b1.41.4[1.2, 1.6]+0.02.01.5[1.3, 1.7]+0.5
Self-explanationOLMo-1b18.49.0[8.5, 9.5]+9.414.77.1[6.6, 7.5]+7.6
Llama-3b10.110.9[10.4, 11.4]−0.813.410.5[10.0, 11.0]+2.9
Gemma-4b8.75.7[5.3, 6.1]+3.06.24.0[3.6, 4.3]+2.3
OLMo-7b9.47.9[7.4, 8.4]+1.57.06.0[5.6, 6.4]+1.0
Llama-8b9.48.2[7.8, 8.7]+1.213.710.9[10.3, 11.4]+2.9
Gemma-27b5.93.8[3.5, 4.2]+2.03.42.6[2.3, 2.9]+0.8
Qwen-72b3.42.9[2.6, 3.2]+0.53.13.1[2.8, 3.4]+0.0✓
Teacher-explanationOLMo-1b24.112.2[11.7, 12.8]+11.820.010.1[9.6, 10.7]+9.9
Llama-3b12.411.5[10.9, 12.0]+0.910.410.4[9.8, 10.9]+0.0✓
Gemma-4b14.18.2[7.8, 8.7]+5.84.73.5[3.2, 3.8]+1.2
OLMo-7b15.510.5[9.9, 11.0]+5.18.57.0[6.6, 7.5]+1.5
Llama-8b10.78.5[8.0, 9.0]+2.212.110.5[9.9, 11.0]+1.7
Gemma-27b9.75.7[5.3, 6.1]+4.02.82.6[2.4, 2.9]+0.2✓
Qwen-72b6.34.1[3.7, 4.5]+2.22.94.2[3.9, 4.6]−1.3✓

Table 13. MedQA: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000B = 1000 sign-randomised draws, with its 95% interval; Δ\Delta: observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.

MethodModelCorr Obs.Corr NullCorr 95% intervalCorr Δ\DeltaBrk Obs.Brk NullBrk 95% intervalBrk Δ\DeltaPass
BlankOLMo-1b2.22.6[2.4, 2.8]−0.31.82.0[1.8, 2.1]−0.1
Llama-3b1.21.5[1.4, 1.6]−0.31.61.9[1.7, 2.0]−0.2
Gemma-4b1.21.1[0.9, 1.2]+0.12.01.5[1.3, 1.7]+0.5
OLMo-7b1.41.6[1.4, 1.7]−0.11.61.5[1.4, 1.7]+0.1
Llama-8b1.41.2[1.0, 1.3]+0.21.01.6[1.4, 1.8]−0.6✓
Gemma-27b1.00.9[0.7, 1.0]+0.11.31.1[0.9, 1.2]+0.2
Qwen-72b0.70.8[0.7, 0.9]−0.10.80.9[0.7, 1.0]−0.1
RepeatOLMo-1b2.73.0[2.8, 3.2]−0.32.12.4[2.2, 2.6]−0.3
Llama-3b2.01.9[1.8, 2.1]+0.02.62.6[2.4, 2.8]+0.0
Gemma-4b1.71.3[1.1, 1.5]+0.42.31.7[1.5, 1.9]+0.5
OLMo-7b2.02.0[1.8, 2.2]+0.02.11.9[1.7, 2.2]+0.1
Llama-8b1.61.8[1.6, 1.9]−0.12.02.1[1.9, 2.3]+0.0
Gemma-27b1.11.1[0.9, 1.2]+0.01.61.4[1.2, 1.6]+0.2
Qwen-72b1.11.2[1.0, 1.3]−0.11.21.3[1.2, 1.5]−0.1
ParaphraseOLMo-1b3.03.3[3.0, 3.5]−0.32.42.5[2.3, 2.7]−0.1
Llama-3b3.43.1[2.9, 3.4]+0.23.33.7[3.5, 4.0]−0.4
Gemma-4b1.71.4[1.3, 1.6]+0.22.51.9[1.7, 2.0]+0.6
OLMo-7b2.72.6[2.4, 2.9]+0.12.62.4[2.2, 2.6]+0.2
Llama-8b1.91.8[1.6, 2.0]+0.11.72.2[2.0, 2.4]−0.5
Gemma-27b1.51.2[1.1, 1.4]+0.31.91.5[1.3, 1.7]+0.4
Qwen-72b1.21.3[1.2, 1.5]−0.21.51.6[1.4, 1.7]−0.1
Self-explanationOLMo-1b4.04.0[3.7, 4.3]−0.13.13.2[3.0, 3.4]−0.1
Llama-3b4.04.1[3.8, 4.4]−0.14.75.0[4.7, 5.3]−0.3
Gemma-4b3.52.4[2.2, 2.6]+1.04.02.8[2.5, 3.0]+1.2
OLMo-7b5.45.0[4.6, 5.3]+0.54.54.3[4.0, 4.6]+0.2✓
Llama-8b4.33.7[3.5, 4.0]+0.53.24.0[3.7, 4.2]−0.8✓
Gemma-27b3.42.5[2.3, 2.7]+0.93.72.6[2.3, 2.8]+1.1
Qwen-72b1.82.2[2.0, 2.4]−0.43.02.8[2.5, 3.0]+0.2
Teacher-explanationOLMo-1b9.812.3[11.8, 12.8]−2.510.69.9[9.4, 10.3]+0.7
Llama-3b5.74.8[4.5, 5.1]+0.94.35.8[5.5, 6.2]−1.5✓
Gemma-4b6.13.8[3.5, 4.0]+2.34.53.3[3.0, 3.5]+1.2
OLMo-7b9.68.0[7.6, 8.3]+1.79.56.9[6.5, 7.3]+2.6
Llama-8b4.84.0[3.7, 4.2]+0.82.94.1[3.8, 4.4]−1.2✓
Gemma-27b4.73.0[2.8, 3.3]+1.63.92.8[2.6, 3.1]+1.0
Qwen-72b2.92.6[2.4, 2.9]+0.23.13.7[3.5, 4.0]−0.7

Table 14. MMLU: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000B = 1000 sign-randomised draws, with its 95% interval; Δ\Delta: observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.

MethodModelCorr Obs.Corr NullCorr 95% intervalCorr Δ\DeltaBrk Obs.Brk NullBrk 95% intervalBrk Δ\DeltaPass
BlankOLMo-1b1.31.6[1.4, 1.7]−0.31.51.3[1.2, 1.5]+0.2
Llama-3b1.41.1[1.0, 1.3]+0.31.11.3[1.1, 1.4]−0.1✓
Gemma-4b1.00.8[0.6, 0.9]+0.21.01.1[0.9, 1.2]−0.1✓
OLMo-7b2.12.7[2.5, 2.9]−0.75.34.4[4.1, 4.6]+1.0
Llama-8b1.11.1[0.9, 1.2]+0.01.31.5[1.3, 1.7]−0.2
Gemma-27b0.30.3[0.2, 0.4]+0.00.50.5[0.4, 0.5]+0.0
Qwen-72b0.20.3[0.2, 0.3]+0.00.50.3[0.2, 0.4]+0.1
RepeatOLMo-1b2.52.1[1.9, 2.3]+0.42.42.3[2.1, 2.5]+0.1✓
Llama-3b2.31.8[1.6, 2.0]+0.51.41.9[1.7, 2.1]−0.5✓
Gemma-4b1.31.1[0.9, 1.2]+0.31.51.3[1.1, 1.5]+0.2
OLMo-7b2.02.5[2.3, 2.7]−0.54.74.0[3.7, 4.2]+0.8
Llama-8b1.51.4[1.2, 1.6]+0.11.72.1[1.9, 2.3]−0.4
Gemma-27b0.60.5[0.4, 0.6]+0.10.70.7[0.6, 0.8]+0.0
Qwen-72b0.80.5[0.5, 0.6]+0.30.40.6[0.4, 0.7]−0.2✓
ParaphraseOLMo-1b2.42.2[2.0, 2.4]+0.22.32.1[1.9, 2.3]+0.2
Llama-3b2.12.1[1.9, 2.2]+0.11.72.7[2.4, 2.9]−1.0
Gemma-4b1.21.0[0.8, 1.1]+0.21.71.4[1.2, 1.6]+0.3
OLMo-7b2.12.6[2.4, 2.8]−0.55.14.3[4.1, 4.6]+0.8
Llama-8b1.61.4[1.3, 1.6]+0.11.71.9[1.7, 2.1]−0.2
Gemma-27b0.50.5[0.4, 0.6]+0.00.80.7[0.6, 0.8]+0.1
Qwen-72b0.80.6[0.5, 0.7]+0.20.40.5[0.4, 0.7]−0.1✓
Self-explanationOLMo-1b4.63.5[3.3, 3.8]+1.14.03.9[3.7, 4.2]+0.1✓
Llama-3b1.81.8[1.6, 1.9]+0.11.92.2[2.0, 2.4]−0.4
Gemma-4b1.51.1[1.0, 1.3]+0.41.61.4[1.3, 1.6]+0.2
OLMo-7b2.12.7[2.5, 2.9]−0.56.05.3[4.9, 5.5]+0.7
Llama-8b2.31.8[1.6, 2.0]+0.51.42.4[2.2, 2.6]−1.0✓
Gemma-27b0.70.6[0.5, 0.7]+0.11.00.8[0.7, 0.9]+0.1
Qwen-72b1.10.7[0.6, 0.8]+0.40.40.6[0.5, 0.7]−0.2✓
Teacher-explanationOLMo-1b2.53.2[2.9, 3.4]−0.64.23.5[3.2, 3.7]+0.7
Llama-3b2.62.1[1.9, 2.2]+0.51.62.4[2.2, 2.6]−0.8✓
Gemma-4b1.91.3[1.1, 1.4]+0.62.11.7[1.5, 1.9]+0.3
OLMo-7b2.63.5[3.3, 3.7]−0.910.69.8[9.4, 10.2]+0.8
Llama-8b2.41.8[1.6, 2.0]+0.61.52.5[2.3, 2.8]−1.0✓
Gemma-27b0.70.7[0.6, 0.8]+0.01.21.0[0.9, 1.2]+0.2
Qwen-72b1.00.7[0.6, 0.8]+0.30.50.6[0.5, 0.7]−0.2✓

Table 15. DRUID: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000B = 1000 sign-randomised draws, with its 95% interval; Δ\Delta: observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.

Margin analysis results on DRUID with Self-explanation and Teacher-explanation

Figure 8. Margin analysis results on DRUID with Self-explanation and Teacher-explanation.

References

  1. [1]Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, and Kam-Fai Wong. 2023. Beyond factuality: A comprehensive evaluation of large language models as knowledge generators. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6325–6341, Singapore. Association for Computational Linguistics.arxiv.org/abs/2310.07289
  2. [2]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonnell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The language model evaluation harness.
  3. [3]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
  4. [4]Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.02226
  5. [5]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie SravanKumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.arxiv.org/abs/2407.21783
  6. [6]Ryan Greenblatt. 2025. Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance. Redwood Research Blog. Accessed: 2026-09-29.
  7. [7]Lovisa Hagström, Sara Vera Marjanovic, Haeun Yu, Arnay Arora, Christina Lioma, Maria Maistro, Pepa Atanassova, and Isabelle Augenstein. 2025. A reality check on context utilisation for retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19691–19730, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2412.17031
  8. [8]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  9. [9]Yoonna Jang, Kisu Yang, and Isabelle Augenstein. 2025. Expanding computation spaces of llms at inference time. Preprint, arXiv:2509.24884.arxiv.org/abs/2509.24884
  10. [10]Di Jin, Eileen Pan, Nassim Outafoulte, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14).
  11. [11]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2025. Prompt repetition improves non-reasoning llms. Preprint, arXiv:2512.19482.
  12. [12]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.arxiv.org/abs/2005.11401
  13. [13]Jiatao Li, Xinyu Hu, Xunjian Yin, and Xiaojun Wan. 2025. Evaluating self-generated documents for enhancing retrieval-augmented generation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2741–2775, Albuquerque, New Mexico. Association for Computational Linguistics.arxiv.org/abs/2410.13192
  14. [14]Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3154–3169, Dublin, Ireland. Association for Computational Linguistics.arxiv.org/abs/2110.08387
  15. [15]Jacob Pfau, William Merrill, and Samuel R. Bowman. 2024. Let’s think dot by dot: Hidden computation in transformer language models. In First Conference on Language Modeling.arxiv.org/abs/2404.15758
  16. [16]Qwen Team. 2024. Qwen2.5: A party of foundation models.
  17. [17]Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2025. Repetition improves language model embeddings. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2402.15449
  18. [18]Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.01296
  19. [19]ZhongXiang Sun, Xiaoxue Zhang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. 2025. RedEP: Detecting hallucination in retrieval-augmented generation via mechanistic interpretability. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.11414
  20. [23]Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxiang He, and Yulia Tsvetkov. 2024. Resolving knowledge conflicts in large language models. In First Conference on Language Modeling.arxiv.org/abs/2310.00935
  21. [24]Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2305.13300
  22. [25]Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. Re-reading improves reasoning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15549–15575, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2309.06275
  23. [26]Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2209.10063
  24. [27]Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hangyu Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, and Jiliang Tang. 2025. Towards knowledge checking in retrieval-augmented generation: A representation perspective. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2952–2969, Albuquerque, New Mexico. Association for Computational Linguistics.arxiv.org/abs/2411.14572
  25. [28]Yanzhe Zhang, Sanmi Koyejo, and Diyi Yang. 2026. The illusion of robustness: Aggregate accuracy hides prediction flips under task-irrelevant context. Preprint, arXiv:2607.12963.arxiv.org/abs/2607.12963
  26. [29]Chenguang Zhu, Yichong Xu, Xiang Ren, Bill Yuchen Lin, Meng Jiang, and Wenhao Yu. 2022. Knowledge-augmented methods for natural language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 12–20, Dublin, Ireland. Association for Computational Linguistics.

Paper details

vlnc:W_mgrhzdda
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents