WorksComputer Science
A Mechanistic Analysis of In-Context Learning and Out-of-Context Hallucination
Abstract
Large language models can infer and follow specific rules through in-context learning (ICL), yet may fail to remain faithful to provided evidence, giving rise to out-of-context (OOC) hallucination. Existing work has largely investigated the underlying mechanisms of ICL and OOC hallucination along separate lines. However, this separation obscures the mechanistic basis for these seemingly opposing outcomes. In this work, we study this contrast through the unified view of functional roles in dynamic context. The key observation is that role information can be internally represented without reliably constraining output. We derive a first-order decomposition to characterize the role effect, showing that its substantial influence depends on two necessary conditions, namely sufficient contribution magnitude and coherent signed aggregation. Causal analysis further shows that role span contributions affect key token choices involving response structure, content selection, and factual values. We further show that role-consistent choices depend on sufficient role span support at these key decisions, while failing choices can be reversed when this support overcomes the deficit. These findings support a common mechanistic view of ICL and OOC hallucination: each functional role defines the pattern of output decisions over which role span contributions must remain effective, and differences in this organization account for the seemingly opposing outcomes.
Introduction
A central strength of LLMs is the ability to condition generation on dynamic instructions. This capability is exemplified by in-context learning (ICL), where models can infer and follow specific rules supplied in the prompt [4, 6]. The same dependence on context, however, also exposes a major reliability challenge. Models may fail to remain faithful to the provided evidence and even produce unsupported claims, a failure mode we refer to as out-of-context (OOC) hallucination [13, 26]. This contrast raises a natural question: why can the same model reliably follow a prompt-specified rule, yet fail to remain faithful to evidence that is likewise supplied in context? Although existing work studies the mechanisms underlying ICL and OOC hallucination separately [11, 23, 31], the mechanistic origin of this contrast remains unclear.
Fundamentally, dynamic context serves different functional roles in generation: rules specify how to respond, whereas evidence specifies what factual content the response should be grounded in. The same content can be interpreted differently according to its function in the current prompt. The same sentence may act as a rule, a distractor, or ordinary background across tasks [20]. While contemporary LLMs receive no explicit supervision for these roles, recent mechanistic studies of ICL suggest that task structure can nevertheless be inferred from context [5].
In this paper, we present a counter-intuitive finding: role information can be internally represented without reliably constraining the resulting response. We decompose this distinction into two questions: (1) how functional roles are internally represented, and (2) how role information affects LLM1 generation. Despite their different outcomes, ICL and OOC hallucination can be analyzed through a unified framework focusing on the role effect, while preserving the distinct functions of rules and evidence. We connect these questions by examining how information from role-bearing spans affects token predictions and choices during generation. Crucially, a role can affect the model’s predictions without determining its final choice.
The effect of dynamic context is inherently non-uniform [1]. The prefix can establish the direction of the continuation and persistently influence subsequent predictions. For instance, rules can shape how the response is organized, whereas evidence can determine which factual content is carried forward [28, 8, 27]. We isolate the attention contributions from role-bearing spans, termed role span contributions, and examine how they affect next-token predictions. Our experiments show that successful role-conditioned generation depends on sufficient contribution support at the relevant token decisions. From this perspective, we show that ICL and OOC hallucination share the same contextual-influence mechanism, while it takes effect at different types of decisions in the two settings. Interested readers can refer to Appendix § 10 for a thorough discussion of related work.
Our contributions are threefold. First, we establish a unified account of contextual influence in ICL and OOC hallucination by distinguishing internal role representation from its effect on generation. Second, we derive a first-order decomposition that identifies two necessary conditions for predictive influence: sufficient contribution magnitude and coherent signed aggregation. Third, we show that ICL and OOC hallucination share a role span contribution mechanism, while their contrasting outcomes depend on effective support at different generation decisions.
Preliminaries
Setup. Let denote the context, the query, and a response trajectory. In ICL, the context supplies a rule for answering the query, whereas in OOC hallucination, it supplies evidence against which the response is evaluated. We index two functional roles, rule and evidence, by , and use to denote the corresponding role span. For an input , let denote its token positions. At decoding step , the model conditions on the prefix and defines a next-token distribution . We refer to this next-token prediction as the decision at step , and to as its choice. As the role span sits in the prompt, its effect on generation is realized through these decisions.
Matched contexts. To isolate the effect of role , we construct a matched counterfactual context that changes only the content of the role span while holding the remaining prompt fixed, as illustrated in Figure 1. The role itself is unchanged, and only what it specifies differs. We write and for the inputs constructed from and , where denotes sequence concatenation.
Figure 1. Examples of matched contexts. Construction details are provided in Appendix § 12.§ 1.
Residual response. Inspired by prior work on LLM internal mechanisms [23], we investigate how a role span is represented in the model and how it affects the internal state. Let denote the residual stream representation at layer and position when the model processes . We define the residual response as , which points from the matched counterfactual state to the original role instantiation.
Role Spans Induce Functional Residual Responses
ICL and OOC hallucination differ in their empirical outcomes, yet both rely on dynamic context within the same autoregressive decoding framework. Despite the shared structure, context serves distinct functional roles in the two settings. This raises a natural question: how does the functional role organize the model’s internal state? We address this through matched role comparisons in the residual stream, linking the role distinction to its internal representational form. We first characterize the organization of the induced state changes and then examine their relation to the decoding process.
Structured Residual Responses from Role Spans
The same sentence can serve different functional roles, such as an instruction, evidence, or distractor. To examine how this distinction appears in internal states, we keep the span content fixed while changing only its function. Specifically, we construct multiple functional framings for each span and train a linear readout to distinguish the corresponding roles, following Belinkov [3]. We find that the readout generalizes consistently across prompt formulations, indicating a stable functional distinction in internal states.
We next examine how functional role structures internal states. To test this, for each role , we fit the top 10 PCA directions on one half of the samples and measure the fraction of energy captured on the other half. Ideally, a high captured fraction would indicate shared organization, whereas isotropic differences would remain near the random baseline. Figure 2 shows that the energy concentration remains well above the baseline. Rule responses exhibit strong concentration, whereas evidence responses are more distributed. These results show that the residual responses induced by functional role exhibit systematic low-dimensional organization.

Figure 2. Structured residual responses. Left panels show rule span responses, and right panels show evidence span responses. Each violin shows the fraction of energy captured by a 10-dimensional PCA subspace; points are colored by the percentile of . The dashed gray line denotes the expected random subspace baseline , with PCA directions and residual dimension.
Residual Responses Toward Role-Consistent Generation
The low-dimensional organization observed above shows that role spans can induce structured representational change during inference. Since the residual stream connects prompt processing to token prediction, we next examine how these changes relate to the output process. To make this relation concrete, we consider two empirical implications: (1) The dominant directions of the residual subspace should align with the unembedding directions of outputs. This follows from the fact that the LLM maps the residual representation to token logits through the final normalization and unembedding matrix . Here, the structured should appear as a larger projection onto the output directions. (2) The representation change induced by a specific role span should preserve relevant role information. Since the residual stream connects intermediate representations with downstream computation, the role information represented by should be maintained in a form that later blocks can read, transform, and propagate.
To examine the first empirical implication, we project through , measuring its projection and logit shift toward the output. We use a matched non-role perturbation as the comparison group. Figure 3 shows that, across all three models, role responses are more strongly aligned with the corresponding output directions, which results in larger logit shifts. Specifically, rule responses show clearer positive projection, while evidence responses separate more through the logit shift. By contrast, the comparison group remains weaker in both projection and logit shift. These results show that, during the decoding process, structured appears as a directional logit shift toward the output direction indicated by the role span.

Figure 3. Final layer decoding projection of . Blue points denote the comparison group.
To examine the second empirical implication, we evaluate behavior recovery in states induced by the matched counterfactual context; details are provided in Appendix § 12.§ 2. As shown in Table 1, matched role responses restore the original behavior associated with the role, showing that the injected residual response remains usable by subsequent computation. Moreover, recovery remains low for the comparison groups since they lack the role conditioned state required for recovery. These results indicate that the induced representational change carries transferable role information that remains usable by downstream computation.
| Injected response | Qwen | Mistral | Llama | Avg. |
| Rule | ||||
| matched role response | 0.94 | 1.00 | 1.00 | 0.98 |
| no role content | 0.17 | 0.08 | 0.00 | 0.08 |
| mismatched role response | 0.10 | 0.31 | 0.05 | 0.15 |
| Evidence | ||||
| matched role response | 0.71 | 0.80 | 0.93 | 0.81 |
| no role content | 0.44 | 0.20 | 0.37 | 0.34 |
| mismatched role response | 0.35 | 0.25 | 0.04 | 0.21 |
Table 1. Behavior recovery rates.
Our findings suggest that functional roles in ICL and OOC hallucination can be characterized in the representation space. Rule and evidence spans induce a common form of structured residual response that carries role-specific information forward for downstream computation.
Role Span Contributions in Generation
So far, we have characterized how role spans are internally represented, yet how they shape LLM generation remains unclear. We examine two questions: (i) whether role information can shift the model’s preference toward specific responses, and (ii) how role spans affect token during generation, leading to distinct generation outcomes. Specifically, we develop a theoretical framework that relates role-induced changes to internal LLM computations and derive a first-order decomposition to characterize the conditions under which role span contributions can influence LLM behavior.
Response Trajectory Score Shift
To characterize how rule or evidence spans influence generation, we examine their effects on next-token scores. Since autoregressive predictions depend on the preceding response history, changes in role information can lead to different response prefixes during generation. Such differences can confound the influence of role information on subsequent predictions. To isolate the influence of a role span at a given position, we evaluate the same token choice under a common response history. For each role , we consider a response trajectory , which can be either a constructed candidate or a response generated autoregressively. In both cases, we evaluate the score of under contexts and , holding the query and response prefix fixed. We defer further details to Appendix § 12.§ 1. Specifically, we write the stepwise score as , and define the resulting score shift as . Based on the autoregressive factorization, the change in the log probability assigned to is the sum of these stepwise differences. We formalize this quantity in the following definition.
Definition 1 (Response trajectory score shift). For a response sequence , define the stepwise score shift as . The response trajectory score shift is , which measures the change in the log probability assigned to the same response under and . Equivalently, using the autoregressive product form,
Here, indicates that yields a higher score than does. In what follows, we relate the stepwise score shifts to forward pass quantities associated with the role span.
Role Span Contributions in the Forward Pass
We mainly consider decoder-only LLMs, where successive attention and MLP modules update the residual stream before the final unembedding. The role-induced responses observed in § 3 are formed and propagated through these updates. To relate this process to the role span, we write the residual update explicitly. For a processed input , the residual state at layer and position is updated as . This reveals that information carried by different roles is accumulated across depth. A difference induced by role can therefore reflect both information inherited from previous layers and the current updates. Recall that the role span is defined by source token positions. Among the two update terms above, the attention output can be explicitly written as a sum over source token positions:
where indexes attention heads and indexes source token positions in that are visible to the target position under the causal mask. Eq. (2) decomposes the attention update at into contributions aggregated from source positions. The key observation is that the role span is also a set of input token positions. We can therefore isolate the contribution of the role span by retaining the terms with , giving
We term the role span contribution to the attention update at layer and position . In decoder-only Transformers, attention directly transfers differences across token positions, whereas normalization and MLP blocks act only within individual positions. Appendix B.1 formalizes this distinction.
First-Order Decomposition of Score Shifts
The response trajectory score shift accumulates the stepwise shifts across response positions. We therefore analyze each stepwise shift through selected role span contribution coordinates and quantify the first-order score change associated with their matched differences. To make this operational, we approximate the stepwise score by its first-order change with respect to the selected coordinates. Specifically, for each decoding step , let denote the role span contribution indices considered in the expansion. For , define . Collecting these differences gives .
The vector , therefore, measures how the matched role comparison changes the selected role span contributions over .
Assumption 1 (Local score regularity). For a fixed response position , let and denote the selected role span contribution vectors induced by and , respectively. Define
as the score obtained by replacing, in layer order, the selected contributions under with the corresponding entries of . By construction, ; let . We assume that is differentiable and has an -Lipschitz gradient on the line segment from to , and write .
Under Assumption 1, a first-order expansion of from to yields the following decomposition.
Theorem 1 (First-order role span score decomposition). Suppose Assumption 1 holds. Then the score shift satisfies
where the gradients are evaluated at , and the remainder satisfies
The detailed derivations are provided in Appendix § 11.§ 2. Eq. (4) expresses the stepwise score shift as selected first-order terms plus a remainder. We use the coordinates to isolate the direct source-indexed role span attention pathway, while effects outside these coordinates remain in . We define the selected first-order score contribution associated with as
Note that is an inner product between the local score gradient and the matched contribution difference, so it gives a directional contribution to the score shift. Hence, Eq. (6) makes two factors explicit: (1) the matched difference captures the role-dependent change in the forward pass quantity, and (2) the gradient captures the local sensitivity of the stepwise score to this quantity. Equivalently, writing the inner product in norm-angle form gives
To this end, we identify three distinct factors: the matched difference , the local score sensitivity, and their directional alignment. Summing Eq. (4) over response positions and using Eq. (6) gives
Together, Eqs. (7) and (8) show how local role span contributions organize into a trajectory-level score shift. Locally, the contribution magnitude depends on both the matched change in and the sensitivity of the stepwise score, while the directional alignment determines whether the term supports or opposes the score shift. Across the generation, the signed terms must aggregate coherently rather than cancel. The first-order decomposition therefore identifies two conditions for a trajectory-level effect, namely sufficient magnitude and coherent signed aggregation.
Experiments
In this section, we investigate the mechanism underlying the common dependence on context and the distinct generation outcomes of ICL and OOC hallucination. We use rule-following tasks derived from CL-bench ([7]) and evidence-grounding examples from RAGTruth ([17]), with settings provided in Appendix § 12. To explain these distinct outcomes, we address two related questions: how role information affects model predictions and how its contributions influence token selection during generation. Our experiments examine three claims: (i) that role span contributions produce role-consistent preference shifts through organized layerwise effects, (ii) that rule and evidence realize this influence through different types of key token decisions, and (iii) that successful choices depend on sufficient contribution support, while failed choices can be reversed when this support overcomes the decision deficit. Appendices § 13.§ 4 and § 13.§ 5 extend the analysis to broader ICL and evidence-grounding settings.
Role Span Contribution Mechanism
In this subsection, we empirically test how role span contributions translate into predictive influence as the first-order decomposition suggests. If a role span can guide predictions toward the specified content, it must increase the relative support for that content over alternatives. To examine this, we compare two candidate responses and , where follows the rule or is supported by the evidence, whereas does not. Since we aim to measure how role information shifts predictive support between these alternatives, we evaluate their conditional probabilities under matched contexts, independently of the model’s generated response. We define to quantify the preference change. Here, indicates that role information increases the relative support for over . Construction details are provided in Appendix § 12.§ 3. As shown in Figure 4(a), we empirically observe that is high for both roles, indicating that the role span contribution consistently favors role-compatible responses over alternatives. More specifically, this preference is expressed as a tendency toward rule-following responses in ICL and evidence-grounded responses in the OOC setting.

Figure 4. Response preference and contribution organization. (a) Fraction of positive candidate margins. For Qwen2.5-7B, (b) layerwise profiles of and , (c) signed layerwise contributions and sign coherence, and (d) preference margin under baseline, role span pathway ablation, and random ablation. More results are in Appendix § 13.§ 1.
Since the contribution propagates through the forward pass, its final effect accumulates across layers. The decomposition in Theorem 1 suggests two conditions for a substantial first-order contribution: (1) the contribution magnitude must be sufficiently large, and (2) the signed layerwise terms must aggregate coherently. Based on Eq. (7), we use to measure contribution magnitude and the cosine term to determine the signed direction. Note that the aggregated first-order terms can track the exact score shift (Appendix § 13.§ 2).
We first examine the layerwise contribution magnitude. As illustrated in Figure 4(c), both rule and evidence exhibit a localized middle-to-late concentration with a clear interior peak, indicating that substantial contribution is confined to a limited depth range. To understand the mechanism behind this peak, we further decompose the contribution magnitude into its two factors. As shown in Figure 4(b), and exhibit opposing trends: increases with depth, whereas decreases. Given these opposing trends, their product is expected to peak at an intermediate depth where the increase in no longer outweighs the decrease in . Consistent with this prediction, the resulting contribution exhibits the localized interior peak shown in Figure 4(c). In early layers, remains close to zero, indicating that the role span has induced a limited change. The contribution therefore remains small despite high downstream sensitivity. Thus, a high contribution appears only when both factors remain sufficiently large.
Notably, magnitude alone is insufficient if the signed terms cancel after aggregation. We further compare targeted contribution ablation with a random control, which controls for generic intervention effects. As shown in Figure 4(d), targeted ablation produces a larger reduction in for both roles, showing that disrupting the contribution organization markedly weakens the resulting preference shift. Taken together, these results show that role span contribution emerges when both magnitude factors remain sufficiently large, while disrupting their signed organization weakens the resulting preference shift.
Role Span Contribution at Key Token Decisions
Contribution localization. The above analysis suggests that role spans can shift predictions toward role-consistent responses through layerwise contributions. During autoregressive generation, this influence must be realized through token choices that affect whether the role’s requirements are satisfied. We therefore expect role span contributions to be particularly consequential at some key decisions. To examine how these decisions contribute to the distinct outcomes of ICL and OOC hallucination, we first measure the token-level first-order contribution at the corresponding output decisions. As shown in Table 2(a), these decisions consistently occur at positions with high role span influence. We then remove the role span pathway during free generation and record the first change in token selection. These first divergences likewise occur at positions with high role span influence. Thus, these findings suggest that role spans shape generation through key token choices.
| Model | (a) First-divergence localization | (b) Retention of role-specified output content (%) | ||||||||||||
| Rule span | Evidence span | Rule span | Evidence span | |||||||||||
| Grounded | Hallucinated | Format | Entity | Grounded | Hallucinated | |||||||||
| Attr. | Dec. | Attr. | Dec. | Attr. | Dec. | Rem. | Rem. | Rem. | Rem. | |||||
| Qwen2.5-7B | 87.5 | 85.0 | 75.0 | 92.5 | 88.0 | 88.2 | 100.0 | 4.8 | 57.1 | 18.1 | 100.0 | 10.2 | 62.3 | 11.3 |
| Mistral-7B | 82.5 | 85.7 | 87.5 | 77.5 | 89.8 | 90.1 | 100.0 | 2.4 | 58.1 | 33.3 | 100.0 | 16.7 | 61.5 | 7.7 |
| Llama-3.1-8B | 80.0 | 90.0 | 85.0 | 95.0 | 92.4 | 91.9 | 100.0 | 4.8 | 53.3 | 18.1 | 100.0 | 10.0 | 61.5 | 13.5 |
Table 2. Role span effects on output decisions. (a) Role span influence at the first divergence and the output decision. (b) Retention rate of the content specified by under the original () and pathway-removed (Rem.) conditions. Additional results are provided in Appendix § 13.§ 6.
Note. Attr. and Dec. report role span influence at the first divergence and output decision positions, respectively; higher values indicate stronger influence. Pathway removal changes token selection in 92–100% of generations. Hallucinated prefixes are taken from RAGTruth. Further details are in Appendix § 13.§ 6.
Role Effects. We next examine the effects of role span contributions at these key token decisions for the free generation. Specifically, we remove the role span pathway during decoding and assess whether the resulting response satisfies each role’s requirements. As shown in Table 2(b), removing the rule pathway disrupts the specified format and reduces retention of the selected entity. For evidence, pathway removal changes the facts stated in originally grounded responses. These results show that role span contributions causally support the key decisions that realize role-specific requirements. One might wonder whether the same pathway also affects factual decisions in real hallucinations. We test this using annotated LLM-generated hallucinations from RAGTruth [17], resuming generation from the prefixes preceding the hallucinated facts under the original and pathway-removed conditions. As shown in Table 2(b), the rate of selecting evidence-supported facts drops from 61.5–62.3% to 7.7–13.5% across models. This finding extends the role effects observed above to factual decisions in real hallucinated generation.
Together, these results reveal distinct decision-level effects of the two roles. Rule span contributions are typically realized through structural and selection decisions, whereas evidence span contributions affect factual content decisions, including continuations from genuine hallucination examples. By shaping different types of key token decisions, the two roles give rise to distinct generation outcomes in ICL and OOC hallucination.
Role Span Contributions to LLM Decision-Making
The preceding analyses establish how role span contributions are organized across layers and causally influence key token choices during free generation. Yet, even when role span contributions influence a decision, they may not be sufficient to make the model select the role-consistent token. We therefore examine when these contributions can sustain a successful choice or change an initially incorrect one. To this end, we first attenuate the signed contribution at initially successful decisions (§ 5.3.1) and then restore the contribution at naturally failing ones (§ 5.3.2).
Dependence of Successful Decisions
We first examine successful decisions in the rule-following and evidence-grounding tasks derived above. In both settings, we analyze the model’s autoregressive generations and identify the key decisions at which the corresponding role requirements are satisfied. To examine whether role span contributions sustain these choices, we measure the advantage of the role-consistent token over its strongest competitor using the target-token margin, with zero marking the decision boundary. We then attenuate the role span contribution by a factor of and track changes in this margin and the resulting token choice.
As shown in Table 3(a), increasing generally narrows the margin under both roles. For rule decisions, the median margin shifts from positive to negative across all models, leaving the rule-specified token selected in only a minority of these key choices. Meanwhile, attenuation substantially erodes the advantage of the evidence-supported token, bringing the corresponding factual choice closer to the decision boundary. In contrast, the non-role comparison shows smaller margin changes, showing that the pronounced changes are driven by role span contributions. Under full attenuation, these margin reductions are accompanied by frequent token flips and performance declines (Table 3(b).)
| Model | Flip (%) | Task (%) | Flip (%) | Task (%) | |||||||
| (a) Target token margin | (b) Token and task outcomes at | ||||||||||
| Targeted attenuation | Non-role comparison | Targeted attenuation | Non-role comparison | ||||||||
| Model | Flip (%) | Task (%) | Flip (%) | Task (%) | |||||||
| Rule span | |||||||||||
| Qwen2.5-7B | 4.62 | 5.00 | -0.38 | -12.89 | -16.25 | 4.62 | 5.00 | 84.0 | 22.3 | 17.0 | 100.0 |
| Mistral-7B | 6.69 | 6.06 | 2.09 | -4.16 | -5.03 | 6.69 | 5.25 | 77.5 | 32.4 | 32.4 | 94.1 |
| Llama-3.1-8B | 4.69 | 3.25 | 2.06 | -1.94 | -4.06 | 4.69 | 2.38 | 76.5 | 28.6 | 25.5 | 86.7 |
| Evidence span, task-defined | |||||||||||
| Qwen2.5-7B | 8.23 | 7.90 | 5.63 | -0.57 | -1.09 | 8.23 | 7.60 | 57.4 | 25.9 | 9.3 | 74.1 |
| Mistral-7B | 10.00 | 9.94 | 6.72 | 2.56 | 1.25 | 10.00 | 9.88 | 44.2 | 42.3 | 0.0 | 98.1 |
| Llama-3.1-8B | 8.62 | 7.62 | 4.50 | 0.50 | -0.69 | 8.62 | 7.75 | 54.9 | 37.3 | 2.0 | 98.0 |
| Evidence span, response-derived | |||||||||||
| Qwen2.5-7B | 14.70 | 14.04 | 7.88 | 1.66 | 0.46 | 14.70 | 12.54 | 50.0 | 45.8 | 0.0 | 100.0 |
| Mistral-7B | 15.91 | 15.54 | 12.62 | 0.96 | 0.98 | 15.91 | 15.19 | 43.5 | 30.4 | 0.0 | 100.0 |
| Llama-3.1-8B | 5.06 | 4.84 | 2.01 | -4.62 | -9.28 | 5.06 | 5.23 | 87.5 | 12.5 | 4.2 | 95.8 |
Table 3. Sustained attenuation of role span contributions. The contribution is scaled by , with denoting complete attenuation. (a) Median target token margin under targeted attenuation and non-role comparison. (b) Token flips and task performance under full attenuation.
To rule out dependence on predefined factual decisions, we further apply the same sustained attenuation to facts correctly generated by the models themselves. As shown in the response-derived rows of Table 3, targeted attenuation substantially reduces the decision margin and frequently changes the selected token and generated factual value, whereas attenuation of non-evidence spans largely preserves the original factual content. These results show that the presence of role information does not guarantee effective support during generation. The contribution must remain sufficiently large when the model makes key choices. For rules, this support must be sustained across repeated structural and selection decisions, whereas for evidence, it is required when the factual value is determined.
Recoverability of Failed Decisions
We have focused on initially successful decisions, where attenuation shows that the local choice depends on sustained role span support. We next examine whether amplifying the same contributions can restore decisions that initially fail. To test the causal effect, we amplify the same contribution at naturally failing decisions and track the task performance.
Considering task-defined cases, we find that amplifying the evidence span contribution restores a substantial subset of naturally failing factual decisions. As shown in Table 4, the restoration rate reaches 43.7% at , accompanied by improved task performance. In contrast, rule failures exhibit much larger initial decision deficits, and their target tokens remain unrecovered even under amplification. These results suggest that whether contribution amplification can reverse a failed decision depends on the initial decision deficit.
| Model | Baseline margin | Margin change | Selection (%) | Task (%) | Metric | |||||||
| (a) Restoration at | (b) Restoration across weights | |||||||||||
| Rule span | ||||||||||||
| Qwen2.5-7B | -14.6 | -0.61 | 0.0 | 0.0 | Selection | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Mistral-7B | -19.0 | +0.06 | 0.0 | 4.8 | Margin | -14.6 | -14.6 | -14.5 | -14.1 | -13.0 | -12.8 | -11.7 |
| Llama-3.1-8B | -11.3 | +0.00 | 0.0 | 0.0 | Task | 0.0 | 0.0 | 1.3 | 2.5 | 6.3 | 3.8 | 6.3 |
| Evidence span, task-defined | ||||||||||||
| Qwen2.5-7B | -4.0 | +0.89 | 29.5 | 34.1 | Selection | 0.0 | 22.3 | 28.2 | 43.7 | 41.7 | 39.8 | 38.8 |
| Mistral-7B | -2.2 | +0.28 | 16.7 | 20.8 | Margin | -3.0 | -2.9 | -2.0 | -1.2 | -0.6 | -1.9 | -2.8 |
| Llama-3.1-8B | -2.7 | +1.69 | 34.3 | 37.1 | Task | 0.0 | 25.2 | 32.0 | 47.6 | 53.4 | 55.3 | 58.3 |
| Evidence span, real-hallucinated | ||||||||||||
| Qwen2.5-7B | -6.0 | +1.18 | 11.1 | 44.4 | Selection | 0.0 | 8.8 | 10.0 | 17.5 | 32.5 | 33.8 | 32.5 |
| Mistral-7B | -5.3 | +1.20 | 11.5 | 42.3 | random | 0.0 | 1.2 | 5.0 | 6.2 | 5.0 | 0.0 | 2.5 |
| Llama-3.1-8B | -4.5 | +0.96 | 7.4 | 37.0 | Task | 28.8 | 37.5 | 41.2 | 43.8 | 57.5 | 57.5 | 61.2 |
| All models | -5.4 | +1.12 | 10.0 | 41.2 | random | 28.8 | 27.5 | 27.5 | 28.8 | 25.0 | 15.0 | 25.0 |
Table 4. Restoration at failing decisions. We restore the contribution vector at the decision, scaled by a weight . (a) Restoring the contribution at . (b) Restoration across multiple weights.
We further examine whether contribution amplification can restore factual decisions in genuine hallucinations from RAGTruth. As shown in the real-hallucinated rows of Table 4, these decisions exhibit larger initial deficits than task-defined evidence decisions. Nevertheless, amplification shifts the decision margin toward the evidence-supported token and improves factual selection. At , target selection reaches 32.5%, with 57.5% of continuations stating the evidence-supported fact, both outperforming random-direction interventions. These results show that evidence span contributions remain effective in genuine hallucinations, although their ability to reverse factual decisions still depends on the decision deficit.
Consequently, these results show that the same local contribution mechanism can extend to naturally failing decisions, including genuine hallucination examples. Whether amplification reverses the choice depends on the balance between the support of role contribution and the decision deficit. This relation further yields a quantitative prediction for failed decisions, which we formalize in Appendix § 11.§ 4 and evaluate in Appendix § 13.7.
Table 4: Restoration at failing decisions. We restore the contribution vector at the decision, scaled by a weight . (a) Restoring the contribution at . (b) Restoration across multiple weights.
Note. Margins follow the definition in Table 3, and ; positive values indicate a shift toward the target token. Real-hallucinated decisions are taken from RAGTruth; their Task is the share of continuations stating the grounded value at the hallucinated slot. random rows replace the contribution vector with a random direction of equal norm.
In summary, we show that dynamic context guides LLM generation through functional role span contributions in both ICL and OOC hallucination. Specifically, rule contributions shape structural and selection choices, whereas evidence contributions shape factual choices. Moreover, effective role influence depends on coherent contribution organization across layers and sufficient support when the key choice is made. The causal interventions further show that a failed choice can be reversed when the corresponding support is sufficient to compensate for the decision deficit.
Conclusion
In this paper, we offer a common mechanistic view of the distinct outcomes of ICL and OOC hallucination. This view treats the functional roles of dynamic context as the organizing principle for model behavior. We connect internal role representations to their effects on decoding through role span contributions. These contributions causally shape key token choices that determine response structure, content selection, and factual value. Each functional role determines where these contributions must remain effective during generation. We derive a first-order decomposition of role span contributions and identify two necessary conditions for reliable influence. At the decision level, rule following requires sustained support across repeated structural and selection choices, whereas evidence grounding relies on support at a smaller set of decisive factual choices. The contrasting outcomes therefore follow from distinct requirements for sustaining role support across generation.
AI USE STATEMENT
In this work, we did not use generative AI tools for tasks requiring mandatory disclosure under the ICLR 2027 AI Policy. Additionally, we used generative AI tools to create or edit software code and to edit the research paper to improve readability. We have reviewed all AI-assisted work. AI-assisted code was verified and tested by the authors, and AI-assisted edits to the manuscript were reviewed by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
REPRODUCIBILITY STATEMENT
The source code used for our experiments is provided in the supplementary materials and will be released in a public repository. Appendix B provides the complete proofs and derivations of the theoretical results. Appendix C provides the experimental details, including the experimental setting, matched role construction, representation and recovery analyses, and contribution and preference analysis. Appendix D reports additional experimental results.
Appendix
Related Work
Context-conditioned generation. Motivated by the contextual adaptability of LLMs, recent work has extensively studied how information supplied at inference time shapes model behavior. In-context learning provides a prominent setting where models can infer a task from demonstrations and instructions without task-specific training [10, 29]. Recent studies further show that this adaptation is sensitive to demonstration selection, ordering, and composition [15, 29, 18, 25]. Meanwhile, a separate line of work has investigated hallucination in LLM generation, where model outputs contain factually incorrect or unsupported information [13, 31]. Retrieval-augmented generation has become a common approach to mitigating hallucination by incorporating external evidence into generation [30, 2]. Nevertheless, LLMs may still produce claims that are unsupported by or contradictory to the retrieved content [17, 26]. Such departures from the supplied context define the out-of-context (OOC) hallucination setting and expose a gap in understanding contextual faithfulness [2, 12].
Mechanistic analyses of inference-time behavior. Recent mechanistic work extends the study of inference-time behavior from observable outcomes to internal representations. In ICL, some studies identify attention circuits that support in-context computation [19], while other work seeks to explicitly characterize in-context task information by recovering task vectors from hidden states [11, 23]. In parallel, mechanistic studies of hallucination associate incorrect or unsupported generation with failures in knowledge retrieval and transfer and with conflicts between parametric knowledge and external evidence [31, 28]. Other studies further localize components involved in retrieving and using contextual information during generation, showing that evidence use can likewise be traced to internal components [27, 16]. Despite this progress, the two behaviors are predominantly examined within specific tasks or failure settings. Across both lines, a common concern is which internal representations of context become consequential for generation.
Attributing contextual influence. To establish this relevance, existing methods connect model predictions to their internal sources. For instance, input attribution and score decomposition quantify how tokens or activations support a prediction [22], while circuit analysis traces it to interpretable computational paths [9]. However, such local analyses rarely make explicit the function of context in the current task [31]. Despite comparable mechanistic accounts, ICL and OOC hallucination exhibit distinct task-level behaviors. To this end, we analyze the distinct functional roles in the two settings and identify the mechanistic conditions underlying task success and failure from a unified perspective.
Proofs and Derivations
Attention-Mediated Role Span Transport
We first formalize why a role-induced difference can enter a new token position only through attention.
Lemma 1 (Position locality). In a decoder-only Transformer, normalization and MLP modules act independently at each token position, whereas the attention update at position aggregates states from the source positions visible to .
Proof. Normalization and MLP modules apply the same map independently at each position. By contrast, Eq. (2) writes the attention update at as a sum over its visible source positions. Position-wise transformations of queries and keys do not alter this distinction.
Fix a decoding step and write , , and for their aligned role span positions. Define
and let denote the source positions visible to . For head and source position , write
We decompose the source-indexed differences as
Proposition 1 (Attention-mediated role span transport). Suppose that and differ only at the aligned role span positions . For every , , and at each layer ,
where
Moreover,
Consequently, if exists, then .
Proof. Since the matched inputs differ only at , their embedding representations coincide at every . Hence, .
We next decompose the attention difference. By Eq. (2),
The visible source positions admit the disjoint partition
Summing over these three sets gives Eq. (9). The first term is the direct role span contribution defined in Eq. (3); the second reads from non-role positions whose residual states already differ.
For a source position in the third set, , and hence its value representation is identical under the two computations:
Thus, contains the remaining changes in attention weights over otherwise unchanged source values. This definition covers any such reweighting without assigning it to a particular query, key, or softmax component.
We next show how the difference propagates through the block. If and , the two computations provide the same local state and the same attention update at position . By Lemma 1, all remaining operations at that position therefore receive identical inputs, which gives .
Now let be the first layer such that . Then . If the attention updates also agreed at layer , the previous implication would yield , a contradiction. Therefore, .
Proposition 1 establishes that attention is the direct channel through which a role-induced difference enters a new token position. Among the resulting terms, is the direct source-indexed role span pathway used in § 4.3, while and are treated in the remainder analysis of Appendix § 11.§ 3.
First-Order Role Span Score Decomposition
We now make the coordinate dependence in Assumption 1 explicit. Fix a decoding step and write
with
We equip this stacked coordinate space with the Euclidean inner product and norm.
Substitution map. For , let denote the score obtained from by processing the model in forward order and replacing each selected role span attention contribution with before it enters the residual stream. Specifically, the term recomputed from the current modified state is removed and replaced by . All replacements within a layer are applied before advancing to the next layer, and the remaining quantities are computed from the resulting states.
Lemma 2 (Exact substitution identity). The substitution map is well defined and satisfies
Consequently, with
the stepwise score shift admits the exact identity
Proof. The forward ordering uniquely determines every modified residual state, so is well defined for any collection .
We next establish the endpoint identity. Set . At the input layer, the modified computation and the computation under have identical states. Suppose that their states agree through layer . Their unmodified attention terms at layer are then identical, and each selected role span term is replaced by its value . Hence, the complete attention updates and the resulting residual states also agree at layer . Induction over the layers gives .
Finally, adding and subtracting in the definition of gives
which is (10).
Proof of Theorem 1. By Lemma 2, it remains to expand . Let
Then we have
Adding and subtracting gives
where
Since has an -Lipschitz gradient on the segment between and ,
Moreover, the stacked inner product decomposes as
Combining this equality with (10) yields (4) with
Finally,
which gives (5).
The term is the Taylor remainder within the selected coordinate space, whereas contains the score change not reproduced by these coordinates. We characterize the latter terms in Appendix B.3.
Remainder Characterization
By Theorem 1, the remainder separates as , where .
Here, is the curvature error introduced by linearizing over the selected coordinates, whereas measures the score change not reproduced by these substitutions. Relative to the decomposition in Appendix B.1, the substitution gap may include omitted direct coordinates and portions of and that are not recovered by the modified computation.
Notably, this characterization does not assume that is small. In Appendix D.3, we compare the direct role span attention pathway with alternative transport terms and MLP updates, and show that attention provides the dominant route for role information to reach the relevant token decisions.
Decision-Level Recovery Prediction
We finally relate the role span contribution to the local decision boundary considered in Section 5.3. For a naturally failing decision, let denote the target token and let
be its strongest competitor before amplification. Following Section 5.3.2, let parameterize amplification of the same role span contribution at the decision position, with denoting the original decision. For the fixed competitor , write
so that . We define the signed support supplied along the amplification direction as
Proposition 2 (Decision-boundary crossing). Suppose that the derivative of is -Lipschitz along the local amplification path. Then
For and , the corresponding first-order prediction of the amplification required to reach the decision boundary is
Moreover, if remains the strongest non-target token along the local path, then any satisfying
guarantees .
Proof. A first-order expansion of the fixed-competitor margin at gives
where the Lipschitz condition yields . This proves Eq. (11).
Ignoring the quadratic term, the local boundary is reached when . For , solving this equality gives Eq. (13). Thus, a larger initial deficit requires stronger amplification, whereas larger positive signed support reduces the required strength.
For the finite- statement, Eq. (11) gives the lower bound
Hence, Eq. (14) implies . If remains the strongest competitor, this pairwise margin equals the target-token margin, and therefore .
Consequently, the first-order decision formulation estimates the amplification required to recover a failed decision. Appendix § 13.7 further examines the extension of this prediction to task-level recovery.
Experimental Details
Experimental setting. For the rule role, we derive a controlled benchmark from the task patterns in CL-bench Dou et al. [7], covering format, selection, and include or omit instructions. For the evidence role, we use the training split of RAGTruth Niu et al. [17]. We evaluate Qwen2.5-7B, Mistral-7B, and Llama-3.1-8B. Free generations use greedy decoding, whereas fixed response trajectories are evaluated with teacher forcing.
Matched Role Construction
To isolate functional role , we compare two prompts that share the query and all non-role content but differ in the role-bearing span. The matched counterfactual changes only this span while preserving the remainder of the prompt. Figure 1 illustrates the matched construction. To ensure alignment, we locate the edited span separately in each tokenized prompt and, for a fixed response trajectory , use the same response prefix under both conditions, with comparisons performed at the same decoding step.
Representation and Recovery Analyses
Subspace concentration. We extract at the final prompt position from every decoder layer. For each model and role, we fit a rank-10 subspace on one half of the samples and evaluate it on the other, then exchange the two splits. We use the matched role comparisons defined in Appendix § 12.§ 1.
Specifically, the held-out concentration is computed as , where projects onto the fitted rank-10 subspace . The random-subspace baseline is , where is the hidden state dimension.
Output alignment and behavior recovery. We assess output alignment by applying the final RMS normalization to the two residual states in each matched pair and then projecting their difference onto the unembedding direction of the token selected under . We report the resulting logit shift and cosine alignment, with a matched non-role perturbation as the control. Separately, we evaluate behavior recovery by adding to the counterfactual state at the final prompt position and recording whether the next-token choice returns to that under .
Contribution and Preference Analysis
Candidate comparisons. Each preference analysis combines a matched context pair with a minimal candidate pair. Both candidates are scored with teacher forcing, and measures the resulting change in relative support. fig:5 illustrates the design.
| Matched contexts | Matched candidates | |
| Rule | : JSON-output rule<br>: Markdown-list rule | : {”name”: ”Alpha”, ”value”: ”120”}<br>: {”label”: ”Alpha”, ”value”: ”120”} |
| Evidence | : Evidence supporting 2024<br>: Matched context, e.g., evidence supporting 2029 | : The value is 2024<br>: The value is 2029 |
Table 5.
fig:5: Matched preference construction. Each context pair differs only in the role-bearing span, while each candidate pair differs only in the choice specified by that role.
Additional Experiments
Additional Model Results for Contribution Organization
Figures 6 and 7 report the corresponding analyses for Mistral-7B and Llama-3.1-8B, both of which exhibit the same overall pattern. Contributions remain concentrated within a limited range of layers, the two magnitude factors vary in opposite directions across depth, and role span pathway ablation reduces more than random ablation. The contribution organization identified in the main analysis is therefore consistent across models.

Figure 6. Organization of contributions for Mistral-7B. (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of and ; and (c) preference margin under the baseline, role span pathway ablation, and random ablation.

Figure 7. Organization of contributions for Llama-3.1-8B. (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of and ; and (c) preference margin under the baseline, role span pathway ablation, and random ablation.
| Model | Rule span Corr. | Rule span Sign (%) | Evidence span Corr. | Evidence span Sign (%) |
| Qwen2.5-7B | 0.93 | 77.1 | 0.86 | 95.0 |
| Mistral-7B | 0.94 | 87.6 | 0.88 | 97.0 |
| Llama-3.1-8B | 0.92 | 91.4 | 0.85 | 94.0 |
Table 5 (PDF p. 20). Correspondence between first-order contributions and exact score shifts.
Note. Corr. is the median Spearman correlation between the magnitudes of the summed first-order contributions and the exact stepwise score shifts. Sign is the fraction of responses whose accumulated contribution matches the sign of the exact trajectory shift.
First-Order Correspondence
Here we examine how the first-order contributions correspond to the exact role effect. Within each response, the magnitudes of the summed first-order contributions track the exact stepwise score shifts, with median Spearman correlations of 0.85–0.94 across models. The sign of the accumulated contribution also matches the exact trajectory shift for most responses.
Role Span Attention Pathway
Appendix B.1 establishes that attention is the direct channel when a role-induced difference enters a new token position. To examine how much of the realized role effect this pathway carries, we study its recoverability across attention terms and its decision effects. Specifically, for each matched context pair, we fix the exact trajectory score effect , and substitute one term at a time. We consider the direct role term , attention reweighting , and a matched non-role term.
Table 6 shows that the direct term recovers 73.5%–93.9% of the exact role effect across models. In contrast, recovery from attention reweighting and matched non-role positions remains near zero. Together with the sustained attenuation results in § 5.3.1, these findings show that the direct role span attention term carries a substantial and behaviorally consequential part of the role effect.
Since it is essential for the role effect to become available at the decision, we next attenuate the carried state at three depths. Table 7(a) shows that attenuation at 0.25 has little effect in any setting. By 0.50, the dependence begins to emerge, and at 0.75, the margin drop is positive for all settings. Thus, the role effect becomes relevant to the decision only after sufficient depth. Table 7(b) shows that attenuating either update family reduces the target margin. Across comparisons, the attention intervention produces a larger effect than the MLP intervention.
| (a) Carry drop by depth | (b) Update families | ||||
| Model | 0.5 | 0.75 | Attention | MLP | |
| Rule | |||||
| Qwen2.5-7B | [, 0.12] | [, 0.14] | 11.25 [7.18, 16.11] | 30.79 [24.51, 37.68] | 26.14 [21.55, 31.06] |
| Mistral-7B | 0.05 [, 0.11] | 0.77 [0.15, 1.42] | 12.56 [9.76, 15.32] | 15.64 [12.20, 19.21] | 13.40 [10.84, 15.97] |
| Llama-3.1-8B | [, 0.03] | 3.25 [2.10, 4.55] | 9.11 [8.20, 10.00] | 13.74 [11.59, 15.96] | 8.77 [7.71, 9.68] |
| Evidence | |||||
| Qwen2.5-7B | 0.02 [, 0.28] | 0.23 [, 0.61] | 4.25 [3.05, 5.59] | 8.76 [6.76, 11.01] | 7.18 [5.43, 9.02] |
| Mistral-7B | [, 0.07] | 1.10 [0.68, 1.53] | 5.00 [3.83, 6.19] | 8.70 [6.85, 10.83] | 7.81 [6.37, 9.27] |
| Llama-3.1-8B | 0.20 [, 0.41] | 2.61 [1.82, 3.46] | 4.68 [3.61, 5.76] | 7.90 [6.22, 9.70] | 4.01 [2.62, 5.54] |
Table 7. Accumulated carry and update-family effects at role-sensitive decisions. (a) The carried state entering the layer at three fractional depths. (b) Each update family at all layers. Note. Drops are computed as matched control minus targeted intervention, so larger values indicate stronger dependence.
Taken together, these results establish the empirical importance of the role span attention pathway. The direct attention term recovers a substantial part of the role effect, and this effect becomes increasingly available from the carried state with depth. Combined with the consistently larger intervention effect of attention than MLP, this pattern supports attention as the dominant route through which role information reaches the relevant token decisions.
| Model | Rule: Direct | Rule: Reweighting | Rule: Non-role | Evidence: Direct | Evidence: Reweighting | Evidence: Non-role |
| Qwen2.5-7B | 73.5 | 0.0 | 0.1 | 85.9 | 1.9 | 1.4 |
| Mistral-7B | 88.3 | 0.6 | 92.0 | 1.9 | ||
| Llama-3.1-8B | 87.7 | 0.8 | 93.9 | 0.4 | 0.8 |
Table 6. Direct recovery of role effects. Median normalized recovery (%) after substituting the direct role term (), attention reweighting (), or matched non-role positions.
Rule Contributions in Few-Shot ICL
Our main experiments place each rule in an explicit prompt span, allowing us to isolate its effect on generation. To test whether the same account extends to few-shot ICL, we use prompts in which labeled examples implicitly specify the rule. The model must infer the underlying task mapping from these examples. We then test whether the inferred mapping makes a measurable causal contribution to the decision and identify where this contribution is realized within the examples.
We use the SST-2 sentiment task from the Stanford Sentiment Treebank Socher et al. [21] and the six-class TREC question classification task Li & Roth [14]. To isolate the effect of the inferred mapping, we preserve the demonstrations, their order, the query, and the prompt format across each prompt pair, varying only the assignment of labels to classes. We use the demonstration blocks as the rule positions and matched positions outside them as controls.
Table 8(a) shows that all three models follow the induced mapping, while breaking the input–label pairing reduces mapping following with the same inputs and labels preserved in the prompt. This indicates that the rule is induced from relations established across the demonstrations. Table 8(b) shows that the inferred rule yields positive contribution margins and that substituting the demonstration positions recovers nearly the full effect, with negligible recovery from the matched control. Table 8(c) further shows that targeted attenuation reduces the label margin in all six settings and significantly lowers task accuracy in five, with the 95% CI. Together, these results show that the inferred mapping produces a measurable role span contribution that affects the query decision. We next examine where this contribution is realized within the demonstrations.
| (a) Rule induction | (b) Contribution | (c) Decision effect | |||||
| Model | Task acc. (%) orig./perm. | Mapping follow (%) | No-demo/broken (%) | Positive margin (%) | Recovery (%) demo/control | Margin drop [95% CI] | Accuracy drop [95% CI] (pp) |
| SST-2 | |||||||
| Qwen2.5-7B | 77.1 / 86.7 | 84.8 | 46.7 / 54.3 | 81.9 | 99.4 / 0.1 | 2.27 [1.7, 2.8] | 25.7 [15.2, 35.2] |
| Mistral-7B | 82.9 / 82.9 | 82.9 | 49.5 / 48.6 | 81.9 | 99.8 / 0.0 | 1.24 [0.8, 1.7] | 20.0 [7.6, 31.4] |
| Llama-3.1-8B | 73.3 / 77.1 | 79.0 | 41.0 / 41.9 | 77.1 | 96.2 / | 0.63 [0.3, 0.9] | 15.2 [6.7, 23.8] |
| TREC | |||||||
| Qwen2.5-7B | 42.9 / 46.7 | 65.7 | 20.0 / 25.7 | 76.2 | 99.4 / | 1.04 [0.4, 1.7] | 8.6 [, 20.0] |
| Mistral-7B | 52.4 / 47.6 | 66.7 | 14.3 / 21.9 | 74.3 | 98.5 / | 1.37 [0.9, 1.9] | 28.6 [18.1, 39.0] |
| Llama-3.1-8B | 35.2 / 37.1 | 73.3 | 14.3 / 17.1 | 66.7 | 97.0 / | 0.69 [0.2, 1.1] | 18.1 [8.6, 27.6] |
Table 8. Role effects for rules induced from demonstrations. The three column groups report rule induction, contribution alignment, and the effect of targeted attenuation. Note. Recovery is evaluated on examples with . Matched positions outside the demonstration blocks form the control. Drop metrics are computed as control minus targeted attenuation, so larger values indicate stronger dependence on the selected positions.
Note. Recovery is evaluated on examples with . Matched positions outside the demonstration blocks form the control. Drop metrics are computed as control minus targeted attenuation, so larger values indicate stronger dependence on the selected positions.
Table 9(a) shows that the label positions recover nearly the full effect of the demonstration blocks. The input positions recover only a small fraction of this effect. Table 9(b) shows the same organization under targeted attenuation. The margin drop at the label positions is close to that of all demonstration positions, while the input positions show a much smaller change. Together with the pairing result in Table 8(a), these findings separate rule formation from its direct realization at the query decision. The rule is induced from relations across demonstrations, and its direct contribution is concentrated at the label positions.
| Model | (a) Recovery (%) All | (a) Recovery (%) Labels | (a) Recovery (%) Inputs | (b) Margin drop vs. control All | (b) Margin drop vs. control Labels | (b) Margin drop vs. control Inputs |
| SST-2 | ||||||
| Qwen2.5-7B | 99.4 | 98.4 | 4.4 | 2.27 | 2.09 | 0.29 |
| Mistral-7B | 99.8 | 98.7 | 2.4 | 1.24 | 1.37 | 0.38 |
| Llama-3.1-8B | 96.2 | 95.1 | 3.2 | 0.63 | 0.76 | 0.23 |
| TREC | ||||||
| Qwen2.5-7B | 99.4 | 98.8 | 3.7 | 1.04 | 0.89 | 0.47 |
| Mistral-7B | 98.5 | 91.1 | 11.1 | 1.37 | 1.35 | 0.14 |
| Llama-3.1-8B | 97.0 | 90.9 | 1.6 | 0.69 | 0.49 | 0.10 |
Table 9. Position-wise organization of induced role effects. Recovery and margin drop are compared across all demonstration tokens, label tokens, and input tokens.
We further test whether failing label decisions can be restored by amplifying the demonstration contribution, using the same operation as in § 5.3.2. Table 10(a) shows that amplification restores a subset of these failing decisions, while random directions of equal norm do not at larger weights. Table 10(b) further shows that, at similar deficits, single-label rule decisions recover less often than evidence decisions. Table 10(c) shows that recovery follows the prediction in Appendix § 11.§ 4. Together, these results show that whether a failing decision can be restored depends on the support relative to the deficit.
| Direction | (a) Selection (%) | (a) Selection (%) | (a) Selection (%) | (b) Matched deficit (%) Group | (b) Matched deficit (%) Recovery | (c) Prediction at (%) Group | (c) Prediction at (%) Recovery |
| Demonstration | 12.9 | 18.8 | 22.5 | Single-label rule | 6.5 | 59.5 | |
| Random | 5.5 | 0.0 | 0.0 | Task-defined evidence | 47.6 | 0.4 | |
| Real-hallucinated | 58.8 |
Table 10. Restoration of failing single-label rule decisions in few-shot ICL. (a) Target selection under amplification. (b) Recovery at matched deficits. (c) Recovery by the predicted threshold.
Note. follows Appendix § 11.§ 4.
Evidence from Multiple Sources
Our main evidence experiments place the relevant fact in a single support span, allowing us to isolate its effect on a factual choice. We next examine evidence use across multiple sources. One setting requires the model to combine information from several passages, while the other presents conflicting facts from two sources. In each setting, we identify which evidence positions affect the selected answer and test whether weakening them changes the factual choice.
Distributed evidence. We first study questions from MuSiQue [24] that require two reasoning steps. Each answer depends on a bridge passage and a passage that contains the final fact. The paired prompts preserve the question, passage order, and all other text, while changing only the answer entity. We use the union of the two passages as the evidence positions and matched distractor positions as the control. This construction lets us compare the evidence needed to establish the answer with the evidence that acts at the final choice.
Note. Recovery difference (nats) is the score recovery at the answer decision from evidence substitution relative to the matched distractor substitution. Larger margin drop indicates stronger dependence.
As shown in Table 11(a), the matched effect favors the supported answer for all three models, and substitution through the evidence positions recovers most of this effect relative to the control. Removing either passage also reduces the answer margin, showing that both passages are needed to establish the answer (Table 11(b)). Their direct effects at the final choice are not equal. Weakening the passage that contains the final fact causes a large margin drop in every model, whereas weakening the bridge passage has little direct effect in Qwen and Mistral and a much smaller effect in Llama. Thus, the answer depends on evidence distributed across the two passages, but the final choice depends mainly on the passage that supplies the final fact.
| (a) Grounded preference and pathway recovery | |||||||
| Grounded preference (%) | Matched effect | Pathway recovery | |||||
| Model | Original | Counterfactual | No evidence | Median | Recovery difference (nats) | ||
| Qwen2.5-7B | 82.7 | 46.9 | 63.3 | 36.47 | 5.08 [4.0, 6.2] | ||
| Mistral-7B | 89.8 | 43.9 | 69.4 | 35.11 | 5.07 [4.2, 6.0] | ||
| Llama-3.1-8B | 98.0 | 50.0 | 76.5 | 26.08 | 4.30 [3.6, 5.0] | ||
| (b) Support dependence and decision-level effects | |||||||
| Prompt dependence: margin drop (nats) | Decision effect: margin drop vs. control | Token flip (%) | |||||
| Model | Bridge removed | Answer removed | All evidence | Answer span | Bridge span | Targeted | Control |
| Qwen2.5-7B | 1.98 [0.9, 3.1] | 5.82 [4.5, 7.2] | 5.89 [4.7, 7.1] | 4.68 [3.7, 5.7] | 0.09 [−0.4, 0.6] | 11.2 | 1.0 |
| Mistral-7B | 1.27 [0.7, 1.9] | 5.46 [4.4, 6.6] | 3.98 [3.2, 4.8] | 3.48 [2.8, 4.2] | 0.14 [−0.1, 0.4] | 8.2 | 0.0 |
| Llama-3.1-8B | 1.07 [0.6, 1.5] | 4.50 [3.7, 5.3] | 4.37 [3.6, 5.1] | 3.42 [2.8, 4.0] | 0.43 [0.1, 0.8] | 12.2 | 0.0 |
Table 11. Evidence contributions under distributed multi-hop grounding. (a) Grounded preference and pathway recovery. (b) Dependence on each supporting passage and its direct effect at the answer decision.
Competing evidence. We next consider contexts in which two sources state different entities for the same fact. The prompt marks one source as primary. The paired prompts exchange the entities stated by the two sources while preserving the question, passage order, and remaining text. A closed-book condition measures the preference that remains without either claim. We then weaken the two sources separately to test how each one affects the same factual choice.
Table 12(a) shows that context-conditioned preference varies across settings. The original and closed-book rates are nearly identical for Qwen, separate more for Mistral, and differ most clearly for Llama. The matched effect remains positive in all three models. Table 12(b) reveals a more consistent source-specific effect. Weakening the primary source lowers the margin of its answer, whereas weakening the competing source can raise it. The primary intervention also changes the selected token far more often than the control. Thus, the models distinguish the effects of the two sources, even though the designated source does not exert the same control over the final answer.
| Model | Primary preference (%) | Matched effect | Pathway recovery | |||
| Original | Swapped | Closed book | Median | Positive (%) | Recovery difference (nats) | |
| Qwen2.5-7B | 28.4 | 25.3 | 26.3 | 1.06 | 53.7 [43.2, 64.2] | 4.99 [4.0, 6.0] |
| Mistral-7B | 31.6 | 13.7 | 22.1 | 1.90 | 65.3 [55.8, 74.7] | 4.63 [3.9, 5.3] |
| Llama-3.1-8B | 34.7 | 5.3 | 20.0 | 4.03 | 81.1 [72.6, 88.4] | 4.70 [4.1, 5.3] |
| Model | Primary drop | Secondary gain | Primary | Control | ||
| Qwen2.5-7B | 6.43 [5.2, 7.7] | 5.30 [4.2, 6.5] | 23.2 | 3.2 | ||
| Mistral-7B | 5.12 [4.3, 5.9] | 3.91 [3.2, 4.6] | 26.3 | 1.1 | ||
| Llama-3.1-8B | 6.30 [5.5, 7.1] | 3.34 [2.8, 3.9] | 31.6 | 0.0 |
Table 12. Evidence contributions under competing factual sources. (a) Context-conditioned preference and pathway recovery. (b) Source-specific effects under targeted attenuation.
Taken together, the two settings show that evidence dependence can be distributed across sources, while direct influence and final control remain source-specific. The context determines which information is required, that information shapes the factual choice, and the resulting influence may or may not determine the final answer. These findings extend the evidence contribution account. beyond a single support span while preserving the distinction between contextual influence and final behavior.
Role Span Influence around Key Decisions
So far, our analysis shows that role span contributions act through the output decisions required by each role. The same pattern extends to rules inferred from demonstrations and evidence drawn from multiple sources in Appendices D.4 and D.5. For rules, these decisions govern structure and selection; for evidence, they determine factual content in the response. One may wonder how the role effect is organized around these semantic decisions.
We answer this by comparing role span influence at the semantic decisions with matched output positions. As shown in Table 13(a), role span influence is consistently higher at the semantic decisions than at the matched positions. The same concentration appears when the decisions are specified from reference responses. These results show that semantic decisions form distinct points of role span influence in the output.
| Model | Decision percentile | Matched control | Median offset | Before dec. (%) | At dec. (%) |
| Rule | |||||
| Qwen2.5-7B | 85.0 | 67.4 | +0 | 46.9 | 53.1 |
| Mistral-7B | 85.7 | 63.6 | −1 | 55.9 | 44.1 |
| Llama-3.1-8B | 90.0 | 63.0 | −5 | 55.3 | 44.7 |
| Evidence | |||||
| Qwen2.5-7B | 92.5 | 50.2 | −8 | 86.0 | 8.0 |
| Mistral-7B | 77.5 | 51.2 | −6 | 87.2 | 4.3 |
| Llama-3.1-8B | 95.0 | 50.5 | −6 | 83.7 | 9.3 |
Table 13. Role span behavior at semantic decisions. (a) Role span influence at semantic decisions: the median within-response percentile of at the decision position and at matched positions in other responses. (b) Position of the first generation change relative to the semantic decision. (a) Role span influence (%) (b) Position of generation changes.
Step further, we ask how the generation change is positioned around these semantic decisions. We measure the first divergence relative to each decision position. As shown in Table 13(b), rule changes occur at or shortly before the decision itself, with 44%–53% of first divergences occurring exactly at that position. In contrast, evidence shows a different pattern: 84%–87% of first divergences occur before the factual value is emitted, with median offsets of several tokens. These results indicate that rule and evidence exhibit different positional patterns around their semantic decisions.
We further examine which pathway carries these earlier changes. On a fixed prefix, we remove only the direct evidence term at a position, or remove the remaining transport terms while keeping it. As shown in Table 14, the earlier first divergences occur at low-margin positions and are mainly carried by the indirect terms. In contrast, removing only the direct evidence term at the factual decision changes the selected fact far more often than the same removal on a matched non-evidence span. These results indicate that indirect transport shapes the tokens preceding the fact, while the factual decision itself remains sensitive to the direct evidence contribution.
| Direct evidence removed | Matched non-evidence | Indirect share median (%) | Margin median | |
| All models | 31.6 | 1.7 | 91.8 | 0.6 |
Table 14. Direct and indirect evidence effects on a fixed prefix. (a) Changes in the selected fact at the factual decision. (b) Pathway share and margin at the first divergence. (a) Factual decision (%) (b) First divergence.
In short, the semantic decisions provide distinct points of role span influence, while generation changes follow different positional patterns around them. Functional roles therefore organize both the output decisions that carry role span influence and the local pattern of generation changes around those decisions.
Note. Primary drop is the margin decrease of the primary asserted candidate under targeted attenuation, relative to the matched control; Secondary gain is the margin increase after attenuating the competing source.
Note. (a) Both columns show the median within-response percentile of ; Matched reads the same positions in other responses. (b) Offset is the first-divergence position minus the semantic-decision position (in tokens); negative values indicate an earlier change.
Note. The prefix is fixed to the original generation. Indirect terms follow the decomposition in Appendix § 11.§ 1.
Predicting Task Recovery
Appendix § 11.§ 4 predicts the recovery of each naturally failing decision under role contribution amplification. We first compare these predictions with the subsequent intervention outcomes. Across the two roles, the predicted and observed decision outcomes agree in 79% of the cases, while the predicted amplification closely tracks the first observed crossing among recovered decisions.
Here, we show that the decision-level predictions can extend to task behavior. Following the role-specific decision organization identified in § 5.2, we map the predicted decision outcomes to task recovery. For evidence, recovery of the factual choice determines the grounding prediction. For rules, the decision-level prediction is not directly comparable, since successful rule following requires sustained support across multiple structural and selection choices.
As shown in Table 15, the observed task recovery follows the predicted trend for evidence, while rule recovery stays near zero, as predicted. For evidence, task success increases from 25.2% to 58.3%, compared with a predicted increase from 23.3% to 69.9%. For rules, both remain near zero over the same range. Thus, we show that the first-order decision formulation can be used to quantitatively predict task recovery in both ICL and OOC hallucination.
| Rule span Predicted (%) | Rule span Observed (%) | Evidence span Predicted (%) | Evidence span Observed (%) | |
| 0.5 | 0.0 | 0.0 | 23.3 | 25.2 |
| 1 | 0.0 | 1.3 | 26.2 | 32.0 |
| 2 | 0.0 | 2.5 | 38.8 | 47.6 |
| 4 | 0.0 | 6.3 | 52.4 | 53.4 |
| 8 | 0.0 | 3.8 | 64.1 | 55.3 |
| 16 | 6.3 | 6.3 | 69.9 | 58.3 |
Table 15. Task-level recovery prediction under role contribution amplification. Predicted versus observed task success across amplification strengths.
References
- [1]Shengnan An, Zexiong Ma, Zèqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188, 2024.arxiv.org/abs/2404.16811
- [2]Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannahneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, pp. 9112–9141, 2024.arxiv.org/abs/2310.11511
- [3]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.arxiv.org/abs/2102.12452
- [4]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dharival, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
- [5]Hakaze Cho, Mariko Kato, Yoshihiro Sakai, and Naoya Inoue. Revisiting in-context learning inference circuit in large language models. In International Conference on Learning Representations, volume 2025, pp. 13081–13117, 2025.arxiv.org/abs/2410.04468
- [6]Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 1107–1128, 2024.
- [7]Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yuqiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. CI-bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587, 2026.arxiv.org/abs/2602.03587
- [8]Kevin Du, Vëstein Snæbjarnarson, Niklas Stoehr, Jennifer White, Aaron Schein, and Ryan Cotterell. Context versus prior knowledge in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13211–13235, 2024.arxiv.org/abs/2404.04633
- [9]Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375–24410, 2024.arxiv.org/abs/2406.11944
- [10]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to learn in context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4849–4870, 2023.
- [11]Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9318–9333, 2023.
- [12]Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yangfan Ye, Weihong Zhong, Yuxuan Gu, Baoxin Wang, Dayong Wu, et al. Improving contextual faithfulness of large language models via retrieval heads-induced optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16896–16913, 2025a.arxiv.org/abs/2501.13573
- [13]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Ruan, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM transactions on information systems, 43(2):1–55, 2025b.DOI
- [14]Xin Li and Dan Roth. Learning question classifiers. In Coling 2002: The 19th international conference on computational linguistics, 2002.
- [15]Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086–8098, 2022.arxiv.org/abs/2104.08786
- [16]Clement Neo, Shay B Cohen, and Fazl Barez. Interpreting context look-ups in transformers: Investigating attention-mlp interactions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16681–16697, 2024.
- [17]Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10862–10878, 2024.arxiv.org/abs/2401.00396
- [18]Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Revisiting demonstration selection strategies in in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9090–9101, 2024.
- [19]Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 6916–6932, 2024.arxiv.org/abs/2402.13055
- [20]Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International conference on machine learning, pp. 31210–31227. PMLR, 2023.arxiv.org/abs/2302.00093
- [21]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.DOI
- [22]Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319–3328. PMLR, 2017.arxiv.org/abs/1703.01365
- [23]Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. In International conference on learning representations, volume 2024, pp. 17282–17333, 2024.arxiv.org/abs/2310.15213
- [24]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022.DOI
- [25]Xubin Wang, Jianfei Wu, Yichen Yuan, Deyu Cai, Mingzhe Li, and Weijia Jia. Demonstration selection for in-context learning via reinforcement learning. arXiv preprint arXiv:2412.03966, 2024.arxiv.org/abs/2412.03966
- [26]Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. Advances in neural information processing systems, 37:33402–33422, 2024.arxiv.org/abs/2404.10198
- [27]Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In International Conference on Learning Representations, volume 2025, pp. 62143–62156, 2025.arxiv.org/abs/2404.15574
- [28]Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In International Conference on Learning Representations, volume 2024, pp. 35623–35646, 2024.arxiv.org/abs/2305.13300
- [29]Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pp. 39818–39833. PMLR, 2023.arxiv.org/abs/2302.05698
- [30]Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In International Conference on Learning Representations, volume 2024, pp. 29862–29883, 2024.arxiv.org/abs/2310.01558
- [31]Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. Mechanistic understanding and mitigation of language model non-factual hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7943–7956, 2024.arxiv.org/abs/2403.18167
