betaFeedback

WorksComputer Science

A Mechanistic Analysis of In-Context Learning and Out-of-Context Hallucination

Abstract

Large language models can infer and follow specific rules through in-context learning (ICL), yet may fail to remain faithful to provided evidence, giving rise to out-of-context (OOC) hallucination. Existing work has largely investigated the underlying mechanisms of ICL and OOC hallucination along separate lines. However, this separation obscures the mechanistic basis for these seemingly opposing outcomes. In this work, we study this contrast through the unified view of functional roles in dynamic context. The key observation is that role information can be internally represented without reliably constraining output. We derive a first-order decomposition to characterize the role effect, showing that its substantial influence depends on two necessary conditions, namely sufficient contribution magnitude and coherent signed aggregation. Causal analysis further shows that role span contributions affect key token choices involving response structure, content selection, and factual values. We further show that role-consistent choices depend on sufficient role span support at these key decisions, while failing choices can be reversed when this support overcomes the deficit. These findings support a common mechanistic view of ICL and OOC hallucination: each functional role defines the pattern of output decisions over which role span contributions must remain effective, and differences in this organization account for the seemingly opposing outcomes.

Introduction

A central strength of LLMs is the ability to condition generation on dynamic instructions. This capability is exemplified by in-context learning (ICL), where models can infer and follow specific rules supplied in the prompt [4, 6]. The same dependence on context, however, also exposes a major reliability challenge. Models may fail to remain faithful to the provided evidence and even produce unsupported claims, a failure mode we refer to as out-of-context (OOC) hallucination [13, 26]. This contrast raises a natural question: why can the same model reliably follow a prompt-specified rule, yet fail to remain faithful to evidence that is likewise supplied in context? Although existing work studies the mechanisms underlying ICL and OOC hallucination separately [11, 23, 31], the mechanistic origin of this contrast remains unclear.

Fundamentally, dynamic context serves different functional roles in generation: rules specify how to respond, whereas evidence specifies what factual content the response should be grounded in. The same content can be interpreted differently according to its function in the current prompt. The same sentence may act as a rule, a distractor, or ordinary background across tasks [20]. While contemporary LLMs receive no explicit supervision for these roles, recent mechanistic studies of ICL suggest that task structure can nevertheless be inferred from context [5].

In this paper, we present a counter-intuitive finding: role information can be internally represented without reliably constraining the resulting response. We decompose this distinction into two questions: (1) how functional roles are internally represented, and (2) how role information affects LLM1 generation. Despite their different outcomes, ICL and OOC hallucination can be analyzed through a unified framework focusing on the role effect, while preserving the distinct functions of rules and evidence. We connect these questions by examining how information from role-bearing spans affects token predictions and choices during generation. Crucially, a role can affect the model’s predictions without determining its final choice.

The effect of dynamic context is inherently non-uniform [1]. The prefix can establish the direction of the continuation and persistently influence subsequent predictions. For instance, rules can shape how the response is organized, whereas evidence can determine which factual content is carried forward [28, 8, 27]. We isolate the attention contributions from role-bearing spans, termed role span contributions, and examine how they affect next-token predictions. Our experiments show that successful role-conditioned generation depends on sufficient contribution support at the relevant token decisions. From this perspective, we show that ICL and OOC hallucination share the same contextual-influence mechanism, while it takes effect at different types of decisions in the two settings. Interested readers can refer to Appendix § 10 for a thorough discussion of related work.

Our contributions are threefold. First, we establish a unified account of contextual influence in ICL and OOC hallucination by distinguishing internal role representation from its effect on generation. Second, we derive a first-order decomposition that identifies two necessary conditions for predictive influence: sufficient contribution magnitude and coherent signed aggregation. Third, we show that ICL and OOC hallucination share a role span contribution mechanism, while their contrasting outcomes depend on effective support at different generation decisions.

Preliminaries

Setup. Let CC denote the context, xx the query, and y=(y1,…,yT)y = (y_1, \ldots, y_T) a response trajectory. In ICL, the context supplies a rule for answering the query, whereas in OOC hallucination, it supplies evidence against which the response is evaluated. We index two functional roles, rule and evidence, by f∈{r,e}f \in\{r, e\}, and use CfC_f to denote the corresponding role span. For an input PP, let Sf(P)S_f(P) denote its token positions. At decoding step tt, the model conditions on the prefix y<ty_{<t} and defines a next-token distribution pθ(⋅∣C,x,y<t)p_\theta(\cdot\mid C, x, y_{<t}). We refer to this next-token prediction as the decision at step tt, and to yty_t as its choice. As the role span sits in the prompt, its effect on generation is realized through these decisions.

Matched contexts. To isolate the effect of role ff, we construct a matched counterfactual context C~f\widetilde{C}_f that changes only the content of the role span while holding the remaining prompt fixed, as illustrated in Figure 1. The role ff itself is unchanged, and only what it specifies differs. We write PfP^f and P~f\widetilde{P}^f for the inputs constructed from Cf⊕xC_f \oplus x and C~f⊕x\widetilde{C}_f \oplus x, where ⊕\oplus denotes sequence concatenation.

Figure 1. Examples of matched contexts. Construction details are provided in Appendix § 12.§ 1.

Residual response. Inspired by prior work on LLM internal mechanisms [23], we investigate how a role span is represented in the model and how it affects the internal state. Let hℓ(P,i)∈Rdh_\ell(P,i) \in\mathbb{R}^d denote the residual stream representation at layer ℓ\ell and position ii when the model processes PP. We define the residual response as Δhℓf(i)=hℓ(Pf,i)−hℓ(P~f,i)\Delta h_\ell^f(i) = h_\ell(P^f,i) - h_\ell(\widetilde{P}^f,i), which points from the matched counterfactual state to the original role instantiation.

Role Spans Induce Functional Residual Responses

ICL and OOC hallucination differ in their empirical outcomes, yet both rely on dynamic context within the same autoregressive decoding framework. Despite the shared structure, context serves distinct functional roles in the two settings. This raises a natural question: how does the functional role organize the model’s internal state? We address this through matched role comparisons in the residual stream, linking the role distinction to its internal representational form. We first characterize the organization of the induced state changes and then examine their relation to the decoding process.

Structured Residual Responses from Role Spans

The same sentence can serve different functional roles, such as an instruction, evidence, or distractor. To examine how this distinction appears in internal states, we keep the span content fixed while changing only its function. Specifically, we construct multiple functional framings for each span and train a linear readout to distinguish the corresponding roles, following Belinkov [3]. We find that the readout generalizes consistently across prompt formulations, indicating a stable functional distinction in internal states.

We next examine how functional role structures internal states. To test this, for each role ff, we fit the top 10 PCA directions on one half of the samples and measure the fraction of Δhℓf\Delta h_{\ell}^{f} energy captured on the other half. Ideally, a high captured fraction would indicate shared organization, whereas isotropic differences would remain near the random baseline. Figure 2 shows that the energy concentration remains well above the baseline. Rule responses exhibit strong concentration, whereas evidence responses are more distributed. These results show that the residual responses induced by functional role exhibit systematic low-dimensional organization.

Structured residual responses

Figure 2. Structured residual responses. Left panels show rule span responses, and right panels show evidence span responses. Each violin shows the fraction of Δhℓf\Delta h_{\ell}^{f} energy captured by a 10-dimensional PCA subspace; points are colored by the percentile of ∥Δh∥\lVert\Delta h\rVert. The dashed gray line denotes the expected random subspace baseline k/H≈0.2%k/H \approx0.2\%, with k=10k = 10 PCA directions and residual dimension.

Residual Responses Toward Role-Consistent Generation

The low-dimensional organization observed above shows that role spans can induce structured representational change during inference. Since the residual stream connects prompt processing to token prediction, we next examine how these changes relate to the output process. To make this relation concrete, we consider two empirical implications: (1) The dominant directions of the residual subspace should align with the unembedding directions of outputs. This follows from the fact that the LLM maps the residual representation to token logits through the final normalization and unembedding matrix WUW_U. Here, the structured Δh\Delta h should appear as a larger projection onto the output directions. (2) The representation change induced by a specific role span should preserve relevant role information. Since the residual stream connects intermediate representations with downstream computation, the role information represented by Δh\Delta h should be maintained in a form that later blocks can read, transform, and propagate.

To examine the first empirical implication, we project Δh\Delta h through WUW_U, measuring its projection and logit shift toward the output. We use a matched non-role perturbation as the comparison group. Figure 3 shows that, across all three models, role responses are more strongly aligned with the corresponding output directions, which results in larger logit shifts. Specifically, rule responses show clearer positive projection, while evidence responses separate more through the logit shift. By contrast, the comparison group remains weaker in both projection and logit shift. These results show that, during the decoding process, structured Δh\Delta h appears as a directional logit shift toward the output direction indicated by the role span.

Final layer decoding projection of $\Delta h$

Figure 3. Final layer decoding projection of Δh\Delta h. Blue points denote the comparison group.

To examine the second empirical implication, we evaluate behavior recovery in states induced by the matched counterfactual context; details are provided in Appendix § 12.§ 2. As shown in Table 1, matched role responses restore the original behavior associated with the role, showing that the injected residual response remains usable by subsequent computation. Moreover, recovery remains low for the comparison groups since they lack the role conditioned state required for recovery. These results indicate that the induced representational change carries transferable role information that remains usable by downstream computation.

Injected responseQwenMistralLlamaAvg.
Rule
matched role response0.941.001.000.98
no role content0.170.080.000.08
mismatched role response0.100.310.050.15
Evidence
matched role response0.710.800.930.81
no role content0.440.200.370.34
mismatched role response0.350.250.040.21

Table 1. Behavior recovery rates.

Our findings suggest that functional roles in ICL and OOC hallucination can be characterized in the representation space. Rule and evidence spans induce a common form of structured residual response that carries role-specific information forward for downstream computation.

Role Span Contributions in Generation

So far, we have characterized how role spans are internally represented, yet how they shape LLM generation remains unclear. We examine two questions: (i) whether role information can shift the model’s preference toward specific responses, and (ii) how role spans affect token during generation, leading to distinct generation outcomes. Specifically, we develop a theoretical framework that relates role-induced changes to internal LLM computations and derive a first-order decomposition to characterize the conditions under which role span contributions can influence LLM behavior.

Response Trajectory Score Shift

To characterize how rule or evidence spans influence generation, we examine their effects on next-token scores. Since autoregressive predictions depend on the preceding response history, changes in role information can lead to different response prefixes during generation. Such differences can confound the influence of role information on subsequent predictions. To isolate the influence of a role span at a given position, we evaluate the same token choice under a common response history. For each role f∈{r,e}f \in\{r,e\}, we consider a response trajectory y=(y1,…,yT)\mathbf{y}=(y_1,\ldots,y_T), which can be either a constructed candidate or a response generated autoregressively. In both cases, we evaluate the score of yty_t under contexts CfC^f and C~f\widetilde{C}^f, holding the query xx and response prefix y<ty_{<t} fixed. We defer further details to Appendix § 12.§ 1. Specifically, we write the stepwise score as ψt(P)=log⁡pθ(yt∣P)\psi_t(P)=\log p_{\theta}(y_t \mid P), and define the resulting score shift as δtf(y)=ψt(Ptf(y))−ψt(P~tf(y))\delta_t^f(\mathbf{y})=\psi_t(P_t^f(\mathbf{y}))-\psi_t(\widetilde{P}_t^f(\mathbf{y})). Based on the autoregressive factorization, the change in the log probability assigned to y\mathbf{y} is the sum of these stepwise differences. We formalize this quantity in the following definition.

Definition 1 (Response trajectory score shift). For a response sequence y=(y1,…,yT)y=(y_1,\ldots,y_T), define the stepwise score shift as δtf(y)=ψt(Ptf(y))−ψt(P~tf(y))\delta_t^f(y)=\psi_t(P_t^f(y))-\psi_t(\widetilde{P}_t^f(y)). The response trajectory score shift is Φf(y)=∑t=1Tδtf(y)\Phi_f(y)=\sum_{t=1}^{T}\delta_t^f(y), which measures the change in the log probability assigned to the same response under CfC_f and C~f\widetilde{C}_f. Equivalently, using the autoregressive product form,

Φf(y)=log⁡pθ(y∣Cf⊕x)−log⁡pθ(y∣C~f⊕x)=∑t=1T[log⁡pθ(yt∣Cf⊕x⊕y<t)−log⁡pθ(yt∣C~f⊕x⊕y<t)].(1)\Phi_f(y)=\log p_\theta(y\mid C_f\oplus x)-\log p_\theta(y\mid\widetilde{C}_f\oplus x) =\sum_{t=1}^{T}\left[\log p_\theta(y_t\mid C_f\oplus x\oplus y_{<t})-\log p_\theta(y_t\mid\widetilde{C}_f\oplus x\oplus y_{<t})\right]. \tag*{(1)}

Here, Φf(y)>0\Phi_f(y)>0 indicates that CfC_f yields a higher score than C~f\widetilde{C}_f does. In what follows, we relate the stepwise score shifts δtf(y)\delta_t^f(y) to forward pass quantities associated with the role span.

Role Span Contributions in the Forward Pass

We mainly consider decoder-only LLMs, where successive attention and MLP modules update the residual stream before the final unembedding. The role-induced responses observed in § 3 are formed and propagated through these updates. To relate this process to the role span, we write the residual update explicitly. For a processed input PP, the residual state at layer ℓ\ell and position ii is updated as hℓ(P,i)=hℓ−1(P,i)+Attn⁡ℓout(P,i)+MLP⁡ℓout(P,i)h_\ell(P,i)=h_{\ell-1}(P,i)+\operatorname{Attn}^{\mathrm{out}}_\ell(P,i)+\operatorname{MLP}^{\mathrm{out}}_\ell(P,i). This reveals that information carried by different roles is accumulated across depth. A difference induced by role ff can therefore reflect both information inherited from previous layers and the current updates. Recall that the role span Sf(P)S_f(P) is defined by source token positions. Among the two update terms above, the attention output can be explicitly written as a sum over source token positions:

Attn⁡ℓout(P,i)=∑a∑sαi←s(a)(P)Vℓ(a)(P,s)WO(a)(2)\operatorname{Attn}^{\mathrm{out}}_\ell(P,i)=\sum_a\sum_s\alpha_{i\leftarrow s}^{(a)}(P)V_\ell^{(a)}(P,s)W_O^{(a)} \tag*{(2)}

where aa indexes attention heads and ss indexes source token positions in PP that are visible to the target position ii under the causal mask. Eq. (2) decomposes the attention update at ii into contributions aggregated from source positions. The key observation is that the role span Sf(P)S_f(P) is also a set of input token positions. We can therefore isolate the contribution of the role span by retaining the terms with s∈Sf(P)s\in S_f(P), giving

Bℓ,if(P)=∑a∑s∈Sf(P)αi←s(a)(P)Vℓ(a)(P,s)WO(a).(3)B_{\ell,i}^f(P)=\sum_a\sum_{s\in S_f(P)}\alpha_{i\leftarrow s}^{(a)}(P)V_\ell^{(a)}(P,s)W_O^{(a)}. \tag*{(3)}

We term Bℓ,if(P)B_{\ell,i}^f(P) the role span contribution to the attention update at layer ℓ\ell and position ii. In decoder-only Transformers, attention directly transfers differences across token positions, whereas normalization and MLP blocks act only within individual positions. Appendix B.1 formalizes this distinction.

First-Order Decomposition of Score Shifts

The response trajectory score shift Φf(y)\Phi_f(y) accumulates the stepwise shifts δtf(y)\delta_t^f(y) across response positions. We therefore analyze each stepwise shift through selected role span contribution coordinates and quantify the first-order score change associated with their matched differences. To make this operational, we approximate the stepwise score by its first-order change with respect to the selected Bℓ,ifB_{\ell,i}^f coordinates. Specifically, for each decoding step tt, let Bf,t⊆{(ℓ,i)}\mathcal{B}_{f,t}\subseteq\{(\ell,i)\} denote the role span contribution indices considered in the expansion. For (ℓ,i)∈Bf,t(\ell,i)\in\mathcal{B}_{f,t}, define ΔBℓ,i,tf(y)=Bℓ,if(Ptf(y))−Bℓ,if(P~tf(y))\Delta B_{\ell,i,t}^f(y)=B_{\ell,i}^f(P_t^f(y))-B_{\ell,i}^f(\widetilde{P}_t^f(y)). Collecting these differences gives ΔBf,tf(y)=(ΔBℓ,i,tf(y))(ℓ,i)∈Bf,t\Delta B_{f,t}^f(y)=\left(\Delta B_{\ell,i,t}^f(y)\right)_{(\ell,i)\in\mathcal{B}_{f,t}}.

The vector ΔBf,tf(y)\Delta B_{f,t}^f(y), therefore, measures how the matched role comparison changes the selected role span contributions over Bf,t\mathcal{B}_{f,t}.

Assumption 1 (Local score regularity). For a fixed response position tt, let b~\tilde b and bfb^f denote the selected role span contribution vectors induced by P~tf(y)\widetilde{P}_t^f(y) and Ptf(y)P_t^f(y), respectively. Define Ψt(b)\Psi_t(b)

as the score obtained by replacing, in layer order, the selected contributions under P~tf(y)\widetilde{P}^{f}_{t}(y) with the corresponding entries of bb. By construction, Ψt(b~)=ψt(P~tf(y))\Psi_{t}(\widetilde{b})=\psi_{t}(\widetilde{P}^{f}_{t}(y)); let Et⊥=ψt(Ptf(y))−Ψt(bf)E^{\perp}_{t}=\psi_{t}(P^{f}_{t}(y))-\Psi_{t}(b^{f}). We assume that Ψt\Psi_{t} is differentiable and has an LtL_{t}-Lipschitz gradient on the line segment from b~\widetilde{b} to bfb^{f}, and write ∇Bℓ,ifψt:=∇b~ℓ,iΨt(b)\nabla_{B^{f}_{\ell,i}}\psi_{t}:=\nabla_{\widetilde{b}_{\ell,i}}\Psi_{t}(b).

Under Assumption 1, a first-order expansion of Ψt\Psi_{t} from b~\widetilde{b} to bfb^{f} yields the following decomposition.

Theorem 1 (First-order role span score decomposition). Suppose Assumption 1 holds. Then the score shift satisfies

δtf(y)=∑(ℓ,i)∈Bf,t⟨∇Bℓ,ifψt,ΔBℓ,i,tf(y)⟩+Rt,(4)\delta^{f}_{t}(y)=\sum_{(\ell,i)\in B_{f,t}}\left\langle\nabla_{B^{f}_{\ell,i}}\psi_{t},\Delta B^{f}_{\ell,i,t}(y)\right\rangle+R_{t}, \tag*{(4)}

where the gradients are evaluated at b~\widetilde{b}, and the remainder satisfies

∣Rt∣≤Lt2∥ΔB:,tf(y)∥2+∣Et⊥∣.(5)|R_{t}|\leq\frac{L_{t}}{2}\left\|\Delta B^{f}_{:,t}(y)\right\|^{2}+|E^{\perp}_{t}|. \tag*{(5)}

The detailed derivations are provided in Appendix § 11.§ 2. Eq. (4) expresses the stepwise score shift as selected first-order terms plus a remainder. We use the Bℓ,ifB^{f}_{\ell,i} coordinates to isolate the direct source-indexed role span attention pathway, while effects outside these coordinates remain in RtR_{t}. We define the selected first-order score contribution associated with (ℓ,i)(\ell,i) as

Cℓ,i,tB,f=⟨∇Bℓ,ifψt,ΔBℓ,i,tf(y)⟩.(6)C^{B,f}_{\ell,i,t}=\left\langle\nabla_{B^{f}_{\ell,i}}\psi_{t},\Delta B^{f}_{\ell,i,t}(y)\right\rangle. \tag*{(6)}

Note that Cℓ,i,tB,fC^{B,f}_{\ell,i,t} is an inner product between the local score gradient and the matched contribution difference, so it gives a directional contribution to the score shift. Hence, Eq. (6) makes two factors explicit: (1) the matched difference ΔBℓ,i,tf(y)\Delta B^{f}_{\ell,i,t}(y) captures the role-dependent change in the forward pass quantity, and (2) the gradient ∇Bℓ,ifψt\nabla_{B^{f}_{\ell,i}}\psi_{t} captures the local sensitivity of the stepwise score to this quantity. Equivalently, writing the inner product in norm-angle form gives

Cℓ,i,tB,f=∥∇Bℓ,ifψt∥∥ΔBℓ,i,tf(y)∥cos⁡∠(∇Bℓ,ifψt,ΔBℓ,i,tf(y)).(7)C^{B,f}_{\ell,i,t}=\left\|\nabla_{B^{f}_{\ell,i}}\psi_{t}\right\|\left\|\Delta B^{f}_{\ell,i,t}(y)\right\|\cos\angle\left(\nabla_{B^{f}_{\ell,i}}\psi_{t},\Delta B^{f}_{\ell,i,t}(y)\right). \tag*{(7)}

To this end, we identify three distinct factors: the matched difference ΔBℓ,i,tf(y)\Delta B^{f}_{\ell,i,t}(y), the local score sensitivity, and their directional alignment. Summing Eq. (4) over response positions and using Eq. (6) gives

Φf(y)=∑t=1T∑(ℓ,i)∈Bf,tCℓ,i,tB,f+R,R=∑t=1TRt.(8)\Phi_{f}(y)=\sum_{t=1}^{T}\sum_{(\ell,i)\in B_{f,t}}C^{B,f}_{\ell,i,t}+R,\qquad R=\sum_{t=1}^{T}R_{t}. \tag*{(8)}

Together, Eqs. (7) and (8) show how local role span contributions organize into a trajectory-level score shift. Locally, the contribution magnitude depends on both the matched change in Bℓ,ifB^{f}_{\ell,i} and the sensitivity of the stepwise score, while the directional alignment determines whether the term supports or opposes the score shift. Across the generation, the signed terms must aggregate coherently rather than cancel. The first-order decomposition therefore identifies two conditions for a trajectory-level effect, namely sufficient magnitude and coherent signed aggregation.

Experiments

In this section, we investigate the mechanism underlying the common dependence on context and the distinct generation outcomes of ICL and OOC hallucination. We use rule-following tasks derived from CL-bench ([7]) and evidence-grounding examples from RAGTruth ([17]), with settings provided in Appendix § 12. To explain these distinct outcomes, we address two related questions: how role information affects model predictions and how its contributions influence token selection during generation. Our experiments examine three claims: (i) that role span contributions produce role-consistent preference shifts through organized layerwise effects, (ii) that rule and evidence realize this influence through different types of key token decisions, and (iii) that successful choices depend on sufficient contribution support, while failed choices can be reversed when this support overcomes the decision deficit. Appendices § 13.§ 4 and § 13.§ 5 extend the analysis to broader ICL and evidence-grounding settings.

Role Span Contribution Mechanism

In this subsection, we empirically test how role span contributions translate into predictive influence as the first-order decomposition suggests. If a role span can guide predictions toward the specified content, it must increase the relative support for that content over alternatives. To examine this, we compare two candidate responses y+y^{+} and y−y^{-}, where y+y^{+} follows the rule or is supported by the evidence, whereas y−y^{-} does not. Since we aim to measure how role information shifts predictive support between these alternatives, we evaluate their conditional probabilities under matched contexts, independently of the model’s generated response. We define Mf=Φf(y+)−Φf(y−)M_f = \Phi_f(y^{+}) - \Phi_f(y^{-}) to quantify the preference change. Here, Mf>0M_f > 0 indicates that role information increases the relative support for y+y^{+} over y−y^{-}. Construction details are provided in Appendix § 12.§ 3. As shown in Figure 4(a), we empirically observe that frac⁡(Mf>0)\operatorname{frac}(M_f > 0) is high for both roles, indicating that the role span contribution consistently favors role-compatible responses over alternatives. More specifically, this preference is expressed as a tendency toward rule-following responses in ICL and evidence-grounded responses in the OOC setting.

Response preference and contribution organization

Figure 4. Response preference and contribution organization. (a) Fraction of positive candidate margins. For Qwen2.5-7B, (b) layerwise profiles of ∣ΔBℓ,t∣\lvert\Delta B_{\ell,t}\rvert and ∣∇ψt∣\lvert\nabla\psi_t\rvert, (c) signed layerwise contributions and sign coherence, and (d) preference margin MfM_f under baseline, role span pathway ablation, and random ablation. More results are in Appendix § 13.§ 1.

Since the contribution propagates through the forward pass, its final effect accumulates across layers. The decomposition in Theorem 1 suggests two conditions for a substantial first-order contribution: (1) the contribution magnitude must be sufficiently large, and (2) the signed layerwise terms must aggregate coherently. Based on Eq. (7), we use ∥∇ψt∥⋅∥ΔBℓ,t∥\lVert\nabla\psi_t\rVert\cdot\lVert\Delta B_{\ell,t}\rVert to measure contribution magnitude and the cosine term to determine the signed direction. Note that the aggregated first-order terms can track the exact score shift (Appendix § 13.§ 2).

We first examine the layerwise contribution magnitude. As illustrated in Figure 4(c), both rule and evidence exhibit a localized middle-to-late concentration with a clear interior peak, indicating that substantial contribution is confined to a limited depth range. To understand the mechanism behind this peak, we further decompose the contribution magnitude into its two factors. As shown in Figure 4(b), ∣ΔBℓ,t∣\lvert\Delta B_{\ell,t}\rvert and ∣∇ψt∣\lvert\nabla\psi_t\rvert exhibit opposing trends: ∣ΔBℓ,t∣\lvert\Delta B_{\ell,t}\rvert increases with depth, whereas ∣∇ψt∣\lvert\nabla\psi_t\rvert decreases. Given these opposing trends, their product is expected to peak at an intermediate depth where the increase in ∣ΔBℓ,t∣\lvert\Delta B_{\ell,t}\rvert no longer outweighs the decrease in ∣∇ψt∣\lvert\nabla\psi_t\rvert. Consistent with this prediction, the resulting contribution exhibits the localized interior peak shown in Figure 4(c). In early layers, ∥ΔBℓ,t∥\lVert\Delta B_{\ell,t}\rVert remains close to zero, indicating that the role span has induced a limited change. The contribution therefore remains small despite high downstream sensitivity. Thus, a high contribution appears only when both factors remain sufficiently large.

Notably, magnitude alone is insufficient if the signed terms cancel after aggregation. We further compare targeted contribution ablation with a random control, which controls for generic intervention effects. As shown in Figure 4(d), targeted ablation produces a larger reduction in MfM_f for both roles, showing that disrupting the contribution organization markedly weakens the resulting preference shift. Taken together, these results show that role span contribution emerges when both magnitude factors remain sufficiently large, while disrupting their signed organization weakens the resulting preference shift.

Role Span Contribution at Key Token Decisions

Contribution localization. The above analysis suggests that role spans can shift predictions toward role-consistent responses through layerwise contributions. During autoregressive generation, this influence must be realized through token choices that affect whether the role’s requirements are satisfied. We therefore expect role span contributions to be particularly consequential at some key decisions. To examine how these decisions contribute to the distinct outcomes of ICL and OOC hallucination, we first measure the token-level first-order contribution δ^t\hat{\delta}_t at the corresponding output decisions. As shown in Table 2(a), these decisions consistently occur at positions with high role span influence. We then remove the role span pathway during free generation and record the first change in token selection. These first divergences likewise occur at positions with high role span influence. Thus, these findings suggest that role spans shape generation through key token choices.

Model(a) First-divergence localization(b) Retention of role-specified output content (%)
Rule spanEvidence spanRule spanEvidence span
GroundedHallucinatedFormatEntityGroundedHallucinated
Attr.Dec.Attr.Dec.Attr.Dec.CfC_fRem.CfC_fRem.CfC_fRem.CfC_fRem.
Qwen2.5-7B87.585.075.092.588.088.2100.04.857.118.1100.010.262.311.3
Mistral-7B82.585.787.577.589.890.1100.02.458.133.3100.016.761.57.7
Llama-3.1-8B80.090.085.095.092.491.9100.04.853.318.1100.010.061.513.5

Table 2. Role span effects on output decisions. (a) Role span influence at the first divergence and the output decision. (b) Retention rate of the content specified by CfC_f under the original (CfC_f) and pathway-removed (Rem.) conditions. Additional results are provided in Appendix § 13.§ 6.

Note. Attr. and Dec. report role span influence at the first divergence and output decision positions, respectively; higher values indicate stronger influence. Pathway removal changes token selection in 92–100% of generations. Hallucinated prefixes are taken from RAGTruth. Further details are in Appendix § 13.§ 6.

Role Effects. We next examine the effects of role span contributions at these key token decisions for the free generation. Specifically, we remove the role span pathway during decoding and assess whether the resulting response satisfies each role’s requirements. As shown in Table 2(b), removing the rule pathway disrupts the specified format and reduces retention of the selected entity. For evidence, pathway removal changes the facts stated in originally grounded responses. These results show that role span contributions causally support the key decisions that realize role-specific requirements. One might wonder whether the same pathway also affects factual decisions in real hallucinations. We test this using annotated LLM-generated hallucinations from RAGTruth [17], resuming generation from the prefixes preceding the hallucinated facts under the original and pathway-removed conditions. As shown in Table 2(b), the rate of selecting evidence-supported facts drops from 61.5–62.3% to 7.7–13.5% across models. This finding extends the role effects observed above to factual decisions in real hallucinated generation.

Together, these results reveal distinct decision-level effects of the two roles. Rule span contributions are typically realized through structural and selection decisions, whereas evidence span contributions affect factual content decisions, including continuations from genuine hallucination examples. By shaping different types of key token decisions, the two roles give rise to distinct generation outcomes in ICL and OOC hallucination.

Role Span Contributions to LLM Decision-Making

The preceding analyses establish how role span contributions are organized across layers and causally influence key token choices during free generation. Yet, even when role span contributions influence a decision, they may not be sufficient to make the model select the role-consistent token. We therefore examine when these contributions can sustain a successful choice or change an initially incorrect one. To this end, we first attenuate the signed contribution at initially successful decisions (§ 5.3.1) and then restore the contribution at naturally failing ones (§ 5.3.2).

Dependence of Successful Decisions

We first examine successful decisions in the rule-following and evidence-grounding tasks derived above. In both settings, we analyze the model’s autoregressive generations and identify the key decisions at which the corresponding role requirements are satisfied. To examine whether role span contributions sustain these choices, we measure the advantage of the role-consistent token over its strongest competitor using the target-token margin, with zero marking the decision boundary. We then attenuate the role span contribution by a factor of 1−α1-\alpha and track changes in this margin and the resulting token choice.

As shown in Table 3(a), increasing α\alpha generally narrows the margin under both roles. For rule decisions, the median margin shifts from positive to negative across all models, leaving the rule-specified token selected in only a minority of these key choices. Meanwhile, attenuation substantially erodes the advantage of the evidence-supported token, bringing the corresponding factual choice closer to the decision boundary. In contrast, the non-role comparison shows smaller margin changes, showing that the pronounced changes are driven by role span contributions. Under full attenuation, these margin reductions are accompanied by frequent token flips and performance declines (Table 3(b).)

Modelα=0\alpha=0α=0.25\alpha=0.25α=0.5\alpha=0.5α=0.75\alpha=0.75α=1\alpha=1α=0\alpha=0α=1\alpha=1Flip (%)Task (%)Flip (%)Task (%)
(a) Target token margin(b) Token and task outcomes at α=1\alpha=1
Targeted attenuationNon-role comparisonTargeted attenuationNon-role comparison
Modelα=0\alpha=0α=0.25\alpha=0.25α=0.5\alpha=0.5α=0.75\alpha=0.75α=1\alpha=1α=0\alpha=0α=1\alpha=1Flip (%)Task (%)Flip (%)Task (%)
Rule span
Qwen2.5-7B4.625.00-0.38-12.89-16.254.625.0084.022.317.0100.0
Mistral-7B6.696.062.09-4.16-5.036.695.2577.532.432.494.1
Llama-3.1-8B4.693.252.06-1.94-4.064.692.3876.528.625.586.7
Evidence span, task-defined
Qwen2.5-7B8.237.905.63-0.57-1.098.237.6057.425.99.374.1
Mistral-7B10.009.946.722.561.2510.009.8844.242.30.098.1
Llama-3.1-8B8.627.624.500.50-0.698.627.7554.937.32.098.0
Evidence span, response-derived
Qwen2.5-7B14.7014.047.881.660.4614.7012.5450.045.80.0100.0
Mistral-7B15.9115.5412.620.960.9815.9115.1943.530.40.0100.0
Llama-3.1-8B5.064.842.01-4.62-9.285.065.2387.512.54.295.8

Table 3. Sustained attenuation of role span contributions. The contribution is scaled by (1−α)(1-\alpha), with α=1\alpha=1 denoting complete attenuation. (a) Median target token margin under targeted attenuation and non-role comparison. (b) Token flips and task performance under full attenuation.

To rule out dependence on predefined factual decisions, we further apply the same sustained attenuation to facts correctly generated by the models themselves. As shown in the response-derived rows of Table 3, targeted attenuation substantially reduces the decision margin and frequently changes the selected token and generated factual value, whereas attenuation of non-evidence spans largely preserves the original factual content. These results show that the presence of role information does not guarantee effective support during generation. The contribution must remain sufficiently large when the model makes key choices. For rules, this support must be sustained across repeated structural and selection decisions, whereas for evidence, it is required when the factual value is determined.

Recoverability of Failed Decisions

We have focused on initially successful decisions, where attenuation shows that the local choice depends on sustained role span support. We next examine whether amplifying the same contributions can restore decisions that initially fail. To test the causal effect, we amplify the same contribution at naturally failing decisions and track the task performance.

Considering task-defined cases, we find that amplifying the evidence span contribution restores a substantial subset of naturally failing factual decisions. As shown in Table 4, the restoration rate reaches 43.7% at β=2\beta=2, accompanied by improved task performance. In contrast, rule failures exhibit much larger initial decision deficits, and their target tokens remain unrecovered even under 16×16 \times amplification. These results suggest that whether contribution amplification can reverse a failed decision depends on the initial decision deficit.

ModelBaseline marginMargin changeSelection (%)Task (%)Metricβ=0\beta=0β=0.5\beta=0.5β=1\beta=1β=2\beta=2β=4\beta=4β=8\beta=8β=16\beta=16
(a) Restoration at β=1\beta=1(b) Restoration across weights
Rule span
Qwen2.5-7B-14.6-0.610.00.0Selection0.00.00.00.00.00.00.0
Mistral-7B-19.0+0.060.04.8Margin-14.6-14.6-14.5-14.1-13.0-12.8-11.7
Llama-3.1-8B-11.3+0.000.00.0Task0.00.01.32.56.33.86.3
Evidence span, task-defined
Qwen2.5-7B-4.0+0.8929.534.1Selection0.022.328.243.741.739.838.8
Mistral-7B-2.2+0.2816.720.8Margin-3.0-2.9-2.0-1.2-0.6-1.9-2.8
Llama-3.1-8B-2.7+1.6934.337.1Task0.025.232.047.653.455.358.3
Evidence span, real-hallucinated
Qwen2.5-7B-6.0+1.1811.144.4Selection0.08.810.017.532.533.832.5
Mistral-7B-5.3+1.2011.542.3random0.01.25.06.25.00.02.5
Llama-3.1-8B-4.5+0.967.437.0Task28.837.541.243.857.557.561.2
All models-5.4+1.1210.041.2random28.827.527.528.825.015.025.0

Table 4. Restoration at failing decisions. We restore the contribution vector at the decision, scaled by a weight β\beta. (a) Restoring the contribution at β=1\beta=1. (b) Restoration across multiple weights.

We further examine whether contribution amplification can restore factual decisions in genuine hallucinations from RAGTruth. As shown in the real-hallucinated rows of Table 4, these decisions exhibit larger initial deficits than task-defined evidence decisions. Nevertheless, amplification shifts the decision margin toward the evidence-supported token and improves factual selection. At β=4\beta= 4, target selection reaches 32.5%, with 57.5% of continuations stating the evidence-supported fact, both outperforming random-direction interventions. These results show that evidence span contributions remain effective in genuine hallucinations, although their ability to reverse factual decisions still depends on the decision deficit.

Consequently, these results show that the same local contribution mechanism can extend to naturally failing decisions, including genuine hallucination examples. Whether amplification reverses the choice depends on the balance between the support of role contribution and the decision deficit. This relation further yields a quantitative prediction for failed decisions, which we formalize in Appendix § 11.§ 4 and evaluate in Appendix § 13.7.

Table 4: Restoration at failing decisions. We restore the contribution vector at the decision, scaled by a weight β\beta. (a) Restoring the contribution at β=1\beta=1. (b) Restoration across multiple weights.

Note. Margins follow the definition in Table 3, and Δmt=mt(β=1)−mt(β=0)\Delta m_t = m_t(\beta=1) - m_t(\beta=0); positive values indicate a shift toward the target token. Real-hallucinated decisions are taken from RAGTruth; their Task is the share of continuations stating the grounded value at the hallucinated slot. random rows replace the contribution vector with a random direction of equal norm.

In summary, we show that dynamic context guides LLM generation through functional role span contributions in both ICL and OOC hallucination. Specifically, rule contributions shape structural and selection choices, whereas evidence contributions shape factual choices. Moreover, effective role influence depends on coherent contribution organization across layers and sufficient support when the key choice is made. The causal interventions further show that a failed choice can be reversed when the corresponding support is sufficient to compensate for the decision deficit.

Conclusion

In this paper, we offer a common mechanistic view of the distinct outcomes of ICL and OOC hallucination. This view treats the functional roles of dynamic context as the organizing principle for model behavior. We connect internal role representations to their effects on decoding through role span contributions. These contributions causally shape key token choices that determine response structure, content selection, and factual value. Each functional role determines where these contributions must remain effective during generation. We derive a first-order decomposition of role span contributions and identify two necessary conditions for reliable influence. At the decision level, rule following requires sustained support across repeated structural and selection choices, whereas evidence grounding relies on support at a smaller set of decisive factual choices. The contrasting outcomes therefore follow from distinct requirements for sustaining role support across generation.

AI USE STATEMENT

In this work, we did not use generative AI tools for tasks requiring mandatory disclosure under the ICLR 2027 AI Policy. Additionally, we used generative AI tools to create or edit software code and to edit the research paper to improve readability. We have reviewed all AI-assisted work. AI-assisted code was verified and tested by the authors, and AI-assisted edits to the manuscript were reviewed by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

REPRODUCIBILITY STATEMENT

The source code used for our experiments is provided in the supplementary materials and will be released in a public repository. Appendix B provides the complete proofs and derivations of the theoretical results. Appendix C provides the experimental details, including the experimental setting, matched role construction, representation and recovery analyses, and contribution and preference analysis. Appendix D reports additional experimental results.

Appendix

Related Work

Context-conditioned generation. Motivated by the contextual adaptability of LLMs, recent work has extensively studied how information supplied at inference time shapes model behavior. In-context learning provides a prominent setting where models can infer a task from demonstrations and instructions without task-specific training [10, 29]. Recent studies further show that this adaptation is sensitive to demonstration selection, ordering, and composition [15, 29, 18, 25]. Meanwhile, a separate line of work has investigated hallucination in LLM generation, where model outputs contain factually incorrect or unsupported information [13, 31]. Retrieval-augmented generation has become a common approach to mitigating hallucination by incorporating external evidence into generation [30, 2]. Nevertheless, LLMs may still produce claims that are unsupported by or contradictory to the retrieved content [17, 26]. Such departures from the supplied context define the out-of-context (OOC) hallucination setting and expose a gap in understanding contextual faithfulness [2, 12].

Mechanistic analyses of inference-time behavior. Recent mechanistic work extends the study of inference-time behavior from observable outcomes to internal representations. In ICL, some studies identify attention circuits that support in-context computation [19], while other work seeks to explicitly characterize in-context task information by recovering task vectors from hidden states [11, 23]. In parallel, mechanistic studies of hallucination associate incorrect or unsupported generation with failures in knowledge retrieval and transfer and with conflicts between parametric knowledge and external evidence [31, 28]. Other studies further localize components involved in retrieving and using contextual information during generation, showing that evidence use can likewise be traced to internal components [27, 16]. Despite this progress, the two behaviors are predominantly examined within specific tasks or failure settings. Across both lines, a common concern is which internal representations of context become consequential for generation.

Attributing contextual influence. To establish this relevance, existing methods connect model predictions to their internal sources. For instance, input attribution and score decomposition quantify how tokens or activations support a prediction [22], while circuit analysis traces it to interpretable computational paths [9]. However, such local analyses rarely make explicit the function of context in the current task [31]. Despite comparable mechanistic accounts, ICL and OOC hallucination exhibit distinct task-level behaviors. To this end, we analyze the distinct functional roles in the two settings and identify the mechanistic conditions underlying task success and failure from a unified perspective.

Proofs and Derivations

Attention-Mediated Role Span Transport

We first formalize why a role-induced difference can enter a new token position only through attention.

Lemma 1 (Position locality). In a decoder-only Transformer, normalization and MLP modules act independently at each token position, whereas the attention update at position ii aggregates states from the source positions visible to ii.

Proof. Normalization and MLP modules apply the same map independently at each position. By contrast, Eq. (2) writes the attention update at ii as a sum over its visible source positions. Position-wise transformations of queries and keys do not alter this distinction.

Fix a decoding step tt and write P=Ptf(y)P = P_t^f(y), P~=P~tf~(y)\widetilde{P} = \widetilde{P}_t^{\widetilde{f}}(y), and SfS_f for their aligned role span positions. Define

dℓ(i)=hℓ(P,i)−hℓ(P~,i),Cℓ={s:dℓ(s)≠0},d_\ell(i) = h_\ell(P,i) - h_\ell(\widetilde{P},i), \qquad C_\ell= \{s : d_\ell(s) \ne0\},

and let V(i)\mathcal{V}(i) denote the source positions visible to ii. For head aa and source position s∈V(i)s \in\mathcal{V}(i), write

gℓ,i,s(a)(P)=αi←s(a)(P)Vℓ(a)(P,s)WO(a),Δgℓ,i,s(a)=gℓ,i,s(a)(P)−gℓ,i,s(a)(P~).g_{\ell,i,s}^{(a)}(P) = \alpha_{i \leftarrow s}^{(a)}(P)V_\ell^{(a)}(P,s)W_O^{(a)}, \qquad\Delta g_{\ell,i,s}^{(a)} = g_{\ell,i,s}^{(a)}(P) - g_{\ell,i,s}^{(a)}(\widetilde{P}).

We decompose the source-indexed differences as

ΔBℓ,if=∑a∑s∈V(i)∩SfΔgℓ,i,s(a).\Delta B_{\ell,i}^{f} = \sum_a \sum_{s \in\mathcal{V}(i) \cap S_f} \Delta g_{\ell,i,s}^{(a)}.
ΔIℓ,i=∑a∑s∈V(i)∩(Cℓ−1∖Sf)Δgℓ,i,s(a).\Delta I_{\ell,i} = \sum_a \sum_{s \in\mathcal{V}(i) \cap(C_{\ell-1} \setminus S_f)} \Delta g_{\ell,i,s}^{(a)}.
ΔWℓ,i=∑a∑s∈V(i)∖(Sf∪Cℓ−1)Δgℓ,i,s(a).\Delta W_{\ell,i} = \sum_a \sum_{s \in\mathcal{V}(i) \setminus(S_f \cup C_{\ell-1})} \Delta g_{\ell,i,s}^{(a)}.

Proposition 1 (Attention-mediated role span transport). Suppose that PP and P~\widetilde{P} differ only at the aligned role span positions SfS_f. For every i∉Sfi \notin S_f, d0(i)=0d_0(i) = 0, and at each layer ℓ≥1\ell\ge1,

ΔAttn⁡ℓ(i)=ΔBℓ,if+ΔIℓ,i+ΔWℓ,i,(9)\Delta\operatorname{Attn}_\ell(i) = \Delta B_{\ell,i}^{f} + \Delta I_{\ell,i} + \Delta W_{\ell,i}, \tag*{(9)}

where

ΔAttn⁡ℓ(i)=Attn⁡ℓout(P,i)−Attn⁡ℓout(P~,i).\Delta\operatorname{Attn}_\ell(i) = \operatorname{Attn}_\ell^{\mathrm{out}}(P,i) - \operatorname{Attn}_\ell^{\mathrm{out}}(\widetilde{P},i).

Moreover,

dℓ−1(i)=0andΔAttn⁡ℓ(i)=0⟹dℓ(i)=0.d_{\ell-1}(i) = 0 \quad\text{and} \quad\Delta\operatorname{Attn}_\ell(i) = 0 \quad\Longrightarrow\quad d_\ell(i) = 0.

Consequently, if ℓ⋆=min⁡{ℓ:dℓ(i)≠0}\ell^\star= \min\{\ell: d_\ell(i) \ne0\} exists, then ΔAttn⁡ℓ⋆(i)≠0\Delta\operatorname{Attn}_{\ell^\star}(i) \ne0.

Proof. Since the matched inputs differ only at SfS_f, their embedding representations coincide at every i∉Sfi \notin S_f. Hence, d0(i)=0d_0(i) = 0.

We next decompose the attention difference. By Eq. (2),

ΔAttn⁡ℓ(i)=∑a∑s∈V(i)Δgℓ,i,s(a).\Delta\operatorname{Attn}_\ell(i) = \sum_a \sum_{s \in\mathcal{V}(i)} \Delta g_{\ell,i,s}^{(a)}.

The visible source positions admit the disjoint partition

V(i)=(V(i)∩Sf)∪(V(i)∩(Cℓ−1∖Sf))∪(V(i)∖(Sf∪Cℓ−1)).\mathcal{V}(i) = \bigl(\mathcal{V}(i) \cap S_f\bigr) \cup\bigl(\mathcal{V}(i) \cap(C_{\ell-1} \setminus S_f)\bigr) \cup\bigl(\mathcal{V}(i) \setminus(S_f \cup C_{\ell-1})\bigr).

Summing over these three sets gives Eq. (9). The first term is the direct role span contribution defined in Eq. (3); the second reads from non-role positions whose residual states already differ.

For a source position in the third set, dℓ−1(s)=0d_{\ell-1}(s) = 0, and hence its value representation is identical under the two computations:

Vℓ(a)(P,s)=Vℓ(a)(P~,s).V_\ell^{(a)}(P,s) = V_\ell^{(a)}(\widetilde{P},s).

Thus, ΔWℓ,i\Delta W_{\ell,i} contains the remaining changes in attention weights over otherwise unchanged source values. This definition covers any such reweighting without assigning it to a particular query, key, or softmax component.

We next show how the difference propagates through the block. If dℓ−1(i)=0d_{\ell-1}(i) = 0 and ΔAttn⁡ℓ(i)=0\Delta\operatorname{Attn}_\ell(i) = 0, the two computations provide the same local state and the same attention update at position ii. By Lemma 1, all remaining operations at that position therefore receive identical inputs, which gives dℓ(i)=0d_\ell(i) = 0.

Now let ℓ⋆\ell^\star be the first layer such that dℓ⋆(i)≠0d_{\ell^\star}(i) \ne0. Then dℓ⋆−1(i)=0d_{\ell^\star-1}(i) = 0. If the attention updates also agreed at layer ℓ⋆\ell^\star, the previous implication would yield dℓ⋆(i)=0d_{\ell^\star}(i) = 0, a contradiction. Therefore, ΔAttn⁡ℓ⋆(i)≠0\Delta\operatorname{Attn}_{\ell^\star}(i) \ne0.

Proposition 1 establishes that attention is the direct channel through which a role-induced difference enters a new token position. Among the resulting terms, ΔBℓ,if\Delta B^{f}_{\ell,i} is the direct source-indexed role span pathway used in § 4.3, while ΔIℓ,i\Delta I_{\ell,i} and ΔWℓ,i\Delta W_{\ell,i} are treated in the remainder analysis of Appendix § 11.§ 3.

First-Order Role Span Score Decomposition

We now make the coordinate dependence in Assumption 1 explicit. Fix a decoding step tt and write

b~=(Bℓ,if(P~tf(y)))(ℓ,i)∈Bf,t,bf=(Bℓ,if(Ptf(y)))(ℓ,i)∈Bf,t,\widetilde{b}=\left(B^{f}_{\ell,i}\left(\widetilde{P}^{f}_{t}(y)\right)\right)_{(\ell,i)\in\mathcal{B}_{f,t}}, \qquad b^{f}=\left(B^{f}_{\ell,i}\left(P^{f}_{t}(y)\right)\right)_{(\ell,i)\in\mathcal{B}_{f,t}},

with

Δb=bf−b~=ΔBB,tf(y).\Delta b=b^{f}-\widetilde{b}=\Delta B^{f}_{B,t}(y).

We equip this stacked coordinate space with the Euclidean inner product and norm.

Substitution map. For b=(bℓ,i)(ℓ,i)∈Bf,tb=(b_{\ell,i})_{(\ell,i)\in\mathcal{B}_{f,t}}, let Ψt(b)\Psi_t(b) denote the score obtained from P~tf(y)\widetilde{P}^{f}_{t}(y) by processing the model in forward order and replacing each selected role span attention contribution with bℓ,ib_{\ell,i} before it enters the residual stream. Specifically, the term recomputed from the current modified state is removed and replaced by bℓ,ib_{\ell,i}. All replacements within a layer are applied before advancing to the next layer, and the remaining quantities are computed from the resulting states.

Lemma 2 (Exact substitution identity). The substitution map is well defined and satisfies

Ψt(b~)=ψt(P~tf(y)).\Psi_t(\widetilde{b})=\psi_t\left(\widetilde{P}^{f}_{t}(y)\right).

Consequently, with

Et⊥=ψt(Ptf(y))−Ψt(bf),E^{\perp}_{t}=\psi_t\left(P^{f}_{t}(y)\right)-\Psi_t\left(b^{f}\right),

the stepwise score shift admits the exact identity

δtf(y)=Ψt(bf)−Ψt(b~)+Et⊥.(10)\delta^{f}_{t}(y)=\Psi_t\left(b^{f}\right)-\Psi_t(\widetilde{b})+E^{\perp}_{t}. \tag*{(10)}

Proof. The forward ordering uniquely determines every modified residual state, so Ψt(b)\Psi_t(b) is well defined for any collection bb.

We next establish the endpoint identity. Set b=b~b=\widetilde{b}. At the input layer, the modified computation and the computation under P~tf(y)\widetilde{P}^{f}_{t}(y) have identical states. Suppose that their states agree through layer ℓ−1\ell-1. Their unmodified attention terms at layer ℓ\ell are then identical, and each selected role span term is replaced by its value Bℓ,if(P~tf(y))B^{f}_{\ell,i}\left(\widetilde{P}^{f}_{t}(y)\right). Hence, the complete attention updates and the resulting residual states also agree at layer ℓ\ell. Induction over the layers gives Ψt(b~)=ψt(P~tf(y))\Psi_t(\widetilde{b})=\psi_t\left(\widetilde{P}^{f}_{t}(y)\right).

Finally, adding and subtracting Ψt(bf)\Psi_t\left(b^{f}\right) in the definition of δtf(y)\delta^{f}_{t}(y) gives

δtf(y)=ψt(Ptf(y))−ψt(P~tf(y))=Ψt(bf)−Ψt(b~)+[ψt(Ptf(y))−Ψt(bf)],\begin{aligned} \delta^{f}_{t}(y)=\psi_t\left(P^{f}_{t}(y)\right)-\psi_t\left(\widetilde{P}^{f}_{t}(y)\right) \\ &=\Psi_t\left(b^{f}\right)-\Psi_t(\widetilde{b})+\left[\psi_t\left(P^{f}_{t}(y)\right)-\Psi_t\left(b^{f}\right)\right], \end{aligned}

which is (10).

Proof of Theorem 1. By Lemma 2, it remains to expand Ψt(bf)−Ψt(b~)\Psi_t\left(b^{f}\right)-\Psi_t(\widetilde{b}). Let

γ(τ)=b~+τΔb,τ∈[0,1].\gamma(\tau)=\widetilde{b}+\tau\Delta b,\qquad\tau\in[0,1].

Then we have

Ψt(bf)−Ψt(b~)=∫01⟨∇Ψt(γ(τ)),Δb⟩ dτ.\Psi_t\left(b^{f}\right)-\Psi_t(\widetilde{b})=\int_{0}^{1}\left\langle\nabla\Psi_t(\gamma(\tau)),\Delta b\right\rangle\,d\tau.

Adding and subtracting ∇Ψt(b~)\nabla\Psi_t(\widetilde{b}) gives

Ψt(bf)−Ψt(b~)=⟨∇Ψt(b~),Δb⟩+rtcurv,(11)\Psi_t\left(b^{f}\right)-\Psi_t(\widetilde{b})=\left\langle\nabla\Psi_t(\widetilde{b}),\Delta b\right\rangle+r^{\mathrm{curv}}_{t}, \tag*{(11)}

where

rtcurv=∫01⟨∇Ψt(γ(τ))−∇Ψt(b~),Δb⟩ dτ.r_t^{\mathrm{curv}}=\int_0^1\left\langle\nabla\Psi_t(\gamma(\tau))-\nabla\Psi_t(\tilde b),\Delta b\right\rangle\,d\tau.

Since Ψt\Psi_t has an LtL_t-Lipschitz gradient on the segment between b~\tilde b and bfb^f,

∣rtcurv∣≤∫01∥∇Ψt(γ(τ))−∇Ψt(b~)∥∥Δb∥ dτ≤∫01Ltτ∥Δb∥2 dτ=Lt2∥Δb∥2.\begin{aligned} \left|r_t^{\mathrm{curv}}\right|\le\int_0^1\left\|\nabla\Psi_t(\gamma(\tau))-\nabla\Psi_t(\tilde b)\right\|\left\|\Delta b\right\|\,d\tau\\ &\le\int_0^1L_t\tau\left\|\Delta b\right\|^2\,d\tau=\frac{L_t}{2}\left\|\Delta b\right\|^2. \end{aligned}

Moreover, the stacked inner product decomposes as

⟨∇Ψt(b~),Δb⟩=∑(ℓ,i)∈Bf,t⟨∇bℓ,iΨt(b~),ΔBℓ,i,tf(y)⟩.\left\langle\nabla\Psi_t(\tilde b),\Delta b\right\rangle=\sum_{(\ell,i)\in\mathcal{B}_{f,t}}\left\langle\nabla_{b_{\ell,i}}\Psi_t(\tilde b),\Delta B^f_{\ell,i,t}(y)\right\rangle.

Combining this equality with (10) yields (4) with

Rt=rtcurv+Et⊥.R_t=r_t^{\mathrm{curv}}+E_t^{\perp}.

Finally,

∣Rt∣≤∣rtcurv∣+∣Et⊥∣≤Lt2∥ΔBB,tf(y)∥2+∣Et⊥∣,|R_t|\le\left|r_t^{\mathrm{curv}}\right|+\left|E_t^{\perp}\right|\le\frac{L_t}{2}\left\|\Delta B^f_{\mathcal{B},t}(y)\right\|^2+\left|E_t^{\perp}\right|,

which gives (5).

The term rtcurvr_t^{\mathrm{curv}} is the Taylor remainder within the selected coordinate space, whereas Et⊥E_t^{\perp} contains the score change not reproduced by these coordinates. We characterize the latter terms in Appendix B.3.

Remainder Characterization

By Theorem 1, the remainder separates as Rt=rtcurv+Et⊥R_t=r_t^{\mathrm{curv}}+E_t^{\perp}, where ∣Rt∣≤Lt2∥ΔBB,tf(y)∥2+∣Et⊥∣|R_t|\le\frac{L_t}{2}\left\|\Delta B^f_{\mathcal{B},t}(y)\right\|^2+|E_t^{\perp}|.

Here, rtcurvr_t^{\mathrm{curv}} is the curvature error introduced by linearizing Ψt\Psi_t over the selected coordinates, whereas Et⊥=ψt(Ptf(y))−Ψt(bf)E_t^{\perp}=\psi_t(P_t^f(y))-\Psi_t(b^f) measures the score change not reproduced by these substitutions. Relative to the decomposition in Appendix B.1, the substitution gap may include omitted direct coordinates and portions of ΔIℓ,i\Delta I_{\ell,i} and ΔWℓ,i\Delta W_{\ell,i} that are not recovered by the modified computation.

Notably, this characterization does not assume that RtR_t is small. In Appendix D.3, we compare the direct role span attention pathway with alternative transport terms and MLP updates, and show that attention provides the dominant route for role information to reach the relevant token decisions.

Decision-Level Recovery Prediction

We finally relate the role span contribution to the local decision boundary considered in Section 5.3. For a naturally failing decision, let yt∗y_t^* denote the target token and let

ct=arg⁡max⁡v≠yt∗ log⁡pθ(v∣Pt;β=0)c_t=\underset{v\ne y_t^*}{\arg\max}\ \log p_\theta(v\mid P_t;\beta=0)

be its strongest competitor before amplification. Following Section 5.3.2, let β\beta parameterize amplification of the same role span contribution Bℓ,if(P)B^f_{\ell,i}(P) at the decision position, with β=0\beta=0 denoting the original decision. For the fixed competitor ctc_t, write

mt(β;ct)=log⁡pθ(yt∗∣Pt;β)−log⁡pθ(ct∣Pt;β),m_t(\beta;c_t)=\log p_\theta(y_t^*\mid P_t;\beta)-\log p_\theta(c_t\mid P_t;\beta),

so that mt(0;ct)=mt(0)<0m_t(0;c_t)=m_t(0)<0. We define the signed support supplied along the amplification direction as

St=ddβmt(β;ct)∣β=0.S_t=\left.\frac{d}{d\beta}m_t(\beta;c_t)\right|_{\beta=0}.

Proposition 2 (Decision-boundary crossing). Suppose that the derivative of mt(β;ct)m_t(\beta; c_t) is LL-Lipschitz along the local amplification path. Then

∣mt(β;ct)−mt(0)−βSt∣≤L2β2.(12)\left|m_t(\beta; c_t)-m_t(0)-\beta S_t\right| \le\frac{L}{2}\beta^2. \tag*{(12)}

For mt(0)<0m_t(0)<0 and St>0S_t>0, the corresponding first-order prediction of the amplification required to reach the decision boundary is

β^t=−mt(0)St.(13)\hat{\beta}_t=-\frac{m_t(0)}{S_t}. \tag*{(13)}

Moreover, if ctc_t remains the strongest non-target token along the local path, then any β>0\beta>0 satisfying

βSt>−mt(0)+L2β2(14)\beta S_t>-m_t(0)+\frac{L}{2}\beta^2 \tag*{(14)}

guarantees mt(β)>0m_t(\beta)>0.

Proof. A first-order expansion of the fixed-competitor margin at β=0\beta=0 gives

mt(β;ct)=mt(0)+βSt+εt(β),m_t(\beta;c_t)=m_t(0)+\beta S_t+\varepsilon_t(\beta),

where the Lipschitz condition yields ∣εt(β)∣≤Lβ2/2|\varepsilon_t(\beta)|\le L\beta^2/2. This proves Eq. (11).

Ignoring the quadratic term, the local boundary is reached when mt(0)+βSt=0m_t(0)+\beta S_t=0. For St>0S_t>0, solving this equality gives Eq. (13). Thus, a larger initial deficit −mt(0)-m_t(0) requires stronger amplification, whereas larger positive signed support StS_t reduces the required strength.

For the finite-β\beta statement, Eq. (11) gives the lower bound

mt(β;ct)≥mt(0)+βSt−L2β2.m_t(\beta;c_t)\ge m_t(0)+\beta S_t-\frac{L}{2}\beta^2.

Hence, Eq. (14) implies mt(β;ct)>0m_t(\beta;c_t)>0. If ctc_t remains the strongest competitor, this pairwise margin equals the target-token margin, and therefore mt(β)>0m_t(\beta)>0.

Consequently, the first-order decision formulation estimates the amplification required to recover a failed decision. Appendix § 13.7 further examines the extension of this prediction to task-level recovery.

Experimental Details

Experimental setting. For the rule role, we derive a controlled benchmark from the task patterns in CL-bench Dou et al. [7], covering format, selection, and include or omit instructions. For the evidence role, we use the training split of RAGTruth Niu et al. [17]. We evaluate Qwen2.5-7B, Mistral-7B, and Llama-3.1-8B. Free generations use greedy decoding, whereas fixed response trajectories are evaluated with teacher forcing.

Matched Role Construction

To isolate functional role ff, we compare two prompts that share the query and all non-role content but differ in the role-bearing span. The matched counterfactual changes only this span while preserving the remainder of the prompt. Figure 1 illustrates the matched construction. To ensure alignment, we locate the edited span separately in each tokenized prompt and, for a fixed response trajectory yy, use the same response prefix y<ty_{<t} under both conditions, with comparisons performed at the same decoding step.

Representation and Recovery Analyses

Subspace concentration. We extract Δhℓf\Delta h_\ell^f at the final prompt position from every decoder layer. For each model and role, we fit a rank-10 subspace on one half of the samples and evaluate it on the other, then exchange the two splits. We use the matched role comparisons defined in Appendix § 12.§ 1.

Specifically, the held-out concentration is computed as Eℓ,if=∥ΠUℓΔhℓf(i)∥2∥Δhℓf(i)∥2\mathcal{E}^{f}_{\ell,i} = \frac{\lVert\Pi_{U_\ell}\Delta h^{f}_{\ell}(i)\rVert^{2}}{\lVert\Delta h^{f}_{\ell}(i)\rVert^{2}}, where ΠUℓ\Pi_{U_\ell} projects onto the fitted rank-10 subspace UℓU_\ell. The random-subspace baseline is Erand=10/H\mathcal{E}_{\mathrm{rand}} = 10/H, where HH is the hidden state dimension.

Output alignment and behavior recovery. We assess output alignment by applying the final RMS normalization to the two residual states in each matched pair and then projecting their difference onto the unembedding direction of the token selected under CfC_f. We report the resulting logit shift and cosine alignment, with a matched non-role perturbation as the control. Separately, we evaluate behavior recovery by adding Δhℓf\Delta h^{f}_{\ell} to the counterfactual state at the final prompt position and recording whether the next-token choice returns to that under CfC_f.

Contribution and Preference Analysis

Candidate comparisons. Each preference analysis combines a matched context pair with a minimal candidate pair. Both candidates are scored with teacher forcing, and MfM_f measures the resulting change in relative support. fig:5 illustrates the design.

Matched contextsMatched candidates
RuleCrC_r: JSON-output rule<br>C~r\widetilde{C}_r: Markdown-list ruley+y^{+}: {”name”: ”Alpha”, ”value”: ”120”}<br>y−y^{-}: {”label”: ”Alpha”, ”value”: ”120”}
EvidenceCeC_e: Evidence supporting 2024<br>C~e\widetilde{C}_e: Matched context, e.g., evidence supporting 2029y+y^{+}: The value is 2024<br>y−y^{-}: The value is 2029

Table 5.

Φf(y)=log⁡pθ(y∣Cf⊕x)−log⁡pθ(y∣C~f⊕x),Mf=Φf(y+)−Φf(y−).\Phi_f(y) = \log p_\theta(y \mid C_f \oplus x) - \log p_\theta(y \mid\widetilde{C}_f \oplus x), \qquad M_f = \Phi_f(y^{+}) - \Phi_f(y^{-}).

fig:5: Matched preference construction. Each context pair differs only in the role-bearing span, while each candidate pair differs only in the choice specified by that role.

Additional Experiments

Additional Model Results for Contribution Organization

Figures 6 and 7 report the corresponding analyses for Mistral-7B and Llama-3.1-8B, both of which exhibit the same overall pattern. Contributions remain concentrated within a limited range of layers, the two magnitude factors vary in opposite directions across depth, and role span pathway ablation reduces MfM_f more than random ablation. The contribution organization identified in the main analysis is therefore consistent across models.

Organization of contributions for Mistral-7B

Figure 6. Organization of contributions for Mistral-7B. (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of ∣ΔBℓ,t∣|\Delta B_{\ell,t}| and ∣∇ψt∣|\nabla\psi_t|; and (c) preference margin MfM_f under the baseline, role span pathway ablation, and random ablation.

Organization of contributions for Llama-3.1-8B

Figure 7. Organization of contributions for Llama-3.1-8B. (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of ∣ΔBℓ,t∣|\Delta B_{\ell,t}| and ∣∇ψt∣|\nabla\psi_t|; and (c) preference margin MfM_f under the baseline, role span pathway ablation, and random ablation.

ModelRule span Corr.Rule span Sign (%)Evidence span Corr.Evidence span Sign (%)
Qwen2.5-7B0.9377.10.8695.0
Mistral-7B0.9487.60.8897.0
Llama-3.1-8B0.9291.40.8594.0

Table 5 (PDF p. 20). Correspondence between first-order contributions and exact score shifts.

Note. Corr. is the median Spearman correlation between the magnitudes of the summed first-order contributions and the exact stepwise score shifts. Sign is the fraction of responses whose accumulated contribution matches the sign of the exact trajectory shift.

First-Order Correspondence

Here we examine how the first-order contributions correspond to the exact role effect. Within each response, the magnitudes of the summed first-order contributions track the exact stepwise score shifts, with median Spearman correlations of 0.85–0.94 across models. The sign of the accumulated contribution also matches the exact trajectory shift for most responses.

Role Span Attention Pathway

Appendix B.1 establishes that attention is the direct channel when a role-induced difference enters a new token position. To examine how much of the realized role effect this pathway carries, we study its recoverability across attention terms and its decision effects. Specifically, for each matched context pair, we fix the exact trajectory score effect Φf\Phi_f, and substitute one term at a time. We consider the direct role term ΔB\Delta B, attention reweighting ΔW\Delta W, and a matched non-role term.

Table 6 shows that the direct term recovers 73.5%–93.9% of the exact role effect across models. In contrast, recovery from attention reweighting and matched non-role positions remains near zero. Together with the sustained attenuation results in § 5.3.1, these findings show that the direct role span attention term carries a substantial and behaviorally consequential part of the role effect.

Since it is essential for the role effect to become available at the decision, we next attenuate the carried state at three depths. Table 7(a) shows that attenuation at 0.25LL has little effect in any setting. By 0.50LL, the dependence begins to emerge, and at 0.75LL, the margin drop is positive for all settings. Thus, the role effect becomes relevant to the decision only after sufficient depth. Table 7(b) shows that attenuating either update family reduces the target margin. Across comparisons, the attention intervention produces a larger effect than the MLP intervention.

(a) Carry drop by depth(b) Update families
Modelℓ/L=0.25\ell/L=0.250.50.75AttentionMLP
Rule
Qwen2.5-7B−0.01-0.01 [−0.13-0.13, 0.12]−0.22-0.22 [−0.64-0.64, 0.14]11.25 [7.18, 16.11]30.79 [24.51, 37.68]26.14 [21.55, 31.06]
Mistral-7B0.05 [−0.01-0.01, 0.11]0.77 [0.15, 1.42]12.56 [9.76, 15.32]15.64 [12.20, 19.21]13.40 [10.84, 15.97]
Llama-3.1-8B−0.04-0.04 [−0.10-0.10, 0.03]3.25 [2.10, 4.55]9.11 [8.20, 10.00]13.74 [11.59, 15.96]8.77 [7.71, 9.68]
Evidence
Qwen2.5-7B0.02 [−0.24-0.24, 0.28]0.23 [−0.11-0.11, 0.61]4.25 [3.05, 5.59]8.76 [6.76, 11.01]7.18 [5.43, 9.02]
Mistral-7B−0.04-0.04 [−0.17-0.17, 0.07]1.10 [0.68, 1.53]5.00 [3.83, 6.19]8.70 [6.85, 10.83]7.81 [6.37, 9.27]
Llama-3.1-8B0.20 [−0.06-0.06, 0.41]2.61 [1.82, 3.46]4.68 [3.61, 5.76]7.90 [6.22, 9.70]4.01 [2.62, 5.54]

Table 7. Accumulated carry and update-family effects at role-sensitive decisions. (a) The carried state entering the layer at three fractional depths. (b) Each update family at all layers. Note. Drops are computed as matched control minus targeted intervention, so larger values indicate stronger dependence.

Taken together, these results establish the empirical importance of the role span attention pathway. The direct attention term recovers a substantial part of the role effect, and this effect becomes increasingly available from the carried state with depth. Combined with the consistently larger intervention effect of attention than MLP, this pattern supports attention as the dominant route through which role information reaches the relevant token decisions.

ModelRule: Direct ΔB\Delta BRule: Reweighting ΔW\Delta WRule: Non-roleEvidence: Direct ΔB\Delta BEvidence: Reweighting ΔW\Delta WEvidence: Non-role
Qwen2.5-7B73.50.00.185.91.91.4
Mistral-7B88.30.6−0.9-0.992.01.9−0.7-0.7
Llama-3.1-8B87.70.8−0.2-0.293.90.40.8

Table 6. Direct recovery of role effects. Median normalized recovery (%) after substituting the direct role term (ΔB\Delta B), attention reweighting (ΔW\Delta W), or matched non-role positions.

Rule Contributions in Few-Shot ICL

Our main experiments place each rule in an explicit prompt span, allowing us to isolate its effect on generation. To test whether the same account extends to few-shot ICL, we use prompts in which labeled examples implicitly specify the rule. The model must infer the underlying task mapping from these examples. We then test whether the inferred mapping makes a measurable causal contribution to the decision and identify where this contribution is realized within the examples.

We use the SST-2 sentiment task from the Stanford Sentiment Treebank Socher et al. [21] and the six-class TREC question classification task Li & Roth [14]. To isolate the effect of the inferred mapping, we preserve the demonstrations, their order, the query, and the prompt format across each prompt pair, varying only the assignment of labels to classes. We use the demonstration blocks as the rule positions and matched positions outside them as controls.

Table 8(a) shows that all three models follow the induced mapping, while breaking the input–label pairing reduces mapping following with the same inputs and labels preserved in the prompt. This indicates that the rule is induced from relations established across the demonstrations. Table 8(b) shows that the inferred rule yields positive contribution margins and that substituting the demonstration positions recovers nearly the full effect, with negligible recovery from the matched control. Table 8(c) further shows that targeted attenuation reduces the label margin in all six settings and significantly lowers task accuracy in five, with the 95% CI. Together, these results show that the inferred mapping produces a measurable role span contribution that affects the query decision. We next examine where this contribution is realized within the demonstrations.

(a) Rule induction(b) Contribution(c) Decision effect
ModelTask acc. (%) orig./perm.Mapping follow (%)No-demo/broken (%)Positive margin (%)Recovery (%) demo/controlMargin drop [95% CI]Accuracy drop [95% CI] (pp)
SST-2
Qwen2.5-7B77.1 / 86.784.846.7 / 54.381.999.4 / 0.12.27 [1.7, 2.8]25.7 [15.2, 35.2]
Mistral-7B82.9 / 82.982.949.5 / 48.681.999.8 / 0.01.24 [0.8, 1.7]20.0 [7.6, 31.4]
Llama-3.1-8B73.3 / 77.179.041.0 / 41.977.196.2 / −1.0-1.00.63 [0.3, 0.9]15.2 [6.7, 23.8]
TREC
Qwen2.5-7B42.9 / 46.765.720.0 / 25.776.299.4 / −0.1-0.11.04 [0.4, 1.7]8.6 [−1.9-1.9, 20.0]
Mistral-7B52.4 / 47.666.714.3 / 21.974.398.5 / −0.5-0.51.37 [0.9, 1.9]28.6 [18.1, 39.0]
Llama-3.1-8B35.2 / 37.173.314.3 / 17.166.797.0 / −0.7-0.70.69 [0.2, 1.1]18.1 [8.6, 27.6]

Table 8. Role effects for rules induced from demonstrations. The three column groups report rule induction, contribution alignment, and the effect of targeted attenuation. Note. Recovery is evaluated on examples with ∣Φr∣>0.5\lvert\Phi_r\rvert> 0.5. Matched positions outside the demonstration blocks form the control. Drop metrics are computed as control minus targeted attenuation, so larger values indicate stronger dependence on the selected positions.

Note. Recovery is evaluated on examples with ∣Φr∣≥0.5\lvert\Phi_r\rvert\ge0.5. Matched positions outside the demonstration blocks form the control. Drop metrics are computed as control minus targeted attenuation, so larger values indicate stronger dependence on the selected positions.

Table 9(a) shows that the label positions recover nearly the full effect of the demonstration blocks. The input positions recover only a small fraction of this effect. Table 9(b) shows the same organization under targeted attenuation. The margin drop at the label positions is close to that of all demonstration positions, while the input positions show a much smaller change. Together with the pairing result in Table 8(a), these findings separate rule formation from its direct realization at the query decision. The rule is induced from relations across demonstrations, and its direct contribution is concentrated at the label positions.

Model(a) Recovery (%) All(a) Recovery (%) Labels(a) Recovery (%) Inputs(b) Margin drop vs. control All(b) Margin drop vs. control Labels(b) Margin drop vs. control Inputs
SST-2
Qwen2.5-7B99.498.44.42.272.090.29
Mistral-7B99.898.72.41.241.370.38
Llama-3.1-8B96.295.13.20.630.760.23
TREC
Qwen2.5-7B99.498.83.71.040.890.47
Mistral-7B98.591.111.11.371.350.14
Llama-3.1-8B97.090.91.60.690.490.10

Table 9. Position-wise organization of induced role effects. Recovery and margin drop are compared across all demonstration tokens, label tokens, and input tokens.

We further test whether failing label decisions can be restored by amplifying the demonstration contribution, using the same operation as in § 5.3.2. Table 10(a) shows that amplification restores a subset of these failing decisions, while random directions of equal norm do not at larger weights. Table 10(b) further shows that, at similar deficits, single-label rule decisions recover less often than evidence decisions. Table 10(c) shows that recovery follows the prediction in Appendix § 11.§ 4. Together, these results show that whether a failing decision can be restored depends on the support relative to the deficit.

Direction(a) Selection (%) β=4\beta=4(a) Selection (%) β=8\beta=8(a) Selection (%) β=16\beta=16(b) Matched deficit (%) Group(b) Matched deficit (%) Recovery(c) Prediction at β=1\beta=1 (%) Group(c) Prediction at β=1\beta=1 (%) Recovery
Demonstration12.918.822.5Single-label rule6.5β^t≤1\hat{\beta}_t \le159.5
Random5.50.00.0Task-defined evidence47.6β^t>1\hat{\beta}_t > 10.4
Real-hallucinated58.8

Table 10. Restoration of failing single-label rule decisions in few-shot ICL. (a) Target selection under amplification. (b) Recovery at matched deficits. (c) Recovery by the predicted threshold.

Note. β^t\hat{\beta}_t follows Appendix § 11.§ 4.

Evidence from Multiple Sources

Our main evidence experiments place the relevant fact in a single support span, allowing us to isolate its effect on a factual choice. We next examine evidence use across multiple sources. One setting requires the model to combine information from several passages, while the other presents conflicting facts from two sources. In each setting, we identify which evidence positions affect the selected answer and test whether weakening them changes the factual choice.

Distributed evidence. We first study questions from MuSiQue [24] that require two reasoning steps. Each answer depends on a bridge passage and a passage that contains the final fact. The paired prompts preserve the question, passage order, and all other text, while changing only the answer entity. We use the union of the two passages as the evidence positions and matched distractor positions as the control. This construction lets us compare the evidence needed to establish the answer with the evidence that acts at the final choice.

Note. Recovery difference (nats) is the score recovery at the answer decision from evidence substitution relative to the matched distractor substitution. Larger margin drop indicates stronger dependence.

As shown in Table 11(a), the matched effect favors the supported answer for all three models, and substitution through the evidence positions recovers most of this effect relative to the control. Removing either passage also reduces the answer margin, showing that both passages are needed to establish the answer (Table 11(b)). Their direct effects at the final choice are not equal. Weakening the passage that contains the final fact causes a large margin drop in every model, whereas weakening the bridge passage has little direct effect in Qwen and Mistral and a much smaller effect in Llama. Thus, the answer depends on evidence distributed across the two passages, but the final choice depends mainly on the passage that supplies the final fact.

(a) Grounded preference and pathway recovery
Grounded preference (%)Matched effectPathway recovery
ModelOriginalCounterfactualNo evidenceMedian MeM_eRecovery difference (nats)
Qwen2.5-7B82.746.963.336.475.08 [4.0, 6.2]
Mistral-7B89.843.969.435.115.07 [4.2, 6.0]
Llama-3.1-8B98.050.076.526.084.30 [3.6, 5.0]
(b) Support dependence and decision-level effects
Prompt dependence: margin drop (nats)Decision effect: margin drop vs. controlToken flip (%)
ModelBridge removedAnswer removedAll evidenceAnswer spanBridge spanTargetedControl
Qwen2.5-7B1.98 [0.9, 3.1]5.82 [4.5, 7.2]5.89 [4.7, 7.1]4.68 [3.7, 5.7]0.09 [−0.4, 0.6]11.21.0
Mistral-7B1.27 [0.7, 1.9]5.46 [4.4, 6.6]3.98 [3.2, 4.8]3.48 [2.8, 4.2]0.14 [−0.1, 0.4]8.20.0
Llama-3.1-8B1.07 [0.6, 1.5]4.50 [3.7, 5.3]4.37 [3.6, 5.1]3.42 [2.8, 4.0]0.43 [0.1, 0.8]12.20.0

Table 11. Evidence contributions under distributed multi-hop grounding. (a) Grounded preference and pathway recovery. (b) Dependence on each supporting passage and its direct effect at the answer decision.

Competing evidence. We next consider contexts in which two sources state different entities for the same fact. The prompt marks one source as primary. The paired prompts exchange the entities stated by the two sources while preserving the question, passage order, and remaining text. A closed-book condition measures the preference that remains without either claim. We then weaken the two sources separately to test how each one affects the same factual choice.

Table 12(a) shows that context-conditioned preference varies across settings. The original and closed-book rates are nearly identical for Qwen, separate more for Mistral, and differ most clearly for Llama. The matched effect remains positive in all three models. Table 12(b) reveals a more consistent source-specific effect. Weakening the primary source lowers the margin of its answer, whereas weakening the competing source can raise it. The primary intervention also changes the selected token far more often than the control. Thus, the models distinguish the effects of the two sources, even though the designated source does not exert the same control over the final answer.

ModelPrimary preference (%)Matched effectPathway recovery
OriginalSwappedClosed bookMedian MeM_ePositive MeM_e (%)Recovery difference (nats)
Qwen2.5-7B28.425.326.31.0653.7 [43.2, 64.2]4.99 [4.0, 6.0]
Mistral-7B31.613.722.11.9065.3 [55.8, 74.7]4.63 [3.9, 5.3]
Llama-3.1-8B34.75.320.04.0381.1 [72.6, 88.4]4.70 [4.1, 5.3]
ModelPrimary dropSecondary gainPrimaryControl
Qwen2.5-7B6.43 [5.2, 7.7]5.30 [4.2, 6.5]23.23.2
Mistral-7B5.12 [4.3, 5.9]3.91 [3.2, 4.6]26.31.1
Llama-3.1-8B6.30 [5.5, 7.1]3.34 [2.8, 3.9]31.60.0

Table 12. Evidence contributions under competing factual sources. (a) Context-conditioned preference and pathway recovery. (b) Source-specific effects under targeted attenuation.

Taken together, the two settings show that evidence dependence can be distributed across sources, while direct influence and final control remain source-specific. The context determines which information is required, that information shapes the factual choice, and the resulting influence may or may not determine the final answer. These findings extend the evidence contribution account. beyond a single support span while preserving the distinction between contextual influence and final behavior.

Role Span Influence around Key Decisions

So far, our analysis shows that role span contributions act through the output decisions required by each role. The same pattern extends to rules inferred from demonstrations and evidence drawn from multiple sources in Appendices D.4 and D.5. For rules, these decisions govern structure and selection; for evidence, they determine factual content in the response. One may wonder how the role effect is organized around these semantic decisions.

We answer this by comparing role span influence at the semantic decisions with matched output positions. As shown in Table 13(a), role span influence is consistently higher at the semantic decisions than at the matched positions. The same concentration appears when the decisions are specified from reference responses. These results show that semantic decisions form distinct points of role span influence in the output.

ModelDecision percentileMatched controlMedian offsetBefore dec. (%)At dec. (%)
Rule
Qwen2.5-7B85.067.4+046.953.1
Mistral-7B85.763.6−155.944.1
Llama-3.1-8B90.063.0−555.344.7
Evidence
Qwen2.5-7B92.550.2−886.08.0
Mistral-7B77.551.2−687.24.3
Llama-3.1-8B95.050.5−683.79.3

Table 13. Role span behavior at semantic decisions. (a) Role span influence at semantic decisions: the median within-response percentile of ∣δ^t∣\lvert\hat{\delta}_t\rvert at the decision position and at matched positions in other responses. (b) Position of the first generation change relative to the semantic decision. (a) Role span influence (%) (b) Position of generation changes.

Step further, we ask how the generation change is positioned around these semantic decisions. We measure the first divergence relative to each decision position. As shown in Table 13(b), rule changes occur at or shortly before the decision itself, with 44%–53% of first divergences occurring exactly at that position. In contrast, evidence shows a different pattern: 84%–87% of first divergences occur before the factual value is emitted, with median offsets of several tokens. These results indicate that rule and evidence exhibit different positional patterns around their semantic decisions.

We further examine which pathway carries these earlier changes. On a fixed prefix, we remove only the direct evidence term at a position, or remove the remaining transport terms while keeping it. As shown in Table 14, the earlier first divergences occur at low-margin positions and are mainly carried by the indirect terms. In contrast, removing only the direct evidence term at the factual decision changes the selected fact far more often than the same removal on a matched non-evidence span. These results indicate that indirect transport shapes the tokens preceding the fact, while the factual decision itself remains sensitive to the direct evidence contribution.

Direct evidence removedMatched non-evidenceIndirect share median (%)Margin median
All models31.61.791.80.6

Table 14. Direct and indirect evidence effects on a fixed prefix. (a) Changes in the selected fact at the factual decision. (b) Pathway share and margin at the first divergence. (a) Factual decision (%) (b) First divergence.

In short, the semantic decisions provide distinct points of role span influence, while generation changes follow different positional patterns around them. Functional roles therefore organize both the output decisions that carry role span influence and the local pattern of generation changes around those decisions.

Note. Primary drop is the margin decrease of the primary asserted candidate under targeted attenuation, relative to the matched control; Secondary gain is the margin increase after attenuating the competing source.

Note. (a) Both columns show the median within-response percentile of ∣δ^t∣\lvert\hat{\delta}_t\rvert; Matched reads the same positions in other responses. (b) Offset is the first-divergence position minus the semantic-decision position (in tokens); negative values indicate an earlier change.

Note. The prefix is fixed to the original generation. Indirect terms follow the decomposition in Appendix § 11.§ 1.

Predicting Task Recovery

Appendix § 11.§ 4 predicts the recovery of each naturally failing decision under role contribution amplification. We first compare these predictions with the subsequent intervention outcomes. Across the two roles, the predicted and observed decision outcomes agree in 79% of the cases, while the predicted amplification closely tracks the first observed crossing among recovered decisions.

Here, we show that the decision-level predictions can extend to task behavior. Following the role-specific decision organization identified in § 5.2, we map the predicted decision outcomes to task recovery. For evidence, recovery of the factual choice determines the grounding prediction. For rules, the decision-level prediction is not directly comparable, since successful rule following requires sustained support across multiple structural and selection choices.

As shown in Table 15, the observed task recovery follows the predicted trend for evidence, while rule recovery stays near zero, as predicted. For evidence, task success increases from 25.2% to 58.3%, compared with a predicted increase from 23.3% to 69.9%. For rules, both remain near zero over the same range. Thus, we show that the first-order decision formulation can be used to quantitatively predict task recovery in both ICL and OOC hallucination.

β\betaRule span Predicted (%)Rule span Observed (%)Evidence span Predicted (%)Evidence span Observed (%)
0.50.00.023.325.2
10.01.326.232.0
20.02.538.847.6
40.06.352.453.4
80.03.864.155.3
166.36.369.958.3

Table 15. Task-level recovery prediction under role contribution amplification. Predicted versus observed task success across amplification strengths.

References

  1. [1]Shengnan An, Zexiong Ma, Zèqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188, 2024.arxiv.org/abs/2404.16811
  2. [2]Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannahneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, pp. 9112–9141, 2024.arxiv.org/abs/2310.11511
  3. [3]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.arxiv.org/abs/2102.12452
  4. [4]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dharival, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  5. [5]Hakaze Cho, Mariko Kato, Yoshihiro Sakai, and Naoya Inoue. Revisiting in-context learning inference circuit in large language models. In International Conference on Learning Representations, volume 2025, pp. 13081–13117, 2025.arxiv.org/abs/2410.04468
  6. [6]Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 1107–1128, 2024.
  7. [7]Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yuqiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. CI-bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587, 2026.arxiv.org/abs/2602.03587
  8. [8]Kevin Du, Vëstein Snæbjarnarson, Niklas Stoehr, Jennifer White, Aaron Schein, and Ryan Cotterell. Context versus prior knowledge in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13211–13235, 2024.arxiv.org/abs/2404.04633
  9. [9]Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375–24410, 2024.arxiv.org/abs/2406.11944
  10. [10]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to learn in context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4849–4870, 2023.
  11. [11]Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9318–9333, 2023.
  12. [12]Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yangfan Ye, Weihong Zhong, Yuxuan Gu, Baoxin Wang, Dayong Wu, et al. Improving contextual faithfulness of large language models via retrieval heads-induced optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16896–16913, 2025a.arxiv.org/abs/2501.13573
  13. [13]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Ruan, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM transactions on information systems, 43(2):1–55, 2025b.DOI
  14. [14]Xin Li and Dan Roth. Learning question classifiers. In Coling 2002: The 19th international conference on computational linguistics, 2002.
  15. [15]Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086–8098, 2022.arxiv.org/abs/2104.08786
  16. [16]Clement Neo, Shay B Cohen, and Fazl Barez. Interpreting context look-ups in transformers: Investigating attention-mlp interactions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16681–16697, 2024.
  17. [17]Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10862–10878, 2024.arxiv.org/abs/2401.00396
  18. [18]Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Revisiting demonstration selection strategies in in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9090–9101, 2024.
  19. [19]Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 6916–6932, 2024.arxiv.org/abs/2402.13055
  20. [20]Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International conference on machine learning, pp. 31210–31227. PMLR, 2023.arxiv.org/abs/2302.00093
  21. [21]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.DOI
  22. [22]Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319–3328. PMLR, 2017.arxiv.org/abs/1703.01365
  23. [23]Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. In International conference on learning representations, volume 2024, pp. 17282–17333, 2024.arxiv.org/abs/2310.15213
  24. [24]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022.DOI
  25. [25]Xubin Wang, Jianfei Wu, Yichen Yuan, Deyu Cai, Mingzhe Li, and Weijia Jia. Demonstration selection for in-context learning via reinforcement learning. arXiv preprint arXiv:2412.03966, 2024.arxiv.org/abs/2412.03966
  26. [26]Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. Advances in neural information processing systems, 37:33402–33422, 2024.arxiv.org/abs/2404.10198
  27. [27]Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In International Conference on Learning Representations, volume 2025, pp. 62143–62156, 2025.arxiv.org/abs/2404.15574
  28. [28]Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In International Conference on Learning Representations, volume 2024, pp. 35623–35646, 2024.arxiv.org/abs/2305.13300
  29. [29]Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pp. 39818–39833. PMLR, 2023.arxiv.org/abs/2302.05698
  30. [30]Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In International Conference on Learning Representations, volume 2024, pp. 29862–29883, 2024.arxiv.org/abs/2310.01558
  31. [31]Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. Mechanistic understanding and mitigation of language model non-factual hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7943–7956, 2024.arxiv.org/abs/2403.18167

Paper details

vlnc:W_fvwy5her
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents