Context or Knowledge: How LLMs Attribute Their Own Answers

Introduction

When a language model answers a question with a document in context, a user may want to know where the answer came from. Did the model use the document, or what it already knew? This matters when the document updates or contradicts the model’s knowledge. We can ask the model directly (Figure 1). Asked for France’s capital, it answers “Paris” with either a supporting or an irrelevant document, but reports context in the first case and knowledge in the second. We call this reply the source report. Our research question is how the model produces it.

Image not extracted

This question is distinct from asking whether the document supports the answer. A source report is a claim about how the answer was produced. In retrieval-augmented systems, both a retrieved document and parametric knowledge may support the same answer. Prior work on attribution evaluates whether generated content is supported by the available context [15]; work on explanation faithfulness likewise shows that a model’s verbal account of its computation need not directly expose the computation itself [19, 7]. Appendix § 10 reviews related work in more detail. We instead ask what internal state the model uses when it produces a source report about an answer that has already been generated.

Our starting point is the transformer’s key–value (KV) cache. During generation, previously processed tokens leave keys and values that later tokens can attend to. These entries contain more than the token identities themselves: the same question tokens can acquire different cached states depending on the document that was processed before them. This gives us a way to separate what the model saw from the state produced by processing it. We let the model answer normally, intervene on selected cache entries, and only then ask it about the source. This setup lets us distinguish three possible mechanisms for the source report:

H1: Re-read the document. The model may inspect the document again when asked about the source. Under this hypothesis, changing the document after answering should change the report.

H2: Judge the answer using parametric knowledge. The model may instead assess whether its answer is compatible with knowledge stored in its parameters. Under this hypothesis, the report can be determined from the answer’s compatibility with parametric knowledge rather than from a record of how the answer was produced.

H3: Read a cached trace of the earlier computation. The model may use source-related information left in the cache while answering. If so, transplanting the Q+A cache between runs should transfer the report, even when the recipient’s question, answer, and document stay unchanged.

We test these mechanisms in sequence (Table 1). First, we change or remove the document’s cache after the answer has been generated while preserving the Q+A cache. The report remains unchanged, even though the replacement document is still readable. This argues against determining the report from the document currently available at report time. Second, we transplant the Q+A cache between runs with identical question and answer tokens. The report follows the transplanted cache, while the recipient’s document remains fixed. This establishes a causal dependence on the state produced during the earlier computation. Third, we fit linear directions in Q+A values and show that steering these values can change the source report across facts. Fitted K+V components also carry nearly the full transplant effect: transferring them nearly reproduces the report change, while holding them fixed nearly prevents it (§ 6). Finally, we apply the same fitted value directions during answer generation, without refitting. They change which source supplies the answer, linking source-related cached state to source selection even under opposing instructions.

Table 1: Hypotheses for the source report and evidence from our interventions. Blue ✓ marks evidence supporting the hypothesis; orange ✗ marks evidence against it. H1 and H2 are tested as sufficient explanations of the report. H3 represents the hypothesis that the report depends on source-related state produced while answering.

Hypothesis	Proposed mechanism	Evidence
H1: Re-read the document	The model re-inspects the document to produce the source report.	✗ Not supported as a sufficient explanation. Changing or removing the document after answering does not change the report (§ 3).
H2: Judge the answer against knowledge	The model determines the report by evaluating the answer against its parametric knowledge.	✗ Not supported as a sufficient explanation. Transplanting the Q+A cache changes the report while keeping the answer tokens and recipient document unchanged (§ 4).
H3: Read a cached trace	The report depends on source-related state produced while answering.	✓ Supported by our interventions. Q+A cache transplants transfer the report, and steering Q+A values changes it (§ 4, § 5). Clamping fitted K+V projections nearly blocks transfer; transferring those projections alone nearly reproduces it (§ 6).

Importantly, H1 and H2 are tested as sufficient explanations of the report, not as claims that document processing and parametric knowledge cannot contribute to the cached state. A successful Q+A transplant does not imply that the trace represents a single discrete “source label”; it shows that the state produced during the earlier computation contains information that causally affects the later report. Hybrid mechanisms may therefore contribute to the computation as well.

Our main contribution is therefore a mechanistic explanation of source reporting in the controlled setting: the source report depends on source-related computational state inherited from the question-answering process, rather than simply on the document that remains available when the report is requested. We call this state a source trace; its role in reporting leaves open whether it encodes evidence availability, actual source use, or both. The experiments do not establish general reliability of source reports, unrestricted introspection, or a complete semantic interpretation of this state.

Setup: hold the words fixed, change their cached state

Facts and documents. We start from 155 candidate facts across eight relations: capital, continent, official language, chemical symbol, author, painter, composer, and inventor. Each fact has seven one-sentence documents: a true fact, two counterfactuals, two irrelevant facts, and two neutral sentences. Document variants have equal token counts within each item under each model’s tokenizer. Every later token therefore sits at the same position in each run of that fact, so cache entries can be copied between runs without padding.

Models. We test OLMo-3-7B and OLMo-3.1-32B (Team Olmo, [17]), and Qwen3-8B and Qwen3-32B (Yang et al., [23]), with Qwen’s thinking disabled. Every dialogue is shorter than 300 tokens and fits each model’s attention window. Steering experiments use the two smaller models.

One question, different routes to an answer (schematic)
Question (every run): What is the capital of France?
Type	Document	Answer	Intact report
True fact	“The capital of France is Paris.”	Paris	context
Counterfactual	“The capital of France is Tokyo.”	Tokyo	context
Irrelevant	“The capital of Canada is Ottawa.”	Paris	knowledge
Neutral	“Snow melted slowly on the hills.”	Paris	knowledge
Source question: “What is the source of the answer you just gave: the provided Document, or your own knowledge? Answer ‘context’ for the Document or ‘knowledge’ for your own knowledge. One word only.”

Dialogue and source report. The model sees a document and a question, and generates its answer greedily. In § 3–§ 6, the LLM is instructed to use a relevant document even when it conflicts with its knowledge, and otherwise to answer from knowledge. The fixed-document follow-up (Appendix § 12.§ 8) and the experiment in § 7 vary this instruction explicitly. After the answer, we ask the source question. We call the model’s reply the source report: context for the document, knowledge for its own knowledge. The report is the label with the higher summed token log-probability. This agrees with the model’s freely generated one-word reply in 99.6–100% of cases.

Eligible runs. The intact report is the report of an unedited run. We keep only eligible runs, whose answer and intact report match those expected for their document type (above), using rules fixed before the experiments. Eligibility is 92–100% by model and condition. The exception is irrelevant-fact runs on OLMo-3-7B (81–82%, mainly refusals). This filter ensures that every edited run starts from a known answer and report, so that a change can be attributed to the edit. Reported success rates are therefore conditional on eligibility.

Intervention timing. For every token and layer the KV cache stores the key and value vectors that later tokens attend to. We divide the dialogue into spans. The document span is the document tokens. The question span ($Q$) contains the question and the template tokens between the document and the answer. The answer span ($A$) contains the generated answer tokens. We edit the cache after the answer is generated and before the source question is processed. We use two types of edits:

1. A drop removes the document’s cache entries. Later tokens keep their original position indices.

(2) A swap (or transplant) replaces the keys and values of a span with those from another run of the same fact that saw a different document. We write “recipient ← donor”: the recipient run receives the donor’s cache. Swapping the answer span requires identical answer tokens in both runs. Our main transplant replaces $Q$ and $A$ together ($Q+A$). Entries we do not edit are kept, not recomputed. A token’s text thus stays the same while its cached state can come from a different processing history.

Outcomes and rates. A flip is a change of the source report to the other label. A source switch (§ 7) is a change of the answer to the other source’s value, e.g., from “Tokyo” to “Paris”. Rates in § 3–§ 4 are over eligible facts. Steering directions are fitted on some relations and evaluated on others. Report-steering rates pool two retained splits (Appendix § 12.§ 5); steering rates are over item–split pairs, not independent facts. The examples are schematic illustrations, not quoted transcripts. Detailed rates appear in Appendix § 12.§ 1 onward.

Changing the document leaves the report in place

Under H1, the report may compare the answer with the document currently in context. If so, changing the document after the model has answered should change the report. The model reads “The capital of France is Tokyo” and answers “Tokyo”. We keep its question-and-answer cache and either remove the document’s entries or replace them with an irrelevant fact. Only then do we ask the source question (Figure 2).

Image not extracted

Document edit	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
Drop counterfactual	0%	0%	0%	0%
Counterfactual → irrelevant	0%	0%	0%	0%
Irrelevant → true fact	0%	0%	0%	0%

In this setting, neither edit flips a report on any model. The edit is available to the model: in every eligible run with a replaced document, the model can quote the new document. The reverse test leads to the same conclusion. After the model answers “Paris” from memory, inserting a true fact stating “Paris” into the document never turns knowledge into context. The report therefore does not track whether the current document supports the answer. DynamicQA Wikipedia paragraphs show the same pattern, with rare exceptions (Appendix § 14). Table 7 gives the full templated-task results.

	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
Counterfactual run (read “Tokyo”, answered “Tokyo”): reports context
intact	100%	100%	100%	100%
DOCUMENT DROP	100%	100%	100%	100%
DOCUMENT SWAP → irrelevant fact	100%	100%	100%	100%
model quotes the new document	100%	100%	100%	100%
Irrelevant-fact run (answered “Paris” from memory): reports context
intact	0%	0%	0%	0%
DOCUMENT SWAP → true fact (its own answer)	0%	0%	0%	0%
DOCUMENT SWAP → counterfactual	0%	0%	0%	0%
QUESTION SWAP (document and answer stay): report becomes . . .
counterfactual ← irrelevant-fact: knowledge	97.9%	100%	100%	14.2%
irrelevant-fact ← counterfactual: context	86.6%	90.6%	6.7%	100%
controls: report changes	0%	0%	0%	0%

The report follows the question-and-answer cache

The previous experiment showed that the report does not change when we replace the document after the answer has been generated. This suggests that the information used for reporting may be retained in the model’s internal state from the earlier question-answering computation. We test this by transplanting the KV cache of the question and answer (Q+A cache) between runs.

We pair runs with identical question and answer tokens but different documents. For example, both runs may answer “Paris”, one with a supporting document and a context report, the other with a neutral document and a knowledge report (Figure 3). We copy the Q+A cache from the donor run into the recipient; the recipient’s document and all other cache entries remain unchanged.

Image not extracted

After the transplant, the report consistently follows the donor’s cache across all four tested models (Table 3). For example, replacing the Q+A cache of a neutral or irrelevant-document run with the cache from a supporting-document run changes the recipient’s report to context, even though the recipient’s original document is not relevant. Conversely, transplanting the Q+A cache from a neutral or irrelevant-document run causes a recipient that still contains supporting evidence to report knowledge. Control transplants, which preserve the source condition, produce no report flips. On DynamicQA Wikipedia paragraphs, joint Q+A transplants similarly produce 97.2–100% donor-following (Appendix § 14).

Recipient $\leftarrow$ Q+A donor	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
True fact $\leftarrow$ neutral	100%	100%	100%	100%
Neutral $\leftarrow$ true fact	100%	100%	100%	100%
True fact $\leftarrow$ irrelevant	100%	100%	100%	100%
Irrelevant $\leftarrow$ true fact	100%	100%	100%	100%

The intervention changes the report while leaving the recipient’s document and answer unchanged. Thus, the report cannot be determined solely by whether the recipient’s document supports the answer, because that document is identical before and after the transplant. Likewise, the answer itself is identical, so the report cannot be explained simply by the answer string or by a difference in the model’s final answer. Instead, the intervention changes only the Q+A cache, indicating that source-related information carried in this state contributes to the report.

Why transplant the question and answer together? The source-related information may be distributed across the computation of both the question and the answer. We therefore transplant their KV entries jointly, which produces a consistent effect across conditions. Results for separate question-only and answer-only transplants are reported in Appendices § 12.§ 2 and § 12.§ 4.

The transplant experiment therefore provides causal evidence that the later source report depends on information encoded in the Q+A cache. We next ask whether this source-related component can be directly manipulated by a shared intervention in the cache.

Steering question-and-answer values changes the report

We next ask whether the source-related state identified by the transplant experiment can be directly manipulated. We construct directions in Q+A value space from the difference between runs with supporting and irrelevant documents.

For each model and relation split, we fit directions on one set of relations and evaluate them on the remaining relations. Fitting examples pair a supporting-fact document that supports the answer with an irrelevant-fact document that does not; answer-span fitting requires identical answer tokens. At each layer, we average the value vectors over the positions of the question or answer span and compute the mean difference between the two conditions:

Here, $s$ specifies whether the intervention is applied to the question or answer span, and $\mathcal{F}_{s}$ is the corresponding fitting set. Values carry the information retrieved by attention; the key/value ablations motivating this choice are in Appendix § 12.§ 3. Fitting uses states recorded before any source question. We then intervene on the Q+A values after the model has already produced its answer, leaving the keys and all other cache entries unchanged. Adding the direction moves the Q+A state toward the supporting-document condition; subtracting it moves the state in the opposite direction. At $\alpha= 1$, the intervention has the magnitude of the mean difference between the two fitting conditions.

As a control, we apply random directions with the same layerwise norms and the same intervention locations and strengths. Each random condition is averaged over three draws.

Image not extracted

The learned direction reliably changes the subsequent source report in both directions (Figure 4). With an irrelevant document, adding the direction produces a context report in 100% of evaluations at $\alpha= 2$ on both models. Conversely, with a counterfactual document, subtracting the direction removes the context report in 100% of evaluations by $\alpha= 3$ on both models. In all of these interventions, the answer tokens were generated before the edit and remain fixed while the subsequent source report changes.

The effect also generalises to neutral and true-fact documents. Across all four document conditions, $\alpha= 3$ flips 100% of reports except on OLMo true facts (98.9%). At $\alpha= 1$–3, random controls produce no flips on Qwen and at most 1.1% on OLMo. Additional results, including the comparison of joint and separate-span interventions, are in Appendices C.5–C.6.

These results show that the Q+A cache contains a component that can be accessed by a shared linear intervention to change the source report. They do not imply that the entire source-related state is one-dimensional: the learned direction captures a manipulable component of the natural Q+A state.

For irrelevant ($\alpha= 2$) and counterfactual documents ($\alpha= 3$), every report flips while answer repetition and document quotation both succeed. Both models also retrieve the counterfactual value in every case. These checks use separate continuations of the same edited cache (Appendix § 12.§ 7).

Interpretation of the contrast. This contrast does not separate evidence availability from source use. A fixed-document contrast also supports report steering (Appendix § 12.§ 8), but neither contrast establishes a pure use signal. The intervention therefore establishes that the state produced during answering is sufficient to change the later source report.

Image not extracted

Document / direction	Model	$\alpha= 1$	$\alpha= 2$	$\alpha= 3$	$\alpha= 4$
Irrelevant fact: create	OLMo-3-7B	42.6%	100%	100%	100%
	Qwen3-8B	4.9%	100%	100%	100%
Neutral sentence: create	OLMo-3-7B	12.8%	88.5%	100%	100%
	Qwen3-8B	0%	2.4%	100%	100%
Counterfactual: remove	OLMo-3-7B	0%	60.5%	100%	100%
	Qwen3-8B	12.5%	100%	100%	100%
True fact: remove	OLMo-3-7B	0%	46.6%	98.9%	100%
	Qwen3-8B	0%	96.6%	100%	100%

Fitted directions capture the report-transfer effect

Copying the Q+A cache makes the source report follow the donor (§ 4). Steering also changes the report (§ 5). Do the fitted directions capture the information that makes the transplant work? We pair true-fact and irrelevant-fact runs with identical Q+A tokens (Figure 5). We reuse the value directions from § 5 and fit key directions from the same contrast. All edits cover Q+A at every layer, after answering and before asking for the source. The document and all dialogue tokens stay fixed.

Image not extracted

Are these components needed for transfer? Copy the donor’s full Q+A cache, then restore its projections onto the fitted directions to those of the recipient. For a value vector, this clamp is

Here $v_D$ and $v_R$ are the original donor and recipient value vectors at the same layer and position; $u$ is the unit direction fitted for that layer and span. The projection $\langle v,u\rangle u$ is the component of $v$ along $u$, with $\langle\cdot,\cdot\rangle$ denoting the dot product. Keys use a norm-preserving rotation instead (Appendix § 13). Joint K+V clamps reduce flips from 100% to 1.1% on OLMo and from 100% to 0% on Qwen.

Is transferring these projections enough? Start from the recipient’s cache and impose the donor’s projections: for values, exchange $D$ and $R$ on the right of Equation (2). Without amplification, this flips 97.9% of OLMo and 100% of Qwen reports, nearly reproducing the full transplant.

Nearly necessary and sufficient for report transfer. By contrast, clamping a control subspace with comparable donor–recipient difference energy on fitting data leaves 100% and 94.7% of reports flipped (Table 4). On these pairs, the fitted K+V components therefore largely account for the full transplant’s effect on the report. Values-only tests and behavioural-check limits are in Appendix § 13.

Intervention	OLMo-3-7B	Qwen3-8B
Full Q+A KV transplant	100%	100%
+ restore recipient K+V projections	1.1%	0%
+ clamp energy-matched subspace	100%	94.7%
No transplant: donor K+V projections only	97.9%	100%

Steering values during answering changes the chosen source

So far we edited the cache only after the answer was complete, which can change the report but not the answer. We now ask whether the same value-steering direction also affects which source the model answers from. To test this, we steer values while the model reads the question and generates its answer. This readout needs no source question. Given the document “The capital of France is Tokyo”, the answer “Tokyo” comes from the document and “Paris” from memory.

Image not extracted

We reuse the fitted value direction of Equation (1) (true-fact minus irrelevant-fact values), without refitting for the instructions tested here. We apply it at $\alpha= 2$ at every layer, first to the question tokens and then to each generated answer token. To make the test stringent, the instruction always opposes the edit. We steer toward memory when the model is told to use the document, and toward the document when it is told the document may contain errors. Recipients are runs that initially answered from the source we steer away from.

Steering overrides the instruction in both directions on both models (Figure 6). Toward memory, it switches 71% of OLMo and 51% of Qwen answers; toward the provided context, 85% and 73%. In these two conditions, random directions of equal norm switch at most 4.1% of answers, except for OLMo toward the context (24%), where the fitted direction still switches 3.6 times as many. At most 9.3% of answers are damaged. Residual-stream steering is weaker than value steering on OLMo and stronger on Qwen (Appendix § 12.9). The same fitted value direction therefore changes both the later source report and which source supplies the answer, even when the instruction opposes the intervention.

Image not extracted

Steer toward	Edit	OLMo-3-7B Switched	OLMo-3-7B Random	OLMo-3-7B Damaged	Qwen3-8B Switched	Qwen3-8B Random	Qwen3-8B Damaged
Memory	Values	70.9	1.6	9.3	51.2	4.1	1.2
Memory	Residual	16.3	0.2	0.6	75.6	3.7	2.4
Document	Values	85.3	23.6	0	72.8	3.2	0
Document	Residual	52.3	5.5	1.8	98	1.2	0

Discussion and limitations

Together, the interventions give a consistent account of source reporting in this setting. Editing the document after answering leaves the report unchanged. Transplanting the question-and-answer cache moves the report while every recipient token stays fixed. A shared value direction in that cache can change the report. K+V projections onto fitted directions also nearly block or reproduce report transfer on true-fact–irrelevant-fact pairs. Applying the same value direction during answering changes which source supplies the answer. The report therefore depends on state computed before the source question is asked. These findings concern eligible runs and one source-question wording. Document edits and transplants extend to selected DynamicQA items; steering and projection tests use only 7–8B models. General attribution reliability in natural retrieval settings remains untested. Three questions remain open:

Meaning and formation of the directions. Our interventions establish a causal role in reporting without resolving whether the trace represents evidence availability, actual source use, or both. The fixed-document comparison leaves this distinction open (Appendix § 12.§ 8). Future work should determine what these directions encode and how they form during computation. H1 and H2 fail as complete accounts, but hybrid mechanisms may contribute.

Roles of keys and values. Joint K+V edits nearly block or reproduce report transfer. Value-only clamps are less effective when keys are also transplanted (Appendix § 13). Explaining how keys and values contribute separately, and how they interact, is a next step. Answer repetition and document read-back remain untested for these joint K+V edits.

Roles of question and answer states. Our main interventions edit Q+A together. Separate-span ablations show that their contributions vary across models and conditions (Appendices § 12.§ 4 and § 12.§ 6). Future work should distinguish what is established while processing the question from what is added during answering. The fitted directions span layers and Q/A positions; they do not define one global source variable.

Reproducibility statement

The appendix reports the joint steering sweep, compact span and fitting-contrast comparisons, controls and the mapping from experiments to scripts and result artifacts. Full sweeps, random controls, behavioural checks and the fitting protocol are preserved in the accompanying data.

AI USE STATEMENT

We used generative AI tools to assist with drafting and editing the manuscript and with writing code. The authors take responsibility for the interpretation of the results, the scientific conclusions, and the final content of this paper.

Related work

Reports about internal computation. Prior work studies uncertainty, truth-related states and the faithfulness of explanations and self-reports [6, 9, 1, 19, 7, 3]. In a close precedent, Lindsey [10] injects a concept before an artificial prefill, making Claude more likely to accept it as intentional. We study naturally generated answers and test the later source report with document edits and Q+A transplants that preserve the question and answer tokens.

Knowledge conflicts and source selection. Contextual and parametric knowledge can compete [11, 22, 14] (Jin et al. 2024). Zhao et al. [25] predict source choice from residual activations; SpARE [26] controls it through sparse autoencoder features. Our central contribution concerns the mechanism of the subsequent source report. § 7 connects source-related state to answer selection.

Attribution to context. ContextCite [4] estimates contextual contributions to generation using context ablations and a sparse linear surrogate. We instead intervene after answering, preserving downstream Q+A state. Invariance of the later report therefore does not imply that the document did not contribute to generation.

Patching and steering. We build on activation patching [20, 13, 24], mean-difference steering [18, 27, 8, 16], and KV-cache steering [2], with matched controls to assess intervention specificity [21].

Generated factual items

Each candidate is a subject–relation–object statement, rendered as “The {phrase} of {subject} is {value}.” and queried with “{Who/What} is the {phrase} of {subject}?” Table 5 describes the original pool of 155 candidates. Model- and intervention-specific eligibility checks determine which items enter each reported result (§ 2).

Relation	Phrase	Candidates
capital	capital	30
symbol	chemical symbol	26
author	author	22
continent	continent	21
language	official language	20
painter	painter	12
composer	composer	12
inventor	inventor	12
Total		155

Counterfactual construction. Counterfactuals use other attested values of the same relation. For candidate $i$, let others list that relation’s distinct values excluding the true value. The counterfactual is chosen deterministically as others[i mod |others|]; the second counterfactual uses the next value in the same cycle. A capital is thus replaced by a capital and an author by an author, preserving the answer type without inventing new strings. Values need not be unique across subjects: the continent relation has six distinct values across 21 candidates.

Parametric-knowledge check. Each candidate is queried with neutral filler in place of an informative document. The check accepts an answer if it contains the true value and does not contain the counterfactual, using normalised string matching over at most 12 greedily decoded tokens. It is applied per model, so eligible item sets may differ. This tests recall of the factual answer; the additional answer and intact-report requirements for each intervention are specified in § 2.

Length matching. The seven document variants described in Appendix § 12.§ 1, including neutral fillers, have equal token counts within an item under each model’s tokenizer. This keeps the later question and answer positions aligned across conditions, allowing cache entries to be exchanged without a positional shift. Length matching is an additional constraint on the candidate inventory, separate from the model’s ability to recall a fact.

Relation-based splits. Directions are fitted on DEV relations and tested on held-out relations. Two candidate splits exchange the groups {capital, continent, language, symbol} and {author, painter, composer, inventor}; the third uses a mixed DEV group. The exact memberships and retained splits for the reported steering experiments are given in Appendix § 12.§ 5. Thus, evaluation relations are absent from the corresponding fitting set, while the extent of domain transfer depends on the split.

Additional results and details

Documents and document interventions

The examples in Table 6 illustrate the conditions. Each item has seven documents: one true fact and two variants of each of the other conditions. Exact token-length equality is enforced within each item in the experimental dataset. Appendix § 11 describes the candidate facts and their construction.

document	example	control variant	role
true fact	The capital of France is Paris.	—	relevant, true value
counterfactual	The capital of France is Tokyo.	. . . is Lima.	relevant, false value
irrelevant fact	The capital of Canada is Ottawa.	. . . of Egypt is Cairo.	true fact, other subject
neutral sentence	Snow melted slowly on the hills.	The library was open until late.	no fact at all

Read-back caveats. Qwen3, after receiving the question KV of an irrelevant-fact run, says the document does not mention the subject when asked to read it (77.4% on Qwen3-8B); on that model the clean dissociation is the counterfactual ← neutral question-swap test (Table 8). After the document’s KV is dropped, Qwen3-32B reconstructs the document from its own answer when asked to repeat it (26.2%); the DOCUMENT SWAP is the clean version of that cell.

recipient $\leftarrow$ donor: report becomes	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
The same true answer “Paris”; QUESTION SWAP
true-fact $\leftarrow$ neutral: knowledge	80.2%	100%	100%	95.8%
neutral $\leftarrow$ true-fact: context	97.4%	100%	73.3%	100%
A false answer “Tokyo” in plain sight; QUESTION SWAP
counterfactual $\leftarrow$ neutral: knowledge	90.4%	100%	68.3%	25.8%
still reads “Tokyo” in the document	83.3%	97.2%	68.3%	25.8%
neutral $\leftarrow$ counterfactual: context	89.5%	97.2%	49.2%	100%
IDENTICAL-TEXT SWAP: question and answer swapped, visible text unchanged
true-fact $\leftarrow$ neutral: knowledge	100%	100%	100%	100%
neutral $\leftarrow$ true-fact: context	100%	100%	100%	100%
true-fact $\leftarrow$ irrelevant-fact: knowledge	100%	100%	100%	100%
irrelevant-fact $\leftarrow$ true-fact: context	100%	100%	100%	100%
controls: report changes	0%	0%	0%	0%

Question and answer transplants

Question-only tests first located a signal downstream of the document, but were not uniformly effective. Figure 7 retains that comparison. The main Q+A result uses identical-answer pairs, a different eligible subset from the counterfactual/memory pairs below.

Image not extracted

Rates are over facts for which both runs qualify; the identical-text intervention additionally requires identical answer tokens. The read-back row measures the joint event: the report changes to knowledge and the model can still read the false value from the document.

Keys and values in question-only transplants

swap of the question span	keys+values	keys only	values only
counterfactual ← irrelevant (remove)	97.9% · 100%	0% · 0%	6.2% · 0%
true-fact ← irrelevant (remove)	92.9% · 100%	0% · 0%	0% · 0%
irrelevant ← counterfactual (create)	86.6% · 90.6%	40.2% · 0.9%	99% · 97.2%
irrelevant ← true-fact (create)	90.9% · 98.3%	54.5% · 5.2%	100% · 100%

Question and answer spans in transplants

Using the pairs from the IDENTICAL-TEXT SWAP experiment (§ 4), we swap the question cache, the answer cache, or both. The joint result establishes sufficiency for transferring the report on these pairs; it does not establish that each span separately encodes an independent source variable.

	question only	answer only	question + answer
OLMo-3-7B, Qwen3-8B, Qwen3-32B (all four directions)	75–100%	0–3%	100%
OLMo-3.1-32B, irrelevant-fact ← true-fact	30%	0%	100%
OLMo-3.1-32B, true-fact ← irrelevant-fact	100%	59%	100%

Joint report steering across strengths

The sweep uses the true-fact versus irrelevant-fact contrast of § 5. Rates are over eligible item–split pairs pooled over splits 0 and 2, not independent facts. The full fitted-direction sweep and random controls are retained in the accompanying data.

Relation splits. The three candidate DEV sets are: split 0, {capital, continent, language, symbol}; split 1, {author, painter, composer, inventor}; and split 2, {capital, language, author, composer}. Evaluation uses the remaining relations. The reported Q+A sweep and the answer-only fitting-contrast comparison use splits 0 and 2. Exact prompts and fitting details are archived in data/steering_protocol.json.

Random means: all Qwen cells are zero. For OLMo, irrelevant-fact Q+A gives 1.1% at $\alpha= 3$ and 1.6% at $\alpha= 4$; neutral Q+A gives 2.2% at $\alpha= 4$. All other joint cells are zero.

Question and answer spans in report steering

Table 12 compares Q, A and Q+A at a common strength. The joint edit is more effective in every model–condition cell, supporting the choice of Q+A in the main text. It does not imply that either span has a single exclusive function.

Document / direction	Model	Q only	A only	Q+A
Irrelevant fact: create	OLMo-3-7B	62.3%	67.2%	100%
	Qwen3-8B	97.5%	3.7%	100%
Neutral sentence: create	OLMo-3-7B	42.3%	62.8%	100%
	Qwen3-8B	8.4%	0%	100%
Counterfactual: remove	OLMo-3-7B	0%	72.1%	100%
	Qwen3-8B	58%	85.2%	100%
True fact: remove	OLMo-3-7B	0%	37.5%	98.9%
	Qwen3-8B	40.9%	23.9%	100%

At $\alpha= 3$, random means are zero except OLMo irrelevant-fact Q (0.5%) and Q+A (1.1%). Answer-only random controls are zero at all four strengths. Across the full sweep, the maximum random rate is 2.2%.

The distinction persists at higher strength in the counterfactual setting: answer-only steering at $\alpha= 4$ flips 86% of OLMo and 90.9% of Qwen reports, whereas Q+A flips 100% on both models. Thus, the complete effect reported in the main text belongs to the joint intervention.

Content checks after report steering

These checks use the same edited Q+A caches and eligible item–split pairs as the report-steering sweep (block 3i). The report, answer repetition, document quotation and counterfactual-value read-back are separate continuations; no evaluations are filtered after intervention. Table 13 shows the operating points used for the two main conditions.

Document / direction	$\alpha$	OLMo-3-7B	Qwen3-8B
Irrelevant fact: create	2	100%	100%
Counterfactual: remove	3	100%	100%

For answer repetition, “Repeat the answer you just gave, exactly.” succeeds if the response contains the original answer. For quotation, “Repeat the Document above word for word.” succeeds if the response contains the full document, ignoring case and a final period. Both checks succeed on every unedited baseline. After steering, quotation succeeds in every evaluation in Table 13. For counterfactual documents, “According to the Document above, $\langle q\rangle$ Reply with the exact word from the Document.” also retrieves the false value in every case; success requires that value and no refusal such as “not mentioned”. These checks establish preservation at the stated settings, not across every strength or intervention. The full supplied checks are archived in data/report_steering_checks.json.

Supporting versus irrelevant and fixed-document fitting contrasts

We compare two fitting procedures while editing only answer values. The original contrast is supporting versus irrelevant documents. The fixed-document contrast changes the instruction: $P_D$ says to use a relevant document even against prior knowledge; $P_M$ warns that the document may be wrong and favours memory.

Construction. Let CF and IRR denote counterfactual and irrelevant documents. At layer $\ell$ and span $s$, we fit

Here $\overline{V}$ averages values over span positions, retaining all KV heads. The second difference subtracts the prompt contrast measured with an irrelevant document. All four answers are generated greedily, up to eight tokens, and their caches are rebuilt from the prompt and generated answer. CF answers differ across prompts; IRR answers need not match either. Each answer is averaged over its own span, so answer content can enter this contrast. The original answer direction instead requires identical answer tokens.

Fitting and evaluation. We retain items that use CF under $P_D$ and memory under $P_M$, and memory with IRR under both prompts, with matching source reports: 48.4% of OLMo and 77.9% of Qwen items. Both directions require at least ten DEV pairs. The answer-only comparison uses splits 0 and 2 (Appendix § 12.§ 5); split 1 fails the DEV-pair minimum for the fixed-document contrast on OLMo and the original contrast on Qwen. Recipients are eligible $P_D$ runs; edits cover answer values at every layer after generation. Exact prompts and selection rules are in data/steering_protocol.json.

The fixed-document direction removes context more effectively on Qwen. On OLMo, neither contrast dominates removal across the full sweep, while the original contrast creates context more effectively. Neither answer-only direction reliably creates context on Qwen. These names describe fitting conditions, not established semantic variables.

Geometry and scope. Norms are $\sqrt{\sum_{\ell}\lVert d_{\ell}\rVert^{2}}$ over all layers and heads, computed per split. Cosines are unweighted means of per-layer cosines, including a near-zero layer-0 contribution: 0.565 and 0.571 for OLMo, and 0.577 and 0.569 for Qwen, on splits 0 and 2, respectively. Per-split norms and full sweeps are archived with the protocol and comparison data. Answer repetition was not measured at $\alpha= 3$, so Table 14 establishes report control at that strength, not preservation of answer readout.

Document / direction	Model	Supporting vs. irrelevant	Fixed document
True fact: remove	OLMo-3-7B	37.5%	30.7%
True fact: remove	Qwen3-8B	23.9%	77.3%
Counterfactual: remove	OLMo-3-7B	72.1%	72.1%
Counterfactual: remove	Qwen3-8B	85.2%	100%
Irrelevant fact: create	OLMo-3-7B	67.2%	37.7%
Irrelevant fact: create	Qwen3-8B	3.7%	2.5%
Neutral sentence: create	OLMo-3-7B	62.8%	23.1%
Neutral sentence: create	Qwen3-8B	0%	0%

Source-choice steering: instructions and controls

We use the two settings in § 7. To steer toward memory, we keep the document-priority instruction used in the preceding experiments. To steer toward the document, we instead warn that the document may contain errors and instruct the model to prefer its own knowledge. This provides recipients that initially answer from memory; otherwise, the model usually already uses the document. In both settings, recipients initially answer from the source we steer away from.

Value steering reuses the fitted directions of § 5, at all layers and without refitting for the instruction. The residual-stream comparison adds a mean hidden-state difference fitted at question positions under a neutral instruction. It uses one layer selected on fitting relations: OLMo layers 12, 16 and 12 across the three splits, Qwen layer 20, out of 32 and 36 layers respectively. Both methods intervene during question processing and answer generation at $\alpha= 2$.

Terminology used in the tables

Flip: the report changed to the other source label. Switched (§ 7): the answer changed to the other source’s value. Damaged: the answer contains neither value. Random: a random direction with the same norm per layer, same positions and $\alpha$, averaged over 3 draws. Held-out: directions fitted on some relations and measured on others; retained splits depend on the experiment.

Where every number comes from

section	script	artifacts
items	90_clean_items.py	items.jsonl
3–4, document and question swaps	91_clean_kv_patching.py	kv_patching.json, kv_patching_items.jsonl
4, IDENTICAL-TEXT SWAP	93_clean_answer_patching.py	answer_patching.json, answer_patching_items.jsonl
tables for 3–4	92_clean_tables.py	tables.md
5, report-steering sweep	author-supplied aggregate table	data/report_steering_update.csv
Appendix C.7	author-supplied block 3i (script 105)	data/report_steering_checks.json
7	96_clean_source_steering.py	source_steering.json, source_steering_items.jsonl
Appendix E	author-supplied aggregate tables	data/dynamic_results.json
Appendix C.8	author-supplied blocks 3e/3g (scripts 100 and 102)	data/answer_direction_comparison.json; data/steering_protocol.json
6, Appendix D	author-supplied blocks 3j–3l	data/transplant_clamp_results.json

Clamps and transfers along fitted directions

Paired runs, directions and readout

Blocks 3j (values), 3k (joint K+V) and 3l (energy controls) use OLMo-3-7B-Instruct and Qwen3-8B with thinking disabled, bf16 and Transformers 4.57.1. OLMo and Qwen have 32 and 36 layers, respectively, with 32 and 8 KV heads of dimension 128.

Pairs follow § 4: a true-fact run reports context, while its irrelevant-document partner gives identical answer tokens but reports knowledge. Equal document lengths align positions. The question span runs from the document’s end to the prompt’s end, including chat-template tokens; the answer span ends at the answer’s last token. Edits cover Q+A at every layer after answering and before the fixed source question. Other cache entries remain unchanged.

Directions are fitted on DEV relations and evaluated on held-out relations in two splits. Rates pool both splits and both directions for each true-fact–irrelevant-fact pair; the unit is a directional evaluation, not a distinct fact. The readout is forced-choice $P(\text{context})$; a flip crosses 0.5 away from the recipient’s original report. These results are not pooled with the report-steering sweep.

For each layer and span, take the mean over DEV items of the position-averaged true-fact minus irrelevant-fact difference, then normalise the resulting direction. Value directions are the same as in § 5, flattened across KV heads: one coordinate per position in 4096 dimensions on OLMo and 1024 on Qwen. Key directions are fitted in the pre-RoPE, post-$k$-norm-gain space. Grouping follows the model’s key normalisation: one 4096-dimensional group on OLMo, but eight 128-dimensional head groups on Qwen. Question and answer positions use their respective fitted directions. Layer 0 has zero difference and is left unchanged. This is a family of local coordinates, not a rank-one intervention on the entire cache.

Two operations with the same coordinates

Let $u$ be a fitted unit direction, $x$ the vector being edited, and $s$ the paired vector supplying the target scalar projection (called a coordinate here). A value edit is

For a clamp, first transplant Q+A K+V, then use the recipient’s original vector as $s$. For donor coordinates only, start from the recipient’s cache and use the donor as $s$. The value component orthogonal to $u$ is unchanged. Both operations use observed coordinates rather than an amplified mean-difference shift.

Keys require a norm-preserving edit. We first undo RoPE using the relevant layer’s rotary cosine and sine, including OLMo’s distinct sliding-window and full-attention rotary modules:

Reported round-trip error is approximately $2 \times10^{-7}$. Within each key group, write $c = \langle y,u\rangle$, $r = y - cu$, and set $c'$ to the source coordinate, clipped to $\pm0.999\lVert y\rVert$. The edit is

We then reapply RoPE and store the result in bf16. This keeps the starting key norm while changing its orientation; unlike the value edit, it rescales the orthogonal component. It does not preserve attention weights. Earlier additive key interventions caused nonspecific report changes even for random directions, motivating the norm constraint.

The rotation is performed after the key-normalisation gain. OLMo has near-zero gains in approximately a quarter of key dimensions, making inversion of the gain numerically unstable. The edit preserves Euclidean norm in this post-gain space, but need not stay on the exact ellipsoid defined by the model’s normalisation and gain.

Value and key contributions

	OLMo-3-7B		Qwen3-8B
	Create	Remove	Create	Remove
K+V transplant	100%	100%	100%	100%
V transplant	100%	10.6%	100%	47.4%
V transplant + V clamp	0%	0%	0%	0%
V transplant + random V clamp	100%	10.6%	100%	47.4%
K transplant	78.7%	0%	39.5%	0%
K+V transplant + V clamp	100%	0%	68.4%	0%
Donor V coordinate only	100%	2.1%	100%	31.6%

Value-coordinate clamps eliminate flips produced by value-only transplants: 0% of reports flip on either model, compared with 55.3% on OLMo and 73.7% on Qwen without the clamp. With donor keys also transplanted, a value-only clamp is less effective, leaving 50% and 34.2% of reports flipped, all when creating context.

	OLMo-3-7B		Qwen3-8B
	Create	Remove	Create	Remove
K+V transplant + V clamp	100%	0%	68.4%	0%
K+V transplant + K+V clamp	2.1%	0%	0%	0%
K+V transplant + V clamp + random K clamp	100%	0%	67.6%	0%
K transplant + K clamp	0%	0%	0%	0%
K transplant + random K clamp	76%	0%	38.7%	0%
Random donor coordinates only	0%	0%	0%	0%

Joint K+V clamps leave 1.1% of reports flipped on OLMo and 0% on Qwen. Residual OLMo flips occur only in the creation direction (2.1%); no flips remain in the removal direction on either model.

Transferring only the donor’s K+V coordinates flips 97.9% of OLMo and 100% of Qwen reports, pooled over both directions (Table 19), matching the main result in Table 4.

Intervention	OLMo-3-7B	Qwen3-8B
Full K+V transplant	100%	100%
+ selected-coordinate clamp	1.1%	0%
+ top-PC clamp	100%	100%
+ energy-matched clamp	100%	94.7%
+ random clamp	100%	100%
V transplant + selected-coordinate clamp	0%	0%
+ energy-matched clamp instead	51.1%	65.8%
+ random clamp instead	55.3%	74.1%
Donor K+V coordinates only: selected	97.9%	100%
top PC instead	0%	0%
energy-matched subspace instead	0%	2.6%
random directions instead	0%	0%

Energy-matched controls

A random one-dimensional subspace captures very little of the donor–recipient difference. To test whether the selected coordinates matter simply because they capture a large change, block 3l also fits a control subspace on DEV. After projecting out the selected direction, it takes the fewest leading principal directions of per-position DEV differences needed to capture at least as much squared difference energy as the selected direction. This choice is made separately for each layer, span and key group. Reported median subspace dimensions range from 3–72 on OLMo and 6–186 on Qwen, versus one selected direction per group. The top-PC control uses only the first such principal direction. Random controls use three draws and the same intervention locations and coordinate-replacement geometry.

Model	Cache	Selected	Matched	Random
OLMo-3-7B	V	11.6%	12.3%	0.0%
OLMo-3-7B	K	25.1%	21.0%	0.0%
Qwen3-8B	V	15.7%	18.5%	0.1%
Qwen3-8B	K	36.2%	49.3%	0.8%

For a subspace projector $P$, the reported held-out energy fraction is

with key-group contributions also summed. This measures projected difference energy, not the norm of the nonlinear key edit. Matching is enforced on DEV only: the held-out OLMo key control captures 21.0%, compared with 25.1% for the selected directions. Qwen’s matched control captures more held-out energy in both values and keys, yet leaves 94.7% of reports flipped after clamping and flips only 2.6% when its donor coordinates are imposed alone.

Scope of the behavioural checks. Answer repetition and document read-back were reported as fully preserved for value-only donor-coordinate edits, but have not been supplied for the joint K+V conditions. Fixed dialogue tokens therefore do not establish preservation of subsequent content readouts. Nor do these interventions establish exclusive coordinate semantics or a complete account of ordinary source reporting.

Replication on DynamicQA Wikipedia paragraphs

We extend the document and cache-transplant experiments to a Wikipedia-paragraph adaptation of DynamicQA (Marjanovic et al., 2024). Each item has seven length-matched document variants. For each model, we restrict the item pool to questions it can answer from memory. This evaluates the interventions on richer, benchmark-derived passages while preserving a known memory answer.

Rates are over eligible items for each condition or pair, rather than item–split pairs or the full memory-known pools. “Uninformative” denotes the condition providing no answer information; “irrelevant” denotes a paragraph about a different fact. Identical-token transplants additionally require matching question and answer tokens. All four model columns use the same order as the main tables.

Document interventions

We edit the document after answering and retain the Q+A cache. Table 21 reports the percentage of context reports, rather than using a single flip metric for interventions with different initial reports.

Document edit	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
Drop counterfactual	100%	98.4%	99.4%	100%
Counterfactual → irrelevant	100%	100%	99.4%	100%
Irrelevant → true fact	0%	0%	0%	0%
Irrelevant → counterfactual	0%	0%	0%	0%

After deletion or irrelevant replacement of a counterfactual paragraph, context is retained in 98.4–100% of items across model–edit cells. Inserting a true or counterfactual paragraph into an irrelevant-document run creates no context reports. Thus, the pattern from § 3 extends to these Wikipedia paragraphs, with rare exceptions to exact invariance.

Question-only transplants

Table 22 combines the question-only tests with irrelevant and uninformative donors. These effects vary by model and direction. The joint read-back rate uses all eligible items; it is not a success rate conditional on the report having flipped.

Recipient ← Q donor: report	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
Counterfactual ← irrelevant: knowledge	54.5%	100%	95.5%	57.6%
Irrelevant ← counterfactual: context	78.8%	72.4%	15.2%	72.9%
True fact ← uninformative: knowledge	94.9%	100%	99.5%	70.8%
Uninformative ← true fact: context	84.7%	77.1%	25.9%	59.8%
Counterfactual ← uninformative: knowledge	79.7%	99.2%	94.5%	16.8%
+ reads the false document value	66.7%	90.7%	93.9%	11.9%
Uninformative ← counterfactual: context	84.1%	72.9%	12.3%	61.6%

Question and answer contributions with identical tokens

For these pairs, the recipient’s document and all question and answer tokens stay fixed. Only the cached state of Q, A, or Q+A is replaced. Each three-row group uses the same item subset, allowing a direct comparison of the edited spans. These subsets differ from the broader question-only tests above.

Edited span	OLMo-3-7B	Qwen3-8B	OLMo-3.1-32B	Qwen3-32B
True fact ← uninformative: report becomes knowledge
Q only	97.6%	100%	99.4%	77.6%
A only	10.6%	8.5%	51.9%	7.2%
Q+A	100%	100%	100%	100%
Uninformative ← true fact: report becomes context
Q only	82.4%	84.5%	27.1%	72.4%
A only	1.2%	0%	0%	3.9%
Q+A	100%	97.2%	99.4%	100%
True fact ← irrelevant: report becomes knowledge
Q only	60.9%	100%	100%	97.3%
A only	6.5%	19.2%	60.6%	19.8%
Q+A	100%	100%	100%	100%
Irrelevant ← true fact: report becomes context
Q only	87%	79.8%	32.4%	80.2%
A only	28.3%	0%	0%	2.7%
Q+A	97.8%	99%	100%	100%

Joint Q+A transplants make 97.2–100% of reports follow the donor across the 16 model–direction cells. The effect is nearly complete, rather than universal as on the templated task. The separate-span results again depend on model and direction, supporting the main paper’s use of the joint span. This replication tests document edits and transplants; steering on DynamicQA remains outside the reported experiments.