betaFeedback

WorksComputer Science

Cross-Prompt Reuse of Entity-Replacement Responses

Abstract

With model weights fixed, we measure how much of an entity-replacement response transfers between English factual questions, beyond whether answers change. We add the hidden-state change from two source questions to a target question about the same entity, comparing the predicted log-probability shift between two relations' published answers, given known answer prefixes, with the measured one. Hidden reuse injects the aligned change at 50% depth before the target's remaining layers; direct reuse adds source output-score changes. With other-relation sources, hidden reuse removes 45% and 49% of no-change error on 0.5-1B Qwen and Llama. Gemma-2-2B removes 22% and passes one of three comparisons in plain text without its default beginning-of-sequence (BOS) token. A later protocol restoring it yields 58-68%, passing every comparison on three cohorts. Larger checkpoints, chat format and new entities show replication and format- or cohort-dependent failures. On small plain-format checkpoints, other-relation hidden reuse mainly reproduces the shift both relations' questions share, without reliably lowering relation-selective error. With target-relation sources, hidden reuse's post hoc advantage over direct reuse is 0.2-0.9 nats outside chat format, unconfirmed and not replacement-specific. The same relation-neutral change at both questions' entity tokens lowers relation-selective error more than random or other-replacement changes, indicating receiving-question dependence.

Introduction

Steering and function vectors reuse recorded activation changes across prompts ([51, 46, 50]). Accounts of factual recall likewise predict that mid-depth changes at an entity’s tokens can be read out by another question’s later layers ([17, 41]). A changed answer, however, does not tell us how much of the receiving question’s response has been reproduced. Entity replacement makes that reference measurable: we can replace the entity in the receiving question itself, which we call the target, and compare its actual response with the transferred one. How faithfully can a response recorded in one question reproduce the response in another, and does the transfer also reproduce the part of that response specific to the relation each question asks about?

An entity-replacement response is the hidden-state or output-score change caused by replacing the named entity. We measure its reproduction in answer scores. Replacing SLIB with JDeveloper in a question about SLIB’s developer shifts the log probability of JDeveloper’s published developer answer, Oracle Corporation, against its programming-language answer, Java; each answer is scored as a complete string, token by token after its known preceding tokens. The question about the programming language scores the same answers in the opposite order. A shift toward Oracle Corporation in both questions is shared; a shift of each question toward its own relation’s answer is relation-selective (§ 3). Overall fidelity concerns the full measured contrast change. Each prediction is compared with predicting no change and with transferring a different replacement’s response. Unlike interchange studies, which swap entity representations between prompts and evaluate the resulting answers ([28, 55]), this reference lets us measure what share of the target’s own signed contrast change a transfer reproduces.

We compare two reuse routes that combine the same two source questions with the same similarity weights (Figure 1). Hidden reuse adds the aligned changes at 50% depth to the unedited target and continues its remaining layers; direct reuse adds source output changes. With other-relation sources, which ask the target’s other relation, direct reuse imports that relation’s read-out, the mapping from states to its answers, whereas hidden reuse lets the target’s own computation read the change out. To reproduce the relation-selective part, that computation must then supply each question’s shift toward its own relation’s answer, not merely a shift both share. Target-relation sources ask the target’s own relation, so their read-out matches; we also call them read-out-matched.

Figure 1. Response reuse; model parameters stay fixed. Both routes reuse each source’s 50%-depth change and differ in which read-out, the remaining layers, transforms it: the source’s for direct reuse, the target’s for hidden reuse. (a) The three answer comparisons, pre-specified block sign tests, ask whether hidden reuse from SOS_O has lower error than each of zero prediction, hidden reuse from SO′S'_O and direct reuse from SOS_O. In the answer studies, analyses with read-out-matched sources (STS_T, ST′S'_T) are descriptive, exploratory or post hoc except one failed pre-specified family (§ 4.1). (b) In the source and target shapes, depth increases upward; fills as in (a). The target’s read-out runs twice, with the hidden change (left) and without it (right). Δs,e\Delta_{s,e} is the source’s 50%-depth change from aa to ee, recaptured at each known answer prefix; hidden reuse aligns its token rows to target positions (Equation (2)). Weights wsw_s use only unedited states. c^hid\hat{c}^{\mathrm{hid}}, c^dir\hat{c}^{\mathrm{dir}}: cc from each route’s predicted scores. The block and example (plain format, cue \nAnswer:\n; Appendix § 9) are from the earlier crossing cohort, not the 35-block answer cohort, chosen retrospectively for readable wording, not outcomes (Appendix § 10.§ 1). (c) Mx(A)M_x(A): log probability mass of group AA’s complete answers on input xx; shares measure contrast reproduction, not answer correctness or free generation.

The answer to the first question is a portable but incomplete response. In the plain format, raw text without a chat template or the beginning-of-sequence (BOS) token that Llama and Gemma add by default, other-relation hidden reuse removes 38–51% of the no-change error on Qwen and Llama across the original and two fresh cohorts of new entities, with format- and cohort-dependent failures. Gemma removes −6 to 22% and passes at most one of three comparisons per cohort, but 58–69% with BOS restored, passing every comparison (Table 1 and Table 2). For the second question, the relation-selective part is not reliably reproduced on the small checkpoints in the plain format: hidden reuse there mainly reproduces the shared shift, and descriptive intervals for selective-error reduction never lie wholly above zero, even where overall reproduction is substantial. On the original cohort, hidden reuse removes 49% of plain-format Llama-3.2-1B’s no-change error, yet its selective-error reduction is −11.4% (Table 2). Pooled over the three cohorts, target-relation sources narrow hidden reuse’s advantage over direct reuse from 0.55–5.05 to 0.20–0.90 nats, except for plain-format Gemma (post hoc; Table 18). Yet the same relation-neutral change, added to both questions about a replacement, lowers relation-selective error more than random or other-replacement changes in pre-specified tests, indicating that these selective effects depend on the receiving question.

StudyMaterialBlocksTestsPassLostMain result
Answer confirmationnew3597345/49/22% of no-change error removed
Larger checkpointsreused3597051/46/14% removed
Chat formatreused3518184All comparisons pass, Gemma’s included
First fresh cohortnew33201447–8B selectivity tests pass in family; Gemma passes none
BOS tokenboth9918180Gemma passes every comparison
Second fresh cohortnew3138143Matched-route test fails on all six
Answer depth and rowsreused35descriptiveShare declines from the input
Testbed and read-outreused64–9930203Single vectors trail span alignment
Steering completionreused64–9924134Best vector trails span alignment on five of six
Identical-content controlsreused9912122Neutral content beats random and other-replacement changes
Earlier next-token studiesearlier10–4545363Same beats different replacement

Table 1.

ModelInputOrig.Fresh 1Fresh 2Gap/natsSel. %
Qwen 0.5Bplain45 (3)42 (3)38 (2)0.328.7
chat43 (3)––0.6321.8
Llama 1Bplain49 (3)50 (3)40 (1)0.24−11.4
+BOS57 (–)66 (–)57 (–)0.208.3
chat56 (3)––0.3740.7
Gemma 2Bplain22 (1)2 (0)18 (1)0.29−9.8
+BOS68 (3)65 (3)58 (3)0.2635.9
chat46 (3)––0.3438.3
Qwen 7Bplain51 (3)42 (3)49 (3)0.9050.2
chat35 (3)––1.2037.2
Llama 8Bplain46 (3)49 (3)50 (3)0.4019.7
+BOS41 (–)51 (–)54 (–)0.362.6
chat20 (3)––0.216.7
Gemma 9Bplain14 (1)−6 (0)9 (1)0.45−7.3
+BOS65 (3)69 (3)66 (3)0.5848.7
chat52 (3)––0.2052.1

Table 2. Answer reuse at 1/2 depth. Overall (other-relation sources): % of the no-change error removed by same-replacement hidden reuse in each cohort (35/33/31 blocks; chat: original blocks), with passes of the three pre-specified comparisons in parentheses (–: untested). Gap (target-relation sources): direct minus hidden error, nats, over 99 blocks (chat: 35); positive favors hidden reuse (Table 18 and Table 16b). Sel. (other-relation sources): selective-error reduction from zero prediction (%), original 35 blocks (Table 24). Gap and Sel. are descriptive. Robust shares: Table 14.

Table 18: Source-relation matching at 1/2 depth with suffix interpolation on the same 99 blocks (35 original, 33 first fresh, 31 second fresh), equally weighted within each encoding. Gap: direct minus hidden mean answer error (nats), using other- or target-relation sources for the same replacement. Brackets: 95% label-resampling sensitivity intervals for the target mean gap and its share of favoring blocks (Fav.), not family-wise tests. All six pre-specified matched-route tests on the 31-block second cohort failed (Table 22).

CheckpointInputOther gapTarget gap [95%]Target Fav. [95%]
Qwen 0.5Bplain1.210.32 [0.10, 0.56]0.64 [0.48, 0.79]
Llama 1Bplain0.550.24 [0.03, 0.45]0.69 [0.51, 0.84]
Gemma 2Bplain0.330.29 [−0.09, 0.66]0.58 [0.41, 0.73]
Qwen 7Bplain5.050.90 [0.18, 1.56]0.70 [0.53, 0.84]
Llama 8Bplain2.700.40 [0.17, 0.65]0.73 [0.56, 0.87]
Gemma 9Bplain0.570.45 [0.09, 0.87]0.61 [0.44, 0.77]
Llama 1B+BOS0.840.20 [0.05, 0.37]0.72 [0.57, 0.84]
Gemma 2B+BOS1.970.26 [0.05, 0.45]0.68 [0.51, 0.83]
Llama 8B+BOS2.670.36 [0.11, 0.64]0.73 [0.58, 0.86]
Gemma 9B+BOS4.200.58 [0.26, 0.94]0.79 [0.64, 0.91]

Table 18.

Depth and construction comparisons show where, and in what form, the response remains reusable. Up to 50% depth, transferred changes at the entity tokens remove somewhat less error than the target’s own change there (Figure 2). Span-aligned addition, which aligns changed entity tokens one to one, outperforms the best tested single-vector construction on five of six checkpoints, four under pooled correction across all 223 tests (Table 3). Replications span model scales, input formats and new entities within largely familiar relation/template families.

CheckpointBestSpanGain [95% interval]
Qwen 0.5B13.347.30.85∗ [0.33, 1.62]
Llama 1B38.365.30.66∗ [0.35, 1.05]
Gemma 2B42.165.20.82∗ [0.41, 1.25]
Qwen 7B29.948.51.38∗† [0.18, 2.50]
Llama 8B45.155.50.48 [−0.11, 1.10]
Gemma 9B48.169.81.09∗ [0.50, 1.77]

Table 3. Best single-vector construction at the 1/2 interface, default encoding. The best of 90 single-vector constructions is selected on the original cohort; Best and Span (span-aligned addition) are evaluated on the same 64 already-studied new-entity blocks. Best/Span: zero-prediction error removed (%). Gain: mean vector minus span error (nats), with 95% label-resampled intervals. ∗: passes its pre-specified six-test Holm family (Appendix § 14); †: fails the 223-test pooled correction. The Qwen 0.5B and Gemma 9B comparisons repeat earlier tests on the same blocks (§ 4.3).

Figure 2. Answer-error reduction relative to zero prediction over 35 blocks, default encoding. Solid: span-aligned addition, with other-relation sources unless labelled target-relation; dashed: target-own edited-minus-clean patches, except the horizontal line, other-relation direct reuse; the legend gives row coverage. Vertical: the confirmatory interface; the curves use span-aligned addition, not the suffix interpolation of Table 2. The terminal interface d=1d = 1 precedes final normalization. Appendix § 13 gives numerical summaries and row definitions, and explains why Qwen2.5-7B uses its saved plain-format scan.

Related Work

Cross-prompt reuse of recorded changes. Activation addition and contrastive activation addition transfer prompt-pair differences [51, 46], and representation engineering and inference-time intervention shift activations along directions estimated from many examples [63, 33]; function, task, binding and relation vectors transfer recorded states, function vectors acting when added at early and middle layers [50, 24, 11, 56]. Request patching distinguishes receiving-prompt computation at middle layers from donor-output copying at late layers [52]. States mapped between prompt conditions recover target-prompt behavior [5]; concepts patched across translation prompts are read out in the receiving language [8]. Entity representations also transfer across queries. RAVEL interchanges entity representations between prompts about different attributes or from generic sentences about the entity [28], and Patchscopes and related methods decode subject and other hidden states in other prompts [18, 45, 3]. REMEDI adds learned edits to entity representations to change downstream generation [25]. Wang and Xu Wang and Xu [55] find that subject-representation interchange, including from prompts about another relation, largely matches textual substitution in answer agreement and probabilities, with subject effects in early and middle layers. Their measures differ from ours, the share of the target’s own signed contrast change that is reproduced, so the two results are not directly comparable; § 4.3 compares state replacement with additive entity-row patches under our measure. Knowledge-editing evaluations ask whether an edited fact generalizes to paraphrases [38] and related facts [7]. We instead measure how much of a replacement’s signed complete-answer change transfers additively across questions and separate shared from relation-selective components. Mean-centering improves steering vectors [30], and steering evaluations study reliability and context dependence [49, 32, 59].

Interventions and factual recall. Causal mediation studies internal pathways, including gender bias [53]; causal tracing localizes factual associations [37], though localization need not indicate where edits work best [22]. Causal abstraction formalizes interchange interventions, also on learned subspaces [13, 14, 15]. Factual recall enriches subjects before extracting attributes [17]. Subject lookup does not depend on the preceding context [41], and both relation-agnostic subject heads and relation-specific components contribute additive updates [6, 39]. Multi-token names may form implicit vocabulary items [12], and heads favoring memorized or in-context answers compete and can be rescaled [61, 43]. Lindsey et al. Lindsey et al. [34] insert California features recorded in another prompt into a Dallas prompt, eliciting Sacramento; Agarwal Agarwal [1] finds entity-token patches effective in early and middle layers and final-token patches only in late layers. These accounts predict the mid-depth transfer whose fidelity we quantify. Zhang and Nanda Zhang and Nanda [62] and Heimersheim and Nanda Heimersheim and Nanda [23] discuss patching metrics and interpretation; our equal-length edits follow the former’s symmetric token replacement. Alternative pathways (Malekov et al., 2024) and token entrainment [42] complicate mechanistic interpretations of agreement.

Perturbations and content controls. Interventions can produce off-distribution representations [19]; random norm-matched vectors can shift distributions more than steering vectors [51 App. G], and content-free directions move behaviors similarly [36], which motivates different-replacement and random controls. Complementary work uses stepwise distributional change and uncertainty to diagnose reasoning failures [4]; we measure the fidelity of replacement-induced changes transferred across questions.

Setting and Response Comparisons

Materials and models. A relation is the queried attribute, and a template is its wording with an entity slot. ZsRE, a zero-shot relation extraction dataset, supplies published factual questions, answers and exact relation labels [31]. Each block crosses two entity replacements with two relations, each with two source questions and one target question; within each cohort, relation labels, entities, known aliases, templates and complete inputs are disjoint across blocks, and selection uses factual records, Wikidata entity identities and aliases [54] and the tokenizers, before any model outputs were measured. Table 1 lists the studies, Appendix § 10.§ 4 four earlier next-token studies, and Appendix § 10 eligibility, overlap and study order. The answer results use three cohorts of new-entity blocks whose relation labels and templates mostly recur from earlier studies (Appendix B.2): the original cohort of 35 blocks (140 targets) and two fresh cohorts of 33 and 31 blocks; pooled analyses use all 99 blocks. The earlier 40-block crossing cohort supplies the example of Figure 1, earlier next-token tests and the descriptive comparison that motivated other-relation sources (Appendices § 10.§ 1, § 10.§ 4 and § 10.§ 5).

We use released Qwen2.5-0.5B-Instruct ((Qwen Team, 2024)), Llama-3.2-1B-Instruct ([40]) and Gemma-2-2B-it ([16]), with same-family 7B, 8B and 9B replications. Changes are injected at an interface, the hidden state at a decoder-layer boundary; confirmatory interventions use the interface at 50% depth. Models run deterministically on CPU in float32; construction and scoring use float64. Determinism here holds for fixed inputs and batch shapes. The plain format, fixed by the original protocol before outcomes, omits chat templates and the default Llama/Gemma BOS token. Later protocols compare chat format and plain text with only BOS restored (plain+BOS). The depth scan and the later testbed, read-out, steering-completion and identical-content studies use each checkpoint’s default encoding: plain for Qwen, whose tokenizer adds no BOS, and plain+BOS for Llama and Gemma. Appendix § 10.§ 3 gives the exact checkpoints, interfaces and input formats.

Response reuse

Let Hℓ(x,a)∈RTx×DH_{\ell}(x,a) \in\mathbb{R}^{T_x \times D} collect hidden states for prompt xx with original entity aa (the unedited, or clean, prompt), and let ee denote its replacement. A source supplies the finite-difference response

Δs,e=Hℓ(s,e)−Hℓ(s,a).(1)\Delta_{s,e} = H_{\ell}(s,e) - H_{\ell}(s,a). \tag*{(1)}

Rows (token positions) before the first changed token are zero. In the earlier next-token studies, the matrix Ws→tsuffixW_{s \to t}^{\mathrm{suffix}} aligns the first-changed-to-final region by relative suffix position with linear interpolation, mapping positions between differently worded prompts. The responses of two sources SS yield

Δ^t,e=∑s∈SwsWs→tsuffixΔs,e.(2)\widehat{\Delta}_{t,e} = \sum_{s \in S} w_s W_{s \to t}^{\mathrm{suffix}} \Delta_{s,e}. \tag*{(2)}

The weights are normalized inverse squared distances between unedited final hidden states (Appendix § 9). Hidden reuse adds this prediction to the unedited target states, runs the remaining layers and subtracts the unedited logits (unnormalized next-token scores). Direct reuse instead adds the weighted source output changes to the unedited target scores, using the same sources and weights (Appendix § 9). Each source’s output change is its own hidden change continued through the source’s remaining layers. Other-relation sources SOS_O, used in the confirmatory tests, ask the target’s other relation; target-relation sources STS_T ask its own relation. Different-replacement source sets SO′S'_O and ST′S'_T are recorded for the block’s other replacement under the same source templates.

We call this hidden construction suffix interpolation. In answer evaluation it covers the rendered prompt only, shared candidate-prefix rows align tokenwise, and each candidate prefix gets its own source capture, the source’s change recorded with that prefix appended, rather than one response reused for the whole answer (Appendix § 9). The answer studies’ tests use it; the depth scan and the testbed of standard constructions (§ 4.3) also use span-aligned addition, which copies the changed entity span tokenwise (§ 4.2). Entity replacements preserve token length; source and target questions differ in wording before the entity. Internal, unregistered protocols fixed the confirmatory comparisons, and predictions were saved before edited-target outcomes (Appendix § 10.§ 2).

Answer contrasts

Answer evaluation asks whether reuse reproduces the replacement-induced shift in relative probability between the two relations’ complete answers. For input xx rendered for answer completion (Appendix § 9), we sum complete-string probabilities within each answer group and compare their log probability masses. The fixed groups ATA_T and AOA_O contain published answers for the replacement entity under the target and other relation. In Figure 1’s illustrative crossing-cohort developer question, these are Oracle Corporation (developer) and Java (programming language), both answers for JDeveloper. We score the same groups before and after replacing SLIB with JDeveloper. The group score, answer contrast, replacement-induced change and prediction error are

Mx(A)=log⁡∑y∈Ap(y∣x),M_x(A) = \log\sum_{y \in A} p(y \mid x),
cx=Mx(AT)−Mx(AO),c_x = M_x(A_T) - M_x(A_O),
δc=ct,e−ct,a,\delta c = c_{t,e} - c_{t,a},
Eans=∣δc^−δc∣.(3)E_{\mathrm{ans}} = \lvert\widehat{\delta c} - \delta c \rvert. \tag*{(3)}

Lower error means closer reproduction of the observed signed contrast change; zero prediction, the no-change baseline, predicts δc^=0\widehat{\delta c}=0, and its mean error is the no-change error; a predictor’s share removed is one minus its mean error divided by the no-change error. At each candidate-token position, the recorded prefix-conditioned response predicts the next-token scores. Full-vocabulary softmax probabilities yield complete-string scores and a predicted contrast c^t,e\widehat{c}_{t,e}, Table 1: Studies and their pre-specified test families. Tests: number of tests; Pass: passes under Holm correction within each family; Lost: passes lost when all 223 confirmatory tests form one Holm family (Table 25). Each family was fixed by an internal written protocol before its outcomes, although for Qwen some testbed arms include blocks already observed (Appendix § 10.§ 5). Material: blocks new to the study, reused from earlier studies, or both (earlier: these studies’ own materials, which precede the answer studies). Blocks hold four targets (components for two earlier next-token studies; Table 6). Protocols named one family primary without making the others conditional on it. Two steering-completion tests repeat testbed tests (Appendix § 14). BOS: beginning-of-sequence token.

StudyRoleMain resultTargetsUnits
Single-token comparisonsPre-spec.Identity contrasts recur; derivative contrasts inconclusive6410 components
Multi-token comparisonsPre-spec.Finite sources beat local derivatives18445 components
Replacement/relation crossingPre-spec.Same beats different replacement in 40 of 40 blocks16040 blocks
Native amplitudePre-spec.Route gain largely not replacement-specific14436 blocks

Table 6. Earlier next-token studies in run order: role, main result, and targets and inference units per checkpoint. Blocks contain four targets; components group relations sharing templates.

TestFamily pHolmp_{\mathrm{Holm}}Pooled (223)
Single-token: Qwen same over different replacement0.0120.081
Single-token: Llama same over different replacement0.0120.081
Single-token: Gemma same over different replacement0.0120.081
Answer study, small, plain: Llama 1B over direct reuse0.0050.081
Answer study, small, plain: Llama 1B over zero0.0120.225
Answer study, small, plain: Gemma 2B over direct reuse0.0250.592
Chat format, small: Qwen 0.5B over different9.4×10−49.4 \times10^{-4}0.081
Chat format, larger: Llama 8B over zero0.0080.592
Chat format, larger: Llama 8B over different0.0060.225
Chat format, larger: Gemma 9B over different0.0030.081
First fresh cohort, small: Qwen 0.5B over zero0.0090.175
First fresh cohort, larger: Qwen 7B over zero0.0030.059
First fresh cohort, selectivity: Qwen 7B0.0010.059
First fresh cohort, selectivity: Llama 8B0.0180.947
Second fresh cohort, larger: Gemma 9B over direct reuse0.0160.390
Second fresh cohort, small: Gemma 2B over different0.0320.390
Second fresh cohort, small: Qwen 0.5B over direct reuse0.0120.131
Testbed, span over tuned vector: Llama 8B0.0340.944
Testbed, span over tuned vector: Qwen 7B0.0340.944
Testbed, identical-content selectivity: Llama 1B0.0221.000
Steering, tuned later-row vector over zero: Llama 1B0.0070.144
Steering, tuned later-row vector over zero: Llama 8B0.0040.068
Steering, span over best vector: Qwen 7B0.0030.125
Steering, norm-matched later rows over zero: Llama 1B0.0010.053
Identical content, neutral over random: Llama 1B0.0130.822
Identical content, neutral over other replacement: Llama 1B0.0020.176

Table 25. All 26 conclusions that change under the final 223-test pooled Holm correction: original family-wise and pooled adjusted pp values. Every other test keeps its conclusion; Table 1 gives the study inventory.

written c^t,ehid\hat{c}_{t,e}^{\mathrm{hid}} or c^t,edir\hat{c}_{t,e}^{\mathrm{dir}} for hidden or direct reuse, with δc^=c^t,e−ct,a\delta\hat{c}=\hat{c}_{t,e}-c_{t,a}. This measures contrast reproduction, not answer correctness or free generation. Faithfully predicting movement toward the other relation’s answers need not improve factual recall. Qwen 0.5B’s original plain-format mean error is 2.869 nats for zero prediction and 1.572 for hidden reuse (Table 7): 45.2% of the no-change error removed. In the original small-checkpoint plain format, the median total probability mass of the scored candidates ranges from about 10−8.410^{-8.4} to 10−4.610^{-4.6} (Table 13). Table 4 in Appendix § 9 lists notation and terms.

TermMeaning
Replacement a→ea \to eSwapping the original entity aa for an entity ee with the same token length in a question
Replacement responseHidden-state or output-score difference between the replaced and original question (Equation (1))
Source, targetQuestions whose recorded responses are reused; the question whose response is predicted
Hidden reuseAligned, weighted source hidden responses added at 50% depth and continued through the target’s remaining layers
Direct (output) reuseWeighted source output-score changes added to the target’s unedited scores
Zero predictionPredicts no change; its answer error, the zero or no-change error, is the size of the observed change
Same, different replacementSources recorded for the target’s own replacement, or for the block’s other replacement under the same source templates
Target-, other-relation sourcesSources that ask about the target’s relation, or about the block’s other relation
Native amplitude; row-norm profileMeasured source magnitudes; a prescribed vector length at each token used to match magnitudes
Donor, referenceIn magnitude matching, the response being rescaled and the response supplying row norms
Block; componentFour target cells forming one inference unit; relations linked by shared templates
KL recovery RKLR_{\mathrm{KL}}One minus the prediction’s Kullback–Leibler error relative to the unedited target’s (Equation (7))
cc, δc\delta c, EansE_{\mathrm{ans}}Answer contrast, its replacement-induced change and the absolute prediction error (Equation (3))
ss, hh; s^\hat{s}Selective and shared components of a relation-swapped pair (Equation (4)); a prediction’s selective component, with selective error ∣s^−s∣|\hat{s}-s|. The subscript ss of Equations (1) and (2) instead indexes source questions
Share removedOne minus a predictor’s mean answer error divided by the zero-prediction error

Table 4. Notation and terms. Source sets STS_T and SOS_O ask about the target’s relation and the other relation; primed sets (ST′S'_T, SO′S'_O) are recorded for the block’s other replacement under the same source templates. c^t,ehid\hat{c}^{\mathrm{hid}}_{t,e} and c^t,edir\hat{c}^{\mathrm{dir}}_{t,e} are the contrasts predicted by hidden and direct reuse.

PredictionQwenLlamaGemma
(a) Small checkpoints
Zero2.8692.8212.961†
Same, direct2.9231.8872.858
Same, hidden1.5721.4272.298
Different, hidden3.2693.0593.382†
Share removed, direct−2%33%3%
Share removed, hidden45%49%22%
(b) 7B, 8B and 9B checkpoints
Zero6.9743.9053.385†
Same, direct8.9234.5363.793†
Same, hidden3.4012.1012.923
Different, hidden6.9783.8293.880
Share removed, direct−28%−16%−12%
Share removed, hidden51%46%14%

Table 7. Mean answer-contrast error with other-relation sources in the plain format (nats; lower is better) on 140 targets in 35 equally weighted blocks; zero predicts no change, and same/different denotes replacement identity. Share removed is the fraction of the zero-prediction error removed. †: Holm-adjusted p>0.05p > 0.05 for same-replacement hidden reuse against this baseline.

Endpoint or predictionQwenLlamaGemma
Unedited target−5.689 [−6.980, −4.429]−4.607 [−5.380, −3.568]−8.390 [−11.532, −6.821]
Replaced target−5.744 [−7.164, −4.195]−4.570 [−5.620, −3.421]−8.397 [−11.195, −6.965]
Same/hidden, other−5.625 [−7.149, −4.371]−4.601 [−5.411, −3.350]−8.387 [−11.252, −6.895]
Same/output, other−5.866 [−7.399, −4.494]−4.609 [−5.528, −3.354]−8.692 [−11.158, −6.849]
Different/hidden, other−5.954 [−7.217, −4.466]−4.609 [−5.553, −3.515]−8.542 [−11.021, −6.656]
Different/output, other−6.195 [−7.879, −4.892]−4.875 [−5.983, −3.864]−8.873 [−11.430, −6.872]
Same/hidden, target−5.780 [−7.323, −4.313]−4.641 [−5.555, −3.306]−8.241 [−10.944, −6.873]
Same/output, target−5.909 [−7.402, −4.298]−4.606 [−5.708, −3.340]−8.796 [−11.397, −6.919]
Different/hidden, target−5.747 [−7.169, −4.685]−4.706 [−5.733, −3.462]−8.645 [−11.193, −6.769]
Different/output, target−6.256 [−7.826, −4.871]−4.967 [−6.006, −3.790]−8.956 [−11.764, −7.070]

Table 13. Absolute complete-candidate mass: median log⁡10\log_{10} mass [25th, 75th percentiles] over all 140 targets per small checkpoint in the original 35-block cohort and plain format. Brackets describe the target distribution, not confidence intervals. Other/target denotes source relation. All four nonzero predictions per source relation are retained.

For each replacement, the two target questions ask different relations, so their answer groups are swapped: a shift toward the first relation’s answers raises the first question’s contrast and lowers the second’s. Half the sum of their changes δc1,δc2\delta c_1,\delta c_2 keeps each question’s movement toward its own relation’s answers and cancels a shift shared by both questions; half the difference keeps that shared shift:

s=δc1+δc22,h=δc1−δc22,(4)s=\frac{\delta c_1+\delta c_2}{2},\qquad h=\frac{\delta c_1-\delta c_2}{2}, \tag*{(4)}

the relation-selective component ss and the shared component hh, oriented by the block’s first relation (the subscript ss in Equations (1) and 2 instead indexes sources). For predicted s^\hat{s} and h^\hat{h}, selective error is ∣s^−s∣|\hat{s}-s| and shared error ∣h^−h∣|\hat{h}-h|; selectivity tests ask whether ∣s∣−∣s^−s∣|s|-|\hat{s}-s|, the reduction from zero prediction, is positive.

Statistical analysis. Paired effects, baseline error minus evaluated-prediction error, average the four targets of each block. Exact sign tests, one-sided unless stated, count blocks with positive and negative effects, assessing how broadly improvement recurs; medians of block effects get distribution-free intervals by inverting the sign test, and means with pointwise bootstrap intervals describe magnitude. Holm correction ([26]) adjusts within each pre-specified family at 0.05; Table 1 gives each study’s tests and passes, and passes in the text are within-family. Pre-specified (confirmatory) comparisons were fixed in a written protocol before their outcomes, and one passes when its Holm-adjusted pp is below 0.05; descriptive results report estimates without tests, and post hoc analyses were chosen after outcomes were seen. With 31–35 nonzero blocks, 80% power requires a true positive-sign probability of about 71–80%, depending on family size and Holm rank (Appendix § 11); a failed test does not show equivalence. Pooling all 223 confirmatory tests into one family (pooled correction) would change 26 conclusions (Table 25). Intervals resample whole blocks 20,000 times (Appendix § 11). Cross-cohort tests assume block independence despite reused relation labels. Label-resampled intervals, which resample relation labels, are post hoc sensitivity summaries that widen even for independent blocks, not dependence-adjusted tests; Holm addresses multiplicity, not this dependence (Appendix § 11). Appendices C–G give the answer families and post hoc checks. The answer depth and position scans are descriptive.

Cross-Prompt Response Reuse

We first ask how much of the target’s response transfers, against which comparators, and whether its relation-selective part is reproduced (§ 4.1); we then ask where in depth and token position, and in which constructions, the changes remain reusable (§ 4.2 and § 4.3).

Answer-conditioned response reproduction

On 140 new-entity targets in 35 disjoint blocks of four, absolute answer-contrast error (Equation (3)) is averaged within blocks. Confirmatory sources ask about the other relation, a condition chosen after a descriptive analysis of the crossing cohort in which it separated the routes most clearly, whereas the route-gap intervals with target-relation sources included zero on Llama and Gemma (Appendix § 10.§ 5). Pre-specified families compare, on each checkpoint, same-replacement hidden reuse from SOS_O with direct reuse from SOS_O, zero prediction and different-replacement hidden reuse from SO′S'_O, which checks that predictions depend on the replacement.

How much transfers. In the plain format on the original cohort, hidden reuse removes 45–51% of the no-change error on Qwen and Llama but 14–22% on Gemma (Table 2). Qwen and Llama pass all three comparisons at both scales, whereas Gemma passes one of three at each scale. With the BOS token restored, both Gemma checkpoints remove 58–69% across the three cohorts and pass every comparison; the token also enlarges the observed changes, so shares are not directly comparable across formats (Table 21 and Table 23). In each checkpoint’s chat template all eighteen comparisons pass (Table 20). On two fresh cohorts with new entities, Qwen2.5-7B and Llama-3.1-8B pass every comparison; the small checkpoints pass none against zero prediction on the second in the plain format, and plain-format Gemma at most one per cohort (Table 20 and Table 21).

CheckpointOver direct reuse: SignsOver direct reuse: GainOver direct reuse: pHolmp_{\mathrm{Holm}}Over zero: SignsOver zero: GainOver zero: pHolmp_{\mathrm{Holm}}Over different: SignsOver different: GainOver different: pHolmp_{\mathrm{Holm}}
Chat format, small checkpoints
Qwen 0.5B29/61.921.2×10−41.2 \times10^{-4}30/51.834.5×10−54.5 \times10^{-5}27/82.039.4×10−49.4 \times10^{-4}
Llama 1B30/52.144.5×10−54.5 \times10^{-5}32/32.441.3×10−61.3 \times10^{-6}34/13.439.4×10−99.4 \times10^{-9}
Gemma 2B32/34.241.3×10−61.3 \times10^{-6}33/22.971.5×10−71.5 \times10^{-7}33/23.541.5×10−71.5 \times10^{-7}
Chat format, larger checkpoints
Qwen 7B32/38.991.7×10−61.7 \times10^{-6}28/73.080.00130/54.406.7×10−56.7 \times10^{-5}
Llama 8B31/46.031.2×10−51.2 \times10^{-5}25/101.450.00826/92.040.006
Gemma 9B34/112.139.4×10−99.4 \times10^{-9}28/74.890.00127/85.560.003
First fresh cohort, small checkpoints
Qwen 0.5B29/41.224.4×10−54.4 \times10^{-5}25/81.050.00929/41.634.4×10−54.4 \times10^{-5}
Llama 1B28/50.581.7×10−41.7 \times10^{-4}29/41.314.4×10−54.4 \times10^{-5}30/31.176.3×10−66.3 \times10^{-6}
Gemma 2B15/18−0.06-0.061.000†1.000^{\dagger}15/180.051.000†1.000^{\dagger}21/120.580.244†0.244^{\dagger}
First fresh cohort, larger checkpoints
Qwen 7B27/64.758.1×10−48.1 \times10^{-4}26/73.120.00328/53.472.0×10−42.0 \times10^{-4}
Llama 8B29/43.004.4×10−54.4 \times10^{-5}29/42.004.4×10−54.4 \times10^{-5}31/22.185.9×10−75.9 \times10^{-7}
Gemma 9B18/150.120.728†0.728^{\dagger}15/18−0.17-0.170.757†0.757^{\dagger}22/110.420.120†0.120^{\dagger}

Table 20. The chat-format and first-fresh-cohort nine-comparison families: same-replacement other-relation hidden reuse at the 1/2 interface against direct reuse, zero prediction and different-replacement hidden reuse. Signs count positive/negative blocks (35 blocks in the chat format, 33 in the first fresh cohort); gains are mean block error reductions (nats); one-sided exact sign tests are Holm-adjusted within each family. †: pHolm>0.05p_{\mathrm{Holm}} > 0.05.

Table 21: Answer-comparison families of Appendix D: same-replacement other-relation hidden reuse at the 1/2 interface against direct reuse, zero prediction and different-replacement hidden reuse. The plain+BOS families cover the two Gemma checkpoints, one six-test family per cohort; the second fresh cohort has one nine-test family per scale. Signs count positive/negative blocks; gains are mean block error reductions (nats); one-sided exact sign tests are Holm-adjusted within each family. †: pHolm>0.05p_{\mathrm{Holm}} > 0.05.

CheckpointOver direct reuse: SignsOver direct reuse: GainOver direct reuse: pHolmp_{\mathrm{Holm}}Over zero: SignsOver zero: GainOver zero: pHolmp_{\mathrm{Holm}}Over different: SignsOver different: GainOver different: pHolmp_{\mathrm{Holm}}
Plain+BOS, original blocks (35 blocks)
Gemma 2B32/31.634.2×10−74.2 \times10^{-7}35/02.921.7×10−101.7 \times10^{-10}32/32.934.2×10−74.2 \times10^{-7}
Gemma 9B34/14.084.2×10−94.2 \times10^{-9}34/13.004.2×10−94.2 \times10^{-9}35/03.201.7×10−101.7 \times10^{-10}
Plain+BOS, first fresh cohort (33 blocks)
Gemma 2B30/32.211.4×10−61.4 \times10^{-6}31/22.352.0×10−72.0 \times10^{-7}29/42.165.5×10−65.5 \times10^{-6}
Gemma 9B32/14.262.0×10−82.0 \times10^{-8}32/13.282.0×10−82.0 \times10^{-8}33/03.237.0×10−107.0 \times10^{-10}
Plain+BOS, second fresh cohort (31 blocks)
Gemma 2B29/22.106.9×10−76.9 \times10^{-7}28/31.994.6×10−64.6 \times10^{-6}30/12.258.9×10−88.9 \times10^{-8}
Gemma 9B30/14.278.9×10−88.9 \times10^{-8}28/33.494.6×10−64.6 \times10^{-6}30/13.168.9×10−88.9 \times10^{-8}
Plain format, second fresh cohort (31 blocks), small checkpoints
Qwen 0.5B24/71.040.01222/90.940.074†0.074^{\dagger}25/61.290.004
Llama 1B22/90.640.074†0.074^{\dagger}21/100.880.074†0.074^{\dagger}28/31.572.1×10−52.1 \times10^{-5}
Gemma 2B22/90.490.074†0.074^{\dagger}20/110.440.075†0.075^{\dagger}23/81.000.032
Plain format, second fresh cohort (31 blocks), larger checkpoints
Qwen 7B27/44.851.0×10−41.0 \times10^{-4}27/43.651.0×10−41.0 \times10^{-4}28/33.412.1×10−52.1 \times10^{-5}
Llama 8B28/32.672.1×10−52.1 \times10^{-5}28/32.132.1×10−52.1 \times10^{-5}27/42.281.0×10−41.0 \times10^{-4}
Gemma 9B23/80.700.01617/140.280.720†0.720^{\dagger}17/140.600.720†0.720^{\dagger}

Table 21.

CheckpointCohortZero errorMassShareSelectiveNew firstSigns, plain+BOS
Llama 1BOriginal2.82 / 2.81−4.61 / −4.6149.4 / 56.5−11.4 / 8.382 / 7627/8, 31/4, 31/4
First fresh2.63 / 2.62−4.86 / −4.7650.0 / 66.021.2 / 41.465 / 6629/4, 30/3, 31/2
Second fresh2.22 / 2.25−4.39 / −4.4439.6 / 56.50.4 / 18.452 / 6028/3, 28/3, 29/2
Gemma 2BOriginal2.96 / 4.31−8.39 / −4.5322.4 / 67.8−9.8 / 35.942 / 9132/3, 35/0, 32/3
First fresh2.53 / 3.63−8.97 / −4.312.1 / 64.6−32.9 / 52.036 / 8330/3, 31/2, 29/4
Second fresh2.49 / 3.45−8.25 / −4.1617.8 / 57.6−8.8 / 40.038 / 7429/2, 28/3, 30/1
Llama 8BOriginal3.90 / 4.21−3.09 / −3.3746.2 / 40.819.7 / 2.6101 / 10029/6, 30/5, 30/5
First fresh4.07 / 4.40−2.98 / −3.1649.3 / 50.835.0 / 26.497 / 10128/5, 29/4, 29/4
Second fresh4.23 / 4.90−2.75 / −3.0050.4 / 53.834.1 / 35.789 / 9529/2, 25/6, 26/5
Gemma 9BOriginal3.38 / 4.62−8.05 / −4.6613.6 / 65.0−7.3 / 48.749 / 9934/1, 34/1, 35/0
First fresh2.71 / 4.76−8.36 / −4.45−6.3 / 68.8−21.9 / 57.241 / 9732/1, 32/1, 33/0
Second fresh3.00 / 5.30−7.87 / −4.299.5 / 65.8−23.5 / 54.338 / 8530/1, 28/3, 30/1

Table 23. Plain format against plain+BOS on the three cohorts (descriptive except the Gemma signs, which are the tests of Table 21). Each pair gives plain / plain+BOS values of the mean zero-prediction error (nats), the median log⁡10\log_{10} candidate mass, the share of the zero error removed by same-replacement other-relation hidden reuse (%), the reduction in selective absolute error relative to zero (%), and the number of targets whose highest-scoring edited candidate is a published target-relation answer of the replacement (of 140, 132 and 124). The last column gives plain+BOS block signs over direct reuse, zero prediction and different-replacement hidden reuse; Llama’s are not tested.

The shares are ratios of means of right-skewed block gains; over the 99 blocks the median block loses 32–51% of its no-change error on Qwen and Llama in the plain format and 61–64% on Gemma with the BOS token (Table 14). Excluding targets whose answers share a word with an entity name lowers plain-format Gemma’s pooled shares to near zero but changes the other pooled shares by at most five points (Table 14). Without the blocks containing those targets, every Qwen and Llama comparison in the original plain format still passes (post hoc; Table 16a), but plain-format Gemma-2-2B’s sole original pass, against direct reuse, no longer does (pHolm=0.025p_{\mathrm{Holm}} = 0.025 to 0.0780.078; Table 8 and Table 16a).

ModelBaselineMean95% intervalSignspHolmp_{\mathrm{Holm}}
QwenOutput1.351[0.882, 1.840]30/5/01.01×10−41.01 \times10^{-4}
QwenZero1.297[0.655, 2.073]28/7/01.52×10−31.52 \times10^{-3}
QwenDifferent hidden1.696[1.107, 2.448]29/6/04.09×10−44.09 \times10^{-4}
LlamaOutput0.459[0.163, 0.767]27/8/04.70×10−34.70 \times10^{-3}
LlamaZero1.393[0.649, 2.295]26/9/01.20×10−21.20 \times10^{-2}
LlamaDifferent hidden1.632[0.875, 2.570]30/5/01.01×10−41.01 \times10^{-4}
GemmaOutput0.560[0.139, 0.977]25/10/02.50×10−22.50 \times10^{-2}
GemmaZero0.663[0.077, 1.363]21/14/01.55×10−11.55 \times10^{-1}
GemmaDifferent hidden1.084[0.465, 1.822]23/12/08.95×10−28.95 \times10^{-2}

Table 8. Pre-specified answer-contrast error improvement over each baseline, in nats. Positive means lower same-replacement/other-relation hidden error. Signs give positive/negative/zero block counts. Intervals describe means pointwise; the corrected sign tests assess prevalence of positive block effects.

Table 14. Robust summaries of the shares in Table 2 (post hoc; no tests). For same-replacement hidden reuse with other-relation sources: the median over blocks of each block’s share of its no-change error removed (Med.), the share after excluding targets whose answers share a word with an entity name (No ov.; 22, 8 and 14 targets in 8, 3 and 5 blocks of the three cohorts), and, over all 99 blocks, the share of Table 2 with an interval resampling blocks within cohorts. Brackets on medians are distribution-free intervals from inverting the sign test (coverage at least 0.95). Chat rows cover the original blocks.

ModelInputOriginal Med.Original No ov.First fresh Med.First fresh No ov.Second fresh Med.Second fresh No ov.All 99 blocks ShareAll 99 blocks Med.All 99 blocks No ov.
Qwen 0.5Bplain35.135.332.340.030.235.741.9 [31.8, 50.5]32.3 [22.5, 47.4]37.1
Qwen 0.5Bchat44.842.1–––––––
Llama 1Bplain39.946.147.248.511.732.746.9 [38.0, 54.6]39.9 [29.8, 46.8]43.4
Llama 1B+BOS46.252.464.863.552.756.659.7 [52.9, 65.6]55.7 [46.2, 62.6]57.7
Llama 1Bchat52.953.8–––––––
Gemma 2Bplain20.57.8−6.1−9.119.315.314.6 [3.9, 24.4]11.6 [−7.4, 22.0]4.3
Gemma 2B+BOS63.165.460.163.260.656.163.9 [58.9, 68.2]61.3 [55.2, 64.5]61.8
Gemma 2Bchat46.150.7–––––––
Qwen 7Bplain55.654.245.538.951.150.947.5 [41.6, 52.9]51.3 [42.1, 55.6]47.8
Qwen 7Bchat35.933.9–––––––
Llama 8Bplain40.843.040.947.447.354.748.6 [42.7, 54.0]42.3 [36.1, 51.5]48.5
Llama 8B+BOS38.540.646.148.155.755.248.5 [42.6, 53.9]45.2 [38.5, 54.9]48.3
Llama 8Bchat19.221.8–––––––
Gemma 9Bplain12.15.3−26.6−18.83.77.56.4 [−4.5, 16.3]6.7 [−4.2, 14.7]−1.5
Gemma 9B+BOS64.663.066.166.659.268.066.5 [62.2, 70.4]63.7 [56.6, 69.7]66.0
Gemma 9Bchat51.654.6–––––––
CheckpointVersus directVersus zeroVersus different hidden
Qwen 0.5B22/5; 6.1 × 10−310^{-3}20/7; 0.03822/5; 6.1 × 10−310^{-3}
Llama 1B22/5; 6.1 × 10−310^{-3}21/6; 0.01524/3; 2.2 × 10−410^{-4}
Gemma 2B19/8; 0.07813/14; 0.70115/12; 0.701
Qwen 7B24/3; 1.7 × 10−410^{-4}26/1; 1.9 × 10−610^{-6}24/3; 1.7 × 10−410^{-4}
Llama 8B25/2; 2.3 × 10−510^{-5}22/5; 3.0 × 10−310^{-3}24/3; 1.7 × 10−410^{-4}
Gemma 9B16/11; 0.24817/10; 0.24822/5; 3.0 × 10−310^{-3}
CheckpointHidden shareDirect shareOther hidden over target directTarget hidden over target direct
Qwen 0.5B42.5−2.1−0.06 [−0.51, 0.39] 15/200.63 [0.25, 1.00] 22/13
Llama 1B56.37.0−0.07 [−0.44, 0.28] 16/190.37 [0.03, 0.68] 26/9
Gemma 2B46.4−19.9−1.08 [−1.60, −0.60] 11/240.34 [−0.11, 0.75] 23/12
Qwen 7B35.2−67.6−0.88 [−2.12, 0.35] 16/191.20 [0.46, 1.94] 27/8
Llama 8B20.0−63.4−3.08 [−4.29, −2.00] 8/270.21 [−0.24, 0.64] 20/15
Gemma 9B52.4−77.5−1.60 [−2.43, −0.79] 9/260.20 [−0.50, 0.98] 21/14
CheckpointSpan over suffix interpolationSpan E over vector at each E rowNative broadcast share (%)
Qwen 0.5B0.20 [0.13, 0.27]0.45 [0.25, 0.67]−239.8
Llama 1B0.12 [0.06, 0.19]0.33 [0.13, 0.54]−198.4
Gemma 2B0.12 [0.05, 0.20]0.30 [0.08, 0.53]−103.4
Qwen 7B0.28 [0.07, 0.50]0.49 [0.12, 0.87]−48.7
Llama 8B0.18 [0.09, 0.28]0.30 [0.07, 0.55]−82.0
Gemma 9B0.14 [0.03, 0.26]0.37 [0.18, 0.56]−38.2

Table 16. Additional comparisons for Sections 4.1 and 4.3, all at 1/2 depth. (a) Original plain-format overlap sensitivity: positive/negative signs and Holm-adjusted pp after removing the eight name-overlap blocks, leaving 27; the three baselines on three checkpoints retain one nine-test Holm family per scale. This reduced analysis is post hoc. (b) Original 35-block chat cohort: the share of no-change error removed by other-relation hidden or direct reuse (%); negative shares mean increased error. The last two columns are target-relation direct error minus other-relation or target-relation hidden error (nats; pointwise 95% block-bootstrap interval; positive/negative blocks), so negative values favor matched-source direct reuse; the last column gives the chat-format gaps of Table 2. Neither route comparison was pre-specified. (c) Descriptive paired error advantages (nats; pointwise 95% intervals resampling blocks within cohorts) over all 99 blocks in default encoding, with other-relation sources and coefficient 1 for the entity-row vector. The last column gives the native-scale vector’s share of no-change error removed when broadcast from the entity onward; negative shares mean larger mean error than zero prediction. E denotes entity rows. (a) Name-overlap exclusions: signs; pHolmp_{\mathrm{Holm}}. (b) Chat-format comparators, original cohort. (c) Coverage and alignment comparisons, 99 blocks.

At the answer cue, before any candidate token is scored, hidden reuse reduces the full-vocabulary Kullback–Leibler divergence from the edited target by 88–97% relative to the unedited target in each checkpoint’s default encoding, against 44–87% for other-relation direct reuse (Table 28). In greedy generation, which follows its own tokens, one question-plus-cue capture (one source recording at the question and cue, without answer tokens) with span-aligned addition from other-relation sources reproduces the edited answer string for 5–23% of targets within 16 tokens (Table 29); construction, capture and scope differ from candidate scoring, so the gap does not isolate autoregressive error accumulation.

Table 28: Full-vocabulary readouts at the answer cue, whose next-token distribution does not depend on the candidate set: KL recovery, one minus the mean block KL divergence from the edited target over that of the clean target (%), and agreement of the most probable token with the edited target’s (%), over the 99 blocks of three cohorts in each checkpoint’s default encoding. Constructions use other-relation sources unless marked target (Tgt.); one vector (Vec.) at native scale. Interp.: suffix interpolation; Span: span-aligned addition; Repl.: entity-state replacement; Dir.: direct output reuse; E: entity rows; later: every row from the entity onward. The clean target’s top-token agreement is the baseline.

Interp.Span allSpan ERepl. EVec. EVec. laterTgt. allDir. oth.Dir. tgt.Clean
KL recovery
Qwen 0.5B8891898942−42924480–
Llama 1B97989797256988793–
Gemma 2B949593934514976788–
Qwen 7B949594943233967688–
Llama 8B949591923836977290–
Gemma 9B959794945251986990–
Top-token agreement
Qwen 0.5B75777676523081466337
Llama 1B88899189322590647020
Gemma 2B77787575552684506944
Qwen 7B76787475211577455613
Llama 8B85868080323087516919
Gemma 9B84858284393189516622

Table 28.

Table 29: Greedy generation of up to 16 tokens after the answer cue, over 396 targets of three cohorts in each checkpoint’s default encoding. Changes are added at the 1/2 interface: span-aligned entity, question and cue rows from one capture (Span, other or target relation), replaced entity states (Repl.), or one vector at every row from the entity on, generated rows included (Vec.). Changed: the edited target’s answer string, the text before the first line break, differs from the clean target’s (%). Exact: answer string identical to the edited target’s (%). In set: greedy answer equal to a scored candidate (%). Contains (post hoc): answer contains a published target-relation answer of the replacement, for the edited target (% of targets, count) and, among those targets, for the Span oth. and Vec. predictions and for the clean target (%).

CheckpointChangedSpan oth.Span tgt.Repl.Vec.CleanEditedEditedSpan oth.Vec.Clean
ExactIn setContains
Qwen 0.5B9618181727611 (42)812931
Llama 1B10055500015 (60)283227
Gemma 2B9617211436721 (82)662921
Qwen 7B9867514418 (72)432814
Llama 8B981011847939 (154)463819
Gemma 9B97232622891131 (123)674118

Table 29.

Against which comparators. How much hidden reuse gains over direct reuse depends on which relation the sources ask. With other-relation sources, direct reuse imports that relation’s read-out: in the plain format it removes almost none of the no-change error on the small Qwen and Gemma checkpoints and adds 12–28% to it at 7–9B; in the chat format it adds 63–78% at 7–9B (Table 7 and Table 16b). With the three cohorts pooled, using target-relation instead of other-relation sources narrows the hidden-over-direct gap from 0.55–5.05 to 0.20–0.90 nats, except for plain-format Gemma (Table 18); this switch changes the recorded response as well as its read-out. Target-relation direct reuse can also outperform other-relation hidden reuse: on the original blocks in chat format it does so on Gemma-2-2B, Llama-3.1-8B and Gemma-2-9B, by up to 3.08 nats with pointwise intervals excluding zero (Table 16b).

With target-relation sources, the residual advantage of hidden reuse is unconfirmed. It is 0.24–0.90 nats pooled over the three plain-format cohorts and 0.20–0.58 nats with the BOS token (Table 18; post hoc). In chat format on the original blocks it is 0.20–1.20 nats, with pointwise intervals including zero on Gemma-2-2B, Llama-3.1-8B and Gemma-2-9B (Table 16b; post hoc). On three plain-format checkpoints, the label-resampled interval for the mean gap includes zero or the interval for the share of favoring blocks includes one half (Table 18). After each route is rescaled, the advantage is 0.11–0.75 nats in the plain format on the fresh cohorts; Gemma-2-2B’s interval includes zero (Table 15). A pre-specified test on the 31-block second cohort failed on all six checkpoints (Table 22). Nor is the route advantage replacement-specific: with target-relation sources, a different replacement gains more from the hidden route than the correct one on every checkpoint in the plain and plain+BOS formats, by 0.24–1.72 nats over the 99 blocks with intervals excluding zero in eight of the ten cases, consistent with the target attenuating wrong injected changes (Table 15).

Table 15. Route comparisons that allow for prediction magnitude and replacement identity (nats; post hoc). Unscaled and Rescaled: mean block error of direct minus hidden reuse (positive favors hidden reuse) on the two fresh cohorts (64 blocks), without and with each route’s predictions multiplied by its least-squares factor fitted on the original cohort. Interaction: over all 99 blocks, the same-replacement route gain minus the different-replacement route gain; positive values mean a larger route gain for the same replacement. Brackets are 95% intervals resampling blocks within cohorts.

SourcesModelInputUnscaledRescaledInteraction
Other relationQwen 0.5Bplain1.14 [0.80, 1.52]0.74 [0.47, 1.04]0.45 [0.07, 0.86]
Other relationLlama 1Bplain0.61 [0.41, 0.82]0.50 [0.32, 0.69]−0.34 [−0.77, 0.05]
Other relationGemma 2Bplain0.20 [−0.10, 0.51]0.13 [−0.07, 0.32]−0.22 [−0.56, 0.10]
Other relationQwen 7Bplain4.80 [3.82, 5.79]2.93 [2.26, 3.59]1.89 [0.69, 3.05]
Other relationLlama 8Bplain2.84 [2.23, 3.48]1.75 [1.37, 2.15]0.57 [−0.10, 1.22]
Other relationGemma 9Bplain0.40 [0.09, 0.74]0.09 [−0.13, 0.32]−0.21 [−0.67, 0.23]
Other relationLlama 1B+BOS0.93 [0.73, 1.14]0.80 [0.61, 1.00]−0.40 [−0.86, −0.01]
Other relationGemma 2B+BOS2.15 [1.65, 2.70]1.74 [1.30, 2.22]0.71 [0.08, 1.31]
Other relationLlama 8B+BOS2.86 [2.24, 3.50]1.81 [1.40, 2.23]0.40 [−0.36, 1.11]
Other relationGemma 9B+BOS4.27 [3.41, 5.17]2.86 [2.28, 3.47]1.72 [0.78, 2.66]
Target relationQwen 0.5Bplain0.23 [0.08, 0.39]0.24 [0.08, 0.41]−0.42 [−0.78, −0.06]
Target relationLlama 1Bplain0.28 [0.12, 0.43]0.27 [0.12, 0.42]−0.72 [−1.09, −0.39]
Target relationGemma 2Bplain0.14 [−0.10, 0.37]0.11 [−0.07, 0.29]−0.30 [−0.64, 0.03]
Target relationQwen 7Bplain0.82 [0.33, 1.29]0.75 [0.31, 1.18]−1.72 [−2.81, −0.69]
Target relationLlama 8Bplain0.44 [0.26, 0.63]0.40 [0.23, 0.57]−1.45 [−2.05, −0.92]
Target relationGemma 9Bplain0.36 [0.13, 0.60]0.22 [0.05, 0.39]−0.24 [−0.70, 0.19]
Target relationLlama 1B+BOS0.26 [0.14, 0.38]0.24 [0.12, 0.36]−0.96 [−1.41, −0.57]
Target relationGemma 2B+BOS0.26 [0.11, 0.41]0.25 [0.10, 0.40]−0.89 [−1.39, −0.46]
Target relationLlama 8B+BOS0.36 [0.19, 0.55]0.38 [0.22, 0.55]−1.66 [−2.26, −1.14]
Target relationGemma 9B+BOS0.55 [0.34, 0.76]0.61 [0.41, 0.82]−1.72 [−2.50, −1.00]

Table 22: Neutral-source and read-out-matched route families on the second fresh cohort (31 blocks, plain format). Other over neutral compares same-replacement hidden reuse from other-relation sources with hidden reuse from the two relation-neutral sources; neutral over zero compares the neutral sources with zero prediction (one family of six per scale). The matched route compares hidden with direct reuse when both use target-relation sources (one family of six). Signs, gains (nats) and Holm adjustment as in Table 21. †: pHolm>0.05p_{\mathrm{Holm}} > 0.05.

CheckpointOther over neutral: SignsOther over neutral: GainOther over neutral: pHolmp_{\mathrm{Holm}}Neutral over zero: SignsNeutral over zero: GainNeutral over zero: pHolmp_{\mathrm{Holm}}Matched route: SignsMatched route: GainMatched route: pHolmp_{\mathrm{Holm}}
Qwen 0.5B21/100.400.142†0.142^{\dagger}19/120.540.422†0.422^{\dagger}21/100.380.212†0.212^{\dagger}
Llama 1B22/90.450.088†0.088^{\dagger}19/120.430.422†0.422^{\dagger}19/120.180.422†0.422^{\dagger}
Gemma 2B22/90.470.088†0.088^{\dagger}19/12−0.030.422†0.422^{\dagger}14/17−0.070.763†0.763^{\dagger}
Qwen 7B18/130.350.710†0.710^{\dagger}26/53.305.8×10−45.8 \times10^{-4}21/100.410.212†0.212^{\dagger}
Llama 8B19/120.220.562†0.562^{\dagger}25/61.910.00221/100.340.212†0.212^{\dagger}
Gemma 9B14/170.071.000†1.000^{\dagger}16/150.221.000†1.000^{\dagger}18/130.510.473†0.473^{\dagger}

Table 22.

What transfers. On the original plain-format blocks, the selective component ss carries 9–38% of the observed variance, less on the small checkpoints than at 7–9B; the rest is shared between the questions (Table 11). Hidden reuse cuts shared error by 34–62% on the small checkpoints (Table 12). Descriptive pointwise 95% intervals for selective-error reduction are never wholly positive on the small plain-format checkpoints in any cohort; on the original cohort they are positive for plain Qwen2.5-7B and Llama-3.1-8B, Gemma with BOS, and every chat checkpoint except Llama-3.1-8B (Table 24). The pre-specified 7–8B selectivity tests followed up these descriptive reductions: both passed on the first fresh cohort, neither under pooled correction, and only Llama-3.1-8B passed on the second (Table 24). A constant push toward each question’s own answers, predicting the same small positive ss for every pair, would pass all four tests, since most observed ss are positive. On these two checkpoints in the fresh cohorts, same-replacement hidden reuse lowers selective error by 34–36%, whereas different-replacement hidden reuse lowers it by only −9 to 5% (Table 24).

CheckpointQwen 0.5BLlama 1BGemma 2BQwen 7BLlama 8BGemma 9B
Selective share20.68.719.138.323.533.3

Table 11. Selective share of observed contrast variance on the original 35 blocks, plain format (%). Balanced relation-swapped pairing uses Var⁡(δc)=Var⁡(s)+E[h2]\operatorname{Var}(\delta c) = \operatorname{Var}(s) + \mathbb{E}[h^2]; the share is Var⁡(s)/Var⁡(δc)\operatorname{Var}(s) / \operatorname{Var}(\delta c), not an attribution of the primary absolute-error gain. The complementary share is shared between questions. Each checkpoint is evaluated on the same original cohort.

SourceModelPredictionrsr_srhr_hMAEs\mathrm{MAE}_sMAEh\mathrm{MAE}_h
OtherQwenSame/hidden0.3000.9171.1811.076
OtherQwenSame/output−0.4990.8852.5381.508
OtherQwenDifferent/hidden0.0360.1381.4992.785
OtherQwenDifferent/output0.0170.0392.0573.154
OtherQwenZero——1.2932.571
OtherLlamaSame/hidden0.5390.9381.0211.046
OtherLlamaSame/output−0.0690.9391.5241.066
OtherLlamaDifferent/hidden0.1480.0351.1232.891
OtherLlamaDifferent/output0.198−0.3781.2883.684
OtherLlamaZero——0.9162.720
OtherGemmaSame/hidden0.3830.8281.5621.673
OtherGemmaSame/output−0.2660.7242.1461.958
OtherGemmaDifferent/hidden0.2240.0041.5822.943
OtherGemmaDifferent/output−0.097−0.1961.9723.872
OtherGemmaZero——1.4232.552
TargetQwenSame/hidden0.8530.9400.7710.900
TargetQwenSame/output0.6860.9191.1971.194
TargetQwenDifferent/hidden0.0840.1061.5482.868
TargetQwenDifferent/output−0.0320.0311.9423.027
TargetQwenZero——1.2932.571
TargetLlamaSame/hidden0.6110.9430.7961.009
TargetLlamaSame/output0.3480.9471.1130.952
TargetLlamaDifferent/hidden0.2130.0781.0412.860
TargetLlamaDifferent/output−0.180−0.3711.5483.669
TargetLlamaZero——0.9162.720
TargetGemmaSame/hidden0.4130.8651.3711.443
TargetGemmaSame/output0.3620.7331.8161.962
TargetGemmaDifferent/hidden0.108−0.0571.6203.052
TargetGemmaDifferent/output0.262−0.2721.7573.913
TargetGemmaZero——1.4232.552

Table 12. Paired contrast components: selective ss and shared hh, 70 relation-swapped pairs per checkpoint, retained in 35 four-target blocks. Errors are in nats; zero has undefined correlation. Correlations use the fixed pair orientation described above.

Table 24. Selectivity checks of same-replacement other-relation hidden reuse (post hoc; no tests). For each listed cohort, encoding and checkpoint: the share of relation-swapped pairs whose observed selective component ss is positive (%); blocks favoring / opposing a vanishingly small constant s^>0\hat{s} > 0 under the selectivity statistic ∣s∣−∣s^−s∣|s| - |\hat{s} - s| (Push; blocks whose two pairs differ in sign tie); the block signs of that statistic for hidden reuse (Hidden); the reduction in selective absolute error relative to zero prediction (%) of same-replacement (Same) and different-replacement (Diff.) hidden reuse, with 95% block-bootstrap intervals; and the mean block advantage in selective absolute error of the same over the different replacement (nats), with its interval and favoring / opposing blocks. In the pre-specified two-test selectivity families (plain format, Qwen 7B and Llama 8B), the constant push would give Holm-adjusted pp of 0.011 and 0.002 on the first fresh cohort and 0.029 and <0.001 on the second; hidden reuse gives 0.001 and 0.018, and 0.075 and <0.001. The final panel reports the original 35-block chat selective-error reductions with pointwise 95% block-bootstrap intervals; these are descriptive, not multiplicity-adjusted tests.

Checkpoints>0s > 0PushHiddenSameDiff.Same over different
Original blocks, plain format
Qwen 0.5B52.911/919/168.7 [−13.5, 28.9]−15.9 [−32.9, 1.7]0.32 [0.01, 0.63] 21/14
Llama 1B58.615/914/21−11.4 [−42.7, 9.9]−22.6 [−46.9, −3.2]0.10 [−0.10, 0.32] 17/18
Gemma 2B50.08/814/21−9.8 [−33.9, 10.3]−11.2 [−28.1, 3.8]0.02 [−0.31, 0.36] 19/16
Qwen 7B62.915/627/850.2 [36.8, 60.8]3.5 [−16.8, 20.2]2.16 [1.28, 3.06] 27/8
Llama 8B74.322/523/1219.7 [1.5, 34.2]−10.5 [−28.3, 3.0]0.65 [0.31, 1.00] 28/7
Gemma 9B48.69/1016/19−7.3 [−25.5, 11.1]−7.4 [−28.6, 11.5]0.00 [−0.59, 0.56] 20/15
First fresh cohort, plain format
Qwen 0.5B62.113/523/1017.2 [−1.6, 32.9]−12.6 [−33.5, 5.8]0.44 [0.05, 0.85] 21/12
Llama 1B53.011/920/1321.2 [−0.4, 38.3]0.1 [−19.0, 14.5]0.26 [−0.03, 0.55] 18/15
Gemma 2B54.58/57/26−32.9 [−58.3, −7.3]−10.0 [−35.0, 10.2]−0.32 [−0.80, 0.09] 17/16
Qwen 7B65.213/326/735.1 [17.7, 49.0]−9.4 [−30.7, 7.2]1.99 [0.97, 3.05] 24/9
Llama 8B74.220/423/1035.0 [21.4, 46.5]−1.8 [−21.7, 15.2]0.93 [0.50, 1.37] 27/6
Gemma 9B43.95/915/18−21.9 [−47.1, −1.5]−8.8 [−26.6, 7.6]−0.20 [−0.64, 0.23] 15/18
Second fresh cohort, plain format
Qwen 0.5B66.115/523/821.8 [−0.4, 40.7]−7.3 [−27.6, 8.8]0.39 [0.05, 0.74] 22/9
Llama 1B56.18/718/130.4 [−28.4, 21.0]−34.7 [−68.1, −9.5]0.29 [0.03, 0.57] 19/12
Gemma 2B41.95/1014/17−8.8 [−30.6, 8.9]−18.6 [−44.5, 2.2]0.13 [−0.20, 0.45] 19/12
Qwen 7B62.911/320/1136.5 [14.0, 52.8]3.6 [−18.2, 20.4]1.43 [0.43, 2.43] 20/11
Llama 8B80.620/126/534.1 [20.7, 45.2]4.6 [−12.3, 21.0]0.69 [0.21, 1.16] 20/11
Gemma 9B46.84/69/22−23.5 [−52.8, 0.3]−21.8 [−48.3, 1.6]−0.03 [−0.55, 0.50] 15/16
Original blocks, plain+BOS
Llama 1B61.412/417/188.3 [−20.4, 31.2]−19.9 [−45.1, 0.6]0.26 [0.03, 0.51] 21/14
Gemma 2B55.710/625/1035.9 [20.9, 48.2]−9.4 [−27.9, 6.0]0.76 [0.44, 1.09] 25/10
Llama 8B81.425/316/192.6 [−20.4, 20.0]−9.6 [−31.9, 8.4]0.27 [−0.17, 0.72] 19/16
Gemma 9B72.920/427/848.7 [32.4, 61.4]−3.5 [−17.1, 8.3]1.21 [0.76, 1.66] 28/7
First fresh cohort, plain+BOS
Llama 1B60.611/424/941.4 [20.7, 55.3]−9.7 [−28.7, 5.4]0.48 [0.26, 0.71] 26/7
Gemma 2B62.113/528/552.0 [36.5, 63.3]−0.3 [−14.6, 12.3]0.97 [0.53, 1.47] 23/10
Llama 8B71.217/324/926.4 [7.0, 40.8]3.0 [−20.9, 21.2]0.60 [0.13, 1.13] 22/11
Gemma 9B74.217/129/457.2 [45.9, 65.7]0.4 [−15.9, 14.2]1.59 [1.01, 2.28] 30/3
Second fresh cohort, plain+BOS
Llama 1B58.111/618/1318.4 [−4.0, 37.3]−34.7 [−58.0, −17.4]0.43 [0.25, 0.61] 24/7
Gemma 2B59.711/522/940.0 [20.2, 55.8]−15.5 [−27.6, −4.5]0.96 [0.61, 1.32] 26/5
Llama 8B75.820/423/835.7 [19.7, 48.5]1.9 [−12.3, 14.2]0.94 [0.44, 1.42] 26/5
Gemma 9B75.818/226/554.3 [42.7, 64.5]−1.6 [−16.8, 11.7]1.49 [1.03, 1.96] 29/2
CheckpointSelective error reduction (%) [95%]
Original 35 blocks, chat format
Qwen 0.5B21.8 [1.2, 38.1]
Llama 1B40.7 [26.5, 51.7]
Gemma 2B38.3 [22.0, 51.4]
Qwen 7B37.2 [20.4, 51.0]
Llama 8B6.7 [−10.4, 19.9]
Gemma 9B52.1 [39.9, 61.7]

Relation-neutral sources, “Tell me about X.” and “What do you know about X?”, continued through the target, remove 39–45% of the no-change error on Qwen2.5-7B and Llama-3.1-8B, versus 42–51% for other-relation sources across the three cohorts (Appendix § 12). The post hoc advantage of other-relation over neutral sources, 0.24–0.45 nats on plain-format Qwen and Llama pooled over cohorts, keeps a mean interval above zero under resampled labels only at 0.5–1B and an interval for the share of favoring blocks above one half only on Qwen2.5-0.5B (Table 17a). The six pre-specified comparisons on the 31-block second cohort did not detect this advantage (Table 22).

Table 17: Label-resampling sensitivity for two distinct 99-block comparisons, with equal block weights in the point estimates (35 original, 33 first fresh, 31 second fresh). Brackets are saved 95% intervals from 20,000 relation-label resamples, seed 20261013; Fav. is the share of positive block effects, not a probability of significance. (a) Plain format: neutral-source hidden error minus other-relation hidden error. These pooled summaries are distinct from the 31-block tests in Table 22. (b) Default encoding: zero-prediction selective error minus error from the same replacement’s identical neutral entity-row content (D2a). These intervals are post hoc sensitivity summaries, not adjusted sign tests.

CheckpointMean gain in nats [95%]Fav. [95%]
(a) Other-relation versus neutral sources, plain format
Qwen 0.5B0.39 [0.10, 0.69]0.70 [0.52, 0.85]
Llama 1B0.45 [0.12, 0.84]0.66 [0.47, 0.82]
Gemma 2B0.06 [−0.32, 0.48]0.52 [0.33, 0.70]
Qwen 7B0.44 [−0.28, 1.18]0.57 [0.40, 0.72]
Llama 8B0.24 [−0.11, 0.57]0.63 [0.45, 0.78]
Gemma 9B−0.09 [−0.57, 0.43]0.47 [0.30, 0.65]
(b) Identical-content selectivity, default encoding
Qwen 0.5B0.40 [0.13, 0.69]0.72 [0.54, 0.87]
Llama 1B0.16 [−0.03, 0.37]0.61 [0.44, 0.76]
Gemma 2B0.75 [0.38, 1.16]0.79 [0.62, 0.92]
Qwen 7B1.69 [0.81, 2.64]0.69 [0.50, 0.85]
Llama 8B0.74 [0.36, 1.20]0.70 [0.54, 0.84]
Gemma 9B1.29 [0.80, 1.90]0.84 [0.70, 0.94]

Table 17.

Injected with equal weights into both relation-swapped targets, the relation-neutral entity-row change is identical for the two questions, so any shift of each question toward its own relation’s answers must come from the receiving question. This identical change lowers relation-selective error in 60–83 of 99 blocks, passing all six pre-specified tests within their family (Table 30). Identical random perturbations with the same row norms, and the other replacement’s neutral change given identically to both targets, lower selective error by at most 0.22 nats; the neutral change beats both on all six checkpoints in pre-specified families (Table 34–Table 36). For all three comparisons, Llama-3.2-1B’s within-family pass does not survive pooled correction. Under label resampling, some Llama-3.2-1B mean-effect intervals include zero, and several favoring-block intervals include one half (Table 17b and Table 35). These tests use the same selective-error readout as the selectivity checks above but inject identical content into both questions; they also differ in construction, encoding (each checkpoint’s default, with BOS for Llama and Gemma) and inference (sign tests over 99 blocks rather than per-cohort intervals).

CheckpointD1aD1bD1cD2aD2b
Qwen 0.5B99/0, 7.61, <0.00155/44, −0.06, 1.00050/14, 0.85, <0.00171/28, 0.40, <0.00141/58, 0.10, 1.000
Llama 1B97/2, 6.76, <0.00150/49, −0.01, 1.00051/13, 0.67, <0.00160/39, 0.16, 0.022†72/27, 0.92, <0.001
Gemma 2B99/0, 6.50, <0.00143/56, −0.06, 1.00051/13, 0.77, <0.00178/21, 0.75, <0.00146/53, −0.03, 1.000
Qwen 7B97/2, 7.26, <0.00151/48, 0.01, 1.00042/22, 1.06, 0.034†68/31, 1.69, <0.00145/54, −0.11, 1.000
Llama 8B95/4, 6.04, <0.00143/56, −0.05, 1.00042/22, 0.87, 0.034†69/30, 0.74, <0.00176/23, 1.39, <0.001
Gemma 9B96/3, 5.26, <0.00156/43, 0.01, 1.00052/12, 1.09, <0.00183/16, 1.29, <0.00134/65, −0.62, 1.000

Table 30. Confirmatory families of the testbed and read-out protocol, each six one-sided (D1a, D2a, D2b) or two-sided (D1b, D1c) exact sign tests with Holm correction within the family, in each checkpoint’s default encoding. Cells give positive / negative block counts, the mean signed error difference in nats and the Holm-adjusted pp. D1a: later-row vector error minus span-aligned error; D1b: state-replacement error minus additive entity-row error on the same rows; D1c: tuned-vector error minus span-aligned error, with tuning on the original cohort (fresh cohorts, 64 blocks); D2a: zero minus neutral selective error for identical entity-row changes; D2b: error with answer-position attention to entity rows blocked minus error with attention to other prompt rows blocked, sparing the first token and final cue row in the latter. Other families use 99 blocks. A dagger marks tests that do not pass the final 223-test pooled Holm correction (Table 25). For 80% power the true share of favoring blocks must reach 0.63–0.66 in the one-sided families, 0.64–0.68 in D1b and 0.68–0.71 in D1c (last to first Holm rank).

CheckpointG1G2
Qwen 0.5B74/25, 0.42, <0.00171/28, 0.46, <0.001
Llama 1B61/38, 0.15, 0.013†64/35, 0.21, 0.002†
Gemma 2B81/18, 0.70, <0.00175/24, 0.77, <0.001
Qwen 7B75/24, 1.54, <0.00168/31, 1.47, <0.001
Llama 8B70/29, 0.67, <0.00168/31, 0.64, <0.001
Gemma 9B81/18, 1.25, <0.00184/15, 1.19, <0.001

Table 34. Confirmatory families of the identical-content controls, each six one-sided exact sign tests over the 99 blocks with Holm correction within the family, in each checkpoint’s default encoding. Cells give blocks favoring / opposing the hypothesis, the mean block difference of selective error in nats (positive favors it) and the Holm-adjusted pp. G1: the same replacement’s identical neutral entity-row change has lower selective error than identical random entity-row perturbations with its row norms, averaged over eight draws. G2: it has lower selective error than the identical neutral change of the block’s other replacement mapped onto the target’s entity span. A dagger marks tests that do not pass when all 223 confirmatory tests form one Holm family (pooled pp: G1 Llama 1B 0.822; G2 Llama 1B 0.176). For 80% power the true share of favoring blocks must reach 0.63–0.66 (last to first Holm rank).

CheckpointG1 WithinG1 LabelsG1 Fav.G2 WithinG2 LabelsG2 Fav.
Qwen 0.5B[0.29, 0.57][0.17, 0.70]0.75 [0.59, 0.88][0.31, 0.62][0.17, 0.76]0.72 [0.52, 0.88]
Llama 1B[0.05, 0.25][−0.02, 0.34]0.62 [0.44, 0.77]†[0.10, 0.33][0.01, 0.42]0.65 [0.48, 0.80]†
Gemma 2B[0.51, 0.90][0.36, 1.09]0.82 [0.66, 0.94][0.56, 1.00][0.40, 1.20]0.76 [0.59, 0.89]
Qwen 7B[1.12, 1.96][0.76, 2.37]0.76 [0.58, 0.90][1.02, 1.93][0.67, 2.35]0.69 [0.51, 0.84]
Llama 8B[0.44, 0.91][0.30, 1.10]0.71 [0.55, 0.85][0.43, 0.86][0.27, 1.06]0.69 [0.49, 0.86]†
Gemma 9B[0.94, 1.57][0.76, 1.83]0.82 [0.67, 0.94][0.92, 1.48][0.73, 1.73]0.85 [0.71, 0.95]

Table 35. Mean block differences of the two control families (nats; positive favors the hypothesis) with 95% intervals resampling blocks within cohorts (Within) or relation labels, each block weighted by the product of its two labels’ counts (Labels; Appendix § 11), and the share of favoring blocks with its label-resampled interval (Fav.); † marks share intervals that include one half.

CheckpointNEUDRDRNRNDRND labelsRND draws
Qwen 0.5B0.40 [0.26, 0.56]−0.06 [−0.17, 0.04]−0.05 [−0.17, 0.06]−0.02 [−0.09, 0.04]−0.15, 0.11−0.10 to 0.05
Llama 1B0.16 [0.06, 0.27]−0.05 [−0.12, 0.02]−0.06 [−0.14, 0.02]0.01 [−0.02, 0.04]−0.04, 0.07−0.01 to 0.03
Gemma 2B0.75 [0.54, 0.96]−0.03 [−0.15, 0.09]−0.02 [−0.13, 0.09]0.04 [0.00, 0.09]−0.04, 0.140.01 to 0.10
Qwen 7B1.69 [1.21, 2.19]0.22 [−0.08, 0.54]0.31 [0.01, 0.63]0.15 [−0.08, 0.39]−0.21, 0.590.03 to 0.33
Llama 8B0.74 [0.50, 1.00]0.11 [−0.02, 0.23]0.12 [−0.01, 0.24]0.07 [0.03, 0.12][0.01, 0.14]0.02 to 0.13
Gemma 9B1.29 [0.98, 1.62]0.11 [−0.05, 0.27]0.11 [−0.06, 0.28]0.05 [0.00, 0.10]‡−0.03, 0.140.02 to 0.07

Table 36. Selective-error reduction from zero prediction, ∣s∣−∣s^−s∣|s| - |\hat{s} - s| (nats), of each identical entity-row construction over the 99 blocks, with 95% intervals resampling blocks within cohorts. NEU: the same replacement’s neutral change, the identical-content construction of Appendix § 15; DR: the other replacement’s neutral change mapped onto the target’s span; DRN: DR with each row rescaled to NEU’s norm; RND: random rows with NEU’s norms, averaged over eight draws, with its interval resampling relation labels (RND labels; Appendix § 11) and the range of the eight draws’ means. ‡: the interval excludes zero although a bound rounds to 0.00.

Blocking attention from the answer positions to entity rows above the interface (Geva et al., [17]) retains only 3–15% of the error reduction from transferred entity rows on Qwen and Llama but more than half on Gemma; blocking the other prompt rows except the first token and the final cue row retains nearly all of Llama’s reduction, about half of Gemma’s and little of Qwen’s. The pre-specified test that blocking entity rows raises block error more than blocking the other rows passes only on Llama (Table 30 and Table 33).

CheckpointKO-EKO-E first halfKO-E second halfKO-QKO-PKO-PQShift
Qwen 0.5B0.03/0.03/0.030.50/0.51/0.450.60/0.62/0.590.77/0.78/0.770.65/0.60/0.640.14/−0.09/0.151.72
Llama 1B0.12/0.12/0.110.42/0.44/0.420.73/0.71/0.720.97/0.96/0.960.94/0.93/0.940.91/0.90/0.901.21
Gemma 2B0.55/0.54/0.550.69/0.69/0.670.90/0.90/0.900.61/0.61/0.601.00/1.00/0.990.54/0.53/0.531.07
Qwen 7B0.15/0.12/0.110.66/0.62/0.580.56/0.59/0.560.58/0.59/0.590.74/0.75/0.740.11/0.16/0.233.89
Llama 8B0.11/0.12/0.120.14/0.16/0.150.95/0.93/0.931.01/1.00/1.000.97/0.96/0.970.93/0.92/0.931.38
Gemma 9B0.70/0.71/0.690.78/0.78/0.780.95/0.95/0.940.69/0.67/0.670.99/1.00/0.990.47/0.46/0.480.98

Table 33. Attention knockouts above the interface: share of the zero-prediction error removed under each knockout divided by the share without it, for other-relation entity rows / identical neutral entity rows / the target’s own entity-row change, over the 99 blocks of three cohorts. Knockouts block attention from the answer positions (final cue row and candidate prefix) to entity rows (E), to the question and cue rows between entity and final cue row (Q), to rows before the entity except the first (P), or to both (PQ), in every remaining layer or in its first or second half; each prediction is measured from the clean target under the same knockout. Blocking attention from every post-entity row to the entity rows removes the entity-row effect entirely, as the causal structure requires; this check is not tabulated. Shift: mean absolute change of the clean target’s answer contrast under KO-E (nats).

Depth and row position of answer reuse

We next ask where in the target computation these responses remain reusable. A pre-specified descriptive scan injects the same responses at nine interfaces from the decoder input (d=0d = 0) to the last layer (d=1d = 1), keeping the source weights from 50% depth. Span-aligned addition copies the changed entity span tokenwise, interpolates only the later question and cue rows, and copies answer-prefix rows. At each replacement endpoint, changed-span token IDs match across source and target questions, so at d=0d = 0 it reproduces the edited target by construction. Row subsets separate entity rows from post-entity question, cue and answer-prefix rows. Oracle patches add the target’s own edited-minus-clean states to the same rows (Appendix § 13).

In each checkpoint’s default encoding the share removed by span-aligned addition declines with depth on every checkpoint, from 100% at d=0d = 0 to 46–70% at d=1/2d = 1/2 (Figure 2). Up to 50% depth, transferred entity rows remove somewhat less error than the target’s own entity-row patch (Table 26). With target-relation sources, all aligned rows remove more error than entity rows alone at d=1/2d = 1/2, with pointwise intervals excluding zero on all five checkpoints with these runs (Table 27). At d=7/8d = 7/8 the target’s own post-entity change removes most of the no-change error, but other-relation responses on those rows remove far less (Table 26). Row effects are not additive: at d=1/2d = 1/2, the target’s own entity-row and post-entity-row changes, continued separately and added as predicted contrast changes, remove 49–69% of the no-change error, whereas patching both together reproduces the edit (Appendix § 13). Omitting answer-prefix rows keeps most of the span-aligned share at d=1/2d = 1/2 (Table 26 and Table 27), so under candidate scoring one question-plus-cue capture carries most of this gain.

CheckpointConstruction01/81/43/81/25/83/47/81Direct
Qwen 0.5BExact, all rows100.077.266.256.252.745.235.121.91.9−1.9
Qwen 0.5BExact, entity rows100.075.547.337.335.926.99.65.00.0
Qwen 0.5BExact, post-entity rows0.0−1.43.13.1−2.4−4.517.819.71.9
Qwen 0.5BTarget relation, all rows100.081.272.169.862.457.352.347.645.1
Qwen 0.5BOwn entity-row change100.085.054.948.240.925.78.64.20.0
Qwen 0.5BOwn post-entity change0.01.011.110.37.015.741.363.7100.0
Llama 1BExact, all rows100.084.072.968.563.355.052.541.634.432.4
Llama 1BExact, entity rows100.082.359.348.740.717.713.812.10.0
Llama 1BExact, post-entity rows0.0−0.27.89.510.417.524.928.834.4
Llama 1BTarget relation, all rows100.087.280.577.275.471.769.470.364.5
Llama 1BOwn entity-row change100.088.467.751.642.417.712.811.60.0
Llama 1BOwn post-entity change0.00.910.013.917.436.148.562.4100.0
Gemma 2BExact, all rows100.087.480.872.369.863.550.039.930.729.9
Gemma 2BExact, entity rows100.084.975.856.952.940.019.88.50.0
Gemma 2BExact, post-entity rows0.03.33.810.914.619.631.938.730.7
Gemma 2BTarget relation, all rows100.090.486.280.579.175.069.771.871.1
Gemma 2BOwn entity-row change100.091.079.561.757.442.620.98.80.0
Gemma 2BOwn post-entity change0.03.74.718.822.438.767.390.4100.0
Qwen 7BExact, all rows100.077.268.258.156.941.326.3−0.7−23.5−27.9
Qwen 7BExact, entity rows100.069.762.340.337.728.018.6−3.10.0
Qwen 7BExact, post-entity rows0.0−0.9−0.28.915.4−1.6−4.2−3.1−23.5
Qwen 7BTarget relation, all rows100.081.674.363.760.956.356.946.842.8
Qwen 7BOwn entity-row change100.079.572.745.944.231.721.1−3.10.0
Qwen 7BOwn post-entity change0.0−0.2−0.324.028.230.043.382.9100.0
Llama 8BExact, all rows100.083.574.866.846.430.313.9−14.6−18.2−14.2
Llama 8BExact, entity rows100.077.461.551.728.31.3−1.8−0.80.0
Llama 8BExact, post-entity rows0.03.47.312.911.99.4−17.0−18.2
Llama 8BTarget relation, all rows100.086.783.278.772.265.163.260.858.5
Llama 8BOwn entity-row change100.081.968.454.932.12.1−1.0−0.70.0
Llama 8BOwn post-entity change0.04.49.414.034.169.581.490.0100.0
Gemma 9BExact, all rows100.091.581.378.268.756.134.113.3−11.8−23.3
Gemma 9BExact, entity rows100.088.576.369.753.937.014.810.10.0
Gemma 9BExact, post-entity rows0.00.02.34.59.321.429.113.4−11.8
Gemma 9BTarget relation, all rows100.093.187.283.778.275.067.668.566.4
Gemma 9BOwn entity-row change100.093.380.972.458.039.316.111.20.0
Gemma 9BOwn post-entity change0.0−0.84.16.213.648.477.186.3100.0

Table 26. Answer reuse across depth in each checkpoint’s default encoding: percentage of the zero-prediction error removed on the 35 original blocks by exact span alignment with other-relation sources (all, entity or post-entity rows), by target-relation sources (all rows), and by the target’s own edited-minus-clean change on entity or post-entity rows. Qwen 7B uses its saved plain-format records from the earlier scan, since its depth re-run failed the consistency check. The last column is direct reuse, which does not depend on depth. The table includes the plotted series in Figure 2 and additional row subsets.

CheckpointddOth. allOth. EQOth. EOth. QATgt. allTgt. ETgt. QANeu. allNeu. EOwn EAll over E
Qwen 0.5B1/466.249.847.33.172.149.86.660.947.954.90.64 [0.34, 1.02]
Qwen 0.5B1/252.736.535.9−2.462.435.3−0.645.135.140.90.78 [0.39, 1.20]
Llama 1B1/472.959.359.37.880.563.18.372.860.767.70.49 [0.28, 0.74]
Llama 1B1/263.343.340.710.475.440.215.454.139.442.40.99 [0.61, 1.41]
Gemma 2B1/480.876.975.83.886.276.04.881.375.479.50.44 [0.27, 0.62]
Gemma 2B1/269.858.752.914.679.153.517.470.451.457.41.10 [0.78, 1.45]
Qwen 7B1/468.263.962.3−0.274.3−−−−72.7−
Qwen 7B1/256.941.937.715.460.9−−−−44.2−
Llama 8B1/474.865.361.57.383.265.66.775.864.068.40.74 [0.48, 1.01]
Llama 8B1/246.430.728.311.972.232.429.552.128.332.11.67 [1.19, 2.22]
Gemma 9B1/481.379.276.32.387.277.54.884.476.680.90.45 [0.29, 0.61]
Gemma 9B1/268.758.553.99.378.255.911.870.953.358.01.03 [0.71, 1.38]

Table 27. Row subsets by source relation at d=1/4d = 1/4 and d=1/2d = 1/2 in the default encoding (35 original blocks): percentage of the zero-prediction error removed by exact alignment on all rows, entity, question and cue rows (EQ), entity rows (E) or post-entity rows (QA) with other-relation, target-relation or relation-neutral sources (neutral entity rows with equal weights), and by the target’s own entity-row change. The last column is the target-relation all-row gain over entity rows (nats, pointwise 95% interval). Qwen 7B uses its saved plain-format records from the earlier scan, which lack the target-relation subsets and neutral sources.

Standard constructions on the testbed

Using the same answer-error reference, we ask whether a single vector can substitute for the aligned change matrix. The testbed protocol compares standard constructions over the 99 blocks at the 1/2 interface in default encoding. Its simplest single vector is the weighted source change at the last entity row, kept at its native scale and added at every target row from the entity onward; contrastive activation addition instead starts after the prompt (Panickssery et al., [46]). This broadcast vector has larger mean error than zero prediction on all six checkpoints (Table 16c), and span-aligned addition beats it in 95–99 blocks (Table 30). At matched entity-row coverage, span-aligned changes lower error by 0.30–0.49 nats relative to a native-scale vector (descriptive; Table 16c). Entity-state replacement (Huang et al., [28]; Wang and Xu, [55]) and additive entity-row patches differ by −0.06 to 0.01 nats, undistinguished by the pre-specified tests. Span-aligned addition also lowers mean error relative to the confirmatory suffix interpolation by 0.12–0.28 nats, with all six pointwise intervals excluding zero (descriptive; Table 16c).

Two protocols then tuned single vectors on the original cohort and tested them on the two fresh cohorts, which earlier studies had already used (64 blocks). The testbed’s tuned comparison (D1c in Table 30) selects the coefficient of one last-entity-row vector added at each entity row, without broadcasting it to later rows. The later steering-completion protocol of Table 1 selects the best of 90 constructions by minimizing mean block answer error on the original cohort; they vary averaging (single-pair or sixteen-context), direction (entity or answer cue), placement and amplitude (Table 3). After tuning, the broadcast vector’s pre-specified test against zero passes on three checkpoints, one under pooled correction. Rowwise norm matching scales one direction to the span-aligned row norms and beats zero on five checkpoints over all 99 blocks, four under pooled correction. Sixteen-context averaging gives no detected gain in its pre-specified comparison. Span-aligned addition beats the best of the 90 constructions (a norm-matched direction on four checkpoints) on all but Llama-3.1-8B under family-wise correction and on four under pooled correction; two of the five passes repeat earlier tests (Appendix § 14).

Discussion

What the quantitative profile adds. For reuse of recorded changes, overall fidelity and relation selectivity can diverge: measured against the target’s own response, overall fidelity is substantial but incomplete, whereas relation selectivity is not reliably reproduced on the small plain-format checkpoints, where the transfer mainly reproduces the shift both questions share. Binary answer outcomes do not quantify the magnitude of the reproduced response or its shared and relation-selective components. Beyond established entity-state transfer (Geva et al., [17]; Nanda et al., [41]), target-relation sources substantially narrow the hidden-over-direct gap except for plain-format Gemma, and the identical relation-neutral change retains selective effects that depend on the receiving question, beyond random and other-replacement controls.

Which changes to inject. Aligned changes on all rows beat entity rows alone at 50% depth (§ 4.2) and, on five of six checkpoints, the best tested single vectors (§ 4.3). Scaling single vectors to span-aligned row norms narrows that gap, but directions, coverage and prefix information also vary, so row structure is not isolated.

Conclusion

With known answer prefixes supplied, entity-replacement responses are partly portable across questions: other-relation hidden reuse removes 38–51% of the no-change error on plain-format Qwen and Llama and 58–69% on Gemma with BOS restored, with format- and cohort-dependent exceptions. On the small checkpoints in the original plain format, other-relation hidden reuse mainly reproduces the shared shift without reliably lowering relation-selective error. The identical relation-neutral change can nevertheless produce selective effects that depend on the receiving question. With the three cohorts pooled, target-relation sources substantially narrow the hidden-over-direct advantage except for plain-format Gemma; the residual advantage is unconfirmed and not replacement-specific. Span-aligned addition beats the best tested single vectors on five of six checkpoints, four under pooled correction. Together, these measurements quantify the magnitude, relation selectivity and construction dependence of cross-question response reuse.

Limitations

Relation labels and question wording vary together, and some labels express overlapping predicates, so they do not isolate semantic relation from wording. The fresh cohorts test new replacements within largely familiar relation and template families (Appendix § 10). Eligibility requires equal-length token-span replacements in English questions or completion prompts. Results cover six instruct checkpoints from three families, with size and training lineage confounded. Chat jointly changes the template, BOS token, answer instruction and scored terminator (end-of-turn token instead of newline). Restoring BOS alone reproduces Gemma’s chat-format improvement but not Llama-3.2-1B’s chat-format selectivity gain, whose source among the other components is unresolved. The three answer comparisons were not tested for Llama with BOS, and fresh cohorts were not run in chat format.

Confirmatory studies use one interface at 50% depth. The descriptive answer depth and position scans reuse the observed 35-block cohort and split constructions by target rows; their target-relation row subsets and neutral-source runs lack Qwen2.5-7B, whose re-run failed its consistency check (Appendix § 13). Suffix interpolation is inexact at the decoder input and lets entity rows receive non-entity source rows. Span-aligned addition applies only to same-replacement sources; different-replacement and random controls use interpolation, except in the identical-content experiment, whose other-replacement control interpolates within the entity span (Appendix § 15). Attention knockouts locate attention paths rather than heads or features and also shift the unedited answer contrast. Construction comparisons jointly vary coverage, direction and prefix information and do not evaluate behavioral steering; norm-matched vectors borrow span-aligned norms (Appendix § 14).

Complete-answer evaluation conditions sources and targets on known published-answer prefixes. Its literal candidate groups carry little absolute probability mass and exclude alternative aliases and formatting, including semantically equivalent strings ([27]); Appendix § 11 gives their masses. The next-token distribution at the answer cue and greedy generation from a single capture are separate measures: one question-plus-cue capture with other-relation sources reproduced the edited greedy answer string for only 5–23% of targets (Appendix § 13). Component splits are algebraic output contrasts, not identified hidden-state mechanisms.

The protocols were internal, not externally registered, and fixed confirmatory comparisons before their own outcomes. Later designs used earlier findings and materials, including the choice of other-relation sources, the BOS ablations after Gemma’s plain-format failures, and the 90-construction comparison, whose distinct selection and evaluation cohorts had all been studied before. Appendix § 10.§ 5 gives the order and scope of each protocol. The neutral-source controls use two fixed English templates, and the different-replacement control changes both the original and replacement entity.

Sign inference assumes independent blocks or relation/template components. Grouping recorded overlaps leaves possible dependence through aliases, recurring answers and relation labels reused across cohorts. Label resampling shows sensitivity for some mean effects and positive-block shares but gives descriptive intervals, not dependence-adjusted family-wise tests (Appendix § 11). Study-wide Holm correction addresses a separate multiplicity question and changes 26 conclusions, including first-fresh-cohort selectivity and some tuned-vector and identical-content results; Llama-3.1-8B’s second-cohort selectivity replication remains (Table 25). Limited power also precludes interpreting failed comparisons as equivalence (Appendix § 11). A constant push would also pass the fresh-cohort selectivity sign tests, so interpretation relies on magnitudes and different-replacement comparisons (Table 24).

Appendix guide

Appendix A defines the constructions and scores and collects notation in Table 4. Appendix B describes the blocks, cohorts, checkpoints and formats, and the order of the studies (Appendix § 10.§ 5). The remaining appendices follow § 4: Appendices C (prediction fidelity, calibration and statistical procedures) and D (scale, format, fresh-cohort and source-relation comparisons) support § 4.1; Appendix E supports § 4.2 and the cue-distribution and generation results of § 4.1; Appendix F supports § 4.3; and Appendix G gives the attention knockouts and identical-content controls of § 4.1. Table 1 inventories every test family, and Table 25 lists the conclusions that change under pooled correction.

Response Construction and Scores

Notation and terms. Table 4 collects the terms. Appendix B.4 defines the earlier next-token direction comparisons.

Suffix correspondence and source weights. Let pxp_x be the first changed position and TxT_x the sequence length. For target row j≥ptj \ge p_t, the source coordinate is

xs→t(j)=ps+(Ts−1−ps)j−ptTt−1−pt.(5)x_{s\to t}(j)=p_s+(T_s-1-p_s)\frac{j-p_t}{T_t-1-p_t}. \tag*{(5)}

Writing q=∣x∣q=|x| and α=x−q\alpha=x-q, the aligned row is (1−α)Δs[q]+αΔs[min⁡(q+1,Ts−1)](1-\alpha)\Delta_s[q]+\alpha\Delta_s[\min(q+1,T_s-1)]. Rows before ptp_t are zero. The map connects the first changed and final rows; it is a positional heuristic rather than a claim of semantic correspondence. For normalized unedited final states zx=Hℓ(x,a)Tx−1/∥Hℓ(x,a)Tx−1∥2z_x=H_\ell(x,a)_{T_x-1}/\|H_\ell(x,a)_{T_x-1}\|_2 and ds=∥zs−zt∥22d_s=\|z_s-z_t\|_2^2, the fixed weights are

ws=(ds+10−12)−1∑r∈S(dr+10−12)−1.(6)w_s=\frac{(d_s+10^{-12})^{-1}}{\sum_{r\in S}(d_r+10^{-12})^{-1}}. \tag*{(6)}

Each source condition has its own unedited-state weights. Both reuse routes from the same source pair use the same numerical weights; comparisons across source conditions use the same weighting rule, not necessarily the same values. No edited target enters alignment or weighting.

Direct reuse. With the same sources and weights, direct reuse predicts logits

ℓ^t,e=ℓ(t,a)+∑sws[ℓ(s,e)−ℓ(s,a)].\widehat{\ell}_{t,e} = \ell(t,a) + \sum_s w_s[\ell(s,e)-\ell(s,a)].

Subtracting the unedited target logits gives its predicted output change. Unlike hidden reuse, this route does not pass the source change through the target’s remaining layers.

Row-norm matching. For a nonzero donor row, magnitude matching uses R(A,B)j=∥Bj∥2Aj/∥Aj∥2\mathcal{R}(A,B)_j = \lVert B_j\rVert_2 A_j/\lVert A_j\rVert_2. A zero reference norm requires a zero matched row; a zero donor norm with positive reference norm cannot be matched. This operation fixes row magnitudes, not content.

Distributional measures. For clean, edited and predicted logits ℓa,ℓe,ℓP\ell_a,\ell_e,\ell_P, set p∙=softmax⁡(ℓ∙)\mathbf{p}_{\bullet} = \operatorname{softmax}(\ell_{\bullet}). KL recovery is

RKL=1−KL⁡(pe∥pP)KL⁡(pe∥pa).(7)R_{\mathrm{KL}} = 1 - \frac{\operatorname{KL}(p_e\mathbin\Vert p_P)}{\operatorname{KL}(p_e\mathbin\Vert p_a)}. \tag*{(7)}

Values 1 and 0 denote exact reproduction and the clean target’s error, respectively; negative values denote larger error. Greedy agreement compares predicted and edited-target top tokens, with the clean target as the zero-change baseline. The cue-level aggregate in Table 28 uses the ratio of mean block divergences, rather than a mean of per-target recovery ratios.

Complete-answer evaluation. In the plain and plain+BOS formats, each question receives the cue \textbackslash nAnswer:\textbackslash n, and candidates retain their released wording and terminal newline. Chat renders a single user message containing the instruction Answer with a short phrase only. followed by a newline and the original question. It uses the released template’s generation prompt instead of this cue. The template’s default system content is retained, with Llama’s default date fixed to 26 July 2024. Each candidate ends with the family’s end-of-turn token rather than a newline, and that token is also scored. For the rendered answer prompt q∗q^*, a complete candidate string yy has score

ℓ(y∣q∗)=∑u=1Tylog⁡p(yu∣q∗,y<u).(8)\ell(y \mid q^*) = \sum_{u=1}^{T_y} \log p(y_u \mid q^*, y_{<u}). \tag*{(8)}

The sum uses full-vocabulary normalization with no length normalization. The plain formats’ newline-terminated sequences are prefix-free within each target’s candidate set on all three tokenizers. Exact duplicate strings enter the candidate set once. For a published answer group AA, the group score of Equation (3) is Mx(A)=log⁡∑y∈Aexp⁡ℓ(y∣q∗)M_x(A) = \log\sum_{y\in A} \exp\ell(y \mid q^*) with x=q∗x=q^*. The measured contrast change δc\delta c is the replacement-induced change in Mx(AT)−Mx(AO)M_x(A_T)-M_x(A_O). Absolute error compares the predicted change with the directly measured edited target’s change, in nats; zero predicts no change.

At candidate position uu, let y<uy_{<u} be the known preceding candidate tokens and sa∗,se∗s_a^*,s_e^* the rendered source prompts. The recorded response is

Δs,e(y<u)=Hℓ(se∗,y<u)−Hℓ(sa∗,y<u).(9)\Delta_{s,e}(y_{<u}) = H_{\ell}(s_e^*,y_{<u}) - H_{\ell}(s_a^*,y_{<u}). \tag*{(9)}

Each prefix therefore has its own source capture, rather than reusing one bare-question matrix for the whole answer. The clean target also receives that prefix. The changed prompt suffix uses interpolation; shared candidate-prefix rows align tokenwise. Source weights use the clean final prompt state and stay fixed across candidates. For either reuse route, predicted logits at each prefix undergo full-vocabulary softmax, giving p^t,e(⋅∣y<u)\widehat{p}_{t,e}(\cdot\mid y_{<u}). Predicted complete-string and group scores are

L^t(y)=∑u=1Tylog⁡p^t,e(yu∣y<u),M^t(A)=log⁡∑y∈Aexp⁡L^t(y),c^t,e=M^t(AT)−M^t(AO).(10)\begin{aligned} \widehat{L}_t(y) &= \sum_{u=1}^{T_y} \log\widehat{p}_{t,e}(y_u \mid y_{<u}),\\ \widehat{M}_t(A) &= \log\sum_{y\in A} \exp\widehat{L}_t(y),\\ \widehat{c}_{t,e} &= \widehat{M}_t(A_T)-\widehat{M}_t(A_O). \tag*{(10)} \end{aligned}

The predicted contrast change is δc^=c^t,e−ct,a\widehat{\delta c} = \widehat{c}_{t,e}-c_{t,a}, using the same clean target contrast as the observed change in Equation (3).

Study Materials

Blocks and eligibility

Each answer-study block crosses two factual entity replacements with two queried relations and six published ZsRE question templates: two sources and one target per relation. Replacements preserve token length on all three tokenizers and change at least two entity-overlapping tokens. Source and target contexts diverge before the entity. Within each cohort, relation labels, entity identities and known aliases, templates and complete inputs are disjoint across blocks; this does not make cohorts independent of one another.

The introductory example and Table 5 come from block 11 of the earlier 40-block crossing, selected retrospectively for readable wording, not its outcomes. They are not examples selected from the 35-block answer cohort. The earlier next-token cohorts and their inference units are summarized in Appendix § 10.§ 4.

Relation labelRolePublished question
developerSource 1By whom was xx developed?
developerSource 2Which company developed xx?
developerTargetWhat is the name of xx’s developer?
programming languageSource 1What is the programming language for xx?
programming languageSource 2What programming language was used to write xx?
programming languageTargetWhich programming language is xx written in?

Table 5. Six published question templates in one complete block.

Table 5: Six published question templates in one complete block. Substitute xx with each endpoint of SLIB to JDeveloper and Apache Derby to OpenLayers. The entity slot is the only replacement in the displayed wording.

Answer cohorts and overlap

The original answer cohort has 35 disjoint four-target blocks, 70 labels, 140 distinct entities and 210 templates. It separates target- and other-relation new-answer strings on all 140 targets, with no recorded dependency edges. Of the 70 relations and 210 templates, 68 and 202 recur from the crossing, and 67 and 197 recur from the native-amplitude study. These are again new-entity rather than new-family materials. Each target’s candidates include published answers for both original and replacement entities under both relations; only the replacement’s groups ATA_T and AOA_O enter the contrast, and the New-first and In-set summaries use the full candidate set (Table 23 and Table 29). There are 532 target/candidate pairs per checkpoint, at most six candidates per target. All sources, clean target states and predictions are frozen for each checkpoint before edited-target answers are measured.

The two fresh cohorts contain 33 and 31 further four-target blocks. They reuse 64 of 66 and 62 of 62 relation labels and 192 of 198 and 185 of 186 templates from earlier cohorts. Thus they assess new entities within largely familiar relation and wording families, not new-family generalization. Selection excludes the earlier cohorts’ entities, aliases and complete inputs, retains equal-length, tokenizer-compatible replacements, and uses the same block and answer-group construction. Only the input rendering and scored terminator change across formats (Appendix § 9).

Artifacts, data and computation

Checkpoints and input formats. The fixed released checkpoints are Qwen2.5-0.5B-Instruct (Qwen Team, [48]), hidden state 12 of 24 (dimension 896); Llama-3.2-1B-Instruct (Meta, [40]), state 8 of 16 (dimension 2,048); and Gemma-2-2B-it (Gemma Team, [16]), state 13 of 26 (dimension 2,304), each at 50% depth, the interface at which changes are injected. A scale replication uses the same-family Qwen2.5-7B-Instruct (Qwen Team, [48]), Llama-3.1-8B-Instruct (Grattafiori et al., [20]) and Gemma-2-9B-it (Gemma Team, [16]), at states 14 of 28, 16 of 32 and 21 of 42; their tokenizers are byte-identical to the small checkpoints’, so the answer inputs are unchanged. Original studies use plain raw text without special tokens, fixed before outcomes: no chat template or Llama/Gemma beginning-of-sequence (BOS) token, although both tokenizers add one by default. The first token acts as an attention sink (Xiao et al., [60]); without it, one-token perturbations spread further (Barbero et al., [2]) and Gemma-2 loses its massive activations and degrades (Owen et al., [44]). Later protocols add each checkpoint’s chat template and plain text with only the BOS token restored (plain+BOS). The tested, steering-completion and identical-content studies use each checkpoint’s default encoding, plain for Qwen, whose tokenizer adds no BOS token, and plain+BOS for Llama and Gemma.

The checkpoints are used under their published terms: Apache 2.0 for Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, the Qwen Research License, which permits non-commercial research use, for Qwen2.5-3B-Instruct, the Llama 3.2 and Llama 3.1 Community Licenses for Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct, and the Gemma Terms of Use for both Gemma-2 checkpoints. Materials come from public research releases of ZsRE, BEAR, T-REx (as distributed with LPAQA), ParaRel and LPAQA, and from public Wikidata metadata. They state facts about named public entities, including birth dates, family members and party membership; no new personal data are collected. All prompts and answers are English. All model runs use CPUs, float32 model weights and float64 scoring, with Python 3.12.9, PyTorch 2.13.0 (Paszke et al., [47]), Transformers 5.14.1 (Wolf et al., [58]) and NumPy 2.5.1 (Harris et al., [21]). The small, 1.5B and 3B checkpoints run with eight threads; the larger checkpoints run as four 48-thread processes per 192-core node. An earlier descriptive answer standard-patch extension on the original cohort, not analysed here, and all 7B, 8B and 9B runs use the official PyTorch 2.13.0+cpu build and the other measurements the macOS arm64 wheel of the same version, with the same scientific settings and the 10−310^{-3} logit replay check: every continuation from an unpatched interface must reproduce the clean logits within 10−310^{-3}. We plan to release the code, block and candidate identifiers, saved predictions, run records and written protocols; text derived from ZsRE and LPAQA will be released only as their terms permit.

BEAR is released under CC BY-SA 4.0, the ParaRel repository under MIT, and Wikidata structured metadata under CC0.1 We could not establish an explicit license for the exact ZsRE release or for LPAQA’s prompt files and bundled T-REx training subset from their release pages.2 Public availability is not treated as a redistribution license; reuse must retain upstream attribution and resolve these terms before repackaging those materials.

The four earlier next-token cohorts and the original answer cohort total 2,076 target/checkpoint evaluations on the small checkpoints; the two fresh cohorts add 256 new targets, while auxiliary studies reuse some targets. The answer standard-patch extension used 23.52 allocated core-hours plus 3.89 for an incomplete launch. The depth/scale, span-aligned-scan/chat/first-fresh-cohort, BOS/neutral/second-cohort, testbed/depth, steering-completion and identical-content studies used, respectively, about 3, 8, 6.5, 14.3, 4.3 and 1.8 reported CPU hours for the smaller-checkpoint runs, and 448, 1,020, 1,178, 3,293, 889 and 404 allocated core-hours for the larger checkpoints. The span-aligned-scan group also used 11.6 allocated core-hours in an incomplete launch; its smaller-checkpoint figure includes the auxiliary descriptive 1.5B and 3B runs, whose outcomes are not analysed here. CPU time and allocated core-hours are distinct quantities; we did not record a project-wide total and do not infer one from target counts. Saved-data checks run no models.

This work is intended for controlled analysis of response transfer, not deployment of factual edits. Public named-entity facts can still be sensitive, outdated or biased; identities are retained because entity replacement is the research variable, not because the records are anonymous. We performed no dedicated offensive-content audit and collected no new participant data. Response reuse could propagate false personal claims or be misused to manipulate answers. The English-only selection and limited checkpoints also preclude population-level fairness claims.

Earlier next-token studies

Four studies preceding the answer evaluation measured next-token logit changes immediately after the input prompt, without an answer cue or BOS token; the multi-token study included 132 complete questions and 52 completion prompts among its 184 edits. They used the three small checkpoints and public ZsRE, BEAR, T-REx, ParaRel and LPAQA materials ([57, 9, 29] (Elshahar et al. 2018)). Table 6 gives their sample sizes and inference units; their 45 tests remain in the study-wide multiplicity accounting of Table 1.

Directional agreement is the cosine between predicted and measured logit-change vectors after subtracting each vector’s vocabulary mean. It is unchanged by positive rescaling, unlike answer-contrast error. A target-local derivative is the Jacobian of the input-to-interface computation at the original input applied to the full embedding displacement of the replacement; a source-local derivative is computed analogously at a source and then aligned and weighted. Neither is a path-integrated derivative. Direction comparisons assign common row-norm profiles before exact continuation through the target’s remaining layers.

The single-token and multi-token studies each test four contrasts on three checkpoints: finite source minus target-local derivative at the finite and derivative profiles, same minus different replacement, and finite source minus source-local derivative. At the finite profile the compared rows use finite-source magnitudes; the derivative profile is globally rescaled to that total norm. Inference averages relation medians equally within connected components of relations linked by shared templates. The single-token sample has 64 BEAR replacements in ten relations and ten components; the multi-token sample has 184 replacements in 46 relations and 45 components. All twelve multi-token comparisons pass their Holm family; only the three identity contrasts pass in the single-token study, whose derivative comparisons remain inconclusive at ten components.

The crossing uses 40 disjoint four-target blocks and a common same-replacement, target-relation row-norm profile for every condition. Its nine tests compare same with different replacements within each source relation and same/other with different/target; all have 40 positive blocks. The native-amplitude study uses 36 further blocks and retains the measured source norms. Its twelve tests assess the hidden-over-direct directional gain for the same replacement and the reduction of the same-minus-different identity margin, on three checkpoints and two source relations; all pass within their family. That margin reduction is specific to the raw cosine scale: five of six mean contrasts change sign under Fisher-z, squared-cosine or remaining-gap rescaling. These direction studies do not establish complete-answer fidelity and do not replace the answer-error comparisons.

Study sequence

The independent single-token, multi-token, 40-block crossing and native-amplitude studies preceded the 35-block answer study. An earlier 157-pair pilot is not evidence: its pair-level tests assumed independence despite recurring subjects and relations. Its identities, aliases and prompts are excluded from later selections. The pilot fixed the 50% interface; the later descriptive all-layer scan on the crossing cohort did not retune it.

Each core study had an internal written protocol before its edited-target outcomes, not an external registration. Later designs were informed by earlier findings: the answer study chose other-relation sources after the crossing cohort’s descriptive answer comparison separated the routes most clearly. With target-relation sources, the descriptive route-gap intervals included zero on Llama and Gemma. After the plain-format answer and scale studies came the span-aligned depth scan, chat format and the first fresh cohort. That cohort’s two pre-specified selectivity tests followed up the descriptive reductions on Qwen2.5-7B and Llama-3.1-8B. BOS ablation, neutral sources and the second fresh cohort followed Gemma’s known plain-format failures. Subsequent protocols fixed the testbed, steering-completion and identical-content comparisons before their own outcomes, but reused studied materials, including already-observed Qwen blocks in some testbed arms. Post-outcome diagnostics do not become pre-specified comparisons.

Prediction Fidelity, Calibration and the Answer Estimand

The primary answer comparisons concern reproduction of a measured change, not factual answer accuracy. Table 7 and Table 8 report the absolute errors and the nine pre-specified answer comparisons against all three baselines, including Gemma’s positive mean gains with failed block sign tests. Mean-effect intervals describe magnitude; sign tests assess how often a block favors a comparison. The original answer mean intervals use 20,000 outer-block percentile resamples with seed 20261007; later tables state their aggregation and resampling scope.

Inference and test inventory. For a paired error comparison, positive effect means baseline error minus evaluated-prediction error; two-sided construction differences state their subtraction order in the table captions. Exact sign tests omit zero effects: under the null, the number of positive signs among the nonzero blocks is binomial with success probability one half. The one-sided test uses its upper tail; the two-sided test doubles the smaller tail, capped at one. Empty or incomplete planned comparisons receive p=1p = 1. Holm adjustment orders the raw pp values within the stated family, multiplies each by the number of remaining tests, and takes successive maxima capped at one. A named primary family does not make the other families conditional on its success.

The original and larger-checkpoint answer families each contain nine tests (three baselines on three checkpoints); chat has one nine-test family per scale. The first fresh cohort adds those eighteen comparisons and two large-checkpoint selectivity tests. The second adds six matched-route tests, eighteen answer comparisons, two selectivity tests and twelve neutral-source tests (other-relation versus neutral, and neutral versus zero). BOS has one six-test Gemma family per cohort; Llama+BOS comparisons are descriptive. The five testbed families, four steering families and two identical-content control families each contain six tests, as defined in Table 30, Table 31 and Table 34. Together with the 45 earlier tests (Appendix § 10.§ 4), these form the 223-test pooled sensitivity family, including two repeated comparisons. Table 25 lists every conclusion altered by that pooling. Power descriptions invert the exact binomial rejection rule at the first and last Holm ranks for a true positive-sign probability; they do not use the observed mean effect as a power estimate. For the one-sided answer families with 31–35 nonzero blocks, the true positive-sign probability needed for 80% power ranges from about 0.71 to 0.80 across family sizes and the last to first Holm ranks. The larger testbed, steering and identical-content control families state their sample sizes and rank-specific thresholds in Table 30, Table 31 and Table 34.

CheckpointE1E2E3E4
Qwen 0.5B38/26, 0.13, 0.25350/14, 0.85, <0.00142/57, −0.08, 0.95448/51, 0.20, 0.656
Llama 1B44/20, 0.45, 0.007†51/13, 0.66, <0.00147/52, −0.04, 1.00066/33, 0.54, 0.001†
Gemma 2B37/27, 0.53, 0.25351/13, 0.82, <0.00156/43, 0.04, 1.00076/23, 1.65, <0.001
Qwen 7B38/26, 0.97, 0.25345/19, 1.38, 0.003†51/48, 0.07, 1.00072/27, 2.12, <0.001
Llama 8B45/19, 1.47, 0.004†38/26, 0.48, 0.16946/53, −0.03, 1.00081/18, 1.79, <0.001
Gemma 9B49/15, 2.05, <0.00152/12, 1.09, <0.00147/52, −0.06, 1.00081/18, 2.21, <0.001

Table 31.

Mean intervals use percentile quantiles of 20,000 resamples of whole inference units, preserving paired predictions. Pooled summaries resample blocks within each of the 35/33/31-block cohorts. Label-resampled sensitivity intervals instead sample the distinct relation labels with replacement and weight each block by the product of its two labels’ sampled counts, normalizing the weighted mean or positive-block share. These post hoc route and content summaries use seed 20261013. The product-weight scheme can widen intervals even for independent blocks: treating label counts as independent with mean and variance one gives a heuristic variance multiplier near three, not a general correction factor. It therefore measures sensitivity to this reweighting, not dependence-adjusted family-wise significance. Median-block intervals invert the binomial sign test to order statistics with coverage at least 0.95.

For route rescaling, each route gets a least-squares multiplicative factor fitted on the original cohort, with no intercept; errors are then evaluated on the two fresh cohorts only. This post hoc magnitude check is separate from the observed-on-predicted calibration fits with an intercept in Table 9, and does not replace the original tests.

SourceModelRouteSlope [95%]Intercept [95%]Bias [95%]
OtherQwenHidden0.900 [0.622, 1.046]0.421 [0.047, 0.899]−0.440 [−0.975, −0.048]
OtherQwenOutput0.460 [0.130, 0.700]0.509 [−0.131, 1.160]−0.812 [−1.905, 0.111]
OtherLlamaHidden0.945 [0.787, 1.016]0.273 [−0.012, 0.553]−0.271 [−0.556, 0.026]
OtherLlamaOutput0.870 [0.667, 0.958]0.552 [0.078, 1.043]−0.589 [−1.102, −0.092]
OtherGemmaHidden0.728 [0.560, 0.823]0.243 [−0.183, 0.701]−0.253 [−0.781, 0.249]
OtherGemmaOutput0.548 [0.218, 0.779]0.253 [−0.285, 0.748]−0.283 [−0.936, 0.365]
TargetQwenHidden0.976 [0.881, 1.045]0.076 [−0.131, 0.273]−0.071 [−0.275, 0.135]
TargetQwenOutput0.818 [0.698, 0.884]−0.010 [−0.308, 0.305]0.068 [−0.275, 0.404]
TargetLlamaHidden0.957 [0.820, 1.023]0.316 [0.047, 0.587]−0.316 [−0.592, −0.046]
TargetLlamaOutput0.953 [0.810, 1.025]0.130 [−0.233, 0.485]−0.121 [−0.490, 0.244]
TargetGemmaHidden0.761 [0.573, 0.894]0.258 [−0.072, 0.591]−0.271 [−0.608, 0.067]
TargetGemmaOutput0.662 [0.438, 0.818]0.010 [−0.388, 0.453]0.096 [−0.409, 0.587]

Table 9. Calibration of same-replacement predictions: observed change on predicted change. Brackets give pointwise 95% block-bootstrap intervals. Intercept and mean prediction minus observed change (bias) are in nats. Other/target denotes the source relation. Zero prediction has no calibration slope.

Individual predictions and calibration. Figure 3 and Figure 4 show every target and block, respectively. Correlations measure covariation, whereas calibration regresses observed change on predicted change with an intercept; no evaluated prediction is recalibrated. A slope near one does not alone establish calibration, because intercept and mean bias also matter (Table 9). Table 10 gives paired uncertainty for the hidden-minus-direct correlation difference.

Figure 3. Both source relations on shared axes, small checkpoints in the original 35-block cohort and plain format; all 140 targets per checkpoint and three reuse conditions. Matching lines are calibration fits of observed on predicted changes; the gray dash-dot line is exact reproduction. Target-relation sources bring the output route closer to hidden reuse, without making the different-replacement control track the target response.

Figure 4. All 35 block gains behind the nine original answer comparisons (other-relation sources). Each point is one block’s mean error reduction by same-replacement hidden reuse relative to direct reuse, zero prediction or different-replacement hidden reuse; open circles are positive and filled diamonds negative. Squares and bars give block means and pointwise 95% intervals; numbers give positive/negative block counts.

SourceModelrhidden−routputr_{\mathrm{hidden}} - r_{\mathrm{output}} [95%]
OtherQwen0.249 [0.117, 0.415]
OtherLlama0.063 [0.017, 0.155]
OtherGemma0.249 [0.139, 0.410]
TargetQwen0.053 [0.013, 0.127]
TargetLlama0.023 [−0.008, 0.070]
TargetGemma0.150 [0.055, 0.277]

Table 10. Paired Pearson-correlation differences for same-replacement hidden minus output reuse. Intervals are pointwise descriptive, not new corrected tests. With target-relation sources, Llama’s interval includes zero.

Shared and relation-selective changes.

The components ss and hh of Equation (4) are algebraic output summaries, not identified hidden-state mechanisms. Balanced pairing gives

Var⁡(δc)=Var⁡(s)+E[h2].\operatorname{Var}(\delta c) = \operatorname{Var}(s) + \mathbb{E}[h^2].

The selective share in Table 11 uses the first term; the shared term is the second moment of hh. For component prediction errors es=s^−se_s = \hat{s} - s and eh=h^−he_h = \hat{h} - h, the mean absolute error of a relation-swapped target pair is

∣es+eh∣+∣es−eh∣2=max⁡(∣es∣,∣eh∣).\frac{\lvert e_s + e_h\rvert+ \lvert e_s - e_h\rvert}{2} = \max(\lvert e_s\rvert,\lvert e_h\rvert).

Thus good total response reproduction need not imply good relation-selective reproduction. Table 12 reports both components’ correlations and absolute errors; selective-error reductions and their intervals appear in Table 24, including nonpositive reductions.

Candidate scope and sensitivity. The complete-string contrast uses full-vocabulary token normalization but only the literal published strings in its answer groups. Their small absolute probability mass (Table 13) limits an interpretation in terms of factual recall or free generation. Table 14 reports median-block summaries and name-overlap exclusions, including the weakened plain-format Gemma results. Table 15 reports route rescaling and replacement-identity comparisons: an advantage of the hidden route can also occur for a wrong replacement and is not by itself evidence of replacement-specific transfer. These checks are post hoc descriptions, not replacements for the original tests. Label-resampled intervals are sensitivity summaries, not dependence-adjusted family-wise tests; interval widening can arise even for independent blocks.

Comparator and sensitivity details. Table 16 separates the overlap exclusions, original-cohort chat comparisons and 99-block construction differences. These scopes are not pooled together. Table 17 distinguishes the three-cohort source comparison from the identical-content selectivity comparison and gives both mean and positive-block-share intervals, including intervals crossing zero or one half.

Replication, Encoding and Source-Relation Comparisons

These studies distinguish replication across checkpoint scales and input formats from generalization to new relation families. The fresh cohorts contain new entities but mostly familiar relations and templates. Table 19, Table 20 and Table 21 report the larger-checkpoint, chat-format and both fresh-cohort families, including failed comparisons. Table 22 reports the second fresh cohort’s failed target-relation route tests and the neutral-source comparisons; failure does not show equivalence. Appendix § 10.§ 3 defines the input formats and Appendix § 10.§ 5 gives their protocol order.

CheckpointBaselineSignsGainpHolmp_{\mathrm{Holm}}
Qwen 7BDirect reuse32/35.5231.7×10−61.7 \times10^{-6}
Qwen 7BZero33/23.5731.7×10−71.7 \times10^{-7}
Qwen 7BDifferent32/33.5771.7×10−61.7 \times10^{-6}
Llama 8BDirect reuse32/32.4341.7×10−61.7 \times10^{-6}
Llama 8BZero30/51.8034.5×10−54.5 \times10^{-5}
Llama 8BDifferent32/31.7281.7×10−61.7 \times10^{-6}
Gemma 9BDirect reuse23/120.8700.090
Gemma 9BZero23/120.4610.090
Gemma 9BDifferent29/60.9571.8×10−41.8 \times10^{-4}

Table 19. Larger-checkpoint confirmation: same-replacement other-relation hidden reuse at the 1/2 interface against three baselines on the 35 answer blocks. Gain is the mean block error reduction (nats); signs count positive/negative blocks. Baselines are direct reuse, zero prediction and different-replacement hidden reuse. One-sided exact sign tests are Holm-adjusted within this nine-test family.

Matching the source relation. Table 18 compares the two source conditions over the same 99 blocks and encoding. Matching substantially narrows the mean route gap except for plain-format Gemma, where the reduction is small. The source response and its read-out both change: this is not an isolated causal effect of matching. The residual matched-route advantages are descriptive, not confirmed by the second cohort’s pre-specified tests; the label-resampled intervals show the sensitivity behind this qualification.

Neutral-source shares by cohort. In the plain format, the other-relation/neutral-source hidden shares of no-change error removed are 51.2/39.6, 42.4/40.9, 48.7/44.1% for Qwen2.5-7B and 46.2/44.2, 49.3/38.7, 50.4/45.1% for Llama-

3.1-8B, in original/first-fresh/second-fresh order (35/33/31 blocks). These are cohort-specific ratios, distinct from the pooled nats in Table 17.

Restoring only the BOS token is distinct from changing to a chat template. Table 23 reports the format-specific results rather than pooling away Gemma’s plain-format failures. Table 24 shows why the selectivity sign tests alone are insufficient: a constant push toward each question’s own answers would also pass the four specified fresh-cohort tests. The magnitudes and different-replacement comparison carry additional information. The study-wide Holm sensitivity in Table 25 lists every conclusion that changes when all 223 tests, which include the two repeated comparisons, are pooled.

E Depth, Row Coverage and Conditional Scoring versus Generation

The descriptive answer scan reuses the original 35 blocks, nine interfaces and fixed source weights from 50% depth. Span-aligned addition copies the same replacement’s entity rows tokenwise, interpolates later question and cue rows and copies candidate-prefix rows. With token IDs matched as in § 4.2, it reproduces the edit at the decoder input by construction. Table 26 and Table 27 report the default-encoding depth and source/row comparisons; Qwen2.5-7B lacks the target-relation row subsets and neutral-source runs. The nine relative interfaces are 0, 1/8, 1/4, 3/8, 1/2, 5/8, 3/4, 7/8, 1, rounded to the nearest layer. Here d=0d = 0 is the effective decoder input and d=1d = 1 the last layer’s output before final normalization. Continuation still applies final normalization and the released head, including Gemma’s logit soft-capping, so hidden and direct reuse need not agree at d=1d = 1 (Table 26). Entity rows (E) cover the minimal changed span; post-entity question rows (Q), the answer cue (C), and candidate-prefix rows (A) partition the later suffix. EQ retains entity, question and cue rows but omits answer-prefix rows. Row-subset patches are zero elsewhere. Oracle patches use the target’s own edited-minus-clean states on those same rows.

Every unpatched continuation must reproduce clean logits within 10−310^{-3}. The separate cross-run tolerance is 10−410^{-4} in token log probability: the Qwen2.5-7B depth re-run differs by 1.05×10−41.05 \times10^{-4} in one of 3,192 construction/candidate comparisons after changing the float32 batch shape from 17 to 11 rows. Its curve therefore uses the earlier saved plain-format scan; the new target-relation row subsets and neutral-source runs are not reported.

Combining row responses. On the original 35 blocks in default encoding at d=1/2d = 1/2, adding the target’s separately continued entity-row and post-entity-row predictions as contrast changes removes 51.7, 55.3, 68.5, 54.5, 48.7, 61.9% of no-change error, in Qwen 0.5B, Llama 1B, Gemma 2B, Qwen 7B, Llama 8B and Gemma 9B order. Patching both row sets together instead removes 100% by reproducing the edit. These are errors evaluated after adding the predicted contrast changes, not sums of the two individual error-removal shares; absolute error is nonlinear.

Table 28 evaluates the full-vocabulary distribution at the answer cue, without choosing a candidate prefix. Table 29 instead lets greedy generation follow its own tokens, using one question-plus-cue capture. These results are not interchangeable with per-prefix candidate scoring: capture, construction and evaluation scope differ, so the page does not isolate autoregressive error accumulation. Low generation fidelity is reported alongside the stronger conditional-score results.

Standard Patches and the Single-Vector Comparison

The default-encoding comparison over 99 blocks favors span-aligned addition, by 0.12–0.28 nats with all six pointwise block-bootstrap intervals excluding zero (Table 16c). Coverage and construction change together, so that comparison does not isolate post-entity rows.

Table 30 reports the tested families, including the undetected entity-addition versus state-replacement differences and the family-specific knockout outcomes. The stronger single-vector extension selects among 90 constructions on the original cohort and evaluates on the two already-studied fresh cohorts; the selection and evaluation sets are distinct but neither is an unseen research setting. Directions, placements, averaging and amplitudes vary, so the comparison is between tested constructions, not an isolated effect of row structure. Table 31 and Table 32 report all six checkpoints and all four families, including failed and pooled-correction-sensitive tests.

CheckpointE1 WithinE1 LabelsE2 WithinE2 LabelsE3 WithinE3 LabelsE4 WithinE4 Labels
Qwen 0.5B[0.00, 0.27][−0.10, 0.37][0.53, 1.23][0.33, 1.62][−0.15, −0.02][−0.19, 0.03][−0.08, 0.50][−0.31, 0.79]
Llama 1B[0.21, 0.69][0.08, 0.87][0.45, 0.88][0.35, 1.05][−0.10, 0.01][−0.14, 0.04][0.27, 0.81][0.12, 0.99]
Gemma 2B[0.01, 1.05][−0.34, 1.41][0.57, 1.07][0.41, 1.25][−0.03, 0.12][−0.09, 0.18][1.24, 2.08][0.93, 2.49]
Qwen 7B[0.13, 1.80][−0.53, 2.45][0.69, 2.05][0.18, 2.50][−0.11, 0.24][−0.24, 0.37][1.55, 2.70][1.15, 3.16]
Llama 8B[0.88, 2.10][0.46, 2.57][0.16, 0.80][−0.11, 1.10][−0.09, 0.02][−0.13, 0.05][1.35, 2.24][1.00, 2.66]
Gemma 9B[1.45, 2.67][1.05, 3.04][0.74, 1.47][0.50, 1.77][−0.13, 0.02][−0.22, 0.08][1.74, 2.70][1.37, 3.06]
Share of blocks with positive differences, labels resampled
Qwen 0.5B0.59 [0.38, 0.79]†0.78 [0.60, 0.93]0.42 [0.27, 0.59]†0.48 [0.30, 0.67]†
Llama 1B0.69 [0.47, 0.88]†0.80 [0.62, 0.94]0.47 [0.32, 0.64]†0.67 [0.51, 0.81]
Gemma 2B0.58 [0.36, 0.78]†0.80 [0.60, 0.95]0.57 [0.42, 0.71]†0.77 [0.63, 0.89]
Qwen 7B0.59 [0.39, 0.78]†0.70 [0.52, 0.87]0.52 [0.34, 0.68]†0.73 [0.57, 0.86]
Llama 8B0.70 [0.49, 0.88]†0.59 [0.38, 0.80]†0.46 [0.30, 0.63]†0.82 [0.66, 0.94]
Gemma 9B0.77 [0.55, 0.93]0.81 [0.62, 0.96]0.47 [0.30, 0.66]†0.82 [0.68, 0.93]

Table 32. Mean signed error differences defined in Table 31 (nats) with 95% intervals resampling blocks within cohorts (Within) or relation labels, each block weighted by the product of its two labels’ counts (Labels; Appendix § 11). The lower panel gives the share of blocks with positive differences and its label-resampled interval; † marks intervals that include one half.

The 90 constructions. The single-pair vector is the inverse-distance-weighted change at the two other-relation sources’ last entity row. Its four placements are the target’s last entity row, every entity row, every row from the entity onward, or answer positions. Seven coefficients, 1/8, 1/4, 1/2, 1, 2, 4, 8, give 28 constructions. The sixteen-context vector averages last-entity-row changes with equal weights over those two sources, the two neutral templates and twelve further relation-neutral templates, giving another 28 at the same placements and coefficients. The twelve templates are Here is a note about X., Describe X., Who or what is X?, Write a few words about X., Give some facts about X., Summarize what is known about X., The topic is X., Consider X., Let us talk about X., I want to learn about X., Name one fact about X. and Have you heard of X?, rendered with the same cue and candidate prefix. Neither vector uses target-relation questions. Taking either vector at the answer cue instead and adding it at answer positions or every later row gives 28 more. The final six use either entity-vector direction at entity rows, every later row or answer positions, with each row scaled to the span-aligned response’s norm. All continue through the target’s remaining layers. The best construction minimizes mean block answer error on the original 35 blocks, with exact ties resolved by the fixed construction order, and is evaluated on the other 64 already-studied blocks. The norm-matched constructions borrow magnitude information from span-aligned reuse; this is not a comparison at equal information. The earlier D1c instead tunes the coefficient of one vector at each entity row.

Table 31: Confirmatory families of the steering-completion protocol, each six exact sign tests with Holm correction within the family, in each checkpoint’s default encoding. Cells give positive / negative block counts, the mean signed error difference in nats and the Holm-adjusted pp. E1 (one-sided, fresh cohorts, 64 blocks): the single-pair vector at every later row, at the coefficient chosen on the original cohort (Qwen 0.5B 1/8, Llama 1B 1/4, Gemma 2B 1/2, Qwen 7B 1/2, Llama 8B 1/2, Gemma 9B 1/2), has lower error than zero prediction. E2 (two-sided, 64 blocks): best-vector error minus span-aligned error, for the best of the 90 single-vector constructions chosen on the original cohort (Qwen 0.5B, single-pair vector at each entity row, coefficient 1; Llama 1B, norm-matched single-pair direction at the entity rows; Gemma 2B, norm-matched single-pair direction at every later row; Qwen 7B, norm-matched sixteen-context direction at every later row; Llama 8B, norm-matched sixteen-context direction at every later row; Gemma 9B, single-pair vector at each entity row, coefficient 1); for Qwen 0.5B and Gemma 9B this is the vector tuned for D1c, so their E2 tests repeat D1c on the same blocks (Table 30). E3 (two-sided, 99 blocks): single-pair error minus sixteen-context entity-vector error, both at each entity row at coefficient 1. E4 (one-sided, 99 blocks): the norm-matched single-pair direction at every later row has lower error than zero prediction. A dagger marks tests that do not pass the final 223-test pooled Holm correction (Table 25). For 80% power the true share of favoring blocks must reach 0.67–0.71 in E1, 0.68–0.71 in E2, 0.64–0.68 in E3 and 0.63–0.66 in E4 (last to first Holm rank).

Receiving-Question Dependence and Content Controls

The knockouts block attention from the answer positions above the interface. KO-E targets entity rows; KO-PQ targets other prompt rows except the first token and the final cue row. Table 33 reports their family-dependent outcomes and the shift of the clean answer contrast caused by KO-E. These are route interventions, not identified heads or features, and their clean-reference changes limit a mechanistic interpretation.

For the identical-content experiment, the two receiving questions get identical relation-neutral entity-row changes. The controls use either the other replacement’s neutral change or eight random perturbations with matched row norms. The original two test families remain in Table 34. NEU averages the changes from Tell me about X. and What do you know about X? with equal weights. DR maps the other replacement’s neutral change to the target’s entity span by relative-position interpolation, copying tokenwise when lengths agree; DRN rescales each DR row to NEU’s norm. RND uses independent Gaussian rows with NEU’s row norms, averaging selective errors over eight seeded draws. Both relation-swapped targets receive the same injected content for a given replacement. G1 tests NEU against RND and G2 against DR. Table 36 and Table 37 report absolute selective-error reductions, random-control sensitivity and predicted magnitudes: random changes are not universally ineffective, and generic route attenuation does not by itself establish content specificity. The same-replacement change outperforms the controls within the specified families, with pooled-correction exceptions in Table 25. Mean/share sensitivity intervals for the control comparisons are in Table 35; those for neutral content against zero are in Table 17b. On the 28–32 blocks per checkpoint where DR needs no span interpolation, NEU beats DR in 18–25 blocks; its uncorrected sign-test pp exceeds 0.05 only on Qwen2.5-7B (p=0.181p = 0.181). For the neutral content itself against zero, Llama-3.2-1B’s label-resampled mean interval is [−0.03, 0.37] nats, despite its original within-family pass (Table 17b).

∣s^∣\lvert\hat{s} \rvert (% of ∣s∣\lvert s \rvert)Share removed (%)
Checkpoint∣s∣\lvert s \rvertNEUDRDRNRNDNEUDRDRNRND
Qwen 0.5B1.3774.153.054.345.132.1−5.8−6.6−6.1
Llama 1B0.8980.548.952.631.144.50.3−0.83.3
Gemma 2B1.7674.738.239.721.450.70.3−0.13.5
Qwen 7B4.4876.443.043.337.637.65.15.24.9
Llama 8B2.5357.326.627.313.636.15.96.23.3
Gemma 9B2.5978.334.937.514.954.64.24.82.3

Table 37. Size of the predicted selective component and the full-metric share of the zero-prediction error removed by each identical entity-row construction over the 99 blocks: the mean observed ∣s∣\lvert s \rvert (nats), each construction’s mean ∣s^∣\lvert\hat{s} \rvert as a percentage of it (for RND, the mean over draws of each draw’s mean ∣s^∣\lvert\hat{s} \rvert), and the share removed on the complete answer contrast (%). Constructions as in Table 36.

References

  1. [1]Divyansh Agarwal. Relation before entity: Deferred commitment in language model factual recall, 2026. URL https://arxiv.org/abs/2609.17537. ICML 2026 Mechanistic Interpretability Workshop.
  2. [2]Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Petar Veličković, Razvan Pascanu, and Michael M. Bronstein. Why do LLMs attend to the first token? In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=tu4dFUsW5z.arxiv.org/abs/2504.02732
  3. [3]Haozhe Chen, Carl Vondrick, and Chengzhi Mao. SelfIE: Self-interpretation of large language model embeddings. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7373–7388, 2024. URL https://proceedings.mlr.press/v235/chen24ao.html.arxiv.org/abs/2403.10949
  4. [4]Jinkun Chen, Fengxiang Cheng, Sijia Han, and Vlado Keseљ. “I may not have articulated myself clearly”: Diagnosing dynamic instability in LLM reasoning at inference time, 2026. URL https://arxiv.org/abs/2602.02863.
  5. [5]Fan L. Cheng and Nikolaus Kriegeskorte. Decomposing how prompting steers behavior, 2026. URL https://arxiv.org/abs/2606.03093.
  6. [6]Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in LLMs, 2024. URL https://arxiv.org/abs/2402.07321. NeurIPS 2023 Workshop on Attributing Model Behavior at Scale.
  7. [7]Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283–298, 2024. doi: 10.1162/tacl_a_00644. URL https://aclanthology.org/2024.tacl-1.16/.DOI
  8. [8]Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31822–31841. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1536. URL https://aclanthology.org/2025.acl-long.1536/.
  9. [9]Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021. doi: 10.1162/tacl_a_00410. URL https://aclanthology.org/2021.tacl-1.60/.DOI
  10. [11]Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/9d1b7fc578c0d2d6431fc26d736ecaf3-Abstract-Conference.html.
  11. [12]Sheridan Feucht, David Atkinson, Byron C. Wallace, and David Bau. Token erasure as a footprint of implicit vocabulary items in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9727–9739. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.emnlp-main.543. URL https://aclanthology.org/2024.emnlp-main.543/.DOI
  12. [13]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems 34, pages 9574–9586, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/4f5c4221f4d49a5a807eda27434231040-Abstract.html.arxiv.org/abs/2106.02997
  13. [14]Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pages 160–187, 2024. URL https://proceedings.mlr.press/v236/geiger24a.html.arxiv.org/abs/2303.02536
  14. [15]Atticus Geiger, DuliGur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83):1–64, 2025. URL https://jmlr.org/papers/v26/23-0058.html.
  15. [16]Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118.
  16. [17]Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.751. URL https://aclanthology.org/2023.emnlp-main.751/.DOI
  17. [18]Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15466–15490, 2024. URL https://proceedings.mlr.press/v235/ghandeharioun24a.html.arxiv.org/abs/2401.06102
  18. [19]Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, and Christopher Potts. Addressing divergent representations from causal interventions on neural networks, 2025. URL https://arxiv.org/abs/2511.04638.
  19. [20]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  20. [21]Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357–362, 2020. doi: 10.1038/s41586-020-2649-2.
  21. [22]Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Advances in Neural Information Processing Systems 36, pages 17643–17668, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/3927bbdcf0e8d1fa8aa23c2358a281-Abstract-Conference.html.arxiv.org/abs/2301.04213
  22. [23]Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching, 2024. URL https://arxiv.org/abs/2404.15255.
  23. [24]Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.624. URL https://aclanthology.org/2023.findings-emnlp.624/.DOI
  24. [25]Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. In First Conference on Language Modeling, 2024. URL https://arxiv.org/abs/2304.00740.
  25. [26]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.DOI
  26. [27]Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.564. URL https://aclanthology.org/2021.emnlp-main.564/.
  27. [28]Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8669–8687. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.470. URL https://aclanthology.org/2024.acl-long.470/.
  28. [29]Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438, 2020. doi: 10.1162/tacl_a_00324. URL https://aclanthology.org/2020.tacl-1.28/.DOI
  29. [30]Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centering, 2023. URL https://arxiv.org/abs/2312.03813.
  30. [31]Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning, pages 333–342. Association for Computational Linguistics, 2017. doi: 10.18653/v1/K17-1034. URL https://aclanthology.org/K17-1034/.DOI
  31. [32]Jiaqian Li, Yanshu Li, and Kuan-Hao Huang. Steering vector fields for context-aware inference-time control in large language models, 2026. URL https://arxiv.org/abs/2602.01654. EMNLP 2026 main conference, to appear.
  32. [33]Kenneth Li, Om Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/81b8390039b7302c9098b769f8b6cd93-Abstract.html.arxiv.org/abs/2306.03341
  33. [34]Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html.
  34. [36]Srikant Malla, Chiho Choi, and Joon Hee Choi. Steering interference reflects the model’s defaults, not the behavior directions, 2026. URL https://arxiv.org/abs/2609.06951.
  35. [37]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35, pages 17359–17372, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html.arxiv.org/abs/2202.05262
  36. [38]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MkbcAHIYgyS.arxiv.org/abs/2210.07229
  37. [39]Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vec-style vector arithmetic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5030–5047. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.281. URL https://aclanthology.org/2024.naacl-long.281/.DOI
  38. [40]Meta. Llama 3.2 1B Instruct: Model card. https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct, 2024.
  39. [41]Neel Nanda, Senthooran Rajamanoharan, János Krámár, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level (Post 1). AI Alignment Forum, 23 December, 2023. URL https://www.alignmentforum.org/posts/iGuWZTHwb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall.
  40. [42]Jingcheng Niu, Xingdi Yuan, Tong Wang, Hamidreza Saghir, and Amir H. Abdi. Llama see, Llama do: A mechanistic perspective on contextual entrainment and distraction in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16218–16239. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.791. URL https://aclanthology.org/2025.acl-long.791/.DOI
  41. [43]Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8420–8436. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.458. URL https://aclanthology.org/2024.acl-long.458/.DOI
  42. [44]Louis Owen, Nilabhra Roy Chowdhury, Abhay Kumar, and Fabian Güra. A refined analysis of massive activations in LLMs, 2025. URL https://arxiv.org/abs/2503.22329.
  43. [45]Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future Lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548–560. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.conll-1.37. URL https://aclanthology.org/2023.conll-1.37/.DOI
  44. [46]Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.828. URL https://aclanthology.org/2024.acl-long.828/.
  45. [47]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alikhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.arxiv.org/abs/1912.01703
  46. [48]Qwen Team. Qwen2.5 technical report, 2024. URL https://arxiv.org/abs/2412.15115.
  47. [49]Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Andria Garriga-Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. In Advances in Neural Information Processing Systems 37, pages 139179–139212, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/fb3ad59a84799bfb8d700e56d19c231b-Abstract-Conference.html.
  48. [50]Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwxytyMwaG.arxiv.org/abs/2310.15213
  49. [51]Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https://arxiv.org/abs/2308.10248v5. Version 5.
  50. [52]Alexandre Variengien and Eric Winsor. Look before you leap: Universal emergent mechanism for retrieval in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://iclr.cc/virtual/2025/poster/28935.
  51. [53]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems 33, pages 12388–12401, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html.
  52. [54]Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Communications of the ACM, 57(10):78–85, 2014. doi: 10.1145/2629489.
  53. [55]Zijian Wang and Chang Xu. Functional abstraction of knowledge recall in large language models, 2025. URL https://arxiv.org/abs/2504.14496.
  54. [56]Zijian Wang, Britney Whyte, and Chang Xu. Locating and extracting relational concepts in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4818–4832. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-acl.287. URL https://aclanthology.org/2024.findings-acl.287/.DOI
  55. [57]Jacek Wiland, Max Ploner, and Alan Akbik. BEAR: A unified framework for evaluating relational knowledge in causal and masked language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2393–2411. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-naacl.155. URL https://aclanthology.org/2024.findings-naacl.155/.DOI
  56. [58]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/.DOI
  57. [59]Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. AxBench: Steering LLMs? Even simple baselines outperform sparse autoencoders. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 67035–67080, 2025. URL https://proceedings.mlr.press/v267/wu25a.html.arxiv.org/abs/2501.17148
  58. [60]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://iclr.cc/virtual/2024/poster/18794.arxiv.org/abs/2309.17453
  59. [61]Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924–9959. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.615. URL https://aclanthology.org/2023.emnlp-main.615/.DOI
  60. [62]Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/06a52a54c8ee03cd86771136b9c91eb1f-Abstract-Conference.html.arxiv.org/abs/2309.16042
  61. [63]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sammi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency, 2023. URL https://arxiv.org/abs/2310.01405.

Paper details

vlnc:W_3h8wdk2f
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents