betaFeedback

WorksComputer Science

RouteVerify: Routing Interventions for Step-Level Verification in Mixture-of-Experts Models

Abstract

Routing states in mixture-of-experts (MoE) models can distinguish correct from erroneous reasoning steps. If changing routing also changes a model's step-level verification judgment, it is tempting to conclude that the judgment follows what routing encodes. We test this inference with RouteVerify, a controlled intervention study that replaces selected internal activations from one solution trace with activations from another, while keeping the evaluated input unchanged. Across nine MoE checkpoints, we compare three intervention interfaces: router logits, MoE outputs, and post-MoE residual states. We vary the layers and token positions at which activations are replaced. We measure changes in a verification margin that quantifies the model's preference for INCORRECT over CORRECT. We also evaluate the same input with activations from correct versus erroneous steps to test whether the resulting judgment depends on the source step's correctness. We report four linked results. First, probes on router logits separate correct from erroneous steps in eight checkpoints (AUROC 0.80-0.88). Second, replacing router logits across all routed layers changes the judgment in all nine checkpoints, and by more than a norm-matched random replacement. Third, this change rarely depends on whether the source step is correct. At one layer, routing from erroneous steps and from correct steps produces indistinguishable margins in all nine checkpoints. Under all-layer routing, their difference is resolved and positive in only one. Fourth, the same difference is positive for local residual replacements in eight of nine checkpoints, so the measure does detect such dependence where it exists. Correctness is therefore readable from routing, and routing replacements move the judgment, but the movement rarely follows correctness.

Introduction

Process verification asks a model to determine whether an intermediate reasoning step is correct, rather than only judging the final answer (Lightman et al., [13]). Recent work shows that internal representations can reveal output correctness (Orgad et al., 2025), while hidden-state interventions can also change step-level verification behavior (Zhou et al., 2026). In mixture-of-experts (MoE) models, routing is a natural place to look for such a signal: routing states can distinguish correct from erroneous steps, and routing can be intervened on through expert selection and weighting. If replacing routing also changes the model’s judgment, it is tempting to conclude that the judgment follows what routing encodes. This paper tests that inference. We ask when and where routing interventions change a step-level verification judgment, and whether the change depends on the correctness of the step.

We investigate this question with RouteVerify (Figure 1), a controlled activation-replacement study across nine MoE checkpoints. We use paired solution traces for the same problem: one is correct, and the other contains its first error at the step being evaluated. The model is asked to judge that step as CORRECT or INCORRECT. We record internal activations while the model processes one trace and replace selected activations in its computation on the other trace. The example supplying the activations is the donor; the example whose activations are replaced is the receiver. The receiver’s input text remains unchanged, and we measure the effect on its verification judgment.

Figure 1. Overview of ROUTEVERIFY. Activations from a donor example replace selected activations in a receiver example, whose input text remains unchanged and whose verification judgment is measured. We vary the intervention interface, layers, and token positions. TT summarizes the signed margin changes from the two opposite-direction replacements within a correct/error pair. DD compares activations from correct and erroneous donors from another problem while holding the receiver fixed. Positive DD means that the erroneous donor produces a more INCORRECT-favoring margin than the correct donor.

We quantify judgment changes using a verification margin: the difference between the model’s length-normalized log probabilities for INCORRECT and CORRECT, with larger values favoring INCORRECT. A large change in this margin, however, does not necessarily mean that the model responds to whether the donor step is correct. For example, suppose the model initially favors the appropriate label for each member of a correct/error pair. An intervention that pushes both judgments toward neutrality moves each margin toward the other example’s label, even if the effect does not depend on the donor step’s correctness. We therefore distinguish changes produced by exchanging activations within a pair from differences produced by correct versus erroneous donors on the same receiver. For the latter comparison, we keep the receiver fixed and obtain both donors from a matched pair associated with another problem.

We compare three intervention interfaces: router logits, which score candidate experts; MoE outputs, which contain the routed experts’ combined contribution; and post-MoE residual states, which contain the representation after the MoE residual update. We vary both the layers and the token positions at which activations are replaced. A token window specifies the selected positions within the evaluated reasoning step, such as its final token, its final eight tokens, or the entire step. This design allows us to examine both the location and the extent of an intervention.

Our results form one finding in four parts. First, the information is readable: out-of-fold probes on router logits distinguish correct from erroneous steps in eight checkpoints, with AUROC from 0.80 to 0.88 (§ 6). Second, routing replacements change the judgment. At the same layer and token window, post-MoE residual replacements produce larger paired margin effects than router-logit or MoE-output replacements in all 36 comparisons across nine checkpoints and four windows. Router-logit interventions become substantially stronger when expanded across routed layers and tokens, and comparing two eight-token windows at different positions reveals systematic position dependence. Across all routed layers, they change the judgment in all nine checkpoints, and by more than a normmatched random control. Third, this change rarely depends on donor correctness. With the receiver fixed, erroneous and correct donors produce indistinguishable margins under single-layer routing in all nine checkpoints. Under all-layer routing, their difference is resolved and positive in only one. In six checkpoints, the upper interval bound of that difference is below 15% of the all-layer routing effect (Appendix § 23.§ 4). Fourth, the same measure detects correctness dependence where it exists: local residual interventions produce more INCORRECT-favoring margins with erroneous than with correct donors in eight of nine checkpoints. Together, these results show that correctness is readable from routing and that routing replacements move the judgment, yet the movement rarely tracks donor correctness. Changing a verification judgment and changing it according to the donor step’s correctness are distinct outcomes, both of which depend on where and how broadly the intervention is applied.

We distinguish the size of a paired judgment change, TT, from its dependence on donor correctness, DD, measured on a fixed receiver. With these two measures, we contribute four linked results:

  • Router logits are decodable: probes on router logits distinguish correct from erroneous steps in eight checkpoints.

  • Router replacements change the judgment: across all routed layers, TT is positive in all nine checkpoints and exceeds a norm-matched random control. We also show how routing effects change with layer coverage and token position, against MoE outputs and residual states at the same layer and tokens.

  • The change does not follow donor correctness: router DD is indistinguishable from zero at one layer in all nine checkpoints and is resolved positive under all-layer routing in only one. A large intervention effect need not be correctness-sensitive.

  • The same measure detects correctness in the residual state: residual DD is positive in eight of nine checkpoints, so an unresolved router DD is not a failure of the measure.

Intervention Design

Verification task and score

Our evaluation set is derived from PRMBench, a benchmark with step-level annotations for evaluating fine-grained error detection in reasoning traces (Song et al., 2025). We construct 290 pairs of solution traces. In each pair, one trace is labeled correct and the other contains its first annotated error at the evaluated step. The model receives the problem, the solution trace through this step, and a fixed instruction to judge the step as CORRECT or INCORRECT. Later steps and analysis metadata are not provided. We write C and E for the correct and error-bearing members of a pair, respectively.

Rather than recording only which label the model prefers, we measure its relative preference using a continuous verification margin:

M(x)=∣yI∣−1log⁡p(yI∣x)−∣yC∣−1log⁡p(yC∣x),(1)M(x) = |y_I|^{-1}\log p(y_I \mid x) - |y_C|^{-1}\log p(y_C \mid x), \tag*{(1)}

where yIy_I and yCy_C are the model-specific token sequences for INCORRECT and CORRECT, and ∣y∣|y| denotes the number of tokens in a sequence. Positive values favor INCORRECT; negative values favor CORRECT. An increase in the margin therefore indicates a shift toward an INCORRECT judgment, although it need not change the preferred label. We use average log probability per token so that the comparison does not directly reflect the summation of different numbers of token log probabilities. We also report total-logprob margins as a sensitivity analysis.

Activation replacement and two intervention measures

Donors, receivers, and replacement. We call the example supplying recorded activations the donor and the example whose activations are replaced the receiver. The receiver retains its original problem, solution text, and verification instruction. Only the selected internal activations are replaced; the model then continues its computation and scores the two judgment labels.

For an intervention setting ss, let hsrh_s^r and hsdh_s^d denote the corresponding receiver and donor activations. We replace the receiver activations according to

h~sr=hsr+α(hsd−hsr),(2)\tilde{h}_s^r = h_s^r + \alpha(h_s^d - h_s^r), \tag*{(2)}

where α\alpha controls the replacement strength. All new experiments use α=1\alpha= 1, corresponding to exact replacement. The setting ss specifies the intervention interface and its layer and token coverage. Let xr←d,sx_{r\leftarrow d,s} denote the receiver evaluated under this intervention. Its margin change is

Δd→r,s=M(xr←d,s)−M(xr).(3)\Delta_{d\to r,s}=M\left(x_{r\leftarrow d,s}\right)-M\left(x_r\right). \tag*{(3)}

Paired directional effect, TT. For each correct/error pair, we perform two replacements: activations from the erroneous trace are inserted into the correct receiver, and activations from the correct trace are inserted into the erroneous receiver. We summarize these two changes as

Ts=12(ΔE→C,s−ΔC→E,s).(4)T_s=\frac{1}{2}\left(\Delta_{E\to C,s}-\Delta_{C\to E,s}\right). \tag*{(4)}

The signs are chosen so that an increase in the correct receiver’s margin and a decrease in the erroneous receiver’s margin both contribute positively. Thus, positive TT indicates an average shift toward the donor step’s label across the two replacements; it does not require both replacements to move in that direction.

However, TT can also be positive when an intervention simply reduces the original separation between the two receivers:

2Ts=[M(xE)−M(xC)]−[M(xE←C,s)−M(xC←E,s)].(5)2T_s=\left[M\left(x_E\right)-M\left(x_C\right)\right]-\left[M\left(x_{E\leftarrow C,s}\right)-M\left(x_{C\leftarrow E,s}\right)\right]. \tag*{(5)}

For example, an intervention that makes both receivers’ margins approach zero can yield positive TT when their original margins favor their respective labels. Consequently, TT measures a signed paired intervention effect, but does not by itself establish that the effect depends on whether the donor step is correct or erroneous.

Fixed-receiver donor contrast, DD. To test this dependence, we compare correct and erroneous donors on the same receiver. For each receiver pair ii, we select an external matched pair jj from another problem. We then evaluate each receiver r∈{Ci,Ei}r\in\{C_i,E_i\} twice at the same intervention setting: once with activations from the correct donor and once with activations from the erroneous donor. We define

Di,r,s=M(xr←Ej,s)−M(xr←Cj,s),Di,common,s=12(Di,Ci,s+Di,Ei,s).(6)D_{i,r,s}=M\left(x_{r\leftarrow E_j,s}\right)-M\left(x_{r\leftarrow C_j,s}\right),\qquad D_{i,\mathrm{common},s}=\frac{1}{2}\left(D_{i,C_i,s}+D_{i,E_i,s}\right). \tag*{(6)}

Positive DD means that, for the same receiver, the erroneous donor produces a more INCORRECT-favoring margin than the correct donor. Because the receiver is held fixed, its baseline margin cancels from this comparison. This contrast tests differences between matched donor steps of different correctness; it does not isolate correctness from every other difference between their activations. Appendix § 23 tests this contrast against a same-label placebo directed by a lexical proxy.

TT and DD therefore answer different questions. TT summarizes signed changes from the two cross-role replacements within a correct/error pair, whereas DD compares two donor roles on an unchanged receiver. Because they use different donor–receiver comparisons, neither D/TD/T nor T−DT-D has a decomposition interpretation.

Interfaces, layer coverage, and token windows

Intervention interfaces. We replace activations at three points in an MoE block. A router-logit intervention replaces the scores assigned to candidate experts before expert selection. The model then uses its native expert selection, weighting, and dispatch procedures. A MoE-output intervention replaces the adapter-defined routed contribution before it is added to the residual stream. A post-MoE residual intervention replaces the resulting residual state, which can already contain information from attention, experts, and earlier layers.

Where an architecture includes a shared-expert or parallel dense branch, that branch remains receiver-derived under the MoE-output intervention (§ 3). Although all three interfaces are evaluated using the same verification margin, they replace different amounts of computation. A residual replacement swaps more of the computation than a router replacement, so a larger residual effect does not show that routing is unimportant. The comparison within the routed-update scope is router against MoE output: routing acts through the routed update, whereas an output replacement directly replaces that update. The three interventions should not be interpreted as additive stages of a single mediation decomposition.

Layer and token coverage. For local comparisons, we use one fixed, model-specific layer, referred to as the anchor layer. At this layer, we compare all three interfaces under four token windows, each specifying the positions within the evaluated reasoning step at which activations are replaced: the last token; the final eight tokens (suffix-8); eight tokens around the first token at which the paired prefixes diverge (divergence-8); and the full step.

We additionally apply a joint all-router intervention, replacing router logits at every routed layer under the same four token windows. Comparing suffix-8 with divergence-8 holds the number of intervened tokens fixed while changing their position. It does not, however, equalize the size of the activation changes or the computation preceding those positions. Exact window construction and donor–receiver token alignment are given in Appendix § 14.

We evaluate DD both locally, at one layer and the final step token, and in a separate experiment that replaces router logits across all routed layers and the full step (Appendix § 17). We also evaluate an all-layer MoE-output intervention at the last token; this experiment is reported with TT only.

Controls and uncertainty

Self-patches, which replace receiver activations with their own recorded values, and unpatched reruns verify that the intervention machinery preserves the receiver when no cross-example replacement is made. Matched external donors come from other problems, prioritizing agreement in error type and proximity of the error position, and remain fixed during analysis. Norm-random controls replace the donor–receiver activation difference with a Gaussian direction having the same ℓ2\ell_2 norm. For TT, control donors preserve the donor role of each cross-role replacement; for DD, both donor roles are evaluated on each fixed receiver.

Pairs receive equal weight. To account for dependence among pairs associated with the same source problem, we resample the K=235K = 235 source-problem components and retain all pairs belonging to each sampled component. Last-token, full-step, DD, and all-layer-output analyses use all N=290N = 290 pairs. Eight-token and direct-contrast analyses use the corresponding common eligible subsets, with unavailable windows treated as missing rather than zero.

Primary intervals are two-sided 95% pointwise percentile intervals from 50,000 bootstrap resamples generated with PCG64. Direct differences are formed within pair before resampling, and reused matched donors are conditioned on rather than resampled independently. Exact seeds and version-specific analysis details are reported in Appendices § 14 and § 17.

The checkpoints are fixed and the discovery sample was assembled retrospectively, so these intervals do not estimate a population distribution over models. We apply no multiplicity correction; post hoc receiver-role and GPT-checkpoint contrasts are reported as secondary analyses.

Models and Intervention Boundaries

We evaluate nine MoE checkpoints: Qwen3.6-35B-A3B, Qwen3-30B-A3B, GLM-4.7-Flash, Nemotron-3.5 Lightning 30B-A3B-BF16, Nemotron-3 Nano 30B-A3B, Gemma 4 26B-A4B, OLMoE-1B-7B, and GPT-OSS-20B/120B [17, 19] (Yang et al. 2025; Z.ai 2026).

Each checkpoint uses a fixed architecture-specific anchor layer for local interventions, while all-layer interventions cover every routed block in that checkpoint. Appendix § 13 reports model sources, anchor indices, and routed-layer counts (Table 2). Qwen3 is the only checkpoint for which the local DD and token-window experiments use different anchors: L36 and L24, respectively. Its probe protocol is described separately.

ModelLocal DD layerWindow layerAll routed sitesProbe layer
Qwen3.6-35B-A3B20204020
Qwen3-30B-A3B36244824
GLM-4.7-Flash24244624
Nemotron-3.5 Lightning27272327
Nemotron-3 Nano27272327
Gemma 4 26B-A4B15153015
OLMoE-1B-7B8816—
GPT-OSS 20B12122412
GPT-OSS 120B18183618

Table 2. Layer coverage in the nine-checkpoint analysis. Layers are zero-indexed absolute decoder-block indices. “All routed sites” counts routed blocks replaced by an all-router or all-layer MoE-output intervention, not every decoder block. “Probe layer” refers only to completed representation analyses in Appendix § 18.

The MoE-output intervention follows each architecture’s native computation boundaries. For GLM, Qwen3.6, and the Nemotron models, we replace the routed experts’ contribution while retaining the receiver’s shared-expert branch. For Gemma, we replace the normalized routed branch while retaining the receiver’s parallel dense branch. For Qwen3, OLMoE, and GPT-OSS, the intervention replaces the native MoE/MLP-module output. Exact adapter definitions and execution checks are reported in Appendices § 13.§ 3 and § 20, respectively.

Interface and Coverage Dependence on PRMBench

Before comparing interventions, we record the unpatched verification performance on this paired assay. Reading M>0M > 0 as an INCORRECT verdict, accuracy ranges from 0.567 [0.548, 0.588] in OLMoE to 0.786 [0.755, 0.816] in Gemma and AUROC from 0.602 to 0.874, so every checkpoint is above the 0.5 chance level of this balanced assay while verification quality differs substantially (Appendix § 16).

Local residual interventions are stronger across token windows

Across nine models and four token windows, post-MoE residual interventions produce larger local responses (TT) than both router-logit and MoE-output interventions: all 36 residual-minus-router and all 36 residual-minus-output direct intervals are positive. This ordering therefore persists beyond the last-token setting and across different token regions within the evaluated step. The magnitude varies substantially across checkpoints; full-step residual TT, for example, ranges from 0.1300 in Nano to 4.8698 in Gemma (Figure 2B). These values compare intervention interfaces within a checkpoint and should not be interpreted as model-quality rankings. Normalizing by the baseline margin gap of the same pairs, R=mean⁡(T)/mean⁡(g)R = \operatorname{mean}(T)/\operatorname{mean}(g) with g=M(xE)−M(xC)g = M(x_E) - M(x_C), places full-step residual effects between 0.24 [0.22, 0.26] in GPT-OSS-120B and 0.90 [0.88, 0.93] in GPT-OSS-20B; Gemma, which has the largest absolute TT, has R=0.69R = 0.69 [0.65, 0.72]. In GPT-OSS-20B, Qwen3, Qwen3.6, and Gemma the RR interval leave zero above 0.5, so the intervention does not merely compress the paired gap in Equation (5) but reverses its sign on average. These four checkpoints have patched accuracy of 0.27 to 0.37 after this intervention (Appendix § 16).

Figure 2. Interface and coverage effects on the verification margin. A: positions of the divergence-8 and suffix-8 windows within an evaluated step. B: full-step TT for single-layer router, all-router, and residual interventions. C: the full-step residual-minus-all-router difference. D: the change in this difference from suffix-8 to divergence-8; negative values indicate a smaller residual-versus-routing gap near the first divergence. Error bars are pointwise 95% source-component percentile intervals from 50,000 resamples; direct contrasts use common eligible pairs. Panels B and D use symmetric-log axes.

Local residual effects depend on donor correctness

We next ask whether the intervention direction depends on whether the replacement activation comes from a correct or erroneous step. At one layer and the last token, residual DD is positive on both correct and error receivers in eight of nine models (Table 1). Residual-minus-router and residual-minus-output contrasts in DcommonD_{\mathrm{common}} are also positive in these models. The comparison within the routed-update scope is router against MoE output (§ 2.3). MoE-output DD is positive on both receiver types in five of nine models, while every single-router DcommonD_{\mathrm{common}} interval includes zero.

ModelResidual DcorrectD_{\mathrm{correct}}Residual DerrorD_{\mathrm{error}}Router DcommonD_{\mathrm{common}}Output DcommonD_{\mathrm{common}}
Qwen3.6†−.0096 [−.0308,.0130].0356 [.0101,.0625]−.0010 [−.0085,.0062]−.0053 [−.0120,.0010]
Qwen3.7282 [.6313,.8269].5921 [.4989,.6879]−.0073 [−.0151,.0006].0519 [.0339,.0698]
GLM-4.7-Flash.1989 [.1594,.2404].1371 [.1077,.1673].0009 [−.0062,.0080].0642 [.0475,.0806]
Lightning.1492 [.1285,.1704].1125 [.0953,.1304]−.0001 [−.0035,.0034]−.0003 [−.0041,.0036]
Nano.0793 [.0680,.0907].0624 [.0525,.0726].0005 [−.0017,.0026].0004 [−.0024,.0032]
Gemma41.1705 [.8775,1.4791].2586 [.1150,.4163]−.0061 [−.0208,.0087].1188 [.0681,.1793]
OLMoE.1368 [.0859,.1925].1586 [.1049,.2169].0050 [−.0011,.0110].0349 [.0211,.0487]
GPT-OSS-20B.1352 [.1016,.1696].1019 [.0726,.1323]−.0017 [−.0060,.0027]−.0014 [−.0108,.0079]
GPT-OSS-120B.2262 [.1895,.2645].1654 [.1342,.1977].0061 [−.0025,.0150].0436 [.0312,.0564]

Table 1. Fixed-receiver DD at the last token. Positive values mean that, on the same receiver, activations from an erroneous step produce a more INCORRECT-favoring margin than activations from a correct step. Residual DD is positive on both receiver types in eight models; Qwen3.6 (†) is the exception. A substantial part of OLMoE’s residual DD is associated with surface content (Appendix L). Router and output columns report DcommonD_{\mathrm{common}}, while residual columns retain the two receiver types. All panels use N=290N = 290 pairs and K=235K = 235 source components with pointwise 95% percentile intervals.

A same-label placebo tests whether the residual contrast exceeds differences associated with a lexical proxy (Appendix § 23). Six of eight eligible models pass. Qwen3.6 has no resolved positive residual DcommonD_{\mathrm{common}}, and a substantial part of OLMoE’s residual DD is associated with surface content. The supported interpretation is dependence on donor correctness beyond this lexical proxy, rather than pure correctness.

Qwen3.6 shows clear receiver dependence. Its residual DD is −0.0096 [−0.0308, 0.0130] on correct receivers and 0.0356 [0.0101, 0.0625] on error receivers, with a direct difference of −0.0452 [−0.0748, −0.0156]. Its DcommonD_{\mathrm{common}} = 0.0130 [−0.0054, 0.0327] therefore does not establish a common positive direction. Additional receiver asymmetries are reported in Appendix § 15.

Broader routing coverage changes the interface ordering

Single-layer router interventions are weak, but this does not bound the effect of intervening on routing more broadly. Full-step all-router TT is positive in all nine models and increases over the corresponding single-router interventions (Figure 2B). At full-step coverage, the residual-minus-all-router gap remains positive in six models, is unresolved in Lightning and Nano, and reverses in GPT-OSS-120B (−0.1083 [−0.1757, −0.0419]; Figure 2C). At divergence-8, all-router also exceeds residual in both Nemotron checkpoints and GPT-OSS-120B. Routing can therefore become comparable to, or stronger than, a local residual intervention when coverage expands across layers and tokens.

Increasing token coverage alone does not produce a common trend in the residual-versus-routing gap. A clearer pattern appears when token count is held fixed. Define Gw=Tres,w−Tall−router,wG_w = T_{\mathrm{res},w} - T_{\mathrm{all-router},w}. Moving from suffix-8 to divergence-8 reduces GwG_w in all nine models, with every direct interval for Gdivergence8−Gsuffix8G_{\mathrm{divergence8}} - G_{\mathrm{suffix8}} below zero (Figure 2D). Thus, the relative effect of routing depends systematically on token position, not only on the number of tokens intervened on.

This positional effect does not imply that the absolute routing response is always larger near divergence. For GPT-OSS-120B, the all-router divergence-minus-suffix difference is −0.0028 [−0.0591, 0.0535], whereas the residual difference is −0.1946 [−0.2315, −0.1589]. Here, the smaller residual-versus-routing gap near divergence is driven primarily by residual sensitivity to position.

The two GPT-OSS checkpoints further separate intervention magnitude from the correctness dependence tested below. GPT-OSS-120B has the larger full-step all-router TT (120B − 20B = 0.1316 [0.0704, 0.1935]), motivating a direct test of whether it also shows the stronger fixed-receiver effect.

Broad routing correctness dependence varies across models and receivers

We extend the fixed-receiver analysis across all routed layers and the full evaluated step. Under this broader routing intervention, correctness-dependent effects are substantially less uniform than the local residual effects above. GPT-OSS-20B has positive Dcommon=0.0699D_{\mathrm{common}} = 0.0699 [0.0181, 0.1226] (Figure 3), with Dcorrect=0.1137D_{\mathrm{correct}} = 0.1137 [0.0480, 0.1803] and Derror=0.0262D_{\mathrm{error}} = 0.0262 [−0.0335, 0.0871]. Gemma also shows a positive correct-receiver effect, 0.3875 [0.0762, 0.7002], although its Dcommon=0.2207D_{\mathrm{common}} = 0.2207 [−0.0114, 0.4545] remains unresolved. The other checkpoints likewise do not establish a common positive effect. Relative to the all-router TT, the upper interval bound of DcommonD_{\mathrm{common}} is below 15% in six checkpoints (Appendix § 23.§ 4).

Figure 3. Fixed-receiver effects for full-step all-router interventions. Positive DD means that, on the same receiver, activations from an erroneous step produce a more INCORRECT-favoring margin than activations from a correct step. Circles and squares show the two receiver types, and diamonds show their average DcommonD_{\mathrm{common}}. Error bars are pointwise 95% source-component percentile intervals (N=290N = 290, K=235K = 235, 50,000 resamples). The lower strip reports direct GPT-OSS checkpoint contrasts for TT and DcommonD_{\mathrm{common}}.

Direct contrasts confirm that response magnitude and correctness dependence can order checkpoints differently. In GPT-OSS-20B, Dcorrect−Derror=0.0874D_{\mathrm{correct}} - D_{\mathrm{error}} = 0.0874 [0.0158, 0.1597], indicating a larger effect on correct receivers. More importantly, the paired GPT-OSS-20B-minus-120B contrast in DcommonD_{\mathrm{common}} is 0.0751 [0.0206, 0.1301], while GPT-OSS-120B itself has Dcommon=−0.0051D_{\mathrm{common}} = -0.0051 [−0.0393, 0.0290]. Thus, GPT-OSS-120B has the larger broad-routing TT, but GPT-OSS-20B has the larger fixed-receiver DD. A larger routing-induced judgment change therefore does not necessarily imply a stronger dependence on the correctness of the replacement activation.

These checkpoint contrasts do not isolate model scale: training, routed depth, expert count, and checkpoint identity also differ. Likewise, local residual DD and broad router DD use different layer and token coverage and therefore do not provide a matched-coverage interface comparison. Additional scoring sensitivities and post hoc contrasts are reported in Appendix F.

All-Layer Output Interventions and Donor Controls

Single-layer MoE-output interventions can be weak, so we next test whether replacing MoE outputs jointly across all routed layers produces a stronger response. At the last token, the all-layer MoE-output intervention has positive TT in all nine checkpoints, ranging from 0.0994 in Qwen3.6 to 1.1326 in Gemma (Table 6). Moreover, every paired-donor-minus-norm-random interval is positive. Thus, the effect of the all-layer output intervention cannot be explained by activation displacement magnitude alone.

Matched external donors from other problems also produce substantial responses. All paired-donor-minus-matched-donor intervals include zero, so the current data do not resolve an advantage for the paired donor over a matched external donor; nor do they establish equivalence. The result instead shows that large MoE-output intervention effects can also arise from donor activations drawn from other problems with matched error type and boundary position.

We apply the same controls to the broader router interventions. Full-step all-router TT exceeds the norm-random control in all nine checkpoints, although the contrast is close to zero for OLMoE (0.0466 [0.0003, 0.0938]). By contrast, matched external donors produce larger TT than the paired donors in eight checkpoints, with OLMoE unresolved. Because matched donors can differ in activation magnitude, context, and structure, this comparison should not be interpreted as evidence that they contain more correctness-related information. Full control results are reported in Appendix § 15.

A decision-level reading gives the same picture. The full-step all-router intervention changes the verdict of 0.06 to 0.23 of receivers under the paired donor, whereas the norm-random control changes 0.03 to 0.11 and the matched external donor 0.07 to 0.35, exceeding the paired donor in all nine checkpoints; a flip rate on its own therefore does not establish donor or routing specificity (Appendix § 16).

These controls distinguish intervention magnitude from two stronger claims. Exceeding the norm-random control shows that the observed response is not explained by displacement norm alone, while the matched-donor comparison does not establish pair-specificity. Neither comparison determines whether the judgment change follows the correctness of the replacement activation; that question requires the fixed-receiver DD analysis in § 4.4. The all-layer MoE-output intervention currently has no corresponding fixed-receiver DD experiment. Because the output, router, and residual experiments also differ in interface and coverage, they should not be interpreted as a router-to-output-to-residual mediation decomposition.

Representational Signals and Natural-Error Interventions

Using out-of-fold L2-logistic probes grouped by source problem, router-logit representations distinguish correct from erroneous steps across eight evaluated checkpoints, with AUROC from 0.801 to 0.884 (Appendix § 18; probes are unavailable for OLMoE). Yet a readable signal does not by itself determine what an intervention on it does. Router DcommonD_{\mathrm{common}} is unresolved at one layer in all nine checkpoints (§ 4.2) and under all-layer routing in eight (§ 4.4), although all-layer routing changes the judgment in every checkpoint (§ 5).

The interface difference also appears on naturally occurring errors: residual interventions produce larger absolute changes than router-logit interventions in all six model-by-direction comparisons. This exploratory ProcessBench analysis uses GPT-OSS-20B, Gemma, and Nemotron-3 Nano with 64 cross-problem pairs; it does not include fixed-receiver DD or token-window analyses (Appendix § 19).

Related Work

Error signals and step-level verification. Process supervision and fine-grained reasoning benchmarks motivate step-level verification (Lightman et al., [13]; Song et al., 2025; Zheng et al., 2025). Prior work shows that internal representations can contain information about truthfulness, reasoning correctness, or answerability that is not directly reflected in model outputs (Orgad et al., 2025; Yuan et al., 2026; Du & Hu, [3]), and intermediate-state probes have been used for self-verification and early exit (Zhang et al., 2025). Yuan et al. also report limited success in improving final-answer accuracy; we measure verification-margin changes on fixed step prefixes. Other work directly intervenes on verifier representations, for example by steering verifier strictness (Zhou et al., 2026). Our work connects these two lines by asking not only whether error-related information is present, but also when changing router, MoE-output, or residual activations alters the verification judgment and whether that change depends on the correctness of the replacement activation.

MoE routing and intervention. MoE routing reflects semantic structure (Olson et al., [18]) and hidden-state geometry (Wang et al., 2026), while other studies analyze expert specialization, the upstream formation of routing decisions, and experts that are important for generation (Herbst et al., [10]; Li et al., [12]; Su et al., 2026). Intervening on experts or routing can also change model behavior (Fayaz et al., 2026; Bandarkar et al., [1]), and recent work has begun to separate routing diagnostics from behavioral influence under ablation, quantization, or controlled gate changes (Engmann et al., [5]; Gu, [8]; Caruzzo et al., [2]). Related analyses study router-visible subspaces, expert allocation, factual recall, and expert selectivity (Ye et al., 2026; Yoon et al., 2026; Lu et al., [14]; Salomone et al., 2026). We complement these studies with a controlled comparison of router-logit, MoE-output, and post-MoE residual interventions, varying both layer coverage and token position under a common step-level verification endpoint.

Interpreting activation replacement. Causal tracing and Patchscopes study model behavior through activation or representation replacement [16, 7] (Vig et al. 2020). A central limitation of such analyses is that information readable from a representation need not be causally used [11, 4] (Sharma et al. 2026), while patching conclusions can depend on the intervention construction, evaluation metric, and the layers and token positions patched (Zhang & Nanda, 2024). Subspace interventions and causal localization introduce further interpretational limitations [15, 9]. Following these concerns, we separately measure the size of an intervention-induced judgment change and, with the receiver held fixed, whether the direction of that change depends on whether the replacement activation comes from a correct or erroneous step. We do not interpret either quantity as a pathway contribution or causal fraction.

Limitations

Activation replacement can move model states off their natural distribution, and its effects depend on the intervention site, evaluation margin, and the amount of computation being replaced [15] (Zhang & Nanda 2024). Moreover, a positive fixed-receiver DD shows that the intervention effect depends on whether the replacement activation comes from a correct or erroneous step, but it does not isolate correctness from correlated differences in text or representation geometry. The same-label placebo supports dependence beyond the tested lexical proxy in six of eight eligible models; it does not rule out other surface features (Appendix L). Our interventions operate at the interface level and do not localize the effect to individual experts. Finally, our nine checkpoints form a fixed, retrospectively assembled set rather than a sample from a model population. Accordingly, our results characterize the tested interventions and settings; they do not identify unique computational pathways or causal fractions of the model’s verification process.

Conclusion

We introduced ROUTEVERIFY to study when and where routing interventions affect step-level verification in MoE models, and whether the change depends on the correctness of the step. Four linked results answer this question. First, correctness is readable from routing: probes on router logits distinguish correct from erroneous steps in eight models, with AUROC from 0.80 to 0.88.

Second, routing replacements change the judgment. At a single layer, routing is the weakest lever: residual replacements change the judgment more than router-logit or MoE-output replacements in all 36 comparisons. Replacing router logits across all routed layers or the full step increases the effect in all nine models and exceeds the local residual effect in GPT-OSS-120B. This all-layer effect is larger than that of a norm-matched random control in all nine models. Token position also matters: moving an eight-token window from the end of the step to the first divergence between the correct and erroneous solutions reduces the residual-versus-all-router gap in all nine models. This is a relative effect; it does not require the absolute routing response to increase near divergence.

Third, this change rarely depends on donor correctness. We measure the dependence with DD: holding the receiver fixed, we compare its judgments under activations from a correct and an erroneous donor step. At one layer and the last token, router DD is indistinguishable from zero in all nine models. Under all-layer routing, GPT-OSS-120B has a larger paired effect TT than GPT-OSS-20B, but only GPT-OSS-20B has a resolved positive DcommonD_{\mathrm{common}}.

Fourth, the same measure detects correctness dependence in the residual state: at the last token, residual DD is positive in eight of nine models. A substantial part of OLMoE’s residual DD, however, is associated with surface content.

Correctness is readable from routing, and routing replacements move the judgment, but the movement rarely follows correctness. The size of an intervention effect must be distinguished from its dependence on donor correctness, and routing cannot be judged from a single layer or token position.

Reproducibility Statement

Experiments record frozen dataset and model revisions, pair hashes, resolved configurations, seeds, commits, validation-access flags, and immutable predictions. Appendix J reports run IDs and hashes; the method and control appendices specify estimands, paired resampling, identity checks, and protocol controls. Appendix F separates the wide-DD primary analysis, total-logprob sensitivity, and post hoc direct contrasts, including their input-identity and recovery records. Code, configurations, pair manifests, and non-weight artifacts are available at https://github.com/yucheng-du/RouteVerify. Recorded revisions identify separately downloadable model weights.

AI Use Statement

Generative AI tools, including OpenAI Codex, Claude Code, and Cursor, assisted with research framing and experimental design; implementation, testing, data selection and reformatting, analysis, and interpretation; and literature review, figure preparation, translation, and manuscript drafting and editing. They were not used to generate study datasets or proofs, and external audit outputs were treated as issue leads rather than evidence. AI-assisted outputs were checked against source code, tests, machine-readable artifacts, and cited sources; no AI system had authority to accept results or determine the scientific conclusions. The authors take responsibility for the final text, claims, code, and artifacts.

Appendix Roadmap

The appendix provides implementation details, supplementary analyses, controls, and reproducibility records for the experiments in the main paper. It is organized by scientific function rather than by experiment chronology:

  1. Appendix B: verification setup, model identifiers, intervention layers, and architecture-specific boundaries;

  2. Appendix C: fixed-receiver contrasts, token windows, donor construction, controls, and statistical analysis;

  3. Appendix D: supplementary nine-checkpoint results and control comparisons;

  4. Appendix E: unpatched verification performance, decision flip rates, and gap-normalized effects;

  5. Appendix F: full-step all-router fixed-receiver analysis, scoring sensitivity, and direct contrasts;

  6. Appendix G: representation-probe analyses;

  7. Appendix H: natural-error evaluation on ProcessBench;

  8. Appendix I: additional implementation checks and control interpretation;

  9. Appendix J: reproducibility and artifact records;

  10. Appendix K: claim boundaries; and

  11. Appendix L: same-label placebos, donor surface differences, and routing manipulation diagnostics.

Model and Verification Setup

Prompt and verification endpoint

The verification prompt contains the mathematical problem, the solution trace through the evaluated step, and a fixed request to choose between CORRECT and INCORRECT. Dataset labels, pair identities, generator metadata, and later solution steps are not shown to the model. The scorer computes the mean conditional log probability of each complete candidate string using the model’s native tokenizer and chat template. Intervention runs retain the receiver prompt and modify only the designated internal activation.

PRMBench pairs share a problem and an identical prefix before the first benchmark-annotated error step. The nine-checkpoint intervention analyses use 290 ordered pairs grouped into 235 canonical source-problem components. ProcessBench is analyzed separately with 64 cross-problem pairs. Each archived analysis records its sample count, pair identity, bootstrap identity, and source-prediction path.

We use the frozen hitsmy/PRMBench_Preview snapshot recorded in the release manifests. Eligible pairs have matching nonempty problems, equal nonzero step counts, valid error indices with an unambiguous indexing base, identical text before the first annotated error, and a changed step at that boundary. Only complete, unique pairs are retained. An error-type-stratified 50/50 split of 580 complete pair groups (seed 20260730) assigns 290 to discovery and 290 to validation. The nine-checkpoint intervention experiments use only the discovery partition. The 235 source components are bootstrap units, not the units of this split. Labels follow the benchmark annotations without independent mathematical review of every trace and error boundary.

Category-exclusion sensitivity. In a post hoc analysis, we exclude the six circular and three redundancy pairs in the Simplicity category, retaining the original predictions, donor assignments, and 50,000-resample source-component bootstrap protocol (seed 20260908). Full-step and last-token panels retain 281 pairs in 230 components; eight-token contrasts use their common eligible subsets. All 72 single-layer residual-minus-router and residual-minus-MoE-output TT intervals remain positive, and all nine divergence-8-minus-suffix-8 gap intervals remain negative. For OLMoE, the full-step all-router paired-minus-norm-random contrast changes from 0.0466 [0.0003, 0.0938] to 0.0450 [−0.0013, 0.0928]: the point estimate changes little, but the interval includes zero. The full sample remains the primary analysis.

Models and intervention layers

The nine evaluated checkpoints are Qwen3.6-35B-A3B, Qwen3-30B-A3B, GLM-4.7-Flash, Nemotron-3.5 Lightning 30B-A3B-BF16, Nemotron-3 Nano 30B-A3B, Gemma 4 26B-A4B, OLMoE-1B-7B, and GPT-OSS-20B/120B. Model identifiers and source links follow.

google/gemma-4-26B-A4B-it, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, Qwen/Qwen3.6-35B-A3B, Qwen/Qwen3-30B-A3B, zai-org/GLM-4.7-Flash, allenai/OLMoE-1B-7B-0924-Instruct, openai/gpt-oss-20b, and openai/gpt-oss-120b. Exact model and tokenizer revisions are retained in the resolved configurations and artifact index.

Architecture-specific intervention boundaries

Router-logit interventions replace the adapter’s pre-dispatch logits while retaining native expert selection and weighting. MoE-output interventions replace the routed contribution at the architecture-specific output boundary, while post-MoE residual interventions replace the corresponding residual activation. These are distinct intervention locations and are not treated as additive stages of a mediation decomposition.

For Qwen3.6 and GLM, the MoE-output boundary precedes addition of the shared branch, so the shared-expert contribution remains receiver-derived. GLM has 47 decoder blocks, with dense block 0 excluded from the 46 routed sites. In both Nemotron checkpoints, the L27 output boundary is the final routed-only aggregate at fc2_latent_proj; the receiver shared-expert branch remains unchanged, and the 23 routed expert blocks lie within 52 hybrid decoder blocks.

Gemma replaces the normalized routed contribution at post_feedforward_layernorm_2 while preserving its parallel dense branch. Its residual boundary is post-FFN. OLMoE exposes the routed aggregate before residual addition. Qwen3 and the GPT-OSS checkpoints expose the native MoE/MLP-module output. GPT-OSS uses 24 and 36 routed blocks for the 20B and 120B checkpoints, respectively, under native MXFP4 execution and routing rules. The executed adapters and resolved configurations define the exact intervention boundaries.

Intervention Protocol Details

Single-layer fixed-receiver correctness contrast

For receiver pair ii, we select an external matched donor pair d(i)d(i) from a different canonical source component. Matching prioritizes agreement in error type, followed by proximity of the first-error position, with a seeded pair hash used for tie-breaking. The same external pair supplies its correct and error activations to both receivers. Donor assignments remain fixed during analysis; the 290 receiver pairs use 204 distinct external donor pairs.

Let Mir←d,sM_{i}^{r\leftarrow d,s} denote the verification margin after replacing the designated last-token receiver activation with the corresponding activation from donor role ss, for receiver role r∈{C,E}r \in\{C,E\}. The four donor/receiver combinations give

Di,r=Mir←d,E−Mir←d,C,(7)D_{i,r}=M_{i}^{r\leftarrow d,E}-M_{i}^{r\leftarrow d,C}, \tag*{(7)}
Di,common=12(Di,C+Di,E).(8)D_{i,\mathrm{common}}=\frac{1}{2}\left(D_{i,C}+D_{i,E}\right). \tag*{(8)}

Because the receiver is identical in the two terms defining Di,rD_{i,r}, its baseline verification margin cancels. Receiver roles are averaged within pair before computing DcommonD_{\mathrm{common}}, so the sample size remains 290 rather than 580. Direct interface contrasts are likewise computed within pair before resampling.

Positive DD means that, for the same receiver, replacing the activation with one from an erroneous step produces a more INCORRECT-favoring margin than replacing it with one from a correct step. It therefore rules out a completely donor-role-independent perturbation as a sufficient explanation for that comparison. These labels remain correlated with text, content, and activation geometry, however. Neither D/TD/T nor T−DT-D has a decomposition interpretation. This local experiment uses one layer and the final step token; the full-step all-router fixed-receiver analysis is reported in Appendix § 17.

Token windows and alignment

The window experiment crosses four intervention conditions—single-layer router logits, single-layer MoE output, single-layer residual, and all-layer router logits—with four token regions: the final token of the evaluated step, the final kk tokens, kk tokens centered on the first token-level divergence between the paired rendered prefixes, and the full evaluated step.

For each tokenizer, we choose the largest k∈{4,8,16,32}k \in\{4, 8, 16, 32\} for which both paired steps contain at least kk tokens in at least 90% of discovery pairs, using token lengths only. All nine checkpoints select k=8k=8. Divergence is defined as the first difference in the complete rendered causal prefixes. The centered window is clipped to remain within the evaluated step without shortening it. If the first divergence lies outside that step, the example is ineligible for this condition. For even kk, the divergence token occupies one of the two central positions before clipping. Suffix and divergence windows may overlap or coincide.

Full-step replacement covers every receiver-step token even when the donor step has a different length. If receiver and donor windows have lengths RR and LL, receiver offset jj maps to donor offset

⌊j(L−1)/(R−1)+1/2⌋;\left\lfloor j(L-1)/(R-1)+1/2\right\rfloor;

when R=1R=1, the final donor token is used. The mapping preserves endpoint ordering and may reuse donor positions. It is an operational alignment rather than a claim of semantic token equivalence.

Control conditions

Each interface/window combination includes paired-donor, self-patch, matched-external-donor, and norm-random controls. Self and matched conditions inherit the paired receiver positions. A matched donor supplies its own corresponding window; unavailable donor windows are recorded as missing. Norm-random interventions use a Gaussian direction whose ℓ2\ell_2 displacement matches the paired donor–receiver displacement separately at each intervention position and layer. Equal token count therefore does not imply equal activation displacement or equal cumulative layer coverage.

A pair contributes to TT only when both replacement directions are available. Direct control and cross-window contrasts use the intersection of their eligible pair sets, and unavailable windows are never treated as zero. Table 3 reports the resulting sample sizes. Last-token and full-step panels use all N=290N = 290 pairs and K=235K = 235 source components.

ModelSuffix-8 PairedSuffix-8 MatchedDivergence-8 PairedDivergence-8 MatchedCommon Paired
Qwen3.6278/226265/217276/224261/213276/224
Qwen3278/226265/217276/224261/213276/224
GLM-4.7-Flash274/222256/209272/220252/205272/220
Lightning278/226265/217276/224261/213276/224
Nano278/226265/217276/224261/213276/224
Gemma 4279/226268/220277/224264/216277/224
OLMoE271/220250/205271/220250/205271/220
GPT-OSS 20B272/220253/206271/219251/204271/219
GPT-OSS 120B272/220253/206271/219251/204271/219

Table 3. Eligible pairs/source components (N/KN/K) for the eight-token windows. Paired-donor, self-patch, and norm-random conditions share the paired mask. “Common” is the suffix/divergence intersection used for the equal-length position contrast. Matched-donor comparisons use their own complete-pair intersection.

All-layer MoE-output intervention

The all-layer MoE-output experiment replaces the last-token routed output at every routed site listed in Table 2. It includes paired donors, matched external donors, and norm-random controls. The norm-random condition matches the paired donor–receiver displacement independently at each routed site. Paired-minus-matched and paired-minus-random contrasts are computed within pair before aggregation. This intervention differs in both interface and coverage from a single-layer MoE-output intervention and a full-step all-router intervention.

Statistical analysis

The nine-checkpoint window analysis reconstructs 161,480 pair-level TT values from the saved scores and includes all four token windows, intervention interfaces, controls, direct interface comparisons, cross-window contrasts, and gap-change contrasts. Uncertainty is estimated by 50,000 PCG64 source-component bootstrap resamples. Each draw samples eligible source components with replacement, retains all pairs belonging to each sampled component, and computes a pair-weighted mean. Analyses sharing the same eligibility mask use the same resamples.

Reported intervals are two-sided pointwise 95% percentile intervals. Inference conditions on the fixed checkpoints, discovery pool, and donor assignments; reused donors are not independently resampled. No multiplicity correction or population-of-models test is applied.

All final single-layer DD, all-layer MoE-output, window, and direct-contrast panels use 50,000 source-component resamples. Exact seeds and analysis identifiers are stored with the archived analysis artifacts. The reported window figures use one common analysis throughout, and direct differences are formed within pair before resampling.

Supplementary Nine-Checkpoint Results

Receiver dependence. Qwen3.6 is the only checkpoint for which the local residual intervention does not show a common positive direction across receiver types. DC=−0.00964D_{\mathrm{C}} = -0.00964

Figure 4. Nine-checkpoint intervention profiles across four token windows and four intervention conditions. Panels use checkpoint-specific linear axes. Points are pair-weighted TT means with two-sided pointwise 95% source-component percentile intervals from 50,000 PCG64 resamples. Last-token and full-step conditions use N=290N = 290, K=235K = 235; eight-token sample sizes are given in Table 3. Local intervention layers are fixed within each checkpoint, whereas the all-router condition replaces router logits at every routed layer.

ModelDcorrectD_{\mathrm{correct}}DerrorD_{\mathrm{error}}DcommonD_{\mathrm{common}}
Router
Qwen3.6.0015 [−.0056, .0093].0015\ [-.0056,\ .0093]−.0035 [−.0168, .0091]-.0035\ [-.0168,\ .0091]−.0010 [−.0085, .0062]-.0010\ [-.0085,\ .0062]
Qwen3−.0054 [−.0157, .0049]-.0054\ [-.0157,\ .0049]−.0091 [−.0197, .0015]-.0091\ [-.0197,\ .0015]−.0073 [−.0151, .0006]-.0073\ [-.0151,\ .0006]
GLM-4.7-Flash−.0005 [−.0109, .0100]-.0005\ [-.0109,\ .0100].0022 [−.0084, .0130].0022\ [-.0084,\ .0130].0009 [−.0062, .0080].0009\ [-.0062,\ .0080]
Lightning.0017 [−.0026, .0061].0017\ [-.0026,\ .0061]−.0018 [−.0072, .0037]-.0018\ [-.0072,\ .0037]−.0001 [−.0035, .0034]-.0001\ [-.0035,\ .0034]
Nano.0018 [−.0012, .0047].0018\ [-.0012,\ .0047]−.0008 [−.0038, .0022]-.0008\ [-.0038,\ .0022].0005 [−.0017, .0026].0005\ [-.0017,\ .0026]
Gemma4−.0106 [−.0328, .0111]-.0106\ [-.0328,\ .0111]−.0016 [−.0202, .0180]-.0016\ [-.0202,\ .0180]−.0061 [−.0208, .0087]-.0061\ [-.0208,\ .0087]
OLMoE.0061 [−.0011, .0132].0061\ [-.0011,\ .0132].0039 [−.0049, .0127].0039\ [-.0049,\ .0127].0050 [−.0011, .0110].0050\ [-.0011,\ .0110]
GPT-OSS-20B−.0004 [−.0049, .0042]-.0004\ [-.0049,\ .0042]−.0029 [−.0103, .0043]-.0029\ [-.0103,\ .0043]−.0017 [−.0060, .0027]-.0017\ [-.0060,\ .0027]
GPT-OSS-120B.0031 [−.0093, .0156].0031\ [-.0093,\ .0156].0092 [−.0044, .0228].0092\ [-.0044,\ .0228].0061 [−.0025, .0150].0061\ [-.0025,\ .0150]
MoE output
Qwen3.6.0009 [−.0063, .0080].0009\ [-.0063,\ .0080]−.0115 [−.0240, .0003]-.0115\ [-.0240,\ .0003]−.0053 [−.0120, .0010]-.0053\ [-.0120,\ .0010]
Qwen3.0498 [.0321, .0676].0498\ [.0321,\ .0676].0540 [.0305, .0771].0540\ [.0305,\ .0771].0519 [.0339, .0698].0519\ [.0339,\ .0698]
GLM-4.7-Flash.0802 [.0598, .1006].0802\ [.0598,\ .1006].0482 [.0316, .0647].0482\ [.0316,\ .0647].0642 [.0475, .0806].0642\ [.0475,\ .0806]
Lightning.0006 [−.0039, .0052].0006\ [-.0039,\ .0052]−.0012 [−.0064, .0040]-.0012\ [-.0064,\ .0040]−.0003 [−.0041, .0036]-.0003\ [-.0041,\ .0036]
Nano.0018 [−.0015, .0051].0018\ [-.0015,\ .0051]−.0010 [−.0048, .0029]-.0010\ [-.0048,\ .0029].0004 [−.0024, .0032].0004\ [-.0024,\ .0032]
Gemma4.1430 [.0727, .2295].1430\ [.0727,\ .2295].0946 [.0459, .1458].0946\ [.0459,\ .1458].1188 [.0681, .1793].1188\ [.0681,\ .1793]
OLMoE.0402 [.0250, .0556].0402\ [.0250,\ .0556].0296 [.0123, .0472].0296\ [.0123,\ .0472].0349 [.0211, .0487].0349\ [.0211,\ .0487]
GPT-OSS-20B.0031 [−.0094, .0154].0031\ [-.0094,\ .0154]−.0059 [−.0167, .0048]-.0059\ [-.0167,\ .0048]−.0014 [−.0108, .0079]-.0014\ [-.0108,\ .0079]
GPT-OSS-120B.0433 [.0290, .0581].0433\ [.0290,\ .0581].0439 [.0268, .0617].0439\ [.0268,\ .0617].0436 [.0312, .0564].0436\ [.0312,\ .0564]
Residual
Qwen3.6−.0096 [−.0308, .0130]-.0096\ [-.0308,\ .0130].0356 [.0101, .0625].0356\ [.0101,\ .0625].0130 [−.0054, .0327].0130\ [-.0054,\ .0327]
Qwen3.7282 [.6313, .8269].7282\ [.6313,\ .8269].5921 [.4989, .6879].5921\ [.4989,\ .6879].6601 [.5712, .7509].6601\ [.5712,\ .7509]
GLM-4.7-Flash.1989 [.1594, .2404].1989\ [.1594,\ .2404].1371 [.1077, .1673].1371\ [.1077,\ .1673].1680 [.1353, .2019].1680\ [.1353,\ .2019]
Lightning.1492 [.1285, .1704].1492\ [.1285,\ .1704].1125 [.0953, .1304].1125\ [.0953,\ .1304].1309 [.1127, .1495].1309\ [.1127,\ .1495]
Nano.0793 [.0680, .0907].0793\ [.0680,\ .0907].0624 [.0525, .0726].0624\ [.0525,\ .0726].0709 [.0608, .0811].0709\ [.0608,\ .0811]
Gemma41.1705 [.8775, 1.4791]1.1705\ [.8775,\ 1.4791].2586 [.1150, .4163].2586\ [.1150,\ .4163].7145 [.5321, .9069].7145\ [.5321,\ .9069]
OLMoE.1368 [.0859, .1925].1368\ [.0859,\ .1925].1586 [.1049, .2169].1586\ [.1049,\ .2169].1477 [.0966, .2038].1477\ [.0966,\ .2038]
GPT-OSS-20B.1352 [.1016, .1696].1352\ [.1016,\ .1696].1019 [.0726, .1323].1019\ [.0726,\ .1323].1186 [.0909, .1473].1186\ [.0909,\ .1473]
GPT-OSS-120B.2262 [.1895, .2645].2262\ [.1895,\ .2645].1654 [.1342, .1977].1654\ [.1342,\ .1977].1958 [.1659, .2271].1958\ [.1659,\ .2271]

Table 4. Last-token fixed-receiver DD at all three local interfaces. Each checkpoint uses N=290N = 290 receiver pairs and K=235K = 235 source components. DcommonD_{\mathrm{common}} averages the two receiver types within pair. Intervals are two-sided pointwise 95% source-component percentile intervals from 50,000 resamples. Qwen3 uses L36 for this experiment and L24 for the token-window experiment. A substantial part of OLMoE’s residual DD is associated with surface content (Appendix § 23).

ModelPaired TTPaired−matchedPaired−norm-random
Qwen3.6.347 [.286, .408]−.168 [−.227, −.111].218 [.170, .267]
Qwen31.576 [1.358, 1.796]−.497 [−.663, −.335].651 [.505, .804]
GLM-4.7-Flash.231 [.189, .275]−.071 [−.102, −.040].165 [.131, .200]
Lightning.221 [.186, .256]−.073 [−.096, −.050].175 [.144, .208]
Nano.143 [.118, .170]−.040 [−.056, −.024].120 [.098, .143]
Gemma41.208 [.958, 1.459]−.926 [−1.199, −.655].915 [.689, 1.141]
OLMoE†.1852 [.1200, .2527]−.0168 [−.0648, .0304].0466 [.0003, .0938]
GPT-OSS-20B.297 [.242, .353]−.161 [−.214, −.109].189 [.146, .233]
GPT-OSS-120B.428 [.350, .509]−.169 [−.229, −.111].214 [.155, .275]

Table 5. Controls for full-step all-router interventions and directly paired contrasts. All panels use N=290N = 290, K=235K = 235 and 50,000 source-component resamples. Norm-random interventions match the paired donor displacement separately at each token position and layer. OLMoE’s (†) paired-minus-norm-random interval lies close to zero; a larger matched-donor TT does not imply more correctness-related information.

[−0.03077, 0.01299], whereas DE=0.03560D_{\mathrm{E}} = 0.03560 [0.01006, 0.06248]. Their direct difference is −0.04524 [−0.07478, −0.01557], establishing receiver dependence. Its DcommonD_{\mathrm{common}} has median −0.00077 and 10%-trimmed mean 0.00217, showing that the positive mean does not correspond to a uniform pairwise direction.

The remaining eight residual panels have positive intervals for both receiver types, although their magnitudes need not match. For example, the correct-minus-error receiver differences for Qwen3 and GLM are 0.1362 [0.0690, 0.2045] and 0.0618 [0.0376, 0.0874], respectively.

Coverage and position. Expanding token coverage from the last token to the full step does not produce a common trend in the residual-minus-all-router gap. The gap increases in Qwen3.6, Qwen3, Gemma 4, and GPT-OSS 20B; decreases in GLM, Lightning, Nano, and GPT-OSS 120B; and is unresolved in OLMoE.

Position is more consistent. With token count fixed at eight, moving from suffix-8 to divergence-8 reduces the residual-minus-all-router gap in all nine checkpoints. This pattern does not imply a larger absolute routing response near divergence. In GPT-OSS 120B, the all-router divergence-minus-suffix difference is −0.0028 [−0.0591, 0.0535], whereas the residual difference is −0.1946 [−0.2315, −0.1589]. The changed ordering there is driven primarily by residual sensitivity to token position.

Qwen3.6’s full-step residual effect also differs from its heavy-tailed last-token result: its mean is 1.2821, median 1.2086, 10%-trimmed mean 1.2246, and 88.3% of pair effects are positive. These statistics summarize the same distribution of pair-level effects; the pair-weighted mean remains the primary summary.

Control comparisons. Full-step all-router paired-minus-matched intervals are negative in eight checkpoints and include zero in OLMoE. By contrast, paired-minus-norm-random intervals are positive in all nine checkpoints, although OLMoE lies close to zero: 0.0466 [0.0003, 0.0938], with norm-random T=0.1387T = 0.1387 and paired-donor T=0.1852T = 0.1852.

For the all-layer MoE-output intervention, every paired-minus-norm-random interval is positive, whereas every paired-minus-matched interval includes zero. The latter result does not establish equivalence between paired and matched external donors.

Two GPT-OSS checkpoints. Matched-pair 120B-minus-20B contrasts are 0.1316 [0.0704, 0.1935] for full-step all-router TT and −0.8498 [−0.9704, −0.7294] for full-step residual TT. Single-layer DcommonD_{\mathrm{common}} differences are 0.0450 [0.0290, 0.0618] at the MoE-output interface and 0.0773 [0.0414, 0.1138] at the residual interface. Effects therefore do not increase uniformly across interfaces in the larger checkpoint.

ModelPaired-donor TTPaired−matchedPaired−norm-random
Qwen3.6.099 [.064, .138].019 [−.007, .048].088 [.051, .130]
Qwen3.933 [.785, 1.083].084 [−.022, .190].848 [.713, .984]
GLM-4.7-Flash.162 [.127, .198].014 [−.012, .040].149 [.116, .183]
Lightning.214 [.181, .249].019 [−.010, .048].196 [.164, .229]
Nano.101 [.083, .120].008 [−.006, .022].088 [.071, .106]
Gemma41.133 [.869, 1.422].136 [−.107, .388].938 [.684, 1.217]
OLMoE.113 [.067, .162].007 [−.035, .050].093 [.055, .135]
GPT-OSS-20B.388 [.331, .446].030 [−.015, .077].364 [.280, .447]
GPT-OSS-120B.563 [.478, .650].022 [−.045, .091].528 [.446, .612]

Table 6. All-layer MoE-output interventions at the last token. Paired-donor TT is positive and exceeds the norm-matched random control in all nine models. “Paired” uses the corresponding correct/error trace as the donor, whereas “matched” uses an external donor from another problem matched by error type and boundary position. All paired-minus-matched intervals include zero. Entries are pair means with pointwise 95% source-component percentile intervals (N=290N = 290, K=235K = 235, 50,000 resamples).

The two checkpoints also differ in local anchor, routed depth, expert count, training, and checkpoint identity. These comparisons therefore do not isolate parameter count or define a scaling law.

Verification Performance, Decision Flips, and Normalized Effects

All three quantities reported here are computed from the saved candidate scores of the nine-checkpoint window runs and, where stated, of the full-step all-router fixed-receiver run; no additional model forwards are required. A receiver receives the verdict INCORRECT when M>0M > 0 and CORRECT when M≤0M \le0, and the truth is CORRECT for the correct receiver and INCORRECT for the error receiver. These are within-study values of a fixed candidate-scoring verdict on the paired PRMBench assay used in this study, not official PRMBench benchmark scores. The mean-logprob margin of (1) is primary, and the total-logprob margin gives the same picture, with accuracy within 0.021 and AUROC within 0.002 of the primary values. Exact ties at zero margin occur only on the total metric, in 5 baseline receivers of Qwen3, and are read as CORRECT. Intervals follow the rest of the paper: two-sided pointwise 95% source-component percentile intervals from 50,000 resamples without multiplicity correction, with the AUROC interval computed from resample-weighted tie-aware rank comparisons.

Unpatched verification performance

Accuracy is the fraction of the 580 receivers whose verdict matches the truth. Error recall is the fraction of error receivers called INCORRECT, and the correct-receiver false-positive rate is the fraction of correct receivers called INCORRECT. AUROC ranks receivers by MM with the error receiver as the positive class. The mean gap g=M(xE)−M(xC)g = M(x_E) - M(x_C) averages the within-pair margin difference. Labels are balanced, so chance is 0.5 for accuracy and for AUROC, and balanced accuracy equals accuracy.

All nine checkpoints are above chance (Table 7). Accuracy ranges from 0.5672 in OLMoE to 0.7862 in Gemma and AUROC from 0.6018 to 0.8744. Two checkpoints are strongly biased in opposite directions: Nano calls 0.621 of correct receivers INCORRECT, whereas OLMoE never does and recalls 0.134 of errors. Within the window runs, unpatched reruns reproduce these scores exactly, with zero verdict flips in every checkpoint.

ModelAccuracyError recallCorrect-receiver FPRAUROCMean gap gg
Qwen3.60.7776 [0.7457, 0.8090]0.6517 [0.5951, 0.7075]0.0966 [0.0605, 0.1359]0.8580 [0.8279, 0.8866]1.5549 [1.3789, 1.7296]
Qwen30.7345 [0.7038, 0.7650]0.5586 [0.5000, 0.6151]0.0897 [0.0559, 0.1263]0.8610 [0.8316, 0.8891]4.6818 [4.1758, 5.1846]
GLM-4.7-Flash0.5845 [0.5621, 0.6075]0.1862 [0.1424, 0.2319]0.0172 [0.0035, 0.0339]0.8058 [0.7748, 0.8360]0.8147 [0.7125, 0.9175]
Lightning0.7621 [0.7298, 0.7937]0.7517 [0.7013, 0.8007]0.2276 [0.1771, 0.2800]0.8475 [0.8185, 0.8755]0.8680 [0.7716, 0.9655]
Nano0.6534 [0.6241, 0.6838]0.9276 [0.8953, 0.9570]0.6207 [0.5586, 0.6812]0.8171 [0.7857, 0.8478]0.5024 [0.4378, 0.5689]
Gemma40.7862 [0.7552, 0.8165]0.8103 [0.7651, 0.8537]0.2379 [0.1852, 0.2941]0.8744 [0.8470, 0.9003]7.0863 [6.3870, 7.7836]
OLMoE0.5672 [0.5481, 0.5875]0.1345 [0.0962, 0.1751]0.0000 [0.0000, 0.0000]0.6018 [0.5701, 0.6334]0.6056 [0.4427, 0.7738]
GPT-OSS-20B0.7155 [0.6858, 0.7451]0.8034 [0.7562, 0.8486]0.3724 [0.3132, 0.4339]0.8202 [0.7901, 0.8497]1.2942 [1.1356, 1.4525]
GPT-OSS-120B0.7103 [0.6781, 0.7425]0.8138 [0.7662, 0.8596]0.3931 [0.3345, 0.4539]0.7785 [0.7460, 0.8104]1.3498 [1.1818, 1.5193]

Table 7. Unpatched verification performance on the paired PRMBench assay used in this study, under the mean-logprob margin and the M>0M > 0 verdict rule. Accuracy, error recall, and the correct-receiver false-positive rate use all 580 receivers of the 290 pairs; AUROC takes the error receiver as the positive class; gg is the within-pair baseline margin gap. These are within-study assay values, not official PRMBench benchmark scores. Intervals are two-sided pointwise 95% source-component percentile intervals from 50,000 resamples (N=290N = 290, K=235K = 235).

Decision flip rates

For each receiver we compare the unpatched verdict with the patched verdict. The four transition classes are correct-to-correct, correct-to-wrong, wrong-to-correct, and wrong-to-wrong, where correct and wrong refer to agreement with the truth. The flip rate is the fraction of receivers whose verdict changes. The harmful rate conditions on baseline-correct receivers and the beneficial rate on baseline-wrong receivers; the two have different denominators and are never subtracted. The change in accuracy equals the unconditional beneficial rate minus the unconditional harmful rate. By construction, the paired design supplies the opposite-role donor, so a donor-aligned flip is a flip away from the truth, and accuracy decreases in every paired residual panel and in every full-step all-router panel. Where flips are few the sign is not forced: in 41 of the remaining 198 paired panels, all of them router-logit, MoE-output, or non-full-step all-router panels, accuracy instead increases by at most 0.0091, a net gain of five receivers or fewer. Positive mean TT indicates an average margin shift toward the opposite-role donors’ labels, but does not determine the accuracy change.

Under the paired donor at full step, the all-router intervention changes 0.0621 (OLMoE) to 0.2293 (GPT-OSS-120B) of verdicts, the norm-random control 0.0310 to 0.1121, and the matched external donor 0.0690 to 0.3517, exceeding the paired donor in all nine checkpoints (Table 8). The paired residual intervention changes 0.0672 to 0.5362 of verdicts and separates into two regimes: Qwen3.6, Qwen3, Gemma, and GPT-OSS-20B fall to patched accuracy 0.2724 to 0.3741, well below chance, whereas GLM-4.7-Flash and OLMoE end at about chance (0.4966 and 0.5000) and Lightning, Nano, and GPT-OSS-120B at 0.5517 to 0.6931; these five lose 0.0672 to 0.1017 of accuracy. Single-layer router-logit interventions flip 0.0052 to 0.0259 of verdicts at full step. Conditional beneficial rates are undefined for OLMoE on correct receivers, whose false-positive rate is exactly zero, and rest on one to five receivers for GLM-4.7-Flash; the unconditional rates and accuracy changes are defined throughout.

The same reading applies to the full-step all-router fixed-receiver run of Appendix F. Holding the receiver fixed and switching the donor from correct to error changes 0.0172 (Nano) to 0.1517 (Qwen3.6) of verdicts, far fewer than patching itself. On correct receivers, GPT-OSS-20B moves 0.1000 of verdicts to INCORRECT and 0.0414 to CORRECT, and Gemma 0.0897 and 0.0586, the categorical counterpart of their positive correct-receiver DD. By contrast, the accuracy change on the receivers has the same sign and a similar size under the correct and the error donor in every checkpoint, for example 0.0552 versus 0.0690 in Lightning and −0.1724 versus −0.1517 in Qwen3.6. Baseline-to-patched flips in that run are therefore dominated by patching itself rather than by donor role.

Table 8: Full-step verdict flips under the mean-logprob margin, on all 580 receivers of the 290 pairs. The first three columns give the all-router flip rate under the paired donor, a matched external donor, and the norm-random control; the last two give the paired residual flip rate and its accuracy change. Self-patch controls produce zero flips in every checkpoint. Intervals are two-sided pointwise 95% source-component percentile intervals from 50,000 resamples (N=290N = 290, K=235K = 235).

ModelAll-router pairedAll-router matchedAll-router norm-randomResidual pairedResidual Δ\Deltaaccuracy
Qwen3.60.1690 [0.1399, 0.1993]0.2690 [0.2352, 0.3032]0.0879 [0.0651, 0.1123]0.5362 [0.4826, 0.5890]−0.5052 [−0.5629, −0.4467]
Qwen30.2155 [0.1831, 0.2491]0.3517 [0.3134, 0.3903]0.1121 [0.0850, 0.1406]0.4397 [0.3867, 0.4929]−0.4155 [−0.4711, −0.3596]
GLM-4.7-Flash0.0966 [0.0722, 0.1224]0.1155 [0.0896, 0.1421]0.0586 [0.0402, 0.0782]0.0983 [0.0732, 0.1242]−0.0879 [−0.1150, −0.0618]
Lightning0.2034 [0.1701, 0.2377]0.3000 [0.2607, 0.3402]0.0586 [0.0412, 0.0769]0.1103 [0.0847, 0.1381]−0.0690 [−0.0956, −0.0423]
Nano0.1414 [0.1107, 0.1738]0.1897 [0.1544, 0.2266]0.0466 [0.0295, 0.0654]0.1121 [0.0857, 0.1398]−0.1017 [−0.1301, −0.0742]
Gemma40.1879 [0.1567, 0.2202]0.3224 [0.2819, 0.3635]0.0552 [0.0372, 0.0748]0.4362 [0.3866, 0.4859]−0.4121 [−0.4653, −0.3582]
OLMoE0.0621 [0.0437, 0.0816]0.0690 [0.0498, 0.0893]0.0310 [0.0179, 0.0451]0.0672 [0.0448, 0.0914]−0.0672 [−0.0914, −0.0448]
GPT-OSS-20B0.1621 [0.1313, 0.1941]0.3172 [0.2752, 0.3595]0.0466 [0.0308, 0.0636]0.4224 [0.3681, 0.4775]−0.4155 [−0.4710, −0.3603]
GPT-OSS-120B0.2293 [0.1926, 0.2668]0.3086 [0.2683, 0.3493]0.1052 [0.0804, 0.1311]0.0983 [0.0741, 0.1237]−0.0845 [−0.1096, −0.0603]

Table 8.

Effects normalized by the baseline gap

We normalize each panel by the baseline margin gap of the same eligible pairs under the same margin, R=mean⁡(T)/mean⁡(g)R = \operatorname{mean}(T)/\operatorname{mean}(g) with g=M(xE)−M(xC)g = M(x_E) - M(x_C). By Equation (5), 2T=g−gafter2T = g - g_{\text{after}} with gafter=M(xE←C)−M(xC←E)g_{\text{after}} = M(x_{E\leftarrow C}) - M(x_{C\leftarrow E}), so R=0.5R = 0.5 means that the mean post-transfer gap is zero and R=1R = 1 an equal-magnitude reversal. RR is a ratio of means. It can be negative or exceed one, it is not a mediation share or an explained variance, and it differs from dividing each receiver’s effect by its own distance to the decision boundary, which is not computed here.

Full-step residual RR runs from 0.2371 in GPT-OSS-120B to 0.9040 in GPT-OSS-20B (Table 9). Its interval lies entirely above 0.5 in GPT-OSS-20B, Qwen3, Qwen3.6, and Gemma4, whose mean post-transfer gaps are −1.0455, −3.7285, −1.0094, and −2.6533. GLM-4.7-Flash (0.5177 [0.4857, 0.5496]) and OLMoE (0.5356 [0.4726, 0.6084]) sit at about 0.5, and Lightning, Nano, and GPT-OSS-120B report RR around 0.25. Full-step all-router RR runs from 0.3355 to 0.3365, reducing the mean gap to between 33% and 66% of baseline, and single-layer router-logit RR stays within [−0.0094, 0.0167] in all windows.

ModelMean gap ggRR, all-router (full step)RR, residual (full step)RR, residual (last token)Pairs with g≤0g \le0
Qwen3.61.5549 [1.3789, 1.7296]0.2229 [0.1933, 0.2523]0.8246 [0.7991, 0.8505]0.0291 [0.0088, 0.0514]0.1379
Qwen34.6818 [4.1758, 5.1846]0.3365 [0.3057, 0.3666]0.8982 [0.8786, 0.9170]0.0892 [0.0737, 0.1050]0.0897
GLM-4.7-Flash0.8147 [0.7125, 0.9175]0.2839 [0.2481, 0.3179]0.5177 [0.4857, 0.5496]0.2856 [0.2566, 0.3149]0.1586
Lightning0.8680 [0.7716, 0.9655]0.2541 [0.2257, 0.2821]0.2503 [0.2351, 0.2657]0.1726 [0.1602, 0.1854]0.1345
Nano0.5024 [0.4378, 0.5689]0.2856 [0.2534, 0.3168]0.2586 [0.2412, 0.2764]0.1726 [0.1612, 0.1845]0.1655
Gemma47.0863 [6.3870, 7.7836]0.1705 [0.1388, 0.2025]0.6872 [0.6520, 0.7226]0.1171 [0.0891, 0.1478]0.0931
OLMoE0.6056 [0.4427, 0.7738]0.3059 [0.2277, 0.3846]0.5356 [0.4726, 0.6084]0.1577 [0.1005, 0.2200]0.3966
GPT-OSS-20B1.2942 [1.1356, 1.4525]0.2293 [0.1932, 0.2663]0.9040 [0.8819, 0.9268]0.1134 [0.0927, 0.1357]0.1310
GPT-OSS-120B1.3498 [1.1818, 1.5193]0.3173 [0.2747, 0.3589]0.2371 [0.2156, 0.2602]0.1550 [0.1386, 0.1724]0.1931

Table 9. Paired-donor effects normalized by the baseline margin gap of the same pairs, R=mean⁡(T)/mean⁡(g)R = \operatorname{mean}(T)/\operatorname{mean}(g), under the mean-logprob margin. The last column gives the fraction of pairs with a non-positive baseline gap. Intervals are two-sided pointwise 95% source-component percentile intervals from 50,000 resamples (N=290N = 290, K=235K = 235).

Within a checkpoint and margin definition, normalization preserves interface orderings when the panels share an eligible-pair set and a positive mean baseline gap. Across windows or controls with different eligible-pair sets, the denominator is recomputed and orderings need not be preserved. On the residual interface the matched external donor gives essentially the same RR as the paired donor, for example 0.2512 versus 0.2503 in Lightning, 0.2589 versus 0.2586 in Nano, and 0.2482 versus 0.2371 in GPT-OSS-120B, so normalization does not change the donor-generic character of that effect. All 1,152 panels have denominators whose intervals exclude zero. OLMoE has non-positive baseline gaps in 0.3966 of pairs, against at most 0.1931 elsewhere, so its RR should not be read as a per-pair recovery fraction.

Unpatched scores of the full-step all-router run

The unpatched candidate scores of the run in Appendix § 17 reproduce the window-run scores receiver by receiver for Qwen3.6, Qwen3, OLMoE, GPT-OSS-20B, and GPT-OSS-120B, but not for GLM-4.7-Flash, Lightning, Nano, and Gemma. Recorded inputs, tokenization, pair identities, and receiver roles agree across the two runs for all nine checkpoints, but the source of the score discrepancies remains unresolved. Unpatched accuracy differs by at most 0.006. Table 7 reports the window-run scores; each run retains its own baseline. The fixed-receiver DD estimates of Appendix § 17 compare patched scores within the wide-DD run and do not use the window-run baseline.

Full-Step All-Router Fixed-Receiver Analysis

Experimental design and primary analysis

This experiment extends the fixed-receiver contrast in Equation (6) to every adapter-defined routed layer listed in Table 2 and every token in the evaluated step, with α=1\alpha= 1. All nine checkpoints use the same 290 receiver pairs, 235 source components, and a fixed assignment to 204 distinct external donor pairs. Matching excludes the receiver pair and its source component, then prioritizes agreement in error type and proximity of the first-error position.

For each receiver, the correct and error members of the matched external donor pair separately replace router logits across all routed layers and the full step. Donor and receiver spans are aligned using the monotone mapping defined in Appendix § 14. Each checkpoint retains its native BF16 or MXFP4 execution and native expert dispatch.

The primary endpoint remains the candidate-wise mean-logprob verification margin in Equation (1). For each receiver pair, the four donor/receiver combinations yield DCD_{\mathrm{C}}, DED_{\mathrm{E}}, and their within-pair average DcommonD_{\mathrm{common}}. All four intervention scores and identity controls are generated under the same loaded model invocation. All 290 pairs are complete for every checkpoint; no values are imputed.

A positive DcommonD_{\mathrm{common}} indicates an average correctness-dependent effect across the two receiver types. It does not require both receiver-specific effects, or every individual pair effect, to be positive.

The primary uncertainty analysis uses 50,000 draws from numpy.random.Generator(PCG64) with seed 20260908. Each draw samples 235 receiver source components with replacement, retains all pairs belonging to each sampled component, and divides the summed pair effects by the sampled pair count. Both receiver roles share each draw, so DcommonD_{\mathrm{common}} has N=290N = 290, not 580. Donors are neither rematched nor resampled independently.

All intervals are two-sided pointwise 95% percentile intervals conditional on the discovery panel and fixed donor assignment, without simultaneous coverage or multiplicity adjustment. Table 10 reports the primary results; Table 11 reports the scoring sensitivity. Counts of intervals excluding zero should not be interpreted as estimates of a model-population success rate.

ModelDcorrectD_{\mathrm{correct}}DerrorD_{\mathrm{error}}DcommonD_{\mathrm{common}}
Qwen3.60.0198 [−0.0199, 0.0595]0.0012 [−0.0616, 0.0644]0.0105 [−0.0282, 0.0486]
Qwen3−0.0954 [−0.2095, 0.0192]−0.0738 [−0.2102, 0.0589]−0.0846 [−0.1872, 0.0166]
GLM-4.7-Flash0.0041 [−0.0273, 0.0369]−0.0292 [−0.0572, −0.0011]−0.0125 [−0.0363, 0.0110]
Lightning0.0213 [−0.0005, 0.0430]0.0043 [−0.0197, 0.0280]0.0128 [−0.0050, 0.0304]
Nano0.0032 [−0.0109, 0.0171]−0.0094 [−0.0219, 0.0028]−0.0031 [−0.0134, 0.0069]
Gemma40.3875 [0.0762, 0.7002]0.0540 [−0.2159, 0.3335]0.2207 [−0.0114, 0.4545]
OLMoE−0.0088 [−0.0609, 0.0435]0.0415 [−0.0435, 0.1270]0.0164 [−0.0376, 0.0713]
GPT-OSS-20B0.1137 [0.0480, 0.1803]0.0262 [−0.0335, 0.0871]0.0699 [0.0181, 0.1226]
GPT-OSS-120B−0.0032 [−0.0489, 0.0435]−0.0071 [−0.0539, 0.0396]−0.0051 [−0.0393, 0.0290]

Table 10. Primary full-step all-router fixed-receiver effects. Each checkpoint uses N=290N = 290 receiver pairs, K=235K = 235 source components, and the same fixed assignment to 204 external donor pairs. Entries are mean-logprob estimates with two-sided pointwise 95% source-component percentile intervals from 50,000 PCG64 resamples at α=1\alpha= 1. DcommonD_{\mathrm{common}} averages the two receiver types within pair. Checkpoint-specific margin scales should not be interpreted as ability rankings.

ModelDcorrectD_{\mathrm{correct}}DerrorD_{\mathrm{error}}DcommonD_{\mathrm{common}}
Qwen3.60.0477 [−0.0541, 0.1509]−0.0055 [−0.1502, 0.1411]0.0211 [−0.0710, 0.1118]
Qwen3−0.2638 [−0.5815, 0.0554]−0.1974 [−0.5702, 0.1644]−0.2306 [−0.5154, 0.0491]
GLM-4.7-Flash0.0147 [−0.0753, 0.1090]−0.0774 [−0.1552, 0.0009]−0.0313 [−0.0986, 0.0356]
Lightning0.0717 [−0.0052, 0.1485]0.0040 [−0.0775, 0.0838]0.0379 [−0.0239, 0.0990]
Nano0.0090 [−0.0375, 0.0549]−0.0307 [−0.0702, 0.0082]−0.0108 [−0.0442, 0.0217]
Gemma40.9582 [0.2172, 1.7054]0.1395 [−0.4944, 0.7933]0.5488 [0.0034, 1.1020]
OLMoE−0.0254 [−0.1765, 0.1265]0.1043 [−0.1255, 0.3361]0.0394 [−0.1092, 0.1911]
GPT-OSS-20B0.2588 [0.1017, 0.4193]0.0719 [−0.0600, 0.2074]0.1653 [0.0463, 0.2868]
GPT-OSS-120B0.0136 [−0.0968, 0.1269]−0.0245 [−0.1340, 0.0832]−0.0055 [−0.0859, 0.0749]

Table 11. Total-logprob sensitivity for the same full-step all-router scores, pairs, donor assignments, and bootstrap draws as Table 10. Candidate totals change the relative weighting of verdict strings when their token counts differ. These results do not replace the primary mean-logprob endpoint and are not an independent replication.

Candidate weighting sensitivity

Total log probability uses the same candidate scores as the primary endpoint but weights candidates by their token counts. It is therefore a scoring sensitivity rather than an independent replication.

Gemma illustrates the distinction. All 1,160 four-cell scoring rows contain two target tokens for CORRECT and three for INCORRECT. Let AA denote the error-minus-correct donor change in the INCORRECT candidate’s mean log probability, averaged first over the two receivers and then over pairs, and let BB denote the corresponding change for CORRECT. Then

Dcommonmean=A−B,Dcommontotal=3A−2B.(9)D_{\mathrm{common}}^{\mathrm{mean}} = A - B, \qquad D_{\mathrm{common}}^{\mathrm{total}} = 3A - 2B. \tag*{(9)}

Here A=0.10743A = 0.10743 and B=−0.11327B = -0.11327. The primary mean-logprob result is 0.2207 [−0.0114, 0.4545], whereas the total-logprob result is 0.5488 [0.0034, 1.1020]. The secondary analysis verifies this reweighting for every pair and every bootstrap draw, with maximum absolute residual 1.78×10−151.78 \times10^{-15}. The change is not a uniform positive rescaling, which would preserve whether an interval contains zero.

Gemma’s correct-receiver interval is positive under both endpoints, while its primary DcommonD_{\mathrm{common}} interval includes zero. GLM’s error-receiver interval is slightly negative under mean scoring and includes zero under total scoring. Mean scoring remains the primary endpoint for all checkpoints.

Post hoc direct contrasts and common-input identity

After inspecting the nine-checkpoint results, we computed two classes of secondary direct contrast from the saved four-cell scores: DC−DED_C - D_E within each checkpoint and GPT-OSS 20B minus 120B in DcommonD_{\mathrm{common}}. Each difference is formed within receiver pair before the same 50,000-component resampling, jointly resampling both receiver types or both checkpoints.

Table 12 reports these contrasts. GPT-OSS 20B has DC−DE=0.0874D_C - D_E = 0.0874 [0.0158, 0.1597]. Gemma and GLM have direct receiver-difference intervals that include zero; different zero-crossing behavior of their marginal intervals therefore does not establish a receiver-type difference. Conversely, an interval containing zero does not establish equivalence.

All nine checkpoints share receiver sample identities, source components, and fixed donor assignments. For the two GPT-OSS checkpoints, the secondary analysis additionally compares 1,160 receiver/donor identity occurrences and 2,320 candidate-conditioned complete inputs. It checks saved rendered-prompt hashes, input formats and hashes, step spans, sample/source identities, target IDs and positions, tensor shapes, and hashes recomputed from the complete input_ids arrays.

The full input-identity projection has the same SHA-256 for both checkpoints:

35c1fb039b9cfa0617565d039650f3d8f6 b00e4016a835ecf440bc703416a136.

The archived identity records do not separately store rendered prompt text, but equality of the saved whole-prompt hashes and freshly hashed complete token arrays establishes identity of the full inputs used in this comparison. This check applies only to the full-step fixed-receiver GPT comparison and does not establish identity with earlier TT, local DD, or representation-probe experiments.

The paired GPT-OSS 20B-minus-120B contrast is 0.0751 [0.0206, 0.1301]. It compares two fixed checkpoints under common inputs and donor assignments. Training, checkpoint identity, expert count, and routed-layer coverage still differ, so the result does not isolate parameter count.

Similarly, comparing an unresolved local-router interval with a positive full-step all-router interval is not a direct test of the effect of expanding coverage. Full-step router and local residual interventions also differ in layer and token coverage. Neither comparison identifies an equal-coverage interface difference, a directional fraction D/TD/T, or a compression component T−DT-D. The all-layer MoE-output intervention has no corresponding fixed-receiver DD experiment.

Model / checkpoint contrastEstimate95% CIBootstrap SE
Receiver-role difference: Dcorrect−DerrorD_{\mathrm{correct}} - D_{\mathrm{error}}
Qwen3.60.0185[−0.0545, 0.0897]0.0369
Qwen3−0.0216[−0.1624, 0.1248]0.0733
GLM-4.7-Flash0.0332[−0.0033, 0.0712]0.0190
Lightning0.0170[−0.0115, 0.0461]0.0147
Nano0.0126[−0.0041, 0.0295]0.0086
Gemma40.3335[−0.0215, 0.6877]0.1804
OLMoE−0.0503[−0.1387, 0.0406]0.0459
GPT-OSS-20B0.0874[0.0158, 0.1597]0.0368
GPT-OSS-120B0.0039[−0.0589, 0.0673]0.0324
Checkpoint difference in DcommonD_{\mathrm{common}}
GPT-OSS-20B − GPT-OSS-120B0.0751[0.0206, 0.1301]0.0279

Table 12. Post hoc mean-logprob contrasts for full-step all-router DD. Receiver-type differences are correct minus error within checkpoint; the checkpoint difference is GPT-OSS 20B minus 120B in DcommonD_{\mathrm{common}}. Differences are formed within pair and jointly resampled over the same 235 receiver source components (N=290N = 290, fixed donors; 50,000 PCG64 draws). Intervals are two-sided pointwise 95% percentile intervals without multiplicity adjustment. The GPT checkpoint contrast uses verified common inputs but does not isolate parameter count.

Execution checks

All nine checkpoints completed preflight, smoke, and full execution stages with accepted atomic-pair, identity-control, and native-dispatch checks. Gemma’s resource interruption retained 44 complete pairs from invocation 29687b42-ec5c-4c96-9129-02e06d0136a2; the resumed invocation 4e46dce0-861b-44c9-8ace-f0f1d925818b supplied the remaining 246. An unfinished pair was recomputed in full rather than joining intervention cells across invocations.

Execution version, scientific source, and configuration remained unchanged. The recovery procedure does not alter the receiver-pair estimate or the source-component resampling unit. Recorded full-run and smoke forwards have a lower bound of 63,488. Work interrupted before completion is not recorded exactly, so the fresh-plan count of 63,216 should not be interpreted as an actual execution total.

The secondary verification reconstructed all 54 original mean/total endpoints and reproduced their stored point estimates, interval endpoints, and bootstrap standard errors before computing additional contrasts. The manuscript analysis reuses this accepted primary analysis and introduces no additional model forwards or protected-validation access. Artifact identities are listed in Appendix § 21.

Representation-Probe Analysis

Error-related information and intervention effects remain distinct. We first ask whether routing states contain information that distinguishes correct from erroneous steps, and whether such information predicts the effect of intervening on routing. Using out-of-fold L2-logistic probes grouped by source problem, router-logit representations distinguish the two classes across eight evaluated checkpoints. In the historical six-model panel, router-logit AUROC ranges from 0.801–0.884, compared with 0.829–0.907 for residual states and 0.837–0.915 for router inputs. The GLM and GPT-OSS-120B extensions show the same pattern, with router-logit AUROCs of 0.818 and 0.860, respectively (details below).

A readable signal does not by itself determine intervention strength. GLM, for example, has a router-logit AUROC of 0.818 but an unresolved local router DD and a positive full-step all-router TT. GPT-OSS-120B similarly contains a strong correct/error signal in its router logits while its broad routing intervention behaves differently from its local residual intervention. Thus, information that can be read from a routing representation does not by itself determine how changing that representation will affect the verification judgment. Protocol-specific probe details and cross-checkpoint comparability limitations are reported below.

Probe protocols and checkpoint details

The representation-probe analysis asks whether internal activations contain information that distinguishes correct from erroneous steps. The probe uses L2-regularized logistic classifiers with training-fold standardization and out-of-fold predictions. Folds group the 290 paired examples by canonical source-problem component so that linked examples are not split across training and evaluation folds.

The six-checkpoint probe panel uses a post hoc mapping of the 290 pairs to 235 source components and is therefore retrospective.

The positive class is the error-bearing step. AUROC measures how well a representation separates correct from erroneous steps; it does not measure causal use of that information or identify a correctness-specific representation. We probe residual states, router inputs, router logits, and expert-aligned routing representations.

ModelProtocol gradeAccess / causal layerResidualRouter inputRouter logitsExpert weight
Gemma 4modern15 / 150.8930.9010.8680.792
Nemotron-3 Nanomodern27 / 270.8550.8650.8310.688
Nemotron-3.5modern27 / 270.8650.8620.8260.733
Lightning
Qwen3-6.35B-A3Bmodern20 / 200.8710.8730.8150.790
GPT-OSS 20Bextended12 / 120.8290.8370.8010.715
Qwen3-30B-A3Bhistorical retrospective24 / 360.9070.9150.8840.750
OLMoE-1B-7Bunavailable— / 8————

Table 13. Middle-anchor matched out-of-fold AUROC under 235 canonical source-problem groups. “Access/causal layer” gives the observational recapture anchor and the headline intervention anchor, respectively. “Expert weight” is the expert-ID-aligned sparse gate-weight representation. These are access measurements at capture interfaces, not intervention effects or a mediation sequence.

Across the six checkpoints in the shared probe table, residual and router-input representations generally have higher AUROC than router logits, which in turn exceed expert-aligned sparse-weight representations. A narrower 238-group recapture-native construction changes every cell by at most 0.031 and preserves this ordering.

Qwen3 uses a different probe/intervention layer pair (L24/L36) and is therefore not treated as an equal-layer intervention comparison. OLMoE has no corresponding recapture manifest or activation tensors and is omitted rather than assigned an imputed value. No additional contrast-feature control is constructed because the archived manifests do not contain predeclared topic, language, token-class, or formatting labels.

GLM. The GLM L24 source-group out-of-fold analysis uses 290 pairs, 580 rows, and 235 source components. Residual AUROC is 0.8646 [0.8387, 0.8918], router-input AUROC is 0.8717 [0.8451, 0.8973], router-logit AUROC is 0.8180 [0.7864, 0.8476], expert-aligned sparse-weight AUROC is 0.7427 [0.7082, 0.7775], and expert-membership AUROC is 0.7010 [0.6644, 0.7397].

The direct router-input-minus-router-logit difference is 0.0537 [0.0293, 0.0781]; residual-minus-input is unresolved at −0.0071 [−0.0161, 0.0017]. Intervals condition on fitted out-of-fold predictions and use 1,000 source-component bootstrap draws with seed 20260829. These probe intervals are separate from the 50,000-draw intervention analyses. GLM’s separate 64-pair behavior subset has AUROC 0.8401.

GPT-OSS 120B. The GPT-OSS 120B probe uses the saved L18 recaptures. Residual AUROC is 0.8813 [0.8543, 0.9059], router-input AUROC is 0.8847 [0.8580, 0.9075], router-logit AUROC is 0.8599 [0.8310, 0.8868], expert-aligned sparse-weight AUROC is 0.7039 [0.6678, 0.7433], and expert-membership AUROC is 0.6704 [0.6326, 0.7128].

The direct input-minus-logit difference is 0.0248 [0.0046, 0.0446], logit-minus-aligned-weight is 0.1560 [0.1132, 0.1920], and aligned-weight-minus-membership is 0.0335 [0.0207, 0.0474]. Residual-minus-input is unresolved at −0.0034 [−0.0112, 0.0042].

The analysis reuses 290 pairs, 580 rows, 235 source components, and five grouped folds, with training-fold preprocessing and L2-logistic fitting. Intervals condition on the fitted out-of-fold predictions and use 1,000 source-component draws with seed 20260829.

The GPT-OSS 20B and 120B recaptures share raw prompt content, pair identities, and verification boundaries but differ in the rendered Harmony system-header date. Their rendered-prompt and token-input hashes therefore differ for all 580 rows. Together with checkpoint and architecture differences, this prevents interpreting their probe difference as a causal effect of model scale.

Text baseline on the same folds. A fixed word unigram/bigram TF–IDF baseline uses the boundary-step text and the same L2-logistic settings as the router probes (C=1C = 1, balanced classes, liblinear, 2,000 maximum iterations), without tuning. Vocabulary and IDF are fit only within each training fold. The comparison uses all 580 samples, 290 pairs, 235 source components, and the same five folds. Identical text inputs in the five paired-first-error protocols yield one fitted baseline reused across checkpoints.

Table 14 reports router-logit comparisons with this text baseline. Intervals jointly resample 235 source components 1,000 times with seed 20260829 while conditioning on the fitted out-of-fold predictions; the classifiers are not refit within each draw. Qwen3’s distinct boundary construction remains separate.

Model (access layer)Text AUROC [95% CI]Router minus text [95% CI]
Gemma 4 (L15)0.743 [0.710, 0.777]+0.124 [0.084, 0.159]
Nemotron-3 Nano (L27)0.743 [0.710, 0.777]+0.088 [0.051, 0.123]
Nemotron-3.5 Lightning (L27)0.743 [0.710, 0.777]+0.083 [0.043, 0.116]
Qwen3.6-35B-A3B (L20)0.743 [0.710, 0.777]+0.072 [0.031, 0.110]
GPT-OSS-20B (L12)0.743 [0.710, 0.777]+0.057 [0.024, 0.092]
Historical boundary sensitivity
Qwen3-30B-A3B (L24)0.734 [0.696, 0.771]+0.150 [0.110, 0.190]

Table 14. Boundary-step text baseline on the identical 580 samples, 235 source components, and five saved interface-access folds. Differences compare the middle router-logit probe with the fixed text baseline; 95% intervals use paired source-component resampling.

A router advantage over this TF–IDF baseline does not establish a correctness-specific representation or remove all surface confounds. Contextual router states contain information unavailable to isolated boundary-step word features, and the representation dimensions and classifier capacities are not matched.

In Qwen3’s separate matched surface comparison, routing and text features reach AUROCs of 0.777 and 0.795, with paired difference −0.019 [−0.047, 0.010]. This result is compatible with readable routing information but does not establish causal or selectively used routing information.

The representation-probe and intervention analyses measure different objects. The probes passively read residual, router-input, router-logit, and expert-aligned routing representations, whereas the intervention analyses replace activations at adapter-defined boundaries and measure changes in the verification margin. Similar interface names do not make these measurements stages of a serial mediation pathway.

Natural-Error Evaluation on ProcessBench

The interface difference also appears on naturally occurring errors. We next test whether the local residual-versus-router difference extends beyond the constructed PRMBench pairs. For this exploratory extension, we construct 64 cross-problem pairs from ProcessBench Zheng et al. (2025), each matching a trace with a naturally occurring first error to an all-correct trace from the same generator at the same step index. We evaluate GPT-OSS-20B, Gemma, and Nemotron-3 Nano using the same local residual and router-logit interventions.

Residual interventions produce larger absolute changes than router-logit interventions in all six model-by-direction comparisons. The paired residual-minus-router TT difference is also positive in all three models: 0.0476 [0.0145, 0.0817] for GPT-OSS-20B, 0.1241 [0.0246, 0.2402] for Gemma, and 0.0575 [0.0396, 0.0755] for Nano. Five of the six residual point estimates move in the direction associated with the replacement activation’s correctness; Gemma’s correct-to-error direction is the exception. An exploratory analysis relates this exception to whether the receiver initially missed the error, but this analysis is observational and subject to baseline conditioning effects (Section H.2).

These results provide complementary evidence that the local interface difference is not limited to the constructed PRMBench errors. However, ProcessBench uses only three models and 64 cross-problem pairs and does not include the fixed-receiver DD or token-window analyses. We therefore treat it as a separate natural-error extension rather than part of the primary nine-model comparison.

Evaluation setup and supplementary results

The ProcessBench extension tests whether the local residual-versus-router difference also appears for naturally occurring reasoning errors. From the public GSM8K subset, 193 of 400 source traces are all-correct and 207 contain a first-error label. We construct 64 same-generator pairs, each linking a trace with a naturally occurring first error to an all-correct trace at the same step index. The resulting 128 traces contain no trace reuse, and all 64 pairs are cross-problem matches.

GPT-OSS-20B, Gemma, and Nemotron-3 Nano use the same processed-data selection, binary verification prompt, boundary policy, two replacement directions, dose α=1\alpha= 1, identity conditions, and outcome metric. The reported tables use a 50,000-resample analysis of the immutable pair-level scores. Architecture-specific anchors remain unchanged. A scoped Nemotron adapter repair changed neither the samples, boundaries, estimand, nor intervention comparisons.

On the balanced 128-trace endpoint, GPT-OSS has accuracy 0.625 and AUROC 0.736; Gemma has accuracy 0.688 and AUROC 0.854; and Nemotron-3 Nano has accuracy 0.641 and AUROC 0.824. Each full run contains 128 behavior rows, 128 recapture rows, and 768 intervention rows. Artifact-integrity and coverage checks are independent of effect direction and statistical significance.

Residual directional intervals exclude zero in four of the six checkpoint-by-direction comparisons, whereas router intervals exclude zero in none. Five of six residual point estimates move in the direction associated with the correctness of the replacement activation. Gemma’s correct-to-error direction is the exception. Nano has the expected sign in both directions; GPT-OSS is clear only in the correct-to-error direction; Gemma has a large expected effect in the reverse direction. These differences motivate the receiver-state analysis below.

ModelSiteMC→\toFEFE→\toMCTTG=Tres−TrouterG = T_{\mathrm{res}} - T_{\mathrm{router}}
GPT-OSS 20Bresidual−.0590-.0590<br>[−.1029,−.0163-.1029, -.0163]+.0242+.0242<br>[−.0344,.0824-.0344, .0824].0416.0416<br>[ .0065,.0775.0065, .0775].0476.0476<br>[ .0145,.0817.0145, .0817]
router+.0051+.0051<br>[−.0060,.0164-.0060, .0164]−.0070-.0070<br>[−.0174,.0035-.0174, .0035]−.0060-.0060<br>[−.0143,.0025-.0143, .0025]
Gemma 4 26B-A4Bresidual+.0442+.0442<br>[−.0813,.1594-.0813, .1594]+.2948+.2948<br>[ .1424,.5035.1424, .5035].1253.1253<br>[ .0224,.2452.0224, .2452].1241.1241<br>[ .0246,.2402.0246, .2402]
router+.0010+.0010<br>[−.0339,.0358-.0339, .0358]+.0034+.0034<br>[−.0203,.0268-.0203, .0268].0012.0012<br>[−.0183,.0205-.0183, .0205]
Nemotron-3 Nanoresidual−.0517-.0517<br>[−.0686,−.0345-.0686, -.0345]+.0667+.0667<br>[ .0454,.0897.0454, .0897].0592.0592<br>[ .0410,.0778.0410, .0778].0575.0575<br>[ .0396,.0755.0396, .0755]
router−.0033-.0033<br>[−.0093,.0027-.0093, .0027]+.0001+.0001<br>[−.0054,.0056-.0054, .0056].0017.0017<br>[−.0024,.0056-.0024, .0056]

Table 15. ProcessBench natural-error transfer on 64 same-generator, cross-problem pairs per model. MC→\toFE uses a matched-correct donor and first-error receiver; FE→\toMC swaps their roles. Each cell shows an estimate above its paired-cluster percentile-bootstrap 95% interval (50,000 resamples; no multiplicity adjustment). TT and GG come directly from paired rows, not marginal intervals.

Receiver-state analysis

We reanalyze the same 64 pairs in all three checkpoints while retaining both replacement directions and both intervention sites. Receivers are grouped by whether their unpatched verification margin correctly judges the evaluated boundary. Group intervals condition on the observed group sizes: 50,000 PCG64 draws with seed 0 independently resample pairs within the two groups, and group differences are computed from the joint draws.

These exploratory, unadjusted intervals describe an observational grouping. The baseline margin used to define the groups also appears with a negative sign in the intervention effect, so the resulting group comparisons are subject to baseline conditioning.

Figure 5 shows that Gemma’s positive correct-to-error point estimate is not attributable to a single pair: 37/64 effects are positive, the median is 0.0781, and all 64 leave-one-out means lie between 0.0281 and 0.0752.

Figure 5. Exploratory correct-to-error residual intervention on ProcessBench. (a) All 64 Gemma pair effects, ordered by value; the dashed line is their mean. (b) Mean effects grouped by whether the receiver initially detects the error, with sample counts in parentheses. Intervals use 50,000 percentile bootstrap draws within the observed receiver-state groups. Negative effects correspond to the expected direction for this intervention.

The mean is 0.3320 for the 11 initially missed errors and −0.0155 for the 53 initially detected errors, a difference of 0.3475 [0.1342, 0.5708]. This does not imply that every positive effect belongs to the missed-error group; 28/53 detected-error receivers also have positive effects. Table 16 reports the complete two-direction decomposition.

ModelDirectionCorrectly judged<br>nn, meanMisjudged<br>nn, meanGroup contrast<br>[95% CI]
GPT-OSSC→\toE46, −.077618, −.0114.0662 [−.0249, .1564]
E→\toC34, .043630, .0022−.0415 [−.1618, .0747]
Gemma 4C→\toE53, −.015511, .3320.3475 [.1342, .5708]
E→\toC35, .312629, .2733−.0392 [−.4320, .2568]
NanoC→\toE59, −.05745, .0164.0738 [−.0272, .1340]
E→\toC23, .101841, .0469−.0549 [−.1085, −.0053]

Table 16. Exploratory residual-transfer effects by the receiver’s unpatched judgment. Entries give mean shifts in nats/target-token; the group contrast is misjudged minus correctly judged, with marginal 95% intervals from 50,000 within-group pair-bootstrap draws. C→\toE is matched correct to first error; E→\toC is the reverse. All original pairs remain included.

Construction and shared-problem sensitivity. The 64 pairs cover 12 generators and first-error step indices zero through five, with counts 8, 17, 18, 12, 5, and 4. Although 49 pairs have equal full-trace step counts, the median absolute difference in visible step-text length is 176 characters; matching the step index therefore does not match the amount of visible text.

The 128 traces represent 125 normalized problems, linking the pairs into 61 connected components: 59 singletons, one two-pair component, and one three-pair component. A sensitivity analysis jointly resamples these components, retains every member pair, and computes pair-weighted means.

With 50,000 PCG64 draws and seed 0, residual-minus-router gaps are 0.0476 [0.0126, 0.0831] for GPT-OSS, 0.1241 [0.0239, 0.2412] for Gemma, and 0.0575 [0.0389, 0.0759] for Nano. These supplementary intervals preserve the positive-gap result without replacing the canonical pair-bootstrap analysis. Gemma’s receiver-state contrast is 0.3475 [0.1299, 0.5779] under the same component resampling.

Routing changes and signed averages. Gemma’s near-zero mean router effect averages over pair-level changes with opposing signs. In the correct-to-error direction, 30 pair effects are positive, 33 are negative, and one is zero. Their mean absolute magnitude is 0.1108, compared with a signed mean of 0.0010; the residual mean absolute magnitude is 0.3197.

A small signed router average therefore does not imply that every individual router intervention leaves the verification judgment unchanged. Native top-kk changes and output magnitudes remain descriptive manipulation diagnostics rather than calibrated causal doses or mediation quantities. No additional donor controls or error-type labels are introduced in this analysis.

Additional Controls and Implementation Checks

Identity controls

Unpatched reruns and self-patch conditions verify that the intervention machinery preserves the receiver when no cross-example activation replacement is made. Failure of an identity condition would invalidate the corresponding cross-example intervention rows. Passing the identity check is an implementation validation and is not evidence that the natural causal contribution of the intervention site is zero.

Native routing execution

The intervention adapters retain each checkpoint’s native expert selection, normalization, and dispatch after router-logit replacement. Execution artifacts record patched logits together with the downstream expert IDs and weights used for dispatch. Machine checks compare these quantities under each checkpoint’s native top-kk and normalization rules. These checks are manipulation validation rather than cross-checkpoint effect-size covariates.

ProcessBench separately contains 256/256 verified dispatch rows per checkpoint. Cross-example router interventions change native top-kk selection in 49/64 GPT-OSS, 63/64 Gemma, and 61/64 Nano rows in each direction.

Matched-donor construction and interpretation

Matched external donors are selected by prioritizing agreement in error type and proximity of the first-error position, with a seeded hash used only for tie-breaking. The matched pair’s correct and error members supply the same donor roles as the corresponding paired-donor condition. Donor assignments are fixed for analysis, and matched donors come from different source-problem components.

These controls test whether using the paired donor produces a larger response than using a matched external donor. The primary comparison is the direct paired-minus-matched difference. A relative comparison, when reported in the archived artifacts, divides the difference of means by the paired-donor effect; it is not a mean of pairwise ratios and should not be interpreted as the fraction of an underlying correctness mechanism.

Positive TT can still arise from compression of the original correct/error judgment gap (Equation (5)). Matched-donor controls do not remove this ambiguity because donor and receiver roles still change together. The fixed-receiver experiments address the separate question of whether the intervention effect changes when the receiver is held constant and the replacement activation changes from a correct to an erroneous step. Neither analysis decomposes TT into mechanistic components.

Norm-random controls

The norm-random intervention adds a Gaussian direction to the receiver activation, scaled to the ℓ2\ell_2 norm of the paired donor–receiver displacement at the corresponding intervention site. It tests whether an intervention of similar magnitude but random direction produces a comparable response. A smaller norm-random effect does not exclude structured but correctness-independent activation replacement.

For full-step all-router interventions, paired-minus-norm-random intervals are positive in all nine checkpoints, with OLMoE close to zero. For the all-layer MoE-output intervention, every paired-minus-norm-random interval is positive. By contrast, paired-minus-matched-donor intervals for the all-layer MoE-output intervention all include zero. These controls therefore show effects beyond random displacement magnitude without resolving paired-example specificity.

Reproducibility and Artifact Records

The nine-checkpoint inputs come from three campaign archives: campaign_20260907_A (Qwen3.6, Qwen3, GLM), campaign_20260907_B (Lightning, Nano, Gemma, OLMoE), and campaign_20260907_C (GPT-OSS 20B and 120B). Their resolved configurations, executed adapters, scores, donor manifests, token geometry, and native analyses define the experimental identities.

The abc_synthesis_20260908 CPU analysis reads these artifacts, reads the saved model outputs and computes the common-window and direct-contrast analyses. It executes no new model forwards and does not alter the discovery selection. The protected validation split is not used.

The versioned window JSON has SHA-256 7543e13c3b3cf0ebba9f8a86df443fcd a5c7fd10b6cd2fb28fd3909d24a1ac; the DD/bundle headline CSV has SHA-256 0be3afb97f570165d0d7a0e662ba32c2f b9c6f8b5e58f55222ce13c86c636971. Machine-readable rows include the source file, analysis version, and JSON pointer or pair identity.

Full-step all-router DD records. The separate campaign_20260908_D/final_v1 archive retains each checkpoint’s atomic four-cell scores, full input identities, resolved configuration, adapter and runner hashes, native 50,000-draw analysis, and recovery records.

Its wide_d_nine_model_v1.json summary has SHA-256

6b1ccd80ecd58d01025ee41f3b2c7be2 74875a831509018244045019fa98e14b.

The wide_d_synthesis_20260908 analysis stores the main summary and a separately identified wide_d_posthoc_secondary_contrasts_v1 artifact. Its receipt records source-member hashes, the complete GPT input-identity projection, Gemma candidate-weight checks, and exact reproduction of the original endpoints. These records do not overwrite the A/B/C campaign artifacts or the representation-probe inputs. No protected-validation split was used in the full-step DD experiment or its secondary analysis.

Representation-probe and ProcessBench analyses have separate input and analysis records. Exact model revisions, pair manifests, fold assignments, seeds, and source hashes are stored in the analysis archives.

Claim Boundaries

The experiments support four main conclusions within the tested settings. First, post-MoE residual interventions produce larger local responses than router-logit and MoE-output interventions across all nine checkpoints and all four token windows. Second, expanding routing interventions across routed layers and tokens can substantially increase their effect and can change the ordering relative to a local residual intervention. Third, fixed-receiver experiments show correctness-dependent local residual effects in eight checkpoints, with a substantial part of OLMoE’s residual DD associated with surface content (Appendix L); the corresponding broad-routing effects vary across checkpoints and receiver types. Fourth, all-layer MoE-output interventions exceed norm-matched random controls, while the current data do not resolve an advantage of paired donors over matched external donors.

More specifically, the full-step all-router fixed-receiver analysis provides a positive DcommonD_{\mathrm{common}} interval for GPT-OSS 20B and a positive correct-receiver interval for Gemma. Post hoc direct contrasts support a larger correct-receiver effect for GPT-OSS 20B and a larger DcommonD_{\mathrm{common}} for GPT-OSS 20B than GPT-OSS 120B. Qwen3.6 remains an explicit exception to the common local-residual fixed-receiver pattern.

These conclusions retain the intervention interface, layer and token coverage, receiver type, verification endpoint, and checkpoint scope under which they were measured. Intervals containing zero do not establish equivalence. The representation-probe and ProcessBench analyses provide complementary evidence under their own protocols and are not pooled with the nine-checkpoint intervention study.

The experiments do not establish that routing is behaviorally irrelevant, that any intervention has exactly zero effect, that the tested interfaces form a formal mediation pathway, that residual states are the unique mechanism of verification, that the observed effects isolate a pure correctness representation, that paired-example identity is required for an intervention effect, or that the findings hold universally across MoE models and verification tasks.

Surface Controls and Routing Diagnostics

The fixed-receiver contrast DD holds the receiver constant, but its two donors differ in both correctness and content. We examine how much of the residual contrast can be reproduced by surface differences between donors with the same label. We then measure whether router interventions change expert selection and whether the two donor roles in DD produce comparable routing perturbations. These analyses use the saved outputs and diagnostics of the existing intervention runs.

Surface-oriented same-label placebo

Construction and inference. The fixed-receiver and token-window runs use the same donor-ranking rule with different seeds for breaking ties. For 195 of the 290 receiver pairs, they selected different external donor pairs. This subset contains 166 receiver source components and is identical across models. For each receiver trace, we compare its patched margins under the two external donors with the same correctness label: both erroneous for a correct receiver, and both correct for an erroneous receiver. The intervention is at the last-token post-MoE residual interface in both runs.

To orient this placebo, we use the out-of-fold error score of the TF-IDF text classifier. The placebo is the margin under the higher-scoring donor minus the margin under the lower-scoring donor. We average the two receiver roles within each pair, as for DcommonD_{\mathrm{common}}. Thus, a positive placebo means that the more error-like surface content produces a more INCORRECT-favoring judgment even though the donor label is fixed. We compare this quantity with real DcommonD_{\mathrm{common}} on the same retained receiver pairs; below, DD denotes DcommonD_{\mathrm{common}}. There are no TF-IDF ties in the eligible subset. Qwen3 is excluded because its fixed-receiver and token-window runs intervene at different layers.

We use 95% receiver source-component percentile intervals with 50,000 PCG64 resamples and seed 0. Real DD, placebo, and their difference share the same resamples, so the difference interval is paired. This is a post hoc analysis of existing runs. Before computing the interval estimates, we fixed three conditions for a model to pass:

  • (a) the lower confidence bound of real DD is above zero;

  • (b) the placebo interval includes zero, or its upper bound is below one quarter of the real-DD point estimate; and

  • (c) the lower confidence bound of real DD minus placebo is above zero.

The aggregate criterion requires at least six of the eight eligible models to pass all three conditions.

Results and sensitivity. Six of eight models pass (Table 17). Qwen3.6 does not have a resolved positive residual DD, consistent with the full-sample result. OLMoE has a positive same-label placebo amounting to about two thirds of its real DD. Its real-minus-placebo contrast is 0.057 [−0.009, 0.124], so a substantial part of its residual DD is associated with surface content.

ModelReal DDPlaceboReal DD − placebo
Qwen3.60.008 [−0.014, 0.031]−0.013 [−0.039, 0.013]0.021 [−0.016, 0.062]
GLM-4.7-Flash0.183 [0.144, 0.222]−0.004 [−0.036, 0.029]0.186 [0.134, 0.238]
Lightning0.140 [0.117, 0.164]0.016 [−0.003, 0.035]0.124 [0.093, 0.156]
Nano0.075 [0.063, 0.088]0.004 [−0.006, 0.014]0.072 [0.055, 0.088]
Gemma40.805 [0.558, 1.068]−0.157 [−0.418, 0.105]0.961 [0.562, 1.381]
OLMoE0.169 [0.102, 0.241]0.111 [0.062, 0.163]0.057 [−0.009, 0.124]
GPT-OSS-20B0.130 [0.095, 0.167]0.030 [−0.012, 0.070]0.100 [0.051, 0.151]
GPT-OSS-120B0.213 [0.175, 0.254]0.019 [−0.015, 0.054]0.194 [0.144, 0.247]

Table 17. Same-label placebo at the last-token residual interface. Real DD, TF-IDF-oriented placebo, and their paired difference use the same 195 receiver pairs in 166 source components. Brackets give 95% source-component percentile intervals. Differences are computed before rounding. All models except Qwen3.6 and OLMoE meet the three conditions in Appendix § 23.§ 1.

Orienting the placebo by character count gives six passing models, whereas word-count orientation and the total-logprob margin each give five. The count-based orientations subtract the shorter donor’s margin from the longer donor’s margin. They drop a receiver pair if either same-label comparison ties, leaving 194 pairs for character count and 183 for word count; real DD is recomputed on the corresponding subset. The total-logprob sensitivity retains TF-IDF orientation and all 195 pairs. GPT-OSS-20B is the additional model that fails the criterion for word count and total log probability: its placebo is positive, but real DD minus placebo remains positive. Character-count orientation is an additional sensitivity analysis, separate from the conditions fixed before estimation.

The category-exclusion sensitivity removes circular and redundancy pairs, leaving 281 pairs overall and 194 placebo-eligible pairs in 166 source components. The TF-IDF-oriented test still passes in six of eight models with the same two exceptions (bootstrap seed 20260908).

This comparison has two limitations. First, eligibility depends on ties in the donor-ranking rule, so these donors and receiver pairs are not a random sample for the placebo test. Second, the two margins come from different runs. For Nano and Lightning, even identical donors produce nonzero cross-run differences, so their placebo includes numerical drift. Both still pass when each patched margin is first differenced from its own run’s unpatched receiver margin. For the models that pass, the supported interpretation is dependence on donor correctness beyond this lexical proxy; the test does not isolate pure correctness or rule out surface features that TF-IDF does not capture.

Residual effects when the donor surface gap is small

The placebo compares different donors with the same label. A complementary analysis asks whether real DD persists when the TF-IDF classifier does not rank the erroneous donor as more error-like than the correct donor. For the external donor pair used in DD, define Δs=s(Ej)−s(Cj)\Delta s = s(E_j) - s(C_j), where ss is the out-of-fold TF-IDF error score. We report the average residual DD on pairs with Δs≤0\Delta s \le0. We also fit an ordinary least-squares regression of pair-level DD on Δs\Delta s using all 290 receiver pairs and report its intercept at zero gap. The regressions are refitted within each of 50,000 source-component resamples with seed 0. Qwen3 is excluded because its archived out-of-fold scores do not align with the evaluated boundary steps.

There are 80 pairs with Δs<0\Delta s < 0. Residual DD remains positive in seven of eight models on this subset (Table 18), including Qwen3.6, whose full-sample contrast is unresolved. OLMoE is the exception: its subset estimate is 0.001 [−0.022, 0.024]. Its fitted zero-gap intercept is nevertheless positive, 0.056 [0.012, 0.102]. The intercept uses a linear fit across the full gap range, whereas the subset mean averages only observations at or below zero. Together with the placebo, these results support a substantial surface-associated component in OLMoE without establishing that all of its residual effect is explained by TF-IDF. The gap analysis is correlational.

Table 18: Residual DD and the TF-IDF donor gap. The left column uses the 80 receiver pairs in 78 source components for which the erroneous donor’s TF-IDF error score is no higher than the correct donor’s. The right column is the zero-gap intercept of a linear regression on all 290 pairs in 235 source components. These are distinct estimands. Brackets give 95% source-component percentile intervals.

ModelDD where Δs≤0\Delta_s \le0Intercept at Δs=0\Delta_s = 0
Qwen3.60.038 [0.003, 0.081]0.021 [−0.002, 0.047]
GLM-4.7-Flash0.124 [0.071, 0.182]0.145 [0.109, 0.182]
Lightning0.104 [0.073, 0.137]0.125 [0.101, 0.149]
Nano0.062 [0.045, 0.081]0.065 [0.052, 0.078]
Gemma41.024 [0.602, 1.478]0.864 [0.614, 1.130]
OLMoE0.001 [−0.022, 0.024]0.056 [0.012, 0.102]
GPT-OSS-20B0.060 [0.027, 0.095]0.076 [0.046, 0.106]
GPT-OSS-120B0.151 [0.095, 0.215]0.158 [0.123, 0.195]

Table 18.

Routing manipulation and donor perturbation size

Measurements. For each router intervention, the saved diagnostics record whether any member of the selected top-kk expert set changes, the fraction of experts retained, and the KL divergence from receiver to replacement router logits after applying softmax over all experts. A changed top-kk set means that at least one selected expert differs; this measure does not detect weight changes within an unchanged set. We average each statistic over the two candidate-scoring forwards and over patched router sites within a receiver trace, then average across receiver traces. The last-token single-layer comparisons have 580 receiver traces per model and treatment.

Single-layer paired router interventions change top-kk membership in approximately 35% to 93% of cases across models (Table 19). In the fixed-receiver runs used for DD, the corresponding changed-set fraction ranges from approximately 45% to 100% across models and donor roles. Thus, unresolved single-layer router DD does not imply that expert selection is unchanged. Across the four windows and the two router intervention scopes, matched external donors produce approximately 1.6 to 2.3 times the mean router KL of paired donors. In contrast, the erroneous-to-correct donor KL ratio in the fixed-receiver DD runs ranges from 0.90 to 1.09 across the single-layer and full-step all-router settings. The two donor roles therefore produce similar average perturbation sizes, although this comparison does not establish that individual interventions are matched in every respect.

Table 19: Routing changes and perturbation-size comparisons. Top-kk changed is the mean changed-set fraction for a paired last-token single-layer router intervention. The matched/paired KL range spans all four token windows and both single-layer and all-router interventions. The last two columns compare the erroneous and correct external donors used in DD: the ratio of their mean KL values at the last-token single layer and under full-step all-router replacement, respectively.

ModelTop-kk changedMatched/paired KL rangeDD donors: E/C KL single layerDD donors: E/C KL all-router
Qwen3.60.9261.715–2.1190.9980.998
Qwen30.6831.585–2.1440.9390.995
GLM-4.7-Flash0.6361.657–2.3310.9991.016
Lightning0.8831.763–2.1600.9441.010
Nano0.8431.738–2.1410.9001.009
Gemma40.8861.560–1.9341.0911.009
OLMoE0.3481.612–2.2131.0301.024
GPT-OSS-20B0.6551.676–2.2241.0490.999
GPT-OSS-120B0.7091.636–2.2020.9551.000

Table 19.

Interpretation boundaries. KL here is a common projection of router logits, not necessarily the divergence of the native dispatch weights. Some models use gate functions other than softmax over all experts, and raw KL values should not be compared across models. Under all-router replacement, the receiver logits at a later layer are computed after earlier layers have already been patched; the diagnostic is relative to that forward’s state, not the native unpatched routing. At multi-token windows, paired donors also share prefix positions with the receiver, whereas matched donors do not. Part of their perturbation-size difference is therefore positional.

Routing effects relative to the residual effect and to TT

Table 20 expresses the full-step interventions on a relative scale. The first three columns give the paired effect TT of the single-layer router, single-layer MoE-output, and all-layer router interventions as a percentage of the single-layer residual TT at the same token window. Each entry is a ratio of means over the same pairs, with a 95% percentile-bootstrap interval in which numerator and denominator share resamples. Over the full step, the single-layer router effect is at most 5.9% of the residual effect, whereas the all-layer router effect ranges from 24.8% to 133.8%.

ModelRouter, one layerMoE output, one layerRouter, all layersAll-router DcommonD_{\mathrm{common}} as % of all-router TT
Qwen3.6−0.2 [−1.0, 0.6]0.3 [−0.8, 1.3]27.0 [23.2, 30.9]3.0 [−8.1, 14.0]
Qwen30.5 [0.0, 1.1]2.8 [1.9, 3.8]37.5 [33.7, 41.2]−5.4 [−11.9, 1.1]
GLM-4.7-Flash1.4 [−0.6, 3.3]23.3 [20.3, 26.3]54.8 [47.1, 62.7]−5.4 [−15.7, 4.8]
Lightning0.1 [−1.6, 1.6]7.1 [4.9, 9.3]101.5 [88.7, 114.9]5.8 [−2.3, 13.8]
Nano−0.4 [−2.6, 1.8]10.3 [7.9, 12.7]110.4 [95.9, 125.4]−2.2 [−9.3, 4.8]
Gemma41.2 [0.4, 2.1]18.5 [15.8, 21.3]24.8 [20.0, 29.8]18.3 [−0.9, 37.6]
OLMoE3.1 [−1.6, 7.8]19.2 [12.5, 26.0]57.1 [41.5, 73.8]8.8 [−20.3, 38.5]
GPT-OSS-20B1.4 [0.5, 2.4]9.7 [7.9, 11.4]25.4 [21.4, 29.5]23.6 [6.1, 41.3]
GPT-OSS-120B5.9 [2.4, 9.4]15.4 [11.8, 18.8]133.8 [112.9, 156.3]−1.2 [−9.2, 6.8]

Table 20. Routing effects on a relative scale (full step). Columns 2 to 4: paired effect TT as a percentage of the single-layer residual TT, with 95% intervals. Column 5: all-router DcommonD_{\mathrm{common}} and its interval ends as a percentage of the all-router TT point estimate.

The last column divides the full-step all-router DcommonD_{\mathrm{common}} of Table 10, and the two ends of its interval, by the full-step all-router TT. If donors with the receiver’s own label left the margin unchanged and matched donors acted like paired donors, DcommonD_{\mathrm{common}} would equal TT under Eqs. 4 and 6. The ratio can therefore be read as the share of the routing effect that depends on donor correctness. Its upper interval end is below 7% in four checkpoints (Qwen3, GLM-4.7-Flash, Nano, and GPT-OSS-120B) and below 15% in two more (Lightning and Qwen3.6). In Gemma4 and OLMoE the interval is too wide to bound the share, and GPT-OSS-20B is the only checkpoint with a resolved positive share.

Interpretation boundaries. These ratios are descriptive and were computed after the main analyses. The denominator of the last column is a point estimate, so its uncertainty is not propagated. TT uses paired donors, whereas DD uses matched donors from another problem. No equivalence margin was specified in advance, and an interval containing zero does not establish equivalence.

References

  1. [1]Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, and Nanyun Peng. Multilingual routing in mixture-of-experts. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/1b558190825286a3defcc78d02fa2189-Paper-Conference.pdf.arxiv.org/abs/2510.04694
  2. [2]Cedric Caruzzo, Donggeun Yoo, and Tae Soo Kim. Routing divergence is not evidence of behavioral influence in same-weight MoE self-distillation, 2026. URL https://arxiv.org/abs/2608.15787.
  3. [3]Yucheng Du and Xiyang Hu. Recognition-refusal misalignment in LLMs: Why models answer structurally unanswerable questions, 2026. URL https://arxiv.org/abs/2608.29109. Accepted to EMNLP 2026 Main Conference.
  4. [4]Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021. doi: 10.1162/tacl_a_00359. URL https://aclanthology.org/2021.tacl-1.10/.DOI
  5. [5]Leonard Engmann, Christian Medeiros Adriano, and Holger Giese. From observation to intervention: A causal audit of expert importance in mixture-of-experts models, 2026. URL https://arxiv.org/abs/2606.10703. Accepted at the ICML 2026 Workshop on Philosophy of Science Meets Machine Learning (PhilML), non-archival.
  6. [7]Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15466–15490. PMLR, 2024. URL https://proceedings.mlr.press/v235/ghandeharioun24a.html.arxiv.org/abs/2401.06102
  7. [8]Parvel Gu. Detecting a route flip is easier than knowing whether to fix it: Causal route-mediated damage in quantized mixture-of-experts, 2026. URL https://arxiv.org/abs/2608.11212.
  8. [9]Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharian. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Advances in Neural Information Processing Systems, volume 36, pp. 17643–17668. Curran Associates, Inc., 2023. doi: 10.52202/075280-0774. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/3927bbdcf0e8d1fa8aaa23c26f358a281-Paper-Conference.pdf.
  9. [10]Jeremy Herbst, Stefan Wermter, and Jae Hee Lee. The expert strikes back: Interpreting mixture-of-experts language models at expert level. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2604.02178.
  10. [11]John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733–2743. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1275. URL https://aclanthology.org/D19-1275/.DOI
  11. [12]Wengang Li, Lingqi Zhang, Toshio Endo, and Mohamed Wahib. Understanding cross-layer contributions to mixture-of-experts routing in LLMs. In International Conference on Learning Representations, pp. 37420–37446, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/3efdd3ccd70da09ec6edb3b3d17d2fd5-Paper-Conference.pdf.
  12. [13]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations, pp. 39578–39601, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/aca97732e30bcf1303bc22ac3924fd16-Paper-Conference.pdf.arxiv.org/abs/2305.20050
  13. [14]Yuetian Lu, Ali Modarressi, Yihong Liu, and Hinrich Schütze. Expert-aware causal tracing of factual recall in sparse MoE language models, 2026. URL https://arxiv.org/abs/2606.03780v2. Version 2, revised September 3, 2026.
  14. [15]Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In International Conference on Learning Representations, volume 2024, pp. 26486–26515, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/70b8505ac793e311756f793cd80eb8d-Paper-Conference.pdf.arxiv.org/abs/2311.17030
  15. [16]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, volume 35, pp. 17359–17372. Curran Associates, Inc., 2022. doi: 10.52202/068431-1262. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf.DOI
  16. [17]Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhatia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah Smith, Pang Wei Koh, Amanpreet Singh, and Hanna Hajishirzi. OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations, pp. 62061–62121, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/9b224ace8963c9385ad5e2b5c9039b97-Paper-Conference.pdf.arxiv.org/abs/2409.02060
  17. [18]Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. Probing semantic routing in large mixture-of-expert models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 18263–18278. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-emnlp.991. URL https://aclanthology.org/2025.findings-emnlp.991/.DOI
  18. [19]OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925.

Paper details

vlnc:W_976stu2q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Claude Code, Cursor

View statement
Looking for related papers…
Contents