Introduction
A mixture-of-experts (MoE) Transformer uses a router to select among neural modules, called experts, when processing an input representation [34 , 21 , 8 , 5 ] . Different layers can maintain separate experts, each with its own parameters, or use a common set of expert parameters across layers, which is called expert sharing [37 , 14 , 2 , 17 , 32 ] . Training information from different layers then shapes the same learned function. For multi-step reasoning, evaluating this choice requires understanding both the accuracy of individual computations and their contribution to the final output.
Choosing which computations should share an expert requires comparing the final-output errors of the resulting shared and separate models. A natural statistical rationale for sharing is that combining information from related contexts can reduce estimation variance, at the cost of bias when their required transformations differ. Multi-task learning provides a foundation for understanding this trade-off [38 , 39 , 6 , 15 ] . However, even when this trade-off favors sharing in average local prediction error, the corresponding final-output comparison remains a separate question. Shared parameters can change the dependence between errors at different calls, while subsequent computations determine how those errors combine. A criterion based on local error therefore needs an additional connection to the accuracy of the complete computation. The comparison we seek must account jointly for how sharing changes estimation errors and how downstream computation transforms them, identifying when a local preference for shared experts remains justified at the final output. We therefore ask:
When does expert sharing reduce average local prediction error, and under what conditions is this advantage preserved at the final output?
Our analysis shows that the preference can reverse even when both models use the same fixed linear propagation map. Sharing can lower average local prediction error while increasing expected squared final-output error. The reason is that a local improvement can coexist with greater bias in directions amplified downstream, or with a change in how errors across calls reinforce or cancel. We derive exact risk comparisons that retain these effects and characterize the propagation conditions under which every strict local improvement is preserved in a specified sharing model.
Beyond its effect on accuracy, sharing raises a question about what an individual expert computes. Repeated selection of an expert may suggest that it implements an operation common to several contexts. Causal abstraction evaluates such interpretations through interventions on internal computation [9 , 10 ] . We apply this perspective by changing expert choices while holding the pre-action context fixed and comparing the resulting effects with proposed operations.
Contributions
Exact risk comparisons for expert sharing. In specified linear regression models, we derive necessary and sufficient conditions for a common estimate to outperform separate estimates by balancing variance savings against bias from differences between targets. The same analysis gives exact criteria for merging expert groups and evaluates fixed combinations of separate and common estimates (Theorems 1 and 2 ), with extensions to cross-layer sharing and general estimation geometry.
When local improvements survive to the output. Under a common fixed linear propagation map, exact expected squared-risk comparisons yield a strict local-to-final reversal (Proposition 47 ). For nontrivial fixed groupwise averaging and centered estimation noise with equal positive variance in every direction, we characterize the propagation maps that preserve every strict local improvement (Section N.4). The condition requires equal positive scaling of squared lengths on the coefficient differences removed by sharing. For general routed computations, we derive conditional error bounds accounting for distribution shifts, changes in expert selection, and errors in surrounding components (§ 5 ).
Separating predictive success from expert-level operations. We construct a model with correct predictions and correct expert labels whose output remains unchanged when a different expert is forced (Proposition 5). We then analyze the state information and intervention coverage needed to evaluate operation-level effects, distinguishing individual expert calls from actions spanning several layers (§ 6 ).
Additionally, we conduct experiments to reproduce the local risk comparisons and realize a local-to-final reversal. In low-data symbolic tasks, trained Transformers show one-step gains when given correct preceding states, while final-output gains remain statistically unresolved when the models use their own predictions (Figure 1 ). Separate intervention and held-out-transition tests examine expert-level attribution and generalization beyond observed transitions. These results provide a basis for evaluating sharing patterns through both local estimation and final-output performance, while identifying the evidence needed to interpret individual experts as executable operations.
Figure 1. Local and final gains in trained Transformers. Positive differences mean lower error for tied experts. (a) At 64 traces, one-step gains are supported, while the final-output difference is not statistically resolved. (b) All forty tying contrasts at 64, 96, 128, and 192 traces. Bars retain the reported simultaneous intervals of the full 240-endpoint follow-up family (N = 32768 N = 32768 N = 32768 , 480 tails, δ = 0.05 \delta= 0.05 δ = 0.05 , radius = 2.3657 = 2.3657 = 2.3657 )
Related Work
We introduce prior work in expert sharing, statistical pooling, and causal abstraction discussing how local sharing gains change under composition, routing, and intervention-based evaluation.
Expert reuse and routed architectures. Classical MoE studies learned gating over specialized predictors [16 , 19 ] ; sparse Transformers scale conditional computation [34 , 21 , 8 , 5 ] . Recent systems share experts across adjacent layers, global pools, or depth groups [37 , 14 , 2 , 17 , 32 ] . Expert-choice, soft dispatch, and sequential expert calls use different execution mechanisms [44 , 31 , 42 ] . Our event-level comparison retains their functional costs; Sections B and I give the architecture representation and cost dictionary.
Statistical sharing and shrinkage. Multi-task learning explains gains from shared structure and negative transfer from misspecified relatedness [38 , 39 , 6 , 15 ] . MoE theory studies generalization, robustness, expressivity, and task-specialized routing [3 , 43 , 30 , 41 , 28 ] . Our local formulas and known-noise Gaussian plug-in guarantee specialize classical shrinkage [1 , 36 , 35 ] . The fixed-rule audit uses unbiased risk estimation, and additive transfer uses the standard Wasserstein coupling argument [33 ] .
Reasoning and causal abstraction. Chain-of-thought theory studies serial computation through intermediate states [27 , 23 ] ; latent-module work formalizes reusable components in reasoning traces [20 ] . Causal abstraction distinguishes behavioral agreement from intervention agreement [9 , 10 ] . Applied to route actions, this requires sufficient pre-action states and distinguishable forced effects. We also analyze randomized action audits via inverse-propensity estimation [12 ] , with a matching lower-bound dependence on per-action coverage.
Routed Roles and Event-Lifted Risk
We distinguish call environments from expert blocks and measure their complete-event effects. This makes tied and separate libraries comparable.
Roles, experts, and the reference. Consider one prediction request: the current state is 3 and the next operation is multiplication by two, so the target next state is 6. In the illustrative two-layer Transformer of Appendix § 14 , both layers process this same request, rather than applying the task operation twice. At the chosen prediction position, operation i = 2 i = 2 i = 2 is labelled by call type k = i + 1 = 3 k = i + 1 = 3 k = i + 1 = 3 , giving two roles ( 1 , 3 ) (1, 3) ( 1 , 3 ) and ( 2 , 3 ) (2, 3) ( 2 , 3 ) .
A role a a a in a finite set J \mathcal{J} J specifies a controlled call environment and its target behavior. Different input numbers can instantiate the same role. The two layers define distinct roles here because their input sources and surrounding computation differ. An expert library E θ = { E θ , e : X e → Y e } e = 1 K \mathcal{E}_\theta= \{E_{\theta,e} : \mathcal{X}_e \to\mathcal{Y}_e\}_{e=1}^{K} E θ = { E θ , e : X e → Y e } e = 1 K contains parameterized functions indexed by e e e . Separate fitting uses two independent parameter blocks for these roles, and reuse assigns the same block. The reused block may receive different inputs across roles. Its outputs and the roles’ target updates may also differ.
In general, a fixed assignment α m : J → [ K m ] \alpha_m : \mathcal{J} \to[K_m] α m : J → [ K m ] identifies the expert serving each role in comparator m ∈ { r e u s e , s e p } m \in\{\mathrm{reuse},\mathrm{sep}\} m ∈ { reuse , sep } , with [ K m ] = { 1 , … , K m } [K_m] = \{1,\ldots,K_m\} [ K m ] = { 1 , … , K m } . Reuse groups J e = { a : α r e u s e ( a ) = e } \mathcal{J}_e = \{a : \alpha_{\mathrm{reuse}}(a) = e\} J e = { a : α reuse ( a ) = e } share functions through compatible interfaces or fixed adapters. The separate model is a function per role. In this role-assignment comparison, expert sharing is the common-function use represented by the reuse comparator. The assignment specifies who should serve a role. The router decides who actually does, without redefining the role.
The reference is the declared target computation, not the fitted separate model. In the example, knowing the target next state is 6 does not determine a layer’s target residual. A local target f a ⋆ f_a^\star f a ⋆ is specified separately to map hidden vectors to residuals, whereas the symbolic task operator F i F_i F i maps task states to task states.
Complete events and aligned instructions. An event is a complete computational block, not necessarily a token or reasoning step. For j = 0 , … , H − 1 j = 0,\ldots,H-1 j = 0 , … , H − 1 , states X j , X ^ j m ∈ S j X_j,\widehat{X}_j^m \in\mathcal{S}_j X j , X j m ∈ S j retain everything needed for execution, including prefixes, hidden-state arrays, and caches. A canonical program Γ j ∈ G j ⋆ \Gamma_j \in\mathcal{G}_j^\star Γ j ∈ G j ⋆ specifies roles, call order, input selection, and combination. A fixed implementation map ι j m : G j ⋆ → G j m \iota_j^m : \mathcal{G}_j^\star\to\mathcal{G}_j^m ι j m : G j ⋆ → G j m translates it into model-specific instructions. Conditional on fitted parameters and external randomness,
X j + 1 = Φ j ⋆ ( X j ; Γ j ) , X ^ j + 1 m = Φ j m ( X ^ j m ; Γ ^ j m ) , Γ ^ j m = R j m ( H j m ) . (1) X_{j+1}=\Phi^\star_j(X_j;\Gamma_j), \qquad\widehat{X}^{m}_{j+1}=\Phi^m_j(\widehat{X}^{m}_j;\widehat{\Gamma}^{m}_j), \qquad\widehat{\Gamma}^{m}_j=\mathcal{R}^{m}_j(\mathcal{H}^{m}_j). \tag*{(1)} X j + 1 = Φ j ⋆ ( X j ; Γ j ) , X j + 1 m = Φ j m ( X j m ; Γ j m ) , Γ j m = R j m ( H j m ) . (1)
The maps Φ j ⋆ \Phi^\star_j Φ j ⋆ and Φ j m \Phi^m_j Φ j m return complete post-event states in S j + 1 \mathcal{S}_{j+1} S j + 1 , including shared paths and aggregation. The router uses available history H j m \mathcal{H}^{m}_j H j m to choose the actual plan Γ ^ j m \widehat{\Gamma}^{m}_j Γ j m , which may differ from the aligned plan ι j m ( Γ j ) \iota^m_j(\Gamma_j) ι j m ( Γ j ) . Decisions use only information available when made. Adaptive internal decisions must be separate events for fixed-sequence replacement, or the whole policy must be one event lift. Section B.1 gives the complete history and randomness interface.
Role risk in its call environment. Role a a a has probe-record space Z a \mathcal{Z}_a Z a , distribution P a P_a P a , and an event lift
T a [ f ] : Z a ⟶ S , (2) \mathcal{T}_a[f]:\mathcal{Z}_a\longrightarrow\mathcal{S}, \tag*{(2)} T a [ f ] : Z a ⟶ S , (2)
which inserts expert function f f f into a specified environment and returns the complete post-event state. For example, T a [ f ] ( h , u , c ) = h + c f ( u ) \mathcal{T}_a[f](h,u,c)=h+cf(u) T a [ f ] ( h , u , c ) = h + c f ( u ) is a residual update with state h h h , expert input u u u , and scalar gain c c c . Dimensions agree after any fixed adapter. Let T a ⋆ \mathcal{T}^{\star}_a T a ⋆ be the target-call counterpart. For a state metric d d d ,
ℓ a ( f ; z ) = d ( T a [ f ] ( z ) , T a ⋆ ( z ) ) , R a ( f ) = E Z ∼ P a ℓ a ( f ; Z ) . (3) \ell_a(f;z)=d(\mathcal{T}_a[f](z),\mathcal{T}^{\star}_a(z)), \qquad R_a(f)=\mathbb{E}_{Z\sim P_a}\ell_a(f;Z). \tag*{(3)} ℓ a ( f ; z ) = d ( T a [ f ] ( z ) , T a ⋆ ( z )) , R a ( f ) = E Z ∼ P a ℓ a ( f ; Z ) . (3)
Both arguments are complete states. The worked example inserts one residual into the full array. The bounded-loss theorem additionally requires ℓ a ∈ [ 0 , 1 ] \ell_a\in[0,1] ℓ a ∈ [ 0 , 1 ] and probe laws, lifts, targets, classes, and permitted expert inputs fixed independently of fitting data, or a valid joint uniform bound.
The Local Sharing Frontier
This section mainly discuss that, with call environments fixed, when should roles share a function. The linear model prices pooling and merges by removable variance versus sharing bias. Beyond that, we compare certificates rather than exact risks.
An exact linear-operator frontier
In a specified linear estimation problem, consider M ≥ 1 M\ge1 M ≥ 1 equally weighted roles with target operators A a ∈ R p × d A_a\in\mathbb{R}^{p\times d} A a ∈ R p × d and responses Y a = X a A a T + E a Y_a=X_aA_a^{\mathsf{T}}+E_a Y a = X a A a T + E a . Each fixed design X a ∈ R n × d X_a\in\mathbb{R}^{n\times d} X a ∈ R n × d stacks the training inputs and satisfies n ≥ d n\ge d n ≥ d , X a T X a = n I d X_a^{\mathsf{T}}X_a=nI_d X a T X a = n I d , which are not the event states X j X_j X j . Designs are held constant as fitting noise varies. Orthogonal columns are a model assumption. All noise rows are mutually independent across observations and roles, centered, and have covariance σ 2 I p \sigma^2I_p σ 2 I p . Gaussianity is not required. Define
A ‾ = 1 M ∑ a = 1 M A a , B = 1 M ∑ a = 1 M ∥ A a − A ‾ ∥ F 2 . (4) \overline{A}=\frac{1}{M}\sum_{a=1}^{M}A_a,\qquad B=\frac{1}{M}\sum_{a=1}^{M}\lVert A_a-\overline{A}\rVert_F^2. \tag*{(4)} A = M 1 a = 1 ∑ M A a , B = M 1 a = 1 ∑ M ∥ A a − A ∥ F 2 . (4)
Thus B B B measures dispersion among the true role targets. Ordinary least squares (OLS) gives A ^ a = Y a T X a / n \widehat{A}_a=Y_a^{\mathsf{T}}X_a/n A a = Y a T X a / n and A ^ p o o l = M − 1 ∑ a A ^ a \widehat{A}_{\mathrm{pool}}=M^{-1}\sum_a\widehat{A}_a A pool = M − 1 ∑ a A a . For a fixed coefficient λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , partial pooling uses
A ^ a ( λ ) = ( 1 − λ ) A ^ a + λ A ^ p o o l . (5) \widehat{A}^{(\lambda)}_a=(1-\lambda)\widehat{A}_a+\lambda\widehat{A}_{\mathrm{pool}}. \tag*{(5)} A a ( λ ) = ( 1 − λ ) A a + λ A pool . (5)
Separate fitting and full sharing with λ = 0 \lambda=0 λ = 0 and 1 1 1 , and intermediate matrices generally remain role-specific. An independent input has second moment E [ x x T ] = I d \mathbb{E}[xx^{\mathsf{T}}]=I_d E [ x x T ] = I d . The excess prediction risk is
R λ = 1 M ∑ a = 1 M E E , x ∥ ( A ^ a ( λ ) − A a ) x ∥ 2 2 . (6) \mathcal{R}_{\lambda}=\frac{1}{M}\sum_{a=1}^{M}\mathbb{E}_{\mathcal{E},x}\left\lVert\left(\widehat{A}^{(\lambda)}_a-A_a\right)x\right\rVert_2^2. \tag*{(6)} R λ = M 1 a = 1 ∑ M E E , x ( A a ( λ ) − A a ) x 2 2 . (6)
It averages fitting noise and test inputs relative to noiseless targets, excluding fresh label noise. Only under this isotropic test geometry does conditional prediction error equal squared Frobenius parameter error. This specified squared loss is distinct from the bounded metric loss in Equation (3) .
Theorem 1 (Exact sharing and partial-pooling risks). Let D = p d D=pd D = p d , V = D σ 2 / n V=D\sigma^2/n V = D σ 2 / n , and C = V ( 1 − 1 / M ) C=V(1-1/M) C = V ( 1 − 1/ M ) . For the excess risk in Equation (6) ,
R s e p = V , (7) \mathcal{R}_{\mathrm{sep}}=V, \tag*{(7)} R sep = V , (7) R s h a r e = B + V M , (8) \mathcal{R}_{\mathrm{share}}=B+\frac{V}{M}, \tag*{(8)} R share = B + M V , (8) R λ = V − 2 C λ + ( B + C ) λ 2 . (9) \mathcal{R}_{\lambda}=V-2C\lambda+(B+C)\lambda^2. \tag*{(9)} R λ = V − 2 C λ + ( B + C ) λ 2 . (9) Consequently, full reuse has smaller expected excess prediction risk than separate estimation if and only if
B < C = p d σ 2 n ( 1 − 1 M ) . (10) B<C=\frac{pd\sigma^2}{n}\left(1-\frac{1}{M}\right). \tag*{(10)} B < C = n p d σ 2 ( 1 − M 1 ) . (10) If B + C > 0 B+C>0 B + C > 0 , the optimal fixed coefficient is λ ⋆ = C / ( B + C ) \lambda^\star=C/(B+C) λ ⋆ = C / ( B + C ) . The minimum risk, endpoint comparisons, and degenerate cases are given in Section D.1.
Here D D D counts coefficients, V V V is separate estimation error, and C = V − V / M C=V-V/M C = V − V / M is the removable variance. Full pooling pays bias B B B while averaging independent fitting errors to variance V / M V/M V / M . The oracle coefficient depends on unknown B B B . Substituting a same-data estimate of λ \lambda λ into the fixed-rule identity is invalid. The classical guarantee in Section D.6 separately requires Gaussian noise, known q = σ 2 / n > 0 q=\sigma^2/n>0 q = σ 2 / n > 0 , and D ( M − 1 ) ≥ 4 D(M-1)\ge4 D ( M − 1 ) ≥ 4 .
For allocation, fix a partition Π \Pi Π into K Π K_\Pi K Π disjoint nonempty groups before observing fitting noise. Group G G G has target mean A ˉ G = ∣ G ∣ − 1 ∑ a ∈ G A a \bar{A}_G=|G|^{-1}\sum_{a\in G}A_a A ˉ G = ∣ G ∣ − 1 ∑ a ∈ G A a and uses the average of its role-wise OLS estimates.
Theorem 2 (Exact partition and merge frontier). Under the assumptions of Theorem 1 , the expected role-averaged excess prediction risk of partition Π \Pi Π is
R ( Π ) = 1 M [ ∑ G ∈ Π ∑ a ∈ G ∥ A a − A ˉ G ∥ F 2 + K Π V ] . (11) \mathcal{R}(\Pi)=\frac{1}{M}\left[\sum_{G\in\Pi}\sum_{a\in G}\lVert A_a-\bar{A}_G\rVert_F^2+K_\Pi V\right]. \tag*{(11)} R ( Π ) = M 1 [ G ∈ Π ∑ a ∈ G ∑ ∥ A a − A ˉ G ∥ F 2 + K Π V ] . (11) If two groups G G G and H H H are merged, the exact risk change is
R ( Π G ∪ H ) − R ( Π ) = 1 M [ ∣ G ∣ ∣ H ∣ ∣ G ∣ + ∣ H ∣ ∥ A ˉ G − A ˉ H ∥ F 2 − V ] . (12) \mathcal{R}(\Pi_{G\cup H})-\mathcal{R}(\Pi)=\frac{1}{M}\left[\frac{|G||H|}{|G|+|H|}\lVert\bar{A}_G-\bar{A}_H\rVert_F^2-V\right]. \tag*{(12)} R ( Π G ∪ H ) − R ( Π ) = M 1 [ ∣ G ∣ + ∣ H ∣ ∣ G ∣∣ H ∣ ∥ A ˉ G − A ˉ H ∥ F 2 − V ] . (12) Thus a merge improves expected risk if and only if its between-centroid functional separation is below the variance V V V saved by removing one independently estimated expert. Optimizing the number and composition of experts is therefore exactly a function-space clustering objective with penalty V V V per fitted expert.
The variance charge saved by a merge is V V V before division by M M M , hence V / M V/M V / M in average risk. Cross-depth patterns and general test geometry are treated in Appendix § 20 and Theorem 39 in Appendix § 25 .
A distribution-free comparison
We now retain the sharing-versus-estimation comparison as a high-probability bound, under explicit sampling and loss conditions. For fixed candidate class F e \mathcal{F}_e F e and positive evaluation weights π e , a \pi_{e,a} π e , a summing to one on J e \mathcal{J}_e J e , define
R s e p , e ∗ = ∑ a ∈ J e π e , a inf f ∈ F e R a ( f ) , (13) R^\ast_{\mathrm{sep},e}=\sum_{a\in\mathcal{J}_e}\pi_{e,a}\inf_{f\in\mathcal{F}_e}R_a(f), \tag*{(13)} R sep , e ∗ = a ∈ J e ∑ π e , a f ∈ F e inf R a ( f ) , (13)
R s h a r e , e ∗ = inf f ∈ F e ∑ a ∈ J e π e , a R a ( f ) , (14) R^\ast_{\mathrm{share},e}=\inf_{f\in\mathcal{F}_e}\sum_{a\in\mathcal{J}_e}\pi_{e,a}R_a(f), \tag*{(14)} R share , e ∗ = f ∈ F e inf a ∈ J e ∑ π e , a R a ( f ) , (14)
B e r e u s e = R s h a r e , e ∗ − R s e p , e ∗ ≥ 0. (15) B^{\mathrm{reuse}}_e=R^\ast_{\mathrm{share},e}-R^\ast_{\mathrm{sep},e}\ge0. \tag*{(15)} B e reuse = R share , e ∗ − R sep , e ∗ ≥ 0. (15)
The first oracle chooses one function per role, and the second must use a common function, incurring reuse bias. With n a n_a n a mutually independent observations per role, weighted pooling has effective sample size
N e e f f = ( ∑ a ∈ J e π e , a 2 n a ) − 1 . (16) N^{\mathrm{eff}}_e=\left(\sum_{a\in\mathcal{J}_e}\frac{\pi_{e,a}^2}{n_a}\right)^{-1}. \tag*{(16)} N e eff = ( a ∈ J e ∑ n a π e , a 2 ) − 1 . (16)
which is M n M n M n for balanced roles. With nonempty finite classes, losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] , fixed objects independent of fitting data, and mutually independent role samples, Theorem 8 (Appendix § 15 .§ 2 ) gives shared/separate ERM bounds R s e p , e ∗ + B e r e u s e + 2 ϵ s h , e R^{*}_{\mathrm{sep},e}+B^{\mathrm{reuse}}_{e}+2\epsilon_{\mathrm{sh},e} R sep , e ∗ + B e reuse + 2 ϵ sh , e and R s e p , e ∗ + 2 ϵ s e p , e R^{*}_{\mathrm{sep},e}+2\epsilon_{\mathrm{sep},e} R sep , e ∗ + 2 ϵ sep , e on one 1 − δ 1-\delta 1 − δ fitting-data event. The shared empirical-risk and weighted role-wise deviations use ϵ s h , e ( δ ) = log ( 4 ∣ F e ∣ / δ ) 2 N e f f \epsilon_{\mathrm{sh},e}(\delta)=\sqrt{\frac{\log(4|\mathcal{F}_{e}|/\delta)}{2N_{\mathrm{eff}}}} ϵ sh , e ( δ ) = 2 N eff l o g ( 4∣ F e ∣/ δ ) and ϵ s e p , e ( δ ) = ∑ a ∈ J e π e , a log ( 4 ∣ J e ∣ ∣ F e ∣ / δ ) 2 n a \epsilon_{\mathrm{sep},e}(\delta)=\sum_{a\in\mathcal{J}_{e}}\pi_{e,a}\sqrt{\frac{\log(4|\mathcal{J}_{e}||\mathcal{F}_{e}|/\delta)}{2n_{a}}} ϵ sep , e ( δ ) = ∑ a ∈ J e π e , a 2 n a l o g ( 4∣ J e ∣∣ F e ∣/ δ ) , respectively (Equations (43) and (44) ), with ∣ F e ∣ |\mathcal{F}_{e}| ∣ F e ∣ counting functions. Variance interpretations require Proposition 7’s class conditions (Appendix § 15 ). Approximate ERM and mismatched training weights retain Proposition 41’s terms (Appendix § 25 ). Infinite classes need uniform bounds, and Equation (52) treats independent traces.
From Local Prediction Error to Final Outputs
In this section, we discuss the main argument that lower local prediction error does not necessarily indicate a better final output. Common fixed linear propagation gives an exact terminal frontier and a reversal. The local squared-risk model in § 4 isolates estimation, and a common fixed propagation map then exposes the effect of composition. The metric-error analysis below tracks the conditions needed for a general routed program. The trained-model study in § 7 measures teacher-forced one-step and free-running final-state errors.
Exact terminal geometry and a reversal
The terminal map determines which local error directions and correlations matter. Let θ \theta θ stack the true role coefficients and Y = θ + ζ Y=\theta+\zeta Y = θ + ζ their estimates, with E ζ = 0 \mathbb{E}\zeta=0 E ζ = 0 and finite covariance Σ \Sigma Σ . For every compared rule , a common matrix G G G , fixed independently of ζ \zeta ζ , maps coefficient error to terminal error. Fix a pooling matrix P P P independently of ζ \zeta ζ , and set Q = I − P Q=I-P Q = I − P and L λ = I − λ Q L_{\lambda}=I-\lambda Q L λ = I − λ Q , 0 ≤ λ ≤ 1 0\le\lambda\le1 0 ≤ λ ≤ 1 . Thus L 0 Y = Y L_{0}Y=Y L 0 Y = Y is separate fitting and L 1 Y = P Y L_{1}Y=PY L 1 Y = P Y is full pooling. For R H ( λ ) = E ∥ G ( L λ Y − θ ) ∥ 2 2 R_{H}(\lambda)=\mathbb{E}\lVert G(L_{\lambda}Y-\theta)\rVert_{2}^{2} R H ( λ ) = E ∥ G ( L λ Y − θ ) ∥ 2 2 , Theorem 46 in Appendix § 26 gives
R H ( λ ) = R H ( 0 ) − 2 a H λ + b H λ 2 , a H = tr ( G T G Q Σ ) , b H = ∥ G Q θ ∥ 2 2 + tr ( G T G Q Σ Q T ) ≥ 0. (17) \begin{aligned}
R_{H}(\lambda)&=R_{H}(0)-2a_{H}\lambda+b_{H}\lambda^{2},\\
a_{H}&=\operatorname{tr}(G^{\mathsf{T}}GQ\Sigma),\\
b_{H}&=\lVert GQ\theta\rVert_{2}^{2}+\operatorname{tr}(G^{\mathsf{T}}GQ\Sigma Q^{\mathsf{T}})\ge0. \tag*{(17)}
\end{aligned} R H ( λ ) a H b H = R H ( 0 ) − 2 a H λ + b H λ 2 , = tr ( G T GQ Σ ) , = ∥ GQθ ∥ 2 2 + tr ( G T GQ Σ Q T ) ≥ 0. (17)
The geometry G T G G^{\mathsf{T}}G G T G weights coefficient errors by their endpoint effects. The coefficient b H b_{H} b H contains squared bias and noise. a H = E ⟨ G ζ , G Q ζ ⟩ a_{H}=\mathbb{E}\langle G\zeta,GQ\zeta\rangle a H = E ⟨ Gζ , GQζ ⟩ measures alignment between original noise and the removed component. It can be negative when pooling destroys beneficial cancellation. Full pooling therefore improves actual expected terminal risk exactly when 2 a H > b H 2a_{H}>b_{H} 2 a H > b H .
A local gain can reverse. In Proposition 47 (Appendix § 26 ), three scalar roles have independent centered fitting errors of variance one, target θ = ( 1 / 2 , − 1 , 1 / 2 ) T \theta=(1/2,-1,1/2)^{\mathsf{T}} θ = ( 1/2 , − 1 , 1/2 ) T , and propagation G = ( 1 , − 2 , 1 ) G=(1,-2,1) G = ( 1 , − 2 , 1 ) . Uniform full pooling reduces local average risk from 1 1 1 to 5 / 6 5/6 5/6 , but raises terminal risk from 6 6 6 to 9 9 9 . The pooled component is annihilated by G 1 = 0 G1=0 G 1 = 0 , leaving squared terminal bias ( G θ ) 2 = 9 (G\theta)^2=9 ( Gθ ) 2 = 9 . Partial pooling can still help: the same example has R H ( λ ) = 6 − 12 λ + 15 λ 2 R_{H}(\lambda)=6-12\lambda+15\lambda^{2} R H ( λ ) = 6 − 12 λ + 15 λ 2 .
Here G G G comes from the three events in Equation (233) , starting at S 0 = 0 S_{0}=0 S 0 = 0 :
S 1 = S 0 + θ 1 , S 2 = − 1 2 S 1 + θ 2 , S 3 = − 2 S 2 + θ 3 . S_{1}=S_{0}+\theta_{1},\qquad S_{2}=-\frac{1}{2}S_{1}+\theta_{2},\qquad S_{3}=-2S_{2}+\theta_{3}. S 1 = S 0 + θ 1 , S 2 = − 2 1 S 1 + θ 2 , S 3 = − 2 S 2 + θ 3 .
Each fitted model uses this same recurrence with its fitted coefficients. Expanding it gives the weights ( 1 , − 2 , 1 ) (1,-2,1) ( 1 , − 2 , 1 ) , so the common propagation is fixed by the computation. Equal pooled coefficients cancel through G 1 = 0 G1=0 G 1 = 0 , while the target satisfies G θ = 3 G\theta=3 Gθ = 3 .
Repeated calls can reuse correlated fitted errors despite independent training data (Propositions 13 and 43; Appendices § 17 , § 25 ). Appendix § 26 .§ 4 ’s uniform-inheritance result requires orthogonal projection and isotropic noise. A full-law L 2 L^{2} L 2 remainder bound yields a same-law true-risk interval (Proposition 49; Appendix § 26 .§ 3 ). Joint Taylor and conditional-moment results retain region, moment, and escape assumptions (Appendix § 27 ). Support replay recomputes gains, and recorded-gain replay pays an extra discrepancy (Appendix § 28 ).
The uniform-inheritance statement concerns a restricted case: a fixed Euclidean orthogonal pooling projection P P P , Q = I − P Q=I-P Q = I − P with positive rank, centered noise with covariance σ I \sigma I σ I , σ > 0 \sigma>0 σ > 0 , and the same fixed G G G for both estimates. In that case, Section N.4 shows that strict local improvement from full pooling implies strict terminal improvement for every target θ \theta θ exactly when Q G T G Q = c Q QG^{\mathsf{T}}GQ=cQ Q G T GQ = c Q for some c > 0 c>0 c > 0 . Thus G G G scales squared lengths by the same positive factor on the discarded contrast space. This all-target condition is not necessary for one particular target to benefit.
Conditional certificates for general routed programs
We now bound aligned implementation and routing errors without an exact common linear map. Use S j = X j S_j=X_j S j = X j , S ^ j m = X ^ j m \widehat{S}^{m}_j=\widehat{X}^{m}_j S j m = X j m from Equation (1) , now represented in a common metric space ( S , d ) (\mathcal{S},d) ( S , d ) . For H ≥ 1 H\geq1 H ≥ 1 events write the reference dynamics as
S j + 1 = T j , Γ j ( S j ) , (18) S_{j+1}=T_{j,\Gamma_j}(S_j), \tag*{(18)} S j + 1 = T j , Γ j ( S j ) , (18)
where T j , γ = Φ ⋅ ⋆ ( ⋅ ; γ ) T_{j,\gamma}=\Phi^{\star}_{\cdot}(\mathord{\cdot};\gamma) T j , γ = Φ ⋅ ⋆ ( ⋅ ; γ ) and T ^ j , γ m = Φ ⋅ m ( ⋅ ; γ ) \widehat{T}^{m}_{j,\gamma}=\Phi^{m}_{\cdot}(\mathord{\cdot};\gamma) T j , γ m = Φ ⋅ m ( ⋅ ; γ ) use their respective program spaces. Each reference fixed-program map has a deterministic L j L_j L j -Lipschitz envelope throughout the compared region, uniformly over admissible programs. This does not assume a hard router is Lipschitz. Let A j = ∏ r = j + 1 H − 1 L r A_j=\prod_{r=j+1}^{H-1}L_r A j = ∏ r = j + 1 H − 1 L r , with empty product one. At the same current model state , define
a j m = d ( T ^ j , ι j m ( Γ j ) m ( S ^ j m ) , T j , Γ j ( S ^ j m ) ) , (19) a_j^m=d\left(\widehat{T}^{m}_{j,\iota_j^m(\Gamma_j)}(\widehat{S}^{m}_j),T_{j,\Gamma_j}(\widehat{S}^{m}_j)\right), \tag*{(19)} a j m = d ( T j , ι j m ( Γ j ) m ( S j m ) , T j , Γ j ( S j m ) ) , (19)
r j m = d ( T ^ j , Γ ^ j m m ( S ^ j m ) , T ^ j , ι j m ( Γ j ) m ( S ^ j m ) ) . (20) r_j^m=d\left(\widehat{T}^{m}_{j,\widehat{\Gamma}^{m}_j}(\widehat{S}^{m}_j),\widehat{T}^{m}_{j,\iota_j^m(\Gamma_j)}(\widehat{S}^{m}_j)\right). \tag*{(20)} r j m = d ( T j , Γ j m m ( S j m ) , T j , ι j m ( Γ j ) m ( S j m ) ) . (20)
The first compares fitted and target implementations under aligned instructions. The second changes only the plan at that same state, avoiding a second charge for earlier state error. With e j m = d ( S ^ j m , S j ) e_j^m=d(\widehat{S}^{m}_j,S_j) e j m = d ( S j m , S j ) , the triangle inequality gives
e j + 1 m ≤ L j e j m + a j m + r j m . e_{j+1}^m\leq L_je_j^m+a_j^m+r_j^m. e j + 1 m ≤ L j e j m + a j m + r j m .
Evaluation expectations condition on fitted models and average fresh trajectories and routing randomness, writing as ρ j m = E e v a l r j m \rho_j^m=\mathbb{E}_{\mathrm{eval}}r_j^m ρ j m = E eval r j m . The random intended program remains inside the discrepancies. Role-probe risk controls rollout error only through a further bridge.
Assumption 3 (Common role-aggregation domination). The role distributions, event-lift losses, target hybrids, and expert classes are fixed independently of the expert-estimation sample. There are deterministic nonnegative weights ω j , a \omega_{j,a} ω j , a , also fixed independently of that sample and valid for both comparators, such that, conditional on the fitted models,
E e v a l a j m ≤ τ j m + ∑ a ∈ J ω j , a R a ( f ^ α m ( a ) m ) , τ j m ≥ 0. (21) \mathbb{E}_{\mathrm{eval}}a_j^m\leq\tau_j^m+\sum_{a\in\mathcal{J}}\omega_{j,a}R_a\left(\widehat{f}^{m}_{\alpha_m(a)}\right),\qquad\tau_j^m\geq0. \tag*{(21)} E eval a j m ≤ τ j m + a ∈ J ∑ ω j , a R a ( f α m ( a ) m ) , τ j m ≥ 0. (21) The weights are fixed domination envelopes, not realized gates. Dynamic routing still requires justified occurrence, sensitivity, transfer, and assembly control. Appendix § 14 .§ 4 gives occupancy/transfer sufficient conditions. Proposition 42 (Appendix M.4) charges additive support mismatch.
The role risks are measured on fixed probes. In the sufficient construction of Appendix § 14 .§ 4 , each κ j , q , a \kappa_{j,q,a} κ j , q , a for controlled occurrence q q q of role a a a bounds both occurrence mass and distribution shift. Summing these bounds yields the common weights ω j , a \omega_{j,a} ω j , a , rather than empirical call frequencies. The term τ j m \tau_j^m τ j m retains surrounding-assembly mismatch after target experts replace the controlled calls. In contrast, ρ j m \rho_j^m ρ j m averages the effect of changing only the plan at the same current fitted-model state. Within-event effects are accounted for by the declared lifts and weights. A j A_j A j propagates the resulting event error through later events.
For each role set w a = ∑ j A j ω j , a w_a=\sum_j A_j\omega_{j,a} w a = ∑ j A j ω j , a , within each reuse group retain w a > 0 w_a>0 w a > 0 , put W e = ∑ a ∈ J e w a W_e=\sum_{a\in\mathcal{J}_e}w_a W e = ∑ a ∈ J e w a , and set π e , a = w a / W e \pi_{e,a}=w_a/W_e π e , a = w a / W e . Only W e > 0 W_e>0 W e > 0 groups enter below. All oracles, reuse biases, radii, and empirical objectives are recomputed on this active support. The full convention, including zero-weight roles influencing training, is in Section E.2.
Theorem 4 (Plan-level terminal certificates). *Let initial states agree and let scalar terminal utility U \mathcal{U} U be L U L_{\mathcal{U}} L U -Lipschitz. Assume Assumption 3 and all conditions of Theorem 8 on every active group: fixed finite classes, losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] , fixed probe laws and performance weights independent of fitting data, at least one observation per active role, and mutually independent role observations. Fit matched performance-weighted ERMs from the same role samples. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) choose δ e > 0 \delta_e>0 δ e > 0 , ∑ e δ e ≤ δ , \sum_e\delta_e\leq\delta, ∑ e δ e ≤ δ ,
and use the radii of Equations (43) and (44) at δ e \delta_e δ e , with ϵ r e u s e , e = ϵ s h , e \epsilon_{\mathrm{reuse},e} = \epsilon_{\mathrm{sh},e} ϵ reuse , e = ϵ sh , e . With probability at least 1 − δ 1-\delta 1 − δ over fitting data, simultaneously for both models,
E e v a l ∣ U ( S ^ H m ) − U ( S H ) ∣ ≤ L U U m , \mathbb{E}_{\mathrm{eval}}\left|\mathcal{U}(\widehat{S}^{m}_{H})-\mathcal{U}(S_H)\right| \le L_{\mathcal{U}}U_m, E eval U ( S H m ) − U ( S H ) ≤ L U U m ,
U m = ∑ j A j ( τ j m + ρ j m ) + ∑ e w e [ R s e p , e ∗ + 1 { m = r e u s e } B e r e u s e + 2 ϵ m , e ( δ e ) ] . (22) U_m = \sum_j A_j\left(\tau_j^m+\rho_j^m\right)
+\sum_e w_e\left[R^*_{\mathrm{sep},e}+\mathbf{1}\{m=\mathrm{reuse}\}B^{\mathrm{reuse}}_e+2\epsilon_{m,e}(\delta_e)\right]. \tag*{(22)} U m = j ∑ A j ( τ j m + ρ j m ) + e ∑ w e [ R sep , e ∗ + 1 { m = reuse } B e reuse + 2 ϵ m , e ( δ e ) ] . (22)
U m U_m U m is a state-error bound, not the utility U \mathcal{U} U . The expanded bounds and probability accounting are in Section E.2. Common oracles cancel in the state-certificate gap P R G H = U s e p − U r e u s e \mathrm{PRG}_H = U_{\mathrm{sep}}-U_{\mathrm{reuse}} PRG H = U sep − U reuse :
P R G H = ∑ e w e [ 2 { ϵ s e p , e ( δ e ) − ϵ s h , e ( δ e ) } − B e r e u s e ] − ∑ j A j [ ( τ j r e u s e − τ j s e p ) + ( ρ j r e u s e − ρ j s e p ) ] . (23) \mathrm{PRG}_H = \sum_e w_e\left[2\{\epsilon_{\mathrm{sep},e}(\delta_e)-\epsilon_{\mathrm{sh},e}(\delta_e)\}-B^{\mathrm{reuse}}_e\right]
-\sum_j A_j\left[(\tau_j^{\mathrm{reuse}}-\tau_j^{\mathrm{sep}})+(\rho_j^{\mathrm{reuse}}-\rho_j^{\mathrm{sep}})\right]. \tag*{(23)} PRG H = e ∑ w e [ 2 { ϵ sep , e ( δ e ) − ϵ sh , e ( δ e )} − B e reuse ] − j ∑ A j [ ( τ j reuse − τ j sep ) + ( ρ j reuse − ρ j sep ) ] . (23)
Thus P R G H > 0 \mathrm{PRG}_H>0 PRG H > 0 orders these state-error bounds. Strict utility-bound ordering also requires L U > 0 L_{\mathcal{U}}>0 L U > 0 . It does not order unknown true risks or directly bound discrete answer accuracy. Model-specific weights prevent this compact cancellation. Approximate ERM, mismatched training weights, and zero-weight training influences retain their explicit algorithmic-deviation terms (Proposition 41 ). Architecture-specific route costs and margin feedback are in Sections I and J. Independent actual-performance evaluation is in Proposition 45 (Appendix § 25 .§ 7 ).
Why Route Labels Do Not Establish Causal Reuse
The preceding comparisons concern predictive risk and error bounds. In this section, we discuss whether behavior can be attributed to experts implementing executable operations. Correct expert labels do not always establish operations. We separate labels from intervention fidelity, sufficient state information, and audit coverage.
Let C C C be the complete executable context and S = Ψ ( C ) S=\Psi(C) S = Ψ ( C ) a macro state extracted by a fixed map. A route action A i A_i A i specifies a non-anticipating policy and bounded stopping time. Execution produces T i ( C ) \mathcal{T}_i(C) T i ( C ) and macro result T i ( C ) = Ψ ( T i ( C ) ) T_i(C)=\Psi(\mathcal{T}_i(C)) T i ( C ) = Ψ ( T i ( C )) . Deterministic statements use greedy execution or a common fixed realization of exogenous randomness. The full policy and stochastic formulations are in Sections F and L. For a candidate operator F i : S → S F_i:S\to S F i : S → S and fixed pre-action context law ν i \nu_i ν i , define
μ i = E C ∼ ν i d ( T i ( C ) , F i ( Ψ ( C ) ) ) . (24) \mu_i=\mathbb{E}_{C\sim\nu_i}d\left(T_i(C),F_i(\Psi(C))\right). \tag*{(24)} μ i = E C ∼ ν i d ( T i ( C ) , F i ( Ψ ( C )) ) . (24)
This is prediction error for an intervened effect using the pre-action state. Small defect also needs non-collapse between actions to support a reusable-operation claim. A single expert call and a multi-layer compound action are different attribution targets.
Proposition 5 (Observational alignment does not imply causal reuse ). For any finite family of target operators { F i } i = 1 K \{F_i\}_{i=1}^{K} { F i } i = 1 K with outputs in a common real vector space, there exists a residual routed model with exact task output and route-label accuracy one, while its output under forced expert index j j j is independent of j j j . Under a distribution over requested index I I I , state S S S , and forced index J J J , its forced-route fidelity to F J F_J F J is exactly
P ( F I ( S ) = F J ( S ) ) , (25) \mathbb{P}\left(F_I(S)=F_J(S)\right), \tag*{(25)} P ( F I ( S ) = F J ( S ) ) , (25) which equals 1 / K 1/K 1/ K when J J J is uniform and independent of ( I , S ) (I,S) ( I , S ) and the K K K operator outputs are pairwise distinct at every evaluated state.
The shared path computes F I ( S ) F_I(S) F I ( S ) and every expert contributes zero, while the natural router selects I I I perfectly. Forcing J J J replaces the requested operation I I I and pre-action prefix unchanged, so the output remains F I ( S ) F_I(S) F I ( S ) . The 1 / K 1/K 1/ K specialization requires both independent uniform forcing and pairwise distinct target outputs at every evaluated state.
A compressed state may also omit information needed to predict an action. Proposition 14 in Appendix § 18 gives a worst-case diameter obstruction, while Proposition 15 gives an average squared-loss conditional-variance identity. The worst pair and probability-weighted average treat rare aliases differently. The extractor, actions, operators, and alignment must be fixed independently of the audit, and off-policy fidelity needs further control to transfer to on-policy contexts.
Randomized action-wise audits require supported actions and known forcing propensities (Section K). Their population comparison concerns action-specific f A ( S ) f_A(S) f A ( S ) versus action-unaware f ( S ) f(S) f ( S ) . It equals a local sharing comparison only with matched inputs, classes, losses, and distributions (Section K.1). Physical tying alone does not remove action information encoded in the expert input. These causal checks remain separate from the paired terminal-performance audit.
Experiments: From Local Gains to Final Outputs
We test whether reducing local error improves the final output, first in controlled estimation, and then in trained modern Transformers architecture. The latter compose four operations on Z 31 \mathbb{Z}_{31} Z 31 using four-layer, width-256 causal models trained at horizons 4 and 8. We distinguish teacher-forced one-step error from free-running final-state error. Methods, full results, and evaluation scope are in Section G.
One-step gains need not yield detectable final-output gains. A training-data sweep compares the same trained model pairs on both endpoints. Figure 1 shows cross-depth tying versus layer-local experts for all ten seeds at all four budgets of the balanced follow-up. At 64 training traces, tying reduces teacher-forced step error by 3.49–7.72 percentage points, with a positive lower bound in every seed, so the local gains are non-trivial. However, final-output differences range only from −0.186 to +0.192 points and every interval crosses zero, which shows that no statistically meaningful output gain is observed. Both models remain near chance on the final endpoint in this low-data regime. This is a concrete example, align with our maths abstract construction, which shows that there exists the case when local improvement without a statistically meaningful final improvement . Other budgets exhibit larger final changes and seed-dependent directions.
Conclusion
Expert sharing must be assessed at three levels: local estimation, final-output performance, and intervention effects. Locally it trades estimation savings against role incompatibility. Common fixed linear propagation makes the terminal trade-off exact and permits local gains to reverse. General routed programs require explicit transfer, assembly, and routing control. Causal reuse additionally requires supported interventions and a sufficient pre-action state. The experiments make the local-to-final distinction concrete: a controlled program reverses the risk ordering, while trained models exhibit supported one-step gains without resolved final-output gains.
AI Use Statement
In this work, generative AI tools were used for language editing, code development, and the drafting of candidate mathematical derivations. The authors independently verified the final mathematical content, including assumptions, derivations, propositions, and proofs. The final proofs, citations, code, and experiment provenance were also manually checked or rewritten by the authors. The authors take full responsibility for the final content of this work, including the text, claims, mathematical results, code, and all other materials produced with the assistance of generative AI.
Ethics Statement
The reported experiments use generated symbolic data and locally trained Transformer models, rather than personal data or a deployed decision system. More efficient conditional computation can reduce resource use, but it can also lower the cost of deploying capable models. Future applications should report both efficiency and capability changes.
Reproducibility Statement
The assumptions and complete proofs are given in the corresponding theoretical appendices, with a consolidated reading guide in the appendix overview. Section G specifies the tasks, model architectures, optimization settings, frozen-feature estimators, interventions, and evaluation procedures. Figure captions and Sections G.6 and G.7 state the sampling units, comparison families, and uncertainty conventions. The supplemented materials preserves the experimental configurations and summaries together with scripts for regenerating the tables and figures.
Appendix overview and reading paths
The appendices separate proofs of the main claims from architectural specializations and evaluation protocols. Readers can use the questions below to choose a path.
Understand the problem settings and argument. Start with the worked Transformer example, then return to § 4 –§ 6 . Consult Section A to distinguish exact risk, a certificate, and an audit.
Check a main claim. Read Section D for local identities, Sections C and E for statistical and terminal bounds, Section N for terminal geometry, and Section F for causal obstructions. Use Section Q alongside the full statements.
Choose an extension or evaluation. Use H–P selectively for the relevant architecture, geometry, or intervention. Section G reports local-to-final experiments, intervention supplements, and evaluation scope. The paired audit in Appendix § 25 .§ 7 addresses actual performance.
Table 2.
Reading Guide: Symbols, Mathematical Terms, and Claim Types
This guide distinguishes risk comparisons, certificates, and audits, and records the paper’s notation. Definitions apply throughout; specialized symbols are also introduced next to their equations, and a reused letter has the scope specified in its section.
Three types of guarantees
A risk is an expected loss on a specified data distribution. A population oracle is the best function in a declared class when that distribution is known; an oracle tuning coefficient minimizes an expected risk using unknown population quantities. An empirical risk replaces the expectation by a sample average, and an empirical risk minimizer (ERM) minimizes that average. An excess prediction risk here measures error relative to the noiseless target operator, excluding irreducible observation noise that would be common to the compared predictors.
An exact expected-risk comparison averages over the specified fitting randomness and is an equality or inequality about that model’s true risk. A high-probability certificate says that a bound is valid for all training samples except a set of probability at most δ \delta δ . A smaller certificate does not imply smaller actual error: if r 1 ≤ 10 r_1 \le10 r 1 ≤ 10 and r 2 ≤ 20 r_2 \le20 r 2 ≤ 20 , either true error can be smaller. A held-out audit estimates the risk difference of already fitted systems on new examples. Theorem 39 illustrates an exact expected-risk comparison; Theorem 4 gives a high-probability certificate; Proposition 45 gives a held-out audit.
Index and notation conventions
Unless stated otherwise, counts are positive integers, sets of roles and experts are finite, all maps and random variables are measurable, and all stated expectations exist. Probability arguments use standard Borel state and probe spaces (measurable spaces equivalent to Borel subsets of a complete separable metric space); finite sets, Euclidean vectors, and finite token sequences satisfy this condition. These routine conditions allow conditional distributions and conditional means to be interpreted consistently.
Table 1.
Table 1.
A hat marks an estimate, fitted object, empirical quantity, or state/route realized by a fitted model; Φ m n \Phi_m^n Φ m n and R m n \mathcal{R}_m^n R m n also belong to the fitted model despite having no hat; ⋆ \star ⋆ marks a reference or oracle, specified locally. arg min \operatorname*{arg\,min} arg min is the set of minimizing arguments; inf \inf inf is the greatest lower bound even when no minimizer is attained. Empty sums are zero and empty products are one. 1 { B } \mathbf{1}\{B\} 1 { B } equals one when event B B B occurs and zero otherwise. Law ( X ) \operatorname{Law}(X) Law ( X ) means the probability distribution of X X X , and “almost surely” means except on an event of probability zero.
Linear-model notation
For a matrix A A A , ∥ A ∥ F \lVert A\rVert_F ∥ A ∥ F denotes its Frobenius norm and ∥ A ∥ 2 \lVert A\rVert_2 ∥ A ∥ 2 its operator norm. The common-Hilbert-risk assumption fixes one shared geometry for all role risks. The design Gram matrix X T X X^\mathsf{T}X X T X and the propagation Gramian P j T P j P_j^\mathsf{T}P_j P j T P j describe different geometries.
Table 3.
Transfer and transport conventions
The 1-Wasserstein distance is
W 1 ( P , Q ; d ) = inf γ ∈ Π ( P , Q ) ∫ d ( x , y ) d γ ( x , y ) (26) W_1(P,Q;d) = \inf_{\gamma\in\Pi(P,Q)} \int d(x,y)\,\mathrm{d}\gamma(x,y) \tag*{(26)} W 1 ( P , Q ; d ) = γ ∈ Π ( P , Q ) inf ∫ d ( x , y ) d γ ( x , y ) (26)
where Π ( P , Q ) \Pi(P,Q) Π ( P , Q ) is the set of couplings. The finite first-moment condition is E d ( X , x 0 ) < ∞ \mathbb{E}d(X,x_0) < \infty E d ( X , x 0 ) < ∞ for one fixed point x 0 x_0 x 0 . The coupling notation Π ( P , Q ) \Pi(P,Q) Π ( P , Q ) is unrelated to a partition Π \Pi Π . In the discrete route transport of Equation (129) , costs are distances between expert outputs, not between their labels.
Intervention conventions
The propensity e a ( C ) e_a(C) e a ( C ) is the probability of actually sampling action a a a at context C C C , not generally the router’s softmax score under deterministic top-k k k selection. Overlap requires e a ( C ) ≥ β > 0 e_a(C) \ge\beta> 0 e a ( C ) ≥ β > 0 on every claimed region/action. Changing the intervention policy or target distribution changes the estimand. A Markov-sufficient macro state makes the transition law depend on the complete context only through that state and the chosen action; otherwise kernels must be defined on the complete context.
Computational vocabulary and comparison objects
A role groups specified call environments with a probe law and target; an event contains the calls and surrounding operations being compared. A reasoning step is a task-level operation, with no assumed one-to-one relation to a token, layer, call, or event. The independent statistical unit is the entire independently sampled record, often a complete trace rather than an individual call.
Support replay fixes selected indices but recomputes their gains at the replay’s own states. Frozen-gain replay also fixes recorded gains; natural execution recomputes its selections. Freezing a feature law fixes both the feature map and its evaluation input distribution.
An estimation-risk frontier averages fitting randomness; a terminal-risk frontier also includes the map carrying errors to the endpoint. An approximation frontier minimizes population discrepancy over a feasible class without charging estimation or optimization.
Technical terms used by the extensions
Function-subspace dimension, matrix rank, factor norms, and the norm of an increment B A − B 0 A 0 BA - B_{0}A_{0} B A − B 0 A 0 are distinct constraints. A fixed core is not optimized, whereas a free core is. A feature second moment is E [ φ φ T ] \mathbb{E}[\varphi\varphi^{\mathsf{T}}] E [ φ φ T ] , not a centered covariance. Conditioning on a selection cell requires recomputing conditional means and covariances. A full-law risk averages both non-escape and escape outcomes; derivative envelopes must hold throughout the specified region, not only at the expansion point.
Routed Computation Graphs, Roles, and Event Lifts
Overview. This appendix explains what is shared and how its effect is compared inside a complete computation. The worked example makes the roles, assignments, and event lifts concrete; the remaining sections give the precise interfaces and sufficient conditions connecting local probes to rollout errors. After the example, Sections B.1–B.2 define programs and role risks, Sections B.3–B.4 give the replacement and transfer arguments, and Section B.5 distinguishes the levels of reuse.
Worked example: two-layer expert reuse
One prediction through two layers. Consider the request “current state 3, next operation multiply by two.” The task in Section G uses F 2 ( s ) = 2 s m o d 31 F_{2}(s)=2s\ \mathrm{mod}\ 31 F 2 ( s ) = 2 s mod 31 , so the target next state is F 2 ( 3 ) = 6 F_{2}(3)=6 F 2 ( 3 ) = 6 . This is one operation within a supplied program. To follow the internal computation, use the illustrative two-layer causal Transformer: width 128, four attention heads, and four candidate SwiGLU experts of intermediate width 384 per layer. Both layers contribute to this one prediction; they do not perform two successive multiplications. Each receives the position’s hidden representation, not the integer 3 as a direct expert input. The primary trained comparators in Section G.1 instead have four layers and hidden width 256. The example uses declared reference functions and probes.
Two roles and their assigned experts. For this request, the controlled call type is k = i + 1 = 3 k=i+1=3 k = i + 1 = 3 , since the task operation index is i = 2 i=2 i = 2 . The chosen prediction position gives role ( 1 , 3 ) (1,3) ( 1 , 3 ) in the first layer and role ( 2 , 3 ) (2,3) ( 2 , 3 ) in the second. The separate model uses each layer’s own local expert 3—two independent parameter blocks, numbered 3 and 7 when flattened. The reuse model assigns shared block 3 to both roles. Its four-expert bank is shared across depths, while routers and normalization remain layer-local; the separate model stores two independent banks. A different input number can instantiate the same role.
This pair belongs to the full eight-role construction. Let k ∈ { 1 , 2 , 3 , 4 } k\in\{1,2,3,4\} k ∈ { 1 , 2 , 3 , 4 } name a controlled call type and ℓ ∈ { 1 , 2 } \ell\in\{1,2\} ℓ ∈ { 1 , 2 } its layer. At symbolic-task prediction positions, k = i + 1 k=i+1 k = i + 1 labels the requested operation i ∈ { 0 , 1 , 2 , 3 } i\in\{0,1,2,3\} i ∈ { 0 , 1 , 2 , 3 } . The full assignment is
J = { ( ℓ , k ) : ℓ ∈ { 1 , 2 } , k ∈ { 1 , 2 , 3 , 4 } } , K s e p = 8 , K r e u s e = 4 , \mathcal{J}=\{(\ell,k):\ell\in\{1,2\},\ k\in\{1,2,3,4\}\},\qquad K_{\mathrm{sep}}=8,\qquad K_{\mathrm{reuse}}=4, J = {( ℓ , k ) : ℓ ∈ { 1 , 2 } , k ∈ { 1 , 2 , 3 , 4 }} , K sep = 8 , K reuse = 4 ,
α s e p ( ℓ , k ) = 4 ( ℓ − 1 ) + k , α r e u s e ( ℓ , k ) = k , J k = { ( 1 , k ) , ( 2 , k ) } . \alpha_{\mathrm{sep}}(\ell,k)=4(\ell-1)+k,\qquad\alpha_{\mathrm{reuse}}(\ell,k)=k,\qquad\mathcal{J}_{k}=\{(1,k),(2,k)\}. α sep ( ℓ , k ) = 4 ( ℓ − 1 ) + k , α reuse ( ℓ , k ) = k , J k = {( 1 , k ) , ( 2 , k )} .
Table 14.
A call occurrence belongs to one role. Paper indices are one-based; code bank positions are k − 1 k-1 k − 1 . Separate indices 5–8 flatten a layer/index pair, not an eight-way router. At role (2, 3), the reuse assignment specifies block 3, but the natural router may select block 1. This remains role (2, 3), not (2, 1).
Reference updates and task outputs. The target next state 6 does not uniquely specify a 128-dimensional residual at either layer. For the local comparison, specify reference weights independently and let f a ⋆ f_a^{\star} f a ⋆ be the corresponding same-interface SwiGLU function. Both fitted models are evaluated against this declared target; the fitted separate model is not the reference. The local function returns a hidden-state residual, whereas F i : Z 31 → Z 31 F_i: \mathbb{Z}_{31} \to\mathbb{Z}_{31} F i : Z 31 → Z 31 returns a task state. These local reference weights are not asserted to form a complete network that outputs 6.
A residual difference you can calculate. For arithmetic only, suppose a valid controlled probe fixes the prefix and all other call contributions. The base row has first two coordinates ( 1 , 0 ) (1,0) ( 1 , 0 ) , and the expert output is multiplied by a fixed gain c = 1 / 2 c=1/2 c = 1/2 . Suppose the candidate residual begins ( 3 , 1 ) (3,1) ( 3 , 1 ) and the reference residual begins ( 2 , 1 ) (2,1) ( 2 , 1 ) , with all remaining output differences zero. The updated row then begins ( 1 , 0 ) + 1 2 ( 3 , 1 ) = ( 2.5 , 0.5 ) (1,0)+\frac{1}{2}(3,1)=(2.5,0.5) ( 1 , 0 ) + 2 1 ( 3 , 1 ) = ( 2.5 , 0.5 ) for the candidate, and ( 1 , 0 ) + 1 2 ( 2 , 1 ) = ( 2 , 0.5 ) (1,0)+\frac{1}{2}(2,1)=(2,0.5) ( 1 , 0 ) + 2 1 ( 2 , 1 ) = ( 2 , 0.5 ) for the reference. Under the complete-state metric below, this gives a single-probe distance of 0.5: the expert returns an update, and the comparison measures its effect after insertion into the surrounding state.
Only the first two coordinates of the 128-dimensional row are displayed; all other coordinate differences are zero. These illustrative values give a single-probe state distance of 0.5.
The complete-state representation. Each E θ , e : R 128 → R 128 E_{\theta,e}: \mathbb{R}^{128} \to\mathbb{R}^{128} E θ , e : R 128 → R 128 returns a residual, with parameters
E θ , e ( u ) = D e [ SiLU ( V e u ) ⊙ ( U e u ) ] , U e , V e ∈ R 384 × 128 , D e ∈ R 128 × 384 . E_{\theta,e}(u)=D_e[\operatorname{SiLU}(V_eu)\odot(U_eu)],\quad U_e,V_e\in\mathbb{R}^{384\times128},\quad D_e\in\mathbb{R}^{128\times384}. E θ , e ( u ) = D e [ SiLU ( V e u ) ⊙ ( U e u )] , U e , V e ∈ R 384 × 128 , D e ∈ R 128 × 384 .
For a fixed-length prefix without dropout, take the event to be one full FFN subblock with residual addition. Its pre-event state is X = ( p , H ) X=(p,H) X = ( p , H ) , where p p p is the prefix and H ∈ R n × 128 H\in\mathbb{R}^{n\times128} H ∈ R n × 128 is the complete pre-FFN array after attention. The expert input is u = LN ( H t , : ) u=\operatorname{LN}(H_{t,:}) u = LN ( H t , : ) at position t t t . Mask and layer boundary are fixed. This full-prefix implementation uses no KV-cache path. Attention, FFN, and readout can be separate events: the horizon H H H is not automatically the layer count. S j S_j S j contains the complete records at boundary j j j ; X j X_j X j and X ^ j m \widehat{X}^{m}_j X j m follow the target and fitted trajectories respectively.
A valid z ∈ Z a z\in\mathcal{Z}_a z ∈ Z a fixes p , H , t , u p,H,t,u p , H , t , u , a gain c c c , and the other call contributions. Let H b a s e ( z ) H_{\mathrm{base}}(z) H base ( z ) retain the residual path and all contributions except the controlled call. Insert t ( v ) \operatorname{Insert}_t(v) Insert t ( v ) is an n × 128 n\times128 n × 128 zero array with row t t t equal to v ⊤ v^\top v ⊤ . The numerical comparison above is an instance of
T a [ f ] ( z ) = ( p , H b a s e ( z ) + Insert t ( c f ( u ) ) ) , T_a[f](z)=(p,H_{\mathrm{base}}(z)+\operatorname{Insert}_t(cf(u))), T a [ f ] ( z ) = ( p , H base ( z ) + Insert t ( c f ( u ))) ,
T a ⋆ ( z ) = ( p , H b a s e ( z ) + Insert t ( c f a ⋆ ( u ) ) ) . T_a^{\star}(z)=(p,H_{\mathrm{base}}(z)+\operatorname{Insert}_t(cf_a^{\star}(u))). T a ⋆ ( z ) = ( p , H base ( z ) + Insert t ( c f a ⋆ ( u ))) .
For same-shaped states use
d ( ( p , H ) , ( p ′ , H ′ ) ) = 1 { p ≠ p ′ } + ∥ H − H ′ ∥ F . d((p,H),(p',H'))=\mathbf{1}\{p\ne p'\}+\lVert H-H'\rVert_F. d (( p , H ) , ( p ′ , H ′ )) = 1 { p = p ′ } + ∥ H − H ′ ∥ F .
Only one row differs on this probe, so ℓ a ( f ; z ) = ∣ c ∣ ∥ f ( u ) − f a ⋆ ( u ) ∥ 2 \ell_a(f;z)=|c|\lVert f(u)-f_a^{\star}(u)\rVert_2 ℓ a ( f ; z ) = ∣ c ∣ ∥ f ( u ) − f a ⋆ ( u ) ∥ 2 . Take P a P_a P a uniform on a valid probe bank independently fixed before expert fitting; R a ( f ) = E P a ℓ a ( f ; Z ) R_a(f)=\mathbb{E}_{P_a}\ell_a(f;Z) R a ( f ) = E P a ℓ a ( f ; Z ) averages these complete-state discrepancies.
Intended and actual execution. A canonical Γ j m ∈ G j ⋆ \Gamma_j^m\in\mathcal{G}_j^{\star} Γ j m ∈ G j ⋆ names roles, positions, gains or gain rules, and combination for all calls in the event. ν j m \nu_j^m ν j m substitutes the assigned model-specific blocks, producing an aligned plan in G j m \mathcal{G}_j^m G j m . The actual plan is Γ ^ j m = R j m ( Γ j m ) \widehat{\Gamma}_j^m=R_j^m(\Gamma_j^m) Γ j m = R j m ( Γ j m ) , using only visible prefix, activations, and already revealed randomness. The execution maps return complete event states. At one common current state, compare
Φ j m ( X ^ j m ; Γ ^ j m ) actual fitted execution, \Phi_j^m(\widehat{X}_j^m;\widehat{\Gamma}_j^m)\qquad\text{actual fitted execution,} Φ j m ( X j m ; Γ j m ) actual fitted execution,
Φ j m ( X ^ j m ; ι j m ( Γ j ) ) aligned fitted execution, \Phi_j^m(\widehat{X}_j^m;\iota_j^m(\Gamma_j))\qquad\text{aligned fitted execution,} Φ j m ( X j m ; ι j m ( Γ j )) aligned fitted execution,
Φ j ⋆ ( X ^ j m ; Γ j ) target at the model state. \Phi_j^{\star}(\widehat{X}_j^m;\Gamma_j)\qquad\text{target at the model state.} Φ j ⋆ ( X j m ; Γ j ) target at the model state.
The first two isolate routing cost; the last two isolate implementation error. The reference trajectory itself evaluates Φ j ⋆ \Phi_j^{\star} Φ j ⋆ at X j X_j X j , not at X ^ j m \widehat{X}_j^m X j m .
The eight illustrated roles cover controlled prediction-position calls, not all errors in a complete rollout; other calls need their own losses or justified assembly terms. The Frobenius metric is generally unbounded. Section B.2 specifies the fixed-probe conditions, and Section P.4 treats fixed-gain replay. See Section A for component terminology and return to § 3 for the general definitions.
Canonical programs and model-specific implementations
Let
E θ = { E θ , e : X e → Y e } e = 1 K (27) \mathcal{E}_{\theta}=\{E_{\theta,e}:\mathcal{X}_{e}\to\mathcal{Y}_{e}\}_{e=1}^{K} \tag*{(27)} E θ = { E θ , e : X e → Y e } e = 1 K (27)
be a library of parameter blocks. A canonical route program γ ∈ G j ⋆ \gamma\in\mathcal{G}^{\star}_{j} γ ∈ G j ⋆ is labelled by functional roles rather than by model-specific expert indices. It contains the controlled calls, their dispatched inputs, mixture weights, call order, shared paths, residual operations, aggregation, and—when needed—a finite non-anticipating policy for internal adaptive choices. Each model m m m has an implementation map
ι j m : G j ⋆ ⟶ G j m (28) \iota^{m}_{j}:\mathcal{G}^{\star}_{j}\longrightarrow\mathcal{G}^{m}_{j} \tag*{(28)} ι j m : G j ⋆ ⟶ G j m (28)
that replaces every canonical role occurrence by the expert index, tied parameter block, or subgraph used by that comparator. This map is part of the comparison protocol and aligns canonical roles with each model’s executable library.
Conditional on an executable plan and the relevant exogenous randomness, the event map
Φ j m : S j × G j m ⟶ S j + 1 (29) \Phi^{m}_{j}:\mathcal{S}_{j}\times\mathcal{G}^{m}_{j}\longrightarrow\mathcal{S}_{j+1} \tag*{(29)} Φ j m : S j × G j m ⟶ S j + 1 (29)
is deterministic. A non-anticipating scheduler produces the actually executed plan Γ ^ j m \widehat{\Gamma}^{m}_{j} Γ j m from information available before its corresponding routing decision. Future random variables may be represented by an exogenous seed in the probability space, but are not made observable to the router merely to obtain a deterministic representation.
Complete interfaces for Equation (1) . A state space S j \mathcal{S}_{j} S j is the set of admissible complete records immediately before event j j j . For a Transformer event, a record can include the prefix, the hidden-state array, attention caches, and any earlier values reused by residual connections. A state X j ∈ S j X_{j}\in\mathcal{S}_{j} X j ∈ S j is one such record, whereas X j + 1 ∈ S j + 1 X_{j+1}\in\mathcal{S}_{j+1} X j + 1 ∈ S j + 1 is the record after the event. The learned state X ^ j m \widehat{X}^{m}_{j} X j m is a separate trajectory value in the same declared comparison representation. The spaces X e , Y e \mathcal{X}_{e},\mathcal{Y}_{e} X e , Y e describe one expert’s interface and need not equal the event-state spaces. If models use different internal representations, the comparison maps and induced event transitions must be specified; equal output dimensions alone do not supply these maps.
The program spaces G j ⋆ \mathcal{G}^{\star}_{j} G j ⋆ and G j m \mathcal{G}^{m}_{j} G j m are sets of admissible instructions. A lowercase γ \gamma γ is a fixed member; Γ j \Gamma_{j} Γ j is the possibly random intended program. A program includes the declared input-selection rules, call order, mixing weights or weight-computation rules, and aggregation. Canonical means that roles are named consistently across models. The implementation map ι j m \iota^{m}_{j} ι j m is fixed as part of this comparison and may send different roles to the same parameter block.
Let H j m \mathcal{H}^{m}_{j} H j m be the space of admissible history records at the routing boundary. Then
H j m ∈ H j m , R j m : H j m ⟶ G j m , Γ ^ j m = R j m ( H j m ) . \mathcal{H}^{m}_{j}\in\mathcal{H}^{m}_{j},\qquad
R^{m}_{j}:\mathcal{H}^{m}_{j}\longrightarrow\mathcal{G}^{m}_{j},\qquad
\widehat{\Gamma}^{m}_{j}=R^{m}_{j}(\mathcal{H}^{m}_{j}). H j m ∈ H j m , R j m : H j m ⟶ G j m , Γ j m = R j m ( H j m ) .
The history record may contain the router’s view of the current state, earlier plans and outputs, and random draws already revealed for this decision. It need not expose every coordinate of the complete event state. It is a random record, not itself a sigma-field. If F j m \mathcal{F}^{m}_{j} F j m is the information allowed before the decision, non-anticipation requires the history and decision to be measurable with respect to F j m \mathcal{F}^{m}_{j} F j m . For an internal adaptivity policy this requirement applies separately at each internal decision.
An external random tape is a collection of random draws used to realize all compared executions on one probability space. Fixing it makes the transitions deterministic as functions of their arguments; only the appropriate revealed portion is visible to a router. The tape argument is suppressed in Φ j ⋆ \Phi^{\star}_{j} Φ j ⋆ and Φ j m \Phi^{m}_{j} Φ j m .
There are three distinct event outputs at a common model state: actual execution uses Φ j m ( X j m ; Γ ^ j m ) \Phi^{m}_{j}(X^{m}_{j};\widehat{\Gamma}^{m}_{j}) Φ j m ( X j m ; Γ j m ) ; aligned execution uses Φ j m ( X j m ; ι j m ( Γ j ) ) \Phi^{m}_{j}(X^{m}_{j};\iota^{m}_{j}(\Gamma_{j})) Φ j m ( X j m ; ι j m ( Γ j )) ; the reference comparison uses Φ j ⋆ ( X j m ; Γ j ) \Phi^{\star}_{j}(X^{m}_{j};\Gamma_{j}) Φ j ⋆ ( X j m ; Γ j ) . The actual reference trajectory instead evaluates the last map at X j X_{j} X j . These insertions are precisely the split used in Equation (74) .
In § 5 all compared stage states are represented in a declared common metric space ( S , d ) (\mathcal{S},d) ( S , d ) , and the notation is
S j = X j , S ^ j m = X ^ j m , T j , γ ( s ) = Φ j ⋆ ( s ; γ ) , T ^ j , γ m ( s ) = Φ j m ( s ; γ ) . S_{j}=X_{j},\qquad
\widehat{S}^{m}_{j}=\widehat{X}^{m}_{j},\qquad
T_{j,\gamma}(s)=\Phi^{\star}_{j}(s;\gamma),\qquad
\widehat{T}^{m}_{j,\gamma}(s)=\Phi^{m}_{j}(s;\gamma). S j = X j , S j m = X j m , T j , γ ( s ) = Φ j ⋆ ( s ; γ ) , T j , γ m ( s ) = Φ j m ( s ; γ ) .
In the first transition γ ∈ G j ⋆ \gamma\in\mathcal{G}^{\star}_{j} γ ∈ G j ⋆ ; in the second it lies in G j m \mathcal{G}^{m}_{j} G j m . The semicolon merely separates state and plan arguments. The intended program may depend on the reference history; recomputing a reference policy on the learned state would define a different comparator.
The abstraction includes the following common cases.
Table 15.
It uses their fixed-program transitions, their implementation maps, and the role-level domination condition stated below. The architecture-specific transport and matching results in § 21 apply only to their stated additive or preselected-sequence specializations.
Roles as fixed call environments
A role a ∈ J a \in\mathcal{J} a ∈ J specifies (i) a probe distribution P a P_a P a and (ii) a measurable event lift T a [ f ] \mathcal{T}_a[f] T a [ f ] . A probe record z z z contains all information needed to execute the controlled call environment: the pre-event state, canonical program, controlled occurrence, revealed randomness, and the fixed values of the other components in the hybrid comparison. The lift returns a post-event model state after inserting candidate function f f f at the controlled occurrence. The target hybrid T a ⋆ \mathcal{T}^{\star}_a T a ⋆ has the same codomain. Therefore
ℓ a ( f ; z ) = d ( T a [ f ] ( z ) , T a ⋆ ( z ) ) (30) \ell_a(f;z) = d\left(\mathcal{T}_a[f](z),\mathcal{T}^{\star}_a(z)\right) \tag*{(30)} ℓ a ( f ; z ) = d ( T a [ f ] ( z ) , T a ⋆ ( z ) ) (30)
is a distance between two event states, even if f f f itself maps a token representation to an FFN residual.
For the finite-sample theorem, P a P_a P a , T a \mathcal{T}_a T a , T a ⋆ \mathcal{T}^{\star}_a T a ⋆ , and the expert class are fixed independently of the samples used to estimate the experts, and ℓ a ∈ [ 0 , 1 ] \ell_a \in[0,1] ℓ a ∈ [ 0 , 1 ] . A frozen surrounding assembly or an independent sample split is sufficient. If the same observations jointly train the backbone, dispatch, or event lift, one must analyze the joint hypothesis class rather than condition on a data-dependent loss without paying its complexity. Scaling a bounded distance is valid; post-hoc clipping of an unbounded distance controls only the clipped quantity and cannot silently be substituted into an unclipped terminal bound.
A reuse assignment α m : J → [ K m ] \alpha_m:\mathcal{J} \to[K_m] α m : J → [ K m ] maps roles to model-specific parameter blocks. The main compact comparison assumes that the canonical roles, evaluation weights, and expert-estimation classes are common across comparators. Data-dependent assignments require either an independent selection split or an additional selection-complexity term.
A deterministic call-replacement lemma
Fix a state s s s and one realized canonical program γ \gamma γ with Q ( γ ) Q(\gamma) Q ( γ ) controlled call occurrences. For a model m m m , let T ~ j , γ m , ( q ) ( s ) \widetilde{T}^{m,(q)}_{j,\gamma}(s) T j , γ m , ( q ) ( s ) be the hybrid event output in which the first q q q canonical occurrences use the fitted expert functions and the remaining occurrences use their target functions, while all non-expert components use the fixed model-m m m assembly. Thus T ~ j , γ m , ( Q ) \widetilde{T}^{m,(Q)}_{j,\gamma} T j , γ m , ( Q ) is the correctly aligned learned event and T ~ m , ( 0 ) \widetilde{T}^{m,(0)} T m , ( 0 ) uses target experts inside the model assembly. Define
τ j , γ m ( s ) = d ( T ~ j , γ m , ( Q ) ( s ) , T j , γ ( s ) ) , (31) \tau^{m}_{j,\gamma}(s) = d\left(\widetilde{T}^{m,(Q)}_{j,\gamma}(s),T_{j,\gamma}(s)\right), \tag*{(31)} τ j , γ m ( s ) = d ( T j , γ m , ( Q ) ( s ) , T j , γ ( s ) ) , (31)
which isolates shared-backbone, dispatch, aggregation, or other assembly mismatch.
Lemma 6 (Call-by-call event replacement). For every s s s , γ \gamma γ , and model m m m ,
d ( T ~ j , γ m , ( Q ) ( s ) , T j , γ ( s ) ) ≤ τ j , γ m ( s ) + ∑ q = 1 Q ( γ ) d ( T ~ j , γ m , ( q ) ( s ) , T ~ j , γ m , ( q − 1 ) ( s ) ) . (32) d\left(\widetilde{T}_{j,\gamma}^{m,(Q)}(s),T_{j,\gamma}(s)\right) \le\tau_{j,\gamma}^{m}(s)+\sum_{q=1}^{Q(\gamma)}d\left(\widetilde{T}_{j,\gamma}^{m,(q)}(s),\widetilde{T}_{j,\gamma}^{m,(q-1)}(s)\right). \tag*{(32)} d ( T j , γ m , ( Q ) ( s ) , T j , γ ( s ) ) ≤ τ j , γ m ( s ) + q = 1 ∑ Q ( γ ) d ( T j , γ m , ( q ) ( s ) , T j , γ m , ( q − 1 ) ( s ) ) . (32) Each summand is an event-lift loss for the role of occurrence q q q under the corresponding hybrid probe record.
Proof. Insert the sequence T ~ m , ( Q − 1 ) , … , T ~ m , ( 0 ) \widetilde{T}^{m,(Q-1)},\ldots,\widetilde{T}^{m,(0)} T m , ( Q − 1 ) , … , T m , ( 0 ) and apply the triangle inequality. By construction, adjacent hybrids differ at exactly one controlled occurrence and have outputs in the same post-event state space. The lemma is a fixed-program hybrid statement. If later routing decisions adapt to earlier expert outputs, refine those decisions into separate routed events or treat the entire adaptive policy as one indivisible event lift. Conditioning on one realized trace while replacing an earlier output is valid only when the later decisions are explicitly frozen as part of the intervention; it is not automatically a decomposition of the original adaptive execution.
A rigorous sufficient occupancy-transfer condition
Condition on the expert-estimation sample and let Z j , q m Z_{j,q}^{m} Z j , q m be the hybrid probe record of the q q q th controlled occurrence when that occurrence exists. Assume a deterministic maximum Q max , j < ∞ Q_{\max,j}<\infty Q m a x , j < ∞ ; a summable version gives the same conclusion for an unbounded count. For each role a a a , define the conditional sub-probability measure
Q j , q , a m ( B ) = P ( Z j , q m ∈ B , q ≤ Q ( Γ j ) , a ( j , Γ j , q ) = a ∣ D ) , (33) Q_{j,q,a}^{m}(B)=\mathbb{P}\left(Z_{j,q}^{m}\in B,\ q\le Q(\Gamma_j),\ a(j,\Gamma_j,q)=a\mid\mathcal{D}\right), \tag*{(33)} Q j , q , a m ( B ) = P ( Z j , q m ∈ B , q ≤ Q ( Γ j ) , a ( j , Γ j , q ) = a ∣ D ) , (33)
where D \mathcal{D} D denotes the information in the fitted sample. Formally this is a sigma-field: the collection of events whose truth can be determined from the sample. Conditioning fixes the fitted parameters, not the fresh evaluation trajectory. Suppose there are deterministic constants κ j , q , a ≥ 0 \kappa_{j,q,a}\ge0 κ j , q , a ≥ 0 , fixed independently of D \mathcal{D} D and valid for both comparators, such that
Q j , q , a m ≪ P a , ∥ d Q j , q , a m d P a ∥ ∞ ≤ κ j , q , a a.s. (34) Q_{j,q,a}^{m}\ll P_a,\qquad\left\|\frac{dQ_{j,q,a}^{m}}{dP_a}\right\|_{\infty}\le\kappa_{j,q,a}\quad\text{a.s.} \tag*{(34)} Q j , q , a m ≪ P a , d P a d Q j , q , a m ∞ ≤ κ j , q , a a.s. (34)
The notation Q ≪ P Q\ll P Q ≪ P means P ( B ) = 0 P(B)=0 P ( B ) = 0 implies Q ( B ) = 0 Q(B)=0 Q ( B ) = 0 . The Radon–Nikodym derivative d Q / d P dQ/dP d Q / d P is the weight function satisfying Q ( B ) = ∫ B ( d Q / d P ) d P Q(B)=\int_B(dQ/dP)\,dP Q ( B ) = ∫ B ( d Q / d P ) d P ; its ∞ \infty ∞ -norm is the smallest almost-sure upper bound. Because Q j , q , a m Q_{j,q,a}^{m} Q j , q , a m has total mass at most one (a sub-probability measure), κ j , q , a \kappa_{j,q,a} κ j , q , a already includes both occurrence mass and distribution shift. Integrating each hybrid loss in Equation (32) yields
E [ a j m ∣ D ] ≤ τ j m + ∑ a ∈ J ω j , a R a ( f ^ α m ( a ) ) , ω j , a = ∑ q = 1 Q max , j κ j , q , a , (35) \mathbb{E}[a_j^m\mid\mathcal{D}]\le\tau_j^m+\sum_{a\in\mathcal{J}}\omega_{j,a}R_a\left(\widehat{f}_{\alpha_m(a)}\right),\qquad\omega_{j,a}=\sum_{q=1}^{Q_{\max,j}}\kappa_{j,q,a}, \tag*{(35)} E [ a j m ∣ D ] ≤ τ j m + a ∈ J ∑ ω j , a R a ( f α m ( a ) ) , ω j , a = q = 1 ∑ Q m a x , j κ j , q , a , (35)
where
τ j m = E [ τ j , Γ j m ( S ^ j m ) ∣ D ] . (36) \tau_j^m=\mathbb{E}\left[\tau_{j,\Gamma_j}^{m}\left(\widehat{S}_{j}^{m}\right)\mid\mathcal{D}\right]. \tag*{(36)} τ j m = E [ τ j , Γ j m ( S j m ) ∣ D ] . (36)
Indeed, ∫ ℓ a ( f ^ ; z ) d Q j , q , a m ( z ) ≤ κ j , q , a R a ( f ^ ) \int\ell_a(\widehat{f};z)\,dQ_{j,q,a}^{m}(z)\le\kappa_{j,q,a}R_a(\widehat{f}) ∫ ℓ a ( f ; z ) d Q j , q , a m ( z ) ≤ κ j , q , a R a ( f ) . This is one sufficient condition for Assumption 3. Proposition 42 permits support mismatch by retaining an additive discrepancy term. More refined lifts can absorb within-event suffix sensitivity and give smaller weights. If only model-specific envelopes κ j , q , a m \kappa_{j,q,a}^{m} κ j , q , a m are credible, each comparator must retain its own terminal weights and full certificate.
Hierarchy of reuse claims
Parameter reuse : identical weights occur in multiple calls.
Statistical reuse : samples from several roles jointly estimate those weights.
Functional reuse : one function has low role risk on the stated role distributions.
Causal reuse : a pre-action route intervention induces a stable macro transition.
Compositional reuse : that transition remains accurate in previously unseen route sequences or horizons.
Only the first item is an architectural fact. Each subsequent item requires additional empirical or mathematical conditions.
Partial pooling versus physical tying. The interpolation in Equation (5) shares statistical information, but
A ^ a ( λ ) − A ^ b ( λ ) = ( 1 − λ ) ( A ^ a − A ^ b ) . \widehat{A}^{(\lambda)}_a-\widehat{A}^{(\lambda)}_b=(1-\lambda)(\widehat{A}_a-\widehat{A}_b). A a ( λ ) − A b ( λ ) = ( 1 − λ ) ( A a − A b ) .
For 0 < λ < 1 0<\lambda<1 0 < λ < 1 , these matrices need not coincide. Writing them as a common core plus role-specific residuals is only a reparameterization unless the residual capacity is restricted and counted. Full pooling and each block of a hard partition do use one common matrix; intermediate shrinkage does not by itself reduce the number of stored role matrices. Theorems about estimation risk, approximation under an update constraint, and physical storage therefore address different objects.
The expert’s information is part of its function class. The common operator in the local model receives x x x , not a separate role identifier. A class of functions f ( x , a ) f(x,a) f ( x , a ) can implement role-dependent operations with one parameter block and is not the same class as the shared f ( x ) f(x) f ( x ) in that model. A hidden feature can also encode role or depth without an explicit identifier. Thus a fixed feature map, its input law, the permitted conditioning information, and any adapter must be declared before interpreting a sharing bias.
Distribution-Free Statistical Theory
Overview. We compare fitting one function for a group with fitting a separate function for each role. Reuse bias measures the population cost of the shared restriction, while concentration controls estimation from independent records. Section C.1 interprets reuse bias, Section C.2 proves the finite-class comparison, Section C.3 examines sample allocation, and Section C.4 gives uniform-deviation and trace-level templates.
Reuse bias and its functional interpretation
Fix a group J e \mathcal{J}_e J e and a class F e \mathcal{F}_e F e . The role distributions, losses, class, weights, and sample counts are fixed independently of the observations used by the ERMs. For weights π e , a > 0 \pi_{e,a}>0 π e , a > 0 summing to one, define the shared and separate population oracles as in Equation (15) . Nonnegativity of B e r e u s e B^{\mathrm{reuse}}_e B e reuse follows because the shared class is the diagonal subset of the product class ∏ a ∈ J e F e \prod_{a\in\mathcal{J}_e}\mathcal{F}_e ∏ a ∈ J e F e . The concentration theorem below assumes ℓ a ( f ; Z ) ∈ [ 0 , 1 ] \ell_a(f;Z)\in[0,1] ℓ a ( f ; Z ) ∈ [ 0 , 1 ] for every role and candidate; if the surrounding event lift is learned from the same sample, it must be included in the hypothesis class or learned on an independent split.
A useful exact interpretation holds under a common quadratic geometry.
Proposition 7 (Reuse bias as functional variance). First take the unrestricted candidate class F e = H \mathcal{F}_e=\mathcal{H} F e = H , where H \mathcal{H} H is a real Hilbert space, and suppose
R a ( f ) = c a + ∥ f − f a ⋆ ∥ H 2 , f ∈ H . (37) R_a(f)=c_a+\lVert f-f_a^\star\rVert_{\mathcal{H}}^2,\qquad f\in\mathcal{H}. \tag*{(37)} R a ( f ) = c a + ∥ f − f a ⋆ ∥ H 2 , f ∈ H . (37) With f ˉ = ∑ a π e , a f a ⋆ \bar{f}=\sum_a\pi_{e,a}f_a^\star f ˉ = ∑ a π e , a f a ⋆ ,
B e r e u s e = ∑ a π e , a ∥ f a ⋆ − f ˉ ∥ H 2 = 1 2 ∑ a , b π e , a π e , b ∥ f a ⋆ − f b ⋆ ∥ H 2 . (38) B_e^{\mathrm{reuse}}=\sum_a\pi_{e,a}\lVert f_a^\star-\bar{f}\rVert_{\mathcal{H}}^2=\frac{1}{2}\sum_{a,b}\pi_{e,a}\pi_{e,b}\lVert f_a^\star-f_b^\star\rVert_{\mathcal{H}}^2. \tag*{(38)} B e reuse = a ∑ π e , a ∥ f a ⋆ − f ˉ ∥ H 2 = 2 1 a , b ∑ π e , a π e , b ∥ f a ⋆ − f b ⋆ ∥ H 2 . (38) More generally, for any nonempty class F ⊆ H \mathcal{F}\subseteq\mathcal{H} F ⊆ H and risks R a ( f ) = c a + ∥ f − t a ∥ H 2 R_a(f)=c_a+\lVert f-t_a\rVert_{\mathcal{H}}^2 R a ( f ) = c a + ∥ f − t a ∥ H 2 , set t ˉ = ∑ a π e , a t a \bar{t}=\sum_a\pi_{e,a}t_a t ˉ = ∑ a π e , a t a and d ( u , F ) = inf f ∈ F ∥ u − f ∥ H d(u,\mathcal{F})=\inf_{f\in\mathcal{F}}\lVert u-f\rVert_{\mathcal{H}} d ( u , F ) = inf f ∈ F ∥ u − f ∥ H . The restricted-class reuse bias is
B F = ∑ a π e , a ∥ t a − t ˉ ∥ H 2 + d ( t ˉ , F ) 2 − ∑ a π e , a d ( t a , F ) 2 . (39) B_{\mathcal{F}}=\sum_a\pi_{e,a}\lVert t_a-\bar{t}\rVert_{\mathcal{H}}^2+d(\bar{t},\mathcal{F})^2-\sum_a\pi_{e,a}d(t_a,\mathcal{F})^2. \tag*{(39)} B F = a ∑ π e , a ∥ t a − t ˉ ∥ H 2 + d ( t ˉ , F ) 2 − a ∑ π e , a d ( t a , F ) 2 . (39) No convexity or attainment of the infima is required. In particular, if every t a ∈ F t_a\in\mathcal{F} t a ∈ F and t ˉ ∈ F \bar{t}\in\mathcal{F} t ˉ ∈ F (or its closure), only the variance term remains.
Proof. Write f ˉ − f a ⋆ = ( f ˉ − f a ⋆ ) \bar{f}-f_a^\star=(\bar{f}-f_a^\star) f ˉ − f a ⋆ = ( f ˉ − f a ⋆ ) . The identity ∑ a π e , a ( f ˉ − f a ⋆ ) = 0 \sum_a\pi_{e,a}(\bar{f}-f_a^\star)=0 ∑ a π e , a ( f ˉ − f a ⋆ ) = 0 cancels the summed cross term, giving
∑ a π e , a ∥ f − f a ⋆ ∥ H 2 = ∥ f − f ˉ ∥ H 2 + ∑ a π e , a ∥ f ˉ − f a ⋆ ∥ H 2 . \sum_a\pi_{e,a}\lVert f-f_a^\star\rVert_{\mathcal{H}}^2=\lVert f-\bar{f}\rVert_{\mathcal{H}}^2+\sum_a\pi_{e,a}\lVert\bar{f}-f_a^\star\rVert_{\mathcal{H}}^2. a ∑ π e , a ∥ f − f a ⋆ ∥ H 2 = ∥ f − f ˉ ∥ H 2 + a ∑ π e , a ∥ f ˉ − f a ⋆ ∥ H 2 .
Only the first term depends on the shared function, so it is minimized at f = f ˉ f = \bar{f} f = f ˉ . The constants c a c_a c a cancel when subtracting the separate oracle risk. Expanding the weighted double sum gives
∑ a , b π e , a π e , b ∥ f a ∗ − f b ∗ ∥ H 2 = 2 ( ∑ a π e , a ∥ f a ∗ ∥ H 2 − ∥ f ˉ ∥ H 2 ) . \sum_{a,b} \pi_{e,a}\pi_{e,b}\lVert f_a^* - f_b^*\rVert_{\mathcal{H}}^2 = 2\left(\sum_a \pi_{e,a}\lVert f_a^*\rVert_{\mathcal{H}}^2 - \lVert\bar{f}\rVert_{\mathcal{H}}^2\right). a , b ∑ π e , a π e , b ∥ f a ∗ − f b ∗ ∥ H 2 = 2 ( a ∑ π e , a ∥ f a ∗ ∥ H 2 − ∥ f ˉ ∥ H 2 ) .
which yields the pairwise form. For the restricted identity, the same completed-square equality with t a t_a t a holds for every f ∈ F f \in\mathcal{F} f ∈ F . Taking its infimum gives the shared risk ∑ a π e , a c a + ∑ a π e , a ∥ t a − t ˉ ∥ 2 + d ( t ˉ , F ˉ ) 2 \sum_a \pi_{e,a}c_a + \sum_a \pi_{e,a}\lVert t_a - \bar{t}\rVert^2 + d(\bar{t},\bar{\mathcal{F}})^2 ∑ a π e , a c a + ∑ a π e , a ∥ t a − t ˉ ∥ 2 + d ( t ˉ , F ˉ ) 2 . Subtract the separate infima ∑ a π e , a [ c a + d ( t a , F ) 2 ] \sum_a \pi_{e,a}[c_a + d(t_a,\mathcal{F})^2] ∑ a π e , a [ c a + d ( t a , F ) 2 ] to obtain Equation (39) .
The proposition motivates functional probes.
A finite-class obstruction to the pure-variance interpretation. Let H = R \mathcal{H} = \mathbb{R} H = R , F = { 0 , 1 } \mathcal{F} = \{0,1\} F = { 0 , 1 } , π 1 = π 2 = 1 / 2 \pi_1 = \pi_2 = 1/2 π 1 = π 2 = 1/2 , and R 1 ( f ) = f 2 R_1(f) = f^2 R 1 ( f ) = f 2 , R 2 ( f ) = ( f − 1 ) 2 R_2(f) = (f - 1)^2 R 2 ( f ) = ( f − 1 ) 2 . Losses on the candidate class are in [ 0 , 1 ] [0,1] [ 0 , 1 ] and both role targets are realizable. Nevertheless,
B F = 1 2 , ∑ a π a ( t a − t ˉ ) 2 = 1 4 , d ( t ˉ , F ) 2 = 1 4 . (40) B_{\mathcal{F}} = \frac{1}{2}, \qquad\sum_a \pi_a(t_a - \bar{t})^2 = \frac{1}{4}, \qquad d(\bar{t},\mathcal{F})^2 = \frac{1}{4}. \tag*{(40)} B F = 2 1 , a ∑ π a ( t a − t ˉ ) 2 = 4 1 , d ( t ˉ , F ) 2 = 4 1 . (40)
The missing mean t ˉ = 1 / 2 \bar{t} = 1/2 t ˉ = 1/2 accounts for the gap. A common squared-loss geometry alone therefore does not imply the pure-variance claim.
Finite-class comparison and proof
Now the question is sampling error: how far can every candidate’s empirical loss be from its population loss? A simultaneous event controls both fitted comparators.
Theorem 8 (Simultaneous finite-class comparison). Assume the losses from Equation (3) lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , F e \mathcal{F}_e F e is finite, n a ≥ 1 n_a \ge1 n a ≥ 1 , and all role samples are mutually independent. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ , the shared and role-specific empirical risk minimizers (ERMs) satisfy
∑ a π e , a R a ( f ^ e ) ≤ R s e p , e ∗ + B e r e u s e + 2 ϵ s h , e , (41) \sum_a \pi_{e,a}R_a(\hat{f}_e) \le R^*_{\mathrm{sep},e} + B^{\mathrm{reuse}}_e + 2\epsilon_{\mathrm{sh},e}, \tag*{(41)} a ∑ π e , a R a ( f ^ e ) ≤ R sep , e ∗ + B e reuse + 2 ϵ sh , e , (41) ∑ a π e , a R a ( f ^ a ) ≤ R s e p , e ∗ + 2 ϵ s e p , e , (42) \sum_a \pi_{e,a}R_a(\hat{f}_a) \le R^*_{\mathrm{sep},e} + 2\epsilon_{\mathrm{sep},e}, \tag*{(42)} a ∑ π e , a R a ( f ^ a ) ≤ R sep , e ∗ + 2 ϵ sep , e , (42) where
ϵ s h , e = log ( 4 ∣ F e ∣ / δ ) 2 N e e f f , (43) \epsilon_{\mathrm{sh},e} = \sqrt{\frac{\log(4|\mathcal{F}_e|/\delta)}{2N^{\mathrm{eff}}_e}}, \tag*{(43)} ϵ sh , e = 2 N e eff log ( 4∣ F e ∣/ δ ) , (43) ϵ s e p , e = ∑ a π e , a log ( 4 ∣ J e ∣ ∣ F e ∣ / δ ) 2 n a . (44) \epsilon_{\mathrm{sep},e} = \sum_a \pi_{e,a}\sqrt{\frac{\log(4|\mathcal{J}_e||\mathcal{F}_e|/\delta)}{2n_a}}. \tag*{(44)} ϵ sep , e = a ∑ π e , a 2 n a log ( 4∣ J e ∣∣ F e ∣/ δ ) . (44) Thus the displayed shared certificate is smaller whenever B e r e u s e < 2 ( ϵ s e p , e − ϵ s h , e ) B^{\mathrm{reuse}}_e < 2(\epsilon_{\mathrm{sep},e} - \epsilon_{\mathrm{sh},e}) B e reuse < 2 ( ϵ sep , e − ϵ sh , e ) .
Proof. For notational simplicity suppress e e e . Role a a a supplies independent observations Z a , 1 , … , Z a , n a ∼ P a Z_{a,1},\ldots,Z_{a,n_a} \sim P_a Z a , 1 , … , Z a , n a ∼ P a , and
R ^ a ( f ) = 1 n a ∑ i = 1 n a ℓ a ( f ; Z a , i ) , R ^ s h ( f ) = ∑ a π a R ^ a ( f ) . (45) \widehat{R}_a(f) = \frac{1}{n_a}\sum_{i=1}^{n_a}\ell_a(f;Z_{a,i}), \qquad\widehat{R}_{\mathrm{sh}}(f) = \sum_a \pi_a\widehat{R}_a(f). \tag*{(45)} R a ( f ) = n a 1 i = 1 ∑ n a ℓ a ( f ; Z a , i ) , R sh ( f ) = a ∑ π a R a ( f ) . (45)
For fixed f f f , weighted Hoeffding gives
P ( ∣ R ^ s h ( f ) − ∑ a π a R a ( f ) ∣ > t ) ≤ 2 exp ( − 2 N e e f f t 2 ) , (46) \mathbb{P}\left(\left|\widehat{R}_{\mathrm{sh}}(f) - \sum_a \pi_aR_a(f)\right| > t\right) \le2\exp\left(-2N^{\mathrm{eff}}_e t^2\right), \tag*{(46)} P ( R sh ( f ) − a ∑ π a R a ( f ) > t ) ≤ 2 exp ( − 2 N e eff t 2 ) , (46)
where ( N e e f f ) − 1 = ∑ a π a 2 / n a (N^{\mathrm{eff}}_e)^{-1} = \sum_a \pi_a^2/n_a ( N e eff ) − 1 = ∑ a π a 2 / n a . A union bound over F e \mathcal{F}_e F e with failure budget δ / 2 \delta/2 δ /2 yields
sup f ∈ F e ∣ R ^ s h ( f ) − ∑ a π a R a ( f ) ∣ ≤ ϵ s h , e . (47) \sup_{f\in\mathcal{F}_e}\left|\widehat{R}_{\mathrm{sh}}(f) - \sum_a \pi_aR_a(f)\right| \le\epsilon_{\mathrm{sh},e}. \tag*{(47)} f ∈ F e sup R sh ( f ) − a ∑ π a R a ( f ) ≤ ϵ sh , e . (47)
If f s h ⋆ f^\star_{\mathrm{sh}} f sh ⋆ is a shared population minimizer and f ^ e \widehat{f}_e f e is a shared ERM, then
∑ a π a R a ( f ^ e ) ≤ R ^ s h ( f ^ e ) + ϵ s h , e ≤ R ^ s h ( f s h ⋆ ) + ϵ s h , e ≤ R s h a r e , e ⋆ + 2 ϵ s h , e = R s e p , e ⋆ + B e r e u s e + 2 ϵ s h , e . (48) \sum_a \pi_a R_a(\widehat{f}_e) \le\widehat{R}_{\mathrm{sh}}(\widehat{f}_e) + \epsilon_{\mathrm{sh},e}
\le\widehat{R}_{\mathrm{sh}}(f^\star_{\mathrm{sh}}) + \epsilon_{\mathrm{sh},e}
\le R^\star_{\mathrm{share},e} + 2\epsilon_{\mathrm{sh},e} = R^\star_{\mathrm{sep},e} + B^{\mathrm{reuse}}_e + 2\epsilon_{\mathrm{sh},e}. \tag*{(48)} a ∑ π a R a ( f e ) ≤ R sh ( f e ) + ϵ sh , e ≤ R sh ( f sh ⋆ ) + ϵ sh , e ≤ R share , e ⋆ + 2 ϵ sh , e = R sep , e ⋆ + B e reuse + 2 ϵ sh , e . (48)
For every role, Hoeffding and a union bound over ∣ J e ∣ ∣ F e ∣ |\mathcal{J}_e||\mathcal{F}_e| ∣ J e ∣∣ F e ∣ with the remaining failure budget give
sup f ∈ F e ∣ R ^ a ( f ) − R a ( f ) ∣ ≤ log ( 4 ∣ J e ∣ ∣ F e ∣ / δ ) 2 n a . (49) \sup_{f\in\mathcal{F}_e} \left|\widehat{R}_a(f)-R_a(f)\right| \le\sqrt{\frac{\log(4|\mathcal{J}_e||\mathcal{F}_e|/\delta)}{2n_a}}. \tag*{(49)} f ∈ F e sup R a ( f ) − R a ( f ) ≤ 2 n a log ( 4∣ J e ∣∣ F e ∣/ δ ) . (49)
The same ERM argument role by role, followed by multiplication by π a \pi_a π a and summation, proves Theorem 8. The two uniform events hold jointly with probability at least 1 − δ 1-\delta 1 − δ .
What the effective sample size optimizes
Proposition 9 (Allocation maximizing the Hoeffding effective sample size). For positive real allocations with N = ∑ a n a N=\sum_a n_a N = ∑ a n a ,
N e e f f = ( ∑ a π a 2 n a ) − 1 ≤ N , (50) N^{\mathrm{eff}}_e=\left(\sum_a\frac{\pi_a^2}{n_a}\right)^{-1}\le N, \tag*{(50)} N e eff = ( a ∑ n a π a 2 ) − 1 ≤ N , (50) with equality if and only if n a = N π a n_a=N\pi_a n a = N π a .
Proof. Cauchy–Schwarz gives 1 = ( ∑ a π a ) 2 ≤ ( ∑ a π a 2 / n a ) ( ∑ a n a ) = N / N e e f f 1=(\sum_a\pi_a)^2\le(\sum_a\pi_a^2/n_a)(\sum_a n_a)=N/N^{\mathrm{eff}}_e 1 = ( ∑ a π a ) 2 ≤ ( ∑ a π a 2 / n a ) ( ∑ a n a ) = N / N e eff . Equality requires π a / n a \pi_a/\sqrt{n_a} π a / n a to be proportional to n a \sqrt{n_a} n a .
This allocation maximizes the particular bounded-loss Hoeffding effective sample size. A Bernstein analysis can produce variance-aware allocations for unequal variances.
The finite class is not essential. If, on one event,
sup f ∣ R ^ s h ( f ) − R s h ( f ) ∣ ≤ u s h , sup f ∣ R ^ a ( f ) − R a ( f ) ∣ ≤ u a , (51) \sup_f\left|\widehat{R}_{\mathrm{sh}}(f)-R_{\mathrm{sh}}(f)\right|\le u_{\mathrm{sh}},\qquad\sup_f\left|\widehat{R}_a(f)-R_a(f)\right|\le u_a, \tag*{(51)} f sup R sh ( f ) − R sh ( f ) ≤ u sh , f sup R a ( f ) − R a ( f ) ≤ u a , (51)
then the same proof gives shared excess 2 u s h 2u_{\mathrm{sh}} 2 u sh and separate excess 2 ∑ a π a u a 2\sum_a\pi_a u_a 2 ∑ a π a u a . Uniform bounds based on Rademacher complexity (fit to random signs) or covering numbers (the size of a finite approximation to a class) may be substituted if they prove exactly Equation (51) . Algorithm-specific stability, compression, and PAC-Bayes bounds are not automatically uniform-deviation statements; using them requires replacing the learning theorem and matching its particular estimator or randomized predictor.
Calls inside one autoregressive trace should not be counted as independent samples. Let Z 1 , … , Z N Z_1,\ldots,Z_N Z 1 , … , Z N be independent traces, not necessarily identically distributed, and let L r ( f ) ∈ [ 0 , 1 ] L_r(f)\in[0,1] L r ( f ) ∈ [ 0 , 1 ] be any aggregate trace loss. For a finite class F \mathcal{F} F , with probability at least 1 − δ 1-\delta 1 − δ ,
sup f ∈ F ∣ 1 N ∑ r = 1 N ( L r ( f ) − E L r ( f ) ) ∣ ≤ log ( 2 ∣ F ∣ / δ ) 2 N . (52) \sup_{f\in\mathcal{F}}\left|\frac{1}{N}\sum_{r=1}^{N}\left(L_r(f)-\mathbb{E}L_r(f)\right)\right|\le\sqrt{\frac{\log(2|\mathcal{F}|/\delta)}{2N}}. \tag*{(52)} f ∈ F sup N 1 r = 1 ∑ N ( L r ( f ) − E L r ( f ) ) ≤ 2 N log ( 2∣ F ∣/ δ ) . (52)
If traces are IID, the centering equals the common population trace risk. No within-trace independence is needed.
Exact Linear Sharing Frontier
Overview. We prove the exact pooling and partition risks in the specified linear model. We also separate fixed-rule identities from data-dependent choices: a same-data plug-in can fail, independent validation needs its own guarantee, and the stated high-dimensional Gaussian specialization is classical shrinkage. Sections D.1–D.3 derive the pooling, merge, and dispersion formulas; Sections D.4–D.6 contrast a failing plug-in, independent validation, and the classical Gaussian shrinkage specialization.
Oracle details and proof of the exact sharing frontier
For B + C > 0 B+C>0 B + C > 0 , the optimal fixed coefficient and its minimum risk in Theorem 1 are
λ ⋆ = C B + C , R λ ⋆ = V − C 2 B + C . (53) \lambda^\star= \frac{C}{B+C}, \qquad\mathcal{R}_{\lambda^\star}=V-\frac{C^2}{B+C}. \tag*{(53)} λ ⋆ = B + C C , R λ ⋆ = V − B + C C 2 . (53)
If B = C = 0 B=C=0 B = C = 0 , every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] gives the same risk V V V . For M > 1 M>1 M > 1 , σ > 0 \sigma>0 σ > 0 , and finite B B B , oracle partial pooling is no worse than either endpoint; it is strictly better than separate estimation and is strictly better than full reuse whenever B > 0 B>0 B > 0 .
Proof of Theorem 1. Write Y a = X a A a T + E a Y_a=X_aA_a^\mathsf{T}+\mathcal{E}_a Y a = X a A a T + E a , where the noise rows have mean zero, covariance σ 2 I p \sigma^2 I_p σ 2 I p , and are independent across observations and roles. Gaussianity is not used in the expected-risk proof. Orthogonality of the fixed design gives
A ^ a = A a + E a , E a = 1 n E a T X a . (54) \widehat{A}_a=A_a+E_a,\qquad E_a=\frac{1}{n}\mathcal{E}_a^\mathsf{T}X_a. \tag*{(54)} A a = A a + E a , E a = n 1 E a T X a . (54)
The p d = D pd=D p d = D entries of E a E_a E a are centered and uncorrelated with variance σ 2 / n \sigma^2/n σ 2 / n , and the matrices E a E_a E a are independent across roles. Indeed, for output coordinates u , v u,v u , v and input coordinates r , t r,t r , t , E [ ( E a ) u r ( E a ) v t ] = ( σ 2 / n ) 1 { u = v , r = t } \mathbb{E}[(E_a)_{ur}(E_a)_{vt}]=(\sigma^2/n)1\{u=v,r=t\} E [( E a ) u r ( E a ) v t ] = ( σ 2 / n ) 1 { u = v , r = t } . Hence
E ∥ E a ∥ F 2 = p d σ 2 n = V . (55) \mathbb{E}\lVert E_a\rVert_F^2=\frac{pd\sigma^2}{n}=V. \tag*{(55)} E ∥ E a ∥ F 2 = n p d σ 2 = V . (55)
For isotropic test input x x x , E x ∥ ( A ^ − A a ) x ∥ 2 2 = ∥ A ^ − A a ∥ F 2 \mathbb{E}_x\lVert(\widehat{A}-A_a)x\rVert_2^2=\lVert\widehat{A}-A_a\rVert_F^2 E x ∥( A − A a ) x ∥ 2 2 = ∥ A − A a ∥ F 2 .
Separate estimation therefore has average risk V V V . Let E ‾ = M − 1 ∑ a E a \overline{E}=M^{-1}\sum_a E_a E = M − 1 ∑ a E a . The pooled estimator is A ^ p o o l = A ‾ + E ‾ \widehat{A}_{\mathrm{pool}}=\overline{A}+\overline{E} A pool = A + E , so
1 M ∑ a E ∥ A ^ p o o l − A a ∥ F 2 = 1 M ∑ a ∥ A ‾ − A a ∥ F 2 + E ∥ E ‾ ∥ F 2 = B + V M . (56) \begin{aligned}
\frac{1}{M}\sum_a\mathbb{E}\lVert\widehat{A}_{\mathrm{pool}}-A_a\rVert_F^2
=
\frac{1}{M}\sum_a\lVert\overline{A}-A_a\rVert_F^2+\mathbb{E}\lVert\overline{E}\rVert_F^2 \\
&=B+\frac{V}{M}. \tag*{(56)}
\end{aligned} M 1 a ∑ E ∥ A pool − A a ∥ F 2 = M 1 a ∑ ∥ A − A a ∥ F 2 + E ∥ E ∥ F 2 = B + M V . (56)
where the cross term vanishes and E ∥ E ‾ ∥ F 2 = V / M \mathbb{E}\lVert\overline{E}\rVert_F^2=V/M E ∥ E ∥ F 2 = V / M .
For partial reuse,
A ^ a ( λ ) − A a = λ ( A ‾ − A a ) + ( 1 − λ ) E a + λ E ‾ . (57) \widehat{A}_a^{(\lambda)}-A_a=\lambda(\overline{A}-A_a)+(1-\lambda)E_a+\lambda\overline{E}. \tag*{(57)} A a ( λ ) − A a = λ ( A − A a ) + ( 1 − λ ) E a + λ E . (57)
The average squared bias is λ 2 B \lambda^2B λ 2 B . For each coefficient, the variance of ( 1 − λ ) E a + λ E ‾ (1-\lambda)E_a+\lambda\overline{E} ( 1 − λ ) E a + λ E is
σ 2 n [ ( 1 − λ ) 2 + 2 λ ( 1 − λ ) + λ 2 M ] . (58) \frac{\sigma^2}{n}\left[(1-\lambda)^2+\frac{2\lambda(1-\lambda)+\lambda^2}{M}\right]. \tag*{(58)} n σ 2 [ ( 1 − λ ) 2 + M 2 λ ( 1 − λ ) + λ 2 ] . (58)
Multiplying by D D D and simplifying yields
R λ = V − 2 V ( 1 − 1 M ) λ + [ B + V ( 1 − 1 M ) ] λ 2 , (59) \mathcal{R}_\lambda
=
V-2V\left(1-\frac{1}{M}\right)\lambda
+\left[B+V\left(1-\frac{1}{M}\right)\right]\lambda^2, \tag*{(59)} R λ = V − 2 V ( 1 − M 1 ) λ + [ B + V ( 1 − M 1 ) ] λ 2 , (59)
which is Equation (9) . The full-sharing comparison follows from R s h a r e − R s e p = B − C \mathcal{R}_{\mathrm{share}}-\mathcal{R}_{\mathrm{sep}}=B-C R share − R sep = B − C . Differentiating the strictly convex quadratic when B + C > 0 B+C>0 B + C > 0 gives λ ⋆ = C / ( B + C ) \lambda^\star=C/(B+C) λ ⋆ = C / ( B + C ) and the stated minimum. In the degenerate case B = C = 0 B=C=0 B = C = 0 , the quadratic is constant in λ \lambda λ . If M > 1 M>1 M > 1 and σ > 0 \sigma>0 σ > 0 , then C > 0 C>0 C > 0 , so the derivative at zero is negative. If additionally B > 0 B>0 B > 0 , then λ ⋆ < 1 \lambda^\star<1 λ ⋆ < 1 and the interior optimum is strictly below full sharing.
Proof of the exact partition and merge frontier
For a group G ⊆ [ M ] G\subseteq[M] G ⊆ [ M ] , write
A ^ G = 1 ∣ G ∣ ∑ a ∈ G A ^ a = A ‾ G + E ‾ G , E ∥ E ‾ G ∥ F 2 = V ∣ G ∣ . (60) \widehat{A}_G=\frac{1}{|G|}\sum_{a\in G}\widehat{A}_a=\overline{A}_G+\overline{E}_G,\qquad\mathbb{E}\lVert\overline{E}_G\rVert_F^2=\frac{V}{|G|}. \tag*{(60)} A G = ∣ G ∣ 1 a ∈ G ∑ A a = A G + E G , E ∥ E G ∥ F 2 = ∣ G ∣ V . (60)
The total expected risk contributed by roles in G G G is therefore
∑ a ∈ G E ∥ A ^ G − A a ∥ F 2 = ∑ a ∈ G ∥ A ‾ G − A a ∥ F 2 + ∣ G ∣ E ∥ ε ‾ G ∥ F 2 = ∑ a ∈ G ∥ A ‾ G − A a ∥ F 2 + V . (61) \begin{aligned}
\sum_{a\in G}\mathbb{E}\left\|\widehat{A}_G-A_a\right\|_F^2
=
\sum_{a\in G}\left\|\overline{A}_G-A_a\right\|_F^2
+
|G|\mathbb{E}\left\|\overline{\varepsilon}_G\right\|_F^2 \\
&=
\sum_{a\in G}\left\|\overline{A}_G-A_a\right\|_F^2+V. \tag*{(61)}
\end{aligned} a ∈ G ∑ E A G − A a F 2 = a ∈ G ∑ A G − A a F 2 + ∣ G ∣ E ∥ ε G ∥ F 2 = a ∈ G ∑ A G − A a F 2 + V . (61)
Summing over the K Π K\Pi K Π blocks and dividing by M M M proves Equation (11) .
For two disjoint groups G G G , H H H , the Hilbert-space variance decomposition gives
∑ a ∈ G ∪ H ∥ A a − A ‾ G ∪ H ∥ F 2 = ∑ a ∈ G ∥ A a − A ‾ G ∥ F 2 + ∑ a ∈ H ∥ A a − A ‾ H ∥ F 2 + ∣ G ∣ ∣ H ∣ ∣ G ∣ + ∣ H ∣ ∥ A ‾ G − A ‾ H ∥ F 2 . (62) \sum_{a\in G\cup H}\left\|A_a-\overline{A}_{G\cup H}\right\|_F^2
=
\sum_{a\in G}\left\|A_a-\overline{A}_G\right\|_F^2
+
\sum_{a\in H}\left\|A_a-\overline{A}_H\right\|_F^2
+
\frac{|G||H|}{|G|+|H|}
\left\|\overline{A}_G-\overline{A}_H\right\|_F^2. \tag*{(62)} a ∈ G ∪ H ∑ A a − A G ∪ H F 2 = a ∈ G ∑ A a − A G F 2 + a ∈ H ∑ A a − A H F 2 + ∣ G ∣ + ∣ H ∣ ∣ G ∣∣ H ∣ A G − A H F 2 . (62)
Merging increases within-group dispersion by the final term and decreases K Π V K\Pi V K Π V by exactly V V V , proving Equation (12) .
The merge separation has an exact sampling law under the additional assumption that all noise entries are mutually Gaussian with the stated independent-row covariance and σ > 0 \sigma>0 σ > 0 . For two pre-specified, disjoint, nonempty groups, let m = ∣ G ∣ m=|G| m = ∣ G ∣ , h = ∣ H ∣ h=|H| h = ∣ H ∣ , c = m h / ( m + h ) c=mh/(m+h) c = mh / ( m + h ) , q = σ 2 / n q=\sigma^2/n q = σ 2 / n , and define
W G , H = c ∥ A ‾ G − A ‾ H ∥ F 2 , W ^ G , H = c ∥ A ^ G − A ^ H ∥ F 2 . (63) W_{G,H}=c\left\|\overline{A}_G-\overline{A}_H\right\|_F^2,
\qquad
\widehat{W}_{G,H}=c\left\|\widehat{A}_G-\widehat{A}_H\right\|_F^2. \tag*{(63)} W G , H = c A G − A H F 2 , W G , H = c A G − A H F 2 . (63)
Because each coefficient of A ^ G − A ^ H \widehat{A}_G-\widehat{A}_H A G − A H has variance q ( 1 / m + 1 / h ) = q / c q(1/m+1/h)=q/c q ( 1/ m + 1/ h ) = q / c ,
W ^ G , H q ∼ χ D ′ 2 ( W G , H q ) , E W ^ G , H = W G , H + V . (64) \frac{\widehat{W}_{G,H}}{q}
\sim
\chi^{\prime2}_D\left(\frac{W_{G,H}}{q}\right),
\qquad
\mathbb{E}\widehat{W}_{G,H}=W_{G,H}+V. \tag*{(64)} q W G , H ∼ χ D ′2 ( q W G , H ) , E W G , H = W G , H + V . (64)
Thus W ^ G , H − V \widehat{W}_{G,H}-V W G , H − V is unbiased for the functional separation governing the merge decision. Exact noncentral-chi-square confidence bounds can be inverted for a pre-specified pair; searching many candidate merges requires a familywise or validation correction.
The heterogeneity is observable up to a known noise correction
The exact model also gives a direct diagnostic for B B B .
Proposition 10 (Noisy role dispersion). Assume M ≥ 2 M\geq2 M ≥ 2 , σ > 0 \sigma>0 σ > 0 , and independent Gaussian observation noise for the distributional identity below. Let q = σ 2 / n q=\sigma^2/n q = σ 2 / n and
B ^ r a w = 1 M ∑ a = 1 M ∥ A ^ a − A ^ p o o l ∥ F 2 . (65) \widehat{B}_{\mathrm{raw}}
=
\frac{1}{M}\sum_{a=1}^{M}
\left\|\widehat{A}_a-\widehat{A}_{\mathrm{pool}}\right\|_F^2. \tag*{(65)} B raw = M 1 a = 1 ∑ M A a − A pool F 2 . (65) Then
E B ^ r a w = B + C , B ^ u n b = B ^ r a w − C (66) \mathbb{E}\widehat{B}_{\mathrm{raw}}=B+C,
\qquad
\widehat{B}_{\mathrm{unb}}=\widehat{B}_{\mathrm{raw}}-C \tag*{(66)} E B raw = B + C , B unb = B raw − C (66) is unbiased for B B B . More precisely,
M B ^ r a w q ∼ χ D ( M − 1 ) ′ 2 ( M B q ) . (67) \frac{M\widehat{B}_{\mathrm{raw}}}{q}
\sim
\chi^{\prime2}_{D(M-1)}
\left(\frac{MB}{q}\right). \tag*{(67)} q M B raw ∼ χ D ( M − 1 ) ′2 ( q M B ) . (67)
Proof. For one matrix coefficient, stack the M M M role estimates and let P = I M − 11 ⊤ / M P=I_M-\mathbf{1}\mathbf{1}^{\top}/M P = I M − 1 1 ⊤ / M . The centered estimate is P ( a + ε ) P(a+\varepsilon) P ( a + ε ) with ε ∼ N ( 0 , q I M ) \varepsilon\sim\mathcal{N}(0,qI_M) ε ∼ N ( 0 , q I M ) . In an orthonormal basis of the rank-( M − 1 ) (M-1) ( M − 1 ) image of P P P , its squared norm divided by q q q is noncentral chi-squared with noncentrality ∥ P a ∥ 2 / q \|Pa\|^2/q ∥ P a ∥ 2 / q . Summing the independent contributions of all D D D coefficients gives Equation (67) ; its mean gives Equation (66) because D q ( M − 1 ) / M = C Dq(M-1)/M=C D q ( M − 1 ) / M = C .
The expectation and unbiasedness formulas need only the second-moment assumptions; the noncentral chi-square law additionally needs Gaussian noise, q > 0 q > 0 q > 0 , and M ≥ 2 M \ge2 M ≥ 2 . When σ = 0 \sigma= 0 σ = 0 , dispersion is observed without noise; when M = 1 M = 1 M = 1 , both dispersions and B , C B, C B , C are zero and no chi-square expression is needed. A noncentral chi-square variable is a sum of squared independent unit-variance normal variables with possibly nonzero means; its degrees of freedom count summands and its noncentrality is the sum of squared means. Thus the same dispersion that determines negative transfer is estimable from role-specific fits after subtracting estimation noise. The oracle coefficient can be written λ ⋆ = C / ( B + C ) \lambda^\star= C/(B + C) λ ⋆ = C / ( B + C ) , suggesting the clipped plug-in rule λ ^ = 1 \widehat{\lambda} = 1 λ = 1 when B ^ r a w = 0 \widehat{B}_{\mathrm{raw}} = 0 B raw = 0 and λ ^ = min { 1 , C / B ^ r a w } \widehat{\lambda} = \min\{1, C/\widehat{B}_{\mathrm{raw}}\} λ = min { 1 , C / B raw } otherwise, when the exact assumptions are approximately credible. It is not justified by substituting λ ^ \widehat{\lambda} λ into the fixed-coefficient risk formula. The low-dimensional Gaussian counterexample below and the contrast-space result give the dimension-dependent conclusions. Section D.6 supplies the classical known-noise Gaussian guarantee when the contrast dimension D ( M − 1 ) D(M - 1) D ( M − 1 ) is at least four.
A same-data plug-in can worsen risk inside the exact model
Take M = 2 M = 2 M = 2 , D = 1 D = 1 D = 1 , q = 1 q = 1 q = 1 , and write Y i = θ i + ε i Y_i = \theta_i + \varepsilon_i Y i = θ i + ε i with independent ε i ∼ N ( 0 , 1 ) \varepsilon_i \sim\mathcal{N}(0, 1) ε i ∼ N ( 0 , 1 ) . In the orthonormal contrast coordinate, Z = ( Y 1 − Y 2 ) / 2 ∼ N ( μ , 1 ) Z = (Y_1 - Y_2)/\sqrt{2} \sim\mathcal{N}(\mu, 1) Z = ( Y 1 − Y 2 ) / 2 ∼ N ( μ , 1 ) , where μ = ( θ 1 − θ 2 ) / 2 \mu= (\theta_1 - \theta_2)/\sqrt{2} μ = ( θ 1 − θ 2 ) / 2 . Since C = 1 / 2 C = 1/2 C = 1/2 and B ^ r a w = Z 2 / 2 \widehat{B}_{\mathrm{raw}} = Z^2/2 B raw = Z 2 /2 , the plug-in has contrast estimate
h ( Z ) = { 0 , ∣ Z ∣ ≤ 1 , Z − 1 / Z , ∣ Z ∣ > 1. (68) h(Z) =
\begin{cases}
0, & |Z| \le1, \\
Z - 1/Z, & |Z| > 1.
\end{cases} \tag*{(68)} h ( Z ) = { 0 , Z − 1/ Z , ∣ Z ∣ ≤ 1 , ∣ Z ∣ > 1. (68)
The common-mean coordinate is unchanged. Put u ( z ) = h ( z ) − z u(z) = h(z) - z u ( z ) = h ( z ) − z ; this function is continuous and piecewise differentiable. Gaussian integration by parts, with the boundary terms cancelling at ± 1 \pm1 ± 1 , gives
E ( h ( Z ) − μ ) 2 − 1 = E { u ( Z ) 2 + 2 u ′ ( Z ) } (69) \mathbb{E}(h(Z) - \mu)^2 - 1 = \mathbb{E}\{u(Z)^2 + 2u'(Z)\} \tag*{(69)} E ( h ( Z ) − μ ) 2 − 1 = E { u ( Z ) 2 + 2 u ′ ( Z )} (69)
= E [ ( Z 2 − 2 ) 1 { ∣ Z ∣ ≤ 1 } + 3 Z 2 1 { ∣ Z ∣ > 1 } ] . (70) = \mathbb{E}\left[(Z^2 - 2)\mathbf{1}_{\{|Z|\le1\}} + \frac{3}{Z^2}\mathbf{1}_{\{|Z|>1\}}\right]. \tag*{(70)} = E [ ( Z 2 − 2 ) 1 { ∣ Z ∣ ≤ 1 } + Z 2 3 1 { ∣ Z ∣ > 1 } ] . (70)
For completeness, the integration identity follows from the normal density derivative ϕ ′ ( z − μ ) = − ( z − μ ) ϕ ( z − μ ) \phi'(z - \mu) = -(z - \mu)\phi(z - \mu) ϕ ′ ( z − μ ) = − ( z − μ ) ϕ ( z − μ ) , so E [ ( Z − μ ) u ( Z ) ] = E u ′ ( Z ) \mathbb{E}[(Z - \mu)u(Z)] = \mathbb{E}u'(Z) E [( Z − μ ) u ( Z )] = E u ′ ( Z ) ; u u u and its derivative are integrable here. At μ = 6 \mu= 6 μ = 6 , Chebyshev’s inequality gives P ( 4 ≤ Z ≤ 8 ) ≥ 3 / 4 \mathbb{P}(4 \le Z \le8) \ge3/4 P ( 4 ≤ Z ≤ 8 ) ≥ 3/4 , whereas the normal fourth moment gives P ( ∣ Z ∣ ≤ 1 ) ≤ 3 / 5 \mathbb{P}(|Z| \le1) \le3/5 P ( ∣ Z ∣ ≤ 1 ) ≤ 3/5 . Consequently,
E ( h ( Z ) − 6 ) 2 − 1 ≥ 3 64 3 4 − 6 625 = 4089 160000 > 0. (71) \mathbb{E}(h(Z) - 6)^2 - 1 \ge\frac{3}{64}\frac{3}{4} - \frac{6}{625} = \frac{4089}{160000} > 0. \tag*{(71)} E ( h ( Z ) − 6 ) 2 − 1 ≥ 64 3 4 3 − 625 6 = 160000 4089 > 0. (71)
The average risk of the two roles increases by half this contrast-risk difference. Unbiased estimation of B B B in Proposition 10 does not imply risk domination after a nonlinear same-data plug-in.
Independent validation selects reuse strength with an oracle inequality
Let Λ ⊂ [ 0 , 1 ] \Lambda\subset[0, 1] Λ ⊂ [ 0 , 1 ] be a finite grid. Fit f ^ λ \widehat{f}_{\lambda} f λ using training data, and evaluate every fitted candidate on an independent weighted validation set with loss in [ 0 , 1 ] [0, 1] [ 0 , 1 ] and effective size N v a l e f f N_{\mathrm{val}}^{\mathrm{eff}} N val eff . Let λ ^ \widehat{\lambda} λ minimize validation loss. Here the weighted validation convention is the one made explicit for finite candidates in Theorem 48 : conditional on the training information G \mathcal{G} G , validation records i = 1 , … , N i = 1,\ldots,N i = 1 , … , N are independent, and nonnegative weights v i v_i v i summing to one are fixed before observing their outcomes. Write ℓ i ( f ^ λ ) ∈ [ 0 , 1 ] \ell_i(\widehat{f}_{\lambda}) \in[0, 1] ℓ i ( f λ ) ∈ [ 0 , 1 ] for the loss on record i i i . Then R ^ ( f ^ λ ) = ∑ i v i ℓ i ( f ^ λ ) \widehat{R}(\widehat{f}_{\lambda}) = \sum_i v_i\ell_i(\widehat{f}_{\lambda}) R ( f λ ) = ∑ i v i ℓ i ( f λ ) , R ( f ^ λ ) = ∑ i v i E [ ℓ i ( f ^ λ ) ∣ G ] R(\widehat{f}_{\lambda}) = \sum_i v_i\mathbb{E}[\ell_i(\widehat{f}_{\lambda}) \mid\mathcal{G}] R ( f λ ) = ∑ i v i E [ ℓ i ( f λ ) ∣ G ] , and N v a l e f f = ( ∑ i v i 2 ) − 1 N_{\mathrm{val}}^{\mathrm{eff}} = (\sum_i v_i^2)^{-1} N val eff = ( ∑ i v i 2 ) − 1 . The selector minimizes this same weighted empirical risk. Nonidentically distributed records retain this declared weighted target; losses of different candidates on the same record need not be independent.
Proposition 11 (Validation-selected partial reuse). Conditional on the training sample, with probability at least 1 − δ 1 - \delta 1 − δ over validation data,
R ( f ^ λ ^ ) ≤ min λ ∈ Λ R ( f ^ λ ) + 2 log ( 2 ∣ Λ ∣ / δ ) 2 N v a l e f f . (72) R(\widehat{f}_{\widehat{\lambda}}) \le\min_{\lambda\in\Lambda} R(\widehat{f}_{\lambda}) + 2\sqrt{\frac{\log(2|\Lambda|/\delta)}{2N_{\mathrm{val}}^{\mathrm{eff}}}}. \tag*{(72)} R ( f λ ) ≤ λ ∈ Λ min R ( f λ ) + 2 2 N val eff log ( 2∣Λ∣/ δ ) . (72)
Proof. Weighted Hoeffding and a union bound give one event on which every validation risk is within the displayed radius of its population risk. Apply the standard two-deviation ERM argument on that event.
This result provides a statistically valid way to choose among fitted candidates for the stated bounded validation loss, without estimating B B B parametrically. The guarantee evaluates the already fitted candidates using independent validation data. Theorem 48 gives a separate finite-family guarantee under explicit tail assumptions, including a Gaussian quadratic-loss specialization.
Classical same-data shrinkage in the contrast space
This calculation explains a dimension-specific exception to the preceding warning; it is an application of classical Baranchik–Stein shrinkage [1 , 36 ] . A directly comparable risk identity and Baranchik’s sufficient conditions are reproduced in Maruyama & Takemura (2024, § 1 , equations (1.5)–(1.6)). Shrinkage toward a fixed subspace, a common mean, or data-selected cluster attractors is also established background [35 ] ; the calculation here fixes the common-mean subspace.
Retain the balanced fixed-design model of Theorem 1 , with M ≥ 2 M \ge2 M ≥ 2 , D = p d D = pd D = p d , mutually Gaussian observation noise, and known q = σ 2 / n > 0 q = \sigma^2/n > 0 q = σ 2 / n > 0 . Stack the M M M vectorized role estimates as Y = θ + ζ ∈ R M D Y = \theta+ \zeta\in\mathbb{R}^{MD} Y = θ + ζ ∈ R M D , so ζ ∼ N ( 0 , q I M D ) \zeta\sim\mathcal{N}(0, qI_{MD}) ζ ∼ N ( 0 , q I M D ) . Define
P = 11 T M ⊗ I D , Q = I M D − P , k = D ( M − 1 ) . P = \frac{\mathbf{1}\mathbf{1}^{\mathsf{T}}}{M} \otimes I_D,\qquad Q = I_{MD} - P,\qquad k = D(M-1). P = M 1 1 T ⊗ I D , Q = I M D − P , k = D ( M − 1 ) .
Here P P P replaces every role vector by their common average, and Q Q Q retains their differences. The integer k k k is the dimension of this contrast space , not the number of selected experts. Choose a fixed matrix U ∈ R M D × k U \in\mathbb{R}^{MD \times k} U ∈ R M D × k whose columns are an orthonormal basis of that space: U T U = I k U^{\mathsf{T}}U = I_k U T U = I k , U U T = Q UU^{\mathsf{T}} = Q U U T = Q . Then
Z = U T Y ∼ N ( μ , q I k ) , μ = U T θ , M B ^ r a w = ∥ Z ∥ 2 , M C = k q . Z = U^{\mathsf{T}}Y \sim\mathcal{N}(\mu, qI_k),\qquad\mu= U^{\mathsf{T}}\theta,\qquad M\widehat{B}_{\mathrm{raw}} = \lVert Z\rVert^2,\qquad MC = kq. Z = U T Y ∼ N ( μ , q I k ) , μ = U T θ , M B raw = ∥ Z ∥ 2 , M C = k q .
The same-data coefficient defined after Proposition 10 leaves P Y PY P Y unchanged and replaces Z Z Z by
h ( Z ) = ( 1 − k q ∥ Z ∥ 2 ) + Z , h ( 0 ) = 0 , ( t ) + = max { t , 0 } . h(Z) = \left(1-\frac{kq}{\lVert Z\rVert^2}\right)_{+}Z,\qquad h(0)=0,\qquad(t)_{+}=\max\{t,0\}. h ( Z ) = ( 1 − ∥ Z ∥ 2 k q ) + Z , h ( 0 ) = 0 , ( t ) + = max { t , 0 } .
Thus the fitted stacked vector is P Y + U h ( Z ) PY + Uh(Z) P Y + U h ( Z ) . Its role-averaged excess prediction risk satisfies
R p l u g − R s e p = 1 M ( E ∥ h ( Z ) − μ ∥ 2 − k q ) < 0 for every fixed θ when k ≥ 4. \mathcal{R}_{\mathrm{plug}}-\mathcal{R}_{\mathrm{sep}}=\frac{1}{M}\left(\mathbb{E}\lVert h(Z)-\mu\rVert^2-kq\right)<0\qquad\text{for every fixed }\theta\text{ when }k\ge4. R plug − R sep = M 1 ( E ∥ h ( Z ) − μ ∥ 2 − k q ) < 0 for every fixed θ when k ≥ 4.
Direct verification of the classical specialization. Let u ( z ) = h ( z ) − z u(z)=h(z)-z u ( z ) = h ( z ) − z and s = ∥ z ∥ 2 s=\lVert z\rVert^2 s = ∥ z ∥ 2 . The divergence div u = ∑ i = 1 k ∂ u i / ∂ z i \operatorname{div}u=\sum_{i=1}^{k}\partial u_i/\partial z_i div u = ∑ i = 1 k ∂ u i / ∂ z i is the sum of coordinate derivatives. Away from the sphere s = k q s=kq s = k q ,
s ≤ k q s > k q u ( z ) − z − k q z / s div u ( z ) − k − k q ( k − 2 ) / s ∥ u ( z ) ∥ 2 s k 2 q 2 / s \begin{array}{c|cc}
& s\le kq & s>kq\\
u(z) & -z & -kqz/s\\
\operatorname{div}u(z) & -k & -kq(k-2)/s\\
\lVert u(z)\rVert^2 & s & k^2q^2/s
\end{array} u ( z ) div u ( z ) ∥ u ( z ) ∥ 2 s ≤ k q − z − k s s > k q − k q z / s − k q ( k − 2 ) / s k 2 q 2 / s
The two formulas for u u u agree on the sphere. Both pieces have bounded first derivatives on their domains, so u u u is weakly differentiable (its integration-by-parts derivative has no surface jump term). It and its derivatives are Gaussian-integrable. Coordinatewise Gaussian integration by parts gives
E [ ( Z i − μ i ) u i ( Z ) ] = q E [ ∂ i u i ( Z ) ] . \mathbb{E}[(Z_i-\mu_i)u_i(Z)]=q\mathbb{E}[\partial_i u_i(Z)]. E [( Z i − μ i ) u i ( Z )] = q E [ ∂ i u i ( Z )] .
Expanding h ( Z ) − μ = Z − μ + u ( Z ) h(Z)-\mu=Z-\mu+u(Z) h ( Z ) − μ = Z − μ + u ( Z ) and summing therefore gives
E ∥ h ( Z ) − μ ∥ 2 − k q = E [ ( s − 2 k q ) 1 { s ≤ k q } + k ( 4 − k ) q 2 s 1 { s > k q } ] . \mathbb{E}\lVert h(Z)-\mu\rVert^2-kq
=
\mathbb{E}\left[\left(s-2kq\right)\mathbf{1}_{\{s\le kq\}}+\frac{k(4-k)q^2}{s}\mathbf{1}_{\{s>kq\}}\right]. E ∥ h ( Z ) − μ ∥ 2 − k q = E [ ( s − 2 k q ) 1 { s ≤ k q } + s k ( 4 − k ) q 2 1 { s > k q } ] .
For k ≥ 4 k\ge4 k ≥ 4 the outside term is nonpositive, and the inside term is at most − k q -kq − k q . Every nondegenerate Gaussian distribution assigns positive probability to s ≤ k q s\le kq s ≤ k q , so the expectation is strictly negative, also at k = 4 k=4 k = 4 . Finally P ζ P\zeta P ζ and U ( h ( Z ) − μ ) U(h(Z)-\mu) U ( h ( Z ) − μ ) lie in orthogonal subspaces for every realization. The common-mean risk is unchanged, and division by M M M converts the contrast-risk difference to the original role-averaged risk.
Relation to classical shrinkage. After standardizing X = Z / q X = Z/\sqrt{q} X = Z / q , the rule is ( 1 − ϕ ( w ) / w ) X (1 - \phi(w)/w)X ( 1 − ϕ ( w ) / w ) X with w = ∥ X ∥ 2 w = \lVert X\rVert^2 w = ∥ X ∥ 2 and ϕ ( w ) = min { w , k } \phi(w) = \min\{w,k\} ϕ ( w ) = min { w , k } . In the cited Stein risk identity, substitution gives w − 2 k w - 2k w − 2 k on w ≤ k w \le k w ≤ k and k ( 4 − k ) / w k(4-k)/w k ( 4 − k ) / w otherwise. It also satisfies Baranchik’s monotonicity and range conditions 0 ≤ ϕ ( w ) ≤ 2 ( k − 2 ) 0 \le\phi(w) \le2(k-2) 0 ≤ ϕ ( w ) ≤ 2 ( k − 2 ) when k ≥ 4 k \ge4 k ≥ 4 . The usual positive-part James–Stein rule uses ( k − 2 ) q (k-2)q ( k − 2 ) q , not k q kq k q , and already dominates the unshrunk estimate for k ≥ 3 k \ge3 k ≥ 3 [35 ] . That is a different estimator; the threshold four here is not a universal dimension barrier.
The strict improvement is pointwise in the fixed target, not a claimed target-uniform positive improvement amount. The low-dimensional counterexample in Equation (71) remains valid. The main non-minimaxity result of Maruyama & Takemura [25 ] concerns rules whose shrinkage is exactly zero outside a finite ball; this is not the present rule, whose ϕ ( w ) \phi(w) ϕ ( w ) stays equal to k k k . Only their introductory classical risk identity is used here.
End-to-End Route-Plan Certificates
Overview. We track how aligned implementation error and functional routing error propagate to the endpoint. The certificate requires explicit role-to-rollout transfer conditions. A separate linear calculation retains error directions and cross-time covariance rather than replacing them by scalar sensitivity bounds. Sections E.1–E.2 establish propagation and the conditional certificate; Sections E.3–E.5 address functional route cost, terminal decision margins, and the separate covariance identity.
Pathwise propagation with random intended programs
All random variables are defined on one coupled evaluation probability space. The intended canonical program Γ j \Gamma_j Γ j may depend on the reference history and exogenous randomness; it is never moved outside expectation. The reference and learned recursions are
S j + 1 = T j , Γ j ( S j ) , S ~ j + 1 m = T ~ j , Γ ~ j m ( S ~ j m ) . (73) S_{j+1}=T_{j,\Gamma_j}(S_j), \qquad\widetilde{S}_{j+1}^{m}=\widetilde{T}_{j,\widetilde{\Gamma}_j^{m}}(\widetilde{S}_j^{m}). \tag*{(73)} S j + 1 = T j , Γ j ( S j ) , S j + 1 m = T j , Γ j m ( S j m ) . (73)
The implementation ι m ( Γ j ) \iota^{m}(\Gamma_j) ι m ( Γ j ) is the model-m m m executable plan aligned with the intended canonical program. Assume T j , γ T_{j,\gamma} T j , γ is L j L_j L j -Lipschitz on the reached region uniformly in admissible canonical γ \gamma γ . Define e j m = d ( S ~ j m , S j ) e_j^{m}=d(\widetilde{S}_j^{m},S_j) e j m = d ( S j m , S j ) . Inserting the aligned learned transition and the reference map at the learned state gives, pathwise,
e j + 1 m ≤ d ( T ~ j , Γ ~ j m ( S ~ j m ) , T ~ j , ι m ( Γ j ) ( S ~ j m ) ) + d ( T ~ j , ι m ( Γ j ) ( S ~ j m ) , T j , Γ j ( S ~ j m ) ) + d ( T j , Γ j ( S ~ j m ) , T j , Γ j ( S j ) ) ≤ r j m + a j m + L j e j m . (74) \begin{aligned}
e_{j+1}^{m}
&\le d\left(\widetilde{T}_{j,\widetilde{\Gamma}_j^{m}}(\widetilde{S}_j^{m}),\widetilde{T}_{j,\iota^{m}(\Gamma_j)}(\widetilde{S}_j^{m})\right) \\
&\quad+d\left(\widetilde{T}_{j,\iota^{m}(\Gamma_j)}(\widetilde{S}_j^{m}),T_{j,\Gamma_j}(\widetilde{S}_j^{m})\right)+d\left(T_{j,\Gamma_j}(\widetilde{S}_j^{m}),T_{j,\Gamma_j}(S_j)\right) \\
&\le r_j^{m}+a_j^{m}+L_j e_j^{m}. \tag*{(74)}
\end{aligned} e j + 1 m ≤ d ( T j , Γ j m ( S j m ) , T j , ι m ( Γ j ) ( S j m ) ) + d ( T j , ι m ( Γ j ) ( S j m ) , T j , Γ j ( S j m ) ) + d ( T j , Γ j ( S j m ) , T j , Γ j ( S j ) ) ≤ r j m + a j m + L j e j m . (74)
Induction yields
e H m ≤ ( ∏ r = 0 H − 1 L r ) e 0 m + ∑ j = 0 H − 1 A j ( a j m + r j m ) , A j = ∏ r = j + 1 H − 1 L r . (75) e_H^{m}\le\left(\prod_{r=0}^{H-1}L_r\right)e_0^{m}+\sum_{j=0}^{H-1}A_j(a_j^{m}+r_j^{m}), \qquad A_j=\prod_{r=j+1}^{H-1}L_r. \tag*{(75)} e H m ≤ ( r = 0 ∏ H − 1 L r ) e 0 m + j = 0 ∑ H − 1 A j ( a j m + r j m ) , A j = r = j + 1 ∏ H − 1 L r . (75)
No independence between route errors and states is used. The intended program is coupled to the reference trajectory; the theorem bounds imitation of that coupled reference computation rather than regret against a state-dependent policy recomputed at the learned state. If sensitivities are random rather than deterministic upper bounds, their products remain inside expectation unless additional moment assumptions justify factorization.
Expanded terminal certificate and probability bookkeeping
The compact statement of Theorem 4 uses the following complete active-support convention:
w a = ∑ j = 0 H − 1 A j w j , a , J e + = { a ∈ J e : w a > 0 } , W e = ∑ a ∈ J e + w a , π e , a = w a W e ( a ∈ J e + ) . (76) \begin{aligned}
w_a&=\sum_{j=0}^{H-1}A_jw_{j,a}, \qquad\mathcal{J}_e^+=\{a\in\mathcal{J}_e:w_a>0\},\\
W_e&=\sum_{a\in\mathcal{J}_e^+}w_a,\qquad\pi_{e,a}=\frac{w_a}{W_e}\quad(a\in\mathcal{J}_e^+). \tag*{(76)}
\end{aligned} w a W e = j = 0 ∑ H − 1 A j w j , a , J e + = { a ∈ J e : w a > 0 } , = a ∈ J e + ∑ w a , π e , a = W e w a ( a ∈ J e + ) . (76)
Here w a w_a w a is role a a a ’s total terminal weight, J e + \mathcal{J}_e^+ J e + retains its positive-weight roles, W e W_e W e is their group total, and π e , a \pi_{e,a} π e , a gives their relative importance. Only groups with W e > 0 W_e>0 W e > 0 enter the certificate. For notational economy, R s e p , e ∗ R^*_{\mathrm{sep},e} R sep , e ∗ , B e r e u s e B_e^{\mathrm{reuse}} B e reuse , and the two radii below denote the quantities from § 4 recomputed on J e + \mathcal{J}_e^+ J e + with weights π e , a \pi_{e,a} π e , a , and the empirical objectives use the same active support. A zero-weight role can be omitted from this comparison; if its data nevertheless influence a shared fitted expert, that influence must be represented by a positive training/certificate weight or by an additional algorithmic-deviation term. Thus these weights incorporate downstream amplification.
Expanded conditions and bounds. Let δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let the initial states agree, and let terminal utility (a scalar score) U : S → R U:\mathcal{S}\to\mathbb{R} U : S → R be L U L_U L U -Lipschitz. On every active group J e + \mathcal{J}_e^+ J e + assume all hypotheses of Theorem 8 : a fixed finite class, losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] , at least one observation per active role, and mutually independent role observations under the stated sampling model. The classes, losses, distributions, and performance weights are fixed independently of these observations. For each group e e e with W e > 0 W_e>0 W e > 0 , choose δ e > 0 \delta_e>0 δ e > 0 with ∑ e δ e ≤ δ \sum_e\delta_e\le\delta ∑ e δ e ≤ δ , and define ϵ s h , e ( δ e ) \epsilon_{\mathrm{sh},e}(\delta_e) ϵ sh , e ( δ e ) and ϵ s e p , e ( δ e ) \epsilon_{\mathrm{sep},e}(\delta_e) ϵ sep , e ( δ e ) by Equation (43) and Equation (44) , respectively, with δ \delta δ replaced by δ e \delta_e δ e . Assume the group estimators minimize the performance-weighted empirical risks induced by Equation (76) . The two comparators are fitted from the same role samples; if they are refitted on independent samples, the confidence allocation must also cover both fitted datasets. Under Assumption 3, with probability at least 1 − δ 1-\delta 1 − δ over the expert-estimation samples,
E e v a l ∣ U ( S ^ H r e u s e ) − U ( S H ) ∣ ≤ L U U r e u s e , (77) \mathbb{E}_{\mathrm{eval}}\left|U(\widehat{S}^{\mathrm{reuse}}_H)-U(S_H)\right|\le L_U U_{\mathrm{reuse}}, \tag*{(77)} E eval U ( S H reuse ) − U ( S H ) ≤ L U U reuse , (77)
U r e u s e = ∑ j A j ( τ j r e u s e + ρ j r e u s e ) + ∑ e W e ( R s e p , e ∗ + B e r e u s e + 2 ϵ s h , e ( δ e ) ) , (78) U_{\mathrm{reuse}}=\sum_j A_j\left(\tau_j^{\mathrm{reuse}}+\rho_j^{\mathrm{reuse}}\right)+\sum_e W_e\left(R^*_{\mathrm{sep},e}+B_e^{\mathrm{reuse}}+2\epsilon_{\mathrm{sh},e}(\delta_e)\right), \tag*{(78)} U reuse = j ∑ A j ( τ j reuse + ρ j reuse ) + e ∑ W e ( R sep , e ∗ + B e reuse + 2 ϵ sh , e ( δ e ) ) , (78)
E e v a l ∣ U ( S ^ H s e p ) − U ( S H ) ∣ ≤ L U U s e p , (79) \mathbb{E}_{\mathrm{eval}}\left|U(\widehat{S}^{\mathrm{sep}}_H)-U(S_H)\right|\le L_U U_{\mathrm{sep}}, \tag*{(79)} E eval U ( S H sep ) − U ( S H ) ≤ L U U sep , (79)
U s e p = ∑ j A j ( τ j s e p + ρ j s e p ) + ∑ e W e ( R s e p , e ∗ + 2 ϵ s e p , e ( δ e ) ) . (80) U_{\mathrm{sep}}=\sum_j A_j\left(\tau_j^{\mathrm{sep}}+\rho_j^{\mathrm{sep}}\right)+\sum_e W_e\left(R^*_{\mathrm{sep},e}+2\epsilon_{\mathrm{sep},e}(\delta_e)\right). \tag*{(80)} U sep = j ∑ A j ( τ j sep + ρ j sep ) + e ∑ W e ( R sep , e ∗ + 2 ϵ sep , e ( δ e ) ) . (80)
The symbols ϵ r e u s e , e \epsilon_{\mathrm{reuse},e} ϵ reuse , e and ϵ s e p , e \epsilon_{\mathrm{sep},e} ϵ sep , e in Equation (22) are respectively the shared and separate radii at the same group budget δ e \delta_e δ e . Its indicator selects the reuse bias only for the shared model, so substituting either value of m m m gives exactly the two expanded bounds above. The common oracle cancels in Equation (23) only because both comparators use these same groups, weights, losses, and evaluation populations. The utility-bound difference is L U P R G H L_U\mathrm{PRG}_H L U PRG H ; strict ordering requires L U > 0 L_U>0 L U > 0 . For L U = 0 L_U=0 L U = 0 both utility bounds vanish.
Proof of Theorem 4. For each active group choose δ e > 0 \delta_e>0 δ e > 0 with ∑ e δ e ≤ δ \sum_e\delta_e\le\delta ∑ e δ e ≤ δ and condition on the simultaneous training event obtained by applying Theorem 8 with failure probability δ e \delta_e δ e to group e e e . Under Assumption 3, common initial states, and ρ j m = E r j m \rho_j^m=\mathbb{E}r_j^m ρ j m = E r j m ,
E R H m ≤ ∑ j A j ( τ j m + ρ j m ) + ∑ a ( ∑ j A j ω j , a ) R a ( f ^ J m ( a ) m ) . (81) \mathbb{E}R_H^m\le\sum_j A_j(\tau_j^m+\rho_j^m)+\sum_a\left(\sum_j A_j\omega_{j,a}\right)R_a\left(\widehat{f}_{\mathcal{J}_m(a)}^m\right). \tag*{(81)} E R H m ≤ j ∑ A j ( τ j m + ρ j m ) + a ∑ ( j ∑ A j ω j , a ) R a ( f J m ( a ) m ) . (81)
The coefficient of role a a a is w a w_a w a . Restrict each reuse group to its active support J e + \mathcal{J}_e^+ J e + from Equation (76) ; then writing w a = W e π e , a w_a=W_e\pi_{e,a} w a = W e π e , a gives
∑ a ∈ J e + w a R a ( f ) = W e ∑ a ∈ J e + π e , a R a ( f ) . (82) \sum_{a\in\mathcal{J}_e^+}w_aR_a(f)=W_e\sum_{a\in\mathcal{J}_e^+}\pi_{e,a}R_a(f). \tag*{(82)} a ∈ J e + ∑ w a R a ( f ) = W e a ∈ J e + ∑ π e , a R a ( f ) . (82)
Because the estimators use these fixed performance weights, the shared part of Theorem 8 gives the reuse term in Equation (78) ; its role-specific part gives Equation (80) . Lipschitz continuity of terminal utility multiplies the state bound by L U L_U L U . Subtracting the two displayed right-hand sides cancels only the genuinely common oracle terms and yields L U P R G H L_U\mathrm{PRG}_H L U PRG H .
If comparator m m m requires model-specific weights ω j , a m \omega_{j,a}^m ω j , a m , define w a m = ∑ j A j ω j , a m w_a^m=\sum_j A_j\omega_{j,a}^m w a m = ∑ j A j ω j , a m and construct each certificate separately. Both remain valid, but the oracle and estimation terms generally have different coefficients and the compact expression Equation (23) is unavailable. This is the appropriate treatment when rollout occupancies differ and no common domination envelope is credible. Likewise, if event lifts or the surrounding assembly are learned from the expert-estimation data, the finite-sample event must be replaced by a uniform bound for the joint class or by sample splitting.
Functional route cost
For a finite canonical plan space and a finite model-specific executable plan space, define
C j m ( γ , γ ′ ) = P ( Γ j = γ , Γ ^ j m = γ ′ ) , (83) C_j^m(\gamma,\gamma') = \mathbb{P}(\Gamma_j = \gamma,\widehat{\Gamma}_j^m = \gamma'), \tag*{(83)} C j m ( γ , γ ′ ) = P ( Γ j = γ , Γ j m = γ ′ ) , (83)
D j m ( γ , γ ′ ) = E [ d ( T ^ j m ( S ^ j m ) , T ^ j , j m m ( γ ) m ( S ^ j m ) ) ∣ Γ j = γ , Γ ^ j m = γ ′ ] , (84) D_j^m(\gamma,\gamma') = \mathbb{E}\left[d\left(\widehat{T}_j^m(\widehat{S}_j^m),\widehat{T}_{j,j_m^m(\gamma)}^m(\widehat{S}_j^m)\right)\mid\Gamma_j = \gamma,\widehat{\Gamma}_j^m = \gamma'\right], \tag*{(84)} D j m ( γ , γ ′ ) = E [ d ( T j m ( S j m ) , T j , j m m ( γ ) m ( S j m ) ) ∣ Γ j = γ , Γ j m = γ ′ ] , (84)
with D j m ( γ , γ ′ ) = 0 D_j^m(\gamma,\gamma')=0 D j m ( γ , γ ′ ) = 0 when the conditioning event has zero probability. Then
ρ j m = ∑ γ , γ ′ C j m ( γ , γ ′ ) D j m ( γ , γ ′ ) . (85) \rho_j^m = \sum_{\gamma,\gamma'} C_j^m(\gamma,\gamma')D_j^m(\gamma,\gamma'). \tag*{(85)} ρ j m = γ , γ ′ ∑ C j m ( γ , γ ′ ) D j m ( γ , γ ′ ) . (85)
This quantity distinguishes harmless index substitutions from rare but severe functional confusions. Index accuracy records only the off-alignment mass and is meaningful only after the canonical-to-model alignment has been specified.
For an additive mixture, a direct gate bound is available.
Lemma 12 (Equal-mass functional gate discrepancy). Let v 1 , … , v K v_1,\ldots,v_K v 1 , … , v K lie in a normed vector space and let g , g ′ g,g' g , g ′ have equal total mass. Then
∥ ∑ e ( g e − g e ′ ) v e ∥ ≤ 1 2 ( max e , e ′ ∥ v e − v e ′ ∥ ) ∥ g − g ′ ∥ 1 . (86) \left\|\sum_e(g_e-g'_e)v_e\right\| \leq\frac{1}{2}\left(\max_{e,e'}\|v_e-v_{e'}\|\right)\|g-g'\|_1. \tag*{(86)} e ∑ ( g e − g e ′ ) v e ≤ 2 1 ( e , e ′ max ∥ v e − v e ′ ∥ ) ∥ g − g ′ ∥ 1 . (86) The constant 1 / 2 1/2 1/2 is sharp.
Proof. The positive and negative parts of g − g ′ g-g' g − g ′ have common mass ∥ g − g ′ ∥ 1 / 2 \|g-g'\|_1/2 ∥ g − g ′ ∥ 1 /2 . Normalize them to two convex combinations of the v e v_e v e ; their distance is at most the diameter. Equality is attained by two points and disjoint one-hot gates.
The lemma applies at one common expert input and with the same expert-output collection. Different states or dispatch-dependent expert inputs require additional state or dispatch sensitivity terms.
Terminal decisions with a margin
Discrete accuracy is not Lipschitz. It can nevertheless be controlled under a terminal margin when the same deterministic readout g g g is applied to the reference and learned terminal states. Let z = g ( S H ) z=g(S_H) z = g ( S H ) , suppose g g g is L g L_g L g -Lipschitz from d d d to ℓ ∞ \ell_\infty ℓ ∞ , and let the reference top-two gap be γ > 0 \gamma>0 γ > 0 . For L g > 0 L_g>0 L g > 0 , the learned argmax is unchanged whenever the pathwise right-hand side of Equation (75) is strictly less than γ / ( 2 L g ) \gamma/(2L_g) γ / ( 2 L g ) ; for L g = 0 L_g=0 L g = 0 , the readout is constant on the compared region and the conclusion is immediate. If the learned and reference readout heads differ, an additional common-state readout discrepancy ∥ g ^ ( S ^ H ) − g ( S ^ H ) ∥ ∞ \|\widehat{g}(\widehat{S}_H)-g(\widehat{S}_H)\|_\infty ∥ g ( S H ) − g ( S H ) ∥ ∞ must also be below the remaining margin budget. These pathwise robustness statements yield expected-accuracy control when combined with a suitable tail bound.
Exact directional amplification
Proposition 13 (Exact propagation of correlated errors). Let e j + 1 = M j e j + ξ j e_{j+1}=M_je_j+\xi_j e j + 1 = M j e j + ξ j in R s \mathbb{R}^s R s , with e 0 = 0 e_0=0 e 0 = 0 , deterministic s × s s\times s s × s matrices M j M_j M j , and square-integrable errors ξ j \xi_j ξ j . Define b j = E ξ j b_j=\mathbb{E}\xi_j b j = E ξ j and cross-covariances Σ j k = E [ ( ξ j − b j ) ( ξ k − b k ) T ] \Sigma_{jk}=\mathbb{E}[(\xi_j-b_j)(\xi_k-b_k)^\mathsf{T}] Σ j k = E [( ξ j − b j ) ( ξ k − b k ) T ] . With P j = M H − 1 ⋯ M j + 1 P_j=M_{H-1}\cdots M_{j+1} P j = M H − 1 ⋯ M j + 1 and P H − 1 = I s P_{H-1}=I_s P H − 1 = I s ,
E ∥ e H ∥ 2 2 = ∥ ∑ j P j b j ∥ 2 2 + ∑ j , k tr ( P j Σ j k P k T ) . (87) \mathbb{E}\|e_H\|_2^2=
\left\|\sum_jP_jb_j\right\|_2^2+
\sum_{j,k}\operatorname{tr}(P_j\Sigma_{jk}P_k^\mathsf{T}). \tag*{(87)} E ∥ e H ∥ 2 2 = j ∑ P j b j 2 2 + j , k ∑ tr ( P j Σ j k P k T ) . (87) Here tr \operatorname{tr} tr sums diagonal matrix entries. For uncorrelated centered errors only j = k j=k j = k terms remain, weighted by P j T P j P_j^\mathsf{T}P_j P j T P j , the Gramian measuring directional amplification. Reusing one fitted expert generally creates nonzero cross-time terms, so that special case cannot be assumed from independent training examples.
Proof of Proposition 13. Unrolling e j + 1 = M j e j + ξ j e_{j+1}=M_j e_j+\xi_j e j + 1 = M j e j + ξ j gives
e H = ∑ j = 0 H − 1 P j ξ j , P j = M H − 1 ⋯ M j + 1 . (88) e_H=\sum_{j=0}^{H-1}P_j\xi_j,\qquad P_j=M_{H-1}\cdots M_{j+1}. \tag*{(88)} e H = j = 0 ∑ H − 1 P j ξ j , P j = M H − 1 ⋯ M j + 1 . (88)
The mean is ∑ j P j b j \sum_j P_jb_j ∑ j P j b j . Expanding the covariance without discarding cross-time terms gives
Cov ( e H ) = ∑ j , k P j Σ j k P k T . (89) \operatorname{Cov}(e_H)=\sum_{j,k}P_j\Sigma_{jk}P_k^{\mathsf T}. \tag*{(89)} Cov ( e H ) = j , k ∑ P j Σ j k P k T . (89)
Use E ∥ X ∥ 2 = ∥ E X ∥ 2 + tr ( Cov X ) \mathbb{E}\lVert X\rVert^2=\lVert\mathbb{E}X\rVert^2+\operatorname{tr}(\operatorname{Cov}X) E ∥ X ∥ 2 = ∥ E X ∥ 2 + tr ( Cov X ) to obtain Proposition 13. This identity follows by writing X = E X + ( X − E X ) X=\mathbb{E}X+(X-\mathbb{E}X) X = E X + ( X − E X ) , expanding its squared norm, and observing that the centered cross term has zero expectation. If all off-diagonal Σ j k \Sigma_{jk} Σ j k vanish, the previous diagonal Gramian formula follows. The double sum is nonnegative as a whole, although individual cross terms may be negative. If M j M_j M j is random and correlated with the errors, the deterministic covariance factorization is not valid: only the pathwise identity e H = ∑ j P j ξ j e_H=\sum_jP_j\xi_j e H = ∑ j P j ξ j is automatic. Proposition 43 shows explicitly how tying parameters generates cross-time terms even in a simple additive model.
Causal Route Actions and Proofs
Overview. We ask what a forced route actually changes, rather than what its natural label suggests. The counterexamples and risk decompositions distinguish observational agreement from executable reuse and show how an insufficient macro state obstructs prediction of intervention effects. Action coverage is treated separately in Section K. Sections F.1–F.3 define actions and prove the observational and state-aliasing obstructions; Sections F.4–F.5 cover rollout, reporting requirements, and the average-risk version of the obstruction.
Non-anticipating route actions
A complete micro context C ∈ C C\in\mathcal{C} C ∈ C contains the prompt, generated prefix, caches, hidden states, router state, and all randomness revealed up to the intervention boundary. A fixed macro extractor Ψ : C → S \Psi:\mathcal{C}\to\mathcal{S} Ψ : C → S produces S = Ψ ( C ) S=\Psi(C) S = Ψ ( C ) . A route action A i = ( ϖ i , σ i ) A_i=(\varpi_i,\sigma_i) A i = ( ϖ i , σ i ) consists of a non-anticipating policy over future route decisions and a stopping time 1 ≤ σ i ≤ B a c t 1\leq\sigma_i\leq B_{\mathrm{act}} 1 ≤ σ i ≤ B act . Every decision made by ϖ i \varpi_i ϖ i is measurable with respect to the information available immediately before that decision; neither the policy nor the stopping rule may inspect unrevealed future tokens or a completed post-hoc route trace.
For pathwise statements, execution is greedy or is conditioned on a common realization of exogenous randomness. Executing A i A_i A i then yields a deterministic micro transition T i ( C ) \mathcal{T}_i(C) T i ( C ) and macro result T i ( C ) = Ψ ( T i ( C ) ) T_i(C)=\Psi(\mathcal{T}_i(C)) T i ( C ) = Ψ ( T i ( C )) . Under stochastic generation, the corresponding object is a transition kernel; the deterministic claims below then apply conditionally, while the Wasserstein formulation in § 24 controls the induced laws.
For a candidate macro operator F i F_i F i , fixed before the evaluation sample is observed, the off-policy defect under a fixed probe distribution ν i \nu_i ν i is
μ i o f f = E C ∼ ν i d ( T i ( C ) , F i ( Ψ ( C ) ) ) . (90) \mu_i^{\mathrm{off}}=\mathbb{E}_{C\sim\nu_i}d\bigl(T_i(C),F_i(\Psi(C))\bigr). \tag*{(90)} μ i off = E C ∼ ν i d ( T i ( C ) , F i ( Ψ ( C )) ) . (90)
If a controller executes actions on-policy, the corresponding occupancy changes with the action library and controller; an on-policy defect must therefore be evaluated under that induced distribution. For K ≥ 2 K\geq2 K ≥ 2 , a small defect is meaningful only together with non-collapse between actions, for example
Sep ( ν ) = E C ∼ ν 2 K ( K − 1 ) ∑ i < i ′ d ( T i ( C ) , T i ′ ( C ) ) . (91) \operatorname{Sep}(\nu)=\mathbb{E}_{C\sim\nu}\frac{2}{K(K-1)}\sum_{i<i'}d\bigl(T_i(C),T_{i'}(C)\bigr). \tag*{(91)} Sep ( ν ) = E C ∼ ν K ( K − 1 ) 2 i < i ′ ∑ d ( T i ( C ) , T i ′ ( C ) ) . (91)
If Ψ \Psi Ψ , the actions, or the operator class are selected using the same records on which the defect is reported, sample splitting, cross-fitting, or a uniform complexity correction is required.
Proof that observational alignment is insufficient
Proof of Proposition 5. Construct a residual routed block whose shared path receives ( S , I ) (S,I) ( S , I ) and returns F I ( S ) F_I(S) F I ( S ) . Set every expert contribution identically to zero, and let the natural router output index I. The model output is exactly F I ( S ) F_I(S) F I ( S ) and natural route-label accuracy is one. Under an intervention replacing only the routed index by J J J , the shared path is unchanged and every expert contribution remains zero, so the output is still F I ( S ) F_I(S) F I ( S ) and is independent of J J J . Forced-route fidelity to the target associated with J J J is therefore exactly P ( F I ( S ) = F J ( S ) ) \mathbb{P}(F_I(S)=F_J(S)) P ( F I ( S ) = F J ( S )) . If J J J is uniform and independent of ( I , S ) (I,S) ( I , S ) and the K K K values F 1 ( S ) , … , F K ( S ) F_1(S),\ldots,F_K(S) F 1 ( S ) , … , F K ( S ) are pairwise distinct at every evaluated state, equality occurs if and only if J = I J=I J = I , giving fidelity 1 / K 1/K 1/ K .
For comparison, a second model with zero shared contribution and experts E i ( S ) = F i ( S ) E_i(S)=F_i(S) E i ( S ) = F i ( S ) has the same natural outputs and the same labels under route I I I , but a forced route J J J returns F J ( S ) F_J(S) F J ( S ) . These two models are observationally equivalent on task outputs and natural route labels and differ under intervention. The construction is compatible with perfect task performance. It establishes a non-identifiability result: behavioral and route-label observations alone do not determine where the operation is implemented.
State-aliasing obstruction and proof
Let C 0 \mathcal{C}_0 C 0 be a nonempty set of contexts. Two contexts are aliased when they have the same extracted state. Define the largest difference between their responses to action i i i by
Ξ i ( Ψ ; C 0 ) = sup C , C ′ ∈ C 0 , C ≠ C ′ Ψ ( C ) = Ψ ( C ′ ) d ( T i ( C ) , T i ( C ′ ) ) . (92) \Xi_i(\Psi;\mathcal{C}_0)=\sup_{\substack{C,C'\in\mathcal{C}_0,\ C\ne C'\\ \Psi(C)=\Psi(C')}}d(T_i(C),T_i(C')). \tag*{(92)} Ξ i ( Ψ ; C 0 ) = C , C ′ ∈ C 0 , C = C ′ Ψ ( C ) = Ψ ( C ′ ) sup d ( T i ( C ) , T i ( C ′ )) . (92)
with value zero when no distinct aliased pair exists. Thus Ξ i \Xi_i Ξ i is the largest possible action-effect discrepancy among contexts that the macro representation cannot distinguish; the supremum need not be attained.
Proposition 14 (State-aliasing obstruction). Every deterministic F i F_i F i satisfies
sup C ∈ C 0 d ( T i ( C ) , F i ( Ψ ( C ) ) ) ≥ 1 2 Ξ i ( Ψ ; C 0 ) . (93) \sup_{C\in\mathcal{C}_0}d(T_i(C),F_i(\Psi(C)))\ge\frac{1}{2}\Xi_i(\Psi;\mathcal{C}_0). \tag*{(93)} C ∈ C 0 sup d ( T i ( C ) , F i ( Ψ ( C ))) ≥ 2 1 Ξ i ( Ψ ; C 0 ) . (93)
Proof of Proposition 14. For any distinct C , C ′ C,C' C , C ′ with Ψ ( C ) = Ψ ( C ′ ) = s \Psi(C)=\Psi(C')=s Ψ ( C ) = Ψ ( C ′ ) = s , the triangle inequality gives
d ( T i ( C ) , T i ( C ′ ) ) ≤ d ( T i ( C ) , F i ( s ) ) + d ( F i ( s ) , T i ( C ′ ) ) . (94) d(T_i(C),T_i(C'))\le d(T_i(C),F_i(s))+d(F_i(s),T_i(C')). \tag*{(94)} d ( T i ( C ) , T i ( C ′ )) ≤ d ( T i ( C ) , F i ( s )) + d ( F i ( s ) , T i ( C ′ )) . (94)
At least one right-hand term is at least half the left-hand side. Taking the supremum over feasible distinct aliased pairs proves the claim. If no such pair exists, the defined diameter is zero and the inequality is trivial.
This is a worst-case obstruction; rare aliased contexts need not prevent small average defect. For squared loss, Proposition 15 instead characterizes the minimum average defect exactly by conditional variance. Non-collapse, pre-action selectability, and on-policy composition remain additional requirements; definitions and rollout bounds are in Section F.
Causal rollout
Let I j I_j I j be the executed action and let I j ⋆ I_j^\star I j ⋆ be a fixed or externally coupled reference action sequence on the same probability space. The reference sequence is held fixed under the coupling; comparing two state-dependent controllers requires an additional policy-stability or selection-mismatch term. Define
ϵ j a b s = d ( T I j ( C j ) , F I j ( S j ) ) , (95) \epsilon_j^{\mathrm{abs}}=d(T_{I_j}(C_j),F_{I_j}(S_j)), \tag*{(95)} ϵ j abs = d ( T I j ( C j ) , F I j ( S j )) , (95)
χ j s e l = d ( F I j ( S j ) , F I j ⋆ ( S j ) ) . (96) \chi_j^{\mathrm{sel}}=d(F_{I_j}(S_j),F_{I_j^\star}(S_j)). \tag*{(96)} χ j sel = d ( F I j ( S j ) , F I j ⋆ ( S j )) . (96)
where S j = Ψ ( C j ) S_j=\Psi(C_j) S j = Ψ ( C j ) . Let S ˉ 0 = S 0 \bar{S}_0=S_0 S ˉ 0 = S 0 and S ˉ j + 1 = F I j ⋆ ( S ˉ j ) \bar{S}_{j+1}=F_{I_j^\star}(\bar{S}_j) S ˉ j + 1 = F I j ⋆ ( S ˉ j ) , and assume F I j ⋆ F_{I_j^\star} F I j ⋆ is L j L_j L j -Lipschitz on a region containing both trajectories. Inserting F I j ( S j ) F_{I_j}(S_j) F I j ( S j ) and F I j ⋆ ( S j ) F_{I_j^\star}(S_j) F I j ⋆ ( S j ) yields
d ( S j + 1 , S ˉ j + 1 ) ≤ ϵ j a b s + χ j s e l + L j d ( S j , S ˉ j ) . (97) d(S_{j+1},\bar{S}_{j+1})\le\epsilon_j^{\mathrm{abs}}+\chi_j^{\mathrm{sel}}+L_jd(S_j,\bar{S}_j). \tag*{(97)} d ( S j + 1 , S ˉ j + 1 ) ≤ ϵ j abs + χ j sel + L j d ( S j , S ˉ j ) . (97)
Induction gives
d ( S H , S ~ H ) ≤ ∑ j = 0 H − 1 ( ∏ r = j + 1 H − 1 L r ) ( ϵ j a b s + χ j s e l ) . (98) d(S_H,\widetilde{S}_H) \le\sum_{j=0}^{H-1}\left(\prod_{r=j+1}^{H-1} L_r\right)\left(\epsilon_j^{\mathrm{abs}}+\chi_j^{\mathrm{sel}}\right). \tag*{(98)} d ( S H , S H ) ≤ j = 0 ∑ H − 1 ( r = j + 1 ∏ H − 1 L r ) ( ϵ j abs + χ j sel ) . (98)
This controls macro-transition mismatch for the coupled reference sequence. Comparing state-feedback controllers additionally requires control of how the reference policy changes when its input state changes.
The supremum in Equation (93) is sensitive to rare contexts. An average-risk conclusion needs their probability, not just their maximum separation. Fix an action i i i , a probe-context distribution, S = Ψ ( C ) S = \Psi(C) S = Ψ ( C ) in a standard Borel space, and a square-integrable actual action outcome Y i = T i ( C ) Y_i = T_i(C) Y i = T i ( C ) in a separable real Hilbert space H \mathcal{H} H . A stochastic outcome can be included as long as its additional randomness is also averaged. Define m i ( S ) = E [ Y i ∣ S ] m_i(S) = \mathbb{E}[Y_i \mid S] m i ( S ) = E [ Y i ∣ S ] .
Proposition 15 (Average state-compression obstruction). Among all measurable predictors F F F with finite second moment,
inf F E ∥ Y i − F ( S ) ∥ H 2 = E ∥ Y i − m i ( S ) ∥ H 2 = 1 2 E ∥ Y i − Y i ′ ∥ H 2 , (99) \inf_F \mathbb{E}\lVert Y_i-F(S)\rVert_{\mathcal{H}}^2 = \mathbb{E}\lVert Y_i-m_i(S)\rVert_{\mathcal{H}}^2 = \frac{1}{2}\mathbb{E}\lVert Y_i-Y_i'\rVert_{\mathcal{H}}^2, \tag*{(99)} F inf E ∥ Y i − F ( S ) ∥ H 2 = E ∥ Y i − m i ( S ) ∥ H 2 = 2 1 E ∥ Y i − Y i ′ ∥ H 2 , (99) where, conditional on S S S , Y i Y_i Y i and Y i ′ Y_i' Y i ′ are independent draws from the same conditional action-outcome law. The optimum is zero exactly when Y i = m i ( S ) Y_i = m_i(S) Y i = m i ( S ) almost surely.
Proof. Insert m i ( S ) m_i(S) m i ( S ) into Y i − F ( S ) Y_i-F(S) Y i − F ( S ) . Conditional on S S S , the residual Y i − m i ( S ) Y_i-m_i(S) Y i − m i ( S ) has mean zero, so its inner product with m i ( S ) − F ( S ) m_i(S)-F(S) m i ( S ) − F ( S ) has zero expectation. The risk equals E ∥ Y i − m i ( S ) ∥ 2 + E ∥ m i ( S ) − F ( S ) ∥ 2 \mathbb{E}\lVert Y_i-m_i(S)\rVert^2+\mathbb{E}\lVert m_i(S)-F(S)\rVert^2 E ∥ Y i − m i ( S ) ∥ 2 + E ∥ m i ( S ) − F ( S ) ∥ 2 . The first term is attained by F = m i F=m_i F = m i . Conditional independence makes the inner product of Y i − m i ( S ) Y_i-m_i(S) Y i − m i ( S ) and Y i ′ − m i ( S ) Y_i'-m_i(S) Y i ′ − m i ( S ) have zero conditional expectation; expanding their squared difference gives twice the first term. A nonnegative squared norm has zero mean exactly when it vanishes almost surely.
For deterministic actions this residual quantifies action-relevant information lost by the state extractor. With stochastic outcomes it also includes irreducible execution noise; the theorem does not label all conditional variance as compression error. Restricting F F F to a neural class adds its approximation error and cannot improve the unrestricted infimum. Unlike the worst-case diameter, Equation (99) weights rare contexts by their probability.
Controlled Experiments: Methods and Results
Overview. This appendix describes three complementary studies: controlled estimators test quantitative risk predictions, trained Transformers compare one-step and final-output performance, and interventions examine the effects attributed to expert calls. The primary experiment family and later extensions retain separate evaluation and confidence scopes. Sections G.1–G.6 specify the task, models, controlled estimation, trained comparisons, interventions, routing diagnostics, and paired performance evaluation. The remaining sections cover data budgets, transition holdouts, state-space size, state-conditioned variability, and general feature geometry.
Shared task, models, and evaluation protocol
The primary experiment family comprises fifteen trained models: three sharing architectures under the task-and-balance objective, and two additional supervision objectives on the tied architecture, each with three training seeds. The tied task-and-balance models are reused as the intervention baseline. Data-budget and generalization studies are described separately with their own settings and evaluation scope.
Task and splits. The task composes four operations on a finite state space,
F 0 ( s ) = s + 1 , F 1 ( s ) = s + 7 , F 2 ( s ) = 2 s , F 3 ( s ) = 3 s + 5 ( m o d 31 ) . F_0(s) = s + 1,\quad F_1(s) = s + 7,\quad F_2(s) = 2s,\quad F_3(s) = 3s + 5 \pmod{31}. F 0 ( s ) = s + 1 , F 1 ( s ) = s + 7 , F 2 ( s ) = 2 s , F 3 ( s ) = 3 s + 5 ( mod 31 ) .
and the stream ⟨ b o s ⟩ , s 0 , z 1 , s 1 , … , z H , s H \langle\mathrm{bos}\rangle, s_0, z_1, s_1, \ldots, z_H, s_H ⟨ bos ⟩ , s 0 , z 1 , s 1 , … , z H , s H , where s t = F z t ( s t − 1 ) s_t = F_{z_t}(s_{t-1}) s t = F z t ( s t − 1 ) , and H H H is the number of operation application. The operation program is externally supplied, not generated by the model. A 37-symbol input vocabulary encodes state s s s as token s + 2 s + 2 s + 2 and operation z z z as token z + 33 z + 33 z + 33 ; a 31-way state head predicts the next state at operation-token positions. Training uses reference prefixes (teacher forcing), whereas free-running evaluation uses greedy state predictions as the subsequent inputs. The first four operations define a family; the protocol hashes their canonical bytes to assign 209 training, 22 development, and 25 test families. All initial states and continuations of one family stay together. Family assignment conditions on the entire four-operation prefix; it does not make those four coordinates independently uniform within a split. This withholds program-prefix families, not every local state–action transition. The strict transition holdout in Section G.8 is a different training and evaluation protocol.
The eight artificial estimation roles, three scalar composition roles, and four symbolic task operations belong to distinct experimental settings.
The primary pool contains 16,384 independently sampled trace records, equally split between training horizons 4 and 8; repeated contents are allowed under this sampling law. Three training seeds share this pool. Within a seed, compared models use the same horizon and minibatch-index schedule. Throughout this experimental appendix, H4, H8, H12, and H16 denote operation application number H = 4 , 8 , 12 , 16 H = 4, 8, 12, 16 H = 4 , 8 , 12 , 16 , respectively. H8 is the primary free-running evaluation. H4 enumerates all 775 held-out-family traces; H12/H16 are descriptive length-extrapolation tests.
Models and fitting. All primary models have four pre-normalized causal Transformer layers, hidden width 256, four attention heads, sinusoidal positions, SwiGLU FFNs of inner width 768, no dropout, and no capacity dropping. Local has four distinct experts per layer; tied reuses one bank of four experts across depth with layer-specific routers and normalization; dense has one unrouted, active-width-matched FFN per layer. Top-1 routing retains the selected softmax gain. Local and tied initialize expert values equally but differ in parameter-object identity. Dense is an active-width control, not an identical initial function.
The training settings below distinguish the fifteen primary architecture/objective runs from the smaller model trained solely to supply frozen features. Follow-up experiments use the changes specified in their respective sections.
Table 1 (PDF p. 36).
Evaluation uses FP32. Trace presentations count repeated uses of the training pool. Training and neural-model evaluation use NVIDIA RTX A5000 24 GB GPUs, with one card for the primary experiments and a second for the follow-ups. Controlled linear calculations use float64. Training time, memory, unique parameters, and dispatched computation are reported separately.
Checkpoint selection and development data. The primary comparisons use all fifteen checkpoints at the final training update, 3,000. Development monitoring uses a fixed 512-trace set, evaluated every 500 updates. The coefficients in Section G.4 specify the three training objectives. Candidate checkpoints, action mappings, evaluation losses, comparison families, and sample sizes are fixed before generating the primary evaluation banks. The mappings are selected using development data as described in Section G.4. Follow-up studies retain their own protocols and are not included retrospectively in the primary comparison family.
Outcomes and independent units. Terminal error is the indicator of an incorrect final state; complete-trace error indicates any incorrect predicted state along a free-running trace. A trace can recover from an intermediate error, so these are different endpoints. Teacher-forced step error averages state-prediction mistakes on reference prefixes; it is not the event-lift role loss of Equation (3) and is not a substitute for free-running performance. A complete trace is the independent evaluation unit. Multiple forced actions on one context are averaged before across-context uncertainty is calculated. Across-seed standard deviations describe training variability, not additional audit samples.
Controlled local estimation and composition
Local estimation gains under controlled sharing. Frozen, nonorthogonal Transformer features define a linear residual-estimation problem. All 54 estimation settings agree with their covariance-aware references within five Monte Carlo standard errors; two repetitions on trained features satisfy the same numerical agreement criterion (Section G.2).
Frozen Transformer feature interface. The frozen-feature estimation study uses a separately trained two-layer, width-128 Transformer after 300 updates. For each probe trace, one prediction position is selected uniformly. The feature is the normalized input to the expert subblock in the second Transformer layer, after its FFN pre-normalization. Features are extracted in evaluation mode without gradients in FP32, converted to float64, and mapped to 32 dimensions by a fixed random orthonormal 128 × 32 projection. No centering, whitening, or post-hoc direction selection is applied. A controlled residual call uses T a [ A ] ( z ) = h ( z ) + J A u ( z ) T_a[A](z) = h(z) + J_Au(z) T a [ A ] ( z ) = h ( z ) + J A u ( z ) with J ⊤ J = I 16 J^\top J = I_{16} J ⊤ J = I 16 , so the squared difference between complete event outputs equals ∥ ( A − A a ) u ( z ) ∥ 2 \lVert(A - A_a)u(z)\rVert^2 ∥( A − A a ) u ( z ) ∥ 2 . Eight artificial statistical roles are defined independently of the symbolic operation, trace content, and layer.
Each role has 64 fixed nonorthogonal design rows. Repeating those rows one, four, or sixteen times, with fresh independent Gaussian label noise of standard deviation 0.1, gives n = 64 n = 64 n = 64 , 256, 1024 independent measurements. The input support does not increase with n n n . Test geometry is the empirical second moment of a separately frozen 8,192-record feature pool. Targets are held constant across the measurement-count sweep and calibrated to three heterogeneity levels c = 0 c = 0 c = 0 , 0.5, 2. Designs, targets, features, surrounding state, and test geometry are fixed before drawing fitting noise. The 32-dimensional test second moment has condition number approximately 1.95 × 10 7 1.95 \times10^7 1.95 × 1 0 7 ; the eight design singular-value ratios range from 1.20 × 10 − 4 1.20 \times10^{-4} 1.20 × 1 0 − 4 to 1.52 × 10 − 4 1.52 \times10^{-4} 1.52 × 1 0 − 4 . The fitting procedure uses neither ridge regularization nor whitening.
More precisely, the calibration sets B ( c ) = c C 256 B(c) = cC_{256} B ( c ) = c C 256 , where B ( c ) B(c) B ( c ) is target sharing bias in this fixed test geometry and C 256 C_{256} C 256 is the estimation variance removable by pooling at n = 256 n = 256 n = 256 . The dimensionless c c c therefore fixes target heterogeneity relative to one reference variance scale. Writing V 0 ( n ) V_0(n) V 0 ( n ) for separate-estimation risk in this geometry, the reported eight-role construction has C = ( 7 / 8 ) V 0 ( n ) C = (7/8)V_0(n) C = ( 7/8 ) V 0 ( n ) . Repeated design rows make these variance quantities scale as 1 / n 1/n 1/ n , whereas B ( c ) B(c) B ( c ) stays fixed. Thus c c c is a target-construction parameter.
Float64 role-wise least squares is solved by reduced QR and followed by the test-geometry projection of Equation (214) . The five fixed coefficients 0, 0.25, 0.5, 0.75, 1 and the population oracle are evaluated on the same noisy observations with each of 128 independent repetitions. Observed risk is computed from the fitted matrices; the analytic reference is stored separately. All 54 estimation settings meet the numerical comparison criterion ∣ m e a n − r e f e r e n c e ∣ ≤ max ( 5 M C S E , 10 − 8 ) |\mathrm{mean} - \mathrm{reference}| \leq\max(5\ \mathrm{MCSE}, 10^{-8}) ∣ mean − reference ∣ ≤ max ( 5 MCSE , 1 0 − 8 ) . Full-pooling ratios cross one between the sampled measurement counts in the nonzero-heterogeneity settings, consistent with the specified reference.
Table 4. Frozen-feature estimation endpoints from 128 fitting-noise repetitions. Parentheses are Monte Carlo standard errors (MCSE). Ratios divide the two means.
Figure 2. Frozen-feature estimation under fixed target heterogeneity. Observed full/separate ratios and the independently selected coefficients are distinct from the analytic lines and population oracle. More measurements of the same feature rows reduce noise without removing fixed target heterogeneity.
A realizable selector uses 4,096 fresh noisy validation records per fit to choose among the five fixed coefficients. Its risk and coefficient frequencies are reported separately in Table 5 ; the oracle target is not made available to selection. The squared Gaussian loss is unbounded, so the bounded primary audit radius is not applied to this estimation study. The trained-feature replication repeats this inserted-estimator design on features from two trained step-3000 checkpoints, with condition numbers about 3,002 and 2,625, respectively; both 54-setting repetitions satisfy the same numerical comparison criterion.
Table 5. Validation-selected pooling (4,096 fresh validation records per fit). Frequencies follow the grid 0, 0.25, 0.5, 0.75, 1 and sum to 128; conditional finite-grid excess is computed after selection.
Table 6. Trained-feature replication repeats the frozen-estimator protocol using features from two completed step-3000 checkpoints. Counts use the five-MCSE numerical-agreement criterion.
Local improvements can reverse at the final output. The executed three-event construction of Proposition 47 evaluates the same fitted coefficients locally and after composition. Full pooling lowers observed role-average squared error from 1.0090 to 0.8359, while final-output squared error rises from 6.0086 to 9.0000 (Figure 3 ). The corresponding analytic optima also differ: λ = 4 / 7 \lambda= 4/7 λ = 4/7 locally versus 2 / 5 2/5 2/5 at the endpoint. Thus a local improvement can accompany worse final output in this controlled program.
Figure 3. Controlled local-to-final risk reversal. The same estimates are evaluated in local and final-output geometries. Markers are reported means with ± 5 \pm5 ± 5 MCSE over 20,000 noise repetitions; dashed curves are analytic references. Full pooling helps locally but hurts the final output, and the optimal pooling strengths differ. The correlated-noise case isolates a further limit of pooling. Panels (a) and (b) use the same three-role estimates; (c) is a separate two-dimensional model. The common-direction normalization and both constructions are specified in Section G.2.
Composition of the same estimates. The controlled composition study executes the existing three-event construction in Proposition 47 , with θ = ( 1 / 2 , − 1 , 1 / 2 ) T \theta= (1/2,-1,1/2)^{\mathsf{T}} θ = ( 1/2 , − 1 , 1/2 ) T , independent unit-variance coefficient noise, and common-mean pooling. The implementation evaluates both the actual recurrence and its fixed linear terminal map on 20,000 noise repetitions. The role-average, common-direction, and terminal geometries are distinct evaluations of the same estimates. Full pooling changes observed local risk from 1.0090 to 0.8359 and terminal risk from 6.0086 to 9.0000, against references 1 → 5 / 6 1 \to5/6 1 → 5/6 and 6 → 9 6 \to9 6 → 9 . The terminal full-pooling output is constant, hence its MCSE is zero.
In Figure 3 (a), the common-direction curve measures the squared scalar error projection
E [ ( 1 T ( L λ Y − θ ) 3 ) 2 ] , 1 = ( 1 , 1 , 1 ) T . \mathbb{E}\left[\left(\frac{\mathbf{1}^{\mathsf{T}}(L_{\lambda}Y-\theta)}{\sqrt{3}}\right)^2\right], \qquad\mathbf{1}=(1,1,1)^{\mathsf{T}}. E [ ( 3 1 T ( L λ Y − θ ) ) 2 ] , 1 = ( 1 , 1 , 1 ) T .
Its analytic value is 1 at every displayed pooling strength. This is a projection onto a unit direction, rather than role-average risk or the squared error of the arithmetic mean. Panels (a) and (b) evaluate the same three-role estimates. Panel (c) instead uses the separately declared two-dimensional model from Appendix § 26 .§ 1 :
θ = 0 , P = diag ( 1 , 0 ) , G = ( 1 , 1 ) , Σ = ( 4 − 3 / 2 − 3 / 2 1 ) . \theta=0,\qquad P=\operatorname{diag}(1,0),\qquad G=(1,1),\qquad\Sigma=\begin{pmatrix}4 & -3/2 \\ -3/2 & 1\end{pmatrix}. θ = 0 , P = diag ( 1 , 0 ) , G = ( 1 , 1 ) , Σ = ( 4 − 3/2 − 3/2 1 ) .
In that model, suppressing a component removes beneficial noise cancellation, so positive pooling increases terminal risk.
Table 7: Controlled composition: selected points; 20,000 noise repetitions for each of the two models. Reference and observed values are separate columns. The full-pooling terminal loss is constant, so its MCSE is zero.
Table 7.
Nine primary runs compare the three architectures at three seeds, all at the full budget. Tied has 3,433,984 distinct parameters versus 10,511,872 for local, a 67.3% reduction; dense has 3,429,888. The reported H8 audit contains one erroneous trace across the nine checkpoints, produced by local seed 1. All twelve architecture intervals cover zero. The tested task is saturated at this budget. Matching active FFN width does not establish equal total compute or elapsed time. The per-seed resource quantities are reported separately below.
Table 8. Shared, unshared, and dense Transformers at training update 3000. Errors are percentages on 32,768 H8 traces per checkpoint; training time excludes I/O. Parameter counts are millions; peak is allocated MiB.
H4 is also solved on its full finite test population. At H12/H16, terminal error ranges approximately from 0.964 to 0.972 and complete-trace error is at least 0.999 in the supplied primary records. The uniform 31-state guessing error is a scale reference. These trained systems do not demonstrate successful length extrapolation under the reported settings.
Table 9. Primary horizon diagnostics (percent error): H4 enumerates all 775 family-held-out traces; H12/H16 use 4,096 traces per checkpoint. Each horizon lists terminal / complete-trace error.
Route labels and executable interventions
Objectives and interventions. The three training objectives are cumulative:
L B = L t a s k + 0.01 L b a l , L R = L B + 0.1 L r o u t e , L I = L R + 0.5 L C I . \begin{aligned}
\mathcal{L}_B &= \mathcal{L}_{\mathrm{task}} + 0.01\mathcal{L}_{\mathrm{bal}},\\
\mathcal{L}_R &= \mathcal{L}_B + 0.1\mathcal{L}_{\mathrm{route}},\\
\mathcal{L}_I &= \mathcal{L}_R + 0.5\mathcal{L}_{\mathrm{CI}}.
\end{aligned} L B L R L I = L task + 0.01 L bal , = L B + 0.1 L route , = L R + 0.5 L CI .
The task loss is cross-entropy for the next state. The balancing term is computed over all input tokens, while route supervision is cross-entropy between natural per-layer router logits and the requested operation at state-prediction positions. The natural token/update budget is matched. I also receives a counterfactual forward pass and extra target per training trace, so neither its total computation nor its supervision is matched to R. B is the tied baseline from the architecture comparison, reused rather than retrained: the primary family has fifteen unique runs.
The intervention loss independently draws t t t uniformly from { 1 , … , H } \{1,\ldots,H\} { 1 , … , H } and operation i i i uniformly from { 0 , 1 , 2 , 3 } \{0,1,2,3\} { 0 , 1 , 2 , 3 } for each training trace. At the corresponding state-prediction position in every primary-model layer, it forces expert i i i with that layer’s detached natural-pass gain; this is a four-layer compound action. The requested operation token and pre-action prefix remain unchanged; the target is F i ( s t − 1 ) F_i(s_{t-1}) F i ( s t − 1 ) , the chosen operation applied to the pre-action state. Future target tokens and post-action reference states are not inserted. For B, which has no route-label supervision, an operation-to-expert permutation is selected separately at each layer on 4,096 development contexts. All 24 permutations are enumerated; the chosen permutation maximizes natural label agreement, with lexicographic tie-breaking. R/I retain their label-defined identity mapping for the main comparison. Neither final forced fidelity nor test data are used to select the mapping.
At the forced position, let z ℓ n a t z_{\ell}^{\mathrm{nat}} z ℓ nat denote layer ℓ \ell ℓ ’s router logits in the original natural pass. Its recorded gain is the natural winner’s probability, q ℓ n a t = max e softmax ( z ℓ n a t ) e q_{\ell}^{\mathrm{nat}} = \max_e \operatorname{softmax}(z_{\ell}^{\mathrm{nat}})_e q ℓ nat = max e softmax ( z ℓ nat ) e . After detachment, this fixed value multiplies the forced expert’s output. The convention preserves the original winner’s gain rather than looking up the forced expert’s original probability or recomputing a gain during intervention. All layers’ reference gains come from the same original natural pass. The forced pass still computes its own activations, while the residual and other shared paths retain their declared computation. The operation-to-expert mapping remains development-selected for B and the identity for R/I. This is the frozen-gain convention of Section P.4, distinct from current-gain support replay in Section P.3.
Evaluation. For each of 32,768 independent H8 teacher-prefix contexts, all four actions are forced and averaged. Forced fidelity is the average indicator that the resulting state equals the declared operation’s output. Cross-action fidelity excludes the originally requested action. Action separation compares outputs of distinct forced actions and must be read alongside target separation on the same states. The uniform-state, uniformly requested and forced collapsed-control reference is 67/248, not 1/4, because several operation outputs coincide; this control distribution is not automatically the distribution of every model-specific probe. Same-index forcing, causal-mask, and explicit-operator controls test implementation rather than learned performance.
The main action is a four-layer compound intervention. Single-layer tests use a separate 2,048-context bank, changing only one call. On-policy diagnostics evaluate each model on its own reached-state distribution. The label-agreement columns use development scores; primary compound fidelity is evaluated on the independent test contexts. These columns therefore describe different evaluation sets.
R and I both have development label agreement 1, but compound fidelities are 0.9467 ± 0.0171 and 1.0000 ± 0.0000, versus 0.2528 ± 0.0079 for B. The I checkpoints have single-layer fidelity 0.3274 ± 0.0247 when averaged over layers. This contrast attribution scopes; it does not prove that individual experts have no useful partial function. All these averages are descriptive over three seeds.
Table 10. Per-seed primary causal results. Raw/aligned label scores use development data; compound fidelity uses final teacher contexts; L0–L3 are separate single-layer secondary interventions.
Table 11. On-policy secondary diagnostics, 2,048 contexts per checkpoint. Each model reaches its own context distribution. Mean ± sample SD over seeds.
Table 11 (PDF p. 42).
Intervention effects are a separate question. Route-supervised models at 64 traces have label agreement 1 but forced-action fidelity 0.3713 ± 0.0267. At the primary budget, intervention supervision gives compound fidelity 1 without establishing single-expert attribution. Separately holdout-trained models retain seen fidelity 1 but reach only 0.0021 ± 0.0037 on withheld transitions. A separately trained follow-up retains the seen/withheld fidelity gap at moduli 97 and 251 (Section G.8.1). These address causal attribution and coverage; full tables remain in Sections G.4 and G.8.
Functional routing, margin, and gain diagnostics
The routing diagnostics use six frozen local/tied checkpoints and 4,096 independent development sites per checkpoint. The nearest, farthest, and random alternative experts are evaluated on one common pre-call input with the same natural gain. The reported local event difference is checked against the residual-output distance. Selecting the nearest alternative makes its local distance smallest by definition; whether it has smaller terminal harm is an empirical question. A logical-site substitution persists through later full-prefix recomputation, while subsequent unforced routes respond naturally. Candidate search is additional diagnostic work.
The measurements show positive distance–harm associations in many accurate primary checkpoints. The lower-budget routing follow-up uses 24 lower-budget checkpoints, 1,024 sites each, and includes repairs as negative loss differences. Associations weaken or change sign in some settings. Downstream first-disagreement summaries refer to the instrumented operation-token positions, not every token and cache event in the computation.
Table 12. Functional expert-replacement diagnostics, 4,096 sites per checkpoint. Harm is intervened minus natural terminal error and is not clipped. Distance ordering is by construction; terminal association is empirical. SE is across independent sampled sites as reported.
Table 13. Lower-budget expert-replacement diagnostics outside the saturated primary regime. Each checkpoint uses 1,024 sites; selected nearest and farthest summaries are shown. Negative harm represents repairs.
The margin-and-gain diagnostic records logits before and after direct score perturbations, common shifts, and hidden-state perturbations. Let γ = z ( 1 ) − z ( 2 ) \gamma= z^{(1)} - z^{(2)} γ = z ( 1 ) − z ( 2 ) be the original winner–runner-up logit gap. A direct perturbation lowers the winner by c γ c\gamma c γ and raises the runner-up by c γ c\gamma c γ , leaving other logits unchanged; this c c c is a perturbation scale, not the target-heterogeneity parameter in Section G.2. Hidden-state perturbations use h ′ = h + α max { ∥ h ∥ 2 , 1 } v h^{\prime} = h + \alpha\max\{\lVert h\rVert_2, 1\}v h ′ = h + α max {∥ h ∥ 2 , 1 } v , where v v v is the perturbation-direction vector, and then recompute normalization and routing. The table gives the tested c c c and α \alpha α values. With ϵ = ∥ z ~ e − z ∥ ∞ \epsilon= \lVert\tilde{z}_e - z\rVert_\infty ϵ = ∥ z ~ e − z ∥ ∞ and tol = 64 ϵ m a c h ( 1 + max e ∣ z e ∣ ) \operatorname{tol} = 64\epsilon_{\mathrm{mach}}(1 + \max_e \lvert z_e\rvert) tol = 64 ϵ mach ( 1 + max e ∣ z e ∣) , where ϵ m a c h \epsilon_{\mathrm{mach}} ϵ mach is the floating-point machine precision, a pair is certified if γ − 2 ϵ > tol \gamma- 2\epsilon> \operatorname{tol} γ − 2 ϵ > tol , boundary if ∣ γ − 2 ϵ ∣ ≤ tol \lvert\gamma- 2\epsilon\rvert\le\operatorname{tol} ∣ γ − 2 ϵ ∣ ≤ tol , and condition-not-met otherwise. It evaluates the strict half-margin condition in Lemma 27 from the actual scores with a floating-point tolerance. Across the measured perturbation pairs, 150,095 satisfy the strict condition with slack exceeding the floating-point tolerance, with zero natural-support violations among them. Boundary cases are retained. The same-support gain measurement also reports nonzero coefficient changes, so support stability is not update equality. This is a sampled local diagnostic, not the trajectory-level conditions or a non-vacuous numerical certificate for the general routed bound.
Table 32. Router-margin and gain summaries pooled over six checkpoints. A certified violation is a natural support change despite the strict half-margin condition with numerical tolerance. Failure of the sufficient condition does not require a support change.
Independent paired performance evaluation
The primary evaluation comprises eighteen paired comparisons: tied/local and dense/local at each of three seeds under terminal and complete-trace error (twelve), plus R/B and I/R under 1 − Fid 1 - \operatorname{Fid} 1 − Fid (six). Loss differences are reference minus candidate. We apply both directions of Proposition 45 , with thirty-six tails, total failure allowance 0.05, and 32,768 independent traces/contexts per checkpoint. The resulting two-sided radius is 2.0039 percentage points.
Architecture and intervention comparisons use two independent evaluation banks, each containing 32,768 held-out-family H8 traces. Within each comparison, candidate and reference predictions are paired on the same records. The resulting intervals cross zero for all twelve architecture comparisons and favor the candidate for all six supervision comparisons. These conclusions concern fixed checkpoints on the declared evaluation distributions. Four forced actions are averaged within each context before across-context uncertainty is computed; neither those actions nor the three training seeds increase the independent sample count. The fixed-family guarantee requires candidates to be fixed before the independent evaluation data are inspected.
Table 33. All 18 primary paired comparisons. Positive differences favor the first model; numbers are percentage points. Every interval has radius 2.0039 pp (36 tails, δ = 0.05 \delta= 0.05 δ = 0.05 , N = 32768 N = 32768 N = 32768 ).
Local-to-final gains across data budgets and seeds
The training-data sweep varies the training-record budget over 17 values from 16 to 16,384, using the same 3,000-update policy. The original three seeds give 153 architecture configurations. The original interval families stay separate from the ten-seed follow-up at budgets 64, 96, 128, and 192.
A configuration specifies an architecture, budget, and seed; a contrast instead specifies a model pair, budget, and seed. Here the three architectures also yield three pairs: tied/local, dense/local, and dense/tied. Thus both counts happen to equal 17 × 3 × 3 = 153 17 \times3 \times3 = 153 17 × 3 × 3 = 153 in the original sweep. Each contrast has one local and one terminal endpoint, giving 306 endpoint comparisons and 612 one-sided tails for two-sided simultaneous intervals.
Budget- and seed-dependent comparisons. The data do not satisfy dense ≤ \le ≤ tied ≤ \le ≤ local at every budget. The original three-seed means already have exceptions at 24, 192, and 256; at 192, teacher-forced step errors are approximately 0 (dense), 0.00109 (local), and 0.00496 (tied). At 64 traces the three original tied/local step improvements are 5.18, 5.16, and 6.49 percentage points, whereas terminal differences are approximately −0.043, −0.006, −0.143 points with intervals covering zero. An established local prediction gain therefore does not imply a detectable gain on that terminal endpoint.
Table 16. Original three-seed architecture data-budget sweep: teacher-forced step error on the reported H8 bank (153 configuration summaries). Values are mean ± \pm ± sample SD over three training seeds.
Table 17. Free-running terminal error for exactly the same original three-seed architecture checkpoints and H8 bank as the preceding step-error table. Values are mean ± \pm ± sample SD.
Table 18. Tied versus local at 64 training traces, original architecture-sweep family. Values and bounds are percentage points. Differences are local minus tied error. The intervals retain the full 153-pair, 306-endpoint, 612-tail sweep budget (N = 32768 N = 32768 N = 32768 , δ = 0.05 \delta= 0.05 δ = 0.05 ; radius 2.39685 percentage points).
In the balanced ten-seed follow-up at 96 traces, the local tied/local contrast is supported in opposite directions: six seeds favor tied and four favor local. At 64, all ten favor tied on the local endpoint. This motivates reporting seeds. The 153 original local/terminal configuration contrasts and 120 follow-up contrasts overlap in 36 configurations, leaving 237 unique configurations and 84 newly added contrasts. Their original confidence budgets are not combined into a new 237-comparison family.
The follow-up uses four budgets (64, 96, 128, 192), seeds 0–9, and the same three model pairs: 4 × 10 × 3 = 120 contrasts, or 240 endpoints and 480 one-sided tails for two-sided intervals. Figure 1 displays the 40 tied/local contrasts and their two endpoints while retaining that full-family interval budget.
Table 19. Balanced ten-seed architecture follow-up; means and sample SD are computed from run summaries. Last column counts lower / higher / inconclusive local risk versus local under the separate follow-up intervals.
Paired one-step and final-output gains. Figure 1 uses all forty tied/local contrasts in the balanced follow-up: all ten seeds at all four budgets (64, 96, 128, 192). Its 64-trace panel repeats ten of those contrasts. Every bar retains its original 240-endpoint, 480-tail family interval with δ = 0.05 \delta= 0.05 δ = 0.05 and 32,768 independent traces per checkpoint.
Across those ten 64-trace seeds, one-step improvements range from 3.4931 to 7.7190 percentage points, with ten positive lower bounds. Final-output differences range from −0.1862 to +0.1923 points, with ten intervals crossing zero. Both models have near-chance final predictions in that low-data region, so the result illustrates a lack of statistically resolved endpoint improvement. Other displayed budgets show larger final changes and seed-dependent directions. No displayed comparison establishes opposite signs on both endpoints. This empirical observable uses reference prefixes locally and model-generated prefixes finally.
The one-step metric uses reference prefixes, whereas the final-state and complete-trace metrics use free-running predictions. The independent-step reference used for complete-trace error concerns failure anywhere along a trace; it is not a final-state error model. Agreement with that reference does not establish independence of actual prediction errors or exclude propagation and recovery.
Supervision at low data. The supervision-budget sweep includes ten budgets (64, 96, 128, 192, 256, 512, 1,024, 2,048, 4,096, 16,384), three objectives, and the three original training seeds, yielding ninety configuration summaries. The B models reuse tied checkpoints from the architecture sweep. Additional seed comparisons retain their separate evaluation families. At 64 traces, R retains raw route agreement 1 while forced fidelity is 0.3713 ± 0.0267 0.3713 \pm0.0267 0.3713 ± 0.0267 . Labels and action effects therefore have different empirical behavior. The extra intervention targets also constitute extra supervision, so observed I improvements are not free at matched total training cost.
Table 20. Supervision-budget sweep over ten training-data budgets, three objectives, and the three original seeds. The baseline reuses tied checkpoints from the architecture sweep. Causal evaluation uses 16,384 contexts.
Strict state–action holdout and capacity checks
The transition-holdout study uses separate B/R/I trainings with a declared set of sixteen of the 124 state–action pairs withheld. Ordinary training traces containing excluded transitions are rejected, and such pairs are excluded from intervention-training targets; input and output state symbols remain available elsewhere. Thus this is stronger than family holdout, but it is still one finite symbolic task and one selected holdout set.
The fixed withheld pairs, written as (pre-action state, operation ID) with IDs 0–3, are
( 1 , 1 ) , ( 3 , 0 ) , ( 3 , 3 ) , ( 6 , 1 ) , ( 7 , 2 ) , ( 8 , 3 ) , ( 10 , 0 ) , ( 10 , 3 ) , (1, 1),\ (3, 0),\ (3, 3),\ (6, 1),\ (7, 2),\ (8, 3),\ (10, 0),\ (10, 3), ( 1 , 1 ) , ( 3 , 0 ) , ( 3 , 3 ) , ( 6 , 1 ) , ( 7 , 2 ) , ( 8 , 3 ) , ( 10 , 0 ) , ( 10 , 3 ) ,
( 11 , 3 ) , ( 13 , 2 ) , ( 18 , 2 ) , ( 24 , 0 ) , ( 25 , 0 ) , ( 25 , 1 ) , ( 28 , 3 ) , ( 29 , 0 ) . (11, 3),\ (13, 2),\ (18, 2),\ (24, 0),\ (25, 0),\ (25, 1),\ (28, 3),\ (29, 0). ( 11 , 3 ) , ( 13 , 2 ) , ( 18 , 2 ) , ( 24 , 0 ) , ( 25 , 0 ) , ( 25 , 1 ) , ( 28 , 3 ) , ( 29 , 0 ) .
The same set is used across this B/R/I holdout family, with at most two operations excluded at any state. The per-state withholding study in Section G.8.1 uses a different design.
Evaluation partitions forced cells by the pre-state and forced action. Evaluation includes 6,987 eligible withheld contexts containing 8,593 withheld action cells; seen statistics use 16,384 contexts and 56,943 seen cells. Context averaging is retained. Natural test traces are separately partitioned into 21,919 using at least one withheld transition and 10,849 clean traces.
The holdout-trained I models have seen fidelity 1 and withheld fidelity 0.0021 ± 0.0037 over three seeds. Natural terminal error on traces using a withheld transition is about 0.984, while clean-trace error is zero. Separately trained full-table I controls have fidelity 1 on both evaluation subsets. These are different checkpoints. The observations show that coverage-conditioned compound success did not generalize in this setting. They are consistent with memorization of seen transitions, but do not uniquely identify the model’s internal implementation or negate measured in-distribution intervention effects.
Table 21. Unseen-transition evaluation: separate trainings excluding the 16 declared state–action pairs. Seen/withheld fidelity is context-averaged within the relevant action subset. Withheld SE is the reported across-context estimate; clean/uses columns partition natural test traces.
Table 22. Holdout-trained models versus separately trained full-table controls, evaluated on the same subset labels. Mean ± sample SD over seeds.
The capacity-and-holdout-size follow-up varies width, depth, training-pool size, training steps, and nested holdout size. Width 256 to 512 is a doubling, though parameter count grows by roughly four. The combined setting has eight width-512 layers, 65,536 training traces, and 12,000 steps. These tested modifications do not yield uniformly high held-out fidelity, but all-run upper bounds must retain exceptions: wide-s0 is 0.02834, long-s1 0.01174, the four-pair holdout at seed 2 is 0.13128, and the eight-pair holdout at seed 2 is 0.01236. Seen fidelity is not exactly one in every run either. All 24 follow-up summaries are shown below.
Table 23. All 24 capacity and holdout-size run summaries, including favorable exceptions and non-perfect seen fidelity. The base 16-pair results remain part of the base transition-holdout study.
State–action generalization across state-space sizes
Question and protocol. The follow-up tests whether the seen/withheld fidelity gap is confined to the original 31-state setting. It applies the same four operations modulo p = 31 , 97 , 251 p = 31, 97, 251 p = 31 , 97 , 251 , giving transition tables with 124, 388, and 1,004 entries. At every state, zero, one, or two of the four operations are withheld, corresponding to 0%, 25%, or 50% of the table. This per-state design differs from the preceding sixteen-pair holdout. The grid contains three training seeds for each of nine settings, giving 27 trained configurations.
The follow-up retains the four-layer, width-256 tied architecture and intervention-supervised objective, with modulus-specific state tokens and a p p p -way output head. Ordinary training traces containing a withheld transition are rejected, and counterfactual targets are restricted to allowed transitions. The no-holdout arm is separately trained at each modulus; it checks task learnability under full coverage.
Table 24 gives the training-pool sizes and update budgets. These budgets target approximately comparable nominal exposure; total compute is not exactly matched, and exposure is nominal rather than a measured post-rejection count for each transition. Parameter counts also change with the vocabulary and output head.
Table 24. State-space follow-up: actual reported budgets and evaluation subset sizes. Each row has three training seeds.
Evaluation and results. Forced-action evaluation uses 16,384 contexts per run and averages the applicable actions within a context before aggregation. At 25% and 50% withholding, every state has respectively one or two excluded actions, giving 16,384 and 32,768 withheld action cells, but still only 16,384 contexts. In the no-holdout control, withheld fidelity is undefined, not zero. Natural task
N c l e a n N_{\mathrm{clean}} N clean and N u s e N_{\mathrm{use}} N use are counts per seed in a 16,384-trace task bank; they are constant across the three seeds of each setting. The latter subset uses at least one withheld transition.
evaluation separately partitions 16,384 traces into clean traces and those using at least one withheld transition; their counts must not be interchanged with the causal-context counts.
All nine full-coverage control checkpoints have observed fidelity one. Across the eighteen holdout-trained checkpoints, withheld fidelity is at most 0.00647, while seen fidelity is at least 0.97763. Table 25 reports every setting, and Table 26 retains every seed. Observed clean-subset terminal error is zero in all 27 evaluated configurations, but at 50% withholding the clean subsets contain only 54, 72, and 70 traces for the three moduli.
Table 25. State-space follow-up: forced-action fidelity and free-running terminal error (mean ± sample SD across three seeds). All values are proportions.
Table 26: All 27 state-space follow-up run summaries. Fidelity and error columns are proportions.
Table 26. Withheld actions are averaged within each eligible context before aggregation; at 25% and 50% withholding there are 16,384 contexts, with one and two withheld actions per context, respectively. Empty subsets are shown as dashes.
All nine full-coverage control checkpoints have observed fidelity one. Clean-subset terminal error is zero in all 27 runs, but the subset counts in Table 24 must be retained. Dashes denote an empty subset. These are descriptive three-seed summaries.
Coverage-dependent generalization. The enlarged state spaces preserve a substantial coverage gap in this finite task family under the reported training recipe. Uniform-state guessing at 1 / p 1/p 1/ p serves as a scale reference. The displayed means and seed standard deviations are descriptive summaries over three training seeds; the primary eighteen-pair audit retains its own confidence budget.
State compression and general-design supplements
State-conditioned intervention variability. The study groups intervention outputs by the declared pre-action state and action, using a one-hot outcome representation across 124 cells. The table reports the centered outcome-variance statistic, cell-weighted fidelity, and fraction of empirically deterministic cells for 63 checkpoints. These are different descriptive quantities. In particular, the corresponding classification-loss analysis uses a conditional mode; it is not the squared-loss conditional-mean identity of Proposition 15 .
The centered and unbiased-pair variance summaries differ by up to 0.00159009, so they are not treated as interchangeable finite-sample estimators. We report the centered statistic descriptively for each fixed checkpoint and state extractor, measuring conditional outcome variability.
Table 27. State-conditioned intervention variability for 63 fixed checkpoints and the specified state extractor. The centered one-hot outcome statistic and cell-weighted fidelity measure different losses.
Pooling under general feature geometry. This supplement uses 512 noise repetitions on the frozen nonorthogonal designs. The 54 partition cells and 54 merge cells are interpreted using the existing covariance-aware fixed-estimator framework in Theorems 38 and 39 . The empirical risks satisfy the numerical comparison criterion, but the roles have unequal estimation variances. Applying a single global equal-variance price outside the balanced assumptions yields four merge-sign mismatches.
Table 28. General-design merging: four sign discrepancies obtained by applying a global equal-variance proxy outside its assumptions. The covariance-aware reference and observed mean agree in sign. Groups i / j i/j i / j denotes the merge G = { i } G = \{i\} G = { i } , H = { j } H = \{j\} H = { j } using zero-based role indices 0, …, 7; 0/4 and 2/6 therefore identify two singleton-group pairs, not group sizes.
The nine plug-in entries show lower observed risk than the separate-risk reference in this particular simulation. Contrast dimension k ≥ 4 k \ge4 k ≥ 4 is not sufficient for the classical guarantee in Section D.6; the spherical Gaussian contrast-noise and matching loss assumptions also matter and are not automatically established by the anisotropic design.
Table 29. General-design same-data shrinkage: Monte Carlo risk versus an analytic separate-risk reference, not a paired significance test.
General Linear Sharing Subspaces and Cross-Depth Architectures
Overview. We extend the local linear comparison to unequal role weights, sample counts, and noise levels, and to fixed cross-depth sharing patterns. The formulas separate the approximation cost of tying from its estimation cost, explaining which role, depth, and interaction effects a declared linear architecture retains. Fixed global-pool assignments and depth-conditioned operator subspaces are instances of this projection comparison. Section H.1 treats unequal roles and sampling, Section H.2 gives the projection formulation, Section H.3 instantiates cross-depth and multi-axis patterns, and Section H.4 collects the proofs.
Unequal roles and heteroscedastic noise
For role a ∈ [ M ] a \in[M] a ∈ [ M ] , let A a ∈ R p × d A_a \in\mathbb{R}^{p \times d} A a ∈ R p × d , assume n a ≥ d n_a \ge d n a ≥ d , and let the fixed design satisfy
X a T X a = n a I d . (100) X_a^{\mathsf{T}}X_a = n_a I_d. \tag*{(100)} X a T X a = n a I d . (100)
The response matrix is Y a = X a A a T + E a Y_a = X_a A_a^{\mathsf{T}} + E_a Y a = X a A a T + E a , where noise rows are mutually independent across observations and roles, centered, and have covariance σ a 2 I p \sigma_a^2 I_p σ a 2 I p . Different σ a 2 \sigma_a^2 σ a 2 are called heteroscedastic noise: some roles have noisier targets than others. Gaussianity is not required for any expectation identity in this section. Let π a > 0 \pi_a > 0 π a > 0 with ∑ a π a = 1 \sum_a \pi_a = 1 ∑ a π a = 1 , put D = p d D = pd D = p d and v a = σ a 2 / n a v_a = \sigma_a^2/n_a v a = σ a 2 / n a , and define
A ˉ π = ∑ a π a A a , B π = ∑ a π a ∥ A a − A ˉ π ∥ F 2 . (101) \bar{A}_{\pi} = \sum_a \pi_a A_a, \qquad B_{\pi} = \sum_a \pi_a \lVert A_a - \bar{A}_{\pi} \rVert_F^2. \tag*{(101)} A ˉ π = a ∑ π a A a , B π = a ∑ π a ∥ A a − A ˉ π ∥ F 2 . (101)
V s e p = D ∑ a π a v a , V p o o l = D ∑ a π a 2 v a , C π = V s e p − V p o o l . (102) V_{\mathrm{sep}} = D \sum_a \pi_a v_a, \qquad V_{\mathrm{pool}} = D \sum_a \pi_a^2 v_a, \qquad C_{\pi} = V_{\mathrm{sep}} - V_{\mathrm{pool}}. \tag*{(102)} V sep = D a ∑ π a v a , V pool = D a ∑ π a 2 v a , C π = V sep − V pool . (102)
The weighted pooled empirical objective is ∑ a ( π a / n a ) ∥ Y a − X a A T ∥ F 2 \sum_a (\pi_a/n_a)\lVert Y_a - X_a A^{\mathsf{T}}\rVert_F^2 ∑ a ( π a / n a ) ∥ Y a − X a A T ∥ F 2 , whose minimizer is
A ^ π = ∑ a π a A ^ a , A ^ a = Y a T X a / n a . (103) \hat{A}_{\pi} = \sum_a \pi_a \hat{A}_a, \qquad\hat{A}_a = Y_a^{\mathsf{T}}X_a/n_a. \tag*{(103)} A ^ π = a ∑ π a A ^ a , A ^ a = Y a T X a / n a . (103)
For a common shrinkage coefficient λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , define A ^ a ( λ ) = ( 1 − λ ) A ^ a + λ A ^ π \hat{A}_a^{(\lambda)} = (1-\lambda)\hat{A}_a + \lambda\hat{A}_{\pi} A ^ a ( λ ) = ( 1 − λ ) A ^ a + λ A ^ π . Its role-weighted excess prediction risk is
R λ = ∑ a π a E E , x [ ∥ ( A ^ a ( λ ) − A a ) x ∥ 2 2 ] , E [ x x T ] = I d . (104) R_{\lambda} = \sum_a \pi_a \mathbb{E}_{E,x}\left[\lVert(\hat{A}_a^{(\lambda)} - A_a)x\rVert_2^2\right], \qquad\mathbb{E}[xx^{\mathsf{T}}] = I_d. \tag*{(104)} R λ = a ∑ π a E E , x [ ∥( A ^ a ( λ ) − A a ) x ∥ 2 2 ] , E [ x x T ] = I d . (104)
Theorem 16 (Exact weighted and heteroscedastic reuse frontier). For the role-weighted excess prediction risk in Equation (104) ,
R λ = V s e p − 2 C π λ + ( B π + C π ) λ 2 . (105) R_{\lambda} = V_{\mathrm{sep}} - 2C_{\pi}\lambda+ (B_{\pi} + C_{\pi})\lambda^2. \tag*{(105)} R λ = V sep − 2 C π λ + ( B π + C π ) λ 2 . (105)
In particular,
R s e p = V s e p , R s h a r e = B π + V p o o l . (106) \mathcal{R}_{\mathrm{sep}}=V_{\mathrm{sep}},\qquad\mathcal{R}_{\mathrm{share}}=B_{\pi}+V_{\mathrm{pool}}. \tag*{(106)} R sep = V sep , R share = B π + V pool . (106)
Full reuse is better than separate estimation if and only if
B π < C π = D ∑ a π a ( 1 − π a ) σ a 2 n a . (107) B_{\pi}<C_{\pi}=D\sum_a\frac{\pi_a(1-\pi_a)\sigma_a^2}{n_a}. \tag*{(107)} B π < C π = D a ∑ n a π a ( 1 − π a ) σ a 2 . (107)
When C π > 0 C_{\pi}>0 C π > 0 , the optimal common shrinkage strength is
λ π ⋆ = C π B π + C π , R λ π ⋆ = V s e p − C π 2 B π + C π . (108) \lambda_{\pi}^{\star}=\frac{C_{\pi}}{B_{\pi}+C_{\pi}},\qquad\mathcal{R}_{\lambda_{\pi}^{\star}}=V_{\mathrm{sep}}-\frac{C_{\pi}^2}{B_{\pi}+C_{\pi}}. \tag*{(108)} λ π ⋆ = B π + C π C π , R λ π ⋆ = V sep − B π + C π C π 2 . (108)
If C π = 0 < B π C_{\pi}=0<B_{\pi} C π = 0 < B π , the optimal coefficient is zero; if C π = B π = 0 C_{\pi}=B_{\pi}=0 C π = B π = 0 , every coefficient is optimal. The quadratic form is retained, with the sharing bias B π B_{\pi} B π , separate-estimation risk V s e p V_{\mathrm{sep}} V sep , and removable variance C π C_{\pi} C π given by their weighted, heteroscedastic counterparts. A role with large evaluation mass but very few or very noisy samples contributes strongly to both the separate risk and the potential value of pooling. Conversely, raw role multiplicity can be large while C π C_{\pi} C π is small when almost all mass lies on one role.
The theorem also distinguishes two sample-allocation problems that are sometimes conflated.
Corollary 17 (Architecture-dependent optimal sampling). Fix the idealized continuous variance-allocation problem ∑ a n a = N \sum_a n_a=N ∑ a n a = N and assume σ a > 0 \sigma_a>0 σ a > 0 . The formulas below apply when the displayed unconstrained allocations satisfy the feasibility requirements of the orthogonal designs, including any lower bounds such as n a ≥ d n_a\ge d n a ≥ d ; otherwise the corresponding constrained KKT solution is used. The allocation minimizing the pooled estimation variance is
n a p o o l = N π a σ a ∑ b π b σ b , min V p o o l = D N ( ∑ a π a σ a ) 2 , (109) n_a^{\mathrm{pool}}=N\frac{\pi_a\sigma_a}{\sum_b\pi_b\sigma_b},\qquad\min V_{\mathrm{pool}}=\frac{D}{N}\left(\sum_a\pi_a\sigma_a\right)^2, \tag*{(109)} n a pool = N ∑ b π b σ b π a σ a , min V pool = N D ( a ∑ π a σ a ) 2 , (109) whereas the allocation minimizing separate-estimation risk is
n a s e p = N π a σ a ∑ b π b σ b , min V s e p = D N ( ∑ a π a σ a ) 2 . (110) n_a^{\mathrm{sep}}=N\frac{\sqrt{\pi_a}\sigma_a}{\sum_b\sqrt{\pi_b}\sigma_b},\qquad\min V_{\mathrm{sep}}=\frac{D}{N}\left(\sum_a\sqrt{\pi_a}\sigma_a\right)^2. \tag*{(110)} n a sep = N ∑ b π b σ b π a σ a , min V sep = N D ( a ∑ π a σ a ) 2 . (110) If each architecture may choose its own allocation, full sharing has lower optimized risk exactly when
B π < D N [ ( ∑ a π a σ a ) 2 − ( ∑ a π a σ a ) 2 ] . (111) B_{\pi}<\frac{D}{N}\left[\left(\sum_a\sqrt{\pi_a}\sigma_a\right)^2-\left(\sum_a\pi_a\sigma_a\right)^2\right]. \tag*{(111)} B π < N D ( a ∑ π a σ a ) 2 − ( a ∑ π a σ a ) 2 . (111) For completeness, minimizing ∑ a c a / n a \sum_a c_a/n_a ∑ a c a / n a with c a > 0 c_a>0 c a > 0 , n a ≥ d n_a\ge d n a ≥ d , and ∑ a n a = N > M d \sum_a n_a=N>Md ∑ a n a = N > M d gives n a = max { d , c a / t } n_a=\max\{d,\sqrt{c_a/t}\} n a = max { d , c a / t } , where t > 0 t>0 t > 0 is chosen to meet the budget. The Karush–Kuhn–Tucker (KKT) conditions are the stationarity and complementary-slackness conditions for this constrained convex problem: a coordinate above its floor has derivative − c a / n a 2 + t = 0 -c_a/n_a^2+t=0 − c a / n a 2 + t = 0 ; a coordinate at the floor need not satisfy that equality. If N = M d N=Md N = M d , every n a = d n_a=d n a = d . Integer allocations require a separate discrete solution.
This result concerns data acquisition or replay allocation.
Any linear sharing architecture is a projection problem
A large class of tying and conditioning rules can be described by a linear subspace of admissible role-wise functions. The following theorem gives an exact bias–dimension decomposition.
Theorem 18 (Exact risk of a linear sharing subspace). Let J J J role cells have true parameter vectors θ 1 , … , θ J ∈ R D \theta_1,\ldots,\theta_J\in\mathbb{R}^D θ 1 , … , θ J ∈ R D . Suppose their unconstrained stacked estimator satisfies
θ ^ = θ + ζ , E ζ = 0 , Cov ( ζ ) = ν I J D . (112) \widehat{\theta}=\theta+\zeta,\qquad\mathbb{E}\zeta=0,\qquad\operatorname{Cov}(\zeta)=\nu I_{JD}. \tag*{(112)} θ = θ + ζ , E ζ = 0 , Cov ( ζ ) = ν I J D . (112) *Here stacking means concatenating the J J J length-D D D vectors in R J D \mathbb{R}^{JD} R J D , and Cov \operatorname{Cov} Cov denotes covariance. The subspace must be chosen before observing this fitting noise. Let U ⊆ R J D \mathcal{U}\subseteq\mathbb{R}^{JD} U ⊆ R J D be a fixed linear sharing
subspace, let P U P_U P U be its Euclidean orthogonal projector, and fit θ ^ U = P U θ ^ \hat{\theta}_U=P_U\hat{\theta} θ ^ U = P U θ ^ . Define the role-averaged squared parameter error by R ( U ) = J − 1 E ∥ θ ^ U − θ ∥ 2 2 \mathcal{R}(U)=J^{-1}\mathbb{E}\left\|\hat{\theta}_U-\theta\right\|_2^2 R ( U ) = J − 1 E θ ^ U − θ 2 2 . Then
R ( U ) = 1 J ∥ ( I − P U ) θ ∥ 2 2 + v J dim ( U ) . (113) \mathcal{R}(U)=\frac{1}{J}\left\|(I-P_U)\theta\right\|_2^2+\frac{v}{J}\dim(U). \tag*{(113)} R ( U ) = J 1 ∥ ( I − P U ) θ ∥ 2 2 + J v dim ( U ) . (113)
The first term is exactly the squared approximation cost imposed by the fixed subspace; the second charges one isotropic-noise degree of freedom for every fitted scalar parameter. The partition result in Theorem 2 is the special case in which U U U contains arrays constant inside every expert group. More generally, tied layers, shared cores, additive depth embeddings, and fixed low-dimensional modulation bases correspond to different choices of U U U .
An exact cross-depth tying frontier
We now instantiate Theorem 18 for architectures that reuse experts across depth. Index a semantic role by a ∈ [ M ] a\in[M] a ∈ [ M ] and depth by ℓ ∈ [ L ] \ell\in[L] ℓ ∈ [ L ] . The true local operator is A a ℓ ∈ R p × d A_{a\ell}\in\mathbb{R}^{p\times d} A a ℓ ∈ R p × d , each cell has an orthogonal design with n ≥ d n\ge d n ≥ d samples and homoscedastic noise σ 2 \sigma^2 σ 2 , with noise independent across cells, and
V = p d σ 2 n (114) V=\frac{pd\sigma^2}{n} \tag*{(114)} V = n p d σ 2 (114)
would be the expected risk of one cell-specific OLS fit. Define
A ˉ = 1 M L ∑ a , ℓ A a ℓ , A ˉ a ⋅ = 1 L ∑ ℓ A a ℓ , A ˉ ⋅ ℓ = 1 M ∑ a A a ℓ , (115) \bar{A}=\frac{1}{ML}\sum_{a,\ell}A_{a\ell},\qquad\bar{A}_{a\cdot}=\frac{1}{L}\sum_{\ell}A_{a\ell},\qquad\bar{A}_{\cdot\ell}=\frac{1}{M}\sum_a A_{a\ell}, \tag*{(115)} A ˉ = M L 1 a , ℓ ∑ A a ℓ , A ˉ a ⋅ = L 1 ℓ ∑ A a ℓ , A ˉ ⋅ ℓ = M 1 a ∑ A a ℓ , (115)
B r o l e = 1 M ∑ a ∥ A ˉ a ⋅ − A ˉ ∥ F 2 , B d e p t h = 1 L ∑ ℓ ∥ A ˉ ⋅ ℓ − A ˉ ∥ F 2 , (116) B_{\mathrm{role}}=\frac{1}{M}\sum_a\left\|\bar{A}_{a\cdot}-\bar{A}\right\|_F^2,\qquad B_{\mathrm{depth}}=\frac{1}{L}\sum_\ell\left\|\bar{A}_{\cdot\ell}-\bar{A}\right\|_F^2, \tag*{(116)} B role = M 1 a ∑ A ˉ a ⋅ − A ˉ F 2 , B depth = L 1 ℓ ∑ A ˉ ⋅ ℓ − A ˉ F 2 , (116)
R a ℓ i n t = A a ℓ − A ˉ a ⋅ − A ˉ ⋅ ℓ + A ˉ , R^{\mathrm{int}}_{a\ell}=A_{a\ell}-\bar{A}_{a\cdot}-\bar{A}_{\cdot\ell}+\bar{A}, R a ℓ int = A a ℓ − A ˉ a ⋅ − A ˉ ⋅ ℓ + A ˉ ,
B i n t = 1 M L ∑ a , ℓ ∥ R a ℓ i n t ∥ F 2 . (117) B_{\mathrm{int}}=\frac{1}{ML}\sum_{a,\ell}\left\|R^{\mathrm{int}}_{a\ell}\right\|_F^2. \tag*{(117)} B int = M L 1 a , ℓ ∑ R a ℓ int F 2 . (117)
The dot in a subscript means averaging over that index. The role effect changes a a a only; the depth effect changes ℓ \ell ℓ only. The interaction R a ℓ i n t R^{\mathrm{int}}_{a\ell} R a ℓ int is what remains when the two effects cannot explain a cell jointly. Under the balanced Frobenius inner product, these components are orthogonal (their pairwise inner products sum to zero). We use this as an algebraic two-way analysis of variance (ANOVA) decomposition.
Theorem 19 (Exact risks for cross-layer reuse patterns). Under the balanced cell model above, the exact role–cell-averaged excess prediction risks are those in Table 30 .
Table 30. Exact cross-depth sharing risks. “Role-tied” means that the same semantic expert is reused at every depth; “additive depth-conditioned” allows a semantic expert main effect plus a depth main effect but no role–depth interaction.
Consequently:
role-tied beats cell-specific ⟺ B d e p t h + B i n t < V ( 1 − 1 L ) , (118) \textit{role-tied beats cell-specific} \Longleftrightarrow B_{\mathrm{depth}} + B_{\mathrm{int}} < V\left(1-\frac{1}{L}\right), \tag*{(118)} role-tied beats cell-specific ⟺ B depth + B int < V ( 1 − L 1 ) , (118)
additive beats role-tied ⟺ B d e p t h > V L − 1 M L , (119) \textit{additive beats role-tied} \Longleftrightarrow B_{\mathrm{depth}} > V\frac{L-1}{ML}, \tag*{(119)} additive beats role-tied ⟺ B depth > V M L L − 1 , (119)
additive beats cell-specific ⟺ B i n t < V ( M − 1 ) ( L − 1 ) M L . (120) \textit{additive beats cell-specific} \Longleftrightarrow B_{\mathrm{int}} < V\frac{(M-1)(L-1)}{ML}. \tag*{(120)} additive beats cell-specific ⟺ B int < V M L ( M − 1 ) ( L − 1 ) . (120)
These formulas separate three reasons that cross-layer reuse may fail. A globally tied pool pays semantic-role variation, depth variation, and their interaction. Tying the same semantic expert across depth removes the role main effect but still pays depth shift and role–depth interaction. A depth-conditioned shared expert removes both main effects; only the interaction remains. Within this additive operator model, depth conditioning is useful precisely when the removed depth effect exceeds the variance cost of its additional ( L − 1 ) D (L-1)D ( L − 1 ) D scalar degrees of freedom. Interpreting a nonlinear depth embedding through this subspace requires an explicit realization or local-linearization argument. If B i n t B_{\mathrm{int}} B int is large, neither global tying nor additive depth conditioning is sufficient, and layer-specific residual capacity is statistically justified.
The same calculation extends to several architectural axes.
Corollary 20 (Multi-axis additive reuse). Suppose role cells are indexed by r r r factors of sizes m 1 , … , m r m_1,\ldots,m_r m 1 , … , m r , with J = ∏ k = 1 r m k J=\prod_{k=1}^{r}m_k J = ∏ k = 1 r m k balanced cells. Let U a d d \mathcal{U}_{\mathrm{add}} U add contain a grand mean plus one zero-sum main effect for every factor, and define the normalized interaction residual
B ≥ 2 = 1 J ∥ ( I − P U a d d ) θ ∥ 2 2 . (121) B_{\geq2}=\frac{1}{J}\left\|(I-P_{\mathcal{U}_{\mathrm{add}}})\theta\right\|_2^2. \tag*{(121)} B ≥ 2 = J 1 ∥ ( I − P U add ) θ ∥ 2 2 . (121) It contains all interactions of order at least two. Then
R ( U a d d ) = B ≥ 2 + V J [ 1 + ∑ k = 1 r ( m k − 1 ) ] . (122) \mathcal{R}(\mathcal{U}_{\mathrm{add}})=B_{\geq2}+\frac{V}{J}\left[1+\sum_{k=1}^{r}(m_k-1)\right]. \tag*{(122)} R ( U add ) = B ≥ 2 + J V [ 1 + k = 1 ∑ r ( m k − 1 ) ] . (122) This covers, for example, semantic role × \times × layer × \times × modality or role × \times × reasoning position × \times × tool state. For the declared linear subspaces, adding a factor-specific correction is worthwhile only when the removed component exceeds its estimation price. Corollary 40 gives the covariance-weighted version without balanced isotropic noise.
Proofs
Proof of Theorem 16. Orthogonality gives A ^ a = A a + E a \widehat{A}_a=A_a+E_a A a = A a + E a , with centered, uncorrelated entries of variance v a v_a v a and independent error matrices across roles. Let E ‾ π = ∑ b π b E b \overline{E}_{\pi}=\sum_b\pi_bE_b E π = ∑ b π b E b . Then
A ^ a ( λ ) − A a = λ ( A ‾ π − A a ) + ( 1 − λ ) E a + λ E ‾ π . (123) \widehat{A}_a^{(\lambda)}-A_a=\lambda(\overline{A}_{\pi}-A_a)+(1-\lambda)E_a+\lambda\overline{E}_{\pi}. \tag*{(123)} A a ( λ ) − A a = λ ( A π − A a ) + ( 1 − λ ) E a + λ E π . (123)
The role-averaged squared bias is λ 2 B π \lambda^2B_{\pi} λ 2 B π . For each scalar coefficient, the weighted variance is
∑ a π a Var ( ( 1 − λ ) E a + λ E ‾ π ) = ( 1 − λ ) 2 ∑ a π a v a + ( 2 λ − λ 2 ) ∑ a π a 2 v a . (124) \sum_a\pi_a\operatorname{Var}\left((1-\lambda)E_a+\lambda\overline{E}_{\pi}\right)
=(1-\lambda)^2\sum_a\pi_av_a+(2\lambda-\lambda^2)\sum_a\pi_a^2v_a. \tag*{(124)} a ∑ π a Var ( ( 1 − λ ) E a + λ E π ) = ( 1 − λ ) 2 a ∑ π a v a + ( 2 λ − λ 2 ) a ∑ π a 2 v a . (124)
Multiplying by D D D and adding the bias gives Equation (105) . Setting λ = 0 , 1 \lambda=0,1 λ = 0 , 1 , subtracting the endpoint risks, and minimizing the convex quadratic prove the remaining statements.
Proof of Corollary 17. Minimizing ∑ a c a / n a \sum_a c_a/n_a ∑ a c a / n a subject to ∑ a n a = N \sum_a n_a=N ∑ a n a = N gives n a ∝ c a n_a\propto\sqrt{c_a} n a ∝ c a by Lagrange multipliers, or equivalently by Cauchy–Schwarz. For pooled estimation, c a = π a 2 σ a 2 c_a=\pi_a^2\sigma_a^2 c a = π a 2 σ a 2 , yielding n a ∝ π a σ a n_a\propto\pi_a\sigma_a n a ∝ π a σ a and the first minimum. For separate estimation, c a = π a σ a 2 c_a=\pi_a\sigma_a^2 c a = π a σ a 2 , yielding n a ∝ π a σ a n_a\propto\sqrt{\pi_a}\sigma_a n a ∝ π a σ a and the second. Comparing the two optimized endpoint risks gives Equation (111) .
Proof of Theorem 18. Write P = P U P = P_{\mathcal{U}} P = P U . Because P θ − θ = − ( I − P ) θ P\theta- \theta= -(I-P)\theta P θ − θ = − ( I − P ) θ is orthogonal to P ζ P\zeta P ζ and E ζ = 0 \mathbb{E}\zeta= 0 E ζ = 0 ,
E ∥ P θ ^ − θ ∥ 2 2 = ∥ ( I − P ) θ ∥ 2 2 + E ∥ P ζ ∥ 2 2 . (125) \mathbb{E}\left\|P\hat{\theta}-\theta\right\|_2^2 = \left\|(I-P)\theta\right\|_2^2 + \mathbb{E}\left\|P\zeta\right\|_2^2. \tag*{(125)} E P θ ^ − θ 2 2 = ∥ ( I − P ) θ ∥ 2 2 + E ∥ P ζ ∥ 2 2 . (125)
Moreover,
E ∥ P ζ ∥ 2 2 = tr ( P v I P T ) = v tr ( P ) = v dim ( U ) . (126) \mathbb{E}\left\|P\zeta\right\|_2^2 = \operatorname{tr}(P vI P^{\mathsf{T}}) = v\operatorname{tr}(P) = v\dim(\mathcal{U}). \tag*{(126)} E ∥ P ζ ∥ 2 2 = tr ( P v I P T ) = v tr ( P ) = v dim ( U ) . (126)
Divide by J J J .
Proof of Theorem 19. Stack the M L ML M L operator vectors and apply Theorem 18 with scalar noise variance v = σ 2 / n v = \sigma^2/n v = σ 2 / n and block dimension D = p d D = pd D = p d . The cell-specific, globally tied, role-tied, depth-specific, and additive subspaces have dimensions M L D MLD M L D , D D D , M D MD M D , L D LD L D , and ( M + L − 1 ) D (M+L-1)D ( M + L − 1 ) D , respectively. Orthogonal two-way ANOVA gives
1 M L ∑ a , ℓ ∥ A a ℓ − A ˉ ∥ F 2 = B r o l e + B d e p t h + B i n t , (127) \frac{1}{ML}\sum_{a,\ell}\left\|A_{a\ell}-\bar{A}\right\|_F^2 = B_{\mathrm{role}} + B_{\mathrm{depth}} + B_{\mathrm{int}}, \tag*{(127)} M L 1 a , ℓ ∑ A a ℓ − A ˉ F 2 = B role + B depth + B int , (127)
while projection onto role means leaves B d e p t h + B i n t B_{\mathrm{depth}} + B_{\mathrm{int}} B depth + B int , projection onto depth means leaves B r o l e + B i n t B_{\mathrm{role}} + B_{\mathrm{int}} B role + B int , and projection onto the additive subspace leaves B i n t B_{\mathrm{int}} B int . Substituting each residual and dimension into Equation (113) gives the table. Pairwise subtraction gives Equations (118) to (120) .
Proof of Corollary 20. The additive subspace has one grand-mean block and ( m k − 1 ) (m_k-1) ( m k − 1 ) independent main-effect blocks for axis k k k , hence dimension D [ 1 + ∑ k ( m k − 1 ) ] D[1+\sum_k(m_k-1)] D [ 1 + ∑ k ( m k − 1 )] . Its orthogonal residual consists of interactions of order at least two. Apply Theorem 18 .
Architecture-Specific Functional Routing Bounds
Overview. A routing change matters through the event output it changes, not just expert-index disagreement. We bound the generic cost ρ j \rho_j ρ j from Equation (85) for additive soft and uniform top-k k k gates, capacity-limited row-separable assignments, additive hierarchies, and preselected chains. Full soft dispatch also changes expert inputs and requires the additional term in Theorem 23 . Each bound applies only to its stated interface. The transport notation follows Peyré & Cuturi [29 ] ; the inequalities are proved directly. Sections I.1–I.5 treat the individual routing interfaces; Section I.6 summarizes their cost terms, and Section I.7 supplies the remaining proofs.
Additive soft and top-k gates as transport over expert functions
Fix a common expert input u u u and let v e ( u ) v_e(u) v e ( u ) be the routed residual produced by expert e e e in a normed vector space. For normalized gates g , h ∈ Δ K : = { w ∈ R K : w e ≥ 0 , ∑ e w e = 1 } g,h \in\Delta_K := \{w \in\mathbb{R}^K : w_e \ge0,\ \sum_e w_e = 1\} g , h ∈ Δ K := { w ∈ R K : w e ≥ 0 , ∑ e w e = 1 } , define the discrete coupling set
Π ( g , h ) = { γ ∈ R + K × K : ∑ e ′ γ e e ′ = g e , ∑ e γ e e ′ = h e ′ } , (128) \Pi(g,h) = \left\{\gamma\in\mathbb{R}_{+}^{K\times K} : \sum_{e'}\gamma_{ee'} = g_e,\ \sum_e\gamma_{ee'} = h_{e'}\right\}, \tag*{(128)} Π ( g , h ) = { γ ∈ R + K × K : e ′ ∑ γ e e ′ = g e , e ∑ γ e e ′ = h e ′ } , (128)
and the input-dependent functional transport cost
W u ( g , h ) = min γ ∈ Π ( g , h ) ∑ e , e ′ γ e e ′ c u ( e , e ′ ) , c u ( e , e ′ ) = ∥ v e ( u ) − v e ′ ( u ) ∥ . (129) W_u(g,h) = \min_{\gamma\in\Pi(g,h)}\sum_{e,e'}\gamma_{ee'}c_u(e,e'),\qquad c_u(e,e') = \left\|v_e(u)-v_{e'}(u)\right\|. \tag*{(129)} W u ( g , h ) = γ ∈ Π ( g , h ) min e , e ′ ∑ γ e e ′ c u ( e , e ′ ) , c u ( e , e ′ ) = ∥ v e ( u ) − v e ′ ( u ) ∥ . (129)
Theorem 21 (Functional transport bound for additive routing). Let
M ( u , g ) = B ( u ) + ∑ e = 1 K g e v e ( u ) , (130) M(u,g) = B(u) + \sum_{e=1}^{K}g_ev_e(u), \tag*{(130)} M ( u , g ) = B ( u ) + e = 1 ∑ K g e v e ( u ) , (130) where B ( u ) B(u) B ( u ) contains every dense or always-active shared path and where changing the gate does not change u u u or the collection { v e ( u ) } \{v_e(u)\} { v e ( u )} . Then
∥ M ( u , g ) − M ( u , h ) ∥ ≤ W u ( g , h ) ≤ D u 2 ∥ g − h ∥ 1 , D u = max e , e ′ c u ( e , e ′ ) . (131) \left\|M(u,g)-M(u,h)\right\| \le W_u(g,h) \le\frac{D_u}{2}\left\|g-h\right\|_1,\qquad D_u = \max_{e,e'}c_u(e,e'). \tag*{(131)} ∥ M ( u , g ) − M ( u , h ) ∥ ≤ W u ( g , h ) ≤ 2 D u ∥ g − h ∥ 1 , D u = e , e ′ max c u ( e , e ′ ) . (131) The first inequality can be strictly sharper than the diameter–total-variation bound when most mismatched mass moves between functionally similar experts.
The shared term B ( u ) B(u) B ( u ) cancels exactly. Thus an always-active expert can improve the event while contributing nothing to the discrepancy between two conditional gates at the same input. Conversely, a dominant shared path can make natural task accuracy insensitive to the routed residual, which is one mechanism behind the causal non-identifiability result in Proposition 5.
Named architecture instances. Mixtral combines two selected SwiGLU experts using a softmax over the selected logits [18 ] ; this is weighted top-2, not the equal-1/2 special case below. DeepSeekMoE combines always-active shared experts with weighted routed experts [4 ] , corresponding to B ( u ) B(u) B ( u ) plus the conditional sum in Equation (130) . The actual gate normalization and selected mass must be retained; sub-probability gates use the null-atom augmentation below. Shared paths cancel only between two gates of the same fixed model at the same input, not between separately trained models.
For uniform top-k k k routing, the transport cost becomes a matching problem.
Corollary 22 (Uniform top-k k k functional matching). Let S = { s 1 , … , s k } S = \{s_1, \ldots, s_k\} S = { s 1 , … , s k } and T = { t 1 , … , t k } T = \{t_1, \ldots, t_k\} T = { t 1 , … , t k } be two k k k -element expert sets, and let both gates assign mass 1 / k 1/k 1/ k to every selected expert. Then
W u ( g S , g T ) = 1 k min π ∈ S k ∑ r = 1 k ∥ v s r ( u ) − v t π ( r ) ( u ) ∥ . (132) W_u(g_S,g_T) = \frac{1}{k} \min_{\pi\in\mathfrak{S}_k} \sum_{r=1}^{k} \lVert v_{s_r}(u) - v_{t_{\pi(r)}}(u) \rVert. \tag*{(132)} W u ( g S , g T ) = k 1 π ∈ S k min r = 1 ∑ k ∥ v s r ( u ) − v t π ( r ) ( u )∥ . (132) Consequently, two disjoint top-k k k supports can have small functional route cost, while replacing one selected expert can be severe if the substitute is functionally distant. Here S k \mathfrak{S}_k S k is the set of the k ! k! k ! permutations of { 1 , … , k } \{1,\ldots,k\} { 1 , … , k } , so the minimum matches each selected expert to one expert in the other set. The formula is invariant to enumeration. For nonuniform within-support weights, use the general transport problem in Equation (129) ; support overlap alone does not determine the route cost.
Capacity drops and row-separable expert-choice assignment
Token-choice and expert-choice routers may drop, pad, or reroute mass under capacity constraints [44 , 22 ] . A zero-residual “drop” atom makes the associated functional cost explicit. For a sub-probability gate g ∈ R + K g \in\mathbb{R}_{+}^{K} g ∈ R + K with ∑ e g e ≤ 1 \sum_e g_e \le1 ∑ e g e ≤ 1 , introduce atom 0 0 0 with v 0 ( u ) = 0 v_0(u)=0 v 0 ( u ) = 0 and augmented mass
g ~ 0 = 1 − ∑ e = 1 K g e , g ~ e = g e ( e ≥ 1 ) . (133) \widetilde{g}_0 = 1 - \sum_{e=1}^{K} g_e,\qquad\widetilde{g}_e = g_e \quad(e \ge1). \tag*{(133)} g 0 = 1 − e = 1 ∑ K g e , g e = g e ( e ≥ 1 ) . (133)
Applying Theorem 21 to g ~ \widetilde{g} g and h ~ \widetilde{h} h charges
c u ( e , 0 ) = ∥ v e ( u ) ∥ (134) c_u(e,0) = \lVert v_e(u) \rVert \tag*{(134)} c u ( e , 0 ) = ∥ v e ( u )∥ (134)
for dropping the contribution of expert e e e . Hence a drop-rate statistic is insufficient unless it is weighted by the magnitude of the dropped residual.
For T T T token or slot states u t u_t u t , suppose an assignment matrix acts row-separably after the u t u_t u t are fixed, so row t t t produces M ( u t , g t ) M(u_t,g_t) M ( u t , g t ) as in Equation (130) . If matrices G , H G,H G , H induce row gates g t , h t g_t,h_t g t , h t , then under the sum norm on the batch output,
∑ t = 1 T ∥ M ( u t , g t ) − M ( u t , h t ) ∥ ≤ ∑ t = 1 T W u t ( g ~ t , h ~ t ) . (135) \sum_{t=1}^{T} \lVert M(u_t,g_t) - M(u_t,h_t) \rVert\le\sum_{t=1}^{T} W_{u_t}(\widetilde{g}_t,\widetilde{h}_t). \tag*{(135)} t = 1 ∑ T ∥ M ( u t , g t ) − M ( u t , h t )∥ ≤ t = 1 ∑ T W u t ( g t , h t ) . (135)
The same statement divided by T T T holds for the average norm, while a maximum norm is controlled by the maximum row cost. Joint capacity constraints may couple how the rows are selected, but no row-independence assumption is used once the two assignment matrices are fixed. Dispatch mechanisms that change expert-slot inputs use the generic fixed-plan discrepancy; Theorem 23 gives a bound when dispatch sensitivity is controlled.
Full Soft MoE constructs weighted token combinations as expert inputs before combining their outputs [31 ] . The common-input gate bound alone is therefore insufficient. Let x 1 , … , x T ∈ R d x_1,\ldots,x_T \in\mathbb{R}^d x 1 , … , x T ∈ R d be fixed token states and let there be Q Q Q slots, meaning Q Q Q intermediate vectors sent to designated expert calls. A dispatch matrix D ∈ R + T × Q D \in\mathbb{R}_{+}^{T \times Q} D ∈ R + T × Q has column sums one; a combine matrix C ∈ R + T × Q C \in\mathbb{R}_{+}^{T \times Q} C ∈ R + T × Q has row sums one. With fixed slot-to-expert assignment e ( q ) e(q) e ( q ) , define
u q ( D ) = ∑ t D t q x t , v q ( D ) = E e ( q ) ( u q ( D ) ) , y t ( D , C ) = ∑ q C t q v q ( D ) . (136) u_q(D) = \sum_t D_{tq}x_t,\qquad v_q(D) = E_{e(q)}(u_q(D)),\qquad y_t(D,C) = \sum_q C_{tq}v_q(D). \tag*{(136)} u q ( D ) = t ∑ D tq x t , v q ( D ) = E e ( q ) ( u q ( D )) , y t ( D , C ) = q ∑ C tq v q ( D ) . (136)
Let ( D ′ , C ′ ) (D',C') ( D ′ , C ′ ) be another pair with the same normalizations and the same expert functions. Expert E e ( q ) E_{e(q)} E e ( q ) is assumed L q L_q L q -Lipschitz on the convex hull of the token states, using the respective input and output norms. The convex hull consists of all their nonnegative weighted averages with total weight one.
Let W D o u t W_D^{\mathrm{out}} W D out be discrete transport over slot outputs { v q ( D ) } \{v_q(D)\} { v q ( D )} , with ground cost ∥ v q ( D ) − v q ′ ( D ) ∥ \lVert v_q(D)-v_q'(D)\rVert ∥ v q ( D ) − v q ′ ( D )∥ . Let W X i n W_X^{\mathrm{in}} W X in be transport over the token states, with cost ∥ x t − x t ′ ∥ \lVert x_t-x_{t'}\rVert ∥ x t − x t ′ ∥ . Define C ‾ q ′ = T − 1 ∑ t C t q ′ \overline{C}'_q=T^{-1}\sum_t C'_{tq} C q ′ = T − 1 ∑ t C tq ′ , the average output-token weight assigned to slot q q q .
Theorem 23 (Dispatch-plus-combine discrepancy for full soft routing). Under the normalization and Lipschitz assumptions above,
1 T ∑ t ∥ y t ( D , C ) − y t ( D ′ , C ′ ) ∥ ≤ 1 T ∑ t W D o u t ( C t , : , C t , : ′ ) + ∑ q C ‾ q ′ L q W X i n ( D : , q , D : , q ′ ) . (137) \frac{1}{T}\sum_t \lVert y_t(D,C)-y_t(D',C')\rVert
\leq\frac{1}{T}\sum_t W_D^{\mathrm{out}}(C_{t,:},C'_{t,:})
+\sum_q \overline{C}'_q L_q W_X^{\mathrm{in}}(D_{:,q},D'_{:,q}). \tag*{(137)} T 1 t ∑ ∥ y t ( D , C ) − y t ( D ′ , C ′ )∥ ≤ T 1 t ∑ W D out ( C t , : , C t , : ′ ) + q ∑ C q ′ L q W X in ( D : , q , D : , q ′ ) . (137) The reverse insertion gives another valid bound using output geometry at D ′ D' D ′ and average combine weights from C C C ; their minimum is valid. If D X = max t , t ′ ∥ x t − x t ′ ∥ D_X=\max_{t,t'}\lVert x_t-x_{t'}\rVert D X = max t , t ′ ∥ x t − x t ′ ∥ , each input transport term is at most $D_X\lVert D_{:,q}-D'_{:,q}\rVert_1/2.
Proof. Insert y t ( D , C ′ ) y_t(D,C') y t ( D , C ′ ) . The first difference keeps slot outputs fixed and is bounded by output transport. The second difference is at most ∑ q C t q ′ ∥ E e ( q ) ( u q ( D ) ) − E e ( q ) ( u q ( D ′ ) ) ∥ \sum_q C'_{tq}\lVert E_{e(q)}(u_q(D))-E_{e(q)}(u_q(D'))\rVert ∑ q C tq ′ ∥ E e ( q ) ( u q ( D )) − E e ( q ) ( u q ( D ′ ))∥ . Use the expert’s Lipschitz bound and the barycenter transport bound ∥ u q ( D ) − u q ( D ′ ) ∥ ≤ W X i n ( D : , q , D : , q ′ ) \lVert u_q(D)-u_q(D')\rVert\leq W_X^{\mathrm{in}}(D_{:,q},D'_{:,q}) ∥ u q ( D ) − u q ( D ′ )∥ ≤ W X in ( D : , q , D : , q ′ ) . Average over t t t . Reversing the insertion proves the alternative; the diameter inequality follows from Theorem 21 applied to token vectors.
This extends the routing analysis beyond the common-expert-input assumption. If dispatch is unchanged, the second term vanishes. If the slot-to-expert assignment or the token states also change, an additional expert-substitution or input-state discrepancy must be included. The theorem bounds the route component of a fixed trained event, not the learned expert approximation error already charged elsewhere.
Hierarchical additive routing separates coarse and fine errors
Consider a two-level additive hierarchy with groups r ∈ [ G ] r \in[G] r ∈ [ G ] , within-group experts e ∈ [ K r ] e \in[K_r] e ∈ [ K r ] , outer gate p ∈ Δ G p \in\Delta_G p ∈ Δ G , and conditional inner gates q r ∈ Δ K r q_r \in\Delta_{K_r} q r ∈ Δ K r . All vectors below are evaluated at the same fixed input. Define
m r ( q r ) = ∑ e q r , e v r , e , M ( p , q ) = ∑ r p r m r ( q r ) . (138) m_r(q_r)=\sum_e q_{r,e}v_{r,e},\qquad M(p,q)=\sum_r p_rm_r(q_r). \tag*{(138)} m r ( q r ) = e ∑ q r , e v r , e , M ( p , q ) = r ∑ p r m r ( q r ) . (138)
Proposition 24 (Hierarchical functional route decomposition). For two hierarchical routes ( p , q ) (p,q) ( p , q ) and ( p ′ , q ′ ) (p',q') ( p ′ , q ′ ) ,
∥ M ( p , q ) − M ( p ′ , q ′ ) ∥ ≤ W q o u t ( p , p ′ ) + ∑ r = 1 G p r ′ W r i n ( q r , q r ′ ) . (139) \lVert M(p,q)-M(p',q')\rVert
\leq W_q^{\mathrm{out}}(p,p')+\sum_{r=1}^{G}p'_rW_r^{\mathrm{in}}(q_r,q'_r). \tag*{(139)} ∥ M ( p , q ) − M ( p ′ , q ′ )∥ ≤ W q out ( p , p ′ ) + r = 1 ∑ G p r ′ W r in ( q r , q r ′ ) . (139) where the outer transport uses
c q o u t ( r , r ′ ) = ∥ m r ( q r ) − m r ′ ( q r ′ ) ∥ , (140) c_q^{\mathrm{out}}(r,r')=\lVert m_r(q_r)-m_{r'}(q_{r'})\rVert, \tag*{(140)} c q out ( r , r ′ ) = ∥ m r ( q r ) − m r ′ ( q r ′ )∥ , (140) and the inner transport in group r r r uses
c r i n ( e , e ′ ) = ∥ v r , e − v r , e ′ ∥ . (141) c_r^{\mathrm{in}}(e,e')=\lVert v_{r,e}-v_{r,e'}\rVert. \tag*{(141)} c r in ( e , e ′ ) = ∥ v r , e − v r , e ′ ∥ . (141) The reverse insertion gives a second valid bound with ( p , q ) (p,q) ( p , q ) and ( p ′ , q ′ ) (p',q') ( p ′ , q ′ ) interchanged; their minimum may be used.
The first term measures a coarse group mistake, and the second measures fine selection errors after the outer mass is fixed. A single flat route-accuracy number cannot reveal which level consumes the reuse gain.
Preselected sequential expert chains have within-event amplification
Sequential architectures call experts one after another inside one routed event [42 ] . The theorem here concerns two preselected, equal-length sequences. If a later expert choice adapts to an earlier expert output, each adaptive choice must instead be made a separate routed event or the entire adaptive policy must be compared through the generic plan-level distance.
Let stage s ∈ [ Q ] s \in[Q] s ∈ [ Q ] map metric space ( X s − 1 , d s − 1 ) (\mathcal{X}_{s-1},d_{s-1}) ( X s − 1 , d s − 1 ) to ( X s , d s ) (\mathcal{X}_s,d_s) ( X s , d s ) through F s , e F_{s,e} F s , e . Consider
γ = ( e 1 , … , e Q ) , γ ′ = ( e 1 ′ , … , e Q ′ ) , (142) \gamma=(e_1,\ldots,e_Q), \qquad\gamma'=(e'_1,\ldots,e'_Q), \tag*{(142)} γ = ( e 1 , … , e Q ) , γ ′ = ( e 1 ′ , … , e Q ′ ) , (142)
and suppose every suffix map F r , e r F_{r,e_r} F r , e r from the first sequence is L r L_r L r -Lipschitz on the reached hybrid region. Let
x s − 1 ′ = F s − 1 , e s − 1 ′ ∘ ⋯ ∘ F 1 , e 1 ′ ( x ) , A s i n = ∏ r = s + 1 Q L r , (143) x'_{s-1}=F_{s-1,e'_{s-1}}\circ\cdots\circ F_{1,e'_1}(x), \qquad A_s^{\mathrm{in}}=\prod_{r=s+1}^{Q}L_r, \tag*{(143)} x s − 1 ′ = F s − 1 , e s − 1 ′ ∘ ⋯ ∘ F 1 , e 1 ′ ( x ) , A s in = r = s + 1 ∏ Q L r , (143)
with the empty prefix and product interpreted as the identity and one.
Theorem 25 (Sequential route-plan discrepancy). The output maps F γ = F Q , e Q ∘ ⋯ ∘ F 1 , e 1 F_\gamma=F_{Q,e_Q}\circ\cdots\circ F_{1,e_1} F γ = F Q , e Q ∘ ⋯ ∘ F 1 , e 1 and F γ ′ F_{\gamma'} F γ ′ satisfy
d Q ( F γ ( x ) , F γ ′ ( x ) ) ≤ ∑ s = 1 Q A s i n d s ( F s , e s ( x s − 1 ′ ) , F s , e s ′ ( x s − 1 ′ ) ) . (144) d_Q(F_\gamma(x),F_{\gamma'}(x))\le\sum_{s=1}^{Q}A_s^{\mathrm{in}}d_s\left(F_{s,e_s}(x'_{s-1}),F_{s,e'_s}(x'_{s-1})\right). \tag*{(144)} d Q ( F γ ( x ) , F γ ′ ( x )) ≤ s = 1 ∑ Q A s in d s ( F s , e s ( x s − 1 ′ ) , F s , e s ′ ( x s − 1 ′ ) ) . (144)
Thus one early substitution can have a large cost in an expansive chain, while several index differences can be harmless when the substituted experts agree on the reached hybrid states. Variable-length chains can be embedded in a common maximum length by padding a terminated chain with an explicit identity/no-op transition on the terminal state. A zero map is generally not valid padding because it changes the state.
Combining Theorem 25 with the outer amplification A j A_j A j in Theorem 4 produces two scales of sensitivity: A s i n A_s^{\mathrm{in}} A s in measures propagation inside one routed event, while A j A_j A j measures propagation through later events. For a call at within-event position s s s of global event j j j , the natural terminal importance is their product.
Architecture-to-cost dictionary
Table 31. Architecture-specific quantities that upper-bound the generic functional routing term under their stated structural assumptions.
These are upper bounds on transition discrepancy. They can be estimated from counterfactual forward passes while preserving the active-expert budget.
Proofs
Proof of Theorem 21. For any γ ∈ Π ( g , h ) \gamma\in\Pi(g,h) γ ∈ Π ( g , h ) ,
∑ e ( g e − h e ) v e = ∑ e , e ′ γ e e ′ ( v e − v e ′ ) . (145) \sum_e (g_e-h_e)v_e = \sum_{e,e'} \gamma_{ee'}(v_e-v_{e'}). \tag*{(145)} e ∑ ( g e − h e ) v e = e , e ′ ∑ γ e e ′ ( v e − v e ′ ) . (145)
The triangle inequality gives
∥ ∑ e ( g e − h e ) v e ∥ ≤ ∑ e , e ′ γ e e ′ c u ( e , e ′ ) . (146) \left\|\sum_e (g_e-h_e)v_e\right\| \le\sum_{e,e'} \gamma_{ee'}c_u(e,e'). \tag*{(146)} e ∑ ( g e − h e ) v e ≤ e , e ′ ∑ γ e e ′ c u ( e , e ′ ) . (146)
Minimizing over couplings proves the first inequality. There exists a coupling that leaves the common mass min ( g e , h e ) \min(g_e,h_e) min ( g e , h e ) on the diagonal and transports only total mass TV ( g , h ) = ∥ g − h ∥ 1 / 2 \operatorname{TV}(g,h)=\lVert g-h\rVert_1/2 TV ( g , h ) = ∥ g − h ∥ 1 /2 off the diagonal. Every moved unit costs at most D u D_u D u , proving the second inequality.
Proof of Corollary 22. After restricting to the two k k k -point uniform measures and multiplying a coupling by k k k , the feasible set consists of nonnegative matrices whose row sums and column sums all equal one. This is the doubly stochastic polytope : a bounded set described by linear equality and inequality constraints. An extreme point cannot be a nontrivial convex combination of two other feasible points. Such points are permutation matrices (one entry equal to one per row and column), and a linear objective attains a minimum at one of them; this is the standard doubly stochastic matrix characterization ([29 ] ). Dividing by k k k gives Equation (132) . Shared atoms simply admit zero-cost matched pairs.
Proof of Proposition 24. Insert M ( p ′ , q ) M(p',q) M ( p ′ , q ) :
∥ M ( p , q ) − M ( p ′ , q ′ ) ∥ ≤ ∥ M ( p , q ) − M ( p ′ , q ) ∥ + ∥ M ( p ′ , q ) − M ( p ′ , q ′ ) ∥ . (147) \lVert M(p,q)-M(p',q')\rVert\le\lVert M(p,q)-M(p',q)\rVert+\lVert M(p',q)-M(p',q')\rVert. \tag*{(147)} ∥ M ( p , q ) − M ( p ′ , q ′ )∥ ≤ ∥ M ( p , q ) − M ( p ′ , q )∥ + ∥ M ( p ′ , q ) − M ( p ′ , q ′ )∥ . (147)
Apply Theorem 21 to the outer mixtures of vectors m r ( q r ) m_r(q_r) m r ( q r ) for the first term. For the second term, use the triangle inequality over groups and apply the same theorem inside each group. The reverse insertion is identical with the two routes exchanged.
Proof of Theorem 25. For s = 0 , … , Q s=0,\ldots,Q s = 0 , … , Q , let H s H_s H s use the prefix ( e 1 ′ , … , e s ′ ) (e'_1,\ldots,e'_s) ( e 1 ′ , … , e s ′ ) from γ ′ \gamma' γ ′ and the suffix ( e s + 1 , … , e Q ) (e_{s+1},\ldots,e_Q) ( e s + 1 , … , e Q ) from γ \gamma γ . Thus H 0 = F γ H_0=F_\gamma H 0 = F γ and H Q = F γ ′ H_Q=F_{\gamma'} H Q = F γ ′ . Adjacent hybrids H s − 1 H_{s-1} H s − 1 and H s H_s H s receive the same state x s − 1 ′ x'_{s-1} x s − 1 ′ at stage s s s , differ only between F s , e s F_{s,e_s} F s , e s and F s , e s ′ F_{s,e'_s} F s , e s ′ , and share the suffix ( e s + 1 , … , e Q ) (e_{s+1},\ldots,e_Q) ( e s + 1 , … , e Q ) . That suffix is A s i n A^{\mathrm{in}}_s A s in -Lipschitz from d s d_s d s to d Q d_Q d Q . The triangle inequality over H 0 , … , H Q H_0,\ldots,H_Q H 0 , … , H Q proves Equation (144) .
Endogenous Routing Feedback and Margin-Stable Traces
Overview. An early state error can change a later route and thereby create more error. We give sufficient support-stability conditions and probabilistic bounds for this feedback, retaining both router margins and expert-output geometry. Conditional margin control and weaker marginal control yield different guarantees. Sections J.1–J.2 treat soft-routing sensitivity and hard-support stability; Sections J.3–J.4 distinguish conditional from marginal margin bounds, followed by implications and proofs in Sections J.5–J.6.
Soft additive routing produces an additional Lipschitz term
Consider an additive routed event
F ( s ) = M ( s , g ( s ) ) = B ( s ) + ∑ e = 1 K g e ( s ) v e ( s ) , (148) F(s)=M(s,g(s))=B(s)+\sum_{e=1}^{K}g_e(s)v_e(s), \tag*{(148)} F ( s ) = M ( s , g ( s )) = B ( s ) + e = 1 ∑ K g e ( s ) v e ( s ) , (148)
with g ( s ) ∈ Δ K g(s) \in\Delta_K g ( s ) ∈ Δ K . The output is in a normed vector space and its distance is the norm of a difference; when one metric d d d is used on both sides, this compatibility is assumed. Otherwise use separate input metric and output norm. Assume that for every fixed gate h h h ,
d ( M ( s , h ) , M ( s ′ , h ) ) ≤ L x d ( s , s ′ ) . (149) d(M(s,h),M(s',h)) \le L_xd(s,s'). \tag*{(149)} d ( M ( s , h ) , M ( s ′ , h )) ≤ L x d ( s , s ′ ) . (149)
At state s ′ s' s ′ , define the functional transport cost W s ′ W_{s'} W s ′ as in Equation (129) , using the collection { v e ( s ′ ) } \{v_e(s')\} { v e ( s ′ )} at the common input s ′ s' s ′ .
Proposition 26 (Closed-loop sensitivity of an additive soft MoE event). If the router is functionally transport-Lipschitz,
W s ′ ( g ( s ) , g ( s ′ ) ) ≤ L W d ( s , s ′ ) , (150) W_{s'}(g(s),g(s')) \le L_Wd(s,s'), \tag*{(150)} W s ′ ( g ( s ) , g ( s ′ )) ≤ L W d ( s , s ′ ) , (150)
then
d ( F ( s ) , F ( s ′ ) ) ≤ ( L x + L W ) d ( s , s ′ ) . (151) d(F(s),F(s')) \le(L_x+L_W)d(s,s'). \tag*{(151)} d ( F ( s ) , F ( s ′ )) ≤ ( L x + L W ) d ( s , s ′ ) . (151)
In particular, if ∥ g ( s ) − g ( s ′ ) ∥ 1 ≤ L g d ( s , s ′ ) \lVert g(s)-g(s')\rVert_1 \le L_gd(s,s') ∥ g ( s ) − g ( s ′ ) ∥ 1 ≤ L g d ( s , s ′ ) and the expert-output diameter at the reached states is uniformly at most D max D_{\max} D m a x , one may take
L W ≤ 1 2 D max L g . (152) L_W \le\frac{1}{2}D_{\max}L_g. \tag*{(152)} L W ≤ 2 1 D m a x L g . (152)
Thus soft routing adds a sensitivity proportional to both gate movement and functional expert separation. This result applies to the additive common-input form; a dispatch mechanism whose gate changes expert inputs requires an additional dispatch-sensitivity term.
Hard top-k support is locally stable inside a margin tube
Assume the reference and learned router coordinates have been placed in one fixed canonical role alignment (or already share the same expert coordinates). Let z : S → R K z:\mathcal{S}\to\mathbb{R}^K z : S → R K be reference router logits, fix 1 ≤ k < K 1\le k<K 1 ≤ k < K , and let I k ( z ) \mathcal{I}_k(z) I k ( z ) denote the indices of the k k k largest coordinates under a fixed tie rule. Write z ( 1 ) ≥ ⋯ ≥ z ( K ) z_{(1)}\ge\cdots\ge z_{(K)} z ( 1 ) ≥ ⋯ ≥ z ( K ) and define the top-k k k support margin
γ k ( s ) = z ( k ) ( s ) − z ( k + 1 ) ( s ) , (153) \gamma_k(s)=z_{(k)}(s)-z_{(k+1)}(s), \tag*{(153)} γ k ( s ) = z ( k ) ( s ) − z ( k + 1 ) ( s ) , (153)
with k = 1 k=1 k = 1 giving the ordinary winner margin.
Lemma 27 (Top-k k k support-margin stability). If another logit vector z ~ \widetilde{z} z satisfies
∥ z ~ − z ( s ) ∥ ∞ < 1 2 γ k ( s ) , (154) \lVert\widetilde{z}-z(s)\rVert_\infty< \frac{1}{2}\gamma_k(s), \tag*{(154)} ∥ z − z ( s ) ∥ ∞ < 2 1 γ k ( s ) , (154)
then I k ( z ~ ) = I k ( z ( s ) ) \mathcal{I}_k(\widetilde{z})=\mathcal{I}_k(z(s)) I k ( z ) = I k ( z ( s )) . More generally, if a learned router z ^ \widehat{z} z satisfies
∥ z ^ ( s ′ ) − z ( s ′ ) ∥ ∞ ≤ β , ∥ z ( s ′ ) − z ( s ) ∥ ∞ ≤ L R d ( s , s ′ ) , (155) \lVert\widehat{z}(s')-z(s')\rVert_\infty\le\beta,\qquad\lVert z(s')-z(s)\rVert_\infty\le L_Rd(s,s'), \tag*{(155)} ∥ z ( s ′ ) − z ( s ′ ) ∥ ∞ ≤ β , ∥ z ( s ′ ) − z ( s ) ∥ ∞ ≤ L R d ( s , s ′ ) , (155)
then the supports agree whenever
γ k ( s ) > 2 ( β + L R d ( s , s ′ ) ) . (156) \gamma_k(s)>2\bigl(\beta+L_Rd(s,s')\bigr). \tag*{(156)} γ k ( s ) > 2 ( β + L R d ( s , s ′ ) ) . (156)
For unit-gain top-1 or uniform hard top-k k k , support agreement determines the entire gate. For top-k k k followed by nonuniform normalized weights, the lemma controls only the support; any remaining within-support weight discrepancy can be bounded by the transport result in Theorem 21. Top-1 with a selected softmax gain also retains this weight variation; Proposition 51 differentiates that gain explicitly.
The margin tube yields a pathwise no-support-cascade condition. Let the reference support at event j j j be I k ( z j ( S j ) ) \mathcal{I}_k(z_j(S_j)) I k ( z j ( S j )) and the learned support be I k ( z ^ j ( S ^ j ) ) \mathcal{I}_k(\widehat{z}_j(\widehat{S}_j)) I k ( z j ( S j )) . Assume the reference logits are L R , j L_{R,j} L R , j -Lipschitz, and that
∥ z ^ j ( s ) − z j ( s ) ∥ ∞ ≤ β j (157) \lVert\widehat{z}_j(s)-z_j(s)\rVert_\infty\le\beta_j \tag*{(157)} ∥ z j ( s ) − z j ( s ) ∥ ∞ ≤ β j (157)
holds on the reached tube. Assume every reference transition with its complete route (support and any route weights) frozen is L j L_j L j -Lipschitz, uniformly over the reference routes reached. Finally, suppose that whenever the learned and reference supports agree, the same-state event discrepancy—including model approximation and, for nonuniform top-k k k , any within-support weight difference—is at most η j \eta_j η j . Define
e ˉ 0 = d ( S ^ 0 , S 0 ) , e ˉ j + 1 = L j e ˉ j + η j . (158) \bar{e}_0=d(\widehat{S}_0,S_0),\qquad\bar{e}_{j+1}=L_j\bar{e}_j+\eta_j. \tag*{(158)} e ˉ 0 = d ( S 0 , S 0 ) , e ˉ j + 1 = L j e ˉ j + η j . (158)
Theorem 28 (Margin-certified absence of support cascades). If, along the reference trace,
γ k , j ( S j ) > 2 ( β j + L R , j e ˉ j ) for every j , (159) \gamma_{k,j}(S_j) > 2(\beta_j + L_{R,j}\bar e_j) \qquad\text{for every } j, \tag*{(159)} γ k , j ( S j ) > 2 ( β j + L R , j e ˉ j ) for every j , (159) then the learned and reference top-k k k supports agree at every event and
d ( S ^ j , S j ) ≤ e ˉ j for all j . (160) d(\widehat S_j,S_j) \le\bar e_j \qquad\text{for all } j. \tag*{(160)} d ( S j , S j ) ≤ e ˉ j for all j . (160) For unit-gain top-1 and uniform hard top-k k k , the corresponding discrete route sequence is unchanged, so no support-change routing penalty is incurred. For nonuniform gates, including selected-softmax-gain top-1, same-support weight error remains inside η j \eta_j η j .
The condition is sufficient rather than necessary. It identifies a support-stability basin: local approximation or within-support weight error may be nonzero, but it cannot change future selected supports while the accumulated state deviation remains inside every margin tube.
A margin-distribution bound for expected support changes
Uniform positive margins are often unrealistic. An expected guarantee follows from a conditional anti-concentration condition. Let
E j = d ( S ^ j , S j ) , B j = ∥ z ^ j ( S ^ j ) − z j ( S ^ j ) ∥ ∞ , U j = B j + L R , j E j . (161) E_j = d(\widehat S_j,S_j), \qquad B_j = \left\lVert\widehat z_j(\widehat S_j)-z_j(\widehat S_j)\right\rVert_\infty, \qquad U_j = B_j + L_{R,j}E_j. \tag*{(161)} E j = d ( S j , S j ) , B j = z j ( S j ) − z j ( S j ) ∞ , U j = B j + L R , j E j . (161)
By Lemma 27, a support mismatch implies γ k , j ( S j ) ≤ 2 U j \gamma_{k,j}(S_j) \le2U_j γ k , j ( S j ) ≤ 2 U j . Assume that there exists a version of the conditional margin CDF such that, almost surely, for every t ≥ 0 t \ge0 t ≥ 0 ,
P ( γ k , j ( S j ) ≤ t ∣ U j ) ≤ min { 1 , c j t α j } (162) \mathbb{P}(\gamma_{k,j}(S_j) \le t \mid U_j) \le\min\{1,c_jt^{\alpha_j}\} \tag*{(162)} P ( γ k , j ( S j ) ≤ t ∣ U j ) ≤ min { 1 , c j t α j } (162)
for constants c j > 0 c_j > 0 c j > 0 and α j > 0 \alpha_j > 0 α j > 0 . This conditional statement is essential: an unconditional margin histogram does not generally control a random, dependent threshold 2 U j 2U_j 2 U j .
Let M j M_j M j be the indicator of a support mismatch. Let r j ≥ 0 r_j \ge0 r j ≥ 0 denote the full route-induced transition discrepancy at event j j j , and assume the pointwise decomposition
r j ≤ ζ j + Δ j M j , ζ j ≥ 0. (163) r_j \le\zeta_j + \Delta_jM_j, \qquad\zeta_j \ge0. \tag*{(163)} r j ≤ ζ j + Δ j M j , ζ j ≥ 0. (163)
Here ζ j \zeta_j ζ j bounds route discrepancy that can remain even when supports agree, such as nonuniform within-support weight changes; for unit-gain top-1 or uniform top-k k k , one may take ζ j = 0 \zeta_j = 0 ζ j = 0 . The deterministic constant Δ j ≥ 0 \Delta_j \ge0 Δ j ≥ 0 bounds the additional severity associated with a support change.
Theorem 29 (Margin-mediated expected support-change cost). Under the preceding conditions, with ρ j = E r j \rho_j = \mathbb{E}r_j ρ j = E r j ,
ρ j ≤ E ζ j + Δ j E min { 1 , c j ( 2 U j ) α j } . (164) \rho_j \le\mathbb{E}\zeta_j + \Delta_j\mathbb{E}\min\{1,c_j(2U_j)^{\alpha_j}\}. \tag*{(164)} ρ j ≤ E ζ j + Δ j E min { 1 , c j ( 2 U j ) α j } . (164) For α j = 1 \alpha_j = 1 α j = 1 and finite first moments, the simpler (possibly looser) bound holds without restricting the range of U j U_j U j :
ρ j ≤ E ζ j + 2 c j Δ j ( E B j + L R , j E E j ) . (165) \rho_j \le\mathbb{E}\zeta_j + 2c_j\Delta_j\left(\mathbb{E}B_j + L_{R,j}\mathbb{E}E_j\right). \tag*{(165)} ρ j ≤ E ζ j + 2 c j Δ j ( E B j + L R , j E E j ) . (165) Consequently, if a pathwise local recursion has the form
E j + 1 ≤ a j + L j E j + r j , (166) E_{j+1} \le a_j + L_jE_j + r_j, \tag*{(166)} E j + 1 ≤ a j + L j E j + r j , (166) then
E E j + 1 ≤ E a j + E ζ j + 2 c j Δ j E B j + ( L j + 2 c j Δ j L R , j ) E E j . (167) \mathbb{E}E_{j+1} \le\mathbb{E}a_j + \mathbb{E}\zeta_j + 2c_j\Delta_j\mathbb{E}B_j + (L_j + 2c_j\Delta_jL_{R,j})\mathbb{E}E_j. \tag*{(167)} E E j + 1 ≤ E a j + E ζ j + 2 c j Δ j E B j + ( L j + 2 c j Δ j L R , j ) E E j . (167)
The extra factor 2 c j Δ j L R , j 2c_j\Delta_jL_{R,j} 2 c j Δ j L R , j is an endogenous support-routing amplification term. It is large when margins concentrate near zero, logits are state-sensitive, or support substitutions are functionally severe. For α j ≠ 1 \alpha_j \ne1 α j = 1 , Equation (164) yields a nonlinear moment recursion involving E U j α j \mathbb{E}U_j^{\alpha_j} E U j α j (or the clipped moment appearing explicitly in the theorem).
A weaker marginal-margin guarantee under arbitrary dependence
Conditioning on U j U_j U j in Equation (162) can be restrictive because U j U_j U j itself depends on the trajectory and router margin. A marginal margin assumption still gives a guarantee, but generally a slower one. Suppress j j j and let γ ≥ 0 \gamma\ge0 γ ≥ 0 be a support margin, U ≥ 0 U \ge0 U ≥ 0 the logit perturbation bound, and M M M the support-mismatch indicator. Suppose M ≤ 1 { γ ≤ 2 U } M \le\mathbf{1}\{\gamma\le2U\} M ≤ 1 { γ ≤ 2 U } , P ( γ ≤ t ) ≤ c t α \mathbb{P}(\gamma\le t) \le ct^\alpha P ( γ ≤ t ) ≤ c t α for all t ≥ 0 t \ge0 t ≥ 0 , and m r = E U r < ∞ m_r = \mathbb{E}U^r < \infty m r = E U r < ∞ for some moment order r > 0 r > 0 r > 0 . The constants c > 0 c > 0 c > 0 , α > 0 \alpha> 0 α > 0 control how much margin probability lies near zero; this is called a margin anti-concentration condition.
Theorem 30 (Dependence-robust marginal-margin bound). No independence between γ \gamma γ and U U U is required for
P ( M = 1 ) ≤ min { 1 , inf t > 0 [ c ( 2 t ) α + m r t r ] } . (168) \mathbb{P}(M=1) \le\min\left\{1,\inf_{t>0}\left[c(2t)^{\alpha}+\frac{m_r}{t^r}\right]\right\}. \tag*{(168)} P ( M = 1 ) ≤ min { 1 , t > 0 inf [ c ( 2 t ) α + t r m r ] } . (168) A sharper explicit bound, not merely the optimization of the preceding split, is
P ( M = 1 ) ≤ min { 1 , K α , r ( 2 α c ) r / ( α + r ) m r α / ( α + r ) } , (169) \mathbb{P}(M=1) \le\min\left\{1,K_{\alpha,r}(2^{\alpha}c)^{r/(\alpha+r)}m_r^{\alpha/(\alpha+r)}\right\}, \tag*{(169)} P ( M = 1 ) ≤ min { 1 , K α , r ( 2 α c ) r / ( α + r ) m r α / ( α + r ) } , (169) where K α , r = ( ( α + r ) / α ) α / ( α + r ) K_{\alpha,r}=\left((\alpha+r)/\alpha\right)^{\alpha/(\alpha+r)} K α , r = ( ( α + r ) / α ) α / ( α + r ) . For m r = 0 m_r=0 m r = 0 , the mismatch probability is zero. If the route loss satisfies r r o u t e ≤ ζ + Δ M r_{\mathrm{route}}\le\zeta+\Delta M r route ≤ ζ + Δ M , multiply the mismatch bound by Δ \Delta Δ and add E ζ \mathbb{E}\zeta E ζ to obtain a functional-cost bound.
Proof. For every fixed threshold t > 0 t>0 t > 0 , a mismatch implies either γ ≤ 2 t \gamma\le2t γ ≤ 2 t or U > t U>t U > t . A union bound and Markov’s inequality give Equation (168) . For the sharper bound, let E = { M = 1 } E=\{M=1\} E = { M = 1 } and p = P ( E ) p=\mathbb{P}(E) p = P ( E ) . On E E E , U ≥ γ / 2 U\ge\gamma/2 U ≥ γ /2 . The tail-integral identity for a nonnegative random variable and P ( E , γ > t ) ≥ p − c t α \mathbb{P}(E,\gamma>t)\ge p-ct^\alpha P ( E , γ > t ) ≥ p − c t α imply, for p > 0 p>0 p > 0 ,
m r ≥ 2 − r E [ γ r 1 E ] (170) m_r\ge2^{-r}\mathbb{E}[\gamma^r1_E] \tag*{(170)} m r ≥ 2 − r E [ γ r 1 E ] (170)
≥ 2 − r ∫ 0 ( p / c ) 1 / α r t r − 1 ( p − c t α ) d t (171) \ge2^{-r}\int_0^{(p/c)^{1/\alpha}}rt^{r-1}(p-ct^\alpha)\,dt \tag*{(171)} ≥ 2 − r ∫ 0 ( p / c ) 1/ α r t r − 1 ( p − c t α ) d t (171)
= 2 − r α α + r c − r / α p 1 + r / α . (172) =2^{-r}\frac{\alpha}{\alpha+r}c^{-r/\alpha}p^{1+r/\alpha}. \tag*{(172)} = 2 − r α + r α c − r / α p 1 + r / α . (172)
Rearranging yields Equation (169) ; the case p = 0 p=0 p = 0 is immediate. If m r = 0 m_r=0 m r = 0 , then U = 0 U=0 U = 0 almost surely and P ( γ = 0 ) = 0 \mathbb{P}(\gamma=0)=0 P ( γ = 0 ) = 0 by the margin assumption. No conditioning or independence is used. For example, α = r = c = 1 \alpha=r=c=1 α = r = c = 1 yields 2 m 1 2\sqrt{m_1} 2 m 1 rather than the looser 2 2 m 1 2\sqrt{2m_1} 2 2 m 1 from optimizing the split. □ \square □
Proposition 31 (Sharp exponent and limiting constant under marginal control). For every α , r , c > 0 \alpha,r,c>0 α , r , c > 0 , every ε > 0 \varepsilon>0 ε > 0 , and every 0 < b ≤ c − 1 / α 0<b\le c^{-1/\alpha} 0 < b ≤ c − 1/ α , there is a two-expert top-1 example with margin CDF P ( γ ≤ t ) = min { 1 , c t α } \mathbb{P}(\gamma\le t)=\min\{1,ct^\alpha\} P ( γ ≤ t ) = min { 1 , c t α } and strict support changes such that
P ( M = 1 ) = [ α + r α ( 2 1 + ε ) r c r / α E U r ] α / ( α + r ) . (173) \mathbb{P}(M=1)=\left[\frac{\alpha+r}{\alpha}\left(\frac{2}{1+\varepsilon}\right)^r c^{r/\alpha}\mathbb{E}U^r\right]^{\alpha/(\alpha+r)}. \tag*{(173)} P ( M = 1 ) = [ α α + r ( 1 + ε 2 ) r c r / α E U r ] α / ( α + r ) . (173) Letting b ↓ 0 b\downarrow0 b ↓ 0 shows that the moment exponent cannot improve. Letting ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 shows that the multiplicative constant in the nontrivial part of Equation (169) cannot be uniformly reduced under the stated assumptions.
Proof. Use logits z = ( γ / 2 , − γ / 2 ) z=(\gamma/2,-\gamma/2) z = ( γ /2 , − γ /2 ) . On { γ ≤ b } \{\gamma\le b\} { γ ≤ b } replace them by z ~ = ( − ε γ / 2 , ε γ / 2 ) \widetilde{z}=(-\varepsilon\gamma/2,\varepsilon\gamma/2) z = ( − ε γ /2 , ε γ /2 ) ; otherwise leave them unchanged. The reference and perturbed winners differ strictly except on the zero-probability event γ = 0 \gamma=0 γ = 0 . Thus
U = 1 + ε 2 γ 1 { γ ≤ b } , P ( M = 1 ) = c b α , E U r = ( 1 + ε 2 ) r c α α + r b α + r . U=\frac{1+\varepsilon}{2}\gamma1_{\{\gamma\le b\}},\qquad\mathbb{P}(M=1)=cb^\alpha,\qquad\mathbb{E}U^r=\left(\frac{1+\varepsilon}{2}\right)^r\frac{c\alpha}{\alpha+r}b^{\alpha+r}. U = 2 1 + ε γ 1 { γ ≤ b } , P ( M = 1 ) = c b α , E U r = ( 2 1 + ε ) r α + r c α b α + r .
Eliminating b b b proves the formula. Choose c b α < 1 cb^\alpha<1 c b α < 1 to avoid saturation and then let ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 for limiting sharpness. Positive ε \varepsilon ε avoids relying on a favorable tie rule. □ \square □
The perturbation in this example uses only currently available logits, not future information. Small-margin states are precisely the ones receiving error. This explains why an unconditional margin histogram cannot justify the faster conditional bound. For a first-moment state certificate and r = 1 r=1 r = 1 , the robust dependence on the error moment has exponent α / ( α + 1 ) \alpha/(\alpha+1) α / ( α + 1 ) , not α \alpha α .
Implications for expert reuse
Cross-layer and global pools can increase the number of candidate experts, which can alter router margins. They may also place functionally redundant experts near one another, reducing Δ j \Delta_j Δ j or the transport cost of a support change. Therefore candidate-pool size alone does not determine route stability. The relevant measurements are jointly:
the top-k k k support-margin distribution under the reached occupancy.
the state-to-logit sensitivity L R , j L_{R,j} L R , j or an empirical local analogue.
same-support weight discrepancy ζ j \zeta_j ζ j when weights are not uniform.
the functional geometry of candidate expert outputs.
the position-dependent downstream amplification.
These quantities turn the otherwise exogenous ρ j \rho_j ρ j into a measurable closed-loop diagnostic. They also explain why route consistency can deteriorate with reasoning length even when the one-step expert estimators remain unchanged.
Proofs
Proof of Proposition 26. Insert M ( s ′ , g ( s ) ) M(s', g(s)) M ( s ′ , g ( s )) :
d ( F ( s ) , F ( s ′ ) ) ≤ d ( M ( s , g ( s ) ) , M ( s ′ , g ( s ) ) ) + d ( M ( s ′ , g ( s ) ) , M ( s ′ , g ( s ′ ) ) ) . (174) d(F(s), F(s')) \leq d(M(s, g(s)), M(s', g(s))) + d(M(s', g(s)), M(s', g(s'))). \tag*{(174)} d ( F ( s ) , F ( s ′ )) ≤ d ( M ( s , g ( s )) , M ( s ′ , g ( s ))) + d ( M ( s ′ , g ( s )) , M ( s ′ , g ( s ′ ))) . (174)
The first term is at most L x d ( s , s ′ ) L_xd(s, s') L x d ( s , s ′ ) by Equation (149) ; the second is at most W s ′ ( g ( s ) , g ( s ′ ) ) W_{s'}(g(s), g(s')) W s ′ ( g ( s ) , g ( s ′ )) by Theorem 21. This proves Equation (151) . The diameter specialization follows from the second inequality in Equation (131) .
Proof of Lemma 27. Every selected logit can decrease by at most ∥ z ~ − z ( s ) ∥ ∞ \lVert\tilde{z}-z(s)\rVert_{\infty} ∥ z ~ − z ( s ) ∥ ∞ , and every unselected logit can increase by at most the same amount. If this perturbation is below γ k ( s ) / 2 \gamma_k(s)/2 γ k ( s ) /2 , the smallest selected coordinate remains strictly larger than the largest unselected coordinate. For the second statement, the triangle inequality bounds ∥ z ~ ( s ′ ) − z ( s ) ∥ ∞ \lVert\tilde{z}(s')-z(s)\rVert_{\infty} ∥ z ~ ( s ′ ) − z ( s ) ∥ ∞ by β + L R d ( s , s ′ ) \beta+L_Rd(s,s') β + L R d ( s , s ′ ) .
Proof of Theorem 28. Proceed by induction. At j = 0 j=0 j = 0 , the error is at most e ˉ 0 \bar{e}_0 e ˉ 0 . If d ( S ^ j , S j ) ≤ e ˉ j d(\hat{S}_j,S_j)\leq\bar{e}_j d ( S ^ j , S j ) ≤ e ˉ j , Equation (159) and Lemma 27 imply identical supports. By the assumed same-support event-discrepancy bound and the Lipschitz property of the reference fixed-support transition,
d ( S ^ j + 1 , S j + 1 ) ≤ η j + L j d ( S ^ j , S j ) ≤ e ˉ j + 1 . (175) d(\hat{S}_{j+1},S_{j+1})\leq\eta_j+L_jd(\hat{S}_j,S_j)\leq\bar{e}_{j+1}. \tag*{(175)} d ( S ^ j + 1 , S j + 1 ) ≤ η j + L j d ( S ^ j , S j ) ≤ e ˉ j + 1 . (175)
Thus support agreement and the error envelope hold at every event.
Proof of Theorem 29. A support mismatch implies γ k , j ( S j ) ≤ 2 U j \gamma_{k,j}(S_j)\leq2U_j γ k , j ( S j ) ≤ 2 U j . Conditioning on U j U_j U j and applying Equation (162) at the random threshold 2 U j 2U_j 2 U j gives
P ( M j = 1 ) ≤ E min { 1 , c j ( 2 U j ) α j } . (176) \mathbb{P}(M_j=1)\leq\mathbb{E}\min\{1,c_j(2U_j)^{\alpha_j}\}. \tag*{(176)} P ( M j = 1 ) ≤ E min { 1 , c j ( 2 U j ) α j } . (176)
Taking expectations in Equation (163) proves Equation (164) . For α j = 1 \alpha_j=1 α j = 1 , use min { 1 , 2 c j U j } ≤ 2 c j U j \min\{1,2c_jU_j\}\leq2c_jU_j min { 1 , 2 c j U j } ≤ 2 c j U j , substitute U j = B j + L R , j E j U_j=B_j+L_{R,j}E_j U j = B j + L R , j E j and use linearity of expectation. Taking expectations in Equation (166) and inserting the resulting route-cost bound proves Equation (167) .
Counterfactual Compatibility and the Sample Complexity of Route Audits
Overview. We separate a population comparison of action effects from the finite-sample task of auditing them. Under a declared randomized intervention law, counterfactual variance measures incompatibility of action means; absent overlap, unsupported effects are not identified. Independent records and positive forcing probabilities support action-specific concentration. Sections K.1–K.2 establish the population comparison and the overlap obstruction; Sections K.3–K.5 give audit bounds and budget allocation, with remaining proofs in Section K.6.
Reuse bias as counterfactual transition variance
Let S S S be a pre-action macro state in a standard Borel space and let A ∈ [ K ] A\in[K] A ∈ [ K ] be drawn from a declared randomized intervention distribution p a ( S ) = P ( A = a ∣ S ) p_a(S)=\mathbb{P}(A=a\mid S) p a ( S ) = P ( A = a ∣ S ) . Let Y a Y^a Y a be the potential post-action transition target for action a a a in a separable real Hilbert space H \mathcal{H} H . Assume every Y a Y^a Y a is Bochner measurable and square-integrable, conditional randomization A ⊥ ( Y 1 , … , Y K ) ∣ S A \perp(Y^{1},\ldots,Y^{K}) \mid S A ⊥ ( Y 1 , … , Y K ) ∣ S , and consistency Y = Y A Y=Y^{A} Y = Y A . Define the Bochner conditional mean
m a ( s ) = E [ Y a ∣ S = s ] . (177) m_a(s)=\mathbb{E}[Y^{a}\mid S=s]. \tag*{(177)} m a ( s ) = E [ Y a ∣ S = s ] . (177)
Values of m a m_a m a outside the support of p a ( S ) p_a(S) p a ( S ) are immaterial to the displayed risks. The evaluation experiment draws A A A and observes Y A Y^{A} Y A . Compare action-specific predictors f A ( S ) f_A(S) f A ( S ) with one shared predictor f ( S ) f(S) f ( S ) under squared loss.
Theorem 32 (Counterfactual decomposition of reuse bias). Under the conditions above, allow all measurable predictors with finite weighted second moment. The action-specific and shared Bayes predictors are
f a ⋆ ( s ) = m a ( s ) , f s h a r e ⋆ ( s ) = m ˉ ( s ) : = ∑ a p a ( s ) m a ( s ) (178) f_a^{\star}(s)=m_a(s), \qquad f_{\mathrm{share}}^{\star}(s)=\bar{m}(s):=\sum_a p_a(s)m_a(s) \tag*{(178)} f a ⋆ ( s ) = m a ( s ) , f share ⋆ ( s ) = m ˉ ( s ) := a ∑ p a ( s ) m a ( s ) (178) on the corresponding supports. Their oracle-risk difference is
B c f r e u s e = E ∥ m A ( S ) − m ˉ ( S ) ∥ H 2 (179) B_{\mathrm{cf}}^{\mathrm{reuse}}=\mathbb{E}\left\|m_A(S)-\bar{m}(S)\right\|_{\mathcal{H}}^2 \tag*{(179)} B cf reuse = E ∥ m A ( S ) − m ˉ ( S ) ∥ H 2 (179) = 1 2 E S ∑ a , b p a ( S ) p b ( S ) ∥ m a ( S ) − m b ( S ) ∥ H 2 . (180) =\frac{1}{2}\mathbb{E}_{S}\sum_{a,b}p_a(S)p_b(S)\left\|m_a(S)-m_b(S)\right\|_{\mathcal{H}}^2. \tag*{(180)} = 2 1 E S a , b ∑ p a ( S ) p b ( S ) ∥ m a ( S ) − m b ( S ) ∥ H 2 . (180)
This is the action-level analogue of Proposition 7 . It separates irreducible transition uncertainty, which appears in both oracle risks, from incompatibility among conditional action means, which is paid only by the shared operator. The intervention distribution is part of the estimand: changing which actions are compared, or where they are forced, changes B c f r e u s e B_{\mathrm{cf}}^{\mathrm{reuse}} B cf reuse .
The identity clarifies the relation between functional and causal reuse. Small observational disagreement among naturally selected experts does not imply small Equation (180) ; the latter compares actions at common pre-action states. Conversely, distinct indices can have negligible causal sharing bias if their forced conditional transition means coincide on the stated intervention occupancy. The theorem is a population identity. A plug-in estimate of B c f r e u s e B_{\mathrm{cf}}^{\mathrm{reuse}} B cf reuse must estimate the conditional means m a m_a m a and evaluate their squared differences on independent or cross-fitted records, or else control the complexity and overfitting of the learned functions.
When the local and causal comparisons are the same. For a supported action set, define π a = P ( A = a ) > 0 \pi_a=\mathbb{P}(A=a)>0 π a = P ( A = a ) > 0 and the local population risk
R a ( f ) = E [ ∥ Y a − f ( S ) ∥ H 2 ∣ A = a ] . R_a(f)=\mathbb{E}\left[\left\|Y^a-f(S)\right\|_{\mathcal{H}}^2\mid A=a\right]. R a ( f ) = E [ ∥ Y a − f ( S ) ∥ H 2 ∣ A = a ] .
Use the same predictors of S S S , the same squared loss, and these weights in the local shared/separate oracle definitions. Consistency and the law of total expectation give ∑ a π a R a ( f ) = E ∥ Y A − f ( S ) ∥ H 2 \sum_a\pi_aR_a(f)=\mathbb{E}\|Y^A-f(S)\|_{\mathcal{H}}^2 ∑ a π a R a ( f ) = E ∥ Y A − f ( S ) ∥ H 2 ; allowing a separate f a f_a f a gives the corresponding action-specific objective. Their population bias is then exactly the bias in Theorem 32 . This alignment uses the action-conditional state laws, which need not equal the marginal law of S S S .
Without this matched experiment and class, the two biases are different quantities. In particular, tying a physical expert does not remove action information encoded in its input. A statistical role such as operation–depth pair ( i , ℓ ) (i,\ell) ( i , ℓ ) identifies a call environment; a route action specifies an intervention on executable calls. Equating those indices requires an explicit map, not merely identical names.
Without overlap, unsupported route effects are not identifiable
Let an observational or intervention policy select action a a a with propensity e a ( C ) = P ( A = a ∣ C ) e_a(C)=\mathbb{P}(A=a\mid C) e a ( C ) = P ( A = a ∣ C ) . Suppose there is a measurable set B \mathcal{B} B with P ( C ∈ B ) > 0 \mathbb{P}(C\in\mathcal{B})>0 P ( C ∈ B ) > 0 on which e a ( C ) = 0 e_a(C)=0 e a ( C ) = 0 .
Proposition 33 (No-overlap non-identifiability). Without structural restrictions linking unsupported and supported action effects, the distribution of T a ( C ) T_a(C) T a ( C ) on B \mathcal{B} B is not identified from data generated by the policy whenever the model class permits two different effects there. A defect or utility depending nontrivially on those differences is likewise not identified; a constant summary need not be affected. In a nondegenerate binary-output model, for example, there exist two action libraries with identical observed-data distributions but different values of
E [ d ( T a ( C ) , F a ( Ψ ( C ) ) ) 1 { C ∈ B } ] . (181) \mathbb{E}\left[d\left(T_a(C),F_a\left(\Psi(C)\right)\right)\mathbf{1}\{C\in\mathcal{B}\}\right]. \tag*{(181)} E [ d ( T a ( C ) , F a ( Ψ ( C ) ) ) 1 { C ∈ B } ] . (181)
The conclusion is not repaired by more samples from the same policy. It requires either a structural extrapolation assumption or interventions that assign positive probability to the unsupported action–context region.
A finite-sample audit of fixed action-specific defects
The macro extractor Ψ \Psi Ψ , action definitions, candidate operators F a F_a F a , and bounded losses in this subsection are fixed independently of the audit records. They may be learned on a separate fitting split and then conditioned upon. Cross-fitting requires a separate conditional bound for each held-out fold and a union bound over folds; if the fitted operators differ across folds, the resulting target is their average fold-specific risk, not automatically the risk of a final refitted model. Treating all cross-fitted scores as IID conditional on every fold fit is not justified. Reusing the same audit observations to select these objects requires a uniform complexity correction.
Suppose the context–potential-outcome records are IID across i = 1 , … , N i=1,\ldots,N i = 1 , … , N . Conditional on context C i C_i C i , an action can have stochastic output T a ( C i ; U i ) T_a(C_i;U_i) T a ( C i ; U i ) , where U i U_i U i is unrevealed execution randomness. Define the bounded potential losses and their conditional means by
L i , a = d ( T a ( C i ; U i ) , F a ( Ψ ( C i ) ) ) ∈ [ 0 , 1 ] , h a ( c ) = E [ L i , a ∣ C i = c ] . (182) L_{i,a}=d(T_a(C_i;U_i),F_a(\Psi(C_i)))\in[0,1],\qquad h_a(c)=\mathbb{E}[L_{i,a}\mid C_i=c]. \tag*{(182)} L i , a = d ( T a ( C i ; U i ) , F a ( Ψ ( C i ))) ∈ [ 0 , 1 ] , h a ( c ) = E [ L i , a ∣ C i = c ] . (182)
Conditional on C i C_i C i , randomize A i A_i A i independently of its potential-loss vector with known propensity e a ( C i ) e_a(C_i) e a ( C i ) , independently across records, and observe L i = L i , A i L_i=L_{i,A_i} L i = L i , A i . Deterministic execution is the special case without U i U_i U i . Let
μ a = E L i , a = E h a ( C i ) (183) \mu_a=\mathbb{E}L_{i,a}=\mathbb{E}h_a(C_i) \tag*{(183)} μ a = E L i , a = E h a ( C i ) (183)
under the probe-context distribution. The Horvitz–Thompson estimator is [12 ]
μ ^ a = 1 N ∑ i = 1 N 1 { A i = a } L i e a ( C i ) . (184) \widehat{\mu}_a=\frac{1}{N}\sum_{i=1}^{N}\frac{\mathbf{1}\{A_i=a\}L_i}{e_a(C_i)}. \tag*{(184)} μ a = N 1 i = 1 ∑ N e a ( C i ) 1 { A i = a } L i . (184)
Theorem 34 (Simultaneous concentration of a randomized route audit). Assume the intervention policy has overlap e a ( C ) ≥ β > 0 e_a(C)\ge\beta>0 e a ( C ) ≥ β > 0 a.s. for every claimed action a ∈ [ K ] a\in[K] a ∈ [ K ] , 0 < β ≤ 1 / K 0<\beta\le1/K 0 < β ≤ 1/ K .
Then each μ ^ a \widehat{\mu}_a μ a is unbiased, and with probability at least 1 − δ 1-\delta 1 − δ ,
max a ∈ [ K ] ∣ μ ^ a − μ a ∣ ≤ 2 log ( 2 K / δ ) N β + log ( 2 K / δ ) 3 N β . (185) \max_{a\in[K]}|\widehat{\mu}_a-\mu_a|\le\sqrt{\frac{2\log(2K/\delta)}{N\beta}}+\frac{\log(2K/\delta)}{3N\beta}. \tag*{(185)} a ∈ [ K ] max ∣ μ a − μ a ∣ ≤ N β 2 log ( 2 K / δ ) + 3 N β log ( 2 K / δ ) . (185) Consequently, N β N\beta N β is the leading effective sample size for a simultaneous claim about the least-covered action. Under uniform random forcing, β = 1 / K \beta=1/K β = 1/ K , so estimating all K K K action defects to fixed accuracy requires order K K K audit records, up to the usual accuracy and logarithmic factors.
The theorem applies to a loss evaluated after one expert call, a compound multi-layer route action, or a bounded stopping-time macro action, provided the action and loss are defined before the audit records are observed. If an action changes active compute, the comparator should include matched-compute null or substitute actions so that the audit isolates functionality rather than budget.
For fixed nonnegative terminal weights w a w_a w a , the same simultaneous event implies the conservative bound
∣ ∑ a w a μ ^ a − ∑ a w a μ a ∣ ≤ ( ∑ a w a ) [ 2 log ( 2 K / δ ) N β + log ( 2 K / δ ) 3 N β ] . (186) \left|\sum_a w_a\widehat{\mu}_a-\sum_a w_a\mu_a\right|
\le\left(\sum_a w_a\right)\left[\sqrt{\frac{2\log(2K/\delta)}{N\beta}}+\frac{\log(2K/\delta)}{3N\beta}\right]. \tag*{(186)} a ∑ w a μ a − a ∑ w a μ a ≤ ( a ∑ w a ) [ N β 2 log ( 2 K / δ ) + 3 N β log ( 2 K / δ ) ] . (186)
This is useful when every action-specific defect must also be reported, but it is not the optimal rate for every aggregate. In particular, for the behavior-policy-weighted target
μ e = E ∑ a e a ( C ) h a ( C ) = E L i , (187) \mu_e=\mathbb{E}\sum_a e_a(C)h_a(C)=\mathbb{E}L_i, \tag*{(187)} μ e = E a ∑ e a ( C ) h a ( C ) = E L i , (187)
the direct estimator μ ^ e = N − 1 ∑ i L i \widehat{\mu}_e=N^{-1}\sum_i L_i μ e = N − 1 ∑ i L i satisfies, by Hoeffding,
∣ μ ^ e − μ e ∣ ≤ log ( 2 / δ ) 2 N . (188) |\widehat{\mu}_e-\mu_e|\le\sqrt{\frac{\log(2/\delta)}{2N}}. \tag*{(188)} ∣ μ e − μ e ∣ ≤ 2 N log ( 2/ δ ) . (188)
with probability at least 1 − δ 1-\delta 1 − δ , without a 1 / β 1/\beta 1/ β factor. Thus N β N\beta N β is the relevant size for uniform action-specific or least-covered-action claims, not a universal sample size for all policy averages. A different target policy q a ( C ) q_a(C) q a ( C ) can be evaluated by weighting with q a ( C ) / e a ( C ) q_a(C)/e_a(C) q a ( C ) / e a ( C ) , whose rate depends on the resulting importance-weight envelope or variance.
The per-action coverage dependence is minimax unavoidable
The 1 / N β 1/\sqrt{N\beta} 1/ N β term is not merely an artifact of inverse-propensity weighting.
Proposition 35 (Per-action coverage lower bound). There is a universal constant c > 0 c>0 c > 0 such that, for any estimator μ ~ a \tilde{\mu}_a μ ~ a based on N N N randomized records with constant propensity e a = β e_a=\beta e a = β , one can choose a bounded binary intervention-outcome distribution for which
E ∣ μ ~ a − μ a ∣ ≥ c min { 1 , 1 N β } . (189) \mathbb{E}\left|\tilde{\mu}_a-\mu_a\right| \ge c\min\left\{1,\frac{1}{\sqrt{N\beta}}\right\}. \tag*{(189)} E ∣ μ ~ a − μ a ∣ ≥ c min { 1 , N β 1 } . (189)
Thus an action that is forced only rarely cannot be audited precisely merely because the total token count is large. The proof is a two-point Le Cam argument [40 ] .
Allocating a finite intervention budget
Uniform forcing is simple but is not generally variance optimal for a performance-weighted aggregate. Suppose the target is μ w = ∑ a w a μ a \mu_w=\sum_a w_a\mu_a μ w = ∑ a w a μ a , constant action propensities e a ≥ 0 e_a\ge0 e a ≥ 0 sum to one, and
s a = E L i , a 2 . (190) s_a=\mathbb{E}L_{i,a}^{2}. \tag*{(190)} s a = E L i , a 2 . (190)
For every action with w a 2 s a > 0 w_a^2s_a>0 w a 2 s a > 0 , require e a > 0 e_a>0 e a > 0 . The unbiased single-record estimator is w A L i / e A w_A L_i/e_A w A L i / e A , with second moment ∑ a w a 2 s a / e a \sum_a w_a^2s_a/e_a ∑ a w a 2 s a / e a over the active set.
Proposition 36 (Variance-optimal route-intervention allocation). Assume w a ≥ 0 w_a\ge0 w a ≥ 0 and not all w a s a w_a\sqrt{s_a} w a s a vanish. Let A + = { a : w a s a > 0 } A_+=\{a:w_a\sqrt{s_a}>0\} A + = { a : w a s a > 0 } . Ignoring positivity requirements for actions outside A + A_+ A + and lower-propensity constraints, the action distribution minimizing the second moment of the weighted estimator is
e a ⋆ = w a s a ∑ b ∈ A + w b s b ( a ∈ A + ) , e a ⋆ = 0 ( a ∉ A + ) , (191) e_a^\star=\frac{w_a\sqrt{s_a}}{\sum_{b\in A_+}w_b\sqrt{s_b}}\quad(a\in A_+),\qquad e_a^\star=0\quad(a\notin A_+), \tag*{(191)} e a ⋆ = ∑ b ∈ A + w b s b w a s a ( a ∈ A + ) , e a ⋆ = 0 ( a ∈ / A + ) , (191) with minimum
( ∑ a ∈ A + w a s a ) 2 . (192) \left(\sum_{a\in A_+}w_a\sqrt{s_a}\right)^2. \tag*{(192)} a ∈ A + ∑ w a s a 2 . (192)
A pilot sample can estimate s a s_a s a , after which an independent second phase can use Equation (191) . If action-specific claims are required for every action, or if future targets may assign them positive weight, impose a floor e a ≥ β e_a\ge\beta e a ≥ β with K β ≤ 1 K\beta\le1 K β ≤ 1 ; if K β < 1 K\beta<1 K β < 1 and some w a s a > 0 w_a\sqrt{s_a}>0 w a s a > 0 , the constrained solution is e a = max { β , w a s a / t } e_a=\max\{\beta,w_a\sqrt{s_a}/t\} e a = max { β , w a s a / t } , where t > 0 t>0 t > 0 is chosen so that ∑ a e a = 1 \sum_a e_a=1 ∑ a e a = 1 . If K β = 1 K\beta=1 K β = 1 , the unique feasible allocation is e a = β e_a=\beta e a = β . These formulas follow from the Lagrange stationarity condition on coordinates above the floor; when all second-moment contributions vanish, every feasible allocation is optimal. Actions with zero contribution to the declared aggregate may receive zero mass only when no claim about their individual effects is made.
Proofs
Proof of Theorem 32. For a supported action a a a , conditional randomization gives E [ Y a ∣ S , A = a ] = m a ( S ) \mathbb{E}[Y^a\mid S,A=a]=m_a(S) E [ Y a ∣ S , A = a ] = m a ( S ) . The Hilbert-space Pythagorean identity for Bochner conditional expectations gives
E [ ∥ Y a − f ( S ) ∥ 2 ∣ S , A = a ] = E [ ∥ Y a − m a ( S ) ∥ 2 ∣ S ] + ∥ m a ( S ) − f ( S ) ∥ 2 . (193) \mathbb{E}\left[\left\|Y^a-f(S)\right\|^2\mid S,A=a\right]
=
\mathbb{E}\left[\left\|Y^a-m_a(S)\right\|^2\mid S\right]
+\left\|m_a(S)-f(S)\right\|^2. \tag*{(193)} E [ ∥ Y a − f ( S ) ∥ 2 ∣ S , A = a ] = E [ ∥ Y a − m a ( S ) ∥ 2 ∣ S ] + ∥ m a ( S ) − f ( S ) ∥ 2 . (193)
Action-specific prediction sets f a = m a f_a=m_a f a = m a and leaves only the first term. The shared problem minimizes ∑ a p a ( S ) ∥ m a ( S ) − f ( S ) ∥ 2 \sum_a p_a(S)\left\|m_a(S)-f(S)\right\|^2 ∑ a p a ( S ) ∥ m a ( S ) − f ( S ) ∥ 2 pointwise, whose minimizer is m ˉ ( S ) \bar{m}(S) m ˉ ( S ) . Subtracting the common irreducible term yields Equation (179) . Applying the weighted Hilbert variance identity conditionally on S S S gives Equation (180) . ▫
Proof of Proposition 33. Construct two action libraries that agree for every action–context pair with positive observation probability and differ only in the value of T a ( C ) T_a(C) T a ( C ) for C ∈ B C \in\mathcal{B} C ∈ B . Because action a a a has zero probability on B \mathcal{B} B , the changed transition is never observed, so the induced observed-data distributions are identical. Choosing the two unsupported transitions to have different distances from F a ( Ψ ( C ) ) F_a(\Psi(C)) F a ( Ψ ( C )) changes Equation (181) . Therefore the estimand is not identified without additional structural restrictions.
Proof of Theorem 34. For fixed a a a , let
X i , a = 1 { A i = a } L i e a ( C i ) . (194) X_{i,a} = \frac{\mathbb{1}\{A_i=a\}L_i}{e_a(C_i)}. \tag*{(194)} X i , a = e a ( C i ) 1 { A i = a } L i . (194)
Conditional randomization gives E [ X i , a ∣ C i ] = h a ( C i ) \mathbb{E}[X_{i,a} \mid C_i] = h_a(C_i) E [ X i , a ∣ C i ] = h a ( C i ) , hence E X i , a = μ a \mathbb{E}X_{i,a} = \mu_a E X i , a = μ a . Moreover 0 ≤ X i , a ≤ 1 / β 0 \le X_{i,a} \le1/\beta 0 ≤ X i , a ≤ 1/ β and
E X i , a 2 ≤ E 1 { A i = a } e a ( C i ) 2 = E 1 e a ( C i ) ≤ 1 β . (195) \mathbb{E}X_{i,a}^{2} \le\mathbb{E}\frac{\mathbb{1}\{A_i=a\}}{e_a(C_i)^2} = \mathbb{E}\frac{1}{e_a(C_i)} \le\frac{1}{\beta}. \tag*{(195)} E X i , a 2 ≤ E e a ( C i ) 2 1 { A i = a } = E e a ( C i ) 1 ≤ β 1 . (195)
Thus Var ( X i , a ) ≤ 1 / β \operatorname{Var}(X_{i,a}) \le1/\beta Var ( X i , a ) ≤ 1/ β , and ∣ X i , a − μ a ∣ ≤ 1 / β \lvert X_{i,a} - \mu_a\rvert\le1/\beta ∣ X i , a − μ a ∣ ≤ 1/ β . To make the Bernstein constant explicit, put Z a = μ ^ a − μ a Z_a = \widehat{\mu}_a - \mu_a Z a = μ a − μ a . Expanding the exponential moment of each centered bounded summand and using k ! ≥ 2 ⋅ 3 k − 2 k! \ge2 \cdot3^{k-2} k ! ≥ 2 ⋅ 3 k − 2 for integers k ≥ 2 k \ge2 k ≥ 2 gives
log E e t Z a ≤ t 2 / ( N β ) 2 ( 1 − ∣ t ∣ / ( 3 N β ) ) , ∣ t ∣ < 3 N β . \log\mathbb{E}e^{tZ_a} \le\frac{t^2/(N\beta)}{2(1-\lvert t\rvert/(3N\beta))}, \qquad\lvert t\rvert< 3N\beta. log E e t Z a ≤ 2 ( 1 − ∣ t ∣ / ( 3 N β )) t 2 / ( N β ) , ∣ t ∣ < 3 N β .
This is an exponential-moment bound for both signs of Z a Z_a Z a . The Chernoff argument (apply Markov’s inequality to e t Z a e^{tZ_a} e t Z a and optimize t t t ) gives P { ∣ Z a ∣ > 2 x / ( N β ) + x / ( 3 N β ) } ≤ 2 e − x \mathbb{P}\{\lvert Z_a\rvert> \sqrt{2x/(N\beta)} + x/(3N\beta)\} \le2e^{-x} P {∣ Z a ∣ > 2 x / ( N β ) + x / ( 3 N β )} ≤ 2 e − x . Take x = log ( 2 K / δ ) x = \log(2K/\delta) x = log ( 2 K / δ ) and union-bound over actions.
Proof of Proposition 35. It suffices to consider a context-free stochastic binary outcome, allowed by the potential-loss model above. Under hypotheses P − P_- P − and P + P_+ P + , action a a a is selected with probability β \beta β , and conditional on selection its outcome is Bernoulli with means 1 / 2 − ε 1/2-\varepsilon 1/2 − ε and 1 / 2 + ε 1/2+\varepsilon 1/2 + ε , respectively; every other observation has the same law. For ε ≤ 1 / 4 \varepsilon\le1/4 ε ≤ 1/4 , the one-record Kullback–Leibler divergence is at most 16 β ε 2 16\beta\varepsilon^2 16 β ε 2 , so the N N N -record divergence is at most 16 N β ε 2 16N\beta\varepsilon^2 16 N β ε 2 . Choosing ε = c 0 min { 1 , ( N β ) − 1 / 2 } \varepsilon= c_0\min\{1,(N\beta)^{-1/2}\} ε = c 0 min { 1 , ( N β ) − 1/2 } with a sufficiently small universal c 0 ≤ 1 / 4 c_0 \le1/4 c 0 ≤ 1/4 keeps total variation bounded away from one by Pinsker’s inequality. Le Cam’s two-point inequality then lower-bounds the maximum expected absolute error by a universal constant times the parameter separation, proving Equation (189) .
Proof of Proposition 36. Restrict to A + A_+ A + . Cauchy–Schwarz gives
( ∑ a ∈ A + w a s a ) 2 = ( ∑ a ∈ A + w a s a e a e a ) 2 ≤ ( ∑ a ∈ A + w a 2 s a e a ) ( ∑ a ∈ A + e a ) . (196) \left(\sum_{a\in A_+}w_a\sqrt{s_a}\right)^2
=
\left(\sum_{a\in A_+}\frac{w_a\sqrt{s_a}}{\sqrt{e_a}}\sqrt{e_a}\right)^2
\le
\left(\sum_{a\in A_+}\frac{w_a^2s_a}{e_a}\right)
\left(\sum_{a\in A_+}e_a\right). \tag*{(196)} a ∈ A + ∑ w a s a 2 = a ∈ A + ∑ e a w a s a e a 2 ≤ a ∈ A + ∑ e a w a 2 s a a ∈ A + ∑ e a . (196)
At an optimum no mass is assigned outside A + A_+ A + , so the second factor is one. Equality holds exactly when e a ∝ w a s a e_a \propto w_a\sqrt{s_a} e a ∝ w a s a on A + A_+ A + , yielding the stated allocation and minimum.
Extensions, Assumption Ledger, and Research Protocol
Overview. This appendix collects complementary extensions and proposed uses of the framework: functional corrections, grouping selection, stochastic macro transitions, and resource-adjusted scores. Sections L.1–L.3 cover functional corrections, grouping, and stochastic dynamics; Sections L.4–L.5 distinguish resource preferences and assumptions.
Partial functional reuse beyond a scalar shrinkage coefficient
The exact linear theorem uses one scalar λ \lambda λ . A function-space shared core plus an r r r -dimensional role correction gives a complementary interpolation.
Proposition 37 (Residual bias of a rank-r r r functional correction). Under the unrestricted Hilbert quadratic setting F e = H \mathcal{F}_e = \mathcal{H} F e = H of Proposition 7 , assume finite-dimensional H \mathcal{H} H , fix an integer 0 ≤ r ≤ dim ( H ) 0 \le r \le\dim(\mathcal{H}) 0 ≤ r ≤ dim ( H ) , and define
Σ = ∑ a π a ( f a ⋆ − f ˉ ) ⊗ ( f a ⋆ − f ˉ ) (197) \Sigma= \sum_a \pi_a (f_a^\star-\bar{f}) \otimes(f_a^\star-\bar{f}) \tag*{(197)} Σ = a ∑ π a ( f a ⋆ − f ˉ ) ⊗ ( f a ⋆ − f ˉ ) (197) where ( u ⊗ u ) v = u ⟨ u , v ⟩ (u \otimes u)v = u\langle u,v\rangle ( u ⊗ u ) v = u ⟨ u , v ⟩ and the eigenvalues are λ 1 ≥ λ 2 ≥ ⋯ ≥ 0 \lambda_1 \ge\lambda_2 \ge\cdots\ge0 λ 1 ≥ λ 2 ≥ ⋯ ≥ 0 . They measure between-role variation along orthogonal function directions, not estimation-noise variance. Among role predictors f a = f ˉ + U z a f_a = \bar{f} + Uz_a f a = f ˉ + U z a with U : R r → H U:\mathbb{R}^r \to\mathcal{H} U : R r → H , U ⊤ U = I r U^\top U = I_r U ⊤ U = I r , and role-specific coordinates z a ∈ R r z_a \in\mathbb{R}^r z a ∈ R r , sharing an r r r -dimensional correction subspace, the minimum average excess risk is
( B e r e u s e ) ( r ) = ∑ k > r λ k . (198) \left(B_e^{\mathrm{reuse}}\right)^{(r)} = \sum_{k>r} \lambda_k. \tag*{(198)} ( B e reuse ) ( r ) = k > r ∑ λ k . (198)
Proof. For fixed subspace U U U , the optimal role correction is the orthogonal projection of f a ⋆ − f ˉ f_a^\star-\bar{f} f a ⋆ − f ˉ onto U U U . The average residual is tr ( ( I − P U ) Σ ) \operatorname{tr}((I-P_U)\Sigma) tr (( I − P U ) Σ ) . The rank-r r r projector maximizing captured trace spans the top r r r eigenvectors, with the usual convention that the tail is zero once r r r reaches the rank.
This is a function-space statement. Connecting it to a rank-r r r LoRA parameterization requires a separate realizability assumption about the network Jacobian or adapter class, and adding estimation variance for the learned role coordinates is necessary for a complete finite-sample adapter comparison.
For a concrete obstruction, take linear maps on R 2 \mathbb{R}^2 R 2 , input second moment I 2 I_2 I 2 , two equally weighted role targets f ± ( x ) = ± I 2 x f_\pm(x)=\pm I_2x f ± ( x ) = ± I 2 x , and a fixed zero shared core. Their centered variation spans one function direction, so a one-dimensional functional correction is exact. But either target matrix has two unit singular values: a matrix-rank-one update to this fixed core has squared Frobenius residual at least one. Thus function-subspace dimension one does not imply parameter-update rank one.
Sections O.5 and O.6 gives exact frozen-feature adapter tails, a two-role free-core result, and initialization-dependent tangent images.
Data-dependent grouping
The fixed-rule identities condition on a reuse assignment independent of fitting noise. Adaptive grouping therefore needs a separate selection argument.
In the exact linear model, Theorem 2 supplies a population objective without an ad hoc clustering metric: minimize within-group Frobenius dispersion plus V V V per fitted expert. Its pairwise merge statistic has the exact law in Equation (64) . Outside that model, the following validation-based construction retains the same logic without assuming that raw parameter distance equals functional heterogeneity.
First, fit a finite candidate family A \mathcal{A} A of groupings using training data and select among those fitted candidates on an independent validation split. Conditional on training, if validation loss is bounded and has effective size N v a l e f f N_{\mathrm{val}}^{\mathrm{eff}} N val eff , validation ERM satisfies the analogue of Equation (72) with ∣ Λ ∣ |\Lambda| ∣Λ∣ replaced by ∣ A ∣ |\mathcal{A}| ∣ A ∣ . Second, if grouping and expert fitting use the same observations, uniform convergence must cover the union of all grouped hypothesis classes; a finite grouping family contributes an additional log ∣ A ∣ \log|\mathcal{A}| log ∣ A ∣ term. An untouched final test is still required. Theorem 48 provides a finite-candidate oracle inequality for explicit loss-tail conditions, including a sufficient Gaussian quadratic model. Equation (71) illustrates the risk of a same-data dispersion plug-in.
A certificate-guided grouping objective may use
U ^ ( α ) = ∑ e W ^ e ( B ^ e + 2 ϵ ^ s h , e ) + ∑ j A ^ j ( τ ^ j + ρ ^ j ) + λ r e s Cost ( α ) . (199) \widehat{U}(\alpha)=\sum_e \widehat{W}_e\left(\widehat{B}_e+2\widehat{\epsilon}_{sh,e}\right)+\sum_j\widehat{A}_j\left(\widehat{\tau}_j+\widehat{\rho}_j\right)+\lambda_{\mathrm{res}}\operatorname{Cost}(\alpha). \tag*{(199)} U ( α ) = e ∑ W e ( B e + 2 ϵ s h , e ) + j ∑ A j ( τ j + ρ j ) + λ res Cost ( α ) . (199)
where the common role-specific oracle term has been omitted only when it is invariant across the compared partitions. Every fitted functional probe and plug-in quantity used to choose α \alpha α must be computed on training/validation data or by cross-fitting, and final evaluation is performed once on a held-out test set.
Stochastic macro dynamics
When the macro state is Markov-sufficient, deterministic transitions can be replaced by Markov kernels on a Polish (complete separable) metric space ( S , d ) (\mathcal{S},d) ( S , d ) with finite first moments. Let W d W_d W d denote the 1-Wasserstein distance with ground metric d d d , and let Q j Q_j Q j be the actual kernel and Q ˉ j \bar{Q}_j Q ˉ j a reference kernel satisfying
sup s W d ( Q j ( s , ⋅ ) , Q ˉ j ( s , ⋅ ) ) ≤ η j , W d ( Q ˉ j ( s , ⋅ ) , Q ˉ j ( s ′ , ⋅ ) ) ≤ L j d ( s , s ′ ) . (200) \sup_s W_d(Q_j(s,\cdot),\bar{Q}_j(s,\cdot)) \leq\eta_j,\qquad W_d(\bar{Q}_j(s,\cdot),\bar{Q}_j(s',\cdot)) \leq L_jd(s,s'). \tag*{(200)} s sup W d ( Q j ( s , ⋅ ) , Q ˉ j ( s , ⋅ )) ≤ η j , W d ( Q ˉ j ( s , ⋅ ) , Q ˉ j ( s ′ , ⋅ )) ≤ L j d ( s , s ′ ) . (200)
For distributions ξ j + 1 = ξ j Q j \xi_{j+1}=\xi_jQ_j ξ j + 1 = ξ j Q j and ξ ˉ j + 1 = ξ ˉ j Q ˉ j \bar{\xi}_{j+1}=\bar{\xi}_j\bar{Q}_j ξ ˉ j + 1 = ξ ˉ j Q ˉ j , conditional coupling and the triangle inequality give
W d ( ξ j + 1 , ξ ˉ j + 1 ) ≤ η j + L j W d ( ξ j , ξ ˉ j ) . (201) W_d(\xi_{j+1},\bar{\xi}_{j+1}) \leq\eta_j + L_jW_d(\xi_j,\bar{\xi}_j). \tag*{(201)} W d ( ξ j + 1 , ξ ˉ j + 1 ) ≤ η j + L j W d ( ξ j , ξ ˉ j ) . (201)
which telescopes exactly as Equation (75) . Standard kernel measurability is assumed. If the one-step discrepancy is state-dependent rather than uniformly bounded, its expectation under ξ j \xi_j ξ j replaces η j \eta_j η j . Without macro-state sufficiency, kernels must be defined on the full micro context before being pushed forward through Ψ \Psi Ψ .
Resource-adjusted reuse
We evaluate statistical gains alongside serving costs. Define every cost difference as
Δ C ∙ = C ∙ r e u s e − C ∙ s e p , (202) \Delta C_{\bullet}=C_{\bullet}^{\mathrm{reuse}}-C_{\bullet}^{\mathrm{sep}}, \tag*{(202)} Δ C ∙ = C ∙ reuse − C ∙ sep , (202)
so a positive value means that reuse is more costly. A deployment score may retain system costs separately:
P R G H r e s o u r c e = P R G H − λ m e m Δ C m e m o r y − λ c o m m Δ C c o m m u n i c a t i o n − λ l a t Δ C l a t e n c y − λ f l o p Δ C f l o p . (203) \mathrm{PRG}_{H}^{\mathrm{resource}}=\mathrm{PRG}_{H}-\lambda_{\mathrm{mem}}\Delta C_{\mathrm{memory}}-\lambda_{\mathrm{comm}}\Delta C_{\mathrm{communication}}-\lambda_{\mathrm{lat}}\Delta C_{\mathrm{latency}}-\lambda_{\mathrm{flop}}\Delta C_{\mathrm{flop}}. \tag*{(203)} PRG H resource = PRG H − λ mem Δ C memory − λ comm Δ C communication − λ lat Δ C latency − λ flop Δ C flop . (203)
The nonnegative coefficients convert the different resource units into the chosen score units and encode deployment preferences rather than statistical constants, and the raw cost changes should be reported alongside the scalarized score.
Assumption dependencies
Section Q audits the assumptions of each core result and extension, including the exact-model restrictions, estimator independence, transfer conditions, route semantics, and causal coverage. It distinguishes mathematical sufficiency from empirical plausibility. The covariance and marginal-margin extensions in Sections J and M are used when the stronger uncorrelated-error or conditional-margin assumptions are not credible.
Stronger Results under Weaker or More Explicit Assumptions
Overview. We identify what remains valid when the simplest assumptions are relaxed. The extensions retain general test geometry and covariance, optimization and transfer costs, and escape events. The final subsection separately explains how held-out paired evaluation can support actual performance improvement. Each extension specifies the object, randomness, and fixed quantities. Sections M.1–M.2 develop the general linear geometry; Sections M.3–M.6 retain optimization, transfer, repeated-error, and escape costs, before the paired audit in Section M.7.
The design Gram describes which coefficients can be estimated; the test geometry describes which errors matter at evaluation. They play different roles, even when both are built from features.
The orthogonal design X a T X a = n a I d X_a^{\mathsf{T}}X_a=n_aI_d X a T X a = n a I d is useful for isolating a scalar variance price, but it is not needed for an exact linear risk calculation. Let J J J role cells have parameter vectors θ a ∈ R D \theta_a\in\mathbb{R}^{D} θ a ∈ R D , stacked into θ ∈ R J D \theta\in\mathbb{R}^{JD} θ ∈ R J D . Suppose an unbiased base estimator has the form
Y = θ + ζ , E ζ = 0 , E [ ζ ζ T ] = Σ ⪰ 0. (204) Y=\theta+\zeta,\qquad\mathbb{E}\zeta=0,\qquad\mathbb{E}[\zeta\zeta^{\mathsf{T}}]=\Sigma\succeq0. \tag*{(204)} Y = θ + ζ , E ζ = 0 , E [ ζ ζ T ] = Σ ⪰ 0. (204)
Here Y Y Y denotes the stacked parameter estimates, not the earlier response matrix Y a Y_a Y a . The covariance Σ \Sigma Σ may have nonzero cross-role blocks. Fix a symmetric positive-semidefinite matrix W ∈ R J D × J D W \in\mathbb{R}^{JD \times JD} W ∈ R J D × J D , and define ∥ u ∥ W 2 = u ⊤ W u \lVert u\rVert_W^2 = u^\top W u ∥ u ∥ W 2 = u ⊤ W u . This is a squared seminorm: directions in the null space of W W W do not affect the chosen evaluation loss. A fixed matrix L ∈ R J D × J D L \in\mathbb{R}^{JD \times JD} L ∈ R J D × J D specifies a linear sharing rule θ ~ L = L Y \tilde{\theta}_L = LY θ ~ L = L Y . The risk is
R W ( L ) = E ∥ L Y − θ ∥ W 2 . (205) R_W(L) = \mathbb{E}\lVert LY - \theta\rVert_W^2. \tag*{(205)} R W ( L ) = E ∥ L Y − θ ∥ W 2 . (205)
“Fixed” means chosen independently of ζ \zeta ζ , or conditioned on an independent pilot sample. No Gaussian distribution is assumed.
Theorem 38. (Exact covariance-aware risk of a fixed linear sharing rule.) Under Equation (204) ,
R W ( L ) = ∥ ( L − I ) θ ∥ W 2 + tr ( W L Σ L ⊤ ) . (206) R_W(L) = \lVert(L-I)\theta\rVert_W^2 + \operatorname{tr}(WL\Sigma L^\top). \tag*{(206)} R W ( L ) = ∥( L − I ) θ ∥ W 2 + tr ( W L Σ L ⊤ ) . (206) If Σ \Sigma Σ is known, then
R ^ W ( L ) = ∥ ( L − I ) Y ∥ W 2 + 2 tr ( W L Σ ) − tr ( W Σ ) (207) \widehat{R}_W(L) = \lVert(L-I)Y\rVert_W^2 + 2\operatorname{tr}(WL\Sigma) - \operatorname{tr}(W\Sigma) \tag*{(207)} R W ( L ) = ∥( L − I ) Y ∥ W 2 + 2 tr ( W L Σ ) − tr ( W Σ ) (207) is unbiased for the risk of this fixed L L L .
Proof. Write L Y − θ = ( L − I ) θ + L ζ LY-\theta=(L-I)\theta+L\zeta L Y − θ = ( L − I ) θ + L ζ . Its deterministic–centered cross term has zero expectation, and E [ ζ ⊤ L ⊤ W L ζ ] = tr ( W L Σ L ⊤ ) \mathbb{E}[\zeta^\top L^\top W L\zeta]=\operatorname{tr}(WL\Sigma L^\top) E [ ζ ⊤ L ⊤ W L ζ ] = tr ( W L Σ L ⊤ ) , proving Equation (206) . Also
E ∥ ( L − I ) Y ∥ W 2 = ∥ ( L − I ) θ ∥ W 2 + tr ( W ( L − I ) Σ ( L − I ) ⊤ ) . \mathbb{E}\lVert(L-I)Y\rVert_W^2=\lVert(L-I)\theta\rVert_W^2+\operatorname{tr}\bigl(W(L-I)\Sigma(L-I)^\top\bigr). E ∥( L − I ) Y ∥ W 2 = ∥( L − I ) θ ∥ W 2 + tr ( W ( L − I ) Σ ( L − I ) ⊤ ) .
Expanding the last trace and using symmetry of W , Σ W,\Sigma W , Σ leaves the risk minus 2 tr ( W L Σ ) + tr ( W Σ ) 2\operatorname{tr}(WL\Sigma)+\operatorname{tr}(W\Sigma) 2 tr ( W L Σ ) + tr ( W Σ ) , proving unbiasedness.
This is the fixed-linear-estimator case of unbiased risk estimation, with an elementary second-moment proof [36 ] . Unbiasedness for each pre-specified L L L does not imply unbiasedness of the minimum over data-selected rules. Selecting a grouping on these scores requires independent evaluation or uniform selection control. Replacing Σ \Sigma Σ by an estimate also adds an estimation problem. The unbiased score may be negative in a sample even though its expected risk is nonnegative.
Instantiation for nonorthogonal operator regression. For each role, let X a ∈ R n a × d X_a \in\mathbb{R}^{n_a \times d} X a ∈ R n a × d have full column rank, define its design Gram matrix G a = X a ⊤ X a ≻ 0 G_a=X_a^\top X_a\succ0 G a = X a ⊤ X a ≻ 0 , and observe Y a = X a A a ⊤ + E a Y_a=X_aA_a^\top+\mathcal{E}_a Y a = X a A a ⊤ + E a . Assume the noise rows are centered, independent, and have output covariance Ω a ∈ R p × p \Omega_a\in\mathbb{R}^{p\times p} Ω a ∈ R p × p . Then
A ^ a = Y a ⊤ X a G a − 1 , Cov ( vec ( A ^ a − A a ) ) = G a − 1 ⊗ Ω a . (208) \widehat{A}_a=Y_a^\top X_aG_a^{-1},\qquad\operatorname{Cov}(\operatorname{vec}(\widehat{A}_a-A_a))=G_a^{-1}\otimes\Omega_a. \tag*{(208)} A a = Y a ⊤ X a G a − 1 , Cov ( vec ( A a − A a )) = G a − 1 ⊗ Ω a . (208)
The operation vec \operatorname{vec} vec stacks a matrix’s columns; here ⊗ \otimes ⊗ is the Kronecker product, which forms a block matrix by multiplying each entry of its first argument by the second matrix. Independence across roles makes the stacked covariance block diagonal; correlated role fits instead require their actual cross-covariances.
Let the independent test input in role a a a have second moment T a = E [ x a x a ⊤ ] ⪰ 0 T_a=\mathbb{E}[x_ax_a^\top]\succeq0 T a = E [ x a x a ⊤ ] ⪰ 0 , and let evaluation probabilities π a > 0 \pi_a>0 π a > 0 sum to one. Since
E x a ∥ B x a ∥ 2 2 = tr ( B T a B ⊤ ) = vec ( B ) ⊤ ( T a ⊗ I p ) vec ( B ) , (209) \mathbb{E}_{x_a}\lVert Bx_a\rVert_2^2=\operatorname{tr}(BT_aB^\top)=\operatorname{vec}(B)^\top(T_a\otimes I_p)\operatorname{vec}(B), \tag*{(209)} E x a ∥ B x a ∥ 2 2 = tr ( B T a B ⊤ ) = vec ( B ) ⊤ ( T a ⊗ I p ) vec ( B ) , (209)
choose
W = blockdiag a = 1 J ( π a ( T a ⊗ I p ) ) . (210) W=\operatorname{blockdiag}_{a=1}^{J}\bigl(\pi_a(T_a\otimes I_p)\bigr). \tag*{(210)} W = blockdiag a = 1 J ( π a ( T a ⊗ I p ) ) . (210)
Then Equation (206) is exactly the test prediction risk. This accommodates anisotropic representations, unequal test frequencies, and ill-conditioned but full-rank designs. A nearly singular G a G_a G a increases estimation variance through G a − 1 G_a^{-1} G a − 1 ; role count alone does not describe this difficulty. For rank-deficient designs, OLS is not an unbiased estimator of all coordinates without an identifiability restriction, so Equation (204) cannot simply be assumed.
The exact pooling frontier survives arbitrary covariance
For this subsection suppose W ≻ 0 W \succ0 W ≻ 0 , so it defines an inner product ⟨ u , v ⟩ W = u ⊤ W v \langle u,v\rangle_W = u^\top Wv ⟨ u , v ⟩ W = u ⊤ W v . A fixed sharing subspace V ⊆ R J D \mathcal{V} \subseteq\mathbb{R}^{JD} V ⊆ R J D consists of all parameter arrays allowed by a tying rule. If the columns of a full-column-rank matrix B V B_{\mathcal{V}} B V form a basis for that subspace, its weighted orthogonal projector is
P = B V ( B V ⊤ W B V ) − 1 B V ⊤ W . (211) P = B_{\mathcal{V}}\left(B_{\mathcal{V}}^\top W B_{\mathcal{V}}\right)^{-1}B_{\mathcal{V}}^\top W. \tag*{(211)} P = B V ( B V ⊤ W B V ) − 1 B V ⊤ W . (211)
For the zero subspace define P = 0 P = 0 P = 0 . In either case P 2 = P P^2 = P P 2 = P , P ⊤ W = W P P^\top W = WP P ⊤ W = W P , and P u Pu P u is the closest admissible vector to u u u in the W W W -norm. It need not be symmetric in ordinary Euclidean coordinates.
Let Q = I − P Q = I - P Q = I − P and define
B W = ∥ Q θ ∥ W 2 , V 0 = tr ( W Σ ) , V P = tr ( W P Σ P ⊤ ) , C W = tr ( W Q Σ Q ⊤ ) ≥ 0. (212) B_W = \lVert Q\theta\rVert_W^2,\qquad V_0 = \operatorname{tr}(W\Sigma),\qquad V_P = \operatorname{tr}(WP\Sigma P^\top),\qquad C_W = \operatorname{tr}(WQ\Sigma Q^\top) \ge0. \tag*{(212)} B W = ∥ Qθ ∥ W 2 , V 0 = tr ( W Σ ) , V P = tr ( W P Σ P ⊤ ) , C W = tr ( W Q Σ Q ⊤ ) ≥ 0. (212)
The bias B W B_W B W is the population cost of the sharing restriction, V 0 V_0 V 0 is the no-sharing estimation risk, and C W C_W C W is the noise variance removed by that restriction.
Theorem 39. (Covariance- and geometry-aware partial pooling). For L λ = I − λ Q L_\lambda= I - \lambda Q L λ = I − λ Q , 0 ≤ λ ≤ 1 0 \le\lambda\le1 0 ≤ λ ≤ 1 , under Equation (204) and W ≻ 0 W \succ0 W ≻ 0 ,
V 0 = V P + C W , V_0 = V_P + C_W, V 0 = V P + C W , R W ( L λ ) = V 0 − 2 C W λ + ( B W + C W ) λ 2 . (213) R_W(L_\lambda) = V_0 - 2C_W\lambda+ (B_W + C_W)\lambda^2. \tag*{(213)} R W ( L λ ) = V 0 − 2 C W λ + ( B W + C W ) λ 2 . (213) Full projection improves expected risk over L = I L = I L = I exactly when B W < C W B_W < C_W B W < C W . If B W + C W > 0 B_W + C_W > 0 B W + C W > 0 , the optimal fixed coefficient is λ ⋆ = C W / ( B W + C W ) \lambda^\star= C_W/(B_W + C_W) λ ⋆ = C W / ( B W + C W ) ; if both vanish, every coefficient has the same risk. No Gaussianity, isotropic covariance, or independence across role estimates is required.
Proof. Weighted orthogonality gives P ⊤ W Q = 0 P^\top WQ = 0 P ⊤ W Q = 0 . Consequently the noise components P ζ P\zeta P ζ and Q ζ Q\zeta Qζ are orthogonal in the evaluation geometry for every realization, even if they are statistically dependent. Therefore V 0 = V P + C W V_0 = V_P + C_W V 0 = V P + C W . The error decomposes as
L λ Y − θ = − λ Q θ + P ζ + ( 1 − λ ) Q ζ . L_\lambda Y - \theta= -\lambda Q\theta+ P\zeta+ (1-\lambda)Q\zeta. L λ Y − θ = − λ Qθ + P ζ + ( 1 − λ ) Qζ .
The bias–noise cross term has expectation zero, while the two projected noise components are pointwise orthogonal. Thus the risk is λ 2 B W + V P + ( 1 − λ ) 2 C W \lambda^2 B_W + V_P + (1-\lambda)^2 C_W λ 2 B W + V P + ( 1 − λ ) 2 C W . Expansion and minimization prove the claims.
This strengthens the scalar frontier. If every role estimate has exactly the same random error and the sharing subspace contains all common-role arrays, then Q ζ = 0 Q\zeta= 0 Qζ = 0 and C W = 0 C_W = 0 C W = 0 : averaging does not remove that common error. Full sharing then only adds bias. Conversely, covariance concentrated in directions excluded by the sharing subspace makes pooling especially useful.
For common-operator sharing in Equation (210) , the fitted shared matrix is
A ^ t e s t = ( ∑ a π a A ^ a T a ) ( ∑ a π a T a ) − 1 , (214) \widehat{A}_{\mathrm{test}} = \left(\sum_a \pi_a\widehat{A}_aT_a\right)\left(\sum_a \pi_aT_a\right)^{-1}, \tag*{(214)} A test = ( a ∑ π a A a T a ) ( a ∑ π a T a ) − 1 , (214)
provided all T a ≻ 0 T_a \succ0 T a ≻ 0 . This is the weighted projection estimator, not necessarily ordinary pooled training OLS. Pooled OLS weights roles by their training design Grams; Equation (214) weights them by the declared test geometry.
Corollary 40. (Exact value of enlarging a sharing architecture). Let U ⊆ V \mathcal{U} \subseteq\mathcal{V} U ⊆ V be two fixed sharing subspaces, with weighted projectors P U , P V P_{\mathcal{U}}, P_{\mathcal{V}} P U , P V . Define Q n e w = P V − P U Q_{\mathrm{new}} = P_{\mathcal{V}} - P_{\mathcal{U}} Q new = P V − P U . Then
R W ( P U ) − R W ( P V ) = ∥ Q n e w θ ∥ W 2 − tr ( W Q n e w Σ Q n e w ⊤ ) . (215) R_W(P_{\mathcal{U}}) - R_W(P_{\mathcal{V}}) = \lVert Q_{\mathrm{new}}\theta\rVert_W^2 - \operatorname{tr}(WQ_{\mathrm{new}}\Sigma Q_{\mathrm{new}}^\top). \tag*{(215)} R W ( P U ) − R W ( P V ) = ∥ Q new θ ∥ W 2 − tr ( W Q new Σ Q new ⊤ ) . (215) Thus an additional depth effect, role correction, or fixed adapter direction is useful exactly when its recovered signal exceeds its covariance-weighted estimation cost.
Proof. Because the subspaces are nested, Q n e w Q_{\mathrm{new}} Q new projects onto the directions in V \mathcal{V} V weighted-orthogonal to U \mathcal{U} U . Both bias and noise split into weighted-orthogonal components. Enlarging the subspace removes squared bias ∥ Q n e w θ ∥ W 2 \lVert Q_{\mathrm{new}}\theta\rVert_W^2 ∥ Q new θ ∥ W 2 and adds variance tr ( W Q n e w Σ Q n e w ⊤ ) \operatorname{tr}(WQ_{\mathrm{new}}\Sigma Q_{\mathrm{new}}^\top) tr ( W Q new Σ Q new ⊤ ) .
The earlier dimension penalty is recovered when W = I / J W = I/J W = I / J and Σ = v I \Sigma= vI Σ = v I . In general, equal parameter counts do not imply equal estimation costs. Learned subspaces or ranks must still be selected independently or analyzed with a selection penalty.
Approximate training and mismatched role weights
Fix one group and the evaluation weight vector π \pi π , allowing zero entries provided the empirical losses are defined for every role used in training. Let ν \nu ν be the normalized weight vector used by the training algorithm. Define
R ^ π ( f ) = ∑ a π a R ^ a ( f ) , R ^ ν ( f ) = ∑ a ν a R ^ a ( f ) , TV ( π , ν ) = 1 2 ∑ a ∣ π a − ν a ∣ . (216) \widehat{R}_{\pi}(f)=\sum_{a}\pi_a\widehat{R}_a(f),\qquad\widehat{R}_{\nu}(f)=\sum_{a}\nu_a\widehat{R}_a(f),\qquad\operatorname{TV}(\pi,\nu)=\frac{1}{2}\sum_{a}\lvert\pi_a-\nu_a\rvert. \tag*{(216)} R π ( f ) = a ∑ π a R a ( f ) , R ν ( f ) = a ∑ ν a R a ( f ) , TV ( π , ν ) = 2 1 a ∑ ∣ π a − ν a ∣ . (216)
Assume 0 ≤ ℓ a ≤ 1 0\leq\ell_a\leq1 0 ≤ ℓ a ≤ 1 and f ^ ∈ F \widehat{f}\in\mathcal{F} f ∈ F satisfies R ^ ν ( f ^ ) ≤ inf f ∈ F R ^ ν ( f ) + η o p t \widehat{R}_{\nu}(\widehat{f})\leq\inf_{f\in\mathcal{F}}\widehat{R}_{\nu}(f)+\eta_{\mathrm{opt}} R ν ( f ) ≤ inf f ∈ F R ν ( f ) + η opt , where η o p t ≥ 0 \eta_{\mathrm{opt}}\geq0 η opt ≥ 0 is an empirical optimization gap. The gap may be computationally unknown and is retained explicitly in the bound.
Proposition 41 (Approximate ERM with explicit weight mismatch). On any event with sup f ∣ R π ( f ) − R ^ π ( f ) ∣ ≤ u π \sup_f\lvert R_{\pi}(f)-\widehat{R}_{\pi}(f)\rvert\leq u_{\pi} sup f ∣ R π ( f ) − R π ( f )∣ ≤ u π ,
R π ( f ^ ) ≤ inf f R π ( f ) + 2 u π + η o p t + 2 TV ( π , ν ) . (217) R_{\pi}(\widehat{f})\leq\inf_f R_{\pi}(f)+2u_{\pi}+\eta_{\mathrm{opt}}+2\operatorname{TV}(\pi,\nu). \tag*{(217)} R π ( f ) ≤ f inf R π ( f ) + 2 u π + η opt + 2 TV ( π , ν ) . (217)
The training weights ν \nu ν may themselves be data dependent: the bound uses their realized distance from the fixed evaluation weights, while the uniform event is over the fixed class and fixed π \pi π .
Proof. For every f f f , bounded losses imply ∣ R ^ π ( f ) − R ^ ν ( f ) ∣ ≤ TV ( π , ν ) \lvert\widehat{R}_{\pi}(f)-\widehat{R}_{\nu}(f)\rvert\leq\operatorname{TV}(\pi,\nu) ∣ R π ( f ) − R ν ( f )∣ ≤ TV ( π , ν ) . Move from R π ( f ^ ) R_{\pi}(\widehat{f}) R π ( f ) to its empirical value, then to the training objective, apply approximate minimization, and reverse the two changes at a population-optimal candidate (or an arbitrarily close one). The two uniform deviations contribute 2 u π 2u_{\pi} 2 u π and the two weight changes contribute 2 TV ( π , ν ) 2\operatorname{TV}(\pi,\nu) 2 TV ( π , ν ) .
In Theorem 4, the shared certificate therefore acquires ∑ e W e [ η o p t , e + 2 TV ( π e , ν e ) ] \sum_e W_e[\eta_{\mathrm{opt},e}+2\operatorname{TV}(\pi_e,\nu_e)] ∑ e W e [ η opt , e + 2 TV ( π e , ν e )] . If separate fits have gaps η a \eta_a η a , their certificate acquires ∑ a w a η a \sum_a w_a\eta_a ∑ a w a η a . The corrected bound difference is the original P R G H \mathrm{PRG}_H PRG H plus the separate correction minus the shared correction. Zero-evaluation-weight roles can influence training through ν \nu ν and are now explicitly charged rather than discarded. For task losses unrelated to the event-lift loss, an optimization-gap or calibration bound remains necessary.
Transfer without identical distribution supports
A bounded density ratio is a strong sufficient condition: it fails if rollout visits states outside the probe distribution’s support. Even on overlapping supports, two shifted Gaussians with equal variance generally have an unbounded density ratio. The following replacement needs no absolute continuity.
Use the conditional occurrence measures Q j , q , a m Q^{m}_{j,q,a} Q j , q , a m from Equation (33) . Let s j , q , a m = Q j , q , a m ( Z a ) s^{m}_{j,q,a}=Q^{m}_{j,q,a}(\mathcal{Z}_a) s j , q , a m = Q j , q , a m ( Z a ) be the probability that occurrence q q q exists and has role a a a . When s j , q , a m > 0 s^{m}_{j,q,a}>0 s j , q , a m > 0 , let ν j , q , a m = Q j , q , a m / s j , q , a m \nu^{m}_{j,q,a}=Q^{m}_{j,q,a}/s^{m}_{j,q,a} ν j , q , a m = Q j , q , a m / s j , q , a m be its conditional probe distribution. Define a discrepancy over the actual loss class by
∂ a ( ν , P a ) = sup f ∈ F a ∣ E Z ∼ ν ℓ a ( f ; Z ) − E Z ∼ P a ℓ a ( f ; Z ) ∣ . (218) \partial_a(\nu,P_a)=\sup_{f\in\mathcal{F}_a}\left\lvert\mathbb{E}_{Z\sim\nu}\ell_a(f;Z)-\mathbb{E}_{Z\sim P_a}\ell_a(f;Z)\right\rvert. \tag*{(218)} ∂ a ( ν , P a ) = f ∈ F a sup ∣ E Z ∼ ν ℓ a ( f ; Z ) − E Z ∼ P a ℓ a ( f ; Z ) ∣ . (218)
The class F a \mathcal{F}_a F a is the applicable fixed expert class for role a a a . This discrepancy measures changes in task loss, not changes in arbitrary features; for losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] it is at most one.
Proposition 42 (Additive occupancy transfer). Assume the call-replacement lemma applies and choose deterministic occurrence bounds s j , q , a m ≤ s ‾ j , q , a s^{m}_{j,q,a}\leq\overline{s}_{j,q,a} s j , q , a m ≤ s j , q , a , valid for both comparators independently of expert-fitting data. Then Assumption 2 holds with
ω j , a = ∑ q s ‾ j , q , a , \omega_{j,a}=\sum_q\overline{s}_{j,q,a}, ω j , a = q ∑ s j , q , a , τ ~ j m = τ j m + ∑ q , a s j , q , a m ∂ a ( ν j , q , a m , P a ) . (219) \widetilde{\tau}^{m}_j=\tau^{m}_j+\sum_{q,a}s^{m}_{j,q,a}\partial_a(\nu^{m}_{j,q,a},P_a). \tag*{(219)} τ j m = τ j m + q , a ∑ s j , q , a m ∂ a ( ν j , q , a m , P a ) . (219)
Zero-mass occurrences contribute zero. If the probe space has metric d Z a d_{\mathcal{Z}_a} d Z a and every ℓ a ( f ; ⋅ ) \ell_a(f;\cdot) ℓ a ( f ; ⋅ ) is uniformly L a p r o b e L_a^{\mathrm{probe}} L a probe -Lipschitz, then, for distributions with finite first moments,
∂ a ( ν , P a ) ≤ L a p r o b e W 1 ( ν , P a ; d Z a ) . (220) \partial_a(\nu,P_a)\leq L_a^{\mathrm{probe}}W_1(\nu,P_a;d_{\mathcal{Z}_a}). \tag*{(220)} ∂ a ( ν , P a ) ≤ L a probe W 1 ( ν , P a ; d Z a ) . (220)
For bounded losses, the minimum of this expression and one is also valid.
Proof. For the fitted function, conditional on its fitting data,
∫ ℓ a ( f ^ ; z ) d Q j , q , a m ( z ) = s j , q , a m E ν j , q , a m ℓ a ( f ^ ; Z ) ≤ s ˉ j , q , a R a ( f ^ ) + s j , q , a m d a ( ν j , q , a m , P a ) . \int\ell_a(\hat{f};z)\,dQ^{m}_{j,q,a}(z)=s^{m}_{j,q,a}\mathbb{E}_{\nu^{m}_{j,q,a}}\ell_a(\hat{f};Z)\le\bar{s}_{j,q,a}R_a(\hat{f})+s^{m}_{j,q,a}d_a(\nu^{m}_{j,q,a},P_a). ∫ ℓ a ( f ^ ; z ) d Q j , q , a m ( z ) = s j , q , a m E ν j , q , a m ℓ a ( f ^ ; Z ) ≤ s ˉ j , q , a R a ( f ^ ) + s j , q , a m d a ( ν j , q , a m , P a ) .
Nonnegativity of R a R_a R a justifies replacing s s s by s ˉ \bar{s} s ˉ only in its coefficient. Sum the call losses. For Equation (220) , couple Z ∼ ν Z\sim\nu Z ∼ ν and Z ′ ∼ P a Z'\sim P_a Z ′ ∼ P a . For every f f f the expectation difference is at most L a p r o b e E d Z a ( Z , Z ′ ) L_a^{\mathrm{probe}}\mathbb{E}_{d_{Z_a}}(Z,Z') L a probe E d Z a ( Z , Z ′ ) . Minimize over couplings and take the supremum over f f f .
The coupling argument is the standard mechanism behind Wasserstein transfer bounds [33 ] ; here it is applied to role-conditioned event probes and their occurrence masses. Small transport distance is useful only when the loss class has controlled smoothness. The additive shift term must be bounded or estimated independently, and its uncertainty reported.
Repeated experts create coherent, rather than fresh, errors
The off-diagonal terms in Equation (87) have a direct interpretation when a fitted parameter is reused. Consider the exact additive-error model
ξ j = b j + B j δ e j + ϵ j . (221) \xi_j=b_j+B_j\delta_{e_j}+\epsilon_j. \tag*{(221)} ξ j = b j + B j δ e j + ϵ j . (221)
Here the expert-index sequence e j e_j e j and matrices M j M_j M j , B j B_j B j are deterministic. Each δ e ∈ R d e \delta_e\in\mathbb{R}^{d_e} δ e ∈ R d e is a centered random fitting error drawn once per expert and reused at every occurrence of that expert, with covariance V e V_e V e . The s × d e j s\times d_{e_j} s × d e j matrix B j B_j B j maps parameter error into a local state perturbation. The errors ϵ j ∈ R s \epsilon_j\in\mathbb{R}^{s} ϵ j ∈ R s are centered with covariances R j R_j R j , pairwise uncorrelated across time and uncorrelated with all δ e \delta_e δ e ; distinct δ e \delta_e δ e are uncorrelated. All second moments are finite. Define
K e = ∑ j : e j = e P j B j . (222) K_e=\sum_{j:e_j=e}P_jB_j. \tag*{(222)} K e = j : e j = e ∑ P j B j . (222)
Thus K e K_e K e adds all uses of expert e e e after mapping them into terminal-state coordinates.
Proposition 43 (Exact covariance cost of repeated expert use). *For the linear recursion in Proposition 13 and Equation (221) ,
E ∥ e H ∥ 2 2 = ∥ ∑ j P j b j ∥ 2 2 + ∑ e tr ( K e V e K e T ) + ∑ j tr ( P j R j P j T ) . (223) \mathbb{E}\lVert e_H\rVert_2^2=
\left\lVert\sum_jP_jb_j\right\rVert_2^2+
\sum_e\operatorname{tr}(K_eV_eK_e^{\mathsf{T}})+
\sum_j\operatorname{tr}(P_jR_jP_j^{\mathsf{T}}). \tag*{(223)} E ∥ e H ∥ 2 2 = j ∑ P j b j 2 2 + e ∑ tr ( K e V e K e T ) + j ∑ tr ( P j R j P j T ) . (223)
Proof. Unroll the recursion and collect repeated occurrences of the same random vector: e H = ∑ j P j b j + ∑ e K e δ e + ∑ j P j ϵ j e_H=\sum_jP_jb_j+\sum_eK_e\delta_e+\sum_jP_j\epsilon_j e H = ∑ j P j b j + ∑ e K e δ e + ∑ j P j ϵ j . The specified cross-covariances vanish between these collected components. Take their covariance and then its trace.
For the scalar example M j = B j = 1 M_j=B_j=1 M j = B j = 1 , b j = ϵ j = 0 b_j=\epsilon_j=0 b j = ϵ j = 0 , one expert used H H H times with fitting-error variance v v v gives E e H 2 = H 2 v \mathbb{E}e_H^2=H^2v E e H 2 = H 2 v . Fresh independent errors with the same variance at each call instead give H v Hv H v . Oppositely signed propagated effects may cancel, so reuse does not always increase the covariance cost. The relevant quantity is K e K_e K e , not merely occurrence count.
Equation (221) is exact for additive residual perturbations. For a nonlinear expert it may be a first-order linearization; applying it then requires a separate remainder bound. Also, conditioning on a route sequence selected using δ e \delta_e δ e generally changes the error means and covariances. The deterministic/exogenous-route specialization must not be justified by conditioning on arbitrary endogenous routes and retaining the original V e V_e V e .
Local stability regions with an explicit escape cost
Uniform control on every possible hidden state can be much stronger than needed. Let Ω \Omega Ω be the event that the reference, learned, and required hybrid states remain in specified deterministic regions where the local recursion and a utility Lipschitz constant L U L_{\mathcal{U}} L U are valid. Assume common initial states and that terminal utility is globally bounded in an interval of width R U R_U R U . Let A j A_j A j be the products of the local reference Lipschitz envelopes. Conditional on the fitted models, suppose
E [ 1 Ω ∑ j A j ( a j + r j ) ] ≤ U l o c , P ( Ω c ) ≤ p e x i t . \mathbb{E}\left[\mathbf{1}_{\Omega}\sum_j A_j(a_j+r_j)\right]\le U_{\mathrm{loc}},\qquad\mathbb{P}(\Omega^c)\le p_{\mathrm{exit}}. E [ 1 Ω j ∑ A j ( a j + r j ) ] ≤ U loc , P ( Ω c ) ≤ p exit .
Here U l o c U_{\mathrm{loc}} U loc controls only paths remaining inside the region, and p e x i t p_{\mathrm{exit}} p exit is the probability of leaving it.
Proposition 44 (Localized utility certificate). Under these conditions,
E ∣ U ( S ^ H ) − U ( S H ) ∣ ≤ L U U l o c + R U p e x i t . (224) \mathbb{E}|U(\widehat{S}_H)-U(S_H)|\le L_U U_{\mathrm{loc}}+R_U p_{\mathrm{exit}}. \tag*{(224)} E ∣ U ( S H ) − U ( S H ) ∣ ≤ L U U loc + R U p exit . (224) Proof. On Ω \Omega Ω , apply the pathwise recursion and the local utility bound. On Ω c \Omega^c Ω c , use the global utility range. Split the expectation into these two events.
The indicator-restricted probe measures can be treated by the same nonnegative-loss aggregation argument as other sub-probability measures. No independence of Ω \Omega Ω and local errors is required. A local analysis is informative only when the escape probability is also controlled.
Expected accuracy and actual improvement are separate claims
A Lipschitz state certificate does not directly bound discrete answer accuracy. Let the same readout g : S → R K g:\mathcal{S}\to\mathbb{R}^K g : S → R K be L g L_g L g -Lipschitz from d d d to ℓ ∞ \ell_\infty ℓ ∞ , let E = d ( S ^ H , S H ) E=d(\widehat{S}_H,S_H) E = d ( S H , S H ) , and let γ ≥ 0 \gamma\ge0 γ ≥ 0 be the reference top-two logit gap. For any deterministic t > 0 t>0 t > 0 ,
P ( different predicted answers ) ≤ P ( γ ≤ 2 L g t ) + P ( E > t ) . (225) \mathbb{P}(\text{different predicted answers})\le\mathbb{P}(\gamma\le2L_g t)+\mathbb{P}(E>t). \tag*{(225)} P ( different predicted answers ) ≤ P ( γ ≤ 2 L g t ) + P ( E > t ) . (225)
This follows by splitting on E ≤ t E\le t E ≤ t and applying the pathwise margin test. It needs no independence between the margin and state error. A moment bound gives P ( E > t ) ≤ E E r / t r \mathbb{P}(E>t)\le\mathbb{E}E^r/t^r P ( E > t ) ≤ E E r / t r for any r > 0 r>0 r > 0 . A model-specific readout adds its common-state logit discrepancy to L g E L_gE L g E . This controls disagreement with the reference; correctness relative to ground truth additionally requires the reference’s own accuracy.
One can instead test actual improvement directly. Fix K c a n d K_{\mathrm{cand}} K cand fitted candidate systems and a separate comparator before observing an independent IID audit set of size N N N . All losses ℓ m ( Z ) \ell_m(Z) ℓ m ( Z ) and ℓ s e p ( Z ) \ell_{\mathrm{sep}}(Z) ℓ sep ( Z ) lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and are measured on the same example Z Z Z . Define the paired differences and their means by
D i , m = ℓ s e p ( Z i ) − ℓ m ( Z i ) , Δ ^ m = 1 N ∑ i D i , m , Δ m = E D i , m . (226) D_{i,m}=\ell_{\mathrm{sep}}(Z_i)-\ell_m(Z_i),\qquad\widehat{\Delta}_m=\frac{1}{N}\sum_i D_{i,m},\qquad\Delta_m=\mathbb{E}D_{i,m}. \tag*{(226)} D i , m = ℓ sep ( Z i ) − ℓ m ( Z i ) , Δ m = N 1 i ∑ D i , m , Δ m = E D i , m . (226)
Positive Δ m \Delta_m Δ m means lower true risk for candidate m m m under this audit distribution.
Proposition 45 (A held-out certificate of actual performance improvement). Conditional on fitted systems, with probability at least 1 − δ 1-\delta 1 − δ ,
Δ m ≥ Δ ^ m − 2 log ( K c a n d / δ ) N simultaneously for all m . (227) \Delta_m\ge\widehat{\Delta}_m-\sqrt{\frac{2\log(K_{\mathrm{cand}}/\delta)}{N}}\qquad\text{simultaneously for all }m. \tag*{(227)} Δ m ≥ Δ m − N 2 log ( K cand / δ ) simultaneously for all m . (227) Thus exceeding this threshold certifies improvement of a fitted system’s true audit-distribution risk, not just improvement of an upper bound.
Proof. Each paired difference lies in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . One-sided Hoeffding gives P ( Δ ^ m − Δ m > t ) ≤ exp ( − N t 2 / 2 ) \mathbb{P}(\widehat{\Delta}_m-\Delta_m>t)\le\exp(-Nt^2/2) P ( Δ m − Δ m > t ) ≤ exp ( − N t 2 /2 ) . Set the probability to δ / K c a n d \delta/K_{\mathrm{cand}} δ / K cand and take a union bound. Pairing requires no independence between the two models’ losses on one example; independence is across audit records.
The same simultaneous event permits selection among this pre-fitted finite candidate family. Repeatedly generating new candidates in response to the audit results invalidates this guarantee without further selection control. For unbounded squared loss, the bounded Hoeffding radius cannot be used; a justified tail model or a separately declared bounded evaluation loss is needed. This audit and P R G H \mathrm{PRG}_H PRG H answer different questions: the latter explains a mechanism under modeling assumptions, whereas the former tests actual performance on a specified population.
Terminal-Risk Frontiers and Valid Adaptive Selection
Overview. We connect local sharing to a common terminal geometry, exhibit a reversal of local improvement, and characterize a restricted setting in which every local full-pooling gain survives. Independent selection and nonlinear approximation require separate guarantees. Fixed matrices may instead be conditioned on an independent pilot, provided the stated conditional moments hold; this does not permit treating a same-data selected rule as fixed. Section N.1 derives the terminal frontier and reversal, Section N.2 treats independent finite-family selection, Section N.3 retains the nonlinear remainder, and Section N.4 characterizes uniform preservation in its stated submodel.
The terminal geometry can change the preferred amount of sharing
Let Y = θ + ζ ∈ R p Y=\theta+\zeta\in\mathbb{R}^{p} Y = θ + ζ ∈ R p collect unbiased estimates of p p p role coefficients, with fixed target θ \theta θ , E ζ = 0 \mathbb{E}\zeta=0 E ζ = 0 , and E [ ζ ζ T ] = Σ ⪰ 0 \mathbb{E}[\zeta\zeta^{\mathsf{T}}]=\Sigma\succeq0 E [ ζ ζ T ] = Σ ⪰ 0 . A fixed linear sharing rule L L L estimates θ \theta θ by L Y LY L Y . Suppose the terminal error of every compared rule is exactly
e H ( L ) = G ( L Y − θ ) , G ∈ R s × p . (228) e_H(L)=G(LY-\theta),\qquad G\in\mathbb{R}^{s\times p}. \tag*{(228)} e H ( L ) = G ( L Y − θ ) , G ∈ R s × p . (228)
The same propagation matrix G G G is used for all comparators and is independent of ζ \zeta ζ . For example, in Proposition 13, if each local injection is ξ j = H j ( L Y − θ ) \xi_j=H_j(LY-\theta) ξ j = H j ( L Y − θ ) with fixed H j H_j H j , then G = ∑ j P j H j G=\sum_jP_jH_j G = ∑ j P j H j . This sums repeated uses before calculating their covariance. Define W H = G T G W_H=G^{\mathsf{T}}G W H = G T G ; it is positive semidefinite and may be singular, so ∥ x ∥ W H 2 = x T W H x \|x\|_{W_H}^2=x^{\mathsf{T}}W_Hx ∥ x ∥ W H 2 = x T W H x is a squared seminorm rather than necessarily a squared norm.
Theorem 46. (Exact sharing frontier in terminal geometry.) Under Equation (228) , for every fixed L L L ,
R H ( L ) : = E ∥ e H ( L ) ∥ 2 2 = ∥ ( L − I ) θ ∥ W H 2 + tr ( W H L Σ L T ) . (229) R_H(L):=\mathbb{E}\|e_H(L)\|_2^2=\|(L-I)\theta\|_{W_H}^2+\operatorname{tr}(W_HL\Sigma L^{\mathsf{T}}). \tag*{(229)} R H ( L ) := E ∥ e H ( L ) ∥ 2 2 = ∥ ( L − I ) θ ∥ W H 2 + tr ( W H L Σ L T ) . (229) Fix a pooling matrix P P P independently of ζ \zeta ζ , put Q = I − P Q=I-P Q = I − P , and let L λ = I − λ Q L_\lambda=I-\lambda Q L λ = I − λ Q for 0 ≤ λ ≤ 1 0\leq\lambda\leq1 0 ≤ λ ≤ 1 . Neither Euclidean orthogonality nor orthogonality in W H W_H W H is presumed. Set
a H = tr ( W H Q Σ ) , b H = ∥ Q θ ∥ W H 2 + tr ( W H Q Σ Q T ) ≥ 0. (230) \begin{aligned}
a_H&=\operatorname{tr}(W_HQ\Sigma),\\
b_H&=\|Q\theta\|_{W_H}^2+\operatorname{tr}(W_HQ\Sigma Q^{\mathsf{T}})\geq0. \tag*{(230)}
\end{aligned} a H b H = tr ( W H Q Σ ) , = ∥ Qθ ∥ W H 2 + tr ( W H Q Σ Q T ) ≥ 0. (230) Then
R H ( L λ ) = R H ( I ) − 2 a H λ + b H λ 2 . (231) R_H(L_\lambda)=R_H(I)-2a_H\lambda+b_H\lambda^2. \tag*{(231)} R H ( L λ ) = R H ( I ) − 2 a H λ + b H λ 2 . (231) Full pooling strictly improves terminal risk if and only if 2 a H > b H 2a_H>b_H 2 a H > b H . If b H > 0 b_H>0 b H > 0 , an optimal coefficient is
λ H ⋆ = min { 1 , max { 0 , a H / b H } } . (232) \lambda_H^\star=\min\{1,\max\{0,a_H/b_H\}\}. \tag*{(232)} λ H ⋆ = min { 1 , max { 0 , a H / b H }} . (232) If b H = 0 b_H=0 b H = 0 , necessarily a H = 0 a_H=0 a H = 0 and all coefficients have the same risk.
Proof. Expand L Y − θ = ( L − I ) θ + L ζ LY-\theta=(L-I)\theta+L\zeta L Y − θ = ( L − I ) θ + L ζ . The centered cross term vanishes and E ∥ G L ζ ∥ 2 = tr ( W H L Σ L T ) \mathbb{E}\|GL\zeta\|^2=\operatorname{tr}(W_HL\Sigma L^{\mathsf{T}}) E ∥ G L ζ ∥ 2 = tr ( W H L Σ L T ) , proving Equation (229) even when W H W_H W H is singular. Substitution of L λ L_\lambda L λ gives a deterministic bias − λ G Q θ -\lambda GQ\theta − λ GQθ and variance
E ∥ G ζ − λ G Q ζ ∥ 2 = E ∥ G ζ ∥ 2 − 2 λ E ⟨ G ζ , G Q ζ ⟩ + λ 2 E ∥ G Q ζ ∥ 2 . \mathbb{E}\|G\zeta-\lambda GQ\zeta\|^2=\mathbb{E}\|G\zeta\|^2-2\lambda\mathbb{E}\langle G\zeta,GQ\zeta\rangle+\lambda^2\mathbb{E}\|GQ\zeta\|^2. E ∥ Gζ − λ GQζ ∥ 2 = E ∥ Gζ ∥ 2 − 2 λ E ⟨ Gζ , GQζ ⟩ + λ 2 E ∥ GQζ ∥ 2 .
The middle expectation equals a H a_H a H by the trace identity and symmetry of W H , Σ W_H,\Sigma W H , Σ . This proves the quadratic and the strict full-pooling comparison. Minimize the convex quadratic on [ 0 , 1 ] [0,1] [ 0 , 1 ] when b H > 0 b_H>0 b H > 0 . When b H = 0 b_H=0 b H = 0 , both G Q θ = 0 GQ\theta=0 GQθ = 0 and G Q ζ = 0 GQ\zeta=0 GQζ = 0 almost surely; hence the middle expectation is zero as well. ▫
Why the linear term need not be a nonnegative variance saving. If P P P is a projection and P T W H Q = 0 P^{\mathsf{T}}W_HQ=0 P T W H Q = 0 , then a H = tr ( W H Q Σ ) = tr ( W H Q Σ Q T ) ≥ 0 a_H=\operatorname{tr}(W_HQ\Sigma)=\operatorname{tr}(W_HQ\Sigma Q^{\mathsf{T}})\geq0 a H = tr ( W H Q Σ ) = tr ( W H Q Σ Q T ) ≥ 0 : its two projected output components are pairwise orthogonal. General pooling need not satisfy this condition. For example, take θ = 0 \theta=0 θ = 0 , P = diag ( 1 , 0 ) P=\operatorname{diag}(1,0) P = diag ( 1 , 0 ) , G = ( 1 , 1 ) G=(1,1) G = ( 1 , 1 ) , and
Σ = ( 4 − 3 / 2 − 3 / 2 1 ) ≻ 0. \Sigma=
\begin{pmatrix}
4 & -3/2\\
-3/2 & 1
\end{pmatrix}
\succ0. Σ = ( 4 − 3/2 − 3/2 1 ) ≻ 0.
Then a H = − 1 / 2 a_H=-1/2 a H = − 1/2 , b H = 1 b_H=1 b H = 1 , and R H ( L λ ) = 2 + λ + λ 2 R_H(L_\lambda)=2+\lambda+\lambda^2 R H ( L λ ) = 2 + λ + λ 2 . Suppressing a component removes beneficial cancellation and hurts every positive sharing strength.
Proposition 47 (Local pooling improvement can reverse at the terminal state). There is a three-role linear model in which full pooling reduces average local mean squared error from 1 to 5/6 but increases terminal mean squared error from 6 to 9. The reversal persists if local evaluation and pooling use weights proportional to the magnitudes of the terminal propagation coefficients.
Proof. Let Y a = θ a + ε a Y_a = \theta_a + \varepsilon_a Y a = θ a + ε a with independent centered errors of variance one and θ = ( 1 / 2 , − 1 , 1 / 2 ) T \theta= (1/2,-1,1/2)^{\mathsf{T}} θ = ( 1/2 , − 1 , 1/2 ) T . Local separate risk is one. For uniform pooling, θ ˉ = 0 \bar{\theta} = 0 θ ˉ = 0 , the average squared role discrepancy is 1 / 2 1/2 1/2 , and the pooled variance is 1 / 3 1/3 1/3 , giving 5 / 6 5/6 5/6 .
Take the reference events, from the common initial state S 0 = 0 S_0 = 0 S 0 = 0 , to be
S 1 = S 0 + θ 1 , S 2 = − 1 2 S 1 + θ 2 , S 3 = − 2 S 2 + θ 3 . (233) S_1 = S_0 + \theta_1,\qquad S_2 = -\frac{1}{2}S_1 + \theta_2,\qquad S_3 = -2S_2 + \theta_3. \tag*{(233)} S 1 = S 0 + θ 1 , S 2 = − 2 1 S 1 + θ 2 , S 3 = − 2 S 2 + θ 3 . (233) Each learned model uses the same events with its fitted coefficients. Thus G = ( 1 , − 2 , 1 ) G = (1,-2,1) G = ( 1 , − 2 , 1 ) , separate terminal risk is ∥ G ∥ 2 2 = 6 \lVert G\rVert_2^2 = 6 ∥ G ∥ 2 2 = 6 , and G 1 = 0 G1 = 0 G 1 = 0 , G θ = 3 G\theta= 3 Gθ = 3 . Full pooling uses Y ˉ 1 \bar{Y}1 Y ˉ 1 and has the constant error G ( Y ˉ 1 − θ ) = − 3 G(\bar{Y}1-\theta) = -3 G ( Y ˉ 1 − θ ) = − 3 , hence risk nine. For partial uniform pooling,
R H ( λ ) = 6 ( 1 − λ ) 2 + 9 λ 2 = 6 − 12 λ + 15 λ 2 , (234) R_H(\lambda) = 6(1-\lambda)^2 + 9\lambda^2 = 6 - 12\lambda+ 15\lambda^2, \tag*{(234)} R H ( λ ) = 6 ( 1 − λ ) 2 + 9 λ 2 = 6 − 12 λ + 15 λ 2 , (234) which is minimized at λ = 2 / 5 \lambda= 2/5 λ = 2/5 with risk 18 / 5 18/5 18/5 .
For the alternative weights π = ( 1 / 4 , 1 / 2 , 1 / 4 ) \pi= (1/4,1/2,1/4) π = ( 1/4 , 1/2 , 1/4 ) , use the weighted pool ∑ a π a Y a \sum_a \pi_aY_a ∑ a π a Y a . Its weighted target mean is − 1 / 4 -1/4 − 1/4 , weighted squared bias is 9 / 16 9/16 9/16 , and variance is ∑ a π a 2 = 3 / 8 \sum_a \pi_a^2 = 3/8 ∑ a π a 2 = 3/8 . Weighted local risk is therefore 15 / 16 < 1 15/16 < 1 15/16 < 1 . Any common scalar pool is still annihilated by G 1 = 0 G1 = 0 G 1 = 0 , so its terminal risk remains nine.
Here the terminal-optimal partial rule substantially outperforms both endpoints.
Selection of finitely many groups, subspaces, or sharing strengths
Let A \mathcal{A} A be a finite set of K ≥ 1 K \ge1 K ≥ 1 fitted candidates, each fixed conditional on a training/pilot sigma-field T \mathcal{T} T . Candidates can have different partitions, subspaces, or sharing strengths. An independent validation sample has records indexed by i = 1 , … , N i = 1,\ldots,N i = 1 , … , N , independent conditional on T \mathcal{T} T . Losses of different candidates on the same record may be dependent. Fix nonnegative weights w i w_i w i summing to one, measurable with respect to T \mathcal{T} T and thus fixed before validation, and define
R ^ m = ∑ i w i ℓ i , m , R m = ∑ i w i E [ ℓ i , m ∣ T ] , m ∈ A . \widehat{R}_m = \sum_i w_i\ell_{i,m},\qquad R_m = \sum_i w_i\mathbb{E}[\ell_{i,m}\mid\mathcal{T}],\qquad m\in\mathcal{A}. R m = i ∑ w i ℓ i , m , R m = i ∑ w i E [ ℓ i , m ∣ T ] , m ∈ A .
The risk is for this declared validation target; nonidentically distributed records do not define an unspecified IID population. The candidates must not be created adaptively from these same validation outcomes.
Theorem 48 (Independent finite-family selection with explicit loss tails). Assume the centered loss X i , m = ℓ i , m − E [ ℓ i , m ∣ T ] X_{i,m}=\ell_{i,m}-\mathbb{E}[\ell_{i,m}\mid\mathcal{T}] X i , m = ℓ i , m − E [ ℓ i , m ∣ T ] obeys, conditional on T \mathcal{T} T ,
log E [ e t X i , m ∣ T ] ≤ v i , m t 2 2 ( 1 − b i , m ∣ t ∣ ) for ∣ t ∣ < 1 / b i , m , (235) \log\mathbb{E}[e^{tX_{i,m}}\mid\mathcal{T}] \le\frac{v_{i,m}t^2}{2(1-b_{i,m}|t|)}\qquad\text{for }|t|<1/b_{i,m}, \tag*{(235)} log E [ e t X i , m ∣ T ] ≤ 2 ( 1 − b i , m ∣ t ∣ ) v i , m t 2 for ∣ t ∣ < 1/ b i , m , (235) where v i , m , b i , m ≥ 0 v_{i,m},b_{i,m}\ge0 v i , m , b i , m ≥ 0 are fixed conditional on T \mathcal{T} T ; when b i , m = 0 b_{i,m}=0 b i , m = 0 the domain is all real t t t . For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) set
x = log ( 2 K / δ ) , V m = ∑ i w i 2 v i , m , B m = max i w i b i , m , (236) x=\log(2K/\delta),\qquad V_m=\sum_i w_i^2v_{i,m},\qquad B_m=\max_i w_ib_{i,m}, \tag*{(236)} x = log ( 2 K / δ ) , V m = i ∑ w i 2 v i , m , B m = i max w i b i , m , (236) r m = 2 V m x + B m x . r_m=\sqrt{2V_mx}+B_mx. r m = 2 V m x + B m x . With conditional probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every candidate, ∣ R ^ m − R m ∣ ≤ r m |\widehat{R}_m-R_m|\le r_m ∣ R m − R m ∣ ≤ r m . On this event the penalized selector m ^ ∈ arg min m ( R ^ m + r m ) \widehat{m}\in\arg\min_m(\widehat{R}_m+r_m) m ∈ arg min m ( R m + r m ) satisfies
R m ^ ≤ min m ∈ A ( R m + 2 r m ) . (237) R_{\widehat{m}}\le\min_{m\in\mathcal{A}}(R_m+2r_m). \tag*{(237)} R m ≤ m ∈ A min ( R m + 2 r m ) . (237) An unpenalized empirical minimizer instead satisfies R m ^ ≤ min m R m + 2 max m r m R_{\widehat{m}}\le\min_m R_m+2\max_m r_m R m ≤ min m R m + 2 max m r m .
Proof. Conditional independence and Equation (235) imply that the log moment generating function of ∑ i w i X i , m \sum_i w_iX_{i,m} ∑ i w i X i , m is at most V m t 2 / [ 2 ( 1 − B m ∣ t ∣ ) ] V_m t^2/[2(1-B_m|t|)] V m t 2 / [ 2 ( 1 − B m ∣ t ∣ )] . For V m > 0 V_m>0 V m > 0 and the upper tail, use Chernoff’s inequality at t = 2 x / V m / ( 1 + B m 2 x / V m ) t=\sqrt{2x/V_m}/(1+B_m\sqrt{2x/V_m}) t = 2 x / V m / ( 1 + B m 2 x / V m ) to get probability at most e − x e^{-x} e − x above 2 V m x + B m x \sqrt{2V_mx}+B_mx 2 V m x + B m x . Apply the same argument to the negative sum. If V m = 0 V_m=0 V m = 0 , all positive-weight centered losses have zero variance and the claim is immediate. A union bound over both tails of the K K K candidates gives the simultaneous event. On that event, for every m m m ,
R m ≤ R ^ m + r m ≤ R ^ m ^ + r m ^ ≤ R m ^ + 2 r m . R_m \le\widehat{R}_m+r_m \le\widehat{R}_{\widehat{m}}+r_{\widehat{m}} \le R_{\widehat{m}}+2r_m. R m ≤ R m + r m ≤ R m + r m ≤ R m + 2 r m .
For the raw minimizer the same argument uses r m ^ + r m ≤ 2 max m r m r_{\widehat{m}}+r_m\le2\max_m r_m r m + r m ≤ 2 max m r m .
The tail hypothesis is often called a two-sided sub-gamma condition: v v v controls the quadratic fluctuation scale and b b b the range over which the exponential moment is controlled. This tail condition is an additional assumption beyond covariance information. For bounded losses in [ 0 , 1 ] [0,1] [ 0 , 1 ] , Hoeffding’s lemma gives v i , m = 1 / 4 v_{i,m}=1/4 v i , m = 1/4 , b i , m = 0 b_{i,m}=0 b i , m = 0 , recovering Proposition 11. To implement the penalized selector, valid radii or deterministic upper bounds on them must be available; replacing unknown parameters by estimates from the same validation outcomes needs a new guarantee.
A sufficient condition for unbounded quadratic validation loss. Conditional on training and fixed validation inputs (include those inputs in the conditioning information), suppose ℓ i , m = ∥ μ i , m + ϵ i ∥ 2 2 \ell_{i,m}=\lVert\mu_{i,m}+\epsilon_i\rVert_2^2 ℓ i , m = ∥ μ i , m + ϵ i ∥ 2 2 , where μ i , m \mu_{i,m} μ i , m is fixed and the validation errors are independent ϵ i ∼ N ( 0 , Ω i ) \epsilon_i\sim\mathcal{N}(0,\Omega_i) ϵ i ∼ N ( 0 , Ω i ) . Errors may be shared across candidates at the same record. Then Equation (235) holds with
v i , m = 2 tr ( Ω i 2 ) + 4 μ i , m T Ω i μ i , m , b i , m = 2 ∥ Ω i ∥ o p . (238) v_{i,m}=2\operatorname{tr}(\Omega_i^2)+4\mu_{i,m}^{\mathsf{T}}\Omega_i\mu_{i,m}, \qquad b_{i,m}=2\lVert\Omega_i\rVert_{\mathrm{op}}. \tag*{(238)} v i , m = 2 tr ( Ω i 2 ) + 4 μ i , m T Ω i μ i , m , b i , m = 2 ∥ Ω i ∥ op . (238)
Here ∥ Ω i ∥ o p \lVert\Omega_i\rVert_{\mathrm{op}} ∥ Ω i ∥ op is its largest eigenvalue. To verify this, diagonalize Ω i \Omega_i Ω i with eigenvalues ω j ≥ 0 \omega_j\ge0 ω j ≥ 0 and corresponding coordinates μ j \mu_j μ j . The centered quadratic log moment generating function is
∑ j [ − 1 2 log ( 1 − 2 t ω j ) − t ω j + 2 t 2 ω j μ j 2 1 − 2 t ω j ] . \sum_j\left[-\frac{1}{2}\log(1-2t\omega_j)-t\omega_j+\frac{2t^2\omega_j\mu_j^2}{1-2t\omega_j}\right]. j ∑ [ − 2 1 log ( 1 − 2 t ω j ) − t ω j + 1 − 2 t ω j 2 t 2 ω j μ j 2 ] .
For ∣ 2 t ω j ∣ < 1 |2t\omega_j|<1 ∣2 t ω j ∣ < 1 , the power series for the logarithm bounds the first two terms by t 2 ω j 2 / ( 1 − 2 ∣ t ∣ ∥ Ω i ∥ o p ) t^2\omega_j^2/(1-2|t|\lVert\Omega_i\rVert_{\mathrm{op}}) t 2 ω j 2 / ( 1 − 2∣ t ∣ ∥ Ω i ∥ op ) ; the last term is bounded with the same denominator. Summing gives Equation (238) . Null eigenvalues cause no difficulty because their centered contributions are zero. Known Ω i \Omega_i Ω i and a bound ∥ μ i , m ∥ ≤ M i , m \lVert\mu_{i,m}\rVert\le M_{i,m} ∥ μ i , m ∥ ≤ M i , m permit the conservative substitution μ i , m T Ω i μ i , m ≤ ∥ Ω i ∥ o p M i , m 2 \mu_{i,m}^{\mathsf{T}}\Omega_i\mu_{i,m}\le\lVert\Omega_i\rVert_{\mathrm{op}}M_{i,m}^2 μ i , m T Ω i μ i , m ≤ ∥ Ω i ∥ op M i , m 2 . Without such information these are theoretical, not automatically computable, radii.
Independent selection and refitting give a different exact identity. If a pilot selects a rule L = L ( T ) L=L(\mathcal{T}) L = L ( T ) and a fresh fitting error satisfies E [ ζ ∣ T ] = 0 \mathbb{E}[\zeta\mid\mathcal{T}]=0 E [ ζ ∣ T ] = 0 , Cov ( ζ ∣ T ) = Σ \operatorname{Cov}(\zeta\mid\mathcal{T})=\Sigma Cov ( ζ ∣ T ) = Σ , then conditioning and averaging Equation (206) yields the average of the fixed-rule risks. The averaging uses fresh fitting errors after pilot selection. Independence of the final audit also remains necessary for Proposition 45.
Terminal risk with controlled nonlinear remainders
The following result converts a controlled first-order remainder into an interval for actual terminal risk.
Proposition 49 (Terminal risk intervals with a controlled remainder). Let the actual terminal error be e = e l i n + r e=e_{\mathrm{lin}}+r e = e lin + r under the same probability law, with square-integrable vectors, R l i n = E ∥ e l i n ∥ 2 2 R_{\mathrm{lin}}=\mathbb{E}\lVert e_{\mathrm{lin}}\rVert_2^2 R lin = E ∥ e lin ∥ 2 2 , and E ∥ r ∥ 2 2 ≤ ε 2 \mathbb{E}\lVert r\rVert_2^2\le\varepsilon^2 E ∥ r ∥ 2 2 ≤ ε 2 . No independence is required. Then
( max { 0 , R l i n − ε } ) 2 ≤ E ∥ e ∥ 2 2 ≤ ( R l i n + ε ) 2 . (239) \left(\max\{0,\sqrt{R_{\mathrm{lin}}}-\varepsilon\}\right)^2\le\mathbb{E}\lVert e\rVert_2^2\le\left(\sqrt{R_{\mathrm{lin}}}+\varepsilon\right)^2. \tag*{(239)} ( max { 0 , R lin − ε } ) 2 ≤ E ∥ e ∥ 2 2 ≤ ( R lin + ε ) 2 . (239) In particular, a reuse model has strictly smaller actual risk than a separate model if the reuse upper endpoint is smaller than the separate lower endpoint, using valid remainder bounds for both.
Proof. Use the triangle and reverse-triangle inequalities for the L 2 L^2 L 2 norm of random vectors: ∥ e ∥ L 2 ≤ ∥ e l i n ∥ L 2 + ∥ r ∥ L 2 \lVert e\rVert_{L^2}\le\lVert e_{\mathrm{lin}}\rVert_{L^2}+\lVert r\rVert_{L^2} ∥ e ∥ L 2 ≤ ∥ e lin ∥ L 2 + ∥ r ∥ L 2 and ∥ e ∥ L 2 ≥ ∥ e l i n ∥ L 2 − ∥ r ∥ L 2 \lVert e\rVert_{L^2}\ge\lVert e_{\mathrm{lin}}\rVert_{L^2}-\lVert r\rVert_{L^2} ∥ e ∥ L 2 ≥ ∥ e lin ∥ L 2 − ∥ r ∥ L 2 . Nonnegativity and squaring give the endpoints; strict endpoint separation gives the comparison.
Two diagnostics against automatic extrapolation. For the scalar nonlinear map e ( δ ) = δ 2 e(\delta)=\delta^2 e ( δ ) = δ 2 at δ = 0 \delta=0 δ = 0 , the linearization is zero, but δ ∼ N ( 0 , v ) \delta\sim\mathcal{N}(0,v) δ ∼ N ( 0 , v ) gives true squared error E δ 4 = 3 v 2 \mathbb{E}\delta^4=3v^2 E δ 4 = 3 v 2 . Mere knowledge of the covariance does not bound a general fourth-order remainder. In an endogenous linear-looking map, take δ ∼ N ( 0 , 1 ) \delta\sim\mathcal{N}(0,1) δ ∼ N ( 0 , 1 ) and G ( δ ) = 1 { ∣ δ ∣ > 1 } G(\delta)=\mathbf{1}_{\{|\delta|>1\}} G ( δ ) = 1 { ∣ δ ∣ > 1 } . If ϕ , Φ \phi,\Phi ϕ , Φ denote the standard normal density and CDF, respectively,
E [ ( G ( δ ) δ ) 2 ] = 2 [ ϕ ( 1 ) + 1 − Φ ( 1 ) ] > 2 [ 1 − Φ ( 1 ) ] = E [ G ( δ ) 2 ] E δ 2 . \mathbb{E}\left[(G(\delta)\delta)^2\right]=2[\phi(1)+1-\Phi(1)]>2[1-\Phi(1)]=\mathbb{E}[G(\delta)^2]\mathbb{E}\delta^2. E [ ( G ( δ ) δ ) 2 ] = 2 [ ϕ ( 1 ) + 1 − Φ ( 1 )] > 2 [ 1 − Φ ( 1 )] = E [ G ( δ ) 2 ] E δ 2 .
The inequality follows by integration by parts over ∣ δ ∣ > 1 |\delta|>1 ∣ δ ∣ > 1 . Averaging a fixed-G G G formula while retaining unconditional noise covariance is therefore invalid when the route selected G G G from the same error. A valid analysis must use the resulting conditional means and covariances or a pathwise remainder bound. For nonlinear neural traces, specifying the expansion point, reached region, derivatives, noise moments, and exit contribution is still a separate modeling and estimation task.
Section O now supplies explicit joint-derivative remainder envelopes and conditional route moments for specified models.
A corollary: when every local full-pooling gain survives
The reversal in Proposition 47 raises a uniform question: which fixed propagation geometries preserve a strict local full-pooling improvement for every possible target? The following is a structural corollary of Theorem 46, proved by an elementary spectral argument. The role of loss geometry in shrinkage is established in, for example, Hansen [11 ] . Matrix-risk ordering, which controls all linear combinations, is also distinct from merely lowering its trace (Matsuda & Strawderman, [26 ] ).
Let Y = θ + ζ ∈ R p Y=\theta+\zeta\in\mathbb{R}^p Y = θ + ζ ∈ R p , with E ζ = 0 \mathbb{E}\zeta=0 E ζ = 0 and Cov ( ζ ) = q I p \operatorname{Cov}(\zeta)=qI_p Cov ( ζ ) = q I p , q > 0 q>0 q > 0 . Let P = P T = P 2 P=P^{\mathsf{T}}=P^2 P = P T = P 2 be a fixed Euclidean orthogonal pooling projection, Q = I p − P Q=I_p-P Q = I p − P , and k = rank ( Q ) > 0 k=\operatorname{rank}(Q)>0 k = rank ( Q ) > 0 . The separate and fully pooled estimates are Y Y Y and P Y PY P Y . Let the same fixed matrix G ∈ R s × p G\in\mathbb{R}^{s\times p} G ∈ R s × p propagate either estimate’s error, as in Equation (228) . Define the pooled-minus-separate risk differences
Δ l o c ( θ ) = E ∥ P Y − θ ∥ 2 − E ∥ Y − θ ∥ 2 , \Delta_{\mathrm{loc}}(\theta)=\mathbb{E}\lVert PY-\theta\rVert^2-\mathbb{E}\lVert Y-\theta\rVert^2, Δ loc ( θ ) = E ∥ P Y − θ ∥ 2 − E ∥ Y − θ ∥ 2 ,
Δ H ( θ ) = E ∥ G ( P Y − θ ) ∥ 2 − E ∥ G ( Y − θ ) ∥ 2 , A = Q G T G Q . \Delta_H(\theta)=\mathbb{E}\lVert G(PY-\theta)\rVert^2-\mathbb{E}\lVert G(Y-\theta)\rVert^2,\qquad A=QG^{\mathsf{T}}GQ. Δ H ( θ ) = E ∥ G ( P Y − θ ) ∥ 2 − E ∥ G ( Y − θ ) ∥ 2 , A = Q G T GQ .
These are unnormalized risks; dividing local risk by a positive role count leaves its sign unchanged. The matrix A A A is positive semidefinite, vanishes on range ( P ) \operatorname{range}(P) range ( P ) , and describes amplification restricted to the contrast space range ( Q ) \operatorname{range}(Q) range ( Q ) . Only the stated second moments, not Gaussianity, are required.
Corollary of Theorem 46. For these fixed P P P , G G G , q q q ,
[ ∀ θ ∈ R p : Δ l o c ( θ ) < 0 ⟹ Δ H ( θ ) < 0 ] ⟺ A = c Q for some c > 0. \left[\forall\theta\in\mathbb{R}^p:\ \Delta_{\mathrm{loc}}(\theta)<0\Longrightarrow\Delta_H(\theta)<0\right]\Longleftrightarrow A=cQ\quad\text{for some }c>0. [ ∀ θ ∈ R p : Δ loc ( θ ) < 0 ⟹ Δ H ( θ ) < 0 ] ⟺ A = c Q for some c > 0.
Equivalently, G G G must be a positive scaled isometry on the discarded contrast space: ∥ G v ∥ 2 = c ∥ v ∥ 2 \lVert Gv\rVert^2=c\lVert v\rVert^2 ∥ G v ∥ 2 = c ∥ v ∥ 2 for every v ∈ range ( Q ) v\in\operatorname{range}(Q) v ∈ range ( Q ) . This defines the term scaled isometry ; it imposes no such requirement on all of R p \mathbb{R}^p R p .
Proof. Since P 2 = P = P T P^2=P=P^{\mathsf{T}} P 2 = P = P T and the noise is isotropic, the fixed-rule risk identities give
Δ l o c ( θ ) = ∥ Q θ ∥ 2 − q k , Δ H ( θ ) = θ T A θ − q tr ( A ) . \Delta_{\mathrm{loc}}(\theta)=\lVert Q\theta\rVert^2-qk,\qquad\Delta_H(\theta)=\theta^{\mathsf{T}}A\theta-q\operatorname{tr}(A). Δ loc ( θ ) = ∥ Qθ ∥ 2 − q k , Δ H ( θ ) = θ T A θ − q tr ( A ) .
For the second equality, tr ( G T G P ) − tr ( G T G ) = − tr ( G T G Q ) = − tr ( A ) \operatorname{tr}(G^{\mathsf{T}}GP)-\operatorname{tr}(G^{\mathsf{T}}G)=-\operatorname{tr}(G^{\mathsf{T}}GQ)=-\operatorname{tr}(A) tr ( G T GP ) − tr ( G T G ) = − tr ( G T GQ ) = − tr ( A ) . Thus no additional assumption P T G T G Q = 0 P^{\mathsf{T}}G^{\mathsf{T}}GQ=0 P T G T GQ = 0 is needed. If A = c Q A=cQ A = c Q with c > 0 c>0 c > 0 , then Δ H = c Δ l o c \Delta_H=c\Delta_{\mathrm{loc}} Δ H = c Δ loc for every target, proving sufficiency.
For necessity, if A = 0 A=0 A = 0 , the target θ = 0 \theta=0 θ = 0 has Δ l o c = − q k < 0 \Delta_{\mathrm{loc}}=-qk<0 Δ loc = − q k < 0 but Δ H = 0 \Delta_H=0 Δ H = 0 , contradicting the strict implication. Otherwise let λ max > 0 \lambda_{\max}>0 λ m a x > 0 be the largest eigenvalue of A A A restricted to its k k k -dimensional contrast space. If this restriction is not a scalar multiple of the identity, its largest eigenvalue exceeds the average:
λ max > tr ( A ) k . \lambda_{\max}>\frac{\operatorname{tr}(A)}{k}. λ m a x > k tr ( A ) .
Choose a unit eigenvector v ∈ range ( Q ) v\in\operatorname{range}(Q) v ∈ range ( Q ) for λ max \lambda_{\max} λ m a x and a scalar t t t satisfying
q tr ( A ) λ max < t < q k . \frac{q\operatorname{tr}(A)}{\lambda_{\max}}<t<qk. λ m a x q tr ( A ) < t < q k .
The interval is nonempty. At θ = t v \theta= \sqrt{t}v θ = t v , the local difference is t − q k < 0 t-qk<0 t − q k < 0 , whereas the terminal difference is t λ max − q tr ( A ) > 0 t\lambda_{\max}-q\operatorname{tr}(A)>0 t λ m a x − q tr ( A ) > 0 . This strict reversal contradicts the uniform implication. Hence the restriction must be c I k cI_k c I k with c > 0 c>0 c > 0 , equivalent to A = c Q A=cQ A = c Q on the full space.
Consequences of the all-target condition. The corollary generalizes the isolated reversal to every nonscalar contrast geometry and identifies exactly when no target can reverse a local gain. At k = 1 k=1 k = 1 , any nonzero restriction is automatically scalar. The cases k = 0 k=0 k = 0 and q = 0 q=0 q = 0 are excluded because no strict local full-pooling gain is possible; the implication would then be vacuous. The case A = 0 A=0 A = 0 preserves only equality of terminal risks, not strict improvement. A single target may still benefit when A ≠ c Q A\ne cQ A = c Q , so this is not a necessary condition for one particular model to benefit. It compares full projection with separate estimation, not all possible partial-pooling coefficients or same-data adaptive rules. An error-dependent G G G , a different G G G for each comparator, anisotropic noise, or nonlinear/naturally rerouted execution requires its own analysis.
Explicit Nonlinear, Endogenous-Route, and Adapter Specializations
Overview. We make three interfaces explicit: how derivatives control a nonlinear remainder, how route selection changes error moments, and which functional corrections a parameterization can realize. These resolve the conditional modeling questions in Propositions 37 and 49 and Theorem 46 . Every specialization fixes its reference before taking expectations. Calculations use real arithmetic; floating-point error, a different architecture, or a changed input distribution needs its own accounting. Sections O.1–O.2 bound nonlinear remainders and block derivatives; Sections O.3–O.4 analyze route-selected moments; Sections O.5–O.6 distinguish adapter realizability from its initialization-dependent tangent image.
A finite-horizon remainder from joint state–parameter derivatives
Use Euclidean norms on states and parameters and the Frobenius norm when a matrix is viewed as a parameter vector. Let d ∈ R p d\in\mathbb{R}^{p} d ∈ R p denote one parameter perturbation, reused at every occurrence of the relevant parameters; it is not redrawn at successive events. Conditional on a fixed input and reference parameters, define
s j + 1 = F j ( s j , 0 ) , s ^ j + 1 = F j ( s ^ j , d ) , s ^ 0 = s 0 . s_{j+1}=F_j(s_j,0),\qquad\widehat{s}_{j+1}=F_j(\widehat{s}_j,d),\qquad\widehat{s}_0=s_0. s j + 1 = F j ( s j , 0 ) , s j + 1 = F j ( s j , d ) , s 0 = s 0 .
Here F j F_j F j is a twice continuously differentiable event, or a smooth extension of one specified hard-routing branch. A branch extension evaluates the chosen expert and its prescribed smooth gate even at states where it would not win. Actual hard-routed execution equals this extension only on a separately certified route-agreement event.
For M j = D s F j ( s j , 0 ) M_j=D_sF_j(s_j,0) M j = D s F j ( s j , 0 ) and B j = D d F j ( s j , 0 ) B_j=D_dF_j(s_j,0) B j = D d F j ( s j , 0 ) define the linear error
z 0 = 0 , z j + 1 = M j z j + B j d , G 0 = 0 , G j + 1 = M j G j + B j , z j = G j d . (240) z_0=0,\qquad z_{j+1}=M_jz_j+B_jd,\qquad G_0=0,\qquad G_{j+1}=M_jG_j+B_j,\qquad z_j=G_jd. \tag*{(240)} z 0 = 0 , z j + 1 = M j z j + B j d , G 0 = 0 , G j + 1 = M j G j + B j , z j = G j d . (240)
The symbol D 2 F j D^2F_j D 2 F j is a bilinear second derivative, with norm sup ∥ v ∥ , ∥ w ∥ ≤ 1 ∥ D 2 F j [ v , w ] ∥ \sup_{\lVert v\rVert,\lVert w\rVert\le1}\lVert D^2F_j[v,w]\rVert sup ∥ v ∥ , ∥ w ∥ ≤ 1 ∥ D 2 F j [ v , w ]∥ , on the product space of state and parameter directions.
Theorem 50 (Localized finite-horizon Taylor remainder). Fix deterministic constants L j , K j , h j ≥ 0 L_j,K_j,h_j\ge0 L j , K j , h j ≥ 0 that bound, respectively, ∥ D s F j ∥ \lVert D_sF_j\rVert ∥ D s F j ∥ , ∥ D d F j ∥ \lVert D_dF_j\rVert ∥ D d F j ∥ , and ∥ D 2 F j ∥ \lVert D^2F_j\rVert ∥ D 2 F j ∥ on specified regions. Let Ω \Omega Ω be an event on which actual execution uses the specified branches and the segment from ( s j , 0 ) (s_j,0) ( s j , 0 ) to ( s ^ j , d ) (\widehat{s}_j,d) ( s j , d ) lies in these regions for every j j j . Define deterministic sequences
c 0 = b 0 = 0 , c j + 1 = L j c j + K j , b j + 1 = ∥ M j ∥ 2 b j + 1 2 h j ( c j 2 + 1 ) . (241) c_0=b_0=0,\qquad c_{j+1}=L_jc_j+K_j,\qquad b_{j+1}=\lVert M_j\rVert_2b_j+\tfrac{1}{2}h_j(c_j^2+1). \tag*{(241)} c 0 = b 0 = 0 , c j + 1 = L j c j + K j , b j + 1 = ∥ M j ∥ 2 b j + 2 1 h j ( c j 2 + 1 ) . (241) Then, on Ω \Omega Ω , for every event,
∥ s ^ j − s j ∥ 2 ≤ c j ∥ d ∥ 2 , ∥ s ^ j − s j − z j ∥ 2 ≤ b j ∥ d ∥ 2 2 . (242) \lVert\widehat{s}_j-s_j\rVert_2\le c_j\lVert d\rVert_2,\qquad\lVert\widehat{s}_j-s_j-z_j\rVert_2\le b_j\lVert d\rVert_2^2. \tag*{(242)} ∥ s j − s j ∥ 2 ≤ c j ∥ d ∥ 2 , ∥ s j − s j − z j ∥ 2 ≤ b j ∥ d ∥ 2 2 . (242) Write e = s ^ H − s H e=\widehat{s}_H-s_H e = s H − s H and r = e − G H d r=e-G_Hd r = e − G H d under the full probability law. If q > 2 q>2 q > 2 , P ( Ω c ) ≤ p e x i t \mathbb{P}(\Omega^c)\le p_{\mathrm{exit}} P ( Ω c ) ≤ p exit , and ∥ e ∥ L q + ∥ G H d ∥ L q ≤ T q < ∞ \lVert e\rVert_{L_q}+\lVert G_Hd\rVert_{L_q}\le T_q<\infty ∥ e ∥ L q + ∥ G H d ∥ L q ≤ T q < ∞ ,
E ∥ r ∥ 2 2 ≤ b H 2 E ∥ d ∥ 2 4 1 Ω + T q 2 p e x i t 1 − 2 / q . (243) \mathbb{E}\lVert r\rVert_2^2\le b_H^2\mathbb{E}\lVert d\rVert_2^4\mathbf{1}_{\Omega}+T_q^2p_{\mathrm{exit}}^{1-2/q}. \tag*{(243)} E ∥ r ∥ 2 2 ≤ b H 2 E ∥ d ∥ 2 4 1 Ω + T q 2 p exit 1 − 2/ q . (243) Thus the square root of this right-hand side is a valid ε \varepsilon ε in Proposition 49 . If P ( Ω ) = 1 \mathbb{P}(\Omega)=1 P ( Ω ) = 1 , only a finite fourth moment is needed and the exit term is absent.
Proof. On the segment in the statement, the integral first-derivative formula gives ∥ s ^ j + 1 − s j + 1 ∥ ≤ L j ∥ s ^ j − s j ∥ + K j ∥ d ∥ \lVert\widehat{s}_{j+1}-s_{j+1}\rVert\le L_j\lVert\widehat{s}_j-s_j\rVert+K_j\lVert d\rVert ∥ s j + 1 − s j + 1 ∥ ≤ L j ∥ s j − s j ∥ + K j ∥ d ∥ . Induction proves the first bound. The integral second-order Taylor formula at ( s j , 0 ) (s_j,0) ( s j , 0 ) has a remainder t j t_j t j bounded by 1 2 h j ( ∥ s ^ j − s j ∥ 2 + ∥ d ∥ 2 ) \frac{1}{2}h_j(\lVert\widehat{s}_j-s_j\rVert^2+\lVert d\rVert^2) 2 1 h j (∥ s j − s j ∥ 2 + ∥ d ∥ 2 ) . Subtracting Equation (240) gives r j + 1 = M j r j + t j r_{j+1}=M_jr_j+t_j r j + 1 = M j r j + t j . The first bound and induction give the stated b j b_j b j . For the full-law bound, split E ∥ r ∥ 2 \mathbb{E}\lVert r\rVert^2 E ∥ r ∥ 2 over Ω \Omega Ω and its complement. Hölder’s inequality gives E [ ∥ r ∥ 2 1 Ω c ] ≤ ∥ r ∥ L q 2 P ( Ω c ) 1 − 2 / q \mathbb{E}[\lVert r\rVert^2\mathbf{1}_{\Omega^c}]\le\lVert r\rVert_{L^q}^2\mathbb{P}(\Omega^c)^{1-2/q} E [∥ r ∥ 2 1 Ω c ] ≤ ∥ r ∥ L q 2 P ( Ω c ) 1 − 2/ q ; the triangle inequality bounds ∥ r ∥ L q \lVert r\rVert_{L^q} ∥ r ∥ L q by T q T_q T q . No independence between d d d , Ω \Omega Ω , e e e , and r r r is used.
Moments, small-noise scaling, and an obstruction. If the hypotheses hold globally and d ∼ N ( μ , Σ ) d\sim\mathcal{N}(\mu,\Sigma) d ∼ N ( μ , Σ ) , then
E ∥ d ∥ 4 = ( ∥ μ ∥ 2 + tr Σ ) 2 + 2 tr ( Σ 2 ) + 4 μ ⊤ Σ μ . (244) \mathbb{E}\lVert d\rVert^4=(\lVert\mu\rVert^2+\operatorname{tr}\Sigma)^2+2\operatorname{tr}(\Sigma^2)+4\mu^\top\Sigma\mu. \tag*{(244)} E ∥ d ∥ 4 = (∥ μ ∥ 2 + tr Σ ) 2 + 2 tr ( Σ 2 ) + 4 μ ⊤ Σ μ . (244)
To verify it, write d = μ + Z d=\mu+Z d = μ + Z , diagonalize Σ \Sigma Σ , use independent centered normal coordinates with fourth moments 3 σ i 4 3\sigma_i^4 3 σ i 4 , and note that odd terms vanish. In particular, for d = σ Z d=\sigma Z d = σ Z and constants b H b_H b H independent of σ \sigma σ , the remainder norm is O ( σ 2 ) O(\sigma^2) O ( σ 2 ) if E ∥ Z ∥ 4 < ∞ \mathbb{E}\lVert Z\rVert^4<\infty E ∥ Z ∥ 4 < ∞ . For centered Z Z Z , the linear risk is σ 2 tr ( G H Cov ( Z ) G H ⊤ ) \sigma^2\operatorname{tr}(G_H\operatorname{Cov}(Z)G_H^\top) σ 2 tr ( G H Cov ( Z ) G H ⊤ ) ; an o ( 1 ) o(1) o ( 1 ) relative-risk error additionally requires this coefficient to be strictly positive. A zero linear risk provides no relative guarantee. Covariance alone cannot replace the fourth moment: let d N d_N d N equal N \sqrt{N} N , − N -\sqrt{N} − N , 0 0 0 with probabilities 1 / ( 2 N ) 1/(2N) 1/ ( 2 N ) , 1 / ( 2 N ) 1/(2N) 1/ ( 2 N ) , 1 − 1 / N 1-1/N 1 − 1/ N . Then E d N 2 = 1 \mathbb{E}d_N^2=1 E d N 2 = 1 but E d N 4 = N \mathbb{E}d_N^4=N E d N 4 = N . The map F ( d ) = d 2 F(d)=d^2 F ( d ) = d 2 has constant Hessian two and unbounded squared remainder along this family. A small escape probability alone is likewise insufficient for unbounded state loss: an error of size p − 1 p^{-1} p − 1 on an event of probability p p p has risk p − 1 p^{-1} p − 1 . A bounded utility does not bound this state-space remainder.
From one prefix to an input population. The statement can be applied conditional on a random input X X X without assuming that the parameter error is independent of X X X . If its conditional premises hold with measurable constants and a finite right-hand side, then
E ∥ r ∥ 2 ≤ E X [ b H ( X ) 2 E [ ∥ d ∥ 4 1 Ω ∣ X ] + T q ( X ) 2 p e x i t ( X ) 1 − 2 / q ] . (245) \mathbb{E}\lVert r\rVert^2\le\mathbb{E}_X\left[b_H(X)^2\mathbb{E}[\lVert d\rVert^4\mathbf{1}_{\Omega}\mid X]+T_q(X)^2p_{\mathrm{exit}}(X)^{1-2/q}\right]. \tag*{(245)} E ∥ r ∥ 2 ≤ E X [ b H ( X ) 2 E [∥ d ∥ 4 1 Ω ∣ X ] + T q ( X ) 2 p exit ( X ) 1 − 2/ q ] . (245)
This is the tower property applied to Equation (243) . The required integrability can fail if small margins or large derivative constants occur on the input population.
Explicit constants for the normalized gated blocks in the preparation code
We now specify a concrete specialization. Freeze the weights of attention, normalization, router, the up/gate matrices, and any depth correction (not their activations); perturb only expert down matrices. This matches the mathematical form of the corresponding subblock in experiments/transformer-core.py. Joint changes to the router/up/gate/normalizer or finite-precision execution require new terms. For one token let
N ( x ) = γ ⊙ Π x ( ∥ Π x ∥ 2 / d x + ϵ N ) 1 / 2 + β , Π = I − 11 T / d x , ϵ N > 0 , N(x)=\gamma\odot\frac{\Pi x}{(\lVert\Pi x\rVert^2/d_x+\epsilon_N)^{1/2}}+\beta,\qquad\Pi=I-\mathbf{1}\mathbf{1}^{\mathsf{T}}/d_x,\qquad\epsilon_N>0, N ( x ) = γ ⊙ (∥ Π x ∥ 2 / d x + ϵ N ) 1/2 Π x + β , Π = I − 1 1 T / d x , ϵ N > 0 ,
h ( u ) = SiLU ( V u ) ⊙ ( U u ) , g e ( u ) = softmax ( R u ) e , h(u)=\operatorname{SiLU}(Vu)\odot(Uu),\qquad g_e(u)=\operatorname{softmax}(Ru)_e, h ( u ) = SiLU ( V u ) ⊙ ( U u ) , g e ( u ) = softmax ( R u ) e ,
F e ( x , Δ ) = x + ( D e + Δ ) [ g e ( N ( x ) ) h ( N ( x ) ) ] + A N ( x ) . (246) F_e(x,\Delta)=x+(D_e+\Delta)[g_e(N(x))h(N(x))]+AN(x). \tag*{(246)} F e ( x , Δ ) = x + ( D e + Δ ) [ g e ( N ( x )) h ( N ( x ))] + A N ( x ) . (246)
Here d x d_x d x is state width, γ , β \gamma,\beta γ , β are fixed LayerNorm scale and bias, U , V U,V U , V are expert up and gate matrices, D e D_e D e is its down matrix, R R R the router matrix, and A A A an optional fixed depth-correction matrix (zero if absent). The entrywise product is ⊙ \odot ⊙ , and SiLU ( t ) = t / ( 1 + e − t ) \operatorname{SiLU}(t)=t/(1+e^{-t}) SiLU ( t ) = t / ( 1 + e − t ) . The gain is the selected softmax probability, not a unit gate; support agreement does not freeze it. A deliberately frozen gain in [ 0 , 1 ] [0,1] [ 0 , 1 ] is also allowed by setting its derivatives below to zero.
Proposition 51 (Explicit smooth-branch derivative envelopes). Put g = ∥ γ ∥ ∞ g=\lVert\gamma\rVert_\infty g = ∥ γ ∥ ∞ , and define
R N = g d x + ∥ β ∥ 2 , L N = g / ϵ N , H N = 6 g / ( d x ϵ N ) , R_N=g\sqrt{d_x}+\lVert\beta\rVert_2,\qquad L_N=g/\sqrt{\epsilon_N},\qquad H_N=6g/(\sqrt{d_x}\epsilon_N), R N = g d x + ∥ β ∥ 2 , L N = g / ϵ N , H N = 6 g / ( d x ϵ N ) , a = ∥ U ∥ 2 , v = ∥ V ∥ 2 , T = v R N , a=\lVert U\rVert_2,\qquad v=\lVert V\rVert_2,\qquad T=vR_N, a = ∥ U ∥ 2 , v = ∥ V ∥ 2 , T = v R N , s 1 = 1 + T / 4 , s 2 = 1 / 2 + T / 4. (247) s_1=1+T/4,\qquad s_2=1/2+T/4. \tag*{(247)} s 1 = 1 + T /4 , s 2 = 1/2 + T /4. (247) Valid bounds for h h h and its first two derivatives on ∥ u ∥ ≤ R N \lVert u\rVert\le R_N ∥ u ∥ ≤ R N are
H 0 = a v R N 2 , H 1 = a v R N ( s 1 + 1 ) , H 2 = a R N s 2 v 2 + 2 s 1 a v . (248) H_0=avR_N^2,\qquad H_1=avR_N(s_1+1),\qquad H_2=aR_Ns_2v^2+2s_1av. \tag*{(248)} H 0 = a v R N 2 , H 1 = a v R N ( s 1 + 1 ) , H 2 = a R N s 2 v 2 + 2 s 1 a v . (248)
Let r R = ∥ R ∥ 2 r_R = \lVert R\rVert_2 r R = ∥ R ∥ 2 , g 1 = 2 r R L N g_1 = 2r_RL_N g 1 = 2 r R L N , and g 2 = 6 r R 2 L N 2 + 2 r R H N g_2 = 6r_R^2L_N^2 + 2r_RH_N g 2 = 6 r R 2 L N 2 + 2 r R H N . For k e ( x ) = g e ( N ( x ) ) h ( N ( x ) ) k_e(x) = g_e(N(x))h(N(x)) k e ( x ) = g e ( N ( x )) h ( N ( x )) , take
k 0 = H 0 , k 1 = H 1 L N + g 1 H 0 , k 2 = H 2 L N 2 + H 1 H N + 2 g 1 H 1 L N + g 2 H 0 . (249) \begin{aligned}
k_0 &= H_0, \qquad k_1 = H_1L_N + g_1H_0,\\
k_2 &= H_2L_N^2 + H_1H_N + 2g_1H_1L_N + g_2H_0. \tag*{(249)}
\end{aligned} k 0 k 2 = H 0 , k 1 = H 1 L N + g 1 H 0 , = H 2 L N 2 + H 1 H N + 2 g 1 H 1 L N + g 2 H 0 . (249)
Uniformly for all states and ∥ Δ ∥ F ≤ t \lVert\Delta\rVert_F \le t ∥ Δ ∥ F ≤ t , the constants
L = 1 + ( ∥ D e ∥ 2 + t ) k 1 + ∥ A ∥ 2 L N , K = k 0 , L = 1 + (\lVert D_e\rVert_2 + t)k_1 + \lVert A\rVert_2L_N,\qquad K = k_0, L = 1 + (∥ D e ∥ 2 + t ) k 1 + ∥ A ∥ 2 L N , K = k 0 ,
h F = ( ∥ D e ∥ 2 + t ) k 2 + ∥ A ∥ 2 H N + 2 k 1 (250) h_F = (\lVert D_e\rVert_2 + t)k_2 + \lVert A\rVert_2H_N + 2k_1 \tag*{(250)} h F = (∥ D e ∥ 2 + t ) k 2 + ∥ A ∥ 2 H N + 2 k 1 (250)
bound ∥ D x F e ∥ \lVert D_xF_e\rVert ∥ D x F e ∥ , ∥ D Δ F e ∥ \lVert D_\Delta F_e\rVert ∥ D Δ F e ∥ , and its joint second derivative. For n n n tokens with rowwise normalization and a fixed support per row, use Frobenius state norms, maximize these per-expert constants over the selected bank, and replace K K K by n max e k 0 , e \sqrt{n}\max_e k_{0,e} n max e k 0 , e . No extra factor n n n is needed in L L L or h F h_F h F , even when the down-matrix perturbations are shared across tokens, provided their stacked Frobenius norm is at most t t t .
Proof. Write z = Π x z = \Pi x z = Π x and q = ∥ z ∥ 2 / d x + ϵ N q = \lVert z\rVert^2/d_x + \epsilon_N q = ∥ z ∥ 2 / d x + ϵ N . Apart from the fixed diagonal scale, the derivative of z / q z/\sqrt{q} z / q is q − 1 / 2 Π − q − 3 / 2 z z ⊤ / d x q^{-1/2}\Pi- q^{-3/2}zz^\top/d_x q − 1/2 Π − q − 3/2 z z ⊤ / d x , whose norm is at most ϵ N − 1 / 2 \epsilon_N^{-1/2} ϵ N − 1/2 . Its second derivative in directions u , w u,w u , w is
− Π u ⟨ z , Π w ⟩ + Π w ⟨ z , Π u ⟩ + z ⟨ Π u , Π w ⟩ d x q 3 / 2 + 3 z ⟨ z , Π u ⟩ ⟨ z , Π w ⟩ d x 2 q 5 / 2 . -\frac{\Pi u\langle z,\Pi w\rangle+\Pi w\langle z,\Pi u\rangle+z\langle\Pi u,\Pi w\rangle}{d_xq^{3/2}}+\frac{3z\langle z,\Pi u\rangle\langle z,\Pi w\rangle}{d_x^2q^{5/2}}. − d x q 3/2 Π u ⟨ z , Π w ⟩ + Π w ⟨ z , Π u ⟩ + z ⟨ Π u , Π w ⟩ + d x 2 q 5/2 3 z ⟨ z , Π u ⟩ ⟨ z , Π w ⟩ .
Since ∥ z ∥ ≤ d x q \lVert z\rVert\le\sqrt{d_xq} ∥ z ∥ ≤ d x q , its norm for unit directions is at most 6 / ( d x ϵ N ) 6/(\sqrt{d_x}\epsilon_N) 6/ ( d x ϵ N ) . This also proves the stated bound on N N N . If p ( t ) = ( 1 + e − t ) − 1 p(t)=(1+e^{-t})^{-1} p ( t ) = ( 1 + e − t ) − 1 , then SiLU ′ = p + t p ( 1 − p ) \operatorname{SiLU}'=p+tp(1-p) SiLU ′ = p + tp ( 1 − p ) and SiLU ′ ′ = 2 p ( 1 − p ) + t p ( 1 − p ) ( 1 − 2 p ) \operatorname{SiLU}''=2p(1-p)+tp(1-p)(1-2p) SiLU ′′ = 2 p ( 1 − p ) + tp ( 1 − p ) ( 1 − 2 p ) . The inequalities p ( 1 − p ) ≤ 1 / 4 p(1-p)\le1/4 p ( 1 − p ) ≤ 1/4 and ∣ t ∣ ≤ T |t|\le T ∣ t ∣ ≤ T give s 1 , s 2 s_1,s_2 s 1 , s 2 . Twice applying the product rule to h h h gives Equation (248) . For a softmax coordinate, the logit gradient is p e ( e e − p ) p_e(e_e-p) p e ( e e − p ) and its Hessian is p e [ ( e e − p ) ( e e − p ) ⊤ − ( diag ( p ) − p p ⊤ ) ] p_e[(e_e-p)(e_e-p)^\top-(\operatorname{diag}(p)-pp^\top)] p e [( e e − p ) ( e e − p ) ⊤ − ( diag ( p ) − p p ⊤ )] . Their norms are bounded by two and six, respectively. Composition with R N RN R N and the product rule prove Equation (249) . Finally D Δ 2 F e = 0 D_\Delta^2F_e=0 D Δ 2 F e = 0 , each state–parameter cross derivative has norm at most k 1 k_1 k 1 , and the state Hessian is bounded by ( ∥ D e ∥ + t ) k 2 + ∥ A ∥ H N (\lVert D_e\rVert+t)k_2+\lVert A\rVert H_N (∥ D e ∥ + t ) k 2 + ∥ A ∥ H N . Sum these three terms on the joint product space. For rowwise maps, the sum of products of squared row-direction norms is at most the product of their sums. The same inequality applies when a repeated parameter direction is selected from the stacked perturbation. The parameter first derivative has at most n n n rows of norm k 0 ∥ Δ ∥ F k_0\lVert\Delta\rVert_F k 0 ∥ Δ ∥ F , giving only the stated n \sqrt{n} n factor in K K K .
Closing the surrounding fixed-prefix attention computation.
We complete the fixed-prefix Transformer analysis with an explicit, conservative bound for attention in the preparation architecture. Fix prefix length n n n , head width d h d_h d h , all weights, and a fixed causal mask with at least one allowed key per row. Softmax is taken only over allowed logits. With normalized input Z Z Z satisfying ∥ Z ∥ F ≤ Z 0 = n R N \lVert Z\rVert_F\le Z_0=nR_N ∥ Z ∥ F ≤ Z 0 = n R N , one attention head is
A h ( Z ) = P h ( Z ) Z W v , h ⊤ , P h ( Z ) = softmax r o w s ( ( Z W q , h ⊤ ) ( Z W k , h ⊤ ) ⊤ / d h ) . A_h(Z)=P_h(Z)ZW_{v,h}^{\top},\qquad P_h(Z)=\operatorname{softmax}_{\mathrm{rows}}\left((ZW_{q,h}^{\top})(ZW_{k,h}^{\top})^\top/\sqrt{d_h}\right). A h ( Z ) = P h ( Z ) Z W v , h ⊤ , P h ( Z ) = softmax rows ( ( Z W q , h ⊤ ) ( Z W k , h ⊤ ) ⊤ / d h ) .
Set q h = ∥ W q , h ∥ 2 q_h=\lVert W_{q,h}\rVert_2 q h = ∥ W q , h ∥ 2 , k h = ∥ W k , h ∥ 2 k_h=\lVert W_{k,h}\rVert_2 k h = ∥ W k , h ∥ 2 , v h = ∥ W v , h ∥ 2 v_h=\lVert W_{v,h}\rVert_2 v h = ∥ W v , h ∥ 2 , and
s 1 , h = 2 q h k h Z 0 / d h , s 2 , h = 2 q h k h / d h , p 1 , h = 2 s 1 , h , p 2 , h = 6 s 1 , h 2 + 2 s 2 , h , a 1 , h = p 1 , h v h Z 0 + n v h , a 2 , h = p 2 , h v h Z 0 + 2 p 1 , h v h . (251) \begin{aligned}
s_{1,h}&=2q_hk_hZ_0/\sqrt{d_h}, & s_{2,h}&=2q_hk_h/\sqrt{d_h},\\
p_{1,h}&=2s_{1,h}, & p_{2,h}&=6s_{1,h}^2+2s_{2,h},\\
a_{1,h}&=p_{1,h}v_hZ_0+\sqrt{n}v_h, & a_{2,h}&=p_{2,h}v_hZ_0+2p_{1,h}v_h. \tag*{(251)}
\end{aligned} s 1 , h p 1 , h a 1 , h = 2 q h k h Z 0 / d h , = 2 s 1 , h , = p 1 , h v h Z 0 + n v h , s 2 , h p 2 , h a 2 , h = 2 q h k h / d h , = 6 s 1 , h 2 + 2 s 2 , h , = p 2 , h v h Z 0 + 2 p 1 , h v h . (251)
For the concatenated heads followed by output matrix W o W_o W o , take a i = ∥ W o ∥ 2 ( ∑ h a i , h 2 ) 1 / 2 a_i=\lVert W_o\rVert_2(\sum_h a_{i,h}^2)^{1/2} a i = ∥ W o ∥ 2 ( ∑ h a i , h 2 ) 1/2 , i = 1 , 2 i=1,2 i = 1 , 2 . An attention-plus-residual event X ↦ X + A ( N ( X ) ) X\mapsto X+A(N(X)) X ↦ X + A ( N ( X )) therefore has
L a t t = 1 + a 1 L N , K a t t = 0 , h a t t = a 2 L N 2 + a 1 H N . (252) L_{\mathrm{att}}=1+a_1L_N,\qquad K_{\mathrm{att}}=0,\qquad h_{\mathrm{att}}=a_2L_N^2+a_1H_N. \tag*{(252)} L att = 1 + a 1 L N , K att = 0 , h att = a 2 L N 2 + a 1 H N . (252)
For completeness, these bounds follow by differentiating the bilinear score map: its first and second derivatives have norms at most s 1 , h , s 2 , h s_{1,h},s_{2,h} s 1 , h , s 2 , h . The row-softmax first and second derivative norms are at most two and six. For the latter, its bilinear form is p ⋅ [ ( u − E p u ) ( w − E p w ) − Cov p ( u , w ) ] p\mathbin{\cdot}[(u-E_pu)(w-E_pw)-\operatorname{Cov}_p(u,w)] p ⋅ [( u − E p u ) ( w − E p w ) − Cov p ( u , w )] ; its Euclidean norm is at most its ℓ 1 \ell_1 ℓ 1 norm, which is at most 5 ∥ u ∥ 2 ∥ w ∥ 2 ≤ 6 ∥ u ∥ 2 ∥ w ∥ 2 5\lVert u\rVert_2\lVert w\rVert_2\le6\lVert u\rVert_2\lVert w\rVert_2 5 ∥ u ∥ 2 ∥ w ∥ 2 ≤ 6 ∥ u ∥ 2 ∥ w ∥ 2 . Combine the rows by the squared-row inequality used above, and note ∥ P h ∥ 2 ≤ ∥ P h ∥ F ≤ n \lVert P_h\rVert_2\le\lVert P_h\rVert_F\le\sqrt{n} ∥ P h ∥ 2 ≤ ∥ P h ∥ F ≤ n . Two product rules give a 1 , h , a 2 , h a_{1,h},a_{2,h} a 1 , h , a 2 , h ; concatenation and composition with LayerNorm give Equation (252) . The terminal LayerNorm followed by a fixed linear head similarly has L = ∥ W h e a d ∥ 2 L N L=\lVert W_{\mathrm{head}}\rVert_2L_N L = ∥ W head ∥ 2 L N , K = 0 K=0 K = 0 , h = ∥ W h e a d ∥ 2 H N h=\lVert W_{\mathrm{head}}\rVert_2H_N h = ∥ W head ∥ 2 H N . Alternating these events with Equation (250) covers a full fixed-prefix forward pass with the specified down-weight perturbations.
A noncircular route and region check. Choose a perturbation radius t t t first, compute the uniform constants above and c j c_j c j by Equation (241) , and evaluate reference router margins before examining perturbed executions. At each FFN event, let γ j \gamma_j γ j be the minimum reference top-one logit gap over its tokens. The sufficient tests
∥ d ∥ ≤ t , γ j > 2 ∥ R j ∥ 2 L N , j c j t for every FFN event (253) \lVert d\rVert\le t,\qquad\gamma_j > 2\lVert R_j\rVert_2 L_{N,j}c_jt \quad\text{for every FFN event} \tag*{(253)} ∥ d ∥ ≤ t , γ j > 2 ∥ R j ∥ 2 L N , j c j t for every FFN event (253)
ensure route agreement by induction and Lemma 27. The derivative bounds are global in the state, so no additional state-region escape occurs in this particular normalized architecture. The test controls support only; the varying gain has already been included in k 1 , k 2 k_1,k_2 k 1 , k 2 . If the tests hold and d d d is bounded by t t t almost surely, set p e x i t = 0 p_{\mathrm{exit}}=0 p exit = 0 . For an unbounded perturbation law, use at least P ( ∥ d ∥ > t ) \mathbb{P}(\lVert d\rVert>t) P (∥ d ∥ > t ) in the exit accounting rather than discard it. For a fixed input with all strict margins, a positive sufficiently small t t t exists because the finitely many constants and recursions are continuous and finite at t = 0 t=0 t = 0 . The admissible radius can depend on the input. Small ϵ N \epsilon_N ϵ N , large weights, long prefixes, or depth can make these conservative constants uninformative.
An explicit tail envelope for this specialization. There is also a global, route-independent growth bound for the down-only perturbation model at a fixed prefix. A normalized attention residual is uniformly bounded by its fixed weights and Z 0 Z_0 Z 0 ; a routed FFN residual is bounded by
n max e [ ( ∥ D e ∥ 2 + ∥ d ∥ ) H 0 , e + ∥ A ∥ 2 R N ] . \sqrt{n}\max_e\left[(\lVert D_e\rVert_2+\lVert d\rVert)H_{0,e}+\lVert A\rVert_2R_N\right]. n e max [ (∥ D e ∥ 2 + ∥ d ∥) H 0 , e + ∥ A ∥ 2 R N ] .
Summing residual magnitudes through finitely many blocks gives ∥ X ^ − X ∥ F ≤ C 0 + C 1 ∥ d ∥ \lVert\widehat{X}-X\rVert_F\le C_0+C_1\lVert d\rVert ∥ X − X ∥ F ≤ C 0 + C 1 ∥ d ∥ with explicit nonnegative constants obtained from these displayed sums for the perturbed and reference passes. If the endpoint is the final normalized logits, the sharper bound ∥ e ∥ F ≤ 2 ∥ W h e a d ∥ 2 n R N , f i n a l \lVert e\rVert_F\le2\lVert W_{\mathrm{head}}\rVert_2\sqrt{n}R_{N,\mathrm{final}} ∥ e ∥ F ≤ 2 ∥ W head ∥ 2 n R N , final is available. Consequently a finite q q q -th parameter moment supplies a finite T q T_q T q in Equation (243) ; for Gaussian perturbations one may use q = 4 q=4 q = 4 and Equation (244) . The actual weights, margins, covariance, and target prefix distribution are still needed to obtain a numerical certificate for a trained model. Theorem 59 and Proposition 60 distinguish current-gain support replay from the preparation code’s frozen-gain intervention.
Conditional moments under endogenous routing
Let ξ ∈ R p \xi\in\mathbb{R}^p ξ ∈ R p be a square-integrable error vector containing all reused parameter or local-injection errors. Let R \mathcal{R} R be a sigma-field that makes the actual coefficient matrix G G G and offset b b b measurable; both may be selected using ξ \xi ξ . It may have to contain more than the expert indices: within-support gains, states, and Jacobians can still vary at a fixed index. Assume b + G ξ b+G\xi b + G ξ and the conditional-moment expressions below are integrable at the indicated squared orders. Finite route families with bounded G , b G,b G , b and ξ ∈ L 2 \xi\in L^2 ξ ∈ L 2 are sufficient.
Theorem 52 (Exact risk with route-selected conditional moments). For μ R = E [ ξ ∣ R ] \mu_{\mathcal{R}}=\mathbb{E}[\xi\mid\mathcal{R}] μ R = E [ ξ ∣ R ] and Σ R = Cov ( ξ ∣ R ) \Sigma_{\mathcal{R}}=\operatorname{Cov}(\xi\mid\mathcal{R}) Σ R = Cov ( ξ ∣ R ) , an exactly route-affine error e = b + G ξ e=b+G\xi e = b + G ξ obeys
E ∥ e ∥ 2 2 = E [ ∥ b + G μ R ∥ 2 2 + tr ( G Σ R G T ) ] . (254) \mathbb{E}\lVert e\rVert_2^2=\mathbb{E}\left[\lVert b+G\mu_{\mathcal{R}}\rVert_2^2+\operatorname{tr}(G\Sigma_{\mathcal{R}}G^{\mathsf{T}})\right]. \tag*{(254)} E ∥ e ∥ 2 2 = E [ ∥ b + G μ R ∥ 2 2 + tr ( G Σ R G T ) ] . (254)
For a finite route variable T T T , with constant G t , b t G_t,b_t G t , b t on each positive-mass cell, this is the sum of these conditional risks weighted by p t = P ( T = t ) p_t=\mathbb{P}(T=t) p t = P ( T = t ) . Neither the conditional means nor the conditional cross-covariances may be replaced by their unconditional values without an additional argument.
A quadratic sharing frontier remains possible under one fixed endogenous route map. Specifically, let Q Q Q be deterministic, and fix the map T = T ( ξ ) T=T(\xi) T = T ( ξ ) and cell matrices G t G_t G t independently of λ \lambda λ . Suppose all compared errors are exactly e λ = G T [ ξ − λ Q ( θ + ξ ) ] e_\lambda=G_T[\xi-\lambda Q(\theta+\xi)] e λ = G T [ ξ − λ Q ( θ + ξ )] . Then
R ( λ ) = R ( 0 ) − 2 a λ + b 2 λ 2 , a = E ⟨ G T ξ , G T Q ( θ + ξ ) ⟩ , b 2 = E ∥ G T Q ( θ + ξ ) ∥ 2 ≥ 0. (255) R(\lambda)=R(0)-2a\lambda+b_2\lambda^2,\qquad a=\mathbb{E}\langle G_T\xi,G_TQ(\theta+\xi)\rangle,\qquad b_2=\mathbb{E}\lVert G_TQ(\theta+\xi)\rVert^2\ge0. \tag*{(255)} R ( λ ) = R ( 0 ) − 2 aλ + b 2 λ 2 , a = E ⟨ G T ξ , G T Q ( θ + ξ )⟩ , b 2 = E ∥ G T Q ( θ + ξ ) ∥ 2 ≥ 0. (255)
With W t = G t T G t W_t=G_t^{\mathsf{T}}G_t W t = G t T G t , μ t = E [ ξ ∣ T = t ] \mu_t=\mathbb{E}[\xi\mid T=t] μ t = E [ ξ ∣ T = t ] , Σ t = Cov ( ξ ∣ T = t ) \Sigma_t=\operatorname{Cov}(\xi\mid T=t) Σ t = Cov ( ξ ∣ T = t ) , the coefficients equal
a = ∑ t p t { μ t T W t Q ( θ + μ t ) + tr ( W t Q Σ t ) } , a=\sum_t p_t\left\{\mu_t^{\mathsf{T}}W_tQ(\theta+\mu_t)+\operatorname{tr}(W_tQ\Sigma_t)\right\}, a = t ∑ p t { μ t T W t Q ( θ + μ t ) + tr ( W t Q Σ t ) } ,
b 2 = ∑ t p t { ∥ Q ( θ + μ t ) ∥ W t 2 + tr ( W t Q Σ t Q T ) } . (256) b_2=\sum_t p_t\left\{\lVert Q(\theta+\mu_t)\rVert_{W_t}^2+\operatorname{tr}(W_tQ\Sigma_tQ^{\mathsf{T}})\right\}. \tag*{(256)} b 2 = t ∑ p t { ∥ Q ( θ + μ t ) ∥ W t 2 + tr ( W t Q Σ t Q T ) } . (256)
Full sharing improves this risk exactly when 2 a > b 2 2a>b_2 2 a > b 2 ; the minimizer on [ 0 , 1 ] [0,1] [ 0 , 1 ] is min { 1 , max { 0 , a / b 2 } } \min\{1,\max\{0,a/b_2\}\} min { 1 , max { 0 , a / b 2 }} if b 2 > 0 b_2>0 b 2 > 0 . If b 2 = 0 b_2=0 b 2 = 0 , then a = 0 a=0 a = 0 and all coefficients have equal risk.
Proof. Write ξ = μ R + η \xi= \mu_{\mathcal{R}} + \eta ξ = μ R + η with E [ η ∣ R ] = 0 \mathbb{E}[\eta\mid\mathcal{R}] = 0 E [ η ∣ R ] = 0 . Conditional on R \mathcal{R} R , expand the squared norm; the cross term vanishes and the remaining quadratic noise term is tr ( G Σ R G T ) \operatorname{tr}(G\Sigma_{\mathcal{R}}G^{\mathsf{T}}) tr ( G Σ R G T ) . Take expectations. For the sharing statement set X = G T ξ X = G_{\mathsf{T}}\xi X = G T ξ and Y = G T Q ( θ + ξ ) Y = G_{\mathsf{T}}Q(\theta+ \xi) Y = G T Q ( θ + ξ ) . Expand E ∥ X − λ Y ∥ 2 \mathbb{E}\lVert X - \lambda Y\rVert^2 E ∥ X − λY ∥ 2 and then condition each coefficient on T T T ; this gives Equation (256) . Minimize the resulting quadratic. If E ∥ Y ∥ 2 = 0 \mathbb{E}\lVert Y\rVert^2 = 0 E ∥ Y ∥ 2 = 0 , then Y = 0 Y = 0 Y = 0 almost surely and E ⟨ X , Y ⟩ = 0 \mathbb{E}\langle X,Y\rangle= 0 E ⟨ X , Y ⟩ = 0 .
This is a conditional second-moment identity. It also applies to fixed linear propagations within a route cell after stacking all local errors, so it retains cross-time dependence. If the cell map or coefficients change with λ \lambda λ , every candidate must use its own conditional law; Equation (255) is no longer one common quadratic. For a nonlinear route cell the identity applies to its linear part only, with the remainder treated by Theorem 50 and Proposition 49 .
General fixed Gaussian route cells via their probabilities. There is a useful identity beyond one half-space. Let Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 be fixed, let C t C_t C t be a fixed Borel selection region, and write p t ( μ ) = Pr ξ ∼ N ( μ , Σ ) ( ξ ∈ C t ) > 0 p_t(\mu) = \Pr_{\xi\sim\mathcal{N}(\mu,\Sigma)}(\xi\in C_t) > 0 p t ( μ ) = Pr ξ ∼ N ( μ , Σ ) ( ξ ∈ C t ) > 0 . When differentiating, hold the region and covariance fixed. Then
μ t = μ + Σ ∇ μ log p t ( μ ) , Σ t = Σ + Σ ∇ μ 2 log p t ( μ ) Σ . (257) \mu_t = \mu+ \Sigma\nabla_\mu\log p_t(\mu), \qquad\Sigma_t = \Sigma+ \Sigma\nabla_\mu^2 \log p_t(\mu)\Sigma. \tag*{(257)} μ t = μ + Σ ∇ μ log p t ( μ ) , Σ t = Σ + Σ ∇ μ 2 log p t ( μ ) Σ. (257)
To prove this, differentiate the Gaussian density inside its integral on C t C_t C t . The first density derivative is Σ − 1 ( ξ − μ ) \Sigma^{-1}(\xi-\mu) Σ − 1 ( ξ − μ ) times the density, and the second is [ Σ − 1 ( ξ − μ ) ( ξ − μ ) T Σ − 1 − Σ − 1 ] [\Sigma^{-1}(\xi-\mu)(\xi-\mu)^{\mathsf{T}}\Sigma^{-1} - \Sigma^{-1}] [ Σ − 1 ( ξ − μ ) ( ξ − μ ) T Σ − 1 − Σ − 1 ] times that density. Divide the integrated expressions by p t p_t p t and subtract the outer product of the conditional mean displacement. Differentiation is justified by locally integrable polynomial-times-Gaussian envelopes even if the region’s boundary is not smooth. For singular covariance, apply the identity in whitened coordinates on its fixed affine support, varying the mean only within that support; ambient mean derivatives need not exist across moving supports. This handles fixed intersections or nonlinear regions analytically in terms of p t p_t p t . Regions depending on the differentiated mean require boundary terms; regions or propagation coefficients changing across sharing candidates require separate calculations.
A complete two-expert Gaussian threshold calculation
Proposition 53 (Half-space route probabilities, means, and covariances). Let ξ ∼ N ( μ , Σ ) \xi\sim\mathcal{N}(\mu,\Sigma) ξ ∼ N ( μ , Σ ) in R p \mathbb{R}^p R p , allowing singular Σ \Sigma Σ , and select route + + + when a T ξ > τ a^{\mathsf{T}}\xi> \tau a T ξ > τ , route − - − otherwise. Assume first that s 2 = a T Σ a > 0 s^2 = a^{\mathsf{T}}\Sigma a > 0 s 2 = a T Σ a > 0 , and put
κ = τ − a T μ s , h = Σ a s , \kappa= \frac{\tau-a^{\mathsf{T}}\mu}{s}, \qquad h = \frac{\Sigma a}{s}, κ = s τ − a T μ , h = s Σ a , p + = 1 − Φ ( κ ) , p − = Φ ( κ ) , m + = ϕ ( κ ) p + , m − = ϕ ( κ ) p − . p_+ = 1-\Phi(\kappa), \qquad p_- = \Phi(\kappa), \qquad m_+ = \frac{\phi(\kappa)}{p_+}, \qquad m_- = \frac{\phi(\kappa)}{p_-}. p + = 1 − Φ ( κ ) , p − = Φ ( κ ) , m + = p + ϕ ( κ ) , m − = p − ϕ ( κ ) . The functions ϕ , Φ \phi,\Phi ϕ , Φ are the standard normal density and CDF. For finite τ \tau τ , both probabilities are positive and
μ + = μ + h m + , Σ + = Σ + h h T ( κ m + − m + 2 ) , \mu_+ = \mu+ hm_+, \qquad\Sigma_+ = \Sigma+ hh^{\mathsf{T}}(\kappa m_+ - m_+^2), μ + = μ + h m + , Σ + = Σ + h h T ( κ m + − m + 2 ) , μ − = μ − h m − , Σ − = Σ − h h T ( κ m − + m − 2 ) . (258) \mu_- = \mu- hm_-, \qquad\Sigma_- = \Sigma- hh^{\mathsf{T}}(\kappa m_- + m_-^2). \tag*{(258)} μ − = μ − h m − , Σ − = Σ − h h T ( κ m − + m − 2 ) . (258) For any fixed offsets b ± b_\pm b ± and matrices G ± G_\pm G ± , these moments and Equation (254) give the exact risk of e = b T + G T T ξ e = b_T + G_T^{\mathsf{T}}\xi e = b T + G T T ξ . If s = 0 s = 0 s = 0 , the score is constant almost surely: use its single realized branch (with the stated tie convention), and the original μ , Σ \mu,\Sigma μ , Σ on that branch; there is no division by s s s or conditioning on a zero-mass cell.
Proof. Let Z = ( a T ξ − a T μ ) / s Z = (a^{\mathsf{T}}\xi-a^{\mathsf{T}}\mu)/s Z = ( a T ξ − a T μ ) / s and η = ξ − μ − h Z \eta= \xi-\mu-hZ η = ξ − μ − h Z . Then Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) and η \eta η is independent of Z Z Z , including in the singular Gaussian case, because their cross-covariance is zero. Moreover Cov ( η ) = Σ − h h T ⪰ 0 \operatorname{Cov}(\eta) = \Sigma-hh^{\mathsf{T}} \succeq0 Cov ( η ) = Σ − h h T ⪰ 0 . Integration by parts using ϕ ′ ( z ) = − z ϕ ( z ) \phi'(z)=-z\phi(z) ϕ ′ ( z ) = − z ϕ ( z ) gives
∫ κ ∞ z ϕ ( z ) d z = ϕ ( κ ) , ∫ κ ∞ z 2 ϕ ( z ) d z = p + + κ ϕ ( κ ) . \int_{\kappa}^{\infty} z\phi(z)\,\mathrm{d}z = \phi(\kappa), \qquad\int_{\kappa}^{\infty} z^2\phi(z)\,\mathrm{d}z = p_+ + \kappa\phi(\kappa). ∫ κ ∞ z ϕ ( z ) d z = ϕ ( κ ) , ∫ κ ∞ z 2 ϕ ( z ) d z = p + + κ ϕ ( κ ) .
Subtract from the unconditional first and second moments to obtain the lower half-space formulas. Substitute in ξ = μ + h Z + η \xi= \mu+ hZ + \eta ξ = μ + h Z + η and subtract the conditional means' outer products to obtain Equation (258) . The variance-zero score is deterministic and is handled directly.
An equivalent form avoiding division by small route probabilities. The unnormalized moments of the upper cell are
u + = E [ ξ 1 T = + ] = p + μ + h ϕ ( κ ) , u_{+}=\mathbb{E}[\xi\mathbf{1}_{T=+}]=p_{+}\mu+h\phi(\kappa), u + = E [ ξ 1 T = + ] = p + μ + h ϕ ( κ ) ,
V + = E [ ξ ξ T 1 T = + ] = p + ( Σ + μ μ T ) + ϕ ( κ ) ( μ h T + h μ T + κ h h T ) . (259) V_{+}=\mathbb{E}[\xi\xi^{\mathsf{T}}\mathbf{1}_{T=+}]
=p_{+}(\Sigma+\mu\mu^{\mathsf{T}})+\phi(\kappa)(\mu h^{\mathsf{T}}+h\mu^{\mathsf{T}}+\kappa hh^{\mathsf{T}}). \tag*{(259)} V + = E [ ξ ξ T 1 T = + ] = p + ( Σ + μ μ T ) + ϕ ( κ ) ( μ h T + h μ T + κh h T ) . (259)
The lower cell has u − = p − μ − h ϕ ( κ ) u_{-}=p_{-}\mu-h\phi(\kappa) u − = p − μ − h ϕ ( κ ) and the corresponding negative sign in the correction to p − ( Σ + μ μ T ) p_{-}(\Sigma+\mu\mu^{\mathsf{T}}) p − ( Σ + μ μ T ) . The exact risk is therefore
∑ t ∈ { + , − } [ p t ∥ b t ∥ 2 + 2 b t T G t u t + tr ( G t V t G t T ) ] . (260) \sum_{t\in\{+,-\}}\left[p_{t}\lVert b_{t}\rVert^{2}+2b_{t}^{\mathsf{T}}G_{t}u_{t}+\operatorname{tr}(G_{t}V_{t}G_{t}^{\mathsf{T}})\right]. \tag*{(260)} t ∈ { + , − } ∑ [ p t ∥ b t ∥ 2 + 2 b t T G t u t + tr ( G t V t G t T ) ] . (260)
These formulas follow from the same two integrals; they avoid inverse small probabilities, although extreme-tail cancellation still requires numerically stable CDF/survival functions and appropriate precision. They specialize classical truncated-normal moment calculations [24 ] .
Why the mean and cross-covariance both matter. For ξ ∼ N ( 0 , 1 ) \xi\sim\mathcal{N}(0,1) ξ ∼ N ( 0 , 1 ) , T = sign ( ξ ) T=\operatorname{sign}(\xi) T = sign ( ξ ) , and G T = T G_{T}=T G T = T , the error is e = ∣ ξ ∣ e=\lvert\xi\rvert e = ∣ ξ ∣ . Its mean is 2 / π \sqrt{2/\pi} 2/ π , variance is 1 − 2 / π 1-2/\pi 1 − 2/ π , and squared risk is one. Keeping only the conditional variance loses the conditional-mean contribution 2 / π 2/\pi 2/ π . For an independent standard bivariate normal ( ξ 1 , ξ 2 ) (\xi_{1},\xi_{2}) ( ξ 1 , ξ 2 ) , select + + + when ξ 1 + ξ 2 > 2 \xi_{1}+\xi_{2}>\sqrt{2} ξ 1 + ξ 2 > 2 . In the notation above, κ = 1 \kappa=1 κ = 1 and h = ( 1 , 1 ) T / 2 h=(1,1)^{\mathsf{T}}/\sqrt{2} h = ( 1 , 1 ) T / 2 , so the off-diagonal conditional covariance is ( m + − m + 2 ) / 2 < 0 (m_{+}-m_{+}^{2})/2<0 ( m + − m + 2 ) /2 < 0 . It is zero before selection. Thus even independently fitted errors need not be independent or uncorrelated after routing.
Recomputing the route can destroy the quadratic frontier. Let ξ ∼ N ( 0 , 1 ) \xi\sim\mathcal{N}(0,1) ξ ∼ N ( 0 , 1 ) , a λ = 1 − λ a_{\lambda}=1-\lambda a λ = 1 − λ , and use two branches whose error is e λ = a λ ξ e_{\lambda}=a_{\lambda}\xi e λ = a λ ξ when ∣ a λ ξ ∣ > τ > 0 \lvert a_{\lambda}\xi\rvert>\tau>0 ∣ a λ ξ ∣ > τ > 0 and zero otherwise. For 0 ≤ λ < 1 0\leq\lambda<1 0 ≤ λ < 1 ,
R ( λ ) = 2 a λ 2 [ τ a λ ϕ ( τ a λ ) + 1 − Φ ( τ a λ ) ] , R ( 1 ) = 0. (261) R(\lambda)=2a_{\lambda}^{2}\left[\frac{\tau}{a_{\lambda}}\phi\left(\frac{\tau}{a_{\lambda}}\right)+1-\Phi\left(\frac{\tau}{a_{\lambda}}\right)\right],\qquad R(1)=0. \tag*{(261)} R ( λ ) = 2 a λ 2 [ a λ τ ϕ ( a λ τ ) + 1 − Φ ( a λ τ ) ] , R ( 1 ) = 0. (261)
The same integral as above proves this formula. It is not a quadratic in λ \lambda λ : as λ ↑ 1 \lambda\uparrow1 λ ↑ 1 it is smaller than every positive power of 1 − λ 1-\lambda 1 − λ , yet it is positive at zero. A nonzero quadratic cannot have that property. This is a controlled example of route-dependent propagation.
Which functional corrections can a low-rank parameter update realize?
LoRA represents a change to a specified frozen linear weight by a matrix product B A BA B A [13 ] . Its matrix rank is at most the factor width. This differs from the dimension of the span of an arbitrary collection of functions in Proposition 37. To make the comparison precise, fix a feature map ϕ ( x ) ∈ R k \phi(x)\in\mathbb{R}^{k} ϕ ( x ) ∈ R k , its input law, and its uncentered second moment C = E [ ϕ ( x ) ϕ ( x ) T ] ⪰ 0 C=\mathbb{E}[\phi(x)\phi(x)^{\mathsf{T}}]\succeq0 C = E [ ϕ ( x ) ϕ ( x ) T ] ⪰ 0 . Let W 0 ∈ R d × k W_{0}\in\mathbb{R}^{d\times k} W 0 ∈ R d × k be the fixed core and W a ⋆ W_{a}^{\star} W a ⋆ a role’s linear target on these features. Write K a = W a ⋆ − W 0 K_{a}=W_{a}^{\star}-W_{0} K a = W a ⋆ − W 0 . Only updates to this output weight are allowed; changing the feature map changes the problem.
Theorem 54 (Exact adapter approximation bias in feature and output geometry). Let S ⪰ 0 S\succeq0 S ⪰ 0 be a fixed output-loss matrix and let r r r be an integer with 0 ≤ r ≤ min ( d , k ) 0\leq r\leq\min(d,k) 0 ≤ r ≤ min ( d , k ) . Define T a = S 1 / 2 K a C 1 / 2 T_{a}=S^{1/2}K_{a}C^{1/2} T a = S 1/2 K a C 1/2 . Then
inf rank ( Δ a ) ≤ r E [ ∥ ( K a − Δ a ) ϕ ( x ) ∥ S 2 ] = ∑ i > r σ i ( T a ) 2 . (262) \inf_{\operatorname{rank}(\Delta_{a})\leq r}\mathbb{E}\left[\lVert(K_{a}-\Delta_{a})\phi(x)\rVert_{S}^{2}\right]=\sum_{i>r}\sigma_{i}(T_{a})^{2}. \tag*{(262)} rank ( Δ a ) ≤ r inf E [ ∥( K a − Δ a ) ϕ ( x ) ∥ S 2 ] = i > r ∑ σ i ( T a ) 2 . (262) Here σ i \sigma_{i} σ i are decreasing singular values, S 1 / 2 S^{1/2} S 1/2 , C 1 / 2 C^{1/2} C 1/2 are the symmetric positive-semidefinite square roots, and ∥ y ∥ S 2 = y T S y \lVert y\rVert_{S}^{2}=y^{\mathsf{T}}Sy ∥ y ∥ S 2 = y T S y . Both S S S and C C C may be singular. If T a , r T_{a,r} T a , r is a truncated singular-value decomposition, a realizing update is
Δ a = ( S 1 / 2 ) † T a , r ( C 1 / 2 ) † , (263) \Delta_{a}=(S^{1/2})^{\dagger}T_{a,r}(C^{1/2})^{\dagger}, \tag*{(263)} Δ a = ( S 1/2 ) † T a , r ( C 1/2 ) † , (263) where † \dagger † denotes the Moore–Penrose inverse (invert the nonzero singular values and leave zero ones zero). The optimal risk is zero exactly when rank ( T a ) ≤ r \operatorname{rank}(T_{a})\leq r rank ( T a ) ≤ r . For independently chosen role adapters, the weighted average optimum is the sum of Equation (262) with the declared role weights; this is an approximation result, not an estimation or optimization guarantee.
Proof. The objective equals ∥ T a − S 1 / 2 Δ a C 1 / 2 ∥ F 2 \|T_a-S^{1/2}\Delta_aC^{1/2}\|_F^2 ∥ T a − S 1/2 Δ a C 1/2 ∥ F 2 . For any matrix Z Z Z of rank at most r r r , let P Z P_Z P Z project onto its column space. Then ∥ T a − Z ∥ F 2 ≥ ∥ ( I − P Z ) T a ∥ F 2 \|T_a-Z\|_F^2\ge\|(I-P_Z)T_a\|_F^2 ∥ T a − Z ∥ F 2 ≥ ∥ ( I − P Z ) T a ∥ F 2 . Diagonalizing T a T a ⊤ T_aT_a^\top T a T a ⊤ shows tr ( P Z T a T a ⊤ ) ≤ ∑ i ≤ r σ i ( T a ) 2 \operatorname{tr}(P_ZT_aT_a^\top)\le\sum_{i\le r}\sigma_i(T_a)^2 tr ( P Z T a T a ⊤ ) ≤ ∑ i ≤ r σ i ( T a ) 2 : in an eigenbasis the diagonal weights of a rank-at-most-r r r orthogonal projector lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and sum to at most r r r . This proves the lower bound. The left and right singular directions of T a T_a T a with nonzero singular values lie in the ranges of S 1 / 2 S^{1/2} S 1/2 and C 1 / 2 C^{1/2} C 1/2 , respectively. Therefore the pseudoinverse lift has rank at most r r r and satisfies S 1 / 2 Δ a C 1 / 2 = T a , r S^{1/2}\Delta_aC^{1/2}=T_{a,r} S 1/2 Δ a C 1/2 = T a , r , attaining the bound. Every such rank-at-most-r r r update can be factored as B A BA B A by its singular-value decomposition, padding the factors with zeros when necessary.
The covariance must be a feature second moment , not its centered covariance when features have nonzero mean. Null directions of C C C are unobserved under this law; they need not remain null after distribution shift. Likewise S S S null directions are invisible to this loss, not necessarily to another output metric or a downstream computation.
The infimum has no update-norm or factor-norm constraint. A pseudoinverse lift can have large norm on poorly excited feature directions. A bounded-radius adapter problem, including a radius needed for a downstream Taylor bound, therefore need not attain this unconstrained tail value. Theorem 57 and Proposition 58 give the radius-constrained specializations and count each distinct updated matrix once.
Targets outside the linear feature span. For an arbitrary square-integrable desired correction d ⋆ ( x ) d_\star(x) d ⋆ ( x ) , let B = E [ d ⋆ φ ⊤ ] B=\mathbb{E}[d_\star\varphi^\top] B = E [ d ⋆ φ ⊤ ] , K = B C † K=BC^\dagger K = B C † , and u = d ⋆ − K φ u=d_\star-K\varphi u = d ⋆ − K φ . A vector in the null space of C C C is orthogonal to φ \varphi φ almost surely, so B = B C † C B=BC^\dagger C B = B C † C and E [ u φ ⊤ ] = 0 \mathbb{E}[u\varphi^\top]=0 E [ u φ ⊤ ] = 0 . Consequently, for any fixed S ⪰ 0 S\succeq0 S ⪰ 0 ,
inf rank ( Δ ) ≤ r E ∥ d ⋆ − Δ φ ∥ S 2 = E ∥ u ∥ S 2 + ∑ i > r σ i ( S 1 / 2 K C 1 / 2 ) 2 . (264) \inf_{\operatorname{rank}(\Delta)\le r}\mathbb{E}\|d_\star-\Delta\varphi\|_S^2
=\mathbb{E}\|u\|_S^2+\sum_{i>r}\sigma_i(S^{1/2}KC^{1/2})^2. \tag*{(264)} rank ( Δ ) ≤ r inf E ∥ d ⋆ − Δ φ ∥ S 2 = E ∥ u ∥ S 2 + i > r ∑ σ i ( S 1/2 K C 1/2 ) 2 . (264)
This follows by expanding the square, eliminating the cross term using the orthogonality just proved, and applying Theorem 54. The feature-misspecification residual is retained for targets outside the feature span.
A shared factor is a different architecture. With Euclidean output loss, role-specific feature second moments C a C_a C a , and one common column factor U ∈ R d × r U\in\mathbb{R}^{d\times r} U ∈ R d × r , U ⊤ U = I r U^\top U=I_r U ⊤ U = I r , write Δ a = U Z a \Delta_a=UZ_a Δ a = U Z a . For fixed U U U the best value is ∑ a π a ∥ ( I − U U ⊤ ) K a C a 1 / 2 ∥ F 2 \sum_a\pi_a\|(I-UU^\top)K_aC_a^{1/2}\|_F^2 ∑ a π a ∥ ( I − U U ⊤ ) K a C a 1/2 ∥ F 2 , attained by Z a = U ⊤ K a Z_a=U^\top K_a Z a = U ⊤ K a . Optimizing U U U gives
inf U , Z a ∑ a π a E ∥ ( K a − U Z a ) φ a ∥ 2 = ∑ i > r λ i ( ∑ a π a K a C a K a ⊤ ) . (265) \inf_{U,Z_a}\sum_a\pi_a\mathbb{E}\|(K_a-UZ_a)\varphi_a\|^2
=\sum_{i>r}\lambda_i\left(\sum_a\pi_aK_aC_aK_a^\top\right). \tag*{(265)} U , Z a inf a ∑ π a E ∥ ( K a − U Z a ) φ a ∥ 2 = i > r ∑ λ i ( a ∑ π a K a C a K a ⊤ ) . (265)
This follows by the same trace maximization as in the proof above. These eigenvectors are output directions; they are not the function-space eigenvectors in Proposition 37, and the constraint is stronger than separate rank-r r r updates. For example, two targets diag ( 1 , 0 ) \operatorname{diag}(1,0) diag ( 1 , 0 ) and diag ( 0 , 1 ) \operatorname{diag}(0,1) diag ( 0 , 1 ) under C = I 2 C=I_2 C = I 2 and zero core admit separate rank-one updates, but a common one-dimensional output factor has average residual 1 / 2 1/2 1/2 .
Two concrete insertion points. For the preparation code’s tied_depth term, take φ ( x ) = N ( x ) \varphi(x)=N(x) φ ( x ) = N ( x ) and Δ = depth_up depth_down \Delta=\text{depth\_up}\,\text{depth\_down} Δ = depth_up depth_down . It is a parallel residual correction, not a modification of an existing expert weight. For a LoRA update to the down weight of a frozen gated expert, take φ ( x ) = SiLU ( V N ( x ) ) ⊙ U N ( x ) \varphi(x)=\operatorname{SiLU}(VN(x))\odot UN(x) φ ( x ) = SiLU ( V N ( x )) ⊙ U N ( x ) , multiplied by its fixed reference gain when that gain is included in the output loss. A state-dependent but frozen gain function can be absorbed into φ \varphi φ ; a route-index indicator can likewise be included if the evaluation averages calls where that expert is not selected. The surrounding input map, gain function, and selection law must remain fixed across the compared adapters. These are exact local feature maps. An adapter may change the inputs or routes of later blocks, for which the preceding Taylor or conditional-route analysis is still required.
Proposition 55 (Two roles with a freely learned common core). For two targets W 1 ⋆ , W 2 ⋆ W_1^\star,W_2^\star W 1 ⋆ , W 2 ⋆ , a common feature second moment C C C , and weights π 1 , π 2 > 0 \pi_1,\pi_2>0 π 1 , π 2 > 0 with π 1 + π 2 = 1 \pi_1+\pi_2=1 π 1 + π 2 = 1 , and integer r ≥ 0 r\ge0 r ≥ 0 , suppose the common core W 0 W_0 W 0 is an arbitrary matrix rather than fixed. Under Euclidean output loss,
inf W 0 , rank ( Δ i ) ≤ r ∑ i = 1 2 π i ∥ ( W i ⋆ − W 0 − Δ i ) C 1 / 2 ∥ F 2 = π 1 π 2 ∑ j > 2 r σ j ( ( W 1 ⋆ − W 2 ⋆ ) C 1 / 2 ) 2 . (266) \inf_{W_0,\operatorname{rank}(\Delta_i)\le r}\sum_{i=1}^{2}\pi_i\|(W_i^\star-W_0-\Delta_i)C^{1/2}\|_F^2
=\pi_1\pi_2\sum_{j>2r}\sigma_j((W_1^\star-W_2^\star)C^{1/2})^2. \tag*{(266)} W 0 , rank ( Δ i ) ≤ r inf i = 1 ∑ 2 π i ∥ ( W i ⋆ − W 0 − Δ i ) C 1/2 ∥ F 2 = π 1 π 2 j > 2 r ∑ σ j (( W 1 ⋆ − W 2 ⋆ ) C 1/2 ) 2 . (266)
In particular, exact realization on the feature law is possible exactly when the displayed difference has rank at most 2 r 2r 2 r .
Proof. For fixed updates, completing the square over the free core gives π 1 π 2 ∥ ( W 1 ⋆ − W 2 ⋆ − Δ 1 + Δ 2 ) C 1 / 2 ∥ F 2 \pi_1\pi_2\lVert(W_1^\star-W_2^\star-\Delta_1+\Delta_2)C^{1/2}\rVert_F^2 π 1 π 2 ∥( W 1 ⋆ − W 2 ⋆ − Δ 1 + Δ 2 ) C 1/2 ∥ F 2 . The difference of two rank-at-most-r r r matrices has rank at most 2 r 2r 2 r , yielding the lower bound by Theorem 54. Conversely, split a best rank-at-most-2 r 2r 2 r truncation Z Z Z of the whitened target difference into two groups of at most r r r singular terms, Z = Z 1 + Z 2 Z=Z_1+Z_2 Z = Z 1 + Z 2 . Take Δ 1 = Z 1 ( C 1 / 2 ) † \Delta_1=Z_1(C^{1/2})^\dagger Δ 1 = Z 1 ( C 1/2 ) † and Δ 2 = − Z 2 ( C 1 / 2 ) † \Delta_2=-Z_2(C^{1/2})^\dagger Δ 2 = − Z 2 ( C 1/2 ) † . Their whitened difference is exactly Z Z Z ; choose W 0 = π 1 ( W 1 ⋆ − Δ 1 ) + π 2 ( W 2 ⋆ − Δ 2 ) W_0=\pi_1(W_1^\star-\Delta_1)+\pi_2(W_2^\star-\Delta_2) W 0 = π 1 ( W 1 ⋆ − Δ 1 ) + π 2 ( W 2 ⋆ − Δ 2 ) . This attains the lower bound, including singular C C C and 2 r 2r 2 r exceeding the matrix dimensions.
For the earlier targets ± I 2 \pm I_2 ± I 2 with C = I 2 C=I_2 C = I 2 , the fixed zero core leaves residual one for each rank-one adapter. A free core W 0 = diag ( 1 , − 1 ) W_0=\operatorname{diag}(1,-1) W 0 = diag ( 1 , − 1 ) instead allows Δ + = diag ( 0 , 2 ) \Delta_+=\operatorname{diag}(0,2) Δ + = diag ( 0 , 2 ) and Δ − = diag ( − 2 , 0 ) \Delta_-=\operatorname{diag}(-2,0) Δ − = diag ( − 2 , 0 ) , both rank one, and zero residual. Thus the fixed-core counterexample cannot be used as a lower bound for jointly learned cores. The two-role formula does not extend to an arbitrary number of roles by checking pairwise ranks: simultaneous realizability adds compatibility constraints. For example, take three targets 0 0 0 , I 2 I_2 I 2 , 2 I 2 2I_2 2 I 2 , C = I 2 C=I_2 C = I 2 , and r = 1 r=1 r = 1 . Every pairwise difference has rank at most 2 r 2r 2 r . Exact simultaneous realization would require det W 0 = 0 \det W_0=0 det W 0 = 0 and det ( I 2 − W 0 ) = 0 \det(I_2-W_0)=0 det ( I 2 − W 0 ) = 0 , hence tr W 0 = 1 \operatorname{tr}W_0=1 tr W 0 = 1 . But then det ( 2 I 2 − W 0 ) = 4 − 2 tr W 0 + det W 0 = 2 \det(2I_2-W_0)=4-2\operatorname{tr}W_0+\det W_0=2 det ( 2 I 2 − W 0 ) = 4 − 2 tr W 0 + det W 0 = 2 , a contradiction. This is an obstruction to exact realization, not a computed positive lower bound on the three-role infimum.
Initialization-dependent low-rank tangent images
Proposition 56. Initialization-dependent tangent image and its missing term. For the feature map above with Euclidean output loss, parameterize an update by B A BA B A . At factors ( B 0 , A 0 ) (B_0,A_0) ( B 0 , A 0 ) its exact change is
( B 0 + B ˙ ) ( A 0 + A ˙ ) − B 0 A 0 = B 0 A ˙ + B ˙ A 0 + B ˙ A ˙ . (267) (B_0+\dot{B})(A_0+\dot{A})-B_0A_0=B_0\dot{A}+\dot{B}A_0+\dot{B}\dot{A}. \tag*{(267)} ( B 0 + B ˙ ) ( A 0 + A ˙ ) − B 0 A 0 = B 0 A ˙ + B ˙ A 0 + B ˙ A ˙ . (267) Thus the first-order parameter-to-function image is { ( B 0 A ˙ + B ˙ A 0 ) φ } \{(B_0\dot{A}+\dot{B}A_0)\varphi\} {( B 0 A ˙ + B ˙ A 0 ) φ } , not all rank-at-most-r r r corrections. For B 0 = 0 B_0=0 B 0 = 0 , put V 0 = A 0 C 1 / 2 V_0=A_0C^{1/2} V 0 = A 0 C 1/2 and let P V 0 P_{V_0} P V 0 be the orthogonal projector onto its row space. If K φ K\varphi K φ is the desired functional change, then
inf B ˙ E ∥ ( K − B ˙ A 0 ) φ ∥ 2 = ∥ K C 1 / 2 ( I − P V 0 ) ∥ F 2 . (268) \inf_{\dot{B}}\mathbb{E}\lVert(K-\dot{B}A_0)\varphi\rVert^2=\lVert KC^{1/2}(I-P_{V_0})\rVert_F^2. \tag*{(268)} B ˙ inf E ∥( K − B ˙ A 0 ) φ ∥ 2 = ∥ K C 1/2 ( I − P V 0 ) ∥ F 2 . (268) attained by B ˙ = K C 1 / 2 V 0 † \dot{B}=KC^{1/2}V_0^\dagger B ˙ = K C 1/2 V 0 † . The omitted term in Equation (267) obeys
∥ B ˙ A ˙ φ ∥ L 2 ≤ ∥ C 1 / 2 ∥ 2 ∥ B ˙ ∥ F ∥ A ˙ ∥ F ≤ 1 2 ∥ C 1 / 2 ∥ 2 ( ∥ B ˙ ∥ F 2 + ∥ A ˙ ∥ F 2 ) . (269) \lVert\dot{B}\dot{A}\varphi\rVert_{L^2}\leq\lVert C^{1/2}\rVert_2\lVert\dot{B}\rVert_F\lVert\dot{A}\rVert_F\leq\frac{1}{2}\lVert C^{1/2}\rVert_2\left(\lVert\dot{B}\rVert_F^2+\lVert\dot{A}\rVert_F^2\right). \tag*{(269)} ∥ B ˙ A ˙ φ ∥ L 2 ≤ ∥ C 1/2 ∥ 2 ∥ B ˙ ∥ F ∥ A ˙ ∥ F ≤ 2 1 ∥ C 1/2 ∥ 2 ( ∥ B ˙ ∥ F 2 + ∥ A ˙ ∥ F 2 ) . (269)
Proof. Multiply the two factors and group linear and quadratic terms. When B 0 = 0 B_0=0 B 0 = 0 , the matrix approximation problem is inf B ˙ ∥ K C 1 / 2 − B ˙ V 0 ∥ F 2 \inf_{\dot{B}}\lVert KC^{1/2}-\dot{B}V_0\rVert_F^2 inf B ˙ ∥ K C 1/2 − B ˙ V 0 ∥ F 2 . Each row of B ˙ V 0 \dot{B}V_0 B ˙ V 0 lies in the row space of V 0 V_0 V 0 , and every row projection is attained by the displayed pseudoinverse, proving the equality. Finally use submultiplicativity of Frobenius and operator norms on B ˙ A ˙ C 1 / 2 \dot{B}\dot{A}C^{1/2} B ˙ A ˙ C 1/2 , followed by 2 a b ≤ a 2 + b 2 2ab\leq a^2+b^2 2 ab ≤ a 2 + b 2 .
At the preparation code’s zero depth-up initialization, the initial functional tangent therefore only uses the row space of the initialized depth-down after feature whitening. It is generally narrower than the class obtained by training both factors. If both factors are zero, the first derivative is zero but nonzero finite updates remain possible through B ˙ A ˙ \dot{B}\dot{A} B ˙ A ˙ . More concretely, with r = 1 r=1 r = 1 , C = I 2 C=I_2 C = I 2 , A 0 = ( 1 , 0 ) A_0=(1,0) A 0 = ( 1 , 0 ) , B 0 = 0 B_0=0 B 0 = 0 , and target K = diag ( 0 , 1 ) K=\operatorname{diag}(0,1) K = diag ( 0 , 1 ) , the tangent residual is one while a finite rank-one update using row factor ( 0 , 1 ) (0,1) ( 0 , 1 ) is exact. A tangent obstruction is not a global impossibility theorem.
A rigorous nonlinear parameter-to-function fallback. For a general insertion point, fix a Hilbert space of functions on a fixed probe law and suppose F ( W ) \mathcal{F}(W) F ( W ) is twice differentiable on the parameter ball ∥ Δ ∥ F ≤ t \lVert\Delta\rVert_F\leq t ∥ Δ ∥ F ≤ t , with a uniform second-derivative norm at most H H H . Let J = D F ( W 0 ) J=D\mathcal{F}(W_0) J = D F ( W 0 ) , let d ⋆ d_\star d ⋆ be a target functional correction, and let S r ( t ) = { Δ : rank ( Δ ) ≤ r , ∥ Δ ∥ F ≤ t } \mathcal{S}_r(t)=\{\Delta:\operatorname{rank}(\Delta)\leq r,\lVert\Delta\rVert_F\leq t\} S r ( t ) = { Δ : rank ( Δ ) ≤ r , ∥ Δ ∥ F ≤ t } . Define the constrained image distance
b r ( t ) = inf Δ ∈ S r ( t ) ∥ d ⋆ − J Δ ∥ H , η = H t 2 / 2. b_r(t)=\inf_{\Delta\in\mathcal{S}_r(t)}\lVert d_\star-J\Delta\rVert_{\mathcal{H}},\qquad\eta=Ht^2/2. b r ( t ) = Δ ∈ S r ( t ) inf ∥ d ⋆ − J Δ ∥ H , η = H t 2 /2.
Uniform Taylor remainder and the triangle inequalities give
( max { 0 , b r ( t ) − η } ) 2 ≤ inf Δ ∈ S r ( t ) ∥ d ⋆ − [ F ( W 0 + Δ ) − F ( W 0 ) ] ∥ H 2 ≤ ( b r ( t ) + η ) 2 . (270) \left(\max\{0,b_r(t)-\eta\}\right)^2 \le\inf_{\Delta\in S_r(t)} \left\|d_\star-\left[F(W_0+\Delta)-F(W_0)\right]\right\|_{\mathcal{H}}^2 \le\left(b_r(t)+\eta\right)^2. \tag*{(270)} ( max { 0 , b r ( t ) − η } ) 2 ≤ Δ ∈ S r ( t ) inf ∥ d ⋆ − [ F ( W 0 + Δ ) − F ( W 0 ) ] ∥ H 2 ≤ ( b r ( t ) + η ) 2 . (270)
Indeed, apply the pointwise norm inequalities for every feasible Δ \Delta Δ and then take infima; an approximating sequence suffices if an infimum is not attained. The image J ( S r ( t ) ) J(S_r(t)) J ( S r ( t )) is generally not a linear subspace, so projecting onto the full range of J J J only supplies a relaxation, not a realizability certificate. Smoothness into L 2 L^2 L 2 requires a uniform square-integrable derivative envelope on the declared input law; pointwise differentiability alone does not supply it. For hard routing one must first certify a common smooth branch or separately pay for route changes.
Applying the conditional bounds. A numerical nonlinear bound still needs a specified checkpoint, reference input population, perturbed weights, perturbation law, and meaningful margin/tail bounds. A multistep endogenous route generally conditions on an intersection of nonlinear decisions, rather than the single half-space solved here; selected indices alone do not fix gains or Jacobians. The radius extension in Section P fixes the local feature law and core, constrains the actual output-matrix update, and separates approximation from terminal performance.
Rank-Constrained Updates and Precisely Defined Branch Replay
Overview. We address two limits of a local approximation argument: the update must fit a budget on actual parameter changes, and fixed-support execution must be distinguished from natural routing. The guarantees separately track matrix rank, total radius, support disagreement, and the extra cost of replaying recorded gains, for fixed features/cores and finite fixed-prefix computations. The matrix arguments extend classical low-rank least-squares geometry [7 ] ; the routing arguments are pathwise couplings, not assumptions of route independence. Sections P.1–P.2 handle one-matrix and total update budgets; Section P.3 compares natural execution with current-gain support replay, and Section P.4 accounts for freezing the gains as well.
One matrix: an exact isotropic radius frontier and semidefinite bounds
Fix a core W 0 ∈ R d × k W_0 \in\mathbb{R}^{d\times k} W 0 ∈ R d × k and target W ⋆ W^\star W ⋆ , and set K = W ⋆ − W 0 K=W^\star-W_0 K = W ⋆ − W 0 . The feature map φ \varphi φ , its probability law, and C = E [ φ φ T ] ⪰ 0 C=\mathbb{E}[\varphi\varphi^{\mathsf{T}}]\succeq0 C = E [ φ φ T ] ⪰ 0 remain fixed as the update changes. For output geometry S ⪰ 0 S\succeq0 S ⪰ 0 , integer 0 ≤ r ≤ min ( d , k ) 0\le r\le\min(d,k) 0 ≤ r ≤ min ( d , k ) and finite t ≥ 0 t\ge0 t ≥ 0 , define
E r , t ( K ; S , C ) = min rank ( Δ ) ≤ r ∥ Δ ∥ F ≤ t ∥ S 1 / 2 ( K − Δ ) C 1 / 2 ∥ F 2 . (271) \mathcal{E}_{r,t}(K;S,C)=\min_{\substack{\operatorname{rank}(\Delta)\le r\\\|\Delta\|_F\le t}}\left\|S^{1/2}(K-\Delta)C^{1/2}\right\|_F^2. \tag*{(271)} E r , t ( K ; S , C ) = rank ( Δ ) ≤ r ∥Δ ∥ F ≤ t min S 1/2 ( K − Δ ) C 1/2 F 2 . (271)
This equals min E ∥ ( K − Δ ) φ ∥ S 2 \min\mathbb{E}\|(K-\Delta)\varphi\|_S^2 min E ∥ ( K − Δ ) φ ∥ S 2 . The constraint is on the actual matrix update, not on a factor’s norm or on a training gradient. The feasible set is nonempty and compact: rank at most r r r is closed (all ( r + 1 ) (r+1) ( r + 1 ) -minors vanish), and the Frobenius ball is closed and bounded. Hence the continuous objective attains its minimum, including for singular S , C S,C S , C . Write ( x ) + = max { x , 0 } (x)_+=\max\{x,0\} ( x ) + = max { x , 0 } and let T r T_r T r denote a truncated SVD retaining at most r r r singular values of T T T ; T 0 = 0 T_0=0 T 0 = 0 .
Theorem 57 (Rank and matrix-radius approximation). In isotropic geometry, put q r = ∥ K r ∥ F = ( ∑ i ≤ r σ i ( K ) 2 ) 1 / 2 q_r=\|K_r\|_F=\left(\sum_{i\le r}\sigma_i(K)^2\right)^{1/2} q r = ∥ K r ∥ F = ( ∑ i ≤ r σ i ( K ) 2 ) 1/2 . Then
E r , t ( K ; I , I ) = ∑ i > r σ i ( K ) 2 + ( q r − t ) + 2 . (272) \mathcal{E}_{r,t}(K;I,I)=\sum_{i>r}\sigma_i(K)^2+(q_r-t)_+^2. \tag*{(272)} E r , t ( K ; I , I ) = i > r ∑ σ i ( K ) 2 + ( q r − t ) + 2 . (272) For q r > 0 q_r>0 q r > 0 , an attaining update is Δ ⋆ = min { 1 , t / q r } K r \Delta^\star=\min\{1,t/q_r\}K_r Δ ⋆ = min { 1 , t / q r } K r ; for q r = 0 q_r=0 q r = 0 , use Δ ⋆ = 0 \Delta^\star=0 Δ ⋆ = 0 .
For arbitrary positive-semidefinite S , C S,C S , C , set
T = S 1 / 2 K C 1 / 2 , q r = ∥ T r ∥ F , E r = ∑ i > r σ i ( T ) 2 , κ = ∥ S 1 / 2 ∥ 2 ∥ C 1 / 2 ∥ 2 , Δ 0 = ( S 1 / 2 ) † T r ( C 1 / 2 ) † , α = { min { 1 , t / ∥ Δ 0 ∥ F } , Δ 0 ≠ 0 , 1 , Δ 0 = 0. (273) \begin{aligned}
T&=S^{1/2}KC^{1/2},\qquad q_r=\|T_r\|_F,\qquad E_r=\sum_{i>r}\sigma_i(T)^2,\qquad\kappa=\|S^{1/2}\|_2\|C^{1/2}\|_2,\\
\Delta_0&=(S^{1/2})^\dagger T_r(C^{1/2})^\dagger,\qquad\alpha=\begin{cases}\min\{1,t/\|\Delta_0\|_F\},&\Delta_0\ne0,\\1,&\Delta_0=0.\end{cases} \tag*{(273)}
\end{aligned} T Δ 0 = S 1/2 K C 1/2 , q r = ∥ T r ∥ F , E r = i > r ∑ σ i ( T ) 2 , κ = ∥ S 1/2 ∥ 2 ∥ C 1/2 ∥ 2 , = ( S 1/2 ) † T r ( C 1/2 ) † , α = { min { 1 , t /∥ Δ 0 ∥ F } , 1 , Δ 0 = 0 , Δ 0 = 0. (273)
Then
E r + ( q r − κ t ) + 2 ≤ E r , t ( K ; S , C ) ≤ E r + ( 1 − α ) 2 q r 2 . (274) E_r + (q_r-\kappa t)_+^2 \le\mathcal{E}_{r,t}(K;S,C) \le E_r + (1-\alpha)^2q_r^2. \tag*{(274)} E r + ( q r − κ t ) + 2 ≤ E r , t ( K ; S , C ) ≤ E r + ( 1 − α ) 2 q r 2 . (274)
The right-hand value is attained by the feasible candidate α Δ 0 \alpha\Delta_0 α Δ 0 , but need not be the minimum. If t ≥ ∥ Δ 0 ∥ F t \ge\|\Delta_0\|_F t ≥ ∥ Δ 0 ∥ F , the two bounds equal E r E_r E r . These conventions cover r = 0 r=0 r = 0 , t = 0 t=0 t = 0 , a zero target, null feature/output geometries, and tied singular values without dividing by zero.
Proof. For any feasible Δ \Delta Δ let P P P project onto its column space and put u = ∥ Δ ∥ F u=\|\Delta\|_F u = ∥Δ ∥ F . Since P Δ = Δ P\Delta=\Delta P Δ = Δ , ⟨ K , Δ ⟩ F = ⟨ P K , Δ ⟩ F ≤ u ∥ P K ∥ F \langle K,\Delta\rangle_F=\langle PK,\Delta\rangle_F\le u\|PK\|_F ⟨ K , Δ ⟩ F = ⟨ P K , Δ ⟩ F ≤ u ∥ P K ∥ F . In an eigenbasis of K K ⊤ KK^\top K K ⊤ , the diagonal entries of P P P lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and sum to at most r r r . It follows that ∥ P K ∥ F 2 = tr ( P K K ⊤ ) ≤ q r 2 \|PK\|_F^2=\operatorname{tr}(PKK^\top)\le q_r^2 ∥ P K ∥ F 2 = tr ( P K K ⊤ ) ≤ q r 2 . Consequently
∥ K − Δ ∥ F 2 ≥ ∥ K ∥ F 2 + u 2 − 2 u q r = ∑ i > r σ i ( K ) 2 + ( u − q r ) 2 . \|K-\Delta\|_F^2\ge\|K\|_F^2+u^2-2uq_r=\sum_{i>r}\sigma_i(K)^2+(u-q_r)^2. ∥ K − Δ ∥ F 2 ≥ ∥ K ∥ F 2 + u 2 − 2 u q r = i > r ∑ σ i ( K ) 2 + ( u − q r ) 2 .
Minimize over 0 ≤ u ≤ t 0\le u\le t 0 ≤ u ≤ t . A scaled K r K_r K r has the optimal norm and attains the inner-product bound; the zero case is immediate.
For the general lower bound, Z = S 1 / 2 Δ C 1 / 2 Z=S^{1/2}\Delta C^{1/2} Z = S 1/2 Δ C 1/2 has rank at most r r r and norm at most κ t \kappa t κ t . Minimizing over all such Z Z Z relaxes the actual image constraint, so the isotropic formula applied to T T T gives the left-hand side of Equation (274) . Every nonzero left/right singular direction of T T T lies in the range of S 1 / 2 / C 1 / 2 S^{1/2}/C^{1/2} S 1/2 / C 1/2 , respectively. Thus S 1 / 2 Δ 0 C 1 / 2 = T r S^{1/2}\Delta_0C^{1/2}=T_r S 1/2 Δ 0 C 1/2 = T r and rank ( Δ 0 ) ≤ r \operatorname{rank}(\Delta_0)\le r rank ( Δ 0 ) ≤ r , also when these matrices are singular. Scaling by α \alpha α makes the update feasible. Orthogonality of T − T r T-T_r T − T r and T r T_r T r gives
∥ T − α T r ∥ F 2 = E r + ( 1 − α ) 2 q r 2 . \|T-\alpha T_r\|_F^2=E_r+(1-\alpha)^2q_r^2. ∥ T − α T r ∥ F 2 = E r + ( 1 − α ) 2 q r 2 .
If t ≥ ∥ Δ 0 ∥ F t\ge\|\Delta_0\|_F t ≥ ∥ Δ 0 ∥ F , this equals E r E_r E r and q r = ∥ S 1 / 2 Δ 0 C 1 / 2 ∥ F ≤ κ t q_r=\|S^{1/2}\Delta_0C^{1/2}\|_F\le\kappa t q r = ∥ S 1/2 Δ 0 C 1/2 ∥ F ≤ κ t , proving the final claim.
A suboptimality example in semidefinite geometry. Take S = I 2 S=I_2 S = I 2 , C = diag ( 10 − 2 , 1 ) C=\operatorname{diag}(10^{-2},1) C = diag ( 1 0 − 2 , 1 ) , K = diag ( 20 , 1 ) K=\operatorname{diag}(20,1) K = diag ( 20 , 1 ) , and r = t = 1 r=t=1 r = t = 1 . Then T = diag ( 2 , 1 ) T=\operatorname{diag}(2,1) T = diag ( 2 , 1 ) and Δ 0 = diag ( 20 , 0 ) \Delta_0=\operatorname{diag}(20,0) Δ 0 = diag ( 20 , 0 ) . The scaled candidate α Δ 0 = diag ( 1 , 0 ) \alpha\Delta_0=\operatorname{diag}(1,0) α Δ 0 = diag ( 1 , 0 ) has loss ( 2 − 0.1 ) 2 + 1 = 4.61 (2-0.1)^2+1=4.61 ( 2 − 0.1 ) 2 + 1 = 4.61 . The feasible rank-one update diag ( 0 , 1 ) \operatorname{diag}(0,1) diag ( 0 , 1 ) has loss 4 4 4 . Thus scaled transformed SVD can be strictly suboptimal under a parameter-norm budget. Replacing the transformed feasible image by the rank-constrained Frobenius ball of radius κ t \kappa t κ t gives a relaxation. Imposing the unscaled radius t t t directly in transformed coordinates is not in general a valid relaxation or a feasible parameter construction. Poorly excited directions can make the inverse lift expensive.
For an arbitrary square-integrable desired correction, the fixed-feature orthogonal decomposition in Equation (264) still applies for every feasible Δ \Delta Δ . Its feature-misspecification residual must be added to Equation (271) .
A total radius over distinct parameter matrices
Let Δ 1 , … , Δ E \Delta_1,\ldots,\Delta_E Δ 1 , … , Δ E be updates to distinct physical parameter matrices , and form d d d by concatenating their vectorized entries once each. Then
∥ d ∥ 2 2 = ∑ e = 1 E ∥ Δ e ∥ F 2 . (275) \|d\|_2^2=\sum_{e=1}^{E}\|\Delta_e\|_F^2. \tag*{(275)} ∥ d ∥ 2 2 = e = 1 ∑ E ∥ Δ e ∥ F 2 . (275)
An expert reused at several tokens or depths contributes one summand, not a fresh matrix or independent random draw at each call. Conversely, giving each of E E E distinct matrices radius t t t permits total radius E t \sqrt{E}\,t E t , not t t t .
Proposition 58 (A common isotropic radius for distinct update blocks). Fix targets K e K_e K e , ranks r e r_e r e within their matrix dimensions, and positive weights w e w_e w e summing to one. Put q e = ∥ ( K e ) r e ∥ F q_e=\|(K_e)_{r_e}\|_F q e = ∥ ( K e ) r e ∥ F , a e = ∑ i > r e σ i ( K e ) 2 a_e=\sum_{i>r_e}\sigma_i(K_e)^2 a e = ∑ i > r e σ i ( K e ) 2 , and Q 0 = ( ∑ e q e 2 ) 1 / 2 Q_0=\left(\sum_e q_e^2\right)^{1/2} Q 0 = ( ∑ e q e 2 ) 1/2 . Then
min rank ( Δ e ) ≤ r e ∑ e ∥ Δ e ∥ F 2 ≤ t 2 ∑ e w e ∥ K e − Δ e ∥ F 2 = ∑ e w e a e + ∑ e w e ( q e − u e ) 2 , (276) \min_{\substack{\operatorname{rank}(\Delta_e)\le r_e\\ \sum_e\|\Delta_e\|_F^2\le t^2}}\sum_e w_e\|K_e-\Delta_e\|_F^2
=\sum_e w_ea_e+\sum_e w_e(q_e-u_e)^2, \tag*{(276)} rank ( Δ e ) ≤ r e ∑ e ∥ Δ e ∥ F 2 ≤ t 2 min e ∑ w e ∥ K e − Δ e ∥ F 2 = e ∑ w e a e + e ∑ w e ( q e − u e ) 2 , (276) where u e = q e u_e=q_e u e = q e if t ≥ Q 0 t\ge Q_0 t ≥ Q 0 , and u e = 0 u_e=0 u e = 0 if t = 0 t=0 t = 0 . For 0 < t < Q 0 0<t<Q_0 0 < t < Q 0 , there is a unique η > 0 \eta>0 η > 0 satisfying
u e = w e q e w e + η , ∑ e u e 2 = t 2 . (277) u_e=\frac{w_eq_e}{w_e+\eta},\qquad\sum_eu_e^2=t^2. \tag*{(277)} u e = w e + η w e q e , e ∑ u e 2 = t 2 . (277) An optimizer is Δ e = ( u e / q e ) ( K e ) r e \Delta_e=(u_e/q_e)(K_e)_{r_e} Δ e = ( u e / q e ) ( K e ) r e when q e > 0 q_e>0 q e > 0 and zero otherwise. This is a deterministic approximation identity; it assumes no independence between parameter errors and proves no optimizer’s convergence.
Proof. The inner-product inequality in Theorem 57 reduces each block to a e + ( q e − u e ) 2 a_e + (q_e-u_e)^2 a e + ( q e − u e ) 2 at prescribed norm u e u_e u e . An optimum need not use u e > q e u_e > q_e u e > q e : replacing it by q e q_e q e reduces both radius and objective. The problem is therefore the scalar quadratic minimization over 0 ≤ u e ≤ q e 0 \le u_e \le q_e 0 ≤ u e ≤ q e , ∑ e u e 2 ≤ t 2 \sum_e u_e^2 \le t^2 ∑ e u e 2 ≤ t 2 , and aligned truncated SVDs attain any such choice. The inactive and zero-radius cases follow directly. For 0 < t < Q 0 0 < t < Q_0 0 < t < Q 0 , the squared norm of ( w e q e / ( w e + η ) ) e (w_e q_e/(w_e+\eta))_e ( w e q e / ( w e + η ) ) e is continuous and strictly decreasing from Q 0 2 Q_0^2 Q 0 2 to zero as η \eta η increases from zero to infinity. The resulting u u u minimizes ∑ e w e ( q e − u e ) 2 + η ∑ e u e 2 \sum_e w_e(q_e-u_e)^2+\eta\sum_e u_e^2 ∑ e w e ( q e − u e ) 2 + η ∑ e u e 2 coordinatewise. For any feasible v v v , this minimization and ∑ e u e 2 = t 2 ≥ ∑ e v e 2 \sum_e u_e^2=t^2\ge\sum_e v_e^2 ∑ e u e 2 = t 2 ≥ ∑ e v e 2 imply ∑ e w e ( q e − u e ) 2 ≤ ∑ e w e ( q e − v e ) 2 \sum_e w_e(q_e-u_e)^2\le\sum_e w_e(q_e-v_e)^2 ∑ e w e ( q e − u e ) 2 ≤ ∑ e w e ( q e − v e ) 2 . This proves global optimality without an assumption about numerical search.
Several roles sharing one actual update. Roles are not automatically distinct variables in Equation (276) . For fixed tying groups J e \mathcal{J}_e J e , common isotropic geometry within each group, and positive role weights π a \pi_a π a summing to one, set w e = ∑ a ∈ J e π a w_e=\sum_{a\in\mathcal{J}_e}\pi_a w e = ∑ a ∈ J e π a and K ˉ e = w e − 1 ∑ a ∈ J e π a K a \bar{K}_e=w_e^{-1}\sum_{a\in\mathcal{J}_e}\pi_aK_a K ˉ e = w e − 1 ∑ a ∈ J e π a K a . Completing the square gives
∑ a π a ∥ K a − Δ α ( a ) ∥ F 2 = ∑ e ∑ a ∈ J e π a ∥ K a − K ˉ e ∥ F 2 + ∑ e w e ∥ K ˉ e − Δ e ∥ F 2 . (278) \sum_a \pi_a\lVert K_a-\Delta_{\alpha(a)}\rVert_F^2
=
\sum_e\sum_{a\in\mathcal{J}_e}\pi_a\lVert K_a-\bar{K}_e\rVert_F^2
+
\sum_e w_e\lVert\bar{K}_e-\Delta_e\rVert_F^2. \tag*{(278)} a ∑ π a ∥ K a − Δ α ( a ) ∥ F 2 = e ∑ a ∈ J e ∑ π a ∥ K a − K ˉ e ∥ F 2 + e ∑ w e ∥ K ˉ e − Δ e ∥ F 2 . (278)
Apply Proposition 58 to the unique matrices and group centroids in the second term. The first term is an irreducible tying bias in this model. For example, targets + 1 +1 + 1 , − 1 -1 − 1 with equal weights and one shared scalar update have loss 1 + Δ 2 ≥ 1 1+\Delta^2\ge1 1 + Δ 2 ≥ 1 , even though separate updates of radius one fit each target exactly. Merely adding separate per-role optima would give an invalid claim of joint realizability. For two distinct targets both equal to 1 1 1 , a total radius one instead gives equal optimal updates 1 / 2 1/\sqrt{2} 1/ 2 and average loss ( 1 − 1 / 2 ) 2 (1-1/\sqrt{2})^2 ( 1 − 1/ 2 ) 2 , rather than zero.
For different semidefinite geometries and distinct variables, any radii t e t_e t e with ∑ e t e 2 ≤ t 2 \sum_e t_e^2\le t^2 ∑ e t e 2 ≤ t 2 yield jointly feasible scaled lifts from Equation (273) . Their weighted upper-bound values add. Their individual lower-bound values add under these individual caps; for only a total cap, a valid lower bound minimizes the sum of individual lower bounds over t e ≥ 0 t_e\ge0 t e ≥ 0 , ∑ e t e 2 ≤ t 2 \sum_e t_e^2\le t^2 ∑ e t e 2 ≤ t 2 . With shared variables, evaluate the same constructed Δ e \Delta_e Δ e in every associated role; separate per-role minima are only a relaxation unless compatibility is proved.
Interface with the nonlinear parameter model. The concatenated d d d in Equation (275) is the same parameter vector used in Equation (240) . Choosing radii with total at most t t t makes the derivative envelopes at that radius applicable, subject to their separate state-region and route hypotheses. For random updates with this bound almost surely, E ∥ d ∥ 4 ≤ t 4 \mathbb{E}\lVert d\rVert^4\le t^4 E ∥ d ∥ 4 ≤ t 4 ; covariance or independence of calls is not needed for this inequality. Repetition still affects G j , c j , b j G_j,c_j,b_j G j , c j , b j , not the count of distinct parameter entries. Applying the radius bound to terminal risk also requires the stated route, remainder, cross-covariance, and exit controls. For factors initialized nontrivially, the actual increment is B A − B 0 A 0 BA-B_0A_0 B A − B 0 A 0 ; factor width r r r does not by itself make this increment rank at most r r r . The rank and radius constraints must hold for the increment, as emphasized by Proposition 56 .
Support replay recomputes the selected gain at its own state
Fix a finite prefix (or a distribution of prefixes with finite maximum length), not a recursively generated token sequence. All states include the full prefix and every value needed by attention, residual paths, and subsequent events. For events j = 0 , … , H − 1 j=0,\ldots,H-1 j = 0 , … , H − 1 , let A j \mathcal{A}_j A j be the finite set of complete support patterns at that event; a pattern may specify one expert per token. A deterministic rule ρ j ( s ) \rho_j(s) ρ j ( s ) , including its tie convention, selects the natural support. Route-free events have a single possible pattern. Let F j , a ( s ) F_{j,a}(s) F j , a ( s ) be the complete branch extension at prescribed support a a a , including the current-state gain, expert inputs, and every shared path. In a top-one residual event it has the form
F j , a ( s ) = B j ( s ) + p j , a ( s ) v j , a ( s ) , p j ( s ) = softmax ( z j ( s ) ) . (279) F_{j,a}(s)=B_j(s)+p_{j,a}(s)v_{j,a}(s),\qquad p_j(s)=\operatorname{softmax}(z_j(s)). \tag*{(279)} F j , a ( s ) = B j ( s ) + p j , a ( s ) v j , a ( s ) , p j ( s ) = softmax ( z j ( s )) . (279)
Here B j B_j B j includes the common residual path. Forcing a a a uses p j , a ( s ) p_{j,a}(s) p j , a ( s ) , not max e p j , e ( s ) \max_e p_{j,e}(s) max e p j , e ( s ) and not a gain recorded at another state. The branch is defined also where a a a would not win.
On one common probability space, fix a prescribed support sequence a j ∘ a_j^\circ a j ∘ (for example, recorded from a declared reference pass), an input, model parameters, and any common external random tape. For each realization these objects are held the same in the following two runs:
n 0 = r 0 = s 0 , n j + 1 = F j , ρ j ( n j ) ( n j ) , r j + 1 = F j , a j ∘ ( r j ) . (280) n_0=r_0=s_0,\qquad n_{j+1}=F_{j,\rho_j(n_j)}(n_j),\qquad r_{j+1}=F_{j,a_j^\circ}(r_j). \tag*{(280)} n 0 = r 0 = s 0 , n j + 1 = F j , ρ j ( n j ) ( n j ) , r j + 1 = F j , a j ∘ ( r j ) . (280)
Thus n n n is natural execution and r r r is support replay . The prescribed sequence is not revised in response to either run’s terminal loss. It may depend on the input and correlated parameters; no independence is assumed. Replay is a defined counterfactual computation. Model parameters are the same within this pair; comparison of two models uses a separate pair for each.
Theorem 59 (Natural execution versus support replay). Let τ \tau τ be the first j j j with ρ j ( r j ) ≠ a j ∘ \rho_j(r_j) \ne a_j^\circ ρ j ( r j ) = a j ∘ , setting τ = H \tau= H τ = H if there is none, and put Ω = { τ = H } \Omega= \{\tau= H\} Ω = { τ = H } and p d i f f = P ( Ω c ) p_{\mathrm{diff}} = \mathbb{P}(\Omega^c) p diff = P ( Ω c ) . Then τ \tau τ is also the first index at which ρ j ( n j ) ≠ a j ∘ \rho_j(n_j) \ne a_j^\circ ρ j ( n j ) = a j ∘ . In particular, n H = r H n_H = r_H n H = r H on Ω \Omega Ω . For a common target Y Y Y and the same measurable loss 0 ≤ ℓ ( s , Y ) ≤ M < ∞ 0 \le\ell(s,Y) \le M < \infty 0 ≤ ℓ ( s , Y ) ≤ M < ∞ in both runs,
∣ R n a t − R r e p ∣ : = ∣ E ℓ ( n H , Y ) − E ℓ ( r H , Y ) ∣ ≤ M p d i f f . (281) \lvert R^{\mathrm{nat}} - R^{\mathrm{rep}} \rvert:= \lvert\mathbb{E}\ell(n_H,Y) - \mathbb{E}\ell(r_H,Y) \rvert\le M p_{\mathrm{diff}}. \tag*{(281)} ∣ R nat − R rep ∣ := ∣ E ℓ ( n H , Y ) − E ℓ ( r H , Y )∣ ≤ M p diff . (281) This bound needs neither smoothness nor independence, and its factor M M M is sharp over the stated class.
For a functional refinement, suppose the prescribed branch is L j L_j L j -Lipschitz between the required pairs n j , r j n_j,r_j n j , r j , and define
f j = ∥ F j , ρ j ( n j ) ( n j ) − F j , a j ∘ ( n j ) ∥ , f_j = \lVert F_{j,\rho_j(n_j)}(n_j) - F_{j,a_j^\circ}(n_j)\rVert, f j = ∥ F j , ρ j ( n j ) ( n j ) − F j , a j ∘ ( n j )∥ , D H = ∑ j = 0 H − 1 ( ∏ k = j + 1 H − 1 L k ) f j . (282) D_H = \sum_{j=0}^{H-1}\left(\prod_{k=j+1}^{H-1} L_k\right) f_j. \tag*{(282)} D H = j = 0 ∑ H − 1 k = j + 1 ∏ H − 1 L k f j . (282) Then ∥ n H − r H ∥ ≤ D H \lVert n_H-r_H\rVert\le D_H ∥ n H − r H ∥ ≤ D H . If also the terminal loss is L ℓ L_\ell L ℓ -Lipschitz in this norm, then
∣ R n a t − R r e p ∣ ≤ E min { M 1 Ω c , L ℓ D H } . (283) \lvert R^{\mathrm{nat}}-R^{\mathrm{rep}}\rvert\le\mathbb{E}\min\{M1_{\Omega^c},L_\ell D_H\}. \tag*{(283)} ∣ R nat − R rep ∣ ≤ E min { M 1 Ω c , L ℓ D H } . (283) Path-dependent finite L j L_j L j are allowed, but their products remain inside expectation. The Lipschitz condition is on branch extensions over the compared states, not an assumption that the discontinuous natural router is Lipschitz.
Proof. Before the first replay mismatch, induction gives n j = r j n_j=r_j n j = r j : the two rules choose a j ∘ a_j^\circ a j ∘ at the same state and apply the identical branch, including its current-state gain. If the first mismatch occurs, the equal pre-event states make it a mismatch in the natural run as well. If none occurs, equality persists to the endpoint. Losses therefore differ only on Ω c \Omega^c Ω c , and their pointwise absolute difference is at most M 1 Ω c M1_{\Omega^c} M 1 Ω c . Taking expectations proves Equation (281) . For sharpness, use one event and a Bernoulli input of probability p p p : the natural rule selects branch 1 on that input and branch 0 otherwise, replay always selects 0, and branch outputs/losses are respectively M M M and zero. The risk difference is M p Mp M p . For the functional statement, insert F j , a j ∘ ( n j ) F_{j,a_j^\circ}(n_j) F j , a j ∘ ( n j ) to get ∥ n j + 1 − r j + 1 ∥ ≤ f j + L j ∥ n j − r j ∥ \lVert n_{j+1}-r_{j+1}\rVert\le f_j+L_j\lVert n_j-r_j\rVert ∥ n j + 1 − r j + 1 ∥ ≤ f j + L j ∥ n j − r j ∥ . Unroll from equal initial states. Both this norm bound times L ℓ L_\ell L ℓ and the bounded-loss indicator bound apply pointwise; take their minimum and expectation.
Expert-output and gain contributions. At a common state in Equation (279) , write e = ρ j ( s ) e=\rho_j(s) e = ρ j ( s ) , a = a j ∘ a=a_j^\circ a = a j ∘ , suppress j , s j,s j , s , and insert p e v a p_e v_a p e v a to obtain
f j ≤ p e ∥ v e − v a ∥ + ∥ p e − p a ∥ ∥ v a ∥ . (284) f_j \le p_e\lVert v_e-v_a\rVert+\lVert p_e-p_a\rVert\lVert v_a\rVert. \tag*{(284)} f j ≤ p e ∥ v e − v a ∥ + ∥ p e − p a ∥ ∥ v a ∥ . (284) The reverse insertion gives p a ∥ v e − v a ∥ + ∥ p e − p a ∥ ∥ v e ∥ p_a\lVert v_e-v_a\rVert+\lVert p_e-p_a\rVert\lVert v_e\rVert p a ∥ v e − v a ∥ + ∥ p e − p a ∥ ∥ v e ∥ ; either is valid. For rowwise routing and Frobenius state norm, combine per-row upper bounds by the square root of their squared sum. Shared paths cancel at this common state. Equal expert outputs alone need not imply equal branch outputs: with v 1 = v 2 = 1 v_1=v_2=1 v 1 = v 2 = 1 , probabilities ( 0.8 , 0.2 ) (0.8,0.2) ( 0.8 , 0.2 ) and prescribed expert 2, the discrepancy is 0.6 despite zero expert-output diameter. The selected masses differ. Thus the equal-mass diameter lemma cannot be used here without a null atom or an explicit gain term. By contrast, if the prescribed expert equals the natural expert at the same state, both support and recomputed gain agree, and f j = 0 f_j=0 f j = 0 .
A first-disagreement margin bound on the correct states. For rowwise top-one routing, sum only over routed events. Let reference logits z j ∘ z_j^\circ z j ∘ select a j ∘ a_j^\circ a j ∘ with winner gap γ j ≥ 0 \gamma_j \ge0 γ j ≥ 0 (minimum over tokens), and put U j = ∥ z j ( r j ) − z j ∘ ∥ ∞ U_j=\lVert z_j(r_j)-z_j^\circ\rVert_\infty U j = ∥ z j ( r j ) − z j ∘ ∥ ∞ over all tokens and coordinates. With the same tie rule,
p d i f f ≤ min { 1 , ∑ j P ( τ ≥ j , γ j ≤ 2 U j ) } p_{\mathrm{diff}} \le\min\left\{1,\sum_j \mathbb{P}(\tau\ge j,\ \gamma_j \le2U_j)\right\} p diff ≤ min { 1 , j ∑ P ( τ ≥ j , γ j ≤ 2 U j ) } ≤ min { 1 , ∑ j inf ν > 0 [ P ( γ j ≤ 2 ν ) + E U j ν ν ν ] } , ν > 0. (285) \le\min\left\{1,\sum_j\inf_{\nu>0}\left[\mathbb{P}(\gamma_j\le2\nu)+\frac{\mathbb{E}U_j^\nu}{\nu^\nu}\right]\right\},\qquad\nu>0. \tag*{(285)} ≤ min { 1 , j ∑ ν > 0 inf [ P ( γ j ≤ 2 ν ) + ν ν E U j ν ] } , ν > 0. (285)
For the first line, a first mismatch implies the displayed margin event by Lemma 27; sum over its disjoint possible locations. For the second line, drop { τ ≥ j } \{\tau\ge j\} { τ ≥ j } , split each margin event at U j ≤ v U_j \le v U j ≤ v , and apply Markov to U j > v U_j > v U j > v . No independence across events or between margin and perturbation is needed. If the marginal CDF satisfies a power bound, Theorem 30 gives its sharper constant instead. Reference inputs with ties remain in the bound; they are not assumed to have zero mass. The moments and margins here are those of support-replay states. For the down-only specialization, Equation (253) is an alternative sufficient condition for p d i f f = 0 p_{\mathrm{diff}} = 0 p diff = 0 under its hypotheses.
Transferring replay-risk comparisons to natural execution. For two models evaluated on the same target, loss and input population, let b m b_m b m be a valid right-hand side of Equation (281) or Equation (283) . Then
R s e p n a t − R r e u s e n a t ≥ R s e p r e p − R r e u s e r e p − b s e p − b r e u s e . (286) R_{\mathrm{sep}}^{\mathrm{nat}} - R_{\mathrm{reuse}}^{\mathrm{nat}} \ge R_{\mathrm{sep}}^{\mathrm{rep}} - R_{\mathrm{reuse}}^{\mathrm{rep}} - b_{\mathrm{sep}} - b_{\mathrm{reuse}}. \tag*{(286)} R sep nat − R reuse nat ≥ R sep rep − R reuse rep − b sep − b reuse . (286)
Consequently a replay-risk gap exceeding the two penalties certifies the corresponding natural-risk ordering. With only replay-risk intervals, use its separate-model lower endpoint minus its reuse-model upper endpoint. Two upper bounds alone do not yield this conclusion. Each model may have its own prescribed sequence and penalty. The fixed-G G G or fixed-selection assumptions of Theorems 46 and 52 remain necessary for their exact quadratics.
Bounded squared loss for normalized fixed-prefix outputs. If the terminal state is the final normalized logits, the frozen head and normalizer in Section O.2 give a common bound ∥ s ∥ ≤ B \lVert s\rVert\le B ∥ s ∥ ≤ B on natural and replay outputs for a bounded prefix length. For a common target with ∥ Y ∥ ≤ B Y \lVert Y\rVert\le B_Y ∥ Y ∥ ≤ B Y , the unclipped square loss ℓ ( s , Y ) = ∥ s − Y ∥ 2 \ell(s,Y) = \lVert s - Y\rVert^2 ℓ ( s , Y ) = ∥ s − Y ∥ 2 obeys M = ( B + B Y ) 2 M = (B + B_Y)^2 M = ( B + B Y ) 2 and L ℓ = 2 ( B + B Y ) L_\ell= 2(B + B_Y) L ℓ = 2 ( B + B Y ) on this output set. Indeed, ∣ ∥ s − Y ∥ 2 − ∥ s ′ − Y ∥ 2 ∣ ≤ ( ∥ s − Y ∥ + ∥ s ′ − Y ∥ ) ∥ s − s ′ ∥ \left|\lVert s - Y\rVert^2 - \lVert s' - Y\rVert^2\right| \le(\lVert s - Y\rVert+ \lVert s' - Y\rVert)\lVert s - s'\rVert ∥ s − Y ∥ 2 − ∥ s ′ − Y ∥ 2 ≤ (∥ s − Y ∥ + ∥ s ′ − Y ∥) ∥ s − s ′ ∥ . Thus the bridge can apply to actual bounded-output square risk; it is not limited to an artificially clipped loss. This conclusion requires the stated output/target bounds.
Frozen-gain replay pays an additional functional term
The preparation code’s force/gain_reference mechanism fixes both the chosen index and the supplied gain at a forced position. The implementation uses frozen-gain replay, defined below. For a top-one event, fix a reference gain q j ∈ [ 0 , 1 ] q_j \in[0,1] q j ∈ [ 0 , 1 ] and define
F ‾ j , a ∘ ( s ) = B j ( s ) + q j v j , a ∘ ( s ) , h 0 = s 0 , h j + 1 = F ‾ j , a ∘ ( h j ) . \overline{F}_{j,a^\circ}(s) = B_j(s) + q_j v_{j,a^\circ}(s), \qquad h_0 = s_0,\qquad h_{j+1} = \overline{F}_{j,a^\circ}(h_j). F j , a ∘ ( s ) = B j ( s ) + q j v j , a ∘ ( s ) , h 0 = s 0 , h j + 1 = F j , a ∘ ( h j ) .
For rowwise routing, q j q_j q j is the vector of prescribed gains. For route-free events let F ‾ j = F j \overline{F}_j = F_j F j = F j .
Proposition 60. (A gain-aware bridge for complete-gate replay). Suppose the current-gain prescribed branch F j , a j ∘ F_{j,a_j^\circ} F j , a j ∘ is L j L_j L j -Lipschitz between r j , h j r_j,h_j r j , h j , and define
c j = ∥ F j , a j ∘ ( h j ) − F ‾ j , a j ∘ ( h j ) ∥ , V H = ∑ j ( ∏ k = j + 1 H − 1 L k ) c j . (287) c_j = \lVert F_{j,a_j^\circ}(h_j) - \overline{F}_{j,a_j^\circ}(h_j)\rVert,\qquad V_H = \sum_j \left(\prod_{k=j+1}^{H-1} L_k\right)c_j. \tag*{(287)} c j = ∥ F j , a j ∘ ( h j ) − F j , a j ∘ ( h j )∥ , V H = j ∑ k = j + 1 ∏ H − 1 L k c j . (287) Then ∥ r H − h H ∥ ≤ V H \lVert r_H - h_H\rVert\le V_H ∥ r H − h H ∥ ≤ V H . In the single-row form, c j = ∣ p j , a j ∘ ( h j ) − q j ∣ ∥ v j , a j ∘ ( h j ) ∥ c_j = \lvert p_{j,a_j^\circ}(h_j) - q_j\rvert\lVert v_{j,a_j^\circ}(h_j)\rVert c j = ∣ p j , a j ∘ ( h j ) − q j ∣ ∥ v j , a j ∘ ( h j )∥ . For a common loss in [ 0 , M ] [0,M] [ 0 , M ] that is also L ℓ L_\ell L ℓ -Lipschitz,
∣ R n a t − R f r o z e n ∣ ≤ M p d i f f + E [ 1 Ω min { M , L ℓ V H } ] . (288) \left|R^{\mathrm{nat}} - R^{\mathrm{frozen}}\right| \le M p_{\mathrm{diff}} + \mathbb{E}\left[1_{\Omega}\min\{M,L_\ell V_H\}\right]. \tag*{(288)} R nat − R frozen ≤ M p diff + E [ 1 Ω min { M , L ℓ V H } ] . (288) Here Ω \Omega Ω and p d i f f p_{\mathrm{diff}} p diff are the support-replay event and probability in Theorem 59 , not agreement measured on h h h . No independent-gain or independent-state assumption is used. If L j L_j L j are random, all products stay inside the expectation.
Proof. Insert F j , a j ∘ ( h j ) F_{j,a_j^\circ}(h_j) F j , a j ∘ ( h j ) to obtain ∥ r j + 1 − h j + 1 ∥ ≤ L j ∥ r j − h j ∥ + c j \lVert r_{j+1} - h_{j+1}\rVert\le L_j\lVert r_j - h_j\rVert+ c_j ∥ r j + 1 − h j + 1 ∥ ≤ L j ∥ r j − h j ∥ + c j and unroll. The formula for c j c_j c j follows by canceling the same-state shared path and expert output. On Ω \Omega Ω , n H = r H n_H = r_H n H = r H , so the natural/frozen loss difference is bounded by min { M , L ℓ V H } \min\{M,L_\ell V_H\} min { M , L ℓ V H } . On its complement it is at most M M M . Splitting the expectation proves the claim. In particular, the right-hand side never exceeds M M M ; it does not factor a gain error from a dependent event. □ \square □
Counterexample to an index-only bound for frozen gain. Let σ ( x ) = ( 1 + e − x ) − 1 \sigma(x) = (1 + e^{-x})^{-1} σ ( x ) = ( 1 + e − x ) − 1 and start from s 0 = 1 s_0 = 1 s 0 = 1 . At the first residual event the winning support is always 1, with constant gain g 0 = σ ( 1 ) g_0 = \sigma(1) g 0 = σ ( 1 ) , and F 0 ( s ; D ) = s + g 0 D F_0(s; D) = s + g_0D F 0 ( s ; D ) = s + g 0 D for a constant expert feature. The reference has D = 0 D = 0 D = 0 ; the compared model changes only this output weight to D = 1 / g 0 D = 1/g_0 D = 1/ g 0 , reaching s 1 = 2 s_1 = 2 s 1 = 2 instead of the reference state 1. At the second event use logits ( s , 0 ) (s, 0) ( s , 0 ) , expert-1 output 1, and residual map s ↦ s + σ ( s ) s \mapsto s+\sigma(s) s ↦ s + σ ( s ) . Every reference, natural, support-replay and frozen-gain run selects expert 1 throughout. Nevertheless the natural/support-replay endpoint is 2 + σ ( 2 ) 2 + \sigma(2) 2 + σ ( 2 ) , while reference-gain replay ends at 2 + σ ( 1 ) 2 + \sigma(1) 2 + σ ( 1 ) . The common bounded Lipschitz loss ℓ ( s ) = min { 1 , max { 0 , s / 4 } } \ell(s) = \min\{1, \max\{0, s/4\}\} ℓ ( s ) = min { 1 , max { 0 , s /4 }} gives risk difference ( σ ( 2 ) − σ ( 1 ) ) / 4 > 0 (\sigma(2) - \sigma(1))/4 > 0 ( σ ( 2 ) − σ ( 1 )) /4 > 0 although p d i f f = 0 p_{\mathrm{diff}} = 0 p diff = 0 . This two-event construction has the selected-softmax-gain semantics of Equation (279) . It disproves the index-only extension.
Conditions for applying the replay bounds. Without bounded loss, a mismatch probability p p p does not bound squared risk: one event may output p − 1 p^{-1} p − 1 only on the mismatching event, giving squared risk p − 1 p^{-1} p − 1 . Conversely, bounded but discontinuous classification loss need not be Lipschitz: outputs − ϵ -\epsilon − ϵ and + ϵ +\epsilon + ϵ can have loss difference one for every ϵ > 0 \epsilon> 0 ϵ > 0 . Thus the gain-to-loss refinement requires its Lipschitz premise or a separate terminal decision-margin argument; boundedness alone still gives the support-replay result. The normalized fixed-prefix branch constants in Proposition 51 can supply the stated branch Lipschitz bounds at a specified total update radius, without freezing attention activations or selected gains. Extra adaptive decisions must be included in the event support/state before invoking these results.
Assumptions and Theorem Dependencies
This table summarizes the dependencies of the existing results. Each alternative retains its own assumptions.
Table 34.
Table 35.
Interpreting diagnostics and applying a result
Table 36.