betaFeedback

WorksComputer Science

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

Abstract

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a 1.30-fold improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.

Introduction

Residual connections [14] are the backbone of modern Transformers. A standard PreNorm layer reads from a normalized copy of the current hidden state, computes an update, and adds that back to the original state [47, 54]. Repeating this operation yields a simple yet effective transport rule: one dd-dimensional state is carried through depth along an identity path, while every attention and feed-forward module contributes an additive update. This design makes deep optimization practical, but it also ties the width of the state carried through depth to the width of the working space of every layer.

Recent residual architectures relax this constraint in different ways. Alternating Updates (AltUp) maintains multiple residual streams, with each layer computing one stream and predicting the remainder [4]. Hyper-Connections (HC) likewise uses multiple streams but learns how layers read from, write to, and combine them with lightweight projectors [59]; Virtual Width Networks (VWN) generalizes this formulation by splitting each stream into finer granularity [38]; Manifold-Constrained Hyper-Connections (mHC) stabilizes HC by constraining its residual mixing operators to a doubly stochastic manifold [53]. Attention Residuals instead retrieves representations from earlier layers using attention over depth [20]. These methods introduce different residual operators with different update rules. Nevertheless, they share two recurring departures from vanilla residual addition: (i) the effective residual state is expanded, and (ii) residual accumulation becomes explicitly regulated, keeping the residual scale in check. This suggests that residual-state capacity is an architectural variable in its own right, rather than a quantity that must always equal model width.

Figure 1. HiRes uses a two-level residual hierarchy. (a) Vanilla PreNorm carries one dd-dimensional residual stream. (b) HiRes carries a DD-dimensional PostNorm outer residual across layer blocks. (c) Each block reads D→dD \to d, runs 2L/B2L/B vanilla sublayers on the dd-dimensional inner residual, and writes back d→Dd \to D. Gray and navy paths have width dd and DD; orange trapezoids are the added dense interfaces, and dashed frames mark repeated residual units.

However, their per-layer expanded states and intricate mixing add activation traffic and memory-bound residual operators, typically requiring fusion, recomputation, overlap, or specialized kernels to recover throughput [53]. If the benefit stems from the wider state itself rather than particular operators, a natural question arises: can Transformers benefit from a wider residual state with as little architectural change as possible?

To this end, we introduce Hierarchical Residuals (HiRes). HiRes groups adjacent Transformer layers into layer blocks and organizes residual computation at two levels. Inside each block, the model runs ordinary dd-dimensional PreNorm Transformer layers. Across blocks, it maintains a DD-dimensional outer residual state with the familiar additive update (D>dD > d). A read projection maps the outer state into a block, and after the inner residual accumulation, a write projection maps the resulting inner state back. The block write is added to the outer residual and the sum is then post-normalized, which regulates the residual scale. Thus, the outer recurrence retains the familiar PostNorm residual, while the inner Transformer layers are unchanged. In this way, the residual expansion ratio D/dD/d can be selected independently of the compute width dd for each inner layer. Because read/write operations occur only at block boundaries, the additional parameters and compute costs are amortized over depth. Figure 1 places the two levels side by side: the outer recurrence is the residual motif of a vanilla PostNorm Transformer, with a whole layer block in place of a sublayer.

Method

Empirically, HiRes consistently outperforms vanilla residual addition across model sizes and depths. Under fitted loss-compute scaling laws, it is 1.30× more FLOP-efficient. As depth scales up at fixed width, HiRes continues to improve through 128 layers where the baseline exhibits diminishing returns. Against HyperConnections, mHC, V!WN, and Attention Residuals, HiRes remains comparable in quality at lower activation memory and I/O costs. On OLMoE-1B-7B [30] pretraining for 500B tokens (§ 4.5), HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. Overall, HiRes provides a minimalist way to expand the residual state without widening per-layer compute. It shows that decoupling residual width from compute width is beneficial, and that this decoupling is available with standard operators.

Method

Vanilla Residual Addition

Consider a decoder-only Transformer with LL layers and hidden width dd. Each layer contains an attention module and an FFN module, giving 2L2L residual sublayers. Let xi−1∈Rdx_{i-1} \in\mathbb{R}^{d} denote the residual state before sublayer ii. A standard PreNorm Transformer can be written as

xi=xi−1+Fi(RMSNorm⁡i(xi−1)),i=1,…,2L,(1)x_i = x_{i-1} + F_i(\operatorname{RMSNorm}_i(x_{i-1})), \qquad i = 1,\ldots,2L, \tag*{(1)}

where Fi:Rd→RdF_i:\mathbb{R}^{d} \to\mathbb{R}^{d} is an attention or FFN transformation. Thus the state carried through depth (residual width, dimension of xix_i) and the state processed by every sublayer (compute width, input and output dimension of FiF_i) share the same width dd.

Decoupling Residual Width and Compute Width

To decouple residual width and compute width, we introduce a wider residual stream hi∈RDh_i \in\mathbb{R}^{D} with D≥dD \ge d and n:=Ddn := \frac{D}{d} denotes the residual expansion ratio. Since every FiF_i keeps operating at compute width dd, there needs a read map Ri:RD→RdR_i:\mathbb{R}^{D} \to\mathbb{R}^{d} to project the carried state into the working space, and a write map Wi:Rd→RDW_i:\mathbb{R}^{d} \to\mathbb{R}^{D} to return the sublayer output to the residual space. So the vanilla residual in Equation (1) becomes

xi=Rihi−1,hi=hi−1+WiFi(RMSNorm⁡i(xi)),i=1,…,2L.(2)x_i = R_i h_{i-1}, \qquad h_i = h_{i-1} + W_i F_i(\operatorname{RMSNorm}_i(x_i)), \qquad i = 1,\ldots,2L. \tag*{(2)}

Setting D=dD = d and Ri=Wi=IR_i = W_i = I recovers the vanilla recurrence.

Residual-expansion methods typically introduce lightweight interfaces by sharing mixing weights across streams or coordinate groups. HiRes instead adopts the simplest choice: dense projections Ri∈Rd×DR_i \in\mathbb{R}^{d \times D} and Wi∈RD×dW_i \in\mathbb{R}^{D \times d}. At fixed DD and dd these are the most general linear interfaces, imposing no structural inductive biases like weight sharing, providing full connectivity between the expanded residual state and the working space. Empirically, HiRes achieves competitive performance at a smaller expansion ratio than methods with structured interfaces, while incurring lower activation memory and I/O costs (§ 4.4).

Hierarchical Residual Recurrence

Applying dense interfaces at every sublayer would add Θ(LDd)\Theta(LDd) parameters and FLOPs, which is comparable to the Θ(Ld2)\Theta(Ld^2) cost of the Transformer layers themselves and thus unacceptable. HiRes therefore invokes them only at block boundaries, yielding a hierarchical structure.

HiRes partitions the LL layers into BB consecutive blocks B1,…,BB\mathcal{B}_1,\ldots,\mathcal{B}_B. Block bb contains ℓb\ell_b layers, where ∑b=1Bℓb=L\sum_{b=1}^{B}\ell_b = L. For simplicity, we use equal-sized contiguous blocks with ℓb=L/B\ell_b = L/B.

Inside each block bb, HiRes keeps the inner PreNorm recurrence unchanged as the vanilla residual in Equation (1):

xb,i=xb,i−1+Fb,i(RMSNorm⁡b,i(xb,i−1)),i=1,…,2ℓb.(3)x_{b,i} = x_{b,i-1} + F_{b,i}(\operatorname{RMSNorm}_{b,i}(x_{b,i-1})), \qquad i = 1,\ldots,2\ell_b. \tag*{(3)}

We write Φb\Phi_b for the composition of these recurrences, so that xb,2ℓb=Φb(xb,0)x_{b,2\ell_b} = \Phi_b(x_{b,0}).

Between blocks, HiRes introduces an outer residual and normalizes it in PostNorm style, transforming the layerwise recurrence in Equation (2) into a blockwise one:

xb,0=Rbhb−1,hb=RMSNorm⁡b(hb−1+WbΦb(xb,0)),b=1,…,B.(4)x_{b,0} = R_b h_{b-1}, \qquad h_b = \operatorname{RMSNorm}_b(h_{b-1} + W_b\Phi_b(x_{b,0})), \qquad b = 1,\ldots,B. \tag*{(4)}

Equations (3) and (4) define HiRes’ two-level residual hierarchy: a dd-dimensional PreNorm inner residual and a DD-dimensional PostNorm outer residual, illustrated in Figure 1. Grouping layers into blocks amortizes the dense-interface overhead across depth, reducing it from Θ(LDd)\Theta(LDd) to Θ(BDd)\Theta(BDd). Layers within each block thus share the same boundary interfaces and dd-dimensional coordinate system: the block pulls a subspace of the outer residual state, transforms it through several layers, and commits the result back.

To connect the embedding and the language modeling head to the outer residual state, we use an embedding write projection E∈RD×dE \in\mathbb{R}^{D \times d} and a head read projection H∈Rd×DH \in\mathbb{R}^{d \times D},

h0=E Embed⁡(t),xfinal=HhB,(5)h_0 = E\,\operatorname{Embed}(t), \qquad x_{\mathrm{final}} = H h_B, \tag*{(5)}

where tt is the input token. Appendix § 12 details the exact parameter, FLOP, memory, and I/O overheads, and Appendix Algorithm 1 provides PyTorch-style pseudocode for the forward pass.

Design Choices

Normalization at block boundaries. Boundary normalization provides residual regulation in HiRes: without it, the outer residual stream would be unbounded as it accumulates block outputs. Keeping PreNorm within blocks and applying PostNorm between them combines PreNorm’s optimization stability during the backward pass [50, 54] with PostNorm’s periodic control of the outer residual scale during the forward pass [19].

Setting D=dD = d and Rb=Wb=IR_b = W_b = I in Equation (4) and removing the outer residual accumulation yields hierarchical normalization (HiNorm):

xb+1,0=RMSNorm⁡b(Φb(xb,0)),b=1,…,B,(6)x_{b+1,0} = \operatorname{RMSNorm}_b\left(\Phi_b(x_{b,0})\right), \qquad b = 1,\ldots,B, \tag*{(6)}

which inserts one RMSNorm into the residual stream every L/BL/B layers. HiNorm isolates regulation from residual expansion and serves as a control in our experiments.

HiNorm alone beats the baseline and reaches minimum loss at an intermediate B=4B = 4 (Appendix Figure 6a). In Appendix Figure 6b, the residual scale grows monotonically in the vanilla baseline and in HiRes without boundary normalization. HC, VWN, and AttnRes [20, 38, 59] regulate the scale implicitly through residual mixing learned at every layer, whereas HiRes regulates it explicitly with boundary normalizations: the scale grows only within each block and is reset at every boundary.

Interface stabilization. We empirically observe that randomly initialized interfaces degrade performance early in training (Appendix § 11.§ 3.2), which we hypothesize arises from a seesaw dynamic: the interfaces learn each block’s working coordinate system while the inner layers simultaneously learn to transform representations within that evolving space. Motivated by prior identity-biased residual gates [3, 46], we stabilize the interfaces by interpolating between identity and learned mappings using learnable gating scalars:

Rb=(1−σ(αbR))Π+σ(αbR)R~b,Wb=(1−σ(αbW))ΠT+σ(αbW)W~b,(7)R_b = \left(1-\sigma(\alpha_b^R)\right)\Pi+ \sigma(\alpha_b^R)\widetilde{R}_b,\qquad W_b = \left(1-\sigma(\alpha_b^W)\right)\Pi^{\mathsf{T}} + \sigma(\alpha_b^W)\widetilde{W}_b, \tag*{(7)}

where Π=[Id×d,0d×(D−d)]∈Rd×D\Pi= [I_{d \times d}, 0_{d \times(D-d)}] \in\mathbb{R}^{d \times D} truncates the outer state to its first dd coordinates, ΠT\Pi^{\mathsf{T}} zero-pads back to width DD, and σ\sigma is the logistic sigmoid. Each interface has an independent learned scalar logit initialized such that σ(α)\sigma(\alpha) is close to 0 (we set it to 0.1 in the experiments).

This parameterization initializes the interfaces near identity, providing the inner layers with a relatively stable coordinate system early in training. The interfaces can then gradually specialize and make increasing use of the expanded residual state parameterized by R~b\widetilde{R}_b and W~b\widetilde{W}_b.

Experiments

Setup

We conduct the primary architecture study using the OLMoE codebase [30]. All models use RMSNorm [56], RoPE [44], grouped-query attention (GQA) [1], and a SwiGLU feed-forward network [39]. We train with AdamW [24] and a cosine learning-rate schedule with warmup, use a 4096-token context, and initialize weights with standard deviation 0.02.

The default HiRes configuration uses residual expansion ratio n=D/d=2n = D/d = 2 and block count B=4B = 4. We ablate both choices in § 5.1.

For architecture comparisons, we average training loss over the final 1000 steps. Validation loss and downstream accuracy are averaged over the final 3 evaluations, 1000 steps apart, and across 11 held-out domains and 12 tasks, respectively. The full experimental setup is detailed in Appendix § 9.

Fitted scaling laws and compute multiplier at matched loss

Figure 2. HiRes achieves better scaling properties. (a) Validation loss versus training FLOPs with fitted laws. At L=3.00\mathcal{L}=3.00 (inset), the baseline needs 1.30× the FLOPs of HiRes with n=2n=2. (b) Compute multiplier at matched loss: baseline FLOPs over family FLOPs, at runs (markers) and along the fitted laws (lines).

Scaling on Model Size

We evaluate model-size scaling over five model sizes, jointly increasing inner width and depth from (d,L)=(512,8)(d,L)=(512,8) to (1536,24)(1536,24) with the parameter count spanning from 70M to 626M. The detailed scaling ladder can be found in Table 5. We compare six fixed architecture families: the vanilla residual baseline; HiRes with n=2n=2, n=3n=3, and n=4n=4; and two reduced forms of HiRes, namely HiNorm, which keeps only the boundary normalization, and equal-width HiRes with n=1n=1, which adds the interfaces and the outer residual without widening. All HiRes variants use B=4B=4.

(d,L)(d, L)TokensMetricBaselineHiNormn=1n = 1n=2n = 2n=3n = 3n=4n = 4
(512,8)(512, 8)30BParams70.264M70.266M72.888M75.511M––
FLOPs (101910^{19})1.86911.86911.91631.9635––
Train loss3.47683.46173.45153.4314––
Val. loss3.47333.45643.44013.4201––
(768,12)(768, 12)50BParams138.433M138.436M144.334M150.235M156.136M162.037M
FLOPs (101910^{19})6.41866.41876.59576.77276.94977.1267
Train loss3.22283.19563.18693.16453.15863.1591
Val. loss3.19413.17143.15733.13003.12423.1251
(1024,16)(1024, 16)100BParams245.403M245.407M255.892M266.382M276.872M287.362M
FLOPs (101910^{19})22.78022.78023.40924.03924.66925.299
Train loss3.01082.98272.97582.96632.95892.9539
Val. loss2.97342.94662.94012.92712.92192.9153
(1280,20)(1280, 20)200BParams403.757M403.762M420.146M436.535M452.924M469.313M
FLOPs (101910^{19})73.62373.62375.59077.55679.52281.488
Train loss2.85242.83382.82452.80972.80152.8065
Val. loss2.81582.79722.78652.76972.75902.7612
(1536,24)(1536, 24)350BParams626.078M626.084M649.677M673.276M––
FLOPs (101910^{19})194.91194.91199.86204.82––
Train loss2.72952.70842.70192.6907––
Val. loss2.68382.66472.66182.6427––

Table 5. Parameters, training FLOPs, and losses for model size scaling. At every model size, loss falls from the baseline to HiNorm, HiRes n=1n = 1, and HiRes n=2n = 2. HiNorm and HiRes use B=4B = 4.

Since HiRes introduces modest parameter overhead, we compare models by fitting a loss-compute scaling law [16, 18] in analytic training FLOPs: C=T(6P+12Lsd)C=T(6P+12Lsd), where PP is the total parameter count, TT is the number of training tokens, LL is the number of layers, ss is the sequence length, and dd is the inner width. We jointly fit the observations with the scaling law Lf(C)=L∞+Af(C/C0)−αf\mathcal{L}_{f}(C)=\mathcal{L}_{\infty}+A_{f}(C/C_{0})^{-\alpha_{f}}, with a shared irreducible loss and family-specific amplitude and exponent. Here C0=1020C_{0}=10^{20} FLOPs is the reference compute and L∞=2.101\mathcal{L}_{\infty}=2.101 is the fitted irreducible loss.

Figure 2 separates the two sources of the HiRes gain: residual regulation and residual expansion. HiNorm consistently shifts the vanilla residual scaling curve downward. Widening the residual state (n=2n=2) brings a further gain at every scale. Equal-width HiRes (n=1n=1) remains close to HiNorm, indicating that the hierarchy interfaces alone add little benefit. A clearer improvement appears when the outer state expands from D=dD=d to D=2dD=2d, while the n=3n=3 and n=4n=4 trajectories remain close to n=2n=2, indicating diminishing returns. Regulation and widening thus contribute complementary gains. HiRes with n=2n=2 combines both, and the fitted scaling laws estimate that the baseline requires 1.30× the FLOPs of HiRes to reach a validation loss of 3.00.

Scaling on Depth

We next examine depth scaling properties of HiRes by fixing the inner width at d=512d = 512 and the training budget at 30B tokens while increasing the number of Transformer layers from 8 to 128.

Figure 3 plots validation loss against depth on a logarithmic axis, where ideal compute scaling should appear as a straight line. HiRes not only consistently outperforms the vanilla residual baseline at every depth, but also maintains better scaling properties. The baseline bends away from its scaling curve at the last doubling from L=64L = 64 to L=128L = 128. With B=4B = 4, HiRes gives up less of its per-doubling reduction than the baseline, especially at n=2n = 2, indicating that regulating residual accumulation and widening the residual state can hold more layers. Raising the block count to B=8B = 8 further preserves the scaling close to a straight line through L=128L = 128, suggesting more blocks for deeper models.

Figure 3. HiRes scales better with depth. HiRes wins at every depth and bends less at L=128L = 128; B=8B = 8 stays nearly straight.

Comparison with Other Residual Expansion Methods

We compare HiRes with other residual expansion methods with the recipe of (d,L)=(768,12)(d, L) = (768, 12) for 50B-token training, namely Hyper-Connections (HC) [59], mHC [53], Virtual Width Networks (VWN) [38], and Attention Residuals [20]. Specifically, we test 2×2\times and 4×4\times wider residual states for HC/mHC/VWN, full and 4-block AttnRes, and n=2n = 2, B=4B = 4 HiRes. We then test mHC-4, VWN-4, Block AttnRes-4, and HiRes configurations on larger backbones. Appendix § 10.§ 4 gives the detailed implementations and complete results.

At (d,L)=(768,12)(d, L) = (768, 12), all residual-expansion methods improve the vanilla residual baseline in loss, and HiRes achieves comparable performance to mHC, VWN, and AttnRes at a lower residual expansion rate, as shown in Table 1.

Table 1 HiRes versus other residual-expansion methods at (d,L)=(768,12)(d, L) = (768, 12) and 50B tokens. Suffixes give the expansion rate for HC/mHC/VWN and the block count for AttnRes. With a 2×2\times wider residual state, HiRes is competitive even with the 4×4\times variants of HC, mHC, and VWN. Bold/underlined: best/second-best among residual-expansion methods. Appendix C.3 details the configurations and lists their parameters and training FLOPs.

MetricBaselineHC-2HC-4mHC-2mHC-4VWN-2VWN-4Full AttnResBlock AttnRes-4HiRes
Train loss3.22283.17733.18703.19413.16253.15843.15783.19383.19433.1645
Val. loss3.19413.16493.15333.18713.14593.14003.12283.15663.16493.1300
Avg. Acc. (%)46.9147.2546.2646.6647.3347.0345.8646.1746.1347.15

Table 1.

Table 2 compares the vanilla residual baseline, mHC, VWN, Block AttnRes, and HiRes across three backbone scales: (d,L)=(768,12)(d, L) = (768, 12), (1024,16)(1024, 16), (1280,20)(1280, 20). HiRes remains within the leading quality range of these recent residual-expansion methods. The same table reports the analytic I/O at each scale (Appendix § 12), together with throughput and memory from a unified measurement of each method’s highly optimized Triton implementation (Appendix § 13). HiRes has the lowest activation I/O among residual-expansion methods and closest to the baseline’s 3d. Despite additional parameters, HiRes retains 91.7–93.1% of baseline throughput at 3.3–3.4% memory overhead, which is on par with optimized AttnRes and more efficient than mHC/VWN. Moreover, HiRes’ standard PyTorch implementation already reaches near-final performance, while other methods rely on mechanism-specific kernel fusion. Appendix § 13 reports the agentic optimization campaign and the latency decomposition behind these numbers.

BackboneMetricBaselinemHC-4VWN-4BlockAttnRes-4HiRes
(768, 12)Train loss3.22283.16253.15783.19433.1645
Val. loss3.19413.14593.12283.16493.1300
Avg. Acc. (%)46.9147.3345.8646.1347.15
Activation I/O3d3d17d17d17d17d5.25d5.25d4.58d4.58d
Peak memory (GiB)16.3119.7120.8917.1816.86
Throughput (M tok/s)0.6820.5900.5560.6390.625
(1024, 16)Train loss3.01082.96332.96782.99122.9663
Val. loss2.97342.92052.92312.95472.9271
Avg. Acc. (%)48.7649.1649.5749.4549.15
Activation I/O3d3d17d17d17d17d5.19d5.19d4.19d4.19d
Peak memory (GiB)21.4927.2828.8422.9622.22
Throughput (M tok/s)0.4190.3700.3540.3980.389
(1280, 20)Train loss2.85242.81572.80682.83662.8097
Val. loss2.81582.76712.75332.78712.7697
Avg. Acc. (%)51.9052.4952.3051.4852.76
Activation I/O3d3d17d17d17d17d5.15d5.15d3.95d3.95d
Peak memory (GiB)28.0736.8938.8530.3328.99
Throughput (M tok/s)0.2790.2400.2240.2630.259

Table 2.

Table 2 HiRes stays competitive across scales. Activation I/O is the analytic count per token per sublayer for optimized implementations (Appendix § 12); peak memory and throughput are measured with each method’s optimized Triton implementation (Appendix § 13), whereas HiRes is nearly as fast with vanilla PyTorch implementation (Appendix Table 13). Bold/underlined: best/second-best among residual-expansion methods.

Long-Horizon MoE Pretraining

To scale up, we evaluate HiRes on the OLMoE-1B-7B pretraining recipe [30], a sparse Mixture-of-Experts model [12, 40]. The model contains 6.9B total parameters and 1.3B active parameters per token, and is trained for 500B tokens. We compare the vanilla residual baseline with HiRes using the n=2n = 2, B=4B = 4 configuration, which adds 83.9M parameters (6.5% of the active parameters and 1.2% of the total parameters); Appendix Table 9 lists the full configuration.

As shown in Figure 4, HiRes achieves 0.0208 lower training loss and 0.0180 lower validation loss than the vanilla residual baseline. It also performs better on most downstream tasks and improves the downstream average accuracy by 1.14 points. These results establish that the HiRes design transfers to long-horizon sparse-MoE pretraining.

Figure 4. HiRes on 500B-token OLMoE-1B-7B pretraining. HiRes ends with 0.0208 lower training loss, 0.0180 lower validation loss, and a 1.14-point higher average accuracy. See Appendix § 10.§ 5 for details.

Ablation Studies

Ablation on Block Count and Expansion Ratio

We ablate the two structural hyperparameters of HiRes: the residual expansion ratio nn and the number of blocks BB. All runs use (d,L)=(768,12)(d, L) = (768, 12) and 50B training tokens. The expansion sweep fixes B=4B = 4 and varies n=D/dn = D/d from 0.75 to 4, while the block sweep fixes n=2n = 2 and varies BB from 1 to 12. Figure 5 reports both sweeps.

Figure 5. Ablation on expansion ratio nn and block count BB in HiRes. Beyond the default n=2n = 2, B=4B = 4 (star), gains roughly follow the shifted scaling law (dashed), which translates the baseline scaling law (dotted) through the star, accounting for the benefit from increasing parameters.

Expansion ratio. In Figure 5a, validation loss rapidly decreases as the expansion ratio increases to n=2n = 2, after which the slope slows. The wider n=3n = 3 and n=4n = 4 settings provide marginal gains, largely following the shifted scaling law, which accounts for the benefit from extra parameters. Thus n=2n = 2 is the empirical knee of the quality-overhead curve: it captures most of the benefit of residual expansion while keeping the interface cost moderate. On the other side, at n=0.75n = 0.75 the outer state is narrower than the layer compute width, and it is the only point in the sweep that fails to improve over HiNorm, even with more parameters. This result identifies residual width, rather than the interfaces themselves, as the main source of the gain.

Number of blocks. In Figure 5b, increasing BB also improves validation loss, with the largest gains occurring as the hierarchy is refined from one to four blocks. Beyond B=4B = 4, more frequent outer-state updates show diminishing returns similar to those of the expansion ratio. We therefore retain B=4B = 4 as the empirical knee.

Architecture Ablations

We subtract one component at a time from the complete n=2n = 2, B=4B = 4 HiRes model, testing boundary normalization, outer residual, and interface learning against the same reference architecture, as shown in Table 3; HiNorm and equal-width HiRes (n=1n = 1) serve as reduced references. All ablations use (d,L)=(768,12)(d, L) = (768, 12) and 50B training tokens.

VariantTrain lossVal. loss
Baseline3.22283.1941
HiNorm3.19563.1714
HiRes n=1n = 1, B=4B = 43.18693.1573
HiRes n=2n = 2, B=4B = 43.16453.1300
– Frozen random R/WR/W3.19053.1614
– w/o boundary RMSNorm3.18743.1552
– w/o outer residual3.18993.1616
– w/o interface stabilization3.16863.1375

Table 3. Ablations on HiRes components. Removing or freezing any one component of HiRes (n=2n = 2, B=4B = 4) hurts.

The ablations support the following conclusions. (i) The interfaces must be learnable. Freezing the read/write projections raises validation loss by 0.031, so widening residual with fixed maps does not work. (ii) Widening helps even without normalization. Removing boundary RMSNorm raises validation loss by 0.025, yet this widening variant still improves on the baseline by 0.039. (iii) Outer residual is necessary. Removing the outer residual raises validation loss by 0.032, because each write-read pair between adjacent blocks then collapses into a single d→D→dd \to D \to d MLP with boundary RMSNorm as its activation, leaving the residual state effectively unexpanded. (iv) Interface stabilization gives a modest final gain. Disabling it raises validation loss by 0.008, a relatively modest gap. Nevertheless, its advantage is much larger early in training (Appendix § 11.§ 3.2), which is the main reason we adopt it.

Related Work

Residual-state expansion. The closest methods expand the residual state, but typically by replacing vanilla addition with a learned, per-layer operator. AltUp widens the representation into sub-blocks and updates one sub-block at backbone width while a lightweight predictor updates the rest [4]. Hyper-Connections (HC) maintains multiple residual streams with learned layer-dependent read, write, and residual mixing weights [59]; Virtual Width Networks (VWN) generalizes HC by constructing fine-grained fractional streams, enabling more expressive mixing [38, 60]; mHC constrains the residual stream transition to a doubly stochastic manifold for stability [53]. Attention Residuals instead learns content-dependent attention over earlier layer or block representations [20]. Relative to this family, HiRes is simpler in mechanism: one wide outer state, dense read/write only at block boundaries, and additive residual transport. It maintains more expressive interfaces between the expanded residual and the fixed-width inner state, and amortizes the cost across depth.

Normalization in Transformers. The original Transformer normalizes after each residual addition (PostNorm), whereas modern language models typically use PreNorm to keep an uninterrupted additive path through depth [2, 31, 47, 54, 56]. There exists a well-known trade-off between PreNorm and PostNorm: PostNorm controls activation scale but can attenuate gradients, while PreNorm eases optimization but lets the residual state grow unbounded [49]. Hybrid placements pursue both, adding normalization inside the residual branches [41], around each sublayer [19], mixed within a sublayer [61], across depth [21], or across coupled PreNorm-like and PostNorm-like streams [22]. HiRes differs in granularity: complete PreNorm layers run inside a block, and PostNorm is applied only after the block-level outer residual addition, preserving ordinary inner updates while periodically resetting the outer-state scale.

Cross-layer connectivity and state aggregation. Prior work enriches information flow across depth through gated shortcuts, identity paths, or dense connectivity [14, 17, 43]. Transformer variants further aggregate earlier representations using static weights, element-wise gates, reused values, or low-rank projections [11, 26, 32, 58], while MUDDFormer mixes decoupled streams and Deep Delta Learning maintains a matrix-valued state [52, 57]. HiRes instead uses hierarchical connectivity, combining a persistent wide outer state across blocks with narrow inner states within each block.

Conclusion

We introduced Hierarchical Residuals (HiRes), a deliberately minimal architecture that decouples residual-state width from per-layer compute width. HiRes carries a wide persistent state across layer blocks, performs ordinary Transformer computation in a narrower inner space, and connects the two through learned read/write projections at block boundaries. This hierarchical design uses only dense linear maps, normalization, and residual addition, requiring neither specialized kernels nor changes to the inner Transformer layers.

Across model sizes and depths, HiRes consistently outperforms the vanilla residual baseline and is competitive with other residual expansion methods including HC, VWN, mHC, and AttnRes. These results demonstrate that residual-state width need not be tied to per-layer compute width and that their conventional equality can be suboptimal. HiRes therefore establishes residual width as a practical scaling variable that can be exploited through a simple yet effective hierarchical design.

Implementation Pseudocode

HiRes can be implemented as a simple extension of a vanilla Transformer by changing fewer than ten lines of code in the model’s forward method, as illustrated in Algorithm 1.

PyTorch-style pseudocode of the HiRes forward pass

Figure 1 (PDF p. 15). PyTorch-style pseudocode of the HiRes forward pass.

def forward(self, input_ids):
    x = self.embed(input_ids)
    x_outer = self.embed_proj(x)  # up-project d -> D
    for b in range(self.num_blocks):
        # Read at block entry, down-project D -> d
        x_inner = self.read_proj[b](x_outer)
        # Standard PreNorm inner residual
        for i in range(self.num_layers // self.num_blocks):
            x_inner = x_inner + self.attn[b][i](self.attn_norm[b][i](x_inner))
            x_inner = x_inner + self.ffn[b][i](self.ffn_norm[b][i](x_inner))
        # Write at block exit, up-project d -> D
        x_outer = self.outer_norm[b](x_outer + self.write_proj[b](x_inner))
    x = self.lm_head_proj(x_outer)  # down-project D -> d
    logits = self.lm_head(self.final_layernorm(x))
    return logits

Experimental Setup

All dense experiments use decoder-only Transformers implemented in the OLMoE codebase [30]. Models use weight-tied input/output embeddings, RMSNorm [56], RoPE [44], SwiGLU [39] with preactivation width dff=4dd_{\mathrm{ff}} = 4d, grouped-query attention [1] with head dimension 64 and 4 KV heads, and per-head QK-norm [9]. Training uses AdamW [24] with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95, ϵ=10−8\epsilon= 10^{-8}, weight decay 0.1, and gradient-norm clipping at 1.0. The learning rate uses cosine decay to 0.1 of the peak rate after a linear warmup over 2% of training tokens. Models use a 4,096-token sequence length and initialization standard deviation 0.02. Table 4 lists the scale-dependent settings.

Configuration (d,L)(d, L)(512,8)(512, 8)(768,12)(768, 12)(1024,16)(1024, 16)(1280,20)(1280, 20)(1536,24)(1536, 24)
Attention heads812162024
KV heads44444
FFN width2,0483,0724,0965,1206,144
Baseline params70.264M138.433M245.403M403.757M626.078M
Global batch size128128256384512
Peak learning rate8.0×10−48.0 \times10^{-4}6.5×10−46.5 \times10^{-4}5.7×10−45.7 \times10^{-4}5.1×10−45.1 \times10^{-4}4.6×10−44.6 \times10^{-4}
Training tokens30B50B100B200B350B

Table 4. Scale-dependent training recipes.

The training corpus is the OLMo2 stage-1 mixture, tokenized with the Dolma2 tokenizer and its 100,352-token vocabulary [48]. All compared architectures at a given scale reuse the same data order and global sample-index array.

Validation loss is averaged over 11 held-out domains from Paloma [25]: C4 [33], Dolma Books, Dolma Common Crawl, Dolma Pes2o, Dolma Reddit, Dolma Stack, Dolma Wiki [42], ICE, M2D2 S2ORC [23, 34], Pile [13], and WikiText-103 [27]. Downstream accuracy is averaged over 12 tasks: MMLU [15], ARC-Challenge and ARC-Easy [7], BoolQ [6], CommonsenseQA [45], COPA [35], HellaSwag [55], OpenBookQA [28], PIQA [5], SciQ [51], SocialIQA [37], and WinoGrande [36].

Detailed Experimental Results

Scaling on Model Size

Following the model-size study in § 4.2, Table 5 provides the complete results at every evaluated scale.

Scaling on Depth

In § 4.3 we study depth scaling at fixed width; Table 6 reports the corresponding results and the follow-up sweep over the number of HiRes blocks.

(d,L)(d, L)MetricBaselinen=1n = 1, B=4B = 4n=1n = 1, B=8B = 8n=2n = 2, B=4B = 4n=2n = 2, B=8B = 8
(512,8)(512, 8)Params70.264M72.888M–75.511M–
FLOPs (101910^{19})1.86911.9163–1.9635–
Train loss3.47683.4515–3.4314–
Val. loss3.47333.4401–3.4201–
(512,16)(512, 16)Params89.148M91.771M–94.395M–
FLOPs (101910^{19})2.81262.8598–2.9071–
Train loss3.33343.2934–3.2774–
Val. loss3.32213.2860–3.2635–
(512,32)(512, 32)Params126.915M129.539M131.638M132.162M136.360M
FLOPs (101910^{19})4.70044.74764.78544.79484.8704
Train loss3.19493.15213.15513.14213.1364
Val. loss3.17903.14073.14213.12853.1182
(512,64)(512, 64)Params202.449M205.073M207.172M207.696M211.895M
FLOPs (101910^{19})8.47598.52328.56098.57048.6459
Train loss3.06953.03233.03183.01853.0162
Val. loss3.05553.02023.02323.00192.9986
(512,128)(512, 128)Params353.518M356.142M358.241M358.765M362.963M
FLOPs (101910^{19})16.02716.07416.11216.12116.197
Train loss2.99182.95362.92582.92322.9141
Val. loss2.97652.93872.91542.91112.9017

Table 6. HiRes beats baseline at every depth, and B=8B = 8 helps most at L=128L = 128. All models use d=512d = 512 and 30B tokens.

Comparison with Other Residual Expansion Methods

Configurations. We detail the configurations compared in § 4.4. HC-mm and mHC-mm maintain mm width-dd residual streams. VWN-mm has the same total virtual width mdmd but uses finer-grained streams: 2m2m streams of width d/2d/2. We evaluate m∈{2,4}m \in\{2, 4\}. Full AttnRes retrieves from all preceding sublayer outputs, whereas Block AttnRes-4 partitions the 2L2L residual sublayers into 4 equal blocks and retains one representation per completed block. HiRes uses expansion ratio n=D/d=2n = D/d = 2 and block count B=4B = 4 throughout. Table 7 lists the parameters and training FLOPs of each configuration.

Method(768,12)(768, 12) Params(768,12)(768, 12) FLOPs(1024,16)(1024, 16) Params(1024,16)(1024, 16) FLOPs(1280,20)(1280, 20) Params(1280,20)(1280, 20) FLOPs
Baseline138.4M6.419245.4M22.78403.8M73.62
HC-2138.5M6.421––––
HC-4138.6M6.423––––
mHC-2138.8M6.429––––
mHC-4140.3M6.474248.7M22.98408.9M74.24
VWN-2140.9M6.492––––
VWN-4143.3M6.564254.0M23.30417.2M75.24
Full AttnRes138.5M6.420––––
Block AttnRes-4138.5M6.420245.5M22.78403.9M73.64
HiRes150.2M6.773266.4M24.04436.5M77.56

Table 7. Parameters and training FLOPs of the compared residual expansion methods. FLOPs are reported in units of 101910^{19}.

Per-task downstream accuracy. Table 8 reports the complete downstream-task breakdown for the four residual-expansion methods transferred across all three backbone scales.

BackboneTaskBaselinemHC-4VWN-4Block AttnRes-4HiRes
(768, 12)MMLU27.8727.24<u>27.28</u>27.4026.59
ARC-Challenge25.9824.7522.97<u>23.63</u>24.75
ARC-Easy50.94<u>51.93</u>50.4152.0551.11
BoolQ59.67<u>55.84</u>51.2756.0655.18
CommonsenseQA32.05<u>34.07</u>31.5931.6134.34
COPA67.00<u>68.00</u>65.0062.3370.67
HellaSwag33.42<u>34.83</u><u>34.83</u>34.2535.26
OpenBookQA30.60<u>29.40</u>29.0730.5329.27
PIQA64.62<u>65.61</u><u>65.61</u>65.5265.96
SciQ76.1081.50<u>78.83</u>77.30<u>78.83</u>
SocialIQA42.73<u>42.84</u>42.3942.1543.09
WinoGrande51.9151.93<u>51.12</u>50.7050.72
Avg. Acc.46.9147.3345.8646.13<u>47.15</u>
(1024, 16)MMLU28.2028.8228.8928.5228.40
ARC-Challenge26.0925.3124.86<u>26.20</u>26.98
ARC-Easy56.4354.56<u>56.43</u>56.9655.73
BoolQ57.2858.28<u>59.94</u>62.0456.38
CommonsenseQA35.5433.58<u>35.05</u>34.2135.93
COPA69.0069.00<u>69.67</u>70.6766.00
HellaSwag40.1842.0242.1440.74<u>42.03</u>
OpenBookQA30.6031.8732.0030.9332.00
PIQA65.92<u>67.99</u>67.2766.4768.61
SciQ82.0382.73<u>82.20</u>81.6081.57
SocialIQA43.06<u>43.52</u>43.5042.5343.72
WinoGrande50.8052.2052.9352.49<u>52.51</u>
Avg. Acc.48.7649.1649.57<u>49.45</u>49.15
(1280, 20)MMLU30.4930.75<u>30.89</u>30.5630.95
ARC-Challenge29.3229.10<u>28.65</u><u>28.65</u>27.65
ARC-Easy59.5958.83<u>61.29</u>58.4862.81
BoolQ58.9260.7157.9658.37<u>59.71</u>
CommonsenseQA37.3737.3735.54<u>38.36</u>39.53
COPA73.3373.0072.0072.00<u>72.67</u>
HellaSwag47.48<u>49.59</u>50.3948.00<u>50.13</u>
OpenBookQA32.2035.47<u>33.20</u>31.7333.07
PIQA70.44<u>70.86</u>70.7869.3371.13
SciQ85.3784.6787.4085.10<u>85.53</u>
SocialIQA43.3643.3644.2343.88<u>44.17</u>
WinoGrande54.9156.1455.2553.33<u>55.75</u>
Avg. Acc.51.90<u>52.49</u>52.3051.4852.76

Table 8. Downstream performance for different residual expansion methods. Best and second-best values are bold and <u>underlined</u>. HiRes remains competitive across model scales.

Across three backbone scales, HiRes improves downstream accuracy over the vanilla baseline by 0.24, 0.39, and 0.86 percentage points from the smallest to the largest model, achieving the highest aggregate accuracy among the compared methods at the largest scale. Individual tasks favor different methods, but HiRes delivers consistent aggregate gains and is competitive with other residual expansion methods.

Long-Horizon MoE Pretraining

For the long-horizon experiment in § 4.5, Table 9 summarizes the MoE configuration and 500B-token results.

SettingValue
Total parameters6.9B
Active parameters1.3B
Transformer layers16
Model width dd2,048
Attention heads16
Context length4,096
MoE experts64
MoE top-kk8
Expert hidden size2,048
Global batch size768
Peak learning rate4.0×10−44.0 \times10^{-4}
Warmup ratio2%
Training tokens500B
HiRes outer width DD4,096
HiRes blocks BB4
HiRes added parameters83.9M
Interface initializationσ(α)=0.1\sigma(\alpha) = 0.1

Table 9. Table 9 On OLMoE-1B-7B, HiRes improves losses and most downstream tasks after 500B tokens. (a) Model and training settings inherited from OLMoE [30], plus the HiRes configuration. (b) Losses and 12-task downstream accuracy (%). Bold: the better of the two. (a) Model and training configuration

MetricBaselineHiRes (n=2n = 2, B=4B = 4)
Train loss2.36632.3455
Val. loss2.34912.3311
MMLU41.5842.19
ARC-Challenge44.7049.61
ARC-Easy76.4378.54
BoolQ66.3970.09
CommonsenseQA49.8849.66
COPA87.6786.67
HellaSwag73.1974.29
OpenBookQA44.3344.80
PIQA78.6579.67
SciQ93.1393.60
SocialIQA47.7848.43
WinoGrande67.1767.01
Avg. Acc.64.2465.38

Table 9. (b) Training and downstream results

Detailed Ablation Results

Ablation on Block Count and Expansion Ratio

Complementing § 5.1, Table 10 reports the expansion-ratio and block-count ablations.

ConfigurationParamsTrain lossVal. loss
Expansion ratio at B=4B = 4
n=0.75n = 0.75142.9M3.20343.1735
n=1n = 1144.3M3.18693.1573
n=1.5n = 1.5147.3M3.17383.1450
n=2n = 2150.2M3.16453.1300
n=2.5n = 2.5153.2M3.16313.1306
n=3n = 3156.1M3.15863.1242
n=4n = 4162.0M3.15913.1251
Block count at n=2n = 2
B=1B = 1143.2M3.21703.1877
B=2B = 2145.5M3.19503.1658
B=3B = 3147.9M3.17473.1426
B=4B = 4150.2M3.16453.1300
B=6B = 6155.0M3.15573.1249
B=12B = 12169.1M3.15403.1172

Table 10. Table 10 Ablation results on expansion ratio nn and block count BB. Loss improves quickly up to n=2n = 2 and B=4B = 4 and slowly beyond. All runs use (d,L)=(768,12)(d, L) = (768, 12) and 50B tokens.

Architecture Ablations

Table 11 extends the architecture ablations in § 5.2 with interface, stabilization, write-rule, and transport variants.

VariantTrain lossVal. loss
Boundary normalization ×\times outer residual
Baseline3.22283.1941
Boundary RMSNorm only (HiNorm)3.19563.1714
Identity R/WR/W + outer residual3.21183.1807
Identity R/WR/W + outer residual + RMSNorm3.19433.1686
Learned interface ×\times scalar stabilization
Learned R/WR/W, stabilized3.16453.1300
Learned R/WR/W, direct3.16863.1375
Frozen random R/WR/W, stabilized3.19053.1614
Frozen random R/WR/W, direct3.21813.1927
Boundary RMSNorm ×\times scalar stabilization
With RMSNorm, stabilized3.16453.1300
Without RMSNorm, stabilized3.18743.1552
With RMSNorm, direct3.16863.1375
Without RMSNorm, direct3.19383.1591
Write rule at n=2n = 2, B=4B = 4
Full-output write3.16453.1300
Increment-only write3.16533.1341
Read/write direction at n=1n = 1, B=4B = 4
Complete read/write interface3.18693.1573
Read-only interface3.19023.1630
Write-only interface3.18973.1614

Table 11. Supplementary interface ablations at (d,L)=(768,12)(d, L) = (768, 12). All models train for 50B tokens.

Boundary RMSNorm and the outer residual each improve over the baseline when introduced separately, and their combination performs best in the identity-interface factorial. Stabilization improves both learned and frozen interfaces, with the larger gain for frozen random maps. Boundary RMSNorm yields a similar improvement with and without stabilization, so its contribution is not contingent on the scalar gates. The increment-only variant replaces the default write WbΦb(xb,0)W_b\Phi_b(x_{b,0}) with Wb[Φb(xb,0)−xb,0]W_b[\Phi_b(x_{b,0}) - x_{b,0}], committing only the update accumulated inside the block, so the outer skip is exactly the identity rather than I+WbRbI + W_bR_b; its result is close to full-output writing. At equal residual and compute width, both one-sided variants underperform the complete interface, indicating that learned transport in both directions is preferable.

Boundary Normalization

Figure 6 shows the boundary-frequency ablation and the per-layer residual scale discussed in § 3.4.

Figure 6. (a) Boundary-frequency ablation. (b) Per-layer residual activation scale.

Figure 6 Boundary normalization lowers loss and regulates residual scale at (d,L)=(768,12)(d, L) = (768, 12). (a) HiNorm training loss under different boundary normalization frequencies. (b) Boundary normalization regulates residual accumulation as in other residual-expansion methods; dotted lines mark the HiRes block boundaries (B=4)(B = 4).

Interface Stabilization Dynamics

The scalar gates in Equation (7) bias each read/write map toward the coordinate-aligned identity at initialization. We compare the default initialization, σ(α)=0.1\sigma(\alpha) = 0.1, with direct dense interfaces. Figure 7 shows that direct interfaces initially train worse than the baseline and only later cross below it. The identity-biased parameterization starts closer to the baseline and remains better throughout the early optimization phase. Stabilization therefore prevents newly initialized interfaces from immediately disrupting residual transport. The final results in Table 11 further show that stabilization is an optimization aid rather than a requirement for the architecture.

Figure 7. Identity-biased initialization stabilizes early interface training. Training loss of the baseline and of HiRes with direct or identity-biased interfaces under (d,L)=(768,12)(d,L)=(768,12). (a) Full training loss trajectories. (b) Loss difference from the baseline. Identity-biased interfaces start near baseline and maintain lower training loss than direct ones.

Parameters, Compute, Memory, and I/O Costs

Parameters, compute, and memory. HiRes leaves all attention and FFN modules unchanged. Its additional parameters come from 2B2B block interfaces, two endpoint projections, BB outer RMSNorm, and 2B+22B+2 stabilization scalars:

ΔP=2BdD⏟Rb,Wb+2dD⏟E,H+BD⏟Norm+(2B+2)⏟scalar gates.(8)\Delta P = \underbrace{2BdD}_{R_b,W_b}+\underbrace{2dD}_{E,H}+\underbrace{BD}_{\mathrm{Norm}}+\underbrace{(2B+2)}_{\text{scalar gates}}. \tag*{(8)}

For the OLMoE-1B-7B model in § 4.5, where (d,L,D,B)=(2048,16,4096,4)(d,L,D,B)=(2048,16,4096,4), this amounts to 83.9M additional parameters, approximately 6.5% of the model’s 1.3B active parameters and 1.2% of its 6.9B total parameters. The dominant additional training compute comes from the forward and backward passes through the 2(B+1)2(B+1) dense interface projections. Over NtrainN_{\mathrm{train}} training tokens, this amounts to

ΔCtrain≈12Ntrain(B+1)dD FLOPs,(9)\Delta C_{\mathrm{train}} \approx12N_{\mathrm{train}}(B+1)dD\ \mathrm{FLOPs}, \tag*{(9)}

excluding the lower-order normalization and scalar-gate operations. Relative to the vanilla residual baseline, HiRes additionally stores only the outer states at the (B+1)(B+1) block boundaries. For a micro-batch of NmbN_{\mathrm{mb}} tokens, this activation-memory overhead is

ΔA=Nmb(B+1)D activation elements.(10)\Delta A=N_{\mathrm{mb}}(B+1)D\ \text{activation elements}. \tag*{(10)}

Activation I/O. Let fl:Rd→Rdf_l:\mathbb{R}^{d}\to\mathbb{R}^{d} denote one sublayer; an LL-layer Transformer has 2L2L sublayers. We count forward activation elements per token, excluding parameter reads and flf_l’s internal traffic. For HiRes, the count includes its two endpoint maps. Following AttnRes [20], an input constructed only for flf_l is passed to it directly rather than written, whereas residual states reused by later merges are written. Table 12 summarizes the result for each method, and the paragraphs below derive each row.

MethodTotal readsTotal writesAverage I/O per sublayerTypical values
Baseline4Ld4Ld2Ld2Ld3d3d3d3d
HC (fused)2L(3m+1)d2L(3m+1)d2Lmd2Lmd(4m+1)d(4m+1)d17d17d
mHC (fused)2L(3m+1)d2L(3m+1)d2Lmd2Lmd(4m+1)d(4m+1)d17d17d
VWN (fused)2L(3m+1)d2L(3m+1)d2Lmd2Lmd(4m+1)d(4m+1)d17d17d
Full AttnRes (two-phase)L(Q+2LQ)dL\left(Q+\frac{2L}{Q}\right)d2Ld2Ld12(Q+2LQ+2)d\frac{1}{2}\left(Q+\frac{2L}{Q}+2\right)d7d7d
Block AttnRes (two-phase)[6L+Q(Q−1)2]d\left[6L+\frac{Q(Q-1)}{2}\right]d4Ld4Ld[5+Q(Q−1)4L]d\left[5+\frac{Q(Q-1)}{4L}\right]d5.2d5.2d
HiRes (fused)[4L+B(2n+1)+n+1]d\left[4L+B(2n+1)+n+1\right]d[2L+B(n+1)+n+1]d\left[2L+B(n+1)+n+1\right]d[3+B(3n+2)+2(n+1)2L]d\left[3+\frac{B(3n+2)+2(n+1)}{2L}\right]d4.2d4.2d

Table 12. HiRes reaches the lowest activation I/O among residual-expansion methods. We compare optimized implementations for all methods and count residual-related activation reads and writes. mm is the HC/mHC/VWN expansion rate, QQ the number of AttnRes blocks, and n=D/dn=D/d and BB the HiRes expansion ratio and block count. The fourth column divides total I/O by the 2L2L sublayers; typical values use L=16L=16, Q=B=4Q=B=4, m=4m=4, and n=2n=2.

Residual and HiRes. A vanilla merge moves 3d3d elements per sublayer, or 6Ld6Ld in total. HiRes retains this cost for inner merges. At each block boundary, the read projection reads DD elements and writes dd elements; fusing the write projection, outer addition, and RMSNorm then reads D+dD+d elements and writes DD elements.

Each block therefore incurs 3D+2d3D+2d elements of traffic, and the two endpoint maps move another 2(D+d)2(D+d), giving the overhead

ΔIO=B(3D+2d)+2(D+d)elements per token(11)\Delta\mathrm{IO}=B(3D+2d)+2(D+d)\quad\text{elements per token} \tag*{(11)}

and the total

IOHiRes=[6L+B(3n+2)+2(n+1)]d,n=D/d.(12)\mathrm{IO}_{\mathrm{HiRes}}=\left[6L+B(3n+2)+2(n+1)\right]d,\qquad n=D/d. \tag*{(12)}

HC, mHC and VWN. HC [59] maintains mm width-dd residual streams, Xl∈Rm×dX_l\in\mathbb{R}^{m\times d}. Its three connections are generated from the current state,

(Hlpre,Hlpost,Hlres)=Gl(Xl),(13)\left(H_l^{\mathrm{pre}},H_l^{\mathrm{post}},H_l^{\mathrm{res}}\right)=G_l(X_l), \tag*{(13)}

with pre-connection Hlpre∈R1×mH_l^{\mathrm{pre}}\in\mathbb{R}^{1\times m}, post-connection Hlpost∈Rm×1H_l^{\mathrm{post}}\in\mathbb{R}^{m\times1}, and a residual connection Hlres∈Rm×mH_l^{\mathrm{res}}\in\mathbb{R}^{m\times m}:

xl=HlpreXl,yl=fl(xl),Xl+1=HlresXl+Hlpostyl.(14)x_l=H_l^{\mathrm{pre}}X_l,\qquad y_l=f_l(x_l),\qquad X_{l+1}=H_l^{\mathrm{res}}X_l+H_l^{\mathrm{post}}y_l. \tag*{(14)}

Following the dynamic HC parameterization, HC constructs three connections by self-gated projections. Taking HlpreH_l^{\mathrm{pre}} as an example:

Hlpre=alpre⋅tanh⁡(Norm⁡lpre(Xl)Wlpre⊤)+blpre,(15)H_l^{\mathrm{pre}}=a_l^{\mathrm{pre}}\cdot\tanh\left(\operatorname{Norm}_l^{\mathrm{pre}}(X_l)W_l^{\mathrm{pre}\top}\right)+b_l^{\mathrm{pre}}, \tag*{(15)}

where alpre∈Ra_l^{\mathrm{pre}}\in\mathbb{R} is a scaling factor, blpre∈R1×mb_l^{\mathrm{pre}}\in\mathbb{R}^{1\times m} is a bias, and Wlpre∈Rd×1W_l^{\mathrm{pre}}\in\mathbb{R}^{d\times1} is the projection weight.

In mHC [53], GlG_l normalizes vec⁡(Xl)\operatorname{vec}(X_l), jointly projects it to m2+2mm^2+2m logits, applies sigmoid transforms to the pre/post logits, and applies Sinkhorn–Knopp iterations to HlresH_l^{\mathrm{res}}.

In VWN [38], each stream is split into rr parts of width d/rd/r, forming Xl∈Rmr×d/rX_l\in\mathbb{R}^{mr\times d/r}. The remaining operations are similar to HC.

The fused schedules below are our analytic constructions from these published recurrences rather than traffic reported by the source papers. For each method, we can pack the three projections in (13) and fold (15) into one kernel, reading mdmd elements once without materializing the normalized and intermediate states. Because HlpreH_l^{\mathrm{pre}} is available only after the projection has reduced over all of XlX_l, the subsequent pre-connection in (14) makes a second mdmd read and writes dd. After flf_l, the residual connection and post-connection are fused into one kernel that reads md+dmd+d and writes mdmd. Thus the activation traffic for each sublayer is

R=(3m+1)d,W=md,IO=(4m+1)d,(16)R=(3m+1)d,\qquad W=md,\qquad\mathrm{IO}=(4m+1)d, \tag*{(16)}

and over the network,

IOHC=IOmHC=IOVWN=2L(4m+1)d.(17)\mathrm{IO}_{\mathrm{HC}}=\mathrm{IO}_{\mathrm{mHC}}=\mathrm{IO}_{\mathrm{VWN}}=2L(4m+1)d. \tag*{(17)}

This omits the lower-order m2+2mm^2+2m coefficient traffic.

AttnRes. Full AttnRes [20] stores every preceding layer output vi∈Rdv_i \in\mathbb{R}^d and uses a learned query ql∈Rdq_l \in\mathbb{R}^d to retrieve across depth:

αi→l=exp⁡(qlTRMSNorm⁡(vi))∑j<lexp⁡(qlTRMSNorm⁡(vj)),hl=∑i<lαi→lvi.(18)\alpha_{i\to l} = \frac{\exp(q_l^{\mathsf{T}}\operatorname{RMSNorm}(v_i))}{\sum_{j<l}\exp(q_l^{\mathsf{T}}\operatorname{RMSNorm}(v_j))}, \qquad h_l = \sum_{i<l}\alpha_{i\to l}v_i. \tag*{(18)}

Block AttnRes instead stores one completed representation per architecture block and includes the current block’s partial sum as an additional source.

For the two-phase schedule, partition the 2L2L sublayers into QQ execution blocks of S=2L/QS = 2L/Q sublayers. Phase 1 batches the SS fixed queries in a block and attends to completed history once, returning one dd-wide output and softmax statistics per sublayer. Phase 2 attends to the sequentially available intra-block history and merges both phases using online softmax [29]. For Full AttnRes, execution block q∈{0,…,Q−1}q \in\{0,\ldots,Q-1\} has qSqS completed outputs. Phase 1 reads these once for all SS queries, contributing qSdqSd elements per block, and writes one dd-wide result per sublayer. Within a block, sublayer jj reads jj earlier intra-block outputs in Phase 2, which also reads back its Phase 1 result for the merge and passes hlh_l directly to flf_l. Summing both triangular histories over depth gives

RFullAttnRes=[S∑q=0Q−1q+Q∑j=0S−1j+2L]d=L(Q+S)d,(19)R_{\mathrm{FullAttnRes}} = \left[S\sum_{q=0}^{Q-1}q + Q\sum_{j=0}^{S-1}j + 2L\right]d = L(Q+S)d, \tag*{(19)}
WFullAttnRes=2Ld,W_{\mathrm{FullAttnRes}} = 2Ld,
IO‾FullAttnRes=Q+S+22d.\overline{\mathrm{IO}}_{\mathrm{FullAttnRes}} = \frac{Q+S+2}{2}d.

Here the two phases each write one dd-wide result per sublayer. Block AttnRes instead reads only the qq completed block summaries in Phase 1. Summing this history gives Q(Q−1)d/2Q(Q-1)d/2 reads; its fused partial-sum attention and online-softmax merge additionally read 3d3d and write the dd-wide partial sum per sublayer. Thus

RBlockAttnRes=[6L+Q(Q−1)2]d,(20)R_{\mathrm{BlockAttnRes}} = \left[6L+\frac{Q(Q-1)}{2}\right]d, \tag*{(20)}
WBlockAttnRes=4Ld,W_{\mathrm{BlockAttnRes}} = 4Ld,
IO‾BlockAttnRes=[5+Q−12S]d.\overline{\mathrm{IO}}_{\mathrm{BlockAttnRes}} = \left[5+\frac{Q-1}{2S}\right]d.

Agent-Driven Optimization of Residual Mechanisms

To assess the practical efficiency and ease of adoption of each residual expansion mechanism, we measure both its final training efficiency and the agent-driven optimization required to reach it, comparing framework-level PyTorch and mechanism-specific Triton implementations.

Experimental Protocol

We evaluate (dmodel,L)∈{(768,12),(1024,16),(1280,20)}(d_{\mathrm{model}}, L) \in\{(768, 12), (1024, 16), (1280, 20)\} on eight GPUs using FSDP FULL_SHARD, bfloat16, sequence length 4096, and per-GPU micro-batch 4. Every measurement includes forward, backward, and the optimizer update; peak memory is reported per GPU. The attention, MLP, normalization, optimizer, precision, and measurement harness are shared, and only mechanism-specific code may change.

Starting from functionally equivalent naive implementations, independent Claude Opus 5 agents receive the same prompt and optimize HiRes, VWN-4, mHC-4, and Block AttnRes-4 in two stages: the PyTorch stage that may leverage graph transformations, torch.compile, batching, and layout changes; and the Triton stage that adds mechanism-specific fused kernels. Every candidate must match the reference outputs and gradients at all three scales. Candidates are measured against the current best and retained only if throughput improves by at least 0.2% at every scale. We report the best implementation from each stage. The final optimized Triton implementation is used to measure the efficiency of each residual expansion mechanism in Table 2.

Optimization Trajectory

We visualize the optimization campaign in Figure 8. Three patterns emerge. First, HiRes maintains the highest throughput among the residual-expansion methods throughout most of the campaign, until the fully optimized Block AttnRes overtakes it near the end. Second, HiRes reaches its best implementation in less optimization time than the other methods in our campaign. Third, HiRes’ PyTorch implementation already matches its final Triton throughput, with custom kernels primarily reducing memory. Other methods instead rely on Triton for substantial throughput gains.

Optimization trajectory showing throughput and peak-memory overhead

Figure 8. Agent-driven optimization campaign. (a) Throughput (% of the frozen baseline). (b) Peak-memory overhead at (d,L)=(768,12)(d, L) = (768, 12) versus optimization time. Dots are accepted improvements; solid and dashed segments are the PyTorch and Triton stages; square, diamond, and star mark the best PyTorch, best Triton, and closing implementations. HiRes reaches its best throughput early and without custom kernels.

Practical Efficiency

Table 13 compares the best PyTorch-stage and fused Triton implementations across scales. Without custom kernels, HiRes already achieves 91.7–93.0% of baseline throughput—the highest among the residual-expansion methods—with only 5.5–5.7% peak-memory overhead. Triton fusion preserves this throughput while reducing the memory overhead to 3.3–3.4%. The resulting implementation approaches the throughput of Block AttnRes-4, exceeds that of mHC-4 and VWN-4, and has the lowest memory overhead among the compared methods.

MethodStageThroughput (% of baseline) (768, 12)Throughput (% of baseline) (1024, 16)Throughput (% of baseline) (1280, 20)Peak memory (% over baseline) (768, 12)Peak memory (% over baseline) (1024, 16)Peak memory (% over baseline) (1280, 20)
mHC-4PyTorch58.156.455.2+19.1+25.2+29.7
Triton86.588.286.1+20.8+27.0+31.4
VWN-4PyTorch68.369.369.2+47.6+58.7+66.2
Triton81.684.480.5+28.1+34.2+38.4
Block AttnRes-4PyTorch80.377.475.1+3.7+4.9+5.8
Triton93.794.994.5+5.3+6.9+8.0
HiResPyTorch91.792.793.0+5.5+5.7+5.5
Triton91.792.893.1+3.4+3.4+3.3

Table 13. Table 13 Best PyTorch- and Triton-stage performance of each method. Throughput and peak-memory overhead are relative to the frozen baseline. Within each stage, best/second-best values are bold/underlined. HiRes leads in throughput and memory without custom Triton kernels.

Latency Decomposition

Figure 9 separates measured kernel time from analytic compute- and bandwidth-limited floors. HiRes spends most of its overhead in the 2B+22B + 2 dense interface GEMMs, whereas the comparison methods spend most of theirs moving and mixing expanded states at every sublayer. This explains why fusion materially benefits the latter, while HiRes is already close to its fused performance with framework-level optimizations.

Figure 9. Figure 9 Extra kernel time per step at (1280, 20) over the shared frozen backbone (242.5 ms). (a) Fusion (best Triton versus best PyTorch) cuts the overhead of the other methods by 2–6× but leaves HiRes nearly unchanged. (b) Fused overhead by kernel type (upper bars) and its compute- and bandwidth-limited floors at the GPU’s advertised bf16 and HBM peaks (lower bars): HiRes spends most of it in cuBLAS GEMMs, the others in their own mixing kernels.

A Latent View

Besides expanding the residual state width, HiRes also admits a dual interpretation: relative to a DD-dimensional persistent hidden state, its attention and FFN computation occurs in a dd-dimensional latent space. Expanding residual width by n=D/dn = D/d is therefore equivalent to compressing compute width by a factor of nn; the wide state preserves information across blocks, while the narrow state performs the transformations within each block.

This view connects HiRes to module-specific latent designs. MLA compresses the attention KV representation [8], while LatentMoE performs routed expert computation in a compressed space [10]. HiRes extends the same separation between state width and compute width to the residual hierarchy. Its key distinction is interaction granularity: instead of exchanging information between hidden and latent spaces at every sublayer, HiRes reads once at block entry and writes once at block exit. Layer grouping therefore reduces hidden-latent transport from per-sublayer to per-block while allowing computation to evolve for multiple layers in the latent space.

Limitations

Our study has the following limitations. First, HiRes decouples residual width from compute width, but we have not jointly optimized these widths and model depth under a fixed compute budget. The best allocation of capacity between carrying representations and transforming them therefore remains unclear. Second, we use equal-sized blocks, so exchanges between the wide state and the inner computation occur at regular intervals. We have not examined whether different parts of the network benefit from more frequent access to the wide state. Finally, our results show the benefits of residual expansion, yet how the wider residual state improves learning remains an open question. In particular, what information is preserved across blocks and how individual sublayers contribute to its transformation and reuse are not yet well understood.

References

  1. [1]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sangh. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298.DOI
  2. [2]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.arxiv.org/abs/1607.06450
  3. [3]Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: fast convergence at large depth. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352–1361. PMLR, 27–30 Jul 2021.arxiv.org/abs/2003.04887
  4. [4]Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating updates for efficient transformers. In A. Oh, T. Naumann, A. Glober­son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 76718–76736. Curran Associates, Inc., 2023. doi: 10.52202/075280-3352.DOI
  5. [5]Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, April 2020. ISSN 2159-5399. doi: 10.1609/aaai.v34i05.6239.DOI
  6. [6]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300.DOI
  7. [7]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  8. [8]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenqiu Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruiyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanmhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. W. Y. K. Li, Y. X. W. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.arxiv.org/abs/2405.04434
  9. [9]Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdlmoushin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Bri­odkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kıp‌f, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7480–7512. PMLR, 23–29 Jul 2023.
  10. [10]Vennmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, and Bita Rouhani. Latentmoe: Toward optimal accuracy per flop and parameter in mixture of experts, 2026.arxiv.org/abs/2601.18089
  11. [11]Yanwen Fang, Yuxi CAI, Jintai Chen, Jingyu Zhao, Guangjian Tian, and Guodong Li. Cross-layer retrospective retrieving via layer attention. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2302.03985
  12. [12]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  13. [13]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.arxiv.org/abs/2101.00027
  14. [14]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.arxiv.org/abs/1512.03385
  15. [15]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
  16. [16]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Henighan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc., 2022. doi: 10.52202/068431-2176.
  17. [17]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.arxiv.org/abs/1608.06993
  18. [18]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.arxiv.org/abs/2001.08361
  19. [19]Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-LN: Revisiting normalization layer in the transformer ar- chitecture. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30400–30436. PMLR, 13–19 Jul 2025.arxiv.org/abs/2502.02732
  20. [20]Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyun Wang, Yucheng Wang, Guangduo Chen, Bohong Yin, Yutian Chen, Junjie Yan, Ming Wei, Y. Zhang, Fanging Meng, Chao Hong, Xiaotong Xie, Shaowei Liu, Enzhe Lu, Yunpeng Tai, Yanru Chen, Xin Men, Haiqing Guo, Y. Charles, Haoyu Lu, Lin Sui, Jinguo Zhu, Zaida Zhou, Weiran He, Weixiao Huang, Xinran Xu, Yuzhi Wang, Guokun Lai, Yulun Du, Yuxin Wu, Zhilin Yang, and Xinyu Zhou. Attention residuals, 2026.arxiv.org/abs/2603.15031
  21. [21]Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-In: Unleashing the power of deeper layers by combining pre-ln and post-ln. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 98244–98258, 2025.arxiv.org/abs/2412.13795
  22. [22]Tianyu Li, Dongchen Han, Zixuan Cao, Haofeng Huang, Mengyu Zhou, Ming Chen, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang, and Gao Huang. Siamesenorm: Breaking the barrier to reconciling pre/post-norm, 2026.arxiv.org/abs/2602.08064
  23. [23]Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.447.DOI
  24. [24]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  25. [25]Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannah Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. In A. Glober­son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 64338–64376. Curran Associates, Inc., 2024. doi: 10.52202/079017-2052.DOI
  26. [26]Gaurav Menghani, Ravi Kumar, and Sanjiv Kumar. LaUReL: Learned augmented residual layer. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 43826–43836. PMLR, 13–19 Jul 2025.arxiv.org/abs/2411.07501
  27. [27]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.arxiv.org/abs/1609.07843
  28. [28]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, October–November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1260.
  29. [29]Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax, 2018.arxiv.org/abs/1805.02867
  30. [30]Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah Smith, Pang Wei Koh, Amanpreet Singh, and Hanna Hajishirzi. Olmo: Open mixture-of-experts language models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 62061–62121, 2025.arxiv.org/abs/2409.02060
  31. [31]Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2–3 2019. Association for Computational Linguistics.DOI
  32. [32]Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 136479–136508. Curran Associates, Inc., 2024. doi: 10.52202/079017-4336.DOI
  33. [33]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
  34. [34]Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. M2D2: A massively multi-domain language modeling dataset. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 964–975, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.63.DOI
  35. [35]Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense Reasoning: Papers from the AAAI Spring Symposium, volume SS-11-06 of AAAI Technical Report, pages 90–95, 2011.
  36. [36]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, August 2021. ISSN 1557-7317. doi: 10.1145/3474381.DOI
  37. [37]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454.DOI
  38. [38]Seed, Baisheng Li, Banggu Wu, Bole Ma, Bowen Xiao, Chaoyi Zhang, Cheng Li, Chengyi Wang, Chengyin Xu, Chi Zhang, Chong Hu, Daoguang Zan, Defa Zhu, Dongyu Xu, Du Li, Faming Wu, Fan Xia, Ge Zhang, Guang Shi, Haobin Chen, Hongyu Zhu, Hongzhi Huang, Huan Zhou, Huanzhang Dou, Jianhui Duan, Jianqiao Lu, Jianyu Jiang, Jiayi Xu, Jiecao Chen, Jin Chen, Jin Ma, Jing Su, Jingji Chen, Jun Wang, Jun Yuan, Juncai Liu, Jundong Zhou, Kai Hua, Kai Shen, Kai Xiang, Kaiyuan Chen, Kang Liu, Ke Shen, Liang Xiang, Lin Yan, Lishu Luo, Mengyao Zhang, Ming Ding, Mofan Zhang, Nian Ning Liang, Peng Li, Penghao Huang, Pengpeng Mu, Qi Huang, Qianli Ma, Qiyang Min, Qiying Yu, Renming Pang, Ru Zhang, Shen Yan, Shen Yan, Shixiong Zhao, Shuaishuai Cao, Shuang Wu, Siyan Chen, Siyu Li, Siyuan Qiao, Tao Sun, Tian Xin, Tiantian Fan, Ting Huang, Ting-Han Fan, Wei Jia, Wenqiang Zhang, Wenxuan Liu, Xiangzhong Wu, Xiaochen Zuo, Xiaoying Jia, Ximing Yang, Xin Liu, Xin Yu, Xingyan Bin, Xintong Hao, Xiongcai Luo, Xujing Li, Xun Zhou, Yanghua Peng, Yangrui Chen, Yi Lin, Yichong Leng, Yinghao Li, Yingshuan Song, Yiyuan Ma, Yong Shan, Yongan Xiang, Yonghui Wu, Yongtao Zhang, Yongzhen Yao, Yu Bao, Yuehang Yang, Yufeng Yuan, Yunshui Li, Yuqiao Xian, Yutao Zeng, Yuxuan Wang, Zehua Hong, Zehua Wang, Zengzhi Wang, Zeyu Yang, Zhengqiang Yin, Zhenyi Lu, Zhexi Zhang, Zhi Chen, Zhi Zhang, Zhiqi Lin, Zihao Huang, Zilin Xu, Ziyun Wei, and Zuo Wang. Virtual width networks, 2025.arxiv.org/abs/2511.11238
  39. [39]Noam Shazeer. Glu variants improve transformer, 2020.arxiv.org/abs/2002.05202
  40. [40]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
  41. [41]Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization, 2021.arxiv.org/abs/2110.09456
  42. [42]Luca Soldaini, Rodney Kinney, Akshita B加ghia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannah Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.840.DOI
  43. [43]Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015.arxiv.org/abs/1505.00387
  44. [44]Jianlin Su, Muratdha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, February 2024. ISSN 0925-2312. doi: 10.1016/j.neucom.2023.127063.DOI
  45. [45]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421.DOI
  46. [46]Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 32–42, October 2021.arxiv.org/abs/2103.17239
  47. [47]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762
  48. [48]Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Validad Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious (COLM’s version). In Second Conference on Language Modeling, 2025.
  49. [49]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, October 2024. ISSN 1939-3539. doi: 10.1109/tpami.2024.3386927.DOI
  50. [50]Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Anna Korhonen, David Traum, and Lluis Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1176.DOI
  51. [51]Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4413.DOI
  52. [52]Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. MUDDFormer: Breaking residual bottlenecks in transformers via multiway dynamic dense connections. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 68440–68458. PMLR, 13–19 Jul 2025.arxiv.org/abs/2502.12170
  53. [53]Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Mingyu Xu, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. mHC: Manifold-constrained hyper-connections. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2512.24880
  54. [54]Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524–10533. PMLR, 13–18 Jul 2020.arxiv.org/abs/2002.04745
  55. [55]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472.DOI
  56. [56]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.arxiv.org/abs/1910.07467
  57. [57]Yifan Zhang, Yifeng Liu, Mengdi Wang, and Quanquan Gu. Deep delta learning, 2026.arxiv.org/abs/2601.00417
  58. [58]Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. Value residual learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1375.
  59. [59]Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 97183–97219, 2025.arxiv.org/abs/2409.19606
  60. [60]Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, and Xun Zhou. Frac-connections: Fractional extension of hyper-connections, 2025.arxiv.org/abs/2503.14125
  61. [61]Zhijian Zhuo, Yutao Zeng, Ya Wang, Sijun Zhang, Jian Yang, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Hybridnorm: Towards stable and efficient transformer training via hybrid normalization. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 25106–25147. Curran Associates, Inc., 2025. doi: 10.52202/085713-0844.DOI

Paper details

vlnc:W_wwy4ppz6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents