WorksComputer Science
Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width
Abstract
Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a 1.30-fold improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.
Introduction
Residual connections [14] are the backbone of modern Transformers. A standard PreNorm layer reads from a normalized copy of the current hidden state, computes an update, and adds that back to the original state [47, 54]. Repeating this operation yields a simple yet effective transport rule: one -dimensional state is carried through depth along an identity path, while every attention and feed-forward module contributes an additive update. This design makes deep optimization practical, but it also ties the width of the state carried through depth to the width of the working space of every layer.
Recent residual architectures relax this constraint in different ways. Alternating Updates (AltUp) maintains multiple residual streams, with each layer computing one stream and predicting the remainder [4]. Hyper-Connections (HC) likewise uses multiple streams but learns how layers read from, write to, and combine them with lightweight projectors [59]; Virtual Width Networks (VWN) generalizes this formulation by splitting each stream into finer granularity [38]; Manifold-Constrained Hyper-Connections (mHC) stabilizes HC by constraining its residual mixing operators to a doubly stochastic manifold [53]. Attention Residuals instead retrieves representations from earlier layers using attention over depth [20]. These methods introduce different residual operators with different update rules. Nevertheless, they share two recurring departures from vanilla residual addition: (i) the effective residual state is expanded, and (ii) residual accumulation becomes explicitly regulated, keeping the residual scale in check. This suggests that residual-state capacity is an architectural variable in its own right, rather than a quantity that must always equal model width.
Figure 1. HiRes uses a two-level residual hierarchy. (a) Vanilla PreNorm carries one -dimensional residual stream. (b) HiRes carries a -dimensional PostNorm outer residual across layer blocks. (c) Each block reads , runs vanilla sublayers on the -dimensional inner residual, and writes back . Gray and navy paths have width and ; orange trapezoids are the added dense interfaces, and dashed frames mark repeated residual units.
However, their per-layer expanded states and intricate mixing add activation traffic and memory-bound residual operators, typically requiring fusion, recomputation, overlap, or specialized kernels to recover throughput [53]. If the benefit stems from the wider state itself rather than particular operators, a natural question arises: can Transformers benefit from a wider residual state with as little architectural change as possible?
To this end, we introduce Hierarchical Residuals (HiRes). HiRes groups adjacent Transformer layers into layer blocks and organizes residual computation at two levels. Inside each block, the model runs ordinary -dimensional PreNorm Transformer layers. Across blocks, it maintains a -dimensional outer residual state with the familiar additive update (). A read projection maps the outer state into a block, and after the inner residual accumulation, a write projection maps the resulting inner state back. The block write is added to the outer residual and the sum is then post-normalized, which regulates the residual scale. Thus, the outer recurrence retains the familiar PostNorm residual, while the inner Transformer layers are unchanged. In this way, the residual expansion ratio can be selected independently of the compute width for each inner layer. Because read/write operations occur only at block boundaries, the additional parameters and compute costs are amortized over depth. Figure 1 places the two levels side by side: the outer recurrence is the residual motif of a vanilla PostNorm Transformer, with a whole layer block in place of a sublayer.
Method
Empirically, HiRes consistently outperforms vanilla residual addition across model sizes and depths. Under fitted loss-compute scaling laws, it is 1.30× more FLOP-efficient. As depth scales up at fixed width, HiRes continues to improve through 128 layers where the baseline exhibits diminishing returns. Against HyperConnections, mHC, V!WN, and Attention Residuals, HiRes remains comparable in quality at lower activation memory and I/O costs. On OLMoE-1B-7B [30] pretraining for 500B tokens (§ 4.5), HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. Overall, HiRes provides a minimalist way to expand the residual state without widening per-layer compute. It shows that decoupling residual width from compute width is beneficial, and that this decoupling is available with standard operators.
Method
Vanilla Residual Addition
Consider a decoder-only Transformer with layers and hidden width . Each layer contains an attention module and an FFN module, giving residual sublayers. Let denote the residual state before sublayer . A standard PreNorm Transformer can be written as
where is an attention or FFN transformation. Thus the state carried through depth (residual width, dimension of ) and the state processed by every sublayer (compute width, input and output dimension of ) share the same width .
Decoupling Residual Width and Compute Width
To decouple residual width and compute width, we introduce a wider residual stream with and denotes the residual expansion ratio. Since every keeps operating at compute width , there needs a read map to project the carried state into the working space, and a write map to return the sublayer output to the residual space. So the vanilla residual in Equation (1) becomes
Setting and recovers the vanilla recurrence.
Residual-expansion methods typically introduce lightweight interfaces by sharing mixing weights across streams or coordinate groups. HiRes instead adopts the simplest choice: dense projections and . At fixed and these are the most general linear interfaces, imposing no structural inductive biases like weight sharing, providing full connectivity between the expanded residual state and the working space. Empirically, HiRes achieves competitive performance at a smaller expansion ratio than methods with structured interfaces, while incurring lower activation memory and I/O costs (§ 4.4).
Hierarchical Residual Recurrence
Applying dense interfaces at every sublayer would add parameters and FLOPs, which is comparable to the cost of the Transformer layers themselves and thus unacceptable. HiRes therefore invokes them only at block boundaries, yielding a hierarchical structure.
HiRes partitions the layers into consecutive blocks . Block contains layers, where . For simplicity, we use equal-sized contiguous blocks with .
Inside each block , HiRes keeps the inner PreNorm recurrence unchanged as the vanilla residual in Equation (1):
We write for the composition of these recurrences, so that .
Between blocks, HiRes introduces an outer residual and normalizes it in PostNorm style, transforming the layerwise recurrence in Equation (2) into a blockwise one:
Equations (3) and (4) define HiRes’ two-level residual hierarchy: a -dimensional PreNorm inner residual and a -dimensional PostNorm outer residual, illustrated in Figure 1. Grouping layers into blocks amortizes the dense-interface overhead across depth, reducing it from to . Layers within each block thus share the same boundary interfaces and -dimensional coordinate system: the block pulls a subspace of the outer residual state, transforms it through several layers, and commits the result back.
To connect the embedding and the language modeling head to the outer residual state, we use an embedding write projection and a head read projection ,
where is the input token. Appendix § 12 details the exact parameter, FLOP, memory, and I/O overheads, and Appendix Algorithm 1 provides PyTorch-style pseudocode for the forward pass.
Design Choices
Normalization at block boundaries. Boundary normalization provides residual regulation in HiRes: without it, the outer residual stream would be unbounded as it accumulates block outputs. Keeping PreNorm within blocks and applying PostNorm between them combines PreNorm’s optimization stability during the backward pass [50, 54] with PostNorm’s periodic control of the outer residual scale during the forward pass [19].
Setting and in Equation (4) and removing the outer residual accumulation yields hierarchical normalization (HiNorm):
which inserts one RMSNorm into the residual stream every layers. HiNorm isolates regulation from residual expansion and serves as a control in our experiments.
HiNorm alone beats the baseline and reaches minimum loss at an intermediate (Appendix Figure 6a). In Appendix Figure 6b, the residual scale grows monotonically in the vanilla baseline and in HiRes without boundary normalization. HC, VWN, and AttnRes [20, 38, 59] regulate the scale implicitly through residual mixing learned at every layer, whereas HiRes regulates it explicitly with boundary normalizations: the scale grows only within each block and is reset at every boundary.
Interface stabilization. We empirically observe that randomly initialized interfaces degrade performance early in training (Appendix § 11.§ 3.2), which we hypothesize arises from a seesaw dynamic: the interfaces learn each block’s working coordinate system while the inner layers simultaneously learn to transform representations within that evolving space. Motivated by prior identity-biased residual gates [3, 46], we stabilize the interfaces by interpolating between identity and learned mappings using learnable gating scalars:
where truncates the outer state to its first coordinates, zero-pads back to width , and is the logistic sigmoid. Each interface has an independent learned scalar logit initialized such that is close to 0 (we set it to 0.1 in the experiments).
This parameterization initializes the interfaces near identity, providing the inner layers with a relatively stable coordinate system early in training. The interfaces can then gradually specialize and make increasing use of the expanded residual state parameterized by and .
Experiments
Setup
We conduct the primary architecture study using the OLMoE codebase [30]. All models use RMSNorm [56], RoPE [44], grouped-query attention (GQA) [1], and a SwiGLU feed-forward network [39]. We train with AdamW [24] and a cosine learning-rate schedule with warmup, use a 4096-token context, and initialize weights with standard deviation 0.02.
The default HiRes configuration uses residual expansion ratio and block count . We ablate both choices in § 5.1.
For architecture comparisons, we average training loss over the final 1000 steps. Validation loss and downstream accuracy are averaged over the final 3 evaluations, 1000 steps apart, and across 11 held-out domains and 12 tasks, respectively. The full experimental setup is detailed in Appendix § 9.

Figure 2. HiRes achieves better scaling properties. (a) Validation loss versus training FLOPs with fitted laws. At (inset), the baseline needs 1.30× the FLOPs of HiRes with . (b) Compute multiplier at matched loss: baseline FLOPs over family FLOPs, at runs (markers) and along the fitted laws (lines).
Scaling on Model Size
We evaluate model-size scaling over five model sizes, jointly increasing inner width and depth from to with the parameter count spanning from 70M to 626M. The detailed scaling ladder can be found in Table 5. We compare six fixed architecture families: the vanilla residual baseline; HiRes with , , and ; and two reduced forms of HiRes, namely HiNorm, which keeps only the boundary normalization, and equal-width HiRes with , which adds the interfaces and the outer residual without widening. All HiRes variants use .
| Tokens | Metric | Baseline | HiNorm | |||||
| 30B | Params | 70.264M | 70.266M | 72.888M | 75.511M | – | – | |
| FLOPs () | 1.8691 | 1.8691 | 1.9163 | 1.9635 | – | – | ||
| Train loss | 3.4768 | 3.4617 | 3.4515 | 3.4314 | – | – | ||
| Val. loss | 3.4733 | 3.4564 | 3.4401 | 3.4201 | – | – | ||
| 50B | Params | 138.433M | 138.436M | 144.334M | 150.235M | 156.136M | 162.037M | |
| FLOPs () | 6.4186 | 6.4187 | 6.5957 | 6.7727 | 6.9497 | 7.1267 | ||
| Train loss | 3.2228 | 3.1956 | 3.1869 | 3.1645 | 3.1586 | 3.1591 | ||
| Val. loss | 3.1941 | 3.1714 | 3.1573 | 3.1300 | 3.1242 | 3.1251 | ||
| 100B | Params | 245.403M | 245.407M | 255.892M | 266.382M | 276.872M | 287.362M | |
| FLOPs () | 22.780 | 22.780 | 23.409 | 24.039 | 24.669 | 25.299 | ||
| Train loss | 3.0108 | 2.9827 | 2.9758 | 2.9663 | 2.9589 | 2.9539 | ||
| Val. loss | 2.9734 | 2.9466 | 2.9401 | 2.9271 | 2.9219 | 2.9153 | ||
| 200B | Params | 403.757M | 403.762M | 420.146M | 436.535M | 452.924M | 469.313M | |
| FLOPs () | 73.623 | 73.623 | 75.590 | 77.556 | 79.522 | 81.488 | ||
| Train loss | 2.8524 | 2.8338 | 2.8245 | 2.8097 | 2.8015 | 2.8065 | ||
| Val. loss | 2.8158 | 2.7972 | 2.7865 | 2.7697 | 2.7590 | 2.7612 | ||
| 350B | Params | 626.078M | 626.084M | 649.677M | 673.276M | – | – | |
| FLOPs () | 194.91 | 194.91 | 199.86 | 204.82 | – | – | ||
| Train loss | 2.7295 | 2.7084 | 2.7019 | 2.6907 | – | – | ||
| Val. loss | 2.6838 | 2.6647 | 2.6618 | 2.6427 | – | – |
Table 5. Parameters, training FLOPs, and losses for model size scaling. At every model size, loss falls from the baseline to HiNorm, HiRes , and HiRes . HiNorm and HiRes use .
Since HiRes introduces modest parameter overhead, we compare models by fitting a loss-compute scaling law [16, 18] in analytic training FLOPs: , where is the total parameter count, is the number of training tokens, is the number of layers, is the sequence length, and is the inner width. We jointly fit the observations with the scaling law , with a shared irreducible loss and family-specific amplitude and exponent. Here FLOPs is the reference compute and is the fitted irreducible loss.
Figure 2 separates the two sources of the HiRes gain: residual regulation and residual expansion. HiNorm consistently shifts the vanilla residual scaling curve downward. Widening the residual state () brings a further gain at every scale. Equal-width HiRes () remains close to HiNorm, indicating that the hierarchy interfaces alone add little benefit. A clearer improvement appears when the outer state expands from to , while the and trajectories remain close to , indicating diminishing returns. Regulation and widening thus contribute complementary gains. HiRes with combines both, and the fitted scaling laws estimate that the baseline requires 1.30× the FLOPs of HiRes to reach a validation loss of 3.00.
Scaling on Depth
We next examine depth scaling properties of HiRes by fixing the inner width at and the training budget at 30B tokens while increasing the number of Transformer layers from 8 to 128.
Figure 3 plots validation loss against depth on a logarithmic axis, where ideal compute scaling should appear as a straight line. HiRes not only consistently outperforms the vanilla residual baseline at every depth, but also maintains better scaling properties. The baseline bends away from its scaling curve at the last doubling from to . With , HiRes gives up less of its per-doubling reduction than the baseline, especially at , indicating that regulating residual accumulation and widening the residual state can hold more layers. Raising the block count to further preserves the scaling close to a straight line through , suggesting more blocks for deeper models.
Figure 3. HiRes scales better with depth. HiRes wins at every depth and bends less at ; stays nearly straight.
Comparison with Other Residual Expansion Methods
We compare HiRes with other residual expansion methods with the recipe of for 50B-token training, namely Hyper-Connections (HC) [59], mHC [53], Virtual Width Networks (VWN) [38], and Attention Residuals [20]. Specifically, we test and wider residual states for HC/mHC/VWN, full and 4-block AttnRes, and , HiRes. We then test mHC-4, VWN-4, Block AttnRes-4, and HiRes configurations on larger backbones. Appendix § 10.§ 4 gives the detailed implementations and complete results.
At , all residual-expansion methods improve the vanilla residual baseline in loss, and HiRes achieves comparable performance to mHC, VWN, and AttnRes at a lower residual expansion rate, as shown in Table 1.
Table 1 HiRes versus other residual-expansion methods at and 50B tokens. Suffixes give the expansion rate for HC/mHC/VWN and the block count for AttnRes. With a wider residual state, HiRes is competitive even with the variants of HC, mHC, and VWN. Bold/underlined: best/second-best among residual-expansion methods. Appendix C.3 details the configurations and lists their parameters and training FLOPs.
| Metric | Baseline | HC-2 | HC-4 | mHC-2 | mHC-4 | VWN-2 | VWN-4 | Full AttnRes | Block AttnRes-4 | HiRes |
| Train loss | 3.2228 | 3.1773 | 3.1870 | 3.1941 | 3.1625 | 3.1584 | 3.1578 | 3.1938 | 3.1943 | 3.1645 |
| Val. loss | 3.1941 | 3.1649 | 3.1533 | 3.1871 | 3.1459 | 3.1400 | 3.1228 | 3.1566 | 3.1649 | 3.1300 |
| Avg. Acc. (%) | 46.91 | 47.25 | 46.26 | 46.66 | 47.33 | 47.03 | 45.86 | 46.17 | 46.13 | 47.15 |
Table 1.
Table 2 compares the vanilla residual baseline, mHC, VWN, Block AttnRes, and HiRes across three backbone scales: , , . HiRes remains within the leading quality range of these recent residual-expansion methods. The same table reports the analytic I/O at each scale (Appendix § 12), together with throughput and memory from a unified measurement of each method’s highly optimized Triton implementation (Appendix § 13). HiRes has the lowest activation I/O among residual-expansion methods and closest to the baseline’s 3d. Despite additional parameters, HiRes retains 91.7–93.1% of baseline throughput at 3.3–3.4% memory overhead, which is on par with optimized AttnRes and more efficient than mHC/VWN. Moreover, HiRes’ standard PyTorch implementation already reaches near-final performance, while other methods rely on mechanism-specific kernel fusion. Appendix § 13 reports the agentic optimization campaign and the latency decomposition behind these numbers.
| Backbone | Metric | Baseline | mHC-4 | VWN-4 | Block | AttnRes-4 | HiRes |
| (768, 12) | Train loss | 3.2228 | 3.1625 | 3.1578 | 3.1943 | 3.1645 | |
| Val. loss | 3.1941 | 3.1459 | 3.1228 | 3.1649 | 3.1300 | ||
| Avg. Acc. (%) | 46.91 | 47.33 | 45.86 | 46.13 | 47.15 | ||
| Activation I/O | |||||||
| Peak memory (GiB) | 16.31 | 19.71 | 20.89 | 17.18 | 16.86 | ||
| Throughput (M tok/s) | 0.682 | 0.590 | 0.556 | 0.639 | 0.625 | ||
| (1024, 16) | Train loss | 3.0108 | 2.9633 | 2.9678 | 2.9912 | 2.9663 | |
| Val. loss | 2.9734 | 2.9205 | 2.9231 | 2.9547 | 2.9271 | ||
| Avg. Acc. (%) | 48.76 | 49.16 | 49.57 | 49.45 | 49.15 | ||
| Activation I/O | |||||||
| Peak memory (GiB) | 21.49 | 27.28 | 28.84 | 22.96 | 22.22 | ||
| Throughput (M tok/s) | 0.419 | 0.370 | 0.354 | 0.398 | 0.389 | ||
| (1280, 20) | Train loss | 2.8524 | 2.8157 | 2.8068 | 2.8366 | 2.8097 | |
| Val. loss | 2.8158 | 2.7671 | 2.7533 | 2.7871 | 2.7697 | ||
| Avg. Acc. (%) | 51.90 | 52.49 | 52.30 | 51.48 | 52.76 | ||
| Activation I/O | |||||||
| Peak memory (GiB) | 28.07 | 36.89 | 38.85 | 30.33 | 28.99 | ||
| Throughput (M tok/s) | 0.279 | 0.240 | 0.224 | 0.263 | 0.259 |
Table 2.
Table 2 HiRes stays competitive across scales. Activation I/O is the analytic count per token per sublayer for optimized implementations (Appendix § 12); peak memory and throughput are measured with each method’s optimized Triton implementation (Appendix § 13), whereas HiRes is nearly as fast with vanilla PyTorch implementation (Appendix Table 13). Bold/underlined: best/second-best among residual-expansion methods.
Long-Horizon MoE Pretraining
To scale up, we evaluate HiRes on the OLMoE-1B-7B pretraining recipe [30], a sparse Mixture-of-Experts model [12, 40]. The model contains 6.9B total parameters and 1.3B active parameters per token, and is trained for 500B tokens. We compare the vanilla residual baseline with HiRes using the , configuration, which adds 83.9M parameters (6.5% of the active parameters and 1.2% of the total parameters); Appendix Table 9 lists the full configuration.
As shown in Figure 4, HiRes achieves 0.0208 lower training loss and 0.0180 lower validation loss than the vanilla residual baseline. It also performs better on most downstream tasks and improves the downstream average accuracy by 1.14 points. These results establish that the HiRes design transfers to long-horizon sparse-MoE pretraining.
Figure 4. HiRes on 500B-token OLMoE-1B-7B pretraining. HiRes ends with 0.0208 lower training loss, 0.0180 lower validation loss, and a 1.14-point higher average accuracy. See Appendix § 10.§ 5 for details.
Ablation Studies
Ablation on Block Count and Expansion Ratio
We ablate the two structural hyperparameters of HiRes: the residual expansion ratio and the number of blocks . All runs use and 50B training tokens. The expansion sweep fixes and varies from 0.75 to 4, while the block sweep fixes and varies from 1 to 12. Figure 5 reports both sweeps.
Figure 5. Ablation on expansion ratio and block count in HiRes. Beyond the default , (star), gains roughly follow the shifted scaling law (dashed), which translates the baseline scaling law (dotted) through the star, accounting for the benefit from increasing parameters.
Expansion ratio. In Figure 5a, validation loss rapidly decreases as the expansion ratio increases to , after which the slope slows. The wider and settings provide marginal gains, largely following the shifted scaling law, which accounts for the benefit from extra parameters. Thus is the empirical knee of the quality-overhead curve: it captures most of the benefit of residual expansion while keeping the interface cost moderate. On the other side, at the outer state is narrower than the layer compute width, and it is the only point in the sweep that fails to improve over HiNorm, even with more parameters. This result identifies residual width, rather than the interfaces themselves, as the main source of the gain.
Number of blocks. In Figure 5b, increasing also improves validation loss, with the largest gains occurring as the hierarchy is refined from one to four blocks. Beyond , more frequent outer-state updates show diminishing returns similar to those of the expansion ratio. We therefore retain as the empirical knee.
Architecture Ablations
We subtract one component at a time from the complete , HiRes model, testing boundary normalization, outer residual, and interface learning against the same reference architecture, as shown in Table 3; HiNorm and equal-width HiRes () serve as reduced references. All ablations use and 50B training tokens.
| Variant | Train loss | Val. loss |
| Baseline | 3.2228 | 3.1941 |
| HiNorm | 3.1956 | 3.1714 |
| HiRes , | 3.1869 | 3.1573 |
| HiRes , | 3.1645 | 3.1300 |
| – Frozen random | 3.1905 | 3.1614 |
| – w/o boundary RMSNorm | 3.1874 | 3.1552 |
| – w/o outer residual | 3.1899 | 3.1616 |
| – w/o interface stabilization | 3.1686 | 3.1375 |
Table 3. Ablations on HiRes components. Removing or freezing any one component of HiRes (, ) hurts.
The ablations support the following conclusions. (i) The interfaces must be learnable. Freezing the read/write projections raises validation loss by 0.031, so widening residual with fixed maps does not work. (ii) Widening helps even without normalization. Removing boundary RMSNorm raises validation loss by 0.025, yet this widening variant still improves on the baseline by 0.039. (iii) Outer residual is necessary. Removing the outer residual raises validation loss by 0.032, because each write-read pair between adjacent blocks then collapses into a single MLP with boundary RMSNorm as its activation, leaving the residual state effectively unexpanded. (iv) Interface stabilization gives a modest final gain. Disabling it raises validation loss by 0.008, a relatively modest gap. Nevertheless, its advantage is much larger early in training (Appendix § 11.§ 3.2), which is the main reason we adopt it.
Related Work
Residual-state expansion. The closest methods expand the residual state, but typically by replacing vanilla addition with a learned, per-layer operator. AltUp widens the representation into sub-blocks and updates one sub-block at backbone width while a lightweight predictor updates the rest [4]. Hyper-Connections (HC) maintains multiple residual streams with learned layer-dependent read, write, and residual mixing weights [59]; Virtual Width Networks (VWN) generalizes HC by constructing fine-grained fractional streams, enabling more expressive mixing [38, 60]; mHC constrains the residual stream transition to a doubly stochastic manifold for stability [53]. Attention Residuals instead learns content-dependent attention over earlier layer or block representations [20]. Relative to this family, HiRes is simpler in mechanism: one wide outer state, dense read/write only at block boundaries, and additive residual transport. It maintains more expressive interfaces between the expanded residual and the fixed-width inner state, and amortizes the cost across depth.
Normalization in Transformers. The original Transformer normalizes after each residual addition (PostNorm), whereas modern language models typically use PreNorm to keep an uninterrupted additive path through depth [2, 31, 47, 54, 56]. There exists a well-known trade-off between PreNorm and PostNorm: PostNorm controls activation scale but can attenuate gradients, while PreNorm eases optimization but lets the residual state grow unbounded [49]. Hybrid placements pursue both, adding normalization inside the residual branches [41], around each sublayer [19], mixed within a sublayer [61], across depth [21], or across coupled PreNorm-like and PostNorm-like streams [22]. HiRes differs in granularity: complete PreNorm layers run inside a block, and PostNorm is applied only after the block-level outer residual addition, preserving ordinary inner updates while periodically resetting the outer-state scale.
Cross-layer connectivity and state aggregation. Prior work enriches information flow across depth through gated shortcuts, identity paths, or dense connectivity [14, 17, 43]. Transformer variants further aggregate earlier representations using static weights, element-wise gates, reused values, or low-rank projections [11, 26, 32, 58], while MUDDFormer mixes decoupled streams and Deep Delta Learning maintains a matrix-valued state [52, 57]. HiRes instead uses hierarchical connectivity, combining a persistent wide outer state across blocks with narrow inner states within each block.
Conclusion
We introduced Hierarchical Residuals (HiRes), a deliberately minimal architecture that decouples residual-state width from per-layer compute width. HiRes carries a wide persistent state across layer blocks, performs ordinary Transformer computation in a narrower inner space, and connects the two through learned read/write projections at block boundaries. This hierarchical design uses only dense linear maps, normalization, and residual addition, requiring neither specialized kernels nor changes to the inner Transformer layers.
Across model sizes and depths, HiRes consistently outperforms the vanilla residual baseline and is competitive with other residual expansion methods including HC, VWN, mHC, and AttnRes. These results demonstrate that residual-state width need not be tied to per-layer compute width and that their conventional equality can be suboptimal. HiRes therefore establishes residual width as a practical scaling variable that can be exploited through a simple yet effective hierarchical design.
Implementation Pseudocode
HiRes can be implemented as a simple extension of a vanilla Transformer by changing fewer than ten lines of code in the model’s forward method, as illustrated in Algorithm 1.

Figure 1 (PDF p. 15). PyTorch-style pseudocode of the HiRes forward pass.
def forward(self, input_ids):
x = self.embed(input_ids)
x_outer = self.embed_proj(x) # up-project d -> D
for b in range(self.num_blocks):
# Read at block entry, down-project D -> d
x_inner = self.read_proj[b](x_outer)
# Standard PreNorm inner residual
for i in range(self.num_layers // self.num_blocks):
x_inner = x_inner + self.attn[b][i](self.attn_norm[b][i](x_inner))
x_inner = x_inner + self.ffn[b][i](self.ffn_norm[b][i](x_inner))
# Write at block exit, up-project d -> D
x_outer = self.outer_norm[b](x_outer + self.write_proj[b](x_inner))
x = self.lm_head_proj(x_outer) # down-project D -> d
logits = self.lm_head(self.final_layernorm(x))
return logits
Experimental Setup
All dense experiments use decoder-only Transformers implemented in the OLMoE codebase [30]. Models use weight-tied input/output embeddings, RMSNorm [56], RoPE [44], SwiGLU [39] with preactivation width , grouped-query attention [1] with head dimension 64 and 4 KV heads, and per-head QK-norm [9]. Training uses AdamW [24] with , , , weight decay 0.1, and gradient-norm clipping at 1.0. The learning rate uses cosine decay to 0.1 of the peak rate after a linear warmup over 2% of training tokens. Models use a 4,096-token sequence length and initialization standard deviation 0.02. Table 4 lists the scale-dependent settings.
| Configuration | |||||
| Attention heads | 8 | 12 | 16 | 20 | 24 |
| KV heads | 4 | 4 | 4 | 4 | 4 |
| FFN width | 2,048 | 3,072 | 4,096 | 5,120 | 6,144 |
| Baseline params | 70.264M | 138.433M | 245.403M | 403.757M | 626.078M |
| Global batch size | 128 | 128 | 256 | 384 | 512 |
| Peak learning rate | |||||
| Training tokens | 30B | 50B | 100B | 200B | 350B |
Table 4. Scale-dependent training recipes.
The training corpus is the OLMo2 stage-1 mixture, tokenized with the Dolma2 tokenizer and its 100,352-token vocabulary [48]. All compared architectures at a given scale reuse the same data order and global sample-index array.
Validation loss is averaged over 11 held-out domains from Paloma [25]: C4 [33], Dolma Books, Dolma Common Crawl, Dolma Pes2o, Dolma Reddit, Dolma Stack, Dolma Wiki [42], ICE, M2D2 S2ORC [23, 34], Pile [13], and WikiText-103 [27]. Downstream accuracy is averaged over 12 tasks: MMLU [15], ARC-Challenge and ARC-Easy [7], BoolQ [6], CommonsenseQA [45], COPA [35], HellaSwag [55], OpenBookQA [28], PIQA [5], SciQ [51], SocialIQA [37], and WinoGrande [36].
Detailed Experimental Results
Scaling on Model Size
Following the model-size study in § 4.2, Table 5 provides the complete results at every evaluated scale.
Scaling on Depth
In § 4.3 we study depth scaling at fixed width; Table 6 reports the corresponding results and the follow-up sweep over the number of HiRes blocks.
| Metric | Baseline | , | , | , | , | |
| Params | 70.264M | 72.888M | – | 75.511M | – | |
| FLOPs () | 1.8691 | 1.9163 | – | 1.9635 | – | |
| Train loss | 3.4768 | 3.4515 | – | 3.4314 | – | |
| Val. loss | 3.4733 | 3.4401 | – | 3.4201 | – | |
| Params | 89.148M | 91.771M | – | 94.395M | – | |
| FLOPs () | 2.8126 | 2.8598 | – | 2.9071 | – | |
| Train loss | 3.3334 | 3.2934 | – | 3.2774 | – | |
| Val. loss | 3.3221 | 3.2860 | – | 3.2635 | – | |
| Params | 126.915M | 129.539M | 131.638M | 132.162M | 136.360M | |
| FLOPs () | 4.7004 | 4.7476 | 4.7854 | 4.7948 | 4.8704 | |
| Train loss | 3.1949 | 3.1521 | 3.1551 | 3.1421 | 3.1364 | |
| Val. loss | 3.1790 | 3.1407 | 3.1421 | 3.1285 | 3.1182 | |
| Params | 202.449M | 205.073M | 207.172M | 207.696M | 211.895M | |
| FLOPs () | 8.4759 | 8.5232 | 8.5609 | 8.5704 | 8.6459 | |
| Train loss | 3.0695 | 3.0323 | 3.0318 | 3.0185 | 3.0162 | |
| Val. loss | 3.0555 | 3.0202 | 3.0232 | 3.0019 | 2.9986 | |
| Params | 353.518M | 356.142M | 358.241M | 358.765M | 362.963M | |
| FLOPs () | 16.027 | 16.074 | 16.112 | 16.121 | 16.197 | |
| Train loss | 2.9918 | 2.9536 | 2.9258 | 2.9232 | 2.9141 | |
| Val. loss | 2.9765 | 2.9387 | 2.9154 | 2.9111 | 2.9017 |
Table 6. HiRes beats baseline at every depth, and helps most at . All models use and 30B tokens.
Comparison with Other Residual Expansion Methods
Configurations. We detail the configurations compared in § 4.4. HC- and mHC- maintain width- residual streams. VWN- has the same total virtual width but uses finer-grained streams: streams of width . We evaluate . Full AttnRes retrieves from all preceding sublayer outputs, whereas Block AttnRes-4 partitions the residual sublayers into 4 equal blocks and retains one representation per completed block. HiRes uses expansion ratio and block count throughout. Table 7 lists the parameters and training FLOPs of each configuration.
| Method | Params | FLOPs | Params | FLOPs | Params | FLOPs |
| Baseline | 138.4M | 6.419 | 245.4M | 22.78 | 403.8M | 73.62 |
| HC-2 | 138.5M | 6.421 | – | – | – | – |
| HC-4 | 138.6M | 6.423 | – | – | – | – |
| mHC-2 | 138.8M | 6.429 | – | – | – | – |
| mHC-4 | 140.3M | 6.474 | 248.7M | 22.98 | 408.9M | 74.24 |
| VWN-2 | 140.9M | 6.492 | – | – | – | – |
| VWN-4 | 143.3M | 6.564 | 254.0M | 23.30 | 417.2M | 75.24 |
| Full AttnRes | 138.5M | 6.420 | – | – | – | – |
| Block AttnRes-4 | 138.5M | 6.420 | 245.5M | 22.78 | 403.9M | 73.64 |
| HiRes | 150.2M | 6.773 | 266.4M | 24.04 | 436.5M | 77.56 |
Table 7. Parameters and training FLOPs of the compared residual expansion methods. FLOPs are reported in units of .
Per-task downstream accuracy. Table 8 reports the complete downstream-task breakdown for the four residual-expansion methods transferred across all three backbone scales.
| Backbone | Task | Baseline | mHC-4 | VWN-4 | Block AttnRes-4 | HiRes |
| (768, 12) | MMLU | 27.87 | 27.24 | <u>27.28</u> | 27.40 | 26.59 |
| ARC-Challenge | 25.98 | 24.75 | 22.97 | <u>23.63</u> | 24.75 | |
| ARC-Easy | 50.94 | <u>51.93</u> | 50.41 | 52.05 | 51.11 | |
| BoolQ | 59.67 | <u>55.84</u> | 51.27 | 56.06 | 55.18 | |
| CommonsenseQA | 32.05 | <u>34.07</u> | 31.59 | 31.61 | 34.34 | |
| COPA | 67.00 | <u>68.00</u> | 65.00 | 62.33 | 70.67 | |
| HellaSwag | 33.42 | <u>34.83</u> | <u>34.83</u> | 34.25 | 35.26 | |
| OpenBookQA | 30.60 | <u>29.40</u> | 29.07 | 30.53 | 29.27 | |
| PIQA | 64.62 | <u>65.61</u> | <u>65.61</u> | 65.52 | 65.96 | |
| SciQ | 76.10 | 81.50 | <u>78.83</u> | 77.30 | <u>78.83</u> | |
| SocialIQA | 42.73 | <u>42.84</u> | 42.39 | 42.15 | 43.09 | |
| WinoGrande | 51.91 | 51.93 | <u>51.12</u> | 50.70 | 50.72 | |
| Avg. Acc. | 46.91 | 47.33 | 45.86 | 46.13 | <u>47.15</u> | |
| (1024, 16) | MMLU | 28.20 | 28.82 | 28.89 | 28.52 | 28.40 |
| ARC-Challenge | 26.09 | 25.31 | 24.86 | <u>26.20</u> | 26.98 | |
| ARC-Easy | 56.43 | 54.56 | <u>56.43</u> | 56.96 | 55.73 | |
| BoolQ | 57.28 | 58.28 | <u>59.94</u> | 62.04 | 56.38 | |
| CommonsenseQA | 35.54 | 33.58 | <u>35.05</u> | 34.21 | 35.93 | |
| COPA | 69.00 | 69.00 | <u>69.67</u> | 70.67 | 66.00 | |
| HellaSwag | 40.18 | 42.02 | 42.14 | 40.74 | <u>42.03</u> | |
| OpenBookQA | 30.60 | 31.87 | 32.00 | 30.93 | 32.00 | |
| PIQA | 65.92 | <u>67.99</u> | 67.27 | 66.47 | 68.61 | |
| SciQ | 82.03 | 82.73 | <u>82.20</u> | 81.60 | 81.57 | |
| SocialIQA | 43.06 | <u>43.52</u> | 43.50 | 42.53 | 43.72 | |
| WinoGrande | 50.80 | 52.20 | 52.93 | 52.49 | <u>52.51</u> | |
| Avg. Acc. | 48.76 | 49.16 | 49.57 | <u>49.45</u> | 49.15 | |
| (1280, 20) | MMLU | 30.49 | 30.75 | <u>30.89</u> | 30.56 | 30.95 |
| ARC-Challenge | 29.32 | 29.10 | <u>28.65</u> | <u>28.65</u> | 27.65 | |
| ARC-Easy | 59.59 | 58.83 | <u>61.29</u> | 58.48 | 62.81 | |
| BoolQ | 58.92 | 60.71 | 57.96 | 58.37 | <u>59.71</u> | |
| CommonsenseQA | 37.37 | 37.37 | 35.54 | <u>38.36</u> | 39.53 | |
| COPA | 73.33 | 73.00 | 72.00 | 72.00 | <u>72.67</u> | |
| HellaSwag | 47.48 | <u>49.59</u> | 50.39 | 48.00 | <u>50.13</u> | |
| OpenBookQA | 32.20 | 35.47 | <u>33.20</u> | 31.73 | 33.07 | |
| PIQA | 70.44 | <u>70.86</u> | 70.78 | 69.33 | 71.13 | |
| SciQ | 85.37 | 84.67 | 87.40 | 85.10 | <u>85.53</u> | |
| SocialIQA | 43.36 | 43.36 | 44.23 | 43.88 | <u>44.17</u> | |
| WinoGrande | 54.91 | 56.14 | 55.25 | 53.33 | <u>55.75</u> | |
| Avg. Acc. | 51.90 | <u>52.49</u> | 52.30 | 51.48 | 52.76 |
Table 8. Downstream performance for different residual expansion methods. Best and second-best values are bold and <u>underlined</u>. HiRes remains competitive across model scales.
Across three backbone scales, HiRes improves downstream accuracy over the vanilla baseline by 0.24, 0.39, and 0.86 percentage points from the smallest to the largest model, achieving the highest aggregate accuracy among the compared methods at the largest scale. Individual tasks favor different methods, but HiRes delivers consistent aggregate gains and is competitive with other residual expansion methods.
Long-Horizon MoE Pretraining
For the long-horizon experiment in § 4.5, Table 9 summarizes the MoE configuration and 500B-token results.
| Setting | Value |
| Total parameters | 6.9B |
| Active parameters | 1.3B |
| Transformer layers | 16 |
| Model width | 2,048 |
| Attention heads | 16 |
| Context length | 4,096 |
| MoE experts | 64 |
| MoE top- | 8 |
| Expert hidden size | 2,048 |
| Global batch size | 768 |
| Peak learning rate | |
| Warmup ratio | 2% |
| Training tokens | 500B |
| HiRes outer width | 4,096 |
| HiRes blocks | 4 |
| HiRes added parameters | 83.9M |
| Interface initialization |
Table 9. Table 9 On OLMoE-1B-7B, HiRes improves losses and most downstream tasks after 500B tokens. (a) Model and training settings inherited from OLMoE [30], plus the HiRes configuration. (b) Losses and 12-task downstream accuracy (%). Bold: the better of the two. (a) Model and training configuration
| Metric | Baseline | HiRes (, ) |
| Train loss | 2.3663 | 2.3455 |
| Val. loss | 2.3491 | 2.3311 |
| MMLU | 41.58 | 42.19 |
| ARC-Challenge | 44.70 | 49.61 |
| ARC-Easy | 76.43 | 78.54 |
| BoolQ | 66.39 | 70.09 |
| CommonsenseQA | 49.88 | 49.66 |
| COPA | 87.67 | 86.67 |
| HellaSwag | 73.19 | 74.29 |
| OpenBookQA | 44.33 | 44.80 |
| PIQA | 78.65 | 79.67 |
| SciQ | 93.13 | 93.60 |
| SocialIQA | 47.78 | 48.43 |
| WinoGrande | 67.17 | 67.01 |
| Avg. Acc. | 64.24 | 65.38 |
Table 9. (b) Training and downstream results
Detailed Ablation Results
Ablation on Block Count and Expansion Ratio
Complementing § 5.1, Table 10 reports the expansion-ratio and block-count ablations.
| Configuration | Params | Train loss | Val. loss |
| Expansion ratio at | |||
| 142.9M | 3.2034 | 3.1735 | |
| 144.3M | 3.1869 | 3.1573 | |
| 147.3M | 3.1738 | 3.1450 | |
| 150.2M | 3.1645 | 3.1300 | |
| 153.2M | 3.1631 | 3.1306 | |
| 156.1M | 3.1586 | 3.1242 | |
| 162.0M | 3.1591 | 3.1251 | |
| Block count at | |||
| 143.2M | 3.2170 | 3.1877 | |
| 145.5M | 3.1950 | 3.1658 | |
| 147.9M | 3.1747 | 3.1426 | |
| 150.2M | 3.1645 | 3.1300 | |
| 155.0M | 3.1557 | 3.1249 | |
| 169.1M | 3.1540 | 3.1172 |
Table 10. Table 10 Ablation results on expansion ratio and block count . Loss improves quickly up to and and slowly beyond. All runs use and 50B tokens.
Architecture Ablations
Table 11 extends the architecture ablations in § 5.2 with interface, stabilization, write-rule, and transport variants.
| Variant | Train loss | Val. loss |
| Boundary normalization outer residual | ||
| Baseline | 3.2228 | 3.1941 |
| Boundary RMSNorm only (HiNorm) | 3.1956 | 3.1714 |
| Identity + outer residual | 3.2118 | 3.1807 |
| Identity + outer residual + RMSNorm | 3.1943 | 3.1686 |
| Learned interface scalar stabilization | ||
| Learned , stabilized | 3.1645 | 3.1300 |
| Learned , direct | 3.1686 | 3.1375 |
| Frozen random , stabilized | 3.1905 | 3.1614 |
| Frozen random , direct | 3.2181 | 3.1927 |
| Boundary RMSNorm scalar stabilization | ||
| With RMSNorm, stabilized | 3.1645 | 3.1300 |
| Without RMSNorm, stabilized | 3.1874 | 3.1552 |
| With RMSNorm, direct | 3.1686 | 3.1375 |
| Without RMSNorm, direct | 3.1938 | 3.1591 |
| Write rule at , | ||
| Full-output write | 3.1645 | 3.1300 |
| Increment-only write | 3.1653 | 3.1341 |
| Read/write direction at , | ||
| Complete read/write interface | 3.1869 | 3.1573 |
| Read-only interface | 3.1902 | 3.1630 |
| Write-only interface | 3.1897 | 3.1614 |
Table 11. Supplementary interface ablations at . All models train for 50B tokens.
Boundary RMSNorm and the outer residual each improve over the baseline when introduced separately, and their combination performs best in the identity-interface factorial. Stabilization improves both learned and frozen interfaces, with the larger gain for frozen random maps. Boundary RMSNorm yields a similar improvement with and without stabilization, so its contribution is not contingent on the scalar gates. The increment-only variant replaces the default write with , committing only the update accumulated inside the block, so the outer skip is exactly the identity rather than ; its result is close to full-output writing. At equal residual and compute width, both one-sided variants underperform the complete interface, indicating that learned transport in both directions is preferable.
Boundary Normalization
Figure 6 shows the boundary-frequency ablation and the per-layer residual scale discussed in § 3.4.
Figure 6. (a) Boundary-frequency ablation. (b) Per-layer residual activation scale.
Figure 6 Boundary normalization lowers loss and regulates residual scale at . (a) HiNorm training loss under different boundary normalization frequencies. (b) Boundary normalization regulates residual accumulation as in other residual-expansion methods; dotted lines mark the HiRes block boundaries .
Interface Stabilization Dynamics
The scalar gates in Equation (7) bias each read/write map toward the coordinate-aligned identity at initialization. We compare the default initialization, , with direct dense interfaces. Figure 7 shows that direct interfaces initially train worse than the baseline and only later cross below it. The identity-biased parameterization starts closer to the baseline and remains better throughout the early optimization phase. Stabilization therefore prevents newly initialized interfaces from immediately disrupting residual transport. The final results in Table 11 further show that stabilization is an optimization aid rather than a requirement for the architecture.
Figure 7. Identity-biased initialization stabilizes early interface training. Training loss of the baseline and of HiRes with direct or identity-biased interfaces under . (a) Full training loss trajectories. (b) Loss difference from the baseline. Identity-biased interfaces start near baseline and maintain lower training loss than direct ones.
Parameters, Compute, Memory, and I/O Costs
Parameters, compute, and memory. HiRes leaves all attention and FFN modules unchanged. Its additional parameters come from block interfaces, two endpoint projections, outer RMSNorm, and stabilization scalars:
For the OLMoE-1B-7B model in § 4.5, where , this amounts to 83.9M additional parameters, approximately 6.5% of the model’s 1.3B active parameters and 1.2% of its 6.9B total parameters. The dominant additional training compute comes from the forward and backward passes through the dense interface projections. Over training tokens, this amounts to
excluding the lower-order normalization and scalar-gate operations. Relative to the vanilla residual baseline, HiRes additionally stores only the outer states at the block boundaries. For a micro-batch of tokens, this activation-memory overhead is
Activation I/O. Let denote one sublayer; an -layer Transformer has sublayers. We count forward activation elements per token, excluding parameter reads and ’s internal traffic. For HiRes, the count includes its two endpoint maps. Following AttnRes [20], an input constructed only for is passed to it directly rather than written, whereas residual states reused by later merges are written. Table 12 summarizes the result for each method, and the paragraphs below derive each row.
| Method | Total reads | Total writes | Average I/O per sublayer | Typical values |
| Baseline | ||||
| HC (fused) | ||||
| mHC (fused) | ||||
| VWN (fused) | ||||
| Full AttnRes (two-phase) | ||||
| Block AttnRes (two-phase) | ||||
| HiRes (fused) |
Table 12. HiRes reaches the lowest activation I/O among residual-expansion methods. We compare optimized implementations for all methods and count residual-related activation reads and writes. is the HC/mHC/VWN expansion rate, the number of AttnRes blocks, and and the HiRes expansion ratio and block count. The fourth column divides total I/O by the sublayers; typical values use , , , and .
Residual and HiRes. A vanilla merge moves elements per sublayer, or in total. HiRes retains this cost for inner merges. At each block boundary, the read projection reads elements and writes elements; fusing the write projection, outer addition, and RMSNorm then reads elements and writes elements.
Each block therefore incurs elements of traffic, and the two endpoint maps move another , giving the overhead
and the total
HC, mHC and VWN. HC [59] maintains width- residual streams, . Its three connections are generated from the current state,
with pre-connection , post-connection , and a residual connection :
Following the dynamic HC parameterization, HC constructs three connections by self-gated projections. Taking as an example:
where is a scaling factor, is a bias, and is the projection weight.
In mHC [53], normalizes , jointly projects it to logits, applies sigmoid transforms to the pre/post logits, and applies Sinkhorn–Knopp iterations to .
In VWN [38], each stream is split into parts of width , forming . The remaining operations are similar to HC.
The fused schedules below are our analytic constructions from these published recurrences rather than traffic reported by the source papers. For each method, we can pack the three projections in (13) and fold (15) into one kernel, reading elements once without materializing the normalized and intermediate states. Because is available only after the projection has reduced over all of , the subsequent pre-connection in (14) makes a second read and writes . After , the residual connection and post-connection are fused into one kernel that reads and writes . Thus the activation traffic for each sublayer is
and over the network,
This omits the lower-order coefficient traffic.
AttnRes. Full AttnRes [20] stores every preceding layer output and uses a learned query to retrieve across depth:
Block AttnRes instead stores one completed representation per architecture block and includes the current block’s partial sum as an additional source.
For the two-phase schedule, partition the sublayers into execution blocks of sublayers. Phase 1 batches the fixed queries in a block and attends to completed history once, returning one -wide output and softmax statistics per sublayer. Phase 2 attends to the sequentially available intra-block history and merges both phases using online softmax [29]. For Full AttnRes, execution block has completed outputs. Phase 1 reads these once for all queries, contributing elements per block, and writes one -wide result per sublayer. Within a block, sublayer reads earlier intra-block outputs in Phase 2, which also reads back its Phase 1 result for the merge and passes directly to . Summing both triangular histories over depth gives
Here the two phases each write one -wide result per sublayer. Block AttnRes instead reads only the completed block summaries in Phase 1. Summing this history gives reads; its fused partial-sum attention and online-softmax merge additionally read and write the -wide partial sum per sublayer. Thus
Agent-Driven Optimization of Residual Mechanisms
To assess the practical efficiency and ease of adoption of each residual expansion mechanism, we measure both its final training efficiency and the agent-driven optimization required to reach it, comparing framework-level PyTorch and mechanism-specific Triton implementations.
Experimental Protocol
We evaluate on eight GPUs using FSDP FULL_SHARD, bfloat16, sequence length 4096, and per-GPU micro-batch 4. Every measurement includes forward, backward, and the optimizer update; peak memory is reported per GPU. The attention, MLP, normalization, optimizer, precision, and measurement harness are shared, and only mechanism-specific code may change.
Starting from functionally equivalent naive implementations, independent Claude Opus 5 agents receive the same prompt and optimize HiRes, VWN-4, mHC-4, and Block AttnRes-4 in two stages: the PyTorch stage that may leverage graph transformations, torch.compile, batching, and layout changes; and the Triton stage that adds mechanism-specific fused kernels. Every candidate must match the reference outputs and gradients at all three scales. Candidates are measured against the current best and retained only if throughput improves by at least 0.2% at every scale. We report the best implementation from each stage. The final optimized Triton implementation is used to measure the efficiency of each residual expansion mechanism in Table 2.
Optimization Trajectory
We visualize the optimization campaign in Figure 8. Three patterns emerge. First, HiRes maintains the highest throughput among the residual-expansion methods throughout most of the campaign, until the fully optimized Block AttnRes overtakes it near the end. Second, HiRes reaches its best implementation in less optimization time than the other methods in our campaign. Third, HiRes’ PyTorch implementation already matches its final Triton throughput, with custom kernels primarily reducing memory. Other methods instead rely on Triton for substantial throughput gains.

Figure 8. Agent-driven optimization campaign. (a) Throughput (% of the frozen baseline). (b) Peak-memory overhead at versus optimization time. Dots are accepted improvements; solid and dashed segments are the PyTorch and Triton stages; square, diamond, and star mark the best PyTorch, best Triton, and closing implementations. HiRes reaches its best throughput early and without custom kernels.
Practical Efficiency
Table 13 compares the best PyTorch-stage and fused Triton implementations across scales. Without custom kernels, HiRes already achieves 91.7–93.0% of baseline throughput—the highest among the residual-expansion methods—with only 5.5–5.7% peak-memory overhead. Triton fusion preserves this throughput while reducing the memory overhead to 3.3–3.4%. The resulting implementation approaches the throughput of Block AttnRes-4, exceeds that of mHC-4 and VWN-4, and has the lowest memory overhead among the compared methods.
| Method | Stage | Throughput (% of baseline) (768, 12) | Throughput (% of baseline) (1024, 16) | Throughput (% of baseline) (1280, 20) | Peak memory (% over baseline) (768, 12) | Peak memory (% over baseline) (1024, 16) | Peak memory (% over baseline) (1280, 20) |
| mHC-4 | PyTorch | 58.1 | 56.4 | 55.2 | +19.1 | +25.2 | +29.7 |
| Triton | 86.5 | 88.2 | 86.1 | +20.8 | +27.0 | +31.4 | |
| VWN-4 | PyTorch | 68.3 | 69.3 | 69.2 | +47.6 | +58.7 | +66.2 |
| Triton | 81.6 | 84.4 | 80.5 | +28.1 | +34.2 | +38.4 | |
| Block AttnRes-4 | PyTorch | 80.3 | 77.4 | 75.1 | +3.7 | +4.9 | +5.8 |
| Triton | 93.7 | 94.9 | 94.5 | +5.3 | +6.9 | +8.0 | |
| HiRes | PyTorch | 91.7 | 92.7 | 93.0 | +5.5 | +5.7 | +5.5 |
| Triton | 91.7 | 92.8 | 93.1 | +3.4 | +3.4 | +3.3 |
Table 13. Table 13 Best PyTorch- and Triton-stage performance of each method. Throughput and peak-memory overhead are relative to the frozen baseline. Within each stage, best/second-best values are bold/underlined. HiRes leads in throughput and memory without custom Triton kernels.
Latency Decomposition
Figure 9 separates measured kernel time from analytic compute- and bandwidth-limited floors. HiRes spends most of its overhead in the dense interface GEMMs, whereas the comparison methods spend most of theirs moving and mixing expanded states at every sublayer. This explains why fusion materially benefits the latter, while HiRes is already close to its fused performance with framework-level optimizations.
Figure 9. Figure 9 Extra kernel time per step at (1280, 20) over the shared frozen backbone (242.5 ms). (a) Fusion (best Triton versus best PyTorch) cuts the overhead of the other methods by 2–6× but leaves HiRes nearly unchanged. (b) Fused overhead by kernel type (upper bars) and its compute- and bandwidth-limited floors at the GPU’s advertised bf16 and HBM peaks (lower bars): HiRes spends most of it in cuBLAS GEMMs, the others in their own mixing kernels.
A Latent View
Besides expanding the residual state width, HiRes also admits a dual interpretation: relative to a -dimensional persistent hidden state, its attention and FFN computation occurs in a -dimensional latent space. Expanding residual width by is therefore equivalent to compressing compute width by a factor of ; the wide state preserves information across blocks, while the narrow state performs the transformations within each block.
This view connects HiRes to module-specific latent designs. MLA compresses the attention KV representation [8], while LatentMoE performs routed expert computation in a compressed space [10]. HiRes extends the same separation between state width and compute width to the residual hierarchy. Its key distinction is interaction granularity: instead of exchanging information between hidden and latent spaces at every sublayer, HiRes reads once at block entry and writes once at block exit. Layer grouping therefore reduces hidden-latent transport from per-sublayer to per-block while allowing computation to evolve for multiple layers in the latent space.
Limitations
Our study has the following limitations. First, HiRes decouples residual width from compute width, but we have not jointly optimized these widths and model depth under a fixed compute budget. The best allocation of capacity between carrying representations and transforming them therefore remains unclear. Second, we use equal-sized blocks, so exchanges between the wide state and the inner computation occur at regular intervals. We have not examined whether different parts of the network benefit from more frequent access to the wide state. Finally, our results show the benefits of residual expansion, yet how the wider residual state improves learning remains an open question. In particular, what information is preserved across blocks and how individual sublayers contribute to its transformation and reuse are not yet well understood.
References
- [1]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sangh. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298.DOI
- [2]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.arxiv.org/abs/1607.06450
- [3]Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: fast convergence at large depth. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352–1361. PMLR, 27–30 Jul 2021.arxiv.org/abs/2003.04887
- [4]Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating updates for efficient transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 76718–76736. Curran Associates, Inc., 2023. doi: 10.52202/075280-3352.DOI
- [5]Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, April 2020. ISSN 2159-5399. doi: 10.1609/aaai.v34i05.6239.DOI
- [6]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300.DOI
- [7]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
- [8]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenqiu Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruiyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanmhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. W. Y. K. Li, Y. X. W. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.arxiv.org/abs/2405.04434
- [9]Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdlmoushin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Briodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kıpf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7480–7512. PMLR, 23–29 Jul 2023.
- [10]Vennmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, and Bita Rouhani. Latentmoe: Toward optimal accuracy per flop and parameter in mixture of experts, 2026.arxiv.org/abs/2601.18089
- [11]Yanwen Fang, Yuxi CAI, Jintai Chen, Jingyu Zhao, Guangjian Tian, and Guodong Li. Cross-layer retrospective retrieving via layer attention. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2302.03985
- [12]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
- [13]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.arxiv.org/abs/2101.00027
- [14]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.arxiv.org/abs/1512.03385
- [15]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
- [16]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Henighan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc., 2022. doi: 10.52202/068431-2176.
- [17]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.arxiv.org/abs/1608.06993
- [18]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.arxiv.org/abs/2001.08361
- [19]Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-LN: Revisiting normalization layer in the transformer ar- chitecture. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30400–30436. PMLR, 13–19 Jul 2025.arxiv.org/abs/2502.02732
- [20]Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyun Wang, Yucheng Wang, Guangduo Chen, Bohong Yin, Yutian Chen, Junjie Yan, Ming Wei, Y. Zhang, Fanging Meng, Chao Hong, Xiaotong Xie, Shaowei Liu, Enzhe Lu, Yunpeng Tai, Yanru Chen, Xin Men, Haiqing Guo, Y. Charles, Haoyu Lu, Lin Sui, Jinguo Zhu, Zaida Zhou, Weiran He, Weixiao Huang, Xinran Xu, Yuzhi Wang, Guokun Lai, Yulun Du, Yuxin Wu, Zhilin Yang, and Xinyu Zhou. Attention residuals, 2026.arxiv.org/abs/2603.15031
- [21]Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-In: Unleashing the power of deeper layers by combining pre-ln and post-ln. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 98244–98258, 2025.arxiv.org/abs/2412.13795
- [22]Tianyu Li, Dongchen Han, Zixuan Cao, Haofeng Huang, Mengyu Zhou, Ming Chen, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang, and Gao Huang. Siamesenorm: Breaking the barrier to reconciling pre/post-norm, 2026.arxiv.org/abs/2602.08064
- [23]Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.447.DOI
- [24]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
- [25]Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannah Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 64338–64376. Curran Associates, Inc., 2024. doi: 10.52202/079017-2052.DOI
- [26]Gaurav Menghani, Ravi Kumar, and Sanjiv Kumar. LaUReL: Learned augmented residual layer. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 43826–43836. PMLR, 13–19 Jul 2025.arxiv.org/abs/2411.07501
- [27]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.arxiv.org/abs/1609.07843
- [28]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, October–November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1260.
- [29]Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax, 2018.arxiv.org/abs/1805.02867
- [30]Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah Smith, Pang Wei Koh, Amanpreet Singh, and Hanna Hajishirzi. Olmo: Open mixture-of-experts language models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 62061–62121, 2025.arxiv.org/abs/2409.02060
- [31]Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2–3 2019. Association for Computational Linguistics.DOI
- [32]Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 136479–136508. Curran Associates, Inc., 2024. doi: 10.52202/079017-4336.DOI
- [33]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
- [34]Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. M2D2: A massively multi-domain language modeling dataset. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 964–975, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.63.DOI
- [35]Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense Reasoning: Papers from the AAAI Spring Symposium, volume SS-11-06 of AAAI Technical Report, pages 90–95, 2011.
- [36]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, August 2021. ISSN 1557-7317. doi: 10.1145/3474381.DOI
- [37]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454.DOI
- [38]Seed, Baisheng Li, Banggu Wu, Bole Ma, Bowen Xiao, Chaoyi Zhang, Cheng Li, Chengyi Wang, Chengyin Xu, Chi Zhang, Chong Hu, Daoguang Zan, Defa Zhu, Dongyu Xu, Du Li, Faming Wu, Fan Xia, Ge Zhang, Guang Shi, Haobin Chen, Hongyu Zhu, Hongzhi Huang, Huan Zhou, Huanzhang Dou, Jianhui Duan, Jianqiao Lu, Jianyu Jiang, Jiayi Xu, Jiecao Chen, Jin Chen, Jin Ma, Jing Su, Jingji Chen, Jun Wang, Jun Yuan, Juncai Liu, Jundong Zhou, Kai Hua, Kai Shen, Kai Xiang, Kaiyuan Chen, Kang Liu, Ke Shen, Liang Xiang, Lin Yan, Lishu Luo, Mengyao Zhang, Ming Ding, Mofan Zhang, Nian Ning Liang, Peng Li, Penghao Huang, Pengpeng Mu, Qi Huang, Qianli Ma, Qiyang Min, Qiying Yu, Renming Pang, Ru Zhang, Shen Yan, Shen Yan, Shixiong Zhao, Shuaishuai Cao, Shuang Wu, Siyan Chen, Siyu Li, Siyuan Qiao, Tao Sun, Tian Xin, Tiantian Fan, Ting Huang, Ting-Han Fan, Wei Jia, Wenqiang Zhang, Wenxuan Liu, Xiangzhong Wu, Xiaochen Zuo, Xiaoying Jia, Ximing Yang, Xin Liu, Xin Yu, Xingyan Bin, Xintong Hao, Xiongcai Luo, Xujing Li, Xun Zhou, Yanghua Peng, Yangrui Chen, Yi Lin, Yichong Leng, Yinghao Li, Yingshuan Song, Yiyuan Ma, Yong Shan, Yongan Xiang, Yonghui Wu, Yongtao Zhang, Yongzhen Yao, Yu Bao, Yuehang Yang, Yufeng Yuan, Yunshui Li, Yuqiao Xian, Yutao Zeng, Yuxuan Wang, Zehua Hong, Zehua Wang, Zengzhi Wang, Zeyu Yang, Zhengqiang Yin, Zhenyi Lu, Zhexi Zhang, Zhi Chen, Zhi Zhang, Zhiqi Lin, Zihao Huang, Zilin Xu, Ziyun Wei, and Zuo Wang. Virtual width networks, 2025.arxiv.org/abs/2511.11238
- [39]Noam Shazeer. Glu variants improve transformer, 2020.arxiv.org/abs/2002.05202
- [40]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
- [41]Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization, 2021.arxiv.org/abs/2110.09456
- [42]Luca Soldaini, Rodney Kinney, Akshita B加ghia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannah Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.840.DOI
- [43]Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015.arxiv.org/abs/1505.00387
- [44]Jianlin Su, Muratdha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, February 2024. ISSN 0925-2312. doi: 10.1016/j.neucom.2023.127063.DOI
- [45]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421.DOI
- [46]Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 32–42, October 2021.arxiv.org/abs/2103.17239
- [47]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762
- [48]Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Validad Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious (COLM’s version). In Second Conference on Language Modeling, 2025.
- [49]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, October 2024. ISSN 1939-3539. doi: 10.1109/tpami.2024.3386927.DOI
- [50]Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Anna Korhonen, David Traum, and Lluis Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1176.DOI
- [51]Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4413.DOI
- [52]Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. MUDDFormer: Breaking residual bottlenecks in transformers via multiway dynamic dense connections. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 68440–68458. PMLR, 13–19 Jul 2025.arxiv.org/abs/2502.12170
- [53]Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Mingyu Xu, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. mHC: Manifold-constrained hyper-connections. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2512.24880
- [54]Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524–10533. PMLR, 13–18 Jul 2020.arxiv.org/abs/2002.04745
- [55]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472.DOI
- [56]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.arxiv.org/abs/1910.07467
- [57]Yifan Zhang, Yifeng Liu, Mengdi Wang, and Quanquan Gu. Deep delta learning, 2026.arxiv.org/abs/2601.00417
- [58]Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. Value residual learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1375.
- [59]Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 97183–97219, 2025.arxiv.org/abs/2409.19606
- [60]Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, and Xun Zhou. Frac-connections: Fractional extension of hyper-connections, 2025.arxiv.org/abs/2503.14125
- [61]Zhijian Zhuo, Yutao Zeng, Ya Wang, Sijun Zhang, Jian Yang, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Hybridnorm: Towards stable and efficient transformer training via hybrid normalization. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 25106–25147. Curran Associates, Inc., 2025. doi: 10.52202/085713-0844.DOI






