WorksComputer Science
M3Former: Fine-Grained Cross-Layer Mixing for Transformers
Abstract
Residual connections give each Transformer block access to all earlier layers only through one fixed sum, broadcast to all of the block's inputs. This broadcast bottleneck imposes three constraints: the query, key, value and residual inputs of a block, which we call its roles, read the same vector; all hidden dimensions within a role share one set of weights over earlier layers; and every earlier layer has the same fixed weight for every token. We propose the Multi-Role Multi-Group Mixing Residual (M3), which removes all three: every role, and every group of hidden dimensions within a role, reads its own token-dependent mixture of all earlier layers. M3 starts as an exact residual connection and adds as little as 1.4% parameters and 1.8% FLOPs. From 100M to 1.4B parameters, M3Former outperforms the Transformer and coarse-grained cross-layer connections at every size, matching a Transformer 1.19-1.34x the size of the one it modifies, trained with 1.4-1.8x the compute. Relaxing only some of the constraints recovers only part of this gain. M3Former also improves sparse mixture-of-experts models of up to 4.6B parameters and trains more stably in deeper networks. Code: https://github.com/Kingchet10/m3former
Introduction
Residual connections {(citep)}(He et al., 2016) allow Transformers {(citep)}(Vaswani et al., 2017) to be trained at great depth. Each block adds its output to a running sum, and the next block reads this sum as its input. In a pre-norm Transformer, the input of a block is thus the token embedding plus the updates of all earlier blocks. This running sum, often called the residual stream, is the only path through which a block can use what earlier layers have computed {(citep)}(Elhage et al., 2021). Although language models have grown to billions of parameters {(citep)}(Brown et al., 2020; Kaplan et al., 2020; Hoffmann et al., 2022), nearly all of them still connect their blocks in this way.
Every block reads only this sum and broadcasts it to all of its inputs, although the readers of the stream need different information. The broadcast constrains a block in three ways (Figure 2a). (1) Roles: queries and keys decide where to attend, values decide what to move, and the residual path carries the state that the block updates {(citep)}(Vaswani et al., 2017; Elhage et al., 2021), yet all four read the same vector. (2) Groups: different subspaces of the stream carry different features {(citep)}(Elhage et al., 2021), and heads and layers specialize in different functions {(citep)}(Voita et al., 2019; Clark et al., 2019b; Tenney et al., 2019; Jawahar et al., 2019), yet all hidden dimensions of a role share one set of weights over earlier layers. (3) Layer weights: every earlier layer enters the sum with the same fixed weight for every token, so a block cannot emphasize the layers it needs; in pre-norm models the growing sum dilutes each layer {(citep)}(Kimi Team, 2026), and many deep layers can be removed with little loss {(citep)}(Gromov et al., 2024). We call these constraints the broadcast bottleneck.

Figure 2. M3: multi-role multi-group mixing. (a) The three constraints: a residual connection gives every role and group the same fixed sum , whereas M3 gives each of the role–group pairs its own token-dependent mixture. (b) Block assembles each of its four role inputs from group-wise mixtures of the history . (c) A role–group mixer weights group of every earlier state by and sums them. (d) A generator computes all coefficients from ; its initialization reproduces the standard Transformer. is shown; coefficient profiles are schematic.
Cross-layer connections relax these constraints one at a time, and each relaxation has improved language modeling. Most relax (3), with static {(citep)}(Wang et al., 2019; Paglialdini et al., 2024) or token-dependent weights over earlier layers {(citep)}(Zhu et al., 2025a; Xie et al., 2025; Kimi Team, 2026), some also relax (1) by forming the query, key and value from separate mixtures {(citep)}(Heddes et al., 2025; Xiao et al., 2025), or (2) by splitting the hidden state into fractions, heads or streams {(citep)}(Zhu et al., 2025b; Luo et al., 2026; Qin et al., 2026). None relaxes all three (Table 1). Yet every group of every role needs different information from different earlier layers, and this need changes from token to token; keeping any one constraint still forces some of them to share a mixture, so removing the bottleneck requires relaxing all three together.
| Method | (1) Roles | (2) Groups | (3) Weights | Sources |
| Residual | – | – | fixed | |
| DLCL, DenseFormer | – | – | static | all |
| Hyper-conn., mHC | – | – | dynamic | streams |
| Frac-connections | – | ✓ | dynamic | |
| AttnRes | – | – | dynamic | blocks |
| Multi-head AttnRes | – | ✓ | dynamic | all |
| Value residual | V | – | static | layer 1 |
| DeepCrossAttn. | Q,K,V | – | static | both |
| MUDDFormer | Q,K,V,R | – | dynamic | all |
| M3 (ours) | Q,K,V,R | ✓ | dynamic | all |
Table 1. Cross-layer connections and the three constraints. Static weights are learned but the same for every token; dynamic weights depend on the token.
In this work, we propose the Multi-Role Multi-Group Mixing Residual (M3), which relaxes all three constraints jointly: every role of a block, and every group of hidden dimensions within a role, combines all earlier layers with its own weights, generated from the current token (Figure 2). Because it starts as an exact residual connection, M3 can replace the residual connections of a Transformer without other changes; we call the result M3Former.
We pre-train M3Former, the Transformer and coarse-grained cross-layer connections from 100M to 1.4B parameters under one recipe, together with mixture-of-experts models of up to 4.6B parameters, and make three contributions:
Problem. We identify the broadcast bottleneck, three constraints that the residual connection places on how a block reads earlier layers, and show that each of them limits the Transformer: relaxing only one or two recovers only part of the gain (§ 5.1).
Method. We propose M3, in which every role–group pair of a block reads its own token-dependent mixture of all earlier layers. One small generator produces all mixtures, and a fused kernel computes them in a single pass over the history. The resulting M3Former starts exactly as the standard Transformer and adds as little as 1.4% parameters and 1.8% FLOPs.
Evidence at scale. At 20 tokens per parameter, M3Former matches a Transformer 1.19–1.34× the size of the one it modifies (Figure 3) and outperforms attention residuals from 100M to 1.4B and MUDDFormer from 100M to 700M (Figure 1). It also improves mixture-of-experts models of up to 4.6B parameters, and it trains to completion at 1.4B, where MUDDFormer diverges under the same recipe, and stays ahead when both use PrePostNorm.

Figure 1. Validation loss at 20 training tokens per parameter (active parameters for mixture-of-experts models). In the bars: losses; red: change of M3Former relative to the Transformer; hatched: diverged run; white slash: bar cut off at the top. Each panel spans 0.12 in loss, the last 0.06.
Figure 3. Equivalent size (§ 3) at 20 tokens per parameter, against the size of the Transformer that each model modifies.
M3: Multi-Role Multi-Group Mixing
We first write the residual connection as a read of a history of hidden states (§ 2.1), then define depth mixing and its granularity (§ 2.2), describe how M3 computes and initializes its mixing coefficients (§ 2.3), and finally analyze its cost (§ 2.4).
The residual connection reads a history
Consider a pre-norm Transformer with blocks and hidden size . Let be the token embedding and the hidden state after block ; all quantities are per token, and we write the token index only where it matters. Block has access to the history , but a standard block reads only its last element:
where , and are RMSNorms, and are the query, key and value projections. Because , reading the latest state is the same as adding the updates of all earlier blocks with equal, fixed weights. The block uses this one vector in four roles: as the input of the query, key and value projections, and as the residual path to which the attention output is added.
M3 gives the four roles separate inputs , :
where reuses the attention norm of Eq. (1), while and use two new RMSNorms, so that each projected role has its own scale. The residual input is added without normalization, and the feed-forward step is unchanged. With for every role and the scales of and equal to that of , Eq. (2) reduces to Eq. (1). Block 1 has only to read and is left unchanged.
Depth mixing and its granularity
We split the channels of every hidden state into contiguous groups of width , , and form each group of each role input as a weighted sum over the history (Figure 2c):
The groups partition the hidden channels before the query, key and value projections, so they are not attention heads: a role input is reassembled from its groups and only then normalized and projected as in Eq. (2).
Eq. (3) defines a family of connections that relax the three constraints of §1 to different degrees (Figure 2a):
Residual connection: , one fixed mixture for all roles and groups; relaxes none.
Dense mixing: , one mixture of all earlier layers shared by all roles and groups; relaxes (3). Its weights are static in DenseFormer (Pagliardini et al., [24]) and token-dependent in attention residuals (Kimi Team, [18]).
Multi-role: , four mixtures, one per role, as in MUDDFormer (Xiao et al., [38]); relaxes (1) and (3).
Multi-group: , mixtures shared by all roles, as in multi-head attention residuals (Luo et al., [23]); relaxes (2) and (3).
M3: no sharing, mixtures, one for every role and group; relaxes all three.
We call the number of independent mixtures per block and token the granularity of a connection. The family is nested: M3 represents every coarser member exactly (Proposition 2), and readers that share a coefficient receive the sum of their gradients for it, which cancel in part when they need different sources. With token-dependent coefficients, existing cross-layer connections relax at most two of the three constraints; DeepCrossAttention (Heddes et al., [12]) makes only its token-independent coefficients specific to channels. Table 1 compares them on all three, and §6 discusses them.
M3 relaxes all three: in it, every coefficient is specific to a role and a group and depends on the token (§2.3).
Token-conditioned weights
A single generator per block produces all coefficients of Eq. (3) from the latest state of the same token (Figure 2d):
computed separately for every token, where collects the coefficients , flattens it into a row vector of length , is an RMSNorm without a learned scale, , , and is a learned static prior. The hidden width is the smallest multiple of 64 that exceeds , so the generator is small. The coefficients are signed and are not normalized over depth: a role can subtract an earlier representation as easily as it adds one, which a convex mixture cannot do (Lemma 1 in Appendix § 9). The hidden state that enters the output layer is mixed in the same way from , with a single role and its own generator.
Initialization. We set , draw from , and set to one on the latest state and zero elsewhere. Then for every role, and M3Former computes exactly the same function as the standard Transformer at the start of training (Proposition 1); every departure from a single shared read is learned.
Number of groups. We count in units of the number of attention heads, : from 100M to 700M we train , that is, groups of width 64, 32 and 16, and at 1.4B ; we report the best at each size (§5.2, Table 5). The groups are not tied to the heads, since they partition the channels of the history before the projections.
Complexity analysis
Per token, block spends multiply–adds in the generator, equal to the number of entries of its weight matrices, and in the mixing of Eq. (3). Finer granularity is cheap for two reasons. First, a group-wise mixture touches the same values as a full-width one, so M3 spends the same multiply–adds on mixing as multi-role mixing; its mixtures cost more only in the output layer of the generator, instead of . Second, every member of the family reads the same earlier states, and our fused Triton kernel reads each once and writes all four roles and groups in one pass, so its reads and writes of hidden states do not grow with .
Summed over the network, M3 adds a fraction of the non-embedding parameters of the Transformer, which grows with depth relative to width (Appendix § 9.§ 4). For the models we report, it adds 1.4–3.9% parameters and 1.8–4.3% multiply–adds per token (Table 5), against 0.6–1.0% and 1.0–1.4% for MUDDFormer. At inference, a token keeps only its own hidden states, so the extra memory does not grow with the context length.
| Size | Params | MACs | Fit | Observed | |
| 100M | 1.44% | 1.85% | 0.0043 | 0.0850 | |
| 300M | 1.40% | 1.78% | 0.0033 | 0.0542 | |
| 500M | 1.40% | 1.80% | 0.0031 | 0.0412 | |
| 700M | 3.86% | 4.25% | 0.0081 | 0.0438 | |
| 1.4B | 1.52% | 1.91% | 0.0029 | 0.0308 |
Table 5. Overhead of M3 at the reported , relative to the non-embedding parameters and per-token weight multiply–adds (MACs) of the Transformer, and loss reduction at : from the extra parameters by Eq. (7) (Fit) and observed over the Transformer.
Experimental Setup
Data and training. We train all models on a fixed sample of FineWeb-Edu ([26]), tokenized with the 32K-token TinyLlama tokenizer ([43]), with sequences of 2,048 tokens and a global batch of 64 sequences. All models use AdamW with a warmup–stable–decay schedule ([14]) and one preset learning rate per size, shared by all methods (Appendix § 10). The training budget of each size is set by the non-embedding parameter count of its Transformer. The main run trains on tokens, close to the compute-optimal ratio of Hoffmann et al. [13], and runs with and tokens decay from checkpoints of the main run. We report the loss on 99.9M held-out tokens.
Models. Table 2 lists the five dense sizes and two mixture-of-experts models. The Transformer is a pre-norm decoder with RMSNorm ([42]), SwiGLU feed-forward layers ([29]) and rotary position embeddings ([31]) (Appendix § 10). Where we report one M3Former per size, we use its best . In the mixture-of-experts models, every feed-forward layer routes each token to 8 of 64 SwiGLU experts, without a shared expert (Table 7); they also train on 20 tokens per parameter, counting active parameters, and M3 uses in both, 64 and 96 groups.
| Model | FFN | ||
| 100M | 1792 | 81M | |
| 300M | 2816 | 308M | |
| 500M | 3200 | 458M | |
| 700M | 3520 | 642M | |
| 1.4B | 4608 | 1363M | |
| 1.8B-A0.38B | 1.76B/0.31B | ||
| 4.6B-A0.89B | 4.49B/0.80B |
Table 2. Models. is the non-embedding parameter count of the Transformer, total/active for the mixture-of-experts models, whose FFN column gives the number and width of experts. All heads have width 64.
| Small | Large | |
| Blocks | 24 | 28 |
| Width | 1024 | 1536 |
| Attention heads (MHA) | 16 | 24 |
| Head dimension | 64 | 64 |
| Experts per layer | 64 | 64 |
| Experts per token | 8 | 8 |
| Expert width | 352 | 512 |
| Expert activation | SwiGLU | SwiGLU |
| Shared experts | none | none |
| Vocabulary | 32,000 | 32,000 |
| Embedding and output | untied | untied |
| Sequence length | 2,048 | 2,048 |
| Total / active params | 1.83B / 0.38B | 4.59B / 0.89B |
| M3 groups | 64 | 96 |
Table 7. Mixture-of-experts architectures. Every feed-forward layer is a mixture-of-experts layer, and experts are not shared across layers.
Baselines. We compare M3Former with two coarse-grained cross-layer connections. MUDDFormer ([38]) lets each role read its own token-dependent mixture of earlier layers, shared by all of its channels; it is M3 with ,
with the same block and normalization. We train it with the same pre-norm recipe as all other models. Attention residuals (Kimi Team, [18]) replace the residual sum with softmax attention over the outputs of earlier layers; we train their block variant, with blocks of four layers, from 100M to 1.4B. For the ablation in § 5.1, we also train multi-group mixing, in which each group reads its own mixture, shared by all roles.
Equivalent size. To express a loss as a model size, we fit to the five Transformers (, , ; RMSE 0.002). The equivalent size of a model is , where is the size of the Transformer that the model modifies (Table 2) and the size of the Transformer that this fit predicts to reach the same loss; since that Transformer is also trained on 20 tokens per parameter, it uses times the training compute (Appendix § 9.§ 5).
Results
Scaling with model size
Figure 1 compares the four models at 20 tokens per parameter, and M3Former reaches the lowest loss at every size. It lowers the loss of the Transformer by 0.085, 0.054, 0.041, 0.044 and 0.031 from 100M to 1.4B, 6–47% more than MUDDFormer up to 700M; at 1.4B, MUDDFormer diverged, and with PrePostNorm for both, M3Former stays 0.010 ahead (§ 5.3). In equivalent size (Figure 3), M3Former matches a Transformer 1.19–1.34× the size of the one it modifies, or 1.41–1.78× its training compute, against 1.14–1.31× for MUDDFormer. Spent on a larger Transformer, its 1.4–3.9% extra parameters would lower the loss by only 0.003–0.008 according to the fit (Appendix § 9.§ 5). Attention residuals help most at 100M (0.047, or 1.17×), but by at most 0.020 from 300M to 1.4B (1.00–1.09×).
Scaling with data
Figure 4 follows each model as its training data grow from 5 to 20 tokens per parameter. The curve of M3Former lies below that of the Transformer at every size and budget, by 0.031–0.109. The gain is not just a faster start: from 300M on, it shrinks by at most 0.013 between 5 and 20 tokens per parameter, while the loss itself falls by 0.16–0.20.
Figure 4. Validation loss against training tokens, at 5, 10 and 20 tokens per parameter for every size. M3 uses the best of each size.
Mixture-of-experts models
Fine-grained mixing also helps sparse models. On the two mixture-of-experts models of Table 2, M3 lowers the loss by 0.010 and 0.008 (Figure 1, right). In the small model, MUDDFormer instead raises the loss by 0.024, and block attention residuals by 0.133. Both use the dense recipe without tuning, whereas Kimi Team [18] report gains for attention residuals with their own mixture-of-experts recipe.
Downstream evaluation
We evaluate zero-shot and five-shot accuracy with the LM Evaluation Harness ([9]) on the tasks used by Xiao et al. [38]: LAMBADA ([25]), PIQA ([2]), WinoGrande ([28]), ARC-Easy and ARC-Challenge ([6]), SciQ ([37]), LogiQA ([20]), BoolQ ([4]), HellaSwag ([41]), and the middle-school part of RACE ([19]). Table 3 reports zero-shot accuracy of the 700M and 1.4B dense models and both mixture-of-experts models, and Table 8 in Appendix § 11 five-shot accuracy. M3 has the highest average zero-shot accuracy in all four models, 0.4–1.3 points above the Transformer and 0.3–0.6 above the strongest other connection, and beats the Transformer on 30 of the 40 model–task pairs. Its largest and most consistent gain is on LAMBADA, 2.5–3.7 points. With five examples, M3 has the highest average at 700M, 1.4B and in the large mixture-of-experts model, 0.4–0.7 points above the Transformer, and is 0.3 below it in the small one.
| Size | Model | PPL↓ | LAMBADA | PIQA | WinoGrande | ARC-E | ARC-C | SciQ | LogiQA | BoolQ | HellaSwag | RACE-M | Avg |
| 700M | Transformer | 9.72 | 30.60 | 68.61 | 54.30 | 60.31 | 29.78 | 88.30 | 23.35 | 59.94 | 45.55 | 39.42 | 50.02 |
| AttnRes | 9.67 | 31.22 | 68.44 | 51.54 | 58.59 | 32.34 | 87.50 | 23.35 | 61.41 | 45.81 | 40.95 | 50.11 | |
| MUDDFormer | 9.38 | 33.38 | 68.61 | 53.91 | 57.70 | 32.34 | 89.90 | 23.96 | 60.58 | 47.05 | 40.53 | 50.80 | |
| M3 | 9.31 | 34.29 | 68.66 | 55.25 | 59.93 | 31.57 | 89.60 | 22.27 | 61.25 | 47.62 | 41.99 | 51.24 | |
| 1.4B | Transformer | 8.33 | 38.29 | 71.38 | 58.64 | 64.65 | 35.49 | 90.40 | 20.74 | 61.04 | 54.56 | 42.97 | 53.82 |
| AttnRes | 8.21 | 39.39 | 71.65 | 59.04 | 62.71 | 37.63 | 91.00 | 24.42 | 62.42 | 54.71 | 42.27 | 54.52 | |
| M3 | 8.07 | 40.97 | 70.84 | 58.88 | 63.76 | 35.92 | 91.70 | 22.89 | 64.28 | 55.92 | 46.17 | 55.13 | |
| 1.8B-A0.38B | Transformer | 9.73 | 27.38 | 67.63 | 52.17 | 56.02 | 30.12 | 84.80 | 21.81 | 61.44 | 44.33 | 35.93 | 48.16 |
| AttnRes | 11.12 | 25.07 | 63.82 | 52.96 | 48.91 | 26.54 | 83.80 | 22.89 | 60.37 | 38.74 | 36.84 | 45.99 | |
| MUDDFormer | 9.97 | 28.90 | 66.59 | 52.49 | 53.91 | 29.78 | 86.40 | 24.42 | 58.50 | 43.03 | 39.14 | 48.32 | |
| M3 | 9.64 | 30.33 | 67.68 | 53.20 | 54.76 | 30.29 | 87.50 | 23.66 | 55.78 | 44.84 | 37.81 | 48.58 | |
| 4.6B-A0.89B | Transformer | 8.07 | 35.51 | 72.74 | 56.99 | 63.05 | 36.26 | 90.00 | 20.28 | 55.41 | 54.16 | 42.41 | 52.68 |
| M3 | 8.00 | 38.00 | 71.06 | 57.06 | 64.23 | 36.01 | 89.80 | 19.35 | 59.85 | 54.56 | 43.25 | 53.32 |
Table 3. Zero-shot downstream accuracy (%) at 20 training tokens per parameter. PIQA, ARC and HellaSwag use length-normalized accuracy. PPL is the perplexity on the held-out FineWeb-Edu tokens. Bold: best at each size.
| Size | Model | LAMBADA | PIQA | WinoGrande | ARC-E | ARC-C | SciQ | LogiQA | BoolQ | HellaSwag | RACE-M | Avg |
| 700M | Transformer | 29.90 | 68.72 | 55.33 | 65.70 | 32.85 | 90.20 | 22.43 | 62.26 | 45.71 | 41.50 | 51.46 |
| AttnRes | 30.04 | 67.41 | 52.80 | 66.37 | 34.04 | 90.60 | 20.58 | 56.73 | 45.37 | 41.57 | 50.55 | |
| MUDDFormer | 30.56 | 68.66 | 54.06 | 68.48 | 34.47 | 90.40 | 25.04 | 58.65 | 47.06 | 42.20 | 51.96 | |
| M3 | 33.71 | 69.04 | 55.96 | 68.69 | 34.90 | 91.80 | 23.35 | 53.15 | 47.84 | 43.38 | 52.18 | |
| 1.4B | Transformer | 36.25 | 72.25 | 59.35 | 72.39 | 39.16 | 93.90 | 22.43 | 62.78 | 54.67 | 45.26 | 55.85 |
| AttnRes | 40.40 | 71.55 | 58.64 | 72.35 | 40.36 | 93.40 | 23.50 | 62.97 | 54.72 | 44.85 | 56.27 | |
| M3 | 40.42 | 71.65 | 61.17 | 72.90 | 39.51 | 94.60 | 21.81 | 59.69 | 56.28 | 47.14 | 56.52 | |
| 1.8B-A0.38B | Transformer | 26.14 | 69.04 | 51.85 | 65.78 | 33.28 | 88.60 | 25.81 | 56.82 | 44.19 | 36.00 | 49.75 |
| AttnRes | 24.65 | 64.64 | 53.59 | 56.40 | 28.24 | 85.80 | 22.58 | 43.46 | 38.07 | 36.00 | 45.34 | |
| MUDDFormer | 24.53 | 67.90 | 54.30 | 63.05 | 32.76 | 89.20 | 25.81 | 59.36 | 42.87 | 39.35 | 49.91 | |
| M3 | 26.57 | 67.57 | 51.38 | 64.65 | 32.34 | 89.10 | 26.73 | 53.06 | 44.84 | 38.65 | 49.49 | |
| 4.6B-A0.89B | Transformer | 36.99 | 73.01 | 56.99 | 71.97 | 40.27 | 92.30 | 21.51 | 60.24 | 54.24 | 43.31 | 55.08 |
| M3 | 36.02 | 71.60 | 56.67 | 71.97 | 38.74 | 93.30 | 23.96 | 61.19 | 55.14 | 46.03 | 55.46 |
Table 8. Five-shot downstream accuracy (%) at 20 training tokens per parameter, with the metrics of Table 3. Bold: best at each size.
Analysis
All three constraints matter
Which of the three constraints limit the Transformer? Table 4 ablates them from 100M to 1.4B with the members of the family in § 2.2, and with block attention residuals for relaxing only (3). These lower the loss of the Transformer by 0.047 at 100M but by at most 0.020 from 300M on. Separating groups as well (multi-group) lowers it by 0.014–0.059, and separating roles instead (multi-role, here MUDDFormer) by 0.028–0.080 up to 700M. Only M3, which relaxes all three, goes further, by 0.005–0.013 beyond the better of the two at every size. At 700M, relaxing (3) alone, (2) and (3), or (1) and (3) recovers 0.005, 0.021 and 0.036 of the 0.044 gain of M3. The gains of separating roles and separating groups are not additive, so they partly address the same limitation, but each removes a part that the other leaves in place. Group-specific mixing helps here even on its own, whereas splitting attention residuals into heads raises their loss ([18]); in M3, each group has its own signed, unnormalized mixture rather than a share of one softmax over depth.
| Residual | AttnRes^a | Multi-group | Multi-role^b | M3 | |
| (1) Roles | ✓ | ✓ | |||
| (2) Groups | ✓ | ✓ | |||
| (3) Weights | ✓ | ✓ | ✓ | ✓ | |
| 100M | 2.798 | 2.751 | 2.739 | 2.717 | 2.713 |
| 300M | 2.437 | 2.417 | 2.405 | 2.394 | 2.383 |
| 500M | 2.345 | 2.345 | 2.331 | 2.317 | 2.304 |
| 700M | 2.275 | 2.269 | 2.253 | 2.238 | 2.231 |
| 1.4B | 2.119 | 2.105 | 2.101 | div. | 2.088 |
Table 4. Ablation of the three constraints: each column relaxes the checked ones. Validation loss at 20 tokens per parameter. ^aBlock attention residuals [18], which use a softmax over blocks of layers and, unlike the other columns, are not in the family of § 2.2. ^bMUDDFormer ([38]), which diverged (div.) at 1.4B.
How fine should the groups be?
Figure 5 varies the number of groups per role, , from 100M to 700M at 20 tokens per parameter; is MUDDFormer. Groups pay off most at the first split: lowers the loss by 0.005–0.013 at every size, three to nine times what its added parameters would give in a larger Transformer according to the fit (Appendix § 9.§ 5). Finer groups help only at 700M, by another 0.002, and with there M3Former is still 0.005 below MUDDFormer, so the choice of barely affects our comparisons.

Figure 5. Validation loss with groups per role, where is the number of attention heads; is MUDDFormer.
Training stability
All models are trained with the same pre-norm recipe and no additional stabilization, and every run from 100M to 700M completed. At 1.4B, MUDDFormer diverged after 177,788 of 208,021 steps: its loss jumped to 10.37, the loss of a uniform prediction over the vocabulary, its gradient norm fell to zero, and it did not recover. M3Former and attention residuals completed the same run and its decay branches. MUDDFormer trains its deep variants with PrePostNorm, which normalizes the inputs of the mixing and the mixed outputs ([38]). With PrePostNorm, both complete the 1.4B run: MUDDFormer reaches 2.083 and M3Former 2.073, so M3Former stays ahead by 0.010; PrePostNorm also lowers the loss of M3Former, from 2.088.
What do roles and groups read?
Figure 6 shows the coefficients that M3 learns at 300M; Appendix § 13 shows all blocks at both sizes.
Figure 6. Coefficients of separately trained 100M and 300M M3 models (), averaged over validation tokens; colors scale as . (a), (b): 300M. (c): increase of the validation loss when the coefficients of one block are averaged over roles, groups or tokens; one point per block, bars: medians.
The four roles of a block read different layers (a): in the last block, R puts 2.55 on the latest state, Q and K spread over the upper half of the network, and V draws mostly on and the embedding while giving the latest state . The groups of a role differ as well (b): every group of V reads , but with weights from 0.31 to 2.15, and the weights of the groups of K on range from to 0.52, so they disagree even in sign. Reimposing one constraint in a single block, by averaging its coefficients over roles, groups or tokens, raises the loss in every block (c; medians 0.16, 0.046 and 0.011 at 300M). Averaging over groups costs more than over tokens in all 23 blocks, most of all for V (Appendix § 14).
Related Work
Residual connections. Residual connections [11] and highway networks [30] made deep networks trainable, and in Transformers the residual stream is the channel through which layers communicate [8]. Much work changes how a block writes to this stream, through normalization placement [40, 21] or rescaled residual branches [1, 35]. M3 leaves the write unchanged and changes how each block reads.
Cross-layer connections. DenseNet connects every layer to all earlier ones [15], and DLCL [36] and DenseFormer [24] bring learned static weights over earlier layers to Transformers. Hyper-connections widen the residual stream into several streams with partly token-dependent connections [45, 39], and frac-connections split the hidden state into fractions [46]; both recombine the current streams rather than read individual earlier layers. Attention residuals attend over the outputs of earlier layers [18], and multi-head and multi-stream variants split this attention by subspace or stream [23, 27]; all of them form one input per block. Value residual learning passes the first-layer values to later layers [44]. Closest to our work, Deep-CrossAttention [12] and MUD-DFormer [38] form several block inputs from separate mixtures. DeepCrossAttention combines static channel-wise weights with a token-dependent weight per layer, and MUDDFormer generates token-dependent weights per role; in both, the token-dependent weights are shared by all channels, and M3 makes them specific to the group as well, applying to the depth axis the idea of multi-head attention [33] and fine-grained experts [7, 22], which split a component into smaller, specialized parts.
Conclusion
M3 removes the broadcast bottleneck of the residual connection by letting every group of every role read its own token-dependent mixture of earlier layers. From 100M to 1.4B parameters, M3Former matches a Transformer 1.19–1.34× the size of the one it modifies, improves on MUDDFormer at every size, including 1.4B, where MUDDFormer needs PrePostNorm to converge, and carries over to mixture-of-experts models. All three constraints matter, and the first split into groups captures most of the gain per parameter.
Limitations
Due to computational constraints, our dense models reach 1.4B parameters and our mixture-of-experts models 4.6B; we leave larger models and longer training to future work. We study decoder-only language models pre-trained on English text, and applying M3 to other architectures, languages and modalities remains to be explored.
Derivations
This appendix proves the properties of M3 that the main text uses and derives its cost, in the notation of §2. A reader is a pair of a role and a group , so a block has readers, and reader of block forms .
Mixing states and mixing updates
Let and for be the update of block , so that .
Lemma 1. For any , with , and is a bijection of . The residual connection, , gives , and a convex combination of states gives .
Proof. Substitute and exchange the sums; the inverse is with . For a convex combination, each is a sum of nonnegative weights that sum to one.
The residual connection thus gives the update of every earlier block the weight one, for every token. A convex mixture, such as a softmax over depth, can only lower the weight of an earlier update, never raise it above one or make it negative, whereas signed, unnormalized coefficients, as in M3, can give the updates any weights .
Exact initialization
Proposition 1. Let , let be one on and zero elsewhere for every reader, initialize the output mixer in the same way, and let the scales of and equal that of . Then, for every input and every value of , M3Former computes the same hidden states and logits as the Transformer with the same remaining parameters.
Proof. Both models share . Suppose they agree on . With , Eq. (4) gives whatever is, so for every reader and . Since and then act as , Eq. (2) reduces to Eq. (1), and the models agree on . By induction they agree on , and the output mixer passes to the output layer.
Nesting of the family
Proposition 2. Let every member of the family of §2.2 use the block of Eq. (2) and a generator of the form of Eq. (4) whose hidden width is at most , and let , , , and be the sets of functions computed by the residual connection, dense, multi-role and multi-group mixing, and M3, the last two with the same . Then and .
Proof. With and on the latest state, every member computes the residual connection, as in Proposition 1. A generator of smaller hidden width is a generator of width whose extra hidden units have zero output weights, so we can take all hidden widths equal to . Given multi-role mixing, whose generator outputs are indexed by , give M3 the same and set its column of and entry of to the column and entry of multi-role mixing for every ; then for every token, and the two models compute the same function. Multi-group mixing is embedded in the same way, by repeating its columns and entries over the four roles, and dense mixing is embedded in multi-role or multi-group mixing by repeating its columns and entries over the roles or the groups. The output mixer is embedded in the same way.
M3 thus computes every function that a coarser member computes, so its lowest attainable training loss is at most theirs; whether training finds the additional functions is what § 5.1 tests. With , M3 is multi-role mixing, which is our MUDDFormer baseline (§ 3). Attention residuals are not members of the family: their weights are a softmax over the outputs of blocks of layers, computed from all earlier states rather than from alone.
Parameters and computation
Summing the generator and the two extra norms of every block and the output mixer, which has one role and groups, M3 adds
parameters, the parameter count of our implementation. Per token, block spends multiply–adds in the generator and in Eq. (3), and the output mixer . If were exactly , the generators would add about parameters, and relative to the non-embedding parameters of a Transformer with feed-forward width ,
so the overhead grows with depth relative to width. In our models, rounding up to a multiple of 64 makes larger than , so we compute the overhead exactly from Eq. (5); Table 5 lists it for the models we report.
Equivalent size and parameter-matched gain
We fit with to the Transformers trained on tokens. A loss is reached by a Transformer of size , and we report .
Training on tokens costs operations, so the Transformer that matches the loss uses times the compute of a Transformer of size .
To compare the gain of M3 with what its extra parameters would give if they were spent on a larger Transformer, we evaluate the fit at , where :
This is an estimate under the fitted Transformer curve, not a bound on what parameters in another form could contribute. Table 5 compares it with the observed gain.
Training Details
Table Table 6 lists the training hyperparameters of the dense models. The dense Transformer is a pre-norm decoder with RMSNorm (), SwiGLU feed-forward layers, rotary position embeddings with base 10,000, and tied input and output embeddings. RMSNorm scales start at one. The query, key and value projections and the gate projection of the feed-forward layer are drawn from a truncated normal distribution with standard deviation 0.02, and the attention output projection and the up and down projections of the feed-forward layer from one with standard deviation in block , counted from one. The tied embedding is drawn from a normal distribution with standard deviation , truncated at . Each backbone parameter is seeded by its name, so all methods start from the same backbone; § 2.3 gives the initialization of the M3 mixers. The 1.4B main run has 208,021 steps and the 700M main run 98,013 steps. Table 7 gives the mixture-of-experts architectures of § 4.3; parameter counts are those of the Transformer baselines, and MUDDFormer and M3 add their mixers.
| Data | FineWeb-Edu (sample-350BT), fixed subset |
| Tokenizer | TinyLlama SentencePiece (32,000) |
| Sequence length | 2,048 |
| Global batch | 64 sequences (131,072 tokens) |
| Optimizer | AdamW, , , , weight decay 0.1 |
| Gradient clipping | global norm 1.0 |
| Schedule | WSD; linear warmup over 1% of the budget; decay to 10% of the peak over the last 20% of each budget |
| Peak learning rate | for 100M, 300M, 500M, 700M, 1.4B |
| Budgets | , , tokens; is the non-embedding parameter count of the Transformer |
| Validation | 99,876,864 tokens |
| Precision | BF16 computation; FP32 master parameters and gradient reduction |
| Hardware | NVIDIA H200 |
Table 6. Training hyperparameters of the dense models, shared by all methods.
Full Results
Table 9 lists all runs of the main setup. Table 10 adds all 700M runs. Table 8 gives five-shot downstream accuracy; the five examples are fixed across models, RACE-M is scored per question, and prompts are not truncated. We compare methods at in the main text; with tokens, MUDDFormer is slightly ahead of M3 at 100M (by 0.003) and of M3 with at 700M (by 0.012).
| Size | Tokens | Transf. | AttnRes | MUDDFormer | M3 | |
| 100M | 0.41B | 3.0899 | 3.0447 | 2.9780 | 2.9811 | |
| 0.81B | 2.9209 | 2.8743 | 2.8289 | 2.8273 | ||
| 1.63B | 2.7976 | 2.7507 | 2.7172 | 2.7126 | ||
| 300M | 1.54B | 2.6395 | 2.6183 | 2.5836 | 2.5724 | |
| 3.08B | 2.5243 | 2.5043 | 2.4774 | 2.4669 | ||
| 6.17B | 2.4367 | 2.4168 | 2.3937 | 2.3825 | ||
| 500M | 2.29B | 2.5368 | 2.5292 | 2.5043 | 2.4878 | |
| 4.58B | 2.4278 | 2.4257 | 2.4002 | 2.3853 | ||
| 9.17B | 2.3452 | 2.3447 | 2.3171 | 2.3039 | ||
| 700M | 3.21B | 2.4566 | 2.4455 | 2.4139 | 2.4055 | |
| 6.42B | 2.3534 | 2.3469 | 2.3159 | 2.3087 | ||
| 12.85B | 2.2747 | 2.2692 | 2.2385 | 2.2309 | ||
| 1.4B | 6.82B | 2.2763 | 2.2594 | – | 2.2428 | |
| 13.63B | 2.1870 | 2.1732 | – | 2.1556 | ||
| 27.27B | 2.1193 | 2.1053 | diverged | 2.0884 |
Table 9. Validation loss of all runs in the main setup. M3 uses the best at each size (Table 5). Attention residuals use blocks of four layers. Multi-group mixing (Table 4) is trained at only, except at 700M (Table 10).
| 700M | |||
| Transformer | 2.4566 | 2.3534 | 2.2747 |
| Attention residuals | 2.4455 | 2.3469 | 2.2692 |
| Multi-group, | 2.4349 | 2.3316 | 2.2533 |
| MUDDFormer | 2.4139 | 2.3159 | 2.2385 |
| M3, | 2.4259 | 2.3108 | 2.2332 |
| M3, | 2.4032 | 2.3099 | 2.2315 |
| M3, | 2.4055 | 2.3087 | 2.2309 |
Table 10. All 700M runs of the main setup ().
Pseudocode
Figure 7 gives a PyTorch-style reference implementation of one M3Former block; our fused kernel computes the same mixture in a single pass over the history.
# H: [h_0, ..., h_{l-1}], each of shape [T, d]
# returns h_l
def m3_block(H, blk, G):
l, (T, d) = len(H), H[-1].shape
# token-conditioned coefficients a_{l,r,g,j}
x = rms_norm(H[-1]) # no learned scale
A = blk.W2(gelu(blk.W1(x))) + blk.a0
A = A.view(T, 4, G, l) # roles Q, K, V, R
# split every state into G groups
S = torch.stack(H, -1).view(T, G, d // G, l)
Z = torch.einsum('trgj,tgdj->trgd', A, S)
zq, zk, zv, zr = Z.reshape(T, 4, d).unbind(1)
u = zr + blk.attn(blk.na(zq), blk.nk(zk),
blk.nv(zv))
return u + blk.ffn(blk.nf(u))
# init: W2 = 0; a0 = 1 on j = l-1, 0 elsewhere
# -> every role reads h_{l-1} (standard block)
fig:7: PyTorch-style pseudocode of an M3Former block. blk.attn includes the query, key, value and output projections.
Learned Coefficients
Figures 8–10 extend Figure 6 with the same models (). Figure 8 shows the group-averaged coefficients of every role in every block, Figure 9 all 16 groups of every role in the last block at 300M, and Figure 10 the weights on the latest state and on the embedding through depth. Through the middle of the network, every role gives the latest state at most 0.73, far below the 1 of the residual connection; only R puts more than 1 on it, and only in the last blocks; V reads the embedding in the first blocks and again in the last third of the network, and K subtracts it in the last block. In the last block, the groups of R agree on the latest state, whereas the groups of Q and K read different layers.
Figure 8. Coefficients of every role on every earlier state in every block, averaged over groups and validation tokens, with the colors of Figure 6. Top: 300M; bottom: 100M.
Figure 9. Coefficients of the 16 groups of every role in the last block of the 300M model, averaged over validation tokens, with the colors of Figure 6.
Figure 10. Weight of every role on the latest state (a) and on the embedding (b), averaged over groups and validation tokens, against relative depth.
Interventions on the Coefficients
Figure 11 details the interventions of Figure 6c. We run the models of Figure 6 on 128 validation sequences of 2048 tokens and replace the coefficients of one block by an average: over its four roles, over the groups of each role, or over tokens, using the token means on 64 other validation sequences. Every sequence sees every intervention, and each intervention of Figure 6c raises the loss by at least 5.9 standard errors over the sequences. Averaging over the four roles, which mixes the residual input with the normalized query, key and value inputs, costs most in the last blocks (a). Averaged one role at a time, the groups matter most for V at both sizes, whose groups read with weights from 0.31 to 2.15 in the last block (Figure 6b), and the token matters least for Q (b, c). The dependence on the token is concentrated: 10% of the tokens carry 92% of the loss increase summed over blocks at 300M and 87% at 100M, and positions whose next token continues the current word lose 2.3 times as much as other positions at both sizes (d), mostly through R. Applied to all blocks at once, averaging over tokens, groups and roles raises the loss by 1.2, 6.2 and 8.4 at 300M.
Figure 11. Interventions of Figure 6c. (a) Loss increase against relative depth; solid: 300M, dashed: 100M. (b), (c) Median over blocks of the loss increase when only one role is averaged over tokens (b) or over its groups (c). (d) Loss increase from averaging over tokens, summed over blocks, at positions whose next token continues the current word (in word) and at other positions. Dark: 300M; light: 100M.
References
- [1]Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. 2021. ReZero is all you need: Fast convergence at large depth. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352–1361.arxiv.org/abs/2003.04887
- [2]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439.arxiv.org/abs/1911.11641
- [4]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2924–2936.arxiv.org/abs/1905.10044
- [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.arxiv.org/abs/1803.05457
- [7]Damai Dai, Chengqi Deng, Chenggang Zhao, Runxin Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297.
- [8]Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
- [9]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonnell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The language model evaluation harness. Zenodo.
- [11]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778.arxiv.org/abs/1512.03385
- [12]Mike Heddes, Adel Javanmard, Kyriakos Axiotis, Gang Fu, MohammadHosse in Bateni, and Vahab Mirrokni. 2025. DeepCrossAttention: Supercharging transformer residual connections. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 22881–22903.arxiv.org/abs/2502.06785
- [13]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35.arxiv.org/abs/2203.15556
- [14]Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling.arxiv.org/abs/2404.06395
- [15]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700–4708.arxiv.org/abs/1608.06993
- [18]Kimi Team. 2026. Attention residuals. arXiv preprint arXiv:2603.15031.
- [19]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale Reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794.arxiv.org/abs/1704.04683
- [20]Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020a. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3622–3628.arxiv.org/abs/2007.08124
- [21]Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. 2020b. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5747–5763.arxiv.org/abs/2004.08249
- [22]Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzywółdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczuр. 2024. Scaling laws for fine-grained mixture of experts. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33270–33288.arxiv.org/abs/2402.07871
- [23]Cheng Luo, Zefan Cai, and Junjie Hu. 2026. Multi-head attention residuals. arXiv preprint arXiv:2607.27230.arxiv.org/abs/2607.27230
- [24]Matteo Pagliardini, Amirkeivan Mohtashami, François Fleuret, and Martin Jaggi. 2024. DenseFormer: Enhancing information flow in transformers via depth weighted averaging. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2402.02622
- [25]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534.arxiv.org/abs/1606.06031
- [26]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37.arxiv.org/abs/2406.17557
- [27]Xiaohan Qin, Xiangdong Zhang, Tuo Dai, Sunan Zhou, Huajun Wu, Yebin Yang, Zhuo Xia, Lin Yao, Yuliang Liu, and Junchi Yan. 2026. Multi-stream attention residuals. arXiv preprint arXiv:2610.13630.arxiv.org/abs/2610.13630
- [28]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740.arxiv.org/abs/1907.10641
- [29]Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202.arxiv.org/abs/2002.05202
- [30]Rupesh K. Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Information Processing Systems, volume 28.arxiv.org/abs/1507.06228
- [31]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.arxiv.org/abs/2104.09864
- [33]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.arxiv.org/abs/1706.03762
- [35]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022. DeepNet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555.arxiv.org/abs/2203.00555
- [36]Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822.arxiv.org/abs/1906.01787
- [37]Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106.arxiv.org/abs/1707.06209
- [38]Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. 2025. MUDFormer: Breaking residual bottlenecks in transformers via multiway dynamic dense connections. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 68440–68458.arxiv.org/abs/2502.12170
- [39]Zhenda Xie, Yixuan Wei, Huangji Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. 2025. mHC: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880.arxiv.org/abs/2512.24880
- [40]Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishui Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524–10533.arxiv.org/abs/2002.04745
- [41]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.arxiv.org/abs/1905.07830
- [42]Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32.arxiv.org/abs/1910.07467
- [43]Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An open-source small language model. arXiv preprint arXiv:2401.02385.arxiv.org/abs/2401.02385
- [44]Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. 2025. Value residual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356.arxiv.org/abs/2410.17897
- [45]Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. 2025a. Hyper-connections. In International Conference on Learning Representations.arxiv.org/abs/2409.19606
- [46]Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, and Xun Zhou. 2025b. Frac-connections: Fractional extension of hyper-connections. arXiv preprint arXiv:2503.14125.arxiv.org/abs/2503.14125






