betaFeedback

WorksComputer Science

KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers

Abstract

Post-LayerNorm Transformers have seen limited practical adoption due to enduring difficulties in scaling them to large depths. While prior research has focused on stabilising Post-LayerNorm by improving conditioning at initialisation, stability often deteriorates when models are trained with large learning rates, forcing additional compromises. In the decoder-only settings we study, existing Post-LayerNorm methods fail to outperform strong Pre-LayerNorm baselines. We propose KiteNorm, a post-norm training method designed to break this trend by constraining hidden-state variance. This approach is motivated by a bound on the LayerNorm Jacobian suggesting that variance growth contributes to gradient vanishing, alongside empirical evidence linking it to rank collapse. KiteNorm combines residual scaling and variance regularisation to maintain stability throughout training, supported by scalarised double-normalisation that provides separately learnable scales for the skip and residual branches. Across decoder-only Transformers up to 1.3B parameters, KiteNorm improves training stability and outperforms leading baselines at their optimal learning rates, with scaling laws favouring KiteNorm across depth, width, batch size, and training budget. Ablations show that each component is necessary for the full stability and performance gains.

Introduction

Post-LayerNorm (post-norm) Transformers (Vaswani et al., [28]) have largely been replaced by Pre-LayerNorm (pre-norm) Transformers (Wang et al., [30]) in large-scale Transformer implementations, owing to their improved gradient conditioning along the uninterrupted skip branch (Xiong et al., [32]). However, pre-norm exhibits persistent weaknesses, such as parameter inefficiency in deep layers (Sun et al., [27]; Gromov et al., [9]) and massive activations (Sun et al., [26]). Residual scaling (Sun et al., [27]; Radford et al., [22]) can mitigate these weaknesses, but it cannot fully eliminate them as the architecture necessarily piles up sublayer contributions on the skip branch.

This has motivated renewed interest in post-norm architectures with the goal of scaling them to a large number of layers. The core challenge is the fact that the Jacobian of post-norm acts on both the residual and skip branches during backpropagation, which leads to gradient vanishing as the scale of the activations and the number of layers rise (Nguyen & Salazar, [18]; Huang et al., [11]). While prior work has analysed the effective dynamics of this phenomenon, the underlying cause remains poorly understood. We extend existing insights by deriving an upper bound that directly relates hidden-state variance to gradient propagation, and by providing empirical evidence linking variance expansion to rank collapse (Dong et al., [7]; Noci et al., [19]).

Recent works have made strides in improving the conditioning of post-norm Transformers through branch scaling and double-normalisation (Wang et al., [29]; Chen & Wei, [3]; Joseph et al., [12]).

While these methods improve optimisation conditions at initialisation, they can be less robust to high learning rates or less competitive at moderate depths. We postulate that methods that rely primarily on favourable initialisation conditions are ultimately prone to degraded performance or stability as large learning rates push models further away from initialisation. Resolving this trade-off is therefore important for improving the scalability and efficiency of Transformer training. Our contributions can be outlined as follows:

  • We extend the theoretical analysis of gradient propagation in post-norm Transformers by highlighting the impact of hidden-state variance, and provide empirical evidence linking variance expansion to rank collapse.

  • We propose KiteNorm, constraining hidden-state variance through a combination of residual scaling, double-normalisation, scalarised LayerNorm and a new variance regulariser, which stabilises optimisation even at large learning rates.

  • We conduct a suite of experiments with decoder-only Transformers up to 1.3B parameters that demonstrate KiteNorm’s improved stability and scaling with respect to model depth, width, batch size, and training budget, even when compared to leading solutions.

  • We provide a series of ablations that demonstrate each component of KiteNorm is essential.

Code is available at https://github.com/l-trochelmann/KiteNorm.

Preliminaries and Related Work

In this section, we establish notation, provide context for our work, and derive the crucial backward pass dynamics of post-norm Transformers.

Notation

Define the normalisation of a vector v∈Rdv \in\mathbb{R}^{d} as follows:

v~=v−μ(v)1σ(v),(1)\tilde{v} = \frac{v-\mu(v)\mathbf{1}}{\sigma(v)}, \tag*{(1)}

where μ\mu and σ\sigma denote the mean and standard deviation, respectively, and 1\mathbf{1} is the size-dd vector of ones. LayerNorm ([1]), as used in Transformers, can then be written as

LN⁡(X)=γ⋅X~+β,(2)\operatorname{LN}(X) = \gamma\cdot\tilde{X} + \beta, \tag*{(2)}

where X∈RS×dX \in\mathbb{R}^{S \times d} is a length-SS sequence of tokens that are embedded in hidden dimension dd. Normalisation is applied row-wise across dd. The parameters γ,β∈Rd\gamma, \beta\in\mathbb{R}^{d} act element-wise across dd and are called the gain and bias of LayerNorm. Their components are uniformly initialised as 1 and 0, respectively. Implementations of LayerNorm often add a small constant ϵ\epsilon in the denominator during the normalisation step to ensure numerical stability, though we omit it for clarity in this section.

The original Transformer ([28]) used normalisation as a component of its sublayers in the following form:

Sublayer⁡post-norm(X)=LN⁡(X+F(X)).(3)\operatorname{Sublayer}_{\mathrm{post\text{-}norm}}(X) = \operatorname{LN}(X + \mathcal{F}(X)). \tag*{(3)}

Sublayer notation. A decoder-only Transformer stacks such sublayers, with F\mathcal{F} alternating between self-attention and an MLP transformation. We denote the number of Transformer layers by LL, so that the model has 2L2L sublayers. We write XlX_l for the hidden states entering the ll-th sublayer, where l∈{1,2,…,2L}l \in\{1,2,\ldots,2L\}, and Fl\mathcal{F}_l for the corresponding residual branch transformation. To differentiate sublayer inputs and outputs, we write Zl=Xl+Fl(Xl)Z_l = X_l + \mathcal{F}_l(X_l). II denotes the identity matrix.

Norms and matrix notation. ∥⋅∥q\|\cdot\|_q describes the LqL_q norm for vectors, and the corresponding induced operator norm for matrices. ∥⋅∥F\|\cdot\|_F describes the Frobenius norm. The ii-th row of a matrix AlA_l is denoted by Al,iA_{l,i}. Since LayerNorm is applied independently to each token’s hidden state, our theoretical derivations omit the sequence length and batch size. We thus write xl∈Rdx_l \in\mathbb{R}^{d} for a row from XlX_l and zl∈Rdz_l \in\mathbb{R}^{d} for a row from ZlZ_l.

Gradient Propagation in Post-Norm Transformers

We now examine the backward pass of post-norm Transformers. Let LNl\mathrm{LN}_l denote the LayerNorm operation in sublayer ll, and let xl+1=LNl(xl+Fl(xl))=LNl(zl)x_{l+1} = \mathrm{LN}_l(x_l + \mathcal{F}_l(x_l)) = \mathrm{LN}_l(z_l). Then

∂xl+1∂xl=∂LNl(zl)∂zl∂zl∂xl=∂LNl(zl)∂zl(I+∂Fl(xl)∂xl).(4)\frac{\partial x_{l+1}}{\partial x_l} = \frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l}\frac{\partial z_l}{\partial x_l} = \frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l}\left(I + \frac{\partial\mathcal{F}_l(x_l)}{\partial x_l}\right). \tag*{(4)}

LayerNorm is a gradient bottleneck. Notably, the LayerNorm Jacobian multiplies both the Jacobian along the skip branch II and the derivative along the residual branch ∂Fl(xl)∂xl\frac{\partial\mathcal{F}_l(x_l)}{\partial x_l}. As highlighted by Nguyen & Salazar [18], this leads to instability if ∂LNl(zl)∂zl\frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l} repeatedly expands or contracts gradients across layers.

Spectral norm bound. We can characterise when LayerNorm can act as a contraction by analysing the spectral norm of ∂LNl(zl)∂zl\frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l}, for which we derive the following upper bound (see Appendix § 8).

Theorem 1 (Spectral norm bound for LayerNorm). Let LNl\mathrm{LN}_l denote LayerNorm applied to the hidden state zlz_l, with learned scale parameter γ\gamma. Then for any σ(zl)>0\sigma(z_l) > 0 the Jacobian of LNl\mathrm{LN}_l with respect to zlz_l satisfies

∥∂LNl(zl)∂zl∥2≤∥γ∥∞σ(zl).(5)\left\lVert\frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l} \right\rVert_2 \le\frac{\lVert\gamma\rVert_\infty}{\sigma(z_l)}. \tag*{(5)}

The crucial implication of Theorem 1, together with equation (4), is that the LayerNorm Jacobian acts as a shared bottleneck for gradient propagation through both the skip and residual branches. In particular, its spectral norm is upper-bounded by the LayerNorm gain divided by σ(zl)\sigma(z_l). Hence, sufficiently large hidden-state variance can cause the LayerNorm Jacobian to contract gradients, contributing to gradient vanishing across depth. This establishes hidden-state variance growth as a concrete source of gradient contraction in post-norm Transformers and motivates constraining it during training.

Rank Collapse

Noci et al. [19] investigate rank collapse in Transformers by measuring the alignment of token representations using a metric originating from Cho & Saul [4] and Nachum et al. [17]. For two token indices i,i′i, i' in the hidden-state matrix ZlZ_l after the residual addition in sublayer ll, define the token correlation as

ρi,i′(Zl)=E⟨Zl,i,Zl,i′⟩E∥Zl,i∥22E∥Zl,i′∥22,(6)\rho_{i,i'}(Z_l) = \frac{\mathbb{E}\langle Z_{l,i}, Z_{l,i'} \rangle}{\sqrt{\mathbb{E}\lVert Z_{l,i}\rVert_2^2\mathbb{E}\lVert Z_{l,i'}\rVert_2^2}}, \tag*{(6)}

where the expectation is taken with respect to the model weights at initialisation.

Following Noci et al. [19], we define token alignment as the average token correlation

τ(Zl)=1S(S−1)∑i≠i′ρi,i′(Zl),(7)\tau(Z_l) = \frac{1}{S(S-1)} \sum_{i \ne i'} \rho_{i,i'}(Z_l), \tag*{(7)}

where SS is the sequence length.

Token representations pointing in the same direction have correlation 1. Thus, as token alignment tends to 1, the representation becomes increasingly rank-collapsed. In practice, we compute this metric over the actual model weights, replacing the expectation with the empirical value computed from the trained model states.

Although other rank-collapse metrics have been proposed (Dong et al. [7]), we find that they behave similarly in practice and focus on token alignment because it has a direct geometric interpretation. Token alignment therefore provides a simple empirical measure of rank collapse in Transformers, which we use in the following section to study its connection to gradient vanishing.

Related Work

Normalisation placement has played a central role in the development of Transformer architectures. The original Transformer used post-norm sublayers, but many large-scale implementations have since adopted pre-norm following Wang et al. [30]. In pre-norm Transformers, LayerNorm is placed before the residual branch transformation:

Sublayer⁡pre-norm(X)=X+F(LN⁡(X)).(8)\operatorname{Sublayer}_{\mathrm{pre\text{-}norm}}(\mathbf{X}) = \mathbf{X} + \mathcal{F}(\operatorname{LN}(\mathbf{X})). \tag*{(8)}

Unlike post-norm, pre-norm leaves the skip connection uninterrupted during backpropagation, allowing gradients to flow directly across depth [30]. Subsequent work linked this structure to improved gradient conditioning in expectation [32].

Pre-norm scaling methods. Several methods for controlling signal growth along the skip branch have been proposed. GPT-2 applies residual output scaling by initialising the output weights of each sublayer with an additional factor of 1/2L1/\sqrt{2L} [22], which compensates for the accumulation of residual updates in deeper models. Later work formalised the importance of such scaling mechanisms, showing that unconstrained signal growth in pre-norm Transformers can reduce the effectiveness of deep layers [27]. Instead of scaling output weights, LayerNorm-Scaling rescales the input to each sublayer after pre-normalisation by a depth-dependent factor.

Additional normalisation. Related approaches such as peri-norm [6, 13] add a second LayerNorm after the residual branch transformation, yielding peri-norm sublayers of the form

Sublayer⁡peri-norm(X)=X+LN⁡(F(LN⁡(X))).(9)\operatorname{Sublayer}_{\mathrm{peri\text{-}norm}}(\mathbf{X}) = \mathbf{X} + \operatorname{LN}(\mathcal{F}(\operatorname{LN}(\mathbf{X}))). \tag*{(9)}

This constrains the magnitude of each residual update before it is added to the skip branch.

Despite these improvements, pre-norm retains architectural limitations. Because each sublayer adds directly to the skip branch, scaling or additional normalisation can mitigate signal accumulation and alignment, but cannot fully avoid them. Recent work has also shown that large activations can propagate through pre-norm models at scale [26]. These issues have motivated renewed interest in post-norm architectures, where LayerNorm is applied after the residual addition and therefore normalises both the skip and residual branches.

Post-norm stabilisation. Several methods attempt to recover the stability of post-norm while preserving its performance advantages. Earlier, T-Fixup [11] also investigated the interaction between LayerNorm and gradient propagation, motivating an alternative initialisation strategy. Recently, mix-norm constructs models from an initial block of post-norm layers followed by pre-norm layers [14]. This reduces the number of post-norm layers over which gradients can vanish, while also reducing the number of pre-norm layers over which signal can accumulate. However, the resulting architecture remains a compromise between the two regimes. DeepNet instead retains a post-norm structure and introduces residual and skip branch scaling to improve optimisation conditions at initialisation [29]. Although effective for very deep encoder-decoder Transformers, we find that such initialisation-based stabilisation is less robust in decoder-only settings trained with large learning rates.

Most recently, Chen & Wei [3] proposed KEEL, a post-norm method that improves decoder-only Transformer training by using double-normalisation:

Sublayer⁡double-norm(X)=LN⁡(X+F(LN⁡(X))).(10)\operatorname{Sublayer}_{\mathrm{double\text{-}norm}}(\mathbf{X}) = \operatorname{LN}(\mathbf{X} + \mathcal{F}(\operatorname{LN}(\mathbf{X}))). \tag*{(10)}

Although the additional normalisation may appear redundant, it allows the model to separately learn input scales for the skip and residual branches within each sublayer. KEEL further employs skip branch scaling to improve gradient propagation. Overall, while KEEL improves the stability of post-norm training, it does not explicitly constrain hidden-state variance during training and still underperforms strong pre-norm baselines in our decoder-only setting.

KiteNorm

KiteNorm combines three techniques to improve both stability and performance. Residual scaling, discussed in § 3.1, is essential to our stabilisation strategy and explains the core dynamic of our method. We further introduce variance regularisation, described in § 3.2, which prevents this stability from degrading over time when models are trained with large learning rates. Finally, to improve the expressivity of this post-norm setup, we leverage scalarised double-normalisation, explained in § 3.3.

Residual Scaling

Variance expansion and rank collapse. While several prior works ([32, 11, 14]) have observed the link between gradient vanishing and activation variance in post-norm Transformers, the underlying pressure that causes the activation scales to explode at high learning rates remains largely unexplored. We offer additional insight into the dynamics of activation variance. Specifically, we observe that the variance expansion of hidden states tends to coincide with rank collapse in early training, as shown in Figure 1a and Figure 1b.

Effect of residual scaling on training failures

Figure 1. Effect of residual scaling on training failures observed when training a 12-layer post-norm Transformer according to the setup in Appendix B with learning rate 10−210^{-2}. We report statistics over the hidden state after the residual addition in the final Attention sublayer. Figures 1a & 1b show the behaviour of vanilla post-norm with residual scaling factor c=1c = 1, whereas figures 1c and 1d show the effects of other choices of cc.

Residual scaling reverses rank collapse. Recent literature on rank collapse ([12]) proposes that weighting the relative contribution of the skip and residual branches in favour of the skip branch can reduce the rate of rank collapse. We observe this in practice by reducing the scale of the residual branch with residual scales as small as 12L\frac{1}{2L}, shown in Figure 1c. While the unscaled model diverged while exhibiting rank collapse, choosing a sufficiently small residual scale can reverse rank collapse before the model diverges, even when using a relatively high learning rate. The reversal of rank collapse coincides with a reversal of hidden-state expansion, shown in Figure 1d.

Equivalence to skip branch scaling. Since prior work specifies scaling the skip branch ([12]), we formalise the equivalence between scaling the skip and residual branches, demonstrating that our method aligns with prior work.

We express skip scaling aa and residual scaling cc generally and relate them to each other. For positive aa, cc and ignoring the stability constant ϵ\epsilon, it holds that

LN⁡(axl+cFl(xl))=LN⁡(ac(1cxl+1aFl(xl)))=LN⁡(1cxl+1aFl(xl)).(11)\operatorname{LN}(a x_l + c\mathcal{F}_l(x_l)) = \operatorname{LN}\left(ac\left(\frac{1}{c}x_l + \frac{1}{a}\mathcal{F}_l(x_l)\right)\right) = \operatorname{LN}\left(\frac{1}{c}x_l + \frac{1}{a}\mathcal{F}_l(x_l)\right). \tag*{(11)}

Thus, the forward pass depends on the relative scale ca\frac{c}{a}. Since this equivalence holds for any xlx_l, it also holds for the corresponding Jacobians. Therefore, skip scaling and residual scaling are equivalent for optimising the model’s output loss, connecting our approach to the mitigation of rank collapse investigated by Joseph et al. [12].

However, the two approaches differ in their effect on the variance entering LayerNorm. Increasing skip scaling also increases this variance, making it unsuitable for our variance regulariser, which we detail in the next section. This motivates our choice of residual scaling with c=12Lc = \frac{1}{2L}, following the relative scaling strength of Chen & Wei [3].

Variance Regularisation

While residual scaling can reverse the initial variance expansion associated with rank collapse, it does not prevent a second form of variance expansion that can occur sporadically later in training. We observe this in Figure 7c of our ablations, where KiteNorm without variance regularisation becomes unstable after an initially stable training period. To prevent this later instability, we introduce variance regularisation, which adds a penalty based on hidden-state variance to the optimisation loss. Specifically, we propose the ReLU regulariser term:

Ablation studies on KiteNorm in decoder-only Transformers

Figure 7. Ablation studies on KiteNorm in decoder-only Transformers.

mean⁡l(ReLU⁡(σ(zl)2−1)).(12)\operatorname{mean}_l\left(\operatorname{ReLU}\left(\sigma(z_l)^2-1\right)\right). \tag*{(12)}

In practice, the resulting ReLU penalties are averaged across the batch, sequence, and sublayer dimensions before being added to the optimisation loss with weight λ\lambda. Notably, this regulariser only affects the loss if the variance of hidden states entering the outer LayerNorm exceeds 1. Hence, the regulariser does not affect optimisation while hidden-state variance remains below this threshold.

Using regularisation strength λ=1\lambda= 1 across our experiments, KiteNorm remains stable at and around its optimal learning rate, with instability occurring only at learning rates that already yield substantially worse performance. We hypothesise that by reversing the initial variance expansion, residual scaling establishes a stable training regime in which the optimiser can respond to subsequent variance growth through the regulariser.

The computational overhead of variance regularisation is small, as the regulariser can reuse the variance already computed by LayerNorm.

Scalarised Double-Normalisation

Variance regularisation provides a signal for constraining hidden-state variance, which the optimiser can already respond to by adjusting the model weights. However, we hypothesise that directly learnable scales make this response easier. To this end, we employ double-norm following Chen & Wei [3], which allows the optimiser to separately adjust the input scales of the skip and residual branches.

To further simplify this control, we propose using double-norm with scalarised LayerNorm:

LNscalarised(X)=γ′⋅X~+β′,(13)\mathrm{LN}_{\mathrm{scalarised}}(X)=\gamma' \cdot\widetilde{X}+\beta', \tag*{(13)}

where γ′,β′∈R\gamma',\beta' \in\mathbb{R} are scalar parameters initialised as 1 and 0, respectively. This provides the optimiser with a small set of parameters that can directly adjust all branch scales. Our ablations in § 4.3 show that double-norm improves both stability and performance, while scalarisation further improves performance.

Experiments

We perform a range of experiments to compare KiteNorm against competitive baselines from existing work. First, in § 4.1, we compare KiteNorm with leading normalisation methods in our setting, evaluating practical performance. Then, in § 4.2, we analyse and compare the scaling properties of KiteNorm against the strongest baseline. Finally, in § 4.3, we document the efficacy of each component of KiteNorm through ablation studies.

Unless otherwise specified, all trained models use the following hyperparameters and training setup. We train decoder-only Transformers on SlimPajama-627B (Soboleva et al. [24]) using the Chinchilla ratio of 20 training tokens per model parameter (Hoffmann et al. [10]). Our optimisation setup uses the AdamW optimiser (Loshchilov & Hutter [15]) with fixed β1=β2\beta_1 = \beta_2 following Orvieto & Gower [20] and gradient clipping. When training KiteNorm, we add the ReLU regulariser from equation (12) to the optimisation loss. Architecturally, our models use rotary position embeddings (RoPE) (Su et al. [25]) and a GLU feed-forward block instead of a standard MLP, following recent industry practice (Shazeer [23]). Appendix § 9.§ 1 provides a complete setup and hyperparameter specification.

We report validation loss, validation perplexity (ppl), and training loss. We additionally evaluate downstream performance on ARC-Easy and PIQA using the Language Model Evaluation Harness (Gao et al. [8]), with the evaluation setup detailed in Appendix § 9.§ 3. In plots, we abbreviate validation as valid and training as train. For some experiments, we display only the tuned results, meaning that we show the run with the lowest final loss or perplexity from a specified learning rate grid.

Comparison Against Leading Baselines

Compared methods. Our criterion for selecting competitive approaches is scalable performance in decoder-only Transformers under our training setup. Our implementations follow the written specifications in each method’s original proposal, with exceptions specified and motivated below. We give a comprehensive implementation breakdown in Appendix § 9.§ 2.

As a strong pre-norm baseline, we use pre-norm with GPT-2 initialisation [22] based on a comparative study described in Appendix § 10.§ 1. Additionally, we compare against the more recent LayerNorm-Scaling (LNS) [27]. We also include peri-norm [6, 13] due to its recent popularity. To represent post-norm, we evaluate against vanilla post-norm [28] and KEEL [3]. For KEEL, where the original specification uses an MLP, we report GLU results in the main comparison because an equal-parameter MLP variant performs worse in Appendix § 10.§ 2. To represent the hybrid approach between pre- and post-norm, we use mix-norm [14] with partial GPT-2 initialisation based on a comparison in Appendix § 10.§ 3.

Learning rate robustness. We compare these methods in a 48-layer Transformer, where depth renders the challenges of normalisation placement more pronounced. Figure 2 shows the results. In the learning rate exploration in Figure 2a, we find that KiteNorm and LayerNorm-Scaling provide the best robustness against high learning rates. Vanilla post-norm, KEEL and mix-norm exhibit comparatively poor robustness to learning rate choice. The odd curve associated with peri-norm is discussed separately in Appendix § 10.§ 4.

Final validation perplexities of KiteNorm and competitive LayerNorm methods in a 48-layer Transformer

Figure 2. Final validation perplexities of KiteNorm and competitive LayerNorm methods in a 48-layer Transformer.

Seed robustness. To obtain a more reliable estimate of the performance difference between these methods, we pick the learning rates yielding the best performance for each method in the exploration and rerun each method twice with new random seeds at its selected learning rate. The resulting performance values are shown in the right panel of Figure 2. Here, we highlight that KiteNorm maintains a substantial performance advantage for both the best and worst values across three seeds.

Scaling baseline selection. Although LayerNorm-Scaling slightly outperforms GPT-2 pre-norm in our 48-layer comparison, additional comparisons at 12 and 96 layers in Appendix § 10.§ 5 show that it falls behind GPT-2 pre-norm at greater depth, where GPT-2 pre-norm also continues to outperform KEEL. Given the computational cost of the scaling experiments we discuss in the next section, we restrict further comparisons to GPT-2 pre-norm, which we select as the more robust baseline across depth.

Downstream performance. We additionally evaluate the best final checkpoints from our 48-layer comparison on ARC-Easy and PIQA in Appendix § 10.§ 6. While the ARC-Easy results follow the validation perplexity comparison, PIQA is a binary-choice task where the scores of these relatively small models are closer to chance, making the small performance differences difficult to distinguish from noise.

Scalability

We analyse the scaling behaviour of KiteNorm against GPT-2 pre-norm across depth, training budget, batch size, and width. We generally keep the base configuration fixed except for the specific axis being varied. For depth and width sweeps, we also adjust the training budget to maintain the Chinchilla-style ratio of 20 tokens per parameter. For each method and sweep point, we report the run with the lowest final validation loss from (10−3,2⋅10−3,5⋅10−3,10−2,2⋅10−2)(10^{-3}, 2 \cdot10^{-3}, 5 \cdot10^{-3}, 10^{-2}, 2 \cdot10^{-2}).

Depth scaling. First, we analyse scaling across depth, where normalisation methods often exhibit their most pronounced failures. To that end, we start with a depth of 3 layers and double it successively until reaching 96 layers. Measuring the final validation loss of each point, we obtain the scaling law displayed in Figure 3.

Figure 3. Depth scaling law.

Both methods achieve substantially lower validation loss as depth and non-embedding parameter counts increase, but KiteNorm maintains its lead across the entire range. A narrowing gap in validation loss should not be interpreted as diminishing practical relevance. At high parameter counts, comparable absolute loss improvements require substantially larger increases in training compute.

Extending the scaling law analysis to data, batch size, and model width, we obtain the additional scaling laws shown in Figure 4.

Scaling behaviour of KiteNorm and GPT-2 pre-norm across training budget, batch size, and width in 12-layer Transformers

Figure 4. Scaling behaviour of KiteNorm and GPT-2 pre-norm across training budget, batch size, and width in 12-layer Transformers.

Training budget scaling. Inspecting the impact of training budget on final model performance, we vary the training budget as multiples of the Chinchilla rule of 20 tokens per parameter. Specifically, we test the multipliers (14,12,1,2,4,6)(\frac{1}{4}, \frac{1}{2}, 1, 2, 4, 6), obtaining the data scaling law in Figure 4a. Here, too, KiteNorm demonstrates superior performance across lower and higher budgets, with especially large relative gains under low budgets.

Batch size scaling. Since larger batch sizes are commonly used to improve hardware utilisation in decoder-only Transformer training, we investigate the performance impact of doubling our batch size from the base value of 128 up to 1024. The resulting batch size scaling law is shown in Figure 4b. KiteNorm demonstrates greater robustness to large batch sizes.

Width scaling. Finally, for completeness, we investigate the scaling of final performance across width by both doubling and halving the hidden representation size dmodeld_{\mathrm{model}}, displaying the resulting width scaling law in Figure 4c. The performance gap favouring KiteNorm is robust to these changes in width.

Combined scaling. To test whether these trends persist when depth, width, batch size, and training budget are scaled together, we train a 1B-parameter decoder-only Transformer based on the Pythia 1B configuration ([2]). Specifically, we use 16 layers, dmodel=2048d_{\mathrm{model}} = 2048, and batch size 1024, with sequence length, number of Attention heads, and number of GLU parameters matched to Figure 5: Performance of 1B-parameter models in Pythia configuration.

Figure 5a. Training loss curves.

MethodLRarc_easy (acc_norm)piqa (acc_norm)
GPT-2 pre-norm2⋅10−32 \cdot10^{-3}0.4430.670
GPT-2 pre-norm5⋅10−35 \cdot10^{-3}0.2840.530
KiteNorm2⋅10−32 \cdot10^{-3}0.4730.675
KiteNorm5⋅10−35 \cdot10^{-3}0.4700.667

Table 5b. Zero-shot downstream evaluation.

LRKiteNormGPT-2 pre-norm
2⋅10−42 \cdot10^{-4}17.38716.044
5⋅10−45 \cdot10^{-4}15.33715.464
1⋅10−31 \cdot10^{-3}14.92915.996
2⋅10−32 \cdot10^{-3}15.09015.728
5⋅10−35 \cdot10^{-3}16.053diverged

Table 6a. Final validation perplexities across learning rates.

Figure 6b. Gradient L2 norms of the GLU output projection in each layer. Measured after the final step of training with learning rate 10−310^{-3}.

Figure 6: Training stability of Llama-style 1.3B-parameter models.

the Pythia 1B specification. For these runs, we use a warm-up-then-constant learning rate schedule [31] with 572 warm-up steps and 7325 total steps for 15B tokens.

The resulting training loss curves are shown in figure 5a. KiteNorm achieves lower training loss at learning rate 2⋅10−32 \cdot10^{-3} and remains stable at learning rate 5⋅10−35 \cdot10^{-3}, where GPT-2 pre-norm diverges. The downstream results computed on the final checkpoints and shown in the right panel of figure 5 further demonstrate that KiteNorm achieves higher accuracy on ARC-Easy and PIQA at the lower learning rate, while still converging at the higher learning rate.

Further robustness tests. To investigate whether KiteNorm’s advantages extend to different architectural specifications and datasets, we adapt the 1B configuration to follow Llama 3.2 1B [16], increasing the number of Attention heads to 32 and setting the GLU expansion factor to 4. For consistency, we do not tie the input and output embeddings, yielding 1.3B-parameter models. We train these models on 22B tokens from FineWeb [21], using a warm-up-then-constant learning rate schedule with a 6% warm-up portion.

The left panel of figure 6 shows the final validation perplexities across learning rates. KiteNorm outperforms GPT-2 pre-norm at their respective optimal learning rates and remains stable across the entire learning rate grid. In contrast, GPT-2 pre-norm diverges at the highest learning rate and exhibits loss spikes at learning rates 10−310^{-3} and 2⋅10−32 \cdot10^{-3}. The training loss curves are shown in Appendix § 10.7.

We also examine gradient propagation across depth, comparing the L2 norms of the GLU output-projection gradients in the final training step of both variants of the Llama model trained with learning rate 10−310^{-3}. Figure 6b shows that KiteNorm exhibits no greater depth-wise gradient decay than GPT-2 pre-norm, with slightly more homogeneous gradient norms across layers.

Ablations

Each component of KiteNorm contributes to the above results, which we demonstrate through ablation studies shown in Figure 7. Figure 7a shows that removing the inner LayerNorm from our architecture, which yields a regularised and residually scaled post-norm with scalarised LayerNorm parameters, degrades both validation performance and stability even in a relatively shallow 12-layer model. We next compare scalarised LayerNorm against standard LayerNorm in the 24- to 96-layer models from our depth-scaling analysis. Figure 7b shows that scalarised LayerNorm improves performance, with the largest gains appearing in deeper models. Finally, we ablate the variance regulariser in a high-learning-rate stress test. As shown in Figure 7c, the 96-layer non-regularised KiteNorm model trained with learning rate 5⋅10−35 \cdot10^{-3} diverges during later training, while variance regularisation prevents this sporadic instability. Appendix § 10.8 further shows that omitting variance regularisation from GPT-2 pre-norm is valid for our comparisons, as the ReLU regulariser does not improve it.

Relation to KEEL. Our analysis of residual scaling in § 3.1 and additional ablations in Appendix § 10.9 suggest that KEEL primarily benefits from two components: branch scaling, which mitigates the initial variance expansion linked to rank collapse, and double-normalisation, which allows separate learned input scales for the two branches. We further find that KEEL’s first-layer exceptions reduce performance. Even without these exceptions, KEEL underperforms KiteNorm in our 48-layer comparison and lacks a mechanism to control the sporadic variance expansion that can occur later in training.

Supplementary ablations. In Appendix § 10.10, we investigate different choices of the residual scaling factor, regularisation strength, normalisation function, warm-up duration, and Adam betas. Our KiteNorm configuration retains its performance advantage across all of these variations.

Conclusion

In this paper, we revisited normalisation placement in deep Transformers, highlighting the trade-off between the stability of pre-norm architectures and the performance potential of post-norm architectures. We proposed KiteNorm, a post-norm variant combining residual scaling, variance regularisation and scalarised double-normalisation. KiteNorm improves both stability and scalable performance over leading baselines, with gains supported by scaling laws across depth, width, training budget, and batch size, as well as experiments on Pythia- and Llama-style models, downstream evaluations, and extensive ablations. Our theoretical analysis motivates variance control as a means of mitigating gradient vanishing, while our empirical findings link early variance expansion to rank collapse and demonstrate how KiteNorm maintains stability throughout training. These results suggest that post-norm architectures remain a viable and underexplored route for Transformer training.

Limitations

Other implementations of variance regularisation could yield further improvements. Our residual scaling rule is motivated by theoretical intuition and sensitivity analyses, rather than a complete theory of rank collapse and gradient vanishing in post-norm Transformers. Furthermore, our theoretical analysis does not establish a general guarantee against gradient vanishing. Our experiments are limited to decoder-only Transformers of up to 1.3B parameters, trained on SlimPajama-627B and FineWeb, leaving generalisation to other architectures and larger models unexplored. Finally, while our ablations demonstrate robustness to variations in several hyperparameters, our training configuration was selected through experimental tuning rather than a principled parameterisation strategy, such as that proposed by Dey et al. [5].

Acknowledgments and Disclosure of Funding

The authors acknowledge the financial support from the Hector Foundation. The authors also acknowledge the Max Planck Institute for Intelligent Systems for providing the computational resources used in this work. Antonio Orvieto is supported by the AI2050 Early Career Fellowship from Schmidt Sciences.

Proof of Theorem 1

We begin with the elementary components of the LayerNorm Jacobian. For brevity, we write μl\mu_l, σl\sigma_l to mean μ(zl)\mu(z_l), σ(zl)\sigma(z_l). We ignore the small stability constant ϵ\epsilon added in implementation.

First, consider the Jacobian of the mean subtraction:

∂(zl−μl1)∂zl=I−1∂μl∂zl=I−1d11T=I−1d11T.(14)\frac{\partial(z_l-\mu_l\mathbf{1})}{\partial z_l} = I-\mathbf{1}\frac{\partial\mu_l}{\partial z_l} = I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} = I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} . \tag*{(14)}

We also require the Jacobian of the standard deviation:

∂σl∂zl=∂∂zl(1d(zl−μl1)T(zl−μl1))=12σl∂∂zl(1d(zl−μl1)T(zl−μl1))=12σl2d(zl−μl1)T∂∂zl(zl−μl1)=12σl2d((I−1d11T)(zl−μl1))T=1dσl((zl−μl1)−1d11T(zl−μl1))T=1dσl((zl−μl1)−1d1(1Tzl−1T(1d1Tzl)1))T=1dσl((zl−μl1)−1d1(1Tzl−(1d1Tzld)))T=1dσl(zl−μl1)T.(15)\begin{aligned} \frac{\partial\sigma_l}{\partial z_l} &=\frac{\partial}{\partial z_l}\left(\sqrt{\frac{1}{d}(z_l-\mu_l\mathbf{1})^{\mathsf{T}}(z_l-\mu_l\mathbf{1})}\right)\\ &=\frac{1}{2\sigma_l}\frac{\partial}{\partial z_l}\left(\frac{1}{d}(z_l-\mu_l\mathbf{1})^{\mathsf{T}}(z_l-\mu_l\mathbf{1})\right)\\ &=\frac{1}{2\sigma_l}\frac{2}{d}(z_l-\mu_l\mathbf{1})^{\mathsf{T}}\frac{\partial}{\partial z_l}(z_l-\mu_l\mathbf{1})\\ &=\frac{1}{2\sigma_l}\frac{2}{d}\left(\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)(z_l-\mu_l\mathbf{1})\right)^{\mathsf{T}}\\ &=\frac{1}{d\sigma_l}\left((z_l-\mu_l\mathbf{1})-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}(z_l-\mu_l\mathbf{1})\right)^{\mathsf{T}}\\ &=\frac{1}{d\sigma_l}\left((z_l-\mu_l\mathbf{1})-\frac{1}{d}\mathbf{1}\left(\mathbf{1}^{\mathsf{T}}z_l-\mathbf{1}^{\mathsf{T}}\left(\frac{1}{d}\mathbf{1}^{\mathsf{T}}z_l\right)\mathbf{1}\right)\right)^{\mathsf{T}}\\ &=\frac{1}{d\sigma_l}\left((z_l-\mu_l\mathbf{1})-\frac{1}{d}\mathbf{1}\left(\mathbf{1}^{\mathsf{T}}z_l-\left(\frac{1}{d}\mathbf{1}^{\mathsf{T}}z_ld\right)\right)\right)^{\mathsf{T}}\\ &=\frac{1}{d\sigma_l}(z_l-\mu_l\mathbf{1})^{\mathsf{T}} . \tag*{(15)} \end{aligned}

Let again z~l=zl−μl1σl\widetilde{z}_l=\frac{z_l-\mu_l\mathbf{1}}{\sigma_l}. We can now calculate the Jacobian of normalisation:

∂z~l∂zl=∂∂zl(σl−1(zl−μl1))=σl−1∂(zl−μl1)∂zl+(zl−μl1)∂(σl−1)∂zl=σl−1(I−1d11T)+(zl−μl1)(−σl−2∂σl∂zl)=σl−1(I−1d11T)−(zl−μl1)σl−2(1dσl(zl−μl1)T)=1σl(I−1d11T)−1dσl3(zl−μl1)(zl−μl1)T=1σl(I−1d11T−1dσl2(zl−μl1)(zl−μl1)T)=1σl(I−1d11T−1d(z~lσl)(z~lσl)T)=1σl(I−1d11T−1dz~lz~lT).(16)\begin{aligned} \frac{\partial\widetilde{z}_l}{\partial z_l} &=\frac{\partial}{\partial z_l}\left(\sigma_l^{-1}(z_l-\mu_l\mathbf{1})\right)\\ &=\sigma_l^{-1}\frac{\partial(z_l-\mu_l\mathbf{1})}{\partial z_l} +(z_l-\mu_l\mathbf{1})\frac{\partial(\sigma_l^{-1})}{\partial z_l}\\ &=\sigma_l^{-1}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right) +(z_l-\mu_l\mathbf{1})\left(-\sigma_l^{-2}\frac{\partial\sigma_l}{\partial z_l}\right)\\ &=\sigma_l^{-1}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right) -(z_l-\mu_l\mathbf{1})\sigma_l^{-2}\left(\frac{1}{d\sigma_l}(z_l-\mu_l\mathbf{1})^{\mathsf{T}}\right)\\ &=\frac{1}{\sigma_l}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right) -\frac{1}{d\sigma_l^3}(z_l-\mu_l\mathbf{1})(z_l-\mu_l\mathbf{1})^{\mathsf{T}}\\ &=\frac{1}{\sigma_l}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} -\frac{1}{d\sigma_l^2}(z_l-\mu_l\mathbf{1})(z_l-\mu_l\mathbf{1})^{\mathsf{T}}\right)\\ &=\frac{1}{\sigma_l}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} -\frac{1}{d}(\widetilde{z}_l\sigma_l)(\widetilde{z}_l\sigma_l)^{\mathsf{T}}\right)\\ &=\frac{1}{\sigma_l}\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} -\frac{1}{d}\widetilde{z}_l\widetilde{z}_l^{\mathsf{T}}\right). \tag*{(16)} \end{aligned}

Finally, consider the Jacobian of LayerNorm itself:

∂LN⁡l(zl)∂zl=∂LN⁡l(zl)∂z~l∂z~l∂zl=Diag⁡(γ)∂z~l∂zl=1σlDiag⁡(γ)(I−1d11T−1dz~lz~lT).(17)\frac{\partial\operatorname{LN}_l(z_l)}{\partial z_l} =\frac{\partial\operatorname{LN}_l(z_l)}{\partial\widetilde{z}_l} \frac{\partial\widetilde{z}_l}{\partial z_l} =\operatorname{Diag}(\gamma)\frac{\partial\widetilde{z}_l}{\partial z_l} =\frac{1}{\sigma_l}\operatorname{Diag}(\gamma)\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\widetilde{z}_l\widetilde{z}_l^{\mathsf{T}}\right). \tag*{(17)}

Since we are interested in how this Jacobian may contract the gradient, we analyse its spectral norm:

∥∂LNl(zl)∂zl∥2≤1σl⋅∥Diag⁡(γ)∥2⋅∥I−1d11T−1dz~lz~lT∥2(18)\left\lVert\frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l} \right\rVert_2 \le\frac{1}{\sigma_l}\cdot\left\lVert\operatorname{Diag}(\gamma) \right\rVert_2 \cdot\left\lVert I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}} \right\rVert_2 \tag*{(18)}

It is trivial that ∥Diag⁡(γ)∥2=∥γ∥∞\left\lVert\operatorname{Diag}(\gamma) \right\rVert_2=\left\lVert\gamma\right\rVert_\infty. Regarding I−1d11T−1dz~lz~lTI-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}, we prove in the following that this is an orthogonal projection matrix.

Since z~l\tilde{z}_l is normalised, we have 1Tz~l=0\mathbf{1}^{\mathsf{T}}\tilde{z}_l=0 and ∥z~l∥22=d\left\lVert\tilde{z}_l \right\rVert_2^2=d. Likewise, ∥1∥22=d\left\lVert\mathbf{1} \right\rVert_2^2=d. First we show that 1d11T\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} is an orthogonal projector:

(1d11T)2=1d211T11T=1d21(d)1T=1d11T.(19)\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)^2=\frac{1}{d^2}\mathbf{1}\mathbf{1}^{\mathsf{T}}\mathbf{1}\mathbf{1}^{\mathsf{T}}=\frac{1}{d^2}\mathbf{1}(d)\mathbf{1}^{\mathsf{T}}=\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}. \tag*{(19)}

As symmetry is trivial, 1d11T\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}} is an orthogonal projector. It is also trivial that the same argument holds for 1dz~lz~lT\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}. Thus they are both orthogonal projectors.

We additionally show that they are also orthogonal to each other, using 1Tz~l=0\mathbf{1}^{\mathsf{T}}\tilde{z}_l=0:

(1dz~lz~lT)(1d11T)=1d2z~l(0)1T=0=1d21(0)z~lT=(1d11T)(1dz~lz~lT).(20)\left(\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right)\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)=\frac{1}{d^2}\tilde{z}_l(0)\mathbf{1}^{\mathsf{T}}=0=\frac{1}{d^2}\mathbf{1}(0)\tilde{z}_l^{\mathsf{T}}=\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)\left(\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right). \tag*{(20)}

Finally, we use this to show that I−1d11T−1dz~lz~lTI-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}} itself is idempotent:

(I−1d11T−1dz~lz~lT)2=I−1d11T−1dz~lz~lT−1d11T+(1d11T)2+(1dz~lz~lT)(1d11T)−1dz~lz~lT+(1dz~lz~lT)2+(1d11T)(1dz~lz~lT)=I−1d11T−1dz~lz~lT−1d11T+1d11T+0−1dz~lz~lT+1dz~lz~lT+0=I−1d11T−1dz~lz~lT.(21)\begin{aligned} \left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right)^2 &=I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\\ &\quad-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}+\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)^2+\left(\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right)\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)\\ &\quad-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}+\left(\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right)^2+\left(\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)\left(\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\right)\\ &=I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}\\ &\quad-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}+\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}+0\\ &\quad-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}+\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}+0 =I-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}}. \tag*{(21)} \end{aligned}

Hence, considering that any sum of symmetric matrices is itself also symmetric, I−1d11T−1dz~lz~lTI-\frac{1}{d}\mathbf{1}\mathbf{1}^{\mathsf{T}}-\frac{1}{d}\tilde{z}_l\tilde{z}_l^{\mathsf{T}} is an orthogonal projection matrix. Its eigenvalues are therefore contained in {0,1}\{0,1\}, so its spectral norm is at most 1.

Consequently, we obtain an upper bound on the spectral norm of the LayerNorm Jacobian:

∥∂LNl(zl)∂zl∥2≤∥γ∥∞σ(zl).(22)\left\lVert\frac{\partial\mathrm{LN}_l(z_l)}{\partial z_l} \right\rVert_2 \le\frac{\left\lVert\gamma\right\rVert_\infty}{\sigma(z_l)}. \tag*{(22)}

Specification

Training Setup Details

All models except the 1B-scale models discussed in § 4.2 were trained on up to 8 NVIDIA A100-SXM4-80GB GPUs. The 1B-scale models were trained on 4 NVIDIA B200 GPUs. Altogether, approximately 1000 NVIDIA A100-SXM4-80GB GPU hours and 250 NVIDIA B200 GPU hours were used for our experiments.

In Table 1 we state the baseline hyperparameters shared between all configurations unless otherwise stated. The learning rate is not a shared hyperparameter, because it is the parameter tuned in the learning-rate explorations.

HYPERPARAMETERSHARED VALUE
training corpusSlimPajama-627B
training objectivenext-token cross entropy
sequence length2048
vocabulary size50280
dmodeld_{\mathrm{model}}512
dGLUd_{\mathrm{GLU}}1536
number of heads8
sublayers per Transformer layerAttention followed by GLU
Attentioncausal self-Attention with RoPE
RoPE base500000
GLU nonlinearitySiLU gate
linear projectionsno bias
tied token embeddingsFalse
dropoutnone
batch size128 sequences per optimiser step
optimiserAdamW
schedulerlinear warm-up followed by cosine decay
warm-up fraction6%
warm-up start lr1⋅10−101 \cdot10^{-10}
decay end lr0.0
weight decay0.1 on matrix parameters and 0.0 on bias and normalisation parameters
dtypebfloat16
TF32enabled
β1\beta_10.95
β2\beta_20.95
LayerNorm ϵ\epsilon1⋅10−61 \cdot10^{-6}
gradient clipping threshold1.0
random seed7 unless reseeding is specified
sampler seed7 unless reseeding is specified

Table 1. List of shared hyperparameters.

The batch size in Table 1 denotes the effective batch size after distributed training and gradient accumulation. Thus, for the 48-layer comparison against leading baselines we use four GPUs, micro-batch size 16 per GPU, and two gradient accumulation steps. For the 12-layer base configuration we use one GPU, micro-batch size 32, and four gradient accumulation steps. The default 12-layer and 48-layer configurations use step budgets 6866 and 12207 respectively, corresponding to the Chinchilla-style token budgets used throughout the paper.

The Pythia 1B configuration uses L=16L = 16, dmodel=2048d_{\mathrm{model}} = 2048, batch size 1024, and 7325 training steps for 15B tokens. Its GLU width is chosen to match the parameter count of the MLP in the Pythia 1B specification. The Llama-style configuration additionally uses 32 Attention heads and a GLU expansion factor of 4, and is trained on 22B tokens. Both configurations use warm-up-then-constant learning rate schedules, with their respective warm-up durations specified in § 4.2.

Implementation Details

We distinguish the Transformer layer index k∈{1,…,L}k \in\{1,\ldots,L\} from the sublayer index l∈{1,…,2L}l \in\{1,\ldots,2L\}. Each Transformer layer contains one Attention sublayer and one GLU sublayer. We write the residual branch function of sublayer ll as FlF_l, and define

Mix⁡a,c(U,V)=aU+cV.(23)\operatorname{Mix}_{a,c}(U,V)=aU+cV. \tag*{(23)}

Fixed branch scalings are applied in the forward pass through Mix⁡a,c\operatorname{Mix}_{a,c}, whereas initialisation scalings are applied once to the output projection matrices immediately after weight initialisation. Unless otherwise stated, all linear and embedding weights are initialised from N(0,0.022)\mathcal{N}(0,0.02^2), all LayerNorm scale parameters are initialised to 1, and all LayerNorm shift parameters are initialised to 0.

LayerNorm statistics are computed over the hidden dimension without Bessel correction. Standard LayerNorm uses vector-valued parameters γ,β∈Rdmodel\gamma,\beta\in\mathbb{R}^{d_{\mathrm{model}}} when the shift is enabled. Scalarised LayerNorm uses scalar parameters γ′,β′∈R\gamma',\beta' \in\mathbb{R} as in equation (13). Matching equation (12), the implemented ReLU variance regulariser used by KiteNorm is

RReLU=12L∑l=12LEb,t[ReLU⁡(σ(zl,b,t)2−1)].(24)\mathcal{R}_{\mathrm{ReLU}}=\frac{1}{2L}\sum_{l=1}^{2L}\mathbb{E}_{b,t}\left[\operatorname{ReLU}\left(\sigma(z_{l,b,t})^2-1\right)\right]. \tag*{(24)}

where zlz_l is the hidden state immediately after the residual addition and immediately before any outer normalisation in sublayer ll, and σ(⋅)\sigma(\cdot) is the variance over the hidden dimension. The training loss for KiteNorm is LCE+λRReLU\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{R}_{\mathrm{ReLU}} with λ=1\lambda=1. The other baselines do not use variance regularisation.

Table 2 states the exact implementation of the scaling and normalisation schemes used in the comparison of competitive approaches.

METHODSUBLAYER IMPLEMENTATIONSCALING DETAILS
GPT-2 pre-normXl+1=Xl+Fl(LN(Xl))X_{l+1}=X_l+\mathcal{F}_l(\mathrm{LN}(X_l)), with a final output LayerNorm. Standard LayerNorm includes the shift parameter.Runtime skip and residual scales are both 1. The Attention output projection and GLU output projection in every Transformer layer are multiplied at initialisation by 12L\frac{1}{\sqrt{2L}}.
LayerNorm-ScalingPre-norm with LN replaced in both sublayers of Transformer layer kk by LNSk(X)=k−1/2LN(X)\mathrm{LNS}_k(X)=k^{-1/2}\mathrm{LN}(X). The final output normalisation is standard LayerNorm, not LayerNorm-Scaling.Runtime skip and residual scales are both 1. No residual output projection scaling is applied at initialisation. Standard LayerNorm includes the shift parameter.
Post-normXl+1=LN(Xl+Fl(Xl))X_{l+1}=\mathrm{LN}(X_l+\mathcal{F}_l(X_l)), without a final output LayerNorm. Standard LayerNorm includes the shift parameter.Runtime skip and residual scales are both 1. No residual output projection scaling is applied at initialisation.
KEELDouble-norm: Xl+1=LN(Mixαl,1(Xl,Fl(LN(Xl))))X_{l+1}=\mathrm{LN}(\mathrm{Mix}_{\alpha_l,1}(X_l,\mathcal{F}_l(\mathrm{LN}(X_l)))). The outer LayerNorm after the first Attention sublayer is omitted. Standard LayerNorm is used without the shift parameter.For the two sublayers belonging to Transformer layer k=1k=1, αl=1\alpha_l=1. For all sublayers belonging to Transformer layers k≥2k\ge2, αl=2L\alpha_l=2L. The residual branch scale is always 1. No residual output projection scaling is applied at initialisation. The main comparison uses GLUs. The equal-parameter MLP variant is only used in Appendix C.2.
Peri-normXl+1=Xl+LN(Fl(LN(Xl)))X_{l+1}=X_l+\mathrm{LN}(\mathcal{F}_l(\mathrm{LN}(X_l))). There is no final output LayerNorm. Standard LayerNorm includes the shift parameter.Runtime skip and residual scales are both 1. No residual output projection scaling is applied at initialisation.
Mix-normThe first ⌊0.25L⌋\lfloor0.25L\rfloor Transformer layers are post-norm layers and the remaining L−⌊0.25L⌋L-\lfloor0.25L\rfloor Transformer layers are pre-norm layers. A final output LayerNorm is used because the model ends with a pre-norm suffix. Standard LayerNorm includes the shift parameter.Runtime skip and residual scales are both 1. Let Lpre=L−⌊0.25L⌋L_{\mathrm{pre}}=L-\lfloor0.25L\rfloor. Only the Attention output projection and GLU output projection in the pre-norm suffix are multiplied at initialisation by 12Lpre\frac{1}{\sqrt{2L_{\mathrm{pre}}}}, following the partial GPT-2 initialisation selected in Appendix C.3.
KiteNormScalarised double-norm: Xl+1=LNscalarised(Xl+12LFl(LNscalarised(Xl)))X_{l+1}=\mathrm{LN}_{\mathrm{scalarised}}\left(X_l+\frac{1}{2L}\mathcal{F}_l(\mathrm{LN}_{\mathrm{scalarised}}(X_l))\right). There is no final output LayerNorm.The skip branch scale is 1 and the residual branch scale is 12L\frac{1}{2L} in every sublayer. No residual output projection scaling is applied at initialisation. The ReLU variance regulariser in equation 24 is used with λ=1\lambda=1.

Table 2. Exact implementation of the normalisation and scaling schemes in the comparison of competitive approaches.

The learning-rate exploration in Figure 2 varies only the maximum learning rate of the scheduler. All other hyperparameters are held fixed according to Table 1 and Table 2. The reseeded runs use the best learning rate from the displayed grid for each method.

Downstream Evaluation

We evaluate the final checkpoints of selected models on ARC-Easy and PIQA using the Language Model Evaluation Harness. Evaluations are performed in the zero-shot setting over the complete test sets, reporting both standard accuracy and length-normalised accuracy. Models are evaluated with their respective training configurations and tokenisers, using a batch size of 64 and a maximum sequence length of 2048. Evaluations are performed on a single NVIDIA A100-SXM4-80GB GPU.

Supplementary Experiments

Unless otherwise specified, all models use the setup and hyperparameters given in Appendix B. In tables, – denotes an untested configuration, while div. indicates training divergence.

Residual Scaling Variants of Pre-Norm

In order to identify the strongest pre-norm baseline to compare against, we conduct learning rate exploration with unscaled pre-norm and a reparameterisation of GPT-2 pre-norm that applies the scaling in the sublayer, rather than at initialisation. Comparing the result against GPT-2 pre-norm, we obtain figures 8a and 8b respectively. For unscaled pre-norm, we note reduced robustness to high learning rates and decreased performance. For pre-norm with fixed scaling, we observe deteriorated performance at depth. These findings motivate us to compare against GPT-2 pre-norm.

Learning-rate explorations comparing GPT-2 pre-norm with unscaled pre-norm and fixed-scale pre-norm

Figure 8. (a) Comparison against unscaled pre-norm in a 12-layer Transformer. (b) Comparison against fixed scale pre-norm in a 48-layer Transformer. Figure 8: Learning-rate explorations for pre-norm variants.

GLU Variant of KEEL

We compare the performance of a KEEL model with our GLU to one with an MLP, where both have the same number of parameters. The result is presented in figure 9, where we find that the GLU variant performs better.

Figure 9. Comparison between KEEL models with GLU and models with MLP in a 12-layer Transformer.

Residual Scaling for Mix-Norm

Given that GPT-2 style residual scaling can improve pre-norm methods, we evaluate whether this also applies to the pre-norm section of mix-norm models. Here, we compare vanilla mix-norm against a variant where GPT-2 style residual scaling is applied in the pre-norm sublayers, displaying the result in Figure 10. Since we observe a clear performance improvement when using GPT-2 style residual scaling, we apply it in all other mix-norm runs.

Figure 10. Comparison between vanilla mix-norm and mix-norm with GPT-2 style residual scaling in a 12-layer Transformer.

Rank Collapse with Peri-Norm

Since we find the learning rate exploration yields an oddly non-hyperbolic curve for peri-norm, we analyse its dynamics and find that peri-norm is prone to rank collapse. We show this empirically by analysing the token alignment in a peri-norm model trained with a learning rate of 10−210^{-2}, seen in Figure 11. Indeed, while the first layer is often an exception in Transformer models, the subsequent layers show a clear tendency to rank collapse, plausibly explaining the poor performance of peri-norm in our experiments.

Figure 11. Rank collapse in hidden states after the residual addition in Attention sublayers when training a 12-layer Transformer with peri-norm.

Additional Baseline Comparisons Across Depth

We extend our comparison against KEEL and LayerNorm-Scaling to 12- and 96-layer Transformers. Table 3 and Table 4 report final validation perplexities across learning rates, following the setup described in Appendix § 9.

Method2e-35e-31e-22e-25e-2
KiteNorm–25.18124.83125.180–
GPT-2 pre-norm–26.90926.660div.–
LayerNorm-Scaling–26.99126.64526.615div.
KEEL28.29627.64428.364––

Table 3. Final validation perplexities across learning rates in 12-layer Transformers.

Method1e-32e-35e-31e-2
KiteNorm–16.37116.18916.719
GPT-2 pre-norm16.99116.75516.99517.616
KEEL18.10117.12616.889div.
LayerNorm-Scaling–18.15317.13317.167

Table 4. Final validation perplexities across learning rates in 96-layer Transformers.

KiteNorm achieves the lowest validation perplexity at both depths. While LayerNorm-Scaling slightly outperforms GPT-2 pre-norm at 12 layers, it falls behind at 96 layers. KEEL also remains behind KiteNorm at both depths.

Downstream Evaluation at 48 Layers

We evaluate the best final checkpoints from the 48-layer comparison in § 4.1 on ARC-Easy and PIQA, following Appendix § 9.§ 3. Table 5 reports standard accuracy and length-normalised accuracy.

MethodARC-EasyPIQA
accacc_normaccacc_norm
KiteNorm0.4550.4040.6280.625
GPT-2 pre-norm0.4260.3830.6090.621
LayerNorm-Scaling0.4360.3810.6330.624
KEEL0.4260.3850.6120.616
Peri-norm0.4140.3810.6250.613
Mix-norm0.4140.3730.6140.604
Post-norm0.4000.3560.5940.591

Table 5. Zero-shot downstream evaluation of the best final checkpoints from the 48-layer comparison.

KiteNorm achieves the highest accuracy on ARC-Easy under both metrics, consistent with its lower validation perplexity. On PIQA, the differences are smaller across all configurations, with KiteNorm achieving the highest length-normalised accuracy and LayerNorm-Scaling achieving the highest standard accuracy.

Complete Llama Training Loss Curves

We show the complete training loss curves for the Llama experiments from section 4.2 in Figure 12. KiteNorm trains without loss spikes across the explored learning rates, whereas GPT-2 pre-norm exhibits loss spikes at learning rates 10−310^{-3} and 2⋅10−32 \cdot10^{-3} and diverges at 5⋅10−35 \cdot10^{-3}.

Figure 12. Complete training loss curves of Llama-style 1.3B-parameter models across learning rates.

GPT-2 Pre-Norm With the ReLU Regulariser

We explore whether our ReLU regulariser has the potential to improve the GPT-2 pre-norm baseline to ensure a fair comparison. In a 12-layer Transformer, we train GPT-2 pre-norm with the ReLU regulariser enabled, yielding the loss curves displayed in Figure 13. The result shows clearly elevated instability when enabling variance regularisation while training GPT-2 pre-norm.

Figure 13. Comparison between training of GPT-2 pre-norm with and without the ReLU regulariser enabled in a 12-layer Transformer.

KEEL Architectural Ablations

To further investigate the relationship between KiteNorm and KEEL, we ablate KEEL’s double-normalisation and first-layer architectural exceptions in the 48-layer comparison setting from § 4.1. The resulting validation perplexities are shown in Table 6.

Configuration2e-35e-31e-22e-2
KEEL20.76620.420div.–
No double-norm21.382div.div.–
No exceptions20.37919.61219.640div.
Neither21.09319.936div.–

Table 6. Final validation perplexities of KEEL architectural variants in a 48-layer Transformer across learning rates.

We find that removing the first-layer exceptions consistently improves performance. Double-normalisation also improves performance and, in some configurations, stability. However, even this improved KEEL variant does not match KiteNorm’s performance in the main 48-layer comparison.

Sensitivity Analyses

Residual scaling. We find that the residual scaling rule presents a trade-off between performance and stability in post-norm models. Table 7 presents the final validation perplexities of 24-layer post-norm Transformers with variance regularisation enabled, but without double-normalisation or scalarised LayerNorm. Larger residual scaling improves performance at the expense of stability. The factor 1/(2L)1/(2L), approximately 0.02 at this depth, provides a practical compromise.

Residual scale2e-35e-31e-2
0.0525.95324.380div.
0.0227.14124.94126.176
0.0128.61526.24227.840

Table 7. Final validation perplexities in 24-layer post-norm models with variance regularisation and varying residual scaling.

We revisit a scaling rule of exactly 1/L1/L in the 48-layer KiteNorm configuration from § 4.1. As shown in Table 8, the alternative rule does not improve performance.

Scaling rule2e-35e-31e-2
1/(2L)1/(2L)19.72619.08019.100
1/L1/L–19.12519.199

Table 8. Final validation perplexities in 48-layer KiteNorm models with varying residual scaling.

Regularisation strength. We vary the regularisation weight λ\lambda in the 48-layer setting of § 4.1. Table 9 shows that λ=1\lambda= 1 achieves the best performance among the tested values, although the method remains effective with both weaker and stronger regularisation.

λ\lambda2e-35e-31e-2
0.119.75519.19019.277
1.019.72619.08019.100
10.020.03619.43119.337

Table 9. Final validation perplexities in 48-layer KiteNorm models with different regularisation weights.

Normalisation choice. We compare KiteNorm using scalarised LayerNorm against a variant with RMSNorm and a scalarised gain parameter. We also compare KEEL using LayerNorm and RMSNorm, following its original normalisation choice. The results in Table 10 show small performance differences between the normalisation variants, while KiteNorm retains its advantage over KEEL.

Configuration2e-35e-31e-22e-2
KiteNorm, scalarLN19.72619.08019.10019.868
KiteNorm, scalarRMSNorm19.81119.15819.25321.113
KEEL, LN20.76620.420div.–
KEEL, RMSNorm20.81620.321div.–

Table 10. Final validation perplexities in 48-layer Transformers with differently parameterised LayerNorm and RMSNorm variants.

Optimisation hyperparameters. We investigate whether longer learning rate warm-up affects the relative performance of KiteNorm and the baselines. Table 11 compares warm-up portions of 6% and 12% in the 48-layer setting. Longer warm-up improves the stability of KEEL and shifts the optimal learning rates, but KiteNorm retains its performance advantage.

Configuration2e-35e-31e-22e-2
KiteNorm, 6%19.72619.08019.10019.868
KiteNorm, 12%19.92019.24019.12319.565
GPT-2 pre-norm, 6%20.37620.24020.951div.
GPT-2 pre-norm, 12%20.26520.17220.13921.743
KEEL, 6%20.76620.420div.–
KEEL, 12%20.75819.82920.115–

Table 11. Final validation perplexities in 48-layer Transformers with differing warm-up durations.

We additionally vary the Adam beta parameters, keeping β1=β2\beta_{1} = \beta_{2} throughout. Table 12 shows that the performance advantage of KiteNorm persists across the tested values.

Configuration2e-35e-31e-2
KiteNorm, 0.920.07219.40019.290
KiteNorm, 0.9519.72619.08019.100
KiteNorm, 0.97519.72519.21919.589
GPT-2 pre-norm, 0.920.69920.515div.
GPT-2 pre-norm, 0.9520.37620.24020.951
GPT-2 pre-norm, 0.97520.76120.56220.707

Table 12. Final validation perplexities in 48-layer Transformers when trained with different Adam beta parameters.

References

  1. [1]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. [2]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afia Kahn, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pp. 2397–2430. PMLR, 2023.arxiv.org/abs/2304.01373
  3. [3]Chen Chen and Lai Wei. Post-layernorm is back: Stable, expressive, and deep. arXiv preprint arXiv:2601.19895, 2026.arxiv.org/abs/2601.19895
  4. [4]Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
  5. [5]Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: Complete enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618, 2025.arxiv.org/abs/2505.01618
  6. [6]Ming Ding, Zhuoyi Yang, Wenyı Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34:19822–19835, 2021.
  7. [7]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pp. 2793–2803. PMLR, 2021.arxiv.org/abs/2103.03404
  8. [8]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL https://zenodo.org/records/12608602.
  9. [9]Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024.arxiv.org/abs/2403.17887
  10. [10]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022.arxiv.org/abs/2203.15556
  11. [11]Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning, pp. 4475–4483. PMLR, 2020.
  12. [12]Federico Arangath Joseph, Jerome Sieber, Melanie N Zeilinger, and Carmen Amo Alonso. Lambda-skip connections: the architectural component that prevents rank collapse. arXiv preprint arXiv:2410.10609, 2024.arxiv.org/abs/2410.10609
  13. [13]Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-ln: Revisiting normalization layer in the transformer architecture. arXiv preprint arXiv:2502.02732, 2025.arxiv.org/abs/2502.02732
  14. [14]Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795, 2024.arxiv.org/abs/2412.13795
  15. [15]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
  16. [16]Meta. Llama 3.2 model card, 2024. URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md.
  17. [17]Ido Nachum, Jan Hązła, Michael Gaspar, and Anatoly Khina. A johnson–lindenstrauss framework for randomly initialized cnns. arXiv preprint arXiv:2111.02155, 2021.arxiv.org/abs/2111.02155
  18. [18]Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Proceedings of the 16th international conference on spoken language translation, 2019.DOI
  19. [19]Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198–27211, 2022.arxiv.org/abs/2206.03126
  20. [20]Antonio Orvieto and Robert M Gower. In search of adam’s secret sauce. arXiv preprint arXiv:2505.21829, 2025.arxiv.org/abs/2505.21829
  21. [21]Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=n6SCkn2qAg.arxiv.org/abs/2406.17557
  22. [22]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  23. [23]Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  24. [24]Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R. Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627b token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. Accessed: 2026-05-06.
  25. [25]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  26. [26]Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498, 2026.
  27. [27]Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models. arXiv preprint arXiv:2502.05795, 2025.
  28. [28]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  29. [29]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024.
  30. [30]Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1810–1822, 2019.
  31. [31]Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024.
  32. [32]Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference on machine learning, pp. 10524–10533. PMLR, 2020.

Paper details

vlnc:W_d7mykym2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents