Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Introduction

Scaling language models typically involves increasing parameters, training data, and computation together Henighan et al., [13]. Looped Transformers offer another dimension for scaling computation: repeatedly applying a shared block increases effective depth without assigning independent parameters to every executed layer [6, 18]. Recent work further demonstrates that recurrent depth can improve reasoning performance through additional computation at test time Geiping et al., [9]. This opportunity makes the organization of recurrence a central design concern. Performance depends not only on how many iterations a model executes, but also on which computations it repeats and how contextual information remains available across iterations.

Recurrence can be organized through different layer topologies, depending on which layers are repeated and which non-recurrent layers precede or follow them Huang et al., [16]. A straightforward choice is to loop the entire Transformer stack, feeding the output of each pass back into the same stack for the next Zhu et al., [32]; we refer to this topology as Full Loop. However, analyses of pretrained language models suggest that different depths support distinct computational roles, including contextualizing input tokens, refining features, and forming predictions Lad et al., [17]. Repeating the entire stack places these roles together within the recurrent computation, without explicitly separating input processing from iterative refinement. Analyses of looped models likewise observe multiple feedforward-like processing stages recurring within each iteration ([2]). These findings motivate organizing recurrence around a division of roles: distinguishing context construction that can be performed once from state refinement to which additional computation can be allocated.

Partial looping has been explored in recurrent language models, repeating only a selected subset of layers while executing the remaining layers once ([9, 16]). Controlled comparisons with matched parameter counts, effective depth, and training budgets report improvements over full-stack looping on several multi-step reasoning benchmarks ([16]). Such designs make the distinction between recurrent and non-recurrent computation explicit, but choosing the scope of recurrence alone does not determine how the two components should work together. We argue that their different computational roles should guide a broader asymmetric design: non-recurrent layers can specialize in context construction, while recurrent layers perform iterative state updates, with information exchange connecting these two types of computation.

We introduce the Asymmetric Looped Transformer (AsymLoop), which connects a single-pass causal encoder to a weight-tied recurrent decoder through joint attention to fixed Global KV and evolving Local KV. A causal encoder runs once and projects its output into per-decoder-layer Global key–value (KV) caches. These caches serve as contextual anchors that remain fixed across all iterations of a forward pass. A weight-tied decoder generates Local KV from its current hidden states at each iteration. The two sources offer complementary views: Global KV preserves access to the context encoded by the encoder, while Local KV reflects the decoder’s current intermediate state. Queries allocate attention jointly across both sources under a single softmax, enabling each update to integrate global contextual information with the current local state. All components are trained end to end.

For optimization, we adopt the heavy-tail-guided layerwise learning-rate method of He et al. (2026) for the encoder and decoder. The same spectral allocation rule is applied to matrices in both components. Encoder–decoder learning-rate differences arise from their measured spectral statistics. We use Asymmetric Learning Rate Schedule (ALRS) as a shorthand for this training configuration and examine the resulting allocation patterns and their empirical effects in our experiments.

Our main contributions are:

• An asymmetric recurrent architecture. We jointly design one-time context encoding and weight-tied recurrent updates, connecting them through joint attention to fixed Global KV and dynamic Local KV.

• Analysis of context access under recurrence. Through component ablations and comparisons across recurrent depths, we examine the roles of fixed Global KV and evolving Local KV in recurrent state refinement.

• Performance gains across model sizes. Across four model sizes, AsymLoop achieves lower validation perplexity and higher average zero-shot accuracy than Full Loop under matched training budgets and effective depth. Perplexity improves by 0.98 and 0.30 at 135M and 350M, respectively, while the seven-task average accuracy at 1B rises from 49.58% to 50.85%.

Related Work

Looped Transformers and Recurrent Depth. Weight-tied recurrence in Transformers traces back to Universal Transformers ([6]) and ALBERT ([18]), which established that parameter sharing across layers can achieve competitive linguistic representations. Theoretical investigations have formalized looped Transformers as programmable computers ([10]) capable of learning complex in-context algorithms ([30]) and latent thoughts ([28]). More recently, scaling up test-time compute through recurrent depth has emerged as a promising alternative to chain-of-thought token generation ([9]). Studies such as Parcae ([25]) and FLT ([7]) explore stability in looped models. Beyond full-stack recurrence, partial-loop architectures already separate recurrent computation from non-recurrent input and output processing ([9]). Our focus is the information exchange between these components: AsymLoop provides the recurrent decoder with layer-specific Global KV from a single-pass encoder, jointly accessed with evolving Local KV at every iteration.

Causal Encoder-Decoder Architectures. Standard autoregressive language modeling typically employs decoder-only stacks. YOCO lets upper layers reuse KV produced by lower causal layers [29]. The Causal Encoder-Decoder (CED) proposed in DeepSeek-V4.1-Flash [5] divides the Transformer into an encoder that generates a static Global KV cache once, and a decoder that attends to this cache alongside standard causal dependencies. While CED demonstrated KV cache savings during inference, it was conceived as a non-recurrent ($K = 1$) feedforward model. Their asymmetry also suggests a way to organize recurrent language models: build a causal representation of the input once, then let a recurrent stack iteratively refine its state while retaining access to that representation.

Heavy-Tailed Self-Regularization and Layerwise Optimization. Heavy-Tailed Self-Regularization (HT-SR) characterizes weight spectra through fitted power-law tails (Martin & Mahoney, 2021; Martin et al., 2021). He et al. (2026) use these spectral exponents to allocate learning rates across layers and modules. We apply their allocation rule to the non-recurrent encoder and the weight-tied decoder. Encoder parameters are used once per forward pass, whereas decoder parameters are reused across iterations and accumulate gradient contributions from each use. This difference motivates allowing their learning rates to vary.

The Asymmetric Looped Architecture

Asymmetric Encoder–Decoder Design

As shown in Figure 1, AsymLoop consists of a non-recurrent causal encoder and a weight-tied recurrent decoder. The encoder uses the same causal Transformer blocks as a standard decoder-only language model, without changing their architecture. We split $L$ physical layers into $E$ encoder layers and $D$ decoder layers, with $E = D = L/2$ in our experiments. A single forward pass through the encoder produces contextual representations $H_E \in\mathbb{R}^{N \times d}$, where $N$ is the sequence length and $d$ is the hidden dimension. The encoder output serves as the input to the first decoder iteration. Separate projection matrices for each decoder layer map this output into layer-specific Global KV caches, which are reused across all $K$ iterations.

The decoder runs for $K$ iterations, feeding the output of each iteration into the next. Denote its $D$ layers by $\mathcal{D}_1, \ldots, \mathcal{D}_D$, and the Global KV for layer $\mathcal{D}_l$ by $\mathcal{G}_l$, defined below. At iteration $t$, $X^{(t)}$ denotes the input to the entire decoder, and $Z_l^{(t)}$ denotes the output of decoder layer $\mathcal{D}_l$. Thus, $Z_D^{(t)}$ is the output of the final decoder layer $\mathcal{D}_D$ and of the entire decoder at that iteration. The first iteration receives the encoder output; each subsequent iteration receives the preceding iteration’s decoder output. Within each iteration, the first layer processes $X^{(t)}$, and the remaining layers process the preceding layer’s output:

Each $\mathcal{D}_l$ has distinct parameters shared across all $K$ iterations. The final state $Z_D^{(K)}$ is used for next-token prediction, and the model is trained end to end through all iterations.

Global–Local Fusion via Joint Attention

Each decoder layer attends to two sources: fixed Global KV from the encoder and evolving Local KV from the current decoder state. For layer $l$, Global KV is obtained using layer-specific projections:

At each iteration, the layer projects its current RMS-normalized input into queries and Local KV. Before attention, we apply QKNorm [14], followed by RoPE, to queries and both Global and Local keys. The resulting Global keys and values form $\mathcal{G}_l = (K_l^{\mathrm{glob}}, V_l^{\mathrm{glob}})$, which remains fixed across all $K$ iterations of a forward pass; queries and Local KV are recomputed at each iteration.

Each attention head reads both KV sources under a single softmax. Omitting layer and iteration indices, the attention output is

where $d_h$ is the head dimension and $M$ is the causal mask. The two sources are concatenated along the key-position axis; a query at position $i$ can access only positions $j \le i$ in either source. The shared softmax allocates attention between encoder context and the current decoder state. Standard output projection, residual connections, and FFN follow.

Application of Spectral Layerwise Learning Rates

We use the layerwise learning-rate method of He et al. (2026) to train AsymLoop. The method uses spectral exponents from Heavy-Tailed Self-Regularization (HT-SR) [19] to assign learning rates to individual weight matrices in the encoder and decoder under the same allocation rule.

For a weight matrix $W_m$, let $\mathcal{T}_m$ contain the largest half of its squared singular values and let $\lambda_{m,\min}$ be the smallest value in this tail. At refresh step $s$, we estimate the exponent of the power-law density $p(\lambda) \propto\lambda^{-\alpha}$ using

Following the allocation rule of He et al. (2026), larger exponents receive larger learning-rate multipliers. With $\alpha_{\min}(s)$ and $\alpha_{\max}(s)$ denoting the extrema across the attention, FFN, and Global KV projection matrices, the target learning rate is

where $\eta_{\mathrm{base}}(u)$ is the common warmup/cosine schedule at training progress $u$. This maps the exponents to multipliers in $[1,5]$. Following He et al. (2026), the input embedding and output head use the upper multiplier; normalization parameters retain the base schedule.

Following He et al. (2026), ALRS periodically updates the learning-rate multipliers during the first 20% of training, then holds them fixed while the base learning-rate schedule continues. Scheduling details are provided in Appendix § 9.§ 1.

Experiments

We evaluate AsymLoop on LLaMA-style architectures at the 60M, 135M, 350M, and 1B scales following He et al. (2026). § 4.1 defines the baselines, training budgets, and effective-depth comparisons, with full configurations and hardware details in Appendix § 8. We then report pre-training results across model sizes (§ 4.2), downstream performance (§ 4.3), the effect of recurrent depth (§ 4.4), ablation studies (§ 4.5), and transfer to an OLMo-style backbone (Appendix § 14).

Experimental Setup

Pre-training. All model families in the main experiments use the LLaMA-style backbone configurations following He et al. (2026) and are trained from scratch on FineWeb-Edu (Penedo et al. [24]). To study the scaling behavior of AsymLoop, we evaluate the 60M, 135M, 350M, and 1B configurations. Training token budgets follow the Chinchilla scaling law (Hoffmann et al. [15]), which prescribes approximately 20 training tokens per parameter, giving 1.68B, 3B, 7B, and 20B tokens at the four model sizes. Detailed data splits, model configurations, and training budgets are provided in Appendix § 8.

Baselines and depth matching. We compare AsymLoop with two vanilla baselines: a standard non-recurrent Transformer (Dense), and a homogeneous looped Transformer that repeats the entire Transformer stack with weights shared across iterations (Full Loop). We compare recurrent models at matched effective depth, defined as the total number of Transformer layers executed per forward pass. The additional parameters and computation introduced by Global KV are detailed in Appendix § 11.

Evaluation. We report validation cross-entropy and perplexity on held-out FineWeb-Edu data to assess language modeling performance. We further evaluate the pretrained models on seven commonsense reasoning benchmarks under zero-shot and five-shot settings: PIQA ([1]), Social IQa ([27]), HellaSwag ([31]), WinoGrande ([26]), ARC-Challenge (ARC-c) ([3]), ARC-Easy (ARC-e) ([3]), and OpenBookQA (OBQA) ([22]). We use the standard prompts from the LM Evaluation Harness ([8]) and report per-task accuracy and the unweighted mean across tasks. Evaluation settings are summarized in Appendix § 8.

Pre-training under the Chinchilla Scaling Law

To examine the scalability of AsymLoop, we jointly increase model size and training token budget following the Chinchilla scaling law. Figure 2(a, Figure 2b) shows that AsymLoop consistently achieves lower validation loss than Dense and Full Loop across all four scales, from 60M to 1B. Its advantage over Full Loop is maintained at matched effective depth and training budgets within each scale. These results demonstrate the effectiveness of AsymLoop across model sizes, with sustained improvements at the billion-parameter scale. These gains also extend to an OLMo-style backbone ([11]), where AsymLoop reduces validation perplexity by 14.8% relative to the depth-matched Full Loop (Appendix § 14).

AsymLoop consistently improves both language-modeling loss and zero-shot downstream accuracy over Full Loop across all evaluated scales. Notably, the downstream advantage grows monotonically with model scale, from 0.13 percentage points at 60M to 1.27 points at 1B. We examine the task-level results in § 4.3.

Downstream Evaluation

We evaluate the pretrained models on seven commonsense reasoning benchmarks to assess their downstream generalization: PIQA (Bisk et al., [1]), Social IQa (Sap et al., [27]), HellaSwag (Zellers et al., [31]), WinoGrande (Sakaguchi et al., [26]), ARC-Challenge (Clark et al., [3]), ARC-Easy (Clark et al., [3]), and OpenBookQA (Mihaylov et al., [22]). Evaluations cover all four model sizes under both zero-shot and five-shot settings, using the standard prompts provided by the LM Evaluation Harness (Gao et al., [8]). Table 1 summarizes the per-task and average accuracies.

Scale	Model	OBQA	WinoGrande	ARC-c	ARC-e	HellaSwag	SIQA	PIQA	Avg.
60M	Dense	17.80	50.83	17.41	45.33	27.39	34.54	59.03	36.05
60M	Full Loop	16.80	50.28	20.22	46.51	27.14	37.00	58.71	36.66
60M	AsymLoop	17.80	50.59	18.17	47.31	27.72	36.49	59.41	36.79
135M	Dense	18.80	48.54	21.42	52.90	29.06	37.41	61.53	38.52
135M	Full Loop	19.80	51.85	22.87	51.52	29.09	38.69	62.19	39.43
135M	AsymLoop	22.00	50.99	22.35	54.50	30.22	38.38	61.97	40.06
350M	Dense	22.20	53.35	26.62	59.39	33.82	39.15	67.30	43.12
350M	Full Loop	24.80	51.78	25.77	58.92	34.57	39.05	66.38	43.04
350M	AsymLoop	23.60	52.41	27.05	62.12	35.59	39.20	67.79	43.97
1B	Dense	28.40	59.19	33.96	69.02	41.47	41.61	71.49	49.31
1B	Full Loop	29.40	57.77	34.64	68.64	42.67	42.53	71.38	49.58
1B	AsymLoop	32.40	57.93	36.69	70.71	44.34	41.81	72.03	50.85
60M	Dense	17.60	49.57	18.43	42.93	27.09	34.54	58.11	35.47
60M	Full Loop	15.40	50.51	20.48	44.44	27.27	35.21	58.32	35.95
60M	AsymLoop	16.60	51.70	18.86	45.71	27.53	36.13	59.47	36.57
135M	Dense	18.00	51.70	22.35	50.21	28.75	36.44	61.64	38.44
135M	Full Loop	17.20	52.09	22.44	50.72	28.85	38.33	63.06	38.96
135M	AsymLoop	21.20	50.28	23.46	53.66	30.03	37.56	62.95	39.88
350M	Dense	23.00	53.28	27.65	59.93	33.41	39.00	66.76	43.29
350M	Full Loop	21.20	52.17	26.88	60.02	34.33	39.15	66.92	42.95
350M	AsymLoop	24.00	54.38	28.92	64.48	35.03	40.69	68.17	45.10
1B	Dense	24.20	57.70	36.35	69.57	41.07	42.48	71.60	48.99
1B	Full Loop	26.60	58.25	36.77	70.54	42.84	43.09	72.25	50.05
1B	AsymLoop	29.00	59.27	38.99	73.40	44.06	43.65	71.76	51.45

As shown in Table 1, AsymLoop consistently outperforms Dense and Full Loop in average accuracy across all four model sizes under both zero-shot and five-shot settings. The improvements extend across multiple benchmarks, with particularly broad gains in five-shot evaluation: AsymLoop achieves the best performance on all seven tasks at 350M and six of the seven tasks at 1B. At 350M, its average accuracy exceeds Full Loop by 2.15 percentage points. These results demonstrate that the pre-training gains of AsymLoop translate into stronger downstream commonsense reasoning across both evaluation settings. We further analyze the training dynamics of downstream performance at the 1B scale in Appendix § 13.

Furthermore, we evaluate the mathematical reasoning performance of the 1B pretrained models through supervised fine-tuning on GSM8K (Cobbe et al., [4]). AsymLoop achieves an accuracy of 11.75%, a 1.52 percentage point improvement over Full Loop. These results suggest that its downstream benefits also extend to supervised fine-tuning; detailed results and training settings are provided in Appendix § 15.

Downstream Performance under a 5× Chinchilla Budget

To investigate whether the benefits of AsymLoop persist under extended training, we increase the training budget of the 350M models from 7B to 35B tokens, corresponding to five times the Chinchilla budget. We compare AsymLoop with Dense and Full Loop under the same token budget, maintaining matched effective depth between the recurrent models. Table 2 presents their zero-shot and five-shot performance on the seven commonsense reasoning benchmarks.

Model	OBQA	WinoGrande	ARC-c	ARC-e	HellaSwag	SIQA	PIQA	Avg.
								Zero-shot
Dense	25.40	54.70	31.06	66.33	37.69	39.61	68.44	46.18
Full Loop ($K = 2$)	27.20	53.43	31.91	66.20	39.04	41.40	68.55	46.82
AsymLoop ($K = 3$, ALRS)	26.80	55.80	32.68	68.35	38.54	40.02	69.04	47.32
								Five-shot
Dense	21.20	54.54	30.72	67.00	37.21	41.20	68.77	45.81
Full Loop ($K = 2$)	23.00	56.51	31.66	66.50	38.53	41.50	68.99	46.67
AsymLoop ($K = 3$, ALRS)	25.40	56.67	32.08	68.56	38.70	41.50	68.99	47.41

As shown in Table 2, AsymLoop maintains the highest average accuracy in both zero-shot and five-shot evaluation under the extended training budget. In five-shot evaluation, it matches or surpasses both baselines on all seven tasks, improving average accuracy over Full Loop from 46.67% to 47.41%. These results demonstrate that the downstream benefits of AsymLoop persist with a fivefold increase in training tokens, supporting its effectiveness under extended pre-training.

Effect of Recurrent Depth

We investigate how recurrent depth affects AsymLoop by varying the number of decoder iterations from $K = 1$ to $K = 3$ and $K = 5$ at the 60M, 135M, and 350M scales. To compare architectures at matched effective depth, these configurations are paired with Dense, Full Loop with $K = 2$, and Full Loop with $K = 3$, respectively, yielding depths of $L$, $2L$, and $3L$. The training token budget is held fixed within each model scale. Table 3 reports validation perplexity for these comparisons.

Scale	Loop / Total Layers	$K$	Effective Depth	Architecture	Budget	PPL
60M	0 / 6	1	6	Dense	1.68B	23.72
60M	3 / 6	1	6	AsymLoop (Ours)	1.68B	21.79
60M	6 / 6	2	12	Full Loop	1.68B	23.03
60M	3 / 6	3	12	AsymLoop (Ours)	1.68B	21.45
60M	6 / 6	3	18	Full Loop	1.68B	22.77
60M	3 / 6	5	18	AsymLoop (Ours)	1.68B	21.30
135M	0 / 12	1	12	Dense	3.00B	17.46
135M	6 / 12	1	12	AsymLoop (Ours)	3.00B	16.54
135M	12 / 12	2	24	Full Loop	3.00B	17.12
135M	6 / 12	3	24	AsymLoop (Ours)	3.00B	16.14
135M	12 / 12	3	36	Full Loop	3.00B	16.81
135M	6 / 12	5	36	AsymLoop (Ours)	3.00B	16.09
350M	0 / 24	1	24	Dense	7.00B	12.55
350M	12 / 24	1	24	AsymLoop (Ours)	7.00B	12.20
350M	24 / 24	2	48	Full Loop	7.00B	12.30
350M	12 / 24	3	48	AsymLoop (Ours)	7.00B	12.00
350M	24 / 24	3	72	Full Loop	7.00B	12.15
350M	12 / 24	5	72	AsymLoop (Ours)	7.00B	11.92
1B	24 / 24	2	48	Full Loop	20.00B	8.91
1B	12 / 24	3	48	AsymLoop (Ours)	20.00B	8.86

As shown in Table 3, AsymLoop achieves lower validation perplexity than the corresponding depth-matched baseline in every comparison. Its advantage is already evident at $K = 1$ and persists as effective depth increases to $2L$ and $3L$. Increasing the number of decoder iterations further reduces perplexity across the 60M, 135M, and 350M scales. For example, at 350M, perplexity decreases from 12.20 at $K = 1$ to 11.92 at $K = 5$. These results demonstrate that AsymLoop remains effective across different depths and benefits from additional recurrent computation.

Ablation Studies and Analysis

We examine the contributions of AsymLoop’s architectural and optimization components. Further analyses of asymmetric learning-rate allocation, Global KV under recurrence, and alternative context-access designs are provided in Appendices B and C.

Component Ablations

We evaluate the contributions of AsymLoop’s attention design, normalization, and learning-rate allocation through component ablations (Table 4). For the architectural components, we individually remove Global KV, Local KV, and QKNorm under uniform learning rates, and include a parameter-matched variant without Global KV. For optimization, we compare ALRS with uniform learning rates and a fixed decoder learning-rate boost, and examine sensitivity to the base learning rate.

Model Variant	PPL $\downarrow$ ($K = 3$)	PPL $\downarrow$ ($K = 5$)
*$a$ Architectural components
AsymLoop (ALRS)	23.751	23.580
AsymLoop (uniform LR)	24.727	24.619
w/o QKNorm	24.984	24.663
w/o Global KV	25.566	25.751
w/o Local KV	25.656	25.465
Baseline with parameter matching	25.339	25.323
*$b$ ALRS and base learning rate
AsymLoop	23.751	23.580
w/o ALRS	24.727	24.619
w/ $0.5\times$ LR	24.059	23.826
w/ $2\times$ LR	24.865	24.278
w/ $4\times$ LR	26.181	25.941
w/ static decoder boost	24.785	24.708
*$c$ Full Loop ($K = 2$)	Uniform LR	ALRS
Validation PPL $\downarrow$	25.571	24.787

Architectural components. Under uniform learning rates, removing either Global KV or Local KV increases validation perplexity at both $K = 3$ and $K = 5$ (Table 4a), indicating that encoder context and evolving decoder states provide complementary information. Retaining Global KV also yields lower perplexity than reallocating its parameters to a wider FFN, suggesting that access to encoder context is more effective than additional FFN capacity in these comparisons. Removing QKNorm increases perplexity at both recurrent depths.

ALRS and base learning rate. At the default base learning rate, ALRS lowers perplexity by approximately one point relative to uniform learning rates at both recurrent depths (Table 4b). It also achieves lower perplexity than a fixed decoder learning-rate boost, indicating the benefit of adapting learning rates at the matrix level. Among the tested base learning rates, 0.001 gives the best results at both $K = 3$ and $K = 5$.

AsymLoop also retains its advantage over a widened Full Loop baseline at matched analytical per-step training FLOPs; detailed comparisons are provided in Appendix § 12.

Conclusion

We presented AsymLoop, an asymmetric looped Transformer that combines single-pass context encoding with recurrent state refinement through joint attention to Global and Local KV. We train this architecture using the spectral layerwise learning-rate method of He et al. (2026) and evaluate its effect through optimization ablations. Across four model scales from 60M to 1B, AsymLoop achieves lower validation perplexity and higher average downstream accuracy than the depth-matched Full Loop baseline. Its language-modeling gains extend to an OLMo-style backbone and sparse MoE feed-forward layers, while supervised fine-tuning on GSM8K also yields higher mathematical reasoning accuracy. Although AsymLoop requires more training FLOPs than the original Full Loop, it achieves lower validation loss than a widened Full Loop baseline under matched training FLOPs.

AI USE STATEMENT

Generative AI tools assisted with polishing the language of this manuscript. The authors reviewed the AI-assisted text and take responsibility for all content and claims in this paper.

REPRODUCIBILITY STATEMENT

The model architecture is described in § 3, and Appendix § 8 provides the training configurations, hyperparameters, and computational resources. All source datasets used in our experiments are publicly available; § 4.1 and the relevant appendices specify our FineWeb-Edu shard selection and other custom training and validation splits. For the seven downstream benchmarks, we use the standard prompts and evaluation protocols of the LM Evaluation Harness [8]; the separate GSM8K protocol is documented in Appendix § 15. The H100 pre-training runs used up to 32 NVIDIA H100 GPUs per run, while hardware for other experiments is identified in their respective protocols. We plan to publicly release the training and evaluation code and the model checkpoints used in this work. Zhou, Mingyu Xu, Mingyu Yang, Mingze Wang, Muyang Chen, Ni Shentu, Ning Wang, Niufang Ning, Panpan Huang, Peixin Cong, Peiyi Wang, Peiyuan Xin, Pengfei Ren, Pengfei Yan, Pengle Zhang, Qi Kang, Qi Tang, Qiancheng Wang, Qiang Li, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qizhou Guo, Rongxian Xu, Rui Ding, Rui Hu, Rui Tian, Rui Yu, Ruidong Zhu, Ruifan Xu, Ruihan Yang, Ruihang Xia, Ruijie Lu, Ruilin Geng, Ruipeng Hong, Ruiqi Ge, Ruisong Zhang, Ruize Sun, Ruizhe Pan, Runji Wang, Runqian Chen, Runxin Xu, Ruohong Tian, Ruomeng Shen, Ruoyu Zhang, Ryan X., S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen, Shengding Hu, Shengkai Lin, Shengwen Ran, Shengyu Liu, Shengyuan Jia, Shi Bai, Shi Feng, Shicheng Xu, Shichun Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shiyuan Feng, Shufan Gong, Shuhan Lin, Shuiping Yu, Shunfeng Zhou, Shuo Yang, Shuomeng Wang, Shuting Guo, Shuting Pan, Shuying Yu, Sinuo Cao, Siyi Lin, Sizhe Chen, Songyang Chen, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tongrui Xiong, Wangding Zeng, Wei Liu, Wei Zhang, Weibin Xu, Weihao Zeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Shao, Wenkai Yang, Wenli Zhang, Wenlu Wang, Wenlve Huang, Wenqian Yan, Wentao Zhang, Xi Gao, Xiang He, Xiang Li, Xiangli Li, Xiangwen Wang, Xiangying Zhang, Xiankui Wei, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojian Qu, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaoyao Zou, Xiaoyuan Li, Xicheng Guo, Xieting Chu, Xin Cheng, Xin Liu, Xin Xie, Xinbo Xu, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xintong Yao, Xinyang Chen, Xinyong Jiang, Xinyu Yang, Xinyu Yang, Xu Chen, Xuanyu Wang, Xubei Zhong, Xuecheng Su, Xuejie Liu, Xuheng Lin, Xujie Fan, Xuncheng Zhao, Xuwei Fu, Y. C. Yan, Y. H. Jiang, Y. T. Wu, Y. W. M., Y. Z. Wang, Yafei Gao, Yang Yang, Yang Zhang, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Meng, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yaoyang Ye, Yehang Yin, Yexinrui Wu, Yi Qian, Yi Tao, Yi Yu, Yichao Zhang, Yichen Jiang, Yicheng Wang, Yifan Ding, Yifan Shi, Yifeng Peng, Yifeng Zhai, Yijia Wu, Yiliang Xiong, Yilun Wang, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yiping Wang, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyao Yang, Yiyuan Liu, Yizai Cai, Yizhen Wei, Yizhi Wang, Yonglun Yang, Yongqi Zhuo, Yongqiang Guo, Yongtong Wu, Yu Wu, Yu Zhang, Yuan Bi, Yuan Cheng, Yuan Ou, Yuan Sun, Yuanfan Xu, Yuanhang Sun, Yuanhao Li, Yuchen Liu, Yuchen Yao, Yudong Han, Yuduan Wang, Yuhan Wu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunchuan Wang, Yunfan Xiao, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian Wang, Yuqing Chen, Yushun Zhang, Yutong Lin, Yuwei Xiao, Yuxian Gu, Yuxiang Chen, Yuxiang Huang, Yuxiang Luo, Yuxiang You, Yuxin Chen, Yuxin Xiang, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhe Guo, Yuzhen Huang, Yuzhuo Bai, Z. Y. Z., Zanlin Ni, Zehao Wang, Zehua Zhao, Zehui Ren, Zejun Zhao, Zhangli Sha, Zhanying Wang, Zhaocheng Zhang, Zhaoshuai Du, Zhe Fu, Zhean Xu, Zhenda Xie, Zheng Liu, Zhengyan Zhang, Zhenhua Dong, Zhewen Hao, Zhibang Wang, Zhibin Gou, Zhicheng Ma, Zhilao Li, Zhihong Shao, Zhihuan Huang, Zhijie Li, Zhirui Lu, Zhixian Huang, Zhixuan Chen, Zhixuan Chen, Zhixuan Pan, Zhiyu Wu, Zhizhou Ren, Zhu He, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zili Zhang, Zilin Li, Zilong Hou, Zilong Lyu, Ziqiao Wang, Ziwei Xie, Ziya Zhang, Ziyi Gao, Zizheng Pan, Zonglin Li, Zongqing Yao, Zui Chen, Zuofan Wu, Zhenchen Ling, Chengyu Hou, Chong Chen, D. Li, Di Qi, Dongjie Ji, Fang Wei, Fanyi Xia, Fei Xie, Feiyi Tan, Hailong Guo, Haiyan Zhai, Hui Zhou, Huihui Tan, Huijie Li, Jia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jiaqing Gao, Jiayi Shao, Jie Chen, Jieyu Yang, Jin Chen, Jingde Zhang, Jingzi Zhou, Jingqian Wang, Jinyang Liu, JinZhao Sun, Junhua Ling, Junmin Zheng, Kaicheng Yang, Ke Xu, Le Su, Leyi Xia, Liangfeng Ding, Lin Zhuo, Linwang Ma, Linyan Zhu, Liyu Cai, Luqi Yao, M. K. Zhang, Meng Li, Miao Lin, Miaojun Wang, Min Zhang, Mingming Li, Mingming Wang, Mingze Yin, Minmin Han, Nan Cao, Ning Wang, Ningxin Ma, Panpan Wang, Peihan Lin, Peng Sun, Peng Zhang, Qian Ying, Qiang Xiang, Qiao Wang, Qingmiao Mao, Qiwei Jiang, Rongli Jin, Ruyi Chen, Sha Tao, Shangmian Sun, Shaoqing Wu, Shichao Zhou, Si Lei, Tianyang Zhang, Tianyu Sun, Tingting Yin, W. L. Xiao, Wei An, Wei Li, Wei Wang, Weiwei Lin, Wenqing Hou, X. Lin, Xiangfei Meng, Xianzhu Huang, Xiao Peng, Xiaoqian Li, Xiaoting Zhang, Xiaowen Sun, Xiaoxiang Wang, Xiaoyu Ye, Xinrou Zhang, Xinyu Zhang, Xue Cao, Xueyin Chen, Yanan Zhou, Yanhong Xu, Yao Xia, Yao Xu, Yi Shao, Yihong Zhang, Yiling Ma, Ying Tang, Yining Lou, Yiru Chen, Yishi Piao, Yixuan Chen, Yong Xiong, Yuchen Xuan, Yuehan Yang, Yuer Xu, Yukun Zha, Yunxian Ma, Yuping Lin, Yuting Yan, Yutong Xie, Yuwen Sheng, Yuxuan Zhu, Zekai Zhang, Zhe Ju, Zhenzhen Lin, Zheren Gao, Zheyang Sun, Zhigang Yan, Zhongyu Wu, Zi Wang, Zihua Qu, Ziling Yan, and Ziyi Wan. Deepseek-v4.1-flash: Pushing the limits of kv cache compression, 2026. URL https://arxiv.org/abs/2609.19969.

Architecture, Training, and Evaluation Details

Table 5 summarizes the architectures and pre-training hyperparameters for the four model configurations. We denote these configurations by 60M, 135M, 350M, and 1B, with the actual parameter counts reported separately for each architecture.

All main pre-training experiments use cleaned and deduplicated FineWeb-Edu shards 00000–00039 for training, with shard 00233 reserved for validation. We use the t5-base tokenizer with a vocabulary size of 32,128 and a sequence length of 1024. Input embeddings and the output head are untied.

Training budgets are matched across architectures within each model configuration. The 60M models are trained for 3,200 optimizer steps, corresponding to approximately 1.68B input token positions including padding. The 135M, 350M, and 1B models are trained on 3B, 7B, and 20B tokens, respectively.

Configuration	60M	135M	350M	1B
Dense parameters	91.8M	134.3M	368.2M	1.34B
AsymLoop parameters	95.4M	141.4M	393.4M	1.44B
Layers $L$ ($E + D$)	6 (3+3)	12 (6+6)	24 (12+12)	24 (12+12)
$d_{\mathrm{model}}$	768	768	1024	2048
$d_{\mathrm{ff}}$	2048	2048	2736	5461
Heads	6	12	16	32
Training budget	1.68B	3.00B	7.00B	20.00B
Peak LR $\eta_0$	$1.0 \times10^{-3}$	$1.0 \times10^{-3}$	$1.0 \times10^{-3}$	$5.0 \times10^{-4}$
Weight decay	0.1	0.1	0.1	0.1
Effective depth ($L/2L/3L$)	6 / 12 / 18	12 / 24 / 36	24 / 48 / 72	24 / 48 / 72

We use AdamW with $(\beta_1, \beta_2) = (0.9, 0.95)$, $\epsilon= 10^{-8}$, and weight decay 0.1. All main pre-training experiments use a global batch size of 512, a global gradient clipping threshold of 1.0 in L2 norm, and random seed 6198. The base learning rate follows linear warmup over the first 10% of training, followed by cosine decay to 10% of its peak. Schedule progress is measured in optimizer steps at 60M and training tokens at larger scales. ALRS scheduling details are provided in Appendix § 9.§ 1.

Large-scale pre-training uses up to 32 NVIDIA H100 GPUs, each with 80 GB of memory.

We report validation cross-entropy (CE) and perplexity (PPL) to assess pre-training performance. For downstream commonsense reasoning, we report zero-shot and five-shot accuracy using the standard prompts provided by the LM Evaluation Harness (Gao et al., [8]).

ALRS Allocation and Weight-Spectrum Diagnostics

ALRS Scheduling Details

Following the allocation and scheduling procedure of He et al. (2026), ALRS assigns matrix-wise learning-rate multipliers in [1, 5]. Allocations are updated at the first optimizer step and every 100 steps thereafter during the first 20% of training, with a 50-step linear transition to each new learning-rate target. After this initial phase, the multipliers remain fixed while the base learning rate continues its cosine decay.

Observed Encoder–Decoder Learning-Rate Differences

We analyze how ALRS allocates learning rates between the encoder and decoder in the 135M and 350M models. Figure 3 shows their mean learning-rate trajectories and decoder-to-encoder ratios separately for Attention and FFN modules.

Figure 3 shows a consistent FFN asymmetry across model scales: after the allocation updates end, the decoder’s mean FFN learning rate is approximately 1.5 times the encoder’s. For Attention, the encoder–decoder difference is more pronounced at 350M than at 135M. These differences arise from applying the same spectral allocation rule to all matrices; the approximately 1.5-fold FFN ratio is an observed outcome rather than a prescribed decoder multiplier.

Weight-Spectrum Analysis

To examine the weight structure learned under ALRS, we compare normalized stable rank between uniform learning rates and ALRS in the 135M and 350M AsymLoop models ($K = 3$). Statistics are averaged equally over the local Q/K/V/O matrices for Attention and the gate/up/down matrices for FFN, separately for the encoder and decoder.

Figure 4 shows that ALRS generally produces lower normalized stable rank, with consistent reductions in Attention and decoder FFN modules at both model scales. This pattern reflects greater relative concentration of spectral energy in the leading singular direction, indicating that differentiated learning-rate allocation is accompanied by changes in the learned weight structure.

Complete Ablation Results and Evaluation Details

Global KV under Recurrence

We compare AsymLoop with and without Global KV at $K = 1, 3, 5$ to examine how its contribution changes with recurrent depth. Both variants retain QKNorm and use uniform learning rates.

Global KV reduces validation cross-entropy at every tested depth, with its gain increasing from 0.022 at $K = 1$ to 0.045 at $K = 5$ (Table 6). This trend supports a greater benefit from encoder context as recurrence deepens.

Model Variant	CE $\downarrow$ ($K = 1$)	CE $\downarrow$ ($K = 3$)	CE $\downarrow$ ($K = 5$)
AsymLoop	3.238735	3.207915	3.203515
w/o Global KV	3.260346	3.241257	3.248483
Global KV gain $G(K)$	0.021611	0.033343	0.044968
Gain beyond $K = 1$	0.000000	0.011732	0.023357

Alternative Context-Access Designs

We compare Global KV with two alternatives: removing this access path or replacing it with additive injection of the encoder output before each decoder iteration. The additive variant computes $z_0 = 0$ and $z_t = D(z_{t-1} + E)$, where $E$ is the encoder output. Full Loop provides an additional baseline. All models use QKNorm and uniform learning rates.

As shown in Table 7, AsymLoop achieves the lowest validation perplexity. Additive injection improves over removing Global KV, while retaining Global KV provides a further gain. These results highlight the benefit of accessing encoder context through attention during recurrent computation.

Model Variant	PPL $\downarrow$
AsymLoop	24.727
w/o Global KV	25.566
w/ additive injection	25.116
Full Loop	25.053

Additional Validation Results

Table 8 reports validation cross-entropy and perplexity for Dense, Full Loop, and AsymLoop without QKNorm under a common evaluation protocol. The AsymLoop results also provide the reference values for the QKNorm ablation in Table 4.

Reference	$K$	Effective depth	CE	PPL
Dense, Uniform	1	6	3.271820	26.359274
Dense, layerwise LR	1	6	3.209192	24.759083
Full Loop, Uniform	2	12	3.241455	25.570904
Full Loop, Uniform	3	18	3.229346	25.263120
AsymLoop w/o QKNorm, Uniform	3	12	3.218239	24.984094
AsymLoop w/o QKNorm, Uniform	5	18	3.205305	24.663015

Computational Cost and Downstream Performance

We analyze AsymLoop’s computational cost and its relationship with downstream performance. Compute-matched comparisons with a widened Full Loop baseline are reported in Appendix § 12.

Cost of Global KV and Joint Attention

Let $N$, $d$, $m$, and $V$ denote the sequence length, hidden dimension, SwiGLU intermediate dimension, and vocabulary size, respectively. We count two FLOPs per multiply–accumulate and use full attention matrices. A standard Transformer block has forward cost

At matched effective depth $J = E + KD = RL$, the Full Loop cost is $F_{\mathrm{Full}} = F_{\mathrm{blk}} + F_{\mathrm{head}}$, where $F_{\mathrm{blk}} = JC(m)$ and $F_{\mathrm{head}} = 2NdV$. Here, $E = D = L/2$ and $R = (K + 1)/2$.

AsymLoop adds Global KV projections computed once per decoder layer and additional attention products at each decoder iteration:

Table 9 summarizes the resulting forward costs. Across the evaluated model sizes, these components add 9.38%–15.23% to the Full Loop baseline.

Model size	Forward GFLOPs/token		Additional cost (%)
	Full Loop	AsymLoop	Global KV proj.	Global attn.	Total
60M	0.256967	0.292356	2.75	11.02	13.77
135M	0.464585	0.535364	3.05	12.19	15.23
350M	1.476657	1.677984	3.41	10.23	13.63
1B	5.365891	5.869208	3.75	5.63	9.38

Compute–Performance Trade-offs Across Recurrent Depths

We investigate the trade-off between training compute and downstream performance across model scales and recurrent depths. Figure 5 compares average zero-shot accuracy against analytical training FLOPs, pairing AsymLoop with Dense or Full Loop at matched effective depths of $L$, $2L$, and $3L$.

Across all three depth settings, AsymLoop consistently achieves higher accuracy than its depth-matched baseline with a modest increase in training compute. Comparing across depths further reveals a favorable compute–performance trade-off: at 60M, 135M, and 350M, AsymLoop with $K = 1$ achieves higher accuracy than Full Loop with either $K = 2$ or $K = 3$, while requiring fewer training FLOPs. These results show that AsymLoop can offer both better downstream performance and lower compute than deeper Full Loop configurations. Complementary comparisons with a widened Full Loop baseline at matched analytical per-step training FLOPs are provided in Appendix § 12.

Performance at Matched Per-Step Training FLOPs

To investigate whether AsymLoop’s performance gains arise solely from additional training FLOPs, we compare it with a widened Full Loop baseline. We increase the baseline’s SwiGLU intermediate dimension while preserving its recurrent architecture, matching AsymLoop’s analytical per-step training FLOPs under a full-block recomputation assumption.

Experimental Setup

We evaluate the original Full Loop, widened Full Loop, and AsymLoop at the 60M and 135M scales. At both scales, the widened baseline increases the FFN dimension from 2048 to 2656. Full Loop uses two iterations, while AsymLoop uses three decoder iterations with ALRS. Within each scale, all models share the training data, token budget, batch size, and distributed training configuration. At 60M, we report validation cross-entropy from one run per configuration; at 135M, we report the mean and sample standard deviation over three training seeds per configuration.

Results

Table 10 shows that widening Full Loop improves validation cross-entropy, closing approximately 40% of its gap to AsymLoop at both model scales. Nevertheless, AsymLoop achieves the lowest cross-entropy in both comparisons. Its advantage also persists across all three random seeds in the 135M experiments. These results suggest that additional training FLOPs alone do not explain AsymLoop’s performance gains.

Model size	Model	FFN	Params	s/step	Peak mem.	Validation CE
60M	Full Loop $K=2$	2048	91.83M	0.232	20.2 GiB	3.0434
60M	+ FLOPs matching	2656	100.23M	0.230	21.6 GiB	3.0139
60M	AsymLoop $K=3$	2048	95.37M	0.216	21.7 GiB	2.9684
135M	Full Loop $K=2$	2048	134.30M	0.286	30.9 GiB	2.81550 ± 0.00221
135M	+ FLOPs matching	2656	151.11M	0.296	33.6 GiB	2.78983 ± 0.00197
135M	AsymLoop $K=3$	2048	141.38M	0.367	33.8 GiB	2.74947 ± 0.00299

At 135M, each configuration is evaluated over three training seeds. The widened Full Loop and AsymLoop achieve validation cross-entropies of 2.78983 ± 0.00197 and 2.74947 ± 0.00299, respectively (mean ± sample standard deviation). The mean difference is 0.04036, and all three AsymLoop runs achieve lower cross-entropy than all three widened Full Loop runs. These observations support the consistency of the improvement within the evaluated training configuration.

Training FLOPs Verification

We measure per-step training matrix FLOPs on H100 GPUs, including forward, backward, and activation recomputation. Table 11 shows that the widened Full Loop uses 1.23%–2.29% fewer FLOPs than AsymLoop, indicating closely matched training compute.

Model size	AsymLoop	Full Loop (wide)	FFN Width	Difference vs. AsymLoop
60M	1134.878	1108.907	2656	−2.29%
135M	2118.157	2092.198	2656	−1.23%

Discussion

AsymLoop achieves lower validation cross-entropy than the widened Full Loop at both model scales while using fewer parameters. Table 10 also shows comparable peak memory usage and per-step training times of the same order of magnitude. These results highlight AsymLoop’s performance advantage at closely matched training FLOPs.

Downstream Training Dynamics at 1B

We track the zero-shot performance of 1B Full Loop and AsymLoop throughout pre-training. Figure 6 presents the accuracy trajectories on seven downstream benchmarks and their average.

Both models improve with training, while AsymLoop develops a clearer advantage in average accuracy during the later stages. This trend is particularly evident on OpenBookQA, ARC-Easy, and HellaSwag, where AsymLoop maintains higher accuracy across successive evaluations toward the end of training.

Transfer to an OLMo-Style Backbone

To assess generalization across backbone architectures, we evaluate AsymLoop on an OLMo-style backbone [11]. We compare Dense, Full Loop, and AsymLoop trained from scratch.

Experimental Setup

Architecture. All models use eight physical Transformer blocks with hidden dimension 512, eight attention heads, and SwiGLU intermediate dimension 2048, together with RMSNorm and RoPE. Full Loop repeats the backbone twice, while AsymLoop combines a four-layer encoder with a four-layer decoder repeated three times. The two looped models therefore have the same effective depth of 16.

Data and evaluation. We train all models on approximately 1.68B tokens from FineWeb-Edu-dedup using the SmolLM tokenizer, with sequence length 1024 and global batch size 512. Validation cross-entropy and perplexity are evaluated on a shared held-out set.

Optimization. We use AdamW with a cosine learning-rate schedule and 10% warmup. AsymLoop applies ALRS multipliers in $[1, 3]$, with allocation updates restricted to the first 20% of training.

Validation Results

Results. Table 12 shows that AsymLoop achieves the lowest validation cross-entropy and perplexity among all three models. Compared with the depth-matched Full Loop, AsymLoop reduces perplexity by 14.8%. These results demonstrate that its language-modeling benefits extend to an OLMo-style backbone, supporting its applicability across backbone architectures.

Table 12: Validation performance on an OLMo-style backbone. Cross-entropy and perplexity ($\downarrow$). Depth denotes effective depth. Darker/lighter blue marks the best/second-best results.

Model	Depth	CE $\downarrow$	PPL $\downarrow$
Dense	8	3.508664	33.403615
Full Loop ($K = 2$)	16	3.501389	33.161466
AsymLoop ($K = 3$)	16	3.341073	28.249416

Supervised Fine-Tuning on GSM8K

To evaluate mathematical reasoning after supervised fine-tuning, we fine-tune the 1B pretrained models on GSM8K ([4]). We compare Dense, Full Loop, and AsymLoop after three epochs of full-parameter adaptation.

Results after Three SFT Epochs

Table 13 shows that AsymLoop achieves the highest GSM8K accuracy among the three models. Under matched fine-tuning schedules, AsymLoop reaches 11.75% accuracy, improving over Full Loop by 1.52 percentage points. These results suggest that AsymLoop’s pre-training benefits also extend to mathematical reasoning after supervised fine-tuning.

Table 13: GSM8K accuracy of 1B models after three epochs of full-parameter supervised fine-tuning. The best result is highlighted in blue.

Model	GSM8K Accuracy (%) $\uparrow$	Relative gain over Dense (%) $\uparrow$
Dense	8.94	–
Full Loop ($K = 2$)	10.24	+14.54
AsymLoop ($K = 3$)	11.75	+31.43

Training and Evaluation Protocols

Models and optimization. We fine-tune the 1B pretrained models using AdamW with a global batch size of 32, a peak learning rate of $10^{-4}$, and cosine decay with 10% warmup. We report results after three epochs using a single random seed.

Data and supervision. We use 6,973 GSM8K training examples for supervised fine-tuning and reserve 500 for development, with a context length of 1024. Training supervises the worked solutions and final answers, excluding question and padding tokens from the loss.

Evaluation. We evaluate all 1,319 test questions using greedy decoding without demonstrations, allowing up to 512 generated tokens. Accuracy is computed by comparing the normalized final numerical answer with the reference.

Sparse Mixture-of-Experts Models

We investigate whether AsymLoop’s benefits extend to sparse mixture-of-experts models [23]. Using the 60M and 135M backbone configurations, we compare MoE variants of Dense, Full Loop, and AsymLoop, where Dense denotes the non-recurrent architecture.

Experimental Setup

We replace each feed-forward layer with eight SwiGLU experts and top-2 routing, without token dropping. Each expert has intermediate dimension 1024, preserving the active expert-matrix parameter count of the original dense FFN. Full Loop repeats the backbone twice, while AsymLoop divides it equally into an encoder and a decoder and repeats the decoder three times, matching the effective depth of Full Loop.

Experts and routers are shared across recurrences, with routing decisions recomputed at each iteration. Training combines cross-entropy with load-balancing and router z-loss terms weighted by 0.01 and 0.001, respectively. Validation cross-entropy excludes these auxiliary losses.

We train all models on FineWeb-Edu-dedup using AdamW with a cosine learning-rate schedule, sequence length 1024, and global batch size 512. The 60M and 135M configurations use 1.6B and 3B training tokens, respectively. We report validation cross-entropy and perplexity on a shared held-out set.

Validation Results

Table 14 shows that AsymLoop achieves the lowest validation cross-entropy and perplexity at both model scales. At 135M, it reduces perplexity by 4.5% relative to the depth-matched Full Loop. These results show that AsymLoop remains effective with sparse experts, extending its applicability beyond dense feed-forward layers.

Model	Effective depth	CE $\downarrow$	PPL $\downarrow$
60M configuration: 1.6B training tokens
Dense + MoE	6	2.9856	19.798
Full Loop K2 + MoE	12	2.9363	18.845
AsymLoop K3 + MoE	12	2.8986	18.149
135M configuration: 3B training tokens
Dense + MoE	12	2.7489	15.625
Full Loop K2 + MoE	24	2.7154	15.111
AsymLoop K3 + MoE	24	2.6690	14.426