betaFeedback

WorksComputer Science

ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Abstract

Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Research: each proposal is recorded with its experiments and reasoning, reviewed by the community, and referenced by later work. To evaluate ideas before costly large-scale training, ArchSpace leverages scaling laws to predict model loss at larger scales from smaller experiments, a practice widely adopted in industry. We provide initial references that make the platform usable: Qwen3.5 and OLMo3 baselines, fitted scaling laws, and a reusable training recipe. Ideas validated through this review can in turn become new baselines, and ArchSpace thereby aims to broaden the range of ideas that shape the next generation of models.

https://www.archspace.live/ · GitHub · Training dataset

Qwen3.5 baseline collection · OLMo3 baseline collection

Introduction

Architecture forms the foundation of a large language model (LLM), shaping its performance, scaling behavior, and efficiency during training and inference. When teams such as DeepSeek, Qwen, and Kimi release new open models, they also introduce architectural changes and innovations. Validating these designs involves costly training experiments. However, the experiments and trade-offs behind most models remain inaccessible, leaving the community with little account of which ideas succeeded, which were abandoned, and why. An open and complete record of validation would therefore be as valuable to subsequent research as a new idea itself.

The value of such records also depends on whether researchers discover the documented ideas and build on them. However, much of the attention to new architectures is concentrated on research from industry: our survey2 shows that academic work accounts for about a third of the papers but receives less than 10% of the citations (Figure 1). Academic ideas need greater visibility and proper validation so that the community can discuss these ideas and draw inspiration from them, and even explore new architectural designs on these bases.

Figure 1. Academic architecture research receives a disproportionately small citation share, motivating ArchSpace’s goal of broader visibility and recognition.

We therefore introduce ArchSpace, a public platform for Recording, Reviewing, and Referencing Architecture Research. Each proposal is recorded with its experiments and reasoning, reviewed by the community to check whether the results support the conclusion, and referenced by later work. To evaluate the architectural changes described in each proposal, the validation framework uses scaling laws [9, 13] to compare models across multiple compute budgets, a systematic and standard practice widely used in industry [5, 6, 11, 16, 17, 20].

As initial references, we provide Qwen3.5 and OLMo3 baselines. Validated architecture innovations can in turn become new baselines for later work. ArchSpace aims to build a community around these contributions, where the research process is openly documented and a wider range of ideas can gain visibility and recognition.

An Open Community for Architecture Research

ArchSpace aims to build an open community for Recording, Reviewing, and Referencing Architecture Research. Each proposal is recorded with its experiments and reasoning, reviewed by the community to check whether the results support the conclusion, and referenced by later work.

Contributing to ArchSpace begins with an Architecture Proposal issue, which states the proposed change, its rationale, the baseline, and the validation plan (Figure 2). Community discussion helps contributors refine the architectural idea and plan a fair comparison with the baseline. Contributors then implement and evaluate the proposed architecture under the validation framework (§ 3) and submit their work for public review through a pull request (PR) linked to the issue. The PR links to the implementation, configurations, checkpoints, training logs, evaluation results, and reproduction instructions. It also records failed trials and search choices so readers can trace how the conclusion was reached. Community reviewers then use this record to check whether the implementation matches the proposal, the comparison is reproducible, and the experiments support the conclusion. The verified PR is then merged into ArchSpace for future reference, and the proposed architecture can serve as a new baseline.

The ArchSpace contribution workflow

Figure 2. The ArchSpace contribution workflow. Phase I compares the proposed architecture with its baseline across the scaling ladder; the three settings differ in their reuse of model size, token count, learning rate (LR), and batch size (BS). Phase II tests whether gains persist through longer training and evaluates downstream capabilities. Reviewed records support future reference. Here, L^\hat{L} denotes predicted loss; CC is training FLOPs, MM training FLOPs per token, DD the training token count, η\eta LR, and BB BS.

Our broader aim is for reviewed PRs to have the same standing as traditional research papers and become the primary way to communicate and cite contributions in ArchSpace. A PR combines a complete account of the contribution with direct links to the resources needed to reproduce and extend the work. Follow-up PRs can link to earlier contributions, so researchers can trace how architectural ideas develop across time. By bringing these contributions together, ArchSpace aims to provide a shared foundation for subsequent architecture research.

The ArchSpace Validation Framework

ArchSpace standardizes architecture comparisons through a two-phase validation framework (Figure 2). Phase I tests whether the proposed architecture improves the compute–loss trend across the five budgets of the scaling ladder. Phase II trains 1B models on 150B+10B tokens to test whether the gain persists through longer training and to evaluate downstream capabilities directly.

To isolate the effect of the architectural change, the proposed architecture and its baseline use the same training data, tokenizer, and validation data in both phases. The ArchSpace training dataset provides official OLMo 3 mixtures pre-tokenized with the OLMo 3 tokenizer [19].3 In our training dataset, the data for pretraining in both phases come from the Dolma 3 Pool 150B sample, and the data for Phase II long-context mid-training come from the Dolma 3 Longmino Mix (50B).

Phase I: Scaling across compute budgets

Architectural gains can vary with the compute budget, so Phase I validates the proposed architecture against its baseline across multiple budgets using fitted scaling laws that estimate each architecture’s lowest achievable loss [9, 13]. The proposed architecture is considered an improvement if its compute-optimal loss curve lies on the Pareto frontier. The validation loss LL used for fitting is computed as the equal-weight average over the eleven domains of the original OLMo 3 validation set [19] (Appendix § 10). Throughout the report, loss refers to this validation loss.

Let MM denote a model’s scale in training FLOPs per token at its training context length,4 DD the number of training tokens, and C=MDC = MD the total training FLOPs. At a fixed budget CC, increasing MM leaves fewer tokens for training, with D=C/MD = C/M. To fit a scaling law for each architecture, ArchSpace sweeps the five compute budgets CjC_j of its scaling ladder: 1×10181 \times10^{18}, 3×10183 \times10^{18}, 1×10191 \times10^{19}, 3×10193 \times10^{19}, and 1×10201 \times10^{20} FLOPs. The largest-budget runs dominate the training cost, while runs at smaller budgets add modest compute. Training models across the scaling ladder therefore enables systematic comparisons between architectures at little additional computational cost.

We provide three settings that differ in the scope of hyperparameter search. Broader searches require more training runs and therefore a larger total FLOPs budget. When the proposed architecture is similar to its baseline, the optimal learning rate (LR) and batch size (BS) can also be similar for the two architectures. The baseline’s search results can then be reused to reduce the cost of validation. The choice of setting therefore depends on the extent of the architectural change and the available compute.

At each budget of the scaling ladder, Setting 1 reuses the baseline’s compute-optimal settings, including model size, training token count, LR, and BS. Setting 2 performs an IsoFLOP sweep over model size and training token count while reusing the baseline’s LR and BS. Setting 3 additionally tunes LR and BS during the IsoFLOP sweep (Figure 2).

Setting 1: baseline-optimal settings. Setting 1 trains one model per budget. The baseline’s scaling law gives the target model scale M∗(Cj)M^*(C_j) (§ 4.2). A realizable model of the proposed architecture rarely has exactly this scale, so the model with training FLOPs per token closest to the target should be selected. The selected model is trained either for the token count given by the baseline’s compute-optimal curve, allowing the budget to shift slightly, or at the fixed budget CjC_j, accepting a small departure from the optimal model size and training token count (Appendix § 10.§ 4). LR and BS should be taken from the best measured baseline recipe at the same budget and closest model scale (Appendix § 9). The five measured losses are then used to fit a power law in compute, following Kaplan et al. [13], with the irreducible loss term used in the Chinchilla law [9]:

L^(C)=L∞+αCC−γC.(1)\hat{L}(C) = L_{\infty} + \alpha_C C^{-\gamma_C}. \tag*{(1)}

The fitting parameters are the irreducible loss L∞L_{\infty}, amplitude αC\alpha_C, and exponent γC\gamma_C.

Setting 2: IsoFLOP sweep over model size and training data. An IsoFLOP sweep compares models of different sizes trained with the same total compute budget, adjusting the training token count for each size [9]. This comparison identifies the combination of model size and training tokens that gives the lowest loss at that budget. Setting 2 searches four to six model sizes at each budget of the scaling ladder. The selected sizes should bracket the expected optimum for the proposed architecture. LR and BS should be reused from the best measured recipe of the nearest baseline model–budget group, as in Setting 1. The losses measured across these IsoFLOP sweeps are used to fit L^(M,D)\hat{L}(M,D) using the coupled form in Equation (3). This law describes how loss varies with model scale and training token count.

Setting 3: LR- and BS-tuned IsoFLOP. At each model–budget configuration in the size sweep, the LR–BS search starts from the baseline’s predicted optima (Appendix § 10.§ 4) or, at an imposed BS, from the optimal LR at that BS (Equation (13)). The grid can be adjusted to the proposed architecture and the available compute; we use a 4×44 \times4 grid as an example. The losses from the sweeps over model size, LR, and BS are used to fit the full scaling law L^(M,D,η,B)\hat{L}(M,D,\eta,B) in Equation (2), where η\eta and BB denote LR and BS.

Phase II: Training 1B models on 150B+10B tokens

Phase II tests whether gains measured on the scaling ladder persist through longer training and context extension, and evaluates downstream capabilities. The framework adopts 1B parameters as a reasonable model size for this evaluation, with pretraining on about 150B tokens at 8K context followed by long-context mid-training on 10B tokens at 64K.5

For pretraining, the LR and BS predicted by the Phase I scaling laws provide a good starting point for hyperparameter search. In practice, the predicted optima differ from the best measured settings, so we conduct a further search over LR and BS. We then use the selected pretrained checkpoint for long-context mid-training and tune LR and BS with the pretraining settings as a reference (§ 5). As in Phase I, researchers can directly reuse the tuned settings, or run their own search when additional compute is available.

Runs are compared at the final checkpoint of each training stage, after its LR schedule completes, because intermediate rankings can reverse during LR decay (Appendix § 12.§ 3). Downstream capabilities are evaluated with OLMES [7]6 using the evaluation suites introduced in OLMo 3 [19]. After pretraining, the Base Main suite tests general capabilities, involving mathematical reasoning and question answering. After long-context mid-training, evaluation focuses on the Base Long-Context suite, with RULER [10] as the primary development signal and HELMET [29] for complementary evaluation. The Base Main suite is also rerun to check whether the capabilities developed during pretraining are retained. The proposed architecture should use the same evaluation configuration as the baseline, so differences in their scores reflect only architectural differences.

Phase I: Baseline Ladders and Scaling Laws

Hybrid architectures have become a mainstream design choice for LLMs. We therefore select Qwen3.5 and OLMo3 as baselines. Qwen3.5 interleaves three Gated DeltaNet layers [28] with one full-attention layer [24], whereas OLMo3 interleaves three sliding-window attention layers with one full-attention layer [19].

For each architecture, we provide a baseline ladder: a series of models trained at the five compute budgets of the scaling ladder, with LR and BS swept within each model–budget group. We fit a scaling law to each ladder’s measured losses to obtain the compute-optimal model size and training token count, together with predicted optima. We then test the loss predictions on extrapolation runs at larger budgets to validate the effectiveness of the fitted scaling laws.

Experimental setup

The models in both ladders are pretrained on the Dolma 3 Pool 150B sample using the shared OLMo 3 tokenizer (§ 3) and 8K-token sequences. Training combines Muon [12, 15] with AdamW (Appendix § 9.§ 1) under a warmup–stable–decay (WSD) schedule [11], with a square-root cooldown over the final 20% of updates [8].

Following Phase I Setting 3 (§ 3.1), we conduct IsoFLOP experiments by sweeping an outer grid of model sizes and compute budgets (Figure 3). Within each model–budget group, we further sweep an inner grid of LR and BS to find the best hyperparameters (Figure 4 and Figure 9). Appendices § 8 and § 9 specify the model sizes and LR–BS grids for each ladder.

Figure 3. Coupled and additive base surfaces for each baseline ladder. Circles mark the lowest measured loss per model–budget group; curves show the surfaces L∗(M,D)L^{*}(M,D) fitted to these losses. Table 1 gives the coupled-surface coefficients, Table 20 compares fit errors, and Table 16 lists the measured minima.

Figure 4. LR–BS dependence of loss in the Qwen3.5 baseline ladder. Colors show the measured loss above each group minimum, and contours show the fitted LR–BS term ΔL\Delta L at intervals of 0.02. Crosses mark predicted optima and stars the best measured configurations; empty panels are outside the sweep. Table 2 gives the fitted laws.

Figure 9. LR–BS dependence of loss in the OLMo3 baseline ladder. Colors show the measured loss above each group minimum, and contours show the fitted LR–BS term ΔL\Delta L at intervals of 0.02. Crosses mark predicted optima and stars the best measured configurations; empty panels are outside the sweep. Table 2 gives the fitted laws.

Analysis method

Fitting the scaling law in two stages. We fit a scaling law in two stages to describe how loss varies with model scale MM (training FLOPs per token), training token count DD, LR η\eta, and BS BB:

L^(M,D,η,B)=L∗(M,D)+ΔL(M,D,η,B),ΔL≥0.(2)\widehat{L}(M,D,\eta,B) = L^*(M,D) + \Delta L(M,D,\eta,B), \qquad\Delta L \ge0. \tag*{(2)}

Here the base surface L∗(M,D)L^*(M,D) gives the minimum loss achievable under the given model scale and training token count. The LR–BS term ΔL\Delta L gives the additional loss when LR or BS departs from its optimal value. At the predicted optimal LR and BS, ΔL=0\Delta L = 0 and L^=L∗\widehat{L} = L^*.

In the first stage, we fit L∗(M,D)L^*(M,D) to the lowest measured loss from each LR–BS inner grid, following OpenEuroLLM [1]. In the second stage, we fit ΔL\Delta L using the losses of all swept runs across both outer and inner grids. We hold the fitted L∗L^* fixed during this stage to preserve the IsoFLOP curves fitted to the group minima (Appendix § 11.§ 2).

Both stages minimize the sum of Huber losses on the differences between predicted and measured log loss. The Huber loss grows quadratically for small differences and linearly once their magnitude exceeds a threshold. Appendix § 10.§ 2 gives the exact definition, threshold, and fitting objectives.

Modeling the minimum loss across model and data scales. For the base surface, we extend the Chinchilla law [9]7 with a coupling exponent γMD\gamma_{MD}, following Skaling [27] and OpenEuroLLM [1]:

L∗(M,D)=L∞+(αMM−γM+αDD−γD)γMD,(3)L^*(M,D) = L_{\infty} + \left(\alpha_M M^{-\gamma_M} + \alpha_D D^{-\gamma_D}\right)^{\gamma_{MD}}, \tag*{(3)}

Here L∞L_{\infty} is the irreducible loss. The amplitudes αM\alpha_M and αD\alpha_D set the magnitudes of the model and data terms. The exponents γM\gamma_M and γD\gamma_D determine how these terms decrease as MM and DD increase, while γMD\gamma_{MD} controls their interaction. The amplitudes and exponents are all positive.

When γMD=1\gamma_{MD} = 1, the two power-law terms add directly, recovering the Chinchilla form with MM as model scale. We call this the additive form and fit it as an ablation to the standard coupled form (Appendix § 11.§ 1). Table 1 gives the fitted coefficients of the coupled base surface for each baseline ladder.

BaselineL∞L_{\infty}αM\alpha_MγM\gamma_MαD\alpha_DγD\gamma_DγMD\gamma_{MD}
Qwen3.51.7226.77×1046.77 \times10^{4}0.5201.90×1041.90 \times10^{4}0.4530.505
OLMo31.4883.96×1053.96 \times10^{5}0.5765.99×1045.99 \times10^{4}0.4750.402

Table 1. Coefficients of the base surface L∗L^* (Equation (3)), fitted to the lowest measured loss of each model–budget group. MM is in FLOPs per token and DD in tokens.

Modeling loss away from the optimal LR and BS. We first model the optimal LR η∗\eta^* and BS B∗B^* as power laws in MM and DD, a form used in StepLaw [14], OpenEuroLLM [1], and Microsoft Law [3]:

η∗=αηMγη,MDγη,D,B∗=αBMγB,MDγB,D,h=αhMγh,MDγh,D.(4)\eta^* = \alpha_{\eta} M^{\gamma_{\eta,M}} D^{\gamma_{\eta,D}}, \qquad B^* = \alpha_B M^{\gamma_{B,M}} D^{\gamma_{B,D}}, \qquad h = \alpha_h M^{\gamma_{h,M}} D^{\gamma_{h,D}}. \tag*{(4)}

We also model the curvature scale h>0h > 0 in the same way, which will be used in the LR–BS term ΔL\Delta L (Equation (5)). The fitting parameters are α□>0\alpha_{\square} > 0 and γ□,△\gamma_{\square,\triangle}, with □∈{η,B,h}\square\in\{\eta, B, h\} and △∈{M,D}\triangle\in\{M,D\}.

To guide the form of ΔL\Delta L, we use the optimization proxy for Muon-type methods [25, 26]. Starting from this proxy, we derive two hyperbolic cosine (cosh) terms that describe its joint dependence on LR and BS in log space (Appendix § 10.§ 1). We therefore model the loss increase away from η∗\eta^* and B∗B^* with two cosh terms. With η~=ln⁡(η/η∗)\widetilde{\eta} = \ln(\eta/\eta^*) and B~=ln⁡(B/B∗)\widetilde{B} = \ln(B/B^*) denoting the deviations of LR and BS from their optima in log space, the LR–BS term is

ΔL(M,D,η,B)=2h[cosh⁡(η~−γη,BB~)−1]+2hρ(cosh⁡B~−1)(5)\Delta L(M,D,\eta,B) = 2h\left[\cosh\left(\widetilde{\eta} - \gamma_{\eta,B}\widetilde{B}\right) - 1\right] + 2h\rho\left(\cosh\widetilde{B} - 1\right) \tag*{(5)}

This decomposition lets us obtain the optimal LR for any chosen BS. At that BS, the second term in Equation (5) gives the minimum loss increase achievable by optimizing LR, while the first term gives the additional increase from using a different LR. The optimal LR therefore makes the first term zero. This minimum occurs at η=η∗(B/B∗)γη,B\eta= \eta^*(B/B^*)^{\gamma_{\eta,B}}, where the curvature in log LR is 2h2h. Thus, γη,B\gamma_{\eta,B} determines how the optimal LR changes with BS, while hh controls how sharply loss rises around that LR.

With LR optimized at each BS, the second term describes how the minimum loss varies with BS. Its curvature in log BS at B∗B^* is 2hρ2h\rho, so ρ>0\rho> 0 is the ratio of this curvature to the curvature 2h2h in log LR. We fit one ρ\rho per baseline ladder. Appendix § 10.§ 1 derives the optimal LR at a fixed BS and the curvature expressions.

Table 2 summarizes the power laws and shape parameters of the LR–BS term for each baseline ladder.

Loss prediction and hyperparameter selection

We evaluate loss predictions across the scaling ladder and at larger compute budgets, then examine how the predicted optima can guide LR–BS searches.

Fit quality across the scaling ladder. We compare the base-surface predictions with each model–budget group’s lowest measured loss (Figure 3) and the full-law predictions with the losses of all swept runs. We report mean absolute percentage error (MAPE) at the group minima (Best-point) and over all swept runs (All-point), as defined in Appendix § 10.§ 3.

We first compare the coupled surface with the additive ablation, which fixes γMD=1\gamma_{MD} = 1 and is fitted to the same group minima. For the fitted range 0<γMD<10 < \gamma_{MD} < 1, the coupled form makes the loss reductions from model scale and training data reinforce each other; the additive form keeps them independent (Appendix § 11.§ 1). Both forms have fitted IsoFLOP minima within the swept range of model sizes, but the additive form deviates most from the measured losses at the smallest and largest budgets (Figure 3, bottom row). Its best-point MAPE is 0.48% for Qwen3.5 and 0.62% for OLMo3, compared with 0.13% and 0.11% for the coupled surface (Table 20). The coupled surface therefore fits the measured IsoFLOP minima more closely across the scaling ladder.

BaselineFormL∞L_{\infty}αM\alpha_{M}γM\gamma_{M}αD\alpha_{D}γD\gamma_{D}γMD\gamma_{MD}MAPE
Qwen3.5Coupled1.7226.77×1046.77 \times10^{4}0.5201.90×1041.90 \times10^{4}0.4530.5050.13%
Qwen3.5Additive2.3003.61×1043.61 \times10^{4}0.5371.28×1031.28 \times10^{3}0.3641.0000.48%
OLMo3Coupled1.4883.96×1053.96 \times10^{5}0.5765.99×1045.99 \times10^{4}0.4750.4020.11%
OLMo3Additive2.3488.52×1048.52 \times10^{4}0.5801.37×1031.37 \times10^{3}0.3651.0000.62%

Table 20. Coupled and additive base surfaces fitted to the same group minima. MAPE is evaluated on those minima; MM is in FLOPs per token and DD in tokens. Figure 3 compares their IsoFLOP curves with measurements.

We then evaluate the full scaling laws, which fit the measured losses of all swept runs with a MAPE of 0.25% for Qwen3.5 and 0.27% for OLMo3 (Table 3). Residuals show no trend across budgets, so the laws do not systematically over- or underpredict loss at any budget (Appendix § 10.§ 3).

BaselineBest-point MAPEAll-point MAPEP95
Qwen3.50.22%0.25%0.80%
OLMo30.17%0.27%0.69%

Table 3. Loss-prediction errors of the full fixed-slope scaling laws on the swept runs. Mean absolute percentage error (MAPE) is reported at group minima (Best-point) and over all runs (All-point); P95 is the 95th percentile of absolute percentage error over all runs.

We also compare the fitted LR–BS term with the measured loss differences in every model–budget group of the Qwen3.5 ladder (Figure 4); Figure 9 in Appendix § 10.§ 3 shows the corresponding OLMo3 results. The measured loss differences follow the fitted contours, including the tilt that couples the optimal LR to the BS.

Loss prediction at larger compute budgets. We next test loss predictions at larger budgets by freezing the fitted coefficients and training models using the ladder training setup and validation data (Figure 5 and Table 4). These runs use the largest swept models and new 32-layer models, trained near the compute-optimal combination of model size and training token count at up to about ten times the largest swept budget.

BaselineModelFLOPsTPPLLL^\hat{L}Error
Qwen3.51.79B2.00×10202.00 \times10^{20}9.12.62902.6337+0.18%
Qwen3.52.64B†4.43×10204.43 \times10^{20}9.42.54182.5498+0.31%
OLMo31.91B2.98×10202.98 \times10^{20}11.82.61312.6053−0.30%
OLMo33.36B†9.80×10209.80 \times10^{20}12.92.48882.4766−0.49%
Mean absolute percentage error0.32%

Table 4. Extrapolation runs beyond the largest swept budget, 1×10201 \times10^{20} FLOPs. Predictions use the frozen coefficients and each run’s LR and BS. Relative error is L^/L−1\hat{L}/L - 1. Daggers mark model sizes beyond the swept range. Table 18 gives the recipes, and Figure 5 compares the measured losses with the compute-optimal curve.

Figure 5. Measured extrapolation losses compared with the compute-optimal loss min⁡MD=CL∗(M,D)\min_{M D=C} L^*(M,D). Labels give the relative prediction errors L^/L−1\hat{L}/L - 1 of the full scaling law at each run’s LR and BS, with the coefficients held fixed (Table 4).

The best measured loss at each swept budget lies within 0.5% of the compute-optimal curve of L∗L^{*}, and the extrapolation losses follow this curve beyond the swept range (Figure 5). At the LR and BS used in training, the frozen scaling laws also predict the extrapolation losses within 0.5%, with a MAPE of 0.32% (Table 4), comparable to the MAPE on the swept runs. Appendix § 10.§ 5 gives the training recipes and compares predictions from L∗L^{*} alone with those from the full scaling law.

Using predicted optima to guide LR–BS searches. Finally, we compare the predicted optimal LR and BS with the best measured configuration in each model–budget group (Table 22). The predicted optimal LR, however, differs from the best measured LR with a MAPE of 35.1% for Qwen3.5 and 23.1% for OLMo3. The Setting 3 search should therefore be initialized with these predicted optima, and the final recipe should be selected using the measured losses. Appendices § 11.§ 3 and § 11.§ 4 examine how alternative LR–BS forms and methods for estimating the optima affect prediction accuracy.

Form of ΔL\Delta LParametersQwen3.5: MAPEOLMo3: MAPE
Best-pointAll-pointη∗\eta^*B∗B^*Best-pointAll-pointη∗\eta^*B∗B^*
Proxy shape150.29%0.46%40.57%73.85%0.26%0.53%28.90%75.58%
Log-quadratic170.21%0.28%34.49%23.19%0.19%0.31%22.47%30.41%
Cosh, fitted slope170.21%0.25%50.04%24.33%0.17%0.27%23.19%26.59%
Cosh, γη,B=1/2\gamma_{\eta,B} = 1/2160.22%0.25%35.10%21.42%0.17%0.27%23.11%26.68%

Table 22. Each alternative LR–BS form is fitted using the same base surface and losses from all swept runs. The Best-point and All-point columns report loss MAPE on group minima and all runs; the η∗\eta^* and B∗B^* columns report MAPE against the best measured configuration of each group. Parameter counts include the base surface. Bold marks the adopted form.

We also use the predicted optima to initialize the LR–BS search for the longer pretraining in Phase II (Figure 6). To compare training length across model sizes, we use tokens per parameter (TPP), D/ND/N, where NN is the parameter count excluding the input embedding. The extrapolation runs follow the fitted compute-optimal path to larger models and budgets while remaining in the TPP range of the ladders. In contrast, Phase II

Figure 6. Model scale and token counts in the baseline ladders, extrapolation runs (§ 4.3), and Phase II pretraining (§ 5). Solid curves show the fitted compute-optimal paths D∗(M)D^{*}(M); dotted lines indicate approximately 10 and 100 tokens per parameter (TPP).

pretraining uses about 130–160 TPP, compared with 10–14 TPP in the longest ladder runs at the Phase II model scales. This combination of model scale and token exposure lies outside the swept range, so we search over LR and BS directly in Phase II (§ 5).

Comparison with scaling laws from prior work

We next compare the coupled and additive base surfaces within each study that reports both (Table 5). In our baseline ladders and in prior work, the coupled form matches the measured losses more closely than the additive form. The fitted coupling exponents are γMD=0.51\gamma_{MD} = 0.51 for Qwen3.5 and 0.40 for OLMo3, compared with 0.31–0.41 for Skaling and 0.41 for OpenEuroLLM.

Within-study comparisons of the coupled form (Equation 3) and the additive Chinchilla form [9]. The table denotes each study’s outer exponent by γND\gamma_{ND}, corresponding to γMD\gamma_{MD} in Equation 3. SK is the Skaling study’s own grid of training runs; RMSE denotes root mean square error. Prediction errors are evaluated on runs excluded from fitting in prior studies and at the swept group minima in this report.

StudyDataγND\gamma_{ND}CheckCoupledAdditive
Skaling [27]Farseer grid0.41Larger NN, DD; MAPE2.31%2.46%
Skaling [27]SK grid0.31Larger NN, DD; MAPE0.70%5.17%
OpenEuroLLM [1]OpenEuroLLM ladder0.41Largest model; RMSE0.00560.0174
This reportQwen3.50.51Group minima; MAPE0.13%0.48%
This reportOLMo30.40Group minima; MAPE0.11%0.62%

Table 5.

We also compare laws for optimal LR and BS, evaluating each law from prior work on the path for which it is defined (Table 6 and Figure 7). At a fixed model scale, optimal BS follows a power law in data size with fitted exponents from 0.46 to 0.57 (Table 6), close to the D1/2D^{1/2} prediction derived from the Muon-type proxy [25] in Appendix § 10.§ 1. The fitted power laws for B∗B^{*} also have a negative exponent in MM for each baseline ladder (Table 2); the optimal BS therefore depends on both MM and DD (Table 19). By contrast, the optimal-LR exponents vary more across studies, whose optimizers, schedules, and parameterizations differ. We therefore fit a separate LR law to each baseline ladder.

ParameterQwen3.5OLMo3
η∗\eta^*421M−0.408D−0.152421M^{-0.408}D^{-0.152}269M−0.389D−0.154269M^{-0.389}D^{-0.154}
B∗B^*36.8M−0.047D+0.48536.8M^{-0.047}D^{+0.485}47.3M−0.036D+0.45747.3M^{-0.036}D^{+0.457}
hh17.9M−0.075D−0.24217.9M^{-0.075}D^{-0.242}14.0M−0.097D−0.20314.0M^{-0.097}D^{-0.203}
(γη,B,ρ)(\gamma_{\eta,B}, \rho)(0.5,1.16)(0.5, 1.16)(0.5,1.15)(0.5, 1.15)

Table 2. Power laws and shape parameters of the LR–BS term (Equations (4) and (5)). The exponent γη,B\gamma_{\eta,B} is fixed at 1/21/2. MM is in FLOPs per token; DD and B∗B^* are in tokens.

Figure 7. Baseline scaling laws for optimal LR and BS compared with laws from prior work. Panels (a, b) compare the laws at 20 TPP with OpenEuroLLM [1], StepLaw [14], and Porian Law [23]; panels (c, d) compare the laws with those of DeepSeek LLM [5] at compute-optimal model sizes and training token counts. Baseline curves span the swept sizes and budgets; curves for prior work use the reported coefficients under each study’s parameter-count convention. Table 6 gives the exponents.

Table 6: Scaling exponents of optimal LR and BS, measured as log–log slopes. The row blocks show exponents for a fixed model with varying data, for models of different sizes at 20 TPP, and for compute-optimal combinations of model size and training token count, respectively. When NN varies, we evaluate the baseline laws at the swept model sizes and the laws from prior work at 0.1B–2B parameters under each study’s parameter-count convention; when CC varies, slopes are measured over 1×10181 \times10^{18}–1×10201 \times10^{20} FLOPs. The proxy row is theoretical; the other rows describe fitted laws. Figure 7 plots the laws as NN or CC varies.

PathLawη∗\eta^* exponentB∗B^* exponent
Fixed model, varying DDShulgin et al. [25] (proxy)−0.250+0.500
OpenEuroLLM [1]+0.086+0.460
StepLaw [14]+0.307+0.571
This report, Qwen3.5−0.152+0.485
This report, OLMo3−0.154+0.457
20 TPP, varying NNOpenEuroLLM [1]−0.331+0.605
StepLaw [14]−0.406+0.571
Porian Law [23]−0.360+0.703
This report, Qwen3.5−0.549+0.440
This report, OLMo3−0.531+0.422
Compute-optimal, varying CCDeepSeek LLM [5]−0.125+0.327
This report, Qwen3.5−0.271+0.238
This report, OLMo3−0.260+0.234

Table 6. Scaling exponents of optimal LR and BS, measured as log–log slopes. The row blocks show exponents for a fixed model with varying data, for models of different sizes at 20 TPP, and for compute-optimal combinations of model size and training token count, respectively. When NN varies, we evaluate the baseline laws at the swept model sizes and the laws from prior work at 0.1B–2B parameters under each study’s parameter-count convention; when CC varies, slopes are measured over 1×10181 \times10^{18}–1×10201 \times10^{20} FLOPs. The proxy row is theoretical; the other rows describe fitted laws. Figure 7 plots the laws as NN or CC varies.

ComponentAlternativesSelected
Base surfaceAdditive form of Chinchilla [9] (γMD=1\gamma_{MD}=1); coupled form of Skaling and OpenEuroLLM [1, 27]Coupled form, Equation (3) (Appendix § 11.§ 1)
Fitting objectiveSquared log-loss residuals; Huber loss on log-loss residuals, as in Skaling and OpenEuroLLMHuber loss with δ=0.001\delta=0.001 when fitting L∗L^{*} and ΔL\Delta L, Equations (18) and (19); best-point MAPE of L∗L^{*} is 0.13% (Qwen3.5) and 0.11% (OLMo3), compared with 0.13% and 0.12% for the squared loss
Fitting orderL∗L^{*} and ΔL\Delta L fitted together; L∗L^{*} fitted first and held fixed while ΔL\Delta L is fittedTwo-stage fit (Appendix § 11.§ 2)
Estimating the optimaQuadratic surfaces fitted to each group’s losses, then power laws fitted to the minima of those surfaces, following OpenEuroLLM; one LR–BS term fitted to losses from all runsOne shared LR–BS term (Appendix § 11.§ 4)
Form of ΔL\Delta LProxy shape of Shulgin et al. [25] with fitted power laws for its optima and amplitude; log-quadratic form based on the proxy’s second-order expansion, with fitted slope and curvature ratio; cosh form with a fitted slope; cosh form with the proxy’s slope γη,B=1/2\gamma_{\eta,B}=1/2Cosh form with γη,B=1/2\gamma_{\eta,B}=1/2, Equation (5) (Appendix § 11.§ 3)
Optimal BSB∗B^{*} depending on DD only, as in StepLaw [14]; B∗B^{*} depending on MM and DD, as in OpenEuroLLMDepends on MM and DD for all baseline ladders

Table 19. Design choices of the scaling law for the baseline ladders.

Implications for the validation framework

The settings of § 3.1 reuse the baseline ladders in two ways. Settings 1 and 2 use the best measured recipes of the nearest model–budget groups (Appendix § 9), and Setting 3 starts its LR–BS search from the predicted optima (Appendix § 10.§ 4). At a BS imposed by hardware, the search starts from the predicted optimal LR at that BS, given by Equation 13 in Appendix § 10.§ 1.

Phase II pretraining uses much higher token counts at the 1B scale (§ 4.3). Its direct LR–BS search also starts from the predicted optima (§ 5).

Phase II: 1B Baselines and Training Recipe

To provide reusable baselines for the longer training and downstream evaluation in Phase II (§ 3.2), we train 1B Qwen3.5 and OLMo3 models on 150B tokens at 8K context, followed by 10B tokens of long-context mid-training at 64K. Both baselines use the same data, token counts, and schedules within each training stage. We tune LR and BS for Qwen3.5, then reuse the selected baseline recipe for OLMo3 without tuning.

Validation loss and downstream task scores favor different recipes. We therefore select recipes from the Pareto frontier using validation loss and the Base Main score for pretraining, and long-context scores and the Base Main score for mid-training. § 5.3 reports the baselines’ downstream evaluation results; Table 8 reports the complete pretraining sweep; Appendix § 12 gives the configurations and detailed analyses, including the complete mid-training sweep.

η\etaBBLLL^\hat{L}ErrorScoreMathMC-STEMMC-NonSTEMGenQA
8e-42M2.50042.5159+0.62%27.32.626.028.851.8
1e-32M2.50882.5171+0.33%26.92.624.229.052.0
2e-32M2.49582.5260+1.21%27.62.925.629.552.3
8e-44M2.51202.5138+0.07%27.12.526.728.650.5
1e-34M2.50552.5140+0.34%27.12.626.428.051.6
2e-34M2.49332.5190+1.03%27.42.725.728.852.3
3e-34M2.48782.5256+1.52%27.93.227.028.852.6
4e-34M2.49412.5328+1.55%28.02.927.429.452.4
5e-34M2.49692.5400+1.73%27.32.925.428.352.5
3e-38M2.49852.5255+1.08%27.42.826.328.651.7
5e-38M2.49202.5352+1.73%27.52.826.728.052.4

Table 8. Qwen3.5 1B pretraining sweep on about 150B tokens: measured loss and Base Main scores, with predictions L^\hat{L} from the frozen scaling law. Error is L^/L−1\hat{L}/L-1. MC denotes multiple choice; GenQA denotes generative question answering.

Pretraining beyond the compute-optimal ladder

The Qwen3.5 pretraining run reaches about 130 TPP, compared with about 10 TPP in the longest ladder run at this model scale. We tune LR and BS for this longer training, starting from the predicted optima of the frozen scaling law (Table 7). At BS 4M, the predicted optimal LR is about 8e-4. Table 8 gives the complete sweep, with loss predictions and Base Main scores.

BaselineModelTPPη∗\eta^*B∗B^*η∗(⋅∣B=4M)\eta^*(\cdot\mid B=4\mathrm{M})LLL^\hat{L}Error
Qwen3.51.21B129.87.49e-43.29M8.26e-42.48782.5256+1.52%
OLMo31.00B156.97.5e-42.61M9.28e-42.53742.5516+0.56%

Table 7. Predicted optima and losses for Phase II pretraining from the frozen scaling laws. η∗\eta^* and B∗B^* are the predicted optimal LR and BS; the conditional LR is evaluated at BS 4M (Equation (13)). LL and L^\hat{L} are the measured and predicted losses for runs using the baseline recipe in Table 23; relative error is L^/L−1\hat{L}/L-1.

The final loss of every pretraining run is within 1% of the lowest measured loss, yet the lowest loss and the highest Base Main score occur at different recipes. The selected recipe uses LR 3e-3 and BS 4M (Table 8); its LR is about 3.6 times the predicted optimum at the same BS. This choice relies on results from the final checkpoints of completed schedules: lower-LR runs have lower loss during the stable phase, but the run with the selected higher-LR recipe has the lowest loss after the final decay (Appendix § 12.§ 3).

The pretraining runs also test predictions outside the fitting regime: they extend beyond their baseline ladders in TPP and use a different warmup schedule (Appendix § 12.§ 2). The frozen Qwen3.5 scaling law overpredicts every pretraining loss, with a MAPE of about 1% (Appendix § 12.§ 2). For OLMo3, the scaling law predicts the baseline loss within 0.6% (Table 7).

Long-context mid-training

We then extend the pretrained model’s context window to 64K through long-context mid-training. The Qwen3.5 sweep tests peak LR values from the final pretraining LR to the pretraining peak, using the pretraining BS of 4M and a doubled BS of 8M. Each run restores the model and optimizer state of the selected pretrained checkpoint and trains on the same 10B tokens.

Using RULER as the primary development signal (§ 3.2), we select LR 1e-3 and BS 4M for mid-training (Figure 8 and Table 25). At 64K, the length targeted by Phase II, this recipe yields the highest RULER score: 57.0, compared with 51.9 when mid-training uses the final pretraining LR (Table 9). The sweep also shows that the LR yielding the highest measured score is higher at BS 8M than at BS 4M for both RULER and HELMET at 64K (Appendix § 12.§ 4). The baseline recipe combines the selected settings for pretraining and mid-training: peak LR 3e-3 with BS 4M for pretraining, followed by 1e-3 with BS 4M for long-context mid-training (Table 23).

Figure 8. Qwen3.5 long-context mid-training sweep: RULER (a) and HELMET (b) scores at each evaluation length, and the Base Main score against the peak LR (c). Base Main input limits are 8K for the pretrained checkpoint and 64K after mid-training. Table 25 and Table 26 give the full results.

CheckpointRULER ↑\uparrowHELMET ↑\uparrow
Qwen3.5
Pretrained checkpoint17.523.3
Mid-trained at the final pretraining LR (3e-4)51.937.8
Mid-trained at the selected LR (1e-3)57.039.1
Highest HELMET (1.5e-3)56.040.0
OLMo3
Pretrained checkpoint2.77.4
Mid-trained with the baseline recipe (1e-3)36.129.6

Table 9. 64K RULER and HELMET scores before and after long-context mid-training at BS 4M. The Qwen3.5 rows compare the selected LR (1e-3), the final pretraining LR (3e-4), and the LR with the highest HELMET score (1.5e-3). OLMo3 reuses the baseline recipe without tuning. Table 25 and Table 28 give the full sweep and the OLMo3 scores after each stage, respectively.

StageContextTokensη\etaBBWarmupFinal η\eta
Pretraining8K157.4B3e-34M2,000 updates3e-4
Long-context mid-training64K10.7B1e-34M200 updates0

Table 23. Phase II baseline recipe, selected for Qwen3.5 and reused for OLMo3 without tuning (Section 5).

η\etaBBScoreMathMC-STEMMC-NonSTEMGenQARULERHELMET
Pretrained checkpoint27.93.227.028.852.617.523.3
3e-44M44.515.549.856.456.251.937.8
5e-44M45.517.350.657.956.352.638.4
1e-34M46.318.651.359.356.057.039.1
1.5e-34M46.319.850.459.255.856.040.0
3e-34M45.520.749.358.054.253.737.5
1e-38M45.917.751.458.356.453.837.2
1.5e-38M46.419.650.759.555.754.738.3
3e-38M45.720.150.057.954.656.440.0

Table 25. Qwen3.5 long-context mid-training sweep: Base Main scores and 64K RULER and HELMET scores for every run.

The Qwen3.5 search across both training stages costs about 1.6×10221.6 \times10^{22} FLOPs. By comparison, reusing the selected recipe for a single run of both stages costs about 1.47×10211.47 \times10^{21} FLOPs for Qwen3.5 and 1.24×10211.24 \times10^{21} FLOPs for OLMo3, each less than a tenth of the search cost.

Baseline capability results

We next present downstream evaluation results for the pretrained and mid-trained checkpoints of both baselines on the Base Main and Base Long-Context suites of OLMo 3 [19]. Qwen3.5 uses the selected recipe, and OLMo3 reuses it without tuning. OLMo3 uses about 16% less compute across the two training stages and a different inference implementation. The analysis therefore focuses on changes across training stages within each baseline.

The Base Main score gives equal weight to Math, multiple-choice STEM (MC-STEM), multiple-choice non-STEM (MC-NonSTEM), and generative QA (GenQA); code tasks are excluded. Within each family, task and subtask scores are averaged with equal weights at each level according to the OLMES hierarchy. Base Main evaluation uses an input limit of 8K after pretraining and 64K after mid-training.

After mid-training, the Base Main score rises from 27.9 to 46.3 for Qwen3.5 and from 26.2 to 40.5 for OLMo3 (Table 10). All four family averages increase. Within GenQA, however, HellaSwag, Jeopardy, and LAMBADA scores decrease for each baseline.

TaskQwen3.5 PretrainedQwen3.5 Mid-trainedOLMo3 PretrainedOLMo3 Mid-trained
Base Main score27.946.326.240.5
Math3.218.62.116.4
GSM8K5.332.03.129.1
GSM Symbolic1.712.31.19.9
MATH2.511.42.110.2
MC-STEM27.051.325.243.8
ARC26.567.224.654.7
MedMCQA31.337.230.934.1
MedQA23.329.921.425.2
MMLU STEM25.742.322.836.2
SciQ28.179.826.368.7
MC-NonSTEM28.859.327.749.2
CoQA (MC)26.677.527.261.2
CSQA21.248.419.539.8
DROP (MC)25.242.625.733.5
Jeopardy (MC)24.660.926.247.3
MMLU Humanities25.751.124.242.4
MMLU Other28.352.125.146.3
MMLU Social Sciences27.354.722.746.8
Natural Questions (MC)25.754.624.444.1
PIQA50.362.550.158.5
Social IQa36.063.933.660.4
SQuAD (MC)25.684.325.961.4
GenQA52.656.049.852.4
Basic Skills71.677.969.075.1
CoQA55.959.354.355.2
DROP25.941.023.635.6
HellaSwag62.860.458.957.0
Jeopardy32.730.127.624.4
LAMBADA60.357.257.954.8
Natural Questions12.915.113.013.7
SQuAD77.688.272.183.9
WinoGrande73.574.771.872.3

Table 10. Base Main task and family scores for both baselines before and after long-context mid-training. Evaluation uses input limits of 8K for the pretrained checkpoints and 64K after mid-training.

For the Base Long-Context suite, we also report RULER and HELMET scores at each evaluation length, using the same tasks, examples, and input limits across checkpoints (Table 11). The pretrained scores are reference measurements rather than selection criteria. RULER averages the scores from its 13 tasks with equal weights. Our HELMET [29] score gives equal weight to five categories: synthetic recall, retrieval-augmented generation, passage re-ranking, many-shot in-context learning, and long-document QA. Each category score averages its task scores equally; tasks scored by a model judge are excluded.

BaselineCheckpointRULER 4KRULER 8KRULER 16KRULER 32KRULER 64KHELMET 8KHELMET 16KHELMET 32KHELMET 64K
Qwen3.5Pretrained65.158.434.023.817.535.924.122.923.3
Qwen3.5Mid-trained81.274.471.265.357.046.342.840.939.1
OLMo3Pretrained69.451.97.94.12.734.68.19.17.4
OLMo3Mid-trained75.363.252.443.336.142.137.134.729.6

Table 11. Base Long-Context suite results [19]: RULER and HELMET before and after long-context mid-training, at each evaluation length. Table 26 and Table 27 give the full sweep and the 64K category scores, respectively.

At 64K, Qwen3.5’s RULER score rises from 17.5 to 57.0 and its HELMET score from 23.3 to 39.1; OLMo3’s scores rise from 2.7 to 36.1 and from 7.4 to 29.6. Both baselines achieve 64K long-context capability with the same two-stage recipe, tuned for Qwen3.5 and reused without tuning for OLMo3. Researchers can reuse the checkpoints, baseline recipe, and Base Main and Base Long-Context results in subsequent Phase II comparisons.

Related Work

Open research resources. Shared baselines and documented evaluations support reproducible model comparisons. OLMo 3 [19] releases training data, recipes, and intermediate checkpoints, while OLMES [7] standardizes evaluation choices that affect comparisons between models. Using OLMo 3 data and OLMES evaluation, ArchSpace connects reusable architecture baselines to a continuing public contribution process: proposals, implementations, experimental records, reasoning, and reviews remain linked for later citation and reuse (§ 2).

Scaling laws for architecture comparison. Kaplan et al. [13] established power-law relationships between language-model loss, model size, data, and compute. Chinchilla [9] used IsoFLOP profiles and an additive loss surface to determine compute-optimal model sizes and training token counts. DeepSeek LLM [5] represented model scale by non-embedding FLOPs per token, while Porian et al. [23] showed how output-layer compute, warmup, and optimizer tuning explain discrepancies in compute-optimal scaling. These studies provide the basis for Phase I’s IsoFLOP comparisons with consistent accounting of training FLOPs (§ 3.1).

The form of the loss surface determines how predictions vary across model sizes and training token counts. Skaling [27] couples the model and data terms through an outer exponent, reducing systematic residuals of the additive Chinchilla form. OpenEuroLLM [1] fitted this coupled surface to the lowest measured loss of each model–budget group. We adopt the coupled form and this fitting target for the base surface L∗(M,D)L^*(M,D), using training FLOPs per token MM as model scale, and add an LR–BS term to describe loss away from the predicted optima (§ 4.2).

Hyperparameter scaling and loss modeling. Hyperparameter scaling laws range from relationships along a prescribed training path to functions of model size and token count. DeepSeek LLM [5] fits optimal LR and BS along the compute-optimal path, whereas the Porian Law [23] describes a path at about 20 TPP. Microsoft Law [3] models optimal LR as a product of powers of model size and token count at a fixed BS, estimating the optima from quadratic fits in log LR. StepLaw [14] uses this product form for LR and a token-count-only power law for BS. OpenEuroLLM [1] first estimates optima from a quadratic surface in the original LR and BS coordinates within each model–budget group, then fits product power laws for both LR and BS. We fit power laws for the optima through the losses of all swept runs and compare prior laws along the paths on which they are defined (§ 4.2 and § 4.4).

Optimization theory provides a complementary basis for modeling the effect of LR and BS on loss. Shulgin et al. [25] derive a proxy for Muon-type optimization, and Su [26] analyzes generalized power-law proxies by minimizing first over LR and then over BS. Starting from the fixed-momentum Muon-type proxy, we derive its cosh representation in log space. This representation and the proxy’s square-root scaling of optimal LR with BS motivate our fixed-slope LR–BS term, which we combine with fitted power laws for the optima and curvature (§ 4.2; Appendix § 10.§ 1).

Architecture evaluation after context extension. OlmPool [2] directly studies how architectural choices affect long-context extension, finding that short-context metrics poorly predict the resulting long-context capabilities. It trains 7B models on about 140B pretraining tokens followed by 10B long-context tokens. Phase II adopts the same sequence of pretraining and long-context mid-training, using 1B models trained on 150B+10B tokens to evaluate architectural gains beyond the compute–loss comparisons of Phase I (§ 3.2). RULER [10] tests retrieval, multi-hop tracing, aggregation, and question answering with synthetic tasks; HELMET [29] covers a broader range of downstream applications. Phase II uses these benchmarks in the Base Long-Context suite and evaluates general capabilities with the Base Main suite (§ 5.3).

Conclusion

ArchSpace records, reviews, and references architecture experiments under a standardized validation framework with shared data and evaluation. In Phase I, loss is compared across the compute budgets of the scaling ladder. Phase II then evaluates downstream capabilities in 1B models after pretraining on 150B tokens and after long-context mid-training on 10B tokens. We provide Qwen3.5 and OLMo3 baseline ladders, fitted scaling laws, measured recipes, and Phase II baselines as initial references for this workflow.

The scaling laws combine a coupled base surface with a fixed-slope cosh LR–BS term and fit the measured losses from both baseline ladders with a MAPE of 0.25%–0.27%. The coupled base surface fits the group minima more accurately than its additive ablation, and the fitted coupling exponents and data exponents of optimal BS are close to those in prior work. The frozen laws predict extrapolation losses at up to ten times the largest swept budget with a MAPE of 0.32%. These laws also give compute-optimal model sizes and training token counts. Locating optimal LR, however, is harder than predicting loss, so we use the predicted optima to initialize direct recipe searches.

In Phase II, Qwen3.5’s selected pretraining LR is about 3.6 times the predicted optimum at the same BS. Loss predictions nevertheless remain close to the measurements for both baselines. Long-context mid-training then raises the Base Main scores and establishes 64K long-context capability for both baselines. The recipe tuned for Qwen3.5 also yields large gains for OLMo3 when reused without tuning.

With these baselines, ArchSpace offers the community an ongoing cycle of architecture research. Contributors introduce an idea through an Architecture Proposal, evaluate it under the validation framework, and submit the resulting record for public review. A Verified contribution can then become a new baseline for the next proposal. The reviewed records let later studies reproduce comparisons and reuse implementations. This cycle addresses the gaps identified at the start of this report: incomplete public records of the experiments and reasoning behind successful and unsuccessful architectural decisions, and research attention concentrated around industrial releases. ArchSpace aims to broaden the range of ideas that receive public review, citation, and recognition, making today’s baselines the starting points for tomorrow’s architecture research. Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Gaggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspergren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molbyog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satyadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Simong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang,

Architecture Configurations

Table 12 lists the Qwen3.5 and OLMo3 model shapes, parameter counts, and per-token training FLOPs. The baseline models use dense SwiGLU feed-forward blocks and untied input embeddings and output heads; all baseline series use the initialization described in Appendix § 9.§ 2.

BaselineModelLayersdddffd_{\mathrm{ff}}NNNtotalN_{\mathrm{total}}MM
Qwen3.593M8512204892,907,872144,288,096676,930,560
202M127682560202,411,608279,481,9441,483,310,592
407M1610243584407,475,072510,235,5202,922,792,960
736M2012804608735,791,320864,241,8805,161,536,000
1.21B24153656321,212,956,2561,367,096,9288,353,115,136
1.79B28179261441,794,704,4081,974,535,19212,231,937,536
2.64B†32204871682,642,291,2002,847,812,09617,765,535,744
OLMo385M8512204884,951,552136,331,776635,489,280
176M127682560176,198,400253,268,7361,340,195,328
346M1610243584346,096,640448,857,0882,579,699,712
614M2012804608613,520,640741,971,2004,467,248,640
1.00B24153656321,003,636,2241,157,776,8967,153,837,056
1.91B28204871681,908,639,7442,114,160,64013,212,758,016
3.36B†32256092163,361,016,3203,617,917,44022,681,697,280

Table 12. Model shapes, parameter counts, and per-token training FLOPs for the Qwen3.5 and OLMo3 baselines. dd and dffd_{\mathrm{ff}} are model and feed-forward widths; NN excludes the input embedding and NtotalN_{\mathrm{total}} includes it. MM is training FLOPs per token and includes the cost of attention at an 8K-token context length. Daggers mark models used only in the extrapolation runs of § 4.3.

Qwen3.5 repeats three Gated DeltaNet layers followed by one gated full-attention layer. Each full-attention head has dimension 256. The query/key–value head counts are 4/1, 6/1, 8/1, 10/2, 12/2, and 14/2 for the six swept sizes in ascending order, and 16/2 for the 2.64B extrapolation model. In Gated DeltaNet layers, the key/value head counts are 4/8, 6/12, 8/16, 10/20, 12/24, 14/28, and 16/32 in the same model order. The key and value heads each have dimension 128, and the convolution kernel has size 4. Qwen3.5 uses rotary position embeddings (RoPE) with base 1×1071 \times10^{7} and rotary fraction 0.25, and RMS normalization (RMSNorm) with ϵ=1×10−6\epsilon= 1 \times10^{-6}.

OLMo3 repeats three sliding-window attention layers with a 4,096-token window followed by one full-attention layer. The numbers of query, key, and value heads are equal. Across the six swept sizes in ascending order, the head counts are 8, 8, 16, 16, 24, and 32, with corresponding head dimensions of 64, 96, 64, 80, 64, and 64. The 3.36B extrapolation model has 32 heads of dimension 80. OLMo3 uses RoPE with base 500,000, query–key normalization, and reordered RMSNorm with ϵ=1×10−6\epsilon= 1 \times10^{-6}.

Training Recipe and Best Measured Recipes

Optimizer and parameter groups

AdamW updates the input embeddings, the output head, and the scalar, vector, and convolution parameters. Muon optimizes all other two-dimensional matrix parameters, including projection and feed-forward matrices (Table 13). AdamW uses betas (0.9, 0.95) and gradient-norm clipping at 1.0; clipping is disabled for the Muon parameters.

ParametersOptimizerWeight decay
Input embedding matrixAdamW0
Output head matrixAdamW0.1
Normalization weightsAdamW0
GDN decay and time-step parametersAdamW0
GDN depthwise convolution weightsAdamW0.1
All other 2D matrix parametersMuon0.1

Table 13. Optimizer assignments and weight decay for Qwen3.5 and OLMo3. Gated DeltaNet (GDN) rows apply only to Qwen3.5.

Muon approximately orthogonalizes gradient momentum, preserving its singular-vector directions while bringing the singular values toward one [12]. We use momentum 0.9 and approximate the orthogonalization with five Newton–Schulz iterations using a fifth-degree polynomial, avoiding an explicit singular-value decomposition. Following the Muon update rule [15], the optimization part of the matrix update is

ΔWt=−ηt0.2max⁡(dout,din) Ot,(6)\Delta W_t = -\eta_t 0.2\sqrt{\max(d_{\mathrm{out}},d_{\mathrm{in}})}\,O_t, \tag*{(6)}

where OtO_t is the approximately orthogonalized momentum, doutd_{\mathrm{out}} and dind_{\mathrm{in}} are the dimensions of the matrix being orthogonalized, and ηt\eta_t is the scheduled LR. The root mean square of OtO_t varies with the matrix dimensions.

The square-root factor compensates for this variation, and the factor 0.2 scales the update to a magnitude near that of AdamW. These factors let the two optimizers share ηt\eta_t; the reported LR specifies the peak of this shared schedule.

Initialization

All baseline series use a scaled truncated-normal initialization adapted from a Megatron-based training recipe. The base initialization scale depends on model width as in Nguyen and Salazar [18]; residual output projections are additionally scaled with depth. For model width dd and block count nlayersn_{\mathrm{layers}}, the scales are

σbase=25d,σout=σbasenlayers.(7)\sigma_{\mathrm{base}} = \sqrt{\frac{2}{5d}}, \qquad\sigma_{\mathrm{out}} = \frac{\sigma_{\mathrm{base}}}{\sqrt{n_{\mathrm{layers}}}}. \tag*{(7)}

Embeddings, input projections, convolution weights, and the output head use a zero-mean normal distribution with scale σbase\sigma_{\mathrm{base}}; residual output projections in the Gated DeltaNet and attention layers and in the feed-forward networks use σout\sigma_{\mathrm{out}}. Each normal distribution is truncated at three standard deviations on either side of zero. Finally, normalization weights are initialized to one.

Gated DeltaNet represents its decay rate with a logarithmic parameter so that the rate remains positive [28]. We initialize this parameter as log⁡A\log A, with AA sampled uniformly between 1×10−21 \times10^{-2} and 16. The time-step bias is initialized by applying the inverse softplus to a value sampled log-uniformly between 1×10−31 \times10^{-3} and 1×10−11 \times10^{-1}.

Schedule and search grids

The shared WSD schedule [11] warms up the LR over 33.6M (2252^{25}) tokens, holds it at the peak, and decays it over the final 20% of updates to 10% of the peak with a square-root profile [8]. The stable phase is independent of the total training length. We therefore reuse this phase across runs that differ only in budget, then apply a separate LR decay for each budget, reducing the cost of the sweeps. The Phase II schedules are specified in Appendix § 12.§ 1.

We choose LR values around the expected optimum, with lower values for deeper models (Table 14). This choice follows the decrease in optimal LR with model size reported by the Porian Law [23] and StepLaw [14]. The tested BS values range from 128K to 2M tokens; for each model–budget pair, they give 500–200,000 updates and keep warmup below 20% of the training tokens.

Layersη\eta grid
82e-3, 5e-3, 7e-3, 1e-2
121e-3, 2e-3, 5e-3, 7e-3
161e-3, 2e-3, 3e-3, 5e-3
208e-4, 1e-3, 2e-3, 3e-3
24 and 285e-4, 7e-4, 1e-3, 2e-3

Table 14. Peak LR values tested in the Qwen3.5 and OLMo3 baseline ladders, grouped by model depth. Each LR value is tested at BS 128K, 256K, 512K, 1M, and 2M tokens where feasible.

The baseline’s Setting 1 curve plots the best measured loss at each budget (Table 15). Setting 1 and Setting 2 reuse the best measured LR and BS from the model–budget groups listed in Table 16. The predicted optima in Table 17 initialize the Setting 3 search.

BaselineFLOPsModelDDη\etaBBLL
Qwen3.51×10181 \times10^{18}93M1.48B1e-2512K3.4743
Qwen3.53×10183 \times10^{18}202M2.02B2e-3512K3.2553
Qwen3.51×10191 \times10^{19}407M3.42B2e-3512K3.0346
Qwen3.53×10193 \times10^{19}736M5.81B3e-3512K2.8768
Qwen3.51×10201 \times10^{20}1.21B12.0B2e-31M2.7127
OLMo31×10181 \times10^{18}176M746M5e-3256K3.5170
OLMo33×10183 \times10^{18}176M2.24B2e-3512K3.2975
OLMo31×10191 \times10^{19}346M3.88B2e-3512K3.0837
OLMo33×10193 \times10^{19}614M6.72B2e-3512K2.9083
OLMo31×10201 \times10^{20}1.00B14.0B1e-3512K2.7468

Table 15. Best measured recipe of each baseline at each compute budget.

FLOPsQwen3.5 ModelQwen3.5 DDQwen3.5 η\etaQwen3.5 BBQwen3.5 LLOLMo3 ModelOLMo3 DDOLMo3 η\etaOLMo3 BBOLMo3 LL
1×10181 \times10^{18}93M1.48B1e-2512K3.474385M1.57B5e-3512K3.5361
202M674M5e-3256K3.4804176M746M5e-3256K3.5170
407M342M2e-3128K3.5826346M388M5e-3128K3.6323
736M194M3e-3128K3.7343614M224M3e-3128K3.7505
3×10183 \times10^{18}93M4.43B1e-21M3.318085M4.72B5e-3512K3.3793
202M2.02B2e-3512K3.2553176M2.24B2e-3512K3.2975
407M1.03B2e-3256K3.2656346M1.16B2e-3256K3.3238
736M581M2e-3256K3.3553614M672M2e-3256K3.3811
1.21B359M1e-3128K3.45681.00B419M2e-3128K3.5039
1.79B245M2e-3128K3.57131.91B227M1e-3128K3.6810
1×10191 \times10^{19}93M14.8B1e-21M3.211585M15.7B5e-31M3.2741
202M6.74B2e-3512K3.0846176M7.46B2e-3512K3.1284
407M3.42B2e-3512K3.0346346M3.88B2e-3512K3.0837
736M1.94B3e-3512K3.0597614M2.24B2e-3512K3.0979
1.21B1.20B1e-3256K3.11611.00B1.40B2e-3256K3.1603
1.79B818M1e-3256K3.18391.91B757M1e-3256K3.2705
3×10193 \times10^{19}93M44.3B5e-32M3.161285M47.2B2e-31M3.2144
202M20.2B2e-31M2.9750176M22.4B2e-31M3.0290
407M10.3B2e-3512K2.8892346M11.6B2e-3512K2.9336
736M5.81B3e-3512K2.8768614M6.72B2e-3512K2.9083
1.21B3.59B1e-3512K2.89531.00B4.19B1e-3512K2.9279
1.79B2.45B1e-3512K2.92711.91B2.27B1e-3512K2.9893
1×10201 \times10^{20}202M67.4B2e-32M2.9017176M74.6B2e-31M2.9575
407M34.2B3e-32M2.7834346M38.8B2e-31M2.8232
736M19.4B2e-31M2.7292614M22.4B2e-31M2.7579
1.21B12.0B2e-31M2.71271.00B14.0B1e-3512K2.7468
1.79B8.18B1e-31M2.72261.91B7.57B1e-3512K2.7593

Table 16. Best measured recipe in each model–budget group.

BaselineFLOPsM∗M^* (GFLOPs/token)D∗D^*η∗\eta^*B∗B^*L∗L^*
Qwen3.51×10181 \times10^{18}1.02978M3.82e-3315K3.4649
Qwen3.53×10183 \times10^{18}1.701.76B2.84e-3409K3.2457
Qwen3.51×10191 \times10^{19}2.993.35B2.05e-3545K3.0370
Qwen3.53×10193 \times10^{19}4.986.02B1.52e-3707K2.8716
Qwen3.51×10201 \times10^{20}8.7311.5B1.10e-3942K2.7142
Qwen3.51×10211 \times10^{21}25.5†39.2B5.87e-41.59M2.4706
OLMo31×10181 \times10^{18}0.9811.02B3.53e-3286K3.5147
OLMo33×10183 \times10^{18}1.611.86B2.65e-3371K3.2949
OLMo31×10191 \times10^{19}2.783.60B1.94e-3491K3.0812
OLMo33×10193 \times10^{19}4.566.58B1.46e-3636K2.9084
OLMo31×10201 \times10^{20}7.8512.7B1.07e-3843K2.7405
OLMo31×10211 \times10^{21}22.2†45.0B5.85e-41.41M2.4726

Table 17. Compute-optimal model sizes and training token counts, together with predicted optima of the adopted fixed-slope scaling laws. Daggers mark model scales beyond the swept range. The predictions at 1×10211 \times10^{21} FLOPs are extrapolations beyond the swept budgets.

Scaling-Law Fitting and Validation

The scaling laws are fitted to validation losses measured at the final checkpoints after WSD decay. Validation loss is averaged with equal weights over eleven domains, as specified in § 3.1: C4 English, Dolma books, Dolma common crawl, Dolma pes20, Dolma Reddit, Dolma Stack, Dolma Wiki, ICE, M2D2 S2ORC, Pile, and WikiText-103 [19].

LR–BS term

Deriving the proxy’s cosh form. To guide the form of ΔL\Delta L, we use convergence analysis to describe how LR and BS affect optimization. Our baseline ladders use Muon, and Pethick et al. [22] show that an idealized form of Muon belongs to the class of methods based on a linear minimization oracle. For these methods, Shulgin et al. [25] construct an optimization proxy from bounds on a gradient-norm measure of stationarity during training, under smoothness and gradient-noise assumptions. With a fixed model, fixed momentum, constant LR, and enough training steps to neglect the transient term, they use the three-term proxy

Q(η,B;D)=a1BDη+a2B+a3η,a1,a2,a3>0.(8)Q(\eta,B;D)=\frac{a_{1}B}{D\eta}+\frac{a_{2}}{\sqrt{B}}+a_{3}\eta,\qquad a_{1},a_{2},a_{3}>0. \tag*{(8)}

The positive coefficients absorb the fixed-model and momentum constants. The first term describes incomplete optimization within the training budget and decreases with the number of training steps D/BD/B and with LR. The second represents gradient noise and decreases with BS. The third comes from the smoothness bound and grows with the size of each update.

At a fixed BS, differentiating with respect to LR gives the minimum at ηQ∗(B)=a1B/(a3D)\eta_{Q}^{*}(B)=\sqrt{a_{1}B/(a_{3}D)}, where the first and third terms are equal. To derive the cosh form, we express LR relative to this optimum as r=η/ηQ∗(B)r=\eta/\eta_{Q}^{*}(B). The sum of the first and third terms is then proportional to r+r−1r+r^{-1}. Using cosh⁡x=(ex+e−x)/2\cosh x=(e^{x}+e^{-x})/2, we obtain the increase above this minimum:

Q(η,B;D)−min⁡ηQ(η,B;D)=a1a3BD(r+r−1−2)=2a1a3BD[cosh⁡(ln⁡ηηQ∗(B))−1].(9)\begin{aligned} Q(\eta,B;D)-\min_{\eta}Q(\eta,B;D)=\sqrt{\frac{a_{1}a_{3}B}{D}}\left(r+r^{-1}-2\right) \\ &=2\sqrt{\frac{a_{1}a_{3}B}{D}}\left[\cosh\left(\ln\frac{\eta}{\eta_{Q}^{*}(B)}\right)-1\right]. \tag*{(9)} \end{aligned}

Thus, cosh expresses the balance between the inverse-LR and proportional-LR terms in log coordinates. Multiplying or dividing the optimal LR by the same factor gives the same increase in the proxy.

Substituting the optimal LR at each BS leaves

min⁡ηQ(η,B;D)=2a1a3DB+a2B.(10)\min_{\eta}Q(\eta,B;D)=2\sqrt{\frac{a_{1}a_{3}}{D}}\sqrt{B}+\frac{a_{2}}{\sqrt{B}}. \tag*{(10)}

The first term increases with BS, while the second decreases. Minimizing their sum gives the optimal BS BQ∗=a2D/(2a1a3)B_{Q}^{*}=a_{2}\sqrt{D}/(2\sqrt{a_{1}a_{3}}) and the corresponding optimal LR ηQ∗=ηQ∗(BQ∗)∝D−1/4\eta_{Q}^{*}=\eta_{Q}^{*}(B_{Q}^{*})\propto D^{-1/4}. This order of minimization follows Shulgin et al. [25] and the general-exponent treatment of Su [26]. With hQ=a3ηQ∗h_{Q}=a_{3}\eta_{Q}^{*} and Q∗=Q(ηQ∗,BQ∗;D)=4hQQ^{*}=Q(\eta_{Q}^{*},B_{Q}^{*};D)=4h_{Q}, we then rewrite the full proxy as

Q−Q∗=4hQ[cosh⁡(12ln⁡BBQ∗)−1]+2hQBBQ∗[cosh⁡(ln⁡ηηQ∗(B))−1].(11)\begin{aligned} Q-Q^{*}=4h_{Q}\left[\cosh\left(\frac{1}{2}\ln\frac{B}{B_{Q}^{*}}\right)-1\right] \\ &+2h_{Q}\sqrt{\frac{B}{B_{Q}^{*}}}\left[\cosh\left(\ln\frac{\eta}{\eta_{Q}^{*}(B)}\right)-1\right]. \tag*{(11)} \end{aligned}

The first term is the increase after optimizing LR at the chosen BS; the second is the additional increase from using another LR at that BS. Equations (9) and (11) are exact identities for the three-term proxy in Equation (8).

The proxy’s second-order expansion gives a local quadratic form for the LR–BS dependence. Writing η~Q=ln⁡(η/ηQ∗)\widetilde{\eta}_{Q}=\ln(\eta/\eta_{Q}^{*}) and B~Q=ln⁡(B/BQ∗)\widetilde{B}_{Q}=\ln(B/B_{Q}^{*}), we obtain

Q−Q∗=hQ[(η~Q−12B~Q)2+12B~Q2]+O((∣η~Q∣+∣B~Q∣)3).(12)Q-Q^{*}=h_{Q}\left[\left(\widetilde{\eta}_{Q}-\frac{1}{2}\widetilde{B}_{Q}\right)^{2}+\frac{1}{2}\widetilde{B}_{Q}^{2}\right]+O\left((|\widetilde{\eta}_{Q}|+|\widetilde{B}_{Q}|)^{3}\right). \tag*{(12)}

Thus the slope of optimal log LR in log BS is 1/21/2, and the curvature in log BS after optimizing LR is half the curvature in log LR at the optimum. The expansion generally has third-order terms because the LR amplitude in Equation (11) varies with BS.

Adapting the proxy to validation loss. Using the proxy’s structure to model final validation loss is a heuristic step. We use the peak LR of the WSD runs in place of the proxy’s constant LR; these runs use approximate Muon updates and auxiliary AdamW (Appendix § 9.§ 1). We fit the validation-loss optima and curvature scale with the power laws in Equation (4). The power laws for the optima have precedents in § 6; the law for hh is our modeling extension of the fixed-model relation hQ=a3η~Q∗∝D−1/4h_Q = a_3\tilde{\eta}_Q^* \propto D^{-1/4}. All exponents in MM and DD are fitted to validation losses.

For the local shape, we retain the proxy’s slope γη,B=1/2\gamma_{\eta,B}=1/2 and fit the curvature ratio ρ>0\rho>0, shared across model sizes and token counts within each baseline ladder. A quadratic dependence of validation loss on log LR at fixed BS also has a fitting precedent in Bjorck et al. [3]. To describe loss farther from the predicted optima, we replace each squared log deviation x2x^2 by 2(cosh⁡x−1)2(\cosh x-1). The expansion 2(cosh⁡x−1)=x2+x4/12+O(x6)2(\cosh x-1)=x^2+x^4/12+O(x^6) shows that, at the same coefficient values, this choice preserves the quadratic terms while specifying faster growth away from the optimum. This replacement in both directions is the heuristic choice that gives Equation (5).

Relative to the exact proxy, the adopted form makes two changes to the BS dependence. It uses an LR amplitude hh shared across BS values at fixed MM, DD, replacing the proxy’s amplitude hQB/BQ∗h_Q\sqrt{B/B_Q^*}. It also uses 2hρ(cosh⁡B~−1)2h\rho(\cosh\widetilde{B}-1) for the BS dependence after optimizing LR, replacing 4hQ[cosh⁡(B~Q/2)−1]4h_Q[\cosh(\widetilde{B}_Q/2)-1]. With ρ=1/2\rho=1/2 and matching optima and curvature scale, the two forms agree through second order; their higher-order terms differ. Appendix § 11.§ 3 evaluates these modeling choices by comparing the complete forms, including a fitted form that retains the proxy shape.

Deriving the optimal LR at a specified BS. For the adopted loss model at fixed MM, DD, BB, the derivative ∂ΔL/∂η~=2hsinh⁡(η~−γη,BB~)\partial\Delta L/\partial\widetilde{\eta}=2h\sinh(\widetilde{\eta}-\gamma_{\eta,B}\widetilde{B}) vanishes at η~=γη,BB~\widetilde{\eta}=\gamma_{\eta,B}\widetilde{B}. The predicted optimal LR at that BS is therefore

η∗(M,D∣B)≡arg min⁡η L^(M,D,η,B)=η∗(M,D)[BB∗(M,D)]γη,B.(13)\begin{aligned} \eta^*(M,D\mid B)\equiv\underset{\eta}{\operatorname{arg\,min}}\ \widehat{L}(M,D,\eta,B) \\ &=\eta^*(M,D)\left[\frac{B}{B^*(M,D)}\right]^{\gamma_{\eta,B}}. \tag*{(13)} \end{aligned}

Thus γη,B\gamma_{\eta,B} is both the power-law exponent of optimal LR in BS and the slope ∂ln⁡η∗(M,D∣B)/∂ln⁡B\partial\ln\eta^*(M,D\mid B)/\partial\ln B. It determines how the best LR changes when a different BS is chosen.

Using η~−γη,BB~=ln⁡[η/η∗(M,D∣B)]\widetilde{\eta}-\gamma_{\eta,B}\widetilde{B}=\ln[\eta/\eta^*(M,D\mid B)], the full scaling law becomes

L^(M,D,η,B)=L∗(M,D)+2hρ(cosh⁡B~−1)\widehat{L}(M,D,\eta,B)=L^*(M,D)+2h\rho(\cosh\widetilde{B}-1)
+2h[cosh⁡(ln⁡ηη∗(M,D∣B))−1].(14)\qquad+2h\left[\cosh\left(\ln\frac{\eta}{\eta^*(M,D\mid B)}\right)-1\right]. \tag*{(14)}

The first line is the minimum predicted loss at the chosen BS, attained by optimizing LR. The second line is the additional loss from choosing a different LR at that BS. Consequently, ρ\rho controls how quickly the minimum predicted loss rises as BS moves away from B∗B^* after LR has been adjusted. If LR is held fixed while BS changes, the optimal LR also shifts, and both terms contribute. The coefficients of the LR–BS term are fitted together in the second stage (Appendix § 10.§ 2). After fitting, Equation (13) gives the optimal LR for each specified BS, as in the fixed-BS analysis of OpenEuroLLM [1].

Curvature and the role of ρ\rho. Expanding Equation (5) around the predicted optima gives

ΔL=h[(η~−γη,BB~)2+ρB~2]+O((∣η~∣+∣B~∣)4).(15)\Delta L=h\left[(\widetilde{\eta}-\gamma_{\eta,B}\widetilde{B})^2+\rho\widetilde{B}^2\right]+O\left((|\widetilde{\eta}|+|\widetilde{B}|)^4\right). \tag*{(15)}

For h,ρ>0h,\rho>0, this quadratic is positive definite and has elliptical contours around the predicted optimum. The Hessian H\mathbf{H} is the matrix of second derivatives with respect to (η~,B~)(\widetilde{\eta},\widetilde{B}), evaluated at this optimum with M,DM,D fixed:

H=2h(1−γη,B−γη,Bγη,B2+ρ).(16)\mathbf{H}=2h\begin{pmatrix}1&-\gamma_{\eta,B}\\-\gamma_{\eta,B}&\gamma_{\eta,B}^2+\rho\end{pmatrix}. \tag*{(16)}

The curvature scale hh equals Hηη/2H_{\eta\eta}/2, half the curvature with respect to LR in log space. The off-diagonal entry describes the LR–BS coupling, with γη,B=−HηB/Hηη\gamma_{\eta,B}=-H_{\eta B}/H_{\eta\eta}. Optimizing LR at each BS leaves $\min_{\widetilde{\eta}}\Delta L = 2hρ(cosh⁡B~−1)2h\rho(\cosh\widetilde{B}-1), whose curvature at B∗B^{*} is 2hρ2h\rho. Hence

ρ=d2dB~2min⁡η~ΔL∣B~=0Hηη=HBB−HηB2/HηηHηη.(17)\rho= \frac{\left.\dfrac{d^{2}}{d\widetilde{B}^{2}}\min_{\widetilde{\eta}}\Delta L\right|_{\widetilde{B}=0}}{H_{\eta\eta}} = \frac{H_{BB}-H_{\eta B}^{2}/H_{\eta\eta}}{H_{\eta\eta}}. \tag*{(17)}

The numerator is the curvature in log BS after optimizing LR; the denominator is the curvature in log LR at the optimum. The numerator is the Schur complement of HηηH_{\eta\eta} [4].

This ratio has a direct interpretation when comparing BS values near B∗B^{*}. If LR is optimized at every BS, the leading loss increase is hρB~2h\rho\widetilde{B}^{2}. If LR is instead held at η∗\eta^{*}, the leading increase is h(γη,B2+ρ)B~2h(\gamma_{\eta,B}^{2}+\rho)\widetilde{B}^{2}. The extra γη,B2\gamma_{\eta,B}^{2} arises because the fixed LR moves away from the optimal LR as BS changes. Fitting ρ\rho therefore allows the remaining sensitivity to BS after optimizing LR to be estimated separately from the fixed slope γη,B\gamma_{\eta,B}.

Fitting procedure

The coefficients are fitted in two stages, one per term of the law. Let gg index the model–budget groups, Lg,min⁡L_{g,\min} denote the lowest measured loss of group gg, and (Mg,Dg)(M_g,D_g) denote its model scale and token count. The first stage fits the base surface of Equation (3) to these group minima by optimizing ϕ=(L∞,αM,γM,αD,γD,γMD)\phi=(L_{\infty},\alpha_M,\gamma_M,\alpha_D,\gamma_D,\gamma_{MD}):

min⁡ϕ∑gHδ(ln⁡L∗(Mg,Dg;ϕ)−ln⁡Lg,min⁡),Hδ(r)={r22,∣r∣≤δ,δ(∣r∣−δ2),∣r∣>δ.(18)\min_{\phi}\sum_g H_{\delta}\left(\ln L^{*}(M_g,D_g;\phi)-\ln L_{g,\min}\right),\qquad H_{\delta}(r)=\begin{cases}\dfrac{r^{2}}{2},&|r|\le\delta,\\ \delta\left(|r|-\dfrac{\delta}{2}\right),&|r|>\delta.\end{cases} \tag*{(18)}

with equal weight for every group. We use the threshold of OpenEuroLLM [1], δ=0.001\delta=0.001. Table 19 compares Huber and squared losses on log-loss residuals.

The second stage holds L∗L^{*} fixed and fits the parameters θ\theta of ΔL\Delta L using the measured loss of every run ii in every group:

min⁡θ∑iHδ(ln⁡[L∗(Mi,Di)+ΔL(Mi,Di,ηi,Bi;θ)]−ln⁡Li),(19)\min_{\theta}\sum_i H_{\delta}\left(\ln\left[L^{*}(M_i,D_i)+\Delta L(M_i,D_i,\eta_i,B_i;\theta)\right]-\ln L_i\right), \tag*{(19)}

with the same Huber loss and threshold as Equation (18) and equal weight for every run. Both fitting stages use bounded nonlinear optimization in log-normalized coordinates, retaining the solution with the lowest objective across several random starts. Table 1 and Table 2 in § 4.2 report the fitted base-surface coefficients and LR–BS parameters.

Additional fit diagnostics

We compare predicted and measured losses for the swept runs used for fitting. The extrapolation runs and Phase II pretraining runs are excluded from fitting; their loss predictions are checked separately in § 4.3 and Appendix § 12.§ 2.

For nn runs with relative errors ei=L^i/Li−1e_i=\widehat{L}_i/L_i-1, the all-point MAPE is (1/n)∑i∣ei∣(1/n)\sum_i|e_i|. The best-point MAPE applies the same formula to the lowest-loss run of each model–budget group, and P95 is the 95th percentile of ∣ei∣|e_i|.

For the OLMo3 ladder, Figure 9 compares the fitted and measured dependence of loss on LR and BS, complementing the Qwen3.5 results in Figure 4. The predictions use the coefficients in Table 2.

Compute-optimal model size, training data, and predicted optima

Following the Chinchilla analysis [9] and its extension in Skaling [27], we minimize L∗(M,D)L^{*}(M,D) subject to MD=CMD=C to obtain the optimal model size and training token count. Substituting D=C/MD=C/M into Equation (3) gives the stationarity condition γMαM(M∗)−γM=γDαD(D∗)−γD\gamma_M\alpha_M(M^{*})^{-\gamma_M}=\gamma_D\alpha_D(D^{*})^{-\gamma_D}. The positive outer exponent does not change the minimizer, so with M∗D∗=CM^{*}D^{*}=C,

M∗(C)=(γMαMγDαD)1/(γM+γD)CγD/(γM+γD),D∗(C)=C/M∗(C),(20)M^{*}(C)=\left(\frac{\gamma_M\alpha_M}{\gamma_D\alpha_D}\right)^{1/(\gamma_M+\gamma_D)}C^{\gamma_D/(\gamma_M+\gamma_D)},\qquad D^{*}(C)=C/M^{*}(C), \tag*{(20)}
D∗(M)=(γDαDγMαM)1/γDMγM/γD.(21)D^{*}(M)=\left(\frac{\gamma_D\alpha_D}{\gamma_M\alpha_M}\right)^{1/\gamma_D}M^{\gamma_M/\gamma_D}. \tag*{(21)}

Equation (20) gives the optimal model scale and token count for a given budget, while Equation (21) gives the token count on the compute-optimal curve at a given model scale. Evaluating Equation (4) at the optimal model size and training token count then gives the predicted optimal LR and BS. If the BS is fixed, minimizing over the LR leaves the term 2ρh[cosh⁡ln⁡(B/B∗)−1]2\rho_h[\cosh\ln(B/B^*)-1], which must also be included when re-optimizing model size and training token count.

Substituting Equation (20) into Equation (3) makes both the model term and the data term proportional to C−γMγD/(γM+γD)C^{-\gamma_M\gamma_D/(\gamma_M+\gamma_D)}. The resulting loss has the form of Equation (1), with γC=γMDγMγD/(γM+γD)\gamma_C=\gamma_{MD}\gamma_M\gamma_D/(\gamma_M+\gamma_D).

Setting 1 of § 3.1 uses the baseline’s compute-optimal curve to choose the model size and training token count. A realizable shape of the proposed architecture has model scale MM, which rarely satisfies M=M∗(Cj)M=M^*(C_j) exactly for a compute budget CjC_j. The training token count is then set in one of two ways:

size first:D=D∗(M),training budget MD∗(M);(22)\begin{aligned} \text{size first:}\qquad&D=D^*(M), &&\text{training budget }MD^*(M); \tag*{(22)} \end{aligned}
budget first:M≈M∗(Cj),D=Cj/M, training budget Cj.(23)\begin{aligned} \text{budget first:}\qquad&M\approx M^*(C_j), &&D=C_j/M,\ \text{training budget }C_j. \tag*{(23)} \end{aligned}

In the size-first construction, the model scale is fixed and the token count should be chosen so that the model scale and token count lie exactly on the baseline’s optimal curve; the training budget can differ from CjC_j. In the budget-first construction, a model scale near the optimum should be chosen, and the token count should be set to keep the budget at CjC_j. The resulting model size and training token count differ slightly from their optimal values.

Table 17 gives the compute-optimal model sizes and training token counts from the fitted base surfaces and the predicted optimal LR and BS of the adopted fixed-slope laws at the five compute budgets and at 1×10211 \times10^{21} FLOPs. These continuous predictions complement the best measured recipes of Appendix B. The predicted model sizes and training token counts should be used to choose model shapes in Setting 1 and the ranges of model sizes searched in Setting 2 and Setting 3. The predicted optima provide the initial LR and BS for the Setting 3 search.

Extrapolation recipes

We test the frozen fixed-slope laws of Section 4.2 by comparing their predictions with the measured losses of the extrapolation runs (Table 4). Table 18 gives the LR and BS used in these runs. A single recipe is used for each model–budget pair, and the predicted LR–BS term ranges from 0.00004 to 0.0018. The base surface L∗L^* alone predicts loss with a MAPE of 0.33% on these runs, so they mainly test the base surface. For the largest swept model of each baseline ladder, we also predict the loss reduction from its best measured 1×10201 \times10^{20} recipe to its extrapolation recipe with a relative error below 6%.

BaselineModelη\etaBB
Qwen3.51.79B8e-41M
Qwen3.52.64B6e-41M
OLMo31.91B8e-41M
OLMo33.36B6e-42M

Table 18. LR and BS used for the extrapolation runs whose measured and predicted losses are compared in Table 4.

Formula Design Choices and Ablations

The ablations examine the base surface, fitting procedure, LR–BS term, and the estimation of the optima to explain the scaling-law choices summarized in Table 19. Within each baseline ladder, alternative base surfaces are fitted to the same group minima, and alternative LR–BS forms are fitted to losses from the same swept runs. Comparisons use the MAPE defined in Appendix C.3.

Coupled versus additive base surface

The coupled surface of Equation (3) reduces to the additive form of Chinchilla [9] when γMD=1\gamma_{MD}=1. Skaling motivates the coupling through the mixed derivative [27]. With X=αMM−γMX=\alpha_{M}M^{-\gamma_{M}} and Y=αDD−γDY=\alpha_{D}D^{-\gamma_{D}},

∂2L∗∂ln⁡M ∂ln⁡D=γMD(γMD−1)γMγDXY(X+Y)γMD−2,(24)\frac{\partial^{2}L^{*}}{\partial\ln M\,\partial\ln D}=\gamma_{MD}(\gamma_{MD}-1)\gamma_{M}\gamma_{D}XY(X+Y)^{\gamma_{MD}-2}, \tag*{(24)}

which vanishes for the additive form and is negative for 0<γMD<10<\gamma_{MD}<1. The coupled form thus allows the effect of model scale to depend on training data. To test the value of this interaction, we fit the additive form to the same group minima, retaining the same LR–BS term.

Table 20 gives the coefficients and best-point MAPE of both base surfaces. Figure 3 compares their IsoFLOP curves with the measured minima, and § 4.3 discusses the comparison.

Fitting order

In the two-stage procedure, we fit L∗L^{*} first and hold it fixed while fitting ΔL\Delta L (Appendix § 10.§ 2). We test whether fitting the two terms simultaneously gives more accurate loss predictions. Both approaches use the same functional forms and equal weighting. The LR–BS term takes the log-quadratic form h(η~−γη,BB~)2+ρB~2h(\tilde{\eta}-\gamma_{\eta,B}\tilde{B})^{2}+\rho\tilde{B}^{2} (Appendix § 10.§ 1); its slope γη,B\gamma_{\eta,B} and curvature ratio ρ\rho are fitted in both approaches. Table 21 reports errors on the swept runs and on runs excluded from fitting.

L∗L^*, best-pointL^\hat{L}, best-pointL^\hat{L}, all-point
BaselineTwo-stageSimultaneousTwo-stageSimultaneousTwo-stageSimultaneous
Swept runs
Qwen3.50.13%0.16%0.21%0.21%0.28%0.27%
OLMo30.11%0.24%0.19%0.18%0.31%0.29%
Runs excluded from fitting
Qwen3.5 extrapolation runs0.27%0.23%
OLMo3 extrapolation runs0.39%0.07%
Qwen3.5 150B-token pretraining0.96%1.17%

Table 21. Fitting-order ablation with a log-quadratic LR–BS term whose slope and curvature ratio are fitted. Two-stage fitting estimates L∗L^* from group minima and holds it fixed while fitting ΔL\Delta L; simultaneous fitting estimates both from the losses of all swept runs.

On the swept runs, the base surface from two-stage fitting predicts the group minima more accurately, while simultaneous fitting gives similar or lower loss error across all points by allowing L∗L^* and ΔL\Delta L to adjust together. For runs excluded from fitting, simultaneous fitting predicts loss more accurately on the extrapolation runs but less accurately on the 150B-token pretraining sweep. We retain two-stage fitting because it preserves the fit of the base surface to the group minima used to determine the IsoFLOP curves and compute-optimal model sizes and training token counts.

Form of the LR–BS term

We compare four LR–BS forms to test how accurately they predict loss and locate the optimal LR and BS (Table 22). The alternatives are a form that retains the proxy shape (Appendix § 10.§ 1), the log-quadratic form, and cosh forms with fitted or fixed LR–BS slopes. The log-quadratic form generalizes the proxy’s second-order expansion in Equation (12) by fitting the slope and curvature ratio. Each alternative uses power laws for its optima and amplitude, with optimal BS depending on both MM and DD (§ 4.4). We fit these forms using the same fitted base surface and losses from all swept runs. Alongside loss MAPE, the table reports the MAPE of the predicted optima relative to the best measured configuration of each model–budget group.

The form that retains the proxy shape has the largest loss error across all swept runs. Compared with the log-quadratic form, both cosh forms reduce all-point loss MAPE from 0.28% to 0.25% for Qwen3.5 and from 0.31% to 0.27% for OLMo3. Fitting the slope gives γη,B=0.77\gamma_{\eta,B} = 0.77 for Qwen3.5 and 0.51 for OLMo3, with nearly the same loss MAPE as fixing it. For Qwen3.5, however, fitting the slope raises the MAPE of the predicted optimal LR to 50.0%, compared with 34.5% for the log-quadratic form. Compared with fitting the cosh slope, fixing it gives an LR MAPE of 35.1% with one fewer parameter and nearly the same loss MAPE. We therefore adopt the cosh form with γη,B=1/2\gamma_{\eta,B} = 1/2 for both baseline ladders. § 4.3 compares the resulting contours with the measured LR–BS dependence.

Shared versus per-group estimation of the optima

We test whether estimating the optima separately for each model–budget group is more accurate than using the shared LR–BS term. OpenEuroLLM estimates the optima by fitting a quadratic surface to the losses in each group [1]. We fit each quadratic in log LR and log BS, retaining fits with a full-rank design matrix, a positive-definite quadratic, and a minimum strictly inside the convex hull of the tested points. These fits cover 24 of the 27 groups for Qwen3.5 and 25 of 27 for OLMo3. We then fit power laws in MM and DD to the retained minima.

When evaluated against these minima, the per-group power laws attain lower MAPE than the shared LR–BS term, which is fitted to losses from all swept runs. The LR errors are 11.3% versus 16.6% for Qwen3.5 and 5.4% versus 12.5% for OLMo3. The BS errors are 9.9% versus 10.5% for Qwen3.5 and 6.2% versus 7.5% for OLMo3.

The per-group power laws, however, predict only the optimal LR and BS. The shared LR–BS term also predicts loss away from its optima and the optimal LR at a fixed BS (Equation (13)). We therefore retain the shared term, using the per-group minima as a separate check of its predicted optima. At an imposed BS, the LR predicted by the shared term should be used to initialize the Setting 3 search (§ 3.1).

Phase II Baselines: Configuration and Results

Configuration and evaluation

The Qwen3.5 and OLMo3 Phase II baselines use the 24-layer shapes in Table 12 and the shared optimizer and initialization settings in Appendix § 9. Pretraining uses 8K context and a 2,000-update warmup. A square-root decay over the final 20% of updates reduces the LR to 10% of its peak.

Each long-context mid-training run then restores the model and optimizer state of the selected pretrained checkpoint and trains on about 10B tokens of the Dolma 3 Longmino Mix (50B) at 64K context. This stage uses a 200-update warmup followed by a stable LR and a square-root decay to zero over the final 20% of updates.

Long-context mid-training retains document boundaries within the 64K sequences [19]. Attention is causal within each document and masked across documents, and rotary positions restart at each boundary.

At 64K context, Qwen3.5 retains its pretraining RoPE base of 1×1071 \times10^{7} and rotary fraction of 0.25 without further positional scaling. By contrast, OLMo3 applies YaRN scaling [21] with factor 8 to the rotary embeddings of its full-attention layers; its sliding-window layers retain their pretraining position encoding.

The LR and BS selected in Appendices § 12.§ 2 and § 12.§ 4 form the Qwen3.5 baseline recipe, which OLMo3 reuses for both stages (Table 23). § 5.3 describes the evaluation of the resulting checkpoints and reports the principal Base Main and Base Long-Context results.

Pretraining sweep

The Qwen3.5 pretraining sweep starts from the predicted optima in Table 7. At BS 4M, the tested peak LR values extend up to 5e-3 and bracket the LR of the lowest-loss recipe. Fewer LR values are tested at BS 2M and 8M. Table 8 lists every run with its Base Main scores and the loss predicted by the frozen scaling law. Figure 6 shows the sweep’s TPP relative to the baseline ladders.

The pretraining sweep extends beyond the baseline ladder in TPP and uses a different warmup schedule. The ladder uses a fixed token count for warmup, whereas the sweep uses a fixed update count. The sweep’s warmup fraction therefore increases with BS (Appendix § 12.§ 1).

The frozen scaling law overpredicts every measured loss, with a MAPE of 1.02%. At the selected recipe, the overprediction is 1.52% for the full law and 1.03% for the base surface alone. The selected LR lies above the predicted optimum at the same BS, so the LR–BS term increases the overprediction (§ 5.1).

Downstream task rankings differ from the ranking by final validation loss. Table 24 compares the lowest-loss recipe with close alternatives on benchmarks evaluated separately from the Base Main suite. A higher-loss alternative performs best on each benchmark shown, and the best recipe varies by task. We consider these task-dependent rankings when selecting the pretraining recipe (§ 5.1).

Table 24: Qwen3.5 benchmark results at the end of pretraining for the selected recipe and alternatives from Table 8. MMLU uses length-normalized accuracy; bits per byte is measured on the reference answer. These evaluations are separate from the Base Main suite.

Metric3e-3/4M2e-3/4M2e-3/2M
Validation loss LL2.48782.49332.4958
ARC-Challenge accuracy ↑\uparrow24.8%25.5%27.2%
MMLU humanities accuracy ↑\uparrow25.4%24.6%25.9%
MMLU social sciences accuracy ↑\uparrow26.4%26.6%30.0%
MMLU STEM accuracy ↑\uparrow25.1%27.0%27.8%
Basic-skills coding accuracy ↑\uparrow60.7%64.3%63.8%
MBPP bits per byte ↓\downarrow0.5900.5880.600
Minerva Math bits per byte ↓\downarrow0.6100.6070.616

Table 24.

Effect of the decay

The pretraining runs change rank during LR decay (Figure 10). Before decay, loss increases with peak LR at every tested BS; across all runs, 8e-4/4M ranks first and 3e-3/4M eighth. During decay, loss falls by 0.09 to 0.18, generally more at higher peak LR, consistent with the sharp loss reduction during the cooldown of WSD schedules [8]. At the final checkpoint, 3e-3/4M ranks first and 8e-4/4M last. This reversal supports comparing the final checkpoints of complete schedules, as required in Phase II.

Figure 10. Loss measured before decay (a) and reduction during decay (b) in the Qwen3.5 pretraining sweep. Subtracting the reduction from the loss before decay gives the final loss in Table 8, showing how recipe rankings change during decay.

Long-context mid-training

The scaling laws are fitted to losses measured in the baseline ladders and do not apply to long-context mid-training. We therefore sweep LR and BS directly for Qwen3.5 rather than starting from predicted optima. At the pretraining BS of 4M, peak LR ranges from the final pretraining LR, 3e-4, to the pretraining peak, 3e-3; fewer LR values are tested at a doubled BS of 8M. Table 25 reports Base Main scores and 64K RULER and HELMET scores for every run.

Table 25: Qwen3.5 long-context mid-training sweep: Base Main scores and 64K RULER and HELMET scores for every run. Long-context scores for the pretrained checkpoint are reference measurements. Base Main input limits are 8K for the pretrained checkpoint and 64K for the mid-trained checkpoints. Figure 8 plots the Base Main and long-context scores, and Table 26 gives long-context results at each evaluation length.

RULERHELMET
η\etaBB4K8K16K32K64K8K16K32K64K
Qwen3.5
Pretrained checkpoint65.158.434.023.817.535.924.122.923.3
3e-44M77.271.969.161.351.945.542.539.937.8
5e-44M78.874.370.161.652.645.741.739.138.4
1e-34M81.274.471.265.357.046.342.840.939.1
1.5e-34M79.275.072.265.656.047.745.542.140.0
3e-34M81.676.971.163.053.745.442.340.037.5
1e-38M80.073.770.262.153.845.443.439.837.2
1.5e-38M80.474.470.564.754.746.644.641.038.3
3e-38M79.375.069.365.756.447.445.443.240.0
OLMo3
Pretrained checkpoint69.451.97.94.12.734.68.19.17.4
1e-34M75.363.252.443.336.142.137.134.729.6

Table 26. Long-context scores for the Qwen3.5 mid-training sweep and the OLMo3 baseline checkpoints.

Table 26: Long-context scores for the Qwen3.5 mid-training sweep and the OLMo3 baseline checkpoints; scores for the pretrained checkpoints are reference measurements. Figure 8a,Figure 8b plots the sweep results; Table 27 breaks down the 64K scores by category.

η\etaBBRetr.Multi-hopAggr.QARecallRAGRerankICLLongQA
Pretrained checkpoint19.89.213.716.56.924.50.169.015.8
3e-44M67.016.421.440.038.144.50.276.430.0
5e-44M68.717.021.637.040.944.80.473.832.1
1e-34M73.518.428.439.044.144.52.275.829.0
1.5e-34M71.518.227.342.041.245.72.478.032.9
3e-34M68.918.422.641.533.844.82.677.229.2
1e-38M69.321.024.837.038.242.21.975.827.8
1.5e-38M70.020.825.640.041.845.21.374.428.6
3e-38M71.716.426.245.040.645.53.479.031.5

Table 27. Qwen3.5 category scores at 64K; scores for the pretrained checkpoint are reference measurements. RULER categories are needle-in-a-haystack retrieval (Retr.), multi-hop tracing (Multi-hop), aggregation (Aggr.), and question answering (QA). HELMET categories are synthetic recall (Recall), retrieval-augmented generation (RAG), passage re-ranking (Rerank), many-shot in-context learning (ICL), and long-document QA (LongQA).

Doubling BS lowers the 64K RULER and HELMET scores at LR 1e-3 but raises them at 3e-3. At BS 8M, both scores increase with LR up to the largest tested value.

Long-context mid-training also preserves RULER performance as the evaluation length grows (Figure 8a and Table 26). At 64K, beyond the 8K pretraining window, the pretrained checkpoint retains 27% of its 4K RULER score, compared with 66%–71% for the mid-trained checkpoints. Among the tested recipes, the selected run has the highest RULER score at 64K, the length targeted by Phase II, while 3e-3/4M scores highest at 4K and 8K. Table 27 gives the 64K scores by category.

We also reuse the Qwen3.5 baseline recipe for OLMo3 without further tuning. Table 28 reports the resulting Base Main scores and the 64K RULER and HELMET scores after each training stage.

CheckpointScoreMathMC-STEMMC-NonSTEMGenQARULERHELMET
Pretrained checkpoint26.22.125.227.749.82.77.4
Mid-trained (1e-3/4M)40.516.443.849.252.436.129.6

Table 28. OLMo3 results with the Qwen3.5 recipe reused without tuning. Base Main input limits change from 8K after pretraining to 64K after mid-training. Long-context scores for the pretrained checkpoint are reference measurements. Table 10 and Table 26 give task-level Base Main results and long-context scores at each evaluation length, respectively.

Literature Survey Methodology

The survey covers language-model architecture studies from November 2023 to October 2026, including multimodal models, with no minimum parameter count. Eligible papers described changes to a model’s internal computation and reported training or evaluation experiments. Studies limited to data, optimization, prompting, or execution efficiency were excluded unless they included architectural changes.

To identify eligible papers, we searched titles and abstracts across all arXiv categories, then used AI-assisted screening and targeted full-text checks.

Figure 1 compares work from universities and independent researchers alone with work involving industry. We classified a paper as work involving industry if at least one author was affiliated with a company, government body, independent research institution, or non-university nonprofit, including dual affiliations. We obtained affiliations from formal publications or arXiv versions and also used verified information about authors’ employment at the time of publication; funding alone did not determine the group.

Each paper was counted once by publication date, using its earliest documented public release or its first arXiv submission when no earlier date was established. The figure spans 23Q4–26Q3, with 2,919 papers in the paper-count plot. The three-month periods are labeled Q1 for February–April, Q2 for May–July, Q3 for August–October, and Q4 for November–January of the following year.

The citation plot contains 117,629 cumulative citations to 2,859 papers from the same periods. We retrieved citation counts from Semantic Scholar on October 31, 2026. We matched papers to Semantic Scholar records only when both the arXiv identifiers and normalized titles agreed; unmatched papers were excluded. In each plot, bars show group totals and the line shows the share of papers or citations for work involving industry. Citation totals are grouped by publication date and are unadjusted for paper count or age.

References

  1. [1]Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, Jörg Franke, Sampo Pyysalo, Jenia Jitsev, and Aaron Klein. Deriving scaling laws for openneurollm models: Learning rate, batch size and loss, 2026. URL https://arxiv.org/abs/2608.28308. 4.2, 4.2, 5, 7, 6, 6, C.1, C.2, 19, D.4
  2. [2]Amanda Bertsch, Luca Soldaini, Matthew R. Gormley, Graham Neubig, Hannaneh Hajishirzi, Kyle Lo, and Dirk Groeneveld. Cracks in the foundation: Seemingly minor architectural choices impact long context extension. In Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=CwCFRqBoH. 6arxiv.org/abs/2608.10296
  3. [3]Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal lr across token horizons. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 83640–83657, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/cffa22c56c0df3b3edbd1df8a9ad67804-Paper-Conference.pdf. 4.2, 6, C.1arxiv.org/abs/2409.19913
  4. [4]Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, March 2004. ISBN 9780511804441. doi: 10.1017/cbo9780511804441. URL http://dx.doi.org/10.1017/CBO9780511804441. C.1
  5. [5]DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shanhhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek llm: Scaling open-source language models with longerism, 2024. URL https://arxiv.org/abs/2401.02954. 1, 6, 7, 6, 6
  6. [6]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Srivankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Cauchteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Diewuke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind ThattaI, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurull, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasudén Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpōukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borbinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yasher Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montavlo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Lisovik, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspergren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Ho U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleiman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  7. [7]Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannah Hajishirzi. OLMES: A standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 5020–5048, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.282. URL https://aclanthology.org/2025.findings-naacl.282/.DOI
  8. [8]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws for compute-optimal training beyond fixed training durations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 76232–76264. Curran Associates, Inc., 2024. doi: 10.52202/079017-2427. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/8b970e15a889bf5d12542810df8eae8fc-Paper-Conference.pdf.
  9. [9]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejо, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc., 2022. doi: 10.52202/068431-2176. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf.
  10. [10]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIObc76Sy.arxiv.org/abs/2404.06654
  11. [11]Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=3X2L2TFr0f.arxiv.org/abs/2404.06395
  12. [12]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
  13. [13]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361.
  14. [14]Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, Shuigen Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part i, step law – optimal hyperparameter scaling law in large language model pretraining, 2025. URL https://arxiv.org/abs/2503.04715.
  15. [15]Jingyuan Liu, Jianlin Su, Xingchen Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianz hou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for llm training, 2025. URL https://arxiv.org/abs/2502.16982.
  16. [16]MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yi, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https://arxiv.org/abs/2501.08313. 1
  17. [17]Mistral AI. Mistral large 4. Technical report, Mistral AI, 2026. URL https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/assets/technical_report.pdf. 1
  18. [18]Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2-3 2019. Association for Computational Linguistics. URL https://aclanthology.org/2019.iwslt-1.17/. B.2DOI
  19. [19]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhatia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zheijang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. Olmo 3, 2026. URL https://arxiv.org/abs/2512.13961. 3, 3.1, 3.2, 4, 5.3, 11, 6, C, E.1
  20. [20]OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Bataltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogenini, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Graffstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschle, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Tedor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavy, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaraghav, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. 1
  21. [21]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 31932–31951, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/874a4d89f2d04b4bcf9a2c19545cf040-Paper-Conference.pdf. E.1arxiv.org/abs/2309.00071
  22. [22]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49069–49104. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/pethick25a.html. C.1arxiv.org/abs/2502.07529
  23. [23]Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 100535–100570. Curran Associates, Inc., 2024. doi: 10.52202/079017-3189. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/b6341525cd84f3be0ef203e4d7cd8556-Paper-Conference.pdf. 7, 6, 6, B.3DOI
  24. [24]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5. 4
  25. [25]Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang, Niccolò Ajroldi, Bernhard Schölkopf, and Antonio Orvieto. Deriving hyperparameter scaling laws via modern optimization theory, 2026. URL https://arxiv.org/abs/2603.15958. 4.2, 4.4, 4.6, C.1, 19
  26. [26]Jianlin Su. Deconstructing scaling laws: A trio of optimization, architecture, and data, Jul 2026. URL https://kexue.fm/archives/11833. 4.2, 6, C.1
  27. [27]Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, and Kartik Ahuja. Scaling: Chinchilla’s exponents meet kaplan’s coupling, 2026. URL https://arxiv.org/abs/2608.07222. 4.2, 5, 6, C.4, 19, D.1
  28. [28]Songlin Yang, Jan Kautz, and Ali Hata­miza­deh. Gated delta networks: Improving mamba2 with delta rule. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 29687–29707, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/4904fad153f6434a7bcf04465d4be2cc-Paper-Conference.pdf. 4, B.2arxiv.org/abs/2412.06464
  29. [29]Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Iszak, Moshe Wasserblat, and Danyi Chen. Helm et: How to evaluate long-context models effectively and thoroughly. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 98914–98965, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/f5332c8273d02729730a9c24dec2135e-Paper-Conference.pdf. 3.2, 5.3, 6arxiv.org/abs/2410.02694

Paper details

vlnc:W_kqdhw5mx
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents