betaFeedback

WorksComputer Science

Rethinking Pre-pretraining

Abstract

We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that PPT gains reported in prior work are sensitive to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% perplexity improvement from PPT under the original initialization scheme. An alternative scheme improves both methods, but the no-PPT baseline more, closing the gap between them. However, when PPT receives additional compute rather than a share of the natural-language budget, gains persist at small natural-language budgets even with improved initialization, but disappear by 1B tokens. Finally, models forget what they learn during PPT in subsequent natural-language training; interleaving the same data throughout training prevents forgetting, but does not significantly improve natural-language performance. Our results suggest limited benefits of synthetic pre-pretraining as a distinct training stage and motivate rethinking how such data are used in language model training.

Introduction

Rich computational structure can be learned from naturally occurring text. The benefits of scaling training on such tokens have yet to hit a ceiling, though increasingly large amounts of data are required for further improvements in performance [15]. By contrast, humans acquire language from far less linguistic experience (Warstadt et al., 2023), and their reasoning abilities are largely dissociable from linguistic proficiency [6]. This contrast has motivated recent work to ask whether useful linguistic and computational priors can instead be acquired from synthetic data. Beyond making language learning more sample-efficient, synthetic distributions supply effectively unlimited training data once high-quality natural language becomes scarce.

This idea is operationalized in prior work through pre-pretraining (PPT): a training phase on synthetic data that precedes conventional pretraining (PT) and is intended to produce an initialization with useful priors for learning language. Different synthetic distributions target different computational priors: kk-Dyck encourages hierarchical dependency tracking [9], neural cellular automata (NCA) expose models to latent transition rules and long-range dependencies [20], and procedural tasks train operations such as copying, sorting, and retrieval (Jiang et al., 2026). These approaches have reported gains in language modeling and context recall, including up to 8.6% lower perplexity for NCA, both when PPT is given additional compute on top of a fixed natural-language budget and when PPT and natural-language training share a fixed total compute budget.

Although prior work has demonstrated gains from synthetic PPT, how robust these gains are across other training configurations and regimes remains unclear. We therefore revisit synthetic PPT across a range of training regimes. We study three synthetic distributions (kk-Dyck, NCA, and declarative reference corpora (DeclRef)), alongside code, which prior work has found can improve downstream reasoning beyond coding tasks ([2]).

We begin by asking whether synthetic PPT can provide useful inductive biases when it is not competing with natural-language training for compute. In this additive-compute setting, where PPT is added on top of a fixed natural-language budget, we first reproduce a prior NCA result as a positive control before testing the setting more broadly. We then ask whether PPT is worth its compute when it competes directly with natural-language training for a fixed FLOP budget. We evaluate this compute-matched condition across three regimes: a data-constrained setting, where the alternative is repeated training on the same 100M natural-language tokens; a data-unconstrained setting, which tests whether any PPT recipe remains competitive as training compute scales; and a high token-to-parameter ratio setting, which probes whether PPT becomes useful far beyond the Chinchilla-optimal frontier. Our findings are as follows:

  1. Gains from PPT are sensitive to the baseline initialization scheme. We reproduce a previously reported NCA result under its original setup, recovering a 5.4% perplexity improvement from PPT. However, changing the token embedding initialization from N(0,1)\mathcal{N}(0, 1) to N(0,0.02)\mathcal{N}(0, 0.02) closes the gap between NCA and the no-PPT baseline.

  1. Additive compute: PPT can help at small natural-language budgets. When each method is tuned over initialization schemes, kk-Dyck and NCA improve over the best no-PPT baseline at 50M and 100M natural-language tokens (budgets reported to parallel the amount of language humans may learn from, (Warstadt et al., 2023)). However, by 1B tokens the advantage disappears. This suggests synthetic PPT can instill useful inductive biases, but its benefits diminish at larger natural-language budgets.

  1. Compute-matched: natural-language training performs as well as or better than PPT across all compute-matched regimes. This includes the strongest scarcity case, where multiple epochs on 100M unique natural-language tokens beat every synthetic curriculum. The same holds on most of our downstream evaluations, both DCLM CORE [21] and BLiMP (Warstadt et al., 2020), with exceptions confined to our smallest compute budget (Table 1).

SettingPPT methodBaseline (0% PPT)PPT score
BLiMP (↑ better)Data-constrained (100M NL, 1×10181 \times10^{18})DeclRef 10%0.721 ± 0.0040.731 ± 0.005
DCLM CORE (↑ better)Additive (50M NL)kk-Dyck0.024 ± 0.0040.032 ± 0.003
DCLM CORE (↑ better)Data-unconstrained (1×10181 \times10^{18})C4 1%0.106 ± 0.004
DCLM CORE (↑ better)Data-unconstrained (1×10181 \times10^{18})Code 1%0.088 ± 0.0060.104 ± 0.004
DCLM CORE (↑ better)Data-unconstrained (1×10181 \times10^{18})DeclRef 1%0.097 ± 0.002
DCLM CORE (↑ better)Data-unconstrained (1×10181 \times10^{18})DeclRef 5%0.108 ± 0.011

Table 1. Downstream benchmark improvement. Across the models reported in Figure 1 and Figure 4, these are the cases whose aggregate BLiMP or DCLM CORE score sits above their baseline. However, C4, the natural-language control, reaches a DCLM CORE score in the same range as the PPT methods. In this table, NL denotes natural-language tokens.

  1. Mechanism: learned PPT priors may not survive the phase transition to natural language. We find that gains acquired during PPT on the PPT distribution disappear over the PPT-to-PT phase boundary. Uniformly interleaving the same data throughout training instead preserves performance on those distributions, but we do not find significant improvement in natural-language validation loss.

These findings highlight the surprising effectiveness of naturally occurring data as a training signal: even when unique natural-language tokens are scarce, repeated training on them remains more useful than training on the synthetic corpora we test. Importantly, we do not take this to imply that synthetic data cannot help. Instead, our results point to front-loading synthetic data as a poor way to use it, since what is learned during PPT is progressively lost once natural-language training begins. It may instead be better integrated gradually, through interleaving that avoids the abrupt distribution shift between PPT and PT. Our results therefore motivate rethinking the PPT phase, rather than ruling out synthetic data as a training corpus.

Background & Related Work

Pre-pretraining

All prior work in synthetic pre-pretraining shares a common hypothesis: training on certain synthetic distributions can instill language-modeling priors that transfer to natural language. Designed to encourage in-context learning, these distributions can be generated procedurally in effectively unlimited quantities. PPT corpora in prior work can be divided broadly into three categories: formal languages [9, 23], algorithmic tasks (Wu et al., 2022; Jiang et al., 2026), and data sampled from randomly initialized neural networks [20, 7]. Of these, we focus on kk-Dyck and NCA, for which prior work reports the largest improvements in natural-language perplexity.

Hu et al. [9] introduce PPT on kk-Dyck languages, formal languages over a vocabulary of kk bracket types whose grammar requires that every opening bracket is eventually closed. For example, at k=2k = 2, with vocabulary {(,),[,]}\{(, ), [, ]\}, the string ([()]) is valid. They find the kk-Shuffle Dyck variant performs best at k=128k = 128, and it is this variant we refer to as kk-Dyck throughout the paper. Lee et al. [20] instead use NCA for PPT. A cellular automaton is a grid of cells, each holding a value from a finite vocabulary—for example, one of ten symbols. The grid evolves over discrete time steps: at each step, every cell takes a new value as a function of the cells in its immediate neighborhood, using a single update rule applied identically to every cell. This can be defined as follows: let xtx_t denote the state of the whole grid at time step tt, then an initial state x0x_0 and an update rule ff give xt=f(xt−1)x_t = f(x_{t-1}). An NCA replaces ff with a randomly initialized convolutional neural network.

To construct an NCA PPT corpus, Lee et al. [20] sample a random initial grid, with each cell assigned a value drawn from the fixed vocabulary, together with a randomly initialized convolutional neural network as the update rule. They then run the simulation forward for a fixed number of time steps. At this point, the resulting rollouts range from low-entropy static grids to more complex, visually interesting ones. Hence, each rollout is further filtered by a proxy-complexity metric, the gzip compression ratio, and only those falling within the highest compression band are kept. Each grid in a retained rollout is then flattened exactly as a 2D image tensor would be, and the flattened grids are concatenated in time step order into a one-dimensional token sequence for training.

The experimental protocol is largely shared across the PPT methods in the literature. The natural-language token budget and training recipe are fixed, and a synthetic data phase is prepended, with the number of PPT tokens varied to optimize the downstream natural-language loss after PT. The usual control replaces the synthetic tokens with an equal number of natural-language tokens, and a gain is reported if PPT reaches a lower downstream loss than this control. Methods are therefore compared at equal total tokens and equal model size.

In their largest-scale experiment, training a 1.6B-parameter model, Lee et al. [20] report that PPT on 164M NCA tokens improves perplexity by 5.7% compared to the no-PPT baseline. In Hu et al. [9]’s largest experiment, pre-pretraining a 1B-parameter model on kk-Dyck before 1.6B natural-language tokens reduces final cross-entropy loss by around 0.6% (∼2%\sim2\% in perplexity). Both works additionally report gains on public downstream benchmarks: reasoning benchmarks for NCA like BIG-bench Lite [20], and BLiMP for kk-Dyck [9]. We evaluate both fronts in this work, scoring every method on BLiMP (Warstadt et al., 2020), and on the DCLM CORE suite [21], which contains BIG-bench tasks, among them a subset of BIG-bench Lite. A recent study reproduced the kk-Dyck gain [9] under the original conditions. However, it found that the gain did not consistently persist across all model sizes, tokenizers, and languages tested (Riazhskykh et al., 2026). These mixed results motivate our study, and since kk-Dyck has already been reproduced, our own reproduction (§ 4) focuses on NCA. Concurrently with our work, Yamaguchi et al. (2026) study PPT on kk-Shuffle Dyck and NCA at larger scales, where they report downstream benchmark gains for PPT methods but do not claim improvements in validation loss. However, like prior work, they also keep the weight-initialization scheme fixed. We find that ablating it significantly improves the natural-language baseline, enough to close its gap with PPT.

Training regimes

How a compute budget should be spent depends on which resource is scarce. When natural language is the constraint, additional compute buys only repeated passes over the same tokens. Muennighoff et al. [24] characterize this setting and find that, for their models and data, up to four epochs of repetition are close to lossless, after which the gains from additional compute rapidly diminish and new data is more useful. The slowrun benchmark (Vegesna et al., 2026) isolates the regime in its extreme form, fixing a 100M-token sample of FineWeb and leaving compute essentially unconstrained. Work in this setting has shown that heavy regularization and ensembling continue to buy substantial data efficiency well past the point at which ordinary training overfits [19]. PPT is motivated by the same scarcity, so these are competing answers to one question: what to do with compute once the natural-language data has run out. When data is instead plentiful, compute becomes the constraint, which is the setting where Hoffmann et al. [8] show that model size and token count should be scaled together. Deployed models, however, are rarely trained at that ratio, because training compute is not the only cost; once inference is accounted for, this moves the optimum toward smaller models trained on more tokens [31, 32]. These are the three compute-matched regimes we study: data-constrained, data-unconstrained, and over-Chinchilla, a comparison that prior PPT work has not explicitly made. Because the compute-optimal model size may itself depend on data quality and curriculum [3], in the data-constrained and data-unconstrained regimes we search model size separately for every method and budget.

Experimental Setup

The research question we explore is: in additive and compute-matched settings, does PPT yield a measurable improvement over a natural-language baseline? We test this question in four settings, one in the additive setting, followed by three compute-matched configurations. In this section we describe the architecture, datasets, hyperparameter optimization, and evaluation protocol shared across all four settings.

Additive compute. We first evaluate the additive setting, where PPT is prepended to a natural-language run rather than substituting any part of its training budget (Lee et al., 2026; Hu et al., 2025; Jiang et al., 2026). We fix model size at 336M parameters, and set the natural-language PT budgets to {50M,100M,1B}\{50\mathrm{M}, 100\mathrm{M}, 1\mathrm{B}\}. Then, for each PT budget, we vary the number of PPT tokens over {5M,20M,50M,100M,200M,500M,1B}\{5\mathrm{M}, 20\mathrm{M}, 50\mathrm{M}, 100\mathrm{M}, 200\mathrm{M}, 500\mathrm{M}, 1\mathrm{B}\}, asking which quantity of PPT minimizes final natural-language validation loss. We ablate over three weight-initialization schemes: the scheme used by Hu et al. [9], which initializes every parameter from a Normal distribution N(0,0.02)\mathcal{N}(0, 0.02); the scheme used by Lee et al. [20], which initializes token embeddings from the standard Normal distribution N(0,1)\mathcal{N}(0, 1) and all remaining parameters from N(0,0.02)\mathcal{N}(0, 0.02); and the spectral-based initialization of NanoChat used throughout the rest of this paper ([16] (Yang et al. 2024b)), which draws linear-layer weights from N(0,σ)\mathcal{N}(0,\sigma) with σ=min⁡(1,dout/din)/din\sigma= \min(1,\sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}})/\sqrt{d_{\mathrm{in}}}, token embeddings from N(0,1)\mathcal{N}(0,1), and zero-initializes the language-modeling head and all residual projection weights. This ablation is additional to prior work, and turns out to be crucial for answering our research question.

Compute-matched. For the rest of the experiments, we instead fix the total compute budget and ask whether PPT can justify a share of it. Each experiment begins from a total training budget CC, measured in FLOPs, and α\alpha giving the fraction of that budget spent on PPT. A run with α\alpha receives αC\alpha C FLOPs of PPT followed by (1−α)C(1-\alpha)C FLOPs of natural-language pretraining. Hence, a run with α=0\alpha= 0 spends the entire budget on natural language. We use α∈{0,0.01,0.05,0.10}\alpha\in\{0, 0.01, 0.05, 0.10\}, with the largest percentage used only in the data-constrained regime: early experiments at the larger budgets showed no benefit that justified the extra runs, so α=0.10\alpha= 0.10 was not run there. Note that α\alpha is a fraction of compute, not of tokens: the number of tokens each phase consumes follows from the allocation rather than being set directly. Our FLOP measurement method is covered in Appendix § 14.

The compute-matched experiments span three configurations. The data-constrained configuration fixes natural language at 100M tokens and the total budget at 1×10181 \times10^{18} FLOPs. The data-unconstrained configuration leaves natural language abundant and sweeps over 1×10181 \times10^{18}, 3×10183 \times10^{18} and 9×10189 \times10^{18} FLOP budgets. At each budget, we search over model size to minimize validation loss so models are trained at compute-optimality. The over-Chinchilla configuration trains a model of 881M parameters on 50B tokens, ∼\sim57 tokens per parameter, evaluating NCA and kk-Dyck, the two curricula from prior work we revisit, each at 1% (500M tokens). In all three compute-matched settings we use only the spectral-based default initialization of the NanoChat version we build on [16], as it has been well tuned by the NanoChat community. Using this initialization is further supported by our additive compute experiments, where at the largest natural-language budget the spectral-based initialization performed best for both the baseline and the PPT methods (Figure 4).

Figure 4. PPT is useful at 50M–100M natural-language tokens, but its gain at 1B disappears under spectral-based initialization. In the additive-compute regime with a 336M-parameter model, NCA and kk-Dyck, each at its lowest-loss configuration, improve on the optimal no-PPT baseline at 50M and 100M natural-language tokens. Under spectral-based initialization this gain disappears by 1B tokens, but under the other initializations it persists at every budget (Appendix § 16.§ 1 shows the full breakdown). Because spectral-based initialization yields the lowest loss for every method at this largest budget, we use it in all compute-matched experiments, which run at comparable FLOP budgets.

Architecture and data. All our models are trained using a fork of NanoChat, with the Muon optimizer (Jordan et al., [14]; Liu et al., [22]; Karpathy, [16]), which updates the transformer blocks’ weight matrices, while the embedding and unembedding tables are trained under AdamW. However, we find that in our smaller-scale experiments the PPT phase often does better under AdamW, while at larger scales Muon performs on par, and so we ablate AdamW, applied to all parameters, only for the additive-compute experiments and the 1×10181 \times10^{18} data-constrained experiment. NanoChat sets the hidden size to 64d64d for a model of depth dd, so the parameter count follows from the depth alone. Depth is therefore the only capacity-related hyperparameter we search.

For our main natural-language corpus we follow NanoChat and use FineWeb [25]. For kk-Dyck and NCA, we generate the data using the same code released by prior work (Appendices § 12.§ 1 and § 12.§ 2). We additionally introduce DeclRef, a synthetic corpus that, like the procedural tasks of Jiang et al. [12], targets retrieval from context (Appendix § 12.§ 3). Examples of these PPT distributions can be seen in Figure 2. Alongside the synthetic curricula, we pre-pretrain on two real-text corpora, C4 [29] as a natural-language control and the GitHub Code dataset [5]. The additive setting is aimed at revisiting the additive-compute claims made for the synthetic distributions of prior work, NCA and kk-Dyck, so for this setting we do not test DeclRef. Our data-constrained experiments add DeclRef but leave out code and C4, since these supply additional real text, which would break the 100M-token constraint on natural language. The data-unconstrained experiments do include these other corpora, while the over-Chinchilla experiments return to NCA and kk-Dyck alone. Following prior work [20], the synthetic languages are encoded with their own, far smaller vocabularies. At the phase boundary we therefore expand the vocabulary to its natural-language size and reinitialize both the token embedding table and the language-modeling head, carrying only the transformer body across; each phase has its own learning-rate schedule and the optimizer state is reset at the boundary. Code and C4 share the natural-language tokenizer, so their embedding table and head carry over unchanged. The C4 control instead runs both its PPT and PT phases under a single learning-rate schedule with no optimizer state reset, since in our experiments resetting the optimizer state at the boundary hurt its final loss (Appendix § 15.§ 3). This deviates from prior implementations, which run the natural-language control as a separate phase with its own learning-rate schedule and an optimizer state reset [9, 20] (Yamaguchi et al. 2026). We recommend this setup for natural-language controls in future PPT work.

Figure 2. Tokenizing synthetic pre-pretraining data. For NCA, each grid of a rollout is split into 2×22 \times2 patches that each become one token, and grids are concatenated in time order with start and end tokens. For kk-Dyck, the opening and closing bracket types are each their own token. For DeclRef, keywords, scope brackets, names and values are unique tokens. In the example shown, a reference inside the opened scope retrieves the value of a name declared in that scope, and after the scope closes a reference retrieves the value of a name declared before the scope. Generation and tokenization details are given in § 2 and Appendix § 12.

Hyperparameter optimization. A subset of our hyperparameters are optimized with LLMTuna, a language-model-based hyperparameter optimizer we release alongside this paper. LLMTuna optimizes held-out validation bits per byte (BPB) by proposing each successive hyperparameter configuration from a rolling context of prior trials and their results, together with any files in the codebase one may want to include as context; this is in line with recent work using language models as hyperparameter optimizers [17] (Yang et al. 2024a; Jiang & Zhao 2026). We use it because we found it more sample-efficient than Bayesian hyperparameter optimization using Optuna [1] (see Appendix D.2). The hyperparameters searched and held fixed in each setting are detailed in Appendix § 15.

Evaluation. Our primary metric is BPB on a held-out split of FineWeb (∼10\sim10M tokens); hyperparameter tuning uses a separate validation set. We additionally evaluate on held-out splits of C4 and GitHub code (Appendix § 16), on the DCLM CORE suite [21], and on BLiMP (Warstadt et al., 2020), the grammaticality benchmark used by Hu et al. [9]. DCLM CORE aggregates 22 benchmarks covering knowledge, commonsense, reading comprehension, and symbolic reasoning (Appendix § 16.§ 5). Every metric is reported as the mean over three seeds, with ±\pm values and error bars giving one sample standard deviation.

Additive Compute

The reproduced NCA advantage vanishes under an alternative initialization. We begin by reproducing one of the experimental setups of Lee et al. [20], from their Section 5.1. The exact setup is not described in § 3, but fits within the additive-compute regime. We train a 400M-parameter Llama model on ∼9\sim9B tokens of OpenWebText, and PPT on 164M NCA tokens. Doing so delivers a 5.4% perplexity improvement over the no-PPT baseline. However, changing only the scale at which the embeddings are initialized [10] improves the baseline by 9.0%—more than the reported gain itself—and applying the same initialization to the NCA method leaves the two indistinguishable (Figure 3). For more details on the reproduction see Appendix § 13. Note that this experiment isolates sensitivity to initialization, but does not identify the best NCA configuration under the alternative initialization, since we do not retune the number of PPT tokens in that setting. However, this experiment is meant to show how much performance gain is available on the baseline before any PPT data is introduced, rather than to establish the best achievable NCA configuration under the original training scheme. The latter is studied in the additive-compute and compute-matched experiments that follow.

Figure 3. Gains from NCA PPT are sensitive to baseline initialization. A 400M-parameter Llama model trained on ∼9\sim9B tokens of OpenWebText, reproducing an NCA PPT experiment of Lee et al. [20]; we plot perplexity to match how prior work reports it. The groups differ only in weight initialization. With N(0,1)\mathcal{N}(0,1) embeddings and N(0,0.02)\mathcal{N}(0,0.02) elsewhere, NCA PPT improves perplexity over the no-PPT baseline by 5.4%. Initializing the embeddings from N(0,0.02)\mathcal{N}(0,0.02) instead improves the baseline itself by 9.0%, and the two methods become inseparable despite the additional compute the NCA phase receives.

PPT helps at small natural-language budgets, but not at larger ones. We next ask whether PPT can help in the additive setting, where the natural-language budget is tightly constrained and the synthetic phase does not compete for it. Figure 4 shows, for each PPT method at each natural-language token budget, the best PPT token count and weight initialization. At 50M and 100M natural-language tokens both PPT methods improve over the best no-PPT baseline, whose winning configuration uses spectral-based initialization. This means that at these low budgets the synthetic languages do instill useful inductive biases. This is also seen in Table 1, where kk-Dyck PPT at the 50M natural-language token budget improves DCLM CORE accuracy over the no-PPT baseline. However, at 1B tokens, under the spectral-based initialization, all PPT methods fall within the baseline’s seed spread. Yet under the alternative baseline initialization, both PPT curricula still appear to improve over the baseline at the 1B token scale. The additive setting therefore shows two overlaid effects: at small natural-language budgets, PPT helps regardless of initialization, whereas at 1B it only helps against baselines without spectral-based initialization. For further details on these additive-compute experiments, see Appendix § 16.§ 1, where Figure 8 breaks the results down by initialization and PPT optimizer.

Full synthetic PPT token sweep across natural-language token budgets, initialization schemes, and PPT methods

Figure 8. Full synthetic PPT token sweep behind Figure 4. FineWeb validation bits-per-byte against the amount of synthetic PPT added on top of a fixed natural-language token budget. Rows are that budget, columns are the initialization scheme: two as used in the prior literature that introduced these curricula, and the spectral-based initialization. The model size here is 336M parameters. Within each panel the leftmost point is that configuration’s own baseline with no PPT, marked by the gray dashed line. The green dashed line and band mark the lowest-loss no-PPT baseline across the three initialization schemes, providing a reference for comparisons across initializations. Solid lines with filled markers are Muon PPT and dashed lines with open markers are AdamW PPT. AdamW performs better than Muon for the PPT methods at the lower natural-language budgets; at 1B tokens, however, the two are indistinguishable under spectral-based initialization. Importantly, if one is to train with the alternative initializations, the PPT benefit would still exist even at this larger budget.

Compute-Matched

Although the previous section shows that PPT can improve performance when compute is added on top of a fixed natural-language budget, this does not establish that synthetic data is the best use of those resources. In practice, the same compute could instead be spent on further natural-language training, including repeated epochs when unique language is scarce. We therefore turn to compute-matched experiments, asking whether PPT can justify diverting part of the training budget away from natural language.

Across the tested regimes, allocating compute to synthetic PPT never beats spending it on natural language. Figure 1 shows results across the three compute-matched configurations. In the data-constrained regime, diverting any fraction of the budget to synthetic PPT increases validation BPB, with the effect leveling off beyond 5%. The compute is better spent on additional epochs over the same natural-language data. In the over-Chinchilla regime, neither NCA nor kk-Dyck improves over the baseline on validation loss or DCLM CORE accuracy. The data-unconstrained scaling confirms this pattern holds across budgets: no PPT method significantly outperforms the baseline in terms of natural-language validation BPB, with 1% closer to the baseline than 5% at the two smaller budgets, and the natural-language C4 control matching the baseline. To check that trend, we fit a power law to each method’s results across these budgets and find that no PPT method has either a lower multiplicative constant or a significantly larger scaling exponent than the baseline (Appendix § 16.§ 2, Figure 9). We therefore find no evidence that these recipes scale better than the baseline.

Figure 1. Across all regimes, allocating compute to PPT never beats spending it on natural language. (1) Data-constrained: FineWeb validation bits per byte (BPB) against the fraction of a fixed 1×10181 \times10^{18}-FLOP budget spent on PPT, with repetition over 100M natural-language tokens. Optimal model sizes range from 561M to 708M parameters. (2) Over-Chinchilla: an 881M-parameter model trained on 50B tokens, ∼12×\sim12 \times the compute-optimal token count, scored on FineWeb BPB and DCLM CORE. The PPT methods spend 0.5B tokens on PPT and 49.5B on FineWeb. (3) Data-unconstrained: the optimal model size found at each budget varies with the PPT method, from 179M to 186M parameters at 1×10181 \times10^{18} FLOPs, 252M to 336M at 3×10183 \times10^{18}, and 438M to 561M at 9×10189 \times10^{18}.

Figure 9. No PPT method scales better than the baseline. Here we plot FineWeb validation BPB against training compute for the data-unconstrained experiments. Each line is a separate power law, BPB=AC−α\mathrm{BPB}=A C^{-\alpha} [15], fitted by least squares on log BPB against log CC over that method’s nine runs (three seeds per scale), with CC in units of 101810^{18} FLOPs, so AA is the fitted BPB at 101810^{18} FLOPs. The table gives AA and α\alpha with their standard errors. The baseline has the lowest AA, and no method’s exponent is significantly larger than the baseline’s (within standard error). Raising PPT from 1% to 5% raises AA for every method. However, its effect on the exponent is unclear as at 5% every exponent is within standard error of the baseline’s. Every synthetic PPT method remains above the baseline at every budget, while C4, independent of percentage, is the same as the baseline. Since we only have three budgets per method, and three seeds, we do not fit an irreducible-loss term as in Hoffmann et al. [8].

The BPB results are broadly reflected in the downstream DCLM CORE and BLiMP benchmarks, with the exceptions shown in Table 1. At the lowest compute budget we study, DeclRef and code show some improvement over the baseline. However, how significant these gains are remains unclear. PPT on C4, the natural-language control, scores similarly to these synthetic PPT methods on DCLM CORE. In addition, these are the only experiments in which we observe an aggregate significant gain; every other method performs at or below the baseline (see Appendix § 16). However, when breaking the two benchmarks into their individual tasks, we find that some PPT methods do improve on specific sub-tasks, but often at the cost of others, which is why most perform on par on aggregate (Appendix § 16.§ 5, Figure 11). This is especially the case for BLiMP, where some phenomena improve under PPT, in line with the BLiMP gains Hu et al. [9] report for kk-Dyck.

Figure 11. Breakdown of downstream performance by benchmark sub-tasks. Each row is a PPT model from Figure 1 and Figure 4, and each column is one of the 13 BLiMP or 22 DCLM CORE tasks. Blue cells are significantly above the baseline, red cells significantly below, and white cells on par. Most cells are white, so on most sub-scores the PPT models perform on par with their baseline. The colored cells show that, even so, the PPT method changes how the model performs on some sub-tasks of these benchmarks.

We also observe no consistent ranking among PPT methods across configurations or scales. For example, DeclRef is marginally better than NCA and kk-Dyck in the data-constrained regime, but in the data-unconstrained regime the ordering varies with both the PPT percentage and the total compute budget. This is accentuated in the over-Chinchilla regime: with large token budgets and parameter counts, final validation loss between the baseline and PPT methods becomes very similar.

Retention Across the PPT-to-PT Transition

One reason PPT may fail to improve over the baseline is that what it learns does not carry over into PT. The shift from PPT to PT is abrupt: the PPT data differs substantially from natural language, and is seen only at the start of training and never again. Ibrahim et al. [11] show that even a shift between two natural languages causes forgetting. Their work shows that continued pretraining on German, after 300B tokens of English, causes forgetting on the original English distribution. The shift from a synthetic distribution to natural language is even larger.

We test this hypothesis by tracking the validation BPB of the NCA, kk-Dyck and code PPT distributions across the phase boundary, at 2.7×10192.7 \times10^{19} FLOPs, a larger budget than the ones run in previous sections. This gives us a longer horizon over which to observe performance on the PPT distribution. The three right-hand panels in Figure 5 show the result: during the PPT phase all three methods learn their respective distributions, but at the PPT-to-PT boundary the BPB immediately starts increasing, signaling that the models forget their ability on the PPT distribution. The same figure suggests that this stems from front-loading the data in the PPT phase. Holding 5% of every batch as PPT data throughout training instead improves performance on the PPT distribution over the entire run. This is despite both schedules spending the same total compute on the PPT distribution. Forgetting at the boundary is therefore preventable by simply mixing in the synthetic distribution throughout PT. Interleaving improves FineWeb BPB for the PPT methods, though not significantly for all, and as these runs are not hyperparameter-tuned at this scale, how large the gain could be remains open.

Figure 5. Uniformly distributing the PPT data across training, rather than front-loading it, prevents forgetting on the PPT distribution. In the interleaved runs, 5% of every batch consists of NCA, kk-Dyck, or code throughout training. In the PPT runs, the same 5% of compute is instead spent up front, at 2.7×10192.7 \times10^{19} FLOPs. Left: FineWeb validation BPB at the end of training. Right: validation BPB on the PPT corpus itself across the whole run, one panel per corpus on its own scale, with the gray dotted line marking 5% of total compute. Under the PPT schedule the models lose this ability at the PPT-to-PT boundary; under interleaving they keep it and continue to improve on it. Elsewhere in this work the vocabulary is replaced at the phase boundary for the synthetic PPT distributions, which means the PPT corpus cannot be scored after the PPT phase. Every run here instead shares a single vocabulary, extended to hold the NCA and kk-Dyck tokens, so the corpus can be tracked throughout training. These runs are not hyperparameter-tuned (configurations in Appendix § 15.§ 5, Table 7), so the FineWeb panel is included only to show that the interleaved runs sit in a similar range to their PPT counterparts.

RunddηPT\eta_{\mathrm{PT}}PT warmdownηPPT\eta_{\mathrm{PPT}}PPT warmdownPPT batch
NCA 5%, interleaved241.5e-20.5–––
kk-Dyck 5%, interleaved241.5e-20.5–––
Code 5%, interleaved241.5e-20.5–––
NCA 5%, PPT phase243e-20.46e-50.4256k
kk-Dyck 5%, PPT phase213e-20.43e-50.4512k
Code 5%, PPT phase241.5e-20.5–––

Table 7. Retention experiment: configurations. The runs of Figure 5 at 2.7×10192.7 \times10^{19} FLOPs. The PPT runs use each method’s 5% configuration, extrapolated from the smaller budgets, with a shared vocabulary. The interleaved runs train under one schedule, so they have no PPT-phase entries, and all three share the code 5% configuration so that only the interleaved corpus differs.

Limitations

Hyperparameter search at larger scales. We could not run a full hyperparameter search above 9×10189 \times10^{18} FLOPs, where each trial becomes expensive; the over-Chinchilla runs, for instance, sweep only the peak learning rates and use other hyperparameters from the smaller budgets (Appendix § 15.§ 6). However, the baseline and PPT methods receive the same reduced sweep, so the comparison between them remains matched. Similarly, the share of compute given to PPT is itself a hyperparameter, which we fix at 1% and 5% in the data-unconstrained setting rather than tune, leaving shares in between, or ones that change with the total budget, untested.

Effect sizes relative to prior work. The differences we report between PPT and the baseline are small, and in some settings smaller than those previously reported. One reason is that we use NanoChat, which uses a different tokenizer, model architecture, and natural-language dataset than prior work. The tokenizer alone has been shown to change the effect of PPT (Riazhskykh et al., 2026). However, within our own experiments, the setup is held fixed across the baseline and the PPT methods, so any difference between them is attributable to PPT rather than to these details. Another reason for discrepancies with prior gains is the metric we report. In order to make our results easily comparable for future work, we evaluate using validation BPB. Prior work often reports results in perplexity, which is the exponential of the loss and hence yields larger relative differences than BPB. Hence, our BPB differences are not directly comparable to the perplexity ones shown in prior work.

Discussion

We set out to test whether PPT improves PT efficiency. Across three synthetic corpora and multiple training regimes, allocating compute to PPT never outperforms spending it on natural language. This holds even under a multi-epoch setting over 100M tokens. However, in the additive setting at the smallest natural-language budgets, a benefit persists even against a well-initialized baseline. Prior studies hold the initialization scheme fixed while varying the curriculum, a natural design choice for isolating the curriculum effect. Our results suggest that future robustness checks should additionally vary this axis for both PPT and no-PPT methods.

Previous work argues that the priors learned during PPT persist through to the PT phase [9, 20]. In our experiments, however, models do not retain their ability on the PPT distribution after the transition (§ 6). If such priors are indeed kept after PT, we would expect the model to keep at least part of this ability. One could argue that losing this ability is expected, and that it may not matter how well the model does on the PPT distribution, as long as the priors survive. Once the PPT data is no longer present in training, the model no longer needs to assign probability to the PPT tokens, so it is likely that the performance on the PPT distribution degrades. However, if this were the sole reason for the drop in performance, degradation on the code distribution would not occur. Unlike the synthetic distributions, most tokens seen during the code PPT phase are also seen during the PT phase, since comments and variable names carry natural-language semantics. Hence, the model keeps learning to predict them during PT, albeit outside of code contexts. Yet, even in this case, loss on the code distribution rises after the phase boundary. This suggests that the priors learned on code-specific data are not fully kept, and are instead progressively overwritten once the distribution shifts abruptly to natural language. We therefore recommend that future work on synthetic data schedules also measure the model’s ability on the synthetic distribution throughout training, as a proxy for whether the inductive biases learned from it are retained.

A natural question is whether PPT could pay off at scales beyond those tested in this work. Though our largest models stay below one billion parameters, our over-Chinchilla experiments train an 881M-parameter model on 50B tokens, a compute budget comparable to or larger than experiments in prior PPT work [9, 20] (Yamaguchi et al. 2026). Even so, we show that neither NCA nor kk-Dyck improves on the baseline at this scale. We also approach the question from a scaling-law perspective, which motivates our data-unconstrained experiments. Such a perspective has not been explicitly analyzed in prior work. There, we search for each method’s compute-optimal recipe at three budgets and fit a power law to each, and find no evidence that any PPT method scales better than the baseline (§ 5, Appendix § 16.§ 2). Taken together, these results suggest that these PPT recipes would not overtake natural-language training at larger scales.

Next, our main conclusions concern natural-language validation loss. On downstream benchmarks, where prior work highlights gains [9, 20] (Yamaguchi et al. 2026), for example on specific BLiMP phenomena [9] and on reasoning benchmarks after fine-tuning [20], the picture is less clear. In our experiments, aggregate BLiMP and DCLM CORE scores mostly match the baseline. However, exceptions exist on individual sub-tasks, including BLiMP phenomena, which shift in both directions, though without a clear pattern across methods or settings (Figure 11). Hence, PPT may still shape narrower capabilities that aggregate benchmarks do not capture, which more targeted evaluations could bring out. Where it does, integrating synthetic data gradually rather than front-loading it could help better retain these capabilities. PPT could similarly prove more useful in other data modalities, which our study, limited to language modeling, does not examine.

Moreover, our findings leave open questions for future work. As natural-language data is increasingly constrained relative to available compute [19], synthetic data is a compelling prospect. That said, our results motivate rethinking whether such data should be integrated via an explicit PPT phase. For such a phase to be worthwhile, the forgetting across the phase boundary must first be solved. Alternatively, this forgetting can be avoided by interleaving synthetic data throughout training. We therefore recommend that future work investigate better interleaving schedules. However, whether such schedules make synthetic data a better use of compute than natural language remains an open question.

Acknowledgements

We thank Dan Lee and Seungwook Han, authors of Lee et al. [20], and Michael Hu, author of Hu et al. [9], for helpful discussions and advice on this work. Specifically, Dan Lee and Seungwook Han suggested comparing AdamW with Muon for the PPT phase in our additive experiments, which improved the PPT methods (Figure 8).

Reproducibility Statement

Links to our code are given in Appendix § 17. The repositories contain the full training factory, the data-preparation and generation scripts for every corpus, and the evaluation code, together with a manifest holding one record per experiment we report.

AI Use Statement

In this work, we used generative AI tools to help write the code in the repositories we release (Appendix § 17), find related work, and polish the writing of this paper, all under close supervision by us, who reviewed and edited all AI-assisted output. In addition, our hyperparameter optimizer, LLMTuna, uses a language model to propose hyperparameter configurations within a purpose-built search loop and not simply a coding agent (Appendix § 15.§ 1). We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Data

k-Dyck

We generate kk-Shuffle Dyck with the code released by Hu et al. [9], at their published settings. At each position the generator opens a new bracket with probability 0.5 or closes one of the currently open brackets, chosen uniformly among the open types, forcing a close once the maximum nesting depth is reached and an open at depth zero. We use the variant and setting they report as best, k=128k = 128 bracket types, which gives a vocabulary of 2k=2562k = 256, with a maximum nesting depth of 16 and sequences of 2048 tokens.

Neural cellular automata

We generate the NCA corpus with the code released by Lee et al. [20]. The automaton is a 12×1212 \times12 grid of cells, each holding one of ten colors, and the update rule is a randomly initialized convolutional network applied identically to every cell. A rule and an initial grid are sampled for every sequence, and a rule is kept only when the gzip compression ratio of its rollout exceeds 50%, which selects for dynamics that are more complex rather than trivially predictable (low entropy).

Grids are not tokenized cell by cell. Each grid is serialized in row-major order as non-overlapping 2×22 \times2 patches, so its 144 cells become 36 tokens, and since a patch holds four cells over ten colors the patch vocabulary is 10410^4 token IDs. A start and an end token delimit each grid, so one grid costs 38 tokens, and successive grids are concatenated into a single sequence tracing the automaton’s evolution.

The context length therefore fixes how many grids a sequence holds. Lee et al. [20] train at 1024 tokens, which is 27 grids (27×38=102627 \times38 = 1026, with the tail truncated), whereas we train at 2048, which is 54 grids (54×38=205254 \times38 = 2052, again with its tail truncated). Relative to a sequence length of 1024, each of our sequences shows twice as much of the automaton’s evolution as a result. This does not apply to the NCA reproduction of Appendix B.

Declarative-reference data

Prior work uses procedural tasks such as copying and retrieval as PPT data [12]. This “declarative-reference” corpus follows that idea, aiming at a distribution with little semantics that forces context recall, but at a higher entropy than a pure copying task.

In this synthetic corpus, each training document is a sequence of statements, built from four keywords:

  • declare declares a new Name with its Value.

  • reference repeats a Name that is already in scope (i.e., that has been declared), followed by its Value.

  • open starts a new Scope, and close ends it. A Scope is the part of a document between a matching open and close. Scopes can be nested, like balanced brackets in kk-Dyck. Names declared inside a Scope are available until that Scope is closed; Names declared in enclosing Scopes remain available.

Here are the first sixteen tokens of a training document, with token IDs above and a possible decoding below. The choice of letters is arbitrary and only makes the example more readable: lowercase letters stand for Name tokens, uppercase for Value tokens.

token IDs2163873102797723053837374773031638731027977
decodeddeclarerQSTdeclarenbmUVreferencerQST

Table 14.

Reading it as three statements:

  1. declare r Q S T. At the start of a document nothing has been declared yet, so a reference is impossible and the generator emits a declare. It then draws a random Name, here the single token rr, and a random Value, here the three tokens QQ SS TT. The length of each is itself drawn at random, from one to three tokens, so statements vary in length.

  2. declare n b m U V. A Name is now in scope, so this time the generator could have emitted a reference. The keyword is chosen at random from those available, and here it is a declare again, giving the Name nn bb mm and the Value UU VV.

  3. reference r Q S T. This time the keyword drawn is a reference. A second random choice picks a Name from those in scope, either rr or nn bb mm, and here picks rr. Once both are made, the most likely continuation is the Value QQ SS TT associated with rr, so a model predicts these three tokens correctly by recalling that pairing from the context.

The Names and Values occupy disjoint blocks of the vocabulary, IDs [4,516)[4, 516) and [516,1028)[516, 1028), and the four keywords occupy IDs 0 to 3, so the letter case in the decoding stands in for a split that is real in the data. No open or close appears in this example window, although both occur elsewhere in the document. When a Scope is closed, Names declared inside it are no longer available for reference; Names declared in enclosing Scopes remain available.

The generator has the following parameters:

  • How often a Scope opens, and how deeply Scopes can nest.

  • How frequently each Name and Value token is drawn. The frequencies follow a Zipf distribution drawn newly for every document.

  • How a reference chooses among the Names currently in scope, which favors recently declared ones.

  • How the mix of declare to reference statements shifts from the start of a document to its end.

  • The fraction of references that pair the Name with a fresh Value instead of repeating the original one, so that no position is perfectly deterministic.

  • How often the Name frequencies are reshuffled part-way through a document: a Name that occurs every few statements early on may, after the reshuffle, become as rare as any other, so frequency statistics inferred from the opening tokens go stale and must be re-estimated from later context.

We choose these parameters with the following goal in mind: that predicting a token should mainly require using the current document rather than memorizing. Name and Value tokens are drawn partly from a fixed table of likely successors shared by every document, as in natural language, where some word sequences are common across all text, and partly at random. The shared part can be learned across documents, while the random part cannot. The random part aims to provide the corpus with entropy. A Value is therefore only partly predictable when it is declared, so a reference must mainly retrieve it from the current document. The Zipf token frequencies are also redrawn for every document, so they must be inferred from the current one. Which Names can be referenced at all depends on the Scopes currently open, so both the answer and the space of possible answers are properties of the local context. The parameter values are provided in the generator code released with this paper.

NCA Reproduction Experiment

We follow one of the configurations described by Lee et al. [20], a 400M-parameter Llama model trained on ∼9\sim9B tokens of OpenWebText, for which the original work reports an 8.6% perplexity improvement over the no-PPT baseline. The exact Llama configuration is not specified for the 400M model, so we estimate it and choose the following: 24 layers, a hidden size of 1024 with 16 attention heads, and a multilayer perceptron intermediate size of 4096. The OpenWebText training and validation splits are not specified either, so we choose these ourselves too. Under their initialization we recover the effect, a 5.4% gain, and we attribute the remaining difference from the reported 8.6% to the choices above. Beyond those choices we make no changes to the released codebase, and the only thing we vary is the weight initialization.

We compare three initializations, shown in Figure 6. The first is that of the original work: the embedding table is drawn from N(0,1)\mathcal{N}(0,1), the transformer body from N(0,0.02)\mathcal{N}(0,0.02), and the language-modeling head keeps PyTorch’s default uniform initialization [26 b], whose standard deviation here is ≈0.02\approx0.02. The second draws both the token embedding table and the language-modeling head from N(0,0.02)\mathcal{N}(0,0.02). The third is the spectral-based initialization from the version of NanoChat we use: linear layers from N(0,σ)\mathcal{N}(0,\sigma) with σ=min⁡(1,dout/din/din)\sigma= \min(1,\sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}}/\sqrt{d_{\mathrm{in}}}), the embedding table from N(0,1)\mathcal{N}(0,1), and the residual output projections and head zeroed. It does not transfer to Llama exactly, as RMSNorm weights have no counterpart and are left at ones.

Every run is initialized under the method being tested, including the PPT phase. NCA therefore trains its PPT phase under that method, and at the transition the new token embedding table and language-modeling head are drawn under the same one, while the body carries over. As shown in Figure 6, the initialization that best suits the baseline is itself architecture-dependent: the spectral-based initialization is the default in the NanoChat version we use, but is not the best-performing of the three here. This provides further motivation to ablate initialization when evaluating PPT.

FLOP Measurement

To convert a FLOP budget into a training configuration, we measure the FLOPs of one forward and backward pass (one optimization step) for the model using the FLOP-counting utilities in PyTorch (PyTorch, 2026c). We then divide the total budget by the FLOPs per step to determine the number of optimization steps and tokens consumed by each phase. Working backwards from a FLOP budget to a configuration is the standard framing of compute-optimal PT [8], and it is what allows a PPT method and its baseline to be compared on equal terms for the compute-matched experiments. We measure FLOPs directly rather than matching token counts for compute matching because the synthetic corpora use much smaller vocabularies than natural language, so the language-modeling head, and with it the FLOPs per token, is smaller during PPT. Given its smaller vocabulary size, the PPT phase is therefore afforded extra steps. However, this difference shrinks as models grow, since the language-modeling head makes up a decreasing share of the total

OpenWebText validation perplexity for the no-PPT baseline and NCA under three initialization methods. The chart compares $\mathcal{N}(0,1)$ embeddings and $\mathcal{N}(0,0.02)$ rest, spectral initialization, and $\mathcal{N}(0,0.02)$ for all parameters; annotations show 5.4% gain, 4.4% gain, and 9.0% gain, with the gap between NCA and the baseline narrowing.Vector (PDF)

caption: The NCA positive control under three baseline weight initializations. A 400M-parameter Llama model trained on 164M NCA tokens, and then ∼9\sim9B tokens of OpenWebText, following Lee et al. [20]. Validation perplexity for the no-PPT baseline and NCA under three initializations, namely that of the original work (N(0,1)\mathcal{N}(0,1) embeddings and N(0,0.02)\mathcal{N}(0,0.02) elsewhere), N(0,0.02)\mathcal{N}(0,0.02) embeddings and head, and the spectral-based initialization of the NanoChat version we use. Under each alternative initialization the baseline improves and the gap between NCA and the baseline narrows.

parameters. For the over-Chinchilla configuration, our largest model, we therefore simply allocate tokens directly, 0.5B to PPT and 49.5B to FineWeb against the baseline’s 50B, so that every method sees the same total number of tokens (Appendix § 15.§ 6).

Hyperparameters

How LLMTuna works

LLMTuna is a language-model-based hyperparameter optimizer that we release with this work. The approach is inspired by prior work that also used language models for hyperparameter optimization and autoresearch more broadly [17] (Yang et al. 2024a; Jiang & Zhao 2026). LLMTuna maintains an ordered history of configurations tried so far and their corresponding scores, which in our experiments are validation BPB. At each step, a language model uses this history together with any additional context provided, such as training code, notes on hyperparameter behavior, or constraints on the search, to propose the next configuration. This gives the model semantic context from the codebase that can inform which hyperparameters are likely to matter. The search space is specified using three parameter types, Float, Int, and Choice, each with a name, a natural-language description, and optional bounds and initial value.

Following the API design of Optuna [1], the optimization loop consists of two calls. suggest provides the accumulated context to the model together with a tool whose schema is generated from the search space, so the proposal is returned as JSON in a fixed shape rather than as prose. observe then records the configuration and its resulting score, adding the trial to the history available to subsequent calls to suggest.

Benchmark against Optuna

We compare LLMTuna against a Tree-structured Parzen Estimator (TPE) search using the Optuna library [1] on one of our experiment configurations: NCA at 1% and a total budget of 1×1081 \times10^{8} FLOPs. Both searches minimize the same objective (validation set BPB) over the same search space and initial hyperparameter configuration.

Optuna is run with the TPE sampler and a budget of 30 trials. The learning rate parameters are sampled in log space for a fair comparison, and the constraint that the warmup and warmdown ratios sum to at most one is enforced by conditional sampling, drawing the warmdown ratio from [0,1−warmup][0, 1-\mathrm{warmup}]. LLMTuna is run under the proposer, context and stopping rule given in Appendix § 15.§ 3 (code repository in Appendix § 17). Results are shown in Figure 7. These results suggest that LLMTuna can find good configurations with relatively few trials under the search constraints considered here.

Figure 7. LLMTuna reaches a lower validation loss in fewer trials than a TPE search. The continuous lines for each method show the best FineWeb validation BPB so far against trial index, while the faint dots are individual trials. LLMTuna reaches 0.9780 at its tenth trial, while TPE reaches 0.9808 at its twenty-second and does not improve over the remaining eight. The axis is broken above 0.992, where the early exploratory trials of both searches sit.

Search configuration

In the additive-compute setting, we run a separate LLMTuna search for each combination of PT natural-language token count, PPT token count, and the three weight-initialization schemes of § 3. In the data-constrained and data-unconstrained settings, a search is run for each combination of budget CC, PPT fraction α\alpha and PPT corpus. Table 2 lists what each search tunes.

SettingRecipeTuned hyperparameters
AdditiveUnifiedPeak matrix learning rate, warmup fraction, warmdown fraction, PT batch size
SeparateThe above, plus PPT peak learning rate, PPT warmup fraction, PPT warmdown fraction, PPT batch size
Compute-matchedUnifiedDepth, peak matrix learning rate, warmup fraction, warmdown fraction
SeparateThe above, plus PPT peak learning rate, PPT warmup fraction, PPT warmdown fraction, PPT batch size

Table 2. Hyperparameters tuned in each setting. Separate is the standard recipe of Section 3: the PPT phase gets its own learning rate, and the optimizer state is reset at the phase boundary. Unified covers the no-PPT baseline, the C4 control, and the unified optimizer code variant, which all run both phases under a single schedule with no reset; for C4, restarting the optimizer at the boundary hurt its final loss. In the additive setting the natural-language token count is fixed, so batch size sets the number of optimizer steps; we therefore tune it there, and fix it at 262,144 tokens per step in the compute-matched settings. The PPT-phase optimizer is not searched; in the additive and data-constrained settings we run one search under each of Muon and AdamW and report the lower-BPB result.

Every search is given a budget of at most 30 trials and stops early once five consecutive proposals fail to improve the best validation BPB by more than 10−310^{-3}. Proposals come from claude-sonnet-4-6 with a thinking budget of 300 tokens and a response cap of 4000, called through a tool interface that forces it to return a value for every tuned hyperparameter. The proposer sees (i) a system prompt naming only the hyperparameters that the study tunes, (ii) a description of each, with its bounds or permitted options, (iii) a summary of the training stack, generated once from ten source files and restricted to what bears on the tuned hyperparameters, and (iv) every trial so far, with its full configuration and validation BPB, in the order the trials were run. In the data-unconstrained setting, each FLOP budget runs its own search, warm-started from the previous budget’s best configuration and carrying that search’s history forward as context, so the proposer can see how an optimum moved as the budget grew.

Proposer prompt

The system prompt for a study tuning all eight hyperparameters begins:

You are a hyperparameter optimizer minimizing val_bpb
(validation bits-per-byte; lower is better).
You see all prior trials in the order they were run.
Balance exploiting the current best config with informative
exploration; the trial budget is small, so every proposal
should either improve the best or teach you something
specific.
Most of the result at a given scale comes from three knobs:
depth, matrix-lr, and ppt-lr. Each has its own optimum, and
each optimum shifts with scale. Getting them near their
minimum for THIS scale matters more than refining the other
knobs, so make sure each has been searched widely enough
that you would notice if it were off.
...

The rest of the prompt covers step size and keeping proposals attributable; the full prompts and hyperparameter definitions are in the released code.

Hyperparameters found

Tables 3–7 list the configuration of every cell the paper reports, one table per setting. Each table lists only the hyperparameters that differ between the cells of its setting; the rest are shared by every run in the paper and are stated here once. All models use a vocabulary of 65,536 tokens (extended to hold the PPT tokens in the shared-vocabulary retention runs) and train at a sequence length of 2048 with a trapezoidal learning-rate schedule (linear warmdown to zero over the stated fraction of the phase, no warmup unless noted), gradient clipping at 1.0 and no weight decay. During natural-language pretraining Muon updates the matrix parameters at the stated peak rate ηPT\eta_{\mathrm{PT}} while the embedding and unembedding tables train under AdamW at 0.3 and 0.004; during a PPT phase all three groups share a single rate ηPPT\eta_{\mathrm{PPT}}. Every value is read from the logged configuration of the three seed replicas, which agree on every field.

Over-Chinchilla configuration

The over-Chinchilla configuration’s 50B tokens are ∼12×\sim12\times the compute-optimal token count for its 881M-parameter model, based on the ≈4.6\approx4.6 tokens per parameter of the baseline’s searched optima at $1 \times 10^{18}$–9×10189 \times10^{18} FLOPs. This is well below the 20 token to parameter ratio reported by Hoffmann et al. [8], as NanoChat’s own fits also find ([18]). This is thanks to differences in data, model architecture, and optimizer. Nonetheless, even by the Chinchilla ratio, the run’s $\approx 57 tokens per parameter is greater than the Chinchilla-optimal ratio. We run the PPT methods at 1% of total compute at this scale, which amounts to 0.5B tokens on PPT and 49.5B on FineWeb.

MethodddηPT\eta_{\mathrm{PT}}PPT optimizerηPPT\eta_{\mathrm{PPT}}PPT batch
Baseline (0% PPT)203e-2–––
NCA 1%203e-2AdamW3e-5512k
NCA 5%203e-2AdamW3e-5512k
NCA 10%222e-2AdamW3e-61M
kk-Dyck 1%203e-2Muon8e-6512k
kk-Dyck 5%203e-2Muon2e-2512k
kk-Dyck 10%223e-2Muon1e-31M
DeclRef 1%203e-2AdamW3e-5256k
DeclRef 5%203e-2AdamW3e-5512k
DeclRef 10%212e-2AdamW1e-6128k

Table 3. Data-constrained setting: reported configurations. 100M unique natural-language tokens at 1×10181 \times10^{18} FLOPs, the experiments of panel (1) of Figure 1, each at the lower-BPB PPT optimizer. Every PPT run warms down over 0.4 of both its PPT and natural-language phases, and the baseline over 0.2 of training.

Methoddd 1×10181 \times10^{18}dd 3×10183 \times10^{18}dd 9×10189 \times10^{18}ηPT\eta_{\mathrm{PT}} 1×10181 \times10^{18}ηPT\eta_{\mathrm{PT}} 3×10183 \times10^{18}ηPT\eta_{\mathrm{PT}} 9×10189 \times10^{18}ηPPT\eta_{\mathrm{PPT}} 1×10181 \times10^{18}ηPPT\eta_{\mathrm{PPT}} 3×10183 \times10^{18}ηPPT\eta_{\mathrm{PPT}} 9×10189 \times10^{18}PT warmdown 1×10181 \times10^{18}PT warmdown 3×10183 \times10^{18}PT warmdown 9×10189 \times10^{18}PPT warmdown 1×10181 \times10^{18}PPT warmdown 3×10183 \times10^{18}PPT warmdown 9×10189 \times10^{18}PPT batch 1×10181 \times10^{18}PPT batch 3×10183 \times10^{18}PPT batch 9×10189 \times10^{18}
Baseline (0% PPT)1216203e-22e-21.5e-2–––0.40.40.5––––––
C4 1%1115193e-21.5e-21.5e-2–––0.40.40.4––––––
C4 5%1215193e-21.5e-21.5e-2–––0.40.40.4––––––
NCA 1%1115192e-22e-22e-26e-32e-21.2e-20.40.40.40.40.40.41M1M1M
NCA 5%1216203e-23e-23e-22e-36e-56e-50.40.40.40.40.40.4256k256k256k
kk-Dyck 1%1116203e-23e-24e-23e-31e-21e-20.40.40.40.40.40.41M1M1M
kk-Dyck 5%1216183e-23e-23e-23e-53e-53e-50.40.40.40.40.40.4512k512k512k
DeclRef 1%1216183e-21.5e-21.5e-21.2e-21.2e-21.2e-20.40.40.40.40.40.4256k256k256k
DeclRef 5%1216202e-21.5e-21.5e-22e-31e-31e-30.40.40.40.40.40.4256k256k256k
Code 1% (unified optimizer)1115183e-23e-21.5e-2–––0.40.50.5––––––
Code 5% (unified optimizer)1216201.5e-21.5e-21.5e-2–––0.40.50.5––––––
Code 1% (separate PPT optimizer)1115183e-23e-22e-21e-43e-43e-40.40.40.40.40.70.71M1M1M
Code 5% (separate PPT optimizer)1214183e-23e-22e-21e-51e-51e-50.40.40.40.40.80.8256k256k512k

Table 4. Data-unconstrained setting: reported configurations. The methods of Table 10 at each budget. Two schedule details have no column: kk-Dyck 1% warms its PPT phase up over the first 0.1 of the phase at every budget, and the unified code 5% method warms up over 0.02 of training at 3×10183 \times10^{18} and 9×10189 \times10^{18}; every other experiment has no warmup.

No LLMTuna search is run at this budget. Training uses no warmup and a linear warmdown over the second half of training, and for the PPT runs the warmdown fraction is set so that decay begins at the same 25B natural-language token mark as in the baseline. We sweep only the peak learning rates, since the optimum moves with the training horizon and this horizon is an order of magnitude longer than any we searched. Every configuration is first run at 0.006, extrapolated from the baseline’s searched optima, and then at three further rates, 0.00748, 0.00563 and 0.00368, obtained by rescaling the peak rate with the training horizon TT as TpT^p for p∈{−0.2,−0.32,−0.5}p \in\{-0.2, -0.32, -0.5\}, a range informed by the power law between optimal learning rate and token horizon reported by Bjorck et al. [4]. Each rate is run at a single seed, and the rate minimizing FineWeb validation BPB is replicated across the remaining seeds: 0.006 for the baseline and kk-Dyck, 0.00748 for NCA.

Full results

This appendix collects the complete results for each setting in turn. For our reported BLiMP results, we score BLiMP following the released evaluation code of Hu et al. [9]. Each of the 67,000 minimal pairs is scored by total sentence log-probability, and a pair is counted correct only if the grammatical sentence scores strictly higher than the ungrammatical one, so ties count as errors. No beginning-of-sequence token is prepended, which means the first token of each sentence is unscored. DCLM CORE is evaluated with the NanoChat evaluation code (Karpathy, [16]).

MethodηPT\eta_{\mathrm{PT}}PT warmdownηPPT\eta_{\mathrm{PPT}}PPT warmupPPT warmdownPPT batch
Baseline (0% PPT)6e-30.5––––
NCA 1%7.5e-30.4951.3e-300.61M
kk-Dyck 1%6e-30.4951e-20.10.41M

Table 6. Over-Chinchilla setting: configurations. Depth 24 (881M parameters) on 50B tokens, the methods of panel (2) of Figure 1. Hyperparameter search details are given in Appendix § 15.§ 6.

Additive setting

Figure 8 shows the full results behind Figure 4, and Table 8 and Table 9 report BLiMP and DCLM CORE accuracy for the same runs, with each curriculum at its lowest-loss PPT token count.

InitializationMethod50M100M1B
Spectral initBaseline0.691±0.0050.691 \pm0.0050.719±0.0010.719 \pm0.0010.774±0.0050.774 \pm0.005
NCA0.691±0.0020.691 \pm0.0020.719±0.0080.719 \pm0.0080.769±0.0070.769 \pm0.007
kk-Dyck0.688±0.0060.688 \pm0.0060.723±0.0070.723 \pm0.0070.778±0.0070.778 \pm0.007
N(0,1)N(0,1) embeddingsBaseline0.687±0.0030.687 \pm0.0030.717±0.0040.717 \pm0.0040.779±0.0030.779 \pm0.003
NCA0.683±0.0030.683 \pm0.0030.717±0.0070.717 \pm0.0070.776±0.0020.776 \pm0.002
kk-Dyck0.695±0.0070.695 \pm0.0070.718±0.0110.718 \pm0.0110.775±0.0010.775 \pm0.001
N(0,0.02)N(0,0.02) allBaseline0.695±0.0050.695 \pm0.0050.720±0.0060.720 \pm0.0060.779±0.0050.779 \pm0.005
NCA0.691±0.0080.691 \pm0.0080.724±0.0060.724 \pm0.0060.774±0.0040.774 \pm0.004
kk-Dyck0.690±0.0050.690 \pm0.0050.722±0.0110.722 \pm0.0110.783±0.0050.783 \pm0.005

Table 8. BLiMP in the additive setting. BLiMP accuracy for the additive-compute experiments of Figure 4: for each natural-language budget and initialization, the baseline and each curriculum at its best (lowest-loss) PPT token count (Table 5 lists the counts). Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above that of the baseline with the same initialization (none are bolded here as all perform similarly to the baseline).

NL tokensMethodInitPPT tokensPT batchPPT optimizerηPPT\eta_{\mathrm{PPT}}PPT warmdownPPT batch
50MBaselineN(0,1)\mathcal{N}(0, 1) emb.–64k––––
50MBaselinespectral–64k––––
50MNCAN(0,0.02)\mathcal{N}(0, 0.02)500M64kMuon4e-31256k
50Mkk-Dyckspectral20M64kAdamW1.2e-4164k
100MBaselineN(0,1)\mathcal{N}(0, 1) emb.–64k––––
100MBaselinespectral–64k––––
100MNCAN(0,0.02)\mathcal{N}(0, 0.02)500M64kMuon6e-31512k
100Mkk-Dyckspectral1B64kAdamW5e-51512k
1BBaselineN(0,1)\mathcal{N}(0, 1) emb.–256k––––
1BBaselinespectral–256k––––
1BNCAspectral100M256kMuon6e-61256k
1Bkk-Dyckspectral5M256kMuon6e-60.4256k

Table 5. Additive setting: reported configurations. The twelve points of Figure 4: per natural-language (NL) budget, the two baselines the figure draws and each curriculum at its lowest-loss initialization, PPT token count and PPT optimizer. Every experiment settled on ηPT=0.02\eta_{\mathrm{PT}} = 0.02 with a warmdown of 0.4, and the PPT phase runs with no warmup.

InitializationMethod50M100M1B
Spectral initBaseline0.024±0.0040.024 \pm0.0040.042±0.0110.042 \pm0.0110.110±0.0090.110 \pm0.009
NCA0.032±0.0010.032 \pm0.0010.046±0.0060.046 \pm0.0060.120±0.0020.120 \pm0.002
kk-Dyck0.032±0.0030.032 \pm0.0030.046±0.0120.046 \pm0.0120.116±0.0030.116 \pm0.003
N(0,1)N(0,1) embeddingsBaseline0.030±0.0020.030 \pm0.0020.040±0.0070.040 \pm0.0070.107±0.0100.107 \pm0.010
NCA0.035±0.0100.035 \pm0.0100.045±0.0090.045 \pm0.0090.117±0.0140.117 \pm0.014
kk-Dyck0.030±0.0050.030 \pm0.0050.044±0.0060.044 \pm0.0060.115±0.0150.115 \pm0.015
N(0,0.02)N(0,0.02) allBaseline0.031±0.0070.031 \pm0.0070.049±0.0080.049 \pm0.0080.121±0.0030.121 \pm0.003
NCA0.027±0.0060.027 \pm0.0060.047±0.0120.047 \pm0.0120.116±0.0060.116 \pm0.006
kk-Dyck0.037±0.0120.037 \pm0.0120.044±0.0040.044 \pm0.0040.117±0.0120.117 \pm0.012

Table 9. DCLM CORE in the additive setting. Centered CORE accuracy for the same experiments as Table 8: for each natural-language budget and initialization, the baseline and each curriculum at its best (lowest-loss) PPT token count (Table 5 lists the counts). Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above that of the baseline with the same initialization.

Data-unconstrained setting

Panel (3) of Figure 1 shows that, at each of the three budgets we ran, spending compute on synthetic PPT does not beat spending it on natural language. A natural follow-up is whether any method improves faster with compute, and so might overtake the baseline at budgets beyond ours. Figure 9 addresses this by fitting a power law to each method’s FineWeb BPB across the same runs. The baseline starts lowest, and no PPT method has a significantly larger scaling exponent, so we find no clear evidence that the gaps in panel (3) close with scale. Table 10 gives the FineWeb BPB behind both figures. To see how these results carry over to other evaluations, Tables 11 and Table 12 report BPB on the held-out C4 and GitHub-code splits for the same checkpoints, and Table 13 and 14 report BLiMP and DCLM CORE accuracy.

Method1×10181 \times10^{18}3×10183 \times10^{18}9×10189 \times10^{18}
Baseline (0% PPT)0.9794±0.00030.9794 \pm0.00030.9234±0.00020.9234 \pm0.00020.8733±0.00020.8733 \pm0.0002
C4 1%0.9799±0.00060.9799 \pm0.00060.9232±0.00050.9232 \pm0.00050.8729±0.00040.8729 \pm0.0004
C4 5%0.9804±0.00040.9804 \pm0.00040.9238±0.00030.9238 \pm0.00030.8735±0.00050.8735 \pm0.0005
NCA 1%0.9803±0.00050.9803 \pm0.00050.9239±0.00010.9239 \pm0.00010.8751±0.00240.8751 \pm0.0024
NCA 5%0.9830±0.00020.9830 \pm0.00020.9272±0.00020.9272 \pm0.00020.8765±0.00020.8765 \pm0.0002
k-Dyck 1%0.9802±0.00020.9802 \pm0.00020.9248±0.00030.9248 \pm0.00030.8765±0.00210.8765 \pm0.0021
k-Dyck 5%0.9828±0.00070.9828 \pm0.00070.9270±0.00010.9270 \pm0.00010.8763±0.00030.8763 \pm0.0003
DeclRef 1%0.9804±0.00040.9804 \pm0.00040.9245±0.00050.9245 \pm0.00050.8772±0.00040.8772 \pm0.0004
DeclRef 5%0.9827±0.00050.9827 \pm0.00050.9265±0.00010.9265 \pm0.00010.8758±0.00030.8758 \pm0.0003
Code 1% (unified optimizer)0.9831±0.00030.9831 \pm0.00030.9266±0.00020.9266 \pm0.00020.8790±0.00560.8790 \pm0.0056
Code 5% (unified optimizer)0.9869±0.00030.9869 \pm0.00030.9305±0.00020.9305 \pm0.00020.8817±0.00160.8817 \pm0.0016
Code 1% (separate PPT optimizer)0.9800±0.00040.9800 \pm0.00040.9242±0.00020.9242 \pm0.00020.8740±0.00010.8740 \pm0.0001
Code 5% (separate PPT optimizer)0.9822±0.00030.9822 \pm0.00030.9270±0.00040.9270 \pm0.00040.8753±0.00050.8753 \pm0.0005

Table 10. Allocating a larger share of a fixed compute budget to synthetic PPT generally increases loss. FineWeb bits-per-byte. Hyperparameters were searched independently for each method at every budget. Code appears under both the separate and the unified optimizer recipe. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Method1×10181 \times10^{18}3×10183 \times10^{18}9×10189 \times10^{18}
Baseline (0% PPT)0.761 ± 0.0040.780 ± 0.0030.794 ± 0.004
C4 1%0.756 ± 0.0070.779 ± 0.0040.791 ± 0.009
C4 5%0.757 ± 0.0090.780 ± 0.0050.794 ± 0.008
NCA 1%0.753 ± 0.0060.781 ± 0.0070.792 ± 0.002
NCA 5%0.759 ± 0.0060.773 ± 0.0100.796 ± 0.002
kk-Dyck 1%0.755 ± 0.0070.782 ± 0.0060.798 ± 0.002
kk-Dyck 5%0.762 ± 0.0060.783 ± 0.0010.797 ± 0.002
DeclRef 1%0.758 ± 0.0130.779 ± 0.0100.791 ± 0.009
DeclRef 5%0.754 ± 0.0020.778 ± 0.0030.797 ± 0.008
Code 1% (unified optimizer)0.753 ± 0.0040.782 ± 0.0020.789 ± 0.007
Code 5% (unified optimizer)0.762 ± 0.0090.784 ± 0.0040.793 ± 0.003
Code 1% (separate PPT optimizer)0.758 ± 0.0070.783 ± 0.0070.796 ± 0.006
Code 5% (separate PPT optimizer)0.755 ± 0.0090.780 ± 0.0050.789 ± 0.004

Table 12. The data-unconstrained comparison on the held-out GitHub-code split. GitHub-code bits-per-byte for the methods of Table 10, same checkpoints, same protocol. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Method1×10181 \times10^{18}3×10183 \times10^{18}9×10189 \times10^{18}
Baseline (0% PPT)0.088 ± 0.0060.121 ± 0.0150.165 ± 0.008
C4 1%0.106 ± 0.0040.131 ± 0.0040.156 ± 0.004
C4 5%0.101 ± 0.0070.137 ± 0.0050.154 ± 0.007
NCA 1%0.092 ± 0.0050.127 ± 0.0110.162 ± 0.007
NCA 5%0.082 ± 0.0030.133 ± 0.0040.155 ± 0.012
kk-Dyck 1%0.096 ± 0.0120.135 ± 0.0100.153 ± 0.006
kk-Dyck 5%0.098 ± 0.0040.135 ± 0.0070.153 ± 0.007
DeclRef 1%0.097 ± 0.0020.130 ± 0.0140.152 ± 0.005
DeclRef 5%0.108 ± 0.0110.125 ± 0.0080.159 ± 0.005
Code 1% (unified optimizer)0.098 ± 0.0050.135 ± 0.0020.159 ± 0.005
Code 5% (unified optimizer)0.103 ± 0.0050.130 ± 0.0080.158 ± 0.007
Code 1% (separate PPT optimizer)0.104 ± 0.0040.132 ± 0.0100.159 ± 0.006
Code 5% (separate PPT optimizer)0.096 ± 0.0080.127 ± 0.0120.156 ± 0.002

Table 13. BLiMP across the data-unconstrained sweep. BLiMP accuracy for the data-unconstrained sweep; higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s at that budget.

Method1×10181 \times10^{18}3×10183 \times10^{18}9×10189 \times10^{18}
Baseline (0% PPT)1.1461±0.00021.1461 \pm0.00021.0900±0.00021.0900 \pm0.00021.0409±0.00041.0409 \pm0.0004
C4 1%1.1465±0.00071.1465 \pm0.00071.0898±0.00071.0898 \pm0.00071.0399±0.00051.0399 \pm0.0005
C4 5%1.1456±0.00041.1456 \pm0.00041.0885±0.00041.0885 \pm0.00041.0382±0.00081.0382 \pm0.0008
NCA 1%1.1471±0.00061.1471 \pm0.00061.0905±0.00031.0905 \pm0.00031.0424±0.00271.0424 \pm0.0027
NCA 5%1.1497±0.00031.1497 \pm0.00031.0937±0.00031.0937 \pm0.00031.0433±0.00021.0433 \pm0.0002
k-Dyck 1%1.1470±0.00021.1470 \pm0.00021.0914±0.00031.0914 \pm0.00031.0436±0.00211.0436 \pm0.0021
k-Dyck 5%1.1493±0.00061.1493 \pm0.00061.0935±0.00011.0935 \pm0.00011.0443±0.00051.0443 \pm0.0005
DeclRef 1%1.1472±0.00061.1472 \pm0.00061.0912±0.00061.0912 \pm0.00061.0446±0.00051.0446 \pm0.0005
DeclRef 5%1.1493±0.00071.1493 \pm0.00071.0931±0.00011.0931 \pm0.00011.0428±0.00051.0428 \pm0.0005
Code 1% (unified optimizer)1.1500±0.00051.1500 \pm0.00051.0935±0.00031.0935 \pm0.00031.0468±0.00541.0468 \pm0.0054
Code 5% (unified optimizer)1.1537±0.00031.1537 \pm0.00031.0973±0.00021.0973 \pm0.00021.0487±0.00171.0487 \pm0.0017
Code 1% (separate PPT optimizer)1.1468±0.00041.1468 \pm0.00041.0907±0.00021.0907 \pm0.00021.0420±0.00051.0420 \pm0.0005
Code 5% (separate PPT optimizer)1.1488±0.00041.1488 \pm0.00041.0938±0.00021.0938 \pm0.00021.0432±0.00061.0432 \pm0.0006

Table 11. The data-unconstrained comparison on the held-out C4 split. C4 bits-per-byte for the methods of Table 10, same checkpoints, same protocol. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Data-constrained setting

Panel (1) of Figure 1 reports, for each curriculum and PPT percentage, the best configuration found by the search. Because we run one search under each PPT-phase optimizer and report the lower-

BPB result, that panel selects over the optimizer as well as over depth and learning rate. Figure 10 separates the two optimizers, and the benchmark scores for the same experiments follow.

Figure 10. Data-constrained results separated by PPT-phase optimizer. The experiments behind panel (1) of Figure 1, with Muon PPT drawn as solid lines with filled markers and AdamW PPT as dashed lines with open markers. Each experiment was searched independently under the same search space and stopping rule, so the two differ in the optimizer used during PPT and in the configuration each search found. Neither optimizer is always better than the other: AdamW is lower for NCA and DeclRef at every PPT percentage, while kk-Dyck is lower under Muon. No experiment of either optimizer reaches the no-PPT baseline.

Method1×10181 \times10^{18}3×10183 \times10^{18}9×10189 \times10^{18}
Baseline (0% PPT)1.9435±0.02601.9435 \pm0.02601.7078±0.07851.7078 \pm0.07851.6702±0.03471.6702 \pm0.0347
C4 1%1.9738±0.04981.9738 \pm0.04981.7890±0.05281.7890 \pm0.05281.6590±0.02861.6590 \pm0.0286
C4 5%1.9012±0.01321.9012 \pm0.01321.7914±0.07321.7914 \pm0.07321.6832±0.02171.6832 \pm0.0217
NCA 1%1.9356±0.08831.9356 \pm0.08831.7793±0.04691.7793 \pm0.04691.7353±0.02951.7353 \pm0.0295
NCA 5%1.9640±0.06031.9640 \pm0.06031.8043±0.08621.8043 \pm0.08621.7272±0.02271.7272 \pm0.0227
kk-Dyck 1%1.9867±0.09291.9867 \pm0.09291.7675±0.05801.7675 \pm0.05801.8248±0.13791.8248 \pm0.1379
kk-Dyck 5%1.9536±0.02811.9536 \pm0.02811.8378±0.10981.8378 \pm0.10981.8205±0.11921.8205 \pm0.1192
DeclRef 1%1.9933±0.06701.9933 \pm0.06701.7080±0.10511.7080 \pm0.10511.7438±0.05281.7438 \pm0.0528
DeclRef 5%1.9250±0.01471.9250 \pm0.01471.7385±0.06281.7385 \pm0.06281.7206±0.05451.7206 \pm0.0545
Code 1% (unified optimizer)1.9601±0.08341.9601 \pm0.08341.8063±0.11561.8063 \pm0.11561.7206±0.03461.7206 \pm0.0346
Code 5% (unified optimizer)1.8892±0.03611.8892 \pm0.03611.7491±0.03041.7491 \pm0.03041.6568±0.05851.6568 \pm0.0585
Code 1% (separate PPT optimizer)1.9697±0.08911.9697 \pm0.08911.7372±0.05101.7372 \pm0.05101.6882±0.03741.6882 \pm0.0374
Code 5% (separate PPT optimizer)1.9426±0.03901.9426 \pm0.03901.7975±0.00991.7975 \pm0.00991.6924±0.06611.6924 \pm0.0661

Table 11 (PDF p. 26).

tbl:14: DCLM CORE across the data-unconstrained sweep. Centered CORE accuracy; higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s.

MethodBLiMPDCLM CORE
Baseline (0% PPT)0.721 ± 0.0040.068 ± 0.012
NCA 1%0.718 ± 0.0030.077 ± 0.007
NCA 5%0.722 ± 0.0010.074 ± 0.006
NCA 10%0.730 ± 0.0060.069 ± 0.002
kk-Dyck 1%0.714 ± 0.0070.068 ± 0.009
kk-Dyck 5%0.724 ± 0.0010.059 ± 0.010
kk-Dyck 10%0.728 ± 0.0060.060 ± 0.008
DeclRef 1%0.727 ± 0.0030.068 ± 0.007
DeclRef 5%0.720 ± 0.0070.070 ± 0.009
DeclRef 10%0.731 ± 0.0050.070 ± 0.004

Table 15. BLiMP and DCLM CORE in the data-constrained regime. 100M unique natural-language tokens at a fixed 1×10181 \times10^{18}-FLOP budget, the experiments of panel (1) of Figure 1, each at the lower-BPB PPT optimizer. Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s.

Over-Chinchilla setting

Benchmark scores for the 881M-parameter runs on 50B tokens.

MethodBLiMPDCLM CORE
Baseline (0% PPT)0.809 ± 0.0060.263 ± 0.008
NCA 1%0.812 ± 0.0050.264 ± 0.009
kk-Dyck 1%0.809 ± 0.0020.262 ± 0.009

Table 16. Over-Chinchilla benchmark summary. BLiMP and DCLM CORE for the over-Chinchilla regime: 881M parameters on 50B tokens, NCA and kk-Dyck at 1%. Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s. No method separates from the baseline on either benchmark.

Downstream benchmarks

The DCLM CORE suite from Li et al. [21] aggregates 22 tasks: AGIEval LSAT-AR, ARC-Easy, ARC-Challenge, six BIG-bench tasks (QA Wikidata, Dyck languages, Operators, Repeat Copy Logic, CS Algorithms, and Language Identification), BoolQ, CommonsenseQA, COPA, CoQA, HellaSwag (zero-shot and 10-shot), Jeopardy, LAMBADA, OpenBookQA, PIQA, SQuAD, the Winograd Schema Challenge, and Winogrande. Please refer to the original paper for more details on each benchmark.

Code release

The training and evaluation code is available at rethinking-ppt, and LLMTuna at llmtuna-hpo.

References

  1. [1]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2623–2631, 2019.arxiv.org/abs/1907.10902
  2. [2]Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code or not to code? exploring impact of code in pre-training. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=zSFeN1uACx.arxiv.org/abs/2408.10914
  3. [3]Xiao Bi, Deli Chen, Guanting Chen, et al. DeepSeek LLM: Scaling open-source language models with longertexts, 2024. URL https://arxiv.org/abs/2401.02954.
  4. [4]Johan Bjorck, Alon Benhaim, Vishray Chaudhary, Furu Wei, and Xia Song. Scaling optimal LR across token horizons. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=WYL4eFLcxG.arxiv.org/abs/2409.19913
  5. [5]CodeParrot. GitHub Code dataset. https://huggingface.co/datasets/codeparrot/github-code, 2022.
  6. [6]Evelina Fedorenko, Anna Ivanova, and Tamar Regev. The language network as a natural kind within the broader landscape of the human brain. Nature Reviews Neuroscience, 25:289–312, 04 2024. doi: 10.1038/s41583-024-00802-4.DOI
  7. [7]Xu Guo, Runyu Peng, Jian Tong, Yunhua Zhou, Haijun Lv, Zhihui Lu, and Qipeng Guo. Synthetic pre-training improves language model robustness to noisy pre-training data, 2026. URL https://arxiv.org/abs/2605.10129.
  8. [8]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther­ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen­nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2203.15556
  9. [9]Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. Between circuits and Chomsky: Pre-pretraining on formal languages imparts linguistic biases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. arXiv:2502.19249.arxiv.org/abs/2502.19249
  10. [10]Hugging Face. transformers: Llama configuration and modeling (v4.40.0). https://github.com/huggingface/transformers/blob/v4.40.0/src/transformers/models/llama/configuration_llama.py, 2024. initializer_range=0.02, applied as N(0, 0.02).
  11. [11]Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2403.08763
  12. [12]Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, and Damien Teney. Procedural pretraining: Warming up language models with abstract data. In International Conference on Machine Learning (ICML), 2026. arXiv:2601.21725.arxiv.org/abs/2601.21725
  13. [14]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024.
  14. [15]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361.
  15. [16]Andre­j Karpathy. nanochat: The best ChatGPT that $100 can buy, 2025. URL https://github.com/karpathy/nanochat.
  16. [17]Andre­j Karpathy. AutoResearch: AI agents running research on single-GPU nanochat training automatically. https://github.com/karpathy/autoresearch, 2026a. URL https://github.com/karpathy/autoresearch.DOI
  17. [18]Andre­j Karpathy. nanochat miniseries v1. https://github.com/karpathy/nanochat/discussions/420, 2026b. GitHub discussion, 7 January 2026.
  18. [19]Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute. arXiv preprint arXiv:2509.14786, 2025.arxiv.org/abs/2509.14786
  19. [20]Dan Lee, Seungwook Han, Akarsh Kumar, and Pulkit Agrawal. Training language models via neural cellular automata. arXiv preprint arXiv:2603.10055, 2026.
  20. [21]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. DataComp-LM: In search of the next generation of training sets for language models. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 37, 2024.arxiv.org/abs/2406.11794
  21. [22]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  22. [23]Masato Mita, Taiga Someya, Ryo Yoshida, and Yohei Oseki. Language acquisition device in large language models. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19564–19577, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.895. URL https://aclanthology.org/2026.acl-long.895/.DOI
  23. [24]Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.16264
  24. [25]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. arXiv:2406.17557.arxiv.org/abs/2406.17557
  25. [26]PyTorch. torch.nn.Linear. https://docs.pytorch.org/docs/2.14/generated/torch.nn.Linear.html, 2026a. PyTorch 2.14 documentation.
  26. [29]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683.
  27. [31]Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. arXiv:2401.00448.arxiv.org/abs/2401.00448
  28. [32]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971

Paper details

vlnc:W_vvmc7fk5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents