WorksComputer Science
OpenGNS: An Open Dataset of the Gradient Noise Scale across Vision, Language, and Diffusion Models
Abstract
The Gradient Noise Scale (GNS) metric measures the signal-to-noise ratio of gradients in stochastic gradient descent and, as shown by McCandlish et al. (2018), can be used to approximate the Critical Batch Size (CBS) during neural network training. Despite its appearance in technical reports of frontier models, no open dataset of GNS measurements across modern architectures exists, limiting systematic study of the metric. We close this gap by releasing OpenGNS, a dataset of more than 400 GNS measurement trajectories spanning three domains: image classification with ResNets, autoregressive language modeling with Generative Pre-trained Transformers (GPTs), and image synthesis with Diffusion Transformers (DiTs), covering models from 11M to 1B parameters trained under Maximal Update Parameterization (muP) with dense batch-size and learning-rate sweeps. Building on this dataset, we study how the GNS depends on hyperparameters, scale, and training progress, and how it relates to the CBS. We find that the GNS behaves as a consistent "temperature of training", remaining stable across model sizes under muP at equivalent loss, and our results suggest task-dependent biases in its relation to the CBS: a tendency to overestimate the CBS for classification and to underestimate it for generative tasks, consistent in direction across common CBS definitions. The dataset is available at https://huggingface.co/datasets/olympique-marcel/OpenGNS
Introduction
The Gradient Noise Scale (GNS), introduced by McCandlish et al. [22], provides a principled approach to understanding the trade-offs inherent in stochastic gradient descent. Intuitively, the GNS quantifies the noise in the stochastic gradient: When gradient noise is high, larger batch sizes are beneficial, as averaging over more samples yields a more accurate approximation of the true gradient. When noise is already low, additional samples merely increase computational cost per update without improving gradient quality. This intuition connects the GNS to the Critical Batch Size (CBS), which is the batch size that balances runtime and computational effort required to reach a given loss level, and is therefore highly relevant to the AutoML community.
Understanding and accurately estimating the CBS has become increasingly important in the era of foundation models. Pre-training autoregressive Large Language Models (LLMs) and diffusion models from the Computer Vision (CV) domain consumes substantial computational resources, often distributed across thousands of Graphics Processing Units (GPUs) [31, 7] (NVIDIA et al. 2025). Making pre-training more efficient, therefore, reduces energy consumption and cost, and helps democratize Artificial Intelligence (AI) research for academic laboratories that lack industrial-scale compute. In the data-parallel training setting, the global batch size directly determines how many GPUs can be utilized in parallel, making batch-size selection a critical decision that affects training efficiency, resource utilization, and final model quality.
Despite its practical importance, the GNS has received limited systematic investigation. While the original work by McCandlish et al. [22] predates the Transformer era, the metric does subsequently appear in technical reports of some frontier models, see, e.g., GPT-3 [2]. Furthermore, open-source implementations exist for measuring the GNS in both autoregressive and diffusion settings [3, 9]. However, no comprehensive analysis has examined how the GNS behaves across modern model architectures, training regimes, and hyperparameter configurations, and no open dataset of GNS measurements is available to the community to study the metric and validate its predictions.
We close this gap with OpenGNS, an open dataset of GNS measurements spanning three domains: image classification with ResNets on CIFAR-10, autoregressive language modeling with Generative Pre-Trained Transformers (GPTs) on FineWeb, and image synthesis with Diffusion Transformers (DiTs) on ImageNet. The dataset covers eleven model configurations from 11M up to 1B parameters under Maximal Update Parameterization (muP), trained over dense batch-size and learning-rate grids. Such a dataset is also directly relevant to the AutoML community. Full hyperparameter sweeps over batch size and learning rate are prohibitively expensive at the scale of modern foundation models, and methods that rely on cheap proxies (smaller models or shorter runs) depend on the assumption that low-fidelity behavior transfers to higher fidelities. Per-step trajectories of loss and GNS across a dense grid of model sizes, batch sizes, and learning rates provide a quantitative basis for studying this assumption and for informing Hyperparameter Optimization (HPO) strategies for batch-size and learning-rate selection without re-running the underlying experiments. Alongside the dataset, we present an empirical study that demonstrates its utility and yields insights that can inform more efficient allocation of computational resources in large-scale training runs. In summary, we make the following three contributions:
The OpenGNS dataset: We release GNS measurements together with training and validation losses for more than 400 training runs spanning three domains, eleven model sizes, and dense batch-size/learning-rate grids under muP. To our knowledge, this is the first open dataset of its kind.
GNS across hyperparameters: The measurement of the GNS depends strongly on the batch-size and learning-rate configuration. However, when both hyperparameters are jointly well-tuned, training runs exhibit consistent GNS trajectories, validating the notion of the GNS as a temperature of training [22] also for modern Transformer architectures.
GNS across loss and model size: Empirical tests examine how well the GNS approximates the CBS across different learning tasks, extending experiments from McCandlish et al. [22] on CIFAR-10 to GPT-style architectures and DiTs. The results suggest systematic differences depending on the task type: Overestimation for classification and underestimation for generative tasks. Scaling up model size under muP, both metrics remain consistent across model sizes, enabling transfer of CBS recommendations across widths at the same loss.
The paper is structured as follows: In § 2 we revisit the derivation of the GNS metric and its measurement process. § 3 introduces the OpenGNS dataset, including the experimental setup, contents, and release format. § 4 presents the empirical findings enabled by the dataset. Related work is covered in § 5. Finally conclusions are drawn in § 6.
Background
In this section, we revisit the theoretical derivation of the GNS following McCandlish et al. [22], show how to measure it without overhead in data-parallel training settings, and define the CBS.
Formal Derivation of the Gradient Noise Scale
The GNS is based on a second-order Taylor expansion of the loss of a Neural Network (NN). Let denote the NN parameters, the true gradient, and the Hessian of the problem. Perturbing the parameters by a step of size along a direction , the expansion of the loss around can be written as
Choosing , the argument of the left-hand side of Eq. (1) becomes , i.e., the update rule of gradient descent with learning rate . With access to the true gradient and Hessian, it is possible to compute the optimal step size that minimizes the loss. Stochastic Gradient Descent (SGD), however, only provides access to a gradient estimate with
The covariance matrix of is defined as
where is the batch size and is the per-example covariance matrix. Computing the expected loss based on this, Eq. (1) can then be rewritten as
It can be shown that in this case the optimal batch size, under assumption of the optimal step size, is attained when the batch size is equal to the GNS, i.e.,
Under the further assumption that the optimization is perfectly conditioned, the Hessian becomes a multiple of identity matrix and Eq. (5) reduces to
which is easier to measure during the training, as computing the Hessian is usually expensive. The quantity then essentially becomes a measure of noise-to-signal of the gradient. When we measure the GNS empirically throughout this work, we make use of the approximation and refer to is as GNS.
Measuring the Gradient Noise Scale
Multiple options for measuring the simplified version of the GNS in Eq. (6) exist. An overview of the different techniques is given in Gray et al. [9]. In this paper, the method presented by McCandlish et al. [22] for data-parallel training environments is used. It is based on the idea that, given gradients at a small and large batch size, the expected norms can be computed as
Solving for and by substituting terms from Eq. (8) into Eq. (7) then yields
Recall that in data-parallel training, the global gradient update is typically computed as
where is the number of devices (GPUs) and is the local gradient update on each GPU. Therefore, by setting , , and , the necessary measurements for computing are available with no additional overhead. To account for the nosiness of these measurements, both values are measured every few training steps and separate Exponential Moving Averages (EMAs) with a decay parameter are fitted to stabilize them, see Sec. B for an analysis of the impact on the measurements. Other options for measuring the GNS in Transformer models include the methods proposed by Gray et al. [9] which measure gradient norms on a per-example basis, but require custom GPU kernels.
Definition of the Critical Batch Size
In the literature, the term CBS is often used with different meanings. In the ideal linear scaling case, an increase of the batch size by a factor results in a decrease by of optimization steps, required to reach a certain loss threshold, by the same factor. The "original" definition by McCandlish et al. [22] defines the CBS as the batch size, where the scaling efficiency drops below 50%. In Zhang et al. (2024), the authors tighten this bound and define the CBS as the batch size, where an overhead of only 20% is allowed. Different from the CBS is the “Optimal Batch Size”, which is sometimes defined as the smallest batch size that can be scaled to without any overhead (Zhang et al., 2024), or the batch size that achieves the lowest validation loss under a given data horizon [6]. Another value is the “Maximum Stable Batch Size”, which is the largest batch size that does not break the training, i.e., the batch size that can recover a certain loss level irresistible of the number of optimization steps/overhead. In general Optimal Batch Size < Critical Batch Size (CBS) < Maximum Stable Batch Size.
OpenGNS Dataset
This section describes the OpenGNS dataset. § 3.1 introduces the three model architectures and experimental setup that produced the measurements, and § 3.2.1 details the contents and release format of the data.
Experimental Setup
To investigate the GNS, we experiment with a supervised learning approach from the image classification domain and two self-supervised learning approaches, one each from the Natural Language Processing (NLP) and CV domains. Specifically, we use a ResNet18 (He et al., 2016) for image classification, an autoregressive language model for NLP, and a DiT image model for
Dataset Contents
CV. All models are trained with the AdamW optimizer [19]. The full hyperparameter grid is summarized in Tab. 1.
The image classification workload trains on the CIFAR-10 dataset [15] for 200 epochs. Following the original muP implementation [33], the channel widths are scaled by factors of 1×, 2×, and 4× relative to the base architecture. We use a cosine learning rate decay schedule that performs one decay step per epoch. For a final generalization test, we leverage the CIFAR-10.1 dataset1. For the autoregressive Transformer model from the NLP domain, we use the nano-muP implementation [4], which is based on NanoGPT [14]. The models are trained for 20B tokens on the FineWeb dataset (100B token version) [29], which roughly matches the compute-optimal training duration for the largest model configuration according to the Chinchilla scaling law [13]. The diffusion models are based on DiTs [28] and use the muP implementation of [34]. Models are trained on ImageNet (Russakovsky et al., 2015) for 51M images, corresponding to ≈ 800,000 steps at batch size . The validation loss is computed on the COCO validation set, following the practice in [5].
All models are trained on the Leonardo Booster High-Performance Computing (HPC) system featuring four NVIDIA A100 GPUs per node, connected via InfiniBand. The GNS is measured by adding a custom hook to the PyTorch-DDP framework, based on the implementations of [3] and [9]. The GPT and DiT models are trained using two nodes (8 GPUs), while the ResNet models use a single node (4 GPUs). In total, ≈ 200,000 GPUs hours were used for development and production runs.
Dataset Contents
Each (model, batch size, learning rate) combination is trained with a single random seed. For the large GPT models (widths 2,048 and 2,560), the small batch sizes , , and are skipped due to long training times. For DiTs, a small number of combinations (9 out of 75) exhibit training instabilities after roughly half the training time and diverge to NaN values, particularly at larger learning rates, in line with what is reported by the original DiT-muP implementation2. Similarly, 12 out of 210 NLP runs diverge through loss spikes, all at the two largest learning rates. We nevertheless include these runs in the dataset, recorded up to the point of divergence, as the onset of instability itself marks a stability boundary of the batch-size–learning-rate grid.
For each training run, we record the training and validation loss together with three GNS-related quantities at regular measurement intervals: The simplified noise scale itself, (stored as gns), the squared gradient norm (gns_norm), and the trace of the per-example gradient covariance (gns_var), see Figure 1. Releasing the two raw components separately allows users to study them separately and reconstruct the GNS themselves.
Figure 1. Left: Dependency of the GNS on the learning-rate hyperparameter, for a fixed batch-size, model-size, and training task (GPT model on FineWeb data). Linear axis scaling is employed for better visibility. Middle and Right: The two raw components of the GNS (the squared gradient norm and the trace of the per-example gradient covariance) over the course of training for a representative GPT model run (Width=512, BS=, LR=).
The dataset is hosted on HuggingFace3 and split into one Parquet file per workload: cv.parquet, nlp.parquet, and diffusion.parquet. All three files share the columns batch_size, width or equivalent model-size identifier, an iteration counter, samples_seen, and the five measurement columns above. nlp.parquet additionally stores both the peak_lr hyperparameter and the instantaneous lr, which varies due to the cosine decay schedule. cv.parquet adds validation and test accuracy. diffusion.parquet stores the learning rate on a scale and includes a second validation loss on COCO images (coco_val/loss).
Empirical Study of the GNS
Using the OpenGNS dataset from § 3, we empirically study the GNS. Section 4.1 analyzes its dependence on batch size and learning rate, while Section 4.2 examines its relationship to the CBS across training loss and model size under muP.
Gradient Noise Scale over Hyperparameters
For the same training objective, GNS measurements can vary substantially depending on the choice of hyperparameters. As explained in the derivation of the GNS in § 2.1, the measurement is sensitive to the choice of learning rate relative to the batch size. Figure 1 depicts this dependence, as for the same model architecture and training task (GPT model on FineWeb data), the measured GNS varies by nearly an order of magnitude when the learning rate is scaled by factors of 8 and 16 from the baseline.
As McCandlish et al. [22] point out, the GNS depends on the learning rate and the batch size through what they term the training temperature. This reflects the idea of the loss function of an NN as a potential energy function. When the learning rate is too small relative to the batch size, the optimizer explores only a narrow region around its current position, leading to artificially inflated GNS measurements. However, when the learning rate is appropriately scaled with the batch size, configurations that reach similar validation losses settle into the same temperature regime and produce comparable GNS trajectories.
The OpenGNS dataset enables us to systematically investigate this dependence by comparing GNS trajectories across a wide variety of batch-size and learning-rate combinations for all three training tasks. We choose for each batch size the learning rate that achieves the lowest validation loss and define this as the optimal configuration. As shown in Figure 2, the GNS trajectory throughout the training depends significantly on the specific hyperparameter configuration chosen. For each task (autoregressive language modeling with a GPT on FineWeb data, image classification with a ResNet-18 on CIFAR-10, and diffusion modeling with a DiT on ImageNet) we observe that configurations reaching worse final validation losses (highlighted in gray) trace GNS trajectories that deviate substantially from those reaching the lowest validation losses.
Figure 2. GNS as a the temperature of training. Over the course of the training run, batch-size and learning-rate combinations that reach similar final validation losses share similar temperature trajectories. Note that for the GPT and DiT it is possible to recover the Adam-specific scaling rules (learning rate ×2, batch size ×4, blue and green lines). A peak in the learning rate that occurs roughly at the final GNS level, as also predicted by Li et al. (2024), can be identified. The GNS curves for smaller batch sizes employ additional smoothing for better visibility.
However, as the colored curves of Figure 2 emphasize, well-optimized training runs share consistent GNS trajectories, which represent configurations that achieve comparably low final validation losses. Despite varying the batch size by up to a factor of 16 (from to tokens for GPT, from to images for DiT, and from to images for ResNet), the GNS trajectories align when each configuration is paired with an appropriate learning rate.
For some hyperparameter combinations, we are also able to recover the approximate optimal learning-rate–batch-size scaling rules. In particular, the GPT and DiT experiments, see Figure 2, show cases where scaling the batch size by a factor of four while doubling the learning rate (blue and green lines) produces nearly identical GNS trajectories. This is consistent with the Adam-specific scaling behavior documented in prior work, where the optimal learning rate scales as , see [12, 21]. Additionally, in line with the predictions of Li et al. (2024), we observe that the learning-rate peak in batch-size–learning-rate scaling for well-optimized GPT runs occurs at batch sizes (, green line; , red line) close to the final GNS values, between and tokens.
Gradient Noise Scale and Critical Batch Size over Loss and Model Size
We next investigate how well the GNS approximates the CBS and how both metrics depend on the model size when scaling width under muP.
The CBS is calculated by examining different loss levels and quantifying the required overhead in optimization steps when, compared to a base batch size, the batch size to reach the same loss level is doubled. Following the definition of Zhang et al. (2024), we define the CBS as the batch size, where the scaling efficiency overhead exceeds 20%, see App. C for a more detailed explanation and App. D for a sensitivity analysis with respect to this threshold choice.
Figure 3 shows the GNS and CBS as a function of the training loss across different model sizes for all three tasks. The GPT models exhibit the highest GNS and CBS values, reaching up to tokens, followed by the image classification ResNets with values up to tens of thousands of images. The diffusion models show comparatively smaller values, on the order of hundreds of images.
Figure 3. The GNS (solid lines) and CBS (dashed, light lines) as a function of training loss across model sizes for well-optimized batch-size and learning-rate combinations. The GNS remains roughly consistent across model sizes at the same training loss, suggesting that measurements from small models can inform batch-size selection for large models. Large models achieve higher GNS values primarily because they reach lower training losses than small models. While the GNS tracks the overall trend of the CBS, it tends to underestimate the CBS for generative tasks (GPT, DiT) and overestimate it for the classification task (ResNet).
A key observation is that the GNS values remain roughly consistent across model sizes at the same training loss level. This is consistent with the theoretical expectation from McCandlish et al. [22], i.e., the noise scale should not exhibit a strong dependence on the model size directly, but rather primarily on the achieved loss. Large models attain higher GNS values mainly because they reach lower training losses than smaller models. This consistency across model sizes under muP suggests that GNS measurements from smaller proxy models could inform both learning-rate and batch-size selection for larger models, potentially reducing the cost of hyperparameter optimization for large-scale training runs.
While the GNS tracks the overall trend of the CBS across all tasks, we observe systematic differences in their relationship. For the generative tasks (GPT and DiT), the GNS tends to underestimate the CBS, consistent with recent findings in Merrill et al. [23]. At the same time, for the classification task (ResNet on CIFAR-10), the GNS tends to overestimate the CBS, aligning with the original measurements reported in McCandlish et al. [22]. One reason for these systematic discrepancies could be the simplified assumption of a well-conditioned Hessian used to derive in Eq. (5), which may hold to different degrees across task types. For the NLP task in particular, the heavy-tailed class imbalance of the token distribution is known to induce exactly this kind of ill-conditioning Kunstner et al. [16]. The direction of the GNS-CBS deviations persists under alternative overhead thresholds of 10%, 30%, and 50%, although their magnitude varies with the threshold and the gap for the classification task largely closes under the original 50% definition of McCandlish et al. [22], see App. § 10.
Related Work on Gradient Noise Scale
Apart from McCandlish et al. [22], an earlier definition of a noise scale of the gradient was proposed by Smith and Le [32], modeling the training process of an NN as a stochastic differential equation. Frameworks like Pollux (Qiao et al. [30]) and KungFu (Mai et al. [20]) feature the capability to use the GNS metric for selecting a suitable batch size during the training, however, they do not perform a systematic study of its dependence on hyperparameters or model size, nor do they release the underlying measurements. In Gray et al. [8, 9], methods to compute the GNS analytically in both approximate and exact forms are introduced, with a focus on LLMs. The authors of Crowson et al. [3] use the GNS for scheduling the batch size during the final 10% of training of a DiT, increasing the batch size up to a factor of 10.
More recently, measurements of the GNS for long training horizons (up to 1T tokens) have been performed in Merrill et al. [23] and Li et al. (2025), making use of pretrained model checkpoints to compute the GNS at different intervals of training. Coupled batch-size and learning-rate schedules were recently introduced in Meterez et al. [24], but work only below the CBS and thus require knowing its value a priori. Common to all of these works is that, while they report GNS measurements for individual training runs, none release the underlying trajectories in a form that allows other researchers to reuse them for follow-up study. OpenGNS differs in this respect: It provides per-step GNS measurements together with loss curves across a dense grid of batch sizes, learning rates, and model sizes, spanning image classification, autoregressive language modeling, and diffusion-based image synthesis. Open releases of intermediate training checkpoints ([1, 10, 26]) provide rich training-trajectory data, but focus exclusively on NLP models and do not include GNS measurements. It should be noted that Merrill et al. [23] found that for LLMs at 1B and 7B parameter scale, the GNS underestimates the CBS by multiple orders of magnitude. This underestimation is consistent with our results for generative tasks, though not in this order of magnitude. Their CBS definition also uses a loss-recovery criterion rather than accounting for a scaling-efficiency overhead as proposed by McCandlish et al. [22], which intuitively aligns more with the “Maximum Stable Batch Size”, see § 2.3.
Summary and Conclusion
This study contributes OpenGNS, an open dataset of GNS measurements across three domains: image classification, autoregressive language modeling, and diffusion-based image synthesis. The empirical study enabled by this dataset confirms prior observations regarding CBS differences across tasks: GPTs benefit from batch sizes in the millions of tokens, while DiTs, despite often reaching billions of parameters in state-of-the-art systems, operate efficiently at much smaller batch sizes on the order of hundreds of images (corresponding to roughly tens of thousands of patch tokens).
Our analysis validates the finding of McCandlish et al. [22] that the GNS behaves as a temperature of training, remaining consistent across runs that reach similar validation losses. This work extends this observation to modern Transformer and diffusion architectures and demonstrates that, combined with μP, the consistency holds across model sizes. We find that the relationship between GNS and CBS is not always as straightforward as the theory might suggest: In our experiments, the GNS appears to act more as a lower bound for the CBS in the generative tasks and more as an upper bound in the classification task. Despite certain limitations in the theoretical assumptions and experimental scale (see App. § 7), the cross-size consistency of the GNS under P suggests that measurements from smaller proxy models could inform batch-size selection for larger ones, reducing the computational cost of hyperparameter optimization at scale, which is relevant for the AutoML community. Releasing the raw gradient norm and variance measurements alongside the noise scale further enables deeper investigation of training dynamics across architectures and training regimes. The increase of the GNS as training progresses also points to a practical avenue for resource allocation: Since smaller global batch sizes correspond to fewer GPUs required for data-parallel training, GNS-guided batch-size scheduling could allow for more efficient resource utilization, using a low number of GPUs in the early phases of training when small batches are sufficient, and scaling up to larger batches and more devices as training progresses. By releasing OpenGNS, we hope to encourage further investigation of such strategies and a deeper theoretical understanding of the GNS metric.
Broader Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. Although our evaluations require significant computational resources, the insights we provide in this work will enable more efficient training practices in the future, ultimately reducing resource waste and energy consumption.
Acknowledgements. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Regular Access call. The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the project “NXT GEN AI METHODS – Generative Methoden für Perzeption, Prädiktion und Planung". The authors would like to thank the consortium for the successful cooperation.
Limitations
The theoretical framework of the GNS relies on several simplifying assumptions. It is derived using a second-order Taylor expansion and assumes a well-conditioned Hessian, see Eq. (5). While this approximation proves empirically useful, the systematic differences observed between the GNS and the CBS across task types (overestimation for classification, underestimation for generative tasks) could be a sign of ill-conditioning. Second, all experiments use the AdamW optimizer rather than plain SGD, for which the original theory was derived. AdamW has largely replaced SGD as the most popular optimizer. However, experiments in [22], show that the GNS values remain consistent across loss levels for both SGD and Adam, suggesting the metric also works for adaptive gradient methods. Some works also make use of a preconditioned or non-Euclidean version of the GNS for this reason [30, 25] (Li et al. 2025). Note that our measurements are taken on the raw gradients before AdamW applies momentum and preconditioning, and the optimizer state is not logged during training. The released data therefore does not capture optimizer-state effects, and preconditioned noise-scale variants cannot be reconstructed from it, which we leave to future work. Third, the GNS is defined purely in terms of the training optimization dynamics and provides no direct information about a generalization. Finally, while the experiments involve an extensive grid search using substantial computational resources to thoroughly explore different hyperparameter and model size configurations, the largest models reach only one billion parameters trained on tens of billions of tokens. State-of-the-art generative models often contain 100B parameters and are trained on trillions of tokens, rendering experiments at such scales infeasible. Nevertheless, the consistency of GNS trajectories across model sizes under P suggests that the findings could be extrapolated to even larger scales.
Gradient Noise Scale Measurement Parameters
As described in § 2.2, the measurement of the GNS using the data-parallel method depends on two parameters: The ratio of and as well as the EMA decay parameter . We first investigate how varying these parameters affects the GNS measurements.
As expected, low EMA values, e.g., , result in noisier approximations that closely track rapid changes, while high values, e.g., , produce smoother estimates but respond slower to variations in the GNS, see Figure 4. Based on these observations, we set between 0.95 and 0.995 depending on the task. Regarding the values of and , [9] find that small values reduce the standard error. As shown in our experiments in Figure 5, the ratio does not significantly affect the measured GNS values for a Transformer model. This robustness simplifies practical deployment as the measurement can be performed reliably across different data-parallel configurations without requiring careful tuning of the batch size ratio. Our experiments use ratios of 4 to 8.
Figure 4. Influence of the EMA decay parameter on GNS measurements (GPT model trained on FineWeb data). Small values () yield noisy estimates, while values close to one () smooth excessively.
Figure 5. Influence of the batch size ratio on GNS measurements with the data-parallel method of [22] for a GPT model trained on FineWeb data. The estimates remain consistent across different ratios, indicating robustness to this parameter choice.
Computing the Critical Batch Size
We compute the CBS as follows. For each run we smooth the training loss with an exponentially weighted moving average whose span scales as , so that the smoothing window is roughly constant in samples seen. For a given target loss , we record the number of samples each run needs to first reach , and for every batch size we keep only the best-performing learning rate, . Using the smallest swept batch size as reference, the scaling efficiency is . Following (Zhang et al., 2024), the CBS is the batch size at which first drops below , i.e. a 20% scaling efficiency overhead, obtained by log-linear interpolation between the two adjacent batch sizes on our grid.
Sensitivity to Different Critical Batch Size Thresholds
Throughout the literature, different definitions of the CBS exist, as discussed in § 2.3. In the following, we therefore recompute the CBS under different overhead assumptions, complementing the 20% threshold used in § 4.2. As can be observed from the plots in Figure 6, a tighter definition of the CBS (i.e., a lower tolerated overhead) moves the CBS measurements closer to the GNS predictions for the generative GPT and DiT models, although the GNS still tends to underestimate the CBS at all three thresholds. In the classification case, the GNS continues to overestimate the CBS at the 10% and 30% overhead thresholds (the tighter the threshold, the larger the overestimation)
Figure 6. GNS (solid lines) and CBS (dashed, lighter lines) for tolerated overheads of 10%, 30% and 50% at the CBS (left to right). Rows show GPT / FineWeb, DiT / ImageNet and ResNet-18 / CIFAR-10. The GNS does not depend on the threshold, so the solid curves are identical within each row.
but aligns closely with the CBS under the original 50% definition of McCandlish et al. [22]. The task-dependent deviation reported in § 4.2 thus never changes sign across thresholds, but its magnitude depends on the tolerated overhead.
Sensitivity to Random Seed
As the computations required to deliver the results proposed in this dataset are expensive and scale linearly with the number of random seeds, mostly only a single random seed is provided. However, for the CV case, we reran the batch size–learning rate grid from Table 1 for an additional seed to compute error bars. We average the training loss and the GNS of the two seeds step by step before smoothing, and we repeat the CBS extraction separately for each seed. In Figure 7, lines show the seed-average, while the shaded band and the error bars show the range covered by the two seeds. With only two seeds we report this range instead of a standard error. The GNS is almost identical for both seeds, whereas the CBS varies by a small factor, with the largest deviations at the lowest target losses, but still tending to be overestimated by the GNS.
| Hyperparameter | ResNet18 | GPT | DiT |
| Dataset | CIFAR-10 | FineWeb | ImageNet |
| Training data | 10M images | 20B tokens | 51M images |
| Validation data | 10,000 images | 100M tokens | 40,504 images |
| Widths | [1, 2, 4] | [256, 512, 1,024, 2,048, 2,560] | [144, 288, 576] |
| Depth | 18 | 12 | 28 |
| Attn. head size | - | 128 | 72 |
| Parameters | 11M, 45M, 179M | 22M, 64M, 203M, 707M, 1B | 11M, 43M, 169M |
| Batch size | – images | – tokens | – images |
| Learning rate | – | – | – |
| Optimizer | AdamW | AdamW | AdamW |
| 0.9 | 0.9 | 0.9 | |
| 0.99 | 0.95 | 0.99 | |
| Weight decay | 0.0 | 0.01 | 0.01 |
| LR schedule | Cosine (epoch based) | Cosine (step based) | Constant |
| Final LR | 10% | 10% | - |
| LR warmup | None | 5% of training duration | None |
| for GNS | 0.995 | 0.95 | 0.995 |
| 4 | 8 | 8 |
Table 1. Hyperparameter grids for the three experimental workloads. There are 180 configurations for the ResNet model, 178 for the GPT model and 75 for the DiT models, for a total of 433 training runs.
Figure 7. CBS and GNS for CV, averaged over two seeds. The shaded band (GNS) and error bars (CBS) span the range of the two seeds.
References
- [1]Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Van Der Wal, O. (2023). Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, ICML ’23. JMLR.org.arxiv.org/abs/2304.01373
- [2]Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.arxiv.org/abs/2005.14165
- [3]Crowson, K., Baumann, S. A., Birch, A., Abraham, T. M., Kaplan, D. Z., and Shipppole, E. (2024). Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9550–9575. PMLR.arxiv.org/abs/2401.11605
- [4]Dey, N., Anthony, Q., and Hestness, J. (2024). The practitioner’s guide to the maximal update parameterization.
- [5]Esser, P., Kulal, S., Blattmann, A., Entenzari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., and Rombach, R. (2024). Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
- [6]Filatov, O., Ebert, J., Wang, J., and Kesselheim, S. (2025). Time transfer: On optimal learning rate and batch size in the infinite data limit.arxiv.org/abs/2410.05838
- [7]Grattafiori, A., Dubey, A., Jauhari, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobou, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Hovan, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuletti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gru, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Lloyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Dam laj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vonmimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. (2024). The llama 3 herd of models.arxiv.org/abs/2407.21783
- [8]Gray, G., Samar, A., and Hestness, J. (2023). Efficient and approximate per-example gradient norms for gradient noise scale. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023).
- [9]Gray, G., Tiwari, A., Bergsma, S., and Hestness, J. (2024). Normalization layer per-example gradients are sufficient to predict gradient noise scale in transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2411.00999
- [10]Groeneveld, D., Beltagy, I., Walsh, E., Bhagavatula, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muenninghoff, N., Naik, A., Nam, C., Peters, M., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Soldaini, L., Smith, N., and Hajishirzi, H. (2024). OLMo: Accelerating the science of language models. In Ku, L.-W., Martins, A., and Srikuumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.00838
- [12]Hilton, J., Cobbe, K., and Schulman, J. (2022). Batch size-invariance for policy optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.arxiv.org/abs/2110.00641
- [13]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennig an, T., Noland, E., Millican, K., van den Dries sche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. (2022). An empirical analysis of compute-optimal large language model training. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.
- [14]Karpathy, A. (2022). NanoGPT. https://github.com/karpathy/nanoGPT.
- [15]Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Available at https: //www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
- [16]Kunstner, F., Yadav, R., Milligan, A., Schmidt, M., and Bietti, A. (2024). Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2402.19449
- [19]Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
- [20]Mai, L., Li, G., Wagenländer, M., Fertakis, K., Brabete, A.-O., and Pietzuch, P. (2020). KungFu: Making training in distributed machine learning adaptive. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 937–954. USENIX Association.
- [21]Malladi, S., Lyu, K., Panigrahi, A., and Arora, S. (2022). On the SDEs and scaling rules for adaptive gradient algorithms. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS’22. Curran Associates, Inc.arxiv.org/abs/2205.10287
- [22]McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. (2018). An empirical model of large-batch training.arxiv.org/abs/1812.06162
- [23]Merrill, W., Arora, S., Groeneveld, D., and Hajishirzi, H. (2025). Critical batch size revisited: A simple empirical approach to large-batch language model training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2505.23971
- [24]Meterez, A., Morwani, D., Wu, J., Oncescu, C.-A., Pehlewan, C., and Kakade, S. (2025). Seesaw: Accelerating training by balancing learning rate and batch size scheduling.arxiv.org/abs/2510.14717
- [25]Naganuma, H., Gupta, S., Briki, Y., Mitliagkas, I., Rish, I., Raman, P., and Shi, H.-J. M. (2026). Adaptive batch sizes using non-euclidean gradient noise scales for stochastic sign and spectral descent.arxiv.org/abs/2602.03001
- [26]Nezhurina, M., Franke, J., Nakamura, T., Carstensen, T., Ajroldi, N., Komulainen, V., Salinas, D., and Jitsev, J. (2025). Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison.
- [28]Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4172–4182.arxiv.org/abs/2212.09748
- [29]Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. (2024). The fineweb datasets: Decanting the web for the finest text data at scale.arxiv.org/abs/2406.17557
- [30]Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., and Xing, E. P. (2021). Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 1–18. USENIX Association.arxiv.org/abs/2008.12260
- [31]Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., Nathan, A., Luo, A., Helyar, A., Madry, A., Efremov, A., Spyra, A., Baker-Whitcomb, A., Beutel, A., Karpenko, A., Makelov, A., Neitz, A., Wei, A., Barr, A., Kirchmeyer, A., Ivanov, A., Christakis, A., Gillespie, A., Tam, A., Bennett, A., Wan, A., Huang, A., Sandjideh, A. M., Yang, A., Kumar, A., Saraiva, A., Vallone, A., Gheorghe, A., Garcia, A. G., Braunstein, A., Liu, A., Schmidt, A., Mereskin, A., Mishchenko, A., Applebaum, A., Rogerson, A., Rajan, A., Wei, A., Kotha, A., Srivastava, A., Agrawal, A., Vijayvergiya, A., Tyra, A., Nair, A., Nayak, A., Eggers, B., Ji, B., Hoover, B., Chen, B., Chen, B., Barak, B., Miniaiev, B., Hao, B., Baker, B., Lightcap, B., McKinzie, B., Wang, B., Quinn, B., Fioca, B., Hsu, B., Yang, B., Yu, B., Zhang, B., Brenner, B., Zetino, C. R., Raymond, C., Lugaresi, C., Paz, C., Hudson, C., Whitney, C., Li, C., Chen, C., Cole, C., Voss, C., Ding, C., Shen, C., Huang, C., Colby, C., Hallacy, C., Koch, C., Lu, C., Kaplan, C., Kim, C., Minott-Henriques, C., Frey, C., Yu, C., Czarnecki, C., Reid, C., Wei, C., Decareaux, C., Scheau, C., Zhang, C., Forbes, C., Tang, D., Goldberg, D., Roberts, D., Palmie, D., Kappler, D., Levine, D., Wright, D., Leo, D., Lin, D., Robinson, D., Grabb, D., Chen, D., Lim, D., Salama, D., Bhattacharjee, D., Tsipras, D., Li, D., Yu, D., Strouse, D., Williams, D., Hunn, D., Bayes, E., Arbus, E., Akyurek, E., Le, E. Y., Widmann, E., Yani, E., Proehl, E., Sert, E., Cheung, E., Schwartz, E., Han, E., Jiang, E., Mitchell, E., Sigler, E., Wallace, E., Ritter, E., Kavanaugh, E., Mays, E., Nikishin, E., Li, F., Such, F. P., de Avila Belbute Peres, F., Raso, F., Bekerman, F., Tsimpourlas, F., Chანტzis, F., Song, F., Zhang, F., Raila, G., McGrath, G., Briggs, G., Yang, G., Parascandolo, G., Chabot, G., Kim, G., Zhao, G., Valiant, G., Leclerc, G., Salman, H., Wang, H., Sheng, H., Jiang, H., Wang, H., Jin, H., Sikchi, H., Schmidt, H., Aspegren, H., Chen, H., Qiu, H., Lightman, H., Covert, I., Kivlichan, I., Silber, I., Sohl, I., Hammoud, I., Clavera, I., Lan, I., Akkaya, I., Kostrikov, I., Kofman, I., Etinger, I., Singal, I., Hehir, J., Huh, J., Pan, J., Wilczynski, J., Pachocki, J., Lee, J., Quinn, J., Kiros, J., Kalra, J., Samaroo, J., Wang, J., Wolfe, J., Chen, J., Wang, J., Harb, J., Han, J., Wang, J., Zhao, J., Chen, J., Yang, J., Tworek, J., Chand, J., Landon, J., Liang, J., Lin, J., Liu, J., Wang, J., Tang, J., Yin, J., Jang, J., Morris, J., Flynn, J., Ferstad, J., Heidecke, J., Fishbein, J., Hallman, J., Grant, J., Chien, J., Gordon, J., Park, J., Liss, J., Kraaijeveld, J., Guay, J., Mo, J., Lawson, J., McGrath, J., Vendrow, J., Jiao, J., Lee, J., Steele, J., Wang, J., Mao, J., Chen, K., Hayashi, K., Xiao, K., Salahi, K., Wu, K., Sekhri, K., Sharma, K., Singhal, K., Li, K., Nguyen, K., Gu-Lemberg, K., King, K., Liu, K., Stone, K., Yu, K., Ying, K., Georgiev, K., Lim, K., Tirumala, K., Miller, K., Ahmad, L., Lv, L., Clare, L., Fauconnet, L., Itow, L., Yang, L., Romaniuk, L., Anise, L., Byron, L., Pathak, L., Maksin, L., Lo, L., Ho, L., Jing, L., Wu, L., Xiong, L., Mamitsuka, L., Yang, L., McCallum, L., Held, L., Bourgeois, L., Engstrom, L., Kuhn, L., Feuvrier, L., Zhang, L., Switzer, L., Kondraciuk, L., Kaiser, L., Joglekar, M., Singh, M., Shah, M., Stratta, M., Williams, M., Chen, M., Sun, M., Cayton, M., Li, M., Zhang, M., Aljubeh, M., Nichols, M., Haines, M., Schwarzer, M., Gupta, M., Shah, M., Huang, M., Dong, M., Wang, M., Glaese, M., Carroll, M., Lampe, M., Malek, M., Sharman, M., Zhang, M., Wang, M., Pokrass, M., Florian, M., Pavlov, M., Wang, M., Chen, M., Wang, M., Feng, M., Bavarian, M., Lin, M., Abdool, M., Rohaninejad, M., Soto, N., Staudacher, N., LaFontaine, N., Marwell, N., Liu, N., Preston, N., Turley, N., Ansman, N., Blades, N., Pancha, N., Mikhaylin, N., Felix, N., Handa, N., Rai, N., Keskar, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Gleeson, O., Mishkin, P., Lesiewicz, P., Baltescu, P., Belov, P., Zhokhov, P., Pronin, P., Guo, P., Thacker, P., Liu, Q., Yuan, Q., Liu, Q., Dias, R., Puckett, R., Arora, R., Mullapudi, R. T., Gaon, R., Miyara, R., Song, R., Aggarwal, R., Marsan, R., Yemiru, R., Xiong, R., Kshirsagar, R., Nuttall, R., Tsiupa, R., Eldan, R., Wang, R., James, R., Ziv, R., Shu, R., Nigmatullin, R., Jain, S., Talaie, S., Altman, S., Arnesen, S., Toizer, S., Toyer, S., Miserendino, S., Agarwal, S., Yoo, S., Heon, S., Ethersmith, S., Grove, S., Taylor, S., Bubeck, S., Banesius, S., Amdo, S., Zhao, S., Wu, S., Santurkar, S., Zhao, S., Chaudhuri, S. R., Krishnaswamy, S., Shuaiqi, Xia, Cheng, S., Anadkat, S., Fishman, S. P., Tobin, S., Fu, S., Jain, S., Mei, S., Egoian, S., Kim, S., Golden, S., Mah, S., Lin, S., Imm, S., Sharpe, S., Yadlowsky, S., Choudhry, S., Eum, S., Sanjeev, S., Khan, T., Stramer, T., Wang, T., Xin, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Degry, T., Shadwell, T., Fu, T., Gao, T., Garipov, T., Sriskandarajah, T., Sherbakov, T., Kaftan, T., Hiratsuka, T., Wang, T., Song, T., Zhao, T., Peterson, T., Kharitonov, V., Chernova, V., Kosaraju, V., Kuo, V., Pong, V., Verma, V., Petrov, V., Jiang, W., Zhang, W., Zhou, W., Xie, W., Zhan, W., McCabe, W., DePue, W., Ellsworth, W., Bain, W., Thompson, W., Chen, X., Qi, X., Xiang, X., Shi, X., Dubois, Y., Yu, Y., Khakbaz, Y., Wu, Y., Qian, Y., Lee, Y. T., Chen, Y., Zhang, Y., Xiong, Y., Tian, Y., Cha, Y., Bai, Y., Yang, Y., Yuan, Y., Li, Y., Zhang, Y., Yang, Y., Jin, Y., Jiang, Y., Wang, Y., Wang, Y., Liu, Y., Stuben̈voll, Z., Dou, Z., Wu, Z., and Wang, Z. (2025). Openai gpt-5 system card.arxiv.org/abs/2601.03267
- [32]Smith, S. L. and Le, Q. V. (2018). A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations.arxiv.org/abs/1710.06451
- [33]Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. (2021). Tuning large neural networks via zero-shot hyperparameter transfer. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 17084–17097. Curran Associates, Inc.
- [34]Zheng, C., Zhang, X., Wang, R., Huang, W., Tian, Z., Huang, W., Zhu, J., and Li, C. (2025). Scaling diffusion transformers efficiently via μp.arxiv.org/abs/2505.15270






