WorksComputer Science
Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute
Abstract
LLM pretraining is increasingly compute-intensive, while substantial inference serving compute remains idle during off-peak periods. However, such serving hardware is difficult to use for conventional first-order training due to its memory, networking and other system requirements. Zeroth-order (ZO) optimization offers a natural interface to this forward-only compute resource, but its gradient estimates break down in pretraining. Unlike fine-tuning, which can confine updates to a small parameter subspace, pretraining must update every parameter, and the alignment of a ZO gradient estimate with the true gradient shrinks as the number of updated parameters grows. We make a different observation: noisy ZO estimates can still provide useful curvature information, even when they are too inaccurate to serve as gradients. Based on this insight, we propose ZO-Curv, which leverages idle forward-only devices to estimate curvature and uses it to adapt and refine the update used during first-order training, thereby accelerating the process. We provide convergence guarantees that characterize the effect of finite-query curvature estimation. Experiments on LLMs from 125M to 1.5B parameters show that ZO-Curv consistently accelerates pretraining. At 770M parameters, ZO-Curv reaches AdamW's 20K-step validation loss in 3.9 hours instead of 7.5 hours, while achieving a lower final loss of 2.70 versus 2.84 under the same step budget. These results show that idle model serving compute can accelerate LLM pretraining when zeroth-order queries are used to estimate curvature rather than gradients directly.
Introduction
LLM pretraining demands substantial compute, with model development requiring repeated runs over architectures, training schedules, data mixtures, and optimizer configurations [19, 18, 16, 48, 44]. Meanwhile, rapidly growing model serving compute, from inference clusters to local devices [4], represents a potentially valuable neglected resource for accelerating pretraining.
Serving compute is abundant but difficult to train on. Much of this serving capacity is underutilized. As shown in Figure 1(a), serving clusters are provisioned for peak demand, leaving substantial capacity idle during off-peak periods [33, 14, 22, 23]. Across tens of thousands of production Kubernetes clusters, Cast AI reports only 5% average GPU utilization from January 2025 to April 2026 [5]. However, this idle compute cannot readily join conventional first-order (FO) training. Training must additionally store activations for backpropagation, gradients, and optimizer states, resulting in a – memory overhead over inference (Figure 3(a)). It also requires optimized backward kernels, ultra high-speed networks and runtime support that serving platforms often lack ([21, 51]).
Figure 1. Landscape of idle serving devices and the pathway of harvesting them. (a) Diurnal inference demand leaves peak-provisioned capacity idle. (b) Production clusters exhibit low average GPU utilization. (c) Harvesting idle devices doubles the pretraining runs a cluster completes per day. (d) Comparison of ZO curvature and ZO gradients on pretraining.
Figure 3. Serving devices cannot join training, and curvature estimation burdens the training node. (a) Forward-only execution fits a 24 GB serving card at every size; AdamW and SophiaG training exceed it from 355M onward. (b) A SophiaG curvature-refresh step costs about twice an AdamW step, while a forward-only computation is much cheaper to obtain.
Zeroth-order optimization: promise and challenge. Zeroth-order (ZO) optimization needs only forward passes, estimating gradients from loss differences under random perturbations ([40, 31]). This makes it a natural fit for forward-only hardware, and it has recently proven practical for fine-tuning LLMs at inference-level memory cost ([28, 49]). Pretraining is harder: the variance of random-direction ZO estimators grows with the parameter dimension ([31, 11, 37]). At the hundreds of millions to billions of parameters of modern LLMs, this leaves low-query estimates extremely noisy. MeZO-style pretraining of GPT-2 355M confirms the effect, stalling near a validation loss of 8.5 while AdamW drops below 3.0 after the same number of training steps. Mixing ZO estimates into the FO gradient inherits the same noise and underperforms AdamW (Figure 1(d)).
Our approach: ZO-Curv. ZO-Curv uses ZO to estimate curvature rather than gradients. Unlike a gradient, curvature need not identify a precise descent direction, which noisy ZO queries cannot provide at pretraining scale; it only sets a coordinate-wise scale for the update, and even a coarse diagonal estimate can tell how aggressively each coordinate should move. ZO-Curv therefore keeps exact FO gradient computation on the training node and outsources only curvature estimation to idle serving devices, which evaluate perturbed losses with forward passes. The training node aggregates these estimates into a diagonal preconditioner for its FO update.
Our contributions are as follows.
We show that ZO estimates, while too noisy to provide reliable descent directions for LLM pretraining, remain effective for estimating curvature to precondition FO gradients.
We propose ZO-Curv, a hybrid optimization framework that keeps exact FO gradient computation on the training node while offloading forward-only curvature estimation to idle serving devices. We further establish convergence guarantees for ZO-Curv.
We demonstrate consistent gains across models from 125M to 1.5B parameters. ZO-Curv reaches AdamW’s target loss up to faster, with larger models benefiting more from additional serving compute. On the downstream task LAMBADA, ZO-Curv improves accuracy by 26.9% over AdamW.
Background and Motivation
Preliminaries
LLM pretraining. Pretraining minimizes the autoregressive negative log-likelihood over a large corpus , where are the transformer parameters. The de facto optimizer is AdamW ([27]): every step runs a forward and a backward pass and maintains two moment buffers alongside parameters and gradients.
Second-order preconditioning. Sophia [24] accelerates pretraining by dividing the gradient EMA (Exponential Moving Average) by an EMA of a diagonal Hessian estimate, followed by coordinate-wise clipping. The curvature estimate needs to be neither exact nor fresh: it is refreshed only every ten steps, and the EMA, the clipping, and a positive floor on the denominator absorb estimation noise. This coarse signal halves the number of steps to reach a given validation loss. Its cost, however, is charged to the training node, since the Gauss–Newton–Bartlett (GNB) estimator computes a minibatch gradient with resampled labels, an additional forward and backward pass. Sophia accounts for this as 5% extra compute per step on average, or roughly 50% extra FLOPs on each refresh step. Measured on GPU, however, a refresh step costs about twice an AdamW step rather than the 1.5× that its FLOPs suggest (§ 2.3).
Zeroth-order optimization. Zeroth-order (ZO) methods optimize with forward evaluations only [40, 31]. The two-point estimator perturbs the parameters along a random direction and forms
which is unbiased for the gradient of a smoothed loss. Its variance, however, grows linearly with the dimension at a fixed number of evaluations [31, 11]. Because each evaluation needs only inference-level memory, ZO has recently proven practical for fine-tuning LLMs [28, 49], which raises the question of whether it can support pretraining as well.
ZO gradients lose directional signal at pretraining scale
The most direct way to use forward-only serving devices is to let ZO replace backpropagation outright. Figure 2(a) shows that it does not work. We pretrain a 355M model in GPT-2 structure from scratch on OpenWebText with a MeZO-style two-point estimator, against AdamW under an identical setup. AdamW drops below a validation loss of 3.0 within the budget, whereas pure ZO stalls near 8.5 after an initial descent. The gap is not a constant-factor slowdown: beyond the first few thousand steps, the ZO run makes essentially no further progress.
Figure 2. ZO gradients fail at pretraining. (a) Pure ZO stalls early while AdamW converges under the same budget. (b) Cosine similarity between the ZO gradient and the exact gradient.
The failure is directional and scales with the number of trainable parameters . Figure 2(b) compares the cosine similarity between the two-point estimate equation 1 and the exact gradient in pretraining () and in LoRA fine-tuning of the same model on SST-2 [39] (). Because the estimate is parallel to its Gaussian direction , this cosine has expected magnitude close to for any gradient, matching both curves: it reflects the dimension, not the loss landscape. Fine-tuning can confine the search to a small subspace, but pretraining must update every parameter, so each ZO estimate carries almost no directional information. Forward-only devices should therefore supply not the descent direction but information that tolerates far noisier measurements.
Serving devices cannot train, and curvature is costly
Could idle serving devices instead join conventional training as ordinary data-parallel workers? Figure 3a compares peak per-device memory for forward-only execution against AdamW and SophiaG training across model sizes. Forward-only execution fits a representative 24 GB serving card at every size, peaking at 8.5 GB for the 1.5B model. Training crosses that limit already at 355M and reaches roughly eight times the forward-only footprint at 1.5B, because it additionally holds activations for backpropagation, gradients, and optimizer state. A device that comfortably serves a model still cannot train it.
Second-order preconditioning strains the training node further, more than its FLOPs accounting indicates. § 2.1 reports Sophia consumes roughly 50% extra FLOPs for the GNB refresh. While measured on GPU (Figure 3b), a step that recomputes the GNB estimate takes 2.12× the time of an AdamW step. Meanwhile, forward-only ZO estimation costs only 66%, and it runs within serving-level memory. The compute that curvature estimation needs is cheap and abundant, but it sits on the wrong side of the cluster.
Together, the two negative results motivate a single split. The first-order gradient is indispensable (§ 2.2) and demands full training capability, so it must stay on the training node. Curvature is auxiliary: it only rescales coordinates, the update rule that consumes it is built to tolerate noise (EMA, clipping, positive floor), and is friendly to ZO estimation on the idle serving nodes.
ZO-Curv: Curvature from Idle Forward-Only Compute
Setting
Figure 4 illustrates the overall ZO-Curv workflow. We consider a service cloud with one strong training node and forward-only serving nodes. The training node supports backpropagation and maintains optimizer states; serving nodes hold the model and evaluate losses with inference-level memory. The nodes are physically interconnected within the same cloud, and we assume that transferring model snapshots and query results is not the dominant cost.
Figure 4. ZO-Curv overview. The strong node runs ordinary first-order pretraining; idle serving devices concurrently evaluate central-difference queries and return their averaged directional estimate, from which the strong node assembles the diagonal curvature that preconditions its update.
Let be the pretraining objective, with . The two pools run concurrently, offset by one step. At step , the training node broadcasts the snapshot to the serving pool and computes the exact gradient with true next-token labels; meanwhile, the pool evaluates its curvature queries at , whose results arrive in time for step . The update at step therefore uses the curvature estimated at : neither side ever waits for the other, and the curvature
EMA and the model are still refreshed at every training step. The one-step offset has no measurable effect on training (Appendix § 13).
Forward-only diagonal curvature estimation
Estimation target. Our target is the Gauss–Newton–Bartlett (GNB) diagonal estimator used by Sophia-G [24]. Given a curvature minibatch , we sample pseudo-labels and define the sampled-label loss . Writing , the reference estimate is . Its expectation over the sampled labels recovers the minibatch Gauss–Newton diagonal [29]. We estimate a damped version of this GNB diagonal from forward passes only, avoiding the additional backward pass needed to compute .
Forward-only queries. At each step, the serving pool draws independent full-parameter directions and evaluates two perturbed losses per direction. All queries share the same snapshot, curvature minibatch, and sampled labels, which are held fixed under perturbation. The returned central differences determine the curvature estimate:
Averaging the directional estimates before squaring yields an unbiased estimate of the GNB diagonal with uniform damping (Proposition 1). The curvature minibatch and pseudo-labels are sampled on the serving nodes. Thus, the downlink carries only the model snapshot and query indices, while the pool aggregates the directional terms internally. A pool leader returns the averaged vector , requiring one gradient-sized transfer per refresh. Overall, the estimator requires perturbed forward evaluations across the pool.
Preconditioned update. The training node keeps two exponential moving averages, both initialized at zero: the gradient momentum and the curvature ; the latter uses the estimate of the previous step, which the pool returns one step after receiving the snapshot (§ 3.1). Dividing the momentum by the floored curvature and clipping the result yields
where scales the curvature, is the weight-decay coefficient, and all operations act coordinate-wise. The floor and the clipping threshold are what make this update forgiving of curvature
Algorithm 1 ZO-Curv: training node
1: Input: initial parameters ; learning rates ; query budget ; EMA coefficients ; curvature scale ; clipping threshold ; denominator floor ; weight decay ; staleness 2: ; ; partition into 3: Broadcast ; 4: for do 5: 6: 7: if then 8: ▷ queries of 9: 10: Broadcast ; ▷ non-blocking; pool runs Alg. 2 11: else 12: 13: end if 14: 15: 16: end for
Algorithm 2 ZO-Curv: serving node
1: Input: snapshot ; query indices ; finite-difference radius 2: Draw and ; form 3: for do 4: 5: 6: 7: end for 8: ▷ local partial sum 9: Send to the pool leader 10: Leader: 11: Return to the training node
error: a positive denominator rescales a coordinate without flipping its sign, and clipping caps the step wherever the estimated curvature is too small. Algorithms 1 and 2 state both procedures for a general staleness , partitioning the query indices into disjoint sets across the pool.
Convergence Analysis
We study a deterministic core of equation 3: replace minibatch gradients with and omit momentum and weight decay; the results hold for staleness (Remark 2). Proofs and explicit constants are in Appendix H.
Proposition 1 (Forward-only damped GNB curvature). Fix a step and condition on its curvature minibatch and sampled labels. As ,
and for a universal constant .
The forward-only estimate is thus unbiased for a GNB diagonal with a Levenberg–Marquardt-type damping that decreases as , and its fluctuation is of the order of its target in every coordinate, independently of . Under Assumption A1, finite differences add a relative error of order .
Let , , be the denominator built from the EMA of at the same iterates, and let be its ZO counterpart. Fresh directions at every refresh let the EMA average the fluctuations: if each coordinate of varies by at most a factor over the horizon, then with probability at least , after warm-up steps, where and (Theorem 3); the positive floor preserves this relative bound. We require .
Theorem 1 (Robust descent). Suppose along each update segment and the above distortion bound holds. For , the deterministic update satisfies
If is bounded below, the minimum of this clipped stationarity measure over steps is .
A positive denominator preserves each gradient coordinate’s sign, and clipping bounds the update magnitude; smoothness and the step-size condition ensure that the finite step decreases the loss.
Theorem 2 (Local linear convergence). Suppose clipping is inactive and the preconditioned PL condition holds in the local region. Under the conditions of Theorem 1, choosing gives
Curvature error enters only through the factor , which is governed by the EMA window and depends on only logarithmically. The budget instead sets the damping of the reference preconditioner: as grows, decreases and approaches the undamped GNB preconditioner of Sophia-G. The serving pool thus supplies coordinate-wise update scales while the training node retains first-order gradient computation.
Experiments
We evaluate ZO-Curv to answer four research questions: RQ1: does forward-only curvature estimation accelerate LLM pretraining end to end, in wall-clock time and across model sizes? RQ2: how does the gain correlate with the query budget ? RQ3: can curvature queries substitute for first-order batch size, exchanging training-node compute for idle serving compute? RQ4: do faster pretraining loss curves translate into downstream quality?
Setup
Models and data. GPT-2–style decoder-only transformers (Radford et al. [34]) with 125M, 355M, and 770M parameters, plus a 1.5B GPT-NeoX configuration, trained from scratch on OpenWebText (Gokaslan & Cohen [13]) with context length 1024 and global batch size 512. SophiaG and ZO-Curv train for 20K steps at 125M–770M and 50K steps at 1.5B; AdamW receives a doubled step budget (40K and 100K, respectively) so that all methods span a comparable wall-clock window.
Baselines. (i) AdamW (Loshchilov & Hutter [27]), the FO standard. (ii) SophiaG (Liu et al. [24]), which computes the GNB curvature by backpropagation on the training node, refreshed at every step so that both second-order methods consume equally fresh curvature. (iii) Pure ZO, a MeZO-style ZO-SGD (Malladi et al. [28]). (iv) InstructedZO (Cai [4]), which mixes ZO gradient estimates into the FO gradient and represents the gradient-level use of forward-only compute. (v) ZO-Curv, with unless stated otherwise. Further details are given in Appendix § 9.
Hardware and timing. All experiments run on a single H100 node. The forward-only pool is emulated on the same hardware: at every step, the perturbed forward evaluations of § 3.2 are computed alongside the first-order step. Wall-clock time for ZO-Curv counts the training-node critical path; the concurrent curvature evaluations are excluded, as they would occupy otherwise-idle serving devices. § 4.2 additionally runs first-order training and the forward-only queries on disjoint GPU groups, and the speedup persists.
RQ1: End-to-end acceleration
Figure 5 plots validation loss against wall-clock time and grants AdamW twice the steps to span a comparable time window. Table 1 instead holds every method to the same 20K-step budget at 770M and measures the time to reach AdamW’s final loss. ZO-Curv reaches this loss in roughly half the wall-clock time. The speedup is consistent from 125M to 1.5B.
| Method | Val. loss ↓ | Steps to target (K) ↓ | Time to target (h) ↓ |
| AdamW | 2.84 | 20.0 | 7.53 |
| SophiaG | 2.69 | 10.0 | 7.98 |
| Pure ZO (MeZO-style) | 8.01 | – | – |
| InstructedZO | 3.37 | – | – |
| ZO-Curv (ours) | 2.70 | 10.5 | 3.98 |
Table 1. Main comparison at GPT-2 770M with a shared 20K-step budget. Steps- and time-to-target measure how quickly each method reaches AdamW’s final validation loss; “–” = never reached.
Table 1 also shows that simply using forward-only compute does not provide this benefit. Pure ZO never reaches the target, while InstructedZO finishes above AdamW. Both use ZO estimates as descent directions. In contrast, ZO-Curv uses forward-only estimates for curvature while retaining the exact first-order gradient. The benefit therefore comes from using forward-only compute for curvature rather than for the descent direction.
The comparison with SophiaG further isolates where curvature is computed. SophiaG and ZO-Curv make nearly identical progress per step (Appendix § 10). However, SophiaG computes its GNB curvature on the training node, where a refresh costs an AdamW step (§ 2.3). Table 1 shows the resulting wall-clock overhead. SophiaG reaches the target in only 10.0K steps but takes 7.98 hours, more than the 7.53 hours AdamW needs for its full run. ZO-Curv reaches the same target in 10.5K steps and takes only 3.98 hours. Its curvature estimation runs concurrently on the idle serving pool, leaving the training-node critical path close to that of AdamW. Refreshing curvature every ten steps, as in the original Sophia, amortizes the overhead only by using staler curvature. Even then, SophiaG remains 6–12% slower per step than ZO-Curv (Appendix § 11). Offloading curvature removes this training-node overhead rather than trading it against freshness.
Figure 6 further validates the speedup under a real concurrent deployment. We partition one H100 node into two disjoint four-GPU groups. One group performs first-order training, while the other executes forward-only curvature queries. The returned curvature has a three-step staleness (). AdamW and SophiaG use only the four training GPUs, so the other four GPUs represent otherwise-idle serving capacity. Unlike the emulated setting above, the figure reports the measured elapsed training time of the concurrent runs. ZO-Curv completes 20K steps in about half the time required by SophiaG. At 20K steps, ZO-Curv reaches a validation loss of 2.81, close to SophiaG’s 2.78, while requiring roughly half the elapsed training time. It also reaches this loss in less than half the time of AdamW. Thus, the wall-clock acceleration persists when curvature estimation and first-order training execute concurrently on disjoint GPU groups.
Figure 6. Real deployment on disjoint GPU groups. The x-axis reports the measured elapsed training time.
RQ2: Effect of the query budget
In Section 3.3, the damping of the curvature target decreases as , bringing the preconditioner closer to the GNB diagonal of SophiaG, and Figure 7(a,b) confirms the effect on the final loss. The loss improves as grows, monotonically until the curve flattens. Where it flattens depends on scale: 125M is essentially saturated by and 355M by , while 770M and 1.5B still improve at . This ordering is consistent with the dimension dependence of the estimation error. Figure 7(c,d) shows the corresponding trajectories at the two largest sizes. Even separates from AdamW within the first few thousand steps, and larger budgets lower the whole curve rather than only its endpoint. Appendix § 12 repeats these panels for all four sizes.
Figure 7. Effect of the query budget across model sizes. (a)(b) Final validation loss for different query budgets; (c)(d) corresponding training trajectories.
The queries are mutually independent, so the budget partitions freely across the pool and bounds how many idle devices can contribute in parallel. Read as a provisioning rule, the sweep says that larger models absorb more idle capacity before the returns flatten. Since a larger budget never costs accuracy, we use in all other experiments.
RQ3: Trading first-order batch for queries
The experiments so far hold the first-order configuration fixed and add queries. We now ask the converse: can queries pay for a smaller first-order batch? Figure 8 varies the training batch size jointly with at 355M, with the step budget held fixed across cells and the column giving plain AdamW. Along the marked pairs, halving the batch and adding a modest query budget recovers the full-batch loss: a batch of 256 with matches a batch of 512 at 3.03. Fewer queries are needed as the batch shrinks, eight when replacing a batch of 256 and a single one when replacing a batch of 128. A first-order step at half the batch costs roughly half the training-node compute, so this substitution shortens every step while the added queries run on serving hardware. ZO-Curv thus offers two ways to spend the same idle capacity: keep the batch and reach the target loss in less time (RQ1), or shrink the batch and cheapen every step at equal loss. Both trade scarce training GPUs for abundant serving compute.
Figure 8. Effect of the training batch size and query budget on final validation loss for GPT-2 355M. The column corresponds to AdamW.
RQ4: Downstream quality
We evaluate the 770M checkpoints at the 20K-step budget of Table 1 on HellaSwag [47], ARC-Easy and ARC-Challenge [8], and LAMBADA [32]. We first evaluate the pretrained checkpoints with few-shot prompting and then apply LoRA finetuning to each task (Figure 9; protocol in Appendix § 14). Before finetuning, ZO-Curv consistently outperforms AdamW across all four tasks and closely tracks SophiaG. The largest gain appears on LAMBADA, where ZO-Curv improves accuracy from 30.8% to 39.1%, a 26.9% relative improvement over AdamW.
Figure 9. Downstream performance of different models. Solid bars: few-shot performance; hatched bars: LoRA finetuning gains.
The advantage persists after finetuning. ZO-Curv achieves an average score of 39.6, compared with 35.2 for AdamW and 39.8 for SophiaG. These results show that the lower pretraining loss of ZO-Curv translates into stronger downstream performance rather than remaining confined to the pretraining objective. Meanwhile, its near-parity with SophiaG shows that curvature estimated on forward-only hardware provides comparable downstream quality to curvature computed by back-propagation.
Conclusion
We show that idle serving compute can accelerate LLM pretraining by estimating curvature rather than gradients. ZO-Curv keeps exact first-order gradients on the training node while using noisy zeroth-order estimates to precondition step sizes. Across 125M–1.5B models, ZO-Curv matches SophiaG step for step and reaches AdamW’s loss in roughly half the wall-clock time. Future work will evaluate ZO-Curv on larger real-world serving clusters.
AI Use Statement
In this work, we used generative AI tools to implement methods, provide feedback on experimental design, interpret results, and assist with translation. We did not use generative AI tools to formulate the convergence results or to develop the core hypothesis or conceptual framework of ZO-Curv. These contributions originated with the authors. Generating synthetic datasets, cleaning or reformatting datasets, and conducting qualitative data analysis are not applicable to this work. Additionally, we used generative AI tools to search for, identify, and summarize related literature, to format references, and to support brainstorming. We have reviewed all AI-assisted work. AI-assisted implementations were reviewed and validated through the training runs reported in the paper. The authors verified the existence and bibliographic details of every cited work and revised all AI-drafted text. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Reproducibility Statement
Section 3.2 and Algorithms 1 and 2 specify ZO-Curv completely, including the curvature estimator and the preconditioned update. Section 3.3 summarizes our theoretical results, and Appendix H states their assumptions and gives complete proofs. Section 4.1 describes the experimental protocol, and Appendix B lists the model configurations, all optimizer and estimator hyperparameters (Table 2), and the profiling protocol. Our implementation builds on the public nanoGPT code base used by Sophia [24], and all data are public: we pretrain on OpenWebText with the nanoGPT split and evaluate on standard benchmarks following the protocol of Appendix G. Appendix ?? describes the deployment on disjoint GPU groups. We will release our code upon acceptance.
Related work
Zeroth-order optimization. ZO methods build gradient surrogates from function evaluations alone, from SPSA [40] to the smoothing analysis of Nesterov & Spokoiny (2017) and its stochastic nonconvex extension [12]. Their rates degrade with the dimension, and this dependence is optimal up to constants for two-point estimators [11, 37]. MeZO [28] made ZO practical for LLM fine-tuning by regenerating perturbations from random seeds, and later work benchmarks ZO fine-tuning [49] and reduces its variance with adaptive momentum [17], sparse perturbations [26], and low-rank subspaces [7, 46]. DeepZero [6] trains convolutional networks from scratch with coordinate-wise estimates and pruning-guided sparsity, and forward gradients [2, 35] replace backpropagation with forward-mode directional derivatives, whose variance likewise grows with the number of parameters. InstructedZO [4] mixes ZO estimates into first-order gradients to pretrain on mobile devices. All of these use the estimate as the descent direction; § 2.2 shows that this fails at LLM pretraining scale, and ZO-Curv uses ZO queries only for curvature.
Zeroth-order curvature estimation. Second-order SPSA [41] estimates the Hessian from loss values alongside the gradient and uses it in a Newton-type update. HiZOO [50] estimates a diagonal Hessian with one extra forward pass per step and uses it to precondition ZO fine-tuning of LLMs. Both still take the descent direction from ZO estimates, in settings that avoid backpropagation altogether. ZO-Curv instead pairs a forward-only curvature estimate with the exact first-order gradient, targets the GNB diagonal of Sophia-G for pretraining, and computes the estimate on separate idle hardware.
Preconditioned optimizers for pretraining. Adaptive methods [20, 27] precondition with gradient moments, Adafactor [38] factors these moments to save memory, and K-FAC [30] and Shampoo [15] use structured curvature, with the generalized Gauss–Newton matrix as the principled curvature for likelihood losses [29]. More recent optimizers such as SOAP [43] and Muon [25] precondition LLM pretraining with matrix-level structure. AdaHessian [45] and Sophia [24] instead use a diagonal Hessian estimate, and Sophia shows that a clipped, EMA-smoothed estimate accelerates LLM pretraining, but these estimators require extra backward passes on the training node. The closest systems precedent is Distributed Shampoo [1], which computes the inverse roots of its preconditioners asynchronously on otherwise underused CPUs and refreshes them only every few hundred steps. Its statistics, however, are still gathered from backpropagated gradients on the accelerators, and only the matrix roots are offloaded. ZO-Curv moves the curvature statistic itself to forward-only devices. It does not propose a new preconditioner: it matches SophiaG step for step (Appendix § 10).
Training on heterogeneous and idle compute. SWARM parallelism [36] and DiLoCo [10] train across poorly connected devices, Petals [3] runs collaborative inference and fine-tuning of large models, and Bamboo [42] trains on preemptible instances. Their training workers must all run backpropagation. On the serving side, inference engines such as vLLM [21] and DistServe [51] are built around forward passes, and Splitwise [33] and Mélange [14] make inference clusters more efficient. Lyra [22] loans idle inference servers to training jobs during low-traffic periods, Valve [23] colocates offline work with online serving, and Runway reclaims idle inference GPUs for research jobs [9]. These systems hand idle serving capacity to other workloads, and loaned servers must run full training. ZO-Curv instead lets forward-only capacity accelerate the same pretraining run, without backpropagation on the serving devices.
Experimental details
Models and data. We build on the nanoGPT implementation used by Sophia [24]: decoder-only transformers with GELU activations, no bias terms, and no dropout, with the configurations of Table 2. For the 355M and larger models we scale attention by the inverse of the layer index, as Sophia does to keep AdamW stable at these sizes. All models are trained from scratch on OpenWebText [13] with the GPT-2 tokenizer and the train and validation split of nanoGPT (9B and 4.4M tokens), at context length 1024 and global batch size 512, in bfloat16.
| Model | Heads | Depth | AdamW lr | SophiaG / ZO-Curv lr | ||
| GPT-2 Small (125M) | 768 | 12 | 12 | 0.05 | ||
| GPT-2 Medium (355M) | 1024 | 16 | 24 | 0.08 | ||
| GPT-2 Large (770M) | 1280 | 20 | 36 | 0.05 | ||
| GPT-NeoX (1.5B) | 1536 | 24 | 48 | 0.05 |
Table 2. Model configurations and peak learning rates. ZO-Curv uses the SophiaG hyperparameters unchanged; is the curvature scale of equation (3).
Optimization. All methods use a cosine learning-rate schedule with 2K warmup steps, decaying to a minimum between and , and gradient clipping at norm 1.0. AdamW uses and weight decay 0.1. ZO-Curv and SophiaG use , weight decay 0.2, clipping threshold , and the curvature scale of Table 2, following Sophia, and refreshes its curvature at every step. Step budgets are given in § 4.1.
Curvature estimator. Unless stated otherwise, ZO-Curv uses queries per refresh with finite-difference radius , and samples the pseudo-labels from the model’s own predictive distribution, as in the GNB estimator. Following the Sophia implementation, the scale in counts the tokens of the curvature batch, and the denominator floor is .
Profiling. The measurements of § 2.3 report peak reserved GPU memory at batch size 8, with forward-only execution in fp16 and training in mixed precision, and per-step time averaged over 100 steps after warmup.
Step-axis comparison with SophiaG
Figure 10 replots the runs of Figure 5 on a step axis. ZO-Curv and SophiaG are nearly indistinguishable at every size, confirming that the forward-only estimator matches the backpropagated GNB estimator in per-step optimization quality. The contribution of ZO-Curv is not a stronger preconditioner but a preconditioner obtained from hardware that could not otherwise participate in training, which is why the two methods separate on the wall-clock axis of Figure 5.
Figure 5. Main result (RQ1): forward-only curvature turns step efficiency into wall-clock speedup (OpenWebText; AdamW receives a doubled step budget). ZO-Curv reaches AdamW’s final loss in roughly half the time at every size, whereas SophiaG spends its step advantage on curvature computed on the training node.
Figure 10. ZO-Curv matches SophiaG’s step efficiency at every scale (first 20K steps of the runs of Figure 5). The two curves are nearly indistinguishable on a step axis: the forward-only estimator matches the backpropagated one in per-step quality, and the advantage of ZO-Curv lies in where the estimate is computed.
SophiaG with a ten-step refresh interval
Our main comparison refreshes the curvature of SophiaG at every step, so that it consumes curvature as fresh as that of ZO-Curv (§ 4.1). The original Sophia instead refreshes every steps to amortize the cost of the GNB estimator. Figure 11 reports the time each method needs for a fixed budget of 20K steps under this setting, using the per-step measurements behind Figure 5. A ten-step interval brings SophiaG to 9–13% above AdamW per step, while ZO-Curv stays within 1–5% of AdamW because its estimate is computed off the training node. SophiaG therefore remains 6–12% slower than ZO-Curv at every size, and it obtains this reduction by refreshing its curvature ten times less often, whereas ZO-Curv refreshes at every step.
Figure 11. SophiaG with a ten-step refresh interval. Wall-clock time to complete 20K steps for AdamW, SophiaG refreshing its curvature every ten steps, and ZO-Curv refreshing at every step. Even with its curvature cost amortized, SophiaG is slower than ZO-Curv at every size.
The staler curvature also costs accuracy (Figure 12). After 20K steps, SophiaG with a ten-step interval ends 0.03–0.04 above its every-step counterpart in the main runs and 0.02–0.04 above ZO-Curv at every size. Amortizing the estimator on the training node thus leaves SophiaG both slower and less accurate than ZO-Curv, which keeps the curvature fresh at nearly the per-step cost of AdamW.
Figure 12. Loss of SophiaG with a ten-step refresh interval. Validation loss after 20K steps for the runs of Figure 11. With its curvature refreshed ten times less often, SophiaG ends above ZO-Curv at every size.
Training trajectories across query budgets
Figure 13 extends the trajectory panels of Figure 7(c,d) to all four model sizes. The qualitative picture is uniform: every query budget separates from AdamW early, and increasing lowers the entire trajectory. The sizes differ only in where the budgets stop mattering, consistent with the saturation points of Figure 7(a,b).
Figure 13. Training trajectories across query budgets, all four sizes (companion to Figure 7). At 125M the budgets collapse onto each other early, whereas at 770M and 1.5B they remain cleanly ordered, matching the scale-dependent saturation of Figure 7(a,Figure 7b).
Effect of the one-step pipeline offset
The pipeline of § 3.1 applies the curvature estimated at to the update at step . Figure 14 compares this deployed variant against a synchronous control that waits for the curvature of the current snapshot . The two loss curves coincide throughout training: consecutive snapshots differ by a single clipped update, and the curvature EMA smooths over far longer horizons than one step, so the offset costs nothing while removing the serving pool from the critical path.
Figure 14. The one-step pipeline offset is free (GPT-2 355M, ). Training with curvature estimated at (pipelined, as deployed) is indistinguishable from using curvature estimated at the current (synchronous).
Full downstream results
Table 3 lists the per-task accuracies behind Figure 9. We evaluate the 770M checkpoint of each method after 20K steps, the shared budget of Table 1.
| Method | HellaSwag Base | HellaSwag FT | ARC-E Base | ARC-E FT | ARC-C Base | ARC-C FT | LAMBADA Base | LAMBADA FT | Average Base | Average FT |
| AdamW | 31.4 | 33.0 | 40.2 | 50.9 | 22.1 | 26.1 | 30.8 | – | 31.1 | 35.2 |
| SophiaG | 35.6 | 37.1 | 47.5 | 54.7 | 25.8 | 28.1 | 39.1 | – | 37.0 | 39.8 |
| ZO-Curv (ours) | 35.2 | 36.3 | 47.4 | 54.2 | 26.8 | 28.8 | 39.0 | – | 37.1 | 39.6 |
Table 3. Full downstream evaluation of the GPT-2 770M checkpoints at 20K steps (companion to Figure 9). Accuracy in %. Base: few-shot before finetuning (10 examples for HellaSwag, 25 for ARC-E and ARC-C, none for LAMBADA). FT: zero-shot after per-task LoRA finetuning. LAMBADA has no public training split and is not finetuned; it enters the FT average with its untuned score.
Before finetuning. We use few-shot prompting with 10 in-context examples for HellaSwag, 25 for ARC-Easy and ARC-Challenge, and none for LAMBADA, drawn from each training split with a fixed seed. For the three multiple-choice tasks, the prediction is the choice with the highest average log-likelihood per answer token, evaluated on the validation split (10,042, 570, and 299 examples). LAMBADA is scored by whether greedy decoding recovers the final word, on the 5,153 test examples of the OpenAI version.
After finetuning. Starting from each checkpoint, we finetune separately on the training split of every multiple-choice task with LoRA (rank 8, , dropout 0.05) on the attention and MLP projections, which trains 5.9M parameters (0.76% of the model). We use AdamW with learning rate and batch size 8 for 2,000 steps, with the loss restricted to the answer tokens, and then evaluate on the validation split. LAMBADA has no public training split, so it is not finetuned.
Results. Before finetuning, ZO-Curv improves over AdamW on every task, by 6.0 points on average (37.1 versus 31.1). It stays within 0.4 points of SophiaG on HellaSwag, ARC-Easy, and LAMBADA, and is 1.0 point higher on ARC-Challenge. Finetuning helps the AdamW checkpoint most, by 4.1 points on average against 2.8 for SophiaG and 2.5 for ZO-Curv, with the largest gain on ARC-Easy (10.7 points). It nonetheless trails both second-order checkpoints on every task after finetuning, while ZO-Curv remains within 0.8 points of SophiaG throughout (39.6 versus 39.8 on average).
Proofs
This appendix provides the full theoretical analysis of ZO-Curv. Our goal is to isolate the effect of replacing the backpropagation-based Gauss–Newton–Bartlett (GNB) curvature estimator in Sophia-G with a forward-only zeroth-order estimator. The analysis proceeds in three stages. First, we identify what the average-then-square ZO estimator estimates: under a finite query budget , it is an unbiased estimate of a damped GNB diagonal whose damping decreases as , and its fluctuation is of the order of this target in every coordinate, independently of the dimension . Second, we show that the curvature EMA averages these fluctuations into a bounded multiplicative perturbation of the damped reference preconditioner, and that such a perturbation does not destroy descent: the update direction remains a descent direction and converges to stationarity. Third, we combine the two results into a convergence guarantee relative to the damped-GNB-preconditioned reference update.
As in the theoretical analysis of Sophia [24], we analyze a deterministic clipped preconditioned core in order to isolate the role of curvature estimation. In particular, we omit gradient momentum and decoupled weight decay from the convergence proof. The curvature EMA is retained and updated at every iteration, matching the per-step estimator in § 3. We first analyze the synchronous core, in which the update at step uses the curvature queried at ; Remark 2 shows that the results carry over unchanged to the pipelined update with any staleness .
Notation and reference curvature estimator
Let
denote the training objective, with parameters . At iteration , let
denote the first-order gradient used to update the model.
The curvature estimator is computed from a sampled-label mini-batch, as in the GNB estimator of Sophia-G. Given a mini-batch , let
and define the sampled-label loss
We denote its gradient at by
Conditioned on the sampled mini-batch and pseudo-labels, is fixed throughout the ZO queries.
The backpropagation-based GNB estimator used as our reference is
where denotes element-wise multiplication. Taking expectation over the sampled labels recovers the diagonal of the mini-batch Gauss–Newton matrix, as established by the Bartlett identities in [24]. Our analysis below compares the ZO estimate with this reference conditioned on the same mini-batch and the same sampled labels.
A finite query budget adds a uniform damping to this target (Proposition 1). We therefore define the damped GNB diagonal
where is the all-ones vector. This is a Levenberg–Marquardt-type damping of the GNB diagonal, of the kind commonly added to curvature preconditioners for numerical stability. Here it is not a tuned hyperparameter but is induced by the finite budget, and as . Adding a constant to every coordinate preserves the ordering of the coordinates by curvature.
The reference is coupled to the ZO-Curv trajectory: at every iteration, is evaluated at the ZO-Curv iterate , on the same mini-batch and with the same sampled labels as the ZO queries. It is never computed by the algorithm and serves only as a comparison point. Define the reference curvature EMA, initialized at zero as in ZO-Curv, as
The corresponding reference denominator is
where is the numerical denominator floor. We use exclusively for the ZO finite-difference radius, to distinguish it from .
ZO approximation of the GNB curvature
At each iteration, ZO-Curv draws independent Gaussian perturbations
Importantly, all queries use the same mini-batch and the same sampled pseudo-labels, and hence share the same curvature gradient .
For direction , define the central-difference directional derivative
The corresponding ZO gradient estimate is
ZO-Curv first averages the gradient estimates,
and then squares the averaged estimate:
We first study the ideal directional-derivative limit .
Lemma 1 (Moments of the averaged ZO gradient). Fix a vector and let . Define
Then, for every coordinate ,
Proof. For a single Gaussian direction,
which proves equation 22.
For the second moment, Isserlis’ identity for a standard Gaussian gives
Therefore
Averaging independent directions divides the variance by , hence
Combining this with yields
Lemma 1 shows that the averaged ZO gradient itself approaches at the canonical stochastic rate. Its second moment equation equation (23) is the source of the damping in equation (13). For the convergence analysis, however, we require a high-probability rather than an expectation bound.
Lemma 2 (High-probability ZO gradient error). Under the setting of Lemma 1, there exists a universal constant such that, for any failure probability , with probability at least ,
Proof. Fix a coordinate and define
We have . Moreover, is a centered Gaussian with variance , and is standard Gaussian. The product of two sub-Gaussian random variables is sub-exponential, even when the two variables are correlated. Therefore, for a universal constant ,
Bernstein’s inequality for independent centered sub-exponential variables then gives
Setting
and applying a union bound over proves equation (31). □
We next account for the finite central-difference radius .
Assumption A1 (Local Hessian Lipschitzness). For every iteration , the sampled-label loss is twice differentiable with an -Lipschitz Hessian on the ball containing all ZO query points:
The query points are with , so the relevant radius is rather than ; we take in Proposition 1 and in Theorem 3, where and bound with high probability.
Lemma 3 (Central-difference error). Under Assumption A1, for any direction with ,
Proof. Define the scalar function
Then and , so Assumption A1 gives
Taylor’s theorem with integral remainder, , then yields
Subtracting the two expansions cancels and , and dividing by gives
This proves the result; no third derivative is required. □
We now combine Lemmas 1–3 into the properties of the curvature estimate used in the rest of the analysis. Write
for the ideal estimate with exact directional derivatives, and for the central-difference estimate equation 20.
Proposition 1 (Forward-only estimation of damped GNB curvature). Fix an iteration and condition on its mini-batch and sampled labels, with .
(i) Unbiasedness. The ideal estimate is unbiased for the damped target equation 13:
(ii) Relative second moment. There is a universal constant such that, for every coordinate ,
(iii) Relative envelope and finite differences. For , define
and suppose and that Assumption A1 holds with . Then, with probability at least , simultaneously for all coordinates ,
where is the constant of Lemma 2.
Proof. For (i), Lemma 1 with gives
For (ii), write with and as in the proof of Lemma 2. Since ,
The are independent and centered, so
By Lemma 1, , and since , also for a universal constant . Hence with , and
with , where the last step uses . For (iii), we split the failure probability into three events. First, Lemma 2 with failure probability , for which , gives for all . Since and for ,
Therefore, using ,
which is equation (46).
Second, Gaussian norm concentration, , and a union bound over the directions give, with probability at least , . Third, the Gaussian tail bound and a union bound over all coordinates give, with probability at least , . On the second event every query point lies in with , so Lemma 3 applies to every direction and
On both events, satisfies . Then
and equation (46) gives equation (47). A union bound over the three events, with total failure probability , completes the proof. □
Proposition 1 shows that, like the GNB estimator of Sophia-G, the forward-only estimate is unbiased; its target is the GNB diagonal with the additional uniform damping . Averaging before squaring is what keeps this damping small: squaring each directional estimate before averaging would give , a damping times larger that does not decrease with the budget. Moreover, by (ii) and (iii), the fluctuation of the estimate around its target is at most of the order of the target itself in every coordinate, with constants independent of , , and . This relative control is what the curvature EMA exploits next. Dividing equation (47) by shows that the finite-difference perturbation is also relative, of order .
Effect of curvature EMA
The optimizer does not use a single raw curvature estimate directly. Instead, both Sophia-G and ZO-Curv maintain a curvature EMA. Because fresh directions are drawn at every refresh, the EMA averages the relative fluctuations of Proposition 1 over its effective window .
At every iteration, define
and let and , so that and . Let denote the history up to and including and the step- mini-batch and sampled labels, but not the step- directions. Then and are -measurable, the directions are independent of , and is -measurable.
Assumption A2 (Slowly varying damped curvature). Over the analyzed horizon , there exist and deterministic such that
almost surely along the ZO-Curv trajectory.
Since , the damping keeps every coordinate away from zero, and the assumption only asks that the damped curvature of each coordinate varies by at most a factor over the horizon. Let .
Lemma 4 (EMA concentration). Suppose Assumptions A1 and A2 hold over iterations. For , define
take in Assumption A1, and suppose . Let and
Then, with probability at least , simultaneously for all and all coordinates ,
Proof. For each , let be the event of Proposition 1(iii) at step with failure probability . Its constants satisfy , , and , so the bounds of Proposition 1(iii) hold with and . Because the step- directions are independent of , the proposition applies conditionally on , and .
Fix a coordinate and decompose
Finite-difference term. On , equation 47 and give
so the first sum is at most .
Truncation. By Proposition 1(i), . Let . Since and are nonnegative and , on we have . Define the truncated martingale differences
Because , the Cauchy–Schwarz inequality and equation 44 bound the conditional mean that is removed:
Hence, on ,
Martingale term. The sequence is a martingale difference sequence with respect to , with and, by Assumption A2 and equation 44,
Freedman’s inequality then gives, with probability at least ,
where we used . Finally, for we have , so and by Assumption A2. Dividing by and collecting the three terms gives equation 61 for this pair . A union bound over the events , with total failure probability , and over the Freedman events, with total failure probability , completes the proof. □
Lemma 4 bounds the error relative to the reference EMA itself. The leading term is governed by the EMA window and depends on , , and only through . It does not require : the query budget enters the analysis through the damping of the target rather than through the estimation error.
From curvature error to preconditioner distortion
Define the actual ZO-Curv denominator
Lemma 5 (Denominator distortion). Suppose that, for some and every coordinate ,
Then
Proof. The map is nondecreasing. For the upper bound,
For the lower bound, since ,
Both matrices are diagonal, which proves equation 71. □
Lemma 5 is the interface between the estimator analysis and the optimization analysis. Because the error bound of Lemma 4 is relative, it passes through the floor without any division by , and the entire effect of the forward-only ZO estimator is summarized by the multiplicative distortion .
Robust descent under approximate curvature
We now analyze the deterministic clipped preconditioned update
where
The clipping operator is applied coordinate-wise:
The implementation corresponds to .
The following assumption is a preconditioned analogue of the usual smoothness assumption.
Assumption A3 (Local preconditioned smoothness). There exists such that, for every iteration and every point on the update segment,
For ordinary gradient descent, smoothness assumes . Assumption A3 replaces the isotropic matrix by the reference curvature preconditioner . Because the reference target is damped, ordinary smoothness already implies this assumption. For (defined in Lemma 4), , so along the update segments implies Assumption A3 with . This is only a sufficient condition; the assumption holds with a smaller whenever the GNB part of captures the curvature of .
Theorem 1 (Robust descent under curvature estimation error). Suppose
for some , and Assumption A3 holds. Define
Then
and the one-step loss satisfies
Therefore, every non-stationary step is a descent step whenever
In particular, if
then
Proof. Since every , clipping preserves the sign of . Hence
Summing over coordinates gives equation 80.
Taylor’s theorem with integral remainder and Assumption A3 imply
It remains to upper-bound the quadratic term. By definition of clipping,
Therefore
From ,
Since and have the same sign,
Thus
Summing over yields
Substituting equation 94 into equation 87 gives equation 81. The remaining claims follow immediately from the corresponding stepsize choices.
The key property in Theorem 1 is that curvature error affects the magnitude of the update but not its coordinate-wise sign. In contrast, replacing itself with a noisy ZO gradient may change the sign of the first-order descent term . This difference formalizes why noisy ZO information is more naturally tolerated as a preconditioner than as a descent direction.
Corollary 1 (Convergence to stationarity). Assume the conditions of Theorem 1, let
and suppose is bounded below by . Then, for any ,
Consequently,
Proof. Equation equation 84 gives
Summing from to ,
Since ,
The minimum is no larger than the average, giving equation 96.
Remark 1 (From to the gradient). The measure controls the gradient whenever the denominator is bounded above. Under Assumption A2, , so on the event of Theorem 3, . Since is increasing in , implies for every coordinate .
Local linear convergence
The global result above does not require clipping to be inactive. Once the optimization enters a sufficiently local region, however, the preconditioned gradient becomes small and clipping no longer activates. In this regime we obtain a sharper linear rate.
Assumption A4 (Preconditioned Polyak–Łojasiewicz condition). There exists such that, whenever lies in a neighborhood of the target solution,
Theorem 2 (Local linear convergence with approximate curvature). Suppose Assumptions A3 and A4 hold, and
Suppose further that
Then clipping is inactive and
For any
we have
In particular, choosing
gives
Proof. From ,
Condition equation 103 therefore implies
so clipping is inactive.
Using Theorem 1 with ,
Since
inversion reverses the PSD order:
Hence
where the second inequality uses Assumption A4. Substituting equation 115 into equation 111 proves equation 106. Setting gives equation 108. □
Theorem 2 makes the effect of curvature estimation error explicit. For exact reference curvature, , the contraction coefficient is
With approximate ZO curvature, the effective progress coefficient is reduced continuously by
Thus curvature estimation error shrinks the stable stepsize and slows the contraction rate, but does not destroy convergence as long as .
Convergence of ZO-Curv
We finally combine the estimator and optimization results over a horizon of iterations, with one curvature estimate per training step. Throughout, , , , , , and are as in Lemma 4.
Theorem 3. (Convergence of ZO-Curv). Suppose Assumptions A1 (with ), A2, and A3 hold over a horizon of iterations, and . Then, with probability at least , simultaneously for every iteration ,
If
and
then
If, in addition, the iterates remain in the neighborhood of Assumption A4 and satisfy the no-clipping condition equation (103) with replaced by , then choosing
gives
Proof. By Lemma 4, with probability at least $1-\zeta$, coordinate-wise for every \delta=\delta_T$ then proves equation (118). Equation (121) follows from Corollary 1 applied to the iterations , and equation (123) follows from Theorem 2 applied at each of these iterations.
The query budget enters Theorem 3 through the reference preconditioner rather than through : is built from the damped target , whose damping decreases as , so larger budgets bring the reference update closer to the undamped GNB-preconditioned update of Sophia-G. The estimation error is controlled by the EMA window and depends on , , and only logarithmically.
Corollary 2. (Controlled penalty relative to reference GNB curvature). Suppose there exists a constant such that . Then the number of local iterations required to reach
satisfies
For exact reference curvature, , this reduces to
Hence the multiplicative iteration penalty of ZO curvature relative to the reference GNB-preconditioned core is bounded by
For small ,
Proof. From Theorem 3,
where
Using
it is sufficient that
which gives equation 125. Equation equation 128 is the Taylor expansion around . □
Remark 2 (Pipelined updates with staleness ). In the deployed pipeline (§ 3.1 and Algorithm 1), the EMA used at step is built from estimates queried at earlier snapshots , , and is held fixed between refreshes. Define the reference EMA with exactly the same schedule, from at the same snapshots, mini-batches, and sampled labels. Proposition 1 is a per-refresh statement. Lemma 4 applies to the sequence of refreshes, with indexing refreshes, and the history before the directions of refresh : these directions are still drawn independently of , and a held step copies both EMA states. Lemma 5 depends only on and . Theorems 1 and 2 are per-step statements about and , with Assumptions A3 and A4 stated relative to the lagged . Theorem 3 unions over at most refreshes. All results therefore hold verbatim for any staleness , including the one-step offset of the main experiments and the deployment of Appendix ??; staleness enters only through the reference , which the assumptions already constrain.
Interpretation
The above results establish the following chain:
The finite query budget has two separate effects. Its systematic effect is the uniform damping of the curvature target (Proposition 1), which averaging before squaring keeps a factor smaller than squaring before averaging, and which vanishes as . Its random effect is a fluctuation of the order of the target in every coordinate, with constants independent of . The curvature EMA averages this fluctuation over its effective window (Lemma 4), and the positive floor preserves the resulting relative bound (Lemma 5), which yields the multiplicative preconditioner distortion of equation 60. As long as , ZO-Curv preserves descent. In the local unclipped regime, the convergence coefficient is reduced from the reference value
to
where and refer to the damped GNB reference preconditioner.
Therefore, curvature-estimation error produces a continuous and controlled degradation of the convergence rate rather than destroying the descent direction. This differs fundamentally from injecting ZO noise directly into the first-order gradient: an erroneous gradient estimate can rotate or reverse the descent direction, whereas a positive curvature estimate only rescales the coordinate-wise first-order update, with clipping additionally bounding the maximum step magnitude.
Thus, replacing the backpropagation-based GNB computation with forward-only ZO queries yields a damped GNB preconditioner: the query budget controls the damping, the curvature EMA controls the estimation error, and the convergence mechanism of clipped curvature preconditioning is retained.
References
- [1]Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
- [2]Atılım Güneş Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation. arXiv preprint arXiv:2202.08587, 2022.arxiv.org/abs/2202.08587
- [3]Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riaibinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 558–568, 2023.
- [4]Dongqi Cai. Federated llm pre-training on mobile phones. In Proceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services, pp. 657–658, 2025.DOI
- [5]Cast AI. 2026 state of kubernetes optimization report. https://cast.ai/reports/kubernetes-optimization-report/, 2026. Analysis of production Kubernetes clusters across AWS, GCP, and Azure.
- [6]Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. DeepZero: Scaling up zeroth-order optimization for deep model training. In International Conference on Learning Representations, volume 2024, pp. 50185–50206, 2024.arxiv.org/abs/2310.02025
- [7]Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In International Conference on Learning Representations, volume 2025, pp. 62581–62607, 2025.arxiv.org/abs/2410.07698
- [8]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
- [9]Brannon Dorsey and Matt Kafonek. Borrowing the night: Reclaiming idle inference gpus for research. https://runway.com/news/engineering/borrowing-the-night-reclaiming-idle-inference-gpus-for-research, July 2026.
- [10]Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023.arxiv.org/abs/2311.08105
- [11]John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015.arxiv.org/abs/1312.2139
- [12]Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368, 2013.arxiv.org/abs/1309.5549
- [13]Aaron Gokaslan and Vanya Cohen. OpenWebText corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
- [14]Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024.arxiv.org/abs/2404.14527
- [15]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp. 1842–1850. PMLR, 2018.arxiv.org/abs/1802.09568
- [16]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
- [17]Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18363–18371, 2024.
- [18]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
- [19]Apoorv Khandelwal, Tian Yun, Nihal V Nayak, Jack Merullo, Stephen H Bach, Chen Sun, and Ellie Pavlick. $100K or 100 days: Trade-offs when pre-training with academic resources. arXiv preprint arXiv:2410.23261, 2024.arxiv.org/abs/2410.23261
- [20]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
- [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023.arxiv.org/abs/2309.06180
- [22]Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang. Lyra: Elastic scheduling for deep learning clusters. In Proceedings of the Eighteenth European Conference on Computer Systems, pp. 835–850, 2023.DOI
- [23]Fangyue Liu, Hua Liu, Xinyuan Lyu, Shuo Ai, Hao Liang, Lingpeng Chen, Ziqian Hu, Chong Zha, Xin Jin, Hanmei Luo, et al. Valve: Production online-offline inference colocation with jointly-bounded preemption latency and rate. arXiv preprint arXiv:2604.07874, 2026a.arxiv.org/abs/2604.07874
- [24]Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International conference on learning representations, volume 2024, pp. 1621–1650, 2024.arxiv.org/abs/2305.14342
- [25]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
- [26]Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. Advances in Neural Information Processing Systems, 38:21139–21169, 2026b.arxiv.org/abs/2402.15751
- [27]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
- [28]Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075, 2023.arxiv.org/abs/2305.17333
- [29]James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76, 2020.
- [30]James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408–2417. PMLR, 2015.arxiv.org/abs/1503.05671
- [31]Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017.
- [32]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzele, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 1525–1534, 2016.arxiv.org/abs/1606.06031
- [33]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132. IEEE, 2024.
- [34]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [35]Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. arXiv preprint arXiv:2210.03310, 2022.arxiv.org/abs/2210.03310
- [36]Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. In International Conference on Machine Learning, pp. 29416–29440. PMLR, 2023.arxiv.org/abs/2301.11913
- [37]Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research, 18(52):1–11, 2017.
- [38]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International conference on machine learning, pp. 4596–4604. PMLR, 2018.arxiv.org/abs/1804.04235
- [39]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.DOI
- [40]James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 1992.DOI
- [41]James C Spall. Adaptive stochastic approximation by the simultaneous perturbation method. IEEE transactions on automatic control, 45(10):1839–1853, 2000.
- [42]John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible instances resilient for affordable training of large {DNNs}. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 497–513, 2023.arxiv.org/abs/2204.12013
- [43]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, volume 2025, pp. 93423–93444, 2025.arxiv.org/abs/2409.11321
- [44]Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084–17097, 2021.
- [45]Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10665–10673, 2021.arxiv.org/abs/2006.00719
- [46]Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. Zeroth-order fine-tuning of llms in random subspaces. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4475–4485. IEEE, 2025.arxiv.org/abs/2410.08989
- [47]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
- [48]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shouhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.arxiv.org/abs/2205.01068
- [49]Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592, 2024.arxiv.org/abs/2402.11592
- [50]Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor Tsang. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. In International Conference on Learning Representations, volume 2025, pp. 43496–43520, 2025.arxiv.org/abs/2402.15173
- [51]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 193–210, 2024.arxiv.org/abs/2401.09670













