betaFeedback

WorksComputer Science

Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

Abstract

LLM pretraining is increasingly compute-intensive, while substantial inference serving compute remains idle during off-peak periods. However, such serving hardware is difficult to use for conventional first-order training due to its memory, networking and other system requirements. Zeroth-order (ZO) optimization offers a natural interface to this forward-only compute resource, but its gradient estimates break down in pretraining. Unlike fine-tuning, which can confine updates to a small parameter subspace, pretraining must update every parameter, and the alignment of a ZO gradient estimate with the true gradient shrinks as the number of updated parameters grows. We make a different observation: noisy ZO estimates can still provide useful curvature information, even when they are too inaccurate to serve as gradients. Based on this insight, we propose ZO-Curv, which leverages idle forward-only devices to estimate curvature and uses it to adapt and refine the update used during first-order training, thereby accelerating the process. We provide convergence guarantees that characterize the effect of finite-query curvature estimation. Experiments on LLMs from 125M to 1.5B parameters show that ZO-Curv consistently accelerates pretraining. At 770M parameters, ZO-Curv reaches AdamW's 20K-step validation loss in 3.9 hours instead of 7.5 hours, while achieving a lower final loss of 2.70 versus 2.84 under the same step budget. These results show that idle model serving compute can accelerate LLM pretraining when zeroth-order queries are used to estimate curvature rather than gradients directly.

Introduction

LLM pretraining demands substantial compute, with model development requiring repeated runs over architectures, training schedules, data mixtures, and optimizer configurations [19, 18, 16, 48, 44]. Meanwhile, rapidly growing model serving compute, from inference clusters to local devices [4], represents a potentially valuable neglected resource for accelerating pretraining.

Serving compute is abundant but difficult to train on. Much of this serving capacity is underutilized. As shown in Figure 1(a), serving clusters are provisioned for peak demand, leaving substantial capacity idle during off-peak periods [33, 14, 22, 23]. Across tens of thousands of production Kubernetes clusters, Cast AI reports only 5% average GPU utilization from January 2025 to April 2026 [5]. However, this idle compute cannot readily join conventional first-order (FO) training. Training must additionally store activations for backpropagation, gradients, and optimizer states, resulting in a 3.53.5–9×9\times memory overhead over inference (Figure 3(a)). It also requires optimized backward kernels, ultra high-speed networks and runtime support that serving platforms often lack ([21, 51]).

Figure 1. Landscape of idle serving devices and the pathway of harvesting them. (a) Diurnal inference demand leaves peak-provisioned capacity idle. (b) Production clusters exhibit low average GPU utilization. (c) Harvesting idle devices doubles the pretraining runs a cluster completes per day. (d) Comparison of ZO curvature and ZO gradients on pretraining.

Figure 3. Serving devices cannot join training, and curvature estimation burdens the training node. (a) Forward-only execution fits a 24 GB serving card at every size; AdamW and SophiaG training exceed it from 355M onward. (b) A SophiaG curvature-refresh step costs about twice an AdamW step, while a forward-only computation is much cheaper to obtain.

Zeroth-order optimization: promise and challenge. Zeroth-order (ZO) optimization needs only forward passes, estimating gradients from loss differences under random perturbations ([40, 31]). This makes it a natural fit for forward-only hardware, and it has recently proven practical for fine-tuning LLMs at inference-level memory cost ([28, 49]). Pretraining is harder: the variance of random-direction ZO estimators grows with the parameter dimension dd ([31, 11, 37]). At the hundreds of millions to billions of parameters of modern LLMs, this leaves low-query estimates extremely noisy. MeZO-style pretraining of GPT-2 355M confirms the effect, stalling near a validation loss of 8.5 while AdamW drops below 3.0 after the same number of training steps. Mixing ZO estimates into the FO gradient inherits the same noise and underperforms AdamW (Figure 1(d)).

Our approach: ZO-Curv. ZO-Curv uses ZO to estimate curvature rather than gradients. Unlike a gradient, curvature need not identify a precise descent direction, which noisy ZO queries cannot provide at pretraining scale; it only sets a coordinate-wise scale for the update, and even a coarse diagonal estimate can tell how aggressively each coordinate should move. ZO-Curv therefore keeps exact FO gradient computation on the training node and outsources only curvature estimation to idle serving devices, which evaluate perturbed losses with forward passes. The training node aggregates these estimates into a diagonal preconditioner for its FO update.

Our contributions are as follows.

  • We show that ZO estimates, while too noisy to provide reliable descent directions for LLM pretraining, remain effective for estimating curvature to precondition FO gradients.

  • We propose ZO-Curv, a hybrid optimization framework that keeps exact FO gradient computation on the training node while offloading forward-only curvature estimation to idle serving devices. We further establish convergence guarantees for ZO-Curv.

  • We demonstrate consistent gains across models from 125M to 1.5B parameters. ZO-Curv reaches AdamW’s target loss up to 2.2×2.2\times faster, with larger models benefiting more from additional serving compute. On the downstream task LAMBADA, ZO-Curv improves accuracy by 26.9% over AdamW.

Background and Motivation

Preliminaries

LLM pretraining. Pretraining minimizes the autoregressive negative log-likelihood F(θ)=Ex∼D[−∑ilog⁡pθ(xi∣x<i)]F(\theta) = \mathbb{E}_{x\sim\mathcal{D}}\left[-\sum_i \log p_\theta(x_i \mid x_{<i})\right] over a large corpus D\mathcal{D}, where θ∈Rd\theta\in\mathbb{R}^d are the transformer parameters. The de facto optimizer is AdamW ([27]): every step runs a forward and a backward pass and maintains two moment buffers alongside parameters and gradients.

Second-order preconditioning. Sophia [24] accelerates pretraining by dividing the gradient EMA (Exponential Moving Average) by an EMA of a diagonal Hessian estimate, followed by coordinate-wise clipping. The curvature estimate needs to be neither exact nor fresh: it is refreshed only every ten steps, and the EMA, the clipping, and a positive floor on the denominator absorb estimation noise. This coarse signal halves the number of steps to reach a given validation loss. Its cost, however, is charged to the training node, since the Gauss–Newton–Bartlett (GNB) estimator computes a minibatch gradient with resampled labels, an additional forward and backward pass. Sophia accounts for this as 5% extra compute per step on average, or roughly 50% extra FLOPs on each refresh step. Measured on GPU, however, a refresh step costs about twice an AdamW step rather than the 1.5× that its FLOPs suggest (§ 2.3).

Zeroth-order optimization. Zeroth-order (ZO) methods optimize with forward evaluations only [40, 31]. The two-point estimator perturbs the parameters along a random direction uu and forms

g^=F(θ+εu)−F(θ−εu)2εu,(1)\hat{g} = \frac{F(\theta+ \varepsilon u) - F(\theta- \varepsilon u)}{2\varepsilon}u, \tag*{(1)}

which is unbiased for the gradient of a smoothed loss. Its variance, however, grows linearly with the dimension dd at a fixed number of evaluations [31, 11]. Because each evaluation needs only inference-level memory, ZO has recently proven practical for fine-tuning LLMs [28, 49], which raises the question of whether it can support pretraining as well.

ZO gradients lose directional signal at pretraining scale

The most direct way to use forward-only serving devices is to let ZO replace backpropagation outright. Figure 2(a) shows that it does not work. We pretrain a 355M model in GPT-2 structure from scratch on OpenWebText with a MeZO-style two-point estimator, against AdamW under an identical setup. AdamW drops below a validation loss of 3.0 within the budget, whereas pure ZO stalls near 8.5 after an initial descent. The gap is not a constant-factor slowdown: beyond the first few thousand steps, the ZO run makes essentially no further progress.

Figure 2. ZO gradients fail at pretraining. (a) Pure ZO stalls early while AdamW converges under the same budget. (b) Cosine similarity between the ZO gradient and the exact gradient.

The failure is directional and scales with the number of trainable parameters dd. Figure 2(b) compares the cosine similarity between the two-point estimate equation 1 and the exact gradient in pretraining (d≈3.5×108d \approx3.5 \times10^{8}) and in LoRA fine-tuning of the same model on SST-2 [39] (d≈7.9×105d \approx7.9 \times10^{5}). Because the estimate is parallel to its Gaussian direction uu, this cosine has expected magnitude close to 2/(πd)\sqrt{2/(\pi d)} for any gradient, matching both curves: it reflects the dimension, not the loss landscape. Fine-tuning can confine the search to a small subspace, but pretraining must update every parameter, so each ZO estimate carries almost no directional information. Forward-only devices should therefore supply not the descent direction but information that tolerates far noisier measurements.

Serving devices cannot train, and curvature is costly

Could idle serving devices instead join conventional training as ordinary data-parallel workers? Figure 3a compares peak per-device memory for forward-only execution against AdamW and SophiaG training across model sizes. Forward-only execution fits a representative 24 GB serving card at every size, peaking at 8.5 GB for the 1.5B model. Training crosses that limit already at 355M and reaches roughly eight times the forward-only footprint at 1.5B, because it additionally holds activations for backpropagation, gradients, and optimizer state. A device that comfortably serves a model still cannot train it.

Second-order preconditioning strains the training node further, more than its FLOPs accounting indicates. § 2.1 reports Sophia consumes roughly 50% extra FLOPs for the GNB refresh. While measured on GPU (Figure 3b), a step that recomputes the GNB estimate takes 2.12× the time of an AdamW step. Meanwhile, forward-only ZO estimation costs only 66%, and it runs within serving-level memory. The compute that curvature estimation needs is cheap and abundant, but it sits on the wrong side of the cluster.

Together, the two negative results motivate a single split. The first-order gradient is indispensable (§ 2.2) and demands full training capability, so it must stay on the training node. Curvature is auxiliary: it only rescales coordinates, the update rule that consumes it is built to tolerate noise (EMA, clipping, positive floor), and is friendly to ZO estimation on the idle serving nodes.

ZO-Curv: Curvature from Idle Forward-Only Compute

Setting

Figure 4 illustrates the overall ZO-Curv workflow. We consider a service cloud with one strong training node and MM forward-only serving nodes. The training node supports backpropagation and maintains optimizer states; serving nodes hold the model and evaluate losses with inference-level memory. The nodes are physically interconnected within the same cloud, and we assume that transferring model snapshots and query results is not the dominant cost.

Figure 4. ZO-Curv overview. The strong node runs ordinary first-order pretraining; idle serving devices concurrently evaluate qq central-difference queries and return their averaged directional estimate, from which the strong node assembles the diagonal curvature that preconditions its update.

Let F(θ)=Ex∼p[ℓ(θ;x)]F(\theta) = \mathbb{E}_{x\sim p}[\ell(\theta;x)] be the pretraining objective, with θ∈Rd\theta\in\mathbb{R}^{d}. The two pools run concurrently, offset by one step. At step tt, the training node broadcasts the snapshot θt\theta_t to the serving pool and computes the exact gradient gt=∇Lt(θt)g_t = \nabla L_t(\theta_t) with true next-token labels; meanwhile, the pool evaluates its curvature queries at θt\theta_t, whose results arrive in time for step t+1t+1. The update at step tt therefore uses the curvature estimated at θt−1\theta_{t-1}: neither side ever waits for the other, and the curvature

EMA and the model are still refreshed at every training step. The one-step offset has no measurable effect on training (Appendix § 13).

Forward-only diagonal curvature estimation

Estimation target. Our target is the Gauss–Newton–Bartlett (GNB) diagonal estimator used by Sophia-G [24]. Given a curvature minibatch Bt={xb}b=1BB_t = \{x_b\}_{b=1}^{B}, we sample pseudo-labels y^b∼pθt(⋅∣xb)\hat{y}_b \sim p_{\theta_t}(\cdot\mid x_b) and define the sampled-label loss ℓtsamp(θ)=B−1∑b=1Bℓce(fθ(xb),y^b)\ell_t^{\mathrm{samp}}(\theta) = B^{-1} \sum_{b=1}^{B} \ell_{\mathrm{ce}}(f_\theta(x_b), \hat{y}_b). Writing ct=∇ℓtsamp(θt)c_t = \nabla\ell_t^{\mathrm{samp}}(\theta_t), the reference estimate is htG=B ct⊙cth_t^{\mathrm{G}} = B\,c_t \odot c_t. Its expectation over the sampled labels recovers the minibatch Gauss–Newton diagonal [29]. We estimate a damped version of this GNB diagonal from forward passes only, avoiding the additional backward pass needed to compute ctc_t.

Forward-only queries. At each step, the serving pool draws qq independent full-parameter directions ut,j∼N(0,Id)u_{t,j} \sim\mathcal{N}(0, I_d) and evaluates two perturbed losses per direction. All queries share the same snapshot, curvature minibatch, and sampled labels, which are held fixed under perturbation. The returned central differences determine the curvature estimate:

Δt,j=ℓtsamp(θt+εut,j)−ℓtsamp(θt−εut,j)2ε,\Delta_{t,j} = \frac{\ell_t^{\mathrm{samp}}(\theta_t + \varepsilon u_{t,j}) - \ell_t^{\mathrm{samp}}(\theta_t - \varepsilon u_{t,j})}{2\varepsilon},
zˉt=1q∑j=1qΔt,jut,j,htZO=B zˉt⊙zˉt.(2)\bar{z}_t = \frac{1}{q}\sum_{j=1}^{q}\Delta_{t,j}u_{t,j}, \qquad h_t^{\mathrm{ZO}} = B\,\bar{z}_t \odot\bar{z}_t. \tag*{(2)}

Averaging the directional estimates before squaring yields an unbiased estimate of the GNB diagonal with uniform damping ∥htG∥1/q\lVert h_t^{\mathrm{G}}\rVert_1/q (Proposition 1). The curvature minibatch and pseudo-labels are sampled on the serving nodes. Thus, the downlink carries only the model snapshot and query indices, while the pool aggregates the directional terms internally. A pool leader returns the averaged vector zˉt\bar{z}_t, requiring one gradient-sized transfer per refresh. Overall, the estimator requires 2q2q perturbed forward evaluations across the pool.

Preconditioned update. The training node keeps two exponential moving averages, both initialized at zero: the gradient momentum mt=β1mt−1+(1−β1)gtm_t = \beta_1m_{t-1} + (1-\beta_1)g_t and the curvature vt=β2vt−1+(1−β2)ht−1ZOv_t = \beta_2v_{t-1} + (1-\beta_2)h_{t-1}^{\mathrm{ZO}}; the latter uses the estimate of the previous step, which the pool returns one step after receiving the snapshot (§ 3.1). Dividing the momentum by the floored curvature d^t=max⁡{γvt,ϵ0}\hat{d}_t = \max\{\gamma v_t, \epsilon_0\} and clipping the result yields

θt+1=(1−ηtλ)θt−ηt clip⁡(mt/d^t,ρ),(3)\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\,\operatorname{clip}(m_t/\hat{d}_t,\rho), \tag*{(3)}

where γ\gamma scales the curvature, λ\lambda is the weight-decay coefficient, and all operations act coordinate-wise. The floor ϵ0\epsilon_0 and the clipping threshold ρ\rho are what make this update forgiving of curvature

Algorithm 1 ZO-Curv: training node

1: Input: initial parameters θ1\theta_1; learning rates {ηt}t=1T\{\eta_t\}_{t=1}^T; query budget qq; EMA coefficients β1,β2\beta_1,\beta_2; curvature scale γ\gamma; clipping threshold ρ\rho; denominator floor ϵ0\epsilon_0; weight decay λ\lambda; staleness ss 2: m0←0m_0 \leftarrow0; v0←0v_0 \leftarrow0; partition {1,…,q}\{1,\ldots,q\} into {Jm}m=1M\{\mathcal{J}_m\}_{m=1}^M 3: Broadcast (θ1,{Jm})(\theta_1,\{\mathcal{J}_m\}); tdisp←1t_{\mathrm{disp}}\leftarrow1 4: for t=1,…,Tt=1,\ldots,T do 5: gt←∇Lt(θt)g_t\leftarrow\nabla L_t(\theta_t) 6: mt←β1mt−1+(1−β1)gtm_t\leftarrow\beta_1m_{t-1}+(1-\beta_1)g_t 7: if t−tdisp=st-t_{\mathrm{disp}}=s then 8: h←Bzˉt−sZOh\leftarrow B\bar{z}_{t-s}^{\mathrm{ZO}} ▷ queries of θt−s\theta_{t-s} 9: vt←β2vt−1+(1−β2)hv_t\leftarrow\beta_2v_{t-1}+(1-\beta_2)h 10: Broadcast (θt,{Jm})(\theta_t,\{\mathcal{J}_m\}); tdisp←tt_{\mathrm{disp}}\leftarrow t ▷ non-blocking; pool runs Alg. 2 11: else 12: vt←vt−1v_t\leftarrow v_{t-1} 13: end if 14: d^t←max⁡{γvt,ϵ0}\hat{d}_t\leftarrow\max\{\gamma v_t,\epsilon_0\} 15: θt+1←(1−ηtλ)θt−ηtclip⁡(mt/d^t,ρ)\theta_{t+1}\leftarrow(1-\eta_t\lambda)\theta_t-\eta_t\operatorname{clip}(m_t/\hat{d}_t,\rho) 16: end for

Algorithm 2 ZO-Curv: serving node mm

1: Input: snapshot θt\theta_t; query indices Jm\mathcal{J}_m; finite-difference radius ϵ\epsilon 2: Draw BB and y^∼pθt\hat{y}\sim p_{\theta_t}; form ℓsamp\ell^{\mathrm{samp}} 3: for j∈Jmj\in\mathcal{J}_m do 4: uj∼N(0,Id)u_j\sim\mathcal{N}(0,I_d) 5: L±←ℓsamp(θt±ϵuj)L^\pm\leftarrow\ell^{\mathrm{samp}}(\theta_t\pm\epsilon u_j) 6: Δj←(L+−L−)/2ϵ\Delta_j\leftarrow(L^+-L^-)/2\epsilon 7: end for 8: zm←∑j∈JmΔjujz_m\leftarrow\sum_{j\in\mathcal{J}_m}\Delta_j u_j ▷ local partial sum 9: Send zmz_m to the pool leader 10: Leader: zˉt←q−1∑m=1Mzm\bar{z}_t\leftarrow q^{-1}\sum_{m=1}^M z_m 11: Return zˉt\bar{z}_t to the training node

error: a positive denominator rescales a coordinate without flipping its sign, and clipping caps the step wherever the estimated curvature is too small. Algorithms 1 and 2 state both procedures for a general staleness ss, partitioning the qq query indices into disjoint sets {Jm}m=1M\{\mathcal{J}_m\}_{m=1}^M across the pool.

Convergence Analysis

We study a deterministic core of equation 3: replace minibatch gradients with gt=∇F(θt)g_t=\nabla F(\theta_t) and omit momentum and weight decay; the results hold for staleness ss (Remark 2). Proofs and explicit constants are in Appendix H.

Proposition 1 (Forward-only damped GNB curvature). Fix a step and condition on its curvature minibatch and sampled labels. As ϵ→0\epsilon\to0,

E[htZO]=h~t:=(1+1q)gtG+λt1,λt=∥htG∥1q,(4)\mathbb{E}[h_t^{\mathrm{ZO}}]=\tilde{h}_t:=\left(1+\frac{1}{q}\right)g_t^G+\lambda_t\mathbf{1},\qquad\lambda_t=\frac{\lVert h_t^G\rVert_1}{q}, \tag*{(4)}

and E[(ht,iZO)2]≤Cvh~t,i2\mathbb{E}[(h_{t,i}^{\mathrm{ZO}})^2]\le C_v\tilde{h}_{t,i}^2 for a universal constant CvC_v.

The forward-only estimate is thus unbiased for a GNB diagonal with a Levenberg–Marquardt-type damping λt\lambda_t that decreases as 1/q1/q, and its fluctuation is of the order of its target in every coordinate, independently of dd. Under Assumption A1, finite differences add a relative error of order ϵ2\epsilon^2.

Let Dt=Diag⁡(dt)D_t=\operatorname{Diag}(d_t), dt=max⁡{γv~t,ϵ0}d_t=\max\{\gamma\tilde{v}_t,\epsilon_0\}, be the denominator built from the EMA v~t\tilde{v}_t of h~t\tilde{h}_t at the same iterates, and let D^t=Diag⁡(d^t)\hat{D}_t=\operatorname{Diag}(\hat{d}_t) be its ZO counterpart. Fresh directions at every refresh let the EMA average the fluctuations: if each coordinate of h~t\tilde{h}_t varies by at most a factor κ\kappa over the horizon, then with probability at least 1−ζ1-\zeta, (1−δ)Dt⪯D^t⪯(1+δ)Dt(1-\delta)D_t\preceq\hat{D}_t\preceq(1+\delta)D_t after O(1/(1−β2))O(1/(1-\beta_2)) warm-up steps, where δ=O(κ[(1−β2)Λ+(1−β2)Λ2]+ζ/T+ϵ2)\delta=O\left(\kappa[\sqrt{(1-\beta_2)\Lambda}+(1-\beta_2)\Lambda^2]+\sqrt{\zeta/T+\epsilon^2}\right) and Λ=log⁡(qdT/ζ)\Lambda=\log(qdT/\zeta) (Theorem 3); the positive floor preserves this relative bound. We require δ<1\delta<1.

Theorem 1 (Robust descent). Suppose ∇2F⪯αDt\nabla^2F\preceq\alpha D_t along each update segment and the above distortion bound holds. For η≤(1−δ)/α\eta\le(1-\delta)/\alpha, the deterministic update θt+1=θt−ηclip⁡(D^t−1gt,ρ)\theta_{t+1}=\theta_t-\eta\operatorname{clip}(\hat{D}_t^{-1}g_t,\rho) satisfies

F(θt+1)≤F(θt)−η2∑imin⁡{gt,i2dt,i,ρ∣gt,i∣}.(5)F(\theta_{t+1})\le F(\theta_t)-\frac{\eta}{2}\sum_i\min\left\{\frac{g_{t,i}^2}{d_{t,i}},\rho|g_{t,i}|\right\}. \tag*{(5)}

If FF is bounded below, the minimum of this clipped stationarity measure over TT steps is O(1/T)O(1/T).

A positive denominator preserves each gradient coordinate’s sign, and clipping bounds the update magnitude; smoothness and the step-size condition ensure that the finite step decreases the loss.

Theorem 2 (Local linear convergence). Suppose clipping is inactive and the preconditioned PL condition gtTDt−1gt≥2μD(F(θt)−F⋆)g_t^{\mathsf{T}}D_t^{-1}g_t \ge2\mu_D(F(\theta_t)-F_\star) holds in the local region. Under the conditions of Theorem 1, choosing η=(1−δ)/α\eta=(1-\delta)/\alpha gives

F(θt+1)−F⋆≤(1−μDα1−δ1+δ)(F(θt)−F⋆).(6)F(\theta_{t+1})-F_\star\le\left(1-\frac{\mu_D}{\alpha}\frac{1-\delta}{1+\delta}\right)(F(\theta_t)-F_\star). \tag*{(6)}

Curvature error enters only through the factor (1−δ)/(1+δ)(1-\delta)/(1+\delta), which is governed by the EMA window and depends on dd only logarithmically. The budget qq instead sets the damping of the reference preconditioner: as qq grows, λt\lambda_t decreases and DtD_t approaches the undamped GNB preconditioner of Sophia-G. The serving pool thus supplies coordinate-wise update scales while the training node retains first-order gradient computation.

Experiments

We evaluate ZO-Curv to answer four research questions: RQ1: does forward-only curvature estimation accelerate LLM pretraining end to end, in wall-clock time and across model sizes? RQ2: how does the gain correlate with the query budget qq? RQ3: can curvature queries substitute for first-order batch size, exchanging training-node compute for idle serving compute? RQ4: do faster pretraining loss curves translate into downstream quality?

Setup

Models and data. GPT-2–style decoder-only transformers (Radford et al. [34]) with 125M, 355M, and 770M parameters, plus a 1.5B GPT-NeoX configuration, trained from scratch on OpenWebText (Gokaslan & Cohen [13]) with context length 1024 and global batch size 512. SophiaG and ZO-Curv train for 20K steps at 125M–770M and 50K steps at 1.5B; AdamW receives a doubled step budget (40K and 100K, respectively) so that all methods span a comparable wall-clock window.

Baselines. (i) AdamW (Loshchilov & Hutter [27]), the FO standard. (ii) SophiaG (Liu et al. [24]), which computes the GNB curvature by backpropagation on the training node, refreshed at every step so that both second-order methods consume equally fresh curvature. (iii) Pure ZO, a MeZO-style ZO-SGD (Malladi et al. [28]). (iv) InstructedZO (Cai [4]), which mixes ZO gradient estimates into the FO gradient and represents the gradient-level use of forward-only compute. (v) ZO-Curv, with q=512q=512 unless stated otherwise. Further details are given in Appendix § 9.

Hardware and timing. All experiments run on a single 8×8\timesH100 node. The forward-only pool is emulated on the same hardware: at every step, the 2q2q perturbed forward evaluations of § 3.2 are computed alongside the first-order step. Wall-clock time for ZO-Curv counts the training-node critical path; the concurrent curvature evaluations are excluded, as they would occupy otherwise-idle serving devices. § 4.2 additionally runs first-order training and the forward-only queries on disjoint GPU groups, and the speedup persists.

RQ1: End-to-end acceleration

Figure 5 plots validation loss against wall-clock time and grants AdamW twice the steps to span a comparable time window. Table 1 instead holds every method to the same 20K-step budget at 770M and measures the time to reach AdamW’s final loss. ZO-Curv reaches this loss in roughly half the wall-clock time. The speedup is consistent from 125M to 1.5B.

MethodVal. loss ↓Steps to target (K) ↓Time to target (h) ↓
AdamW2.8420.07.53
SophiaG2.6910.07.98
Pure ZO (MeZO-style)8.01––
InstructedZO3.37––
ZO-Curv (ours)2.7010.53.98

Table 1. Main comparison at GPT-2 770M with a shared 20K-step budget. Steps- and time-to-target measure how quickly each method reaches AdamW’s final validation loss; “–” = never reached.

Table 1 also shows that simply using forward-only compute does not provide this benefit. Pure ZO never reaches the target, while InstructedZO finishes above AdamW. Both use ZO estimates as descent directions. In contrast, ZO-Curv uses forward-only estimates for curvature while retaining the exact first-order gradient. The benefit therefore comes from using forward-only compute for curvature rather than for the descent direction.

The comparison with SophiaG further isolates where curvature is computed. SophiaG and ZO-Curv make nearly identical progress per step (Appendix § 10). However, SophiaG computes its GNB curvature on the training node, where a refresh costs 2.12×2.12\times an AdamW step (§ 2.3). Table 1 shows the resulting wall-clock overhead. SophiaG reaches the target in only 10.0K steps but takes 7.98 hours, more than the 7.53 hours AdamW needs for its full run. ZO-Curv reaches the same target in 10.5K steps and takes only 3.98 hours. Its curvature estimation runs concurrently on the idle serving pool, leaving the training-node critical path close to that of AdamW. Refreshing curvature every ten steps, as in the original Sophia, amortizes the overhead only by using staler curvature. Even then, SophiaG remains 6–12% slower per step than ZO-Curv (Appendix § 11). Offloading curvature removes this training-node overhead rather than trading it against freshness.

Figure 6 further validates the speedup under a real concurrent deployment. We partition one 8×8\timesH100 node into two disjoint four-GPU groups. One group performs first-order training, while the other executes forward-only curvature queries. The returned curvature has a three-step staleness (s=3s = 3). AdamW and SophiaG use only the four training GPUs, so the other four GPUs represent otherwise-idle serving capacity. Unlike the emulated setting above, the figure reports the measured elapsed training time of the concurrent runs. ZO-Curv completes 20K steps in about half the time required by SophiaG. At 20K steps, ZO-Curv reaches a validation loss of 2.81, close to SophiaG’s 2.78, while requiring roughly half the elapsed training time. It also reaches this loss in less than half the time of AdamW. Thus, the wall-clock acceleration persists when curvature estimation and first-order training execute concurrently on disjoint GPU groups.

Figure 6. Real deployment on disjoint GPU groups. The x-axis reports the measured elapsed training time.

RQ2: Effect of the query budget

In Section 3.3, the damping of the curvature target decreases as 1/q1/q, bringing the preconditioner closer to the GNB diagonal of SophiaG, and Figure 7(a,b) confirms the effect on the final loss. The loss improves as qq grows, monotonically until the curve flattens. Where it flattens depends on scale: 125M is essentially saturated by q≈32q \approx32 and 355M by q≈256q \approx256, while 770M and 1.5B still improve at q=512q = 512. This ordering is consistent with the dimension dependence of the estimation error. Figure 7(c,d) shows the corresponding trajectories at the two largest sizes. Even q=4q = 4 separates from AdamW within the first few thousand steps, and larger budgets lower the whole curve rather than only its endpoint. Appendix § 12 repeats these panels for all four sizes.

Figure 7. Effect of the query budget qq across model sizes. (a)(b) Final validation loss for different query budgets; (c)(d) corresponding training trajectories.

The qq queries are mutually independent, so the budget partitions freely across the pool and bounds how many idle devices can contribute in parallel. Read as a provisioning rule, the sweep says that larger models absorb more idle capacity before the returns flatten. Since a larger budget never costs accuracy, we use q=512q = 512 in all other experiments.

RQ3: Trading first-order batch for queries

The experiments so far hold the first-order configuration fixed and add queries. We now ask the converse: can queries pay for a smaller first-order batch? Figure 8 varies the training batch size jointly with qq at 355M, with the step budget held fixed across cells and the q=0q=0 column giving plain AdamW. Along the marked pairs, halving the batch and adding a modest query budget recovers the full-batch loss: a batch of 256 with q=32q=32 matches a batch of 512 at 3.03. Fewer queries are needed as the batch shrinks, eight when replacing a batch of 256 and a single one when replacing a batch of 128. A first-order step at half the batch costs roughly half the training-node compute, so this substitution shortens every step while the added queries run on serving hardware. ZO-Curv thus offers two ways to spend the same idle capacity: keep the batch and reach the target loss in less time (RQ1), or shrink the batch and cheapen every step at equal loss. Both trade scarce training GPUs for abundant serving compute.

Figure 8. Effect of the training batch size BB and query budget qq on final validation loss for GPT-2 355M. The q=0q=0 column corresponds to AdamW.

RQ4: Downstream quality

We evaluate the 770M checkpoints at the 20K-step budget of Table 1 on HellaSwag [47], ARC-Easy and ARC-Challenge [8], and LAMBADA [32]. We first evaluate the pretrained checkpoints with few-shot prompting and then apply LoRA finetuning to each task (Figure 9; protocol in Appendix § 14). Before finetuning, ZO-Curv consistently outperforms AdamW across all four tasks and closely tracks SophiaG. The largest gain appears on LAMBADA, where ZO-Curv improves accuracy from 30.8% to 39.1%, a 26.9% relative improvement over AdamW.

Figure 9. Downstream performance of different models. Solid bars: few-shot performance; hatched bars: LoRA finetuning gains.

The advantage persists after finetuning. ZO-Curv achieves an average score of 39.6, compared with 35.2 for AdamW and 39.8 for SophiaG. These results show that the lower pretraining loss of ZO-Curv translates into stronger downstream performance rather than remaining confined to the pretraining objective. Meanwhile, its near-parity with SophiaG shows that curvature estimated on forward-only hardware provides comparable downstream quality to curvature computed by back-propagation.

Conclusion

We show that idle serving compute can accelerate LLM pretraining by estimating curvature rather than gradients. ZO-Curv keeps exact first-order gradients on the training node while using noisy zeroth-order estimates to precondition step sizes. Across 125M–1.5B models, ZO-Curv matches SophiaG step for step and reaches AdamW’s loss in roughly half the wall-clock time. Future work will evaluate ZO-Curv on larger real-world serving clusters.

AI Use Statement

In this work, we used generative AI tools to implement methods, provide feedback on experimental design, interpret results, and assist with translation. We did not use generative AI tools to formulate the convergence results or to develop the core hypothesis or conceptual framework of ZO-Curv. These contributions originated with the authors. Generating synthetic datasets, cleaning or reformatting datasets, and conducting qualitative data analysis are not applicable to this work. Additionally, we used generative AI tools to search for, identify, and summarize related literature, to format references, and to support brainstorming. We have reviewed all AI-assisted work. AI-assisted implementations were reviewed and validated through the training runs reported in the paper. The authors verified the existence and bibliographic details of every cited work and revised all AI-drafted text. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.

Reproducibility Statement

Section 3.2 and Algorithms 1 and 2 specify ZO-Curv completely, including the curvature estimator and the preconditioned update. Section 3.3 summarizes our theoretical results, and Appendix H states their assumptions and gives complete proofs. Section 4.1 describes the experimental protocol, and Appendix B lists the model configurations, all optimizer and estimator hyperparameters (Table 2), and the profiling protocol. Our implementation builds on the public nanoGPT code base used by Sophia [24], and all data are public: we pretrain on OpenWebText with the nanoGPT split and evaluate on standard benchmarks following the protocol of Appendix G. Appendix ?? describes the deployment on disjoint GPU groups. We will release our code upon acceptance.

Related work

Zeroth-order optimization. ZO methods build gradient surrogates from function evaluations alone, from SPSA [40] to the smoothing analysis of Nesterov & Spokoiny (2017) and its stochastic nonconvex extension [12]. Their rates degrade with the dimension, and this dependence is optimal up to constants for two-point estimators [11, 37]. MeZO [28] made ZO practical for LLM fine-tuning by regenerating perturbations from random seeds, and later work benchmarks ZO fine-tuning [49] and reduces its variance with adaptive momentum [17], sparse perturbations [26], and low-rank subspaces [7, 46]. DeepZero [6] trains convolutional networks from scratch with coordinate-wise estimates and pruning-guided sparsity, and forward gradients [2, 35] replace backpropagation with forward-mode directional derivatives, whose variance likewise grows with the number of parameters. InstructedZO [4] mixes ZO estimates into first-order gradients to pretrain on mobile devices. All of these use the estimate as the descent direction; § 2.2 shows that this fails at LLM pretraining scale, and ZO-Curv uses ZO queries only for curvature.

Zeroth-order curvature estimation. Second-order SPSA [41] estimates the Hessian from loss values alongside the gradient and uses it in a Newton-type update. HiZOO [50] estimates a diagonal Hessian with one extra forward pass per step and uses it to precondition ZO fine-tuning of LLMs. Both still take the descent direction from ZO estimates, in settings that avoid backpropagation altogether. ZO-Curv instead pairs a forward-only curvature estimate with the exact first-order gradient, targets the GNB diagonal of Sophia-G for pretraining, and computes the estimate on separate idle hardware.

Preconditioned optimizers for pretraining. Adaptive methods [20, 27] precondition with gradient moments, Adafactor [38] factors these moments to save memory, and K-FAC [30] and Shampoo [15] use structured curvature, with the generalized Gauss–Newton matrix as the principled curvature for likelihood losses [29]. More recent optimizers such as SOAP [43] and Muon [25] precondition LLM pretraining with matrix-level structure. AdaHessian [45] and Sophia [24] instead use a diagonal Hessian estimate, and Sophia shows that a clipped, EMA-smoothed estimate accelerates LLM pretraining, but these estimators require extra backward passes on the training node. The closest systems precedent is Distributed Shampoo [1], which computes the inverse roots of its preconditioners asynchronously on otherwise underused CPUs and refreshes them only every few hundred steps. Its statistics, however, are still gathered from backpropagated gradients on the accelerators, and only the matrix roots are offloaded. ZO-Curv moves the curvature statistic itself to forward-only devices. It does not propose a new preconditioner: it matches SophiaG step for step (Appendix § 10).

Training on heterogeneous and idle compute. SWARM parallelism [36] and DiLoCo [10] train across poorly connected devices, Petals [3] runs collaborative inference and fine-tuning of large models, and Bamboo [42] trains on preemptible instances. Their training workers must all run backpropagation. On the serving side, inference engines such as vLLM [21] and DistServe [51] are built around forward passes, and Splitwise [33] and Mélange [14] make inference clusters more efficient. Lyra [22] loans idle inference servers to training jobs during low-traffic periods, Valve [23] colocates offline work with online serving, and Runway reclaims idle inference GPUs for research jobs [9]. These systems hand idle serving capacity to other workloads, and loaned servers must run full training. ZO-Curv instead lets forward-only capacity accelerate the same pretraining run, without backpropagation on the serving devices.

Experimental details

Models and data. We build on the nanoGPT implementation used by Sophia [24]: decoder-only transformers with GELU activations, no bias terms, and no dropout, with the configurations of Table 2. For the 355M and larger models we scale attention by the inverse of the layer index, as Sophia does to keep AdamW stable at these sizes. All models are trained from scratch on OpenWebText [13] with the GPT-2 tokenizer and the train and validation split of nanoGPT (9B and 4.4M tokens), at context length 1024 and global batch size 512, in bfloat16.

Modeldmodeld_{\mathrm{model}}HeadsDepthAdamW lrSophiaG / ZO-Curv lrγ\gamma
GPT-2 Small (125M)76812126×10−46 \times10^{-4}6×10−46 \times10^{-4}0.05
GPT-2 Medium (355M)102416243×10−43 \times10^{-4}7×10−47 \times10^{-4}0.08
GPT-2 Large (770M)128020362×10−42 \times10^{-4}3×10−43 \times10^{-4}0.05
GPT-NeoX (1.5B)153624481.5×10−41.5 \times10^{-4}1.2×10−41.2 \times10^{-4}0.05

Table 2. Model configurations and peak learning rates. ZO-Curv uses the SophiaG hyperparameters unchanged; γ\gamma is the curvature scale of equation (3).

Optimization. All methods use a cosine learning-rate schedule with 2K warmup steps, decaying to a minimum between 10−510^{-5} and 6×10−56 \times10^{-5}, and gradient clipping at norm 1.0. AdamW uses β=(0.9,0.95)\beta= (0.9, 0.95) and weight decay 0.1. ZO-Curv and SophiaG use β=(0.965,0.99)\beta= (0.965, 0.99), weight decay 0.2, clipping threshold ρ=1\rho= 1, and the curvature scale γ\gamma of Table 2, following Sophia, and refreshes its curvature at every step. Step budgets are given in § 4.1.

Curvature estimator. Unless stated otherwise, ZO-Curv uses q=512q = 512 queries per refresh with finite-difference radius ε=10−4\varepsilon= 10^{-4}, and samples the pseudo-labels from the model’s own predictive distribution, as in the GNB estimator. Following the Sophia implementation, the scale BB in hZO=Bzˉ⊙z~h^{\mathrm{ZO}} = B \bar{z} \odot\tilde{z} counts the tokens of the curvature batch, and the denominator floor is ϵ0=10−15\epsilon_0 = 10^{-15}.

Profiling. The measurements of § 2.3 report peak reserved GPU memory at batch size 8, with forward-only execution in fp16 and training in mixed precision, and per-step time averaged over 100 steps after warmup.

Step-axis comparison with SophiaG

Figure 10 replots the runs of Figure 5 on a step axis. ZO-Curv and SophiaG are nearly indistinguishable at every size, confirming that the forward-only estimator matches the backpropagated GNB estimator in per-step optimization quality. The contribution of ZO-Curv is not a stronger preconditioner but a preconditioner obtained from hardware that could not otherwise participate in training, which is why the two methods separate on the wall-clock axis of Figure 5.

Figure 5. Main result (RQ1): forward-only curvature turns step efficiency into wall-clock speedup (OpenWebText; AdamW receives a doubled step budget). ZO-Curv reaches AdamW’s final loss in roughly half the time at every size, whereas SophiaG spends its step advantage on curvature computed on the training node.

Figure 10. ZO-Curv matches SophiaG’s step efficiency at every scale (first 20K steps of the runs of Figure 5). The two curves are nearly indistinguishable on a step axis: the forward-only estimator matches the backpropagated one in per-step quality, and the advantage of ZO-Curv lies in where the estimate is computed.

SophiaG with a ten-step refresh interval

Our main comparison refreshes the curvature of SophiaG at every step, so that it consumes curvature as fresh as that of ZO-Curv (§ 4.1). The original Sophia instead refreshes every k=10k = 10 steps to amortize the cost of the GNB estimator. Figure 11 reports the time each method needs for a fixed budget of 20K steps under this setting, using the per-step measurements behind Figure 5. A ten-step interval brings SophiaG to 9–13% above AdamW per step, while ZO-Curv stays within 1–5% of AdamW because its estimate is computed off the training node. SophiaG therefore remains 6–12% slower than ZO-Curv at every size, and it obtains this reduction by refreshing its curvature ten times less often, whereas ZO-Curv refreshes at every step.

Figure 11. SophiaG with a ten-step refresh interval. Wall-clock time to complete 20K steps for AdamW, SophiaG refreshing its curvature every ten steps, and ZO-Curv refreshing at every step. Even with its curvature cost amortized, SophiaG is slower than ZO-Curv at every size.

The staler curvature also costs accuracy (Figure 12). After 20K steps, SophiaG with a ten-step interval ends 0.03–0.04 above its every-step counterpart in the main runs and 0.02–0.04 above ZO-Curv at every size. Amortizing the estimator on the training node thus leaves SophiaG both slower and less accurate than ZO-Curv, which keeps the curvature fresh at nearly the per-step cost of AdamW.

Figure 12. Loss of SophiaG with a ten-step refresh interval. Validation loss after 20K steps for the runs of Figure 11. With its curvature refreshed ten times less often, SophiaG ends above ZO-Curv at every size.

Training trajectories across query budgets

Figure 13 extends the trajectory panels of Figure 7(c,d) to all four model sizes. The qualitative picture is uniform: every query budget separates from AdamW early, and increasing qq lowers the entire trajectory. The sizes differ only in where the budgets stop mattering, consistent with the saturation points of Figure 7(a,b).

Figure 13. Training trajectories across query budgets, all four sizes (companion to Figure 7). At 125M the budgets collapse onto each other early, whereas at 770M and 1.5B they remain cleanly ordered, matching the scale-dependent saturation of Figure 7(a,Figure 7b).

Effect of the one-step pipeline offset

The pipeline of § 3.1 applies the curvature estimated at θt−1\theta_{t-1} to the update at step tt. Figure 14 compares this deployed variant against a synchronous control that waits for the curvature of the current snapshot θt\theta_t. The two loss curves coincide throughout training: consecutive snapshots differ by a single clipped update, and the curvature EMA smooths over far longer horizons than one step, so the offset costs nothing while removing the serving pool from the critical path.

Figure 14. The one-step pipeline offset is free (GPT-2 355M, q=512q = 512). Training with curvature estimated at θt−1\theta_{t-1} (pipelined, as deployed) is indistinguishable from using curvature estimated at the current θt\theta_t (synchronous).

Full downstream results

Table 3 lists the per-task accuracies behind Figure 9. We evaluate the 770M checkpoint of each method after 20K steps, the shared budget of Table 1.

MethodHellaSwag BaseHellaSwag FTARC-E BaseARC-E FTARC-C BaseARC-C FTLAMBADA BaseLAMBADA FTAverage BaseAverage FT
AdamW31.433.040.250.922.126.130.8–31.135.2
SophiaG35.637.147.554.725.828.139.1–37.039.8
ZO-Curv (ours)35.236.347.454.226.828.839.0–37.139.6

Table 3. Full downstream evaluation of the GPT-2 770M checkpoints at 20K steps (companion to Figure 9). Accuracy in %. Base: few-shot before finetuning (10 examples for HellaSwag, 25 for ARC-E and ARC-C, none for LAMBADA). FT: zero-shot after per-task LoRA finetuning. LAMBADA has no public training split and is not finetuned; it enters the FT average with its untuned score.

Before finetuning. We use few-shot prompting with 10 in-context examples for HellaSwag, 25 for ARC-Easy and ARC-Challenge, and none for LAMBADA, drawn from each training split with a fixed seed. For the three multiple-choice tasks, the prediction is the choice with the highest average log-likelihood per answer token, evaluated on the validation split (10,042, 570, and 299 examples). LAMBADA is scored by whether greedy decoding recovers the final word, on the 5,153 test examples of the OpenAI version.

After finetuning. Starting from each checkpoint, we finetune separately on the training split of every multiple-choice task with LoRA (rank 8, α=16\alpha= 16, dropout 0.05) on the attention and MLP projections, which trains 5.9M parameters (0.76% of the model). We use AdamW with learning rate 2×10−52 \times10^{-5} and batch size 8 for 2,000 steps, with the loss restricted to the answer tokens, and then evaluate on the validation split. LAMBADA has no public training split, so it is not finetuned.

Results. Before finetuning, ZO-Curv improves over AdamW on every task, by 6.0 points on average (37.1 versus 31.1). It stays within 0.4 points of SophiaG on HellaSwag, ARC-Easy, and LAMBADA, and is 1.0 point higher on ARC-Challenge. Finetuning helps the AdamW checkpoint most, by 4.1 points on average against 2.8 for SophiaG and 2.5 for ZO-Curv, with the largest gain on ARC-Easy (10.7 points). It nonetheless trails both second-order checkpoints on every task after finetuning, while ZO-Curv remains within 0.8 points of SophiaG throughout (39.6 versus 39.8 on average).

Proofs

This appendix provides the full theoretical analysis of ZO-Curv. Our goal is to isolate the effect of replacing the backpropagation-based Gauss–Newton–Bartlett (GNB) curvature estimator in Sophia-G with a forward-only zeroth-order estimator. The analysis proceeds in three stages. First, we identify what the average-then-square ZO estimator estimates: under a finite query budget qq, it is an unbiased estimate of a damped GNB diagonal whose damping decreases as 1/q1/q, and its fluctuation is of the order of this target in every coordinate, independently of the dimension dd. Second, we show that the curvature EMA averages these fluctuations into a bounded multiplicative perturbation of the damped reference preconditioner, and that such a perturbation does not destroy descent: the update direction remains a descent direction and converges to stationarity. Third, we combine the two results into a convergence guarantee relative to the damped-GNB-preconditioned reference update.

As in the theoretical analysis of Sophia [24], we analyze a deterministic clipped preconditioned core in order to isolate the role of curvature estimation. In particular, we omit gradient momentum and decoupled weight decay from the convergence proof. The curvature EMA is retained and updated at every iteration, matching the per-step estimator in § 3. We first analyze the synchronous core, in which the update at step tt uses the curvature queried at θt\theta_t; Remark 2 shows that the results carry over unchanged to the pipelined update with any staleness ss.

Notation and reference curvature estimator

Let

F(θ)(7)F(\theta) \tag*{(7)}

denote the training objective, with parameters θ∈Rd\theta\in\mathbb{R}^d. At iteration tt, let

gt:=∇F(θt)(8)g_t := \nabla F(\theta_t) \tag*{(8)}

denote the first-order gradient used to update the model.

The curvature estimator is computed from a sampled-label mini-batch, as in the GNB estimator of Sophia-G. Given a mini-batch {xb}b=1B\{x_b\}_{b=1}^{B}, let

y^b∼softmax⁡(f(θt,xb)),b=1,…,B,(9)\hat{y}_b \sim\operatorname{softmax}(f(\theta_t, x_b)), \qquad b=1,\ldots,B, \tag*{(9)}

and define the sampled-label loss

ℓtsamp(θ):=1B∑b=1Bℓce(f(θ,xb),y^b).(10)\ell_t^{\mathrm{samp}}(\theta) := \frac{1}{B}\sum_{b=1}^{B}\ell_{\mathrm{ce}}(f(\theta,x_b),\hat{y}_b). \tag*{(10)}

We denote its gradient at θt\theta_t by

ct:=∇ℓtsamp(θt).(11)c_t := \nabla\ell_t^{\mathrm{samp}}(\theta_t). \tag*{(11)}

Conditioned on the sampled mini-batch and pseudo-labels, ctc_t is fixed throughout the qq ZO queries.

The backpropagation-based GNB estimator used as our reference is

htG:=B ct⊙ct,(12)h_t^{G} := B\,c_t \odot c_t, \tag*{(12)}

where ⊙\odot denotes element-wise multiplication. Taking expectation over the sampled labels recovers the diagonal of the mini-batch Gauss–Newton matrix, as established by the Bartlett identities in [24]. Our analysis below compares the ZO estimate with this reference conditioned on the same mini-batch and the same sampled labels.

A finite query budget adds a uniform damping to this target (Proposition 1). We therefore define the damped GNB diagonal

h~t:=(1+1q)htG+λt1,λt:=∥htG∥1q=B∥ct∥22q,(13)\widetilde{h}_t := \left(1+\frac{1}{q}\right)h_t^{G}+\lambda_t\mathbf{1}, \qquad\lambda_t := \frac{\lVert h_t^{G}\rVert_1}{q}=\frac{B\lVert c_t\rVert_2^2}{q}, \tag*{(13)}

where 1∈Rd\mathbf{1}\in\mathbb{R}^d is the all-ones vector. This is a Levenberg–Marquardt-type damping of the GNB diagonal, of the kind commonly added to curvature preconditioners for numerical stability. Here it is not a tuned hyperparameter but is induced by the finite budget, and h~t→htG\widetilde{h}_t \to h_t^{G} as q→∞q\to\infty. Adding a constant to every coordinate preserves the ordering of the coordinates by curvature.

The reference is coupled to the ZO-Curv trajectory: at every iteration, h~t\widetilde{h}_t is evaluated at the ZO-Curv iterate θt\theta_t, on the same mini-batch and with the same sampled labels as the ZO queries. It is never computed by the algorithm and serves only as a comparison point. Define the reference curvature EMA, initialized at zero as in ZO-Curv, as

v~t=β2v~t−1+(1−β2)h~t,v~0=0.(14)\widetilde{v}_t = \beta_2\widetilde{v}_{t-1}+(1-\beta_2)\widetilde{h}_t,\qquad\widetilde{v}_0=0. \tag*{(14)}

The corresponding reference denominator is

dt,i:=max⁡{v~t,i,ϵ0},Dt:=Diag⁡(dt).(15)d_{t,i}:=\max\{\widetilde{v}_{t,i},\epsilon_0\},\qquad D_t:=\operatorname{Diag}(d_t). \tag*{(15)}

where ϵ0>0\epsilon_0>0 is the numerical denominator floor. We use ϵ\epsilon exclusively for the ZO finite-difference radius, to distinguish it from ϵ0\epsilon_0.

ZO approximation of the GNB curvature

At each iteration, ZO-Curv draws qq independent Gaussian perturbations

ut,j∼i.i.d.N(0,Id),j=1,…,q.(16)u_{t,j} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0,I_d), \qquad j=1,\ldots,q. \tag*{(16)}

Importantly, all qq queries use the same mini-batch and the same sampled pseudo-labels, and hence share the same curvature gradient ctc_t.

For direction ut,ju_{t,j}, define the central-difference directional derivative

Δt,jε:=ℓtsamp(θt+εut,j)−ℓtsamp(θt−εut,j)2ε.(17)\Delta_{t,j}^{\varepsilon} := \frac{\ell_t^{\mathrm{samp}}(\theta_t+\varepsilon u_{t,j})-\ell_t^{\mathrm{samp}}(\theta_t-\varepsilon u_{t,j})}{2\varepsilon}. \tag*{(17)}

The corresponding ZO gradient estimate is

z~t,jε:=Δt,jεut,j.(18)\tilde{z}_{t,j}^{\varepsilon} := \Delta_{t,j}^{\varepsilon}u_{t,j}. \tag*{(18)}

ZO-Curv first averages the qq gradient estimates,

zˉtε:=1q∑j=1qz~t,jε,(19)\bar{z}_t^{\varepsilon} := \frac{1}{q}\sum_{j=1}^{q}\tilde{z}_{t,j}^{\varepsilon}, \tag*{(19)}

and then squares the averaged estimate:

htZO:=Bzˉtε⊙zˉtε.(20)h_t^{\mathrm{ZO}} := B\bar{z}_t^{\varepsilon}\odot\bar{z}_t^{\varepsilon}. \tag*{(20)}

We first study the ideal directional-derivative limit ε→0\varepsilon\to0.

Lemma 1 (Moments of the averaged ZO gradient). Fix a vector c∈Rdc\in\mathbb{R}^{d} and let u1,…,uq∼i.i.d.N(0,Id)u_1,\ldots,u_q\overset{\mathrm{i.i.d.}}{\sim}\mathcal{N}(0,I_d). Define

zj:=(cTuj)uj,zˉ:=1q∑j=1qzj.(21)z_j := (c^{\mathsf{T}}u_j)u_j,\qquad\bar{z}:=\frac{1}{q}\sum_{j=1}^{q}z_j. \tag*{(21)}

Then, for every coordinate ii,

E[zˉi∣c]=ci,(22)\mathbb{E}[\bar{z}_i\mid c]=c_i, \tag*{(22)}
E[zˉi2∣c]=(1+1q)ci2+1q∥c∥22,(23)\mathbb{E}[\bar{z}_i^2\mid c]=\left(1+\frac{1}{q}\right)c_i^2+\frac{1}{q}\lVert c\rVert_2^2, \tag*{(23)}
Var⁡(zˉi∣c)=∥c∥22+ci2q.(24)\operatorname{Var}(\bar{z}_i\mid c)=\frac{\lVert c\rVert_2^2+c_i^2}{q}. \tag*{(24)}

Proof. For a single Gaussian direction,

E[(cTu)ui]=∑r=1dcrE[urui]=ci,(25)\mathbb{E}[(c^{\mathsf{T}}u)u_i]=\sum_{r=1}^{d}c_r\mathbb{E}[u_ru_i]=c_i, \tag*{(25)}

which proves equation 22.

For the second moment, Isserlis’ identity for a standard Gaussian gives

E[(cTu)2ui2]=∥c∥22+2ci2.(26)\mathbb{E}[(c^{\mathsf{T}}u)^2u_i^2]=\lVert c\rVert_2^2+2c_i^2. \tag*{(26)}

Therefore

Var⁡((cTu)ui)=∥c∥22+ci2.(27)\operatorname{Var}((c^{\mathsf{T}}u)u_i)=\lVert c\rVert_2^2+c_i^2. \tag*{(27)}

Averaging qq independent directions divides the variance by qq, hence

Var⁡(zˉi∣c)=∥c∥22+ci2q.(28)\operatorname{Var}(\bar{z}_i\mid c)=\frac{\lVert c\rVert_2^2+c_i^2}{q}. \tag*{(28)}

Combining this with E[zˉi∣c]=ci\mathbb{E}[\bar{z}_i\mid c]=c_i yields

E[zˉi2∣c]=ci2+∥c∥22+ci2q(29)\mathbb{E}[\bar{z}_i^2\mid c]=c_i^2+\frac{\lVert c\rVert_2^2+c_i^2}{q} \tag*{(29)}
=(1+1q)ci2+1q∥c∥22.(30)=\left(1+\frac{1}{q}\right)c_i^2+\frac{1}{q}\lVert c\rVert_2^2. \tag*{(30)}

Lemma 1 shows that the averaged ZO gradient itself approaches cc at the canonical q−1/2q^{-1/2} stochastic rate. Its second moment equation equation (23) is the source of the damping in equation (13). For the convergence analysis, however, we require a high-probability rather than an expectation bound.

Lemma 2 (High-probability ZO gradient error). Under the setting of Lemma 1, there exists a universal constant C0>0C_0 > 0 such that, for any failure probability ζ∈(0,1)\zeta\in(0,1), with probability at least 1−ζ1-\zeta,

∥zˉ−c∥∞≤C0∥c∥2[log⁡(2d/ζ)q+log⁡(2d/ζ)q].(31)\lVert\bar{z}-c\rVert_{\infty} \le C_0\lVert c\rVert_2\left[\sqrt{\frac{\log(2d/\zeta)}{q}}+\frac{\log(2d/\zeta)}{q}\right]. \tag*{(31)}

Proof. Fix a coordinate ii and define

Xj,i:=(c⊤uj)uj,i−ci.(32)X_{j,i}:=(c^{\top}u_j)u_{j,i}-c_i. \tag*{(32)}

We have E[Xj,i]=0\mathbb{E}[X_{j,i}]=0. Moreover, c⊤ujc^{\top}u_j is a centered Gaussian with variance ∥c∥22\lVert c\rVert_2^2, and uj,iu_{j,i} is standard Gaussian. The product of two sub-Gaussian random variables is sub-exponential, even when the two variables are correlated. Therefore, for a universal constant C>0C > 0,

∥Xj,i∥ψ1≤C∥c∥2.(33)\lVert X_{j,i}\rVert_{\psi_1}\le C\lVert c\rVert_2. \tag*{(33)}

Bernstein’s inequality for independent centered sub-exponential variables then gives

Pr⁡[∣1q∑j=1qXj,i∣>C0∥c∥2(sq+sq)]≤2e−s.(34)\Pr\left[\left\lvert\frac{1}{q}\sum_{j=1}^{q}X_{j,i}\right\rvert>C_0\lVert c\rVert_2\left(\sqrt{\frac{s}{q}}+\frac{s}{q}\right)\right]\le2e^{-s}. \tag*{(34)}

Setting

s=log⁡(2d/ζ)(35)s=\log(2d/\zeta) \tag*{(35)}

and applying a union bound over i=1,…,di=1,\ldots,d proves equation (31). □

We next account for the finite central-difference radius ε\varepsilon.

Assumption A1 (Local Hessian Lipschitzness). For every iteration tt, the sampled-label loss ℓtsamp\ell_t^{\mathrm{samp}} is twice differentiable with an LHL_H-Lipschitz Hessian on the ball Ut:={θ:∥θ−θt∥2≤εR}\mathcal{U}_t:=\{\theta:\lVert\theta-\theta_t\rVert_2\le\varepsilon R\} containing all ZO query points:

∥∇2ℓtsamp(θ)−∇2ℓtsamp(θ′)∥2≤LH∥θ−θ′∥2,θ,θ′∈Ut.(36)\lVert\nabla^2\ell_t^{\mathrm{samp}}(\theta)-\nabla^2\ell_t^{\mathrm{samp}}(\theta')\rVert_2\le L_H\lVert\theta-\theta'\rVert_2,\qquad\theta,\theta'\in\mathcal{U}_t. \tag*{(36)}

The query points are θt±εut,j\theta_t\pm\varepsilon u_{t,j} with ∥ut,j∥2≈d\lVert u_{t,j}\rVert_2\approx\sqrt{d}, so the relevant radius is εmax⁡j∥ut,j∥2\varepsilon\max_j\lVert u_{t,j}\rVert_2 rather than ε\varepsilon; we take R=RqR=R_q in Proposition 1 and R=RTR=R_T in Theorem 3, where RqR_q and RTR_T bound max⁡j∥ut,j∥2\max_j\lVert u_{t,j}\rVert_2 with high probability.

Lemma 3 (Central-difference error). Under Assumption A1, for any direction uu with ∥u∥2≤R\lVert u\rVert_2\le R,

∣ℓtsamp(θt+εu)−ℓtsamp(θt−εu)2ε−ct⊤u∣≤LHε26∥u∥23.(37)\left\lvert\frac{\ell_t^{\mathrm{samp}}(\theta_t+\varepsilon u)-\ell_t^{\mathrm{samp}}(\theta_t-\varepsilon u)}{2\varepsilon}-c_t^{\top}u\right\rvert\le\frac{L_H\varepsilon^2}{6}\lVert u\rVert_2^3. \tag*{(37)}

Proof. Define the scalar function

ϕ(s):=ℓtsamp(θt+su),∣s∣≤ε.(38)\phi(s):=\ell_t^{\mathrm{samp}}(\theta_t+su),\qquad\lvert s\rvert\le\varepsilon. \tag*{(38)}

Then ϕ′(0)=ct⊤u\phi'(0)=c_t^{\top}u and ϕ′′(s)=u⊤∇2ℓtsamp(θt+su)u\phi''(s)=u^{\top}\nabla^2\ell_t^{\mathrm{samp}}(\theta_t+su)u, so Assumption A1 gives

∣ϕ′′(s)−ϕ′′(0)∣≤LH∣s∣∥u∥23.(39)\lvert\phi''(s)-\phi''(0)\rvert\le L_H\lvert s\rvert\lVert u\rVert_2^3. \tag*{(39)}

Taylor’s theorem with integral remainder, ϕ(±ε)=ϕ(0)±εϕ′(0)+∫0±ε(±ε−s)ϕ′′(s) ds\phi(\pm\varepsilon)=\phi(0)\pm\varepsilon\phi'(0)+\int_0^{\pm\varepsilon}(\pm\varepsilon-s)\phi''(s)\,ds, then yields

∣ϕ(±ε)−ϕ(0)∓εϕ′(0)−ε22ϕ′′(0)∣≤LH∥u∥23∫0ε(ε−s)s ds=LHε36∥u∥23.(40)\left\lvert\phi(\pm\varepsilon)-\phi(0)\mp\varepsilon\phi'(0)-\frac{\varepsilon^2}{2}\phi''(0)\right\rvert\le L_H\lVert u\rVert_2^3\int_0^\varepsilon(\varepsilon-s)s\,ds=\frac{L_H\varepsilon^3}{6}\lVert u\rVert_2^3. \tag*{(40)}

Subtracting the two expansions cancels ϕ(0)\phi(0) and ϕ′′(0)\phi''(0), and dividing by 2ε2\varepsilon gives

∣ϕ(ε)−ϕ(−ε)2ε−ϕ′(0)∣≤LHε26∥u∥23.(41)\left\lvert\frac{\phi(\varepsilon)-\phi(-\varepsilon)}{2\varepsilon}-\phi'(0)\right\rvert\le\frac{L_H\varepsilon^2}{6}\lVert u\rVert_2^3. \tag*{(41)}

This proves the result; no third derivative is required. □

We now combine Lemmas 1–3 into the properties of the curvature estimate used in the rest of the analysis. Write

htZO,0:=B zˉt0⊙zˉt0,zˉt0:=1q∑j=1q(ctTut,j)ut,j,(42)h_t^{\mathrm{ZO},0} := B\,\bar{z}_t^0 \odot\bar{z}_t^0,\qquad\bar{z}_t^0 := \frac{1}{q}\sum_{j=1}^{q}(c_t^\mathsf{T}u_{t,j})u_{t,j}, \tag*{(42)}

for the ideal estimate with exact directional derivatives, and htZOh_t^{\mathrm{ZO}} for the central-difference estimate equation 20.

Proposition 1 (Forward-only estimation of damped GNB curvature). Fix an iteration tt and condition on its mini-batch and sampled labels, with ct≠0c_t \ne0.

(i) Unbiasedness. The ideal estimate is unbiased for the damped target equation 13:

E[ht,iZO,0]=h~t,i.(43)\mathbb{E}\left[h_{t,i}^{\mathrm{ZO},0}\right] = \tilde{h}_{t,i}. \tag*{(43)}

(ii) Relative second moment. There is a universal constant Cv≥1C_v \ge1 such that, for every coordinate ii,

E[(ht,iZO,0)2]≤Cvh~t,i2.(44)\mathbb{E}\left[\left(h_{t,i}^{\mathrm{ZO},0}\right)^2\right] \le C_v\tilde{h}_{t,i}^2. \tag*{(44)}

(iii) Relative envelope and finite differences. For ζ∈(0,1)\zeta\in(0,1), define

Λ:=log⁡4dζ,Rq:=d+2log⁡4qζ,\Lambda:= \log\frac{4d}{\zeta},\qquad R_q := \sqrt{d}+\sqrt{2\log\frac{4q}{\zeta}},
Mq,d:=2log⁡8qdζ,ϑ:=LHε26Rq3Mq,d,(45)M_{q,d} := \sqrt{2\log\frac{8qd}{\zeta}},\qquad\vartheta:= \frac{L_H\varepsilon^2}{6}R_q^3M_{q,d}, \tag*{(45)}

and suppose q≥Λq \ge\Lambda and that Assumption A1 holds with R=RqR=R_q. Then, with probability at least 1−ζ1-\zeta, simultaneously for all coordinates ii,

ht,iZO,0≤CbΛh~t,i,Cb:=max⁡{2,8C02},(46)h_{t,i}^{\mathrm{ZO},0} \le C_b\Lambda\tilde{h}_{t,i},\qquad C_b := \max\{2,8C_0^2\}, \tag*{(46)}
∣ht,iZO−ht,iZO,0∣≤2CbΛBϑ2h~t,i+Bϑ2,(47)\left|h_{t,i}^{\mathrm{ZO}}-h_{t,i}^{\mathrm{ZO},0}\right| \le2\sqrt{C_b\Lambda B\vartheta^2\tilde{h}_{t,i}+B\vartheta^2}, \tag*{(47)}

where C0C_0 is the constant of Lemma 2.

Proof. For (i), Lemma 1 with c=ctc=c_t gives

E[ht,iZO,0]=BE[(zˉt,i0)2]=(1+1q)Bct,i2+B∥ct∥22q=h~t,i.(48)\mathbb{E}\left[h_{t,i}^{\mathrm{ZO},0}\right]=B\mathbb{E}\left[(\bar{z}_{t,i}^0)^2\right]=\left(1+\frac{1}{q}\right)Bc_{t,i}^2+\frac{B\|c_t\|_2^2}{q}=\tilde{h}_{t,i}. \tag*{(48)}

For (ii), write zˉt,i0=ct,i+ai\bar{z}_{t,i}^0=c_{t,i}+a_i with ai:=q−1∑j=1qXj,ia_i:=q^{-1}\sum_{j=1}^{q}X_{j,i} and Xj,iX_{j,i} as in the proof of Lemma 2. Since (x+y)4≤8(x4+y4)(x+y)^4\le8(x^4+y^4),

E[(zˉt,i0)4]≤8ct,i4+8E[ai4].(49)\mathbb{E}\left[(\bar{z}_{t,i}^0)^4\right]\le8c_{t,i}^4+8\mathbb{E}[a_i^4]. \tag*{(49)}

The Xj,iX_{j,i} are independent and centered, so

E[ai4]=qE[X1,i4]+3q(q−1)(E[X1,i2])2q4≤E[X1,i4]+3(E[X1,i2])2q2.(50)\mathbb{E}[a_i^4]=\frac{q\mathbb{E}[X_{1,i}^4]+3q(q-1)\left(\mathbb{E}[X_{1,i}^2]\right)^2}{q^4}\le\frac{\mathbb{E}[X_{1,i}^4]+3\left(\mathbb{E}[X_{1,i}^2]\right)^2}{q^2}. \tag*{(50)}

By Lemma 1, E[X1,i2]=∥ct∥22+ct,i2≤2∥ct∥22\mathbb{E}[X_{1,i}^2]=\|c_t\|_2^2+c_{t,i}^2\le2\|c_t\|_2^2, and since ∥X1,i∥ψ1≤C∥ct∥2\|X_{1,i}\|_{\psi_1}\le C\|c_t\|_2, also E[X1,i4]≤C′∥ct∥24\mathbb{E}[X_{1,i}^4]\le C'\|c_t\|_2^4 for a universal constant C′C'. Hence E[ai4]≤C′′∥ct∥24/q2\mathbb{E}[a_i^4]\le C''\|c_t\|_2^4/q^2 with C′′:=C′+12C'':=C'+12, and

E[(ht,iZO,0)2]=B2E[(zˉt,i0)4]≤8(Bct,i2)2+8C′′λt2(51)\mathbb{E}\left[\left(h_{t,i}^{\mathrm{ZO},0}\right)^2\right]=B^2\mathbb{E}\left[(\bar{z}_{t,i}^0)^4\right]\le8(Bc_{t,i}^2)^2+8C''\lambda_t^2 \tag*{(51)}
≤Cv(Bct,i2+λt)2≤Cvh~t,i2,(52)\le C_v(Bc_{t,i}^2+\lambda_t)^2\le C_v\tilde{h}_{t,i}^2, \tag*{(52)}

with Cv:=8max⁡{1,C′′}C_v:=8\max\{1,C''\}, where the last step uses Bct,i2+λt≤h~t,iBc_{t,i}^2+\lambda_t\le\tilde{h}_{t,i}. For (iii), we split the failure probability ζ\zeta into three events. First, Lemma 2 with failure probability ζ/2\zeta/2, for which log⁡(2d/(ζ/2))=Λ\log(2d/(\zeta/2))=\Lambda, gives ∣ai∣≤C0∥ct∥2(Λ/q+Λ/q)|a_i|\le C_0\lVert c_t\rVert_2(\sqrt{\Lambda/q}+\Lambda/q) for all ii. Since q≥Λq\ge\Lambda and (x+x)2≤4x(\sqrt{x}+x)^2\le4x for x≤1x\le1,

Bai2≤4C02ΛB∥ct∥22q=4C02Λλt.(53)Ba_i^2\le4C_0^2\Lambda\frac{B\lVert c_t\rVert_2^2}{q}=4C_0^2\Lambda\lambda_t. \tag*{(53)}

Therefore, using Λ≥1\Lambda\ge1,

ht,iZO,0=B(ct,i+ai)2≤2Bct,i2+8C02Λλt≤CbΛh~t,i,(54)h_{t,i}^{\mathrm{ZO},0}=B(c_{t,i}+a_i)^2\le2Bc_{t,i}^2+8C_0^2\Lambda\lambda_t\le C_b\Lambda\widetilde h_{t,i}, \tag*{(54)}

which is equation (46).

Second, Gaussian norm concentration, Pr⁡[∥u∥2≥d+s]≤e−s2/2\Pr[\lVert u\rVert_2\ge\sqrt{d}+s]\le e^{-s^2/2}, and a union bound over the qq directions give, with probability at least 1−ζ/41-\zeta/4, max⁡j≤q∥ut,j∥2≤Rq\max_{j\le q}\lVert u_{t,j}\rVert_2\le R_q. Third, the Gaussian tail bound Pr⁡[∣uj,i∣≥s]≤2e−s2/2\Pr[|u_{j,i}|\ge s]\le2e^{-s^2/2} and a union bound over all qdqd coordinates give, with probability at least 1−ζ/41-\zeta/4, max⁡j≤q, i≤d∣ut,j,i∣≤Mq,d\max_{j\le q,\,i\le d}|u_{t,j,i}|\le M_{q,d}. On the second event every query point lies in Ut\mathcal U_t with R=RqR=R_q, so Lemma 3 applies to every direction and

z~t,iε−z~t,i0=1q∑j=1qet,j,iFD,∣et,j,iFD∣≤LHε26∥ut,j∥22∣ut,j,i∣.(55)\widetilde z_{t,i}^{\varepsilon}-\widetilde z_{t,i}^{0}=\frac{1}{q}\sum_{j=1}^{q}e_{t,j,i}^{\mathrm{FD}},\qquad|e_{t,j,i}^{\mathrm{FD}}|\le\frac{LH\varepsilon^2}{6}\lVert u_{t,j}\rVert_2^2|u_{t,j,i}|. \tag*{(55)}

On both events, ei:=z~t,iε−z~t,i0e_i:=\widetilde z_{t,i}^{\varepsilon}-\widetilde z_{t,i}^{0} satisfies ∣ei∣≤ϑ|e_i|\le\vartheta. Then

∣ht,iZO−ht,iZO,0∣=B∣2z~t,iZO,0ei+ei2∣≤2Bht,iZO,0 ϑ+Bϑ2,(56)|h_{t,i}^{\mathrm{ZO}}-h_{t,i}^{\mathrm{ZO},0}|=B|2\widetilde z_{t,i}^{\mathrm{ZO},0}e_i+e_i^2|\le2\sqrt{Bh_{t,i}^{\mathrm{ZO},0}}\,\vartheta+B\vartheta^2, \tag*{(56)}

and equation (46) gives equation (47). A union bound over the three events, with total failure probability ζ/2+ζ/4+ζ/4=ζ\zeta/2+\zeta/4+\zeta/4=\zeta, completes the proof. □

Proposition 1 shows that, like the GNB estimator of Sophia-G, the forward-only estimate is unbiased; its target is the GNB diagonal with the additional uniform damping λt=∥htG∥1/q\lambda_t=\lVert h_t^G\rVert_1/q. Averaging before squaring is what keeps this damping small: squaring each directional estimate before averaging would give E[B(Δt,j0ut,j,i)2]=2ht,iG+∥htG∥1\mathbb E[B(\Delta_{t,j}^{0}u_{t,j,i})^2]=2h_{t,i}^G+\lVert h_t^G\rVert_1, a damping qq times larger that does not decrease with the budget. Moreover, by (ii) and (iii), the fluctuation of the estimate around its target is at most of the order of the target itself in every coordinate, with constants independent of dd, qq, and ctc_t. This relative control is what the curvature EMA exploits next. Dividing equation (47) by h~t,i\widetilde h_{t,i} shows that the finite-difference perturbation is also relative, of order ΛBϑ2/h~t,i=O(ε2)\sqrt{\Lambda B\vartheta^2/\widetilde h_{t,i}}=O(\varepsilon^2).

Effect of curvature EMA

The optimizer does not use a single raw curvature estimate directly. Instead, both Sophia-G and ZO-Curv maintain a curvature EMA. Because fresh directions are drawn at every refresh, the EMA averages the relative fluctuations of Proposition 1 over its effective window 1/(1−β2)1/(1-\beta_2).

At every iteration, define

vtZO=β2vt−1ZO+(1−β2)htZO,v0ZO=0,(57)v_t^{\mathrm{ZO}}=\beta_2v_{t-1}^{\mathrm{ZO}}+(1-\beta_2)h_t^{\mathrm{ZO}},\qquad v_0^{\mathrm{ZO}}=0, \tag*{(57)}

and let wt,s:=(1−β2)β2t−sw_{t,s}:=(1-\beta_2)\beta_2^{t-s} and Wt:=∑s=1twt,s=1−β2tW_t:=\sum_{s=1}^{t}w_{t,s}=1-\beta_2^t, so that vtZO=∑s≤twt,shsZOv_t^{\mathrm{ZO}}=\sum_{s\le t}w_{t,s}h_s^{\mathrm{ZO}} and v~t=∑s≤twt,sh~s\widetilde v_t=\sum_{s\le t}w_{t,s}\widetilde h_s. Let Gs\mathcal G_s denote the history up to and including θs\theta_s and the step-ss mini-batch and sampled labels, but not the step-ss directions. Then csc_s and h~s\widetilde h_s are Gs\mathcal G_s-measurable, the directions us,1,…,us,qu_{s,1},\ldots,u_{s,q} are independent of Gs\mathcal G_s, and hsZOh_s^{\mathrm{ZO}} is Gs+1\mathcal G_{s+1}-measurable.

Assumption A2 (Slowly varying damped curvature). Over the analyzed horizon s≤Ts\le T, there exist κ≥1\kappa\ge1 and deterministic H‾i>0\overline H_i>0 such that

H‾iκ≤h~s,i≤H‾i,s≤T, i=1,…,d.(58)\frac{\overline H_i}{\kappa}\le\widetilde h_{s,i}\le\overline H_i,\qquad s\le T,\ i=1,\ldots,d. \tag*{(58)}

almost surely along the ZO-Curv trajectory.

Since h~s,i≥λs\tilde{h}_{s,i} \ge\lambda_s, the damping keeps every coordinate away from zero, and the assumption only asks that the damped curvature of each coordinate varies by at most a factor κ\kappa over the horizon. Let H:=min⁡iHˉiH := \min_i \bar{H}_i.

Lemma 4 (EMA concentration). Suppose Assumptions A1 and A2 hold over TT iterations. For ζ∈(0,1)\zeta\in(0,1), define

ΛT:=log⁡16qdTζ,RT:=d+2log⁡8qTζ,ϑT:=LHϵ26RT32ΛT,ϱT:=κBϑT2H.(59)\begin{aligned} \Lambda_T &:= \log\frac{16qdT}{\zeta}, & R_T &:= \sqrt{d}+\sqrt{2\log\frac{8qT}{\zeta}},\\ \vartheta_T &:= \frac{L_H\epsilon^2}{6}R_T^3\sqrt{2\Lambda_T}, & \varrho_T &:= \frac{\kappa B\vartheta_T^2}{H}. \tag*{(59)} \end{aligned}

take R=RTR=R_T in Assumption A1, and suppose q≥ΛTq \ge\Lambda_T. Let t0:=⌈ln⁡2/(1−β2)⌉t_0 := \lceil\ln2/(1-\beta_2)\rceil and

δT:=2κ[2Cv(1−β2)ΛT+43Cb(1−β2)ΛT2]+Cvζ2T+2CbΛTϱT+ϱT.(60)\delta_T := 2\kappa\left[\sqrt{2C_v(1-\beta_2)\Lambda_T}+\frac{4}{3}C_b(1-\beta_2)\Lambda_T^2\right]+\sqrt{\frac{C_v\zeta}{2T}}+2\sqrt{C_b\Lambda_T\varrho_T}+\varrho_T. \tag*{(60)}

Then, with probability at least 1−ζ1-\zeta, simultaneously for all t0≤t≤Tt_0 \le t \le T and all coordinates ii,

∣vt,iZO−v~t,i∣≤δTv~t,i.(61)\left|v^{\mathrm{ZO}}_{t,i}-\tilde{v}_{t,i}\right|\le\delta_T\tilde{v}_{t,i}. \tag*{(61)}

Proof. For each s≤Ts\le T, let Es\mathcal{E}_s be the event of Proposition 1(iii) at step ss with failure probability ζ/(2T)\zeta/(2T). Its constants satisfy log⁡(8dT/ζ)≤ΛT\log(8dT/\zeta)\le\Lambda_T, Rq=RTR_q=R_T, and Mq,d=2log⁡(16qdT/ζ)=2ΛTM_{q,d}=\sqrt{2\log(16qdT/\zeta)}=\sqrt{2\Lambda_T}, so the bounds of Proposition 1(iii) hold with ΛT\Lambda_T and ϑT\vartheta_T. Because the step-ss directions are independent of Gs\mathcal{G}_s, the proposition applies conditionally on Gs\mathcal{G}_s, and Pr⁡[Es∣Gs]≤ζ/(2T)\Pr[\mathcal{E}_s\mid\mathcal{G}_s]\le\zeta/(2T).

Fix a coordinate ii and decompose

vt,iZO−v~t,i=∑s≤twt,s(hs,iZO−hs,iZO,0)+∑s≤twt,sξs,i,ξs,i:=hs,iZO,0−h~s,i.(62)v^{\mathrm{ZO}}_{t,i}-\tilde{v}_{t,i} =\sum_{s\le t}w_{t,s}\left(h^{\mathrm{ZO}}_{s,i}-h^{\mathrm{ZO},0}_{s,i}\right)+\sum_{s\le t}w_{t,s}\xi_{s,i}, \qquad \xi_{s,i}:=h^{\mathrm{ZO},0}_{s,i}-\tilde{h}_{s,i}. \tag*{(62)}

Finite-difference term. On Es\mathcal{E}_s, equation 47 and h~s,i≥Hˉi/κ≥H/κ\tilde{h}_{s,i}\ge\bar{H}_i/\kappa\ge H/\kappa give

∣hs,iZO−hs,iZO,0∣≤h~s,i[2CbΛTBϑT2h~s,i+BϑT2h~s,i]≤(2CbΛTϱT+ϱT)h~s,i,(63)\left|h^{\mathrm{ZO}}_{s,i}-h^{\mathrm{ZO},0}_{s,i}\right| \le\tilde{h}_{s,i}\left[2\sqrt{\frac{C_b\Lambda_TB\vartheta_T^2}{\tilde{h}_{s,i}}}+\frac{B\vartheta_T^2}{\tilde{h}_{s,i}}\right] \le(2\sqrt{C_b\Lambda_T\varrho_T}+\varrho_T)\tilde{h}_{s,i}, \tag*{(63)}

so the first sum is at most (2CbΛTϱT+ϱT)v~t,i(2\sqrt{C_b\Lambda_T\varrho_T}+\varrho_T)\tilde{v}_{t,i}.

Truncation. By Proposition 1(i), E[ξs,i∣Gs]=0\mathbb{E}[\xi_{s,i}\mid\mathcal{G}_s]=0. Let bi:=CbΛTHˉib_i:=C_b\Lambda_T\bar{H}_i. Since hs,iZO,0h^{\mathrm{ZO},0}_{s,i} and h~s,i\tilde{h}_{s,i} are nonnegative and CbΛT≥1C_b\Lambda_T\ge1, on Es\mathcal{E}_s we have ∣ξs,i∣≤CbΛTh~s,i≤bi|\xi_{s,i}|\le C_b\Lambda_T\tilde{h}_{s,i}\le b_i. Define the truncated martingale differences

ξs,i′:=ξs,i1{∣ξs,i∣≤bi}−E[ξs,i1{∣ξs,i∣≤bi}∣Gs].(64)\xi'_{s,i}:=\xi_{s,i}\mathbbm{1}_{\{|\xi_{s,i}|\le b_i\}}-\mathbb{E}\left[\xi_{s,i}\mathbbm{1}_{\{|\xi_{s,i}|\le b_i\}}\mid\mathcal{G}_s\right]. \tag*{(64)}

Because E[ξs,i∣Gs]=0\mathbb{E}[\xi_{s,i}\mid\mathcal{G}_s]=0, the Cauchy–Schwarz inequality and equation 44 bound the conditional mean that is removed:

∣E[ξs,i1{∣ξs,i∣≤bi}∣Gs]∣=∣E[ξs,i1{∣ξs,i∣>bi}∣Gs]∣≤Cvh~s,iζ2T.(65)\left|\mathbb{E}\left[\xi_{s,i}\mathbbm{1}_{\{|\xi_{s,i}|\le b_i\}}\mid\mathcal{G}_s\right]\right| =\left|\mathbb{E}\left[\xi_{s,i}\mathbbm{1}_{\{|\xi_{s,i}|>b_i\}}\mid\mathcal{G}_s\right]\right| \le\sqrt{C_v}\tilde{h}_{s,i}\sqrt{\frac{\zeta}{2T}}. \tag*{(65)}

Hence, on ⋂s≤TEs\bigcap_{s\le T}\mathcal{E}_s,

∣∑s≤twt,sξs,i∣≤∣∑s≤twt,sξs,i′∣+Cvζ2Tv~t,i.(66)\left|\sum_{s\le t}w_{t,s}\xi_{s,i}\right| \le\left|\sum_{s\le t}w_{t,s}\xi'_{s,i}\right|+\sqrt{\frac{C_v\zeta}{2T}}\tilde{v}_{t,i}. \tag*{(66)}

Martingale term. The sequence Ys:=wt,sξs,i′Y_s:=w_{t,s}\xi'_{s,i} is a martingale difference sequence with respect to (Gs+1)s(\mathcal{G}_{s+1})_s, with ∣Ys∣≤2(1−β2)bi|Y_s|\le2(1-\beta_2)b_i and, by Assumption A2 and equation 44,

∑s≤tE[Ys2∣Gs]≤CvHˉi2∑s≤twt,s2≤Cv(1−β2)Hˉi2.(67)\sum_{s\le t}\mathbb{E}[Y_s^2\mid\mathcal{G}_s] \le C_v\bar{H}_i^2\sum_{s\le t}w_{t,s}^2 \le C_v(1-\beta_2)\bar{H}_i^2. \tag*{(67)}

Freedman’s inequality then gives, with probability at least 1−ζ/(2dT)1-\zeta/(2dT),

∣∑s≤tYs∣≤Hˉi[2Cv(1−β2)ΛT+43Cb(1−β2)ΛT2].(68)\left|\sum_{s\le t}Y_s\right| \le\bar{H}_i\left[\sqrt{2C_v(1-\beta_2)\Lambda_T}+\frac{4}{3}C_b(1-\beta_2)\Lambda_T^2\right]. \tag*{(68)}

where we used log⁡(4dT/ζ)≤ΛT\log(4dT/\zeta) \le\Lambda_T. Finally, for t≥t0t \ge t_0 we have β2t≤e−(1−β2)t≤1/2\beta_2^t \le e^{-(1-\beta_2)t} \le1/2, so Wt≥1/2W_t \ge1/2 and v~t,i≥WtH‾i/κ≥H‾i/(2κ)\widetilde{v}_{t,i} \ge W_t\overline{H}_i/\kappa\ge\overline{H}_i/(2\kappa) by Assumption A2. Dividing by v~t,i\widetilde{v}_{t,i} and collecting the three terms gives equation 61 for this pair (t,i)(t,i). A union bound over the events Es\mathcal{E}_s, with total failure probability ζ/2\zeta/2, and over the dTdT Freedman events, with total failure probability ζ/2\zeta/2, completes the proof. □

Lemma 4 bounds the error relative to the reference EMA itself. The leading term is governed by the EMA window 1/(1−β2)1/(1-\beta_2) and depends on dd, qq, and TT only through ΛT\Lambda_T. It does not require q→∞q \to\infty: the query budget enters the analysis through the damping λt\lambda_t of the target rather than through the estimation error.

From curvature error to preconditioner distortion

Define the actual ZO-Curv denominator

d^t,i:=max⁡{γvt,iZO,ϵ0},D^t:=Diag⁡(d^t).(69)\widehat{d}_{t,i} := \max\{\gamma v^{\mathrm{ZO}}_{t,i},\epsilon_0\}, \qquad\widehat{D}_t := \operatorname{Diag}(\widehat{d}_t). \tag*{(69)}

Lemma 5 (Denominator distortion). Suppose that, for some δ∈[0,1)\delta\in[0,1) and every coordinate ii,

(1−δ)v~t,i≤vt,iZO≤(1+δ)v~t,i.(70)(1-\delta)\widetilde{v}_{t,i} \le v^{\mathrm{ZO}}_{t,i} \le(1+\delta)\widetilde{v}_{t,i}. \tag*{(70)}

Then

(1−δ)Dt⪯D^t⪯(1+δ)Dt.(71)(1-\delta)D_t \preceq\widehat{D}_t \preceq(1+\delta)D_t. \tag*{(71)}

Proof. The map x↦max⁡{γx,ϵ0}x \mapsto\max\{\gamma x,\epsilon_0\} is nondecreasing. For the upper bound,

d^t,i≤max⁡{(1+δ)γv~t,i,ϵ0}≤(1+δ)max⁡{γv~t,i,ϵ0}=(1+δ)dt,i.(72)\widehat{d}_{t,i} \le\max\{(1+\delta)\gamma\widetilde{v}_{t,i},\epsilon_0\} \le(1+\delta)\max\{\gamma\widetilde{v}_{t,i},\epsilon_0\} = (1+\delta)d_{t,i}. \tag*{(72)}

For the lower bound, since ϵ0≥(1−δ)ϵ0\epsilon_0 \ge(1-\delta)\epsilon_0,

d^t,i≥max⁡{(1−δ)γv~t,i,ϵ0}≥(1−δ)max⁡{γv~t,i,ϵ0}=(1−δ)dt,i.(73)\widehat{d}_{t,i} \ge\max\{(1-\delta)\gamma\widetilde{v}_{t,i},\epsilon_0\} \ge(1-\delta)\max\{\gamma\widetilde{v}_{t,i},\epsilon_0\} = (1-\delta)d_{t,i}. \tag*{(73)}

Both matrices are diagonal, which proves equation 71. □

Lemma 5 is the interface between the estimator analysis and the optimization analysis. Because the error bound of Lemma 4 is relative, it passes through the floor without any division by ϵ0\epsilon_0, and the entire effect of the forward-only ZO estimator is summarized by the multiplicative distortion δ\delta.

Robust descent under approximate curvature

We now analyze the deterministic clipped preconditioned update

θt+1=θt−ηst,(74)\theta_{t+1} = \theta_t - \eta s_t, \tag*{(74)}

where

st:=clip⁡(D^t−1gt,ρ).(75)s_t := \operatorname{clip}\left(\widehat{D}_t^{-1}g_t,\rho\right). \tag*{(75)}

The clipping operator is applied coordinate-wise:

clip⁡(x,ρ):=sign⁡(x)min⁡{∣x∣,ρ}.(76)\operatorname{clip}(x,\rho) := \operatorname{sign}(x)\min\{|x|,\rho\}. \tag*{(76)}

The implementation corresponds to ρ=1\rho= 1.

The following assumption is a preconditioned analogue of the usual smoothness assumption.

Assumption A3 (Local preconditioned smoothness). There exists α>0\alpha> 0 such that, for every iteration tt and every point on the update segment,

∇2F(θt−τηst)⪯αDt,∀τ∈[0,1].(77)\nabla^2 F(\theta_t-\tau\eta s_t) \preceq\alpha D_t,\qquad\forall\tau\in[0,1]. \tag*{(77)}

For ordinary gradient descent, smoothness assumes ∇2F⪯LI\nabla^2F \preceq LI. Assumption A3 replaces the isotropic matrix LILI by the reference curvature preconditioner DtD_t. Because the reference target is damped, ordinary smoothness already implies this assumption. For t≥t0t \ge t_0 (defined in Lemma 4), dt,i≥γv~t,i≥γWtmin⁡s≤tλs≥γ2min⁡s≤tλsd_{t,i} \ge\gamma\widetilde{v}_{t,i} \ge\gamma W_t\min_{s\le t}\lambda_s \ge\frac{\gamma}{2}\min_{s\le t}\lambda_s, so ∇2F⪯LI\nabla^2F \preceq LI along the update segments implies Assumption A3 with α=2L/(γmin⁡s≤tλs)\alpha= 2L/(\gamma\min_{s\le t}\lambda_s). This is only a sufficient condition; the assumption holds with a smaller α\alpha whenever the GNB part of DtD_t captures the curvature of FF.

Theorem 1 (Robust descent under curvature estimation error). Suppose

(1−δ)Dt⪯D^t⪯(1+δ)Dt(78)(1-\delta)D_t \preceq\widehat{D}_t \preceq(1+\delta)D_t \tag*{(78)}

for some δ∈[0,1)\delta\in[0,1), and Assumption A3 holds. Define

Ψt:=gtTst.(79)\Psi_t := g_t^{\mathsf{T}}s_t. \tag*{(79)}

Then

Ψt=∑i=1dmin⁡{gt,i2d^t,i,ρ∣gt,i∣}≥0,(80)\Psi_t = \sum_{i=1}^{d} \min\left\{\frac{g_{t,i}^{2}}{\widehat{d}_{t,i}},\rho\lvert g_{t,i}\rvert\right\} \ge0, \tag*{(80)}

and the one-step loss satisfies

F(θt+1)≤F(θt)−η(1−αη2(1−δ))Ψt.(81)F(\theta_{t+1}) \le F(\theta_t)-\eta\left(1-\frac{\alpha\eta}{2(1-\delta)}\right)\Psi_t. \tag*{(81)}

Therefore, every non-stationary step is a descent step whenever

0<η<2(1−δ)α.(82)0 < \eta< \frac{2(1-\delta)}{\alpha}. \tag*{(82)}

In particular, if

η≤1−δα,(83)\eta\le\frac{1-\delta}{\alpha}, \tag*{(83)}

then

F(θt+1)≤F(θt)−η2Ψt.(84)F(\theta_{t+1}) \le F(\theta_t)-\frac{\eta}{2}\Psi_t. \tag*{(84)}

Proof. Since every d^t,i>0\widehat{d}_{t,i}>0, clipping preserves the sign of gt,ig_{t,i}. Hence

gt,ist,i=∣gt,i∣min⁡{∣gt,i∣d^t,i,ρ}(85)g_{t,i}s_{t,i}=\lvert g_{t,i}\rvert\min\left\{\frac{\lvert g_{t,i}\rvert}{\widehat{d}_{t,i}},\rho\right\} \tag*{(85)}
=min⁡{gt,i2d^t,i,ρ∣gt,i∣}≥0.(86)=\min\left\{\frac{g_{t,i}^{2}}{\widehat{d}_{t,i}},\rho\lvert g_{t,i}\rvert\right\}\ge0. \tag*{(86)}

Summing over coordinates gives equation 80.

Taylor’s theorem with integral remainder and Assumption A3 imply

F(θt−ηst)≤F(θt)−ηgtTst+αη22stTDtst.(87)F(\theta_t-\eta s_t)\le F(\theta_t)-\eta g_t^{\mathsf{T}}s_t+\frac{\alpha\eta^2}{2}s_t^{\mathsf{T}}D_ts_t. \tag*{(87)}

It remains to upper-bound the quadratic term. By definition of clipping,

∣st,i∣≤∣gt,i∣d^t,i.(88)\lvert s_{t,i}\rvert\le\frac{\lvert g_{t,i}\rvert}{\widehat{d}_{t,i}}. \tag*{(88)}

Therefore

dt,ist,i2≤dt,i∣st,i∣∣gt,i∣d^t,i(89)d_{t,i}s_{t,i}^{2}\le d_{t,i}\lvert s_{t,i}\rvert\frac{\lvert g_{t,i}\rvert}{\widehat{d}_{t,i}} \tag*{(89)}
=dt,id^t,i∣gt,i∣∣st,i∣.(90)=\frac{d_{t,i}}{\widehat{d}_{t,i}}\lvert g_{t,i}\rvert\lvert s_{t,i}\rvert. \tag*{(90)}

From d^t,i≥(1−δ)dt,i\widehat{d}_{t,i}\ge(1-\delta)d_{t,i},

dt,id^t,i≤11−δ.(91)\frac{d_{t,i}}{\widehat{d}_{t,i}}\le\frac{1}{1-\delta}. \tag*{(91)}

Since gt,ig_{t,i} and st,is_{t,i} have the same sign,

∣gt,i∣∣st,i∣=gt,ist,i.(92)\lvert g_{t,i}\rvert\lvert s_{t,i}\rvert=g_{t,i}s_{t,i}. \tag*{(92)}

Thus

dt,ist,i2≤11−δgt,ist,i.(93)d_{t,i}s_{t,i}^{2} \le\frac{1}{1-\delta}g_{t,i}s_{t,i}. \tag*{(93)}

Summing over ii yields

stTDtst≤11−δgtTst=11−δΨt.(94)s_t^{\mathsf T}D_ts_t \le\frac{1}{1-\delta}g_t^{\mathsf T}s_t = \frac{1}{1-\delta}\Psi_t. \tag*{(94)}

Substituting equation 94 into equation 87 gives equation 81. The remaining claims follow immediately from the corresponding stepsize choices.

The key property in Theorem 1 is that curvature error affects the magnitude of the update but not its coordinate-wise sign. In contrast, replacing gtg_t itself with a noisy ZO gradient may change the sign of the first-order descent term gtTg^tg_t^{\mathsf T}\hat{g}_t. This difference formalizes why noisy ZO information is more naturally tolerated as a preconditioner than as a descent direction.

Corollary 1 (Convergence to stationarity). Assume the conditions of Theorem 1, let

η≤1−δα,(95)\eta\le\frac{1-\delta}{\alpha}, \tag*{(95)}

and suppose FF is bounded below by F⋆F_\star. Then, for any T≥1T \ge1,

min⁡0≤t<TΨt≤2(F(θ0)−F⋆)ηT.(96)\min_{0\le t<T}\Psi_t \le\frac{2(F(\theta_0)-F_\star)}{\eta T}. \tag*{(96)}

Consequently,

min⁡0≤t<TΨt=O(T−1).(97)\min_{0\le t<T}\Psi_t = O(T^{-1}). \tag*{(97)}

Proof. Equation equation 84 gives

F(θt)−F(θt+1)≥η2Ψt.(98)F(\theta_t)-F(\theta_{t+1}) \ge\frac{\eta}{2}\Psi_t. \tag*{(98)}

Summing from t=0t=0 to T−1T-1,

F(θ0)−F(θT)≥η2∑t=0T−1Ψt.(99)F(\theta_0)-F(\theta_T) \ge\frac{\eta}{2}\sum_{t=0}^{T-1}\Psi_t. \tag*{(99)}

Since F(θT)≥F⋆F(\theta_T) \ge F_\star,

∑t=0T−1Ψt≤2(F(θ0)−F⋆)η.(100)\sum_{t=0}^{T-1}\Psi_t \le\frac{2(F(\theta_0)-F_\star)}{\eta}. \tag*{(100)}

The minimum is no larger than the average, giving equation 96.

Remark 1 (From Ψt\Psi_t to the gradient). The measure Ψt\Psi_t controls the gradient whenever the denominator is bounded above. Under Assumption A2, v~t,i≤WtHˉi≤Hˉi\tilde{v}_{t,i} \le W_t\bar{H}_i \le\bar{H}_i, so on the event of Theorem 3, d^t,i≤(1+δT)dt,i≤d^max⁡:=(1+δT)max⁡{γmax⁡iHˉi,ϵ0}\hat{d}_{t,i} \le(1+\delta_T)d_{t,i} \le\hat{d}_{\max} := (1+\delta_T)\max\{\gamma\max_i\bar{H}_i,\epsilon_0\}. Since a↦min⁡{a2/d^max⁡,ρ∣a∣}a \mapsto\min\{a^2/\hat{d}_{\max},\rho|a|\} is increasing in ∣a∣|a|, Ψt≤ψ\Psi_t \le\psi implies ∣gt,i∣≤max⁡{d^max⁡ψ,ψ/ρ}|g_{t,i}| \le\max\{\sqrt{\hat{d}_{\max}\psi},\psi/\rho\} for every coordinate ii.

Local linear convergence

The global result above does not require clipping to be inactive. Once the optimization enters a sufficiently local region, however, the preconditioned gradient becomes small and clipping no longer activates. In this regime we obtain a sharper linear rate.

Assumption A4 (Preconditioned Polyak–Łojasiewicz condition). There exists μD∈(0,α]\mu_D \in(0,\alpha] such that, whenever θt\theta_t lies in a neighborhood N\mathcal{N} of the target solution,

gtTDt−1gt≥2μD(F(θt)−F⋆).(101)g_t^{\mathsf T}D_t^{-1}g_t \ge2\mu_D\left(F(\theta_t)-F_\star\right). \tag*{(101)}

Theorem 2 (Local linear convergence with approximate curvature). Suppose Assumptions A3 and A4 hold, and

(1−δ)Dt⪯D^t⪯(1+δ)Dt,0≤δ<1.(102)(1-\delta)D_t \preceq\hat{D}_t \preceq(1+\delta)D_t,\qquad0\le\delta<1. \tag*{(102)}

Suppose further that

∥D^t−1gt∥∞≤(1−δ)ρ.(103)\lVert\widehat{D}_t^{-1}g_t\rVert_\infty\le(1-\delta)\rho. \tag*{(103)}

Then clipping is inactive and

st=D^t−1gt.(104)s_t=\widehat{D}_t^{-1}g_t. \tag*{(104)}

For any

0<η<2(1−δ)α,(105)0<\eta<\frac{2(1-\delta)}{\alpha}, \tag*{(105)}

we have

F(θt+1)−F⋆≤[1−2ημD1+δ(1−αη2(1−δ))](F(θt)−F⋆).(106)F(\theta_{t+1})-F_\star\le\left[1-\frac{2\eta\mu_D}{1+\delta}\left(1-\frac{\alpha\eta}{2(1-\delta)}\right)\right](F(\theta_t)-F_\star). \tag*{(106)}

In particular, choosing

η=1−δα(107)\eta=\frac{1-\delta}{\alpha} \tag*{(107)}

gives

F(θt+1)−F⋆≤[1−μDα1−δ1+δ](F(θt)−F⋆).(108)F(\theta_{t+1})-F_\star\le\left[1-\frac{\mu_D}{\alpha}\frac{1-\delta}{1+\delta}\right](F(\theta_t)-F_\star). \tag*{(108)}

Proof. From D^t⪰(1−δ)Dt\widehat{D}_t\succeq(1-\delta)D_t,

∣gt,i∣d^t,i≤∣gt,i∣(1−δ)dt,i.(109)\frac{|g_{t,i}|}{\widehat{d}_{t,i}}\le\frac{|g_{t,i}|}{(1-\delta)d_{t,i}}. \tag*{(109)}

Condition equation 103 therefore implies

∣gt,i∣d^t,i≤ρ,(110)\frac{|g_{t,i}|}{\widehat{d}_{t,i}}\le\rho, \tag*{(110)}

so clipping is inactive.

Using Theorem 1 with st=D^t−1gts_t=\widehat{D}_t^{-1}g_t,

F(θt+1)≤F(θt)−η(1−αη2(1−δ))gtTD^t−1gt.(111)F(\theta_{t+1})\le F(\theta_t)-\eta\left(1-\frac{\alpha\eta}{2(1-\delta)}\right)g_t^{\mathsf T}\widehat{D}_t^{-1}g_t. \tag*{(111)}

Since

D^t⪯(1+δ)Dt,(112)\widehat{D}_t\preceq(1+\delta)D_t, \tag*{(112)}

inversion reverses the PSD order:

D^t−1⪰11+δDt−1.(113)\widehat{D}_t^{-1}\succeq\frac{1}{1+\delta}D_t^{-1}. \tag*{(113)}

Hence

gtTD^t−1gt≥11+δgtTDt−1gt(114)g_t^{\mathsf T}\widehat{D}_t^{-1}g_t\ge\frac{1}{1+\delta}g_t^{\mathsf T}D_t^{-1}g_t \tag*{(114)}
≥2μD1+δ(F(θt)−F⋆),(115)\ge\frac{2\mu_D}{1+\delta}(F(\theta_t)-F_\star), \tag*{(115)}

where the second inequality uses Assumption A4. Substituting equation 115 into equation 111 proves equation 106. Setting η=(1−δ)/α\eta=(1-\delta)/\alpha gives equation 108. □

Theorem 2 makes the effect of curvature estimation error explicit. For exact reference curvature, δ=0\delta=0, the contraction coefficient is

1−μDα.(116)1-\frac{\mu_D}{\alpha}. \tag*{(116)}

With approximate ZO curvature, the effective progress coefficient is reduced continuously by

1−δ1+δ.(117)\frac{1-\delta}{1+\delta}. \tag*{(117)}

Thus curvature estimation error shrinks the stable stepsize and slows the contraction rate, but does not destroy convergence as long as δ<1\delta<1.

Convergence of ZO-Curv

We finally combine the estimator and optimization results over a horizon of TT iterations, with one curvature estimate per training step. Throughout, ΛT\Lambda_T, RTR_T, ϑT\vartheta_T, ϱT\varrho_T, t0t_0, and δT\delta_T are as in Lemma 4.

Theorem 3. (Convergence of ZO-Curv). Suppose Assumptions A1 (with R=RTR = R_T), A2, and A3 hold over a horizon of TT iterations, and q≥ΛTq \ge\Lambda_T. Then, with probability at least 1−ζ1-\zeta, simultaneously for every iteration t0≤t<Tt_0 \le t < T,

(1−δT)Dt⪯D^t⪯(1+δT)Dt.(118)(1-\delta_T)D_t \preceq\widehat{D}_t \preceq(1+\delta_T)D_t. \tag*{(118)}

If

δT<1(119)\delta_T < 1 \tag*{(119)}

and

η≤1−δTα,(120)\eta\le\frac{1-\delta_T}{\alpha}, \tag*{(120)}

then

min⁡t0≤t<TΨt≤2(F(θt0)−F⋆)η(T−t0).(121)\min_{t_0\le t<T}\Psi_t \le\frac{2(F(\theta_{t_0})-F_\star)}{\eta(T-t_0)}. \tag*{(121)}

If, in addition, the iterates θt0,…,θT−1\theta_{t_0},\ldots,\theta_{T-1} remain in the neighborhood N\mathcal{N} of Assumption A4 and satisfy the no-clipping condition equation (103) with δ\delta replaced by δT\delta_T, then choosing

η=1−δTα(122)\eta= \frac{1-\delta_T}{\alpha} \tag*{(122)}

gives

F(θT)−F⋆≤[1−μDα1−δT1+δT]T−t0(F(θt0)−F⋆).(123)F(\theta_T)-F_\star\le\left[1-\frac{\mu_D}{\alpha}\frac{1-\delta_T}{1+\delta_T}\right]^{T-t_0}(F(\theta_{t_0})-F_\star). \tag*{(123)}

Proof. By Lemma 4, with probability at least $1-\zeta$, (1−δT)v~t≤vtZO≤(1+δT)v~t(1-\delta_T)\widetilde{v}_t \le v_t^{\mathrm{ZO}} \le(1+\delta_T)\widetilde{v}_t coordinate-wise for every t0≤t≤T.xref‘Lemma5‘witht_0 \le t \le T. {xref}`Lemma 5` with\delta=\delta_T$ then proves equation (118). Equation (121) follows from Corollary 1 applied to the iterations t0,…,T−1t_0,\ldots,T-1, and equation (123) follows from Theorem 2 applied at each of these iterations.

The query budget enters Theorem 3 through the reference preconditioner rather than through δT\delta_T: DtD_t is built from the damped target h~t\widetilde{h}_t, whose damping λt=∥htG∥1/q\lambda_t=\lVert h_t^G\rVert_1/q decreases as 1/q1/q, so larger budgets bring the reference update closer to the undamped GNB-preconditioned update of Sophia-G. The estimation error δT\delta_T is controlled by the EMA window and depends on dd, qq, and TT only logarithmically.

Corollary 2. (Controlled penalty relative to reference GNB curvature). Suppose there exists a constant δˉ<1\bar{\delta}<1 such that δT≤δˉ\delta_T\le\bar{\delta}. Then the number of local iterations T−t0T-t_0 required to reach

F(θT)−F⋆≤ϵopt(124)F(\theta_T)-F_\star\le\epsilon_{\mathrm{opt}} \tag*{(124)}

satisfies

T−t0=O(αμD1+δˉ1−δˉlog⁡F(θt0)−F⋆ϵopt).(125)T-t_0=O\left(\frac{\alpha}{\mu_D}\frac{1+\bar{\delta}}{1-\bar{\delta}}\log\frac{F(\theta_{t_0})-F_\star}{\epsilon_{\mathrm{opt}}}\right). \tag*{(125)}

For exact reference curvature, δˉ=0\bar{\delta}=0, this reduces to

TG=O(αμDlog⁡F(θt0)−F⋆ϵopt).(126)T_G=O\left(\frac{\alpha}{\mu_D}\log\frac{F(\theta_{t_0})-F_\star}{\epsilon_{\mathrm{opt}}}\right). \tag*{(126)}

Hence the multiplicative iteration penalty of ZO curvature relative to the reference GNB-preconditioned core is bounded by

TZOTG≤1+δˉ1−δˉ.(127)\frac{T_{\mathrm{ZO}}}{T_G}\le\frac{1+\bar{\delta}}{1-\bar{\delta}}. \tag*{(127)}

For small δˉ\bar{\delta},

1+δˉ1−δˉ=1+2δˉ+O(δˉ2).(128)\frac{1+\bar{\delta}}{1-\bar{\delta}}=1+2\bar{\delta}+O(\bar{\delta}^2). \tag*{(128)}

Proof. From Theorem 3,

F(θT)−F⋆≤(1−ν)T−t0(F(θt0)−F⋆),(129)F(\theta_T)-F_\star\le(1-\nu)^{T-t_0}\left(F(\theta_{t_0})-F_\star\right), \tag*{(129)}

where

ν≔μDα1−δˉ1+δˉ.(130)\nu\coloneqq\frac{\mu_D}{\alpha}\frac{1-\bar{\delta}}{1+\bar{\delta}}. \tag*{(130)}

Using

1−ν≤e−ν,(131)1-\nu\le e^{-\nu}, \tag*{(131)}

it is sufficient that

T−t0≥1νlog⁡F(θt0)−F⋆ϵopt,(132)T-t_0 \ge\frac{1}{\nu}\log\frac{F(\theta_{t_0})-F_\star}{\epsilon_{\mathrm{opt}}}, \tag*{(132)}

which gives equation 125. Equation equation 128 is the Taylor expansion around δˉ=0\bar{\delta}=0. □

Remark 2 (Pipelined updates with staleness ss). In the deployed pipeline (§ 3.1 and Algorithm 1), the EMA vtZOv_t^{\mathrm{ZO}} used at step tt is built from estimates ht′ZOh_{t'}^{\mathrm{ZO}} queried at earlier snapshots θt′\theta_{t'}, t′≤t−st' \le t-s, and is held fixed between refreshes. Define the reference EMA v~t\tilde{v}_t with exactly the same schedule, from h~t′\tilde{h}_{t'} at the same snapshots, mini-batches, and sampled labels. Proposition 1 is a per-refresh statement. Lemma 4 applies to the sequence of refreshes, with ss indexing refreshes, and Gs\mathcal{G}_s the history before the directions of refresh ss: these directions are still drawn independently of Gs\mathcal{G}_s, and a held step copies both EMA states. Lemma 5 depends only on vtZOv_t^{\mathrm{ZO}} and v~t\tilde{v}_t. Theorems 1 and 2 are per-step statements about DtD_t and D~t\tilde{D}_t, with Assumptions A3 and A4 stated relative to the lagged DtD_t. Theorem 3 unions over at most TT refreshes. All results therefore hold verbatim for any staleness ss, including the one-step offset of the main experiments and the s=3s=3 deployment of Appendix ??; staleness enters only through the reference DtD_t, which the assumptions already constrain.

Interpretation

The above results establish the following chain:

q, ε⟹h~t (unbiased target)⟹δT (EMA)⟹optimization rate.(133)q,\ \varepsilon\Longrightarrow\tilde{h}_t\ \text{(unbiased target)} \Longrightarrow\delta_T\ \text{(EMA)} \Longrightarrow\text{optimization rate}. \tag*{(133)}

The finite query budget has two separate effects. Its systematic effect is the uniform damping λt=∥htG∥1/q\lambda_t=\lVert h_t^G\rVert_1/q of the curvature target (Proposition 1), which averaging before squaring keeps a factor qq smaller than squaring before averaging, and which vanishes as q→∞q\to\infty. Its random effect is a fluctuation of the order of the target in every coordinate, with constants independent of dd. The curvature EMA averages this fluctuation over its effective window (Lemma 4), and the positive floor preserves the resulting relative bound (Lemma 5), which yields the multiplicative preconditioner distortion δT\delta_T of equation 60. As long as δT<1\delta_T<1, ZO-Curv preserves descent. In the local unclipped regime, the convergence coefficient is reduced from the reference value

μDα(134)\frac{\mu_D}{\alpha} \tag*{(134)}

to

μDα1−δT1+δT,(135)\frac{\mu_D}{\alpha}\frac{1-\delta_T}{1+\delta_T}, \tag*{(135)}

where μD\mu_D and α\alpha refer to the damped GNB reference preconditioner.

Therefore, curvature-estimation error produces a continuous and controlled degradation of the convergence rate rather than destroying the descent direction. This differs fundamentally from injecting ZO noise directly into the first-order gradient: an erroneous gradient estimate can rotate or reverse the descent direction, whereas a positive curvature estimate only rescales the coordinate-wise first-order update, with clipping additionally bounding the maximum step magnitude.

Thus, replacing the backpropagation-based GNB computation with forward-only ZO queries yields a damped GNB preconditioner: the query budget controls the damping, the curvature EMA controls the estimation error, and the convergence mechanism of clipped curvature preconditioning is retained.

References

  1. [1]Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
  2. [2]Atılım Güneş Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation. arXiv preprint arXiv:2202.08587, 2022.arxiv.org/abs/2202.08587
  3. [3]Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riaibinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 558–568, 2023.
  4. [4]Dongqi Cai. Federated llm pre-training on mobile phones. In Proceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services, pp. 657–658, 2025.DOI
  5. [5]Cast AI. 2026 state of kubernetes optimization report. https://cast.ai/reports/kubernetes-optimization-report/, 2026. Analysis of production Kubernetes clusters across AWS, GCP, and Azure.
  6. [6]Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. DeepZero: Scaling up zeroth-order optimization for deep model training. In International Conference on Learning Representations, volume 2024, pp. 50185–50206, 2024.arxiv.org/abs/2310.02025
  7. [7]Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In International Conference on Learning Representations, volume 2025, pp. 62581–62607, 2025.arxiv.org/abs/2410.07698
  8. [8]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  9. [9]Brannon Dorsey and Matt Kafonek. Borrowing the night: Reclaiming idle inference gpus for research. https://runway.com/news/engineering/borrowing-the-night-reclaiming-idle-inference-gpus-for-research, July 2026.
  10. [10]Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023.arxiv.org/abs/2311.08105
  11. [11]John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015.arxiv.org/abs/1312.2139
  12. [12]Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368, 2013.arxiv.org/abs/1309.5549
  13. [13]Aaron Gokaslan and Vanya Cohen. OpenWebText corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  14. [14]Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024.arxiv.org/abs/2404.14527
  15. [15]Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp. 1842–1850. PMLR, 2018.arxiv.org/abs/1802.09568
  16. [16]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
  17. [17]Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18363–18371, 2024.
  18. [18]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  19. [19]Apoorv Khandelwal, Tian Yun, Nihal V Nayak, Jack Merullo, Stephen H Bach, Chen Sun, and Ellie Pavlick. $100K or 100 days: Trade-offs when pre-training with academic resources. arXiv preprint arXiv:2410.23261, 2024.arxiv.org/abs/2410.23261
  20. [20]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
  21. [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023.arxiv.org/abs/2309.06180
  22. [22]Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang. Lyra: Elastic scheduling for deep learning clusters. In Proceedings of the Eighteenth European Conference on Computer Systems, pp. 835–850, 2023.DOI
  23. [23]Fangyue Liu, Hua Liu, Xinyuan Lyu, Shuo Ai, Hao Liang, Lingpeng Chen, Ziqian Hu, Chong Zha, Xin Jin, Hanmei Luo, et al. Valve: Production online-offline inference colocation with jointly-bounded preemption latency and rate. arXiv preprint arXiv:2604.07874, 2026a.arxiv.org/abs/2604.07874
  24. [24]Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International conference on learning representations, volume 2024, pp. 1621–1650, 2024.arxiv.org/abs/2305.14342
  25. [25]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  26. [26]Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. Advances in Neural Information Processing Systems, 38:21139–21169, 2026b.arxiv.org/abs/2402.15751
  27. [27]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
  28. [28]Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075, 2023.arxiv.org/abs/2305.17333
  29. [29]James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76, 2020.
  30. [30]James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408–2417. PMLR, 2015.arxiv.org/abs/1503.05671
  31. [31]Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017.
  32. [32]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzele, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 1525–1534, 2016.arxiv.org/abs/1606.06031
  33. [33]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132. IEEE, 2024.
  34. [34]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  35. [35]Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. arXiv preprint arXiv:2210.03310, 2022.arxiv.org/abs/2210.03310
  36. [36]Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. In International Conference on Machine Learning, pp. 29416–29440. PMLR, 2023.arxiv.org/abs/2301.11913
  37. [37]Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research, 18(52):1–11, 2017.
  38. [38]Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International conference on machine learning, pp. 4596–4604. PMLR, 2018.arxiv.org/abs/1804.04235
  39. [39]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.DOI
  40. [40]James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 1992.DOI
  41. [41]James C Spall. Adaptive stochastic approximation by the simultaneous perturbation method. IEEE transactions on automatic control, 45(10):1839–1853, 2000.
  42. [42]John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible instances resilient for affordable training of large {DNNs}. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 497–513, 2023.arxiv.org/abs/2204.12013
  43. [43]Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, volume 2025, pp. 93423–93444, 2025.arxiv.org/abs/2409.11321
  44. [44]Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084–17097, 2021.
  45. [45]Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10665–10673, 2021.arxiv.org/abs/2006.00719
  46. [46]Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. Zeroth-order fine-tuning of llms in random subspaces. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4475–4485. IEEE, 2025.arxiv.org/abs/2410.08989
  47. [47]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
  48. [48]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shouhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.arxiv.org/abs/2205.01068
  49. [49]Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592, 2024.arxiv.org/abs/2402.11592
  50. [50]Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor Tsang. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. In International Conference on Learning Representations, volume 2025, pp. 43496–43520, 2025.arxiv.org/abs/2402.15173
  51. [51]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 193–210, 2024.arxiv.org/abs/2401.09670

Paper details

vlnc:W_hwggtz8k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents