Introduction
Large language model fine-tuning relies on first-order (FO) optimizers [12 ] . However, FO training requires storing activations, gradients, and optimizer states, making it memory-intensive. Zeroth-order (ZO) optimization [13 ] is therefore an appealing alternative: it uses only forward passes and reduces memory to near inference level. Still, in the full LLM parameter space, ZO methods suffer from high variance and high query cost [15 ] .
LoRA [6 ] offers a lower-dimensional alternative by restricting updates to a low-rank subspace, improving efficiency and usually reducing variance. But this also discards fine-grained curvature information from the full space, which can slow convergence and degrade performance.
This leads to a central dilemma in zeroth-order optimization (Table 1 ): full-space optimization is informative but high-variance, while subspace optimization is low-variance but blind. Recent curvature-aware ZO methods, such as LOREN [17 ] , reduce estimator variance through anisotropic perturbations [10 ] . Even so, existing methods usually stay in one space only: either the full parameter space or a low-rank subspace such as LoRA. They therefore gain only one side of the trade-off. Full-space methods retain richer information but are statistically costly, while subspace methods are more efficient but inevitably lose part of the curvature structure that is present in the full space. This motivates a simple question: can we keep the efficiency of low-dimensional sampling while still using curvature information from the full space? We approach this question through Variational MultiScale (VMS) theory [7 , 8 ] from computational mechanics. VMS separates a problem into a resolved coarse scale and an unresolved fine scale, then uses a correction term [9 ] to account for the effect of the unresolved part. This viewpoint suggests a natural way to think about low-dimensional and high-dimensional spaces in zeroth-order optimization.
Table 1. Trade-offs between full-space and subspace ZO optimization. ✓ denotes a relative strength, and × a weakness.
Based on this idea, we propose ENGINE (Endogenous Variational MultiScale). We treat the low-dimensional LoRA parameterization as the resolved coarse scale and incorporate unresolved full-space curvature through a fine-scale correction. Unlike classical VMS, the optimization operator evolves with the optimization state, so the interaction between the two spaces must be updated dynamically. ENGINE addresses this with an endogenous V-cycle that alternates between the two spaces during training. Through a closed-form subgrid correction, it uses fine-scale information from the high-dimensional space to correct the efficient low-dimensional update.
Experiments on standard LLM optimization benchmarks show that ENGINE consistently improves over state-of-the-art zeroth-order baselines, including LOREN. It achieves better downstream accuracy, faster convergence, and up to a 2.61× speed-up in forward passes (Figure 1 ), while keeping peak memory usage unchanged.
Figure 1. Left: Forward-pass and wall-clock speed-up of ENGINE over LOREN to reach LOREN’s best accuracy, across four representative (model, task) settings. Right: Validation accuracy convergence on LLaMA-3-8B (CB).
Our contributions are summarised as follows.
The framework of ENGINE. We propose ENGINE, an endogenous Variational MultiScale framework for zeroth-order optimization that combines low-rank sampling with a closed-form subgrid correction, thereby reintroducing unresolved full-space curvature into the coarse update. Under appropriate assumptions, we rigorously analyze the bias and variance of the corrected estimator. We also establish a stationarity bound for coarse updates.
Figure 2. Test accuracy on all 16 (model, task) combinations. Top row: GPT-2-XL (GLUE, FP32) and OPT-2.7B (GLUE, FP32). Bottom row: LLaMA-3-8B (SuperGLUE, BF16) and OPT-13B (SuperGLUE, BF16). ENGINE is shown in blue hatched. Full numerical values are in Appendix E.1.
Preliminaries & Problem Setup
To address the single-scale limitation of ZO methods discussed in § 1 , ENGINE couples a low-dimensional LoRA scale with the high-dimensional full parameter space via Variational MultiScale (VMS). § 2.1 first introduces the curvature-aware ZO framework adopted from LOREN, after which § 2.2 recalls the Variational MultiScale (VMS) viewpoint underlying our method.
Zeroth-Order Setting & Notation Setup
Notation. We denote vectors by lowercase bold and matrices by uppercase bold, where ⊗ \otimes ⊗ is the Kronecker product and ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ the Euclidean norm. For a weight matrix X ∈ R m × n \mathbf{X} \in\mathbb{R}^{m \times n} X ∈ R m × n , let x : = vec ( X ) ∈ R d \mathbf{x} := \operatorname{vec}(\mathbf{X}) \in\mathbb{R}^{d} x := vec ( X ) ∈ R d with d = m n d = mn d = mn (multi-layer models are handled block-wise). We minimize f ( x ) : = E ξ ∼ P d a t a [ ℓ ( x , ξ ) ] f(\mathbf{x}) := \mathbb{E}_{\xi\sim P_{\mathrm{data}}}[\ell(\mathbf{x}, \xi)] f ( x ) := E ξ ∼ P data [ ℓ ( x , ξ )] using only function-value queries, where P d a t a P_{\mathrm{data}} P data is the data distribution.
Rank-1 Kronecker curvature. Full Hessian storage costs O ( m 2 n 2 ) O(m^{2}n^{2}) O ( m 2 n 2 ) memory. K-FAC [14 ] and Shampoo [5 ] exploit matrix structure to reduce preconditioning costs. Following LOREN, we use the compact vector parameterization:
H ~ = I m ⊗ ( ρ I n + v v T ) , v ∈ R n , ρ > 0 , (1) \widetilde{\mathbf{H}} = \mathbf{I}_{m} \otimes\left(\rho\mathbf{I}_{n} + \mathbf{v}\mathbf{v}^{\mathsf{T}}\right), \qquad\mathbf{v} \in\mathbb{R}^{n},\ \rho> 0, \tag*{(1)} H = I m ⊗ ( ρ I n + v v T ) , v ∈ R n , ρ > 0 , (1)
where v v T \mathbf{v}\mathbf{v}^{\mathsf{T}} v v T captures anisotropy shared across blocks, and damping ensures positive definiteness. Storing v \mathbf{v} v costs O ( n ) O(n) O ( n ) . The inverse and inverse square root admit closed forms with O ( m n ) O(mn) O ( mn ) application cost, avoiding dense matrices. § 3 extends this approximation to multiple scales.
Curvature as covariance. For u ∼ N ( 0 , I d ) \mathbf{u} \sim\mathcal{N}(\mathbf{0}, \mathbf{I}_{d}) u ∼ N ( 0 , I d ) and smoothing radius ϵ > 0 \epsilon> 0 ϵ > 0 , the symmetric finite-difference estimator [18 ] is g ^ ϵ ( x , u ) = f ( x + ϵ u ) − f ( x − ϵ u ) 2 ϵ u \widehat{\mathbf{g}}_{\epsilon}(\mathbf{x}, \mathbf{u}) = \frac{f(\mathbf{x}+\epsilon\mathbf{u})-f(\mathbf{x}-\epsilon\mathbf{u})}{2\epsilon}\mathbf{u} g ϵ ( x , u ) = 2 ϵ f ( x + ϵ u ) − f ( x − ϵ u ) u . Its expectation is the gradient of the Gaussian-smoothed loss. Generalized Gaussian smoothing extends this to anisotropic perturbations u ∼ N ( 0 , Σ ) \mathbf{u} \sim\mathcal{N}(\mathbf{0}, \mathbf{\Sigma}) u ∼ N ( 0 , Σ ) , and setting Σ = H ~ − 1 \mathbf{\Sigma} = \widetilde{\mathbf{H}}^{-1} Σ = H − 1 yields the NES identity [21 ]
H ~ − 1 ∇ f ( x ) = E u ~ ∼ N ( 0 , H ~ − 1 ) [ ⟨ ∇ f ( x ) , u ~ ⟩ u ~ ] , \widetilde{H}^{-1}\nabla f(x)=\mathbb{E}_{\widetilde{u}\sim\mathcal{N}(0,\widetilde{H}^{-1})}\left[\langle\nabla f(x),\widetilde{u}\rangle\widetilde{u}\right], H − 1 ∇ f ( x ) = E u ∼ N ( 0 , H − 1 ) [ ⟨ ∇ f ( x ) , u ⟩ u ] ,
so preconditioning by H ~ − 1 \widetilde{H}^{-1} H − 1 corresponds to larger ZO perturbations along directions of lower estimated curvature.
Variational MultiScale Method
The Variational MultiScale (VMS) method decomposes problems into resolved (coarse) and unresolved (fine) scales. Let Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d be a bounded domain, V V V a function space over Ω \Omega Ω with dual V ∗ V^* V ∗ , and L : V → V ∗ \mathcal{L}:V\to V^* L : V → V ∗ a linear operator, where the symmetric bilinear form a ( w , u ) : = ( w , L u ) a(w,u):=(w,\mathcal{L}u) a ( w , u ) := ( w , L u ) uses the L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) inner product ( w , u ) : = ∫ Ω w ( x ) u ( x ) d x (w,u):=\int_{\Omega}w(x)u(x)\,dx ( w , u ) := ∫ Ω w ( x ) u ( x ) d x . The weak problem seeks u ∈ V u\in V u ∈ V such that a ( w , u ) = ( w , f ) a(w,u)=(w,f) a ( w , u ) = ( w , f ) for all test functions w ∈ V w\in V w ∈ V . VMS decomposes V = V ‾ ⊕ V ′ V=\overline{V}\oplus V' V = V ⊕ V ′ into coarse V ‾ \overline{V} V and fine V ′ V' V ′ subspaces, so that u = u ‾ + u ′ u=\overline{u}+u' u = u + u ′ and w = w ‾ + w ′ w=\overline{w}+w' w = w + w ′ with u ‾ , w ‾ ∈ V ‾ \overline{u},\overline{w}\in\overline{V} u , w ∈ V and u ′ , w ′ ∈ V ′ u',w'\in V' u ′ , w ′ ∈ V ′ . The fine scale is driven by the coarse residual L u ′ = − ( L u ‾ − f ) \mathcal{L}u'=-(\mathcal{L}\overline{u}-f) L u ′ = − ( L u − f ) . Following the sign convention of Hughes [1995], the fine-scale solution operator M : V ∗ → V ′ M:V^*\to V' M : V ∗ → V ′ includes the negative sign, so u ′ = M ( L u ‾ − f ) u'=M(\mathcal{L}\overline{u}-f) u ′ = M ( L u − f ) . Substitution yields the closed coarse equation:
a ( w ‾ , u ‾ ) + ( L ∗ w ‾ , M ( L u ‾ − f ) ) = ( w ‾ , f ) , (2) a(\overline{w},\overline{u})+(\mathcal{L}^*\overline{w},M(\mathcal{L}\overline{u}-f))=(\overline{w},f), \tag*{(2)} a ( w , u ) + ( L ∗ w , M ( L u − f )) = ( w , f ) , (2)
where L ∗ : V → V ∗ \mathcal{L}^*:V\to V^* L ∗ : V → V ∗ is the adjoint of L \mathcal{L} L , and the second term in (2) is the subgrid correction encoding unresolved-scale feedback. In our setting, u ‾ \overline{u} u , u ′ u' u ′ , and L \mathcal{L} L correspond to the LoRA-space update, unresolved full-space curvature, and endogenous Hessian operator H ~ \widetilde{H} H , respectively. The subgrid correction corresponds to the VMS-EO fine-scale curvature injection developed in § 3 .
Method
Building on the rank-1 curvature primitive ( H ~ , v ) (\widetilde{H},v) ( H , v ) from Eq. (1) and the VMS formulation coupling coarse and fine scales from Eq. (2) , this section develops and analyzes ENGINE (Figure 3 ). §3.1 contrasts the rigid linear operator L \mathcal{L} L of classical VMS with the dynamic Hessian H ( θ ) H(\theta) H ( θ ) of optimization, motivating why classical VMS cannot be ported verbatim. §3.2 derives the central correction τ s g s \tau_{\mathrm{sgs}} τ sgs and its rank-1 LoRA-branch surrogate. §3.3 embeds VMS-EO into a V-cycle so that the endogenous state v f v_f v f is refreshed across phases. §3.4 then quantifies how coarse sampling and exact subgrid correction affect variance and bias relative to a common target.
Figure 3. ENGINE overview. Left: the gap between the restricted fine direction R ( Σ f V f ) \mathcal{R}(\Sigma_f V_f) R ( Σ f V f ) and the coarse direction Σ c R ( V f ) \Sigma_c\mathcal{R}(V_f) Σ c R ( V f ) defines the subgrid correction τ s g s \tau_{\mathrm{sgs}} τ sgs . Middle: V-cycle alternates Coarse (ϕ \phi ϕ ) and Fine (W W W ) phases, with every S S S steps a fine probe refreshing v f ← v f − ν g ( v f ) v_f \leftarrow v_f-\nu g(v_f) v f ← v f − ν g ( v f ) and λ \lambda λ (Hessian refresh), while VMS-EO (τ s g s \tau_{\mathrm{sgs}} τ sgs ) injects the subgrid correction into each coarse step. Right: classical VMS uses a fixed operator L \mathcal{L} L , while ENGINE uses the dynamic Hessian H ( θ ) H(\theta) H ( θ ) .
The multiscale state is carried by three rank-1 curvature vectors: the full-space v f ∈ R n v_f\in\mathbb{R}^n v f ∈ R n and the two LoRA-branch curvatures v B ∈ R r v_B\in\mathbb{R}^r v B ∈ R r , v A ∈ R n v_A\in\mathbb{R}^n v A ∈ R n . We instantiate the resolved coarse scale by a rank-r r r LoRA factorization Δ W = B A \Delta W=BA Δ W = B A with B ∈ R m × r B\in\mathbb{R}^{m\times r} B ∈ R m × r , A ∈ R r × n A\in\mathbb{R}^{r\times n} A ∈ R r × n , and r ≪ min ( m , n ) r\ll\min(m,n) r ≪ min ( m , n ) , so the coarse parameter ϕ = ( B , A ) \phi=(B,A) ϕ = ( B , A ) has dimension d c = r ( m + n ) d_c=r(m+n) d c = r ( m + n ) , much smaller than d f = m n d_f=mn d f = mn . Fine-scale information is obtained from the full parameter space and used to correct curvature effects not captured by the coarse model.
Endogenous Operator: Why Classical VMS Fails?
Classical VMS starts from a linear problem L u = f \mathcal{L}u=f L u = f with operator L \mathcal{L} L fixed by the underlying physics, which is what allows the unresolved component u ′ u' u ′ to be eliminated analytically in closed form (Figure 3 Right, top). In optimization the analogous linear equation is the Newton step H ( θ ) Δ θ = − ∇ f ( θ ) \mathbf{H}(\theta)\Delta\theta=-\nabla f(\theta) H ( θ ) Δ θ = − ∇ f ( θ ) , giving the correspondence ( L , u , f ) ↔ ( H , Δ θ , − ∇ f ) (\mathcal{L},u,f)\leftrightarrow(\mathbf{H},\Delta\theta,-\nabla f) ( L , u , f ) ↔ ( H , Δ θ , − ∇ f ) . Unlike L \mathcal{L} L , however, H ( θ ) \mathbf{H}(\theta) H ( θ ) co-evolves with the iterate (Figure 3 Right, bottom), so the closure cannot be derived once and for all but must be refreshed periodically by the V-cycle of § 3.3 . We call this state-dependent operator an endogenous operator and the resulting subgrid correction VMS-EO.
Coarse increments are represented by changes in the LoRA factors. The prolongation operator P : ( Δ B , Δ A ) ↦ B Δ A + Δ B A ∈ R m × n \mathcal{P}:(\Delta\mathbf{B},\Delta\mathbf{A})\mapsto\mathbf{B}\Delta\mathbf{A}+\Delta\mathbf{B}\mathbf{A}\in\mathbb{R}^{m\times n} P : ( Δ B , Δ A ) ↦ B Δ A + Δ BA ∈ R m × n lifts a coarse update to the full space (the differential of the LoRA product), and the restriction operator R = ( R B , R A ) \mathcal{R}=(\mathcal{R}_{\mathbf{B}},\mathcal{R}_{\mathbf{A}}) R = ( R B , R A ) with R B ( X ) = X A T ∈ R m × r \mathcal{R}_{\mathbf{B}}(\mathbf{X})=\mathbf{X}\mathbf{A}^{\mathsf{T}}\in\mathbb{R}^{m\times r} R B ( X ) = X A T ∈ R m × r and R A ( X ) = B T X ∈ R r × n \mathcal{R}_{\mathbf{A}}(\mathbf{X})=\mathbf{B}^{\mathsf{T}}\mathbf{X}\in\mathbb{R}^{r\times n} R A ( X ) = B T X ∈ R r × n maps a full-space matrix X \mathbf{X} X to the two LoRA gradient blocks. The two operators are formal adjoints under the Frobenius inner product, R = P ∗ \mathcal{R}=\mathcal{P}^{*} R = P ∗ , i.e., ⟨ P ( Δ B , Δ A ) , X ⟩ F = ⟨ ( Δ B , Δ A ) , R ( X ) ⟩ F \langle\mathcal{P}(\Delta\mathbf{B},\Delta\mathbf{A}),\mathbf{X}\rangle_{F}=\langle(\Delta\mathbf{B},\Delta\mathbf{A}),\mathcal{R}(\mathbf{X})\rangle_{F} ⟨ P ( Δ B , Δ A ) , X ⟩ F = ⟨( Δ B , Δ A ) , R ( X ) ⟩ F for all X ∈ R m × n \mathbf{X}\in\mathbb{R}^{m\times n} X ∈ R m × n .
At LLM scale, H ∈ R d f × d f \mathbf{H}\in\mathbb{R}^{d_f\times d_f} H ∈ R d f × d f cannot be formed, so ENGINE inherits LOREN’s rank-1 Kronecker H ~ = I m ⊗ ( ρ I n + v v T ) \widetilde{\mathbf{H}}=\mathbf{I}_{m}\otimes(\rho\mathbf{I}_{n}+\mathbf{v}\mathbf{v}^{\mathsf{T}}) H = I m ⊗ ( ρ I n + v v T ) as the practical operator with preconditioner Σ = H ~ − 1 \boldsymbol{\Sigma}=\widetilde{\mathbf{H}}^{-1} Σ = H − 1 . Given parameter x \mathbf{x} x and curvature v \mathbf{v} v , LOREN draws K K K anisotropic perturbations z k = x + ε Σ 1 / 2 u k \mathbf{z}_{k}=\mathbf{x}+\varepsilon\boldsymbol{\Sigma}^{1/2}\mathbf{u}_{k} z k = x + ε Σ 1/2 u k where u k ∼ N ( 0 , I ) \mathbf{u}_{k}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) u k ∼ N ( 0 , I ) to simultaneously form RLOO estimators for both parameter and curvature gradients:
g ( x ) = 1 ε ( K − 1 ) ∑ k = 1 K ( f k − f ˉ ) Σ 1 / 2 u k , g ( v ) = 1 K − 1 ∑ k = 1 K ( f k − f ˉ ) ∇ v log p θ ( z k ) , (3) g(\mathbf{x})=\frac{1}{\varepsilon(K-1)}\sum_{k=1}^{K}(f_{k}-\bar{f})\boldsymbol{\Sigma}^{1/2}\mathbf{u}_{k},\qquad g(\mathbf{v})=\frac{1}{K-1}\sum_{k=1}^{K}(f_{k}-\bar{f})\nabla_{\mathbf{v}}\log p_{\theta}(\mathbf{z}_{k}), \tag*{(3)} g ( x ) = ε ( K − 1 ) 1 k = 1 ∑ K ( f k − f ˉ ) Σ 1/2 u k , g ( v ) = K − 1 1 k = 1 ∑ K ( f k − f ˉ ) ∇ v log p θ ( z k ) , (3)
where f k : = f ( z k ) f_{k}:=f(\mathbf{z}_{k}) f k := f ( z k ) , f ˉ : = 1 K ∑ j f j \bar{f}:=\frac{1}{K}\sum_{j}f_{j} f ˉ := K 1 ∑ j f j , and the closed form of the score function ∇ v log p θ ( z ) \nabla_{\mathbf{v}}\log p_{\theta}(\mathbf{z}) ∇ v log p θ ( z ) is given in LOREN Eq. (10). The two estimators share the same batch { u k , f k } \{\mathbf{u}_{k},f_{k}\} { u k , f k } , so each step costs only K K K forward passes. ENGINE samples in LoRA coordinates with (3) to avoid d f d_f d f -scale variance, while injecting full-space curvature feedback as a subgrid correction back into the coarse phase.
VMS-EO Subgrid Correction
To construct the correction in Figure 3 (Left), we first identify where coarse and fine directions differ. Lemma 3.1 establishes exact restriction of first-order gradients. Proposition 3.2 then gives the curvature mismatch introduced by preconditioning. This residual supplies the corrected LoRA updates in Eqs. (7) and (8) .
Closed-Form Curvature Correction. Let G ∈ R m × n \mathbf{G}\in\mathbb{R}^{m\times n} G ∈ R m × n denote a full-space gradient-like matrix. Deterministically G = ∇ W f \mathbf{G}=\nabla_{\mathbf{W}}f G = ∇ W f , while in the ZO implementation it is replaced by the RLOO estimator.
Lemma 3.1 (chain rule). LoRA restriction commutes with first-order differentiation: R ( ∇ W f ) = ∇ ( B , A ) f \mathcal{R}(\nabla_{\mathbf{W}}f)=\nabla_{(\mathbf{B},\mathbf{A})}f R ( ∇ W f ) = ∇ ( B , A ) f . Proof in Appendix B.1.
Lemma 3.1 shows that restricting the exact fine-space gradient gives the coarse-space gradient. Curvature preconditioning introduces a mismatch between the two directions. Let Σ f , Σ B , Σ A \boldsymbol{\Sigma}_{f},\boldsymbol{\Sigma}_{B},\boldsymbol{\Sigma}_{A} Σ f , Σ B , Σ A be the rank-1 Kronecker covariances built from v f , v B , v A \mathbf{v}_{f},\mathbf{v}_{B},\mathbf{v}_{A} v f , v B , v A (Sherman-Morrison forms in Appendix B.2), and Σ c : = diag ( Σ B , Σ A ) \boldsymbol{\Sigma}_{c}:=\operatorname{diag}(\boldsymbol{\Sigma}_{B},\boldsymbol{\Sigma}_{A}) Σ c := diag ( Σ B , Σ A ) . In general R ( Σ f G ) ≠ Σ c R ( G ) \mathcal{R}(\boldsymbol{\Sigma}_{f}\mathbf{G})\ne\boldsymbol{\Sigma}_{c}\mathcal{R}(\mathbf{G}) R ( Σ f G ) = Σ c R ( G ) , so the two preconditioned directions can differ in both magnitude and orientation. This gap defines the subgrid residual :
τ s g s : = R ( Σ f G ) − Σ c R ( G ) , (4) \boldsymbol{\tau}_{\mathrm{sgs}}:=\mathcal{R}(\boldsymbol{\Sigma}_{f}\mathbf{G})-\boldsymbol{\Sigma}_{c}\mathcal{R}(\mathbf{G}), \tag*{(4)} τ sgs := R ( Σ f G ) − Σ c R ( G ) , (4)
with branch components τ s g s , B ∈ R m × r \boldsymbol{\tau}_{\mathrm{sgs},B}\in\mathbb{R}^{m\times r} τ sgs , B ∈ R m × r and τ s g s , A ∈ R r × n \boldsymbol{\tau}_{\mathrm{sgs},A}\in\mathbb{R}^{r\times n} τ sgs , A ∈ R r × n . The following result expresses each branch residual through the rank-1 curvature vectors.
Proposition 3.2 (closed-form subgrid residual). Under the rank-1 surrogates above with ρ > 0 \rho>0 ρ > 0 , the branch residuals are:
τ s g s , B = 1 ρ [ G A T v B v B T ρ + ∥ v B ∥ 2 − G v f ( A v f ) T ρ + ∥ v f ∥ 2 ] , (5) \boldsymbol{\tau}_{\mathrm{sgs},B}=\frac{1}{\rho}\left[\frac{\mathbf{G}\mathbf{A}^{\mathsf{T}}\mathbf{v}_{B}\mathbf{v}_{B}^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_{B}\rVert^{2}}-\frac{\mathbf{G}\mathbf{v}_{f}(\mathbf{A}\mathbf{v}_{f})^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_{f}\rVert^{2}}\right], \tag*{(5)} τ sgs , B = ρ 1 [ ρ + ∥ v B ∥ 2 G A T v B v B T − ρ + ∥ v f ∥ 2 G v f ( A v f ) T ] , (5) τ s g s , A = 1 ρ [ B T G v A v A T ρ + ∥ v A ∥ 2 − B T G v f v f T ρ + ∥ v f ∥ 2 ] . (6) \boldsymbol{\tau}_{\mathrm{sgs},A}=\frac{1}{\rho}\left[\frac{\mathbf{B}^{\mathsf{T}}\mathbf{G}\mathbf{v}_{A}\mathbf{v}_{A}^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_{A}\rVert^{2}}-\frac{\mathbf{B}^{\mathsf{T}}\mathbf{G}\mathbf{v}_{f}\mathbf{v}_{f}^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_{f}\rVert^{2}}\right]. \tag*{(6)} τ sgs , A = ρ 1 [ ρ + ∥ v A ∥ 2 B T G v A v A T − ρ + ∥ v f ∥ 2 B T G v f v f T ] . (6)
Proof in Appendix B.2.
Lemma 3.1 and Proposition 3.2 together identify τ s g s \tau_{\mathrm{sgs}} τ sgs as the second-order curvature gap that VMS-EO must inject. We now describe the corrected coarse update.
Rank-1 Surrogate for LoRA Branches. Let s c : = Σ c R ( G ) s_{\mathrm{c}} := \Sigma_{\mathrm{c}}\mathcal{R}(\mathbf{G}) s c := Σ c R ( G ) be the pure coarse preconditioned direction, which equals the LoRA-space LOREN direction without any VMS-EO correction. VMS-EO simply adds a branch-weighted subgrid correction s c + λ τ s g s s_{\mathrm{c}}+\lambda\tau_{\mathrm{sgs}} s c + λ τ sgs , where λ = 0 \lambda=0 λ = 0 recovers LoRA-space LOREN and a larger λ \lambda λ injects more fine-grid curvature feedback. The two branches carry independent weights λ B , λ A ∈ [ 0 , 1 ] \lambda_{\mathrm{B}},\lambda_{\mathrm{A}}\in[0,1] λ B , λ A ∈ [ 0 , 1 ] .
VMS-EO corrected updates. Let g ^ c , B ∈ R m × r \widehat{\mathbf{g}}_{\mathrm{c},\mathrm{B}}\in\mathbb{R}^{m\times r} g c , B ∈ R m × r and g ^ c , A ∈ R r × n \widehat{\mathbf{g}}_{\mathrm{c},\mathrm{A}}\in\mathbb{R}^{r\times n} g c , A ∈ R r × n denote the preconditioned RLOO directions for the two LoRA branches from Eq. (3) . With branch step sizes η B , η A > 0 \eta_{\mathrm{B}},\eta_{\mathrm{A}}>0 η B , η A > 0 , the corrected updates are:
Δ B = − η B ( g ^ c , B + λ B τ s g s , B ) . (7) \Delta\mathbf{B}=-\eta_{\mathrm{B}}\left(\widehat{\mathbf{g}}_{\mathrm{c},\mathrm{B}}+\lambda_{\mathrm{B}}\tau_{\mathrm{sgs},\mathrm{B}}\right). \tag*{(7)} Δ B = − η B ( g c , B + λ B τ sgs , B ) . (7)
Δ A = − η A ( g ^ c , A + λ A τ s g s , A ) . (8) \Delta\mathbf{A}=-\eta_{\mathrm{A}}\left(\widehat{\mathbf{g}}_{\mathrm{c},\mathrm{A}}+\lambda_{\mathrm{A}}\tau_{\mathrm{sgs},\mathrm{A}}\right). \tag*{(8)} Δ A = − η A ( g c , A + λ A τ sgs , A ) . (8)
Implementation details are given in Appendix D.1.
Adaptive mixing weights. Proposition C.7 characterizes an oracle mixing weight that minimizes the estimation MSE. Since it depends on unknown correction and error statistics, we use normalized angular disagreement between coarse and fine curvature directions as a practical proxy for selecting λ B , λ A ∈ [ 0 , 1 ] \lambda_{\mathrm{B}},\lambda_{\mathrm{A}}\in[0,1] λ B , λ A ∈ [ 0 , 1 ] . For the B-branch we use the formula below, and the A-branch is analogous after replacing v B \mathbf{v}_{\mathrm{B}} v B with v A \mathbf{v}_{\mathrm{A}} v A and the projection A v f A\mathbf{v}_{\mathrm{f}} A v f with v f \mathbf{v}_{\mathrm{f}} v f :
δ B : = 1 2 ∥ v B ∥ v B ∥ − A v f ∥ A v f ∥ ∥ , λ B = clip ( δ B , λ min , λ max ) . (9) \delta_{\mathrm{B}}:=\frac{1}{2}\left\lVert\frac{\mathbf{v}_{\mathrm{B}}}{\lVert\mathbf{v}_{\mathrm{B}}\rVert}-\frac{A\mathbf{v}_{\mathrm{f}}}{\lVert A\mathbf{v}_{\mathrm{f}}\rVert}\right\rVert,\qquad\lambda_{\mathrm{B}}=\operatorname{clip}(\delta_{\mathrm{B}},\lambda_{\min},\lambda_{\max}). \tag*{(9)} δ B := 2 1 ∥ v B ∥ v B − ∥ A v f ∥ A v f , λ B = clip ( δ B , λ m i n , λ m a x ) . (9)
When δ \delta δ approaches zero the two curvature directions become aligned, so λ \lambda λ falls to λ min \lambda_{\min} λ m i n and the correction receives its minimum mixing weight. When δ \delta δ approaches one the directions diverge, so λ \lambda λ rises to λ max \lambda_{\max} λ m a x and injects more fine-grid feedback.
Engine: V-cycle Coupling
ENGINE alternates between M M M -epoch coarse (ϕ \phi ϕ ) and N N N -epoch fine (W \mathbf{W} W ) phases (Figure 3 Middle), updating the base W 0 \mathbf{W}_0 W 0 in W = W 0 + B A \mathbf{W}=\mathbf{W}_0+\mathbf{B}\mathbf{A} W = W 0 + BA only at phase boundaries. A cyclic schedule couples them, using P \mathcal{P} P and R \mathcal{R} R operators to transfer parameters, momentum, and curvature.
Two-phase schedule. Coarse phase. ENGINE updates ϕ = ( B , A ) \phi=(\mathbf{B},\mathbf{A}) ϕ = ( B , A ) via Eqs. (7) and (8) and uses (3) to take one curvature step per branch with learning rate ν > 0 \nu>0 ν > 0 (Algorithm 1, line 6). Crucially, the K K K RLOO samples { u k , f k } \{u_k,f_k\} { u k , f k } are reused from those already drawn for ( Δ B , Δ A ) (\Delta\mathbf{B},\Delta\mathbf{A}) ( Δ B , Δ A ) , so this curvature step costs no additional forward passes. The operator H ~ = H ~ ( θ , v f ) \widetilde{\mathbf{H}}=\widetilde{\mathbf{H}}(\theta,\mathbf{v}_{\mathrm{f}}) H = H ( θ , v f ) is endogenous, however, so v f \mathbf{v}_{\mathrm{f}} v f goes stale across coarse steps and causes τ s g s \tau_{\mathrm{sgs}} τ sgs to inject outdated information through its v f \mathbf{v}_{\mathrm{f}} v f dependence. We therefore trigger a Hessian refresh every S S S coarse steps as illustrated in Algorithm 1, where K K K extra full-space perturbations update v f \mathbf{v}_{\mathrm{f}} v f using (3) with learning rate ν \nu ν together with the projection scalars in Proposition 3.2 while θ \theta θ , B \mathbf{B} B , and A \mathbf{A} A remain unchanged. This refresh-and-reuse strategy parallels Lazy Hessian [[2 ] ], which also recomputes curvature every S S S steps and reuses it in between to cut arithmetic cost without sacrificing convergence.
Fine phase. ENGINE returns to the full parameter space and applies Σ f \Sigma_{\mathrm{f}} Σ f -preconditioned LOREN steps from (3) to W \mathbf{W} W , simultaneously refreshing v f \mathbf{v}_{\mathrm{f}} v f for the next coarse phase.
Prolongation P \mathcal{P} P from coarse to fine. The coarse update B A \mathbf{B}\mathbf{A} BA is absorbed into the base weight W 0 \mathbf{W}_0 W 0 . We save the current A \mathbf{A} A as A o l d \mathbf{A}_{\mathrm{old}} A old before reinitialization. This cached matrix maps vectors from R n \mathbb{R}^{n} R n to R r \mathbb{R}^{r} R r during the next assimilation from fine to coarse. Branch momentum combine into m f \mathbf{m}_{\mathrm{f}} m f via the LoRA product rule, and curvature is EMA-blended with weight β \beta β . Algorithm 1 specifies the transfers.
Restriction R \mathcal{R} R from fine to coarse. We set the base weight W 0 \mathbf{W}_0 W 0 to W f i n e _ b e s t \mathbf{W}_{\mathrm{fine\_best}} W fine_best , rescale B \mathbf{B} B , and reset A \mathbf{A} A to zero so that W 0 + B A = W 0 \mathbf{W}_0+\mathbf{B}\mathbf{A}=\mathbf{W}_0 W 0 + BA = W 0 holds exactly at cycle start. Momentum is projected onto the two branches via R \mathcal{R} R with default damping factor α = 0.5 \alpha=0.5 α = 0.5 , and the two branch curvatures use the cached basis A o l d \mathbf{A}_{\mathrm{old}} A old . Algorithm 1 gives these transfer formulas.
Algorithm 1 ENGINE (see Appendix § 10 for details).
Require: frozen W W W , dataset D D D , rank r r r , cycles V V V , coarse/fine epochs M M M , N N N , queries K K K , smoothing ϵ \epsilon ϵ , damping ρ \rho ρ , learning rates η B \eta_B η B , η A \eta_A η A , η f \eta_f η f , ν \nu ν , mixing range [ λ min , λ max ] [\lambda_{\min}, \lambda_{\max}] [ λ m i n , λ m a x ] , transfer weight β \beta β , fine-probe interval S S S (every S S S coarse steps, refresh v f v_f v f and λ \lambda λ ).
Initialize B B B randomly, set A ← 0 A \leftarrow0 A ← 0 , and initialize v B v_B v B , v A v_A v A , v f v_f v f .
for v = 1 , … , V v = 1, \ldots, V v = 1 , … , V do
Coarse phase (M M M epochs in ϕ = ( B , A ) \phi= (B, A) ϕ = ( B , A ) ): ▷ Coarse space
for t = 1 , … , M ⋅ steps/epoch t = 1, \ldots, M \cdot\text{steps/epoch} t = 1 , … , M ⋅ steps/epoch do
Draw K K K LoRA perturbations and estimate g ^ c , B \widehat{g}_{c,B} g c , B , g ^ c , A \widehat{g}_{c,A} g c , A via (3) .
v B ← v B − ν g ( v B ) v_B \leftarrow v_B - \nu g(v_B) v B ← v B − ν g ( v B ) , v A ← v A − ν g ( v A ) v_A \leftarrow v_A - \nu g(v_A) v A ← v A − ν g ( v A ) on the same K K K samples.
if t m o d S = 0 t \bmod S = 0 t mod S = 0 then
v f ← v f − ν g ( v f ) v_f \leftarrow v_f - \nu g(v_f) v f ← v f − ν g ( v f ) via (3) . ▷ Hessian refresh, K K K extra fwd
Update λ B \lambda_B λ B , λ A \lambda_A λ A via (9) .
end if
Compute τ s g s \tau_{\mathrm{sgs}} τ sgs via Prop. 3.2 (Eqs. (5) and (6) ). ▷ VMS-EO
Update ( B , A ) (B, A) ( B , A ) via Eqs. (7) and (8) .
end for
Prolongation P P P (C → F C \to F C → F ): W 0 ← W 0 + B A W_0 \leftarrow W_0 + BA W 0 ← W 0 + B A , A o l d ← A A_{\mathrm{old}} \leftarrow A A old ← A , m f = m B A + B m A m_f = m_BA + Bm_A m f = m B A + B m A , v f ← normalize ( ( 1 − β ) v f + β normalize ( A T v B + v A ) ) v_f \leftarrow\operatorname{normalize}((1-\beta)v_f + \beta\operatorname{normalize}(A^{\mathsf{T}}v_B+v_A)) v f ← normalize (( 1 − β ) v f + β normalize ( A T v B + v A )) .
Fine phase (N N N epochs in W W W ): LOREN on W W W with lr η f \eta_f η f , K K K queries via (3) , updating v f v_f v f . ▷ Fine space
Restriction R R R (F → C F \to C F → C ): W 0 ← W f i n e _ b e s t W_0 \leftarrow W_{\mathrm{fine\_best}} W 0 ← W fine_best , rescale B B B , A ← 0 A \leftarrow0 A ← 0 , m A = α B T m f m_A = \alpha B^{\mathsf{T}}m_f m A = α B T m f , m B = α m f A o l d T m_B = \alpha m_f A_{\mathrm{old}}^{\mathsf{T}} m B = α m f A old T , v A = normalize ( v f ) v_A = \operatorname{normalize}(v_f) v A = normalize ( v f ) , v B = normalize ( A o l d v f ) v_B = \operatorname{normalize}(A_{\mathrm{old}}v_f) v B = normalize ( A old v f ) .
end for
return W W W .
ENGINE maintains only the LoRA factors ( B , A ) (B, A) ( B , A ) and the rank-1 curvature vectors ( v f , v B , v A ) (v_f, v_B, v_A) ( v f , v B , v A ) , never explicitly storing a full Hessian. Algorithm 1 summarizes the procedure, and implementation details are in Appendix § 10 .
Statistical and Convergence Analysis
We provide a detailed analysis of the trade-off between bias and variance in ENGINE. Under appropriate assumptions, coarse sampling reduces the dimension-dependent variance bound and exact correction eliminates structural bias in the local-linear limit. We also establish convergence bounds for coarse updates.
Fix the current weights, LoRA factors, and covariances. Subscripts f f f and c c c denote fine and coarse spaces. In row-wise vector coordinates, let g f r a w g_f^{\mathrm{raw}} g f raw be the full-space gradient and g c r a w = R ( g f r a w ) g_c^{\mathrm{raw}} = \mathcal{R}(g_f^{\mathrm{raw}}) g c raw = R ( g f raw ) the gradient with respect to ϕ \phi ϕ . The coarse direction is s c = Σ c g c r a w s_c = \Sigma_c g_c^{\mathrm{raw}} s c = Σ c g c raw , and the common comparison target is g ⋆ = R ( Σ f g f r a w ) g_{\star} = \mathcal{R}(\Sigma_f g_f^{\mathrm{raw}}) g ⋆ = R ( Σ f g f raw ) . Thus Eq. (4) gives τ s g s = g ⋆ − s c \tau_{\mathrm{sgs}} = g_{\star} - s_c τ sgs = g ⋆ − s c .
We use a Gaussian estimator with b b b independent directions and smoothing radius ϵ \epsilon ϵ . Write F ( ϕ ) = f ( W 0 + B A ) F(\phi) = f(W_0 + BA) F ( ϕ ) = f ( W 0 + B A ) and draw z i ∼ N ( 0 , Σ c ) z_i \sim\mathcal{N}(0, \Sigma_c) z i ∼ N ( 0 , Σ c ) independently. Define:
g ^ c , ϵ , b = 1 b ∑ i = 1 b F ( ϕ + ϵ z i ) − F ( ϕ ) ϵ z i . (10) \widehat{g}_{c,\epsilon,b} = \frac{1}{b} \sum_{i=1}^{b} \frac{F(\phi+\epsilon z_i)-F(\phi)}{\epsilon}z_i. \tag*{(10)} g c , ϵ , b = b 1 i = 1 ∑ b ϵ F ( ϕ + ϵ z i ) − F ( ϕ ) z i . (10)
Here z i = [ z B , i T , z A , i T ] T ∈ R d c z_i = [z_{B,i}^{\mathsf{T}}, z_{A,i}^{\mathsf{T}}]^{\mathsf{T}} \in\mathbb{R}^{d_c} z i = [ z B , i T , z A , i T ] T ∈ R d c stacks B- and A-branch perturbations of dimensions m r mr m r and r n rn r n , with d c = r ( m + n ) d_c = r(m+n) d c = r ( m + n ) . The covariance Σ c = diag ( Σ B , Σ A ) ∈ R d c × d c \Sigma_c = \operatorname{diag}(\Sigma_B, \Sigma_A) \in\mathbb{R}^{d_c \times d_c} Σ c = diag ( Σ B , Σ A ) ∈ R d c × d c has blocks of sizes m r × m r mr \times mr m r × m r and r n × r n rn \times rn r n × r n , respectively. Implementation uses RLOO from Eq. (3) . The next result isolates the effect of exact correction in the local-linear limit, where the smoothing bias vanishes.
Proposition 3.3 (Common-target error with exact correction). Suppose F F F has a Lipschitz continuous gradient and Σ c \Sigma_c Σ c is fixed and positive definite. As ϵ → 0 \epsilon\to0 ϵ → 0 with b b b fixed, Eq. (10) converges in mean square to g ^ c , 0 , b = b − 1 ∑ i z i z i T g c r a w \widehat{g}_{c,0,b} = b^{-1}\sum_i z_i z_i^{\mathsf{T}}g_c^{\mathrm{raw}} g c , 0 , b = b − 1 ∑ i z i z i T g c raw . Its mean is s c s_c s c and its trace variance is:
V c , 0 = tr ( Σ c ) ( g c r a w ) T Σ c g c r a w + ( g c r a w ) T Σ c 2 g c r a w b . (11) V_{c,0} = \frac{\operatorname{tr}(\Sigma_c)(g_c^{\mathrm{raw}})^{\mathsf{T}}\Sigma_c g_c^{\mathrm{raw}} + (g_c^{\mathrm{raw}})^{\mathsf{T}}\Sigma_c^2 g_c^{\mathrm{raw}}}{b}. \tag*{(11)} V c , 0 = b tr ( Σ c ) ( g c raw ) T Σ c g c raw + ( g c raw ) T Σ c 2 g c raw . (11)
With exact deterministic correction and both branch weights equal to one, the common-target errors satisfy:
E ∥ g ^ c , 0 , b − g ⋆ ∥ 2 = V c , 0 + ∥ τ s g s ∥ 2 , (12) \mathbb{E}\lVert\widehat{\mathbf{g}}_{c,0,b}-\mathbf{g}_\star\rVert^2 = V_{c,0} + \lVert\tau\mathbf{s}_{\mathrm{gs}}\rVert^2, \tag*{(12)} E ∥ g c , 0 , b − g ⋆ ∥ 2 = V c , 0 + ∥ τ s gs ∥ 2 , (12)
E ∥ g ^ c , 0 , b + τ s g s − g ⋆ ∥ 2 = V c , 0 . (13) \mathbb{E}\lVert\widehat{\mathbf{g}}_{c,0,b}+\tau\mathbf{s}_{\mathrm{gs}}-\mathbf{g}_\star\rVert^2 = V_{c,0}. \tag*{(13)} E ∥ g c , 0 , b + τ s gs − g ⋆ ∥ 2 = V c , 0 . (13)
Theorem C.3 proves the stated mean-square limit of the Gaussian estimator g ^ c , ϵ , b \widehat{\mathbf{g}}_{c,\epsilon,b} g c , ϵ , b . Lemma C.2 gives Eq. (11) , and Proposition C.4 yields Eqs. (12) and (13) in the limit. Proof in Appendix § 9 .§ 3.3 .
Variance reduction and bias correction. For isotropic sampling with Σ c = I d c \boldsymbol{\Sigma}_c = I_{d_c} Σ c = I d c , Eq. (11) reduces to V c , 0 = d c + 1 b ∥ g r a w ∥ 2 V_{c,0} = \frac{d_c+1}{b}\lVert\mathbf{g}^{\mathrm{raw}}\rVert^2 V c , 0 = b d c + 1 ∥ g raw ∥ 2 , compared with the fine-space factor d f + 1 b \frac{d_f+1}{b} b d f + 1 . For anisotropic sampling with fixed positive-definite Σ c \boldsymbol{\Sigma}_c Σ c and ∥ Σ c ∥ o p ≤ M c \lVert\boldsymbol{\Sigma}_c\rVert_{\mathrm{op}} \le M_c ∥ Σ c ∥ op ≤ M c , the same equation gives the upper bound V c , 0 ≤ d c + 1 b M c 2 ∥ g r a w ∥ 2 V_{c,0} \le\frac{d_c+1}{b}M_c^2\lVert\mathbf{g}^{\mathrm{raw}}\rVert^2 V c , 0 ≤ b d c + 1 M c 2 ∥ g raw ∥ 2 . Under comparable gradient and spectral scales, d c ≪ d f d_c \ll d_f d c ≪ d f reduces the dimension-dependent variance bound. In the local-linear limit, Proposition C.4 shows that exact correction eliminates the structural bias relative to g ⋆ \mathbf{g}_\star g ⋆ while preserving the coarse sampling variance, decreasing MSE by ∥ τ s g s ∥ 2 \lVert\tau\mathbf{s}_{\mathrm{gs}}\rVert^2 ∥ τ s gs ∥ 2 . Ideal ENGINE thus combines coarse-space variance reduction with structural-bias elimination. Appendix § 9 .§ 3 gives finite-radius bounds and a common-target comparison in Table 4 .
Table 4.
For finite smoothing radius ϵ \epsilon ϵ and L c L_c L c -smooth F F F , Theorem C.3 in Appendix § 9 .§ 2 gives ∥ b c , ϵ ∥ = O ( ϵ d c ) \lVert\mathbf{b}_{c,\epsilon}\rVert= \mathcal{O}(\epsilon\sqrt{d_c}) ∥ b c , ϵ ∥ = O ( ϵ d c ) with L c L_c L c and M c M_c M c held fixed. Here b c , ϵ = E g ^ c , ϵ , b − s c \mathbf{b}_{c,\epsilon} = \mathbb{E}\widehat{\mathbf{g}}_{c,\epsilon,b} - \mathbf{s}_c b c , ϵ = E g c , ϵ , b − s c is the residual smoothing bias. Proposition C.4 gives the MSE reduction ∥ τ s g s ∥ 2 − 2 ⟨ b c , ϵ , τ s g s ⟩ \lVert\tau\mathbf{s}_{\mathrm{gs}}\rVert^2 - 2\langle\mathbf{b}_{c,\epsilon}, \tau\mathbf{s}_{\mathrm{gs}}\rangle ∥ τ s gs ∥ 2 − 2 ⟨ b c , ϵ , τ s gs ⟩ , quantifying the effect of this remaining bias. For approximate corrections, Proposition C.5 accounts for their bias, variance, and dependence on the coarse estimate. Proposition C.6 in Appendix § 9 .§ 4 separates row compression error from scalar coefficient mismatch.
Convergence analysis. Consider the ideal Gaussian update ϕ t + 1 = ϕ t − η ( g ^ c , ϵ , b , t + τ s g s , t ) \phi_{t+1} = \phi_t - \eta(\widehat{\mathbf{g}}_{c,\epsilon,b,t} + \tau\mathbf{s}_{\mathrm{gs},t}) ϕ t + 1 = ϕ t − η ( g c , ϵ , b , t + τ s gs , t ) on a fixed coarse objective F F F . Under the assumptions in Appendix § 9 .6, Theorem C.9 gives the following bound for the prescribed η = Θ ( T − 1 / 2 ) \eta= \Theta(T^{-1/2}) η = Θ ( T − 1/2 ) :
1 T ∑ t = 0 T − 1 E ∥ ∇ F ( ϕ t ) ∥ 2 ≤ O ( T − 1 / 2 ) + O ( ϵ 2 ) . (14) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\lVert\nabla F(\phi_t)\rVert^2 \le O(T^{-1/2}) + O(\epsilon^2). \tag*{(14)} T 1 t = 0 ∑ T − 1 E ∥ ∇ F ( ϕ t ) ∥ 2 ≤ O ( T − 1/2 ) + O ( ϵ 2 ) . (14)
Here T T T counts iterations. The bound separates optimization decay from the smoothing residual, with coarse-dimensional sampling constants. Corollary C.10 extends the bound to approximate corrections, with correction bias affecting the residual and correction variance affecting the sampling constants.
Experiments
We evaluate ENGINE on 16 (model, task) combinations, assessing accuracy, efficiency, and the contributions of V-cycle and VMS-EO.
Experimental Setup and Main Results
Setup. We evaluate four autoregressive LLMs of size 1.5B to 13B, namely GPT-2-XL [16 ] , OPT-2.7B and OPT-13B [22 ] , and LLaMA-3-8B [4 ] , on 16 prompt-free classification tasks drawn from GLUE [19 ] and SuperGLUE [20 ] . We compare ENGINE against LOREN together with five other ZO baselines, whose full list and background appear in Appendix § 7 . All baselines are independently reproduced under LOREN’s setup. On top of this protocol, ENGINE introduces only two new hyperparameters. The C60F60 V-cycle alternates 60 coarse epochs with 60 fine epochs, and the curvature refresh interval S S S is set per (model, task) combination as listed in Table 8 . The LoRA rank r r r scales with model size. We report mean accuracy (%) over five seeds for all tasks. See Appendices § 11 and § 12 for details.
Table 8. Specific hyperparameters for ENGINE. Each row gives a model’s learning rate, rank, and refresh intervals S S S in coarse steps. All four models use ν = 10 − 3 \nu= 10^{-3} ν = 1 0 − 3 , ρ = 10 − 3 \rho= 10^{-3} ρ = 1 0 − 3 , C60F60, and mixing bounds [ 0.3 , 0.8 ] [0.3, 0.8] [ 0.3 , 0.8 ] .
Main results. Table 2 and Figure 2 show that ENGINE outperforms LOREN and other baseline methods on every (model, task) cell. Representative training curves appear in Figure 4 .
Table 2. Main results using accuracy (%) for all tasks. Bold and cyan mark the best and second best per column. Highlighted rows give ENGINE and its absolute gain over LOREN. Full results in Appendix § 11 .§ 1 .
Figure 4. Visualization of validation accuracy on GPT-2-XL across three GLUE tasks (MNLI, QNLI, SST-2). ENGINE reaches higher peak accuracy than all baselines on every task. See Appendix § 11 .§ 2 for training loss curves and all four models.
Efficiency and memory. Figure 1 further shows that on every setting, ENGINE reaches LOREN’s best accuracy with substantially fewer forward passes and less wall-clock time, achieving up to a 2.61× \times × forward-pass speed-up and a 2.58× \times × wall-clock speed-up on LLaMA-3-8B (CB). Meanwhile,
ENGINE’s peak GPU memory matches LOREN’s, because the fine phase is equivalent to vanilla LOREN and determines the peak, while the coarse phase only stores LoRA-scale optimizer and curvature states. Table 11 compares all methods at common target accuracies.
Table 11: Query and wall-clock speed-ups at common target accuracies. Each entry divides the largest query count or time in its column by that method’s value, assigning the least efficient method 1.0×. Higher is better, and bold marks the best result.
Table 11.
Ablation: Where does ENGINE’s gain originate from?
Setup and design. We compare LOREN, ENGINE w/o VMS-EO, and ENGINE on the same 16 settings, with other settings matched (Appendix § 12 ). The variant without VMS-EO retains the V-cycle and disables the correction. Its comparison with LOREN isolates the V-cycle’s contribution, while its comparison with full ENGINE isolates the correction’s contribution to performance.
Analysis: Attributing the Dual-Scale Benefits of Engine
Result. Table 3 yields two observations. First, ENGINE w/o VMS-EO has a mix of wins and losses against LOREN across the 16 settings. Second, ENGINE achieves higher accuracy than both LOREN and ENGINE w/o VMS-EO across all four models. Table 6 gives the full mean± \pm ± std results, and Figure 9 shows the ablation convergence curves.
Table 3. Ablation accuracy (%) on all 16 settings. Bold marks the best per column, and cyan marks the second best. The highlighted row is our method, with the green Δ \Delta Δ vs LOREN row showing the absolute gain. Full results in Appendix § 11 .
Table 6. VMS-EO ablation over five seeds, reported as mean accuracy (%) ± standard deviation.
Figure 9. VMS-EO ablation across four models and 16 tasks. Validation accuracy versus forward passes for ENGINE (red), ENGINE w/o VMS-EO (grey), and LOREN (orange). From top to bottom, the rows show LLaMA-3-8B, OPT-13B, GPT-2-XL, and OPT-2.7B. In the first two rows, columns show RTE, BoolQ, WiC, and CB. In the last two rows, columns show MNLI, QNLI, SST-2, and CoLA.
Analysis. The two observations identify distinct contributions.
The V-cycle alone achieves mixed gains against LOREN. Coarse sampling in d c ≪ d f d_c \ll d_f d c ≪ d f dimensions reduces the variance bound under comparable gradient and spectral scales (Proposition 3.3). The second-order restriction mismatch in Proposition 3.2 biases the uncorrected coarse direction and can offset this variance benefit. Alternating fine phases retain full-space curvature-aware optimization, while coarse phases reduce per-step cost.
Incorporating VMS-EO yields consistent performance gains. By feeding full-space curvature information back into the subspace through Eqs. (7) and (8) , VMS-EO corrects the structural bias. Proposition 3.3 shows that exact correction eliminates this bias without increasing coarse sampling variance in the local-linear limit. Combining coarse sampling with curvature correction thus helps explain ENGINE’s gains over LOREN and the uncorrected V-cycle.
Conclusions
In this paper, we propose ENGINE, a zeroth-order LLM fine-tuning framework inspired by Variational MultiScale (VMS) theory from computational mechanics. From the VMS perspective, we treat the LoRA parameter space as the coarse space and the full parameter space as the fine space. By reformulating VMS with an endogenous, state-dependent Hessian, we derive the VMS-EO subgrid correction to incorporate fine-space curvature information into coarse updates. ENGINE integrates this correction into an alternating V-cycle that couples the two spaces through mutual feedback. Our statistical analysis shows that, under the stated assumptions, coarse sampling reduces the dimension-dependent variance bound and exact subgrid correction eliminates curvature-mismatch bias in the local-linear limit. We further derive nonconvex stationarity bounds for corrected coarse updates. Across 16 model-task settings, ENGINE consistently outperforms state-of-the-art ZO baselines, achieving substantial forward-pass and wall-clock speed-ups with no additional peak memory overhead. More broadly, this work provides a multiscale perspective on optimization, showing how efficient low-dimensional updates can be guided by full-space curvature information. We hope this perspective inspires further connections between multiscale modeling and optimization. Appendix H discusses limitations and future work.
Acknowledgments and Disclosure of Funding
We gratefully acknowledge the support of Westlake University Research Center for Industries of the Future and Westlake University Center for High-performance Computing.
The authors declare no competing interests.
Related Work
ENGINE connects research on zeroth-order optimization, low-rank adaptation, and Variational MultiScale theory. These three lines of research motivate its sampling strategy, parameter spaces, and correction mechanism.
Zeroth-order LLM fine-tuning. Zeroth-order methods estimate gradients from function evaluations and reduce the memory required for LLM fine-tuning. MeZO uses in-place parameter perturbations to achieve memory usage comparable to inference. Its gradient estimates have high variance, which can slow optimization. MeZO-SVRG [3 ] uses periodic reference gradients to reduce this variance, with additional memory for stored parameters and gradients. LOZO [1 ] exploits the low-rank structure of gradients to reduce the dimension of gradient estimation. Curvature information provides another way to improve the search direction. HiZOO [23 ] estimates the diagonal Hessian and uses it to precondition the update. LOREN uses a low-rank Kronecker approximation to incorporate curvature into perturbation sampling. ENGINE builds on this curvature representation, combining LoRA sampling with full-parameter updates. Its subgrid correction injects full-space curvature information into coarse updates.
Low-rank adaptation. LoRA represents a weight update as the product of two small matrices and optimizes their entries while keeping the base weight fixed. This reduces the number of trainable parameters and provides the coarse coordinates used by ENGINE. ReLoRA [11 ] periodically merges the low-rank update into the base weight and reinitializes the low-rank factors. Successive updates can therefore accumulate in the full weight matrix. ReLoRA uses this procedure within first-order training. ENGINE combines parameter merging with alternating LoRA and full-space ZO updates, transferring momentum and curvature between the two spaces.
Variational MultiScale theory. Variational MultiScale (VMS) theory originates in computational mechanics. It separates a problem into resolved coarse scales and unresolved fine scales. The effect of the fine scales enters the coarse equation through a subgrid correction. In the classical linear formulation, the governing operator is fixed by the physical problem and determines this correction. ENGINE brings this perspective into zeroth-order optimization. The LoRA coordinates define the coarse space, and the correction accounts for curvature information lost when forming the coarse update. The curvature operator is estimated online and evolves during optimization. This state dependence motivates VMS with an Endogenous Operator.
Structural Derivations and Proofs
In this appendix, we prove the LoRA chain rule and derive the subgrid residual formulas.
Proof of Lemma 3.1 (LoRA chain rule)
Proof. With W 0 \mathbf{W}_0 W 0 fixed, differentiating W = W 0 + B A \mathbf{W} = \mathbf{W}_0 + \mathbf{B}\mathbf{A} W = W 0 + BA gives:
d W = d B A + B d A . \mathrm{d}\mathbf{W} = \mathrm{d}\mathbf{B}\,\mathbf{A} + \mathbf{B}\,\mathrm{d}\mathbf{A}. d W = d B A + B d A .
The Frobenius inner product and cyclic trace identity give:
d f = ⟨ ∇ W f , d W ⟩ F = ⟨ ( ∇ W f ) A T , d B ⟩ F + ⟨ B T ∇ W f , d A ⟩ F . \begin{aligned}
\mathrm{d}f &= \langle\nabla_{\mathbf{W}}f,\mathrm{d}\mathbf{W}\rangle_F \\
&= \langle(\nabla_{\mathbf{W}}f)\mathbf{A}^{\mathsf{T}},\mathrm{d}\mathbf{B}\rangle_F + \langle\mathbf{B}^{\mathsf{T}}\nabla_{\mathbf{W}}f,\mathrm{d}\mathbf{A}\rangle_F.
\end{aligned} d f = ⟨ ∇ W f , d W ⟩ F = ⟨( ∇ W f ) A T , d B ⟩ F + ⟨ B T ∇ W f , d A ⟩ F .
Matching the coefficients of the independent differentials gives the components of R ( ∇ W f ) \mathcal{R}(\nabla_{\mathbf{W}}f) R ( ∇ W f ) :
∂ f ∂ B = ( ∇ W f ) A T , ∂ f ∂ A = B T ∇ W f . \begin{aligned}
\frac{\partial f}{\partial\mathbf{B}} &= (\nabla_{\mathbf{W}}f)\mathbf{A}^{\mathsf{T}}, \\
\frac{\partial f}{\partial\mathbf{A}} &= \mathbf{B}^{\mathsf{T}}\nabla_{\mathbf{W}}f.
\end{aligned} ∂ B ∂ f ∂ A ∂ f = ( ∇ W f ) A T , = B T ∇ W f .
Proof of Proposition 3.2 (subgrid residual)
Proof. For ρ > 0 \rho> 0 ρ > 0 and v ∈ R d \mathbf{v} \in\mathbb{R}^{d} v ∈ R d , the Sherman-Morrison identity gives:
( ρ I d + v v T ) − 1 = 1 ρ ( I d − v v T ρ + ∥ v ∥ 2 ) . (15) (\rho\mathit{I}_{d}+\mathbf{v}\mathbf{v}^{\mathsf{T}})^{-1}=\frac{1}{\rho}\left(\mathit{I}_{d}-\frac{\mathbf{v}\mathbf{v}^{\mathsf{T}}}{\rho+\lVert\mathbf{v}\rVert^{2}}\right). \tag*{(15)} ( ρ I d + v v T ) − 1 = ρ 1 ( I d − ρ + ∥ v ∥ 2 v v T ) . (15)
The Kronecker structure of Σ f \Sigma_f Σ f applies ( ρ I n + v f v f T ) − 1 (\rho\mathit{I}_{n}+\mathbf{v}_f\mathbf{v}_f^{\mathsf{T}})^{-1} ( ρ I n + v f v f T ) − 1 independently to each row of G G G . Equation (B.1) therefore gives:
Σ f G = 1 ρ ( G − G v f v f T ρ + ∥ v f ∥ 2 ) . \Sigma_fG=\frac{1}{\rho}\left(G-\frac{G\mathbf{v}_f\mathbf{v}_f^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_f\rVert^{2}}\right). Σ f G = ρ 1 ( G − ρ + ∥ v f ∥ 2 G v f v f T ) .
Applying R B ( X ) = X A T \mathcal{R}_B(X)=X\mathit{A}^{\mathsf{T}} R B ( X ) = X A T yields:
R B ( Σ f G ) = 1 ρ ( G A T − G v f ( A v f ) T ρ + ∥ v f ∥ 2 ) . \mathcal{R}_B(\Sigma_fG)=\frac{1}{\rho}\left(G\mathit{A}^{\mathsf{T}}-\frac{G\mathbf{v}_f(\mathit{A}\mathbf{v}_f)^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_f\rVert^{2}}\right). R B ( Σ f G ) = ρ 1 ( G A T − ρ + ∥ v f ∥ 2 G v f ( A v f ) T ) .
For the coarse direction, apply Eq. (B.1) with d = r d=r d = r and v = v B \mathbf{v}=\mathbf{v}_B v = v B to each row of R B ( G ) \mathcal{R}_B(G) R B ( G ) to obtain:
Σ B R B ( G ) = 1 ρ ( G A T − G A T v B v B T ρ + ∥ v B ∥ 2 ) . \Sigma_B\mathcal{R}_B(G)=\frac{1}{\rho}\left(G\mathit{A}^{\mathsf{T}}-\frac{G\mathit{A}^{\mathsf{T}}\mathbf{v}_B\mathbf{v}_B^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_B\rVert^{2}}\right). Σ B R B ( G ) = ρ 1 ( G A T − ρ + ∥ v B ∥ 2 G A T v B v B T ) .
Subtracting the coarse direction from the restricted fine direction cancels the G A T G\mathit{A}^{\mathsf{T}} G A T terms and gives:
τ s g s , B = 1 ρ [ G A T v B v B T ρ + ∥ v B ∥ 2 − G v f ( A v f ) T ρ + ∥ v f ∥ 2 ] , \tau_{\mathrm{sgs},B}=\frac{1}{\rho}\left[\frac{G\mathit{A}^{\mathsf{T}}\mathbf{v}_B\mathbf{v}_B^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_B\rVert^{2}}-\frac{G\mathbf{v}_f(\mathit{A}\mathbf{v}_f)^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_f\rVert^{2}}\right], τ sgs , B = ρ 1 [ ρ + ∥ v B ∥ 2 G A T v B v B T − ρ + ∥ v f ∥ 2 G v f ( A v f ) T ] ,
which proves Eq. (5) .
The same calculation with R A ( X ) = B T X \mathcal{R}_A(X)=\mathit{B}^{\mathsf{T}}X R A ( X ) = B T X and the A-branch covariance gives:
τ s g s , A = 1 ρ [ B T G v A v A T ρ + ∥ v A ∥ 2 − B T G v f v f T ρ + ∥ v f ∥ 2 ] , \tau_{\mathrm{sgs},A}=\frac{1}{\rho}\left[\frac{\mathit{B}^{\mathsf{T}}G\mathbf{v}_A\mathbf{v}_A^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_A\rVert^{2}}-\frac{\mathit{B}^{\mathsf{T}}G\mathbf{v}_f\mathbf{v}_f^{\mathsf{T}}}{\rho+\lVert\mathbf{v}_f\rVert^{2}}\right], τ sgs , A = ρ 1 [ ρ + ∥ v A ∥ 2 B T G v A v A T − ρ + ∥ v f ∥ 2 B T G v f v f T ] ,
which proves Eq. (6) .
Theoretical Analysis
This appendix presents the statistical and convergence analysis of ENGINE. Section C.1 defines the notation and estimation setting. Section C.2 derives Gaussian bias and variance bounds. Section C.3 compares estimation errors at a common target. Section C.4 bounds scalar projection errors. Section C.5 derives the optimal mixing weight. Section C.6 establishes convergence bounds for exact and approximate correction.
Setup and assumptions
Fine and coarse spaces. Subscripts f f f and c c c denote fine and coarse spaces, corresponding to full parameters and LoRA factors, respectively. Write W = W 0 + B A W=W_0+BA W = W 0 + B A , with fixed W 0 W_0 W 0 , B ∈ R m × r B\in\mathbb{R}^{m\times r} B ∈ R m × r , and A ∈ R r × n A\in\mathbb{R}^{r\times n} A ∈ R r × n . Using row-wise vectorization, define w = vec ( W ) ∈ R d f w=\operatorname{vec}(W)\in\mathbb{R}^{d_f} w = vec ( W ) ∈ R d f and ϕ = ( vec ( B ) T , vec ( A ) T ) T ∈ R d c \phi=(\operatorname{vec}(B)^{\mathsf{T}},\operatorname{vec}(A)^{\mathsf{T}})^{\mathsf{T}}\in\mathbb{R}^{d_c} ϕ = ( vec ( B ) T , vec ( A ) T ) T ∈ R d c , where d f = m n d_f=mn d f = mn and d c = r ( m + n ) d_c=r(m+n) d c = r ( m + n ) . The fine-space loss is f ( w ) f(w) f ( w ) , and its LoRA parameterization is F ( ϕ ) = f ( w ( ϕ ) ) F(\phi)=f(w(\phi)) F ( ϕ ) = f ( w ( ϕ )) . We omit boldface on matrices and directions in this appendix.
Gradients and restriction. The superscript raw denotes exact gradients before preconditioning. We write g f r a w = ∇ w f ( w ) ∈ R d f g_f^{\mathrm{raw}}=\nabla_w f(w)\in\mathbb{R}^{d_f} g f raw = ∇ w f ( w ) ∈ R d f and g c r a w = ∇ ϕ F ( ϕ ) ∈ R d c g_c^{\mathrm{raw}}=\nabla_\phi F(\phi)\in\mathbb{R}^{d_c} g c raw = ∇ ϕ F ( ϕ ) ∈ R d c . Let J = ∂ w ∂ ϕ J=\frac{\partial w}{\partial\phi} J = ∂ ϕ ∂ w and R = J T ∈ R d c × d f R=J^{\mathsf{T}}\in\mathbb{R}^{d_c\times d_f} R = J T ∈ R d c × d f represent prolongation and restriction. The chain rule gives g c r a w = R g f r a w g_c^{\mathrm{raw}}=R g_f^{\mathrm{raw}} g c raw = R g f raw . For G ∈ R m × n G\in\mathbb{R}^{m\times n} G ∈ R m × n , restriction acts as R vec ( G ) = ( vec ( G A T ) T , vec ( B T G ) T ) T R\operatorname{vec}(G)=(\operatorname{vec}(G A^{\mathsf{T}})^{\mathsf{T}},\operatorname{vec}(B^{\mathsf{T}}G)^{\mathsf{T}})^{\mathsf{T}} R vec ( G ) = ( vec ( G A T ) T , vec ( B T G ) T ) T . For the gradient matrix G = ∇ W f G=\nabla_W f G = ∇ W f , we have g f r a w = vec ( G ) g_f^{\mathrm{raw}}=\operatorname{vec}(G) g f raw = vec ( G ) .
Preconditioned directions and correction. Let Σ f ∈ R d f × d f \Sigma_f \in\mathbb{R}^{d_f \times d_f} Σ f ∈ R d f × d f and Σ c = diag ( Σ B , Σ A ) ∈ R d c × d c \Sigma_c = \operatorname{diag}(\Sigma_B, \Sigma_A) \in\mathbb{R}^{d_c \times d_c} Σ c = diag ( Σ B , Σ A ) ∈ R d c × d c be symmetric positive-definite preconditioners and perturbation covariances. The branch blocks have sizes m r × m r m_r \times m_r m r × m r and r n × r n r_n \times r_n r n × r n , respectively. Define the preconditioned directions g f = Σ f g f r a w g_f = \Sigma_f g_f^{\mathrm{raw}} g f = Σ f g f raw and g c = Σ c g c r a w g_c = \Sigma_c g_c^{\mathrm{raw}} g c = Σ c g c raw , the common target g ⋆ = R g f g_\star= Rg_f g ⋆ = R g f , and the exact subgrid residual τ = g ⋆ − g c \tau= g_\star- g_c τ = g ⋆ − g c . The common target g ⋆ ∈ R d c g_\star\in\mathbb{R}^{d_c} g ⋆ ∈ R d c applies fine-space preconditioning before restriction, while g c g_c g c applies coarse-space preconditioning after restriction. Here g c g_c g c and τ \tau τ are vector forms of s c s_c s c and τ s g s \tau_{\mathrm{sgs}} τ sgs .
Estimation setting and errors. Let g ^ c \hat{g}_c g ^ c denote the preconditioned coarse estimate and d ^ = g ^ c + λ τ \hat{d} = \hat{g}_c + \lambda\tau d ^ = g ^ c + λ τ the corrected direction. The mean squared error and bias relative to g ⋆ g_\star g ⋆ are E ∥ d ^ − g ⋆ ∥ 2 \mathbb{E}\|\hat{d} - g_\star\|^2 E ∥ d ^ − g ⋆ ∥ 2 and E d ^ − g ⋆ \mathbb{E}\hat{d} - g_\star E d ^ − g ⋆ , respectively. For a random vector X X X , Var ( X ) = E ∥ X − E X ∥ 2 \operatorname{Var}(X) = \mathbb{E}\|X - \mathbb{E}X\|^2 Var ( X ) = E ∥ X − E X ∥ 2 denotes trace variance. Vector norms are Euclidean. Expectations condition on current parameters and preconditioners and average over fresh perturbations. For simplicity, we analyze Gaussian estimators with perturbation radius ε \varepsilon ε and b b b independent directions, where b b b plays the role of K K K in the method. Exact correction uses conditionally deterministic τ \tau τ and a fixed mixing weight λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] shared by both branches. Section C.6 specifies the conditions for history-dependent weights.
Bias and variance of Gaussian estimators
Consider an L L L -smooth objective φ \varphi φ on R d \mathbb{R}^d R d , g = ∇ φ ( x ) g = \nabla\varphi(x) g = ∇ φ ( x ) , a fixed SPD covariance Σ \Sigma Σ with ∥ Σ ∥ o p ≤ M \|\Sigma\|_{\mathrm{op}} \le M ∥Σ ∥ op ≤ M , and independent z i ∼ N ( 0 , Σ ) z_i \sim\mathcal{N}(0,\Sigma) z i ∼ N ( 0 , Σ ) . With deterministic function evaluations, define:
g ^ ε , b = 1 b ∑ i = 1 b φ ( x + ε z i ) − φ ( x ) ε z i , φ ε , Σ ( x ) = E φ ( x + ε z ) . (16) \hat{g}_{\varepsilon,b} = \frac{1}{b}\sum_{i=1}^{b} \frac{\varphi(x+\varepsilon z_i)-\varphi(x)}{\varepsilon}z_i,\qquad\varphi_{\varepsilon,\Sigma}(x) = \mathbb{E}\varphi(x+\varepsilon z). \tag*{(16)} g ^ ε , b = b 1 i = 1 ∑ b ε φ ( x + ε z i ) − φ ( x ) z i , φ ε , Σ ( x ) = E φ ( x + ε z ) . (16)
For fixed x x x , Σ \Sigma Σ , and b b b , define the actual trace variance by V ε = E ∥ g ^ ε , b − E g ^ ε , b ∥ 2 V_\varepsilon= \mathbb{E}\|\hat{g}_{\varepsilon,b} - \mathbb{E}\hat{g}_{\varepsilon,b}\|^2 V ε = E ∥ g ^ ε , b − E g ^ ε , b ∥ 2 . This quantity is defined at every positive smoothing radius. The subscript 0 below refers to the estimator obtained in the limit ε → 0 \varepsilon\to0 ε → 0 , with the number of random directions b b b held fixed. We first derive the mean and bound the smoothing bias separately from sampling variance.
Lemma C.1 (Gaussian smoothing identity). Under the setting above , the estimator has mean E g ^ ε , b = Σ ∇ φ ε , Σ ( x ) \mathbb{E}\hat{g}_{\varepsilon,b} = \Sigma\nabla\varphi_{\varepsilon,\Sigma}(x) E g ^ ε , b = Σ∇ φ ε , Σ ( x ) . Its bias b ε = E g ^ ε , b − Σ g b_\varepsilon= \mathbb{E}\hat{g}_{\varepsilon,b} - \Sigma g b ε = E g ^ ε , b − Σ g obeys ∥ b ε ∥ ≤ L ε ∥ Σ ∥ o p tr ( Σ ) ≤ L ε d M 3 \|b_\varepsilon\| \le L\varepsilon\|\Sigma\|_{\mathrm{op}}\sqrt{\operatorname{tr}(\Sigma)} \le L\varepsilon\sqrt{dM^3} ∥ b ε ∥ ≤ L ε ∥Σ ∥ op tr ( Σ ) ≤ L ε d M 3 .
Proof. Write z = Σ 1 / 2 u z = \Sigma^{1/2}u z = Σ 1/2 u with u ∼ N ( 0 , I d ) u \sim\mathcal{N}(0,I_d) u ∼ N ( 0 , I d ) . Since E z = 0 \mathbb{E}z = 0 E z = 0 , the constant baseline φ ( x ) \varphi(x) φ ( x ) contributes zero to the mean. Gaussian integration by parts, applied componentwise, gives:
E [ u φ ( x + ε Σ 1 / 2 u ) ] = ε Σ 1 / 2 E ∇ φ ( x + ε Σ 1 / 2 u ) . \mathbb{E}\left[u\varphi\left(x+\varepsilon\Sigma^{1/2}u\right)\right] = \varepsilon\Sigma^{1/2}\mathbb{E}\nabla\varphi\left(x+\varepsilon\Sigma^{1/2}u\right). E [ u φ ( x + ε Σ 1/2 u ) ] = ε Σ 1/2 E ∇ φ ( x + ε Σ 1/2 u ) .
Global L L L -smoothness bounds gradient growth linearly and function growth quadratically, justifying integration by parts and differentiation under the Gaussian expectation. Therefore, we have:
E g ^ ε , b = ε − 1 Σ 1 / 2 E [ u φ ( x + ε Σ 1 / 2 u ) ] = Σ E ∇ φ ( x + ε z ) = Σ ∇ φ ε , Σ ( x ) . \begin{aligned}
\mathbb{E}\hat{g}_{\varepsilon,b} = \varepsilon^{-1}\Sigma^{1/2}\mathbb{E}\left[u\varphi\left(x+\varepsilon\Sigma^{1/2}u\right)\right] \\
&= \Sigma\mathbb{E}\nabla\varphi(x+\varepsilon z) \\
&= \Sigma\nabla\varphi_{\varepsilon,\Sigma}(x).
\end{aligned} E g ^ ε , b = ε − 1 Σ 1/2 E [ u φ ( x + ε Σ 1/2 u ) ] = Σ E ∇ φ ( x + ε z ) = Σ∇ φ ε , Σ ( x ) .
Averaging b b b samples preserves this mean. Subtracting Σ g \Sigma g Σ g and using the Lipschitz continuity of the gradient yields:
∥ b ε ∥ = ∥ Σ E [ ∇ φ ( x + ε z ) − ∇ φ ( x ) ] ∥ ≤ ∥ Σ ∥ o p E ∥ ∇ φ ( x + ε z ) − ∇ φ ( x ) ∥ ≤ L ε ∥ Σ ∥ o p E ∥ z ∥ ≤ L ε ∥ Σ ∥ o p E ∥ z ∥ 2 . \begin{aligned}
\|b_\varepsilon\| = \|\Sigma\mathbb{E}[\nabla\varphi(x+\varepsilon z)-\nabla\varphi(x)]\| \\
&\le\|\Sigma\|_{\mathrm{op}}\mathbb{E}\|\nabla\varphi(x+\varepsilon z)-\nabla\varphi(x)\| \\
&\le L\varepsilon\|\Sigma\|_{\mathrm{op}}\mathbb{E}\|z\| \\
&\le L\varepsilon\|\Sigma\|_{\mathrm{op}}\sqrt{\mathbb{E}\|z\|^2}.
\end{aligned} ∥ b ε ∥ = ∥Σ E [ ∇ φ ( x + ε z ) − ∇ φ ( x )] ∥ ≤ ∥Σ ∥ op E ∥∇ φ ( x + ε z ) − ∇ φ ( x ) ∥ ≤ L ε ∥Σ ∥ op E ∥ z ∥ ≤ L ε ∥Σ ∥ op E ∥ z ∥ 2 .
Using E ∥ z ∥ 2 = tr ( Σ ) ≤ d M \mathbb{E}\|z\|^2 = \operatorname{tr}(\Sigma) \le dM E ∥ z ∥ 2 = tr ( Σ ) ≤ d M and ∥ Σ ∥ o p ≤ M \|\Sigma\|_{\mathrm{op}} \le M ∥Σ ∥ op ≤ M completes the bias bound.
The next identity gives the moments of the linear term in the finite-difference estimator. These moments enter the finite-radius variance bound.
Lemma C.2 (Gaussian covariance identity). For z ∼ N ( 0 , Σ ) z \sim\mathcal{N}(0,\Sigma) z ∼ N ( 0 , Σ ) and fixed g g g , let Y = ( g T z ) z Y=(g^{\mathsf{T}}z)z Y = ( g T z ) z . Then we have:
E Y = Σ g , Cov ( Y ) = ( g T Σ g ) Σ + Σ g g T Σ . \mathbb{E}Y=\Sigma g,\qquad\operatorname{Cov}(Y)=(g^{\mathsf{T}}\Sigma g)\Sigma+\Sigma gg^{\mathsf{T}}\Sigma. E Y = Σ g , Cov ( Y ) = ( g T Σ g ) Σ + Σ g g T Σ. The covariance of the average of b b b independent copies is Cov ( Y ) b \frac{\operatorname{Cov}(Y)}{b} b Cov ( Y ) .
Proof. Since E [ z z T ] = Σ \mathbb{E}[zz^{\mathsf{T}}]=\Sigma E [ z z T ] = Σ and g g g is fixed, the mean is E Y = Σ g \mathbb{E}Y=\Sigma g E Y = Σ g . For coordinates i , j i,j i , j , expand the second moment and use the Gaussian fourth-moment identity to obtain:
E [ Y i Y j ] = ∑ k , l g k g l E [ z i z j z k z l ] = ∑ k , l g k g l ( Σ i j Σ k l + Σ i k Σ j l + Σ i l Σ j k ) = Σ i j ( g T Σ g ) + 2 ( Σ g ) i ( Σ g ) j . \begin{aligned}
\mathbb{E}[Y_iY_j]=\sum_{k,l}g_kg_l\mathbb{E}[z_iz_jz_kz_l] \\
&=\sum_{k,l}g_kg_l(\Sigma_{ij}\Sigma_{kl}+\Sigma_{ik}\Sigma_{jl}+\Sigma_{il}\Sigma_{jk}) \\
&=\Sigma_{ij}(g^{\mathsf{T}}\Sigma g)+2(\Sigma g)_i(\Sigma g)_j.
\end{aligned} E [ Y i Y j ] = k , l ∑ g k g l E [ z i z j z k z l ] = k , l ∑ g k g l ( Σ ij Σ k l + Σ ik Σ j l + Σ i l Σ j k ) = Σ ij ( g T Σ g ) + 2 ( Σ g ) i ( Σ g ) j .
Subtracting ( Σ g ) ( Σ g ) T (\Sigma g)(\Sigma g)^{\mathsf{T}} ( Σ g ) ( Σ g ) T gives the stated covariance. For independent copies Y ( 1 ) , … , Y ( b ) Y^{(1)},\ldots,Y^{(b)} Y ( 1 ) , … , Y ( b ) , cross-covariances vanish, so Cov ( b − 1 ∑ i = 1 b Y ( i ) ) = Cov ( Y ) / b \operatorname{Cov}(b^{-1}\sum_{i=1}^{b}Y^{(i)})=\operatorname{Cov}(Y)/b Cov ( b − 1 ∑ i = 1 b Y ( i ) ) = Cov ( Y ) / b . Taking its trace yields:
E ∥ 1 b ∑ i = 1 b Y ( i ) − Σ g ∥ 2 = tr ( Σ ) g T Σ g + g T Σ 2 g b . \mathbb{E}\left\|\frac{1}{b}\sum_{i=1}^{b}Y^{(i)}-\Sigma g\right\|^{2}
=\frac{\operatorname{tr}(\Sigma)g^{\mathsf{T}}\Sigma g+g^{\mathsf{T}}\Sigma^{2}g}{b}. E b 1 i = 1 ∑ b Y ( i ) − Σ g 2 = b tr ( Σ ) g T Σ g + g T Σ 2 g .
These identities and a Taylor remainder bound yield the finite-radius estimation error bound.
Theorem C.3 (Gaussian estimation error). The mean is Σ ∇ φ ε , Σ ( x ) \Sigma\nabla\varphi_{\varepsilon,\Sigma}(x) Σ∇ φ ε , Σ ( x ) and its bias relative to Σ g \Sigma g Σ g satisfies ∥ b ε ∥ ≤ L ε d M 3 \|b_{\varepsilon}\|\le L\varepsilon\sqrt{dM^{3}} ∥ b ε ∥ ≤ L ε d M 3 . For finite ε \varepsilon ε , the MSE satisfies:
E ∥ g ^ ε , b − Σ g ∥ 2 ≤ 2 ( d + 1 ) M 2 b ∥ g ∥ 2 + L 2 ε 2 M 3 [ d ( d + 2 ) ( d + 4 ) 2 b + d ] . (17) \mathbb{E}\|\widehat{g}_{\varepsilon,b}-\Sigma g\|^{2}
\le\frac{2(d+1)M^{2}}{b}\|g\|^{2}+L^{2}\varepsilon^{2}M^{3}\left[\frac{d(d+2)(d+4)}{2b}+d\right]. \tag*{(17)} E ∥ g ε , b − Σ g ∥ 2 ≤ b 2 ( d + 1 ) M 2 ∥ g ∥ 2 + L 2 ε 2 M 3 [ 2 b d ( d + 2 ) ( d + 4 ) + d ] . (17) For fixed x x x , Σ \Sigma Σ , and b b b , the estimator converges in mean square as ε → 0 \varepsilon\to0 ε → 0 to g ^ 0 , b = 1 b ∑ i = 1 b ( g T z i ) z i \widehat{g}_{0,b}=\frac{1}{b}\sum_{i=1}^{b}(g^{\mathsf{T}}z_i)z_i g 0 , b = b 1 ∑ i = 1 b ( g T z i ) z i .
Proof. The mean and bias bound follow from Lemma C.1. We use Lemma C.2 to control the linear term and bound the remaining finite-difference error below.
Step 1. Control the Taylor remainder. For one sample, let Y = ( g T z ) z Y=(g^{\mathsf{T}}z)z Y = ( g T z ) z and write:
X ε = φ ( x + ε z ) − φ ( x ) ε z = Y + U ε , \begin{aligned}
X_{\varepsilon}=\frac{\varphi(x+\varepsilon z)-\varphi(x)}{\varepsilon}z \\
&=Y+U_{\varepsilon},
\end{aligned} X ε = ε φ ( x + ε z ) − φ ( x ) z = Y + U ε ,
U ε = ( φ ( x + ε z ) − φ ( x ) ε − g T z ) z . U_{\varepsilon}=\left(\frac{\varphi(x+\varepsilon z)-\varphi(x)}{\varepsilon}-g^{\mathsf{T}}z\right)z. U ε = ( ε φ ( x + ε z ) − φ ( x ) − g T z ) z .
The smoothness inequality gives ∣ φ ( x + ε z ) − φ ( x ) − ε g T z ∣ ≤ L ε 2 ∥ z ∥ 2 2 |\varphi(x+\varepsilon z)-\varphi(x)-\varepsilon g^{\mathsf{T}}z|\le\frac{L\varepsilon^{2}\|z\|^{2}}{2} ∣ φ ( x + ε z ) − φ ( x ) − ε g T z ∣ ≤ 2 L ε 2 ∥ z ∥ 2 , and consequently:
E ∥ U ε ∥ 2 ≤ L 2 ε 2 4 E ∥ z ∥ 6 ≤ L 2 ε 2 M 3 4 d ( d + 2 ) ( d + 4 ) . \begin{aligned}
\mathbb{E}\|U_{\varepsilon}\|^{2}\le\frac{L^{2}\varepsilon^{2}}{4}\mathbb{E}\|z\|^{6} \\
&\le\frac{L^{2}\varepsilon^{2}M^{3}}{4}d(d+2)(d+4).
\end{aligned} E ∥ U ε ∥ 2 ≤ 4 L 2 ε 2 E ∥ z ∥ 6 ≤ 4 L 2 ε 2 M 3 d ( d + 2 ) ( d + 4 ) .
Here we use ∥ Σ 2 u ∥ 2 ≤ M ∥ u ∥ 2 \|\Sigma^{2}u\|^{2}\le M\|u\|^{2} ∥ Σ 2 u ∥ 2 ≤ M ∥ u ∥ 2 and the Gaussian moment E ∥ u ∥ 6 = d ( d + 2 ) ( d + 4 ) \mathbb{E}\|u\|^{6}=d(d+2)(d+4) E ∥ u ∥ 6 = d ( d + 2 ) ( d + 4 ) . Using the same sampled directions, Jensen’s inequality gives:
E ∥ g ^ ε , b − g ^ 0 , b ∥ 2 = E ∥ 1 b ∑ i = 1 b U ε , i ∥ 2 ≤ 1 b ∑ i = 1 b E ∥ U ε , i ∥ 2 ⟶ 0 as ε → 0. \begin{aligned}
\mathbb{E}\|\widehat{g}_{\varepsilon,b}-\widehat{g}_{0,b}\|^{2}
=\mathbb{E}\left\|\frac{1}{b}\sum_{i=1}^{b}U_{\varepsilon,i}\right\|^{2} \\
&\le\frac{1}{b}\sum_{i=1}^{b}\mathbb{E}\|U_{\varepsilon,i}\|^{2}\longrightarrow0
\quad\text{as }\varepsilon\to0.
\end{aligned} E ∥ g ε , b − g 0 , b ∥ 2 = E b 1 i = 1 ∑ b U ε , i 2 ≤ b 1 i = 1 ∑ b E ∥ U ε , i ∥ 2 ⟶ 0 as ε → 0.
For fixed b b b , this proves mean-square convergence to g ^ 0 , b \widehat{g}_{0,b} g 0 , b and V ε → V 0 V_{\varepsilon} \to V_0 V ε → V 0 as ε → 0 \varepsilon\to0 ε → 0 .
Step 2. Bound the centered variance. Applying ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 \lVert a+b\rVert^2 \le2\lVert a\rVert^2+2\lVert b\rVert^2 ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 to the centered Y Y Y and U ε U_{\varepsilon} U ε gives:
Var ( X ε ) ≤ 2 Var ( Y ) + 2 Var ( U ε ) ≤ 2 [ tr ( Σ ) g T Σ g + g T Σ 2 g ] + 2 E ∥ U ε ∥ 2 ≤ 2 ( d + 1 ) M 2 ∥ g ∥ 2 + L 2 ε 2 M 3 2 d ( d + 2 ) ( d + 4 ) . \begin{aligned}
\operatorname{Var}(X_{\varepsilon}) &\le2\operatorname{Var}(Y)+2\operatorname{Var}(U_{\varepsilon})\\
&\le2\left[\operatorname{tr}(\Sigma)g^{\mathsf T}\Sigma g+g^{\mathsf T}\Sigma^2g\right]+2\mathbb{E}\lVert U_{\varepsilon}\rVert^2\\
&\le2(d+1)M^2\lVert g\rVert^2+\frac{L^2\varepsilon^2M^3}{2}d(d+2)(d+4).
\end{aligned} Var ( X ε ) ≤ 2 Var ( Y ) + 2 Var ( U ε ) ≤ 2 [ tr ( Σ ) g T Σ g + g T Σ 2 g ] + 2 E ∥ U ε ∥ 2 ≤ 2 ( d + 1 ) M 2 ∥ g ∥ 2 + 2 L 2 ε 2 M 3 d ( d + 2 ) ( d + 4 ) .
Here g T Σ g ≤ M ∥ g ∥ 2 g^{\mathsf T}\Sigma g\le M\lVert g\rVert^2 g T Σ g ≤ M ∥ g ∥ 2 , g T Σ 2 g ≤ M 2 ∥ g ∥ 2 g^{\mathsf T}\Sigma^2g\le M^2\lVert g\rVert^2 g T Σ 2 g ≤ M 2 ∥ g ∥ 2 , and tr ( Σ ) ≤ d M \operatorname{tr}(\Sigma)\le dM tr ( Σ ) ≤ d M . Averaging b b b independent X ε X_{\varepsilon} X ε samples divides this variance bound by b b b .
Step 3. Add the squared smoothing bias. Write g ^ ε , b − Σ g = ( g ^ ε , b − E g ^ ε , b ) + b ε \widehat{g}_{\varepsilon,b}-\Sigma g=(\widehat{g}_{\varepsilon,b}-\mathbb{E}\widehat{g}_{\varepsilon,b})+b_{\varepsilon} g ε , b − Σ g = ( g ε , b − E g ε , b ) + b ε . The centered term has zero mean, so:
E ∥ g ^ ε , b − Σ g ∥ 2 = Var ( g ^ ε , b ) + ∥ b ε ∥ 2 ≤ 2 ( d + 1 ) M 2 b ∥ g ∥ 2 + L 2 ε 2 M 3 2 b d ( d + 2 ) ( d + 4 ) + L 2 ε 2 d M 3 . \begin{aligned}
\mathbb{E}\lVert\widehat{g}_{\varepsilon,b}-\Sigma g\rVert^2
&=\operatorname{Var}(\widehat{g}_{\varepsilon,b})+\lVert b_{\varepsilon}\rVert^2\\
&\le\frac{2(d+1)M^2}{b}\lVert g\rVert^2+\frac{L^2\varepsilon^2M^3}{2b}d(d+2)(d+4)+L^2\varepsilon^2dM^3.
\end{aligned} E ∥ g ε , b − Σ g ∥ 2 = Var ( g ε , b ) + ∥ b ε ∥ 2 ≤ b 2 ( d + 1 ) M 2 ∥ g ∥ 2 + 2 b L 2 ε 2 M 3 d ( d + 2 ) ( d + 4 ) + L 2 ε 2 d M 3 .
This is (16). The variance decreases with b b b , while the smoothing-bias term is controlled by ε \varepsilon ε .
Local-linear limit. By Theorem C.3, b ε → 0 b_{\varepsilon}\to0 b ε → 0 and g ^ ε , b → g ^ 0 , b \widehat{g}_{\varepsilon,b}\to\widehat{g}_{0,b} g ε , b → g 0 , b in mean square as ε → 0 \varepsilon\to0 ε → 0 with b b b fixed. Applying Lemma C.2 to this limiting estimator gives E g ^ 0 , b = Σ g \mathbb{E}\widehat{g}_{0,b}=\Sigma g E g 0 , b = Σ g . Its covariance and trace variance are:
Cov ( g ^ 0 , b ) = ( g T Σ g ) Σ + Σ g g T Σ b , \operatorname{Cov}(\widehat{g}_{0,b})=\frac{(g^{\mathsf T}\Sigma g)\Sigma+\Sigma gg^{\mathsf T}\Sigma}{b}, Cov ( g 0 , b ) = b ( g T Σ g ) Σ + Σ g g T Σ ,
V 0 = tr ( Σ ) g T Σ g + g T Σ 2 g b (18) V_0=\frac{\operatorname{tr}(\Sigma)g^{\mathsf T}\Sigma g+g^{\mathsf T}\Sigma^2g}{b} \tag*{(18)} V 0 = b tr ( Σ ) g T Σ g + g T Σ 2 g (18)
The limiting MSE relative to Σ g \Sigma g Σ g is therefore V 0 V_0 V 0 . For Σ = I d \Sigma=I_d Σ = I d , substitution gives V 0 = ( d + 1 ) ∥ g ∥ 2 b V_0=\frac{(d+1)\lVert g\rVert^2}{b} V 0 = b ( d + 1 ) ∥ g ∥ 2 . All constants here concern the Gaussian estimator in Eq. (16) .
Common-target MSE and exact correction
Section C.3.1 treats finite perturbations with exact correction, and Section C.3.2 includes correction error. Section C.3.3 then specializes these results to the local-linear limit.
Which estimators and which targets? We distinguish isotropic LoRA ZO from LoRA+LOREN. The former samples with covariance I d c I_{d_c} I d c and targets g c r a w g_c^{\mathrm{raw}} g c raw , whereas the latter samples with covariance Σ c \Sigma_c Σ c and targets g c = Σ c g c r a w g_c=\Sigma_cg_c^{\mathrm{raw}} g c = Σ c g c raw . Full-space LOREN samples with covariance Σ f \Sigma_f Σ f and targets g f = Σ f g f r a w g_f=\Sigma_fg_f^{\mathrm{raw}} g f = Σ f g f raw . Ideal ENGINE uses the same coarse estimator as LoRA+LOREN, shifted by the exact residual τ \tau τ . The comparison uses the Gaussian estimator models of (16) , which isolate the effects of sampling covariance and subgrid correction. For s ∈ { f , c } s\in\{f,c\} s ∈ { f , c } , superscripts iso and ani denote sampling covariances I d s I_{d_s} I d s and Σ s \Sigma_s Σ s , respectively. We abbreviate the Gaussian coarse estimator as g ^ c = g ^ c a n i \widehat{g}_c=\widehat{g}_c^{\mathrm{ani}} g c = g c ani , suppressing ε \varepsilon ε and b b b . This is g ^ c , ε , b \widehat{g}_{c,\varepsilon,b} g c , ε , b from Eq. (10) .
Finite-radius analysis
We first keep ε > 0 \varepsilon>0 ε > 0 . Let f f f and F F F have smoothness constants L f , L c L_f,L_c L f , L c , and let ∥ Σ s ∥ o p ≤ M s \lVert\Sigma_s\rVert_{\mathrm{op}}\le M_s ∥ Σ s ∥ op ≤ M s for s ∈ { f , c } s\in\{f,c\} s ∈ { f , c } . Theorem C.3 gives the variance and smoothing-bias bounds:
Var ( g ^ s a n i ) ≤ 2 ( d s + 1 ) M s 2 b ∥ g s r a w ∥ 2 + L s 2 ε 2 M s 3 2 b d s ( d s + 2 ) ( d s + 4 ) , (19) \operatorname{Var}(\widehat{g}_s^{\mathrm{ani}})\le\frac{2(d_s+1)M_s^2}{b}\lVert g_s^{\mathrm{raw}}\rVert^2+\frac{L_s^2\varepsilon^2M_s^3}{2b}d_s(d_s+2)(d_s+4), \tag*{(19)} Var ( g s ani ) ≤ b 2 ( d s + 1 ) M s 2 ∥ g s raw ∥ 2 + 2 b L s 2 ε 2 M s 3 d s ( d s + 2 ) ( d s + 4 ) , (19)
∥ E g ^ s a n i − g s ∥ ≤ L s ε d s M s 3 . (20) \lVert\mathbb{E}\widehat{g}_s^{\mathrm{ani}}-g_s\rVert\le L_s\varepsilon\sqrt{d_sM_s^3}. \tag*{(20)} ∥ E g s ani − g s ∥ ≤ L s ε d s M s 3 . (20)
For isotropic sampling, set Σ s = I d s \Sigma_s=I_{d_s} Σ s = I d s and M s = 1 M_s=1 M s = 1 , with target g s r a w g_s^{\mathrm{raw}} g s raw . All comparisons use the same direction count b b b and radius ε \varepsilon ε .
For any deterministic target a a a , expanding the centered error gives:
E ∥ X − a ∥ 2 = Var ( X ) + ∥ E X − a ∥ 2 , (21) \mathbb{E}\lVert X-a\rVert^2=\operatorname{Var}(X)+\lVert\mathbb{E}X-a\rVert^2, \tag*{(21)} E ∥ X − a ∥ 2 = Var ( X ) + ∥ E X − a ∥ 2 , (21)
since E ( X − E X ) = 0 \mathbb{E}(X-\mathbb{E}X)=0 E ( X − E X ) = 0 makes the cross term vanish. For isotropic full-space and coarse estimators, use targets g f r a w g_f^{\mathrm{raw}} g f raw and g c r a w g_c^{\mathrm{raw}} g c raw , respectively. Full-space anisotropic sampling targets g f g_f g f . To compare coarse updates with and without correction, we use the common target g ⋆ g_\star g ⋆ below.
For fixed b b b and ε > 0 \varepsilon>0 ε > 0 , the variance bound contains a term linear in d s d_s d s and a remainder of order ε 2 d c 3 \varepsilon^2d_c^3 ε 2 d c 3 . With comparable gradient, covariance, and smoothness scales bounded independently of dimension, coarse sampling reduces these dimension factors from d f = m n d_f=mn d f = mn to d c = r ( m + n ) d_c=r(m+n) d c = r ( m + n ) when d c ≪ d f d_c\ll d_f d c ≪ d f .
Exact correction at a common target. We next quantify how exact correction changes the bias and MSE while preserving the coarse sampling variance. Let g ^ c \widehat{g}_c g c denote the finite-radius coarse estimator, with smoothing bias b c , ε = E g ^ c − g c b_{c,\varepsilon}=\mathbb{E}\widehat{g}_c-g_c b c , ε = E g c − g c and trace variance V c = Var ( g ^ c ) V_c=\operatorname{Var}(\widehat{g}_c) V c = Var ( g c ) . For fixed λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , the corrected direction is d ^ λ = g ^ c + λ τ \widehat{d}_\lambda=\widehat{g}_c+\lambda\tau d λ = g c + λ τ .
Proposition C.4 (Exact-correction benefit). The ideal estimator satisfies:
Var ( d ^ λ ) = V c , E ∥ d ^ λ − g ⋆ ∥ 2 = V c + ∥ b c , ε − ( 1 − λ ) τ ∥ 2 . (22) \operatorname{Var}(\widehat{d}_\lambda)=V_c,\qquad
\mathbb{E}\lVert\widehat{d}_\lambda-g_\star\rVert^2
=V_c+\lVert b_{c,\varepsilon}-(1-\lambda)\tau\rVert^2. \tag*{(22)} Var ( d λ ) = V c , E ∥ d λ − g ⋆ ∥ 2 = V c + ∥ b c , ε − ( 1 − λ ) τ ∥ 2 . (22) The MSE reduction relative to the uncorrected estimator is:
E ∥ g ^ c − g ⋆ ∥ 2 − E ∥ d ^ λ − g ⋆ ∥ 2 = λ ( 2 − λ ) ∥ τ ∥ 2 − 2 λ ⟨ b c , ε , τ ⟩ . \mathbb{E}\lVert\widehat{g}_c-g_\star\rVert^2-\mathbb{E}\lVert\widehat{d}_\lambda-g_\star\rVert^2
=\lambda(2-\lambda)\lVert\tau\rVert^2-2\lambda\langle b_{c,\varepsilon},\tau\rangle. E ∥ g c − g ⋆ ∥ 2 − E ∥ d λ − g ⋆ ∥ 2 = λ ( 2 − λ ) ∥ τ ∥ 2 − 2 λ ⟨ b c , ε , τ ⟩ .
Proof. Conditional on the state, τ \tau τ and λ \lambda λ are deterministic, so:
Var ( d ^ λ ) = Var ( g ^ c ) , \operatorname{Var}(\widehat{d}_\lambda)=\operatorname{Var}(\widehat{g}_c), Var ( d λ ) = Var ( g c ) ,
E d ^ λ − g ⋆ = b c , ε − ( 1 − λ ) τ . \mathbb{E}\widehat{d}_\lambda-g_\star=b_{c,\varepsilon}-(1-\lambda)\tau. E d λ − g ⋆ = b c , ε − ( 1 − λ ) τ .
Applying the bias-variance identity in Eq. (21) proves Eq. (22) . Subtracting the result at λ \lambda λ from that at λ = 0 \lambda=0 λ = 0 gives:
∥ b c , ε − τ ∥ 2 − ∥ b c , ε − ( 1 − λ ) τ ∥ 2 = λ ( 2 − λ ) ∥ τ ∥ 2 − 2 λ ⟨ b c , ε , τ ⟩ . \lVert b_{c,\varepsilon}-\tau\rVert^2-\lVert b_{c,\varepsilon}-(1-\lambda)\tau\rVert^2
=\lambda(2-\lambda)\lVert\tau\rVert^2-2\lambda\langle b_{c,\varepsilon},\tau\rangle. ∥ b c , ε − τ ∥ 2 − ∥ b c , ε − ( 1 − λ ) τ ∥ 2 = λ ( 2 − λ ) ∥ τ ∥ 2 − 2 λ ⟨ b c , ε , τ ⟩ .
For finite ε \varepsilon ε and full correction λ = 1 \lambda=1 λ = 1 , the same identity gives:
MSE L o R A + L O R E N − MSE i d e a l = ∥ τ ∥ 2 − 2 ⟨ b c , ε , τ ⟩ ≥ ∥ τ ∥ ( ∥ τ ∥ − 2 L c ε d c M c 3 ) . \operatorname{MSE}_{\mathrm{LoRA+LOREN}}-\operatorname{MSE}_{\mathrm{ideal}}
=\lVert\tau\rVert^2-2\langle b_{c,\varepsilon},\tau\rangle
\ge\lVert\tau\rVert\left(\lVert\tau\rVert-2L_c\varepsilon\sqrt{d_cM_c^3}\right). MSE LoRA + LOREN − MSE ideal = ∥ τ ∥ 2 − 2 ⟨ b c , ε , τ ⟩ ≥ ∥ τ ∥ ( ∥ τ ∥ − 2 L c ε d c M c 3 ) .
The inequality follows from Cauchy-Schwarz and Eq. (20) . Thus ∥ τ ∥ > 2 L c ε d c M c 3 \lVert\tau\rVert>2L_c\varepsilon\sqrt{d_cM_c^3} ∥ τ ∥ > 2 L c ε d c M c 3 is sufficient for strictly smaller squared bias and MSE, with unchanged coarse sampling variance.
:::
Approximate correction
At finite ε \varepsilon ε , an approximate correction can add bias and sampling noise. The following decomposition includes their dependence on the coarse estimate and provides the objective for choosing the mixing weight. Here g ^ c \widehat{g}_c g c denotes the finite-radius coarse estimator, with trace variance V c V_c V c , smoothing bias b c , ε b_{c,\varepsilon} b c , ε , and centered fluctuation ξ c = g ^ c − E g ^ c \xi_c=\widehat{g}_c-\mathbb{E}\widehat{g}_c ξ c = g c − E g c .
For a general correction τ ^ = τ + b τ + ξ τ \widehat{\tau}=\tau+b_\tau+\xi_\tau τ = τ + b τ + ξ τ , define b τ = E τ ^ − τ b_\tau=\mathbb{E}\widehat{\tau}-\tau b τ = E τ − τ and ξ τ = τ ^ − E τ ^ \xi_\tau=\widehat{\tau}-\mathbb{E}\widehat{\tau} ξ τ = τ − E τ , so E ξ τ = 0 \mathbb{E}\xi_\tau=0 E ξ τ = 0 . Let V τ = E ∥ ξ τ ∥ 2 V_\tau=\mathbb{E}\lVert\xi_\tau\rVert^2 V τ = E ∥ ξ τ ∥ 2 and C c , τ = E ⟨ ξ c , ξ τ ⟩ C_{c,\tau}=\mathbb{E}\langle\xi_c,\xi_\tau\rangle C c , τ = E ⟨ ξ c , ξ τ ⟩ .
Proposition C.5 (MSE with an approximate correction). For fixed λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] and square-integrable estimates, the MSE is:
E ( λ ) : = E ∥ g ^ c + λ τ ^ − g ⋆ ∥ 2 = V c + λ 2 V τ + 2 λ C c , τ + ∥ b c , ε − ( 1 − λ ) τ + λ b τ ∥ 2 . (23) \mathcal{E}(\lambda):=\mathbb{E}\lVert\widehat{g}_c+\lambda\widehat{\tau}-g_\star\rVert^2
=V_c+\lambda^2V_\tau+2\lambda C_{c,\tau}
+\lVert b_{c,\varepsilon}-(1-\lambda)\tau+\lambda b_\tau\rVert^2. \tag*{(23)} E ( λ ) := E ∥ g c + λ τ − g ⋆ ∥ 2 = V c + λ 2 V τ + 2 λ C c , τ + ∥ b c , ε − ( 1 − λ ) τ + λ b τ ∥ 2 . (23)
Proof. The error splits into a centered fluctuation and a deterministic bias:
g ^ c + λ τ ^ − g ⋆ = ξ c + λ ξ τ + b c , ε − ( 1 − λ ) τ + λ b τ . \widehat{g}_c+\lambda\widehat{\tau}-g_\star
=\xi_c+\lambda\xi_\tau+b_{c,\varepsilon}-(1-\lambda)\tau+\lambda b_\tau. g c + λ τ − g ⋆ = ξ c + λ ξ τ + b c , ε − ( 1 − λ ) τ + λ b τ .
The first two terms sum to a zero-mean vector. Their squared norm has expectation:
E ∥ ξ c + λ ξ τ ∥ 2 = E ∥ ξ c ∥ 2 + λ 2 E ∥ ξ τ ∥ 2 + 2 λ E ⟨ ξ c , ξ τ ⟩ = V c + λ 2 V τ + 2 λ C c , τ . \mathbb{E}\lVert\xi_c+\lambda\xi_\tau\rVert^2
=\mathbb{E}\lVert\xi_c\rVert^2+\lambda^2\mathbb{E}\lVert\xi_\tau\rVert^2+2\lambda\mathbb{E}\langle\xi_c,\xi_\tau\rangle
=V_c+\lambda^2V_\tau+2\lambda C_{c,\tau}. E ∥ ξ c + λ ξ τ ∥ 2 = E ∥ ξ c ∥ 2 + λ 2 E ∥ ξ τ ∥ 2 + 2 λ E ⟨ ξ c , ξ τ ⟩ = V c + λ 2 V τ + 2 λ C c , τ .
The expected cross term vanishes, so adding the squared bias yields Eq. (23) .
:::
The covariance term C c , τ C_{c,\tau} C c , τ accounts for dependence between the two estimates. For history-measurable weights, the formula applies conditional on that history. For weights computed from the same samples, the analysis applies directly to the joint quantity λ τ ^ \lambda\hat{\tau} λ τ ^ .
Local-linear limit and exact comparisons
All estimators and variances in this subsection refer to the limit ε → 0 \varepsilon\to0 ε → 0 with b b b fixed.
Proof of Proposition 3.3. Apply Theorem C.3 with φ = F \varphi= F φ = F and Σ = Σ c \Sigma= \Sigma_c Σ = Σ c to obtain mean-square convergence and b c , ε → 0 b_{c,\varepsilon} \to0 b c , ε → 0 . Lemma C.2 gives the limiting mean g c g_c g c and trace variance V c , 0 V_{c,0} V c , 0 in Eq. (11) . Setting b c , ε = 0 b_{c,\varepsilon}=0 b c , ε = 0 in Proposition C.4 gives:
E ∥ d ^ λ − g ⋆ ∥ 2 = V c + ( 1 − λ ) 2 ∥ τ ∥ 2 . \mathbb{E}\lVert\hat{d}_{\lambda}-g_{\star}\rVert^{2}=V_c+(1-\lambda)^2\lVert\tau\rVert^2. E ∥ d ^ λ − g ⋆ ∥ 2 = V c + ( 1 − λ ) 2 ∥ τ ∥ 2 .
Here V c V_c V c denotes V c , 0 V_{c,0} V c , 0 in the main text. Setting λ = 0 \lambda=0 λ = 0 gives Eq. (12) , and λ = 1 \lambda=1 λ = 1 gives Eq. (13) .
More generally, the MSE reduction is λ ( 2 − λ ) ∥ τ ∥ 2 \lambda(2-\lambda)\lVert\tau\rVert^2 λ ( 2 − λ ) ∥ τ ∥ 2 , which is positive for 0 < λ ≤ 1 0<\lambda\leq1 0 < λ ≤ 1 and τ ≠ 0 \tau\ne0 τ = 0 . For approximate corrections with mean-square limits, substituting b c , ε = 0 b_{c,\varepsilon}=0 b c , ε = 0 and the limiting correction moments into Proposition C.5 gives the corresponding local result.
Dimension dependence of sampling variance. For fixed b b b and comparable gradient and covariance scales bounded independently of dimension, the local-linear variance bounds give V f a n i = O ( d f ) V_f^{\mathrm{ani}}=O(d_f) V f ani = O ( d f ) and V c a n i = O ( d c ) V_c^{\mathrm{ani}}=O(d_c) V c ani = O ( d c ) . Here V f a n i V_f^{\mathrm{ani}} V f ani is the variance measured in full parameter coordinates. The linear dimension factor explains the sampling advantage of d c ≪ d f d_c\ll d_f d c ≪ d f . We next restrict the full-space output to compare all estimators in the same coarse coordinates.
Bias and variance at a common target. To compare full-space LOREN with coarse estimators, first restrict its output to the same coordinates using R g ^ f a n i R\hat{g}_f^{\mathrm{ani}} R g ^ f ani . Define δ i s o = g ⋆ − g c r a w \delta_{\mathrm{iso}}=g_{\star}-g_c^{\mathrm{raw}} δ iso = g ⋆ − g c raw . Let V c i s o V_c^{\mathrm{iso}} V c iso and V c a n i V_c^{\mathrm{ani}} V c ani denote the trace variances of g ^ c , 0 , b i s o \hat{g}_{c,0,b}^{\mathrm{iso}} g ^ c , 0 , b iso and g ^ c , 0 , b a n i \hat{g}_{c,0,b}^{\mathrm{ani}} g ^ c , 0 , b ani , respectively, and let V ⋆ d i r e c t V_{\star}^{\mathrm{direct}} V ⋆ direct denote that of R g ^ f , 0 , b a n i R\hat{g}_{f,0,b}^{\mathrm{ani}} R g ^ f , 0 , b ani . Substituting the coarse gradients and covariances into Eq. (18) gives the first two expressions below. For the third, apply the same identity with the fine-space quantities and use Cov ( R X ) = R Cov ( X ) R ⊤ \operatorname{Cov}(RX)=R\operatorname{Cov}(X)R^{\top} Cov ( R X ) = R Cov ( X ) R ⊤ . Their exact expressions are:
V c i s o = ( d c + 1 ) ∥ g c r a w ∥ 2 b , (24) V_c^{\mathrm{iso}}=\frac{(d_c+1)\lVert g_c^{\mathrm{raw}}\rVert^2}{b}, \tag*{(24)} V c iso = b ( d c + 1 ) ∥ g c raw ∥ 2 , (24)
V c a n i = tr ( Σ c ) ( g c r a w ) ⊤ Σ c g c r a w + ( g c r a w ) ⊤ Σ c 2 g c r a w b , (25) V_c^{\mathrm{ani}}=\frac{\operatorname{tr}(\Sigma_c)(g_c^{\mathrm{raw}})^{\top}\Sigma_cg_c^{\mathrm{raw}}+(g_c^{\mathrm{raw}})^{\top}\Sigma_c^2g_c^{\mathrm{raw}}}{b}, \tag*{(25)} V c ani = b tr ( Σ c ) ( g c raw ) ⊤ Σ c g c raw + ( g c raw ) ⊤ Σ c 2 g c raw , (25)
V ⋆ d i r e c t = ( ( g f r a w ) ⊤ Σ f g f r a w ) tr ( R Σ f R ⊤ ) + ∥ g ⋆ ∥ 2 b . (26) V_{\star}^{\mathrm{direct}}=\frac{((g_f^{\mathrm{raw}})^{\top}\Sigma_fg_f^{\mathrm{raw}})\operatorname{tr}(R\Sigma_fR^{\top})+\lVert g_{\star}\rVert^2}{b}. \tag*{(26)} V ⋆ direct = b (( g f raw ) ⊤ Σ f g f raw ) tr ( R Σ f R ⊤ ) + ∥ g ⋆ ∥ 2 . (26)
Here R Σ f g f r a w = g ⋆ R\Sigma_fg_f^{\mathrm{raw}}=g_{\star} R Σ f g f raw = g ⋆ , and taking the trace of the restricted covariance gives V ⋆ d i r e c t V_{\star}^{\mathrm{direct}} V ⋆ direct .
Table 4 : Exact comparison in the local-linear limit ε → 0 \varepsilon\to0 ε → 0 , with b b b held fixed, against the same target g ⋆ g_{\star} g ⋆ , conditional on the current state. Bias is mean minus target. All methods use b b b independent directions. ENGINE rows use exact, conditionally deterministic correction.
Table 4 follows by inserting the respective means g c r a w g_c^{\mathrm{raw}} g c raw , g c g_c g c , g ⋆ g_{\star} g ⋆ , and g c + λ τ g_c+\lambda\tau g c + λ τ into (20). Ideal ENGINE with λ = 1 \lambda=1 λ = 1 has smaller MSE than restricted full-space LOREN precisely when V c a n i < V ⋆ d i r e c t V_c^{\mathrm{ani}}<V_{\star}^{\mathrm{direct}} V c ani < V ⋆ direct , and smaller MSE than isotropic LoRA precisely when V c a n i < V c i s o + ∥ δ i s o ∥ 2 V_c^{\mathrm{ani}}<V_c^{\mathrm{iso}}+\lVert\delta_{\mathrm{iso}}\rVert^2 V c ani < V c iso + ∥ δ iso ∥ 2 . These conditions account for gradients, covariances, and restriction in each comparison.
Combined benefit of ideal ENGINE. The two benefits act on separate terms of the MSE decomposition. Coarse sampling reduces the dimension-dependent variance under the stated scale assumptions. Exact correction recovers g ⋆ g_{\star} g ⋆ and removes the structural bias, reducing V c a n i + ∥ τ ∥ 2 V_{c}^{\mathrm{ani}} + \lVert\tau\rVert^{2} V c ani + ∥ τ ∥ 2 to V c a n i V_{c}^{\mathrm{ani}} V c ani .
Scalar approximation error
The exact branch residual has the form τ ℓ = ρ − 1 ( c ℓ q ℓ T − c f , ℓ s ℓ T ) \tau_{\ell} = \rho^{-1}(c_{\ell}q_{\ell}^{\mathsf{T}} - c_{f,\ell}s_{\ell}^{\mathsf{T}}) τ ℓ = ρ − 1 ( c ℓ q ℓ T − c f , ℓ s ℓ T ) , where p B = m p_{B}=m p B = m , p A = r p_{A}=r p A = r , c ℓ , c f , ℓ ∈ R p ℓ c_{\ell},c_{f,\ell} \in\mathbb{R}^{p_{\ell}} c ℓ , c f , ℓ ∈ R p ℓ , and ρ > 0 \rho> 0 ρ > 0 is the same damping parameter as in the rank-1 preconditioner. Here v B v_{B} v B , v A v_{A} v A , and v f v_{f} v f denote the method’s curvature vectors v B v_{B} v B , v A v_{A} v A , and v f v_{f} v f , respectively. The branch residual τ ℓ \tau_{\ell} τ ℓ is the matrix form of τ s g s , ℓ \tau_{\mathrm{sgs},\ell} τ sgs , ℓ , for ℓ ∈ { B , A } \ell\in\{B,A\} ℓ ∈ { B , A } . Specifically:
q B = v B , s B = A v f , q A = v A , s A = v f , q_{B}=v_{B},\quad s_{B}=Av_{f},\quad q_{A}=v_{A},\quad s_{A}=v_{f}, q B = v B , s B = A v f , q A = v A , s A = v f ,
c B = G A T v B ρ + ∥ v B ∥ 2 , c f , B = G v f ρ + ∥ v f ∥ 2 , c_{B}=\frac{GA^{\mathsf{T}}v_{B}}{\rho+\lVert v_{B}\rVert^{2}},\quad c_{f,B}=\frac{Gv_{f}}{\rho+\lVert v_{f}\rVert^{2}}, c B = ρ + ∥ v B ∥ 2 G A T v B , c f , B = ρ + ∥ v f ∥ 2 G v f ,
c A = B T G v A ρ + ∥ v A ∥ 2 , c f , A = B T G v f ρ + ∥ v f ∥ 2 . c_{A}=\frac{B^{\mathsf{T}}Gv_{A}}{\rho+\lVert v_{A}\rVert^{2}},\quad c_{f,A}=\frac{B^{\mathsf{T}}Gv_{f}}{\rho+\lVert v_{f}\rVert^{2}}. c A = ρ + ∥ v A ∥ 2 B T G v A , c f , A = ρ + ∥ v f ∥ 2 B T G v f .
These are the exact coefficients of Proposition 3.2. Their row averages define the scalar-broadcast approximation studied below. Let 1 p \mathbf{1}_{p} 1 p be the all-ones vector, Π p = 1 p 1 p T p \Pi_{p} = \frac{\mathbf{1}_{p}\mathbf{1}_{p}^{\mathsf{T}}}{p} Π p = p 1 p 1 p T and P p = I p − Π p P_{p}=I_{p}-\Pi_{p} P p = I p − Π p . For p = p ℓ p=p_{\ell} p = p ℓ , define the arithmetic coefficient means c ˉ ℓ = 1 p T c ℓ p \bar{c}_{\ell}=\frac{\mathbf{1}_{p}^{\mathsf{T}}c_{\ell}}{p} c ˉ ℓ = p 1 p T c ℓ and c ˉ f , ℓ = 1 p T c f , ℓ p \bar{c}_{f,\ell}=\frac{\mathbf{1}_{p}^{\mathsf{T}}c_{f,\ell}}{p} c ˉ f , ℓ = p 1 p T c f , ℓ .
The next proposition separates row compression error from scalar coefficient mismatch.
Proposition C.6 (Row compression and scalar approximation error). For p = p ℓ p=p_{\ell} p = p ℓ , the row mean τ ℓ m e a n = Π p τ ℓ \tau_{\ell}^{\mathrm{mean}}=\Pi_{p}\tau_{\ell} τ ℓ mean = Π p τ ℓ is the unique closest matrix to τ ℓ \tau_{\ell} τ ℓ among matrices with identical rows, in the Frobenius norm. Let D ℓ = ∥ P p τ ℓ ∥ F D_{\ell}=\lVert P_{p}\tau_{\ell}\rVert_{F} D ℓ = ∥ P p τ ℓ ∥ F . For any matrix Z Z Z with identical rows, orthogonality gives:
∥ τ ℓ − Z ∥ F 2 = D ℓ 2 + ∥ τ ℓ m e a n − Z ∥ F 2 . (27) \lVert\tau_{\ell}-Z\rVert_{F}^{2}=D_{\ell}^{2}+\lVert\tau_{\ell}^{\mathrm{mean}}-Z\rVert_{F}^{2}. \tag*{(27)} ∥ τ ℓ − Z ∥ F 2 = D ℓ 2 + ∥ τ ℓ mean − Z ∥ F 2 . (27) In particular, for scalar coefficients c ~ ℓ , c ~ f , ℓ \widetilde{c}_{\ell},\widetilde{c}_{f,\ell} c ℓ , c f , ℓ , set τ ~ ℓ = 1 ρ 1 p ( c ~ ℓ q ℓ T − c ~ f , ℓ s ℓ T ) \widetilde{\tau}_{\ell}=\frac{1}{\rho}\mathbf{1}_{p}(\widetilde{c}_{\ell}q_{\ell}^{\mathsf{T}}-\widetilde{c}_{f,\ell}s_{\ell}^{\mathsf{T}}) τ ℓ = ρ 1 1 p ( c ℓ q ℓ T − c f , ℓ s ℓ T ) . Then we have:
∥ τ ℓ − τ ~ ℓ ∥ F 2 = D ℓ 2 + p ρ 2 ∥ ( c ~ ℓ − c ˉ ℓ ) q ℓ − ( c ~ f , ℓ − c ˉ f , ℓ ) s ℓ ∥ 2 . (28) \lVert\tau_{\ell}-\widetilde{\tau}_{\ell}\rVert_{F}^{2}=D_{\ell}^{2}+\frac{p}{\rho^{2}}\left\lVert(\widetilde{c}_{\ell}-\bar{c}_{\ell})q_{\ell}-(\widetilde{c}_{f,\ell}-\bar{c}_{f,\ell})s_{\ell}\right\rVert^{2}. \tag*{(28)} ∥ τ ℓ − τ ℓ ∥ F 2 = D ℓ 2 + ρ 2 p ∥ ( c ℓ − c ˉ ℓ ) q ℓ − ( c f , ℓ − c ˉ f , ℓ ) s ℓ ∥ 2 . (28)
Proof. Row averaging gives τ ℓ m e a n = 1 ρ 1 p ( c ˉ ℓ q ℓ T − c ˉ f , ℓ s ℓ T ) \tau_{\ell}^{\mathrm{mean}}=\frac{1}{\rho}\mathbf{1}_{p}(\bar{c}_{\ell}q_{\ell}^{\mathsf{T}}-\bar{c}_{f,\ell}s_{\ell}^{\mathsf{T}}) τ ℓ mean = ρ 1 1 p ( c ˉ ℓ q ℓ T − c ˉ f , ℓ s ℓ T ) . The rows of P p τ ℓ P_{p}\tau_{\ell} P p τ ℓ sum to zero, so this residual is orthogonal to every matrix with identical rows. Therefore, we have:
∥ τ ℓ − Z ∥ F 2 = ∥ P p τ ℓ + ( τ ℓ m e a n − Z ) ∥ F 2 = ∥ P p τ ℓ ∥ F 2 + ∥ τ ℓ m e a n − Z ∥ F 2 . \begin{aligned}
\lVert\tau_{\ell}-Z\rVert_{F}^{2}=\lVert P_{p}\tau_{\ell}+(\tau_{\ell}^{\mathrm{mean}}-Z)\rVert_{F}^{2} \\
&=\lVert P_{p}\tau_{\ell}\rVert_{F}^{2}+\lVert\tau_{\ell}^{\mathrm{mean}}-Z\rVert_{F}^{2}.
\end{aligned} ∥ τ ℓ − Z ∥ F 2 = ∥ P p τ ℓ + ( τ ℓ mean − Z ) ∥ F 2 = ∥ P p τ ℓ ∥ F 2 + ∥ τ ℓ mean − Z ∥ F 2 .
The second term vanishes exactly at Z = τ ℓ m e a n Z=\tau_{\ell}^{\mathrm{mean}} Z = τ ℓ mean , proving optimality and uniqueness. For Z = τ ~ ℓ Z=\widetilde{\tau}_{\ell} Z = τ ℓ , its value is:
∥ τ ~ ℓ − τ ℓ m e a n ∥ F 2 = p ρ 2 ∥ ( c ~ ℓ − c ˉ ℓ ) q ℓ − ( c ~ f , ℓ − c ˉ f , ℓ ) s ℓ ∥ 2 , \lVert\widetilde{\tau}_{\ell}-\tau_{\ell}^{\mathrm{mean}}\rVert_{F}^{2}=\frac{p}{\rho^{2}}\left\lVert(\widetilde{c}_{\ell}-\bar{c}_{\ell})q_{\ell}-(\widetilde{c}_{f,\ell}-\bar{c}_{f,\ell})s_{\ell}\right\rVert^{2}, ∥ τ ℓ − τ ℓ mean ∥ F 2 = ρ 2 p ∥ ( c ℓ − c ˉ ℓ ) q ℓ − ( c f , ℓ − c ˉ f , ℓ ) s ℓ ∥ 2 ,
since ∥ 1 p ∥ 2 = p \lVert\mathbf{1}_{p}\rVert^{2}=p ∥ 1 p ∥ 2 = p . Substitution proves Eq. (28) .
The two branches have minimum total compression error D B 2 + D A 2 D_{B}^{2}+D_{A}^{2} D B 2 + D A 2 . For general scalar coefficients, the total squared correction error is the sum of both right-hand sides of Eq. (28) . Appendix D.1 specifies how the implementation’s normalization and scaling determine these coefficients.
Oracle-optimal mixing
The MSE is quadratic in λ \lambda λ . Define τ ˉ = E τ ^ = τ + b τ \bar{\tau}=\mathbb{E}\hat{\tau}=\tau+b_{\tau} τ ˉ = E τ ^ = τ + b τ .
Proposition C.7 (Oracle-optimal mixing weight). If ∥ τ ˉ ∥ 2 + V τ > 0 \lVert\bar{\tau}\rVert^{2}+V_{\tau}>0 ∥ τ ˉ ∥ 2 + V τ > 0 , minimizing (23) over fixed λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] gives:
λ ⋆ = clip [ 0 , 1 ] ( ⟨ τ − b c , ε , τ ˉ ⟩ − C c , τ ∥ τ ˉ ∥ 2 + V τ ) . (29) \lambda^{\star}=\operatorname{clip}_{[0,1]}\left(\frac{\langle\tau-b_{c,\varepsilon},\bar{\tau}\rangle-C_{c,\tau}}{\lVert\bar{\tau}\rVert^{2}+V_{\tau}}\right). \tag*{(29)} λ ⋆ = clip [ 0 , 1 ] ( ∥ τ ˉ ∥ 2 + V τ ⟨ τ − b c , ε , τ ˉ ⟩ − C c , τ ) . (29) If the denominator vanishes, τ ^ = 0 \hat{\tau}=0 τ ^ = 0 almost surely and every feasible weight is optimal.
Proof. Set a τ = ∥ τ ‾ ∥ 2 + V τ a_{\tau} = \lVert\overline{\tau} \rVert^{2} + V_{\tau} a τ = ∥ τ ∥ 2 + V τ and q τ = ⟨ τ − b c , ε , τ ‾ ⟩ − C c , τ q_{\tau} = \langle\tau- b_{c,\varepsilon}, \overline{\tau} \rangle- C_{c,\tau} q τ = ⟨ τ − b c , ε , τ ⟩ − C c , τ . Expanding the squared bias in Eq. (23) gives:
E ( λ ) = E ( 0 ) + a τ λ 2 − 2 q τ λ . \mathcal{E}(\lambda) = \mathcal{E}(0) + a_{\tau}\lambda^{2} - 2q_{\tau}\lambda. E ( λ ) = E ( 0 ) + a τ λ 2 − 2 q τ λ .
For a τ > 0 a_{\tau} > 0 a τ > 0 , this strictly convex quadratic is minimized at q τ / a τ q_{\tau}/a_{\tau} q τ / a τ on R \mathbb{R} R , and clipping gives the minimum on [ 0 , 1 ] [0,1] [ 0 , 1 ] . The same argument applies to any closed subinterval [ λ min , λ max ] [\lambda_{\min}, \lambda_{\max}] [ λ m i n , λ m a x ] . If a τ = E ∥ τ ‾ ∥ 2 = 0 a_{\tau} = \mathbb{E}\lVert\overline{\tau}\rVert^{2} = 0 a τ = E ∥ τ ∥ 2 = 0 , then τ ‾ = 0 \overline{\tau} = 0 τ = 0 almost surely and E \mathcal{E} E is independent of λ \lambda λ .
For unbiased estimators with uncorrelated noise, set b c , ε = 0 b_{c,\varepsilon} = 0 b c , ε = 0 , b τ = 0 b_{\tau} = 0 b τ = 0 , and C c , τ = 0 C_{c,\tau} = 0 C c , τ = 0 . When ∥ τ ∥ 2 + V τ > 0 \lVert\tau\rVert^{2} + V_{\tau} > 0 ∥ τ ∥ 2 + V τ > 0 , substituting these conditions into Proposition C.7 and Eq. (23) gives:
λ ∗ = ∥ τ ∥ 2 ∥ τ ∥ 2 + V τ , E ( λ ∗ ) = V c + ∥ τ ∥ 2 V τ ∥ τ ∥ 2 + V τ . \lambda^{*} = \frac{\lVert\tau\rVert^{2}}{\lVert\tau\rVert^{2} + V_{\tau}}, \qquad\mathcal{E}(\lambda^{*}) = V_{c} + \frac{\lVert\tau\rVert^{2}V_{\tau}}{\lVert\tau\rVert^{2} + V_{\tau}}. λ ∗ = ∥ τ ∥ 2 + V τ ∥ τ ∥ 2 , E ( λ ∗ ) = V c + ∥ τ ∥ 2 + V τ ∥ τ ∥ 2 V τ .
For fixed positive correction energy, this optimal weight decreases as the correction variance grows.
Convergence under positive alignment
We analyze updates on the fixed coarse objective F ( ϕ ) = f ( w ( ϕ ) ) F(\phi) = f(w(\phi)) F ( ϕ ) = f ( w ( ϕ )) with W 0 W_{0} W 0 held fixed. Stationarity is measured by the gradient with respect to the stacked LoRA factors, g c , t r a w = ∇ ϕ F ( ϕ t ) g^{\mathrm{raw}}_{c,t} = \nabla_{\phi}F(\phi_{t}) g c , t raw = ∇ ϕ F ( ϕ t ) . Let F t \mathcal{F}_{t} F t be the history before fresh step-t t t perturbations, and E t = E [ ⋅ ∣ F t ] \mathbb{E}_{t} = \mathbb{E}[\cdot\mid\mathcal{F}_{t}] E t = E [ ⋅ ∣ F t ] . All current parameters, preconditioners, and exact targets are F t \mathcal{F}_{t} F t -measurable.
Assumptions for ENGINE convergence. The following assumptions hold uniformly over the coarse steps under analysis.
(A1) Smoothness and lower boundedness. The coarse objective F F F is L c L_{c} L c -smooth with L c > 0 L_{c} > 0 L c > 0 and F ( ϕ ) ≥ F inf F(\phi) \ge F_{\inf} F ( ϕ ) ≥ F i n f .
(A2) Bounded positive-definite preconditioning. The coarse preconditioner satisfies m c I ⪯ Σ c , t ⪯ M c I m_{c}I \preceq\Sigma_{c,t} \preceq M_{c}I m c I ⪯ Σ c , t ⪯ M c I for constants 0 < m c ≤ M c 0 < m_{c} \le M_{c} 0 < m c ≤ M c .
(A3) Target alignment and norm control. The restricted full-space target satisfies:
⟨ g c , t r a w , g ⋆ , t ⟩ ≥ κ ⋆ ∥ g c , t r a w ∥ 2 , ∥ g ⋆ , t ∥ ≤ M ⋆ ∥ g c , t r a w ∥ , κ ⋆ > 0. (30) \langle g^{\mathrm{raw}}_{c,t}, g_{\star,t}\rangle\ge\kappa_{\star}\lVert g^{\mathrm{raw}}_{c,t}\rVert^{2}, \qquad\lVert g_{\star,t}\rVert\le M_{\star}\lVert g^{\mathrm{raw}}_{c,t}\rVert, \qquad\kappa_{\star} > 0. \tag*{(30)} ⟨ g c , t raw , g ⋆ , t ⟩ ≥ κ ⋆ ∥ g c , t raw ∥ 2 , ∥ g ⋆ , t ∥ ≤ M ⋆ ∥ g c , t raw ∥ , κ ⋆ > 0. (30)
(A4) Predictable mixing. The weight λ t ∈ [ 0 , 1 ] \lambda_{t} \in[0,1] λ t ∈ [ 0 , 1 ] is F t \mathcal{F}_{t} F t -measurable, so it is fixed before the fresh step-t t t perturbations are sampled.
(A5) Controlled estimator bias and variance. For nonnegative constants β c \beta_{c} β c , ν c \nu_{c} ν c , ω c \omega_{c} ω c , the coarse estimator satisfies:
∥ E t g ^ c , t − g c , t ∥ ≤ β c , Var t ( g ^ c , t ) ≤ ν c ∥ g c , t r a w ∥ 2 + ω c . \lVert\mathbb{E}_{t}\widehat{g}_{c,t} - g_{c,t}\rVert\le\beta_{c}, \qquad\operatorname{Var}_{t}(\widehat{g}_{c,t}) \le\nu_{c}\lVert g^{\mathrm{raw}}_{c,t}\rVert^{2} + \omega_{c}. ∥ E t g c , t − g c , t ∥ ≤ β c , Var t ( g c , t ) ≤ ν c ∥ g c , t raw ∥ 2 + ω c .
Empirical alignment proxy. In our diagnostic runs, the cosine between the raw gradient estimate and the deployed pre-momentum direction after correction and clipping was positive at every measured coarse step. This measures alignment with the estimated gradient, while Assumption A3 concerns uniform alignment and norm control of the exact target.
The following descent lemma provides a common bound for exact and approximate correction.
Lemma C.8 (Conditional descent with a biased stochastic direction). Suppose F F F is L c L_{c} L c -smooth, L c > 0 L_{c} > 0 L c > 0 , and ϕ t + 1 = ϕ t − η d ^ t \phi_{t+1} = \phi_{t} - \eta\widehat{d}_{t} ϕ t + 1 = ϕ t − η d t . Let an F t \mathcal{F}_{t} F t -measurable reference direction d t d_{t} d t satisfy:
⟨ g c , t r a w , d t ⟩ ≥ κ ∥ g c , t r a w ∥ 2 , \langle g^{\mathrm{raw}}_{c,t}, d_{t}\rangle\ge\kappa\lVert g^{\mathrm{raw}}_{c,t}\rVert^{2}, ⟨ g c , t raw , d t ⟩ ≥ κ ∥ g c , t raw ∥ 2 , ∥ d t ∥ ≤ M ∥ g c , t r a w ∥ , κ > 0. \lVert d_{t}\rVert\le M\lVert g^{\mathrm{raw}}_{c,t}\rVert, \qquad\kappa> 0. ∥ d t ∥ ≤ M ∥ g c , t raw ∥ , κ > 0. Assume ∥ E t d ^ t − d t ∥ ≤ β \lVert\mathbb{E}_{t}\widehat{d}_{t} - d_{t}\rVert\le\beta ∥ E t d t − d t ∥ ≤ β and Var t ( d ^ t ) ≤ ν ∥ g c , t r a w ∥ 2 + ω \operatorname{Var}_{t}(\widehat{d}_{t}) \le\nu\lVert g^{\mathrm{raw}}_{c,t}\rVert^{2} + \omega Var t ( d t ) ≤ ν ∥ g c , t raw ∥ 2 + ω , where β , ν , ω \beta,\nu,\omega β , ν , ω are nonnegative constants. Set A = 2 M 2 + ν > 0 \mathcal{A} = 2M^{2} + \nu> 0 A = 2 M 2 + ν > 0 and B = 2 β 2 + ω \mathcal{B} = 2\beta^{2} + \omega B = 2 β 2 + ω . For 0 < η ≤ κ 2 L c A 0 < \eta\le\frac{\kappa}{2L_{c}\mathcal{A}} 0 < η ≤ 2 L c A κ , the one-step bound is:
E t F ( ϕ t + 1 ) ≤ F ( ϕ t ) − η κ 4 ∥ g c , t r a w ∥ 2 + η β 2 2 κ + L c η 2 B 2 . (31) \mathbb{E}_{t}F(\phi_{t+1}) \le F(\phi_{t}) - \frac{\eta\kappa}{4}\lVert g^{\mathrm{raw}}_{c,t}\rVert^{2} + \frac{\eta\beta^{2}}{2\kappa} + \frac{L_{c}\eta^{2}\mathcal{B}}{2}. \tag*{(31)} E t F ( ϕ t + 1 ) ≤ F ( ϕ t ) − 4 η κ ∥ g c , t raw ∥ 2 + 2 κ η β 2 + 2 L c η 2 B . (31)
Proof. Step 1. Lower-bound the mean descent term. Let e t = E t d ^ t − d t e_t = \mathbb{E}_t\hat{d}_t - d_t e t = E t d ^ t − d t . The Cauchy-Schwarz inequality and ∥ e t ∥ ≤ β \lVert e_t\rVert\le\beta ∥ e t ∥ ≤ β imply:
⟨ g c , t r a w , E t d ^ t ⟩ = ⟨ g c , t r a w , d t ⟩ + ⟨ g c , t r a w , e t ⟩ ≥ κ ∥ g c , t r a w ∥ 2 − β ∥ g c , t r a w ∥ . \langle g_{c,t}^{\mathrm{raw}},\mathbb{E}_t\hat{d}_t\rangle= \langle g_{c,t}^{\mathrm{raw}},d_t\rangle+ \langle g_{c,t}^{\mathrm{raw}},e_t\rangle
\ge\kappa\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 - \beta\lVert g_{c,t}^{\mathrm{raw}}\rVert. ⟨ g c , t raw , E t d ^ t ⟩ = ⟨ g c , t raw , d t ⟩ + ⟨ g c , t raw , e t ⟩ ≥ κ ∥ g c , t raw ∥ 2 − β ∥ g c , t raw ∥ .
Young’s inequality in the form β ∥ g c , t r a w ∥ ≤ κ ∥ g c , t r a w ∥ 2 2 + β 2 2 κ \beta\lVert g_{c,t}^{\mathrm{raw}}\rVert\le\frac{\kappa\lVert g_{c,t}^{\mathrm{raw}}\rVert^2}{2} + \frac{\beta^2}{2\kappa} β ∥ g c , t raw ∥ ≤ 2 κ ∥ g c , t raw ∥ 2 + 2 κ β 2 gives:
⟨ g c , t r a w , E t d ^ t ⟩ ≥ κ 2 ∥ g c , t r a w ∥ 2 − β 2 2 κ . \langle g_{c,t}^{\mathrm{raw}},\mathbb{E}_t\hat{d}_t\rangle\ge\frac{\kappa}{2}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 - \frac{\beta^2}{2\kappa}. ⟨ g c , t raw , E t d ^ t ⟩ ≥ 2 κ ∥ g c , t raw ∥ 2 − 2 κ β 2 .
Step 2. Upper-bound the second moment. Separating conditional variance from the squared conditional mean yields:
E t ∥ d ^ t ∥ 2 = ∥ E t d ^ t ∥ 2 + Var t ( d ^ t ) = ∥ d t + e t ∥ 2 + Var t ( d ^ t ) \begin{aligned}
\mathbb{E}_t\lVert\hat{d}_t\rVert^2 = \lVert\mathbb{E}_t\hat{d}_t\rVert^2 + \operatorname{Var}_t(\hat{d}_t) \\
&= \lVert d_t + e_t\rVert^2 + \operatorname{Var}_t(\hat{d}_t)
\end{aligned} E t ∥ d ^ t ∥ 2 = ∥ E t d ^ t ∥ 2 + Var t ( d ^ t ) = ∥ d t + e t ∥ 2 + Var t ( d ^ t )
≤ 2 ∥ d t ∥ 2 + 2 ∥ e t ∥ 2 + ν ∥ g c , t r a w ∥ 2 + ω ≤ ( 2 M 2 + ν ) ∥ g c , t r a w ∥ 2 + 2 β 2 + ω = A ∥ g c , t r a w ∥ 2 + B . \begin{aligned}
\le2\lVert d_t\rVert^2 + 2\lVert e_t\rVert^2 + \nu\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 + \omega\\
&\le(2M^2+\nu)\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 + 2\beta^2+\omega\\
&= \mathcal{A}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 + \mathcal{B}.
\end{aligned} ≤ 2 ∥ d t ∥ 2 + 2 ∥ e t ∥ 2 + ν ∥ g c , t raw ∥ 2 + ω ≤ ( 2 M 2 + ν ) ∥ g c , t raw ∥ 2 + 2 β 2 + ω = A ∥ g c , t raw ∥ 2 + B .
Step 3. Apply smoothness and the step-size condition. The update and L c L_c L c -smoothness imply:
E t F ( ϕ t + 1 ) ≤ F ( ϕ t ) − η ⟨ g c , t r a w , E t d ^ t ⟩ + L c η 2 2 E t ∥ d ^ t ∥ 2 ≤ F ( ϕ t ) − ( η κ 2 − L c η 2 A 2 ) ∥ g c , t r a w ∥ 2 + η β 2 2 κ + L c η 2 B 2 . \begin{aligned}
\mathbb{E}_tF(\phi_{t+1}) \le F(\phi_t) - \eta\langle g_{c,t}^{\mathrm{raw}},\mathbb{E}_t\hat{d}_t\rangle+ \frac{L_c\eta^2}{2}\mathbb{E}_t\lVert\hat{d}_t\rVert^2 \\
&\le F(\phi_t) - \left(\frac{\eta\kappa}{2}-\frac{L_c\eta^2\mathcal{A}}{2}\right)\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 + \frac{\eta\beta^2}{2\kappa}+\frac{L_c\eta^2\mathcal{B}}{2}.
\end{aligned} E t F ( ϕ t + 1 ) ≤ F ( ϕ t ) − η ⟨ g c , t raw , E t d ^ t ⟩ + 2 L c η 2 E t ∥ d ^ t ∥ 2 ≤ F ( ϕ t ) − ( 2 η κ − 2 L c η 2 A ) ∥ g c , t raw ∥ 2 + 2 κ η β 2 + 2 L c η 2 B .
Since L c η 2 A 2 ≤ η κ 4 \frac{L_c\eta^2\mathcal{A}}{2}\le\frac{\eta\kappa}{4} 2 L c η 2 A ≤ 4 η κ , the coefficient of the squared gradient is at least η κ 4 \frac{\eta\kappa}{4} 4 η κ . This proves (30).
For ideal ENGINE, use the exact correction d ^ t = g ^ c , t + λ t τ t \hat{d}_t = \hat{g}_{c,t}+\lambda_t\tau_t d ^ t = g ^ c , t + λ t τ t . Set κ = min { m c , κ ⋆ } \kappa=\min\{m_c,\kappa_\star\} κ = min { m c , κ ⋆ } , M = max { M c , M ⋆ } M=\max\{M_c,M_\star\} M = max { M c , M ⋆ } , A = 2 M 2 + ν c \mathcal{A}=2M^2+\nu_c A = 2 M 2 + ν c and B = 2 β c 2 + ω c \mathcal{B}=2\beta_c^2+\omega_c B = 2 β c 2 + ω c .
Let T ≥ 1 T\ge1 T ≥ 1 be the number of optimization steps. The next theorem bounds the average squared coarse gradient after these steps and identifies the residual error due to estimator bias.
Theorem C.9 (Ideal ENGINE stationarity). Under Assumptions A1 to A5, for ϕ t + 1 = ϕ t − η d ^ t \phi_{t+1}=\phi_t-\eta\hat{d}_t ϕ t + 1 = ϕ t − η d ^ t , η = κ 2 L c A T \eta=\frac{\kappa}{2L_c\mathcal{A}\sqrt{T}} η = 2 L c A T κ and T ≥ 1 T\ge1 T ≥ 1 , the iterates satisfy:
1 T ∑ t = 0 T − 1 E ∥ g c , t r a w ∥ 2 ≤ 8 L c A Δ F κ 2 T + B A T + 2 β c 2 κ 2 , Δ F = F ( ϕ 0 ) − F i n f . (32) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 \le\frac{8L_c\mathcal{A}\Delta_F}{\kappa^2\sqrt{T}}+\frac{\mathcal{B}}{\mathcal{A}\sqrt{T}}+\frac{2\beta_c^2}{\kappa^2}, \qquad\Delta_F=F(\phi_0)-F_{\mathrm{inf}}. \tag*{(32)} T 1 t = 0 ∑ T − 1 E ∥ g c , t raw ∥ 2 ≤ κ 2 T 8 L c A Δ F + A T B + κ 2 2 β c 2 , Δ F = F ( ϕ 0 ) − F inf . (32)
Proof. Step 1. Verify alignment and the norm bound for the mixed target. The mixed target satisfies:
d t = g c , t + λ t τ t = ( 1 − λ t ) g c , t + λ t g ⋆ , t . \begin{aligned}
d_t=g_{c,t}+\lambda_t\tau_t \\
&=(1-\lambda_t)g_{c,t}+\lambda_tg_{\star,t}.
\end{aligned} d t = g c , t + λ t τ t = ( 1 − λ t ) g c , t + λ t g ⋆ , t .
Since g c , t = Σ c , t g c , t r a w g_{c,t}=\Sigma_{c,t}g_{c,t}^{\mathrm{raw}} g c , t = Σ c , t g c , t raw , the spectral lower bound and (30) give:
⟨ g c , t r a w , d t ⟩ = ( 1 − λ t ) ( g c , t r a w ) ⊤ Σ c , t g c , t r a w + λ t ⟨ g c , t r a w , g ⋆ , t ⟩ ≥ [ ( 1 − λ t ) m c + λ t κ ⋆ ] ∥ g c , t r a w ∥ 2 ≥ κ ∥ g c , t r a w ∥ 2 . \begin{aligned}
\langle g_{c,t}^{\mathrm{raw}},d_t\rangle=(1-\lambda_t)(g_{c,t}^{\mathrm{raw}})^\top\Sigma_{c,t}g_{c,t}^{\mathrm{raw}}+\lambda_t\langle g_{c,t}^{\mathrm{raw}},g_{\star,t}\rangle\\
&\ge[(1-\lambda_t)m_c+\lambda_t\kappa_\star]\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 \\
&\ge\kappa\lVert g_{c,t}^{\mathrm{raw}}\rVert^2.
\end{aligned} ⟨ g c , t raw , d t ⟩ = ( 1 − λ t ) ( g c , t raw ) ⊤ Σ c , t g c , t raw + λ t ⟨ g c , t raw , g ⋆ , t ⟩ ≥ [( 1 − λ t ) m c + λ t κ ⋆ ] ∥ g c , t raw ∥ 2 ≥ κ ∥ g c , t raw ∥ 2 .
Similarly, the triangle inequality and the upper bounds yield:
∥ d t ∥ ≤ ( 1 − λ t ) ∥ Σ c , t g c , t r a w ∥ + λ t ∥ g ⋆ , t ∥ ≤ [ ( 1 − λ t ) M c + λ t M ⋆ ] ∥ g c , t r a w ∥ ≤ M ∥ g c , t r a w ∥ . \begin{aligned}
\lVert d_t\rVert\le(1-\lambda_t)\lVert\Sigma_{c,t}g_{c,t}^{\mathrm{raw}}\rVert+\lambda_t\lVert g_{\star,t}\rVert\\
&\le[(1-\lambda_t)M_c+\lambda_tM_\star]\lVert g_{c,t}^{\mathrm{raw}}\rVert\\
&\le M\lVert g_{c,t}^{\mathrm{raw}}\rVert.
\end{aligned} ∥ d t ∥ ≤ ( 1 − λ t ) ∥ Σ c , t g c , t raw ∥ + λ t ∥ g ⋆ , t ∥ ≤ [( 1 − λ t ) M c + λ t M ⋆ ] ∥ g c , t raw ∥ ≤ M ∥ g c , t raw ∥ .
Both comparisons use 0 ≤ λ t ≤ 1 0 \le\lambda_t \le1 0 ≤ λ t ≤ 1 .
Step 2. Apply the exact-correction result conditionally. Predictability makes λ t \lambda_t λ t fixed conditional on F t \mathcal{F}_t F t . The same holds for τ t \tau_t τ t , so Proposition C.4 gives:
E t d ^ t − d t = E t g ^ c , t − g c , t , \mathbb{E}_t\widehat{d}_t-d_t=\mathbb{E}_t\widehat{g}_{c,t}-g_{c,t}, E t d t − d t = E t g c , t − g c , t ,
Var t ( d ^ t ) = Var t ( g ^ c , t ) . \operatorname{Var}_t(\widehat{d}_t)=\operatorname{Var}_t(\widehat{g}_{c,t}). Var t ( d t ) = Var t ( g c , t ) .
Thus Lemma C.8 applies with β = β c \beta=\beta_c β = β c , ν = ν c \nu=\nu_c ν = ν c , and ω = ω c \omega=\omega_c ω = ω c . The chosen step size is admissible since T ≥ 1 T\ge1 T ≥ 1 .
Step 3. Sum the expected descent inequalities. Take unconditional expectations in (31) , use the tower property, and rearrange to get:
η κ 4 E ∥ g c , t r a w ∥ 2 ≤ E F ( ϕ t ) − E F ( ϕ t + 1 ) + η β c 2 2 κ + L c η 2 B 2 . \frac{\eta\kappa}{4}\mathbb{E}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2 \le\mathbb{E}F(\phi_t)-\mathbb{E}F(\phi_{t+1})+\frac{\eta\beta_c^2}{2\kappa}+\frac{L_c\eta^2\mathcal{B}}{2}. 4 η κ E ∥ g c , t raw ∥ 2 ≤ E F ( ϕ t ) − E F ( ϕ t + 1 ) + 2 κ η β c 2 + 2 L c η 2 B .
Summing from t = 0 t=0 t = 0 to T − 1 T-1 T − 1 telescopes the objective terms. For a fixed initial state and F ≥ F i n f F\ge F_{\mathrm{inf}} F ≥ F inf this gives:
η κ 4 ∑ t = 0 T − 1 E ∥ g c , t r a w ∥ 2 ≤ Δ F + T η β c 2 2 κ + T L c η 2 B 2 . \frac{\eta\kappa}{4}\sum_{t=0}^{T-1}\mathbb{E}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2\le\Delta_F+\frac{T\eta\beta_c^2}{2\kappa}+\frac{TL_c\eta^2\mathcal{B}}{2}. 4 η κ t = 0 ∑ T − 1 E ∥ g c , t raw ∥ 2 ≤ Δ F + 2 κ T η β c 2 + 2 T L c η 2 B .
Dividing by η κ T 4 \frac{\eta\kappa T}{4} 4 η κ T gives the general constant-step bound:
1 T ∑ t = 0 T − 1 E ∥ g c , t r a w ∥ 2 ≤ 4 Δ F η κ T + 2 β c 2 κ 2 + 2 L c η B κ . (33) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2\le\frac{4\Delta_F}{\eta\kappa T}+\frac{2\beta_c^2}{\kappa^2}+\frac{2L_c\eta\mathcal{B}}{\kappa}. \tag*{(33)} T 1 t = 0 ∑ T − 1 E ∥ g c , t raw ∥ 2 ≤ η κ T 4 Δ F + κ 2 2 β c 2 + κ 2 L c η B . (33)
Substituting η = κ 2 L c A T \eta=\frac{\kappa}{2L_c\mathcal{A}\sqrt{T}} η = 2 L c A T κ into Eq. (33) proves Eq. (32) . □
For Gaussian RGE with b c b_c b c independent coarse-space directions, Theorem C.3 permits:
β c 2 ≤ L c 2 ε 2 d c M c 3 , ν c = 2 ( d c + 1 ) M c 2 b c , ω c = L c 2 ε 2 M c 3 2 b c d c ( d c + 2 ) ( d c + 4 ) . \beta_c^2\le L_c^2\varepsilon^2d_cM_c^3,\qquad
\nu_c=\frac{2(d_c+1)M_c^2}{b_c},\qquad
\omega_c=\frac{L_c^2\varepsilon^2M_c^3}{2b_c}d_c(d_c+2)(d_c+4). β c 2 ≤ L c 2 ε 2 d c M c 3 , ν c = b c 2 ( d c + 1 ) M c 2 , ω c = 2 b c L c 2 ε 2 M c 3 d c ( d c + 2 ) ( d c + 4 ) .
Hence the ideal model has O ( T − 1 / 2 ) + O ( ε 2 ) O(T^{-1/2})+O(\varepsilon^2) O ( T − 1/2 ) + O ( ε 2 ) stationarity, with coarse-dimensional sampling constants. Here T T T is the number of optimization steps, and S S S is the curvature refresh interval used in the method.
We now extend the same descent argument to approximate corrections. Their mean error changes the residual term, while their variance changes the sampling constants in the stationarity bound.
Additional assumption for approximate correction.
(A6) Controlled correction error. The predictable weight satisfies 0 ≤ λ t ≤ λ max ≤ 1 0\le\lambda_t\le\lambda_{\max}\le1 0 ≤ λ t ≤ λ m a x ≤ 1 , and the correction estimate obeys, uniformly in t t t :
∥ E t τ ^ t − τ t ∥ ≤ β τ , Var t ( τ ^ t ) ≤ ν τ ∥ g c , t r a w ∥ 2 + ω τ \lVert\mathbb{E}_t\widehat{\tau}_t-\tau_t\rVert\le\beta_\tau,\qquad
\operatorname{Var}_t(\widehat{\tau}_t)\le\nu_\tau\lVert g_{c,t}^{\mathrm{raw}}\rVert^2+\omega_\tau ∥ E t τ t − τ t ∥ ≤ β τ , Var t ( τ t ) ≤ ν τ ∥ g c , t raw ∥ 2 + ω τ
for nonnegative constants β τ , ν τ , ω τ \beta_\tau,\nu_\tau,\omega_\tau β τ , ν τ , ω τ .
Corollary C.10 (Convergence with an approximate correction). Under Assumptions A1 to A6 , update with d ^ t = g ^ c , t + λ t τ ^ t \widehat{d}_t=\widehat{g}_{c,t}+\lambda_t\widehat{\tau}_t d t = g c , t + λ t τ t . Define:
β a p p = β c + λ max β τ , \beta_{\mathrm{app}}=\beta_c+\lambda_{\max}\beta_\tau, β app = β c + λ m a x β τ ,
ν a p p = 2 ν c + 2 λ max 2 ν τ , ω a p p = 2 ω c + 2 λ max 2 ω τ , \nu_{\mathrm{app}}=2\nu_c+2\lambda_{\max}^2\nu_\tau,\qquad
\omega_{\mathrm{app}}=2\omega_c+2\lambda_{\max}^2\omega_\tau, ν app = 2 ν c + 2 λ m a x 2 ν τ , ω app = 2 ω c + 2 λ m a x 2 ω τ ,
A a p p = 2 M 2 + ν a p p , B a p p = 2 β a p p 2 + ω a p p . \mathcal{A}_{\mathrm{app}}=2M^2+\nu_{\mathrm{app}},\qquad
\mathcal{B}_{\mathrm{app}}=2\beta_{\mathrm{app}}^2+\omega_{\mathrm{app}}. A app = 2 M 2 + ν app , B app = 2 β app 2 + ω app .
For η = κ 2 L c A a p p T \eta=\frac{\kappa}{2L_c\mathcal{A}_{\mathrm{app}}\sqrt{T}} η = 2 L c A app T κ , the bound is:
1 T ∑ t = 0 T − 1 E ∥ g c , t r a w ∥ 2 ≤ 8 L c A a p p Δ F κ 2 T + B a p p A a p p T + 2 β a p p 2 κ 2 . (34) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\lVert g_{c,t}^{\mathrm{raw}}\rVert^2\le
\frac{8L_c\mathcal{A}_{\mathrm{app}}\Delta_F}{\kappa^2\sqrt{T}}
+\frac{\mathcal{B}_{\mathrm{app}}}{\mathcal{A}_{\mathrm{app}}\sqrt{T}}
+\frac{2\beta_{\mathrm{app}}^2}{\kappa^2}. \tag*{(34)} T 1 t = 0 ∑ T − 1 E ∥ g c , t raw ∥ 2 ≤ κ 2 T 8 L c A app Δ F + A app T B app + κ 2 2 β app 2 . (34)
Proof. The reference direction remains d t = ( 1 − λ t ) g c , t + λ t g ⋆ , t \mathbf{d}_t=(1-\lambda_t)\mathbf{g}_{c,t}+\lambda_t\mathbf{g}_{\star,t} d t = ( 1 − λ t ) g c , t + λ t g ⋆ , t , so its alignment and norm bounds are unchanged. The conditional mean error is:
E t d ^ t − d t = ( E t g ^ c , t − g c , t ) + λ t ( E t τ ^ t − τ t ) . \mathbb{E}_t\widehat{\mathbf{d}}_t-\mathbf{d}_t=(\mathbb{E}_t\widehat{\mathbf{g}}_{c,t}-\mathbf{g}_{c,t})+\lambda_t(\mathbb{E}_t\widehat{\boldsymbol{\tau}}_t-\boldsymbol{\tau}_t). E t d t − d t = ( E t g c , t − g c , t ) + λ t ( E t τ t − τ t ) .
Its norm is at most β c + λ max β τ = β a p p \beta_c+\lambda_{\max}\beta_\tau=\beta_{\mathrm{app}} β c + λ m a x β τ = β app . Let ξ c , t \boldsymbol{\xi}_{c,t} ξ c , t and ξ τ , t \boldsymbol{\xi}_{\tau,t} ξ τ , t be the two centered estimates, and denote their trace variances by V c , t V_{c,t} V c , t and V τ , t V_{\tau,t} V τ , t . Then we have:
Var t ( d ^ t ) = V c , t + λ t 2 V τ , t + 2 λ t E t ⟨ ξ c , t , ξ τ , t ⟩ ≤ V c , t + λ t 2 V τ , t + 2 λ t V c , t V τ , t ≤ 2 V c , t + 2 λ t 2 V τ , t ≤ ν a p p ∥ g c , t r a w ∥ 2 + ω a p p . \begin{aligned}
\operatorname{Var}_t(\widehat{\mathbf{d}}_t)&=V_{c,t}+\lambda_t^2V_{\tau,t}+2\lambda_t\mathbb{E}_t\langle\boldsymbol{\xi}_{c,t},\boldsymbol{\xi}_{\tau,t}\rangle\\
&\le V_{c,t}+\lambda_t^2V_{\tau,t}+2\lambda_t\sqrt{V_{c,t}V_{\tau,t}}\\
&\le2V_{c,t}+2\lambda_t^2V_{\tau,t}\\
&\le\nu_{\mathrm{app}}\lVert\mathbf{g}_{c,t}^{\mathrm{raw}}\rVert^2+\omega_{\mathrm{app}}.
\end{aligned} Var t ( d t ) = V c , t + λ t 2 V τ , t + 2 λ t E t ⟨ ξ c , t , ξ τ , t ⟩ ≤ V c , t + λ t 2 V τ , t + 2 λ t V c , t V τ , t ≤ 2 V c , t + 2 λ t 2 V τ , t ≤ ν app ∥ g c , t raw ∥ 2 + ω app .
The first inequality is conditional Cauchy-Schwarz, and the second uses 2 a b ≤ a 2 + b 2 2ab\le a^2+b^2 2 ab ≤ a 2 + b 2 . Together they control the cross-covariance for jointly sampled estimators. Lemma C.8 now applies with the approximate constants. Summing its inequality and substituting the new step size exactly as in the preceding theorem gives (34) . □
For a deterministic scalar-broadcast correction, the squared mean error is the sum over the two branches of (28) and the conditional correction variance is zero. A uniform upper bound on that sum therefore supplies β τ 2 \beta_\tau^2 β τ 2 in the corollary. For random scalar estimates, their mean error and variance must instead be bounded separately as in Proposition C.5 . Thus the projection result supplies a correction-error bound, while the descent lemma explains how this error enters the stationarity bound.
Implementation Details
This appendix specifies the scalar correction used by ENGINE and its relation to the approximation error in Section C.4.
Scalar correction and update
Branch notation. Boldface c ℓ , c f , ℓ ∈ R p ℓ \mathbf{c}_\ell,\mathbf{c}_{f,\ell}\in\mathbb{R}^{p_\ell} c ℓ , c f , ℓ ∈ R p ℓ denotes the exact per-row coefficients in Section C.4, while c ℓ , c f , ℓ c_\ell,c_{f,\ell} c ℓ , c f , ℓ denote scalar surrogates. For ℓ ∈ { B , A } \ell\in\{B,A\} ℓ ∈ { B , A } , define X B = B X_B=B X B = B and X A = A X_A=A X A = A . For the B-branch, p B = m p_B=m p B = m , G B = G A ⊤ G_B=GA^\top G B = G A ⊤ , h B = G v f h_B=Gv_f h B = G v f , q B = v B q_B=v_B q B = v B , and s B = A v f s_B=Av_f s B = A v f . For the A-branch, p A = r p_A=r p A = r , G A = B ⊤ G G_A=B^\top G G A = B ⊤ G , h A = B ⊤ G v f h_A=B^\top Gv_f h A = B ⊤ G v f , q A = v A q_A=v_A q A = v A , and s A = v f s_A=v_f s A = v f . Here G = ∇ W f G=\nabla W_f G = ∇ W f defines the reference contractions. The matrices 1 p ℓ q ℓ ⊤ \mathbf{1}_{p_\ell}\mathbf{q}_\ell^\top 1 p ℓ q ℓ ⊤ and 1 p ℓ s ℓ ⊤ \mathbf{1}_{p_\ell}\mathbf{s}_\ell^\top 1 p ℓ s ℓ ⊤ have the same shape as X ℓ X_\ell X ℓ .
Scalar coefficients. The denominator-adjusted reference scalars are:
c ℓ = 1 p ℓ ⊤ G ℓ q ℓ ρ + p ℓ ∥ q ℓ ∥ 2 , c f , ℓ = 1 p ℓ ⊤ h ℓ ρ + p ℓ ∥ v f ∥ 2 . (35) c_\ell=\frac{\mathbf{1}_{p_\ell}^{\top}G_\ell q_\ell}{\rho+p_\ell\lVert q_\ell\rVert^2},\qquad c_{f,\ell}=\frac{\mathbf{1}_{p_\ell}^{\top}h_\ell}{\rho+p_\ell\lVert v_f\rVert^2}. \tag*{(35)} c ℓ = ρ + p ℓ ∥ q ℓ ∥ 2 1 p ℓ ⊤ G ℓ q ℓ , c f , ℓ = ρ + p ℓ ∥ v f ∥ 2 1 p ℓ ⊤ h ℓ . (35)
These scalars differ from the arithmetic row means c ˉ ℓ \bar{c}_\ell c ˉ ℓ and c ˉ f , ℓ \bar{c}_{f,\ell} c ˉ f , ℓ in Section C.4. For the exact preconditioned direction g c , ℓ = Σ ℓ G ℓ \mathbf{g}_{c,\ell}=\Sigma_\ell G_\ell g c , ℓ = Σ ℓ G ℓ , Eq. (B.1) gives:
⟨ 1 p ℓ q ℓ ⊤ , g c , ℓ ⟩ F = 1 p ℓ ⊤ G ℓ q ℓ ρ + ∥ q ℓ ∥ 2 . (36) \langle\mathbf{1}_{p_\ell}\mathbf{q}_\ell^\top,\mathbf{g}_{c,\ell}\rangle_F=\frac{\mathbf{1}_{p_\ell}^{\top}G_\ell q_\ell}{\rho+\lVert q_\ell\rVert^2}. \tag*{(36)} ⟨ 1 p ℓ q ℓ ⊤ , g c , ℓ ⟩ F = ρ + ∥ q ℓ ∥ 2 1 p ℓ ⊤ G ℓ q ℓ . (36)
When c ℓ ≠ 0 c_\ell\ne0 c ℓ = 0 , the ratio of this contraction to c ℓ c_\ell c ℓ is ( ρ + p ℓ ∥ q ℓ ∥ 2 ) / ( ρ + ∥ q ℓ ∥ 2 ) (\rho+p_\ell\lVert q_\ell\rVert^2)/(\rho+\lVert q_\ell\rVert^2) ( ρ + p ℓ ∥ q ℓ ∥ 2 ) / ( ρ + ∥ q ℓ ∥ 2 ) , which approaches p ℓ p_\ell p ℓ when ∥ q ℓ ∥ 2 ≫ ρ \lVert q_\ell\rVert^2\gg\rho ∥ q ℓ ∥ 2 ≫ ρ . For the preconditioned RLOO estimate, we use:
c ℓ i m p l = ⟨ 1 p ℓ q ℓ ⊤ , g ^ c , ℓ ⟩ F ρ + p ℓ ∥ q ℓ ∥ 2 + 10 − 8 . (37) c_\ell^{\mathrm{impl}}=\frac{\langle\mathbf{1}_{p_\ell}\mathbf{q}_\ell^\top,\widehat{\mathbf{g}}_{c,\ell}\rangle_F}{\rho+p_\ell\lVert q_\ell\rVert^2+10^{-8}}. \tag*{(37)} c ℓ impl = ρ + p ℓ ∥ q ℓ ∥ 2 + 1 0 − 8 ⟨ 1 p ℓ q ℓ ⊤ , g c , ℓ ⟩ F . (37)
For a fine probe, let Z k ∈ R m × n Z_k\in\mathbb{R}^{m\times n} Z k ∈ R m × n be the perturbation matrix for this weight block, f k f_k f k the loss at the k k k -th perturbed model, and f ˉ = K − 1 ∑ k = 1 K f k \bar{f}=K^{-1}\sum_{k=1}^{K}f_k f ˉ = K − 1 ∑ k = 1 K f k , with K > 1 K>1 K > 1 . Let v f , k v_{f,k} v f , k denote the curvature vector when the k k k -th contribution is accumulated, before its next curvature update. The sampled fine coefficients used in the correction are:
c f , B i m p l = EMA b c [ 1 ε ( K − 1 ) ∑ k = 1 K ( f k − f ˉ ) 1 m T Z k v f , k ρ + m ∥ v f , k ∥ 2 + 10 − 8 ] , c f , A i m p l = EMA b c [ 1 ε ( K − 1 ) ∑ k = 1 K ( f k − f ˉ ) 1 r T B T Z k v f , k ρ + r ∥ v f , k ∥ 2 + 10 − 8 ] . (38) \begin{aligned}
c_{f,B}^{\mathrm{impl}} &= \operatorname{EMA}_{\mathrm{bc}}\left[\frac{1}{\varepsilon(K-1)}\sum_{k=1}^{K}\frac{(f_k-\bar{f})\mathbf{1}_m^{\mathsf{T}}Z_kv_{f,k}}{\rho+m\lVert v_{f,k}\rVert^2+10^{-8}}\right],\\
c_{f,A}^{\mathrm{impl}} &= \operatorname{EMA}_{\mathrm{bc}}\left[\frac{1}{\varepsilon(K-1)}\sum_{k=1}^{K}\frac{(f_k-\bar{f})\mathbf{1}_r^{\mathsf{T}}B^{\mathsf{T}}Z_kv_{f,k}}{\rho+r\lVert v_{f,k}\rVert^2+10^{-8}}\right]. \tag*{(38)}
\end{aligned} c f , B impl c f , A impl = EMA bc [ ε ( K − 1 ) 1 k = 1 ∑ K ρ + m ∥ v f , k ∥ 2 + 1 0 − 8 ( f k − f ˉ ) 1 m T Z k v f , k ] , = EMA bc [ ε ( K − 1 ) 1 k = 1 ∑ K ρ + r ∥ v f , k ∥ 2 + 1 0 − 8 ( f k − f ˉ ) 1 r T B T Z k v f , k ] . (38)
Here EMA b c \operatorname{EMA}_{\mathrm{bc}} EMA bc is an exponential moving average with decay 0.9, initialized at zero and divided by 1 − 0.9 j 1-0.9^j 1 − 0. 9 j after j j j fine probes in the current coarse phase. Its state is reset at each new coarse phase and retained between probes. The 10 − 8 10^{-8} 1 0 − 8 term stabilizes the sampled-coefficient denominators. The superscript impl identifies the coefficients used in the update, including their estimation, normalization, and smoothing errors.
Correction and update. The scalar-corrected step, before clipping and momentum processing, is:
τ ^ ℓ i m p l = 1 p ℓ ( c ℓ i m p l q ℓ T − c f , ℓ i m p l s ℓ T ) , Δ X ℓ = − η ℓ ( g ^ c , ℓ + λ ℓ τ ^ ℓ i m p l ) . (39) \begin{aligned}
\widehat{\tau}_{\ell}^{\mathrm{impl}} &= \mathbf{1}_{p_\ell}\left(c_{\ell}^{\mathrm{impl}}q_{\ell}^{\mathsf{T}}-c_{f,\ell}^{\mathrm{impl}}s_{\ell}^{\mathsf{T}}\right),\\
\Delta X_{\ell} &= -\eta_{\ell}\left(\widehat{g}_{c,\ell}+\lambda_{\ell}\widehat{\tau}_{\ell}^{\mathrm{impl}}\right). \tag*{(39)}
\end{aligned} τ ℓ impl Δ X ℓ = 1 p ℓ ( c ℓ impl q ℓ T − c f , ℓ impl s ℓ T ) , = − η ℓ ( g c , ℓ + λ ℓ τ ℓ impl ) . (39)
With the current-basis directions defined above, this correction has the form τ ~ ℓ \widetilde{\tau}_{\ell} τ ℓ in Proposition C.6 with c ~ ℓ = ρ c ℓ i m p l \widetilde{c}_{\ell}=\rho c_{\ell}^{\mathrm{impl}} c ℓ = ρ c ℓ impl and c ~ f , ℓ = ρ c f , ℓ i m p l \widetilde{c}_{f,\ell}=\rho c_{f,\ell}^{\mathrm{impl}} c f , ℓ = ρ c f , ℓ impl . These factors account for the 1 / ρ 1/\rho 1/ ρ in the exact residual. Substituting them into Eq. (28) includes both coefficient normalization and correction scaling in the error decomposition.
Initialization safeguards. When A \mathbf{A} A is reset to zero, B-branch feedback uses A o l d v f \mathbf{A}_{\mathrm{old}}v_f A old v f if a cached basis is available and max i , j ∣ A i j ∣ < 10 − 10 \max_{i,j}|A_{ij}|<10^{-10} max i , j ∣ A ij ∣ < 1 0 − 10 . Transferred curvature vectors are normalized only when their norms exceed 10 − 10 10^{-10} 1 0 − 10 . Otherwise, their initialized values are retained. The cached-basis fallback introduces a directional approximation in addition to the scalar coefficient error analyzed above.
Detailed Experiments and Results
This appendix reports complete results and learning curves. Experimental settings and hyperparameters are collected in Appendix § 12 .
Full Results and Ablation
Table 5 and Table 6 expand the main results and ablation in Table 2 and Table 3 , reporting five-seed means and standard deviations. All results use accuracy (%). Baselines are independently reproduced under the LOREN protocol. The ablation compares ENGINE with the same V-cycle without VMS-EO correction. Bold and cyan bold mark the best and second-best results in each column. The highlighted rows report ENGINE and its absolute accuracy gain over LOREN.
Table 5. Full results over five seeds, reported as mean accuracy (%) ± standard deviation. Avg. averages the four task means.
Additional Convergence Curves
Figure 5 to Figure 8 extend Figure 4 to all 16 model-task settings, reporting validation accuracy and training loss. Figure 9 shows how the ablation results in Table 6 evolve with the forward-pass budget.
Figure 5. GPT-2-XL (1.5B) training dynamics on four GLUE tasks (MNLI, QNLI, SST-2, CoLA). Top row: validation accuracy. Bottom row: training loss.
Figure 8. OPT-13B training dynamics on four SuperGLUE tasks (RTE, BoolQ, WiC, CB). Top row: validation accuracy. Bottom row: training loss.
Figure 6. OPT-2.7B training dynamics on four GLUE tasks (MNLI, QNLI, SST-2, CoLA). Top row: validation accuracy. Bottom row: training loss.
Figure 7. LLaMA-3-8B training dynamics on four SuperGLUE tasks (RTE, BoolQ, WiC, CB). Top row: validation accuracy. Bottom row: training loss.
Experimental Setup and Hyperparameters
Tasks and metrics. We follow the LOREN protocol for datasets, splits, and training. GPT-2-XL and OPT-2.7B use FP32 on MNLI, QNLI, SST-2, and CoLA. LLaMA-3-8B and OPT-13B use BF16 on RTE, BoolQ, WiC, and CB. All tasks use accuracy (%) for the main results, ablation, learning curves, and sensitivity analyses. Avg. is the arithmetic mean of the four task accuracies for each model.
Training budget. Each run uses one NVIDIA A100 80GB GPU. All methods use batch size 64, perturbation radius ϵ = 10 − 3 \epsilon= 10^{-3} ϵ = 1 0 − 3 , and direction count K = 6 K = 6 K = 6 . Training lasts 8,000 steps for GPT-2-XL and OPT-2.7B and 4,000 steps for LLaMA-3-8B and OPT-13B.
Method configurations. Across models, MeZO-Adam uses betas (0.9, 0.999). MeZO-SVRG uses full-batch update interval 2. LOZO searches ranks { 2 , 4 , 8 } \{2, 4, 8\} { 2 , 4 , 8 } and intervals { 50 , 100 } \{50, 100\} { 50 , 100 } . HiZOO uses Hessian smoothing 10 − 8 10^{-8} 1 0 − 8 . LOREN uses curvature learning rate ν = 10 − 3 \nu= 10^{-3} ν = 1 0 − 3 and damping 10 − 1 10^{-1} 1 0 − 1 .
Tables 7 and 8 give the baseline and ENGINE settings. Baseline settings follow LOREN, and bold entries mark selected values. The ablation uses the same ENGINE settings with λ = 0 \lambda= 0 λ = 0 .
Table 7. Specific hyperparameters for the baselines. Learning rate search grids are listed by model, with selected values in bold. The two MeZO-SVRG rows give full-batch and mini-batch rates.
Hyperparameter Sensitivity
Let I = { 5 , 10 , 20 , 100 , 1000 } \mathcal{I} = \{5, 10, 20, 100, 1000\} I = { 5 , 10 , 20 , 100 , 1000 } be the candidate refresh intervals. The index j j j identifies one of the 16 model and task settings, and a j ( S ) a_j(S) a j ( S ) denotes its raw validation accuracy at refresh interval S S S . For a positive accuracy range across the candidates, we define the normalized score:
q j ( S ) = a j ( S ) − min u ∈ I a j ( u ) max u ∈ I a j ( u ) − min u ∈ I a j ( u ) , S ∈ I . (40) q_j(S) = \frac{a_j(S) - \min_{u \in\mathcal{I}} a_j(u)}{\max_{u \in\mathcal{I}} a_j(u) - \min_{u \in\mathcal{I}} a_j(u)}, \qquad S \in\mathcal{I}. \tag*{(40)} q j ( S ) = max u ∈ I a j ( u ) − min u ∈ I a j ( u ) a j ( S ) − min u ∈ I a j ( u ) , S ∈ I . (40)
Table 9 reports these per-setting Min-Max scores. At S = 20 S = 20 S = 20 , they range from 0.91 to 1.00 across all 16 settings, supporting a common default of 20 coarse steps. The per-task intervals used for the main results are listed in Table 8 .
Table 9: Sensitivity to the refresh interval S S S in coarse steps. Entries are the normalized scores q j ( S ) q_j(S) q j ( S ) in Eq. (40) . Bold marks the highest score for each model and task.
Table 9.
We also vary the coarse and fine phase lengths M M M , N ∈ { 10 , 30 , 60 } N \in\{10, 30, 60\} N ∈ { 10 , 30 , 60 } epochs, following Algorithm 1. In this schedule, each epoch comprises eight minibatch updates, so C60F60 allocates 480 coarse and 480 fine updates per cycle. For each schedule, we average validation accuracy over the 16 model and task settings, then apply Min-Max normalization across the nine schedules. Table 10 shows that C60F60 achieves the highest average validation accuracy.
Table 10: Sensitivity to the V-cycle schedule. Rows give coarse phase length M M M and columns give fine phase length N N N , both in epochs. Scores are Min-Max normalized averages over 16 model and task settings. Bold marks the best schedule.
Table 10.
Efficiency Analysis
In Table 11 , ENGINE has the highest query efficiency in all four settings and the highest wall-clock efficiency on GPT-2-XL and OPT-2.7B.
Limitations and Future Work
Adaptive query allocation. The refresh interval S S S and coarse/fine phase lengths are fixed in advance. Curvature drift can make cached fine-space information stale, while the relative benefit of each phase can change during training. Adapting refreshes and phase lengths to curvature drift or validation-loss plateaus could improve query allocation.
Learning-rate selection. The branch learning rates η B \eta_{\mathrm{B}} η B , η A \eta_{\mathrm{A}} η A , and η f \eta_{\mathrm{f}} η f share one value to limit tuning, leaving separate rates to be explored.
Access requirements and task coverage. ENGINE requires access to model parameters to perturb weights and transfer states between spaces, limiting its use through prediction-only APIs. Our experiments cover classification fine-tuning across four models and 16 model-task settings. Evaluation on generation tasks would establish whether the observed benefits extend to other fine-tuning objectives.