Introduction
Large language models (LLMs) have achieved remarkable success across a wide range of domains [7 , 2 ] . Fine-tuning these models on downstream tasks is crucial for their practical deployment. Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) [16 ] have emerged as a practical solution, training low-rank updates while keeping pretrained weights frozen [16 , 21 , 14 , 20 , 38 ] (Zhang et al. 2023). However, first-order optimization still incurs substantial memory overhead from storing activations for backpropagation, especially in long-context settings where activations dominate memory. To enhance memory efficiency, zeroth-order (ZO) fine-tuning methods [24 , 8 ] (Zhang et al. 2024) estimate descent directions from function values without backpropagated gradients, avoiding the activation storage required by backward passes [33 ] .
Recent low-rank ZO fine-tuning methods combine these two directions by restricting the finite-difference search through low-rank perturbations, adapter factors, gradient estimators, or random subspaces [9 , 39 , 32 ] (Yu et al. 2025). Factor-based approaches constrain the learned update through adapter coordinates, whereas low-rank perturbation methods constrain individual ZO steps. These two branches of methods differ in the way of constructing low-rank additive update matrices when optimizing a fixed-rank update matrix. Factor-based adaptation uses a factorized update Δ W = B A \Delta W = BA Δ W = B A and optimizes over ( B , A ) (B, A) ( B , A ) [16 ] . This representation is nonunique, since ( B Q ) ( Q − 1 A ) (BQ)(Q^{-1}A) ( B Q ) ( Q − 1 A ) gives the same update for any invertible Q Q Q . Low-rank perturbation and random-subspace methods instead use restricted finite-difference directions or estimators to construct each ZO step [9 , 39 ] (Yu et al. 2025). For example, low-rank perturbation methods form each step from a sampled rank-r r r direction U V T UV^{\mathsf{T}} U V T , while random-subspace variants restrict the step direction to a prescribed low-dimensional subspace [9 ] (Yu et al. 2025). As a result, factor-based methods optimize a non-unique parameterization, while perturbation and subspace methods restrict each ZO search direction.
To tackle these issues, we propose to optimize Δ W \Delta W Δ W as a fixed-rank matrix variable formulated on a Riemannian manifold. Consider one selected weight matrix with frozen pretrained value W 0 W_0 W 0 . At iteration t t t , the adapted weight is written as W t = W 0 + Δ W t W_t = W_0 + \Delta W_t W t = W 0 + Δ W t , with
Δ W t ∈ M r m × n , t = 0 , … , T , (1) \Delta W_t \in\mathcal{M}^{m \times n}_r,\qquad t = 0,\ldots,T, \tag*{(1)} Δ W t ∈ M r m × n , t = 0 , … , T , (1)
where M r m × n \mathcal{M}^{m \times n}_r M r m × n denotes the manifold of rank-r r r matrices in R m × n \mathbb{R}^{m \times n} R m × n , with r ≪ min { m , n } r \ll\min\{m,n\} r ≪ min { m , n } . Thus, for this block, the update trajectory stays on the same fixed-rank manifold. This turns fixed-rank structure from a property of an individual ZO step into a constraint on the persistent update trajectory. Building on this fixed-rank update trajectory, we propose a forward-only Riemannian ZO fine-tuning framework. At each step, we project a random matrix onto the current tangent space, normalize it, and retract the update back to the rank-r r r manifold to build the ZO Riemannian gradient. We use this gradient to design Zeroth-order Riemannian Gradient Descent (ZO-RGD) and a smoothed variant (ZO-RGD-S). Figure 1 illustrates the tangent random matrix and the retracted update for one representative matrix block. We summarize the contributions as follows:
Figure 1. Two loss evaluations determine a fixed-rank ZO-RGD update. The orthogonal projection onto T Δ W t M r T_{\Delta W_t}\mathcal{M}_r T Δ W t M r gives Z ^ t = P T Δ W t M r ( Z t ) \widehat{Z}_t=P_{T_{\Delta W_t}\mathcal{M}_r}(Z_t) Z t = P T Δ W t M r ( Z t ) , and normalization gives Z ~ t = m n Z ^ t / ∥ Z ^ t ∥ F \widetilde{Z}_t=\sqrt{mn}\,\widehat{Z}_t/\lVert\widehat{Z}_t\rVert_{\mathrm{F}} Z t = mn Z t / ∥ Z t ∥ F . The two points Δ W t ± ϵ Z ~ t \Delta W_t\pm\epsilon\widetilde{Z}_t Δ W t ± ϵ Z t along these directions jointly perturb all selected blocks to obtain F ± F_{\pm} F ± . Their difference gives the directional derivative s ^ t \hat{s}_t s ^ t and thus the zeroth-order Riemannian gradient G ^ t \widehat{G}_t G t . Truncated SVD retracts Δ W t − η G ^ t \Delta W_t-\eta\widehat{G}_t Δ W t − η G t back to the rank-r r r manifold.
We formulate forward-only LLM fine-tuning as optimization of persistent additive updates on fixed-rank matrix manifolds. For each selected matrix block, the update trajectory is maintained on a prescribed rank-r r r manifold, so the rank constraint is imposed on the optimization state rather than only on an individual ZO step.
We show that tangent projection makes the finite-difference perturbation scale depend on the rank and layer shape, and correct this mismatch with a normalized tangent estimator that needs only two forward passes and no additional retractions per step. Building on this estimator, we develop ZO-RGD and its smoothed variant ZO-RGD-S, which recursively smooths the finite-difference scalar at the same forward cost.
We prove that the implemented ambient finite difference agrees with its retracted counterpart to leading order, and that the normalized estimator recovers the Riemannian gradient in expectation up to an O ( ϵ 2 ) O(\epsilon^2) O ( ϵ 2 ) bias. Building on these properties, we establish a nonasymptotic convergence guarantee for ZO-RGD, conditional on regularity of the realized trajectory.
We evaluate the proposed approaches across model families and downstream tasks against existing forward-only baselines. The results show competitive performance, with gains in several settings.
Related Work
Low-rank adaptation. LoRA adapts a frozen pretrained model by adding a low-rank update to selected weight matrices [16 ] . This idea has led to a large family of PEFT methods, including weight-decomposed adaptation [21 ] , adaptive rank allocation and learning-rate asymmetry [14 ] (Zhang et al. 2023), nonzero initialization and analyses of low-rank training dynamics [20 , 41 ] , tensorized update parameterizations [40 ] , rank-one or high-rank structured variants [31 , 17 ] , low-rank gradient projection (Zhao et al., 2024a), and broader PEFT surveys [38 ] . These methods establish low-rank updates as an effective adaptation object. In a forward-only ZO setting, we ask a different question: whether the update matrix itself can be the constrained optimization state, rather than representing it through nonunique factor coordinates.
Zeroth-order fine-tuning. Classical ZO theory studies simultaneous perturbation, Gaussian or spherical smoothing, and two-point randomized estimators [33 , 13 , 12 , 27 , 30 , 4 , 3 ] . MeZO showed that such forward-only estimates can fine-tune LLMs while avoiding activation-gradient storage [24 ] . Subsequent work has expanded the optimizer and benchmark landscape through large-scale studies [8 ] , benchmark-oriented analyses (Zhang et al., 2024), random-subspace or low-rank perturbation methods [9 , 39 ] (Yu et al. 2025), and variants that use curvature or projected-alignment information [36 , 25 ] (Zhao et al. 2024b). Other approaches explore sparse perturbations [22 ] and step-size adaptation based on batch-loss variability [10 ] . LOZO [9 ] is the closest predecessor because it introduces low-rank structure into ZO estimation for LLMs. The distinction is where the structure lives: LOZO forms a low-rank estimator or subspace update, whereas the proposed algorithms construct their random matrices in the tangent space of the current fixed-rank updates.
Riemannian optimization on fixed-rank matrix manifolds. Fixed-rank matrix manifolds, tangent projections, and retractions are standard tools for optimizing low-rank matrices without exposing redundant factor gauges [1 , 34 , 37 ] . Riemannian ZO optimization studies stochastic optimization over Riemannian submanifolds embedded in Euclidean space when only noisy function evaluations are available [18 ] . Recent deep-learning work uses this geometry to optimize low-rank weights directly or to design Riemannian low-rank optimizers and preconditioners [5 , 6 , 26 ] (Zhang & Pilanci 2024). RoZO [32 ] is a closely related concurrent work that combines ZO optimization with geometry induced by low-rank adapter factors, while our method takes the matrix update itself as the optimization variable on the fixed-rank manifold. Riemannian ZO methods establish convergence for exponential-map or retraction-based finite differences under geometric and smoothness assumptions [19 , 15 ] . Structure-preserving metrics have also been proposed to address geodesic incompleteness [23 ] .
Preliminaries
Zeroth-order optimization
For a stochastic objective f ( X ) = E ξ [ F ( X ; ξ ) ] f(X) = \mathbb{E}_{\xi}[F(X;\xi)] f ( X ) = E ξ [ F ( X ; ξ )] , ZO methods estimate descent directions from function values. Given a data realization ξ \xi ξ and a random matrix Z Z Z with i.i.d. standard Gaussian entries, the two-point estimator is
∇ ^ F ( X ; ξ ) = F ( X + ϵ Z ; ξ ) − F ( X − ϵ Z ; ξ ) 2 ϵ Z , (2) \widehat{\nabla}F(X;\xi) = \frac{F(X+\epsilon Z;\xi)-F(X-\epsilon Z;\xi)}{2\epsilon}Z, \tag*{(2)} ∇ F ( X ; ξ ) = 2 ϵ F ( X + ϵ Z ; ξ ) − F ( X − ϵ Z ; ξ ) Z , (2)
where ϵ > 0 \epsilon> 0 ϵ > 0 controls the perturbation radius [12 , 27 , 30 , 4 , 3 ] . The scalar difference estimates a directional derivative, together with Z Z Z , gives a randomized gradient estimate. MeZO uses two forward passes and regenerates Z Z Z from seeds to avoid backpropagation and storing perturbations [24 ] .
Fixed-rank manifold geometry
For a matrix block, we can define the fixed-rank manifold as
M r m × n = { X ∈ R m × n : rank ( X ) = r } , d r = r ( m + n − r ) , (3) \mathcal{M}_{r}^{m \times n} = \{X \in\mathbb{R}^{m \times n} : \operatorname{rank}(X) = r\}, \qquad d_r = r(m+n-r), \tag*{(3)} M r m × n = { X ∈ R m × n : rank ( X ) = r } , d r = r ( m + n − r ) , (3)
with the Frobenius metric ⟨ G 1 , G 2 ⟩ F = tr ( G 1 T G 2 ) \langle G_1,G_2\rangle_{\mathrm{F}} = \operatorname{tr}(G_1^{\mathsf{T}}G_2) ⟨ G 1 , G 2 ⟩ F = tr ( G 1 T G 2 ) . We write M r \mathcal{M}_r M r when the block dimensions are clear. At X = U Σ V T X = U\Sigma V^{\mathsf{T}} X = U Σ V T , the thin-SVD factors U , V U,V U , V have orthonormal columns and Σ \Sigma Σ contains positive singular values. The Frobenius-orthogonal projection of a matrix Z Z Z onto the tangent space T X M r T_X\mathcal{M}_r T X M r is [1 , 34 , 37 ] denoted as
P T X M r ( Z ) = U U T Z + Z V V T − U U T Z V V T . (4) P_{T_X\mathcal{M}_r}(Z) = UU^{\mathsf{T}}Z + ZVV^{\mathsf{T}} - UU^{\mathsf{T}}ZVV^{\mathsf{T}}. \tag*{(4)} P T X M r ( Z ) = U U T Z + Z V V T − U U T Z V V T . (4)
The remaining matrix is an admissible first-order direction at the current update, and the tangent space changes as that update moves. For a step Y ∈ T X M r Y \in T_X\mathcal{M}_r Y ∈ T X M r , we can return it to M r \mathcal{M}_r M r using the rank-r r r truncated-SVD map H r \mathcal{H}_r H r :
R X ( Y ) = H r ( X + Y ) . (5) R_X(Y) = \mathcal{H}_r(X+Y). \tag*{(5)} R X ( Y ) = H r ( X + Y ) . (5)
The map retains the leading r r r singular triplets and gives the closest rank-r r r matrix in Frobenius distance locally. It is a smooth local retraction on suitable neighborhoods away from the rank-deficient boundary [1 , 34 , 5 ] .
Proposed method
Problem formulation
Let S \mathcal{S} S index the selected matrix blocks, with frozen pretrained values W 0 , ℓ \mathbf{W}_{0,\ell} W 0 , ℓ and learned additive updates Δ W ℓ ∈ M r ℓ m ℓ × n ℓ \Delta\mathbf{W}_{\ell} \in\mathcal{M}_{r_{\ell}}^{m_{\ell} \times n_{\ell}} Δ W ℓ ∈ M r ℓ m ℓ × n ℓ . The effective weights are W 0 , ℓ + Δ W ℓ \mathbf{W}_{0,\ell} + \Delta\mathbf{W}_{\ell} W 0 , ℓ + Δ W ℓ . Writing E = ∏ ℓ ∈ S R m ℓ × n ℓ \mathcal{E} = \prod_{\ell\in\mathcal{S}} \mathbb{R}^{m_{\ell} \times n_{\ell}} E = ∏ ℓ ∈ S R m ℓ × n ℓ , the matrix updates lie on the product manifold:
M = ∏ ℓ ∈ S M r ℓ m ℓ × n ℓ ⊂ E . (6) \mathcal{M} = \prod_{\ell\in\mathcal{S}} \mathcal{M}_{r_{\ell}}^{m_{\ell} \times n_{\ell}} \subset\mathcal{E}. \tag*{(6)} M = ℓ ∈ S ∏ M r ℓ m ℓ × n ℓ ⊂ E . (6)
Trainable one-dimensional parameters, including biases and LayerNorm parameters, form a Euclidean factor b ∈ R d 0 b \in\mathbb{R}^{d_0} b ∈ R d 0 , while all unselected parameters remain frozen. Thus the full trainable state belongs to the product manifold X = M × R d 0 \mathcal{X} = \mathcal{M} \times\mathbb{R}^{d_0} X = M × R d 0 . Unlike Euclidean factor coordinates, this representation is used only to compute tangent projections and retractions of the matrix variable, and the method is invariant to the choice of representative (see Appendix § 8 .§ 1 ). The loss function is defined on the surrounding Euclidean space:
F ( Δ W , b ; ξ ) : = L ( W 0 + Δ W , b ; ξ ) , ( Δ W , b ) ∈ E × R d 0 . (7) F(\Delta\mathbf{W}, b; \xi) := \mathcal{L}(\mathbf{W}_0 + \Delta\mathbf{W}, b; \xi), \quad(\Delta\mathbf{W}, b) \in\mathcal{E} \times\mathbb{R}^{d_0}. \tag*{(7)} F ( Δ W , b ; ξ ) := L ( W 0 + Δ W , b ; ξ ) , ( Δ W , b ) ∈ E × R d 0 . (7)
It couples all blocks and need not decompose over them. We propose to minimize its expectation formulated on the product manifold X \mathcal{X} X subject to the mixed constraint:
min ( Δ W , b ) ∈ X f ( Δ W , b ) : = E ξ [ F ( Δ W , b ; ξ ) ] . (8) \min_{(\Delta\mathbf{W},b)\in\mathcal{X}} f(\Delta\mathbf{W}, b) := \mathbb{E}_{\xi}[F(\Delta\mathbf{W}, b; \xi)]. \tag*{(8)} ( Δ W , b ) ∈ X min f ( Δ W , b ) := E ξ [ F ( Δ W , b ; ξ )] . (8)
ZO Riemannian Gradient Estimation
We first describe one representative matrix block, suppressing its index. In the full model, independent blockwise random matrices are applied jointly to the coupled loss, and two forward evaluations produce one scalar coefficient shared by all updates.
Finite-difference scale mismatch: At Δ W t ∈ M r \Delta\mathbf{W}_t \in\mathcal{M}_r Δ W t ∈ M r , we draw a Gaussian matrix Z t ∈ R m × n \mathbf{Z}_t \in\mathbb{R}^{m \times n} Z t ∈ R m × n with i.i.d. standard normal entries and project it using the orthogonal projection in (4) :
Z ^ t = P T Δ W t M r ( Z t ) . (9) \widehat{\mathbf{Z}}_t = P_{T_{\Delta\mathbf{W}_t}\mathcal{M}_r}(\mathbf{Z}_t). \tag*{(9)} Z t = P T Δ W t M r ( Z t ) . (9)
This gives a valid tangent direction, but changes the scale of the finite differences. The ambient Gaussian satisfies ∥ Z t ∥ F 2 ∼ χ m n 2 \lVert\mathbf{Z}_t \rVert_F^2 \sim\chi^2_{mn} ∥ Z t ∥ F 2 ∼ χ mn 2 , whereas its tangent projection has ∥ Z ^ t ∥ F 2 ∼ χ d r 2 \lVert\widehat{\mathbf{Z}}_t \rVert_F^2 \sim\chi^2_{d_r} ∥ Z t ∥ F 2 ∼ χ d r 2 , where d r = r ( m + n − r ) d_r = r(m+n-r) d r = r ( m + n − r ) . In the low-rank regime, d r ≪ m n d_r \ll mn d r ≪ mn , so a fixed ϵ \epsilon ϵ yields a typical perturbation radius ϵ d r \epsilon\sqrt{d_r} ϵ d r instead of ϵ m n \epsilon\sqrt{mn} ϵ mn . Consequently, changing the rank or layer shape also changes the perturbation scale, even when the nominal finite-difference parameter remains fixed.
Normalized gradient estimator: To remove this dependence of the perturbation scale on the rank and layer shape, we decouple the scale from the tangent projection by normalizing Z ^ t \widehat{\mathbf{Z}}_t Z t :
Z ~ t = m n Z ^ t ∥ Z ^ t ∥ F . (10) \widetilde{\mathbf{Z}}_t = \sqrt{mn}\frac{\widehat{\mathbf{Z}}_t}{\lVert\widehat{\mathbf{Z}}_t \rVert_F}. \tag*{(10)} Z t = mn ∥ Z t ∥ F Z t . (10)
The normalized random matrix remains in T Δ W t M r T_{\Delta\mathbf{W}_t}\mathcal{M}_r T Δ W t M r and has fixed norm m n \sqrt{mn} mn , matching the standard ambient Gaussian scale while retaining the tangent-space geometry. Given the same sampled realization ξ t \xi_t ξ t for both evaluations, we form
s ^ t = F ( Δ W t + ϵ Z ~ t ; ξ t ) − F ( Δ W t − ϵ Z ~ t ; ξ t ) 2 ϵ . (11) \widehat{s}_t = \frac{F(\Delta\mathbf{W}_t + \epsilon\widetilde{\mathbf{Z}}_t; \xi_t) - F(\Delta\mathbf{W}_t - \epsilon\widetilde{\mathbf{Z}}_t; \xi_t)}{2\epsilon}. \tag*{(11)} s t = 2 ϵ F ( Δ W t + ϵ Z t ; ξ t ) − F ( Δ W t − ϵ Z t ; ξ t ) . (11)
G ^ t = s ^ t Z ~ t ∈ T Δ W t M r . (12) \widehat{\mathbf{G}}_t = \widehat{s}_t \widetilde{\mathbf{Z}}_t \in T_{\Delta\mathbf{W}_t}\mathcal{M}_r. \tag*{(12)} G t = s t Z t ∈ T Δ W t M r . (12)
The scalar s ^ t \widehat{s}_t s t estimates the directional derivative of the sampled loss along Z ~ t \widetilde{\mathbf{Z}}_t Z t , and multiplying it by Z ~ t \widetilde{\mathbf{Z}}_t Z t yields a Riemannian gradient estimator on the tangent space. In the product setting, the two evaluations of F F F perturb all selected blocks jointly rather than one block at a time. Although $\widetilde{\mathbf{Z}}_t Algorithm 1 ZO-RGD
Require: Frozen weights W 0 W_0 W 0 ; initial update Δ W 0 ∈ M \Delta W_0\in\mathcal{M} Δ W 0 ∈ M ; learning rate η \eta η ; perturbation radius ϵ \epsilon ϵ .
for t = 0 , 1 , 2 , … t=0,1,2,\ldots t = 0 , 1 , 2 , … do
Sample a realization ξ t \xi_t ξ t , Gaussian random matrices for all selected blocks, and z 0 , t z_{0,t} z 0 , t .
Project and normalize each random matrix using (9) –(10) .
Compute s ^ t \hat{s}_t s ^ t with two joint forward evaluations as in (11) .
Form the ZO Riemannian gradient G ^ t , ℓ = s ^ t Z ~ t , ℓ \widehat{G}_{t,\ell}=\hat{s}_t\widetilde{Z}_{t,\ell} G t , ℓ = s ^ t Z t , ℓ for ℓ ∈ S \ell\in\mathcal{S} ℓ ∈ S as in (12) .
Retract each block as in (13) and set b t + 1 = b t − η s ^ t z 0 , t b_{t+1}=b_t-\eta\hat{s}_t z_{0,t} b t + 1 = b t − η s ^ t z 0 , t .
end for
lies in the current tangent space, the evaluation points Δ W t ± ϵ Z ~ t \Delta W_t\pm\epsilon\widetilde{Z}_t Δ W t ± ϵ Z t in (11) generally do not lie on M r \mathcal{M}_r M r . These temporary points are nevertheless admissible, because the loss in (7) is defined on the ambient space and only the persistent update is returned to the manifold by retraction. This choice saves two retractions per step. In § 5 , we will show that it agrees with retracted finite differences to leading order (Proposition 1), and the normalized estimator (12) remains aligned in expectation with the Riemannian gradient (Proposition 2).
ZO Riemannian Optimization Algorithms
As illustrated in Figure 1 , the estimator in (12) provides a search direction in the current tangent space, but a step along it is located on the tangent space. We now combine it with the retraction (5) to obtain two algorithms on the product manifold in (6) : ZO-RGD, the basic retracted gradient step, and ZO-RGD-S, which adds scalar smoothing at the same cost of two forward passes per iteration.
ZO-RGD: ZO-RGD turns the normalized tangent estimator into a forward-only update. After the two joint evaluations determine s ^ t \hat{s}_t s ^ t , each matrix block takes a step along its own random matrix. The fixed-rank retraction (5) then maps the trial point back to the constraint set:
Δ W t + 1 , ℓ = R Δ W t , ℓ ( − η G ^ t , ℓ ) , ℓ ∈ S . (13) \Delta W_{t+1,\ell}=\mathcal{R}_{\Delta W_{t,\ell}}\left(-\eta\widehat{G}_{t,\ell}\right),\qquad\ell\in\mathcal{S}. \tag*{(13)} Δ W t + 1 , ℓ = R Δ W t , ℓ ( − η G t , ℓ ) , ℓ ∈ S . (13)
The same scalar s ^ t \hat{s}_t s ^ t drives every matrix block, while the tangent spaces and retractions remain block-specific. The Euclidean factor takes the plain step b t + 1 = b t − η s ^ t z 0 , t b_{t+1}=b_t-\eta\hat{s}_t z_{0,t} b t + 1 = b t − η s ^ t z 0 , t along its raw Gaussian vector z 0 , t ∈ R d 0 z_{0,t}\in\mathbb{R}^{d_0} z 0 , t ∈ R d 0 . We initialize the update (13) at a near-zero fixed-rank point (see Appendix § 8 .§ 2 ), which yields the complete sequence summarized in Algorithm 1 .
ZO-RGD with smoothing: ZO-RGD-S uses the same random matrices and two forward evaluations, but recursively smooths the scalar coefficient in a fixed matrix-block order ℓ 1 , … , ℓ K \ell_1,\ldots,\ell_K ℓ 1 , … , ℓ K :
m t , k = β m t , k − 1 + ( 1 − β ) s ^ t , k = 1 , … , K , (14) m_{t,k}=\beta m_{t,k-1}+(1-\beta)\hat{s}_t,\qquad k=1,\ldots,K, \tag*{(14)} m t , k = β m t , k − 1 + ( 1 − β ) s ^ t , k = 1 , … , K , (14)
G ^ t , ℓ k S = m t , k Z ~ t , ℓ k , (15) \widehat{G}^{\mathrm{S}}_{t,\ell_k}=m_{t,k}\widetilde{Z}_{t,\ell_k}, \tag*{(15)} G t , ℓ k S = m t , k Z t , ℓ k , (15)
Δ W t + 1 , ℓ k = R Δ W t , ℓ k ( − η G ^ t , ℓ k S ) . (16) \Delta W_{t+1,\ell_k}=R_{\Delta W_{t,\ell_k}}\left(-\eta\widehat{G}^{\mathrm{S}}_{t,\ell_k}\right). \tag*{(16)} Δ W t + 1 , ℓ k = R Δ W t , ℓ k ( − η G t , ℓ k S ) . (16)
The scalar state passes between steps as m t , 0 = m t − 1 , K m_{t,0}=m_{t-1,K} m t , 0 = m t − 1 , K for t ≥ 1 t\ge1 t ≥ 1 . On first use, we set m 0 , 0 = s ^ 0 m_{0,0}=\hat{s}_0 m 0 , 0 = s ^ 0 , so the first-step coefficients equal those of ZO-RGD. For subsequent steps, the recursion progressively blends the previous state toward s ^ t \hat{s}_t s ^ t along the fixed block order, so different blocks generally use different smoothed coefficients within the same iteration. Algorithm 2 in Appendix § 8 .§ 3 gives the full pseudocode.
Blockwise implementation and memory: Each matrix block stores a thin-SVD update with O ( ( m ℓ + n ℓ ) r ℓ ) O((m_\ell+n_\ell)r_\ell) O (( m ℓ + n ℓ ) r ℓ ) parameters, and ZO-RGD-S adds only the scalar smoothing state. Random matrices are regenerated from the same seed for both evaluations and the update, so dense perturbations need not be retained [24 , 9 ] . The projection and retraction (QR factorizations and an SVD of a 2 r ℓ × 2 r ℓ 2r_\ell\times2r_\ell 2 r ℓ × 2 r ℓ core) act on dense tangent matrices and cost O ( m ℓ n ℓ r ℓ ) O(m_\ell n_\ell r_\ell) O ( m ℓ n ℓ r ℓ ) per block (see Appendix § 8 .§ 4 ).
Theoretical results
This section supports the estimator and algorithm of § 4 with three results: the ambient finite difference matches its retracted counterpart to leading order (Proposition 1), the normalized estimator recovers the Riemannian gradient in expectation up to a known scaling (Proposition 2), and ZO-RGD converges under trajectory-regularity conditions (Theorem 1). Proofs of these two propositions and one theorem are deferred to Appendices § 9 and § 10 , respectively.
Proposition 1 (Ambient and retracted finite differences). Let F ( ⋅ ; ξ t ) F(\cdot;\xi_t) F ( ⋅ ; ξ t ) be differentiable near Δ W t ∈ M r \Delta W_t\in\mathcal{M}_r Δ W t ∈ M r , and let R Δ W t ( τ Y ) = Δ W t + τ Y + O ( τ 2 ) R_{\Delta W_t}(\tau Y)=\Delta W_t+\tau Y+O(\tau^2) R Δ W t ( τ Y ) = Δ W t + τ Y + O ( τ 2 ) as τ → 0 \tau\to0 τ → 0 for every tangent Y Y Y . If s ^ t r e t r \hat{s}^{\mathrm{retr}}_t s ^ t retr denotes (11) evaluated at R Δ W t ( ± ϵ Z ~ t ) R_{\Delta W_t}(\pm\epsilon\widetilde{Z}_t) R Δ W t ( ± ϵ Z t ) , then s ^ t = s ^ t r e t r + o ( 1 ) \hat{s}_t=\hat{s}^{\mathrm{retr}}_t+o(1) s ^ t = s ^ t retr + o ( 1 ) as ϵ → 0 \epsilon\to0 ϵ → 0 .
A retraction follows the tangent line to first order, so both finite differences share the same limiting directional derivative. At finite ϵ \epsilon ϵ , they evaluate different points and can give different coefficients, therefore, the analysis below concerns the implemented ambient estimator.
Proposition 2 (Expected gradient alignment). Fix Δ W t ∈ M r \Delta W_t\in\mathcal{M}_r Δ W t ∈ M r and ξ t \xi_t ξ t , and assume F ( ⋅ ; ξ t ) F(\cdot;\xi_t) F ( ⋅ ; ξ t ) has a locally Lipschitz Hessian near Δ W t \Delta W_t Δ W t . For the estimator in (12) and the Frobenius metric, as ϵ → 0 \epsilon\to0 ϵ → 0 ,
E Z t [ G ^ t ∣ Δ W t , ξ t ] = m n d r grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) . (17) \mathbb{E}_{Z_t}\left[\widehat{G}_t\mid\Delta W_t,\xi_t\right]=\frac{mn}{d_r}\operatorname{grad}_{\mathcal{M}_r}F(\Delta W_t;\xi_t)+O(\epsilon^2). \tag*{(17)} E Z t [ G t ∣ Δ W t , ξ t ] = d r mn grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) . (17) If E ξ t [ F ] = f \mathbb{E}_{\xi_t}[F]=f E ξ t [ F ] = f and differentiation commutes with expectation, the same holds with f f f in place of F F F .
Proposition 2 shows that, in expectation, the estimator of each fixed-rank block is its Riemannian gradient scaled by C ℓ = m ℓ n ℓ / d r ℓ ≥ 1 \mathcal{C}_\ell=m_\ell n_\ell/d_{r_\ell}\ge1 C ℓ = m ℓ n ℓ / d r ℓ ≥ 1 . Measuring stationarity in the corresponding weighted norm ∥ ⋅ ∥ D \|\cdot\|_{\mathcal{D}} ∥ ⋅ ∥ D (Appendix § 10 .§ 1 ), which dominates the plain gradient norm, we obtain the following convergence result, conditional on the trajectory regularity in Assumptions 1–3 (Appendix § 10 .§ 2 ). The formal statement with explicit stepsize and constants is in Appendix § 10 .§ 3 .
Theorem 1 (Conditional convergence of ZO-RGD, informal). Suppose Assumptions 1–3 hold along the realized trajectory. Then, with a constant stepsize η = O ( T − 1 / 2 ) \eta=\mathcal{O}(T^{-1/2}) η = O ( T − 1/2 ) , ZO-RGD satisfies
1 T ∑ t = 0 T − 1 E [ ∥ grad f ( x t ) ∥ D 2 ] = O ( T − 1 / 2 ) + O ( T − 1 ) + O ( ϵ 4 ) , (18) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\left[\|\operatorname{grad}f(x_t)\|_{\mathcal{D}}^2\right]=\mathcal{O}(T^{-1/2})+\mathcal{O}(T^{-1})+\mathcal{O}(\epsilon^4), \tag*{(18)} T 1 t = 0 ∑ T − 1 E [ ∥ grad f ( x t ) ∥ D 2 ] = O ( T − 1/2 ) + O ( T − 1 ) + O ( ϵ 4 ) , (18) where the hidden constants depend on the initial gap, the smoothness and variance constants, and the trainable dimension.
Experiments
This section evaluates ZO-RGD and ZO-RGD-S as forward-only fine-tuning methods. § 6.2 compares them against cited MeZO and LOZO results on the eleven-task OPT suite at 13B and 30B, the scale at which those baselines report their main results. § 6.3 removes the reporting asymmetries of cited numbers with a self-run comparison in which every method shares the same protocol, checkpoint rule, and seeds. § 6.4 tests whether the gains carry over to Llama-3.1-8B and Qwen2.5-7B. § 6.5 isolates the geometric ingredients of the method through controlled ablations and reports the computational price of projection and retraction. Full configurations, per-seed values, the complete OPT-1.3B comparison, and further diagnostics are in Appendix § 11 .
Setup
We evaluate the fixed-rank update formulation on OPT-1.3B, OPT-13B, and OPT-30B (Zhang et al. (2022)), using the eleven-task MeZO/LOZO suite and its task metrics (Malladi et al. [24 ] ; Chen et al. [9 ] ; Wang et al. [35 ] ; Rajpurkar et al. [28 ] ; Dua et al. [11 ] ), and test portability on Llama-3.1-8B and Qwen2.5-7B. Our runs use 1000 training examples and 20,000 updates, and each update uses two forward passes.
Broad-suite comparison with cited baselines
OPT-13B. Table 1 follows the OPT-13B table format of LOZO. The cited rows are kept in the same order, and ZO-RGD and ZO-RGD-S are added under the setup above. This is the main large-model comparison in the paper because it uses the full eleven-task OPT suite at the scale where MeZO and LOZO report their main OPT results.
Table 1. OPT-13B main comparison on the LOZO eleven-task suite. The results of Zero-shot, ICL, MeZO, MeZO-LoRA, and LOZO are from Chen et al. [9 ] ; ZO-RGD and ZO-RGD-S are evaluated under the setup described above, with each cell selected by the development metric within that variant’s 20,000-step sweep cells. Best and second-best results in each column are shown in bold and <u>underlined</u>, respectively. Fail denotes that the ZO-RGD’s ReCoRD run diverged and produced no valid evaluation (see Appendix § 11 .18 for details).
Against the strongest cited ZO training row among MeZO, MeZO-LoRA, and LOZO, ZO-RGD-S improves on 6 of the 11 tasks and ZO-RGD on 4 of its 10 reported tasks. On 8 of the 11 tasks at least one of the two variants is above the strongest cited row: SST-2, RTE, CB, BoolQ, WiC, MultiRC, ReCoRD, and SQuAD, spanning sentiment classification, entailment, semantic reasoning, and extractive QA. WSC, COPA, and DROP remain stronger for the cited baselines, so the advantage is broad but not uniform across all tasks.
OPT-30B. Table 2 follows the OPT-30B part of the LOZO scaling table. We keep the zero-shot, ICL, MeZO, and LOZO rows from the cited table and add ZO-RGD and ZO-RGD-S. This table evaluates whether the same fixed-rank additive-update formulation remains effective at OPT-30B scale. Because checkpoint selection is not fully matched for OPT-30B, we use this table as scaling evidence, with run-level reporting details given in Appendix § 11 .§ 1 .
Table 2. OPT-30B scaling comparison on the LOZO task subset. Zero-shot, ICL, MeZO, and LOZO are cited from Chen et al. [9 ] ; ZO-RGD and ZO-RGD-S are evaluated with the same training budget, with checkpoint-selection details reported in Appendix § 11 .§ 1 . Best and second-best results in each column are shown in bold and <u>underlined</u>, respectively. SQuAD uses F1, and the other columns use the OPT task metrics.
At OPT-30B, the better of ZO-RGD and ZO-RGD-S improves over the cited LOZO row on SST-2, RTE, and WiC, ties LOZO on BoolQ, and remains competitive on WSC and SQuAD. Together with the OPT-13B results, this scaling study supports the central method claim: fixed-rank constraints can be imposed directly on learned additive updates while preserving the forward-only fine-tuning
pattern. The complete local forward-only comparison at OPT-1.3B is in Appendix § 11 .§ 3 , where the two Riemannian variants give the best forward-only result on 7 of 11 tasks.
Controlled comparison under a symmetric protocol
Table 3 complements the broad-suite results with self-run baselines, method-specific development-selected learning rates fixed across report seeds, and symmetric development-loss checkpoint selection. ZO-RGD-S reaches 69.4 ± 3.3 69.4 \pm3.3 69.4 ± 3.3 on RTE and 84.9 ± 1.3 84.9 \pm1.3 84.9 ± 1.3 on SQuAD, exceeding MeZO and LOZO on both OPT-13B tasks. The first-order LoRA+AdamW reference is stronger on RTE but runs out of A100-80G memory on SQuAD at batch size 16 without activation checkpointing. The OPT-30B BoolQ column is a single-seed spot-check under the same symmetric protocol (see Appendix § 11 .§ 6 ). Per-seed values and a shared-learning-rate regime, including loss trajectories against forward passes, are in Appendices § 11 .§ 7 and § 11 .9.
Table 3. Controlled OPT comparison with fixed report-seed hyperparameters and symmetric development-loss checkpoint selection. OPT-13B: mean ± \pm ± sd over three seeds (RTE: accuracy; SQuAD: F1). OPT-30B BoolQ: single-seed accuracy at learning rate 10 − 7 10^{-7} 1 0 − 7 . ZO-RGD uses r = 4 r = 4 r = 4 , ZO-RGD-S uses r = 2 r = 2 r = 2 . Passes/update counts forward (F) and backward (B) evaluations; bold marks the best forward-only result. NR denotes a matched result not reported in this comparison; OOM denotes an out-of-memory failure.
Scalar smoothing at matched rank. At matched rank (r = 2 r = 2 r = 2 ) and learning rate on OPT-13B (Appendix § 11 .8, Table 14 ), ZO-RGD-S has a higher mean than ZO-RGD on RTE (69.4 versus 64.5), while ZO-RGD has a slightly higher mean on SQuAD (85.4 versus 84.9). Neither paired difference is statistically significant (two-sided paired t t t -tests, p = 0.28 / 0.47 p = 0.28/0.47 p = 0.28/0.47 ); these three-seed comparisons do not establish an advantage for either variant.
Table 14: Scalar smoothing at fixed rank and learning rate on OPT-13B. Both methods use r = 2 r = 2 r = 2 and paired seeds { 1 , 2 , 3 } \{1, 2, 3\} { 1 , 2 , 3 } . The rates were selected on the ZO-RGD-S development grid and fixed before the ZO-RGD runs. Scores are mean ± \pm ± sd (%); all runs completed.
Table 14.
Portability beyond OPT
Table 4 extends the recipe to Llama-3.1-8B and Qwen2.5-7B across four tasks. Results support portability across model families, with task-dependent performance. Learning rates come from method-specific development pilots. A seed-0 Llama BoolQ learning-rate diagnostic (Appendix § 11 .10, Table 17 ) shows a strong learning-rate effect for all three methods: at 10 − 7 10^{-7} 1 0 − 7 , LOZO reaches 80.9, and both Riemannian variants reach 85.5. This single-seed observation is separate from the three-seed panel and does not establish a method ranking.
Table 4. Modern-model results (mean ± \pm ± sd, three seeds, %; RTE/BoolQ: accuracy; SQuAD: F1; GSM8K: exact match on n = 200 n=200 n = 200 ).
Table 17 : Llama-3.1-8B BoolQ learning-rate diagnostic (accuracy, %). All entries use seed 0 and 20,000 updates; each method retains its panel rank.
Table 17.
Geometry and computational cost
Geometry and parameterization. The controls in Table 5 test the finite-difference rule and the optimization variable in separate blocks. Normalized tangent finite differences outperform the tested alternatives on SST-2, including learning-rate-compensated ambient random matrices. The separate parameterization comparison favors the manifold update on SST-2 and SQuAD, but not RTE at the pilot-selected rates. A separate three-seed SST-2 check at the manifold arm’s 20,000-step grid rate 10 − 7 10^{-7} 1 0 − 7 gives 92.7 ± 0.7 92.7 \pm0.7 92.7 ± 0.7 , illustrating sensitivity to the tuning budget. Further controls, negative results, ablations, and stability diagnostics are in Appendices § 11 .11–§ 11 .15.
Table 5. OPT-1.3B geometry controls (%; mean ± \pm ± sd, three paired seeds). Left: finite-difference geometry on SST-2. Right: manifold versus BA-factor ZO with matched budgets (accuracy; SQuAD: F1). Learning rates are selected separately for each panel; compare within a panel. Right-panel rates use a shared 2,000-step development pilot: 10 − 6 / 10 − 7 / 10 − 6 10^{-6}/10^{-7}/10^{-6} 1 0 − 6 /1 0 − 7 /1 0 − 6 for the manifold arm on SST-2/RTE/SQuAD and 10 − 4 10^{-4} 1 0 − 4 for BA-factor.
Runtime and memory. Across two OPT-13B SQuAD profiles on different A100-80G cards (1,000 steps per method per card), ZO-RGD/ZO-RGD-S average 3.51/3.54 seconds per update versus LOZO’s 1.93, all at approximately 39.4 GB peak reserved memory as shown in Table 6 . Thus comparable memory comes with higher per-step cost. Per-step times differ by less than 1% between the two hardware repeats, and peak reserved memory agrees at the logged precision (see Table 23 , Appendix § 11 .16). LoRA+AdamW runs out of memory at its first forward pass in both allocations under the same batch-size-16 configuration; the fixed-rank update state itself is negligible (see Appendix § 11 .17).
Table 6. Measured cost at OPT-13B SQuAD, averaged over the two A100-80G hardware repeats in Table 23 . LoRA+AdamW exhausts the A100 on SQuAD (batch 16, no activation checkpointing); its 126.2 GB entry is a separate three-run RTE mean on H200.
Table 23. Repeated OPT-13B SQuAD cost profile. Each repeat runs all four methods back to back for 1,000 steps each in one allocation, on a different A100-80G card, with intermediate evaluation disabled. These are hardware repeats, not training seeds.
Conclusion
We formulate forward-only LLM fine-tuning as optimization of persistent additive updates on fixed-rank matrix manifolds, so that the rank constraint is imposed on the optimization state rather than on individual ZO steps. ZO-RGD estimates the Riemannian gradient from two forward passes with normalized tangent finite differences and retracts each update onto the manifold. ZO-RGD-S adds recursive smoothing of the finite-difference scalar at the same cost. We show that the ambient finite difference agrees with its retracted counterpart to leading order, that the normalized estimator recovers the Riemannian gradient in expectation up to an O ( ϵ 2 ) O(\epsilon^2) O ( ϵ 2 ) bias, and that ZO-RGD admits a nonasymptotic convergence guarantee conditional on trajectory regularity. Both variants are competitive with strong forward-only baselines on the OPT suite, under a controlled symmetric protocol, and on Llama-3.1-8B and Qwen2.5-7B, with gains in several settings that mechanism tests attribute to tangent normalization and the manifold parameterization.
AI Use Statement
In this work, we used generative AI tools to assist in implementing the methods, proving the conditional convergence result, and to improve the readability of the paper. We have not used generative AI tools for any other task with required disclosure. We have reviewed all AI-assisted work: the code was tested by the authors, and all text edits were checked to preserve the technical content. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Ethics Statement
This paper proposes a memory-efficient, forward-only optimizer for fine-tuning large language models and evaluates it on publicly available pretrained models (OPT, Llama-3.1, and Qwen2.5) and public benchmark data sets. The work involves no human subjects, no newly collected data, and no private or personally identifiable information. Lowering the memory required for fine-tuning broadens access to model adaptation; this carries the dual-use considerations common to all fine-tuning methods, since adapted models can serve beneficial or harmful ends, but our method introduces no risk beyond that of existing fine-tuning approaches.
Reproducibility Statement
The proposed methods are fully specified: ZO-RGD in Algorithm 1 (§ 4.3 ) and ZO-RGD-S in Algorithm 2 (Appendix § 8 .§ 3 ), with the stored representation, initialization, retraction, and per-step cost detailed in Appendix § 8 . All assumptions underlying the theoretical results are stated in § 5 and Appendix § 10 .§ 2 ; complete proofs of the propositions and of the convergence theorem are given in Appendices § 9 and § 10 , respectively. The evaluation protocol is described in § 6.1 and Appendix § 11 .§ 1 , compute resources in Appendix § 11 .§ 2 , and per-seed values in Appendix § 11 .§ 7 . All experiments use publicly available pretrained models and public data sets. The source code will be released upon acceptance, together with a manifest that maps every reported cell to its job configuration, selection rule, seeds, hardware, and completion status, including failed runs.
Algorithmic and Implementation Details
This appendix follows the order of § 4 : the thin-SVD representation (Appendix § 8 .§ 1 ), the near-zero initialization of the fixed-rank state (Appendix § 8 .§ 2 ), the full pseudocode of ZO-RGD-S (Appendix § 8 .§ 3 ), and the complexity of the implementation (Appendix § 8 .§ 4 ).
Thin-SVD representation
The optimization variable is the matrix Δ W ℓ \Delta W_{\ell} Δ W ℓ itself, and the stored thin-SVD triple ( U ℓ , Σ ℓ , V ℓ ) (U_{\ell},\Sigma_{\ell},V_{\ell}) ( U ℓ , Σ ℓ , V ℓ ) is a numerical representation of that point. The representation is unique only up to simultaneous sign changes of paired singular vectors, and, when singular values coincide, up to joint rotations of the corresponding subspaces. None of this affects the algorithm or the analysis, because projection, normalization, and retraction depend on the represented point only: they are invariant to the choice of representative. This is the precise sense in which the method has no factor-gauge degrees of freedom, in contrast to optimizing LoRA factor coordinates ( B , A ) (B,A) ( B , A ) .
Near-zero initialization
The fixed-rank constraint excludes the zero update when r ℓ > 0 r_{\ell} > 0 r ℓ > 0 . To preserve the pretrained initialization while remaining feasible for ZO-RGD and ZO-RGD-S, we initialize each trainable update block at a near-zero fixed-rank point:
Δ W i n i t , ℓ = U i n i t , ℓ ( σ i n i t , ℓ I r ℓ ) V i n i t , ℓ T , ℓ ∈ S , (19) \Delta W_{\mathrm{init},\ell}=U_{\mathrm{init},\ell}(\sigma_{\mathrm{init},\ell}I_{r_\ell})V_{\mathrm{init},\ell}^{\mathsf{T}}, \qquad\ell\in\mathcal{S}, \tag*{(19)} Δ W init , ℓ = U init , ℓ ( σ init , ℓ I r ℓ ) V init , ℓ T , ℓ ∈ S , (19)
where 0 < σ i n i t , ℓ ≪ 1 0<\sigma_{\mathrm{init},\ell}\ll1 0 < σ init , ℓ ≪ 1 , and U i n i t , ℓ U_{\mathrm{init},\ell} U init , ℓ and V i n i t , ℓ V_{\mathrm{init},\ell} V init , ℓ have orthonormal columns. The factors are obtained by QR orthonormalization of Gaussian matrices (computed in single precision and cast to the model precision), and σ i n i t , ℓ = 10 − 6 \sigma_{\mathrm{init},\ell}=10^{-6} σ init , ℓ = 1 0 − 6 by default. This keeps each update block on its prescribed fixed-rank manifold while making W 0 + Δ W i n i t W_0+\Delta W_{\mathrm{init}} W 0 + Δ W init numerically close to the pretrained model.
ZO-RGD-S pseudocode
Algorithm 2 expands the scalar recursion in (14) . As in Algorithm 1, the pseudocode shows the matrix-block updates.
Algorithm 2 ZO-RGD-S
Require: Frozen weights W 0 W_0 W 0 ; initial update Δ W 0 ∈ M \Delta W_0\in\mathcal{M} Δ W 0 ∈ M ; learning rate η \eta η ; perturbation radius ϵ \epsilon ϵ ; smoothing parameter β ∈ [ 0 , 1 ) \beta\in[0,1) β ∈ [ 0 , 1 ) .
Fix an order ℓ 1 , … , ℓ K \ell_1,\ldots,\ell_K ℓ 1 , … , ℓ K of the trainable matrix blocks.
for t = 0 , 1 , 2 , … t=0,1,2,\ldots t = 0 , 1 , 2 , … do
Sample a realization ξ t \xi_t ξ t , construct normalized tangent random matrices, and compute s ^ t \hat{s}_t s ^ t with the same two forward passes as ZO-RGD.
if t = 0 t=0 t = 0 then
m 0 , 0 = s ^ 0 m_{0,0}=\hat{s}_0 m 0 , 0 = s ^ 0 .
else
m t , 0 = m t − 1 , K m_{t,0}=m_{t-1,K} m t , 0 = m t − 1 , K .
end if
for k = 1 , … , K k=1,\ldots,K k = 1 , … , K do
m t , k = β m t , k − 1 + ( 1 − β ) s ^ t m_{t,k}=\beta m_{t,k-1}+(1-\beta)\hat{s}_t m t , k = β m t , k − 1 + ( 1 − β ) s ^ t .
G ^ t , ℓ k S = m t , k Z ~ t , ℓ k \widehat{G}^{\mathsf{S}}_{t,\ell_k}=m_{t,k}\widetilde{Z}_{t,\ell_k} G t , ℓ k S = m t , k Z t , ℓ k .
Δ W t + 1 , ℓ k = R Δ W t , ℓ k ( − η G ^ t , ℓ k S ) \Delta W_{t+1,\ell_k}=R_{\Delta W_{t,\ell_k}}\left(-\eta\widehat{G}^{\mathsf{S}}_{t,\ell_k}\right) Δ W t + 1 , ℓ k = R Δ W t , ℓ k ( − η G t , ℓ k S ) .
end for
end for
Complexity of the implementation
Two costs are distinguished: the persistent trainable state and the per-step work.
Persistent state. At each selected two-dimensional block, the trainable update is stored as the thin-SVD triple ( U ℓ , Σ ℓ , V ℓ ) (U_\ell,\Sigma_\ell,V_\ell) ( U ℓ , Σ ℓ , V ℓ ) , with U ℓ ∈ R m ℓ × r ℓ U_\ell\in\mathbb{R}^{m_\ell\times r_\ell} U ℓ ∈ R m ℓ × r ℓ , diagonal Σ ℓ ∈ R r ℓ × r ℓ \Sigma_\ell\in\mathbb{R}^{r_\ell\times r_\ell} Σ ℓ ∈ R r ℓ × r ℓ , and V ℓ ∈ R n ℓ × r ℓ V_\ell\in\mathbb{R}^{n_\ell\times r_\ell} V ℓ ∈ R n ℓ × r ℓ . The per-block state therefore has r ℓ ( m ℓ + n ℓ + 1 ) r_\ell(m_\ell+n_\ell+1) r ℓ ( m ℓ + n ℓ + 1 ) parameters, i.e. O ( ( m ℓ + n ℓ ) r ℓ ) O((m_\ell+n_\ell)r_\ell) O (( m ℓ + n ℓ ) r ℓ ) , much smaller than a dense m ℓ n ℓ m_\ell n_\ell m ℓ n ℓ trainable update when r ℓ ≪ min ( m ℓ , n ℓ ) r_\ell\ll\min(m_\ell,n_\ell) r ℓ ≪ min ( m ℓ , n ℓ ) . The training loop stores no first-order optimizer state and no backpropagation graph; random matrices are regenerated from seeds, and the thin-SVD triple is the only trainable representation. ZO-RGD-S adds only a scalar smoothing variable and does not maintain a matrix-valued momentum buffer. Thus the forward-only memory profile remains in the MeZO/LOZO regime, with an additional fixed-rank update state that scales linearly in rank; Appendix § 11 .17 quantifies this state for OPT-1.3B.
Per-step work. The per-step work, however, is not low-rank: the current implementation materializes dense m ℓ × n ℓ m_\ell\times n_\ell m ℓ × n ℓ intermediates, the Gaussian random matrix, its tangent projection, and the update direction, so each block incurs O ( m ℓ n ℓ r ℓ ) O(m_\ell n_\ell r_\ell) O ( m ℓ n ℓ r ℓ ) work in the projection and retraction products plus O ( m ℓ n ℓ ) O(m_\ell n_\ell) O ( m ℓ n ℓ ) transient memory, in addition to the two forward passes; only the core SVD of the retraction is small ( 2 r ℓ × 2 r ℓ ) (2r_\ell\times2r_\ell) ( 2 r ℓ × 2 r ℓ ) . In two allocations on different A100-80G chips, profiling all methods back to back gives roughly 1.8 × 1.8\times 1.8 × the per-step time of LOZO at the identical forward-pass count (Table 6 ). Projection and retraction add computation while retaining comparable peak memory. A fused factored application that samples random matrices directly in factor coordinates could avoid the dense intermediates in principle, but it is an engineering change we have not implemented or measured, and a “lazy” retraction applied every few steps would change the algorithm itself (intermediate iterates would leave the manifold and random matrices would use a stale tangent space), requiring its own bias and convergence analysis.
Proofs for Tangent-Normalized ZO Estimation
This appendix proves Propositions 1 and 2 of § 5 . For the reader’s convenience, each proposition is restated under its original number immediately before its proof.
Proposition 1 (Ambient and retracted finite differences). Let F ( ⋅ ; ξ t ) F(\cdot;\xi_t) F ( ⋅ ; ξ t ) be differentiable near Δ W t ∈ M r \Delta W_t \in\mathcal{M}_r Δ W t ∈ M r , and let R Δ W t ( τ Y ) = Δ W t + τ Y + O ( τ 2 ) R_{\Delta W_t}(\tau Y)=\Delta W_t+\tau Y+O(\tau^2) R Δ W t ( τ Y ) = Δ W t + τ Y + O ( τ 2 ) as τ → 0 \tau\to0 τ → 0 for every tangent Y Y Y . If s ^ t r e t r \hat{s}^{\mathrm{retr}}_t s ^ t retr denotes (11) evaluated at R Δ W t ( ± ϵ Z ~ t ) R_{\Delta W_t}(\pm\epsilon\widetilde{Z}_t) R Δ W t ( ± ϵ Z t ) , then s ^ t = s ^ t r e t r + o ( 1 ) \hat{s}_t=\hat{s}^{\mathrm{retr}}_t+o(1) s ^ t = s ^ t retr + o ( 1 ) as ϵ → 0 \epsilon\to0 ϵ → 0 .
Write the implemented scalar s ^ t \hat{s}_t s ^ t of (11) as s ^ t l i n \hat{s}^{\mathrm{lin}}_t s ^ t lin , and its retracted counterpart as s ^ t r e t r \hat{s}^{\mathrm{retr}}_t s ^ t retr .
s ^ t l i n = F ( Δ W t + ϵ Z ~ t ; ξ t ) − F ( Δ W t − ϵ Z ~ t ; ξ t ) 2 ϵ , (20) \hat{s}^{\mathrm{lin}}_t=\frac{F(\Delta W_t+\epsilon\widetilde{Z}_t;\xi_t)-F(\Delta W_t-\epsilon\widetilde{Z}_t;\xi_t)}{2\epsilon}, \tag*{(20)} s ^ t lin = 2 ϵ F ( Δ W t + ϵ Z t ; ξ t ) − F ( Δ W t − ϵ Z t ; ξ t ) , (20)
s ^ t r e t r = F ( R Δ W t ( ϵ Z ~ t ) ; ξ t ) − F ( R Δ W t ( − ϵ Z ~ t ) ; ξ t ) 2 ϵ . (21) \hat{s}^{\mathrm{retr}}_t=\frac{F(R_{\Delta W_t}(\epsilon\widetilde{Z}_t);\xi_t)-F(R_{\Delta W_t}(-\epsilon\widetilde{Z}_t);\xi_t)}{2\epsilon}. \tag*{(21)} s ^ t retr = 2 ϵ F ( R Δ W t ( ϵ Z t ) ; ξ t ) − F ( R Δ W t ( − ϵ Z t ) ; ξ t ) . (21)
Proof. The linearized finite difference satisfies
s ^ t l i n = ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z ~ t ⟩ F + o ( 1 ) . \hat{s}^{\mathrm{lin}}_t=\left\langle\nabla_{\Delta W}F(\Delta W_t;\xi_t),\widetilde{Z}_t\right\rangle_F+o(1). s ^ t lin = ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z t ⟩ F + o ( 1 ) .
By the retraction property,
R Δ W t ( ± ϵ Z ~ t ) = Δ W t ± ϵ Z ~ t + O ( ϵ 2 ) . R_{\Delta W_t}(\pm\epsilon\widetilde{Z}_t)=\Delta W_t\pm\epsilon\widetilde{Z}_t+O(\epsilon^2). R Δ W t ( ± ϵ Z t ) = Δ W t ± ϵ Z t + O ( ϵ 2 ) .
Substituting this expansion into the retracted finite difference gives the same leading directional derivative. Hence
s ^ t r e t r = ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z ~ t ⟩ F + o ( 1 ) , \hat{s}^{\mathrm{retr}}_t=\left\langle\nabla_{\Delta W}F(\Delta W_t;\xi_t),\widetilde{Z}_t\right\rangle_F+o(1), s ^ t retr = ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z t ⟩ F + o ( 1 ) ,
which proves the claim. □
Proposition 2 (Expected gradient alignment). Fix Δ W t ∈ M r \Delta W_t\in\mathcal{M}_r Δ W t ∈ M r and ξ t \xi_t ξ t , and assume F ( ⋅ ; ξ t ) F(\cdot;\xi_t) F ( ⋅ ; ξ t ) has a locally Lipschitz Hessian near Δ W t \Delta W_t Δ W t . For the estimator in (12) and the Frobenius metric, as ϵ → 0 \epsilon\to0 ϵ → 0 ,
E Z t [ G ~ t ∣ Δ W t , ξ t ] = m n d r grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) . \mathbb{E}_{Z_t}[\widetilde{G}_t\mid\Delta W_t,\xi_t]=\frac{mn}{d_r}\operatorname{grad}_{\mathcal{M}_r}F(\Delta W_t;\xi_t)+O(\epsilon^2). E Z t [ G t ∣ Δ W t , ξ t ] = d r mn grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) . If E ξ t [ F ] = f \mathbb{E}_{\xi_t}[F]=f E ξ t [ F ] = f and differentiation commutes with expectation, the same holds with f f f in place of F F F .
Proof. Let
T t : = T Δ W t M r , d r = dim ( T t ) = r ( m + n − r ) . T_t:=T_{\Delta W_t}\mathcal{M}_r,\qquad d_r=\dim(T_t)=r(m+n-r). T t := T Δ W t M r , d r = dim ( T t ) = r ( m + n − r ) .
Since Z ~ t = P T t ( Z t ) \widetilde{Z}_t=P_{T_t}(Z_t) Z t = P T t ( Z t ) is the projection of an isotropic Gaussian matrix onto the tangent space, the normalized vector
u t : = Z ~ t ∥ Z ~ t ∥ F u_t:=\frac{\widetilde{Z}_t}{\|\widetilde{Z}_t\|_F} u t := ∥ Z t ∥ F Z t
is isotropic on the unit sphere of T t T_t T t . Thus
Z ~ t = m n u t . \widetilde{Z}_t=\sqrt{mn}\,u_t. Z t = mn u t .
Assuming that F ( ⋅ ; ξ t ) F(\cdot;\xi_t) F ( ⋅ ; ξ t ) is sufficiently smooth in a neighborhood of Δ W t \Delta W_t Δ W t , a Taylor expansion for fixed ξ t \xi_t ξ t gives
s ^ t = F ( Δ W t + ϵ Z ~ t ; ξ t ) − F ( Δ W t − ϵ Z ~ t ; ξ t ) 2 ϵ (22) \hat{s}_t=\frac{F(\Delta W_t+\epsilon\widetilde{Z}_t;\xi_t)-F(\Delta W_t-\epsilon\widetilde{Z}_t;\xi_t)}{2\epsilon} \tag*{(22)} s ^ t = 2 ϵ F ( Δ W t + ϵ Z t ; ξ t ) − F ( Δ W t − ϵ Z t ; ξ t ) (22)
= ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z ~ t ⟩ F + O ( ϵ 2 ) . (23) =\left\langle\nabla_{\Delta W}F(\Delta W_t;\xi_t),\widetilde{Z}_t\right\rangle_F+O(\epsilon^2). \tag*{(23)} = ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , Z t ⟩ F + O ( ϵ 2 ) . (23)
Let
g t : = grad M r F ( Δ W t ; ξ t ) = P T t ( ∇ Δ W F ( Δ W t ; ξ t ) ) g_t := \operatorname{grad}_{\mathcal{M}_r} F(\Delta W_t;\xi_t) = P_{T_t}\left(\nabla_{\Delta W} F(\Delta W_t;\xi_t)\right) g t := grad M r F ( Δ W t ; ξ t ) = P T t ( ∇ Δ W F ( Δ W t ; ξ t ) )
be the Riemannian gradient under the Frobenius metric. Since u t ∈ T t u_t \in T_t u t ∈ T t , we have
⟨ ∇ Δ W F ( Δ W t ; ξ t ) , u t ⟩ F = ⟨ g t , u t ⟩ F . \left\langle\nabla_{\Delta W}F(\Delta W_t;\xi_t),u_t\right\rangle_F = \left\langle g_t,u_t\right\rangle_F. ⟨ ∇ Δ W F ( Δ W t ; ξ t ) , u t ⟩ F = ⟨ g t , u t ⟩ F .
Therefore,
G ^ t = s ^ t Z ~ t = m n ⟨ g t , u t ⟩ F u t + O ( ϵ 2 ) , (24) \widehat{G}_t = \widehat{s}_t\widetilde{Z}_t = mn\left\langle g_t,u_t\right\rangle_F u_t + O(\epsilon^2), \tag*{(24)} G t = s t Z t = mn ⟨ g t , u t ⟩ F u t + O ( ϵ 2 ) , (24)
where the remainder is in the Frobenius norm. By isotropy on the d r d_r d r -dimensional tangent space,
E Z t [ ⟨ g t , u t ⟩ F u t ∣ Δ W t , ξ t ] = 1 d r g t . \mathbb{E}_{Z_t}\left[\left\langle g_t,u_t\right\rangle_F u_t \mid\Delta W_t,\xi_t\right] = \frac{1}{d_r}g_t. E Z t [ ⟨ g t , u t ⟩ F u t ∣ Δ W t , ξ t ] = d r 1 g t .
Hence
E Z t [ G ^ t ∣ Δ W t , ξ t ] = m n d r grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) . \mathbb{E}_{Z_t}\left[\widehat{G}_t \mid\Delta W_t,\xi_t\right] = \frac{mn}{d_r}\operatorname{grad}_{\mathcal{M}_r}F(\Delta W_t;\xi_t) + O(\epsilon^2). E Z t [ G t ∣ Δ W t , ξ t ] = d r mn grad M r F ( Δ W t ; ξ t ) + O ( ϵ 2 ) .
If E ξ t [ F ( Δ W ; ξ t ) ] = f ( Δ W ) \mathbb{E}_{\xi_t}[F(\Delta W;\xi_t)] = f(\Delta W) E ξ t [ F ( Δ W ; ξ t )] = f ( Δ W ) and differentiation commutes with expectation, then taking expectation over ξ t \xi_t ξ t gives the second statement.
Convergence Analysis of ZO-RGD
This appendix proves the results of § 5 for ZO-RGD exactly as implemented: the mixed product of fixed-rank blocks and a Euclidean factor of one-dimensional parameters, blockwise normalized tangent random matrices, one shared finite-difference scalar per step, ambient tangent-line evaluation points, and update-only truncated-SVD retraction. Proposition 2 is the single-block specialization of Lemmas 1–3. Throughout, “ZO-RGD” means the base method of Algorithm 1 with one joint random direction per step, zero weight decay, no singular-value clamping, no gradient accumulation, and a differentiable training loss.
Algorithm restatement
Blocks and constants. The trainable two-dimensional blocks are indexed by ℓ ∈ { 1 , … , N } \ell\in\{1,\ldots,N\} ℓ ∈ { 1 , … , N } ; block ℓ \ell ℓ has ambient shape m ℓ × n ℓ m_\ell\times n_\ell m ℓ × n ℓ and fixed rank 1 ≤ r ℓ ≤ min ( m ℓ , n ℓ ) 1 \le r_\ell\le\min(m_\ell,n_\ell) 1 ≤ r ℓ ≤ min ( m ℓ , n ℓ ) . (Throughout, N N N is the block count; the symbol L L L is reserved for the gradient-Lipschitz constant of Assumption 2, and ρ 3 \rho_3 ρ 3 for its third-derivative constant, distinct from the normalization radii ρ ℓ \rho_\ell ρ ℓ .) Write M ℓ : = M r ℓ m ℓ × n ℓ \mathcal{M}_\ell:= \mathcal{M}_{r_\ell}^{m_\ell\times n_\ell} M ℓ := M r ℓ m ℓ × n ℓ for the fixed-rank manifold of block ℓ \ell ℓ ,
d ℓ : = dim M ℓ = r ℓ ( m ℓ + n ℓ − r ℓ ) , ρ ℓ 2 : = m ℓ n ℓ , c ℓ : = ρ ℓ 2 d ℓ . d_\ell:= \dim\mathcal{M}_\ell= r_\ell(m_\ell+n_\ell-r_\ell), \qquad\rho_\ell^2 := m_\ell n_\ell, \qquad c_\ell:= \frac{\rho_\ell^2}{d_\ell}. d ℓ := dim M ℓ = r ℓ ( m ℓ + n ℓ − r ℓ ) , ρ ℓ 2 := m ℓ n ℓ , c ℓ := d ℓ ρ ℓ 2 .
The trainable one-dimensional parameters (biases and LayerNorm parameters) form one Euclidean factor R d 0 \mathbb{R}^{d_0} R d 0 ; set c 0 : = 1 c_0 := 1 c 0 := 1 . c ℓ ≥ 1 c_\ell\ge1 c ℓ ≥ 1 , because m ℓ n ℓ − d ℓ = ( m ℓ − r ℓ ) ( n ℓ − r ℓ ) ≥ 0 m_\ell n_\ell-d_\ell=(m_\ell-r_\ell)(n_\ell-r_\ell)\ge0 m ℓ n ℓ − d ℓ = ( m ℓ − r ℓ ) ( n ℓ − r ℓ ) ≥ 0 , with equality if and only if r ℓ = min ( n ℓ , m ℓ ) r_\ell=\min(n_\ell,m_\ell) r ℓ = min ( n ℓ , m ℓ ) .
State and dimensions. The trainable state is
x = ( Δ W 1 , … , Δ W N , b ) ∈ X : = ( ∏ ℓ = 1 N M ℓ ) × R d 0 , x=(\Delta W_1,\ldots,\Delta W_N,b)\in\mathcal{X}:=\left(\prod_{\ell=1}^{N}\mathcal{M}_\ell\right)\times\mathbb{R}^{d_0}, x = ( Δ W 1 , … , Δ W N , b ) ∈ X := ( ℓ = 1 ∏ N M ℓ ) × R d 0 ,
embedded in the ambient Euclidean product
E m i x : = ( ∏ ℓ R m ℓ × n ℓ ) × R d 0 \mathcal{E}_{\mathrm{mix}}:=\left(\prod_{\ell}\mathbb{R}^{m_\ell\times n_\ell}\right)\times\mathbb{R}^{d_0} E mix := ( ℓ ∏ R m ℓ × n ℓ ) × R d 0
with the direct-sum Frobenius inner product. The total ambient trainable dimension is D ‾ : = ∑ ℓ m ℓ n ℓ + d 0 \overline{D}:=\sum_\ell m_\ell n_\ell+d_0 D := ∑ ℓ m ℓ n ℓ + d 0 .
Loss function. F ( x ; ξ ) : = L ( W 0 + Δ W , b ; ξ ) F(x;\xi):=\mathcal{L}(W_0+\Delta W,b;\xi) F ( x ; ξ ) := L ( W 0 + Δ W , b ; ξ ) , defined on an open neighborhood of X \mathcal{X} X in E m i x \mathcal{E}_{\mathrm{mix}} E mix : the finite-difference points below leave X \mathcal{X} X , so F F F must be (and, in the implementation, is) an ambient function. Set f ( x ) : = E ξ F ( x ; ξ ) f(x):=\mathbb{E}_{\xi}F(x;\xi) f ( x ) := E ξ F ( x ; ξ ) .
Tangent spaces and gradients. T x : = ( ∏ ℓ T ℓ ) × R d 0 T_x:=\left(\prod_\ell T_\ell\right)\times\mathbb{R}^{d_0} T x := ( ∏ ℓ T ℓ ) × R d 0 with T ℓ : = T Δ W ℓ M ℓ T_\ell:=T_{\Delta W_\ell}\mathcal{M}_\ell T ℓ := T Δ W ℓ M ℓ , and P ℓ P_\ell P ℓ denotes the orthogonal projector onto T ℓ T_\ell T ℓ from (4) . The Riemannian gradient in the product Frobenius metric is
grad f ( x ) = ( P 1 ∇ 1 f , … , P N ∇ N f , ∇ 0 f ) , \operatorname{grad}f(x)=\left(P_1\nabla_1f,\ldots,P_N\nabla_Nf,\nabla_0f\right), grad f ( x ) = ( P 1 ∇ 1 f , … , P N ∇ N f , ∇ 0 f ) ,
where ∇ ℓ ( ∇ 0 ) \nabla_\ell(\nabla_0) ∇ ℓ ( ∇ 0 ) is the Euclidean partial gradient with respect to block ℓ \ell ℓ (the 1D factor).
Weighted norm. Define the linear map D : = blkdiag ( c 1 I T 1 , … , c N I T N , I d 0 ) D := \operatorname{blkdiag}(c_1 I_{T_1},\ldots,c_N I_{T_N},I_{d_0}) D := blkdiag ( c 1 I T 1 , … , c N I T N , I d 0 ) on T x T_x T x and, for v = ( v 1 , … , v N , v 0 ) ∈ T x v=(v_1,\ldots,v_N,v_0)\in T_x v = ( v 1 , … , v N , v 0 ) ∈ T x , the weighted norm
∥ v ∥ D 2 : = ∑ ℓ c ℓ ∥ v ℓ ∥ F 2 + ∥ v 0 ∥ 2 . \|v\|_D^2 := \sum_{\ell} c_\ell\|v_\ell\|_F^2+\|v_0\|^2. ∥ v ∥ D 2 := ℓ ∑ c ℓ ∥ v ℓ ∥ F 2 + ∥ v 0 ∥ 2 .
Since every c ℓ ≥ 1 c_\ell\ge1 c ℓ ≥ 1 and c 0 = 1 c_0=1 c 0 = 1 , we have ∥ v ∥ D 2 ≥ ∥ v ∥ 2 \|v\|_D^2\ge\|v\|^2 ∥ v ∥ D 2 ≥ ∥ v ∥ 2 (plain product norm).
At step t t t , given x t = ( Δ W 1 , t , … , Δ W L , t , b t ) x_t=(\Delta W_{1,t},\ldots,\Delta W_{L,t},b_t) x t = ( Δ W 1 , t , … , Δ W L , t , b t ) , we restate the proposed algorithm as follows:
Draw Z ℓ ∼ N ( 0 , I m ℓ × n ℓ ) Z_\ell\sim\mathcal{N}(0,I_{m_\ell\times n_\ell}) Z ℓ ∼ N ( 0 , I m ℓ × n ℓ ) i.i.d. across ℓ \ell ℓ and z 0 ∼ N ( 0 , I d 0 ) z_0\sim\mathcal{N}(0,I_{d_0}) z 0 ∼ N ( 0 , I d 0 ) , independent of the past and of ξ t \xi_t ξ t .
Set Z ~ ℓ : = P ℓ Z ℓ \widetilde{Z}_\ell:=P_\ell Z_\ell Z ℓ := P ℓ Z ℓ and Z ~ ℓ : = ρ ℓ Z ^ ℓ / ∥ Z ^ ℓ ∥ F \widetilde{Z}_\ell:=\rho_\ell\widehat{Z}_\ell/\|\widehat{Z}_\ell\|_F Z ℓ := ρ ℓ Z ℓ /∥ Z ℓ ∥ F (almost surely well defined). The joint random direction is u t : = ( Z ~ 1 , … , Z ~ N , z 0 ) ∈ T x t u_t:=(\widetilde{Z}_1,\ldots,\widetilde{Z}_N,z_0)\in T_{x_t} u t := ( Z 1 , … , Z N , z 0 ) ∈ T x t .
Two function evaluations on the ambient line, sharing the same minibatch ξ t \xi_t ξ t : s ^ t : = [ F ( x t + ϵ u t ; ξ t ) − F ( x t − ϵ u t ; ξ t ) ] / ( 2 ϵ ) \widehat{s}_t:=[F(x_t+\epsilon u_t;\xi_t)-F(x_t-\epsilon u_t;\xi_t)]/(2\epsilon) s t := [ F ( x t + ϵ u t ; ξ t ) − F ( x t − ϵ u t ; ξ t )] / ( 2 ϵ ) .
Estimator G ^ t : = s ^ t u t \widehat{G}_t:=\widehat{s}_t u_t G t := s t u t , so G ^ t , ℓ = s ^ t Z ~ ℓ ∈ T ℓ \widehat{G}_{t,\ell}=\widehat{s}_t\widetilde{Z}_\ell\in T_\ell G t , ℓ = s t Z ℓ ∈ T ℓ and G ^ t , 0 = s ^ t z 0 \widehat{G}_{t,0}=\widehat{s}_t z_0 G t , 0 = s t z 0 .
Update: Δ W ℓ , t + 1 = H r ℓ ( Δ W ℓ , t − η G ^ t , ℓ ) \Delta W_{\ell,t+1}=\mathcal{H}_{r_\ell}(\Delta W_{\ell,t}-\eta\widehat{G}_{t,\ell}) Δ W ℓ , t + 1 = H r ℓ ( Δ W ℓ , t − η G t , ℓ ) for each block and b t + 1 = b t − η G ^ t , 0 b_{t+1}=b_t-\eta\widehat{G}_{t,0} b t + 1 = b t − η G t , 0 ; that is, x t + 1 = R x t ( − η G ^ t ) x_{t+1}=\mathcal{R}_{x_t}(-\eta\widehat{G}_t) x t + 1 = R x t ( − η G t ) with the product retraction R = ( H r 1 , … , H r L , id ) \mathcal{R}=(\mathcal{H}_{r_1},\ldots,\mathcal{H}_{r_L},\operatorname{id}) R = ( H r 1 , … , H r L , id ) .
Remark 2 (randomness model). Let F t : = σ ( x 0 , … , x t ) \mathcal{F}_t:=\sigma(x_0,\ldots,x_t) F t := σ ( x 0 , … , x t ) ; the draws ( Z 1 , … , Z N , z 0 , ξ t ) (Z_1,\ldots,Z_N,z_0,\xi_t) ( Z 1 , … , Z N , z 0 , ξ t ) are independent of F t \mathcal{F}_t F t . Three standard idealizations bridge the implementation and the probabilistic model. (i) Per-step seeds come from one pseudorandom stream; we model them as i.i.d. (ii) Assumption 3 treats minibatches as conditionally unbiased given F t \mathcal{F}_t F t , whereas the data loader shuffles a finite dataset and samples without replacement within an epoch; this is the standard stochastic-oracle idealization shared with the MeZO and LOZO analyses. (iii) The common-random-numbers cancellation in Lemma 2 requires the forward pass to be deterministic given ( x , ξ ) (x,\xi) ( x , ξ ) , which holds in the implementation (evaluation mode; no dropout). Conditional expectations below are justified by the finite second moment (Lemma 4) and the F t \mathcal{F}_t F t -measurability of grad f ( x t ) \operatorname{grad} f(x_t) grad f ( x t ) .
Assumptions and technical lemmas
Assumption 1 (trajectory regularity). There exist σ min > 0 \sigma_{\min}>0 σ m i n > 0 , G ∞ < ∞ G_\infty<\infty G ∞ < ∞ , and f ∗ > − ∞ f_* >-\infty f ∗ > − ∞ such that, almost surely along the realized trajectory, finite-difference segments, and update segments: (i) σ r ℓ ( Δ W ℓ , t ) ≥ σ min \sigma_{r_\ell}(\Delta W_{\ell,t})\ge\sigma_{\min} σ r ℓ ( Δ W ℓ , t ) ≥ σ m i n for every block and step; (ii) ∥ ∇ f ∥ ≤ G ∞ \|\nabla f\|\le G_\infty ∥∇ f ∥ ≤ G ∞ ; (iii) f ≥ f ∗ f\ge f_* f ≥ f ∗ ; (iv) η ∥ G ^ t , ℓ ∥ F ≤ σ min / 2 \eta\|\widehat{G}_{t,\ell}\|_F\le\sigma_{\min}/2 η ∥ G t , ℓ ∥ F ≤ σ m i n /2 for every step t t t and 2D block ℓ \ell ℓ .
Assumption 2 (smoothness). F ( ⋅ ; ξ ) F(\cdot;\xi) F ( ⋅ ; ξ ) is differentiable with L L L -Lipschitz gradient, uniformly in ξ \xi ξ , and three times differentiable along the realized finite-difference segments with third directional derivatives bounded: ∣ ϕ ′ ′ ′ ( τ ) ∣ ≤ ρ 3 ∥ v ∥ 3 |\phi'''(\tau)|\le\rho_3\|v\|^3 ∣ ϕ ′′′ ( τ ) ∣ ≤ ρ 3 ∥ v ∥ 3 for ϕ ( τ ) : = F ( x + τ v ; ξ ) \phi(\tau):=F(x+\tau v;\xi) ϕ ( τ ) := F ( x + τ v ; ξ ) .
Assumption 3 (stochastic model). E ξ ∇ F ( x ; ξ ) = ∇ f ( x ) \mathbb{E}_\xi\nabla F(x;\xi)=\nabla f(x) E ξ ∇ F ( x ; ξ ) = ∇ f ( x ) , with blockwise variances E ξ ∥ P ℓ ( ∇ F − ∇ f ) ∥ F 2 ≤ σ ℓ 2 \mathbb{E}_\xi\|\mathcal{P}_\ell(\nabla F-\nabla f)\|_F^2\le\sigma_\ell^2 E ξ ∥ P ℓ ( ∇ F − ∇ f ) ∥ F 2 ≤ σ ℓ 2 and E ξ ∥ ∇ 0 F − ∇ 0 f ∥ 2 ≤ σ 0 2 \mathbb{E}_\xi\|\nabla_0F-\nabla_0f\|^2\le\sigma_0^2 E ξ ∥ ∇ 0 F − ∇ 0 f ∥ 2 ≤ σ 0 2 ; both function evaluations of step t t t share the same minibatch ξ t \xi_t ξ t .
Assumptions 1–3 in Section 5 are used with the following precise domain. Let X r e g ⊂ X \mathcal{X}_{\mathrm{reg}}\subset\mathcal{X} X reg ⊂ X be a set containing all iterates almost surely, and define its enlargement
X r e g ′ : = { x t + τ u t : ∣ τ ∣ ≤ ϵ } ∪ { x t − τ η G ^ t : τ ∈ [ 0 , 1 ] } ∪ { segments between x t − η G ^ t and R x t ( − η G ^ t ) } . (25) \mathcal{X}'_{\mathrm{reg}}:=\{x_t+\tau u_t:|\tau|\le\epsilon\}\cup\{x_t-\tau\eta\widehat{G}_t:\tau\in[0,1]\}
\cup\{\text{segments between }x_t-\eta\widehat{G}_t\text{ and }\mathcal{R}_{x_t}(-\eta\widehat{G}_t)\}. \tag*{(25)} X reg ′ := { x t + τ u t : ∣ τ ∣ ≤ ϵ } ∪ { x t − τ η G t : τ ∈ [ 0 , 1 ]} ∪ { segments between x t − η G t and R x t ( − η G t )} . (25)
over all steps t t t and realized draws. Assumption 1 requires: (i) σ r ℓ ( Δ W ℓ , t ) ≥ σ min > 0 \sigma_{r_\ell}(\Delta W_{\ell,t})\ge\sigma_{\min}>0 σ r ℓ ( Δ W ℓ , t ) ≥ σ m i n > 0 for every block and step on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ ; (ii) ∥ ∇ f ∥ ≤ G ∞ \|\nabla f\|\le G_\infty ∥∇ f ∥ ≤ G ∞ on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ ; (iii) f ≥ f ∗ f\ge f_* f ≥ f ∗ on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ ; (iv) almost surely, η ∥ G ^ t , ℓ ∥ F ≤ σ min / 2 \eta\|\widehat{G}_{t,\ell}\|_F\le\sigma_{\min}/2 η ∥ G t , ℓ ∥ F ≤ σ m i n /2 for every t t t and every 2D block ℓ \ell ℓ . Assumption 2 requires F ( ⋅ ; ξ ) F(\cdot;\xi) F ( ⋅ ; ξ ) differentiable with L L L -Lipschitz gradient on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ , uniformly in ξ \xi ξ , and, along every realized finite-difference segment, ϕ ( τ ) : = F ( x + τ v ; ξ ) \phi(\tau):=F(x+\tau v;\xi) ϕ ( τ ) := F ( x + τ v ; ξ ) three times differentiable with ∣ ϕ ′ ′ ′ ( τ ) ∣ ≤ ρ 3 ∥ v ∥ 3 |\phi'''(\tau)|\le\rho_3\|v\|^3 ∣ ϕ ′′′ ( τ ) ∣ ≤ ρ 3 ∥ v ∥ 3 . Assumption 3 requires E ξ ∇ F ( x ; ξ ) = ∇ f ( x ) \mathbb{E}_\xi\nabla F(x;\xi)=\nabla f(x) E ξ ∇ F ( x ; ξ ) = ∇ f ( x ) on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ and the blockwise variance bounds E ξ ∥ P ℓ ( ∇ F − ∇ f ) ∥ F 2 ≤ σ ℓ 2 \mathbb{E}_\xi\|\mathcal{P}_\ell(\nabla F-\nabla f)\|_F^2\le\sigma_\ell^2 E ξ ∥ P ℓ ( ∇ F − ∇ f ) ∥ F 2 ≤ σ ℓ 2 and E ξ ∥ ∇ 0 F − ∇ 0 f ∥ 2 ≤ σ 0 2 \mathbb{E}_\xi\|\nabla_0F-\nabla_0f\|^2\le\sigma_0^2 E ξ ∥ ∇ 0 F − ∇ 0 f ∥ 2 ≤ σ 0 2 on X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ , with both function evaluations sharing the same ξ t \xi_t ξ t . As a consequence of Assumptions 2 and 3, f = E ξ F f = \mathbb{E}_{\xi}F f = E ξ F is differentiable with L L L -Lipschitz gradient on X r e g ′ \mathcal{X}_{\mathrm{reg}}' X reg ′ (dominated convergence plus uniformity in ξ \xi ξ ); this is used in Lemma 6. Write
σ D 2 : = ∑ ℓ c ℓ ρ ℓ 2 + σ 0 2 , Δ 0 : = f ( x 0 ) − f ∗ . \sigma_D^2 := \sum_{\ell} c_{\ell}\rho_{\ell}^2 + \sigma_0^2,\qquad\Delta_0 := f(x_0)-f_*. σ D 2 := ℓ ∑ c ℓ ρ ℓ 2 + σ 0 2 , Δ 0 := f ( x 0 ) − f ∗ .
Remark 3 (what Assumption 1 does and does not assert). (a) Regular region. Each M ℓ \mathcal{M}_{\ell} M ℓ is noncompact: its closure adds the rank-deficient boundary, on which the tangent spaces and the retraction degenerate. Assumption 1 confines the analysis to a region away from that boundary; it does not prove that the iterates stay there. This is the standard form of assumption in fixed-rank optimization (Vandereycken, [34 ] ; Schneider & Uschmajew, [29 ] ). Clause (i) is checkable because the state stores Σ ℓ \Sigma_{\ell} Σ ℓ explicitly; since the near-zero initialization (19) starts at distance σ i n i t , ℓ \sigma_{\mathrm{init},\ell} σ init , ℓ from the boundary, the clause is most demanding early in training, and Appendix § 11 .15 checks it empirically on our runs. (b) Step regularity. Clause (iv) is not enforced by the algorithm. A deterministic sufficient condition: if ∣ F ∣ ≤ B F |F| \le B_F ∣ F ∣ ≤ B F on X r e g ′ \mathcal{X}_{\mathrm{reg}}' X reg ′ , then ∣ s ^ t ∣ ≤ B F / ϵ |\hat{s}_t| \le B_F/\epsilon ∣ s ^ t ∣ ≤ B F / ϵ and ∥ Z ~ ℓ ∥ F = ρ ℓ \|\widetilde{Z}_{\ell}\|_F = \rho_{\ell} ∥ Z ℓ ∥ F = ρ ℓ , so η ≤ ϵ σ min / ( 2 B F max ℓ ρ ℓ ) \eta\le\epsilon\sigma_{\min}/(2B_F\max_{\ell}\rho_{\ell}) η ≤ ϵ σ m i n / ( 2 B F max ℓ ρ ℓ ) implies (iv). (c) 1D segments. On the Euclidean factor, ∥ G ~ t , 0 ∥ = ∣ s ^ t ∣ ∥ z 0 ∥ \|\widetilde{G}_{t,0}\| = |\hat{s}_t|\|z_0\| ∥ G t , 0 ∥ = ∣ s ^ t ∣∥ z 0 ∥ has no deterministic bound, so uniform constants ( L , ρ 3 , G ∞ ) (L,\rho_3,G_{\infty}) ( L , ρ 3 , G ∞ ) on X r e g ′ \mathcal{X}_{\mathrm{reg}}' X reg ′ are a nontrivial requirement in the 1D directions. Alternatively, one can condition on the event ∥ z 0 ∥ ≤ d 0 + 2 log ( T / δ ) \|z_0\| \le\sqrt{d_0+2\log(T/\delta)} ∥ z 0 ∥ ≤ d 0 + 2 log ( T / δ ) , of probability at least 1 − δ 1-\delta 1 − δ , and absorb its complement into an additive δ \delta δ -term; we state the assumption version for cleanliness.
Lemma 1 (random-direction law and moments). Condition on F t \mathcal{F}_t F t (equivalently, fix the tangent spaces T ℓ T_{\ell} T ℓ ; note T ℓ T_{\ell} T ℓ is a random, F t \mathcal{F}_t F t -measurable subspace — only its dimension d ℓ d_{\ell} d ℓ is deterministic, since Assumption 1(i) keeps every iterate at exact rank r ℓ r_{\ell} r ℓ ). Then Z ^ ℓ = P ℓ Z ℓ \widehat{Z}_{\ell}=P_{\ell}Z_{\ell} Z ℓ = P ℓ Z ℓ is a standard Gaussian on the d ℓ d_{\ell} d ℓ -dimensional subspace T ℓ T_{\ell} T ℓ , u ℓ : = Z ^ ℓ / ∥ Z ^ ℓ ∥ F u_{\ell}:=\widehat{Z}_{\ell}/\|\widehat{Z}_{\ell}\|_F u ℓ := Z ℓ /∥ Z ℓ ∥ F is uniform on the unit sphere of T ℓ T_{\ell} T ℓ , and Z ~ ℓ = ρ ℓ u ℓ \widetilde{Z}_{\ell}=\rho_{\ell}u_{\ell} Z ℓ = ρ ℓ u ℓ is uniform on the sphere of radius ρ ℓ \rho_{\ell} ρ ℓ in T ℓ T_{\ell} T ℓ ; in particular ∥ Z ~ ℓ ∥ F = ρ ℓ \|\widetilde{Z}_{\ell}\|_F=\rho_{\ell} ∥ Z ℓ ∥ F = ρ ℓ deterministically. Conditionally on F t \mathcal{F}_t F t , the factors ( u 1 , … , u N , z 0 ) (u_1,\ldots,u_N,z_0) ( u 1 , … , u N , z 0 ) are mutually independent. For fixed g ∈ T ℓ g\in T_{\ell} g ∈ T ℓ and h ∈ R d 0 h\in\mathbb{R}^{d_0} h ∈ R d 0 :
E [ ⟨ g , u ℓ ⟩ u ℓ ] = g d ℓ , hence E [ ⟨ g , Z ~ ℓ ⟩ Z ~ ℓ ] = c ℓ g , E [ ⟨ g , Z ~ ℓ ⟩ 2 ] = c ℓ ∥ g ∥ F 2 , (26) \mathbb{E}[\langle g,u_{\ell}\rangle u_{\ell}] = \frac{g}{d_{\ell}},\qquad
\text{hence}\qquad
\mathbb{E}[\langle g,\widetilde{Z}_{\ell}\rangle\widetilde{Z}_{\ell}] = c_{\ell}g,\qquad
\mathbb{E}[\langle g,\widetilde{Z}_{\ell}\rangle^2] = c_{\ell}\|g\|_F^2, \tag*{(26)} E [⟨ g , u ℓ ⟩ u ℓ ] = d ℓ g , hence E [⟨ g , Z ℓ ⟩ Z ℓ ] = c ℓ g , E [⟨ g , Z ℓ ⟩ 2 ] = c ℓ ∥ g ∥ F 2 , (26)
E [ ⟨ h , z 0 ⟩ z 0 ] = h , E [ ⟨ h , z 0 ⟩ 2 ] = ∥ h ∥ 2 , E [ ⟨ h , z 0 ⟩ 2 ∥ z 0 ∥ 2 ] = ( d 0 + 2 ) ∥ h ∥ 2 , (27) \mathbb{E}[\langle h,z_0\rangle z_0] = h,\qquad
\mathbb{E}[\langle h,z_0\rangle^2] = \|h\|^2,\qquad
\mathbb{E}[\langle h,z_0\rangle^2\|z_0\|^2] = (d_0+2)\|h\|^2, \tag*{(27)} E [⟨ h , z 0 ⟩ z 0 ] = h , E [⟨ h , z 0 ⟩ 2 ] = ∥ h ∥ 2 , E [⟨ h , z 0 ⟩ 2 ∥ z 0 ∥ 2 ] = ( d 0 + 2 ) ∥ h ∥ 2 , (27)
E ∥ z 0 ∥ 2 k = d 0 ( d 0 + 2 ) ⋯ ( d 0 + 2 k − 2 ) ≤ ( d 0 + 2 k − 2 ) k , k ≥ 1. (28) \mathbb{E}\|z_0\|^{2k} = d_0(d_0+2)\cdots(d_0+2k-2)\le(d_0+2k-2)^k,\qquad k\ge1. \tag*{(28)} E ∥ z 0 ∥ 2 k = d 0 ( d 0 + 2 ) ⋯ ( d 0 + 2 k − 2 ) ≤ ( d 0 + 2 k − 2 ) k , k ≥ 1. (28)
With Q : = ∑ ℓ ρ ℓ 2 Q:=\sum_{\ell}\rho_{\ell}^2 Q := ∑ ℓ ρ ℓ 2 (deterministic), ∥ u t ∥ 2 = Q + ∥ z 0 ∥ 2 \|u_t\|^2=Q+\|z_0\|^2 ∥ u t ∥ 2 = Q + ∥ z 0 ∥ 2 , and consequently
E ∥ u t ∥ 4 ≤ ( D ‾ + 2 ) 2 , E ∥ u t ∥ 8 ≤ ( D ‾ + 6 ) 4 . (29) \mathbb{E}\|u_t\|^4\le(\overline{D}+2)^2,\qquad
\mathbb{E}\|u_t\|^8\le(\overline{D}+6)^4. \tag*{(29)} E ∥ u t ∥ 4 ≤ ( D + 2 ) 2 , E ∥ u t ∥ 8 ≤ ( D + 6 ) 4 . (29)
Proof. The orthogonal projection of an isotropic Gaussian onto a subspace is a standard Gaussian on that subspace, and normalizing a standard Gaussian on T ℓ T_{\ell} T ℓ yields the uniform distribution on its unit sphere; scaling by ρ ℓ \rho_{\ell} ρ ℓ gives the sphere of radius ρ ℓ \rho_{\ell} ρ ℓ . Mutual independence holds because the blocks use independent Gaussians and the projectors are fixed under the conditioning. For the sphere moments, isotropy of u ℓ u_{\ell} u ℓ on T ℓ T_{\ell} T ℓ implies that the covariance operator g ↦ E [ ⟨ g , u ℓ ⟩ u ℓ ] g\mapsto\mathbb{E}[\langle g,u_{\ell}\rangle u_{\ell}] g ↦ E [⟨ g , u ℓ ⟩ u ℓ ] is a multiple of the identity on T ℓ T_{\ell} T ℓ ; taking the trace and using ∥ u ℓ ∥ = 1 \|u_{\ell}\|=1 ∥ u ℓ ∥ = 1 identifies the multiple as 1 / d ℓ 1/d_{\ell} 1/ d ℓ , which gives the first identity in (26) , and taking the inner product with g g g gives the second; multiplying by ρ ℓ 2 = c ℓ d ℓ \rho_{\ell}^2=c_{\ell}d_{\ell} ρ ℓ 2 = c ℓ d ℓ converts both to Z ~ ℓ \widetilde{Z}_{\ell} Z ℓ . The Gaussian identities in (27) are classical; for the third, expand ∑ i , k h i h k E [ z i z k ∥ z ∥ 2 ] \sum_{i,k}h_i h_k\mathbb{E}[z_i z_k\|z\|^2] ∑ i , k h i h k E [ z i z k ∥ z ∥ 2 ] and use E z 1 4 = 3 \mathbb{E}z_1^4=3 E z 1 4 = 3 and E z 1 2 z j 2 = 1 \mathbb{E}z_1^2z_j^2=1 E z 1 2 z j 2 = 1 for j ≠ 1 j\ne1 j = 1 . For (28) , ∥ z 0 ∥ 2 ∼ χ d 0 2 \|z_0\|^2\sim\chi^2_{d_0} ∥ z 0 ∥ 2 ∼ χ d 0 2 has E ( χ d 0 2 ) k = d 0 ( d 0 + 2 ) ⋯ ( d 0 + 2 k − 2 ) \mathbb{E}(\chi^2_{d_0})^k=d_0(d_0+2)\cdots(d_0+2k-2) E ( χ d 0 2 ) k = d 0 ( d 0 + 2 ) ⋯ ( d 0 + 2 k − 2 ) , and each factor is at most d 0 + 2 k − 2 d_0+2k-2 d 0 + 2 k − 2 . For (29) , expand E ( Q + X ) k \mathbb{E}(Q+X)^k E ( Q + X ) k with X : = ∥ z 0 ∥ 2 X:=\|z_0\|^2 X := ∥ z 0 ∥ 2 binomially; each E X j ≤ ( d 0 + 2 k − 2 ) j \mathbb{E}X^j\le(d_0+2k-2)^j E X j ≤ ( d 0 + 2 k − 2 ) j for j ≤ k j\le k j ≤ k , so the binomial sum is at most ( Q + d 0 + 2 k − 2 ) k = ( D ‾ + 2 k − 2 ) k (Q+d_0+2k-2)^k=(\overline{D}+2k-2)^k ( Q + d 0 + 2 k − 2 ) k = ( D + 2 k − 2 ) k ; take k = 2 k=2 k = 2 and k = 4 k=4 k = 4 . □ \square □
Lemma 2 (two-point bias). Under Assumption 2, with s t l i n : = ⟨ ∇ F ( x t ; ξ t ) , u t ⟩ s_t^{\mathrm{lin}}:=\langle\nabla F(x_t;\xi_t),u_t\rangle s t lin := ⟨ ∇ F ( x t ; ξ t ) , u t ⟩ ,
∣ s ^ t − s t l i n ∣ ≤ β 3 6 ϵ 2 ∥ u t ∥ 3 . (30) |\hat{s}_t-s_t^{\mathrm{lin}}|\le\frac{\beta_3}{6}\epsilon^2\|u_t\|^3. \tag*{(30)} ∣ s ^ t − s t lin ∣ ≤ 6 β 3 ϵ 2 ∥ u t ∥ 3 . (30)
Proof. By Assumption 2, ϕ ( τ ) : = F ( x t + τ u t ; ξ t ) \phi(\tau):=F(x_t+\tau u_t;\xi_t) ϕ ( τ ) := F ( x t + τ u t ; ξ t ) is three times differentiable on [ − ϵ , ϵ ] [-\epsilon,\epsilon] [ − ϵ , ϵ ] (it is C 2 C^2 C 2 with ϕ ′ ′ ′ \phi''' ϕ ′′′ existing pointwise, which suffices for the Lagrange remainder). Taylor expansion gives ϕ ( ± ϵ ) = ϕ ( 0 ) ± ϵ ϕ ′ ( 0 ) + ϵ 2 2 ϕ ′ ′ ( 0 ) ± ϵ 3 6 ϕ ′ ′ ′ ( τ ± ) \phi(\pm\epsilon)=\phi(0)\pm\epsilon\phi'(0)+\frac{\epsilon^2}{2}\phi''(0)\pm\frac{\epsilon^3}{6}\phi'''(\tau_{\pm}) ϕ ( ± ϵ ) = ϕ ( 0 ) ± ϵ ϕ ′ ( 0 ) + 2 ϵ 2 ϕ ′′ ( 0 ) ± 6 ϵ 3 ϕ ′′′ ( τ ± ) for some τ ± ∈ ( − ϵ , ϵ ) \tau_{\pm}\in(-\epsilon,\epsilon) τ ± ∈ ( − ϵ , ϵ ) . The symmetric difference cancels ϕ ′ ′ ( 0 ) \phi''(0) ϕ ′′ ( 0 ) : s ^ t = ϕ ′ ( 0 ) + ϵ 2 12 ( ϕ ′ ′ ′ ( τ + ) + ϕ ′ ′ ′ ( τ − ) ) \hat{s}_t=\phi'(0)+\frac{\epsilon^2}{12}(\phi'''(\tau_+)+\phi'''(\tau_-)) s ^ t = ϕ ′ ( 0 ) + 12 ϵ 2 ( ϕ ′′′ ( τ + ) + ϕ ′′′ ( τ − )) , and ∣ ϕ ′ ′ ′ ∣ ≤ β 3 ∥ u t ∥ 3 |\phi'''|\le\beta_3\|u_t\|^3 ∣ ϕ ′′′ ∣ ≤ β 3 ∥ u t ∥ 3 . □ \square □
Remark 4 (structure of the bias). The constant 1 / 6 1/6 1/6 is sharp: it is attained by ϕ ( τ ) = τ 3 \phi(\tau)=\tau^3 ϕ ( τ ) = τ 3 . Because the implementation evaluates the loss on the ambient tangent line rather than along a retracted curve, no retraction curvature enters this bound: it is the classical Euclidean symmetric-difference bound. First-order geometry (tangent projection, d ℓ d_\ell d ℓ , c ℓ c_\ell c ℓ ) enters the analysis through the random-direction law of Lemma 1; second-order manifold geometry enters only through the retraction step (Lemmas 5 and 6). Note also that the single scalar s ^ t \hat{s}_t s ^ t carries the finite-difference bias of all N + 1 N+1 N + 1 factors at once — the coupling induced by the shared two-point evaluation — which is why the bias constant in Lemma 3 involves the total dimension D ˉ \bar{D} D ˉ rather than a per-block quantity.
Lemma 3 (first moment). Under Assumptions 1–3,
E [ G ^ t ∣ F t ] = D g r a d f ( x t ) + β t , ∥ β t ∥ ≤ B ε : = ρ 3 6 ε 2 ( D ˉ + 2 ) 2 . (31) \mathbb{E}[\widehat{G}_t\mid\mathcal{F}_t]=D\mathop{\rm grad} f(x_t)+\beta_t,\qquad\|\beta_t\|\le B_\varepsilon:=\frac{\rho_3}{6}\varepsilon^2(\bar{D}+2)^2. \tag*{(31)} E [ G t ∣ F t ] = D grad f ( x t ) + β t , ∥ β t ∥ ≤ B ε := 6 ρ 3 ε 2 ( D ˉ + 2 ) 2 . (31)
Proof. Condition on F t \mathcal{F}_t F t and ξ t \xi_t ξ t . Write s ^ t = s t lin + δ t \hat{s}_t=s_t^{\operatorname{lin}}+\delta_t s ^ t = s t lin + δ t with ∣ δ t ∣ ≤ ρ 3 6 ε 2 ∥ u t ∥ 3 |\delta_t|\le\frac{\rho_3}{6}\varepsilon^2\|u_t\|^3 ∣ δ t ∣ ≤ 6 ρ 3 ε 2 ∥ u t ∥ 3 by Lemma 2, and decompose s t lin = ∑ k = 1 N a k + a 0 s_t^{\operatorname{lin}}=\sum_{k=1}^{N}a_k+a_0 s t lin = ∑ k = 1 N a k + a 0 with a k : = ⟨ P k ∇ k F , Z ~ k ⟩ a_k:=\langle P_k\nabla_kF,\widetilde{Z}_k\rangle a k := ⟨ P k ∇ k F , Z k ⟩ (tangency of Z ~ k \widetilde{Z}_k Z k lets us insert P k P_k P k ) and a 0 : = ⟨ ∇ 0 F , z 0 ⟩ a_0:=\langle\nabla_0F,z_0\rangle a 0 := ⟨ ∇ 0 F , z 0 ⟩ .
Component ℓ \ell ℓ . E [ s t lin Z ~ ℓ ] = E [ a ℓ Z ~ ℓ ] \mathbb{E}[s_t^{\operatorname{lin}}\widetilde{Z}_\ell]=\mathbb{E}[a_\ell\widetilde{Z}_\ell] E [ s t lin Z ℓ ] = E [ a ℓ Z ℓ ] : for k ≠ ℓ k\ne\ell k = ℓ , E [ a k Z ~ ℓ ] = E [ a k ] E [ Z ~ ℓ ] \mathbb{E}[a_k\widetilde{Z}_\ell]=\mathbb{E}[a_k]\mathbb{E}[\widetilde{Z}_\ell] E [ a k Z ℓ ] = E [ a k ] E [ Z ℓ ] by conditional independence (Lemma 1), and both factors are zero-mean, so the cross terms vanish. Lemma 1 then gives E [ a ℓ Z ~ ℓ ] = c ℓ P ℓ ∇ ℓ F \mathbb{E}[a_\ell\widetilde{Z}_\ell]=c_\ell P_\ell\nabla_\ell F E [ a ℓ Z ℓ ] = c ℓ P ℓ ∇ ℓ F . Component 0: E [ s t lin z 0 ] = ∇ 0 F \mathbb{E}[s_t^{\operatorname{lin}}z_0]=\nabla_0F E [ s t lin z 0 ] = ∇ 0 F by the same argument. Hence E [ s t lin u t ] = D g r a d F ( x t ; ξ t ) \mathbb{E}[s_t^{\operatorname{lin}}u_t]=D\mathop{\rm grad}F(x_t;\xi_t) E [ s t lin u t ] = D grad F ( x t ; ξ t ) .
Bias. ∥ E [ δ t u t ] ∥ ≤ E [ ∣ δ t ∣ ∥ u t ∥ ] ≤ ρ 3 6 ε 2 E ∥ u t ∥ 4 ≤ ρ 3 6 ε 2 ( D ˉ + 2 ) 2 \|\mathbb{E}[\delta_tu_t]\|\le\mathbb{E}[|\delta_t|\|u_t\|]\le\frac{\rho_3}{6}\varepsilon^2\mathbb{E}\|u_t\|^4\le\frac{\rho_3}{6}\varepsilon^2(\bar{D}+2)^2 ∥ E [ δ t u t ] ∥ ≤ E [ ∣ δ t ∣∥ u t ∥ ] ≤ 6 ρ 3 ε 2 E ∥ u t ∥ 4 ≤ 6 ρ 3 ε 2 ( D ˉ + 2 ) 2 by Lemma 1. Finally take E ξ \mathbb{E}_{\xi} E ξ , using Assumption 3 and P ℓ E ξ ∇ ℓ F = P ℓ ∇ ℓ f P_\ell\mathbb{E}_{\xi}\nabla_\ell F=P_\ell\nabla_\ell f P ℓ E ξ ∇ ℓ F = P ℓ ∇ ℓ f .
Remark 5 (interpretation). The implemented estimator is an (ε \varepsilon ε -biased) stochastic preconditioned Riemannian gradient: the preconditioner D D D has per-factor gains c ℓ = m ℓ n ℓ / d ℓ ≥ 1 c_\ell=m_\ell n_\ell/d_\ell\ge1 c ℓ = m ℓ n ℓ / d ℓ ≥ 1 on 2 2 2 D blocks and exactly 1 1 1 on the 1 1 1 D factor. The post-projection normalization to ∥ Z ~ ℓ ∥ F = m ℓ n ℓ \|\widetilde{Z}_\ell\|_F=\sqrt{m_\ell n_\ell} ∥ Z ℓ ∥ F = m ℓ n ℓ is what produces c ℓ c_\ell c ℓ ; the single-block specialization has N = 1 N=1 N = 1 and d 0 = 0 d_0=0 d 0 = 0 , for which D ˉ = m n \bar{D}=mn D ˉ = mn and the bias bound reads ( ρ 3 / 6 ) ε 2 ( m n ) 2 (\rho_3/6)\varepsilon^2(mn)^2 ( ρ 3 /6 ) ε 2 ( mn ) 2 since ∥ u t ∥ = m n \|u_t\|=\sqrt{mn} ∥ u t ∥ = mn is deterministic — the ( D ˉ + 2 ) 2 (\bar{D}+2)^2 ( D ˉ + 2 ) 2 constant is not needed in that case. D D D is a fixed, deterministic, dimension-derived rescaling; it involves no tuning.
Lemma 4 (second moment). Under Assumptions 1–3,
E [ ∥ G ^ t ∥ 2 ∣ F t ] ≤ 2 ( D ˉ + 2 ) ( ∥ g r a d f ( x t ) ∥ D 2 + σ D 2 ) + ρ 3 2 18 ε 4 ( D ˉ + 6 ) 4 . (32) \mathbb{E}[\|\widehat{G}_t\|^2\mid\mathcal{F}_t]\le2(\bar{D}+2)\left(\|\mathop{\rm grad} f(x_t)\|_D^2+\sigma_D^2\right)+\frac{\rho_3^2}{18}\varepsilon^4(\bar{D}+6)^4. \tag*{(32)} E [ ∥ G t ∥ 2 ∣ F t ] ≤ 2 ( D ˉ + 2 ) ( ∥ grad f ( x t ) ∥ D 2 + σ D 2 ) + 18 ρ 3 2 ε 4 ( D ˉ + 6 ) 4 . (32)
Proof. E [ ∥ G ^ t ∥ 2 ] = E [ s ^ t 2 ∥ u t ∥ 2 ] \mathbb{E}[\|\widehat{G}_t\|^2]=\mathbb{E}[\hat{s}_t^2\|u_t\|^2] E [ ∥ G t ∥ 2 ] = E [ s ^ t 2 ∥ u t ∥ 2 ] and s ^ t 2 ≤ 2 ( s t lin ) 2 + 2 δ t 2 \hat{s}_t^2\le2(s_t^{\operatorname{lin}})^2+2\delta_t^2 s ^ t 2 ≤ 2 ( s t lin ) 2 + 2 δ t 2 . Condition on F t \mathcal{F}_t F t and ξ t \xi_t ξ t throughout; write s t lin = A + a 0 s_t^{\operatorname{lin}}=A+a_0 s t lin = A + a 0 with A : = ∑ ℓ a ℓ A:=\sum_\ell a_\ell A := ∑ ℓ a ℓ , which is conditionally independent of z 0 z_0 z 0 .
(i) With ∥ u t ∥ 2 = Q + ∥ z 0 ∥ 2 \|u_t\|^2=Q+\|z_0\|^2 ∥ u t ∥ 2 = Q + ∥ z 0 ∥ 2 ,
E [ ( s t lin ) 2 ( Q + ∥ z 0 ∥ 2 ) ] = Q E [ ( s t lin ) 2 ] + E [ A 2 ] E ∥ z 0 ∥ 2 + 2 E [ A ] E [ a 0 ∥ z 0 ∥ 2 ] + E [ a 0 2 ∥ z 0 ∥ 2 ] = Q S + d 0 ∑ ℓ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 + 0 + ( d 0 + 2 ) ∥ ∇ 0 F ∥ 2 ≤ ( Q + d 0 + 2 ) S = ( D ˉ + 2 ) S , \begin{aligned}
\mathbb{E}[(s_t^{\operatorname{lin}})^2(Q+\|z_0\|^2)]=Q\mathbb{E}[(s_t^{\operatorname{lin}})^2]+\mathbb{E}[A^2]\mathbb{E}\|z_0\|^2+2\mathbb{E}[A]\mathbb{E}[a_0\|z_0\|^2]+\mathbb{E}[a_0^2\|z_0\|^2] \\
&=QS+d_0\sum_\ell c_\ell\|P_\ell\nabla_\ell F\|_F^2+0+(d_0+2)\|\nabla_0F\|^2 \\
&\le(Q+d_0+2)S=(\bar{D}+2)S,
\end{aligned} E [( s t lin ) 2 ( Q + ∥ z 0 ∥ 2 )] = Q E [( s t lin ) 2 ] + E [ A 2 ] E ∥ z 0 ∥ 2 + 2 E [ A ] E [ a 0 ∥ z 0 ∥ 2 ] + E [ a 0 2 ∥ z 0 ∥ 2 ] = QS + d 0 ℓ ∑ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 + 0 + ( d 0 + 2 ) ∥ ∇ 0 F ∥ 2 ≤ ( Q + d 0 + 2 ) S = ( D ˉ + 2 ) S ,
where S : = E [ ( s t lin ) 2 ] = ∑ ℓ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 + ∥ ∇ 0 F ∥ 2 = ∥ g r a d F ( x t ; ξ t ) ∥ D 2 S:=\mathbb{E}[(s_t^{\operatorname{lin}})^2]=\sum_\ell c_\ell\|P_\ell\nabla_\ell F\|_F^2+\|\nabla_0F\|^2=\|\mathop{\rm grad}F(x_t;\xi_t)\|_D^2 S := E [( s t lin ) 2 ] = ∑ ℓ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 + ∥ ∇ 0 F ∥ 2 = ∥ grad F ( x t ; ξ t ) ∥ D 2 (Lemma 1; cross terms vanish as in Lemma 3, and E [ A ] = 0 \mathbb{E}[A]=0 E [ A ] = 0 kills the middle term). The final inequality follows by merging the last two terms: since ∑ ℓ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 = S − ∥ ∇ 0 F ∥ 2 \sum_\ell c_\ell\|P_\ell\nabla_\ell F\|_F^2=S-\|\nabla_0F\|^2 ∑ ℓ c ℓ ∥ P ℓ ∇ ℓ F ∥ F 2 = S − ∥ ∇ 0 F ∥ 2 , the sum equals Q S + d 0 S + 2 ∥ ∇ 0 F ∥ 2 ≤ ( Q + d 0 + 2 ) S QS+d_0S+2\|\nabla_0F\|^2\le(Q+d_0+2)S QS + d 0 S + 2∥ ∇ 0 F ∥ 2 ≤ ( Q + d 0 + 2 ) S , using ∥ ∇ 0 F ∥ 2 ≤ S \|\nabla_0F\|^2\le S ∥ ∇ 0 F ∥ 2 ≤ S .
(ii) E [ δ t 2 ∥ u t ∥ 2 ] ≤ ( ρ 3 6 ) 2 ε 4 E ∥ u t ∥ 8 ≤ ρ 3 2 36 ε 4 ( D ˉ + 6 ) 4 \mathbb{E}[\delta_t^2\|u_t\|^2]\le\left(\frac{\rho_3}{6}\right)^2\varepsilon^4\mathbb{E}\|u_t\|^8\le\frac{\rho_3^2}{36}\varepsilon^4(\bar{D}+6)^4 E [ δ t 2 ∥ u t ∥ 2 ] ≤ ( 6 ρ 3 ) 2 ε 4 E ∥ u t ∥ 8 ≤ 36 ρ 3 2 ε 4 ( D ˉ + 6 ) 4 by Lemmas 2 and 1.
Combining, E [ ∥ G ^ t ∥ 2 ] ≤ 2 ( D ˉ + 2 ) E ξ ∥ g r a d F ∥ D 2 + ρ 3 2 18 ε 4 ( D ˉ + 6 ) 4 \mathbb{E}[\|\widehat{G}_t\|^2]\le2(\bar{D}+2)\mathbb{E}_{\xi}\|\mathop{\rm grad}F\|_D^2+\frac{\rho_3^2}{18}\varepsilon^4(\bar{D}+6)^4 E [ ∥ G t ∥ 2 ] ≤ 2 ( D ˉ + 2 ) E ξ ∥ grad F ∥ D 2 + 18 ρ 3 2 ε 4 ( D ˉ + 6 ) 4 . Finally, per factor, E ξ ∥ P ℓ ∇ ℓ F ∥ F 2 = ∥ P ℓ ∇ ℓ f ∥ F 2 + E ξ ∥ P ℓ ( ∇ ℓ F − ∇ ℓ f ) ∥ F 2 \mathbb{E}_{\xi}\|P_\ell\nabla_\ell F\|_F^2=\|P_\ell\nabla_\ell f\|_F^2+\mathbb{E}_{\xi}\|P_\ell(\nabla_\ell F-\nabla_\ell f)\|_F^2 E ξ ∥ P ℓ ∇ ℓ F ∥ F 2 = ∥ P ℓ ∇ ℓ f ∥ F 2 + E ξ ∥ P ℓ ( ∇ ℓ F − ∇ ℓ f ) ∥ F 2 (the cross term vanishes by Assumption 3 since E ξ P ℓ ( ∇ ℓ F − ∇ ℓ f ) = 0 \mathbb{E}_{\xi}P_\ell(\nabla_\ell F-\nabla_\ell f)=0 E ξ P ℓ ( ∇ ℓ F − ∇ ℓ f ) = 0 ), so E ξ ∥ g r a d F ∥ D 2 ≤ ∥ g r a d f ∥ D 2 + σ D 2 \mathbb{E}_{\xi}\|\mathop{\rm grad}F\|_D^2\le\|\mathop{\rm grad}f\|_D^2+\sigma_D^2 E ξ ∥ grad F ∥ D 2 ≤ ∥ grad f ∥ D 2 + σ D 2 .
Lemma 5 (retraction error). Let X ∈ M ℓ X \in\mathcal{M}_{\ell} X ∈ M ℓ with σ r ℓ ( X ) ≥ σ min \sigma_{r_{\ell}}(X) \ge\sigma_{\min} σ r ℓ ( X ) ≥ σ m i n and Y ∈ T X M ℓ Y \in T_X\mathcal{M}_{\ell} Y ∈ T X M ℓ with ∥ Y ∥ F ≤ σ min / 2 \lVert Y\rVert_F \le\sigma_{\min}/2 ∥ Y ∥ F ≤ σ m i n /2 . Assume m ℓ , n ℓ ≥ 2 r ℓ m_{\ell}, n_{\ell} \ge2r_{\ell} m ℓ , n ℓ ≥ 2 r ℓ (our settings have m ℓ , n ℓ ≥ 2048 ≫ 2 r ℓ m_{\ell}, n_{\ell} \ge2048 \gg2r_{\ell} m ℓ , n ℓ ≥ 2048 ≫ 2 r ℓ ; the degenerate case is discussed in the proof). Then
∥ H r ℓ ( X + Y ) − ( X + Y ) ∥ F ≤ ∥ Y ∥ F 2 σ min . (33) \left\lVert\mathcal{H}_{r_{\ell}}(X+Y)-(X+Y)\right\rVert_F \le\frac{\lVert Y\rVert_F^2}{\sigma_{\min}}. \tag*{(33)} ∥ H r ℓ ( X + Y ) − ( X + Y ) ∥ F ≤ σ m i n ∥ Y ∥ F 2 . (33) On the Euclidean factor the retraction is the identity, with zero error.
Proof. Drop the block index. Write X = U Σ V ⊤ X=U\Sigma V^{\top} X = U Σ V ⊤ (thin SVD) and decompose the tangent vector as Y = U M V ⊤ + U p V ⊤ + U V p ⊤ Y=UMV^{\top}+U_pV^{\top}+UV_p^{\top} Y = U M V ⊤ + U p V ⊤ + U V p ⊤ with U ⊤ U p = 0 U^{\top}U_p=0 U ⊤ U p = 0 and V ⊤ V p = 0 V^{\top}V_p=0 V ⊤ V p = 0 . The three components are mutually Frobenius-orthogonal (all cross traces vanish via U ⊤ U p = 0 U^{\top}U_p=0 U ⊤ U p = 0 , V ⊤ V p = 0 V^{\top}V_p=0 V ⊤ V p = 0 ), so
∥ Y ∥ F 2 = ∥ M ∥ F 2 + ∥ U p V ⊤ ∥ F 2 + ∥ U V p ⊤ ∥ F 2 . (34) \lVert Y\rVert_F^2=\lVert M\rVert_F^2+\lVert U_pV^{\top}\rVert_F^2+\lVert UV_p^{\top}\rVert_F^2. \tag*{(34)} ∥ Y ∥ F 2 = ∥ M ∥ F 2 + ∥ U p V ⊤ ∥ F 2 + ∥ U V p ⊤ ∥ F 2 . (34)
Choose Q U Q_U Q U as any r r r orthonormal columns of col ( U ) ⊥ \operatorname{col}(U)^\perp col ( U ) ⊥ whose span contains col ( U p ) \operatorname{col}(U_p) col ( U p ) (if m < 2 r m<2r m < 2 r , the second block shrinks to dim ≤ m − r \dim\le m-r dim ≤ m − r columns and the argument below is unchanged); choose Q V Q_V Q V analogously. In the extended bases [ U , Q U ] [U,Q_U] [ U , Q U ] and [ V , Q V ] [V,Q_V] [ V , Q V ] ,
X + Y = [ U , Q U ] K [ V , Q V ] ⊤ , K = [ Σ + M R 1 R 2 0 ] ∈ R 2 r × 2 r , X+Y=[U,Q_U]K[V,Q_V]^{\top},\qquad K=\begin{bmatrix}\Sigma+M & R_1\\ R_2 & 0\end{bmatrix}\in\mathbb{R}^{2r\times2r}, X + Y = [ U , Q U ] K [ V , Q V ] ⊤ , K = [ Σ + M R 2 R 1 0 ] ∈ R 2 r × 2 r ,
with ∥ R 1 ∥ F = ∥ U V p ⊤ ∥ F \lVert R_1\rVert_F=\lVert UV_p^{\top}\rVert_F ∥ R 1 ∥ F = ∥ U V p ⊤ ∥ F and ∥ R 2 ∥ F = ∥ U p V ⊤ ∥ F \lVert R_2\rVert_F=\lVert U_pV^{\top}\rVert_F ∥ R 2 ∥ F = ∥ U p V ⊤ ∥ F (the orthonormal-basis factors preserve the Frobenius norms of the two normal components). Whenever U p U_p U p and V p V_p V p have full column rank r r r —an almost-sure event for the Gaussian-derived tangent random matrices—this K K K is exactly the 2 r × 2 r 2r\times2r 2 r × 2 r core assembled by the efficient retraction of § 4.3 .
Since ∥ M ∥ 2 ≤ ∥ M ∥ F ≤ ∥ Y ∥ F ≤ σ min / 2 \lVert M\rVert_2\le\lVert M\rVert_F\le\lVert Y\rVert_F\le\sigma_{\min}/2 ∥ M ∥ 2 ≤ ∥ M ∥ F ≤ ∥ Y ∥ F ≤ σ m i n /2 , Weyl’s inequality gives σ min ( Σ + M ) ≥ σ min / 2 \sigma_{\min}(\Sigma+M)\ge\sigma_{\min}/2 σ m i n ( Σ + M ) ≥ σ m i n /2 , i.e. ∥ ( Σ + M ) − 1 ∥ 2 ≤ 2 / σ min \lVert(\Sigma+M)^{-1}\rVert_2\le2/\sigma_{\min} ∥( Σ + M ) − 1 ∥ 2 ≤ 2/ σ m i n . Define the rank-≤ r \le r ≤ r competitor
K 0 : = [ R 2 ( Σ + M ) − 1 ] ( Σ + M ) [ I ( Σ + M ) − 1 R 1 ] = [ Σ + M R 1 R 2 R 2 ( Σ + M ) − 1 R 1 ] . K_0:=\left[R_2(\Sigma+M)^{-1}\right](\Sigma+M)\left[I\quad(\Sigma+M)^{-1}R_1\right]=\begin{bmatrix}\Sigma+M & R_1\\ R_2 & R_2(\Sigma+M)^{-1}R_1\end{bmatrix}. K 0 := [ R 2 ( Σ + M ) − 1 ] ( Σ + M ) [ I ( Σ + M ) − 1 R 1 ] = [ Σ + M R 2 R 1 R 2 ( Σ + M ) − 1 R 1 ] .
Because H r ( K ) \mathcal{H}_r(K) H r ( K ) is the best rank-r r r approximation of K K K in Frobenius norm and K 0 K_0 K 0 has rank at most r r r ,
∥ K − H r ( K ) ∥ F ≤ ∥ K − K 0 ∥ F = ∥ R 2 ( Σ + M ) − 1 R 1 ∥ F ≤ ∥ R 2 ∥ F ∥ ( Σ + M ) − 1 ∥ 2 ∥ R 1 ∥ F ≤ 2 σ min ∥ R 1 ∥ F ∥ R 2 ∥ F . \begin{aligned}
\lVert K-\mathcal{H}_r(K)\rVert_F
&\le\lVert K-K_0\rVert_F
=\left\lVert R_2(\Sigma+M)^{-1}R_1\right\rVert_F\\
&\le\lVert R_2\rVert_F\lVert(\Sigma+M)^{-1}\rVert_2\lVert R_1\rVert_F
\le\frac{2}{\sigma_{\min}}\lVert R_1\rVert_F\lVert R_2\rVert_F.
\end{aligned} ∥ K − H r ( K ) ∥ F ≤ ∥ K − K 0 ∥ F = R 2 ( Σ + M ) − 1 R 1 F ≤ ∥ R 2 ∥ F ∥( Σ + M ) − 1 ∥ 2 ∥ R 1 ∥ F ≤ σ m i n 2 ∥ R 1 ∥ F ∥ R 2 ∥ F .
By the AM–GM inequality and (34) , ∥ R 1 ∥ F ∥ R 2 ∥ F ≤ 1 2 ( ∥ R 1 ∥ F 2 + ∥ R 2 ∥ F 2 ) ≤ 1 2 ∥ Y ∥ F 2 \lVert R_1\rVert_F\lVert R_2\rVert_F\le\frac{1}{2}(\lVert R_1\rVert_F^2+\lVert R_2\rVert_F^2)\le\frac{1}{2}\lVert Y\rVert_F^2 ∥ R 1 ∥ F ∥ R 2 ∥ F ≤ 2 1 (∥ R 1 ∥ F 2 + ∥ R 2 ∥ F 2 ) ≤ 2 1 ∥ Y ∥ F 2 , so ∥ K − H r ( K ) ∥ F ≤ ∥ Y ∥ F 2 / σ min \lVert K-\mathcal{H}_r(K)\rVert_F\le\lVert Y\rVert_F^2/\sigma_{\min} ∥ K − H r ( K ) ∥ F ≤ ∥ Y ∥ F 2 / σ m i n . Since [ U , Q U ] [U,Q_U] [ U , Q U ] and [ V , Q V ] [V,Q_V] [ V , Q V ] have orthonormal columns, X + Y X+Y X + Y and K K K share the same nonzero singular values, so their best rank-r r r approximation errors coincide, which transfers the bound from K K K to X + Y X+Y X + Y .
Under Assumption 1(iv), the radius condition ∥ η G ^ t , ℓ ∥ F ≤ σ min / 2 \lVert\eta\widehat{G}_{t,\ell}\rVert_F\le\sigma_{\min}/2 ∥ η G t , ℓ ∥ F ≤ σ m i n /2 holds at every step and 2D block, so Lemma 5 applies to every realized update.
Lemma 6 (descent inequality). Under Assumptions 1–3, with L R : = L + 2 G ∞ / σ min L_R:=L+2G_{\infty}/\sigma_{\min} L R := L + 2 G ∞ / σ m i n , for every step t t t , almost surely,
f ( x t + 1 ) ≤ f ( x t ) − η ⟨ grad f ( x t ) , G ^ t ⟩ + L R 2 η 2 ∥ G ^ t ∥ 2 . (35) f(x_{t+1})\le f(x_t)-\eta\langle\operatorname{grad} f(x_t),\widehat{G}_t\rangle+\frac{L_R}{2}\eta^2\lVert\widehat{G}_t\rVert^2. \tag*{(35)} f ( x t + 1 ) ≤ f ( x t ) − η ⟨ grad f ( x t ) , G t ⟩ + 2 L R η 2 ∥ G t ∥ 2 . (35)
Proof. (i) L L L -smoothness of f f f (the consequence of Assumptions 2 and 3 noted in Appendix § 10 .§ 2 ) along the ambient update segment [ x t , x t − η G ^ t ] ⊂ X r e g ′ [x_t,x_t-\eta\widehat{G}_t]\subset\mathcal{X}'_{\mathrm{reg}} [ x t , x t − η G t ] ⊂ X reg ′ gives f ( x t − η G ^ t ) ≤ f ( x t ) − η ⟨ ∇ f ( x t ) , G ^ t ⟩ + L 2 η 2 ∥ G ^ t ∥ 2 f(x_t-\eta\widehat{G}_t)\le f(x_t)-\eta\langle\nabla f(x_t),\widehat{G}_t\rangle+\frac{L}{2}\eta^2\lVert\widehat{G}_t\rVert^2 f ( x t − η G t ) ≤ f ( x t ) − η ⟨ ∇ f ( x t ) , G t ⟩ + 2 L η 2 ∥ G t ∥ 2 ; every component of G ^ t \widehat{G}_t G t is tangent, so ⟨ ∇ f , G ^ t ⟩ = ⟨ grad f , G ^ t ⟩ \langle\nabla f,\widehat{G}_t\rangle=\langle\operatorname{grad} f,\widehat{G}_t\rangle ⟨ ∇ f , G t ⟩ = ⟨ grad f , G t ⟩ . (ii) Retraction correction:
∥ x t + 1 − ( x t − η G ^ t ) ∥ = ( ∑ ℓ ∥ H r ℓ ( Δ W ℓ , t − η G ^ t , ℓ ) − ( Δ W ℓ , t − η G ^ t , ℓ ) ∥ F 2 ) 1 / 2 ≤ η 2 σ min ( ∑ ℓ ∥ G ^ t , ℓ ∥ F 4 ) 1 / 2 ≤ η 2 σ min ∥ G ^ t ∥ 2 , \begin{aligned}
\left\lVert x_{t+1}-(x_t-\eta\widehat{G}_t)\right\rVert
=\left(\sum_{\ell}\left\lVert\mathcal{H}_{r_{\ell}}(\Delta W_{\ell,t}-\eta\widehat{G}_{t,\ell})-(\Delta W_{\ell,t}-\eta\widehat{G}_{t,\ell})\right\rVert_F^2\right)^{1/2} \\
&\le\frac{\eta^2}{\sigma_{\min}}\left(\sum_{\ell}\lVert\widehat{G}_{t,\ell}\rVert_F^4\right)^{1/2}
\le\frac{\eta^2}{\sigma_{\min}}\lVert\widehat{G}_t\rVert^2,
\end{aligned} x t + 1 − ( x t − η G t ) = ( ℓ ∑ H r ℓ ( Δ W ℓ , t − η G t , ℓ ) − ( Δ W ℓ , t − η G t , ℓ ) F 2 ) 1/2 ≤ σ m i n η 2 ( ℓ ∑ ∥ G t , ℓ ∥ F 4 ) 1/2 ≤ σ m i n η 2 ∥ G t ∥ 2 ,
using Lemma 5 with Assumption 1(iv), and ∑ ℓ a ℓ 2 ≤ ∑ ℓ a ℓ \sqrt{\sum_{\ell}a_{\ell}^{2}}\leq\sum_{\ell}a_{\ell} ∑ ℓ a ℓ 2 ≤ ∑ ℓ a ℓ for a ℓ ≥ 0 a_{\ell}\geq0 a ℓ ≥ 0 . The mean value theorem on the third branch of X r e g ′ \mathcal{X}'_{\mathrm{reg}} X reg ′ with Assumption 1(ii) gives f ( x t + 1 ) − f ( x t − η G ^ t ) ≤ G ∞ ( η 2 / σ min ) ∥ G ^ t ∥ 2 f(x_{t+1})-f(x_t-\eta\widehat{G}_t)\leq G_{\infty}(\eta^{2}/\sigma_{\min})\|\widehat{G}_t\|^{2} f ( x t + 1 ) − f ( x t − η G t ) ≤ G ∞ ( η 2 / σ m i n ) ∥ G t ∥ 2 . Summing (i) and (ii) yields the coefficient L / 2 + G ∞ / σ min = L R / 2 L/2+G_{\infty}/\sigma_{\min}=L_R/2 L /2 + G ∞ / σ m i n = L R /2 .
The following is the formal version of Theorem 1.
Theorem 2 (Conditional convergence of ZO-RGD, formal). Let Assumptions 1–3 hold, and let L R : = L + 2 G ∞ / σ min L_R:=L+2G_{\infty}/\sigma_{\min} L R := L + 2 G ∞ / σ m i n , with D ˉ \bar{D} D ˉ , σ D 2 \sigma_D^2 σ D 2 , and Δ 0 \Delta_0 Δ 0 as defined in Appendices § 10 .§ 1 and § 10 .§ 2 . Run base ZO-RGD (Algorithm 1) for T T T steps with the constant stepsize
η = min { 1 2 L R ( D ˉ + 2 ) , Δ 0 L R ( D ˉ + 2 ) σ D 2 T } , (36) \eta=\min\left\{\frac{1}{2L_R(\bar{D}+2)},\sqrt{\frac{\Delta_0}{L_R(\bar{D}+2)\sigma_D^2T}}\right\}, \tag*{(36)} η = min { 2 L R ( D ˉ + 2 ) 1 , L R ( D ˉ + 2 ) σ D 2 T Δ 0 } , (36) where the second candidate is read as + ∞ +\infty + ∞ if σ D = 0 \sigma_D=0 σ D = 0 . Then
1 T ∑ t = 0 T − 1 E [ ∥ grad f ( x t ) ∥ D 2 ] ≤ 8 Δ 0 L R ( D ˉ + 2 ) σ D 2 T + 8 Δ 0 L R ( D ˉ + 2 ) T + ρ 3 2 ϵ 4 9 [ ( D ˉ + 2 ) 4 + ( D ˉ + 6 ) 4 2 ( D ˉ + 2 ) ] . (37) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\left[\|\operatorname{grad} f(x_t)\|_D^2\right]\leq8\sqrt{\frac{\Delta_0L_R(\bar{D}+2)\sigma_D^2}{T}}+\frac{8\Delta_0L_R(\bar{D}+2)}{T}+\frac{\rho_3^2\epsilon^4}{9}\left[(\bar{D}+2)^4+\frac{(\bar{D}+6)^4}{2(\bar{D}+2)}\right]. \tag*{(37)} T 1 t = 0 ∑ T − 1 E [ ∥ grad f ( x t ) ∥ D 2 ] ≤ 8 T Δ 0 L R ( D ˉ + 2 ) σ D 2 + T 8 Δ 0 L R ( D ˉ + 2 ) + 9 ρ 3 2 ϵ 4 [ ( D ˉ + 2 ) 4 + 2 ( D ˉ + 2 ) ( D ˉ + 6 ) 4 ] . (37) Since ∥ v ∥ ≤ ∥ v ∥ D \|v\|\leq\|v\|_D ∥ v ∥ ≤ ∥ v ∥ D for every tangent vector v v v , the same right-hand side bounds the average of E ∥ grad f ( x t ) ∥ 2 \mathbb{E}\|\operatorname{grad} f(x_t)\|^2 E ∥ grad f ( x t ) ∥ 2 .
Proof. Take E [ ⋅ ∣ F t ] \mathbb{E}[\cdot\mid\mathcal{F}_t] E [ ⋅ ∣ F t ] of Lemma 6 and insert Lemmas 3 and 4:
E [ f ( x t + 1 ) ∣ F t ] ≤ f ( x t ) − η ⟨ grad f , D grad f + β t ⟩ + L R 2 η 2 [ 2 ( D ˉ + 2 ) ( ∥ grad f ∥ D 2 + σ D 2 ) + ρ 3 2 18 ϵ 4 ( D ˉ + 6 ) 4 ] , \mathbb{E}[f(x_{t+1})\mid\mathcal{F}_t]\leq f(x_t)-\eta\langle\operatorname{grad} f,D\operatorname{grad} f+\beta_t\rangle+\frac{L_R}{2}\eta^2\left[2(\bar{D}+2)\left(\|\operatorname{grad} f\|_D^2+\sigma_D^2\right)+\frac{\rho_3^2}{18}\epsilon^4(\bar{D}+6)^4\right], E [ f ( x t + 1 ) ∣ F t ] ≤ f ( x t ) − η ⟨ grad f , D grad f + β t ⟩ + 2 L R η 2 [ 2 ( D ˉ + 2 ) ( ∥ grad f ∥ D 2 + σ D 2 ) + 18 ρ 3 2 ϵ 4 ( D ˉ + 6 ) 4 ] ,
where all gradients are evaluated at x t x_t x t . The leading term is − η ⟨ grad f , D grad f ⟩ = − η ∥ grad f ∥ D 2 -\eta\langle\operatorname{grad} f,D\operatorname{grad} f\rangle=-\eta\|\operatorname{grad} f\|_D^2 − η ⟨ grad f , D grad f ⟩ = − η ∥ grad f ∥ D 2 . For the cross term, Young’s inequality (a b ≤ a 2 / 4 + b 2 ab\leq a^2/4+b^2 ab ≤ a 2 /4 + b 2 ) with ∥ grad f ∥ ≤ ∥ grad f ∥ D \|\operatorname{grad} f\|\leq\|\operatorname{grad} f\|_D ∥ grad f ∥ ≤ ∥ grad f ∥ D gives − η ⟨ grad f , β t ⟩ ≤ η ∥ grad f ∥ B ϵ ≤ η 4 ∥ grad f ∥ D 2 + η B ϵ 2 -\eta\langle\operatorname{grad} f,\beta_t\rangle\leq\eta\|\operatorname{grad} f\|B_\epsilon\leq\frac{\eta}{4}\|\operatorname{grad} f\|_D^2+\eta B_\epsilon^2 − η ⟨ grad f , β t ⟩ ≤ η ∥ grad f ∥ B ϵ ≤ 4 η ∥ grad f ∥ D 2 + η B ϵ 2 . The condition η ≤ 1 / ( 2 L R ( D ˉ + 2 ) ) \eta\leq1/(2L_R(\bar{D}+2)) η ≤ 1/ ( 2 L R ( D ˉ + 2 )) makes L R η 2 ( D ˉ + 2 ) ∥ grad f ∥ D 2 ≤ η 2 ∥ grad f ∥ D 2 L_R\eta^2(\bar{D}+2)\|\operatorname{grad} f\|_D^2\leq\frac{\eta}{2}\|\operatorname{grad} f\|_D^2 L R η 2 ( D ˉ + 2 ) ∥ grad f ∥ D 2 ≤ 2 η ∥ grad f ∥ D 2 . Hence
E [ f ( x t + 1 ) ∣ F t ] ≤ f ( x t ) − η 4 ∥ grad f ( x t ) ∥ D 2 + η B ϵ 2 + η 2 L R ( D ˉ + 2 ) σ D 2 + η 2 L R ρ 3 2 36 ϵ 4 ( D ˉ + 6 ) 4 . \mathbb{E}[f(x_{t+1})\mid\mathcal{F}_t]\leq f(x_t)-\frac{\eta}{4}\|\operatorname{grad} f(x_t)\|_D^2+\eta B_\epsilon^2+\eta^2L_R(\bar{D}+2)\sigma_D^2+\eta^2\frac{L_R\rho_3^2}{36}\epsilon^4(\bar{D}+6)^4. E [ f ( x t + 1 ) ∣ F t ] ≤ f ( x t ) − 4 η ∥ grad f ( x t ) ∥ D 2 + η B ϵ 2 + η 2 L R ( D ˉ + 2 ) σ D 2 + η 2 36 L R ρ 3 2 ϵ 4 ( D ˉ + 6 ) 4 .
Telescoping over t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , taking full expectations, using f ≥ f ∗ f\geq f_* f ≥ f ∗ , and multiplying by 4 / ( η T ) 4/(\eta T) 4/ ( η T ) :
1 T ∑ t = 0 T − 1 E ∥ grad f ( x t ) ∥ D 2 ≤ 4 Δ 0 η T + 4 B ϵ 2 + 4 η L R ( D ˉ + 2 ) σ D 2 + η L R ρ 3 2 9 ϵ 4 ( D ˉ + 6 ) 4 . (38) \frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\|\operatorname{grad} f(x_t)\|_D^2\leq\frac{4\Delta_0}{\eta T}+4B_\epsilon^2+4\eta L_R(\bar{D}+2)\sigma_D^2+\eta\frac{L_R\rho_3^2}{9}\epsilon^4(\bar{D}+6)^4. \tag*{(38)} T 1 t = 0 ∑ T − 1 E ∥ grad f ( x t ) ∥ D 2 ≤ η T 4 Δ 0 + 4 B ϵ 2 + 4 η L R ( D ˉ + 2 ) σ D 2 + η 9 L R ρ 3 2 ϵ 4 ( D ˉ + 6 ) 4 . (38)
With η = min { 1 / ( 2 L R ( D ˉ + 2 ) ) , Δ 0 / ( L R ( D ˉ + 2 ) σ D 2 T ) } \eta=\min\{1/(2L_R(\bar{D}+2)),\sqrt{\Delta_0/(L_R(\bar{D}+2)\sigma_D^2T)}\} η = min { 1/ ( 2 L R ( D ˉ + 2 )) , Δ 0 / ( L R ( D ˉ + 2 ) σ D 2 T ) } , the standard argument (1 / min ≤ 1/\min\leq 1/ min ≤ sum of reciprocals, and the second candidate balances the first and third terms of (38) ) gives
4 Δ 0 η T + 4 η L R ( D ˉ + 2 ) σ D 2 ≤ 8 Δ 0 L R ( D ˉ + 2 ) σ D 2 T + 8 Δ 0 L R ( D ˉ + 2 ) T , \frac{4\Delta_0}{\eta T}+4\eta L_R(\bar{D}+2)\sigma_D^2\leq8\sqrt{\frac{\Delta_0L_R(\bar{D}+2)\sigma_D^2}{T}}+\frac{8\Delta_0L_R(\bar{D}+2)}{T}, η T 4 Δ 0 + 4 η L R ( D ˉ + 2 ) σ D 2 ≤ 8 T Δ 0 L R ( D ˉ + 2 ) σ D 2 + T 8 Δ 0 L R ( D ˉ + 2 ) ,
which gives the first two terms of (37) . For the remaining two terms of (38) , the definition of B ϵ B_\epsilon B ϵ gives 4 B ϵ 2 = ρ 3 2 9 ϵ 4 ( D ˉ + 2 ) 4 4B_\epsilon^2=\frac{\rho_3^2}{9}\epsilon^4(\bar{D}+2)^4 4 B ϵ 2 = 9 ρ 3 2 ϵ 4 ( D ˉ + 2 ) 4 , and η ≤ 1 / ( 2 L R ( D ˉ + 2 ) ) \eta\leq1/(2L_R(\bar{D}+2)) η ≤ 1/ ( 2 L R ( D ˉ + 2 )) gives η L R ρ 3 2 9 ϵ 4 ( D ˉ + 6 ) 4 ≤ ρ 3 2 18 ϵ 4 ( D ˉ + 6 ) 4 / ( D ˉ + 2 ) \eta\frac{L_R\rho_3^2}{9}\epsilon^4(\bar{D}+6)^4\leq\frac{\rho_3^2}{18}\epsilon^4(\bar{D}+6)^4/(\bar{D}+2) η 9 L R ρ 3 2 ϵ 4 ( D ˉ + 6 ) 4 ≤ 18 ρ 3 2 ϵ 4 ( D ˉ + 6 ) 4 / ( D ˉ + 2 ) . Summing yields (37) . Since ∥ v ∥ ≤ ∥ v ∥ D \|v\|\leq\|v\|_D ∥ v ∥ ≤ ∥ v ∥ D for every tangent vector, the same right-hand side bounds the plain-norm average.
η r e g : = ϵ σ min 2 B F max ℓ ρ ℓ \eta_{\mathrm{reg}} := \frac{\epsilon\sigma_{\min}}{2B_F \max_{\ell} \rho_{\ell}} η reg := 2 B F m a x ℓ ρ ℓ ϵ σ m i n and choose the three-way minimum η = min { 1 2 L R ( D ˉ + 2 ) , Δ 0 L R ( D ˉ + 2 ) σ D 2 T , η r e g } \eta= \min\left\{\frac{1}{2L_R(\bar{D}+2)}, \sqrt{\frac{\Delta_0}{L_R(\bar{D}+2)\sigma_D^2T}}, \eta_{\mathrm{reg}}\right\} η = min { 2 L R ( D ˉ + 2 ) 1 , L R ( D ˉ + 2 ) σ D 2 T Δ 0 , η reg } . Since 1 / min ≤ 1/\min\le 1/ min ≤ the sum of reciprocals, the 4 Δ 0 / ( η T ) 4\Delta_0/(\eta T) 4 Δ 0 / ( η T ) term contributes at most one extra summand 4 Δ 0 / ( η r e g T ) = 8 Δ 0 B F max ℓ ρ ℓ / ( ϵ σ min T ) 4\Delta_0/(\eta_{\mathrm{reg}}T) = 8\Delta_0B_F\max_{\ell}\rho_{\ell}/(\epsilon\sigma_{\min}T) 4 Δ 0 / ( η reg T ) = 8 Δ 0 B F max ℓ ρ ℓ / ( ϵ σ m i n T ) , which is O ( 1 / T ) O(1/T) O ( 1/ T ) and does not affect the leading 1 / T \sqrt{1/T} 1/ T rate.
Remark 7 (function-evaluation complexity and the ϵ \epsilon ϵ -condition). Each step uses two forward passes. In the regime ϵ 2 ≤ σ D 2 \epsilon^2 \le\sigma_D^2 ϵ 2 ≤ σ D 2 , to reach ( 1 / T ) ∑ t E ∥ grad f ( x t ) ∥ 2 ≤ ϵ 2 (1/T)\sum_t \mathbb{E}\|\operatorname{grad} f(x_t)\|^2 \le\epsilon^2 ( 1/ T ) ∑ t E ∥ grad f ( x t ) ∥ 2 ≤ ϵ 2 it suffices to take T = O ( Δ 0 L R ( D ˉ + 2 ) σ D 2 / ϵ 4 ) T = O(\Delta_0L_R(\bar{D}+2)\sigma_D^2/\epsilon^4) T = O ( Δ 0 L R ( D ˉ + 2 ) σ D 2 / ϵ 4 ) together with ϵ 2 ≤ 3 ϵ s / ( 2 ρ 3 ( D ˉ + 2 ) 2 ) \epsilon^2 \le3\epsilon_s/(\sqrt{2}\rho_3(\bar{D}+2)^2) ϵ 2 ≤ 3 ϵ s / ( 2 ρ 3 ( D ˉ + 2 ) 2 ) , which makes 4 B ϵ 2 ≤ ϵ s 2 / 2 4B_\epsilon^2 \le\epsilon_s^2/2 4 B ϵ 2 ≤ ϵ s 2 /2 and renders the ϵ 4 \epsilon^4 ϵ 4 term subdominant. The ϵ \epsilon ϵ -condition carries ambient-dimension powers, as in standard ambient ZO analyses (e.g., the μ \mu μ -condition in the MeZO analysis [24 ] ); ρ 3 \rho_3 ρ 3 is a worst-case directional constant along the realized segments. We state the sufficient scaling explicitly rather than absorbing it into asymptotic notation.
Additional Experimental Details
This appendix follows the order of § 6 . Appendices D.1 and D.2 give the training protocol and compute resources. Appendices D.3–D.6 supplement the broad-suite comparison with the complete OPT-1.3B table, its unified-protocol rerun, the trainability floor, and the OPT-30B spot-check. Appendices D.7–D.9 supplement the controlled comparison with per-seed values, the matched-rank comparison, and a shared-learning-rate regime with convergence against forward passes. Appendix D.10 gives the learning-rate diagnostic for the modern-model panel. Appendices D.11–D.15 collect the geometry controls, negative results, ablations, and stability diagnostics; Appendices D.16 and D.17 give runtime and memory; and Appendix D.18 is the failure register.
Training protocol and configurations
For ZO-RGD and ZO-RGD-S, each ZO update uses one normalized tangent random matrix, so the two-point finite-difference scalar is obtained with two forward passes. Unless otherwise noted, runs use FP16 checkpoints, 20,000 training steps, constant learning rate, ϵ = 10 − 3 \epsilon= 10^{-3} ϵ = 1 0 − 3 , and weight decay 0. Seeds are paired: the training-subset seed and the trainer seed move together. Hyperparameter selection uses a dedicated grid seed (0) on OPT and a disjoint pilot seed (7) on the modern-model panel; report seeds are { 1 , 2 , 3 } \{1, 2, 3\} { 1 , 2 , 3 } on the OPT panels and { 0 , 1 , 2 } \{0, 1, 2\} { 0 , 1 , 2 } on the modern-model panel (the two panels were executed independently; neither report set overlaps its selection seed, and the per-panel seed sets are disclosed here because they differ).
Each run samples three disjoint subsets with the paired seeds: up to 1000 training examples, up to 500 development examples held out from the training data (100 for CB and COPA), and up to 1000 evaluation examples from the task’s validation split; smaller datasets are capped by the available split sizes, and reported n n n is the actual subset size of the run. Training operates only on the training subset. Within each run, the kept checkpoint is the one with the lowest development loss (eval_loss), under the same cadence for every method; development loss is the selection quantity because it is the only development-set signal produced by every trainer in the comparison, so a single selection rule can be applied symmetrically. The evaluation subset is read exactly once per run, after all selection. Table 7 summarizes the per-method configurations, and Table 8 audits the baseline implementations and their deviations.
Table 7. Unified-protocol configuration card. All methods use two forward passes per step; ranks are each method’s published setting; the shared grid is the MeZO grid, verbatim; selection is development-loss only (grid/pilot seed disjoint from report seeds).
Table 8. Baseline implementation audit for the unified-protocol tables. All baselines run through the same data, prompt, and evaluation path as our methods, from the local baseline implementations, with the deviations column exhaustive.
Checkpoint conventions of the retained broad-suite tables. The three broad-suite tables of § 6.2 retain their original reporting conventions, which differ from the unified protocol above. At OPT-1.3B (Table 9 ), our rows use best-development checkpoints while the four baseline rows report the final step. At OPT-1.3B (Table 1 ), our cells are selected by development metric within each variant’s completed 20,000-step sweep, while the baseline rows are cited from LOZO. At OPT-30B (Table 2 ), the cited rows follow the LOZO reporting and our cells were not selected under the unified rule, so the table serves as scaling evidence only; the symmetric OPT-30B evidence is the BoolQ spot-check of Appendix D.6. Every controlled claim in the paper rests on the unified-protocol tables.
Table 9. OPT-1.3B complete local comparison on the eleven-task suite. Best and second-best results in each column are shown in bold and underlined, respectively.
Compute resources and reproducibility
ZO-RGD and ZO-RGD-S experiments were conducted on a shared academic GPU cluster with heterogeneous NVIDIA GPUs (V100, A100-80G, H100, H200); the unified-protocol OPT-13B cells ran on A100-80G and H200, with per-cell hardware recorded in the manifest. Wall-clock is compared only within one GPU model (Appendix § 11 .16), and no cross-node timing ratio is used anywhere in the paper. Every reported cell carries a manifest row, job ID, result file, configuration, selection rule, seeds, hardware, status (including failures), and, for runs after provenance instrumentation landed, the exact code commit, released with the code together with the per-seed tables of Appendix § 11 .§ 7 .
OPT-1.3B: complete forward-only comparison
Table 9 reports OPT-1.3B results on the full eleven-task suite. This scale provides a complete local comparison among forward-only methods, complementing the cited large-model tables of § 6.2 . It tests whether the fixed-rank additive-update formulation is already effective at a scale where the full baseline set can be evaluated. The table is retained under its original convention: our rows use best-development checkpoints and the baseline rows use the final step (Appendix § 11 .§ 1 ); the symmetric unified-protocol rerun of the same suite is Table 10 in Appendix § 11 .§ 4 .
Table 10. OPT-1.3B breadth, unified protocol, single fresh seed per cell (%). No bolding: single-seed cells are not ranked. SQuAD/DROP report F1.
The two Riemannian ZO variants give the best forward-only result on 7 of the 11 tasks. The gains are distributed across task families, including entailment and semantic classification tasks such as RTE and CB, reasoning tasks such as MultiRC and COPA, and extractive QA tasks such as SQuAD and DROP. This result suggests that optimizing the persistent fixed-rank update is competitive across the complete suite, rather than benefiting only one benchmark type.
OPT-1.3B breadth under the unified protocol
Table 10 reruns the full eleven-task suite under the unified protocol (shared grid, development-loss selection, evaluation set read once), at a single fresh report seed per cell; it revisits the same eleven tasks as the retained Table 9 under the symmetric convention. Cells are single-seed and therefore not ranked statistically. A selection-stability check over the 66 (method, task) cells, formal fresh-seed value minus grid-seed value, gives mean − 0.60 -0.60 − 0.60 , sd 2.31 2.31 2.31 , with ∣ Δ ∣ > 4 |\Delta| > 4 ∣Δ∣ > 4 points only on small-evaluation-set tasks and in both directions, so the grid stage does not systematically inflate the fresh-seed numbers. Read alongside the trainability floor (Appendix § 11 .§ 5 ): on WiC, COPA, and ReCoRD the best trained method sits within two standard deviations of the untrained model, so differences there are mostly noise.
Zero-shot trainability floor at OPT-1.3B
Table 11 tests whether each task responds to forward-only training at this scale. It compares the self-run untrained (zero-shot) score with the best trained cell of the historical eleven-task pool (over which the floor analysis was originally built), normalizing the best-case gain by evaluation sampling noise alone, so the ratio is an optimistic upper bound on signal. Three tasks fall below two standard deviations of the untrained model, ReCoRD (1.5 sd), COPA (0.6), WiC (0.4), and on COPA and WiC at least one trained row scores at or below the untrained model. Recomputing the ratios against the unified-protocol breadth table changes the numbers slightly (WiC 2.0, COPA 0.8, ReCoRD 1.3) and no conclusion: columns on these tasks measure noise more than method, and we do not interpret differences on them anywhere in this paper.
Table 11. Trainability floor at OPT-1.3B (%). Δ \Delta Δ /sd normalizes the best-case trained gain by evaluation sampling noise; seed-to-seed training variance is not included, so every ratio is optimistic.
OPT-30B single-seed spot-check
The OPT-30B BoolQ column of Table 3 reports the unified-protocol BoolQ spot-check (shared grid, development-loss winner at 10−7 for all four methods, grid seed 0). An earlier pre-locked fixed-configuration triple is consistent (ZO-RGD/ZO-RGD-S/LOZO: 72.00/72.50/68.70). These cells are single-seed, carry no uncertainty estimate, and are not ranked statistically; they are the only symmetric 30B evidence and update the corresponding cells of the retained Table 2 . The 30B SQuAD column was not run: all four methods exceed 80 GB at the first forward pass on A100-80G, and the available longer-wall-clock partitions did not admit the run.
Per-seed values and provenance
The controlled table compares matched self-run rows at OPT-13B and a uniform-convention OPT-30B spot-check. HiZOO is shown only for the verified three-seed RTE cell; unavailable matched cells are not inferred. Table 12 and Table 13 report the per-seed values behind the controlled comparison and the modern-model panel. Untrained zero-shot reference evaluations under the same harness give RTE 60.29/59.57 (two runs) and SQuAD 46.23/48.35/46.59/48.19/46.29 (five runs). Every reported cell of this paper maps to one cluster job ID, one result file, and one manifest row (selection rule, learning rate, seeds, hardware, and status, including failed runs); the manifest is released with the code, and the per-seed values for the mechanism blocks of Table 5 are recorded there in the same format. The HiZOO dagger row uses RTE seeds { 0 , 1 , 2 } \{0, 1, 2\} { 0 , 1 , 2 } , with per-seed accuracies 61.37/59.93/64.98; a matched SQuAD run is unavailable.
Table 12: Per-seed values (%) for the non-dagger rows of the OPT-13B controlled comparison (Table 3 ), report seeds { 1 , 2 , 3 } \{1, 2, 3\} { 1 , 2 , 3 } . NR: no formal LoRA+AdamW SQuAD seed results; the A100 grid attempt ran out of memory.
Table 12.
Table 13: Per-seed values (%) for the modern-model panel (Table 4 ), report seeds { 0 , 1 , 2 } \{0, 1, 2\} { 0 , 1 , 2 } .
Table 13.
Matched-rank comparison at OPT-13B
Table 14 reports the matched-rank comparison summarized in § 6.3 , and Table 15 gives the paired values behind it. Both variants use rank 2, seeds { 1 , 2 , 3 } \{1, 2, 3\} { 1 , 2 , 3 } , 20,000 updates, and development-loss checkpoint selection. The learning rates (RTE: 10 − 6 10^{-6} 1 0 − 6 ; SQuAD: 10 − 7 10^{-7} 1 0 − 7 ) were selected for ZO-RGD-S and fixed before the additional ZO-RGD runs; they were not independently tuned for ZO-RGD in this comparison.
Table 15. Matched-rank OPT-13B scores (%). All twelve runs completed. Differences are ZO-RGD-S minus ZO-RGD, calculated before rounding.
The mean paired differences are +4.93 points on RTE and − 0.50 -0.50 − 0.50 on SQuAD, with paired-difference standard deviations 5.76 and 0.96, respectively (two-sided paired t t t -tests: p = 0.276 p = 0.276 p = 0.276 and 0.466). All three RTE seeds favor ZO-RGD-S, but excluding seed 2 reduces the mean difference to +1.62. SQuAD favors ZO-RGD-S on one of three seeds. The observations are task-dependent; neither test establishes a difference, and nonsignificance does not establish equivalence.
Shared-learning-rate controlled core at OPT-13B
A companion regime to the shared-grid protocol of § 6.1 : a single pre-locked learning rate (10 − 7 10^{-7} 1 0 − 7 , fixed in advance for every method and both tasks) applied identically to ZO-RGD, ZO-RGD-S, and LOZO, with no method receiving any tuning, on five paired seeds { 0 , … , 4 } \{0,\ldots,4\} { 0 , … , 4 } . No method receives any tuning in this regime; its value is the paired structure. Two deviations from the main-text protocol are disclosed: the LOZO runs of this earlier core used rank 1 with resampling interval ν = 50 \nu= 50 ν = 50 (an internal setting, verified from the run logs; the official ν = 100 \nu= 100 ν = 100 is used in every unified-protocol table), and the report seeds are { 0 , … , 4 } \{0,\ldots,4\} { 0 , … , 4 } rather than { 1 , 2 , 3 } \{1,2,3\} { 1 , 2 , 3 } . Table 16 reports the cells and the paired comparisons. Divergences stay in the denominator: ZO-RGD diverged (FP16 NaN) on SQuAD seeds 0 and 3 and reports 3/5; its RTE cell reports 4/5 after one timeout, and all other cells report 5/5. Figure 2 complements these endpoint cells with a forward-pass-efficiency view of the same regime: RTE development loss against cumulative forward passes at the shared learning rate, truncated at the common 40K-forward budget, with HiZOO added at the same rate.
Figure 2. OPT-13B RTE development loss against cumulative forward passes at the shared learning rate 10 − 7 10^{-7} 1 0 − 7 , truncated at the common 40K-forward budget. Lines are means and bands are ranges over completed runs. HiZOO uses four forward passes per step; the other methods use two.
Table 16 : Shared-learning-rate core (%; mean ± \pm ± sd over completed seeds; completed/attempted in parentheses). Paired differences vs. LOZO use only seeds where both methods completed; the sign test is exact and one-sided. “Strict separation” means the worst seed of ours beats the best seed of LOZO on the paired subset.
Table 16.
Learning-rate diagnostic for the modern-model panel
Table 4 in § 6.4 tests whether the fixed-rank update formulation transfers beyond OPT under one protocol and three report seeds. Each method selects its own learning rate in a development-only pilot of 2,000 steps. On Llama-3.1-8B BoolQ the pilot chose 10 − 7 10^{-7} 1 0 − 7 for ZO-RGD-S and 10 − 6 10^{-6} 1 0 − 6 for ZO-RGD and LOZO. Table 17 compares both rates at seed 0 for all three methods over the full 20,000-update budget. Changing the rate from 10 − 6 10^{-6} 1 0 − 6 to 10 − 7 10^{-7} 1 0 − 7 increases accuracy by 17.9, 19.8, and 18.2 points for LOZO, ZO-RGD, and ZO-RGD-S, respectively. At the shared rate 10 − 7 10^{-7} 1 0 − 7 , both Riemannian variants reach 85.5, versus LOZO’s 80.9; this remaining 4.6-point gap is a single-seed observation with method-specific ranks. The diagnostic identifies learning-rate sensitivity in the panel and shows that the short pilot selected a rate for ZO-RGD and LOZO that performed worse at the full budget on this task. It does not establish a ranking across seeds. The original three-seed panel remains unchanged; its per-seed values are in Appendix § 11 .§ 7 .
Additional geometry controls and negative results
Table 5 in § 6.5 gives the complete finite-difference-geometry and parameterization controls. This appendix adds a working-point diagnostic for the baselines, the traversal-order check, the HiZOO released-implementation diagnostic, and negative results for additional variants.
Baselines at ZO-RGD’s working point. Table 18 reruns MeZO and LOZO at ZO-RGD’s working point. Its single-seed ZO-RGD reference is descriptive; it is not a three-seed paired comparison.
Table 18: Baselines at ZO-RGD’s working point on OPT-1.3B (%). MeZO and LOZO report mean ± \pm ± sd over three seeds; ZO-RGD is the single fresh-seed breadth result in Appendix § 11 .§ 4 . This diagnostic uses a different working point from the parameterization comparison in Table 5 .
Table 18.
Block traversal order. Rerunning ZO-RGD-S under three different blockwise traversal orders (name-keyed RNG, three paired seeds) spreads SST-2 accuracy by 0.15 points (91.63/91.78/91.70), far below the seed standard deviation — the fixed traversal order that its scalar recursion follows is not a hidden tuning axis.
LOZO accumulated-update rank. The fixed-rank constraint is a real restriction on the update only because the closest baseline’s persistent update is not rank-limited in practice. Instrumenting LOZO’s accumulated update W T − W 0 W_T-W_0 W T − W 0 with per-layer SVD sidecars (official r = 1 r=1 r = 1 configuration, RTE) shows its effective rank at 90% of the energy reaching a median of 1019 of the 2048 model dimension across 146 tracked OPT-1.3B layers, and 2466 of 5120 across 242 OPT-13B layers, already by step 4,000 and essentially flat thereafter (1019 and 2466 at step 20,000). The baseline’s learned update is effectively half full-rank; ours is exactly rank r r r .
HiZOO released-implementation diagnostic. We include HiZOO ((Zhao et al., 2024b)) in the OPT-13B RTE comparison and convergence plot using the released implementation at the shared learning rate 10 − 7 10^{-7} 1 0 − 7 . At its published OPT learning rate, however, the same implementation scores at chance on our RTE setup at both 1.3B and 13B (52.1 ± \pm ± 3.8 / 52.5 ± \pm ± 1.1). Instrumentation shows that its Hessian-information coefficient is inactive (α = 0 \alpha= 0 α = 0 ) in the released setting, at which point the update rule reduces to MeZO’s; matched-learning-rate comparisons overlap with MeZO within seed noise at both scales. A modified activated configuration (α > 0 \alpha> 0 α > 0 , FP32 Hessian state) trains in smoke tests, but it is not the published configuration and is therefore not substituted for the released method in the main comparison.
Negative results on our own variants. Two internal variants underperform and are reported as such. A tangent-space (matrix) momentum variant loses to the scalar recursion of ZO-RGD-S on four of six tasks at three paired seeds (pilot: one of six, mean − 2.0 -2.0 − 2.0 points), and a per-step EMA variant wins on one of three tasks; the scalar recursion along the fixed traversal order remains the recommended smoothing. A lazy-retraction variant (retract every F F F steps, F ∈ { 10 , 50 , 200 } F \in\{10, 50, 200\} F ∈ { 10 , 50 , 200 } ) is slower than retracting every step by 17–19%: phase profiling shows retraction is only 18.4% of a step while random-matrix generation and projection account for ≈ 60 % \approx60\% ≈ 60% , and deferring retraction forces materialization of the accumulated update. Lazy retraction is a separate algorithm outside the scope of the base theorem, and its measured effect is negative on both axes; runtime engineering should target the random-matrix generation and projection phases instead.
Effect of normalizing the tangent random matrix
We test the central design choice of the estimator: post-projection normalization that restores the ambient finite-difference radius after tangent projection. Without normalization, the raw projected random matrix has Frobenius norm concentrated at d r \sqrt{d_r} d r rather than m n \sqrt{mn} mn . For OPT-1.3B square blocks with m = n = 2048 m = n = 2048 m = n = 2048 and r = 4 r = 4 r = 4 , the ratio is d r / ( m n ) ≈ 0.063 \sqrt{d_r/(mn)} \approx0.063 d r / ( mn ) ≈ 0.063 , so the unnormalized two-point finite difference evaluates the loss within an effective radius of about 6% of the standard MeZO/LOZO scale at the same ϵ \epsilon ϵ .
Table 19 reports controlled OPT-1.3B ablations with ZO-RGD (r = 4 r = 4 r = 4 , η = 10 − 7 \eta= 10^{-7} η = 1 0 − 7 , batch size 16, 20K steps). The first comparison uses the raw projected random matrix at ϵ = 10 − 3 \epsilon= 10^{-3} ϵ = 1 0 − 3 on SST-2, RTE, and SQuAD; the rescaled-radius comparison uses ϵ = 1.6 × 10 − 2 \epsilon= 1.6 \times10^{-2} ϵ = 1.6 × 1 0 − 2 , the square-block compensation value, to test whether one global radius can replace per-block normalization.
Table 19. Normalization ablation on OPT-1.3B. The normalized reference reproduces the historical ZO-RGD r = 4 r = 4 r = 4 row (Table 9 ). SQuAD reports F1; the other tasks report accuracy. † ^{\dagger} † The SQuAD unnormalized run diverged and was cancelled before completion; intermediate checkpoints at steps 4K, 8K, and 12K all gave F1 0, and the entry is a truncated run reported at its collapsed value, not a completed evaluation. A rerun cannot repair it: with normalization off, the SQuAD ablation diverges deterministically (first evaluation NaN on all seeds; Appendix § 11 .18), and the divergence itself is the ablation conclusion.
The raw projected random matrix performs poorly under the standard ϵ = 10 − 3 \epsilon= 10^{-3} ϵ = 1 0 − 3 setting on all tested tasks: SST-2 drops by roughly 30 points, RTE stays near chance, and SQuAD yields zero F1 at every measured intermediate evaluation. The ϵ = 1.6 × 10 − 2 \epsilon= 1.6 \times10^{-2} ϵ = 1.6 × 1 0 − 2 rescaled-radius comparison does not recover the SST-2 result. This is consistent with the structure of OPT blocks, which mix square attention matrices with non-square feed-forward matrices; a single global ϵ \epsilon ϵ cannot simultaneously calibrate the per-block d r ℓ / ( m ℓ n ℓ ) \sqrt{d_{r_\ell}/(m_\ell n_\ell)} d r ℓ / ( m ℓ n ℓ ) ratios. Normalization is therefore a per-block rank- and shape-dependent calibration of the finite-difference radius, not a global scalar rescaling.
Rank ablation on OPT-1.3B
Historical convention: the runs in this subsection predate the unified protocol (final-step evaluation, single seed) and support no main-text claim; the protocol-conformant rank evidence is the rank axis of the stability atlas (Appendix § 11 .14).
Rank controls both the intrinsic tangent dimension and the size of the thin-SVD update state. Table 20 swaps the historical-table ranks for ZO-RGD (r = 4 r = 4 r = 4 ) and ZO-RGD-S (r = 2 r = 2 r = 2 ) on five tasks; the largest observed shift is 1.4 points and most paired results are within 1 point. An SST-2 rank-sweep table that appeared in the earlier version of this manuscript is removed: five of its six cells cannot be reconstructed from any job log; the reconstructible rank evidence under the unified protocol is in the stability atlas. These results indicate that the OPT-1.3B conclusions are not tied to a single isolated rank choice.
Table 20 : OPT-1.3B cross-rank auxiliary runs for the two ranks used in the main table. These runs use 20K steps, η = 10 − 7 \eta= 10^{-7} η = 1 0 − 7 , and final-step evaluation; they are separate from the main-table checkpoint-selection convention. SQuAD and DROP report F1; the remaining tasks report accuracy. Bold marks the better rank for each available pair.
Table 20.
Stability atlas at OPT-1.3B
One-factor-at-a-time robustness of ZO-RGD around the default configuration (SST-2, two paired seeds per setting; Table 21 ). Rank is insensitive across r ∈ { 1 , 2 , 8 } r \in\{1, 2, 8\} r ∈ { 1 , 2 , 8 } (91.4–92.6 against base 92.6); the perturbation radius is stable one decade down (ϵ = 10 − 4 \epsilon= 10^{-4} ϵ = 1 0 − 4 : 91.3) and breaks one decade up (ϵ = 10 − 2 \epsilon= 10^{-2} ϵ = 1 0 − 2 : 69.8); and the initialization scale is irrelevant across four orders of magnitude (σ i n i t = 10 − 4 \sigma_{\mathrm{init}} = 10^{-4} σ init = 1 0 − 4 : 92.6; 10 − 8 10^{-8} 1 0 − 8 : 92.2). In FP16, 10 − 8 10^{-8} 1 0 − 8 underflows to zero, so this configuration effectively starts at Σ = 0 \Sigma= 0 Σ = 0 ; training is unaffected because the normalized tangent step does not scale with Σ \Sigma Σ — the first step has ∥ ξ ∥ ≈ 5.5 × 10 − 3 \lVert\xi\rVert\approx5.5 \times10^{-3} ∥ ξ ∥ ≈ 5.5 × 1 0 − 3 and moves the state off zero, and the step-0 diagnostics of the 10 − 6 10^{-6} 1 0 − 6 and 10 − 8 10^{-8} 1 0 − 8 runs are bitwise identical. The method is therefore insensitive to under-initialization by construction. Precision is the one sensitive axis: bf16 scores 77.6 (− 14.9 -14.9 − 14.9 points), and FP16 is the default for this estimator (see also the failure register).
Table 21 : Stability atlas (SST-2 accuracy, %; mean ± \pm ± sd over two paired seeds; one factor varied per row).
Table 21.
Singular-value diagnostics
Every 500 steps, each ZO-RGD and ZO-RGD-S run of the shared-learning-rate core (Appendix § 11 .9; 20 runs) and of the two OPT-30B BoolQ configurations (the unified spot-check and the earlier pre-locked triple of Appendix § 11 .§ 6 ; 4 runs) records the manifold-wide minimum singular value σ min = min σ σ r ( Δ W ℓ ) \sigma_{\min} = \min_{\sigma} \sigma_r(\Delta W_{\ell}) σ m i n = min σ σ r ( Δ W ℓ ) , condition-number summaries over the adapted blocks, and one representative layer’s spectrum — 24 runs at two scales in total (Table 22 ; ranges across completed seeds). Three readings: σ min \sigma_{\min} σ m i n moves away from the rank-deficient boundary by five to six orders of magnitude; no singular value approaches zero, so the retraction is never evaluated near the degenerate set; and condition numbers stay within 7% of 1, so no block effectively loses rank by spectral collapse. Of the two ZO-RGD SQuAD seeds that diverge under the shared learning rate (Appendix § 11 .18), seed 0 goes non-finite at step 500, before a second diagnostic is logged; seed 3, which diverges at step 1,500, shows the complementary signature: ∥ Σ ∥ \lVert\Sigma\rVert ∥ Σ ∥ grows (0.07 → \to → 2.58 → \to → 3.70 over the diagnostics logged between steps 1,000 and 1,500) while σ min \sigma_{\min} σ m i n does not collapse before the loss goes non-finite — a step-size and dynamic-range failure, not a boundary failure. The instrument makes the boundary clause of Assumption 1(i) empirically checkable on our own runs; it does not monitor the step-size clause (iv), and it is a diagnostic, not a certificate of the theorem.
Table 22. Manifold-wide σ min \sigma_{\min} σ m i n and conditioning across the instrumented runs (ranges over completed seeds). All runs start at σ min = 1.013 × 10 − 6 \sigma_{\min} = 1.013 \times10^{-6} σ m i n = 1.013 × 1 0 − 6 by construction.
Runtime and memory
Table 6 in § 6.5 separates forward-pass count, per-step time, and peak memory; it records the computational price of projection and retraction without turning endpoint accuracy into a wall-clock claim. Its forward-only rows average the two hardware repeats in Table 23 , which runs all four methods back to back on two different A100-80G cards.
Memory analysis
The persistent trainable state is the thin-SVD triple of Appendix § 8 .§ 4 , with r ℓ ( m ℓ + n ℓ + 1 ) r_\ell(m_\ell+n_\ell+1) r ℓ ( m ℓ + n ℓ + 1 ) parameters per selected block. For OPT-1.3B with model dimension 2048, rank r = 4 r=4 r = 4 , and all 146 selected two-dimensional blocks, the total FP16 fixed-rank update state is approximately
∑ ℓ r ℓ ( m ℓ + n ℓ + 1 ) ⋅ 2 bytes ≈ ( 24 [ 4 ⋅ 4 ( 2048 + 2048 + 1 ) + 2 ⋅ 4 ( 2048 + 8192 + 1 ) ] \sum_{\ell} r_\ell(m_\ell+n_\ell+1)\cdot2\ \text{bytes} \approx\left(24[4\cdot4(2048+2048+1)+2\cdot4(2048+8192+1)] \right. ℓ ∑ r ℓ ( m ℓ + n ℓ + 1 ) ⋅ 2 bytes ≈ ( 24 [ 4 ⋅ 4 ( 2048 + 2048 + 1 ) + 2 ⋅ 4 ( 2048 + 8192 + 1 )]
+ 4 ( 50272 + 2048 + 1 ) + 4 ( 2050 + 2048 + 1 ) ) ⋅ 2 bytes \left. +4(50272+2048+1)+4(2050+2048+1)\right)\cdot2\ \text{bytes} + 4 ( 50272 + 2048 + 1 ) + 4 ( 2050 + 2048 + 1 ) ) ⋅ 2 bytes
≈ 7.5 MB . \approx7.5\ \text{MB}. ≈ 7.5 MB .
The 146 blocks are the matrices updated by the implementation: six projection matrices per transformer layer across 24 OPT layers, plus the token-embedding ( 50272 × 2048 ) (50272 \times2048) ( 50272 × 2048 ) and positional-embedding ( 2050 × 2048 ) (2050 \times2048) ( 2050 × 2048 ) matrices outside the repeated layer stack. This estimate therefore accounts for the attention projections, MLP projections, and the two embedding matrices in OPT-1.3B. It is negligible relative to the frozen pretrained weights (∼ 2.6 \sim2.6 ∼ 2.6 GB in FP16). A short empirical measurement (100 training steps, NVIDIA V100, batch 16) reported a 7.2 MB fixed-rank update overhead and a total peak GPU memory of ∼ 6.07 \sim6.07 ∼ 6.07 GB, dominated by frozen weights and forward-pass activations rather than by the trainable update state.
Failure register
Every failed run stays in the denominator of its cell or is disclosed here; the manifest retains 63 failed, cancelled, timed-out, or numerically divergent run records alongside 1552 successful result records. The material failures for interpretation are: (i) FP16 divergence of ZO-RGD on OPT-13B SQuAD under the shared learning rate, seeds 0 and 3, first non-finite diagnostic at steps 500 and 1500 respectively, projected-gradient NaN with FPII collapsing to zero, while ZO-RGD-S completed all five seeds in the same regime; one additional ZO-RGD SQuAD attempt in the grid-regime rerun diverged the same way on A100 (from step ≈ 12,000 \approx12{,}000 ≈ 12 , 000 ) and was superseded by a clean completed rerun. These failures occurred in the listed configurations; both variants completed all runs in the matched rank-2 comparison (Table 14 ). (ii) bf16: three attempts at a bf16 forensic matrix produced zero completed runs; together with the −14.9-point bf16 atlas entry, FP16 is the supported precision, and a small 2 × 2 2 \times2 2 × 2 precision forensic shows the divergence onset step shifts with dtype while the mode itself is structural. (iii) The normalization-off ablation diverges deterministically on SQuAD (first evaluation NaN on all seeds); the divergence itself is the ablation conclusion and no rerun can repair it. (iv) OPT-13B ReCoRD, ZO-RGD rank 4 (the single landed sweep cell): manifold-state NaN propagated from step 4,000, and the strict checkpoint loader refused the corrupt state after the 20,000-step run, so the cell produced no valid evaluation and is marked Fail in Table 1 ; an earlier pre-safeguard attempt of the same configuration collapsed to 15.1. ZO-RGD-S completed the same task at 82.10.