betaFeedback

WorksComputer Science

Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

Abstract

Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update remains rank-constrained throughout fine-tuning. On this geometry, we build a Riemannian ZO gradient estimator that projects a random matrix onto the tangent space, normalizes it to correct the scale mismatch induced by projection, and further propose Zeroth-order Riemannian Gradient Descent (ZO-RGD) and its smoothed variant, ZO-RGD-S. We show that the ambient finite difference matches its retracted counterpart, the normalized estimator recovers the Riemannian gradient in expectation, and ZO-RGD converges under trajectory-regularity conditions. Experiments on LLM fine-tuning tasks show that the proposed algorithms are competitive with strong forward-only baselines across model families and tasks, with gains in several settings.

Introduction

Large language models (LLMs) have achieved remarkable success across a wide range of domains [7, 2]. Fine-tuning these models on downstream tasks is crucial for their practical deployment. Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) [16] have emerged as a practical solution, training low-rank updates while keeping pretrained weights frozen [16, 21, 14, 20, 38] (Zhang et al. 2023). However, first-order optimization still incurs substantial memory overhead from storing activations for backpropagation, especially in long-context settings where activations dominate memory. To enhance memory efficiency, zeroth-order (ZO) fine-tuning methods [24, 8] (Zhang et al. 2024) estimate descent directions from function values without backpropagated gradients, avoiding the activation storage required by backward passes [33].

Recent low-rank ZO fine-tuning methods combine these two directions by restricting the finite-difference search through low-rank perturbations, adapter factors, gradient estimators, or random subspaces [9, 39, 32] (Yu et al. 2025). Factor-based approaches constrain the learned update through adapter coordinates, whereas low-rank perturbation methods constrain individual ZO steps. These two branches of methods differ in the way of constructing low-rank additive update matrices when optimizing a fixed-rank update matrix. Factor-based adaptation uses a factorized update ΔW=BA\Delta W = BA and optimizes over (B,A)(B, A) [16]. This representation is nonunique, since (BQ)(Q−1A)(BQ)(Q^{-1}A) gives the same update for any invertible QQ. Low-rank perturbation and random-subspace methods instead use restricted finite-difference directions or estimators to construct each ZO step [9, 39] (Yu et al. 2025). For example, low-rank perturbation methods form each step from a sampled rank-rr direction UVTUV^{\mathsf{T}}, while random-subspace variants restrict the step direction to a prescribed low-dimensional subspace [9] (Yu et al. 2025). As a result, factor-based methods optimize a non-unique parameterization, while perturbation and subspace methods restrict each ZO search direction.

To tackle these issues, we propose to optimize ΔW\Delta W as a fixed-rank matrix variable formulated on a Riemannian manifold. Consider one selected weight matrix with frozen pretrained value W0W_0. At iteration tt, the adapted weight is written as Wt=W0+ΔWtW_t = W_0 + \Delta W_t, with

ΔWt∈Mrm×n,t=0,…,T,(1)\Delta W_t \in\mathcal{M}^{m \times n}_r,\qquad t = 0,\ldots,T, \tag*{(1)}

where Mrm×n\mathcal{M}^{m \times n}_r denotes the manifold of rank-rr matrices in Rm×n\mathbb{R}^{m \times n}, with r≪min⁡{m,n}r \ll\min\{m,n\}. Thus, for this block, the update trajectory stays on the same fixed-rank manifold. This turns fixed-rank structure from a property of an individual ZO step into a constraint on the persistent update trajectory. Building on this fixed-rank update trajectory, we propose a forward-only Riemannian ZO fine-tuning framework. At each step, we project a random matrix onto the current tangent space, normalize it, and retract the update back to the rank-rr manifold to build the ZO Riemannian gradient. We use this gradient to design Zeroth-order Riemannian Gradient Descent (ZO-RGD) and a smoothed variant (ZO-RGD-S). Figure 1 illustrates the tangent random matrix and the retracted update for one representative matrix block. We summarize the contributions as follows:

Two loss evaluations determine a fixed-rank ZO-RGD update

Figure 1. Two loss evaluations determine a fixed-rank ZO-RGD update. The orthogonal projection onto TΔWtMrT_{\Delta W_t}\mathcal{M}_r gives Z^t=PTΔWtMr(Zt)\widehat{Z}_t=P_{T_{\Delta W_t}\mathcal{M}_r}(Z_t), and normalization gives Z~t=mn Z^t/∥Z^t∥F\widetilde{Z}_t=\sqrt{mn}\,\widehat{Z}_t/\lVert\widehat{Z}_t\rVert_{\mathrm{F}}. The two points ΔWt±ϵZ~t\Delta W_t\pm\epsilon\widetilde{Z}_t along these directions jointly perturb all selected blocks to obtain F±F_{\pm}. Their difference gives the directional derivative s^t\hat{s}_t and thus the zeroth-order Riemannian gradient G^t\widehat{G}_t. Truncated SVD retracts ΔWt−ηG^t\Delta W_t-\eta\widehat{G}_t back to the rank-rr manifold.

  1. We formulate forward-only LLM fine-tuning as optimization of persistent additive updates on fixed-rank matrix manifolds. For each selected matrix block, the update trajectory is maintained on a prescribed rank-rr manifold, so the rank constraint is imposed on the optimization state rather than only on an individual ZO step.

  1. We show that tangent projection makes the finite-difference perturbation scale depend on the rank and layer shape, and correct this mismatch with a normalized tangent estimator that needs only two forward passes and no additional retractions per step. Building on this estimator, we develop ZO-RGD and its smoothed variant ZO-RGD-S, which recursively smooths the finite-difference scalar at the same forward cost.

  1. We prove that the implemented ambient finite difference agrees with its retracted counterpart to leading order, and that the normalized estimator recovers the Riemannian gradient in expectation up to an O(ϵ2)O(\epsilon^2) bias. Building on these properties, we establish a nonasymptotic convergence guarantee for ZO-RGD, conditional on regularity of the realized trajectory.

  1. We evaluate the proposed approaches across model families and downstream tasks against existing forward-only baselines. The results show competitive performance, with gains in several settings.

Related Work

Low-rank adaptation. LoRA adapts a frozen pretrained model by adding a low-rank update to selected weight matrices [16]. This idea has led to a large family of PEFT methods, including weight-decomposed adaptation [21], adaptive rank allocation and learning-rate asymmetry [14] (Zhang et al. 2023), nonzero initialization and analyses of low-rank training dynamics [20, 41], tensorized update parameterizations [40], rank-one or high-rank structured variants [31, 17], low-rank gradient projection (Zhao et al., 2024a), and broader PEFT surveys [38]. These methods establish low-rank updates as an effective adaptation object. In a forward-only ZO setting, we ask a different question: whether the update matrix itself can be the constrained optimization state, rather than representing it through nonunique factor coordinates.

Zeroth-order fine-tuning. Classical ZO theory studies simultaneous perturbation, Gaussian or spherical smoothing, and two-point randomized estimators [33, 13, 12, 27, 30, 4, 3]. MeZO showed that such forward-only estimates can fine-tune LLMs while avoiding activation-gradient storage [24]. Subsequent work has expanded the optimizer and benchmark landscape through large-scale studies [8], benchmark-oriented analyses (Zhang et al., 2024), random-subspace or low-rank perturbation methods [9, 39] (Yu et al. 2025), and variants that use curvature or projected-alignment information [36, 25] (Zhao et al. 2024b). Other approaches explore sparse perturbations [22] and step-size adaptation based on batch-loss variability [10]. LOZO [9] is the closest predecessor because it introduces low-rank structure into ZO estimation for LLMs. The distinction is where the structure lives: LOZO forms a low-rank estimator or subspace update, whereas the proposed algorithms construct their random matrices in the tangent space of the current fixed-rank updates.

Riemannian optimization on fixed-rank matrix manifolds. Fixed-rank matrix manifolds, tangent projections, and retractions are standard tools for optimizing low-rank matrices without exposing redundant factor gauges [1, 34, 37]. Riemannian ZO optimization studies stochastic optimization over Riemannian submanifolds embedded in Euclidean space when only noisy function evaluations are available [18]. Recent deep-learning work uses this geometry to optimize low-rank weights directly or to design Riemannian low-rank optimizers and preconditioners [5, 6, 26] (Zhang & Pilanci 2024). RoZO [32] is a closely related concurrent work that combines ZO optimization with geometry induced by low-rank adapter factors, while our method takes the matrix update itself as the optimization variable on the fixed-rank manifold. Riemannian ZO methods establish convergence for exponential-map or retraction-based finite differences under geometric and smoothness assumptions [19, 15]. Structure-preserving metrics have also been proposed to address geodesic incompleteness [23].

Preliminaries

Zeroth-order optimization

For a stochastic objective f(X)=Eξ[F(X;ξ)]f(X) = \mathbb{E}_{\xi}[F(X;\xi)], ZO methods estimate descent directions from function values. Given a data realization ξ\xi and a random matrix ZZ with i.i.d. standard Gaussian entries, the two-point estimator is

∇^F(X;ξ)=F(X+ϵZ;ξ)−F(X−ϵZ;ξ)2ϵZ,(2)\widehat{\nabla}F(X;\xi) = \frac{F(X+\epsilon Z;\xi)-F(X-\epsilon Z;\xi)}{2\epsilon}Z, \tag*{(2)}

where ϵ>0\epsilon> 0 controls the perturbation radius [12, 27, 30, 4, 3]. The scalar difference estimates a directional derivative, together with ZZ, gives a randomized gradient estimate. MeZO uses two forward passes and regenerates ZZ from seeds to avoid backpropagation and storing perturbations [24].

Fixed-rank manifold geometry

For a matrix block, we can define the fixed-rank manifold as

Mrm×n={X∈Rm×n:rank⁡(X)=r},dr=r(m+n−r),(3)\mathcal{M}_{r}^{m \times n} = \{X \in\mathbb{R}^{m \times n} : \operatorname{rank}(X) = r\}, \qquad d_r = r(m+n-r), \tag*{(3)}

with the Frobenius metric ⟨G1,G2⟩F=tr⁡(G1TG2)\langle G_1,G_2\rangle_{\mathrm{F}} = \operatorname{tr}(G_1^{\mathsf{T}}G_2). We write Mr\mathcal{M}_r when the block dimensions are clear. At X=UΣVTX = U\Sigma V^{\mathsf{T}}, the thin-SVD factors U,VU,V have orthonormal columns and Σ\Sigma contains positive singular values. The Frobenius-orthogonal projection of a matrix ZZ onto the tangent space TXMrT_X\mathcal{M}_r is [1, 34, 37] denoted as

PTXMr(Z)=UUTZ+ZVVT−UUTZVVT.(4)P_{T_X\mathcal{M}_r}(Z) = UU^{\mathsf{T}}Z + ZVV^{\mathsf{T}} - UU^{\mathsf{T}}ZVV^{\mathsf{T}}. \tag*{(4)}

The remaining matrix is an admissible first-order direction at the current update, and the tangent space changes as that update moves. For a step Y∈TXMrY \in T_X\mathcal{M}_r, we can return it to Mr\mathcal{M}_r using the rank-rr truncated-SVD map Hr\mathcal{H}_r:

RX(Y)=Hr(X+Y).(5)R_X(Y) = \mathcal{H}_r(X+Y). \tag*{(5)}

The map retains the leading rr singular triplets and gives the closest rank-rr matrix in Frobenius distance locally. It is a smooth local retraction on suitable neighborhoods away from the rank-deficient boundary [1, 34, 5].

Proposed method

Problem formulation

Let S\mathcal{S} index the selected matrix blocks, with frozen pretrained values W0,ℓ\mathbf{W}_{0,\ell} and learned additive updates ΔWℓ∈Mrℓmℓ×nℓ\Delta\mathbf{W}_{\ell} \in\mathcal{M}_{r_{\ell}}^{m_{\ell} \times n_{\ell}}. The effective weights are W0,ℓ+ΔWℓ\mathbf{W}_{0,\ell} + \Delta\mathbf{W}_{\ell}. Writing E=∏ℓ∈SRmℓ×nℓ\mathcal{E} = \prod_{\ell\in\mathcal{S}} \mathbb{R}^{m_{\ell} \times n_{\ell}}, the matrix updates lie on the product manifold:

M=∏ℓ∈SMrℓmℓ×nℓ⊂E.(6)\mathcal{M} = \prod_{\ell\in\mathcal{S}} \mathcal{M}_{r_{\ell}}^{m_{\ell} \times n_{\ell}} \subset\mathcal{E}. \tag*{(6)}

Trainable one-dimensional parameters, including biases and LayerNorm parameters, form a Euclidean factor b∈Rd0b \in\mathbb{R}^{d_0}, while all unselected parameters remain frozen. Thus the full trainable state belongs to the product manifold X=M×Rd0\mathcal{X} = \mathcal{M} \times\mathbb{R}^{d_0}. Unlike Euclidean factor coordinates, this representation is used only to compute tangent projections and retractions of the matrix variable, and the method is invariant to the choice of representative (see Appendix § 8.§ 1). The loss function is defined on the surrounding Euclidean space:

F(ΔW,b;ξ):=L(W0+ΔW,b;ξ),(ΔW,b)∈E×Rd0.(7)F(\Delta\mathbf{W}, b; \xi) := \mathcal{L}(\mathbf{W}_0 + \Delta\mathbf{W}, b; \xi), \quad(\Delta\mathbf{W}, b) \in\mathcal{E} \times\mathbb{R}^{d_0}. \tag*{(7)}

It couples all blocks and need not decompose over them. We propose to minimize its expectation formulated on the product manifold X\mathcal{X} subject to the mixed constraint:

min⁡(ΔW,b)∈Xf(ΔW,b):=Eξ[F(ΔW,b;ξ)].(8)\min_{(\Delta\mathbf{W},b)\in\mathcal{X}} f(\Delta\mathbf{W}, b) := \mathbb{E}_{\xi}[F(\Delta\mathbf{W}, b; \xi)]. \tag*{(8)}

ZO Riemannian Gradient Estimation

We first describe one representative matrix block, suppressing its index. In the full model, independent blockwise random matrices are applied jointly to the coupled loss, and two forward evaluations produce one scalar coefficient shared by all updates.

Finite-difference scale mismatch: At ΔWt∈Mr\Delta\mathbf{W}_t \in\mathcal{M}_r, we draw a Gaussian matrix Zt∈Rm×n\mathbf{Z}_t \in\mathbb{R}^{m \times n} with i.i.d. standard normal entries and project it using the orthogonal projection in (4):

Z^t=PTΔWtMr(Zt).(9)\widehat{\mathbf{Z}}_t = P_{T_{\Delta\mathbf{W}_t}\mathcal{M}_r}(\mathbf{Z}_t). \tag*{(9)}

This gives a valid tangent direction, but changes the scale of the finite differences. The ambient Gaussian satisfies ∥Zt∥F2∼χmn2\lVert\mathbf{Z}_t \rVert_F^2 \sim\chi^2_{mn}, whereas its tangent projection has ∥Z^t∥F2∼χdr2\lVert\widehat{\mathbf{Z}}_t \rVert_F^2 \sim\chi^2_{d_r}, where dr=r(m+n−r)d_r = r(m+n-r). In the low-rank regime, dr≪mnd_r \ll mn, so a fixed ϵ\epsilon yields a typical perturbation radius ϵdr\epsilon\sqrt{d_r} instead of ϵmn\epsilon\sqrt{mn}. Consequently, changing the rank or layer shape also changes the perturbation scale, even when the nominal finite-difference parameter remains fixed.

Normalized gradient estimator: To remove this dependence of the perturbation scale on the rank and layer shape, we decouple the scale from the tangent projection by normalizing Z^t\widehat{\mathbf{Z}}_t:

References

  1. [1]P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008. ISBN 9780691132983. doi: 10.1515/9781400830244. URL https://press.princeton.edu/books/hardcover/9780691132983/optimization-algorithms-on-matrix-manifolds.DOI
  2. [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
  3. [3]Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computa- tional Mathematics, 22(1):35–76, 2022. doi: 10.1007/s10208-021-09499-8. URL https://doi.org/10.1007/s10208-021-09499-8.
  4. [4]Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507–560, 2022. doi: 10.1007/s10208-021-09513-z. URL https://doi.org/10.1007/s10208-021-09513-z.
  5. [5]Fengmiao Bian, Jinyang Zheng, Ziyun Liu, Jianzhuo Luo, and Jian-Feng Cai. Finding low-rank matrix weights in DNNs via Riemannian optimization: RAdamW and RAdamW. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=tiGFiCrmKm.DOI
  6. [6]Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba. LoRA meets Riemannion: Muon optimizer for parametrization-independent low-rank adapters. arXiv:2507.12142, July 2025. URL https://arxiv.org/abs/2507.12142.
  7. [7]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  8. [8]Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kaulkhura, and Sijia Liu. DeepZero: Scaling up zeroth-order optimization for deep model training. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=qBWhjsNPEY.arxiv.org/abs/2310.02025
  9. [9]Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=9BiVepgmWW.arxiv.org/abs/2410.07698
  10. [10]Sizhe Dang, Haishan Ye, Yangyang Guo, Xiaodong Zheng, Yanjun Zhao, Guang Dai, et al. FZOO: Fast zeroth-order optimizer for fine-tuning large language models towards Adam-scale speed. arXiv preprint arXiv:2506.09034, 2025.arxiv.org/abs/2506.09034
  11. [11]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. URL https://aclanthology.org/N19-1246/.DOI
  12. [12]John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations. IEEE Trans. Inf. Theory, 61(5):2788–2806, Mar. 2015. doi: 10.1109/TIT.2015.2409256. URL https://doi.org/10.1109/TIT.2015.2409256.
  13. [13]Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. doi: 10.1137/120880811. URL https://doi.org/10.1137/120880811.
  14. [14]Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 17783–17806. PMLR, July 2024. URL https://proceedings.mlr.press/v235/hayou24a.html.arxiv.org/abs/2402.12354
  15. [15]Chang He, Zhaoye Pan, Xiao Wang, and Bo Jiang. Riemannian accelerated zeroth-order algorithm: Improved robustness and lower query complexity. arXiv preprint arXiv:2405.05713, 2024.arxiv.org/abs/2405.05713
  16. [16]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  17. [17]Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient Hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TwJrTz9CRS.
  18. [18]Jiaxiang Li, Krishnakumar Balasubramanian, and Shiqian Ma. Stochastic zeroth-order Riemannian derivative estimation and optimization. Mathematics of Operations Research, 48(2):1183–1211, 2022. doi: 10.1287/moor.2022.1302. URL https://pubsonline.informs.org/doi/abs/10.1287/moor.2022.1302.DOI
  19. [19]Jiaxiang Li, Krishnakumar Balasubramanian, and Shiqian Ma. Zeroth-order riemannian averaging stochastic approximation algorithms. arXiv preprint arXiv:2309.14506, 2023.arxiv.org/abs/2309.14506
  20. [20]Shiwei Li, Xiandi Luo, Xing Tang, Haozhao Wang, Hao Chen, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Beyond zero initialization: Investigating the impact of non-zero initialization on LoRA fine-tuning dynamics. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 35519–35535. PMLR, July 2025. URL https://proceedings.mlr.press/v267/li25bm.html.arxiv.org/abs/2505.23194
  21. [21]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32100–32121. PMLR, July 2024a. URL https://proceedings.mlr.press/v235/liu24bn.html.arxiv.org/abs/2402.09353
  22. [22]Yong Liu, Zirui Zhu, Chaoyou Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse MeZO: Less parameters for better performance in zeroth-order LLM fine-tuning. arXiv preprint arXiv:2402.15751, 2024b.arxiv.org/abs/2402.15751
  23. [23]Shaocong Ma and Heng Huang. Riemannian zeroth-order gradient estimation with structure-preserving metrics for geodesically incomplete manifolds. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2601.08039
  24. [24]Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems, volume 36, pp. 53038–53075, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/a627810151be4d13f9907ac889ff7e948-Abstract-Conference.html.arxiv.org/abs/2305.17333
  25. [25]Zhendong Mi, Qitao Tan, Grace Li Zhang, Zhaozhuo Xu, Geng Yuan, and Shaoyi Huang. Towards fast LLM fine-tuning through zeroth-order optimization with projected gradient-aligned perturbations. arXiv:2510.18228, Oct. 2025. URL https://arxiv.org/abs/2510.18228.
  26. [26]Zhanfeng Mo, Long-Kai Huang, and Sinno Jialin Pan. Parameter and memory efficient pretraining via low-rank Riemannian optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=i0zzO7Hslk.
  27. [27]Yurii Nesterov and Vladimir Spokoiny. Random Gradient-Free Minimization of Convex Functions. Found. Comput. Math., 17(2):527–566, Apr. 2017. ISSN 1615-3383. doi: 10.1007/s10208-015-9296-2. URL https://doi.org/10.1007/s10208-015-9296-2.
  28. [28]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264/.DOI
  29. [29]Reinhold Schneider and André Uschmajew. Convergence results for projected line-search methods on varieties of low-rank matrices via Łojasiewicz inequality. SIAM Journal on Optimization, 25(1):622–646, 2015. doi: 10.1137/140957822. URL https://doi.org/10.1137/140957822.
  30. [30]Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research, 18(52):1–11, 2017. URL https://jmlr.org/papers/v18/16-632.html.
  31. [31]Hao Nan Sheng, Zhi-Yong Wang, Mingrui Yang, and Hing Cheung So. AROMA: Autonomous rank-one matrix adaptation. arXiv:2504.05343, Apr. 2025. URL https://arxiv.org/abs/2504.05343.
  32. [32]Zichen Song and Weijia Li. RoZO: Geometry-aware zeroth-order fine-tuning on low-rank adapters for black-box large language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1773–1783, Rabat, Morocco, Mar. 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.eacl-long.80. URL https://aclanthology.org/2026.eacl-long.80/.DOI
  33. [33]James C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341, Mar. 1992. doi: 10.1109/9.119632. URL https://doi.org/10.1109/9.119632.
  34. [34]Bart Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 23(2):1214–1236, 2013. doi: 10.1137/110845768. URL https://doi.org/10.1137/110845768.
  35. [35]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html.arxiv.org/abs/1905.00537
  36. [36]Shuo Wang, Ziyu Chen, and Ming Tang. CurvZO: Adaptive curvature-guided sparse zeroth-order optimization for efficient LLM fine-tuning. arXiv:2603.21725, Mar. 2026. URL https://arxiv.org/abs/2603.21725.
  37. [37]Ke Wei, Jian-Feng Cai, Tony F. Chan, and Shingyu Leung. Guarantees of Riemannian optimization for low rank matrix recovery. SIAM Journal on Matrix Analysis and Applications, 37(3):1198–1222, 2016. doi: 10.1137/15M1050525. URL https://doi.org/10.1137/15M1050525.
  38. [38]Menglin Yang, Jialin Chen, Jinkai Tao, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, QiYao Ma, Harshit Verma, Regina Zhang, Min Zhou, Irwin King, and Rex Ying. Low-rank adaptation for foundation models: A comprehensive review. arXiv:2501.00365, Dec. 2024a. URL https://arxiv.org/abs/2501.00365.
  39. [39]Yifan Yang, Kai Zhen, Ershad Banijamali, Athanasios Mouchartis, and Zheng Zhang. AdaZeta: Adaptive zeroth-order tensor-train adaptation for memory-efficient large language models fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 977–995, Miami, Florida, USA, Nov. 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.56. URL https://aclanthology.org/2024.emnlp-main.56/.
  40. [40]Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. LoRETTA: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3161–3176, Mexico City, Mexico, June 2024c. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.174. URL https://aclanthology.org/2024.naacl-long.174/.
  41. [41]Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparameterized low-rank learning and adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 56946–56965. PMLR, July 2024. URL https://proceedings.mlr.press/v235/yaras24a.html.arxiv.org/abs/2406.04112

Paper details

vlnc:W_4p9tdb7j
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents