betaFeedback

WorksComputer Science

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Abstract

Training deep neural networks requires optimizers that balance computational efficiency with formal data privacy. While orthogonalized update rules, such as the recently introduced Muon optimizer, have shown strong empirical performance and scalability, their behavior under Differential Privacy (DP) is not yet understood. In this paper, we propose DP-Muon, a differentially private variant of the Muon optimizer. By analyzing Muon's orthogonalized updates through the lens of a Linear Minimization Oracle (LMO), we show that the leading effect of per-example clipping is a positive rescaling of the gradient, to which the LMO is blind, so that only a small directional residual controlled by the centered gradient variance enters the optimization error. This yields an O(K−1/4)\mathcal O(K^{-1/4}) convergence rate, up to logarithmic factors, to a clipping-dependent neighborhood at a fixed noise scale and, with noise calibrated to a fixed (ε,δ)(\varepsilon,δ) budget, the standard (d/(nε))1/2(\sqrt d/(n\varepsilon))^{1/2} privacy–utility bound of DP-SGD, without assuming bounded gradients. Furthermore, we investigate the hyperparameter landscape of DP-Muon and document how the optimal learning rate, momentum, and batch size change under privacy. Empirically, on image classification DP-Muon matches or improves on DP-SGD and DP-Adam, whereas on language modeling the best optimizer depends on the batch size, with DP-Muon best at small batches and DP-Adam best near the privacy-optimal batch size.

Introduction

Adaptive methods such as Adam ([37]) and AdamW ([50]) remain the default choice for training large models ([2, 46]). A prominent recent alternative is Muon ([32]), which applies an approximate orthogonalization to the momentum of matrix-valued parameters. Muon has attracted considerable attention due to its strong empirical performance, efficient GPU implementation, and reduced optimizer-state memory; its scalability has been demonstrated by training the Moonlight 16B Mixture-of-Experts model on 5.7 trillion tokens ([47]). Its update can be interpreted through stochastic spectral descent ([14, 41]) and, more generally, as a step along a linear minimization oracle (LMO) over a norm ball ([30, 66]). The LMO view gives a unified treatment of non-Euclidean methods with momentum, recovering normalized SGD ([43, 16]), SignSGD and Signum ([10, 75]), Scion ([66]), and Muon as special cases.

In this work, we study whether, and how, the advantages of momentum orthogonalization carry over to differential privacy (DP). We consider the standard central, example-level setting of DP-SGD ((Abadi et al., 2016): each per-example gradient is clipped to norm cc, Gaussian noise is added to the average, and the result is passed to the optimizer; the momentum and LMO steps are post-processing and do not affect privacy [21]. The optimization problem, however, becomes substantially more delicate. Active clipping introduces a state-dependent bias, privacy noise is injected at every step, momentum aggregates gradients along a changing trajectory, and the noisy momentum is then passed through a nonlinear map. Existing analyses of private nonconvex optimization typically sidestep active clipping: the sharpest rates assume Lipschitz losses, that is, uniformly bounded per-example gradients, so that clipping at a threshold above the bound is inactive [8, 79, 83, 7, 52], and analyses of private distributed methods make similar assumptions (Appendix § 13). Generic analyses of biased stochastic gradients [4, 20] measure the full additive bias and do not exploit the structure of the update.

Key observation: for an LMO, clipping is (almost) a rescaling. Let μ(g):=E[clip⁡c(g+ξ)]\mu(g) := \mathbb{E}[\operatorname{clip}_{c}(g + \xi)] be the expected clipped gradient, where gg is the true gradient and ξ\xi is zero-mean sampling noise with variance σ2\sigma^2. We show (Lemma 1) that

μ(g)=λ(g)g+r(g),λ(g)≍cmax⁡{c,∥g∥2},∥r(g)∥2≲λ(g)min⁡{σ,σ2c}.(1)\begin{aligned} \mu(g) &= \lambda(g)g + r(g), \qquad\lambda(g) \asymp\frac{c}{\max\{c,\lVert g\rVert_2\}},\\ \lVert r(g)\rVert_2 &\lesssim\lambda(g)\min\left\{\sigma,\frac{\sigma^2}{c}\right\}. \tag*{(1)} \end{aligned}

The leading effect of clipping is thus a positive rescaling of the true gradient, and an LMO is blind to it: LMO⁡(λg)=LMO⁡(g)\operatorname{LMO}(\lambda g) = \operatorname{LMO}(g) for every λ>0\lambda> 0. Only the residual r(g)r(g), which changes the direction, enters the optimization error. This lets us analyze momentum-LMO methods under active clipping assuming only bounded centered variance, without bounded gradients and without a clipping threshold that grows with the horizon.

Contributions. Our answer to the question in the title has a theoretical and an empirical part.

  • Clipping as rescaling. We prove the positive-ray decomposition (1) and a matching bound on the variance of clipped gradients relative to λ(g)\lambda(g) (§ 3).

  • Convergence of private momentum-LMO methods. For smooth nonconvex objectives in any finite-dimensional norm (covering Muon, Scion, sign descent, and normalized SGD), we prove a high-probability bound that separates optimization error, momentum tracking, sampling noise, privacy noise, and a clipping floor ρ1min⁡{σ,σ2/c}\rho_1\min\{\sigma,\sigma^2/c\} (Theorem 1). The floor is the dual-norm version of the tight Euclidean floor of clipped SGD [38] and vanishes as σ→0\sigma\to0; passing through the LMO thus costs at most the norm-conversion constant ρ1\rho_1 relative to clipped SGD, and the guarantee does not show that it improves on it. At a fixed noise scale, the bound gives the rate O~(K−1/4)\widetilde{O}(K^{-1/4}), where O~\widetilde{O} hides factors logarithmic in KK and in the failure probability; and with noise calibrated to an (ε,δ)(\varepsilon,\delta) budget it gives the familiar O((cE∥Z∥∗log⁡(1/δ)/(nε))1/2)O((c\mathbb{E}\lVert Z\rVert_*\sqrt{\log(1/\delta)}/(n\varepsilon))^{1/2}) dependence (Corollary 1, Appendix § 10.§ 2).

  • What changes under DP. On CNN and a 30M-parameter GPT model, we sweep the batch size, learning rate, and momentum for DP-SGD, DP-Adam, and DP-Muon (and Scion). Under DP, the optimal batch size grows by one to two orders of magnitude and the ranking of optimizers depends on the batch size, unlike in non-private training, where Muon is best at every batch size. On CIFAR-10, DP-Muon attains the lowest mean test loss at all but one batch size, with the gaps within one standard deviation at intermediate batch sizes; on language modeling, DP-Muon is best at small batch sizes, DP-Adam near the optimal batch size, and Scion at large batch sizes (§ 4).

Concurrent work. Two concurrent and independent works also study private Muon2. Kim and Fan (2026) analyze a Muon variant with finite-step Newton–Schulz (NS) orthogonalization and propose a bias-

Algorithm 1 DP-Muon (private momentum-LMO)

Require: stepsize γ\gamma, momentum α∈(0,1)\alpha\in(0,1), clipping threshold cc, noise scale σDP\sigma_{\mathrm{DP}}, batch size BB, iterations KK

  1. m0←0m_0 \gets0

  2. for k=0,…,K−1k = 0,\ldots,K-1 do

  3. Sample Sk⊆[n]S_k \subseteq[n], ∣Sk∣=B|S_k| = B, uniformly at random

  4. g^k←1B∑i∈Skclip⁡c(∇fi(xk))+zk, zk∼N(0,c2σDP2I)\hat{g}_k \gets\frac{1}{B}\sum_{i\in S_k}\operatorname{clip}_c(\nabla f_i(x_k)) + z_k,\ z_k \sim\mathcal{N}(0,c^2\sigma_{\mathrm{DP}}^2 I)

  5. mk+1←(1−α)mk+αg^km_{k+1} \gets(1-\alpha)m_k + \alpha\hat{g}_k

  6. xk+1←xk+γ LMO⁡(mk+1)x_{k+1} \gets x_k + \gamma\,\operatorname{LMO}(m_{k+1})

  7. end for

corrected DP-MuonBC; their vanishing-rate corollaries assume moment bounds on the raw (uncentered) per-example gradients and let the clipping threshold grow with the horizon, while their general bound contains a clipping term that persists even without sampling noise. Shanmugam et al. [70] study when orthogonalization recovers useful singular directions of privatized gradients through a Wedin-type signal-to-noise argument and report large gains of DP-Muon over DP-SGD in vision at large batch sizes. Our analysis is complementary: it covers general LMO geometries, requires only centered variance, and isolates the part of the clipping bias that an LMO actually sees; a detailed comparison is given in § 3.3 and Appendix § 12.

Setup and Algorithm

We consider min⁡x∈Xf(x):=1n∑i=1nfi(x)\min_{x\in\mathcal{X}} f(x) := \frac{1}{n}\sum_{i=1}^{n} f_i(x) over a finite-dimensional normed space (X,∥⋅∥)(\mathcal{X}, \|\cdot\|) with dual norm ∥v∥⋆:=sup⁡∥d∥≤1⟨v,d⟩\|v\|_\star:= \sup_{\|d\|\leq1}\langle v,d\rangle. For the unit ball C={v:∥v∥≤1}C = \{v : \|v\| \leq1\}, the linear minimization oracle is LMO⁡(g)∈arg⁡min⁡v∈C⟨g,v⟩\operatorname{LMO}(g) \in\arg\min_{v\in C}\langle g,v\rangle, so that ⟨g,LMO⁡(g)⟩=−∥g∥⋆\langle g,\operatorname{LMO}(g)\rangle= -\|g\|_\star and LMO⁡(λg)=LMO⁡(g)\operatorname{LMO}(\lambda g) = \operatorname{LMO}(g) for λ>0\lambda> 0. The spectral norm on a weight matrix gives Muon, LMO⁡(G)=−UV⊤\operatorname{LMO}(G) = -UV^\top for G=UΣV⊤G = U\Sigma V^\top; the ℓ∞\ell_\infty norm gives sign descent; the ℓ2\ell_2 norm gives normalized gradient descent; and layer-wise combinations give Scion (Pethick et al. [66]) (see Table 2 in the appendix). Since X\mathcal{X} is finite-dimensional, there are constants ρ1,ρ2\rho_1,\rho_2 with ∥v∥⋆≤ρ1∥v∥2\|v\|_\star\leq\rho_1\|v\|_2 and ∥v∥2≤ρ2∥v∥⋆\|v\|_2 \leq\rho_2\|v\|_\star; for the spectral norm on an m×n′m \times n' matrix, ∥⋅∥⋆\|\cdot\|_\star is the nuclear norm, ρ1=min⁡{m,n′}\rho_1 = \sqrt{\min\{m,n'\}}, and ρ2=1\rho_2 = 1. Clipping is Euclidean, clip⁡c(v):=vmin⁡{1,c/∥v∥2}\operatorname{clip}_c(v) := v\min\{1,c/\|v\|_2\}, as in DP-SGD.

References

  1. [1]Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS’16, page 308–318. ACM, October 2016. doi: 10.1145/2976749.2978318. URL http://dx.doi.org/10.1145/2976749.2978318. (Cited on page 1, 3, 4, and 38)
  2. [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. (Cited on page 1)arxiv.org/abs/2303.08774
  3. [3]Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025. (Cited on page 38)arxiv.org/abs/2504.05295
  4. [4]Ahmad Ajalloeian and Sebastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051, 2020. (Cited on page 2)arxiv.org/abs/2008.00051
  5. [5]Ya I Alber, Alfredo N Iusem, and Mikhail V Solodov. On the projected subgradient method for nonsmooth convex optimization in a hilbert space. Mathematical Programming, 81(1):23–35, 1998. (Cited on page 38)DOI
  6. [6]Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199, 2023. (Cited on page 6 and 37)arxiv.org/abs/1912.02365
  7. [7]Raman Arora, Raef Bassily, Tomás González, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Faster rates of convergence to stationary points in differentially private optimization. In International Conference on Machine Learning, 2023. (Cited on page 2, 34, and 38)arxiv.org/abs/2206.00846
  8. [8]Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. In Advances in Neural Information Processing Systems, 2021. (Cited on page 2 and 38)arxiv.org/abs/2107.05585
  9. [9]Jeremy Bernstein. Deriving muon. URL: https://jeremybernste.in/writing/deriving-muon, 2025. (Cited on page 38)
  10. [10]Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadnesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International conference on machine learning, pages 560–569. PMLR, 2018. (Cited on page 1)arxiv.org/abs/1802.04434
  11. [11]Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization, 2021. URL https://arxiv.org/abs/2102.06171. (Cited on page 38)
  12. [12]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165. (Cited on page 38)
  13. [13]Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. In Advances in Neural Information Processing Systems, 2023. (Cited on page 9 and 38)arxiv.org/abs/2206.07136
  14. [14]David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted boltzmann machines. In Artificial intelligence and statistics, pages 111–119. PMLR, 2015. (Cited on page 1 and 38)
  15. [15]Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. Advances in Neural Information Processing Systems, 33:13773–13782, 2020. (Cited on page 7, 37, and 38)arxiv.org/abs/2006.15429
  16. [16]Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pages 2260–2268. PMLR, 2020. (Cited on page 1, 5, 6, and 38)arxiv.org/abs/2002.03305
  17. [17]Ashok Cutkosky and Harsh Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. In Advances in Neural Information Processing Systems, 2021. (Cited on page 6 and 38)arxiv.org/abs/2106.14343
  18. [18]Rudrajit Das, Abolfazl Hashemi, Sujay Sanghavi, and Inderjit S Dhillon. On the convergence of differentially private federated learning on non-lipschitz objectives, and with normalized client updates. arXiv preprint arXiv:2106.07094, 2021. (Cited on page 38)arxiv.org/abs/2106.07094
  19. [19]Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022. (Cited on page 7 and 9)arxiv.org/abs/2204.13650
  20. [20]Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richtárik. A guide through the zoo of biased sgd. Advances in Neural Information Processing Systems, 36:23158–23171, 2023. (Cited on page 2)arxiv.org/abs/2305.16296
  21. [21]Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014. (Cited on page 2, 3, and 4)
  22. [22]Yuri Ermoliev. Stochastic quasigradient methods. numerical techniques for stochastic optimization. Springer Series in Computational Mathematics, (10):141–185, 1988. (Cited on page 38)
  23. [23]Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557, 2017. (Cited on page 38)arxiv.org/abs/1712.07557
  24. [24]Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems, 33:15042–15053, 2020. (Cited on page 38)arxiv.org/abs/2005.10785
  25. [25]Eduard Gorbunov, Nazarii Tuptitsa, Sayantan Choudhury, Alen Aliev, Peter Richtárik, Samuel Horváth, and Martin Takáč. Methods for convex (l-0, l-1)-smooth optimization: Clipping, acceleration, and adaptivity. arXiv preprint arXiv:2409.14989, 2024. (Cited on page 38)arxiv.org/abs/2409.14989
  26. [26]Kaja Gruntkowska, Alexander Gaponov, Zhirayr Tovmasyan, and Peter Richtárik. Error feedback for muon and friends. arXiv preprint arXiv:2510.00643, 2025a. (Cited on page 38)arxiv.org/abs/2510.00643
  27. [27]Kaja Gruntkowska, Yassine Maziane, Zheng Qu, and Peter Richtárik. Drop-muon: Update less, converge faster. arXiv preprint arXiv:2510.02239, 2025b. (Cited on page 38)arxiv.org/abs/2510.02239
  28. [28]Florian Hübler, Junchi Yang, Xiang Li, and Niao He. Parameter-agnostic optimization under relaxed smoothness. In International Conference on Artificial Intelligence and Statistics, pages 4861–4869. PMLR, 2024. (Cited on page 38)arxiv.org/abs/2311.03252
  29. [29]Rustem Islamov, Grigory Malinovsky, Alexander Gaponov, Aurelien Lucchi, Peter Richtárik, and Eduard Gorbunov. Byzantine-robust and differentially private federated optimization under weaker assumptions. arXiv preprint arXiv:2603.23472, 2026. (Cited on page 38)arxiv.org/abs/2603.23472
  30. [30]Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning, pages 427–435. PMLR, 2013. (Cited on page 1)
  31. [32]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiac heng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon, 6(3):4, 2024. (Cited on page 1 and 7)
  32. [33]Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for byzantine robust optimization. In International conference on machine learning, pages 5311–5319. PMLR, 2021. (Cited on page 38)arxiv.org/abs/2012.10333
  33. [34]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. Muonbp: Faster muon via block-periodic orthogonalization. arXiv preprint arXiv:2510.16981, 2025. (Cited on page 38)arxiv.org/abs/2510.16981
  34. [37]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. (Cited on page 1)arxiv.org/abs/1412.6980
  35. [38]Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In International Conference on Machine Learning, pages 17343–17363. PMLR, 2023. (Cited on page 2, 4, 6, 7, 37, and 38)
  36. [40]Dmitry Kovalev and Ekaterina Borodich. Non-euclidean sgd for structured optimization: Unified analysis and improved rates. arXiv preprint arXiv:2511.11466, 2025. (Cited on page 9 and 38)arxiv.org/abs/2511.11466
  37. [41]Dmitry Kovalev, Peter Richtarik, Eduard Gorbunov, and Elnur Gasanov. Stochastic spectral and conjugate descent methods. Advances in Neural Information Processing Systems, 31, 2018. (Cited on page 1 and 38)arxiv.org/abs/1802.03703
  38. [42]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. (Cited on page 6)
  39. [43]Kfir Y Levy. The power of normalization: Faster evasion of saddle points. arXiv preprint arXiv:1611.04831, 2016. (Cited on page 1)arxiv.org/abs/1611.04831
  40. [44]Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners, 2022a. URL https://arxiv.org/abs/2110.05679. (Cited on page 10)
  41. [45]Zhize Li, Haoyu Zhao, Boyue Li, and Yuejie Chi. Soteriafl: A unified framework for private federated learning with communication compression. Advances in Neural Information Processing Systems, 35:4285–4300, 2022b. (Cited on page 38)arxiv.org/abs/2206.09888
  42. [46]Aixin Liu, Bei Feng, Bingxue Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. (Cited on page 1)arxiv.org/abs/2412.19437
  43. [47]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. (Cited on page 1)arxiv.org/abs/2502.16982
  44. [48]Mingrui Liu, Zhenxun Zhuang, Yunwen Lei, and Chunyang Liao. A communication-efficient distributed gradient clipping algorithm for training deep neural networks. Advances in Neural Information Processing Systems, 35:26204–26217, 2022. (Cited on page 38)arxiv.org/abs/2205.05040
  45. [49]Aleksandr Lobanov, Alexander Gasnikov, Eduard Gorbunov, and Martin Takáč. Linear convergence rate in convex setup is possible! gradient descent method variants under (l_0,l_1)-smoothness. arXiv preprint arXiv:2412.17050, 2024. (Cited on page 38)arxiv.org/abs/2412.17050
  46. [50]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. (Cited on page 1)arxiv.org/abs/1711.05101
  47. [51]Andrew Lowy, Ali Ghafelebashi, and Meisam Razaviyayn. Private non-convex federated learning without a trusted server. In International Conference on Artificial Intelligence and Statistics, pages 5749–5786. PMLR, 2023. (Cited on page 38)arxiv.org/abs/2203.06735
  48. [52]Andrew Lowy, Jonathan Ullman, and Stephen J. Wright. How to make the gradients small privately: Improved rates for differentially private non-convex optimization. In International Conference on Machine Learning, 2024. (Cited on page 2, 34, and 38)arxiv.org/abs/2402.11173
  49. [53]Grigory Malinovsky, Peter Richtárik, Samuel Horváth, and Eduard Gorbunov. Byzantine robustness and partial participation can be achieved at once: Just clip gradient differences. Advances in Neural Information Processing Systems, 37:34900–34979, 2024. (Cited on page 38)arxiv.org/abs/2311.14127
  50. [54]Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, et al. Scaling laws for differentially private language models. arXiv preprint arXiv:2501.18914, 2025. (Cited on page 9)arxiv.org/abs/2501.18914
  51. [55]H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018. (Cited on page 38)
  52. [56]Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models, 2017. URL https://arxiv.org/abs/1708.02182. (Cited on page 38)
  53. [57]Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov, Aurelien Lucchi, and Anastasiia Koloskova. Adaptive methods are preferable in high privacy settings: An SDE perspective. In International Conference on Learning Representations, 2026. (Cited on page 9 and 38)
  54. [58]Tomoya Murata and Taiji Suzuki. Diff2: Differential private optimization via gradient differences for nonconvex distributed learning. In International Conference on Machine Learning, pages 25523–25548. PMLR, 2023. (Cited on page 38)arxiv.org/abs/2302.03884
  55. [59]Ta Duy Nguyen, Thien H Nguyen, Alina Ene, and Huy Nguyen. Improved convergence in high probability of clipped gradient methods with heavy tailed noise. Advances in Neural Information Processing Systems, 36:24191–24222, 2023. (Cited on page 38)
  56. [60]Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially private federated learning on heterogeneous data. In International conference on artificial intelligence and statistics, pages 10110–10145. PMLR, 2022. (Cited on page 38)
  57. [61]Ashwinee Panda, Xinyu Tang, Saeed Mahloujifar, Vikash Sehwan, and Prateek Mittal. A new linear scaling rule for private adaptive hyperparameter optimization. In International Conference on Machine Learning, 2024. (Cited on page 9)
  58. [62]Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with Rényi differential privacy. In International Conference on Learning Representations, 2022. (Cited on page 7 and 9)
  59. [63]Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks, 2013. URL https://arxiv.org/abs/1211.5063. (Cited on page 38)
  60. [64]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. (Cited on page 39)arxiv.org/abs/1912.01703
  61. [65]Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557. (Cited on page 7)
  62. [66]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529, 2025. (Cited on page 1, 3, 5, and 38)arxiv.org/abs/2502.07529
  63. [67]Xun Qian, Alexander Gaponov, Grigory Malinovsky, and Peter Richtárik. Communication-efficient gluon in federated learning. arXiv preprint arXiv:2604.10689, 2026. (Cited on page 38)arxiv.org/abs/2604.10689
  64. [68]Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms). arXiv preprint arXiv:2505.13416, 2025. (Cited on page 9 and 38)arxiv.org/abs/2505.13416
  65. [69]Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without a burn: Scaling laws of dp-sgd, 2023. URL https://arxiv.org/abs/2210.03403. (Cited on page 7 and 9)
  66. [70]Sabari Shanmugam, Nick Barnes, and Kerry Taylor. Spectral gradient orthogonalization improves differentially private training at scale. arXiv preprint arXiv:2608.17415, 2026. (Cited on page 3, 8, and 38)arxiv.org/abs/2608.17415
  67. [71]Naum Zuselevich Shor. Minimization methods for non-differentiable functions. Springer Science & Business Media, 2012. (Cited on page 38)
  68. [72]Egor Shulgin and Peter Richtárik. On the convergence of DP-SGD with adaptive clipping. In NeurIPS 2024 Workshop on Optimization for Machine Learning, 2024. arXiv:2412.19916. (Cited on page 7 and 37)arxiv.org/abs/2412.19916
  69. [73]Egor Shulgin, Sultan AlRashed, Francesco Orobono, and Peter Richtárik. Beyond the ideal: Analyzing the inexact muon update. arXiv preprint arXiv:2510.19933, 2025. (Cited on page 9 and 38)arxiv.org/abs/2510.19933
  70. [74]Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang, Niccolò Ajroldi, Bernhard Schölkopf, and Antonio Orvieto. Deriving hyperparameter scaling laws via modern optimization theory. arXiv preprint arXiv:2603.15958, 2026. (Cited on page 9)arxiv.org/abs/2603.15958
  71. [75]Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signsgd under weaker assumptions. In International Conference on Machine Learning, pages 33077–33099. PMLR, 2023. (Cited on page 1)
  72. [76]Qiaoyu Tang, Frederick Shpilevskiy, and Mathias Lécuyer. DP-AdamBC: Your DP-Adam is actually DP-SGD (unless you apply bias correction). In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. (Cited on page 9 and 38)arxiv.org/abs/2312.14334
  73. [77]Benjamin Thérien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky. Muoco: Muon is a practical inner optimizer for diloco. arXiv preprint arXiv:2505.29275, 2025. (Cited on page 38)arxiv.org/abs/2505.23725
  74. [78]Zhirayr Tovmasyan, Grigory Malinovsky, Laurent Condat, and Peter Richtárik. Revisiting stochastic proximal point methods: Generalized smoothness and similarity. arXiv preprint arXiv:2502.03401, 2025. (Cited on page 38)arxiv.org/abs/2502.03401
  75. [79]Hoang Tran and Ashok Cutkosky. Momentum aggregation for private non-convex ERM. In Advances in Neural Information Processing Systems, 2022. (Cited on page 2, 34, and 38)arxiv.org/abs/2210.06328
  76. [80]Aleksei Triastcyn and Boi Faltings. Federated learning with bayesian differential privacy. In 2019 IEEE International Conference on Big Data (Big Data), pages 2587–2596. IEEE, 2019. (Cited on page 38)DOI
  77. [81]Daniil Vankov, Anton Rodomanov, Angelia Nedich, Lalitha Sankar, and Sebastian U Stich. Optimizing (l-0, l-1)-smooth functions by gradient methods. arXiv preprint arXiv:2410.10800, 2024. (Cited on page 38)arxiv.org/abs/2410.10800
  78. [82]Saleh Vatan Khah, Savellii Chezhegov, Shahrokh Farahmand, Samuel Horváth, and Eduard Gorbunov. Differentially private clipped-SGD: High-probability convergence with arbitrary clipping level. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS), 2026. arXiv:2507.23512. (Cited on page 7 and 37)
  79. [83]Lingxiao Wang, Bargav Javaraman, David Evans, and Quanquan Gu. Efficient privacy-preserving stochastic nonconvex optimization. In Uncertainty in Artificial Intelligence, pages 2203–2213. PMLR, 2023. (Cited on page 2 and 38)
  80. [84]Lingxiao Wang, Xingyu Zhou, Kumar Kshitij Patel, Lawrence Tang, and Aadipa Saha. Efficient private federated non-convex optimization with shuffled model. In Privacy Regulation and Protection in Machine Learning, 2024. (Cited on page 38)
  81. [85]Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled Rényi differential privacy and analytical moments accountant. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), 2019. (Cited on page 3, 4, and 34)arxiv.org/abs/1808.00087
  82. [86]Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas. A theory to instruct differentially-private learning via clipping bias reduction. In IEEE Symposium on Security and Privacy (SP), 2023. (Cited on page 4)DOI
  83. [87]Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Normalized/clipped sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXiv:2206.13033, 2022. (Cited on page 7, 9, 33, 37, and 38)arxiv.org/abs/2206.13033
  84. [88]Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298, 2021. (Cited on page 7 and 39)arxiv.org/abs/2109.12298
  85. [89]Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019. (Cited on page 38)arxiv.org/abs/1905.11881
  86. [90]Meifan Zhang, Zhanhong Xie, and Lihua Yin. Private and communication-efficient federated learning based on differentially private sketches. arXiv preprint arXiv:2410.05733, 2024. (Cited on page 38)arxiv.org/abs/2410.05733
  87. [91]Xin Zhang, Minghong Fang, Jia Liu, and Zhengyuan Zhu. Private and communication-efficient edge learning: A sparse differential gaussian-masking distributed sgd approach. In Proceedings of the Twenty-First International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pages 261–270, 2020. (Cited on page 38)arxiv.org/abs/2001.03836
  88. [92]Xinwei Zhang, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Jinfeng Yi. Understanding clipping for federated learning: Convergence and client-level differential privacy. In International Conference on Machine Learning, ICML 2022, 2022. (Cited on page 38)arxiv.org/abs/2106.13673

Paper details

vlnc:W_e4ywb7hr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents