betaFeedback

WorksComputer Science

Byzantine-Robust Muon: Resilient Momentum Orthogonalization for Distributed Learning

Abstract

Optimization methods, including the Muon optimizer based on the Linear Minimization Oracle (LMO) and momentum orthogonalization, have shown strong empirical performance in large-scale and distributed learning, yet their robustness to adversarial behavior and data heterogeneity is not well understood. We study these methods within a unified first-order trust-region framework and analyze Muon-based updates with robust aggregation for Byzantine-robust distributed and federated learning. Our analysis does not rely on the i.i.d. assumption and introduces a non-Euclidean notion of heterogeneity consistent with the trust-region geometry. Under this framework, we establish convergence guarantees for both general non-convex and star-convex objectives. Our results provide a theoretical characterization of Byzantine-robust Muon-based optimization, with an additive heterogeneity error floor consistent with known impossibility results, under explicit aggregation assumptions. Under an explicit additive oracle-error condition, the guarantees also cover inexact orthogonalization, with the non-convex Byzantine terms in the bound's numerator unchanged but a smaller progress denominator. In simulated federated training with ResNet-18 and ViT-Tiny on CIFAR-10, additional image benchmarks, and a 38M-parameter byte-level language model on TinyStories, the tested Muon configuration attains higher attacked accuracy (lower perplexity) than the tested SGD and normalized-SGD configurations, and than sign updates on CIFAR-10. Fixed-honest controls further show a substantial RFA advantage over averaging under a bounded RFA-query attack, higher attacked accuracy for Muon-NS than tensorwise Euclidean normalization from shared checkpoints, and lower held-out TinyStories perplexity than a tuned AdamW baseline.

Introduction

Deep learning has become a dominant paradigm in modern machine learning [58], particularly with the remarkable success of large language models (LLMs) [88]. Training such models requires substantial computational and memory resources, so optimization methods play a central role: they can significantly accelerate training and improve stability [56]. In practice, adaptive methods, most notably Adam [50] and AdamW [62], have become the standard choice due to their fast and reliable convergence across a wide range of architectures [1, 60, 33, 20].

Recently, a new class of optimizers, such as Muon [44] and Scion [65], has been proposed, leveraging geometry-aware adaptation via momentum orthogonalization, with notable improvements on large language model (LLM) tasks [61]. At its core, Muon can be viewed as stochastic spectral descent (SSD) [17, 18, 19] — steepest descent with respect to the spectral norm [12] — augmented with momentum, where the singular value decomposition required for exact spectral descent is replaced by efficient Newton–Schulz iterations [53, 14, 11, 7, 73, 65]. From a theoretical perspective, the Muon optimizer admits several interpretations. One line of work connects Muon to linear minimization oracle (LMO) methods, such as Frank–Wolfe [43]: Pethick et al. [2025a] propose a general framework for norm-constrained optimization with a unified analysis of non-Euclidean stochastic gradient descent with momentum under arbitrary norms, recovering SignSGD [13], Signum [76], and Muon, with clear connections to adaptive methods; subsequent works extend the framework to gradient clipping and generalized smoothness [66, 70].

A related interpretation and generalization is proposed in Kovalev [ [52], which analyzes Muon within the framework of non-Euclidean trust-region methods [24], explains why orthogonalizing the momentum rather than the raw gradient [80] is beneficial, recovers normalized SGD with momentum [21], SignSGD, Signum, and Muon, and naturally incorporates weight decay; subsequent work demonstrates advantages of non-Euclidean optimization arising from the structure of stochastic gradient noise [52]. Recent works [85, 49, 68] develop variance-reduced variants along with explicit connections between Muon and adaptive methods [38] such as ASGO/One-sided Shampoo [8] and AdaGrad [74, 59].

The core idea of Muon has been widely adopted and adapted [Gruntkowska and Richtárik, 2025], e.g., in MuonBP for model-parallel training [48] and in Drop-Muon, which updates only a random subset of layers per step [37]. One of the most significant application areas is distributed and federated learning, where multiple workers or clients collaboratively optimize a global model [45] and several Muon-type methods improve convergence speed, communication efficiency, and scalability [78, 2, 36, 69]. Despite these advances, existing approaches largely overlook robustness to workers that behave maliciously or send adversarial updates [77], which can significantly degrade the performance of distributed or federated optimization [40].

Distributed and federated learning are commonly viewed as forms of collaborative training whose objective is to learn a single shared model [64, 25]. In these settings, however, the learning process is vulnerable to Byzantine workers [55, 75] — participants that deviate from the prescribed protocol due to malicious intent or system failures, transmitting corrupted information such as adversarial gradients to slow down or completely derail training [10]; they may collude and possess partial knowledge of other users’ private data. Because robustness is typically required in the worst-case sense, much of the literature places no restrictions on the adversary’s capabilities [32, 63]. Under these conditions, methods relying on simple aggregation, such as averaging stochastic gradients, are fundamentally fragile: even a single Byzantine worker can inject an arbitrarily large update and drive the iterates arbitrarily far from the desired solution [9, 81].

Consequently, such settings require secure, Byzantine-robust optimization methods [3, 4]. Designing methods with provable guarantees is nontrivial: stochastic gradients of honest workers are inherently diverse due to randomness and data heterogeneity [89, 5], which makes benign updates hard to distinguish from malicious ones, while adversaries can craft messages resembling legitimate gradients or inject small, temporally correlated perturbations [84]. As demonstrated by Baruch et al. [ [10], Karimireddy et al. [ [46], such attacks bypass many widely used defenses combining robust aggregation with distributed SGD [15, 87, 67] (Damasakos et al. 2019). Moreover, for broad classes of heterogeneous problems, achieving arbitrary accuracy is fundamentally impossible under Byzantine failures [46, 47].

The lower bounds of Karimireddy et al. [ [46] demonstrate limitations of memoryless, permutation-invariant aggregation-based optimization under heterogeneous Byzantine data. These results motivate exploiting temporal structure, such as worker momentum or error correction [42]; our guarantees are established under the explicit assumptions given below. rather than claiming that all memoryless algorithms fail on every problem.

Robust aggregation of worker momentum followed by a normalized server update predates our work: Yang et al. [ [86] analyze the Euclidean method with i.i.d. workers and study batch-size effects. Bolatov et al. [ [16] extend the Euclidean analysis to heterogeneous data under (L0,L1)(L_0, L_1)-smoothness and explicitly identify an arbitrary-norm LMO, including operator-norm Byzantine-robust Muon, as a possible extension. Our contribution is a non-Euclidean convergence analysis under relative first-moment heterogeneity, including non-convex and star-convex guarantees and extensions to extrapolation and inexact LMO steps. For fixed good-worker messages {zi}i∈G\{z_i\}_{i\in\mathcal{G}} in the same dual norm, let G=∣G∣≥2G = |\mathcal{G}| \ge2, zˉ=G−1∑i∈Gzi\bar{z} = G^{-1}\sum_{i\in\mathcal{G}} z_i, and define

DG:=1G∑i∈G∥zi−zˉ∥∗,rG:=1G(G−1)∑i,l∈Gi≠l∥zi−zl∥∗.D_{\mathcal{G}} := \frac{1}{G}\sum_{i\in\mathcal{G}}\lVert z_i-\bar{z}\rVert_*, \qquad r_{\mathcal{G}} := \frac{1}{G(G-1)}\sum_{\substack{i,l\in\mathcal{G}\\i\ne l}}\lVert z_i-z_l\rVert_*.
12rG≤DG≤G−1GrG.\frac{1}{2}r_{\mathcal{G}} \le D_{\mathcal{G}} \le\frac{G-1}{G}r_{\mathcal{G}}.

The last inequality follows from Jensen’s inequality and the triangle inequality and also holds after taking expectations. The pairwise dispersion in Definition 3.1 and the mean deviation used by Bolatov et al. [ [16] are thus equivalent up to constants when measured in the same norm; comparing different norms introduces the corresponding norm-equivalence factors. Table 1 summarizes the distinct proved guarantees.

WorkGeometry and smoothnessHeterogeneityAggregation premiseUpdate oracleConvergence criterion
Yang et al. [ [86]Euclidean LL-smoothi.i.d.Euclidean second momentexact normalizationaverage E∥∇f∥22\mathbb{E}\lVert\nabla f\rVert_2^2
Bolatov et al. [ [16]Euclidean (L0,L1)(L_0,L_1)-smoothadditive second momentEuclidean first momentexact normalizationminimum E∥∇f∥22\mathbb{E}\lVert\nabla f\rVert_2^2
This workarbitrary norm LL-smoothrelative/additive first momentdual-norm first momentexact/inexact LMOaverage E∥∇f∥∗2\mathbb{E}\lVert\nabla f\rVert_*^2; star-convex gap

Table 1. Comparison of the main theoretical guarantees for robustly aggregated worker momentum. The arbitrary-norm LMO of Bolatov et al. [ [16] is discussed as an extension, rather than covered by their theorem.

Contribution

We make the following contributions.

  • We study Byzantine-Robust Muon, a distributed Muon-type method using robust aggregation in place of standard averaging while preserving the momentum-orthogonalization mechanism of Muon. The method maintains worker-level momentum buffers, following the worker-momentum design of Karimireddy et al. [ [46], Farhadkhani et al. [ [28], and therefore uses temporal information that is absent from memoryless permutation-invariant schemes.

  • We formulate the method in a non-Euclidean trust-region framework, which separates the robust aggregation step from the geometry of the update, supports arbitrary norms, and specializes to Muon under the operator/nuclear-norm geometry; other norm choices recover normalized SGD- and sign-type momentum methods.

  • We prove convergence guarantees under Byzantine workers and heterogeneous data: a bound on the average gradient dual norm for general non-convex objectives, and a linear contraction to a stochastic and Byzantine noise neighborhood for star-convex objectives with weight decay, under a non-Euclidean heterogeneity condition aligned with the trust-region geometry.

  • We extend both guarantees under a separate second-moment aggregation premise, with concrete bucketed rules subject to their stated conditions, improve the non-convex rate under second-order smoothness via gradient extrapolation (O(ε−3.5)\mathcal{O}(\varepsilon^{-3.5}) at δ=0\delta= 0; the star-convex counterpart is discussed in Remark E.6), and extend all results to an inexact Muon update satisfying an additive oracle-error condition with parameter δˉ\bar{\delta} [73]: progress deflates by (1−δˉ)(1-\bar{\delta}) while the non-convex Byzantine terms in the numerator remain unchanged (Theorem 4.4).

  • We evaluate the same robust aggregation protocol with ResNet-18 and ViT-Tiny on CIFAR-10, a 38M-parameter byte-level language model on TinyStories, and additional image benchmarks. Five-seed comparisons on CIFAR-10 (five attacks) and TinyStories (four attacks), and three-seed comparisons on the transfer benchmarks (three attacks), cover clean and attacked training; on CIFAR-10 ResNet-18 we additionally ablate the Byzantine fraction, heterogeneity, worker momentum weight, aggregator, bucketing, and nearest-neighbor mixing (Appendix § 8.1).

Preliminaries

This section presents the formal problem setup, along with the main definitions and assumptions used throughout the analysis. We consider the following distributed finite-sum optimization problem:

min⁡x∈X[f(x):=1G∑i∈G{fi(x):=Eξi[Fi(x;ξi)]}],(1)\min_{x \in\mathcal{X}} \left[f(x) := \frac{1}{G}\sum_{i \in\mathcal{G}}\{f_i(x) := \mathbb{E}_{\xi_i}[F_i(x;\xi_i)]\}\right], \tag*{(1)}

where X\mathcal{X} is a finite-dimensional vector space endowed with the inner product ⟨⋅,⋅⟩:X×X→R\langle\cdot,\cdot\rangle: \mathcal{X} \times\mathcal{X} \to\mathbb{R}, ff is a differentiable loss function bounded from below, with finf⁡:=inf⁡x∈Xf(x)>−∞f_{\inf} := \inf_{x \in\mathcal{X}} f(x) > -\infty, and G\mathcal{G} denotes the set of regular clients, with cardinality G:=∣G∣G := |\mathcal{G}|. We assume that the set of all participating clients is [n]:={1,2,…,n}[n] := \{1,2,\ldots,n\}, with G⊆[n]\mathcal{G} \subseteq[n]. The complementary set B:=[n]∖G\mathcal{B} := [n] \setminus\mathcal{G} consists of Byzantine clients. Let ∣B∣=δrealn|\mathcal{B}| = \delta_{\mathrm{real}}n, where δreal\delta_{\mathrm{real}} denotes the true fraction of Byzantine clients and δ\delta is a known upper bound such that ∣B∣≤δn|\mathcal{B}| \leq\delta n; we assume throughout that at least two clients are regular, G≥2G \geq2. (We reserve the symbol BB for the relative heterogeneity constant of Assumption 3.6 below.) We further assume that 0≤δreal≤δ<120 \leq\delta_{\mathrm{real}} \leq\delta< \frac{1}{2}, since if Byzantine clients form a majority, the problem becomes unsolvable in general. In the context of distributed learning, fi:X→Rf_i:\mathcal{X} \to\mathbb{R} represents the loss function associated with the data of a regular client ii.

Robust aggregation. One of the central components of our approach is the concept of a (δ,c)(\delta,c)-robust aggregator. Because our analysis measures all momentum errors by first moments of the dual norm ∥⋅∥∗\|\cdot\|_*, we state the aggregation property directly in that geometry. It is a first-moment variant of the definitions proposed by Gorbunov et al. [ [32], which extend and generalize earlier notions introduced by Karimireddy et al. [ [46, 47].

Definition 3.1 ((δ,c\delta,c)-Robust Aggregator). Let δ<1/2\delta< 1/2 and c≥0c \geq0. Assume that {z1,…,zn}⊂X\{z_1,\ldots,z_n\} \subset\mathcal{X} are random vectors such that there exists a subset G⊆[n]\mathcal{G} \subseteq[n] of size G:=∣G∣≥(1−δ)nG := |\mathcal{G}| \geq(1-\delta)n whose average pairwise dispersion is bounded as

1G(G−1)∑i,ℓ∈Gi≠ℓE∥zi−zℓ∥∗≤r(2)\frac{1}{G(G-1)}\sum_{\substack{i,\ell\in\mathcal{G}\\i\ne\ell}}\mathbb{E}\|z_i-z_\ell\|_* \leq r \tag*{(2)}

for some r≥0r \geq0. We say that z^\hat{z} is a (δ,c)(\delta,c)-Robust Aggregator and write z^=ARAg⁡(z1,…,zn)\hat{z} = \operatorname{ARAg}(z_1,\ldots,z_n) if

E∥z^−zˉ∥∗≤cδr,where zˉ=1G∑i∈Gzi,(3)\mathbb{E}\|\hat{z}-\bar{z}\|_* \leq c\delta r,\qquad\text{where }\bar{z}=\frac{1}{G}\sum_{i \in\mathcal{G}}z_i, \tag*{(3)}

and z^\hat{z} is computed without knowledge of the dispersion level rr (the aggregator is agnostic).

The definition asks the aggregator to approximate the ideal good-worker average (unattainable exactly, since Byzantine identities are unknown) with an error proportional to the dispersion of the honest inputs; the input δ\delta must satisfy δreal≤δ<1/2\delta_{\mathrm{real}} \le\delta< 1/2. The product cδc\delta is best read as a single aggregation-quality factor: in this first-moment form it cannot in general be made small (worst case cδ≥1/2c\delta\ge1/2; Remark B.1), unlike the second-moment setting of Karimireddy et al. [ [46], where error of order δ\delta times the squared dispersion is achievable and optimal. Under the corruption, bucket-size, and dimension conditions of their respective results, bucketed Krum, geometric median, and coordinate-wise median provide second-moment Euclidean aggregation guarantees [47, 32]. Centered clipping requires a dispersion-dependent radius in the cited analysis and is not an agnostic realization of this premise. The second-moment property is distinct from Definition 3.1: Jensen and Assumption 3.4 convert its output error to the dual norm, while Appendix § 8.4 derives separate guarantees under second-moment heterogeneity.

Non-Euclidean Trust Region Framework. We use the first-order non-Euclidean trust-region viewpoint of Kovalev [ [52]. Given a norm ∥⋅∥\lVert\cdot\rVert with dual norm ∥⋅∥∗\lVert\cdot\rVert_{*} and a radius η>0\eta> 0, the update generated by a model vector vkv_k is

xk+1∈arg⁡min⁡x∈X{⟨vk,x⟩:∥x−xk∥≤η}.x^{k+1} \in\mathop{\arg\min}_{x \in\mathcal{X}} \left\{ \langle v_k, x \rangle: \lVert x - x^k \rVert\le\eta\right\}.

This is a first-order trust-region step: rather than minimizing a quadratic majorizer, the method minimizes a linear model over a norm ball. The geometry of the norm determines the steepest-descent direction, while the scalar radius η\eta controls the step length. In the Euclidean case, choosing vk=∇f(xk)v_k = \nabla f(x^k) recovers normalized gradient descent [72, 34].

The same template becomes Muon when applied to matrix-valued parameters with the operator norm as the primal norm and the nuclear norm as the dual norm: the steepest-descent direction is the polar/orthogonal factor solving the associated linear minimization oracle, which Muon applies to a momentum buffer and approximates by Newton–Schulz iterations [44, 65, 52]. The trust-region formulation thus covers normalized SGD, sign-based methods, and Muon-style momentum orthogonalization in a single framework, which is particularly useful in Byzantine settings: robust aggregation controls the submitted momentum vectors, whereas the trust-region subproblem controls the geometry and magnitude of the final server update.

Assumptions. Throughout the paper we employ several standard assumptions. In what follows, Fk\mathcal{F}^k denotes the σ\sigma-algebra generated by the first k+1k+1 stochastic gradient draws {ξit}i∈G,s≤k\{\xi_i^t\}_{i \in\mathcal{G},s \le k} of the algorithms of § 4, together with the Byzantine messages and aggregator randomness through mkm^k and the selection randomness of the trust-region solver through xk+1x^{k+1}; the round-(k+1)(k+1) gradient draws are not Fk\mathcal{F}^k-measurable. We set F−1\mathcal{F}^{-1} trivial; in particular, xk+1x^{k+1} is Fk\mathcal{F}^k-measurable.

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. (Cited on page 1)arxiv.org/abs/2303.08774
  2. [2]Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025. (Cited on page 2)arxiv.org/abs/2504.05295
  3. [3]Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. Advances in Neural Information Processing Systems, 31, 2018. (Cited on page 2)
  4. [4]Zeyuan Allen-Zhu, Faaze Ebrahimian, Jerry Li, and Dan Alistarh. Byzantine-resilient non-convex stochastic gradient descent. In International Conference on Learning Representations, 2021. (Cited on page 2)
  5. [5]Youssef Allouah, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. Fixing by mixing: A recipe for optimal Byzantine ML under heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 1232–1300. PMLR, 2023a. (Cited on pages 2 and 26)arxiv.org/abs/2302.01772
  6. [6]Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. On the privacy-robustness-utility trilemma in distributed learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023b. (Cited on page 6)arxiv.org/abs/2302.04787
  7. [7]Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. arXiv preprint arXiv:2505.16932, 2025. (Cited on pages 1, 9, 47, 48, and 51)arxiv.org/abs/2505.16932
  8. [8]Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762, 2025. (Cited on page 2)arxiv.org/abs/2503.20762
  9. [9]Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938–2948. PMLR, 2020. (Cited on page 2)
  10. [10]Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019. (Cited on pages 2, 10, and 22)
  11. [11]Jeremy Bernstein. Deriving Muon. https://jeremybernste.in/writing/deriving-muon, 2025. (Cited on page 1)
  12. [12]Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024. (Cited on pages 1 and 10)arxiv.org/abs/2409.20325
  13. [13]Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizadenesheli, and Animashree Anandkumar. signSGD: Compressed optimization for non-convex problems. In International Conference on Machine Learning, pages 560–569. PMLR, 2018. (Cited on page 2)arxiv.org/abs/1802.04434
  14. [14]Åke Björck and Clazett Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix. SIAM Journal on Numerical Analysis, 8(2):358–364, 1971. (Cited on page 1)DOI
  15. [15]Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017. (Cited on pages 2 and 25)
  16. [16]Arman Bolatov, Samuel Horváth, Martin Takáč, and Eduard Gorbunov. Byzantine-robust optimization under (l_0,l_1)-smoothness. arXiv preprint arXiv:2603.12512, 2026. (Cited on page 3)arxiv.org/abs/2603.12512
  17. [17]David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted Boltzmann machines. In Artificial Intelligence and Statistics, pages 111–119. PMLR, 2015a. (Cited on page 1)
  18. [18]David Carlson, Ya-Ping Hsieh, Edo Collins, Lawrence Carin, and Volkan Cevher. Stochastic spectral descent for discrete graphical models. IEEE Journal of Selected Topics in Signal Processing, 10(2):296–311, 2016. (Cited on page 1)DOI
  19. [19]David E. Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Preconditioned spectral descent for deep learning. Advances in Neural Information Processing Systems, 28, 2015b. (Cited on page 1)
  20. [20]Michael Crawshaw, Chirag Modi, Mingrui Liu, and Robert M Gower. An exploration of non-Euclidean gradient descent: Muon and its many variants. arXiv preprint arXiv:2510.09827, 2025. (Cited on page 1)arxiv.org/abs/2510.09827
  21. [21]Ashok Cutkosky and Harsh Mehta. Momentum improves normalized SGD. In International Conference on Machine Learning, pages 2260–2268. PMLR, 2020. (Cited on pages 2, 5, 6, 8, 9, 44, and 47)arxiv.org/abs/2002.03305
  22. [23]Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richtárik. A guide through the zoo of biased SGD. Advances in Neural Information Processing Systems, 36:23158–23171, 2023. (Cited on page 5)arxiv.org/abs/2305.16296
  23. [24]Yury Demidovich, Abhishek Chakraborty, Grigory Malinovsky, Angelia Nedić, and Peter Richtárik. Distance-aware muon: Adaptive step scaling for normalized optimization. arXiv preprint arXiv:2605.18999, 2026. (Cited on page 2)arxiv.org/abs/2605.18999
  24. [25]Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Anton Sinitsin, Dmitry Popov, Dmitry V. Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, et al. Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems, 34:7879–7897, 2021. (Cited on page 2)arxiv.org/abs/2106.10207
  25. [26]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. (Cited on pages 10 and 21)
  26. [27]Ronen Eldan and Yuanzhi Li. TinyStories: How small can language models be and still speak coherent English? arXiv preprint arXiv:2305.07759, 2023. (Cited on pages 13 and 22)arxiv.org/abs/2305.07759
  27. [28]Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. Byzantine machine learning made easy by resilient averaging of momentums. In International Conference on Machine Learning, pages 6246–6283. PMLR, 2022. (Cited on pages 3 and 10)arxiv.org/abs/2205.12173
  28. [29]Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. doi: 10.1137/120880811. (Cited on pages 5 and 6)DOI
  29. [30]Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, 156(1–2):59–99, 2016. doi: 10.1007/s10107-015-0871-8. (Cited on page 5)DOI
  30. [31]Eduard Gorbunov, Dmitry Kovalev, Dmitry Makarenko, and Peter Richtárik. Linearly converging error compensated SGD. Advances in Neural Information Processing Systems, 33, 2020. (Cited on page 6)arxiv.org/abs/2010.12292
  31. [32]Eduard Gorbunov, Samuel Horváth, Peter Richtárik, and Gauthier Gidel. Variance reduction is an antidote to Byzantines: Better rates, weaker assumptions and communication compression as a cherry on the top. In International Conference on Learning Representations, 2023. (Cited on pages 2, 4, 5, 6, 29, and 41)arxiv.org/abs/2206.00529
  32. [33]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. (Cited on page 1)arxiv.org/abs/2407.21783
  33. [34]Benjamin Grimmer. Convergence rates for deterministic and stochastic subgradient methods without Lipschitz continuity. SIAM Journal on Optimization, 29(2):1350–1365, 2019. (Cited on page 5)arxiv.org/abs/1712.04104
  34. [36]Kaja Gruntkowska, Alexander Gaponov, Zhirayr Tovmasyan, and Peter Richtárik. Error feedback for Muon and friends. arXiv preprint arXiv:2510.00643, 2025a. (Cited on page 2)arxiv.org/abs/2510.00643
  35. [37]Kaja Gruntkowska, Yassine Maziane, Zheng Qu, and Peter Richtárik. Drop-Muon: Update less, converge faster. arXiv preprint arXiv:2510.02239, 2025b. (Cited on page 2)arxiv.org/abs/2510.02239
  36. [38]Vineet Gupta, Tomer Koren, and Yoram Singer. A unified approach to adaptive regularization in online and stochastic optimization. arXiv preprint arXiv:1706.06569, 2017. (Cited on page 2)arxiv.org/abs/1706.06569
  37. [39]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. (Cited on pages 10 and 21)arxiv.org/abs/1512.03385
  38. [40]Lie He, Sai Praneeth Karimireddy, and Martin Jaggi. Secure Byzantine-robust machine learning. arXiv preprint arXiv:2006.04747, 2020. (Cited on page 2)arxiv.org/abs/2006.04747
  39. [41]Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019. (Cited on pages 10 and 21)arxiv.org/abs/1909.06335
  40. [42]Rustem Islamov, Grigory Malinovsky, Alexander Gaponov, Aurelien Lucchi, Peter Richtárik, and Eduard Gorbunov. Byzantine-robust and differentially private federated optimization under weaker assumptions. arXiv preprint arXiv:2603.23472, 2026. (Cited on page 2)arxiv.org/abs/2603.23472
  41. [43]Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In International Conference on Machine Learning, pages 427–435. PMLR, 2013. (Cited on page 2)
  42. [44]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. (Cited on pages 1, 5, 9, and 47)
  43. [45]Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021. (Cited on page 2)arxiv.org/abs/1912.04977
  44. [46]Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for Byzantine robust optimization. In International Conference on Machine Learning, pages 5311–5319. PMLR, 2021. (Cited on pages 2, 3, 4, 5, 10, 25, 28, and 29)arxiv.org/abs/2012.10333
  45. [47]Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=jXKKDEi5vJt. (Cited on pages 2, 4, 5, 8, 10, 22, 26, 28, 29, 36, 41, and 43)
  46. [48]Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. arXiv preprint arXiv:2510.16981, 2025. (Cited on page 2)arxiv.org/abs/2510.16981
  47. [49]Sarit Khirirat, Abdurakhmon Sadiev, Yury Demidovich, and Peter Richtárik. Better LMO-based momentum methods with second-order information. arXiv preprint arXiv:2512.13227, 2025. (Cited on page 2)arxiv.org/abs/2512.13227
  48. [50]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. (Cited on page 1)arxiv.org/abs/1412.6980
  49. [52]Dmitry Kovalev and Ekaterina Borodich. Non-Euclidean SGD for structured optimization: Unified analysis and improved rates. arXiv preprint arXiv:2511.11466, 2025. (Cited on page 2)arxiv.org/abs/2511.11466
  50. [53]Zdislav Kovarik. Some iterative methods for improving orthonormality. SIAM Journal on Numerical Analysis, 7(3):386–389, 1970. (Cited on page 1)DOI
  51. [54]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. (Cited on page 21)
  52. [55]Leslie Lamport, Robert Shostak, and Marshall Pease. The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3):382–401, 1982. (Cited on page 2)
  53. [56]Quoc V Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y Ng. On optimization methods for deep learning. In Proceedings of the 28th International Conference on Machine Learning, pages 265–272, 2011. (Cited on page 1)
  54. [57]Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. (Cited on page 21)
  55. [58]Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. (Cited on page 1)DOI
  56. [59]Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 983–992. PMLR, 2019. (Cited on page 2)arxiv.org/abs/1805.08114
  57. [60]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. (Cited on page 1)arxiv.org/abs/2412.19437
  58. [61]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025. (Cited on page 1)arxiv.org/abs/2502.16982
  59. [62]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. (Cited on page 1)
  60. [63]Grigory Malinovsky, Peter Richtárik, Samuel Horváth, and Eduard Gorbunov. Byzantine robustness and partial participation can be achieved at once: Just clip gradient differences. Advances in Neural Information Processing Systems, 37:34900–34979, 2024. (Cited on page 2)
  61. [64]Priyanka Mary Mammen. Federated learning: Opportunities and challenges. arXiv preprint arXiv:2101.05428, 2021. (Cited on page 2)arxiv.org/abs/2101.05428
  62. [65]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning. PMLR, 2025a. (Cited on pages 1, 2, and 5)arxiv.org/abs/2502.07529
  63. [66]Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Tony Silveti-Falls, and Volkan Cevher. Generalized gradient norm clipping & non-Euclidean (L₀, L₁)-smoothness. arXiv preprint arXiv:2506.01913, 2025b. (Cited on page 2)
  64. [67]Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142–1154, 2022. (Cited on pages 2 and 10)DOI
  65. [68]Xun Qian, Hussein Rammal, Dmitry Kovalev, and Peter Richtárik. Muon is provably faster with momentum variance reduction. arXiv preprint arXiv:2512.16598, 2025. (Cited on page 2)arxiv.org/abs/2512.16598
  66. [69]Xun Qian, Alexander Gaponov, Grigory Malinovsky, and Peter Richtárik. Communication-efficient gluo in federated learning. arXiv preprint arXiv:2604.10689, 2026. (Cited on page 2)arxiv.org/abs/2604.10689
  67. [70]Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making Muon & Scion great again! (Bridging theory and practice of LMO-based optimizers for LLMs). arXiv preprint arXiv:2505.13416, 2025. (Cited on page 2)arxiv.org/abs/2505.13416
  68. [71]Abdurakhmon Sadiev, Yury Demidovich, Igor Sokolov, Grigory Malinovsky, Sarit Khirirat, and Peter Richtárik. Improved convergence in parameter-agnostic error feedback through momentum. arXiv preprint arXiv:2511.14501, 2025. (Cited on page 9)arxiv.org/abs/2511.14501
  69. [72]Naum Zuselevich Shor. Minimization Methods for Non-Differentiable Functions, volume 3 of Springer Series in Computational Mathematics. Springer, Berlin, Heidelberg, 1985. (Cited on page 5)DOI
  70. [73]Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact Muon update. arXiv preprint arXiv:2510.19933, 2025. (Cited on pages 1, 4, 9, 47, 48, and 51)arxiv.org/abs/2510.19933
  71. [74]Matthew Streeter and H Brendan McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862, 2010. (Cited on page 2)arxiv.org/abs/1002.4862
  72. [75]Lili Su and Nitin H Vaidya. Fault-tolerant multi-agent optimization: Optimal iterative distributed algorithms. In Proceedings of the 2016 ACM Symposium on Principles of Distributed Computing, pages 425–434, 2016. (Cited on page 2)
  73. [76]Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signSGD under weaker assumptions. In International Conference on Machine Learning, pages 33077–33099. PMLR, 2023. (Cited on pages 2 and 5)
  74. [77]Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.07963, 2019. (Cited on page 2)arxiv.org/abs/1911.07963
  75. [78]Benjamin Thérien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky. MuLoCo: Muon is a practical inner optimizer for DiLoCo. arXiv preprint arXiv:2505.23725, 2025. (Cited on page 2)arxiv.org/abs/2505.23725
  76. [79]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021. (Cited on pages 10 and 21)
  77. [80]Mark Tuddenham, Adam Prügel-Bennett, and Jonathan Hare. Orthogonalising gradients to speed up neural network optimisation. arXiv preprint arXiv:2202.07052, 2022. (Cited on page 2)arxiv.org/abs/2202.07052
  78. [81]Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jyoung Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 33:16070–16084, 2020. (Cited on page 2)arxiv.org/abs/2007.05084
  79. [82]Zhaoxian Wu, Qing Ling, Tianyi Chen, and Georgios B Giannakis. Federated variance-reduced stochastic gradient descent with robustness to Byzantine attacks. IEEE Transactions on Signal Processing, 68:4583–4596, 2020. (Cited on page 6)DOI
  80. [83]Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. (Cited on page 21)arxiv.org/abs/1708.07747
  81. [84]Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking Byzantine-tolerant SGD by inner product manipulation. In Uncertainty in Artificial Intelligence, pages 261–270. PMLR, 2020. (Cited on pages 2, 10, and 22)arxiv.org/abs/1903.03936
  82. [85]Shuo Xie, Tianhao Wang, Sashank Reddi, Sanjiv Kumar, and Zhiyuan Li. Structured preconditioners in adaptive optimization: A unified analysis. arXiv preprint arXiv:2503.10537, 2025. (Cited on page 2)arxiv.org/abs/2503.10537
  83. [86]Yi-Rui Yang, Chang-Wei Shi, and Wu-Jun Li. On the effect of batch size in byzantine-robust distributed learning. In International Conference on Learning Representations, volume 2024, pages 27380–27414, 2024. (Cited on page 3)arxiv.org/abs/2305.13856
  84. [87]Dong Yin, Yudong Chen, Kannan Ramchandran, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pages 5650–5659. PMLR, 2018. (Cited on pages 2 and 25)arxiv.org/abs/1803.01498
  85. [88]Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023. (Cited on page 1)DOI
  86. [89]Heng Zhu and Qing Ling. BROADCAST: Reducing both stochastic and compression noise to robustify communication-efficient federated learning. arXiv preprint arXiv:2104.06685, 2021. (Cited on page 2)DOI

Paper details

vlnc:W_34k53pbn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents