Code: https://github.com/RidaAssalouh/BMuon
Introduction
Modern Transformer models incur substantial memory and computational costs at inference time, motivating compression for more efficient deployment [17 ] . Pruning is a natural approach to this problem, but practical gains depend on the sparsity structure: unstructured sparsity often requires specialized kernels or hardware, whereas structured pruning removes entire computational units and translates directly into efficiency gains [31 , 2 ] . Feed-forward modules can account for roughly two thirds of Transformer parameters [10 ] , making their hidden channels a natural target for structured compression. Global pruning ranks channels across layers jointly, allowing different layers to retain different numbers of channels [19 , 2 ] . Post-pruning fine-tuning adds computational cost, especially for large models [17 , 27 ] , motivating compressibility to emerge during training so that pruning requires little or no recovery.
Muon has demonstrated strong training efficiency across model scales, from small vision and language-model benchmarks to billion-parameter language models [16 , 20 , 26 ] . Its matrix update can be viewed as a linear minimization oracle over a spectral-norm ball, giving Muon a steepest-descent interpretation in spectral-norm geometry [24 ] . Prior work has shown that implicit bias can depend on optimization geometry [13 ] . For Muon, recent theory identifies a spectral-norm margin bias in linear and smooth homogeneous models [9 , 12 ] . Yet we find that Muon-trained Transformers degrade rapidly under structured MLP pruning, suggesting that Muon’s spectral geometry does not by itself encourage structured compressibility. As shown in Figure 1 , a Muon-trained ViT-S/16 on ImageNet-1K collapses after only 30% global MLP channel pruning, with Top-1 accuracy falling from 75.23% to 3.17%. In contrast, our proposed Budget Muon (BMuon) algorithm retains 74.08% Top-1 accuracy after 70% pruning, nearly matching its 74.17% accuracy before pruning, with no post-pruning recovery. This raises the central question: can Muon’s spectral geometry be modified to encourage structured compressibility?
Figure 1. Post-training global MLP pruning on ImageNet-1K with ViT-S/16. Mean ± \pm ± std over 3 seeds. See Appendix E.§ 1 .
To address this question, we introduce BMuon, which reweights Muon’s spectral geometry under a single global budget K K K shared across all hidden channels of all MLP blocks. For hidden channel j j j in block ℓ \ell ℓ , BMuon assigns an allocation μ j ℓ ∈ [ 0 , 1 ] \mu^\ell_j \in[0, 1] μ j ℓ ∈ [ 0 , 1 ] to the corresponding row of the up-projection matrix and column of the down-projection matrix, with ∑ ℓ , j μ j ℓ = K \sum_{\ell,j}\mu^\ell_j = K ∑ ℓ , j μ j ℓ = K . For a fixed allocation, BMuon introduces a diagonal reweighting inside Muon’s spectral-norm constraint, with the components of the update variable corresponding to channel j j j in block ℓ \ell ℓ weighted according to its allocation μ j ℓ \mu^\ell_j μ j ℓ . BMuon chooses the allocation globally, intuitively asking where the update budget should be spent to obtain the greatest first-order decrease in the loss.
NuMuon [8 ] is the closest prior compression-aware variant of Muon. It imposes a nuclear-norm constraint on Muon’s update direction to improve low-rank compressibility. BMuon instead targets structured compressibility over hidden channels by introducing a global channel budget into Muon’s spectral-norm geometry. Prior work has also incorporated compressibility into training. Lu et al. [21 ] train networks for unstructured pruning with Frank–Wolfe under fixed layerwise constraints on the weights, while Zimmer et al. [37 ] use Frank–Wolfe under fixed norm constraints for convolutional filter pruning and low-rank compression. BMuon instead leaves the weights unconstrained and modifies Muon’s spectral update geometry for global hidden-channel compression. Resource-constrained structure learning and global structured pruning have been studied across CNNs and Transformers [11 , 33 , 32 , 2 , 29 ] . GRASPrune [30 ] learns gates post hoc on a frozen LLM to select MLP channels and attention KV-head groups under a single global parameter budget, while DABP [18 ] jointly trains CNN weights and continuous channel gates under a global resource budget. BMuon uses no forward gates; its budget instead controls the channel-wise reweighting of Muon’s spectral update constraint during training.
Theoretically, we show that the BMuon oracle is exactly normalized steepest descent under an induced norm Ω α , K \Omega_{\alpha,K} Ω α , K , where α \alpha α controls the strength of channel reweighting and K K K is the global budget over the N N N hidden channels. This family recovers Muon when α = 0 \alpha= 0 α = 0 or K = N K = N K = N . At α = 1 \alpha= 1 α = 1 , its unit ball is the convex hull of operator-norm-bounded updates supported on at most K K K coupled hidden channels. This is related to group k k k -support norms through the use of structured support [3 , 25 ] , but its atoms are bounded in operator norm rather than Euclidean norm. It is also related to spectral k k k -support norms [22 , 23 ] , but differs in the notion of sparsity: spectral k k k -support uses sparsity in singular values, whereas BMuon uses MLP-channel sparsity. At α = K = 1 \alpha= K = 1 α = K = 1 , the norm reduces to an ℓ 1 \ell_1 ℓ 1 aggregation over coupled channel groups, yielding a group-lasso-like geometry [34 ] . We then characterize the representation cost induced by BMuon. Following Dai et al. [7 ] , this is the minimum parameter-space cost among exact two-matrix factorizations that realize the same linear predictor. We prove that, under an explicit budget-scarcity condition, every minimum-cost BMuon factorization is width-minimal, whereas Muon’s minimum cost can also be attained by factorizations with redundant hidden width.
Solving the exact allocation problem at every training step is expensive. We therefore derive a minorization–maximization surrogate that yields a tractable allocation update, followed by the corresponding weighted Muon update. On CIFAR-100 and ImageNet-1K, BMuon outperforms Muon under both gradual and post-training global MLP-channel pruning, even when both use the same magnitude-based pruning criterion. Under gradual pruning, using the learned allocations themselves to rank channels further improves performance, showing that they also provide a useful measure of channel importance. The non-uniform architectures identified by BMuon in the gradual-pruning experiments also outperform matched uniform-width architectures when retrained from scratch, providing evidence that the learned allocation captures useful differences in how MLP capacity should be distributed across layers. Finally, on a 1B-parameter LLaMA-style model trained on FineWeb, BMuon improves robustness to post-training global MLP-channel pruning, showing that the effect extends to language modeling at substantially larger scale.
Contributions. To summarize, our main contributions are:
We characterize the induced norm family Ω α , K \Omega_{\alpha,K} Ω α , K and its relation to Muon and structured sparse geometries in § 2 . In the linear factorization setting of § 3 , we show that for a sufficiently scarce budget, every minimum-cost BMuon factorization is width-minimal, whereas Muon’s minimum cost is also attained by redundant-width factorizations.
In § 5 , we show across CIFAR-100, ImageNet-1K, and a 1B-parameter LLaMA-style model that BMuon improves structured MLP-channel compressibility relative to Muon. In the vision experiments, the learned allocations also provide effective pruning scores and identify non-uniform MLP architectures that outperform matched uniform-width architectures when retrained from scratch.
Budget Muon: Oracle and Geometry
Setup and Muon Preliminaries
MLP channels and global pruning. We consider L L L single-hidden-layer MLP blocks, as in Transformer feed-forward modules, with block ℓ \ell ℓ given by
x ↦ W 2 ℓ ϕ ( W 1 ℓ x + b ℓ ) , W 1 ℓ ∈ R m ℓ × d ℓ , W 2 ℓ ∈ R p ℓ × m ℓ , (1) x \mapsto W_2^\ell\phi\left(W_1^\ell x + b^\ell\right), \qquad W_1^\ell\in\mathbb{R}^{m_\ell\times d_\ell}, \qquad W_2^\ell\in\mathbb{R}^{p_\ell\times m_\ell}, \tag*{(1)} x ↦ W 2 ℓ ϕ ( W 1 ℓ x + b ℓ ) , W 1 ℓ ∈ R m ℓ × d ℓ , W 2 ℓ ∈ R p ℓ × m ℓ , (1)
where m ℓ m_\ell m ℓ is the hidden width. Hidden channel j j j is associated with its incoming weights W 1 , : , j ℓ W_{1,:,j}^\ell W 1 , : , j ℓ , bias b j ℓ b_j^\ell b j ℓ , and outgoing weights W 2 , : , j ℓ W_{2,:,j}^\ell W 2 , : , j ℓ . Pruning the channel removes these components jointly. The matrix parameters W 1 ℓ , W 2 ℓ W_1^\ell, W_2^\ell W 1 ℓ , W 2 ℓ are optimized by Muon, while biases are optimized separately by AdamW. We write W : = ( ( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L W := ((W_1^\ell, W_2^\ell))_{\ell=1}^L W := (( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L for the collection of MLP weight matrices and N : = ∑ ℓ = 1 L m ℓ N := \sum_{\ell=1}^L m_\ell N := ∑ ℓ = 1 L m ℓ for the total number of hidden channels across all blocks. Given channel scores s j ℓ s_j^\ell s j ℓ , global pruning ranks all N N N channels jointly and removes the lowest-scoring ones until the target global sparsity is reached. The resulting hidden width in each block is therefore determined by this global ranking.
Muon as spectral-norm steepest descent. For an MLP matrix W i ℓ W_i^\ell W i ℓ , let M i ℓ M_i^\ell M i ℓ denote the momentum matrix used by Muon, following Appendix A. For any matrix A A A with compact singular value decomposition (SVD) A = U Σ V ⊤ A = U\Sigma V^\top A = U Σ V ⊤ , define its polar factor by Polar ( A ) : = U V ⊤ \operatorname{Polar}(A) := UV^\top Polar ( A ) := U V ⊤ . The exact Muon direction is Δ i , M u o n ℓ = Polar ( M i ℓ ) \Delta_{i,\mathrm{Muon}}^\ell= \operatorname{Polar}(M_i^\ell) Δ i , Muon ℓ = Polar ( M i ℓ ) , and W i ℓ W_i^\ell W i ℓ is updated in the opposite direction. In practice, Muon approximates this direction using a finite Newton–Schulz iteration [16 ] ; throughout the theoretical analysis, we use the exact polar factor. By spectral–nuclear norm duality, the exact Muon direction satisfies
Δ i , M u o n ℓ = Polar ( M i ℓ ) ∈ arg max ∥ Δ i ℓ ∥ o p ≤ 1 ⟨ M i ℓ , Δ i ℓ ⟩ , max ∥ Δ i ℓ ∥ o p ≤ 1 ⟨ M i ℓ , Δ i ℓ ⟩ = ∥ M i ℓ ∥ ∗ . \Delta_{i,\mathrm{Muon}}^\ell= \operatorname{Polar}(M_i^\ell) \in\underset{\lVert\Delta_i^\ell\rVert_{\mathrm{op}}\leq1}{\operatorname{arg\,max}}\left\langle M_i^\ell,\Delta_i^\ell\right\rangle,\qquad\underset{\lVert\Delta_i^\ell\rVert_{\mathrm{op}}\leq1}{\max}\left\langle M_i^\ell,\Delta_i^\ell\right\rangle= \lVert M_i^\ell\rVert_*. Δ i , Muon ℓ = Polar ( M i ℓ ) ∈ ∥ Δ i ℓ ∥ op ≤ 1 arg max ⟨ M i ℓ , Δ i ℓ ⟩ , ∥ Δ i ℓ ∥ op ≤ 1 max ⟨ M i ℓ , Δ i ℓ ⟩ = ∥ M i ℓ ∥ ∗ .
Here ⟨ A , B ⟩ = tr ( A T B ) \langle A, B\rangle= \operatorname{tr}(A^{\mathsf{T}}B) ⟨ A , B ⟩ = tr ( A T B ) is the Frobenius inner product, ∥ A ∥ o p = σ max ( A ) \lVert A\rVert_{\mathrm{op}} = \sigma_{\max}(A) ∥ A ∥ op = σ m a x ( A ) is the operator norm, and ∥ A ∥ ∗ = ∑ j σ j ( A ) \lVert A\rVert_{*} = \sum_j \sigma_j(A) ∥ A ∥ ∗ = ∑ j σ j ( A ) is the nuclear norm. Thus exact-polar Muon is normalized steepest descent under the spectral norm [4 , 24 ] . For all MLP matrices, write M : = ( ( M 1 ℓ , M 2 ℓ ) ) ℓ = 1 L M := ((M_1^\ell, M_2^\ell))_{\ell=1}^{L} M := (( M 1 ℓ , M 2 ℓ ) ) ℓ = 1 L and Δ : = ( ( Δ 1 ℓ , Δ 2 ℓ ) ) ℓ = 1 L \Delta:= ((\Delta_1^\ell, \Delta_2^\ell))_{\ell=1}^{L} Δ := (( Δ 1 ℓ , Δ 2 ℓ ) ) ℓ = 1 L . We equip this product space with the inner-product ⟨ M , Δ ⟩ : = ∑ ℓ = 1 L ( ⟨ M 1 ℓ , Δ 1 ℓ ⟩ + ⟨ M 2 ℓ , Δ 2 ℓ ⟩ ) \langle M, \Delta\rangle:= \sum_{\ell=1}^{L}(\langle M_1^\ell, \Delta_1^\ell\rangle+ \langle M_2^\ell, \Delta_2^\ell\rangle) ⟨ M , Δ ⟩ := ∑ ℓ = 1 L (⟨ M 1 ℓ , Δ 1 ℓ ⟩ + ⟨ M 2 ℓ , Δ 2 ℓ ⟩) and define Muon’s product-space unit ball A M u o n : = { Δ : ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } \mathcal{A}_{\mathrm{Muon}} := \{\Delta: \lVert\Delta_1^\ell\rVert_{\mathrm{op}} \le1,\ \lVert\Delta_2^\ell\rVert_{\mathrm{op}} \le1,\ \forall\ell\} A Muon := { Δ : ∥ Δ 1 ℓ ∥ op ≤ 1 , ∥ Δ 2 ℓ ∥ op ≤ 1 , ∀ ℓ } . Equivalently, A M u o n = { Δ : ∥ Δ ∥ M u o n ≤ 1 } \mathcal{A}_{\mathrm{Muon}} = \{\Delta: \lVert\Delta\rVert_{\mathrm{Muon}} \le1\} A Muon = { Δ : ∥ Δ ∥ Muon ≤ 1 } , where ∥ Δ ∥ M u o n : = max ℓ max { ∥ Δ 1 ℓ ∥ o p , ∥ Δ 2 ℓ ∥ o p } \lVert\Delta\rVert_{\mathrm{Muon}} := \max_\ell\max\{\lVert\Delta_1^\ell\rVert_{\mathrm{op}},\lVert\Delta_2^\ell\rVert_{\mathrm{op}}\} ∥ Δ ∥ Muon := max ℓ max {∥ Δ 1 ℓ ∥ op , ∥ Δ 2 ℓ ∥ op } . Muon’s update direction over all MLP matrices is then
Δ M u o n ∈ arg max Δ ∈ A M u o n ⟨ M , Δ ⟩ . (2) \Delta_{\mathrm{Muon}} \in\underset{\Delta\in\mathcal{A}_{\mathrm{Muon}}}{\arg\max}\ \langle M,\Delta\rangle. \tag*{(2)} Δ Muon ∈ Δ ∈ A Muon arg max ⟨ M , Δ ⟩ . (2)
Since the objective and constraints separate across matrices, this is equivalent to applying the single-matrix Muon direction independently to each W i ℓ W_i^\ell W i ℓ .
The BMuon Oracle
Global channel budget. For each block ℓ \ell ℓ of width m ℓ m_\ell m ℓ , let μ ℓ ∈ [ 0 , 1 ] m ℓ \mu^\ell\in[0,1]^{m_\ell} μ ℓ ∈ [ 0 , 1 ] m ℓ assign an allocation to each hidden channel, and let D μ ℓ : = diag ( μ ℓ ) D_{\mu^\ell} := \operatorname{diag}(\mu^\ell) D μ ℓ := diag ( μ ℓ ) . Writing μ : = ( μ 1 , … , μ L ) ∈ [ 0 , 1 ] N \mu:= (\mu^1,\ldots,\mu^L) \in[0,1]^N μ := ( μ 1 , … , μ L ) ∈ [ 0 , 1 ] N , for a budget K ∈ ( 0 , N ] K \in(0,N] K ∈ ( 0 , N ] , define K K : = { μ ∈ [ 0 , 1 ] N : ∑ ℓ , j μ j ℓ = K } \mathcal{K}_K := \{\mu\in[0,1]^N : \sum_{\ell,j}\mu_j^\ell= K\} K K := { μ ∈ [ 0 , 1 ] N : ∑ ℓ , j μ j ℓ = K } . All hidden channels share the budget K K K , so channels across blocks compete directly for allocation.
Channel-weighted Muon directions. Fix α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] . For a fixed allocation μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , BMuon reweights the spectral constraints in A M u o n \mathcal{A}_{\mathrm{Muon}} A Muon channel-wise, yielding
A α ( μ ) : = { Δ : ∃ { P 1 ℓ , P 2 ℓ } ℓ = 1 L , Δ 1 ℓ = D μ ℓ α P 1 ℓ , Δ 2 ℓ = P 2 ℓ D μ ℓ α , ∥ P 1 ℓ ∥ o p ≤ 1 , ∥ P 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } . \mathcal{A}_{\alpha}(\mu) := \left\{\Delta: \exists\{P_1^\ell,P_2^\ell\}_{\ell=1}^{L},\ \Delta_1^\ell= D_{\mu^\ell}^{\alpha}P_1^\ell,\ \Delta_2^\ell= P_2^\ell D_{\mu^\ell}^{\alpha},\ \lVert P_1^\ell\rVert_{\mathrm{op}}\le1,\ \lVert P_2^\ell\rVert_{\mathrm{op}}\le1,\ \forall\ell\right\}. A α ( μ ) := { Δ : ∃ { P 1 ℓ , P 2 ℓ } ℓ = 1 L , Δ 1 ℓ = D μ ℓ α P 1 ℓ , Δ 2 ℓ = P 2 ℓ D μ ℓ α , ∥ P 1 ℓ ∥ op ≤ 1 , ∥ P 2 ℓ ∥ op ≤ 1 , ∀ ℓ } .
The same factor ( μ j ℓ ) α (\mu_j^\ell)^\alpha ( μ j ℓ ) α scales row j j j of Δ 1 ℓ \Delta_1^\ell Δ 1 ℓ and column j j j of Δ 2 ℓ \Delta_2^\ell Δ 2 ℓ , coupling the two components of hidden channel j j j . When every μ j ℓ > 0 \mu_j^\ell> 0 μ j ℓ > 0 , this is equivalently ∥ D μ ℓ − α Δ 1 ℓ ∥ o p ≤ 1 \lVert D_{\mu^\ell}^{-\alpha}\Delta_1^\ell\rVert_{\mathrm{op}} \le1 ∥ D μ ℓ − α Δ 1 ℓ ∥ op ≤ 1 and ∥ Δ 2 ℓ D μ ℓ − α ∥ o p ≤ 1 \lVert\Delta_2^\ell D_{\mu^\ell}^{-\alpha}\rVert_{\mathrm{op}} \le1 ∥ Δ 2 ℓ D μ ℓ − α ∥ op ≤ 1 for all ℓ \ell ℓ . Thus BMuon applies Muon’s spectral-norm constraint after diagonally reweighting the hidden channels. If μ j ℓ = 0 \mu_j^\ell= 0 μ j ℓ = 0 , the corresponding row of Δ 1 ℓ \Delta_1^\ell Δ 1 ℓ and column of Δ 2 ℓ \Delta_2^\ell Δ 2 ℓ are forced to be zero, making the connection to channel-sparse updates explicit.
BMuon oracle. Muon maximizes the alignment ⟨ M , Δ ⟩ \langle M,\Delta\rangle ⟨ M , Δ ⟩ over a fixed constraint set, ∥ Δ ∥ M u o n ≤ 1 \lVert\Delta\rVert_{\mathrm{Muon}} \le1 ∥ Δ ∥ Muon ≤ 1 . In BMuon, the admissible set for a fixed allocation is A α ( μ ) \mathcal{A}_{\alpha}(\mu) A α ( μ ) , and the oracle chooses the allocation and direction jointly:
max μ ∈ K K max Δ ∈ A α ( μ ) ⟨ M , Δ ⟩ . (3) \max_{\mu\in\mathcal{K}_K}\ \max_{\Delta\in\mathcal{A}_{\alpha}(\mu)}\ \langle M,\Delta\rangle. \tag*{(3)} μ ∈ K K max Δ ∈ A α ( μ ) max ⟨ M , Δ ⟩ . (3)
The budget is allocated to maximize the attainable alignment with the momentum; when M M M is the gradient rather than the momentum, the oracle jointly chooses μ \mu μ and Δ \Delta Δ to maximize the first-order decrease in the loss. By spectral–nuclear norm duality, the oracle reduces to the allocation problem
max μ ∈ K K Φ M ( μ ) , Φ M ( μ ) : = ∑ ℓ = 1 L [ ∥ D μ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ℓ α ∥ ∗ ] . (4) \max_{\mu\in\mathcal{K}_K}\ \Phi_M(\mu),\qquad\Phi_M(\mu) := \sum_{\ell=1}^{L}\left[\lVert D_{\mu^\ell}^{\alpha}M_1^\ell\rVert_{*} + \lVert M_2^\ell D_{\mu^\ell}^{\alpha}\rVert_{*}\right]. \tag*{(4)} μ ∈ K K max Φ M ( μ ) , Φ M ( μ ) := ℓ = 1 ∑ L [ ∥ D μ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ℓ α ∥ ∗ ] . (4)
whose maximum is attained since K K \mathcal{K}_K K K is compact and Φ M \Phi_M Φ M is continuous. Given a maximizer μ ⋆ \mu^\star μ ⋆ , an optimal direction is, for every ℓ \ell ℓ ,
Δ 1 ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , Δ 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α . (5) \Delta_{1}^{\ell,\star}=D_{\mu_{\ell,\star}}^\alpha\operatorname{Polar}\left(D_{\mu_{\ell,\star}}^\alpha M_1^\ell\right), \qquad\Delta_{2}^{\ell,\star}=\operatorname{Polar}\left(M_2^\ell D_{\mu_{\ell,\star}}^\alpha\right)D_{\mu_{\ell,\star}}^\alpha. \tag*{(5)} Δ 1 ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , Δ 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α . (5)
The derivation is given in Appendix B.1. The parameters are then updated as W i ℓ ← W i ℓ − η Δ i ℓ , ⋆ W_i^\ell\leftarrow W_i^\ell-\eta\Delta_i^{\ell,\star} W i ℓ ← W i ℓ − η Δ i ℓ , ⋆ . We call this an oracle because it solves the allocation problem in Eq. (4) exactly at every step; the practical approximation is introduced in Section 4.
Allocation curvature. The exponent α \alpha α controls the curvature of the allocation objective in Eq. (4) .
Theorem 1 (Curvature transition). For any momentum matrices, Φ M \Phi_M Φ M is concave in μ \mu μ for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 , convex in μ \mu μ for α = 1 \alpha= 1 α = 1 , and in general neither convex nor concave for 1 2 < α < 1 \frac{1}{2} < \alpha< 1 2 1 < α < 1 .
For 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 , Eq. (4) maximizes a concave function over the convex set K K \mathcal{K}_K K K , and is therefore a convex program; every local maximizer is global, and optimal allocations may lie in the relative interior of K K \mathcal{K}_K K K , hence may be fractional. By contrast, at α = 1 \alpha= 1 α = 1 and integer K K K , a convex function is maximized over the polytope K K \mathcal{K}_K K K , so a maximizer can be chosen at an extreme point of K K \mathcal{K}_K K K , i.e., as a binary allocation selecting exactly K K K channels. The proof and details are given in Appendix B.3.
Induced Optimization Geometry
Joint maximization over μ \mu μ and Δ \Delta Δ in Eq. (3) is equivalent to maximizing ⟨ M , Δ ⟩ \langle M,\Delta\rangle ⟨ M , Δ ⟩ over Δ ∈ A α , K \Delta\in\mathcal{A}_{\alpha,K} Δ ∈ A α , K , where A α , K : = ⋃ μ ∈ K K A α ( μ ) \mathcal{A}_{\alpha,K} := \bigcup_{\mu\in\mathcal{K}_K}\mathcal{A}_{\alpha}(\mu) A α , K := ⋃ μ ∈ K K A α ( μ ) . Since a linear functional has the same maximum over a set and its convex hull, the BMuon oracle maximizes ⟨ M , Δ ⟩ \langle M,\Delta\rangle ⟨ M , Δ ⟩ over Δ ∈ B α , K \Delta\in\mathcal{B}_{\alpha,K} Δ ∈ B α , K , where B α , K : = conv ( A α , K ) \mathcal{B}_{\alpha,K} := \operatorname{conv}(\mathcal{A}_{\alpha,K}) B α , K := conv ( A α , K ) .
As shown in Appendix B.4, B α , K \mathcal{B}_{\alpha,K} B α , K is compact, convex, centrally symmetric, and has the origin in its interior. Its Minkowski functional Ω α , K ( Δ ) : = inf { t > 0 : Δ ∈ t B α , K } \Omega_{\alpha,K}(\Delta) := \inf\{t > 0 : \Delta\in t\mathcal{B}_{\alpha,K}\} Ω α , K ( Δ ) := inf { t > 0 : Δ ∈ t B α , K } is therefore a norm whose unit ball is B α , K \mathcal{B}_{\alpha,K} B α , K . Thus, for fixed α \alpha α , K K K , BMuon is exactly normalized steepest descent under Ω α , K \Omega_{\alpha,K} Ω α , K :
Δ ⋆ ∈ arg max Ω α , K ( Δ ) ≤ 1 ⟨ M , Δ ⟩ . (6) \Delta^\star\in\arg\max_{\Omega_{\alpha,K}(\Delta)\le1}\langle M,\Delta\rangle. \tag*{(6)} Δ ⋆ ∈ arg Ω α , K ( Δ ) ≤ 1 max ⟨ M , Δ ⟩ . (6)
In comparison with Eq. (2) , BMuon uses a modified norm constraint, hence a different geometry.
From Muon to Sparse Geometries
We prove that the family Ω α , K \Omega_{\alpha,K} Ω α , K interpolates between Muon’s spectral geometry and channel-sparse geometries. Proofs of these special cases are given in Appendix B.5.
Recovering Muon. If K = N K=N K = N , the constraints μ j ℓ ≤ 1 \mu_j^\ell\le1 μ j ℓ ≤ 1 and ∑ ℓ , j μ j ℓ = N \sum_{\ell,j}\mu_j^\ell=N ∑ ℓ , j μ j ℓ = N force μ j ℓ = 1 \mu_j^\ell=1 μ j ℓ = 1 for every channel, so B α , N = A M u o n \mathcal{B}_{\alpha,N}=\mathcal{A}_{\mathrm{Muon}} B α , N = A Muon and Ω α , N = ∥ ⋅ ∥ M u o n \Omega_{\alpha,N}=\|\cdot\|_{\mathrm{Muon}} Ω α , N = ∥ ⋅ ∥ Muon . Muon is also recovered for any K K K by extending the family to α = 0 \alpha=0 α = 0 with D μ 0 = I D_\mu^0=I D μ 0 = I .
A structured K K K -support geometry. Assume K K K is an integer and α = 1 \alpha= 1 α = 1 . Define supp c h ( Δ ) : = { ( ℓ , j ) : Δ 1 , j , : ℓ ≠ 0 or Δ 2 , : , j ℓ ≠ 0 } \operatorname{supp}_{\mathrm{ch}}(\Delta) := \{(\ell,j) : \Delta^{\ell}_{1,j,:} \ne0 \text{ or } \Delta^{\ell}_{2,:,j} \ne0\} supp ch ( Δ ) := {( ℓ , j ) : Δ 1 , j , : ℓ = 0 or Δ 2 , : , j ℓ = 0 } . The induced unit ball is
B 1 , K = conv { Δ : ∣ supp c h ( Δ ) ∣ ≤ K , ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } . \mathcal{B}_{1,K} = \operatorname{conv}\left\{\Delta: \left|\operatorname{supp}_{\mathrm{ch}}(\Delta)\right| \le K,\ \left\|\Delta^{\ell}_{1}\right\|_{\mathrm{op}} \le1,\ \left\|\Delta^{\ell}_{2}\right\|_{\mathrm{op}} \le1,\ \forall\ell\right\}. B 1 , K = conv { Δ : ∣ supp ch ( Δ ) ∣ ≤ K , Δ 1 ℓ op ≤ 1 , Δ 2 ℓ op ≤ 1 , ∀ ℓ } .
Thus Ω 1 , K \Omega_{1,K} Ω 1 , K is closely related to k k k -support and group k k k -support norms [3 , 25 ] , but BMuon uses coupled hidden-channel support and operator-norm constraints rather than Euclidean ones. It is also related to spectral k k k -support norms [22 , 23 ] , but sparsity acts on hidden channels rather than singular directions.
A coupled group-ℓ 1 \ell_1 ℓ 1 geometry. At α = K = 1 \alpha= K = 1 α = K = 1 , Ω 1 , 1 ( Δ ) = ∑ ℓ , j max { ∥ Δ 1 , j , : ℓ ∥ 2 , ∥ Δ 2 , : , j ℓ ∥ 2 } \Omega_{1,1}(\Delta) = \sum_{\ell,j} \max\{\|\Delta^{\ell}_{1,j,:}\|_2, \|\Delta^{\ell}_{2,:,j}\|_2\} Ω 1 , 1 ( Δ ) = ∑ ℓ , j max { ∥ Δ 1 , j , : ℓ ∥ 2 , ∥ Δ 2 , : , j ℓ ∥ 2 } . This is an ℓ 1 \ell_1 ℓ 1 aggregation over coupled hidden channels and therefore promotes channel sparsity.
Representation Cost and Intrinsic Width
Section 2 showed that the BMuon oracle performs steepest descent under Ω α , K \Omega_{\alpha,K} Ω α , K . We now ask whether minimizing Ω α , K \Omega_{\alpha,K} Ω α , K among parameterizations that realize the same predictor favors representations with fewer active hidden channels. This question is motivated by prior work showing that the norm underlying steepest descent can directly shape its implicit bias [13 , 28 , 9 , 15 ] . Rather than establishing an implicit-bias result for BMuon, we characterize the exact parameterizations of a fixed predictor that minimize Ω α , K \Omega_{\alpha,K} Ω α , K , following the representation-cost view of [7 ] (Jagadeesan et al. 2022).
We fix predictors X = ( X ℓ ) ℓ = 1 L X = (X_\ell)_{\ell=1}^{L} X = ( X ℓ ) ℓ = 1 L , where each X ℓ ∈ R p ℓ × d ℓ X_\ell\in\mathbb{R}^{p_\ell\times d_\ell} X ℓ ∈ R p ℓ × d ℓ is nonzero, and denote r ℓ : = rank ( X ℓ ) r_\ell:= \operatorname{rank}(X_\ell) r ℓ := rank ( X ℓ ) . We consider linear two-factor blocks W = ( ( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L W = ((W^\ell_1,W^\ell_2))_{\ell=1}^{L} W = (( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L , with W 1 ℓ ∈ R m ℓ × d ℓ W^\ell_1 \in\mathbb{R}^{m_\ell\times d_\ell} W 1 ℓ ∈ R m ℓ × d ℓ , W 2 ℓ ∈ R p ℓ × m ℓ W^\ell_2 \in\mathbb{R}^{p_\ell\times m_\ell} W 2 ℓ ∈ R p ℓ × m ℓ , and m ℓ ≥ r ℓ m_\ell\ge r_\ell m ℓ ≥ r ℓ . Each block realizes the matrix factorization W 2 ℓ W 1 ℓ W^\ell_2W^\ell_1 W 2 ℓ W 1 ℓ , corresponding to the setup in Eq. (1) with the identity activation and no bias. For any parameter norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ , define its representation cost
R ∥ ⋅ ∥ ( X ) : = inf { ∥ W ∥ 2 : W 2 ℓ W 1 ℓ = X ℓ , ∀ ℓ } . \mathcal{R}_{\|\cdot\|}(X) := \inf\left\{\|W\|^2 : W^\ell_2W^\ell_1 = X_\ell,\ \forall\ell\right\}. R ∥ ⋅ ∥ ( X ) := inf { ∥ W ∥ 2 : W 2 ℓ W 1 ℓ = X ℓ , ∀ ℓ } .
We write R α , K : = R Ω α , K \mathcal{R}_{\alpha,K} := \mathcal{R}_{\Omega_{\alpha,K}} R α , K := R Ω α , K for BMuon and R M u o n : = R ∥ ⋅ ∥ M u o n \mathcal{R}_{\mathrm{Muon}} := \mathcal{R}_{\|\cdot\|_{\mathrm{Muon}}} R Muon := R ∥ ⋅ ∥ Muon for Muon. For a factorization W W W , let a ℓ ( W ) a_\ell(W) a ℓ ( W ) denote the active width of block ℓ \ell ℓ , i.e., the number of hidden channels j j j for which W 1 , j , : ℓ ≠ 0 W^\ell_{1,j,:} \ne0 W 1 , j , : ℓ = 0 or W 2 , : , j ℓ ≠ 0 W^\ell_{2,:,j} \ne0 W 2 , : , j ℓ = 0 . Every exact factorization of X X X satisfies a ℓ ( W ) ≥ r ℓ a_\ell(W) \ge r_\ell a ℓ ( W ) ≥ r ℓ for all ℓ \ell ℓ , so r ℓ r_\ell r ℓ is the minimum possible active width of block ℓ \ell ℓ .
Let σ ℓ , 1 ≥ ⋯ ≥ σ ℓ , r ℓ > 0 \sigma_{\ell,1} \ge\cdots\ge\sigma_{\ell,r_\ell} > 0 σ ℓ , 1 ≥ ⋯ ≥ σ ℓ , r ℓ > 0 denote the nonzero singular values of X ℓ X_\ell X ℓ , let σ ⋆ : = max ℓ , i σ ℓ , i \sigma_\star:= \max_{\ell,i}\sigma_{\ell,i} σ ⋆ := max ℓ , i σ ℓ , i , and define the α \alpha α -effective spectral dimension d e f f ( α ) ( X ) : = ∑ ℓ = 1 L ∑ i = 1 r ℓ ( σ ℓ , i / σ ⋆ ) 1 / ( 2 α ) d_{\mathrm{eff}}^{(\alpha)}(X) := \sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}(\sigma_{\ell,i}/\sigma_\star)^{1/(2\alpha)} d eff ( α ) ( X ) := ∑ ℓ = 1 L ∑ i = 1 r ℓ ( σ ℓ , i / σ ⋆ ) 1/ ( 2 α ) .
Theorem 2 (Representation cost and intrinsic-width recovery) Let 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 and 0 < K ≤ N 0 < K \le N 0 < K ≤ N . Both representation-cost infima are attained, and
R M u o n ( X ) = σ ⋆ , R α , K ( X ) = σ ⋆ max { 1 , ( d e f f ( α ) ( X ) / K ) 2 α } . \mathcal{R}_{\mathrm{Muon}}(X) = \sigma_\star,\qquad\mathcal{R}_{\alpha,K}(X) = \sigma_\star\max\left\{1,\left(d_{\mathrm{eff}}^{(\alpha)}(X)/K\right)^{2\alpha}\right\}. R Muon ( X ) = σ ⋆ , R α , K ( X ) = σ ⋆ max { 1 , ( d eff ( α ) ( X ) / K ) 2 α } . Moreover, Muon admits a minimizer W ⋆ W^\star W ⋆ with a ℓ ( W ⋆ ) = m ℓ a_\ell(W^\star) = m_\ell a ℓ ( W ⋆ ) = m ℓ for every ℓ \ell ℓ , whereas if K ≤ d e f f ( α ) ( X ) K \le d_{\mathrm{eff}}^{(\alpha)}(X) K ≤ d eff ( α ) ( X ) , every BMuon minimizer W ⋆ W^\star W ⋆ satisfies a ℓ ( W ⋆ ) = r ℓ a_\ell(W^\star) = r_\ell a ℓ ( W ⋆ ) = r ℓ for all $\ell.
Proof is given in Appendix C.1.
Interpretation. Muon’s geometric cost is insensitive to hidden-channel redundancy, since its minimum can be attained at full hidden width. In contrast, BMuon is sensitive to redundant width: when K ≤ d e f f ( α ) ( X ) K \le d_{\mathrm{eff}}^{(\alpha)}(X) K ≤ d eff ( α ) ( X ) , every minimum-cost factorization is width-minimal, so any redundancy incurs additional geometric cost. The budget raises the representation cost only when K < d e f f ( α ) ( X ) K < d_{\mathrm{eff}}^{(\alpha)}(X) K < d eff ( α ) ( X ) , in which case it is σ ⋆ ( d e f f ( α ) ( X ) / K ) 2 α \sigma_\star\left(d_{\mathrm{eff}}^{(\alpha)}(X)/K\right)^{2\alpha} σ ⋆ ( d eff ( α ) ( X ) / K ) 2 α ; otherwise, it is σ ⋆ \sigma_\star σ ⋆ . Since d e f f ( α ) ( X ) d_{\mathrm{eff}}^{(\alpha)}(X) d eff ( α ) ( X ) is nondecreasing in α \alpha α , increasing α \alpha α can only enlarge the budget regime in which width-minimality is guaranteed.
Effective spectral dimension. The α \alpha α -effective spectral dimension recovers familiar spectral quantities. Let X b d : = diag ( X 1 , … , X L ) X_{\mathrm{bd}} := \operatorname{diag}(X_1,\ldots,X_L) X bd := diag ( X 1 , … , X L ) . For α = 1 4 \alpha= \frac{1}{4} α = 4 1 , d e f f ( α ) ( X ) = ∥ X b d ∥ F 2 / ∥ X b d ∥ o p 2 d_{\mathrm{eff}}^{(\alpha)}(X) = \lVert X_{\mathrm{bd}}\rVert_F^2/\lVert X_{\mathrm{bd}}\rVert_{\mathrm{op}}^2 d eff ( α ) ( X ) = ∥ X bd ∥ F 2 / ∥ X bd ∥ op 2 , the stable rank of X b d X_{\mathrm{bd}} X bd , while for α = 1 2 \alpha= \frac{1}{2} α = 2 1 , d e f f ( α ) ( X ) = ∥ X b d ∥ ∗ / ∥ X b d ∥ o p d_{\mathrm{eff}}^{(\alpha)}(X) = \lVert X_{\mathrm{bd}}\rVert_\ast/\lVert X_{\mathrm{bd}}\rVert_{\mathrm{op}} d eff ( α ) ( X ) = ∥ X bd ∥ ∗ / ∥ X bd ∥ op .
Practical BMuon
Exactly solving the allocation problem in Eq. (4) at every optimizer step is expensive. We instead perform a minorize–maximize (MM) update [14 ] , replacing the exact allocation problem by the maximization of a surrogate. For fixed momenta M M M and exponent α \alpha α , given an allocation μ ˉ \bar{\mu} μ ˉ , let P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) \bar{P}^{\ell}_{1} = \operatorname{Polar}(D_{\bar{\mu}^{\ell}}^{\alpha}M_{1}^{\ell}) P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) and P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) \bar{P}^{\ell}_{2} = \operatorname{Polar}(M_{2}^{\ell}D_{\bar{\mu}^{\ell}}^{\alpha}) P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) . As shown in Appendix D.§ 1 , we have the tight minorization
Φ M ( μ ) ≥ Q ( μ ∣ μ ˉ ) : = ∑ ℓ , j a j ℓ ( μ j ℓ ) α , a j ℓ = ⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ , (7) \Phi_M(\mu) \ge Q(\mu\mid\bar{\mu}) := \sum_{\ell,j} a_j^\ell(\mu_j^\ell)^\alpha,\qquad a_j^\ell= \langle M_{1,:,j}^\ell,\bar{P}_{1,:,j}^\ell\rangle+ \langle M_{2,:,j}^\ell,\bar{P}_{2,:,j}^\ell\rangle, \tag*{(7)} Φ M ( μ ) ≥ Q ( μ ∣ μ ˉ ) := ℓ , j ∑ a j ℓ ( μ j ℓ ) α , a j ℓ = ⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ , (7)
with equality at μ = μ ˉ \mu= \bar{\mu} μ = μ ˉ . At the boundary μ ˉ j ℓ = 0 \bar{\mu}_j^\ell= 0 μ ˉ j ℓ = 0 , row j j j of D μ ˉ ℓ α M 1 ℓ D_{\bar{\mu}^\ell}^{\alpha}M_1^\ell D μ ˉ ℓ α M 1 ℓ and column j j j of M 2 ℓ D μ ˉ ℓ α M_2^\ell D_{\bar{\mu}^\ell}^{\alpha} M 2 ℓ D μ ˉ ℓ α vanish, giving a j ℓ = 0 a_j^\ell= 0 a j ℓ = 0 , so the surrogate no longer depends on μ j ℓ \mu_j^\ell μ j ℓ . We therefore impose a small floor μ j ℓ ≥ ε \mu_j^\ell\ge\varepsilon μ j ℓ ≥ ε in practice and maximize the surrogate over K K , ε : = { μ ∈ [ ε , 1 ] N : ∑ ℓ , j μ j ℓ = K } \mathcal{K}_{K,\varepsilon} := \{\mu\in[\varepsilon,1]^{N} : \sum_{\ell,j}\mu_j^\ell= K\} K K , ε := { μ ∈ [ ε , 1 ] N : ∑ ℓ , j μ j ℓ = K } , where 0 < ε < K / N 0 < \varepsilon< K/N 0 < ε < K / N . Proposition 10 bounds the gap between the floored and original oracle objectives. The scores a j ℓ a_j^\ell a j ℓ are nonnegative in general and, if for every ( ℓ , j ) (\ell,j) ( ℓ , j ) , M 1 , : , j ℓ ≠ 0 M_{1,:,j}^\ell\ne0 M 1 , : , j ℓ = 0 or M 2 , : , j ℓ ≠ 0 M_{2,:,j}^\ell\ne0 M 2 , : , j ℓ = 0 , they are strictly positive for every μ ˉ ∈ K K , ε \bar{\mu} \in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε . Hence, for 0 < α < 1 0 < \alpha< 1 0 < α < 1 , the surrogate is strictly concave and has a unique maximizer. The KKT conditions, derived in Appendix D.§ 2 , give
( μ j ℓ ) + = clip [ ε , 1 ] [ ( α a j ℓ λ μ ) 1 / ( 1 − α ) ] , (8) (\mu_j^\ell)^+ = \operatorname{clip}_{[\varepsilon,1]}\left[\left(\frac{\alpha a_j^\ell}{\lambda_\mu}\right)^{1/(1-\alpha)}\right], \tag*{(8)} ( μ j ℓ ) + = clip [ ε , 1 ] ( λ μ α a j ℓ ) 1/ ( 1 − α ) , (8)
where clip [ ε , 1 ] ( x ) : = min { 1 , max { ε , x } } \operatorname{clip}_{[\varepsilon,1]}(x) := \min\{1,\max\{\varepsilon,x\}\} clip [ ε , 1 ] ( x ) := min { 1 , max { ε , x }} , and λ μ > 0 \lambda_\mu> 0 λ μ > 0 is found by bisection from ∑ ℓ , j ( μ j ℓ ) + = K \sum_{\ell,j}(\mu_j^\ell)^+ = K ∑ ℓ , j ( μ j ℓ ) + = K . Thus, the MM allocation step introduces no additional step-size hyperparameter. For fixed momenta, each MM step is monotone since Φ M ( μ + ) ≥ Q ( μ + ∣ μ ˉ ) ≥ Q ( μ ˉ ∣ μ ˉ ) = Φ M ( μ ˉ ) \Phi_M(\mu^+) \ge Q(\mu^+ \mid\bar{\mu}) \ge Q(\bar{\mu} \mid\bar{\mu}) = \Phi_M(\bar{\mu}) Φ M ( μ + ) ≥ Q ( μ + ∣ μ ˉ ) ≥ Q ( μ ˉ ∣ μ ˉ ) = Φ M ( μ ˉ ) . Moreover, for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 , repeated MM steps converge in objective value to the global optimum over K K , ε \mathcal{K}_{K,\varepsilon} K K , ε , and every accumulation point of the allocation sequence is globally optimal. The proof is given in Appendix D.§ 3 .
Algorithm 1 summarizes the resulting update. When μ \mu μ is updated, the first polar evaluation constructs the MM surrogate at the current allocation. After the MM step, the polar factors are recomputed using the new allocation to form the parameter update.
The budget K K K can be scheduled during training for example by annealing it from N N N to a target value. Parameters outside the MLP blocks are updated as in the corresponding Muon baseline.
The special cases in § 2.4 place Muon at the α = 0 \alpha=0 α = 0 endpoint and channel-sparse geometries at α = 1 \alpha=1 α = 1 , suggesting that larger α \alpha α corresponds to more sparsity-favoring geometries. Moreover, away from the clipping boundaries, Eq. (8) implies ( μ i ) + / ( μ j ) + = ( a i / a j ) 1 / ( 1 − α ) (\mu_i)^+/(\mu_j)^+=(a_i/a_j)^{1/(1-\alpha)} ( μ i ) + / ( μ j ) + = ( a i / a j ) 1/ ( 1 − α ) , so larger α \alpha α amplifies differences between channel scores and favors more concentrated allocations. At the same time, α = 1 / 2 \alpha=1/2 α = 1/2 is the largest exponent for which the allocation objective is guaranteed to remain concave by Theorem 1 and for which repeated MM steps are guaranteed to converge in objective value by Theorem 9 . We therefore use α = 1 / 2 \alpha=1/2 α = 1/2 throughout our experiments.
In practice, we approximate each polar factor by a finite Newton–Schulz iteration, as in standard Muon, and update the allocation by a single MM step every Δ T μ \Delta T_\mu Δ T μ optimizer steps, keeping it fixed between these updates. We assess these practical approximations empirically in Appendix D.§ 5 , using Δ T μ = 5 \Delta T_\mu=5 Δ T μ = 5 and a certified reference allocation obtained by repeatedly applying the exact-polar MM update until its relative objective loss from the exact optimum is certified to be below a small threshold. In these diagnostics, the relative loss in the allocation objective incurred by the practical one-step Newton–Schulz-based update is certified to remain on the order of 10 − 4 10^{-4} 1 0 − 4 to 10 − 3 10^{-3} 1 0 − 3 , while keeping the allocation fixed for the four intervening optimizer steps keeps the corresponding certified deficit on the order of 10 − 3 10^{-3} 1 0 − 3 . In the reported comparison, replacing the practical allocation update every five optimizer steps by the certified reference allocation leaves test accuracy and post-training compressibility unchanged within the observed seed-to-seed variability.
Relative to Muon, BMuon adds only channelwise scalings on the budgeted MLP matrices and, every Δ T μ \Delta T_\mu Δ T μ steps, one extra polar evaluation per such matrix; all other parameters are updated as in Muon. In our ImageNet-1K and 1B-parameter LLaMA experiments in Sections 5.2 and 5.3, respectively, we observe no measurable increase in overall training time relative to Muon (Appendix D.§ 6 ).
Experiments
Post-Training Compression
Setup. We first test whether BMuon produces models that can be compressed directly after training without post-compression fine-tuning. We train a small Vision Transformer on CIFAR-100 for 200 epochs over three seeds. We compare Muon, Muon with a proximal Group-Lasso penalty on coupled MLP channels (Muon+GL) [34 , 31 ] , NuMuon [8 ] , and BMuon. NuMuon is particularly relevant as a direct comparison: it modifies the Muon update to improve compressibility, but targets low-rank structure through the singular spectrum, whereas BMuon targets structured MLP-channel compression. BMuon uses α = 1 / 2 \alpha= 1/2 α = 1/2 and a budget annealed to 20% of the total MLP-channel mass. Full experimental details in Appendix E.§ 2 .
Post-training compression. We restrict the compression to the ViT MLP blocks. Muon, Muon+GL, and BMuon are compressed by globally pruning hidden channels across all blocks. For magnitude-based pruning, channel j j j in block ℓ \ell ℓ is ranked by
s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 , : , j ℓ ∥ 2 2 s_{j}^{\ell} = \sqrt{\lVert W_{1:,j}^{\ell}\rVert_{2}^{2} + \lVert W_{2,:,j}^{\ell}\rVert_{2}^{2}} s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 , : , j ℓ ∥ 2 2
BMuon is pruned using both this score (BM-mag) and its final learned allocation μ \mu μ (BM-μ \mu μ ). NuMuon instead compresses each MLP weight matrix independently using truncated SVD, with ranks chosen to approximately match the parameter count of the corresponding channel-pruned model.
Results. Figure 2 shows the post-training fragility that motivates this work: Muon falls to 5.72% test accuracy after only 30% MLP-channel pruning. Muon+GL remains robust through 50% pruning, while NuMuon degrades more gradually under low-rank compression. Under magnitude-based pruning, BMuon is substantially more compressible than the structured baselines while matching Muon’s dense accuracy: BMuon reaches 71.46%, versus 71.49% for Muon, 71.25% for Muon+GL, and 69.95% for NuMuon. Using μ \mu μ for ranking provides a further gain: at 80% pruning, BM-μ \mu μ retains 70.57% accuracy, compared with 60.84% for BM-mag, while Muon, Muon+GL, and NuMuon have fallen to 1.93%, 6.01%, and 3.84%, respectively. The advantage of BM-mag under the shared magnitude criterion isolates the compressibility benefit of BMuon training, while the further gain from BM-μ \mu μ indicates that the learned allocation also carries useful channel-importance information.
Figure 2. CIFAR-100 ViT post-training MLP compression. Mean test accuracy ± \pm ± sample std. over three seeds after compression without recovery. Muon, Muon+GL, and BM-mag use global MLP-channel pruning ranked by magnitude; BM-μ \mu μ uses μ \mu μ for ranking, while NuMuon uses parameter-matched truncated-SVD compression of the MLP weights.
Gradual Structured Pruning
We next ask whether BMuon’s compressibility advantage persists when channel pruning is interleaved with training and recovery [1 ] , and whether its learned allocation identifies useful architectures when retrained from scratch.
Setup. We evaluate gradual MLP-channel pruning during training from scratch on a ViT for CIFAR-100 and a ViT-S/16 for ImageNet-1K, targeting final pruning levels of 90% and 95%. All results are reported over three seeds. Each run starts dense and alternates between training and pruning. After each training phase, we rank all surviving MLP channels globally, permanently remove the lowest-ranked ones to reach the next sparsity target, and resume training from the surviving weights. We repeat this train–prune cycle until the final pruning level is reached, and then continue training for recovery. We compare three methods. Muon-mag trains with Muon and ranks channels by the coupled weight-magnitude score introduced in § 5.1 . BMuon-mag trains the MLPs with BMuon (α = 1 / 2 \alpha= 1/2 α = 1/2 ) and uses the same magnitude ranking. BMuon-μ \mu μ uses the same BMuon training but ranks channels by the learned allocation μ \mu μ . During each BMuon training phase, the global budget is annealed to the next target channel count over the first half of the phase and held fixed for the second half.
As an additional control, we test whether the architecture selected by BMuon-μ \mu μ is useful on its own. We keep only its final per-block widths, initialize a new model with those widths, and train it from scratch with standard Muon for the same training horizon. As a baseline, we also train a uniform-width model from scratch with nearly the same total number of MLP channels. Full details are given in Appendix E.§ 4 .
Table 1 (PDF p. 11). Gradual global MLP-channel pruning. Final Top-1 accuracy (%), reported as mean ± sample std. over three seeds. The upper block reports the final pruned models after recovery; the lower block trains the architectures selected by BMuon-μ \mu μ from scratch with Muon and compares their accuracy with uniform-width models having nearly the same total number of MLP channels. Details in Appendix E.§ 4 .
Gradual pruning performance. Table 1 shows that BMuon consistently improves over Muon under gradual pruning. Using the same magnitude-based pruning criterion, BMuon-mag exceeds Muon-mag by 1.78 and 3.18 percentage points on CIFAR-100 at 90% and 95% pruning, and by 3.45 and 4.53 points on ImageNet-1K. Using μ \mu μ as the pruning score then provides an additional gain of 2.46 and 1.03 points on CIFAR-100, and 0.35 and 1.27 points on ImageNet-1K.
Table 1.
Architecture discovery. When trained from scratch, the architectures selected by BMuon-μ \mu μ outperform the nearly channel-matched uniform models in all four settings. Because both the discovered and uniform architectures are initialized from scratch and trained with standard Muon, this comparison isolates the effect of the architecture’s width profile. The results therefore provide evidence that BMuon-μ \mu μ identifies useful non-uniform architectures.
Structured Compressibility at Language-Model Scale
The BMuon construction on MLP hidden channels extends naturally to gated MLPs such as SwiGLU, as used in LLaMA-style models. The same allocation acts on the rows of the gate and up projections and on the columns of the down projection; see Appendix B.§ 2 for details.
To test whether BMuon’s structured-compressibility advantage extends to large-scale language modeling, we train a 1B-parameter LLaMA-style model on 13.1B FineWeb tokens over three seeds. BMuon uses α = 1 / 2 \alpha= 1/2 α = 1/2 and K = 0.2 N K = 0.2N K = 0.2 N . After training, we globally prune SwiGLU hidden channels without recovery, using weight magnitude and an activation-aware score combining root-mean-square activation and output-weight norm; for BMuon, we also rank channels by μ \mu μ . The pruning curves are shown in Figure 3 ; full details are given in Appendix E.§ 5 .
Figure 3. Post-training MLP-channel pruning at language-model scale. Validation perplexity after global structured pruning of a 1B-parameter LLaMA-style model trained on FineWeb. Mean ± std. over three seeds.
On the unpruned models, Muon achieves lower perplexity (10.92 versus 12.84 for BMuon), but its performance degrades rapidly under pruning. In contrast, BMuon preserves its unpruned performance up to 80% pruning, reaching 12.86 with activation ranking and 12.85 with μ \mu μ ranking. BMuon substantially outperforms Muon when both use the same magnitude or activation criterion at moderate and high pruning levels, while μ \mu μ again provides an effective pruning score. Overall, BMuon trades some unpruned quality for substantially greater robustness to structured channel pruning.
Conclusion
Muon-trained Transformers can degrade sharply under post-training MLP-channel pruning, motivating the question of whether Muon’s spectral geometry can be modified to favor structured compressibility. BMuon addresses this question by reweighting hidden channels within Muon’s spectral-norm constraints and making them compete for a global budget. We show that the resulting norm family recovers Muon and connects it to channel-sparse geometries. In linear two-factor models, Muon admits minimum-cost factorizations with redundant hidden width, whereas under a sufficiently scarce BMuon budget every minimum-cost factorization is width-minimal. For practical training, we derive a tractable minorization–maximization approximation to the global allocation oracle. Across vision and language models, BMuon is substantially more robust than Muon to magnitude-based MLP-channel pruning. The learned allocation also provides an effective pruning score and identifies useful non-uniform MLP architectures when retrained from scratch.
Several questions remain open. Our theoretical results characterize representation cost in linear two-factor models; studying the implicit bias of BMuon training, particularly beyond the linear setting, is an important direction for future work. Another direction is to design a practical BMuon algorithm with provable approximation guarantees to the oracle. Our experiments focus on MLP hidden channels, and extending the global budget to other structured components of Transformers remains to be explored. Finally, BMuon can trade dense-model quality for greater pruning robustness, as observed in our language-model experiments; characterizing this tradeoff at larger model scales is a natural direction for future work.
Acknowledgments
Acknowledgments. YI gratefully acknowledges the support of Alberta Innovates (ALLRP 600038-24), the Natural Sciences and Engineering Research Council of Canada (NSERC) (RGPIN-2022-03120, DGECR-2022-00358), and NSERC/Agence Nationale de la Recherche (ANR) (ALLRP 602719-24). This project was undertaken thanks to funding from IVADO and the Canada First Research Excellence Fund. This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).
RB and TJ gratefully acknowledge funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116395 SPARSE-ML.
U.S. is partially supported by the French government under the management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and the grant GHOST ANR-25-CE23-4985, and by the European Research Council Starting Grant DYNASTY – 101039676.
Appendix
Muon Algorithm
Muon applies momentum followed by approximate matrix orthogonalization to eligible matrix parameters Jordan et al., [16 ] . Algorithm 2 gives the update for one matrix parameter using the Nesterov-style momentum variant.
Algorithm 2 Muon
Require: Learning rates { η t } \{\eta_t\} { η t } , momentum β \beta β
Initialize B 0 ← 0 B_0 \leftarrow0 B 0 ← 0
for t = 1 , 2 , … t = 1, 2, \ldots t = 1 , 2 , … do
G t ← ∇ W L t ( W t − 1 ) G_t \leftarrow\nabla_W L_t(W_{t-1}) G t ← ∇ W L t ( W t − 1 )
B t ← β B t − 1 + G t B_t \leftarrow\beta B_{t-1} + G_t B t ← β B t − 1 + G t
M t ← G t + β B t M_t \leftarrow G_t + \beta B_t M t ← G t + β B t
O t ← NewtonSchulz5 ( M t ) O_t \leftarrow\operatorname{NewtonSchulz5}(M_t) O t ← NewtonSchulz5 ( M t )
W t ← W t − 1 − η t O t W_t \leftarrow W_{t-1} - \eta_t O_t W t ← W t − 1 − η t O t
end for
The non-Nesterov variant instead uses M t = B t M_t = B_t M t = B t . Throughout the paper, momentum refers to M t M_t M t , the matrix that Muon orthogonalizes; our geometric results are stated for arbitrary M t M_t M t and therefore do not depend on the momentum variant used to produce it. We specify the variant used in each experimental setup.
Practical Muon implementations may additionally rescale each matrix update. The public Muon implementation uses the aspect-ratio correction max { 1 , m / n } \sqrt{\max\{1,m/n\}} max { 1 , m / n } for W ∈ R m × n W \in\mathbb{R}^{m \times n} W ∈ R m × n Jordan et al., [16 ] , while the large-scale RMS-matching convention of Liu et al. [20 ] uses 0.2 max { m , n } 0.2\sqrt{\max\{m,n\}} 0.2 max { m , n } . Whenever such a rescaling is used, we state it in the corresponding experimental setup; otherwise, no rescaling is applied. We omit it from the geometric analysis.
Parameters not optimized by Muon, including biases, normalization parameters, embeddings, and classifier or output heads, are optimized separately with AdamW. With decoupled weight decay, the final update becomes
W t = ( 1 − η t λ ) W t − 1 − η t γ O t . W_t = (1-\eta_t\lambda)W_{t-1}-\eta_t\gamma O_t. W t = ( 1 − η t λ ) W t − 1 − η t γ O t .
Newton–Schulz orthogonalization. For M ∈ R m × n M \in\mathbb{R}^{m \times n} M ∈ R m × n , let M ~ = M \widetilde{M}=M M = M if m ≤ n m\le n m ≤ n and M ~ = M T \widetilde{M}=M^{\mathsf{T}} M = M T otherwise, and set X 0 = M ~ / ( ∥ M ~ ∥ F + 10 − 7 ) X_0=\widetilde{M}/(\lVert\widetilde{M}\rVert_F+10^{-7}) X 0 = M / (∥ M ∥ F + 1 0 − 7 ) . Muon then applies five iterations
X k + 1 = a X k + b ( X k X k T ) X k + c ( X k X k T ) 2 X k , k = 0 , … , 4. X_{k+1}=aX_k+b(X_kX_k^{\mathsf{T}})X_k+c(X_kX_k^{\mathsf{T}})^2X_k,\qquad k=0,\ldots,4. X k + 1 = a X k + b ( X k X k T ) X k + c ( X k X k T ) 2 X k , k = 0 , … , 4.
with ( a , b , c ) = ( 3.4445 , − 4.7750 , 2.0315 ) (a,b,c)=(3.4445,-4.7750,2.0315) ( a , b , c ) = ( 3.4445 , − 4.7750 , 2.0315 ) Jordan et al., [16 ] . We define NewtonSchulz5 ( M ) = X 5 \operatorname{NewtonSchulz5}(M)=X_5 NewtonSchulz5 ( M ) = X 5 if m ≤ n m\le n m ≤ n , and NewtonSchulz5 ( M ) = X 5 T \operatorname{NewtonSchulz5}(M)=X_5^{\mathsf{T}} NewtonSchulz5 ( M ) = X 5 T otherwise. This finite iteration approximately orthogonalizes M M M ; the main text uses the exact polar direction for the theoretical analysis.
BMuon Oracle and Geometry
This appendix provides the derivations and proofs underlying § 2 . Appendix B.§ 1 derives the BMuon allocation problem and update in Eqs. (4) –(5) from the joint oracle in Eq. (3) . Appendix B.§ 2 extends the construction to gated MLPs such as SwiGLU. Appendix B.§ 3 proves the curvature transition of Theorem 1 . Appendix B.§ 4 establishes the induced norm geometry Ω α , K \Omega_{\alpha,K} Ω α , K and its characterization for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 . Finally, Appendix B.§ 5 proves the geometric special cases stated in § 2.4 .
Notation and setup. We consider L L L coupled MLP blocks with W 1 ℓ ∈ R m ℓ × d ℓ W_1^\ell\in\mathbb{R}^{m_\ell\times d_\ell} W 1 ℓ ∈ R m ℓ × d ℓ and W 2 ℓ ∈ R p ℓ × m ℓ W_2^\ell\in\mathbb{R}^{p_\ell\times m_\ell} W 2 ℓ ∈ R p ℓ × m ℓ , and write N : = ∑ ℓ = 1 L m ℓ N := \sum_{\ell=1}^{L} m_\ell N := ∑ ℓ = 1 L m ℓ for the total number of hidden channels. Their Muon momentum matrices are denoted M 1 ℓ M_1^\ell M 1 ℓ and M 2 ℓ M_2^\ell M 2 ℓ , with corresponding update directions Δ 1 ℓ \Delta_1^\ell Δ 1 ℓ and Δ 2 ℓ \Delta_2^\ell Δ 2 ℓ . We write M : = ( ( M 1 ℓ , M 2 ℓ ) ) ℓ = 1 L M := ((M_1^\ell,M_2^\ell))_{\ell=1}^{L} M := (( M 1 ℓ , M 2 ℓ ) ) ℓ = 1 L and Δ : = ( ( Δ 1 ℓ , Δ 2 ℓ ) ) ℓ = 1 L \Delta:= ((\Delta_1^\ell,\Delta_2^\ell))_{\ell=1}^{L} Δ := (( Δ 1 ℓ , Δ 2 ℓ ) ) ℓ = 1 L , with ⟨ M , Δ ⟩ : = ∑ ℓ = 1 L ( ⟨ M 1 ℓ , Δ 1 ℓ ⟩ + ⟨ M 2 ℓ , Δ 2 ℓ ⟩ ) \langle M,\Delta\rangle:= \sum_{\ell=1}^{L}(\langle M_1^\ell,\Delta_1^\ell\rangle+\langle M_2^\ell,\Delta_2^\ell\rangle) ⟨ M , Δ ⟩ := ∑ ℓ = 1 L (⟨ M 1 ℓ , Δ 1 ℓ ⟩ + ⟨ M 2 ℓ , Δ 2 ℓ ⟩) . For a global budget K ∈ ( 0 , N ] K \in(0,N] K ∈ ( 0 , N ] , BMuon uses allocations μ = ( μ 1 , … , μ L ) \mu=(\mu^1,\ldots,\mu^L) μ = ( μ 1 , … , μ L ) in K K : = { μ ∈ [ 0 , 1 ] N : ∑ ℓ , j μ j ℓ = K } \mathcal{K}_K := \left\{\mu\in[0,1]^N : \sum_{\ell,j}\mu_j^\ell=K\right\} K K := { μ ∈ [ 0 , 1 ] N : ∑ ℓ , j μ j ℓ = K } . For μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , let D μ ℓ : = diag ( μ ℓ ) D_\mu^\ell:= \operatorname{diag}(\mu^\ell) D μ ℓ := diag ( μ ℓ ) .
For fixed α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] , the admissible BMuon directions are
A α ( μ ) : = { Δ : ∃ { P 1 ℓ , P 2 ℓ } ℓ = 1 L , Δ 1 ℓ = D μ α , ℓ P 1 ℓ , Δ 2 ℓ = P 2 ℓ D μ α , ℓ , ∥ P 1 ℓ ∥ o p ≤ 1 , ∥ P 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } . \mathcal{A}_\alpha(\mu):=\left\{\Delta:\exists\{P_1^\ell,P_2^\ell\}_{\ell=1}^{L},\ \Delta_1^\ell=D_\mu^{\alpha,\ell}P_1^\ell,\ \Delta_2^\ell=P_2^\ell D_\mu^{\alpha,\ell},\ \lVert P_1^\ell\rVert_{\mathrm{op}}\le1,\ \lVert P_2^\ell\rVert_{\mathrm{op}}\le1,\ \forall\ell\right\}. A α ( μ ) := { Δ : ∃ { P 1 ℓ , P 2 ℓ } ℓ = 1 L , Δ 1 ℓ = D μ α , ℓ P 1 ℓ , Δ 2 ℓ = P 2 ℓ D μ α , ℓ , ∥ P 1 ℓ ∥ op ≤ 1 , ∥ P 2 ℓ ∥ op ≤ 1 , ∀ ℓ } .
The BMuon oracle solves
max μ ∈ K K max Δ ∈ A α ( μ ) ⟨ M , Δ ⟩ , \max_{\mu\in\mathcal{K}_K}\ \max_{\Delta\in\mathcal{A}_\alpha(\mu)}\ \langle M,\Delta\rangle, μ ∈ K K max Δ ∈ A α ( μ ) max ⟨ M , Δ ⟩ ,
and its allocation objective is
Φ M ( μ ) = ∑ ℓ = 1 L [ ∥ D μ α , ℓ M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ α , ℓ ∥ ∗ ] . \Phi_M(\mu)=\sum_{\ell=1}^{L}\left[\left\|D_\mu^{\alpha,\ell}M_1^\ell\right\|_*+\left\|M_2^\ell D_\mu^{\alpha,\ell}\right\|_*\right]. Φ M ( μ ) = ℓ = 1 ∑ L [ D μ α , ℓ M 1 ℓ ∗ + M 2 ℓ D μ α , ℓ ∗ ] .
Derivation of the BMuon Oracle
We derive Eqs. (4) –(5) from Eq. (3) . Fix μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K . By definition of A α ( μ ) \mathcal{A}_\alpha(\mu) A α ( μ ) ,
max Δ ∈ A α ( μ ) ⟨ M , Δ ⟩ = max ∥ P 1 ℓ ∥ o p ≤ 1 ∥ P 2 ℓ ∥ o p ≤ 1 , ∀ ℓ ∑ ℓ = 1 L [ ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ + ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ ] = ∑ ℓ = 1 L [ max ∥ P 1 ℓ ∥ o p ≤ 1 ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ + max ∥ P 2 ℓ ∥ o p ≤ 1 ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ ] , \begin{aligned}
\max_{\Delta\in\mathcal{A}_\alpha(\mu)}\langle M,\Delta\rangle
&=\max_{\substack{\lVert P_1^\ell\rVert_{\mathrm{op}}\le1\\ \lVert P_2^\ell\rVert_{\mathrm{op}}\le1,\ \forall\ell}}\sum_{\ell=1}^{L}\left[\left\langle M_1^\ell,D_\mu^{\alpha,\ell}P_1^\ell\right\rangle+\left\langle M_2^\ell,P_2^\ell D_\mu^{\alpha,\ell}\right\rangle\right]\\
&=\sum_{\ell=1}^{L}\left[\max_{\lVert P_1^\ell\rVert_{\mathrm{op}}\le1}\left\langle M_1^\ell,D_\mu^{\alpha,\ell}P_1^\ell\right\rangle+\max_{\lVert P_2^\ell\rVert_{\mathrm{op}}\le1}\left\langle M_2^\ell,P_2^\ell D_\mu^{\alpha,\ell}\right\rangle\right],
\end{aligned} Δ ∈ A α ( μ ) max ⟨ M , Δ ⟩ = ∥ P 1 ℓ ∥ op ≤ 1 ∥ P 2 ℓ ∥ op ≤ 1 , ∀ ℓ max ℓ = 1 ∑ L [ ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ + ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ ] = ℓ = 1 ∑ L [ ∥ P 1 ℓ ∥ op ≤ 1 max ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ + ∥ P 2 ℓ ∥ op ≤ 1 max ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ ] ,
where the second equality follows because the objective is a sum over matrices and the constraints are independent across matrices.
Since D μ α , ℓ D_\mu^{\alpha,\ell} D μ α , ℓ is diagonal, ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ = ⟨ D μ α , ℓ M 1 ℓ , P 1 ℓ ⟩ \left\langle M_1^\ell,D_\mu^{\alpha,\ell}P_1^\ell\right\rangle=\left\langle D_\mu^{\alpha,\ell}M_1^\ell,P_1^\ell\right\rangle ⟨ M 1 ℓ , D μ α , ℓ P 1 ℓ ⟩ = ⟨ D μ α , ℓ M 1 ℓ , P 1 ℓ ⟩ and ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ = ⟨ M 2 ℓ D μ α , ℓ , P 2 ℓ ⟩ \left\langle M_2^\ell,P_2^\ell D_\mu^{\alpha,\ell}\right\rangle=\left\langle M_2^\ell D_\mu^{\alpha,\ell},P_2^\ell\right\rangle ⟨ M 2 ℓ , P 2 ℓ D μ α , ℓ ⟩ = ⟨ M 2 ℓ D μ α , ℓ , P 2 ℓ ⟩ . Spectral–nuclear norm duality therefore gives
max Δ ∈ A α ( μ ) ⟨ M , Δ ⟩ = ∑ ℓ = 1 L [ ∥ D μ α , ℓ M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ α , ℓ ∥ ∗ ] = Φ M ( μ ) . \max_{\Delta\in\mathcal{A}_\alpha(\mu)}\langle M,\Delta\rangle=\sum_{\ell=1}^{L}\left[\left\|D_\mu^{\alpha,\ell}M_1^\ell\right\|_*+\left\|M_2^\ell D_\mu^{\alpha,\ell}\right\|_*\right]=\Phi_M(\mu). Δ ∈ A α ( μ ) max ⟨ M , Δ ⟩ = ℓ = 1 ∑ L [ D μ α , ℓ M 1 ℓ ∗ + M 2 ℓ D μ α , ℓ ∗ ] = Φ M ( μ ) .
Maximizing this value over μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K yields Eq. (4) .
Moreover, Φ M \Phi_M Φ M is continuous on K K \mathcal{K}_K K K : for α > 0 \alpha> 0 α > 0 , the map μ ↦ D μ α \mu\mapsto D^\alpha_\mu μ ↦ D μ α is continuous on [ 0 , 1 ] N [0,1]^N [ 0 , 1 ] N , and the nuclear norm is continuous. Since K K \mathcal{K}_K K K is compact, a maximizer μ ⋆ \mu^\star μ ⋆ exists.
Fix such a maximizer. By spectral–nuclear duality, Polar ( A ) ∈ arg max ∥ P ∥ o p ≤ 1 ⟨ A , P ⟩ \operatorname{Polar}(A) \in\arg\max_{\lVert P\rVert_{\mathrm{op}}\le1}\langle A,P\rangle Polar ( A ) ∈ arg max ∥ P ∥ op ≤ 1 ⟨ A , P ⟩ for any matrix A A A . Therefore, for every ℓ \ell ℓ , we can choose
P 1 ℓ , ⋆ = Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , P 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) . P_1^{\ell,\star} = \operatorname{Polar}\left(D^\alpha_{\mu_{\ell,\star}}M_1^\ell\right), \qquad P_2^{\ell,\star} = \operatorname{Polar}\left(M_2^\ell D^\alpha_{\mu_{\ell,\star}}\right). P 1 ℓ , ⋆ = Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , P 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) .
Returning to the update variables through the parameterization Δ 1 ℓ = D μ ℓ α P 1 ℓ \Delta_1^\ell= D^\alpha_{\mu_\ell}P_1^\ell Δ 1 ℓ = D μ ℓ α P 1 ℓ and Δ 2 ℓ = P 2 ℓ D μ ℓ α \Delta_2^\ell= P_2^\ell D^\alpha_{\mu_\ell} Δ 2 ℓ = P 2 ℓ D μ ℓ α , substituting μ = μ ⋆ \mu= \mu^\star μ = μ ⋆ and the maximizing P 1 ℓ , ⋆ , P 2 ℓ , ⋆ P_1^{\ell,\star},P_2^{\ell,\star} P 1 ℓ , ⋆ , P 2 ℓ , ⋆ gives, for every ℓ \ell ℓ ,
Δ 1 ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , Δ 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α . \Delta_1^{\ell,\star} = D^\alpha_{\mu_{\ell,\star}}\operatorname{Polar}\left(D^\alpha_{\mu_{\ell,\star}}M_1^\ell\right), \qquad\Delta_2^{\ell,\star} = \operatorname{Polar}\left(M_2^\ell D^\alpha_{\mu_{\ell,\star}}\right)D^\alpha_{\mu_{\ell,\star}}. Δ 1 ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M 1 ℓ ) , Δ 2 ℓ , ⋆ = Polar ( M 2 ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α .
which is Eq. (5) . This Δ ⋆ \Delta^\star Δ ⋆ attains max Δ ∈ A α ( μ ⋆ ) ⟨ M , Δ ⟩ = Φ M ( μ ⋆ ) \max_{\Delta\in\mathcal{A}_\alpha(\mu^\star)}\langle M,\Delta\rangle= \Phi_M(\mu^\star) max Δ ∈ A α ( μ ⋆ ) ⟨ M , Δ ⟩ = Φ M ( μ ⋆ ) . Since μ ⋆ \mu^\star μ ⋆ maximizes Φ M \Phi_M Φ M over K K \mathcal{K}_K K K , the pair ( μ ⋆ , Δ ⋆ ) (\mu^\star,\Delta^\star) ( μ ⋆ , Δ ⋆ ) solves the joint oracle in Eq. (3) .
Extension to Gated MLPs
A SwiGLU block, omitting biases, has the form
x ↦ W d o w n ℓ ( SiLU ( W g a t e ℓ x ) ⊙ W u p ℓ x ) . x \mapsto W^\ell_{\mathrm{down}}\left(\operatorname{SiLU}(W^\ell_{\mathrm{gate}}x)\odot W^\ell_{\mathrm{up}}x\right). x ↦ W down ℓ ( SiLU ( W gate ℓ x ) ⊙ W up ℓ x ) .
where ⊙ \odot ⊙ denotes elementwise multiplication. Hidden channel j j j therefore couples row j j j of W g a t e ℓ W^\ell_{\mathrm{gate}} W gate ℓ , row j j j of W u p ℓ W^\ell_{\mathrm{up}} W up ℓ , and column j j j of W d o w n ℓ W^\ell_{\mathrm{down}} W down ℓ . BMuon assigns the same allocation μ j ℓ \mu^\ell_j μ j ℓ to these three components. Let M g a t e ℓ M^\ell_{\mathrm{gate}} M gate ℓ , M u p ℓ M^\ell_{\mathrm{up}} M up ℓ , and M d o w n ℓ M^\ell_{\mathrm{down}} M down ℓ denote their momentum matrices. For a fixed allocation μ \mu μ , the admissible directions are parameterized as
Δ g a t e ℓ = D μ ℓ α P g a t e ℓ , Δ u p ℓ = D μ ℓ α P u p ℓ , Δ d o w n ℓ = P d o w n ℓ D μ ℓ α , \Delta^\ell_{\mathrm{gate}} = D^\alpha_{\mu_\ell}P^\ell_{\mathrm{gate}}, \qquad\Delta^\ell_{\mathrm{up}} = D^\alpha_{\mu_\ell}P^\ell_{\mathrm{up}}, \qquad\Delta^\ell_{\mathrm{down}} = P^\ell_{\mathrm{down}}D^\alpha_{\mu_\ell}, Δ gate ℓ = D μ ℓ α P gate ℓ , Δ up ℓ = D μ ℓ α P up ℓ , Δ down ℓ = P down ℓ D μ ℓ α ,
with ∥ P g a t e ℓ ∥ o p , ∥ P u p ℓ ∥ o p , ∥ P d o w n ℓ ∥ o p ≤ 1 \lVert P^\ell_{\mathrm{gate}}\rVert_{\mathrm{op}},\lVert P^\ell_{\mathrm{up}}\rVert_{\mathrm{op}},\lVert P^\ell_{\mathrm{down}}\rVert_{\mathrm{op}}\le1 ∥ P gate ℓ ∥ op , ∥ P up ℓ ∥ op , ∥ P down ℓ ∥ op ≤ 1 .
The same spectral–nuclear duality argument then gives the block contribution
∥ D μ ℓ α M g a t e ℓ ∥ ∗ + ∥ D μ ℓ α M u p ℓ ∥ ∗ + ∥ M d o w n ℓ D μ ℓ α ∥ ∗ . \lVert D^\alpha_{\mu_\ell}M^\ell_{\mathrm{gate}}\rVert_*+\lVert D^\alpha_{\mu_\ell}M^\ell_{\mathrm{up}}\rVert_*+\lVert M^\ell_{\mathrm{down}}D^\alpha_{\mu_\ell}\rVert_*. ∥ D μ ℓ α M gate ℓ ∥ ∗ + ∥ D μ ℓ α M up ℓ ∥ ∗ + ∥ M down ℓ D μ ℓ α ∥ ∗ .
Summing these terms over blocks gives the allocation objective under the same global constraint μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K . For an optimal allocation μ ⋆ \mu^\star μ ⋆ , an optimal direction is
Δ g a t e ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M g a t e ℓ ) , \Delta^{\ell,\star}_{\mathrm{gate}} = D^\alpha_{\mu_{\ell,\star}}\operatorname{Polar}\left(D^\alpha_{\mu_{\ell,\star}}M^\ell_{\mathrm{gate}}\right), Δ gate ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M gate ℓ ) ,
Δ u p ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M u p ℓ ) , \Delta^{\ell,\star}_{\mathrm{up}} = D^\alpha_{\mu_{\ell,\star}}\operatorname{Polar}\left(D^\alpha_{\mu_{\ell,\star}}M^\ell_{\mathrm{up}}\right), Δ up ℓ , ⋆ = D μ ℓ , ⋆ α Polar ( D μ ℓ , ⋆ α M up ℓ ) ,
Δ d o w n ℓ , ⋆ = Polar ( M d o w n ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α . \Delta^{\ell,\star}_{\mathrm{down}} = \operatorname{Polar}\left(M^\ell_{\mathrm{down}}D^\alpha_{\mu_{\ell,\star}}\right)D^\alpha_{\mu_{\ell,\star}}. Δ down ℓ , ⋆ = Polar ( M down ℓ D μ ℓ , ⋆ α ) D μ ℓ , ⋆ α .
Thus the only change from the two-matrix case is that each hidden-channel allocation is shared by three coupled matrix components.
Allocation Curvature
We prove Theorem 1 by treating the three regimes separately.
Concavity for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 . It suffices to consider one left-scaled term F ( μ ) : = ∥ D μ α M ∥ ∗ F(\mu) := \lVert D_\mu^\alpha M\rVert_\ast F ( μ ) := ∥ D μ α M ∥ ∗ . Using ∥ A ∥ ∗ = tr ( ( A ⊤ A ) 1 / 2 ) \lVert A\rVert_\ast= \operatorname{tr}((A^\top A)^{1/2}) ∥ A ∥ ∗ = tr (( A ⊤ A ) 1/2 ) ,
F ( μ ) = tr [ ( M ⊤ D μ 2 α M ) 1 / 2 ] . F(\mu) = \operatorname{tr}\left[\left(M^\top D_\mu^{2\alpha} M\right)^{1/2}\right]. F ( μ ) = tr [ ( M ⊤ D μ 2 α M ) 1/2 ] .
Define
G ( x ) : = tr [ ( M ⊤ D x M ) 1 / 2 ] , x ∈ R + m . G(x) := \operatorname{tr}\left[\left(M^\top D_x M\right)^{1/2}\right], \qquad x \in\mathbb{R}_+^m. G ( x ) := tr [ ( M ⊤ D x M ) 1/2 ] , x ∈ R + m .
Then F ( μ ) = G ( μ 2 α ) F(\mu) = G(\mu^{2\alpha}) F ( μ ) = G ( μ 2 α ) , where the power is applied coordinatewise. The matrix square-root map is operator concave and operator monotone on the positive semidefinite cone [6 ] . Since x ↦ M ⊤ D x M x \mapsto M^\top D_xM x ↦ M ⊤ D x M is linear, G G G is concave. Moreover, G G G is coordinatewise nondecreasing. If x ≤ y x \le y x ≤ y , then D x ⪯ D y D_x \preceq D_y D x ⪯ D y , hence M ⊤ D x M ⪯ M ⊤ D y M M^\top D_xM \preceq M^\top D_yM M ⊤ D x M ⪯ M ⊤ D y M . Operator monotonicity of the square root and monotonicity of the trace therefore give G ( x ) ≤ G ( y ) G(x) \le G(y) G ( x ) ≤ G ( y ) .
For any μ , ν ∈ R + m \mu,\nu\in\mathbb{R}_+^m μ , ν ∈ R + m and θ ∈ [ 0 , 1 ] \theta\in[0,1] θ ∈ [ 0 , 1 ] , since 0 < 2 α ≤ 1 0 < 2\alpha\le1 0 < 2 α ≤ 1 , concavity of t ↦ t 2 α t \mapsto t^{2\alpha} t ↦ t 2 α gives
( θ μ + ( 1 − θ ) ν ) 2 α ≥ θ μ 2 α + ( 1 − θ ) ν 2 α (\theta\mu+ (1-\theta)\nu)^{2\alpha} \ge\theta\mu^{2\alpha} + (1-\theta)\nu^{2\alpha} ( θ μ + ( 1 − θ ) ν ) 2 α ≥ θ μ 2 α + ( 1 − θ ) ν 2 α
coordinatewise. Hence, using first the monotonicity and then the concavity of G G G ,
F ( θ μ + ( 1 − θ ) ν ) = G ( ( θ μ + ( 1 − θ ) ν ) 2 α ) ≥ G ( θ μ 2 α + ( 1 − θ ) ν 2 α ) ≥ θ G ( μ 2 α ) + ( 1 − θ ) G ( ν 2 α ) = θ F ( μ ) + ( 1 − θ ) F ( ν ) . \begin{aligned}
F(\theta\mu+ (1-\theta)\nu) &= G((\theta\mu+ (1-\theta)\nu)^{2\alpha}) \\
&\ge G(\theta\mu^{2\alpha} + (1-\theta)\nu^{2\alpha}) \\
&\ge\theta G(\mu^{2\alpha}) + (1-\theta)G(\nu^{2\alpha}) \\
&= \theta F(\mu) + (1-\theta)F(\nu).
\end{aligned} F ( θ μ + ( 1 − θ ) ν ) = G (( θ μ + ( 1 − θ ) ν ) 2 α ) ≥ G ( θ μ 2 α + ( 1 − θ ) ν 2 α ) ≥ θ G ( μ 2 α ) + ( 1 − θ ) G ( ν 2 α ) = θ F ( μ ) + ( 1 − θ ) F ( ν ) .
Thus μ ↦ ∥ D μ α M ∥ ∗ \mu\mapsto\lVert D_\mu^\alpha M\rVert_\ast μ ↦ ∥ D μ α M ∥ ∗ is concave for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 . Since ∥ M D μ α ∥ ∗ = ∥ D μ α M ⊤ ∥ ∗ \lVert MD_\mu^\alpha\rVert_\ast= \lVert D_\mu^\alpha M^\top\rVert_\ast ∥ M D μ α ∥ ∗ = ∥ D μ α M ⊤ ∥ ∗ , the same argument applies to right-scaled terms. Summing over all blocks and both terms in Eq. (4) proves that Φ M \Phi_M Φ M is concave in μ \mu μ .
Convexity at α = 1 \alpha= 1 α = 1 . At α = 1 \alpha= 1 α = 1 , both maps μ ↦ D μ M \mu\mapsto D_\mu M μ ↦ D μ M and μ ↦ M D μ \mu\mapsto MD_\mu μ ↦ M D μ are linear. Since the nuclear norm is convex, μ ↦ ∥ D μ M ∥ ∗ \mu\mapsto\lVert D_\mu M\rVert_\ast μ ↦ ∥ D μ M ∥ ∗ and μ ↦ ∥ M D μ ∥ ∗ \mu\mapsto\lVert MD_\mu\rVert_\ast μ ↦ ∥ M D μ ∥ ∗ are convex. Summing these terms over all blocks therefore shows that Φ M \Phi_M Φ M is convex.
Intermediate regime 1 2 < α < 1 \frac{1}{2} < \alpha< 1 2 1 < α < 1 . Take the special case L = 1 L = 1 L = 1 , N = 2 N = 2 N = 2 , K = 1 K = 1 K = 1 , and M 2 = 0 M_2 = 0 M 2 = 0 .
For M 1 = I 2 M_1 = I_2 M 1 = I 2 , Φ M ( μ ) = μ 1 α + μ 2 α \Phi_M(\mu) = \mu_1^\alpha+ \mu_2^\alpha Φ M ( μ ) = μ 1 α + μ 2 α . Hence, for every α < 1 \alpha< 1 α < 1 ,
Φ M ( 1 2 , 1 2 ) = 2 1 − α > Φ M ( 1 , 0 ) + Φ M ( 0 , 1 ) 2 = 1 , \Phi_M\left(\frac{1}{2},\frac{1}{2}\right) = 2^{1-\alpha} > \frac{\Phi_M(1,0) + \Phi_M(0,1)}{2} = 1, Φ M ( 2 1 , 2 1 ) = 2 1 − α > 2 Φ M ( 1 , 0 ) + Φ M ( 0 , 1 ) = 1 ,
so Φ M \Phi_M Φ M is not convex.
Now let M 1 = ( 1 0 1 0 ) M_1 = \begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix} M 1 = ( 1 1 0 0 ) . Then Φ M ( μ ) = ( μ 1 2 α + μ 2 2 α ) 1 / 2 \Phi_M(\mu) = (\mu_1^{2\alpha} + \mu_2^{2\alpha})^{1/2} Φ M ( μ ) = ( μ 1 2 α + μ 2 2 α ) 1/2 , and for every α > 1 2 \alpha> \frac{1}{2} α > 2 1 ,
Φ M ( 1 2 , 1 2 ) = 2 1 / 2 − α < Φ M ( 1 , 0 ) + Φ M ( 0 , 1 ) 2 = 1 , \Phi_M\left(\frac{1}{2},\frac{1}{2}\right) = 2^{1/2-\alpha} < \frac{\Phi_M(1,0) + \Phi_M(0,1)}{2} = 1, Φ M ( 2 1 , 2 1 ) = 2 1/2 − α < 2 Φ M ( 1 , 0 ) + Φ M ( 0 , 1 ) = 1 ,
so Φ M \Phi_M Φ M is not concave.
Thus, for 1 2 < α < 1 \frac{1}{2} < \alpha< 1 2 1 < α < 1 , Φ M \Phi_M Φ M is in general neither convex nor concave. These examples show that the thresholds in Theorem 1 are sharp: for $0 < \alpha \le 1$, Φ M \Phi_M Φ M is concave for all momenta if and only if α ≤ 1 2 \alpha\le\frac{1}{2} α ≤ 2 1 , and convex for all momenta if and only if $\alpha = 1`.
Extreme-point structure at α = 1 \alpha= 1 α = 1 . Assume K K K integer. The extreme points of K K \mathcal{K}_{K} K K are exactly the binary vectors with K K K ones. Indeed, every such vector is extreme because all its coordinates lie at the endpoints of [ 0 , 1 ] [0,1] [ 0 , 1 ] . Conversely, any non-binary μ ∈ K K \mu\in\mathcal{K}_{K} μ ∈ K K has at least two fractional coordinates: it cannot have exactly one since ∑ i μ i = K \sum_i \mu_i = K ∑ i μ i = K is an integer. For two fractional coordinates i ≠ j i \ne j i = j , sufficiently small ε > 0 \varepsilon> 0 ε > 0 gives μ ± = μ ± ε ( e i − e j ) ∈ K K \mu^{\pm} = \mu\pm\varepsilon(e_i-e_j) \in\mathcal{K}_{K} μ ± = μ ± ε ( e i − e j ) ∈ K K with μ = 1 2 ( μ + + μ − ) \mu= \frac{1}{2}(\mu^{+}+\mu^{-}) μ = 2 1 ( μ + + μ − ) , so μ \mu μ is not extreme.
Since Φ M \Phi_M Φ M is continuous and convex at α = 1 \alpha= 1 α = 1 , a maximizer over the polytope K K \mathcal{K}_{K} K K can be chosen at an extreme point. Thus an optimal allocation can be chosen binary, selecting exactly K K K channels. Writing S ℓ : = { j : ( ℓ , j ) ∈ S } S_\ell:= \{j : (\ell,j) \in S\} S ℓ := { j : ( ℓ , j ) ∈ S } and D S ℓ : = diag ( 1 S ℓ ) D_{S_\ell} := \operatorname{diag}(\mathbf{1}_{S_\ell}) D S ℓ := diag ( 1 S ℓ ) , the allocation problem becomes
max μ ∈ K K Φ M ( μ ) = max S ⊆ { ( ℓ , j ) : 1 ≤ ℓ ≤ L , 1 ≤ j ≤ m ℓ } ∣ S ∣ = K ∑ ℓ = 1 L [ ∥ D S ℓ M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D S ℓ ∥ ∗ ] . \max_{\mu\in\mathcal{K}_{K}} \Phi_M(\mu) =
\max_{\substack{S \subseteq\{(\ell,j):1 \le\ell\le L,\ 1 \le j \le m_\ell\}\\ |S|=K}}
\sum_{\ell=1}^{L}\left[\left\|D_{S_\ell}M^\ell_1\right\|_*+\left\|M^\ell_2D_{S_\ell}\right\|_*\right]. μ ∈ K K max Φ M ( μ ) = S ⊆ {( ℓ , j ) : 1 ≤ ℓ ≤ L , 1 ≤ j ≤ m ℓ } ∣ S ∣ = K max ℓ = 1 ∑ L [ D S ℓ M 1 ℓ ∗ + M 2 ℓ D S ℓ ∗ ] .
Hence, at α = 1 \alpha= 1 α = 1 , the continuous allocation problem admits an equivalent combinatorial formulation over subsets of K K K global channels.
Norm Geometry of BMuon
Using K K \mathcal{K}_{K} K K and A α ( μ ) \mathcal{A}_{\alpha}(\mu) A α ( μ ) as defined in the notation and setup at the beginning of Appendix B, define
A α , K : = ⋃ μ ∈ K K A α ( μ ) , B α , K : = conv ( A α , K ) . \mathcal{A}_{\alpha,K}:=\bigcup_{\mu\in\mathcal{K}_{K}}\mathcal{A}_{\alpha}(\mu),\qquad\mathcal{B}_{\alpha,K}:=\operatorname{conv}(\mathcal{A}_{\alpha,K}). A α , K := μ ∈ K K ⋃ A α ( μ ) , B α , K := conv ( A α , K ) .
Recall Muon’s product norm and unit ball:
∥ Δ ∥ M u o n : = max ℓ max { ∥ Δ 1 ℓ ∥ o p , ∥ Δ 2 ℓ ∥ o p } , A M u o n : = { Δ : ∥ Δ ∥ M u o n ≤ 1 } . \|\Delta\|_{\mathrm{Muon}}:=\max_{\ell}\max\left\{\|\Delta^\ell_1\|_{\mathrm{op}},\|\Delta^\ell_2\|_{\mathrm{op}}\right\},\qquad\mathcal{A}_{\mathrm{Muon}}:=\{\Delta:\|\Delta\|_{\mathrm{Muon}}\le1\}. ∥Δ ∥ Muon := ℓ max max { ∥ Δ 1 ℓ ∥ op , ∥ Δ 2 ℓ ∥ op } , A Muon := { Δ : ∥Δ ∥ Muon ≤ 1 } .
Lemma 3 (BMuon norm and comparison with Muon). ** Let 0 < α ≤ 1 0<\alpha\le1 0 < α ≤ 1 and 0 < K ≤ N 0<K\le N 0 < K ≤ N . Then B α , K \mathcal{B}_{\alpha,K} B α , K is compact, convex, centrally symmetric, and has the origin in its interior. Consequently, its Minkowski functional Ω α , K \Omega_{\alpha,K} Ω α , K is a norm with unit ball B α , K \mathcal{B}_{\alpha,K} B α , K . Moreover, for every Δ \Delta Δ ,
∥ Δ ∥ M u o n ≤ Ω α , K ( Δ ) ≤ ( N K ) α ∥ Δ ∥ M u o n . \|\Delta\|_{\mathrm{Muon}}\le\Omega_{\alpha,K}(\Delta)\le\left(\frac{N}{K}\right)^\alpha\|\Delta\|_{\mathrm{Muon}}. ∥Δ ∥ Muon ≤ Ω α , K ( Δ ) ≤ ( K N ) α ∥Δ ∥ Muon .
Proof. Fix μ ∈ K K \mu\in\mathcal{K}_{K} μ ∈ K K and Δ ∈ A α ( μ ) \Delta\in\mathcal{A}_{\alpha}(\mu) Δ ∈ A α ( μ ) . By definition, for every ℓ \ell ℓ , there exist P 1 ℓ , P 2 ℓ P^\ell_1,P^\ell_2 P 1 ℓ , P 2 ℓ with ∥ P 1 ℓ ∥ o p , ∥ P 2 ℓ ∥ o p ≤ 1 \|P^\ell_1\|_{\mathrm{op}},\|P^\ell_2\|_{\mathrm{op}}\le1 ∥ P 1 ℓ ∥ op , ∥ P 2 ℓ ∥ op ≤ 1 such that Δ 1 ℓ = D μ ℓ α P 1 ℓ \Delta^\ell_1=D^\alpha_{\mu^\ell}P^\ell_1 Δ 1 ℓ = D μ ℓ α P 1 ℓ and Δ 2 ℓ = P 2 ℓ D μ ℓ α \Delta^\ell_2=P^\ell_2D^\alpha_{\mu^\ell} Δ 2 ℓ = P 2 ℓ D μ ℓ α . Since 0 ≤ μ j ℓ ≤ 1 0\le\mu^\ell_j\le1 0 ≤ μ j ℓ ≤ 1 , ∥ D μ ℓ α ∥ o p ≤ 1 \|D^\alpha_{\mu^\ell}\|_{\mathrm{op}}\le1 ∥ D μ ℓ α ∥ op ≤ 1 . Therefore, by submultiplicativity of the operator norm, ∥ Δ 1 ℓ ∥ o p ≤ 1 \|\Delta^\ell_1\|_{\mathrm{op}}\le1 ∥ Δ 1 ℓ ∥ op ≤ 1 , and ∥ Δ 2 ℓ ∥ o p ≤ 1 \|\Delta^\ell_2\|_{\mathrm{op}}\le1 ∥ Δ 2 ℓ ∥ op ≤ 1 for every ℓ \ell ℓ . Hence A α ( μ ) ⊆ A M u o n \mathcal{A}_{\alpha}(\mu)\subseteq\mathcal{A}_{\mathrm{Muon}} A α ( μ ) ⊆ A Muon for every μ ∈ K K \mu\in\mathcal{K}_{K} μ ∈ K K , so A α , K ⊆ A M u o n \mathcal{A}_{\alpha,K}\subseteq\mathcal{A}_{\mathrm{Muon}} A α , K ⊆ A Muon . Since the Muon unit ball is convex, B α , K ⊆ A M u o n \mathcal{B}_{\alpha,K}\subseteq\mathcal{A}_{\mathrm{Muon}} B α , K ⊆ A Muon .
Conversely, consider the uniform allocation μ ˉ j ℓ = K / N \bar{\mu}^{\ell}_j=K/N μ ˉ j ℓ = K / N , which belongs to K K \mathcal{K}_{K} K K . Then D μ ˉ ℓ α = ( K / N ) α I D^\alpha_{\bar{\mu}^\ell}=(K/N)^\alpha I D μ ˉ ℓ α = ( K / N ) α I , and therefore A α ( μ ˉ ) = ( K / N ) α A M u o n \mathcal{A}_{\alpha}(\bar{\mu})=(K/N)^\alpha\mathcal{A}_{\mathrm{Muon}} A α ( μ ˉ ) = ( K / N ) α A Muon . Since A α ( μ ˉ ) ⊆ B α , K \mathcal{A}_{\alpha}(\bar{\mu})\subseteq\mathcal{B}_{\alpha,K} A α ( μ ˉ ) ⊆ B α , K ,
( K N ) α A M u o n ⊆ B α , K ⊆ A M u o n . (9) \left(\frac{K}{N}\right)^\alpha\mathcal{A}_{\mathrm{Muon}}\subseteq\mathcal{B}_{\alpha,K}\subseteq\mathcal{A}_{\mathrm{Muon}}. \tag*{(9)} ( N K ) α A Muon ⊆ B α , K ⊆ A Muon . (9)
It remains to verify the geometric properties of B α , K \mathcal{B}_{\alpha,K} B α , K . For compactness, let
X : = K K × ∏ ℓ = 1 L ( { P 1 ℓ : ∥ P 1 ℓ ∥ o p ≤ 1 } × { P 2 ℓ : ∥ P 2 ℓ ∥ o p ≤ 1 } ) , \mathcal{X} := \mathcal{K}_{K} \times\prod_{\ell=1}^{L} \left(\left\{P_{1}^{\ell} : \lVert P_{1}^{\ell}\rVert_{\mathrm{op}} \le1\right\} \times\left\{P_{2}^{\ell} : \lVert P_{2}^{\ell}\rVert_{\mathrm{op}} \le1\right\}\right), X := K K × ℓ = 1 ∏ L ( { P 1 ℓ : ∥ P 1 ℓ ∥ op ≤ 1 } × { P 2 ℓ : ∥ P 2 ℓ ∥ op ≤ 1 } ) ,
which is compact in finite dimension, and define F ( μ , { P 1 ℓ , P 2 ℓ } ) : = ( ( D μ ℓ α P 1 ℓ , P 2 ℓ D μ ℓ α ) ) ℓ = 1 L F(\mu,\{P_{1}^{\ell},P_{2}^{\ell}\}) := ((D_{\mu_{\ell}}^{\alpha}P_{1}^{\ell},P_{2}^{\ell}D_{\mu_{\ell}}^{\alpha}))_{\ell=1}^{L} F ( μ , { P 1 ℓ , P 2 ℓ }) := (( D μ ℓ α P 1 ℓ , P 2 ℓ D μ ℓ α ) ) ℓ = 1 L . The map F F F is continuous for α > 0 \alpha> 0 α > 0 , and A α , K = F ( X ) \mathcal{A}_{\alpha,K} = F(\mathcal{X}) A α , K = F ( X ) ; hence A α , K \mathcal{A}_{\alpha,K} A α , K is compact. Its convex hull B α , K \mathcal{B}_{\alpha,K} B α , K is therefore compact in finite dimension and is convex by definition. Moreover, A α , K \mathcal{A}_{\alpha,K} A α , K is centrally symmetric, since replacing every P i ℓ P_{i}^{\ell} P i ℓ by − P i ℓ -P_{i}^{\ell} − P i ℓ replaces Δ \Delta Δ by − Δ -\Delta − Δ ; the convex hull of a centrally symmetric set is centrally symmetric. Finally, the left inclusion in Eq. (9) shows that B α , K \mathcal{B}_{\alpha,K} B α , K contains a neighborhood of the origin.
Thus the Minkowski functional of B α , K \mathcal{B}_{\alpha,K} B α , K is a norm whose unit ball is B α , K \mathcal{B}_{\alpha,K} B α , K . Since a larger set has a smaller Minkowski functional, the two inclusions in Eq. (9) give ∥ Δ ∥ M u o n ≤ Ω α , K ( Δ ) ≤ ( N / K ) α ∥ Δ ∥ M u o n \lVert\Delta\rVert_{\mathrm{Muon}} \le\Omega_{\alpha,K}(\Delta) \le(N/K)^{\alpha}\lVert\Delta\rVert_{\mathrm{Muon}} ∥ Δ ∥ Muon ≤ Ω α , K ( Δ ) ≤ ( N / K ) α ∥ Δ ∥ Muon .
The following Lemma gives a characterization of the norm Ω α , K \Omega_{\alpha,K} Ω α , K in the regime 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 .
Lemma 4 (Characterization of the BMuon norm). Let 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 and 0 < K ≤ N 0 < K \le N 0 < K ≤ N . Then A α , K \mathcal{A}_{\alpha,K} A α , K is convex, and hence B α , K = A α , K \mathcal{B}_{\alpha,K} = \mathcal{A}_{\alpha,K} B α , K = A α , K . For μ ∈ K K \mu\in\mathcal{K}_{K} μ ∈ K K , define
ρ μ ( Δ ) : = max ℓ max { ∥ D μ ℓ − α Δ 1 ℓ ∥ o p , ∥ Δ 2 ℓ D μ ℓ − α ∥ o p } , \rho_{\mu}(\Delta) := \max_{\ell}\max\left\{\left\lVert D_{\mu_{\ell}}^{-\alpha}\Delta_{1}^{\ell}\right\rVert_{\mathrm{op}},\left\lVert\Delta_{2}^{\ell}D_{\mu_{\ell}}^{-\alpha}\right\rVert_{\mathrm{op}}\right\}, ρ μ ( Δ ) := ℓ max max { D μ ℓ − α Δ 1 ℓ op , Δ 2 ℓ D μ ℓ − α op } , with the following convention: if μ j ℓ = 0 \mu_{j}^{\ell} = 0 μ j ℓ = 0 , finiteness requires Δ 1 ; j ℓ = 0 \Delta_{1;j}^{\ell} = 0 Δ 1 ; j ℓ = 0 and Δ 2 , : , j ℓ = 0 \Delta_{2,:,j}^{\ell} = 0 Δ 2 , : , j ℓ = 0 ; the corresponding zero rows and columns are then omitted when evaluating the two operator norms. If either is nonzero, set ρ μ ( Δ ) = + ∞ \rho_{\mu}(\Delta) = +\infty ρ μ ( Δ ) = + ∞ . Then
Ω α , K ( Δ ) = min μ ∈ K K ρ μ ( Δ ) . \Omega_{\alpha,K}(\Delta) = \min_{\mu\in\mathcal{K}_{K}} \rho_{\mu}(\Delta). Ω α , K ( Δ ) = μ ∈ K K min ρ μ ( Δ ) .
Proof. Set β : = 2 α ∈ ( 0 , 1 ] \beta:= 2\alpha\in(0,1] β := 2 α ∈ ( 0 , 1 ] , and define
φ β ( z , s ) : = { z 2 s − β , s > 0 , 0 , ( z , s ) = ( 0 , 0 ) , + ∞ , s = 0 , z ≠ 0. \varphi_{\beta}(z,s) :=
\begin{cases}
z^{2}s^{-\beta}, & s>0,\\
0, & (z,s)=(0,0),\\
+\infty, & s=0,\ z\ne0.
\end{cases} φ β ( z , s ) := ⎩ ⎨ ⎧ z 2 s − β , 0 , + ∞ , s > 0 , ( z , s ) = ( 0 , 0 ) , s = 0 , z = 0.
For s > 0 s > 0 s > 0 , the Hessian of φ β \varphi_{\beta} φ β is positive semidefinite: ∂ z z 2 φ β = 2 s − β > 0 \partial_{zz}^{2}\varphi_{\beta} = 2s^{-\beta} > 0 ∂ z z 2 φ β = 2 s − β > 0 and
det ∇ 2 φ β ( z , s ) = 2 β ( 1 − β ) z 2 s − 2 β − 2 ≥ 0 , \det\nabla^{2}\varphi_{\beta}(z,s) = 2\beta(1-\beta)z^{2}s^{-2\beta-2} \ge0, det ∇ 2 φ β ( z , s ) = 2 β ( 1 − β ) z 2 s − 2 β − 2 ≥ 0 ,
since β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] . Hence φ β \varphi_{\beta} φ β is jointly convex on R × ( 0 , ∞ ) \mathbb{R} \times(0,\infty) R × ( 0 , ∞ ) .
It remains to check the boundary s = 0 s = 0 s = 0 . The only boundary point at which φ β \varphi_{\beta} φ β is finite is ( 0 , 0 ) (0,0) ( 0 , 0 ) . For s > 0 s > 0 s > 0 and θ ∈ [ 0 , 1 ] \theta\in[0,1] θ ∈ [ 0 , 1 ] ,
φ β ( θ z , θ s ) = θ 2 − β φ β ( z , s ) ≤ θ φ β ( z , s ) , \varphi_{\beta}(\theta z,\theta s) = \theta^{2-\beta}\varphi_{\beta}(z,s) \le\theta\varphi_{\beta}(z,s), φ β ( θ z , θ s ) = θ 2 − β φ β ( z , s ) ≤ θ φ β ( z , s ) ,
because 2 − β ≥ 1 2-\beta\ge1 2 − β ≥ 1 , which is exactly the convexity inequality between ( z , s ) (z,s) ( z , s ) and ( 0 , 0 ) (0,0) ( 0 , 0 ) . If a boundary point has s = 0 s = 0 s = 0 and z ≠ 0 z \ne0 z = 0 , its value is + ∞ +\infty + ∞ , so the convexity inequality is automatic. Thus φ β \varphi_{\beta} φ β is convex on R × [ 0 , ∞ ) \mathbb{R} \times[0,\infty) R × [ 0 , ∞ ) . It is also lower semicontinuous. It is continuous for s > 0 s > 0 s > 0 . At ( 0 , 0 ) (0,0) ( 0 , 0 ) , nonnegativity gives
lim inf ( z , s ) → ( 0 , 0 ) φ β ( z , s ) ≥ 0 = φ β ( 0 , 0 ) . \liminf_{(z,s)\to(0,0)} \varphi_{\beta}(z,s) \ge0 = \varphi_{\beta}(0,0). ( z , s ) → ( 0 , 0 ) lim inf φ β ( z , s ) ≥ 0 = φ β ( 0 , 0 ) .
If z 0 ≠ 0 z_0 \ne0 z 0 = 0 , then as ( z , s ) → ( z 0 , 0 ) (z,s) \to(z_0,0) ( z , s ) → ( z 0 , 0 ) , we eventually have ∣ z ∣ ≥ ∣ z 0 ∣ / 2 |z| \ge|z_0|/2 ∣ z ∣ ≥ ∣ z 0 ∣/2 . If s = 0 s=0 s = 0 , then φ β ( z , s ) = + ∞ \varphi_{\beta}(z,s)=+\infty φ β ( z , s ) = + ∞ ; if s > 0 s>0 s > 0 , then
φ β ( z , s ) = z 2 s − β ≥ z 0 2 4 s − β ⟶ + ∞ . \varphi_{\beta}(z,s)=z^2s^{-\beta}\ge\frac{z_0^2}{4}s^{-\beta}\longrightarrow+\infty. φ β ( z , s ) = z 2 s − β ≥ 4 z 0 2 s − β ⟶ + ∞.
Hence
lim inf ( z , s ) → ( z 0 , 0 ) φ β ( z , s ) = + ∞ = φ β ( z 0 , 0 ) . \liminf_{(z,s)\to(z_0,0)}\varphi_{\beta}(z,s)=+\infty=\varphi_{\beta}(z_0,0). ( z , s ) → ( z 0 , 0 ) lim inf φ β ( z , s ) = + ∞ = φ β ( z 0 , 0 ) .
Therefore φ β \varphi_{\beta} φ β is lower semicontinuous on R × [ 0 , ∞ ) \mathbb{R}\times[0,\infty) R × [ 0 , ∞ ) .
For every block ℓ \ell ℓ ,
∥ D μ ℓ − α Δ 1 ℓ ∥ o p 2 = sup ∥ x ∥ 2 = 1 ∥ D μ ℓ − α Δ 1 ℓ x ∥ 2 2 = sup ∥ x ∥ 2 = 1 ∑ j = 1 m ℓ ( μ j ℓ ) − 2 α ⟨ Δ 1 , j : ℓ , x ⟩ 2 = sup ∥ x ∥ 2 = 1 ∑ j = 1 m ℓ φ β ( ⟨ Δ 1 , j : ℓ , x ⟩ , μ j ℓ ) . \begin{aligned}
\left\|D_{\mu^\ell}^{-\alpha}\Delta_1^\ell\right\|_{\mathrm{op}}^2
=\sup_{\|x\|_2=1}\left\|D_{\mu^\ell}^{-\alpha}\Delta_1^\ell x\right\|_2^2 \\
&=\sup_{\|x\|_2=1}\sum_{j=1}^{m_\ell}(\mu_j^\ell)^{-2\alpha}\langle\Delta_{1,j:}^\ell,x\rangle^2 \\
&=\sup_{\|x\|_2=1}\sum_{j=1}^{m_\ell}\varphi_{\beta}\left(\langle\Delta_{1,j:}^\ell,x\rangle,\mu_j^\ell\right).
\end{aligned} D μ ℓ − α Δ 1 ℓ op 2 = ∥ x ∥ 2 = 1 sup D μ ℓ − α Δ 1 ℓ x 2 2 = ∥ x ∥ 2 = 1 sup j = 1 ∑ m ℓ ( μ j ℓ ) − 2 α ⟨ Δ 1 , j : ℓ , x ⟩ 2 = ∥ x ∥ 2 = 1 sup j = 1 ∑ m ℓ φ β ( ⟨ Δ 1 , j : ℓ , x ⟩ , μ j ℓ ) .
This interpretation agrees with the zero-allocation convention in the definition of ρ μ \rho_\mu ρ μ . Indeed, if μ j ℓ = 0 \mu_j^\ell=0 μ j ℓ = 0 and Δ 1 , j : ℓ = 0 \Delta_{1,j:}^\ell=0 Δ 1 , j : ℓ = 0 , then φ β ( 0 , 0 ) = 0 \varphi_{\beta}(0,0)=0 φ β ( 0 , 0 ) = 0 , which amounts to omitting that row from the operator norm. If instead μ j ℓ = 0 \mu_j^\ell=0 μ j ℓ = 0 and Δ 1 , j : ℓ ≠ 0 \Delta_{1,j:}^\ell\ne0 Δ 1 , j : ℓ = 0 , there exists x x x such that ⟨ Δ 1 , j : ℓ , x ⟩ ≠ 0 \langle\Delta_{1,j:}^\ell,x\rangle\ne0 ⟨ Δ 1 , j : ℓ , x ⟩ = 0 , and hence the supremum is + ∞ +\infty + ∞ , matching the zero-allocation convention for ρ μ \rho_\mu ρ μ .
Similarly,
∥ Δ 2 ℓ D μ ℓ − α ∥ o p 2 = sup ∥ y ∥ 2 = 1 ∑ j = 1 m ℓ φ β ( ⟨ Δ 2 : , j ℓ , y ⟩ , μ j ℓ ) . \left\|\Delta_2^\ell D_{\mu^\ell}^{-\alpha}\right\|_{\mathrm{op}}^2
=\sup_{\|y\|_2=1}\sum_{j=1}^{m_\ell}\varphi_{\beta}\left(\langle\Delta_{2:,j}^\ell,y\rangle,\mu_j^\ell\right). Δ 2 ℓ D μ ℓ − α op 2 = ∥ y ∥ 2 = 1 sup j = 1 ∑ m ℓ φ β ( ⟨ Δ 2 : , j ℓ , y ⟩ , μ j ℓ ) .
For fixed ℓ \ell ℓ , j j j , and x x x , the map ( Δ , μ ) ↦ φ β ( ⟨ Δ 1 , j : ℓ , x ⟩ , μ j ℓ ) (\Delta,\mu)\mapsto\varphi_{\beta}(\langle\Delta_{1,j:}^\ell,x\rangle,\mu_j^\ell) ( Δ , μ ) ↦ φ β (⟨ Δ 1 , j : ℓ , x ⟩ , μ j ℓ ) is jointly convex and lower semicontinuous, since φ β \varphi_{\beta} φ β has these properties and its arguments depend linearly on ( Δ , μ ) (\Delta,\mu) ( Δ , μ ) . Finite sums preserve both properties, and so does taking the pointwise supremum over x x x . Hence ( Δ , μ ) ↦ ∥ D μ ℓ − α Δ 1 ℓ ∥ o p 2 (\Delta,\mu)\mapsto\|D_{\mu^\ell}^{-\alpha}\Delta_1^\ell\|_{\mathrm{op}}^2 ( Δ , μ ) ↦ ∥ D μ ℓ − α Δ 1 ℓ ∥ op 2 is jointly convex and lower semicontinuous. The same argument applies to ( Δ , μ ) ↦ ∥ Δ 2 ℓ D μ ℓ − α ∥ o p 2 (\Delta,\mu)\mapsto\|\Delta_2^\ell D_{\mu^\ell}^{-\alpha}\|_{\mathrm{op}}^2 ( Δ , μ ) ↦ ∥ Δ 2 ℓ D μ ℓ − α ∥ op 2 . Finally, g ( Δ , μ ) : = ρ μ ( Δ ) 2 g(\Delta,\mu):=\rho_\mu(\Delta)^2 g ( Δ , μ ) := ρ μ ( Δ ) 2 is the finite maximum of these functions over the blocks ℓ \ell ℓ , and is therefore jointly convex and lower semicontinuous.
We now show that
A α ( μ ) = { Δ : ρ μ ( Δ ) ≤ 1 } . (10) \mathcal{A}_{\alpha}(\mu)=\{\Delta:\rho_\mu(\Delta)\le1\}. \tag*{(10)} A α ( μ ) = { Δ : ρ μ ( Δ ) ≤ 1 } . (10)
If Δ ∈ A α ( μ ) \Delta\in\mathcal{A}_{\alpha}(\mu) Δ ∈ A α ( μ ) , then there exist { P 1 ℓ , P 2 ℓ } ℓ = 1 L \{P_1^\ell,P_2^\ell\}_{\ell=1}^L { P 1 ℓ , P 2 ℓ } ℓ = 1 L such that, for every ℓ \ell ℓ , Δ 1 ℓ = D μ ℓ α P 1 ℓ \Delta_1^\ell=D_{\mu^\ell}^{\alpha}P_1^\ell Δ 1 ℓ = D μ ℓ α P 1 ℓ and Δ 2 ℓ = P 2 ℓ D μ ℓ α \Delta_2^\ell=P_2^\ell D_{\mu^\ell}^{\alpha} Δ 2 ℓ = P 2 ℓ D μ ℓ α , with ∥ P 1 ℓ ∥ o p , ∥ P 2 ℓ ∥ o p ≤ 1 \|P_1^\ell\|_{\mathrm{op}},\|P_2^\ell\|_{\mathrm{op}}\le1 ∥ P 1 ℓ ∥ op , ∥ P 2 ℓ ∥ op ≤ 1 . Let S ℓ : = { j : μ j ℓ > 0 } S_\ell:=\{j:\mu_j^\ell>0\} S ℓ := { j : μ j ℓ > 0 } . If j ∉ S ℓ j\notin S_\ell j ∈ / S ℓ , then Δ 1 , j : ℓ = Δ 2 : , j ℓ = 0 \Delta_{1,j:}^\ell=\Delta_{2:,j}^\ell=0 Δ 1 , j : ℓ = Δ 2 : , j ℓ = 0 . In particular, if S ℓ = ∅ S_\ell=\varnothing S ℓ = ∅ , then Δ 1 ℓ = Δ 2 ℓ = 0 \Delta_1^\ell=\Delta_2^\ell=0 Δ 1 ℓ = Δ 2 ℓ = 0 , so both operator norms appearing in ρ μ \rho_\mu ρ μ for this block are zero under the zero-allocation convention.
If S ℓ ≠ ∅ S_\ell\ne\varnothing S ℓ = ∅ , then
D μ S ℓ ℓ − α Δ 1 , S ℓ : ℓ = P 1 , S ℓ : ℓ , Δ 2 : , S ℓ ℓ D μ S ℓ ℓ − α = P 2 : , S ℓ ℓ . D_{\mu_{S_\ell}^\ell}^{-\alpha}\Delta_{1,S_\ell:}^\ell=P_{1,S_\ell:}^\ell,\qquad
\Delta_{2:,S_\ell}^\ell D_{\mu_{S_\ell}^\ell}^{-\alpha}=P_{2:,S_\ell}^\ell. D μ S ℓ ℓ − α Δ 1 , S ℓ : ℓ = P 1 , S ℓ : ℓ , Δ 2 : , S ℓ ℓ D μ S ℓ ℓ − α = P 2 : , S ℓ ℓ .
Removing rows or columns cannot increase the operator norm, so ∥ P 1 , S ℓ ℓ ∥ o p ≤ ∥ P 1 ℓ ∥ o p ≤ 1 \lVert P^{\ell}_{1,S_\ell}\rVert_{\mathrm{op}} \le\lVert P^\ell_1\rVert_{\mathrm{op}} \le1 ∥ P 1 , S ℓ ℓ ∥ op ≤ ∥ P 1 ℓ ∥ op ≤ 1 and ∥ P 2 , : , S ℓ ℓ ∥ o p ≤ ∥ P 2 ℓ ∥ o p ≤ 1 \lVert P^{\ell}_{2,:,S_\ell}\rVert_{\mathrm{op}} \le\lVert P^\ell_2\rVert_{\mathrm{op}} \le1 ∥ P 2 , : , S ℓ ℓ ∥ op ≤ ∥ P 2 ℓ ∥ op ≤ 1 . By the zero-allocation convention defining ρ μ \rho_\mu ρ μ , the rows and columns with μ j ℓ = 0 \mu^\ell_j=0 μ j ℓ = 0 , which are zero in Δ \Delta Δ , are omitted from the operator-norm computation. Hence ∥ D μ S ℓ − α Δ 1 ℓ ∥ o p = ∥ D μ S ℓ − α Δ 1 , S ℓ ℓ ∥ o p = ∥ P 1 , S ℓ ℓ ∥ o p ≤ 1 \lVert D_{\mu_{S_\ell}}^{-\alpha}\Delta^\ell_1\rVert_{\mathrm{op}}=\lVert D_{\mu_{S_\ell}}^{-\alpha}\Delta^\ell_{1,S_\ell}\rVert_{\mathrm{op}}=\lVert P^\ell_{1,S_\ell}\rVert_{\mathrm{op}}\le1 ∥ D μ S ℓ − α Δ 1 ℓ ∥ op = ∥ D μ S ℓ − α Δ 1 , S ℓ ℓ ∥ op = ∥ P 1 , S ℓ ℓ ∥ op ≤ 1 , and similarly ∥ Δ 2 ℓ D μ S ℓ − α ∥ o p ≤ 1 \lVert\Delta^\ell_2D_{\mu_{S_\ell}}^{-\alpha}\rVert_{\mathrm{op}}\le1 ∥ Δ 2 ℓ D μ S ℓ − α ∥ op ≤ 1 . Since this holds for every ℓ \ell ℓ , we conclude ρ μ ( Δ ) ≤ 1 \rho_\mu(\Delta)\le1 ρ μ ( Δ ) ≤ 1 .
Conversely, if ρ μ ( Δ ) ≤ 1 \rho_\mu(\Delta)\le1 ρ μ ( Δ ) ≤ 1 , finiteness forces Δ 1 , j : ℓ = Δ 2 , : , j ℓ = 0 \Delta^\ell_{1,j:}=\Delta^\ell_{2,:,j}=0 Δ 1 , j : ℓ = Δ 2 , : , j ℓ = 0 whenever j ∉ S ℓ j\notin S_\ell j ∈ / S ℓ . Define P 1 , S ℓ ℓ : = D μ S ℓ − α Δ 1 , S ℓ ℓ P^\ell_{1,S_\ell}:=D_{\mu_{S_\ell}}^{-\alpha}\Delta^\ell_{1,S_\ell} P 1 , S ℓ ℓ := D μ S ℓ − α Δ 1 , S ℓ ℓ and P 2 , : , S ℓ ℓ : = Δ 2 , : , S ℓ ℓ D μ S ℓ − α P^\ell_{2,:,S_\ell}:=\Delta^\ell_{2,:,S_\ell}D_{\mu_{S_\ell}}^{-\alpha} P 2 , : , S ℓ ℓ := Δ 2 , : , S ℓ ℓ D μ S ℓ − α , and set the remaining rows of P 1 ℓ P^\ell_1 P 1 ℓ and columns of P 2 ℓ P^\ell_2 P 2 ℓ to zero. By definition of ρ μ \rho_\mu ρ μ , ∥ P 1 ℓ ∥ o p ≤ 1 \lVert P^\ell_1\rVert_{\mathrm{op}}\le1 ∥ P 1 ℓ ∥ op ≤ 1 and ∥ P 2 ℓ ∥ o p ≤ 1 \lVert P^\ell_2\rVert_{\mathrm{op}}\le1 ∥ P 2 ℓ ∥ op ≤ 1 , while Δ 1 ℓ = D μ α P 1 ℓ \Delta^\ell_1=D^\alpha_\mu P^\ell_1 Δ 1 ℓ = D μ α P 1 ℓ and Δ 2 ℓ = P 2 ℓ D μ α \Delta^\ell_2=P^\ell_2D^\alpha_\mu Δ 2 ℓ = P 2 ℓ D μ α . Thus Δ ∈ A α ( μ ) \Delta\in A_\alpha(\mu) Δ ∈ A α ( μ ) .
Therefore, (10) is proved.
Now take Δ , Δ ′ ∈ A α , K \Delta,\Delta'\in A_{\alpha,K} Δ , Δ ′ ∈ A α , K . By definition of A α , K A_{\alpha,K} A α , K , there exist μ , μ ′ ∈ K K \mu,\mu'\in\mathcal{K}_K μ , μ ′ ∈ K K such that Δ ∈ A α ( μ ) \Delta\in A_\alpha(\mu) Δ ∈ A α ( μ ) and Δ ′ ∈ A α ( μ ′ ) \Delta'\in A_\alpha(\mu') Δ ′ ∈ A α ( μ ′ ) . For θ ∈ [ 0 , 1 ] \theta\in[0,1] θ ∈ [ 0 , 1 ] , set μ θ = θ μ + ( 1 − θ ) μ ′ ∈ K K \mu_\theta=\theta\mu+(1-\theta)\mu'\in\mathcal{K}_K μ θ = θ μ + ( 1 − θ ) μ ′ ∈ K K . Joint convexity of g ( Δ , μ ) = ρ μ ( Δ ) 2 g(\Delta,\mu)=\rho_\mu(\Delta)^2 g ( Δ , μ ) = ρ μ ( Δ ) 2 gives
g ( θ Δ + ( 1 − θ ) Δ ′ , μ θ ) ≤ θ g ( Δ , μ ) + ( 1 − θ ) g ( Δ ′ , μ ′ ) ≤ 1. g\bigl(\theta\Delta+(1-\theta)\Delta',\mu_\theta\bigr)\le\theta g(\Delta,\mu)+(1-\theta)g(\Delta',\mu')\le1. g ( θ Δ + ( 1 − θ ) Δ ′ , μ θ ) ≤ θ g ( Δ , μ ) + ( 1 − θ ) g ( Δ ′ , μ ′ ) ≤ 1.
Hence θ Δ + ( 1 − θ ) Δ ′ ∈ A α ( μ θ ) ⊆ A α , K \theta\Delta+(1-\theta)\Delta'\in A_\alpha(\mu_\theta)\subseteq A_{\alpha,K} θ Δ + ( 1 − θ ) Δ ′ ∈ A α ( μ θ ) ⊆ A α , K . Therefore A α , K A_{\alpha,K} A α , K is convex, and thus B α , K = A α , K B_{\alpha,K}=A_{\alpha,K} B α , K = A α , K .
Using the definition of the Minkowski functional and the positive homogeneity of ρ μ \rho_\mu ρ μ ,
Ω α , K ( Δ ) = inf { t > 0 : Δ / t ∈ A α , K } = inf { t > 0 : ∃ μ ∈ K K , ρ μ ( Δ ) ≤ t } = inf μ ∈ K K ρ μ ( Δ ) . \begin{aligned}
\Omega_{\alpha,K}(\Delta)&=\inf\{t>0:\Delta/t\in A_{\alpha,K}\}\\
&=\inf\{t>0:\exists\ \mu\in\mathcal{K}_K,\ \rho_\mu(\Delta)\le t\}\\
&=\inf_{\mu\in\mathcal{K}_K}\rho_\mu(\Delta).
\end{aligned} Ω α , K ( Δ ) = inf { t > 0 : Δ/ t ∈ A α , K } = inf { t > 0 : ∃ μ ∈ K K , ρ μ ( Δ ) ≤ t } = μ ∈ K K inf ρ μ ( Δ ) .
Finally, for fixed Δ \Delta Δ , g ( Δ , μ ) g(\Delta,\mu) g ( Δ , μ ) is lower semicontinuous in μ \mu μ . Since K K \mathcal{K}_K K K is compact, it attains its minimum, and the same allocation minimizes ρ μ ( Δ ) = g ( Δ , μ ) \rho_\mu(\Delta)=\sqrt{g(\Delta,\mu)} ρ μ ( Δ ) = g ( Δ , μ ) . Thus the last infimum is a minimum.
Proofs for the Geometric Special Cases
We prove the special cases stated in § 2.4 .
Recovery of Muon. If K = N K=N K = N , then 0 ≤ μ j ℓ ≤ 1 0\le\mu^\ell_j\le1 0 ≤ μ j ℓ ≤ 1 and ∑ ℓ , j μ j ℓ = N \sum_{\ell,j}\mu^\ell_j=N ∑ ℓ , j μ j ℓ = N imply μ j ℓ = 1 \mu^\ell_j=1 μ j ℓ = 1 for every channel. Hence D μ ℓ α = I D^\alpha_{\mu_\ell}=I D μ ℓ α = I for every ℓ \ell ℓ , so A α , N = A M u o n A_{\alpha,N}=A_{\mathrm{Muon}} A α , N = A Muon . Since A M u o n A_{\mathrm{Muon}} A Muon is convex, B α , N = A M u o n B_{\alpha,N}=A_{\mathrm{Muon}} B α , N = A Muon , and therefore Ω α , N = ∥ ⋅ ∥ M u o n \Omega_{\alpha,N}=\lVert\cdot\rVert_{\mathrm{Muon}} Ω α , N = ∥ ⋅ ∥ Muon .
Likewise, under the extension D μ 0 : = I D^0_\mu:=I D μ 0 := I , A 0 ( μ ) = A M u o n A_0(\mu)=A_{\mathrm{Muon}} A 0 ( μ ) = A Muon for every μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K . Thus B 0 , K = A M u o n B_{0,K}=A_{\mathrm{Muon}} B 0 , K = A Muon and Ω 0 , K = ∥ ⋅ ∥ M u o n \Omega_{0,K}=\lVert\cdot\rVert_{\mathrm{Muon}} Ω 0 , K = ∥ ⋅ ∥ Muon .
**Structured K K K -support geometry at α = 1. ∗ ∗ A s s u m e \alpha=1.** Assume α = 1. ∗ ∗ A ss u m e K$ is an integer and let V K : = { 1 S : ∣ S ∣ = K } \mathcal{V}_K:=\{\mathbf{1}_S:|S|=K\} V K := { 1 S : ∣ S ∣ = K } , where S ⊆ { ( ℓ , j ) : 1 ≤ ℓ ≤ L , 1 ≤ j ≤ m ℓ } S\subseteq\{(\ell,j):1\le\ell\le L,\ 1\le j\le m_\ell\} S ⊆ {( ℓ , j ) : 1 ≤ ℓ ≤ L , 1 ≤ j ≤ m ℓ } and 1 S ℓ \mathbf{1}^{\ell}_S 1 S ℓ denotes the block-ℓ \ell ℓ component of 1 S \mathbf{1}_S 1 S , with ( 1 S ℓ ) j = 1 { ( ℓ , j ) ∈ S } (\mathbf{1}^{\ell}_S)_j=\mathbf{1}\{(\ell,j)\in S\} ( 1 S ℓ ) j = 1 {( ℓ , j ) ∈ S } . As shown in Appendix B.§ 3 , the extreme points of K K \mathcal{K}_K K K are exactly V K \mathcal{V}_K V K . Since K K \mathcal{K}_K K K is a polytope, K K = conv ( V K ) \mathcal{K}_K=\operatorname{conv}(\mathcal{V}_K) K K = conv ( V K ) .
Let μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , it can be written as μ = ∑ r λ r 1 S r \mu=\sum_r\lambda_r\mathbf{1}_{S_r} μ = ∑ r λ r 1 S r , where λ r ≥ 0 \lambda_r\ge0 λ r ≥ 0 , ∑ r λ r = 1 \sum_r\lambda_r=1 ∑ r λ r = 1 , and ∣ S r ∣ = K |S_r|=K ∣ S r ∣ = K . Now let Δ ∈ A 1 ( μ ) \Delta\in A_1(\mu) Δ ∈ A 1 ( μ ) . Then, for some P 1 ℓ , P 2 ℓ P^\ell_1,P^\ell_2 P 1 ℓ , P 2 ℓ satisfying ∥ P 1 ℓ ∥ o p , ∥ P 2 ℓ ∥ o p ≤ 1 , \lVert P^\ell_1\rVert_{\mathrm{op}},\lVert P^\ell_2\rVert_{\mathrm{op}}\le1, ∥ P 1 ℓ ∥ op , ∥ P 2 ℓ ∥ op ≤ 1 ,
Δ = ( D μ ℓ P 1 ℓ , P 2 ℓ D μ ℓ ) ℓ = 1 L . \Delta= \left(D_{\mu^\ell}P_1^\ell,P_2^\ell D_{\mu^\ell}\right)_{\ell=1}^{L}. Δ = ( D μ ℓ P 1 ℓ , P 2 ℓ D μ ℓ ) ℓ = 1 L .
Since D μ ℓ D_{\mu^\ell} D μ ℓ is linear in μ ℓ \mu^\ell μ ℓ , Δ = ∑ r λ r ( D 1 S r ℓ P 1 ℓ , P 2 ℓ D 1 S r ℓ ) ℓ = 1 L \Delta= \sum_r \lambda_r\left(D_{1_{S_r}}^\ell P_1^\ell,P_2^\ell D_{1_{S_r}}^\ell\right)_{\ell=1}^{L} Δ = ∑ r λ r ( D 1 S r ℓ P 1 ℓ , P 2 ℓ D 1 S r ℓ ) ℓ = 1 L , and each term belongs to A 1 ( 1 S r ) \mathcal{A}_1(1_{S_r}) A 1 ( 1 S r ) . Therefore
⋃ μ ∈ K K A 1 ( μ ) ⊆ conv ( ⋃ ∣ S ∣ = K A 1 ( 1 S ) ) . \bigcup_{\mu\in\mathcal{K}_K}\mathcal{A}_1(\mu)\subseteq\operatorname{conv}\left(\bigcup_{|S|=K}\mathcal{A}_1(1_S)\right). μ ∈ K K ⋃ A 1 ( μ ) ⊆ conv ∣ S ∣ = K ⋃ A 1 ( 1 S ) .
Conversely, ∣ S ∣ = K |S|=K ∣ S ∣ = K implies 1 S ∈ K K 1_S\in\mathcal{K}_K 1 S ∈ K K . Hence, after taking convex hulls,
B 1 , K = conv ( ⋃ μ ∈ K K A 1 ( μ ) ) = conv ( ⋃ ∣ S ∣ = K A 1 ( 1 S ) ) . \mathcal{B}_{1,K}=\operatorname{conv}\left(\bigcup_{\mu\in\mathcal{K}_K}\mathcal{A}_1(\mu)\right)=\operatorname{conv}\left(\bigcup_{|S|=K}\mathcal{A}_1(1_S)\right). B 1 , K = conv μ ∈ K K ⋃ A 1 ( μ ) = conv ∣ S ∣ = K ⋃ A 1 ( 1 S ) .
For a binary allocation 1 S 1_S 1 S ,
A 1 ( 1 S ) = { Δ : Δ 1 ℓ = D 1 S ℓ P 1 ℓ , Δ 2 ℓ = P 2 ℓ D 1 S ℓ , ∥ P 1 ℓ ∥ o p ≤ 1 , ∥ P 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } = { Δ : Δ 1 , : , j ℓ = 0 , Δ 2 , : , j ℓ = 0 for all ( ℓ , j ) ∉ S , ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } = { Δ : supp c h ( Δ ) ⊆ S , ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } . \begin{aligned}
\mathcal{A}_1(1_S)
&=\left\{\Delta:\ \Delta_1^\ell=D_{1_S}^\ell P_1^\ell,\ \Delta_2^\ell=P_2^\ell D_{1_S}^\ell,\ \left\|P_1^\ell\right\|_{\mathrm{op}}\leq1,\ \left\|P_2^\ell\right\|_{\mathrm{op}}\leq1,\ \forall\ell\right\}\\
&=\left\{\Delta:\ \Delta_{1,:,j}^\ell=0,\ \Delta_{2,:,j}^\ell=0\text{ for all }(\ell,j)\notin S,\ \left\|\Delta_1^\ell\right\|_{\mathrm{op}}\leq1,\ \left\|\Delta_2^\ell\right\|_{\mathrm{op}}\leq1,\ \forall\ell\right\}\\
&=\left\{\Delta:\ \operatorname{supp}_{\mathrm{ch}}(\Delta)\subseteq S,\ \left\|\Delta_1^\ell\right\|_{\mathrm{op}}\leq1,\ \left\|\Delta_2^\ell\right\|_{\mathrm{op}}\leq1,\ \forall\ell\right\}.
\end{aligned} A 1 ( 1 S ) = { Δ : Δ 1 ℓ = D 1 S ℓ P 1 ℓ , Δ 2 ℓ = P 2 ℓ D 1 S ℓ , P 1 ℓ op ≤ 1 , P 2 ℓ op ≤ 1 , ∀ ℓ } = { Δ : Δ 1 , : , j ℓ = 0 , Δ 2 , : , j ℓ = 0 for all ( ℓ , j ) ∈ / S , Δ 1 ℓ op ≤ 1 , Δ 2 ℓ op ≤ 1 , ∀ ℓ } = { Δ : supp ch ( Δ ) ⊆ S , Δ 1 ℓ op ≤ 1 , Δ 2 ℓ op ≤ 1 , ∀ ℓ } .
The second equality uses that multiplication by D 1 S ℓ D_{1_S}^\ell D 1 S ℓ zeros exactly the channels outside S S S and does not increase the operator norm; conversely, one may take P i ℓ = Δ i ℓ P_i^\ell=\Delta_i^\ell P i ℓ = Δ i ℓ .
Consequently,
B 1 , K = conv ( ⋃ ∣ S ∣ = K { Δ : supp c h ( Δ ) ⊆ S , ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } ) = conv { Δ : ∣ supp c h ( Δ ) ∣ ≤ K , ∥ Δ 1 ℓ ∥ o p ≤ 1 , ∥ Δ 2 ℓ ∥ o p ≤ 1 , ∀ ℓ } . \begin{aligned}
\mathcal{B}_{1,K}
&=\operatorname{conv}\left(\bigcup_{|S|=K}\left\{\Delta:\ \operatorname{supp}_{\mathrm{ch}}(\Delta)\subseteq S,\ \left\|\Delta_1^\ell\right\|_{\mathrm{op}}\leq1,\ \left\|\Delta_2^\ell\right\|_{\mathrm{op}}\leq1,\ \forall\ell\right\}\right)\\
&=\operatorname{conv}\left\{\Delta:\ |\operatorname{supp}_{\mathrm{ch}}(\Delta)|\leq K,\ \left\|\Delta_1^\ell\right\|_{\mathrm{op}}\leq1,\ \left\|\Delta_2^\ell\right\|_{\mathrm{op}}\leq1,\ \forall\ell\right\}.
\end{aligned} B 1 , K = conv ∣ S ∣ = K ⋃ { Δ : supp ch ( Δ ) ⊆ S , Δ 1 ℓ op ≤ 1 , Δ 2 ℓ op ≤ 1 , ∀ ℓ } = conv { Δ : ∣ supp ch ( Δ ) ∣ ≤ K , Δ 1 ℓ op ≤ 1 , Δ 2 ℓ op ≤ 1 , ∀ ℓ } .
Explicit norm for α = K = 1 \alpha=K=1 α = K = 1 . Since B α , K \mathcal{B}_{\alpha,K} B α , K is the unit ball of Ω α , K \Omega_{\alpha,K} Ω α , K ,
Ω α , K ∗ ( M ) = max Δ ∈ B α , K ⟨ M , Δ ⟩ = max μ ∈ K K Φ M ( μ ) . \Omega_{\alpha,K}^*(M)=\max_{\Delta\in\mathcal{B}_{\alpha,K}}\langle M,\Delta\rangle=\max_{\mu\in\mathcal{K}_K}\Phi_M(\mu). Ω α , K ∗ ( M ) = Δ ∈ B α , K max ⟨ M , Δ ⟩ = μ ∈ K K max Φ M ( μ ) .
As shown in Appendix B.3, at α = 1 \alpha=1 α = 1 a maximizer can be chosen at an extreme point of K K \mathcal{K}_K K K . For K = 1 K=1 K = 1 , these extreme points are exactly the binary vectors with a single nonzero entry. Therefore
Ω 1 , 1 ∗ ( M ) = max μ ∈ K 1 ∑ r = 1 L [ ∥ D μ r M 1 r ∥ ∗ + ∥ M 2 r D μ r ∥ ∗ ] = max ℓ , j ∑ r = 1 L [ ∥ D 1 { ( ℓ , j ) } M 1 r ∥ ∗ + ∥ M 2 r D 1 { ( ℓ , j ) } ∥ ∗ ] = max ℓ , j [ ∥ D 1 { j } M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D 1 { j } ∥ ∗ ] = max ℓ , j [ ∥ M 1 , : , j ℓ ∥ 2 + ∥ M 2 , : , j ℓ ∥ 2 ] . \begin{aligned}
\Omega_{1,1}^*(M)
&=\max_{\mu\in\mathcal{K}_1}\sum_{r=1}^{L}\left[\left\|D_{\mu_r}M_1^r\right\|_*+\left\|M_2^rD_{\mu_r}\right\|_*\right]\\
&=\max_{\ell,j}\sum_{r=1}^{L}\left[\left\|D_{1_{\{(\ell,j)\}}}M_1^r\right\|_*+\left\|M_2^rD_{1_{\{(\ell,j)\}}}\right\|_*\right]\\
&=\max_{\ell,j}\left[\left\|D_{1_{\{j\}}}M_1^\ell\right\|_*+\left\|M_2^\ell D_{1_{\{j\}}}\right\|_*\right]\\
&=\max_{\ell,j}\left[\left\|M_{1,:,j}^\ell\right\|_2+\left\|M_{2,:,j}^\ell\right\|_2\right].
\end{aligned} Ω 1 , 1 ∗ ( M ) = μ ∈ K 1 max r = 1 ∑ L [ ∥ D μ r M 1 r ∥ ∗ + ∥ M 2 r D μ r ∥ ∗ ] = ℓ , j max r = 1 ∑ L [ D 1 {( ℓ , j )} M 1 r ∗ + M 2 r D 1 {( ℓ , j )} ∗ ] = ℓ , j max [ D 1 { j } M 1 ℓ ∗ + M 2 ℓ D 1 { j } ∗ ] = ℓ , j max [ M 1 , : , j ℓ 2 + M 2 , : , j ℓ 2 ] .
The last equality holds because the two matrices in the third equality have respectively a single nonzero row and a single nonzero column, and the nuclear norm of each equals the Euclidean norm of its nonzero row or column. Dualizing gives the result
Ω 1 , 1 ( Δ ) = ∑ ℓ = 1 L ∑ j = 1 m ℓ max { ∥ Δ 1 , j , : ℓ ∥ 2 , ∥ Δ 2 , : , j ℓ ∥ 2 } . \Omega_{1,1}(\Delta)=\sum_{\ell=1}^{L}\sum_{j=1}^{m_{\ell}}\max\left\{\left\|\Delta^{\ell}_{1,j,:}\right\|_{2},\left\|\Delta^{\ell}_{2,:,j}\right\|_{2}\right\}. Ω 1 , 1 ( Δ ) = ℓ = 1 ∑ L j = 1 ∑ m ℓ max { Δ 1 , j , : ℓ 2 , Δ 2 , : , j ℓ 2 } .
Compact-Representation Theory
Appendix C.1 proves Theorem 2, showing that a sufficiently scarce BMuon budget forces every minimum-cost linear factorization to have minimal hidden width, whereas Muon admits minimum-cost representations at full width. Appendix C.2 gives a complementary weight-decay result: channel norms are controlled by their allocation histories, and channels whose allocations vanish are driven to zero.
Intrinsic-Width Recovery
Let X = ( X ℓ ) ℓ = 1 L X=(X_\ell)_{\ell=1}^{L} X = ( X ℓ ) ℓ = 1 L , where X ℓ ∈ R p ℓ × d ℓ X_\ell\in\mathbb{R}^{p_\ell\times d_\ell} X ℓ ∈ R p ℓ × d ℓ and X ℓ ≠ 0 X_\ell\ne0 X ℓ = 0 for every ℓ \ell ℓ . Let r ℓ : = rank ( X ℓ ) ≥ 1 r_\ell:=\operatorname{rank}(X_\ell)\ge1 r ℓ := rank ( X ℓ ) ≥ 1 , and let σ ℓ , 1 ≥ ⋯ ≥ σ ℓ , r ℓ > 0 \sigma_{\ell,1}\ge\cdots\ge\sigma_{\ell,r_\ell}>0 σ ℓ , 1 ≥ ⋯ ≥ σ ℓ , r ℓ > 0 denote the nonzero singular values of X ℓ X_\ell X ℓ . Set σ ⋆ : = max ℓ , i σ ℓ , i \sigma_\star:=\max_{\ell,i}\sigma_{\ell,i} σ ⋆ := max ℓ , i σ ℓ , i .
Each block ℓ \ell ℓ consists of W 1 ℓ ∈ R m ℓ × d ℓ W_1^\ell\in\mathbb{R}^{m_\ell\times d_\ell} W 1 ℓ ∈ R m ℓ × d ℓ and W 2 ℓ ∈ R p ℓ × m ℓ W_2^\ell\in\mathbb{R}^{p_\ell\times m_\ell} W 2 ℓ ∈ R p ℓ × m ℓ . Assume m ℓ ≥ r ℓ m_\ell\ge r_\ell m ℓ ≥ r ℓ for every ℓ \ell ℓ . We consider the linear, bias-free setting, so block ℓ \ell ℓ represents the map W 2 ℓ W 1 ℓ W_2^\ell W_1^\ell W 2 ℓ W 1 ℓ . Let N : = ∑ ℓ = 1 L m ℓ N:=\sum_{\ell=1}^{L}m_\ell N := ∑ ℓ = 1 L m ℓ denote the total number of hidden channels across blocks. An exact factorization W W W satisfies W 2 ℓ W 1 ℓ = X ℓ W_2^\ell W_1^\ell=X_\ell W 2 ℓ W 1 ℓ = X ℓ for every ℓ \ell ℓ .
Recall that, for W = ( ( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L W=((W_1^\ell,W_2^\ell))_{\ell=1}^{L} W = (( W 1 ℓ , W 2 ℓ ) ) ℓ = 1 L , ∥ W ∥ M u o n : = max 1 ≤ ℓ ≤ L max { ∥ W 1 ℓ ∥ o p , ∥ W 2 ℓ ∥ o p } \lVert W\rVert_{\mathrm{Muon}}:=\max_{1\le\ell\le L}\max\{\lVert W_1^\ell\rVert_{\mathrm{op}},\lVert W_2^\ell\rVert_{\mathrm{op}}\} ∥ W ∥ Muon := max 1 ≤ ℓ ≤ L max {∥ W 1 ℓ ∥ op , ∥ W 2 ℓ ∥ op } , and define
R M u o n ( X ) : = inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ ∥ W ∥ M u o n 2 , R α , K ( X ) : = inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ Ω α , K ( W ) 2 . \mathcal{R}_{\mathrm{Muon}}(X):=\inf_{\substack{W_2^\ell W_1^\ell=X_\ell\\\forall\ell}}\lVert W\rVert_{\mathrm{Muon}}^2,\qquad
\mathcal{R}_{\alpha,K}(X):=\inf_{\substack{W_2^\ell W_1^\ell=X_\ell\\\forall\ell}}\Omega_{\alpha,K}(W)^2. R Muon ( X ) := W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf ∥ W ∥ Muon 2 , R α , K ( X ) := W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf Ω α , K ( W ) 2 .
For an exact factorization W W W , let a ℓ ( W ) a_\ell(W) a ℓ ( W ) denote the number of channels j j j such that W 1 , j : ℓ ≠ 0 W_{1,j:}^\ell\ne0 W 1 , j : ℓ = 0 or W 2 , : , j ℓ ≠ 0 W_{2,:,j}^\ell\ne0 W 2 , : , j ℓ = 0 . Finally, recall d e f f ( α ) ( X ) : = ∑ ℓ = 1 L ∑ i = 1 r ℓ ( σ ℓ , i / σ ⋆ ) 1 / ( 2 α ) d_{\mathrm{eff}}^{(\alpha)}(X):=\sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}(\sigma_{\ell,i}/\sigma_\star)^{1/(2\alpha)} d eff ( α ) ( X ) := ∑ ℓ = 1 L ∑ i = 1 r ℓ ( σ ℓ , i / σ ⋆ ) 1/ ( 2 α ) .
Theorem 2 (restated). Let 0 < α ≤ 1 2 0<\alpha\le\frac{1}{2} 0 < α ≤ 2 1 and 0 < K ≤ N 0<K\le N 0 < K ≤ N . Both representation-cost infima are attained, and
R M u o n ( X ) = σ ⋆ , R α , K ( X ) = σ ⋆ max { 1 , ( d e f f ( α ) ( X ) K ) 2 α } . \mathcal{R}_{\mathrm{Muon}}(X)=\sigma_\star,\qquad
\mathcal{R}_{\alpha,K}(X)=\sigma_\star\max\left\{1,\left(\frac{d_{\mathrm{eff}}^{(\alpha)}(X)}{K}\right)^{2\alpha}\right\}. R Muon ( X ) = σ ⋆ , R α , K ( X ) = σ ⋆ max ⎩ ⎨ ⎧ 1 , ( K d eff ( α ) ( X ) ) 2 α ⎭ ⎬ ⎫ . Moreover, Muon admits a minimizer W ⋆ W^\star W ⋆ satisfying a ℓ ( W ⋆ ) = m ℓ a_\ell(W^\star)=m_\ell a ℓ ( W ⋆ ) = m ℓ for every ℓ \ell ℓ , whereas if K ≤ d e f f ( α ) ( X ) K\le d_{\mathrm{eff}}^{(\alpha)}(X) K ≤ d eff ( α ) ( X ) , every BMuon minimizer W ⋆ W^\star W ⋆ satisfies a ℓ ( W ⋆ ) = r ℓ a_\ell(W^\star)=r_\ell a ℓ ( W ⋆ ) = r ℓ for every $\ell.
For μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , recall the fixed-allocation radius
ρ μ ( W ) : = max 1 ≤ ℓ ≤ L max { ∥ D μ − α W 1 ℓ ∥ o p , ∥ W 2 ℓ D μ − α ∥ o p } . \rho_\mu(W):=\max_{1\le\ell\le L}\max\left\{\lVert D_\mu^{-\alpha}W_1^\ell\rVert_{\mathrm{op}},\lVert W_2^\ell D_\mu^{-\alpha}\rVert_{\mathrm{op}}\right\}. ρ μ ( W ) := 1 ≤ ℓ ≤ L max max { ∥ D μ − α W 1 ℓ ∥ op , ∥ W 2 ℓ D μ − α ∥ op } .
with the zero-allocation convention of Lemma 4. By that lemma, Ω α , K ( W ) = min μ ∈ K K ρ μ ( W ) \Omega_{\alpha,K}(W)=\min_{\mu\in\mathcal{K}_K}\rho_\mu(W) Ω α , K ( W ) = min μ ∈ K K ρ μ ( W ) .
We first consider the following single-block case.
Lemma 5 (Fixed-allocation factorization) Let α > 0 \alpha>0 α > 0 . Consider a single block (L = 1 L=1 L = 1 ). Let X ∈ R p × d X\in\mathbb{R}^{p\times d} X ∈ R p × d have rank r ≥ 1 r\ge1 r ≥ 1 and nonzero singular values σ 1 ≥ ⋯ ≥ σ r > 0 \sigma_1\ge\cdots\ge\sigma_r>0 σ 1 ≥ ⋯ ≥ σ r > 0 . For μ ∈ [ 0 , 1 ] m \mu\in[0,1]^m μ ∈ [ 0 , 1 ] m , m ≥ r m\ge r m ≥ r , let μ ( 1 ) ≥ ⋯ ≥ μ ( m ) \mu_{(1)}\ge\cdots\ge\mu_{(m)} μ ( 1 ) ≥ ⋯ ≥ μ ( m ) denote its ordered coordinates. For W = ( W 1 , W 2 ) W=(W_1,W_2) W = ( W 1 , W 2 ) , define
ρ μ ( W ) : = max { ∥ D μ − α W 1 ∥ o p , ∥ W 2 D μ − α ∥ o p } , \rho_\mu(W):=\max\left\{\lVert D_\mu^{-\alpha}W_1\rVert_{\mathrm{op}},\lVert W_2D_\mu^{-\alpha}\rVert_{\mathrm{op}}\right\}, ρ μ ( W ) := max { ∥ D μ − α W 1 ∥ op , ∥ W 2 D μ − α ∥ op } ,
with the zero-allocation convention of Lemma 4 . Then
inf W 2 W 1 = X ρ μ ( W ) 2 = max 1 ≤ i ≤ r σ i μ ( i ) 2 α . \inf_{W_2W_1=X} \rho_\mu(W)^2 = \max_{1\leq i\leq r} \frac{\sigma_i}{\mu_{(i)}^{2\alpha}}. W 2 W 1 = X inf ρ μ ( W ) 2 = 1 ≤ i ≤ r max μ ( i ) 2 α σ i .
The right-hand side is + ∞ +\infty + ∞ if μ ( i ) = 0 \mu_{(i)}=0 μ ( i ) = 0 for some 1 ≤ i ≤ r 1\leq i\leq r 1 ≤ i ≤ r . Whenever it is finite, the infimum is attained.
Proof. If μ ( r ) = 0 \mu_{(r)}=0 μ ( r ) = 0 , then fewer than r r r coordinates satisfy μ j > 0 \mu_j>0 μ j > 0 . Suppose a W = ( W 1 , W 2 ) W=(W_1,W_2) W = ( W 1 , W 2 ) is an exact factorization of X X X , (i.e. W 2 W 1 = X W_2W_1=X W 2 W 1 = X ) having finite fixed-allocation radius ρ μ ( W ) \rho_\mu(W) ρ μ ( W ) . Finiteness of the radius forces W 1 , j : = 0 W_{1,j:}=0 W 1 , j : = 0 and W 2 , : , j = 0 W_{2,:,j}=0 W 2 , : , j = 0 whenever μ j = 0 \mu_j=0 μ j = 0 (convention of Lemma 4 ). Hence X = ∑ j : μ j > 0 W 2 , : , j W 1 , j : X=\sum_{j:\mu_j>0}W_{2,:,j}W_{1,j:} X = ∑ j : μ j > 0 W 2 , : , j W 1 , j : is a sum of fewer than r r r rank-one matrices. Therefore rank ( X ) < r \operatorname{rank}(X)<r rank ( X ) < r , contradicting rank ( X ) = r \operatorname{rank}(X)=r rank ( X ) = r . Thus, when μ ( r ) = 0 \mu_{(r)}=0 μ ( r ) = 0 , every W W W satisfying exact factorization has ρ μ ( W ) = + ∞ \rho_\mu(W)=+\infty ρ μ ( W ) = + ∞ . Since the right-hand side is also + ∞ +\infty + ∞ by convention when μ ( r ) = 0 \mu_{(r)}=0 μ ( r ) = 0 , the claimed equality holds in this case.
Assume now μ ( r ) > 0 \mu_{(r)}>0 μ ( r ) > 0 , and let W = ( W 1 , W 2 ) W=(W_1,W_2) W = ( W 1 , W 2 ) be an exact factorization of X X X , i.e. W 2 W 1 = X W_2W_1=X W 2 W 1 = X , with finite fixed-allocation radius ρ μ ( W ) < ∞ \rho_\mu(W)<\infty ρ μ ( W ) < ∞ . Let S : = { j : μ j > 0 } S:=\{j:\mu_j>0\} S := { j : μ j > 0 } . Finite radius implies W 1 , S c : = 0 W_{1,S^c:}=0 W 1 , S c : = 0 and W 2 , : , S c = 0 W_{2,:,S^c}=0 W 2 , : , S c = 0 (convention of Lemma 4 ), hence X = ( W 2 , : , S D μ S − α ) D μ S 2 α ( D μ S − α W 1 , S : ) X=(W_{2,:,S}D_{\mu_S}^{-\alpha})D_{\mu_S}^{2\alpha}(D_{\mu_S}^{-\alpha}W_{1,S:}) X = ( W 2 , : , S D μ S − α ) D μ S 2 α ( D μ S − α W 1 , S : ) . Using σ i ( A B C ) ≤ ∥ A ∥ o p σ i ( B ) ∥ C ∥ o p \sigma_i(ABC)\leq\|A\|_{\mathrm{op}}\sigma_i(B)\|C\|_{\mathrm{op}} σ i ( A B C ) ≤ ∥ A ∥ op σ i ( B ) ∥ C ∥ op , we obtain, for i ≤ r i\leq r i ≤ r ,
σ i ≤ max { ∥ D μ − α W 1 ∥ o p 2 , ∥ W 2 D μ − α ∥ o p 2 } μ ( i ) 2 α . \sigma_i\leq\max\left\{\|D_\mu^{-\alpha}W_1\|_{\mathrm{op}}^2,\|W_2D_\mu^{-\alpha}\|_{\mathrm{op}}^2\right\}\mu_{(i)}^{2\alpha}. σ i ≤ max { ∥ D μ − α W 1 ∥ op 2 , ∥ W 2 D μ − α ∥ op 2 } μ ( i ) 2 α .
The weighted operator norms are interpreted according to the zero-allocation convention in the definition of ρ μ \rho_\mu ρ μ in Lemma 4 .
So, for any exact factorization with finite fixed-allocation radius, for every i ≤ r i\leq r i ≤ r , ρ μ ( W ) 2 ≥ σ i μ ( i ) 2 α \rho_\mu(W)^2\geq\frac{\sigma_i}{\mu_{(i)}^{2\alpha}} ρ μ ( W ) 2 ≥ μ ( i ) 2 α σ i , hence
ρ μ ( W ) 2 ≥ max 1 ≤ i ≤ r σ i μ ( i ) 2 α . \rho_\mu(W)^2\geq\max_{1\leq i\leq r}\frac{\sigma_i}{\mu_{(i)}^{2\alpha}}. ρ μ ( W ) 2 ≥ 1 ≤ i ≤ r max μ ( i ) 2 α σ i .
The same bound is trivial when ρ μ ( W ) = + ∞ \rho_\mu(W)=+\infty ρ μ ( W ) = + ∞ . Taking the infimum over all exact factorizations W W W gives
inf W 2 W 1 = X ρ μ ( W ) 2 ≥ max 1 ≤ i ≤ r σ i μ ( i ) 2 α . \inf_{W_2W_1=X}\rho_\mu(W)^2\geq\max_{1\leq i\leq r}\frac{\sigma_i}{\mu_{(i)}^{2\alpha}}. W 2 W 1 = X inf ρ μ ( W ) 2 ≥ 1 ≤ i ≤ r max μ ( i ) 2 α σ i .
For the reverse inequality, permutation of the hidden coordinates leaves both W 2 W 1 W_2W_1 W 2 W 1 and ∥ D μ − α W 1 ∥ o p , ∥ W 2 D μ − α ∥ o p \|D_\mu^{-\alpha}W_1\|_{\mathrm{op}},\|W_2D_\mu^{-\alpha}\|_{\mathrm{op}} ∥ D μ − α W 1 ∥ op , ∥ W 2 D μ − α ∥ op unchanged. So assume without loss of generality that μ 1 ≥ ⋯ ≥ μ m \mu_1\geq\cdots\geq\mu_m μ 1 ≥ ⋯ ≥ μ m . Let X = U diag ( σ 1 , … , σ r ) V ⊤ X=U\operatorname{diag}(\sigma_1,\ldots,\sigma_r)V^\top X = U diag ( σ 1 , … , σ r ) V ⊤ be a compact SVD. Set W 1 , i : = σ i v i ⊤ W_{1,i:}=\sqrt{\sigma_i}v_i^\top W 1 , i : = σ i v i ⊤ and W 2 , : , i = σ i u i W_{2,:,i}=\sqrt{\sigma_i}u_i W 2 , : , i = σ i u i for i ≤ r i\leq r i ≤ r , with all remaining rows and columns equal to zero. Then W 2 W 1 = X W_2W_1=X W 2 W 1 = X , and since U U U and V V V have orthonormal columns, the nonzero singular values of both D μ − α W 1 D_\mu^{-\alpha}W_1 D μ − α W 1 and W 2 D μ − α W_2D_\mu^{-\alpha} W 2 D μ − α are { σ i / μ i α } i = 1 r \{\sqrt{\sigma_i}/\mu_i^\alpha\}_{i=1}^r { σ i / μ i α } i = 1 r . Hence
∥ D μ − α W 1 ∥ o p 2 = ∥ W 2 D μ − α ∥ o p 2 = max 1 ≤ i ≤ r σ i μ i 2 α , \|D_\mu^{-\alpha}W_1\|_{\mathrm{op}}^2=\|W_2D_\mu^{-\alpha}\|_{\mathrm{op}}^2=\max_{1\leq i\leq r}\frac{\sigma_i}{\mu_i^{2\alpha}}, ∥ D μ − α W 1 ∥ op 2 = ∥ W 2 D μ − α ∥ op 2 = 1 ≤ i ≤ r max μ i 2 α σ i ,
which gives the reverse inequality and completes the proof. ■
We now prove Theorem 2.
Proof. We first consider Muon. Recall that an exact factorization W = ( W ℓ ) ℓ = 1 L W = (W^\ell)_{\ell=1}^{L} W = ( W ℓ ) ℓ = 1 L , with W ℓ = ( W 1 ℓ , W 2 ℓ ) W^\ell= (W^\ell_1,W^\ell_2) W ℓ = ( W 1 ℓ , W 2 ℓ ) , satisfies W 2 ℓ W 1 ℓ = X ℓ W^\ell_2W^\ell_1 = X_\ell W 2 ℓ W 1 ℓ = X ℓ for every ℓ ∈ { 1 , … , L } \ell\in\{1,\ldots,L\} ℓ ∈ { 1 , … , L } . For any such factorization W W W ,
σ ℓ , 1 = ∥ X ℓ ∥ o p ≤ ∥ W 2 ℓ ∥ o p ∥ W 1 ℓ ∥ o p ≤ ∥ W ∥ M u o n 2 , \sigma_{\ell,1} = \lVert X_\ell\rVert_{\mathrm{op}} \le\lVert W^\ell_2\rVert_{\mathrm{op}}\lVert W^\ell_1\rVert_{\mathrm{op}} \le\lVert W\rVert_{\mathrm{Muon}}^2, σ ℓ , 1 = ∥ X ℓ ∥ op ≤ ∥ W 2 ℓ ∥ op ∥ W 1 ℓ ∥ op ≤ ∥ W ∥ Muon 2 ,
hence ∥ W ∥ M u o n 2 ≥ σ ⋆ \lVert W\rVert_{\mathrm{Muon}}^2 \ge\sigma_\star ∥ W ∥ Muon 2 ≥ σ ⋆ .
To attain this bound at full width, let X ℓ = U ℓ Σ ℓ V ℓ T X_\ell= U_\ell\Sigma_\ell V_\ell^{\mathsf{T}} X ℓ = U ℓ Σ ℓ V ℓ T be a compact SVD and choose A ℓ ∈ R m ℓ × r ℓ A_\ell\in\mathbb{R}^{m_\ell\times r_\ell} A ℓ ∈ R m ℓ × r ℓ such that A ℓ T A ℓ = I A_\ell^{\mathsf{T}}A_\ell= I A ℓ T A ℓ = I and every row of A ℓ A_\ell A ℓ is nonzero. For example, take its first column to be m ℓ − 1 / 2 1 m_\ell^{-1/2}\mathbf{1} m ℓ − 1/2 1 and complete it to an orthonormal family. Define W 1 ℓ = A ℓ Σ ℓ 1 / 2 V ℓ T W^\ell_1 = A_\ell\Sigma_\ell^{1/2}V_\ell^{\mathsf{T}} W 1 ℓ = A ℓ Σ ℓ 1/2 V ℓ T and W 2 ℓ = U ℓ Σ ℓ 1 / 2 A ℓ T W^\ell_2 = U_\ell\Sigma_\ell^{1/2}A_\ell^{\mathsf{T}} W 2 ℓ = U ℓ Σ ℓ 1/2 A ℓ T , then W 2 ℓ W 1 ℓ = X ℓ W^\ell_2W^\ell_1 = X_\ell W 2 ℓ W 1 ℓ = X ℓ and ∥ W 1 ℓ ∥ o p 2 = ∥ W 2 ℓ ∥ o p 2 = σ ℓ , 1 \lVert W^\ell_1\rVert_{\mathrm{op}}^2 = \lVert W^\ell_2\rVert_{\mathrm{op}}^2 = \sigma_{\ell,1} ∥ W 1 ℓ ∥ op 2 = ∥ W 2 ℓ ∥ op 2 = σ ℓ , 1 . Thus R M u o n ( X ) = σ ⋆ R_{\mathrm{Muon}}(X) = \sigma_\star R Muon ( X ) = σ ⋆ . Since every row of A ℓ A_\ell A ℓ is nonzero and Σ ℓ \Sigma_\ell Σ ℓ has strictly positive diagonal entries, every row of W 1 ℓ W^\ell_1 W 1 ℓ and every column of W 2 ℓ W^\ell_2 W 2 ℓ is nonzero. Hence this minimizer satisfies a ℓ ( W ) = m ℓ a_\ell(W) = m_\ell a ℓ ( W ) = m ℓ for every ℓ \ell ℓ , which concludes the Muon part of the proof.
We now turn to BMuon. Lemma 4 implies:
R α , K ( X ) = inf μ ∈ K K inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ ρ μ ( W ) 2 . (11) R_{\alpha,K}(X) = \inf_{\mu\in\mathcal{K}_K}\inf_{\substack{W^\ell_2W^\ell_1=X_\ell\\ \forall\ell}} \rho_\mu(W)^2. \tag*{(11)} R α , K ( X ) = μ ∈ K K inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf ρ μ ( W ) 2 . (11)
For fixed μ \mu μ , denote
c ℓ : = inf W 2 ℓ W 1 ℓ = X ℓ max { ∥ D μ ℓ − α W 1 ℓ ∥ o p 2 , ∥ W 2 ℓ D μ ℓ − α ∥ o p 2 } . c_\ell:= \inf_{W^\ell_2W^\ell_1=X_\ell} \max\left\{\lVert D_{\mu^\ell}^{-\alpha}W^\ell_1\rVert_{\mathrm{op}}^2,\lVert W^\ell_2D_{\mu^\ell}^{-\alpha}\rVert_{\mathrm{op}}^2\right\}. c ℓ := W 2 ℓ W 1 ℓ = X ℓ inf max { ∥ D μ ℓ − α W 1 ℓ ∥ op 2 , ∥ W 2 ℓ D μ ℓ − α ∥ op 2 } .
We want to prove that, for this fixed allocation μ \mu μ ,
inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ ρ μ ( W ) 2 = max 1 ≤ ℓ ≤ L c ℓ . (12) \inf_{\substack{W^\ell_2W^\ell_1=X_\ell\\ \forall\ell}} \rho_\mu(W)^2 = \max_{1\le\ell\le L}c_\ell. \tag*{(12)} W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf ρ μ ( W ) 2 = 1 ≤ ℓ ≤ L max c ℓ . (12)
For every exact factorization W W W and every ℓ ∈ { 1 , … , L } \ell\in\{1,\ldots,L\} ℓ ∈ { 1 , … , L } ,
max { ∥ D μ ℓ − α W 1 ℓ ∥ o p 2 , ∥ W 2 ℓ D μ ℓ − α ∥ o p 2 } ≥ c ℓ . \max\left\{\lVert D_{\mu^\ell}^{-\alpha}W^\ell_1\rVert_{\mathrm{op}}^2,\lVert W^\ell_2D_{\mu^\ell}^{-\alpha}\rVert_{\mathrm{op}}^2\right\} \ge c_\ell. max { ∥ D μ ℓ − α W 1 ℓ ∥ op 2 , ∥ W 2 ℓ D μ ℓ − α ∥ op 2 } ≥ c ℓ .
Hence ρ μ ( W ) 2 ≥ max 1 ≤ ℓ ≤ L c ℓ \rho_\mu(W)^2 \ge\max_{1\le\ell\le L}c_\ell ρ μ ( W ) 2 ≥ max 1 ≤ ℓ ≤ L c ℓ , and thus
inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ ρ μ ( W ) 2 ≥ max 1 ≤ ℓ ≤ L c ℓ . \inf_{\substack{W^\ell_2W^\ell_1=X_\ell\\ \forall\ell}} \rho_\mu(W)^2 \ge\max_{1\le\ell\le L}c_\ell. W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf ρ μ ( W ) 2 ≥ 1 ≤ ℓ ≤ L max c ℓ .
Conversely, suppose first that every c ℓ < ∞ c_\ell< \infty c ℓ < ∞ . Lemma 5 shows that, for every ℓ \ell ℓ , the infimum defining c ℓ c_\ell c ℓ is attained by some exact factorization W ℓ , ⋆ W^{\ell,\star} W ℓ , ⋆ . Let W ⋆ : = ( W ℓ , ⋆ ) ℓ = 1 L W^\star:= (W^{\ell,\star})_{\ell=1}^{L} W ⋆ := ( W ℓ , ⋆ ) ℓ = 1 L , we have ρ μ ( W ⋆ ) 2 = max 1 ≤ ℓ ≤ L c ℓ \rho_\mu(W^\star)^2 = \max_{1\le\ell\le L}c_\ell ρ μ ( W ⋆ ) 2 = max 1 ≤ ℓ ≤ L c ℓ . Therefore
inf W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ ρ μ ( W ) 2 ≤ max 1 ≤ ℓ ≤ L c ℓ . \inf_{\substack{W^\ell_2W^\ell_1=X_\ell\\ \forall\ell}} \rho_\mu(W)^2 \le\max_{1\le\ell\le L}c_\ell. W 2 ℓ W 1 ℓ = X ℓ ∀ ℓ inf ρ μ ( W ) 2 ≤ 1 ≤ ℓ ≤ L max c ℓ .
The same inequality holds if for some ℓ \ell ℓ , c ℓ = + ∞ c_\ell= +\infty c ℓ = + ∞ . Hence, in all cases, (12) holds. For this fixed μ \mu μ , let μ ( 1 ) ℓ ≥ ⋯ ≥ μ ( m ℓ ) ℓ \mu^{\ell}_{(1)} \ge\cdots\ge\mu^{\ell}_{(m_\ell)} μ ( 1 ) ℓ ≥ ⋯ ≥ μ ( m ℓ ) ℓ denote the ordered coordinates of μ ℓ \mu^\ell μ ℓ . By Lemma 5, c ℓ = max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ( i ) ℓ ) 2 α c_\ell= \max_{1\le i\le r_\ell}\frac{\sigma_{\ell,i}}{(\mu^\ell_{(i)})^{2\alpha}} c ℓ = max 1 ≤ i ≤ r ℓ ( μ ( i ) ℓ ) 2 α σ ℓ , i . Therefore, combining Eqs. (11) and (12) ,
R α , K ( X ) = inf μ ∈ K K max 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ( i ) ℓ ) 2 α . R_{\alpha,K}(X)=\inf_{\mu\in\mathcal{K}_K}\max_{1\leq\ell\leq L}\max_{1\leq i\leq r_\ell}\frac{\sigma_{\ell,i}}{(\mu^\ell_{(i)})^{2\alpha}}. R α , K ( X ) = μ ∈ K K inf 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ max ( μ ( i ) ℓ ) 2 α σ ℓ , i .
Set q : = 1 / ( 2 α ) q:=1/(2\alpha) q := 1/ ( 2 α ) . For any μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , denote M ( μ ) : = max 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ( i ) ℓ ) 2 α M(\mu):=\max_{1\leq\ell\leq L}\max_{1\leq i\leq r_\ell}\frac{\sigma_{\ell,i}}{(\mu^\ell_{(i)})^{2\alpha}} M ( μ ) := max 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ ( μ ( i ) ℓ ) 2 α σ ℓ , i .
Assume first that M ( μ ) < ∞ M(\mu)<\infty M ( μ ) < ∞ . Since μ ( i ) ℓ ≤ 1 \mu^\ell_{(i)}\leq1 μ ( i ) ℓ ≤ 1 , we have M ( μ ) ≥ σ ℓ , i M(\mu)\geq\sigma_{\ell,i} M ( μ ) ≥ σ ℓ , i for every ℓ \ell ℓ and i ≤ r ℓ i\leq r_\ell i ≤ r ℓ . Hence M ( μ ) ≥ σ ⋆ M(\mu)\geq\sigma_\star M ( μ ) ≥ σ ⋆ . Moreover, for every ℓ \ell ℓ and i ≤ r ℓ i\leq r_\ell i ≤ r ℓ , M ( μ ) ≥ σ ℓ , i / ( μ ( i ) ℓ ) 2 α M(\mu)\geq\sigma_{\ell,i}/(\mu^\ell_{(i)})^{2\alpha} M ( μ ) ≥ σ ℓ , i / ( μ ( i ) ℓ ) 2 α implies
μ ( i ) ℓ ≥ ( σ ℓ , i M ( μ ) ) q . \mu^\ell_{(i)}\geq\left(\frac{\sigma_{\ell,i}}{M(\mu)}\right)^q. μ ( i ) ℓ ≥ ( M ( μ ) σ ℓ , i ) q .
Therefore
K = ∑ ℓ = 1 L ∑ j = 1 m ℓ μ j ℓ ≥ ∑ ℓ = 1 L ∑ i = 1 r ℓ μ ( i ) ℓ ≥ 1 M ( μ ) q ∑ ℓ = 1 L ∑ i = 1 r ℓ σ ℓ , i q , K=\sum_{\ell=1}^{L}\sum_{j=1}^{m_\ell}\mu^\ell_j\geq\sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}\mu^\ell_{(i)}\geq\frac{1}{M(\mu)^q}\sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}\sigma_{\ell,i}^q, K = ℓ = 1 ∑ L j = 1 ∑ m ℓ μ j ℓ ≥ ℓ = 1 ∑ L i = 1 ∑ r ℓ μ ( i ) ℓ ≥ M ( μ ) q 1 ℓ = 1 ∑ L i = 1 ∑ r ℓ σ ℓ , i q ,
and thus
M ( μ ) ≥ ( ∑ ℓ , i σ ℓ , i q K ) 1 / q . M(\mu)\geq\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^q}{K}\right)^{1/q}. M ( μ ) ≥ ( K ∑ ℓ , i σ ℓ , i q ) 1/ q .
Hence M ( μ ) ≥ max { σ ⋆ , ( ∑ ℓ , i σ ℓ , i q K ) 1 / q } M(\mu)\geq\max\left\{\sigma_\star,\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^q}{K}\right)^{1/q}\right\} M ( μ ) ≥ max { σ ⋆ , ( K ∑ ℓ , i σ ℓ , i q ) 1/ q } , which also holds directly when M ( μ ) = + ∞ M(\mu)=+\infty M ( μ ) = + ∞ .
Since this holds for every μ ∈ K K \mu\in\mathcal{K}_K μ ∈ K K , and R α , K ( X ) = inf μ ∈ K K M ( μ ) R_{\alpha,K}(X)=\inf_{\mu\in\mathcal{K}_K}M(\mu) R α , K ( X ) = inf μ ∈ K K M ( μ ) , we obtain
R α , K ( X ) ≥ max { σ ⋆ , ( ∑ ℓ , i σ ℓ , i q K ) 1 / q } . (13) R_{\alpha,K}(X)\geq\max\left\{\sigma_\star,\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^q}{K}\right)^{1/q}\right\}. \tag*{(13)} R α , K ( X ) ≥ max ⎩ ⎨ ⎧ σ ⋆ , ( K ∑ ℓ , i σ ℓ , i q ) 1/ q ⎭ ⎬ ⎫ . (13)
We now prove the reverse inequality. Set T : = max { σ ⋆ , ( ∑ ℓ , i σ ℓ , i q K ) 1 / q } T:=\max\left\{\sigma_\star,\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^q}{K}\right)^{1/q}\right\} T := max { σ ⋆ , ( K ∑ ℓ , i σ ℓ , i q ) 1/ q } . For each block ℓ \ell ℓ , set μ i ℓ = ( σ ℓ , i / T ) q \mu^\ell_i=(\sigma_{\ell,i}/T)^q μ i ℓ = ( σ ℓ , i / T ) q for 1 ≤ i ≤ r ℓ 1\leq i\leq r_\ell 1 ≤ i ≤ r ℓ , and set the remaining coordinates to zero. Since T ≥ σ ⋆ T\geq\sigma_\star T ≥ σ ⋆ , all these coordinates lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Moreover,
∑ ℓ = 1 L ∑ i = 1 r ℓ μ i ℓ = ∑ ℓ , i σ ℓ , i q T q ≤ K . \sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}\mu^\ell_i=\frac{\sum_{\ell,i}\sigma_{\ell,i}^q}{T^q}\leq K. ℓ = 1 ∑ L i = 1 ∑ r ℓ μ i ℓ = T q ∑ ℓ , i σ ℓ , i q ≤ K .
Let S : = ∑ ℓ = 1 L ∑ i = 1 r ℓ μ i ℓ S:=\sum_{\ell=1}^{L}\sum_{i=1}^{r_\ell}\mu^\ell_i S := ∑ ℓ = 1 L ∑ i = 1 r ℓ μ i ℓ . The current allocation has total mass S S S , so the amount still to be added to obtain an allocation in K K \mathcal{K}_K K K is K − S K-S K − S . The total mass that can be added without violating the constraint μ j ℓ ≤ 1 \mu^\ell_j\leq1 μ j ℓ ≤ 1 is ∑ ℓ = 1 L ∑ j = 1 m ℓ ( 1 − μ j ℓ ) = N − S \sum_{\ell=1}^{L}\sum_{j=1}^{m_\ell}(1-\mu^\ell_j)=N-S ∑ ℓ = 1 L ∑ j = 1 m ℓ ( 1 − μ j ℓ ) = N − S . Since K ≤ N K\leq N K ≤ N , we have K − S ≤ N − S K-S\leq N-S K − S ≤ N − S . Hence we can increase the coordinates, without making any of them exceed 1 1 1 , until the total mass becomes K K K . Denote the resulting allocation by μ ~ \widetilde{\mu} μ . Then μ ~ ∈ K K \widetilde{\mu}\in\mathcal{K}_K μ ∈ K K , and since μ ~ j ℓ ≥ μ j ℓ \widetilde{\mu}^\ell_j\geq\mu^\ell_j μ j ℓ ≥ μ j ℓ for every ℓ , j \ell,j ℓ , j ,
μ ~ ( i ) ℓ ≥ μ ( i ) ℓ = ( σ ℓ , i T ) q , 1 ≤ i ≤ r ℓ . \widetilde{\mu}^\ell_{(i)}\geq\mu^\ell_{(i)}=\left(\frac{\sigma_{\ell,i}}{T}\right)^q,\qquad1\leq i\leq r_\ell. μ ( i ) ℓ ≥ μ ( i ) ℓ = ( T σ ℓ , i ) q , 1 ≤ i ≤ r ℓ .
We have therefore constructed an allocation μ ~ ∈ K K \widetilde{\mu}\in\mathcal{K}_K μ ∈ K K such that
M ( μ ~ ) = max 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ~ ( i ) ℓ ) 2 α ≤ T . M(\widetilde{\mu})=\max_{1\leq\ell\leq L}\max_{1\leq i\leq r_\ell}\frac{\sigma_{\ell,i}}{(\widetilde{\mu}^\ell_{(i)})^{2\alpha}}\leq T. M ( μ ) = 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ max ( μ ( i ) ℓ ) 2 α σ ℓ , i ≤ T .
Hence
R α , K ( X ) ≤ T . (14) \mathcal{R}_{\alpha,K}(X) \le T. \tag*{(14)} R α , K ( X ) ≤ T . (14)
By Eq. (13) and Eq. (14) ,
R α , K ( X ) = max { σ ⋆ , ( ∑ ℓ , i σ ℓ , i q K ) 1 / q } . \mathcal{R}_{\alpha,K}(X)=\max\left\{\sigma_{\star},\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^{q}}{K}\right)^{1/q}\right\}. R α , K ( X ) = max ⎩ ⎨ ⎧ σ ⋆ , ( K ∑ ℓ , i σ ℓ , i q ) 1/ q ⎭ ⎬ ⎫ .
Since ∑ ℓ , i σ ℓ , i q = σ ⋆ q d e f f ( α ) ( X ) \sum_{\ell,i}\sigma_{\ell,i}^{q}=\sigma_{\star}^{q}d_{\mathrm{eff}}^{(\alpha)}(X) ∑ ℓ , i σ ℓ , i q = σ ⋆ q d eff ( α ) ( X ) and 1 / q = 2 α 1/q=2\alpha 1/ q = 2 α , we obtain
R α , K ( X ) = σ ⋆ max { 1 , ( d e f f ( α ) ( X ) K ) 2 α } . \mathcal{R}_{\alpha,K}(X)=\sigma_{\star}\max\left\{1,\left(\frac{d_{\mathrm{eff}}^{(\alpha)}(X)}{K}\right)^{2\alpha}\right\}. R α , K ( X ) = σ ⋆ max ⎩ ⎨ ⎧ 1 , ( K d eff ( α ) ( X ) ) 2 α ⎭ ⎬ ⎫ .
We now prove attainment. Let μ ~ ∈ K K \widetilde{\mu}\in\mathcal{K}_{K} μ ∈ K K be the allocation constructed above. By Lemma 5, for every block ℓ \ell ℓ , there exists an exact factorization W ℓ , ⋆ W^{\ell,\star} W ℓ , ⋆ attaining
inf W 2 ℓ W 1 ℓ = X ℓ max { ∥ D μ ~ ℓ − α W 1 ℓ ∥ o p 2 , ∥ W 2 ℓ D μ ~ ℓ − α ∥ o p 2 } . \inf_{W_{2}^{\ell}W_{1}^{\ell}=X_{\ell}}\max\left\{\left\|D_{\widetilde{\mu}_{\ell}}^{-\alpha}W_{1}^{\ell}\right\|_{\mathrm{op}}^{2},\left\|W_{2}^{\ell}D_{\widetilde{\mu}_{\ell}}^{-\alpha}\right\|_{\mathrm{op}}^{2}\right\}. W 2 ℓ W 1 ℓ = X ℓ inf max { D μ ℓ − α W 1 ℓ op 2 , W 2 ℓ D μ ℓ − α op 2 } .
Let W ⋆ : = ( W ℓ , ⋆ ) ℓ = 1 L W^{\star}:=(W^{\ell,\star})_{\ell=1}^{L} W ⋆ := ( W ℓ , ⋆ ) ℓ = 1 L , then W ⋆ W^{\star} W ⋆ is an exact factorization and
ρ μ ~ ( W ⋆ ) 2 = max 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ~ ( i ) ℓ ) 2 α ≤ T = R α , K ( X ) . \rho_{\widetilde{\mu}}(W^{\star})^{2}=\max_{1\le\ell\le L}\max_{1\le i\le r_{\ell}}\frac{\sigma_{\ell,i}}{(\widetilde{\mu}_{(i)}^{\ell})^{2\alpha}}\le T=\mathcal{R}_{\alpha,K}(X). ρ μ ( W ⋆ ) 2 = 1 ≤ ℓ ≤ L max 1 ≤ i ≤ r ℓ max ( μ ( i ) ℓ ) 2 α σ ℓ , i ≤ T = R α , K ( X ) .
Since Ω α , K ( W ⋆ ) = min ν ∈ K K ρ ν ( W ⋆ ) \Omega_{\alpha,K}(W^{\star})=\min_{\nu\in\mathcal{K}_{K}}\rho_{\nu}(W^{\star}) Ω α , K ( W ⋆ ) = min ν ∈ K K ρ ν ( W ⋆ ) (Lemma 4),
R α , K ( X ) ≤ Ω α , K ( W ⋆ ) 2 ≤ ρ μ ~ ( W ⋆ ) 2 ≤ R α , K ( X ) . \mathcal{R}_{\alpha,K}(X)\le\Omega_{\alpha,K}(W^{\star})^{2}\le\rho_{\widetilde{\mu}}(W^{\star})^{2}\le\mathcal{R}_{\alpha,K}(X). R α , K ( X ) ≤ Ω α , K ( W ⋆ ) 2 ≤ ρ μ ( W ⋆ ) 2 ≤ R α , K ( X ) .
Hence all these terms are equal, and W ⋆ W^{\star} W ⋆ attains the infimum defining R α , K ( X ) \mathcal{R}_{\alpha,K}(X) R α , K ( X ) .
We now prove intrinsic-width recovery. Assume K ≤ d e f f ( α ) ( X ) K\le d_{\mathrm{eff}}^{(\alpha)}(X) K ≤ d eff ( α ) ( X ) . Then R α , K ( X ) = ( ∑ ℓ , i σ ℓ , i q K ) 1 / q \mathcal{R}_{\alpha,K}(X)=\left(\frac{\sum_{\ell,i}\sigma_{\ell,i}^{q}}{K}\right)^{1/q} R α , K ( X ) = ( K ∑ ℓ , i σ ℓ , i q ) 1/ q .
Let W ⋆ W^{\star} W ⋆ be any minimizer of R α , K ( X ) \mathcal{R}_{\alpha,K}(X) R α , K ( X ) . By Lemma 4, there exists μ ⋆ ∈ K K \mu^{\star}\in\mathcal{K}_{K} μ ⋆ ∈ K K such that Ω α , K ( W ⋆ ) = ρ μ ⋆ ( W ⋆ ) \Omega_{\alpha,K}(W^{\star})=\rho_{\mu^{\star}}(W^{\star}) Ω α , K ( W ⋆ ) = ρ μ ⋆ ( W ⋆ ) . Since W ⋆ W^{\star} W ⋆ is a minimizer, ρ μ ⋆ ( W ⋆ ) 2 = R α , K ( X ) \rho_{\mu^{\star}}(W^{\star})^{2}=\mathcal{R}_{\alpha,K}(X) ρ μ ⋆ ( W ⋆ ) 2 = R α , K ( X ) . By Lemma 5 applied to each block ℓ \ell ℓ ,
max 1 ≤ i ≤ r ℓ σ ℓ , i ( μ ( i ) ℓ , ⋆ ) 2 α ≤ R α , K ( X ) . \max_{1\le i\le r_{\ell}}\frac{\sigma_{\ell,i}}{(\mu_{(i)}^{\ell,\star})^{2\alpha}}\le\mathcal{R}_{\alpha,K}(X). 1 ≤ i ≤ r ℓ max ( μ ( i ) ℓ , ⋆ ) 2 α σ ℓ , i ≤ R α , K ( X ) .
Thus, for every i ≤ r ℓ i\le r_{\ell} i ≤ r ℓ ,
μ ( i ) ℓ , ⋆ ≥ ( σ ℓ , i R α , K ( X ) ) q . \mu_{(i)}^{\ell,\star}\ge\left(\frac{\sigma_{\ell,i}}{\mathcal{R}_{\alpha,K}(X)}\right)^{q}. μ ( i ) ℓ , ⋆ ≥ ( R α , K ( X ) σ ℓ , i ) q .
Therefore
K = ∑ ℓ , i ( σ ℓ , i R α , K ( X ) ) q ≤ ∑ ℓ = 1 L ∑ i = 1 r ℓ μ ( i ) ℓ , ⋆ ≤ ∑ ℓ = 1 L ∑ j = 1 m ℓ μ j ℓ , ⋆ = K . K=\sum_{\ell,i}\left(\frac{\sigma_{\ell,i}}{\mathcal{R}_{\alpha,K}(X)}\right)^{q}\le\sum_{\ell=1}^{L}\sum_{i=1}^{r_{\ell}}\mu_{(i)}^{\ell,\star}\le\sum_{\ell=1}^{L}\sum_{j=1}^{m_{\ell}}\mu_{j}^{\ell,\star}=K. K = ℓ , i ∑ ( R α , K ( X ) σ ℓ , i ) q ≤ ℓ = 1 ∑ L i = 1 ∑ r ℓ μ ( i ) ℓ , ⋆ ≤ ℓ = 1 ∑ L j = 1 ∑ m ℓ μ j ℓ , ⋆ = K .
Hence all the terms are equal and μ ( j ) ℓ , ⋆ = 0 \mu_{(j)}^{\ell,\star}=0 μ ( j ) ℓ , ⋆ = 0 for every j > r ℓ j>r_\ell j > r ℓ . Since ρ μ ⋆ ( W ⋆ ) < ∞ \rho_{\mu^\star}(W^\star)<\infty ρ μ ⋆ ( W ⋆ ) < ∞ , every channel with zero allocation has both its corresponding row of W 1 ℓ , ⋆ W_1^{\ell,\star} W 1 ℓ , ⋆ and column of W 2 ℓ , ⋆ W_2^{\ell,\star} W 2 ℓ , ⋆ equal to zero (convention of Lemma 4). Therefore
a ℓ ( W ⋆ ) ≤ r ℓ . a_\ell(W^\star)\le r_\ell. a ℓ ( W ⋆ ) ≤ r ℓ .
On the other hand, X ℓ = ∑ j : c h a n n e l j a c t i v e W 2 , : , j ℓ , ⋆ W 1 , j , : ℓ , ⋆ X_\ell=\sum_{j:\,\mathrm{channel}\ j\ \mathrm{active}}W_{2,:,j}^{\ell,\star}W_{1,j,:}^{\ell,\star} X ℓ = ∑ j : channel j active W 2 , : , j ℓ , ⋆ W 1 , j , : ℓ , ⋆ is a sum of at most a ℓ ( W ⋆ ) a_\ell(W^\star) a ℓ ( W ⋆ ) rank-one matrices. Hence r ℓ = rank ( X ℓ ) ≤ a ℓ ( W ⋆ ) r_\ell=\operatorname{rank}(X_\ell)\le a_\ell(W^\star) r ℓ = rank ( X ℓ ) ≤ a ℓ ( W ⋆ ) . Thus
a ℓ ( W ⋆ ) = r ℓ for every ℓ , a_\ell(W^\star)=r_\ell\qquad\text{for every }\ell, a ℓ ( W ⋆ ) = r ℓ for every ℓ ,
which completes the proof. ■
BMuon with Weight-Decay
We show in this section that, when BMuon is combined with decoupled weight decay, its channel allocations have a direct effect on the learned weights. Consider the BMuon oracle with fixed budget K K K and decoupled weight decay,
W i , t + 1 ℓ = ( 1 − η t λ ) W i , t ℓ − η t Δ i , t ℓ , ⋆ , W_{i,t+1}^{\ell}=(1-\eta_t\lambda)W_{i,t}^{\ell}-\eta_t\Delta_{i,t}^{\ell,\star}, W i , t + 1 ℓ = ( 1 − η t λ ) W i , t ℓ − η t Δ i , t ℓ , ⋆ ,
where λ > 0 \lambda>0 λ > 0 , η t ≥ 0 \eta_t\ge0 η t ≥ 0 , and 0 ≤ λ η t < 1 0\le\lambda\eta_t<1 0 ≤ λ η t < 1 . At each step t t t , let μ t ⋆ ∈ K K \mu_t^\star\in\mathcal{K}_K μ t ⋆ ∈ K K denote a solution of Eq. (4) . Then, by Eq. (5) ,
Δ 1 , t ℓ , ⋆ = D μ ℓ , t ⋆ α Polar ( D μ ℓ , t ⋆ α M 1 , t ℓ ) , Δ 2 , t ℓ , ⋆ = Polar ( M 2 , t ℓ D μ ℓ , t ⋆ α ) D μ ℓ , t ⋆ α . \Delta_{1,t}^{\ell,\star}=D_{\mu_{\ell,t}^\star}^{\alpha}\operatorname{Polar}\left(D_{\mu_{\ell,t}^\star}^{\alpha}M_{1,t}^{\ell}\right),\qquad
\Delta_{2,t}^{\ell,\star}=\operatorname{Polar}\left(M_{2,t}^{\ell}D_{\mu_{\ell,t}^\star}^{\alpha}\right)D_{\mu_{\ell,t}^\star}^{\alpha}. Δ 1 , t ℓ , ⋆ = D μ ℓ , t ⋆ α Polar ( D μ ℓ , t ⋆ α M 1 , t ℓ ) , Δ 2 , t ℓ , ⋆ = Polar ( M 2 , t ℓ D μ ℓ , t ⋆ α ) D μ ℓ , t ⋆ α .
For a channel c = ( ℓ , j ) c=(\ell,j) c = ( ℓ , j ) , write μ c , t : = μ j , t ℓ , ⋆ \mu_{c,t}:=\mu_{j,t}^{\ell,\star} μ c , t := μ j , t ℓ , ⋆ and r c , t : = ( ∥ W 1 , t , j : ℓ ∥ 2 2 + ∥ W 2 , t , : , j ℓ ∥ 2 2 ) 1 / 2 r_{c,t}:=\left(\lVert W_{1,t,j:}^{\ell}\rVert_2^2+\lVert W_{2,t,:,j}^{\ell}\rVert_2^2\right)^{1/2} r c , t := ( ∥ W 1 , t , j : ℓ ∥ 2 2 + ∥ W 2 , t , : , j ℓ ∥ 2 2 ) 1/2 . For 0 ≤ s ≤ T 0\le s\le T 0 ≤ s ≤ T , define D s , T : = ∏ u = s T − 1 ( 1 − λ η u ) D_{s,T}:=\prod_{u=s}^{T-1}(1-\lambda\eta_u) D s , T := ∏ u = s T − 1 ( 1 − λ η u ) , with D T , T : = 1 D_{T,T}:=1 D T , T := 1 .
Theorem 6 (Allocation controls channel norms) Let 0 < α ≤ 1 0<\alpha\le1 0 < α ≤ 1 . For every channel c c c and every T ≥ 1 T\ge1 T ≥ 1 ,
r c , T ≤ D 0 , T r c , 0 + 2 ∑ t = 0 T − 1 η t D t + 1 , T μ c , t α . (15) r_{c,T}\le D_{0,T}r_{c,0}+\sqrt{2}\sum_{t=0}^{T-1}\eta_tD_{t+1,T}\mu_{c,t}^{\alpha}. \tag*{(15)} r c , T ≤ D 0 , T r c , 0 + 2 t = 0 ∑ T − 1 η t D t + 1 , T μ c , t α . (15) Moreover, writing r T : = ( r c , T ) c = 1 N r_T:=(r_{c,T})_{c=1}^{N} r T := ( r c , T ) c = 1 N ,
∥ r T ∥ 1 / α ≤ D 0 , T ∥ r 0 ∥ 1 / α + 2 K α λ ( 1 − D 0 , T ) . (16) \lVert r_T\rVert_{1/\alpha}\le D_{0,T}\lVert r_0\rVert_{1/\alpha}+\frac{\sqrt{2}K^\alpha}{\lambda}(1-D_{0,T}). \tag*{(16)} ∥ r T ∥ 1/ α ≤ D 0 , T ∥ r 0 ∥ 1/ α + λ 2 K α ( 1 − D 0 , T ) . (16) If ∑ t = 0 ∞ η t = ∞ \sum_{t=0}^{\infty}\eta_t=\infty ∑ t = 0 ∞ η t = ∞ , then
lim sup T → ∞ ∥ r T ∥ 1 / α ≤ 2 K α λ . (17) \limsup_{T\to\infty}\lVert r_T\rVert_{1/\alpha}\le\frac{\sqrt{2}K^\alpha}{\lambda}. \tag*{(17)} T → ∞ lim sup ∥ r T ∥ 1/ α ≤ λ 2 K α . (17)
Proof. For c = ( ℓ , j ) c=(\ell,j) c = ( ℓ , j ) , let P 1 , t ℓ = Polar ( D μ t ⋆ α M 1 , t ℓ ) P_{1,t}^{\ell}=\operatorname{Polar}(D_{\mu_t^\star}^{\alpha}M_{1,t}^{\ell}) P 1 , t ℓ = Polar ( D μ t ⋆ α M 1 , t ℓ ) and P 2 , t ℓ = Polar ( M 2 , t ℓ D μ t ⋆ α ) P_{2,t}^{\ell}=\operatorname{Polar}(M_{2,t}^{\ell}D_{\mu_t^\star}^{\alpha}) P 2 , t ℓ = Polar ( M 2 , t ℓ D μ t ⋆ α ) . Since ∥ P 1 , t ℓ ∥ o p ≤ 1 \lVert P_{1,t}^{\ell}\rVert_{\mathrm{op}}\le1 ∥ P 1 , t ℓ ∥ op ≤ 1 and ∥ P 2 , t ℓ ∥ o p ≤ 1 \lVert P_{2,t}^{\ell}\rVert_{\mathrm{op}}\le1 ∥ P 2 , t ℓ ∥ op ≤ 1 ,
∥ Δ 1 , t , j : ℓ , ⋆ ∥ 2 ≤ μ c , t α , ∥ Δ 2 , t , : , j ℓ , ⋆ ∥ 2 ≤ μ c , t α . \lVert\Delta_{1,t,j:}^{\ell,\star}\rVert_2\le\mu_{c,t}^{\alpha},\qquad\lVert\Delta_{2,t,:,j}^{\ell,\star}\rVert_2\le\mu_{c,t}^{\alpha}. ∥ Δ 1 , t , j : ℓ , ⋆ ∥ 2 ≤ μ c , t α , ∥ Δ 2 , t , : , j ℓ , ⋆ ∥ 2 ≤ μ c , t α .
Using the weight-decay update and the triangle inequality,
r c , t + 1 = ( ∥ ( 1 − λ η t ) W 1 , t , j : ℓ − η t Δ 1 , t , j : ℓ , ⋆ ∥ 2 2 + ∥ ( 1 − λ η t ) W 2 , t , : , j ℓ − η t Δ 2 , t , : , j ℓ , ⋆ ∥ 2 2 ) 1 / 2 ≤ ( 1 − λ η t ) ( ∥ W 1 , t , j : ℓ ∥ 2 2 + ∥ W 2 , t , : , j ℓ ∥ 2 2 ) 1 / 2 + η t ( ∥ Δ 1 , t , j : ℓ , ⋆ ∥ 2 2 + ∥ Δ 2 , t , : , j ℓ , ⋆ ∥ 2 2 ) 1 / 2 ≤ ( 1 − λ η t ) r c , t + 2 η t μ c , t α . (18) \begin{aligned}
r_{c,t+1} = \left(\left\|(1-\lambda\eta_t)W^{\ell}_{1,t,j:}-\eta_t\Delta^{\ell,\star}_{1,t,j:}\right\|_2^2+\left\|(1-\lambda\eta_t)W^{\ell}_{2,t,:,j}-\eta_t\Delta^{\ell,\star}_{2,t,:,j}\right\|_2^2\right)^{1/2} \\
&\leq(1-\lambda\eta_t)\left(\left\|W^{\ell}_{1,t,j:}\right\|_2^2+\left\|W^{\ell}_{2,t,:,j}\right\|_2^2\right)^{1/2}+\eta_t\left(\left\|\Delta^{\ell,\star}_{1,t,j:}\right\|_2^2+\left\|\Delta^{\ell,\star}_{2,t,:,j}\right\|_2^2\right)^{1/2} \\
&\leq(1-\lambda\eta_t)r_{c,t}+\sqrt{2}\,\eta_t\mu_{c,t}^{\alpha}. \tag*{(18)}
\end{aligned} r c , t + 1 = ( ( 1 − λ η t ) W 1 , t , j : ℓ − η t Δ 1 , t , j : ℓ , ⋆ 2 2 + ( 1 − λ η t ) W 2 , t , : , j ℓ − η t Δ 2 , t , : , j ℓ , ⋆ 2 2 ) 1/2 ≤ ( 1 − λ η t ) ( W 1 , t , j : ℓ 2 2 + W 2 , t , : , j ℓ 2 2 ) 1/2 + η t ( Δ 1 , t , j : ℓ , ⋆ 2 2 + Δ 2 , t , : , j ℓ , ⋆ 2 2 ) 1/2 ≤ ( 1 − λ η t ) r c , t + 2 η t μ c , t α . (18)
Iterating Eq. (18) gives Eq. (15) .
Set p : = 1 / α p:=1/\alpha p := 1/ α . Applying the ℓ p \ell_p ℓ p norm to the componentwise inequality in Eq. (18) and using the triangle inequality,
∥ r t + 1 ∥ p ≤ ( 1 − λ η t ) ∥ r t ∥ p + 2 η t ∥ μ t α ∥ p . \left\|r_{t+1}\right\|_p\leq(1-\lambda\eta_t)\left\|r_t\right\|_p+\sqrt{2}\,\eta_t\left\|\mu_t^\alpha\right\|_p. ∥ r t + 1 ∥ p ≤ ( 1 − λ η t ) ∥ r t ∥ p + 2 η t ∥ μ t α ∥ p .
Since p = 1 / α p=1/\alpha p = 1/ α and ∑ c μ c , t = K \sum_c\mu_{c,t}=K ∑ c μ c , t = K ,
∥ μ t α ∥ 1 / α = ( ∑ c μ c , t ) α = K α . \left\|\mu_t^\alpha\right\|_{1/\alpha}=\left(\sum_c\mu_{c,t}\right)^\alpha=K^\alpha. ∥ μ t α ∥ 1/ α = ( c ∑ μ c , t ) α = K α .
Iterating this inequality and using
λ ∑ t = 0 T − 1 η t D t + 1 , T = 1 − D 0 , T (19) \lambda\sum_{t=0}^{T-1}\eta_tD_{t+1,T}=1-D_{0,T} \tag*{(19)} λ t = 0 ∑ T − 1 η t D t + 1 , T = 1 − D 0 , T (19)
gives Eq. (16) .
Finally, D 0 , T ≤ exp ( − λ ∑ t = 0 T − 1 η t ) D_{0,T}\leq\exp\left(-\lambda\sum_{t=0}^{T-1}\eta_t\right) D 0 , T ≤ exp ( − λ ∑ t = 0 T − 1 η t ) . Hence ∑ t = 0 ∞ η t = ∞ \sum_{t=0}^{\infty}\eta_t=\infty ∑ t = 0 ∞ η t = ∞ implies D 0 , T → 0 D_{0,T}\to0 D 0 , T → 0 , and Eq. (17) follows.
The channelwise bound also shows that sustained under-allocation suppresses the corresponding channel.
Corollary 7 (Persistent under-allocation). Assume ∑ t = 0 ∞ η t = ∞ \sum_{t=0}^{\infty}\eta_t=\infty ∑ t = 0 ∞ η t = ∞ . If, for some channel c c c , there exist t 0 ≥ 0 t_0\geq0 t 0 ≥ 0 and μ ˉ c ∈ [ 0 , 1 ] \bar{\mu}_c\in[0,1] μ ˉ c ∈ [ 0 , 1 ] such that μ c , t ≤ μ ˉ c \mu_{c,t}\leq\bar{\mu}_c μ c , t ≤ μ ˉ c for every t ≥ t 0 t\geq t_0 t ≥ t 0 , then
lim sup T → ∞ r c , T ≤ 2 λ μ ˉ c α . (20) \limsup_{T\to\infty}r_{c,T}\leq\frac{\sqrt{2}}{\lambda}\bar{\mu}_c^\alpha. \tag*{(20)} T → ∞ lim sup r c , T ≤ λ 2 μ ˉ c α . (20) In particular, if μ c , t → 0 \mu_{c,t}\to0 μ c , t → 0 , then r c , t → 0 r_{c,t}\to0 r c , t → 0 .
Proof. Applying Eq. (15) from t 0 t_0 t 0 gives
r c , T ≤ D t 0 , T r c , t 0 + 2 ∑ t = t 0 T − 1 η t D t + 1 , T μ c , t α . r_{c,T}\leq D_{t_0,T}r_{c,t_0}+\sqrt{2}\sum_{t=t_0}^{T-1}\eta_tD_{t+1,T}\mu_{c,t}^{\alpha}. r c , T ≤ D t 0 , T r c , t 0 + 2 t = t 0 ∑ T − 1 η t D t + 1 , T μ c , t α .
Since μ c , t ≤ μ ˉ c \mu_{c,t}\leq\bar{\mu}_c μ c , t ≤ μ ˉ c for every t ≥ t 0 t\geq t_0 t ≥ t 0 ,
r c , T ≤ D t 0 , T r c , t 0 + 2 μ ˉ c α ∑ t = t 0 T − 1 η t D t + 1 , T . r_{c,T}\leq D_{t_0,T}r_{c,t_0}+\sqrt{2}\,\bar{\mu}_c^\alpha\sum_{t=t_0}^{T-1}\eta_tD_{t+1,T}. r c , T ≤ D t 0 , T r c , t 0 + 2 μ ˉ c α t = t 0 ∑ T − 1 η t D t + 1 , T .
Using Eq. (19) from t 0 t_0 t 0 ,
r c , T ≤ D t 0 , T r c , t 0 + 2 μ ˉ c α λ ( 1 − D t 0 , T ) . r_{c,T} \le D_{t_0,T}r_{c,t_0} + \frac{\sqrt{2}\bar{\mu}_c^\alpha}{\lambda}(1-D_{t_0,T}). r c , T ≤ D t 0 , T r c , t 0 + λ 2 μ ˉ c α ( 1 − D t 0 , T ) .
Since D t 0 , T ≤ exp ( − λ ∑ t = t 0 T − 1 η t ) → 0 D_{t_0,T} \le\exp\left(-\lambda\sum_{t=t_0}^{T-1}\eta_t\right) \to0 D t 0 , T ≤ exp ( − λ ∑ t = t 0 T − 1 η t ) → 0 , we get Eq. (20) . If μ c , t → 0 \mu_{c,t} \to0 μ c , t → 0 , then for every ε > 0 \varepsilon> 0 ε > 0 there exists t 0 t_0 t 0 such that μ c , t α ≤ ε \mu_{c,t}^\alpha\le\varepsilon μ c , t α ≤ ε for every t ≥ t 0 t \ge t_0 t ≥ t 0 . Hence lim sup T → ∞ r c , T ≤ 2 ε / λ \limsup_{T\to\infty} r_{c,T} \le\sqrt{2}\varepsilon/\lambda lim sup T → ∞ r c , T ≤ 2 ε / λ . Since this holds for every ε > 0 \varepsilon> 0 ε > 0 and r c , T ≥ 0 r_{c,T} \ge0 r c , T ≥ 0 , we get r c , T → 0 r_{c,T} \to0 r c , T → 0 . ■ \blacksquare ■
Thus, weight decay makes BMuon’s allocation persistent in parameter space: channel norms are controlled by their discounted allocation histories, implying that channels receiving vanishing allocations are driven toward zero.
MM Allocation Update
This appendix provides the derivations and proofs underlying the MM allocation update of § 4 . Appendix D.1 proves the tight minorization in Eq. (7) and establishes the sign properties of the allocation scores a j ℓ a^\ell_j a j ℓ . Appendix D.2 solves the surrogate maximization problem and derives the closed-form update in Eq. (8) , including the determination of λ μ \lambda_\mu λ μ by bisection. Appendix D.3 proves monotonicity of repeated MM updates and, for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 , convergence in objective value to the global optimum over K K , ε \mathcal{K}_{K,\varepsilon} K K , ε together with global optimality of every accumulation point. Appendix D.4 bounds the effect of replacing the original feasible set K K \mathcal{K}_K K K by K K , ε \mathcal{K}_{K,\varepsilon} K K , ε . Appendix D.5 quantifies the two approximations used in practice: performing a single MM update at each allocation refresh and keeping the allocation fixed between refreshes, and compares the practical update with a certified allocation solve end-to-end. Finally, Appendix D.6 characterizes the computational overhead of the practical BMuon update and reports training-time measurements for ImageNet-1K and 1B-parameter LLaMA.
Throughout this appendix, the momenta M = { M 1 ℓ , M 2 ℓ } ℓ = 1 L M = \{M^\ell_1, M^\ell_2\}_{\ell=1}^{L} M = { M 1 ℓ , M 2 ℓ } ℓ = 1 L are fixed and all polar factors are exact. There are N N N globally budgeted channels, indexed by ( ℓ , j ) (\ell,j) ( ℓ , j ) , and
K K : = { μ ∈ [ 0 , 1 ] N : ∑ ℓ , j μ j ℓ = K } , K K , ε : = { μ ∈ [ ε , 1 ] N : ∑ ℓ , j μ j ℓ = K } , \mathcal{K}_K := \left\{ \mu\in[0,1]^N : \sum_{\ell,j} \mu^\ell_j = K \right\}, \qquad\mathcal{K}_{K,\varepsilon} := \left\{ \mu\in[\varepsilon,1]^N : \sum_{\ell,j} \mu^\ell_j = K \right\}, K K := ⎩ ⎨ ⎧ μ ∈ [ 0 , 1 ] N : ℓ , j ∑ μ j ℓ = K ⎭ ⎬ ⎫ , K K , ε := ⎩ ⎨ ⎧ μ ∈ [ ε , 1 ] N : ℓ , j ∑ μ j ℓ = K ⎭ ⎬ ⎫ ,
where 0 < K < N 0 < K < N 0 < K < N and 0 < ε < K / N 0 < \varepsilon< K/N 0 < ε < K / N . For fixed M M M , the oracle allocation objective of Eq. (4) is
Φ M ( μ ) = ∑ ℓ = 1 L [ ∥ D μ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ℓ α ∥ ∗ ] , 0 < α < 1. \Phi_M(\mu) = \sum_{\ell=1}^{L} \left[ \left\|D^\alpha_{\mu^\ell} M^\ell_1\right\|_* + \left\|M^\ell_2 D^\alpha_{\mu^\ell}\right\|_* \right], \qquad0 < \alpha< 1. Φ M ( μ ) = ℓ = 1 ∑ L [ D μ ℓ α M 1 ℓ ∗ + M 2 ℓ D μ ℓ α ∗ ] , 0 < α < 1.
Given μ ˉ ∈ K K , ε \bar{\mu} \in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε , the MM surrogate is
Q ( μ ∣ μ ˉ ) = ∑ ℓ , j a j ℓ ( μ ˉ ) ( μ j ℓ ) α , Q(\mu\mid\bar{\mu}) = \sum_{\ell,j} a^\ell_j(\bar{\mu})(\mu^\ell_j)^\alpha, Q ( μ ∣ μ ˉ ) = ℓ , j ∑ a j ℓ ( μ ˉ ) ( μ j ℓ ) α ,
where
a j ℓ ( μ ˉ ) = ⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ , a^\ell_j(\bar{\mu}) = \langle M^\ell_{1,:,j}, \bar{P}^\ell_{1,:,j}\rangle+ \langle M^\ell_{2,:,j}, \bar{P}^\ell_{2,:,j}\rangle, a j ℓ ( μ ˉ ) = ⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ ,
with P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) \bar{P}^\ell_1 = \operatorname{Polar}(D^\alpha_{\bar{\mu}^\ell} M^\ell_1) P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) and P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) \bar{P}^\ell_2 = \operatorname{Polar}(M^\ell_2 D^\alpha_{\bar{\mu}^\ell}) P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) . One MM step maximizes Q ( ⋅ ∣ μ ˉ ) Q(\cdot\mid\bar{\mu}) Q ( ⋅ ∣ μ ˉ ) over K K , ε \mathcal{K}_{K,\varepsilon} K K , ε .
Minorization and allocation scores
We first prove the tight minorization in Eq. (7) . Fix an allocation μ ˉ ∈ K K , ε \bar{\mu} \in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε and, for each block, let
P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) , P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) , \bar{P}^\ell_1 = \operatorname{Polar}(D^\alpha_{\bar{\mu}^\ell} M^\ell_1), \qquad\bar{P}^\ell_2 = \operatorname{Polar}(M^\ell_2 D^\alpha_{\bar{\mu}^\ell}), P ˉ 1 ℓ = Polar ( D μ ˉ ℓ α M 1 ℓ ) , P ˉ 2 ℓ = Polar ( M 2 ℓ D μ ˉ ℓ α ) ,
as in § 4 . By spectral–nuclear duality, ∥ A ∥ ∗ = max ∥ P ∥ o p ≤ 1 ⟨ A , P ⟩ \|A\|_* = \max_{\|P\|_{\mathrm{op}}\le1}\langle A, P\rangle ∥ A ∥ ∗ = max ∥ P ∥ op ≤ 1 ⟨ A , P ⟩ . We will use the following equality characterization both here and in the convergence analysis.
Lemma 8 Let ∥ P ∥ o p ≤ 1 \lVert P\rVert_{\mathrm{op}} \le1 ∥ P ∥ op ≤ 1 . If ⟨ A , P ⟩ = ∥ A ∥ ∗ \langle A,P\rangle=\lVert A\rVert_{*} ⟨ A , P ⟩ = ∥ A ∥ ∗ , then
A P T = ( A A T ) 1 / 2 , P T A = ( A T A ) 1 / 2 . AP^{\mathsf{T}}=(AA^{\mathsf{T}})^{1/2},\qquad P^{\mathsf{T}}A=(A^{\mathsf{T}}A)^{1/2}. A P T = ( A A T ) 1/2 , P T A = ( A T A ) 1/2 . Proof Let A = U Σ V T A=U\Sigma V^{\mathsf{T}} A = U Σ V T be a compact SVD. Then
⟨ A , P ⟩ = ∑ r σ r u r T P v r , ∣ u r T P v r ∣ ≤ 1. \langle A,P\rangle=\sum_{r}\sigma_{r}u_{r}^{\mathsf{T}}Pv_{r},\qquad\lvert u_{r}^{\mathsf{T}}Pv_{r}\rvert\le1. ⟨ A , P ⟩ = r ∑ σ r u r T P v r , ∣ u r T P v r ∣ ≤ 1. Since σ r > 0 \sigma_{r}>0 σ r > 0 , equality ⟨ A , P ⟩ = ∑ r σ r \langle A,P\rangle=\sum_{r}\sigma_{r} ⟨ A , P ⟩ = ∑ r σ r implies u r T P v r = 1 u_{r}^{\mathsf{T}}Pv_{r}=1 u r T P v r = 1 for every r r r . Moreover, ∥ P v r ∥ 2 ≤ 1 \lVert Pv_{r}\rVert_{2}\le1 ∥ P v r ∥ 2 ≤ 1 , so equality in Cauchy–Schwarz gives P v r = u r Pv_{r}=u_{r} P v r = u r . Similarly, ∥ P T u r ∥ 2 ≤ 1 \lVert P^{\mathsf{T}}u_{r}\rVert_{2}\le1 ∥ P T u r ∥ 2 ≤ 1 gives P T u r = v r P^{\mathsf{T}}u_{r}=v_{r} P T u r = v r . Hence P V = U PV=U P V = U and P T U = V P^{\mathsf{T}}U=V P T U = V , and therefore
A P T = U Σ U T = ( A A T ) 1 / 2 , P T A = V Σ V T = ( A T A ) 1 / 2 . AP^{\mathsf{T}}=U\Sigma U^{\mathsf{T}}=(AA^{\mathsf{T}})^{1/2},\qquad P^{\mathsf{T}}A=V\Sigma V^{\mathsf{T}}=(A^{\mathsf{T}}A)^{1/2}. A P T = U Σ U T = ( A A T ) 1/2 , P T A = V Σ V T = ( A T A ) 1/2 .
Since ∥ P ˉ 1 ℓ ∥ o p ≤ 1 \lVert\bar{P}^{\ell}_{1}\rVert_{\mathrm{op}}\le1 ∥ P ˉ 1 ℓ ∥ op ≤ 1 and ∥ P ˉ 2 ℓ ∥ o p ≤ 1 \lVert\bar{P}^{\ell}_{2}\rVert_{\mathrm{op}}\le1 ∥ P ˉ 2 ℓ ∥ op ≤ 1 , for any allocation μ \mu μ ,
Φ M ( μ ) = ∑ ℓ = 1 L [ ∥ D μ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ℓ α ∥ ∗ ] ≥ ∑ ℓ = 1 L [ ⟨ D μ ℓ α M 1 ℓ , P ˉ 1 ℓ ⟩ + ⟨ M 2 ℓ D μ ℓ α , P ˉ 2 ℓ ⟩ ] = ∑ ℓ , j ( ⟨ M 1 , j , : ℓ , P ˉ 1 , j , : ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ ) ( μ j ℓ ) α = Q ( μ ∣ μ ˉ ) . (21) \begin{aligned}
\Phi_{M}(\mu)
&=\sum_{\ell=1}^{L}\left[\lVert D_{\mu^{\ell}}^{\alpha}M_{1}^{\ell}\rVert_{*}+\lVert M_{2}^{\ell}D_{\mu^{\ell}}^{\alpha}\rVert_{*}\right]\\
&\ge\sum_{\ell=1}^{L}\left[\left\langle D_{\mu^{\ell}}^{\alpha}M_{1}^{\ell},\bar{P}_{1}^{\ell}\right\rangle+\left\langle M_{2}^{\ell}D_{\mu^{\ell}}^{\alpha},\bar{P}_{2}^{\ell}\right\rangle\right]\\
&=\sum_{\ell,j}\left(\left\langle M_{1,j,:}^{\ell},\bar{P}_{1,j,:}^{\ell}\right\rangle+\left\langle M_{2,:,j}^{\ell},\bar{P}_{2,:,j}^{\ell}\right\rangle\right)(\mu_{j}^{\ell})^{\alpha}=Q(\mu\mid\bar{\mu}). \tag*{(21)}
\end{aligned} Φ M ( μ ) = ℓ = 1 ∑ L [ ∥ D μ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ℓ α ∥ ∗ ] ≥ ℓ = 1 ∑ L [ ⟨ D μ ℓ α M 1 ℓ , P ˉ 1 ℓ ⟩ + ⟨ M 2 ℓ D μ ℓ α , P ˉ 2 ℓ ⟩ ] = ℓ , j ∑ ( ⟨ M 1 , j , : ℓ , P ˉ 1 , j , : ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ ) ( μ j ℓ ) α = Q ( μ ∣ μ ˉ ) . (21)
At μ = μ ˉ \mu=\bar{\mu} μ = μ ˉ , using ⟨ A , Polar ( A ) ⟩ = ∥ A ∥ ∗ \langle A,\operatorname{Polar}(A)\rangle=\lVert A\rVert_{*} ⟨ A , Polar ( A )⟩ = ∥ A ∥ ∗ , Q ( μ ˉ ∣ μ ˉ ) = ∑ ℓ = 1 L [ ∥ D μ ˉ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ˉ ℓ α ∥ ∗ ] = Φ M ( μ ˉ ) Q(\bar{\mu}\mid\bar{\mu})=\sum_{\ell=1}^{L}\left[\lVert D_{\bar{\mu}^{\ell}}^{\alpha}M_{1}^{\ell}\rVert_{*}+\lVert M_{2}^{\ell}D_{\bar{\mu}^{\ell}}^{\alpha}\rVert_{*}\right]=\Phi_{M}(\bar{\mu}) Q ( μ ˉ ∣ μ ˉ ) = ∑ ℓ = 1 L [ ∥ D μ ˉ ℓ α M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ ˉ ℓ α ∥ ∗ ] = Φ M ( μ ˉ ) . Hence the minorization is tight, proving Eq. (7) .
We now establish the sign properties of the scores.
Fix ( ℓ , j ) (\ell,j) ( ℓ , j ) and consider first the row-side term. Write A = D μ ˉ ℓ α M 1 ℓ A=D_{\bar{\mu}^{\ell}}^{\alpha}M_{1}^{\ell} A = D μ ˉ ℓ α M 1 ℓ , P ˉ 1 ℓ = Polar ( A ) \bar{P}_{1}^{\ell}=\operatorname{Polar}(A) P ˉ 1 ℓ = Polar ( A ) , and S = ( A A T ) 1 / 2 S=(AA^{\mathsf{T}})^{1/2} S = ( A A T ) 1/2 . Since P ˉ 1 ℓ \bar{P}_{1}^{\ell} P ˉ 1 ℓ attains the spectral–nuclear dual maximum, Lemma 8 gives A ( P ˉ 1 ℓ ) T = S ⪰ 0 A(\bar{P}_{1}^{\ell})^{\mathsf{T}}=S\succeq0 A ( P ˉ 1 ℓ ) T = S ⪰ 0 ,
S j j = ( μ ˉ j ℓ ) α ⟨ M 1 , j , : ℓ , P ˉ 1 , j , : ℓ ⟩ ≥ 0. S_{jj}=(\bar{\mu}_{j}^{\ell})^{\alpha}\left\langle M_{1,j,:}^{\ell},\bar{P}_{1,j,:}^{\ell}\right\rangle\ge0. S j j = ( μ ˉ j ℓ ) α ⟨ M 1 , j , : ℓ , P ˉ 1 , j , : ℓ ⟩ ≥ 0.
Since μ ˉ ∈ K K , ε \bar{\mu}\in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε , we have μ ˉ j ℓ ≥ ε > 0 \bar{\mu}_{j}^{\ell}\ge\varepsilon>0 μ ˉ j ℓ ≥ ε > 0 . Applying the same argument to the column-side term gives
⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ = [ ( D μ ˉ ℓ α ( M 1 ℓ ) T M 1 ℓ D μ ˉ ℓ α ) 1 / 2 ] j j ( μ ˉ j ℓ ) α , \left\langle M_{1,:,j}^{\ell},\bar{P}_{1,:,j}^{\ell}\right\rangle
=
\frac{\left[\left(D_{\bar{\mu}^{\ell}}^{\alpha}(M_{1}^{\ell})^{\mathsf{T}}M_{1}^{\ell}D_{\bar{\mu}^{\ell}}^{\alpha}\right)^{1/2}\right]_{jj}}{(\bar{\mu}_{j}^{\ell})^{\alpha}}, ⟨ M 1 , : , j ℓ , P ˉ 1 , : , j ℓ ⟩ = ( μ ˉ j ℓ ) α [ ( D μ ˉ ℓ α ( M 1 ℓ ) T M 1 ℓ D μ ˉ ℓ α ) 1/2 ] j j ,
⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ = [ ( D μ ˉ ℓ α ( M 2 ℓ ) T M 2 ℓ D μ ˉ ℓ α ) 1 / 2 ] j j ( μ ˉ j ℓ ) α . (22) \left\langle M_{2,:,j}^{\ell},\bar{P}_{2,:,j}^{\ell}\right\rangle
=
\frac{\left[\left(D_{\bar{\mu}^{\ell}}^{\alpha}(M_{2}^{\ell})^{\mathsf{T}}M_{2}^{\ell}D_{\bar{\mu}^{\ell}}^{\alpha}\right)^{1/2}\right]_{jj}}{(\bar{\mu}_{j}^{\ell})^{\alpha}}. \tag*{(22)} ⟨ M 2 , : , j ℓ , P ˉ 2 , : , j ℓ ⟩ = ( μ ˉ j ℓ ) α [ ( D μ ˉ ℓ α ( M 2 ℓ ) T M 2 ℓ D μ ˉ ℓ α ) 1/2 ] j j . (22)
Both numerators in Eq. (22) are diagonal entries of positive semidefinite matrices. Hence both score contributions are nonnegative, and therefore
a j ℓ = ⟨ M 1 , j : ℓ , P ‾ 1 , j : ℓ ⟩ + ⟨ M 2 , : , j ℓ , P ‾ 2 , : , j ℓ ⟩ ≥ 0. a_j^\ell= \langle M_{1,j:}^\ell,\overline{P}_{1,j:}^\ell\rangle+ \langle M_{2,:,j}^\ell,\overline{P}_{2,:,j}^\ell\rangle\ge0. a j ℓ = ⟨ M 1 , j : ℓ , P 1 , j : ℓ ⟩ + ⟨ M 2 , : , j ℓ , P 2 , : , j ℓ ⟩ ≥ 0.
We next establish strict positivity when all channels are nontrivial. If M 1 , j : ℓ ≠ 0 M_{1,j:}^\ell\ne0 M 1 , j : ℓ = 0 , then
( S 2 ) j j = ( A A ⊤ ) j j = ( μ ˉ j ℓ ) 2 α ∥ M 1 , j : ℓ ∥ 2 2 > 0. (S^2)_{jj} = (AA^\top)_{jj} = (\bar{\mu}_j^\ell)^{2\alpha}\lVert M_{1,j:}^\ell\rVert_2^2 > 0. ( S 2 ) j j = ( A A ⊤ ) j j = ( μ ˉ j ℓ ) 2 α ∥ M 1 , j : ℓ ∥ 2 2 > 0.
Since S S S is symmetric positive semidefinite, S 2 ⪯ ∥ S ∥ op S S^2 \preceq\lVert S\rVert_{\operatorname{op}}S S 2 ⪯ ∥ S ∥ op S . Hence ( S 2 ) j j ≤ ∥ S ∥ op S j j (S^2)_{jj} \le\lVert S\rVert_{\operatorname{op}}S_{jj} ( S 2 ) j j ≤ ∥ S ∥ op S j j , so ( S 2 ) j j > 0 (S^2)_{jj} > 0 ( S 2 ) j j > 0 implies S j j > 0 S_{jj} > 0 S j j > 0 . Since μ ˉ j ℓ > 0 \bar{\mu}_j^\ell> 0 μ ˉ j ℓ > 0 , we obtain ⟨ M 1 , j : ℓ , P ‾ 1 , j : ℓ ⟩ > 0 \langle M_{1,j:}^\ell,\overline{P}_{1,j:}^\ell\rangle> 0 ⟨ M 1 , j : ℓ , P 1 , j : ℓ ⟩ > 0 .
The same argument applied to the column-side term shows that M 2 , : , j ℓ ≠ 0 M_{2,:,j}^\ell\ne0 M 2 , : , j ℓ = 0 implies ⟨ M 2 , : , j ℓ , P ‾ 2 , : , j ℓ ⟩ > 0 \langle M_{2,:,j}^\ell,\overline{P}_{2,:,j}^\ell\rangle> 0 ⟨ M 2 , : , j ℓ , P 2 , : , j ℓ ⟩ > 0 . Therefore, if M 1 , j : ℓ ≠ 0 M_{1,j:}^\ell\ne0 M 1 , j : ℓ = 0 or M 2 , : , j ℓ ≠ 0 M_{2,:,j}^\ell\ne0 M 2 , : , j ℓ = 0 , then a j ℓ > 0 a_j^\ell> 0 a j ℓ > 0 . In particular, under the nontrivial-channel condition (i.e., M 1 , j : ℓ ≠ 0 M_{1,j:}^\ell\ne0 M 1 , j : ℓ = 0 or M 2 , : , j ℓ ≠ 0 M_{2,:,j}^\ell\ne0 M 2 , : , j ℓ = 0 for every ( ℓ , j ) (\ell,j) ( ℓ , j ) ), all scores are strictly positive for every μ ˉ ∈ K K , ε \bar{\mu} \in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε .
Closed-form surrogate maximization
We now derive the closed-form update in Eq. (8) . Under the nontrivial-channel condition, Appendix D.1 gives a j ℓ > 0 a_j^\ell> 0 a j ℓ > 0 for every ( ℓ , j ) (\ell,j) ( ℓ , j ) . Since 0 < α < 1 0 < \alpha< 1 0 < α < 1 , each function μ j ℓ ↦ a j ℓ ( μ j ℓ ) α \mu_j^\ell\mapsto a_j^\ell(\mu_j^\ell)^\alpha μ j ℓ ↦ a j ℓ ( μ j ℓ ) α is strictly concave on [ ε , 1 ] [\varepsilon,1] [ ε , 1 ] . Hence Q ( μ ∣ μ ˉ ) Q(\mu\mid\bar{\mu}) Q ( μ ∣ μ ˉ ) is strictly concave, and its maximization over the convex set K K , ε \mathcal{K}_{K,\varepsilon} K K , ε has a unique maximizer, which we denote by μ + \mu^+ μ + .
Since 0 < ε < K / N < 1 0 < \varepsilon< K/N < 1 0 < ε < K / N < 1 , the uniform allocation μ j ℓ = K / N \mu_j^\ell= K/N μ j ℓ = K / N satisfies the budget constraint and both bound constraints strictly. Hence Slater’s condition holds, and the KKT conditions are necessary and sufficient. Introduce a multiplier λ μ \lambda_\mu λ μ for the budget constraint and multipliers ρ j ℓ , σ j ℓ ≥ 0 \rho_j^\ell,\sigma_j^\ell\ge0 ρ j ℓ , σ j ℓ ≥ 0 for the lower and upper bounds, respectively. The stationarity and complementary-slackness conditions are
α a j ℓ ( μ j ℓ ) α − 1 − λ μ + ρ j ℓ − σ j ℓ = 0 , (23) \alpha a_j^\ell(\mu_j^\ell)^{\alpha-1} - \lambda_\mu+ \rho_j^\ell- \sigma_j^\ell= 0, \tag*{(23)} α a j ℓ ( μ j ℓ ) α − 1 − λ μ + ρ j ℓ − σ j ℓ = 0 , (23)
ρ j ℓ ( μ j ℓ − ε ) = 0 , σ j ℓ ( 1 − μ j ℓ ) = 0. (24) \rho_j^\ell(\mu_j^\ell-\varepsilon)=0,\qquad\sigma_j^\ell(1-\mu_j^\ell)=0. \tag*{(24)} ρ j ℓ ( μ j ℓ − ε ) = 0 , σ j ℓ ( 1 − μ j ℓ ) = 0. (24)
Because K < N K < N K < N , at least one coordinate of any feasible allocation is strictly smaller than 1. For such a coordinate σ j ℓ = 0 \sigma_j^\ell=0 σ j ℓ = 0 , and Eq. (23) implies λ μ > 0 \lambda_\mu> 0 λ μ > 0 .
For a fixed ( ℓ , j ) (\ell,j) ( ℓ , j ) , define
μ ^ j ℓ = ( α a j ℓ λ μ ) 1 / ( 1 − α ) . \hat{\mu}_j^\ell= \left(\frac{\alpha a_j^\ell}{\lambda_\mu}\right)^{1/(1-\alpha)}. μ ^ j ℓ = ( λ μ α a j ℓ ) 1/ ( 1 − α ) .
If ε < μ j ℓ < 1 \varepsilon< \mu_j^\ell< 1 ε < μ j ℓ < 1 , both bound multipliers vanish and μ j ℓ = μ ^ j ℓ \mu_j^\ell=\hat{\mu}_j^\ell μ j ℓ = μ ^ j ℓ . If μ j ℓ = ε \mu_j^\ell=\varepsilon μ j ℓ = ε , then σ j ℓ = 0 \sigma_j^\ell=0 σ j ℓ = 0 and stationarity gives λ μ ≥ α a j ℓ ε α − 1 \lambda_\mu\ge\alpha a_j^\ell\varepsilon^{\alpha-1} λ μ ≥ α a j ℓ ε α − 1 , equivalently μ ^ j ℓ ≤ ε \hat{\mu}_j^\ell\le\varepsilon μ ^ j ℓ ≤ ε . Similarly, if μ j ℓ = 1 \mu_j^\ell=1 μ j ℓ = 1 , then ρ j ℓ = 0 \rho_j^\ell=0 ρ j ℓ = 0 and stationarity gives λ μ ≤ α a j ℓ \lambda_\mu\le\alpha a_j^\ell λ μ ≤ α a j ℓ , equivalently μ ^ j ℓ ≥ 1 \hat{\mu}_j^\ell\ge1 μ ^ j ℓ ≥ 1 . Hence the unique maximizer is
( μ j ℓ ) + = clip [ ε , 1 ] [ ( α a j ℓ λ μ ) 1 / ( 1 − α ) ] , (\mu_j^\ell)^+ = \operatorname{clip}_{[\varepsilon,1]}\left[\left(\frac{\alpha a_j^\ell}{\lambda_\mu}\right)^{1/(1-\alpha)}\right], ( μ j ℓ ) + = clip [ ε , 1 ] ( λ μ α a j ℓ ) 1/ ( 1 − α ) ,
which proves Eq. (8) .
It remains to determine λ μ \lambda_\mu λ μ . Define
H ( λ ) : = ∑ ℓ , j clip [ ε , 1 ] [ ( α a j ℓ λ ) 1 / ( 1 − α ) ] , λ > 0. H(\lambda) := \sum_{\ell,j} \operatorname{clip}_{[\varepsilon,1]}\left[\left(\frac{\alpha a_j^\ell}{\lambda}\right)^{1/(1-\alpha)}\right], \qquad\lambda> 0. H ( λ ) := ℓ , j ∑ clip [ ε , 1 ] ( λ α a j ℓ ) 1/ ( 1 − α ) , λ > 0.
Since every a j ℓ > 0 a_j^\ell> 0 a j ℓ > 0 , H H H is continuous and nonincreasing, with H ( λ ) → N H(\lambda) \to N H ( λ ) → N as λ ↓ 0 \lambda\downarrow0 λ ↓ 0 and H ( λ ) → N ε H(\lambda) \to N\varepsilon H ( λ ) → N ε as λ ↑ ∞ \lambda\uparrow\infty λ ↑ ∞ . Because N ε < K < N N\varepsilon< K < N N ε < K < N , there exists λ μ > 0 \lambda_\mu> 0 λ μ > 0 such that H ( λ μ ) = K H(\lambda_\mu) = K H ( λ μ ) = K , and such a value can be found by bisection. This completes the derivation of the MM allocation update.
Convergence of repeated MM updates
We now prove the convergence statement of § 4 .
Theorem 9 (Convergence of repeated MM updates). Assume the nontrivial-channel condition (i.e., M 1 , j ℓ ≠ 0 M_{1,j}^\ell\ne0 M 1 , j ℓ = 0 or M 2 , : , j ℓ ≠ 0 M_{2,:,j}^\ell\ne0 M 2 , : , j ℓ = 0 for every ( ℓ , j ) (\ell,j) ( ℓ , j ) ), and let 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 . Starting from any μ ( 0 ) ∈ K K , ε \mu^{(0)} \in\mathcal{K}_{K,\varepsilon} μ ( 0 ) ∈ K K , ε , define
μ ( t + 1 ) = arg max μ ∈ K K , ε Q ( μ ∣ μ ( t ) ) . \mu^{(t+1)} = \underset{\mu\in\mathcal{K}_{K,\varepsilon}}{\arg\max}\ Q(\mu\mid\mu^{(t)}). μ ( t + 1 ) = μ ∈ K K , ε arg max Q ( μ ∣ μ ( t ) ) . Then
Φ M ( μ ( t ) ) ⟶ Φ M ⋆ : = max μ ∈ K K , ε Φ M ( μ ) , \Phi_M(\mu^{(t)}) \longrightarrow\Phi_M^\star:= \max_{\mu\in\mathcal{K}_{K,\varepsilon}} \Phi_M(\mu), Φ M ( μ ( t ) ) ⟶ Φ M ⋆ := μ ∈ K K , ε max Φ M ( μ ) , and every accumulation point of { μ ( t ) } t ≥ 0 \{\mu^{(t)}\}_{t\ge0} { μ ( t ) } t ≥ 0 is a global maximizer of Φ M \Phi_M Φ M over K K , ε \mathcal{K}_{K,\varepsilon} K K , ε .
Proof. By Appendix D.1, the surrogate is a tight minorizer: Q ( μ ∣ μ ˉ ) ≤ Φ M ( μ ) Q(\mu\mid\bar{\mu}) \le\Phi_M(\mu) Q ( μ ∣ μ ˉ ) ≤ Φ M ( μ ) for every μ \mu μ , with equality at μ = μ ˉ \mu= \bar{\mu} μ = μ ˉ . Since μ ( t + 1 ) \mu^{(t+1)} μ ( t + 1 ) maximizes Q ( ⋅ ∣ μ ( t ) ) Q(\cdot\mid\mu^{(t)}) Q ( ⋅ ∣ μ ( t ) ) ,
Φ M ( μ ( t + 1 ) ) ≥ Q ( μ ( t + 1 ) ∣ μ ( t ) ) ≥ Q ( μ ( t ) ∣ μ ( t ) ) = Φ M ( μ ( t ) ) . \Phi_M(\mu^{(t+1)}) \ge Q(\mu^{(t+1)} \mid\mu^{(t)}) \ge Q(\mu^{(t)} \mid\mu^{(t)}) = \Phi_M(\mu^{(t)}). Φ M ( μ ( t + 1 ) ) ≥ Q ( μ ( t + 1 ) ∣ μ ( t ) ) ≥ Q ( μ ( t ) ∣ μ ( t ) ) = Φ M ( μ ( t ) ) .
Hence ( Φ M ( μ ( t ) ) ) t (\Phi_M(\mu^{(t)}))_t ( Φ M ( μ ( t ) ) ) t is nondecreasing. Since K K , ε \mathcal{K}_{K,\varepsilon} K K , ε is compact and Φ M \Phi_M Φ M is continuous, Φ M ( μ ( t ) ) \Phi_M(\mu^{(t)}) Φ M ( μ ( t ) ) converges.
Define the MM map T ( μ ˉ ) : = arg max μ ∈ K K , ε Q ( μ ∣ μ ˉ ) T(\bar{\mu}) := \underset{\mu\in\mathcal{K}_{K,\varepsilon}}{\arg\max}\ Q(\mu\mid\bar{\mu}) T ( μ ˉ ) := μ ∈ K K , ε arg max Q ( μ ∣ μ ˉ ) . By Appendix D.2, under the nontrivial-channel condition the surrogate has a unique maximizer. Thus T T T is well defined. We next show that T T T is continuous. By Eq. (22) ,
a j ℓ ( μ ˉ ) = [ ( D μ ˉ ℓ α M 1 ℓ ( M 1 ℓ ) T D μ ˉ ℓ α ) 1 / 2 ] j j ( μ ˉ j ℓ ) α + [ ( D μ ˉ ℓ α ( M 2 ℓ ) T M 2 ℓ D μ ˉ ℓ α ) 1 / 2 ] j j ( μ ˉ j ℓ ) α . a_j^\ell(\bar{\mu}) =
\frac{\left[\left(D_{\bar{\mu}^\ell}^{\alpha}M_1^\ell(M_1^\ell)^{\mathsf T}D_{\bar{\mu}^\ell}^{\alpha}\right)^{1/2}\right]_{jj}}{(\bar{\mu}_j^\ell)^\alpha}
+
\frac{\left[\left(D_{\bar{\mu}^\ell}^{\alpha}(M_2^\ell)^{\mathsf T}M_2^\ell D_{\bar{\mu}^\ell}^{\alpha}\right)^{1/2}\right]_{jj}}{(\bar{\mu}_j^\ell)^\alpha}. a j ℓ ( μ ˉ ) = ( μ ˉ j ℓ ) α [ ( D μ ˉ ℓ α M 1 ℓ ( M 1 ℓ ) T D μ ˉ ℓ α ) 1/2 ] j j + ( μ ˉ j ℓ ) α [ ( D μ ˉ ℓ α ( M 2 ℓ ) T M 2 ℓ D μ ˉ ℓ α ) 1/2 ] j j .
Since μ ˉ ∈ K K , ε \bar{\mu} \in\mathcal{K}_{K,\varepsilon} μ ˉ ∈ K K , ε , we have μ ˉ j ℓ ≥ ε > 0 \bar{\mu}_j^\ell\ge\varepsilon> 0 μ ˉ j ℓ ≥ ε > 0 . Moreover, μ ˉ ↦ D μ ˉ ℓ α \bar{\mu} \mapsto D_{\bar{\mu}^\ell}^{\alpha} μ ˉ ↦ D μ ˉ ℓ α is continuous, and the matrix square root is continuous on the set of positive semidefinite matrices. Therefore μ ˉ ↦ a j ℓ ( μ ˉ ) \bar{\mu} \mapsto a_j^\ell(\bar{\mu}) μ ˉ ↦ a j ℓ ( μ ˉ ) is continuous for every ( ℓ , j ) (\ell,j) ( ℓ , j ) . Hence ( μ , μ ˉ ) ↦ Q ( μ ∣ μ ˉ ) = ∑ ℓ , j a j ℓ ( μ ˉ ) ( μ j ℓ ) α (\mu,\bar{\mu}) \mapsto Q(\mu\mid\bar{\mu}) = \sum_{\ell,j} a_j^\ell(\bar{\mu})(\mu_j^\ell)^\alpha ( μ , μ ˉ ) ↦ Q ( μ ∣ μ ˉ ) = ∑ ℓ , j a j ℓ ( μ ˉ ) ( μ j ℓ ) α is continuous on K K , ε × K K , ε \mathcal{K}_{K,\varepsilon} \times\mathcal{K}_{K,\varepsilon} K K , ε × K K , ε . Since K K , ε \mathcal{K}_{K,\varepsilon} K K , ε is compact and Q ( ⋅ ∣ μ ˉ ) Q(\cdot\mid\bar{\mu}) Q ( ⋅ ∣ μ ˉ ) has a unique maximizer, Berge’s maximum theorem, together with uniqueness of the maximizer, implies that T T T is continuous. Since K K , ε \mathcal{K}_{K,\varepsilon} K K , ε is compact, the sequence { μ ( t ) } \{\mu^{(t)}\} { μ ( t ) } has at least one accumulation point. Let μ ⋆ \mu^\star μ ⋆ be any accumulation point. There exists a subsequence μ ( t k ) → μ ⋆ \mu^{(t_k)} \to\mu^\star μ ( t k ) → μ ⋆ . Since Φ M ( μ ( t ) ) \Phi_M(\mu^{(t)}) Φ M ( μ ( t ) ) converges,
Φ M ( μ ( t k + 1 ) ) − Φ M ( μ ( t k ) ) ⟶ 0. \Phi_M(\mu^{(t_k+1)})-\Phi_M(\mu^{(t_k)}) \longrightarrow0. Φ M ( μ ( t k + 1 ) ) − Φ M ( μ ( t k ) ) ⟶ 0.
Using μ ( t k + 1 ) = T ( μ ( t k ) ) \mu^{(t_k+1)}=T(\mu^{(t_k)}) μ ( t k + 1 ) = T ( μ ( t k ) ) and continuity of T T T and Φ M \Phi_M Φ M gives Φ M ( T ( μ ⋆ ) ) = Φ M ( μ ⋆ ) \Phi_M(T(\mu^\star))=\Phi_M(\mu^\star) Φ M ( T ( μ ⋆ )) = Φ M ( μ ⋆ ) .
Suppose that T ( μ ⋆ ) ≠ μ ⋆ T(\mu^\star)\ne\mu^\star T ( μ ⋆ ) = μ ⋆ . Since T ( μ ⋆ ) T(\mu^\star) T ( μ ⋆ ) is the unique maximizer of Q ( ⋅ ∣ μ ⋆ ) Q(\cdot\mid\mu^\star) Q ( ⋅ ∣ μ ⋆ ) , Q ( T ( μ ⋆ ) ∣ μ ⋆ ) > Q ( μ ⋆ ∣ μ ⋆ ) Q(T(\mu^\star)\mid\mu^\star)>Q(\mu^\star\mid\mu^\star) Q ( T ( μ ⋆ ) ∣ μ ⋆ ) > Q ( μ ⋆ ∣ μ ⋆ ) . By minorization and tightness,
Φ M ( T ( μ ⋆ ) ) ≥ Q ( T ( μ ⋆ ) ∣ μ ⋆ ) > Q ( μ ⋆ ∣ μ ⋆ ) = Φ M ( μ ⋆ ) , \Phi_M(T(\mu^\star))\ge Q(T(\mu^\star)\mid\mu^\star)>Q(\mu^\star\mid\mu^\star)=\Phi_M(\mu^\star), Φ M ( T ( μ ⋆ )) ≥ Q ( T ( μ ⋆ ) ∣ μ ⋆ ) > Q ( μ ⋆ ∣ μ ⋆ ) = Φ M ( μ ⋆ ) ,
which is a contradiction. Therefore T ( μ ⋆ ) = μ ⋆ T(\mu^\star)=\mu^\star T ( μ ⋆ ) = μ ⋆ .
It remains to show that every fixed point is globally optimal. We first show first-order tangency in directional derivatives.
For a function F F F , write F ′ ( x ; d ) : = lim s ↓ 0 [ F ( x + s d ) − F ( x ) ] / s F'(x;d):=\lim_{s\downarrow0}[F(x+sd)-F(x)]/s F ′ ( x ; d ) := lim s ↓ 0 [ F ( x + s d ) − F ( x )] / s for its one-sided directional derivative.
By spectral–nuclear duality, ∥ D μ ℓ α M 1 ℓ ∥ ∗ = max ∥ P ∥ o p ≤ 1 ⟨ D μ ℓ α M 1 ℓ , P ⟩ \lVert D^\alpha_{\mu^\ell}M^\ell_1\rVert_*=\max_{\lVert P\rVert_{\mathrm{op}}\le1}\langle D^\alpha_{\mu^\ell}M^\ell_1,P\rangle ∥ D μ ℓ α M 1 ℓ ∥ ∗ = max ∥ P ∥ op ≤ 1 ⟨ D μ ℓ α M 1 ℓ , P ⟩ . For fixed P P P , the directional derivative at μ ˉ \bar\mu μ ˉ in direction d d d is
∑ j α ( μ ˉ j ℓ ) α − 1 d j ℓ ⟨ M 1 , j : ℓ , P j : ⟩ . \sum_j\alpha(\bar\mu^\ell_j)^{\alpha-1}d^\ell_j\langle M^\ell_{1,j:},P_{j:}\rangle. j ∑ α ( μ ˉ j ℓ ) α − 1 d j ℓ ⟨ M 1 , j : ℓ , P j : ⟩ .
Let A = D μ ˉ α M 1 ℓ A=D^\alpha_{\bar\mu}M^\ell_1 A = D μ ˉ α M 1 ℓ . By Lemma 8, every maximizing P P P satisfies A P T = ( A A T ) 1 / 2 AP^\mathsf{T}=(AA^\mathsf{T})^{1/2} A P T = ( A A T ) 1/2 , so ( μ ˉ j ℓ ) α ⟨ M 1 , j : ℓ , P j : ⟩ = [ ( A A T ) 1 / 2 ] j j (\bar\mu^\ell_j)^\alpha\langle M^\ell_{1,j:},P_{j:}\rangle=[(AA^\mathsf{T})^{1/2}]_{jj} ( μ ˉ j ℓ ) α ⟨ M 1 , j : ℓ , P j : ⟩ = [( A A T ) 1/2 ] j j . Thus the directional derivative above is the same for every maximizing P P P . Since the operator-norm unit ball is compact and the inner objective is continuously differentiable in μ \mu μ on the positive orthant, Danskin’s theorem, together with the identical argument for the column-side term, gives
Φ M ′ ( μ ˉ ; d ) = ∑ ℓ , j α a j ℓ ( μ ˉ ) ( μ ˉ j ℓ ) α − 1 d j ℓ = Q ′ ( μ ˉ ∣ μ ˉ ; d ) . \Phi'_M(\bar\mu;d)=\sum_{\ell,j}\alpha a^\ell_j(\bar\mu)(\bar\mu^\ell_j)^{\alpha-1}d^\ell_j=Q'(\bar\mu\mid\bar\mu;d). Φ M ′ ( μ ˉ ; d ) = ℓ , j ∑ α a j ℓ ( μ ˉ ) ( μ ˉ j ℓ ) α − 1 d j ℓ = Q ′ ( μ ˉ ∣ μ ˉ ; d ) .
Since T ( μ ⋆ ) = μ ⋆ T(\mu^\star)=\mu^\star T ( μ ⋆ ) = μ ⋆ , for every μ ∈ K K , ε \mu\in\mathcal{K}_{K,\varepsilon} μ ∈ K K , ε ,
Q ′ ( μ ⋆ ∣ μ ⋆ ; μ − μ ⋆ ) ≤ 0. Q'(\mu^\star\mid\mu^\star;\mu-\mu^\star)\le0. Q ′ ( μ ⋆ ∣ μ ⋆ ; μ − μ ⋆ ) ≤ 0.
By first-order tangency, Φ M ′ ( μ ⋆ ; μ − μ ⋆ ) ≤ 0 \Phi'_M(\mu^\star;\mu-\mu^\star)\le0 Φ M ′ ( μ ⋆ ; μ − μ ⋆ ) ≤ 0 . For 0 < α ≤ 1 2 0<\alpha\le\frac{1}{2} 0 < α ≤ 2 1 , Φ M \Phi_M Φ M is concave by Theorem 1, and therefore
Φ M ( μ ) ≤ Φ M ( μ ⋆ ) + Φ M ′ ( μ ⋆ ; μ − μ ⋆ ) ≤ Φ M ( μ ⋆ ) . \Phi_M(\mu)\le\Phi_M(\mu^\star)+\Phi'_M(\mu^\star;\mu-\mu^\star)\le\Phi_M(\mu^\star). Φ M ( μ ) ≤ Φ M ( μ ⋆ ) + Φ M ′ ( μ ⋆ ; μ − μ ⋆ ) ≤ Φ M ( μ ⋆ ) .
Thus μ ⋆ \mu^\star μ ⋆ is a global maximizer of Φ M \Phi_M Φ M over K K , ε \mathcal{K}_{K,\varepsilon} K K , ε . Since μ ( t k ) → μ ⋆ \mu^{(t_k)}\to\mu^\star μ ( t k ) → μ ⋆ and Φ M \Phi_M Φ M is continuous, Φ M ( μ ( t k ) ) → Φ M ( μ ⋆ ) = Φ ε ⋆ \Phi_M(\mu^{(t_k)})\to\Phi_M(\mu^\star)=\Phi^\star_\varepsilon Φ M ( μ ( t k ) ) → Φ M ( μ ⋆ ) = Φ ε ⋆ . Because the full sequence Φ M ( μ ( t ) ) \Phi_M(\mu^{(t)}) Φ M ( μ ( t ) ) converges, its limit must also be Φ ε ⋆ \Phi^\star_\varepsilon Φ ε ⋆ . Hence Φ M ( μ ( t ) ) → Φ ε ⋆ \Phi_M(\mu^{(t)})\to\Phi^\star_\varepsilon Φ M ( μ ( t ) ) → Φ ε ⋆ .
Effect of the allocation floor
We finally quantify the effect of replacing the original feasible set K K \mathcal{K}_K K K by K K , ε \mathcal{K}_{K,\varepsilon} K K , ε . Let
Φ ⋆ : = max μ ∈ K K Φ M ( μ ) , Φ ε ⋆ : = max μ ∈ K K , ε Φ M ( μ ) . \Phi^\star:=\max_{\mu\in\mathcal{K}_K}\Phi_M(\mu),\qquad
\Phi^\star_\varepsilon:=\max_{\mu\in\mathcal{K}_{K,\varepsilon}}\Phi_M(\mu). Φ ⋆ := μ ∈ K K max Φ M ( μ ) , Φ ε ⋆ := μ ∈ K K , ε max Φ M ( μ ) .
Proposition 10 (Effect of the allocation floor) Let 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 and 0 < ε < K / N 0 < \varepsilon< K/N 0 < ε < K / N . Then
0 ≤ Φ ⋆ − Φ ε ⋆ ≤ ε N K [ Φ ⋆ − Φ M ( K N 1 ) ] . 0 \le\Phi^\star- \Phi^\star_\varepsilon\le\frac{\varepsilon N}{K}\left[\Phi^\star- \Phi_M\left(\frac{K}{N}\mathbf{1}\right)\right]. 0 ≤ Φ ⋆ − Φ ε ⋆ ≤ K εN [ Φ ⋆ − Φ M ( N K 1 ) ] . Proof Since K K , ε ⊆ K K \mathcal{K}_{K,\varepsilon} \subseteq\mathcal{K}_K K K , ε ⊆ K K , we immediately have Φ ε ⋆ ≤ Φ ⋆ \Phi^\star_\varepsilon\le\Phi^\star Φ ε ⋆ ≤ Φ ⋆ .
Let μ ⋆ ∈ K K \mu^\star\in\mathcal{K}_K μ ⋆ ∈ K K be a maximizer of Φ M \Phi_M Φ M , and define δ : = ε N / K \delta:= \varepsilon N/K δ := εN / K and u : = ( K / N ) 1 u := (K/N)\mathbf{1} u := ( K / N ) 1 . Since 0 < ε < K / N 0 < \varepsilon< K/N 0 < ε < K / N , we have 0 < δ < 1 0 < \delta< 1 0 < δ < 1 . Consider μ ~ : = ( 1 − δ ) μ ⋆ + δ u \widetilde{\mu} := (1-\delta)\mu^\star+ \delta u μ := ( 1 − δ ) μ ⋆ + δ u . For every ( ℓ , j ) (\ell,j) ( ℓ , j ) , μ ~ j ℓ ≥ δ K / N = ε \widetilde{\mu}^{\ell}_j \ge\delta K/N = \varepsilon μ j ℓ ≥ δ K / N = ε , while ∑ ℓ , j μ ~ j ℓ = K \sum_{\ell,j}\widetilde{\mu}^{\ell}_j = K ∑ ℓ , j μ j ℓ = K and μ ~ j ℓ ≤ 1 \widetilde{\mu}^{\ell}_j \le1 μ j ℓ ≤ 1 . Hence μ ~ ∈ K K , ε \widetilde{\mu} \in\mathcal{K}_{K,\varepsilon} μ ∈ K K , ε .
By Theorem 1, Φ M \Phi_M Φ M is concave for 0 < α ≤ 1 2 0 < \alpha\le\frac{1}{2} 0 < α ≤ 2 1 . Therefore
Φ ε ⋆ ≥ Φ M ( μ ~ ) ≥ ( 1 − δ ) Φ M ( μ ⋆ ) + δ Φ M ( u ) = ( 1 − δ ) Φ ⋆ + δ Φ M ( K N 1 ) . \Phi^\star_\varepsilon\ge\Phi_M(\widetilde{\mu}) \ge(1-\delta)\Phi_M(\mu^\star) + \delta\Phi_M(u) = (1-\delta)\Phi^\star+ \delta\Phi_M\left(\frac{K}{N}\mathbf{1}\right). Φ ε ⋆ ≥ Φ M ( μ ) ≥ ( 1 − δ ) Φ M ( μ ⋆ ) + δ Φ M ( u ) = ( 1 − δ ) Φ ⋆ + δ Φ M ( N K 1 ) . Rearranging and using δ = ε N / K \delta= \varepsilon N/K δ = εN / K gives
Φ ⋆ − Φ ε ⋆ ≤ ε N K [ Φ ⋆ − Φ M ( K N 1 ) ] , \Phi^\star- \Phi^\star_\varepsilon\le\frac{\varepsilon N}{K}\left[\Phi^\star- \Phi_M\left(\frac{K}{N}\mathbf{1}\right)\right], Φ ⋆ − Φ ε ⋆ ≤ K εN [ Φ ⋆ − Φ M ( N K 1 ) ] , which completes the proof.
Approximation of the allocation oracle
As in Algorithm 1, BMuon does not recompute the allocation μ \mu μ at every optimizer step. Instead, every five optimizer steps it performs an allocation refresh : μ \mu μ is recomputed and then kept fixed for the next four optimizer steps. The budget may be scheduled over training, so we write K t K_t K t for the budget at optimizer step t t t . At a refresh step t t t , the new allocation is computed under the current budget K t K_t K t .
The practical implementation therefore introduces two approximations. First, at each allocation refresh, we perform only one MM update instead of solving the allocation problem to convergence. Second, we refresh μ \mu μ only once every five optimizer steps, rather than at every optimizer step. We measure these two errors separately, and then test whether replacing the practical one-step update by a certified solve at each refresh changes the training and compression results.
Two paired training runs. We use the same CIFAR-100 ViT and training setup as in Appendix E.2, except that we train for 100 epochs. As in our training experiments, we use α = 1 / 2 \alpha= 1/2 α = 1/2 . For each seed, we run two paired versions of BMuon that differ only in how the allocation μ \mu μ is computed at an allocation refresh. The MLPs contain N = 4608 N = 4608 N = 4608 hidden channels in total. The budget K t K_t K t decreases linearly from N N N to 0.3 N 0.3N 0.3 N during the first third of training and is then kept fixed. In this experiment, K t K_t K t changes only at allocation refreshes, so it is constant during the four intervening optimizer steps. In both runs, μ \mu μ is refreshed every five optimizer steps and is held fixed between two consecutive refreshes.
The two runs differ only in how μ \mu μ is computed at a refresh step t t t . In the 1-step refresh run , we start from the allocation carried from the previous refresh, compute the MM scores using NS5 _5 5 , the five-step Newton–Schulz approximation defined in Appendix A, and perform one MM update under the current budget K t K_t K t . In the certified refresh run , we instead use exact polar factors and repeat the MM update until the stopping condition Eq. (25) is satisfied. In both runs, the allocation computed at step t t t satisfies the current constraint ∑ i μ i = K t \sum_i \mu_i = K_t ∑ i μ i = K t .
Apart from this difference, the two runs use the same BMuon weight update. For each seed, they also use the same initialization, data order, augmentation stream, and budget schedule.
Notation and certified error bound. Let M t M_t M t denote all MLP momentum matrices at optimizer step t t t , and define C t = { μ ∈ [ ε , 1 ] N : 1 ⊤ μ = K t } C_t = \{\mu\in[\varepsilon,1]^N : \mathbf{1}^{\top}\mu= K_t\} C t = { μ ∈ [ ε , 1 ] N : 1 ⊤ μ = K t } . For α = 1 2 \alpha= \frac{1}{2} α = 2 1 , the allocation objective is
Φ M ( μ ) = ∑ ℓ = 1 L ( ∥ D μ 1 / 2 M 1 ℓ ∥ ∗ + ∥ M 2 ℓ D μ 1 / 2 ∥ ∗ ) . \Phi_M(\mu) = \sum_{\ell=1}^{L} \left( \left\|D_{\mu}^{1/2}M_1^{\ell}\right\|_* + \left\|M_2^{\ell}D_{\mu}^{1/2}\right\|_* \right). Φ M ( μ ) = ℓ = 1 ∑ L ( D μ 1/2 M 1 ℓ ∗ + M 2 ℓ D μ 1/2 ∗ ) .
We write Φ t ⋆ = max μ ∈ C t Φ M t ( μ ) \Phi_t^\star= \max_{\mu\in C_t}\Phi_{M_t}(\mu) Φ t ⋆ = max μ ∈ C t Φ M t ( μ ) .
For any feasible μ \mu μ , define g t ( μ ) = max ν ∈ C t ⟨ ∇ Φ M t ( μ ) , ν − μ ⟩ g_t(\mu) = \max_{\nu\in C_t}\langle\nabla\Phi_{M_t}(\mu),\nu-\mu\rangle g t ( μ ) = max ν ∈ C t ⟨ ∇ Φ M t ( μ ) , ν − μ ⟩ . We call g t ( μ ) g_t(\mu) g t ( μ ) the Frank–Wolfe gap. Since Φ M t \Phi_{M_t} Φ M t is concave (Theorem 1 ), for every ν ∈ C t \nu\in C_t ν ∈ C t , Φ M t ( ν ) ≤ Φ M t ( μ ) + ⟨ ∇ Φ M t ( μ ) , ν − μ ⟩ \Phi_{M_t}(\nu) \le\Phi_{M_t}(\mu) + \langle\nabla\Phi_{M_t}(\mu),\nu-\mu\rangle Φ M t ( ν ) ≤ Φ M t ( μ ) + ⟨ ∇ Φ M t ( μ ) , ν − μ ⟩ . Maximizing over ν \nu ν gives Φ t ⋆ − Φ M t ( μ ) ≤ g t ( μ ) \Phi_t^\star-\Phi_{M_t}(\mu) \le g_t(\mu) Φ t ⋆ − Φ M t ( μ ) ≤ g t ( μ ) , which implies in particular that g t ( μ ) ≥ 0 g_t(\mu) \ge0 g t ( μ ) ≥ 0 for every feasible μ \mu μ .
At optimizer step t t t , let μ t r e f ∈ C t \mu_t^{\mathrm{ref}} \in C_t μ t ref ∈ C t denote the allocation obtained by repeatedly applying the exact-polar MM update over C t C_t C t (Theorem 9 shows that the repeated MM updates converge) until
g t ( μ t r e f ) Φ M t ( μ t r e f ) ≤ 10 − 4 . (25) \frac{g_t(\mu_t^{\mathrm{ref}})}{\Phi_{M_t}(\mu_t^{\mathrm{ref}})} \le10^{-4}. \tag*{(25)} Φ M t ( μ t ref ) g t ( μ t ref ) ≤ 1 0 − 4 . (25)
Define U t = Φ M t ( μ t r e f ) + g t ( μ t r e f ) U_t = \Phi_{M_t}(\mu_t^{\mathrm{ref}}) + g_t(\mu_t^{\mathrm{ref}}) U t = Φ M t ( μ t ref ) + g t ( μ t ref ) . The previous inequality gives U t ≥ Φ t ⋆ U_t \ge\Phi_t^\star U t ≥ Φ t ⋆ .
For any feasible candidate μ ~ \tilde{\mu} μ ~ , we therefore define d t ( μ ~ ) = 1 − Φ M t ( μ ~ ) / U t d_t(\tilde{\mu}) = 1 - \Phi_{M_t}(\tilde{\mu})/U_t d t ( μ ~ ) = 1 − Φ M t ( μ ~ ) / U t . Because U t ≥ Φ t ⋆ U_t \ge\Phi_t^\star U t ≥ Φ t ⋆ ,
1 − Φ M t ( μ ~ ) Φ t ⋆ ≤ 1 − Φ M t ( μ ~ ) U t = d t ( μ ~ ) . 1-\frac{\Phi_{M_t}(\tilde{\mu})}{\Phi_t^\star} \le1-\frac{\Phi_{M_t}(\tilde{\mu})}{U_t}=d_t(\tilde{\mu}). 1 − Φ t ⋆ Φ M t ( μ ~ ) ≤ 1 − U t Φ M t ( μ ~ ) = d t ( μ ~ ) .
Thus d t ( μ ~ ) d_t(\tilde{\mu}) d t ( μ ~ ) is a certified upper bound on the relative loss in the exact allocation objective incurred by using μ ~ \tilde{\mu} μ ~ instead of an optimal allocation.
In particular, for μ t r e f \mu_t^{\mathrm{ref}} μ t ref ,
d t ( μ t r e f ) = g t ( μ t r e f ) Φ M t ( μ t r e f ) + g t ( μ t r e f ) ≤ g t ( μ t r e f ) Φ M t ( μ t r e f ) ≤ 10 − 4 , d_t(\mu_t^{\mathrm{ref}})=\frac{g_t(\mu_t^{\mathrm{ref}})}{\Phi_{M_t}(\mu_t^{\mathrm{ref}})+g_t(\mu_t^{\mathrm{ref}})}\le\frac{g_t(\mu_t^{\mathrm{ref}})}{\Phi_{M_t}(\mu_t^{\mathrm{ref}})}\le10^{-4}, d t ( μ t ref ) = Φ M t ( μ t ref ) + g t ( μ t ref ) g t ( μ t ref ) ≤ Φ M t ( μ t ref ) g t ( μ t ref ) ≤ 1 0 − 4 ,
where the last inequality follows from Eq. (25) .
Hence μ t r e f \mu_t^{\mathrm{ref}} μ t ref is certified to have relative objective loss at most 10 − 4 10^{-4} 1 0 − 4 from the exact optimum. We therefore use μ t r e f \mu_t^{\mathrm{ref}} μ t ref as a certified approximation, in objective value, to an optimal allocation at step t t t , and call it the reference allocation . In the certified refresh run , at each refresh step t t t , we use μ t r e f \mu_t^{\mathrm{ref}} μ t ref as the BMuon allocation and keep it fixed until the next refresh.
For each run and seed, we perform the diagnostics described below at ten refresh steps: the first refresh at which K t / N K_t/N K t / N reaches approximately 0.9, 0.8, … \ldots … , 0.3, followed by refreshes near 50%, 75%, and 99% of training. We denote this set of ten refresh steps by D D D .
The two approximation errors. Figure 4 measures separately the two approximations used by the practical implementation.
Figure 4. Accuracy of the practical allocation approximation. (a) Certified relative deficit of one practical NS5 _5 5 -scored MM update. The same update is evaluated on states generated by the 1-step refresh run and by the certified refresh run. At each training stage, the line is the median over three seeds and the shaded region is their minimum–maximum range. The vertical dashed line marks the beginning of the fixed-budget phase K t / N = 0.3 K_t/N = 0.3 K t / N = 0.3 . (b) Certified relative deficit caused by keeping the allocation installed by the certified refresh run fixed for s = 0 , … , 4 s = 0,\ldots,4 s = 0 , … , 4 optimizer steps. For each seed and each s s s , we first take the largest deficit over the ten sampled windows; the line is the median of these three seed-wise maxima and the shaded region is their minimum–maximum range.
At each t ∈ D t \in\mathcal{D} t ∈ D , let μ ^ t \hat{\mu}_t μ ^ t denote the allocation obtained by applying the practical update rule once: one MM step with scores computed using N S 5 \mathrm{NS}_5 NS 5 , starting from the allocation carried from the previous refresh. Figure 4 (a) reports d t ( μ ^ t ) d_t(\hat{\mu}_t) d t ( μ ^ t ) .
For the red curve, we apply the practical one-step allocation update to the 1-step refresh run at step t t t . For the blue curve, we apply the same update to the certified refresh run at step t t t , only for this diagnostic; the resulting allocation is not used to continue the certified refresh run.
At each sampled training stage t ∈ D t \in\mathcal{D} t ∈ D , the plotted value for each curve is the median over the three seeds, and the shaded region is their minimum–maximum range. The vertical dashed line marks the end of the budget decrease, after which K t / N = 0.3 K_t/N = 0.3 K t / N = 0.3 is fixed.
The certified deficit of one practical allocation update remains of order 10 − 4 10^{-4} 1 0 − 4 to 10 − 3 10^{-3} 1 0 − 3 over the sampled states. Its largest sampled value is about 1.2 × 10 − 3 1.2 \times10^{-3} 1.2 × 1 0 − 3 on states from the 1-step refresh run and about 1.1 × 10 − 3 1.1 \times10^{-3} 1.1 × 1 0 − 3 on states from the certified refresh run. The similar behavior on the two trajectories shows that the accuracy of the one-step rule is not specific to states produced by that same rule.
Figure 4 (b) measures the second approximation: keeping μ \mu μ fixed between two refreshes. We use the certified refresh run for this experiment. For a diagnostic refresh t ∈ D t \in\mathcal{D} t ∈ D , BMuon uses μ t r e f \mu_t^{\mathrm{ref}} μ t ref and keeps it fixed for the following five optimizer steps. For s ∈ { 0 , 1 , 2 , 3 , 4 } s \in\{0,1,2,3,4\} s ∈ { 0 , 1 , 2 , 3 , 4 } , we evaluate this same allocation under the current momentum state M t + s M_{t+s} M t + s and report d t + s ( μ t r e f ) d_{t+s}(\mu_t^{\mathrm{ref}}) d t + s ( μ t ref ) .
This quantity upper-bounds the relative objective loss incurred by continuing to use the allocation computed at step t t t , instead of recomputing an optimal allocation for the current momentum state M t + s M_{t+s} M t + s . Thus s = 0 s=0 s = 0 corresponds to the refresh step, while s = 4 s=4 s = 4 corresponds to the last optimizer step before the next refresh.
To avoid hiding a difficult sampled window by averaging over different training steps, Figure 4 (b) first takes the worst case over t ∈ D t \in\mathcal{D} t ∈ D . For seed j j j and offset s s s , define w j , s = max t ∈ D d t + s r e f , ( j ) ( μ t r e f ) w_{j,s} = \max_{t \in\mathcal{D}} d_{t+s}^{\mathrm{ref},(j)}(\mu_t^{\mathrm{ref}}) w j , s = max t ∈ D d t + s ref , ( j ) ( μ t ref ) . The plotted value at offset s s s is the median of w 0 , s w_{0,s} w 0 , s , w 1 , s w_{1,s} w 1 , s , w 2 , s w_{2,s} w 2 , s , and the shaded region is their minimum–maximum range.
Even under this worst-over-sampled-windows summary, the certified deficit after four optimizer steps remains on the order of 10 − 3 10^{-3} 1 0 − 3 . Thus, at the sampled states, keeping μ \mu μ fixed for the four steps between two refreshes produces only a small additional error in the allocation objective.
End-to-end effect. The previous experiments measure the approximation error in the allocation objective. We now test whether this error changes the trained model. Figure 5 compares the two runs. Recall that both refresh μ \mu μ every five optimizer steps: the 1-step refresh run uses one practical MM update, while the certified refresh run uses μ t r e f \mu_t^{\mathrm{ref}} μ t ref .
Figure 5. End-to-end effect of the allocation approximation. We compare the 1-step refresh run with the certified refresh run; both refresh μ \mu μ every five optimizer steps. (a) CIFAR-100 test accuracy during 100 epochs of training. (b) Test accuracy after global structured MLP-channel pruning of the final trained model, using its final μ \mu μ to rank channels, without fine-tuning. For each curve, the line is the median over three seeds and the shaded region is their minimum–maximum range.
Figure 5 (a) shows CIFAR-100 test accuracy during training. Figure 5 (b) shows the accuracy of the final models after structured MLP-channel pruning, without fine-tuning. For each model, we rank all MLP channels globally by the corresponding entries of its final allocation μ \mu μ and remove those with the smallest values. In both panels, the plotted value for each curve is the median over the three seeds, and the shaded region is their minimum–maximum range.
Using μ t r e f \mu_t^{\mathrm{ref}} μ t ref at each refresh leaves test accuracy and post-training compressibility unchanged within the observed seed-to-seed variability. At the final epoch, the absolute differences between the paired runs are 0.69, 0.26, and 0.11 percentage points for seeds 0, 1, and 2, respectively. At 70% pruning, they are 0.61, 0.22, and 0.46 percentage points. For comparison, changing the seed of the 1-step refresh run gives differences of 0.09–0.80 percentage points at the final epoch and 0.05–0.87 percentage points at 70% pruning. The sign of the paired difference is not consistent across seeds.
Scope of the comparison. Both runs refresh μ \mu μ every five optimizer steps. Thus, the end-to-end comparison isolates the difference between one practical MM update and the certified allocation μ t r e f \mu_t^{\mathrm{ref}} μ t ref at each refresh; it does not compare against a run that recomputes a certified allocation at every optimizer step. Figure 4 (b) separately shows that keeping μ \mu μ fixed during the four intervening steps introduces only a small objective error at the sampled states.
Computational Overhead
Relative to Muon, BMuon’s additional computation is confined to the budgeted matrices (the MLP weights; Algorithm 1); all other parameters are updated exactly as in Muon. On steps where the allocation is not updated, BMuon uses the same number of Newton–Schulz polar evaluations as Muon, with only additional channelwise row/column scalings by D μ α D_\mu^\alpha D μ α . On an allocation-update step, BMuon performs one additional polar evaluation per budgeted matrix to construct the MM surrogate, before recomputing the weighted polar factor used for the parameter update. Updating the allocation every Δ T μ \Delta T_\mu Δ T μ steps therefore raises the amortized number of polar evaluations per budgeted matrix from 1 to 1 + 1 / Δ T μ 1+1/\Delta T_\mu 1 + 1/Δ T μ per optimizer step, i.e. 1.2 for Δ T μ = 5 \Delta T_\mu=5 Δ T μ = 5 . Once the surrogate polar factors are available, the channel scores require only row/column inner products, and the allocation update consists of channelwise operations and a scalar bisection over the global budget. The only persistent additional optimizer state is the allocation vector μ ∈ R N \mu\in\mathbb{R}^{N} μ ∈ R N , one scalar per budgeted channel, which is negligible next to the momentum buffers.
We verify empirically that this overhead is small in practice by comparing end-to-end training times against Muon at the allocation-update frequency used throughout our experiments, Δ T μ = 5 \Delta T_\mu=5 Δ T μ = 5 , on the ImageNet-1K ViT-S/16 and 1B-parameter LLaMA settings of Appendices E.4 and E.5. All measurements use eight H100 GPUs and the same Muon and BMuon implementations as the corresponding training runs.
ImageNet-1K. Table 2 reports the mean epoch time for each full 150-epoch gradual-pruning run (training only, excluding evaluation). Averaged over three seeds, BMuon and Muon require 92.5 ± 0.4 92.5\pm0.4 92.5 ± 0.4 and 91.9 ± 1.2 91.9\pm1.2 91.9 ± 1.2 seconds per epoch at 90% final pruning, and 92.1 ± 0.5 92.1\pm0.5 92.1 ± 0.5 and 91.9 ± 0.6 91.9\pm0.6 91.9 ± 0.6 seconds at 95%. The mean differences (0.6% and 0.3%) are comparable to the seed-to-seed variation, so the runs do not resolve an end-to-end slowdown of BMuon relative to Muon.
Table 2. ImageNet-1K training time. Mean epoch time in seconds over each full 150-epoch gradual-pruning run, excluding evaluation. Differences between optimizers are comparable to seed-to-seed variation.
LLaMA. Table 3 reports timing measurements from the 13.1B-token LLaMA training runs. For each run, the reported time per step is the total measured training time divided by the 6,242 training steps. The training time excludes evaluation and checkpointing. Averaged over three seeds, a step takes 4858 ± 42 4858\pm42 4858 ± 42 ms with BMuon and 4918 ± 22 4918\pm22 4918 ± 22 ms with Muon. The 1.2% difference is comparable to the seed-to-seed variation and can also be affected by normal system-level variation between separate jobs, such as job placement, per-GPU performance, and communication, so we do not attribute it to the optimizer. The runs show no end-to-end slowdown of BMuon relative to Muon.
Table 3. LLaMA training time. Mean time per training step and total training time for each full 13.1B-token run, excluding evaluation and checkpointing. The difference between optimizers is comparable to seed-to-seed variation and does not indicate a slowdown of BMuon.
Experimental Details
This section provides the experimental details and additional results for § 5 . Appendix E.1 gives the ImageNet-1K post-training pruning setup and full pruning results. Appendix E.2 gives the CIFAR-100 post-training compression setup and full pruning results. Appendix E.3 studies the effect of the global budget K K K . Appendix E.4 gives the gradual-pruning setup and architecture controls for CIFAR-100 and ImageNet-1K. Appendix E.5 provides the experimental details and full pruning results for the LLaMA/FineWeb experiments.
ImageNet-1K Post-Training Structured Pruning
Architecture and training setup. We use a custom PyTorch ViT-S/16 implementation following the plain-ViT design of Beyer et al. [5 ] : 224 × 224 224 \times224 224 × 224 inputs, 16 × 16 16 \times16 16 × 16 patches, embedding dimension 384, 12 Transformer blocks, six attention heads, MLP width 1536, fixed two-dimensional sinusoidal positional embeddings, global average pooling over patch tokens, and a pre-logits representation layer. The model therefore contains 12 × 1536 = 18,432 12 \times1536 = 18{,}432 12 × 1536 = 18 , 432 MLP hidden channels. Our ImageNet-1K optimization setup follows the ViT-S/16 configuration of Zhong et al. [36 ] . Matrix-valued parameters within the Transformer blocks are optimized with Muon, while the remaining parameters are optimized with AdamW. We use global batch size 4096, Muon learning rate 3 3 × 10 − 3 3\sqrt{3} \times10^{-3} 3 3 × 1 0 − 3 , AdamW learning rate 3 × 10 − 3 3 \times10^{-3} 3 × 1 0 − 3 , weight decay 0.3, Muon momentum 0.95 with Nesterov, and dropout 0.1. Training uses sigmoid binary cross entropy with label smoothing 0.1, random resized crops, and horizontal flips, without Mixup, CutMix, RandAugment, random erasing, or stochastic depth. Muon uses five Newton–Schulz iterations following Jordan et al. [16 ] and the Moonlight-style RMS-matching scale 0.2 max ( d o u t , d i n ) 0.2\sqrt{\max(d_{\mathrm{out}},d_{\mathrm{in}})} 0.2 max ( d out , d in ) (Liu et al. [20 ] ). All models are trained for 150 epochs with 5000 warmup steps followed by cosine learning-rate decay to zero, using a half-length adaptation of the 300-epoch, 10,000-step ViT-S/16 schedule of Zhong et al. [36 ] .
BMuon training. BMuon is applied only to the MLP matrices; the remaining Muon-eligible Transformer matrices use ordinary Muon. We use α = 1 / 2 \alpha= 1/2 α = 1/2 , one MM allocation update every five optimizer steps, and ε = 10 − 6 \varepsilon= 10^{-6} ε = 1 0 − 6 . Let N = 18,432 N = 18{,}432 N = 18 , 432 denote the total number of MLP hidden channels. The global allocation budget is initialized at K ( 0 ) = N K(0) = N K ( 0 ) = N and linearly annealed to K t a r g e t = 5,530 ≈ 0.30 N K_{\mathrm{target}} = 5{,}530 \approx0.30N K target = 5 , 530 ≈ 0.30 N during the first 25% of training, after which it is held fixed for the remaining 75%.
Post-training pruning. After training, we take the final epoch-150 checkpoint and perform one-shot global structured pruning of coupled MLP hidden channels. Each pruning level is evaluated independently from the same dense checkpoint; pruning is therefore not cumulative, and no post-pruning fine-tuning or recovery is performed. For channel j j j in block ℓ \ell ℓ , we use the coupled magnitude score s j ℓ = ∥ W 1 , j ℓ ∥ 2 2 + ∥ W 2 , j ℓ ∥ 2 2 s_j^\ell= \sqrt{\lVert W_{1,j}^\ell\rVert_2^2 + \lVert W_{2,j}^\ell\rVert_2^2} s j ℓ = ∥ W 1 , j ℓ ∥ 2 2 + ∥ W 2 , j ℓ ∥ 2 2 . All 18,432 channels are ranked jointly across the 12 blocks, subject to a minimum of four surviving channels per block. We evaluate global MLP-channel pruning levels 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 95%. Figure 1 compares ordinary Muon and BMuon using this same magnitude-based pruning rule.
Evaluation. We run seeds 0, 1, 2 and evaluate Top-1 accuracy on the full 50,000-image ImageNet-1K validation set. Reported curves show the mean and sample standard deviation across the three seeds. Table 4 reports the exact results corresponding to Figure 1 .
Table 4. ImageNet-1K post-training structured pruning. Top-1 accuracy (%) after global MLP-channel pruning of ViT-S/16 using coupled weight-magnitude ranking. Results are mean ± sample standard deviation over three seeds. No post-pruning recovery is performed.
CIFAR-100 Post-Training Compression
Architecture and shared training. We use a six-block Vision Transformer on CIFAR-100 with 4 × 4 4 \times4 4 × 4 patches, embedding dimension 192, three attention heads, and MLP width 768. All methods are trained for 200 epochs with seeds { 0 , 1 , 2 } \{0, 1, 2\} { 0 , 1 , 2 } using batch size 256. For the matrix optimizer, we use peak learning rate 0.02, momentum 0.95, weight decay 0.05, and five Newton–Schulz steps. Biases, LayerNorm parameters, the CLS token, positional embeddings, patch-embedding layer, and classifier head use AdamW with peak learning rate 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 , betas (0.9, 0.999), and weight decay 0.05. We use five warmup epochs followed by cosine decay. Training uses standard CIFAR-100 augmentations, including random cropping, horizontal flipping, RandAugment, random erasing, Mixup, and CutMix. All reported compression results are recomputed from the final epoch-200 checkpoint. Evaluation uses the CIFAR-100 test set with batch size 512 and no test-time augmentation. Reported uncertainties are sample standard deviations over the three seeds.
BMUon. BMuon uses α = 1 / 2 \alpha= 1/2 α = 1/2 and a global budget over all 4608 MLP hidden channels. The budget is linearly annealed from 4608 to 922, corresponding approximately to 20% of the total channel mass, over the first third of training and then held fixed. The allocation is updated every five optimizer steps using one MM update, with ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 . Non-MLP Transformer matrices use ordinary Muon.
Muon+GL. Muon+GL applies an exact proximal Group-Lasso step with coefficient λ G L \lambda_{\mathrm{GL}} λ GL to coupled MLP hidden-channel groups ( W 1 , : , j ℓ , W 2 , j , : ℓ ) (W^{\ell}_{1,:,j}, W^{\ell}_{2,j,:}) ( W 1 , : , j ℓ , W 2 , j , : ℓ ) , using the group norm ( ∥ W 1 , : , j ℓ ∥ 2 2 + ∥ W 2 , j , : ℓ ∥ 2 2 ) 1 / 2 (\lVert W^{\ell}_{1,:,j}\rVert_{2}^{2}+\lVert W^{\ell}_{2,j,:}\rVert_{2}^{2})^{1/2} (∥ W 1 , : , j ℓ ∥ 2 2 + ∥ W 2 , j , : ℓ ∥ 2 2 ) 1/2 . We tune λ G L ∈ { 0.01 , 0.02 , 0.05 , 0.1 , 0.2 , 0.5 } \lambda_{\mathrm{GL}} \in\{0.01, 0.02, 0.05, 0.1, 0.2, 0.5\} λ GL ∈ { 0.01 , 0.02 , 0.05 , 0.1 , 0.2 , 0.5 } , obtaining mean dense accuracies of 71.50, 71.58, 71.25, 70.03, 69.23, and 61.21%, respectively. Increasing $\lambda_{\mathrm{GL}} strengthens the sparsity-inducing penalty and can improve post-training compressibility at the expense of dense accuracy. To compare at similar pre-compression accuracy, we restrict attention here to settings within one percentage point of BMuon’s 71.46%, leaving λ G L ∈ { 0.01 , 0.02 , 0.05 } \lambda_{\mathrm{GL}} \in\{0.01, 0.02, 0.05\} λ GL ∈ { 0.01 , 0.02 , 0.05 } . At 50% MLP-channel pruning, these settings retain 9.71%, 16.30%, and 71.25% test accuracy, respectively. We therefore use λ G L = 0.05 \lambda_{\mathrm{GL}} = 0.05 λ GL = 0.05 throughout this experiment.
NuMuon. NuMuon [8 ] is applied during training to all Muon-eligible Transformer weight matrices: the fused QKV projection, attention output projection, and the two MLP projections in each of the six blocks, for 24 matrices in total. Following the authors’ cosine rank-scheduling strategy, we use a 10% initial full-rank period and, in our implementation, cosine-anneal the relative update rank from 1.0 to 0.25 between 10% and 80% of training, holding it at 0.25 thereafter. Since all eligible matrices have q = min ( d i n , d o u t ) = 192 q = \min(d_{\mathrm{in}}, d_{\mathrm{out}}) = 192 q = min ( d in , d out ) = 192 , this corresponds to a final update rank of 48. Full-rank steps use the standard Newton–Schulz Muon update, while reduced-rank steps use randomized Block Krylov with two Krylov iterations. The NuMuon paper reports an oversampling setting of 8; in our implementation, we realize this as a Krylov working dimension b = min ( q , k + 8 ) b = \min(q, k + 8) b = min ( q , k + 8 ) , where k = ⌈ r ( t ) q ⌉ k = \lceil r(t)q \rceil k = ⌈ r ( t ) q ⌉ is the target update rank.
Post-training compression. Compression is restricted to the ViT MLP blocks. We evaluate compression ratios { 0 , 0.1 , … , 0.9 , 0.95 } \{0, 0.1, \ldots, 0.9, 0.95\} { 0 , 0.1 , … , 0.9 , 0.95 } independently from the same final checkpoint, without cumulative pruning or post-compression fine-tuning. Muon, Muon+GL, and BMuon use global hidden-channel pruning over all six blocks. Magnitude pruning ranks channel j j j in block ℓ \ell ℓ by s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 : , j ℓ ∥ 2 2 s^\ell_j = \sqrt{\lVert W^\ell_{1:,j}\rVert_2^2 + \lVert W^\ell_{2:,j}\rVert_2^2} s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 : , j ℓ ∥ 2 2 . BMuon is evaluated with both this ranking and its final learned allocation μ \mu μ . During global pruning, we skip any candidate removal that would leave an MLP block with fewer than four hidden channels.
NuMuon compression and matching. At evaluation, NuMuon compresses only the 12 MLP weight matrices using truncated SVD; attention weights remain dense. We distribute the retained rank budget as evenly as possible across matrices, with minimum rank 4. At each compression level, we choose the NuMuon rank so that the resulting whole-model parameter count is as close as possible to that of the corresponding structured-pruned model.
Table 5 reports the exact post-training compression results corresponding to Figure 2 .
Effect of the Global Budget
We ablate the global budget K K K on the CIFAR-100 post-training compression setup of Appendix E.2.
Setup. We use the same six-block ViT, 200-epoch training recipe, and optimization settings as in Appendix E.2. We train BMuon with α = 1 / 2 \alpha= 1/2 α = 1/2 and K / N ∈ { 0.1 , 0.2 , … , 0.9 } K/N \in\{0.1, 0.2, \ldots, 0.9\} K / N ∈ { 0.1 , 0.2 , … , 0.9 } , with no budget schedule and holding all other hyperparameters fixed. After training, we perform global MLP-channel pruning without recovery, ranking channels either by coupled weight magnitude, s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 : , j ℓ ∥ 2 2 s^\ell_j = \sqrt{\lVert W^\ell_{1:,j}\rVert_2^2 + \lVert W^\ell_{2:,j}\rVert_2^2} s j ℓ = ∥ W 1 : , j ℓ ∥ 2 2 + ∥ W 2 : , j ℓ ∥ 2 2 , or by the final learned allocation μ \mu μ .
Figure 6 shows that the budget directly controls the compression level favored during training. With magnitude pruning, models remain robust approximately up to pruning a fraction 1 − K / N 1-K/N 1 − K / N of the channels, and degrade sharply beyond this point. The same pattern holds with μ \mu μ ranking, but the degradation is generally more gradual, further showing that μ \mu μ provides a useful channel-importance signal. For the most restrictive budgets, particularly K / N = 0.1 K/N = 0.1 K / N = 0.1 and 0.2 0.2 0.2 , the accuracy of the unpruned model also decreases slightly, indicating a tradeoff between aggressive compressibility and performance before pruning.
Figure 6. Effect of the budget on post-training compression. CIFAR-100 test accuracy after global MLP-channel pruning for different budget ratios K / N K/N K / N . Panel (a) uses coupled weight-magnitude ranking, while panel (b) uses the final learned allocation μ \mu μ . Mean ± sample std. over three seeds.
Table 5. CIFAR-100 post-training compression results. Test accuracy (%) reported as mean ± sample standard deviation over three seeds. All compression levels are evaluated independently from the final epoch-200 checkpoint, without post-compression fine-tuning.
Gradual Structured Pruning Experiment
All gradual-pruning experiments use seeds 0, 1, 2. Only MLP hidden channels are pruned, and all surviving channels are ranked globally across blocks at each pruning event. We enforce at least four surviving channels per block. Muon-mag and BMuon-mag use the coupled magnitude score s j ℓ = ∥ W 1 , : , j ℓ ∥ 2 2 + ∥ W 2 , : , j ℓ ∥ 2 2 s^\ell_j = \sqrt{\lVert W^\ell_{1,:,j}\rVert_2^2 + \lVert W^\ell_{2,:,j}\rVert_2^2} s j ℓ = ∥ W 1 , : , j ℓ ∥ 2 2 + ∥ W 2 , : , j ℓ ∥ 2 2 , while BMuon-μ \mu μ uses s j ℓ , μ = μ j ℓ s^{\ell,\mu}_j = \mu^\ell_j s j ℓ , μ = μ j ℓ . After pruning, training resumes from the surviving weights and optimizer state; pruned channels never return.
BMuon schedule. On both CIFAR-100 and ImageNet-1K, BMuon is applied only to the MLP matrices, with α = 1 / 2 \alpha= 1/2 α = 1/2 , one MM step every five optimizer steps, and ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 . At the start of each training phase i i i , let N s t a r t , i N_{\mathrm{start},i} N start , i denote the current number of surviving channels and let N e n d , i N_{\mathrm{end},i} N end , i denote the number of channels to retain at the next pruning event. During the first half of phase i i i , the BMuon budget is linearly annealed from N s t a r t , i N_{\mathrm{start},i} N start , i to N e n d , i N_{\mathrm{end},i} N end , i ; during the second half, it is held fixed at N e n d , i N_{\mathrm{end},i} N end , i . At the end of the phase, pruning reduces the network to exactly N e n d , i N_{\mathrm{end},i} N end , i surviving channels. After pruning, the allocation coordinates μ \mu μ of all surviving channels are reset to one.
CIFAR-100. We use a six-block ViT with 32 × 32 32 \times32 32 × 32 inputs, 4 × 4 4 \times4 4 × 4 patches, embedding dimension 112, four attention heads, and MLP width 448, giving 2688 prunable channels. Training lasts 200 epochs with batch size 256. Muon uses learning rate 0.02, momentum 0.95 without Nesterov, weight decay 0.05, and five Newton–Schulz steps; auxiliary parameters use AdamW with learning rate 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 and weight decay 0.05. We use five epochs of linear warmup followed by cosine decay to 1% of the initial learning rate. Training uses random cropping and horizontal flipping, RandAugment (2, 9), random erasing with probability 0.1, and Mixup/CutMix with parameters 0.8/1.0 and total probability 0.5.
For the 90% experiment, cumulative pruning levels of 20, 40, 60, 75, 85, and 90% are applied after epochs 30, 60, 90, 120, 150, and 180, respectively, followed by 20 recovery epochs. The corresponding surviving channel counts are 2150, 1613, 1075, 672, 403, and 269. For the 95% experiment, cumulative pruning levels of 20, 40, 60, 75, 85, 90, and 95% are applied after epochs 26, 52, 78, 104, 130, 155, and 180, respectively, followed by 20 recovery epochs. The corresponding surviving channel counts are 2150, 1613, 1075, 672, 403, 269, and 134.
ImageNet-1K. We use the same ViT-S/16 architecture, ImageNet-1K data pipeline, base optimizer hyperparameters, and 150-epoch learning-rate schedule as in Appendix E.§ 1 . Unlike the post-training pruning experiment, here physical MLP-channel removal is interleaved with training according to the gradual-pruning protocol described above.
For the 90% experiment, cumulative pruning levels of 20, 40, 60, 70, 80, and 90% are applied after epochs 22, 44, 66, 87, 108, and 129, respectively, followed by 21 recovery epochs. The corresponding surviving channel counts are 14,746, 11,059, 7,373, 5,530, 3,686, and 1,843. For the 95% experiment, cumulative pruning levels of 20, 40, 60, 70, 80, 90, and 95% are applied after epochs 19, 38, 57, 76, 95, 114, and 132, respectively, followed by 18 recovery epochs. The corresponding surviving channel counts are 14,746, 11,059, 7,373, 5,530, 3,686, 1,843, and 922.
Evaluation. For CIFAR-100, we report final Top-1 accuracy on the test set at epoch 200. For ImageNet-1K, we report final Top-1 accuracy on the full 50,000-image validation set at epoch 150. In both cases, evaluation is performed after the final recovery stage. Reported means and standard deviations are over seeds 0, 1, 2.
Architecture controls. For each BMuon-μ \mu μ pruning run, we record the final per-block MLP widths. To evaluate the selected architecture on its own, we instantiate a new model with these widths and train it from scratch with standard Muon, without pruning, using the same training recipe and horizon. As a baseline, we also train a uniform-width model from scratch with standard Muon, with its total MLP channel count matched as closely as possible to the corresponding architecture discovered by BMuon-μ \mu μ . Seed pairing is one-to-one: the architecture discovered with seed i i i is retrained from scratch using seed i i i . Thus, for seeds 0, 1, 2, we perform three retraining runs rather than a 3 × 3 3 \times3 3 × 3 crossing of discovery and retraining seeds.
The selected architectures are reported in Table 6 and Table 7 .
Table 6. BMuon-μ \mu μ architectures on CIFAR-100. Per-block MLP widths for blocks 1–6, selected at the end of gradual pruning.
Table 7. BMuon-μ \mu μ architectures on ImageNet-1K. Per-block MLP widths for blocks 1–12, selected at the end of gradual pruning.
On CIFAR-100, the uniform baseline uses width 45 in each of the six blocks at 90% pruning and width 22 at 95%, giving 270 and 132 total MLP channels, respectively, compared with 269 and 134 channels in the corresponding BMuon-μ \mu μ architectures. On ImageNet-1K, the uniform baseline uses width 154 in each of the 12 blocks at 90% pruning and width 77 at 95%, giving 1848 and 924 channels, compared with 1843 and 922 channels in the corresponding BMuon-μ \mu μ architectures.
LLaMA/FineWeb Experimental Details
Implementation provenance. Our LLaMA/FineWeb experiments are based on the LLaMA-1B setup of Muon+ (Zhang et al. [35 ] ).1 We adopt its architecture and training recipe, reducing only the number of Transformer blocks from 24 to 17 to obtain a 986,746,880-parameter model; all other architectural hyperparameters are unchanged. The Muon+ codebase builds on Muon (Jordan et al. [16 ] ) and the modded-nanogpt training framework.2 We retain its LLaMA-2 tokenizer and FineWeb preprocessing and sharding conventions, Muon parameter routing, auxiliary AdamW-style updates for non-Muon parameters, momentum/Nesterov convention, five-step Newton–Schulz polar approximation, matrix learning-rate shape/RMS scaling, warmup–cosine schedule, and BF16 training conventions. For our experiments, we integrate BMuon into the SwiGLU MLPs.
Model and data. We use a 17-layer pre-norm LLaMA-style causal decoder with hidden dimension 2048, 32 attention and 32 key–value heads, SwiGLU width 5461, context length 4096, vocabulary size 32,000, RMSNorm (ϵ = 10 − 5 \epsilon= 10^{-5} ϵ = 1 0 − 5 ), and RoPE with θ = 10,000 \theta= 10{,}000 θ = 10 , 000 . The token embedding and language-model head are untied. The model has 986,746,880 parameters. Each SwiGLU layer contains gate, up, and down projections W g , W u ∈ R 5461 × 2048 W_g, W_u \in\mathbb{R}^{5461 \times2048} W g , W u ∈ R 5461 × 2048 and W d ∈ R 2048 × 5461 W_d \in\mathbb{R}^{2048 \times5461} W d ∈ R 2048 × 5461 , giving N = 17 × 5461 = 92,837 N = 17 \times5461 = 92{,}837 N = 17 × 5461 = 92 , 837 structural MLP channels in total. We train on 13,090,422,784 tokens from the sample–100BT configuration of FineWeb, using the LLaMA-2 tokenizer.
Training. Training uses BF16 with an effective global batch of 512 sequences, corresponding to 2,097,152 tokens per optimizer step. We use seeds { 0 , 1 , 2 } \{0, 1, 2\} { 0 , 1 , 2 } for all methods. For Muon matrices, we use learning rate 0.02, momentum 0.95 with Nesterov momentum, five Newton–Schulz iterations, decoupled weight decay 0.1, gradient clipping at 1.0, a 10% linear warmup, and cosine decay to zero. Embeddings, normalization parameters, and the language-model head use the auxiliary AdamW-style branch with β 1 = β 2 = 0.95 \beta_1 = \beta_2 = 0.95 β 1 = β 2 = 0.95 and ϵ = 10 − 8 \epsilon= 10^{-8} ϵ = 1 0 − 8 . In the Muon baseline, attention and MLP matrices use Muon. For BMuon, the three SwiGLU projection matrices in each block are instead updated by BMuon, while attention matrices continue to use standard Muon.
BMuon configuration. We use α = 1 / 2 \alpha= 1/2 α = 1/2 and a budget K = 18,567 ≈ 0.2 N K = 18{,}567 \approx0.2N K = 18 , 567 ≈ 0.2 N throughout training, with no budget schedule. The allocation is updated every five optimizer steps with one MM iteration, using a numerical floor ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 .
Post-training pruning. We prune only SwiGLU hidden channels. Pruning channel j j j in layer ℓ \ell ℓ jointly removes W g ℓ [ j , : ] W_g^\ell[j,:] W g ℓ [ j , : ] , W u ℓ [ j , : ] W_u^\ell[j,:] W u ℓ [ j , : ] , and W d ℓ [ : , j ] W_d^\ell[:,j] W d ℓ [ : , j ] . For each pruning ratio r ∈ { 0 , 0.1 , … , 0.9 } r \in\{0, 0.1, \ldots, 0.9\} r ∈ { 0 , 0.1 , … , 0.9 } , we restore the unpruned checkpoint and prune directly to ratio r r r . Scores are ranked globally across all Transformer blocks, with a minimum of 10 surviving channels per layer.
For both Muon and BMuon, the magnitude score is
s ℓ j m a g = ( ∥ W g ℓ [ j , : ] ∥ 2 2 + ∥ W u ℓ [ j , : ] ∥ 2 2 + ∥ W d ℓ [ : , j ] ∥ 2 2 ) 1 / 2 . s_{\ell j}^{\mathrm{mag}} = \left(\lVert W_g^\ell[j,:]\rVert_2^2 + \lVert W_u^\ell[j,:]\rVert_2^2 + \lVert W_d^\ell[:,j]\rVert_2^2\right)^{1/2}. s ℓ j mag = ( ∥ W g ℓ [ j , : ] ∥ 2 2 + ∥ W u ℓ [ j , : ] ∥ 2 2 + ∥ W d ℓ [ : , j ] ∥ 2 2 ) 1/2 .
For BMuon, we additionally use
s ℓ j μ = μ j ℓ . s_{\ell j}^{\mu} = \mu_j^\ell. s ℓ j μ = μ j ℓ .
For the activation-aware criterion, let h ℓ j ( x ) = SiLU ( W g ℓ [ j , : ] x ) ( W u ℓ [ j , : ] x ) h_{\ell j}(x) = \operatorname{SiLU}(W_g^\ell[j,:]x)(W_u^\ell[j,:]x) h ℓ j ( x ) = SiLU ( W g ℓ [ j , : ] x ) ( W u ℓ [ j , : ] x ) denote the hidden activation of channel j j j . We score it by
s ℓ j a c t = E x [ h ℓ j ( x ) 2 ] ∥ W d ℓ [ : , j ] ∥ 2 . s_{\ell j}^{\mathrm{act}} = \sqrt{\mathbb{E}_x[h_{\ell j}(x)^2]}\lVert W_d^\ell[:,j]\rVert_2. s ℓ j act = E x [ h ℓ j ( x ) 2 ] ∥ W d ℓ [ : , j ] ∥ 2 .
Activation statistics are accumulated in FP32 over 1,048,576 calibration tokens. Each pruned model is evaluated without recovery on 5,046,272 validation tokens.
Table 8. Post-training structured pruning of the 1B-parameter LLaMA-style model. Validation perplexity after global SwiGLU-channel pruning. Entries report mean ± \pm ± sample std. over seeds {0, 1, 2}. No post-pruning recovery is performed.