Introduction
Modern deep learning owes much of its success to optimization heuristics refined through extensive experimentation [1 , 2 , 3 ] . Notably, state-of-the-art results often depend on surprisingly simple algorithms such as gradient descent (GD), whose practical effectiveness persistently outpaces our theoretical understanding. One particularly intriguing illustration of this gap is the common use of large learning rates. Classical optimization theory traditionally advocates for small learning rates to ensure stable monotonic convergence [4 , 5 ] . Yet, practitioners routinely exceed these conservative bounds, achieving faster convergence and often better generalization [6 , 7 , 8 , 9 , 10 ] . This counterintuitive regime, known as the edge of stability (EoS), is characterized by non-monotonic loss and oscillatory dynamics [11 , 12 ] . The EoS can arise either from initializing training with a large learning rate, or through progressive sharpening, whereby the local curvature grows along the training trajectory until it reaches the stability boundary [12 , 13 , 14 ] .
These empirical successes and observations raise fundamental theoretical questions: Why does operating near instability lead to faster convergence? Can we provide rigorous theoretical guarantees for convergence under these oscillatory dynamics?
Several theoretical perspectives have emerged in recent years [15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 ] . Among the most insightful is the self-stabilization mechanism of [26 ] , which shows that GD naturally adjusts to local curvature in response to large-step-size instabilities, preventing divergence. However, their assumptions are difficult to verify analytically and have only been confirmed numerically on specific architectures. Moreover, this framework does not establish convergence rates. Wu et al. [27 , 28 ] take a complementary approach: they derive convergence rates for logistic-type losses through a direct analysis of the gradient structure. However, their arguments do not reveal the mechanistic role of self-stabilization and do not extend to other loss families without a new specialized analysis for each. MacDonald et al. [29 ] unify both perspectives for overparameterized least squares via a Taylor expansion around a manifold of minimizers, establishing a parallel-orthogonal decomposition that makes the self-stabilization mechanism explicit and simultaneously yields convergence rates. However, this approach fundamentally relies on the existence of a minimizer manifold , which is absent in the classification losses we target.
In this work, we extend this unified approach to exponential-tail losses, a class that includes logistic and cross-entropy losses. The high-level takeaway is:
We develop a unified framework for gradient descent on exponential-tail losses that makes self-stabilization quantitatively explicit and yields convergence rates for the iterates, while revealing a three-phase damping mechanism and a geometry-dependent transient implicit bias.
Specifically, we establish the following:
We identify a subspace defined by the curvature structure of the loss as the geometric anchor for analyzing GD in the exponential-tail setting, and show that decomposing the dynamics relative to it yields a two-equation system: one governs the oscillations orthogonal to the subspace, and the other captures parallel drift along the subspace, where a third-order coupling acts as implicit gradient descent on the curvature, a key feature of the self-stabilization mechanism.
Under certain regularity conditions, which we verify for logistic and cross-entropy under symmetric separable data, we provide a fine-grained convergence analysis with explicit rates for the iterates, not just the loss, revealing a three-phase damping mechanism for the oscillations.
We identify a transient implicit bias of large step sizes: certain orthogonal components can be amplified during the unstable phase and persist at any finite horizon despite asymptotic directional convergence to the max-margin solution.
The remainder of the paper is structured as follows: we begin in § 2 with a review of the related literature. Sections 3–4 develop the general framework; Section 5 verifies it for logistic regression and multiclass classification with the cross-entropy loss; Section 6 empirically validates the theory.
Notation. Throughout, scalars are denoted by lowercase letters (e.g., η , λ \eta, \lambda η , λ ) and vectors by lowercase bold letters (e.g., x , u \mathbf{x}, \mathbf{u} x , u ). We use ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ for the Euclidean norm and ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ for the standard Euclidean inner product. For a subspace S ⊆ R d S \subseteq\mathbb{R}^{d} S ⊆ R d , S ⊥ S^{\perp} S ⊥ denotes its orthogonal complement, Π S \Pi_{S} Π S the orthogonal projector onto S S S , and I S I_{S} I S the identity operator on S S S . We write f ≍ g f \asymp g f ≍ g on a set S S S if there exist constants c , C > 0 c,C > 0 c , C > 0 such that c g ( x ) ≤ f ( x ) ≤ C g ( x ) c g(x) \leq f(x) \leq C g(x) c g ( x ) ≤ f ( x ) ≤ C g ( x ) for all x ∈ S x \in S x ∈ S . We use f ≲ g f \lesssim g f ≲ g and f ≳ g f \gtrsim g f ≳ g for the corresponding one-sided inequalities up to constants.
Related Work
The EoS phenomenon has motivated a rapidly growing literature on large-step-size training, progressive sharpening, benign oscillations, catapult dynamics, and step-size-dependent implicit bias [12 , 30 , 10 , 31 , 32 , 33 , 9 , 11 , 13 , 14 , 34 , 20 , 17 , 35 , 23 , 22 , 15 , 18 , 16 , 36 , 19 , 25 , 37 ] . We defer a broader discussion of this literature to Appendix § 9 and focus here on the works closest to ours:
General self-stabilization mechanisms. This line of work seeks architecture-independent mechanisms explaining why GD remains stable when the local quadratic model predicts instability. The most influential is the self-stabilization framework of [26 ] . The key analytical tool is a cubic Taylor
Table 1. High-level comparison of related theoretical analyses of GD at the edge of stability.
expansion of the loss, centered at a constrained set defined by the sharpness threshold 2 / η 2/\eta 2/ η , which shows that the third-order term induces a negative feedback on the sharpness, causing GD to behave like projected gradient descent under λ max ( θ ) ≤ 2 / η \lambda_{\max}(\theta) \le2/\eta λ m a x ( θ ) ≤ 2/ η . While this provides a compelling explanation of EoS, it is not designed to yield end-to-end convergence rates and its hypotheses involve local, trajectory-dependent quantities that are difficult to verify analytically.
Quantitative convergence analyses for exponential-tail losses. Wu et al. [38 , 27 , 28 ] show that, for logistic regression and related classification losses, large-step GD can preserve the max-margin directional bias while achieving accelerated loss decay, with extensions to regularized objectives, NTK-regime models, and online SGD. Crawshaw and Liu [39 ] sharpen this in two dimensions, giving tight unstable-to-stable transition-time estimates via a fine-grained analysis of orthogonal oscillations. Our work takes a different perspective: rather than deriving rates from margin-based recursions, we connect convergence to the underlying self-stabilization mechanism. This connection enables a fine-grained analysis of the dynamics at the level of the iterates, not just the loss, while simultaneously providing a mechanistic interpretation of convergence. In particular, it reveals a new phenomenon: when the support vectors do not span the full data subspace, the dynamics can exhibit a transient implicit bias, with persistent, and in some cases amplified, orthogonal components.
Relationship to [29 ] . Our work is closest in spirit to MacDonald et al. [29 ] , who bridge the gap between mechanistic and quantitative approaches for overparameterized least squares. Using a Taylor expansion of the gradient around a manifold of minimizers, they establish a parallel-orthogonal decomposition that makes the self-stabilization mechanism transparent while simultaneously yielding convergence rates. We extend this framework in two key directions. First, we replace the minimizer manifold with a curvature-defined reference subspace, allowing the analysis to apply in settings where no finite minimizer exists. This shift moves the theory beyond the overparameterized least-squares regime to exponential-tail losses such as logistic and cross-entropy. Second, our assumptions are formulated at the level of structural properties of the loss and its geometry, rather than tied to a specific objective, thereby enabling a unified treatment of a broader class of problems.
Setup and Motivation
We study gradient descent θ t + 1 = θ t − η ∇ L ( θ t ) \theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}(\theta_t) θ t + 1 = θ t − η ∇ L ( θ t ) on a smooth loss L : R d → R \mathcal{L} : \mathbb{R}^{d} \to\mathbb{R} L : R d → R with step size η > 0 \eta> 0 η > 0 . Two classical facts motivate the analysis. First, if ∇ L \nabla\mathcal{L} ∇ L is L L L -Lipschitz on a convex set containing θ t \theta_t θ t and θ t + 1 \theta_{t+1} θ t + 1 , the descent lemma gives
L ( θ t + 1 ) ≤ L ( θ t ) − η ( 1 − η L 2 ) ∥ ∇ L ( θ t ) ∥ 2 , \mathcal{L}(\theta_{t+1}) \le\mathcal{L}(\theta_t) - \eta\left(1 - \frac{\eta L}{2}\right) \lVert\nabla\mathcal{L}(\theta_t) \rVert^2, L ( θ t + 1 ) ≤ L ( θ t ) − η ( 1 − 2 η L ) ∥ ∇ L ( θ t ) ∥ 2 ,
so the loss decreases for 0 < η < 2 / L 0 < \eta< 2/L 0 < η < 2/ L whenever the gradient is nonzero [4 ] . Second, the Hessian governs the linearized GD map, whose Jacobian is I − η ∇ 2 L ( θ ) I - \eta\nabla^2 \mathcal{L}(\theta) I − η ∇ 2 L ( θ ) : a positive-curvature eigenmode with eigenvalue λ \lambda λ has multiplier 1 − η λ 1 - \eta\lambda 1 − η λ and is expansive when η λ > 2 \eta\lambda> 2 η λ > 2 . The largest eigenvalue λ max ( ∇ 2 L ( θ ) ) \lambda_{\max}(\nabla^2 \mathcal{L}(\theta)) λ m a x ( ∇ 2 L ( θ )) , the sharpness at θ \theta θ , therefore sets the local stability threshold 2 / η 2/\eta 2/ η . Under large step sizes, such as in the EoS regime, the sharpness exceeds 2 / η 2/\eta 2/ η ; the top modes are locally expansive and the loss oscillates non-monotonically [11 ] . Our analysis establishes convergence in exactly this setting, where classical theory provides no convergence guarantee [11 ] . Yet, the losses we study share a geometric structure that makes this possible: the parameter space decomposes into a low-curvature parallel direction along which the loss decreases, and high-curvature orthogonal directions along which the iterates oscillate. The interplay between these components produces a coupled dynamical system, whose structure we develop in this section and use to prove convergence guarantees in Section 4.
Figure 1. GD dynamics illustrating the flat subspace M \mathcal{M} M , orthogonal oscillations, evolution of the loss, and parameter magnitudes.
Warmup Example
Consider the 2D loss function: L ( x , y ) = e − x y 2 \mathcal{L}(x,y)=e^{-x}y^2 L ( x , y ) = e − x y 2 . The gradient and Hessian are given by:
∇ L ( x , y ) = e − x ( − y 2 2 y ) , ∇ 2 L ( x , y ) = e − x ( y 2 − 2 y − 2 y 2 ) . (1) \nabla\mathcal{L}(x,y)=e^{-x}\begin{pmatrix}-y^2\\2y\end{pmatrix},\qquad\nabla^2\mathcal{L}(x,y)=e^{-x}\begin{pmatrix}y^2&-2y\\-2y&2\end{pmatrix}. \tag*{(1)} ∇ L ( x , y ) = e − x ( − y 2 2 y ) , ∇ 2 L ( x , y ) = e − x ( y 2 − 2 y − 2 y 2 ) . (1)
When y = 0 y=0 y = 0 , the curvature vanishes in the x x x direction and the loss is identically zero along x x x . This suggests a natural decomposition of the parameter space into a low-curvature subspace M \mathcal{M} M , the x x x -axis, and its orthogonal complement, the y y y -axis. Using this decomposition, we analyze GD with learning rate η \eta η and initialization y 0 ≠ 0 y_0\ne0 y 0 = 0 :
x t + 1 = x t + η e − x t y t 2 , y t + 1 = y t ( 1 − 2 η e − x t ) . (2) x_{t+1}=x_t+\eta e^{-x_t}y_t^2,\qquad y_{t+1}=y_t(1-2\eta e^{-x_t}). \tag*{(2)} x t + 1 = x t + η e − x t y t 2 , y t + 1 = y t ( 1 − 2 η e − x t ) . (2)
These coupled dynamics reveal the self-stabilization phenomenon. Define the stability threshold x ∗ : = log ( η ) x_*:=\log(\eta) x ∗ := log ( η ) , so that 2 η e − x ∗ = 2 2\eta e^{-x_*}=2 2 η e − x ∗ = 2 and let x 0 < x ∗ x_0<x_* x 0 < x ∗ . The three regimes are then:
Unstable phase (2 η e − x t > 2 2\eta e^{-x_t}>2 2 η e − x t > 2 , i.e. x t < x ∗ x_t<x_* x t < x ∗ ). Since ∣ 1 − 2 η e − x t ∣ > 1 \lvert1-2\eta e^{-x_t}\rvert>1 ∣ 1 − 2 η e − x t ∣ > 1 , ∣ y t ∣ \lvert y_t\rvert ∣ y t ∣ grows while alternating sign at each step. However, this growth is self-limiting: the first equation can be interpreted as gradient descent on e − x t e^{-x_t} e − x t with η y t 2 \eta y_t^2 η y t 2 acting as an effective step size. A large ∣ y t ∣ \lvert y_t\rvert ∣ y t ∣ therefore accelerates forward movement in the x x x direction, which simultaneously decreases e − x t e^{-x_t} e − x t and weakens the instability.
Transition to stability (x t x_t x t crosses x ∗ x_* x ∗ ). Once x t ≥ x ∗ x_t\ge x_* x t ≥ x ∗ , we have 2 η e − x t ≤ 2 2\eta e^{-x_t}\le2 2 η e − x t ≤ 2 and hence ∣ 1 − 2 η e − x t ∣ ≤ 1 \lvert1-2\eta e^{-x_t}\rvert\le1 ∣ 1 − 2 η e − x t ∣ ≤ 1 : the y y y -update becomes non-expanding. For 1 < 2 η e − x t < 2 1<2\eta e^{-x_t}<2 1 < 2 η e − x t < 2 , the iterates still oscillate in sign but now contract. This stage can be described as an oscillatory stable regime .
Non-oscillatory contraction (2 η e − x t < 1 2\eta e^{-x_t}<1 2 η e − x t < 1 , i.e. x t > x ∗ + log 2 x_t>x_*+\log2 x t > x ∗ + log 2 ). The multiplier 1 − 2 η e − x t ∈ ( 0 , 1 ) 1-2\eta e^{-x_t}\in(0,1) 1 − 2 η e − x t ∈ ( 0 , 1 ) , so y t y_t y t contracts without sign changes.
This illustrates the essential components of the unstable regime: oscillation in sharp directions, slow drift in flat directions, and an emergent coupling that stabilizes the dynamics. The subspace M \mathcal{M} M (x x x -axis), plays a central role in separating these behaviors. Two structural features of this example carry over to the general analysis. First, the orthogonal update y t + 1 = ( 1 − 2 η e − x t ) y t y_{t+1}=(1-2\eta e^{-x_t})y_t y t + 1 = ( 1 − 2 η e − x t ) y t is an exact multiplicative map : the contraction/expansion factor depends on x t x_t x t but not on y t y_t y t . Second, the sharpness on the parallel direction decays exponentially along the parallel direction, converting the qualitative observation sharpness drops below 2 / η 2/\eta 2/ η into quantitative rates. One feature does not carry over: the loss restricted to the parallel direction L ( x , 0 ) \mathcal{L}(x,0) L ( x , 0 ) vanishes identically, so forward progress comes entirely from the curvature-reduction coupling. In the classification losses we analyze, the parallel gradient is nonvanishing and provides additional forward drift; the curvature-reduction term acts as an accelerant during the unstable phase, not the sole engine.
Subspace Decomposition and Coupled Dynamics
We now extend the warmup’s two-component structure to a more general setting. The guiding principle is that, near a distinguished one-dimensional subspace, the GD dynamics decompose into a drift parallel to the subspace and oscillations orthogonal to it. Expanding the gradient in these two directions reveals that the reduced dynamics are governed by a reduced loss along the parallel direction and a curvature operator in the orthogonal directions.
Subspace decomposition. Let v ∈ R d v \in\mathbb{R}^{d} v ∈ R d be a unit vector along which the curvature vanishes or decays and set M : = span { v } \mathcal{M} := \operatorname{span}\{v\} M := span { v } . The subspace M \mathcal{M} M is determined by the geometry of the objective: in the warmup example this was the x x x -axis; in the symmetric classification models studied here, it is the max-margin direction. The losses we study have the property that ∇ L ( θ ) \nabla\mathcal{L}(\theta) ∇ L ( θ ) lies in a fixed subspace1 ^{1} 1 N ⊆ R d \mathcal{N} \subseteq\mathbb{R}^{d} N ⊆ R d containing M \mathcal{M} M for all θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d . Without loss of generality, we assume the dynamics are initialized in N \mathcal{N} N and define U : = N ∩ M ⊥ \mathcal{U} := \mathcal{N} \cap\mathcal{M}^{\perp} U := N ∩ M ⊥ , so that N = M ⊕ U \mathcal{N} = \mathcal{M} \oplus\mathcal{U} N = M ⊕ U and every iterate can be written as
θ = θ ∥ v + θ ⊥ , θ ∥ : = ⟨ θ , v ⟩ ∈ R , θ ⊥ : = Π U θ ∈ U . (3) \theta= \theta^{\parallel}v + \theta^{\perp}, \qquad\theta^{\parallel} := \langle\theta, v\rangle\in\mathbb{R}, \qquad\theta^{\perp} := \Pi_{\mathcal{U}}\theta\in\mathcal{U}. \tag*{(3)} θ = θ ∥ v + θ ⊥ , θ ∥ := ⟨ θ , v ⟩ ∈ R , θ ⊥ := Π U θ ∈ U . (3)
Similarly, the gradient decomposes into two components: ∇ L ∥ : = Π M ∇ L \nabla\mathcal{L}^{\parallel} := \Pi_{\mathcal{M}}\nabla\mathcal{L} ∇ L ∥ := Π M ∇ L and ∇ L ⊥ : = Π U ∇ L \nabla\mathcal{L}^{\perp} := \Pi_{\mathcal{U}}\nabla\mathcal{L} ∇ L ⊥ := Π U ∇ L denoting the orthogonal projections onto M \mathcal{M} M and U \mathcal{U} U respectively.
Taylor expansion and coupled dynamics. Recall the GD update θ t + 1 = θ t − η ∇ L ( θ t ) \theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}(\theta_t) θ t + 1 = θ t − η ∇ L ( θ t ) and consider a second-order Taylor expansion of the gradient ∇ L ( θ t ) \nabla\mathcal{L}(\theta_t) ∇ L ( θ t ) around the parallel direction point θ t ∥ v \theta_t^{\parallel}v θ t ∥ v .2 ^{2} 2 Projecting onto M \mathcal{M} M and U \mathcal{U} U under regularity conditions formalized in the next section, we obtain a coupled two-equation system for the GD iterates:
θ t + 1 ⊥ = ( I − η H ( θ t ∥ ) ) θ t ⊥ + O ( ∥ θ t ⊥ ∥ 2 ) , (4) \theta_{t+1}^{\perp} = \left(I - \eta H(\theta_t^{\parallel})\right)\theta_t^{\perp} + O(\lVert\theta_t^{\perp}\rVert^2), \tag*{(4)} θ t + 1 ⊥ = ( I − η H ( θ t ∥ ) ) θ t ⊥ + O (∥ θ t ⊥ ∥ 2 ) , (4)
θ t + 1 ∥ = θ t ∥ − η ℓ ′ ( θ t ∥ ) − η 2 ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ + O ( ∥ θ t ⊥ ∥ 3 ) , (5) \theta_{t+1}^{\parallel} = \theta_t^{\parallel} - \eta\ell'(\theta_t^{\parallel}) - \frac{\eta}{2}(\theta_t^{\perp})^{\mathsf{T}} H'(\theta_t^{\parallel})\theta_t^{\perp} + O(\lVert\theta_t^{\perp}\rVert^3), \tag*{(5)} θ t + 1 ∥ = θ t ∥ − η ℓ ′ ( θ t ∥ ) − 2 η ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ + O (∥ θ t ⊥ ∥ 3 ) , (5)
where the expansion produces two objects that encode the parallel geometry:
ℓ ( θ ∥ ) : = L ( θ ∥ v ) , \ell(\theta^{\parallel}) := \mathcal{L}(\theta^{\parallel}v), ℓ ( θ ∥ ) := L ( θ ∥ v ) ,
the loss restricted to the parallel direction, {#eq:6}
H ( θ ∥ ) : = Π U ∇ 2 L ( θ ∥ v ) Π U , H(\theta^{\parallel}) := \Pi_{\mathcal{U}}\nabla^2\mathcal{L}(\theta^{\parallel}v)\Pi_{\mathcal{U}}, H ( θ ∥ ) := Π U ∇ 2 L ( θ ∥ v ) Π U ,
the Hessian restricted to the parallel direction, {#eq:7}
where H ′ ( θ ∥ ) : = d d θ ∥ H ( θ ∥ ) H'(\theta^{\parallel}) := \frac{d}{d\theta^{\parallel}}H(\theta^{\parallel}) H ′ ( θ ∥ ) := d θ ∥ d H ( θ ∥ ) and ℓ ′ ( θ ∥ ) : = d d θ ∥ ℓ ( θ ∥ ) \ell'(\theta^{\parallel}) := \frac{d}{d\theta^{\parallel}}\ell(\theta^{\parallel}) ℓ ′ ( θ ∥ ) := d θ ∥ d ℓ ( θ ∥ ) . We regard H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) as an operator on U \mathcal{U} U and define λ 1 ( θ ∥ ) : = λ max ( H ( θ ∥ ) ∣ U ) \lambda_1(\theta^{\parallel}) := \lambda_{\max}(H(\theta^{\parallel})|_{\mathcal{U}}) λ 1 ( θ ∥ ) := λ m a x ( H ( θ ∥ ) ∣ U ) . All eigenvalues and Loewner inequalities involving H H H below are understood on U \mathcal{U} U . We call λ 1 \lambda_1 λ 1 the restricted sharpness; it need not equal the largest eigenvalue of the full Hessian at the current iterate.
The structure mirrors the warmup dynamics. (4) governs the orthogonal dynamics : if we neglect the higher order terms, then θ t ⊥ \theta_t^{\perp} θ t ⊥ expands or contracts depending on whether the eigenvalues of η H ( θ t ∥ ) \eta H(\theta_t^{\parallel}) η H ( θ t ∥ ) exceed or fall below 1. (5) governs the parallel dynamics : the parallel direction gradient − η ℓ ′ ( θ t ∥ ) -\eta\ell'(\theta_t^{\parallel}) − η ℓ ′ ( θ t ∥ ) provides one potential source of forward drift, while the quadratic coupling − η 2 ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ -\frac{\eta}{2}(\theta_t^{\perp})^{\mathsf{T}}H'(\theta_t^{\parallel})\theta_t^{\perp} − 2 η ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ converts orthogonal oscillation energy into additional forward progress.
The coupling term − η 2 ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ -\frac{\eta}{2}(\theta_t^{\perp})^{\mathsf{T}}H'(\theta_t^{\parallel})\theta_t^{\perp} − 2 η ( θ t ⊥ ) T H ′ ( θ t ∥ ) θ t ⊥ is the source of self-stabilization : this third-order derivative of the loss translates orthogonal oscillations into forward motion along M \mathcal{M} M , which in turn reduces the curvature and damps the oscillations.
The remainders in (4)–(5) are a potential obstacle if θ t ⊥ \theta_t^{\perp} θ t ⊥ grows during the unstable phase. The next section introduces structural assumptions that control these remainders and convert the approximate coupled system into a form amenable to a rigorous convergence analysis.
Convergence Analysis
The warmup example and the subspace decomposition in § 3 identified several ingredients that drive the self-stabilization mechanism. We distill them into five assumptions, organized in two categories (Table 2 ). The first encodes the geometric structure of the loss landscape: a clean subspace decomposition, where the curvature is well-approximated by its subspace value inside a tubular neighborhood, and a parallel direction along which both the loss and the sharpness decrease. The second category provides the quantitative decay conditions needed to establish convergence rates: a local guarantee that the curvature cannot plateau in the unstable phase, and an asymptotic decay condition for the stable phase.
Table 2. Summary of the structural assumptions used in the convergence analysis.
Assumption A (Valley Geometry)
A.1. Subspace Alignment. L \mathcal{L} L is C 4 C^4 C 4 . Moreover, for all θ ∥ ∈ R \theta^\parallel\in\mathbb{R} θ ∥ ∈ R ,
∇ L ( θ ∥ v ) ∈ M , Π U ∇ 2 L ( θ ∥ v ) v = 0. (6) \nabla\mathcal{L}(\theta^\parallel v) \in\mathcal{M}, \qquad\Pi_U \nabla^2 \mathcal{L}(\theta^\parallel v)v = 0. \tag*{(6)} ∇ L ( θ ∥ v ) ∈ M , Π U ∇ 2 L ( θ ∥ v ) v = 0. (6) A.2. Relative Curvature Comparability. There exists θ ~ ∥ \widetilde{\theta}^\parallel θ ∥ such that for every sufficiently small r > 0 r > 0 r > 0 , there exists δ r ∈ [ 0 , 1 ) \delta_r \in[0,1) δ r ∈ [ 0 , 1 ) such that for all θ ∥ ≥ θ ~ ∥ \theta^\parallel\geq\widetilde{\theta}^\parallel θ ∥ ≥ θ ∥ and θ ⊥ ∈ U \theta^\perp\in U θ ⊥ ∈ U with ∥ θ ⊥ ∥ ≤ r \lVert\theta^\perp\rVert\leq r ∥ θ ⊥ ∥ ≤ r ,
( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ∥ v + θ ⊥ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) . (7) (1-\delta_r)H(\theta^\parallel) \preceq D_{\theta^\perp}\nabla\mathcal{L}^\perp(\theta^\parallel v+\theta^\perp) \preceq(1+\delta_r)H(\theta^\parallel). \tag*{(7)} ( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ∥ v + θ ⊥ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) . (7) A.3. Loss–Curvature Compatibility. There exists θ ‾ ∥ \overline{\theta}^\parallel θ ∥ such that for all θ ∥ ≥ θ ‾ ∥ \theta^\parallel\geq\overline{\theta}^\parallel θ ∥ ≥ θ ∥ : (a) − ℓ ′ ( θ ∥ ) > 0 -\ell'(\theta^\parallel)>0 − ℓ ′ ( θ ∥ ) > 0 (the parallel direction gradient drives θ ∥ \theta^\parallel θ ∥ forward); (b) H ( θ ∥ ) ≻ 0 H(\theta^\parallel)\succ0 H ( θ ∥ ) ≻ 0 and H ′ ( θ ∥ ) ⪯ 0 H'(\theta^\parallel)\preceq0 H ′ ( θ ∥ ) ⪯ 0 (restricted hessian is positive definite and non-increasing on M \mathcal{M} M ).
A.1. gives the correct expansion orders: linear in θ ⊥ \theta^\perp θ ⊥ for the orthogonal update and quadratic for the parallel one. A.2. upgrades the orthogonal dynamics to the exact multiplicative form
θ t + 1 ⊥ = ( I − η H ~ t ) θ t ⊥ , ( 1 − δ r ) H ( θ t ∥ ) ⪯ H ~ t ⪯ ( 1 + δ r ) H ( θ t ∥ ) , \theta_{t+1}^\perp=(I-\eta\widetilde{H}_t)\theta_t^\perp,\qquad(1-\delta_r)H(\theta_t^\parallel)\preceq\widetilde{H}_t\preceq(1+\delta_r)H(\theta_t^\parallel), θ t + 1 ⊥ = ( I − η H t ) θ t ⊥ , ( 1 − δ r ) H ( θ t ∥ ) ⪯ H t ⪯ ( 1 + δ r ) H ( θ t ∥ ) ,
where δ r \delta_r δ r measures the tubular neighborhood-curvature distortion. A.3. provides self-stabilization: forward motion in θ ∥ \theta^\parallel θ ∥ decreases the curvature until it falls below the threshold 2 / η 2/\eta 2/ η . Together with the remainder control, local curvature decay, and tail assumptions below, these properties yield finite entry into a region of orthogonal nonexpansion. In the warmup, δ r = 0 \delta_r=0 δ r = 0 and the loss restricted to the parallel direction vanishes, so forward progress comes solely from the curvature-reduction coupling.
Assumption B (Decay Profile)
B.1. Local Curvature Decay. For every bounded interval J ⊆ [ θ ‾ ∥ , ∞ ) J\subseteq[\overline{\theta}^\parallel,\infty) J ⊆ [ θ ∥ , ∞ ) , there exists β J > 0 \beta_J>0 β J > 0 such that − H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) -H'(\theta^\parallel)\succeq\beta_JH(\theta^\parallel) − H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) for all θ ∥ ∈ J \theta^\parallel\in J θ ∥ ∈ J .
B.2. Exponential Tail. For every tubular neighborhood radius r > 0 r>0 r > 0 , there exists an asymptotic rate β t a i l ( r ) > 0 \beta_{\mathrm{tail}}(r)>0 β tail ( r ) > 0 and a threshold θ t a i l ∥ ( r ) \theta_{\mathrm{tail}}^\parallel(r) θ tail ∥ ( r ) such that, uniformly for all ∥ θ ⊥ ∥ ≤ r \lVert\theta^\perp\rVert\leq r ∥ θ ⊥ ∥ ≤ r ,
− ⟨ ∇ L ∥ ( θ ) , v ⟩ ≍ r e − β t a i l ( r ) ( θ ∥ − θ t a i l ∥ ( r ) ) , λ 1 ( θ ∥ ) ≍ r e − β t a i l ( r ) ( θ ∥ − θ t a i l ∥ ( r ) ) (8) -\langle\nabla\mathcal{L}^\parallel(\theta),v\rangle\asymp_r e^{-\beta_{\mathrm{tail}}(r)(\theta^\parallel-\theta_{\mathrm{tail}}^\parallel(r))},\qquad\lambda_1(\theta^\parallel)\asymp_r e^{-\beta_{\mathrm{tail}}(r)(\theta^\parallel-\theta_{\mathrm{tail}}^\parallel(r))} \tag*{(8)} − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≍ r e − β tail ( r ) ( θ ∥ − θ tail ∥ ( r )) , λ 1 ( θ ∥ ) ≍ r e − β tail ( r ) ( θ ∥ − θ tail ∥ ( r )) (8) for all θ ∥ ≥ θ t a i l ∥ ( r ) \theta^\parallel\geq\theta_{\mathrm{tail}}^\parallel(r) θ ∥ ≥ θ tail ∥ ( r ) . Here f ≍ r g f\asymp_r g f ≍ r g means that there exist constants 0 < c r ≤ C r < ∞ 0<c_r\leq C_r<\infty 0 < c r ≤ C r < ∞ , depending only on r r r , such that c r g ≤ f ≤ C r g c_rg\leq f\leq C_rg c r g ≤ f ≤ C r g on the stated domain.
B.1. strengthens A.3.(b) by requiring the curvature to decrease at a definite rate on every compact interval, ruling out plateaus before the stability threshold. B.2. requires that the parallel gradient and sharpness decay exponentially in θ ∥ \theta^\parallel θ ∥ at the same rate β t a i l \beta_{\mathrm{tail}} β tail .
Remark 1 (Uniform conditioning). When dim U > 1 \dim U>1 dim U > 1 , contraction of the orthogonal component requires that the restricted Hessian satisfy for all θ ∥ \theta^\parallel θ ∥ : H ( θ ∥ ) ⪰ ρ λ 1 ( θ ∥ ) I U H(\theta^\parallel)\succeq\rho\lambda_1(\theta^\parallel)I_U H ( θ ∥ ) ⪰ ρ λ 1 ( θ ∥ ) I U for some ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) , i.e., κ ( H ( θ ∥ ) ∣ U ) ≤ ρ − 1 \kappa(H(\theta^\parallel)|_U)\leq\rho^{-1} κ ( H ( θ ∥ ) ∣ U ) ≤ ρ − 1 . This fails in geometrically interesting cases (§ 5 ), so rather than imposing it as a blanket assumption, we state it as an explicit hypothesis whenever it is needed.
We now state the main convergence theorem, which gives a quantitative version of the coupled dynamics (4)–(5) in § 3 . Equation (5) implicitly performs gradient descent on the restricted curvature: forward motion in θ t ∥ \theta_t^{\parallel} θ t ∥ decreases the sharpness and eventually the stability threshold is reached. Equation (4) then translates this change in the curvature into orthogonal dynamics, distinguishing contraction, expansion, and nonexpansion according to the spectrum of I − η H ( θ t ∥ ) I-\eta H(\theta_t^{\parallel}) I − η H ( θ t ∥ ) . Figure 2 illustrates the different phases of the dynamics.
Admissibility. Fix a tubular neighborhood radius r > 0 r>0 r > 0 and define the stability threshold
θ ∗ ∥ ( r , η ) : = inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } . (9) \theta_*^{\parallel}(r,\eta):=\inf\{\theta^{\parallel}\geq\theta_0^{\parallel}:\eta(1+\delta_r)\lambda_1(\theta^{\parallel})\leq2\}. \tag*{(9)} θ ∗ ∥ ( r , η ) := inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } . (9)
Since r r r and η \eta η are fixed throughout the analysis, we write simply θ ∗ ∥ \theta_*^{\parallel} θ ∗ ∥ hereafter. The triple ( η , r , θ 0 ) (\eta,r,\theta_0) ( η , r , θ 0 ) is called admissible if several requirements hold simultaneously: the initialization is unstable η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) > 2 \eta(1+\delta_r)\lambda_1(\theta_0^{\parallel})>2 η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) > 2 and starts inside the tubular neighborhood; the radius r r r is small enough that the Taylor remainder in equation (5) does not overwhelm the curvature-reduction mechanism; and all the orthogonal oscillations fit inside the tubular neighborhood. These requirements impose a large but not too large step size: η \eta η must induce oscillations while keeping them within the perturbative region where curvature comparability and Taylor approximations hold. Precise constraints on r r r and η \eta η appear in Appendix § 10 .§ 2 .
Figure 2. Roadmap of Theorem 1.
Theorem 1 (Convergence of GD). Suppose Assumptions A and B hold with θ ∗ ∥ ≥ θ t a i l ( r ) \theta_*^{\parallel}\geq\theta_{\mathrm{tail}}(r) θ ∗ ∥ ≥ θ tail ( r ) and that the triple ( η , r , θ 0 ) (\eta,r,\theta_0) ( η , r , θ 0 ) is admissible with θ 0 ∥ ≥ max ( θ ˉ ∥ , θ ^ ∥ ) \theta_0^{\parallel}\geq\max(\bar{\theta}^{\parallel},\hat{\theta}^{\parallel}) θ 0 ∥ ≥ max ( θ ˉ ∥ , θ ^ ∥ ) . The dynamics have two main phases, separated by the first time the parallel coordinate reaches the stability threshold, t s t a b : = inf { t ≥ 0 : θ t ∥ ≥ θ ∗ ∥ } t_{\mathrm{stab}}:=\inf\{t\geq0:\theta_t^{\parallel}\geq\theta_*^{\parallel}\} t stab := inf { t ≥ 0 : θ t ∥ ≥ θ ∗ ∥ } .
I. Unstable phase (0 ≤ t ≤ t s t a b 0\leq t\leq t_{\mathrm{stab}} 0 ≤ t ≤ t stab ). The iterates remain in the tubular neighborhood: ∥ θ t ⊥ ∥ ≤ r \|\theta_t^{\perp}\|\leq r ∥ θ t ⊥ ∥ ≤ r for all t ≤ t s t a b t\leq t_{\mathrm{stab}} t ≤ t stab . The parallel coordinate grows at least linearly and the sharpness decays exponentially:
θ t ∥ − θ 0 ∥ ≳ t , λ 1 ( θ t ∥ ) ≲ λ 1 ( θ 0 ∥ ) e − c t , 0 ≤ t ≤ t s t a b , (10) \theta_t^{\parallel}-\theta_0^{\parallel}\gtrsim t,\qquad\lambda_1(\theta_t^{\parallel})\lesssim\lambda_1(\theta_0^{\parallel})e^{-ct},\qquad0\leq t\leq t_{\mathrm{stab}}, \tag*{(10)} θ t ∥ − θ 0 ∥ ≳ t , λ 1 ( θ t ∥ ) ≲ λ 1 ( θ 0 ∥ ) e − c t , 0 ≤ t ≤ t stab , (10) for a constant c > 0 c>0 c > 0 . In particular, t s t a b < ∞ t_{\mathrm{stab}}<\infty t stab < ∞ and η ( 1 + δ r ) λ 1 ( θ t s t a b ∥ ) ≤ 2 \eta(1+\delta_r)\lambda_1(\theta_{t_{\mathrm{stab}}}^{\parallel})\leq2 η ( 1 + δ r ) λ 1 ( θ t stab ∥ ) ≤ 2 .
II. Stable phase (t ≥ t s t a b t\geq t_{\mathrm{stab}} t ≥ t stab ). The parallel coordinate grows at least logarithmically and the sharpness decays sublinearly:
θ t ∥ − θ 0 ∥ = Ω ( log ( t ) ) , λ 1 ( θ t ∥ ) = O ( 1 / t ) . (11) \theta_t^{\parallel}-\theta_0^{\parallel}=\Omega(\log(t)),\qquad\lambda_1(\theta_t^{\parallel})=\mathcal{O}(1/t). \tag*{(11)} θ t ∥ − θ 0 ∥ = Ω ( log ( t )) , λ 1 ( θ t ∥ ) = O ( 1/ t ) . (11) Assume in addition that there exists ρ t a i l ∈ ( 0 , 1 ] \rho_{\mathrm{tail}}\in(0,1] ρ tail ∈ ( 0 , 1 ] such that H ( θ ∥ ) ⪰ ρ t a i l λ 1 ( θ ∥ ) I U H(\theta^{\parallel})\succeq\rho_{\mathrm{tail}}\lambda_1(\theta^{\parallel})I_U H ( θ ∥ ) ⪰ ρ tail λ 1 ( θ ∥ ) I U for all θ ∥ ≥ θ ∗ ∥ \theta^{\parallel}\geq\theta_*^{\parallel} θ ∥ ≥ θ ∗ ∥ . Then the stable phase can be further divided into three regimes according to the corrected sharpness. Let ε ∗ : = ρ t a i l ( 1 − δ r ) / ( 1 + δ r ) \varepsilon_*:=\rho_{\mathrm{tail}}(1-\delta_r)/(1+\delta_r) ε ∗ := ρ tail ( 1 − δ r ) / ( 1 + δ r ) be the contraction margin, and let t l i n t_{\mathrm{lin}} t lin and t p o l y t_{\mathrm{poly}} t poly denote the first times after stabilization at which the corrected sharpness is at most 2 − ε ∗ 2-\varepsilon_* 2 − ε ∗ and 1 1 1 , respectively:
t l i n : = inf { t ≥ t s t a b : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 2 − ε ∗ } , t p o l y : = inf { t ≥ t s t a b : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 1 } . t_{\mathrm{lin}}:=\inf\{t\geq t_{\mathrm{stab}}:\eta(1+\delta_r)\lambda_1(\theta_t^{\parallel})\leq2-\varepsilon_*\},\qquad
t_{\mathrm{poly}}:=\inf\{t\geq t_{\mathrm{stab}}:\eta(1+\delta_r)\lambda_1(\theta_t^{\parallel})\leq1\}. t lin := inf { t ≥ t stab : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 2 − ε ∗ } , t poly := inf { t ≥ t stab : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 1 } . These times satisfy t s t a b ≤ t l i n ≤ t p o l y < ∞ t_{\mathrm{stab}}\leq t_{\mathrm{lin}}\leq t_{\mathrm{poly}}<\infty t stab ≤ t lin ≤ t poly < ∞ , with equality possible at either boundary.
(i) Transition Phase t ∈ [ t s t a b , t l i n ] t\in[t_{\mathrm{stab}},t_{\mathrm{lin}}] t ∈ [ t stab , t lin ] : the oscillations are nonexpanding, ∥ θ t ⊥ ∥ ≤ ∥ θ t s t a b ⊥ ∥ \|\theta_t^{\perp}\|\leq\|\theta_{t_{\mathrm{stab}}}^{\perp}\| ∥ θ t ⊥ ∥ ≤ ∥ θ t stab ⊥ ∥ .
(ii) Linear Contraction Phase t ∈ [ t l i n , t p o l y ] t\in[t_{\mathrm{lin}},t_{\mathrm{poly}}] t ∈ [ t lin , t poly ] : ∥ θ t ⊥ ∥ ≤ ( 1 − ε ∗ ) t − t l i n ∥ θ t l i n ⊥ ∥ \|\theta_t^{\perp}\|\leq(1-\varepsilon_*)^{t-t_{\mathrm{lin}}}\|\theta_{t_{\mathrm{lin}}}^{\perp}\| ∥ θ t ⊥ ∥ ≤ ( 1 − ε ∗ ) t − t lin ∥ θ t lin ⊥ ∥ with ε ∗ : = ρ t a i l ( 1 − δ r ) 1 + δ r \varepsilon_*:=\dfrac{\rho_{\mathrm{tail}}(1-\delta_r)}{1+\delta_r} ε ∗ := 1 + δ r ρ tail ( 1 − δ r ) .
(iii) Polynomial Decay Phase ( t ≥ t p o l y ) (t\geq t_{\mathrm{poly}}) ( t ≥ t poly ) : ∥ θ t ⊥ ∥ = O ( ( t − t p o l y ) − γ r ) \|\theta_t^{\perp}\|=\mathcal{O}((t-t_{\mathrm{poly}})^{-\gamma_r}) ∥ θ t ⊥ ∥ = O (( t − t poly ) − γ r ) with γ r = ρ t a i l ( 1 − δ r ) C t a i l ( r ) \gamma_r=\rho_{\mathrm{tail}}(1-\delta_r)C_{\mathrm{tail}}(r) γ r = ρ tail ( 1 − δ r ) C tail ( r ) and C t a i l ( r ) C_{\mathrm{tail}}(r) C tail ( r ) a positive constant determined by the tail bounds and the tubular neighborhood tolerance.
After the iterates stabilize, the orthogonal component initially contracts linearly, but this rate is self-limiting: the parallel motion that stabilizes the dynamics simultaneously drives the restricted
curvature to zero, yielding the polynomial tail t − γ r t^{-\gamma_r} t − γ r . The phase boundaries are determined by the corrected sharpness η ( 1 + δ r ) λ 1 ( θ t ∥ ) \eta(1+\delta_r)\lambda_1(\theta_t^{\parallel}) η ( 1 + δ r ) λ 1 ( θ t ∥ ) . During the unstable phase, this quantity exceeds 2; the orthogonal dynamics can therefore be locally expansive. Self-stabilization drives θ t ∥ \theta_t^{\parallel} θ t ∥ forward and decreases the sharpness until the threshold 2 is crossed at t s t a b t_{\mathrm{stab}} t stab . After stabilization, crossing 2 − ε ∗ 2-\varepsilon_* 2 − ε ∗ at t l i n t_{\mathrm{lin}} t lin provides a uniform contraction margin, yielding geometric decay of the orthogonal component. Finally, when the corrected sharpness falls below 1 at t p o l y t_{\mathrm{poly}} t poly , the per-step contraction becomes positive but progressively weaker as the sharpness tends to zero; the resulting product of contraction factors yields the polynomial tail. The conditioning constant ρ t a i l \rho_{\mathrm{tail}} ρ tail separates two usually conflated effects: λ 1 \lambda_1 λ 1 determines when the iterates stabilize, while ρ t a i l \rho_{\mathrm{tail}} ρ tail determines whether all orthogonal directions are damped comparably. This distinction is central in Section 5. Finally, we note that the iterate schedules can be converted into convergence rates for the loss (see Appendix B.5).
Instantiations of the Theory and Their Consequences
This section instantiates our general framework on canonical models from machine learning. Specifically, we verify the structural assumptions of Section 4 for standard losses such as logistic and cross-entropy, and illustrate how the general theory yields sharper, model-specific insights, including geometry-dependent effects not visible at the level of the abstract theory.
Verification for pair-symmetric logistic regression. Fix a unit vector v ∈ R d v \in\mathbb{R}^d v ∈ R d . Consider N N N points x i + = a i v + u i x_i^+ = a_i v + u_i x i + = a i v + u i with a i > 0 a_i > 0 a i > 0 , u i ∈ span { v } ⊥ ∖ { 0 } u_i \in\operatorname{span}\{v\}^{\perp} \setminus\{0\} u i ∈ span { v } ⊥ ∖ { 0 } , labeled + 1 +1 + 1 , and their reflections across span { v } ⊥ \operatorname{span}\{v\}^{\perp} span { v } ⊥ , x i − = − a i v + u i x_i^- = -a_i v + u_i x i − = − a i v + u i , labeled − 1 -1 − 1 . Writing θ = θ ∥ v + θ ⊥ \theta= \theta^{\parallel}v + \theta^{\perp} θ = θ ∥ v + θ ⊥ with θ ⊥ ∈ U : = span { u 1 , … , u N } \theta^{\perp} \in U := \operatorname{span}\{u_1,\ldots,u_N\} θ ⊥ ∈ U := span { u 1 , … , u N } , the logistic loss is
L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ( a i θ ∥ + ⟨ u i , θ ⊥ ⟩ ) + ϕ ( a i θ ∥ − ⟨ u i , θ ⊥ ⟩ ) ] , ϕ ( z ) : = log ( 1 + e − z ) , (12) \mathcal{L}(\theta) = \frac{1}{2N}\sum_{i=1}^{N}\left[\phi\left(a_i\theta^{\parallel}+\langle u_i,\theta^{\perp}\rangle\right)+\phi\left(a_i\theta^{\parallel}-\langle u_i,\theta^{\perp}\rangle\right)\right],\qquad\phi(z):=\log(1+e^{-z}), \tag*{(12)} L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ( a i θ ∥ + ⟨ u i , θ ⊥ ⟩ ) + ϕ ( a i θ ∥ − ⟨ u i , θ ⊥ ⟩ ) ] , ϕ ( z ) := log ( 1 + e − z ) , (12)
The pair symmetry θ ⊥ ↦ − θ ⊥ \theta^{\perp}\mapsto-\theta^{\perp} θ ⊥ ↦ − θ ⊥ leaves each summand in (12) invariant, so the subspace M = span { v } \mathcal{M}=\operatorname{span}\{v\} M = span { v } is precisely the max-margin line. The restricted Hessian matrix is
H ( θ ∥ ) = 1 N ∑ i ϕ ′ ′ ( a i θ ∥ ) u i u i T . (13) H(\theta^{\parallel})=\frac{1}{N}\sum_i\phi^{\prime\prime}(a_i\theta^{\parallel})u_i u_i^{\mathsf{T}}. \tag*{(13)} H ( θ ∥ ) = N 1 i ∑ ϕ ′′ ( a i θ ∥ ) u i u i T . (13)
Set a min : = min i a i a_{\min}:=\min_i a_i a m i n := min i a i and u max : = max i ∥ u i ∥ u_{\max}:=\max_i\lVert u_i\rVert u m a x := max i ∥ u i ∥ . The set I ∗ : = { i : a i = a min } I_*:=\{i:a_i=a_{\min}\} I ∗ := { i : a i = a m i n } is, by construction, the set of indices corresponding to support vectors. Although our numerical illustrations use d = 3 d=3 d = 3 for visualization, the pair-symmetric construction and the verification below hold in arbitrary ambient dimension; in particular, the active orthogonal subspace U = span { u i } U=\operatorname{span}\{u_i\} U = span { u i } may have arbitrary dimension.
Proposition 1 (Logistic Verification). For every fixed θ ∥ > 0 \theta^{\parallel}>0 θ ∥ > 0 , the pair-symmetric logistic loss (14) satisfies Assumptions A and B with this choice of lower threshold in Assumptions A.3–B.1. The following constants may be used:
RC tolerance: δ r = cosh ( u max r ) − 1 = 1 2 ( u max r ) 2 + O ( r 4 ) \delta_r=\cosh(u_{\max}r)-1=\frac{1}{2}(u_{\max}r)^2+O(r^4) δ r = cosh ( u m a x r ) − 1 = 2 1 ( u m a x r ) 2 + O ( r 4 ) , with 0 < r < arccosh ( 2 ) u max 0<r<\frac{\operatorname{arccosh}(2)}{u_{\max}} 0 < r < u m a x arccosh ( 2 ) .
Exponential tail rate: β t a i l = a min \beta_{\mathrm{tail}}=a_{\min} β tail = a m i n .
Uniform conditioning: If the spanning condition U = span { u i : i ∈ I ∗ } U=\operatorname{span}\{u_i:i\in I_*\} U = span { u i : i ∈ I ∗ } holds, then H ( θ ∥ ) ⪰ ρ t a i l λ 1 ( θ ∥ ) I U H(\theta^{\parallel})\succeq\rho_{\mathrm{tail}}\lambda_1(\theta^{\parallel})I_U H ( θ ∥ ) ⪰ ρ tail λ 1 ( θ ∥ ) I U for all θ ∥ ≥ 0 \theta^{\parallel}\geq0 θ ∥ ≥ 0 with
ρ t a i l = λ min ( Σ ∗ ∣ U ) λ max ( Σ ∣ U ) , \rho_{\mathrm{tail}}=\frac{\lambda_{\min}(\Sigma_*\vert_U)}{\lambda_{\max}(\Sigma\vert_U)}, ρ tail = λ m a x ( Σ ∣ U ) λ m i n ( Σ ∗ ∣ U ) , where Σ : = 1 N ∑ i u i u i T \Sigma:=\frac{1}{N}\sum_i u_i u_i^{\mathsf{T}} Σ := N 1 ∑ i u i u i T , Σ ∗ : = 1 N ∑ i ∈ I ∗ u i u i T \Sigma_*:=\frac{1}{N}\sum_{i\in I_*}u_i u_i^{\mathsf{T}} Σ ∗ := N 1 ∑ i ∈ I ∗ u i u i T .
Explicit expressions for all remaining constants are given in Appendix C.
Extension to multiclass cross-entropy. In the multiclass model, the two binary class directions ± v \pm v ± v are replaced by the vertices r 1 , … , r K r_1,\ldots,r_K r 1 , … , r K of a centered regular simplex. For each i i i , the examples x c , i = b i r c + u i x_{c,i}=b_i r_c+u_i x c , i = b i r c + u i share the same offset component u i u_i u i across all classes. Writing R = [ r 1 , … , r K ] R=[r_1,\ldots,r_K] R = [ r 1 , … , r K ] , the parallel direction is now the line M = span { R } \mathcal{M}=\operatorname{span}\{R\} M = span { R } in weight-matrix space. Its positive ray is generated by the Crammer–Singer hard-margin solution [40 , 41 ] .
By symmetry, the gradient on the line is parallel to R R R , so the parallel dynamics again reduce to a single scalar. The orthogonal dynamics are richer: even when u i = 0 u_i = 0 u i = 0 for every i i i , perturbations within the simplex span can change the relative orientations of the class weights. Their curvature is governed by the softmax covariance of the class-logit perturbations, which plays the role of the scalar logistic curvature in the binary analysis. Full statements and proofs are in Appendix § 12 .
Effects of spanning geometry on the dynamics. The convergence behavior of the orthogonal component depends on whether the orthogonal components of the support vectors span the orthogonal subspace U U U . To quantify this, define the conditioning ratio ρ ( θ ℓ ) : = λ min ( H ( θ ℓ ) ) / λ max ( H ( θ ℓ ) ) ∈ ( 0 , 1 ] \rho(\theta^{\ell}) := \lambda_{\min}(H(\theta^{\ell}))/\lambda_{\max}(H(\theta^{\ell})) \in(0,1] ρ ( θ ℓ ) := λ m i n ( H ( θ ℓ )) / λ m a x ( H ( θ ℓ )) ∈ ( 0 , 1 ] . In the spanning case , ρ \rho ρ stays bounded away from zero and the top eigenvalue λ 1 \lambda_1 λ 1 governs contraction of the full orthogonal component. In the non-spanning case , ρ → 0 \rho\to0 ρ → 0 as ∥ θ ℓ ∥ → ∞ \|\theta^{\ell}\| \to\infty ∥ θ ℓ ∥ → ∞ : different orthogonal directions contract at different rates (Figure 4 ), and the slowest may receive only a finite cumulative contraction budget. Both pair-symmetric logistic regression and its multiclass cross-entropy analogue exhibit this dichotomy; we present the binary derivation and defer the multiclass case to Appendix § 12 .
Figure 4. Non-spanning logistic regression at step sizes η / η c ∈ { 1 , 1.5 , 2 , 2.5 , 3 } \eta/\eta_c \in\{1, 1.5, 2, 2.5, 3\} η / η c ∈ { 1 , 1.5 , 2 , 2.5 , 3 } , where η c : = 2 / λ max ( ∇ 2 L ( θ 0 ) ) \eta_c := 2/\lambda_{\max}(\nabla^2\mathcal{L}(\theta_0)) η c := 2/ λ m a x ( ∇ 2 L ( θ 0 )) . (a) Loss. (b) Angle to max-margin line M \mathcal{M} M . (c) Conditioning ratio ρ t \rho_t ρ t . (d) Weak component ∣ c w k ∣ |c_{\mathrm{wk}}| ∣ c wk ∣ . (e) Strong component ∣ c s v ∣ |c_{\mathrm{sv}}| ∣ c sv ∣ . (f) Parallel coordinate.
Let α 1 < ⋯ < α L \alpha_1 < \cdots< \alpha_L α 1 < ⋯ < α L be the distinct values among a 1 , … , a N a_1, \ldots, a_N a 1 , … , a N , and define the nested subspaces
U ( 1 ) ⊆ U ( 2 ) ⊆ ⋯ ⊆ U ( L ) = U , U ( j ) : = span { u i : a i ≤ α j } U^{(1)} \subseteq U^{(2)} \subseteq\cdots\subseteq U^{(L)} = U,\qquad U^{(j)} := \operatorname{span}\{u_i : a_i \le\alpha_j\} U ( 1 ) ⊆ U ( 2 ) ⊆ ⋯ ⊆ U ( L ) = U , U ( j ) := span { u i : a i ≤ α j }
Let j ∗ : = min { j : U ( j ) = U } j_* := \min\{j : U^{(j)} = U\} j ∗ := min { j : U ( j ) = U } . Recalling that the support vectors of the max-margin classifier are exactly the examples { x i ± : i ∈ I ∗ } \{x_i^{\pm} : i \in I_*\} { x i ± : i ∈ I ∗ } , we have j ∗ = 1 j_* = 1 j ∗ = 1 if and only if the orthogonal components of the support vectors span U U U .
Theorem 2 (Spanning vs. non-spanning geometry ). In the stable phase, the following holds:
(i) If j ∗ = 1 j_* = 1 j ∗ = 1 , then ρ ( θ t ℓ ) ≍ 1 \rho(\theta_t^{\ell}) \asymp1 ρ ( θ t ℓ ) ≍ 1 and the three-phase structure of Theorem 1(II) applies;
(ii) If j ∗ > 1 j_* > 1 j ∗ > 1 , then ρ ( θ t ℓ ) ≍ t − ( α j ∗ − α 1 ) / α 1 \rho(\theta_t^{\ell}) \asymp t^{-(\alpha_{j_*}-\alpha_1)/\alpha_1} ρ ( θ t ℓ ) ≍ t − ( α j ∗ − α 1 ) / α 1 . Under an additional simultaneous diagonalizability assumption, explicit componentwise rates can be derived (see Appendix § 11 .§ 4 ).
Transient implicit bias induced by non-spanning geometry. A key implication of our analysis is the emergence of a transient implicit bias: although GD converges asymptotically to the max-margin direction, the iterates can exhibit persistent deviations due to orthogonal components that are amplified during the unstable phase and decay only partially in the stable phase. This phenomenon arises specifically in the non-spanning case (j ∗ > 1 j_* > 1 j ∗ > 1 ): Theorem 2 shows that the conditioning ratio ρ ( θ t ℓ ) \rho(\theta_t^{\ell}) ρ ( θ t ℓ ) decays fast enough that the cumulative contraction budget ∑ t ρ ( θ t ℓ ) λ 1 ( θ t ℓ ) \sum_t \rho(\theta_t^{\ell})\lambda_1(\theta_t^{\ell}) ∑ t ρ ( θ t ℓ ) λ 1 ( θ t ℓ ) remains finite, in contrast to the spanning case where it diverges and drives ∥ θ t ℓ ∥ → 0 \|\theta_t^{\ell}\| \to0 ∥ θ t ℓ ∥ → 0 . As a result, GD may temporarily favor directions outside the max-margin subspace, leading to a geometry-dependent bias controlled by the step size η \eta η : two competing effects — oscillatory amplification and early stable-phase contraction — can make ∥ θ t ℓ ∥ \|\theta_t^{\ell}\| ∥ θ t ℓ ∥ increasing, decreasing, or non-monotone in η \eta η depending on the data geometry and initialization. We exhibit the increasing case in § 6 and defer the others to Appendix § 13 .
Empirical Validation
Trajectory structure and three-phase decay. We illustrate the trajectory structure predicted by Theorem 1 on a spanning logistic model in R 3 \mathbb{R}^3 R 3 with N = 24 N = 24 N = 24 data pairs and dim U = 2 \dim U = 2 dim U = 2 (setup details in Appendix § 13 .§ 1 ). Figure 3 confirms the full predicted trajectory: oscillatory growth inside the tubular neighborhood, followed by linear contraction at rate ( 1 − ε ∗ ) t (1-\varepsilon_*)^t ( 1 − ε ∗ ) t and polynomial decay with exponent γ r = 0.84 \gamma_r = 0.84 γ r = 0.84 . The linear bound is tight initially but loosens as η λ 1 \eta\lambda_1 η λ 1 shrinks below the worst-case threshold.
Figure 3. Unstable and stable phases in a spanning logistic model
Non-spanning geometry and transient implicit bias. We next illustrate the geometry analyzed in Theorem 2 on a non-spanning dataset where the support vectors do not span U U U (setup in Appendix § 13 .§ 1 ). Figure 4 reveals a componentwise split: the support-vector component ∣ c s v ∣ |c_{\mathrm{sv}}| ∣ c sv ∣ contracts through the three-phase mechanism and decays to numerical precision, while the weak component ∣ c w k ∣ |c_{\mathrm{wk}}| ∣ c wk ∣ plateaus at a nonzero value. Crucially, larger η \eta η amplifies ∣ c w k ∣ |c_{\mathrm{wk}}| ∣ c wk ∣ during the unstable phase, and the non-spanning tail dynamics cannot remove this amplification — larger steps leave a larger weak residual. This residual is invisible from the loss alone but manifests as a larger finite-time angle to the max-margin direction, confirming the transient implicit bias predicted by the theory. We leave a systematic investigation of connections to generalization and robustness to future work. Our experiments are intended as controlled tests of the predicted dynamics rather than large-scale validation on deep networks. Exact Hessian formation is not needed by GD itself, and dominant eigenvalues can be estimated using Hessian–vector products; however, identifying and verifying the appropriate reference subspace in modern networks remains an open problem.
Conclusion
We introduced a unified framework for analyzing gradient descent at the edge of stability that converts the qualitative self-stabilization picture into quantitative convergence guarantees for exponential tail losses. The framework not only recovers prior results for logistic regression and cross-entropy classification, but also extends them by providing a unified mechanism that explains and quantifies their large-step dynamics. Crucially, it reveals a phenomenon invisible to prior analyses of GD: under certain data geometries, the step size induces a transient implicit bias that affects the alignment with the max-margin direction. We leave the characterization of this dependence and its role in generalization and robustness to future work, as well as extensions to asymmetric data or deeper architectures that could further broaden the scope of the framework.
Acknowledgments and Disclosure of Funding
This research was supported by NSF 2031985 and Simons Foundation 814201 (Theorinet), ONR MURI 503405-78051 and University of Pennsylvania Startup Funds.
Appendix Roadmap
Appendix A: additional related work.
Appendix B: full proof of Theorem 1.
Appendix C: verification for pair-symmetric logistic regression.
Appendix D: multiclass cross-entropy extension.
Appendix E: additional experiments and setup details.
Appendix F: approximate symmetry.
Additional Related Work on Large Step Size Gradient Descent
Edge of Stability (EoS). The empirical success of training neural networks with learning rates near or above classical stability limits has generated substantial interest in understanding GD beyond the regime predicted by standard smoothness-based analysis. Earlier work had already connected step size, local curvature, and sharpness in neural-network training [12 , 30 ] , related large-learning-rate phenomena such as the catapult mechanism [10 ] , and showed that training can transiently violate monotone descent while still recovering and converging [31 , 32 , 33 , 9 ] . The seminal work of Cohen et al. [11 ] conducted a systematic empirical study of this behavior, showing that GD on deep networks routinely operates near the classical threshold of 2 / λ max 2/\lambda_{\max} 2/ λ m a x , with sharpness rising until it hovers around the stability boundary; they termed this phenomenon the edge of stability . Along with subsequent work [13 , 14 ] , they also documented progressive sharpening, whereby local curvature increases along the training trajectory and pushes the system toward instability even at a fixed step size.
Model-specific analyses and implicit biases. A substantial body of work analyzes the EoS in specific architectures or simplified settings. Studies of diagonal linear and related homogeneous models have shown that large but stable step sizes can change the implicit bias of GD/SGD [34 , 20 , 17 ] , while analyses of two-layer non-homogeneous networks reveal additional step-size-dependent biases [35 ] . Chen and Bruna [23 ] analyze convergence to period-2 orbits through two-step gradient updates in one-dimensional functions, single-neuron models, and matrix factorization, focusing on local mechanisms rather than end-to-end convergence rates. Other works have used minimalist examples or bifurcation theory to probe the dynamics near the stability boundary [22 , 15 , 18 , 16 ] , and analyses of SGD have explored benign oscillations and step-size-dependent selection of global minima [36 ] . Complex phenomena including cycles and chaotic behavior have been documented beyond the EoS [19 , 25 ] , and [37 ] provides impossibility results for global convergence from arbitrary initializations with very large step sizes, while identifying specific conditions under which convergence still occurs. These analyses yield rich insights into particular models but rely on architectural details that make it difficult to extract a unified mechanism.
Proof of Theorem 1
This section gives the full quantitative version of Theorem 1 and its proof. Section B.1 collects the derived constants and states the admissibility conditions that the tubular neighborhood radius r r r , the step size η \eta η and the initialization θ 0 \theta_0 θ 0 must satisfy. Section B.3 establishes tubular neighborhood invariance, monotone drift, and finite entry to the corrected stability regime (Theorem 3). Section B.4 derives the stable regime schedules and the three-phase orthogonal decay (Theorem 4). Together, these two theorems imply Theorem 1 as explained in Remark 2.
Setup and admissibility
Throughout this appendix, Assumptions A and B are in force. Fix a step size η > 0 \eta> 0 η > 0 , an initial condition θ 0 = θ 0 ∥ v + θ 0 ⊥ \theta_0 = \theta_0^{\parallel} v + \theta_0^{\perp} θ 0 = θ 0 ∥ v + θ 0 ⊥ with θ 0 ⊥ ∈ U \theta_0^{\perp} \in U θ 0 ⊥ ∈ U and θ 0 ∥ ≥ θ 0 ∥ \theta_0^{\parallel} \ge\theta_0^{\parallel} θ 0 ∥ ≥ θ 0 ∥ , and a tubular neighborhood radius r > 0 r > 0 r > 0 .
Definition B.1 (Stability threshold and oscillation slab). The stability threshold is
θ ∗ ∥ : = inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } , (14) \theta_*^{\parallel} := \inf\left\{\theta^{\parallel} \ge\theta_0^{\parallel} : \eta(1+\delta_r)\lambda_1(\theta^{\parallel}) \le2\right\}, \tag*{(14)} θ ∗ ∥ := inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } , (14)
and the oscillation slab is
T p r e : = { θ ∥ v + θ ⊥ : θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] , θ ⊥ ∈ U , ∥ θ ⊥ ∥ ≤ r } . \mathcal{T}_{\mathrm{pre}} := \left\{\theta^{\parallel}v+\theta^{\perp} : \theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}],\ \theta^{\perp}\in U,\ \|\theta^{\perp}\|\le r\right\}. T pre := { θ ∥ v + θ ⊥ : θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] , θ ⊥ ∈ U , ∥ θ ⊥ ∥ ≤ r } .
We assume throughout that θ ∗ ∥ < ∞ \theta_*^{\parallel}<\infty θ ∗ ∥ < ∞ . This is automatic under the tail sharpness assumption, since λ 1 ( θ ∥ ) → 0 \lambda_1(\theta^{\parallel})\to0 λ 1 ( θ ∥ ) → 0 exponentially. Since T p r e \mathcal{T}_{\mathrm{pre}} T pre is compact and L \mathcal{L} L is C 4 C^4 C 4 by Assumption A.1., the following constants are finite and determined by L \mathcal{L} L , η \eta η , r r r , and θ 0 ∥ \theta_0^{\parallel} θ 0 ∥ :
m ℓ : = inf θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] ( − ℓ ′ ( θ ∥ ) ) > 0 , (15) m_{\ell} := \inf_{\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}]} \left(-\ell'(\theta^{\parallel})\right)>0, \tag*{(15)} m ℓ := θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] inf ( − ℓ ′ ( θ ∥ ) ) > 0 , (15)
G ∥ : = sup T p r e ⟨ − ∇ L ( θ ) , v ⟩ < ∞ , (16) G_{\parallel} := \sup_{\mathcal{T}_{\mathrm{pre}}}\langle-\nabla\mathcal{L}(\theta),v\rangle<\infty, \tag*{(16)} G ∥ := T pre sup ⟨ − ∇ L ( θ ) , v ⟩ < ∞ , (16)
β : = β [ θ 0 ∥ , θ ∗ ∥ ] , (17) \beta:= \beta_{[\theta_0^{\parallel},\theta_*^{\parallel}]}, \tag*{(17)} β := β [ θ 0 ∥ , θ ∗ ∥ ] , (17)
the log-derivative constant from Assumption B.1..
The positivity of m ℓ m_{\ell} m ℓ follows from A.3. and the compactness of [ θ 0 ∥ , θ ∗ ∥ ] [\theta_0^{\parallel},\theta_*^{\parallel}] [ θ 0 ∥ , θ ∗ ∥ ] . Let c ∥ , r > 0 c_{\parallel,r}>0 c ∥ , r > 0 be a constant such that for all ( θ ∥ , θ ⊥ ) ∈ [ θ 0 ∥ , θ ∗ ∥ ] × U (\theta^{\parallel},\theta^{\perp})\in[\theta_0^{\parallel},\theta_*^{\parallel}]\times U ( θ ∥ , θ ⊥ ) ∈ [ θ 0 ∥ , θ ∗ ∥ ] × U with ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\|\le r ∥ θ ⊥ ∥ ≤ r ,
⟨ ∇ L ( θ ∥ v + θ ⊥ ) , v ⟩ = ℓ ′ ( θ ∥ ) + 1 2 ⟨ H ′ ( θ ∥ ) θ ⊥ , θ ⊥ ⟩ + R ∥ ( θ ∥ , θ ⊥ ) , ∣ R ∥ ( θ ∥ , θ ⊥ ) ∣ ≤ c ∥ , r ∥ θ ⊥ ∥ 3 . (18) \left\langle\nabla\mathcal{L}(\theta^{\parallel}v+\theta^{\perp}),v\right\rangle
=\ell'(\theta^{\parallel})+\frac{1}{2}\left\langle H'(\theta^{\parallel})\theta^{\perp},\theta^{\perp}\right\rangle+R^{\parallel}(\theta^{\parallel},\theta^{\perp}),
\qquad
\left|R^{\parallel}(\theta^{\parallel},\theta^{\perp})\right|\le c_{\parallel,r}\|\theta^{\perp}\|^3. \tag*{(18)} ⟨ ∇ L ( θ ∥ v + θ ⊥ ) , v ⟩ = ℓ ′ ( θ ∥ ) + 2 1 ⟨ H ′ ( θ ∥ ) θ ⊥ , θ ⊥ ⟩ + R ∥ ( θ ∥ , θ ⊥ ) , R ∥ ( θ ∥ , θ ⊥ ) ≤ c ∥ , r ∥ θ ⊥ ∥ 3 . (18)
Such a constant exists by the C 4 C^4 C 4 regularity of L \mathcal{L} L and compactness of T p r e \mathcal{T}_{\mathrm{pre}} T pre .
Uniform conditioning in the unstable phase. By Assumption A.3.(b), H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) is positive definite on U U U . Continuity and compactness therefore give
ρ : = min θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] λ min ( H ( θ ∥ ) ∣ U ) λ 1 ( θ ∥ ) > 0. \rho:= \min_{\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}]}
\frac{\lambda_{\min}(H(\theta^{\parallel})|_U)}{\lambda_1(\theta^{\parallel})}>0. ρ := θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] min λ 1 ( θ ∥ ) λ m i n ( H ( θ ∥ ) ∣ U ) > 0.
Consequently,
H ( θ ∥ ) ⪰ ρ λ 1 ( θ ∥ ) I U ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] . (19) H(\theta^{\parallel})\succeq\rho\lambda_1(\theta^{\parallel})I_U
\qquad\forall\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}]. \tag*{(19)} H ( θ ∥ ) ⪰ ρ λ 1 ( θ ∥ ) I U ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] . (19)
If dim U = 1 \dim U=1 dim U = 1 , then ρ = 1 \rho=1 ρ = 1 . Unlike uniform conditioning on an unbounded tail, this compact-slab bound follows directly from positive definiteness.
Definition B.2 (Admissibility). Define the remainder-smallness parameter
χ : = β − η ( 1 + δ r ) c ∥ , r r ρ , (20) \chi:= \beta-\frac{\eta(1+\delta_r)c_{\parallel,r}r}{\rho}, \tag*{(20)} χ := β − ρ η ( 1 + δ r ) c ∥ , r r , (20)
and the initial instability surplus
C u n s t : = η ( 1 + δ r ) [ η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2 ] . (21) C_{\mathrm{unst}} := \eta(1+\delta_r)\left[\eta(1+\delta_r)\lambda_1(\theta_0^{\parallel})-2\right]. \tag*{(21)} C unst := η ( 1 + δ r ) [ η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2 ] . (21)
The triple ( η , r , θ 0 ) (\eta,r,\theta_0) ( η , r , θ 0 ) is admissible if:
(A1) η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) > 2 \eta(1+\delta_r)\lambda_1(\theta_0^{\parallel})>2 η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) > 2 and ∥ θ 0 ⊥ ∥ ≤ r \|\theta_0^{\perp}\|\le r ∥ θ 0 ⊥ ∥ ≤ r , (corrected instability)
(A2) χ > 0 \chi>0 χ > 0 , (remainder smallness)
(A3) ∥ θ 0 ⊥ ∥ 2 + 2 C u n s t η χ ( θ ∗ ∥ − θ 0 ∥ + η G ∥ ) ≤ r 2 \|\theta_0^{\perp}\|^2+\frac{2C_{\mathrm{unst}}}{\eta\chi}\left(\theta_*^{\parallel}-\theta_0^{\parallel}+\eta G_{\parallel}\right)\le r^2 ∥ θ 0 ⊥ ∥ 2 + η χ 2 C unst ( θ ∗ ∥ − θ 0 ∥ + η G ∥ ) ≤ r 2 . (tubular neighborhood-closing budget)
Non-emptiness of the admissible window
Admissibility carves out a moderately unstable regime: large enough to trigger oscillatory dynamics, but not so large that the local Taylor and curvature-comparability estimates break down. The three conditions are mutually compatible. Conditions (A1) and (A2) together require
2 λ 1 ( θ 0 ∥ ) < η ( 1 + δ r ) < ρ β c ∥ , r r 2 , (22) \frac{2}{\lambda_1(\theta_0^\parallel)} < \eta(1+\delta_r) < \frac{\rho\beta}{c_{\parallel,r}r^2}, \tag*{(22)} λ 1 ( θ 0 ∥ ) 2 < η ( 1 + δ r ) < c ∥ , r r 2 ρβ , (22)
a non-empty open interval whenever
r < ρ β λ 1 ( θ 0 ∥ ) 2 c ∥ , r . r < \frac{\rho\beta\lambda_1(\theta_0^\parallel)}{2c_{\parallel,r}}. r < 2 c ∥ , r ρβ λ 1 ( θ 0 ∥ ) .
In terms of η \eta η itself, this corresponds to
η ∈ ( 2 ( 1 + δ r ) λ 1 ( θ 0 ∥ ) , ρ β ( 1 + δ r ) c ∥ , r r 2 ) . \eta\in\left(\frac{2}{(1+\delta_r)\lambda_1(\theta_0^\parallel)},\frac{\rho\beta}{(1+\delta_r)c_{\parallel,r}r^2}\right). η ∈ ( ( 1 + δ r ) λ 1 ( θ 0 ∥ ) 2 , ( 1 + δ r ) c ∥ , r r 2 ρβ ) .
The constant c ∥ , r c_{\parallel,r} c ∥ , r in (20) depends a priori on the slab [ θ 0 ∥ , θ ∗ ∥ ] [\theta_0^\parallel,\theta_*^\parallel] [ θ 0 ∥ , θ ∗ ∥ ] , hence on η \eta η through θ ∗ ∥ \theta_*^\parallel θ ∗ ∥ . Fix θ 0 ∥ ≥ max { θ ˉ ∥ , θ ~ ∥ } \theta_0^\parallel\ge\max\{\bar{\theta}^\parallel,\tilde{\theta}^\parallel\} θ 0 ∥ ≥ max { θ ˉ ∥ , θ ~ ∥ } and a reference interval
J = [ θ 0 ∥ , θ 0 ∥ + Δ ] , Δ > 0. J=[\theta_0^\parallel,\theta_0^\parallel+\Delta],\qquad\Delta>0. J = [ θ 0 ∥ , θ 0 ∥ + Δ ] , Δ > 0.
Choose a reference radius r 0 > 0 r_0>0 r 0 > 0 on which relative curvature comparability is available. On the fixed reference slab-tubular neighborhood, choose constants β J > 0 \beta_J>0 β J > 0 , ρ J > 0 \rho_J>0 ρ J > 0 , c J < ∞ c_J<\infty c J < ∞ , and G J < ∞ G_J<\infty G J < ∞ such that
− H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) , H ( θ ∥ ) ⪰ ρ J λ 1 ( θ ∥ ) I U ( θ ∥ ∈ J ) , -H'(\theta^\parallel)\succeq\beta_J H(\theta^\parallel),\qquad H(\theta^\parallel)\succeq\rho_J\lambda_1(\theta^\parallel)I_U\qquad(\theta^\parallel\in J), − H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) , H ( θ ∥ ) ⪰ ρ J λ 1 ( θ ∥ ) I U ( θ ∥ ∈ J ) ,
and the parallel Taylor remainder and drift satisfy
∣ R ∥ ( θ ∥ , θ ⊥ ) ∣ ≤ c J ∥ θ ⊥ ∥ 3 , − ⟨ ∇ L ( θ ∥ v + θ ⊥ ) , v ⟩ ≤ G J |R^\parallel(\theta^\parallel,\theta^\perp)|\le c_J\|\theta^\perp\|^3,\qquad-\langle\nabla\mathcal{L}(\theta^\parallel v+\theta^\perp),v\rangle\le G_J ∣ R ∥ ( θ ∥ , θ ⊥ ) ∣ ≤ c J ∥ θ ⊥ ∥ 3 , − ⟨ ∇ L ( θ ∥ v + θ ⊥ ) , v ⟩ ≤ G J
for θ ∥ ∈ J \theta^\parallel\in J θ ∥ ∈ J and ∥ θ ⊥ ∥ ≤ r 0 \|\theta^\perp\|\le r_0 ∥ θ ⊥ ∥ ≤ r 0 .
Write λ 0 : = λ 1 ( θ 0 ∥ ) \lambda_0:=\lambda_1(\theta_0^\parallel) λ 0 := λ 1 ( θ 0 ∥ ) . Choose an RC-valid radius r ∈ ( 0 , r 0 ] r\in(0,r_0] r ∈ ( 0 , r 0 ] with δ r < 1 \delta_r<1 δ r < 1 and
χ c : = β J − 2 c J r ρ J λ 0 > 0. \chi_c:=\beta_J-\frac{2c_Jr}{\rho_J\lambda_0}>0. χ c := β J − ρ J λ 0 2 c J r > 0.
Define
η c ( r ) : = 2 ( 1 + δ r ) λ 0 . \eta_c(r):=\frac{2}{(1+\delta_r)\lambda_0}. η c ( r ) := ( 1 + δ r ) λ 0 2 .
For η > η c ( r ) \eta>\eta_c(r) η > η c ( r ) sufficiently close to η c ( r ) \eta_c(r) η c ( r ) , local curvature decay gives
λ 1 ( θ ∥ ) ≤ λ 0 e − β J ( θ ∥ − θ 0 ∥ ) ( θ ∥ ∈ J ) . \lambda_1(\theta^\parallel)\le\lambda_0e^{-\beta_J(\theta^\parallel-\theta_0^\parallel)}\qquad(\theta^\parallel\in J). λ 1 ( θ ∥ ) ≤ λ 0 e − β J ( θ ∥ − θ 0 ∥ ) ( θ ∥ ∈ J ) .
It follows that
0 < θ ∗ ∥ − θ 0 ∥ ≤ 1 β J log ( η η c ( r ) ) < Δ . 0<\theta_*^\parallel-\theta_0^\parallel\le\frac{1}{\beta_J}\log\left(\frac{\eta}{\eta_c(r)}\right)<\Delta. 0 < θ ∗ ∥ − θ 0 ∥ ≤ β J 1 log ( η c ( r ) η ) < Δ.
Thus the true unstable-phase slab lies inside J J J , and the reference constants can be used in the admissibility conditions.
With these constants,
χ ( η ) = β J − η ( 1 + δ r ) c J r ρ J ⟶ χ c > 0 as η ↓ η c ( r ) . \chi(\eta)=\beta_J-\frac{\eta(1+\delta_r)c_Jr}{\rho_J}\longrightarrow\chi_c>0\qquad\text{as }\eta\downarrow\eta_c(r). χ ( η ) = β J − ρ J η ( 1 + δ r ) c J r ⟶ χ c > 0 as η ↓ η c ( r ) .
Hence χ ( η ) ≥ χ c / 2 \chi(\eta)\ge\chi_c/2 χ ( η ) ≥ χ c /2 for η \eta η sufficiently close to η c ( r ) \eta_c(r) η c ( r ) . Moreover, C u n s t ( η ) → 0 C_{\mathrm{unst}}(\eta)\to0 C unst ( η ) → 0 , while θ ∗ ∥ − θ 0 ∥ + η G J \theta_*^\parallel-\theta_0^\parallel+\eta G_J θ ∗ ∥ − θ 0 ∥ + η G J remains bounded. Therefore
2 C u n s t ( η ) η χ ( η ) ( θ ∗ ∥ − θ 0 ∥ + η G J ) ⟶ 0. \frac{2C_{\mathrm{unst}}(\eta)}{\eta\chi(\eta)}\left(\theta_*^\parallel-\theta_0^\parallel+\eta G_J\right)\longrightarrow0. η χ ( η ) 2 C unst ( η ) ( θ ∗ ∥ − θ 0 ∥ + η G J ) ⟶ 0.
For every initialization with ∥ θ 0 ⊥ ∥ ≤ r / 2 \|\theta_0^\perp\|\le r/2 ∥ θ 0 ⊥ ∥ ≤ r /2 , the tubular neighborhood-closing budget consequently holds with strict slack when η > η c ( r ) \eta>\eta_c(r) η > η c ( r ) is sufficiently close to η c ( r ) \eta_c(r) η c ( r ) .
This proves that, for each such fixed radius, admissible step sizes contain a nonempty interval immediately above η c ( r ) \eta_c(r) η c ( r ) . If additionally θ 0 ∥ ≥ θ t a i l ∥ ( r ) \theta_0^\parallel\ge\theta_{\mathrm{tail}}^\parallel(r) θ 0 ∥ ≥ θ tail ∥ ( r ) , then θ ∗ ∥ ≥ θ 0 ∥ \theta_*^\parallel\ge\theta_0^\parallel θ ∗ ∥ ≥ θ 0 ∥ also guarantees the tail-entry condition required by the stable-phase theorem.
Unstable phase: tubular neighborhood invariance and finite time entry to the stable phase
Theorem 3 (Unstable phase). Suppose ( η , r , θ 0 ) (\eta, r, \theta_0) ( η , r , θ 0 ) is admissible (Definition B.2). Let t s t a b : = inf { t ≥ 0 : θ t ∥ ≥ θ ∗ ∥ } t_{\mathrm{stab}} := \inf\{t \ge0 : \theta_t^{\parallel} \ge\theta_*^{\parallel}\} t stab := inf { t ≥ 0 : θ t ∥ ≥ θ ∗ ∥ } . Then:
(i) Tubular neighborhood invariance: ∥ θ t ⊥ ∥ ≤ r \|\theta_t^{\perp}\| \le r ∥ θ t ⊥ ∥ ≤ r for all 0 ≤ t ≤ t s t a b 0 \le t \le t_{\mathrm{stab}} 0 ≤ t ≤ t stab .
(ii) Monotone forward drift: θ t + 1 ∥ − θ t ∥ ≥ η m ℓ > 0 \theta_{t+1}^{\parallel} - \theta_t^{\parallel} \ge\eta m_{\ell} > 0 θ t + 1 ∥ − θ t ∥ ≥ η m ℓ > 0 for all 0 ≤ t < t s t a b 0 \le t < t_{\mathrm{stab}} 0 ≤ t < t stab .
(iii) Exponential sharpness decay: for all 0 ≤ t < t s t a b 0 \le t < t_{\mathrm{stab}} 0 ≤ t < t stab ,
λ 1 ( θ t ∥ ) ≤ λ 1 ( θ 0 ∥ ) e − β η m ℓ t . \lambda_1(\theta_t^{\parallel}) \le\lambda_1(\theta_0^{\parallel})e^{-\beta\eta m_{\ell}t}. λ 1 ( θ t ∥ ) ≤ λ 1 ( θ 0 ∥ ) e − β η m ℓ t . (iv) Finite entry time: t s t a b ≤ ⌈ ( θ ∗ ∥ − θ 0 ∥ ) / ( η m ℓ ) ⌉ < + ∞ t_{\mathrm{stab}} \le\lceil(\theta_*^{\parallel} - \theta_0^{\parallel})/(\eta m_{\ell})\rceil< +\infty t stab ≤ ⌈( θ ∗ ∥ − θ 0 ∥ ) / ( η m ℓ )⌉ < + ∞ .
(v) Amplitude at handoff: ∥ θ t s t a b ⊥ ∥ ≤ r \|\theta_{t_{\mathrm{stab}}}^{\perp}\| \le r ∥ θ t stab ⊥ ∥ ≤ r
(vi) Overshoot control: 0 ≤ θ t s t a b ∥ − θ ∗ ∥ ≤ η G ∥ 0 \le\theta_{t_{\mathrm{stab}}}^{\parallel} - \theta_*^{\parallel} \le\eta G_{\parallel} 0 ≤ θ t stab ∥ − θ ∗ ∥ ≤ η G ∥ .
Proof. By admissibility, all assumptions needed below hold on the oscillation slab T p r e \mathcal{T}_{\mathrm{pre}} T pre defined above. In particular, Assumption A.2. gives an exact multiplicative representation of the orthogonal update:
θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ , \theta_{t+1}^{\perp} = (I-\eta M_t)\theta_t^{\perp}, θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ ,
where
( 1 − δ r ) H ( θ t ∥ ) ⪯ M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) (23) (1-\delta_r)H(\theta_t^{\parallel}) \preceq M_t \preceq(1+\delta_r)H(\theta_t^{\parallel}) \tag*{(23)} ( 1 − δ r ) H ( θ t ∥ ) ⪯ M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) (23)
whenever θ t ∈ T p r e \theta_t \in\mathcal{T}_{\mathrm{pre}} θ t ∈ T pre .
The proof is a bootstrap on the tubular neighborhood constraint. Let
τ : = inf { t ≥ 0 : ∥ θ t ⊥ ∥ > r or θ t ∥ < θ 0 ∥ or θ t ∥ ≥ θ ∗ ∥ } . \tau:= \inf\{t \ge0 : \|\theta_t^{\perp}\| > r \text{ or } \theta_t^{\parallel} < \theta_0^{\parallel} \text{ or } \theta_t^{\parallel} \ge\theta_*^{\parallel}\}. τ := inf { t ≥ 0 : ∥ θ t ⊥ ∥ > r or θ t ∥ < θ 0 ∥ or θ t ∥ ≥ θ ∗ ∥ } .
We prove that the tubular neighborhood cannot be exited before the stability threshold is reached.
Step 1: Orthogonal energy growth is controlled by curvature. Fix t < τ t < \tau t < τ . Then θ t ∈ T p r e \theta_t \in\mathcal{T}_{\mathrm{pre}} θ t ∈ T pre and θ t ∥ < θ ∗ ∥ \theta_t^{\parallel} < \theta_*^{\parallel} θ t ∥ < θ ∗ ∥ . Using the exact update,
∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 = ∥ ( I − η M t ) θ t ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 (24) \|\theta_{t+1}^{\perp}\|^2-\|\theta_t^{\perp}\|^2
= \|(I-\eta M_t)\theta_t^{\perp}\|^2-\|\theta_t^{\perp}\|^2 \tag*{(24)} ∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 = ∥ ( I − η M t ) θ t ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 (24)
= − 2 η ( θ t ⊥ ) T M t θ t ⊥ + η 2 ( θ t ⊥ ) T M t 2 θ t ⊥ . (25) = -2\eta(\theta_t^{\perp})^{\mathsf{T}}M_t\theta_t^{\perp}
+\eta^2(\theta_t^{\perp})^{\mathsf{T}}M_t^2\theta_t^{\perp}. \tag*{(25)} = − 2 η ( θ t ⊥ ) T M t θ t ⊥ + η 2 ( θ t ⊥ ) T M t 2 θ t ⊥ . (25)
Since M t ⪰ 0 M_t \succeq0 M t ⪰ 0 and
M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) I U , (26) M_t \preceq(1+\delta_r)H(\theta_t^{\parallel})
\preceq(1+\delta_r)\lambda_1(\theta_t^{\parallel})I_U, \tag*{(26)} M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) I U , (26)
we have
M t 2 ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) M t . (27) M_t^2 \preceq(1+\delta_r)\lambda_1(\theta_t^{\parallel})M_t. \tag*{(27)} M t 2 ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) M t . (27)
Therefore
∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ η ( η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ) ( θ t ⊥ ) T M t θ t ⊥ . (28) \|\theta_{t+1}^{\perp}\|^2-\|\theta_t^{\perp}\|^2
\le\eta\bigl(\eta(1+\delta_r)\lambda_1(\theta_t^{\parallel})-2\bigr)
(\theta_t^{\perp})^{\mathsf{T}}M_t\theta_t^{\perp}. \tag*{(28)} ∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ η ( η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ) ( θ t ⊥ ) T M t θ t ⊥ . (28)
Using M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) M_t \preceq(1+\delta_r)H(\theta_t^{\parallel}) M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) , we obtain
∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ η ( 1 + δ r ) ( η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ) Q t . (29) \|\theta_{t+1}^{\perp}\|^2-\|\theta_t^{\perp}\|^2
\le\eta(1+\delta_r)\bigl(\eta(1+\delta_r)\lambda_1(\theta_t^{\parallel})-2\bigr)Q_t. \tag*{(29)} ∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ η ( 1 + δ r ) ( η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ) Q t . (29)
Since λ 1 \lambda_1 λ 1 is nonincreasing and θ t ∥ ≥ θ 0 ∥ \theta_t^{\parallel} \ge\theta_0^{\parallel} θ t ∥ ≥ θ 0 ∥ ,
η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ≤ η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2. (30) \eta(1+\delta_r)\lambda_1(\theta_t^{\parallel})-2
\le\eta(1+\delta_r)\lambda_1(\theta_0^{\parallel})-2. \tag*{(30)} η ( 1 + δ r ) λ 1 ( θ t ∥ ) − 2 ≤ η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2. (30)
Thus, with
C u n s t : = η ( 1 + δ r ) ( η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2 ) , C_{\mathrm{unst}} := \eta(1+\delta_r)\bigl(\eta(1+\delta_r)\lambda_1(\theta_0^\parallel)-2\bigr), C unst := η ( 1 + δ r ) ( η ( 1 + δ r ) λ 1 ( θ 0 ∥ ) − 2 ) ,
we have the energy-growth bound
∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ C u n s t Q t , (31) \lVert\theta_{t+1}^{\perp}\rVert^2-\lVert\theta_t^{\perp}\rVert^2\le C_{\mathrm{unst}}Q_t, \tag*{(31)} ∥ θ t + 1 ⊥ ∥ 2 − ∥ θ t ⊥ ∥ 2 ≤ C unst Q t , (31)
where Q t : = θ t ⊥ ⊤ H ( θ t ) θ t ⊥ Q_t:=\theta_t^{\perp\top}H(\theta_t)\theta_t^\perp Q t := θ t ⊥ ⊤ H ( θ t ) θ t ⊥ .
Step 2: Parallel drift pays for orthogonal growth. By the parallel Taylor expansion, we have
⟨ ∇ L ∥ ( θ t ) , v ⟩ = ℓ ′ ( θ t ∥ ) + 1 2 θ t ⊥ ⊤ H ′ ( θ t ∥ ) θ t ⊥ + R t ∥ , (32) \langle\nabla\mathcal{L}^{\parallel}(\theta_t),\mathbf{v}\rangle
=\ell'(\theta_t^\parallel)+\frac{1}{2}\theta_t^{\perp\top}H'(\theta_t^\parallel)\theta_t^\perp+R_t^\parallel, \tag*{(32)} ⟨ ∇ L ∥ ( θ t ) , v ⟩ = ℓ ′ ( θ t ∥ ) + 2 1 θ t ⊥ ⊤ H ′ ( θ t ∥ ) θ t ⊥ + R t ∥ , (32)
with
∣ R t ∥ ∣ ≤ c ∥ , r ∥ θ t ⊥ ∥ 3 . \lvert R_t^\parallel\rvert\le c_{\parallel,r}\lVert\theta_t^\perp\rVert^3. ∣ R t ∥ ∣ ≤ c ∥ , r ∥ θ t ⊥ ∥ 3 .
Hence the parallel GD update satisfies
θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ∥ ( θ t ) , v ⟩ = − η ℓ ′ ( θ t ∥ ) − η 2 θ t ⊥ ⊤ H ′ ( θ t ∥ ) θ t ⊥ − η R t ∥ . \begin{aligned}
\theta_{t+1}^{\parallel}-\theta_t^\parallel
=-\eta\langle\nabla\mathcal{L}^{\parallel}(\theta_t),\mathbf{v}\rangle\\
&=-\eta\ell'(\theta_t^\parallel)-\frac{\eta}{2}\theta_t^{\perp\top}H'(\theta_t^\parallel)\theta_t^\perp-\eta R_t^\parallel.
\end{aligned} θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ∥ ( θ t ) , v ⟩ = − η ℓ ′ ( θ t ∥ ) − 2 η θ t ⊥ ⊤ H ′ ( θ t ∥ ) θ t ⊥ − η R t ∥ .
By Assumptions A.3. and B.1.,
− ℓ ′ ( θ t ∥ ) ≥ m ℓ , − H ′ ( θ t ∥ ) ⪰ β H ( θ t ∥ ) , -\ell'(\theta_t^\parallel)\ge m_\ell,\qquad
-H'(\theta_t^\parallel)\succeq\beta H(\theta_t^\parallel), − ℓ ′ ( θ t ∥ ) ≥ m ℓ , − H ′ ( θ t ∥ ) ⪰ β H ( θ t ∥ ) ,
therefore
θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + η β 2 Q t − η c ∥ , r ∥ θ t ⊥ ∥ 3 . (33) \theta_{t+1}^{\parallel}-\theta_t^\parallel
\ge\eta m_\ell+\frac{\eta\beta}{2}Q_t-\eta c_{\parallel,r}\lVert\theta_t^\perp\rVert^3. \tag*{(33)} θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + 2 η β Q t − η c ∥ , r ∥ θ t ⊥ ∥ 3 . (33)
In the Admissibility definition B.2, the margin χ > 0 \chi>0 χ > 0 is chosen precisely so that the cubic remainder can be absorbed into the curvature-reduction term. More explicitly, recall the compact-slab conditioning bound (21)
H ( θ t ∥ ) ⪰ ρ λ 1 ( θ t ∥ ) I U H(\theta_t^\parallel)\succeq\rho\lambda_1(\theta_t^\parallel)I_U H ( θ t ∥ ) ⪰ ρ λ 1 ( θ t ∥ ) I U
on the oscillation slab. Since t < τ t<\tau t < τ , we also have
η ( 1 + δ r ) λ 1 ( θ t ∥ ) > 2. \eta(1+\delta_r)\lambda_1(\theta_t^\parallel)>2. η ( 1 + δ r ) λ 1 ( θ t ∥ ) > 2.
Thus
Q t = θ t ⊥ ⊤ H ( θ t ∥ ) θ t ⊥ ≥ ρ λ 1 ( θ t ∥ ) ∥ θ t ⊥ ∥ 2 ≥ 2 ρ η ( 1 + δ r ) ∥ θ t ⊥ ∥ 2 . (34) Q_t=\theta_t^{\perp\top}H(\theta_t^\parallel)\theta_t^\perp
\ge\rho\lambda_1(\theta_t^\parallel)\lVert\theta_t^\perp\rVert^2
\ge\frac{2\rho}{\eta(1+\delta_r)}\lVert\theta_t^\perp\rVert^2. \tag*{(34)} Q t = θ t ⊥ ⊤ H ( θ t ∥ ) θ t ⊥ ≥ ρ λ 1 ( θ t ∥ ) ∥ θ t ⊥ ∥ 2 ≥ η ( 1 + δ r ) 2 ρ ∥ θ t ⊥ ∥ 2 . (34)
Since ∥ θ t ⊥ ∥ ≤ r \lVert\theta_t^\perp\rVert\le r ∥ θ t ⊥ ∥ ≤ r , this implies
∥ θ t ⊥ ∥ 3 ≤ η ( 1 + δ r ) r 2 ρ Q t . (35) \lVert\theta_t^\perp\rVert^3
\le\frac{\eta(1+\delta_r)r}{2\rho}Q_t. \tag*{(35)} ∥ θ t ⊥ ∥ 3 ≤ 2 ρ η ( 1 + δ r ) r Q t . (35)
Consequently,
θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + η 2 ( β − η ( 1 + δ r ) c ∥ , r r ρ ) Q t . (36) \theta_{t+1}^{\parallel}-\theta_t^\parallel
\ge\eta m_\ell+\frac{\eta}{2}\left(\beta-\frac{\eta(1+\delta_r)c_{\parallel,r}r}{\rho}\right)Q_t. \tag*{(36)} θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + 2 η ( β − ρ η ( 1 + δ r ) c ∥ , r r ) Q t . (36)
The quantity in parentheses is precisely χ > 0 \chi>0 χ > 0 . Therefore
θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + η χ 2 Q t ≥ η m ℓ > 0. (37) \theta_{t+1}^{\parallel}-\theta_t^\parallel
\ge\eta m_\ell+\frac{\eta\chi}{2}Q_t\ge\eta m_\ell>0. \tag*{(37)} θ t + 1 ∥ − θ t ∥ ≥ η m ℓ + 2 η χ Q t ≥ η m ℓ > 0. (37)
This proves the monotone forward drift as long as the bootstrap hypotheses hold. Moreover, rearranging (39) gives
Q t ≤ 2 η χ ( θ t + 1 ∥ − θ t ∥ ) . (38) Q_t\le\frac{2}{\eta\chi}\left(\theta_{t+1}^{\parallel}-\theta_t^\parallel\right). \tag*{(38)} Q t ≤ η χ 2 ( θ t + 1 ∥ − θ t ∥ ) . (38)
Combining (33) and (40), we obtain the key self-stabilization inequality
∥ θ t + 1 ∥ ∥ 2 − ∥ θ t ∥ ∥ 2 ≤ 2 C u n s t η χ ( θ t + 1 ∥ − θ t ∥ ) . (39) \lVert\theta_{t+1}^{\parallel}\rVert^2-\lVert\theta_t^{\parallel}\rVert^2\le\frac{2C_{\mathrm{unst}}}{\eta\chi}\left(\theta_{t+1}^{\parallel}-\theta_t^{\parallel}\right). \tag*{(39)} ∥ θ t + 1 ∥ ∥ 2 − ∥ θ t ∥ ∥ 2 ≤ η χ 2 C unst ( θ t + 1 ∥ − θ t ∥ ) . (39)
Step 3: Closure at the stopping time. For every finite integer t ≤ τ t\le\tau t ≤ τ , summing (41) gives
∥ θ t ∥ ∥ 2 ≤ ∥ θ 0 ∥ ∥ 2 + 2 C u n s t η χ ( θ t ∥ − θ 0 ∥ ) . (40) \lVert\theta_t^{\parallel}\rVert^2\le\lVert\theta_0^{\parallel}\rVert^2+\frac{2C_{\mathrm{unst}}}{\eta\chi}\left(\theta_t^{\parallel}-\theta_0^{\parallel}\right). \tag*{(40)} ∥ θ t ∥ ∥ 2 ≤ ∥ θ 0 ∥ ∥ 2 + η χ 2 C unst ( θ t ∥ − θ 0 ∥ ) . (40)
Also, (39) gives
θ t ∥ ≥ θ 0 ∥ + t η m ℓ for every finite t ≤ τ . \theta_t^{\parallel}\ge\theta_0^{\parallel}+t\eta m_\ell\qquad\text{for every finite }t\le\tau. θ t ∥ ≥ θ 0 ∥ + t η m ℓ for every finite t ≤ τ .
If τ = ∞ \tau=\infty τ = ∞ , this contradicts θ t ∥ < θ ∗ ∥ \theta_t^{\parallel}<\theta_*^{\parallel} θ t ∥ < θ ∗ ∥ for all t t t . Hence τ < ∞ \tau<\infty τ < ∞ .
Admissibility implies τ ≥ 1 \tau\ge1 τ ≥ 1 . Since the source iterate θ τ − 1 \theta_{\tau-1} θ τ − 1 belongs to T p r e \mathcal{T}_{\mathrm{pre}} T pre ,
θ τ ∥ ≤ θ τ − 1 ∥ + η ∥ G ∥ < θ ∗ ∥ + η ∥ G ∥ . \theta_\tau^{\parallel}\le\theta_{\tau-1}^{\parallel}+\eta\lVert G\rVert<\theta_*^{\parallel}+\eta\lVert G\rVert. θ τ ∥ ≤ θ τ − 1 ∥ + η ∥ G ∥ < θ ∗ ∥ + η ∥ G ∥ .
Consequently,
∥ θ τ ∥ ∥ 2 ≤ ∥ θ 0 ∥ ∥ 2 + 2 C u n s t η χ ( θ ∗ ∥ − θ 0 ∥ + η ∥ G ∥ ) ≤ r 2 . \lVert\theta_\tau^{\parallel}\rVert^2\le\lVert\theta_0^{\parallel}\rVert^2+\frac{2C_{\mathrm{unst}}}{\eta\chi}\left(\theta_*^{\parallel}-\theta_0^{\parallel}+\eta\lVert G\rVert\right)\le r^2. ∥ θ τ ∥ ∥ 2 ≤ ∥ θ 0 ∥ ∥ 2 + η χ 2 C unst ( θ ∗ ∥ − θ 0 ∥ + η ∥ G ∥ ) ≤ r 2 .
Furthermore, positive forward drift gives θ τ ∥ ≥ θ 0 ∥ \theta_\tau^{\parallel}\ge\theta_0^{\parallel} θ τ ∥ ≥ θ 0 ∥ . Thus neither tubular neighborhood exit nor a decrease below θ 0 ∥ \theta_0^{\parallel} θ 0 ∥ can trigger τ \tau τ . Necessarily θ τ ∥ ≥ θ ∗ ∥ \theta_\tau^{\parallel}\ge\theta_*^{\parallel} θ τ ∥ ≥ θ ∗ ∥ , so τ = t s t a b \tau=t_{\mathrm{stab}} τ = t stab .
This proves tubular neighborhood invariance through the handoff, finite entry, and the overshoot bound
0 ≤ θ t s t a b ∥ − θ ∗ ∥ ≤ η ∥ G ∥ . 0\le\theta_{t_{\mathrm{stab}}}^{\parallel}-\theta_*^{\parallel}\le\eta\lVert G\rVert. 0 ≤ θ t stab ∥ − θ ∗ ∥ ≤ η ∥ G ∥ .
Step 4: Exponential sharpness decay. By the local curvature decay assumption on the oscillation slab,
− H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] . -H'(\theta^{\parallel})\succeq\beta H(\theta^{\parallel})\qquad\forall\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}]. − H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] .
Equivalently,
H ′ ( θ ∥ ) + β H ( θ ∥ ) ⪯ 0. H'(\theta^{\parallel})+\beta H(\theta^{\parallel})\preceq0. H ′ ( θ ∥ ) + β H ( θ ∥ ) ⪯ 0.
Therefore
d d θ ∥ ( e β θ ∥ H ( θ ∥ ) ) = e β θ ∥ ( H ′ ( θ ∥ ) + β H ( θ ∥ ) ) ⪯ 0. \frac{d}{d\theta^{\parallel}}\left(e^{\beta\theta^{\parallel}}H(\theta^{\parallel})\right)=e^{\beta\theta^{\parallel}}\left(H'(\theta^{\parallel})+\beta H(\theta^{\parallel})\right)\preceq0. d θ ∥ d ( e β θ ∥ H ( θ ∥ ) ) = e β θ ∥ ( H ′ ( θ ∥ ) + β H ( θ ∥ ) ) ⪯ 0.
Hence, for every θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] \theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}] θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] ,
e β θ ∥ H ( θ ∥ ) ⪯ e β θ 0 ∥ H ( θ 0 ∥ ) , e^{\beta\theta^{\parallel}}H(\theta^{\parallel})\preceq e^{\beta\theta_0^{\parallel}}H(\theta_0^{\parallel}), e β θ ∥ H ( θ ∥ ) ⪯ e β θ 0 ∥ H ( θ 0 ∥ ) ,
or equivalently,
H ( θ ∥ ) ⪯ e − β ( θ ∥ − θ 0 ∥ ) H ( θ 0 ∥ ) . H(\theta^{\parallel})\preceq e^{-\beta(\theta^{\parallel}-\theta_0^{\parallel})}H(\theta_0^{\parallel}). H ( θ ∥ ) ⪯ e − β ( θ ∥ − θ 0 ∥ ) H ( θ 0 ∥ ) .
Taking largest eigenvalues gives
λ 1 ( θ ∥ ) ≤ e − β ( θ ∥ − θ 0 ∥ ) λ 1 ( θ 0 ∥ ) . \lambda_1(\theta^{\parallel})\le e^{-\beta(\theta^{\parallel}-\theta_0^{\parallel})}\lambda_1(\theta_0^{\parallel}). λ 1 ( θ ∥ ) ≤ e − β ( θ ∥ − θ 0 ∥ ) λ 1 ( θ 0 ∥ ) .
Applying this with θ ∥ = θ t ∥ \theta^{\parallel}=\theta_t^{\parallel} θ ∥ = θ t ∥ and using the monotone drift bound
θ t ∥ − θ 0 ∥ = ∑ s = 0 t − 1 ( θ s + 1 ∥ − θ s ∥ ) ≥ η m ℓ t \theta_t^{\parallel}-\theta_0^{\parallel}=\sum_{s=0}^{t-1}\left(\theta_{s+1}^{\parallel}-\theta_s^{\parallel}\right)\ge\eta m_\ell t θ t ∥ − θ 0 ∥ = s = 0 ∑ t − 1 ( θ s + 1 ∥ − θ s ∥ ) ≥ η m ℓ t
for 0 ≤ t < t s t a b 0\le t<t_{\mathrm{stab}} 0 ≤ t < t stab , we obtain
λ 1 ( θ t ∥ ) ≤ λ 1 ( θ 0 ∥ ) e − β η m ℓ t . \lambda_1(\theta_t^{\parallel})\le\lambda_1(\theta_0^{\parallel})e^{-\beta\eta m_\ell t}. λ 1 ( θ t ∥ ) ≤ λ 1 ( θ 0 ∥ ) e − β η m ℓ t .
Step 5: Finite entry time. By (39), for every 0 ≤ t < t s t a b 0\le t<t_{\mathrm{stab}} 0 ≤ t < t stab ,
θ t + 1 ∥ − θ t ∥ ≥ η m ℓ . \theta_{t+1}^{\parallel}-\theta_t^{\parallel}\ge\eta m_\ell. θ t + 1 ∥ − θ t ∥ ≥ η m ℓ .
Thus, for every n < t s t a b n<t_{\mathrm{stab}} n < t stab ,
θ n ∥ ≥ θ 0 ∥ + n η m ℓ . \theta_n^{\parallel}\ge\theta_0^{\parallel}+n\eta m_\ell. θ n ∥ ≥ θ 0 ∥ + n η m ℓ .
Consequently, once
n ≥ θ ∗ ∥ − θ 0 ∥ η m ℓ , n \ge\frac{\theta_*^{\parallel}-\theta_0^{\parallel}}{\eta m_\ell}, n ≥ η m ℓ θ ∗ ∥ − θ 0 ∥ ,
we must have θ n ∥ ≥ θ ∗ ∥ \theta_n^{\parallel} \ge\theta_*^{\parallel} θ n ∥ ≥ θ ∗ ∥ . Therefore
t s t a b ≤ ⌈ θ ∗ ∥ − θ 0 ∥ η m ℓ ⌉ . t_{\mathrm{stab}} \le\left\lceil\frac{\theta_*^{\parallel}-\theta_0^{\parallel}}{\eta m_\ell} \right\rceil. t stab ≤ ⌈ η m ℓ θ ∗ ∥ − θ 0 ∥ ⌉ .
Step 6: Orthogonal amplitude at handoff. The stopping-time argument already establishes
∥ θ t s t a b ⊥ ∥ 2 ≤ ∥ θ 0 ⊥ ∥ 2 + 2 C u n s t η χ ( θ ∗ ∥ − θ 0 ∥ + η G ∥ ) ≤ r 2 . \left\|\theta_{t_{\mathrm{stab}}}^{\perp}\right\|^2 \le\left\|\theta_0^{\perp}\right\|^2 + \frac{2C_{\mathrm{unst}}}{\eta\chi}\left(\theta_*^{\parallel}-\theta_0^{\parallel}+\eta G_{\parallel}\right) \le r^2. θ t stab ⊥ 2 ≤ θ 0 ⊥ 2 + η χ 2 C unst ( θ ∗ ∥ − θ 0 ∥ + η G ∥ ) ≤ r 2 .
Step 7: Corrected stability at handoff. By definition,
θ ∗ ∥ : = inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } . \theta_*^{\parallel} := \inf\left\{\theta^{\parallel} \ge\theta_0^{\parallel} : \eta(1+\delta_r)\lambda_1(\theta^{\parallel}) \le2\right\}. θ ∗ ∥ := inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } .
Since λ 1 \lambda_1 λ 1 is nonincreasing and θ t s t a b ∥ ≥ θ ∗ ∥ \theta_{t_{\mathrm{stab}}}^{\parallel} \ge\theta_*^{\parallel} θ t stab ∥ ≥ θ ∗ ∥ , we conclude that
η ( 1 + δ r ) λ 1 ( θ t s t a b ∥ ) ≤ 2. \eta(1+\delta_r)\lambda_1\left(\theta_{t_{\mathrm{stab}}}^{\parallel}\right) \le2. η ( 1 + δ r ) λ 1 ( θ t stab ∥ ) ≤ 2.
This proves all claims.
Stable phase: schedules and three-phase damping
We now analyze the trajectory for t ≥ t s t a b t \ge t_{\mathrm{stab}} t ≥ t stab . Throughout this subsection, assume θ ∗ ∥ ≥ θ t a i l ∥ ( r ) \theta_*^{\parallel} \ge\theta_{\mathrm{tail}}^{\parallel}(r) θ ∗ ∥ ≥ θ tail ∥ ( r ) . The re-anchored tail bounds below therefore hold for every θ ∥ ≥ θ ∗ ∥ \theta^{\parallel} \ge\theta_*^{\parallel} θ ∥ ≥ θ ∗ ∥ .
Re-anchored tail constants. Assumption B.2. gives exponential-tail bounds centered at θ t a i l ∥ ( r ) \theta_{\mathrm{tail}}^{\parallel}(r) θ tail ∥ ( r ) . In the stable-phase proof it is more convenient to center the same bounds at the stability threshold θ ∗ ∥ \theta_*^{\parallel} θ ∗ ∥ . We therefore use the following re-anchored constants. If
A ± t a i l , λ 1 , ± t a i l , β t a i l A_{\pm}^{\mathrm{tail}}, \qquad\lambda_{1,\pm}^{\mathrm{tail}}, \qquad\beta_{\mathrm{tail}} A ± tail , λ 1 , ± tail , β tail
denote constants for which
A − t a i l e − β t a i l ( θ ∥ − θ t a i l ∥ ) ≤ − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ A + t a i l e − β t a i l ( θ ∥ − θ t a i l ∥ ) A_-^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})}
\le-\langle\nabla L^{\parallel}(\theta),v\rangle
\le A_+^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})} A − tail e − β tail ( θ ∥ − θ tail ∥ ) ≤ − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ A + tail e − β tail ( θ ∥ − θ tail ∥ )
and
λ 1 , − t a i l e − β t a i l ( θ ∥ − θ t a i l ∥ ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + t a i l e − β t a i l ( θ ∥ − θ t a i l ∥ ) \lambda_{1,-}^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})}
\le\lambda_1(\theta^{\parallel})
\le\lambda_{1,+}^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})} λ 1 , − tail e − β tail ( θ ∥ − θ tail ∥ ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + tail e − β tail ( θ ∥ − θ tail ∥ )
for θ ∥ ≥ θ t a i l ∥ ( r ) \theta^{\parallel} \ge\theta_{\mathrm{tail}}^{\parallel}(r) θ ∥ ≥ θ tail ∥ ( r ) and ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\| \le r ∥ θ ⊥ ∥ ≤ r , then define
A ± ∗ : = A ± t a i l e − β t a i l ( θ ∗ ∥ − θ t a i l ∥ ) , λ 1 , ± ∗ : = λ 1 , ± t a i l e − β t a i l ( θ ∗ ∥ − θ t a i l ∥ ) . (41) A_{\pm}^* := A_{\pm}^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta_*^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})},
\qquad
\lambda_{1,\pm}^* := \lambda_{1,\pm}^{\mathrm{tail}}e^{-\beta_{\mathrm{tail}}(\theta_*^{\parallel}-\theta_{\mathrm{tail}}^{\parallel})}. \tag*{(41)} A ± ∗ := A ± tail e − β tail ( θ ∗ ∥ − θ tail ∥ ) , λ 1 , ± ∗ := λ 1 , ± tail e − β tail ( θ ∗ ∥ − θ tail ∥ ) . (41)
Equivalently, for all θ ∥ ≥ θ ∗ ∥ \theta^{\parallel} \ge\theta_*^{\parallel} θ ∥ ≥ θ ∗ ∥ and ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\| \le r ∥ θ ⊥ ∥ ≤ r ,
A − ∗ e − β t a i l ( θ ∥ − θ ∗ ∥ ) ≤ − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ A + ∗ e − β t a i l ( θ ∥ − θ ∗ ∥ ) , (42) A_-^*e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{\parallel})}
\le-\langle\nabla L^{\parallel}(\theta),v\rangle
\le A_+^*e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{\parallel})}, \tag*{(42)} A − ∗ e − β tail ( θ ∥ − θ ∗ ∥ ) ≤ − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ A + ∗ e − β tail ( θ ∥ − θ ∗ ∥ ) , (42)
λ 1 , − ∗ e − β t a i l ( θ ∥ − θ ∗ ∥ ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + ∗ e − β t a i l ( θ ∥ − θ ∗ ∥ ) . (43) \lambda_{1,-}^*e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{\parallel})}
\le\lambda_1(\theta^{\parallel})
\le\lambda_{1,+}^*e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{\parallel})}. \tag*{(43)} λ 1 , − ∗ e − β tail ( θ ∥ − θ ∗ ∥ ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + ∗ e − β tail ( θ ∥ − θ ∗ ∥ ) . (43)
Throughout the stable-phase proof, A ± ∗ A_{\pm}^* A ± ∗ and λ 1 , ± ∗ \lambda_{1,\pm}^* λ 1 , ± ∗ refer to these re-anchored constants. Define
Ψ ( B ) : = { e B − 1 B , B > 0 , 1 , B = 0 , B r : = 2 β t a i l 1 + δ r A + ∗ λ 1 , − ∗ . \Psi(B) :=
\begin{cases}
\dfrac{e^B-1}{B}, & B>0,\\
1, & B=0,
\end{cases}
\qquad
B_r := \frac{2\beta_{\mathrm{tail}}}{1+\delta_r}\frac{A_+^*}{\lambda_{1,-}^*}. Ψ ( B ) := ⎩ ⎨ ⎧ B e B − 1 , 1 , B > 0 , B = 0 , B r := 1 + δ r 2 β tail λ 1 , − ∗ A + ∗ .
y ˉ = e β t a i l ( θ s t a b ∥ − θ ∗ ∥ ) , α − = β t a i l η A − ∗ , α + = β t a i l η A + ∗ Ψ ( B r ) . \bar{y}=e^{\beta_{\mathrm{tail}}(\theta_{\mathrm{stab}}^{\parallel}-\theta_*^{\parallel})},
\qquad
\alpha_-=\beta_{\mathrm{tail}}\eta A_-^*,
\qquad
\alpha_+=\beta_{\mathrm{tail}}\eta A_+^*\Psi(B_r). y ˉ = e β tail ( θ stab ∥ − θ ∗ ∥ ) , α − = β tail η A − ∗ , α + = β tail η A + ∗ Ψ ( B r ) .
Theorem 4 (Stable phase). *Under the hypotheses of Theorem 3, assume additionally that θ ∗ ∥ ≥ θ t a i l ∥ ( r ) \theta_*^{\parallel} \ge\theta_{\mathrm{tail}}^{\parallel}(r) θ ∗ ∥ ≥ θ tail ∥ ( r ) . Then for all t ≥ t s t a b t \ge t_{\mathrm{stab}} t ≥ t stab :
(i) Nonexpansion: ∥ θ t ∥ ∥ ≤ ∥ θ t s t a b ∥ ∥ ≤ r \lVert\theta_t^\parallel\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^\parallel\rVert\le r ∥ θ t ∥ ∥ ≤ ∥ θ t stab ∥ ∥ ≤ r .
(ii) Logarithmic parallel growth:
1 β t a i l log ( y ˉ + α − ( t − t s t a b ) ) ≤ θ t ∥ − θ ∗ ∥ ≤ 1 β t a i l log ( y ˉ + α + ( t − t s t a b ) ) . \frac{1}{\beta_{\mathrm{tail}}}\log\left(\bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})\right)\le\theta_t^\parallel-\theta_*^\parallel\le\frac{1}{\beta_{\mathrm{tail}}}\log\left(\bar{y}+\alpha_{+}(t-t_{\mathrm{stab}})\right). β tail 1 log ( y ˉ + α − ( t − t stab ) ) ≤ θ t ∥ − θ ∗ ∥ ≤ β tail 1 log ( y ˉ + α + ( t − t stab ) ) .
(iii) Sharpness schedule:
λ 1 , − ∗ y ˉ + α + ( t − t s t a b ) ≤ λ 1 ( θ t ∥ ) ≤ λ 1 , + ∗ y ˉ + α − ( t − t s t a b ) . \frac{\lambda_{1,-}^{*}}{\bar{y}+\alpha_{+}(t-t_{\mathrm{stab}})}\le\lambda_1(\theta_t^\parallel)\le\frac{\lambda_{1,+}^{*}}{\bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})}. y ˉ + α + ( t − t stab ) λ 1 , − ∗ ≤ λ 1 ( θ t ∥ ) ≤ y ˉ + α − ( t − t stab ) λ 1 , + ∗ .
Assume further the uniform conditioning H ( θ ∥ ) ⪰ ρ t a i l λ 1 ( θ ∥ ) I U H(\theta^\parallel)\succeq\rho_{\mathrm{tail}}\lambda_1(\theta^\parallel)I_U H ( θ ∥ ) ⪰ ρ tail λ 1 ( θ ∥ ) I U for all θ ∥ ≥ θ ∗ ∥ \theta^\parallel\ge\theta_*^\parallel θ ∥ ≥ θ ∗ ∥ . Fix 0 < ε ≤ ε ∗ : = ρ t a i l ( 1 − δ r ) / ( 1 + δ r ) 0<\varepsilon\le\varepsilon_*:=\rho_{\mathrm{tail}}(1-\delta_r)/(1+\delta_r) 0 < ε ≤ ε ∗ := ρ tail ( 1 − δ r ) / ( 1 + δ r ) , and define
t l i n : = inf { t ≥ t s t a b : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 2 − ε } , t p o l y : = inf { t ≥ t s t a b : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 1 } . (44) t_{\mathrm{lin}}:=\inf\{t\ge t_{\mathrm{stab}}:\eta(1+\delta_r)\lambda_1(\theta_t^\parallel)\le2-\varepsilon\},\qquad
t_{\mathrm{poly}}:=\inf\{t\ge t_{\mathrm{stab}}:\eta(1+\delta_r)\lambda_1(\theta_t^\parallel)\le1\}. \tag*{(44)} t lin := inf { t ≥ t stab : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 2 − ε } , t poly := inf { t ≥ t stab : η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ 1 } . (44)
Then t s t a b ≤ t l i n ≤ t p o l y < ∞ t_{\mathrm{stab}}\le t_{\mathrm{lin}}\le t_{\mathrm{poly}}<\infty t stab ≤ t lin ≤ t poly < ∞ , and:
(iv) Transition phase (t s t a b ≤ t ≤ t l i n t_{\mathrm{stab}}\le t\le t_{\mathrm{lin}} t stab ≤ t ≤ t lin ): ∥ θ t ⊥ ∥ ≤ ∥ θ t s t a b ⊥ ∥ \lVert\theta_t^\perp\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^\perp\rVert ∥ θ t ⊥ ∥ ≤ ∥ θ t stab ⊥ ∥ .
(v) Linear contraction phase (t l i n ≤ t ≤ t p o l y t_{\mathrm{lin}}\le t\le t_{\mathrm{poly}} t lin ≤ t ≤ t poly ): ∥ θ t ⊥ ∥ ≤ ( 1 − ε ) t − t l i n ∥ θ t l i n ⊥ ∥ \lVert\theta_t^\perp\rVert\le(1-\varepsilon)^{t-t_{\mathrm{lin}}}\lVert\theta_{t_{\mathrm{lin}}}^\perp\rVert ∥ θ t ⊥ ∥ ≤ ( 1 − ε ) t − t lin ∥ θ t lin ⊥ ∥ . At t = t l i n t=t_{\mathrm{lin}} t = t lin , the bound is interpreted as equality, including when ε = 1 \varepsilon=1 ε = 1 .
(vi) Polynomial decay phase (t ≥ t p o l y t\ge t_{\mathrm{poly}} t ≥ t poly ): with A : = ⌈ y ˉ / α + ⌉ + 1 A:=\lceil\bar{y}/\alpha_{+}\rceil+1 A := ⌈ y ˉ / α + ⌉ + 1 and
γ r : = ρ t a i l ( 1 − δ r ) λ 1 , − ∗ β t a i l Ψ ( B r ) A + ∗ , \gamma_r:=\frac{\rho_{\mathrm{tail}}(1-\delta_r)\lambda_{1,-}^{*}}{\beta_{\mathrm{tail}}\Psi(B_r)A_{+}^{*}}, γ r := β tail Ψ ( B r ) A + ∗ ρ tail ( 1 − δ r ) λ 1 , − ∗ ,
∥ θ t ⊥ ∥ ≤ ∥ θ t p o l y ⊥ ∥ ( A + t p o l y − t s t a b A + t − t s t a b ) γ r . \lVert\theta_t^\perp\rVert\le\lVert\theta_{t_{\mathrm{poly}}}^\perp\rVert\left(\frac{A+t_{\mathrm{poly}}-t_{\mathrm{stab}}}{A+t-t_{\mathrm{stab}}}\right)^{\gamma_r}. ∥ θ t ⊥ ∥ ≤ ∥ θ t poly ⊥ ∥ ( A + t − t stab A + t poly − t stab ) γ r .
Proof. Since θ t s t a b ∥ ≥ θ t a i l ∥ ( r ) \theta_{t_{\mathrm{stab}}}^\parallel\ge\theta_{\mathrm{tail}}^\parallel(r) θ t stab ∥ ≥ θ tail ∥ ( r ) , the exponential-tail assumptions apply from time t s t a b t_{\mathrm{stab}} t stab onward as long as the iterates remain inside the tubular neighborhood. We first prove nonexpansion, which closes this bootstrap.
Step 1: Nonexpansion. By Assumptions A.2. and A.1. , the orthogonal update admits the exact multiplicative representation
θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ , \theta_{t+1}^\perp=(I-\eta M_t)\theta_t^\perp, θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ ,
where
( 1 − δ r ) H ( θ t ∥ ) ⪯ M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) (45) (1-\delta_r)H(\theta_t^\parallel)\preceq M_t\preceq(1+\delta_r)H(\theta_t^\parallel) \tag*{(45)} ( 1 − δ r ) H ( θ t ∥ ) ⪯ M t ⪯ ( 1 + δ r ) H ( θ t ∥ ) (45)
whenever ∥ θ t ∥ ∥ ≤ r \lVert\theta_t^\parallel\rVert\le r ∥ θ t ∥ ∥ ≤ r .
We claim that
∥ θ t ⊥ ∥ ≤ ∥ θ t s t a b ⊥ ∥ ∀ t ≥ t s t a b . \lVert\theta_t^\perp\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^\perp\rVert\qquad\forall\,t\ge t_{\mathrm{stab}}. ∥ θ t ⊥ ∥ ≤ ∥ θ t stab ⊥ ∥ ∀ t ≥ t stab .
Indeed, suppose inductively that ∥ θ t ⊥ ∥ ≤ ∥ θ t s t a b ⊥ ∥ \lVert\theta_t^\perp\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^\perp\rVert ∥ θ t ⊥ ∥ ≤ ∥ θ t stab ⊥ ∥ . Since the tail parallel drift satisfies
− ⟨ ∇ L ∥ ( θ t ) , v ⟩ > 0 , -\langle\nabla L^\parallel(\theta_t),v\rangle>0, − ⟨ ∇ L ∥ ( θ t ) , v ⟩ > 0 ,
we have
θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ∥ ( θ t ) , v ⟩ > 0. \theta_{t+1}^\parallel-\theta_t^\parallel=-\eta\langle\nabla L^\parallel(\theta_t),v\rangle>0. θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ∥ ( θ t ) , v ⟩ > 0.
Thus θ t ∥ \theta_t^\parallel θ t ∥ is nondecreasing for t ≥ t s t a b t\ge t_{\mathrm{stab}} t ≥ t stab . Since λ 1 \lambda_1 λ 1 is nonincreasing,
η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ η ( 1 + δ r ) λ 1 ( θ t s t a b ∥ ) ≤ 2. \eta(1+\delta_r)\lambda_1(\theta_t^\parallel)\le\eta(1+\delta_r)\lambda_1(\theta_{t_{\mathrm{stab}}}^\parallel)\le2. η ( 1 + δ r ) λ 1 ( θ t ∥ ) ≤ η ( 1 + δ r ) λ 1 ( θ t stab ∥ ) ≤ 2.
Using (47),
0 ⪯ M t ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) I U , 0\preceq M_t\preceq(1+\delta_r)\lambda_1(\theta_t^\parallel)I_U, 0 ⪯ M t ⪯ ( 1 + δ r ) λ 1 ( θ t ∥ ) I U ,
so every eigenvalue m m m of M t M_t M t satisfies
0 ≤ η m ≤ 2. 0\le\eta m\le2. 0 ≤ η m ≤ 2.
Therefore
∥ I − η M t ∥ o p ≤ 1 , \lVert I-\eta M_t\rVert_{\mathrm{op}}\le1, ∥ I − η M t ∥ op ≤ 1 ,
and hence
∥ θ t + 1 ∥ ∥ ≤ ∥ θ t ∥ ∥ . \lVert\theta_{t+1}^{\parallel}\rVert\le\lVert\theta_t^{\parallel}\rVert. ∥ θ t + 1 ∥ ∥ ≤ ∥ θ t ∥ ∥ .
The induction allows us to conclude that
∥ θ t ∥ ∥ ≤ ∥ θ t s t a b ∥ ∥ ∀ t ≥ t s t a b . \lVert\theta_t^{\parallel}\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^{\parallel}\rVert\qquad\forall\,t\ge t_{\mathrm{stab}}. ∥ θ t ∥ ∥ ≤ ∥ θ t stab ∥ ∥ ∀ t ≥ t stab .
Thus the tail assumptions apply globally for all t ≥ t s t a b t\ge t_{\mathrm{stab}} t ≥ t stab .
Step 2: Parallel recursion. By the re-anchored exponential-tail drift bound (44) and the GD update,
θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ( θ t ) , v ⟩ , \theta_{t+1}^{\parallel}-\theta_t^{\parallel}=-\eta\langle\nabla\mathcal{L}(\theta_t),v\rangle, θ t + 1 ∥ − θ t ∥ = − η ⟨ ∇ L ( θ t ) , v ⟩ ,
we have
η A ∗ − e − β t a i l ( θ t ∥ − θ ∗ ∥ ) ≤ θ t + 1 ∥ − θ t ∥ ≤ η A ∗ + e − β t a i l ( θ t ∥ − θ ∗ ∥ ) . \eta A_*^{-}e^{-\beta_{\mathrm{tail}}(\theta_t^{\parallel}-\theta_*^{\parallel})}\le\theta_{t+1}^{\parallel}-\theta_t^{\parallel}\le\eta A_*^{+}e^{-\beta_{\mathrm{tail}}(\theta_t^{\parallel}-\theta_*^{\parallel})}. η A ∗ − e − β tail ( θ t ∥ − θ ∗ ∥ ) ≤ θ t + 1 ∥ − θ t ∥ ≤ η A ∗ + e − β tail ( θ t ∥ − θ ∗ ∥ ) .
Equivalently, since
y t : = e β t a i l ( θ t ∥ − θ ∗ ∥ ) , y_t:=e^{\beta_{\mathrm{tail}}(\theta_t^{\parallel}-\theta_*^{\parallel})}, y t := e β tail ( θ t ∥ − θ ∗ ∥ ) ,
η A ∗ − y t ≤ θ t + 1 ∥ − θ t ∥ ≤ η A ∗ + y t . \frac{\eta A_*^{-}}{y_t}\le\theta_{t+1}^{\parallel}-\theta_t^{\parallel}\le\frac{\eta A_*^{+}}{y_t}. y t η A ∗ − ≤ θ t + 1 ∥ − θ t ∥ ≤ y t η A ∗ + .
We write
y t + 1 = y t e β t a i l ( θ t + 1 ∥ − θ t ∥ ) . y_{t+1}=y_te^{\beta_{\mathrm{tail}}(\theta_{t+1}^{\parallel}-\theta_t^{\parallel})}. y t + 1 = y t e β tail ( θ t + 1 ∥ − θ t ∥ ) .
For the lower bound, using e z ≥ 1 + z e^z\ge1+z e z ≥ 1 + z ,
y t + 1 ≥ y t ( 1 + β t a i l ( θ t + 1 ∥ − θ t ∥ ) ) ≥ y t + β t a i l η A ∗ − . y_{t+1}\ge y_t\left(1+\beta_{\mathrm{tail}}(\theta_{t+1}^{\parallel}-\theta_t^{\parallel})\right)\ge y_t+\beta_{\mathrm{tail}}\eta A_*^{-}. y t + 1 ≥ y t ( 1 + β tail ( θ t + 1 ∥ − θ t ∥ ) ) ≥ y t + β tail η A ∗ − .
Thus, with
α − : = β t a i l η A ∗ − , \alpha_-:=\beta_{\mathrm{tail}}\eta A_*^{-}, α − := β tail η A ∗ − ,
we get
y t ≥ y ˉ + α − ( t − t s t a b ) . y_t\ge\bar{y}+\alpha_-(t-t_{\mathrm{stab}}). y t ≥ y ˉ + α − ( t − t stab ) .
For the upper bound, set
z t : = β t a i l ( θ t + 1 ∥ − θ t ∥ ) . z_t:=\beta_{\mathrm{tail}}(\theta_{t+1}^{\parallel}-\theta_t^{\parallel}). z t := β tail ( θ t + 1 ∥ − θ t ∥ ) .
The re-centered tail drift bound gives
0 ≤ z t ≤ β t a i l η A ∗ + y t . 0\le z_t\le\frac{\beta_{\mathrm{tail}}\eta A_*^{+}}{y_t}. 0 ≤ z t ≤ y t β tail η A ∗ + .
Since y t ≥ 1 y_t\ge1 y t ≥ 1 , it remains to bound β t a i l η A ∗ + \beta_{\mathrm{tail}}\eta A_*^{+} β tail η A ∗ + . Evaluating the re-centered tail sharpness lower bound at θ ∥ = θ ∗ ∥ \theta^{\parallel}=\theta_*^{\parallel} θ ∥ = θ ∗ ∥ gives
λ 1 , − ∗ ≤ λ 1 ( θ ∗ ∥ ) . \lambda_{1,-}^{*}\le\lambda_1(\theta_*^{\parallel}). λ 1 , − ∗ ≤ λ 1 ( θ ∗ ∥ ) .
By the definition of the stability threshold,
η ( 1 + δ r ) λ 1 ( θ ∗ ∥ ) ≤ 2. \eta(1+\delta_r)\lambda_1(\theta_*^{\parallel})\le2. η ( 1 + δ r ) λ 1 ( θ ∗ ∥ ) ≤ 2.
Therefore
β t a i l η A ∗ + = β t a i l A ∗ + λ 1 , − ∗ η λ 1 , − ∗ ≤ 2 β t a i l 1 + δ r A ∗ + λ 1 , − ∗ = B r . \beta_{\mathrm{tail}}\eta A_*^{+}
=\beta_{\mathrm{tail}}\frac{A_*^{+}}{\lambda_{1,-}^{*}}\eta\lambda_{1,-}^{*}
\le\frac{2\beta_{\mathrm{tail}}}{1+\delta_r}\frac{A_*^{+}}{\lambda_{1,-}^{*}}
=B_r. β tail η A ∗ + = β tail λ 1 , − ∗ A ∗ + η λ 1 , − ∗ ≤ 1 + δ r 2 β tail λ 1 , − ∗ A ∗ + = B r .
Hence z t ∈ [ 0 , B r ] z_t\in[0,B_r] z t ∈ [ 0 , B r ] . By definition of Ψ \Psi Ψ ,
e z t − 1 ≤ z t Ψ ( B r ) . e^{z_t}-1\le z_t\Psi(B_r). e z t − 1 ≤ z t Ψ ( B r ) .
Thus
y t + 1 − y t = y t ( e z t − 1 ) ≤ y t z t Ψ ( B r ) ≤ β t a i l η A ∗ + Ψ ( B r ) = α + . \begin{aligned}
y_{t+1}-y_t&=y_t(e^{z_t}-1)\\
&\le y_tz_t\Psi(B_r)\\
&\le\beta_{\mathrm{tail}}\eta A_*^{+}\Psi(B_r)=\alpha_+.
\end{aligned} y t + 1 − y t = y t ( e z t − 1 ) ≤ y t z t Ψ ( B r ) ≤ β tail η A ∗ + Ψ ( B r ) = α + .
Therefore
y t ≤ y ˉ + α + ( t − t s t a b ) . y_t\le\bar{y}+\alpha_+(t-t_{\mathrm{stab}}). y t ≤ y ˉ + α + ( t − t stab ) .
Combining the two bounds gives
y ˉ + α − ( t − t s t a b ) ≤ y t ≤ y ˉ + α + ( t − t s t a b ) . (46) \bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})\le y_t\le\bar{y}+\alpha_{+}(t-t_{\mathrm{stab}}). \tag*{(46)} y ˉ + α − ( t − t stab ) ≤ y t ≤ y ˉ + α + ( t − t stab ) . (46)
Taking logarithms and dividing by β t a i l \beta_{\mathrm{tail}} β tail yields
1 β t a i l log ( y ˉ + α − ( t − t s t a b ) ) ≤ θ t ℓ − θ ∗ ℓ ≤ 1 β t a i l log ( y ˉ + α + ( t − t s t a b ) ) . \frac{1}{\beta_{\mathrm{tail}}}\log\left(\bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})\right)\le\theta_t^{\ell}-\theta_*^{\ell}\le\frac{1}{\beta_{\mathrm{tail}}}\log\left(\bar{y}+\alpha_{+}(t-t_{\mathrm{stab}})\right). β tail 1 log ( y ˉ + α − ( t − t stab ) ) ≤ θ t ℓ − θ ∗ ℓ ≤ β tail 1 log ( y ˉ + α + ( t − t stab ) ) .
This proves the logarithmic parallel growth.
Step 3: sharpness schedule. The re-anchored exponential-tail sharpness bounds (45) give
λ 1 , − ∗ e − β t a i l ( θ t ℓ − θ ∗ ℓ ) ≤ λ 1 ( θ t ℓ ) ≤ λ 1 , + ∗ e − β t a i l ( θ t ℓ − θ ∗ ℓ ) . \lambda_{1,-}^{*}e^{-\beta_{\mathrm{tail}}(\theta_t^{\ell}-\theta_*^{\ell})}\le\lambda_1(\theta_t^{\ell})\le\lambda_{1,+}^{*}e^{-\beta_{\mathrm{tail}}(\theta_t^{\ell}-\theta_*^{\ell})}. λ 1 , − ∗ e − β tail ( θ t ℓ − θ ∗ ℓ ) ≤ λ 1 ( θ t ℓ ) ≤ λ 1 , + ∗ e − β tail ( θ t ℓ − θ ∗ ℓ ) .
Since y t = e β t a i l ( θ t ℓ − θ ∗ ℓ ) y_t=e^{\beta_{\mathrm{tail}}(\theta_t^{\ell}-\theta_*^{\ell})} y t = e β tail ( θ t ℓ − θ ∗ ℓ ) , this is
λ 1 , − ∗ y t ≤ λ 1 ( θ t ℓ ) ≤ λ 1 , + ∗ y t . \frac{\lambda_{1,-}^{*}}{y_t}\le\lambda_1(\theta_t^{\ell})\le\frac{\lambda_{1,+}^{*}}{y_t}. y t λ 1 , − ∗ ≤ λ 1 ( θ t ℓ ) ≤ y t λ 1 , + ∗ .
Using (48), we obtain
λ 1 , − ∗ y ˉ + α + ( t − t s t a b ) ≤ λ 1 ( θ t ℓ ) ≤ λ 1 , + ∗ y ˉ + α − ( t − t s t a b ) . \frac{\lambda_{1,-}^{*}}{\bar{y}+\alpha_{+}(t-t_{\mathrm{stab}})}\le\lambda_1(\theta_t^{\ell})\le\frac{\lambda_{1,+}^{*}}{\bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})}. y ˉ + α + ( t − t stab ) λ 1 , − ∗ ≤ λ 1 ( θ t ℓ ) ≤ y ˉ + α − ( t − t stab ) λ 1 , + ∗ .
This proves the sharpness schedule.
Step 4: Transition times are finite and ordered. Because y t → ∞ y_t\to\infty y t → ∞ at least linearly by (48), the sharpness upper bound implies
λ 1 ( θ t ℓ ) ≤ λ 1 , + ∗ y ˉ + α − ( t − t s t a b ) ⟶ 0. \lambda_1(\theta_t^{\ell})\le\frac{\lambda_{1,+}^{*}}{\bar{y}+\alpha_{-}(t-t_{\mathrm{stab}})}\longrightarrow0. λ 1 ( θ t ℓ ) ≤ y ˉ + α − ( t − t stab ) λ 1 , + ∗ ⟶ 0.
Hence both thresholds
η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ϵ and η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 1 \eta(1+\delta_r)\lambda_1(\theta_t^{\ell})\le2-\epsilon
\qquad\text{and}\qquad
\eta(1+\delta_r)\lambda_1(\theta_t^{\ell})\le1 η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ϵ and η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 1
are reached in finite time. Since 2 − ϵ ≥ 1 2-\epsilon\ge1 2 − ϵ ≥ 1 for 0 < ϵ ≤ 1 0<\epsilon\le1 0 < ϵ ≤ 1 , we have
t s t a b ≤ t l i n ≤ t p o l y < ∞ . t_{\mathrm{stab}}\le t_{\mathrm{lin}}\le t_{\mathrm{poly}}<\infty. t stab ≤ t lin ≤ t poly < ∞.
Step 5: Transition phase. For t s t a b ≤ t ≤ t l i n t_{\mathrm{stab}}\le t\le t_{\mathrm{lin}} t stab ≤ t ≤ t lin , nonexpansion from Step 1 already gives
∥ θ t ⊥ ∥ ≤ ∥ θ t s t a b ⊥ ∥ . \lVert\theta_t^{\perp}\rVert\le\lVert\theta_{t_{\mathrm{stab}}}^{\perp}\rVert. ∥ θ t ⊥ ∥ ≤ ∥ θ t stab ⊥ ∥ .
This proves the transition-phase bound.
Step 6: Linear contraction phase. Assume now that t l i n ≤ t < t p o l y t_{\mathrm{lin}}\le t<t_{\mathrm{poly}} t lin ≤ t < t poly . By definition of t l i n t_{\mathrm{lin}} t lin ,
η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ϵ . \eta(1+\delta_r)\lambda_1(\theta_t^{\ell})\le2-\epsilon. η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ϵ .
By definition of t p o l y t_{\mathrm{poly}} t poly , for t < t p o l y t<t_{\mathrm{poly}} t < t poly ,
η ( 1 + δ r ) λ 1 ( θ t ℓ ) > 1. \eta(1+\delta_r)\lambda_1(\theta_t^{\ell})>1. η ( 1 + δ r ) λ 1 ( θ t ℓ ) > 1.
Using tail conditioning and relative curvature comparability,
M t ⪰ ( 1 − δ r ) H ( θ t ℓ ) ⪰ ρ t a i l ( 1 − δ r ) λ 1 ( θ t ℓ ) I U . M_t\succeq(1-\delta_r)H(\theta_t^{\ell})\succeq\rho_{\mathrm{tail}}(1-\delta_r)\lambda_1(\theta_t^{\ell})I_U. M t ⪰ ( 1 − δ r ) H ( θ t ℓ ) ⪰ ρ tail ( 1 − δ r ) λ 1 ( θ t ℓ ) I U .
Therefore every eigenvalue m m m of M t M_t M t satisfies
η m ≥ η ρ t a i l ( 1 − δ r ) λ 1 ( θ t ℓ ) = ρ t a i l ( 1 − δ r ) 1 + δ r η ( 1 + δ r ) λ 1 ( θ t ℓ ) > ρ t a i l ( 1 − δ r ) 1 + δ r ≕ ϵ ∗ . \begin{aligned}
\eta m\ge\eta\rho_{\mathrm{tail}}(1-\delta_r)\lambda_1(\theta_t^{\ell})
=\frac{\rho_{\mathrm{tail}}(1-\delta_r)}{1+\delta_r}\eta(1+\delta_r)\lambda_1(\theta_t^{\ell}) \\
&>\frac{\rho_{\mathrm{tail}}(1-\delta_r)}{1+\delta_r}\eqqcolon\epsilon_*.
\end{aligned} η m ≥ η ρ tail ( 1 − δ r ) λ 1 ( θ t ℓ ) = 1 + δ r ρ tail ( 1 − δ r ) η ( 1 + δ r ) λ 1 ( θ t ℓ ) > 1 + δ r ρ tail ( 1 − δ r ) = : ϵ ∗ .
Since ϵ ≤ ϵ ∗ \epsilon\le\epsilon_* ϵ ≤ ϵ ∗ , we have
η m ≥ ϵ . \eta m\ge\epsilon. η m ≥ ϵ .
The upper bound gives
η m ≤ η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ε . \eta m \le\eta(1+\delta_r)\lambda_1(\theta_t^\ell) \le2-\varepsilon. η m ≤ η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 2 − ε .
Thus all eigenvalues of η M t \eta M_t η M t lie in
[ ε , 2 − ε ] . [\varepsilon,2-\varepsilon]. [ ε , 2 − ε ] .
Hence
∥ I − η M t ∥ o p ≤ 1 − ε . \lVert I-\eta M_t\rVert_{\mathrm{op}}\le1-\varepsilon. ∥ I − η M t ∥ op ≤ 1 − ε .
Therefore
∥ θ t + 1 ± ∥ ≤ ( 1 − ε ) ∥ θ t ± ∥ . \lVert\theta_{t+1}^{\pm}\rVert\le(1-\varepsilon)\lVert\theta_t^{\pm}\rVert. ∥ θ t + 1 ± ∥ ≤ ( 1 − ε ) ∥ θ t ± ∥ .
Iterating from t l i n t_{\mathrm{lin}} t lin to t t t yields
∥ θ t ± ∥ ≤ ( 1 − ε ) t − t l i n ∥ θ t l i n ± ∥ ( t l i n ≤ t ≤ t p o l y ) . \lVert\theta_t^{\pm}\rVert\le(1-\varepsilon)^{t-t_{\mathrm{lin}}}\lVert\theta_{t_{\mathrm{lin}}}^{\pm}\rVert\qquad(t_{\mathrm{lin}}\le t\le t_{\mathrm{poly}}). ∥ θ t ± ∥ ≤ ( 1 − ε ) t − t lin ∥ θ t lin ± ∥ ( t lin ≤ t ≤ t poly ) .
Step 7: Polynomial decay phase. For t ≥ t p o l y t\ge t_{\mathrm{poly}} t ≥ t poly , the definition of t p o l y t_{\mathrm{poly}} t poly gives
η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 1. \eta(1+\delta_r)\lambda_1(\theta_t^\ell)\le1. η ( 1 + δ r ) λ 1 ( θ t ℓ ) ≤ 1.
Hence
0 ⪯ η M t ⪯ I U . 0\preceq\eta M_t\preceq I_U. 0 ⪯ η M t ⪯ I U .
Moreover, by tail conditioning and relative curvature comparability,
M t ⪰ ρ t a i l ( 1 − δ r ) λ 1 ( θ t ℓ ) I U . M_t\succeq\rho_{\mathrm{tail}}(1-\delta_r)\lambda_1(\theta_t^\ell)I_U. M t ⪰ ρ tail ( 1 − δ r ) λ 1 ( θ t ℓ ) I U .
Thus
∥ I − η M t ∥ o p ≤ 1 − η ρ t a i l ( 1 − δ r ) λ 1 ( θ t ℓ ) . \lVert I-\eta M_t\rVert_{\mathrm{op}}\le1-\eta\rho_{\mathrm{tail}}(1-\delta_r)\lambda_1(\theta_t^\ell). ∥ I − η M t ∥ op ≤ 1 − η ρ tail ( 1 − δ r ) λ 1 ( θ t ℓ ) .
Therefore, for s ≥ t p o l y s\ge t_{\mathrm{poly}} s ≥ t poly ,
∥ θ s + 1 ± ∥ ≤ ( 1 − η ρ t a i l ( 1 − δ r ) λ 1 ( θ s ℓ ) ) ∥ θ s ± ∥ . \lVert\theta_{s+1}^{\pm}\rVert\le\left(1-\eta\rho_{\mathrm{tail}}(1-\delta_r)\lambda_1(\theta_s^\ell)\right)\lVert\theta_s^{\pm}\rVert. ∥ θ s + 1 ± ∥ ≤ ( 1 − η ρ tail ( 1 − δ r ) λ 1 ( θ s ℓ ) ) ∥ θ s ± ∥ .
Iterating from s = t p o l y s=t_{\mathrm{poly}} s = t poly to s = t − 1 s=t-1 s = t − 1 , and using 1 − a ≤ e − a 1-a\le e^{-a} 1 − a ≤ e − a , gives
∥ θ t ± ∥ ≤ ∥ θ t p o l y ± ∥ exp ( − η ρ t a i l ( 1 − δ r ) ∑ s = t p o l y t − 1 λ 1 ( θ s ℓ ) ) . \lVert\theta_t^{\pm}\rVert\le\lVert\theta_{t_{\mathrm{poly}}}^{\pm}\rVert\exp\left(-\eta\rho_{\mathrm{tail}}(1-\delta_r)\sum_{s=t_{\mathrm{poly}}}^{t-1}\lambda_1(\theta_s^\ell)\right). ∥ θ t ± ∥ ≤ ∥ θ t poly ± ∥ exp − η ρ tail ( 1 − δ r ) s = t poly ∑ t − 1 λ 1 ( θ s ℓ ) .
From the sharpness lower schedule,
λ 1 ( θ s ℓ ) ≥ λ 1 , − ∗ y ˉ + α + ( s − t s t a b ) . \lambda_1(\theta_s^\ell)\ge\frac{\lambda_{1,-}^*}{\bar y+\alpha_+(s-t_{\mathrm{stab}})}. λ 1 ( θ s ℓ ) ≥ y ˉ + α + ( s − t stab ) λ 1 , − ∗ .
Define
A : = ⌈ y ˉ α + ⌉ + 1. A:=\left\lceil\frac{\bar y}{\alpha_+}\right\rceil+1. A := ⌈ α + y ˉ ⌉ + 1.
Then
y ˉ ≤ α + ( A − 1 ) , \bar y\le\alpha_+(A-1), y ˉ ≤ α + ( A − 1 ) ,
so
y ˉ + α + ( s − t s t a b ) ≤ α + ( A + s − t s t a b ) . \bar y+\alpha_+(s-t_{\mathrm{stab}})\le\alpha_+(A+s-t_{\mathrm{stab}}). y ˉ + α + ( s − t stab ) ≤ α + ( A + s − t stab ) .
Consequently,
λ 1 ( θ s ℓ ) ≥ λ 1 , − ∗ α + ( A + s − t s t a b ) . \lambda_1(\theta_s^\ell)\ge\frac{\lambda_{1,-}^*}{\alpha_+(A+s-t_{\mathrm{stab}})}. λ 1 ( θ s ℓ ) ≥ α + ( A + s − t stab ) λ 1 , − ∗ .
Therefore
∑ s = t p o l y t − 1 λ 1 ( θ s ℓ ) ≥ λ 1 , − ∗ α + ∑ s = t p o l y t − 1 1 A + s − t s t a b ≥ λ 1 , − ∗ α + log ( A + t − t s t a b A + t p o l y − t s t a b ) . \begin{aligned}
\sum_{s=t_{\mathrm{poly}}}^{t-1}\lambda_1(\theta_s^\ell)
\ge\frac{\lambda_{1,-}^*}{\alpha_+}\sum_{s=t_{\mathrm{poly}}}^{t-1}\frac{1}{A+s-t_{\mathrm{stab}}} \\
&\ge\frac{\lambda_{1,-}^*}{\alpha_+}\log\left(\frac{A+t-t_{\mathrm{stab}}}{A+t_{\mathrm{poly}}-t_{\mathrm{stab}}}\right).
\end{aligned} s = t poly ∑ t − 1 λ 1 ( θ s ℓ ) ≥ α + λ 1 , − ∗ s = t poly ∑ t − 1 A + s − t stab 1 ≥ α + λ 1 , − ∗ log ( A + t poly − t stab A + t − t stab ) .
Substituting into the exponential estimate yields
∥ θ t ± ∥ ≤ ∥ θ t p o l y ± ∥ ( A + t p o l y − t s t a b A + t − t s t a b ) γ r , \lVert\theta_t^{\pm}\rVert\le\lVert\theta_{t_{\mathrm{poly}}}^{\pm}\rVert\left(\frac{A+t_{\mathrm{poly}}-t_{\mathrm{stab}}}{A+t-t_{\mathrm{stab}}}\right)^{\gamma_r}, ∥ θ t ± ∥ ≤ ∥ θ t poly ± ∥ ( A + t − t stab A + t poly − t stab ) γ r ,
where
γ r : = ρ t a i l ( 1 − δ r ) η λ 1 , − ∗ α + . \gamma_r:=\frac{\rho_{\mathrm{tail}}(1-\delta_r)\eta\lambda_{1,-}^*}{\alpha_+}. γ r := α + ρ tail ( 1 − δ r ) η λ 1 , − ∗ .
This is exactly the claimed polynomial decay estimate.
Remark 2 (Recovering Theorem 1). Theorem 1 in the main text is the union of Theorems 3 and 4.
Remark 3 (Relationship between ε \varepsilon ε and ε ∗ \varepsilon_* ε ∗ ). The parameter ε \varepsilon ε in the linear contraction phase is free within ( 0 , ε ∗ ] (0,\varepsilon_*] ( 0 , ε ∗ ] . Choosing ε = ε ∗ \varepsilon=\varepsilon_* ε = ε ∗ minimizes the contraction factor 1 − ε 1-\varepsilon 1 − ε but defines the latest possible t l i n t_{\mathrm{lin}} t lin . Choosing ε < ε ∗ \varepsilon<\varepsilon_* ε < ε ∗ gives a weaker per-step bound but an earlier onset of contraction, which can produce a tighter overall bound depending on the sharpness schedule. The main theorem fixes ε = ε ∗ \varepsilon=\varepsilon_* ε = ε ∗ for simplicity.
Convergence rates for the loss
(LE) Exponential loss envelope. There exist constants B L > 0 B_L>0 B L > 0 such that, on the tubular neighborhood traversed by the trajectory,
L ( θ ) ≤ B L e − β t a i l ( θ ∥ − θ 0 ∥ ) \mathcal{L}(\theta) \le B_L e^{-\beta_{\mathrm{tail}}(\theta^\parallel-\theta_0^\parallel)} L ( θ ) ≤ B L e − β tail ( θ ∥ − θ 0 ∥ )
for all θ ∥ ≥ θ 0 ∥ \theta^\parallel\ge\theta_0^\parallel θ ∥ ≥ θ 0 ∥ , ∥ θ ⊥ ∥ ≤ r \lVert\theta^\perp\rVert\le r ∥ θ ⊥ ∥ ≤ r .
Condition (LE) is an additional assumption on the normalization and decay of the loss itself; it does not follow from derivative bounds alone. It holds for the normalized logistic and multiclass cross-entropy losses considered below.
Corollary B.1 (Convergence of the Loss). Suppose Assumptions A and B hold, and suppose moreover that the exponential loss envelope (LE) holds on the slab-tubular neighborhood traversed by the trajectory. Then the following bounds hold.
(i) Averaged loss during the unstable phase. For every integer $1\le T\le t_{\mathrm{stab}}+1,
1 T ∑ t = 0 T − 1 L ( θ t ) ≤ B L T 1 − e − β t a i l η m ℓ T 1 − e − β t a i l η m ℓ ≤ B L T ( 1 − e − β t a i l η m ℓ ) . (47) \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)
\le\frac{B_L}{T}\frac{1-e^{-\beta_{\mathrm{tail}}\eta m_\ell T}}{1-e^{-\beta_{\mathrm{tail}}\eta m_\ell}}
\le\frac{B_L}{T(1-e^{-\beta_{\mathrm{tail}}\eta m_\ell})}. \tag*{(47)} T 1 t = 0 ∑ T − 1 L ( θ t ) ≤ T B L 1 − e − β tail η m ℓ 1 − e − β tail η m ℓ T ≤ T ( 1 − e − β tail η m ℓ ) B L . (47) In particular,
1 T ∑ t = 0 T − 1 L ( θ t ) = O ( T − 1 ) uniformly for 1 ≤ T ≤ t s t a b + 1. \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)=O(T^{-1})
\qquad\text{uniformly for }1\le T\le t_{\mathrm{stab}}+1. T 1 t = 0 ∑ T − 1 L ( θ t ) = O ( T − 1 ) uniformly for 1 ≤ T ≤ t stab + 1. (ii) Last-iterate loss after stabilization. For every t ≥ t s t a b t\ge t_{\mathrm{stab}} t ≥ t stab ,
L ( θ t ) ≤ B ~ L y ˉ + α − ( t − t s t a b ) ≤ B ~ L 1 + β t a i l η A ∗ − ( t − t s t a b ) , (48) \mathcal{L}(\theta_t)\le\frac{\widetilde B_L}{\bar y+\alpha_-(t-t_{\mathrm{stab}})}
\le\frac{\widetilde B_L}{1+\beta_{\mathrm{tail}}\eta A_*^-(t-t_{\mathrm{stab}})}, \tag*{(48)} L ( θ t ) ≤ y ˉ + α − ( t − t stab ) B L ≤ 1 + β tail η A ∗ − ( t − t stab ) B L , (48) where
B ~ L : = B L e − β t a i l ( θ ∗ − θ 0 ∥ ) , \widetilde B_L:=B_L e^{-\beta_{\mathrm{tail}}(\theta_*-\theta_0^\parallel)}, B L := B L e − β tail ( θ ∗ − θ 0 ∥ ) , and the last inequality uses y ˉ ≥ 1 \bar y\ge1 y ˉ ≥ 1 . Consequently,
L ( θ t ) = O ( ( 1 + t − t s t a b ) − 1 ) . \mathcal{L}(\theta_t)=O\left((1+t-t_{\mathrm{stab}})^{-1}\right). L ( θ t ) = O ( ( 1 + t − t stab ) − 1 ) .
Proof. For part (i), Theorem 1 implies that throughout the unstable phase the trajectory remains in the slab-tubular neighborhood and satisfies
θ t ∥ ≥ θ 0 ∥ + η m ℓ t ( 0 ≤ t ≤ t s t a b ) . \theta_t^\parallel\ge\theta_0^\parallel+\eta m_\ell t
\qquad(0\le t\le t_{\mathrm{stab}}). θ t ∥ ≥ θ 0 ∥ + η m ℓ t ( 0 ≤ t ≤ t stab ) .
Applying the loss envelope (LE) at θ t \theta_t θ t gives
L ( θ t ) ≤ B L e − β t a i l ( θ t ∥ − θ 0 ∥ ) ≤ B L e − β t a i l η m ℓ t . \mathcal{L}(\theta_t)\le B_L e^{-\beta_{\mathrm{tail}}(\theta_t^\parallel-\theta_0^\parallel)}
\le B_L e^{-\beta_{\mathrm{tail}}\eta m_\ell t}. L ( θ t ) ≤ B L e − β tail ( θ t ∥ − θ 0 ∥ ) ≤ B L e − β tail η m ℓ t .
Therefore, for every $1\le T\le t_{\mathrm{stab}}+1,
1 T ∑ t = 0 T − 1 L ( θ t ) ≤ B L T ∑ t = 0 T − 1 e − β t a i l η m ℓ t = B L T 1 − e − β t a i l η m ℓ T 1 − e − β t a i l η m ℓ . \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)
\le\frac{B_L}{T}\sum_{t=0}^{T-1}e^{-\beta_{\mathrm{tail}}\eta m_\ell t}
=\frac{B_L}{T}\frac{1-e^{-\beta_{\mathrm{tail}}\eta m_\ell T}}{1-e^{-\beta_{\mathrm{tail}}\eta m_\ell}}. T 1 t = 0 ∑ T − 1 L ( θ t ) ≤ T B L t = 0 ∑ T − 1 e − β tail η m ℓ t = T B L 1 − e − β tail η m ℓ 1 − e − β tail η m ℓ T .
Since 1 − e − β t a i l η m ℓ T ≤ 1 1-e^{-\beta_{\mathrm{tail}}\eta m_\ell T}\le1 1 − e − β tail η m ℓ T ≤ 1 , this proves (49).
For part (ii), rewrite the same loss envelope relative to θ ∗ \theta_* θ ∗ . For t ≥ t s t a b t\ge t_{\mathrm{stab}} t ≥ t stab , we have
L ( θ t ) ≤ B L e − β t a i l ( θ t ∥ − θ 0 ∥ ) . (49) \mathcal{L}(\theta_t)\le B_L e^{-\beta_{\mathrm{tail}}(\theta_t^\parallel-\theta_0^\parallel)}. \tag*{(49)} L ( θ t ) ≤ B L e − β tail ( θ t ∥ − θ 0 ∥ ) . (49)
Splitting the exponent gives
θ t ∥ − θ 0 ∥ = ( θ t ∥ − θ ∗ ) + ( θ ∗ − θ 0 ∥ ) , \theta_t^{\parallel}-\theta_0^{\parallel}=(\theta_t^{\parallel}-\theta_\ast)+(\theta_\ast-\theta_0^{\parallel}), θ t ∥ − θ 0 ∥ = ( θ t ∥ − θ ∗ ) + ( θ ∗ − θ 0 ∥ ) ,
and hence
L ( θ t ) ≤ B L e − β t a i l ( θ ∗ − θ 0 ∥ ) e − β t a i l ( θ t ∥ − θ ∗ ) . \mathcal{L}(\theta_t)\le B\mathcal{L}e^{-\beta_{\mathrm{tail}}(\theta_\ast-\theta_0^{\parallel})}e^{-\beta_{\mathrm{tail}}(\theta_t^{\parallel}-\theta_\ast)}. L ( θ t ) ≤ B L e − β tail ( θ ∗ − θ 0 ∥ ) e − β tail ( θ t ∥ − θ ∗ ) .
By definition,
B ~ L : = B L e − β t a i l ( θ ∗ − θ 0 ∥ ) \widetilde{B}_{\mathcal{L}}:=B\mathcal{L}e^{-\beta_{\mathrm{tail}}(\theta_\ast-\theta_0^{\parallel})} B L := B L e − β tail ( θ ∗ − θ 0 ∥ )
and
y t : = e β t a i l ( θ t ∥ − θ ∗ ) . y_t:=e^{\beta_{\mathrm{tail}}(\theta_t^{\parallel}-\theta_\ast)}. y t := e β tail ( θ t ∥ − θ ∗ ) .
Therefore,
L ( θ t ) ≤ B ~ L y t . \mathcal{L}(\theta_t)\le\frac{\widetilde{B}_{\mathcal{L}}}{y_t}. L ( θ t ) ≤ y t B L .
Using the previous inequality (from the stable phase proof)
y t ≥ y ‾ + α − ( t − t s t a b ) ( t ≥ t s t a b ) , y_t\ge\overline{y}+\alpha_-(t-t_{\mathrm{stab}})\qquad(t\ge t_{\mathrm{stab}}), y t ≥ y + α − ( t − t stab ) ( t ≥ t stab ) ,
we obtain
L ( θ t ) ≤ B ~ L y ‾ + α − ( t − t s t a b ) . \mathcal{L}(\theta_t)\le\frac{\widetilde{B}_{\mathcal{L}}}{\overline{y}+\alpha_-(t-t_{\mathrm{stab}})}. L ( θ t ) ≤ y + α − ( t − t stab ) B L .
In the stable phase y ‾ ≥ 1 \overline{y}\ge1 y ≥ 1 , which further implies
L ( θ t ) ≤ B ~ L 1 + α − ( t − t s t a b ) . \mathcal{L}(\theta_t)\le\frac{\widetilde{B}_{\mathcal{L}}}{1+\alpha_-(t-t_{\mathrm{stab}})}. L ( θ t ) ≤ 1 + α − ( t − t stab ) B L .
This proves (50). □
Remark 4 Comparison with sharper logistic loss-rate estimates. The averaged unstable phase bound in Corollary B.1 is intentionally stated under a minimal loss-envelope assumption. It only uses the facts that the trajectory remains in the tubular neighborhood and that the parallel coordinate makes uniform forward progress. Consequently, the proof bounds the loss by a geometric envelope along the trajectory, leading to
1 T ∑ t = 0 T − 1 L ( θ t ) ≤ B L T ( 1 − e − β t a i l η m ℓ ) . \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)\le\frac{B_{\mathcal{L}}}{T(1-e^{-\beta_{\mathrm{tail}}\eta m_{\ell}})}. T 1 t = 0 ∑ T − 1 L ( θ t ) ≤ T ( 1 − e − β tail η m ℓ ) B L .
This estimate is robust but not sharp in the large-step regime.
Sharper rates, such as the logarithmic averaged rates obtained in logistic-tail analyses of [27 , 39 ] , use additional structure: the loss must be comparable to the parallel drift throughout the relevant part of the oscillation tubular neighborhood. Namely, if on this portion of the tubular neighborhood one has
L ( θ ) ≤ C L [ − ⟨ ∇ L ( θ ) , v ⟩ ] , − ⟨ ∇ L ( θ ) , v ⟩ ≤ G ∥ e − β t a i l ( θ ∥ − θ 0 ∥ ) , \mathcal{L}(\theta)\le C_{\mathcal{L}}[-\langle\nabla\mathcal{L}(\theta),v\rangle],\qquad-\langle\nabla\mathcal{L}(\theta),v\rangle\le G_{\parallel}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_0^{\parallel})}, L ( θ ) ≤ C L [ − ⟨ ∇ L ( θ ) , v ⟩] , − ⟨ ∇ L ( θ ) , v ⟩ ≤ G ∥ e − β tail ( θ ∥ − θ 0 ∥ ) ,
then the GD update gives the exact telescoping relation
L ( θ t ) ≤ C L η ( θ t + 1 ∥ − θ t ∥ ) . \mathcal{L}(\theta_t)\le\frac{C_{\mathcal{L}}}{\eta}(\theta_{t+1}^{\parallel}-\theta_t^{\parallel}). L ( θ t ) ≤ η C L ( θ t + 1 ∥ − θ t ∥ ) .
Summing over t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 yields
1 T ∑ t = 0 T − 1 L ( θ t ) ≤ C L η T ( θ T ∥ − θ 0 ∥ ) . \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)\le\frac{C_{\mathcal{L}}}{\eta T}(\theta_T^{\parallel}-\theta_0^{\parallel}). T 1 t = 0 ∑ T − 1 L ( θ t ) ≤ η T C L ( θ T ∥ − θ 0 ∥ ) .
The second tail bound implies
θ T ∥ − θ 0 ∥ ≲ 1 β t a i l log ( 1 + η T ) , \theta_T^{\parallel}-\theta_0^{\parallel}\lesssim\frac{1}{\beta_{\mathrm{tail}}}\log(1+\eta T), θ T ∥ − θ 0 ∥ ≲ β tail 1 log ( 1 + η T ) ,
and hence
1 T ∑ t = 0 T − 1 L ( θ t ) ≲ log ( 1 + η T ) η T . \frac{1}{T}\sum_{t=0}^{T-1}\mathcal{L}(\theta_t)\lesssim\frac{\log(1+\eta T)}{\eta T}. T 1 t = 0 ∑ T − 1 L ( θ t ) ≲ η T log ( 1 + η T ) .
Thus the corollary should be viewed as a general envelope estimate, whereas the sharper Wu-type rate requires a loss–parallel-gradient comparability condition on the unstable portion of the tubular neighborhood. Logistic and cross-entropy losses satisfy this additional tail structure once the trajectory lies in the corresponding exponential-tail region.
Logistic Regression
Preliminaries
Proposition 2 (Full Hessian and its restriction to the parallel direction). Consider the high-dimensional symmetric logistic loss
L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ( ⟨ θ , x i ⟩ ) + ϕ ( − ⟨ θ , x i ′ ⟩ ) ] , x i = a i v + u i , x i ′ = − a i v + u i , (50) \mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi(\langle\theta,x_i\rangle)+\phi(-\langle\theta,x_i'\rangle)\right],\qquad x_i=a_i v+u_i,\qquad x_i'=-a_i v+u_i, \tag*{(50)} L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ (⟨ θ , x i ⟩) + ϕ ( − ⟨ θ , x i ′ ⟩) ] , x i = a i v + u i , x i ′ = − a i v + u i , (50) where v ∈ R d v\in\mathbb{R}^{d} v ∈ R d is a unit vector,
u i ∈ v ⊥ ∖ { 0 } , a i > 0 , i = 1 , … , N . u_i\in v^\perp\setminus\{0\},\qquad a_i>0,\qquad i=1,\ldots,N. u i ∈ v ⊥ ∖ { 0 } , a i > 0 , i = 1 , … , N . Then the full Hessian of L \mathcal{L} L at an arbitrary point θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d is
∇ 2 L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ′ ′ ( ⟨ θ , x i ⟩ ) x i x i ⊤ + ϕ ′ ′ ( − ⟨ θ , x i ′ ⟩ ) x i ′ ( x i ′ ) ⊤ ] . (51) \nabla^2\mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi''(\langle\theta,x_i\rangle)x_i x_i^\top+\phi''(-\langle\theta,x_i'\rangle)x_i'(x_i')^\top\right]. \tag*{(51)} ∇ 2 L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ′′ (⟨ θ , x i ⟩) x i x i ⊤ + ϕ ′′ ( − ⟨ θ , x i ′ ⟩) x i ′ ( x i ′ ) ⊤ ] . (51) In particular, along the parallel direction θ = θ ∥ v \theta=\theta^{\|}v θ = θ ∥ v ,
∇ 2 L ( θ ∥ v ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) ( a i 2 v v ⊤ + u i u i ⊤ ) . (52) \nabla^2\mathcal{L}(\theta^{\|}v)=\frac{1}{N}\sum_{i=1}^{N}\phi''(a_i\theta^{\|})\left(a_i^2vv^\top+u_i u_i^\top\right). \tag*{(52)} ∇ 2 L ( θ ∥ v ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) ( a i 2 v v ⊤ + u i u i ⊤ ) . (52) Therefore, the restriction of the Hessian to the active normal space
U : = span { u 1 , … , u N } ⊆ v ⊥ U:=\operatorname{span}\{u_1,\ldots,u_N\}\subseteq v^\perp U := span { u 1 , … , u N } ⊆ v ⊥ is
H ( θ ∥ ) : = Π U ∇ 2 L ( θ ∥ v ) Π U = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) u i u i ⊤ . (53) H(\theta^{\|}):=\Pi_U\nabla^2\mathcal{L}(\theta^{\|}v)\Pi_U=\frac{1}{N}\sum_{i=1}^{N}\phi''(a_i\theta^{\|})u_i u_i^\top. \tag*{(53)} H ( θ ∥ ) := Π U ∇ 2 L ( θ ∥ v ) Π U = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) u i u i ⊤ . (53)
Proof. We first compute the gradient on the full space:
∇ L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ′ ( ⟨ θ , x i ⟩ ) x i − ϕ ′ ( − ⟨ θ , x i ′ ⟩ ) x i ′ ] . \nabla\mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi'(\langle\theta,x_i\rangle)x_i-\phi'(-\langle\theta,x_i'\rangle)x_i'\right]. ∇ L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ′ (⟨ θ , x i ⟩) x i − ϕ ′ ( − ⟨ θ , x i ′ ⟩) x i ′ ] .
Since ϕ ′ ′ ( − s ) = ϕ ′ ′ ( s ) \phi''(-s)=\phi''(s) ϕ ′′ ( − s ) = ϕ ′′ ( s ) , differentiating once more gives
∇ 2 L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ′ ′ ( ⟨ θ , x i ⟩ ) x i x i ⊤ + ϕ ′ ′ ( − ⟨ θ , x i ′ ⟩ ) x i ′ ( x i ′ ) ⊤ ] , \nabla^2\mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi''(\langle\theta,x_i\rangle)x_i x_i^\top+\phi''(-\langle\theta,x_i'\rangle)x_i'(x_i')^\top\right], ∇ 2 L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ′′ (⟨ θ , x i ⟩) x i x i ⊤ + ϕ ′′ ( − ⟨ θ , x i ′ ⟩) x i ′ ( x i ′ ) ⊤ ] ,
which is exactly (52).
Now evaluate at θ = θ ∥ v \theta=\theta^{\|}v θ = θ ∥ v . Since u i ⊥ v u_i\perp v u i ⊥ v ,
⟨ θ ∥ v , x i ⟩ = a i θ ∥ , − ⟨ θ ∥ v , x i ′ ⟩ = − ( − a i θ ∥ ) = a i θ ∥ . \langle\theta^{\|}v,x_i\rangle=a_i\theta^{\|},\qquad-\langle\theta^{\|}v,x_i'\rangle=-(-a_i\theta^{\|})=a_i\theta^{\|}. ⟨ θ ∥ v , x i ⟩ = a i θ ∥ , − ⟨ θ ∥ v , x i ′ ⟩ = − ( − a i θ ∥ ) = a i θ ∥ .
Using ϕ ′ ′ ( − s ) = ϕ ′ ′ ( s ) \phi''(-s)=\phi''(s) ϕ ′′ ( − s ) = ϕ ′′ ( s ) , we obtain
∇ 2 L ( θ ∥ v ) = 1 2 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) [ ( a i v + u i ) ( a i v + u i ) ⊤ + ( − a i v + u i ) ( − a i v + u i ) ⊤ ] . \nabla^2\mathcal{L}(\theta^{\|}v)=\frac{1}{2N}\sum_{i=1}^{N}\phi''(a_i\theta^{\|})\left[(a_i v+u_i)(a_i v+u_i)^\top+(-a_i v+u_i)(-a_i v+u_i)^\top\right]. ∇ 2 L ( θ ∥ v ) = 2 N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) [ ( a i v + u i ) ( a i v + u i ) ⊤ + ( − a i v + u i ) ( − a i v + u i ) ⊤ ] .
Expanding the two rank-one matrices,
( a i v + u i ) ( a i v + u i ) ⊤ = a i 2 v v ⊤ + a i ( v u i ⊤ + u i v ⊤ ) + u i u i ⊤ , (a_i v+u_i)(a_i v+u_i)^\top=a_i^2vv^\top+a_i(vu_i^\top+u_i v^\top)+u_i u_i^\top, ( a i v + u i ) ( a i v + u i ) ⊤ = a i 2 v v ⊤ + a i ( v u i ⊤ + u i v ⊤ ) + u i u i ⊤ ,
and
( − a i v + u i ) ( − a i v + u i ) ⊤ = a i 2 v v ⊤ − a i ( v u i ⊤ + u i v ⊤ ) + u i u i ⊤ . (-a_i v+u_i)(-a_i v+u_i)^\top=a_i^2vv^\top-a_i(vu_i^\top+u_i v^\top)+u_i u_i^\top. ( − a i v + u i ) ( − a i v + u i ) ⊤ = a i 2 v v ⊤ − a i ( v u i ⊤ + u i v ⊤ ) + u i u i ⊤ .
Adding them cancels the mixed terms:
( a i v + u i ) ( a i v + u i ) ⊤ + ( − a i v + u i ) ( − a i v + u i ) ⊤ = 2 a i 2 v v ⊤ + 2 u i u i ⊤ . (a_i v+u_i)(a_i v+u_i)^\top+(-a_i v+u_i)(-a_i v+u_i)^\top=2a_i^2vv^\top+2u_i u_i^\top. ( a i v + u i ) ( a i v + u i ) ⊤ + ( − a i v + u i ) ( − a i v + u i ) ⊤ = 2 a i 2 v v ⊤ + 2 u i u i ⊤ .
Substituting back yields (53).
Finally, since u i ∈ U ⊆ v ⊥ u_i\in U\subseteq v^\perp u i ∈ U ⊆ v ⊥ , we have
Π U v v ⊤ Π U = 0 , Π U u i u i ⊤ Π U = u i u i ⊤ , \Pi_Uvv^\top\Pi_U=0,\qquad\Pi_Uu_i u_i^\top\Pi_U=u_i u_i^\top, Π U v v ⊤ Π U = 0 , Π U u i u i ⊤ Π U = u i u i ⊤ ,
so projecting ∇ 2 L ( θ ∥ v ) \nabla^2\mathcal{L}(\theta^{\|}v) ∇ 2 L ( θ ∥ v ) onto U U U gives (54).
Proposition 3 (Full third derivative tensor and its restriction to the parallel direction). Consider the high-dimensional symmetric logistic loss
L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ( ⟨ θ , x i ⟩ ) + ϕ ( − ⟨ θ , x i ′ ⟩ ) ] , x i = a i v + u i , x i ′ = − a i v + u i , (54) \mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi(\langle\theta,x_i\rangle)+\phi(-\langle\theta,x_i'\rangle)\right],\qquad x_i=a_i v+u_i,\qquad x_i'=-a_i v+u_i, \tag*{(54)} L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ (⟨ θ , x i ⟩) + ϕ ( − ⟨ θ , x i ′ ⟩) ] , x i = a i v + u i , x i ′ = − a i v + u i , (54) where v ∈ R d v\in\mathbb{R}^{d} v ∈ R d is a unit vector,
u i ∈ v ⊥ ∖ { 0 } , a i > 0 , i = 1 , … , N . u_i\in v^\perp\setminus\{0\},\qquad a_i>0,\qquad i=1,\ldots,N. u i ∈ v ⊥ ∖ { 0 } , a i > 0 , i = 1 , … , N . Then the full third derivative tensor of L \mathcal{L} L at an arbitrary point θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d is
D 3 L ( θ ) [ h , k , ℓ ] = 1 2 N ∑ i = 1 N [ ϕ ′ ′ ′ ( ⟨ θ , x i ⟩ ) ⟨ h , x i ⟩ ⟨ k , x i ⟩ ⟨ ℓ , x i ⟩ − ϕ ′ ′ ′ ( − ⟨ θ , x i ′ ⟩ ) ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ ⟨ ℓ , x i ′ ⟩ ] (55) D^3\mathcal{L}(\theta)[h,k,\ell]=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi'''(\langle\theta,x_i\rangle)\langle h,x_i\rangle\langle k,x_i\rangle\langle\ell,x_i\rangle-\phi'''(-\langle\theta,x_i'\rangle)\langle h,x_i'\rangle\langle k,x_i'\rangle\langle\ell,x_i'\rangle\right] \tag*{(55)} D 3 L ( θ ) [ h , k , ℓ ] = 2 N 1 i = 1 ∑ N [ ϕ ′′′ (⟨ θ , x i ⟩) ⟨ h , x i ⟩ ⟨ k , x i ⟩ ⟨ ℓ , x i ⟩ − ϕ ′′′ ( − ⟨ θ , x i ′ ⟩) ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ ⟨ ℓ , x i ′ ⟩ ] (55) for all h , k , ℓ ∈ R d h,k,\ell\in\mathbb{R}^{d} h , k , ℓ ∈ R d .
Equivalently, the vector obtained by contracting in two slots is
D 3 L ( θ ) [ h , k ] = 1 2 N ∑ i = 1 N [ ϕ ′ ′ ′ ( ⟨ θ , x i ⟩ ) ⟨ h , x i ⟩ ⟨ k , x i ⟩ x i − ϕ ′ ′ ′ ( − ⟨ θ , x i ′ ⟩ ) ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ x i ′ ] . (56) D^3\mathcal{L}(\theta)[h,k]=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi'''(\langle\theta,x_i\rangle)\langle h,x_i\rangle\langle k,x_i\rangle x_i-\phi'''(-\langle\theta,x_i'\rangle)\langle h,x_i'\rangle\langle k,x_i'\rangle x_i'\right]. \tag*{(56)} D 3 L ( θ ) [ h , k ] = 2 N 1 i = 1 ∑ N [ ϕ ′′′ (⟨ θ , x i ⟩) ⟨ h , x i ⟩ ⟨ k , x i ⟩ x i − ϕ ′′′ ( − ⟨ θ , x i ′ ⟩) ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ x i ′ ] . (56) In particular, along the parallel direction θ = θ ∥ v \theta=\theta^\parallel v θ = θ ∥ v , one has
D 3 L ( θ ∥ v ) [ h , k ] = 1 2 N ∑ i = 1 N ϕ ′ ′ ′ ( a i θ ∥ ) [ ⟨ h , x i ⟩ ⟨ k , x i ⟩ x i − ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ x i ′ ] . (57) D^3\mathcal{L}(\theta^\parallel v)[h,k]=\frac{1}{2N}\sum_{i=1}^{N}\phi'''(a_i\theta^\parallel)\left[\langle h,x_i\rangle\langle k,x_i\rangle x_i-\langle h,x_i'\rangle\langle k,x_i'\rangle x_i'\right]. \tag*{(57)} D 3 L ( θ ∥ v ) [ h , k ] = 2 N 1 i = 1 ∑ N ϕ ′′′ ( a i θ ∥ ) [ ⟨ h , x i ⟩ ⟨ k , x i ⟩ x i − ⟨ h , x i ′ ⟩ ⟨ k , x i ′ ⟩ x i ′ ] . (57) Moreover, for every θ ⊥ ∈ U : = span { u 1 , … , u N } ⊆ v ⊥ \theta^\perp\in U:=\operatorname{span}\{u_1,\ldots,u_N\}\subseteq v^\perp θ ⊥ ∈ U := span { u 1 , … , u N } ⊆ v ⊥ , the pure normal contraction satisfies
D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 v . (58) D^3\mathcal{L}(\theta^\parallel v)[\theta^\perp,\theta^\perp]=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta^\parallel)\langle\theta^\perp,u_i\rangle^2v. \tag*{(58)} D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 v . (58) Hence D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] D^3\mathcal{L}(\theta^\parallel v)[\theta^\perp,\theta^\perp] D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] is purely parallel.
Equivalently,
⟨ D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] , v ⟩ = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 = ( θ ⊥ ) T H ′ ( θ ∥ ) θ ⊥ . (59) \left\langle D^3\mathcal{L}(\theta^\parallel v)[\theta^\perp,\theta^\perp],v\right\rangle=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta^\parallel)\langle\theta^\perp,u_i\rangle^2=(\theta^\perp)^{\mathsf T}H'(\theta^\parallel)\theta^\perp. \tag*{(59)} ⟨ D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] , v ⟩ = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 = ( θ ⊥ ) T H ′ ( θ ∥ ) θ ⊥ . (59)
Proof. For a single linear form L ( θ ) = ϕ ( ⟨ θ , a ⟩ ) L(\theta)=\phi(\langle\theta,a\rangle) L ( θ ) = ϕ (⟨ θ , a ⟩) , repeated application of the chain rule gives
∇ L ( θ ) = ϕ ′ ( ⟨ θ , a ⟩ ) a , \nabla L(\theta)=\phi'(\langle\theta,a\rangle)a, ∇ L ( θ ) = ϕ ′ (⟨ θ , a ⟩) a ,
∇ 2 L ( θ ) = ϕ ′ ′ ( ⟨ θ , a ⟩ ) a a T , \nabla^2L(\theta)=\phi''(\langle\theta,a\rangle)aa^{\mathsf T}, ∇ 2 L ( θ ) = ϕ ′′ (⟨ θ , a ⟩) a a T ,
and
D 3 L ( θ ) [ h , k , ℓ ] = ϕ ′ ′ ′ ( ⟨ θ , a ⟩ ) ⟨ h , a ⟩ ⟨ k , a ⟩ ⟨ ℓ , a ⟩ . D^3L(\theta)[h,k,\ell]=\phi'''(\langle\theta,a\rangle)\langle h,a\rangle\langle k,a\rangle\langle\ell,a\rangle. D 3 L ( θ ) [ h , k , ℓ ] = ϕ ′′′ (⟨ θ , a ⟩) ⟨ h , a ⟩ ⟨ k , a ⟩ ⟨ ℓ , a ⟩ .
Applying this to ϕ ( ⟨ θ , x i ⟩ ) \phi(\langle\theta,x_i\rangle) ϕ (⟨ θ , x i ⟩) and to ϕ ( − ⟨ θ , x i ′ ⟩ ) \phi(-\langle\theta,x_i'\rangle) ϕ ( − ⟨ θ , x i ′ ⟩) yields (56); the minus sign comes from the product of the three factors − 1 -1 − 1 contributed by the chain rule for ϕ ( − ⟨ θ , x i ′ ⟩ ) \phi(-\langle\theta,x_i'\rangle) ϕ ( − ⟨ θ , x i ′ ⟩) . The vector formula (57) is the equivalent representation obtained by contracting in the first two slots.
Now evaluate at θ = θ ∥ v \theta=\theta^\parallel v θ = θ ∥ v . Since u i ⊥ v u_i\perp v u i ⊥ v ,
⟨ θ ∥ v , x i ⟩ = a i θ ∥ , − ⟨ θ ∥ v , x i ′ ⟩ = a i θ ∥ . \langle\theta^\parallel v,x_i\rangle=a_i\theta^\parallel,\qquad-\langle\theta^\parallel v,x_i'\rangle=a_i\theta^\parallel. ⟨ θ ∥ v , x i ⟩ = a i θ ∥ , − ⟨ θ ∥ v , x i ′ ⟩ = a i θ ∥ .
Using ϕ ′ ′ ′ ( − s ) = − ϕ ′ ′ ′ ( s ) \phi'''(-s)=-\phi'''(s) ϕ ′′′ ( − s ) = − ϕ ′′′ ( s ) is not needed here because the argument of the second term is already a i θ ∥ a_i\theta^\parallel a i θ ∥ . We obtain (58).
Finally, let θ ⊥ ∈ U ⊆ v ⊥ \theta^\perp\in U\subseteq v^\perp θ ⊥ ∈ U ⊆ v ⊥ . Then
⟨ θ ⊥ , x i ⟩ = ⟨ θ ⊥ , u i ⟩ , ⟨ θ ⊥ , x i ′ ⟩ = ⟨ θ ⊥ , u i ⟩ , \langle\theta^\perp,x_i\rangle=\langle\theta^\perp,u_i\rangle,\qquad\langle\theta^\perp,x_i'\rangle=\langle\theta^\perp,u_i\rangle, ⟨ θ ⊥ , x i ⟩ = ⟨ θ ⊥ , u i ⟩ , ⟨ θ ⊥ , x i ′ ⟩ = ⟨ θ ⊥ , u i ⟩ ,
so the two scalar factors coincide:
⟨ θ ⊥ , x i ⟩ 2 = ⟨ θ ⊥ , x i ′ ⟩ 2 = ⟨ θ ⊥ , u i ⟩ 2 . \langle\theta^\perp,x_i\rangle^2=\langle\theta^\perp,x_i'\rangle^2=\langle\theta^\perp,u_i\rangle^2. ⟨ θ ⊥ , x i ⟩ 2 = ⟨ θ ⊥ , x i ′ ⟩ 2 = ⟨ θ ⊥ , u i ⟩ 2 .
Therefore
D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = 1 2 N ∑ i = 1 N ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 [ ( a i v + u i ) − ( − a i v + u i ) ] . D^3\mathcal{L}(\theta\|v)[\theta^\perp,\theta^\perp]=\frac{1}{2N}\sum_{i=1}^{N}\phi'''(a_i\theta\|)\langle\theta^\perp,u_i\rangle^2[(a_iv+u_i)-(-a_iv+u_i)]. D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = 2 N 1 i = 1 ∑ N ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 [( a i v + u i ) − ( − a i v + u i )] .
and the u i u_i u i -terms cancel:
( a i v + u i ) − ( − a i v + u i ) = 2 a i v . (a_iv+u_i)-(-a_iv+u_i)=2a_iv. ( a i v + u i ) − ( − a i v + u i ) = 2 a i v .
Hence
D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 v , D^3\mathcal{L}(\theta\|v)[\theta^\perp,\theta^\perp]=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta\|)\langle\theta^\perp,u_i\rangle^2v, D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 v ,
which proves (59).
Taking the inner product with v v v gives
⟨ D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] , v ⟩ = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 . \langle D^3\mathcal{L}(\theta\|v)[\theta^\perp,\theta^\perp],v\rangle=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta\|)\langle\theta^\perp,u_i\rangle^2. ⟨ D 3 L ( θ ∥ v ) [ θ ⊥ , θ ⊥ ] , v ⟩ = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 .
On the other hand,
( θ ⊥ ) ⊤ H ′ ( θ ∥ ) θ ⊥ = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ( θ ⊥ ) ⊤ ( u i u i ⊤ ) θ ⊥ = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 . (\theta^\perp)^\top H'(\theta\|)\theta^\perp=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta\|)(\theta^\perp)^\top(u_iu_i^\top)\theta^\perp=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'''(a_i\theta\|)\langle\theta^\perp,u_i\rangle^2. ( θ ⊥ ) ⊤ H ′ ( θ ∥ ) θ ⊥ = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ( θ ⊥ ) ⊤ ( u i u i ⊤ ) θ ⊥ = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) ⟨ θ ⊥ , u i ⟩ 2 .
Thus (60) holds. □
Verification of Assumptions A-B
Proposition 4 (Verification of assumptions for logistic regression). Consider the pair-symmetric logistic loss
L ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ( ⟨ θ , x i + ⟩ ) + ϕ ( ⟨ θ , − x i − ⟩ ) ] , ϕ ( s ) : = log ( 1 + e − s ) , (60) \mathcal{L}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi(\langle\theta,x_i^+\rangle)+\phi(\langle\theta,-x_i^-\rangle)\right],\qquad\phi(s):=\log(1+e^{-s}), \tag*{(60)} L ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ (⟨ θ , x i + ⟩) + ϕ (⟨ θ , − x i − ⟩) ] , ϕ ( s ) := log ( 1 + e − s ) , (60) where
x i + = a i v + u i , x i − = − a i v + u i , a i > 0 , u i ∈ v ⊥ ∖ { 0 } . x_i^+=a_iv+u_i,\qquad x_i^-=-a_iv+u_i,\qquad a_i>0,\qquad u_i\in v^\perp\setminus\{0\}. x i + = a i v + u i , x i − = − a i v + u i , a i > 0 , u i ∈ v ⊥ ∖ { 0 } . Let
U : = span { u 1 , … , u N } , u max : = max 1 ≤ i ≤ N ∥ u i ∥ , a min : = min 1 ≤ i ≤ N a i , U:=\operatorname{span}\{u_1,\ldots,u_N\},\qquad u_{\max}:=\max_{1\le i\le N}\|u_i\|,\qquad a_{\min}:=\min_{1\le i\le N}a_i, U := span { u 1 , … , u N } , u m a x := 1 ≤ i ≤ N max ∥ u i ∥ , a m i n := 1 ≤ i ≤ N min a i , and define the slowest-layer index set and frame operators
I ∗ : = { i : a i = a min } , Σ : = 1 N ∑ i = 1 N u i u i ⊤ , Σ ∗ : = 1 N ∑ i ∈ I ∗ u i u i ⊤ . I_*:=\{i:a_i=a_{\min}\},\qquad\Sigma:=\frac{1}{N}\sum_{i=1}^{N}u_iu_i^\top,\qquad\Sigma_*:=\frac{1}{N}\sum_{i\in I_*}u_iu_i^\top. I ∗ := { i : a i = a m i n } , Σ := N 1 i = 1 ∑ N u i u i ⊤ , Σ ∗ := N 1 i ∈ I ∗ ∑ u i u i ⊤ . Then the logistic loss satisfies Assumptions A and B with the following:
(i) Subspace restricted objects. Writing θ = θ ∥ v + θ ⊥ \theta=\theta^\parallel v+\theta^\perp θ = θ ∥ v + θ ⊥ with θ ⊥ ∈ U \theta^\perp\in U θ ⊥ ∈ U , the loss and Hessian matrix restricted to the parallel direction are
ℓ ( θ ∥ ) = 1 N ∑ i = 1 N ϕ ( a i θ ∥ ) , H ( θ ∥ ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) u i u i ⊤ . \ell(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}\phi(a_i\theta^\parallel),\qquad H(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}\phi''(a_i\theta^\parallel)u_iu_i^\top. ℓ ( θ ∥ ) = N 1 i = 1 ∑ N ϕ ( a i θ ∥ ) , H ( θ ∥ ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) u i u i ⊤ . Moreover,
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i q ( a i θ ∥ ) , q ( s ) : = 1 1 + e s . -\ell'(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta^\parallel),\qquad q(s):=\frac{1}{1+e^s}. − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q ( a i θ ∥ ) , q ( s ) := 1 + e s 1 .
(ii) RC tolerance. For every tubular neighborhood radius r > 0 r > 0 r > 0 , relative curvature comparability holds with
δ r = cosh ( u max r ) − 1 = 1 2 ( u max r ) 2 + O ( r 4 ) . \delta_r = \cosh(u_{\max}r) - 1 = \frac{1}{2}(u_{\max}r)^2 + O(r^4). δ r = cosh ( u m a x r ) − 1 = 2 1 ( u m a x r ) 2 + O ( r 4 ) .
That is, for all relevant θ ∥ \theta^{\parallel} θ ∥ and all ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\| \le r ∥ θ ⊥ ∥ ≤ r ,
( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ∥ v + θ ⊥ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) . (1-\delta_r)H(\theta^{\parallel}) \preceq D_{\theta^{\perp}}\nabla L^{\perp}(\theta^{\parallel}v+\theta^{\perp}) \preceq(1+\delta_r)H(\theta^{\parallel}). ( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ∥ v + θ ⊥ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) .
To verify Assumption A.2. with δ r < 1 \delta_r < 1 δ r < 1 , we restrict to
0 < r < arcosh ( 2 ) u max . 0 < r < \frac{\operatorname{arcosh}(2)}{u_{\max}}. 0 < r < u m a x arcosh ( 2 ) .
(iii) Local Curvature Decay. Fix any θ ˉ ∥ > 0 \bar{\theta}^{\parallel} > 0 θ ˉ ∥ > 0 . For every bounded interval J = [ θ 0 ∥ , θ 1 ∥ ] ⊆ [ θ ˉ ∥ , ∞ ) J=[\theta_0^{\parallel},\theta_1^{\parallel}] \subseteq[\bar{\theta}^{\parallel},\infty) J = [ θ 0 ∥ , θ 1 ∥ ] ⊆ [ θ ˉ ∥ , ∞ ) ,
− H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) ∀ θ ∥ ∈ J , -H'(\theta^{\parallel}) \succeq\beta_J H(\theta^{\parallel}) \qquad\forall\theta^{\parallel}\in J, − H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) ∀ θ ∥ ∈ J ,
where
β J : = a min tanh ( a min θ 0 ∥ 2 ) > 0. \beta_J := a_{\min}\tanh\left(\frac{a_{\min}\theta_0^{\parallel}}{2}\right) > 0. β J := a m i n tanh ( 2 a m i n θ 0 ∥ ) > 0.
In particular, on the oscillation slab [ θ 0 ∥ , θ ∗ ∥ ] [\theta_0^{\parallel},\theta_*^{\parallel}] [ θ 0 ∥ , θ ∗ ∥ ] , one may take
β : = a min tanh ( a min θ 0 ∥ 2 ) . \beta:= a_{\min}\tanh\left(\frac{a_{\min}\theta_0^{\parallel}}{2}\right). β := a m i n tanh ( 2 a m i n θ 0 ∥ ) .
(iv) Unstable drift constants. On the slab [ θ 0 ∥ , θ ∗ ∥ ] [\theta_0^{\parallel},\theta_*^{\parallel}] [ θ 0 ∥ , θ ∗ ∥ ] , the parallel drift satisfies
− ℓ ′ ( θ ∥ ) ≥ m ℓ , m ℓ : = 1 N ∑ i = 1 N a i q ( a i θ 0 ∥ ) . -\ell'(\theta^{\parallel}) \ge m_{\ell}, \qquad m_{\ell} := \frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta_0^{\parallel}). − ℓ ′ ( θ ∥ ) ≥ m ℓ , m ℓ := N 1 i = 1 ∑ N a i q ( a i θ 0 ∥ ) .
Moreover, throughout the radius-r r r tubular neighborhood,
− ⟨ ∇ L ( θ ) , v ⟩ ≤ G ∥ , G ∥ : = 1 N ∑ i = 1 N a i . -\langle\nabla L(\theta),v\rangle\le G_{\parallel}, \qquad G_{\parallel} := \frac{1}{N}\sum_{i=1}^{N}a_i. − ⟨ ∇ L ( θ ) , v ⟩ ≤ G ∥ , G ∥ := N 1 i = 1 ∑ N a i .
(v) Exponential tail rate. The exponential-tail rate is
β t a i l = a min . \beta_{\mathrm{tail}} = a_{\min}. β tail = a m i n .
(vi) Exponential-tail drift constants. Choose the tail threshold so that
θ t a i l ∥ ( r ) ≥ u max r a min . \theta_{\mathrm{tail}}^{\parallel}(r) \ge\frac{u_{\max}r}{a_{\min}}. θ tail ∥ ( r ) ≥ a m i n u m a x r .
Then for all θ ∥ ≥ θ t a i l ∥ ( r ) \theta^{\parallel} \ge\theta_{\mathrm{tail}}^{\parallel}(r) θ ∥ ≥ θ tail ∥ ( r ) and ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\| \le r ∥ θ ⊥ ∥ ≤ r ,
A − ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) ≤ − ⟨ ∇ L ( θ ) , v ⟩ ≤ A + ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) , A_-(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))} \le-\langle\nabla L(\theta),v\rangle\le A_+(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))}, A − ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r )) ≤ − ⟨ ∇ L ( θ ) , v ⟩ ≤ A + ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r )) ,
where one may take
A − ( r ) : = ∣ I ∗ ∣ a min 2 N e − a min θ t a i l ∥ ( r ) , A + ( r ) : = cosh ( u max r ) N ( ∑ i = 1 N a i ) e − a min θ t a i l ∥ ( r ) . A_-(r) := \frac{|I_*|a_{\min}}{2N}e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}, \qquad A_+(r) := \frac{\cosh(u_{\max}r)}{N}\left(\sum_{i=1}^{N}a_i\right)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. A − ( r ) := 2 N ∣ I ∗ ∣ a m i n e − a m i n θ tail ∥ ( r ) , A + ( r ) := N cosh ( u m a x r ) ( i = 1 ∑ N a i ) e − a m i n θ tail ∥ ( r ) .
(vii) Exponential-tail sharpness constants. Under the same radius condition,
λ 1 , − ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) , \lambda_{1,-}(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))} \le\lambda_1(\theta^{\parallel}) \le\lambda_{1,+}(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))}, λ 1 , − ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r )) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r )) ,
where one may take
λ 1 , − ( r ) : = 1 4 λ max ( Σ ∗ ∣ U ) e − a min θ t a i l ∥ ( r ) , λ 1 , + ( r ) : = λ max ( Σ ∣ U ) e − a min θ t a i l ∥ ( r ) . \lambda_{1,-}(r) := \frac{1}{4}\lambda_{\max}(\Sigma_*|_U)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}, \qquad\lambda_{1,+}(r) := \lambda_{\max}(\Sigma|_U)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. λ 1 , − ( r ) := 4 1 λ m a x ( Σ ∗ ∣ U ) e − a m i n θ tail ∥ ( r ) , λ 1 , + ( r ) := λ m a x ( Σ ∣ U ) e − a m i n θ tail ∥ ( r ) .
(viii) Tail uniform conditioning. If the spanning condition
U = span { u i : i ∈ I ∗ } U = \operatorname{span}\{u_i : i \in I_\ast\} U = span { u i : i ∈ I ∗ }
holds, then H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) is uniformly conditioned on the tail:
H ( θ ∥ ) ⪰ ρ t a i l λ 1 ( θ ∥ ) I U ∀ θ ∥ ≥ θ t a i l ( r ) , H(\theta^{\parallel}) \succeq\rho_{\mathrm{tail}}\lambda_1(\theta^{\parallel})I_U \qquad\forall\theta^{\parallel} \ge\theta_{\mathrm{tail}}(r), H ( θ ∥ ) ⪰ ρ tail λ 1 ( θ ∥ ) I U ∀ θ ∥ ≥ θ tail ( r ) ,
with the explicit choice
ρ t a i l : = λ min ( Σ ∗ ∣ U ) λ max ( Σ ∣ U ) . \rho_{\mathrm{tail}} := \frac{\lambda_{\min}(\Sigma_\ast|_U)}{\lambda_{\max}(\Sigma|_U)}. ρ tail := λ m a x ( Σ ∣ U ) λ m i n ( Σ ∗ ∣ U ) .
In fact, this bound holds for all θ ∥ ≥ 0 \theta^{\parallel} \ge0 θ ∥ ≥ 0 .
Proof. We verify each claim in turn.
Step 1: Active-coordinate form of the loss. Write
θ = θ ∥ v + θ ⊥ + θ i n a c t , θ ⊥ ∈ U , θ i n a c t ∈ U ⊥ ∩ v ⊥ . \theta= \theta^{\parallel}v + \theta^{\perp} + \theta^{\mathrm{inact}}, \qquad\theta^{\perp} \in U,\qquad\theta^{\mathrm{inact}} \in U^{\perp} \cap v^{\perp}. θ = θ ∥ v + θ ⊥ + θ inact , θ ⊥ ∈ U , θ inact ∈ U ⊥ ∩ v ⊥ .
Since u i ∈ U u_i \in U u i ∈ U , we have
⟨ θ i n a c t , u i ⟩ = 0 , ⟨ θ i n a c t , v ⟩ = 0. \langle\theta^{\mathrm{inact}},u_i\rangle= 0,\qquad\langle\theta^{\mathrm{inact}},v\rangle= 0. ⟨ θ inact , u i ⟩ = 0 , ⟨ θ inact , v ⟩ = 0.
Therefore the loss depends only on ( θ ∥ , θ ⊥ ) (\theta^{\parallel},\theta^{\perp}) ( θ ∥ , θ ⊥ ) . Indeed,
⟨ θ , x i + ⟩ = ⟨ θ ∥ v + θ ⊥ , a i v + u i ⟩ = a i θ ∥ + ⟨ θ ⊥ , u i ⟩ , \langle\theta,x_i^+\rangle= \langle\theta^{\parallel}v+\theta^{\perp},a_i v+u_i\rangle= a_i\theta^{\parallel}+\langle\theta^{\perp},u_i\rangle, ⟨ θ , x i + ⟩ = ⟨ θ ∥ v + θ ⊥ , a i v + u i ⟩ = a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ,
and
⟨ θ , − x i − ⟩ = ⟨ θ ∥ v + θ ⊥ , a i v − u i ⟩ = a i θ ∥ − ⟨ θ ⊥ , u i ⟩ . \langle\theta,-x_i^-\rangle= \langle\theta^{\parallel}v+\theta^{\perp},a_i v-u_i\rangle= a_i\theta^{\parallel}-\langle\theta^{\perp},u_i\rangle. ⟨ θ , − x i − ⟩ = ⟨ θ ∥ v + θ ⊥ , a i v − u i ⟩ = a i θ ∥ − ⟨ θ ⊥ , u i ⟩ .
Thus
L ( θ ∥ , θ ⊥ ) = 1 2 N ∑ i = 1 N [ ϕ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + ϕ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] . (61) \mathcal{L}(\theta^{\parallel},\theta^{\perp}) = \frac{1}{2N}\sum_{i=1}^{N}\left[\phi(a_i\theta^{\parallel}+\langle\theta^{\perp},u_i\rangle)+\phi(a_i\theta^{\parallel}-\langle\theta^{\perp},u_i\rangle)\right]. \tag*{(61)} L ( θ ∥ , θ ⊥ ) = 2 N 1 i = 1 ∑ N [ ϕ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩) + ϕ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩) ] . (61)
In particular, the inactive component θ i n a c t \theta^{\mathrm{inact}} θ inact is frozen under GD, and all nontrivial dynamics occur in span { v } ⊕ U \operatorname{span}\{v\}\oplus U span { v } ⊕ U .
Step 2: Subspace Alignment. The expression (61) is even in θ ⊥ \theta^{\perp} θ ⊥ :
L ( θ ∥ , θ ⊥ ) = L ( θ ∥ , − θ ⊥ ) . \mathcal{L}(\theta^{\parallel},\theta^{\perp}) = \mathcal{L}(\theta^{\parallel},-\theta^{\perp}). L ( θ ∥ , θ ⊥ ) = L ( θ ∥ , − θ ⊥ ) .
Therefore its U U U -gradient vanishes at θ ⊥ = 0 \theta^{\perp}=0 θ ⊥ = 0 :
∇ L ⊥ ( θ ∥ v ) = 0. \nabla\mathcal{L}^{\perp}(\theta^{\parallel}v)=0. ∇ L ⊥ ( θ ∥ v ) = 0.
Moreover, the loss is independent of the inactive component θ i n a c t \theta^{\mathrm{inact}} θ inact , so the inactive gradient also vanishes. Hence
∇ L ( θ ∥ v ) ∈ span { v } = M . \nabla\mathcal{L}(\theta^{\parallel}v)\in\operatorname{span}\{v\}=\mathcal{M}. ∇ L ( θ ∥ v ) ∈ span { v } = M .
It remains to verify the mixed Hessian alignment condition. Since ∇ L ⊥ ( θ ∥ v ) = 0 \nabla\mathcal{L}^{\perp}(\theta^{\parallel}v)=0 ∇ L ⊥ ( θ ∥ v ) = 0 for every θ ∥ \theta^{\parallel} θ ∥ , differentiating this identity with respect to θ ∥ \theta^{\parallel} θ ∥ gives
0 = d d θ ∥ Π U ∇ L ( θ ∥ v ) = Π U ∇ 2 L ( θ ∥ v ) v . 0=\frac{d}{d\theta^{\parallel}}\Pi_U\nabla\mathcal{L}(\theta^{\parallel}v)=\Pi_U\nabla^2\mathcal{L}(\theta^{\parallel}v)v. 0 = d θ ∥ d Π U ∇ L ( θ ∥ v ) = Π U ∇ 2 L ( θ ∥ v ) v .
Equivalently, using the full Hessian computation in Proposition 2, along the parallel direction
∇ 2 L ( θ ∥ v ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) ( a i 2 v v T + u i u i T ) . \nabla^2\mathcal{L}(\theta^{\parallel}v)=\frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta^{\parallel})\left(a_i^2vv^{\mathsf T}+u_i u_i^{\mathsf T}\right). ∇ 2 L ( θ ∥ v ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) ( a i 2 v v T + u i u i T ) .
Therefore
Π U ∇ 2 L ( θ ∥ v ) v = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) ( a i 2 Π U v v T v + Π U u i u i T v ) = 0 , \Pi_U\nabla^2\mathcal{L}(\theta^{\parallel}v)v=\frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta^{\parallel})\left(a_i^2\Pi_Uvv^{\mathsf T}v+\Pi_Uu_i u_i^{\mathsf T}v\right)=0, Π U ∇ 2 L ( θ ∥ v ) v = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) ( a i 2 Π U v v T v + Π U u i u i T v ) = 0 ,
because Π U v = 0 \Pi_Uv=0 Π U v = 0 and ⟨ u i , v ⟩ = 0 \langle u_i,v\rangle=0 ⟨ u i , v ⟩ = 0 . This verifies both parts of Subspace Alignment.
Step 3: Restricted loss and sharpness matrix. Setting θ ⊥ = 0 \theta^\perp=0 θ ⊥ = 0 in (62) gives
ℓ ( θ ∥ ) = L ( θ ∥ ∣ V ) = 1 2 N ∑ i = 1 N [ ϕ ( a i θ ∥ ) + ϕ ( a i θ ∥ ) ] = 1 N ∑ i = 1 N ϕ ( a i θ ∥ ) . \ell(\theta^\parallel)=\mathcal{L}(\theta^\parallel|_V)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi(a_i\theta^\parallel)+\phi(a_i\theta^\parallel)\right]=\frac{1}{N}\sum_{i=1}^{N}\phi(a_i\theta^\parallel). ℓ ( θ ∥ ) = L ( θ ∥ ∣ V ) = 2 N 1 i = 1 ∑ N [ ϕ ( a i θ ∥ ) + ϕ ( a i θ ∥ ) ] = N 1 i = 1 ∑ N ϕ ( a i θ ∥ ) .
Since
ϕ ′ ( s ) = − 1 1 + e s = − q ( s ) , \phi'(s)=-\frac{1}{1+e^s}=-q(s), ϕ ′ ( s ) = − 1 + e s 1 = − q ( s ) ,
we get
ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i ϕ ′ ( a i θ ∥ ) = − 1 N ∑ i = 1 N a i q ( a i θ ∥ ) . \ell'(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}a_i\phi'(a_i\theta^\parallel)=-\frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta^\parallel). ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i ϕ ′ ( a i θ ∥ ) = − N 1 i = 1 ∑ N a i q ( a i θ ∥ ) .
Hence
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i q ( a i θ ∥ ) . -\ell'(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta^\parallel). − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q ( a i θ ∥ ) .
Next, differentiate the restricted-normal gradient. From (62), the U U U -gradient is
∇ L ⊥ ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ′ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) − ϕ ′ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] u i . \nabla\mathcal{L}^{\perp}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi'(a_i\theta^\parallel+\langle\theta^\perp,u_i\rangle)-\phi'(a_i\theta^\parallel-\langle\theta^\perp,u_i\rangle)\right]u_i. ∇ L ⊥ ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ′ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩) − ϕ ′ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩) ] u i .
Differentiating once more in θ ⊥ \theta^\perp θ ⊥ gives
D θ ⊥ ∇ L ⊥ ( θ ) = 1 2 N ∑ i = 1 N [ ϕ ′ ′ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + ϕ ′ ′ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] u i u i T . D_{\theta^\perp}\nabla\mathcal{L}^{\perp}(\theta)=\frac{1}{2N}\sum_{i=1}^{N}\left[\phi''(a_i\theta^\parallel+\langle\theta^\perp,u_i\rangle)+\phi''(a_i\theta^\parallel-\langle\theta^\perp,u_i\rangle)\right]u_i u_i^{\mathsf{T}}. D θ ⊥ ∇ L ⊥ ( θ ) = 2 N 1 i = 1 ∑ N [ ϕ ′′ ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩) + ϕ ′′ ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩) ] u i u i T .
At θ ⊥ = 0 \theta^\perp=0 θ ⊥ = 0 , this becomes
H ( θ ∥ ) = D θ ⊥ ∇ L ⊥ ( θ ∥ ∣ V ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) u i u i T . H(\theta^\parallel)=D_{\theta^\perp}\nabla\mathcal{L}^{\perp}(\theta^\parallel|_V)=\frac{1}{N}\sum_{i=1}^{N}\phi''(a_i\theta^\parallel)u_i u_i^{\mathsf{T}}. H ( θ ∥ ) = D θ ⊥ ∇ L ⊥ ( θ ∥ ∣ V ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) u i u i T .
Step 4: cosh bounds for the RC Bound. We now prove the cosh-type relative curvature bound. The key point is that the two terms with orthogonal shifts + s +s + s and − s -s − s are averaged.
Recall that
ϕ ′ ′ ( z ) = e z ( 1 + e z ) 2 = 1 2 + e z + e − z = 1 2 ( 1 + cosh z ) . \phi''(z)=\frac{e^z}{(1+e^z)^2}=\frac{1}{2+e^z+e^{-z}}=\frac{1}{2(1+\cosh z)}. ϕ ′′ ( z ) = ( 1 + e z ) 2 e z = 2 + e z + e − z 1 = 2 ( 1 + cosh z ) 1 .
Fix z ∈ R z\in\mathbb{R} z ∈ R and s ∈ R s\in\mathbb{R} s ∈ R , and define
R ( z , s ) : = ϕ ′ ′ ( z + s ) + ϕ ′ ′ ( z − s ) 2 ϕ ′ ′ ( z ) . R(z,s):=\frac{\phi''(z+s)+\phi''(z-s)}{2\phi''(z)}. R ( z , s ) := 2 ϕ ′′ ( z ) ϕ ′′ ( z + s ) + ϕ ′′ ( z − s ) .
We claim that
2 − cosh ( s ) ≤ R ( z , s ) ≤ cosh ( s ) . (62) 2-\cosh(s)\le R(z,s)\le\cosh(s). \tag*{(62)} 2 − cosh ( s ) ≤ R ( z , s ) ≤ cosh ( s ) . (62)
The upper bound is the main useful part. It follows from the elementary identity
cosh ( z + s ) + cosh ( z − s ) = 2 cosh z cosh s . \cosh(z+s)+\cosh(z-s)=2\cosh z\cosh s. cosh ( z + s ) + cosh ( z − s ) = 2 cosh z cosh s .
Since
ϕ ′ ′ ( w ) = 1 2 ( 1 + cosh w ) , \phi''(w)=\frac{1}{2(1+\cosh w)}, ϕ ′′ ( w ) = 2 ( 1 + cosh w ) 1 ,
we have
ϕ ′ ′ ( z + s ) ϕ ′ ′ ( z ) = 1 + cosh z 1 + cosh ( z + s ) \frac{\phi''(z+s)}{\phi''(z)}=\frac{1+\cosh z}{1+\cosh(z+s)} ϕ ′′ ( z ) ϕ ′′ ( z + s ) = 1 + cosh ( z + s ) 1 + cosh z
and similarly for z − s z-s z − s . Hence
R ( z , s ) = 1 + cosh z 2 ( 1 1 + cosh ( z + s ) + 1 1 + cosh ( z − s ) ) . R(z,s)=\frac{1+\cosh z}{2}\left(\frac{1}{1+\cosh(z+s)}+\frac{1}{1+\cosh(z-s)}\right). R ( z , s ) = 2 1 + cosh z ( 1 + cosh ( z + s ) 1 + 1 + cosh ( z − s ) 1 ) .
A direct simplification gives
R ( z , s ) = ( 1 + cosh z ) ( 1 + cosh z cosh s ) ( 1 + cosh z cosh s ) 2 − sinh 2 z sinh 2 s . R(z,s)=\frac{(1+\cosh z)(1+\cosh z\cosh s)}{(1+\cosh z\cosh s)^2-\sinh^2 z\sinh^2 s}. R ( z , s ) = ( 1 + cosh z cosh s ) 2 − sinh 2 z sinh 2 s ( 1 + cosh z ) ( 1 + cosh z cosh s ) .
The denominator can be rewritten as
( 1 + cosh z cosh s ) 2 − sinh 2 z sinh 2 s = ( cosh z + cosh s ) 2 . (1+\cosh z\cosh s)^2-\sinh^2 z\sinh^2 s=(\cosh z+\cosh s)^2. ( 1 + cosh z cosh s ) 2 − sinh 2 z sinh 2 s = ( cosh z + cosh s ) 2 .
Therefore
R ( z , s ) = ( 1 + cosh z ) ( 1 + cosh z cosh s ) ( cosh z + cosh s ) 2 . (63) R(z,s)=\frac{(1+\cosh z)(1+\cosh z\cosh s)}{(\cosh z+\cosh s)^2}. \tag*{(63)} R ( z , s ) = ( cosh z + cosh s ) 2 ( 1 + cosh z ) ( 1 + cosh z cosh s ) . (63)
Let c : = cosh z ≥ 1 c:=\cosh z\ge1 c := cosh z ≥ 1 and d : = cosh s ≥ 1 d:=\cosh s\ge1 d := cosh s ≥ 1 . Then
R ( z , s ) = ( 1 + c ) ( 1 + c d ) ( c + d ) 2 . R(z,s)=\frac{(1+c)(1+cd)}{(c+d)^2}. R ( z , s ) = ( c + d ) 2 ( 1 + c ) ( 1 + c d ) .
We prove the upper bound R ( z , s ) ≤ d R(z,s)\le d R ( z , s ) ≤ d . This is equivalent to
( 1 + c ) ( 1 + c d ) ≤ d ( c + d ) 2 . (1+c)(1+cd)\le d(c+d)^2. ( 1 + c ) ( 1 + c d ) ≤ d ( c + d ) 2 .
Expanding and subtracting the left-hand side from the right-hand side,
d ( c + d ) 2 − ( 1 + c ) ( 1 + c d ) = d c 2 + 2 c d 2 + d 3 − 1 − c − c d − c 2 d = 2 c d 2 + d 3 − 1 − c − c d = ( d − 1 ) ( d 2 + d + 1 + c ( 2 d + 1 ) ) ≥ 0. \begin{aligned}
d(c+d)^2-(1+c)(1+cd)&=dc^2+2cd^2+d^3-1-c-cd-c^2d\\
&=2cd^2+d^3-1-c-cd\\
&=(d-1)(d^2+d+1+c(2d+1))\ge0.
\end{aligned} d ( c + d ) 2 − ( 1 + c ) ( 1 + c d ) = d c 2 + 2 c d 2 + d 3 − 1 − c − c d − c 2 d = 2 c d 2 + d 3 − 1 − c − c d = ( d − 1 ) ( d 2 + d + 1 + c ( 2 d + 1 )) ≥ 0.
Thus R ( z , s ) ≤ cosh s R(z,s)\le\cosh s R ( z , s ) ≤ cosh s .
For the lower bound, the exact formula (64) shows that R ( z , s ) > 0 R(z,s)>0 R ( z , s ) > 0 . If cosh s ≥ 2 \cosh s\ge2 cosh s ≥ 2 , then 2 − cosh s ≤ 0 2-\cosh s\le0 2 − cosh s ≤ 0 , and the lower bound is immediate. If 1 ≤ cosh s < 2 1\le\cosh s<2 1 ≤ cosh s < 2 , one checks from the same one-variable formula
R = ( 1 + c ) ( 1 + c d ) ( c + d ) 2 , c ≥ 1 , d ∈ [ 1 , 2 ) , R=\frac{(1+c)(1+cd)}{(c+d)^2},\qquad c\ge1,\quad d\in[1,2), R = ( c + d ) 2 ( 1 + c ) ( 1 + c d ) , c ≥ 1 , d ∈ [ 1 , 2 ) ,
that
Indeed,
( 1 + c ) ( 1 + c d ) − ( 2 − d ) ( c + d ) 2 = ( d − 1 ) ( 2 c 2 + ( 2 d − 1 ) c + d 2 − d − 1 ) ≥ 0. (64) (1+c)(1+cd)-(2-d)(c+d)^2=(d-1)(2c^2+(2d-1)c+d^2-d-1)\ge0. \tag*{(64)} ( 1 + c ) ( 1 + c d ) − ( 2 − d ) ( c + d ) 2 = ( d − 1 ) ( 2 c 2 + ( 2 d − 1 ) c + d 2 − d − 1 ) ≥ 0. (64)
The bracket is nonnegative for c ≥ 1 c\ge1 c ≥ 1 and d ≥ 1 d\ge1 d ≥ 1 . Indeed,
2 c 2 + ( 2 d − 1 ) c + d 2 − d − 1 2c^2+(2d-1)c+d^2-d-1 2 c 2 + ( 2 d − 1 ) c + d 2 − d − 1
is increasing in c c c on c ≥ 1 c\ge1 c ≥ 1 , since its derivative with respect to c c c is 4 c + 2 d − 1 > 0 4c+2d-1>0 4 c + 2 d − 1 > 0 . Therefore its minimum over c ≥ 1 c\ge1 c ≥ 1 occurs at c = 1 c=1 c = 1 , where 2 + ( 2 d − 1 ) + d 2 − d − 1 = d 2 + d ≥ 0 2+(2d-1)+d^2-d-1=d^2+d\ge0 2 + ( 2 d − 1 ) + d 2 − d − 1 = d 2 + d ≥ 0 .
Therefore (63) holds.
Step 5: Matrix RC bound. For each i i i , set
z i : = a i θ ∥ , s i : = ⟨ θ ⊥ , u i ⟩ . z_i:=a_i\theta^\parallel,\qquad s_i:=\langle\theta^\perp,u_i\rangle. z i := a i θ ∥ , s i := ⟨ θ ⊥ , u i ⟩ .
If ∥ θ ⊥ ∥ ≤ r \lVert\theta^\perp\rVert\le r ∥ θ ⊥ ∥ ≤ r , then
∣ s i ∣ ≤ ∥ θ ⊥ ∥ ∥ u i ∥ ≤ u max r . |s_i|\le\lVert\theta^\perp\rVert\lVert u_i\rVert\le u_{\max}r. ∣ s i ∣ ≤ ∥ θ ⊥ ∥ ∥ u i ∥ ≤ u m a x r .
Since s ↦ cosh s s\mapsto\cosh s s ↦ cosh s is even and increasing for $s\ge 0,
cosh ( s i ) ≤ cosh ( u max r ) . \cosh(s_i)\le\cosh(u_{\max}r). cosh ( s i ) ≤ cosh ( u m a x r ) .
Applying (63) with z = z i z=z_i z = z i and s = s i s=s_i s = s i , we get
2 − cosh ( u max r ) ≤ ϕ ′ ′ ( z i + s i ) + ϕ ′ ′ ( z i − s i ) 2 ϕ ′ ′ ( z i ) ≤ cosh ( u max r ) . 2-\cosh(u_{\max}r)\le\frac{\phi''(z_i+s_i)+\phi''(z_i-s_i)}{2\phi''(z_i)}\le\cosh(u_{\max}r). 2 − cosh ( u m a x r ) ≤ 2 ϕ ′′ ( z i ) ϕ ′′ ( z i + s i ) + ϕ ′′ ( z i − s i ) ≤ cosh ( u m a x r ) .
Equivalently,
( 1 − δ r ) ϕ ′ ′ ( z i ) ≤ 1 2 [ ϕ ′ ′ ( z i + s i ) + ϕ ′ ′ ( z i − s i ) ] ≤ ( 1 + δ r ) ϕ ′ ′ ( z i ) . (1-\delta_r)\phi^{\prime\prime}(z_i) \le\frac{1}{2}\left[\phi^{\prime\prime}(z_i+s_i)+\phi^{\prime\prime}(z_i-s_i)\right] \le(1+\delta_r)\phi^{\prime\prime}(z_i). ( 1 − δ r ) ϕ ′′ ( z i ) ≤ 2 1 [ ϕ ′′ ( z i + s i ) + ϕ ′′ ( z i − s i ) ] ≤ ( 1 + δ r ) ϕ ′′ ( z i ) .
where
δ r : = cosh ( u max r ) − 1. \delta_r := \cosh(u_{\max}r)-1. δ r := cosh ( u m a x r ) − 1.
Multiplying by N − 1 u i u i T N^{-1}u_i u_i^{\mathsf{T}} N − 1 u i u i T and summing over i i i , we obtain
( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) . (1-\delta_r)H(\theta^{\parallel}) \preceq D_{\theta^\perp}\nabla\mathcal{L}^{\perp}(\theta) \preceq(1+\delta_r)H(\theta^{\parallel}). ( 1 − δ r ) H ( θ ∥ ) ⪯ D θ ⊥ ∇ L ⊥ ( θ ) ⪯ ( 1 + δ r ) H ( θ ∥ ) .
This proves the RC bound.
Finally, the Taylor expansion of cosh gives
δ r = cosh ( u max r ) − 1 = 1 2 ( u max r ) 2 + O ( r 4 ) . \delta_r=\cosh(u_{\max}r)-1=\frac{1}{2}(u_{\max}r)^2+O(r^4). δ r = cosh ( u m a x r ) − 1 = 2 1 ( u m a x r ) 2 + O ( r 4 ) .
Step 6: Local Curvature Decay. Differentiate
H ( θ ∥ ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) u i u i T H(\theta^{\parallel})=\frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta^{\parallel})u_i u_i^{\mathsf{T}} H ( θ ∥ ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) u i u i T
with respect to θ ∥ \theta^{\parallel} θ ∥ . This gives
H ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i ϕ ′ ′ ′ ( a i θ ∥ ) u i u i T . H^{\prime}(\theta^{\parallel})=\frac{1}{N}\sum_{i=1}^{N}a_i\phi^{\prime\prime\prime}(a_i\theta^{\parallel})u_i u_i^{\mathsf{T}}. H ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i ϕ ′′′ ( a i θ ∥ ) u i u i T .
For the logistic loss,
− ϕ ′ ′ ′ ( s ) = ϕ ′ ′ ( s ) tanh ( s / 2 ) . -\phi^{\prime\prime\prime}(s)=\phi^{\prime\prime}(s)\tanh(s/2). − ϕ ′′′ ( s ) = ϕ ′′ ( s ) tanh ( s /2 ) .
Thus
− H ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i ϕ ′ ′ ( a i θ ∥ ) tanh ( a i θ ∥ / 2 ) u i u i T . -H^{\prime}(\theta^{\parallel})=\frac{1}{N}\sum_{i=1}^{N}a_i\phi^{\prime\prime}(a_i\theta^{\parallel})\tanh(a_i\theta^{\parallel}/2)u_i u_i^{\mathsf{T}}. − H ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i ϕ ′′ ( a i θ ∥ ) tanh ( a i θ ∥ /2 ) u i u i T .
Now suppose θ ∥ ∈ J = [ θ 0 ∥ , θ 1 ∥ ] ⊆ [ 0 , ∞ ) \theta^{\parallel}\in J=[\theta_0^{\parallel},\theta_1^{\parallel}]\subseteq[0,\infty) θ ∥ ∈ J = [ θ 0 ∥ , θ 1 ∥ ] ⊆ [ 0 , ∞ ) . Since a i ≥ a min a_i\ge a_{\min} a i ≥ a m i n and θ ∥ ≥ θ 0 ∥ \theta^{\parallel}\ge\theta_0^{\parallel} θ ∥ ≥ θ 0 ∥ ,
a i tanh ( a i θ ∥ / 2 ) ≥ a min tanh ( a min θ 0 ∥ / 2 ) = β J . a_i\tanh(a_i\theta^{\parallel}/2)\ge a_{\min}\tanh(a_{\min}\theta_0^{\parallel}/2)=\beta_J. a i tanh ( a i θ ∥ /2 ) ≥ a m i n tanh ( a m i n θ 0 ∥ /2 ) = β J .
Therefore, term by term,
a i ϕ ′ ′ ( a i θ ∥ ) tanh ( a i θ ∥ / 2 ) u i u i T ⪰ β J ϕ ′ ′ ( a i θ ∥ ) u i u i T . a_i\phi^{\prime\prime}(a_i\theta^{\parallel})\tanh(a_i\theta^{\parallel}/2)u_i u_i^{\mathsf{T}}\succeq\beta_J\phi^{\prime\prime}(a_i\theta^{\parallel})u_i u_i^{\mathsf{T}}. a i ϕ ′′ ( a i θ ∥ ) tanh ( a i θ ∥ /2 ) u i u i T ⪰ β J ϕ ′′ ( a i θ ∥ ) u i u i T .
Summing gives
− H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) . -H^{\prime}(\theta^{\parallel})\succeq\beta_JH(\theta^{\parallel}). − H ′ ( θ ∥ ) ⪰ β J H ( θ ∥ ) .
This proves the stated monotone-drift sharpness decay condition.
Step 7: Unstable drift constants. Along the parallel direction,
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a i q ( a i θ ∥ ) . -\ell^{\prime}(\theta^{\parallel})=\frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta^{\parallel}). − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q ( a i θ ∥ ) .
The function q ( s ) = ( 1 + e s ) − 1 q(s)=(1+e^s)^{-1} q ( s ) = ( 1 + e s ) − 1 is positive and decreasing in s s s . Therefore, for θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] \theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{\parallel}] θ ∥ ∈ [ θ 0 ∥ , θ ∗ ∥ ] ,
q ( a i θ ∥ ) ≥ q ( a i θ ∗ ∥ ) , q(a_i\theta^{\parallel})\ge q(a_i\theta_*^{\parallel}), q ( a i θ ∥ ) ≥ q ( a i θ ∗ ∥ ) ,
and hence
− ℓ ′ ( θ ∥ ) ≥ 1 N ∑ i = 1 N a i q ( a i θ ∗ ∥ ) = m ℓ . -\ell^{\prime}(\theta^{\parallel})\ge\frac{1}{N}\sum_{i=1}^{N}a_iq(a_i\theta_*^{\parallel})=m_{\ell}. − ℓ ′ ( θ ∥ ) ≥ N 1 i = 1 ∑ N a i q ( a i θ ∗ ∥ ) = m ℓ .
For the upper drift bound in the tubular neighborhood, use
− ⟨ ∇ L ∥ ( θ ) , v ⟩ = 1 2 N ∑ i = 1 N a i [ q ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + q ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] . -\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle=\frac{1}{2N}\sum_{i=1}^{N}a_i\left[q(a_i\theta^{\parallel}+\langle\theta^\perp,u_i\rangle)+q(a_i\theta^{\parallel}-\langle\theta^\perp,u_i\rangle)\right]. − ⟨ ∇ L ∥ ( θ ) , v ⟩ = 2 N 1 i = 1 ∑ N a i [ q ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩) + q ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩) ] .
Since 0 < q ( s ) ≤ 1 0<q(s)\leq1 0 < q ( s ) ≤ 1 for all s s s , we get
− ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ 1 2 N ∑ i = 1 N a i ( 1 + 1 ) = 1 N ∑ i = 1 N a i = G ∥ . -\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle\leq\frac{1}{2N}\sum_{i=1}^{N}a_i(1+1)=\frac{1}{N}\sum_{i=1}^{N}a_i=G_{\parallel}. − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ 2 N 1 i = 1 ∑ N a i ( 1 + 1 ) = N 1 i = 1 ∑ N a i = G ∥ .
Step 8: Exponential-tail drift constants. Assume
a min θ t a i l ∥ ( r ) ≥ u max r . a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)\geq u_{\max}r. a m i n θ tail ∥ ( r ) ≥ u m a x r .
Then for every θ ∥ ≥ θ t a i l ∥ ( r ) \theta^{\parallel}\geq\theta_{\mathrm{tail}}^{\parallel}(r) θ ∥ ≥ θ tail ∥ ( r ) and ∥ θ ⊥ ∥ ≤ r \|\theta^{\perp}\|\leq r ∥ θ ⊥ ∥ ≤ r ,
a i θ ∥ ± ⟨ θ ⊥ , u i ⟩ ≥ a min θ t a i l ∥ ( r ) − u max r ≥ 0. a_i\theta^{\parallel}\pm\langle\theta^{\perp},u_i\rangle\geq a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)-u_{\max}r\geq0. a i θ ∥ ± ⟨ θ ⊥ , u i ⟩ ≥ a m i n θ tail ∥ ( r ) − u m a x r ≥ 0.
Thus all margins appearing in the tail tubular neighborhood are nonnegative.
For s ≥ 0 s\geq0 s ≥ 0 , the logistic tail satisfies
1 2 e − s ≤ q ( s ) ≤ e − s . \frac{1}{2}e^{-s}\leq q(s)\leq e^{-s}. 2 1 e − s ≤ q ( s ) ≤ e − s .
Using the exact formula for the parallel drift,
− ⟨ ∇ L ∥ ( θ ) , v ⟩ = 1 2 N ∑ i = 1 N a i [ q ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + q ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] . -\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle
=\frac{1}{2N}\sum_{i=1}^{N}a_i\left[q\left(a_i\theta^{\parallel}+\langle\theta^{\perp},u_i\rangle\right)+q\left(a_i\theta^{\parallel}-\langle\theta^{\perp},u_i\rangle\right)\right]. − ⟨ ∇ L ∥ ( θ ) , v ⟩ = 2 N 1 i = 1 ∑ N a i [ q ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + q ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] .
For the lower bound, restrict the sum to the slowest layer I ∗ I_* I ∗ . For i ∈ I ∗ i\in I_* i ∈ I ∗ , a i = a min a_i=a_{\min} a i = a m i n . Hence
− ⟨ ∇ L ∥ ( θ ) , v ⟩ ≥ 1 2 N ∑ i ∈ I ∗ a min [ 1 2 e − ( a min θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + 1 2 e − ( a min θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] = a min 2 N ∑ i ∈ I ∗ e − a min θ ∥ cosh ( ⟨ θ ⊥ , u i ⟩ ) ≥ ∣ I ∗ ∣ a min 2 N e − a min θ ∥ . \begin{aligned}
-\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle
&\geq\frac{1}{2N}\sum_{i\in I_*}a_{\min}\left[\frac{1}{2}e^{-\left(a_{\min}\theta^{\parallel}+\langle\theta^{\perp},u_i\rangle\right)}+\frac{1}{2}e^{-\left(a_{\min}\theta^{\parallel}-\langle\theta^{\perp},u_i\rangle\right)}\right]\\
&=\frac{a_{\min}}{2N}\sum_{i\in I_*}e^{-a_{\min}\theta^{\parallel}}\cosh\left(\langle\theta^{\perp},u_i\rangle\right)\\
&\geq\frac{|I_*|a_{\min}}{2N}e^{-a_{\min}\theta^{\parallel}}.
\end{aligned} − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≥ 2 N 1 i ∈ I ∗ ∑ a m i n [ 2 1 e − ( a m i n θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + 2 1 e − ( a m i n θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] = 2 N a m i n i ∈ I ∗ ∑ e − a m i n θ ∥ cosh ( ⟨ θ ⊥ , u i ⟩ ) ≥ 2 N ∣ I ∗ ∣ a m i n e − a m i n θ ∥ .
Since
e − a min θ ∥ = e − a min θ t a i l ∥ ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) , e^{-a_{\min}\theta^{\parallel}}=e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}e^{-a_{\min}\left(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r)\right)}, e − a m i n θ ∥ = e − a m i n θ tail ∥ ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r ) ) ,
we may take
A − ( r ) = ∣ I ∗ ∣ a min 2 N e − a min θ t a i l ∥ ( r ) . A_-(r)=\frac{|I_*|a_{\min}}{2N}e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. A − ( r ) = 2 N ∣ I ∗ ∣ a m i n e − a m i n θ tail ∥ ( r ) .
For the upper bound, use q ( s ) ≤ e − s q(s)\leq e^{-s} q ( s ) ≤ e − s . Then
− ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ 1 2 N ∑ i = 1 N a i [ e − ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + e − ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] = 1 N ∑ i = 1 N a i e − a i θ ∥ cosh ( ⟨ θ ⊥ , u i ⟩ ) . \begin{aligned}
-\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle
\leq\frac{1}{2N}\sum_{i=1}^{N}a_i\left[e^{-\left(a_i\theta^{\parallel}+\langle\theta^{\perp},u_i\rangle\right)}+e^{-\left(a_i\theta^{\parallel}-\langle\theta^{\perp},u_i\rangle\right)}\right] \\
&=\frac{1}{N}\sum_{i=1}^{N}a_i e^{-a_i\theta^{\parallel}}\cosh\left(\langle\theta^{\perp},u_i\rangle\right).
\end{aligned} − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ 2 N 1 i = 1 ∑ N a i [ e − ( a i θ ∥ + ⟨ θ ⊥ , u i ⟩ ) + e − ( a i θ ∥ − ⟨ θ ⊥ , u i ⟩ ) ] = N 1 i = 1 ∑ N a i e − a i θ ∥ cosh ( ⟨ θ ⊥ , u i ⟩ ) .
Since a i ≥ a min a_i\geq a_{\min} a i ≥ a m i n , θ ∥ ≥ 0 \theta^{\parallel}\geq0 θ ∥ ≥ 0 , and
∣ ⟨ θ ⊥ , u i ⟩ ∣ ≤ u max r , \left|\langle\theta^{\perp},u_i\rangle\right|\leq u_{\max}r, ⟨ θ ⊥ , u i ⟩ ≤ u m a x r ,
we have
e − a i θ ∥ ≤ e − a min θ ∥ , cosh ( ⟨ θ ⊥ , u i ⟩ ) ≤ cosh ( u max r ) . e^{-a_i\theta^{\parallel}}\leq e^{-a_{\min}\theta^{\parallel}},\qquad
\cosh\left(\langle\theta^{\perp},u_i\rangle\right)\leq\cosh(u_{\max}r). e − a i θ ∥ ≤ e − a m i n θ ∥ , cosh ( ⟨ θ ⊥ , u i ⟩ ) ≤ cosh ( u m a x r ) .
Therefore
− ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ cosh ( u max r ) N ( ∑ i = 1 N a i ) e − a min θ ∥ . -\langle\nabla\mathcal{L}^{\parallel}(\theta),v\rangle
\leq\frac{\cosh(u_{\max}r)}{N}\left(\sum_{i=1}^{N}a_i\right)e^{-a_{\min}\theta^{\parallel}}. − ⟨ ∇ L ∥ ( θ ) , v ⟩ ≤ N cosh ( u m a x r ) ( i = 1 ∑ N a i ) e − a m i n θ ∥ .
Thus
A + ( r ) = cosh ( u max r ) N ( ∑ i = 1 N a i ) e − a min θ t a i l ∥ ( r ) . A_+(r)=\frac{\cosh(u_{\max}r)}{N}\left(\sum_{i=1}^{N}a_i\right)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. A + ( r ) = N cosh ( u m a x r ) ( i = 1 ∑ N a i ) e − a m i n θ tail ∥ ( r ) .
Step 9: Exponential-tail sharpness constants. For s ≥ 0 s \ge0 s ≥ 0 ,
ϕ ′ ′ ( s ) = e s ( 1 + e s ) 2 = e − s ( 1 + e − s ) 2 . \phi^{\prime\prime}(s) = \frac{e^s}{(1+e^s)^2} = \frac{e^{-s}}{(1+e^{-s})^2}. ϕ ′′ ( s ) = ( 1 + e s ) 2 e s = ( 1 + e − s ) 2 e − s .
Since 0 < e − s ≤ 1 0 < e^{-s} \le1 0 < e − s ≤ 1 , we have
1 4 e − s ≤ ϕ ′ ′ ( s ) ≤ e − s . \frac{1}{4}e^{-s} \le\phi^{\prime\prime}(s) \le e^{-s}. 4 1 e − s ≤ ϕ ′′ ( s ) ≤ e − s .
Now for θ ∥ ≥ θ t a i l ∥ ( r ) \theta^{\parallel} \ge\theta_{\mathrm{tail}}^{\parallel}(r) θ ∥ ≥ θ tail ∥ ( r ) ,
H ( θ ∥ ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ∥ ) u i u i T . H(\theta^{\parallel}) = \frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta^{\parallel})u_i u_i^{\mathsf{T}}. H ( θ ∥ ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ∥ ) u i u i T .
For the lower bound on λ 1 ( θ ∥ ) \lambda_1(\theta^{\parallel}) λ 1 ( θ ∥ ) , restrict to the slowest layer:
H ( θ ∥ ) ⪰ 1 N ∑ i ∈ I ∗ ϕ ′ ′ ( a min θ ∥ ) u i u i T H(\theta^{\parallel}) \succeq\frac{1}{N}\sum_{i\in I_*}\phi^{\prime\prime}(a_{\min}\theta^{\parallel})u_i u_i^{\mathsf{T}} H ( θ ∥ ) ⪰ N 1 i ∈ I ∗ ∑ ϕ ′′ ( a m i n θ ∥ ) u i u i T
⪰ 1 4 e − a min θ ∥ 1 N ∑ i ∈ I ∗ u i u i T = 1 4 e − a min θ ∥ Σ ∗ . \begin{aligned}
\succeq\frac{1}{4}e^{-a_{\min}\theta^{\parallel}}\frac{1}{N}\sum_{i\in I_*}u_i u_i^{\mathsf{T}} \\
&= \frac{1}{4}e^{-a_{\min}\theta^{\parallel}}\Sigma_*.
\end{aligned} ⪰ 4 1 e − a m i n θ ∥ N 1 i ∈ I ∗ ∑ u i u i T = 4 1 e − a m i n θ ∥ Σ ∗ .
Taking the largest eigenvalue on U U U gives
λ 1 ( θ ∥ ) ≥ 1 4 e − a min θ ∥ λ max ( Σ ∗ ∣ U ) . \lambda_1(\theta^{\parallel}) \ge\frac{1}{4}e^{-a_{\min}\theta^{\parallel}}\lambda_{\max}(\Sigma_*|_U). λ 1 ( θ ∥ ) ≥ 4 1 e − a m i n θ ∥ λ m a x ( Σ ∗ ∣ U ) .
Equivalently,
λ 1 ( θ ∥ ) ≥ λ 1 , − ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) \lambda_1(\theta^{\parallel}) \ge\lambda_{1,-}(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))} λ 1 ( θ ∥ ) ≥ λ 1 , − ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r ))
with
λ 1 , − ( r ) = 1 4 λ max ( Σ ∗ ∣ U ) e − a min θ t a i l ∥ ( r ) . \lambda_{1,-}(r) = \frac{1}{4}\lambda_{\max}(\Sigma_*|_U)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. λ 1 , − ( r ) = 4 1 λ m a x ( Σ ∗ ∣ U ) e − a m i n θ tail ∥ ( r ) .
For the upper bound,
ϕ ′ ′ ( a i θ ∥ ) ≤ e − a i θ ∥ ≤ e − a min θ ∥ . \phi^{\prime\prime}(a_i\theta^{\parallel}) \le e^{-a_i\theta^{\parallel}} \le e^{-a_{\min}\theta^{\parallel}}. ϕ ′′ ( a i θ ∥ ) ≤ e − a i θ ∥ ≤ e − a m i n θ ∥ .
Therefore
H ( θ ∥ ) ⪯ e − a min θ ∥ 1 N ∑ i = 1 N u i u i T = e − a min θ ∥ Σ . H(\theta^{\parallel}) \preceq e^{-a_{\min}\theta^{\parallel}}\frac{1}{N}\sum_{i=1}^{N}u_i u_i^{\mathsf{T}} = e^{-a_{\min}\theta^{\parallel}}\Sigma. H ( θ ∥ ) ⪯ e − a m i n θ ∥ N 1 i = 1 ∑ N u i u i T = e − a m i n θ ∥ Σ.
Taking the largest eigenvalue gives
λ 1 ( θ ∥ ) ≤ e − a min θ ∥ λ max ( Σ ∣ U ) . \lambda_1(\theta^{\parallel}) \le e^{-a_{\min}\theta^{\parallel}}\lambda_{\max}(\Sigma|_U). λ 1 ( θ ∥ ) ≤ e − a m i n θ ∥ λ m a x ( Σ ∣ U ) .
Equivalently,
λ 1 ( θ ∥ ) ≤ λ 1 , + ( r ) e − a min ( θ ∥ − θ t a i l ∥ ( r ) ) \lambda_1(\theta^{\parallel}) \le\lambda_{1,+}(r)e^{-a_{\min}(\theta^{\parallel}-\theta_{\mathrm{tail}}^{\parallel}(r))} λ 1 ( θ ∥ ) ≤ λ 1 , + ( r ) e − a m i n ( θ ∥ − θ tail ∥ ( r ))
with
λ 1 , + ( r ) = λ max ( Σ ∣ U ) e − a min θ t a i l ∥ ( r ) . \lambda_{1,+}(r) = \lambda_{\max}(\Sigma|_U)e^{-a_{\min}\theta_{\mathrm{tail}}^{\parallel}(r)}. λ 1 , + ( r ) = λ m a x ( Σ ∣ U ) e − a m i n θ tail ∥ ( r ) .
Step 10: Tail uniform conditioning under spanning condition. Assume
U = span { u i : i ∈ I ∗ } . U = \operatorname{span}\{u_i : i \in I_*\}. U = span { u i : i ∈ I ∗ } .
Then Σ ∗ ∣ U \Sigma_*|_U Σ ∗ ∣ U is positive definite, so
λ min ( Σ ∗ ∣ U ) > 0. \lambda_{\min}(\Sigma_*|_U) > 0. λ m i n ( Σ ∗ ∣ U ) > 0.
For θ ∥ ≥ 0 \theta^{\parallel} \ge0 θ ∥ ≥ 0 , the logistic curvature ϕ ′ ′ \phi^{\prime\prime} ϕ ′′ is decreasing on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . Since a i ≥ a min a_i \ge a_{\min} a i ≥ a m i n , we have
ϕ ′ ′ ( a i θ ∥ ) ≤ ϕ ′ ′ ( a min θ ∥ ) ∀ i . \phi^{\prime\prime}(a_i\theta^{\parallel}) \le\phi^{\prime\prime}(a_{\min}\theta^{\parallel}) \qquad\forall i. ϕ ′′ ( a i θ ∥ ) ≤ ϕ ′′ ( a m i n θ ∥ ) ∀ i .
Therefore
H ( θ ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ) u i u i T ⪯ ϕ ′ ′ ( a min θ ) Σ , H(\theta) = \frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta)u_i u_i^{\mathsf T} \preceq\phi^{\prime\prime}(a_{\min}\theta)\Sigma, H ( θ ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ) u i u i T ⪯ ϕ ′′ ( a m i n θ ) Σ ,
and hence
λ 1 ( θ ) ≤ ϕ ′ ′ ( a min θ ) λ max ( Σ ∣ U ) . \lambda_1(\theta) \leq\phi^{\prime\prime}(a_{\min}\theta)\lambda_{\max}(\Sigma|_U). λ 1 ( θ ) ≤ ϕ ′′ ( a m i n θ ) λ m a x ( Σ ∣ U ) .
On the other hand, restricting the sum defining H ( θ ) H(\theta) H ( θ ) to the support vector components gives
H ( θ ) ⪰ 1 N ∑ i ∈ I ∗ ϕ ′ ′ ( a min θ ) u i u i T = ϕ ′ ′ ( a min θ ) Σ ∗ . H(\theta) \succeq\frac{1}{N}\sum_{i\in I_*}\phi^{\prime\prime}(a_{\min}\theta)u_i u_i^{\mathsf T} = \phi^{\prime\prime}(a_{\min}\theta)\Sigma_*. H ( θ ) ⪰ N 1 i ∈ I ∗ ∑ ϕ ′′ ( a m i n θ ) u i u i T = ϕ ′′ ( a m i n θ ) Σ ∗ .
Thus
H ( θ ) ⪰ ϕ ′ ′ ( a min θ ) λ min ( Σ ∗ ∣ U ) I U . H(\theta) \succeq\phi^{\prime\prime}(a_{\min}\theta)\lambda_{\min}(\Sigma_*|_U)I_U. H ( θ ) ⪰ ϕ ′′ ( a m i n θ ) λ m i n ( Σ ∗ ∣ U ) I U .
Combining the last two estimates yields
H ( θ ) ⪰ λ min ( Σ ∗ ∣ U ) λ max ( Σ ∣ U ) λ 1 ( θ ) I U . H(\theta) \succeq\frac{\lambda_{\min}(\Sigma_*|_U)}{\lambda_{\max}(\Sigma|_U)}\lambda_1(\theta)I_U. H ( θ ) ⪰ λ m a x ( Σ ∣ U ) λ m i n ( Σ ∗ ∣ U ) λ 1 ( θ ) I U .
Thus the tail conditioning condition holds with
ρ t a i l = λ min ( Σ ∗ ∣ U ) λ max ( Σ ∣ U ) . \rho_{\mathrm{tail}} = \frac{\lambda_{\min}(\Sigma_*|_U)}{\lambda_{\max}(\Sigma|_U)}. ρ tail = λ m a x ( Σ ∣ U ) λ m i n ( Σ ∗ ∣ U ) .
This completes the verification.
Proof of Theorem 2 (spanning vs. non-spanning)
We prove the spectral statement behind Theorem 2. Recall that
H ( θ ) = 1 N ∑ i = 1 N ϕ ′ ′ ( a i θ ) u i u i T on U = span { u 1 , … , u N } . H(\theta) = \frac{1}{N}\sum_{i=1}^{N}\phi^{\prime\prime}(a_i\theta)u_i u_i^{\mathsf T}
\qquad\text{on } U = \operatorname{span}\{u_1,\ldots,u_N\}. H ( θ ) = N 1 i = 1 ∑ N ϕ ′′ ( a i θ ) u i u i T on U = span { u 1 , … , u N } .
Let
α 1 < α 2 < ⋯ < α L \alpha_1 < \alpha_2 < \cdots< \alpha_L α 1 < α 2 < ⋯ < α L
be the distinct values among a 1 , … , a N a_1,\ldots,a_N a 1 , … , a N , and define the layer index sets
I j : = { i : a i = α j } , j = 1 , … , L . I_j := \{i : a_i = \alpha_j\}, \qquad j=1,\ldots,L. I j := { i : a i = α j } , j = 1 , … , L .
For each layer, define the positive semidefinite frame operator
Σ j : = 1 N ∑ i ∈ I j u i u i T , \Sigma_j := \frac{1}{N}\sum_{i\in I_j}u_i u_i^{\mathsf T}, Σ j := N 1 i ∈ I j ∑ u i u i T ,
so that
H ( θ ) = ∑ j = 1 L ϕ ′ ′ ( α j θ ) Σ j . H(\theta) = \sum_{j=1}^{L}\phi^{\prime\prime}(\alpha_j\theta)\Sigma_j. H ( θ ) = j = 1 ∑ L ϕ ′′ ( α j θ ) Σ j .
We also recall the nested subspaces
U ( j ) : = span { u i : a i ≤ α j } , U ( 1 ) ⊆ ⋯ ⊆ U ( L ) = U , U^{(j)} := \operatorname{span}\{u_i : a_i \leq\alpha_j\}, \qquad U^{(1)} \subseteq\cdots\subseteq U^{(L)} = U, U ( j ) := span { u i : a i ≤ α j } , U ( 1 ) ⊆ ⋯ ⊆ U ( L ) = U ,
and
j ∗ : = min { j : U ( j ) = U } . j_* := \min\{j : U^{(j)} = U\}. j ∗ := min { j : U ( j ) = U } .
We assume throughout this subsection that the support layer is transversely active, i.e.
λ max ( Σ 1 ∣ U ) > 0. \lambda_{\max}(\Sigma_1|_U) > 0. λ m a x ( Σ 1 ∣ U ) > 0.
The degenerate case Σ 1 ∣ U = 0 \Sigma_1|_U = 0 Σ 1 ∣ U = 0 only changes the denominator rate in the conditioning ratio and is omitted for notational clarity.
We will use the elementary logistic tail bounds
c − e − s ≤ ϕ ′ ′ ( s ) ≤ c + e − s , s ≥ s 0 , (65) c_-e^{-s} \leq\phi^{\prime\prime}(s) \leq c_+e^{-s}, \qquad s \geq s_0, \tag*{(65)} c − e − s ≤ ϕ ′′ ( s ) ≤ c + e − s , s ≥ s 0 , (65)
for some constants 0 < c − ≤ c + < ∞ 0 < c_- \leq c_+ < \infty 0 < c − ≤ c + < ∞ . For instance, once s ≥ 0 s \geq0 s ≥ 0 , one may take c − = 1 / 4 c_- = 1/4 c − = 1/4 and c + = 1 c_+ = 1 c + = 1 . Since all α j > 0 \alpha_j > 0 α j > 0 , these bounds for all sufficiently large θ \theta θ .
Lemma 1 (Top eigenvalue is governed by the support vectors). For all sufficiently large θ ℓ \theta^\ell θ ℓ ,
λ max ( H ( θ ℓ ) ∣ U ) ≍ e − α 1 θ ℓ . \lambda_{\max}\left(H(\theta^\ell)|_U\right)\asymp e^{-\alpha_1\theta^\ell}. λ m a x ( H ( θ ℓ ) ∣ U ) ≍ e − α 1 θ ℓ .
Proof. For the upper bound, using α j ≥ α 1 \alpha_j\ge\alpha_1 α j ≥ α 1 and (66),
H ( θ ℓ ) ⪯ c + ∑ j = 1 L e − α j θ ℓ Σ j ⪯ c + e − α 1 θ ℓ ∑ j = 1 L Σ j . H(\theta^\ell)\preceq c_+\sum_{j=1}^{L}e^{-\alpha_j\theta^\ell}\Sigma_j\preceq c_+e^{-\alpha_1\theta^\ell}\sum_{j=1}^{L}\Sigma_j. H ( θ ℓ ) ⪯ c + j = 1 ∑ L e − α j θ ℓ Σ j ⪯ c + e − α 1 θ ℓ j = 1 ∑ L Σ j .
Therefore
λ max ( H ( θ ℓ ) ∣ U ) ≤ c + e − α 1 θ ℓ λ max ( ∑ j = 1 L Σ j ∣ U ) . \lambda_{\max}\left(H(\theta^\ell)|_U\right)\le c_+e^{-\alpha_1\theta^\ell}\lambda_{\max}\left(\left.\sum_{j=1}^{L}\Sigma_j\right|_U\right). λ m a x ( H ( θ ℓ ) ∣ U ) ≤ c + e − α 1 θ ℓ λ m a x j = 1 ∑ L Σ j U .
For the lower bound, since U ( 1 ) ≠ { 0 } U^{(1)}\ne\{0\} U ( 1 ) = { 0 } , the operator Σ 1 ∣ U \left.\Sigma_1\right|_U Σ 1 ∣ U has positive top eigenvalue. Hence
H ( θ ℓ ) ⪰ φ ′ ′ ( α 1 θ ℓ ) Σ 1 ⪰ c − e − α 1 θ ℓ Σ 1 H(\theta^\ell)\succeq\varphi^{\prime\prime}(\alpha_1\theta^\ell)\Sigma_1\succeq c_-e^{-\alpha_1\theta^\ell}\Sigma_1 H ( θ ℓ ) ⪰ φ ′′ ( α 1 θ ℓ ) Σ 1 ⪰ c − e − α 1 θ ℓ Σ 1
for all sufficiently large θ ℓ \theta^\ell θ ℓ . Taking top eigenvalues gives
λ max ( H ( θ ℓ ) ∣ U ) ≥ c − e − α 1 θ ℓ λ max ( Σ 1 ∣ U ) . \lambda_{\max}\left(H(\theta^\ell)|_U\right)\ge c_-e^{-\alpha_1\theta^\ell}\lambda_{\max}\left(\left.\Sigma_1\right|_U\right). λ m a x ( H ( θ ℓ ) ∣ U ) ≥ c − e − α 1 θ ℓ λ m a x ( Σ 1 ∣ U ) .
Combining the two estimates proves the claim. □
Lemma 2 (Smallest eigenvalue is governed by the first spanning vectors). For all sufficiently large θ ℓ \theta^\ell θ ℓ ,
λ min ( H ( θ ℓ ) ∣ U ) ≲ e − α j ∗ θ ℓ . \lambda_{\min}\left(H(\theta^\ell)|_U\right)\lesssim e^{-\alpha_{j_*}\theta^\ell}. λ m i n ( H ( θ ℓ ) ∣ U ) ≲ e − α j ∗ θ ℓ .
Proof. Let
Σ ≤ j ∗ : = ∑ k = 1 j ∗ Σ k . \Sigma_{\leq j_*}:=\sum_{k=1}^{j_*}\Sigma_k. Σ ≤ j ∗ := k = 1 ∑ j ∗ Σ k .
By definition of j ∗ j_* j ∗ , the subspace U ( j ∗ ) = U U^{(j_*)}=U U ( j ∗ ) = U . Since each Σ k \Sigma_k Σ k is positive semidefinite and range ( Σ ≤ j ) = U ( j ) \operatorname{range}(\Sigma_{\leq j})=U^{(j)} range ( Σ ≤ j ) = U ( j ) , the identity U ( j ∗ ) = U U^{(j_*)}=U U ( j ∗ ) = U implies that Σ ≤ j ∗ ∣ U \left.\Sigma_{\leq j_*}\right|_U Σ ≤ j ∗ ∣ U is positive definite. Therefore
λ min ( Σ ≤ j ∗ ∣ U ) > 0. \lambda_{\min}\left(\left.\Sigma_{\leq j_*}\right|_U\right)>0. λ m i n ( Σ ≤ j ∗ ∣ U ) > 0.
For the lower bound, since α k ≤ α j ∗ \alpha_k\leq\alpha_{j_*} α k ≤ α j ∗ for k ≤ j ∗ k\leq j_* k ≤ j ∗ , the tail estimate (66) gives, for sufficiently large θ ℓ \theta^\ell θ ℓ ,
H ( θ ℓ ) ⪰ ∑ k = 1 j ∗ c − e − α k θ ℓ Σ k H(\theta^\ell)\succeq\sum_{k=1}^{j_*}c_-e^{-\alpha_k\theta^\ell}\Sigma_k H ( θ ℓ ) ⪰ k = 1 ∑ j ∗ c − e − α k θ ℓ Σ k
⪰ c − e − α j ∗ θ ℓ ∑ k = 1 j ∗ Σ k = c − e − α j ∗ θ ℓ Σ ≤ j ∗ . \succeq c_-e^{-\alpha_{j_*}\theta^\ell}\sum_{k=1}^{j_*}\Sigma_k=c_-e^{-\alpha_{j_*}\theta^\ell}\Sigma_{\leq j_*}. ⪰ c − e − α j ∗ θ ℓ k = 1 ∑ j ∗ Σ k = c − e − α j ∗ θ ℓ Σ ≤ j ∗ .
Thus
λ min ( H ( θ ℓ ) ∣ U ) ≥ c − e − α j ∗ θ ℓ λ min ( Σ ≤ j ∗ ∣ U ) . \lambda_{\min}\left(H(\theta^\ell)|_U\right)\geq c_-e^{-\alpha_{j_*}\theta^\ell}\lambda_{\min}\left(\left.\Sigma_{\leq j_*}\right|_U\right). λ m i n ( H ( θ ℓ ) ∣ U ) ≥ c − e − α j ∗ θ ℓ λ m i n ( Σ ≤ j ∗ ∣ U ) .
For the upper bound, first consider the case j ∗ > 1 j_*>1 j ∗ > 1 . Since j ∗ j_* j ∗ is minimal, U ( j ∗ − 1 ) ⊊ U U^{(j_*-1)}\subsetneq U U ( j ∗ − 1 ) ⊊ U . Hence there exists a unit vector
w ∈ U ∩ ( U ( j ∗ − 1 ) ) ⊥ . w\in U\cap\left(U^{(j_*-1)}\right)^\perp. w ∈ U ∩ ( U ( j ∗ − 1 ) ) ⊥ .
For every i i i with α i ≤ α j ∗ − 1 \alpha_i\leq\alpha_{j_*-1} α i ≤ α j ∗ − 1 , we have u i ∈ U ( j ∗ − 1 ) u_i\in U^{(j_*-1)} u i ∈ U ( j ∗ − 1 ) , so ⟨ w , u i ⟩ = 0 \langle w,u_i\rangle=0 ⟨ w , u i ⟩ = 0 . Therefore the vectors for which k < j ∗ k<j_* k < j ∗ do not contribute to the Rayleigh quotient of w w w :
w ⊤ H ( θ ℓ ) w = ∑ k = j ∗ L φ ′ ′ ( α k θ ℓ ) w ⊤ Σ k w ≤ c + ∑ k = j ∗ L e − α k θ ℓ w ⊤ Σ k w ≤ c + e − α j ∗ θ ℓ w ⊤ ( ∑ k = j ∗ L Σ k ) w . \begin{aligned}
w^\top H(\theta^\ell)w=\sum_{k=j_*}^{L}\varphi^{\prime\prime}(\alpha_k\theta^\ell)w^\top\Sigma_kw \\
&\leq c_+\sum_{k=j_*}^{L}e^{-\alpha_k\theta^\ell}w^\top\Sigma_kw \\
&\leq c_+e^{-\alpha_{j_*}\theta^\ell}w^\top\left(\sum_{k=j_*}^{L}\Sigma_k\right)w.
\end{aligned} w ⊤ H ( θ ℓ ) w = k = j ∗ ∑ L φ ′′ ( α k θ ℓ ) w ⊤ Σ k w ≤ c + k = j ∗ ∑ L e − α k θ ℓ w ⊤ Σ k w ≤ c + e − α j ∗ θ ℓ w ⊤ k = j ∗ ∑ L Σ k w .
Since ∥ w ∥ = 1 \lVert w\rVert= 1 ∥ w ∥ = 1 , the final factor is bounded by λ max ( ∑ k = j ∗ L Σ k ∣ U ) \lambda_{\max}\left(\sum_{k=j_*}^{L}\Sigma_k|_U\right) λ m a x ( ∑ k = j ∗ L Σ k ∣ U ) . Thus
λ min ( H ( θ ∥ ) ∣ U ) ≤ w ⊤ H ( θ ∥ ) w ≲ e − α j ∗ θ ∥ . \lambda_{\min}\left(H(\theta^{\parallel})|_U\right) \le w^\top H(\theta^{\parallel})w \lesssim e^{-\alpha_{j_*}\theta^{\parallel}}. λ m i n ( H ( θ ∥ ) ∣ U ) ≤ w ⊤ H ( θ ∥ ) w ≲ e − α j ∗ θ ∥ .
If j ∗ = 1 j_* = 1 j ∗ = 1 , the same upper bound follows more simply from
λ min ( H ( θ ∥ ) ∣ U ) ≤ λ max ( H ( θ ∥ ) ∣ U ) ≍ e − α 1 θ ∥ , \lambda_{\min}\left(H(\theta^{\parallel})|_U\right) \le\lambda_{\max}\left(H(\theta^{\parallel})|_U\right) \asymp e^{-\alpha_1\theta^{\parallel}}, λ m i n ( H ( θ ∥ ) ∣ U ) ≤ λ m a x ( H ( θ ∥ ) ∣ U ) ≍ e − α 1 θ ∥ ,
using Lemma 1. This completes the proof. □
Proposition 5 (Conditioning ratio asymptotics). For all sufficiently large θ ∥ \theta^{\parallel} θ ∥ ,
ρ ( θ ∥ ) : = λ min ( H ( θ ∥ ) ∣ U ) λ max ( H ( θ ∥ ) ∣ U ) ≳ e − ( α j ∗ − α 1 ) θ ∥ . (66) \rho(\theta^{\parallel}) := \frac{\lambda_{\min}\left(H(\theta^{\parallel})|_U\right)}{\lambda_{\max}\left(H(\theta^{\parallel})|_U\right)} \gtrsim e^{-(\alpha_{j_*}-\alpha_1)\theta^{\parallel}}. \tag*{(66)} ρ ( θ ∥ ) := λ m a x ( H ( θ ∥ ) ∣ U ) λ m i n ( H ( θ ∥ ) ∣ U ) ≳ e − ( α j ∗ − α 1 ) θ ∥ . (66) In particular:
(i) if j ∗ = 1 j_* = 1 j ∗ = 1 , then ρ ( θ ∥ ) ≳ 1 \rho(\theta^{\parallel}) \gtrsim1 ρ ( θ ∥ ) ≳ 1 ;
(ii) if j ∗ > 1 j_* > 1 j ∗ > 1 , then ρ ( θ ∥ ) → 0 \rho(\theta^{\parallel}) \to0 ρ ( θ ∥ ) → 0 exponentially in θ ∥ \theta^{\parallel} θ ∥ .
Proof. Divide the estimate of Lemma 2 by the estimate of Lemma 1. This gives
ρ ( θ ∥ ) ≲ e − α j ∗ θ ∥ e − α 1 θ ∥ = e − ( α j ∗ − α 1 ) θ ∥ . \rho(\theta^{\parallel}) \lesssim\frac{e^{-\alpha_{j_*}\theta^{\parallel}}}{e^{-\alpha_1\theta^{\parallel}}} = e^{-(\alpha_{j_*}-\alpha_1)\theta^{\parallel}}. ρ ( θ ∥ ) ≲ e − α 1 θ ∥ e − α j ∗ θ ∥ = e − ( α j ∗ − α 1 ) θ ∥ . The two cases follow immediately. □
We now translate the θ ∥ \theta^{\parallel} θ ∥ -asymptotics into time asymptotics along the GD trajectory. Importantly, the logarithmic parallel growth in Theorem 4(ii) does not use the uniform tail-conditioning hypothesis; it only uses the tail drift bounds and the post-entry nonexpansion estimate. Therefore it applies in both the spanning and non-spanning cases.
Applying Theorem 4(ii) with β t a i l = α 1 \beta_{\mathrm{tail}} = \alpha_1 β tail = α 1 , there exist constants 0 < c y ≤ C y < ∞ 0 < c_y \le C_y < \infty 0 < c y ≤ C y < ∞ such that, for all sufficiently large t t t ,
c y ( 1 + t − t s t a b ) ≤ e α 1 ( θ t ∥ − θ ∗ ) ≤ C y ( 1 + t − t s t a b ) . c_y(1 + t - t_{\mathrm{stab}}) \le e^{\alpha_1(\theta_t^{\parallel}-\theta_*)} \le C_y(1 + t - t_{\mathrm{stab}}). c y ( 1 + t − t stab ) ≤ e α 1 ( θ t ∥ − θ ∗ ) ≤ C y ( 1 + t − t stab ) .
Consequently, for every Δ ≥ 0 \Delta\ge0 Δ ≥ 0 ,
e − Δ θ t ∥ = e − Δ θ ∗ ( e α 1 ( θ t ∥ − θ ∗ ) ) − Δ / α 1 ≍ ( 1 + t − t s t a b ) − Δ / α 1 . e^{-\Delta\theta_t^{\parallel}} = e^{-\Delta\theta_*}\left(e^{\alpha_1(\theta_t^{\parallel}-\theta_*)}\right)^{-\Delta/\alpha_1}
\asymp(1 + t - t_{\mathrm{stab}})^{-\Delta/\alpha_1}. e − Δ θ t ∥ = e − Δ θ ∗ ( e α 1 ( θ t ∥ − θ ∗ ) ) − Δ/ α 1 ≍ ( 1 + t − t stab ) − Δ/ α 1 .
Taking Δ = α j ∗ − α 1 \Delta= \alpha_{j_*} - \alpha_1 Δ = α j ∗ − α 1 in (66) gives
ρ ( θ t ∥ ) ≳ ( 1 + t − t s t a b ) − ( α j ∗ − α 1 ) / α 1 . \rho(\theta_t^{\parallel}) \gtrsim(1 + t - t_{\mathrm{stab}})^{-(\alpha_{j_*}-\alpha_1)/\alpha_1}. ρ ( θ t ∥ ) ≳ ( 1 + t − t stab ) − ( α j ∗ − α 1 ) / α 1 .
In particular, for fixed η \eta η , replacing 1 + t − t s t a b 1+t-t_{\mathrm{stab}} 1 + t − t stab by t t t only changes the implicit constants for sufficiently large t t t . Hence, in the non-spanning case j ∗ > 1 j_* > 1 j ∗ > 1 ,
ρ ( θ t ∥ ) ≳ t − ( α j ∗ − α 1 ) / α 1 . \rho(\theta_t^{\parallel}) \gtrsim t^{-(\alpha_{j_*}-\alpha_1)/\alpha_1}. ρ ( θ t ∥ ) ≳ t − ( α j ∗ − α 1 ) / α 1 .
Componentwise rates under simultaneous diagonalizability
Theorem 2 shows that, when the support vectors do not span the full active normal space, the full conditioning ratio ρ ( θ ∥ ) \rho(\theta^{\parallel}) ρ ( θ ∥ ) decays to zero. Consequently, the top-eigenvalue tail schedule does not imply contraction of the entire active normal vector. In this subsection we record a complementary positive statement: if the operators Σ j \Sigma_j Σ j are simultaneously diagonalizable and the effective GD maps preserve the same coordinates, then each coordinate admits its own tail contraction rate.
Recall the decomposition
H ( θ ∥ ) = ∑ j = 1 L φ ′ ′ ( α j θ ∥ ) Σ j , α 1 < α 2 < ⋯ < α L , H(\theta^{\parallel}) = \sum_{j=1}^{L}\varphi''(\alpha_j\theta^{\parallel})\Sigma_j,\qquad\alpha_1 < \alpha_2 < \cdots< \alpha_L, H ( θ ∥ ) = j = 1 ∑ L φ ′′ ( α j θ ∥ ) Σ j , α 1 < α 2 < ⋯ < α L ,
where
Σ j : = 1 N ∑ i ∈ I j u i u i ⊤ , I j : = { i : a i = α j } . \Sigma_j := \frac{1}{N}\sum_{i\in I_j}u_i u_i^\top,\qquad I_j := \{i : a_i = \alpha_j\}. Σ j := N 1 i ∈ I j ∑ u i u i ⊤ , I j := { i : a i = α j } .
Assumption 1 (Simultaneous diagonalizability and coordinatewise updates). There exists an orthonormal basis e 1 , … , e q e_1,\ldots,e_q e 1 , … , e q of U U U , where q : = dim U q := \dim U q := dim U , such that:
(i) every operator Σ j \Sigma_j Σ j is diagonal in this basis:
Σ j e k = σ j , k e k , σ j , k ≥ 0 ; \Sigma_j e_k = \sigma_{j,k}e_k,\qquad\sigma_{j,k} \ge0; Σ j e k = σ j , k e k , σ j , k ≥ 0 ; (ii) for all t ≥ t s t a b t \ge t_{\mathrm{stab}} t ≥ t stab , the effective orthogonal update matrix M t M_t M t from
θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ \theta_{t+1}^{\perp} = (I-\eta M_t)\theta_t^{\perp} θ t + 1 ⊥ = ( I − η M t ) θ t ⊥ is diagonal in the same basis:
M t e k = m k , t e k ; M_t e_k = m_{k,t}e_k; M t e k = m k , t e k ; (iii) the coordinatewise RC bounds hold:
( 1 − δ r ) h k ( θ t ∥ ) ≤ m k , t ≤ ( 1 + δ r ) h k ( θ t ∥ ) , (1-\delta_r)h_k(\theta_t^{\parallel}) \le m_{k,t} \le(1+\delta_r)h_k(\theta_t^{\parallel}), ( 1 − δ r ) h k ( θ t ∥ ) ≤ m k , t ≤ ( 1 + δ r ) h k ( θ t ∥ ) , where
h k ( θ ∥ ) : = ⟨ e k , H ( θ ∥ ) e k ⟩ = ∑ j = 1 L σ j , k ϕ ′ ′ ( α j θ ∥ ) . h_k(\theta^{\parallel}) := \langle e_k,H(\theta^{\parallel})e_k\rangle= \sum_{j=1}^{L}\sigma_{j,k}\phi^{\prime\prime}(\alpha_j\theta^{\parallel}). h k ( θ ∥ ) := ⟨ e k , H ( θ ∥ ) e k ⟩ = j = 1 ∑ L σ j , k ϕ ′′ ( α j θ ∥ ) .
For each coordinate e k e_k e k , define its first active index
j ( k ) : = min { j : σ j , k > 0 } . j(k) := \min\{j:\sigma_{j,k}>0\}. j ( k ) := min { j : σ j , k > 0 } .
Equivalently, j ( k ) j(k) j ( k ) is the first to see the coordinate e k e_k e k . The associated coordinate tail exponent is
p k : = α j ( k ) α 1 ≥ 1. p_k := \frac{\alpha_{j(k)}}{\alpha_1} \ge1. p k := α 1 α j ( k ) ≥ 1.
Thus p k = 1 p_k=1 p k = 1 exactly for coordinates seen by the support-vector layer I 1 = I ∗ I_1=I_* I 1 = I ∗ , while p k > 1 p_k>1 p k > 1 for coordinates invisible to the support vectors and first appearing in a later layer.
Write the coordinate decomposition
θ t ⊥ = ∑ k = 1 q c k , t e k , c k , t : = ⟨ θ t ⊥ , e k ⟩ . \theta_t^{\perp}=\sum_{k=1}^{q}c_{k,t}e_k,\qquad c_{k,t}:=\langle\theta_t^{\perp},e_k\rangle. θ t ⊥ = k = 1 ∑ q c k , t e k , c k , t := ⟨ θ t ⊥ , e k ⟩ .
Under Assumption 1, the orthogonal update is coordinatewise:
c k , t + 1 = ( 1 − η m k , t ) c k , t . (67) c_{k,t+1}=(1-\eta m_{k,t})c_{k,t}. \tag*{(67)} c k , t + 1 = ( 1 − η m k , t ) c k , t . (67)
Lemma 3 (Coordinatewise sharpness tails). For every coordinate k k k , there exist constants 0 < h k , − ≤ h k , + < ∞ 0<h_{k,-}\le h_{k,+}<\infty 0 < h k , − ≤ h k , + < ∞ and a threshold θ k , t a i l \theta_{k,\mathrm{tail}} θ k , tail such that, for all θ ∥ ≥ θ k , t a i l \theta^{\parallel}\ge\theta_{k,\mathrm{tail}} θ ∥ ≥ θ k , tail ,
h k , − e − α j ( k ) θ ∥ ≤ h k ( θ ∥ ) ≤ h k , + e − α j ( k ) θ ∥ . (68) h_{k,-}e^{-\alpha_{j(k)}\theta^{\parallel}}\le h_k(\theta^{\parallel})\le h_{k,+}e^{-\alpha_{j(k)}\theta^{\parallel}}. \tag*{(68)} h k , − e − α j ( k ) θ ∥ ≤ h k ( θ ∥ ) ≤ h k , + e − α j ( k ) θ ∥ . (68) Consequently, along the stable-tail trajectory,
h k ( θ t ∥ ) ≍ ( 1 + t − t s t a b ) − p k . (69) h_k(\theta_t^{\parallel})\asymp(1+t-t_{\mathrm{stab}})^{-p_k}. \tag*{(69)} h k ( θ t ∥ ) ≍ ( 1 + t − t stab ) − p k . (69)
Proof. By definition,
h k ( θ ∥ ) = ∑ j = 1 L σ j , k ϕ ′ ′ ( α j θ ∥ ) . h_k(\theta^{\parallel})=\sum_{j=1}^{L}\sigma_{j,k}\phi^{\prime\prime}(\alpha_j\theta^{\parallel}). h k ( θ ∥ ) = j = 1 ∑ L σ j , k ϕ ′′ ( α j θ ∥ ) .
For j < j ( k ) j<j(k) j < j ( k ) , σ j , k = 0 \sigma_{j,k}=0 σ j , k = 0 . Since σ j ( k ) , k > 0 \sigma_{j(k),k}>0 σ j ( k ) , k > 0 , the leading nonzero term is
σ j ( k ) , k ϕ ′ ′ ( α j ( k ) θ ∥ ) . \sigma_{j(k),k}\phi^{\prime\prime}(\alpha_{j(k)}\theta^{\parallel}). σ j ( k ) , k ϕ ′′ ( α j ( k ) θ ∥ ) .
Using the logistic tail bounds
c − e − s ≤ ϕ ′ ′ ( s ) ≤ c + e − s c_-e^{-s}\le\phi^{\prime\prime}(s)\le c_+e^{-s} c − e − s ≤ ϕ ′′ ( s ) ≤ c + e − s
for all sufficiently large s s s , we get the lower bound
h k ( θ ∥ ) ≥ c − σ j ( k ) , k e − α j ( k ) θ ∥ . h_k(\theta^{\parallel})\ge c_-\sigma_{j(k),k}e^{-\alpha_{j(k)}\theta^{\parallel}}. h k ( θ ∥ ) ≥ c − σ j ( k ) , k e − α j ( k ) θ ∥ .
For the upper bound,
h k ( θ ℓ ) ≤ c + ∑ j = j ( k ) L σ j , k e − α j θ ℓ ≤ c + ( ∑ j = j ( k ) L σ j , k ) e − α j ( k ) θ ℓ . h_k(\theta^\ell) \le c_+ \sum_{j=j(k)}^L \sigma_{j,k}e^{-\alpha_j\theta^\ell} \le c_+ \left(\sum_{j=j(k)}^L \sigma_{j,k}\right)e^{-\alpha_{j(k)}\theta^\ell}. h k ( θ ℓ ) ≤ c + j = j ( k ) ∑ L σ j , k e − α j θ ℓ ≤ c + j = j ( k ) ∑ L σ j , k e − α j ( k ) θ ℓ .
This proves (68) .
Now use the stable-tail schedule from Theorem 4:
e α 1 ( θ t ℓ − θ ∗ ) ≍ 1 + t − t s t a b . e^{\alpha_1(\theta_t^\ell-\theta_*)} \asymp1+t-t_{\mathrm{stab}}. e α 1 ( θ t ℓ − θ ∗ ) ≍ 1 + t − t stab .
Therefore
e − α j ( k ) θ t ℓ ≍ ( 1 + t − t s t a b ) − α j ( k ) / α 1 = ( 1 + t − t s t a b ) − p k , e^{-\alpha_{j(k)}\theta_t^\ell} \asymp(1+t-t_{\mathrm{stab}})^{-\alpha_{j(k)}/\alpha_1}=(1+t-t_{\mathrm{stab}})^{-p_k}, e − α j ( k ) θ t ℓ ≍ ( 1 + t − t stab ) − α j ( k ) / α 1 = ( 1 + t − t stab ) − p k ,
which proves (69) .
Proposition 6 (Componentwise contraction budgets). Assume Assumption 1. Then for every coordinate k k k , there exist constants 0 < b k , − ≤ b k , + < ∞ 0<b_{k,-}\le b_{k,+}<\infty 0 < b k , − ≤ b k , + < ∞ and a time t k ≥ t s t a b t_k\ge t_{\mathrm{stab}} t k ≥ t stab such that, for all t ≥ t k t\ge t_k t ≥ t k ,
b k , − ( 1 + t − t s t a b ) − p k ≤ η m k , t ≤ b k , + ( 1 + t − t s t a b ) − p k . (70) b_{k,-}(1+t-t_{\mathrm{stab}})^{-p_k}\le\eta m_{k,t}\le b_{k,+}(1+t-t_{\mathrm{stab}})^{-p_k}. \tag*{(70)} b k , − ( 1 + t − t stab ) − p k ≤ η m k , t ≤ b k , + ( 1 + t − t stab ) − p k . (70) Moreover, after increasing t k t_k t k if necessary, one has 0 ≤ η m k , t ≤ 1 0\le\eta m_{k,t}\le1 0 ≤ η m k , t ≤ 1 for all t ≥ t k t\ge t_k t ≥ t k , and hence
∣ c k , t ∣ ≤ ∣ c k , t k ∣ exp ( − ∑ s = t k t − 1 η m k , s ) . (71) |c_{k,t}|\le|c_{k,t_k}|\exp\left(-\sum_{s=t_k}^{t-1}\eta m_{k,s}\right). \tag*{(71)} ∣ c k , t ∣ ≤ ∣ c k , t k ∣ exp ( − s = t k ∑ t − 1 η m k , s ) . (71)
Proof. The bounds (70) follow immediately from the coordinatewise RC inequality
( 1 − δ r ) h k ( θ t ℓ ) ≤ m k , t ≤ ( 1 + δ r ) h k ( θ t ℓ ) (1-\delta_r)h_k(\theta_t^\ell)\le m_{k,t}\le(1+\delta_r)h_k(\theta_t^\ell) ( 1 − δ r ) h k ( θ t ℓ ) ≤ m k , t ≤ ( 1 + δ r ) h k ( θ t ℓ )
and Lemma 3. Since p k ≥ 1 p_k\ge1 p k ≥ 1 , the upper bound in (70) tends to zero. Hence, after increasing t k t_k t k , we may assume 0 ≤ η m k , t ≤ 1 0\le\eta m_{k,t}\le1 0 ≤ η m k , t ≤ 1 for all t ≥ t k t\ge t_k t ≥ t k . The coordinate update
c k , t + 1 = ( 1 − η m k , t ) c k , t c_{k,t+1}=(1-\eta m_{k,t})c_{k,t} c k , t + 1 = ( 1 − η m k , t ) c k , t
then gives
∣ c k , t + 1 ∣ = ( 1 − η m k , t ) ∣ c k , t ∣ . |c_{k,t+1}|=(1-\eta m_{k,t})|c_{k,t}|. ∣ c k , t + 1 ∣ = ( 1 − η m k , t ) ∣ c k , t ∣.
Iterating and using 1 − a ≤ e − a 1-a\le e^{-a} 1 − a ≤ e − a yields (71) .
Theorem 5 (Componentwise tail dichotomy). Assume Assumption 1. For each coordinate k k k , let
p k : = α j ( k ) α 1 . p_k:=\frac{\alpha_{j(k)}}{\alpha_1}. p k := α 1 α j ( k ) . Then the following hold.
Support-vector coordinates (p k = 1 p_k=1 p k = 1 ). For each k k k , there exists γ k > 0 \gamma_k>0 γ k > 0 and a time t k t_k t k such that, for all t ≥ t k t\ge t_k t ≥ t k ,
∣ c k , t ∣ ≤ ∣ c k , t k ∣ ( 1 + t k − t s t a b 1 + t − t s t a b ) γ k . (72) |c_{k,t}|\le|c_{k,t_k}|\left(\frac{1+t_k-t_{\mathrm{stab}}}{1+t-t_{\mathrm{stab}}}\right)^{\gamma_k}. \tag*{(72)} ∣ c k , t ∣ ≤ ∣ c k , t k ∣ ( 1 + t − t stab 1 + t k − t stab ) γ k . (72) In particular,
c k , t → 0. c_{k,t}\to0. c k , t → 0. Later-layer coordinates (p k > 1 p_k>1 p k > 1 ). There exist constants 0 < C k , − ≤ C k , + < ∞ 0<C_{k,-}\le C_{k,+}<\infty 0 < C k , − ≤ C k , + < ∞ such that, for all t ≥ t k t\ge t_k t ≥ t k ,
C k , − ∣ c k , t k ∣ ≤ ∣ c k , t ∣ ≤ C k , + ∣ c k , t k ∣ . (73) C_{k,-}|c_{k,t_k}|\le|c_{k,t}|\le C_{k,+}|c_{k,t_k}|. \tag*{(73)} C k , − ∣ c k , t k ∣ ≤ ∣ c k , t ∣ ≤ C k , + ∣ c k , t k ∣. (73) In particular, the tail contraction budget is finite:
∑ s = t k ∞ η m k , s < ∞ . (74) \sum_{s=t_k}^{\infty}\eta m_{k,s}<\infty. \tag*{(74)} s = t k ∑ ∞ η m k , s < ∞. (74) Thus the stable tail alone need not drive c k , t c_{k,t} c k , t to zero. More precisely, the coordinate has a finite limit
c k , ∞ = c k , t k ∏ s = t k ∞ ( 1 − η m k , s ) , c_{k,\infty}=c_{k,t_k}\prod_{s=t_k}^{\infty}(1-\eta m_{k,s}), c k , ∞ = c k , t k s = t k ∏ ∞ ( 1 − η m k , s ) , and this limit is nonzero whenever c k , t k ≠ 0 c_{k,t_k}\ne0 c k , t k = 0 and $0\le \eta m_{k,s}<1f o r a l l for all f or a l l s\ge t_k.
Proof. By Proposition 6, for t ≥ t k t \ge t_k t ≥ t k ,
η m k , s ≍ ( 1 + t − t s t a b ) − p k . \eta m_{k,s} \asymp(1+t-t_{\mathrm{stab}})^{-p_k}. η m k , s ≍ ( 1 + t − t stab ) − p k .
First suppose p k = 1 p_k=1 p k = 1 . Then there is a constant b k , − > 0 b_{k,-}>0 b k , − > 0 such that
η m k , s ≥ b k , − 1 + s − t s t a b ∀ s ≥ t k . \eta m_{k,s} \ge\frac{b_{k,-}}{1+s-t_{\mathrm{stab}}}
\qquad\forall s \ge t_k. η m k , s ≥ 1 + s − t stab b k , − ∀ s ≥ t k .
Therefore
∑ s = t k t − 1 η m k , s ≥ b k , − ∑ s = t k t − 1 1 1 + s − t s t a b ≥ b k , − log ( 1 + t − t s t a b 1 + t k − t s t a b ) . \sum_{s=t_k}^{t-1}\eta m_{k,s}
\ge b_{k,-}\sum_{s=t_k}^{t-1}\frac{1}{1+s-t_{\mathrm{stab}}}
\ge b_{k,-}\log\left(\frac{1+t-t_{\mathrm{stab}}}{1+t_k-t_{\mathrm{stab}}}\right). s = t k ∑ t − 1 η m k , s ≥ b k , − s = t k ∑ t − 1 1 + s − t stab 1 ≥ b k , − log ( 1 + t k − t stab 1 + t − t stab ) .
Substituting into (71) gives
∣ c k , t ∣ ≤ ∣ c k , t k ∣ ( 1 + t k − t s t a b 1 + t − t s t a b ) b k , − . |c_{k,t}| \le|c_{k,t_k}|\left(\frac{1+t_k-t_{\mathrm{stab}}}{1+t-t_{\mathrm{stab}}}\right)^{b_{k,-}}. ∣ c k , t ∣ ≤ ∣ c k , t k ∣ ( 1 + t − t stab 1 + t k − t stab ) b k , − .
Thus (72) holds with γ k = b k , − \gamma_k=b_{k,-} γ k = b k , − , and c k , t → 0 c_{k,t}\to0 c k , t → 0 .
Now suppose p k > 1 p_k>1 p k > 1 . Then
∑ s = t k ∞ ( 1 + s − t s t a b ) − p k < ∞ . \sum_{s=t_k}^{\infty}(1+s-t_{\mathrm{stab}})^{-p_k}<\infty. s = t k ∑ ∞ ( 1 + s − t stab ) − p k < ∞.
Using the upper bound in (70) , we get
∑ s = t k ∞ η m k , s < ∞ , \sum_{s=t_k}^{\infty}\eta m_{k,s}<\infty, s = t k ∑ ∞ η m k , s < ∞ ,
which proves (74) . The upper bound in (73) is immediate from nonexpansion:
∣ c k , t ∣ ≤ ∣ c k , t k ∣ . |c_{k,t}| \le|c_{k,t_k}|. ∣ c k , t ∣ ≤ ∣ c k , t k ∣.
For the lower bound, increase t k t_k t k if necessary so that 0 ≤ η m k , s ≤ 1 / 2 0\le\eta m_{k,s}\le1/2 0 ≤ η m k , s ≤ 1/2 for all s ≥ t k s\ge t_k s ≥ t k . Then
log ( 1 − η m k , s ) ≥ − 2 η m k , s . \log(1-\eta m_{k,s})\ge-2\eta m_{k,s}. log ( 1 − η m k , s ) ≥ − 2 η m k , s .
Thus, for t ≥ t k t\ge t_k t ≥ t k ,
∣ c k , t ∣ = ∣ c k , t k ∣ ∏ s = t k t − 1 ( 1 − η m k , s ) ≥ ∣ c k , t k ∣ exp ( − 2 ∑ s = t k t − 1 η m k , s ) ≥ ∣ c k , t k ∣ exp ( − 2 ∑ s = t k ∞ η m k , s ) . \begin{aligned}
|c_{k,t}|=|c_{k,t_k}|\prod_{s=t_k}^{t-1}(1-\eta m_{k,s}) \\
&\ge|c_{k,t_k}|\exp\left(-2\sum_{s=t_k}^{t-1}\eta m_{k,s}\right) \\
&\ge|c_{k,t_k}|\exp\left(-2\sum_{s=t_k}^{\infty}\eta m_{k,s}\right).
\end{aligned} ∣ c k , t ∣ = ∣ c k , t k ∣ s = t k ∏ t − 1 ( 1 − η m k , s ) ≥ ∣ c k , t k ∣ exp ( − 2 s = t k ∑ t − 1 η m k , s ) ≥ ∣ c k , t k ∣ exp ( − 2 s = t k ∑ ∞ η m k , s ) .
The final exponential is a strictly positive constant. This gives (73) with
C k , − : = exp ( − 2 ∑ s = t k ∞ η m k , s ) , C k , + : = 1. C_{k,-}:=\exp\left(-2\sum_{s=t_k}^{\infty}\eta m_{k,s}\right),
\qquad C_{k,+}:=1. C k , − := exp ( − 2 s = t k ∑ ∞ η m k , s ) , C k , + := 1.
Finally, since
∑ s = t k ∞ η m k , s < ∞ and 0 ≤ η m k , s < 1 , \sum_{s=t_k}^{\infty}\eta m_{k,s}<\infty
\qquad\text{and}\qquad
0\le\eta m_{k,s}<1, s = t k ∑ ∞ η m k , s < ∞ and 0 ≤ η m k , s < 1 ,
the infinite product
∏ s = t k ∞ ( 1 − η m k , s ) \prod_{s=t_k}^{\infty}(1-\eta m_{k,s}) s = t k ∏ ∞ ( 1 − η m k , s )
converges to a strictly positive number. Therefore
c k , t → c k , t k ∏ s = t k ∞ ( 1 − η m k , s ) , c_{k,t}\to c_{k,t_k}\prod_{s=t_k}^{\infty}(1-\eta m_{k,s}), c k , t → c k , t k s = t k ∏ ∞ ( 1 − η m k , s ) ,
which is nonzero whenever c k , t k ≠ 0 c_{k,t_k}\ne0 c k , t k = 0 .
Corollary C.1 (Spanning versus non-spanning component behavior). Under Assumption 1 , the stable tail separates the active coordinates into two classes:
U s v : = span { e k : j ( k ) = 1 } , U l a t e : = span { e k : j ( k ) > 1 } . U_{\mathrm{sv}} := \operatorname{span}\{e_k : j(k) = 1\}, \qquad U_{\mathrm{late}} := \operatorname{span}\{e_k : j(k) > 1\}. U sv := span { e k : j ( k ) = 1 } , U late := span { e k : j ( k ) > 1 } . Writing
θ t ⊥ = θ t ⊥ s v + θ t ⊥ l a t e , \theta_t^\perp= \theta_t^{\perp\,\mathrm{sv}} + \theta_t^{\perp\,\mathrm{late}}, θ t ⊥ = θ t ⊥ sv + θ t ⊥ late , one has
θ t ⊥ s v → 0 , \theta_t^{\perp\,\mathrm{sv}} \to0, θ t ⊥ sv → 0 , with coordinatewise polynomial rates. By contrast, each nonzero component of θ t ⊥ l a t e \theta_t^{\perp\,\mathrm{late}} θ t ⊥ late generally converges to a nonzero finite limit determined by the entry value and the finite post-stability contraction budget.
In particular, if j ∗ = 1 j_\ast= 1 j ∗ = 1 , then U l a t e = { 0 } U_{\mathrm{late}} = \{0\} U late = { 0 } , and the whole active normal component decays. If j ∗ > 1 j_\ast> 1 j ∗ > 1 , then U l a t e ≠ { 0 } U_{\mathrm{late}} \ne\{0\} U late = { 0 } , and the stable tail cannot in general remove the later-layer component.
Proof. The subspace U s v U_{\mathrm{sv}} U sv consists exactly of the coordinates with p k = 1 p_k = 1 p k = 1 , so Theorem 5(i) gives coordinatewise convergence to zero. The subspace U l a t e U_{\mathrm{late}} U late consists exactly of the coordinates with p k > 1 p_k > 1 p k > 1 , so Theorem 5(ii) gives a finite contraction budget and, generically, nonzero limiting coordinates.
If j ∗ = 1 j_\ast= 1 j ∗ = 1 , then the first layer spans all of U U U , so every coordinate has j ( k ) = 1 j(k) = 1 j ( k ) = 1 . Hence U l a t e = { 0 } U_{\mathrm{late}} = \{0\} U late = { 0 } . If j ∗ > 1 j_\ast> 1 j ∗ > 1 , some coordinate is not seen by the first layer, so U l a t e ≠ { 0 } U_{\mathrm{late}} \ne\{0\} U late = { 0 } .
Figure 5. Non-spanning configuration with α 2 = 4 \alpha_2 = 4 α 2 = 4 .
Figure 6. Non-spanning configuration with α 2 = 3 \alpha_2 = 3 α 2 = 3 .
Multiclass Cross-Entropy
Setup
The multiclass cross-entropy analysis mirrors the binary logistic case of Appendix C step for step. The main difference is that the orthogonal dynamics now have two sources. As in the binary high-dimensional model, shared off-axis components u i u_i u i create data-induced active directions. In addition, even in the collinear case u i = 0 u_i = 0 u i = 0 , the multi-column structure of the softmax weights creates nontrivial matrix-valued orthogonal directions inside the simplex span.
Data model. Fix K ≥ 3 K \ge3 K ≥ 3 and assume m ≥ K − 1 m \ge K - 1 m ≥ K − 1 . Let r 1 , … , r K ∈ R m r_1, \ldots, r_K \in\mathbb{R}^m r 1 , … , r K ∈ R m be the unit-norm vertices of a centered regular simplex:
⟨ r c , r d ⟩ = { 1 , c = d , − 1 K − 1 , c ≠ d , ∑ c = 1 K r c = 0. \langle r_c, r_d \rangle=
\begin{cases}
1, & c = d, \\
-\frac{1}{K-1}, & c \ne d,
\end{cases}
\qquad
\sum_{c=1}^{K} r_c = 0. ⟨ r c , r d ⟩ = { 1 , − K − 1 1 , c = d , c = d , c = 1 ∑ K r c = 0.
Let
S : = span { r 1 , … , r K } , dim S = K − 1. S := \operatorname{span}\{r_1, \ldots, r_K\}, \qquad\dim S = K - 1. S := span { r 1 , … , r K } , dim S = K − 1.
For each i = 1 , … , N i = 1, \ldots, N i = 1 , … , N , choose a margin scale b i > 0 b_i > 0 b i > 0 and a shared orthogonal vector
u i ∈ S ⊥ . u_i \in S^\perp. u i ∈ S ⊥ .
For each class c ∈ { 1 , … , K } c \in\{1, \ldots, K\} c ∈ { 1 , … , K } , define
x c , i : = b i r c + u i , y c , i : = c . x_{c,i} := b_i r_c + u_i, \qquad y_{c,i} := c. x c , i := b i r c + u i , y c , i := c .
Thus each index i i i defines a full K K K -class orbit: the class-dependent component b i r c b_i r_c b i r c rotates across the simplex directions, while the orthogonal component u i u_i u i is shared across classes.
This is the multiclass analogue of the binary pair-symmetric construction
x i + = a i v + u i , x i − = − a i v + u i . x_i^+ = a_i v + u_i,\qquad x_i^- = -a_i v + u_i. x i + = a i v + u i , x i − = − a i v + u i .
The scalar b i b_i b i plays the role of the binary margin scale a i a_i a i , and the shared vector u i ∈ S ⊥ u_i \in S^\perp u i ∈ S ⊥ plays the same role as the binary off-axis component u i u_i u i . The collinear simplex model is recovered as the special case u i = 0 u_i = 0 u i = 0 for all i i i .
Parameterization and loss. We parameterize the classifier by a weight matrix W = [ w 1 , … , w K ] ∈ R m × K W = [w_1,\ldots,w_K] \in\mathbb{R}^{m \times K} W = [ w 1 , … , w K ] ∈ R m × K , working on the centered linear space
C : = { W ∈ R m × K : ∑ c = 1 K w c = 0 } , \mathcal{C} := \left\{W \in\mathbb{R}^{m \times K} : \sum_{c=1}^{K} w_c = 0\right\}, C := { W ∈ R m × K : c = 1 ∑ K w c = 0 } ,
equipped with the Frobenius inner product ⟨ A , B ⟩ F : = tr ( A ⊤ B ) \langle A,B\rangle_F := \operatorname{tr}(A^\top B) ⟨ A , B ⟩ F := tr ( A ⊤ B ) . Gradient descent reads
W t + 1 = W t − η ∇ L ( W t ) , W_{t+1} = W_t - \eta\nabla\mathcal{L}(W_t), W t + 1 = W t − η ∇ L ( W t ) ,
where the empirical cross-entropy loss is
L ( W ) : = 1 K N ∑ c = 1 K ∑ i = 1 N [ − ⟨ w c , x c , i ⟩ + log ∑ d = 1 K e ⟨ w d , x c , i ⟩ ] . (75) \mathcal{L}(W) := \frac{1}{KN}\sum_{c=1}^{K}\sum_{i=1}^{N}\left[-\langle w_c,x_{c,i}\rangle+ \log\sum_{d=1}^{K}e^{\langle w_d,x_{c,i}\rangle}\right]. \tag*{(75)} L ( W ) := K N 1 c = 1 ∑ K i = 1 ∑ N [ − ⟨ w c , x c , i ⟩ + log d = 1 ∑ K e ⟨ w d , x c , i ⟩ ] . (75)
Active slice and inactive directions. Let
U d a t a : = span { u 1 , … , u N } ⊆ S ⊥ , U_{\mathrm{data}} := \operatorname{span}\{u_1,\ldots,u_N\} \subseteq S^\perp, U data := span { u 1 , … , u N } ⊆ S ⊥ ,
and define the full data span
E : = S ⊕ U d a t a . E := S \oplus U_{\mathrm{data}}. E := S ⊕ U data .
Since every data point has the form
x c , i = b i r c + u i ∈ E , x_{c,i} = b_i r_c + u_i \in E, x c , i = b i r c + u i ∈ E ,
the loss depends on each weight vector w d w_d w d only through its projection onto E E E . We decompose the centered parameter space C \mathcal{C} C orthogonally as
C = A ⊕ I , \mathcal{C} = \mathcal{A} \oplus\mathcal{I}, C = A ⊕ I ,
where
A : = { W ∈ C : w c ∈ E ∀ c } , I : = { W ∈ C : w c ∈ E ⊥ ∀ c } . (76) \mathcal{A} := \{W \in\mathcal{C} : w_c \in E\ \forall c\},\qquad\mathcal{I} := \{W \in\mathcal{C} : w_c \in E^\perp\ \forall c\}. \tag*{(76)} A := { W ∈ C : w c ∈ E ∀ c } , I := { W ∈ C : w c ∈ E ⊥ ∀ c } . (76)
The space A \mathcal{A} A is the active slice: it consists of centered class-weight matrices whose columns lie in the span of the data. The space I \mathcal{I} I contains the inactive directions, which are orthogonal to all data vectors and hence invisible to the loss.
Proposition 7 (Inactive directions are frozen). For every W ∈ C W \in\mathcal{C} W ∈ C , one has
∇ L ( W ) ∈ A . \nabla\mathcal{L}(W) \in\mathcal{A}. ∇ L ( W ) ∈ A . Consequently, writing
W t = W t a c t + W t i n a c t , W t a c t ∈ A , W t i n a c t ∈ I , W_t = W_t^{\mathrm{act}} + W_t^{\mathrm{inact}},\qquad W_t^{\mathrm{act}} \in\mathcal{A},\qquad W_t^{\mathrm{inact}} \in\mathcal{I}, W t = W t act + W t inact , W t act ∈ A , W t inact ∈ I , gradient descent freezes the inactive component:
W t + 1 i n a c t = W t i n a c t ∀ t ≥ 0. W_{t+1}^{\mathrm{inact}} = W_t^{\mathrm{inact}}\qquad\forall t \ge0. W t + 1 inact = W t inact ∀ t ≥ 0.
Proof. Fix a sample ( x c , i , c ) (x_{c,i},c) ( x c , i , c ) . For this sample, the cross-entropy term is
− ⟨ w c , x c , i ⟩ + log ( ∑ e = 1 K e ⟨ w e , x c , i ⟩ ) . -\langle w_c,x_{c,i}\rangle+ \log\left(\sum_{e=1}^{K}e^{\langle w_e,x_{c,i}\rangle}\right). − ⟨ w c , x c , i ⟩ + log ( e = 1 ∑ K e ⟨ w e , x c , i ⟩ ) .
Its gradient with respect to the d d d -th weight column is
∂ w d = ( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i , \partial_{w_d} = \left(p_{d\mid c,i}(W) - \mathbb{1}_{\{d=c\}}\right)x_{c,i}, ∂ w d = ( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i ,
where
p d ∣ c , i ( W ) : = e ⟨ w d , x c , i ⟩ ∑ e = 1 K e ⟨ w e , x c , i ⟩ . p_{d|c,i}(W) := \frac{e^{\langle w_d,x_{c,i}\rangle}}{\sum_{e=1}^{K} e^{\langle w_e,x_{c,i}\rangle}}. p d ∣ c , i ( W ) := ∑ e = 1 K e ⟨ w e , x c , i ⟩ e ⟨ w d , x c , i ⟩ .
Since x c , i = b i r c + u i ∈ E x_{c,i}=b_i r_c+u_i\in E x c , i = b i r c + u i ∈ E , every column of this sample gradient belongs to E E E . Moreover, summing the sample gradient over columns gives
∑ d = 1 K ( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i = ( ∑ d = 1 K p d ∣ c , i ( W ) − 1 ) x c , i = 0. \sum_{d=1}^{K}\left(p_{d|c,i}(W)-\mathbb{1}_{\{d=c\}}\right)x_{c,i}
=\left(\sum_{d=1}^{K}p_{d|c,i}(W)-1\right)x_{c,i}=0. d = 1 ∑ K ( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i = ( d = 1 ∑ K p d ∣ c , i ( W ) − 1 ) x c , i = 0.
Thus the sample gradient is centered and has all columns in E E E . Hence it belongs to A \mathcal{A} A . Averaging over c c c and i i i gives
∇ L ( W ) ∈ A . \nabla\mathcal{L}(W)\in\mathcal{A}. ∇ L ( W ) ∈ A .
Therefore the I \mathcal{I} I -component of gradient descent is unchanged:
W t + 1 i n a c t = Π I ( W t − η ∇ L ( W t ) ) = W t i n a c t , W_{t+1}^{\mathrm{inact}}=\Pi_{\mathcal{I}}\left(W_t-\eta\nabla\mathcal{L}(W_t)\right)=W_t^{\mathrm{inact}}, W t + 1 inact = Π I ( W t − η ∇ L ( W t ) ) = W t inact ,
because ∇ L ( W t ) ∈ A ⊥ I \nabla\mathcal{L}(W_t)\in\mathcal{A}\perp\mathcal{I} ∇ L ( W t ) ∈ A ⊥ I .
The loss depends only on the active component W a c t W^{\mathrm{act}} W act . Indeed, for every inactive column w d i n a c t ∈ E ⊥ w_d^{\mathrm{inact}}\in E^\perp w d inact ∈ E ⊥ and every data point x c , i ∈ E x_{c,i}\in E x c , i ∈ E ,
⟨ w d i n a c t , x c , i ⟩ = 0. \langle w_d^{\mathrm{inact}},x_{c,i}\rangle=0. ⟨ w d inact , x c , i ⟩ = 0.
Hence
⟨ w d , x c , i ⟩ = ⟨ w d a c t , x c , i ⟩ . \langle w_d,x_{c,i}\rangle=\langle w_d^{\mathrm{act}},x_{c,i}\rangle. ⟨ w d , x c , i ⟩ = ⟨ w d act , x c , i ⟩ .
Combined with Proposition 7 , this means that the inactive component is dynamically irrelevant: it is fixed by gradient descent and never affects the loss. Thus, after subtracting the fixed inactive component, the nontrivial dynamics are completely described by the active variable W t a c t W_t^{\mathrm{act}} W t act . We therefore analyze the reduced dynamics on A \mathcal{A} A , equivalently assuming W 0 ∈ A W_0\in\mathcal{A} W 0 ∈ A .
Valley line and active subspace. Inside A \mathcal{A} A , we take the symmetric parallel direction to be the configuration in which each weight column points along its own class axis. Setting
R : = [ r 1 , … , r K ] ∈ R m × K , R := [r_1,\ldots,r_K]\in\mathbb{R}^{m\times K}, R := [ r 1 , … , r K ] ∈ R m × K ,
we have R ∈ A R\in\mathcal{A} R ∈ A , since each r c ∈ S ⊂ E r_c\in S\subset E r c ∈ S ⊂ E and ∑ c = 1 K r c = 0 \sum_{c=1}^{K}r_c=0 ∑ c = 1 K r c = 0 . Moreover,
∥ R ∥ F 2 = ∑ c = 1 K ∥ r c ∥ 2 = K . \lVert R\rVert_F^2=\sum_{c=1}^{K}\lVert r_c\rVert^2=K. ∥ R ∥ F 2 = c = 1 ∑ K ∥ r c ∥ 2 = K .
Thus the unit-norm parallel generator is
v : = R K ∈ A , ∥ v ∥ F = 1. v:=\frac{R}{\sqrt{K}}\in\mathcal{A},\qquad\lVert v\rVert_F=1. v := K R ∈ A , ∥ v ∥ F = 1.
We define the parallel line and active orthogonal space by
M : = span { v } , U : = { Z ∈ A : ⟨ Z , v ⟩ F = 0 } . \mathcal{M}:=\operatorname{span}\{v\},\qquad
\mathcal{U}:=\{Z\in\mathcal{A}:\langle Z,v\rangle_F=0\}. M := span { v } , U := { Z ∈ A : ⟨ Z , v ⟩ F = 0 } .
Then
A = M ⊕ U \mathcal{A}=\mathcal{M}\oplus\mathcal{U} A = M ⊕ U
orthogonally. Every active iterate admits the canonical decomposition
W t = θ t ∥ v + Z t , θ t ∥ : = ⟨ W t , v ⟩ F , Z t ∈ U . (77) W_t=\theta_t^{\parallel}v+Z_t,\qquad
\theta_t^{\parallel}:=\langle W_t,v\rangle_F,\qquad
Z_t\in\mathcal{U}. \tag*{(77)} W t = θ t ∥ v + Z t , θ t ∥ := ⟨ W t , v ⟩ F , Z t ∈ U . (77)
This is the multiclass analogue of the binary decomposition θ = θ ∥ v + θ ⊥ \theta=\theta^{\parallel}v+\theta^\perp θ = θ ∥ v + θ ⊥ , with the matrix-valued active orthogonal space U \mathcal{U} U playing the role of the active normal space.
Furthermore, we can show that this subspace corresponds to the Crammer—Singer hard-margin max-margin direction of our symmetric data configuration.
Proposition 8 (Crammer–Singer max-margin direction). Consider the multiclass data model of Appendix D.1, and let
b min : = min 1 ≤ i ≤ N b i . b_{\min} := \min_{1 \le i \le N} b_i. b m i n := 1 ≤ i ≤ N min b i . Consider the centered Crammer–Singer hard-margin SVM problem
min W ∈ C 1 2 ∥ W ∥ F 2 subject to ⟨ w c − w d , x c , i ⟩ ≥ 1 ∀ i , ∀ c , ∀ d ≠ c . (78) \min_{W \in\mathcal{C}} \frac{1}{2}\lVert W\rVert_F^2 \quad\text{subject to} \quad\langle w_c-w_d,x_{c,i}\rangle\ge1 \quad\forall i,\ \forall c,\ \forall d \ne c. \tag*{(78)} W ∈ C min 2 1 ∥ W ∥ F 2 subject to ⟨ w c − w d , x c , i ⟩ ≥ 1 ∀ i , ∀ c , ∀ d = c . (78) Then its unique solution is
W S V M = K − 1 K b min R . (79) W_{\mathrm{SVM}}=\frac{K-1}{K b_{\min}}R. \tag*{(79)} W SVM = K b m i n K − 1 R . (79) Equivalently, since v = R / K v=R/\sqrt{K} v = R / K , the Crammer–Singer max-margin direction is v v v , and hence the max-margin line is precisely
M = span { v } . \mathcal{M}=\operatorname{span}\{v\}. M = span { v } . Proof. Let
α ∗ : = K − 1 K b min . \alpha_* := \frac{K-1}{K b_{\min}}. α ∗ := K b m i n K − 1 . We first show that W = α ∗ R W=\alpha_*R W = α ∗ R is feasible. For a sample x c , i = b i r c + u i x_{c,i}=b_i r_c+u_i x c , i = b i r c + u i and a competing class d ≠ c d\ne c d = c ,
⟨ α ∗ r c − α ∗ r d , b i r c + u i ⟩ = α ∗ b i ⟨ r c − r d , r c ⟩ + α ∗ ⟨ r c − r d , u i ⟩ . \langle\alpha_*r_c-\alpha_*r_d,b_i r_c+u_i\rangle
=\alpha_*b_i\langle r_c-r_d,r_c\rangle+\alpha_*\langle r_c-r_d,u_i\rangle. ⟨ α ∗ r c − α ∗ r d , b i r c + u i ⟩ = α ∗ b i ⟨ r c − r d , r c ⟩ + α ∗ ⟨ r c − r d , u i ⟩ . Since u i ∈ S ⊥ u_i\in S^\perp u i ∈ S ⊥ and r c − r d ∈ S r_c-r_d\in S r c − r d ∈ S , the second term vanishes. Using the simplex identities,
⟨ r c − r d , r c ⟩ = 1 − ( − 1 K − 1 ) = K K − 1 . \langle r_c-r_d,r_c\rangle
=1-\left(-\frac{1}{K-1}\right)
=\frac{K}{K-1}. ⟨ r c − r d , r c ⟩ = 1 − ( − K − 1 1 ) = K − 1 K . Therefore
⟨ α ∗ r c − α ∗ r d , x c , i ⟩ = α ∗ b i K K − 1 = b i b min ≥ 1. \langle\alpha_*r_c-\alpha_*r_d,x_{c,i}\rangle
=\alpha_*b_i\frac{K}{K-1}
=\frac{b_i}{b_{\min}}\ge1. ⟨ α ∗ r c − α ∗ r d , x c , i ⟩ = α ∗ b i K − 1 K = b m i n b i ≥ 1. Thus α ∗ R \alpha_*R α ∗ R is feasible.
It remains to prove optimality. Let W = [ w 1 , … , w K ] ∈ C W=[w_1,\ldots,w_K]\in\mathcal{C} W = [ w 1 , … , w K ] ∈ C be any feasible point. Choose an index i ∗ i_* i ∗ such that b i ∗ = b min b_{i_*}=b_{\min} b i ∗ = b m i n , and write u ∗ : = u i ∗ u_*:=u_{i_*} u ∗ := u i ∗ . For each fixed class c c c , feasibility gives
⟨ w c − w d , b min r c + u ∗ ⟩ ≥ 1 ∀ d ≠ c . \langle w_c-w_d,b_{\min}r_c+u_*\rangle\ge1
\qquad\forall d\ne c. ⟨ w c − w d , b m i n r c + u ∗ ⟩ ≥ 1 ∀ d = c . Summing over all d ≠ c d\ne c d = c , we obtain
⟨ ∑ d ≠ c ( w c − w d ) , b min r c + u ∗ ⟩ ≥ K − 1. \left\langle\sum_{d\ne c}(w_c-w_d),b_{\min}r_c+u_*\right\rangle\ge K-1. ⟨ d = c ∑ ( w c − w d ) , b m i n r c + u ∗ ⟩ ≥ K − 1. Since W ∈ C W\in\mathcal{C} W ∈ C , we have ∑ d = 1 K w d = 0 \sum_{d=1}^{K}w_d=0 ∑ d = 1 K w d = 0 , and hence
∑ d ≠ c ( w c − w d ) = K w c . \sum_{d\ne c}(w_c-w_d)=Kw_c. d = c ∑ ( w c − w d ) = K w c . Thus
K ⟨ w c , b min r c + u ∗ ⟩ ≥ K − 1 , K\langle w_c,b_{\min}r_c+u_*\rangle\ge K-1, K ⟨ w c , b m i n r c + u ∗ ⟩ ≥ K − 1 , or equivalently
⟨ w c , b min r c + u ∗ ⟩ ≥ K − 1 K . \langle w_c,b_{\min}r_c+u_*\rangle\ge\frac{K-1}{K}. ⟨ w c , b m i n r c + u ∗ ⟩ ≥ K K − 1 . Summing this inequality over c = 1 , … , K c=1,\ldots,K c = 1 , … , K yields
∑ c = 1 K ⟨ w c , b min r c + u ∗ ⟩ ≥ K − 1. \sum_{c=1}^{K}\langle w_c,b_{\min}r_c+u_*\rangle\ge K-1. c = 1 ∑ K ⟨ w c , b m i n r c + u ∗ ⟩ ≥ K − 1. The shared off-axis term cancels because W ∈ C W\in\mathcal{C} W ∈ C :
∑ c = 1 K ⟨ w c , u ∗ ⟩ = ⟨ ∑ c = 1 K w c , u ∗ ⟩ = 0. \sum_{c=1}^{K}\langle w_c,u_*\rangle
=\left\langle\sum_{c=1}^{K}w_c,u_*\right\rangle=0. c = 1 ∑ K ⟨ w c , u ∗ ⟩ = ⟨ c = 1 ∑ K w c , u ∗ ⟩ = 0.
Therefore
b min ∑ c = 1 K ⟨ w c , r c ⟩ = b min ⟨ W , R ⟩ F ≥ K − 1. b_{\min}\sum_{c=1}^{K}\langle w_c,r_c\rangle=b_{\min}\langle W,R\rangle_F\ge K-1. b m i n c = 1 ∑ K ⟨ w c , r c ⟩ = b m i n ⟨ W , R ⟩ F ≥ K − 1.
Hence
⟨ W , R ⟩ F ≥ K − 1 b min . \langle W,R\rangle_F\ge\frac{K-1}{b_{\min}}. ⟨ W , R ⟩ F ≥ b m i n K − 1 .
By Cauchy–Schwarz and ∥ R ∥ F = K \lVert R\rVert_F=\sqrt{K} ∥ R ∥ F = K ,
∥ W ∥ F K = ∥ W ∥ F ∥ R ∥ F ≥ ⟨ W , R ⟩ F ≥ K − 1 b min . \lVert W\rVert_F\sqrt{K}=\lVert W\rVert_F\lVert R\rVert_F\ge\langle W,R\rangle_F\ge\frac{K-1}{b_{\min}}. ∥ W ∥ F K = ∥ W ∥ F ∥ R ∥ F ≥ ⟨ W , R ⟩ F ≥ b m i n K − 1 .
Thus every feasible W W W satisfies
∥ W ∥ F ≥ K − 1 K b min . \lVert W\rVert_F\ge\frac{K-1}{\sqrt{K}\,b_{\min}}. ∥ W ∥ F ≥ K b m i n K − 1 .
On the other hand,
∥ α ∗ R ∥ F = α ∗ K = K − 1 K b min K = K − 1 K b min . \lVert\alpha_*R\rVert_F=\alpha_*\sqrt{K}=\frac{K-1}{Kb_{\min}}\sqrt{K}=\frac{K-1}{\sqrt{K}\,b_{\min}}. ∥ α ∗ R ∥ F = α ∗ K = K b m i n K − 1 K = K b m i n K − 1 .
So α ∗ R \alpha_*R α ∗ R attains the minimum norm.
Finally, equality in Cauchy–Schwarz forces W W W to be a nonnegative scalar multiple of R R R . The margin constraints then force the scalar to be exactly α ∗ \alpha_* α ∗ . Hence the centered hard-margin SVM solution is unique and equals α ∗ R \alpha_*R α ∗ R .
Binary–multiclass dictionary. The setup above maps onto the binary case as follows.
Table 6.
Thus the one-dimensional restricted loss has the same form as in the binary case, with the effective margin scale
a ~ i : = κ b i K \widetilde{a}_i := \frac{\kappa b_i}{\sqrt{K}} a i := K κ b i
replacing a i a_i a i in the restricted loss. The next subsection computes the corresponding active sharpness operator H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) , whose block structure reflects both the simplex-span matrix directions and the shared orthogonal features u i u_i u i .
Closed-form subspace restricted objects
We now compute the subspace-restricted loss, parallel drift, and active sharpness operator for the multiclass simplex model with shared orthogonal features. Throughout this subsection, recall that
v = R K , R = [ r 1 , … , r K ] , κ : = K K − 1 . v = \frac{R}{\sqrt{K}}, \qquad R = [r_1,\ldots,r_K], \qquad\kappa:= \frac{K}{K-1}. v = K R , R = [ r 1 , … , r K ] , κ := K − 1 K .
For s ∈ R s \in\mathbb{R} s ∈ R , define the one-dimensional multiclass profile
ϕ K ( s ) : = log ( 1 + ( K − 1 ) e − s ) , q K ( s ) : = K − 1 e s + K − 1 = − ϕ K ′ ( s ) . \phi_K(s) := \log\left(1+(K-1)e^{-s}\right), \qquad q_K(s) := \frac{K-1}{e^s+K-1} = -\phi_K'(s). ϕ K ( s ) := log ( 1 + ( K − 1 ) e − s ) , q K ( s ) := e s + K − 1 K − 1 = − ϕ K ′ ( s ) .
We also write
a ~ i : = κ b i K . \widetilde{a}_i := \frac{\kappa b_i}{\sqrt{K}}. a i := K κ b i .
For c ∈ { 1 , … , K } c \in\{1,\ldots,K\} c ∈ { 1 , … , K } and s ∈ R s \in\mathbb{R} s ∈ R , define the softmax probability vector p ( c ) ( s ) ∈ R K p^{(c)}(s) \in\mathbb{R}^K p ( c ) ( s ) ∈ R K by
p d ( c ) ( s ) : = { 1 − q K ( s ) , d = c , q K ( s ) K − 1 , d ≠ c . p_d^{(c)}(s) :=
\begin{cases}
1-q_K(s), & d=c,\\
\frac{q_K(s)}{K-1}, & d\ne c.
\end{cases} p d ( c ) ( s ) := { 1 − q K ( s ) , K − 1 q K ( s ) , d = c , d = c .
Equivalently, p ( c ) ( s ) p^{(c)}(s) p ( c ) ( s ) is the softmax distribution obtained when the correct class logit exceeds each incorrect class logit by s s s .
For a vector p ∈ R K p \in\mathbb{R}^K p ∈ R K with nonnegative entries summing to one, let
Cov ( p ) : = Diag ( p ) − p p T . \operatorname{Cov}(p) := \operatorname{Diag}(p)-pp^{\mathsf{T}}. Cov ( p ) := Diag ( p ) − p p T .
Thus for h ∈ R K h \in\mathbb{R}^K h ∈ R K ,
h T Cov ( p ) h = ∑ d = 1 K p d h d 2 − ( ∑ d = 1 K p d h d ) 2 = Var d ∼ p ( h d ) . h^{\mathsf{T}}\operatorname{Cov}(p)h
= \sum_{d=1}^{K}p_dh_d^2-\left(\sum_{d=1}^{K}p_dh_d\right)^2
= \operatorname{Var}_{d\sim p}(h_d). h T Cov ( p ) h = d = 1 ∑ K p d h d 2 − ( d = 1 ∑ K p d h d ) 2 = Var d ∼ p ( h d ) .
Proposition 9 (Closed-form restricted objects). For the multiclass cross-entropy loss (76), restricted to the active slice A \mathcal{A} A , the following identities hold.
(a) Valley-restricted loss. For every θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R ,
ℓ ( θ ∥ ) : = L ( θ ∥ v ) = 1 N ∑ i = 1 N ϕ K ( a ~ i θ ∥ ) . (80) \ell(\theta^{\parallel}) := \mathcal{L}(\theta^{\parallel}v)
= \frac{1}{N}\sum_{i=1}^{N}\phi_K\left(\widetilde{a}_i\theta^{\parallel}\right). \tag*{(80)} ℓ ( θ ∥ ) := L ( θ ∥ v ) = N 1 i = 1 ∑ N ϕ K ( a i θ ∥ ) . (80) (b) Valley drift. For every θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R ,
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a ~ i q K ( a ~ i θ ∥ ) . (81) -\ell'(\theta^{\parallel})
= \frac{1}{N}\sum_{i=1}^{N}\widetilde{a}_i q_K\left(\widetilde{a}_i\theta^{\parallel}\right). \tag*{(81)} − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q K ( a i θ ∥ ) . (81) In particular, − ℓ ′ ( θ ∥ ) > 0 -\ell'(\theta^{\parallel})>0 − ℓ ′ ( θ ∥ ) > 0 for all θ ∥ \theta^{\parallel} θ ∥ .
(c) Gradient alignment along the parallel direction. For every θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R ,
∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v . (82) \nabla\mathcal{L}(\theta^{\parallel}v)=\ell'(\theta^{\parallel})v. \tag*{(82)} ∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v . (82) Equivalently, the active orthogonal gradient vanishes along the parallel direction:
Π U ∇ L ( θ ∥ v ) = 0. \Pi_U\nabla\mathcal{L}(\theta^{\parallel}v)=0. Π U ∇ L ( θ ∥ v ) = 0.
(d) Active sharpness operator. Let
H ( θ ∥ ) : = Π U ∇ 2 L ( θ ∥ v ) Π U H(\theta^{\parallel}) := \Pi_{\mathcal{U}}\nabla^2\mathcal{L}(\theta^{\parallel}v)\Pi_{\mathcal{U}} H ( θ ∥ ) := Π U ∇ 2 L ( θ ∥ v ) Π U
be the Hessian restricted to the active orthogonal space. Then for every Z ∈ U Z \in\mathcal{U} Z ∈ U ,
⟨ Z , H ( θ ∥ ) Z ⟩ F = 1 K N ∑ i = 1 N ∑ c = 1 K h c , i ( Z ) ⊤ Cov ( p ( c ) ( a ~ i θ ∥ ) ) h c , i ( Z ) , (83) \langle Z,H(\theta^{\parallel})Z\rangle_F=\frac{1}{KN}\sum_{i=1}^{N}\sum_{c=1}^{K}h_{c,i}(Z)^{\top}\operatorname{Cov}\left(p^{(c)}(\widetilde{a}_i\theta^{\parallel})\right)h_{c,i}(Z), \tag*{(83)} ⟨ Z , H ( θ ∥ ) Z ⟩ F = K N 1 i = 1 ∑ N c = 1 ∑ K h c , i ( Z ) ⊤ Cov ( p ( c ) ( a i θ ∥ ) ) h c , i ( Z ) , (83)
where
h c , i ( Z ) : = [ ⟨ z 1 , x c , i ⟩ ⋮ ⟨ z K , x c , i ⟩ ] ∈ R K , Z = [ z 1 , … , z K ] . h_{c,i}(Z):=\begin{bmatrix}\langle z_1,x_{c,i}\rangle\\ \vdots\\ \langle z_K,x_{c,i}\rangle\end{bmatrix}\in\mathbb{R}^{K},\qquad Z=[z_1,\ldots,z_K]. h c , i ( Z ) := ⟨ z 1 , x c , i ⟩ ⋮ ⟨ z K , x c , i ⟩ ∈ R K , Z = [ z 1 , … , z K ] .
In particular, H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) is symmetric positive semidefinite on U \mathcal{U} U .
(e) Derivative of the active sharpness. For every Z ∈ U Z \in\mathcal{U} Z ∈ U ,
⟨ Z , H ′ ( θ ∥ ) Z ⟩ F = 1 K N ∑ i = 1 N ∑ c = 1 K a ~ i h c , i ( Z ) ⊤ Cov ′ ( p ( c ) ( a ~ i θ ∥ ) ) h c , i ( Z ) , (84) \langle Z,H'(\theta^{\parallel})Z\rangle_F=\frac{1}{KN}\sum_{i=1}^{N}\sum_{c=1}^{K}\widetilde{a}_i h_{c,i}(Z)^{\top}\operatorname{Cov}'\left(p^{(c)}(\widetilde{a}_i\theta^{\parallel})\right)h_{c,i}(Z), \tag*{(84)} ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F = K N 1 i = 1 ∑ N c = 1 ∑ K a i h c , i ( Z ) ⊤ Cov ′ ( p ( c ) ( a i θ ∥ ) ) h c , i ( Z ) , (84)
where Cov ′ ( p ( c ) ( s ) ) \operatorname{Cov}'(p^{(c)}(s)) Cov ′ ( p ( c ) ( s )) denotes the derivative with respect to the scalar margin variable s s s :
Cov ′ ( p ( c ) ( s ) ) = Diag ( ( p ( c ) ) ′ ( s ) ) − ( p ( c ) ) ′ ( s ) p ( c ) ( s ) ⊤ − p ( c ) ( s ) ( ( p ( c ) ) ′ ( s ) ) ⊤ , \operatorname{Cov}'(p^{(c)}(s))=\operatorname{Diag}\left((p^{(c)})'(s)\right)-(p^{(c)})'(s)p^{(c)}(s)^{\top}-p^{(c)}(s)((p^{(c)})'(s))^{\top}, Cov ′ ( p ( c ) ( s )) = Diag ( ( p ( c ) ) ′ ( s ) ) − ( p ( c ) ) ′ ( s ) p ( c ) ( s ) ⊤ − p ( c ) ( s ) (( p ( c ) ) ′ ( s ) ) ⊤ ,
with
( p d ( c ) ) ′ ( s ) = { q K ( s ) ( 1 − q K ( s ) ) , d = c , − q K ( s ) ( 1 − q K ( s ) ) K − 1 , d ≠ c . (p_d^{(c)})'(s)=
\begin{cases}
q_K(s)(1-q_K(s)), & d=c,\\
-\dfrac{q_K(s)(1-q_K(s))}{K-1}, & d\ne c.
\end{cases} ( p d ( c ) ) ′ ( s ) = ⎩ ⎨ ⎧ q K ( s ) ( 1 − q K ( s )) , − K − 1 q K ( s ) ( 1 − q K ( s )) , d = c , d = c .
Proof. Fix i i i and c c c . Along the parallel direction,
W = θ ∥ v = θ ∥ K R , w d = θ ∥ K r d . W=\theta^{\parallel}v=\frac{\theta^{\parallel}}{\sqrt{K}}R,\qquad w_d=\frac{\theta^{\parallel}}{\sqrt{K}}r_d. W = θ ∥ v = K θ ∥ R , w d = K θ ∥ r d .
Since u i ∈ S ⊥ u_i\in S^{\perp} u i ∈ S ⊥ and r d ∈ S r_d\in S r d ∈ S , we have
⟨ r d , u i ⟩ = 0. \langle r_d,u_i\rangle=0. ⟨ r d , u i ⟩ = 0.
Therefore
⟨ w d , x c , i ⟩ = θ ∥ K ⟨ r d , b i r c + u i ⟩ = θ ∥ b i K ⟨ r d , r c ⟩ . \langle w_d,x_{c,i}\rangle=\frac{\theta^{\parallel}}{\sqrt{K}}\langle r_d,b_i r_c+u_i\rangle=\frac{\theta^{\parallel}b_i}{\sqrt{K}}\langle r_d,r_c\rangle. ⟨ w d , x c , i ⟩ = K θ ∥ ⟨ r d , b i r c + u i ⟩ = K θ ∥ b i ⟨ r d , r c ⟩ .
Using the simplex identities,
⟨ r d , r c ⟩ = { 1 , d = c , − 1 K − 1 , d ≠ c , \langle r_d,r_c\rangle=
\begin{cases}
1, & d=c,\\
-\dfrac{1}{K-1}, & d\ne c,
\end{cases} ⟨ r d , r c ⟩ = ⎩ ⎨ ⎧ 1 , − K − 1 1 , d = c , d = c ,
we obtain
⟨ w c , x c , i ⟩ = θ ∥ b i K , ⟨ w d , x c , i ⟩ = − θ ∥ b i ( K − 1 ) K ( d ≠ c ) . \langle w_c,x_{c,i}\rangle=\frac{\theta^{\parallel}b_i}{\sqrt{K}},\qquad\langle w_d,x_{c,i}\rangle=-\frac{\theta^{\parallel}b_i}{(K-1)\sqrt{K}}\quad(d\ne c). ⟨ w c , x c , i ⟩ = K θ ∥ b i , ⟨ w d , x c , i ⟩ = − ( K − 1 ) K θ ∥ b i ( d = c ) .
Thus the correct-class logit exceeds each incorrect-class logit by
θ ∥ b i K + θ ∥ b i ( K − 1 ) K = K b i ( K − 1 ) K θ ∥ = a ~ i θ ∥ . \frac{\theta^{\parallel}b_i}{\sqrt{K}}+\frac{\theta^{\parallel}b_i}{(K-1)\sqrt{K}}=\frac{K b_i}{(K-1)\sqrt{K}}\theta^{\parallel}=\widetilde{a}_i\theta^{\parallel}. K θ ∥ b i + ( K − 1 ) K θ ∥ b i = ( K − 1 ) K K b i θ ∥ = a i θ ∥ .
The per-sample cross-entropy is therefore
θ ∥ b i K + log ( e θ ∥ b i / K + ( K − 1 ) e − θ ∥ b i / ( ( K − 1 ) K ) ) . \frac{\theta^{\parallel}b_i}{\sqrt{K}}+\log\left(e^{\theta^{\parallel}b_i/\sqrt{K}}+(K-1)e^{-\theta^{\parallel}b_i/((K-1)\sqrt{K})}\right). K θ ∥ b i + log ( e θ ∥ b i / K + ( K − 1 ) e − θ ∥ b i / (( K − 1 ) K ) ) .
Factoring out e b i θ ∥ / K e^{b_i\theta^\parallel/\sqrt{K}} e b i θ ∥ / K gives
log ( 1 + ( K − 1 ) e − a ~ i θ ∥ ) = ϕ K ( a ~ i θ ∥ ) . \log\left(1+(K-1)e^{-\tilde{a}_i\theta^\parallel}\right)=\phi_K(\tilde{a}_i\theta^\parallel). log ( 1 + ( K − 1 ) e − a ~ i θ ∥ ) = ϕ K ( a ~ i θ ∥ ) .
Averaging over c c c and i i i proves (81). Differentiating yields
ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a ~ i ϕ K ′ ( a ~ i θ ∥ ) = − 1 N ∑ i = 1 N a ~ i q K ( a ~ i θ ∥ ) , \ell'(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}\tilde{a}_i\phi_K'(\tilde{a}_i\theta^\parallel)=-\frac{1}{N}\sum_{i=1}^{N}\tilde{a}_iq_K(\tilde{a}_i\theta^\parallel), ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a ~ i ϕ K ′ ( a ~ i θ ∥ ) = − N 1 i = 1 ∑ N a ~ i q K ( a ~ i θ ∥ ) ,
which proves (82).
We next prove the parallel-gradient alignment. For a sample ( x c , i , c ) (x_{c,i},c) ( x c , i , c ) , the gradient with respect to the d d d -th column is
( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i . \left(p_{d\mid c,i}(W)-\boldsymbol{1}_{\{d=c\}}\right)x_{c,i}. ( p d ∣ c , i ( W ) − 1 { d = c } ) x c , i .
Along the parallel direction, the softmax probabilities are precisely
p d ∣ c , i ( θ ∥ v ) = p d ( c ) ( a ~ i θ ∥ ) . p_{d\mid c,i}(\theta^\parallel v)=p_d^{(c)}(\tilde{a}_i\theta^\parallel). p d ∣ c , i ( θ ∥ v ) = p d ( c ) ( a ~ i θ ∥ ) .
Fix i i i and average the sample gradient over the K K K classes c c c . The u i u_i u i -component of the d d d -th column is proportional to
∑ c = 1 K ( p d ( c ) ( a ~ i θ ∥ ) − 1 { d = c } ) . \sum_{c=1}^{K}\left(p_d^{(c)}(\tilde{a}_i\theta^\parallel)-\boldsymbol{1}_{\{d=c\}}\right). c = 1 ∑ K ( p d ( c ) ( a ~ i θ ∥ ) − 1 { d = c } ) .
For c = d c=d c = d , this coefficient is ( 1 − q K ) − 1 = − q K (1-q_K)-1=-q_K ( 1 − q K ) − 1 = − q K , while for each c ≠ d c\ne d c = d it is q K / ( K − 1 ) q_K/(K-1) q K / ( K − 1 ) . Hence the sum is
− q K + ( K − 1 ) q K K − 1 = 0. -q_K+(K-1)\frac{q_K}{K-1}=0. − q K + ( K − 1 ) K − 1 q K = 0.
Thus all shared orthogonal components cancel on the full class orbit.
For the simplex component, again fixing d d d , the d d d -th gradient column contributed by radius b i b_i b i is
b i K ∑ c = 1 K ( p d ( c ) ( a ~ i θ ∥ ) − 1 { d = c } ) r c . \frac{b_i}{K}\sum_{c=1}^{K}\left(p_d^{(c)}(\tilde{a}_i\theta^\parallel)-\boldsymbol{1}_{\{d=c\}}\right)r_c. K b i c = 1 ∑ K ( p d ( c ) ( a ~ i θ ∥ ) − 1 { d = c } ) r c .
Using ∑ c ≠ d r c = − r d \sum_{c\ne d}r_c=-r_d ∑ c = d r c = − r d , this equals
b i K ( − q K r d + q K K − 1 ∑ c ≠ d r c ) = − b i q K K ( 1 + 1 K − 1 ) r d = − b i q K K − 1 r d . \frac{b_i}{K}\left(-q_Kr_d+\frac{q_K}{K-1}\sum_{c\ne d}r_c\right)
=-\frac{b_iq_K}{K}\left(1+\frac{1}{K-1}\right)r_d
=-\frac{b_iq_K}{K-1}r_d. K b i − q K r d + K − 1 q K c = d ∑ r c = − K b i q K ( 1 + K − 1 1 ) r d = − K − 1 b i q K r d .
Averaging over i i i , we obtain
∇ w d L ( θ ∥ v ) = − 1 N ∑ i = 1 N b i q K ( a ~ i θ ∥ ) K − 1 r d . \nabla_{w_d}\mathcal{L}(\theta^\parallel v)
=-\frac{1}{N}\sum_{i=1}^{N}\frac{b_iq_K(\tilde{a}_i\theta^\parallel)}{K-1}r_d. ∇ w d L ( θ ∥ v ) = − N 1 i = 1 ∑ N K − 1 b i q K ( a ~ i θ ∥ ) r d .
Since
v = 1 K [ r 1 , … , r K ] , v=\frac{1}{\sqrt{K}}[r_1,\ldots,r_K], v = K 1 [ r 1 , … , r K ] ,
and
ℓ ′ ( θ ∥ ) = − 1 N ∑ i = 1 N K b i K q K ( a ~ i θ ∥ ) = − 1 N ∑ i = 1 N K b i K − 1 q K ( a ~ i θ ∥ ) , \ell'(\theta^\parallel)=-\frac{1}{N}\sum_{i=1}^{N}\frac{Kb_i}{\sqrt{K}}q_K(\tilde{a}_i\theta^\parallel)
=-\frac{1}{N}\sum_{i=1}^{N}\frac{\sqrt{K}b_i}{K-1}q_K(\tilde{a}_i\theta^\parallel), ℓ ′ ( θ ∥ ) = − N 1 i = 1 ∑ N K K b i q K ( a ~ i θ ∥ ) = − N 1 i = 1 ∑ N K − 1 K b i q K ( a ~ i θ ∥ ) ,
we have
ℓ ′ ( θ ∥ ) v = [ − 1 N ∑ i = 1 N b i q K ( a ~ i θ ∥ ) K − 1 r 1 , … , − 1 N ∑ i = 1 N b i q K ( a ~ i θ ∥ ) K − 1 r K ] , \ell'(\theta^\parallel)v
=\left[-\frac{1}{N}\sum_{i=1}^{N}\frac{b_iq_K(\tilde{a}_i\theta^\parallel)}{K-1}r_1,\ldots,
-\frac{1}{N}\sum_{i=1}^{N}\frac{b_iq_K(\tilde{a}_i\theta^\parallel)}{K-1}r_K\right], ℓ ′ ( θ ∥ ) v = [ − N 1 i = 1 ∑ N K − 1 b i q K ( a ~ i θ ∥ ) r 1 , … , − N 1 i = 1 ∑ N K − 1 b i q K ( a ~ i θ ∥ ) r K ] ,
which proves (83). Since U ⊥ v \mathcal{U}\perp v U ⊥ v , this also gives Π U ∇ L ( θ ∥ v ) = 0 \Pi_{\mathcal{U}}\nabla\mathcal{L}(\theta^\parallel v)=0 Π U ∇ L ( θ ∥ v ) = 0 .
It remains to compute the Hessian. For a perturbation Z = [ z 1 , … , z K ] Z=[z_1,\ldots,z_K] Z = [ z 1 , … , z K ] , the logit perturbation vector for sample ( x c , i , c ) (x_{c,i},c) ( x c , i , c ) is
h c , i ( Z ) = ( ⟨ z 1 , x c , i ⟩ , … , ⟨ z K , x c , i ⟩ ) T . h_{c,i}(Z)=\left(\langle z_1,x_{c,i}\rangle,\ldots,\langle z_K,x_{c,i}\rangle\right)^{\mathsf{T}}. h c , i ( Z ) = ( ⟨ z 1 , x c , i ⟩ , … , ⟨ z K , x c , i ⟩ ) T .
The second directional derivative of the sample cross-entropy in the direction Z Z Z is the softmax covariance of these logit perturbations:
h c , i ( Z ) ⊤ Cov ( p ( c ) ( a ~ i θ ∥ ) ) h c , i ( Z ) . h_{c,i}(Z)^\top\operatorname{Cov}\left(p^{(c)}(\tilde a_i\theta^{\parallel})\right)h_{c,i}(Z). h c , i ( Z ) ⊤ Cov ( p ( c ) ( a ~ i θ ∥ ) ) h c , i ( Z ) .
Averaging over c c c and i i i , and restricting to Z ∈ U Z \in\mathcal{U} Z ∈ U , gives (83) . This also shows that H ( θ ∥ ) H(\theta^{\parallel}) H ( θ ∥ ) is symmetric positive semidefinite.
Finally, differentiating (83) with respect to θ ∥ \theta^{\parallel} θ ∥ gives (84) , because the only θ ∥ \theta^{\parallel} θ ∥ -dependence is through s i = a ~ i θ ∥ s_i=\tilde a_i\theta^{\parallel} s i = a ~ i θ ∥ . The derivative of p ( c ) p^{(c)} p ( c ) follows from
q K ′ ( s ) = − q K ( s ) ( 1 − q K ( s ) ) . q_K^{\prime}(s)=-q_K(s)(1-q_K(s)). q K ′ ( s ) = − q K ( s ) ( 1 − q K ( s )) .
This completes the proof.
□ \square □
Verification of the abstract assumptions
We now verify the constants entering the abstract pre- and post-stability theorems for the multiclass simplex model. The verification is completely parallel to the binary logistic case, except that the active sharpness is matrix-valued. Accordingly, the full contraction statement in the post-stability phase requires a tail conditioning assumption on the active sharpness operator.
Recall the effective margin scales
a ~ i : = κ b i K , κ : = K K − 1 . \tilde a_i:=\frac{\kappa b_i}{\sqrt{K}},\qquad\kappa:=\frac{K}{K-1}. a ~ i := K κ b i , κ := K − 1 K .
and define
a ~ min : = min 1 ≤ i ≤ N a ~ i , a ~ max : = max 1 ≤ i ≤ N a ~ i , I ∗ : = { i : a ~ i = a ~ min } . \tilde a_{\min}:=\min_{1\le i\le N}\tilde a_i,\qquad\tilde a_{\max}:=\max_{1\le i\le N}\tilde a_i,\qquad I_*:=\{i:\tilde a_i=\tilde a_{\min}\}. a ~ m i n := 1 ≤ i ≤ N min a ~ i , a ~ m a x := 1 ≤ i ≤ N max a ~ i , I ∗ := { i : a ~ i = a ~ m i n } .
We also define the maximum data norm
X max : = max 1 ≤ i ≤ N , 1 ≤ c ≤ K ∥ x c , i ∥ . X_{\max}:=\max_{\substack{1\le i\le N,\,1\le c\le K}}\lVert x_{c,i}\rVert. X m a x := 1 ≤ i ≤ N , 1 ≤ c ≤ K max ∥ x c , i ∥ .
Since x c , i = b i r c + u i x_{c,i}=b_i r_c+u_i x c , i = b i r c + u i with u i ⊥ r c u_i\perp r_c u i ⊥ r c and ∥ r c ∥ = 1 \lVert r_c\rVert=1 ∥ r c ∥ = 1 , this is equivalently
X max = max 1 ≤ i ≤ N b i 2 + ∥ u i ∥ 2 . X_{\max}=\max_{1\le i\le N}\sqrt{b_i^2+\lVert u_i\rVert^2}. X m a x = 1 ≤ i ≤ N max b i 2 + ∥ u i ∥ 2 .
We use the multiclass profile
ϕ K ( s ) : = log ( 1 + ( K − 1 ) e − s ) , q K ( s ) : = K − 1 e s + K − 1 . \phi_K(s):=\log(1+(K-1)e^{-s}),\qquad q_K(s):=\frac{K-1}{e^s+K-1}. ϕ K ( s ) := log ( 1 + ( K − 1 ) e − s ) , q K ( s ) := e s + K − 1 K − 1 .
Its curvature satisfies
ϕ K ′ ′ ( s ) = ( K − 1 ) e s ( e s + K − 1 ) 2 , \phi_K^{\prime\prime}(s)=\frac{(K-1)e^s}{(e^s+K-1)^2}, ϕ K ′′ ( s ) = ( e s + K − 1 ) 2 ( K − 1 ) e s ,
and
− d d s log ϕ K ′ ′ ( s ) = tanh ( s − log ( K − 1 ) 2 ) . -\frac{d}{ds}\log\phi_K^{\prime\prime}(s)=\tanh\left(\frac{s-\log(K-1)}{2}\right). − d s d log ϕ K ′′ ( s ) = tanh ( 2 s − log ( K − 1 ) ) .
Thus the curvature is decreasing once s ≥ log ( K − 1 ) s\ge\log(K-1) s ≥ log ( K − 1 ) .
For the oscillation slab, fix r > 0 r>0 r > 0 , an initial value θ 0 ∥ \theta_0^{\parallel} θ 0 ∥ , and the corrected stability anchor
θ ∗ ( r ) : = inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } . \theta_*^{(r)}:=\inf\left\{\theta^{\parallel}\ge\theta_0^{\parallel}:\eta(1+\delta_r)\lambda_1(\theta^{\parallel})\le2\right\}. θ ∗ ( r ) := inf { θ ∥ ≥ θ 0 ∥ : η ( 1 + δ r ) λ 1 ( θ ∥ ) ≤ 2 } .
where
λ 1 ( θ ∥ ) : = λ max ( H ( θ ∥ ) ∣ U ) . \lambda_1(\theta^{\parallel}):=\lambda_{\max}\left(H(\theta^{\parallel})|_{\mathcal U}\right). λ 1 ( θ ∥ ) := λ m a x ( H ( θ ∥ ) ∣ U ) .
Assume throughout this verification that the oscillation slab lies to the right of the multiclass curvature threshold:
a ~ min θ 0 ∥ ≥ log ( K − 1 ) . (85) \tilde a_{\min}\theta_0^{\parallel}\ge\log(K-1). \tag*{(85)} a ~ m i n θ 0 ∥ ≥ log ( K − 1 ) . (85)
Proposition 10 (Subspace alignment and parallel expansion). Let
G U ( θ ∥ , Z ) : = Π U ∇ L ( θ ∥ v + Z ) , g ∥ ( θ ∥ , Z ) : = ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F , G_U(\theta^{\parallel}, Z) := \Pi_U\nabla\mathcal{L}(\theta^{\parallel}v + Z), \qquad g_{\parallel}(\theta^{\parallel}, Z) := \langle\nabla\mathcal{L}(\theta^{\parallel}v + Z), v\rangle_F, G U ( θ ∥ , Z ) := Π U ∇ L ( θ ∥ v + Z ) , g ∥ ( θ ∥ , Z ) := ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F , for θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R and Z ∈ U Z \in\mathcal{U} Z ∈ U . Then the following hold.
(a) Valley alignment. For every θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R ,
∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v . \nabla\mathcal{L}(\theta^{\parallel}v) = \ell'(\theta^{\parallel})v. ∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v . Equivalently,
G U ( θ ∥ , 0 ) = 0. G_U(\theta^{\parallel},0) = 0. G U ( θ ∥ , 0 ) = 0. (b) Active orthogonal Jacobian along the parallel direction. For every θ ∥ ∈ R \theta^{\parallel} \in\mathbb{R} θ ∥ ∈ R ,
D Z G U ( θ ∥ , 0 ) = H ( θ ∥ ) , D_ZG_U(\theta^{\parallel},0) = H(\theta^{\parallel}), D Z G U ( θ ∥ , 0 ) = H ( θ ∥ ) , where
H ( θ ∥ ) : = Π U ∇ 2 L ( θ ∥ v ) Π U . H(\theta^{\parallel}) := \Pi_U\nabla^2\mathcal{L}(\theta^{\parallel}v)\Pi_U. H ( θ ∥ ) := Π U ∇ 2 L ( θ ∥ v ) Π U . (c) Parallel Taylor expansion. Fix a compact interval J ⊂ R J \subset\mathbb{R} J ⊂ R and r > 0 r > 0 r > 0 . Then there exists a finite constant c ∥ , r , J < ∞ c_{\parallel,r,J} < \infty c ∥ , r , J < ∞ such that, for every θ ∥ ∈ J \theta^{\parallel} \in J θ ∥ ∈ J and every Z ∈ U Z \in\mathcal{U} Z ∈ U with ∥ Z ∥ F ≤ r \lVert Z\rVert_F \le r ∥ Z ∥ F ≤ r ,
g ∥ ( θ ∥ , Z ) = ℓ ′ ( θ ∥ ) + 1 2 ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F + R ∥ ( θ ∥ , Z ) , (86) g_{\parallel}(\theta^{\parallel}, Z) = \ell'(\theta^{\parallel}) + \frac{1}{2}\langle Z,H'(\theta^{\parallel})Z\rangle_F + R_{\parallel}(\theta^{\parallel}, Z), \tag*{(86)} g ∥ ( θ ∥ , Z ) = ℓ ′ ( θ ∥ ) + 2 1 ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F + R ∥ ( θ ∥ , Z ) , (86) where
∣ R ∥ ( θ ∥ , Z ) ∣ ≤ c ∥ , r , J ∥ Z ∥ F 3 . (87) \lvert R_{\parallel}(\theta^{\parallel}, Z)\rvert\le c_{\parallel,r,J}\lVert Z\rVert_F^3. \tag*{(87)} ∣ R ∥ ( θ ∥ , Z )∣ ≤ c ∥ , r , J ∥ Z ∥ F 3 . (87)
Proof. Part (a) is exactly the gradient-alignment identity from Proposition 9 . In particular,
Π U ∇ L ( θ ∥ v ) = Π U ( ℓ ′ ( θ ∥ ) v ) = 0 , \Pi_U\nabla\mathcal{L}(\theta^{\parallel}v) = \Pi_U(\ell'(\theta^{\parallel})v) = 0, Π U ∇ L ( θ ∥ v ) = Π U ( ℓ ′ ( θ ∥ ) v ) = 0 ,
because $\mathcal{U} \perp v.
For part (b), differentiate
G U ( θ ∥ , Z ) = Π U ∇ L ( θ ∥ v + Z ) G_U(\theta^{\parallel}, Z) = \Pi_U\nabla\mathcal{L}(\theta^{\parallel}v + Z) G U ( θ ∥ , Z ) = Π U ∇ L ( θ ∥ v + Z )
with respect to Z Z Z at Z = 0 Z = 0 Z = 0 . Since Π U \Pi_U Π U is constant and Z ∈ U Z \in\mathcal{U} Z ∈ U , this gives
D Z G U ( θ ∥ , 0 ) = Π U ∇ 2 L ( θ ∥ v ) Π U = H ( θ ∥ ) . D_ZG_U(\theta^{\parallel},0) = \Pi_U\nabla^2\mathcal{L}(\theta^{\parallel}v)\Pi_U = H(\theta^{\parallel}). D Z G U ( θ ∥ , 0 ) = Π U ∇ 2 L ( θ ∥ v ) Π U = H ( θ ∥ ) .
It remains to prove the parallel expansion. Fix θ ∥ ∈ J \theta^{\parallel} \in J θ ∥ ∈ J and define
F θ ∥ ( Z ) : = g ∥ ( θ ∥ , Z ) = ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F , Z ∈ U . F_{\theta^{\parallel}}(Z) := g_{\parallel}(\theta^{\parallel}, Z) = \langle\nabla\mathcal{L}(\theta^{\parallel}v + Z),v\rangle_F,\qquad Z \in\mathcal{U}. F θ ∥ ( Z ) := g ∥ ( θ ∥ , Z ) = ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F , Z ∈ U .
Since the cross-entropy loss is smooth, F θ ∥ F_{\theta^{\parallel}} F θ ∥ is smooth in Z Z Z . Taylor’s theorem at Z = 0 Z = 0 Z = 0 gives
F θ ∥ ( Z ) = F θ ∥ ( 0 ) + D Z F θ ∥ ( 0 ) [ Z ] + 1 2 D Z 2 F θ ∥ ( 0 ) [ Z , Z ] + R ∥ ( θ ∥ , Z ) . F_{\theta^{\parallel}}(Z) = F_{\theta^{\parallel}}(0) + D_ZF_{\theta^{\parallel}}(0)[Z] + \frac{1}{2}D_Z^2F_{\theta^{\parallel}}(0)[Z,Z] + R_{\parallel}(\theta^{\parallel},Z). F θ ∥ ( Z ) = F θ ∥ ( 0 ) + D Z F θ ∥ ( 0 ) [ Z ] + 2 1 D Z 2 F θ ∥ ( 0 ) [ Z , Z ] + R ∥ ( θ ∥ , Z ) .
The zeroth-order term is
F θ ∥ ( 0 ) = ⟨ ∇ L ( θ ∥ v ) , v ⟩ F = ℓ ′ ( θ ∥ ) . F_{\theta^{\parallel}}(0) = \langle\nabla\mathcal{L}(\theta^{\parallel}v),v\rangle_F = \ell'(\theta^{\parallel}). F θ ∥ ( 0 ) = ⟨ ∇ L ( θ ∥ v ) , v ⟩ F = ℓ ′ ( θ ∥ ) .
The first-order term vanishes. Indeed,
D Z F θ ∥ ( 0 ) [ Z ] = ⟨ ∇ 2 L ( θ ∥ v ) Z , v ⟩ F = ⟨ Z , ∇ 2 L ( θ ∥ v ) v ⟩ F . D_ZF_{\theta^{\parallel}}(0)[Z] = \langle\nabla^2\mathcal{L}(\theta^{\parallel}v)Z,v\rangle_F = \langle Z,\nabla^2\mathcal{L}(\theta^{\parallel}v)v\rangle_F. D Z F θ ∥ ( 0 ) [ Z ] = ⟨ ∇ 2 L ( θ ∥ v ) Z , v ⟩ F = ⟨ Z , ∇ 2 L ( θ ∥ v ) v ⟩ F .
But
∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v . \nabla\mathcal{L}(\theta^{\parallel}v) = \ell'(\theta^{\parallel})v. ∇ L ( θ ∥ v ) = ℓ ′ ( θ ∥ ) v .
Differentiating this identity with respect to θ ∥ \theta^{\parallel} θ ∥ gives
∇ 2 L ( θ ∥ v ) v = ℓ ′ ′ ( θ ∥ ) v . \nabla^2\mathcal{L}(\theta^{\parallel}v)v = \ell''(\theta^{\parallel})v. ∇ 2 L ( θ ∥ v ) v = ℓ ′′ ( θ ∥ ) v .
Since Z ∈ U ⊥ v Z \in\mathcal{U} \perp v Z ∈ U ⊥ v , it follows that
D Z F θ ∥ ( 0 ) [ Z ] = 0. D_ZF_{\theta^{\parallel}}(0)[Z] = 0. D Z F θ ∥ ( 0 ) [ Z ] = 0.
::: For the quadratic term, note that
D Z 2 F θ ∥ ( 0 ) [ Z , Z ] = ⟨ D 3 L ( θ ∥ v ) [ Z , Z ] , v ⟩ F . D_Z^2F_{\theta^{\|}}(0)[Z,Z]=\left\langle D^3\mathcal{L}(\theta^{\|}v)[Z,Z],v\right\rangle_F. D Z 2 F θ ∥ ( 0 ) [ Z , Z ] = ⟨ D 3 L ( θ ∥ v ) [ Z , Z ] , v ⟩ F .
On the other hand,
H ( θ ∥ ) = Π U ∇ 2 L ( θ ∥ v ) Π U . H(\theta^{\|})=\Pi_U\nabla^2\mathcal{L}(\theta^{\|}v)\Pi_U. H ( θ ∥ ) = Π U ∇ 2 L ( θ ∥ v ) Π U .
Therefore, for Z ∈ U Z\in U Z ∈ U ,
⟨ Z , H ( θ ∥ ) Z ⟩ F = ⟨ Z , ∇ 2 L ( θ ∥ v ) Z ⟩ F . \left\langle Z,H(\theta^{\|})Z\right\rangle_F=\left\langle Z,\nabla^2\mathcal{L}(\theta^{\|}v)Z\right\rangle_F. ⟨ Z , H ( θ ∥ ) Z ⟩ F = ⟨ Z , ∇ 2 L ( θ ∥ v ) Z ⟩ F .
Differentiating with respect to θ ∥ \theta^{\|} θ ∥ yields
⟨ Z , H ′ ( θ ∥ ) Z ⟩ F = ⟨ D 3 L ( θ ∥ v ) [ v , Z ] , Z ⟩ F = ⟨ D 3 L ( θ ∥ v ) [ Z , Z ] , v ⟩ F , \left\langle Z,H'(\theta^{\|})Z\right\rangle_F=\left\langle D^3\mathcal{L}(\theta^{\|}v)[v,Z],Z\right\rangle_F=\left\langle D^3\mathcal{L}(\theta^{\|}v)[Z,Z],v\right\rangle_F, ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F = ⟨ D 3 L ( θ ∥ v ) [ v , Z ] , Z ⟩ F = ⟨ D 3 L ( θ ∥ v ) [ Z , Z ] , v ⟩ F ,
where the last equality uses symmetry of the third derivative tensor. Thus
D Z 2 F θ ∥ ( 0 ) [ Z , Z ] = ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F . D_Z^2F_{\theta^{\|}}(0)[Z,Z]=\left\langle Z,H'(\theta^{\|})Z\right\rangle_F. D Z 2 F θ ∥ ( 0 ) [ Z , Z ] = ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F .
This proves the expansion up to third order.
Finally, because L \mathcal{L} L is smooth and the set
{ θ ∥ v + s Z : θ ∥ ∈ J , ∥ Z ∥ F ≤ r , s ∈ [ 0 , 1 ] } \left\{\theta^{\|}v+sZ:\theta^{\|}\in J,\ \|Z\|_F\le r,\ s\in[0,1]\right\} { θ ∥ v + s Z : θ ∥ ∈ J , ∥ Z ∥ F ≤ r , s ∈ [ 0 , 1 ] }
is compact, the third derivative of F θ ∥ F_{\theta^{\|}} F θ ∥ with respect to Z Z Z is uniformly bounded there. Hence Taylor’s theorem gives a constant c ∥ , r , J < ∞ c_{\|,r,J}<\infty c ∥ , r , J < ∞ such that
∣ R ∥ ( θ ∥ , Z ) ∣ ≤ c ∥ , r , J ∥ Z ∥ F 3 . \left|R_{\|}(\theta^{\|},Z)\right|\le c_{\|,r,J}\|Z\|_F^3. R ∥ ( θ ∥ , Z ) ≤ c ∥ , r , J ∥ Z ∥ F 3 .
This completes the proof.
Proposition 11 (Verification constants for multiclass cross-entropy ). Assume (86). Then the following constants verify the abstract assumptions for the multiclass simplex model on the tubular neighborhood
{ θ ∥ v + Z : θ ∥ ∈ [ θ ∗ ∥ , θ ∗ ( r ) ] , ∥ Z ∥ F ≤ r } \left\{\theta^{\|}v+Z:\theta^{\|}\in[\theta_*^{\|},\theta_*^{(r)}],\ \|Z\|_F\le r\right\} { θ ∥ v + Z : θ ∥ ∈ [ θ ∗ ∥ , θ ∗ ( r ) ] , ∥ Z ∥ F ≤ r } and on the corresponding tail half-tubular neighborhood
{ θ ∥ v + Z : θ ∥ ≥ θ ∗ ( r ) , ∥ Z ∥ F ≤ r } . \left\{\theta^{\|}v+Z:\theta^{\|}\ge\theta_*^{(r)},\ \|Z\|_F\le r\right\}. { θ ∥ v + Z : θ ∥ ≥ θ ∗ ( r ) , ∥ Z ∥ F ≤ r } . (a) Relative curvature comparability. A sufficient RC tolerance is
δ r : = e 2 X max r − 1. (88) \delta_r:=e^{\sqrt{2}X_{\max}r}-1. \tag*{(88)} δ r := e 2 X m a x r − 1. (88) In particular, the admissibility condition δ r < 1 \delta_r<1 δ r < 1 is implied by
2 X max r < log 2. \sqrt{2}X_{\max}r<\log2. 2 X m a x r < log 2. (b) Matrix log-derivative lower bound. On the oscillation slab,
− H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U , -H'(\theta^{\|})\succeq\beta H(\theta^{\|})\qquad\text{on }U, − H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U , where
β : = a ~ min tanh ( a ~ min θ 0 ∥ − log ( K − 1 ) 2 ) . \beta:=\widetilde{a}_{\min}\tanh\left(\frac{\widetilde{a}_{\min}\theta_0^{\|}-\log(K-1)}{2}\right). β := a m i n tanh ( 2 a m i n θ 0 ∥ − log ( K − 1 ) ) . Under (86), this gives β ≥ 0 \beta\ge0 β ≥ 0 , and it is strictly positive if
a ~ min θ 0 ∥ > log ( K − 1 ) . \widetilde{a}_{\min}\theta_0^{\|}>\log(K-1). a m i n θ 0 ∥ > log ( K − 1 ) . (c) Valley drift lower bound. The restricted loss satisfies
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a ~ i q K ( a ~ i θ ∥ ) , -\ell'(\theta^{\|})=\frac{1}{N}\sum_{i=1}^{N}\widetilde{a}_i q_K(\widetilde{a}_i\theta^{\|}), − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q K ( a i θ ∥ ) , so on the oscillation slab one may take
m ℓ : = 1 N ∑ i = 1 N a ~ i q K ( a ~ i θ ∗ ( r ) ) . (89) m_{\ell}:=\frac{1}{N}\sum_{i=1}^{N}\widetilde{a}_i q_K(\widetilde{a}_i\theta_*^{(r)}). \tag*{(89)} m ℓ := N 1 i = 1 ∑ N a i q K ( a i θ ∗ ( r ) ) . (89)
(d) Uniform upper bound on tubular neighborhood-level parallel drift. For every θ ∥ ∈ R \theta^{\parallel}\in\mathbb{R} θ ∥ ∈ R and every Z ∈ U Z\in\mathcal{U} Z ∈ U ,
− ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F ≤ G ∥ , G ∥ : = 1 N ∑ i = 1 N a ~ i . -\langle\nabla\mathcal{L}(\theta^{\parallel}v+Z),v\rangle_F\leq G_{\parallel},\qquad G_{\parallel}:=\frac{1}{N}\sum_{i=1}^{N}\widetilde{a}_i. − ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F ≤ G ∥ , G ∥ := N 1 i = 1 ∑ N a i .
(e) Tail drift constants. Set β t a i l : = a ~ min \beta_{\mathrm{tail}}:=\widetilde{a}_{\min} β tail := a m i n , and τ r : = 2 X max r \tau_r:=\sqrt{2}X_{\max}r τ r := 2 X m a x r . If a ~ min θ ∗ ( r ) − τ r ≥ log ( K − 1 ) \widetilde{a}_{\min}\theta_*^{(r)}-\tau_r\geq\log(K-1) a m i n θ ∗ ( r ) − τ r ≥ log ( K − 1 ) , then for every θ ∥ ≥ θ ∗ ( r ) \theta^{\parallel}\geq\theta_*^{(r)} θ ∥ ≥ θ ∗ ( r ) , and every Z ∈ U Z\in\mathcal{U} Z ∈ U with ∥ Z ∥ F ≤ r \lVert Z\rVert_F\leq r ∥ Z ∥ F ≤ r , the parallel drift satisfies
A − e − β t a i l ( θ ∥ − θ ∗ ( r ) ) ≤ − ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F ≤ A + e − β t a i l ( θ ∥ − θ ∗ ( r ) ) , A_-e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{(r)})}\leq-\langle\nabla\mathcal{L}(\theta^{\parallel}v+Z),v\rangle_F\leq A_+e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{(r)})}, A − e − β tail ( θ ∥ − θ ∗ ( r ) ) ≤ − ⟨ ∇ L ( θ ∥ v + Z ) , v ⟩ F ≤ A + e − β tail ( θ ∥ − θ ∗ ( r ) ) ,
with
A − : = e − τ r K − 1 2 N ∑ i ∈ I ∗ a ~ i e − a ~ i θ ∗ ( r ) . (90) A_-:=e^{-\tau_r}\frac{K-1}{2N}\sum_{i\in I_*}\widetilde{a}_i e^{-\widetilde{a}_i\theta_*^{(r)}}. \tag*{(90)} A − := e − τ r 2 N K − 1 i ∈ I ∗ ∑ a i e − a i θ ∗ ( r ) . (90)
A + : = e τ r K − 1 N ∑ i = 1 N a ~ i e − a ~ min θ ∗ ( r ) . (91) A_+:=e^{\tau_r}\frac{K-1}{N}\sum_{i=1}^{N}\widetilde{a}_i e^{-\widetilde{a}_{\min}\theta_*^{(r)}}. \tag*{(91)} A + := e τ r N K − 1 i = 1 ∑ N a i e − a m i n θ ∗ ( r ) . (91)
(f) Tail sharpness constants. For i = 1 , … , N i=1,\ldots,N i = 1 , … , N , define the positive semidefinite operator B i : U → U B_i:\mathcal{U}\to\mathcal{U} B i : U → U by the quadratic form
⟨ Z , B i Z ⟩ F : = 1 K N ∑ c = 1 K ∑ d ≠ c ( ⟨ z d − z c , x c , i ⟩ ) 2 , Z = [ z 1 , … , z K ] ∈ U . (92) \langle Z,B_iZ\rangle_F:=\frac{1}{KN}\sum_{c=1}^{K}\sum_{d\neq c}\bigl(\langle z_d-z_c,x_{c,i}\rangle\bigr)^2,\qquad Z=[z_1,\ldots,z_K]\in\mathcal{U}. \tag*{(92)} ⟨ Z , B i Z ⟩ F := K N 1 c = 1 ∑ K d = c ∑ ( ⟨ z d − z c , x c , i ⟩ ) 2 , Z = [ z 1 , … , z K ] ∈ U . (92)
Let
B ∗ : = ∑ i ∈ I ∗ B i , B a l l : = ∑ i = 1 N B i . B_*:=\sum_{i\in I_*}B_i,\qquad B_{\mathrm{all}}:=\sum_{i=1}^{N}B_i. B ∗ := i ∈ I ∗ ∑ B i , B all := i = 1 ∑ N B i .
Then, for every θ ∥ ≥ θ ∗ ( r ) \theta^{\parallel}\geq\theta_*^{(r)} θ ∥ ≥ θ ∗ ( r ) ,
λ 1 , − e − β t a i l ( θ ∥ − θ ∗ ( r ) ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + e − β t a i l ( θ ∥ − θ ∗ ( r ) ) , \lambda_{1,-}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{(r)})}\leq\lambda_1(\theta^{\parallel})\leq\lambda_{1,+}e^{-\beta_{\mathrm{tail}}(\theta^{\parallel}-\theta_*^{(r)})}, λ 1 , − e − β tail ( θ ∥ − θ ∗ ( r ) ) ≤ λ 1 ( θ ∥ ) ≤ λ 1 , + e − β tail ( θ ∥ − θ ∗ ( r ) ) ,
where one may take
λ 1 , − : = 1 4 e − a ~ min θ ∗ ( r ) λ max ( B ∗ ∣ U ) , (93) \lambda_{1,-}:=\frac{1}{4}e^{-\widetilde{a}_{\min}\theta_*^{(r)}}\lambda_{\max}(B_*|_{\mathcal{U}}), \tag*{(93)} λ 1 , − := 4 1 e − a m i n θ ∗ ( r ) λ m a x ( B ∗ ∣ U ) , (93)
λ 1 , + : = 2 e − a ~ min θ ∗ ( r ) λ max ( B a l l ∣ U ) . (94) \lambda_{1,+}:=2e^{-\widetilde{a}_{\min}\theta_*^{(r)}}\lambda_{\max}(B_{\mathrm{all}}|_{\mathcal{U}}). \tag*{(94)} λ 1 , + := 2 e − a m i n θ ∗ ( r ) λ m a x ( B all ∣ U ) . (94)
(g) Unstable phase conditioning, when needed. If
H ( θ ∥ ) ∣ U ≻ 0 ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] , H(\theta^{\parallel})|_{\mathcal{U}}\succ0\qquad\forall\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{(r)}], H ( θ ∥ ) ∣ U ≻ 0 ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] ,
then compactness and continuity imply that
ρ p r e : = inf θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] λ min ( H ( θ ∥ ) ∣ U ) λ max ( H ( θ ∥ ) ∣ U ) > 0 , \rho_{\mathrm{pre}}:=\inf_{\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{(r)}]}\frac{\lambda_{\min}(H(\theta^{\parallel})|_{\mathcal{U}})}{\lambda_{\max}(H(\theta^{\parallel})|_{\mathcal{U}})}>0, ρ pre := θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] inf λ m a x ( H ( θ ∥ ) ∣ U ) λ m i n ( H ( θ ∥ ) ∣ U ) > 0 ,
and hence
H ( θ ∥ ) ⪰ ρ p r e λ 1 ( θ ∥ ) I U ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] . H(\theta^{\parallel})\succeq\rho_{\mathrm{pre}}\lambda_1(\theta^{\parallel})I_{\mathcal{U}}\qquad\forall\theta^{\parallel}\in[\theta_0^{\parallel},\theta_*^{(r)}]. H ( θ ∥ ) ⪰ ρ pre λ 1 ( θ ∥ ) I U ∀ θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] .
(h) Tail conditioning, when applicable. If the slowest-layer operator is positive definite on the active orthogonal space,
B ∗ ∣ U ≻ 0 , (95) B_*|_{\mathcal{U}}\succ0, \tag*{(95)} B ∗ ∣ U ≻ 0 , (95)
then the active sharpness satisfies the tail conditioning bound
H ( θ ∥ ) ⪰ ρ t a i l λ 1 ( θ ∥ ) I U ∀ θ ∥ ≥ θ ∗ ( r ) , H(\theta^{\parallel})\succeq\rho_{\mathrm{tail}}\lambda_1(\theta^{\parallel})I_{\mathcal{U}}\qquad\forall\theta^{\parallel}\geq\theta_*^{(r)}, H ( θ ∥ ) ⪰ ρ tail λ 1 ( θ ∥ ) I U ∀ θ ∥ ≥ θ ∗ ( r ) ,
with
ρ t a i l : = λ min ( B ∗ ∣ U ) 8 λ max ( B a l l ∣ U ) . (96) \rho_{\mathrm{tail}}:=\frac{\lambda_{\min}(B_*|_{\mathcal{U}})}{8\lambda_{\max}(B_{\mathrm{all}}|_{\mathcal{U}})}. \tag*{(96)} ρ tail := 8 λ m a x ( B all ∣ U ) λ m i n ( B ∗ ∣ U ) . (96)
Consequently, the full three-phase post-stability theorem applies under (96).
Proof. We verify each item.
Relative curvature comparability. Fix a sample ( x c , i , c ) (x_{c,i}, c) ( x c , i , c ) . For a perturbation Z = [ z 1 , … , z K ] Z = [z_1,\ldots,z_K] Z = [ z 1 , … , z K ] , define the logit perturbations
Δ d : = ⟨ z d , x c , i ⟩ , d = 1 , … , K . \Delta_d := \langle z_d,x_{c,i}\rangle,\qquad d=1,\ldots,K. Δ d := ⟨ z d , x c , i ⟩ , d = 1 , … , K .
Softmax probabilities and their covariance are invariant under adding the same scalar to all logits. Hence only the oscillation of Δ = ( Δ 1 , … , Δ K ) \Delta=(\Delta_1,\ldots,\Delta_K) Δ = ( Δ 1 , … , Δ K ) matters. For every d , e d,e d , e ,
∣ Δ d − Δ e ∣ = ∣ ⟨ z d − z e , x c , i ⟩ ∣ ≤ ∥ z d − z e ∥ ∥ x c , i ∥ ≤ 2 ∥ Z ∥ F X max ≤ 2 X max r . |\Delta_d-\Delta_e|=|\langle z_d-z_e,x_{c,i}\rangle|\leq\|z_d-z_e\|\|x_{c,i}\|\leq\sqrt{2}\|Z\|_F X_{\max}\leq\sqrt{2}X_{\max}r. ∣ Δ d − Δ e ∣ = ∣ ⟨ z d − z e , x c , i ⟩ ∣ ≤ ∥ z d − z e ∥∥ x c , i ∥ ≤ 2 ∥ Z ∥ F X m a x ≤ 2 X m a x r .
Thus, after subtracting a common scalar from the perturbed logits, every logit perturbation lies in an interval of length at most 2 X max r \sqrt{2}X_{\max}r 2 X m a x r . Consequently the perturbed softmax distribution is within density ratio
C r : = e 2 X max r C_r:=e^{\sqrt{2}X_{\max}r} C r := e 2 X m a x r
of the parallel softmax distribution.
Next, we will need to invoke the following lemma (proof is one line using h T Cov ( p ) h = min a ∈ R ∑ d p d ( h d − a ) 2 h^\mathsf{T}\operatorname{Cov}(p)h=\min_{a\in\mathbb{R}}\sum_d p_d(h_d-a)^2 h T Cov ( p ) h = min a ∈ R ∑ d p d ( h d − a ) 2 :
Lemma 4 (Covariance comparability under bounded density ratios). Let p , p ~ ∈ Δ K p,\widetilde{p}\in\Delta_K p , p ∈ Δ K satisfy
C − 1 ≤ p ~ d p d ≤ C ∀ d . C^{-1}\leq\frac{\widetilde{p}_d}{p_d}\leq C\qquad\forall d. C − 1 ≤ p d p d ≤ C ∀ d . Then for every h ∈ R K h\in\mathbb{R}^K h ∈ R K ,
C − 1 h T Cov ( p ) h ≤ h T Cov ( p ~ ) h ≤ C h T Cov ( p ) h . C^{-1}h^\mathsf{T}\operatorname{Cov}(p)h\leq h^\mathsf{T}\operatorname{Cov}(\widetilde{p})h\leq Ch^\mathsf{T}\operatorname{Cov}(p)h. C − 1 h T Cov ( p ) h ≤ h T Cov ( p ) h ≤ C h T Cov ( p ) h .
By covariance comparability under bounded density ratios,
C r − 1 h T Cov ( p ) h ≤ h T Cov ( p ~ ) h ≤ C r h T Cov ( p ) h C_r^{-1}h^\mathsf{T}\operatorname{Cov}(p)h\leq h^\mathsf{T}\operatorname{Cov}(\widetilde{p})h\leq C_rh^\mathsf{T}\operatorname{Cov}(p)h C r − 1 h T Cov ( p ) h ≤ h T Cov ( p ) h ≤ C r h T Cov ( p ) h
for every h ∈ R K h\in\mathbb{R}^K h ∈ R K . Applying this samplewise and averaging over ( c , i ) (c,i) ( c , i ) gives
e − 2 X max r H ( θ ∥ ) ⪯ D Z G U ( θ ∥ , Z ) ⪯ e 2 X max r H ( θ ∥ ) . e^{-\sqrt{2}X_{\max}r}H(\theta^\parallel)\preceq D_ZG_U(\theta^\parallel,Z)\preceq e^{\sqrt{2}X_{\max}r}H(\theta^\parallel). e − 2 X m a x r H ( θ ∥ ) ⪯ D Z G U ( θ ∥ , Z ) ⪯ e 2 X m a x r H ( θ ∥ ) .
Therefore the matrix RC condition holds with
δ r : = e 2 X max r − 1 , \delta_r:=e^{\sqrt{2}X_{\max}r}-1, δ r := e 2 X m a x r − 1 ,
since
1 − δ r = 2 − e 2 X max r ≤ e − 2 X max r . 1-\delta_r=2-e^{\sqrt{2}X_{\max}r}\leq e^{-\sqrt{2}X_{\max}r}. 1 − δ r = 2 − e 2 X m a x r ≤ e − 2 X m a x r .
Log-derivative lower bound. We prove the matrix inequality
− H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U . -H'(\theta^\parallel)\succeq\beta H(\theta^\parallel)\qquad\text{on }U. − H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U .
Fix a class c c c and write q = q K ( s ) q=q_K(s) q = q K ( s ) . Then
p c ( c ) ( s ) = 1 − q , p d ( c ) ( s ) = q K − 1 ( d ≠ c ) . p_c^{(c)}(s)=1-q,\qquad p_d^{(c)}(s)=\frac{q}{K-1}\quad(d\neq c). p c ( c ) ( s ) = 1 − q , p d ( c ) ( s ) = K − 1 q ( d = c ) .
For every h ∈ R K h\in\mathbb{R}^K h ∈ R K , the softmax covariance admits the pairwise representation
h T Cov ( p ( c ) ( s ) ) h = ∑ d < e p d ( c ) ( s ) p e ( c ) ( s ) ( h d − h e ) 2 . h^\mathsf{T}\operatorname{Cov}(p^{(c)}(s))h=\sum_{d<e}p_d^{(c)}(s)p_e^{(c)}(s)(h_d-h_e)^2. h T Cov ( p ( c ) ( s )) h = d < e ∑ p d ( c ) ( s ) p e ( c ) ( s ) ( h d − h e ) 2 .
There are two types of pair weights. If one index is the correct class c c c , then
p c ( c ) ( s ) p d ( c ) ( s ) = ( 1 − q ) q K − 1 ( d ≠ c ) , p_c^{(c)}(s)p_d^{(c)}(s)=(1-q)\frac{q}{K-1}\qquad(d\neq c), p c ( c ) ( s ) p d ( c ) ( s ) = ( 1 − q ) K − 1 q ( d = c ) ,
and therefore
− d d s log ( ( 1 − q ) q ) = 1 − 2 q = tanh ( s − log ( K − 1 ) 2 ) . -\frac{d}{ds}\log((1-q)q)=1-2q=\tanh\left(\frac{s-\log(K-1)}{2}\right). − d s d log (( 1 − q ) q ) = 1 − 2 q = tanh ( 2 s − log ( K − 1 ) ) .
If both indices are incorrect, then
p d ( c ) ( s ) p e ( c ) ( s ) = q 2 ( K − 1 ) 2 ( d , e ≠ c , d ≠ e ) , p_d^{(c)}(s)p_e^{(c)}(s)=\frac{q^2}{(K-1)^2}\qquad(d,e\ne c,\ d\ne e), p d ( c ) ( s ) p e ( c ) ( s ) = ( K − 1 ) 2 q 2 ( d , e = c , d = e ) ,
and hence
− d d s log ( q 2 ) = 2 ( 1 − q ) . -\frac{d}{ds}\log(q^2)=2(1-q). − d s d log ( q 2 ) = 2 ( 1 − q ) .
On the region s ≥ log ( K − 1 ) s\ge\log(K-1) s ≥ log ( K − 1 ) , we have q ≤ 1 / 2 q\le1/2 q ≤ 1/2 , so
2 ( 1 − q ) ≥ 1 ≥ 1 − 2 q . 2(1-q)\ge1\ge1-2q. 2 ( 1 − q ) ≥ 1 ≥ 1 − 2 q .
Thus every pair weight in the covariance representation satisfies
− d d s log ( p d ( c ) ( s ) p e ( c ) ( s ) ) ≥ tanh ( s − log ( K − 1 ) 2 ) . -\frac{d}{ds}\log\left(p_d^{(c)}(s)p_e^{(c)}(s)\right)\ge\tanh\left(\frac{s-\log(K-1)}{2}\right). − d s d log ( p d ( c ) ( s ) p e ( c ) ( s ) ) ≥ tanh ( 2 s − log ( K − 1 ) ) .
Consequently, for every h ∈ R K h\in\mathbb{R}^K h ∈ R K ,
− h ⊤ d d s Cov ( p ( c ) ( s ) ) h ≥ tanh ( s − log ( K − 1 ) 2 ) h ⊤ Cov ( p ( c ) ( s ) ) h . -h^\top\frac{d}{ds}\operatorname{Cov}(p^{(c)}(s))h\ge\tanh\left(\frac{s-\log(K-1)}{2}\right)h^\top\operatorname{Cov}(p^{(c)}(s))h. − h ⊤ d s d Cov ( p ( c ) ( s )) h ≥ tanh ( 2 s − log ( K − 1 ) ) h ⊤ Cov ( p ( c ) ( s )) h .
Now take s i ( θ ∥ ) = a ~ i θ ∥ s_i(\theta^\parallel)=\widetilde a_i\theta^\parallel s i ( θ ∥ ) = a i θ ∥ . On the oscillation slab, assumption
a ~ min θ 0 ∥ ≥ log ( K − 1 ) \widetilde a_{\min}\theta_0^\parallel\ge\log(K-1) a m i n θ 0 ∥ ≥ log ( K − 1 )
implies s i ( θ ∥ ) ≥ log ( K − 1 ) s_i(\theta^\parallel)\ge\log(K-1) s i ( θ ∥ ) ≥ log ( K − 1 ) for all i i i and all θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] \theta^\parallel\in[\theta_0^\parallel,\theta_*^{(r)}] θ ∥ ∈ [ θ 0 ∥ , θ ∗ ( r ) ] . Hence
− h ⊤ d d θ ∥ Cov ( p ( c ) ( s i ( θ ∥ ) ) ) h ≥ a ~ i tanh ( a ~ i θ ∥ − log ( K − 1 ) 2 ) h ⊤ Cov ( p ( c ) ( s i ( θ ∥ ) ) ) h . -h^\top\frac{d}{d\theta^\parallel}\operatorname{Cov}(p^{(c)}(s_i(\theta^\parallel)))h\ge\widetilde a_i\tanh\left(\frac{\widetilde a_i\theta^\parallel-\log(K-1)}{2}\right)h^\top\operatorname{Cov}(p^{(c)}(s_i(\theta^\parallel)))h. − h ⊤ d θ ∥ d Cov ( p ( c ) ( s i ( θ ∥ ))) h ≥ a i tanh ( 2 a i θ ∥ − log ( K − 1 ) ) h ⊤ Cov ( p ( c ) ( s i ( θ ∥ ))) h .
Since the function
a ↦ a tanh ( a θ ∥ − log ( K − 1 ) 2 ) a\mapsto a\tanh\left(\frac{a\theta^\parallel-\log(K-1)}{2}\right) a ↦ a tanh ( 2 a θ ∥ − log ( K − 1 ) )
is increasing for a > 0 a>0 a > 0 whenever a θ ∥ ≥ log ( K − 1 ) a\theta^\parallel\ge\log(K-1) a θ ∥ ≥ log ( K − 1 ) , and θ ∥ ≥ θ 0 ∥ \theta^\parallel\ge\theta_0^\parallel θ ∥ ≥ θ 0 ∥ , the right-hand coefficient is bounded below by
β : = a ~ min tanh ( a ~ min θ 0 ∥ − log ( K − 1 ) 2 ) . \beta:=\widetilde a_{\min}\tanh\left(\frac{\widetilde a_{\min}\theta_0^\parallel-\log(K-1)}{2}\right). β := a m i n tanh ( 2 a m i n θ 0 ∥ − log ( K − 1 ) ) .
Substituting into the quadratic-form expression for H H H and summing over i i i and c c c gives
− ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F ≥ β ⟨ Z , H ( θ ∥ ) Z ⟩ F ∀ Z ∈ U . -\langle Z,H'(\theta^\parallel)Z\rangle_F\ge\beta\langle Z,H(\theta^\parallel)Z\rangle_F\qquad\forall Z\in\mathcal{U}. − ⟨ Z , H ′ ( θ ∥ ) Z ⟩ F ≥ β ⟨ Z , H ( θ ∥ ) Z ⟩ F ∀ Z ∈ U .
Equivalently,
− H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U . -H'(\theta^\parallel)\succeq\beta H(\theta^\parallel)\quad\text{on }\mathcal{U}. − H ′ ( θ ∥ ) ⪰ β H ( θ ∥ ) on U .
This proves the stated log-derivative lower bound.
Valley drift constants. By Proposition 9,
− ℓ ′ ( θ ∥ ) = 1 N ∑ i = 1 N a ~ i q K ( a ~ i θ ∥ ) . -\ell'(\theta^\parallel)=\frac{1}{N}\sum_{i=1}^{N}\widetilde a_iq_K(\widetilde a_i\theta^\parallel). − ℓ ′ ( θ ∥ ) = N 1 i = 1 ∑ N a i q K ( a i θ ∥ ) .
Since q K q_K q K is positive and decreasing, the infimum on [ θ 0 ∥ , θ ∗ ( r ) ] [\theta_0^\parallel,\theta_*^{(r)}] [ θ 0 ∥ , θ ∗ ( r ) ] is attained at θ ∗ ( r ) \theta_*^{(r)} θ ∗ ( r ) , giving (90). The upper bound follows from q K ( s ) ≤ 1 q_K(s)\le1 q K ( s ) ≤ 1 .
Tail drift on the tubular neighborhood. Fix θ ∥ ≥ θ ∗ ( r ) \theta^\parallel\ge\theta_*^{(r)} θ ∥ ≥ θ ∗ ( r ) and Z = [ z 1 , … , z K ] ∈ U Z=[z_1,\ldots,z_K]\in\mathcal{U} Z = [ z 1 , … , z K ] ∈ U with ∥ Z ∥ F ≤ r \lVert Z\rVert_F\le r ∥ Z ∥ F ≤ r . For a sample ( x c , i , c ) (x_{c,i},c) ( x c , i , c ) , define the softmax probabilities at W = θ ∥ v + Z W=\theta^\parallel v+Z W = θ ∥ v + Z by
p d ∣ c , i ( W ) : = exp ( ⟨ w d , x c , i ⟩ ) ∑ e = 1 K exp ( ⟨ w e , x c , i ⟩ ) . p_{d\mid c,i}(W):=\frac{\exp(\langle w_d,x_{c,i}\rangle)}{\sum_{e=1}^{K}\exp(\langle w_e,x_{c,i}\rangle)}. p d ∣ c , i ( W ) := ∑ e = 1 K exp (⟨ w e , x c , i ⟩) exp (⟨ w d , x c , i ⟩) .
We first compute the contribution to the parallel drift. Since v = R / K v = R/\sqrt{K} v = R / K , one has
− ⟨ ∇ L ( W ) , v ⟩ F = 1 K N ∑ i = 1 N ∑ c = 1 K b i K ∑ d = 1 K ( 1 { d = c } − p d ∣ c , i ( W ) ) ⟨ r d , r c ⟩ . -\langle\nabla\mathcal{L}(W),v\rangle_F = \frac{1}{KN}\sum_{i=1}^{N}\sum_{c=1}^{K}\frac{b_i}{\sqrt{K}}\sum_{d=1}^{K}\left(\mathbb{1}_{\{d=c\}}-p_{d|c,i}(W)\right)\langle r_d,r_c\rangle. − ⟨ ∇ L ( W ) , v ⟩ F = K N 1 i = 1 ∑ N c = 1 ∑ K K b i d = 1 ∑ K ( 1 { d = c } − p d ∣ c , i ( W ) ) ⟨ r d , r c ⟩ .
Using the simplex identities,
∑ d = 1 K ( 1 { d = c } − p d ∣ c , i ( W ) ) ⟨ r d , r c ⟩ = K K − 1 ( 1 − p c ∣ c , i ( W ) ) . \sum_{d=1}^{K}\left(\mathbb{1}_{\{d=c\}}-p_{d|c,i}(W)\right)\langle r_d,r_c\rangle= \frac{K}{K-1}\left(1-p_{c|c,i}(W)\right). d = 1 ∑ K ( 1 { d = c } − p d ∣ c , i ( W ) ) ⟨ r d , r c ⟩ = K − 1 K ( 1 − p c ∣ c , i ( W ) ) .
Therefore
− g ∥ ( θ ∥ , Z ) = 1 K N ∑ i = 1 N ∑ c = 1 K a ~ i ( 1 − p c ∣ c , i ( θ ∥ v + Z ) ) . (97) -g_{\parallel}(\theta^{\parallel},Z)=\frac{1}{KN}\sum_{i=1}^{N}\sum_{c=1}^{K}\tilde{a}_i\left(1-p_{c|c,i}(\theta^{\parallel}v+Z)\right). \tag*{(97)} − g ∥ ( θ ∥ , Z ) = K N 1 i = 1 ∑ N c = 1 ∑ K a ~ i ( 1 − p c ∣ c , i ( θ ∥ v + Z ) ) . (97)
It remains to bound the correct-class error probability. For each incorrect class d ≠ c d\ne c d = c , the correct-vs-incorrect logit gap is
⟨ w c − w d , x c , i ⟩ = a ~ i θ ∥ + ⟨ z c − z d , x c , i ⟩ . \langle w_c-w_d,x_{c,i}\rangle=\tilde{a}_i\theta^{\parallel}+\langle z_c-z_d,x_{c,i}\rangle. ⟨ w c − w d , x c , i ⟩ = a ~ i θ ∥ + ⟨ z c − z d , x c , i ⟩ .
Because ∥ Z ∥ F ≤ r \lVert Z\rVert_F\le r ∥ Z ∥ F ≤ r ,
∥ z c − z d ∥ ≤ 2 ∥ Z ∥ F ≤ 2 r , \lVert z_c-z_d\rVert\le\sqrt{2}\lVert Z\rVert_F\le\sqrt{2}r, ∥ z c − z d ∥ ≤ 2 ∥ Z ∥ F ≤ 2 r ,
and hence
∣ ⟨ z c − z d , x c , i ⟩ ∣ ≤ 2 X max r = τ r . \left|\langle z_c-z_d,x_{c,i}\rangle\right|\le\sqrt{2}X_{\max}r=\tau_r. ∣ ⟨ z c − z d , x c , i ⟩ ∣ ≤ 2 X m a x r = τ r .
Thus every incorrect-class exponential ratio satisfies
e − ( a ~ i θ ∥ + τ r ) ≤ exp ( − ⟨ w c − w d , x c , i ⟩ ) ≤ e − ( a ~ i θ ∥ − τ r ) . e^{-(\tilde{a}_i\theta^{\parallel}+\tau_r)}\le\exp\left(-\langle w_c-w_d,x_{c,i}\rangle\right)\le e^{-(\tilde{a}_i\theta^{\parallel}-\tau_r)}. e − ( a ~ i θ ∥ + τ r ) ≤ exp ( − ⟨ w c − w d , x c , i ⟩ ) ≤ e − ( a ~ i θ ∥ − τ r ) .
Set
S c , i ( W ) : = ∑ d ≠ c exp ( − ⟨ w c − w d , x c , i ⟩ ) . S_{c,i}(W):=\sum_{d\ne c}\exp\left(-\langle w_c-w_d,x_{c,i}\rangle\right). S c , i ( W ) := d = c ∑ exp ( − ⟨ w c − w d , x c , i ⟩ ) .
Then
( K − 1 ) e − a ~ i θ ∥ − τ r ≤ S c , i ( W ) ≤ ( K − 1 ) e − a ~ i θ ∥ + τ r . (K-1)e^{-\tilde{a}_i\theta^{\parallel}-\tau_r}\le S_{c,i}(W)\le(K-1)e^{-\tilde{a}_i\theta^{\parallel}+\tau_r}. ( K − 1 ) e − a ~ i θ ∥ − τ r ≤ S c , i ( W ) ≤ ( K − 1 ) e − a ~ i θ ∥ + τ r .
Moreover,
1 − p c ∣ c , i ( W ) = S c , i ( W ) 1 + S c , i ( W ) . 1-p_{c|c,i}(W)=\frac{S_{c,i}(W)}{1+S_{c,i}(W)}. 1 − p c ∣ c , i ( W ) = 1 + S c , i ( W ) S c , i ( W ) .
The tail tubular neighborhood margin condition
a ~ min θ ∗ ( r ) − τ r ≥ log ( K − 1 ) \tilde{a}_{\min}\theta_*^{(r)}-\tau_r\ge\log(K-1) a ~ m i n θ ∗ ( r ) − τ r ≥ log ( K − 1 )
implies
S c , i ( W ) ≤ 1 S_{c,i}(W)\le1 S c , i ( W ) ≤ 1
for all θ ∥ ≥ θ ∗ ( r ) \theta^{\parallel}\ge\theta_*^{(r)} θ ∥ ≥ θ ∗ ( r ) , all i i i , and all c c c . Therefore
1 2 S c , i ( W ) ≤ 1 − p c ∣ c , i ( W ) ≤ S c , i ( W ) . \frac{1}{2}S_{c,i}(W)\le1-p_{c|c,i}(W)\le S_{c,i}(W). 2 1 S c , i ( W ) ≤ 1 − p c ∣ c , i ( W ) ≤ S c , i ( W ) .
Consequently,
K − 1 2 e − a ~ i θ ∥ − τ r ≤ 1 − p c ∣ c , i ( W ) ≤ ( K − 1 ) e − a ~ i θ ∥ + τ r . (98) \frac{K-1}{2}e^{-\tilde{a}_i\theta^{\parallel}-\tau_r}\le1-p_{c|c,i}(W)\le(K-1)e^{-\tilde{a}_i\theta^{\parallel}+\tau_r}. \tag*{(98)} 2 K − 1 e − a ~ i θ ∥ − τ r ≤ 1 − p c ∣ c , i ( W ) ≤ ( K − 1 ) e − a ~ i θ ∥ + τ r . (98)
Substituting (99) into (98), and restricting the lower bound to i ∈ I ∗ i\in I_* i ∈ I ∗ , gives
− g ∥ ( θ ∥ , Z ) ≥ e − τ r K − 1 2 N ∑ i ∈ I ∗ a ~ i e − a ~ i θ ∥ . -g_{\parallel}(\theta^{\parallel},Z)\ge e^{-\tau_r}\frac{K-1}{2N}\sum_{i\in I_*}\tilde{a}_i e^{-\tilde{a}_i\theta^{\parallel}}. − g ∥ ( θ ∥ , Z ) ≥ e − τ r 2 N K − 1 i ∈ I ∗ ∑ a ~ i e − a ~ i θ ∥ .
Since a ~ i = a ~ min \tilde{a}_i=\tilde{a}_{\min} a ~ i = a ~ m i n for i ∈ I ∗ i\in I_* i ∈ I ∗ , this equals
A − t u b u l a r n e i g h b o r h o o d e − a ~ min ( θ ∥ − θ ∗ ( r ) ) . A_-^{\mathrm{tubularneighborhood}}e^{-\tilde{a}_{\min}(\theta^{\parallel}-\theta_*^{(r)})}. A − tubularneighborhood e − a ~ m i n ( θ ∥ − θ ∗ ( r ) ) .
Similarly,
− g ∥ ( θ ℓ , Z ) ≤ e τ r K − 1 N ∑ i = 1 N a ~ i e − a ~ i θ ℓ ≤ e τ r K − 1 N ∑ i = 1 N a ~ i e − a ~ min θ ℓ , -g_{\parallel}(\theta^{\ell}, Z) \le e^{\tau_r}\frac{K-1}{N}\sum_{i=1}^{N}\tilde{a}_i e^{-\tilde{a}_i\theta^{\ell}} \le e^{\tau_r}\frac{K-1}{N}\sum_{i=1}^{N}\tilde{a}_i e^{-\tilde{a}_{\min}\theta^{\ell}}, − g ∥ ( θ ℓ , Z ) ≤ e τ r N K − 1 i = 1 ∑ N a ~ i e − a ~ i θ ℓ ≤ e τ r N K − 1 i = 1 ∑ N a ~ i e − a ~ m i n θ ℓ ,
which is
A + t u b u l a r n e i g h b o r h o o d e − a ~ min ( θ ℓ − θ ( r ) ) . A_{+}^{\mathrm{tubularneighborhood}}e^{-\tilde{a}_{\min}(\theta^{\ell}-\theta^{(r)})}. A + tubularneighborhood e − a ~ m i n ( θ ℓ − θ ( r ) ) .
This proves the tubular neighborhood-level tail drift bounds.
Tail sharpness. Let p ( c ) ( s ) p^{(c)}(s) p ( c ) ( s ) be the parallel softmax distribution for class c c c , with correct-class error probability q K ( s ) q_K(s) q K ( s ) . For any vector h ∈ R K h \in\mathbb{R}^{K} h ∈ R K , the softmax covariance admits the pairwise representation
h T Cov ( p ( c ) ( s ) ) h = ∑ d < e p d ( c ) ( s ) p e ( c ) ( s ) ( h d − h e ) 2 . h^{\mathsf{T}}\operatorname{Cov}(p^{(c)}(s))h=\sum_{d<e}p_d^{(c)}(s)p_e^{(c)}(s)(h_d-h_e)^2. h T Cov ( p ( c ) ( s )) h = d < e ∑ p d ( c ) ( s ) p e ( c ) ( s ) ( h d − h e ) 2 .
When s ≥ log ( K − 1 ) s \ge\log(K-1) s ≥ log ( K − 1 ) , we have q K ( s ) ≤ 1 / 2 q_K(s) \le1/2 q K ( s ) ≤ 1/2 . Hence
h T Cov ( p ( c ) ( s ) ) h ≥ q K ( s ) 2 ( K − 1 ) ∑ d ≠ c ( h d − h c ) 2 , h^{\mathsf{T}}\operatorname{Cov}(p^{(c)}(s))h \ge\frac{q_K(s)}{2(K-1)}\sum_{d\ne c}(h_d-h_c)^2, h T Cov ( p ( c ) ( s )) h ≥ 2 ( K − 1 ) q K ( s ) d = c ∑ ( h d − h c ) 2 ,
and also
h T Cov ( p ( c ) ( s ) ) h ≤ 2 q K ( s ) K − 1 ∑ d ≠ c ( h d − h c ) 2 . h^{\mathsf{T}}\operatorname{Cov}(p^{(c)}(s))h \le\frac{2q_K(s)}{K-1}\sum_{d\ne c}(h_d-h_c)^2. h T Cov ( p ( c ) ( s )) h ≤ K − 1 2 q K ( s ) d = c ∑ ( h d − h c ) 2 .
Combining these inequalities with
K − 1 2 e − s ≤ q K ( s ) ≤ ( K − 1 ) e − s \frac{K-1}{2}e^{-s}\le q_K(s)\le(K-1)e^{-s} 2 K − 1 e − s ≤ q K ( s ) ≤ ( K − 1 ) e − s
gives
1 4 e − s ∑ d ≠ c ( h d − h c ) 2 ≤ h T Cov ( p ( c ) ( s ) ) h ≤ 2 e − s ∑ d ≠ c ( h d − h c ) 2 . \frac{1}{4}e^{-s}\sum_{d\ne c}(h_d-h_c)^2\le h^{\mathsf{T}}\operatorname{Cov}(p^{(c)}(s))h\le2e^{-s}\sum_{d\ne c}(h_d-h_c)^2. 4 1 e − s d = c ∑ ( h d − h c ) 2 ≤ h T Cov ( p ( c ) ( s )) h ≤ 2 e − s d = c ∑ ( h d − h c ) 2 .
Substituting
h d = ⟨ z d , x c , i ⟩ , s = a ~ i θ ℓ , h_d=\langle z_d,x_{c,i}\rangle,\qquad s=\tilde{a}_i\theta^{\ell}, h d = ⟨ z d , x c , i ⟩ , s = a ~ i θ ℓ ,
and averaging over c c c and i i i , we obtain the operator inequalities
1 4 ∑ i = 1 N e − a ~ i θ ℓ B i ⪯ H ( θ ℓ ) ⪯ 2 ∑ i = 1 N e − a ~ i θ ℓ B i . \frac{1}{4}\sum_{i=1}^{N}e^{-\tilde{a}_i\theta^{\ell}}B_i\preceq H(\theta^{\ell})\preceq2\sum_{i=1}^{N}e^{-\tilde{a}_i\theta^{\ell}}B_i. 4 1 i = 1 ∑ N e − a ~ i θ ℓ B i ⪯ H ( θ ℓ ) ⪯ 2 i = 1 ∑ N e − a ~ i θ ℓ B i .
The lower bound restricted to i ∈ I ∗ i\in I_* i ∈ I ∗ gives
H ( θ ℓ ) ≥ 1 4 e − a ~ min θ ℓ B ∗ , H(\theta^{\ell})\ge\frac{1}{4}e^{-\tilde{a}_{\min}\theta^{\ell}}B_*, H ( θ ℓ ) ≥ 4 1 e − a ~ m i n θ ℓ B ∗ ,
and the upper bound gives
H ( θ ℓ ) ≤ 2 e − a ~ min θ ℓ B a l l . H(\theta^{\ell})\le2e^{-\tilde{a}_{\min}\theta^{\ell}}B_{\mathrm{all}}. H ( θ ℓ ) ≤ 2 e − a ~ m i n θ ℓ B all .
Taking top eigenvalues yields (94) and (95).
Tail conditioning. If B ∗ ∣ U ≻ 0 B_*|_U\succ0 B ∗ ∣ U ≻ 0 , then
H ( θ ℓ ) ⪰ 1 4 e − a ~ min θ ℓ λ min ( B ∗ ∣ U ) I U . H(\theta^{\ell})\succeq\frac{1}{4}e^{-\tilde{a}_{\min}\theta^{\ell}}\lambda_{\min}(B_*|_U)I_U. H ( θ ℓ ) ⪰ 4 1 e − a ~ m i n θ ℓ λ m i n ( B ∗ ∣ U ) I U .
Meanwhile,
λ 1 ( θ ℓ ) ≤ 2 e − a ~ min θ ℓ λ max ( B a l l ∣ U ) . \lambda_1(\theta^{\ell})\le2e^{-\tilde{a}_{\min}\theta^{\ell}}\lambda_{\max}(B_{\mathrm{all}}|_U). λ 1 ( θ ℓ ) ≤ 2 e − a ~ m i n θ ℓ λ m a x ( B all ∣ U ) .
Combining the two inequalities gives
H ( θ ℓ ) ⪰ λ min ( B ∗ ∣ U ) 8 λ max ( B a l l ∣ U ) λ 1 ( θ ℓ ) I U , H(\theta^{\ell})\succeq\frac{\lambda_{\min}(B_*|_U)}{8\lambda_{\max}(B_{\mathrm{all}}|_U)}\lambda_1(\theta^{\ell})I_U, H ( θ ℓ ) ⪰ 8 λ m a x ( B all ∣ U ) λ m i n ( B ∗ ∣ U ) λ 1 ( θ ℓ ) I U ,
which is (97). This proves the proposition.
Transient Implicit Bias for Multiclass Cross-Entropy
Theorem 6 (Spanning versus non-spanning geometry for multiclass cross-entropy). Consider the multiclass cross-entropy model of Appendix D. Let
α ~ i : = κ b i K , κ : = K K − 1 , \tilde{\alpha}_i := \frac{\kappa b_i}{\sqrt{K}}, \qquad\kappa:= \frac{K}{K-1}, α ~ i := K κ b i , κ := K − 1 K , and let
α ~ 1 < α ~ 2 < ⋯ < α ~ L \tilde{\alpha}_1 < \tilde{\alpha}_2 < \cdots< \tilde{\alpha}_L α ~ 1 < α ~ 2 < ⋯ < α ~ L be the distinct values among α ~ 1 , … , α ~ N \tilde{\alpha}_1,\ldots,\tilde{\alpha}_N α ~ 1 , … , α ~ N . Define
I j : = { i : α ~ i = α ~ j } , B j : = ∑ i ∈ I j B i , B ≤ j : = ∑ k = 1 j B k , I_j := \{i : \tilde{\alpha}_i = \tilde{\alpha}_j\}, \qquad B_j := \sum_{i \in I_j} B_i, \qquad B_{\leq j} := \sum_{k=1}^{j} B_k, I j := { i : α ~ i = α ~ j } , B j := i ∈ I j ∑ B i , B ≤ j := k = 1 ∑ j B k , where B i : U → U B_i : \mathcal{U} \to\mathcal{U} B i : U → U is the positive semidefinite operator defined in (92) . Assume the active nondegeneracy condition
B ≤ L ∣ U ≻ 0. B_{\leq L}|_{\mathcal{U}} \succ0. B ≤ L ∣ U ≻ 0. Define the first spanning layer index
j ∗ C E : = min { j : B ≤ j ∣ U ≻ 0 } . j_*^{\mathrm{CE}} := \min\{j : B_{\leq j}|_{\mathcal{U}} \succ0\}. j ∗ CE := min { j : B ≤ j ∣ U ≻ 0 } . Then, for all sufficiently large θ ∥ \theta^{\parallel} θ ∥ ,
λ max ( H ( θ ∥ ) ∣ U ) ≍ e − α ~ 1 θ ∥ , \lambda_{\max}\left(H(\theta^{\parallel})|_{\mathcal{U}}\right) \asymp e^{-\tilde{\alpha}_1\theta^{\parallel}}, λ m a x ( H ( θ ∥ ) ∣ U ) ≍ e − α ~ 1 θ ∥ , and
λ min ( H ( θ ∥ ) ∣ U ) ≍ e − α ~ j ∗ C E θ ∥ . \lambda_{\min}\left(H(\theta^{\parallel})|_{\mathcal{U}}\right) \asymp e^{-\tilde{\alpha}_{j_*^{\mathrm{CE}}}\theta^{\parallel}}. λ m i n ( H ( θ ∥ ) ∣ U ) ≍ e − α ~ j ∗ CE θ ∥ . Consequently, the tail conditioning ratio
ρ C E ( θ ∥ ) : = λ min ( H ( θ ∥ ) ∣ U ) λ max ( H ( θ ∥ ) ∣ U ) \rho_{\mathrm{CE}}(\theta^{\parallel}) := \frac{\lambda_{\min}\left(H(\theta^{\parallel})|_{\mathcal{U}}\right)}{\lambda_{\max}\left(H(\theta^{\parallel})|_{\mathcal{U}}\right)} ρ CE ( θ ∥ ) := λ m a x ( H ( θ ∥ ) ∣ U ) λ m i n ( H ( θ ∥ ) ∣ U ) satisfies
ρ C E ( θ ∥ ) ≍ e − ( α ~ j ∗ C E − α ~ 1 ) θ ∥ . \rho_{\mathrm{CE}}(\theta^{\parallel}) \asymp e^{-(\tilde{\alpha}_{j_*^{\mathrm{CE}}}-\tilde{\alpha}_1)\theta^{\parallel}}. ρ CE ( θ ∥ ) ≍ e − ( α ~ j ∗ CE − α ~ 1 ) θ ∥ . Along the stable-tail GD trajectory, since
e α ~ 1 ( θ t ∥ − θ ∗ ( r ) ) ≍ t , e^{\tilde{\alpha}_1(\theta_t^{\parallel}-\theta_*^{(r)})} \asymp t, e α ~ 1 ( θ t ∥ − θ ∗ ( r ) ) ≍ t , one has
ρ C E ( θ t ∥ ) ≍ t − ( α ~ j ∗ C E − α ~ 1 ) / α ~ 1 . \rho_{\mathrm{CE}}(\theta_t^{\parallel}) \asymp t^{-(\tilde{\alpha}_{j_*^{\mathrm{CE}}}-\tilde{\alpha}_1)/\tilde{\alpha}_1}. ρ CE ( θ t ∥ ) ≍ t − ( α ~ j ∗ CE − α ~ 1 ) / α ~ 1 . In particular:
(i) If j ∗ C E = 1 j_*^{\mathrm{CE}} = 1 j ∗ CE = 1 , equivalently
B 1 ∣ U = B ∗ ∣ U ≻ 0 , B_1|_{\mathcal{U}} = B_*|_{\mathcal{U}} \succ0, B 1 ∣ U = B ∗ ∣ U ≻ 0 , then ρ C E ( θ ∥ ) ≍ 1 \rho_{\mathrm{CE}}(\theta^{\parallel}) \asymp1 ρ CE ( θ ∥ ) ≍ 1 , so the tail conditioning hypothesis of Theorem 4 holds and the full three-phase contraction theorem applies.
(ii) If j ∗ C E > 1 j_*^{\mathrm{CE}} > 1 j ∗ CE > 1 , then
ρ C E ( θ t ∥ ) ≍ t − ( α ~ j ∗ C E − α ~ 1 ) / α ~ 1 → 0. \rho_{\mathrm{CE}}(\theta_t^{\parallel}) \asymp t^{-(\tilde{\alpha}_{j_*^{\mathrm{CE}}}-\tilde{\alpha}_1)/\tilde{\alpha}_1} \to0. ρ CE ( θ t ∥ ) ≍ t − ( α ~ j ∗ CE − α ~ 1 ) / α ~ 1 → 0. Thus the top sharpness still decays on the t − 1 t^{-1} t − 1 scale, but it no longer controls all active orthogonal directions uniformly.
Proof. By the tail sharpness estimates in Proposition 11, for all sufficiently large θ ℓ \theta^\ell θ ℓ there exist constants 0 < c − ≤ c + < ∞ 0 < c_- \le c_+ < \infty 0 < c − ≤ c + < ∞ such that, in Loewner order on U \mathcal{U} U ,
c − ∑ j = 1 L e − α ~ j θ ℓ B j ⪯ H ( θ ℓ ) ⪯ c + ∑ j = 1 L e − α ~ j θ ℓ B j . c_- \sum_{j=1}^{L} e^{-\widetilde{\alpha}_j\theta^\ell} B_j \preceq H(\theta^\ell) \preceq c_+ \sum_{j=1}^{L} e^{-\widetilde{\alpha}_j\theta^\ell} B_j. c − j = 1 ∑ L e − α j θ ℓ B j ⪯ H ( θ ℓ ) ⪯ c + j = 1 ∑ L e − α j θ ℓ B j .
For the top eigenvalue, the upper bound follows from
H ( θ ℓ ) ⪯ c + e − α ~ 1 θ ℓ ∑ j = 1 L B j , H(\theta^\ell) \preceq c_+ e^{-\widetilde{\alpha}_1\theta^\ell} \sum_{j=1}^{L} B_j, H ( θ ℓ ) ⪯ c + e − α 1 θ ℓ j = 1 ∑ L B j ,
so
λ max ( H ( θ ℓ ) ∣ U ) ≲ e − α ~ 1 θ ℓ . \lambda_{\max}(H(\theta^\ell)|_{\mathcal{U}}) \lesssim e^{-\widetilde{\alpha}_1\theta^\ell}. λ m a x ( H ( θ ℓ ) ∣ U ) ≲ e − α 1 θ ℓ .
For the lower bound, since B 1 ≠ 0 B_1 \ne0 B 1 = 0 on the active directions seen by the slowest layer,
H ( θ ℓ ) ⪰ c − e − α ~ 1 θ ℓ B 1 , H(\theta^\ell) \succeq c_-e^{-\widetilde{\alpha}_1\theta^\ell}B_1, H ( θ ℓ ) ⪰ c − e − α 1 θ ℓ B 1 ,
and hence
λ max ( H ( θ ℓ ) ∣ U ) ≳ e − α ~ 1 θ ℓ . \lambda_{\max}(H(\theta^\ell)|_{\mathcal{U}}) \gtrsim e^{-\widetilde{\alpha}_1\theta^\ell}. λ m a x ( H ( θ ℓ ) ∣ U ) ≳ e − α 1 θ ℓ .
Thus
λ max ( H ( θ ℓ ) ∣ U ) ≍ e − α ~ 1 θ ℓ . \lambda_{\max}(H(\theta^\ell)|_{\mathcal{U}}) \asymp e^{-\widetilde{\alpha}_1\theta^\ell}. λ m a x ( H ( θ ℓ ) ∣ U ) ≍ e − α 1 θ ℓ .
For the smallest eigenvalue, let
j ∗ C E = min { j : B ≤ j ∣ U ≻ 0 } . j_*^{\mathrm{CE}} = \min\{j : B_{\leq j}|_{\mathcal{U}} \succ0\}. j ∗ CE = min { j : B ≤ j ∣ U ≻ 0 } .
Since B ≤ j ∗ C E ∣ U ≻ 0 B_{\leq j_*^{\mathrm{CE}}}|_{\mathcal{U}} \succ0 B ≤ j ∗ CE ∣ U ≻ 0 , we have
λ min ( B ≤ j ∗ C E ∣ U ) > 0. \lambda_{\min}(B_{\leq j_*^{\mathrm{CE}}}|_{\mathcal{U}}) > 0. λ m i n ( B ≤ j ∗ CE ∣ U ) > 0.
Using α ~ k ≤ α ~ j ∗ C E \widetilde{\alpha}_k \leq\widetilde{\alpha}_{j_*^{\mathrm{CE}}} α k ≤ α j ∗ CE for k ≤ j ∗ C E k \leq j_*^{\mathrm{CE}} k ≤ j ∗ CE , the lower Loewner bound gives
H ( θ ℓ ) ⪰ c − e − α ~ j ∗ C E θ ℓ B ≤ j ∗ C E , H(\theta^\ell) \succeq c_-e^{-\widetilde{\alpha}_{j_*^{\mathrm{CE}}}\theta^\ell}B_{\leq j_*^{\mathrm{CE}}}, H ( θ ℓ ) ⪰ c − e − α j ∗ CE θ ℓ B ≤ j ∗ CE ,
and therefore
λ min ( H ( θ ℓ ) ∣ U ) ≳ e − α ~ j ∗ C E θ ℓ . \lambda_{\min}(H(\theta^\ell)|_{\mathcal{U}}) \gtrsim e^{-\widetilde{\alpha}_{j_*^{\mathrm{CE}}}\theta^\ell}. λ m i n ( H ( θ ℓ ) ∣ U ) ≳ e − α j ∗ CE θ ℓ .
For the matching upper bound, if j ∗ C E > 1 j_*^{\mathrm{CE}} > 1 j ∗ CE > 1 , minimality of j ∗ C E j_*^{\mathrm{CE}} j ∗ CE implies that B ≤ j ∗ C E − 1 ∣ U B_{\leq j_*^{\mathrm{CE}}-1}|_{\mathcal{U}} B ≤ j ∗ CE − 1 ∣ U is not positive definite. Hence there exists a unit vector
Z ∈ U such that ⟨ Z , B ≤ j ∗ C E − 1 Z ⟩ F = 0. Z \in\mathcal{U}\quad\text{such that}\quad\langle Z,B_{\leq j_*^{\mathrm{CE}}-1}Z\rangle_F = 0. Z ∈ U such that ⟨ Z , B ≤ j ∗ CE − 1 Z ⟩ F = 0.
Since each B j ⪰ 0 B_j \succeq0 B j ⪰ 0 , this implies
⟨ Z , B j Z ⟩ F = 0 ∀ j < j ∗ C E . \langle Z,B_jZ\rangle_F = 0 \qquad\forall j < j_*^{\mathrm{CE}}. ⟨ Z , B j Z ⟩ F = 0 ∀ j < j ∗ CE .
Therefore
⟨ Z , H ( θ ℓ ) Z ⟩ F ≤ c + ∑ j = j ∗ C E L e − α ~ j θ ℓ ⟨ Z , B j Z ⟩ F ≲ e − α ~ j ∗ C E θ ℓ . \langle Z,H(\theta^\ell)Z\rangle_F \leq c_+\sum_{j=j_*^{\mathrm{CE}}}^{L} e^{-\widetilde{\alpha}_j\theta^\ell}\langle Z,B_jZ\rangle_F \lesssim e^{-\widetilde{\alpha}_{j_*^{\mathrm{CE}}}\theta^\ell}. ⟨ Z , H ( θ ℓ ) Z ⟩ F ≤ c + j = j ∗ CE ∑ L e − α j θ ℓ ⟨ Z , B j Z ⟩ F ≲ e − α j ∗ CE θ ℓ .
Taking the minimum over unit Z ∈ U Z \in\mathcal{U} Z ∈ U gives
λ min ( H ( θ ℓ ) ∣ U ) ≲ e − α ~ j ∗ C E θ ℓ . \lambda_{\min}(H(\theta^\ell)|_{\mathcal{U}}) \lesssim e^{-\widetilde{\alpha}_{j_*^{\mathrm{CE}}}\theta^\ell}. λ m i n ( H ( θ ℓ ) ∣ U ) ≲ e − α j ∗ CE θ ℓ .
The case j ∗ C E = 1 j_*^{\mathrm{CE}} = 1 j ∗ CE = 1 follows from
λ min ( H ) ≤ λ max ( H ) ≍ e − α ~ 1 θ ℓ \lambda_{\min}(H) \leq\lambda_{\max}(H) \asymp e^{-\widetilde{\alpha}_1\theta^\ell} λ m i n ( H ) ≤ λ m a x ( H ) ≍ e − α 1 θ ℓ
together with the lower bound already proved.
Dividing the smallest-eigenvalue estimate by the top-eigenvalue estimate gives
ρ C E ( θ ℓ ) ≍ e − ( α ~ j ∗ C E − α ~ 1 ) θ ℓ . \rho_{\mathrm{CE}}(\theta^\ell) \asymp e^{-(\widetilde{\alpha}_{j_*^{\mathrm{CE}}}-\widetilde{\alpha}_1)\theta^\ell}. ρ CE ( θ ℓ ) ≍ e − ( α j ∗ CE − α 1 ) θ ℓ .
Finally, the stable-tail schedule gives
e α ~ 1 ( θ t ℓ − θ ∗ ( r ) ) ≲ t , e^{\widetilde{\alpha}_1(\theta_t^\ell-\theta_*^{(r)})} \lesssim t, e α 1 ( θ t ℓ − θ ∗ ( r ) ) ≲ t ,
so
e − ( α ~ j ∗ C E − α ~ 1 ) θ t ℓ ≲ t − ( α ~ j ∗ C E − α ~ 1 ) / α ~ 1 . e^{-(\widetilde{\alpha}_{j_*^{\mathrm{CE}}}-\widetilde{\alpha}_1)\theta_t^\ell} \lesssim t^{-(\widetilde{\alpha}_{j_*^{\mathrm{CE}}}-\widetilde{\alpha}_1)/\widetilde{\alpha}_1}. e − ( α j ∗ CE − α 1 ) θ t ℓ ≲ t − ( α j ∗ CE − α 1 ) / α 1 .
This proves the theorem.
Experimental Details
Compute resources. All experiments are low-dimensional synthetic simulations run with full-batch gradient descent. They require only standard CPU computation, no GPU or specialized accelerator, and use negligible memory; each run completes in under a few seconds on a typical laptop.
Experimental Setup
Spanning experiment (§ 6 , Figure 3 ). We use pair-symmetric logistic regression in R 3 \mathbb{R}^{3} R 3 , with v = e 1 v=e_{1} v = e 1 and active orthogonal space U = span { e 2 , e 3 } U=\operatorname{span}\{e_{2},e_{3}\} U = span { e 2 , e 3 } . The dataset has N = 24 N=24 N = 24 reflected pairs. The slowest layer has N s l o w = 10 N_{\mathrm{slow}}=10 N slow = 10 pairs with a i = a min = 0.20 a_i=a_{\min}=0.20 a i = a m i n = 0.20 , and the faster layers have N f a s t = 14 N_{\mathrm{fast}}=14 N fast = 14 pairs with
a i ∈ { 0.35 , 0.45 , 0.55 } . a_i \in\{0.35,0.45,0.55\}. a i ∈ { 0.35 , 0.45 , 0.55 } .
The slowest layer is constructed to span U U U : two slow vectors are fixed as
u 1 = ( 1.8 , 0 ) , u 2 = ( 0 , 0.5 ) , u_1=(1.8,0),\qquad u_2=(0,0.5), u 1 = ( 1.8 , 0 ) , u 2 = ( 0 , 0.5 ) ,
and the remaining slow vectors, together with the faster-layer vectors, are sampled using random seed 1. Specifically, their directions are uniform on [ 0 , 2 π ] [0,2\pi] [ 0 , 2 π ] , their radii are uniform on [ 0.5 , 1.8 ] [0.5,1.8] [ 0.5 , 1.8 ] , and the faster-layer a i a_i a i ’s are sampled uniformly from { 0.35 , 0.45 , 0.55 } \{0.35,0.45,0.55\} { 0.35 , 0.45 , 0.55 } .
We use step size η = 15 \eta=15 η = 15 , initialize at
θ 0 ∥ = 0 , θ 0 ⊥ = ( 0.01 , 0.006 ) , \theta_{0}^{\parallel}=0,\qquad\theta_{0}^{\perp}=(0.01,0.006), θ 0 ∥ = 0 , θ 0 ⊥ = ( 0.01 , 0.006 ) ,
and run full-batch gradient descent for T = 100 T=100 T = 100 iterations. The prescribed tubular neighborhood radius is r t u b u l a r n e i g h b o r h o o d = 0.15 r_{\mathrm{tubularneighborhood}}=0.15 r tubularneighborhood = 0.15 . The RC tolerance is computed as
δ r = cosh ( U max r t u b u l a r n e i g h b o r h o o d ) − 1 , U max : = max i ∥ u i ∥ , \delta_r=\cosh(U_{\max}r_{\mathrm{tubularneighborhood}})-1,\qquad U_{\max}:=\max_i\lVert u_i\rVert, δ r = cosh ( U m a x r tubularneighborhood ) − 1 , U m a x := i max ∥ u i ∥ ,
and the tail-conditioning constant is
ρ t a i l = λ min ( Σ s l o w ) λ max ( Σ U ) , Σ U = 1 N ∑ i = 1 N u i u i T , Σ s l o w = 1 N ∑ i : a i = a min u i u i T . \rho_{\mathrm{tail}}=\frac{\lambda_{\min}(\Sigma_{\mathrm{slow}})}{\lambda_{\max}(\Sigma_U)},\qquad
\Sigma_U=\frac{1}{N}\sum_{i=1}^{N}u_i u_i^{\mathsf T},\qquad
\Sigma_{\mathrm{slow}}=\frac{1}{N}\sum_{i:a_i=a_{\min}}u_i u_i^{\mathsf T}. ρ tail = λ m a x ( Σ U ) λ m i n ( Σ slow ) , Σ U = N 1 i = 1 ∑ N u i u i T , Σ slow = N 1 i : a i = a m i n ∑ u i u i T .
For this instance, δ r = 0.037 \delta_r=0.037 δ r = 0.037 , ρ t a i l = 0.200 \rho_{\mathrm{tail}}=0.200 ρ tail = 0.200 , ϵ ∗ = 0.186 \epsilon_*=0.186 ϵ ∗ = 0.186 , and the observed maximum tubular neighborhood radius is r o b s = 0.13 < r t u b u l a r n e i g h b o r h o o d r_{\mathrm{obs}}=0.13<r_{\mathrm{tubularneighborhood}} r obs = 0.13 < r tubularneighborhood . The phase boundaries are computed from the corrected stability product
η ( 1 + δ r ) λ 1 ( θ t ∥ ) , \eta(1+\delta_r)\lambda_1(\theta_t^{\parallel}), η ( 1 + δ r ) λ 1 ( θ t ∥ ) ,
with t s t a b t_{\mathrm{stab}} t stab the first time this product is at most 2, t l i n t_{\mathrm{lin}} t lin the first time it is at most 2 − ϵ ∗ 2-\epsilon_* 2 − ϵ ∗ , and t p o l y t_{\mathrm{poly}} t poly the first time it is at most 1.
Non-spanning experiment (§ 6 , Figure 4 ). We use pair-symmetric logistic regression in R 3 \mathbb{R}^{3} R 3 , with signed-feature coordinates z = y x z=yx z = y x and empirical loss
L ( θ ) = 1 M ∑ j = 1 M log ( 1 + exp ( − ⟨ θ , z j ⟩ ) ) . \mathcal{L}(\theta)=\frac{1}{M}\sum_{j=1}^{M}\log(1+\exp(-\langle\theta,z_j\rangle)). L ( θ ) = M 1 j = 1 ∑ M log ( 1 + exp ( − ⟨ θ , z j ⟩)) .
The max-margin direction is v = e 1 v=e_1 v = e 1 , and the active orthogonal space is U = span { e 2 , e 3 } U=\operatorname{span}\{e_2,e_3\} U = span { e 2 , e 3 } . The training set consists of reflected signed-feature pairs z i , ± = a i e 1 ± u i z_{i,\pm}=a_i e_1\pm u_i z i , ± = a i e 1 ± u i . We use 5 support pairs with a i = α 1 = 0.1 a_i=\alpha_1=0.1 a i = α 1 = 0.1 and u i = r i e 2 u_i=r_i e_2 u i = r i e 2 , and 5 weak-layer pairs with a i = α 2 = 1.0 a_i=\alpha_2=1.0 a i = α 2 = 1.0 and u i = r i e 3 u_i=r_i e_3 u i = r i e 3 , for M = 20 M=20 M = 20 signed features in total. The radii are generated with random seed 3 by
r i = 2 ( 0.5 + 0.5 ξ i ) , ξ i ∼ Unif [ 0 , 1 ] , r_i=2(0.5+0.5\xi_i),\qquad\xi_i\sim\operatorname{Unif}[0,1], r i = 2 ( 0.5 + 0.5 ξ i ) , ξ i ∼ Unif [ 0 , 1 ] ,
independently for each reflected pair. Thus the orthogonal components of the support vectors span only the e 2 e_2 e 2 -direction, while the weak e 3 e_3 e 3 -direction appears only in the faster-decaying layer. We initialize at
θ 0 = ( 0.8 , 0.20 , 0.05 ) \theta_0=(0.8,0.20,0.05) θ 0 = ( 0.8 , 0.20 , 0.05 )
and run full-batch gradient descent for T = 150 T=150 T = 150 iterations. The baseline step size is
η c : = 2 λ max ( ∇ 2 L ( θ 0 ) ∣ U ) . \eta_c:=\frac{2}{\lambda_{\max}(\left.\nabla^2\mathcal{L}(\theta_0)\right|_U)}. η c := λ m a x ( ∇ 2 L ( θ 0 ) ∣ U ) 2 .
where the Hessian is restricted to the active orthogonal coordinates ( e 2 , e 3 ) (e_2,e_3) ( e 2 , e 3 ) . We sweep η / η c ∈ { 1 , 1.5 , 2 , 2.5 , 3 } \eta/\eta_c \in\{1, 1.5, 2, 2.5, 3\} η / η c ∈ { 1 , 1.5 , 2 , 2.5 , 3 } .
At each time t t t , we record the loss, the parallel coordinate θ t ∥ = ⟨ θ t , e 1 ⟩ \theta_t^{\parallel} = \langle\theta_t, e_1\rangle θ t ∥ = ⟨ θ t , e 1 ⟩ , the support-vector component c s v , t = ⟨ θ t , e 2 ⟩ c_{\mathrm{sv},t} = \langle\theta_t, e_2\rangle c sv , t = ⟨ θ t , e 2 ⟩ , and the weak component c w k , t = ⟨ θ t , e 3 ⟩ c_{\mathrm{wk},t} = \langle\theta_t, e_3\rangle c wk , t = ⟨ θ t , e 3 ⟩ . The restricted Hessian is evaluated at ( θ t ∥ , 0 , 0 ) (\theta_t^{\parallel},0,0) ( θ t ∥ , 0 , 0 ) , and the conditioning ratio is computed as
ρ t = λ min ( H ( θ t ∥ ) ∣ U ) λ max ( H ( θ t ∥ ) ∣ U ) . \rho_t = \frac{\lambda_{\min}(H(\theta_t^{\parallel})|_U)}{\lambda_{\max}(H(\theta_t^{\parallel})|_U)}. ρ t = λ m a x ( H ( θ t ∥ ) ∣ U ) λ m i n ( H ( θ t ∥ ) ∣ U ) .
The angle to the max-margin line is
arctan ( c s v , t 2 + c w k , t 2 ∣ θ t ∥ ∣ ) , \arctan\left(\frac{\sqrt{c_{\mathrm{sv},t}^{2}+c_{\mathrm{wk},t}^{2}}}{|\theta_t^{\parallel}|}\right), arctan ∣ θ t ∥ ∣ c sv , t 2 + c wk , t 2 ,
reported in degrees.
Geometry-Dependent Step-Size Effects in Non-Spanning Configurations
Section 6 illustrates one regime of step-size behavior in non-spanning geometries: larger η \eta η amplifies the persistent orthogonal residual measured at a finite horizon T T T . This appendix complements that experiment with two additional regimes, both arising in non-spanning configurations: residual suppression by larger η \eta η (Figure 5) and non-monotone dependence with an interior minimum (Figure 6). Together with the main-text experiment, these three panels demonstrate that the step-size dependence of the residual is governed by the data geometry rather than by a single universal law. All three experiments use the same pair-symmetric logistic regression template as in Section 6: d = 3 d = 3 d = 3 , N = 20 N = 20 N = 20 data points, and identical initialization protocol. The configurations differ only in the values of α 2 \alpha_2 α 2 . Specific parameter values are listed in the figure captions. Across all panels we sweep five step sizes spanning the unstable range; the smallest is in the classically stable regime and the largest is well above 2 / λ max ( θ 0 ) 2/\lambda_{\max}(\theta_0) 2/ λ m a x ( θ 0 ) .
Downstream effect of the orthogonal residual
The non-spanning experiments above show that weakly controlled orthogonal components can persist at any finite horizon. To illustrate that this residual can have a downstream effect, we construct a simple distribution-shift test on the same non-spanning logistic geometry used in § 6 . We use the pair-symmetric model in R 3 \mathbb{R}^3 R 3 with support vectors spanning only e 2 e_2 e 2 (margin scale α 1 \alpha_1 α 1 ) and non-support vectors along e 3 e_3 e 3 (margin scale α 2 > α 1 \alpha_2 > \alpha_1 α 2 > α 1 ), so that e 3 e_3 e 3 is the weakly controlled “spur” direction. An informal analogy: e 1 e_1 e 1 encodes a robust class feature (e.g., animal shape in a cats-vs-dogs classifier), while e 3 e_3 e 3 encodes a spurious cue (e.g., background texture) that correlates with the label in training but not at test time. We evaluate the trained classifier on a shifted test distribution in which the sign of the e 3 e_3 e 3 component is flipped relative to training: a classifier with spur residual θ s p u r = ⟨ θ T , e 3 ⟩ \theta_{\mathrm{spur}} = \langle\theta_T, e_3 \rangle θ spur = ⟨ θ T , e 3 ⟩ sees its effective margin reduced by 2 ∣ θ s p u r ∣ 2|\theta_{\mathrm{spur}}| 2∣ θ spur ∣ on this shifted distribution. Figure 7 shows that the non-spanning geometry leaves a larger spur residual and a larger finite-time angle to the max-margin direction, while the spanning control suppresses this residual. This illustrates, in a controlled low-dimensional example, how a finite-time spur residual can affect downstream test margins under a shift that reverses the spur coordinate. We emphasize that this is an illustrative low-dimensional example; understanding the scope of this phenomenon in more realistic settings is left to future work.
Figure 7. Non-spanning support geometry produces learning-rate-dependent residuals in pair-symmetric logistic regression (R 3 \mathbb{R}^{3} R 3 , T = 20,000 T = 20{,}000 T = 20 , 000 steps). (A ) Signed-feature schematics. Reflected pairs z i , ± = a i e 1 ± u i z_{i,\pm} = a_i e_1 \pm u_i z i , ± = a i e 1 ± u i are connected by dashed lines. The slow support group (orange, a = 1 a = 1 a = 1 ) spans only e 2 e_2 e 2 ; the fast spur group (blue, a = 4 a = 4 a = 4 ) spans e 3 e_3 e 3 . In the spanning control, an additional slow group (purple) adds support in e 3 e_3 e 3 . Both geometries share the same max-margin line e 1 e_1 e 1 . (B ) Final spur residual ∣ ⟨ θ T , e 3 ⟩ ∣ |\langle\theta_T,e_3\rangle| ∣ ⟨ θ T , e 3 ⟩ ∣ versus learning rate. The non-spanning geometry retains a systematically larger residual, with the gap widening at large η \eta η where unstable amplification is strongest. (C ) Angle between θ T \theta_T θ T and the max-margin direction. The non-spanning residual translates into a larger finite-time deviation from the asymptotic classifier, confirming that top-eigenvalue stabilization alone does not guarantee uniform contraction of all orthogonal components.
Implementation details. We use pair-symmetric logistic regression in R 3 \mathbb{R}^3 R 3 , with v = e 1 v = e_1 v = e 1 as the max-margin direction and U = span { e 2 , e 3 } U = \operatorname{span}\{e_2,e_3\} U = span { e 2 , e 3 } as the active orthogonal space. We work in signed-feature coordinates z = y x z = yx z = y x , so the logistic loss is
L ( θ ) = 1 M ∑ j = 1 M ϕ ( ⟨ θ , z j ⟩ ) , ϕ ( s ) = log ( 1 + e − s ) . \mathcal{L}(\theta) = \frac{1}{M}\sum_{j=1}^{M}\phi(\langle\theta,z_j\rangle), \qquad\phi(s) = \log(1 + e^{-s}). L ( θ ) = M 1 j = 1 ∑ M ϕ (⟨ θ , z j ⟩) , ϕ ( s ) = log ( 1 + e − s ) .
The training set consists of reflected signed-feature pairs z i , ± = a i e 1 ± u i z_{i,\pm} = a_i e_1 \pm u_i z i , ± = a i e 1 ± u i . In the non-spanning geometry, the slow support group has a 1 = 1 a_1 = 1 a 1 = 1 and u 1 = e 2 u_1 = e_2 u 1 = e 2 , with 40 reflected pairs (80 signed features), while the fast spur group has a 2 = 4 a_2 = 4 a 2 = 4 and u 2 = 2 e 3 u_2 = 2e_3 u 2 = 2 e 3 , also with 40 reflected pairs (80 signed features). Thus the non-spanning training set has M = 160 M = 160 M = 160 signed features. The spanning control appends a third group at the slow decay rate a 3 = 1 a_3 = 1 a 3 = 1 with u 3 = 0.8 e 3 u_3 = 0.8e_3 u 3 = 0.8 e 3 , using 40 reflected pairs, bringing the total to M = 240 M = 240 M = 240 signed features.
In both geometries, the max-margin direction is e 1 e_1 e 1 by symmetry. The two settings differ only in whether the slowest-decay support group spans the spur direction e 3 e_3 e 3 . We initialize at
θ 0 = ( 0.1 , 0.15 , 0.15 ) \theta_0 = (0.1, 0.15, 0.15) θ 0 = ( 0.1 , 0.15 , 0.15 )
and run full-batch gradient descent for T = 20,000 T = 20{,}000 T = 20 , 000 steps at each of ten learning rates
η ∈ { 0.01 , 0.02 , 0.05 , 0.1 , 0.2 , 0.5 , 1 , 2 , 5 , 10 } . \eta\in\{0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10\}. η ∈ { 0.01 , 0.02 , 0.05 , 0.1 , 0.2 , 0.5 , 1 , 2 , 5 , 10 } .
For the downstream test, we evaluate two near-boundary signed-feature clouds. The aligned cloud consists of 600 signed features sampled from a Gaussian with mean ( 1 , 0 , + 2 ) (1, 0, +2) ( 1 , 0 , + 2 ) and coordinatewise standard deviation 0.02. The shifted cloud is sampled analogously with mean ( 1 , 0 , − 2 ) (1, 0, -2) ( 1 , 0 , − 2 ) , so its spur coordinate has the opposite sign. We report the final spur residual
∣ θ s p u r ( T ) ∣ = ∣ ⟨ θ T , e 3 ⟩ ∣ , |\theta_{\mathrm{spur}}(T)| = |\langle\theta_T,e_3\rangle|, ∣ θ spur ( T ) ∣ = ∣ ⟨ θ T , e 3 ⟩ ∣ ,
the angle to the max-margin line
arctan ( ∥ Π U θ T ∥ ∣ ⟨ θ T , e 1 ⟩ ∣ ) , \arctan\left(\frac{\lVert\Pi_U\theta_T\rVert}{|\langle\theta_T,e_1\rangle|}\right), arctan ( ∣ ⟨ θ T , e 1 ⟩ ∣ ∥ Π U θ T ∥ ) ,
and the empirical logistic loss on the aligned and shifted test clouds.
Approximate Symmetry and Extensions
Exact pair-symmetry yields the clean multiplicative recursion θ t + 1 ⊥ = ( I − η H ~ ) θ t ⊥ \theta_{t+1}^{\perp} = (I-\eta\widetilde{H})\theta_t^{\perp} θ t + 1 ⊥ = ( I − η H ) θ t ⊥ by making the max-margin direction coincide with the subspace span { v } \operatorname{span}\{v\} span { v } and freezing the inactive subspace. For a general separable dataset, one instead decomposes along the hard-margin SVM direction v s v m v_{\mathrm{svm}} v svm , producing a perturbed recursion
θ t + 1 ⊥ = ( I − η H ( θ t ∥ ) ) θ t ⊥ + Err t , \theta_{t+1}^{\perp} = \left(I-\eta H(\theta_t^{\parallel})\right)\theta_t^{\perp}+\operatorname{Err}_t, θ t + 1 ⊥ = ( I − η H ( θ t ∥ ) ) θ t ⊥ + Err t ,
where Err t \operatorname{Err}_t Err t measures the failure of exact block-alignment, controlled by the off-diagonal quantities ∥ Π v s v m ⊥ ∇ L ( θ ∥ v s v m ) ∥ \lVert\Pi_{v_{\mathrm{svm}}}^{\perp}\nabla\mathcal{L}(\theta^{\parallel}v_{\mathrm{svm}})\rVert ∥ Π v svm ⊥ ∇ L ( θ ∥ v svm )∥ and ∥ Π v s v m ∥ ∇ 2 L ( θ ∥ v s v m ) Π v s v m ⊥ ∥ \lVert\Pi_{v_{\mathrm{svm}}}^{\parallel}\nabla^2\mathcal{L}(\theta^{\parallel}v_{\mathrm{svm}})\Pi_{v_{\mathrm{svm}}}^{\perp}\rVert ∥ Π v svm ∥ ∇ 2 L ( θ ∥ v svm ) Π v svm ⊥ ∥ . If these remain small on the relevant tubular neighborhood, the same analytical strategy applies, and the orthogonal component converges to an asymmetry-dependent residual rather than to zero. The symmetric setting is thus the exactly aligned limit of a broader perturbative theory whose development we leave to future work. To test whether this mechanism is brittle, we also consider a small asymmetric perturbation of the pair-symmetric logistic construction. We compute the hard-margin SVM direction v s v m v_{\mathrm{svm}} v svm of the perturbed dataset and decompose
θ t = θ t ∥ v s v m + θ t ⊥ , θ t ⊥ ⊥ v s v m . \theta_t=\theta_t^{\parallel}v_{\mathrm{svm}}+\theta_t^{\perp},\qquad\theta_t^{\perp}\perp v_{\mathrm{svm}}. θ t = θ t ∥ v svm + θ t ⊥ , θ t ⊥ ⊥ v svm .
In this setting the SVM line is no longer exactly invariant. Therefore the orthogonal component need not contract to zero. Instead, the linearized orthogonal dynamics have the instantaneous center
c t l i n = − H ⊥ ( θ t ∥ ) − 1 Π v s v m ⊥ ∇ L ( θ t ∥ v s v m ) , c_t^{\mathrm{lin}}=-H_{\perp}(\theta_t^{\parallel})^{-1}\Pi_{v_{\mathrm{svm}}}^{\perp}\nabla\mathcal{L}(\theta_t^{\parallel}v_{\mathrm{svm}}), c t lin = − H ⊥ ( θ t ∥ ) − 1 Π v svm ⊥ ∇ L ( θ t ∥ v svm ) ,
where H ⊥ ( θ t ∥ ) H_{\perp}(\theta_t^{\parallel}) H ⊥ ( θ t ∥ ) is the Hessian restricted to v s v m ⊥ v_{\mathrm{svm}}^{\perp} v svm ⊥ . Figure 8 shows that the stability product still self-stabilizes, while the centered residual ∥ θ t ⊥ − c t l i n ∥ \lVert\theta_t^{\perp}-c_t^{\mathrm{lin}}\rVert ∥ θ t ⊥ − c t lin ∥ contracts. Thus the exact symmetric analysis appears to describe the aligned limit of a perturbed recursion, where stability controls deviations from a moving center rather than the raw orthogonal norm itself. This experiment is illustrative and is not covered by the theorem.
Figure 8. Small asymmetric perturbation of the pair-symmetric logistic model.
Experimental setup. We study a controlled synthetic aligned-feature logistic-regression problem in three dimensions, where each example is represented in aligned coordinates as Z i = y i x i ∈ R 3 Z_i=y_ix_i\in\mathbb{R}^3 Z i = y i x i ∈ R 3 . The base dataset is pair-symmetric around the e 1 e_1 e 1 direction. It contains a slow layer of six anchor points with parallel coordinate a s l o w = 0.08 a_{\mathrm{slow}}=0.08 a slow = 0.08 and transverse coordinates ( ± 1 , 0 ) (\pm1,0) ( ± 1 , 0 ) , ( 0 , ± 1 ) (0,\pm1) ( 0 , ± 1 ) , and ( ± 0.7 , ± 0.7 ) (\pm0.7,\pm0.7) ( ± 0.7 , ± 0.7 ) , together with a faster layer of eight reflected pairs with parallel coordinate a f a s t = 0.28 a_{\mathrm{fast}}=0.28 a fast = 0.28 . For each fast pair, the orthogonal direction is sampled from a normalized Gaussian draw and its radius is sampled uniformly from [ 0.25 , 1.10 ] [0.25,1.10] [ 0.25 , 1.10 ] . This produces a dataset of N = 22 N=22 N = 22 aligned examples.
To move away from the exactly symmetric regime, we introduce a controlled asymmetry parameter ε a s y m = 0.04 \varepsilon_{\mathrm{asym}}=0.04 ε asym = 0.04 . Specifically, we perturb both the parallel coordinate and the transverse coordinates of every example by Gaussian noise scaled by ε a s y m \varepsilon_{\mathrm{asym}} ε asym , while enforcing that all parallel coordinates remain positive so that the dataset stays separable. We then compute an approximate hard-margin direction v s v m v_{\mathrm{svm}} v svm directly in aligned coordinates and decompose the iterate as
θ t = θ t ∥ v s v m + B θ t ⊥ , \theta_t=\theta_t^{\parallel}v_{\mathrm{svm}}+B\theta_t^{\perp}, θ t = θ t ∥ v svm + B θ t ⊥ ,
where the columns of B B B form an orthonormal basis of v s v m ⊥ v_{\mathrm{svm}}^{\perp} v svm ⊥ .
We optimize the mean logistic loss with full-batch gradient descent using a fixed step size η = 35 \eta=35 η = 35 for T = 100 T=100 T = 100 iterations. Initialization is set to zero in the parallel direction and to a random orthogonal component of norm 0.025 0.025 0.025 . Unless otherwise stated, all experiments use random seed 3.