Introduction
The implicit bias or implicit regularization of optimizers refers to their tendency to favor solutions with certain properties, often associated with generalization, even when no explicit regularization is used in training. On classification tasks and homogeneous models (e.g. deep ReLU networks), optimizers such as gradient descent and the steepest descent family have been shown to possess margin-maximization implicit biases, selecting among all perfect-accuracy solutions those which maximize a certain confidence margin ([35 , 45 ] ).
Adam ([27 ] ) and its derivatives are popular optimizers for training large language models (LLMs) and other deep neural networks. An improved understanding of their implicit biases is therefore essential to strengthening the theoretical foundations of modern AI. Earlier works ([53 , 15 , 19 ] ) on linear and smooth homogeneous models trained with exponentially-tailed losses showed that Adam has a margin-maximization bias towards the ℓ ∞ \ell_\infty ℓ ∞ margin when the learning rate schedule η ( t ) \eta(t) η ( t ) undergoes a strong decay η ( t ) = o ( t 1 / L − 1 ) \eta(t) = o\left(t^{1/L-1}\right) η ( t ) = o ( t 1/ L − 1 ) . Here, L L L is the homogeneity degree, equal to the network depth for linear or ReLU networks. Moreover, these works show similarity between late-phase Adam and sign gradient descent (sign-GD ), which also has a bias towards ℓ ∞ \ell_\infty ℓ ∞ -margin maximization ([45 ] ). More broadly, the similarity between Adam and sign-GD (including variations with momentum) is assumed, derived or empirically tested in many prior works on optimization geometry, generalization, and optimizer design ([3 , 54 , 34 , 31 , 30 , 33 , 6 ] ).
The schedule η ( t ) \eta(t) η ( t ) plays a key role, since it changes the effective memory horizon along the trajectory when the momentum hyperparameters β 1 , β 2 \beta_1,\beta_2 β 1 , β 2 remain fixed ([5 ] ), as is the case in practice. Motivated by the common constant and weakly decaying schedules, we extend the characterization of Zhang et al. [53 ] ; Gronich and Vardi [19 ] to η ( t ) ≍ t α \eta(t) \asymp t^\alpha η ( t ) ≍ t α with α ≥ 1 L − 1 \alpha\ge\frac{1}{L} - 1 α ≥ L 1 − 1 , obtaining the following surprising results:
When α > 1 L − 1 2 \alpha> \frac{1}{L} - \frac{1}{2} α > L 1 − 2 1 , including a constant learning rate for L > 2 L > 2 L > 2 , we prove nonexistence results for implicit regularizers of Adam. We formalize this by studying regularizers inspired by squared norms: locally Lipschitz functions R ( θ ) R(\theta) R ( θ ) , where θ \theta θ are the network’s parameters, satisfying for all separable datasets { ( x i , y i ) } \{(x_i,y_i)\} {( x i , y i )} and Adam trajectories that directional limit points of θ \theta θ are proportional to KKT points of
min θ R ( θ ) s.t. ∀ i : y i f ( x i ; θ ) ≥ 1. \min_{\theta} R(\theta) \quad\text{s.t.} \quad\forall i : y_i f(x_i;\theta) \ge1 . θ min R ( θ ) s.t. ∀ i : y i f ( x i ; θ ) ≥ 1.
(a) We first show that no non-constant R ( θ ) R(\theta) R ( θ ) satisfies this definition for width-1 linear networks. Thus, Adam does not maximize any margin in width-1 linear networks.
(b) We then lift the construction to a deep linear network with general widths, proving triviality of any convex regularizer.
Both results rule out squared norms as implicit regularizers of Adam, implying there is no margin-maximization bias in the usual sense. In particular, Adam’s late-stage behavior differs significantly from that of sign-GD.
Despite the above results implying that no single R ( θ ) R(\theta) R ( θ ) regularizes all trajectories, we show on the flip side that Adam has a trajectory-dependent margin-maximization implicit bias. Specifically, directional limit points of Adam are KKT points of a weighted-ℓ 2 \ell_2 ℓ 2 margin-maximization objective defined by a trajectory-dependent preconditioning diagonal matrix. This positive result, as well the negative ones, rely on a key mechanism of convergence of Adam’s second-moment preconditioner to a trajectory-dependent limit.
In the intermediate range 1 L − 1 ≤ α ≤ 1 L − 1 2 \frac{1}{L} - 1 \le\alpha\le\frac{1}{L} - \frac{1}{2} L 1 − 1 ≤ α ≤ L 1 − 2 1 , we extend the result of Gronich and Vardi [19 ] , and prove that Adam on homogeneous models maximizes the ℓ ∞ \ell_\infty ℓ ∞ margin. Our result is conditional on gradient directional convergence to a full-support vector.
Of independent interest, we establish a loss decay rate for (full batch) Adam on general homogeneous models, whose leading order depends only on the second-moment parameter:
− log L = c 2 2 t ± o ( t ) , ( α > 1 L − 1 ) , (1) -\log\mathcal{L} = \frac{c_2}{2}t \pm o(t), \qquad\left(\alpha> \frac{1}{L} - 1\right), \tag*{(1)} − log L = 2 c 2 t ± o ( t ) , ( α > L 1 − 1 ) , (1)
− log L = c 2 2 t ± O ( log t ) , ( α > 1 L − 1 2 ) , (2) -\log\mathcal{L} = \frac{c_2}{2}t \pm O(\log t), \qquad\left(\alpha> \frac{1}{L} - \frac{1}{2}\right), \tag*{(2)} − log L = 2 c 2 t ± O ( log t ) , ( α > L 1 − 2 1 ) , (2)
where c 2 c_2 c 2 is the continuous analogue of − log β 2 -\log\beta_2 − log β 2 (see § 3 for definitions). This both shows a direct connection between the value of Adam’s β 2 \beta_2 β 2 parameter and the loss decay rate, and demonstrates that this asymptotic rate is essentially independent of the learning rate schedule. In contrast, earlier results for the regime α < 1 L − 1 \alpha< \frac{1}{L} - 1 α < L 1 − 1 give the schedule-dependent rate − log L ≍ t L ( α + 1 ) = o ( t ) -\log\mathcal{L} \asymp t^{L(\alpha+1)} = o(t) − log L ≍ t L ( α + 1 ) = o ( t ) ([53 , 19 ] ); our analysis also derives this rate, under weaker assumptions.
Related work
An ℓ 2 \ell_2 ℓ 2 -margin-maximization implicit bias of gradient descent was first proved on linear predictors by Soudry et al. [43 ] , and subsequent work investigated how architecture and initialization shape the implicit bias of certain deep networks ([24 , 21 , 52 ] (Moroshko et al. 2020)). General homogeneous models were studied by Lyu and Li [35 ] , who showed that any directional limit point of the trajectory is the direction of a KKT point of the ℓ 2 \ell_2 ℓ 2 -max-margin problem, while Ji and Telgarsky [25 ] proved for homogeneous models definable in an o-minimal structure that the parameter direction converges under gradient descent. See Vardi [46 ] for a survey of the implicit bias of gradient descent.
A common family of non-Euclidean optimizers is steepest descent , whose special cases include sign gradient descent and spectral descent (with momentum: Signum [7 ] and idealized Muon [26 ] , respectively). Gunasekar et al. [20 ] proved a margin-maximization bias for steepest descent on linear models, where the margin geometry is not ℓ 2 \ell_2 ℓ 2 but the norm defining the optimizer. This was generalized by Tsilivis et al. [45 ] to homogeneous models, showing that any limit point of θ t ∥ θ t ∥ \frac{\theta_t}{\lVert\theta_t\rVert} ∥ θ t ∥ θ t is the direction of a KKT point of the appropriate max-margin problem. Specifically, this result implies that sign-GD has a bias towards ℓ ∞ \ell_\infty ℓ ∞ -margin maximization.
On Adam, Wang et al. [48 ] and Wang et al. [49 ] first established a bias toward ℓ 2 \ell_2 ℓ 2 -margin maximization. Their analysis considers a regime in which Adam’s stability constant ε \varepsilon ε asymptotically dominates the denominator, leading to gradient-descent-like behavior. Analyses omitting the stability constant include the works of Zhang et al. [53 ] and Fan et al. [15 ] establishing ℓ ∞ \ell_\infty ℓ ∞ -margin maximization on linear models (in the binary and multiclass settings, respectively), with a decaying learning rate, while Baek et al. [1 ] showed that deterministic batching can induce an implicit bias different from that of full-batch Adam. Recently, Gronich and Vardi [19 ] established an ℓ ∞ \ell_\infty ℓ ∞ bias for Adam on smooth, homogeneous networks under a learning-rate schedule satisfying η ( t ) = o ( t 1 L − 1 ) \eta(t)=o\left(t^{\frac{1}{L}-1}\right) η ( t ) = o ( t L 1 − 1 ) , where L L L denotes the degree of homogeneity.
For AdamW, which incorporates explicit weight decay, Xie and Li [51 ] studied smooth models and losses and showed that trajectory limit points satisfy the KKT conditions for loss minimization subject to a bound on the ℓ ∞ \ell_\infty ℓ ∞ norm of the parameters.
Impossibility results for implicit bias have been proved before by Vardi and Shamir [47 ] for squared-loss regression with gradient flow on ReLU networks, and by Dauber et al. [13 ] ; Suggala et al. [44 ] for other learning problems. Qian and Qian [42 ] and Wang et al. [48 ] observed a converging, trajectory-dependent preconditioner for AdaGrad in binary classification (on linear and homogeneous models, respectively), and Bai et al. [2 ] noticed the mechanism of second-moment convergence (which they term “decoupling”) in Adam on highly degenerate polynomials.
Problem setting
Throughout this work, we consider a binary classification task defined by a dataset { ( x i , y i ) } i = 1 m ⊆ R d × { − 1 , 1 } \{(\mathbf{x}_i,y_i)\}_{i=1}^{m}\subseteq\mathbb{R}^d\times\{-1,1\} {( x i , y i ) } i = 1 m ⊆ R d × { − 1 , 1 } . We study models f ( x ; θ ) f(\mathbf{x};\theta) f ( x ; θ ) , parameterized by θ ∈ R p \theta\in\mathbb{R}^p θ ∈ R p , which satisfy the following assumptions:
(M1) f f f is locally Lipschitz in θ \theta θ and admits a chain rule (see Appendix § 10 ).
(M2) f f f is L L L -homogeneous for L ≥ 1 L\geq1 L ≥ 1 : ∀ z > 0 , x ∈ R d , θ ∈ R p : f ( x ; z θ ) = z L f ( x ; θ ) \forall z>0,\mathbf{x}\in\mathbb{R}^d,\theta\in\mathbb{R}^p: \ f(\mathbf{x};z\theta)=z^L f(\mathbf{x};\theta) ∀ z > 0 , x ∈ R d , θ ∈ R p : f ( x ; z θ ) = z L f ( x ; θ ) .
For example, deep linear networks, ReLU networks and convolutional networks (all without bias terms) of depth L L L satisfy these conditions. The classification margin of a homogeneous model, with respect to a norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ , is defined as
γ ( θ ) = min i ∈ [ m ] y i f ( x i ; θ ∥ θ ∥ ) . (3) \gamma(\theta)=\min_{i\in[m]}y_i f\left(\mathbf{x}_i;\frac{\theta}{\lVert\theta\rVert}\right). \tag*{(3)} γ ( θ ) = i ∈ [ m ] min y i f ( x i ; ∥ θ ∥ θ ) . (3)
We write q i ( θ ) = y i f ( x i ; θ ) q_i(\theta)=y_i f(\mathbf{x}_i;\theta) q i ( θ ) = y i f ( x i ; θ ) and q min ( θ ) = min i q i ( θ ) q_{\min}(\theta)=\min_i q_i(\theta) q m i n ( θ ) = min i q i ( θ ) as shorthand.
We analyze Adam, where we follow Zhang et al. [53 ] ; Fan et al. [15 ] ; Gronich and Vardi [19 ] ; Baek et al. [1 ] , among others, by setting the stability constant ε \varepsilon ε to 0 0 0 . This choice is motivated by the fact that in asymptotic analysis, a positive ε \varepsilon ε dominates the decaying denominator v ^ t \sqrt{\hat{v}_t} v ^ t , while in practice ε \varepsilon ε typically remains negligible throughout optimization. We also follow Barakat and Bianchi [4 ] ; Ma et al. [36 ] ; Belotto da Silva and Gazeau [5 ] ; Wang et al. [48 ] ; Gronich and Vardi [19 ] in working with a continuous-time version of Adam, defined as follows. Consider an empirical loss L ( θ ) \mathcal{L}(\theta) L ( θ ) and a choice of subgradients g t ∈ ∂ L ( θ t ) g_t\in\partial\mathcal{L}(\theta_t) g t ∈ ∂ L ( θ t ) , then:
d θ t d t = − η ( t ) m ^ t v ^ t , (4) \frac{d\theta_t}{dt}=-\eta(t)\frac{\hat{m}_t}{\sqrt{\hat{v}_t}}, \tag*{(4)} d t d θ t = − η ( t ) v ^ t m ^ t , (4)
m t = EMA c 1 [ g t ] , m ^ t = EMA c 1 [ g t ] , v t = EMA c 2 [ g t 2 ] , v ^ t = EMA c 2 [ g t 2 ] . (5) m_t=\operatorname{EMA}_{c_1}[g_t],\quad\hat{m}_t=\operatorname{EMA}_{c_1}[g_t],\quad v_t=\operatorname{EMA}_{c_2}[g_t^2],\quad\hat{v}_t=\operatorname{EMA}_{c_2}[g_t^2]. \tag*{(5)} m t = EMA c 1 [ g t ] , m ^ t = EMA c 1 [ g t ] , v t = EMA c 2 [ g t 2 ] , v ^ t = EMA c 2 [ g t 2 ] . (5)
Here we denote by EMA c [ g ] \operatorname{EMA}_{c}[g] EMA c [ g ] , EMA ^ c [ g ] \widehat{\operatorname{EMA}}_{c}[g] EMA c [ g ] (respectively) the plain and bias-corrected exponential moving average (momentum) term for g g g with a continuous smoothing parameter c > 0 c>0 c > 0 analogous to the discrete parameter β ∈ [ 0 , 1 ) \beta\in[0,1) β ∈ [ 0 , 1 ) (or more precisely, analogous to − log β -\log\beta − log β ; see Appendix § 11 for definitions and references). Note that division, square root and the EMA operations are applied coordinate-wise. Throughout this work, we assume c 1 ≥ c 2 c_1 \ge c_2 c 1 ≥ c 2 , analogous to the common setting β 1 ≤ β 2 \beta_1 \le\beta_2 β 1 ≤ β 2 . For any j ∈ [ p ] j \in[p] j ∈ [ p ] with v ^ t [ j ] = 0 \hat{v}_t[j] = 0 v ^ t [ j ] = 0 we set d θ t d t [ j ] = 0 \frac{d\theta_t}{dt}[j] = 0 d t d θ t [ j ] = 0 ; this is consistent with the practical ε > 0 \varepsilon> 0 ε > 0 setting of Adam, since v ^ t [ j ] = 0 \hat{v}_t[j] = 0 v ^ t [ j ] = 0 implies m ^ t [ j ] = 0 \hat{m}_t[j] = 0 m ^ t [ j ] = 0 . We consider a polynomial learning rate schedule of the form
η ( t ) = η 0 ( 1 + t ) α , α > − 1 , η 0 > 0. (6) \eta(t) = \eta_0(1+t)^\alpha,\qquad\alpha> -1,\qquad\eta_0 > 0 . \tag*{(6)} η ( t ) = η 0 ( 1 + t ) α , α > − 1 , η 0 > 0. (6)
Such learning rate schedules are ubiquitous in practice as well as theoretical analyses.
The losses we consider, following Gronich and Vardi [19 ] , are log-concave, exponentially-tailed losses . Namely, we observe losses L ( θ ) : R p → R \mathcal{L}(\theta) : \mathbb{R}^p \to\mathbb{R} L ( θ ) : R p → R of the form
L ( θ ) = ∑ i = 1 m ℓ ( y i f ( x i ; θ ) ) , ℓ ∈ L e , (7) \mathcal{L}(\theta) = \sum_{i=1}^{m} \ell(y_i f(x_i;\theta)),\qquad\ell\in\mathcal{L}_{\mathrm{e}}, \tag*{(7)} L ( θ ) = i = 1 ∑ m ℓ ( y i f ( x i ; θ )) , ℓ ∈ L e , (7)
where L e \mathcal{L}_{\mathrm{e}} L e is the class of functions of the form ℓ ( u ) = e − φ ( u ) \ell(u) = e^{-\varphi(u)} ℓ ( u ) = e − φ ( u ) for φ ∈ C 2 ( R ) \varphi\in C^2(\mathbb{R}) φ ∈ C 2 ( R ) which is convex, strictly increasing and has bounded first and second derivatives (see Appendix § 10 ). Notably, the exponential and logistic losses (ℓ ( u ) = e − u \ell(u) = e^{-u} ℓ ( u ) = e − u and ℓ ( u ) = log ( 1 + e − u ) \ell(u) = \log(1+e^{-u}) ℓ ( u ) = log ( 1 + e − u ) , respectively) are in L e \mathcal{L}_{\mathrm{e}} L e .
We introduce two trajectory assumptions:
(T1) Positive-margin interpolation: ∃ t 0 ≥ 0 \exists t_0 \ge0 ∃ t 0 ≥ 0 , γ 0 > 0 \gamma_0 > 0 γ 0 > 0 , ∀ t ≥ t 0 : γ 2 ( θ t ) : = q min ( θ t ) ∥ θ t ∥ 2 ≥ γ 0 \forall t \ge t_0 : \gamma_2(\theta_t) := \frac{q_{\min}(\theta_t)}{\lVert\theta_t\rVert_2} \ge\gamma_0 ∀ t ≥ t 0 : γ 2 ( θ t ) := ∥ θ t ∥ 2 q m i n ( θ t ) ≥ γ 0 .
(T2) Loss decay: L ( θ t ) → t → ∞ 0 \mathcal{L}(\theta_t) \xrightarrow[t\to\infty]{} 0 L ( θ t ) t → ∞ 0 .
The lower bound γ 0 \gamma_0 γ 0 on the margin may be arbitrarily small, and the choice of norm here is inconsequential; we also note that (T2) may be weakened to require a sufficiently small loss on a sufficiently long window (see Lemma C.15 ).
Our main question of interest relates to implicit margin regularizers , which we define below:
Definition 3.1 (Implicit Margin Regularizer for Homogeneous Models in the KKT-Stationarity Sense ). Let p ≥ 1 p \ge1 p ≥ 1 , let f ( x ; θ ) : R d × R p → R f(x;\theta) : \mathbb{R}^{d} \times\mathbb{R}^{p} \to\mathbb{R} f ( x ; θ ) : R d × R p → R satisfy (M1), (M2) and let ℓ ( u ) : R → R \ell(u) : \mathbb{R} \to\mathbb{R} ℓ ( u ) : R → R be a loss function. We say that a locally Lipschitz R ( θ ) : R p → R R(\theta) : \mathbb{R}^{p} \to\mathbb{R} R ( θ ) : R p → R that admits a chain rule is an implicit margin regularizer of an optimizer O O O on f f f with the loss ℓ \ell ℓ , under a set of trajectory assumptions T \mathcal{T} T , if R R R satisfies the following:
For any binary classification dataset { ( x i , y i ) } i = 1 m ⊆ R d × { ± 1 } \{(x_i,y_i)\}_{i=1}^{m} \subseteq\mathbb{R}^{d}\times\{\pm1\} {( x i , y i ) } i = 1 m ⊆ R d × { ± 1 } , if θ t \theta_t θ t is optimized with O O O and the empirical loss L ( θ ) = ∑ i = 1 m ℓ ( y i f ( x i ; θ ) ) \mathcal{L}(\theta) = \sum_{i=1}^{m}\ell(y_i f(x_i;\theta)) L ( θ ) = ∑ i = 1 m ℓ ( y i f ( x i ; θ )) on f f f , satisfies the trajectory assumptions T \mathcal{T} T and (T1),(T2), and if θ t ∥ θ t ∥ 2 → t → ∞ θ ˉ \frac{\theta_t}{\lVert\theta_t\rVert_2} \xrightarrow[t\to\infty]{} \bar{\theta} ∥ θ t ∥ 2 θ t t → ∞ θ ˉ then θ ^ : = θ ˉ q min ( θ ˉ ) 1 / L \hat{\theta} := \frac{\bar{\theta}}{q_{\min}(\bar{\theta})^{1/L}} θ ^ := q m i n ( θ ˉ ) 1/ L θ ˉ is a KKT point of
min θ R ( θ ) s.t. ∀ i ∈ [ m ] : y i f ( x i ; θ ) ≥ 1. (8) \min_{\theta} R(\theta)\quad\text{s.t.}\quad\forall i \in[m] : y_i f(x_i;\theta) \ge1 . \tag*{(8)} θ min R ( θ ) s.t. ∀ i ∈ [ m ] : y i f ( x i ; θ ) ≥ 1. (8) Throughout this work we often write “implicit regularizer” or just “regularizer” for short; no other types of regularizers are considered.
Requiring Assumptions (T1) and (T2), together with directional convergence θ t ∥ θ t ∥ 2 → t → ∞ θ ˉ \frac{\theta_t}{\lVert\theta_t\rVert_2} \xrightarrow[t\to\infty]{} \bar{\theta} ∥ θ t ∥ 2 θ t t → ∞ θ ˉ , in Definition 3.1 only strengthens our nonexistence results: no implicit regularizer exists even when restricted to trajectories satisfying these conditions.
Nevertheless, all three conditions are natural in the study of the implicit bias of homogeneous models. Positive-margin interpolation, as required by Assumption (T1), is a form of the realizability premise underlying late-phase implicit-bias analysis. Some works assume momentary realizability (Lyu and Li, [35 ] ; Tsilivis et al., [45 ] ) while others impose a stronger requirement of directional convergence to a positive-margin limit (Gunasekar et al., [21 ] ; Nacson et al., [38 ] ; Gronich and Vardi, [19 ] ). Loss decay (Assumption (T2)) is likewise standard in max-margin analyses of separable classification, where convergence of the training loss to zero is either assumed (Gunasekar et al., [20 ] ; Nacson et al., [38 ] ) or proved (Soudry et al., [43 ] ; Lyu and Li, [35 ] ; Tsilivis et al., [45 ] ; Gronich and Vardi, [19 ] ). Directional convergence is also explicitly assumed in several works (Gunasekar et al., [21 ] ; Nacson et al., [38 ] ; Gronich and Vardi, [19 ] ) and proved in some works on gradient descent and adaptive methods (Ji and Telgarsky, [25 ] ; Wang et al., 2021), and provides a natural sufficient condition for convergence of the predictor up to scale, since the parameter direction determines a homogeneous predictor up to an overall scaling. Indeed, since the goal in studying the implicit bias is to understand which interpolators gradient methods prefer, it is natural to assume convergence to zero loss and to a specific predictor (up to scaling), and to analyze the properties of this interpolator.
Our result for the range 1 L − 1 ≤ α ≤ 1 L − 1 2 \frac{1}{L}-1 \le\alpha\le\frac{1}{L}-\frac{1}{2} L 1 − 1 ≤ α ≤ L 1 − 2 1 (Theorem 4.5 ) assumes directional convergence of the gradients to a vector with full support, which, in our setting, implies directional convergence of the parameters. Our α > 1 L − 1 2 \alpha> \frac{1}{L}-\frac{1}{2} α > L 1 − 2 1 positive result (Theorem 4.4 ) requires no directional convergence and instead applies to every directional limit point.
Using Definition 3.1 , we ask: is there an implicit margin regularizer of Adam on homogeneous networks in the learning rate regime α > 1 L − 1 \alpha> \frac{1}{L}-1 α > L 1 − 1 ?
Notation. We use standard asymptotic notations O O O , Ω \Omega Ω , Θ \Theta Θ as well as ≲ \lesssim ≲ , ≳ \gtrsim ≳ , ≍ \asymp ≍ (with the same respective meanings), and o o o . We focus on behavior as t → ∞ t \to\infty t → ∞ , and so asymptotic notations hide all multiplicative constants independent of t t t , unless stated otherwise. By log u \log u log u we refer to the natural logarithm unless stated otherwise. Vectors are bold-faced; time-dependent vectors are written as w t \mathbf{w}_t w t , and their coordinates w t [ j ] \mathbf{w}_t[j] w t [ j ] or w j ( t ) w_j(t) w j ( t ) interchangeably. ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p is the standard ℓ p \ell_p ℓ p norm for 1 ≤ p ≤ ∞ 1 \le p \le\infty 1 ≤ p ≤ ∞ . For n ∈ N n \in\mathbb{N} n ∈ N , we denote [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } . For a scalar z z z we use sign ( z ) = 1 \operatorname{sign}(z)=1 sign ( z ) = 1 if z > 0 z>0 z > 0 , and − 1 -1 − 1 if z < 0 z<0 z < 0 . For convenience we define sign ( 0 ) = 0 \operatorname{sign}(0)=0 sign ( 0 ) = 0 . For a vector g ∈ R p \mathbf{g}\in\mathbb{R}^p g ∈ R p we denote by sign ( g ) \operatorname{sign}(\mathbf{g}) sign ( g ) the vector sign ( g ) [ j ] : = sign ( g [ j ] ) \operatorname{sign}(\mathbf{g})[j]:=\operatorname{sign}(g[j]) sign ( g ) [ j ] := sign ( g [ j ]) for j ∈ [ p ] j\in[p] j ∈ [ p ] . We denote the p − 1 p-1 p − 1 dimensional ℓ 2 \ell_2 ℓ 2 unit-sphere by S p − 1 : = { u ∈ R p ∣ ∥ u ∥ 2 = 1 } S^{p-1}:=\{\mathbf{u}\in\mathbb{R}^p\mid\|\mathbf{u}\|_2=1\} S p − 1 := { u ∈ R p ∣ ∥ u ∥ 2 = 1 } . The notation C p C^p C p refers to the class of p p p -times continuously differentiable functions.
Main results
In this section we provide our main findings. We identify three regimes, which we name slow , intermediate and fast , referring to the potential for parameter movement as measured by ∫ 0 t η \int_0^t \eta ∫ 0 t η . Interestingly, while the slow and intermediate regimes share an ℓ ∞ \ell_\infty ℓ ∞ bias under certain conditions, the fast regime differs in its implicit bias but shares a loss decay rate of L ∼ exp ( − c 2 2 t ± o ( t ) ) \mathcal{L}\sim\exp\left(-\frac{c_2}{2}t\pm o(t)\right) L ∼ exp ( − 2 c 2 t ± o ( t ) ) with the intermediate regime (excluding its lower endpoint).
We denote for convenience the following quantity, noting its value ranges in the three regimes:
ζ = α − ( 1 L − 1 ) , Slow: ζ < 0 , Intermediate: 0 ≤ ζ ≤ 1 2 , Fast: ζ > 1 2 . (9) \zeta=\alpha-\left(\frac{1}{L}-1\right),\qquad\text{Slow: }\zeta<0,\qquad\text{Intermediate: }0\le\zeta\le\frac{1}{2},\qquad\text{Fast: }\zeta>\frac{1}{2}. \tag*{(9)} ζ = α − ( L 1 − 1 ) , Slow: ζ < 0 , Intermediate: 0 ≤ ζ ≤ 2 1 , Fast: ζ > 2 1 . (9)
Slow Regime. The first theorem is due with minor changes in assumptions to Gronich and Vardi [19 ] ; we state it here for completeness.
Theorem 4.1 (Adam in the slow regime admits an ℓ ∞ \ell_\infty ℓ ∞ -max-margin implicit regularizer; Paraphrased from Gronich and Vardi [19 ] ). For any α < 1 L − 1 \alpha< \frac{1}{L}-1 α < L 1 − 1 (i.e. ζ < 0 \zeta<0 ζ < 0 ), and p ≥ 1 p\ge1 p ≥ 1 , the function R ( θ ) = 1 2 ∥ θ ∥ 2 R(\theta)=\frac{1}{2}\|\theta\|^2 R ( θ ) = 2 1 ∥ θ ∥ 2 is an implicit margin regularizer for Adam on any C 1 C^1 C 1 model f f f satisfying (M2), and with any ℓ ∈ L e \ell\in L_e ℓ ∈ L e (Equation (7) ). Moreover, when θ t ∥ θ t ∥ \frac{\theta_t}{\|\theta_t\|} ∥ θ t ∥ θ t converges and under (T1), (T2), it holds for any j ∈ [ p ] j\in[p] j ∈ [ p ] with ∣ g t [ j ] ∣ ≥ Ω ( ∥ g t ∥ 1 ) |g_t[j]|\ge\Omega(\|g_t\|_1) ∣ g t [ j ] ∣ ≥ Ω ( ∥ g t ∥ 1 ) that
∣ m ^ t [ j ] ∣ v ^ t [ j ] → t → ∞ 1. (10) \frac{|\hat{m}_t[j]|}{\sqrt{\hat{v}_t[j]}} \xrightarrow[t\to\infty]{} 1. \tag*{(10)} v ^ t [ j ] ∣ m ^ t [ j ] ∣ t → ∞ 1. (10)
Fast Regime. In stark contrast, in the fast regime (which notably includes a constant learning rate for any L > 2 L>2 L > 2 ), we prove that late-phase behavior is history-dependent in a way that precludes a universal implicit regularizer. An explicit construction shows that there is no implicit regularizer even for the simple width-1 linear network with input dimension 2:
∇ θ = ( u , b ) = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 : f ( x ; θ ) : = P ( b ) ⟨ u , x ⟩ , P ( b ) : = ∏ j = 1 L − 1 b j . (11) \nabla\theta=(\mathbf{u},\mathbf{b})=(u_1,u_2,b_1,\ldots,b_{L-1})\in\mathbb{R}^{L+1}:\qquad f(\mathbf{x};\theta):=P(\mathbf{b})\langle\mathbf{u},\mathbf{x}\rangle,\qquad P(\mathbf{b}):=\prod_{j=1}^{L-1}b_j. \tag*{(11)} ∇ θ = ( u , b ) = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 : f ( x ; θ ) := P ( b ) ⟨ u , x ⟩ , P ( b ) := j = 1 ∏ L − 1 b j . (11)
Theorem 4.2 (Adam in the fast regime admits no regularizer for the width-1 model.). Let p ≥ 2 p \ge2 p ≥ 2 and let f p f_p f p be the width-1 model of depth L L L with input dimension d = 2 d=2 d = 2 , as in (11) (here L = p − 1 L=p-1 L = p − 1 ). When α > 1 L − 1 2 \alpha> \frac{1}{L}-\frac{1}{2} α > L 1 − 2 1 (i.e. ζ > 1 2 \zeta> \frac{1}{2} ζ > 2 1 ), any implicit margin regularizer (Theorem 3.1) R R R of Adam on f p f_p f p with the exponential loss is constant.
The proof of Theorem 4.2 shows that θ ˉ = lim t → ∞ θ t ∥ θ t ∥ 2 \bar{\theta}=\lim_{t\to\infty}\frac{\theta_t}{\|\theta_t\|_2} θ ˉ = lim t → ∞ ∥ θ t ∥ 2 θ t varies continuously on fixed datasets by varying initialization only. Indeed, there exist open domains U 1 , … , U 2 L + 2 ⊆ S p − 1 U_1,\ldots,U_{2L+2}\subseteq S^{p-1} U 1 , … , U 2 L + 2 ⊆ S p − 1 , whose union covers S p − 1 S^{p-1} S p − 1 up to a null set, so that for each U i U_i U i and any compact subset A ⊆ U i A\subseteq U_i A ⊆ U i , there exists a two-point dataset S A S_A S A with θ t ∥ θ t ∥ 2 \frac{\theta_t}{\|\theta_t\|_2} ∥ θ t ∥ 2 θ t converging to any point in A A A with choice of initialization (see Corollary D.13 for details and Appendix § 15 for empirical validation).
While already a surprising result, Theorem 4.2 does not rule out an implicit regularizer for wide linear networks, if it is trivial when restricted to the width-1 model. Hence we extend the result of Theorem 4.2 to linear networks of arbitrary widths and input dimensions, showing that no convex regularizer exists.
Theorem 4.3 (Adam in the fast regime admits no convex regularizer for arbitrary-width deep linear networks). Let p ≥ 2 p \ge2 p ≥ 2 and let f p f_p f p be a scalar-output deep linear network with input dimension d ≥ 2 d \ge2 d ≥ 2 , p p p parameters, arbitrary depth L L L and arbitrary hidden widths. When α > 1 L − 1 2 \alpha> \frac{1}{L}-\frac{1}{2} α > L 1 − 2 1 (i.e. ζ > 1 2 \zeta> \frac{1}{2} ζ > 2 1 ), any convex implicit margin regularizer (Theorem 3.1) R R R of Adam on f p f_p f p with the exponential loss is constant.
In particular, Theorems 4.2 , 4.3 show that no squared norm is an implicit regularizer in the fast regime, hence there is no max-margin bias in the usual sense.
Theorems 4.2 , 4.3 follow from a mechanism of convergence of the normalized second moments e c 2 t v t [ j ] e^{c_2t}v_t[j] e c 2 t v t [ j ] to trajectory-dependent limits ω j \omega_j ω j in the fast regime. Using this same mechanism, one can positively characterize the implicit bias of Adam in the fast regime as maximizing a margin whose geometry is controlled by the limits ω j \omega_j ω j . The following theorem holds for a general homogeneous model, not necessarily a linear network.
Theorem 4.4 (Adam in the fast regime maximizes a trajectory-dependent margin.). Let f f f satisfy (M1), (M2), and let ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e (Equation (7) ). When α > 1 L − 1 2 \alpha> \frac{1}{L}-\frac{1}{2} α > L 1 − 2 1 (i.e. ζ > 1 2 \zeta> \frac{1}{2} ζ > 2 1 ), it holds under (T1), (T2) for any j ∈ [ p ] j \in[p] j ∈ [ p ] that
∣ m ^ t [ j ] ∣ v ^ t [ j ] ≤ O ( t − ζ ) , e c 2 t v t [ j ] → t → ∞ ω j < ∞ , ω j − e c 2 t v t [ j ] ≤ O ( t − 2 ( ζ − 1 2 ) ) . (12) \frac{|\hat{m}_t[j]|}{\sqrt{\hat{v}_t[j]}} \le O\left(t^{-\zeta}\right), \qquad e^{c_2t}v_t[j]\xrightarrow[t\to\infty]{}\omega_j<\infty,\qquad\omega_j-e^{c_2t}v_t[j]\le O\left(t^{-2(\zeta-\frac{1}{2})}\right). \tag*{(12)} v ^ t [ j ] ∣ m ^ t [ j ] ∣ ≤ O ( t − ζ ) , e c 2 t v t [ j ] t → ∞ ω j < ∞ , ω j − e c 2 t v t [ j ] ≤ O ( t − 2 ( ζ − 2 1 ) ) . (12)
Consequently, Adam asymptotically follows a preconditioned momentum gradient descent trajectory. Namely, it holds for a diagonal matrix D ω D_\omega D ω that
d θ t d t = − D ω η ( t ) e c 2 t m t ( 1 + o ( 1 ) ) , ( D ω ) j , j = { ω j − 1 / 2 : ω j > 0 , 1 : ω j = 0. (13) \frac{d\theta_t}{dt}=-D_\omega\eta(t)e^{c_2t}\mathbf{m}_t(1+o(1)),\qquad(D_\omega)_{j,j}=
\begin{cases}
\omega_j^{-1/2} & :\omega_j>0,\\
1 & :\omega_j=0.
\end{cases} \tag*{(13)} d t d θ t = − D ω η ( t ) e c 2 t m t ( 1 + o ( 1 )) , ( D ω ) j , j = { ω j − 1/2 1 : ω j > 0 , : ω j = 0. (13)
Additionally, for any limit point θ ˉ \bar{\theta} θ ˉ of θ t ∥ θ t ∥ 1 \frac{\theta_t}{\|\theta_t\|_1} ∥ θ t ∥ 1 θ t , we have that θ ^ : = θ ˉ q min ( θ ˉ ) 1 / L \hat{\theta}:=\frac{\bar{\theta}}{q_{\min}(\bar{\theta})^{1/L}} θ ^ := q m i n ( θ ˉ ) 1/ L θ ˉ is a KKT point of
min θ 1 2 θ T D ω − 1 θ s.t. ∀ i ∈ [ m ] : y i f ( x i ; θ ) ≥ 1. (14) \min_{\theta}\ \frac{1}{2}\theta^\mathsf{T}D_\omega^{-1}\theta\qquad\text{s.t.}\qquad\forall i\in[m]:\ y_i f(x_i;\theta)\ge1. \tag*{(14)} θ min 2 1 θ T D ω − 1 θ s.t. ∀ i ∈ [ m ] : y i f ( x i ; θ ) ≥ 1. (14)
Here the choice of ( D ω ) j , j = 1 (D_\omega)_{j,j}=1 ( D ω ) j , j = 1 for inactive coordinates is arbitrary.
Intermediate Regime. In the intermediate regime, our result extends the ℓ ∞ \ell_\infty ℓ ∞ bias of the slow regime, conditional on gradient directional convergence to a full-support direction; proving or refuting an ℓ ∞ \ell_\infty ℓ ∞ bias with no such assumption remains open. We separate this regime from the slow regime due to the qualitative difference in the Adam ratio ∣ m ^ t [ j ] ∣ v ^ t [ j ] \frac{|\hat{m}_t[j]|}{\sqrt{\hat{v}_t[j]}} v ^ t [ j ] ∣ m ^ t [ j ] ∣ , which is a decaying polynomial instead of asymptotically tending to 1 1 1 , and due to the difference in the loss decay rate which we discuss later. It is also distinct from the fast regime since the assumption of directional convergence of the gradient to a full-support vector holds in the construction of Theorems 4.2 , 4.3 ; thus, the fast regime does not have an ℓ ∞ \ell_\infty ℓ ∞ bias even under this assumption.
Theorem 4.5 (Adam in the intermediate regime admits an ℓ ∞ \ell_\infty ℓ ∞ -max-margin implicit regularizer under gradient convergence to a full-support vector). For any 1 L − 1 ≤ α ≤ 1 L − 1 2 \frac{1}{L}-1\le\alpha\le\frac{1}{L}-\frac{1}{2} L 1 − 1 ≤ α ≤ L 1 − 2 1 (i.e. 0 ≤ ζ ≤ 1 2 0\le\zeta\le\frac{1}{2} 0 ≤ ζ ≤ 2 1 ) and p ≥ 1 p\ge1 p ≥ 1 , the function
R ( θ ) = 1 2 ∥ θ ∥ ∞ 2 R(\theta)=\frac{1}{2}\lVert\theta\rVert_{\infty}^{2} R ( θ ) = 2 1 ∥ θ ∥ ∞ 2 is an implicit margin regularizer (Theorem 3.1) for Adam on any f f f satisfying (M1), (M2) and with any ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e (Equation (7) ), under the trajectory assumption ∀ j ∈ [ p ] : g t [ j ] ∥ g t ∥ 1 → t → ∞ g ˉ ∞ [ j ] ≠ 0 \forall j\in[p]:\frac{g_t[j]}{\lVert g_t\rVert_1}\xrightarrow[t\to\infty]{}\bar{g}_{\infty}[j]\ne0 ∀ j ∈ [ p ] : ∥ g t ∥ 1 g t [ j ] t → ∞ g ˉ ∞ [ j ] = 0 . Moreover, under this assumption and (T1), (T2) it holds for any j ∈ [ p ] j\in[p] j ∈ [ p ] that
∣ m ^ t [ j ] ∣ v ^ t [ j ] ≤ O ( t − ζ ) . (15) \frac{|\hat{m}_t[j]|}{\sqrt{\hat{v}_t[j]}}\le O\left(t^{-\zeta}\right). \tag*{(15)} v ^ t [ j ] ∣ m ^ t [ j ] ∣ ≤ O ( t − ζ ) . (15)
Loss Decay Rates. We provide loss decay rates, showing that − log L ( θ t ) = c 2 2 t ± o ( t ) -\log\mathcal{L}(\theta_t)=\frac{c_2}{2}t\pm o(t) − log L ( θ t ) = 2 c 2 t ± o ( t ) in the intermediate and fast regimes (whenever ζ > 0 \zeta>0 ζ > 0 ), thus uncovering a direct connection between the rate and the value of c 2 c_2 c 2 (analogous to − log β 2 -\log\beta_2 − log β 2 in discrete Adam). In particular, this demonstrates that Adam adapts to its learning rate schedule: the leading order c 2 2 t \frac{c_2}{2}t 2 c 2 t is independent of the schedule within the intermediate and fast regimes. This result contrasts with the slow regime, in which we show that − log L ( θ t ) ≍ t L ( α + 1 ) = o ( t ) -\log\mathcal{L}(\theta_t)\asymp t^{L(\alpha+1)}=o(t) − log L ( θ t ) ≍ t L ( α + 1 ) = o ( t ) – a rate which is both sublinear and schedule-dependent. Similar slow-regime results were previously shown in the proof of Lemma C.19 in Gronich and Vardi [19 ] , assuming f ∈ C 1 f\in C^1 f ∈ C 1 and directional convergence of the parameters (assumptions which we omit), and in Theorem 4.5 in Zhang et al. [53 ] for discrete Adam in the special case of linear classifiers. For more precise error terms, see Theorem C.13 .
Theorem 4.6. Let f f f satisfy (M1) (M2), let ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e (Equation (7) ), and let θ t \theta_t θ t be an Adam trajectory of f f f assuming (T1), (T2). Writing U ( t ) : = − log L ( θ t ) U(t):=-\log\mathcal{L}(\theta_t) U ( t ) := − log L ( θ t ) , it holds that
U ( t ) = c 2 2 t ± O ( log t ) , ζ > 1 2 , U ( t ) = c 2 2 t ± o ( t ) , 0 < ζ ≤ 1 2 , U ( t ) = Θ ( t L ( α + 1 ) ) , ζ ≤ 0. (16) \begin{aligned}
U(t)&=\frac{c_2}{2}t\pm O(\log t), && \zeta>\frac{1}{2},\\
U(t)&=\frac{c_2}{2}t\pm o(t), && 0<\zeta\le\frac{1}{2},\\
U(t)&=\Theta\left(t^{L(\alpha+1)}\right), && \zeta\le0. \tag*{(16)}
\end{aligned} U ( t ) U ( t ) U ( t ) = 2 c 2 t ± O ( log t ) , = 2 c 2 t ± o ( t ) , = Θ ( t L ( α + 1 ) ) , ζ > 2 1 , 0 < ζ ≤ 2 1 , ζ ≤ 0. (16)
Proof ideas
In this section, we discuss some main proof ideas. All formal proofs are deferred to the appendix.
Second-moment convergence
Recalling the Adam update d θ t d t = − η ( t ) m ^ t v ^ t \frac{d\theta_t}{dt}=-\eta(t)\frac{\hat{m}_t}{\sqrt{\hat{v}_t}} d t d θ t = − η ( t ) v ^ t m ^ t , studying the asymptotic properties of Adam boils down to the behavior of m t , v t m_t,v_t m t , v t , which can be written explicitly as:
m t = c 1 e − c 1 t ∫ 0 t e c 1 s g s d s , v t = c 2 e − c 2 t ∫ 0 t e c 2 s g s 2 d s . m_t=c_1e^{-c_1t}\int_0^t e^{c_1s}g_s\,\mathrm{d}s,\qquad v_t=c_2e^{-c_2t}\int_0^t e^{c_2s}g_s^2\,\mathrm{d}s. m t = c 1 e − c 1 t ∫ 0 t e c 1 s g s d s , v t = c 2 e − c 2 t ∫ 0 t e c 2 s g s 2 d s .
We discover a critical transition at ζ = 1 2 \zeta=\frac{1}{2} ζ = 2 1 , separating the fast regime from the intermediate and slow regimes:
Proposition 5.1. Under assumptions (T1), (T2) for Adam, when ζ > 1 2 \zeta>\frac{1}{2} ζ > 2 1 , e c 2 t v t [ j ] → t → ∞ ω j < ∞ e^{c_2t}v_t[j]\xrightarrow[t\to\infty]{}\omega_j<\infty e c 2 t v t [ j ] t → ∞ ω j < ∞ for all j ∈ [ p ] j\in[p] j ∈ [ p ] , and when ζ ≤ 1 2 \zeta\le\frac{1}{2} ζ ≤ 2 1 , e c 2 t v t [ j ] → t → ∞ ∞ e^{c_2t}v_t[j]\xrightarrow[t\to\infty]{}\infty e c 2 t v t [ j ] t → ∞ ∞ for all j ∈ [ p ] j\in[p] j ∈ [ p ] with ∣ g t [ j ] ∣ ≥ Ω ( ∥ g t ∥ 1 ) |g_t[j]|\ge\Omega(\lVert g_t\rVert_1) ∣ g t [ j ] ∣ ≥ Ω (∥ g t ∥ 1 ) .
Proposition 5.1 is proved in Appendix C in Corollary C.10 and Proposition C.11. We now use Proposition 5.1 to analyze the three regimes.
Three regimes
Slow regime: Adam ratio approximates sign-GD. Suppose − 1 < α < 1 L − 1 -1<\alpha<\frac{1}{L}-1 − 1 < α < L 1 − 1 (i.e. ζ < 0 \zeta<0 ζ < 0 ). Under the assumption that f ∈ C 1 f\in C^1 f ∈ C 1 and parameter direction converges, Lemma C.19 from Gronich and Vardi [19 ] gives, through slow variation of the model outputs ∣ d g j f ( x i ; θ t ) d t ∣ = o ( 1 ) \left|\frac{d g_j f(x_i;\theta_t)}{dt}\right|=o(1) d t d g j f ( x i ; θ t ) = o ( 1 ) , that
m t = g t + o ( ∥ g t ∥ 2 ) , v t = g t 2 + o ( ∥ g t ∥ 2 2 ) . (17) m_t=g_t+o(\lVert g_t\rVert_2),\qquad v_t=g_t^2+o\left(\lVert g_t\rVert_2^2\right). \tag*{(17)} m t = g t + o (∥ g t ∥ 2 ) , v t = g t 2 + o ( ∥ g t ∥ 2 2 ) . (17)
Now fix a coordinate j j j satisfying the eventual lower bound ∣ g t [ j ] ∣ ≥ a j ∥ g t ∥ 2 |g_t[j]| \ge a_j \|g_t\|_2 ∣ g t [ j ] ∣ ≥ a j ∥ g t ∥ 2 , a j > 0 a_j > 0 a j > 0 . Continuity of g t g_t g t and (17) then imply
m ^ t [ j ] g t [ j ] → t → ∞ 1 , v ^ t [ j ] g t 2 [ j ] → t → ∞ 1 , m ^ t [ j ] v ^ t [ j ] = sign ( g t [ j ] ) ( 1 + o ( 1 ) ) , (18) \frac{\hat{m}_t[j]}{g_t[j]} \xrightarrow{t\to\infty} 1,\qquad\frac{\hat{v}_t[j]}{g_t^2[j]} \xrightarrow{t\to\infty} 1,\qquad\frac{\hat{m}_t[j]}{\sqrt{\hat{v}_t[j]}} = \operatorname{sign}(g_t[j])\left(1+o(1)\right), \tag*{(18)} g t [ j ] m ^ t [ j ] t → ∞ 1 , g t 2 [ j ] v ^ t [ j ] t → ∞ 1 , v ^ t [ j ] m ^ t [ j ] = sign ( g t [ j ]) ( 1 + o ( 1 ) ) , (18)
inducing asymptotic sign-GD behavior, hence an ℓ ∞ \ell_\infty ℓ ∞ bias.
Intermediate regime: sign-GD with a scalar envelope. Now suppose 1 L − 1 ≤ α ≤ 1 L − 1 2 \frac{1}{L}-1 \le\alpha\le\frac{1}{L}-\frac{1}{2} L 1 − 1 ≤ α ≤ L 1 − 2 1 , (i.e. 0 ≤ ζ ≤ 1 2 0 \le\zeta\le\frac{1}{2} 0 ≤ ζ ≤ 2 1 ), and assume
∀ j ∈ [ p ] : g t [ j ] ∥ g t ∥ 1 → t → ∞ g ˉ ∞ [ j ] ≠ 0. \forall j \in[p]:\qquad\frac{g_t[j]}{\|g_t\|_1} \xrightarrow{t\to\infty} \bar{g}_\infty[j] \ne0. ∀ j ∈ [ p ] : ∥ g t ∥ 1 g t [ j ] t → ∞ g ˉ ∞ [ j ] = 0.
Since c 1 ≥ c 2 c_1 \ge c_2 c 1 ≥ c 2 and ∥ g t ∥ 1 → 0 \|g_t\|_1 \to0 ∥ g t ∥ 1 → 0 , Proposition 5.1 gives ∫ 0 t e c 1 s ∥ g s ∥ 1 d s ≳ ∫ 0 t e c 2 s ∥ g s ∥ 1 2 d s → ∞ \int_0^t e^{c_1s}\|g_s\|_1\,ds \gtrsim\int_0^t e^{c_2s}\|g_s\|_1^2\,ds \to\infty ∫ 0 t e c 1 s ∥ g s ∥ 1 d s ≳ ∫ 0 t e c 2 s ∥ g s ∥ 1 2 d s → ∞ , implying that the directional limit transfers to the exponential moving average (Lemma B.4):
m ^ t [ j ] EMA c 1 [ ∥ g t ∥ 1 ] → t → ∞ g ˉ ∞ [ j ] , v ^ t [ j ] EMA c 2 [ ∥ g t ∥ 1 2 ] → t → ∞ g ˉ ∞ 2 [ j ] . \frac{\hat{m}_t[j]}{\operatorname{EMA}_{c_1}[\|g_t\|_1]} \xrightarrow{t\to\infty} \bar{g}_\infty[j],\qquad\frac{\hat{v}_t[j]}{\operatorname{EMA}_{c_2}[\|g_t\|_1^2]} \xrightarrow{t\to\infty} \bar{g}_\infty^2[j]. EMA c 1 [ ∥ g t ∥ 1 ] m ^ t [ j ] t → ∞ g ˉ ∞ [ j ] , EMA c 2 [ ∥ g t ∥ 1 2 ] v ^ t [ j ] t → ∞ g ˉ ∞ 2 [ j ] .
m ^ t v ^ t = ( EMA c 1 [ ∥ g t ∥ 1 ] EMA c 2 [ ∥ g t ∥ 1 2 ] ) sign ( g ˉ ∞ ) ( 1 + o ( 1 ) ) . \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}} = \left(\frac{\operatorname{EMA}_{c_1}[\|g_t\|_1]}{\sqrt{\operatorname{EMA}_{c_2}[\|g_t\|_1^2]}}\right)\operatorname{sign}(\bar{\mathbf{g}}_\infty)(1+o(1)). v ^ t m ^ t = ( EMA c 2 [ ∥ g t ∥ 1 2 ] EMA c 1 [ ∥ g t ∥ 1 ] ) sign ( g ˉ ∞ ) ( 1 + o ( 1 )) .
In other words, θ t \theta_t θ t asymptotically follows a sign-GD trajectory with a scalar envelope.
Fast regime: weighted momentum GD. Proposition 5.1 gives v t [ j ] = e − c 2 t ( ω j + o ( 1 ) ) v_t[j] = e^{-c_2t}(\omega_j + o(1)) v t [ j ] = e − c 2 t ( ω j + o ( 1 )) , ω j < ∞ \omega_j < \infty ω j < ∞ , so Adam sees an asymptotically frozen diagonal preconditioner for momentum gradient descent:
d θ t d t [ j ] = − η ( t ) m ^ t [ j ] v ^ t [ j ] = − η ( t ) e c 2 2 t ω j − 1 / 2 m t [ j ] ( 1 + o ( 1 ) ) . \frac{d\theta_t}{dt}[j] = -\eta(t)\frac{\hat{m}_t[j]}{\sqrt{\hat{v}_t[j]}} = -\eta(t)e^{\frac{c_2}{2}t}\omega_j^{-1/2}\mathbf{m}_t[j](1+o(1)). d t d θ t [ j ] = − η ( t ) v ^ t [ j ] m ^ t [ j ] = − η ( t ) e 2 c 2 t ω j − 1/2 m t [ j ] ( 1 + o ( 1 )) .
In the following subsection we show why this rules out a universal implicit bias.
No universal implicit regularizer in the fast regime
We introduce the explicit width-1 construction for the fast regime, which serves to prove Theorem 4.2 . For a full proof see Appendix § 13 .§ 1 . The core idea is as follows: large initialization-dependent gradients early in the trajectory set the frozen direction of the second moment v t v_t v t , while the first moment m t m_t m t asymptotically tracks the late-phase gradients regardless of initialization. Thus, the late-phase update direction, and hence the limiting parameter direction, is made arbitrary. This implies the triviality of the regularizer R R R . Theorem 4.3 then follows from a lifting argument to an arbitrarily wide network.
Concretely, observe the model (11) and the following dataset parameterized by M > B > 1 , Λ > 0 M > B > 1,\Lambda> 0 M > B > 1 , Λ > 0 ,
x 1 = 1 Λ ( B , 1 ) , x 2 = 1 Λ ( M , 0 ) , y 1 = y 2 = 1. (19) x_1 = \frac{1}{\Lambda}(B,1), \qquad x_2 = \frac{1}{\Lambda}(M,0), \qquad y_1 = y_2 = 1 . \tag*{(19)} x 1 = Λ 1 ( B , 1 ) , x 2 = Λ 1 ( M , 0 ) , y 1 = y 2 = 1. (19)
The main argument shows that, allowing a continuously parameterized initialization θ 0 = θ 0 ( ρ , z , π 1 , … , π L − 1 ) \boldsymbol{\theta}_0 = \boldsymbol{\theta}_0(\rho,z,\pi_1,\ldots,\pi_{L-1}) θ 0 = θ 0 ( ρ , z , π 1 , … , π L − 1 ) :
Proposition 5.2 (Arbitrary margin-normalized convergence to a hypersurface). For any B > 1 , Λ > 0 B > 1,\Lambda> 0 B > 1 , Λ > 0 , denote the hypersurface
Σ B , Λ = { θ ∈ R L + 1 ∣ 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] b j > 0 , f ( x 1 ; θ ) = 1 } . (20) \Sigma_{B,\Lambda} = \left\{\boldsymbol{\theta} \in\mathbb{R}^{L+1} \mid0 < u_1 < u_2,\ \forall j \in[L-1]\ b_j > 0,\ f(x_1;\boldsymbol{\theta}) = 1\right\}. \tag*{(20)} Σ B , Λ = { θ ∈ R L + 1 ∣ 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] b j > 0 , f ( x 1 ; θ ) = 1 } . (20) Then for any θ ^ ∈ Σ B , Λ \hat{\boldsymbol{\theta}} \in\Sigma_{B,\Lambda} θ ^ ∈ Σ B , Λ , and for any large enough M > 0 M > 0 M > 0 there exists an initialization θ 0 \boldsymbol{\theta}_0 θ 0 for which
θ t q min 1 / L ( θ t ) ⟶ t → ∞ θ ^ . \frac{\boldsymbol{\theta}_t}{q_{\min}^{1/L}(\boldsymbol{\theta}_t)} \mathop{\longrightarrow}_{t\to\infty} \hat{\boldsymbol{\theta}} . q m i n 1/ L ( θ t ) θ t ⟶ t → ∞ θ ^ . Moreover, x 1 x_1 x 1 is the unique minimum-margin example for θ ^ \hat{\boldsymbol{\theta}} θ ^ , that is, f ( x 2 ; θ ^ ) > 1 f(x_2;\hat{\boldsymbol{\theta}}) > 1 f ( x 2 ; θ ^ ) > 1 .
Corollary 5.3. Any implicit margin regularizer R R R on f p f_p f p (Equation (11) ) with the exponential loss is constant on Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ .
Proof. Take any C 1 C^1 C 1 path ψ ( s ) ⊆ Σ B , Λ \psi(s) \subseteq\Sigma_{B,\Lambda} ψ ( s ) ⊆ Σ B , Λ . Fixing s s s and invoking Proposition 5.2 and the definition of R R R implies the existence of k s ∈ ∂ R ( ψ ( s ) ) , h s ∈ ∂ f ( x 1 ; ψ ( s ) ) k_s \in\partial R(\psi(s)),h_s \in\partial f(x_1;\psi(s)) k s ∈ ∂ R ( ψ ( s )) , h s ∈ ∂ f ( x 1 ; ψ ( s )) , for which
k s = λ s h s . k_s = \lambda_s h_s . k s = λ s h s .
But ⟨ h s , ψ ˙ ( s ) ⟩ = 0 \langle h_s,\dot{\psi}(s)\rangle= 0 ⟨ h s , ψ ˙ ( s )⟩ = 0 since Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ is a level set of f ( x 1 ; ⋅ ) f(x_1;\cdot) f ( x 1 ; ⋅ ) , so by the chain rule
( R ∘ ψ ) ′ ( s ) = ⟨ k s , ψ ˙ ( s ) ⟩ = 0 for almost every s . (R\circ\psi)'(s) = \langle k_s,\dot{\psi}(s)\rangle= 0 \quad\text{for almost every }s . ( R ∘ ψ ) ′ ( s ) = ⟨ k s , ψ ˙ ( s )⟩ = 0 for almost every s .
Figure 1 describes the main triviality argument and demonstrates arbitrary initialization-dependent directional convergence in experiments (see Appendix § 15 for full experimental details).
Figure 1. Initialization-dependent limiting direction forces constancy of R R R in the fast regime. (a) Varying the initialization produces distinct margin-normalized limits on the hypersurface Σ = { θ ∣ f ( x 1 ; θ ) = 1 } \Sigma= \{\theta\mid f(x_1;\theta) = 1\} Σ = { θ ∣ f ( x 1 ; θ ) = 1 } , due to large early-phase gradients. Since only x 1 x_1 x 1 has an active margin there, KKT forces k ⊥ Σ \mathbf{k} \perp\Sigma k ⊥ Σ for the subgradient k ∈ ∂ R ( θ ^ ) \mathbf{k} \in\partial R(\hat{\theta}) k ∈ ∂ R ( θ ^ ) , implying R R R is constant on Σ \Sigma Σ . (b) Initialization-dependent parameter directions on the unit sphere, reached in experiments running Adam on the width-1 model (11) (with L = 2 L = 2 L = 2 ), each marker representing a different initialization. Blue and orange markers denote the u 1 < u 2 u_1 < u_2 u 1 < u 2 and u 2 < u 1 u_2 < u_1 u 2 < u 1 sectors, respectively, obtained from two distinct yet symmetric datasets.
Continuously varying B , Λ B,\Lambda B , Λ , and utilizing common boundary points of the hypersurfaces Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ , gives constancy of R R R on the whole open sector where 0 < u 1 < u 2 0 < u_1 < u_2 0 < u 1 < u 2 and b j > 0 b_j > 0 b j > 0 for all j j j , and a symmetry argument proves constancy on the whole of R L + 1 \mathbb{R}^{L+1} R L + 1 , proving Theorem 4.2 . Then, a lifting argument is used to generalize the result to networks of arbitrary width and input dimension (see Appendix § 13 .§ 2 ); since the lifted hypersurfaces do not cover the whole space, but do linearly span it, convexity of R R R is required to imply constancy (Theorem 4.3 ).
Experiments
We empirically validate our results by training 3-layer fully-connected ReLU networks to classify m = 1024 m = 1024 m = 1024 MNIST (LeCun et al. [ [32 ] ) images (even vs. odd digits) with the logistic loss. Here the critical schedule rates are 1 L − 1 ≃ − 0.67 \frac{1}{L}-1 \simeq-0.67 L 1 − 1 ≃ − 0.67 , 1 L − 1 2 ≃ − 0.17 \frac{1}{L}-\frac{1}{2} \simeq-0.17 L 1 − 2 1 ≃ − 0.17 . We compare margin values for three regimes of Adam: slow (t − 0.7 t^{-0.7} t − 0.7 ), intermediate (t − 0.5 , t − 0.3 t^{-0.5},t^{-0.3} t − 0.5 , t − 0.3 ) and fast (t − 0.1 , t 0 t^{-0.1},t^0 t − 0.1 , t 0 ), with a common base learning rate 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 . For experimental details see Appendix § 15 .1
For our first experiment we train the networks with ε = 0 \varepsilon= 0 ε = 0 until training loss reaches a small value of 10 − 20 10^{-20} 1 0 − 20 (as explained in § 3 , we set the update to 0 for coordinates with v ^ t = 0 \hat{v}_t = 0 v ^ t = 0 ). Initial parameters are multiplied by a factor of 10 − 4 10^{-4} 1 0 − 4 , since a small initialization is known to accelerate arrival to the asymptotic regime (Moroshko et al., 2020; Woodworth et al., [50 ] ). Results are shown in Figure 2 . Late-phase slopes of − ln L -\ln\mathcal{L} − ln L closely match the predicted value c 2 2 \frac{c_2}{2} 2 c 2 (i.e. − ln ( β 2 ) 2 -\frac{\ln(\beta_2)}{2} − 2 l n ( β 2 ) ) for α ≥ − 0.5 \alpha\ge-0.5 α ≥ − 0.5 , while the slow schedule α = − 0.7 \alpha= -0.7 α = − 0.7 displays a considerably slower rate. The ℓ ∞ \ell_\infty ℓ ∞ margin, denoted γ ∞ \gamma_\infty γ ∞ , separates the three regimes according to our implicit bias characterization: the slow schedule achieves the largest margin, with fast schedules displaying small, decreasing margins. The margins of the intermediate schedules lie between them. We also plot the trajectory-dependent weighted-ℓ 2 \ell_2 ℓ 2 margin, denoted here γ ω \gamma_\omega γ ω , approximating the asymptotic weights ω \omega ω by calculating them from the trajectory endpoint t e n d t_{\mathrm{end}} t end . To make margin scales comparable, we normalize ω 1 / 2 \omega^{1/2} ω 1/2 to unit ℓ 2 \ell_2 ℓ 2 norm; for each fixed ω \omega ω , this only rescales margin values by a uniform constant. Explicitly,2
Figure 2. Adam loss decay rate and margins on a 3-layer network with ε = 0 \varepsilon= 0 ε = 0 and initialization scale 10 − 4 10^{-4} 1 0 − 4 . (a) Late-phase slope of − ln L -\ln\mathcal{L} − ln L on the 10,000 last steps of each trajectory, and a reference at − ln ( β 2 ) / 2 -\ln(\beta_2)/2 − ln ( β 2 ) /2 . (b) The ℓ ∞ \ell_\infty ℓ ∞ margin. (c)–(d) For α = 0 \alpha= 0 α = 0 and α = − 0.7 \alpha= -0.7 α = − 0.7 , respectively, the weighted margin γ ω \gamma_\omega γ ω using weights from one reference seed’s endpoint. In (a)–(b), lines are means across 5 seeds and shading denotes one standard error.
γ ω ( θ ) = q min ( θ ) ∥ θ ∥ ω , ∥ θ ∥ ω = ( ∑ j ω j 1 / 2 θ j 2 ) 1 / 2 , ω = β 2 − t e n d v t e n d , ω 1 / 2 = ω 1 / 2 ∥ ω 1 / 2 ∥ 2 . (21) \gamma_\omega(\theta) = \frac{q_{\min}(\theta)}{\lVert\theta\rVert_\omega}, \qquad\lVert\theta\rVert_\omega= \left(\sum_j \omega_j^{1/2}\theta_j^2\right)^{1/2}, \qquad\omega= \beta_2^{-t_{\mathrm{end}}}v_{t_{\mathrm{end}}}, \qquad\omega^{1/2} = \frac{\omega^{1/2}}{\lVert\omega^{1/2}\rVert_2}. \tag*{(21)} γ ω ( θ ) = ∥ θ ∥ ω q m i n ( θ ) , ∥ θ ∥ ω = ( j ∑ ω j 1/2 θ j 2 ) 1/2 , ω = β 2 − t end v t end , ω 1/2 = ∥ ω 1/2 ∥ 2 ω 1/2 . (21)
For each of the schedules α = 0 \alpha= 0 α = 0 and α = − 0.7 \alpha= -0.7 α = − 0.7 , we select a reference run, fix ω \omega ω using its endpoint weights, and evaluate the resulting margin along its trajectory and those of four additional seeds. In the fast regime, the reference trajectory attains the largest margin under its own endpoint metric among the trajectories compared, consistent with our characterization of the trajectory-dependent implicit bias. In the slow regime, the weighted margin is not maximized by its own trajectory relative to others, consistent with the different implicit bias predicted for this regime. See Appendix § 15 for corresponding plots with the remaining schedules and reference seeds.
Figure 3. Adam margins for ε = 0 \varepsilon= 0 ε = 0 , 10 − 8 10^{-8} 1 0 − 8 on a 3-layer network with initialization scale 1. (a)–(b) ℓ 2 \ell_2 ℓ 2 and ℓ ∞ \ell_\infty ℓ ∞ margins, respectively. Solid curves and plain bands correspond to ε = 0 \varepsilon= 0 ε = 0 ; dashed curves and hatched bands correspond to ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 . Curves show means across five seeds; shading denotes one standard error, with boundaries outlined. (c)–(d) For α = 0 \alpha= 0 α = 0 and α = − 0.7 \alpha= -0.7 α = − 0.7 , respectively, the weighted margin γ ω \gamma_\omega γ ω using weights from one reference seed’s endpoint.
Figure 3 shows the results of our second experiment, comparing ε = 0 \varepsilon= 0 ε = 0 and ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 (the default PyTorch (Paszke et al. [41 ] ) value) in a setting with standard initialization. Here we train until a loss of 10 − 10 10^{-10} 1 0 − 10 or up to 1 million steps. For ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 , ε \varepsilon ε eventually dominates v ^ t \sqrt{\hat{v}_t} v ^ t in the denominator, so that all trajectories are expected to have an implicit bias towards ℓ 2 \ell_2 ℓ 2 margin maximization (see Wang et al. [48 ] for an analysis covering some cases). See Appendix § 15 for an analysis of the effect of ε \varepsilon ε on the Adam update along the trajectory.
The ε = 0 \varepsilon= 0 ε = 0 and ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 runs exhibit similar margins through most of training, consistent with ε \varepsilon ε having negligible effect during this stage. In the late phase of the ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 trajectory, corresponding to a more significant effect of ε \varepsilon ε , we see a decrease in γ ∞ \gamma_\infty γ ∞ for α ≤ − 0.5 \alpha\leq-0.5 α ≤ − 0.5 , and an increase in γ 2 \gamma_2 γ 2 . Nevertheless, γ ∞ \gamma_\infty γ ∞ shows a clear ordering between the slow, intermediate and fast regimes until the end of training; this suggests a significant lingering ℓ ∞ \ell_\infty ℓ ∞ maximization effect from the stage where ε \varepsilon ε is negligible.
The weighted-ℓ 2 \ell_2 ℓ 2 margin comparison shows the same pattern with ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 and standard initialization: the reference trajectory attains the largest margin under its own endpoint metric among the trajectories compared for α = 0 \alpha= 0 α = 0 , but not for α = − 0.7 \alpha= -0.7 α = − 0.7 . For additional seeds, schedules and ε = 0 \varepsilon= 0 ε = 0 see Appendix § 15 .
Discussion and future directions
Sign-based descent has served as a tractable surrogate for Adam, informing explanations of its optimization geometry and performance, as well as motivating optimizer design and training recommendations [3 , 54 , 34 , 31 , 33 ] . On the other hand, theoretical and empirical studies have identified differences between Adam and sign-based methods, including Signum [11 , 39 , 1 ] . Our results show that similarity between implicit biases of Adam and sign gradient descent on homogeneous models depends on the learning rate schedule, further highlighting that transferring conclusions from sign descent to Adam requires careful justification for the regime under consideration.
For the different learning-rate schedule regimes, we analyzed the implicit bias of Adam and obtained both negative and positive results. The trajectory-dependent weighted-ℓ 2 \ell_2 ℓ 2 bias of Adam in the fast regime naturally raises the question of transferability of implications of the ℓ 2 \ell_2 ℓ 2 KKT bias of gradient descent. These include, notably, a vulnerability to data leakage: previous works demonstrated that training data can be reconstructed from networks trained with gradient descent [23 , 9 , 40 , 18 ] , while a recent work argues reconstruction is difficult for Adam because of its supposed ℓ ∞ \ell_\infty ℓ ∞ bias [22 ] . Since we show that in some common settings Adam does not maximize the ℓ ∞ \ell_\infty ℓ ∞ margin, and instead maximizes a trajectory-dependent weighted ℓ 2 \ell_2 ℓ 2 margin, it remains open whether its implicit bias can be leveraged to build a data privacy attack. Additional implications of ℓ 2 \ell_2 ℓ 2 KKT conditions include positive implications in some settings for benign overfitting and generalization (Frei et al., 2023a,b; Kornowski et al., 2023).
Finally, examining the implicit bias of Adam on non-homogeneous models, as in Nacson et al. [37 ] ; Kunin et al. [29 ] ; Cai et al. [10 ] , is a promising and highly motivated line of future inquiry, due to the prevalence of these models in practice.
AI Disclosure
We were assisted by AI tools for this work. AI was used to help ideate and simplify proofs, to write code and run experiments, and to suggest improvements in writing and literature review. We have reviewed and verified all AI-assisted work.
Acknowledgments
This research is supported by the Israel Science Foundation (grant No. 2574/25), by a research grant from Mortimer Zuckerman (the Zuckerman STEM Leadership Program), and by research grants from the Center for New Scientists at the Weizmann Institute of Science, and the Shimon and Golde Picker – Weizmann Annual Grant.
Clarke Subgradients and Loss Families
We use the standard Clarke subdifferential ([12 ] ), which is well defined for any locally Lipschitz function R R R and denoted ∂ R \partial R ∂ R . For further discussion of Clarke subdifferentials for classes of regular functions, see [14 , 8 ] . We also use the following standard definition (e.g. [14 ] ):
Definition A.1. We say a locally Lipschitz function R : R p → R R : \mathbb{R}^{p} \to\mathbb{R} R : R p → R admits a chain rule if for any locally absolutely continuous path ψ : [ 0 , 1 ] → R p \psi: [0,1] \to\mathbb{R}^{p} ψ : [ 0 , 1 ] → R p , for almost any s ∈ [ 0 , 1 ] s \in[0,1] s ∈ [ 0 , 1 ] and for any h ∈ ∂ R ( ψ ( s ) ) h \in\partial R(\psi(s)) h ∈ ∂ R ( ψ ( s )) it holds that
d R ( ψ ( s ) ) d s = ⟨ ψ ˙ ( s ) , h ⟩ . \frac{dR(\psi(s))}{ds} = \langle\dot{\psi}(s), h \rangle. d s d R ( ψ ( s )) = ⟨ ψ ˙ ( s ) , h ⟩ .
Recall Equation (7) for the definition of the family of log-concave, exponentially tailed losses L e \mathcal{L}_{e} L e .
Lemma A.2. Let θ t : [ 0 , ∞ ) → R p \theta_t : [0,\infty) \to\mathbb{R}^{p} θ t : [ 0 , ∞ ) → R p be locally absolutely continuous, let { x i , y i } i = 1 m ⊆ R d × { ± 1 } \{x_i,y_i\}_{i=1}^{m} \subseteq\mathbb{R}^{d} \times\{\pm1\} { x i , y i } i = 1 m ⊆ R d × { ± 1 } and let L ( θ ) = ∑ i = 1 m ℓ ( y i f ( x i ; θ ) ) \mathcal{L}(\theta) = \sum_{i=1}^{m}\ell(y_i f(x_i;\theta)) L ( θ ) = ∑ i = 1 m ℓ ( y i f ( x i ; θ )) where ℓ = e − φ ∈ L e \ell= e^{-\varphi} \in\mathcal{L}_{e} ℓ = e − φ ∈ L e and f f f is locally Lipschitz and admits a chain rule. For almost every t t t , any subgradient g t ∈ ∂ L ( θ t ) g_t \in\partial\mathcal{L}(\theta_t) g t ∈ ∂ L ( θ t ) can be written as
g t = − ∑ i = 1 m φ ′ ( q i ( t ) ) e − φ ( q i ( t ) ) h i ( t ) , q i ( t ) : = y i f ( x i ; θ t ) , h i ( t ) ∈ ∂ q i ( θ t ) , (22) g_t = -\sum_{i=1}^{m}\varphi'(q_i(t))e^{-\varphi(q_i(t))}h_i(t), \qquad q_i(t) := y_i f(x_i;\theta_t), \qquad h_i(t) \in\partial q_i(\theta_t), \tag*{(22)} g t = − i = 1 ∑ m φ ′ ( q i ( t )) e − φ ( q i ( t )) h i ( t ) , q i ( t ) := y i f ( x i ; θ t ) , h i ( t ) ∈ ∂ q i ( θ t ) , (22) and it holds that
d d t L ( θ t ) = ⟨ g t , θ ˙ t ⟩ . (23) \frac{d}{dt}\mathcal{L}(\theta_t) = \langle g_t,\dot{\theta}_t\rangle. \tag*{(23)} d t d L ( θ t ) = ⟨ g t , θ ˙ t ⟩ . (23)
Proof. Equation (22) follows, for instance, from Corollary A.2 of Gronich and Vardi [19 ] . (23) follows since f ( x i ; ⋅ ) f(x_i;\cdot) f ( x i ; ⋅ ) , and therefore L \mathcal{L} L , admit a chain rule (for closure under composition see Lyu and Li [35 ] , Theorem I.4).
EMA - Definition and Lemmas
We follow Barakat and Bianchi [4 ] ; Ma et al. [36 ] ; Belotto da Silva and Gazeau (2020); Wang et al. [48 ] ; Gronich and Vardi [19 ] in working with continuous-time versions of momentum and Adam.
We define for a locally integrable g ∈ L l o c 1 g \in L^{1}_{\mathrm{loc}} g ∈ L loc 1 ,
EMA c [ g ] ( t ) : = c ∫ 0 t e − c ( t − s ) g ( s ) d s , (24) \operatorname{EMA}_{c}[g](t) := c\int_{0}^{t} e^{-c(t-s)}g(s)\,ds , \tag*{(24)} EMA c [ g ] ( t ) := c ∫ 0 t e − c ( t − s ) g ( s ) d s , (24)
EMA ^ c [ g ] ( t ) : = { ( 1 − e − c t ) − 1 EMA c [ g ] ( t ) , t > 0 , g ( 0 ) , t = 0 , (25) \widehat{\operatorname{EMA}}_{c}[g](t) :=
\begin{cases}
(1-e^{-ct})^{-1}\operatorname{EMA}_{c}[g](t), & t>0,\\
g(0), & t=0,
\end{cases} \tag*{(25)} EMA c [ g ] ( t ) := { ( 1 − e − c t ) − 1 EMA c [ g ] ( t ) , g ( 0 ) , t > 0 , t = 0 , (25)
And for a locally square-integrable g ∈ L l o c 2 g \in L^{2}_{\mathrm{loc}} g ∈ L loc 2
A c 1 , c 2 [ g ] ( t ) : = { EMA ^ c 1 [ g ] ( t ) EMA ^ c 2 [ g 2 ] ( t ) , EMA ^ c 2 [ g 2 ] ( t ) > 0 , 0 , otherwise . (26) A_{c_1,c_2}[g](t) :=
\begin{cases}
\dfrac{\widehat{\operatorname{EMA}}_{c_1}[g](t)}
{\sqrt{\widehat{\operatorname{EMA}}_{c_2}[g^{2}](t)}}, & \widehat{\operatorname{EMA}}_{c_2}[g^{2}](t)>0,\\
0, & \text{otherwise}.
\end{cases} \tag*{(26)} A c 1 , c 2 [ g ] ( t ) := ⎩ ⎨ ⎧ EMA c 2 [ g 2 ] ( t ) EMA c 1 [ g ] ( t ) , 0 , EMA c 2 [ g 2 ] ( t ) > 0 , otherwise . (26)
We define continuous-flow Adam for a measurable choice of subgradients g t ∈ ∂ L ( θ t ) g_t \in\partial\mathcal{L}(\theta_t) g t ∈ ∂ L ( θ t ) , smoothing constants c 1 , c 2 > 0 c_1,c_2 > 0 c 1 , c 2 > 0 and a learning rate schedule η ( t ) \eta(t) η ( t ) with the following ODE:
d θ t d t = − η ( t ) A c 1 , c 2 [ g t ] , (27) \frac{d\theta_t}{dt} = -\eta(t)A_{c_1,c_2}[g_t], \tag*{(27)} d t d θ t = − η ( t ) A c 1 , c 2 [ g t ] , (27)
for almost every $t.
Discrete Adam ([27 ] ) with time steps n ≥ 0 n \ge0 n ≥ 0 and subgradients g n ∈ ∂ L ( θ n − 1 ) \mathbf{g}_n \in\partial\mathcal{L}(\theta_{n-1}) g n ∈ ∂ L ( θ n − 1 ) for n ≥ 1 n \ge1 n ≥ 1 is defined as follows for β 1 , β 2 ∈ [ 0 , 1 ) \beta_1,\beta_2 \in[0,1) β 1 , β 2 ∈ [ 0 , 1 ) and ε ≥ 0 \varepsilon\ge0 ε ≥ 0 :
m n = β 1 m n − 1 + ( 1 − β 1 ) g n ( n ≥ 1 ) , m 0 = 0 , m ^ n = ( 1 − β 1 n ) − 1 m n (28) \mathbf{m}_n = \beta_1\mathbf{m}_{n-1} + (1-\beta_1)\mathbf{g}_n \quad(n \ge1), \qquad\mathbf{m}_0 = 0, \qquad\widehat{\mathbf{m}}_n = (1-\beta_1^n)^{-1}\mathbf{m}_n \tag*{(28)} m n = β 1 m n − 1 + ( 1 − β 1 ) g n ( n ≥ 1 ) , m 0 = 0 , m n = ( 1 − β 1 n ) − 1 m n (28)
v n = β 2 v n − 1 + ( 1 − β 2 ) g n 2 ( n ≥ 1 ) , v 0 = 0 , v ^ n = ( 1 − β 2 n ) − 1 v n (29) \mathbf{v}_n = \beta_2\mathbf{v}_{n-1} + (1-\beta_2)\mathbf{g}_n^2 \quad(n \ge1), \qquad\mathbf{v}_0 = 0, \qquad\widehat{\mathbf{v}}_n = (1-\beta_2^n)^{-1}\mathbf{v}_n \tag*{(29)} v n = β 2 v n − 1 + ( 1 − β 2 ) g n 2 ( n ≥ 1 ) , v 0 = 0 , v n = ( 1 − β 2 n ) − 1 v n (29)
θ n = θ n − 1 − η n m ^ n v ^ n + ε ( n ≥ 1 ) . (30) \theta_n = \theta_{n-1} - \eta_n \frac{\widehat{\mathbf{m}}_n}{\sqrt{\widehat{\mathbf{v}}_n}+\varepsilon} \quad(n \ge1). \tag*{(30)} θ n = θ n − 1 − η n v n + ε m n ( n ≥ 1 ) . (30)
Here the square, square root and division are taken coordinate-wise.
The definitions of continuous momentum follow previous works. See the discussion on the connection between c i c_i c i and β i \beta_i β i in Gronich and Vardi [19 ] , which explains why c i ≈ − log ( β i ) c_i \approx-\log(\beta_i) c i ≈ − log ( β i ) when matching the continuous time clock to the iteration count t ↔ n t \leftrightarrow n t ↔ n . Note that while the works of Barakat and Bianchi [4 ] ; Belotto da Silva and Gazeau (2020) consider a training regime in which the hyperparameters β i \beta_i β i are adjusted during training to account for learning rate decay, we adopt the practical setting in which β i \beta_i β i remain fixed, therefore fixing c i c_i c i and adding an external learning rate schedule η ( t ) \eta(t) η ( t ) to the definition of Adam.
We proceed to prove useful lemmas about EMA and the Adam ratio.
Lemma B.1 (Continuous EMA ODE). For any g ∈ L l o c 1 g \in L^1_{\mathrm{loc}} g ∈ L loc 1 , and any c > 0 c>0 c > 0 , it holds almost everywhere that
d d t EMA c [ g ] = c ( g ( t ) − EMA c [ g ] ( t ) ) \frac{d}{dt}\operatorname{EMA}_c[g] = c\bigl(g(t)-\operatorname{EMA}_c[g](t)\bigr) d t d EMA c [ g ] = c ( g ( t ) − EMA c [ g ] ( t ) ) Proof. Follows from direct differentiation.
Lemma B.2 (Adam Ratio is Universally Bounded). Let c 1 > c 2 / 2 > 0 c_1>c_2/2>0 c 1 > c 2 /2 > 0 , and let g ∈ L l o c 2 g \in L^2_{\mathrm{loc}} g ∈ L loc 2 . Then, for every t ≥ 0 t \ge0 t ≥ 0 ,
∣ A c 1 , c 2 [ g ] ( t ) ∣ ≤ K A ( c 1 , c 2 ) : = c 1 c 2 ( 2 c 1 − c 2 ) . \left|A_{c_1,c_2}[g](t)\right| \le K_A(c_1,c_2) := \frac{c_1}{\sqrt{c_2(2c_1-c_2)}}. ∣ A c 1 , c 2 [ g ] ( t ) ∣ ≤ K A ( c 1 , c 2 ) := c 2 ( 2 c 1 − c 2 ) c 1 . Proof. At t = 0 t=0 t = 0 , the bound follows from ∣ A [ g ] ( 0 ) ∣ ≤ 1 ≤ K A \left|A[g](0)\right|\le1\le K_A ∣ A [ g ] ( 0 ) ∣ ≤ 1 ≤ K A . Hence assume t > 0 t>0 t > 0 .
If EMA ^ c 2 [ g 2 ] ( t ) = 0 \widehat{\operatorname{EMA}}_{c_2}[g^2](t)=0 EMA c 2 [ g 2 ] ( t ) = 0 then A c 1 , c 2 [ g ] ( t ) = 0 A_{c_1,c_2}[g](t)=0 A c 1 , c 2 [ g ] ( t ) = 0 by definition. Otherwise, Cauchy–Schwarz gives
∣ ∫ 0 t e − c 1 ( t − s ) g ( s ) d s ∣ 2 ≤ ( ∫ 0 t e − c 2 ( t − s ) g ( s ) 2 d s ) ( ∫ 0 t e − ( 2 c 1 − c 2 ) ( t − s ) d s ) . \left|\int_0^t e^{-c_1(t-s)}g(s)\,ds\right|^2 \le\left(\int_0^t e^{-c_2(t-s)}g(s)^2\,ds\right)\left(\int_0^t e^{-(2c_1-c_2)(t-s)}\,ds\right). ∫ 0 t e − c 1 ( t − s ) g ( s ) d s 2 ≤ ( ∫ 0 t e − c 2 ( t − s ) g ( s ) 2 d s ) ( ∫ 0 t e − ( 2 c 1 − c 2 ) ( t − s ) d s ) . Substituting the normalization factors in the definition of A c 1 , c 2 A_{c_1,c_2} A c 1 , c 2 yields
∣ A c 1 , c 2 [ g ] ( t ) ∣ ≤ c 1 c 2 ( 2 c 1 − c 2 ) ( 1 − e − c 2 t ) ( 1 − e − ( 2 c 1 − c 2 ) t ) 1 − e − c 1 t . \left|A_{c_1,c_2}[g](t)\right| \le\frac{c_1}{\sqrt{c_2(2c_1-c_2)}}\frac{\sqrt{(1-e^{-c_2t})(1-e^{-(2c_1-c_2)t})}}{1-e^{-c_1t}}. ∣ A c 1 , c 2 [ g ] ( t ) ∣ ≤ c 2 ( 2 c 1 − c 2 ) c 1 1 − e − c 1 t ( 1 − e − c 2 t ) ( 1 − e − ( 2 c 1 − c 2 ) t ) . Now,
( 1 − e − c 1 t ) 2 − ( 1 − e − c 2 t ) ( 1 − e − ( 2 c 1 − c 2 ) t ) = ( e − c 2 t / 2 − e − ( 2 c 1 − c 2 ) t / 2 ) 2 ≥ 0. (1-e^{-c_1t})^2-(1-e^{-c_2t})(1-e^{-(2c_1-c_2)t})=\left(e^{-c_2t/2}-e^{-(2c_1-c_2)t/2}\right)^2\ge0. ( 1 − e − c 1 t ) 2 − ( 1 − e − c 2 t ) ( 1 − e − ( 2 c 1 − c 2 ) t ) = ( e − c 2 t /2 − e − ( 2 c 1 − c 2 ) t /2 ) 2 ≥ 0. The last fraction is therefore at most one, as required.
Lemma B.3 (Time Derivative of Uncorrected Adam Ratio). Let g ∈ L l o c 2 g \in L^2_{\mathrm{loc}} g ∈ L loc 2 and denote m = EMA c 1 [ g ] m=\operatorname{EMA}_{c_1}[g] m = EMA c 1 [ g ] , v = EMA c 2 [ g 2 ] v=\operatorname{EMA}_{c_2}[g^2] v = EMA c 2 [ g 2 ] . The quotient m / v m/\sqrt{v} m / v is locally absolutely continuous on { v > 0 } \{v>0\} { v > 0 } , and the following identity holds for almost every t > 0 t>0 t > 0 with v ( t ) > 0 v(t)>0 v ( t ) > 0 :
d d t ( m v ) = c 1 g v − ( c 1 − c 2 2 + c 2 g 2 2 v ) m v . \frac{d}{dt}\left(\frac{m}{\sqrt{v}}\right)=c_1\frac{g}{\sqrt{v}}-\left(c_1-\frac{c_2}{2}+\frac{c_2g^2}{2v}\right)\frac{m}{\sqrt{v}}. d t d ( v m ) = c 1 v g − ( c 1 − 2 c 2 + 2 v c 2 g 2 ) v m . Proof. Direct differentiation, using definitions of m , v m,v m , v :
d d t ( m v ) = c 1 ( g − m ) v − m c 2 ( g 2 − v ) 2 v v . \frac{d}{dt}\left(\frac{m}{\sqrt{v}}\right)=\frac{c_1(g-m)\sqrt{v}-m\frac{c_2(g^2-v)}{2\sqrt{v}}}{v}. d t d ( v m ) = v c 1 ( g − m ) v − m 2 v c 2 ( g 2 − v ) .
Lemma B.4 (Weighted Cesàro principle). Let W : [ 0 , ∞ ) → [ 0 , ∞ ) ∈ L l o c 1 W : [0,\infty) \to[0,\infty) \in L^1_{\mathrm{loc}} W : [ 0 , ∞ ) → [ 0 , ∞ ) ∈ L loc 1 with ∫ 0 t W ( s ) d s → ∞ \int_0^t W(s)\,ds \to\infty ∫ 0 t W ( s ) d s → ∞ , and let h h h be so that W h ∈ L l o c 1 Wh \in L^1_{\mathrm{loc}} W h ∈ L loc 1 . If h ( t ) → h ∞ ∈ ( − ∞ , ∞ ) h(t) \to h_\infty\in(-\infty,\infty) h ( t ) → h ∞ ∈ ( − ∞ , ∞ ) , then
∫ 0 t W ( s ) h ( s ) d s ∫ 0 t W ( s ) d s ⟶ h ∞ . (31) \frac{\int_0^t W(s)h(s)\,ds}{\int_0^t W(s)\,ds} \longrightarrow h_\infty. \tag*{(31)} ∫ 0 t W ( s ) d s ∫ 0 t W ( s ) h ( s ) d s ⟶ h ∞ . (31)
Proof. Fix ε > 0 \varepsilon> 0 ε > 0 and choose T > 0 T > 0 T > 0 such that ∣ h ( s ) − h ∞ ∣ ≤ ε / 2 |h(s)-h_\infty| \le\varepsilon/2 ∣ h ( s ) − h ∞ ∣ ≤ ε /2 for all s ≥ T s \ge T s ≥ T . Since W W W and W h Wh W h are locally integrable,
C T : = ∫ 0 T W ( s ) ∣ h ( s ) − h ∞ ∣ d s < ∞ . C_T := \int_0^T W(s)|h(s)-h_\infty|\,ds < \infty. C T := ∫ 0 T W ( s ) ∣ h ( s ) − h ∞ ∣ d s < ∞.
For all sufficiently large t ≥ T t \ge T t ≥ T , the denominator ∫ 0 t W ( s ) d s \int_0^t W(s)\,ds ∫ 0 t W ( s ) d s is positive. Splitting the integral at T T T and using W ≥ 0 W \ge0 W ≥ 0 , we obtain
∣ ∫ 0 t W ( s ) h ( s ) d s ∫ 0 t W ( s ) d s − h ∞ ∣ = ∣ ∫ 0 t W ( s ) ( h ( s ) − h ∞ ) d s ∣ ∫ 0 t W ( s ) d s ≤ ∫ 0 T W ( s ) ∣ h ( s ) − h ∞ ∣ d s ∫ 0 t W ( s ) d s + ∫ T t W ( s ) ∣ h ( s ) − h ∞ ∣ d s ∫ 0 t W ( s ) d s ≤ C T ∫ 0 t W ( s ) d s + ε 2 ∫ T t W ( s ) d s ∫ 0 t W ( s ) d s ≤ C T ∫ 0 t W ( s ) d s + ε 2 . \begin{aligned}
\left|\frac{\int_0^t W(s)h(s)\,ds}{\int_0^t W(s)\,ds}-h_\infty\right|
=
\frac{\left|\int_0^t W(s)(h(s)-h_\infty)\,ds\right|}{\int_0^t W(s)\,ds} \\
&\le
\frac{\int_0^T W(s)|h(s)-h_\infty|\,ds}{\int_0^t W(s)\,ds}
+
\frac{\int_T^t W(s)|h(s)-h_\infty|\,ds}{\int_0^t W(s)\,ds} \\
&\le
\frac{C_T}{\int_0^t W(s)\,ds}
+
\frac{\varepsilon}{2}\frac{\int_T^t W(s)\,ds}{\int_0^t W(s)\,ds} \\
&\le
\frac{C_T}{\int_0^t W(s)\,ds}+\frac{\varepsilon}{2}.
\end{aligned} ∫ 0 t W ( s ) d s ∫ 0 t W ( s ) h ( s ) d s − h ∞ = ∫ 0 t W ( s ) d s ∫ 0 t W ( s ) ( h ( s ) − h ∞ ) d s ≤ ∫ 0 t W ( s ) d s ∫ 0 T W ( s ) ∣ h ( s ) − h ∞ ∣ d s + ∫ 0 t W ( s ) d s ∫ T t W ( s ) ∣ h ( s ) − h ∞ ∣ d s ≤ ∫ 0 t W ( s ) d s C T + 2 ε ∫ 0 t W ( s ) d s ∫ T t W ( s ) d s ≤ ∫ 0 t W ( s ) d s C T + 2 ε .
Since C T C_T C T is finite and ∫ 0 t W ( s ) d s ⟶ t → ∞ ∞ \int_0^t W(s)\,ds \mathrel{\mathop{\longrightarrow}\limits_{t\to\infty}} \infty ∫ 0 t W ( s ) d s t → ∞ ⟶ ∞ , there exists T 1 ≥ T T_1 \ge T T 1 ≥ T such that
C T ∫ 0 t W ( s ) d s ≤ ε 2 for all t ≥ T 1 . \frac{C_T}{\int_0^t W(s)\,ds} \le\frac{\varepsilon}{2}
\qquad\text{for all }t \ge T_1. ∫ 0 t W ( s ) d s C T ≤ 2 ε for all t ≥ T 1 .
Thus the absolute difference in (31) is at most ε \varepsilon ε for every t ≥ T 1 t \ge T_1 t ≥ T 1 . Since ε > 0 \varepsilon> 0 ε > 0 was arbitrary, the result follows. □
Corollary B.5 (Weighted EMA). Let 0 ≤ g ∈ L l o c 1 0 \le g \in L^1_{\mathrm{loc}} 0 ≤ g ∈ L loc 1 , g h ∈ L l o c 1 gh \in L^1_{\mathrm{loc}} g h ∈ L loc 1 , and assume h ( t ) ⟶ t → ∞ h ∞ ∈ ( − ∞ , ∞ ) h(t) \mathrel{\mathop{\longrightarrow}\limits_{t\to\infty}} h_\infty\in(-\infty,\infty) h ( t ) t → ∞ ⟶ h ∞ ∈ ( − ∞ , ∞ ) and e c t EMA c [ g ] ( t ) ⟶ t → ∞ ∞ e^{ct}\operatorname{EMA}_c[g](t) \mathrel{\mathop{\longrightarrow}\limits_{t\to\infty}} \infty e c t EMA c [ g ] ( t ) t → ∞ ⟶ ∞ . Then
EMA c [ g h ] ( t ) EMA c [ g ] ( t ) ⟶ t → ∞ h ∞ , EMA ^ c [ g h ] ( t ) EMA ^ c [ g ] ( t ) ⟶ t → ∞ h ∞ . \frac{\operatorname{EMA}_c[gh](t)}{\operatorname{EMA}_c[g](t)}
\mathrel{\mathop{\longrightarrow}\limits_{t\to\infty}} h_\infty,
\qquad
\frac{\widehat{\operatorname{EMA}}_c[gh](t)}{\widehat{\operatorname{EMA}}_c[g](t)}
\mathrel{\mathop{\longrightarrow}\limits_{t\to\infty}} h_\infty. EMA c [ g ] ( t ) EMA c [ g h ] ( t ) t → ∞ ⟶ h ∞ , EMA c [ g ] ( t ) EMA c [ g h ] ( t ) t → ∞ ⟶ h ∞ .
Proof. Follows by definition from B.4, after cancelling e − c t e^{-ct} e − c t and bias correction terms. □
Lemma B.6 (EMA Weighted by Slowly Varying Multiplier). Let 0 < g ∈ L l o c 1 0 < g \in L^1_{\mathrm{loc}} 0 < g ∈ L loc 1 and let A : [ 0 , ∞ ) → ( 0 , ∞ ) A : [0,\infty) \to(0,\infty) A : [ 0 , ∞ ) → ( 0 , ∞ ) be locally absolutely continuous. Let ε : [ 0 , ∞ ) → R \varepsilon: [0,\infty) \to\mathbb{R} ε : [ 0 , ∞ ) → R be bounded and measurable with ε ( t ) → 0 \varepsilon(t) \to0 ε ( t ) → 0 . Let c > 0 c > 0 c > 0 . Suppose that for some μ > 0 \mu> 0 μ > 0
log ( e c t g ( t ) ) = μ t ± o ( t ) , ∣ d d t log A ( t ) ∣ ≤ O ( 1 t ) . \log\left(e^{ct}g(t)\right)=\mu t \pm o(t),
\qquad
\left|\frac{d}{dt}\log A(t)\right|\le O\left(\frac{1}{t}\right). log ( e c t g ( t ) ) = μ t ± o ( t ) , d t d log A ( t ) ≤ O ( t 1 ) . Then,
EMA c [ A ( 1 + ε ) g ] ( t ) EMA c [ g ] ( t ) = A ( t ) ( 1 + o ( 1 ) ) . \frac{\operatorname{EMA}_c[A(1+\varepsilon)g](t)}{\operatorname{EMA}_c[g](t)}
=A(t)(1+o(1)). EMA c [ g ] ( t ) EMA c [ A ( 1 + ε ) g ] ( t ) = A ( t ) ( 1 + o ( 1 )) . Proof. Intuitively, the exponential factor in g g g makes both EMAs depend asymptotically only on times close to t t t , while the logarithmic derivative bound makes A A A nearly constant on that window. To make this precise, denote
g ~ ( t ) = e c t g ( t ) , J ( t ) = ∫ 0 t g ~ ( s ) d s , E t ( s ) = A ( s ) A ( t ) ( 1 + ε ( s ) ) − 1. \widetilde{g}(t)=e^{ct}g(t),\qquad J(t)=\int_0^t \widetilde{g}(s)\,ds,\qquad E_t(s)=\frac{A(s)}{A(t)}(1+\varepsilon(s))-1. g ( t ) = e c t g ( t ) , J ( t ) = ∫ 0 t g ( s ) d s , E t ( s ) = A ( t ) A ( s ) ( 1 + ε ( s )) − 1.
Note that the required statement is equivalent, after cancelling shared exponentials and normalizing constants, to:
∫ 0 t g ~ ( s ) A ( s ) A ( t ) ( 1 + ε ( s ) ) d s = ( 1 + o ( 1 ) ) J ( t ) , \int_{0}^{t} \widetilde{g}(s)\frac{A(s)}{A(t)}(1+\varepsilon(s))\,\mathrm{d}s=(1+o(1))J(t), ∫ 0 t g ( s ) A ( t ) A ( s ) ( 1 + ε ( s )) d s = ( 1 + o ( 1 )) J ( t ) ,
and hence also to:
∫ 0 t g ~ ( s ) E t ( s ) d s = o ( J ( t ) ) . \int_{0}^{t}\widetilde{g}(s)E_t(s)\,\mathrm{d}s=o(J(t)). ∫ 0 t g ( s ) E t ( s ) d s = o ( J ( t )) .
By the assumption, log g ~ ( t ) = μ t + o ( t ) \log\widetilde{g}(t)=\mu t+o(t) log g ( t ) = μ t + o ( t ) ; this implies log J ( t ) = μ t + o ( t ) \log J(t)=\mu t+o(t) log J ( t ) = μ t + o ( t ) as well. Indeed, for any 0 < δ < μ 0<\delta<\mu 0 < δ < μ , eventually e ( μ − δ ) t ≤ g ~ ( t ) ≤ e ( μ + δ ) t e^{(\mu-\delta)t}\leq\widetilde{g}(t)\leq e^{(\mu+\delta)t} e ( μ − δ ) t ≤ g ( t ) ≤ e ( μ + δ ) t for t ≥ t δ t\geq t_\delta t ≥ t δ , so
e ( μ − δ ) t ≲ ∫ t δ t e ( μ − δ ) s d s ≤ J ( t ) ≤ ∫ 0 t δ g ~ ( s ) d s + ∫ t δ t e ( μ + δ ) s d s ≲ e ( μ + δ ) t . e^{(\mu-\delta)t}\lesssim\int_{t_\delta}^{t}e^{(\mu-\delta)s}\,\mathrm{d}s\leq J(t)\leq\int_{0}^{t_\delta}\widetilde{g}(s)\,\mathrm{d}s+\int_{t_\delta}^{t}e^{(\mu+\delta)s}\,\mathrm{d}s\lesssim e^{(\mu+\delta)t}. e ( μ − δ ) t ≲ ∫ t δ t e ( μ − δ ) s d s ≤ J ( t ) ≤ ∫ 0 t δ g ( s ) d s + ∫ t δ t e ( μ + δ ) s d s ≲ e ( μ + δ ) t .
Taking logarithm and δ → 0 \delta\to0 δ → 0 proves the claim. Next, by the logarithmic derivative assumption, there exist constants K > 0 K>0 K > 0 and T 0 ≥ 1 T_0\geq1 T 0 ≥ 1 such that ∣ d d t log A ∣ ≤ K t \left|\frac{\mathrm{d}}{\mathrm{d}t}\log A\right|\leq\frac{K}{t} d t d log A ≤ t K for t ≥ T 0 t\geq T_0 t ≥ T 0 , so
( s t ) K ≤ A ( s ) A ( t ) ≤ ( t s ) K , T 0 ≤ s ≤ t . (32) \left(\frac{s}{t}\right)^{K}\leq\frac{A(s)}{A(t)}\leq\left(\frac{t}{s}\right)^{K},\qquad T_0\leq s\leq t. \tag*{(32)} ( t s ) K ≤ A ( t ) A ( s ) ≤ ( s t ) K , T 0 ≤ s ≤ t . (32)
For 0 ≤ s < T 0 ≤ t 0\leq s<T_0\leq t 0 ≤ s < T 0 ≤ t , we first apply (32) at s = T 0 s=T_0 s = T 0 to obtain
A ( T 0 ) A ( t ) ≤ ( t T 0 ) K . \frac{A(T_0)}{A(t)}\leq\left(\frac{t}{T_0}\right)^{K}. A ( t ) A ( T 0 ) ≤ ( T 0 t ) K .
Since A A A is continuous and positive, the constants
M : = max 0 ≤ s ≤ T 0 A ( s ) , B : = sup s ≥ 0 ∣ ε ( s ) ∣ M:=\max_{0\leq s\leq T_0}A(s),\qquad B:=\sup_{s\geq0}|\varepsilon(s)| M := 0 ≤ s ≤ T 0 max A ( s ) , B := s ≥ 0 sup ∣ ε ( s ) ∣
are finite, and A ( T 0 ) > 0 A(T_0)>0 A ( T 0 ) > 0 . Consequently,
A ( s ) A ( t ) = A ( s ) A ( T 0 ) A ( T 0 ) A ( t ) ≤ M A ( T 0 ) ( t T 0 ) K , s < T 0 ≤ t . (33) \frac{A(s)}{A(t)}=\frac{A(s)}{A(T_0)}\frac{A(T_0)}{A(t)}\leq\frac{M}{A(T_0)}\left(\frac{t}{T_0}\right)^{K},\qquad s<T_0\leq t. \tag*{(33)} A ( t ) A ( s ) = A ( T 0 ) A ( s ) A ( t ) A ( T 0 ) ≤ A ( T 0 ) M ( T 0 t ) K , s < T 0 ≤ t . (33)
Combining (32) –(33) gives for some C > 0 C>0 C > 0 and sufficiently large T 1 T_1 T 1 ,
∣ E t ( s ) ∣ ≤ A ( s ) A ( t ) ( 1 + ∣ ε ( s ) ∣ ) + 1 ≤ C t K , 0 ≤ s ≤ t , T 1 ≤ t . |E_t(s)|\leq\frac{A(s)}{A(t)}(1+|\varepsilon(s)|)+1\leq Ct^{K},\qquad0\leq s\leq t,\quad T_1\leq t. ∣ E t ( s ) ∣ ≤ A ( t ) A ( s ) ( 1 + ∣ ε ( s ) ∣ ) + 1 ≤ C t K , 0 ≤ s ≤ t , T 1 ≤ t .
Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . On [ 0 , ( 1 − δ ) t ] [0,(1-\delta)t] [ 0 , ( 1 − δ ) t ] for t ≥ T 1 t\geq T_1 t ≥ T 1 , we have
∣ 1 J ( t ) ∫ 0 ( 1 − δ ) t g ~ ( s ) E t ( s ) d s ∣ ≤ C t K J ( ( 1 − δ ) t ) J ( t ) = e − μ δ t + o ( t ) → t → ∞ 0. \left|\frac{1}{J(t)}\int_{0}^{(1-\delta)t}\widetilde{g}(s)E_t(s)\,\mathrm{d}s\right|\leq Ct^{K}\frac{J((1-\delta)t)}{J(t)}=e^{-\mu\delta t+o(t)}\xrightarrow[t\to\infty]{}0. J ( t ) 1 ∫ 0 ( 1 − δ ) t g ( s ) E t ( s ) d s ≤ C t K J ( t ) J (( 1 − δ ) t ) = e − μ δ t + o ( t ) t → ∞ 0.
On [ ( 1 − δ ) t , t ] [(1-\delta)t,t] [( 1 − δ ) t , t ] , (32) gives via the triangle inequality,
sup ( 1 − δ ) t ≤ s ≤ t ∣ E t ( s ) ∣ ≤ ( 1 − δ ) − K − 1 + ( 1 − δ ) − K sup s ≥ ( 1 − δ ) t ∣ ε ( s ) ∣ → t → ∞ ( 1 − δ ) − K − 1. \sup_{(1-\delta)t\leq s\leq t}|E_t(s)|\leq(1-\delta)^{-K}-1+(1-\delta)^{-K}\sup_{s\geq(1-\delta)t}|\varepsilon(s)|\xrightarrow[t\to\infty]{}(1-\delta)^{-K}-1. ( 1 − δ ) t ≤ s ≤ t sup ∣ E t ( s ) ∣ ≤ ( 1 − δ ) − K − 1 + ( 1 − δ ) − K s ≥ ( 1 − δ ) t sup ∣ ε ( s ) ∣ t → ∞ ( 1 − δ ) − K − 1.
Which yields
lim sup t → ∞ ∣ 1 J ( t ) ∫ ( 1 − δ ) t t g ~ ( s ) E t ( s ) d s ∣ ≤ ( 1 − δ ) − K − 1. \limsup_{t\to\infty}\left|\frac{1}{J(t)}\int_{(1-\delta)t}^{t}\widetilde{g}(s)E_t(s)\,\mathrm{d}s\right|\leq(1-\delta)^{-K}-1. t → ∞ lim sup J ( t ) 1 ∫ ( 1 − δ ) t t g ( s ) E t ( s ) d s ≤ ( 1 − δ ) − K − 1.
Taking δ → 0 \delta\to0 δ → 0 and combining with the estimate on [ 0 , ( 1 − δ ) t ] [0,(1-\delta)t] [ 0 , ( 1 − δ ) t ] gives the desired result. □ \square □
Lemma B.7 (L p L^p L p bounds). Let 1 ≤ p ≤ ∞ 1\leq p\leq\infty 1 ≤ p ≤ ∞ , g ∈ L l o c p g\in L^p_{\mathrm{loc}} g ∈ L loc p , and 0 < λ < c 0<\lambda<c 0 < λ < c . Then
( ∫ 0 t ∣ e λ s EMA c [ g ] ( s ) ∣ p d s ) 1 / p ≤ c c − λ ( ∫ 0 t ∣ e λ s g ( s ) ∣ p d s ) 1 / p . \left(\int_{0}^{t}\left|e^{\lambda s}\operatorname{EMA}_c[g](s)\right|^{p}\,\mathrm{d}s\right)^{1/p}\leq\frac{c}{c-\lambda}\left(\int_{0}^{t}\left|e^{\lambda s}g(s)\right|^{p}\,\mathrm{d}s\right)^{1/p}. ( ∫ 0 t e λ s EMA c [ g ] ( s ) p d s ) 1/ p ≤ c − λ c ( ∫ 0 t e λ s g ( s ) p d s ) 1/ p .
Proof. Fix t > 0 t > 0 t > 0 . Define h ( u ) = e λ u g ( u ) h(u) = e^{\lambda u}g(u) h ( u ) = e λ u g ( u ) for 0 ≤ u ≤ t 0 \le u \le t 0 ≤ u ≤ t , and set h ( u ) = 0 h(u) = 0 h ( u ) = 0 otherwise. For 0 ≤ s ≤ t 0 \le s \le t 0 ≤ s ≤ t ,
e λ s EMA c [ g ] ( s ) = c ∫ 0 s e − ( c − λ ) ( s − u ) h ( u ) d u = c ∫ 0 ∞ e − ( c − λ ) v h ( s − v ) d v . e^{\lambda s}\operatorname{EMA}_{c}[g](s) = c \int_{0}^{s} e^{-(c-\lambda)(s-u)}h(u)\,du = c \int_{0}^{\infty} e^{-(c-\lambda)v}h(s-v)\,dv. e λ s EMA c [ g ] ( s ) = c ∫ 0 s e − ( c − λ ) ( s − u ) h ( u ) d u = c ∫ 0 ∞ e − ( c − λ ) v h ( s − v ) d v .
Minkowski’s integral inequality states that, for p ≥ 1 p \ge1 p ≥ 1 ,
( ∫ 0 t ∣ ∫ 0 ∞ F ( s , v ) d v ∣ p d s ) 1 / p ≤ ∫ 0 ∞ ( ∫ 0 t ∣ F ( s , v ) ∣ p d s ) 1 / p d v . \left(\int_{0}^{t}\left|\int_{0}^{\infty}F(s,v)\,dv\right|^{p}ds\right)^{1/p} \le\int_{0}^{\infty}\left(\int_{0}^{t}|F(s,v)|^{p}ds\right)^{1/p}dv. ( ∫ 0 t ∫ 0 ∞ F ( s , v ) d v p d s ) 1/ p ≤ ∫ 0 ∞ ( ∫ 0 t ∣ F ( s , v ) ∣ p d s ) 1/ p d v .
We apply it with F ( s , v ) = c e − ( c − λ ) v h ( s − v ) F(s,v) = ce^{-(c-\lambda)v}h(s-v) F ( s , v ) = c e − ( c − λ ) v h ( s − v ) . Since h h h vanishes outside [ 0 , t ] [0,t] [ 0 , t ] , a change of variables gives, for every v ≥ 0 v \ge0 v ≥ 0 ,
∫ 0 t ∣ h ( s − v ) ∣ p d s = ∫ − v t − v ∣ h ( u ) ∣ p d u ≤ ∫ 0 t e p λ u ∣ g ( u ) ∣ p d u . \int_{0}^{t}|h(s-v)|^{p}\,ds = \int_{-v}^{t-v}|h(u)|^{p}\,du \le\int_{0}^{t}e^{p\lambda u}|g(u)|^{p}\,du. ∫ 0 t ∣ h ( s − v ) ∣ p d s = ∫ − v t − v ∣ h ( u ) ∣ p d u ≤ ∫ 0 t e p λ u ∣ g ( u ) ∣ p d u .
Consequently,
( ∫ 0 t ∣ e λ s EMA c [ g ] ( s ) ∣ p d s ) 1 / p ≤ c ∫ 0 ∞ e − ( c − λ ) v ( ∫ 0 t ∣ h ( s − v ) ∣ p d s ) 1 / p d v ≤ c c − λ ( ∫ 0 t e p λ u ∣ g ( u ) ∣ p d u ) 1 / p . \begin{aligned}
\left(\int_{0}^{t}|e^{\lambda s}\operatorname{EMA}_{c}[g](s)|^{p}\,ds\right)^{1/p}
\le c\int_{0}^{\infty}e^{-(c-\lambda)v}\left(\int_{0}^{t}|h(s-v)|^{p}\,ds\right)^{1/p}dv \\
&\le\frac{c}{c-\lambda}\left(\int_{0}^{t}e^{p\lambda u}|g(u)|^{p}\,du\right)^{1/p}.
\end{aligned} ( ∫ 0 t ∣ e λ s EMA c [ g ] ( s ) ∣ p d s ) 1/ p ≤ c ∫ 0 ∞ e − ( c − λ ) v ( ∫ 0 t ∣ h ( s − v ) ∣ p d s ) 1/ p d v ≤ c − λ c ( ∫ 0 t e p λ u ∣ g ( u ) ∣ p d u ) 1/ p .
Adam Dynamics Analysis
Define the momentum-corrected loss L A \mathcal{L}_{A} L A , introduced by Barakat and Bianchi [4 ] :
H ( t ) : = ∑ j = 1 p m j ( t ) 2 v j ( t ) = ⟨ m t , m t v t ⟩ , η ^ ( t ) : = 1 − e − c 2 t 1 − e − c 1 t η ( t ) , L A ( t ) : = L ( θ t ) + η ^ ( t ) 2 c 1 H ( t ) . (34) H(t) := \sum_{j=1}^{p}\frac{m_{j}(t)^{2}}{\sqrt{v_{j}(t)}} = \left\langle\mathbf{m}_{t},\frac{\mathbf{m}_{t}}{\sqrt{\mathbf{v}_{t}}}\right\rangle,\qquad\hat{\eta}(t) := \frac{\sqrt{1-e^{-c_{2}t}}}{1-e^{-c_{1}t}}\eta(t),\qquad\mathcal{L}_{A}(t) := \mathcal{L}(\theta_{t})+\frac{\hat{\eta}(t)}{2c_{1}}H(t). \tag*{(34)} H ( t ) := j = 1 ∑ p v j ( t ) m j ( t ) 2 = ⟨ m t , v t m t ⟩ , η ^ ( t ) := 1 − e − c 1 t 1 − e − c 2 t η ( t ) , L A ( t ) := L ( θ t ) + 2 c 1 η ^ ( t ) H ( t ) . (34)
Summands with v t [ j ] = 0 v_{t}[j] = 0 v t [ j ] = 0 are interpreted as zero.
Set
μ ( t ) : = 2 c 1 − c 2 2 − d d t log η ^ ( t ) . (35) \mu(t) := 2c_{1}-\frac{c_{2}}{2}-\frac{d}{dt}\log\hat{\eta}(t). \tag*{(35)} μ ( t ) := 2 c 1 − 2 c 2 − d t d log η ^ ( t ) . (35)
For the polynomial schedule η ( t ) = η 0 ( 1 + t ) α \eta(t)=\eta_{0}(1+t)^{\alpha} η ( t ) = η 0 ( 1 + t ) α it holds that μ ( t ) → μ ∞ : = 2 c 1 − c 2 / 2 ≥ 3 2 c 1 \mu(t)\to\mu_{\infty}:=2c_{1}-c_{2}/2\ge\frac{3}{2}c_{1} μ ( t ) → μ ∞ := 2 c 1 − c 2 /2 ≥ 2 3 c 1 . Therefore, let t A > 0 t_{A}>0 t A > 0 with
μ ( t ) ≥ μ 0 , c 1 < μ 0 < μ ∞ , ( t ≥ t A ) . \mu(t)\ge\mu_{0},\qquad c_{1}<\mu_{0}<\mu_{\infty},\qquad(t\ge t_{A}). μ ( t ) ≥ μ 0 , c 1 < μ 0 < μ ∞ , ( t ≥ t A ) .
Also, with the exception of Lemmas C.1, C.2 which require no assumptions, and Lemma C.15 which requires only (T1), we work under the assumptions ∀ t ≥ t 0 : γ 2 ( θ t ) ≥ γ 0 > 0 \forall t\ge t_{0}:\gamma_{2}(\theta_{t})\ge\gamma_{0}>0 ∀ t ≥ t 0 : γ 2 ( θ t ) ≥ γ 0 > 0 and L ( θ t ) ⟶ t → ∞ 0 \mathcal{L}(\theta_{t})\mathop{\longrightarrow}_{t\to\infty}0 L ( θ t ) ⟶ t → ∞ 0 (Assumptions (T1), (T2)). Lemma C.2 shows that Assumption (T2) implies L A ( t ) ⟶ t → ∞ 0 \mathcal{L}_{A}(t)\mathop{\longrightarrow}_{t\to\infty}0 L A ( t ) ⟶ t → ∞ 0 . Therefore, wherever we assume (T2) we assume w.l.o.g that t 0 t_{0} t 0 is large enough so that t 0 ≥ t A t_{0}\ge t_{A} t 0 ≥ t A and
L ( θ t ) ≤ L A ( t ) < 1 , ( t ≥ t 0 ) . (36) \mathcal{L}(\theta_{t})\le\mathcal{L}_{A}(t)<1,\qquad(t\ge t_{0}). \tag*{(36)} L ( θ t ) ≤ L A ( t ) < 1 , ( t ≥ t 0 ) . (36)
We also observe the following useful quantities on t > t 0 t>t_{0} t > t 0 :
U ( t ) = − log L ( θ t ) , U A ( t ) = − log L A ( t ) , (37) U(t)=-\log\mathcal{L}(\theta_{t}),\qquad U_{A}(t)=-\log\mathcal{L}_{A}(t), \tag*{(37)} U ( t ) = − log L ( θ t ) , U A ( t ) = − log L A ( t ) , (37)
G ( t ) = U 1 − 1 L L , G A ( t ) = U A 1 − 1 L L A , (38) G(t)=U^{1-\frac{1}{L}}\mathcal{L},\qquad G_{A}(t)=U_{A}^{1-\frac{1}{L}}\mathcal{L}_{A}, \tag*{(38)} G ( t ) = U 1 − L 1 L , G A ( t ) = U A 1 − L 1 L A , (38)
Q ( t ) = − log G ( t ) , Q A ( t ) = − log G A ( t ) , (39) Q(t)=-\log G(t),\qquad Q_{A}(t)=-\log G_{A}(t), \tag*{(39)} Q ( t ) = − log G ( t ) , Q A ( t ) = − log G A ( t ) , (39)
V ( t ) = ∫ t 0 t e c 2 s G 2 ( s ) d s , V A ( t ) = ∫ t 0 t e c 2 s G A 2 ( s ) d s , (40) V(t)=\int_{t_{0}}^{t}e^{c_{2}s}G^{2}(s)\,ds,\qquad V_{A}(t)=\int_{t_{0}}^{t}e^{c_{2}s}G_{A}^{2}(s)\,ds, \tag*{(40)} V ( t ) = ∫ t 0 t e c 2 s G 2 ( s ) d s , V A ( t ) = ∫ t 0 t e c 2 s G A 2 ( s ) d s , (40)
r A ( t ) = ( d d t log V A ) 1 / 2 . (41) r_{A}(t)=\left(\frac{d}{dt}\log V_{A}\right)^{1/2}. \tag*{(41)} r A ( t ) = ( d t d log V A ) 1/2 . (41)
Lemma C.1 (Corrected-loss and radial identities). For almost any t t t ,
H ˙ ( t ) = 2 c 1 ⟨ g t , m t v t ⟩ − ( 2 c 1 − c 2 2 ) H ( t ) − c 2 2 ∑ j = 1 p m j ( t ) 2 g j ( t ) 2 v j ( t ) 3 / 2 , (42) \dot{H}(t)=2c_1\left\langle g_t,\frac{m_t}{\sqrt{v_t}}\right\rangle-\left(2c_1-\frac{c_2}{2}\right)H(t)-\frac{c_2}{2}\sum_{j=1}^{p}\frac{m_j(t)^2g_j(t)^2}{v_j(t)^{3/2}}, \tag*{(42)} H ˙ ( t ) = 2 c 1 ⟨ g t , v t m t ⟩ − ( 2 c 1 − 2 c 2 ) H ( t ) − 2 c 2 j = 1 ∑ p v j ( t ) 3/2 m j ( t ) 2 g j ( t ) 2 , (42) and the following dissipation identity holds:
L ˙ A ( t ) = − μ ( t ) ( L A ( t ) − L ( θ t ) ) − c 2 η ^ ( t ) 4 c 1 ∑ j = 1 p m j ( t ) 2 g j ( t ) 2 v j ( t ) 3 / 2 ≤ − μ ( t ) ( L A ( t ) − L ( θ t ) ) . (43) \dot{\mathcal{L}}_A(t)=-\mu(t)\left(\mathcal{L}_A(t)-\mathcal{L}(\theta_t)\right)-\frac{c_2\widehat{\eta}(t)}{4c_1}\sum_{j=1}^{p}\frac{m_j(t)^2g_j(t)^2}{v_j(t)^{3/2}}\leq-\mu(t)\left(\mathcal{L}_A(t)-\mathcal{L}(\theta_t)\right). \tag*{(43)} L ˙ A ( t ) = − μ ( t ) ( L A ( t ) − L ( θ t ) ) − 4 c 1 c 2 η ( t ) j = 1 ∑ p v j ( t ) 3/2 m j ( t ) 2 g j ( t ) 2 ≤ − μ ( t ) ( L A ( t ) − L ( θ t ) ) . (43) In particular, L A \mathcal{L}_A L A is nonincreasing on every interval on which μ ≥ 0 \mu\geq0 μ ≥ 0 . In addition,
( d d t + c 1 ) ( − ⟨ θ t , m t ⟩ ) = − c 1 ⟨ θ t , g t ⟩ + η ^ ( t ) H ( t ) . (44) \left(\frac{d}{dt}+c_1\right)(-\langle\theta_t,m_t\rangle)=-c_1\langle\theta_t,g_t\rangle+\widehat{\eta}(t)H(t). \tag*{(44)} ( d t d + c 1 ) ( − ⟨ θ t , m t ⟩) = − c 1 ⟨ θ t , g t ⟩ + η ( t ) H ( t ) . (44)
Proof. Differentiate H = ∑ j m j 2 v j − 1 / 2 H=\sum_jm_j^2v_j^{-1/2} H = ∑ j m j 2 v j − 1/2 and substitute m ˙ j = c 1 ( g j − m j ) \dot{m}_j=c_1(g_j-m_j) m ˙ j = c 1 ( g j − m j ) and v ˙ j = c 2 ( g j 2 − v j ) \dot{v}_j=c_2(g_j^2-v_j) v ˙ j = c 2 ( g j 2 − v j ) to obtain (42) . By the chain rule,
L ˙ ( θ t ) = ⟨ g t , d θ t d t ⟩ = − η ^ ( t ) ⟨ g t , m t v t ⟩ . \dot{\mathcal{L}}(\theta_t)=\left\langle g_t,\frac{d\theta_t}{dt}\right\rangle=-\widehat{\eta}(t)\left\langle g_t,\frac{m_t}{\sqrt{v_t}}\right\rangle. L ˙ ( θ t ) = ⟨ g t , d t d θ t ⟩ = − η ( t ) ⟨ g t , v t m t ⟩ .
When L A \mathcal{L}_A L A is differentiated, this term cancels the first term in (42) . Using η ^ H / ( 2 c 1 ) = L A − L \widehat{\eta}H/(2c_1)=\mathcal{L}_A-\mathcal{L} η H / ( 2 c 1 ) = L A − L gives (43) . Finally,
d d t ( − ⟨ θ t , m t ⟩ ) = − ⟨ θ ˙ t , m t ⟩ − ⟨ θ t , m ˙ t ⟩ = η ^ H − c 1 ⟨ θ t , g t ⟩ + c 1 ⟨ θ t , m t ⟩ , \frac{d}{dt}(-\langle\theta_t,m_t\rangle)=-\langle\dot{\theta}_t,m_t\rangle-\langle\theta_t,\dot{m}_t\rangle=\widehat{\eta}H-c_1\langle\theta_t,g_t\rangle+c_1\langle\theta_t,m_t\rangle, d t d ( − ⟨ θ t , m t ⟩) = − ⟨ θ ˙ t , m t ⟩ − ⟨ θ t , m ˙ t ⟩ = η H − c 1 ⟨ θ t , g t ⟩ + c 1 ⟨ θ t , m t ⟩ ,
which is (44) .
Lemma C.2 (Equivalence of ordinary and corrected loss decay). It holds that
L ( θ t ) ⟶ t → ∞ 0 ⟺ L A ( t ) ⟶ t → ∞ 0. \mathcal{L}(\theta_t)\mathop{\longrightarrow}_{t\to\infty}0\quad\Longleftrightarrow\quad\mathcal{L}_A(t)\mathop{\longrightarrow}_{t\to\infty}0. L ( θ t ) ⟶ t → ∞ 0 ⟺ L A ( t ) ⟶ t → ∞ 0.
Proof. The reverse implication follows immediately from 0 < L ≤ L A 0<\mathcal{L}\leq\mathcal{L}_A 0 < L ≤ L A . Conversely, suppose that L ( t ) → 0 \mathcal{L}(t)\to0 L ( t ) → 0 . Lemma C.1 makes L A \mathcal{L}_A L A nonincreasing on a tail, so it has a limit A ∞ ≥ 0 A_\infty\geq0 A ∞ ≥ 0 . If A ∞ > 0 A_\infty>0 A ∞ > 0 , then
w ( t ) : = L A ( t ) − L ( t ) L A ( t ) ⟶ 1. w(t):=\frac{\mathcal{L}_A(t)-\mathcal{L}(t)}{\mathcal{L}_A(t)}\longrightarrow1. w ( t ) := L A ( t ) L A ( t ) − L ( t ) ⟶ 1.
For U A ( t ) : = − log L A ( t ) U_A(t):=-\log\mathcal{L}_A(t) U A ( t ) := − log L A ( t ) , (43) gives for t ≥ t A t\geq t_A t ≥ t A ,
U ˙ A ( t ) ≥ μ 0 w ( t ) . \dot{U}_A(t)\geq\mu_0w(t). U ˙ A ( t ) ≥ μ 0 w ( t ) .
Thus U ˙ A ( t ) ≥ μ 0 / 2 \dot{U}_A(t)\geq\mu_0/2 U ˙ A ( t ) ≥ μ 0 /2 eventually, contradicting U A ( t ) → − log A ∞ < ∞ U_A(t)\to-\log A_\infty<\infty U A ( t ) → − log A ∞ < ∞ . Hence A ∞ = 0 A_\infty=0 A ∞ = 0 .
The next lemma translates loss decay into radial and gradient scales.
Lemma C.3 (Asymptotic Rates). Under Assumptions (T1), (T2), the following hold:
U ( t ) ≍ q min ( t ) ≍ ∥ θ t ∥ 2 L , (45) U(t)\asymp q_{\min}(t)\asymp\|\theta_t\|_2^L, \tag*{(45)} U ( t ) ≍ q m i n ( t ) ≍ ∥ θ t ∥ 2 L , (45) ∥ g t ∥ 2 ≍ G ( t ) ≍ ∥ θ t ∥ L − 1 L ( θ t ) , (46) \|g_t\|_2\asymp G(t)\asymp\|\theta_t\|^{L-1}\mathcal{L}(\theta_t), \tag*{(46)} ∥ g t ∥ 2 ≍ G ( t ) ≍ ∥ θ t ∥ L − 1 L ( θ t ) , (46) L ( U ( t ) − φ ( 0 ) ) L ( θ t ) ≤ − ⟨ θ t , g t ⟩ ≲ U ( t ) L ( θ t ) . (47) L\bigl(U(t)-\varphi(0)\bigr)\mathcal{L}(\theta_t)\leq-\langle\theta_t,g_t\rangle\lesssim U(t)\mathcal{L}(\theta_t). \tag*{(47)} L ( U ( t ) − φ ( 0 ) ) L ( θ t ) ≤ − ⟨ θ t , g t ⟩ ≲ U ( t ) L ( θ t ) . (47)
Proof. Since ℓ \ell ℓ is decreasing,
e − φ ( q min ( t ) ) ≤ L ( θ t ) ≤ m e − φ ( q min ( t ) ) . e^{-\varphi(q_{\min}(t))}\leq\mathcal{L}(\theta_t)\leq me^{-\varphi(q_{\min}(t))}. e − φ ( q m i n ( t )) ≤ L ( θ t ) ≤ m e − φ ( q m i n ( t )) .
Loss decay therefore gives q min ( t ) → ∞ q_{\min}(t)\to\infty q m i n ( t ) → ∞ . Convexity, positivity, and boundedness of φ ′ \varphi' φ ′ imply φ ( q min ) ≍ q min \varphi(q_{\min})\asymp q_{\min} φ ( q m i n ) ≍ q m i n on [ t 0 , ∞ ) [t_0,\infty) [ t 0 , ∞ ) , since
0 < φ ′ ( q min ( t 0 ) ) ≤ φ ( q min ( t ) ) − φ ( q min ( t 0 ) ) q min ( t ) − q min ( t 0 ) ≤ sup R φ ′ . 0<\varphi'(q_{\min}(t_0))\leq\frac{\varphi(q_{\min}(t))-\varphi(q_{\min}(t_0))}{q_{\min}(t)-q_{\min}(t_0)}\leq\sup_{\mathbb{R}}\varphi'. 0 < φ ′ ( q m i n ( t 0 )) ≤ q m i n ( t ) − q m i n ( t 0 ) φ ( q m i n ( t )) − φ ( q m i n ( t 0 )) ≤ R sup φ ′ .
Thus − log L ≲ q min -\log\mathcal{L} \lesssim q_{\min} − log L ≲ q m i n . Denote i ∗ = i ∗ ( t ) ∈ [ m ] i_* = i_*(t) \in[m] i ∗ = i ∗ ( t ) ∈ [ m ] for which q min ( t ) = q i ∗ ( t ) q_{\min}(t) = q_{i_*}(t) q m i n ( t ) = q i ∗ ( t ) . The assumed lower bound on γ 2 ( θ t ) \gamma_2(\theta_t) γ 2 ( θ t ) , homogeneity and continuity of f f f on the unit ball give, for all i ∈ [ m ] i \in[m] i ∈ [ m ] ,
γ 0 ∥ θ t ∥ 2 L ≤ q min ( t ) = y i ∗ f ( x i ∗ ; θ ) ≤ y i f ( x i ; θ ) ≤ ∥ θ t ∥ 2 L sup ∥ u ∥ = 1 y i f ( x i ; u ) , (48) \gamma_0 \lVert\theta_t\rVert_2^L \le q_{\min}(t) = y_{i_*}f(x_{i_*};\theta) \le y_i f(x_i;\theta) \le\lVert\theta_t\rVert_2^L \sup_{\lVert u\rVert=1} y_i f(x_i;u), \tag*{(48)} γ 0 ∥ θ t ∥ 2 L ≤ q m i n ( t ) = y i ∗ f ( x i ∗ ; θ ) ≤ y i f ( x i ; θ ) ≤ ∥ θ t ∥ 2 L ∥ u ∥ = 1 sup y i f ( x i ; u ) , (48)
so q i ≍ q min ≍ ∥ θ t ∥ 2 L q_i \asymp q_{\min} \asymp\lVert\theta_t\rVert_2^L q i ≍ q m i n ≍ ∥ θ t ∥ 2 L , proving (45) .
L − 1 L-1 L − 1 -homogeneity of h i ( t ) ∈ ∂ q i h_i(t) \in\partial q_i h i ( t ) ∈ ∂ q i , local Lipschitzness of f f f (implying h i h_i h i are bounded on the unit sphere) and the Euler identity give for some C > 0 C > 0 C > 0
∥ h i ( t ) ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 , ⟨ h i ( t ) , θ t ⟩ = L q i ( t ) . \lVert h_i(t)\rVert_2 \le C\lVert\theta_t\rVert_2^{L-1}, \qquad\langle h_i(t),\theta_t\rangle= Lq_i(t). ∥ h i ( t ) ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 , ⟨ h i ( t ) , θ t ⟩ = L q i ( t ) .
For some C > 0 C > 0 C > 0 , the first relation and (22) give
∥ g t ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 L ( θ t ) . \lVert g_t\rVert_2 \le C\lVert\theta_t\rVert_2^{L-1}\mathcal{L}(\theta_t). ∥ g t ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 L ( θ t ) .
Convexity of φ \varphi φ gives
φ ( 0 ) ≥ φ ( q i ( t ) ) − q i ( t ) φ ′ ( q i ( t ) ) . \varphi(0) \ge\varphi(q_i(t))-q_i(t)\varphi'(q_i(t)). φ ( 0 ) ≥ φ ( q i ( t )) − q i ( t ) φ ′ ( q i ( t )) .
Moreover, e − φ ( q i ( t ) ) ≤ L ( θ t ) = e − U ( t ) e^{-\varphi(q_i(t))}\le\mathcal{L}(\theta_t)=e^{-U(t)} e − φ ( q i ( t )) ≤ L ( θ t ) = e − U ( t ) , so φ ( q i ( t ) ) ≥ U ( t ) \varphi(q_i(t))\ge U(t) φ ( q i ( t )) ≥ U ( t ) . The Euler identity and Cauchy–Schwarz therefore give
L ( U ( t ) − φ ( 0 ) ) L ( θ t ) ≤ L ∑ i = 1 m ( φ ( q i ( t ) ) − φ ( 0 ) ) e − φ ( q i ( t ) ) ≤ L ∑ i = 1 m q i ( t ) φ ′ ( q i ( t ) ) e − φ ( q i ( t ) ) = − ⟨ θ t , g t ⟩ ≤ ∥ θ t ∥ 2 ∥ g t ∥ 2 . \begin{aligned}
L(U(t)-\varphi(0))\mathcal{L}(\theta_t) \le L\sum_{i=1}^{m}(\varphi(q_i(t))-\varphi(0))e^{-\varphi(q_i(t))} \\
&\le L\sum_{i=1}^{m}q_i(t)\varphi'(q_i(t))e^{-\varphi(q_i(t))} \\
&= -\langle\theta_t,g_t\rangle\le\lVert\theta_t\rVert_2\lVert g_t\rVert_2.
\end{aligned} L ( U ( t ) − φ ( 0 )) L ( θ t ) ≤ L i = 1 ∑ m ( φ ( q i ( t )) − φ ( 0 )) e − φ ( q i ( t )) ≤ L i = 1 ∑ m q i ( t ) φ ′ ( q i ( t )) e − φ ( q i ( t )) = − ⟨ θ t , g t ⟩ ≤ ∥ θ t ∥ 2 ∥ g t ∥ 2 .
Combining this with ∥ g t ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 L ( θ t ) \lVert g_t\rVert_2\le C\lVert\theta_t\rVert_2^{L-1}\mathcal{L}(\theta_t) ∥ g t ∥ 2 ≤ C ∥ θ t ∥ 2 L − 1 L ( θ t ) , U ( t ) − φ ( 0 ) ≍ U ( t ) U(t)-\varphi(0)\asymp U(t) U ( t ) − φ ( 0 ) ≍ U ( t ) , and (45) proves both (46) and (47) . □
We show that G A G_A G A is an upper bound on G G G , but that their respective accumulated second moments V A V_A V A and V V V are asymptotically comparable.
Lemma C.4 (Accumulated gradient scales). Suppose Assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) hold. Then, for all sufficiently large t t t ,
G ( t ) ≤ G A ( t ) . G(t)\le G_A(t). G ( t ) ≤ G A ( t ) . Moreover,
V A ( t ) ≍ V ( t ) . (49) V_A(t)\asymp V(t). \tag*{(49)} V A ( t ) ≍ V ( t ) . (49)
Proof. Lemma C.2 gives U A ( t ) → ∞ U_A(t)\to\infty U A ( t ) → ∞ . Since U ≥ U A U\ge U_A U ≥ U A and the function u ↦ u 1 − 1 / L e − u u\mapsto u^{1-1/L}e^{-u} u ↦ u 1 − 1/ L e − u is decreasing for sufficiently large u u u , we have G ≤ G A G\le G_A G ≤ G A eventually.
Choose T T T sufficiently large, with V ( T ) , V A ( T ) > 0 V(T),V_A(T)>0 V ( T ) , V A ( T ) > 0 . The bound G ≤ G A G\le G_A G ≤ G A gives
V ( t ) ≤ V ( T ) + V A ( t ) − V A ( T ) ≲ V A ( t ) . V(t)\le V(T)+V_A(t)-V_A(T)\lesssim V_A(t). V ( t ) ≤ V ( T ) + V A ( t ) − V A ( T ) ≲ V A ( t ) .
For the reverse bound, since U A 1 − 1 / L L ≤ G ≤ G A U_A^{1-1/L}L\le G\le G_A U A 1 − 1/ L L ≤ G ≤ G A , μ 0 > c 2 \mu_0>c_2 μ 0 > c 2 , and U A → ∞ U_A\to\infty U A → ∞ , L ˙ A ≤ − μ 0 ( L A − L ) \dot{\mathcal{L}}_A\le-\mu_0(\mathcal{L}_A-\mathcal{L}) L ˙ A ≤ − μ 0 ( L A − L ) by (43) gives, after increasing T T T if necessary,
G ˙ A = ( 1 − 1 − 1 / L U A ) U A 1 − 1 / L L ˙ A ≤ − μ 0 ( 1 − 1 − 1 / L U A ) ( G A − U A 1 − 1 / L L ) ≤ − c 2 ( G A − G ) . \begin{aligned}
\dot{G}_A=\left(1-\frac{1-1/L}{U_A}\right)U_A^{1-1/L}\dot{\mathcal{L}}_A \\
&\le-\mu_0\left(1-\frac{1-1/L}{U_A}\right)(G_A-U_A^{1-1/L}\mathcal{L}) \\
&\le-c_2(G_A-G).
\end{aligned} G ˙ A = ( 1 − U A 1 − 1/ L ) U A 1 − 1/ L L ˙ A ≤ − μ 0 ( 1 − U A 1 − 1/ L ) ( G A − U A 1 − 1/ L L ) ≤ − c 2 ( G A − G ) .
Set g T ( u ) : = G ( T + u ) ≥ 0 g_T(u):=G(T+u)\ge0 g T ( u ) := G ( T + u ) ≥ 0 for u ≥ 0 u\ge0 u ≥ 0 . Integrating the inequality from T T T to T + u T+u T + u gives
G A ( T + u ) ≤ e − c 2 u G A ( T ) + EMA c 2 [ g T ] ( u ) . G_A(T+u)\le e^{-c_2u}G_A(T)+\operatorname{EMA}_{c_2}[g_T](u). G A ( T + u ) ≤ e − c 2 u G A ( T ) + EMA c 2 [ g T ] ( u ) .
Lemma B.7 with p = 2 p=2 p = 2 , λ = c 2 / 2 \lambda=c_{2}/2 λ = c 2 /2 , and c = c 2 c=c_{2} c = c 2 , together with the triangle inequality in L 2 L^{2} L 2 , now yields
V A ( t ) − V A ( T ) = ( ∫ 0 t − T e c 2 ( T + u ) G A 2 ( T + u ) d u ) 1 / 2 ≤ e c 2 T / 2 G A ( T ) ( ∫ 0 t − T e − c 2 u d u ) 1 / 2 + e c 2 T / 2 ( ∫ 0 t − T ∣ e c 2 u / 2 EMA c 2 [ g T ] ( u ) ∣ 2 d u ) 1 / 2 ≤ e c 2 T / 2 G A ( T ) c 2 + 2 e c 2 T / 2 ( ∫ 0 t − T e c 2 u g T ( u ) 2 d u ) 1 / 2 = e c 2 T / 2 G A ( T ) c 2 + 2 V ( t ) − V ( T ) . \begin{aligned}
\sqrt{V_{A}(t)-V_{A}(T)}=\left(\int_{0}^{t-T}e^{c_{2}(T+u)}G_{A}^{2}(T+u)\,\mathrm{d}u\right)^{1/2} \\
&\le e^{c_{2}T/2}G_{A}(T)\left(\int_{0}^{t-T}e^{-c_{2}u}\,\mathrm{d}u\right)^{1/2}+e^{c_{2}T/2}\left(\int_{0}^{t-T}\left|e^{c_{2}u/2}\operatorname{EMA}_{c_{2}}[g_{T}](u)\right|^{2}\,\mathrm{d}u\right)^{1/2} \\
&\le\frac{e^{c_{2}T/2}G_{A}(T)}{\sqrt{c_{2}}}+2e^{c_{2}T/2}\left(\int_{0}^{t-T}e^{c_{2}u}g_{T}(u)^{2}\,\mathrm{d}u\right)^{1/2} \\
&= \frac{e^{c_{2}T/2}G_{A}(T)}{\sqrt{c_{2}}}+2\sqrt{V(t)-V(T)}.
\end{aligned} V A ( t ) − V A ( T ) = ( ∫ 0 t − T e c 2 ( T + u ) G A 2 ( T + u ) d u ) 1/2 ≤ e c 2 T /2 G A ( T ) ( ∫ 0 t − T e − c 2 u d u ) 1/2 + e c 2 T /2 ( ∫ 0 t − T e c 2 u /2 EMA c 2 [ g T ] ( u ) 2 d u ) 1/2 ≤ c 2 e c 2 T /2 G A ( T ) + 2 e c 2 T /2 ( ∫ 0 t − T e c 2 u g T ( u ) 2 d u ) 1/2 = c 2 e c 2 T /2 G A ( T ) + 2 V ( t ) − V ( T ) .
Consequently, ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^{2}\le2a^{2}+2b^{2} ( a + b ) 2 ≤ 2 a 2 + 2 b 2 gives
V A ( t ) ≤ V A ( T ) + 2 e c 2 T G A ( T ) 2 c 2 + 8 ( V ( t ) − V ( T ) ) ≲ V ( t ) . V_{A}(t)\le V_{A}(T)+\frac{2e^{c_{2}T}G_{A}(T)^{2}}{c_{2}}+8\bigl(V(t)-V(T)\bigr)\lesssim V(t). V A ( t ) ≤ V A ( T ) + c 2 2 e c 2 T G A ( T ) 2 + 8 ( V ( t ) − V ( T ) ) ≲ V ( t ) .
Lemma C.5 (Corrected Loss Asymptotics). Assume (T1), (T2). There exists C > 0 C>0 C > 0 so that for all large enough t t t ,
∥ g t ∥ 2 ≤ C e − Q A ( t ) , v t [ j ] ≤ C e − c 2 t V A ( t ) , r A ( t ) 2 = V ˙ A ( t ) V A ( t ) , (50) \lVert g_{t}\rVert_{2}\le Ce^{-Q_{A}(t)},\qquad v_{t}[j]\le Ce^{-c_{2}t}V_{A}(t),\qquad r_{A}(t)^{2}=\frac{\dot{V}_{A}(t)}{V_{A}(t)}, \tag*{(50)} ∥ g t ∥ 2 ≤ C e − Q A ( t ) , v t [ j ] ≤ C e − c 2 t V A ( t ) , r A ( t ) 2 = V A ( t ) V ˙ A ( t ) , (50) U ˙ A ≥ μ 0 L A − L L A = μ 0 η ^ H 2 c 1 L A ≥ 0 , (51) \dot{U}_{A}\ge\mu_{0}\frac{\mathcal{L}_{A}-\mathcal{L}}{\mathcal{L}_{A}}=\frac{\mu_{0}\widehat{\eta}H}{2c_{1}\mathcal{L}_{A}}\ge0, \tag*{(51)} U ˙ A ≥ μ 0 L A L A − L = 2 c 1 L A μ 0 η H ≥ 0 , (51) Q ˙ A = ( 1 − 1 − 1 / L U A ) U ˙ A ≥ 0 , Q A = U A − ( 1 − 1 L ) log U A ∼ U A → t → ∞ ∞ . (52) \dot{Q}_{A}=\left(1-\frac{1-1/L}{U_{A}}\right)\dot{U}_{A}\ge0,\qquad Q_{A}=U_{A}-\left(1-\frac{1}{L}\right)\log U_{A}\sim U_{A}\xrightarrow{t\to\infty}\infty. \tag*{(52)} Q ˙ A = ( 1 − U A 1 − 1/ L ) U ˙ A ≥ 0 , Q A = U A − ( 1 − L 1 ) log U A ∼ U A t → ∞ ∞. (52)
Proof. Lemma C.2 gives U A ( t ) → ∞ U_{A}(t)\to\infty U A ( t ) → ∞ . Rearranging equation (43) gives (51) , and using the definition of Q A Q_{A} Q A gives (52) . Thus Q A Q_{A} Q A is eventually nondecreasing and tends to infinity. By Lemmas C.3, C.4, for some C > 0 C>0 C > 0
∥ g t ∥ 2 ≤ C G ( t ) ≤ C e − Q A ( t ) . \lVert g_{t}\rVert_{2}\le CG(t)\le Ce^{-Q_{A}(t)}. ∥ g t ∥ 2 ≤ C G ( t ) ≤ C e − Q A ( t ) .
The second-moment formula then gives the asserted bound on v t [ j ] v_{t}[j] v t [ j ] . The identity for r A r_{A} r A follows directly from the definition.
Next, we show an additional desirable property of L A \mathcal{L}_{A} L A , namely that a similar relation to (47) holds between − ⟨ θ t , m t ⟩ -\langle\theta_{t},m_{t}\rangle − ⟨ θ t , m t ⟩ and U A L A U_{A}\mathcal{L}_{A} U A L A .
Lemma C.6 (Radial Momentum). Assume (T1), (T2). There exists a > 0 a>0 a > 0 , T m > 0 T_{m}>0 T m > 0 with
− ⟨ θ t , m t ⟩ ≥ a U A ( t ) L A ( t ) ( t > T m ) . (53) -\langle\theta_{t},m_{t}\rangle\ge aU_{A}(t)\mathcal{L}_{A}(t)\qquad(t>T_{m}). \tag*{(53)} − ⟨ θ t , m t ⟩ ≥ a U A ( t ) L A ( t ) ( t > T m ) . (53)
Proof. By Lemma C.3 and using U ≥ U A U\ge U_{A} U ≥ U A , choose a > 0 a>0 a > 0 and T m T_{m} T m so that − c 1 ⟨ θ t , g t ⟩ ≥ a μ 0 U A L -c_{1}\langle\theta_{t},g_{t}\rangle\ge a\mu_{0}U_{A}\mathcal{L} − c 1 ⟨ θ t , g t ⟩ ≥ a μ 0 U A L . Since μ 0 > c 1 \mu_{0}>c_{1} μ 0 > c 1 and U A → ∞ U_{A}\to\infty U A → ∞ , we may increase the tail time so that U A ≥ 2 μ 0 μ 0 − c 1 > 1 U_{A}\ge\frac{2\mu_{0}}{\mu_{0}-c_{1}}>1 U A ≥ μ 0 − c 1 2 μ 0 > 1 . Using ( U A L A ) ′ = − ( U A − 1 ) L A U ˙ A (U_{A}\mathcal{L}_{A})'=-(U_{A}-1)\mathcal{L}_{A}\dot{U}_{A} ( U A L A ) ′ = − ( U A − 1 ) L A U ˙ A and the radial identity (44) , we obtain, almost everywhere on this tail,
( d d t + c 1 ) ( − ⟨ θ t , m t ⟩ − a U A L A ) = − c 1 ⟨ θ t , g t ⟩ + η ^ H + a ( U A − 1 ) L A U ˙ A − a c 1 U A L A ≥ a μ 0 U A L + a μ 0 ( U A − 1 ) ( L A − L ) − a c 1 U A L A = a ( [ μ 0 ( U A − 1 ) − c 1 U A ] L A + μ 0 [ U A − ( U A − 1 ) ] L ) = a ( [ ( μ 0 − c 1 ) U A − μ 0 ] L A + μ 0 L ) ≥ a 2 ( μ 0 − c 1 ) U A L A . \begin{aligned}
\left(\frac{\mathrm{d}}{\mathrm{d}t}+c_{1}\right)\bigl(-\langle\theta_{t},m_{t}\rangle-aU_{A}\mathcal{L}_{A}\bigr)
=-c_{1}\langle\theta_{t},g_{t}\rangle+\widehat{\eta}H+a(U_{A}-1)\mathcal{L}_{A}\dot{U}_{A}-ac_{1}U_{A}\mathcal{L}_{A} \\
&\ge a\mu_{0}U_{A}\mathcal{L}+a\mu_{0}(U_{A}-1)(\mathcal{L}_{A}-\mathcal{L})-ac_{1}U_{A}\mathcal{L}_{A} \\
&=a\bigl([\mu_{0}(U_{A}-1)-c_{1}U_{A}]\mathcal{L}_{A}+\mu_{0}[U_{A}-(U_{A}-1)]\mathcal{L}\bigr) \\
&=a\bigl([(\mu_{0}-c_{1})U_{A}-\mu_{0}]\mathcal{L}_{A}+\mu_{0}\mathcal{L}\bigr) \\
&\ge\frac{a}{2}(\mu_{0}-c_{1})U_{A}\mathcal{L}_{A}.
\end{aligned} ( d t d + c 1 ) ( − ⟨ θ t , m t ⟩ − a U A L A ) = − c 1 ⟨ θ t , g t ⟩ + η H + a ( U A − 1 ) L A U ˙ A − a c 1 U A L A ≥ a μ 0 U A L + a μ 0 ( U A − 1 ) ( L A − L ) − a c 1 U A L A = a ( [ μ 0 ( U A − 1 ) − c 1 U A ] L A + μ 0 [ U A − ( U A − 1 )] L ) = a ( [( μ 0 − c 1 ) U A − μ 0 ] L A + μ 0 L ) ≥ 2 a ( μ 0 − c 1 ) U A L A .
Where the first inequality uses the choice of a a a , discards the nonnegative term η H \eta H η H , and applies (51) in the form
L A U ˙ A ≥ μ 0 ( L A − L ) . \mathcal{L}_A\dot{U}_A \ge\mu_0(\mathcal{L}_A-\mathcal{L}). L A U ˙ A ≥ μ 0 ( L A − L ) .
The RHS has an infinite e c 1 t e^{c_1t} e c 1 t -weighted integral. Indeed, if ∫ T ∞ e c 1 t U A L A d t < ∞ \int_T^\infty e^{c_1t}U_A\mathcal{L}_A\,dt<\infty ∫ T ∞ e c 1 t U A L A d t < ∞ , then (50) , together with e − Q A ≤ U A L A e^{-Q_A}\le U_A\mathcal{L}_A e − Q A ≤ U A L A , gives ∫ T ∞ e c 1 t ∥ g t ∥ 2 2 d t < ∞ \int_T^\infty e^{c_1t}\lVert g_t\rVert_2^2\,dt<\infty ∫ T ∞ e c 1 t ∥ g t ∥ 2 2 d t < ∞ . The first-moment formula then implies ∥ m t ∥ 2 = O ( e − c 1 t ) \lVert m_t\rVert_2=O(e^{-c_1t}) ∥ m t ∥ 2 = O ( e − c 1 t ) . For each coordinate whose second moment is positive at some time, there exists a j > 0 a_j>0 a j > 0 such that v t [ j ] ≥ a j e − c 2 t v_t[j]\ge a_je^{-c_2t} v t [ j ] ≥ a j e − c 2 t on a common tail; all other coordinates have zero velocity. Hence there exists C > 0 C>0 C > 0 such that
∥ θ ˙ t ∥ 2 ≤ C ( 1 + t ) α e − ( c 1 − c 2 / 2 ) t , \lVert\dot{\theta}_t\rVert_2\le C(1+t)^\alpha e^{-(c_1-c_2/2)t}, ∥ θ ˙ t ∥ 2 ≤ C ( 1 + t ) α e − ( c 1 − c 2 /2 ) t ,
which is integrable, contradicting ∥ θ t ∥ 2 → ∞ \lVert\theta_t\rVert_2\to\infty ∥ θ t ∥ 2 → ∞ . Multiplying the preceding differential inequality by e c 1 t e^{c_1t} e c 1 t and integrating from T T T to t t t gives
e c 1 t ( − ⟨ θ t , m t ⟩ − a U A ( t ) L A ( t ) ) ≥ e c 1 T ( − ⟨ θ T , m T ⟩ − a U A ( T ) L A ( T ) ) + a 2 ( μ 0 − c 1 ) ∫ T t e c 1 s U A ( s ) L A ( s ) d s . e^{c_1t}\bigl(-\langle\theta_t,m_t\rangle-aU_A(t)\mathcal{L}_A(t)\bigr)
\ge e^{c_1T}\bigl(-\langle\theta_T,m_T\rangle-aU_A(T)\mathcal{L}_A(T)\bigr)+\frac{a}{2}(\mu_0-c_1)\int_T^t e^{c_1s}U_A(s)\mathcal{L}_A(s)\,ds. e c 1 t ( − ⟨ θ t , m t ⟩ − a U A ( t ) L A ( t ) ) ≥ e c 1 T ( − ⟨ θ T , m T ⟩ − a U A ( T ) L A ( T ) ) + 2 a ( μ 0 − c 1 ) ∫ T t e c 1 s U A ( s ) L A ( s ) d s .
The initial term is a fixed finite number, whereas the integral tends to infinity. Since μ 0 > c 1 \mu_0>c_1 μ 0 > c 1 , the right-hand side is therefore positive for all sufficiently large t t t . Dividing by e c 1 t > 0 e^{c_1t}>0 e c 1 t > 0 yields (53) .
The following lemma uses the estimates produced so far to reveal a negative feedback mechanism forcing r A r_A r A to be small and Q A Q_A Q A to be large: large r A r_A r A , corresponding to a momentary large “corrected gradient” G A G_A G A , would force growth of Q A Q_A Q A and hence suppress the growth of r A r_A r A .
Lemma C.7 (Lower-bound feedback). Suppose assumptions (T1), (T2) hold. Then, for all sufficiently large t t t , Q A Q_A Q A is nondecreasing, Q A ( t ) → ∞ Q_A(t)\to\infty Q A ( t ) → ∞ , and there are constants k Q A > 0 k_{Q_A}>0 k Q A > 0 and δ > c 2 / 2 \delta>c_2/2 δ > c 2 /2 such that
d d t log r A ( t ) = c 2 2 − Q ˙ A ( t ) − 1 2 r A ( t ) 2 , (54) \frac{d}{dt}\log r_A(t)=\frac{c_2}{2}-\dot{Q}_A(t)-\frac{1}{2}r_A(t)^2, \tag*{(54)} d t d log r A ( t ) = 2 c 2 − Q ˙ A ( t ) − 2 1 r A ( t ) 2 , (54) Q ˙ A ( t ) ≥ min { δ , k Q A ( 1 + t ) α r A ( t ) Q A ( t ) 1 − 1 L } . (55) \dot{Q}_A(t)\ge\min\left\{\delta,k_{Q_A}(1+t)^\alpha r_A(t)Q_A(t)^{1-\frac{1}{L}}\right\}. \tag*{(55)} Q ˙ A ( t ) ≥ min { δ , k Q A ( 1 + t ) α r A ( t ) Q A ( t ) 1 − L 1 } . (55) The differential relations hold almost everywhere.
Proof. (54) follows from differentiating the definition of r A r_A r A . We proceed to prove (55) . First consider times when L < L A / 2 \mathcal{L}<\mathcal{L}_A/2 L < L A /2 . For such times, (51) gives U ˙ A ≥ μ 0 / 2 \dot{U}_A\ge\mu_0/2 U ˙ A ≥ μ 0 /2 . Since μ 0 > c 1 ≥ c 2 \mu_0>c_1\ge c_2 μ 0 > c 1 ≥ c 2 and using (52) , increasing the tail time gives Q ˙ A ≥ δ \dot{Q}_A\ge\delta Q ˙ A ≥ δ for some δ > c 2 / 2 \delta>c_2/2 δ > c 2 /2 .
At times when L ≥ L A / 2 \mathcal{L}\ge\mathcal{L}_A/2 L ≥ L A /2 , we have U = U A + O ( 1 ) U=U_A+O(1) U = U A + O ( 1 ) and ∥ θ t ∥ 2 2 ≍ U A \lVert\theta_t\rVert_2^2\asymp U_A ∥ θ t ∥ 2 2 ≍ U A . Lemma C.6 and Cauchy–Schwarz imply that there exists C > 0 C>0 C > 0 such that
a 2 T A 2 L A 2 ≤ ⟨ θ t , m t ⟩ 2 = ⟨ θ t v t , m t v t ⟩ 2 ≤ ( ∑ j = 1 p θ t [ j ] 2 v t [ j ] ) H ≤ C U A 2 e − c 2 t / 2 V A H . a^2T_A^2\mathcal{L}_A^2\le\langle\theta_t,m_t\rangle^2=\left\langle\theta_t\sqrt{v_t},\frac{m_t}{\sqrt{v_t}}\right\rangle^2\le\left(\sum_{j=1}^p\theta_t[j]^2\sqrt{v_t[j]}\right)H\le CU_A^2e^{-c_2t/2}\sqrt{V_A}\,H. a 2 T A 2 L A 2 ≤ ⟨ θ t , m t ⟩ 2 = ⟨ θ t v t , v t m t ⟩ 2 ≤ ( j = 1 ∑ p θ t [ j ] 2 v t [ j ] ) H ≤ C U A 2 e − c 2 t /2 V A H . By the definitions of r A r_A r A , V A V_A V A , and G A G_A G A ,
r A = V ˙ A V A = e c 2 t / 2 G A V A = U A 1 − 1 L e c 2 t / 2 L A V A . r_A=\sqrt{\frac{\dot{V}_A}{V_A}}=\frac{e^{c_2t/2}G_A}{\sqrt{V_A}}=U_A^{1-\frac{1}{L}}\frac{e^{c_2t/2}\mathcal{L}_A}{\sqrt{V_A}}. r A = V A V ˙ A = V A e c 2 t /2 G A = U A 1 − L 1 V A e c 2 t /2 L A . Substituting this identity into the preceding bound yields
H L A ≥ a 2 C U A 1 − 1 L r A . \frac{H}{\mathcal{L}_A}\ge\frac{a^2}{C}U_A^{1-\frac{1}{L}}r_A. L A H ≥ C a 2 U A 1 − L 1 r A . Since U A → ∞ U_A\to\infty U A → ∞ , we may increase the tail time so that 1 − 1 − 1 / L U A ≥ 1 2 1-\frac{1-1/L}{U_A}\ge\frac12 1 − U A 1 − 1/ L ≥ 2 1 . Together with U ˙ A ≥ 0 \dot{U}_A\ge0 U ˙ A ≥ 0 and (51) , this shows that there exists k Q A > 0 k_{Q_A}>0 k Q A > 0 such that, almost everywhere at the times under consideration,
Q ˙ A = ( 1 − 1 − 1 / L U A ) U ˙ A ≥ 1 2 U ˙ A ≥ μ 0 4 c 1 η H L A ≥ a 2 μ 0 4 c 1 C η U A 1 − 1 L r A \dot{Q}_A=\left(1-\frac{1-1/L}{U_A}\right)\dot{U}_A\ge\frac12\dot{U}_A\ge\frac{\mu_0}{4c_1}\eta\frac{H}{\mathcal{L}_A}\ge\frac{a^2\mu_0}{4c_1C}\eta U_A^{1-\frac{1}{L}}r_A Q ˙ A = ( 1 − U A 1 − 1/ L ) U ˙ A ≥ 2 1 U ˙ A ≥ 4 c 1 μ 0 η L A H ≥ 4 c 1 C a 2 μ 0 η U A 1 − L 1 r A ≥ k Q A ( 1 + t ) α r A Q A 1 − 1 L , \ge k_{Q_A}(1+t)^\alpha r_AQ_A^{1-\frac{1}{L}}, ≥ k Q A ( 1 + t ) α r A Q A 1 − L 1 ,
where the last inequality uses η ~ ≍ ( 1 + t ) α \tilde{\eta}\asymp(1+t)^\alpha η ~ ≍ ( 1 + t ) α and Q A ≍ U A Q_A\asymp U_A Q A ≍ U A , with the fixed positive constants absorbed into k Q A > 0 k_{Q_A}>0 k Q A > 0 .
Lemma C.8 (Upper-bound feedback). Suppose assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) hold, and set
s A ( t ) : = ∥ m t ∥ ∥ v t ∥ 2 , ν : = 1 − 1 L , ζ : = α + ν , λ : = c 1 − c 2 2 > 0 , s_A(t):=\frac{\lVert\mathbf{m}_t\rVert}{\lVert\sqrt{\mathbf{v}_t}\rVert_2},\qquad\nu:=1-\frac{1}{L},\qquad\zeta:=\alpha+\nu,\qquad\lambda:=c_1-\frac{c_2}{2}>0, s A ( t ) := ∥ v t ∥ 2 ∥ m t ∥ , ν := 1 − L 1 , ζ := α + ν , λ := c 1 − 2 c 2 > 0 , where coordinates with zero second moment contribute zero. On a sufficiently late tail, s A s_A s A is locally absolutely continuous, s A = O ( 1 ) s_A=O(1) s A = O ( 1 ) , and there exists C > 0 C>0 C > 0 with
s ˙ A ≤ − λ s A + C e c 2 t / 2 ∥ g t ∥ 2 ≤ − λ s A + C e c 2 t / 2 − Q , (56) \dot{s}_A\leq-\lambda s_A+Ce^{c_2t/2}\lVert\mathbf{g}_t\rVert_2\leq-\lambda s_A+Ce^{c_2t/2-Q}, \tag*{(56)} s ˙ A ≤ − λ s A + C e c 2 t /2 ∥ g t ∥ 2 ≤ − λ s A + C e c 2 t /2 − Q , (56) ∣ d d t Q 1 / L ∣ ≤ C η s A . (57) \left\lvert\frac{d}{dt}Q^{1/L}\right\rvert\leq C\eta s_A. \tag*{(57)} d t d Q 1/ L ≤ C η s A . (57) The differential inequalities hold almost everywhere.
Proof. Coordinates with identically zero second moment have zero velocity and can be omitted. For each remaining coordinate, the second-moment formula gives v j ( t ) ≥ a j e − c 2 t v_j(t)\geq a_j e^{-c_2t} v j ( t ) ≥ a j e − c 2 t eventually, with a j > 0 a_j>0 a j > 0 . Since there are finitely many coordinates, these bounds hold on a common tail. The coordinate ratios, and hence s A s_A s A , are locally absolutely continuous on that tail. Lemma B.2 and the bounded bias-correction factors imply that there exists C > 0 C>0 C > 0 such that s A ≤ C s_A\leq C s A ≤ C . The moment equations and Cauchy–Schwarz give
1 2 d d t s A 2 = c 1 ∑ j m j g j v j − λ s A 2 − c 2 2 ∑ j m j 2 g j 2 v j 2 ≤ c 1 s A ∥ g t v t ∥ 2 − λ s A 2 . \begin{aligned}
\frac{1}{2}\frac{d}{dt}s_A^2=c_1\sum_j\frac{m_jg_j}{v_j}-\lambda s_A^2-\frac{c_2}{2}\sum_j\frac{m_j^2g_j^2}{v_j^2} \\
&\leq c_1s_A\left\lVert\frac{\mathbf{g}_t}{\sqrt{\mathbf{v}_t}}\right\rVert_2-\lambda s_A^2.
\end{aligned} 2 1 d t d s A 2 = c 1 j ∑ v j m j g j − λ s A 2 − 2 c 2 j ∑ v j 2 m j 2 g j 2 ≤ c 1 s A v t g t 2 − λ s A 2 .
Dividing by s A s_A s A where it is positive, and using s ˙ A = 0 \dot{s}_A=0 s ˙ A = 0 almost everywhere on { s A = 0 } \{s_A=0\} { s A = 0 } , gives the first inequality in (56) . The second follows from Lemma C.3 and the identity G = U ν L = e − Q G=U^\nu\mathcal{L}=e^{-Q} G = U ν L = e − Q . For some C > 0 C>0 C > 0 , the loss chain rule and the same gradient-scale estimate give
∣ U ˙ ∣ = η ⟨ g t , m t / v t ⟩ L ( θ t ) ≤ C η U ν s A . (58) |\dot{U}|=\eta\frac{\left\langle\mathbf{g}_t,\mathbf{m}_t/\sqrt{\mathbf{v}_t}\right\rangle}{\mathcal{L}(\theta_t)}\leq C\eta U^\nu s_A. \tag*{(58)} ∣ U ˙ ∣ = η L ( θ t ) ⟨ g t , m t / v t ⟩ ≤ C η U ν s A . (58)
Since Q = U − ν log U ∼ U > 0 Q=U-\nu\log U\sim U>0 Q = U − ν log U ∼ U > 0 and U → ∞ U\to\infty U → ∞ , the chain rule gives, on a sufficiently late tail,
∣ d d t Q 1 / L ∣ = Q − ν L ( 1 − ν U ) ∣ U ˙ ∣ ≤ C η ( U Q ) ν s A ≤ C η s A . \left\lvert\frac{d}{dt}Q^{1/L}\right\rvert=\frac{Q^{-\nu}}{L}\left(1-\frac{\nu}{U}\right)|\dot{U}|\leq C\eta\left(\frac{U}{Q}\right)^\nu s_A\leq C\eta s_A. d t d Q 1/ L = L Q − ν ( 1 − U ν ) ∣ U ˙ ∣ ≤ C η ( Q U ) ν s A ≤ C η s A .
This proves (57) .
Lemma C.9 (Coarse Decay Bounds). Assume ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) . Then r A r_A r A is bounded on a tail. If ζ ≥ 0 \zeta\geq0 ζ ≥ 0 , then
Q ( t ) ≥ Q A ≳ t , r A ( t ) ≲ t − ζ . (59) Q(t)\geq Q_A\gtrsim t,\qquad r_A(t)\lesssim t^{-\zeta}. \tag*{(59)} Q ( t ) ≥ Q A ≳ t , r A ( t ) ≲ t − ζ . (59)
Proof. Write ν = 1 − 1 / L \nu=1-1/L ν = 1 − 1/ L , and observe a tail where preceding lemmas hold. Since Q ˙ A ≥ 0 \dot{Q}_A\geq0 Q ˙ A ≥ 0 , (54) gives
d log r A d t ≤ c 2 2 − 1 2 r A 2 , \frac{d\log r_A}{dt}\leq\frac{c_2}{2}-\frac{1}{2}r_A^2, d t d log r A ≤ 2 c 2 − 2 1 r A 2 ,
so r A r_A r A is bounded by a simple barrier argument.
Suppose ζ ≥ 0 \zeta\geq0 ζ ≥ 0 . Boundedness of r A r_A r A and (55) , considering the two entries in the minimum, imply that there exists C > 0 C>0 C > 0 such that
r A 2 ≤ C ( 1 + ( 1 + t ) − α Q A − ν ) Q ˙ A . r_A^2\leq C\left(1+(1+t)^{-\alpha}Q_A^{-\nu}\right)\dot{Q}_A. r A 2 ≤ C ( 1 + ( 1 + t ) − α Q A − ν ) Q ˙ A .
Indeed, let M M M be an upper bound for r A r_A r A . If k Q A ( 1 + t ) α r A Q A ν ≥ δ k_{Q_A}(1+t)^\alpha r_AQ_A^\nu\geq\delta k Q A ( 1 + t ) α r A Q A ν ≥ δ , then Q ˙ A ≥ δ \dot{Q}_A\geq\delta Q ˙ A ≥ δ , so r A 2 ≤ M 2 ≤ ( M 2 / δ ) Q ˙ A r_A^2\leq M^2\leq(M^2/\delta)\dot{Q}_A r A 2 ≤ M 2 ≤ ( M 2 / δ ) Q ˙ A . Otherwise, Q ˙ A ≥ k Q A ( 1 + t ) α r A Q A ν \dot{Q}_A\geq k_{Q_A}(1+t)^\alpha r_AQ_A^\nu Q ˙ A ≥ k Q A ( 1 + t ) α r A Q A ν , giving r A 2 ≤ M r A ≤ ( M / k Q A ) ( 1 + t ) − α Q A − ν Q ˙ A r_A^2\leq Mr_A\leq(M/k_{Q_A})(1+t)^{-\alpha}Q_A^{-\nu}\dot{Q}_A r A 2 ≤ M r A ≤ ( M / k Q A ) ( 1 + t ) − α Q A − ν Q ˙ A . For T ≤ s ≤ t T \le s \le t T ≤ s ≤ t , the inequality α ≥ − ν \alpha\ge-\nu α ≥ − ν implies ( 1 + s ) − α ≤ ( 1 + t ) ν (1+s)^{-\alpha} \le(1+t)^\nu ( 1 + s ) − α ≤ ( 1 + t ) ν . Integrating the preceding bound and using (54) therefore yields for some C > 0 C>0 C > 0
c 2 2 t − C ≤ Q A ( t ) + 1 2 ∫ T t r A ( s ) 2 d s ≤ C Q A ( t ) + C ( 1 + t ) ν Q A ( t ) 1 − ν . \frac{c_2}{2}t-C \le Q_A(t)+\frac{1}{2}\int_T^t r_A(s)^2\,ds \le C Q_A(t)+C(1+t)^\nu Q_A(t)^{1-\nu}. 2 c 2 t − C ≤ Q A ( t ) + 2 1 ∫ T t r A ( s ) 2 d s ≤ C Q A ( t ) + C ( 1 + t ) ν Q A ( t ) 1 − ν .
Dividing by 1 + t 1+t 1 + t shows that Q A ( t ) / ( 1 + t ) Q_A(t)/(1+t) Q A ( t ) / ( 1 + t ) is bounded away from zero. Together with G ≤ G A G\le G_A G ≤ G A , this proves Q ≥ Q A ≳ t Q\ge Q_A\gtrsim t Q ≥ Q A ≳ t .
Finally, set r ~ A ( t ) = ( 1 + t ) ζ r A ( t ) \widetilde r_A(t)=(1+t)^\zeta r_A(t) r A ( t ) = ( 1 + t ) ζ r A ( t ) . The lower bound on Q A Q_A Q A and Lemma C.7 give for some c > 0 c>0 c > 0
d log r ~ A d t ≤ c 2 2 + ζ 1 + t − min { δ , c r ~ A } . \frac{d\log\widetilde r_A}{dt}\le\frac{c_2}{2}+\frac{\zeta}{1+t}-\min\{\delta,c\widetilde r_A\}. d t d log r A ≤ 2 c 2 + 1 + t ζ − min { δ , c r A } .
Since δ > c 2 / 2 \delta>c_2/2 δ > c 2 /2 , this is negative on a sufficiently late tail whenever r ~ A \widetilde r_A r A exceeds a fixed constant. Thus r ~ A \widetilde r_A r A is bounded, proving r A ≲ t − ζ r_A\lesssim t^{-\zeta} r A ≲ t − ζ .
:::
Corollary C.10 (Second-moment freezing). Assume ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) . If ζ > 1 2 \zeta>\frac{1}{2} ζ > 2 1 then V , V A V,V_A V , V A are bounded, and
∀ j ∈ [ p ] : e c 2 t v t [ j ] → t → ∞ ω j < ∞ , ω j − c 2 ∫ 0 t e c 2 s g j ( s ) 2 d s ≤ O ( t − 2 ζ + 1 ) . \forall j\in[p]:\ e^{c_2t}v_t[j]\xrightarrow[t\to\infty]{}\omega_j<\infty,\qquad\omega_j-c_2\int_0^t e^{c_2s}g_j(s)^2\,ds\le O\left(t^{-2\zeta+1}\right). ∀ j ∈ [ p ] : e c 2 t v t [ j ] t → ∞ ω j < ∞ , ω j − c 2 ∫ 0 t e c 2 s g j ( s ) 2 d s ≤ O ( t − 2 ζ + 1 ) . Proof. Lemma C.9 gives
log V A ( t ) V A ( T ) = ∫ T t r A ( s ) 2 d s ≲ ∫ T ∞ s − 2 ζ d s < ∞ . \log\frac{V_A(t)}{V_A(T)}=\int_T^t r_A(s)^2\,ds\lesssim\int_T^\infty s^{-2\zeta}\,ds<\infty. log V A ( T ) V A ( t ) = ∫ T t r A ( s ) 2 d s ≲ ∫ T ∞ s − 2 ζ d s < ∞. Thus V A V_A V A is bounded, and so is V V V by Lemma C.4 . Since e c 2 t v t [ j ] e^{c_2t}v_t[j] e c 2 t v t [ j ] is nondecreasing and e c 2 t v t [ j ] ≲ V ( t ) e^{c_2t}v_t[j]\lesssim V(t) e c 2 t v t [ j ] ≲ V ( t ) , it has a finite limit. Moreover, again due to Lemma C.4 , e c 2 t g t [ j ] 2 ≲ e c 2 t G A ( t ) 2 ≲ V A ( t ) r A ( t ) 2 ≲ t − 2 ζ e^{c_2t}g_t[j]^2\lesssim e^{c_2t}G_A(t)^2\lesssim V_A(t)r_A(t)^2\lesssim t^{-2\zeta} e c 2 t g t [ j ] 2 ≲ e c 2 t G A ( t ) 2 ≲ V A ( t ) r A ( t ) 2 ≲ t − 2 ζ , so there exists C > 0 C>0 C > 0 such that
0 ≤ ω j − c 2 ∫ 0 t e c 2 s g j ( s ) 2 d s = c 2 ∫ t ∞ e c 2 s g s [ j ] 2 d s ≤ C ∫ t ∞ ( 1 + s ) − 2 ζ d s = O ( ( 1 + t ) − 2 ζ + 1 ) 0\le\omega_j-c_2\int_0^t e^{c_2s}g_j(s)^2\,ds=c_2\int_t^\infty e^{c_2s}g_s[j]^2\,ds\le C\int_t^\infty(1+s)^{-2\zeta}\,ds=O\left((1+t)^{-2\zeta+1}\right) 0 ≤ ω j − c 2 ∫ 0 t e c 2 s g j ( s ) 2 d s = c 2 ∫ t ∞ e c 2 s g s [ j ] 2 d s ≤ C ∫ t ∞ ( 1 + s ) − 2 ζ d s = O ( ( 1 + t ) − 2 ζ + 1 ) = O ( ( 1 + t ) − 2 ( α − ( 1 L − 1 2 ) ) ) . =O\left((1+t)^{-2\left(\alpha-\left(\frac{1}{L}-\frac{1}{2}\right)\right)}\right). = O ( ( 1 + t ) − 2 ( α − ( L 1 − 2 1 ) ) ) .
On the flip side, we can show that in the intermediate and slow regimes the second (and first) moments diverge:
Proposition C.11 (No moment freezing in the intermediate and slow regimes). Suppose Assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) hold and − 1 < α ≤ 1 / L − 1 / 2 -1<\alpha\le1/L-1/2 − 1 < α ≤ 1/ L − 1/2 . Then
∫ 0 ∞ e c 2 s ∥ g s ∥ 2 2 d s = ∞ , ∫ 0 ∞ e c 1 s ∥ g s ∥ 1 d s = ∞ . (60) \int_0^\infty e^{c_2s}\lVert g_s\rVert_2^2\,ds=\infty,\qquad\int_0^\infty e^{c_1s}\lVert g_s\rVert_1\,ds=\infty. \tag*{(60)} ∫ 0 ∞ e c 2 s ∥ g s ∥ 2 2 d s = ∞ , ∫ 0 ∞ e c 1 s ∥ g s ∥ 1 d s = ∞. (60) Proof. Assume, toward a contradiction, that
∫ 0 ∞ e c 2 s ∥ g s ∥ 2 2 d s < ∞ . (61) \int_0^\infty e^{c_2s}\lVert g_s\rVert_2^2\,ds<\infty. \tag*{(61)} ∫ 0 ∞ e c 2 s ∥ g s ∥ 2 2 d s < ∞. (61) There exist T > 0 T>0 T > 0 , { a j } j ∈ [ p ] \{a_j\}_{j\in[p]} { a j } j ∈ [ p ] so that for any j j j with ω j > 0 \omega_j>0 ω j > 0 , it holds that ∀ t ≥ T \forall t\ge T ∀ t ≥ T : v t [ j ] ≥ a j e − c 2 t / 2 \sqrt{v_t[j]}\ge a_j e^{-c_2t/2} v t [ j ] ≥ a j e − c 2 t /2 , thus
∫ T ∞ m s [ j ] 2 v s [ j ] d s ≤ a j − 2 ∫ T ∞ e c 2 s m s [ j ] 2 d s ≤ a j − 2 ( c 1 c 1 − c 2 2 ) 2 ∫ 0 ∞ e c 2 s g s [ j ] 2 d s < ∞ . (62) \int_T^\infty\frac{m_s[j]^2}{v_s[j]}\,ds\le a_j^{-2}\int_T^\infty e^{c_2s}m_s[j]^2\,ds\le a_j^{-2}\left(\frac{c_1}{c_1-\frac{c_2}{2}}\right)^2\int_0^\infty e^{c_2s}g_s[j]^2\,ds<\infty. \tag*{(62)} ∫ T ∞ v s [ j ] m s [ j ] 2 d s ≤ a j − 2 ∫ T ∞ e c 2 s m s [ j ] 2 d s ≤ a j − 2 ( c 1 − 2 c 2 c 1 ) 2 ∫ 0 ∞ e c 2 s g s [ j ] 2 d s < ∞. (62) Where the last inequality is (61) , and the second-to-last is Lemma B.7 with p = 2 p=2 p = 2 , c = c 1 c=c_1 c = c 1 , λ = c 2 2 \lambda=\frac{c_2}{2} λ = 2 c 2 . Therefore, integrating the Adam equation, applying Cauchy–Schwarz and (62) ,
∥ θ t ∥ 2 ≤ ∥ θ T ∥ 2 + ( ∫ T t η ^ ( s ) 2 d s ) 1 / 2 ( ∫ T ∞ ∥ m s v s ∥ 2 2 d s ) 1 / 2 ≤ O ( ( ∫ T t η ^ ( s ) 2 d s ) 1 / 2 ) . (63) \lVert\theta_t\rVert_2\le\lVert\theta_T\rVert_2+\left(\int_T^t\widehat\eta(s)^2\,ds\right)^{1/2}\left(\int_T^\infty\left\lVert\frac{m_s}{\sqrt{v_s}}\right\rVert_2^2\,ds\right)^{1/2}\le O\left(\left(\int_T^t\widehat\eta(s)^2\,ds\right)^{1/2}\right). \tag*{(63)} ∥ θ t ∥ 2 ≤ ∥ θ T ∥ 2 + ( ∫ T t η ( s ) 2 d s ) 1/2 ( ∫ T ∞ v s m s 2 2 d s ) 1/2 ≤ O ( ( ∫ T t η ( s ) 2 d s ) 1/2 ) . (63)
Where,
( ∫ T t η ^ ( s ) 2 d s ) 1 / 2 = { O ( t α + 1 / 2 ) , α > − 1 / 2 , O ( log t ) , α = − 1 / 2 , O ( 1 ) , α < − 1 / 2. (64) \left(\int_T^t \widehat{\eta}(s)^2\,ds\right)^{1/2} =
\begin{cases}
\mathcal{O}\left(t^{\alpha+1/2}\right), & \alpha> -1/2,\\
\mathcal{O}\left(\sqrt{\log t}\right), & \alpha= -1/2,\\
\mathcal{O}(1), & \alpha< -1/2.
\end{cases} \tag*{(64)} ( ∫ T t η ( s ) 2 d s ) 1/2 = ⎩ ⎨ ⎧ O ( t α + 1/2 ) , O ( log t ) , O ( 1 ) , α > − 1/2 , α = − 1/2 , α < − 1/2. (64)
If α < 1 L − 1 2 \alpha< \frac{1}{L}-\frac{1}{2} α < L 1 − 2 1 , the above bounds imply
∥ θ t ∥ 2 = o ( t 1 / L ) . (65) \lVert\theta_t\rVert_2 = o\left(t^{1/L}\right). \tag*{(65)} ∥ θ t ∥ 2 = o ( t 1/ L ) . (65)
For the endpoint α = 1 L − 1 2 \alpha= \frac{1}{L}-\frac{1}{2} α = L 1 − 2 1 , fix S ≥ T S \ge T S ≥ T sufficiently large. Applying Cauchy–Schwarz as in (63) shows that there exists C > 0 C > 0 C > 0 , independent of S S S and t ≥ S t \ge S t ≥ S , such that
∥ θ t ∥ 2 ≤ ∥ θ S ∥ 2 + ( ∫ S t η ^ ( s ) 2 d s ) 1 / 2 ( ∫ S ∞ ∥ m s v s ∥ 2 2 d s ) 1 / 2 ≤ ∥ θ S ∥ 2 + C t 1 / L ( ∫ S ∞ ∥ m s v s ∥ 2 2 d s ) 1 / 2 , \begin{aligned}
\lVert\theta_t\rVert_2 \le\lVert\theta_S\rVert_2 + \left(\int_S^t \widehat{\eta}(s)^2\,ds\right)^{1/2}\left(\int_S^\infty\left\lVert\frac{\mathbf{m}_s}{\sqrt{v_s}}\right\rVert_2^2\,ds\right)^{1/2} \\
&\le\lVert\theta_S\rVert_2 + Ct^{1/L}\left(\int_S^\infty\left\lVert\frac{\mathbf{m}_s}{\sqrt{v_s}}\right\rVert_2^2\,ds\right)^{1/2},
\end{aligned} ∥ θ t ∥ 2 ≤ ∥ θ S ∥ 2 + ( ∫ S t η ( s ) 2 d s ) 1/2 ( ∫ S ∞ v s m s 2 2 d s ) 1/2 ≤ ∥ θ S ∥ 2 + C t 1/ L ( ∫ S ∞ v s m s 2 2 d s ) 1/2 ,
where C > 0 C > 0 C > 0 is independent of S S S and t ≥ S t \ge S t ≥ S , since η ^ ( s ) 2 = O ( s 2 / L − 1 ) \widehat{\eta}(s)^2 = \mathcal{O}\left(s^{2/L-1}\right) η ( s ) 2 = O ( s 2/ L − 1 ) . Dividing by t 1 / L t^{1/L} t 1/ L and taking the limit superior as t → ∞ t \to\infty t → ∞ gives
lim sup t → ∞ ∥ θ t ∥ 2 t 1 / L ≤ C ( ∫ S ∞ ∥ m s v s ∥ 2 2 d s ) 1 / 2 . \limsup_{t\to\infty}\frac{\lVert\theta_t\rVert_2}{t^{1/L}} \le C\left(\int_S^\infty\left\lVert\frac{\mathbf{m}_s}{\sqrt{v_s}}\right\rVert_2^2\,ds\right)^{1/2}. t → ∞ lim sup t 1/ L ∥ θ t ∥ 2 ≤ C ( ∫ S ∞ v s m s 2 2 d s ) 1/2 .
Letting S → ∞ S \to\infty S → ∞ and using (62) , we obtain (65) also at the endpoint.
This further implies
q i ( θ t ) = o ( t ) , U ( t ) = − log L ( θ t ) = o ( t ) . q_i(\theta_t)=o(t), \qquad U(t)=-\log\mathcal{L}(\theta_t)=o(t). q i ( θ t ) = o ( t ) , U ( t ) = − log L ( θ t ) = o ( t ) .
But this is a contradiction to (61) , since this implies divergence of e c 2 t ∥ g t ∥ 1 2 e^{c_2t}\lVert g_t\rVert_1^2 e c 2 t ∥ g t ∥ 1 2 :
e c 2 t ∥ g t ∥ 1 2 ≳ exp ( c 2 t − 2 U ( t ) ) ⟶ ∞ . (66) e^{c_2t}\lVert g_t\rVert_1^2 \gtrsim\exp\left(c_2t-2U(t)\right)\longrightarrow\infty. \tag*{(66)} e c 2 t ∥ g t ∥ 1 2 ≳ exp ( c 2 t − 2 U ( t ) ) ⟶ ∞. (66)
As for the first moment, recall that L → 0 \mathcal{L}\to0 L → 0 implies ∥ g t ∥ 1 → 0 \lVert g_t\rVert_1\to0 ∥ g t ∥ 1 → 0 . Also, c 1 ≥ c 2 c_1\ge c_2 c 1 ≥ c 2 . Together this implies for large enough t t t ,
e c 1 t ∥ g t ∥ 1 ≥ e c 2 t ∥ g t ∥ 1 2 , e^{c_1t}\lVert g_t\rVert_1 \ge e^{c_2t}\lVert g_t\rVert_1^2, e c 1 t ∥ g t ∥ 1 ≥ e c 2 t ∥ g t ∥ 1 2 ,
hence ∫ ∞ e c 1 s ∥ g s ∥ 1 d s = ∞ \int^\infty e^{c_1s}\lVert g_s\rVert_1\,ds=\infty ∫ ∞ e c 1 s ∥ g s ∥ 1 d s = ∞ by comparison.
Corollary C.12 (Tracking and Speed Estimates). Assume ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) and ζ ≥ 0 \zeta\ge0 ζ ≥ 0 and fix j ∈ [ p ] j \in[p] j ∈ [ p ] . If ζ > 1 2 \zeta> \frac{1}{2} ζ > 2 1 or if ∣ g t [ j ] ∣ ∥ g t ∥ 1 \frac{|g_t[j]|}{\lVert g_t\rVert_1} ∥ g t ∥ 1 ∣ g t [ j ] ∣ is eventually bounded away from 0 0 0 , then
∣ m t [ j ] ∣ v t [ j ] ≤ O ( t − ζ ) , g t 2 [ j ] v t [ j ] ≤ O ( t − 2 ζ ) . (67) \frac{|m_t[j]|}{\sqrt{v_t[j]}} \le\mathcal{O}\left(t^{-\zeta}\right), \qquad\frac{g_t^2[j]}{v_t[j]} \le\mathcal{O}\left(t^{-2\zeta}\right). \tag*{(67)} v t [ j ] ∣ m t [ j ] ∣ ≤ O ( t − ζ ) , v t [ j ] g t 2 [ j ] ≤ O ( t − 2 ζ ) . (67)
Proof. Coordinates with identically zero second moment are immediate under the convention. Otherwise, set V j ( t ) : = e c 2 t v t [ j ] V_j(t):=e^{c_2t}v_t[j] V j ( t ) := e c 2 t v t [ j ] and choose T T T sufficiently large that V j ( T ) > 0 V_j(T)>0 V j ( T ) > 0 .
We first show that V A / V j V_A/V_j V A / V j is bounded. If ζ > 1 / 2 \zeta> 1/2 ζ > 1/2 , this follows from Corollary C.10 and V j ( t ) ≥ V j ( T ) > 0 V_j(t)\ge V_j(T)>0 V j ( t ) ≥ V j ( T ) > 0 . Under the second condition, Lemma C.3 gives ∣ g t [ j ] ∣ ≳ G ( t ) |g_t[j]|\gtrsim G(t) ∣ g t [ j ] ∣ ≳ G ( t ) , so for some c > 0 c>0 c > 0
V j ( t ) = V j ( T ) + c 2 ∫ T t e c 2 s g s [ j ] 2 d s ≥ V j ( T ) + c ( V ( t ) − V ( T ) ) ≳ V ( t ) ≍ V A ( t ) . V_j(t)=V_j(T)+c_2\int_T^t e^{c_2s}g_s[j]^2\,ds \ge V_j(T)+c\left(V(t)-V(T)\right)\gtrsim V(t)\asymp V_A(t). V j ( t ) = V j ( T ) + c 2 ∫ T t e c 2 s g s [ j ] 2 d s ≥ V j ( T ) + c ( V ( t ) − V ( T ) ) ≳ V ( t ) ≍ V A ( t ) .
In either case, the gradient-scale estimate (50) gives
g t [ j ] 2 v t [ j ] ≲ e c 2 t G A ( t ) 2 V j ( t ) = V A ( t ) V j ( t ) r A ( t ) 2 ≲ t − 2 ζ . \frac{g_t[j]^2}{v_t[j]} \lesssim\frac{e^{c_2t}G_A(t)^2}{V_j(t)}=\frac{V_A(t)}{V_j(t)}r_A(t)^2\lesssim t^{-2\zeta}. v t [ j ] g t [ j ] 2 ≲ V j ( t ) e c 2 t G A ( t ) 2 = V j ( t ) V A ( t ) r A ( t ) 2 ≲ t − 2 ζ .
The last inequality follows from Lemma C.9 , using boundedness of r A r_A r A when ζ = 0 \zeta=0 ζ = 0 . Next, z j : = ∣ m t [ j ] ∣ / v t [ j ] z_j := |m_t[j]|/\sqrt{v_t[j]} z j := ∣ m t [ j ] ∣/ v t [ j ] is locally absolutely continuous on this tail. Lemma B.3 implies that there exists C > 0 C>0 C > 0 such that, almost everywhere,
z ˙ j ≤ − λ z j + c 1 ∣ g t [ j ] ∣ v t [ j ] ≤ − λ z j + C t − ζ , λ = c 1 − c 2 2 > 0. \dot{z}_j \le-\lambda z_j+c_1\frac{|g_t[j]|}{\sqrt{v_t[j]}} \le-\lambda z_j+Ct^{-\zeta},\qquad\lambda=c_1-\frac{c_2}{2}>0. z ˙ j ≤ − λ z j + c 1 v t [ j ] ∣ g t [ j ] ∣ ≤ − λ z j + C t − ζ , λ = c 1 − 2 c 2 > 0.
Integrating yields
z j ( t ) ≤ e − λ ( t − T ) z j ( T ) + C ∫ T t e − λ ( t − s ) s − ζ d s ≲ t − ζ , z_j(t)\le e^{-\lambda(t-T)}z_j(T)+C\int_T^t e^{-\lambda(t-s)}s^{-\zeta}\,ds\lesssim t^{-\zeta}, z j ( t ) ≤ e − λ ( t − T ) z j ( T ) + C ∫ T t e − λ ( t − s ) s − ζ d s ≲ t − ζ ,
as required. □
Theorem C.13 (Loss decay rate). Let f f f satisfy ( M 1 ) (M1) ( M 1 ) , ( M 2 ) (M2) ( M 2 ) , let ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e and let θ t \theta_t θ t be an Adam trajectory of f f f assuming ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) . Writing U ( t ) : = − log L ( θ t ) U(t):=-\log L(\theta_t) U ( t ) := − log L ( θ t ) and ν : = 1 − 1 / L \nu:=1-1/L ν := 1 − 1/ L , we have
U ( t ) = c 2 2 t + ( ζ + ν ) log t ± O ( 1 ) , ζ > 1 2 , c 2 2 t − O ( log t ) ≤ U ( t ) ≤ c 2 2 t + O ( t ) , ζ = 1 2 , c 2 2 t − O ( t 1 − 2 ζ ) ≤ U ( t ) ≤ c 2 2 t + O ( t ζ ) , 0 < ζ < 1 2 , Ω ( t ) ≤ U ( t ) ≤ c 2 2 t + ν log t + O ( 1 ) , ζ = 0 , U ( t ) = Θ ( t L ( α + 1 ) ) , ζ < 0. (68) \begin{aligned}
U(t)&=\frac{c_2}{2}t+(\zeta+\nu)\log t\pm O(1), &&\zeta>\frac{1}{2},\\
\frac{c_2}{2}t-O(\log t)&\le U(t)\le\frac{c_2}{2}t+O(\sqrt{t}), &&\zeta=\frac{1}{2},\\
\frac{c_2}{2}t-O\left(t^{1-2\zeta}\right)&\le U(t)\le\frac{c_2}{2}t+O(t^\zeta), &&0<\zeta<\frac{1}{2},\\
\Omega(t)&\le U(t)\le\frac{c_2}{2}t+\nu\log t+O(1), &&\zeta=0,\\
U(t)&=\Theta\left(t^{L(\alpha+1)}\right), &&\zeta<0. \tag*{(68)}
\end{aligned} U ( t ) 2 c 2 t − O ( log t ) 2 c 2 t − O ( t 1 − 2 ζ ) Ω ( t ) U ( t ) = 2 c 2 t + ( ζ + ν ) log t ± O ( 1 ) , ≤ U ( t ) ≤ 2 c 2 t + O ( t ) , ≤ U ( t ) ≤ 2 c 2 t + O ( t ζ ) , ≤ U ( t ) ≤ 2 c 2 t + ν log t + O ( 1 ) , = Θ ( t L ( α + 1 ) ) , ζ > 2 1 , ζ = 2 1 , 0 < ζ < 2 1 , ζ = 0 , ζ < 0. (68)
Here asymptotic notation hides constants which may depend on hyperparameters, data or initialization (but not on t t t ).
Proof. First suppose ζ < 0 \zeta<0 ζ < 0 . Lemma C.8 and Q ∼ U Q\sim U Q ∼ U give U ( t ) = O ( t L ( α + 1 ) ) U(t)=O\left(t^{L(\alpha+1)}\right) U ( t ) = O ( t L ( α + 1 ) ) . Equation (58) gives
∣ U ˙ ∣ ≲ η U ν = O ( t L ( α + 1 ) − 1 ) = o ( 1 ) . |\dot{U}|\lesssim\eta U^\nu=O\left(t^{L(\alpha+1)-1}\right)=o(1). ∣ U ˙ ∣ ≲ η U ν = O ( t L ( α + 1 ) − 1 ) = o ( 1 ) .
Consequently,
d d t log G = ( ν L − 1 ) U ˙ = o ( 1 ) , d d t ( e c 2 t G 2 ) = ( c 2 + o ( 1 ) ) V ˙ . \frac{d}{dt}\log G=\left(\frac{\nu}{L}-1\right)\dot{U}=o(1),\qquad\frac{d}{dt}\left(e^{c_2tG^2}\right)=(c_2+o(1))\dot{V}. d t d log G = ( L ν − 1 ) U ˙ = o ( 1 ) , d t d ( e c 2 t G 2 ) = ( c 2 + o ( 1 )) V ˙ .
Since U = o ( t ) U=o(t) U = o ( t ) , we have e c 2 t G 2 → ∞ e^{c_2tG^2}\to\infty e c 2 t G 2 → ∞ ; integration gives e c 2 t G 2 ∼ c 2 V e^{c_2tG^2}\sim c_2V e c 2 t G 2 ∼ c 2 V .
Lemma C.4 now yields
r A 2 = e c 2 t G A 2 V A ≳ e c 2 t G 2 V ≳ 1. r_A^2=\frac{e^{c_2tG_A^2}}{V_A}\gtrsim\frac{e^{c_2tG^2}}{V}\gtrsim1. r A 2 = V A e c 2 t G A 2 ≳ V e c 2 t G 2 ≳ 1.
Since η Q A ν ≤ η U ν = o ( 1 ) \eta Q_A^\nu\le\eta U^\nu=o(1) η Q A ν ≤ η U ν = o ( 1 ) , (55) implies
Q ˙ A ≳ η Q A ν , d d t Q A 1 / L ≳ η . \dot{Q}_A\gtrsim\eta Q_A^\nu,\qquad\frac{d}{dt}Q_A^{1/L}\gtrsim\eta. Q ˙ A ≳ η Q A ν , d t d Q A 1/ L ≳ η .
Integrating gives U ( t ) ≥ Q A ( t ) ≳ t L ( α + 1 ) U(t)\ge Q_A(t)\gtrsim t^{L(\alpha+1)} U ( t ) ≥ Q A ( t ) ≳ t L ( α + 1 ) , which proves the slow-regime estimate.
For the remaining cases, suppose ζ ≥ 0 \zeta\ge0 ζ ≥ 0 .
We use the feedback mechanisms to define barrier variables on both sides. We first phrase all bounds using Q Q Q , and conclude with the relation between Q Q Q and U = − log L U=-\log L U = − log L . Recall ν = 1 − 1 L \nu=1-\frac{1}{L} ν = 1 − L 1 , ζ = α + ν \zeta=\alpha+\nu ζ = α + ν , and set
B ( t ) : = c 2 2 t + ζ log ( 1 + t ) . B(t):=\frac{c_2}{2}t+\zeta\log(1+t). B ( t ) := 2 c 2 t + ζ log ( 1 + t ) .
We first obtain the lower bound from the coarse estimates, then construct an upper barrier.
Define
b ( t ) : = 1 2 log V A ( t ) , Q − ( t ) : = Q A ( t ) + b ( t ) . b(t):=\frac{1}{2}\log V_A(t),\qquad Q_-(t):=Q_A(t)+b(t). b ( t ) := 2 1 log V A ( t ) , Q − ( t ) := Q A ( t ) + b ( t ) .
The identity r A = e c 2 t / 2 − Q A / V A r_A=e^{c_2t/2-Q_A}/\sqrt{V_A} r A = e c 2 t /2 − Q A / V A and Lemma C.9 imply that there exists C > 0 C>0 C > 0 such that
Q − ( t ) = c 2 2 t − log r A ( t ) ≥ B ( t ) − C , Q_-(t)=\frac{c_2}{2}t-\log r_A(t)\ge B(t)-C, Q − ( t ) = 2 c 2 t − log r A ( t ) ≥ B ( t ) − C ,
and
Q ( t ) ≥ Q A ( t ) = Q − ( t ) − b ( t ) ≥ B ( t ) − b ( t ) − C . Q(t)\ge Q_A(t)=Q_-(t)-b(t)\ge B(t)-b(t)-C. Q ( t ) ≥ Q A ( t ) = Q − ( t ) − b ( t ) ≥ B ( t ) − b ( t ) − C .
Moreover, b ˙ = r A 2 / 2 \dot{b}=r_A^2/2 b ˙ = r A 2 /2 , and by Lemma C.9 , r A = O ( t − ζ ) r_A=O(t^{-\zeta}) r A = O ( t − ζ ) whenever ζ > 0 \zeta>0 ζ > 0 , so b ( t ) = O ( t 1 − 2 ζ ) b(t)=O(t^{1-2\zeta}) b ( t ) = O ( t 1 − 2 ζ ) whenever 1 2 > ζ > 0 \frac{1}{2}>\zeta>0 2 1 > ζ > 0 and b ( t ) = O ( log t ) b(t)=O(\log t) b ( t ) = O ( log t ) when ζ = 1 2 \zeta=\frac{1}{2} ζ = 2 1 . If ζ > 1 2 \zeta>\frac{1}{2} ζ > 2 1 , then b ( t ) = O ( 1 ) b(t)=O(1) b ( t ) = O ( 1 ) by Corollary C.10. When ζ = 0 \zeta=0 ζ = 0 , Q ( t ) ≥ Q A ( t ) ≳ t Q(t)\ge Q_A(t)\gtrsim t Q ( t ) ≥ Q A ( t ) ≳ t by Lemma C.9 . This finishes the lower bound.
For the upper bound, there exists C s > 0 C_s>0 C s > 0 such that, on a common tail,
s A ( t ) ≤ S ( t ) : = { C s , 0 ≤ ζ ≤ 1 / 2 , C s ( 1 + t ) − ζ , ζ > 1 / 2. s_A(t)\le S(t):=
\begin{cases}
C_s, & 0\le\zeta\le1/2,\\
C_s(1+t)^{-\zeta}, & \zeta>1/2.
\end{cases} s A ( t ) ≤ S ( t ) := { C s , C s ( 1 + t ) − ζ , 0 ≤ ζ ≤ 1/2 , ζ > 1/2.
The first case follows from Lemma C.8 . The second follows by applying Corollary C.12 to every coordinate. Let C 0 C_0 C 0 be the constant in (57) , write λ = c 1 − c 2 / 2 \lambda=c_1-c_2/2 λ = c 1 − c 2 /2 , and fix A > C 0 / λ A>C_0/\lambda A > C 0 / λ . Define the corrected upper variable
Q + ( t ) : = ( Q ( t ) 1 / L + A η ( t ) s A ( t ) ) L . Q_+(t):=\left(Q(t)^{1/L}+A\eta(t)s_A(t)\right)^L. Q + ( t ) := ( Q ( t ) 1/ L + A η ( t ) s A ( t ) ) L .
By the upper-feedback inequalities, there exists C A > 0 C_A>0 C A > 0 such that, almost everywhere on a sufficiently late tail,
d d t Q + 1 / L ≤ ( C 0 − A λ + A α 1 + t ) η s A + C A η e c 2 t / 2 − Q ≤ C A η e c 2 t / 2 − Q . \frac{d}{dt}Q_+^{1/L}\le\left(C_0-A\lambda+\frac{A\alpha}{1+t}\right)\eta s_A+C_A\eta e^{c_2t/2-Q}
\le C_A\eta e^{c_2t/2-Q}. d t d Q + 1/ L ≤ ( C 0 − A λ + 1 + t A α ) η s A + C A η e c 2 t /2 − Q ≤ C A η e c 2 t /2 − Q .
For a constant D > 0 D>0 D > 0 to be chosen, define
B + ( t ) : = ( ( B ( t ) + D ) 1 / L + A η ( t ) S ( t ) ) L . B_+(t):=\left((B(t)+D)^{1/L}+A\eta(t)S(t)\right)^L. B + ( t ) := ( ( B ( t ) + D ) 1/ L + A η ( t ) S ( t ) ) L .
The explicit choice of S S S gives, in both regimes,
η S = o ( t 1 / L ) , d d t ( η S ) = o ( t − ν ) . \eta S=o\left(t^{1/L}\right),\qquad\frac{d}{dt}(\eta S)=o\left(t^{-\nu}\right). η S = o ( t 1/ L ) , d t d ( η S ) = o ( t − ν ) .
It follows that
d d t B + 1 / L = ( ( c 2 / 2 ) 1 / L L + o ( 1 ) ) ( 1 + t ) − ν . \frac{d}{dt}B_+^{1/L}=\left(\frac{(c_2/2)^{1/L}}{L}+o(1)\right)(1+t)^{-\nu}. d t d B + 1/ L = ( L ( c 2 /2 ) 1/ L + o ( 1 ) ) ( 1 + t ) − ν .
Whenever Q + ≥ B + Q_+\ge B_+ Q + ≥ B + , the inequality s A ≤ S s_A\le S s A ≤ S implies Q ≥ B + D Q\ge B+D Q ≥ B + D . Hence, at such times,
d d t Q + 1 / L ≤ C A η e c 2 t / 2 − B − D = C A η 0 e − D ( 1 + t ) − ν . \frac{d}{dt}Q_+^{1/L}\le C_A\eta e^{c_2t/2-B-D}=C_A\eta_0e^{-D}(1+t)^{-\nu}. d t d Q + 1/ L ≤ C A η e c 2 t /2 − B − D = C A η 0 e − D ( 1 + t ) − ν .
Choose D D D sufficiently large. There exists c > 0 c>0 c > 0 such that, on a sufficiently late tail,
d d t ( Q + 1 / L − B + 1 / L ) ≤ − c ( 1 + t ) − ν whenever Q + ≥ B + . \frac{d}{dt}\left(Q_+^{1/L}-B_+^{1/L}\right)\le-c(1+t)^{-\nu}\qquad\text{whenever }Q_+\ge B_+. d t d ( Q + 1/ L − B + 1/ L ) ≤ − c ( 1 + t ) − ν whenever Q + ≥ B + .
Since ν < 1 \nu<1 ν < 1 , the integral of ( 1 + t ) − ν (1+t)^{-\nu} ( 1 + t ) − ν diverges. Thus Q + Q_+ Q + eventually enters and remains below B + B_+ B + , so Q ≤ Q + ≤ B + Q\le Q_+\le B_+ Q ≤ Q + ≤ B + on a sufficiently late tail. Since B ( t ) + D ≍ t B(t)+D\asymp t B ( t ) + D ≍ t and η S = o ( t 1 / L ) \eta S=o(t^{1/L}) η S = o ( t 1/ L ) , we have
B + ( t ) 1 / L = ( B ( t ) + D ) 1 / L + A η ( t ) S ( t ) ≍ t 1 / L . B_+(t)^{1/L}=(B(t)+D)^{1/L}+A\eta(t)S(t)\asymp t^{1/L}. B + ( t ) 1/ L = ( B ( t ) + D ) 1/ L + A η ( t ) S ( t ) ≍ t 1/ L .
For any fixed t t t , the mean value theorem on the interval [ ( B ( t ) + D ) 1 / L , ( B ( t ) + D ) 1 / L + A η ( t ) S ( t ) ] [(B(t)+D)^{1/L},(B(t)+D)^{1/L}+A\eta(t)S(t)] [( B ( t ) + D ) 1/ L , ( B ( t ) + D ) 1/ L + A η ( t ) S ( t )] , using L ≥ 1 L\ge1 L ≥ 1 , gives, for some C > 0 C>0 C > 0 ,
0 ≤ B + ( t ) − ( B ( t ) + D ) ≤ L A η ( t ) S ( t ) ( ( B ( t ) + D ) 1 / L + A η ( t ) S ( t ) ) L − 1 ≤ C t 1 − 1 / L η ( t ) S ( t ) = C t ν η ( t ) S ( t ) . \begin{aligned}
0\le B_+(t)-(B(t)+D)\le LA\eta(t)S(t)\left((B(t)+D)^{1/L}+A\eta(t)S(t)\right)^{L-1} \\
&\le Ct^{1-1/L}\eta(t)S(t)=Ct^\nu\eta(t)S(t).
\end{aligned} 0 ≤ B + ( t ) − ( B ( t ) + D ) ≤ L A η ( t ) S ( t ) ( ( B ( t ) + D ) 1/ L + A η ( t ) S ( t ) ) L − 1 ≤ C t 1 − 1/ L η ( t ) S ( t ) = C t ν η ( t ) S ( t ) .
If 0 ≤ ζ ≤ 1 / 2 0 \le\zeta\le1/2 0 ≤ ζ ≤ 1/2 , then S = C s S=C_s S = C s , so
t ν η ( t ) S ( t ) = η 0 C s t ν ( 1 + t ) α = O ( t ν + α ) = O ( t ζ ) . t^\nu\eta(t)S(t)=\eta_0C_st^\nu(1+t)^\alpha=O\left(t^{\nu+\alpha}\right)=O\left(t^\zeta\right). t ν η ( t ) S ( t ) = η 0 C s t ν ( 1 + t ) α = O ( t ν + α ) = O ( t ζ ) .
If ζ > 1 / 2 \zeta>1/2 ζ > 1/2 , then S = C s ( 1 + t ) − ζ S=C_s(1+t)^{-\zeta} S = C s ( 1 + t ) − ζ , and α − ζ = − ν \alpha-\zeta=-\nu α − ζ = − ν gives
t ν η ( t ) S ( t ) = η 0 C s t ν ( 1 + t ) α − ζ = O ( 1 ) . t^\nu\eta(t)S(t)=\eta_0C_st^\nu(1+t)^{\alpha-\zeta}=O(1). t ν η ( t ) S ( t ) = η 0 C s t ν ( 1 + t ) α − ζ = O ( 1 ) .
Consequently, absorbing the fixed constant D D D into the error terms,
Q ( t ) ≤ B + ( t ) = { B ( t ) + O ( t ζ ) , 0 ≤ ζ ≤ 1 / 2 , B ( t ) + O ( 1 ) , ζ > 1 / 2. Q(t)\le B_+(t)=
\begin{cases}
B(t)+O\left(t^\zeta\right), & 0\le\zeta\le1/2,\\
B(t)+O(1), & \zeta>1/2.
\end{cases} Q ( t ) ≤ B + ( t ) = { B ( t ) + O ( t ζ ) , B ( t ) + O ( 1 ) , 0 ≤ ζ ≤ 1/2 , ζ > 1/2.
Finally, Q = U − ν log U Q=U-\nu\log U Q = U − ν log U and U → ∞ U\to\infty U → ∞ imply Q / U → 1 Q/U\to1 Q / U → 1 . Therefore
U = Q + ν log ( Q ( 1 + o ( 1 ) ) ) = Q + ν log Q + o ( 1 ) . U=Q+\nu\log\left(Q(1+o(1))\right)=Q+\nu\log Q+o(1). U = Q + ν log ( Q ( 1 + o ( 1 )) ) = Q + ν log Q + o ( 1 ) .
The lower and upper bounds give Q ( t ) ≍ t Q(t)\asymp t Q ( t ) ≍ t , so log Q ( t ) = log t + O ( 1 ) \log Q(t)=\log t+O(1) log Q ( t ) = log t + O ( 1 ) . If ζ > 1 / 2 \zeta>1/2 ζ > 1/2 , this yields
U ( t ) = c 2 2 t + ( ζ + ν ) log t + O ( 1 ) . U(t)=\frac{c_2}{2}t+(\zeta+\nu)\log t+O(1). U ( t ) = 2 c 2 t + ( ζ + ν ) log t + O ( 1 ) .
For 0 < ζ ≤ 1 / 2 0<\zeta\le1/2 0 < ζ ≤ 1/2 , the logarithmic terms are absorbed into O ( t ζ ) O\left(t^\zeta\right) O ( t ζ ) in the upper bound. When ζ = 0 \zeta=0 ζ = 0 , the bounds Q ( t ) ≳ t Q(t)\gtrsim t Q ( t ) ≳ t and Q ( t ) ≤ ( c 2 / 2 ) t + O ( 1 ) Q(t)\le(c_2/2)t+O(1) Q ( t ) ≤ ( c 2 /2 ) t + O ( 1 ) give
U ( t ) ≳ t , U ( t ) ≤ c 2 2 t + ν log t + O ( 1 ) . U(t)\gtrsim t,\qquad U(t)\le\frac{c_2}{2}t+\nu\log t+O(1). U ( t ) ≳ t , U ( t ) ≤ 2 c 2 t + ν log t + O ( 1 ) .
This completes the estimates.
Lemma C.14 (Uniform early-phase dominance of the second moment). Fix L ≥ 1 L\ge1 L ≥ 1 , α ∈ R \alpha\in\mathbb{R} α ∈ R with α > 1 L − 1 2 \alpha>\frac{1}{L}-\frac{1}{2} α > L 1 − 2 1 , and c 2 , k , K > 0 c_2,k,K>0 c 2 , k , K > 0 . There exist constants ε 0 , C > 0 \varepsilon_0,C>0 ε 0 , C > 0 , depending only on L , α , c 2 , k , K , η 0 L,\alpha,c_2,k,K,\eta_0 L , α , c 2 , k , K , η 0 , with the following property. Fix j ∈ [ p ] j\in[p] j ∈ [ p ] and write
ν : = 1 − 1 L , ζ : = α + ν , I j ( t ) : = e c 2 t v j ( t ) . \nu:=1-\frac{1}{L},\qquad\zeta:=\alpha+\nu,\qquad I_j(t):=e^{c_2t}v_j(t). ν := 1 − L 1 , ζ := α + ν , I j ( t ) := e c 2 t v j ( t ) . Assume that:
Every gradient coordinate has a fixed sign, and t ↦ ∣ g i ( t ) ∣ t\mapsto|g_i(t)| t ↦ ∣ g i ( t ) ∣ is nonincreasing on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) for every i ∈ [ p ] i\in[p] i ∈ [ p ] .
The following bounds hold:
U ( 1 / 2 ) ≥ max { 1 , 4 ν } , k G ( t ) ≤ ∣ g j ( t ) ∣ ≤ K G ( t ) for almost every t ≥ 1 / 2. U(1/2)\ge\max\{1,4\nu\},\qquad kG(t)\le|g_j(t)|\le KG(t)\qquad\text{for almost every }t\ge1/2. U ( 1/2 ) ≥ max { 1 , 4 ν } , k G ( t ) ≤ ∣ g j ( t ) ∣ ≤ K G ( t ) for almost every t ≥ 1/2. Then, if L = 1 L=1 L = 1 , or if L > 1 L>1 L > 1 and I j ( 1 ) ≤ ε 0 I_j(1)\le\varepsilon_0 I j ( 1 ) ≤ ε 0 ,
ω j : = lim t → ∞ I j ( t ) ≤ C I j ( 1 ) < ∞ , \omega_j:=\lim_{t\to\infty}I_j(t)\le CI_j(1)<\infty, ω j := t → ∞ lim I j ( t ) ≤ C I j ( 1 ) < ∞ , and
I ˙ j ( t ) I j ( t ) ≤ C ( 1 + t ) − 2 ζ for almost every t ≥ 1. (69) \frac{\dot I_j(t)}{I_j(t)}\le C(1+t)^{-2\zeta}\qquad\text{for almost every }t\ge1. \tag*{(69)} I j ( t ) I ˙ j ( t ) ≤ C ( 1 + t ) − 2 ζ for almost every t ≥ 1. (69) Moreover, when L > 1 L>1 L > 1 ,
ω j = I j ( 1 ) ( 1 + o ( 1 ) ) as I j ( 1 ) → 0 , \omega_j=I_j(1)(1+o(1))\qquad\text{as }I_j(1)\to0, ω j = I j ( 1 ) ( 1 + o ( 1 )) as I j ( 1 ) → 0 , uniformly over all trajectories satisfying these assumptions.
One also has
0 ≤ ω j − I j ( t ) ≤ C I j ( 1 ) ( 1 + t ) 1 − 2 ζ , t ≥ 1. 0\le\omega_j-I_j(t)\le CI_j(1)(1+t)^{1-2\zeta},\qquad t\ge1. 0 ≤ ω j − I j ( t ) ≤ C I j ( 1 ) ( 1 + t ) 1 − 2 ζ , t ≥ 1.
Proof. All constants below depend only on L , α , c 2 , k , K , η 0 L,\alpha,c_2,k,K,\eta_0 L , α , c 2 , k , K , η 0 . The coordinatewise gradient assumptions and zero initial momentum give ∀ i ∈ [ p ] : ∣ m ^ i ( t ) ∣ ≥ ∣ g i ( t ) ∣ \forall i\in[p]:|\hat m_i(t)|\geq|g_i(t)| ∀ i ∈ [ p ] : ∣ m ^ i ( t ) ∣ ≥ ∣ g i ( t ) ∣ , therefore g i ( t ) m ^ i ( t ) ≥ g i ( t ) 2 g_i(t)\hat m_i(t)\geq g_i(t)^2 g i ( t ) m ^ i ( t ) ≥ g i ( t ) 2 . Consequently,
U ˙ ( t ) = ⟨ g t , η ( t ) m ^ t v ^ t ⟩ L ( θ t ) ≥ η ( t ) g j ( t ) m ^ j ( t ) L ( θ t ) v ^ j ( t ) ≥ η ( t ) 1 − e − c 2 t e c 2 t / 2 g j ( t ) 2 L ( θ t ) I j ( t ) . \begin{aligned}
\dot U(t)&=\frac{\left\langle\mathbf g_t,\eta(t)\frac{\hat{\mathbf m}_t}{\sqrt{\hat{\mathbf v}_t}}\right\rangle}{\mathcal{L}(\theta_t)}\\
&\geq\frac{\eta(t)g_j(t)\hat m_j(t)}{\mathcal{L}(\theta_t)\sqrt{\hat v_j(t)}}\\
&\geq\eta(t)\sqrt{1-e^{-c_2t}}\frac{e^{c_2t/2}g_j(t)^2}{\mathcal{L}(\theta_t)\sqrt{I_j(t)}}.
\end{aligned} U ˙ ( t ) = L ( θ t ) ⟨ g t , η ( t ) v ^ t m ^ t ⟩ ≥ L ( θ t ) v ^ j ( t ) η ( t ) g j ( t ) m ^ j ( t ) ≥ η ( t ) 1 − e − c 2 t L ( θ t ) I j ( t ) e c 2 t /2 g j ( t ) 2 .
In particular, U U U is nondecreasing, so U ( t ) ≥ max { 1 , 4 ν } U(t)\geq\max\{1,4\nu\} U ( t ) ≥ max { 1 , 4 ν } for t ≥ 1 / 2 t\geq1/2 t ≥ 1/2 . The gradient lower bound g j ( t ) 2 ≥ k 2 G 2 ( t ) = k 2 L 2 U 2 ν g_j(t)^2\geq k^2G^2(t)=k^2L^2U^{2\nu} g j ( t ) 2 ≥ k 2 G 2 ( t ) = k 2 L 2 U 2 ν and e U = L − 1 e^U=L^{-1} e U = L − 1 therefore give, for some c > 0 c>0 c > 0 ,
d d t ( e U ( t ) U ( t ) 2 ν ) = e U ( t ) U ( t ) 2 ν ( 1 − 2 ν U ( t ) ) U ˙ ( t ) ≥ c η ( t ) e c 2 t / 2 I j ( t ) . \frac{d}{dt}\left(\frac{e^{U(t)}}{U(t)^{2\nu}}\right)=\frac{e^{U(t)}}{U(t)^{2\nu}}\left(1-\frac{2\nu}{U(t)}\right)\dot U(t)\geq c\eta(t)\frac{e^{c_2t/2}}{\sqrt{I_j(t)}}. d t d ( U ( t ) 2 ν e U ( t ) ) = U ( t ) 2 ν e U ( t ) ( 1 − U ( t ) 2 ν ) U ˙ ( t ) ≥ cη ( t ) I j ( t ) e c 2 t /2 .
For t ≥ 1 t\geq1 t ≥ 1 , integrate over [ t − 1 2 , t ] [t-\frac12,t] [ t − 2 1 , t ] . Since I j I_j I j is nondecreasing and η ( s ) ≍ η ( t ) \eta(s)\asymp\eta(t) η ( s ) ≍ η ( t ) on this interval, for some c > 0 c>0 c > 0 , we obtain
e U ( t ) U ( t ) 2 ν ≥ c η ( t ) e c 2 t / 2 I j ( t ) . (70) \frac{e^{U(t)}}{U(t)^{2\nu}}\geq c\eta(t)\frac{e^{c_2t/2}}{\sqrt{I_j(t)}}. \tag*{(70)} U ( t ) 2 ν e U ( t ) ≥ cη ( t ) I j ( t ) e c 2 t /2 . (70)
The second-moment equation and the gradient upper bound give
I ˙ j ( t ) = c 2 e c 2 t g j ( t ) 2 ≤ c 2 K 2 e c 2 t U ( t ) 2 ν e − 2 U ( t ) . \dot I_j(t)=c_2e^{c_2t}g_j(t)^2\leq c_2K^2e^{c_2t}U(t)^{2\nu}e^{-2U(t)}. I ˙ j ( t ) = c 2 e c 2 t g j ( t ) 2 ≤ c 2 K 2 e c 2 t U ( t ) 2 ν e − 2 U ( t ) .
Combining this with (70) shows that there exists C > 0 C>0 C > 0 such that
I ˙ j ( t ) I j ( t ) ≤ C ( 1 + t ) − 2 α U ( t ) − 2 ν , t ≥ 1. (71) \frac{\dot I_j(t)}{I_j(t)}\leq C(1+t)^{-2\alpha}U(t)^{-2\nu},\qquad t\geq1. \tag*{(71)} I j ( t ) I ˙ j ( t ) ≤ C ( 1 + t ) − 2 α U ( t ) − 2 ν , t ≥ 1. (71)
If L = 1 L=1 L = 1 , then ν = 0 \nu=0 ν = 0 and α > 1 / 2 \alpha>1/2 α > 1/2 , so this estimate and its integral prove the logarithmic growth and multiplicative bounds.
Suppose L > 1 L>1 L > 1 , and take logarithms in (70) . For some C > 0 C>0 C > 0 , the bounds U ≥ 1 U\geq1 U ≥ 1 and α log ( 1 + t ) ≥ − c 2 t / 4 − C \alpha\log(1+t)\geq-c_2t/4-C α log ( 1 + t ) ≥ − c 2 t /4 − C give, for all t ≥ 1 t\geq1 t ≥ 1 ,
U ( t ) ≥ c 2 4 t − 1 2 log I j ( t ) − C . U(t)\geq\frac{c_2}{4}t-\frac{1}{2}\log I_j(t)-C. U ( t ) ≥ 4 c 2 t − 2 1 log I j ( t ) − C .
Consequently, as long as I j ( t ) ≤ 2 I j ( 1 ) I_j(t)\leq2I_j(1) I j ( t ) ≤ 2 I j ( 1 ) , a sufficiently large B : = log ( 1 / I j ( 1 ) ) B:=\log(1/I_j(1)) B := log ( 1/ I j ( 1 )) ensures for some c > 0 c>0 c > 0
U ( t ) ≥ c ( t + B ) . (72) U(t)\geq c(t+B). \tag*{(72)} U ( t ) ≥ c ( t + B ) . (72)
On any such interval, (71) implies for some C > 0 C>0 C > 0
log I j ( t ) I j ( 1 ) ≤ C ∫ 1 ∞ ( 1 + s ) − 2 α ( s + B ) − 2 ν d s = : C D ( B ) . (73) \log\frac{I_j(t)}{I_j(1)}\leq C\int_1^\infty(1+s)^{-2\alpha}(s+B)^{-2\nu}\,ds=:CD(B). \tag*{(73)} log I j ( 1 ) I j ( t ) ≤ C ∫ 1 ∞ ( 1 + s ) − 2 α ( s + B ) − 2 ν d s =: C D ( B ) . (73)
Since ν > 0 \nu>0 ν > 0 , the integrand tends pointwise to zero as B → ∞ B\to\infty B → ∞ . For B ≥ 1 B\geq1 B ≥ 1 , it is bounded by the integrable function ( 1 + s ) − 2 α (1+s)^{-2\alpha} ( 1 + s ) − 2 α . Thus D ( B ) ⟶ B → ∞ 0 D(B)\mathop{\longrightarrow}_{B\to\infty}0 D ( B ) ⟶ B → ∞ 0 .
Choose B 0 ≥ 1 B_0\geq1 B 0 ≥ 1 large enough that the preceding lower bound on U U U holds and C D ( B 0 ) < log 2 CD(B_0)<\log2 C D ( B 0 ) < log 2 , and set ε 0 : = e − B 0 \varepsilon_0:=e^{-B_0} ε 0 := e − B 0 . A first time with I j ( t ) = 2 I j ( 1 ) I_j(t)=2I_j(1) I j ( t ) = 2 I j ( 1 ) would contradict 73. Hence (72) , (73) hold for all t ≥ 1 t\geq1 t ≥ 1 , and
1 ≤ ω j I j ( 1 ) ≤ e C D ( B ) = 1 + o ( 1 ) . 1\leq\frac{\omega_j}{I_j(1)}\leq e^{CD(B)}=1+o(1). 1 ≤ I j ( 1 ) ω j ≤ e C D ( B ) = 1 + o ( 1 ) .
Substituting U ( t ) ≥ c ( t + B ) U(t)\geq c(t+B) U ( t ) ≥ c ( t + B ) into (71) also gives for some C > 0 C>0 C > 0
I ˙ j ( t ) I j ( t ) ≤ C ( 1 + t ) − 2 ζ , t ≥ 1. \frac{\dot I_j(t)}{I_j(t)}\leq C(1+t)^{-2\zeta},\qquad t\geq1. I j ( t ) I ˙ j ( t ) ≤ C ( 1 + t ) − 2 ζ , t ≥ 1.
Finally, I j I_j I j is nondecreasing. Thus, for some C > 0 C>0 C > 0 , its finite limit ω j \omega_j ω j satisfies
0 ≤ ω j − I j ( t ) = ∫ t ∞ I ˙ j ( s ) d s ≤ C ω j ∫ t ∞ ( 1 + s ) − 2 ζ d s ≤ C I j ( 1 ) ( 1 + t ) 1 − 2 ζ , t ≥ 1 , \begin{aligned}
0\leq\omega_j-I_j(t)=\int_t^\infty\dot{I}_j(s)\,\mathrm{d}s \\
&\leq C\omega_j\int_t^\infty(1+s)^{-2\zeta}\,\mathrm{d}s \\
&\leq CI_j(1)(1+t)^{1-2\zeta},\qquad t\geq1,
\end{aligned} 0 ≤ ω j − I j ( t ) = ∫ t ∞ I ˙ j ( s ) d s ≤ C ω j ∫ t ∞ ( 1 + s ) − 2 ζ d s ≤ C I j ( 1 ) ( 1 + t ) 1 − 2 ζ , t ≥ 1 ,
which proves the remaining assertion.
The following lemma softens the loss decay assumption to a finite-time low-loss condition.
Lemma C.15 (A finite low-loss window implies loss decay). Assume (T1). Define
K A : = c 1 c 2 ( 2 c 1 − c 2 ) , R p r e : = ∥ θ 0 ∥ 2 + p K A ∫ 0 t 0 η ( s ) d s , K_A:=\frac{c_1}{\sqrt{c_2(2c_1-c_2)}},\qquad R_{\mathrm{pre}}:=\|\theta_0\|_2+\sqrt{p}K_A\int_0^{t_0}\eta(s)\,\mathrm{d}s, K A := c 2 ( 2 c 1 − c 2 ) c 1 , R pre := ∥ θ 0 ∥ 2 + p K A ∫ 0 t 0 η ( s ) d s , and
M h = sup i ∈ [ m ] sup ∥ u ∥ 2 = 1 sup h ∈ ∂ q i ( u ) ∥ h ∥ 1 < ∞ , M_h=\sup_{i\in[m]}\sup_{\|u\|_2=1}\sup_{h\in\partial q_i(u)}\|h\|_1<\infty, M h = i ∈ [ m ] sup ∥ u ∥ 2 = 1 sup h ∈ ∂ q i ( u ) sup ∥ h ∥ 1 < ∞ , G γ 0 : = m γ 0 1 L − 1 M h sup q > 0 q 1 − 1 L φ ′ ( q ) e − φ ( q ) < ∞ , (74) G_{\gamma_0}:=m\gamma_0^{\frac{1}{L}-1}M_h\sup_{q>0}q^{1-\frac{1}{L}}\varphi'(q)e^{-\varphi(q)}<\infty, \tag*{(74)} G γ 0 := m γ 0 L 1 − 1 M h q > 0 sup q 1 − L 1 φ ′ ( q ) e − φ ( q ) < ∞ , (74) G p r e : = sup ∥ θ ∥ 2 ≤ R p r e sup ξ ∈ ∂ L ( θ ) ∥ ξ ∥ 1 < ∞ . G_{\mathrm{pre}}:=\sup_{\|\theta\|_2\leq R_{\mathrm{pre}}}\sup_{\xi\in\partial\mathcal{L}(\theta)}\|\xi\|_1<\infty. G pre := ∥ θ ∥ 2 ≤ R pre sup ξ ∈ ∂ L ( θ ) sup ∥ ξ ∥ 1 < ∞. Choose t 1 > max { t 0 , t A } t_1>\max\{t_0,t_A\} t 1 > max { t 0 , t A } sufficiently large that
e − c 1 ( t 1 − t 0 ) ( 1 − e − c 1 t 0 ) ≤ 1 2 . (75) e^{-c_1(t_1-t_0)}(1-e^{-c_1t_0})\leq\frac{1}{2}. \tag*{(75)} e − c 1 ( t 1 − t 0 ) ( 1 − e − c 1 t 0 ) ≤ 2 1 . (75) Suppose that t 2 > t 1 t_2>t_1 t 2 > t 1 satisfies
t 2 − t 1 > 1 μ 0 log max { 1 , η ^ ( t 1 ) ( 1 2 G p r e + G γ 0 ) ℓ ( 0 ) c 2 ( 2 c 1 − c 2 ) } (76) t_2-t_1>\frac{1}{\mu_0}\log\max\left\{1,\frac{\widehat{\eta}(t_1)\left(\frac{1}{2}G_{\mathrm{pre}}+G_{\gamma_0}\right)}{\ell(0)\sqrt{c_2(2c_1-c_2)}}\right\} \tag*{(76)} t 2 − t 1 > μ 0 1 log max { 1 , ℓ ( 0 ) c 2 ( 2 c 1 − c 2 ) η ( t 1 ) ( 2 1 G pre + G γ 0 ) } (76) and
L ( θ t ) ≤ 1 2 ℓ ( 0 ) ( t ∈ [ t 1 , t 2 ] ) , (77) \mathcal{L}(\theta_t)\leq\frac{1}{2}\ell(0)\qquad(t\in[t_1,t_2]), \tag*{(77)} L ( θ t ) ≤ 2 1 ℓ ( 0 ) ( t ∈ [ t 1 , t 2 ]) , (77) then
L A ( t 2 ) < ℓ ( 0 ) , (78) \mathcal{L}_A(t_2)<\ell(0), \tag*{(78)} L A ( t 2 ) < ℓ ( 0 ) , (78) and this implies
L ( θ t ) → t → ∞ 0. (79) \mathcal{L}(\theta_t)\xrightarrow[t\to\infty]{}0. \tag*{(79)} L ( θ t ) t → ∞ 0. (79)
Proof. Lemma B.2 gives ∥ θ ˙ t ∥ 2 ≤ p K A η ( t ) \|\dot{\theta}_t\|_2\leq\sqrt{p}K_A\eta(t) ∥ θ ˙ t ∥ 2 ≤ p K A η ( t ) . Hence ∥ θ t ∥ 2 ≤ R p r e \|\theta_t\|_2\leq R_{\mathrm{pre}} ∥ θ t ∥ 2 ≤ R pre for 0 ≤ t ≤ t 0 0\leq t\leq t_0 0 ≤ t ≤ t 0 , and local Lipschitzness of L \mathcal{L} L gives ∥ g t ∥ 1 ≤ G p r e < ∞ \|g_t\|_1\leq G_{\mathrm{pre}}<\infty ∥ g t ∥ 1 ≤ G pre < ∞ there. Splitting the momentum integral at t 0 t_0 t 0 and using (74) and (75) , we obtain
∥ m t 1 ∥ 1 ≤ e − c 1 ( t 1 − t 0 ) ( 1 − e − c 1 t 0 ) G p r e + G γ 0 ≤ 1 2 G p r e + G γ 0 . \|\mathbf{m}_{t_1}\|_1\leq e^{-c_1(t_1-t_0)}(1-e^{-c_1t_0})G_{\mathrm{pre}}+G_{\gamma_0}\leq\frac{1}{2}G_{\mathrm{pre}}+G_{\gamma_0}. ∥ m t 1 ∥ 1 ≤ e − c 1 ( t 1 − t 0 ) ( 1 − e − c 1 t 0 ) G pre + G γ 0 ≤ 2 1 G pre + G γ 0 .
Lemma B.2 gives ∣ m j ( t ) ∣ ≤ K A v j ( t ) |m_j(t)|\leq K_A\sqrt{v_j(t)} ∣ m j ( t ) ∣ ≤ K A v j ( t ) , and therefore H ( t 1 ) ≤ K A ∥ m t 1 ∥ 1 H(t_1)\leq K_A\|\mathbf{m}_{t_1}\|_1 H ( t 1 ) ≤ K A ∥ m t 1 ∥ 1 .
Since L A ≥ L \mathcal{L}_A\geq\mathcal{L} L A ≥ L and μ ( t ) ≥ μ 0 \mu(t)\geq\mu_0 μ ( t ) ≥ μ 0 on [ t 1 , t 2 ] [t_1,t_2] [ t 1 , t 2 ] , (43) and (77) give, for almost every t ∈ [ t 1 , t 2 ] t\in[t_1,t_2] t ∈ [ t 1 , t 2 ] ,
L ˙ A ( t ) ≤ − μ 0 ( L A ( t ) − L ( θ t ) ) ≤ − μ 0 ( L A ( t ) − 1 2 ℓ ( 0 ) ) . \dot{\mathcal{L}}_A(t)\leq-\mu_0\bigl(\mathcal{L}_A(t)-\mathcal{L}(\theta_t)\bigr)\leq-\mu_0\left(\mathcal{L}_A(t)-\frac{1}{2}\ell(0)\right). L ˙ A ( t ) ≤ − μ 0 ( L A ( t ) − L ( θ t ) ) ≤ − μ 0 ( L A ( t ) − 2 1 ℓ ( 0 ) ) .
For almost every t ∈ [ t 1 , t 2 ] t\in[t_1,t_2] t ∈ [ t 1 , t 2 ] ,
d d t [ e μ 0 ( t − t 1 ) ( L A ( t ) − 1 2 ℓ ( 0 ) ) ] = e μ 0 ( t − t 1 ) [ L ˙ A ( t ) + μ 0 ( L A ( t ) − 1 2 ℓ ( 0 ) ) ] ≤ 0. \frac{\mathrm{d}}{\mathrm{d}t}\left[e^{\mu_0(t-t_1)}\left(\mathcal{L}_A(t)-\frac{1}{2}\ell(0)\right)\right]
=e^{\mu_0(t-t_1)}\left[\dot{\mathcal{L}}_A(t)+\mu_0\left(\mathcal{L}_A(t)-\frac{1}{2}\ell(0)\right)\right]\leq0. d t d [ e μ 0 ( t − t 1 ) ( L A ( t ) − 2 1 ℓ ( 0 ) ) ] = e μ 0 ( t − t 1 ) [ L ˙ A ( t ) + μ 0 ( L A ( t ) − 2 1 ℓ ( 0 ) ) ] ≤ 0.
By absolute continuity, integrating over [ t 1 , t 2 ] [t_1,t_2] [ t 1 , t 2 ] gives
e μ 0 ( t 2 − t 1 ) ( L A ( t 2 ) − 1 2 ℓ ( 0 ) ) ≤ L A ( t 1 ) − 1 2 ℓ ( 0 ) . e^{\mu_0(t_2-t_1)}\left(\mathcal{L}_A(t_2)-\frac{1}{2}\ell(0)\right)\leq\mathcal{L}_A(t_1)-\frac{1}{2}\ell(0). e μ 0 ( t 2 − t 1 ) ( L A ( t 2 ) − 2 1 ℓ ( 0 ) ) ≤ L A ( t 1 ) − 2 1 ℓ ( 0 ) .
and hence
L A ( t 2 ) ≤ 1 2 ℓ ( 0 ) + e − μ 0 ( t 2 − t 1 ) ( L A ( t 1 ) − 1 2 ℓ ( 0 ) ) . \mathcal{L}_A(t_2)\leq\frac{1}{2}\ell(0)+e^{-\mu_0(t_2-t_1)}\left(\mathcal{L}_A(t_1)-\frac{1}{2}\ell(0)\right). L A ( t 2 ) ≤ 2 1 ℓ ( 0 ) + e − μ 0 ( t 2 − t 1 ) ( L A ( t 1 ) − 2 1 ℓ ( 0 ) ) .
By the definition of L A \mathcal{L}_A L A and the assumption L ( θ t 1 ) ≤ ℓ ( 0 ) / 2 \mathcal{L}(\theta_{t_1})\leq\ell(0)/2 L ( θ t 1 ) ≤ ℓ ( 0 ) /2 ,
L A ( t 1 ) − 1 2 ℓ ( 0 ) = L ( θ t 1 ) − 1 2 ℓ ( 0 ) + η ^ ( t 1 ) 2 c 1 H ( t 1 ) ≤ η ^ ( t 1 ) 2 c 1 H ( t 1 ) . \mathcal{L}_A(t_1)-\frac{1}{2}\ell(0)=\mathcal{L}(\theta_{t_1})-\frac{1}{2}\ell(0)+\frac{\widehat{\eta}(t_1)}{2c_1}H(t_1)\leq\frac{\widehat{\eta}(t_1)}{2c_1}H(t_1). L A ( t 1 ) − 2 1 ℓ ( 0 ) = L ( θ t 1 ) − 2 1 ℓ ( 0 ) + 2 c 1 η ( t 1 ) H ( t 1 ) ≤ 2 c 1 η ( t 1 ) H ( t 1 ) .
Furthermore, the preceding momentum bound gives
H ( t 1 ) ≤ K A ∥ m t 1 ∥ 1 ≤ c 1 c 2 ( 2 c 1 − c 2 ) ( 1 2 G p r e + G γ 0 ) . H(t_1)\leq K_A\lVert m_{t_1}\rVert_1\leq\frac{c_1}{\sqrt{c_2(2c_1-c_2)}}\left(\frac{1}{2}G_{\mathrm{pre}}+G_{\gamma_0}\right). H ( t 1 ) ≤ K A ∥ m t 1 ∥ 1 ≤ c 2 ( 2 c 1 − c 2 ) c 1 ( 2 1 G pre + G γ 0 ) .
Substituting these bounds and using (76) yields
L A ( t 2 ) ≤ 1 2 ℓ ( 0 ) + e − μ 0 ( t 2 − t 1 ) η ^ ( t 1 ) ( 1 2 G p r e + G γ 0 ) 2 c 2 ( 2 c 1 − c 2 ) < ℓ ( 0 ) . \mathcal{L}_A(t_2)\leq\frac{1}{2}\ell(0)+e^{-\mu_0(t_2-t_1)}\frac{\widehat{\eta}(t_1)\left(\frac{1}{2}G_{\mathrm{pre}}+G_{\gamma_0}\right)}{2\sqrt{c_2(2c_1-c_2)}}<\ell(0). L A ( t 2 ) ≤ 2 1 ℓ ( 0 ) + e − μ 0 ( t 2 − t 1 ) 2 c 2 ( 2 c 1 − c 2 ) η ( t 1 ) ( 2 1 G pre + G γ 0 ) < ℓ ( 0 ) .
By absolute continuity and (43) , L A \mathcal{L}_A L A is nonincreasing on [ t 2 , ∞ ) [t_2,\infty) [ t 2 , ∞ ) , with a limit A ∞ ≥ 0 A_\infty\geq0 A ∞ ≥ 0 . Suppose that A ∞ > 0 A_\infty>0 A ∞ > 0 . Since ℓ ( q i ( t ) ) ≤ L A ( t 2 ) < ℓ ( 0 ) \ell(q_i(t))\leq\mathcal{L}_A(t_2)<\ell(0) ℓ ( q i ( t )) ≤ L A ( t 2 ) < ℓ ( 0 ) , convexity gives
q i ( t ) φ ′ ( q i ( t ) ) ≥ φ ( q i ( t ) ) − φ ( 0 ) ≥ log ℓ ( 0 ) L A ( t 2 ) . q_i(t)\varphi'(q_i(t))\geq\varphi(q_i(t))-\varphi(0)\geq\log\frac{\ell(0)}{\mathcal{L}_A(t_2)}. q i ( t ) φ ′ ( q i ( t )) ≥ φ ( q i ( t )) − φ ( 0 ) ≥ log L A ( t 2 ) ℓ ( 0 ) .
Set k : = min { 2 , L log ( ℓ ( 0 ) / L A ( t 2 ) ) } > 0 k:=\min\{2,L\log(\ell(0)/\mathcal{L}_A(t_2))\}>0 k := min { 2 , L log ( ℓ ( 0 ) / L A ( t 2 ))} > 0 . The Euler identity yields
− ⟨ θ t , g t ⟩ = L ∑ i = 1 m q i ( t ) φ ′ ( q i ( t ) ) e − φ ( q i ( t ) ) ≥ k L ( θ t ) . -\langle\theta_t,g_t\rangle=L\sum_{i=1}^{m}q_i(t)\varphi'(q_i(t))e^{-\varphi(q_i(t))}\geq k\mathcal{L}(\theta_t). − ⟨ θ t , g t ⟩ = L i = 1 ∑ m q i ( t ) φ ′ ( q i ( t )) e − φ ( q i ( t )) ≥ k L ( θ t ) .
Thus (44) yields
( d d t + c 1 ) ( − ⟨ θ t , m t ⟩ ) ≥ c 1 k L ( θ t ) + η ^ ( t ) H ( t ) ≥ c 1 k L A ( t ) ≥ c 1 k A ∞ . \left(\frac{d}{dt}+c_1\right)(-\langle\theta_t,m_t\rangle)\geq c_1k\mathcal{L}(\theta_t)+\widehat{\eta}(t)H(t)\geq c_1k\mathcal{L}_A(t)\geq c_1kA_\infty. ( d t d + c 1 ) ( − ⟨ θ t , m t ⟩) ≥ c 1 k L ( θ t ) + η ( t ) H ( t ) ≥ c 1 k L A ( t ) ≥ c 1 k A ∞ .
Consequently, multiplying by e c 1 ( t − t 2 ) e^{c_1(t-t_2)} e c 1 ( t − t 2 ) and integrating implies − ⟨ θ t , m t ⟩ ≥ b : = k A ∞ / 2 > 0 -\langle\theta_t,m_t\rangle\geq b:=kA_\infty/2>0 − ⟨ θ t , m t ⟩ ≥ b := k A ∞ /2 > 0 eventually. The positive-margin condition implies that there exists c > 0 c>0 c > 0 such that
L ( θ t ) ≲ e − c ∥ θ t ∥ 2 2 , ∥ θ t ∥ 2 2 L ( θ t ) ≲ 1. \mathcal{L}(\theta_t)\lesssim e^{-c\lVert\theta_t\rVert_2^2},\qquad\lVert\theta_t\rVert_2^2\mathcal{L}(\theta_t)\lesssim1. L ( θ t ) ≲ e − c ∥ θ t ∥ 2 2 , ∥ θ t ∥ 2 2 L ( θ t ) ≲ 1.
Also, (74) and the second-moment formula, with finite v j ( t 2 ) v_j(t_2) v j ( t 2 ) , give v j ( t ) ≲ 1 \sqrt{v_j(t)}\lesssim1 v j ( t ) ≲ 1 , and hence ∥ m t ∥ 2 2 ≲ H ( t ) \lVert m_t\rVert_2^2\lesssim H(t) ∥ m t ∥ 2 2 ≲ H ( t ) . Cauchy–Schwarz therefore implies, for all sufficiently large t t t ,
b 2 L ( θ t ) ≤ ∥ θ t ∥ 2 2 ∥ m t ∥ 2 2 L ( θ t ) ≲ H ( t ) . b^2\mathcal{L}(\theta_t)\leq\lVert\theta_t\rVert_2^2\lVert m_t\rVert_2^2\mathcal{L}(\theta_t)\lesssim H(t). b 2 L ( θ t ) ≤ ∥ θ t ∥ 2 2 ∥ m t ∥ 2 2 L ( θ t ) ≲ H ( t ) .
Hence L ( θ t ) ≤ C H ( t ) \mathcal{L}(\theta_t)\leq CH(t) L ( θ t ) ≤ C H ( t ) for some C > 0 C>0 C > 0 and L A ( t ) ≤ ( C + η ^ ( t ) / ( 2 c 1 ) ) H ( t ) \mathcal{L}_A(t)\leq(C+\widehat{\eta}(t)/(2c_1))H(t) L A ( t ) ≤ ( C + η ( t ) / ( 2 c 1 )) H ( t ) . By (43) and recalling η ^ H / ( 2 c 1 ) = L A − L \widehat{\eta}H/(2c_1)=\mathcal{L}_A-\mathcal{L} η H / ( 2 c 1 ) = L A − L , almost everywhere on a tail,
− L ˙ A ( t ) ≥ μ 0 2 c 1 η ^ ( t ) H ( t ) ≥ μ 0 A ∞ 2 c 1 η ^ ( t ) C + η ^ ( t ) / ( 2 c 1 ) ≳ η ^ ( t ) 1 + η ^ ( t ) . -\dot{\mathcal{L}}_A(t)\geq\frac{\mu_0}{2c_1}\widehat{\eta}(t)H(t)\geq\frac{\mu_0A_\infty}{2c_1}\frac{\widehat{\eta}(t)}{C+\widehat{\eta}(t)/(2c_1)}\gtrsim\frac{\widehat{\eta}(t)}{1+\widehat{\eta}(t)}. − L ˙ A ( t ) ≥ 2 c 1 μ 0 η ( t ) H ( t ) ≥ 2 c 1 μ 0 A ∞ C + η ( t ) / ( 2 c 1 ) η ( t ) ≳ 1 + η ( t ) η ( t ) .
The right-hand side is comparable to ( 1 + t ) min { α , 0 } (1+t)^{\min\{\alpha,0\}} ( 1 + t ) m i n { α , 0 } , whose integral diverges because α > − 1 \alpha>-1 α > − 1 , contradicting L A ≥ 0 \mathcal{L}_A\geq0 L A ≥ 0 . Thus A ∞ = 0 A_\infty=0 A ∞ = 0 , and 0 < L ( θ t ) ≤ L A ( t ) 0<\mathcal{L}(\theta_t)\leq\mathcal{L}_A(t) 0 < L ( θ t ) ≤ L A ( t ) proves the remaining claim.
Adam - Fast Regime
Width-1 Construction
Recall the width-1 model used for the construction:
∀ θ = ( u , b ) = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 : f ( x ; θ ) : = P ( b ) ⟨ u , x ⟩ , P ( b ) : = ∏ j = 1 L − 1 b j . (80) \forall\theta= (\mathbf{u},\mathbf{b}) = (u_1,u_2,b_1,\ldots,b_{L-1}) \in\mathbb{R}^{L+1}:\qquad f(x;\theta) := P(\mathbf{b})\langle\mathbf{u},x\rangle,\qquad P(\mathbf{b}) := \prod_{j=1}^{L-1} b_j. \tag*{(80)} ∀ θ = ( u , b ) = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 : f ( x ; θ ) := P ( b ) ⟨ u , x ⟩ , P ( b ) := j = 1 ∏ L − 1 b j . (80)
And the following dataset, for M > B > 1 , Λ > 0 M>B>1,\Lambda>0 M > B > 1 , Λ > 0 ,
x 1 = 1 Λ ( B , 1 ) , x 2 = 1 Λ ( M , 0 ) , y 1 = y 2 = 1 (81) x_1=\frac{1}{\Lambda}(B,1),\qquad x_2=\frac{1}{\Lambda}(M,0),\qquad y_1=y_2=1 \tag*{(81)} x 1 = Λ 1 ( B , 1 ) , x 2 = Λ 1 ( M , 0 ) , y 1 = y 2 = 1 (81)
Where x 1 x_1 x 1 is the unique minimum margin point for all trajectories, and x 2 x_2 x 2 is the point supplying large early-time gradients.
We use the exponential loss L ( θ ) = ∑ i ∈ [ 2 ] e − y i f ( x i ; θ ) \mathcal{L}(\theta)=\sum_{i\in[2]}e^{-y_i f(x_i;\theta)} L ( θ ) = ∑ i ∈ [ 2 ] e − y i f ( x i ; θ ) .
Theorem D.1 (Regularizer is constant). Let L ≥ 1 , α > 1 L − 1 2 L\geq1,\alpha>\frac{1}{L}-\frac{1}{2} L ≥ 1 , α > L 1 − 2 1 . If R ( θ ) : R L + 1 → R R(\theta):\mathbb{R}^{L+1}\to\mathbb{R} R ( θ ) : R L + 1 → R is an implicit margin regularizer for Adam on the model (80) with η ( t ) = η 0 ( 1 + t ) α \eta(t)=\eta_0(1+t)^\alpha η ( t ) = η 0 ( 1 + t ) α and the exponential loss, then R ( θ ) R(\theta) R ( θ ) is constant on R L + 1 \mathbb{R}^{L+1} R L + 1 .
Theorem D.1 follows from proving constancy of R R R on the following hypersurface (and its symmetries), defined for B > 1 , Λ > 0 B>1,\Lambda>0 B > 1 , Λ > 0 :
Σ B , Λ = { θ = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 ∣ 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] : b j > 0 , f ( x 1 ; θ ) = 1 } . (82) \Sigma_{B,\Lambda}=\left\{\theta=(u_1,u_2,b_1,\ldots,b_{L-1})\in\mathbb{R}^{L+1}\mid0<u_1<u_2,\ \forall j\in[L-1]:b_j>0,\ f(x_1;\theta)=1\right\}. \tag*{(82)} Σ B , Λ = { θ = ( u 1 , u 2 , b 1 , … , b L − 1 ) ∈ R L + 1 ∣ 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] : b j > 0 , f ( x 1 ; θ ) = 1 } . (82)
Lemma D.2 (Constancy on Hypersurfaces). For any B > 1 , Λ > 0 B>1,\Lambda>0 B > 1 , Λ > 0 , R R R is constant on Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ .
First we prove Theorem D.1 assuming Lemma D.2.
Proof of Theorem D.1. We prove R R R is constant on Θ + = { 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] : b j > 0 } \Theta_+=\{0<u_1<u_2,\ \forall j\in[L-1]:b_j>0\} Θ + = { 0 < u 1 < u 2 , ∀ j ∈ [ L − 1 ] : b j > 0 } . Other sign patterns and order relations between u 1 , u 2 u_1,u_2 u 1 , u 2 follow from data and initialization symmetries; see Corollary D.14. This allows showing that R R R is constant on each such open domain; continuity of R R R then forces constancy on the whole of R L + 1 \mathbb{R}^{L+1} R L + 1 .
Let θ ( 1 ) = ( u ( 1 ) , b ( 1 ) ) , θ ( 2 ) = ( u ( 2 ) , b ( 2 ) ) ∈ Θ + \theta^{(1)}=(\mathbf{u}^{(1)},\mathbf{b}^{(1)}),\theta^{(2)}=(\mathbf{u}^{(2)},\mathbf{b}^{(2)})\in\Theta_+ θ ( 1 ) = ( u ( 1 ) , b ( 1 ) ) , θ ( 2 ) = ( u ( 2 ) , b ( 2 ) ) ∈ Θ + . For any Λ > 0 \Lambda>0 Λ > 0 there exist unique B ( 1 ) , B ( 2 ) B^{(1)},B^{(2)} B ( 1 ) , B ( 2 ) with
f ( Λ − 1 ( B ( 1 ) , 1 ) ; θ ( 1 ) ) = P ( b ( 1 ) ) ⟨ u ( 1 ) , Λ − 1 ( B ( 1 ) , 1 ) ⟩ = 1 = P ( b ( 2 ) ) ⟨ u ( 2 ) , Λ − 1 ( B ( 2 ) , 1 ) ⟩ = f ( Λ − 1 ( B ( 2 ) , 1 ) ; θ ( 2 ) ) . f\left(\Lambda^{-1}(B^{(1)},1);\theta^{(1)}\right)=P(\mathbf{b}^{(1)})\left\langle\mathbf{u}^{(1)},\Lambda^{-1}(B^{(1)},1)\right\rangle=1=P(\mathbf{b}^{(2)})\left\langle\mathbf{u}^{(2)},\Lambda^{-1}(B^{(2)},1)\right\rangle=f\left(\Lambda^{-1}(B^{(2)},1);\theta^{(2)}\right). f ( Λ − 1 ( B ( 1 ) , 1 ) ; θ ( 1 ) ) = P ( b ( 1 ) ) ⟨ u ( 1 ) , Λ − 1 ( B ( 1 ) , 1 ) ⟩ = 1 = P ( b ( 2 ) ) ⟨ u ( 2 ) , Λ − 1 ( B ( 2 ) , 1 ) ⟩ = f ( Λ − 1 ( B ( 2 ) , 1 ) ; θ ( 2 ) ) .
In fact,
B ( i ) = 1 u 1 ( i ) ( Λ P ( b ( i ) ) − u 2 ( i ) ) . B^{(i)}=\frac{1}{u_1^{(i)}}\left(\frac{\Lambda}{P(\mathbf{b}^{(i)})}-u_2^{(i)}\right). B ( i ) = u 1 ( i ) 1 ( P ( b ( i ) ) Λ − u 2 ( i ) ) .
Fix large enough Λ > 0 \Lambda>0 Λ > 0 with B ( 1 ) , B ( 2 ) > 1 B^{(1)},B^{(2)}>1 B ( 1 ) , B ( 2 ) > 1 . By definition θ ( i ) ∈ Σ B ( i ) , Λ \theta^{(i)}\in\Sigma_{B^{(i)},\Lambda} θ ( i ) ∈ Σ B ( i ) , Λ , and by Lemma D.2, R R R is constant on Σ B ( 1 ) , Λ \Sigma_{B^{(1)},\Lambda} Σ B ( 1 ) , Λ and on Σ B ( 2 ) , Λ \Sigma_{B^{(2)},\Lambda} Σ B ( 2 ) , Λ . The point ( ( 0 , Λ ) , 1 ) ((0,\Lambda),1) (( 0 , Λ ) , 1 ) is a boundary point of both Σ B ( 1 ) , Λ , Σ B ( 2 ) , Λ \Sigma_{B^{(1)},\Lambda},\Sigma_{B^{(2)},\Lambda} Σ B ( 1 ) , Λ , Σ B ( 2 ) , Λ , so by continuity of R R R ,
R ( θ ( 1 ) ) = R ( ( 0 , Λ ) , 1 ) = R ( θ ( 2 ) ) . R\left(\theta^{(1)}\right)=R((0,\Lambda),1)=R\left(\theta^{(2)}\right). R ( θ ( 1 ) ) = R (( 0 , Λ ) , 1 ) = R ( θ ( 2 ) ) .
In turn, Lemma D.2 follows from the following major object of the proof. Compared to Proposition 5.2, we strengthen the result by showing that a single dataset suffices for any fixed compact target set.
Proposition D.3 (Arbitrary Directional Convergence). For any B > 1 , Λ > 0 B>1,\Lambda>0 B > 1 , Λ > 0 , and nonempty compact K ⊆ Σ B , Λ K\subseteq\Sigma_{B,\Lambda} K ⊆ Σ B , Λ , there exists M ∗ = M ∗ ( B , Λ , K ) M_* = M_*(B,\Lambda,K) M ∗ = M ∗ ( B , Λ , K ) such that, for every M ≥ M ∗ M\geq M_* M ≥ M ∗ , every θ ^ ∈ K \hat{\theta}\in K θ ^ ∈ K is reached on the fixed dataset (81) by a suitable initialization:
θ t q min 1 / L ( θ t ) → t → ∞ θ ^ . \frac{\theta_t}{q_{\min}^{1/L}(\theta_t)}\xrightarrow[t\to\infty]{}\hat{\theta}. q m i n 1/ L ( θ t ) θ t t → ∞ θ ^ . These initializations can be chosen from a compact family on which the margin-normalized limit exists and depends continuously on initialization.
We show how Lemma D.2 follows from Proposition D.3.
Proof of Lemma D.2. The map
( δ , b ) ↦ ( Λ δ P ( b ) ( 1 + B δ ) , Λ P ( b ) ( 1 + B δ ) , b ) , δ ∈ ( 0 , 1 ) , b ∈ ( 0 , ∞ ) L − 1 , (\delta,\mathbf{b}) \mapsto\left(\frac{\Lambda\delta}{P(\mathbf{b})(1+B\delta)},\frac{\Lambda}{P(\mathbf{b})(1+B\delta)},\mathbf{b}\right), \qquad\delta\in(0,1),\ \mathbf{b} \in(0,\infty)^{L-1}, ( δ , b ) ↦ ( P ( b ) ( 1 + B δ ) Λ δ , P ( b ) ( 1 + B δ ) Λ , b ) , δ ∈ ( 0 , 1 ) , b ∈ ( 0 , ∞ ) L − 1 ,
is a smooth parameterization of Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ . Hence any two points can be joined by a piecewise C 1 C^1 C 1 (and therefore absolutely continuous) path ψ : [ 0 , 1 ] → Σ B , Λ \psi:[0,1]\to\Sigma_{B,\Lambda} ψ : [ 0 , 1 ] → Σ B , Λ .
Fix a point ψ ( s ) = ( u , b ) ≠ 0 \psi(s)=(\mathbf{u},\mathbf{b})\ne0 ψ ( s ) = ( u , b ) = 0 on such a path and choose M M M so large that
q 2 ( ψ ( s ) ) = M P ( b ) u 1 Λ > 1 = P ( b ) ( B u 1 + u 2 ) Λ = q 1 ( ψ ( s ) ) . q_2(\psi(s))=\frac{MP(\mathbf{b})u_1}{\Lambda}>1=\frac{P(\mathbf{b})(Bu_1+u_2)}{\Lambda}=q_1(\psi(s)). q 2 ( ψ ( s )) = Λ M P ( b ) u 1 > 1 = Λ P ( b ) ( B u 1 + u 2 ) = q 1 ( ψ ( s )) .
Proposition D.3 gives an initialization θ 0 \theta_0 θ 0 for which
θ ^ = θ t q min ( θ t ) 1 / L → t → ∞ ψ ( s ) , \widehat{\theta}=\frac{\theta_t}{q_{\min}(\theta_t)^{1/L}}\xrightarrow[t\to\infty]{}\psi(s), θ = q m i n ( θ t ) 1/ L θ t t → ∞ ψ ( s ) ,
and hence also directional convergence θ t ∥ θ t ∥ 2 → ψ ( s ) ∥ ψ ( s ) ∥ 2 \frac{\theta_t}{\lVert\theta_t\rVert_2}\to\frac{\psi(s)}{\lVert\psi(s)\rVert_2} ∥ θ t ∥ 2 θ t → ∥ ψ ( s ) ∥ 2 ψ ( s ) . Thus, since R R R is an implicit margin regularizer, there are k s ∈ ∂ R ( ψ ( s ) ) \mathbf{k}_s\in\partial R(\psi(s)) k s ∈ ∂ R ( ψ ( s )) and λ 1 , λ 2 ≥ 0 \lambda_1,\lambda_2\ge0 λ 1 , λ 2 ≥ 0 such that
k s = λ 1 ∇ q 1 ( ψ ( s ) ) + λ 2 ∇ q 2 ( ψ ( s ) ) , λ i ( q i ( ψ ( s ) ) − 1 ) = 0. \mathbf{k}_s=\lambda_1\nabla q_1(\psi(s))+\lambda_2\nabla q_2(\psi(s)),\qquad\lambda_i(q_i(\psi(s))-1)=0. k s = λ 1 ∇ q 1 ( ψ ( s )) + λ 2 ∇ q 2 ( ψ ( s )) , λ i ( q i ( ψ ( s )) − 1 ) = 0.
Since q 2 ( ψ ( s ) ) > q 1 ( ψ ( s ) ) q_2(\psi(s))>q_1(\psi(s)) q 2 ( ψ ( s )) > q 1 ( ψ ( s )) , it holds that λ 2 = 0 \lambda_2=0 λ 2 = 0 . At almost every s s s where ψ ˙ ( s ) \dot{\psi}(s) ψ ˙ ( s ) exists, differentiation of q 1 ( ψ ( s ) ) ≡ 1 q_1(\psi(s))\equiv1 q 1 ( ψ ( s )) ≡ 1 gives ⟨ ∇ q 1 ( ψ ( s ) ) , ψ ˙ ( s ) ⟩ = 0 \langle\nabla q_1(\psi(s)),\dot{\psi}(s)\rangle=0 ⟨ ∇ q 1 ( ψ ( s )) , ψ ˙ ( s )⟩ = 0 . The chain rule now gives
( R ∘ ψ ) ′ ( s ) = ⟨ k s , ψ ˙ ( s ) ⟩ = 0 (R\circ\psi)'(s)=\langle\mathbf{k}_s,\dot{\psi}(s)\rangle=0 ( R ∘ ψ ) ′ ( s ) = ⟨ k s , ψ ˙ ( s )⟩ = 0
for almost every s s s .
The locally Lipschitz composition R ∘ ψ R\circ\psi R ∘ ψ is absolutely continuous, so it is constant. Path connectedness proves the lemma.
The remainder of the subsection is dedicated to proving Proposition D.3.
Notations and Basic Properties
We denote
g t = ∇ L ( θ t ) , p t = − g t . \mathbf{g}_t=\nabla\mathcal{L}(\theta_t),\qquad\mathbf{p}_t=-\mathbf{g}_t. g t = ∇ L ( θ t ) , p t = − g t .
And we denote g u 1 , g u 2 , g b 1 , … , g b L − 1 g_{u_1},g_{u_2},g_{b_1},\ldots,g_{b_{L-1}} g u 1 , g u 2 , g b 1 , … , g b L − 1 for the scalar gradients (and similarly for p , m , v \mathbf{p},\mathbf{m},\mathbf{v} p , m , v ). We also use a vector notation θ t [ i ] \theta_t[i] θ t [ i ] , i ∈ [ L + 1 ] i\in[L+1] i ∈ [ L + 1 ] , where θ t [ 1 ] = u 1 \theta_t[1]=u_1 θ t [ 1 ] = u 1 , θ t [ 2 ] = u 2 \theta_t[2]=u_2 θ t [ 2 ] = u 2 and θ t [ i ] = b i − 2 \theta_t[i]=b_{i-2} θ t [ i ] = b i − 2 for i ∈ { 3 , … , L + 1 } i\in\{3,\ldots,L+1\} i ∈ { 3 , … , L + 1 } . We use the same notation for g t [ i ] \mathbf{g}_t[i] g t [ i ] , p t [ i ] \mathbf{p}_t[i] p t [ i ] , ω i \omega_i ω i , I i I_i I i (defined below).
We denote
I u 1 ( t ) = c 2 ∫ 0 t e c 2 s p u 1 2 ( s ) d s , I u 2 ( t ) = c 2 ∫ 0 t e c 2 s p u 2 2 ( s ) d s , I b j = c 2 ∫ 0 t e c 2 s p b j 2 ( s ) d s , ( j ∈ [ L − 1 ] ) , I_{u_1}(t)=c_2\int_0^t e^{c_2s}p_{u_1}^2(s)\,ds,\qquad I_{u_2}(t)=c_2\int_0^t e^{c_2s}p_{u_2}^2(s)\,ds,\qquad I_{b_j}=c_2\int_0^t e^{c_2s}p_{b_j}^2(s)\,ds,\quad(j\in[L-1]), I u 1 ( t ) = c 2 ∫ 0 t e c 2 s p u 1 2 ( s ) d s , I u 2 ( t ) = c 2 ∫ 0 t e c 2 s p u 2 2 ( s ) d s , I b j = c 2 ∫ 0 t e c 2 s p b j 2 ( s ) d s , ( j ∈ [ L − 1 ]) ,
and
ω u 1 = lim t → ∞ I u 1 ( t ) , ω u 2 = lim t → ∞ I u 2 ( t ) , ω b j = lim t → ∞ I b j ( t ) ( j ∈ [ L − 1 ] ) . \omega_{u_1}=\lim_{t\to\infty}I_{u_1}(t),\qquad\omega_{u_2}=\lim_{t\to\infty}I_{u_2}(t),\qquad\omega_{b_j}=\lim_{t\to\infty}I_{b_j}(t)\quad(j\in[L-1]). ω u 1 = t → ∞ lim I u 1 ( t ) , ω u 2 = t → ∞ lim I u 2 ( t ) , ω b j = t → ∞ lim I b j ( t ) ( j ∈ [ L − 1 ]) .
Define
ξ 1 = ⟨ x 1 , u t ⟩ , ξ 2 = ⟨ x 2 , u t ⟩ , q 1 = P ( b ) ξ 1 = f ( x 1 ; θ t ) , q 2 = P ( b ) ξ 2 = f ( x 2 ; θ t ) . \xi_1=\langle\mathbf{x}_1,\mathbf{u}_t\rangle,\qquad\xi_2=\langle\mathbf{x}_2,\mathbf{u}_t\rangle,\qquad q_1=P(\mathbf{b})\xi_1=f(\mathbf{x}_1;\theta_t),\qquad q_2=P(\mathbf{b})\xi_2=f(\mathbf{x}_2;\theta_t). ξ 1 = ⟨ x 1 , u t ⟩ , ξ 2 = ⟨ x 2 , u t ⟩ , q 1 = P ( b ) ξ 1 = f ( x 1 ; θ t ) , q 2 = P ( b ) ξ 2 = f ( x 2 ; θ t ) .
The initialization family we consider is parameterized (in addition to B , Λ , M B,\Lambda,M B , Λ , M ) by ρ ≥ 1 \rho\ge1 ρ ≥ 1 , z ∈ [ 0 , 3 log M ] z\in[0,3\log M] z ∈ [ 0 , 3 log M ] , and π = ( π 1 , … , π L − 1 ) ∈ ( 0 , ∞ ) L − 1 \boldsymbol{\pi}=(\pi_1,\ldots,\pi_{L-1})\in(0,\infty)^{L-1} π = ( π 1 , … , π L − 1 ) ∈ ( 0 , ∞ ) L − 1 .
∀ j ∈ [ L − 1 ] : b j ( 0 ) = ρ π j , u ( 0 ) = ρ π 0 ( a , 1 − B a ) , P π : = ∏ j = 1 L − 1 π j , π 0 : = Λ P π , a : = 1 + z / M M . (83) \forall j\in[L-1]:\ b_j(0)=\rho\pi_j,\qquad\mathbf{u}(0)=\rho\pi_0(a,1-Ba),\qquad P_{\boldsymbol{\pi}}:=\prod_{j=1}^{L-1}\pi_j,\qquad\pi_0:=\frac{\Lambda}{P_{\boldsymbol{\pi}}},\qquad a:=\frac{1+z/M}{M}. \tag*{(83)} ∀ j ∈ [ L − 1 ] : b j ( 0 ) = ρ π j , u ( 0 ) = ρ π 0 ( a , 1 − B a ) , P π := j = 1 ∏ L − 1 π j , π 0 := P π Λ , a := M 1 + z / M . (83)
In the L = 1 L = 1 L = 1 case we set as a convention P ( b ) ≡ 1 P(\mathbf{b}) \equiv1 P ( b ) ≡ 1 , P π = 1 P_{\pi} = 1 P π = 1 , π 0 = Λ \pi_0 = \Lambda π 0 = Λ . For M > B + 1 M > B + 1 M > B + 1 , taking
ρ > max { 1 , ( 3 ( B + 1 ) log M M − B − 1 ) 1 / L } = : ρ 0 ( M , B ) \rho> \max\left\{1,\left(\frac{3(B+1)\log M}{M-B-1}\right)^{1/L}\right\} =: \rho_0(M,B) ρ > max { 1 , ( M − B − 1 3 ( B + 1 ) log M ) 1/ L } =: ρ 0 ( M , B )
gives 0 < u 1 ( 0 ) < u 2 ( 0 ) 0 < u_1(0) < u_2(0) 0 < u 1 ( 0 ) < u 2 ( 0 ) for any z , π z,\pi z , π . The parameter z z z controls the initial difference in confidence:
q 1 ( 0 ) = P ( b ( 0 ) ) 1 Λ ⋅ Λ ρ P π ⟨ ( B , 1 ) , ( a , 1 − B a ) ⟩ = ρ L , q_1(0) = P(\mathbf{b}(0))\frac{1}{\Lambda}\cdot\frac{\Lambda\rho}{P_{\pi}}\langle(B,1),(a,1-Ba)\rangle= \rho^L, q 1 ( 0 ) = P ( b ( 0 )) Λ 1 ⋅ P π Λ ρ ⟨( B , 1 ) , ( a , 1 − B a )⟩ = ρ L ,
q 2 ( 0 ) = P ( b ( 0 ) ) 1 Λ Λ ρ P π ⋅ M a = ρ L ( 1 + z ρ L ) = ρ L + z . q_2(0) = P(\mathbf{b}(0))\frac{1}{\Lambda}\frac{\Lambda\rho}{P_{\pi}}\cdot Ma = \rho^L\left(1+\frac{z}{\rho^L}\right)=\rho^L+z. q 2 ( 0 ) = P ( b ( 0 )) Λ 1 P π Λ ρ ⋅ M a = ρ L ( 1 + ρ L z ) = ρ L + z .
It holds with these notations that
L ( θ t ) = e − q 1 + e − q 2 , \mathcal{L}(\theta_t)=e^{-q_1}+e^{-q_2}, L ( θ t ) = e − q 1 + e − q 2 ,
p u 2 = 1 Λ P ( b ) e − q 1 , p_{u_2}=\frac{1}{\Lambda}P(\mathbf{b})e^{-q_1}, p u 2 = Λ 1 P ( b ) e − q 1 ,
p u 1 = B Λ P ( b ) e − q 1 + M Λ P ( b ) e − q 2 = B p u 2 + M Λ P ( b ) e − q 2 , p_{u_1}=\frac{B}{\Lambda}P(\mathbf{b})e^{-q_1}+\frac{M}{\Lambda}P(\mathbf{b})e^{-q_2}=Bp_{u_2}+\frac{M}{\Lambda}P(\mathbf{b})e^{-q_2}, p u 1 = Λ B P ( b ) e − q 1 + Λ M P ( b ) e − q 2 = B p u 2 + Λ M P ( b ) e − q 2 ,
∀ j ∈ [ L − 1 ] : p b j = q 1 b j e − q 1 + q 2 b j e − q 2 . \forall j\in[L-1]:\quad p_{b_j}=\frac{q_1}{b_j}e^{-q_1}+\frac{q_2}{b_j}e^{-q_2}. ∀ j ∈ [ L − 1 ] : p b j = b j q 1 e − q 1 + b j q 2 e − q 2 .
Since all coordinates of p t \mathbf{p}_t p t are strictly positive, all coordinates of θ t \boldsymbol{\theta}_t θ t are strictly increasing, hence so are ξ 1 , ξ 2 , q 1 , q 2 \xi_1,\xi_2,q_1,q_2 ξ 1 , ξ 2 , q 1 , q 2 . It is also simple to show that in fact p i p_i p i are nonincreasing, since (recalling d d t ξ 1 > 0 \frac{d}{dt}\xi_1>0 d t d ξ 1 > 0 , d d t b j > 0 \frac{d}{dt}b_j>0 d t d b j > 0 and q 1 ≥ 1 q_1\geq1 q 1 ≥ 1 )
− d d t log ( P ( b ) e − q 1 ) = q ˙ 1 − d d t log P ( b ) = q ˙ 1 − d d t ( log q 1 − log ξ 1 ) ≥ q ˙ 1 − q ˙ 1 q 1 ≥ 0. (84) -\frac{d}{dt}\log\left(P(\mathbf{b})e^{-q_1}\right)=\dot q_1-\frac{d}{dt}\log P(\mathbf{b})=\dot q_1-\frac{d}{dt}(\log q_1-\log\xi_1)\geq\dot q_1-\frac{\dot q_1}{q_1}\geq0. \tag*{(84)} − d t d log ( P ( b ) e − q 1 ) = q ˙ 1 − d t d log P ( b ) = q ˙ 1 − d t d ( log q 1 − log ξ 1 ) ≥ q ˙ 1 − q 1 q ˙ 1 ≥ 0. (84)
− d d t log q 1 b j e − q 1 = − d d t log q 1 e − q 1 + d d t log b j ≥ q ˙ 1 − q ˙ 1 q 1 ≥ 0. (85) -\frac{d}{dt}\log\frac{q_1}{b_j}e^{-q_1}=-\frac{d}{dt}\log q_1e^{-q_1}+\frac{d}{dt}\log b_j\geq\dot q_1-\frac{\dot q_1}{q_1}\geq0. \tag*{(85)} − d t d log b j q 1 e − q 1 = − d t d log q 1 e − q 1 + d t d log b j ≥ q ˙ 1 − q 1 q ˙ 1 ≥ 0. (85)
An identical calculation holds for terms with q 2 q_2 q 2 .
Lemma D.4 (Ordering of margins). For every B > 1 B>1 B > 1 there exists some M 0 ( B ) M_0(B) M 0 ( B ) so that for every M ≥ M 0 ( B ) M\geq M_0(B) M ≥ M 0 ( B ) ,
∀ t ≥ 0 : ξ 2 ( t ) ≥ ξ 1 ( t ) , q 2 ( t ) ≥ q 1 ( t ) ≥ 1 , \forall t\geq0:\quad\xi_2(t)\geq\xi_1(t),\quad q_2(t)\geq q_1(t)\geq1, ∀ t ≥ 0 : ξ 2 ( t ) ≥ ξ 1 ( t ) , q 2 ( t ) ≥ q 1 ( t ) ≥ 1 , And the differences ξ 2 − ξ 1 , q 2 − q 1 \xi_2-\xi_1,q_2-q_1 ξ 2 − ξ 1 , q 2 − q 1 are strictly increasing. It therefore holds that q min ( t ) = q 1 ( t ) q_{\min}(t)=q_1(t) q m i n ( t ) = q 1 ( t ) . Also,
∀ t ≥ 0 : q 2 e − q 2 ≤ q 1 e − q 1 . \forall t\geq0:\quad q_2e^{-q_2}\leq q_1e^{-q_1}. ∀ t ≥ 0 : q 2 e − q 2 ≤ q 1 e − q 1 .
Proof. Denote ξ Δ ( t ) = ξ 2 ( t ) − ξ 1 ( t ) \xi_{\Delta}(t)=\xi_2(t)-\xi_1(t) ξ Δ ( t ) = ξ 2 ( t ) − ξ 1 ( t ) . At initialization,
ξ Δ ( 0 ) = ρ P π M a − ρ P π = ρ P π z ρ L ≥ 0 , \xi_{\Delta}(0)=\frac{\rho}{P_{\pi}}Ma-\frac{\rho}{P_{\pi}}=\frac{\rho}{P_{\pi}}\frac{z}{\rho^L}\geq0, ξ Δ ( 0 ) = P π ρ M a − P π ρ = P π ρ ρ L z ≥ 0 ,
and since u ˙ 1 ( 0 ) = u ˙ 2 ( 0 ) = η 0 \dot u_1(0)=\dot u_2(0)=\eta_0 u ˙ 1 ( 0 ) = u ˙ 2 ( 0 ) = η 0 ,
ξ ˙ Δ ( 0 ) = M Λ u ˙ 1 ( 0 ) − 1 Λ ( B u ˙ 1 ( 0 ) + u ˙ 2 ( 0 ) ) = η 0 M − B − 1 Λ > 0. \dot\xi_{\Delta}(0)=\frac{M}{\Lambda}\dot u_1(0)-\frac{1}{\Lambda}\left(B\dot u_1(0)+\dot u_2(0)\right)=\eta_0\frac{M-B-1}{\Lambda}>0. ξ ˙ Δ ( 0 ) = Λ M u ˙ 1 ( 0 ) − Λ 1 ( B u ˙ 1 ( 0 ) + u ˙ 2 ( 0 ) ) = η 0 Λ M − B − 1 > 0.
So there exists an initial interval on which ξ Δ ( t ) ≥ 0 \xi_{\Delta}(t)\geq0 ξ Δ ( t ) ≥ 0 . We show that on any such interval, we have ξ ˙ Δ > 0 \dot\xi_{\Delta}>0 ξ ˙ Δ > 0 . Indeed, we have ξ 2 ≥ ξ 1 \xi_2\geq\xi_1 ξ 2 ≥ ξ 1 and q 2 ≥ q 1 q_2\geq q_1 q 2 ≥ q 1 , so
B p u 2 ≤ p u 1 ≤ B p u 2 + M Λ P ( b ) e − q 1 = ( B + M ) p u 2 Bp_{u_2}\leq p_{u_1}\leq Bp_{u_2}+\frac{M}{\Lambda}P(\mathbf{b})e^{-q_1}=(B+M)p_{u_2} B p u 2 ≤ p u 1 ≤ B p u 2 + Λ M P ( b ) e − q 1 = ( B + M ) p u 2
Therefore,
B ( − m u 2 ) ≤ − m u 1 ≤ ( B + M ) ( − m u 2 ) , v u 1 ≤ ( B + M ) 2 v u 2 . B(-m_{u_2})\leq-m_{u_1}\leq(B+M)(-m_{u_2}),\qquad v_{u_1}\leq(B+M)^2v_{u_2}. B ( − m u 2 ) ≤ − m u 1 ≤ ( B + M ) ( − m u 2 ) , v u 1 ≤ ( B + M ) 2 v u 2 .
And since the bias correction terms cancel,
u ˙ 1 u ˙ 2 ≥ B B + M \frac{\dot{u}_1}{\dot{u}_2} \ge\frac{B}{B+M} u ˙ 2 u ˙ 1 ≥ B + M B
So
ξ ˙ Δ = M Λ u ˙ 1 − 1 Λ ( B u ˙ 1 + u ˙ 2 ) = 1 Λ ( ( M − B ) u ˙ 1 − u ˙ 2 ) ≥ 1 Λ u ˙ 2 ( B ( M − B ) B + M − 1 ) > 0 \dot{\xi}_{\Delta}=\frac{M}{\Lambda}\dot{u}_1-\frac{1}{\Lambda}(B\dot{u}_1+\dot{u}_2)=\frac{1}{\Lambda}\left((M-B)\dot{u}_1-\dot{u}_2\right)\ge\frac{1}{\Lambda}\dot{u}_2\left(\frac{B(M-B)}{B+M}-1\right)>0 ξ ˙ Δ = Λ M u ˙ 1 − Λ 1 ( B u ˙ 1 + u ˙ 2 ) = Λ 1 ( ( M − B ) u ˙ 1 − u ˙ 2 ) ≥ Λ 1 u ˙ 2 ( B + M B ( M − B ) − 1 ) > 0
for large enough M M M when B > 1 B>1 B > 1 . Therefore, a first-crossing argument proves ξ Δ ≥ 0 \xi_{\Delta}\ge0 ξ Δ ≥ 0 everywhere (otherwise there would be a first crossing point t 0 t_0 t 0 where ξ Δ ( t 0 ) < 0 \xi_{\Delta}(t_0)<0 ξ Δ ( t 0 ) < 0 , but we have proved that on [ 0 , t 0 ] [0,t_0] [ 0 , t 0 ] we have ξ Δ ≥ 0 , ξ ˙ Δ > 0 \xi_{\Delta}\ge0,\dot{\xi}_{\Delta}>0 ξ Δ ≥ 0 , ξ ˙ Δ > 0 , a contradiction). Hence also ξ ˙ Δ > 0 \dot{\xi}_{\Delta}>0 ξ ˙ Δ > 0 everywhere. Because q 2 − q 1 = P ( b ) ξ Δ q_2-q_1=P(b)\xi_{\Delta} q 2 − q 1 = P ( b ) ξ Δ , with P > 0 P>0 P > 0 increasing, this difference is strictly increasing as well. Now, since q 1 , q 2 q_1,q_2 q 1 , q 2 are increasing, q 2 ≥ q 1 q_2\ge q_1 q 2 ≥ q 1 and q 1 ( 0 ) = ρ L ≥ 1 q_1(0)=\rho^L\ge1 q 1 ( 0 ) = ρ L ≥ 1 , and since x ↦ x e − x x\mapsto xe^{-x} x ↦ x e − x is strictly decreasing on [ 1 , ∞ ) [1,\infty) [ 1 , ∞ ) we get the final claim.
Lemma D.5 (Coordinate Comparability). For any compact Π ⊂ ( 0 , ∞ ) L − 1 \Pi\subset(0,\infty)^{L-1} Π ⊂ ( 0 , ∞ ) L − 1 , there exist constants 0 < k j ≤ K j < ∞ 0<k_j\le K_j<\infty 0 < k j ≤ K j < ∞ , j ∈ [ L + 1 ] j\in[L+1] j ∈ [ L + 1 ] , so that for any ρ ≥ ρ 0 ≥ 1 , z ∈ [ 0 , 3 log M ] \rho\ge\rho_0\ge1,z\in[0,3\log M] ρ ≥ ρ 0 ≥ 1 , z ∈ [ 0 , 3 log M ] and π ∈ Π \pi\in\Pi π ∈ Π , it holds that
k j ∥ θ t ∥ 2 ≤ θ j ( t ) ≤ K j ∥ θ t ∥ 2 , k j ∥ p t ∥ 2 ≤ p j ( t ) ≤ K j ∥ p t ∥ 2 , ( t ≥ 0 ) . k_j\|\theta_t\|_2\le\theta_j(t)\le K_j\|\theta_t\|_2,\qquad k_j\|p_t\|_2\le p_j(t)\le K_j\|p_t\|_2,\qquad(t\ge0). k j ∥ θ t ∥ 2 ≤ θ j ( t ) ≤ K j ∥ θ t ∥ 2 , k j ∥ p t ∥ 2 ≤ p j ( t ) ≤ K j ∥ p t ∥ 2 , ( t ≥ 0 ) . As a result, such constants exist for − m t , v t -m_t,v_t − m t , v t .
Proof. First note that u 1 ≍ u 2 u_1\asymp u_2 u 1 ≍ u 2 since B p u 2 ≤ p u 1 ≤ ( B + M ) p u 2 Bp u_2\le p u_1\le(B+M)p u_2 B p u 2 ≤ p u 1 ≤ ( B + M ) p u 2 implies B B + M ≤ u ˙ 1 u ˙ 2 ≤ B + M B \frac{B}{B+M}\le\frac{\dot{u}_1}{\dot{u}_2}\le\frac{B+M}{B} B + M B ≤ u ˙ 2 u ˙ 1 ≤ B B + M , and 1 M − B ≤ u 1 ( 0 ) u 2 ( 0 ) ≤ 1 \frac{1}{M-B}\le\frac{u_1(0)}{u_2(0)}\le1 M − B 1 ≤ u 2 ( 0 ) u 1 ( 0 ) ≤ 1 . We claim that this fact and Lemma D.4 imply that θ j p j ≍ q 1 e − q 1 \theta_jp_j\asymp q_1e^{-q_1} θ j p j ≍ q 1 e − q 1 for all j j j , and in particular that there exist k p , K p > 0 k_p,K_p>0 k p , K p > 0 with
k p θ j θ i ≤ p i p j ≤ K p θ j θ i , t ≥ 0 , i , j ∈ [ L + 1 ] . (86) k_p\frac{\theta_j}{\theta_i}\le\frac{p_i}{p_j}\le K_p\frac{\theta_j}{\theta_i},\qquad t\ge0,\quad i,j\in[L+1]. \tag*{(86)} k p θ i θ j ≤ p j p i ≤ K p θ i θ j , t ≥ 0 , i , j ∈ [ L + 1 ] . (86)
Indeed,
u 1 p u 1 = B u 1 B u 1 + u 2 q 1 e − q 1 + q 2 e − q 2 , u 2 p u 2 = u 2 B u 1 + u 2 q 1 e − q 1 , b j p b j = q 1 e − q 1 + q 2 e − q 2 , u_1p_{u_1}=\frac{Bu_1}{Bu_1+u_2}q_1e^{-q_1}+q_2e^{-q_2},\qquad u_2p_{u_2}=\frac{u_2}{Bu_1+u_2}q_1e^{-q_1},\qquad b_jp_{b_j}=q_1e^{-q_1}+q_2e^{-q_2}, u 1 p u 1 = B u 1 + u 2 B u 1 q 1 e − q 1 + q 2 e − q 2 , u 2 p u 2 = B u 1 + u 2 u 2 q 1 e − q 1 , b j p b j = q 1 e − q 1 + q 2 e − q 2 ,
and recall that q 2 e − q 2 ≤ q 1 e − q 1 q_2e^{-q_2}\le q_1e^{-q_1} q 2 e − q 2 ≤ q 1 e − q 1 . Note the constants k p , K p k_p,K_p k p , K p in (86) do not depend on ρ , z , π \rho,z,\pi ρ , z , π , so proving comparability of parameters suffices for comparability of gradients.
All constants below may depend on M , Π M,\Pi M , Π and the structural parameters, but not on ρ , z , π \rho,z,\pi ρ , z , π . Set δ = ( c 1 − c 2 / 2 ) / 2 > 0 \delta=(c_1-c_2/2)/2>0 δ = ( c 1 − c 2 /2 ) /2 > 0 , and choose T > 0 T>0 T > 0 such that ( log η ^ ) ′ ≤ δ (\log\hat{\eta})'\le\delta ( log η ^ ) ′ ≤ δ for t ≥ T t\ge T t ≥ T . The initialization and the universal velocity bound imply that there exist c T , C T > 0 c_T,C_T>0 c T , C T > 0 such that
c T ρ ≤ θ i ( t ) ≤ C T ρ , 0 ≤ t ≤ T , θ ˙ i ( T ) ≤ C T . c_T\rho\le\theta_i(t)\le C_T\rho,\qquad0\le t\le T,\qquad\dot{\theta}_i(T)\le C_T. c T ρ ≤ θ i ( t ) ≤ C T ρ , 0 ≤ t ≤ T , θ ˙ i ( T ) ≤ C T .
For t ≥ T t\ge T t ≥ T , write
θ i R M S ( t ) : = θ i ( T ) + ∫ T t η ^ ( s ) p i ( s ) v i ( s ) d s . \theta_i^{\mathrm{RMS}}(t):=\theta_i(T)+\int_T^t\hat{\eta}(s)\frac{p_i(s)}{\sqrt{v_i(s)}}\,ds. θ i RMS ( t ) := θ i ( T ) + ∫ T t η ^ ( s ) v i ( s ) p i ( s ) d s .
Lemma B.3 and the choice of T T T give
θ ¨ i + δ θ ˙ i ≤ c 1 θ ˙ i R M S . \ddot{\theta}_i+\delta\dot{\theta}_i\le c_1\dot{\theta}_i^{\mathrm{RMS}}. θ ¨ i + δ θ ˙ i ≤ c 1 θ ˙ i RMS .
Also, monotonicity of p i p_i p i gives − m i ( t ) ≥ ( 1 − e − c 1 t ) p i ( t ) -m_i(t)\ge(1-e^{-c_1t})p_i(t) − m i ( t ) ≥ ( 1 − e − c 1 t ) p i ( t ) . Integrating these two inequalities yields
θ i ( t ) ≤ θ i ( T ) + θ ˙ i ( T ) δ + c 1 δ ( θ i R M S ( t ) − θ i R M S ( T ) ) , \theta_i(t)\le\theta_i(T)+\frac{\dot{\theta}_i(T)}{\delta}+\frac{c_1}{\delta}\left(\theta_i^{\mathrm{RMS}}(t)-\theta_i^{\mathrm{RMS}}(T)\right), θ i ( t ) ≤ θ i ( T ) + δ θ ˙ i ( T ) + δ c 1 ( θ i RMS ( t ) − θ i RMS ( T ) ) ,
θ i ( t ) ≥ θ i ( T ) + ( 1 − e − c 1 T ) ( θ i R M S ( t ) − θ i R M S ( T ) ) . \theta_i(t)\ge\theta_i(T)+(1-e^{-c_1T})\left(\theta_i^{\mathrm{RMS}}(t)-\theta_i^{\mathrm{RMS}}(T)\right). θ i ( t ) ≥ θ i ( T ) + ( 1 − e − c 1 T ) ( θ i RMS ( t ) − θ i RMS ( T ) ) .
There exists c > 0 c>0 c > 0 such that θ i ( T ) ≥ c ρ ≥ c \theta_i(T)\ge c\rho\ge c θ i ( T ) ≥ c ρ ≥ c for every i i i , and hence there exist uniform constants c ∗ , C ∗ > 0 c_*,C_*>0 c ∗ , C ∗ > 0 such that
c ∗ θ i R M S ( t ) ≤ θ i ( t ) ≤ C ∗ θ i R M S ( t ) , t ≥ T . c_*\theta_i^{\mathrm{RMS}}(t)\le\theta_i(t)\le C_*\theta_i^{\mathrm{RMS}}(t),\qquad t\ge T. c ∗ θ i RMS ( t ) ≤ θ i ( t ) ≤ C ∗ θ i RMS ( t ) , t ≥ T .
Define for i , j i,j i , j ,
K i j ( t ) : = max { K 0 , max T ≤ s ≤ t θ i R M S ( s ) θ j R M S ( s ) } , K 0 : = max i , j sup 0 ≤ s ≤ T θ i ( s ) θ j ( s ) . \mathcal{K}_{ij}(t):=\max\left\{K_0,\max_{T\le s\le t}\frac{\theta_i^{\mathrm{RMS}}(s)}{\theta_j^{\mathrm{RMS}}(s)}\right\},\qquad K_0:=\max_{i,j}\sup_{0\le s\le T}\frac{\theta_i(s)}{\theta_j(s)}. K ij ( t ) := max { K 0 , T ≤ s ≤ t max θ j RMS ( s ) θ i RMS ( s ) } , K 0 := i , j max 0 ≤ s ≤ T sup θ j ( s ) θ i ( s ) .
For every 0 ≤ s ≤ t 0\le s\le t 0 ≤ s ≤ t , the preceding bounds imply θ i ( s ) / θ j ( s ) ≤ C K i j ( t ) \theta_i(s)/\theta_j(s)\le C\mathcal{K}_{ij}(t) θ i ( s ) / θ j ( s ) ≤ C K ij ( t ) for some C > 0 C>0 C > 0 . Thus, for some C > 0 C>0 C > 0 , (86) and the second-moment formula give
v j ( s ) v i ( s ) ≤ C K i j ( t ) , T ≤ s ≤ t . \sqrt{\frac{v_j(s)}{v_i(s)}}\le C\mathcal{K}_{ij}(t),\qquad T\le s\le t. v i ( s ) v j ( s ) ≤ C K ij ( t ) , T ≤ s ≤ t .
Using (86) once more, there exists C 1 > 0 C_1>0 C 1 > 0 such that
θ i R M S ( s ) θ ˙ i R M S ( s ) θ j R M S ( s ) θ ˙ j R M S ( s ) = θ i R M S ( s ) θ j R M S ( s ) p i ( s ) p j ( s ) v j ( s ) v i ( s ) ≤ C 1 K i j ( t ) . \frac{\theta_i^{\mathrm{RMS}}(s)\dot{\theta}_i^{\mathrm{RMS}}(s)}{\theta_j^{\mathrm{RMS}}(s)\dot{\theta}_j^{\mathrm{RMS}}(s)}
=\frac{\theta_i^{\mathrm{RMS}}(s)}{\theta_j^{\mathrm{RMS}}(s)}\frac{p_i(s)}{p_j(s)}\sqrt{\frac{v_j(s)}{v_i(s)}}\le C_1\mathcal{K}_{ij}(t). θ j RMS ( s ) θ ˙ j RMS ( s ) θ i RMS ( s ) θ ˙ i RMS ( s ) = θ j RMS ( s ) θ i RMS ( s ) p j ( s ) p i ( s ) v i ( s ) v j ( s ) ≤ C 1 K ij ( t ) .
So,
d d s ( θ i R M S ( s ) ) 2 ≤ C 1 K i j ( t ) d d s ( θ j R M S ( s ) ) 2 . \frac{d}{ds}\left(\theta_i^{\mathrm{RMS}}(s)\right)^2\le C_1\mathcal{K}_{ij}(t)\frac{d}{ds}\left(\theta_j^{\mathrm{RMS}}(s)\right)^2. d s d ( θ i RMS ( s ) ) 2 ≤ C 1 K ij ( t ) d s d ( θ j RMS ( s ) ) 2 .
Fix t > T t>T t > T . If K i j ( t ) > K 0 \mathcal{K}_{ij}(t)>K_0 K ij ( t ) > K 0 , choose s ∗ ∈ [ T , t ] s_*\in[T,t] s ∗ ∈ [ T , t ] attaining its maximum. Integrating on [ T , s ∗ ] [T,s_*] [ T , s ∗ ] gives
( θ i R M S ( s ∗ ) ) 2 − ( θ i R M S ( T ) ) 2 ≤ C 1 K i j ( t ) [ ( θ j R M S ( s ∗ ) ) 2 − ( θ j R M S ( T ) ) 2 ] . \left(\theta_i^{\mathrm{RMS}}(s_*)\right)^2-\left(\theta_i^{\mathrm{RMS}}(T)\right)^2\le C_1\mathcal{K}_{ij}(t)\left[\left(\theta_j^{\mathrm{RMS}}(s_*)\right)^2-\left(\theta_j^{\mathrm{RMS}}(T)\right)^2\right]. ( θ i RMS ( s ∗ ) ) 2 − ( θ i RMS ( T ) ) 2 ≤ C 1 K ij ( t ) [ ( θ j RMS ( s ∗ ) ) 2 − ( θ j RMS ( T ) ) 2 ] .
And the choice of s ∗ s_* s ∗ gives
θ i R M S ( s ∗ ) = K i j ( t ) θ j R M S ( s ∗ ) , \theta_i^{\mathrm{RMS}}(s_*)=\mathcal{K}_{ij}(t)\theta_j^{\mathrm{RMS}}(s_*), θ i RMS ( s ∗ ) = K ij ( t ) θ j RMS ( s ∗ ) ,
K i j ( t ) 2 ( θ j R M S ( s ∗ ) ) 2 ≤ ( θ i R M S ( T ) ) 2 + C 1 K i j ( t ) ( ( θ j R M S ( s ∗ ) ) 2 − ( θ j R M S ( T ) ) 2 ) . \mathcal{K}_{ij}(t)^2\left(\theta_j^{\mathrm{RMS}}(s_*)\right)^2\le\left(\theta_i^{\mathrm{RMS}}(T)\right)^2+C_1\mathcal{K}_{ij}(t)\left(\left(\theta_j^{\mathrm{RMS}}(s_*)\right)^2-\left(\theta_j^{\mathrm{RMS}}(T)\right)^2\right). K ij ( t ) 2 ( θ j RMS ( s ∗ ) ) 2 ≤ ( θ i RMS ( T ) ) 2 + C 1 K ij ( t ) ( ( θ j RMS ( s ∗ ) ) 2 − ( θ j RMS ( T ) ) 2 ) .
Since θ i R M S ( T ) ≤ K 0 θ j R M S ( T ) \theta_i^{\mathrm{RMS}}(T)\le K_0\theta_j^{\mathrm{RMS}}(T) θ i RMS ( T ) ≤ K 0 θ j RMS ( T ) and θ j R M S \theta_j^{\mathrm{RMS}} θ j RMS is increasing,
K i j ( t ) 2 ≤ K 0 2 + C 1 K i j ( t ) , \mathcal{K}_{ij}(t)^2\le K_0^2+C_1\mathcal{K}_{ij}(t), K ij ( t ) 2 ≤ K 0 2 + C 1 K ij ( t ) ,
and hence K i j ( t ) ≤ K 0 + C 1 \mathcal{K}_{ij}(t)\le K_0+C_1 K ij ( t ) ≤ K 0 + C 1 . This bound also holds when K i j ( t ) = K 0 \mathcal{K}_{ij}(t)=K_0 K ij ( t ) = K 0 . Applying this to every ordered pair proves uniform comparability of the RMSprop coordinates, and hence of the parameters. Equation (86) then proves comparability of the gradients. Positivity of the EMA kernels gives the corresponding comparisons for − m t -\mathbf{m}_t − m t and v t \mathbf{v}_t v t .
Lemma D.6 (Loss Decay and Positive Margin). It holds that L ( θ t ) → t → ∞ 0 \mathcal{L}(\theta_t)\xrightarrow{t\to\infty}0 L ( θ t ) t → ∞ 0 and there exist γ 0 , t 0 > 0 \gamma_0,t_0>0 γ 0 , t 0 > 0 with ∀ t ≥ t 0 : γ ( θ t ) ≥ γ 0 > 0 \forall t\ge t_0:\gamma(\theta_t)\ge\gamma_0>0 ∀ t ≥ t 0 : γ ( θ t ) ≥ γ 0 > 0 .
Proof. We begin with loss decay. Since all parameters are positive and increasing, so are P ( b ) P(\mathbf{b}) P ( b ) , q 1 q_1 q 1 . If towards a contradiction q 1 → t → ∞ Z < ∞ q_1\xrightarrow{t\to\infty}Z<\infty q 1 t → ∞ Z < ∞ , this implies a positive lower bound p u 2 ( t ) ≥ p ∗ > 0 p_{u_2}(t)\ge p_*>0 p u 2 ( t ) ≥ p ∗ > 0 , and since additionally p u 2 p_{u_2} p u 2 is nonincreasing,
− m ^ u 2 ( t ) ≥ p ∗ , v ^ u 2 ( t ) ≤ p u 2 ( 0 ) , u ˙ 2 ( t ) ≥ p ∗ p u 2 ( 0 ) η ( t ) . -\widehat{m}_{u_2}(t)\ge p_*,\qquad\sqrt{\widehat{v}_{u_2}(t)}\le p_{u_2}(0),\qquad\dot{u}_2(t)\ge\frac{p_*}{p_{u_2}(0)}\eta(t). − m u 2 ( t ) ≥ p ∗ , v u 2 ( t ) ≤ p u 2 ( 0 ) , u ˙ 2 ( t ) ≥ p u 2 ( 0 ) p ∗ η ( t ) .
Since η ( t ) = η 0 ( 1 + t ) α \eta(t)=\eta_0(1+t)^\alpha η ( t ) = η 0 ( 1 + t ) α , α > − 1 \alpha>-1 α > − 1 , this implies u 2 ( t ) → t → ∞ ∞ u_2(t)\xrightarrow{t\to\infty}\infty u 2 ( t ) t → ∞ ∞ , but this in turn implies q 1 ( t ) → t → ∞ ∞ q_1(t)\xrightarrow{t\to\infty}\infty q 1 ( t ) t → ∞ ∞ , a contradiction. Thus L ( θ t ) ≤ 2 e − q 1 ( t ) → t → ∞ 0 \mathcal{L}(\theta_t)\le2e^{-q_1(t)}\xrightarrow{t\to\infty}0 L ( θ t ) ≤ 2 e − q 1 ( t ) t → ∞ 0 .
The margin lower bound follows from parameter coordinate comparability: q min ( θ t ) = q 1 ( t ) = P ( b ) B u 1 + u 2 Λ ≳ ∥ θ t ∥ 2 L q_{\min}(\theta_t)=q_1(t)=P(\mathbf{b})\frac{B^{u_1+u_2}}{\Lambda}\gtrsim\lVert\theta_t\rVert_2^L q m i n ( θ t ) = q 1 ( t ) = P ( b ) Λ B u 1 + u 2 ≳ ∥ θ t ∥ 2 L since each parameter is comparable to the norm.
:::
Corollary D.7 (Second Moment Freezing and Asymptotics). Suppose α > 1 L − 1 2 \alpha> \frac{1}{L} - \frac{1}{2} α > L 1 − 2 1 . For all j ∈ [ L + 1 ] j \in[L+1] j ∈ [ L + 1 ] ,
0 < ω j < ∞ , (87) 0 < \omega_j < \infty, \tag*{(87)} 0 < ω j < ∞ , (87) θ ˙ j ( t ) = O ( t 1 / L − 1 ) . (88) \dot{\theta}_j(t) = O\left(t^{1/L-1}\right). \tag*{(88)} θ ˙ j ( t ) = O ( t 1/ L − 1 ) . (88) Moreover,
q 1 ( t ) = c 2 2 t + o ( t ) , q 2 ( t ) − q 1 ( t ) ⟶ ∞ . (89) q_1(t) = \frac{c_2}{2}t + o(t), \qquad q_2(t) - q_1(t) \longrightarrow\infty. \tag*{(89)} q 1 ( t ) = 2 c 2 t + o ( t ) , q 2 ( t ) − q 1 ( t ) ⟶ ∞. (89) Proof. Lemma D.6 provides the conditions of Theorem C.13 , which gives
U ( t ) = c 2 2 t + ( α + 2 − 2 L ) log t + O ( 1 ) . U(t) = \frac{c_2}{2}t + \left(\alpha+ 2 - \frac{2}{L}\right)\log t + O(1). U ( t ) = 2 c 2 t + ( α + 2 − L 2 ) log t + O ( 1 ) . Since q 1 − log 2 ≤ U ≤ q 1 q_1-\log2 \le U \le q_1 q 1 − log 2 ≤ U ≤ q 1 , this proves the asserted leading rate of q 1 q_1 q 1 .
Lemma C.3 and coordinate comparability give
θ j ( t ) ≍ t 1 / L , p j ( t ) ≤ ∥ g t ∥ 2 2 ≲ U ( t ) 1 − 1 / L e − U ( t ) = O ( e − c 2 t / 2 t − ζ ) , \theta_j(t) \asymp t^{1/L}, \qquad p_j(t) \le\lVert g_t\rVert_2^2 \lesssim U(t)^{1-1/L}e^{-U(t)} = O\left(e^{-c_2t/2}t^{-\zeta}\right), θ j ( t ) ≍ t 1/ L , p j ( t ) ≤ ∥ g t ∥ 2 2 ≲ U ( t ) 1 − 1/ L e − U ( t ) = O ( e − c 2 t /2 t − ζ ) , where ζ = α + 1 − 1 / L > 1 / 2 \zeta= \alpha+ 1 - 1/L > 1/2 ζ = α + 1 − 1/ L > 1/2 . Consequently,
ω j = c 2 ∫ 0 ∞ e c 2 t p j ( t ) 2 d t < ∞ , \omega_j = c_2\int_0^\infty e^{c_2t}p_j(t)^2\,\mathrm{d}t < \infty, ω j = c 2 ∫ 0 ∞ e c 2 t p j ( t ) 2 d t < ∞ , and positivity follows from p j > 0 p_j > 0 p j > 0 . Since c 1 > c 2 / 2 c_1 > c_2/2 c 1 > c 2 /2 , the first-moment formula gives
∣ m ^ j ( t ) ∣ = O ( e − c 2 t / 2 t − ζ ) . |\widehat{m}_j(t)| = O\left(e^{-c_2t/2}t^{-\zeta}\right). ∣ m j ( t ) ∣ = O ( e − c 2 t /2 t − ζ ) . Also, I j ( t ) → ω j > 0 I_j(t) \to\omega_j > 0 I j ( t ) → ω j > 0 implies v ^ j ( t ) ≍ e − c 2 t / 2 \sqrt{\widehat{v}_j(t)} \asymp e^{-c_2t/2} v j ( t ) ≍ e − c 2 t /2 . Thus
θ ˙ j ( t ) = η ( t ) ∣ m ^ j ( t ) ∣ v ^ j ( t ) = O ( t α − ζ ) = O ( t 1 / L − 1 ) . \dot{\theta}_j(t) = \eta(t)\frac{|\widehat{m}_j(t)|}{\sqrt{\widehat{v}_j(t)}} = O\left(t^{\alpha-\zeta}\right) = O\left(t^{1/L-1}\right). θ ˙ j ( t ) = η ( t ) v j ( t ) ∣ m j ( t ) ∣ = O ( t α − ζ ) = O ( t 1/ L − 1 ) . Finally, Lemma D.4 gives ξ ˙ Δ ≥ k u 2 \dot{\xi}_{\Delta} \ge k u_2 ξ ˙ Δ ≥ k u 2 for some k > 0 k > 0 k > 0 . Since u 2 ( t ) ≍ t 1 / L → ∞ u_2(t) \asymp t^{1/L} \to\infty u 2 ( t ) ≍ t 1/ L → ∞ , it follows that ξ Δ ( t ) → ∞ \xi_{\Delta}(t) \to\infty ξ Δ ( t ) → ∞ .
Therefore,
q 2 ( t ) − q 1 ( t ) = P ( b ( t ) ) ξ Δ ( t ) ≥ P ( b ( 0 ) ) ξ Δ ( t ) ⟶ ∞ . q_2(t) - q_1(t) = P(b(t))\xi_{\Delta}(t) \ge P(b(0))\xi_{\Delta}(t) \longrightarrow\infty. q 2 ( t ) − q 1 ( t ) = P ( b ( t )) ξ Δ ( t ) ≥ P ( b ( 0 )) ξ Δ ( t ) ⟶ ∞.
Lemma D.8 (Bounds on the Second-Moment Weight Ratios). For every compact set Π ⊆ ( 0 , ∞ ) L − 1 \Pi\subseteq(0,\infty)^{L-1} Π ⊆ ( 0 , ∞ ) L − 1 , there exists k Π > 0 k_{\Pi} > 0 k Π > 0 , independent of M M M , such that for every M ≥ M 0 ( B ) M \ge M_0(B) M ≥ M 0 ( B ) there exists ρ 1 = ρ 1 ( M , Π ) \rho_1 = \rho_1(M,\Pi) ρ 1 = ρ 1 ( M , Π ) with the following property. For every ρ ≥ ρ 1 \rho\ge\rho_1 ρ ≥ ρ 1 , z ∈ [ 0 , 3 log M ] z \in[0,3\log M] z ∈ [ 0 , 3 log M ] , and π ∈ Π \pi\in\Pi π ∈ Π ,
1 8 ≤ π 0 2 ω u 2 π j 2 ω b j ≤ 8 , j ∈ [ L − 1 ] , (90) \frac{1}{8} \le\frac{\pi_0^2\omega_{u_2}}{\pi_j^2\omega_{b_j}} \le8, \qquad j \in[L-1], \tag*{(90)} 8 1 ≤ π j 2 ω b j π 0 2 ω u 2 ≤ 8 , j ∈ [ L − 1 ] , (90) and
B 2 ≤ ω u 1 ω u 2 ≤ ( B + M e − z ) 2 . (91) B^2 \le\frac{\omega_{u_1}}{\omega_{u_2}} \le\left(B+Me^{-z}\right)^2. \tag*{(91)} B 2 ≤ ω u 2 ω u 1 ≤ ( B + M e − z ) 2 . (91) Moreover, when z = 0 z=0 z = 0 ,
ω u 1 ω u 2 ≥ k Π M . (92) \frac{\omega_{u_1}}{\omega_{u_2}} \ge k_{\Pi}M. \tag*{(92)} ω u 2 ω u 1 ≥ k Π M . (92) For L = 1 L=1 L = 1 , the bounds in (90) are void, and the dependence on Π \Pi Π is omitted.
Proof. Recall that
I j ( t ) = c 2 ∫ 0 t e c 2 s p j ( s ) 2 d s , ω j = I j ( ∞ ) . I_j(t)=c_2\int_0^t e^{c_2s}p_j(s)^2\,\mathrm{d}s,\qquad\omega_j=I_j(\infty). I j ( t ) = c 2 ∫ 0 t e c 2 s p j ( s ) 2 d s , ω j = I j ( ∞ ) .
By Corollary D.7 , all these weights are positive and finite. All limits as ρ → ∞ \rho\to\infty ρ → ∞ below are uniform for z ∈ [ 0 , 3 log M ] z\in[0,3\log M] z ∈ [ 0 , 3 log M ] and π ∈ Π \pi\in\Pi π ∈ Π , with M , Π M,\Pi M , Π fixed. Constants indexed only by Π \Pi Π are independent of M M M .
First, Lemma D.4 gives q 2 ( t ) − q 1 ( t ) ≥ z q_2(t)-q_1(t)\ge z q 2 ( t ) − q 1 ( t ) ≥ z , and hence
B ≤ p u 1 ( t ) p u 2 ( t ) = B + M e − ( q 2 ( t ) − q 1 ( t ) ) ≤ B + M e − z . B\le\frac{p_{u_1}(t)}{p_{u_2}(t)}=B+Me^{-(q_2(t)-q_1(t))}\le B+Me^{-z}. B ≤ p u 2 ( t ) p u 1 ( t ) = B + M e − ( q 2 ( t ) − q 1 ( t )) ≤ B + M e − z .
Squaring and integrating proves (91) .
We next compare the hidden-layer weights with ω u 2 \omega_{u_2} ω u 2 . The universal Adam-ratio bound (Lemma B.2 ) implies that there exists K > 0 K>0 K > 0 such that
0 ≤ θ j ( t ) − θ j ( 0 ) ≤ K t , 0\le\theta_j(t)-\theta_j(0)\le Kt, 0 ≤ θ j ( t ) − θ j ( 0 ) ≤ K t ,
where K K K is independent of the initialization. Since B u 1 ( 0 ) + u 2 ( 0 ) = ρ π 0 Bu_1(0)+u_2(0)=\rho\pi_0 B u 1 ( 0 ) + u 2 ( 0 ) = ρ π 0 and b j ( 0 ) = ρ π j b_j(0)=\rho\pi_j b j ( 0 ) = ρ π j , it follows for 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 that
0 ≤ B u 1 ( t ) + u 2 ( t ) ρ π 0 − 1 ≤ ( B + 1 ) K ρ π 0 , 0\le\frac{Bu_1(t)+u_2(t)}{\rho\pi_0}-1\le\frac{(B+1)K}{\rho\pi_0}, 0 ≤ ρ π 0 B u 1 ( t ) + u 2 ( t ) − 1 ≤ ρ π 0 ( B + 1 ) K ,
0 ≤ b j ( t ) ρ π j − 1 ≤ K ρ π j , j ∈ [ L − 1 ] . 0\le\frac{b_j(t)}{\rho\pi_j}-1\le\frac{K}{\rho\pi_j},\qquad j\in[L-1]. 0 ≤ ρ π j b j ( t ) − 1 ≤ ρ π j K , j ∈ [ L − 1 ] .
Compactness of Π \Pi Π bounds π 0 , π 1 , … , π L − 1 \pi_0,\pi_1,\ldots,\pi_{L-1} π 0 , π 1 , … , π L − 1 above and away from zero, so these relative errors are O ( ρ − 1 ) O(\rho^{-1}) O ( ρ − 1 ) , uniformly on [ 0 , 1 ] [0,1] [ 0 , 1 ] , with constants independent of M M M . Taking the product of the normalized b j b_j b j coordinates, and using ∏ j = 1 L − 1 π j = Λ / π 0 \prod_{j=1}^{L-1}\pi_j=\Lambda/\pi_0 ∏ j = 1 L − 1 π j = Λ/ π 0 , gives the first estimate below. Taking the quotient of the normalized B u 1 + u 2 Bu_1+u_2 B u 1 + u 2 and b j b_j b j gives the second:
P ( b ( t ) ) Λ = π 0 − 1 ρ L − 1 ( 1 + O ( ρ − 1 ) ) , \frac{P(b(t))}{\Lambda}=\pi_0^{-1}\rho^{L-1}(1+O(\rho^{-1})), Λ P ( b ( t )) = π 0 − 1 ρ L − 1 ( 1 + O ( ρ − 1 )) ,
B u 1 ( t ) + u 2 ( t ) b j ( t ) = π 0 π j ( 1 + O ( ρ − 1 ) ) , j ∈ [ L − 1 ] . (93) \frac{Bu_1(t)+u_2(t)}{b_j(t)}=\frac{\pi_0}{\pi_j}(1+O(\rho^{-1})),\qquad j\in[L-1]. \tag*{(93)} b j ( t ) B u 1 ( t ) + u 2 ( t ) = π j π 0 ( 1 + O ( ρ − 1 )) , j ∈ [ L − 1 ] . (93)
uniformly on [ 0 , 1 ] [0,1] [ 0 , 1 ] . The constants here may depend on Π \Pi Π , but not on M M M .
For L > 1 L>1 L > 1 , the exact gradient formulas give
p b j p u 2 = B u 1 + u 2 b j ( 1 + q 2 q 1 e − ( q 2 − q 1 ) ) . \frac{p_{b_j}}{p_{u_2}}=\frac{Bu_1+u_2}{b_j}\left(1+\frac{q_2}{q_1}e^{-(q_2-q_1)}\right). p u 2 p b j = b j B u 1 + u 2 ( 1 + q 1 q 2 e − ( q 2 − q 1 ) ) .
The parenthesis lies in [ 1 , 2 ] [1,2] [ 1 , 2 ] by Lemma D.4 . Thus the second line of (93) , followed by squaring and integration, gives
1 4 − o ( 1 ) ≤ π 0 2 I u 2 ( 1 ) π j 2 I b j ( 1 ) ≤ 1 + o ( 1 ) . (94) \frac{1}{4}-o(1)\le\frac{\pi_0^2I_{u_2}(1)}{\pi_j^2I_{b_j}(1)}\le1+o(1). \tag*{(94)} 4 1 − o ( 1 ) ≤ π j 2 I b j ( 1 ) π 0 2 I u 2 ( 1 ) ≤ 1 + o ( 1 ) . (94)
It remains to control the contribution after time 1 1 1 . In the following tail estimates, constants may depend on M , Π M,\Pi M , Π , but not on ρ , z , π \rho,z,\pi ρ , z , π .
Lemma D.5 shows that parameter, gradient and moment coordinates are comparable with constants independent of ρ , z , π \rho,z,\pi ρ , z , π . Consequently, homogeneity and Euler’s identity give
q 1 ≍ ∥ θ t ∥ L , ∥ p t ∥ 2 ≍ ∥ θ t ∥ L − 1 L ≲ U ν e − U . (95) q_1\asymp\lVert\theta_t\rVert^L,\qquad\lVert p_t\rVert_2\asymp\lVert\theta_t\rVert^{L-1}\mathcal{L}\lesssim U^\nu e^{-U}. \tag*{(95)} q 1 ≍ ∥ θ t ∥ L , ∥ p t ∥ 2 ≍ ∥ θ t ∥ L − 1 L ≲ U ν e − U . (95)
We verify the conditions of Lemma C.14. Increasing the threshold for ρ \rho ρ , we may assume throughout that U ( t ) ≥ ρ L − log 2 ≥ max { 1 , 4 ν } U(t)\ge\rho^L-\log2\ge\max\{1,4\nu\} U ( t ) ≥ ρ L − log 2 ≥ max { 1 , 4 ν } . All gradients have fixed signs and are monotone decreasing in absolute value. Lemma D.5 and (95) imply that there exist k , K > 0 k,K>0 k , K > 0 such that
k U ( t ) ν e − U ( t ) ≤ p j ( t ) ≤ K U ( t ) ν e − U ( t ) , t ≥ 1 2 , j ∈ [ L + 1 ] , (96) kU(t)^\nu e^{-U(t)}\le p_j(t)\le KU(t)^\nu e^{-U(t)},\qquad t\ge\frac{1}{2},\quad j\in[L+1], \tag*{(96)} k U ( t ) ν e − U ( t ) ≤ p j ( t ) ≤ K U ( t ) ν e − U ( t ) , t ≥ 2 1 , j ∈ [ L + 1 ] , (96)
with k , K > 0 k,K>0 k , K > 0 independent of ρ , z , π \rho,z,\pi ρ , z , π . Finally, increasing K K K if necessary, the explicit initial gradients and their monotonicity give, for every coordinate j j j ,
I j ( 1 ) ≤ ( e c 2 − 1 ) p j ( 0 ) 2 ≤ K ρ 2 ( L − 1 ) e − 2 ρ L ⟶ 0. (97) I_j(1)\leq(e^{c_2}-1)p_j(0)^2\leq K\rho^{2(L-1)}e^{-2\rho^L}\longrightarrow0. \tag*{(97)} I j ( 1 ) ≤ ( e c 2 − 1 ) p j ( 0 ) 2 ≤ K ρ 2 ( L − 1 ) e − 2 ρ L ⟶ 0. (97)
For L > 1 L>1 L > 1 , Lemma C.14 yields, uniformly in z ∈ [ 0 , 3 log M ] z\in[0,3\log M] z ∈ [ 0 , 3 log M ] and π ∈ Π \pi\in\Pi π ∈ Π with M M M fixed,
ω j − I j ( 1 ) = o ( I j ( 1 ) ) ( ρ → ∞ ) . (98) \omega_j-I_j(1)=o\left(I_j(1)\right)\qquad(\rho\to\infty). \tag*{(98)} ω j − I j ( 1 ) = o ( I j ( 1 ) ) ( ρ → ∞ ) . (98)
Therefore,
ω j = I j ( 1 ) ( 1 + o ( 1 ) ) . (99) \omega_j=I_j(1)(1+o(1)). \tag*{(99)} ω j = I j ( 1 ) ( 1 + o ( 1 )) . (99)
Combining this with (94) proves (90) , after increasing ρ 1 ( M , Π ) \rho_1(M,\Pi) ρ 1 ( M , Π ) . In particular,
ω u 2 ≤ 2 I u 2 ( 1 ) \omega_{u_2}\leq2I_{u_2}(1) ω u 2 ≤ 2 I u 2 ( 1 )
for all sufficiently large ρ \rho ρ .
For L = 1 L=1 L = 1 , the gradient-scale constants for the u 2 u_2 u 2 coordinate can be chosen independently of M M M . Indeed, p u 2 = Λ − 1 e − q 1 p_{u_2}=\Lambda^{-1}e^{-q_1} p u 2 = Λ − 1 e − q 1 and q 2 ≥ q 1 q_2\geq q_1 q 2 ≥ q 1 , so
1 2 Λ L ( θ t ) ≤ p u 2 ( t ) ≤ 1 Λ L ( θ t ) . \frac{1}{2\Lambda}\mathcal{L}(\theta_t)\leq p_{u_2}(t)\leq\frac{1}{\Lambda}\mathcal{L}(\theta_t). 2Λ 1 L ( θ t ) ≤ p u 2 ( t ) ≤ Λ 1 L ( θ t ) .
Lemma C.14, applied to the u 2 u_2 u 2 coordinate, therefore implies that there exist K , K ∗ > 0 K,K_*>0 K , K ∗ > 0 such that
I ˙ u 2 ( t ) I u 2 ( t ) ≤ K η ( t ) − 2 , t ≥ 1 , ω u 2 ≤ K ∗ I u 2 ( 1 ) . \frac{\dot{I}_{u_2}(t)}{I_{u_2}(t)}\leq K\eta(t)^{-2},\qquad t\geq1,\qquad\omega_{u_2}\leq K_*I_{u_2}(1). I u 2 ( t ) I ˙ u 2 ( t ) ≤ K η ( t ) − 2 , t ≥ 1 , ω u 2 ≤ K ∗ I u 2 ( 1 ) .
with K , K ∗ K,K_* K , K ∗ independent of M , ρ , z M,\rho,z M , ρ , z . Together with the preceding argument for L > 1 L>1 L > 1 , we have proved, in every depth, with K ∗ > 0 K_*>0 K ∗ > 0 enlarged if necessary,
ω u 2 ≤ K ∗ I u 2 ( 1 ) , (100) \omega_{u_2}\leq K_*I_{u_2}(1), \tag*{(100)} ω u 2 ≤ K ∗ I u 2 ( 1 ) , (100)
where K ∗ K_* K ∗ is independent of Π , M , ρ , z \Pi,M,\rho,z Π , M , ρ , z (Π \Pi Π is void for L = 1 L=1 L = 1 ).
It remains to prove the enhancement when z = 0 z=0 z = 0 . Set
τ : = ρ 1 − L , p 0 : = p u 2 ( 0 ) = π 0 − 1 ρ L − 1 e − ρ L . \tau:=\rho^{1-L},\qquad p_0:=p_{u_2}(0)=\pi_0^{-1}\rho^{L-1}e^{-\rho^L}. τ := ρ 1 − L , p 0 := p u 2 ( 0 ) = π 0 − 1 ρ L − 1 e − ρ L .
We first bound I u 2 ( 1 ) I_{u_2}(1) I u 2 ( 1 ) from above. Since p u 2 p_{u_2} p u 2 is nonincreasing, its bias-corrected moments satisfy − m ^ u 2 ( t ) ≥ p u 2 ( t ) -\widehat{m}_{u_2}(t)\geq p_{u_2}(t) − m u 2 ( t ) ≥ p u 2 ( t ) , v ^ u 2 ( t ) ≤ p 0 \sqrt{\widehat{v}_{u_2}(t)}\leq p_0 v u 2 ( t ) ≤ p 0 , and bounding η \eta η from below on [ 0 , 1 ] [0,1] [ 0 , 1 ] implies that there exists c > 0 c>0 c > 0 such that
u ˙ 2 ( t ) ≥ c p u 2 ( t ) p 0 = c P ( b ( t ) ) P ( b ( 0 ) ) e − ( q 1 ( t ) − ρ L ) ≥ c e − ( q 1 ( t ) − ρ L ) . \dot{u}_2(t)\geq c\frac{p_{u_2}(t)}{p_0}=c\frac{P(\mathbf{b}(t))}{P(\mathbf{b}(0))}e^{-(q_1(t)-\rho^L)}\geq ce^{-(q_1(t)-\rho^L)}. u ˙ 2 ( t ) ≥ c p 0 p u 2 ( t ) = c P ( b ( 0 )) P ( b ( t )) e − ( q 1 ( t ) − ρ L ) ≥ c e − ( q 1 ( t ) − ρ L ) .
Since P P P is nondecreasing and P ( b ( 0 ) ) / Λ = π 0 − 1 τ − 1 P(\mathbf{b}(0))/\Lambda=\pi_0^{-1}\tau^{-1} P ( b ( 0 )) /Λ = π 0 − 1 τ − 1 , there exists c Π > 0 c_\Pi>0 c Π > 0 such that
q ˙ 1 ( t ) ≥ P ( b ( t ) ) Λ u ˙ 2 ( t ) ≥ c Π τ − 1 e − ( q 1 ( t ) − ρ L ) . \dot{q}_1(t)\geq\frac{P(\mathbf{b}(t))}{\Lambda}\dot{u}_2(t)\geq c_\Pi\tau^{-1}e^{-(q_1(t)-\rho^L)}. q ˙ 1 ( t ) ≥ Λ P ( b ( t )) u ˙ 2 ( t ) ≥ c Π τ − 1 e − ( q 1 ( t ) − ρ L ) .
On the other hand, (93) implies that there exists C Π > 0 C_\Pi>0 C Π > 0 such that
p u 2 ( t ) = p 0 P ( b ( t ) ) P ( b ( 0 ) ) e − ( q 1 ( t ) − ρ L ) ≤ C Π p 0 e − ( q 1 ( t ) − ρ L ) . p_{u_2}(t)=p_0\frac{P(\mathbf{b}(t))}{P(\mathbf{b}(0))}e^{-(q_1(t)-\rho^L)}\leq C_\Pi p_0e^{-(q_1(t)-\rho^L)}. p u 2 ( t ) = p 0 P ( b ( 0 )) P ( b ( t )) e − ( q 1 ( t ) − ρ L ) ≤ C Π p 0 e − ( q 1 ( t ) − ρ L ) .
Consequently, for some C Π > 0 C_\Pi>0 C Π > 0 ,
I u 2 ( 1 ) ≤ C Π p 0 2 ∫ 0 1 e − 2 ( q 1 ( t ) − ρ L ) d t ≤ C Π τ p 0 2 ∫ 0 1 q ˙ 1 ( t ) e − ( q 1 ( t ) − ρ L ) d t ≤ C Π τ p 0 2 . I_{u_2}(1)\leq C_\Pi p_0^2\int_0^1e^{-2(q_1(t)-\rho^L)}\,dt\leq C_\Pi\tau p_0^2\int_0^1\dot{q}_1(t)e^{-(q_1(t)-\rho^L)}\,dt\leq C_\Pi\tau p_0^2. I u 2 ( 1 ) ≤ C Π p 0 2 ∫ 0 1 e − 2 ( q 1 ( t ) − ρ L ) d t ≤ C Π τ p 0 2 ∫ 0 1 q ˙ 1 ( t ) e − ( q 1 ( t ) − ρ L ) d t ≤ C Π τ p 0 2 .
By (100) , for some C Π > 0 C_\Pi>0 C Π > 0 , this implies
ω u 2 ≤ C Π τ p 0 2 . (101) \omega_{u_2}\leq C_\Pi\tau p_0^2. \tag*{(101)} ω u 2 ≤ C Π τ p 0 2 . (101)
Now assume z = 0 z=0 z = 0 , so that q 2 ( 0 ) = ρ L q_2(0)=\rho^L q 2 ( 0 ) = ρ L . Since q 2 ≤ ( M / B ) q 1 q_2\le(M/B)q_1 q 2 ≤ ( M / B ) q 1 , for some C Π > 0 C_{\Pi}>0 C Π > 0 , the fixed-time parameter estimates give
P ( b ( t ) ) Λ ≤ C Π τ − 1 , q 2 ( t ) b j ( t ) ≤ M B q 1 ( t ) b j ( t ) ≤ C Π M τ − 1 . \frac{P(\mathbf{b}(t))}{\Lambda}\le C_{\Pi}\tau^{-1},\qquad\frac{q_2(t)}{b_j(t)}\le\frac{M}{B}\frac{q_1(t)}{b_j(t)}\le C_{\Pi}M\tau^{-1}. Λ P ( b ( t )) ≤ C Π τ − 1 , b j ( t ) q 2 ( t ) ≤ B M b j ( t ) q 1 ( t ) ≤ C Π M τ − 1 .
For some C Π > 0 C_{\Pi}>0 C Π > 0 , the universal velocity bound therefore yields
q ˙ 2 ( t ) = M Λ P ( b ( t ) ) u ˙ 1 ( t ) + ∑ j = 1 L − 1 q 2 ( t ) b j ( t ) b ˙ j ( t ) ≤ C Π M τ − 1 , 0 ≤ t ≤ 1. \dot{q}_2(t)=\frac{M}{\Lambda}P(\mathbf{b}(t))\dot{u}_1(t)+\sum_{j=1}^{L-1}\frac{q_2(t)}{b_j(t)}\dot{b}_j(t)\le C_{\Pi}M\tau^{-1},\qquad0\le t\le1. q ˙ 2 ( t ) = Λ M P ( b ( t )) u ˙ 1 ( t ) + j = 1 ∑ L − 1 b j ( t ) q 2 ( t ) b ˙ j ( t ) ≤ C Π M τ − 1 , 0 ≤ t ≤ 1.
There exists b Π ∈ ( 0 , 1 ] b_{\Pi}\in(0,1] b Π ∈ ( 0 , 1 ] sufficiently small such that
q 2 ( t ) ≤ ρ L + 1 , 0 ≤ t ≤ b Π τ / M . q_2(t)\le\rho^L+1,\qquad0\le t\le b_{\Pi}\tau/M. q 2 ( t ) ≤ ρ L + 1 , 0 ≤ t ≤ b Π τ / M .
Since P P P is nondecreasing, on this interval
p u 1 ( t ) ≥ M Λ P ( b ( t ) ) e − q 2 ( t ) ≥ e − 1 M p 0 . p_{u_1}(t)\ge\frac{M}{\Lambda}P(\mathbf{b}(t))e^{-q_2(t)}\ge e^{-1}Mp_0. p u 1 ( t ) ≥ Λ M P ( b ( t )) e − q 2 ( t ) ≥ e − 1 M p 0 .
Hence
ω u 1 ≥ c 2 ∫ 0 b Π τ / M p u 1 ( t ) 2 d t ≥ c 2 b Π e − 2 M τ p 0 2 . \omega_{u_1}\ge c_2\int_0^{b_{\Pi}\tau/M}p_{u_1}(t)^2\,dt\ge c_2b_{\Pi}e^{-2}M\tau p_0^2. ω u 1 ≥ c 2 ∫ 0 b Π τ / M p u 1 ( t ) 2 d t ≥ c 2 b Π e − 2 M τ p 0 2 .
Dividing by (101) proves (92) . All constants are independent of M M M . For L = 1 L=1 L = 1 , the same argument applies with τ = 1 \tau=1 τ = 1 and with the empty products and sums interpreted as 1 1 1 and 0 0 0 , respectively.
Lemma D.9 (Continuity of Second Moment Weights). Fix M ≥ M 0 ( B ) M\ge M_0(B) M ≥ M 0 ( B ) and a compact Π ⊆ ( 0 , ∞ ) L − 1 \Pi\subseteq(0,\infty)^{L-1} Π ⊆ ( 0 , ∞ ) L − 1 . If ρ ≥ ρ 1 ( M , Π ) \rho\ge\rho_1(M,\Pi) ρ ≥ ρ 1 ( M , Π ) , then ω u 1 , ω u 2 \omega_{u_1},\omega_{u_2} ω u 1 , ω u 2 and every ω b j \omega_{b_j} ω b j are continuous in ( z , π ) ∈ [ 0 , 3 log M ] × Π (z,\pi)\in[0,3\log M]\times\Pi ( z , π ) ∈ [ 0 , 3 log M ] × Π .
Proof. Write σ = ( z , π ) ∈ [ 0 , 3 log M ] × Π \sigma=(z,\pi)\in[0,3\log M]\times\Pi σ = ( z , π ) ∈ [ 0 , 3 log M ] × Π . We first show continuity of the finite-time weights, and then show that they converge uniformly to ω i \omega_i ω i .
Fix T < ∞ T<\infty T < ∞ . The initialization depends continuously on σ \sigma σ , and the universal Adam-ratio bound gives
0 ≤ θ ˙ i ( t ; σ ) ≤ K A η ( t ) . 0\le\dot{\theta}_i(t;\sigma)\le K_A\eta(t). 0 ≤ θ ˙ i ( t ; σ ) ≤ K A η ( t ) .
Thus all trajectories on [ 0 , T ] [0,T] [ 0 , T ] lie in one compact rectangle R T ⊆ ( 0 , ∞ ) L + 1 R_T\subseteq(0,\infty)^{L+1} R T ⊆ ( 0 , ∞ ) L + 1 . On this rectangle the gradient coordinates p i p_i p i are Lipschitz, and there exist a T , b T > 0 a_T,b_T>0 a T , b T > 0 such that
0 < a T ≤ p i ≤ b T < ∞ . 0<a_T\le p_i\le b_T<\infty. 0 < a T ≤ p i ≤ b T < ∞.
The normalized EMA is a contraction in the supremum norm. Moreover, if p , p ~ p,\widetilde{p} p , p take values in [ a T , b T ] [a_T,b_T] [ a T , b T ] , then their second-moment EMAs are at least a T 2 a_T^2 a T 2 . Consequently, denoting m = EMA ^ c 1 [ p ] m=\widehat{\operatorname{EMA}}_{c_1}[p] m = EMA c 1 [ p ] , v = EMA ^ c 2 [ p 2 ] v=\widehat{\operatorname{EMA}}_{c_2}[p^2] v = EMA c 2 [ p 2 ] and m ~ = EMA ^ c 1 [ p ~ ] \widetilde{m}=\widehat{\operatorname{EMA}}_{c_1}[\widetilde{p}] m = EMA c 1 [ p ] , v ~ = EMA ^ c 2 [ p ~ 2 ] \widetilde{v}=\widehat{\operatorname{EMA}}_{c_2}[\widetilde{p}^2] v = EMA c 2 [ p 2 ] ,
∣ A c 1 , c 2 [ p ] ( t ) − A c 1 , c 2 [ p ~ ] ( t ) ∣ ≤ ∣ m − m ~ ∣ v + ∣ m ~ ∣ ∣ 1 v − 1 v ~ ∣ ≤ 1 a T ∥ p − p ~ ∥ ∞ , [ 0 , t ] + b T 2 a T 3 ∥ p 2 − p ~ 2 ∥ ∞ , [ 0 , t ] ≤ ( 1 a T + b T 2 a T 3 ) ∥ p − p ~ ∥ ∞ , [ 0 , t ] . (102) \begin{aligned}
\left|A_{c_1,c_2}[p](t)-A_{c_1,c_2}[\widetilde{p}](t)\right|
&\le\frac{|m-\widetilde{m}|}{\sqrt{v}}+|\widetilde{m}|\left|\frac{1}{\sqrt{v}}-\frac{1}{\sqrt{\widetilde{v}}}\right|\\
&\le\frac{1}{a_T}\|p-\widetilde{p}\|_{\infty,[0,t]}+\frac{b_T}{2a_T^3}\|p^2-\widetilde{p}^2\|_{\infty,[0,t]}\\
&\le\left(\frac{1}{a_T}+\frac{b_T^2}{a_T^3}\right)\|p-\widetilde{p}\|_{\infty,[0,t]}. \tag*{(102)}
\end{aligned} ∣ A c 1 , c 2 [ p ] ( t ) − A c 1 , c 2 [ p ] ( t ) ∣ ≤ v ∣ m − m ∣ + ∣ m ∣ v 1 − v 1 ≤ a T 1 ∥ p − p ∥ ∞ , [ 0 , t ] + 2 a T 3 b T ∥ p 2 − p 2 ∥ ∞ , [ 0 , t ] ≤ ( a T 1 + a T 3 b T 2 ) ∥ p − p ∥ ∞ , [ 0 , t ] . (102)
At t = 0 t=0 t = 0 , both Adam ratios are defined to be 1 1 1 , so the same estimate holds there. Using the integral representation
θ i ( t ; σ ) = θ i ( 0 ; σ ) + ∫ 0 t η ( s ) A c 1 , c 2 [ p i ( ⋅ ; σ ) ] ( s ) d s . (103) \theta_i(t;\sigma)=\theta_i(0;\sigma)+\int_0^t\eta(s)A_{c_1,c_2}[p_i(\cdot;\sigma)](s)\,ds. \tag*{(103)} θ i ( t ; σ ) = θ i ( 0 ; σ ) + ∫ 0 t η ( s ) A c 1 , c 2 [ p i ( ⋅ ; σ )] ( s ) d s . (103)
set
ε ( t ) : = sup 0 ≤ s ≤ t ∥ θ ( s ; σ ) − θ ( s ; σ ~ ) ∥ ∞ . \varepsilon(t) := \sup_{0 \le s \le t} \lVert\theta(s;\sigma)-\theta(s;\widetilde{\sigma}) \rVert_{\infty}. ε ( t ) := 0 ≤ s ≤ t sup ∥ θ ( s ; σ ) − θ ( s ; σ ) ∥ ∞ .
Subtracting the two integral equations and using (102) and Lipschitzness of the p i p_i p i shows that there exists K T > 0 K_T>0 K T > 0 such that
ε ( t ) ≤ ε ( 0 ) + K T ∫ 0 t η ( s ) ε ( s ) d s . \varepsilon(t) \le\varepsilon(0) + K_T \int_0^t \eta(s)\varepsilon(s)\,ds. ε ( t ) ≤ ε ( 0 ) + K T ∫ 0 t η ( s ) ε ( s ) d s .
Gronwall’s inequality therefore yields
ε ( T ) ≤ ε ( 0 ) exp ( K T ∫ 0 T η ( s ) d s ) → σ → σ ~ 0. \varepsilon(T) \le\varepsilon(0)\exp\left(K_T\int_0^T\eta(s)\,ds\right)\xrightarrow{\sigma\to\widetilde{\sigma}}0. ε ( T ) ≤ ε ( 0 ) exp ( K T ∫ 0 T η ( s ) d s ) σ → σ 0.
Thus the trajectories, and hence the gradients, depend continuously on σ \sigma σ in the supremum norm on [ 0 , T ] [0,T] [ 0 , T ] . In particular, every map
σ ⟼ I i ( T ; σ ) = c 2 ∫ 0 T e c 2 t p i ( t ; σ ) 2 d t \sigma\longmapsto I_i(T;\sigma)=c_2\int_0^T e^{c_2t}p_i(t;\sigma)^2\,dt σ ⟼ I i ( T ; σ ) = c 2 ∫ 0 T e c 2 t p i ( t ; σ ) 2 d t
is continuous.
For the tail, Lemma C.14, whose hypotheses were verified in the preceding proof, implies that there exists K > 0 K>0 K > 0 such that, uniformly in σ \sigma σ , for every coordinate i i i ,
I ˙ i ( t ; σ ) I i ( t ; σ ) ≤ K ( 1 + t ) − 2 ζ , t ≥ 1. (104) \frac{\dot I_i(t;\sigma)}{I_i(t;\sigma)}\le K(1+t)^{-2\zeta},\qquad t\ge1. \tag*{(104)} I i ( t ; σ ) I ˙ i ( t ; σ ) ≤ K ( 1 + t ) − 2 ζ , t ≥ 1. (104)
Finite-time continuity and compactness give sup σ , i I i ( 1 ; σ ) < ∞ \sup_{\sigma,i}I_i(1;\sigma)<\infty sup σ , i I i ( 1 ; σ ) < ∞ . The same lemma gives, for every coordinate i i i and T ≥ 1 T\ge1 T ≥ 1 ,
0 ≤ ω i ( σ ) − I i ( T ; σ ) ≤ K I i ( 1 ; σ ) ( 1 + T ) 1 − 2 ζ ≤ K sup σ , i I i ( 1 ; σ ) ( 1 + T ) 1 − 2 ζ . 0\le\omega_i(\sigma)-I_i(T;\sigma)\le K I_i(1;\sigma)(1+T)^{1-2\zeta}
\le K\sup_{\sigma,i}I_i(1;\sigma)(1+T)^{1-2\zeta}. 0 ≤ ω i ( σ ) − I i ( T ; σ ) ≤ K I i ( 1 ; σ ) ( 1 + T ) 1 − 2 ζ ≤ K σ , i sup I i ( 1 ; σ ) ( 1 + T ) 1 − 2 ζ .
The right-hand side tends to zero independently of σ \sigma σ . Therefore ω i \omega_i ω i is a uniform limit of the continuous maps I i ( T ; ⋅ ) I_i(T;\cdot) I i ( T ; ⋅ ) , and is continuous. □
Lemma D.10 (Frozen weights determine the limiting direction). Let δ 0 = B ω u 2 ω u 1 \delta_0=B\sqrt{\frac{\omega_{u_2}}{\omega_{u_1}}} δ 0 = B ω u 1 ω u 2 and let M ≥ M 0 ( B ) M\ge M_0(B) M ≥ M 0 ( B ) . Then
θ t u 2 ( t ) → t → ∞ ( δ 0 , 1 , δ 1 , … , δ L − 1 ) , δ j = 1 + B δ 0 ( ω u 2 ω b j ) 1 / 4 ( j ≥ 1 ) . \frac{\theta_t}{u_2(t)}\xrightarrow{t\to\infty}(\delta_0,1,\delta_1,\ldots,\delta_{L-1}),\qquad
\delta_j=\sqrt{1+B\delta_0}\left(\frac{\omega_{u_2}}{\omega_{b_j}}\right)^{1/4}\quad(j\ge1). u 2 ( t ) θ t t → ∞ ( δ 0 , 1 , δ 1 , … , δ L − 1 ) , δ j = 1 + B δ 0 ( ω b j ω u 2 ) 1/4 ( j ≥ 1 ) . Therefore the margin-normalized limit is
θ t q min 1 / L ( t ) = θ t q 1 1 / L ( t ) → t → ∞ ( ∏ j = 1 L − 1 δ j B δ 0 + 1 Λ ) − 1 / L ( δ 0 , 1 , δ 1 , … , δ L − 1 ) . \frac{\theta_t}{q_{\min}^{1/L}(t)}
=
\frac{\theta_t}{q_1^{1/L}(t)}
\xrightarrow{t\to\infty}
\left(\prod_{j=1}^{L-1}\delta_j\frac{B\delta_0+1}{\Lambda}\right)^{-1/L}
(\delta_0,1,\delta_1,\ldots,\delta_{L-1}). q m i n 1/ L ( t ) θ t = q 1 1/ L ( t ) θ t t → ∞ ( j = 1 ∏ L − 1 δ j Λ B δ 0 + 1 ) − 1/ L ( δ 0 , 1 , δ 1 , … , δ L − 1 ) .
Proof. By Corollary D.7 ,
p u 1 p u 2 = B + M e − ( q 2 − q 1 ) t → t → ∞ B . \frac{p_{u_1}}{p_{u_2}}=B+Me^{-(q_2-q_1)t}\xrightarrow{t\to\infty}B. p u 2 p u 1 = B + M e − ( q 2 − q 1 ) t t → ∞ B .
Moreover, q 1 = c 2 t / 2 + o ( t ) q_1=c_2t/2+o(t) q 1 = c 2 t /2 + o ( t ) , so Lemma D.5 and q 1 = P ( b ) ( B u 1 + u 2 ) / Λ q_1=P(\mathbf b)(Bu_1+u_2)/\Lambda q 1 = P ( b ) ( B u 1 + u 2 ) /Λ give
θ j = Θ ( t 1 / L ) , j ∈ [ L + 1 ] . \theta_j=\Theta(t^{1/L}),\qquad j\in[L+1]. θ j = Θ ( t 1/ L ) , j ∈ [ L + 1 ] .
Since p u 2 = P ( b ) e − q 1 / Λ p_{u_2}=P(\mathbf b)e^{-q_1}/\Lambda p u 2 = P ( b ) e − q 1 /Λ and P ( b ) = Θ ( t 1 − 1 / L ) P(\mathbf b)=\Theta(t^{1-1/L}) P ( b ) = Θ ( t 1 − 1/ L ) , we have
log ( e c 1 t p u 2 ( t ) ) = ( c 1 − c 2 2 ) t + o ( t ) . \log\left(e^{c_1t}p_{u_2}(t)\right)=\left(c_1-\frac{c_2}{2}\right)t+o(t). log ( e c 1 t p u 2 ( t ) ) = ( c 1 − 2 c 2 ) t + o ( t ) .
As c 1 > c 2 / 2 c_1 > c_2/2 c 1 > c 2 /2 , this implies ∫ 0 ∞ e c 1 s p u 2 ( s ) d s = ∞ \int_0^\infty e^{c_1s}p_{u_2}(s)\,\mathrm{d}s = \infty ∫ 0 ∞ e c 1 s p u 2 ( s ) d s = ∞ . It therefore holds that
m u 1 m u 2 → t → ∞ B . \frac{m_{u_1}}{m_{u_2}} \xrightarrow{t\to\infty} B. m u 2 m u 1 t → ∞ B .
And so
u ˙ 1 u ˙ 2 → t → ∞ B ω u 2 ω u 1 = δ 0 . \frac{\dot{u}_1}{\dot{u}_2} \xrightarrow{t\to\infty} B\sqrt{\frac{\omega_{u_2}}{\omega_{u_1}}} = \delta_0. u ˙ 2 u ˙ 1 t → ∞ B ω u 1 ω u 2 = δ 0 .
Recall u 1 , u 2 → t → ∞ ∞ u_1,u_2 \xrightarrow{t\to\infty} \infty u 1 , u 2 t → ∞ ∞ , implying by L’Hospital’s rule that
u 1 u 2 → t → ∞ δ 0 . \frac{u_1}{u_2} \xrightarrow{t\to\infty} \delta_0. u 2 u 1 t → ∞ δ 0 .
For the parameters b j b_j b j , recall that b j = Θ ( t 1 / L ) b_j = \Theta(t^{1/L}) b j = Θ ( t 1/ L ) for each j j j . Moreover,
p b j p u 2 = Λ ξ 1 b j ( 1 + q 2 q 1 e − ( q 2 − q 1 ) ) = Λ ξ 1 b j ( 1 + o ( 1 ) ) . \frac{p_{b_j}}{p_{u_2}} = \frac{\Lambda\xi_1}{b_j}\left(1+\frac{q_2}{q_1}e^{-(q_2-q_1)}\right) = \frac{\Lambda\xi_1}{b_j}(1+o(1)). p u 2 p b j = b j Λ ξ 1 ( 1 + q 1 q 2 e − ( q 2 − q 1 ) ) = b j Λ ξ 1 ( 1 + o ( 1 )) .
Indeed q 2 = O ( t ) q_2 = O(t) q 2 = O ( t ) by the parameter bounds while q 1 = Θ ( t ) q_1 = \Theta(t) q 1 = Θ ( t ) , so q 2 / q 1 = O ( 1 ) q_2/q_1 = O(1) q 2 / q 1 = O ( 1 ) ; the extra term tends to zero because q 2 − q 1 → ∞ q_2-q_1 \to\infty q 2 − q 1 → ∞ . We justify the use of Lemma B.6 with g = p u 2 g=p_{u_2} g = p u 2 and A = Λ ξ 1 b j A=\frac{\Lambda\xi_1}{b_j} A = b j Λ ξ 1 . The preceding logarithmic estimate verifies its first condition. Also, since parameter time derivatives and parameters are respectively O ( t 1 / L − 1 ) , Ω ( t 1 / L ) O(t^{1/L-1}),\Omega(t^{1/L}) O ( t 1/ L − 1 ) , Ω ( t 1/ L ) ,
∣ d d t log ξ 1 b j ∣ ≤ ∣ d d t log ξ 1 ∣ + ∣ d d t log b j ∣ ≤ O ( 1 t ) . \left|\frac{d}{dt}\log\frac{\xi_1}{b_j}\right| \le\left|\frac{d}{dt}\log\xi_1\right|+\left|\frac{d}{dt}\log b_j\right| \le O\left(\frac{1}{t}\right). d t d log b j ξ 1 ≤ d t d log ξ 1 + d t d log b j ≤ O ( t 1 ) .
Therefore, by Lemma B.6,
m b j m u 2 = Λ ξ 1 b j ( 1 + o ( 1 ) ) . \frac{m_{b_j}}{m_{u_2}} = \frac{\Lambda\xi_1}{b_j}(1+o(1)). m u 2 m b j = b j Λ ξ 1 ( 1 + o ( 1 )) .
Thus,
b ˙ j u ˙ 2 = ω u 2 ω b j Λ ξ 1 b j ( 1 + o ( 1 ) ) . \frac{\dot{b}_j}{\dot{u}_2} = \sqrt{\frac{\omega_{u_2}}{\omega_{b_j}}}\frac{\Lambda\xi_1}{b_j}(1+o(1)). u ˙ 2 b ˙ j = ω b j ω u 2 b j Λ ξ 1 ( 1 + o ( 1 )) .
Hence,
d d t b j 2 d d t u 2 2 = b j b ˙ j u 2 u ˙ 2 = ω u 2 ω b j Λ ξ 1 u 2 ( 1 + o ( 1 ) ) → t → ∞ ( B δ 0 + 1 ) ω u 2 ω b j . \frac{\frac{d}{dt}b_j^2}{\frac{d}{dt}u_2^2} = \frac{b_j\dot{b}_j}{u_2\dot{u}_2} = \sqrt{\frac{\omega_{u_2}}{\omega_{b_j}}}\frac{\Lambda\xi_1}{u_2}(1+o(1)) \xrightarrow{t\to\infty} (B\delta_0+1)\sqrt{\frac{\omega_{u_2}}{\omega_{b_j}}}. d t d u 2 2 d t d b j 2 = u 2 u ˙ 2 b j b ˙ j = ω b j ω u 2 u 2 Λ ξ 1 ( 1 + o ( 1 )) t → ∞ ( B δ 0 + 1 ) ω b j ω u 2 .
Both b j 2 b_j^2 b j 2 and u 2 2 u_2^2 u 2 2 tend to infinity, so again applying L’Hospital’s rule and taking the root gives
b j u 2 → t → ∞ B δ 0 + 1 ( ω u 2 ω b j ) 1 / 4 = δ j . \frac{b_j}{u_2} \xrightarrow{t\to\infty} \sqrt{B\delta_0+1}\left(\frac{\omega_{u_2}}{\omega_{b_j}}\right)^{1/4} = \delta_j. u 2 b j t → ∞ B δ 0 + 1 ( ω b j ω u 2 ) 1/4 = δ j .
Finally, recalling that q min = q 1 q_{\min}=q_1 q m i n = q 1 , the margin-normalized limit is simply obtained by normalizing θ ˉ = ( δ 0 , 1 , δ 1 , … , δ L − 1 ) \bar{\theta}=(\delta_0,1,\delta_1,\ldots,\delta_{L-1}) θ ˉ = ( δ 0 , 1 , δ 1 , … , δ L − 1 ) by q min 1 / L ( θ ˉ ) = q 1 1 / L ( θ ˉ ) q_{\min}^{1/L}(\bar{\theta})=q_1^{1/L}(\bar{\theta}) q m i n 1/ L ( θ ˉ ) = q 1 1/ L ( θ ˉ ) :
θ ^ = lim t → ∞ θ t q min 1 / L ( θ t ) = θ ˉ q 1 1 / L ( θ ˉ ) = ( 1 Λ ( B δ 0 + 1 ) ∏ j = 1 L − 1 δ j ) − 1 / L ( δ 0 , 1 , δ 1 , … , δ L − 1 ) . \hat{\theta}=\lim_{t\to\infty}\frac{\theta_t}{q_{\min}^{1/L}(\theta_t)}=\frac{\bar{\theta}}{q_1^{1/L}(\bar{\theta})}=\left(\frac{1}{\Lambda}(B\delta_0+1)\prod_{j=1}^{L-1}\delta_j\right)^{-1/L}(\delta_0,1,\delta_1,\ldots,\delta_{L-1}). θ ^ = t → ∞ lim q m i n 1/ L ( θ t ) θ t = q 1 1/ L ( θ ˉ ) θ ˉ = ( Λ 1 ( B δ 0 + 1 ) j = 1 ∏ L − 1 δ j ) − 1/ L ( δ 0 , 1 , δ 1 , … , δ L − 1 ) .
□
Direction Reachability
We now prove Proposition D.3. Note that we have already proved a dependence of the convergent direction on the initialization, but we are still missing an argument showing that all desired directions on Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ are indeed reached. We use (without proof) the Poincare-Miranda Theorem, a generalization of the intermediate value theorem:
Theorem D.11 (Poincare-Miranda). Let F = ( F 1 , … , F n ) : B → R n F=(F_1,\ldots,F_n):\mathcal{B}\to\mathbb{R}^n F = ( F 1 , … , F n ) : B → R n be continuous where B = ∏ j = 1 n [ a j , b j ] ⊆ R n \mathcal{B}=\prod_{j=1}^{n}[a_j,b_j]\subseteq\mathbb{R}^n B = ∏ j = 1 n [ a j , b j ] ⊆ R n for a j < b j a_j<b_j a j < b j , j ∈ [ n ] j\in[n] j ∈ [ n ] . Let y ∈ R n y\in\mathbb{R}^n y ∈ R n . Assume
∀ j ∈ [ n ] , x = ( x 1 , … , x n ) ∈ B : x j = a j ⟹ F j ( x ) ≤ y j x j = b j ⟹ F j ( x ) ≥ y j \begin{aligned}
&\forall j\in[n],\ \mathbf{x}=(x_1,\ldots,x_n)\in\mathcal{B}: x_j=a_j \Longrightarrow F_j(\mathbf{x})\le y_j\\
&x_j=b_j \Longrightarrow F_j(\mathbf{x})\ge y_j
\end{aligned} ∀ j ∈ [ n ] , x = ( x 1 , … , x n ) ∈ B : x j = a j ⟹ F j ( x ) ≤ y j x j = b j ⟹ F j ( x ) ≥ y j Then there exists x ∈ B \mathbf{x}\in\mathcal{B} x ∈ B with F ( x ) = y F(\mathbf{x})=y F ( x ) = y .
Proof of Proposition D.3. Fix B > 1 B>1 B > 1 , Λ > 0 \Lambda>0 Λ > 0 , and a nonempty compact K ⊆ Σ B , Λ K\subseteq\Sigma_{B,\Lambda} K ⊆ Σ B , Λ . For θ ^ = ( u ^ 1 , u ^ 2 , b ^ 1 , … , b ^ L − 1 ) ∈ K \hat{\theta}=(\hat{u}_1,\hat{u}_2,\hat{b}_1,\ldots,\hat{b}_{L-1})\in K θ ^ = ( u ^ 1 , u ^ 2 , b ^ 1 , … , b ^ L − 1 ) ∈ K , set
δ 0 : = u ^ 1 u ^ 2 ∈ ( 0 , 1 ) , δ j : = b ^ j u ^ 2 > 0 , j ∈ [ L − 1 ] , \delta_0:=\frac{\hat{u}_1}{\hat{u}_2}\in(0,1),\qquad\delta_j:=\frac{\hat{b}_j}{\hat{u}_2}>0,\qquad j\in[L-1], δ 0 := u ^ 2 u ^ 1 ∈ ( 0 , 1 ) , δ j := u ^ 2 b ^ j > 0 , j ∈ [ L − 1 ] ,
and denote
θ ˉ : = ( δ 0 , 1 , δ 1 , … , δ L − 1 ) . \bar{\theta}:=(\delta_0,1,\delta_1,\ldots,\delta_{L-1}). θ ˉ := ( δ 0 , 1 , δ 1 , … , δ L − 1 ) .
The ratios depend continuously on θ ^ ∈ K \hat{\theta}\in K θ ^ ∈ K . Hence
0 < δ − : = min K δ 0 ≤ max K δ 0 = : δ + < 1. 0<\delta_-:=\min_K\delta_0\le\max_K\delta_0=:\delta_+<1. 0 < δ − := K min δ 0 ≤ K max δ 0 =: δ + < 1.
We will choose an initialization whose limiting direction is θ ˉ \bar{\theta} θ ˉ , and then apply Lemma D.10 to obtain the desired margin-normalized limit.
We first restrict ourselves to a compact set of initialization parameters. Use the coordinates
ι j : = 1 2 log π j π 0 , π 0 = Λ ∏ j = 1 L − 1 π j . \iota_j:=\frac{1}{2}\log\frac{\pi_j}{\pi_0},\qquad\pi_0=\frac{\Lambda}{\prod_{j=1}^{L-1}\pi_j}. ι j := 2 1 log π 0 π j , π 0 = ∏ j = 1 L − 1 π j Λ .
This change of coordinates is invertible, with
π 0 ( ι ) = Λ 1 / L exp ( − 2 L ∑ j = 1 L − 1 ι j ) , π j ( ι ) = π 0 ( ι ) e 2 ι j . \pi_0(\iota)=\Lambda^{1/L}\exp\left(-\frac{2}{L}\sum_{j=1}^{L-1}\iota_j\right),\qquad\pi_j(\iota)=\pi_0(\iota)e^{2\iota_j}. π 0 ( ι ) = Λ 1/ L exp ( − L 2 j = 1 ∑ L − 1 ι j ) , π j ( ι ) = π 0 ( ι ) e 2 ι j .
For each j ∈ [ L − 1 ] j\in[L-1] j ∈ [ L − 1 ] , set
ι j ∗ ( θ ^ ) : = log δ j − 1 2 log ( 1 + B δ 0 ) , \iota_j^*(\hat{\theta}):=\log\delta_j-\frac{1}{2}\log(1+B\delta_0), ι j ∗ ( θ ^ ) := log δ j − 2 1 log ( 1 + B δ 0 ) ,
and choose
A j > 1 2 log ( 1 + B ) + 1 4 log 8. A_j>\frac{1}{2}\log(1+B)+\frac{1}{4}\log8. A j > 2 1 log ( 1 + B ) + 4 1 log 8.
Define
J j : = [ min K ι j ∗ − A j , max K ι j ∗ + A j ] , Π : = { π ( ι ) ∣ ι ∈ ∏ j = 1 L − 1 J j } . J_j:=\left[\min_K\iota_j^*-A_j,\max_K\iota_j^*+A_j\right],\qquad\Pi:=\left\{\pi(\iota)\mid\iota\in\prod_{j=1}^{L-1}J_j\right\}. J j := [ K min ι j ∗ − A j , K max ι j ∗ + A j ] , Π := { π ( ι ) ∣ ι ∈ j = 1 ∏ L − 1 J j } .
The set Π \Pi Π is a compact subset of ( 0 , ∞ ) L − 1 (0,\infty)^{L-1} ( 0 , ∞ ) L − 1 and is independent of M M M . Let k Π > 0 k_\Pi>0 k Π > 0 be the constant from Lemma D.8. Since this constant is independent of M M M , we may choose M ∗ > max { 1 , M 0 ( B ) } M_*> \max\{1,M_0(B)\} M ∗ > max { 1 , M 0 ( B )} so that, for every M ≥ M ∗ M\ge M_* M ≥ M ∗ ,
B k Π M < δ − , 1 1 + ( B M 2 ) − 1 > δ + . \frac{B}{\sqrt{k_\Pi M}}<\delta_-,\qquad\frac{1}{1+(BM^2)^{-1}}>\delta_+. k Π M B < δ − , 1 + ( B M 2 ) − 1 1 > δ + .
Fix any such M M M , and then fix any ρ ≥ ρ 1 ( M , Π ) \rho\ge\rho_1(M,\Pi) ρ ≥ ρ 1 ( M , Π ) . We now construct the continuous map to which we apply Theorem D.11 . On the box
B : = [ 0 , 3 log M ] × ∏ j = 1 L − 1 J j , \mathcal{B} := [0,3\log M] \times\prod_{j=1}^{L-1} J_j, B := [ 0 , 3 log M ] × j = 1 ∏ L − 1 J j ,
write ω i ( z , ι ) \omega_i(z,\iota) ω i ( z , ι ) for the weight corresponding to the initialization ( ρ , z , π ( ι ) ) (\rho,z,\pi(\iota)) ( ρ , z , π ( ι )) , and define
d 0 ( z , ι ) : = B ω u 2 ( z , ι ) ω u 1 ( z , ι ) , d_0(z,\iota) := B\sqrt{\frac{\omega_{u_2}(z,\iota)}{\omega_{u_1}(z,\iota)}}, d 0 ( z , ι ) := B ω u 1 ( z , ι ) ω u 2 ( z , ι ) ,
d j ( z , ι ) : = 1 + B d 0 ( z , ι ) ( ω u 2 ( z , ι ) ω b j ( z , ι ) ) 1 / 4 , j ∈ [ L − 1 ] . d_j(z,\iota) := \sqrt{1+Bd_0(z,\iota)}\left(\frac{\omega_{u_2}(z,\iota)}{\omega_{b_j}(z,\iota)}\right)^{1/4}, \qquad j\in[L-1]. d j ( z , ι ) := 1 + B d 0 ( z , ι ) ( ω b j ( z , ι ) ω u 2 ( z , ι ) ) 1/4 , j ∈ [ L − 1 ] .
All weights are positive and continuous by Lemma D.9 , so d = ( d 0 , … , d L − 1 ) d=(d_0,\ldots,d_{L-1}) d = ( d 0 , … , d L − 1 ) is continuous on B \mathcal{B} B .
Now fix any target θ ^ ∈ K \hat{\theta}\in K θ ^ ∈ K , suppressing its dependence in δ j \delta_j δ j and ι j ∗ \iota_j^* ι j ∗ . We first check the two faces in the z z z coordinate. Lemma D.8 gives
d 0 ( 0 , ι ) ≤ B κ Π M < δ 0 . d_0(0,\iota)\le\frac{B}{\sqrt{\kappa_{\Pi}M}}<\delta_0. d 0 ( 0 , ι ) ≤ κ Π M B < δ 0 .
At z = 3 log M z=3\log M z = 3 log M , the same lemma gives
ω u 1 ω u 2 ≤ ( B + M e − 3 log M ) 2 = ( B + M − 2 ) 2 , \frac{\omega_{u_1}}{\omega_{u_2}}\le(B+Me^{-3\log M})^2=(B+M^{-2})^2, ω u 2 ω u 1 ≤ ( B + M e − 3 l o g M ) 2 = ( B + M − 2 ) 2 ,
and hence
d 0 ( 3 log M , ι ) ≥ B B + M − 2 = 1 1 + ( B M 2 ) − 1 > δ 0 . d_0(3\log M,\iota)\ge\frac{B}{B+M^{-2}}=\frac{1}{1+(BM^2)^{-1}}>\delta_0. d 0 ( 3 log M , ι ) ≥ B + M − 2 B = 1 + ( B M 2 ) − 1 1 > δ 0 .
Moreover, the lower bound ω u 1 / ω u 2 ≥ B 2 \omega_{u_1}/\omega_{u_2}\ge B^2 ω u 1 / ω u 2 ≥ B 2 shows that 0 < d 0 ≤ 1 0<d_0\le1 0 < d 0 ≤ 1 throughout B \mathcal{B} B .
For the remaining faces, the definition of ι j ∗ \iota_j^* ι j ∗ gives
log d j − log δ j = ι j − ι j ∗ + 1 2 log 1 + B d 0 1 + B δ 0 + 1 4 log π 0 2 ω u 2 π j 2 ω b j . \log d_j-\log\delta_j=\iota_j-\iota_j^*+\frac{1}{2}\log\frac{1+Bd_0}{1+B\delta_0}+\frac{1}{4}\log\frac{\pi_0^2\omega_{u_2}}{\pi_j^2\omega_{b_j}}. log d j − log δ j = ι j − ι j ∗ + 2 1 log 1 + B δ 0 1 + B d 0 + 4 1 log π j 2 ω b j π 0 2 ω u 2 .
By Lemma D.8,
1 8 ≤ π 0 2 ω u 2 π j 2 ω b j ≤ 8. \frac{1}{8}\le\frac{\pi_0^2\omega_{u_2}}{\pi_j^2\omega_{b_j}}\le8. 8 1 ≤ π j 2 ω b j π 0 2 ω u 2 ≤ 8.
Together with 0 < d 0 ≤ 1 0<d_0\le1 0 < d 0 ≤ 1 , this implies
∣ 1 2 log 1 + B d 0 1 + B δ 0 + 1 4 log π 0 2 ω u 2 π j 2 ω b j ∣ ≤ 1 2 log ( 1 + B ) + 1 4 log 8 < A j . \left|\frac{1}{2}\log\frac{1+Bd_0}{1+B\delta_0}+\frac{1}{4}\log\frac{\pi_0^2\omega_{u_2}}{\pi_j^2\omega_{b_j}}\right|
\le\frac{1}{2}\log(1+B)+\frac{1}{4}\log8<A_j. 2 1 log 1 + B δ 0 1 + B d 0 + 4 1 log π j 2 ω b j π 0 2 ω u 2 ≤ 2 1 log ( 1 + B ) + 4 1 log 8 < A j .
Since min K ι j ∗ ≤ ι j ∗ ≤ max K ι j ∗ \min_K\iota_j^*\le\iota_j^*\le\max_K\iota_j^* min K ι j ∗ ≤ ι j ∗ ≤ max K ι j ∗ , the same estimate gives
ι j = min K ι j ∗ − A j ⟹ d j < δ j , \iota_j=\min_K\iota_j^*-A_j\Longrightarrow d_j<\delta_j, ι j = K min ι j ∗ − A j ⟹ d j < δ j ,
ι j = max K ι j ∗ + A j ⟹ d j > δ j . \iota_j=\max_K\iota_j^*+A_j\Longrightarrow d_j>\delta_j. ι j = K max ι j ∗ + A j ⟹ d j > δ j .
All face inequalities required by Theorem D.11 now hold. Thus there exists a common point ( z † , ι † ) ∈ B (z^\dagger,\iota^\dagger)\in\mathcal{B} ( z † , ι † ) ∈ B such that
d j ( z † , ι † ) = δ j , j = 0 , … , L − 1. d_j(z^\dagger,\iota^\dagger)=\delta_j,\qquad j=0,\ldots,L-1. d j ( z † , ι † ) = δ j , j = 0 , … , L − 1.
Use the initialization corresponding to π † = π ( ι † ) \pi^\dagger=\pi(\iota^\dagger) π † = π ( ι † ) and z † z^\dagger z † . Lemma D.10 applies and gives
θ t u 2 ( t ) → t → ∞ θ ˉ , θ t q min 1 / L ( θ t ) → t → ∞ θ ˉ f ( x 1 ; θ ˉ ) 1 / L . \frac{\theta_t}{u_2(t)} \xrightarrow[t\to\infty]{} \bar{\theta}, \qquad\frac{\theta_t}{q_{\min}^{1/L}(\theta_t)} \xrightarrow[t\to\infty]{} \frac{\bar{\theta}}{f(x_1;\bar{\theta})^{1/L}}. u 2 ( t ) θ t t → ∞ θ ˉ , q m i n 1/ L ( θ t ) θ t t → ∞ f ( x 1 ; θ ˉ ) 1/ L θ ˉ .
Finally, θ ˉ = θ ^ / u ^ 2 \bar{\theta}=\hat{\theta}/\hat{u}_2 θ ˉ = θ ^ / u ^ 2 and f ( x 1 ; θ ^ ) = 1 f(x_1;\hat{\theta})=1 f ( x 1 ; θ ^ ) = 1 . By positive homogeneity,
f ( x 1 ; θ ˉ ) = u ^ 2 − L , θ ˉ f ( x 1 ; θ ˉ ) 1 / L = θ ^ . f(x_1;\bar{\theta})=\hat{u}_2^{-L}, \qquad\frac{\bar{\theta}}{f(x_1;\bar{\theta})^{1/L}}=\hat{\theta}. f ( x 1 ; θ ˉ ) = u ^ 2 − L , f ( x 1 ; θ ˉ ) 1/ L θ ˉ = θ ^ .
This proves the desired convergence for every M ≥ M ∗ M\ge M_\ast M ≥ M ∗ .
For continuous dependence, Lemma D.10 gives the margin-normalized limit throughout B B B as
θ ^ ( z , ι ) : = ( d 0 , 1 , d 1 , … , d L − 1 ) f ( x 1 ; ( d 0 , 1 , d 1 , … , d L − 1 ) ) 1 / L , \hat{\theta}(z,\iota):=\frac{(d_0,1,d_1,\ldots,d_{L-1})}{f(x_1;(d_0,1,d_1,\ldots,d_{L-1}))^{1/L}}, θ ^ ( z , ι ) := f ( x 1 ; ( d 0 , 1 , d 1 , … , d L − 1 ) ) 1/ L ( d 0 , 1 , d 1 , … , d L − 1 ) ,
which is continuous in z , ι z,\iota z , ι and has image containing K K K . Finally, with B , Λ , M , ρ B,\Lambda,M,\rho B , Λ , M , ρ fixed, ( z , ι ) (z,\iota) ( z , ι ) can be recovered continuously from θ 0 \theta_0 θ 0 through (83) . Hence the limiting direction depends continuously on initialization within this family.
For L = 1 L=1 L = 1 , the hidden-layer coordinates and their face conditions are absent, π 0 = Λ \pi_0=\Lambda π 0 = Λ , and the same argument reduces to the intermediate value theorem for d 0 d_0 d 0 on [ 0 , 3 log M ] [0,3\log M] [ 0 , 3 log M ] .
Signed Permutation Symmetries
We now extend the construction from the positive sector to every sign-and-order sector of parameter space. In this section, the model (80) remains fixed; we transform only the dataset and the initialization. All other ingredients of training, including the loss, the learning-rate schedule, and the Adam parameters, remain unchanged.
Recall the positive sector
Θ + : = { ( u , b ) ∈ R L + 1 ∣ 0 < u 1 < u 2 , b j > 0 for every j ∈ [ L − 1 ] } . (105) \Theta_+ := \left\{(\mathbf{u},\mathbf{b})\in\mathbb{R}^{L+1}\mathrel{\big|}0<u_1<u_2,\quad b_j>0\text{ for every }j\in[L-1]\right\}. \tag*{(105)} Θ + := { ( u , b ) ∈ R L + 1 0 < u 1 < u 2 , b j > 0 for every j ∈ [ L − 1 ] } . (105)
Let
S 2 ± : = { Ξ ∈ { 0 , − 1 , 1 } 2 × 2 ∣ Ξ ⊤ Ξ = I 2 } (106) S_2^\pm:=\left\{\Xi\in\{0,-1,1\}^{2\times2}\mathrel{\big|}\Xi^\top\Xi=I_2\right\} \tag*{(106)} S 2 ± := { Ξ ∈ { 0 , − 1 , 1 } 2 × 2 Ξ ⊤ Ξ = I 2 } (106)
be the set of signed permutation matrices on the two input coordinates. Thus, Ξ ∈ S 2 ± \Xi\in S_2^\pm Ξ ∈ S 2 ± may exchange the two coordinates, change either of their signs, or do both. For σ = ( σ 1 , … , σ L − 1 ) ∈ { − 1 , 1 } L − 1 \sigma=(\sigma_1,\ldots,\sigma_{L-1})\in\{-1,1\}^{L-1} σ = ( σ 1 , … , σ L − 1 ) ∈ { − 1 , 1 } L − 1 , define
χ σ : = ∏ r = 1 L − 1 σ r , T Ξ , σ : = diag ( Ξ , σ 1 , … , σ L − 1 ) . (107) \chi_\sigma:=\prod_{r=1}^{L-1}\sigma_r,\qquad T_{\Xi,\sigma}:=\operatorname{diag}(\Xi,\sigma_1,\ldots,\sigma_{L-1}). \tag*{(107)} χ σ := r = 1 ∏ L − 1 σ r , T Ξ , σ := diag ( Ξ , σ 1 , … , σ L − 1 ) . (107)
Equivalently,
T Ξ , σ ( u , b ) = ( Ξ u , σ ⊙ b ) . T_{\Xi,\sigma}(\mathbf{u},\mathbf{b})=(\Xi\mathbf{u},\sigma\odot\mathbf{b}). T Ξ , σ ( u , b ) = ( Ξ u , σ ⊙ b ) .
When L = 1 L=1 L = 1 , the vector σ \sigma σ is empty and the empty product is understood to be χ σ = 1 \chi_\sigma=1 χ σ = 1 . Finally, let
Π Ξ , σ : = T Ξ , σ ⊙ T Ξ , σ ; (108) \Pi_{\Xi,\sigma}:=T_{\Xi,\sigma}\odot T_{\Xi,\sigma}; \tag*{(108)} Π Ξ , σ := T Ξ , σ ⊙ T Ξ , σ ; (108)
this is the unsigned permutation matrix obtained by deleting the signs from T Ξ , σ T_{\Xi,\sigma} T Ξ , σ .
Given a dataset S = { ( x i , y i ) } i = 1 m S=\{(x_i,y_i)\}_{i=1}^m S = {( x i , y i ) } i = 1 m , define
S Ξ , σ : = { ( Ξ x i , χ σ y i ) } i = 1 m . (109) S^{\Xi,\sigma}:=\{(\Xi x_i,\chi_\sigma y_i)\}_{i=1}^m. \tag*{(109)} S Ξ , σ := {( Ξ x i , χ σ y i ) } i = 1 m . (109)
Thus the signs and order of the first-layer parameters are absorbed into the input coordinates, while all labels are flipped exactly when an odd number of the scalar-layer parameters are sign-flipped. For Ξ ∈ S 2 ± \Xi\in S_2^\pm Ξ ∈ S 2 ± and σ ∈ { − 1 , 1 } L − 1 \sigma\in\{-1,1\}^{L-1} σ ∈ { − 1 , 1 } L − 1 , define the corresponding sign-and-order sector by
Θ Ξ , σ : = T Ξ , σ Θ + . (110) \Theta_{\Xi,\sigma}:=T_{\Xi,\sigma}\Theta_+. \tag*{(110)} Θ Ξ , σ := T Ξ , σ Θ + . (110)
The sets Θ Ξ , σ \Theta_{\Xi,\sigma} Θ Ξ , σ are exactly the open sectors obtained by fixing the signs of all parameters and fixing which of ∣ u 1 ∣ |u_1| ∣ u 1 ∣ and ∣ u 2 ∣ |u_2| ∣ u 2 ∣ is smaller.
Lemma D.12 (Dataset symmetries imply trajectory symmetries). Let ( θ t , m t , v t ) (\theta_t,\mathbf{m}_t,\mathbf{v}_t) ( θ t , m t , v t ) solve the bias-corrected Adam equations for the fixed model (80) on a dataset S S S , with initialization θ 0 \theta_0 θ 0 and zero moment buffers. Fix Ξ ∈ S 2 ± \Xi\in S_2^{\pm} Ξ ∈ S 2 ± and σ ∈ { − 1 , 1 } L − 1 \sigma\in\{-1,1\}^{L-1} σ ∈ { − 1 , 1 } L − 1 , and set
θ ~ t : = T Ξ , σ θ t , m ~ t : = T Ξ , σ m t , v ~ t : = Π Ξ , σ v t . (111) \widetilde{\theta}_t := T_{\Xi,\sigma}\theta_t,\qquad\widetilde{\mathbf{m}}_t := T_{\Xi,\sigma}\mathbf{m}_t,\qquad\widetilde{\mathbf{v}}_t := \Pi_{\Xi,\sigma}\mathbf{v}_t. \tag*{(111)} θ t := T Ξ , σ θ t , m t := T Ξ , σ m t , v t := Π Ξ , σ v t . (111) Then ( θ ~ t , m ~ t , v ~ t ) (\widetilde{\theta}_t,\widetilde{\mathbf{m}}_t,\widetilde{\mathbf{v}}_t) ( θ t , m t , v t ) solves the same Adam equations for the same model f f f on the transformed dataset S Ξ , σ S^{\Xi,\sigma} S Ξ , σ , with initialization T Ξ , σ θ 0 T_{\Xi,\sigma}\theta_0 T Ξ , σ θ 0 .
Moreover, for every θ = ( u , b ) \theta=(\mathbf{u},\mathbf{b}) θ = ( u , b ) and every ( x i , y i ) ∈ S (\mathbf{x}_i,y_i)\in S ( x i , y i ) ∈ S ,
( χ σ y i ) f ( Ξ x i ; T Ξ , σ θ ) = y i f ( x i ; θ ) . (112) (\chi_\sigma y_i)f(\Xi\mathbf{x}_i;T_{\Xi,\sigma}\theta)=y_i f(\mathbf{x}_i;\theta). \tag*{(112)} ( χ σ y i ) f ( Ξ x i ; T Ξ , σ θ ) = y i f ( x i ; θ ) . (112) Consequently, if
q min S ( θ ) : = min i ∈ [ m ] y i f ( x i ; θ ) , q_{\min}^{S}(\theta):=\min_{i\in[m]} y_i f(\mathbf{x}_i;\theta), q m i n S ( θ ) := i ∈ [ m ] min y i f ( x i ; θ ) , then
q min S Ξ , σ ( T Ξ , σ θ ) = q min S ( θ ) . (113) q_{\min}^{S^{\Xi,\sigma}}(T_{\Xi,\sigma}\theta)=q_{\min}^{S}(\theta). \tag*{(113)} q m i n S Ξ , σ ( T Ξ , σ θ ) = q m i n S ( θ ) . (113) In particular, directional limits with respect to ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 and margin-normalized limits are transported by T Ξ , σ T_{\Xi,\sigma} T Ξ , σ , and each of the assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) holds for the trajectory θ t \theta_t θ t if and only if it holds for θ ~ t \widetilde{\theta}_t θ t .
Proof. Since Ξ T Ξ = I 2 \Xi^{\mathsf{T}}\Xi=I_2 Ξ T Ξ = I 2 and P ( σ ⊙ b ) = χ σ P ( b ) P(\sigma\mathbin{\odot}\mathbf{b})=\chi_\sigma P(\mathbf{b}) P ( σ ⊙ b ) = χ σ P ( b ) , we have
( χ σ y i ) f ( Ξ x i ; T Ξ , σ θ ) = ( χ σ y i ) P ( σ ⊙ b ) ⟨ Ξ u , Ξ x i ⟩ = χ σ 2 y i P ( b ) ⟨ u , x i ⟩ = y i f ( x i ; θ ) , \begin{aligned}
(\chi_\sigma y_i)f(\Xi\mathbf{x}_i;T_{\Xi,\sigma}\theta)
&=(\chi_\sigma y_i)P(\sigma\mathbin{\odot}\mathbf{b})\langle\Xi\mathbf{u},\Xi\mathbf{x}_i\rangle\\
&=\chi_\sigma^2y_iP(\mathbf{b})\langle\mathbf{u},\mathbf{x}_i\rangle
=y_i f(\mathbf{x}_i;\theta),
\end{aligned} ( χ σ y i ) f ( Ξ x i ; T Ξ , σ θ ) = ( χ σ y i ) P ( σ ⊙ b ) ⟨ Ξ u , Ξ x i ⟩ = χ σ 2 y i P ( b ) ⟨ u , x i ⟩ = y i f ( x i ; θ ) , which proves (112) . Hence, if L S \mathcal{L}_S L S and L S Ξ , σ \mathcal{L}_{S^{\Xi,\sigma}} L S Ξ , σ denote the empirical losses of the original and transformed datasets, respectively, then
L S Ξ , σ ( T Ξ , σ θ ) = L S ( θ ) . (114) \mathcal{L}_{S^{\Xi,\sigma}}(T_{\Xi,\sigma}\theta)=\mathcal{L}_S(\theta). \tag*{(114)} L S Ξ , σ ( T Ξ , σ θ ) = L S ( θ ) . (114) Differentiating (114) gives
T Ξ , σ T ∇ L S Ξ , σ ( T Ξ , σ θ ) = ∇ L S ( θ ) . T_{\Xi,\sigma}^{\mathsf{T}}\nabla\mathcal{L}_{S^{\Xi,\sigma}}(T_{\Xi,\sigma}\theta)=\nabla\mathcal{L}_S(\theta). T Ξ , σ T ∇ L S Ξ , σ ( T Ξ , σ θ ) = ∇ L S ( θ ) . Because T Ξ , σ T_{\Xi,\sigma} T Ξ , σ is orthogonal,
∇ L S Ξ , σ ( T Ξ , σ θ ) = T Ξ , σ ∇ L S ( θ ) . (115) \nabla\mathcal{L}_{S^{\Xi,\sigma}}(T_{\Xi,\sigma}\theta)=T_{\Xi,\sigma}\nabla\mathcal{L}_S(\theta). \tag*{(115)} ∇ L S Ξ , σ ( T Ξ , σ θ ) = T Ξ , σ ∇ L S ( θ ) . (115) Therefore, writing g t = ∇ L S ( θ t ) \mathbf{g}_t=\nabla\mathcal{L}_S(\theta_t) g t = ∇ L S ( θ t ) and g ~ t = ∇ L S Ξ , σ ( θ ~ t ) \widetilde{\mathbf{g}}_t=\nabla\mathcal{L}_{S^{\Xi,\sigma}}(\widetilde{\theta}_t) g t = ∇ L S Ξ , σ ( θ t ) ,
g ~ t = T Ξ , σ g t , g ~ t 2 = Π Ξ , σ g t 2 , (116) \widetilde{\mathbf{g}}_t=T_{\Xi,\sigma}\mathbf{g}_t,\qquad\widetilde{\mathbf{g}}_t^2=\Pi_{\Xi,\sigma}\mathbf{g}_t^2, \tag*{(116)} g t = T Ξ , σ g t , g t 2 = Π Ξ , σ g t 2 , (116) where the square is coordinate-wise. It follows that
m ~ ˙ t = T Ξ , σ m ˙ t = c 1 ( g ~ t − m ~ t ) , \dot{\widetilde{\mathbf{m}}}_t=T_{\Xi,\sigma}\dot{\mathbf{m}}_t=c_1\left(\widetilde{\mathbf{g}}_t-\widetilde{\mathbf{m}}_t\right), m ˙ t = T Ξ , σ m ˙ t = c 1 ( g t − m t ) , v ~ ˙ t = Π Ξ , σ v ˙ t = c 2 ( g ~ t 2 − v ~ t ) . \dot{\widetilde{\mathbf{v}}}_t=\Pi_{\Xi,\sigma}\dot{\mathbf{v}}_t=c_2\left(\widetilde{\mathbf{g}}_t^2-\widetilde{\mathbf{v}}_t\right). v ˙ t = Π Ξ , σ v ˙ t = c 2 ( g t 2 − v t ) . The zero initial conditions are preserved, and the scalar bias-correction factors commute with the two transformations, so
m ~ ^ t = T Ξ , σ m ^ t , v ~ ^ t = Π Ξ , σ v ^ t . \widehat{\widetilde{\mathbf{m}}}_t=T_{\Xi,\sigma}\widehat{\mathbf{m}}_t,\qquad\widehat{\widetilde{\mathbf{v}}}_t=\Pi_{\Xi,\sigma}\widehat{\mathbf{v}}_t. m t = T Ξ , σ m t , v t = Π Ξ , σ v t . For every signed permutation matrix T T T and every vector z \mathbf{z} z and coordinate-wise positive vector w \mathbf{w} w ,
T z ( T ⊙ T ) w = T z w . \frac{T\mathbf{z}}{\sqrt{(T\mathbin{\odot}T)\mathbf{w}}}=T\frac{\mathbf{z}}{\sqrt{\mathbf{w}}}. ( T ⊙ T ) w T z = T w z . Applying this identity with T = T Ξ , σ T=T_{\Xi,\sigma} T = T Ξ , σ yields, for t > 0 t>0 t > 0 ,
m ~ ^ t v ~ ^ t = T Ξ , σ m ^ t v ^ t . (117) \frac{\widehat{\widetilde{\mathbf{m}}}_t}{\sqrt{\widehat{\widetilde{\mathbf{v}}}_t}}
=T_{\Xi,\sigma}\frac{\widehat{\mathbf{m}}_t}{\sqrt{\widehat{\mathbf{v}}_t}}. \tag*{(117)} v t m t = T Ξ , σ v t m t . (117)
If a second-moment coordinate is zero, its corresponding first moment is also zero, and (117) holds under the zero-denominator convention. Thus
θ ~ ˙ t = T Ξ , σ θ ˙ t = − η ( t ) m ~ t v ~ t . \dot{\widetilde{\theta}}_t=T_{\Xi,\sigma}\dot{\theta}_t=-\eta(t)\frac{\widetilde{m}_t}{\sqrt{\widetilde{v}_t}}. θ ˙ t = T Ξ , σ θ ˙ t = − η ( t ) v t m t .
which is precisely the Adam parameter equation for the transformed dataset. Equation (113) follows immediately from (112) . Since T Ξ , σ T_{\Xi,\sigma} T Ξ , σ is orthogonal, whenever θ t ≠ 0 \theta_t \ne0 θ t = 0 ,
θ ~ t ∥ θ ~ t ∥ 2 = T Ξ , σ θ t ∥ θ t ∥ 2 . \frac{\widetilde{\theta}_t}{\lVert\widetilde{\theta}_t\rVert_2}
=
T_{\Xi,\sigma}\frac{\theta_t}{\lVert\theta_t\rVert_2}. ∥ θ t ∥ 2 θ t = T Ξ , σ ∥ θ t ∥ 2 θ t .
Whenever the minimum margin is positive, we likewise have
θ ~ t q min S Ξ , σ ( θ ~ t ) 1 / L = T Ξ , σ θ t q min S ( θ t ) 1 / L . (118) \frac{\widetilde{\theta}_t}{q_{\min}^{S^{\Xi,\sigma}}(\widetilde{\theta}_t)^{1/L}}
=
T_{\Xi,\sigma}\frac{\theta_t}{q_{\min}^{S}(\theta_t)^{1/L}}. \tag*{(118)} q m i n S Ξ , σ ( θ t ) 1/ L θ t = T Ξ , σ q m i n S ( θ t ) 1/ L θ t . (118)
This proves the transportation of directional and margin-normalized limits. For equivalence of assumptions (T1), (T2), the losses agree by (114) , while orthogonality and (113) give, whenever θ t ≠ 0 \theta_t \ne0 θ t = 0 ,
q min S Ξ , σ ( θ ~ t ) ∥ θ ~ t ∥ 2 L = q min S ( θ t ) ∥ θ t ∥ 2 L . \frac{q_{\min}^{S^{\Xi,\sigma}}(\widetilde{\theta}_t)}{\lVert\widetilde{\theta}_t\rVert_2^L}
=
\frac{q_{\min}^{S}(\theta_t)}{\lVert\theta_t\rVert_2^L}. ∥ θ t ∥ 2 L q m i n S Ξ , σ ( θ t ) = ∥ θ t ∥ 2 L q m i n S ( θ t ) .
Thus the two trajectories have identical losses and normalized margins, proving both equivalences. □
Corollary D.13 (Continuous directional limits on a fixed dataset). Under the assumptions of Theorem 4.2 , let p = L + 1 p=L+1 p = L + 1 and define
U Ξ , σ : = Θ Ξ , σ ∩ S p − 1 , Ξ ∈ S 2 ± , σ ∈ { − 1 , 1 } L − 1 . U_{\Xi,\sigma}:=\Theta_{\Xi,\sigma}\cap\mathbb{S}^{p-1},\qquad\Xi\in\mathcal{S}_2^{\pm},\qquad\sigma\in\{-1,1\}^{L-1}. U Ξ , σ := Θ Ξ , σ ∩ S p − 1 , Ξ ∈ S 2 ± , σ ∈ { − 1 , 1 } L − 1 . These 2 L + 2 2^{L+2} 2 L + 2 sets are open in the sphere, and their union has full surface measure. For every compact A ⊆ U Ξ , σ A\subseteq U_{\Xi,\sigma} A ⊆ U Ξ , σ , there is one fixed two-point dataset S A S_A S A such that every point of A A A is the limit of θ t / ∥ θ t ∥ 2 \theta_t/\lVert\theta_t\rVert_2 θ t / ∥ θ t ∥ 2 for a suitable initialization. Moreover, the initializations can be taken from a compact family on which this limiting direction exists and depends continuously on the initialization.
Proof. The sectors fix the signs of all coordinates and the strict order of ∣ u 1 ∣ , ∣ u 2 ∣ \lvert u_1\rvert,\lvert u_2\rvert ∣ u 1 ∣ , ∣ u 2 ∣ , giving 2 L + 1 ⋅ 2 = 2 L + 2 2^{L+1}\cdot2=2^{L+2} 2 L + 1 ⋅ 2 = 2 L + 2 open spherical sets. Their complement has zero measure.
By Lemma D.12 , it suffices to consider a nonempty compact A ⊆ Θ + ∩ S p − 1 A\subseteq\Theta_+\cap\mathbb{S}^{p-1} A ⊆ Θ + ∩ S p − 1 . Fix any B > 1 B>1 B > 1 , Λ > 0 \Lambda>0 Λ > 0 . Since f ( x 1 ; θ ˉ ) > 0 f(\mathbf{x}_1;\bar{\theta})>0 f ( x 1 ; θ ˉ ) > 0 on this sector,
K : = { θ ˉ f ( x 1 ; θ ˉ ) 1 / L | θ ˉ ∈ A } ⊆ Σ B , Λ K:=\left\{\frac{\bar{\theta}}{f(\mathbf{x}_1;\bar{\theta})^{1/L}}\ \middle|\ \bar{\theta}\in A\right\}\subseteq\Sigma_{B,\Lambda} K := { f ( x 1 ; θ ˉ ) 1/ L θ ˉ θ ˉ ∈ A } ⊆ Σ B , Λ
is compact. Proposition D.3 supplies one fixed M M M and a compact initialization family whose continuous margin-normalized limits contain K K K . The margin-normalized limits have minimum margin 1 1 1 , so they are nonzero, and normalizing them to unit norm preserves continuity and reaches every point of A A A . □
Corollary D.14 (Constancy on all sign-and-order sectors). Assume α > 1 L = 1 2 \alpha>\frac{1}{L}=\frac{1}{2} α > L 1 = 2 1 . For every Ξ ∈ S 2 ± \Xi\in\mathcal{S}_2^{\pm} Ξ ∈ S 2 ± and σ ∈ { − 1 , 1 } L − 1 \sigma\in\{-1,1\}^{L-1} σ ∈ { − 1 , 1 } L − 1 , any implicit regularizer R R R of Adam on the width-1 model with the exponential loss is constant on Θ Ξ , σ \Theta_{\Xi,\sigma} Θ Ξ , σ . All of these sector constants equal R ( 0 ) R(0) R ( 0 ) . Consequently, R R R is constant on R L + 1 \mathbb{R}^{L+1} R L + 1 .
Proof. Let R R R be an implicit margin regularizer for f f f with the exponential loss. Fix Ξ ∈ S 2 ± \Xi\in\mathcal{S}_2^{\pm} Ξ ∈ S 2 ± and σ ∈ { − 1 , 1 } L − 1 \sigma\in\{-1,1\}^{L-1} σ ∈ { − 1 , 1 } L − 1 , and write T : = T Ξ , σ T:=T_{\Xi,\sigma} T := T Ξ , σ . Define
R T ( θ ) : = R ( T θ ) . R_T(\theta):=R(T\theta). R T ( θ ) := R ( T θ ) .
We first verify that R T R_T R T is an implicit margin regularizer for f f f . Since T T T is invertible, R T R_T R T is locally Lipschitz and admits a chain rule, with
∂ R T ( θ ) = T ⊤ ∂ R ( T θ ) . \partial R_T(\theta)=T^{\top}\partial R(T\theta). ∂ R T ( θ ) = T ⊤ ∂ R ( T θ ) .
Consider any Adam trajectory θ t \theta_t θ t on a dataset S S S satisfying (T1), (T2) in Definition 3.1 , with margin-normalized limit θ ^ \hat{\theta} θ ^ . By Lemma D.12 , T θ t T\theta_t T θ t is an Adam trajectory on S Ξ , σ S^{\Xi,\sigma} S Ξ , σ , with margin-normalized limit T θ ^ T\hat{\theta} T θ ^ , and satisfying (T1), (T2). Writing q i q_i q i and q ~ i \tilde{q}_i q ~ i for the original and transformed margins, respectively, we have
q ~ i ( T θ ) = q i ( θ ) , T ⊤ ∇ q ~ i ( T θ ) = ∇ q i ( θ ) . \tilde{q}_i(T\theta)=q_i(\theta),\qquad T^{\top}\nabla\tilde{q}_i(T\theta)=\nabla q_i(\theta). q ~ i ( T θ ) = q i ( θ ) , T ⊤ ∇ q ~ i ( T θ ) = ∇ q i ( θ ) .
Because R R R is an implicit regularizer, its KKT conditions at T θ ^ T\hat{\theta} T θ ^ give
k = ∑ i λ i ∇ q ~ i ( T θ ^ ) , k ∈ ∂ R ( T θ ^ ) , \mathbf{k}=\sum_i\lambda_i\nabla\tilde{q}_i(T\hat{\theta}),\qquad\mathbf{k}\in\partial R(T\hat{\theta}), k = i ∑ λ i ∇ q ~ i ( T θ ^ ) , k ∈ ∂ R ( T θ ^ ) ,
∀ i : q ~ i ( T θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q ~ i ( T θ ^ ) − 1 ) = 0. \forall i:\qquad\tilde{q}_i(T\hat{\theta})\ge1,\qquad\lambda_i\ge0,\qquad\lambda_i\bigl(\tilde{q}_i(T\hat{\theta})-1\bigr)=0. ∀ i : q ~ i ( T θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q ~ i ( T θ ^ ) − 1 ) = 0.
Multiplying stationarity by T ⊤ T^{\top} T ⊤ and using q ~ i ( T θ ^ ) = q i ( θ ^ ) \tilde{q}_i(T\hat{\theta})=q_i(\hat{\theta}) q ~ i ( T θ ^ ) = q i ( θ ^ ) yields
T ⊤ k = ∑ i λ i ∇ q i ( θ ^ ) , T ⊤ k ∈ ∂ R T ( θ ^ ) , T^{\top}\mathbf{k}=\sum_i\lambda_i\nabla q_i(\hat{\theta}),\qquad T^{\top}\mathbf{k}\in\partial R_T(\hat{\theta}), T ⊤ k = i ∑ λ i ∇ q i ( θ ^ ) , T ⊤ k ∈ ∂ R T ( θ ^ ) ,
∀ i : q i ( θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q i ( θ ^ ) − 1 ) = 0. \forall i:\qquad q_i(\hat{\theta})\ge1,\qquad\lambda_i\ge0,\qquad\lambda_i\bigl(q_i(\hat{\theta})-1\bigr)=0. ∀ i : q i ( θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q i ( θ ^ ) − 1 ) = 0.
Thus θ ^ \hat{\theta} θ ^ is KKT for the original margin constraints with objective R T R_T R T , proving that R T R_T R T is an implicit regularizer.
Lemma D.2, applied to R T R_T R T , makes R T R_T R T constant on every Σ B , Λ \Sigma_{B,\Lambda} Σ B , Λ . The positive-sector argument in the proof of Theorem D.1 , applied to R T R_T R T , which uses only Lemma D.2 and continuity of R T R_T R T , therefore makes R T R_T R T constant on Θ + \Theta_+ Θ + . Hence R R R is constant on T Θ + = Θ Ξ , σ T\Theta_+=\Theta_{\Xi,\sigma} T Θ + = Θ Ξ , σ .
For any θ ∈ Θ + \theta\in\Theta_+ θ ∈ Θ + , the entire ray { r θ : r > 0 } \{r\theta:r>0\} { r θ : r > 0 } lies in Θ + \Theta_+ Θ + , so continuity gives
R ( T θ ) = R T ( θ ) = lim r ↓ 0 R T ( r θ ) = R ( 0 ) . R(T\theta)=R_T(\theta)=\lim_{r\downarrow0}R_T(r\theta)=R(0). R ( T θ ) = R T ( θ ) = r ↓ 0 lim R T ( r θ ) = R ( 0 ) .
Thus all sector constants equal R ( 0 ) R(0) R ( 0 ) . The sectors have dense union in R L + 1 \mathbb{R}^{L+1} R L + 1 , and continuity extends this equality to the whole space. □ \square □
Arbitrary Widths and Input Dimension
In this section we generalize the width-1 construction to a scalar-output linear network of arbitrary width and input dimension.
Consider a depth-L L L scalar-output linear network. Let its input dimension be d d d , and let its hidden widths be n 1 , … , n L − 1 n_1,\ldots,n_{L-1} n 1 , … , n L − 1 . Its parameter space for L ≥ 2 L\ge2 L ≥ 2 is
V = R n 1 × d × ∏ ℓ = 2 L − 1 R n ℓ × n ℓ − 1 × R 1 × n L − 1 . \mathcal{V}=\mathbb{R}^{n_1\times d}\times\prod_{\ell=2}^{L-1}\mathbb{R}^{n_\ell\times n_{\ell-1}}\times\mathbb{R}^{1\times n_{L-1}}. V = R n 1 × d × ℓ = 2 ∏ L − 1 R n ℓ × n ℓ − 1 × R 1 × n L − 1 .
For L = 1 L=1 L = 1 , set V = R 1 × d \mathcal{V}=\mathbb{R}^{1\times d} V = R 1 × d . Assume throughout that L ≥ 1 L\ge1 L ≥ 1 , d ≥ 2 d\ge2 d ≥ 2 , and n ℓ ∈ N > 0 n_\ell\in\mathbb{N}_{>0} n ℓ ∈ N > 0 for every 1 ≤ ℓ < L 1\le\ell<L 1 ≤ ℓ < L . Set once and for all
κ : = ∏ r = 1 L − 1 n r . \kappa:=\prod_{r=1}^{L-1}n_r. κ := r = 1 ∏ L − 1 n r .
Define the narrow parameter space and distinguish its parameter from the wide one by
Θ : = R 2 × R L − 1 ≅ R L + 1 , θ = ( u , b ) ∈ Θ , ϑ = ( W 1 , … , W L ) ∈ V . \Theta:=\mathbb{R}^{2}\times\mathbb{R}^{L-1}\cong\mathbb{R}^{L+1},\qquad\theta=(\mathbf{u},b)\in\Theta,\qquad\vartheta=(W_1,\ldots,W_L)\in\mathcal{V}. Θ := R 2 × R L − 1 ≅ R L + 1 , θ = ( u , b ) ∈ Θ , ϑ = ( W 1 , … , W L ) ∈ V .
We retain the notation f ( x ; θ ) = P ( b ) u ⊤ x f(x;\theta)=P(b)\mathbf{u}^{\top}x f ( x ; θ ) = P ( b ) u ⊤ x for the width-1 model, and denote the wide model by F ( ξ ; ϑ ) F(\xi;\vartheta) F ( ξ ; ϑ ) :
F ( ξ ; ϑ ) : = W L ⋯ W 1 ξ . F(\xi;\vartheta):=W_L\cdots W_1\xi. F ( ξ ; ϑ ) := W L ⋯ W 1 ξ .
When L = 1 L=1 L = 1 , b b b and the sign tuple s \mathbf{s} s below are empty, and the empty products give P = κ = 1 P=\kappa=1 P = κ = 1 .
The lifting is based on the following principle: choose a sign vector s ℓ ∈ { ± 1 } n ℓ \mathbf{s}_{\ell}\in\{\pm1\}^{n_{\ell}} s ℓ ∈ { ± 1 } n ℓ for every hidden layer. Its i i i -th scalar entry is denoted by ( s ℓ ) i (\mathbf{s}_{\ell})_i ( s ℓ ) i . During training, all neurons in a layer carry signed copies of the same narrow scalar. For the input-dimension lift, additionally choose c ∈ [ d ] : = { 1 , … , d } c\in[d]:=\{1,\ldots,d\} c ∈ [ d ] := { 1 , … , d } and, for x , u ∈ R 2 \mathbf{x},\mathbf{u}\in\mathbb{R}^{2} x , u ∈ R 2 , define
U c , X c : R 2 → R d , U c ( u ) = u 1 e c + ∑ c ′ ≠ c u 2 e c ′ , X c ( x ) = x 1 e c + 1 d − 1 ∑ c ′ ≠ c x 2 e c ′ . (119) U_c,X_c:\mathbb{R}^{2}\to\mathbb{R}^{d},\qquad U_c(\mathbf{u})=u_1\mathbf{e}_c+\sum_{c'\ne c}u_2\mathbf{e}_{c'},\qquad X_c(\mathbf{x})=x_1\mathbf{e}_c+\frac{1}{d-1}\sum_{c'\ne c}x_2\mathbf{e}_{c'}. \tag*{(119)} U c , X c : R 2 → R d , U c ( u ) = u 1 e c + c ′ = c ∑ u 2 e c ′ , X c ( x ) = x 1 e c + d − 1 1 c ′ = c ∑ x 2 e c ′ . (119)
The two maps are paired so that
U c ( u ) T X c ( x ) = u 1 x 1 + u 2 x 2 = u T x . (120) U_c(\mathbf{u})^{\mathsf{T}}X_c(\mathbf{x})=u_1x_1+u_2x_2=\mathbf{u}^{\mathsf{T}}\mathbf{x}. \tag*{(120)} U c ( u ) T X c ( x ) = u 1 x 1 + u 2 x 2 = u T x . (120)
Formally, for s = ( s 1 , … , s L − 1 ) ∈ { ± 1 } n 1 × ⋯ × { ± 1 } n L − 1 \mathbf{s}=(\mathbf{s}_1,\ldots,\mathbf{s}_{L-1})\in\{\pm1\}^{n_1}\times\cdots\times\{\pm1\}^{n_{L-1}} s = ( s 1 , … , s L − 1 ) ∈ { ± 1 } n 1 × ⋯ × { ± 1 } n L − 1 , define the linear map E s , c : Θ → V E_{\mathbf{s},c}:\Theta\to\mathcal{V} E s , c : Θ → V , θ ↦ ϑ \theta\mapsto\vartheta θ ↦ ϑ , by the following formulas when L ≥ 2 L\ge2 L ≥ 2 :
W 1 = ( E s , c θ ) 1 = s 1 U c ( u ) T , (121) W_1=(E_{\mathbf{s},c}\theta)_1=\mathbf{s}_1U_c(\mathbf{u})^{\mathsf{T}}, \tag*{(121)} W 1 = ( E s , c θ ) 1 = s 1 U c ( u ) T , (121)
W ℓ = ( E s , c θ ) ℓ = b ℓ − 1 s ℓ s ℓ − 1 T , 2 ≤ ℓ ≤ L − 1 , (122) W_{\ell}=(E_{\mathbf{s},c}\theta)_{\ell}=b_{\ell-1}\mathbf{s}_{\ell}\mathbf{s}_{\ell-1}^{\mathsf{T}},\qquad2\le\ell\le L-1, \tag*{(122)} W ℓ = ( E s , c θ ) ℓ = b ℓ − 1 s ℓ s ℓ − 1 T , 2 ≤ ℓ ≤ L − 1 , (122)
W L = ( E s , c θ ) L = b L − 1 s L − 1 T . (123) W_L=(E_{\mathbf{s},c}\theta)_L=b_{L-1}\mathbf{s}_{L-1}^{\mathsf{T}}. \tag*{(123)} W L = ( E s , c θ ) L = b L − 1 s L − 1 T . (123)
For L = 1 L=1 L = 1 , define instead E s , c ( u ) : = U c ( u ) T E_{\mathbf{s},c}(\mathbf{u}):=U_c(\mathbf{u})^{\mathsf{T}} E s , c ( u ) := U c ( u ) T (note s \mathbf{s} s has no role here). Denote the image of E s , c E_{\mathbf{s},c} E s , c by
V s , c : = im E s , c . \mathcal{V}_{\mathbf{s},c}:=\operatorname{im}E_{\mathbf{s},c}. V s , c := im E s , c .
The lift is injective. Indeed, for L ≥ 2 L\ge2 L ≥ 2 , its first block determines U c ( u ) U_c(\mathbf{u}) U c ( u ) , hence u \mathbf{u} u , every middle block determines the corresponding scalar b r b_r b r , and the final block determines b L − 1 b_{L-1} b L − 1 . For L = 1 L=1 L = 1 , this follows from injectivity of U c U_c U c . Thus E s , c : Θ → V s , c E_{\mathbf{s},c}:\Theta\to\mathcal{V}_{\mathbf{s},c} E s , c : Θ → V s , c is a linear bijection.
Lemma D.15 (Spanning of the combined copy subspaces). The subspaces of the lift span the full wide parameter space:
span ( ⋃ c = 1 d ⋃ s V s , c ) = V . (124) \operatorname{span}\left(\bigcup_{c=1}^{d}\bigcup_{\mathbf{s}}\mathcal{V}_{\mathbf{s},c}\right)=\mathcal{V}. \tag*{(124)} span ( c = 1 ⋃ d s ⋃ V s , c ) = V . (124) Proof. For L = 1 L=1 L = 1 , E ∅ , c ( ( 1 , 0 ) ) = e c T E_{\varnothing,c}((1,0))=\mathbf{e}_c^{\mathsf{T}} E ∅ , c (( 1 , 0 )) = e c T , so varying c c c spans V \mathcal{V} V . Assume henceforth that L ≥ 2 L\ge2 L ≥ 2 . Since U c ( ( 1 , 0 ) ) = e c U_c((1,0))=\mathbf{e}_c U c (( 1 , 0 )) = e c ,
E s , c ( ( 1 , 0 ) , 0 , … , 0 ) = ( s 1 e c T , 0 , … , 0 ) . E_{\mathbf{s},c}((1,0),0,\ldots,0)=(\mathbf{s}_1\mathbf{e}_c^{\mathsf{T}},0,\ldots,0). E s , c (( 1 , 0 ) , 0 , … , 0 ) = ( s 1 e c T , 0 , … , 0 ) . For fixed c c c , varying s 1 \mathbf{s}_1 s 1 spans the c c c -th input column, and varying c c c spans the whole first matrix. For 1 ≤ ℓ ≤ L − 2 1\le\ell\le L-2 1 ≤ ℓ ≤ L − 2 , setting only b ℓ b_{\ell} b ℓ nonzero isolates b ℓ s ℓ + 1 s ℓ T b_{\ell}\mathbf{s}_{\ell+1}\mathbf{s}_{\ell}^{\mathsf{T}} b ℓ s ℓ + 1 s ℓ T in matrix ℓ + 1 \ell+1 ℓ + 1 ; setting only b L − 1 b_{L-1} b L − 1 nonzero isolates b L − 1 s L − 1 T b_{L-1}\mathbf{s}_{L-1}^{\mathsf{T}} b L − 1 s L − 1 T in the output matrix.
Sign vectors span every coordinate space because, for a basis vector e i ( n ) ∈ R n \mathbf{e}^{(n)}_i\in\mathbb{R}^{n} e i ( n ) ∈ R n ,
e i ( n ) = 1 2 ( 1 n − ( 1 n − 2 e i ( n ) ) ) , \mathbf{e}^{(n)}_i=\frac{1}{2}\left(\mathbf{1}_n-\left(\mathbf{1}_n-2\mathbf{e}^{(n)}_i\right)\right), e i ( n ) = 2 1 ( 1 n − ( 1 n − 2 e i ( n ) ) ) , and both vectors on the right are sign vectors. Products of these representations express every middle-layer matrix unit e i ( ℓ ) e j ( ℓ − 1 ) T \mathbf{e}^{(\ell)}_i\mathbf{e}^{(\ell-1)\mathsf{T}}_j e i ( ℓ ) e j ( ℓ − 1 ) T as a linear combination of four signed outer products. Hence every matrix unit in every block belongs to the span in (124) . □
Lemma D.16 (Convex propagation). Let R : V → R R:\mathcal{V}\to\mathbb{R} R : V → R be finite and convex. If
R ∣ V s , c ≡ R ( 0 ) for every s and c , R|_{\mathcal{V}_{\mathbf{s},c}}\equiv R(0)\qquad\text{for every }\mathbf{s}\text{ and }c, R ∣ V s , c ≡ R ( 0 ) for every s and c , then R R R is constant on V \mathcal{V} V .
Proof. By Lemma D.15 , write any X ∈ V X \in\mathcal{V} X ∈ V as X = ∑ a = 1 N X a X = \sum_{a=1}^{N} X_a X = ∑ a = 1 N X a , with each X a X_a X a in one lift subspace. Since those subspaces are linear, convexity gives
R ( X ) = R ( 1 N ∑ a = 1 N N X a ) ≤ 1 N ∑ a = 1 N R ( N X a ) = R ( 0 ) . R(X) = R\left(\frac{1}{N}\sum_{a=1}^{N}NX_a\right) \le\frac{1}{N}\sum_{a=1}^{N}R(NX_a) = R(0). R ( X ) = R ( N 1 a = 1 ∑ N N X a ) ≤ N 1 a = 1 ∑ N R ( N X a ) = R ( 0 ) .
The same inequality holds for − X -X − X . Convexity at the midpoint of X X X and − X -X − X then forces equality, so R ( X ) = R ( 0 ) R(X)=R(0) R ( X ) = R ( 0 ) .
The next lemma identifies the wide gradient by combining the chain rule with the fact that the gradient lies in the copy subspace.
Lemma D.17 (Model and gradient identities). Fix s s s and c ∈ [ d ] c \in[d] c ∈ [ d ] , let E : = E s , c E := E_{s,c} E := E s , c , and put
E s , c ♯ : = E ♯ : = E ( E ⊤ E ) − 1 . E_{s,c}^{\sharp} := E^{\sharp} := E(E^{\top}E)^{-1}. E s , c ♯ := E ♯ := E ( E ⊤ E ) − 1 . Let S n , S c , w S^{\mathrm{n}}, S^{c,\mathrm{w}} S n , S c , w denote the narrow and wide datasets,
S n : = { ( x i , y i ) } i = 1 m ⊆ R 2 × { ± 1 } , S c , w : = { ( X c ( x i ) / κ , y i ) } i = 1 m , S^{\mathrm{n}} := \{(x_i,y_i)\}_{i=1}^{m} \subseteq\mathbb{R}^{2} \times\{\pm1\}, \qquad S^{c,\mathrm{w}} := \{(X_c(x_i)/\kappa,y_i)\}_{i=1}^{m}, S n := {( x i , y i ) } i = 1 m ⊆ R 2 × { ± 1 } , S c , w := {( X c ( x i ) / κ , y i ) } i = 1 m , Define
L S n n ( θ ) : = ∑ i = 1 m ℓ ( y i f ( x i ; θ ) ) , \mathcal{L}_{S^{\mathrm{n}}}^{\mathrm{n}}(\theta) := \sum_{i=1}^{m}\ell(y_i f(x_i;\theta)), L S n n ( θ ) := i = 1 ∑ m ℓ ( y i f ( x i ; θ )) , L S c , w w ( ϑ ) : = ∑ i = 1 m ℓ ( y i F ( X c ( x i ) / κ ; ϑ ) ) . \mathcal{L}_{S^{c,\mathrm{w}}}^{\mathrm{w}}(\vartheta) := \sum_{i=1}^{m}\ell(y_i F(X_c(x_i)/\kappa;\vartheta)). L S c , w w ( ϑ ) := i = 1 ∑ m ℓ ( y i F ( X c ( x i ) / κ ; ϑ )) . Then
F ( X c ( x ) κ ; E s , c θ ) = f ( x ; θ ) for every x ∈ R 2 , (125) F\left(\frac{X_c(x)}{\kappa};E_{s,c}\theta\right)=f(x;\theta)\qquad\text{for every }x\in\mathbb{R}^{2}, \tag*{(125)} F ( κ X c ( x ) ; E s , c θ ) = f ( x ; θ ) for every x ∈ R 2 , (125) and, at corresponding parameters, the gradients
g w : = ∇ ϑ L S c , w w ( E s , c θ ) , g n : = ∇ θ L S n n ( θ ) g^{\mathrm{w}} := \nabla_{\vartheta}\mathcal{L}_{S^{c,\mathrm{w}}}^{\mathrm{w}}(E_{s,c}\theta),\qquad
g^{\mathrm{n}} := \nabla_{\theta}\mathcal{L}_{S^{\mathrm{n}}}^{\mathrm{n}}(\theta) g w := ∇ ϑ L S c , w w ( E s , c θ ) , g n := ∇ θ L S n n ( θ ) satisfy
g w = E s , c ♯ g n . (126) g^{\mathrm{w}}=E_{s,c}^{\sharp}g^{\mathrm{n}}. \tag*{(126)} g w = E s , c ♯ g n . (126) Equivalently, for L ≥ 2 L \ge2 L ≥ 2 , their blocks are
g W 1 w = 1 n 1 S 1 X c ( g u n ) ⊤ , (127) g_{W_1}^{\mathrm{w}}=\frac{1}{n_1}S_1X_c(g^{\mathrm{n}}_u)^{\top}, \tag*{(127)} g W 1 w = n 1 1 S 1 X c ( g u n ) ⊤ , (127) g W ℓ w = g b ℓ − 1 n n ℓ n ℓ − 1 S ℓ S ℓ − 1 ⊤ , 2 ≤ ℓ ≤ L − 1 , (128) g_{W_\ell}^{\mathrm{w}}=\frac{g_{b_{\ell-1}}^{\mathrm{n}}}{n_\ell n_{\ell-1}}S_\ell S_{\ell-1}^{\top},\qquad2\le\ell\le L-1, \tag*{(128)} g W ℓ w = n ℓ n ℓ − 1 g b ℓ − 1 n S ℓ S ℓ − 1 ⊤ , 2 ≤ ℓ ≤ L − 1 , (128) g W L w = g b L − 1 n n L − 1 S L − 1 ⊤ . (129) g_{W_L}^{\mathrm{w}}=\frac{g_{b_{L-1}}^{\mathrm{n}}}{n_{L-1}}S_{L-1}^{\top}. \tag*{(129)} g W L w = n L − 1 g b L − 1 n S L − 1 ⊤ . (129) Here g u n = ( g u 1 n , g u 2 n ) ⊤ g^{\mathrm{n}}_u=(g^{\mathrm{n}}_{u_1},g^{\mathrm{n}}_{u_2})^{\top} g u n = ( g u 1 n , g u 2 n ) ⊤ , and the middle identity is vacuous when L = 2 L=2 L = 2 . For L = 1 L=1 L = 1 , the single block is g W 1 w = X c ( g u n ) ⊤ g_{W_1}^{\mathrm{w}}=X_c(g_u^{\mathrm{n}})^{\top} g W 1 w = X c ( g u n ) ⊤ . In particular, every such wide gradient belongs to V s , c V_{s,c} V s , c . Thus parameters lift by E E E , whereas gradients and linear momentum buffers lift by E ♯ E^{\sharp} E ♯ .
Proof. Set D : = E ⊤ E D:=E^{\top}E D := E ⊤ E . For L = 1 L=1 L = 1 , D = diag ( 1 , d − 1 ) D=\operatorname{diag}(1,d-1) D = diag ( 1 , d − 1 ) and E ♯ u = X c ( u ) ⊤ E^{\sharp}u=X_c(u)^{\top} E ♯ u = X c ( u ) ⊤ , so both identities follow from (120) and direct differentiation. The remaining computations assume L ≥ 2 L\ge2 L ≥ 2 . Contracting adjacent sign vectors in the copied weights and using (120) gives
W L ⋯ W 1 = κ P ( b ) U c ( u ) ⊤ , W_L\cdots W_1=\kappa P(b)U_c(u)^{\top}, W L ⋯ W 1 = κ P ( b ) U c ( u ) ⊤ ,
F ( X c ( x ) / κ ; E θ ) = P ( b ) U c ( u ) ⊤ X c ( x ) = f ( x ; θ ) . F(X_c(x)/\kappa;E\theta)=P(b)U_c(u)^{\top}X_c(x)=f(x;\theta). F ( X c ( x ) / κ ; E θ ) = P ( b ) U c ( u ) ⊤ X c ( x ) = f ( x ; θ ) .
Consequently, the losses agree under the lift, i.e. L S n = L S c , w w ∘ E \mathcal{L}^{\mathrm{n}}_{S}=\mathcal{L}^{\mathrm{w}}_{S_c,w}\circ E L S n = L S c , w w ∘ E , and the chain rule gives
g n = E T g w . (130) \mathbf{g}^{\mathrm{n}}=E^{\mathsf{T}}\mathbf{g}^{\mathrm{w}}. \tag*{(130)} g n = E T g w . (130)
The wide gradient lies in im E \operatorname{im} E im E . Indeed, for ξ = X c ( x ) / κ \xi=X_c(x)/\kappa ξ = X c ( x ) / κ , the standard layer derivative is
∇ W ℓ F ( ξ ; ϑ ) = ( W L ⋯ W ℓ + 1 ) T ( W ℓ − 1 ⋯ W 1 ξ ) T , \nabla_{W_\ell}F(\xi;\vartheta)=(W_L\cdots W_{\ell+1})^{\mathsf{T}}(W_{\ell-1}\cdots W_1\xi)^{\mathsf{T}}, ∇ W ℓ F ( ξ ; ϑ ) = ( W L ⋯ W ℓ + 1 ) T ( W ℓ − 1 ⋯ W 1 ξ ) T ,
with empty matrix products interpreted as identities. At ϑ = E θ \vartheta=E\theta ϑ = E θ , every hidden forward or backward vector is a scalar multiple of the corresponding sign vector s ℓ s_\ell s ℓ , and ξ ∈ im U c \xi\in\operatorname{im}U_c ξ ∈ im U c . Thus the gradient blocks have the copy forms in (121) –(123) , so each model gradient belongs to im E \operatorname{im} E im E . The same holds for g w \mathbf{g}^{\mathrm{w}} g w , which is a weighted sum of these gradients. Writing g w = E h \mathbf{g}^{\mathrm{w}}=E h g w = E h , multiplying by E T E^{\mathsf{T}} E T and using (130) now yields E T E h = g n E^{\mathsf{T}}Eh=\mathbf{g}^{\mathrm{n}} E T E h = g n , therefore
g w = E ( E T E ) − 1 g n = E # g n , \mathbf{g}^{\mathrm{w}}=E(E^{\mathsf{T}}E)^{-1}\mathbf{g}^{\mathrm{n}}=E^\#\mathbf{g}^{\mathrm{n}}, g w = E ( E T E ) − 1 g n = E # g n ,
where E T E E^{\mathsf{T}}E E T E is invertible because E E E is injective.
For the explicit block formulas, we compute the adjoint E T : V → Θ E^{\mathsf{T}}:\mathcal{V}\to\Theta E T : V → Θ . Put r c : = ∑ k ≠ c e k r_c:=\sum_{k\ne c}e_k r c := ∑ k = c e k , and use the standard Euclidean inner product on Θ \Theta Θ and the sum of Frobenius inner products on V \mathcal{V} V . For any tuple ϑ = ( W 1 , … , W L ) \vartheta=(W_1,\ldots,W_L) ϑ = ( W 1 , … , W L ) , the adjoint is characterized by
⟨ E T ϑ , θ ⟩ Θ = ⟨ ϑ , E θ ⟩ V , for every θ ∈ Θ . \left\langle E^{\mathsf{T}}\vartheta,\theta\right\rangle_{\Theta}=\langle\vartheta,E\theta\rangle_{\mathcal{V}},\qquad\text{for every }\theta\in\Theta. ⟨ E T ϑ , θ ⟩ Θ = ⟨ ϑ , E θ ⟩ V , for every θ ∈ Θ.
We find its coordinates by expanding the right-hand side and comparing the coefficients of the coordinates of θ \theta θ .
For a matrix W W W and column vectors a , b a,b a , b of compatible dimensions,
⟨ W , a b T ⟩ F = ∑ i , j W i j a i b j = a T W b . \left\langle W,ab^{\mathsf{T}}\right\rangle_F=\sum_{i,j}W_{ij}a_i b_j=a^{\mathsf{T}}Wb. ⟨ W , a b T ⟩ F = i , j ∑ W ij a i b j = a T W b .
Write θ = ( u , b 1 , … , b L − 1 ) \theta=(u,b_1,\ldots,b_{L-1}) θ = ( u , b 1 , … , b L − 1 ) . Since U c ( u ) = u 1 e c + u 2 r c U_c(u)=u_1e_c+u_2r_c U c ( u ) = u 1 e c + u 2 r c , the first block of the lift is
( E θ ) 1 = u 1 s 1 e c T + u 2 s 1 r c T . (E\theta)_1=u_1s_1e_c^{\mathsf{T}}+u_2s_1r_c^{\mathsf{T}}. ( E θ ) 1 = u 1 s 1 e c T + u 2 s 1 r c T .
Its contribution to the inner product is therefore
⟨ W 1 , ( E θ ) 1 ⟩ F = u 1 ⟨ W 1 , s 1 e c T ⟩ F + u 2 ⟨ W 1 , s 1 r c T ⟩ F = u 1 s 1 T W 1 e c + u 2 s 1 T W 1 r c . \left\langle W_1,(E\theta)_1\right\rangle_F=u_1\left\langle W_1,s_1e_c^{\mathsf{T}}\right\rangle_F+u_2\left\langle W_1,s_1r_c^{\mathsf{T}}\right\rangle_F
=u_1s_1^{\mathsf{T}}W_1e_c+u_2s_1^{\mathsf{T}}W_1r_c. ⟨ W 1 , ( E θ ) 1 ⟩ F = u 1 ⟨ W 1 , s 1 e c T ⟩ F + u 2 ⟨ W 1 , s 1 r c T ⟩ F = u 1 s 1 T W 1 e c + u 2 s 1 T W 1 r c .
Applying the same matrix identity to each middle block and to the output block gives the complete expansion
⟨ ϑ , E θ ⟩ V = u 1 s 1 T W 1 e c + u 2 s 1 T W 1 r c + ∑ r = 1 L − 2 b r s r + 1 T W r + 1 s r + b L − 1 W L s L − 1 . \langle\vartheta,E\theta\rangle_{\mathcal{V}}=u_1s_1^{\mathsf{T}}W_1e_c+u_2s_1^{\mathsf{T}}W_1r_c+\sum_{r=1}^{L-2}b_rs_{r+1}^{\mathsf{T}}W_{r+1}s_r+b_{L-1}W_Ls_{L-1}. ⟨ ϑ , E θ ⟩ V = u 1 s 1 T W 1 e c + u 2 s 1 T W 1 r c + r = 1 ∑ L − 2 b r s r + 1 T W r + 1 s r + b L − 1 W L s L − 1 .
Because the defining adjoint identity holds for every θ \theta θ , the coefficients of u 1 , u 2 , b 1 , … , b L − 1 u_1,u_2,b_1,\ldots,b_{L-1} u 1 , u 2 , b 1 , … , b L − 1 are exactly the coordinates of E T ϑ E^{\mathsf{T}}\vartheta E T ϑ . Thus
E T ϑ = ( s 1 T W 1 e c , s 1 T W 1 r c , ( s r + 1 T W r + 1 s r ) r = 1 L − 2 , W L s L − 1 ) . (131) E^{\mathsf{T}}\vartheta=\left(s_1^{\mathsf{T}}W_1e_c,\ s_1^{\mathsf{T}}W_1r_c,\ \left(s_{r+1}^{\mathsf{T}}W_{r+1}s_r\right)_{r=1}^{L-2},\ W_Ls_{L-1}\right). \tag*{(131)} E T ϑ = ( s 1 T W 1 e c , s 1 T W 1 r c , ( s r + 1 T W r + 1 s r ) r = 1 L − 2 , W L s L − 1 ) . (131)
Since ∥ s ℓ ∥ 2 = n ℓ \lVert s_\ell\rVert^2=n_\ell ∥ s ℓ ∥ 2 = n ℓ and ∥ r c ∥ 2 = d − 1 \lVert r_c\rVert^2=d-1 ∥ r c ∥ 2 = d − 1 , applying this to E θ E\theta E θ gives
E T E ( u , b 1 , … , b L − 1 ) = ( n 1 u 1 , n 1 ( d − 1 ) u 2 , ( n r + 1 n r b r ) r = 1 L − 2 , n L − 1 b L − 1 ) . (132) E^{\mathsf{T}}E(u,b_1,\ldots,b_{L-1})=\left(n_1u_1,n_1(d-1)u_2,\left(n_{r+1}n_rb_r\right)_{r=1}^{L-2},n_{L-1}b_{L-1}\right). \tag*{(132)} E T E ( u , b 1 , … , b L − 1 ) = ( n 1 u 1 , n 1 ( d − 1 ) u 2 , ( n r + 1 n r b r ) r = 1 L − 2 , n L − 1 b L − 1 ) . (132)
Substituting this diagonal map into E ( E T E ) − 1 g n E(E^{\mathsf{T}}E)^{-1}\mathbf{g}^{\mathrm{n}} E ( E T E ) − 1 g n gives (127) –(129) . The indexed middle tuple is empty when L = 2 L=2 L = 2 .
Corollary D.18 (The Adam trajectory is exactly lifted). Fix s s s , c ∈ [ d ] c \in[d] c ∈ [ d ] , a differentiable margin loss ℓ \ell ℓ , and a finite dataset
S n = { ( x i , y i ) } i = 1 m ⊆ R 2 × { ± 1 } . S^{\mathrm{n}}=\{(\mathbf{x}_i,y_i)\}_{i=1}^{m}\subseteq\mathbb{R}^{2}\times\{\pm1\}. S n = {( x i , y i ) } i = 1 m ⊆ R 2 × { ± 1 } . Use the corresponding wide dataset
S c , w : = { ( X c ( x i ) / κ , y i ) } i = 1 m . S^{c,\mathrm{w}}:=\{(X_c(\mathbf{x}_i)/\kappa,y_i)\}_{i=1}^{m}. S c , w := {( X c ( x i ) / κ , y i ) } i = 1 m . If θ t \theta_t θ t is the Adam trajectory on S n S^{\mathrm{n}} S n from θ 0 \theta_0 θ 0 , then E s , c θ t E_{s,c}\theta_t E s , c θ t is the Adam trajectory on S c , w S^{c,\mathrm{w}} S c , w from E s , c θ 0 E_{s,c}\theta_0 E s , c θ 0 .
Proof. Let
g n ( t ) : = ∇ θ L S n ( θ t ) , g w ( t ) : = ∇ ϑ L S c , w w ( E s , c θ t ) . \mathbf{g}^{\mathrm{n}}(t):=\nabla_{\theta}\mathcal{L}^{\mathrm{n}}_{S}(\theta_t),\qquad\mathbf{g}^{\mathrm{w}}(t):=\nabla_{\vartheta}\mathcal{L}^{\mathrm{w}}_{S^{c,\mathrm{w}}}(E_{s,c}\theta_t). g n ( t ) := ∇ θ L S n ( θ t ) , g w ( t ) := ∇ ϑ L S c , w w ( E s , c θ t ) . The definition of E s , c E_{s,c} E s , c and Equations (127) –(129) (together with the L = 1 L=1 L = 1 formula in Lemma D.17 ) give for each wide coordinate e e e a narrow coordinate r ( e ) r(e) r ( e ) and a fixed nonzero constant a e a_e a e such that
g e w ( t ) = a e g r ( e ) n ( t ) , (133) g^{\mathrm{w}}_e(t)=a_e g^{\mathrm{n}}_{r(e)}(t), \tag*{(133)} g e w ( t ) = a e g r ( e ) n ( t ) , (133) and
( E s , c θ ) e = sign ( a e ) θ r ( e ) . (134) (E_{s,c}\theta)_e=\operatorname{sign}(a_e)\theta_{r(e)}. \tag*{(134)} ( E s , c θ ) e = sign ( a e ) θ r ( e ) . (134) Let m r n , v r n m^{\mathrm{n}}_r,v^{\mathrm{n}}_r m r n , v r n be the narrow moments. Define wide moments along the candidate trajectory by
m e w ( t ) : = a e m r ( e ) n ( t ) , v e w ( t ) : = a e 2 v r ( e ) n ( t ) . (135) m^{\mathrm{w}}_e(t):=a_e m^{\mathrm{n}}_{r(e)}(t),\qquad v^{\mathrm{w}}_e(t):=a_e^2v^{\mathrm{n}}_{r(e)}(t). \tag*{(135)} m e w ( t ) := a e m r ( e ) n ( t ) , v e w ( t ) := a e 2 v r ( e ) n ( t ) . (135) By linearity of the EMA operators, these are the moments associated with g w \mathbf{g}^{\mathrm{w}} g w . The bias corrections give
m ^ e w v ^ e w = sign ( a e ) m ^ r ( e ) n v ^ r ( e ) n . (136) \frac{\widehat{m}^{\mathrm{w}}_e}{\sqrt{\widehat{v}^{\mathrm{w}}_e}}=\operatorname{sign}(a_e)\frac{\widehat{m}^{\mathrm{n}}_{r(e)}}{\sqrt{\widehat{v}^{\mathrm{n}}_{r(e)}}}. \tag*{(136)} v e w m e w = sign ( a e ) v r ( e ) n m r ( e ) n . (136) This identity also holds at zero second-moment coordinates under the zero-denominator convention. Hence, combining (134) , (136) ,
d d t ( E s , c θ t ) = E s , c θ ˙ t = E s , c ( − η ( t ) m ^ t n v ^ t n ) = − η ( t ) m ^ t w v ^ t w . \frac{d}{dt}(E_{s,c}\theta_t)=E_{s,c}\dot{\theta}_t=E_{s,c}\left(-\eta(t)\frac{\widehat{\mathbf{m}}^{\mathrm{n}}_t}{\sqrt{\widehat{\mathbf{v}}^{\mathrm{n}}_t}}\right)=-\eta(t)\frac{\widehat{\mathbf{m}}^{\mathrm{w}}_t}{\sqrt{\widehat{\mathbf{v}}^{\mathrm{w}}_t}}. d t d ( E s , c θ t ) = E s , c θ ˙ t = E s , c ( − η ( t ) v t n m t n ) = − η ( t ) v t w m t w . So E s , c θ t E_{s,c}\theta_t E s , c θ t solves the Adam equations.
Proposition D.19 (Wide KKT points pull back). Fix s s s and c ∈ [ d ] c \in[d] c ∈ [ d ] , and abbreviate E : = E s , c E:=E_{s,c} E := E s , c . For a function R : V → R R:V\to\mathbb{R} R : V → R , set R s , c : = R ∘ E R_{s,c}:=R\circ E R s , c := R ∘ E . For a dataset S = { ( x i , y i ) } i = 1 m S=\{(\mathbf{x}_i,y_i)\}_{i=1}^{m} S = {( x i , y i ) } i = 1 m , write
q i n ( θ ) : = y i f ( x i ; θ ) , q i w ( ϑ ) : = y i F ( X c ( x i ) / κ ; ϑ ) . q_i^{\mathrm{n}}(\theta):=y_i f(\mathbf{x}_i;\theta),\qquad q_i^{\mathrm{w}}(\vartheta):=y_iF(X_c(\mathbf{x}_i)/\kappa;\vartheta). q i n ( θ ) := y i f ( x i ; θ ) , q i w ( ϑ ) := y i F ( X c ( x i ) / κ ; ϑ ) . Consider the narrow problem
min θ ∈ Θ R s , c ( θ ) subject to q i n ( θ ) ≥ 1 , 1 ≤ i ≤ m . (137) \min_{\theta\in\Theta}R_{s,c}(\theta)\qquad\text{subject to}\qquad q_i^{\mathrm{n}}(\theta)\geq1,\quad1\leq i\leq m. \tag*{(137)} θ ∈ Θ min R s , c ( θ ) subject to q i n ( θ ) ≥ 1 , 1 ≤ i ≤ m . (137) If R R R is convex and ϑ ^ = E θ ^ \widehat{\vartheta}=E\widehat{\theta} ϑ = E θ is a KKT point of
min ϑ ∈ V R ( ϑ ) subject to q i w ( ϑ ) ≥ 1 , 1 ≤ i ≤ m , \min_{\vartheta\in V}R(\vartheta)\qquad\text{subject to}\qquad q_i^{\mathrm{w}}(\vartheta)\geq1,\quad1\leq i\leq m, ϑ ∈ V min R ( ϑ ) subject to q i w ( ϑ ) ≥ 1 , 1 ≤ i ≤ m , then θ ^ \widehat{\theta} θ is a KKT point of (137) .
Proof. The model and gradient identities give
q i w ( E θ ) = q i n ( θ ) , E ⊤ ∇ q i w ( E θ ) = ∇ q i n ( θ ) . (138) q_i^{\mathrm{w}}(E\theta)=q_i^{\mathrm{n}}(\theta),\qquad E^{\top}\nabla q_i^{\mathrm{w}}(E\theta)=\nabla q_i^{\mathrm{n}}(\theta). \tag*{(138)} q i w ( E θ ) = q i n ( θ ) , E ⊤ ∇ q i w ( E θ ) = ∇ q i n ( θ ) . (138) By the wide KKT conditions, there are ζ ∈ ∂ R ( E θ ^ ) \zeta\in\partial R(E\widehat{\theta}) ζ ∈ ∂ R ( E θ ) and λ i ≥ 0 \lambda_i\geq0 λ i ≥ 0 such that
q i w ( E θ ^ ) ≥ 1 , ζ − ∑ i = 1 m λ i ∇ q i w ( E θ ^ ) = 0 , λ i ( q i w ( E θ ^ ) − 1 ) = 0. (139) q_i^{\mathrm{w}}(E\widehat{\theta})\geq1,\qquad\zeta-\sum_{i=1}^{m}\lambda_i\nabla q_i^{\mathrm{w}}(E\widehat{\theta})=0,\qquad\lambda_i\bigl(q_i^{\mathrm{w}}(E\widehat{\theta})-1\bigr)=0. \tag*{(139)} q i w ( E θ ) ≥ 1 , ζ − i = 1 ∑ m λ i ∇ q i w ( E θ ) = 0 , λ i ( q i w ( E θ ) − 1 ) = 0. (139)
For every θ ′ ∈ Θ \theta' \in\Theta θ ′ ∈ Θ , the subgradient inequality for a convex R R R gives
R s , c ( θ ′ ) = R ( E θ ′ ) ≥ R ( E θ ^ ) + ⟨ ζ , E ( θ ′ − θ ^ ) ⟩ = R s , c ( θ ^ ) + ⟨ E ⊤ ζ , θ ′ − θ ^ ⟩ . \begin{aligned}
R_{s,c}(\theta') &= R(E\theta') \\
&\ge R(E\hat{\theta}) + \langle\zeta, E(\theta' - \hat{\theta}) \rangle\\
&= R_{s,c}(\hat{\theta}) + \langle E^{\top}\zeta, \theta' - \hat{\theta} \rangle.
\end{aligned} R s , c ( θ ′ ) = R ( E θ ′ ) ≥ R ( E θ ^ ) + ⟨ ζ , E ( θ ′ − θ ^ )⟩ = R s , c ( θ ^ ) + ⟨ E ⊤ ζ , θ ′ − θ ^ ⟩ .
Thus E ⊤ ζ ∈ ∂ R s , c ( θ ^ ) E^{\top}\zeta\in\partial R_{s,c}(\hat{\theta}) E ⊤ ζ ∈ ∂ R s , c ( θ ^ ) . Applying E ⊤ E^{\top} E ⊤ to the middle equation in (139) and using (138) gives
E ⊤ ζ − ∑ i = 1 m λ i ∇ q i n ( θ ^ ) = 0. E^{\top}\zeta- \sum_{i=1}^{m}\lambda_i\nabla q_i^n(\hat{\theta}) = 0. E ⊤ ζ − i = 1 ∑ m λ i ∇ q i n ( θ ^ ) = 0.
The margin identity in (138) transfers feasibility and complementarity. Hence θ ^ \hat{\theta} θ ^ satisfies the KKT conditions for (137) .
Lemma D.20 (Convex regularizer triviality for the wide model). Let O V , O Θ O_{\mathcal{V}}, O_{\Theta} O V , O Θ be optimizers on the wide and narrow parameter spaces respectively . Let ℓ : R → R \ell: \mathbb{R} \to\mathbb{R} ℓ : R → R . Assume the following two properties.
Trajectory lifting: for every trajectory θ t \theta_t θ t of O Θ O_{\Theta} O Θ on f f f and a dataset S n S^n S n , and every sign pattern s s s and c ∈ [ d ] c \in[d] c ∈ [ d ] , E s , c θ t E_{s,c}\theta_t E s , c θ t is a trajectory of O V O_{\mathcal{V}} O V on F F F and the dataset S c , w S^{c,w} S c , w .
Triviality of the narrow regularizer: every convex implicit regularizer of f f f for the narrow optimizer O Θ O_{\Theta} O Θ with the loss ℓ \ell ℓ is constant on Θ \Theta Θ .
Then every convex implicit regularizer of F F F for O V O_{\mathcal{V}} O V with the loss ℓ \ell ℓ is constant on V \mathcal{V} V .
Proof. Let R : V → R R : \mathcal{V} \to\mathbb{R} R : V → R be such a wide regularizer. Fix s , c s,c s , c , abbreviate E : = E s , c E := E_{s,c} E := E s , c , and set R s , c : = R ∘ E R_{s,c} := R \circ E R s , c := R ∘ E . Since R R R is convex, so is R s , c R_{s,c} R s , c ; we show that R s , c R_{s,c} R s , c is an implicit regularizer for O Θ O_{\Theta} O Θ .
Let θ t \theta_t θ t be a trajectory of O Θ O_{\Theta} O Θ satisfying Assumptions (T1), (T2), and suppose
θ t ∥ θ t ∥ 2 ⟶ θ ˉ ≠ 0 , q min n ( θ ˉ ) > 0. \frac{\theta_t}{\|\theta_t\|_2} \longrightarrow\bar{\theta} \ne0,\qquad q_{\min}^n(\bar{\theta}) > 0. ∥ θ t ∥ 2 θ t ⟶ θ ˉ = 0 , q m i n n ( θ ˉ ) > 0.
By the first assumption, ϑ t = E θ t \vartheta_t = E\theta_t ϑ t = E θ t is a legal wide trajectory on S c , w S^{c,w} S c , w . Injectivity gives E θ ˉ ≠ 0 E\bar{\theta} \ne0 E θ ˉ = 0 , and linearity of E E E gives
ϑ t ∥ ϑ t ∥ 2 = E ( θ t / ∥ θ t ∥ 2 ) ∥ E ( θ t / ∥ θ t ∥ 2 ) ∥ 2 → t → ∞ E θ ˉ ∥ E θ ˉ ∥ 2 = : ϑ ˉ . \frac{\vartheta_t}{\|\vartheta_t\|_2}
=
\frac{E(\theta_t/\|\theta_t\|_2)}{\|E(\theta_t/\|\theta_t\|_2)\|_2}
\xrightarrow{t\to\infty}
\frac{E\bar{\theta}}{\|E\bar{\theta}\|_2}
=: \bar{\vartheta}. ∥ ϑ t ∥ 2 ϑ t = ∥ E ( θ t /∥ θ t ∥ 2 ) ∥ 2 E ( θ t /∥ θ t ∥ 2 ) t → ∞ ∥ E θ ˉ ∥ 2 E θ ˉ =: ϑ ˉ .
Homogeneity and the model equivalence therefore give
q min w ( ϑ ˉ ) = q min n ( θ ˉ ) ∥ E θ ˉ ∥ 2 L > 0. q_{\min}^{w}(\bar{\vartheta})
=
\frac{q_{\min}^{n}(\bar{\theta})}{\|E\bar{\theta}\|_2^{L}}
> 0. q m i n w ( ϑ ˉ ) = ∥ E θ ˉ ∥ 2 L q m i n n ( θ ˉ ) > 0.
Consequently, the wide margin-normalized limit is
ϑ ^ : = ϑ ˉ ( min i q i w ( ϑ ˉ ) ) 1 / L = E θ ^ . \hat{\vartheta} := \frac{\bar{\vartheta}}{\left(\min_i q_i^w(\bar{\vartheta})\right)^{1/L}} = E\hat{\theta}. ϑ ^ := ( min i q i w ( ϑ ˉ ) ) 1/ L ϑ ˉ = E θ ^ .
Assumptions (T1), (T2) hold for ϑ t \vartheta_t ϑ t : L w ( ϑ t ) = L n ( θ t ) → t → ∞ 0 \mathcal{L}^{w}(\vartheta_t) = \mathcal{L}^{n}(\theta_t) \xrightarrow{t\to\infty} 0 L w ( ϑ t ) = L n ( θ t ) t → ∞ 0 and γ w ( ϑ t ) = γ n ( θ t ) ⋅ ( ∥ θ t ∥ 2 ∥ ϑ t ∥ 2 ) L ≥ γ 0 ∥ E ∥ 2 L > 0 \gamma^{w}(\vartheta_t) = \gamma^{n}(\theta_t) \cdot\left(\frac{\|\theta_t\|_2}{\|\vartheta_t\|_2}\right)^L \ge\frac{\gamma_0}{\|E\|_2^L} > 0 γ w ( ϑ t ) = γ n ( θ t ) ⋅ ( ∥ ϑ t ∥ 2 ∥ θ t ∥ 2 ) L ≥ ∥ E ∥ 2 L γ 0 > 0 .
Therefore, by definition of R R R , E θ ^ E\hat{\theta} E θ ^ is a KKT point for the wide problem. Proposition D.19 then makes θ ^ \hat{\theta} θ ^ KKT for the narrow problem with objective R s , c R_{s,c} R s , c . Hence R s , c R_{s,c} R s , c is a regularizer for the narrow model.
The second assumption makes R s , c R_{s,c} R s , c constant on Θ \Theta Θ . Its value is R s , c ( 0 ) = R ( 0 ) R_{s,c}(0) = R(0) R s , c ( 0 ) = R ( 0 ) , and E E E maps Θ \Theta Θ onto V s , c \mathcal{V}_{s,c} V s , c . Therefore
R ∣ V s , c ≡ R ( 0 ) for every s and c . R|_{\mathcal{V}_{s,c}} \equiv R(0) \qquad\text{for every } s \text{ and } c. R ∣ V s , c ≡ R ( 0 ) for every s and c .
Lemma D.16 now makes R R R constant on all of V \mathcal{V} V .
Corollary D.21 (No nonconstant convex wide Adam regularizer). Let α > 1 L − 1 2 \alpha> \frac{1}{L} - \frac{1}{2} α > L 1 − 2 1 . For any scalar-output linear network F F F of depth L L L , input dimension d ≥ 2 d \ge2 d ≥ 2 , any choice of positive hidden widths and a total of p p p parameters, every convex implicit regularizer of Adam on F F F with the exponential loss is constant on R p \mathbb{R}^{p} R p .
Proof. This is precisely the statement of Lemma D.20 , following from Corollary D.18 and the narrow (width-1) construction.
General Homogeneous Models
Preconditioned Momentum GD
Proposition D.22. Let α > 1 L − 1 2 \alpha> \frac{1}{L} - \frac{1}{2} α > L 1 − 2 1 , f f f satisfying ( M 1 ) (M1) ( M 1 ) , ( M 2 ) (M2) ( M 2 ) and ℓ ∈ L e \ell\in\mathcal{L}_{e} ℓ ∈ L e . Under Assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) , for every j ∈ [ p ] j \in[p] j ∈ [ p ] ,
ω j : = c 2 ∫ 0 ∞ e c 2 s g s [ j ] 2 d s < ∞ , ω j − c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s ≤ O ( t − 2 ( α − ( 1 L − 1 2 ) ) ) (140) \omega_j := c_2 \int_{0}^{\infty} e^{c_2s}g_s[j]^2\,ds < \infty,\qquad\omega_j - c_2\int_{0}^{t} e^{c_2s}g_s[j]^2\,ds \le O\left(t^{-2\left(\alpha-\left(\frac{1}{L}-\frac{1}{2}\right)\right)}\right) \tag*{(140)} ω j := c 2 ∫ 0 ∞ e c 2 s g s [ j ] 2 d s < ∞ , ω j − c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s ≤ O ( t − 2 ( α − ( L 1 − 2 1 ) ) ) (140) Therefore Adam asymptotically follows a preconditioned momentum gradient descent trajectory. Namely, it holds that
d θ t d t = − D ω η ( t ) e c 2 t 2 m t ( 1 + o ( 1 ) ) , ( D ω ) j , j = { ω j − 1 / 2 : ω j > 0 , 1 : ω j = 0. \frac{d\theta_t}{dt}=-D_{\omega}\eta(t)e^{\frac{c_2t}{2}}m_t(1+o(1)),\qquad(D_{\omega})_{j,j}=\begin{cases}\omega_j^{-1/2} & :\omega_j>0,\\ 1 & :\omega_j=0.\end{cases} d t d θ t = − D ω η ( t ) e 2 c 2 t m t ( 1 + o ( 1 )) , ( D ω ) j , j = { ω j − 1/2 1 : ω j > 0 , : ω j = 0.
Proof. Equation (140) is exactly the result of Corollary C.10. Set
I j ( t ) : = c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s . I_j(t):=c_2\int_{0}^{t}e^{c_2s}g_s[j]^2\,ds. I j ( t ) := c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s .
For every j j j with ω j > 0 \omega_j>0 ω j > 0 , it holds that I j ( t ) > 0 I_j(t)>0 I j ( t ) > 0 for all sufficiently large t t t . Since v j ( t ) = e − c 2 t I j ( t ) v_j(t)=e^{-c_2t}I_j(t) v j ( t ) = e − c 2 t I j ( t ) and
b ^ ( t ) = 1 − e − c 2 t 1 − e − c 1 t → 1 , \widehat b(t)=\frac{\sqrt{1-e^{-c_2t}}}{1-e^{-c_1t}}\to1, b ( t ) = 1 − e − c 1 t 1 − e − c 2 t → 1 ,
define
( E t ) j , j : = { b ^ ( t ) I j ( t ) − 1 / 2 − ( D ω ) j , j : ω j > 0 , 0 : ω j = 0. (E_t)_{j,j}:=\begin{cases}\widehat b(t)I_j(t)^{-1/2}-(D_{\omega})_{j,j} & :\omega_j>0,\\ 0 & :\omega_j=0.\end{cases} ( E t ) j , j := { b ( t ) I j ( t ) − 1/2 − ( D ω ) j , j 0 : ω j > 0 , : ω j = 0.
Then ∥ E t ∥ o p → 0 \|E_t\|_{\mathrm{op}}\to0 ∥ E t ∥ op → 0 . Note that if ω j = 0 \omega_j=0 ω j = 0 then m t [ j ] = v t [ j ] = 0 m_t[j]=v_t[j]=0 m t [ j ] = v t [ j ] = 0 for all t t t , so the Adam equation is exactly
θ ˙ t = − η ( t ) e c 2 t 2 ( D ω + E t ) m t . \dot{\theta}_t=-\eta(t)e^{\frac{c_2t}{2}}(D_{\omega}+E_t)m_t. θ ˙ t = − η ( t ) e 2 c 2 t ( D ω + E t ) m t .
Equivalently, for each coordinate, either ( D ω + E t ) j , j = ( D ω ) j , j (D_{\omega}+E_t)_{j,j}=(D_{\omega})_{j,j} ( D ω + E t ) j , j = ( D ω ) j , j when ω j = 0 \omega_j=0 ω j = 0 or
( D ω + E t ) j , j = ( D ω ) j , j [ 1 + b ^ ( t ) ω j I j ( t ) − 1 ] , (D_{\omega}+E_t)_{j,j}=(D_{\omega})_{j,j}\left[1+\widehat b(t)\sqrt{\frac{\omega_j}{I_j(t)}}-1\right], ( D ω + E t ) j , j = ( D ω ) j , j [ 1 + b ( t ) I j ( t ) ω j − 1 ] ,
when ω j > 0 \omega_j>0 ω j > 0 , where the expression in square brackets is 1 + o ( 1 ) 1+o(1) 1 + o ( 1 ) .
:::
Trajectory-dependent KKT
We show that every directional limit point satisfies the trajectory-dependent KKT conditions.
Theorem D.23 (Every directional limit point is KKT). Let f f f satisfy ( M 1 ) (M1) ( M 1 ) , ( M 2 ) (M2) ( M 2 ) and let α > 1 / L − 1 / 2 \alpha> 1/L-1/2 α > 1/ L − 1/2 , ℓ ∈ L e \ell\in\mathcal{L}_{e} ℓ ∈ L e . Let θ t \theta_t θ t be an Adam trajectory under assumptions ( T 1 ) (T1) ( T 1 ) , ( T 2 ) (T2) ( T 2 ) and let θ ˉ \bar{\theta} θ ˉ be any limit point of θ t / ∥ θ t ∥ 2 \theta_t/\|\theta_t\|_2 θ t /∥ θ t ∥ 2 as t → ∞ t\to\infty t → ∞ . Then q min ( θ ˉ ) > 0 q_{\min}(\bar{\theta})>0 q m i n ( θ ˉ ) > 0 , and θ ^ : = θ ˉ / q min ( θ ˉ ) 1 / L \hat{\theta}:=\bar{\theta}/q_{\min}(\bar{\theta})^{1/L} θ ^ := θ ˉ / q m i n ( θ ˉ ) 1/ L is a Clarke–KKT point of
min θ 1 2 θ T D ω − 1 θ s.t. q i ( θ ) = y i f ( x i ; θ ) ≥ 1 ( i ∈ [ m ] ) . (141) \min_{\theta}\frac{1}{2}\theta^{\mathsf{T}}D_{\omega}^{-1}\theta\qquad\text{s.t.}\qquad q_i(\theta)=y_i f(x_i;\theta)\ge1\quad(i\in[m]). \tag*{(141)} θ min 2 1 θ T D ω − 1 θ s.t. q i ( θ ) = y i f ( x i ; θ ) ≥ 1 ( i ∈ [ m ]) . (141)
For brevity, write D = D ω D = D_{\omega} D = D ω . The second-moment freezing result makes D D D a fixed positive definite diagonal matrix, including when some ω j = 0 \omega_j = 0 ω j = 0 . We use the weighted inner product and norm
⟨ u , v ⟩ D − 1 : = u T D − 1 v , ∥ u ∥ D − 1 2 : = ⟨ u , u ⟩ D − 1 . \langle\mathbf{u},\mathbf{v}\rangle_{D^{-1}} := \mathbf{u}^{\mathsf{T}}D^{-1}\mathbf{v}, \qquad\|\mathbf{u}\|_{D^{-1}}^{2} := \langle\mathbf{u},\mathbf{u}\rangle_{D^{-1}}. ⟨ u , v ⟩ D − 1 := u T D − 1 v , ∥ u ∥ D − 1 2 := ⟨ u , u ⟩ D − 1 .
For the remainder, work under Assumptions (T1), (T2) and α > 1 / L − 1 / 2 \alpha> 1/L - 1/2 α > 1/ L − 1/2 , and set
ν : = 1 − 1 L , ζ : = α + ν > 1 2 , λ : = c 1 − c 2 2 > 0. \nu:= 1-\frac{1}{L}, \qquad\zeta:= \alpha+\nu>\frac{1}{2}, \qquad\lambda:= c_{1}-\frac{c_{2}}{2}>0. ν := 1 − L 1 , ζ := α + ν > 2 1 , λ := c 1 − 2 c 2 > 0.
Let
I j ( t ) : = c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s , a ( t ) : = η 0 t α e c 2 2 t , ( B t ) j , j : = { ω j / I j ( t ) , ω j > 0 , 1 , ω j = 0. I_j(t):=c_2\int_0^t e^{c_2s}g_s[j]^2\,\mathrm{d}s,\qquad a(t):=\eta_0t^\alpha e^{\frac{c_2}{2}t},\qquad(B_t)_{j,j}:=
\begin{cases}
\sqrt{\omega_j/I_j(t)}, & \omega_j>0,\\
1, & \omega_j=0.
\end{cases} I j ( t ) := c 2 ∫ 0 t e c 2 s g s [ j ] 2 d s , a ( t ) := η 0 t α e 2 c 2 t , ( B t ) j , j := { ω j / I j ( t ) , 1 , ω j > 0 , ω j = 0.
These definitions are valid on a sufficiently late tail. Set
A t : = η ^ ( t ) η 0 t α B t . A_t:=\frac{\widehat{\eta}(t)}{\eta_0t^\alpha}B_t. A t := η 0 t α η ( t ) B t .
The exact frozen dynamics and the estimates established in the second-moment freezing give for some C > 0 C>0 C > 0
θ ˙ t = − a ( t ) A t D m t , m ˙ t = c 1 ( g t − m t ) , (142) \dot{\theta}_t=-a(t)A_tD\mathbf{m}_t,\qquad\dot{\mathbf{m}}_t=c_1(\mathbf{g}_t-\mathbf{m}_t), \tag*{(142)} θ ˙ t = − a ( t ) A t D m t , m ˙ t = c 1 ( g t − m t ) , (142)
∥ g t ∥ 2 ≤ C e − c 2 2 t t − ζ , ∥ B t − I ∥ o p ≤ C t − ( 2 ζ − 1 ) . (143) \|\mathbf{g}_t\|_2\le Ce^{-\frac{c_2}{2}t}t^{-\zeta},\qquad\|B_t-I\|_{\mathrm{op}}\le Ct^{-(2\zeta-1)}. \tag*{(143)} ∥ g t ∥ 2 ≤ C e − 2 c 2 t t − ζ , ∥ B t − I ∥ op ≤ C t − ( 2 ζ − 1 ) . (143)
If ω j = 0 \omega_j=0 ω j = 0 , then g t [ j ] = 0 g_t[j]=0 g t [ j ] = 0 almost everywhere, m t [ j ] = v t [ j ] = 0 m_t[j]=v_t[j]=0 m t [ j ] = v t [ j ] = 0 , and θ t [ j ] \theta_t[j] θ t [ j ] is constant; hence the same equations apply. We also use the selected decomposition and chain rule
− g t = ∑ i = 1 m φ ′ ( q i ( θ t ) ) ℓ ( q i ( θ t ) ) h i ( t ) , h i ( t ) ∈ ∂ q i ( θ t ) , (144) -\mathbf{g}_t=\sum_{i=1}^{m}\varphi'(q_i(\theta_t))\ell(q_i(\theta_t))\mathbf{h}_i(t),\qquad\mathbf{h}_i(t)\in\partial q_i(\theta_t), \tag*{(144)} − g t = i = 1 ∑ m φ ′ ( q i ( θ t )) ℓ ( q i ( θ t )) h i ( t ) , h i ( t ) ∈ ∂ q i ( θ t ) , (144)
d d t L ( θ t ) = ⟨ g t , θ ˙ t ⟩ . (145) \frac{\mathrm{d}}{\mathrm{d}t}\mathcal{L}(\theta_t)=\langle\mathbf{g}_t,\dot{\theta}_t\rangle. \tag*{(145)} d t d L ( θ t ) = ⟨ g t , θ ˙ t ⟩ . (145)
Define the discrepancy from purely radial growth and the error in the frozen dynamics by
E t : = − a ( t ) D m t − θ t L t , e t : = − a ( t ) ( A t − I ) D m t . (146) \mathcal{E}_t:=-a(t)D\mathbf{m}_t-\frac{\theta_t}{Lt},\qquad\mathbf{e}_t:=-a(t)(A_t-I)D\mathbf{m}_t. \tag*{(146)} E t := − a ( t ) D m t − L t θ t , e t := − a ( t ) ( A t − I ) D m t . (146)
Thus
θ ˙ t = − a ( t ) D m t + e t = θ t L t + E t + e t . (147) \dot{\theta}_t=-a(t)D\mathbf{m}_t+\mathbf{e}_t=\frac{\theta_t}{Lt}+\mathcal{E}_t+\mathbf{e}_t. \tag*{(147)} θ ˙ t = − a ( t ) D m t + e t = L t θ t + E t + e t . (147)
Lemma D.24 (Bounds for the dynamics). There exist T , c , C > 0 T,c,C>0 T , c , C > 0 such that, for all t ≥ T t\ge T t ≥ T ,
c t 1 / L ≤ ∥ θ t ∥ 2 ≤ C t 1 / L , q min ( θ t ) ≥ c t , ∥ m t ∥ 2 ≤ C e − c 2 2 t t − ζ , (148) ct^{1/L}\le\|\theta_t\|_2\le Ct^{1/L},\qquad q_{\min}(\theta_t)\ge ct,\qquad\|\mathbf{m}_t\|_2\le Ce^{-\frac{c_2}{2}t}t^{-\zeta}, \tag*{(148)} c t 1/ L ≤ ∥ θ t ∥ 2 ≤ C t 1/ L , q m i n ( θ t ) ≥ c t , ∥ m t ∥ 2 ≤ C e − 2 c 2 t t − ζ , (148) L ( θ t ) ≤ C e − c 2 2 t t − ( ζ + ν ) , ∥ E t ∥ 2 ≤ C t − ν , \mathcal{L}(\theta_t)\le Ce^{-\frac{c_2}{2}t}t^{-(\zeta+\nu)},\qquad\|\mathcal{E}_t\|_2\le Ct^{-\nu}, L ( θ t ) ≤ C e − 2 c 2 t t − ( ζ + ν ) , ∥ E t ∥ 2 ≤ C t − ν , and
∥ e t ∥ 2 ≤ C t − ν − δ , δ : = min { 1 , 2 ζ − 1 } > 0. (149) \|\mathbf{e}_t\|_2\le Ct^{-\nu-\delta},\qquad\delta:=\min\{1,2\zeta-1\}>0. \tag*{(149)} ∥ e t ∥ 2 ≤ C t − ν − δ , δ := min { 1 , 2 ζ − 1 } > 0. (149) Almost everywhere on this tail,
∥ g t ∥ 2 ≤ C e − c 2 2 t t − ζ ≤ C t − ν . (150) \|\mathbf{g}_t\|_2\le Ce^{-\frac{c_2}{2}t}t^{-\zeta}\le Ct^{-\nu}. \tag*{(150)} ∥ g t ∥ 2 ≤ C e − 2 c 2 t t − ζ ≤ C t − ν . (150)
Proof. Since ζ > 1 / 2 \zeta>1/2 ζ > 1/2 , Theorem C.13 gives for some C > 0 C>0 C > 0 , with U ( t ) : = − log L ( θ t ) U(t):=-\log\mathcal{L}(\theta_t) U ( t ) := − log L ( θ t ) ,
U ( t ) = c 2 2 t + ( ζ + ν ) log t + O ( 1 ) , U(t)=\frac{c_2}{2}t+(\zeta+\nu)\log t+O(1), U ( t ) = 2 c 2 t + ( ζ + ν ) log t + O ( 1 ) ,
L ( θ t ) ≤ C e − c 2 2 t t − ( ζ + ν ) . (151) \mathcal{L}(\theta_t)\le Ce^{-\frac{c_2}{2}t}t^{-(\zeta+\nu)}. \tag*{(151)} L ( θ t ) ≤ C e − 2 c 2 t t − ( ζ + ν ) . (151)
Combining this with Lemma C.3 yields for some C > 0 C > 0 C > 0
∥ θ t ∥ 2 ≍ t 1 / L , q min ( θ t ) ≲ t , ∥ g t ∥ 2 ≤ C e − c 2 2 t t − ζ , \lVert\theta_t\rVert_2 \asymp t^{1/L}, \qquad q_{\min}(\theta_t) \lesssim t, \qquad\lVert g_t\rVert_2 \le C e^{-\frac{c_2}{2}t}t^{-\zeta}, ∥ θ t ∥ 2 ≍ t 1/ L , q m i n ( θ t ) ≲ t , ∥ g t ∥ 2 ≤ C e − 2 c 2 t t − ζ ,
where the gradient estimate holds almost everywhere. Corollary C.12 and v t [ j ] = e − c 2 t I j ( t ) ≤ ω j e − c 2 t v_t[j] = e^{-c_2t}I_j(t) \le\omega_j e^{-c_2t} v t [ j ] = e − c 2 t I j ( t ) ≤ ω j e − c 2 t imply for some C > 0 C > 0 C > 0
∥ m t ∥ 2 ≤ C t − ζ ( ∑ j v t [ j ] ) 1 / 2 ≤ C e − c 2 2 t t − ζ . \lVert m_t\rVert_2 \le Ct^{-\zeta}\left(\sum_j v_t[j]\right)^{1/2} \le Ce^{-\frac{c_2}{2}t}t^{-\zeta}. ∥ m t ∥ 2 ≤ C t − ζ ( j ∑ v t [ j ] ) 1/2 ≤ C e − 2 c 2 t t − ζ .
Coordinates with ω j = 0 \omega_j = 0 ω j = 0 contribute zero. Boundedness of A t A_t A t , D D D and (142) therefore give for some C > 0 C > 0 C > 0 , almost everywhere,
∥ θ ˙ t ∥ 2 ≤ C a ( t ) ∥ m t ∥ 2 ≤ C t α − ζ = C t − ν . \lVert\dot{\theta}_t\rVert_2 \le Ca(t)\lVert m_t\rVert_2 \le Ct^{\alpha-\zeta}=Ct^{-\nu}. ∥ θ ˙ t ∥ 2 ≤ C a ( t ) ∥ m t ∥ 2 ≤ C t α − ζ = C t − ν .
Finally, since η ^ ( t ) / ( η 0 t α ) = 1 + O ( t − 1 ) \widehat{\eta}(t)/(\eta_0t^\alpha)=1+O(t^{-1}) η ( t ) / ( η 0 t α ) = 1 + O ( t − 1 ) , the freezing estimate in (143) gives ∥ A t − I ∥ o p ≤ C t − δ \lVert A_t-I\rVert_{\mathrm{op}}\le Ct^{-\delta} ∥ A t − I ∥ op ≤ C t − δ . The definitions in (146) now yield for some C > 0 C > 0 C > 0
∥ e t ∥ 2 ≤ a ( t ) ∥ A t − I ∥ o p ∥ D m t ∥ 2 ≤ C t − ν − δ , \lVert e_t\rVert_2 \le a(t)\lVert A_t-I\rVert_{\mathrm{op}}\lVert Dm_t\rVert_2 \le Ct^{-\nu-\delta}, ∥ e t ∥ 2 ≤ a ( t ) ∥ A t − I ∥ op ∥ D m t ∥ 2 ≤ C t − ν − δ ,
∥ E t ∥ 2 ≤ a ( t ) ∥ D m t ∥ 2 + ∥ θ t ∥ 2 L t ≤ C t − ν . \lVert E_t\rVert_2 \le a(t)\lVert Dm_t\rVert_2+\frac{\lVert\theta_t\rVert_2}{Lt}\le Ct^{-\nu}. ∥ E t ∥ 2 ≤ a ( t ) ∥ D m t ∥ 2 + L t ∥ θ t ∥ 2 ≤ C t − ν .
All bounds hold on a common sufficiently late tail.
Although pointwise we only have the bound ∥ E t ∥ 2 ≲ t − ν \lVert E_t\rVert_2\lesssim t^{-\nu} ∥ E t ∥ 2 ≲ t − ν , in fact we can show summability of t 2 ν − 1 ∥ E t ∥ 2 t^{2\nu-1}\lVert E_t\rVert^2 t 2 ν − 1 ∥ E t ∥ 2 .
Lemma D.25 (Finite energy of the radial discrepancy). For sufficiently large T > 0 T>0 T > 0 ,
∫ T ∞ t 2 ν ∥ E t ∥ D − 1 2 d t t < ∞ . (152) \int_T^\infty t^{2\nu}\lVert E_t\rVert_{D^{-1}}^2\frac{dt}{t}<\infty. \tag*{(152)} ∫ T ∞ t 2 ν ∥ E t ∥ D − 1 2 t d t < ∞. (152) Proof. Fix a sufficiently large T T T as in Lemma D.24, and let T ′ ≥ T T' \ge T T ′ ≥ T . All constants below are independent of T ′ T' T ′ . All differential identities and inequalities hold almost everywhere. Define the nonnegative, locally absolutely continuous function
C ( t ) : = ∥ θ t ∥ D − 1 2 2 t 2 / L + L 2 λ t 2 ν − 1 ∥ E t ∥ D − 1 2 + c 1 L λ t 2 ν − 1 a ( t ) L ( θ t ) . C(t):=\frac{\lVert\theta_t\rVert_{D^{-1}}^2}{2t^{2/L}}+\frac{L}{2\lambda}t^{2\nu-1}\lVert E_t\rVert_{D^{-1}}^2+\frac{c_1L}{\lambda}t^{2\nu-1}a(t)\mathcal{L}(\theta_t). C ( t ) := 2 t 2/ L ∥ θ t ∥ D − 1 2 + 2 λ L t 2 ν − 1 ∥ E t ∥ D − 1 2 + λ c 1 L t 2 ν − 1 a ( t ) L ( θ t ) . We show that for some C > 0 C > 0 C > 0 ,
C ˙ ( t ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C t − 1 − δ , (153) \dot{C}(t)\le-Lt^{2\nu-1}\lVert E_t\rVert_{D^{-1}}^2+Ct^{-1-\delta}, \tag*{(153)} C ˙ ( t ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C t − 1 − δ , (153) which implies the lemma’s claim: integrating and using C ( T ′ ) ≥ 0 C(T')\ge0 C ( T ′ ) ≥ 0 , we conclude that
L ∫ T T ′ t 2 ν ∥ E t ∥ D − 1 2 d t t ≤ C ( T ) − C ( T ′ ) + C ∫ T T ′ t − 1 − δ d t ≤ C ( T ) + C δ T − δ . L\int_T^{T'}t^{2\nu}\lVert E_t\rVert_{D^{-1}}^2\frac{dt}{t}\le C(T)-C(T')+C\int_T^{T'}t^{-1-\delta}dt\le C(T)+\frac{C}{\delta}T^{-\delta}. L ∫ T T ′ t 2 ν ∥ E t ∥ D − 1 2 t d t ≤ C ( T ) − C ( T ′ ) + C ∫ T T ′ t − 1 − δ d t ≤ C ( T ) + δ C T − δ . Letting T ′ → ∞ T'\to\infty T ′ → ∞ proves (152) . We proceed to prove (153) . First, Equation (147) and the identity θ t = L t ( − a ( t ) D m t − E t ) \theta_t=Lt(-a(t)Dm_t-E_t) θ t = L t ( − a ( t ) D m t − E t ) give
d d t ∥ θ t ∥ D − 1 2 2 t 2 / L = t − 2 / L ⟨ θ t , θ ˙ t ⟩ D − 1 − 1 L t − 2 / L − 1 ∥ θ t ∥ D − 1 2 = t − 2 / L ⟨ θ t , θ ˙ t − θ t / ( L t ) ⟩ D − 1 = t − 2 / L ⟨ θ t , E ˙ t ⟩ D − 1 + t − 2 / L ⟨ θ t , e t ⟩ D − 1 = L t 1 − 2 / L ⟨ − a ( t ) D m t − E t , E t ⟩ D − 1 + t − 2 / L ⟨ θ t , e t ⟩ D − 1 = L t 2 ν − 1 ⟨ − a ( t ) D m t , E t ⟩ D − 1 − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + t − 2 / L ⟨ θ t , e t ⟩ D − 1 , \begin{aligned}
\frac{d}{dt}\frac{\lVert\theta_t\rVert_{D^{-1}}^2}{2t^{2/L}}
&=t^{-2/L}\langle\theta_t,\dot{\theta}_t\rangle_{D^{-1}}-\frac{1}{L}t^{-2/L-1}\lVert\theta_t\rVert_{D^{-1}}^2\\
&=t^{-2/L}\langle\theta_t,\dot{\theta}_t-\theta_t/(Lt)\rangle_{D^{-1}}\\
&=t^{-2/L}\langle\theta_t,\dot{E}_t\rangle_{D^{-1}}+t^{-2/L}\langle\theta_t,e_t\rangle_{D^{-1}}\\
&=Lt^{1-2/L}\langle-a(t)Dm_t-E_t,E_t\rangle_{D^{-1}}+t^{-2/L}\langle\theta_t,e_t\rangle_{D^{-1}}\\
&=Lt^{2\nu-1}\langle-a(t)Dm_t,E_t\rangle_{D^{-1}}-Lt^{2\nu-1}\lVert E_t\rVert_{D^{-1}}^2+t^{-2/L}\langle\theta_t,e_t\rangle_{D^{-1}},
\end{aligned} d t d 2 t 2/ L ∥ θ t ∥ D − 1 2 = t − 2/ L ⟨ θ t , θ ˙ t ⟩ D − 1 − L 1 t − 2/ L − 1 ∥ θ t ∥ D − 1 2 = t − 2/ L ⟨ θ t , θ ˙ t − θ t / ( L t ) ⟩ D − 1 = t − 2/ L ⟨ θ t , E ˙ t ⟩ D − 1 + t − 2/ L ⟨ θ t , e t ⟩ D − 1 = L t 1 − 2/ L ⟨ − a ( t ) D m t − E t , E t ⟩ D − 1 + t − 2/ L ⟨ θ t , e t ⟩ D − 1 = L t 2 ν − 1 ⟨ − a ( t ) D m t , E t ⟩ D − 1 − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + t − 2/ L ⟨ θ t , e t ⟩ D − 1 ,
where 1 − 2 / L = 2 ν − 1 1-2/L=2\nu-1 1 − 2/ L = 2 ν − 1 .
For the second term, the momentum equation in (142) and a ′ ( t ) / a ( t ) = c 2 / 2 + α / t a'(t)/a(t)=c_2/2+\alpha/t a ′ ( t ) / a ( t ) = c 2 /2 + α / t give
d d t ( − a ( t ) D m t ) = − c 1 a ( t ) D g t − λ ( − a ( t ) D m t ) + α t ( − a ( t ) D m t ) . (154) \frac{d}{dt}\left(-a(t)D m_t\right)=-c_1a(t)D g_t-\lambda\left(-a(t)D m_t\right)+\frac{\alpha}{t}\left(-a(t)D m_t\right). \tag*{(154)} d t d ( − a ( t ) D m t ) = − c 1 a ( t ) D g t − λ ( − a ( t ) D m t ) + t α ( − a ( t ) D m t ) . (154)
Differentiating (146) and using (154) gives
ε ˙ t = d d t ( − a ( t ) D m t ) − θ ˙ t L t + θ t L t 2 = − c 1 a ( t ) D g t − λ ( − a ( t ) D m t ) + α t ( − a ( t ) D m t ) − 1 L t ( θ ˙ t − θ t t ) . \begin{aligned}
\dot{\varepsilon}_t=\frac{d}{dt}\left(-a(t)D m_t\right)-\frac{\dot{\theta}_t}{Lt}+\frac{\theta_t}{Lt^2} \\
&=-c_1a(t)D g_t-\lambda\left(-a(t)D m_t\right)+\frac{\alpha}{t}\left(-a(t)D m_t\right)-\frac{1}{Lt}\left(\dot{\theta}_t-\frac{\theta_t}{t}\right).
\end{aligned} ε ˙ t = d t d ( − a ( t ) D m t ) − L t θ ˙ t + L t 2 θ t = − c 1 a ( t ) D g t − λ ( − a ( t ) D m t ) + t α ( − a ( t ) D m t ) − L t 1 ( θ ˙ t − t θ t ) .
Thus, by the product rule,
d d t ( L 2 λ t 2 ν − 1 ∥ ε t ∥ D − 1 2 ) = L λ t 2 ν − 1 ⟨ ε ˙ t , ε t ⟩ D − 1 + L ( 2 ν − 1 ) 2 λ t 2 ν − 2 ∥ ε t ∥ D − 1 2 = c 1 L λ t 2 ν − 1 ⟨ − a ( t ) D g t , ε t ⟩ D − 1 − L t 2 ν − 1 ⟨ − a ( t ) D m t , ε t ⟩ D − 1 + α L λ t 2 ν − 2 ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − 1 λ t 2 ν − 2 ⟨ θ ˙ t − θ t t , ε t ⟩ D − 1 + L ( 2 ν − 1 ) 2 λ t 2 ν − 2 ∥ ε t ∥ D − 1 2 . (155) \begin{aligned}
\frac{d}{dt}\left(\frac{L}{2\lambda}t^{2\nu-1}\lVert\varepsilon_t\rVert_{D^{-1}}^2\right)
={}&\frac{L}{\lambda}t^{2\nu-1}\langle\dot{\varepsilon}_t,\varepsilon_t\rangle_{D^{-1}}
+\frac{L(2\nu-1)}{2\lambda}t^{2\nu-2}\lVert\varepsilon_t\rVert_{D^{-1}}^2\\
={}&\frac{c_1L}{\lambda}t^{2\nu-1}\langle-a(t)Dg_t,\varepsilon_t\rangle_{D^{-1}}\\
&-Lt^{2\nu-1}\langle-a(t)Dm_t,\varepsilon_t\rangle_{D^{-1}}\\
&+\frac{\alpha L}{\lambda}t^{2\nu-2}\langle-a(t)Dm_t,\varepsilon_t\rangle_{D^{-1}}\\
&-\frac{1}{\lambda}t^{2\nu-2}\left\langle\dot{\theta}_t-\frac{\theta_t}{t},\varepsilon_t\right\rangle_{D^{-1}}\\
&+\frac{L(2\nu-1)}{2\lambda}t^{2\nu-2}\lVert\varepsilon_t\rVert_{D^{-1}}^2. \tag*{(155)}
\end{aligned} d t d ( 2 λ L t 2 ν − 1 ∥ ε t ∥ D − 1 2 ) = = λ L t 2 ν − 1 ⟨ ε ˙ t , ε t ⟩ D − 1 + 2 λ L ( 2 ν − 1 ) t 2 ν − 2 ∥ ε t ∥ D − 1 2 λ c 1 L t 2 ν − 1 ⟨ − a ( t ) D g t , ε t ⟩ D − 1 − L t 2 ν − 1 ⟨ − a ( t ) D m t , ε t ⟩ D − 1 + λ α L t 2 ν − 2 ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − λ 1 t 2 ν − 2 ⟨ θ ˙ t − t θ t , ε t ⟩ D − 1 + 2 λ L ( 2 ν − 1 ) t 2 ν − 2 ∥ ε t ∥ D − 1 2 . (155)
For the third term, the product rule, a ′ ( t ) / a ( t ) = c 2 / 2 + α / t a'(t)/a(t)=c_2/2+\alpha/t a ′ ( t ) / a ( t ) = c 2 /2 + α / t , and the chain rule (145) give
d d t ( c 1 L λ t 2 ν − 1 a ( t ) L ( θ t ) ) = c 1 L ( 2 ν − 1 ) λ t 2 ν − 2 a ( t ) L ( θ t ) + c 1 L λ t 2 ν − 1 a ′ ( t ) L ( θ t ) + c 1 L λ t 2 ν − 1 a ( t ) d d t L ( θ t ) = c 1 L λ t 2 ν − 1 a ( t ) ⟨ g t , θ ˙ t ⟩ + c 1 L c 2 2 λ t 2 ν − 1 a ( t ) L ( θ t ) + c 1 L ( α + 2 ν − 1 ) λ t 2 ν − 2 a ( t ) L ( θ t ) . (156) \begin{aligned}
\frac{d}{dt}\left(\frac{c_1L}{\lambda}t^{2\nu-1}a(t)\mathcal{L}(\theta_t)\right)
={}&\frac{c_1L(2\nu-1)}{\lambda}t^{2\nu-2}a(t)\mathcal{L}(\theta_t)\\
&+\frac{c_1L}{\lambda}t^{2\nu-1}a'(t)\mathcal{L}(\theta_t)\\
&+\frac{c_1L}{\lambda}t^{2\nu-1}a(t)\frac{d}{dt}\mathcal{L}(\theta_t)\\
={}&\frac{c_1L}{\lambda}t^{2\nu-1}a(t)\langle g_t,\dot{\theta}_t\rangle\\
&+\frac{c_1Lc_2}{2\lambda}t^{2\nu-1}a(t)\mathcal{L}(\theta_t)\\
&+\frac{c_1L(\alpha+2\nu-1)}{\lambda}t^{2\nu-2}a(t)\mathcal{L}(\theta_t). \tag*{(156)}
\end{aligned} d t d ( λ c 1 L t 2 ν − 1 a ( t ) L ( θ t ) ) = = λ c 1 L ( 2 ν − 1 ) t 2 ν − 2 a ( t ) L ( θ t ) + λ c 1 L t 2 ν − 1 a ′ ( t ) L ( θ t ) + λ c 1 L t 2 ν − 1 a ( t ) d t d L ( θ t ) λ c 1 L t 2 ν − 1 a ( t ) ⟨ g t , θ ˙ t ⟩ + 2 λ c 1 L c 2 t 2 ν − 1 a ( t ) L ( θ t ) + λ c 1 L ( α + 2 ν − 1 ) t 2 ν − 2 a ( t ) L ( θ t ) . (156)
Adding the three derivatives gives
C ˙ ( t ) = − L t 2 ν − 1 ∥ ε t ∥ D − 1 2 + L t 2 ν − 1 [ ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − ⟨ − a ( t ) D m t , ε t ⟩ D − 1 ] + c 1 L λ t 2 ν − 1 [ ⟨ − a ( t ) D g t , ε t ⟩ D − 1 + a ( t ) ⟨ g t , θ ˙ t ⟩ + c 2 2 a ( t ) L ( θ t ) ] + t − 2 / L ⟨ θ t , ε t ⟩ D − 1 + t 2 ν − 2 [ α L λ ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − 1 λ ⟨ θ ˙ t − θ t t , ε t ⟩ D − 1 + L ( 2 ν − 1 ) 2 λ ∥ ε t ∥ D − 1 2 + c 1 L ( α + 2 ν − 1 ) λ a ( t ) L ( θ t ) ] . \begin{aligned}
\dot{C}(t)={}&-Lt^{2\nu-1}\lVert\varepsilon_t\rVert_{D^{-1}}^2\\
&+Lt^{2\nu-1}\left[\langle-a(t)Dm_t,\varepsilon_t\rangle_{D^{-1}}-\langle-a(t)Dm_t,\varepsilon_t\rangle_{D^{-1}}\right]\\
&+\frac{c_1L}{\lambda}t^{2\nu-1}\left[\langle-a(t)Dg_t,\varepsilon_t\rangle_{D^{-1}}+a(t)\langle g_t,\dot{\theta}_t\rangle+\frac{c_2}{2}a(t)\mathcal{L}(\theta_t)\right]\\
&+t^{-2/L}\langle\theta_t,\varepsilon_t\rangle_{D^{-1}}\\
&+t^{2\nu-2}\left[\frac{\alpha L}{\lambda}\langle-a(t)Dm_t,\varepsilon_t\rangle_{D^{-1}}-\frac{1}{\lambda}\left\langle\dot{\theta}_t-\frac{\theta_t}{t},\varepsilon_t\right\rangle_{D^{-1}}+\frac{L(2\nu-1)}{2\lambda}\lVert\varepsilon_t\rVert_{D^{-1}}^2+\frac{c_1L(\alpha+2\nu-1)}{\lambda}a(t)\mathcal{L}(\theta_t)\right].
\end{aligned} C ˙ ( t ) = − L t 2 ν − 1 ∥ ε t ∥ D − 1 2 + L t 2 ν − 1 [ ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − ⟨ − a ( t ) D m t , ε t ⟩ D − 1 ] + λ c 1 L t 2 ν − 1 [ ⟨ − a ( t ) D g t , ε t ⟩ D − 1 + a ( t ) ⟨ g t , θ ˙ t ⟩ + 2 c 2 a ( t ) L ( θ t ) ] + t − 2/ L ⟨ θ t , ε t ⟩ D − 1 + t 2 ν − 2 [ λ α L ⟨ − a ( t ) D m t , ε t ⟩ D − 1 − λ 1 ⟨ θ ˙ t − t θ t , ε t ⟩ D − 1 + 2 λ L ( 2 ν − 1 ) ∥ ε t ∥ D − 1 2 + λ c 1 L ( α + 2 ν − 1 ) a ( t ) L ( θ t ) ] .
The two terms in (*) coming from the first and second derivatives, cancel exactly. For (**), Equation (147) gives
( ∗ ∗ ) = c 1 L λ t 2 ν − 1 a ( t ) [ − ⟨ g t , E t ⟩ + ⟨ g t , θ ˙ t ⟩ + c 2 2 L ( θ t ) ] = c 1 L λ t 2 ν − 1 a ( t ) [ 1 L t ⟨ g t , θ t ⟩ + c 2 2 L ( θ t ) + ⟨ g t , e t ⟩ ] . (**) = \frac{c_1L}{\lambda}t^{2\nu-1}a(t)\left[-\langle g_t,\mathcal{E}_t\rangle+\langle g_t,\dot{\theta}_t\rangle+\frac{c_2}{2}\mathcal{L}(\theta_t)\right]
= \frac{c_1L}{\lambda}t^{2\nu-1}a(t)\left[\frac{1}{Lt}\langle g_t,\theta_t\rangle+\frac{c_2}{2}\mathcal{L}(\theta_t)+\langle g_t,e_t\rangle\right]. ( ∗ ∗ ) = λ c 1 L t 2 ν − 1 a ( t ) [ − ⟨ g t , E t ⟩ + ⟨ g t , θ ˙ t ⟩ + 2 c 2 L ( θ t ) ] = λ c 1 L t 2 ν − 1 a ( t ) [ L t 1 ⟨ g t , θ t ⟩ + 2 c 2 L ( θ t ) + ⟨ g t , e t ⟩ ] .
Equations (47) and (151) imply, after enlarging T T T if necessary,
− 1 L ⟨ g t , θ t ⟩ ≥ ( U ( t ) − φ ( 0 ) ) L ( θ t ) ≥ c 2 2 t L ( θ t ) ( t ≥ T ) . (157) -\frac{1}{L}\langle g_t,\theta_t\rangle\geq\left(U(t)-\varphi(0)\right)\mathcal{L}(\theta_t)\geq\frac{c_2}{2}t\mathcal{L}(\theta_t)\qquad(t\geq T). \tag*{(157)} − L 1 ⟨ g t , θ t ⟩ ≥ ( U ( t ) − φ ( 0 ) ) L ( θ t ) ≥ 2 c 2 t L ( θ t ) ( t ≥ T ) . (157)
Thus the first two terms in the bracket have nonpositive sum. Equations (150) and (149) , together with α − ζ = − ν \alpha-\zeta=-\nu α − ζ = − ν , therefore give, for some C > 0 C>0 C > 0 ,
( ∗ ∗ ) ≤ c 1 L λ t 2 ν − 1 a ( t ) ⟨ g t , e t ⟩ ≤ C t 2 ν − 1 a ( t ) ∥ g t ∥ 2 ∥ e t ∥ 2 ≤ C t 2 ν − 1 + α − ζ − ν − δ = C t − 1 − δ . (**)\leq\frac{c_1L}{\lambda}t^{2\nu-1}a(t)\langle g_t,e_t\rangle\leq Ct^{2\nu-1}a(t)\|g_t\|_2\|e_t\|_2\leq Ct^{2\nu-1+\alpha-\zeta-\nu-\delta}=Ct^{-1-\delta}. ( ∗ ∗ ) ≤ λ c 1 L t 2 ν − 1 a ( t ) ⟨ g t , e t ⟩ ≤ C t 2 ν − 1 a ( t ) ∥ g t ∥ 2 ∥ e t ∥ 2 ≤ C t 2 ν − 1 + α − ζ − ν − δ = C t − 1 − δ .
For (***), Equations (148) , (150) , (146) , and (151) imply, for some C > 0 C>0 C > 0 ,
∥ a ( t ) D m t ∥ D − 1 ≤ C a ( t ) ∥ m t ∥ 2 ≤ C t α e c 2 t / 2 e − c 2 t / 2 t − ζ = C t − ν , \|a(t)Dm_t\|_{D^{-1}}\leq Ca(t)\|m_t\|_2\leq Ct^\alpha e^{c_2t/2}e^{-c_2t/2}t^{-\zeta}=Ct^{-\nu}, ∥ a ( t ) D m t ∥ D − 1 ≤ C a ( t ) ∥ m t ∥ 2 ≤ C t α e c 2 t /2 e − c 2 t /2 t − ζ = C t − ν ,
∥ E t ∥ D − 1 ≤ ∥ a ( t ) D m t ∥ D − 1 + ∥ θ t ∥ D − 1 L t ≤ C t − ν + C t 1 / L − 1 ≤ C t − ν , \|\mathcal{E}_t\|_{D^{-1}}\leq\|a(t)Dm_t\|_{D^{-1}}+\frac{\|\theta_t\|_{D^{-1}}}{Lt}\leq Ct^{-\nu}+Ct^{1/L-1}\leq Ct^{-\nu}, ∥ E t ∥ D − 1 ≤ ∥ a ( t ) D m t ∥ D − 1 + L t ∥ θ t ∥ D − 1 ≤ C t − ν + C t 1/ L − 1 ≤ C t − ν ,
∥ θ ˙ t − θ t / t ∥ D − 1 ≤ ∥ θ ˙ t ∥ D − 1 + ∥ θ t ∥ D − 1 t ≤ C t − ν + C t 1 / L − 1 ≤ C t − ν , \|\dot{\theta}_t-\theta_t/t\|_{D^{-1}}\leq\|\dot{\theta}_t\|_{D^{-1}}+\frac{\|\theta_t\|_{D^{-1}}}{t}\leq Ct^{-\nu}+Ct^{1/L-1}\leq Ct^{-\nu}, ∥ θ ˙ t − θ t / t ∥ D − 1 ≤ ∥ θ ˙ t ∥ D − 1 + t ∥ θ t ∥ D − 1 ≤ C t − ν + C t 1/ L − 1 ≤ C t − ν ,
a ( t ) L ( θ t ) ≤ C t α − ζ − ν = C t − 2 ν . a(t)\mathcal{L}(\theta_t)\leq Ct^{\alpha-\zeta-\nu}=Ct^{-2\nu}. a ( t ) L ( θ t ) ≤ C t α − ζ − ν = C t − 2 ν .
Here we used α − ζ = − ν \alpha-\zeta=-\nu α − ζ = − ν and 1 / L − 1 = − ν 1/L-1=-\nu 1/ L − 1 = − ν . By Cauchy–Schwarz and (149) , for some C > 0 C>0 C > 0 ,
∣ ( ∗ ∗ ∗ ) ∣ ≤ C t − 2 / L ∥ θ t ∥ 2 ∥ e t ∥ 2 + C t 2 ν − 2 [ ∥ a ( t ) D m t ∥ D − 1 ∥ E t ∥ D − 1 + ∥ θ ˙ t − θ t / t ∥ D − 1 ∥ E t ∥ D − 1 + ∥ E t ∥ D − 1 2 + a ( t ) L ( θ t ) ] ≤ C t − 2 / L t 1 / L t − ν − δ + C t 2 ν − 2 t − 2 ν = C ( t − 1 − δ + t − 2 ) . \begin{aligned}
|( ***)|&\leq Ct^{-2/L}\|\theta_t\|_2\|e_t\|_2\\
&\quad+Ct^{2\nu-2}\left[\|a(t)Dm_t\|_{D^{-1}}\|\mathcal{E}_t\|_{D^{-1}}+\|\dot{\theta}_t-\theta_t/t\|_{D^{-1}}\|\mathcal{E}_t\|_{D^{-1}}+\|\mathcal{E}_t\|_{D^{-1}}^2+a(t)\mathcal{L}(\theta_t)\right]\\
&\leq Ct^{-2/L}t^{1/L}t^{-\nu-\delta}+Ct^{2\nu-2}t^{-2\nu}\\
&=C\left(t^{-1-\delta}+t^{-2}\right).
\end{aligned} ∣ ( ∗ ∗ ∗ ) ∣ ≤ C t − 2/ L ∥ θ t ∥ 2 ∥ e t ∥ 2 + C t 2 ν − 2 [ ∥ a ( t ) D m t ∥ D − 1 ∥ E t ∥ D − 1 + ∥ θ ˙ t − θ t / t ∥ D − 1 ∥ E t ∥ D − 1 + ∥ E t ∥ D − 1 2 + a ( t ) L ( θ t ) ] ≤ C t − 2/ L t 1/ L t − ν − δ + C t 2 ν − 2 t − 2 ν = C ( t − 1 − δ + t − 2 ) .
Combining the estimates for (), (), and ( ), and using 0 < δ ≤ 1 0<\delta\leq1 0 < δ ≤ 1 , we obtain, for some C > 0 C>0 C > 0 ,
C ˙ ( t ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C ( t − 2 + t − 1 − δ ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C t − 1 − δ , \dot{\mathcal{C}}(t)\leq-Lt^{2\nu-1}\|\mathcal{E}_t\|_{D^{-1}}^2+C\left(t^{-2}+t^{-1-\delta}\right)\leq-Lt^{2\nu-1}\|\mathcal{E}_t\|_{D^{-1}}^2+Ct^{-1-\delta}, C ˙ ( t ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C ( t − 2 + t − 1 − δ ) ≤ − L t 2 ν − 1 ∥ E t ∥ D − 1 2 + C t − 1 − δ ,
which proves (153) .
Lemma D.26 (Finite energy implies KKT). Assume (T1), (T2). Let θ ˉ \bar{\theta} θ ˉ be any limit point of θ t / ∥ θ t ∥ 2 \theta_t/\|\theta_t\|_2 θ t /∥ θ t ∥ 2 , and define
γ ∗ : = q min ( θ ˉ ) > 0 , κ ∗ : = ∥ θ ˉ ∥ D − 1 2 > 0 , \gamma_*:=q_{\min}(\bar{\theta})>0,\qquad\kappa_*:=\|\bar{\theta}\|_{D^{-1}}^2>0, γ ∗ := q m i n ( θ ˉ ) > 0 , κ ∗ := ∥ θ ˉ ∥ D − 1 2 > 0 , I ∗ : = { i ∈ [ m ] : q i ( θ ˉ ) = γ ∗ } , K ∗ : = conv ⋃ i ∈ I ∗ ∂ q i ( θ ˉ ) . I_*:=\{i\in[m]:q_i(\bar{\theta})=\gamma_*\},\qquad K_*:=\operatorname{conv}\bigcup_{i\in I_*}\partial q_i(\bar{\theta}). I ∗ := { i ∈ [ m ] : q i ( θ ˉ ) = γ ∗ } , K ∗ := conv i ∈ I ∗ ⋃ ∂ q i ( θ ˉ ) . Then
L γ ∗ κ ∗ D − 1 θ ˉ ∈ K ∗ . (158) \frac{L\gamma_*}{\kappa_*}D^{-1}\bar{\theta}\in K_*. \tag*{(158)} κ ∗ L γ ∗ D − 1 θ ˉ ∈ K ∗ . (158) Proof. Choose t n → ∞ t_n\to\infty t n → ∞ such that θ t n / ∥ θ t n ∥ 2 → θ ˉ \theta_{t_n}/\|\theta_{t_n}\|_2\to\bar{\theta} θ t n /∥ θ t n ∥ 2 → θ ˉ , and set J n : = [ t n , 2 t n ] J_n:=[t_n,2t_n] J n := [ t n , 2 t n ] . We show the following in succession:
θ 2 t n − θ t n ∥ θ t n ∥ 2 ⟶ ( 2 1 / L − 1 ) θ ˉ , \frac{\theta_{2t_n}-\theta_{t_n}}{\|\theta_{t_n}\|_2}\longrightarrow(2^{1/L}-1)\bar{\theta}, ∥ θ t n ∥ 2 θ 2 t n − θ t n ⟶ ( 2 1/ L − 1 ) θ ˉ ,
− 1 ∥ θ t n ∥ 2 ∫ J n a ( t ) g t d t ⟶ λ ( 2 1 / L − 1 ) c 1 D − 1 θ ˉ . -\frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n} a(t)g_t\,dt \longrightarrow\frac{\lambda(2^{1/L}-1)}{c_1}D^{-1}\bar{\theta}. − ∥ θ t n ∥ 2 1 ∫ J n a ( t ) g t d t ⟶ c 1 λ ( 2 1/ L − 1 ) D − 1 θ ˉ .
The latter, after suppressing the inactive margins, will supply the desired inclusion.
For every s ∈ J n s\in J_n s ∈ J n ,
θ s − ( s t n ) 1 / L θ t n = s 1 / L ∫ t n s t − 1 / L ( E t + e t ) d t . \theta_s-\left(\frac{s}{t_n}\right)^{1/L}\theta_{t_n}=s^{1/L}\int_{t_n}^{s}t^{-1/L}(\mathcal{E}_t+e_t)\,dt. θ s − ( t n s ) 1/ L θ t n = s 1/ L ∫ t n s t − 1/ L ( E t + e t ) d t .
The lower bound ∥ θ t ∥ 2 ≥ c t 1 / L \lVert\theta_t\rVert_2\ge ct^{1/L} ∥ θ t ∥ 2 ≥ c t 1/ L implies s 1 / L ∥ θ t n ∥ 2 ≤ ( 2 t n ) 1 / L c t n 1 / L = 2 1 / L c \frac{s^{1/L}}{\lVert\theta_{t_n}\rVert_2}\le\frac{(2t_n)^{1/L}}{ct_n^{1/L}}=\frac{2^{1/L}}{c} ∥ θ t n ∥ 2 s 1/ L ≤ c t n 1/ L ( 2 t n ) 1/ L = c 2 1/ L . This, (149) , and Cauchy–Schwarz therefore imply that there exists a constant C > 0 C>0 C > 0 such that
∥ θ s − ( s / t n ) 1 / L θ t n ∥ 2 ∥ θ t n ∥ 2 ≤ C ∫ J n t − 1 / L ( ∥ E t ∥ D − 1 + ∥ e t ∥ D − 1 ) d t ≤ C ∫ J n t ν ∥ E t ∥ D − 1 d t t + C ∫ t n 2 t n t − 1 / L − ν − δ d t ≤ C ( ∫ J n t 2 ν ∥ E t ∥ D − 1 2 d t t ) 1 / 2 ( ∫ J n 2 d t t ) 1 / 2 + C ∫ t n 2 t n t − 1 − δ d t ≤ C log 2 ( ∫ J n t 2 ν ∥ E t ∥ D − 1 2 d t t ) 1 / 2 + C t n − δ ⟶ 0. \begin{aligned}
\frac{\left\lVert\theta_s-(s/t_n)^{1/L}\theta_{t_n}\right\rVert_2}{\lVert\theta_{t_n}\rVert_2}
\le C\int_{J_n}t^{-1/L}\left(\lVert\mathcal{E}_t\rVert_{D^{-1}}+\lVert e_t\rVert_{D^{-1}}\right)\,dt \\
&\le C\int_{J_n}t^\nu\lVert\mathcal{E}_t\rVert_{D^{-1}}\frac{dt}{t}
+C\int_{t_n}^{2t_n}t^{-1/L-\nu-\delta}\,dt \\
&\le C\left(\int_{J_n}t^{2\nu}\lVert\mathcal{E}_t\rVert_{D^{-1}}^2\frac{dt}{t}\right)^{1/2}
\left(\int_{J_n}2\frac{dt}{t}\right)^{1/2}
+C\int_{t_n}^{2t_n}t^{-1-\delta}\,dt \\
&\le C\sqrt{\log2}\left(\int_{J_n}t^{2\nu}\lVert\mathcal{E}_t\rVert_{D^{-1}}^2\frac{dt}{t}\right)^{1/2}
+Ct_n^{-\delta}\longrightarrow0.
\end{aligned} ∥ θ t n ∥ 2 θ s − ( s / t n ) 1/ L θ t n 2 ≤ C ∫ J n t − 1/ L ( ∥ E t ∥ D − 1 + ∥ e t ∥ D − 1 ) d t ≤ C ∫ J n t ν ∥ E t ∥ D − 1 t d t + C ∫ t n 2 t n t − 1/ L − ν − δ d t ≤ C ( ∫ J n t 2 ν ∥ E t ∥ D − 1 2 t d t ) 1/2 ( ∫ J n 2 t d t ) 1/2 + C ∫ t n 2 t n t − 1 − δ d t ≤ C log 2 ( ∫ J n t 2 ν ∥ E t ∥ D − 1 2 t d t ) 1/2 + C t n − δ ⟶ 0.
Here the integral over J n J_n J n tends to zero by Lemma D.25. Thus θ t / ∥ θ t ∥ 2 → θ ˉ \theta_t/\lVert\theta_t\rVert_2\to\bar{\theta} θ t / ∥ θ t ∥ 2 → θ ˉ uniformly on J n J_n J n , and
θ 2 t n − θ t n ∥ θ t n ∥ 2 ⟶ ( 2 1 / L − 1 ) θ ˉ . \frac{\theta_{2t_n}-\theta_{t_n}}{\lVert\theta_{t_n}\rVert_2}\longrightarrow(2^{1/L}-1)\bar{\theta}. ∥ θ t n ∥ 2 θ 2 t n − θ t n ⟶ ( 2 1/ L − 1 ) θ ˉ .
Integrating (154) over J n J_n J n , and using − a ( t ) D m t = θ ˙ t − e t -a(t)Dm_t=\dot{\theta}_t-e_t − a ( t ) D m t = θ ˙ t − e t , gives
− c 1 ∥ θ t n ∥ 2 ∫ J n a ( t ) D g t d t = λ θ 2 t n − θ t n ∥ θ t n ∥ 2 − [ a ( t ) D m t ] t n 2 t n ∥ θ t n ∥ 2 + α ∥ θ t n ∥ 2 ∫ J n a ( t ) D m t t d t − λ ∥ θ t n ∥ 2 ∫ J n e t d t . -\frac{c_1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}a(t)Dg_t\,dt
=\lambda\frac{\theta_{2t_n}-\theta_{t_n}}{\lVert\theta_{t_n}\rVert_2}
-\frac{[a(t)Dm_t]_{t_n}^{2t_n}}{\lVert\theta_{t_n}\rVert_2}
+\frac{\alpha}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\frac{a(t)Dm_t}{t}\,dt
-\frac{\lambda}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}e_t\,dt. − ∥ θ t n ∥ 2 c 1 ∫ J n a ( t ) D g t d t = λ ∥ θ t n ∥ 2 θ 2 t n − θ t n − ∥ θ t n ∥ 2 [ a ( t ) D m t ] t n 2 t n + ∥ θ t n ∥ 2 α ∫ J n t a ( t ) D m t d t − ∥ θ t n ∥ 2 λ ∫ J n e t d t .
The momentum and parameter bounds make the boundary term and the integral containing α \alpha α of order O ( t n − 1 ) O(t_n^{-1}) O ( t n − 1 ) . The error bound makes the last term O ( t n − δ ) O(t_n^{-\delta}) O ( t n − δ ) . Therefore
− 1 ∥ θ t n ∥ 2 ∫ J n a ( t ) g t d t ⟶ λ ( 2 1 / L − 1 ) c 1 D − 1 θ ˉ . (159) -\frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}a(t)g_t\,dt\longrightarrow\frac{\lambda(2^{1/L}-1)}{c_1}D^{-1}\bar{\theta}. \tag*{(159)} − ∥ θ t n ∥ 2 1 ∫ J n a ( t ) g t d t ⟶ c 1 λ ( 2 1/ L − 1 ) D − 1 θ ˉ . (159)
Next we show that we can focus only on the active margins. Homogeneity and (144) give
h i ( t ) ∥ θ t ∥ 2 L − 1 ∈ ∂ q i ( θ t ∥ θ t ∥ 2 ) , − a ( t ) g t = ∑ i = 1 m b i ( t ) h i ( t ) ∥ θ t ∥ 2 L − 1 , \frac{\mathbf{h}_i(t)}{\lVert\theta_t\rVert_2^{L-1}}\in\partial q_i\left(\frac{\theta_t}{\lVert\theta_t\rVert_2}\right),\qquad
-a(t)g_t=\sum_{i=1}^{m}b_i(t)\frac{\mathbf{h}_i(t)}{\lVert\theta_t\rVert_2^{L-1}}, ∥ θ t ∥ 2 L − 1 h i ( t ) ∈ ∂ q i ( ∥ θ t ∥ 2 θ t ) , − a ( t ) g t = i = 1 ∑ m b i ( t ) ∥ θ t ∥ 2 L − 1 h i ( t ) ,
where
b i ( t ) : = a ( t ) ∥ θ t ∥ 2 L − 1 φ ′ ( q i ( θ t ) ) ℓ ( q i ( θ t ) ) ≥ 0. b_i(t):=a(t)\lVert\theta_t\rVert_2^{L-1}\varphi'(q_i(\theta_t))\ell(q_i(\theta_t))\ge0. b i ( t ) := a ( t ) ∥ θ t ∥ 2 L − 1 φ ′ ( q i ( θ t )) ℓ ( q i ( θ t )) ≥ 0.
The upper bound φ ′ ≤ Φ M ′ \varphi'\le\Phi'_M φ ′ ≤ Φ M ′ , the bounds in (148) and 1 − ν = 1 L 1-\nu=\frac{1}{L} 1 − ν = L 1 imply that there exist constants B , C > 0 B,C>0 B , C > 0 with
∑ i b i ( t ) ≤ Φ M ′ a ( t ) ∥ θ t ∥ 2 L − 1 L ( θ t ) ≤ C t − ν , \sum_i b_i(t)\le\Phi'_M a(t)\lVert\theta_t\rVert_2^{L-1}\mathcal{L}(\theta_t)\le Ct^{-\nu}, i ∑ b i ( t ) ≤ Φ M ′ a ( t ) ∥ θ t ∥ 2 L − 1 L ( θ t ) ≤ C t − ν ,
1 ∥ θ t n ∥ 2 ∫ J n ∑ i b i ( t ) d t ≤ C c ( 1 − ν ) t n − 1 / L ( ( 2 t n ) 1 − ν − t n 1 − ν ) ≤ B . (160) \frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_i b_i(t)\,dt
\le\frac{C}{c(1-\nu)}t_n^{-1/L}\left((2t_n)^{1-\nu}-t_n^{1-\nu}\right)\le B. \tag*{(160)} ∥ θ t n ∥ 2 1 ∫ J n i ∑ b i ( t ) d t ≤ c ( 1 − ν ) C t n − 1/ L ( ( 2 t n ) 1 − ν − t n 1 − ν ) ≤ B . (160)
For any i ∉ I ∗ i \notin I_* i ∈ / I ∗ , fix j ∈ I ∗ j \in I_* j ∈ I ∗ . Uniform convergence of the directions on J n J_n J n and continuity of the margins imply that there exists d i > 0 d_i > 0 d i > 0 such that, for all large n n n ,
q i ( θ t ∥ θ t ∥ 2 ) − q j ( θ t ∥ θ t ∥ 2 ) ≥ d i > 0 ( t ∈ J n ) . q_i\left(\frac{\theta_t}{\lVert\theta_t\rVert_2}\right)-q_j\left(\frac{\theta_t}{\lVert\theta_t\rVert_2}\right)\ge d_i>0 \qquad(t\in J_n). q i ( ∥ θ t ∥ 2 θ t ) − q j ( ∥ θ t ∥ 2 θ t ) ≥ d i > 0 ( t ∈ J n ) .
Since q i ( θ t ) = ∥ θ t ∥ 2 L q i ( θ t / ∥ θ t ∥ 2 ) q_i(\theta_t)=\lVert\theta_t\rVert_2^Lq_i(\theta_t/\lVert\theta_t\rVert_2) q i ( θ t ) = ∥ θ t ∥ 2 L q i ( θ t / ∥ θ t ∥ 2 ) and φ ′ \varphi' φ ′ has a positive lower bound on the attained positive margin range, there exists a constant c > 0 c>0 c > 0 such that, for some C > 0 C>0 C > 0 ,
b i ( t ) b j ( t ) = φ ′ ( q i ( θ t ) ) φ ′ ( q j ( θ t ) ) e − φ ( q i ( θ t ) ) + φ ( q j ( θ t ) ) ≤ C e − c d i t ( t ∈ J n ) . \frac{b_i(t)}{b_j(t)}=\frac{\varphi'(q_i(\theta_t))}{\varphi'(q_j(\theta_t))}e^{-\varphi(q_i(\theta_t))+\varphi(q_j(\theta_t))}\le Ce^{-c d_i t}\qquad(t\in J_n). b j ( t ) b i ( t ) = φ ′ ( q j ( θ t )) φ ′ ( q i ( θ t )) e − φ ( q i ( θ t )) + φ ( q j ( θ t )) ≤ C e − c d i t ( t ∈ J n ) .
Together with (160) , this gives
1 ∥ θ t n ∥ 2 ∫ J n ∑ i ∉ I ∗ b i ( t ) d t ⟶ 0. \frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_{i\notin I_*}b_i(t)\,dt\longrightarrow0. ∥ θ t n ∥ 2 1 ∫ J n i ∈ / I ∗ ∑ b i ( t ) d t ⟶ 0.
For each i ∈ I ∗ i\in I_* i ∈ I ∗ , let h i ∗ ( t ) \mathbf{h}_i^*(t) h i ∗ ( t ) be the Euclidean projection of h i ( t ) / ∥ θ t ∥ 2 L − 1 \mathbf{h}_i(t)/\lVert\theta_t\rVert_2^{L-1} h i ( t ) / ∥ θ t ∥ 2 L − 1 onto ∂ q i ( θ ˉ ) \partial q_i(\bar{\theta}) ∂ q i ( θ ˉ ) . Upper semicontinuity of ∂ q i \partial q_i ∂ q i and uniform convergence of the directions on J n J_n J n give ε n → 0 \varepsilon_n\to0 ε n → 0 such that
h i ∗ ( t ) ∈ ∂ q i ( θ ˉ ) , ∥ h i ( t ) ∥ θ t ∥ 2 L − 1 − h i ∗ ( t ) ∥ 2 ≤ ε n ( i ∈ I ∗ ) . \mathbf{h}_i^*(t)\in\partial q_i(\bar{\theta}),\qquad\left\lVert\frac{\mathbf{h}_i(t)}{\lVert\theta_t\rVert_2^{L-1}}-\mathbf{h}_i^*(t)\right\rVert_2\le\varepsilon_n\qquad(i\in I_*). h i ∗ ( t ) ∈ ∂ q i ( θ ˉ ) , ∥ θ t ∥ 2 L − 1 h i ( t ) − h i ∗ ( t ) 2 ≤ ε n ( i ∈ I ∗ ) .
for almost every t ∈ J n t\in J_n t ∈ J n . Equation (160) and local boundedness of the normalized subgradients therefore give, for some C > 0 C>0 C > 0 ,
1 ∥ θ t n ∥ 2 ∫ J n ∥ − a ( t ) g t − ∑ i ∈ I ∗ b i ( t ) h i ∗ ( t ) ∥ 2 d t ≤ B ε n + C ∥ θ t n ∥ 2 ∫ J n ∑ i ∉ I ∗ b i ( t ) d t ⟶ 0. \frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\left\lVert-a(t)\mathbf{g}_t-\sum_{i\in I_*}b_i(t)\mathbf{h}_i^*(t)\right\rVert_2\,dt
\le B\varepsilon_n+\frac{C}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_{i\notin I_*}b_i(t)\,dt\longrightarrow0. ∥ θ t n ∥ 2 1 ∫ J n − a ( t ) g t − i ∈ I ∗ ∑ b i ( t ) h i ∗ ( t ) 2 d t ≤ B ε n + ∥ θ t n ∥ 2 C ∫ J n i ∈ / I ∗ ∑ b i ( t ) d t ⟶ 0.
Equation (159) therefore gives
1 ∥ θ t n ∥ 2 ∫ J n ∑ i ∈ I ∗ b i ( t ) h i ∗ ( t ) d t ⟶ λ ( 2 1 / L − 1 ) c 1 D − 1 θ ˉ . \frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_{i\in I_*}b_i(t)\mathbf{h}_i^*(t)\,dt\longrightarrow\frac{\lambda(2^{1/L}-1)}{c_1}D^{-1}\bar{\theta}. ∥ θ t n ∥ 2 1 ∫ J n i ∈ I ∗ ∑ b i ( t ) h i ∗ ( t ) d t ⟶ c 1 λ ( 2 1/ L − 1 ) D − 1 θ ˉ .
It remains to normalize the coefficients. Define
a n : = 1 ∥ θ t n ∥ 2 ∫ J n ∑ i ∈ I ∗ b i ( t ) d t . a_n:=\frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_{i\in I_*}b_i(t)\,dt. a n := ∥ θ t n ∥ 2 1 ∫ J n i ∈ I ∗ ∑ b i ( t ) d t .
Euler’s identity gives ⟨ θ ˉ , h i ∗ ( t ) ⟩ = L γ ∗ \langle\bar{\theta},\mathbf{h}_i^*(t)\rangle=L\gamma_* ⟨ θ ˉ , h i ∗ ( t )⟩ = L γ ∗ for every active index. Taking the inner product of the preceding limit with θ ˉ \bar{\theta} θ ˉ , we obtain
L γ ∗ a n = ⟨ θ ˉ , 1 ∥ θ t n ∥ 2 ∫ J n ∑ i ∈ I ∗ b i ( t ) h i ∗ ( t ) d t ⟩ ⟶ λ ( 2 1 / L − 1 ) κ ∗ c 1 . L\gamma_*a_n=\left\langle\bar{\theta},\frac{1}{\lVert\theta_{t_n}\rVert_2}\int_{J_n}\sum_{i\in I_*}b_i(t)\mathbf{h}_i^*(t)\,dt\right\rangle\longrightarrow\frac{\lambda(2^{1/L}-1)\kappa_*}{c_1}. L γ ∗ a n = ⟨ θ ˉ , ∥ θ t n ∥ 2 1 ∫ J n i ∈ I ∗ ∑ b i ( t ) h i ∗ ( t ) d t ⟩ ⟶ c 1 λ ( 2 1/ L − 1 ) κ ∗ .
Thus
a n ⟶ λ ( 2 1 / L − 1 ) κ ∗ c 1 L γ ∗ > 0. a_n\longrightarrow\frac{\lambda(2^{1/L}-1)\kappa_*}{c_1L\gamma_*}>0. a n ⟶ c 1 L γ ∗ λ ( 2 1/ L − 1 ) κ ∗ > 0.
For all sufficiently large n n n , the normalized integral
1 ∥ θ t n ∥ 2 a n ∫ J n ∑ i ∈ I ∗ b i ( t ) h i ∗ ( t ) d t \frac{1}{\lVert\theta_{t_n}\rVert_2a_n}\int_{J_n}\sum_{i\in I_*}b_i(t)\mathbf{h}_i^*(t)\,dt ∥ θ t n ∥ 2 a n 1 ∫ J n i ∈ I ∗ ∑ b i ( t ) h i ∗ ( t ) d t
is a weighted average of elements of K ∗ K_* K ∗ , with nonnegative weights of total mass one, and hence belongs to K ∗ K_* K ∗ . Since K ∗ K_* K ∗ is closed, taking the limit yields
L γ ∗ κ ∗ D − 1 θ ˉ ∈ K ∗ . \frac{L\gamma_*}{\kappa_*}D^{-1}\bar{\theta}\in K_*. κ ∗ L γ ∗ D − 1 θ ˉ ∈ K ∗ .
This proves (158) .
The inclusion proved in Lemma D.26 provides, up to bookkeeping, the KKT conditions.
Proof of Theorem D.23 . Let θ ˉ \bar{\theta} θ ˉ be a limit point of θ t / ∥ θ t ∥ 2 \theta_t/\lVert\theta_t\rVert_2 θ t / ∥ θ t ∥ 2 . By Lemma D.26,
L γ ∗ κ ∗ D − 1 θ ˉ ∈ conv ⋃ i ∈ I ∗ ∂ q i ( θ ˉ ) , γ ∗ : = q min ( θ ˉ ) > 0 , κ ∗ : = ∥ θ ˉ ∥ D − 1 2 > 0. \frac{L\gamma_*}{\kappa_*}D^{-1}\bar{\theta}\in\operatorname{conv}\bigcup_{i\in I_*}\partial q_i(\bar{\theta}),\qquad\gamma_*:=q_{\min}(\bar{\theta})>0,\qquad\kappa_*:=\lVert\bar{\theta}\rVert_{D^{-1}}^2>0. κ ∗ L γ ∗ D − 1 θ ˉ ∈ conv i ∈ I ∗ ⋃ ∂ q i ( θ ˉ ) , γ ∗ := q m i n ( θ ˉ ) > 0 , κ ∗ := ∥ θ ˉ ∥ D − 1 2 > 0.
Convexity of each active Clarke subdifferential gives
L γ ∗ κ ∗ D − 1 θ ˉ = ∑ i = 1 m λ ~ i h ~ i , h ~ i ∈ ∂ q i ( θ ˉ ) , λ ~ i ≥ 0 , ∑ i λ ~ i = 1 , \frac{L\gamma_*}{\kappa_*}D^{-1}\bar{\theta}=\sum_{i=1}^{m}\widetilde{\lambda}_i\widetilde{h}_i,\qquad\widetilde{h}_i\in\partial q_i(\bar{\theta}),\qquad\widetilde{\lambda}_i\ge0,\qquad\sum_i\widetilde{\lambda}_i=1, κ ∗ L γ ∗ D − 1 θ ˉ = i = 1 ∑ m λ i h i , h i ∈ ∂ q i ( θ ˉ ) , λ i ≥ 0 , i ∑ λ i = 1 ,
with λ ~ i = 0 \widetilde{\lambda}_i=0 λ i = 0 for i ∉ I ∗ i\notin I_* i ∈ / I ∗ . Set
θ ^ : = γ ∗ − 1 / L θ ˉ , h ^ i : = γ ∗ − ( L − 1 ) / L h ~ i ∈ ∂ q i ( θ ^ ) , λ i : = κ ∗ λ ~ i L γ ∗ 2 / L . \hat{\theta}:=\gamma_*^{-1/L}\bar{\theta},\qquad\hat{h}_i:=\gamma_*^{-(L-1)/L}\widetilde{h}_i\in\partial q_i(\hat{\theta}),\qquad\lambda_i:=\frac{\kappa_*\widetilde{\lambda}_i}{L\gamma_*^{2/L}}. θ ^ := γ ∗ − 1/ L θ ˉ , h ^ i := γ ∗ − ( L − 1 ) / L h i ∈ ∂ q i ( θ ^ ) , λ i := L γ ∗ 2/ L κ ∗ λ i .
The preceding representation and homogeneity give
D − 1 θ ^ = ∑ i = 1 m λ i h ^ i , q i ( θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q i ( θ ^ ) − 1 ) = 0. D^{-1}\hat{\theta}=\sum_{i=1}^{m}\lambda_i\hat{h}_i,\qquad q_i(\hat{\theta})\ge1,\qquad\lambda_i\ge0,\qquad\lambda_i\bigl(q_i(\hat{\theta})-1\bigr)=0. D − 1 θ ^ = i = 1 ∑ m λ i h ^ i , q i ( θ ^ ) ≥ 1 , λ i ≥ 0 , λ i ( q i ( θ ^ ) − 1 ) = 0.
These are the Clarke–KKT conditions for (141) .
Since γ ∗ = q min ( θ ˉ ) \gamma_*=q_{\min}(\bar{\theta}) γ ∗ = q m i n ( θ ˉ ) , we have θ ^ = θ ˉ / q min ( θ ˉ ) 1 / L \hat{\theta}=\bar{\theta}/q_{\min}(\bar{\theta})^{1/L} θ ^ = θ ˉ / q m i n ( θ ˉ ) 1/ L . So θ ˉ / q min ( θ ˉ ) 1 / L \bar{\theta}/q_{\min}(\bar{\theta})^{1/L} θ ˉ / q m i n ( θ ˉ ) 1/ L is a KKT point as required.
Adam - Intermediate Regime
The main theorem of this section in fact also holds for the slow regime under the same assumptions; compared to Gronich and Vardi [19 ] , it assumes gradient directional convergence with no zero coordinate (which implies parameter directional convergence), but does not assume convergence of subgradients (Assumption (T3) in Gronich and Vardi [19 ] ).
The key distinction between this regime and the fast regime is the non-freezing of the second moments, which implies also non-freezing of the first moments (Proposition C.11).
Theorem E.1 (Intermediate-and-slow regime ℓ ∞ \ell_\infty ℓ ∞ implicit bias). Let L ≥ 1 L\ge1 L ≥ 1 and − 1 < α ≤ 1 L − 1 2 -1<\alpha\le\frac{1}{L}-\frac{1}{2} − 1 < α ≤ L 1 − 2 1 . Let f f f satisfy (M1), (M2), and let ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e . Assume that θ t : [ 0 , ∞ ) → R p \theta_t:[0,\infty)\to\mathbb{R}^p θ t : [ 0 , ∞ ) → R p is an Adam trajectory satisfying assumptions (T1), (T2), and the assumption
g t ∥ g t ∥ 1 ⟶ g ˉ , ∥ g ˉ ∥ 1 = 1 , g ˉ j ≠ 0 for every j ∈ [ p ] . (161) \frac{g_t}{\lVert g_t\rVert_1}\longrightarrow\bar{g},\qquad\lVert\bar{g}\rVert_1=1,\qquad\bar{g}_j\ne0\quad\text{for every }j\in[p]. \tag*{(161)} ∥ g t ∥ 1 g t ⟶ g ˉ , ∥ g ˉ ∥ 1 = 1 , g ˉ j = 0 for every j ∈ [ p ] . (161) Then
θ t ∥ θ t ∥ ∞ ⟶ θ ˉ : = − sign ( g ˉ ) ∈ { − 1 , 1 } p ; (162) \frac{\theta_t}{\lVert\theta_t\rVert_\infty}\longrightarrow\bar{\theta}:=-\operatorname{sign}(\bar{g})\in\{-1,1\}^p; \tag*{(162)} ∥ θ t ∥ ∞ θ t ⟶ θ ˉ := − sign ( g ˉ ) ∈ { − 1 , 1 } p ; (162) And θ ^ : = θ ˉ q min ( θ ˉ ) 1 / L \hat{\theta}:=\frac{\bar{\theta}}{q_{\min}(\bar{\theta})^{1/L}} θ ^ := q m i n ( θ ˉ ) 1/ L θ ˉ is a KKT point of the ℓ ∞ \ell_\infty ℓ ∞ max-margin problem.
Thus, by Definition 3.1 , R ( θ ) = 1 2 ∥ θ ∥ ∞ 2 R(\theta)=\frac{1}{2}\lVert\theta\rVert_\infty^2 R ( θ ) = 2 1 ∥ θ ∥ ∞ 2 is an implicit regularizer of Adam when − 1 < α ≤ 1 L − 1 2 -1<\alpha\le\frac{1}{L}-\frac{1}{2} − 1 < α ≤ L 1 − 2 1 for any model f f f satisfying (M1), (M2) and with any ℓ ∈ L e \ell\in\mathcal{L}_{\mathrm{e}} ℓ ∈ L e , under trajectory assumption (160). Moreover, under (T1), (T2) and for any j ∈ [ p ] j\in[p] j ∈ [ p ] with ∣ g t [ j ] ∣ ∥ g t ∥ 1 ≥ Ω ( 1 ) \frac{|g_t[j]|}{\lVert g_t\rVert_1}\ge\Omega(1) ∥ g t ∥ 1 ∣ g t [ j ] ∣ ≥ Ω ( 1 ) , it holds that
∣ m ^ t [ j ] ∣ v t [ j ] = O ( t − ζ ) . (163) \frac{|\hat{m}_t[j]|}{\sqrt{v_t[j]}}=\mathcal{O}\left(t^{-\zeta}\right). \tag*{(163)} v t [ j ] ∣ m ^ t [ j ] ∣ = O ( t − ζ ) . (163)
Proof of Theorem E.1 . Denote for j ∈ [ p ] j \in[p] j ∈ [ p ] ,
lim t → ∞ g t [ j ] ∥ g t ∥ 1 = g ˉ [ j ] ≠ 0. \lim_{t\to\infty} \frac{g_t[j]}{\lVert g_t\rVert_1} = \bar{g}[j] \ne0. t → ∞ lim ∥ g t ∥ 1 g t [ j ] = g ˉ [ j ] = 0.
Noting that m ^ t [ j ] = EMA ^ c 1 [ g t [ j ] ] = EMA ^ c 1 [ g t [ j ] ∥ g t ∥ 1 ∥ g t ∥ 1 ] \hat{m}_t[j] = \widehat{\operatorname{EMA}}_{c_1}[g_t[j]] = \widehat{\operatorname{EMA}}_{c_1}\left[\frac{g_t[j]}{\lVert g_t\rVert_1}\lVert g_t\rVert_1\right] m ^ t [ j ] = EMA c 1 [ g t [ j ]] = EMA c 1 [ ∥ g t ∥ 1 g t [ j ] ∥ g t ∥ 1 ] (and similarly for v ^ t [ j ] \hat{v}_t[j] v ^ t [ j ] ), Proposition C.11 and Lemma B.5 imply
m ^ t [ j ] EMA ^ c 1 [ ∥ g t ∥ 1 ] → t → ∞ g ˉ [ j ] , v ^ t [ j ] EMA ^ c 2 [ ∥ g t ∥ 1 2 ] → t → ∞ g ˉ [ j ] 2 . \frac{\hat{m}_t[j]}{\widehat{\operatorname{EMA}}_{c_1}[\lVert g_t\rVert_1]} \xrightarrow{t\to\infty} \bar{g}[j], \qquad\frac{\hat{v}_t[j]}{\widehat{\operatorname{EMA}}_{c_2}[\lVert g_t\rVert_1^2]} \xrightarrow{t\to\infty} \bar{g}[j]^2. EMA c 1 [∥ g t ∥ 1 ] m ^ t [ j ] t → ∞ g ˉ [ j ] , EMA c 2 [∥ g t ∥ 1 2 ] v ^ t [ j ] t → ∞ g ˉ [ j ] 2 .
Thus,
m ^ t [ j ] v ^ t [ j ] = A c 1 , c 2 [ ∥ g t ∥ 1 ] sign ( g ˉ [ j ] ) ( 1 + o ( 1 ) ) . (164) \frac{\hat{m}_t[j]}{\sqrt{\hat{v}_t[j]}} = A_{c_1,c_2}[\lVert g_t\rVert_1]\operatorname{sign}(\bar{g}[j])(1+o(1)). \tag*{(164)} v ^ t [ j ] m ^ t [ j ] = A c 1 , c 2 [∥ g t ∥ 1 ] sign ( g ˉ [ j ]) ( 1 + o ( 1 )) . (164)
Denote ν ( t ) : = η ( t ) A c 1 , c 2 [ ∥ g t ∥ 1 ] \nu(t) := \eta(t)A_{c_1,c_2}[\lVert g_t\rVert_1] ν ( t ) := η ( t ) A c 1 , c 2 [∥ g t ∥ 1 ] . We first prove
∫ 0 ∞ ν ( s ) d s = ∞ . \int_0^\infty\nu(s)\,ds = \infty. ∫ 0 ∞ ν ( s ) d s = ∞.
Note that ∥ d θ t d t ∥ ∞ ≤ 2 ν ( t ) \left\lVert\frac{d\theta_t}{dt}\right\rVert_\infty\le2\nu(t) d t d θ t ∞ ≤ 2 ν ( t ) for large enough t t t , so if towards a contradiction ∫ 0 ∞ ν < ∞ \int_0^\infty\nu< \infty ∫ 0 ∞ ν < ∞ then ∫ 0 ∞ ∥ d θ t d t ∥ d t < ∞ \int_0^\infty\left\lVert\frac{d\theta_t}{dt}\right\rVert\,dt < \infty ∫ 0 ∞ d t d θ t d t < ∞ , implying the trajectory converges to a finite point, contradicting L → 0 \mathcal{L}\to0 L → 0 .
Taking T T T so that for any j ∈ [ p ] j\in[p] j ∈ [ p ] , t ≥ T t\ge T t ≥ T Equation (164) holds up to a residual e j ( t ) e_j(t) e j ( t ) with sup j , t ≥ T ∣ e j ( t ) ∣ < 1 2 \sup_{j,t\ge T}|e_j(t)|<\frac{1}{2} sup j , t ≥ T ∣ e j ( t ) ∣ < 2 1 , e j ( t ) → t → ∞ 0 e_j(t)\xrightarrow{t\to\infty}0 e j ( t ) t → ∞ 0 , we have
θ t [ j ] = θ T [ j ] − sign ( g ˉ [ j ] ) ∫ T t ν ( s ) ( 1 + e j ( s ) ) d s . \theta_t[j] = \theta_T[j] - \operatorname{sign}(\bar{g}[j])\int_T^t \nu(s)(1+e_j(s))\,ds. θ t [ j ] = θ T [ j ] − sign ( g ˉ [ j ]) ∫ T t ν ( s ) ( 1 + e j ( s )) d s .
By Lemma B.4 we get
θ t [ j ] ∫ T t ν ( s ) d s → t → ∞ − sign ( g ˉ [ j ] ) . \frac{\theta_t[j]}{\int_T^t \nu(s)\,ds} \xrightarrow{t\to\infty} -\operatorname{sign}(\bar{g}[j]). ∫ T t ν ( s ) d s θ t [ j ] t → ∞ − sign ( g ˉ [ j ]) .
Since ∫ 0 ∞ ν ( s ) d s = ∞ \int_0^\infty\nu(s)\,ds=\infty ∫ 0 ∞ ν ( s ) d s = ∞ , the contribution on [ 0 , T ] [0,T] [ 0 , T ] is negligible, proving
∥ θ t ∥ ∞ ∫ 0 t ν ( s ) d s → t → ∞ 1 , θ t ∥ θ t ∥ ∞ → t → ∞ − sign ( g ˉ ) . \frac{\lVert\theta_t\rVert_\infty}{\int_0^t \nu(s)\,ds}\xrightarrow{t\to\infty}1,\qquad\frac{\theta_t}{\lVert\theta_t\rVert_\infty}\xrightarrow{t\to\infty}-\operatorname{sign}(\bar{g}). ∫ 0 t ν ( s ) d s ∥ θ t ∥ ∞ t → ∞ 1 , ∥ θ t ∥ ∞ θ t t → ∞ − sign ( g ˉ ) .
Consequently,
⟨ θ t ∥ θ t ∥ ∞ , − g t ∥ g t ∥ 1 ⟩ → t → ∞ ⟨ sign ( g ˉ ) , g ˉ ⟩ = ∥ g ˉ ∥ 1 = 1. \left\langle\frac{\theta_t}{\lVert\theta_t\rVert_\infty},-\frac{g_t}{\lVert g_t\rVert_1}\right\rangle\xrightarrow{t\to\infty}\langle\operatorname{sign}(\bar{g}),\bar{g}\rangle=\lVert\bar{g}\rVert_1=1. ⟨ ∥ θ t ∥ ∞ θ t , − ∥ g t ∥ 1 g t ⟩ t → ∞ ⟨ sign ( g ˉ ) , g ˉ ⟩ = ∥ g ˉ ∥ 1 = 1.
Assumption (T1) gives q min ( θ ˉ ) > 0 q_{\min}(\bar{\theta})>0 q m i n ( θ ˉ ) > 0 . Together with Assumption (T2) and the preceding alignment, this proves the KKT claim by the argument of Theorem C.8 in Gronich and Vardi [19 ] , with ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert=\lVert\cdot\rVert_\infty ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ . Although that theorem is stated under Whitney C 1 C^1 C 1 -stratifiability, its proof applies under our assumptions: the loss-subgradient decomposition, homogeneous subgradient identities, local boundedness, and closed-graph property of the Clarke subdifferentials used there all hold here.
Equation (163) is exactly the result of Corollary C.12 when ζ ≥ 0 \zeta\ge0 ζ ≥ 0 ; when ζ < 0 \zeta<0 ζ < 0 , Equation (163) follows from the universal Adam bound B.2.
Experimental Details
Experimental Validation of Width-1 Construction
The parameter directions appearing in Figure 1 b are obtained by training the width-1 model (11) with L = 2 L=2 L = 2 on the dataset (19), with B = 2 B=2 B = 2 , Λ = 1 \Lambda=1 Λ = 1 , and M = 35 M=35 M = 35 . We use initialization (83) with ρ = 1.01 > ρ 0 ( 35 , 2 ) = 1 \rho=1.01>\rho_0(35,2)=1 ρ = 1.01 > ρ 0 ( 35 , 2 ) = 1 . We use 142 pairs ( z , π 1 ) (z,\pi_1) ( z , π 1 ) with z ∈ [ 0 , 9.528876 ] ⊂ [ 0 , 3 log 35 ] z\in[0,9.528876]\subset[0,3\log35] z ∈ [ 0 , 9.528876 ] ⊂ [ 0 , 3 log 35 ] and π 1 ∈ [ 0.091242 , 14.412188 ] \pi_1\in[0.091242,14.412188] π 1 ∈ [ 0.091242 , 14.412188 ] , and π 0 = 1 / π 1 \pi_0=1/\pi_1 π 0 = 1/ π 1 . These were selected using preliminary experiments to obtain broad, well-spaced coverage of the attainable parameter directions. Exchanging the two input coordinates and u 1 ( 0 ) , u 2 ( 0 ) u_1(0), u_2(0) u 1 ( 0 ) , u 2 ( 0 ) gives the corresponding trajectories on the symmetric dataset, yielding 284 directions in total.
We minimize the summed exponential loss L ( θ ) = ∑ i = 1 2 exp ( − y i f ( x i ; θ ) ) \mathcal{L}(\theta)=\sum_{i=1}^{2}\exp(-y_i f(x_i;\theta)) L ( θ ) = ∑ i = 1 2 exp ( − y i f ( x i ; θ )) using full-batch, bias-corrected Adam with β 1 = 0.9 \beta_1=0.9 β 1 = 0.9 , β 2 = 0.999 \beta_2=0.999 β 2 = 0.999 , ε = 0 \varepsilon=0 ε = 0 , zero initial moments, and learning rate η k = 10 − 3 k 0.15 \eta_k=10^{-3}k^{0.15} η k = 1 0 − 3 k 0.15 for update k ≥ 1 k\geq1 k ≥ 1 , that is α = 0.15 > 1 L − 1 2 = 0 \alpha=0.15>\frac{1}{L}-\frac{1}{2}=0 α = 0.15 > L 1 − 2 1 = 0 . All trajectories run for 600,000 updates in float64 arithmetic. Trajectory length was chosen to ensure stability of the parameter direction: each trajectory’s parameter direction is stable up to an error of 0.012 throughout the entire 100,000 last steps, that is:
max 500 , 000 ≤ s , t ≤ 600 , 000 ∥ θ s ∥ θ s ∥ 2 − θ t ∥ θ t ∥ 2 ∥ 2 < 0.012. \max_{500,000\leq s,t\leq600,000}\left\|\frac{\theta_s}{\|\theta_s\|_2}-\frac{\theta_t}{\|\theta_t\|_2}\right\|_2<0.012. 500 , 000 ≤ s , t ≤ 600 , 000 max ∥ θ s ∥ 2 θ s − ∥ θ t ∥ 2 θ t 2 < 0.012.
Margin Comparisons on MNIST
We conduct two experiments on binary classification of MNIST: one long-horizon experiment with ε = 0 \varepsilon=0 ε = 0 only, using a bias-free network and small initialization scale, and a shorter experiment comparing ε = 0 \varepsilon=0 ε = 0 and ε = 10 − 8 \varepsilon=10^{-8} ε = 1 0 − 8 on a more practical setting: a network with biases and standard initialization.
For both experiments we use 1,024 images from the MNIST training set: the first 512 even-digit and first 512 odd-digit examples in the original dataset order, labeled − 1 -1 − 1 and + 1 +1 + 1 , respectively. Pixels are divided by 255, centered by the per-pixel mean of these selected examples, and flattened. The network is a multilayer perceptron with widths 784 → 512 → 512 → 1 784\to512\to512\to1 784 → 512 → 512 → 1 , with ReLU activations after the two hidden layers. For the ε \varepsilon ε comparison experiment only, we incorporate biases in every layer for ease of training (even though such networks are not precisely homogeneous, they are nearly homogeneous, see Cai et al. [10 ] ). We minimize the full-batch mean logistic loss, L ( θ ) = 1024 − 1 ∑ i log ( 1 + exp ( − y i f ( x i ; θ ) ) ) \mathcal{L}(\theta)=1024^{-1}\sum_i\log(1+\exp(-y_i f(x_i;\theta))) L ( θ ) = 102 4 − 1 ∑ i log ( 1 + exp ( − y i f ( x i ; θ ))) , using float64 arithmetic. Each layer’s weights and biases are initialized independently from U ( − 1 / n i n , 1 / n i n ) \mathcal{U}(-1/\sqrt{n_{\mathrm{in}}},1/\sqrt{n_{\mathrm{in}}}) U ( − 1/ n in , 1/ n in ) and then multiplied by a common factor of 10 − 4 10^{-4} 1 0 − 4 in the long-horizon ε = 0 \varepsilon=0 ε = 0 experiment (a small initialization is known to accelerate arrival to the asymptotic regime ([50 ] (Moroshko et al. 2020)). In both experiments, when running Adam with ε = 0 \varepsilon=0 ε = 0 , we assign zero updates to coordinates whose gradients have remained identically zero.
In all runs we use β 1 = 0.9 \beta_1=0.9 β 1 = 0.9 and β 2 = 0.999 \beta_2=0.999 β 2 = 0.999 (PyTorch ([41 ] defaults). At update k ≥ 1 k\geq1 k ≥ 1 , the learning rate is η k = 3 × 10 − 4 k α \eta_k=3\times10^{-4}k^\alpha η k = 3 × 1 0 − 4 k α , where α ∈ { 0 , − 0.1 , − 0.3 , − 0.5 , − 0.7 } \alpha\in\{0,-0.1,-0.3,-0.5,-0.7\} α ∈ { 0 , − 0.1 , − 0.3 , − 0.5 , − 0.7 } . We repeat each schedule with five initialization seeds, 2026100901, …, 2026100905, using identical initial weights across schedules for each seed. For the long-horizon ε = 0 \varepsilon=0 ε = 0 experiment, every run terminates at the first iterate satisfying L ≤ 10 − 20 \mathcal{L}\leq10^{-20} L ≤ 1 0 − 20 , and for the comparison experiment, every run terminates at the first iterate satisfying L ≤ 10 − 10 \mathcal{L}\leq10^{-10} L ≤ 1 0 − 10 or at 1 million steps (the α = − 0.7 , − 0.5 \alpha=-0.7,-0.5 α = − 0.7 , − 0.5 trajectories finish at a slightly larger loss than 10 − 10 10^{-10} 1 0 − 10 ). Experiments use Python 3.12.14 and PyTorch 2.13.0+cu130 on NVIDIA GPUs.
Below, we present supplementary figures to the main text.
Long-horizon, small-initialization Adam with no stability constant (ε = 0 \varepsilon=0 ε = 0 ).
In Figure 4 we focus on the ℓ ∞ \ell_\infty ℓ ∞ margin for α = − 0.3 \alpha=-0.3 α = − 0.3 by seed. This schedule is contained in the intermediate regime and hence we expect ℓ ∞ \ell_\infty ℓ ∞ margin maximization, but this bias could be expressed late in the trajectory. In the experiment, some seeds show monotone increasing margin, others have a slight decrease and recovery of the margin, and yet others show no recovery during the trajectory after achieving a peak value.
Figure 4. ℓ ∞ \ell_\infty ℓ ∞ margins for α = − 0.3 \alpha=-0.3 α = − 0.3 , for five seeds.
In Figure 6 we show the weighted-ℓ 2 \ell_2 ℓ 2 margin of each seed and schedule, evaluated on all seeds. The fast regime α = 0 \alpha=0 α = 0 shows the strongest trajectory-dependent weighted-ℓ 2 \ell_2 ℓ 2 margin increase, with this property becoming less pronounced as the schedule becomes slower.
Figure 6. Endpoint-weighted margins for ε = 0 \varepsilon= 0 ε = 0 in the long-horizon experiment. Rows correspond to learning-rate schedules η t ∝ t α \eta_t \propto t^{\alpha} η t ∝ t α ; columns identify the reference seed whose endpoint metric is held fixed. Each panel shows all five trajectories under that schedule.
Comparison of ε = 0 , 10 − 8 \varepsilon=0,10^{-8} ε = 0 , 1 0 − 8 in a practical setting.
For ε = 10 − 8 \varepsilon=10^{-8} ε = 1 0 − 8 , to identify the onset of the effect of ε \varepsilon ε , we measure the relative error of the update with and without ε \varepsilon ε ,
e ε ( t ) = ∥ m ^ t v ^ t − m ^ t v ^ t + ε ∥ 2 ∥ m ^ t v ^ t ∥ 2 , e_{\varepsilon}(t)=\frac{\left\|\frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}}-\frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\varepsilon}\right\|_2}{\left\|\frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}}\right\|_2}, e ε ( t ) = v ^ t m ^ t 2 v ^ t m ^ t − v ^ t + ε m ^ t 2 ,
which tends to 1 when t → ∞ t\to\infty t → ∞ , since for every j ∈ [ p ] j\in[p] j ∈ [ p ] for which v t [ j ] > 0 v_t[j]>0 v t [ j ] > 0 eventually, it holds that $v_t[j]\xrightarrow{t\to\infty}0, and so
m ^ t [ j ] v ^ t [ j ] + ϵ ∼ ϵ − 1 m ^ t [ j ] = o ( m ^ t [ j ] v ^ t [ j ] ) . \frac{\hat{m}_t[j]}{\sqrt{\hat{v}_t[j]}+\epsilon} \sim\epsilon^{-1}\hat{m}_t[j] = o\left(\frac{\hat{m}_t[j]}{\sqrt{\hat{v}_t[j]}}\right). v ^ t [ j ] + ϵ m ^ t [ j ] ∼ ϵ − 1 m ^ t [ j ] = o ( v ^ t [ j ] m ^ t [ j ] ) .
To avoid momentary effects, we observe in Figure 5 a moving average
Figure 5. Relative update error for Adam with ϵ = 10 − 8 \epsilon=10^{-8} ϵ = 1 0 − 8 . The quantity e ϵ ‾ ( t ) \overline{e_\epsilon}(t) e ϵ ( t ) denotes the trailing 1,000-update moving average of e ϵ ( t ) e_\epsilon(t) e ϵ ( t ) . Curves show means across five seeds; shading denotes one standard error.
e ϵ ‾ ( t ) = 1 1000 ∑ i = 0 999 e ϵ ( t − i ) , t ≥ 1,000 , \overline{e_\epsilon}(t)=\frac{1}{1000}\sum_{i=0}^{999}e_\epsilon(t-i), \qquad t\geq1{,}000, e ϵ ( t ) = 1000 1 i = 0 ∑ 999 e ϵ ( t − i ) , t ≥ 1 , 000 ,
for all five schedules in the ϵ = 10 − 8 \epsilon=10^{-8} ϵ = 1 0 − 8 experiment. Note that some curves start at an advanced loss value at update 1,000. As expected, the curves begin near 0 and tend to 1 towards the end of training.
In Figure 7 , Figure 8 we show the weighted-ℓ 2 \ell_2 ℓ 2 margin of each seed and schedule, evaluated on all seeds, for ϵ = 10 − 8 \epsilon=10^{-8} ϵ = 1 0 − 8 and ϵ = 0 \epsilon=0 ϵ = 0 respectively. The fast regime α = 0 \alpha=0 α = 0 shows the strongest trajectory-dependent weighted-ℓ 2 \ell_2 ℓ 2 margin increase, with this property becoming less pronounced as the schedule becomes slower.
Figure 7. Endpoint-weighted margins for ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 in the comparison experiment. Rows correspond to learning-rate schedules η t ∝ t α \eta_t \propto t^\alpha η t ∝ t α ; columns identify the reference seed whose endpoint metric is held fixed. Each panel shows all five trajectories under that schedule.
Figure 8. Endpoint-weighted margins for ε = 0 \varepsilon= 0 ε = 0 in the comparison experiment. Rows correspond to learning-rate schedules η t ∝ t α \eta_t \propto t^\alpha η t ∝ t α ; columns identify the reference seed whose endpoint metric is held fixed. Each panel shows all five trajectories under that schedule.