Introduction
Ever since the introduction of Transformers Vaswani et al., [37 ] , they have been widely used for autoregressive sequence generation, and have come to dominate many areas of machine learning [3 , 8 ] (Ji et al. 2025). Recently, State Space Models (SSMs) have emerged as a promising alternative. By offering improved computational efficiency while maintaining competitive performance, SSMs are increasingly being adopted across a range of tasks [27 , 6 ] (Gu et al. 2021; Gu & Dao 2023).
Autoregressive models reuse the same parameters across repeated computation steps. When examples have different sequence lengths or require different numbers of rollout steps, the final prediction for each example may be a different function of the same shared parameters. Even if the per-step computation is linear, the resulting margin is generally a non-homogeneous polynomial in the parameters, with a degree that depends on the number of autoregressive applications.
This creates a setting that is not captured by the existing max-margin implicit-bias theory. Classical results show that gradient descent on separable linear models with an exponential-type loss converges in direction to a hard-margin classifier Soudry et al., [30 ] , and later work extends this picture to homogeneous and some non-homogeneous parameterizations [13 , 23 , 20 , 2 ] (Ji & Telgarsky 2020b). In those settings, however, the training examples are typically evaluated by a single shared predictor, or by predictors with a common homogeneity structure. In contrast, variable-step autoregressive models lead to multiple polynomial predictors, one per example or per length class, all sharing parameters but potentially growing at different asymptotic rates during training.
We study this phenomenon through the lens of shared-parameter, linear-in-the-input polynomial predictors, in a classification setting. Our main observation is that the relevant degree of a predictor is not necessarily its algebraic degree. Along a gradient-descent trajectory, leading homogeneous components may vanish in the limiting direction, causing a high-degree margin polynomial to grow like a lower-degree polynomial. We call this trajectory-dependent quantity the effective degree. Since exponential-loss gradients weight examples by their current margins, examples with larger effective degree can become asymptotically suppressed, while examples with smaller effective degree can determine the limiting classifier.
Our main result gives a conditional max-margin characterization for this setting. Assuming that the loss converges to zero, the parameter direction converges, and each margin has a well-defined integer effective degree along the trajectory, we prove that the limiting direction of gradient descent satisfies the KKT conditions of an effective-degree max-margin problem. The unit-margin constraints are imposed only by examples with minimum effective degree, and higher-effective-degree examples enter only through feasibility constraints. Thus the limiting classifier can be governed by the slowest-growing margins rather than by all examples equally.
We instantiate the theorem in scalar linear autoregressive models, linear state-space models (SSMs), and linear autoregressive SSMs. These models are not intended to capture all features of modern sequence architectures; rather, they isolate the effect of parameter reuse across variable numbers of steps. Indeed, the implicit bias of linear models and linear neural networks has been extensively studied as a testbed for understanding the implicit bias in more complex architectures [34 ] , and we adopt this approach here by considering linear autoregressive models and linear SSMs. In our simplified settings, when additional recurrent or autoregressive steps increase effective degree, our theorem predicts a form of length-sensitive implicit bias: shorter or lower-effective-degree examples can dominate the asymptotic margin solution.
Our contribution:
Under standard asymptotic convergence assumptions and an effective-degree regularity assumption, we prove a max-margin/KKT characterization of the gradient-descent limit for exponential loss. This characterization is significantly different from the standard max-margin characterization of homogeneous models.
Related Work
Implicit bias in homogeneous and near-homogeneous models. There is a large body of work studying the implicit bias of gradient-based optimization. Soudry et al. [30 ] initiated this line of work by showing that gradient descent on separable data with the logistic loss converges in direction to the max-margin classifier. Subsequent works extended this perspective to broader classes of models, including linear convolutional networks and homogeneous polynomials [13 a] . For homogeneous models, Lyu & Li [23 ] showed that any limit point of the optimization path is in the direction of a KKT point of the corresponding margin-constrained Euclidean-norm minimization problem, and Ji & Telgarsky (2020b) established directional convergence of parameters and gradients under additional structural assumptions. Ji & Telgarsky (2020a) used the induced dynamics on dual variables to characterize convergence rates to the implicit bias in linear models; we use a related observation to prove directional convergence of the gradient along a subsequence.
Several works go beyond standard homogeneity. Nacson et al. [25 ] studied regularization paths for sums of homogeneous models with different degrees, but their setting uses separate parameters for each homogeneous component and does not analyze gradient descent dynamics. Kunin et al. [20 ] generalized this framework to quasi-homogeneous models , where different parameters have different homogeneity degrees, and showed that the resulting implicit bias can be asymmetric. Our polynomial models, as well as their multidimensional extensions, do not fall under this quasi-homogeneous framework; see Section A.2. Lippl et al. [22 ] studied gradient descent on parameter-sharing homogeneous polynomials, allowing the polynomial to vary across examples.
Most closely related to our treatment of non-homogeneous structure is Cai et al. [2 ] , who established implicit bias results for nearly-homogeneous models . Although polynomials satisfy their definition of near-homogeneity, their analysis considers a single predictor shared across all data points, whereas we study a more general setting in which each data point may induce a different predictor. Moreover, their result relies on a strong assumption on the optimization dynamics, which we relax for polynomials; see Section 4.2 for a detailed comparison.
Implicit bias in attention and sequence models. Recent works have begun to study implicit bias in sequence models. For self-attention, Tarzanagh et al. (2023); Ataee Tarzanagh et al. (2023); Vasudeva et al. [36 ] analyze the implicit bias and convergence behavior of attention-based architectures. These works are complementary to ours: they focus on attention mechanisms, whereas our analysis concerns polynomial and state-space/autoregressive structures.
State-space models and linear autoregressive models. Several recent theoretical works study state-space models and related linear autoregressive models in settings close to ours. Merrill et al. [24 ] analyze the expressivity of state-space models. Slutzky et al. [28 ] study the effect of data poisoning on the implicit bias of diagonal SSMs with the square loss for sequences of fixed length. Other works analyze SSMs with structured transition matrices or related architectures (Gu et al. (2021); Dao & Gu [6 ] ), while Smékal et al. [29 ] study the learning dynamics of SSMs under square loss. For autoregressive models, Joshi et al. [17 , 18 ] establish computational learning results for autoregressive linear threshold models in a setting similar to ours, and Thrampoulidis [32 ] study implicit optimization bias in linear autoregressive next-token prediction with fixed-length sequences and cross-entropy loss. Finally, Emami et al. [7 ] analyze linear RNNs at the NTK limit, showing that the effect of input tokens decays geometrically; the model they consider is reminiscent of the SSMs studied in § 3.3 .
Preliminaries and Problem Definition
Notations. We denote vectors v ∈ R d \mathbf{v} \in\mathbb{R}^{d} v ∈ R d by boldface letters i.e. v = ( v 1 , … , v d ) \mathbf{v} = (v_{1}, \ldots, v_{d}) v = ( v 1 , … , v d ) , and v ˉ : = v / ∥ v ∥ \bar{\mathbf{v}} := \mathbf{v}/\lVert\mathbf{v}\rVert v ˉ := v / ∥ v ∥ .
We denote the sign function sgn ( x ) : = { 1 x ≥ 0 − 1 else \operatorname{sgn}(x) := \begin{cases} 1 & x \ge0 \\ -1 & \text{else} \end{cases} sgn ( x ) := { 1 − 1 x ≥ 0 else . We denote by ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ the Euclidean norm, and [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } . For ν ∈ N \nu\in\mathbb{N} ν ∈ N , we denote by P ν \mathcal{P}^{\nu} P ν the sum of (multivariate, vector valued) monomials in P \mathcal{P} P of degree ν \nu ν , this is a ν \nu ν -homogeneous polynomial, which we call the ν \nu ν -homogenization of P \mathcal{P} P .
Problem Definition
Model and Data. We consider a dataset { X i , y i } i = 1 N \{X_i,y_i\}_{i=1}^{N} { X i , y i } i = 1 N where X i ∈ R d × ℓ i X_i \in\mathbb{R}^{d \times\ell_i} X i ∈ R d × ℓ i and y i ∈ { ± 1 } y_i \in\{\pm1\} y i ∈ { ± 1 } . We view each input sample X i X_i X i as ℓ i \ell_i ℓ i tokens, each of dimension d d d . Note that we allow for each sample to have a different number of tokens. The difficulty in determining the implicit bias of our models already appears in the case of d = 1 d=1 d = 1 . Hence, for brevity, in the main part of the paper, we consider the case of d = 1 d=1 d = 1 , namely, each token is a scalar in R \mathbb{R} R , and we denote the input sequence as x i ∈ R ℓ i x_i \in\mathbb{R}^{\ell_i} x i ∈ R ℓ i . All our results are readily extended to the case where d d d is some constant larger than 1 1 1 , and we prove these extensions formally in Section B.
Let { x i , y i } i = 1 N \{x_i,y_i\}_{i=1}^{N} { x i , y i } i = 1 N be a dataset where x i ∈ R ℓ i x_i \in\mathbb{R}^{\ell_i} x i ∈ R ℓ i , y i ∈ { ± 1 } y_i \in\{\pm1\} y i ∈ { ± 1 } . We define a set of models { M i : R D → R ℓ i } i = 1 N \{\mathcal{M}_i : \mathbb{R}^{D} \to\mathbb{R}^{\ell_i}\}_{i=1}^{N} { M i : R D → R ℓ i } i = 1 N , that share the same parameters w ∈ R D \mathbf{w} \in\mathbb{R}^{D} w ∈ R D , but might have different output dimensions. These models act linearly on the data x i x_i x i , namely their output is y ^ i : = ⟨ M i ( w ) , x i ⟩ \hat{y}_i := \langle\mathcal{M}_i(\mathbf{w}), x_i\rangle y ^ i := ⟨ M i ( w ) , x i ⟩ where ℓ i \ell_i ℓ i is the dimension of the i i i -th sample, i.e. x i ∈ R ℓ i x_i \in\mathbb{R}^{\ell_i} x i ∈ R ℓ i , or in other words, the number of one dimensional tokens x i x_i x i has. Note that although { M i : R D → R ℓ i } i = 1 N \{\mathcal{M}_i : \mathbb{R}^{D} \to\mathbb{R}^{\ell_i}\}_{i=1}^{N} { M i : R D → R ℓ i } i = 1 N indexes N N N models, they may not all be distinct, i.e., we allow M i = M j \mathcal{M}_i = \mathcal{M}_j M i = M j for some i ≠ j i \ne j i = j . This multi-model setting is a generalization of the standard implicit bias setting where all the samples share the same predictor M \mathcal{M} M . This framing aligns with the setting of SSMs and autoregressive models, where each sample may consist of a different number of input tokens or autoregressive steps. In this paper, we focus on two such models: (1) a linear autoregressive model and (2) a linear state-space model (SSM), formally defined in § 3.2 and § 3.3 . Although the models acts linearly on the input data, their parameterizations can be highly non-linear, leading to implicit biases that differ substantially from those of standard linear predictors.
Training. In this paper we will consider the exponential loss and denote L ( w ) = ∑ i = 1 n exp ( − y i M i ( w ) T x i ) \mathcal{L}(\mathbf{w}) = \sum_{i=1}^{n}\exp(-y_i\mathcal{M}_i(\mathbf{w})^{\mathsf{T}}\mathbf{x}_i) L ( w ) = ∑ i = 1 n exp ( − y i M i ( w ) T x i ) . Our results can be readily extended to the logistic loss as well. Throughout the paper, we consider gradient descent w.r.t. vectors or matrices of the parameters. We denote by w ( 0 ) \mathbf{w}^{(0)} w ( 0 ) the parameters at initialization, and
w ( t ) = w ( t − 1 ) − η t ∇ L ( w ( t − 1 ) ) , (1) \mathbf{w}^{(t)}=\mathbf{w}^{(t-1)}-\eta_t\nabla\mathcal{L}(\mathbf{w}^{(t-1)}), \tag*{(1)} w ( t ) = w ( t − 1 ) − η t ∇ L ( w ( t − 1 ) ) , (1)
as the parameter vector after the t th t^{\text{th}} t th step of gradient descent, where η t \eta_t η t denotes the learning rate at time t t t . Our results can be applied to general learning rate schedulers, and we only assume the learning rate sequence { η t } t = 1 ∞ \{\eta_t\}_{t=1}^{\infty} { η t } t = 1 ∞ does not increase. We denote by w ˉ ∞ = lim t → ∞ w ( t ) ∥ w ( t ) ∥ \bar{\mathbf{w}}^{\infty}=\lim_{t\to\infty}\frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert} w ˉ ∞ = lim t → ∞ ∥ w ( t ) ∥ w ( t ) the directional limit of w \mathbf{w} w when it exists.
Linear Autoregressive Model
The first model we consider is a linear autoregressive model. This type of models is common in many academic fields for the prediction of time series Gujarati & Porter [11 ] ; Wei [38 ] ; Kirchgässner et al. [19 ] ; Dama & Sinoquet [5 ] ; Chen et al. [4 ] . Moreover, it is a basic simplification of large language models, where we isolate the autoregressive part of transformers from the non-linear parts which are the self-attention and MLP. The core mechanism is that the model generates new tokens, and then apply itself autoregressively on its own output.
We consider an example x ∈ R l \mathbf{x}\in\mathbb{R}^{l} x ∈ R l which we interpret as an input prompt, i.e., the initial part of some sequence. Here we will consider parameters w ∈ R D \mathbf{w}\in\mathbb{R}^{D} w ∈ R D and assume for simplicity D = l D=l D = l , thus x T = x 1 : D T = ( x 1 , … , x D ) \mathbf{x}^{\mathsf{T}}=\mathbf{x}_{1:D}^{\mathsf{T}}=(x_1,\ldots,x_D) x T = x 1 : D T = ( x 1 , … , x D ) . We compute a sequence of T T T autoregressive steps, where the final token’s sign is the label. Formally, the model computes:
x D + 1 = ⟨ w , x ⟩ , x D + 2 = ⟨ w , x 2 : D + 1 ⟩ , … , x D + T = ⟨ w , x T : D + T − 1 ⟩ . (2) x_{D+1}=\langle\mathbf{w},\mathbf{x}\rangle,\quad x_{D+2}=\langle\mathbf{w},\mathbf{x}_{2:D+1}\rangle,\quad\ldots,\quad x_{D+T}=\langle\mathbf{w},\mathbf{x}_{T:D+T-1}\rangle. \tag*{(2)} x D + 1 = ⟨ w , x ⟩ , x D + 2 = ⟨ w , x 2 : D + 1 ⟩ , … , x D + T = ⟨ w , x T : D + T − 1 ⟩ . (2)
The label of the sequence is defined as sgn ( x D + T ) \operatorname{sgn}(x_{D+T}) sgn ( x D + T ) . More generally, we may allow l ≠ D l\ne D l = D : in the case l < D l<D l < D , we pad x = ( 0 , … , x 1 , … , x l ) \mathbf{x}=(0,\ldots,x_1,\ldots,x_l) x = ( 0 , … , x 1 , … , x l ) , and in the case l > D l>D l > D , apply w \mathbf{w} w only to the last D D D elements of x \mathbf{x} x . This can be thought of as the input prompt being longer than the model’s context length.
We denote the vector of coefficients of x 1 , … , x D x_1,\ldots,x_D x 1 , … , x D in x D + T x_{D+T} x D + T as P \mathcal{P} P , a polynomial in w \mathbf{w} w . Using this notation, we have that ⟨ P ( w ) , x ⟩ = x D + T \langle\mathcal{P}(\mathbf{w}),\mathbf{x}\rangle=x_{D+T} ⟨ P ( w ) , x ⟩ = x D + T . See Figure 1 for a demonstration.
Figure 1. Movie-strip view of the autoregressive construction.
We can interpret the setup in the context of language models as having l l l initial one-dimensional tokens, x 1 , … , x l x_1,\ldots,x_l x 1 , … , x l , which serve as the starting sequence. In standard transformers, and autoregressive language models in general, text is generated token by token: the model predicts the next token based on the current sequence, appends this token, and then feeds the entire updated sequence back into the model to produce the next one, T T T times. Our framework mirrors this process but uses a linear autoregressive model, which is more tractable to analyze. We note that in our work the polynomials are usually associated with an indexed example x i ∈ R l i \mathbf{x}_i\in\mathbb{R}^{l_i} x i ∈ R l i , hence we denote them P i \mathcal{P}_i P i . Note that crucially for a dataset { x i , y i } i = 1 N \{x_i,y_i\}_{i=1}^{N} { x i , y i } i = 1 N , we have a number of autoregressive steps T i T_i T i for each example, which together with l i l_i l i changes P i \mathcal{P}_i P i . That is, both elements embodied in the polynomial predictor. This also means that we may have P i = P j \mathcal{P}_i=\mathcal{P}_j P i = P j for i ≠ j i\ne j i = j , if both examples i , j i,j i , j are labeled using the same number T i T_i T i of autoregressive steps and share the same input length l i l_i l i . We note that in this model our training setup corresponds to training directly on the output after T i T_i T i steps, rather than training on the next-token-prediction.
Linear State Space Models
The second model we consider, is a Linear State Space Model (SSM). SSMs are used to model natural phenomena and have been lately adopted by the Machine Learning community as a promising candidate for Large Language Models to replace and complement Transformers (Gu et al., 2021; Gu & Dao, 2023; [6 ] ).
We consider a sequence of tokens of variable lengths X ∈ R d × l X \in\mathbb{R}^{d \times l} X ∈ R d × l where l l l is the length of the sequence. SSMs consist of parameter matrices A ∈ R d A × d A A \in\mathbb{R}^{d_A \times d_A} A ∈ R d A × d A , B ∈ R d A × d B \in\mathbb{R}^{d_A \times d} B ∈ R d A × d , C ∈ R d A × d C C \in\mathbb{R}^{d_A \times d_C} C ∈ R d A × d C where the model is given recursively by a sequence of “hidden” states h h h , i.e., for 1 ≤ i ≤ l 1 \leq i \leq l 1 ≤ i ≤ l , h i = A ⋅ h i − 1 + B ( X ) i h_i = A \cdot h_{i-1} + B(X)_i h i = A ⋅ h i − 1 + B ( X ) i , where ( X ) i (X)_i ( X ) i denotes the i t h i^{\mathrm{th}} i th column of X X X , with h 0 = 0 h_0 = 0 h 0 = 0 , and with the output being y = sgn ( C T h l ) y = \operatorname{sgn}(C^\mathsf{T}h_l) y = sgn ( C T h l ) . Note that we consider binary classification and assume d C = 1 d_C = 1 d C = 1 . For the rest of the main part of the paper, we consider the case d = 1 d = 1 d = 1 , and let B B B , C C C be vectors. We provide the generalization for a matrix C C C and full proof for the more general case of d ≥ 1 d \geq1 d ≥ 1 in Section B.
A useful convolutional way to denote it overall is
y = sgn ( C T ∑ i = 1 l A l − i B ( X ) i ) . y = \operatorname{sgn}\left(C^\mathsf{T}\sum_{i=1}^{l} A^{l-i}B(X)_i\right). y = sgn ( C T i = 1 ∑ l A l − i B ( X ) i ) .
Note that SSMs in practice usually use nonlinearities, which we omit, and sometimes use special matrix structure for A A A , such as diagonal, or diagonal up to some low rank. We analyze the general case and note that in principle our analysis should apply to those cases as well.
With a dataset { x i , y i } i = 1 N \{x_i, y_i\}_{i=1}^{N} { x i , y i } i = 1 N , denote the number of tokens of the k t h k^{\mathrm{th}} k th sample as l k l_k l k , and consider the polynomial
Q k ( A , B , C ) = ( C T A l k − 1 B ⋮ C T A B C T B ) . Q_k(A, B, C) =
\begin{pmatrix}
C^\mathsf{T}A^{l_k-1}B \\
\vdots\\
C^\mathsf{T}AB \\
C^\mathsf{T}B
\end{pmatrix}. Q k ( A , B , C ) = C T A l k − 1 B ⋮ C T A B C T B .
i.e. y k = ⟨ Q k ( A , B , C ) , x k ⟩ y_k = \langle Q_k(A, B, C), x_k\rangle y k = ⟨ Q k ( A , B , C ) , x k ⟩ , note that Q k Q_k Q k depends only on the number of steps taken by the SSM, that is the model works for a variable length input x k ∈ R l k x_k \in\mathbb{R}^{l_k} x k ∈ R l k . Similarly to the linear autoregressive case, although each sample k k k is associated with a different polynomial Q k Q_k Q k , we may have Q k = Q j Q_k = Q_j Q k = Q j for k ≠ j k \ne j k = j if both examples k , j k, j k , j have the same length, and l k l_k l k is embodied in Q k Q_k Q k .
Linear Autoregressive SSM
Note that the SSM from the previous section does not use its own outputs as inputs for the next steps. That is, the SSM simply runs over the input sequence but does not extend it. Hence it is not autoregressive in this sense. We now introduce the Autoregressive SSM . Consider an example x ∈ R l x \in\mathbb{R}^l x ∈ R l , that we can think of as an input prompt, or as a beginning of some sequence x T = x 1 : l T = ( x 1 , … , x l ) x^\mathsf{T} = x_{1:l}^\mathsf{T} = (x_1, \ldots, x_l) x T = x 1 : l T = ( x 1 , … , x l ) , we compute a sequence of T T T autoregressive steps:
x l + 1 = C T ∑ i = 1 l A l − i B x i , x l + 2 = C T ∑ i = 1 l + 1 A l + 1 − i B x i , … , x l + T = C T ∑ i = 1 l + T − 1 A l + T − 1 − i B x i . (3) x_{l+1} = C^\mathsf{T}\sum_{i=1}^{l} A^{l-i}Bx_i,\quad x_{l+2} = C^\mathsf{T}\sum_{i=1}^{l+1} A^{l+1-i}Bx_i,\quad\ldots,\quad x_{l+T} = C^\mathsf{T}\sum_{i=1}^{l+T-1} A^{l+T-1-i}Bx_i. \tag*{(3)} x l + 1 = C T i = 1 ∑ l A l − i B x i , x l + 2 = C T i = 1 ∑ l + 1 A l + 1 − i B x i , … , x l + T = C T i = 1 ∑ l + T − 1 A l + T − 1 − i B x i . (3)
The label of the sequence x x x is defined as sgn ( x l + T ) \operatorname{sgn}(x_{l+T}) sgn ( x l + T ) , and we write the polynomial coefficients as a vector Q ( A , B , C ) Q(A, B, C) Q ( A , B , C ) , i.e. y = sgn ( ⟨ Q ( A , B , C ) , x ⟩ ) y = \operatorname{sgn}(\langle Q(A, B, C), x\rangle) y = sgn (⟨ Q ( A , B , C ) , x ⟩) . Note that Q Q Q depends both on the number of autoregressive steps T T T and the number of input tokens l l l . As in previous sections, Q i Q_i Q i implicitly embeds l i l_i l i and T i T_i T i , and we allow Q i = Q j Q_i = Q_j Q i = Q j for i ≠ j i \ne j i = j .
Assumptions on the Dynamics
In this section, we present the assumptions about the optimization dynamics that we will use to prove our main results. Before introducing our assumptions, we define the “effective degree” of a polynomial over an optimization trajectory:
Definition 4.1 (Effective Degree). Let p : R D → R p : \mathbb{R}^{D} \to\mathbb{R} p : R D → R be a polynomial with deg ( p ) = k \operatorname{deg}(p) = k deg ( p ) = k . Let w ( t ) ∈ R D w^{(t)} \in\mathbb{R}^{D} w ( t ) ∈ R D be some trajectory of the polynomial’s inputs, such that ∥ w ( t ) ∥ → ∞ \lVert w^{(t)} \rVert\to\infty ∥ w ( t ) ∥ → ∞ and lim t → ∞ w ( t ) ∥ w ( t ) ∥ = w ˉ ∞ \lim_{t \to\infty} \frac{w^{(t)}}{\lVert w^{(t)} \rVert} = \bar{w}^{\infty} lim t → ∞ ∥ w ( t ) ∥ w ( t ) = w ˉ ∞ . We say p p p has an effective degree of m m m over the trajectory w ( t ) \mathbf{w}^{(t)} w ( t ) (denoted effdeg w ( t ) ( p ) \operatorname{effdeg}_{\mathbf{w}^{(t)}}(p) effdeg w ( t ) ( p ) , or simply effdeg ( p ) \operatorname{effdeg}(p) effdeg ( p ) when w ( t ) \mathbf{w}^{(t)} w ( t ) is implicit) if
lim t → ∞ p ( w ( t ) ) ∥ w ( t ) ∥ j = { C p j ( w ˉ ∞ ) ≠ 0 j = m , 0 j > m , \lim_{t\to\infty}\frac{p(\mathbf{w}^{(t)})}{\lVert\mathbf{w}(t)\rVert^j} =
\begin{cases}
C p^j(\bar{\mathbf{w}}^\infty) \ne0 & j=m,\\
0 & j>m,
\end{cases} t → ∞ lim ∥ w ( t ) ∥ j p ( w ( t ) ) = { C p j ( w ˉ ∞ ) = 0 0 j = m , j > m ,
and similarly
lim t → ∞ ∇ p ( w ( t ) ) ∥ w ( t ) ∥ j − 1 = { C ∇ p j ( w ˉ ∞ ) ≠ 0 j = m , 0 j > m . \lim_{t\to\infty}\frac{\nabla p(\mathbf{w}^{(t)})}{\lVert\mathbf{w}(t)\rVert^{j-1}} =
\begin{cases}
C\nabla p^j(\bar{\mathbf{w}}^\infty) \ne0 & j=m,\\
0 & j>m.
\end{cases} t → ∞ lim ∥ w ( t ) ∥ j − 1 ∇ p ( w ( t ) ) = { C ∇ p j ( w ˉ ∞ ) = 0 0 j = m , j > m .
For some constant C > 0 C>0 C > 0 .
Intuitively, the definition states that along the trajectory, w ( t ) \mathbf{w}^{(t)} w ( t ) moves in a direction that cancels out higher degree monomials (if such monomials exist). A simple example would be the trajectory w ( t ) = ( t , t , t ) \mathbf{w}^{(t)}=(t,t,t) w ( t ) = ( t , t , t ) with the polynomial p ( w ) = w 1 + ( w 2 − w 3 ) 2 p(\mathbf{w})=w_1+(w_2-w_3)^2 p ( w ) = w 1 + ( w 2 − w 3 ) 2 , then p ( w ( t ) ) = t = w i p(\mathbf{w}^{(t)})=t=w_i p ( w ( t ) ) = t = w i for all i i i . From now on, for a polynomial p p p , we refer to deg ( p ) \deg(p) deg ( p ) as the algebraic degree of p p p , as to not confuse it with its effective degree.
We use the following assumptions throughout the paper.
Assumption 1 (Loss minimization). The iterates w ( t ) \mathbf{w}^{(t)} w ( t ) minimize the objective, i.e. L ( w ( t ) ) → t → ∞ 0 \mathcal{L}(\mathbf{w}^{(t)})\xrightarrow[t\to\infty]{}0 L ( w ( t ) ) t → ∞ 0 .
Assumption 2 (Convergence in direction). The model’s parameters converge in direction, i.e.
lim t → ∞ w ( t ) ∥ w ( t ) ∥ = w ˉ ∞ . \lim_{t\to\infty}\frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert}=\bar{\mathbf{w}}^\infty. t → ∞ lim ∥ w ( t ) ∥ w ( t ) = w ˉ ∞ .
Assumption 3. ∀ i \forall i ∀ i , effdeg w ( t ) ( Φ x i ⊤ y i ) = ν i \operatorname{effdeg}_{\mathbf{w}^{(t)}}(\Phi^\top_{\mathbf{x}_i}y_i)=\nu_i effdeg w ( t ) ( Φ x i ⊤ y i ) = ν i for ν i ∈ N 1 \nu_i\in\mathbb{N}^{1} ν i ∈ N 1 , with some C i > 0 C_i>0 C i > 0 .
In our first assumption, we assume the iterates of the parameters w ( t ) \mathbf{w}^{(t)} w ( t ) minimize the objective, i.e. we assume that the loss converges to zero. This is a practical assumption as in many cases deep learning exhibits complete interpolation i.e. zero or close to zero error on the data. It also allows us to separate the question of whether gradient descent interpolates from the question of where gradient descent converges when it interpolates. As such, this assumption is also common in literature (e.g., Gunasekar et al. [13 ] ; Vardi & Shamir [35 ] ; Timor et al. [33 ] ).
Our second assumption states that the parameter vector w ( t ) \mathbf{w}^{(t)} w ( t ) converges in direction. This assumption means that the trajectory of w ( t ) \mathbf{w}^{(t)} w ( t ) is “well behaved”, and does not oscillate indefinitely. This is a standard assumption in the literature, and although some works, such as Ji & Telgarsky (2020b); Cai et al. [2 ] have managed to prove it under some assumptions, many works use it implicitly or explicitly as an assumption (e.g., Kunin et al. [20 ] ; Gunasekar et al. [13 ] ; Lyu & Li [23 ] ). Studying when directional convergence can be proved in our setting is a natural and important open question; we leave it to future work and focus here on characterizing the limit.
Our third assumption is more elaborate, but it is natural when we think of polynomials. If we fix some unit vector w ˉ ∞ \bar{\mathbf{w}}^\infty w ˉ ∞ , then for any polynomial p ( w ) p(\mathbf{w}) p ( w ) , the univariate polynomial q ( ∥ w ( t ) ∥ ) : = p ( ∥ w ( t ) ∥ w ˉ ∞ ) = Θ ( p k ( ∥ w ( t ) ∥ w ˉ ∞ ) ) q(\lVert\mathbf{w}^{(t)}\rVert):=p(\lVert\mathbf{w}^{(t)}\rVert\bar{\mathbf{w}}^\infty)=\Theta\left(p^k(\lVert\mathbf{w}^{(t)}\rVert\bar{\mathbf{w}}^\infty)\right) q (∥ w ( t ) ∥) := p (∥ w ( t ) ∥ w ˉ ∞ ) = Θ ( p k (∥ w ( t ) ∥ w ˉ ∞ ) ) when ∥ w ( t ) ∥ → ∞ \lVert\mathbf{w}^{(t)}\rVert\to\infty ∥ w ( t ) ∥ → ∞ , for the maximal k k k for which p k ( w ˉ ∞ ) ≠ 0 p^k(\bar{\mathbf{w}}^\infty)\ne0 p k ( w ˉ ∞ ) = 0 . In words, when we plug in a fixed direction to a multivariate polynomial, it starts behaving like a univariate polynomial, hence grows with its degree. In this case, its degree would be the highest degree that does not cancel when plugging the direction in. The gradient part of the definition ensures that the optimization does not go too far off this rather intuitive behavior.
Initialization-Dependent Effective Degree
Having the effective degree dependent on the gradient descent trajectory may seem like a burdensome complication, which makes the calculation of the implicit bias highly contextual. We show that under the setting of our non-homogeneous polynomials, the effective degree may be different for initializations in two different open sets. Hence, instead of only depending on the model and training examples, it depends on the initialization as well.
Lemma 4.2. There exist a polynomial predictor P : R D → R n \mathcal{P} : \mathbb{R}^{D} \to\mathbb{R}^{n} P : R D → R n , a dataset { x i , y i } i = 1 n \{x_i, y_i\}_{i=1}^{n} { x i , y i } i = 1 n and two disjoint open sets U 2 , U 4 ⊂ R D U_2, U_4 \subset\mathbb{R}^{D} U 2 , U 4 ⊂ R D , such that when trained with gradient descent Equation (1) with a constant step size η > 0 \eta> 0 η > 0 , the margins m i = y i ⟨ P , x i ⟩ m_i = y_i \langle\mathcal{P}, x_i\rangle m i = y i ⟨ P , x i ⟩ have:
effdeg ( m i ) = { 2 i w ( 0 ) ∈ U 2 4 i w ( 0 ) ∈ U 4 \operatorname{effdeg}(m_i) =
\begin{cases}
2i & \mathbf{w}^{(0)} \in U_2 \\
4i & \mathbf{w}^{(0)} \in U_4
\end{cases} effdeg ( m i ) = { 2 i 4 i w ( 0 ) ∈ U 2 w ( 0 ) ∈ U 4
The full proof is in Section D.1. The proof holds for every n n n , and for any subset of the above dataset, and it can be extended to a larger family of predictors beyond that in the lemma. The example above requires careful analysis of the optimization trajectory, but this is not always the case. We provide a concrete, nontrivial family of predictors and datasets, for which the effective degree follows from a short calculation, in Section D.2.
Comparison with bartlett
At this point, one might think that the implicit bias in our setting can be analyzed using the results from Cai et al. [2 ] , at least for the case where all examples share the same predictor. Cai et al. [2 ] established a result on the implicit bias for the class of nearly-homogeneous models , which include our polynomial setting. Perhaps surprisingly, their result is restrictive, as the assumptions of Cai et al. [2 ] implicitly imply that the effective degree of the model equals its algebraic degree. Thus, we can think of Assumption 3 as a relaxation which captures a broader variety of dynamical behaviors.
Assumption 4 (Cai et al. [2 ] , simplified for our setting). At some point t 0 t_0 t 0 during optimization, L ( w ( t 0 ) ) ≤ exp ( − p ( t 0 ) ) \mathcal{L}(\mathbf{w}^{(t_0)}) \leq\exp(-p(t_0)) L ( w ( t 0 ) ) ≤ exp ( − p ( t 0 )) for some p ( x ) p(x) p ( x ) . If the predictor is a polynomial of the form f ( w ; x ) = ⟨ q ( w ) , x ⟩ f(\mathbf{w}; \mathbf{x}) = \langle q(\mathbf{w}), \mathbf{x}\rangle f ( w ; x ) = ⟨ q ( w ) , x ⟩ , then 0 ≤ deg ( p ) ≤ deg ( q ) − 1 0 \leq\deg(p) \leq\deg(q)-1 0 ≤ deg ( p ) ≤ deg ( q ) − 1 , where for the special case of an homogeneous q q q , p ≡ 0 p \equiv0 p ≡ 0
Assumption 4 is presented in Cai et al. [2 ] as a strong separation assumption on the data. However, it is essentially an assumption on the optimization rate. Since it includes t 0 t_0 t 0 on both sides of the inequality, it requires that at some point, the optimization reaches a certain rate; specifically, it requires that at some point all margins behave as if they were some polynomial w.r.t. the model’s parameters. The problem is that the condition implies that all margins progress at some maximal rate. Is it possible that this maximal rate is never reached, and hence Assumption 4 does not hold? As we shall see next, the answer is positive even for a very simple setting with our linear autoregressive model.
Lemma 4.3. Consider the linear autoregressive setting from Section 3.2, trained using gradient descent Equation (1) with a constant step size η > 0 \eta> 0 η > 0 and initialization at 0 0 0 . Then, there exists a size-1 dataset { ( x , y ) } \{(x,y)\} {( x , y )} , such that Assumption 4 from Cai et al. [2 ] does not hold, while our Assumptions 1 to 3 hold.
The example we provide has a margin (polynomial w.r.t. w \mathbf{w} w ), with algebraic degree 4, but an effective degree of 1. This is in contradiction to Assumption 4 that implies the effective degree is higher. Moreover, if we do not insist on the setting from Section 3.2, we can find such examples for every even algebraic degree d d d , and k < d k < d k < d effective degree (see Section C.3), which in the case of even k k k work for almost all initializations. Therefore, a more nuanced approach is necessary to capture the range of possible dynamics of optimization, Assumption 3 is one such approach. In fact, in the following lemma we prove that in the case of polynomials, Assumption 4 implies Assumptions 1 to 3 (see Section A.4 for the proof):
Lemma 4.4. In the setting where the predictor is polynomial, Assumption 4 implies Assumptions 1 to 3.
Assumption 4 essentially requires that the effective degree for every example is equal to the algebraic degree, while Assumption 3 allows any effective degree. As we will see in Section 5, the implicit bias of gradient descent is different than the one presented in Cai et al. [2 ] for the cases where Assumption 4 does not hold. In Section 6, we empirically verify that our assumptions (including Assumption 3) indeed hold, at least in some cases.
We further emphasize that while the above discussion shows that for the case where all examples share the same predictor P i \mathcal{P}_i P i (i.e., the same l i l_i l i and T i T_i T i ), our Assumption 3 is weaker than Assumption 4 of Cai et al. [2 ] , cases where l i l_i l i or T i T_i T i (and hence P i \mathcal{P}_i P i ) are not uniform for all examples are not covered by the setting in Cai et al. [2 ] at all. On the other hand, Cai et al. [2 ] cover non-polynomial model, which we do not consider.
Finally, we note that it is theoretically possible that the dynamics do not satisfy our assumptions. This can happen when the effective degree of the polynomial along the trajectory behaves as a “fractional-degree”. We elaborate on these cases in Section A.3. Studying such dynamics introduces a challenging analysis since the margin in these cases, as well as the KKT conditions, is not well-defined. We leave investigating those cases for future work.
Main Results
We will now present our main result, and show how it applies to the various models we consider.
Theorem 5.1. Let { P i } i = 1 N \{\mathcal{P}_i\}_{i=1}^{N} { P i } i = 1 N , a collection of polynomials with parameters w ∈ R D \mathbf{w} \in\mathbb{R}^{D} w ∈ R D . For a dataset { x i , y i } i = 1 N \{x_i,y_i\}_{i=1}^{N} { x i , y i } i = 1 N where y i = sgn ( ⟨ P i ( w ) , x i ⟩ ) y_i = \operatorname{sgn}\left(\langle\mathcal{P}_i(\mathbf{w}),x_i\rangle\right) y i = sgn ( ⟨ P i ( w ) , x i ⟩ ) , consider the sequence of gradient updates (Equation (1) ) with a non-increasing sequence of step sizes { η t } t \{\eta_t\}_t { η t } t , minimizing the empirical risk. If Assumptions 1 to 3 hold, we have that w ˉ ∞ : = lim t → ∞ w ( t ) ∥ w ( t ) ∥ \bar{\mathbf{w}}^{\infty} := \lim_{t\to\infty} \frac{\mathbf{w}^{(t)}}{\|\mathbf{w}^{(t)}\|} w ˉ ∞ := lim t → ∞ ∥ w ( t ) ∥ w ( t ) is a positive scaling of a first order stationary point (KKT point) of the following optimization problem:
min w ∈ R D ∥ w ∥ 2 2 s.t. ∀ i , y i ⟨ x i , P i ν i ( w ) ⟩ ≥ 0 ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w ) ⟩ ≥ 1 , (4) \begin{aligned}
\min_{\mathbf{w}\in\mathbb{R}^{D}}\ &\|\mathbf{w}\|_2^2 \quad\text{s.t. } \forall i,\ y_i\left\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\right\rangle\ge0 \\
&\forall i\in Z,\ y_i C_i\left\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\right\rangle\ge1, \tag*{(4)}
\end{aligned} w ∈ R D min ∥ w ∥ 2 2 s.t. ∀ i , y i ⟨ x i , P i ν i ( w ) ⟩ ≥ 0 ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w ) ⟩ ≥ 1 , (4) where Z = arg min i ∈ [ n ] { effdeg w y i ⟨ P i , x i ⟩ } Z = \arg\min_{i\in[n]}\{\operatorname{effdeg}_{\mathbf{w}} y_i\langle\mathcal{P}_i,x_i\rangle\} Z = arg min i ∈ [ n ] { effdeg w y i ⟨ P i , x i ⟩} .
The idea behind the proof is to establish the dominance of the lowest effective-degree margins on the gradient, as higher effective degrees decay faster. Then, analyzing the induced gradient-descent updates on the dual weights, i.e., the dual certificate, to show a subsequence of these weights converges. Finally, we use a convexity argument to show the gradient under this subsequence converges in direction to w ˉ ∞ \bar{\mathbf{w}}^{\infty} w ˉ ∞ . An extended proof intuition and the full proof are in Section E.
Note that this result is different in numerous ways from the implicit bias of homogeneous models, given by
min w ∥ w ∥ s.t. ∀ i , y i f ( w ) ≥ 1. \min_{\mathbf{w}}\|\mathbf{w}\| \quad\text{s.t. } \forall i,\ y_i f(\mathbf{w})\ge1 . w min ∥ w ∥ s.t. ∀ i , y i f ( w ) ≥ 1.
The positive-margin constraint is only w.r.t. some of the samples, specifically the ones with lowest effective degree. The constraint on the rest of the samples only roughly requires that they will be classified correctly.
Each sample is associated with a different predictor.
Importantly, the margins y i C i ⟨ x i , P i ν i ( w ) ⟩ y_i C_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle y i C i ⟨ x i , P i ν i ( w )⟩ use different predictors than those being optimized (P i \mathcal{P}_i P i rather than P i ν i \mathcal{P}_i^{\nu_i} P i ν i ), that is, the direction of convergence is towards a KKT point of an optimization problem which may a-priori seem unrelated to the optimized objective. This is similar to Cai et al. [2 ] ; however, they use the algebraic-degree homogenization of the optimized predictors instead of the effective-degree homogenization.
The implicit bias is different depending on the dynamics of gradient descent and depends on the effective degree, which is optimization dependent. This is similar in spirit to Ataee et al. (2023); Tarzanagh et al. (2023), where multiple implicit biases are possible depending on the optimization dynamics.
Note that Theorem 5.1 is phrased in a rather general sense for polynomial predictors, but since we know each of our models is in fact a polynomial, we can see how it applies to them.
Corollary 5.2. Under the settings of Theorem 5.1 , w ˉ ∞ \bar{\mathbf{w}}^{\infty} w ˉ ∞ is a positive scaling of a first order stationary point of Equation (OPT), where:
Interpretation of the max-margin solution. We split the interpretation into two cases: (1) All polynomials have their maximal possible effective degree, i.e. ∀ i , deg ( P i ) = effdeg ( P i ) \forall i,\ \deg(\mathcal{P}_i) = \operatorname{effdeg}(\mathcal{P}_i) ∀ i , deg ( P i ) = effdeg ( P i ) . (2) Some polynomials do not achieve their maximal effective degree, i.e. ∃ i , deg ( P i ) > effdeg ( P i ) \exists i,\ \deg(\mathcal{P}_i) > \operatorname{effdeg}(\mathcal{P}_i) ∃ i , deg ( P i ) > effdeg ( P i ) .
In the first case, our results imply that the implicit bias for a max-margin solution is affected mainly by the shortest sequences seen during training (i.e. polynomials with lowest degree). The other sequences are only required to obtain a correct classification. In the second case, the solution is mainly affected by the examples with the lowest effective degree. This very roughly extends the interpretation of the first case, in the sense that fewer autoregressive steps, or lower-degree polynomials, are most likely to have a lower effective degree. For example, in the case of polynomials corresponding to only one autoregressive step, these polynomials are guaranteed to have the minimal effective degree, and hence have greater effect on the max-margin.
Experiments
The discussion in Section 4.2 alludes that our assumptions may not hold in general, hence it’s customary to check them empirically. We performed experiments with autoregressive linear models and SSMs both on synthetic data and on MNIST [21 ] , and found many cases for which our assumptions hold. In the main text, we present our experiments on autoregressive linear models with a synthetic teacher-student dataset. Here, we generate a random teacher and use it to label Gaussian examples, with different numbers of autoregressive steps T ∈ { 1 , 3 , 5 } T \in\{1, 3, 5\} T ∈ { 1 , 3 , 5 } . We refer the reader to Section G for experiments with SSMs and linear autoregressive models in other settings, and for more information about the experiments.
Verifying Our Assumptions
In Figure 2 and Figure 3 we verify our assumptions. First, we plot the loss to verify Assumption 1. We see that the loss approaches 0. Next, to verify Assumption 2 we consider the correlation of the parameter direction with the final predictor, i.e. ⟨ w ˉ ( t ) , w ˉ ∞ ⟩ \left\langle\bar{\mathbf{w}}^{(t)}, \bar{\mathbf{w}}^\infty\right\rangle ⟨ w ˉ ( t ) , w ˉ ∞ ⟩ . Here, since we do not have access to the directional limit w ˉ ∞ \bar{\mathbf{w}}^\infty w ˉ ∞ , we use the last iterate instead. We see that iterates become close to this direction quite early in training. For Assumption 3, we plot the following ratios for each training example. We plot ∣ m i effdeg ( m i ) ( w ( t ) ) m i ( w ( t ) ) ∣ \left|\frac{m_i^{\operatorname{effdeg}(m_i)}(\mathbf{w}^{(t)})}{m_i(\mathbf{w}^{(t)})}\right| m i ( w ( t ) ) m i effdeg ( m i ) ( w ( t ) ) , likewise ∥ ∇ m i effdeg ( m i ) ( w ( t ) ) ∥ ∥ ∇ m i ( w ( t ) ) ∥ \frac{\left\|\nabla_{m_i}^{\operatorname{effdeg}(m_i)}(\mathbf{w}^{(t)})\right\|}{\left\|\nabla m_i(\mathbf{w}^{(t)})\right\|} ∥ ∇ m i ( w ( t ) ) ∥ ∇ m i effdeg ( m i ) ( w ( t ) ) , and ∣ m i effdeg ( m i ) ( w ( t ) ) / m i ( w ( t ) ) ∣ ∥ ∇ m i effdeg ( m i ) ( w ( t ) ) ∥ / ∥ ∇ m i ( w ( t ) ) ∥ \frac{\left|m_i^{\operatorname{effdeg}(m_i)}(\mathbf{w}^{(t)})/m_i(\mathbf{w}^{(t)})\right|}{\left\|\nabla_{m_i}^{\operatorname{effdeg}(m_i)}(\mathbf{w}^{(t)})\right\|/\left\|\nabla m_i(\mathbf{w}^{(t)})\right\|} ∇ m i effdeg ( m i ) ( w ( t ) ) / ∥ ∇ m i ( w ( t ) ) ∥ m i effdeg ( m i ) ( w ( t ) ) / m i ( w ( t ) ) where m i ( w ( t ) ) : = y i ⟨ P i ( w ( t ) ) , x i ⟩ m_i(\mathbf{w}^{(t)}) := y_i\left\langle\mathcal{P}_i(\mathbf{w}^{(t)}),\mathbf{x}_i\right\rangle m i ( w ( t ) ) := y i ⟨ P i ( w ( t ) ) , x i ⟩ . Assumption 3 requires the first and second ratios to converge to the same constants, with the third ratio converging to 1. For details about finding the effective degree, see Section G.
Figure 2. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale.
Figure 3. Left: Ratio between the effective degree homogenization of the margin and the margin, per example. Middle: The corresponding ratio for the norms of the gradients of the homogenization of the margin, and the margin. Right: The ratio of the two former ratios. Y-axes are log-scale.
Convergence to Max-Margin
In Figure 4 we plot the minimum margins per effective degree. We see that margins of examples with the minimal effective degree indeed converge to higher values, in accordance to the constraints of Equation (OPT) and the results of Theorem 5.1 . We also plot the outputs per effective degree after training. This verifies that examples with lower effective degree have margins bounded away from zero, and shows the concentration of margins in higher-effective degrees, which matches Equation (OPT).
Figure 4. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins.
Discussion and future work
In this work, we characterized the implicit bias of gradient descent for polynomial predictors where each sample is labeled by a different predictor. It implies implicit bias results for Linear Autoregressive Models and linear SSMs, including linear Autoregressive SSMs. Finally, we shed light on the implicit bias of non-homogeneous models through the lens of polynomials, demonstrating that previous assumptions about their dynamics [2 ] may be overly strong in certain settings.
Some interesting questions remain for future work: First, can we completely characterize the implicit bias for the multi-polynomial case? We found that sometimes, the effective degree may be fractional (Section A.3). What is the implicit bias in those cases? Second, how can we generalize these polynomial results to general non-homogeneous models? We consider only polynomials in this paper, but similar results may apply to other classes of predictors and be of much interest.
Acknowledgements
GK is funded by the European Research Council (ERC) under the European Union’s Horizon 2022 research and innovation program (grant agreement No. 101041711), and the Simons Foundation (as part of the Collaboration on the Mathematical and Scientific Foundations of Deep Learning). GV is supported by the Israel Science Foundation (grant No. 2574/25), by a research grant from Mortimer Zuckerman (the Zuckerman STEM Leadership Program), and by research grants from the Center for New Scientists at the Weizmann Institute of Science, and the Shimon and Golde Picker – Weizmann Annual Grant.
References
Extra Discussions and Comparisons
Concrete Examples for Models
Linear Autoregressive
Let x ∈ R l \mathbf{x} \in\mathbb{R}^{l} x ∈ R l , w ∈ R D \mathbf{w} \in\mathbb{R}^{D} w ∈ R D . Suppose D = l = 2 D = l = 2 D = l = 2 and consider T = 2 T = 2 T = 2 autoregressive steps. We have that:
x 3 = ⟨ w , x ⟩ = w 1 x 1 + w 2 x 2 x_{3} = \langle\mathbf{w}, \mathbf{x} \rangle= w_{1}x_{1} + w_{2}x_{2} x 3 = ⟨ w , x ⟩ = w 1 x 1 + w 2 x 2
x 4 = ⟨ w , x 2 : D + T − 1 ⟩ = ⟨ w , x 2 : 3 ⟩ = w 1 w 2 x 1 + ( w 1 + w 2 2 ) x 2 x_{4} = \langle\mathbf{w}, \mathbf{x}_{2:D+T-1} \rangle= \langle\mathbf{w}, \mathbf{x}_{2:3} \rangle
= w_{1}w_{2}x_{1} + (w_{1} + w_{2}^{2})x_{2} x 4 = ⟨ w , x 2 : D + T − 1 ⟩ = ⟨ w , x 2 : 3 ⟩ = w 1 w 2 x 1 + ( w 1 + w 2 2 ) x 2
For our example the polynomial vector is P ( w ) = ( w 1 w 2 w 1 + w 2 2 ) \mathcal{P}(\mathbf{w}) = \begin{pmatrix} w_{1}w_{2} \\ w_{1} + w_{2}^{2} \end{pmatrix} P ( w ) = ( w 1 w 2 w 1 + w 2 2 ) , and in our example x 4 = ( w 1 w 2 , w 1 + w 2 2 ) x x_{4} = (w_{1}w_{2}, w_{1} + w_{2}^{2})\mathbf{x} x 4 = ( w 1 w 2 , w 1 + w 2 2 ) x .
Autoregressive SSM
For a concrete example, consider l = 2 , T = 2 l = 2, T = 2 l = 2 , T = 2 . Then we have
x 3 = C T A B x 1 + C T B x 2 x_{3} = C^{\mathsf{T}}ABx_{1} + C^{\mathsf{T}}Bx_{2} x 3 = C T A B x 1 + C T B x 2
x 4 = C T A 2 B x 1 + C T A B x 2 + C T B x 3 = C T A 2 B x 1 + C T A B x 2 + C T B ( C T A B x 1 + C T B x 2 ) = ( C T A 2 B + C T B C T A B ) x 1 + ( C T A B + ( C T B ) 2 ) x 2 \begin{aligned}
x_{4} = C^{\mathsf{T}}A^{2}Bx_{1} + C^{\mathsf{T}}ABx_{2} + C^{\mathsf{T}}Bx_{3}
= C^{\mathsf{T}}A^{2}Bx_{1} + C^{\mathsf{T}}ABx_{2} + C^{\mathsf{T}}B\left(C^{\mathsf{T}}ABx_{1} + C^{\mathsf{T}}Bx_{2}\right) \\
&= \left(C^{\mathsf{T}}A^{2}B + C^{\mathsf{T}}BC^{\mathsf{T}}AB\right)x_{1} + \left(C^{\mathsf{T}}AB + (C^{\mathsf{T}}B)^{2}\right)x_{2}
\end{aligned} x 4 = C T A 2 B x 1 + C T A B x 2 + C T B x 3 = C T A 2 B x 1 + C T A B x 2 + C T B ( C T A B x 1 + C T B x 2 ) = ( C T A 2 B + C T B C T A B ) x 1 + ( C T A B + ( C T B ) 2 ) x 2
That is the corresponding polynomial would be
Q ( A , B , C ) = ( C T A 2 B + C T B C T A B C T A B + ( C T B ) 2 ) Q(A, B, C) = \begin{pmatrix}
C^{\mathsf{T}}A^{2}B + C^{\mathsf{T}}BC^{\mathsf{T}}AB \\
C^{\mathsf{T}}AB + (C^{\mathsf{T}}B)^{2}
\end{pmatrix} Q ( A , B , C ) = ( C T A 2 B + C T B C T A B C T A B + ( C T B ) 2 )
Comparison to kunin2022asymmetric
Kunin et al. [20 ] define a model to be Λ \Lambda Λ -quasi-homogeneous if:
f ( x , e α Λ w ) = e α f ( x , w ) f(\mathbf{x}, e^{\alpha\Lambda}\mathbf{w}) = e^{\alpha}f(\mathbf{x}, \mathbf{w}) f ( x , e α Λ w ) = e α f ( x , w )
For any scalar α > 0 \alpha> 0 α > 0 , input x \mathbf{x} x and parameters w \mathbf{w} w . These models satisfy a type of Euler-Homogeneous-Theorem. i.e.
⟨ ∇ w f ( x , w ) , Λ w ⟩ = f ( x , w ) \langle\nabla_{\mathbf{w}} f(\mathbf{x}, \mathbf{w}), \Lambda\mathbf{w} \rangle= f(\mathbf{x}, \mathbf{w}) ⟨ ∇ w f ( x , w ) , Λ w ⟩ = f ( x , w )
Under the setting of § 3.2 , let us consider the model with two autoregressive steps, with an example in 2D, i.e. f ( x , w ) = x 1 w 1 w 2 + x 2 ( w 1 + w 2 2 ) f(\mathbf{x}, \mathbf{w}) = x_{1}w_{1}w_{2} + x_{2}(w_{1} + w_{2}^{2}) f ( x , w ) = x 1 w 1 w 2 + x 2 ( w 1 + w 2 2 ) . Since f f f is linear in x \mathbf{x} x , and the identity must hold for any x \mathbf{x} x , then the identities
⟨ ∇ w w 1 w 2 , Λ w ⟩ = w 1 w 2 \langle\nabla_{\mathbf{w}}w_{1}w_{2}, \Lambda\mathbf{w} \rangle= w_{1}w_{2} ⟨ ∇ w w 1 w 2 , Λ w ⟩ = w 1 w 2
⟨ ∇ w w 1 + w 2 2 , Λ w ⟩ = w 1 + w 2 2 \langle\nabla_{\mathbf{w}}w_{1} + w_{2}^{2}, \Lambda\mathbf{w} \rangle= w_{1} + w_{2}^{2} ⟨ ∇ w w 1 + w 2 2 , Λ w ⟩ = w 1 + w 2 2
must hold. Let Λ = ( a b c d ) \Lambda= \begin{pmatrix} a & b \\ c & d \end{pmatrix} Λ = ( a c b d ) . We will start with the first identity
⟨ ∇ w w 1 w 2 , Λ w ⟩ = c w 1 2 + ( a + d ) w 1 w 2 + b w 2 2 . \langle\nabla_{\mathbf{w}}w_{1}w_{2}, \Lambda\mathbf{w} \rangle= cw_{1}^{2} + (a + d)w_{1}w_{2} + bw_{2}^{2}. ⟨ ∇ w w 1 w 2 , Λ w ⟩ = c w 1 2 + ( a + d ) w 1 w 2 + b w 2 2 .
Hence b = c = 0 , a + d = 1 b = c = 0, a + d = 1 b = c = 0 , a + d = 1 . For the second identity, we have
⟨ ∇ w w 1 + w 2 2 , Λ w ⟩ = a w 1 + b w 2 + 2 c w 1 w 2 + 2 d w 2 2 . \langle\nabla_{\mathbf{w}}w_{1} + w_{2}^{2}, \Lambda\mathbf{w} \rangle= aw_{1} + bw_{2} + 2cw_{1}w_{2} + 2dw_{2}^{2}. ⟨ ∇ w w 1 + w 2 2 , Λ w ⟩ = a w 1 + b w 2 + 2 c w 1 w 2 + 2 d w 2 2 .
Hence a = 1 , d = 1 / 2 , b = c = 0 a = 1, d = 1/2, b = c = 0 a = 1 , d = 1/2 , b = c = 0 . But now we have a + d = 3 / 2 ≠ 1 a + d = 3/2 \ne1 a + d = 3/2 = 1 , contradiction to the first identity’s requirements. Hence there exists no Λ \Lambda Λ s.t. f f f is quasi-homogeneous.
Fractional Effective Degree
It is important to note that while Assumptions 1 to 3 capture a wider variety of dynamical behaviors than Cai et al. [2 ] , they still do not capture the full array of possible dynamics. For example, our effective degree assumption does not always hold.
Lemma A.1. There exists an example ( x , y ) (\mathbf{x},y) ( x , y ) , under the setting in § 3.2 , using Equation (1) after initializing at w ( 0 ) = ( − 2 , 1 ) \mathbf{w}^{(0)}=(-2,1) w ( 0 ) = ( − 2 , 1 ) , with a constant step size η > 0 \eta>0 η > 0 s.t. its margin m ( w ( t ) ) = Θ ( ∥ w ( t ) ∥ 3 / 2 ) m(\mathbf{w}^{(t)})=\Theta\left(\lVert\mathbf{w}^{(t)}\rVert^{3/2}\right) m ( w ( t ) ) = Θ ( ∥ w ( t ) ∥ 3/2 ) , violating both Assumption 4 and Assumption 3.
We take x = ( 0 − 1 / 3 ) \mathbf{x}=\begin{pmatrix}0\\-1/3\end{pmatrix} x = ( 0 − 1/3 ) , y = 1 y=1 y = 1 , and
P ( w ) = ( w 1 2 + w 1 w 2 2 2 w 1 w 2 + w 2 3 ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}w_1^2+w_1w_2^2\\2w_1w_2+w_2^3\end{pmatrix} P ( w ) = ( w 1 2 + w 1 w 2 2 2 w 1 w 2 + w 2 3 )
corresponding to 3 autoregressive steps. We have L ( w ) = exp ( 2 3 w 1 w 2 + 1 3 w 2 3 ) \mathcal{L}(\mathbf{w})=\exp\left(\frac{2}{3}w_1w_2+\frac{1}{3}w_2^3\right) L ( w ) = exp ( 3 2 w 1 w 2 + 3 1 w 2 3 ) . The idea is that the optimization path follows a coordinate-relation of w 1 ∼ − w 2 2 w_1\sim-w_2^2 w 1 ∼ − w 2 2 , so we have w ‾ ∞ = ( − 1 0 ) \overline{\mathbf{w}}^\infty=\begin{pmatrix}-1\\0\end{pmatrix} w ∞ = ( − 1 0 ) and ∥ w ∥ ∼ w 2 2 \lVert\mathbf{w}\rVert\sim w_2^2 ∥ w ∥ ∼ w 2 2 , both since w 1 w_1 w 1 dominates w 2 w_2 w 2 . Similarly, substituting w 1 w_1 w 1 with − w 2 2 -w_2^2 − w 2 2 we get that the margin ∼ w 2 3 \sim w_2^3 ∼ w 2 3 . Hence effectively behaves like a 3/2 degree polynomial - which is not an integer. A way to think about what happens is that w ( t ) / ∥ w ( t ) ∥ \mathbf{w}^{(t)}/\lVert\mathbf{w}^{(t)}\rVert w ( t ) / ∥ w ( t ) ∥ converges slow enough, that the radial speed in the direction of w ‾ ∞ \overline{\mathbf{w}}^\infty w ∞ is significantly slower than the radial speed of w \mathbf{w} w . The full proof is in Section A.3.1. Note that in this example Assumption 4 does not hold as well, as by Theorem C.1 it requires the margin to scale like Θ ( ∥ w ( t ) ∥ 2 ) \Theta\left(\lVert\mathbf{w}^{(t)}\rVert^2\right) Θ ( ∥ w ( t ) ∥ 2 ) .
A natural question arises as to whether Assumption 3 can be generalized to fractional-effective degrees. This is an interesting future research direction. We note that it cannot simply be extended using our proof technique. First, note that there is no trivial generalization for homogenization to a fractional degree of a polynomial with integer powers, and Assumption 3 requires the limit to equal a homogenization of the margin. Second, if we relax Assumption 4 to only require lim t → ∞ m ( w ( t ) ) ∥ w ( t ) ∥ k = C ≠ 0 \lim_{t\to\infty}\frac{m(\mathbf{w}^{(t)})}{\lVert\mathbf{w}^{(t)}\rVert^k}=C\ne0 lim t → ∞ ∥ w ( t ) ∥ k m ( w ( t ) ) = C = 0 for some fractional k k k , a new problem arises. Recall that by Assumption 2 we have that w ( t ) ∥ w ( t ) ∥ = w ‾ ∞ + δ ( t ) \frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert}=\overline{\mathbf{w}}^\infty+\delta^{(t)} ∥ w ( t ) ∥ w ( t ) = w ∞ + δ ( t ) with δ ( t ) → 0 \delta^{(t)}\to0 δ ( t ) → 0 , then using taylor approximation we have
∇ m ( w ( t ) ) = ∑ i = 1 d ∥ w ( t ) ∥ i − 1 ∇ m i ( w ( t ) ∥ w ( t ) ∥ ) = ∑ i = 1 d ∥ w ( t ) ∥ i − 1 ∇ m i ( w ‾ ∞ + δ ( t ) ) = ∑ i = 1 d ∥ w ( t ) ∥ i − 1 ∑ j = 1 i ∇ j m i ( w ‾ ∞ ) [ ( δ ( t ) ) ⊗ j − 1 ] \begin{aligned}
\nabla m\left(\mathbf{w}^{(t)}\right)=\sum_{i=1}^{d}\lVert\mathbf{w}^{(t)}\rVert^{i-1}\nabla m^i\left(\frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert}\right) \\
&=\sum_{i=1}^{d}\lVert\mathbf{w}^{(t)}\rVert^{i-1}\nabla m^i\left(\overline{\mathbf{w}}^\infty+\delta^{(t)}\right) \\
&=\sum_{i=1}^{d}\lVert\mathbf{w}^{(t)}\rVert^{i-1}\sum_{j=1}^{i}\nabla^j m^i\left(\overline{\mathbf{w}}^\infty\right)\left[(\delta^{(t)})^{\otimes j-1}\right]
\end{aligned} ∇ m ( w ( t ) ) = i = 1 ∑ d ∥ w ( t ) ∥ i − 1 ∇ m i ( ∥ w ( t ) ∥ w ( t ) ) = i = 1 ∑ d ∥ w ( t ) ∥ i − 1 ∇ m i ( w ∞ + δ ( t ) ) = i = 1 ∑ d ∥ w ( t ) ∥ i − 1 j = 1 ∑ i ∇ j m i ( w ∞ ) [ ( δ ( t ) ) ⊗ j − 1 ]
Where ( δ ( t ) ) ⊗ j = ( δ ( t ) , … , δ ( t ) ) (\delta^{(t)})^{\otimes j}=(\delta^{(t)},\ldots,\delta^{(t)}) ( δ ( t ) ) ⊗ j = ( δ ( t ) , … , δ ( t ) ) i i i times. Now ∥ ∇ j m i ( w ‾ ∞ ) [ ( δ ( t ) ) ⊗ j ] ∥ = O ( ∥ δ ( t ) ∥ j − 1 ) \lVert\nabla^j m^i(\overline{\mathbf{w}}^\infty)[(\delta^{(t)})^{\otimes j}]\rVert=O\left(\lVert\delta^{(t)}\rVert^{j-1}\right) ∥ ∇ j m i ( w ∞ ) [( δ ( t ) ) ⊗ j ]∥ = O ( ∥ δ ( t ) ∥ j − 1 ) , and hence if w ‾ ∞ \overline{\mathbf{w}}^\infty w ∞ zeros out some of the top homogenizations of m m m and their gradients, the dominant terms may come from either the expansions of lower degree homogenizations, or expansions of those canceled top homogenizations, i.e. δ ( t ) ∥ w ( t ) ∥ k ≭ ∥ w ( t ) ∥ k − 1 \delta^{(t)}\lVert\mathbf{w}^{(t)}\rVert^k\not\asymp\lVert\mathbf{w}^{(t)}\rVert^{k-1} δ ( t ) ∥ w ( t ) ∥ k ≍ ∥ w ( t ) ∥ k − 1 depending on how fast or slow δ ( t ) → 0 \delta^{(t)}\to0 δ ( t ) → 0 . All in all, we find that there’s a wide range of possible dynamical behaviors when optimizing polynomials with the exponential loss, some of which change the resulting implicit bias as we see in § 5 . We leave it for future work to investigate these cases, and focus on Assumption 3. We verify all of our assumptions experimentally in § 6 .
Proof of lemma:allfail
In the following example, we demonstrate that in a simple setting, our effective degree assumption may fail. Our example is in 2 D 2D 2 D and 3 linear autoregressive steps. We take x = ( 0 − 1 / 3 ) \mathbf{x}=\begin{pmatrix}0\\-1/3\end{pmatrix} x = ( 0 − 1/3 ) , y = 1 y=1 y = 1 , and
P ( w ) = ( w 1 2 + w 1 w 2 2 2 w 1 w 2 + w 2 2 ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}w_1^2+w_1w_2^2\\2w_1w_2+w_2^2\end{pmatrix} P ( w ) = ( w 1 2 + w 1 w 2 2 2 w 1 w 2 + w 2 2 )
So we have L ( w ) = exp ( 2 3 w 1 w 2 + 1 3 w 2 3 ) \mathcal{L}(\mathbf{w})=\exp(\frac{2}{3}w_1w_2+\frac{1}{3}w_2^3) L ( w ) = exp ( 3 2 w 1 w 2 + 3 1 w 2 3 ) . The idea is that the optimization path follows a relation of w 1 ∼ − w 2 2 w_1\sim-w_2^2 w 1 ∼ − w 2 2 , so we have w ˉ ∞ = ( − 1 0 ) \bar{\mathbf{w}}^\infty=\begin{pmatrix}-1\\0\end{pmatrix} w ˉ ∞ = ( − 1 0 ) and ∥ w ∥ ∼ w 2 2 \lVert\mathbf{w}\rVert\sim w_2^2 ∥ w ∥ ∼ w 2 2 while the margin ∼ w 2 3 \sim w_2^3 ∼ w 2 3 , hence the effective degree is 3 / 2 3/2 3/2 , not an integer. A way to think about what happens is that w ( t ) / ∥ w ( t ) ∥ \mathbf{w}^{(t)}/\lVert\mathbf{w}^{(t)}\rVert w ( t ) / ∥ w ( t ) ∥ converges slow enough, that the radial speed in the direction of w ˉ ∞ \bar{\mathbf{w}}^\infty w ˉ ∞ is significantly slower than the radial speed of w \mathbf{w} w . Let us analyze this formally:
∂ L ∂ w 1 = 2 3 L ( w ) w 2 \frac{\partial\mathcal{L}}{\partial w_1}=\frac{2}{3}\mathcal{L}(\mathbf{w})w_2 ∂ w 1 ∂ L = 3 2 L ( w ) w 2
∂ L ∂ w 2 = L ( w ) ( 2 3 w 1 + w 2 2 ) \frac{\partial\mathcal{L}}{\partial w_2}=\mathcal{L}(\mathbf{w})\left(\frac{2}{3}w_1+w_2^2\right) ∂ w 2 ∂ L = L ( w ) ( 3 2 w 1 + w 2 2 )
For a fixed learning rate η \eta η , denote α ( t ) = η L ( w ( t ) ) \alpha^{(t)}=\eta\mathcal{L}(\mathbf{w}^{(t)}) α ( t ) = η L ( w ( t ) ) , then the gradient descent update gives
w 1 ( t + 1 ) = w 1 ( t ) − α ( t ) 2 3 w 2 ( t ) w_1^{(t+1)}=w_1^{(t)}-\alpha^{(t)}\frac{2}{3}w_2^{(t)} w 1 ( t + 1 ) = w 1 ( t ) − α ( t ) 3 2 w 2 ( t )
w 2 ( t + 1 ) = w 2 ( t ) − α ( t ) ( 2 3 w 1 ( t ) + ( w 2 ( t ) ) 2 ) w_2^{(t+1)}=w_2^{(t)}-\alpha^{(t)}\left(\frac{2}{3}w_1^{(t)}+(w_2^{(t)})^2\right) w 2 ( t + 1 ) = w 2 ( t ) − α ( t ) ( 3 2 w 1 ( t ) + ( w 2 ( t ) ) 2 )
Let us start the optimization from w = ( − 2 1 ) \mathbf{w}=\begin{pmatrix}-2\\1\end{pmatrix} w = ( − 2 1 ) . Note that if d : = − 2 3 w 1 − w 2 2 ≥ 0 d:=-\frac{2}{3}w_1-w_2^2\geq0 d := − 3 2 w 1 − w 2 2 ≥ 0 and w 2 ≥ 0 w_2\geq0 w 2 ≥ 0 , then w 2 w_2 w 2 is non-decreasing. We will show d ( t ) ≥ 0 ∀ t d^{(t)}\geq0\ \forall t d ( t ) ≥ 0 ∀ t . Clearly d ( 0 ) = 1 / 3 d^{(0)}=1/3 d ( 0 ) = 1/3 . Let us see how d d d evolves:
d ( t + 1 ) = − 2 3 w 1 ( t + 1 ) − ( w 2 ( t + 1 ) ) 2 = − 2 3 w 1 ( t ) + α ( t ) 4 9 w 2 ( t ) − ( w 2 ( t ) + α ( t ) d ( t ) ) 2 = − 2 3 w 1 ( t ) + α ( t ) 4 9 w 2 ( t ) − ( w 2 ( t ) ) 2 − 2 w 2 ( t ) α ( t ) d ( t ) − ( α ( t ) d ( t ) ) 2 = d ( t ) − ( α ( t ) d ( t ) ) 2 + α ( t ) w 2 ( t ) ( 4 9 − 2 d ( t ) ) (5) \begin{aligned}
d^{(t+1)}&=-\frac{2}{3}w_1^{(t+1)}-(w_2^{(t+1)})^2\\
&=-\frac{2}{3}w_1^{(t)}+\alpha^{(t)}\frac{4}{9}w_2^{(t)}-\left(w_2^{(t)}+\alpha^{(t)}d^{(t)}\right)^2\\
&=-\frac{2}{3}w_1^{(t)}+\alpha^{(t)}\frac{4}{9}w_2^{(t)}-(w_2^{(t)})^2-2w_2^{(t)}\alpha^{(t)}d^{(t)}-\left(\alpha^{(t)}d^{(t)}\right)^2\\
&=d^{(t)}-\left(\alpha^{(t)}d^{(t)}\right)^2+\alpha^{(t)}w_2^{(t)}\left(\frac{4}{9}-2d^{(t)}\right) \tag*{(5)}
\end{aligned} d ( t + 1 ) = − 3 2 w 1 ( t + 1 ) − ( w 2 ( t + 1 ) ) 2 = − 3 2 w 1 ( t ) + α ( t ) 9 4 w 2 ( t ) − ( w 2 ( t ) + α ( t ) d ( t ) ) 2 = − 3 2 w 1 ( t ) + α ( t ) 9 4 w 2 ( t ) − ( w 2 ( t ) ) 2 − 2 w 2 ( t ) α ( t ) d ( t ) − ( α ( t ) d ( t ) ) 2 = d ( t ) − ( α ( t ) d ( t ) ) 2 + α ( t ) w 2 ( t ) ( 9 4 − 2 d ( t ) ) (5)
So whenever d ( t ) = 0 d^{(t)}=0 d ( t ) = 0 , d ( t + 1 ) = 4 9 α ( t ) w 2 ( t ) > 0 d^{(t+1)}=\frac{4}{9}\alpha^{(t)}w_2^{(t)}>0 d ( t + 1 ) = 9 4 α ( t ) w 2 ( t ) > 0 as long as w 2 ( t ) w_2^{(t)} w 2 ( t ) is positive. First we will show that d ( t ) d^{(t)} d ( t ) is upper bounded. If d ( t ) ≥ 4 / 9 d^{(t)}\geq4/9 d ( t ) ≥ 4/9 , then 4 / 9 − 2 d ( t ) ≤ − d ( t ) 4/9-2d^{(t)}\leq-d^{(t)} 4/9 − 2 d ( t ) ≤ − d ( t ) and so by Equation (5) , d ( t + 1 ) ≤ d ( t ) − α ( t ) w 2 ( t ) d ( t ) ≤ d ( t ) d^{(t+1)}\leq d^{(t)}-\alpha^{(t)}w_2^{(t)}d^{(t)}\leq d^{(t)} d ( t + 1 ) ≤ d ( t ) − α ( t ) w 2 ( t ) d ( t ) ≤ d ( t ) as long as w 2 ( t ) w_2^{(t)} w 2 ( t ) is positive. Hence d ( t ) ≤ max { 4 / 9 , d ( 0 ) } = 4 / 9 d^{(t)}\leq\max\{4/9,d^{(0)}\}=4/9 d ( t ) ≤ max { 4/9 , d ( 0 ) } = 4/9 .
Now whenever d ( t ) ≥ 0 d^{(t)}\geq0 d ( t ) ≥ 0 , we have
d ( t + 1 ) ≥ d ( t ) − ( α ( t ) d ( t ) ) 2 − 2 α ( t ) d ( t ) w 2 ( t ) = d ( t ) ( 1 − d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) ) d^{(t+1)}\geq d^{(t)}-\left(\alpha^{(t)}d^{(t)}\right)^2-2\alpha^{(t)}d^{(t)}w_2^{(t)}=d^{(t)}\left(1-d^{(t)}(\alpha^{(t)})^2-2\alpha^{(t)}w_2^{(t)}\right) d ( t + 1 ) ≥ d ( t ) − ( α ( t ) d ( t ) ) 2 − 2 α ( t ) d ( t ) w 2 ( t ) = d ( t ) ( 1 − d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) )
so it’s enough to show − d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) ≥ − 1 -d^{(t)}(\alpha^{(t)})^2-2\alpha^{(t)}w_2^{(t)}\geq-1 − d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) ≥ − 1 for d ( t ) d^{(t)} d ( t ) to stay non-negative. Note that we can rewrite 2 3 w 1 ( t ) w 2 ( t ) + 1 3 ( w 2 ( t ) ) 3 = − 2 3 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) \frac{2}{3}w_1^{(t)}w_2^{(t)}+\frac{1}{3}(w_2^{(t)})^3=-\frac{2}{3}(w_2^{(t)})^3-d^{(t)}w_2^{(t)} 3 2 w 1 ( t ) w 2 ( t ) + 3 1 ( w 2 ( t ) ) 3 = − 3 2 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) . Let us bound
α ( t ) w 2 ( t ) ≤ η w 2 ( t ) exp ( 2 3 w 1 ( t ) w 2 ( t ) + 1 3 ( w 2 ( t ) ) 3 ) = η w 2 ( t ) exp ( − 2 3 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) ) ≤ η w 2 ( t ) exp ( − 2 3 ( w 2 ( t ) ) 3 ) ≤ 0.6 η (6) \begin{aligned}
\alpha^{(t)}w_2^{(t)}&\leq\eta w_2^{(t)}\exp\left(\frac{2}{3}w_1^{(t)}w_2^{(t)}+\frac{1}{3}(w_2^{(t)})^3\right)\\
&=\eta w_2^{(t)}\exp\left(-\frac{2}{3}(w_2^{(t)})^3-d^{(t)}w_2^{(t)}\right)\\
&\leq\eta w_2^{(t)}\exp\left(-\frac{2}{3}(w_2^{(t)})^3\right)\leq0.6\eta \tag*{(6)}
\end{aligned} α ( t ) w 2 ( t ) ≤ η w 2 ( t ) exp ( 3 2 w 1 ( t ) w 2 ( t ) + 3 1 ( w 2 ( t ) ) 3 ) = η w 2 ( t ) exp ( − 3 2 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) ) ≤ η w 2 ( t ) exp ( − 3 2 ( w 2 ( t ) ) 3 ) ≤ 0.6 η (6)
Where the last inequality is by finding the maximum of w 2 ( t ) exp ( − 2 3 ( w 2 ( t ) ) 3 ) w_{2}^{(t)}\exp\left(-\frac{2}{3}\left(w_{2}^{(t)}\right)^{3}\right) w 2 ( t ) exp ( − 3 2 ( w 2 ( t ) ) 3 ) for w 2 ( t ) ≥ 0 w_{2}^{(t)} \ge0 w 2 ( t ) ≥ 0 . Hence choosing for example η = 0.1 \eta=0.1 η = 0.1 we get
− d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) ≥ − 4 / 9 η 2 − 2 α ( t ) w 2 ( t ) ≥ − 0.1 − 0.2 ≥ − 1 -d^{(t)}\left(\alpha^{(t)}\right)^{2}-2\alpha^{(t)}w_{2}^{(t)} \ge-4/9\eta^{2}-2\alpha^{(t)}w_{2}^{(t)} \ge-0.1-0.2 \ge-1 − d ( t ) ( α ( t ) ) 2 − 2 α ( t ) w 2 ( t ) ≥ − 4/9 η 2 − 2 α ( t ) w 2 ( t ) ≥ − 0.1 − 0.2 ≥ − 1
So we have that d ( t ) ≥ 0 d^{(t)} \ge0 d ( t ) ≥ 0 , and so w 2 ( t ) w_{2}^{(t)} w 2 ( t ) is non-decreasing. We want to show that it’s unbounded. Assume towards contradiction that it is bounded, i.e. w 2 ( t ) ≤ U w_{2}^{(t)} \le U w 2 ( t ) ≤ U , then we have that L ( w ( t ) ) = exp ( − 2 3 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) ) ≥ L inf > 0 \mathcal{L}(\mathbf{w}^{(t)})=\exp\left(-\frac{2}{3}\left(w_{2}^{(t)}\right)^{3}-d^{(t)}w_{2}^{(t)}\right)\ge\mathcal{L}_{\inf}>0 L ( w ( t ) ) = exp ( − 3 2 ( w 2 ( t ) ) 3 − d ( t ) w 2 ( t ) ) ≥ L i n f > 0 . Now
w 1 ( t ) − 2 3 α ( t ) w 2 ( t ) ≤ w 1 ( t ) − 2 3 α ( t ) w_{1}^{(t)}-\frac{2}{3}\alpha^{(t)}w_{2}^{(t)}\le w_{1}^{(t)}-\frac{2}{3}\alpha^{(t)} w 1 ( t ) − 3 2 α ( t ) w 2 ( t ) ≤ w 1 ( t ) − 3 2 α ( t )
Since we start from w 2 ( 0 ) = 1 w_{2}^{(0)}=1 w 2 ( 0 ) = 1 and it’s non-decreasing. Finally, we get
w 1 ( t + 1 ) ≤ w 1 ( t ) − 2 3 η L inf w_{1}^{(t+1)}\le w_{1}^{(t)}-\frac{2}{3}\eta\mathcal{L}_{\inf} w 1 ( t + 1 ) ≤ w 1 ( t ) − 3 2 η L i n f
So w 1 ( t ) → − ∞ w_{1}^{(t)}\to-\infty w 1 ( t ) → − ∞ , and hence d ( t ) = − 2 3 w 1 ( t ) − ( w 2 ( t ) ) 2 → ∞ d^{(t)}=-\frac{2}{3}w_{1}^{(t)}-\left(w_{2}^{(t)}\right)^{2}\to\infty d ( t ) = − 3 2 w 1 ( t ) − ( w 2 ( t ) ) 2 → ∞ , contradicting its upper bound. Hence we have w 2 ( t ) → ∞ w_{2}^{(t)}\to\infty w 2 ( t ) → ∞ , and similarly, L ( w ( t ) ) → 0 \mathcal{L}(\mathbf{w}^{(t)})\to0 L ( w ( t ) ) → 0 .
Now consider the following sequence
e ( t ) = d ( t ) − 2 / 9 e^{(t)}=d^{(t)}-2/9 e ( t ) = d ( t ) − 2/9
We want to show it converges to 0. Note that
e ( t + 1 ) − e ( t ) = − ( α ( t ) d ( t ) ) 2 + α ( t ) w 2 ( t ) ( 4 9 − 2 d ( t ) ) = − ( α ( t ) d ( t ) ) 2 − 2 α ( t ) w 2 ( t ) e ( t ) e^{(t+1)}-e^{(t)}=-\left(\alpha^{(t)}d^{(t)}\right)^{2}+\alpha^{(t)}w_{2}^{(t)}\left(\frac{4}{9}-2d^{(t)}\right)=-\left(\alpha^{(t)}d^{(t)}\right)^{2}-2\alpha^{(t)}w_{2}^{(t)}e^{(t)} e ( t + 1 ) − e ( t ) = − ( α ( t ) d ( t ) ) 2 + α ( t ) w 2 ( t ) ( 9 4 − 2 d ( t ) ) = − ( α ( t ) d ( t ) ) 2 − 2 α ( t ) w 2 ( t ) e ( t )
Hence
e ( t + 1 ) = e ( t ) ( 1 − 2 α ( t ) w 2 ( t ) ) − ( α ( t ) d ( t ) ) 2 e^{(t+1)}=e^{(t)}\left(1-2\alpha^{(t)}w_{2}^{(t)}\right)-\left(\alpha^{(t)}d^{(t)}\right)^{2} e ( t + 1 ) = e ( t ) ( 1 − 2 α ( t ) w 2 ( t ) ) − ( α ( t ) d ( t ) ) 2
Now from Equation (6) , and positivity of w 2 ( t ) w_{2}^{(t)} w 2 ( t ) we have that for large enough t t t , ( 1 − 2 α ( t ) w 2 ( t ) ) ∈ ( 0 , 1 ) \left(1-2\alpha^{(t)}w_{2}^{(t)}\right)\in(0,1) ( 1 − 2 α ( t ) w 2 ( t ) ) ∈ ( 0 , 1 ) . We want to show that ∑ i ( α ( t ) d ( t ) ) 2 < ∞ \sum_i\left(\alpha^{(t)}d^{(t)}\right)^2<\infty ∑ i ( α ( t ) d ( t ) ) 2 < ∞ . Recall that
w 2 ( t + 1 ) − w 2 ( t ) = α ( t ) d ( t ) ≤ 4 9 η exp ( − 2 3 ( w 2 ( t ) ) 3 ) w_{2}^{(t+1)}-w_{2}^{(t)}=\alpha^{(t)}d^{(t)}\le\frac{4}{9}\eta\exp\left(-\frac{2}{3}\left(w_{2}^{(t)}\right)^{3}\right) w 2 ( t + 1 ) − w 2 ( t ) = α ( t ) d ( t ) ≤ 9 4 η exp ( − 3 2 ( w 2 ( t ) ) 3 )
Let I n : = { t : w 2 ( t ) ∈ [ n , n + 1 ] } I_n:=\{t:w_{2}^{(t)}\in[n,n+1]\} I n := { t : w 2 ( t ) ∈ [ n , n + 1 ]} , then for large enough n n n , we have that ∑ t ∈ I n ( w 2 ( t + 1 ) − w 2 ( t ) ) = ∑ t ∈ I n α ( t ) d ( t ) ≤ 2 \sum_{t\in I_n}\left(w_{2}^{(t+1)}-w_{2}^{(t)}\right)=\sum_{t\in I_n}\alpha^{(t)}d^{(t)}\le2 ∑ t ∈ I n ( w 2 ( t + 1 ) − w 2 ( t ) ) = ∑ t ∈ I n α ( t ) d ( t ) ≤ 2 . Moreover, max t ∈ I n α ( t ) d ( t ) ≤ 4 9 η exp ( − 2 3 n 3 ) \max_{t\in I_n}\alpha^{(t)}d^{(t)}\le\frac{4}{9}\eta\exp\left(-\frac{2}{3}n^3\right) max t ∈ I n α ( t ) d ( t ) ≤ 9 4 η exp ( − 3 2 n 3 ) so overall
∑ t ∈ I n ( α ( t ) d ( t ) ) 2 ≤ max t ∈ I n α ( t ) d ( t ) ∑ t ∈ I n ( α ( t ) d ( t ) ) ≤ 8 9 η exp ( − 2 3 n 3 ) \sum_{t\in I_n}\left(\alpha^{(t)}d^{(t)}\right)^2\le\max_{t\in I_n}\alpha^{(t)}d^{(t)}\sum_{t\in I_n}\left(\alpha^{(t)}d^{(t)}\right)\le\frac{8}{9}\eta\exp\left(-\frac{2}{3}n^3\right) t ∈ I n ∑ ( α ( t ) d ( t ) ) 2 ≤ t ∈ I n max α ( t ) d ( t ) t ∈ I n ∑ ( α ( t ) d ( t ) ) ≤ 9 8 η exp ( − 3 2 n 3 )
and overall
∑ t ( α ( t ) d ( t ) ) 2 = ∑ n ∑ t ∈ I n ( α ( t ) d ( t ) ) 2 ≤ ∑ n 8 9 η exp ( − 2 3 n 3 ) < ∞ \sum_t\left(\alpha^{(t)}d^{(t)}\right)^2=\sum_n\sum_{t\in I_n}\left(\alpha^{(t)}d^{(t)}\right)^2\le\sum_n\frac{8}{9}\eta\exp\left(-\frac{2}{3}n^3\right)<\infty t ∑ ( α ( t ) d ( t ) ) 2 = n ∑ t ∈ I n ∑ ( α ( t ) d ( t ) ) 2 ≤ n ∑ 9 8 η exp ( − 3 2 n 3 ) < ∞
Note that since − ( w 2 ( t ) ) 2 − 2 3 w 1 ( t ) = d ( t ) ≥ 0 -\left(w_{2}^{(t)}\right)^2-\frac{2}{3}w_{1}^{(t)}=d^{(t)}\ge0 − ( w 2 ( t ) ) 2 − 3 2 w 1 ( t ) = d ( t ) ≥ 0 we have that w 1 ( t ) → − ∞ w_{1}^{(t)}\to-\infty w 1 ( t ) → − ∞ . And
w 1 ( t ) = w 1 ( 0 ) − 2 3 ∑ k = 0 t − 1 α ( k ) w 2 ( k ) → − ∞ w_{1}^{(t)}=w_{1}^{(0)}-\frac{2}{3}\sum_{k=0}^{t-1}\alpha^{(k)}w_{2}^{(k)}\to-\infty w 1 ( t ) = w 1 ( 0 ) − 3 2 k = 0 ∑ t − 1 α ( k ) w 2 ( k ) → − ∞
So overall ∑ k = 0 t − 1 α ( k ) w 2 ( k ) → ∞ \sum_{k=0}^{t-1}\alpha^{(k)}w_{2}^{(k)}\to\infty ∑ k = 0 t − 1 α ( k ) w 2 ( k ) → ∞ . Now let us define e ( t ) : = d ( t ) − 2 / 9 e^{(t)}:=d^{(t)}-2/9 e ( t ) := d ( t ) − 2/9 , we want to use Theorem F.4 to show e ( t ) → 0 e^{(t)}\to0 e ( t ) → 0 . We have
e ( t + 1 ) = e ( t ) ( 1 − 2 α ( t ) w 2 ( t ) ) − ( α ( t ) d ( t ) ) 2 e^{(t+1)}=e^{(t)}\left(1-2\alpha^{(t)}w_{2}^{(t)}\right)-\left(\alpha^{(t)}d^{(t)}\right)^2 e ( t + 1 ) = e ( t ) ( 1 − 2 α ( t ) w 2 ( t ) ) − ( α ( t ) d ( t ) ) 2
so indeed using Theorem F.4 we get d ( t ) → 2 / 9 d^{(t)} \to2/9 d ( t ) → 2/9 . But now
w 1 ( t ) = − 3 2 ( w 2 ( t ) ) 2 − 1 / 3 + o ( 1 ) w_1^{(t)}=-\frac{3}{2}\left(w_2^{(t)}\right)^2-1/3+o(1) w 1 ( t ) = − 2 3 ( w 2 ( t ) ) 2 − 1/3 + o ( 1 )
and so
2 3 ( w 2 ( t ) ) 3 + d ( t ) w 2 ( t ) = Θ ( ( w 2 ( t ) ) 3 ) \frac{2}{3}\left(w_2^{(t)}\right)^3+d^{(t)}w_2^{(t)}=\Theta\left(\left(w_2^{(t)}\right)^3\right) 3 2 ( w 2 ( t ) ) 3 + d ( t ) w 2 ( t ) = Θ ( ( w 2 ( t ) ) 3 )
but ∥ w ( t ) ∥ = Θ ( ( w 2 ( t ) ) 2 ) \lVert w^{(t)}\rVert=\Theta\left(\left(w_2^{(t)}\right)^2\right) ∥ w ( t ) ∥ = Θ ( ( w 2 ( t ) ) 2 ) , so the margin is Θ ( ∥ w ( t ) ∥ 3 / 2 ) \Theta\left(\lVert w^{(t)}\rVert^{3/2}\right) Θ ( ∥ w ( t ) ∥ 3/2 )
Proof of Theorem 4.4
Theorem C.1 implies Assumption 1. Theorem 6.2 in Cai et al. [2 ] (Theorem F.2) implies Assumption 2.
For the case where q q q is M M M -homogeneous these two imply Assumption 3 immediately by homogeneity.
For the case where q q q is not homogeneous and deg ( p ) ≥ 1 \deg(p) \ge1 deg ( p ) ≥ 1 let us bound the loss explicitly:
L ( w ( t ) ) = ∑ i = 1 N exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ≥ max i ∈ [ N ] exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) (7) \begin{aligned}
\mathcal{L}(w^{(t)})=\sum_{i=1}^{N}\exp\left(-y_i\langle q(w^{(t)}),x_i\rangle\right) \\
&\ge\max_{i\in[N]}\exp\left(-y_i\langle q(w^{(t)}),x_i\rangle\right) \tag*{(7)}
\end{aligned} L ( w ( t ) ) = i = 1 ∑ N exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ≥ i ∈ [ N ] max exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) (7)
We have the margin as defined in Cai et al. [2 ] is,
γ ( t ) : = exp ( Φ ( exp ( p ( ∥ w ( t ) ∥ ) ) L ( w ( t ) ) / N ) ) ∥ w ( t ) ∥ M \gamma(t):=\frac{\exp\left(\Phi\left(\exp\left(p\left(\lVert w^{(t)}\rVert\right)\right)\mathcal{L}\left(w^{(t)}\right)/N\right)\right)}{\lVert w^{(t)}\rVert^M} γ ( t ) := ∥ w ( t ) ∥ M exp ( Φ ( exp ( p ( ∥ w ( t ) ∥ ) ) L ( w ( t ) ) / N ) )
Where M = deg ( q ) M=\deg(q) M = deg ( q ) , and Φ ( x ) = log ( log ( 1 / N x ) ) + 2 / log ( N x ) \Phi(x)=\log(\log(1/Nx))+2/\log(Nx) Φ ( x ) = log ( log ( 1/ N x )) + 2/ log ( N x ) . Since exp ∘ Φ ( x ) \exp\circ\Phi(x) exp ∘ Φ ( x ) is monotonous decreasing, using Equation (7) and for large enough t t t s.t. p ( ∥ w ( t ) ∥ ) ≥ 1 p\left(\lVert w^{(t)}\rVert\right)\ge1 p ( ∥ w ( t ) ∥ ) ≥ 1 ,
γ ( t ) ≤ exp ( Φ ( max i ∈ [ N ] exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) / N ) ) ∥ w ( t ) ∥ M = exp ( log log ( 1 max i ∈ [ N ] exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) + 2 log ( max i ∈ [ N ] exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) ) ∥ w ( t ) ∥ M = exp ( log ( min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ ) + 2 max i ∈ [ N ] ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) ∥ w ( t ) ∥ M ≤ exp ( log ( min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) ∥ w ( t ) ∥ M (8) \begin{aligned}
\gamma(t)\le\frac{\exp\left(\Phi\left(\max_{i\in[N]}\exp\left(-y_i\langle q(w^{(t)}),x_i\rangle\right)/N\right)\right)}{\lVert w^{(t)}\rVert^M} \\
&=\frac{\exp\left(\log\log\left(\frac{1}{\max_{i\in[N]}\exp\left(-y_i\langle q(w^{(t)}),x_i\rangle\right)}\right)+\frac{2}{\log\left(\max_{i\in[N]}\exp\left(-y_i\langle q(w^{(t)}),x_i\rangle\right)\right)}\right)}{\lVert w^{(t)}\rVert^M} \\
&=\frac{\exp\left(\log\left(\min_{i\in[N]}y_i\langle q(w^{(t)}),x_i\rangle\right)+\frac{2}{\max_{i\in[N]}\left(-y_i\langle q(w^{(t)}),x_i\rangle\right)}\right)}{\lVert w^{(t)}\rVert^M} \\
&\le\frac{\exp\left(\log\left(\min_{i\in[N]}y_i\langle q(w^{(t)}),x_i\rangle\right)\right)}{\lVert w^{(t)}\rVert^M} \tag*{(8)}
\end{aligned} γ ( t ) ≤ ∥ w ( t ) ∥ M exp ( Φ ( max i ∈ [ N ] exp ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) / N ) ) = ∥ w ( t ) ∥ M exp ( log log ( m a x i ∈ [ N ] e x p ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) 1 ) + l o g ( m a x i ∈ [ N ] e x p ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) 2 ) = ∥ w ( t ) ∥ M exp ( log ( min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ ) + m a x i ∈ [ N ] ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) 2 ) ≤ ∥ w ( t ) ∥ M exp ( log ( min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ ) ) (8)
= min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ ∥ w ( t ) ∥ M (9) =\frac{\min_{i\in[N]}y_i\langle q(w^{(t)}),x_i\rangle}{\lVert w^{(t)}\rVert^M} \tag*{(9)} = ∥ w ( t ) ∥ M min i ∈ [ N ] y i ⟨ q ( w ( t ) ) , x i ⟩ (9)
Where Equation (8) is due to 2 max i ∈ [ N ] ( − y i ⟨ q ( w ( t ) ) , x i ⟩ ) \frac{2}{\max_{i\in[N]}(-y_i\langle q(w^{(t)}),x_i\rangle)} m a x i ∈ [ N ] ( − y i ⟨ q ( w ( t ) ) , x i ⟩) 2 being negative for large enough t t t . Taking the limit on both sides of Equation (9) , and using Theorem F.1 showing that γ ( t ) \gamma(t) γ ( t ) is positive and increasing for large enough t t t , and since we have that deg ( q ) = M \deg(q)=M deg ( q ) = M , then the normalized margin converges to some positive constant. For simplicity let us denote the margin polynomials m i ( w ) : = y i ⟨ q ( w ( t ) ) , x i ⟩ m_i(w):=y_i\langle q(w^{(t)}),x_i\rangle m i ( w ) := y i ⟨ q ( w ( t ) ) , x i ⟩ , then using homogeneity we have
m ( w ( t ) ) ∥ w ( t ) ∥ M = ∑ i = 0 M ∥ w ( t ) ∥ i − M m i ( w ( t ) ∥ w ( t ) ∥ ) \frac{m(w^{(t)})}{\lVert w^{(t)}\rVert^M}=\sum_{i=0}^{M}\lVert w^{(t)}\rVert^{i-M}m^i\left(\frac{w^{(t)}}{\lVert w^{(t)}\rVert}\right) ∥ w ( t ) ∥ M m ( w ( t ) ) = i = 0 ∑ M ∥ w ( t ) ∥ i − M m i ( ∥ w ( t ) ∥ w ( t ) )
Taking the limit, one summand must not go to zero (because we know the limit is positive), hence it must be the last one, and we get lim t → ∞ m ( w ( t ) ) ∥ w ( t ) ∥ M = m M ( w ‾ ∞ ) > 0 \lim_{t\to\infty} \frac{m^{(w(t))}}{\lVert w^{(t)}\rVert^{M}}=m^{M}(\overline{w}^{\infty})>0 lim t → ∞ ∥ w ( t ) ∥ M m ( w ( t )) = m M ( w ∞ ) > 0 . Similarly, from homogeneity we get that ⟨ w ‾ ∞ , ∇ m M ( w ‾ ∞ ) ⟩ = M m M ( w ‾ ∞ ) > 0 \langle\overline{w}^{\infty},\nabla m^{M}(\overline{w}^{\infty})\rangle=Mm^{M}(\overline{w}^{\infty})>0 ⟨ w ∞ , ∇ m M ( w ∞ )⟩ = M m M ( w ∞ ) > 0 , hence ∇ m M − 1 ( w ‾ ∞ ) ≠ 0 \nabla m^{M-1}(\overline{w}^{\infty})\ne0 ∇ m M − 1 ( w ∞ ) = 0 , and must dominate the gradient. Thus, we have concluded Assumption 3.
For the case where q q q is not homogeneous and deg ( p ) = 0 \deg(p)=0 deg ( p ) = 0 We have that p ( ∥ w ( t ) ∥ ) p(\lVert w^{(t)}\rVert) p (∥ w ( t ) ∥) is a constant and hence
γ ( t ) : = exp ( Φ ( C L ( w ( t ) ) / N ) ) ∥ w ( t ) ∥ M \gamma(t):=\frac{\exp\left(\Phi\left(C\mathcal{L}(w^{(t)})/N\right)\right)}{\lVert w^{(t)}\rVert^{M}} γ ( t ) := ∥ w ( t ) ∥ M exp ( Φ ( C L ( w ( t ) ) / N ) )
For some C > 0 C>0 C > 0 . The rest of the argument follows exactly as above, with a trailing constant C C C , that does not change the sign of the limits, leading to the same conclusion.
Multidimensional Tokens
Linear Autoregressive Models
Given an example X ∈ R l × D X\in\mathbb{R}^{l\times D} X ∈ R l × D which we interpret as the initial part of some sequence X = X [ 1 : D ] = ( x 1 , … , x D ) X=X_{[1:D]}=(x_1,\ldots,x_D) X = X [ 1 : D ] = ( x 1 , … , x D ) (i.e x i x_i x i denote the i t h i^{th} i t h column of X X X ), we compute a sequence of T T T autoregressive steps, the final’s sign being our label. Formally, the first step will be x D + 1 = X w x_{D+1}=Xw x D + 1 = X w . The second step x D + 2 = X 2 : D + 1 w x_{D+2}=X_{2:D+1}w x D + 2 = X 2 : D + 1 w , and at the T t h T^{th} T t h step we’ll have x D + T = X T : D + T − 1 w x_{D+T}=X_{T:D+T-1}w x D + T = X T : D + T − 1 w . For example, let D = 2 D=2 D = 2 and let us consider two autoregressive steps (T = 2 T=2 T = 2 ). We have that
x 3 = X w = ( w 1 x 1 , 1 + w 2 x 1 , 2 w 1 x 2 , 1 + w 2 x 2 , 2 ) x_3=Xw=\begin{pmatrix}w_1x_{1,1}+w_2x_{1,2}\\w_1x_{2,1}+w_2x_{2,2}\end{pmatrix} x 3 = X w = ( w 1 x 1 , 1 + w 2 x 1 , 2 w 1 x 2 , 1 + w 2 x 2 , 2 )
x 4 = X [ 2 : D + T − 1 ] w = X [ 2 : 3 ] w = ( w 1 x 1 , 2 + w 2 x 1 , 3 w 1 x 2 , 2 + w 2 x 2 , 3 ) = ( w 1 w 2 x 1 , 1 + ( w 1 + w 2 2 ) x 1 , 2 w 1 w 2 x 2 , 1 + ( w 1 + w 2 2 ) x 2 , 2 ) x_4=X_{[2:D+T-1]}w=X_{[2:3]}w
=\begin{pmatrix}w_1x_{1,2}+w_2x_{1,3}\\w_1x_{2,2}+w_2x_{2,3}\end{pmatrix}
=\begin{pmatrix}w_1w_2x_{1,1}+(w_1+w_2^2)x_{1,2}\\w_1w_2x_{2,1}+(w_1+w_2^2)x_{2,2}\end{pmatrix} x 4 = X [ 2 : D + T − 1 ] w = X [ 2 : 3 ] w = ( w 1 x 1 , 2 + w 2 x 1 , 3 w 1 x 2 , 2 + w 2 x 2 , 3 ) = ( w 1 w 2 x 1 , 1 + ( w 1 + w 2 2 ) x 1 , 2 w 1 w 2 x 2 , 1 + ( w 1 + w 2 2 ) x 2 , 2 )
Note that in contrast to § 3.2 , the resulting x 4 x_4 x 4 cannot serve as a label by itself, and hence we must fix, or learn a projection vector p ∈ R l p\in\mathbb{R}^{l} p ∈ R l , s.t. y = sgn ( ⟨ p , x 4 ⟩ ) y=\operatorname{sgn}(\langle p,x_4\rangle) y = sgn (⟨ p , x 4 ⟩) . We denote the vector of coefficients of x 1 , … , x D x_1,\ldots,x_D x 1 , … , x D in x D + T x_{D+T} x D + T as P \mathcal{P} P , a polynomial in w w w , and optionally p p p . For our running example that vector would be P ( w , p ) = ( w 1 w 2 p 1 w 1 p 1 + w 2 2 p 1 w 1 w 2 p 2 w 1 p 2 + w 2 2 p 2 ) \mathcal{P}(w,p)=\begin{pmatrix}w_1w_2p_1&w_1p_1+w_2^2p_1\\w_1w_2p_2&w_1p_2+w_2^2p_2\end{pmatrix} P ( w , p ) = ( w 1 w 2 p 1 w 1 w 2 p 2 w 1 p 1 + w 2 2 p 1 w 1 p 2 + w 2 2 p 2 ) . Using this notation, we have that ⟨ P ( w , p ) , X ⟩ = x D + T \langle\mathcal{P}(w,p),X\rangle=x_{D+T} ⟨ P ( w , p ) , X ⟩ = x D + T , and in our example x 4 = Tr ( ( w 1 w 2 p 1 w 1 p 1 + w 2 2 p 1 w 1 w 2 p 2 w 1 p 2 + w 2 2 p 2 ) ( x 1 , 1 x 1 , 2 x 2 , 1 x 2 , 2 ) T ) x_4=\operatorname{Tr}\left(\begin{pmatrix}w_1w_2p_1&w_1p_1+w_2^2p_1\\w_1w_2p_2&w_1p_2+w_2^2p_2\end{pmatrix}\begin{pmatrix}x_{1,1}&x_{1,2}\\x_{2,1}&x_{2,2}\end{pmatrix}^{\mathsf{T}}\right) x 4 = Tr ( ( w 1 w 2 p 1 w 1 w 2 p 2 w 1 p 1 + w 2 2 p 1 w 1 p 2 + w 2 2 p 2 ) ( x 1 , 1 x 2 , 1 x 1 , 2 x 2 , 2 ) T ) .
Note that for a dataset { X i , y i } i = 1 N \{X_i,y_i\}_{i=1}^{N} { X i , y i } i = 1 N we denote the polynomial corresponding to the i t h i^{th} i t h -example as P i \mathcal{P}_i P i , but in this autoregressive setting, it is the case that P i = P j \mathcal{P}_i=\mathcal{P}_j P i = P j for i ≠ j i\ne j i = j , if both examples i , j i,j i , j are labeled by the same number T T T of autoregressive steps. We state the theorem for the case of learning p p p , but it can easily be amended for fixed p p p .
Theorem B.1 (Implicit Bias of Multidimensional Linear Autoregressive Models). For parameters w ∈ R D w\in\mathbb{R}^{D} w ∈ R D and p ∈ R l p\in\mathbb{R}^{l} p ∈ R l denote θ : = ( w , p ) \theta:=(w,p) θ := ( w , p ) , let { X i , y i } i = 1 N \{X_i,y_i\}_{i=1}^{N} { X i , y i } i = 1 N , where y i = sgn ⟨ P i ( θ ) , X i ⟩ y_i=\operatorname{sgn}\langle\mathcal{P}_i(\theta),X_i\rangle y i = sgn ⟨ P i ( θ ) , X i ⟩ . Consider the sequence of gradient updates according to Equation (1) , with any non-increasing sequence of step sizes { η t } \{\eta_t\} { η t } , minimizing the empirical risk. If Assumptions 1 1 1 to 3 3 3 , then θ ˉ ∞ : = lim t → ∞ θ ( t ) ∥ θ ( t ) ∥ \bar{\theta}^{\infty}:=\lim_{t\to\infty}\frac{\theta^{(t)}}{\lVert\theta^{(t)}\rVert} θ ˉ ∞ := lim t → ∞ ∥ θ ( t ) ∥ θ ( t ) is a positive scaling of a first order stationary point of the following optimization problem:
min θ ∈ R D + l ∥ θ ∥ 2 2 s.t. ∀ i , y i ⟨ X i , P i V ( θ ) ⟩ ≥ 0 \min_{\theta\in\mathbb{R}^{D+l}}\lVert\theta\rVert_2^2\quad\text{s.t.}\ \forall i,\ y_i\langle X_i,\mathcal{P}_i^{\mathcal{V}}(\theta)\rangle\ge0 θ ∈ R D + l min ∥ θ ∥ 2 2 s.t. ∀ i , y i ⟨ X i , P i V ( θ )⟩ ≥ 0 ∀ i ∈ Z , y i C i ⟨ X i , P i V ( θ ) ⟩ ≥ 1 \forall i\in Z,\ y_i C_i\langle X_i,\mathcal{P}_i^{\mathcal{V}}(\theta)\rangle\ge1 ∀ i ∈ Z , y i C i ⟨ X i , P i V ( θ )⟩ ≥ 1 where Z = arg min i ∈ [ N ] { effdeg θ ( ⟨ P i , x i y i ⟩ ) } Z=\operatorname*{arg\,min}_{i\in[N]}\{\operatorname{effdeg}_{\theta}(\langle\mathcal{P}_i,x_i y_i\rangle)\} Z = arg min i ∈ [ N ] { effdeg θ (⟨ P i , x i y i ⟩)} .
Proof. The proof follows immediately from Theorem 5.1 , by concatenating θ = ( w , p ) \theta=(\mathbf{w},\mathbf{p}) θ = ( w , p ) into one vector, concatenating each X i X_i X i ’s columns into a vector v i \mathbf{v}_i v i , and concatenating P i \mathcal{P}_i P i ’s columns into a vector P ~ i \widetilde{\mathcal{P}}_i P i the same way, such that they match and then ⟨ v i , P ~ i ( θ ) ⟩ = ⟨ X i , P i ( w , p ) ⟩ \langle\mathbf{v}_i,\widetilde{\mathcal{P}}_i(\theta)\rangle=\langle X_i,\mathcal{P}_i(\mathbf{w},\mathbf{p})\rangle ⟨ v i , P i ( θ )⟩ = ⟨ X i , P i ( w , p )⟩ . □
Autoregressive SSMs
Here we consider the full Autoregressive SSMs, as it subsumes the simpler SSM case. We repeat the notation for convenience. We denote the parameters of the model by A ∈ R d A × d A A\in\mathbb{R}^{d_A\times d_A} A ∈ R d A × d A , B ∈ R d A × d B\in\mathbb{R}^{d_A\times d} B ∈ R d A × d , C ∈ R d A × d C\in\mathbb{R}^{d_A\times d} C ∈ R d A × d . Consider an example X ∈ R d × l X\in\mathbb{R}^{d\times l} X ∈ R d × l , that we can think of as an input prompt, or as a beginning of some sequence X = X 1 : l = ( x 1 , 1 ⋯ x 1 , l ⋮ ⋱ ⋮ x d , 1 ⋯ x d , l ) X=X_{1:l}=\begin{pmatrix}x_{1,1}&\cdots&x_{1,l}\\ \vdots&\ddots&\vdots\\ x_{d,1}&\cdots&x_{d,l}\end{pmatrix} X = X 1 : l = x 1 , 1 ⋮ x d , 1 ⋯ ⋱ ⋯ x 1 , l ⋮ x d , l , we compute a sequence of T T T autoregressive steps:
( X ) l + 1 = C T ∑ i = 1 l A l − i B ( X ) i , ( X ) l + 2 = C T ∑ i = 1 l + 1 A l + 1 − i B ( X ) i , … , ( X ) l + T = C T ∑ i = 1 l + T − 1 A l + T − 1 − i B ( X ) i . (10) (X)_{l+1}=C^{\mathsf{T}}\sum_{i=1}^{l}A^{l-i}B(X)_i,\qquad(X)_{l+2}=C^{\mathsf{T}}\sum_{i=1}^{l+1}A^{l+1-i}B(X)_i,\qquad\ldots,\qquad(X)_{l+T}=C^{\mathsf{T}}\sum_{i=1}^{l+T-1}A^{l+T-1-i}B(X)_i. \tag*{(10)} ( X ) l + 1 = C T i = 1 ∑ l A l − i B ( X ) i , ( X ) l + 2 = C T i = 1 ∑ l + 1 A l + 1 − i B ( X ) i , … , ( X ) l + T = C T i = 1 ∑ l + T − 1 A l + T − 1 − i B ( X ) i . (10)
Where ( X ) i (X)_i ( X ) i denotes the i t h i^{\mathrm{th}} i th column of $X. The label of the sequence x x x is defined to be sgn ( ⟨ x l + T , p ⟩ ) \operatorname{sgn}(\langle x_{l+T},p\rangle) sgn (⟨ x l + T , p ⟩) , where P \mathcal{P} P is some projection vector that is required in order to get a scalar prediction. This vector may be learned or fixed. We phrase the theorem for the general case, where P \mathcal{P} P can be learned.
Theorem B.2 (Implicit bias of SSMs). For parameters A ∈ R d A × d A A\in\mathbb{R}^{d_A\times d_A} A ∈ R d A × d A , B ∈ R d A × d B\in\mathbb{R}^{d_A\times d} B ∈ R d A × d , C ∈ R d A × d C\in\mathbb{R}^{d_A\times d} C ∈ R d A × d and p ∈ R d p\in\mathbb{R}^{d} p ∈ R d denoted for simplicity by w : = ( A , B , C , p ) \mathbf{w}:=(A,B,C,p) w := ( A , B , C , p ) , let { X i , y i } i = 1 n \{X_i,y_i\}_{i=1}^{n} { X i , y i } i = 1 n , where y i = sgn ⟨ Q i ( w ) , X i ⟩ y_i=\operatorname{sgn}\langle Q_i(\mathbf{w}),X_i\rangle y i = sgn ⟨ Q i ( w ) , X i ⟩ . Consider the sequence of gradient updates according to Equation (1) , with a non-increasing sequence of step sizes { η t } \{\eta_t\} { η t } , minimizing the empirical risk. If Assumptions 1 to 3, w ˉ ∞ : = lim t → ∞ w ( t ) ∥ w ( t ) ∥ \bar{\mathbf{w}}^\infty:=\lim_{t\to\infty}\frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert} w ˉ ∞ := lim t → ∞ ∥ w ( t ) ∥ w ( t ) is a positive scaling of a first order stationary point of the following optimization problem:
min w ∈ R d A × d A + 2 d A × d + d ∥ w ∥ 2 s.t. ∀ i , y i ⟨ X i , Q i ν i ( w ) ⟩ ≥ 0 \min_{\mathbf{w}\in\mathbb{R}^{d_A\times d_A+2d_A\times d+d}}\lVert\mathbf{w}\rVert^2\quad\text{s.t.}\quad\forall i,\ y_i\langle X_i,Q_i^{\nu_i}(\mathbf{w})\rangle\ge0 w ∈ R d A × d A + 2 d A × d + d min ∥ w ∥ 2 s.t. ∀ i , y i ⟨ X i , Q i ν i ( w )⟩ ≥ 0 ∀ i ∈ Z , y i C i ⟨ X i , Q i ν i ( w ) ⟩ ≥ 1 \forall i\in Z,\ y_iC_i\langle X_i,Q_i^{\nu_i}(\mathbf{w})\rangle\ge1 ∀ i ∈ Z , y i C i ⟨ X i , Q i ν i ( w )⟩ ≥ 1 where Z = arg min i ∈ [ n ] { effdeg w ( ⟨ Q i , x i ⟩ ) } Z=\arg\min_{i\in[n]}\{\operatorname{effdeg}_{\mathbf{w}}(\langle Q_i,x_i\rangle)\} Z = arg min i ∈ [ n ] { effdeg w (⟨ Q i , x i ⟩)} .
Proof. The proof follows immediately from Theorem 5.1 , by concatenating w = ( A , B , C , p ) \mathbf{w}=(A,B,C,p) w = ( A , B , C , p ) into one vector, and noting that since at each step we only multiply by the parameters and not by X i X_i X i , we can write a coefficient vector for every x i , j x_{i,j} x i , j in terms of A A A , B B B , C C C and p p p only, then using that vector as our polynomial, and concatenating each X i X_i X i to a long vector respecting these coefficients. □
Proof of Theorem 4.3
Consider the following simple toy example in 4D: Take one example x = ( 0 , 0 , 0 , 1 ) x=(0,0,0,1) x = ( 0 , 0 , 0 , 1 ) , y = 1 y=1 y = 1 with 4-autoregressive steps. We have that
x 5 = w 1 x 1 + w 2 x 2 + w 3 x 3 + w 4 x 4 x_5=w_1x_1+w_2x_2+w_3x_3+w_4x_4 x 5 = w 1 x 1 + w 2 x 2 + w 3 x 3 + w 4 x 4
x 6 = w 1 x 2 + w 2 x 3 + w 3 x 4 + w 4 x 5 = w 3 + w 4 2 x_6=w_1x_2+w_2x_3+w_3x_4+w_4x_5=w_3+w_4^2 x 6 = w 1 x 2 + w 2 x 3 + w 3 x 4 + w 4 x 5 = w 3 + w 4 2
x 7 = w 1 x 3 + w 2 x 4 + w 3 x 5 + w 4 x 6 = w 2 + 2 w 3 w 4 + w 4 3 x_7=w_1x_3+w_2x_4+w_3x_5+w_4x_6=w_2+2w_3w_4+w_4^3 x 7 = w 1 x 3 + w 2 x 4 + w 3 x 5 + w 4 x 6 = w 2 + 2 w 3 w 4 + w 4 3
x 8 = w 1 x 4 + w 2 x 5 + w 3 x 6 + w 4 x 7 = w 1 + 2 w 2 w 4 + w 3 2 + 3 w 3 w 4 2 + w 4 4 x_8=w_1x_4+w_2x_5+w_3x_6+w_4x_7=w_1+2w_2w_4+w_3^2+3w_3w_4^2+w_4^4 x 8 = w 1 x 4 + w 2 x 5 + w 3 x 6 + w 4 x 7 = w 1 + 2 w 2 w 4 + w 3 2 + 3 w 3 w 4 2 + w 4 4
So we have that
L ( w ) = exp ( − w 1 − 2 w 2 w 4 − w 3 2 − 3 w 3 w 4 2 − w 4 4 ) \mathcal{L}(\mathbf{w})=\exp(-w_1-2w_2w_4-w_3^2-3w_3w_4^2-w_4^4) L ( w ) = exp ( − w 1 − 2 w 2 w 4 − w 3 2 − 3 w 3 w 4 2 − w 4 4 )
The idea behind this toy example is that
− ∇ L ( w ) = L ( w ) ⋅ ( 1 2 w 4 2 w 3 + 3 w 4 2 2 w 2 + 6 w 3 w 4 + 4 w 4 3 ) -\nabla\mathcal{L}(\mathbf{w})=\mathcal{L}(\mathbf{w})\cdot\begin{pmatrix}1\\2w_4\\2w_3+3w_4^2\\2w_2+6w_3w_4+4w_4^3\end{pmatrix} − ∇ L ( w ) = L ( w ) ⋅ 1 2 w 4 2 w 3 + 3 w 4 2 2 w 2 + 6 w 3 w 4 + 4 w 4 3
Hence whenever w 2 = w 3 = w 4 = 0 w_{2}=w_{3}=w_{4}=0 w 2 = w 3 = w 4 = 0 we get that
− ∇ L ( w ) = L ( w ) ⋅ ( 1 0 0 0 ) -\nabla\mathcal{L}(\mathbf{w})=\mathcal{L}(\mathbf{w})\cdot\begin{pmatrix}1\\0\\0\\0\end{pmatrix} − ∇ L ( w ) = L ( w ) ⋅ 1 0 0 0
That is, w 2 , w 3 , w 4 w_{2},w_{3},w_{4} w 2 , w 3 , w 4 remain 0. This produces the dynamic that while deg ( P ( w ) ⊤ x y ) = 4 \deg(\mathcal{P}(\mathbf{w})^{\top}\mathbf{x}y)=4 deg ( P ( w ) ⊤ x y ) = 4 , we have that effdeg w ( P ( w ) ⊤ x y ) = 1 \operatorname{effdeg}_{\mathbf{w}}(\mathcal{P}(\mathbf{w})^{\top}\mathbf{x}y)=1 effdeg w ( P ( w ) ⊤ x y ) = 1 . Effectively the higher parts are not used during optimization.
We will first show why the assumptions of Cai Cai et al. [2 ] fail, and proceed to show why our assumptions hold.
Proof of lemma:bartlettfails
Proof of Theorem 4.3
Let us first restate the assumption:
Assumption 5 (Cai Cai et al. [2 ] , simplified for our example). At some point t 0 t_{0} t 0 during optimization, L ( w ( t 0 ) ) ≤ exp ( − p ( ∥ w ( t 0 ) ∥ ) ) \mathcal{L}(\mathbf{w}^{(t_{0})})\leq\exp\left(-p\left(\left\|\mathbf{w}^{(t_{0})}\right\|\right)\right) L ( w ( t 0 ) ) ≤ exp ( − p ( w ( t 0 ) ) ) for some p ( x ) p(x) p ( x ) . If the predictor is a polynomial of the form f ( w ; x ) = ⟨ q ( w ) , x ⟩ f(\mathbf{w};\mathbf{x})=\langle q(\mathbf{w}),\mathbf{x}\rangle f ( w ; x ) = ⟨ q ( w ) , x ⟩ , then deg ( p ) = deg ( q ) − 1 \deg(p)=\deg(q)-1 deg ( p ) = deg ( q ) − 1 .
They later use this assumption to prove the following:
Lemma C.1 (Cai Cai et al. [2 ] , simplified). ∀ t ≥ t 0 \forall t\geq t_{0} ∀ t ≥ t 0 we have that L ( w ( t ) ) ≤ exp ( − p ( ∥ w ( t ) ∥ ) ) \mathcal{L}(\mathbf{w}^{(t)})\leq\exp\left(-p\left(\left\|\mathbf{w}^{(t)}\right\|\right)\right) L ( w ( t ) ) ≤ exp ( − p ( w ( t ) ) )
Let us analyze the example in light of Assumption 4. First, recall that
L ( w ) = exp ( − w 1 − 2 w 2 w 4 − w 3 2 − 3 w 3 w 4 2 − w 4 4 ) \mathcal{L}(\mathbf{w})=\exp(-w_{1}-2w_{2}w_{4}-w_{3}^{2}-3w_{3}w_{4}^{2}-w_{4}^{4}) L ( w ) = exp ( − w 1 − 2 w 2 w 4 − w 3 2 − 3 w 3 w 4 2 − w 4 4 )
and
− ∇ L ( w ) = L ( w ) ⋅ ( 1 2 w 4 2 w 3 + 3 w 4 2 2 w 2 + 6 w 3 w 4 + 4 w 4 3 ) -\nabla\mathcal{L}(\mathbf{w})=\mathcal{L}(\mathbf{w})\cdot\begin{pmatrix}1\\2w_{4}\\2w_{3}+3w_{4}^{2}\\2w_{2}+6w_{3}w_{4}+4w_{4}^{3}\end{pmatrix} − ∇ L ( w ) = L ( w ) ⋅ 1 2 w 4 2 w 3 + 3 w 4 2 2 w 2 + 6 w 3 w 4 + 4 w 4 3
Initializing at w ( 0 ) = ( 0 0 0 0 ) \mathbf{w}^{(0)}=\begin{pmatrix}0\\0\\0\\0\end{pmatrix} w ( 0 ) = 0 0 0 0 and whenever w 2 = w 3 = w 4 = 0 w_{2}=w_{3}=w_{4}=0 w 2 = w 3 = w 4 = 0 , we get
− ∇ L ( w ) = L ( w ) ⋅ ( 1 0 0 0 ) (11) -\nabla\mathcal{L}(\mathbf{w})=\mathcal{L}(\mathbf{w})\cdot\begin{pmatrix}1\\0\\0\\0\end{pmatrix} \tag*{(11)} − ∇ L ( w ) = L ( w ) ⋅ 1 0 0 0 (11)
So assuming a constant step size η \eta η ,
w 1 ( t + 1 ) = w 1 ( t ) + η L ( w ( t ) ) w j ( t + 1 ) = w j ( t ) + 0 j ∈ { 2 , 3 , 4 } (12) \begin{aligned}
w_{1}^{(t+1)}&=w_{1}^{(t)}+\eta\mathcal{L}(\mathbf{w}^{(t)})\\
w_{j}^{(t+1)}&=w_{j}^{(t)}+0 \quad j\in\{2,3,4\} \tag*{(12)}
\end{aligned} w 1 ( t + 1 ) w j ( t + 1 ) = w 1 ( t ) + η L ( w ( t ) ) = w j ( t ) + 0 j ∈ { 2 , 3 , 4 } (12)
Where the updates follow from inductively w 2 ( t ) = w 3 ( t ) = w 4 ( t ) = 0 w_{2}^{(t)}=w_{3}^{(t)}=w_{4}^{(t)}=0 w 2 ( t ) = w 3 ( t ) = w 4 ( t ) = 0 staying zero for any t t t . This means that during training we have:
∥ w ( t ) ∥ = ∣ w 1 ( t ) ∣ (13) \left\|\mathbf{w}^{(t)}\right\|=\left|w_{1}^{(t)}\right| \tag*{(13)} w ( t ) = w 1 ( t ) (13)
Furthermore, w 1 ( t ) w_{1}^{(t)} w 1 ( t ) is strictly increasing. We claim that − w 1 ( t ) → − ∞ -w_{1}^{(t)}\to-\infty − w 1 ( t ) → − ∞ . Assume otherwise, then w 1 ( t ) → L ∈ R w_{1}^{(t)}\to L\in\mathbb{R} w 1 ( t ) → L ∈ R , but then the steps can be bounded from below i.e. w 1 ( t ) − w 1 ( t − 1 ) ≥ η exp − L > 0 w_{1}^{(t)}-w_{1}^{(t-1)}\geq\eta\exp^{-L}>0 w 1 ( t ) − w 1 ( t − 1 ) ≥ η exp − L > 0 , which is a contradiction to w 1 ( t ) → L w_{1}^{(t)}\to L w 1 ( t ) → L . So we have
− w 1 ( t ) → − ∞ (14) -w_{1}^{(t)}\to-\infty \tag*{(14)} − w 1 ( t ) → − ∞ (14)
But then L ( w ( t ) ) = exp ( − w 1 ( t ) ) = exp ( − ∥ w ( t ) ∥ ) \mathcal{L}(\mathbf{w}^{(t)})=\exp(-w_{1}^{(t)})=\exp\left(-\left\|\mathbf{w}^{(t)}\right\|\right) L ( w ( t ) ) = exp ( − w 1 ( t ) ) = exp ( − w ( t ) ) . Now assume towards contradiction that Assumption 4 holds, then by Theorem C.1 we get that there exists some t 0 t_{0} t 0 s.t. ∀ t ≥ t 0 \forall t\geq t_{0} ∀ t ≥ t 0 , L ( w ( t ) ) ≤ exp ( − p ( ∥ w ∥ ) ) \mathcal{L}(\mathbf{w}^{(t)})\leq\exp(-p(\|\mathbf{w}\|)) L ( w ( t ) ) ≤ exp ( − p ( ∥ w ∥ )) , but L ( w ( t ) ) = exp ( − ∥ w ( t ) ∥ ) ≥ exp ( − p ( ∥ w ∥ ) ) \mathcal{L}(\mathbf{w}^{(t)})=\exp\left(-\left\|\mathbf{w}^{(t)}\right\|\right)\geq\exp(-p(\|\mathbf{w}\|)) L ( w ( t ) ) = exp ( − w ( t ) ) ≥ exp ( − p ( ∥ w ∥ )) contradiction, since ∥ w ( t ) ∥ → ∞ \left\|\mathbf{w}^{(t)}\right\|\to\infty w ( t ) → ∞ and so for any p p p s.t. deg ( p ) = 3 \deg(p)=3 deg ( p ) = 3 with positive leading coefficient, we have ∥ w ( t ) ∥ = o ( p ( ∥ w ∥ ) ) \left\|\mathbf{w}^{(t)}\right\|=o(p(\|\mathbf{w}\|)) w ( t ) = o ( p ( ∥ w ∥ )) . Consequently Assumption 4 must not hold.
Our Assumptions Hold
We want to show Assumptions 1 to 3 hold. By Equation (12) and Equation (11) , we have that Assumption 2 holds, since ∇ L ( w ) \nabla\mathcal{L}(\mathbf{w}) ∇ L ( w ) always points in the same direction. From Equation (14) and Equation (12) we have that L ( w ( t ) ) = exp ( − w 1 ( t ) ) → 0 \mathcal{L}(\mathbf{w}^{(t)}) = \exp(-w_1^{(t)}) \to0 L ( w ( t ) ) = exp ( − w 1 ( t ) ) → 0 so Assumption 1 holds. To show Assumption 3 holds true, note that
lim t → ∞ w 1 ( t ) + 2 w 2 ( t ) w 4 ( t ) + ( w 3 ( t ) ) 2 + 3 w 3 ( t ) ( w 4 ( t ) ) 2 + ( w 4 ( t ) ) 4 ∥ w ( t ) ∥ 2 = w 1 ( t ) ( w 1 ( t ) ) 2 → 0 \lim_{t\to\infty}\frac{w_1^{(t)}+2w_2^{(t)}w_4^{(t)}+(w_3^{(t)})^2+3w_3^{(t)}(w_4^{(t)})^2+(w_4^{(t)})^4}{\lVert\mathbf{w}^{(t)}\rVert^2}=\frac{w_1^{(t)}}{(w_1^{(t)})^2}\to0 t → ∞ lim ∥ w ( t ) ∥ 2 w 1 ( t ) + 2 w 2 ( t ) w 4 ( t ) + ( w 3 ( t ) ) 2 + 3 w 3 ( t ) ( w 4 ( t ) ) 2 + ( w 4 ( t ) ) 4 = ( w 1 ( t ) ) 2 w 1 ( t ) → 0
On the other hand:
lim t → ∞ w 1 ( t ) + 2 w 2 ( t ) w 4 ( t ) + ( w 3 ( t ) ) 2 + 3 w 3 ( t ) ( w 4 ( t ) ) 2 + ( w 4 ( t ) ) 4 ∥ w ( t ) ∥ = w 1 ( t ) ∣ w 1 ( t ) ∣ = 1 = P 1 ( w ‾ ∞ ) T x y \lim_{t\to\infty}\frac{w_1^{(t)}+2w_2^{(t)}w_4^{(t)}+(w_3^{(t)})^2+3w_3^{(t)}(w_4^{(t)})^2+(w_4^{(t)})^4}{\lVert\mathbf{w}^{(t)}\rVert}=\frac{w_1^{(t)}}{\lvert w_1^{(t)}\rvert}=1=\mathcal{P}^{1}(\overline{\mathbf{w}}^{\infty})^{\mathsf{T}}\mathbf{x}y t → ∞ lim ∥ w ( t ) ∥ w 1 ( t ) + 2 w 2 ( t ) w 4 ( t ) + ( w 3 ( t ) ) 2 + 3 w 3 ( t ) ( w 4 ( t ) ) 2 + ( w 4 ( t ) ) 4 = ∣ w 1 ( t ) ∣ w 1 ( t ) = 1 = P 1 ( w ∞ ) T x y
Now for the corresponding gradient,
lim t → ∞ ( 1 0 0 0 ) ∥ w ( t ) ∥ 0 = ( 1 0 0 0 ) = ∇ P 1 ( w ‾ ∞ ) T x y \lim_{t\to\infty}\frac{\begin{pmatrix}1\\0\\0\\0\end{pmatrix}}{\lVert\mathbf{w}^{(t)}\rVert^0}=\begin{pmatrix}1\\0\\0\\0\end{pmatrix}=\nabla\mathcal{P}^{1}(\overline{\mathbf{w}}^{\infty})^{\mathsf{T}}\mathbf{x}y t → ∞ lim ∥ w ( t ) ∥ 0 1 0 0 0 = 1 0 0 0 = ∇ P 1 ( w ∞ ) T x y
And hence Assumptions 1 to 3 all hold.
Constructing General Negative Examples to Assumption 4
We can use the principles from the proof techniques of Theorem 4.3, i.e. Section C, to create polynomials for which similar dynamics hold. For simplicity, assume we have one example x = ( 1 1 ) \mathbf{x}=\begin{pmatrix}1\\1\end{pmatrix} x = ( 1 1 ) , y = 1 y=1 y = 1 . For every even d ≥ 2 d\ge2 d ≥ 2 and k < d k<d k < d we construct a polynomial such that the margin would be with degree d d d and effective degree k k k : Take P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}-\left(\sum_{i\in A}w_i\right)^k\\\left(\sum_{i\in B}w_i\right)^d\end{pmatrix} P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) , for some A ∩ B = ∅ A\cap B=\emptyset A ∩ B = ∅ . The exponential loss would be:
L ( w ) = exp ( − ( ∑ i ∈ A w i ) k + ( ∑ i ∈ B w i ) d ) . \mathcal{L}(\mathbf{w})=\exp\left(-\left(\sum_{i\in A}w_i\right)^k+\left(\sum_{i\in B}w_i\right)^d\right). L ( w ) = exp − ( i ∈ A ∑ w i ) k + ( i ∈ B ∑ w i ) d .
This will yield a similar type of dynamics. In fact, this will yield a stronger dynamic, since this will have effective degree k k k for almost all initializations in the case of even k k k , and a constant portion of initializations in the case of odd k k k .
Lemma C.2 (Assumption 4 does not hold for almost all initializations). Let d ≥ 2 d\ge2 d ≥ 2 be an even integer, and 0 < k < d 0<k<d 0 < k < d another even integer. Let x = ( 1 1 ) \mathbf{x}=\begin{pmatrix}1\\1\end{pmatrix} x = ( 1 1 ) , y = 1 y=1 y = 1 , and consider P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}-\left(\sum_{i\in A}w_i\right)^k\\\left(\sum_{i\in B}w_i\right)^d\end{pmatrix} P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) , for some A ∩ B = ∅ A\cap B=\emptyset A ∩ B = ∅ . Then for almost all initializations, optimizing
L ( w ) = exp ( − ( ∑ i ∈ A w i ) k + ( ∑ i ∈ B w i ) d ) . \mathcal{L}(\mathbf{w})=\exp\left(-\left(\sum_{i\in A}w_i\right)^k+\left(\sum_{i\in B}w_i\right)^d\right). L ( w ) = exp − ( i ∈ A ∑ w i ) k + ( i ∈ B ∑ w i ) d . With gradient descent yields L ( w ( t ) ) = exp ( − Θ ( ∥ w ( t ) ∥ k ) ) \mathcal{L}(\mathbf{w}^{(t)})=\exp\left(-\Theta\left(\lVert\mathbf{w}^{(t)}\rVert^k\right)\right) L ( w ( t ) ) = exp ( − Θ ( ∥ w ( t ) ∥ k ) )
This can be further generalized to larger families of polynomials, building on the same insight. Importantly, our main result about implicit bias for non-homogeneous polynomials applies to all of these constructions Theorem 5.1 .
Proof. First note that the gradient is of rank-2. Denote u ( w ) = ∑ i ∈ A w i u(\mathbf{w}) = \sum_{i\in A} w_i u ( w ) = ∑ i ∈ A w i , v ( w ) = ∑ i ∈ B w i v(\mathbf{w}) = \sum_{i\in B} w_i v ( w ) = ∑ i ∈ B w i . Then for every i ∈ A i \in A i ∈ A we have
∂ L ∂ w i ( w ) = L ( w ) ∂ ( − u k + v d ) ∂ w i ( w ) = − L ( w ) k u k − 1 \frac{\partial\mathcal{L}}{\partial w_i}(\mathbf{w}) = \mathcal{L}(\mathbf{w})\frac{\partial(-u^k+v^d)}{\partial w_i}(\mathbf{w}) = -\mathcal{L}(\mathbf{w})ku^{k-1} ∂ w i ∂ L ( w ) = L ( w ) ∂ w i ∂ ( − u k + v d ) ( w ) = − L ( w ) k u k − 1
Similarly for i ∈ B i \in B i ∈ B , we have
∂ L ∂ w i ( w ) = L ( w ) ∂ ( − u k + v d ) ∂ w i ( w ) = L ( w ) d v d − 1 \frac{\partial\mathcal{L}}{\partial w_i}(\mathbf{w}) = \mathcal{L}(\mathbf{w})\frac{\partial(-u^k+v^d)}{\partial w_i}(\mathbf{w}) = \mathcal{L}(\mathbf{w})dv^{d-1} ∂ w i ∂ L ( w ) = L ( w ) ∂ w i ∂ ( − u k + v d ) ( w ) = L ( w ) d v d − 1
Whenever i ∉ A ∪ B i \notin A\cup B i ∈ / A ∪ B ,
∂ L ∂ w i = 0 \frac{\partial\mathcal{L}}{\partial w_i}=0 ∂ w i ∂ L = 0
Hence ∀ i , i ′ ∈ A \forall i,i' \in A ∀ i , i ′ ∈ A , w i − w i ′ w_i-w_{i'} w i − w i ′ is constant. Similarly, ∀ i , i ′ ∈ B \forall i,i' \in B ∀ i , i ′ ∈ B , w i − w i ′ w_i-w_{i'} w i − w i ′ is constant. So we can re-write the update per u u u , v v v , i.e.
u t + 1 = u t + η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) v t + 1 = u t + η ∣ B ∣ d v t d − 1 exp ( − u t k + v t d ) (15) \begin{aligned}
u_{t+1} &= u_t+\eta|A|ku_t^{k-1}\exp(-u_t^k+v_t^d) \\
v_{t+1} &= u_t+\eta|B|dv_t^{d-1}\exp(-u_t^k+v_t^d) \tag*{(15)}
\end{aligned} u t + 1 v t + 1 = u t + η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) = u t + η ∣ B ∣ d v t d − 1 exp ( − u t k + v t d ) (15)
The rest of w \mathbf{w} w is fixed. Hence optimizing u u u and v v v will yield similar dynamics to optimizing w \mathbf{w} w . Denote R : = ∣ v 0 ∣ + 1 R:=|v_0|+1 R := ∣ v 0 ∣ + 1 , M : = max { 0 , − u 0 k } M:=\max\{0,-u_0^k\} M := max { 0 , − u 0 k } , and choose η ≤ exp ( − R d − M ) ∣ B ∣ d R d − 2 \eta\le\frac{\exp(-R^d-M)}{|B|dR^{d-2}} η ≤ ∣ B ∣ d R d − 2 e x p ( − R d − M ) .
Now, since u t + 1 = u t + η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) u_{t+1}=u_t+\eta|A|ku_t^{k-1}\exp(-u_t^k+v_t^d) u t + 1 = u t + η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) , we have that since k k k is even, u t k ≥ 0 u_t^k\ge0 u t k ≥ 0 and so − u t k ≤ 0 ≤ M -u_t^k\le0\le M − u t k ≤ 0 ≤ M hence exp ( − u t k + v t d ) ≤ exp ( M + v t d ) \exp(-u_t^k+v_t^d)\le\exp(M+v_t^d) exp ( − u t k + v t d ) ≤ exp ( M + v t d ) .
Now,
∣ v t + 1 ∣ = ∣ v t ∣ ∣ 1 − η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d ) ∣ |v_{t+1}|=|v_t|\left|1-\eta d|v_t|^{d-2}|B|\exp(-u_t^k-v_t^d)\right| ∣ v t + 1 ∣ = ∣ v t ∣ 1 − η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d )
Assume ∣ v t ∣ ≤ R |v_t|\le R ∣ v t ∣ ≤ R , then η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d ) ≤ 1 \eta d|v_t|^{d-2}|B|\exp(-u_t^k-v_t^d)\le1 η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d ) ≤ 1 from our choice of η \eta η , hence ∣ 1 − η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d ) ∣ ∈ [ 0 , 1 ] |1-\eta d|v_t|^{d-2}|B|\exp(-u_t^k-v_t^d)|\in[0,1] ∣1 − η d ∣ v t ∣ d − 2 ∣ B ∣ exp ( − u t k − v t d ) ∣ ∈ [ 0 , 1 ] . So ∣ v t + 1 ∣ ≤ ∣ v t ∣ ≤ R |v_{t+1}|\le|v_t|\le R ∣ v t + 1 ∣ ≤ ∣ v t ∣ ≤ R , hence by induction ∣ v t ∣ ≤ R |v_t|\le R ∣ v t ∣ ≤ R for all t t t .
Now since k k k is even, k − 1 k-1 k − 1 is odd and from Equation (15) we have that u t u_t u t never changes sign along training and ∣ u t ∣ |u_t| ∣ u t ∣ is strictly increasing. Now assume toward contradiction that ∣ u t ∣ |u_t| ∣ u t ∣ is bounded, then we can bound its update by η ∣ A ∣ k u 0 k − 1 exp ( − B − R d ) ≤ η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) \eta|A|ku_0^{k-1}\exp(-B-R^d)\le\eta|A|ku_t^{k-1}\exp(-u_t^k+v_t^d) η ∣ A ∣ k u 0 k − 1 exp ( − B − R d ) ≤ η ∣ A ∣ k u t k − 1 exp ( − u t k + v t d ) , i.e. the steps of u t u_t u t are lower bounded by a constant, in contradiction to it being bounded. Finally, since u t k = ∑ i ∈ A w i u_t^k=\sum_{i\in A}w_i u t k = ∑ i ∈ A w i and those are the only coordinates that diverge, we have that L ( w ( t ) ) ≤ exp ( − Θ ( ∥ w ( t ) ∥ k ) ) \mathcal{L}(\mathbf{w}^{(t)})\le\exp\left(-\Theta\left(\|\mathbf{w}^{(t)}\|^k\right)\right) L ( w ( t ) ) ≤ exp ( − Θ ( ∥ w ( t ) ∥ k ) ) .
Lemma C.3. Let d ≥ 2 d\ge2 d ≥ 2 be an even integer, and 0 < k < d 0<k<d 0 < k < d an odd integer. Let x = ( 1 1 ) \mathbf{x}=\begin{pmatrix}1\\1\end{pmatrix} x = ( 1 1 ) , y = 1 \mathbf{y}=1 y = 1 , and consider P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}-\left(\sum_{i\in A}w_i\right)^k\\ \left(\sum_{i\in B}w_i\right)^d\end{pmatrix} P ( w ) = ( − ( ∑ i ∈ A w i ) k ( ∑ i ∈ B w i ) d ) , for some A ∩ B = ∅ A\cap B=\varnothing A ∩ B = ∅ . Further assume that ( ∑ i ∈ A w i ) k > 0 \left(\sum_{i\in A}w_i\right)^k>0 ( ∑ i ∈ A w i ) k > 0 at initialization. Then, optimizing
L ( w ) = exp ( − ( ∑ i ∈ A w i ) k + ( ∑ i ∈ B w i ) d ) . \mathcal{L}(\mathbf{w})=\exp\left(-\left(\sum_{i\in A}w_i\right)^k+\left(\sum_{i\in B}w_i\right)^d\right). L ( w ) = exp − ( i ∈ A ∑ w i ) k + ( i ∈ B ∑ w i ) d . With gradient descent yields L ( w ( t ) ) = exp ( − Θ ( ∥ w ( t ) ∥ k ) ) \mathcal{L}(\mathbf{w}^{(t)})=\exp\left(-\Theta\left(\|\mathbf{w}^{(t)}\|^k\right)\right) L ( w ( t ) ) = exp ( − Θ ( ∥ w ( t ) ∥ k ) ) .
Proof. The proof is similar to the proof of Theorem C.2 , except the boundedness argument, in which case:
If k k k is even, u t k ≥ 0 u_t^k\ge0 u t k ≥ 0 and so − u t k ≤ 0 ≤ M -u_t^k\le0\le M − u t k ≤ 0 ≤ M
If k k k is odd, u t k − 1 ≥ 0 u_t^{k-1}\ge0 u t k − 1 ≥ 0 , and hence u t u_t u t is non-decreasing, so − u t k ≤ − u 0 k ≤ − M -u_t^k\le-u_0^k\le-M − u t k ≤ − u 0 k ≤ − M .
The rest of the proof is similar since we assume u 0 > 0 u_0>0 u 0 > 0 . □
Effective Degree Characteristics
The Effective Degree Depends on the Model, Data, and Initialization
First we formalize Theorem 4.2:
Lemma D.1. Let w = ( u , v ) ∈ R 2 \mathbf{w}=(u,v)\in\mathbb{R}^{2} w = ( u , v ) ∈ R 2 , define the polynomial predictor P ( w ) = ( q ( w ) q ( w ) 2 ⋮ q ( w ) R ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}q(\mathbf{w})\\q(\mathbf{w})^{2}\\\vdots\\q(\mathbf{w})^{R}\end{pmatrix} P ( w ) = q ( w ) q ( w ) 2 ⋮ q ( w ) R , with
q ( w ) = q ( u , v ) = u 2 + v 4 − u 2 v 2 . q(\mathbf{w})=q(u,v)=u^{2}+v^{4}-u^{2}v^{2}. q ( w ) = q ( u , v ) = u 2 + v 4 − u 2 v 2 . For r ∈ [ R ] r\in[R] r ∈ [ R ] , let x r = e r \mathbf{x}_{r}=\mathbf{e}_{r} x r = e r , y r = 1 y_{r}=1 y r = 1 . Then, m r ( w ) = y r ⟨ P ( w ) , x r ⟩ = q ( w ) r m_{r}(\mathbf{w})=y_{r}\langle\mathcal{P}(\mathbf{w}),\mathbf{x}_{r}\rangle=q(\mathbf{w})^{r} m r ( w ) = y r ⟨ P ( w ) , x r ⟩ = q ( w ) r . Consider gradient descent with a sufficiently small constant step size on L ( w ) = ∑ i = 1 R exp ( − q ( w ) r ) \mathcal{L}(\mathbf{w})=\sum_{i=1}^{R}\exp(-q(\mathbf{w})^{r}) L ( w ) = ∑ i = 1 R exp ( − q ( w ) r ) . For the open sets of initializations
U 2 = { u > 2 , ∣ v ∣ < 1 / 2 } , U 4 = { ∣ u ∣ < 1 / 2 , v > 2 } , U_{2}=\{u>2,\lvert v\rvert<1/2\},\qquad U_{4}=\{\lvert u\rvert<1/2,v>2\}, U 2 = { u > 2 , ∣ v ∣ < 1/2 } , U 4 = {∣ u ∣ < 1/2 , v > 2 } , effdeg ( m r ) = { 2 r w ( 0 ) ∈ U 2 4 r w ( 0 ) ∈ U 4 . \operatorname{effdeg}(m_{r})=
\begin{cases}
2r & \mathbf{w}^{(0)}\in U_{2}\\
4r & \mathbf{w}^{(0)}\in U_{4}.
\end{cases} effdeg ( m r ) = { 2 r 4 r w ( 0 ) ∈ U 2 w ( 0 ) ∈ U 4 .
Proof. Let us denote p 2 = u 2 p_{2}=u^{2} p 2 = u 2 , p 4 = v 4 − u 2 v 2 p_{4}=v^{4}-u^{2}v^{2} p 4 = v 4 − u 2 v 2 i.e. q = p 2 + p 4 q=p_{2}+p_{4} q = p 2 + p 4 , these polynomials will be the effective polynomials in the two regimes.
By Theorem D.2 we have that for w ( 0 ) ∈ U 2 \mathbf{w}^{(0)}\in U_{2} w ( 0 ) ∈ U 2 , u t → ∞ u_{t}\to\infty u t → ∞ , v t → 0 v_{t}\to0 v t → 0 , u t v t → 0 u_{t}v_{t}\to0 u t v t → 0 , hence w ( t ) ∥ w ( t ) ∥ → e 1 \frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert}\to\mathbf{e}_{1} ∥ w ( t ) ∥ w ( t ) → e 1 .
q ( w ( t ) ) ∥ w ( t ) ∥ 2 → 1 = p 2 ( e 1 ) . \frac{q(\mathbf{w}^{(t)})}{\lVert\mathbf{w}^{(t)}\rVert^{2}}\to1=p_{2}(\mathbf{e}_{1}). ∥ w ( t ) ∥ 2 q ( w ( t ) ) → 1 = p 2 ( e 1 ) .
We have
∇ q ( u , v ) = ( 2 u ( 1 − v 2 ) , 2 v ( 2 v 2 − u 2 ) ) . \nabla q(u,v)=(2u(1-v^{2}),2v(2v^{2}-u^{2})). ∇ q ( u , v ) = ( 2 u ( 1 − v 2 ) , 2 v ( 2 v 2 − u 2 )) .
And so,
∇ q ( w ( t ) ) ∥ w ( t ) ∥ → ( 2 , 0 ) = ∇ p 2 ( e 1 ) . \frac{\nabla q(\mathbf{w}^{(t)})}{\lVert\mathbf{w}^{(t)}\rVert}\to(2,0)=\nabla p_{2}(\mathbf{e}_{1}). ∥ w ( t ) ∥ ∇ q ( w ( t ) ) → ( 2 , 0 ) = ∇ p 2 ( e 1 ) .
That is, effdeg ( q ) = 2 \operatorname{effdeg}(q)=2 effdeg ( q ) = 2 . In a similar manner, by Theorem D.2 if w ( 0 ) ∈ U 4 \mathbf{w}^{(0)}\in U_{4} w ( 0 ) ∈ U 4 , then u t → 0 u_{t}\to0 u t → 0 , v t = ∞ v_{t}=\infty v t = ∞ . So, w ( t ) ∥ w ( t ) ∥ → e 2 \frac{\mathbf{w}^{(t)}}{\lVert\mathbf{w}^{(t)}\rVert}\to\mathbf{e}_{2} ∥ w ( t ) ∥ w ( t ) → e 2 . Furthermore,
q ( w ( t ) ) ∥ w ( t ) ∥ 4 → 1 = p 4 ( e 2 ) , \frac{q(\mathbf{w}^{(t)})}{\lVert\mathbf{w}^{(t)}\rVert^{4}}\to1=p_{4}(\mathbf{e}_{2}), ∥ w ( t ) ∥ 4 q ( w ( t ) ) → 1 = p 4 ( e 2 ) ,
and
∇ q ( w ( t ) ) ∥ w ( t ) ∥ 3 → ( 0 , 4 ) = ∇ p 4 ( e 2 ) , \frac{\nabla q(\mathbf{w}^{(t)})}{\lVert\mathbf{w}^{(t)}\rVert^{3}}\to(0,4)=\nabla p_{4}(\mathbf{e}_{2}), ∥ w ( t ) ∥ 3 ∇ q ( w ( t ) ) → ( 0 , 4 ) = ∇ p 4 ( e 2 ) ,
that is, effdeg ( q ) = 4 \operatorname{effdeg}(q)=4 effdeg ( q ) = 4 . Finally, recall that m r = q r m_{r}=q^{r} m r = q r . Using the fact that m r ( w ) ∥ w ∥ r k = ( q ( w ) ∥ w ∥ k ) r \frac{m_{r}(\mathbf{w})}{\lVert\mathbf{w}\rVert^{rk}}=\left(\frac{q(\mathbf{w})}{\lVert\mathbf{w}\rVert^{k}}\right)^{r} ∥ w ∥ r k m r ( w ) = ( ∥ w ∥ k q ( w ) ) r , and
m r ( w ) ∥ w ∥ r k − 1 = r ( q ( w ) ∥ w ∥ k ) r − 1 ∇ q ( w ) ∥ w ∥ k − 1 \frac{m_{r}(\mathbf{w})}{\lVert\mathbf{w}\rVert^{rk-1}}=r\left(\frac{q(\mathbf{w})}{\lVert\mathbf{w}\rVert^{k}}\right)^{r-1}\frac{\nabla q(\mathbf{w})}{\lVert\mathbf{w}\rVert^{k-1}} ∥ w ∥ r k − 1 m r ( w ) = r ( ∥ w ∥ k q ( w ) ) r − 1 ∥ w ∥ k − 1 ∇ q ( w )
we finish the proof. □
Lemma D.2. Let w = ( u , v ) ∈ R 2 \mathbf{w}=(u,v)\in\mathbb{R}^{2} w = ( u , v ) ∈ R 2 , define the polynomial predictor P ( w ) = ( q ( w ) q ( w ) 2 ⋮ q ( w ) R ) \mathcal{P}(\mathbf{w})=\begin{pmatrix}q(\mathbf{w})\\q(\mathbf{w})^{2}\\\vdots\\q(\mathbf{w})^{R}\end{pmatrix} P ( w ) = q ( w ) q ( w ) 2 ⋮ q ( w ) R , with
q ( w ) = q ( u , v ) = u 2 + v 4 − u 2 v 2 . q(\mathbf{w})=q(u,v)=u^{2}+v^{4}-u^{2}v^{2}. q ( w ) = q ( u , v ) = u 2 + v 4 − u 2 v 2 . For r ∈ [ R ] r\in[R] r ∈ [ R ] , let x r = e r \mathbf{x}_{r}=\mathbf{e}_{r} x r = e r , y r = 1 y_{r}=1 y r = 1 . Then, m r ( w ) = y r ⟨ P ( w ) , x r ⟩ = q ( w ) r m_{r}(\mathbf{w})=y_{r}\langle\mathcal{P}(\mathbf{w}),\mathbf{x}_{r}\rangle=q(\mathbf{w})^{r} m r ( w ) = y r ⟨ P ( w ) , x r ⟩ = q ( w ) r . Consider gradient descent with a sufficiently small constant step size on L ( w ) = ∑ i = 1 R exp ( − q ( w ) r ) \mathcal{L}(\mathbf{w})=\sum_{i=1}^{R}\exp(-q(\mathbf{w})^{r}) L ( w ) = ∑ i = 1 R exp ( − q ( w ) r ) . For the open sets of initializations
U 2 = { u > 2 , ∣ v ∣ < 1 / 2 } , U 4 = { ∣ u ∣ < 1 / 2 , v > 2 } , U_{2}=\{u>2,\lvert v\rvert<1/2\},\qquad U_{4}=\{\lvert u\rvert<1/2,v>2\}, U 2 = { u > 2 , ∣ v ∣ < 1/2 } , U 4 = {∣ u ∣ < 1/2 , v > 2 } , and a sufficiently small learning rate η > 0 \eta>0 η > 0 , if w ( 0 ) ∈ U 2 \mathbf{w}^{(0)}\in U_{2} w ( 0 ) ∈ U 2 , w ( t ) ∈ U 2 \mathbf{w}^{(t)}\in U_{2} w ( t ) ∈ U 2 , u t → ∞ u_{t}\to\infty u t → ∞ , v t → 0 v_{t}\to0 v t → 0 , u t v t → 0 u_{t}v_{t}\to0 u t v t → 0 . If w ( 0 ) ∈ U 4 \mathbf{w}^{(0)}\in U_{4} w ( 0 ) ∈ U 4 , w ( t ) ∈ U 4 \mathbf{w}^{(t)}\in U_{4} w ( t ) ∈ U 4 , u t → 0 u_{t}\to0 u t → 0 , v t → ∞ v_{t}\to\infty v t → ∞ .
The proof follows by showing the path is invariant to U 2 U_2 U 2 and U 4 U_4 U 4 , and the dynamic in each one forces the corresponding asymptotic behavior of the coordinates.
Proof. Note that ∇ m r ( w ) = r q ( b w ) r − 1 ∇ q ( w ) \nabla m_r(\mathbf{w}) = rq(b\mathbf{w})^{r-1}\nabla q(\mathbf{w}) ∇ m r ( w ) = r q ( b w ) r − 1 ∇ q ( w ) , hence ∇ L ( w ) \nabla\mathcal{L}(\mathbf{w}) ∇ L ( w ) is always a scalar multiple of ∇ q ( w ) \nabla q(\mathbf{w}) ∇ q ( w ) , i.e.
− ∇ L ( w ) = a ( q ( w ) ) ∇ q ( w ) -\nabla\mathcal{L}(\mathbf{w}) = a(q(\mathbf{w}))\nabla q(\mathbf{w}) − ∇ L ( w ) = a ( q ( w )) ∇ q ( w )
For a ( s ) = ∑ r = 1 R r s r − 1 exp ( − s r ) a(s) = \sum_{r=1}^{R} r s^{r-1}\exp(-s^r) a ( s ) = ∑ r = 1 R r s r − 1 exp ( − s r ) , which is positive whenever t > 0 t > 0 t > 0 , and in that case we can think of w ( t ) \mathbf{w}^{(t)} w ( t ) following a gradient-ascent path w.r.t. q q q , for appropriate step sizes. The coordinate updates are then:
u t + 1 = u t ( 1 + 2 η ⋅ a ( q ( w ( t ) ) ) ( 1 − v t 2 ) ) , v t + 1 = v t ( 1 + 2 η ⋅ a ( q ( w ( t ) ) ) ( 2 v t 2 − u t 2 ) ) (16) u_{t+1} = u_t\left(1 + 2\eta\cdot a(q(\mathbf{w}^{(t)}))(1-v_t^2)\right), \qquad v_{t+1} = v_t\left(1 + 2\eta\cdot a(q(\mathbf{w}^{(t)}))(2v_t^2-u_t^2)\right) \tag*{(16)} u t + 1 = u t ( 1 + 2 η ⋅ a ( q ( w ( t ) )) ( 1 − v t 2 ) ) , v t + 1 = v t ( 1 + 2 η ⋅ a ( q ( w ( t ) )) ( 2 v t 2 − u t 2 ) ) (16)
Note that a ( s ) a(s) a ( s ) decays exponentially, so we can find
M 1 = sup s ≥ 3 s ⋅ a ( s ) < ∞ , M 1 / 2 = sup s ≥ 3 s 1 / 2 ⋅ a ( s ) < ∞ . M_1 = \sup_{s\geq3}s\cdot a(s)<\infty,\qquad M_{1/2}=\sup_{s\geq3}s^{1/2}\cdot a(s)<\infty. M 1 = s ≥ 3 sup s ⋅ a ( s ) < ∞ , M 1/2 = s ≥ 3 sup s 1/2 ⋅ a ( s ) < ∞.
We will choose η > 0 \eta>0 η > 0 s.t.
8 3 M 1 < 1 / 2 , 8 15 η M 1 / 2 < 1 / 2. \frac{8}{3}M_1<1/2,\qquad\frac{8}{\sqrt{15}}\eta M_{1/2}<1/2. 3 8 M 1 < 1/2 , 15 8 η M 1/2 < 1/2.
If w ( t ) ∈ U 2 \mathbf{w}^{(t)}\in U_2 w ( t ) ∈ U 2 : Then q ( w ( t ) ) = u t 2 ( 1 − v t 2 ) + v t 4 ≥ 3 4 u t 2 > 3 q(\mathbf{w}^{(t)})=u_t^2(1-v_t^2)+v_t^4\geq\frac{3}{4}u_t^2>3 q ( w ( t ) ) = u t 2 ( 1 − v t 2 ) + v t 4 ≥ 4 3 u t 2 > 3 , and so u t 2 ≤ 4 3 q ( w ( t ) ) u_t^2\leq\frac{4}{3}q(\mathbf{w}^{(t)}) u t 2 ≤ 3 4 q ( w ( t ) ) . Furthermore, 1 − v t 2 > 0 1-v_t^2>0 1 − v t 2 > 0 , so u t + 1 > u t u_{t+1}>u_t u t + 1 > u t , and since 2 v t 2 − u t 2 < 0 2v_t^2-u_t^2<0 2 v t 2 − u t 2 < 0 , we have
0 > 2 η ⋅ a ( q ( w ( t ) ) ) ( 2 v t 2 − u t 2 ) > − 2 η ⋅ a ( q ( w ( t ) ) ) u t 2 ≥ − 8 3 η ⋅ a ( q ( w ( t ) ) ) > − 1 / 2 , 0>2\eta\cdot a(q(\mathbf{w}^{(t)}))(2v_t^2-u_t^2)>-2\eta\cdot a(q(\mathbf{w}^{(t)}))u_t^2\geq-\frac{8}{3}\eta\cdot a(q(\mathbf{w}^{(t)}))>-1/2, 0 > 2 η ⋅ a ( q ( w ( t ) )) ( 2 v t 2 − u t 2 ) > − 2 η ⋅ a ( q ( w ( t ) )) u t 2 ≥ − 3 8 η ⋅ a ( q ( w ( t ) )) > − 1/2 ,
from the choice of η \eta η and u t 2 ≤ 4 3 q ( w ( t ) ) u_t^2\leq\frac{4}{3}q(\mathbf{w}^{(t)}) u t 2 ≤ 3 4 q ( w ( t ) ) . Plugging in to the update for v t + 1 v_{t+1} v t + 1 we get ∣ v t + 1 ∣ < ∣ v t ∣ |v_{t+1}|<|v_t| ∣ v t + 1 ∣ < ∣ v t ∣ , so w ( t + 1 ) ∈ U 2 \mathbf{w}^{(t+1)}\in U_2 w ( t + 1 ) ∈ U 2 . We got that u t u_t u t is increasing. If we assume it is bounded, we get that q ( w ( t ) ) q(\mathbf{w}^{(t)}) q ( w ( t ) ) must be bounded, but then we can lower bound a ( q ( w ( t ) ) ) ≥ C > 0 a(q(\mathbf{w}^{(t)}))\geq C>0 a ( q ( w ( t ) )) ≥ C > 0 . Together with 1 − v t 2 ≥ 3 / 4 1-v_t^2\geq3/4 1 − v t 2 ≥ 3/4 we get u t + 1 ≥ u t ( 1 + 3 2 η C ) u_{t+1}\geq u_t(1+\frac{3}{2}\eta C) u t + 1 ≥ u t ( 1 + 2 3 η C ) , contradiction. So u t → ∞ u_t\to\infty u t → ∞ .
Now we claim that
∑ i = 0 ∞ 2 η a ( q ( w ( t ) ) ) ( u t 2 − 2 v t 2 ) = ∞ (17) \sum_{i=0}^{\infty}2\eta a(q(\mathbf{w}^{(t)}))(u_t^2-2v_t^2)=\infty \tag*{(17)} i = 0 ∑ ∞ 2 η a ( q ( w ( t ) )) ( u t 2 − 2 v t 2 ) = ∞ (17)
Indeed,
log u t + 1 u t = log ( 1 + 2 a ( q ( w ( t ) ) ) ) ≤ 2 a ( q ( w ( t ) ) ) , (18) \log\frac{u_{t+1}}{u_t}=\log(1+2a(q(\mathbf{w}^{(t)})))\leq2a(q(\mathbf{w}^{(t)})), \tag*{(18)} log u t u t + 1 = log ( 1 + 2 a ( q ( w ( t ) ))) ≤ 2 a ( q ( w ( t ) )) , (18)
and since the left hand side sums to infinity, so must the right hand side. On U 2 U_2 U 2 we have that u t 2 − 2 v t 2 > 2 / 7 u_t^2-2v_t^2>2/7 u t 2 − 2 v t 2 > 2/7 , and hence the original sum must diverge.
Now for w ( t ) ∈ U 2 \mathbf{w}^{(t)}\in U_2 w ( t ) ∈ U 2 we have that
0 < 2 η a ( q ( w ( t ) ) ) ( u t 2 − 2 v t 2 ) < 2 η a ( q ( w ( t ) ) ) u t 2 ≤ 2 η a ( q ( w ( t ) ) ) 4 3 q ( w ( t ) ) = 8 3 q ( w ( t ) ) η a ( q ( w ( t ) ) ) ≤ 8 3 η M 1 < 1 / 2. \begin{aligned}
0&<2\eta a(q(\mathbf{w}^{(t)}))(u_t^2-2v_t^2)\\
&<2\eta a(q(\mathbf{w}^{(t)}))u_t^2\\
&\leq2\eta a(q(\mathbf{w}^{(t)}))\frac{4}{3}q(\mathbf{w}^{(t)})\\
&=\frac{8}{3}q(\mathbf{w}^{(t)})\eta a(q(\mathbf{w}^{(t)}))\\
&\leq\frac{8}{3}\eta M_1<1/2.
\end{aligned} 0 < 2 η a ( q ( w ( t ) )) ( u t 2 − 2 v t 2 ) < 2 η a ( q ( w ( t ) )) u t 2 ≤ 2 η a ( q ( w ( t ) )) 3 4 q ( w ( t ) ) = 3 8 q ( w ( t ) ) η a ( q ( w ( t ) )) ≤ 3 8 η M 1 < 1/2.
Recall that ∣ v t + 1 ∣ = ∣ v t ∣ ( 1 − 2 η a ( q ( w ( t ) ) ) ( u t 2 − 2 v t 2 ) ) |v_{t+1}|=|v_t|(1-2\eta a(q(\mathbf{w}^{(t)}))(u_t^2-2v_t^2)) ∣ v t + 1 ∣ = ∣ v t ∣ ( 1 − 2 η a ( q ( w ( t ) )) ( u t 2 − 2 v t 2 )) . Together with Eq. (17) and the bound on the constant multiplying ∣ v t ∣ |v_t| ∣ v t ∣ , we get that v t → 0 v_t\to0 v t → 0 . Finally, by Eq. (16)
u t + 1 ∣ v t + 1 ∣ = u t ∣ v t ∣ ( 1 + 2 η ⋅ a ( q ( w ( t ) ) ) ( 2 v t 2 − u t 2 ) ) ( 1 + 2 η ⋅ a ( q ( w ( t ) ) ) ( 1 − v t 2 ) ) . u_{t+1}|v_{t+1}|=u_t|v_t|\left(1+2\eta\cdot a(q(\mathbf{w}^{(t)}))(2v_t^2-u_t^2)\right)\left(1+2\eta\cdot a(q(\mathbf{w}^{(t)}))(1-v_t^2)\right). u t + 1 ∣ v t + 1 ∣ = u t ∣ v t ∣ ( 1 + 2 η ⋅ a ( q ( w ( t ) )) ( 2 v t 2 − u t 2 ) ) ( 1 + 2 η ⋅ a ( q ( w ( t ) )) ( 1 − v t 2 ) ) .
Using log ( 1 + x ) ≤ x \log(1+x) \le x log ( 1 + x ) ≤ x ,
log u t + 1 ∣ v t + 1 ∣ u t ∣ v t ∣ ≤ ( 1 + 2 η ⋅ a ( q ( w ( t ) ) ) ( 2 v t 2 − u t 2 ) ) + 2 η ⋅ a ( q ( w ( t ) ) ) ( 1 − v t 2 ) < − 11 2 η a ( q ( w ( t ) ) ) \log\frac{u_{t+1}\lvert v_{t+1}\rvert}{u_t\lvert v_t\rvert} \le\left(1+2\eta\cdot a(q(w^{(t)}))(2v_t^2-u_t^2)\right)+2\eta\cdot a(q(w^{(t)}))(1-v_t^2)<-\frac{11}{2}\eta a(q(w^{(t)})) log u t ∣ v t ∣ u t + 1 ∣ v t + 1 ∣ ≤ ( 1 + 2 η ⋅ a ( q ( w ( t ) )) ( 2 v t 2 − u t 2 ) ) + 2 η ⋅ a ( q ( w ( t ) )) ( 1 − v t 2 ) < − 2 11 η a ( q ( w ( t ) ))
Where the last inequality is due to w ( t ) ∈ U 2 w^{(t)}\in U_2 w ( t ) ∈ U 2 . Now summing the above equation over t t t the r.h.s. diverges to − ∞ -\infty − ∞ , because the sum over t t t in Equation (18) diverges to infinity. The bound on the l.h.s. sum yields ∣ v t ∣ u t → 0 \lvert v_t\rvert u_t\to0 ∣ v t ∣ u t → 0 .
If w ( t ) ∈ U 4 w^{(t)}\in U_4 w ( t ) ∈ U 4 : then u t 2 < 1 / 4 u_t^2<1/4 u t 2 < 1/4 , v t > 2 v_t>2 v t > 2 . Furthermore,
0 < 2 η a ( q ( w ( t ) ) ) ( v t 2 − 1 ) < 2 η a ( q ( w ( t ) ) ) v t 2 ≤ 8 15 η q ( w ( t ) ) 1 / 2 a ( q ( w ( t ) ) ) < 1 / 2. \begin{aligned}
0&<2\eta a(q(w^{(t)}))(v_t^2-1)\\
&<2\eta a(q(w^{(t)}))v_t^2\\
&\le\frac{8}{\sqrt{15}}\eta q(w^{(t)})^{1/2}a(q(w^{(t)}))<1/2.
\end{aligned} 0 < 2 η a ( q ( w ( t ) )) ( v t 2 − 1 ) < 2 η a ( q ( w ( t ) )) v t 2 ≤ 15 8 η q ( w ( t ) ) 1/2 a ( q ( w ( t ) )) < 1/2.
Where (*) is due to the fact that under U 4 U_4 U 4 ,
q ( w ( t ) ) ≥ 15 16 v t 4 . (19) q(w^{(t)})\ge\frac{15}{16}v_t^4. \tag*{(19)} q ( w ( t ) ) ≥ 16 15 v t 4 . (19)
Recalling the update rule in Equation (16) , we get ∣ u t + 1 ∣ < ∣ u t ∣ \lvert u_{t+1}\rvert<\lvert u_t\rvert ∣ u t + 1 ∣ < ∣ u t ∣ , and since 2 v t 2 − u t 2 > 0 2v_t^2-u_t^2>0 2 v t 2 − u t 2 > 0 , then v t + 1 > v t v_{t+1}>v_t v t + 1 > v t . That is, w ( t + 1 ) ∈ U 4 w^{(t+1)}\in U_4 w ( t + 1 ) ∈ U 4 .
Similarly to the prior case, if v t v_t v t were bounded then so were q ( w ( t ) ) q(w^{(t)}) q ( w ( t ) ) , forcing geometric growth for v t v_t v t , which is a contradiction. Hence v t → ∞ v_t\to\infty v t → ∞ . In a similar technique to the prior case, we would like to show ∑ t = 1 ∞ 2 η a ( q ( w ( t ) ) ) ( v t 2 − 1 ) \sum_{t=1}^{\infty}2\eta a(q(w^{(t)}))(v_t^2-1) ∑ t = 1 ∞ 2 η a ( q ( w ( t ) )) ( v t 2 − 1 ) . First note that 2 η a ( q ( w ( t ) ) ) ( v t 2 − 1 ) ≥ 2 3 η a ( q ( w ( t ) ) ) v t 2 2\eta a(q(w^{(t)}))(v_t^2-1)\ge\frac{2}{3}\eta a(q(w^{(t)}))v_t^2 2 η a ( q ( w ( t ) )) ( v t 2 − 1 ) ≥ 3 2 η a ( q ( w ( t ) )) v t 2 , so showing that summing the r.h.s. over t t t diverges suffices. We have
log v t + 1 v t ≤ log ( 1 + 2 η a ( q ( w ( t ) ) ) ( 2 v t 2 − u t 2 ) ) ≤ 4 η a ( q ( w ( t ) ) ) v t 2 . \log\frac{v_{t+1}}{v_t}\le\log\left(1+2\eta a(q(w^{(t)}))(2v_t^2-u_t^2)\right)\le4\eta a(q(w^{(t)}))v_t^2. log v t v t + 1 ≤ log ( 1 + 2 η a ( q ( w ( t ) )) ( 2 v t 2 − u t 2 ) ) ≤ 4 η a ( q ( w ( t ) )) v t 2 .
Since summing the l.h.s. diverges to infinity, so does summing the r.h.s., and hence every scalar multiplication of it. Now ∣ u t + 1 ∣ = ∣ u t ∣ ( 1 − 2 η a ( q ( w ( t ) ) ) ( v t 2 − 1 ) ) \lvert u_{t+1}\rvert=\lvert u_t\rvert(1-2\eta a(q(w^{(t)}))(v_t^2-1)) ∣ u t + 1 ∣ = ∣ u t ∣ ( 1 − 2 η a ( q ( w ( t ) )) ( v t 2 − 1 )) . From the divergence above and the fact that
0 < 2 η a ( q ( w ( t ) ) ) ( v t 2 − 1 ) ≤ 2 η a ( q ( w ( t ) ) ) v t 2 ≤ Eq. (18) 8 15 η q ( w ( t ) ) 1 / 2 a ( q ( w ( t ) ) ) ≤ 8 15 η M 1 / 2 < 1 / 2 , \begin{aligned}
0&<2\eta a(q(w^{(t)}))(v_t^2-1)\\
&\le2\eta a(q(w^{(t)}))v_t^2\\
&\mathrel{\overset{\text{\scriptsize Eq. (18)}}{\le}}\frac{8}{\sqrt{15}}\eta q(w^{(t)})^{1/2}a(q(w^{(t)}))\\
&\le\frac{8}{\sqrt{15}}\eta M_{1/2}<1/2,
\end{aligned} 0 < 2 η a ( q ( w ( t ) )) ( v t 2 − 1 ) ≤ 2 η a ( q ( w ( t ) )) v t 2 ≤ Eq. (18) 15 8 η q ( w ( t ) ) 1/2 a ( q ( w ( t ) )) ≤ 15 8 η M 1/2 < 1/2 ,
we have that u t → 0 u_t\to0 u t → 0 . □
Determining the Effective Degree Without Solving the Dynamics
We next provide a polynomial predictor P : R D → R n \mathcal{P}:\mathbb{R}^{D}\to\mathbb{R}^{n} P : R D → R n , and a dataset { x i , y i } i = 1 n \{x_i,y_i\}_{i=1}^{n} { x i , y i } i = 1 n , such that when trained with gradient descent (Equation (1) ) with a constant step size η > 0 \eta>0 η > 0 , the margins m i = y i ⟨ P , x i ⟩ m_i=y_i\langle\mathcal{P},x_i\rangle m i = y i ⟨ P , x i ⟩ have different effective degrees, and effdeg ( m i ) \operatorname{effdeg}(m_i) effdeg ( m i ) can be derived by a short calculation.
Note that the proof works for any subset of the above dataset, with arbitrary labels, and the polynomial predictor can be extended to a family of polynomials for which the above holds. So this yields in fact a family of such examples.
Consider the parameter vector ( u , v ) = w ∈ R 2 (u,v)=w\in\mathbb{R}^{2} ( u , v ) = w ∈ R 2 and the polynomial P : R 2 → R N \mathcal{P}:\mathbb{R}^{2}\to\mathbb{R}^{N} P : R 2 → R N
P ( u , v ) = ( u ν 1 − v d u ν 2 − v d ⋮ u ν N − v d ) \mathcal{P}(u,v)=
\begin{pmatrix}
u^{\nu_1}-v^d\\
u^{\nu_2}-v^d\\
\vdots\\
u^{\nu_N}-v^d
\end{pmatrix} P ( u , v ) = u ν 1 − v d u ν 2 − v d ⋮ u ν N − v d
for 2 ≤ ν 1 < … < ν N < d 2 \le\nu_1 < \ldots< \nu_N < d 2 ≤ ν 1 < … < ν N < d , with d d d even. Let y i ∈ { ± 1 } y_i \in\{\pm1\} y i ∈ { ± 1 } some labels, choose x i = y i e i ∈ R N x_i = y_i e_i \in\mathbb{R}^N x i = y i e i ∈ R N .
Now m i ( u , v ) = u ν i − v d m_i(u,v) = u^{\nu_i} - v^d m i ( u , v ) = u ν i − v d . For 0 < η < 1 d N 0 < \eta< \frac{1}{dN} 0 < η < d N 1 and initializing with u 0 > 1 u_0 > 1 u 0 > 1 , and ∣ v 0 ∣ < 1 |v_0| < 1 ∣ v 0 ∣ < 1 , we will show effdeg ( m i ) = ν i \operatorname{effdeg}(m_i) = \nu_i effdeg ( m i ) = ν i . The gradient updates are:
u t + 1 = u t + η ∑ i = 1 N ν i u t ν i − 1 exp ( v t d − u t ν i ) u_{t+1} = u_t + \eta\sum_{i=1}^{N} \nu_i u_t^{\nu_i-1} \exp\left(v_t^d-u_t^{\nu_i}\right) u t + 1 = u t + η i = 1 ∑ N ν i u t ν i − 1 exp ( v t d − u t ν i )
v t + 1 = v t − η d v t d − 1 ∑ i = 1 N exp ( v t d − u t ν i ) = v t ( 1 − η d ∣ v t ∣ d − 2 ∑ i = 1 N exp ( v t d − u t ν i ) ) ( d is even ) \begin{aligned}
v_{t+1} = v_t - \eta d v_t^{d-1} \sum_{i=1}^{N} \exp\left(v_t^d-u_t^{\nu_i}\right) \\
&= v_t\left(1-\eta d|v_t|^{d-2}\sum_{i=1}^{N}\exp\left(v_t^d-u_t^{\nu_i}\right)\right) \qquad(d\text{ is even})
\end{aligned} v t + 1 = v t − η d v t d − 1 i = 1 ∑ N exp ( v t d − u t ν i ) = v t ( 1 − η d ∣ v t ∣ d − 2 i = 1 ∑ N exp ( v t d − u t ν i ) ) ( d is even )
Now while u t ≥ 1 u_t \ge1 u t ≥ 1 and ∣ v t ∣ ≤ 1 |v_t| \le1 ∣ v t ∣ ≤ 1 , we have v t d − u t ν i ≤ 0 v_t^d-u_t^{\nu_i} \le0 v t d − u t ν i ≤ 0 . So
0 ≤ η d ∣ v t ∣ d − 2 ∑ i = 1 N exp ( v t d − u t ν i ) ≤ η d N ≤ 1 0 \le\eta d|v_t|^{d-2}\sum_{i=1}^{N}\exp\left(v_t^d-u_t^{\nu_i}\right) \le\eta dN \le1 0 ≤ η d ∣ v t ∣ d − 2 i = 1 ∑ N exp ( v t d − u t ν i ) ≤ η d N ≤ 1
So overall ∣ v t + 1 ∣ < ∣ v t ∣ |v_{t+1}| < |v_t| ∣ v t + 1 ∣ < ∣ v t ∣ , and u t + 1 ≥ u t u_{t+1} \ge u_t u t + 1 ≥ u t since all of the u u u updates are positive. We have that the optimization stays in the open interval u t > u 0 ≥ 1 u_t > u_0 \ge1 u t > u 0 ≥ 1 , ∣ v t ∣ < ∣ v 0 ∣ < 1 |v_t| < |v_0| < 1 ∣ v t ∣ < ∣ v 0 ∣ < 1 .
If we assume u t ≤ M u_t \le M u t ≤ M , then
u t + 1 − u t ≥ η ν 1 u t ν 1 − 1 exp ( v t d − u t ν 1 ) ≥ η ν 1 u 0 ν 1 − 1 e − M ν 1 > 0. u_{t+1}-u_t \ge\eta\nu_1 u_t^{\nu_1-1}\exp\left(v_t^d-u_t^{\nu_1}\right) \ge\eta\nu_1 u_0^{\nu_1-1}e^{-M^{\nu_1}} > 0. u t + 1 − u t ≥ η ν 1 u t ν 1 − 1 exp ( v t d − u t ν 1 ) ≥ η ν 1 u 0 ν 1 − 1 e − M ν 1 > 0.
Hence u t → ∞ u_t \to\infty u t → ∞ , contradiction. A straight forward calculation concludes the proof.
Proof of thm:main
Proof Intuition
We have that w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ exists from Assumption 2. We use Assumption 1 to show this implies w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ must create positive margins, and hence primal feasibility. Assumption 3 implies two different things:
The margins are dominated by their ν i \nu_i ν i -homogeneous parts, and hence ∥ P i ∥ \|\mathcal{P}_i\| ∥ P i ∥ scale like ∥ w ∥ ν i \|w\|^{\nu_i} ∥ w ∥ ν i .
The margins’ gradients are dominated by their ν i − 1 \nu_i-1 ν i − 1 -homogeneous parts, and hence ∥ ∇ P i ∥ \|\nabla\mathcal{P}_i\| ∥∇ P i ∥ scale like ∥ w ∥ ν i − 1 \|w\|^{\nu_i-1} ∥ w ∥ ν i − 1 .
The first is then used to transform the non-homogeneous margin m i : = y i ⟨ P i , x i ⟩ m_i := y_i\langle\mathcal{P}_i,x_i\rangle m i := y i ⟨ P i , x i ⟩ into the homogeneous polynomial in the KKT conditions, i.e. m ν i = ⟨ y i C i P i ν i , x i ⟩ m^{\nu_i} = \langle y_i C_i \mathcal{P}_i^{\nu_i},x_i\rangle m ν i = ⟨ y i C i P i ν i , x i ⟩ , and also to show that the gradients of lower-effective-degree margins dominate the gradient, since their corresponding loss-exponent decays slower (the exponent of every loss-per-sample decays as ∥ w ∥ ν i \|w\|^{\nu_i} ∥ w ∥ ν i ). Among the lowest effdeg margins, we further show that those that converge to a lower normalized margin dominate their higher-normalized-margin counterparts. Overall this tells us that ∇ L = ∑ i ∈ S α i ∇ m i + o ( ∑ i ∈ S α i ∇ m i ) \nabla L = \sum_{i\in S}\alpha_i\nabla m_i + o\left(\sum_{i\in S}\alpha_i\nabla m_i\right) ∇ L = ∑ i ∈ S α i ∇ m i + o ( ∑ i ∈ S α i ∇ m i ) , where S S S consists of the lowest effective degree polynomials with the lowest normalized margin, i.e. the gradient is dominated by a possibly dynamic convex sum of margins from S S S . From Assumption 3, ∇ m i ≈ C i ∇ m i ν Z \nabla m_i \approx C_i\nabla m_i^{\nu_Z} ∇ m i ≈ C i ∇ m i ν Z where ν Z \nu_Z ν Z is the minimal effective degree and hence ( w ˉ ∞ ) ⊤ ∑ i ∈ S α i ∇ m i ≈ γ min ν Z (\bar{w}^{\infty})^\top\sum_{i\in S}\alpha_i\nabla m_i \approx\gamma_{\min}\nu_Z ( w ˉ ∞ ) ⊤ ∑ i ∈ S α i ∇ m i ≈ γ m i n ν Z , where γ min \gamma_{\min} γ m i n is the minimum margin in that defines S S S . i.e. the loss has a constant projection onto w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ . We then attempt to show that the projection onto the space orthogonal to w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ vanishes, at least in some subsequence of the gradients. Towards this we show that the GD updates induce an approximate exponential-multiplicative-weights step on α i \alpha_i α i , which can be thought of as the dual variables. We further show that this multiplicative-weights update is one that minimizes the gradient norm along some subsequence. Finally, we show that the minimizer must have no component orthogonal to w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ . This implies that there exists a subsequence for which ∇ L ∝ w ˉ ∞ \nabla L \propto\bar{w}^{\infty} ∇ L ∝ w ˉ ∞ , which shows optimality and complementary slackness. Dual feasibility is due to α i \alpha_i α i being weights of a convex sum.
Proof
For convenience, let us restate the theorem:
Theorem E.1. Let { P i } i = 1 N \{\mathcal{P}_i\}_{i=1}^{N} { P i } i = 1 N a collection of polynomials with parameters w ∈ R D \mathbf{w} \in\mathbb{R}^{D} w ∈ R D . For a dataset { x i , y i } i = 1 N \{x_i,y_i\}_{i=1}^{N} { x i , y i } i = 1 N where y i = sgn ( ⟨ P i ( w ) , x i ⟩ ) y_i=\operatorname{sgn}(\langle\mathcal{P}_i(\mathbf{w}),x_i\rangle) y i = sgn (⟨ P i ( w ) , x i ⟩) , consider the sequence of gradient updates (Equation (1) ) with a non-increasing sequence of step sizes { η t } t \{\eta_t\}_t { η t } t , minimizing the empirical risk. If Assumptions 1 to 3 hold we have that w ˉ ∞ : = lim t → ∞ w ( t ) ∥ w ( t ) ∥ \bar{\mathbf{w}}^{\infty}:=\lim_{t\to\infty}\frac{\mathbf{w}^{(t)}}{\|\mathbf{w}^{(t)}\|} w ˉ ∞ := lim t → ∞ ∥ w ( t ) ∥ w ( t ) is a positive scaling of a first order stationary point (KKT point) of the following optimization problem:
min w ∈ R d ∥ w ∥ 2 2 s.t. ∀ i , y i ⟨ x i , P i ν i ( w ) ⟩ ≥ 0 \min_{\mathbf{w}\in\mathbb{R}^{d}}\|\mathbf{w}\|_2^2 \quad\text{s.t.}\ \forall i,\ y_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle\ge0 w ∈ R d min ∥ w ∥ 2 2 s.t. ∀ i , y i ⟨ x i , P i ν i ( w )⟩ ≥ 0 ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w ) ⟩ ≥ 1 , \forall i\in Z,\ y_i C_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle\ge1, ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w )⟩ ≥ 1 , where Z = arg min i ∈ [ n ] { effdeg w y i ⟨ P i , x i ⟩ } Z=\arg\min_{i\in[n]}\{\operatorname{effdeg}_{\mathbf{w}} y_i\langle\mathcal{P}_i,x_i\rangle\} Z = arg min i ∈ [ n ] { effdeg w y i ⟨ P i , x i ⟩} .
We would like to show that w ˉ ∞ \bar{\mathbf{w}}^{\infty} w ˉ ∞ is in a direction of a KKT point of Equation (OPT). First, let us state the KKT conditions:
Primal feasibility: ∀ i , y i ⟨ x i , P i ν i ( w ) ⟩ ≥ 0 , ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w ) ⟩ ≥ 1 \forall i,\ y_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle\ge0,\ \forall i\in Z,\ y_iC_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle\ge1 ∀ i , y i ⟨ x i , P i ν i ( w )⟩ ≥ 0 , ∀ i ∈ Z , y i C i ⟨ x i , P i ν i ( w )⟩ ≥ 1
Dual feasibility and complementary slackness: There are { μ i } i = 1 N \{\mu_i\}_{i=1}^{N} { μ i } i = 1 N where μ i ≥ 0 \mu_i\ge0 μ i ≥ 0 and
∀ i ∉ Z \forall i\notin Z ∀ i ∈ / Z , if y i ⟨ x i , P i ν i ( w ) ⟩ > 0 y_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle>0 y i ⟨ x i , P i ν i ( w )⟩ > 0 , μ i = 0 \mu_i=0 μ i = 0
∀ i ∈ Z \forall i\in Z ∀ i ∈ Z , if y i C i ⟨ x i , P i ν i ( w ) ⟩ > 1 y_iC_i\langle x_i,\mathcal{P}_i^{\nu_i}(\mathbf{w})\rangle>1 y i C i ⟨ x i , P i ν i ( w )⟩ > 1 , μ i = 0 \mu_i=0 μ i = 0
Optimality: w = ∑ i = 1 N μ i ⋅ C i ∇ P i ν i ( w ) T y i x i \mathbf{w}=\sum_{i=1}^{N}\mu_i\cdot C_i\nabla\mathcal{P}_i^{\nu_i}(\mathbf{w})^{\mathsf{T}}y_i x_i w = ∑ i = 1 N μ i ⋅ C i ∇ P i ν i ( w ) T y i x i
First we will show that any positive scaling of w ˉ ∞ \bar{\mathbf{w}}^{\infty} w ˉ ∞ is a feasible solution.
Lemma E.2 (Primal Feasibility). w ˉ ∞ \bar{\mathbf{w}}^{\infty} w ˉ ∞ classifies all examples correctly, and for C > 0 C>0 C > 0 large enough, C w ˉ ∞ C\bar{\mathbf{w}}^{\infty} C w ˉ ∞ is a feasible solution to Equation (OPT) under Assumptions 1 to 3.
Proof. By Assumption 3 we have that P i ( w ( t ) ) T x i ∥ w ( t ) ∥ ν i = C i P i ν i ( w ˉ ∞ ) T x i − δ i ( t ) ≠ 0 \frac{\mathcal{P}_i(\mathbf{w}^{(t)})^{\mathsf{T}}x_i}{\|\mathbf{w}^{(t)}\|^{\nu_i}}=C_i\mathcal{P}_i^{\nu_i}(\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i-\delta_i^{(t)}\ne0 ∥ w ( t ) ∥ ν i P i ( w ( t ) ) T x i = C i P i ν i ( w ˉ ∞ ) T x i − δ i ( t ) = 0 with δ i ( t ) → 0 \delta_i^{(t)}\to0 δ i ( t ) → 0 . Hence from homogeneity C i P i ν i ( C w ˉ ∞ ) T x i ≠ 0 C_i\mathcal{P}_i^{\nu_i}(C\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i\ne0 C i P i ν i ( C w ˉ ∞ ) T x i = 0 . Assume towards contradiction that C i P i ν i ( C w ˉ ∞ ) T x i y i < 0 C_i\mathcal{P}_i^{\nu_i}(C\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i y_i<0 C i P i ν i ( C w ˉ ∞ ) T x i y i < 0 , then we have
P i ( w ( t ) ) T x i y i = ( C i P i ν i ( C w ˉ ∞ ) T x i y i C ν i + δ i ( t ) ) ∥ w ( t ) ∥ ν i → − ∞ \mathcal{P}_i(\mathbf{w}^{(t)})^{\mathsf{T}}x_i y_i=\left(\frac{C_i\mathcal{P}_i^{\nu_i}(C\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i y_i}{C^{\nu_i}}+\delta_i^{(t)}\right)\|\mathbf{w}^{(t)}\|^{\nu_i}\to-\infty P i ( w ( t ) ) T x i y i = ( C ν i C i P i ν i ( C w ˉ ∞ ) T x i y i + δ i ( t ) ) ∥ w ( t ) ∥ ν i → − ∞ Since δ i ( t ) → 0 \delta_i^{(t)}\to0 δ i ( t ) → 0 , contradicting Assumption 1. All that is left is scaling C C C such that all margins are in the feasible region.
To show the rest of the KKT conditions, we want to understand the asymptotic direction of the gradient.
Lemma E.3 (Minimum Margin of Minimum Effective Degree Dominates the Gradient). − ∇ L ( w ˉ ( t ) ) + δ ∇ ( t ) ∝ ∑ i ∈ S α i ( t ) C i ∇ P i ν i ( w ( t ) ) T x i y i -\nabla\mathcal{L}(\bar{\mathbf{w}}^{(t)})+\delta_{\nabla}^{(t)}\propto\sum_{i\in S}\alpha_i^{(t)}C_i\nabla\mathcal{P}_i^{\nu_i}(\mathbf{w}^{(t)})^{\mathsf{T}}x_i y_i − ∇ L ( w ˉ ( t ) ) + δ ∇ ( t ) ∝ ∑ i ∈ S α i ( t ) C i ∇ P i ν i ( w ( t ) ) T x i y i for δ ∇ ( t ) → 0 \delta_{\nabla}^{(t)}\to0 δ ∇ ( t ) → 0 , where S = arg min i ∈ Z { C i P i ν i ( w ˉ ∞ ) T x i y i } S=\arg\min_{i\in Z}\{C_i\mathcal{P}_i^{\nu_i}(\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i y_i\} S = arg min i ∈ Z { C i P i ν i ( w ˉ ∞ ) T x i y i } , where S S S is a set, and Z = arg min i ∈ [ n ] { effdeg ( P i ν i ( w ˉ ∞ ) x i y i ) } Z=\arg\min_{i\in[n]}\{\operatorname{effdeg}(\mathcal{P}_i^{\nu_i}(\bar{\mathbf{w}}^{\infty})x_i y_i)\} Z = arg min i ∈ [ n ] { effdeg ( P i ν i ( w ˉ ∞ ) x i y i )} .
Proof. Consider the gradient of the loss:
− ∇ L ( w ( t ) ) = ∑ i = 1 n ∇ P i ( w ( t ) ) T x i y i exp ( − P i ( w ( t ) ) T x i y i ) -\nabla\mathcal{L}\left(\mathbf{w}^{(t)}\right)=\sum_{i=1}^{n}\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^{\mathsf{T}}x_i y_i\exp\left(-\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^{\mathsf{T}}x_i y_i\right) − ∇ L ( w ( t ) ) = i = 1 ∑ n ∇ P i ( w ( t ) ) T x i y i exp ( − P i ( w ( t ) ) T x i y i ) = ∑ i = 1 n ∇ P i ( w ( t ) ) T x i y i exp ( − g ν i ( t ) ( C i P i ν i ( w ˉ ∞ ) T x i y i + δ i ( t ) y i ) ) (by assumption 3) =\sum_{i=1}^{n}\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^{\mathsf{T}}x_i y_i\exp\left(-g^{\nu_i}(t)\left(C_i\mathcal{P}_i^{\nu_i}\left(\bar{\mathbf{w}}^{\infty}\right)^{\mathsf{T}}x_i y_i+\delta_i^{(t)}y_i\right)\right)\quad\text{(by assumption 3)} = i = 1 ∑ n ∇ P i ( w ( t ) ) T x i y i exp ( − g ν i ( t ) ( C i P i ν i ( w ˉ ∞ ) T x i y i + δ i ( t ) y i ) ) (by assumption 3) Now denoting γ i : = C i P i ν i ( w ˉ ∞ ) T x i y i > 0 \gamma_i:=C_i\mathcal{P}_i^{\nu_i}(\bar{\mathbf{w}}^{\infty})^{\mathsf{T}}x_i y_i>0 γ i := C i P i ν i ( w ˉ ∞ ) T x i y i > 0 (this is positive from Theorem E.2), we get
− ∇ L ( w ( t ) ) = ∑ i = 1 n ∇ P i ( w ( t ) ) T x i y i exp ( − g ν i ( t ) ( γ i + δ i ( t ) y i ) ) -\nabla\mathcal{L}\left(\mathbf{w}^{(t)}\right)=\sum_{i=1}^{n}\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^{\mathsf{T}}x_i y_i\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}y_i\right)\right) − ∇ L ( w ( t ) ) = i = 1 ∑ n ∇ P i ( w ( t ) ) T x i y i exp ( − g ν i ( t ) ( γ i + δ i ( t ) y i ) )
Now we want to divide the loss into two separate parts. Denote v Z : = min i { effdeg ( P i ( ⋅ ) ⊤ x i ) } v_Z := \min_i\{\operatorname{effdeg}(\mathcal{P}_i(\cdot)^\top x_i)\} v Z := min i { effdeg ( P i ( ⋅ ) ⊤ x i )} (Z Z Z is chosen because for every k ∈ Z k \in Z k ∈ Z effdeg ( P i ( ⋅ ) ⊤ x i ) = v Z \operatorname{effdeg}(\mathcal{P}_i(\cdot)^\top x_i)=v_Z effdeg ( P i ( ⋅ ) ⊤ x i ) = v Z ). Let us write
A ( t ) = ∑ i ∈ Z ∇ P i ( w ( t ) ) x i y i ⊤ exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) ) A(t) = \sum_{i\in Z} \nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)\mathbf{x}_i y_i^\top\exp\left(-g^{v_Z}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right) A ( t ) = i ∈ Z ∑ ∇ P i ( w ( t ) ) x i y i ⊤ exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) )
and
B ( t ) = ∑ i ∉ Z ∇ P i ( w ( t ) ) x i y i ⊤ exp ( − g v i ( t ) ( γ i + δ i ( t ) ) ) B(t) = \sum_{i\notin Z} \nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)\mathbf{x}_i y_i^\top\exp\left(-g^{v_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right) B ( t ) = i ∈ / Z ∑ ∇ P i ( w ( t ) ) x i y i ⊤ exp ( − g v i ( t ) ( γ i + δ i ( t ) ) )
Note that A ( t ) + B ( t ) = − ∇ L ( w ( t ) ) A(t)+B(t)=-\nabla\mathcal{L}\left(\mathbf{w}^{(t)}\right) A ( t ) + B ( t ) = − ∇ L ( w ( t ) ) . We will now further divide A ( t ) A(t) A ( t ) into two parts. First denote γ min : = min i γ i \gamma_{\min}:=\min_i\gamma_i γ m i n := min i γ i , (recall they are all equal and minimal), and let min i ∈ Z \ S { γ i } + γ min 2 = γ ^ \frac{\min_{i\in Z\backslash S}\{\gamma_i\}+\gamma_{\min}}{2}=\hat{\gamma} 2 m i n i ∈ Z \ S { γ i } + γ m i n = γ ^ . The idea is that we define γ ^ \hat{\gamma} γ ^ to be the average between the minimum margin (of the minimum effective degree), and the second-smallest margin (of the minimum effdeg). Then, when we will have for i ∈ Z \ S i\in Z\backslash S i ∈ Z \ S γ i − γ ^ > 0 \gamma_i-\hat{\gamma}>0 γ i − γ ^ > 0 while γ min − γ ^ < 0 \gamma_{\min}-\hat{\gamma}<0 γ m i n − γ ^ < 0 . Concretely, denote:
I ( t ) : = ∑ i ∈ S exp ( − g v Z ( t ) ( γ min + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i exp ( − γ ^ ⋅ g v Z ( t ) ) \mathrm{I}(t):=\frac{\sum_{i\in S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}+\delta_i^{(t)}\right)\right)\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^\top\mathbf{x}_i y_i}{\exp\left(-\hat{\gamma}\cdot g^{v_Z}(t)\right)} I ( t ) := exp ( − γ ^ ⋅ g v Z ( t ) ) ∑ i ∈ S exp ( − g v Z ( t ) ( γ m i n + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i
I I ( t ) : = ∑ i ∈ Z \ S exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i exp ( − γ ^ ⋅ g v Z ( t ) ) \mathrm{II}(t):=\frac{\sum_{i\in Z\backslash S}\exp\left(-g^{v_Z}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^\top\mathbf{x}_i y_i}{\exp\left(-\hat{\gamma}\cdot g^{v_Z}(t)\right)} II ( t ) := exp ( − γ ^ ⋅ g v Z ( t ) ) ∑ i ∈ Z \ S exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i
We chose I, II such that A ( t ) = exp ( − γ ^ ⋅ g v Z ( t ) ) ( I ( t ) + I I ( t ) ) A(t)=\exp\left(-\hat{\gamma}\cdot g^{v_Z}(t)\right)(\mathrm{I}(t)+\mathrm{II}(t)) A ( t ) = exp ( − γ ^ ⋅ g v Z ( t ) ) ( I ( t ) + II ( t )) . We can then write
∥ A ( t ) ∥ ≥ exp ( − γ ^ ⋅ g v Z ( t ) ) ⋅ ∥ I ( t ) ∥ − ∥ I I ( t ) ∥ (20) \left\|A(t)\right\|\geq\exp\left(-\hat{\gamma}\cdot g^{v_Z}(t)\right)\cdot\left\|\mathrm{I}(t)\right\|-\left\|\mathrm{II}(t)\right\| \tag*{(20)} ∥ A ( t ) ∥ ≥ exp ( − γ ^ ⋅ g v Z ( t ) ) ⋅ ∥ I ( t ) ∥ − ∥ II ( t ) ∥ (20)
Inspecting the norm
∥ I I ( t ) ∥ = ∥ ∑ i ∈ Z \ S exp ( − g v Z ( t ) ( γ min − γ ^ + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i ∥ \left\|\mathrm{II}(t)\right\|=\left\|\sum_{i\in Z\backslash S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}\right)\right)\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^\top\mathbf{x}_i y_i\right\| ∥ II ( t ) ∥ = i ∈ Z \ S ∑ exp ( − g v Z ( t ) ( γ m i n − γ ^ + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i
We note that for large enough t t t since δ i ( t ) → 0 \delta_i^{(t)}\to0 δ i ( t ) → 0 and by definition of γ ^ \hat{\gamma} γ ^
γ min − γ ^ + δ i ( t ) < 0 (21) \gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}<0 \tag*{(21)} γ m i n − γ ^ + δ i ( t ) < 0 (21)
We want to use this to show ∥ I I ( t ) ∥ → ∞ \left\|\mathrm{II}(t)\right\|\to\infty ∥ II ( t ) ∥ → ∞ . Here, a subtle point is to verify the linear combination doesn’t asymptotically cancel out. Indeed
I ( t ) = ∑ i ∈ S exp ( − g v Z ( t ) ( γ min − γ ^ + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i = ∑ i ∈ S exp ( − g v Z ( t ) ( γ min − γ ^ + δ i ( t ) ) ) ( C i ∇ P i v Z ( w ˉ ∞ ) ⊤ x i y i + δ ∇ , i ( t ) ) (xref‘Assumption 3‘) \begin{aligned}
\mathrm{I}(t)=\sum_{i\in S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}\right)\right)\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^\top\mathbf{x}_i y_i \\
&=\sum_{i\in S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}\right)\right)\left(C_i\nabla\mathcal{P}_i^{v_Z}\left(\bar{\mathbf{w}}^\infty\right)^\top\mathbf{x}_i y_i+\delta_{\nabla,i}^{(t)}\right) \quad\text{({xref}`Assumption 3`)}
\end{aligned} I ( t ) = i ∈ S ∑ exp ( − g v Z ( t ) ( γ m i n − γ ^ + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i = i ∈ S ∑ exp ( − g v Z ( t ) ( γ m i n − γ ^ + δ i ( t ) ) ) ( C i ∇ P i v Z ( w ˉ ∞ ) ⊤ x i y i + δ ∇ , i ( t ) ) ( xref ‘Assumption 3‘)
Now from homogeneity we have that
( w ˉ ∞ ) ⊤ C i ∇ P i v Z ( w ˉ ∞ ) y i x i = v Z C i P i v Z ( w ˉ ∞ ) y i x i = v Z γ i y i > 0 \left(\bar{\mathbf{w}}^\infty\right)^\top C_i\nabla\mathcal{P}_i^{v_Z}\left(\bar{\mathbf{w}}^\infty\right)y_i\mathbf{x}_i
=v_Z C_i\mathcal{P}_i^{v_Z}\left(\bar{\mathbf{w}}^\infty\right)y_i\mathbf{x}_i
=v_Z\gamma_i y_i>0 ( w ˉ ∞ ) ⊤ C i ∇ P i v Z ( w ˉ ∞ ) y i x i = v Z C i P i v Z ( w ˉ ∞ ) y i x i = v Z γ i y i > 0
Hence
( w ˉ ∞ ) ⊤ I ( t ) ∥ I I ( t ) ∥ = ∑ i ∈ S exp ( − g v Z ( t ) ( γ min − γ ^ + δ i ( t ) ) ) ∣ ∑ i ∈ S exp ( − g v Z ( t ) ( γ min − γ ^ + δ i ( t ) ) ) ( C i ∇ P i v Z ( w ˉ ∞ ) ⊤ x i y i + δ ∇ , i ( t ) ) ∣ ( v Z γ min + δ ∇ , i ( t ) ) \frac{\left(\bar{\mathbf{w}}^\infty\right)^\top\mathrm{I}(t)}{\left\|\mathrm{II}(t)\right\|}
=
\frac{\sum_{i\in S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}\right)\right)}
{\left|\sum_{i\in S}\exp\left(-g^{v_Z}(t)\left(\gamma_{\min}-\hat{\gamma}+\delta_i^{(t)}\right)\right)\left(C_i\nabla\mathcal{P}_i^{v_Z}\left(\bar{\mathbf{w}}^\infty\right)^\top\mathbf{x}_i y_i+\delta_{\nabla,i}^{(t)}\right)\right|}
\left(v_Z\gamma_{\min}+\delta_{\nabla,i}^{(t)}\right) ∥ II ( t ) ∥ ( w ˉ ∞ ) ⊤ I ( t ) = ∑ i ∈ S exp ( − g v Z ( t ) ( γ m i n − γ ^ + δ i ( t ) ) ) ( C i ∇ P i v Z ( w ˉ ∞ ) ⊤ x i y i + δ ∇ , i ( t ) ) ∑ i ∈ S exp ( − g v Z ( t ) ( γ m i n − γ ^ + δ i ( t ) ) ) ( v Z γ m i n + δ ∇ , i ( t ) )
Is bounded away from 0, and so is I ( t ) ∥ I I ( t ) ∥ \frac{\mathrm{I}(t)}{\left\|\mathrm{II}(t)\right\|} ∥ II ( t ) ∥ I ( t ) . Together with Equation (21) , this yields ∥ I I ( t ) ∥ → ∞ \left\|\mathrm{II}(t)\right\|\to\infty ∥ II ( t ) ∥ → ∞ .
Now for I I ( t ) \mathrm{II}(t) II ( t ) ,
∥ I I ( t ) ∥ = ∥ ∑ i ∈ Z \ S exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i exp ( − γ ^ ⋅ g v Z ( t ) ) ∥ \left\|\mathrm{II}(t)\right\|=\left\|\frac{\sum_{i\in Z\backslash S}\exp\left(-g^{v_Z}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)\nabla\mathcal{P}_i\left(\mathbf{w}^{(t)}\right)^\top\mathbf{x}_i y_i}{\exp\left(-\hat{\gamma}\cdot g^{v_Z}(t)\right)}\right\| ∥ II ( t ) ∥ = exp ( − γ ^ ⋅ g v Z ( t ) ) ∑ i ∈ Z \ S exp ( − g v Z ( t ) ( γ i + δ i ( t ) ) ) ∇ P i ( w ( t ) ) ⊤ x i y i
here again by the definition of γ ^ \hat{\gamma} γ ^ and for large enough t t t , we have that for any i ∈ Z \ S i \in Z \backslash S i ∈ Z \ S
γ i − γ ^ + δ i ( t ) > 0 \gamma_i-\hat{\gamma}+\delta_i^{(t)}>0 γ i − γ ^ + δ i ( t ) > 0
therefore we can bound ∥ I I ( t ) ∥ ≤ C \|\mathrm{II}(t)\| \le C ∥ II ( t ) ∥ ≤ C for large enough t t t . Now consider the expression
∥ B ( t ) ∥ = ∥ ∑ i ∉ Z ∇ P i ( w ( t ) ) x i y i T exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ∥ (22) \|B(t)\|=\left\|\sum_{i\notin Z}\nabla\mathcal{P}_i\left(w^{(t)}\right)x_i y_i^{\mathsf{T}}\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)\right\| \tag*{(22)} ∥ B ( t ) ∥ = i ∈ / Z ∑ ∇ P i ( w ( t ) ) x i y i T exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) (22)
≤ ∑ i ∉ Z exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ∥ ∇ P i ( w ( t ) ) x i ∥ (23) \leq\sum_{i\notin Z}\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)\left\|\nabla\mathcal{P}_i\left(w^{(t)}\right)x_i\right\| \tag*{(23)} ≤ i ∈ / Z ∑ exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ∇ P i ( w ( t ) ) x i (23)
≤ ∑ i ∉ Z g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ( r max + C ) (24) \leq\sum_{i\notin Z}g^{\nu_i-1}(t)\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)(r_{\max}+C) \tag*{(24)} ≤ i ∈ / Z ∑ g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ( r m a x + C ) (24)
For some constant C > 0 C>0 C > 0 , and r max = max i ∥ C i ∇ P i ν i ( w ˉ ∞ ) T x i ∥ r_{\max}=\max_i\left\|C_i\nabla\mathcal{P}_i^{\nu_i}(\bar{w}^{\infty})^{\mathsf{T}}x_i\right\| r m a x = max i C i ∇ P i ν i ( w ˉ ∞ ) T x i . Overall using the bound on ∥ A ( t ) ∥ \|A(t)\| ∥ A ( t ) ∥ (20) , we get that
∥ B ( t ) ∥ ∥ A ( t ) ∥ ≤ ∑ i ∉ Z g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ( r max + C ) exp ( − γ ^ g ν Z ( t ) ) ⋅ ∣ ∥ I ( t ) ∥ − ∥ I I ( t ) ∥ ∣ = ∑ i ∉ Z g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) + γ ^ g ν Z ( t ) ) ( r max + C ) ∣ ∥ I ( t ) ∥ − ∥ I I ( t ) ∥ ∣ \begin{aligned}
\frac{\|B(t)\|}{\|A(t)\|}\leq\frac{\sum_{i\notin Z}g^{\nu_i-1}(t)\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)\right)(r_{\max}+C)}{\exp\left(-\hat{\gamma}g^{\nu_Z}(t)\right)\cdot\left|\|I(t)\|-\|\mathrm{II}(t)\|\right|} \\
&=\frac{\sum_{i\notin Z}g^{\nu_i-1}(t)\exp\left(-g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)+\hat{\gamma}g^{\nu_Z}(t)\right)(r_{\max}+C)}{\left|\|I(t)\|-\|\mathrm{II}(t)\|\right|}
\end{aligned} ∥ A ( t ) ∥ ∥ B ( t ) ∥ ≤ exp ( − γ ^ g ν Z ( t ) ) ⋅ ∣ ∥ I ( t ) ∥ − ∥ II ( t ) ∥ ∣ ∑ i ∈ / Z g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) ) ( r m a x + C ) = ∣ ∥ I ( t ) ∥ − ∥ II ( t ) ∥ ∣ ∑ i ∈ / Z g ν i − 1 ( t ) exp ( − g ν i ( t ) ( γ i + δ i ( t ) ) + γ ^ g ν Z ( t ) ) ( r m a x + C )
Now for i ∉ Z i\notin Z i ∈ / Z , − g ν i ( t ) ( γ i + δ i ( t ) ) + γ ^ g ν Z ( t ) -g^{\nu_i}(t)\left(\gamma_i+\delta_i^{(t)}\right)+\hat{\gamma}g^{\nu_Z}(t) − g ν i ( t ) ( γ i + δ i ( t ) ) + γ ^ g ν Z ( t ) is negative for large enough t t t . Since γ ^ \hat{\gamma} γ ^ , γ i + δ i ( t ) \gamma_i+\delta_i^{(t)} γ i + δ i ( t ) are bounded and positive for large enough t t t , and g ν i ( t ) g^{\nu_i}(t) g ν i ( t ) grows asymptotically faster than g ν Z ( t ) g^{\nu_Z}(t) g ν Z ( t ) . Therefore the expression in the exponent is negative for large enough t t t , hence the numerator goes to zero. The denominator goes to infinity due to ∥ I ( t ) ∥ → ∞ \|I(t)\|\to\infty ∥ I ( t ) ∥ → ∞ , and that ∥ I I ( t ) ∥ \|\mathrm{II}(t)\| ∥ II ( t ) ∥ is bounded, so overall:
− ∇ L ( w ( t ) ) ∥ ∇ L ( w ( t ) ) ∥ = A ( t ) + B ( t ) ∥ A ( t ) + B ( t ) ∥ = A ( t ) ∥ A ( t ) ∥ + δ ~ ( t ) -\frac{\nabla\mathcal{L}(w^{(t)})}{\|\nabla\mathcal{L}(w^{(t)})\|}=\frac{A(t)+B(t)}{\|A(t)+B(t)\|}=\frac{A(t)}{\|A(t)\|}+\tilde{\delta}^{(t)} − ∥∇ L ( w ( t ) ) ∥ ∇ L ( w ( t ) ) = ∥ A ( t ) + B ( t ) ∥ A ( t ) + B ( t ) = ∥ A ( t ) ∥ A ( t ) + δ ~ ( t )
For some δ ~ ( t ) → 0 \tilde{\delta}^{(t)}\to0 δ ~ ( t ) → 0 . Now similarly consider
exp ( − γ ^ g ν Z ( t ) ) ⋅ ∥ I I ( t ) ∥ exp ( − γ ^ g ν Z ( t ) ) ⋅ ∥ I ( t ) ∥ = ∥ I I ( t ) ∥ ∥ I ( t ) ∥ → 0 , \frac{\exp(-\hat{\gamma}g^{\nu_Z}(t))\cdot\|\mathrm{II}(t)\|}{\exp(-\hat{\gamma}g^{\nu_Z}(t))\cdot\|I(t)\|}=\frac{\|\mathrm{II}(t)\|}{\|I(t)\|}\to0, exp ( − γ ^ g ν Z ( t )) ⋅ ∥ I ( t ) ∥ exp ( − γ ^ g ν Z ( t )) ⋅ ∥ II ( t ) ∥ = ∥ I ( t ) ∥ ∥ II ( t ) ∥ → 0 ,
hence
A ( t ) ∥ A ( t ) ∥ = I ( t ) ∥ I ( t ) ∥ + δ ^ ( t ) \frac{A(t)}{\|A(t)\|}=\frac{I(t)}{\|I(t)\|}+\hat{\delta}^{(t)} ∥ A ( t ) ∥ A ( t ) = ∥ I ( t ) ∥ I ( t ) + δ ^ ( t )
For δ ^ ( t ) → 0 \hat{\delta}^{(t)}\to0 δ ^ ( t ) → 0 . Overall
− ∇ L ( w ( t ) ) ∥ ∇ L ( w ( t ) ) ∥ → I ( t ) ∥ I ( t ) ∥ -\frac{\nabla\mathcal{L}(w^{(t)})}{\|\nabla\mathcal{L}(w^{(t)})\|}\to\frac{I(t)}{\|I(t)\|} − ∥∇ L ( w ( t ) ) ∥ ∇ L ( w ( t ) ) → ∥ I ( t ) ∥ I ( t )
i.e. the normalized gradient is dominated by I ( t ) I(t) I ( t ) . □
For simplicity of notation, let us denote m i ( w ) : = P i ( w ) T x i y i m_i(w):=\mathcal{P}_i(w)^{\mathsf{T}}x_i y_i m i ( w ) := P i ( w ) T x i y i . Let us define ∀ i ∈ S \forall i\in S ∀ i ∈ S , α i ( t ) : = exp ( − m i ( w ( t ) ) ) L ∣ S ( w ( t ) ) \alpha_i^{(t)}:=\frac{\exp(-m_i(w^{(t)}))}{\mathcal{L}_{|S}(w^{(t)})} α i ( t ) := L ∣ S ( w ( t ) ) e x p ( − m i ( w ( t ) )) , where L ∣ S = ∑ i ∈ S exp ( − m i ) ≤ L \mathcal{L}_{|S}=\sum_{i\in S}\exp(-m_i)\leq\mathcal{L} L ∣ S = ∑ i ∈ S exp ( − m i ) ≤ L . By Assumption 3, we have that ∇ m i ( w ( t ) ) = 1 ∥ w ( t ) ∥ ν i ( C i ∇ m i ν i ( w ˉ ∞ ) + o ( 1 ) ) \nabla m_i(w^{(t)})=\frac{1}{\|w^{(t)}\|^{\nu_i}}\left(C_i\nabla m_i^{\nu_i}(\bar{w}^{\infty})+o(1)\right) ∇ m i ( w ( t ) ) = ∥ w ( t ) ∥ ν i 1 ( C i ∇ m i ν i ( w ˉ ∞ ) + o ( 1 ) ) . So overall
− ∇ L ( w ( t ) ) = L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) + o ( 1 ) ) -\nabla\mathcal{L}(w^{(t)})=\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t)\left(\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})+o(1)\right) − ∇ L ( w ( t ) ) = L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) ( i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) + o ( 1 ) )
Now note that Δ w ( t ) : = w ( t + 1 ) − w ( t ) = − η t ∇ L ( w ( t ) ) → 0 \Delta w^{(t)}:=w^{(t+1)}-w^{(t)}=-\eta_t\nabla\mathcal{L}(w^{(t)})\to0 Δ w ( t ) := w ( t + 1 ) − w ( t ) = − η t ∇ L ( w ( t ) ) → 0 , hence using taylor approximation we have
m i ( w ( t + 1 ) ) = m i ( w ( t ) ) + ⟨ ∇ m i ( w ( t ) ) , Δ w ( t ) ⟩ + E i ( t ) m_i(w^{(t+1)})=m_i(w^{(t)})+\left\langle\nabla m_i(w^{(t)}),\Delta w^{(t)}\right\rangle+E_i^{(t)} m i ( w ( t + 1 ) ) = m i ( w ( t ) ) + ⟨ ∇ m i ( w ( t ) ) , Δ w ( t ) ⟩ + E i ( t )
With
∣ E i ( t ) ∣ ≤ K i ∥ Δ w ( t ) ∥ g 2 deg ( m i ) − 2 ( t ) = K i η t 2 ∥ ∇ L ( w ( t ) ) ∥ g 2 deg ( m i ) − 2 ( t ) ≤ K i B 2 η t 2 L ∣ S ( w ( t ) ) 2 g 2 ν Z + deg ( m i ) − 4 ( t ) \left|E_i^{(t)}\right| \le K_i\left\|\Delta w^{(t)}\right\|_g^2 \deg(m_i)^{-2}(t)
=K_i\eta_t^2\left\|\nabla\mathcal{L}(w^{(t)})\right\|_g^2\deg(m_i)^{-2}(t)
\le K_iB^2\eta_t^2\mathcal{L}_{\lvert S}(w^{(t)})^2g^{2\nu_Z+\deg(m_i)-4}(t) E i ( t ) ≤ K i Δ w ( t ) g 2 deg ( m i ) − 2 ( t ) = K i η t 2 ∇ L ( w ( t ) ) g 2 deg ( m i ) − 2 ( t ) ≤ K i B 2 η t 2 L ∣ S ( w ( t ) ) 2 g 2 ν Z + d e g ( m i ) − 4 ( t )
For some constants K i K_i K i , and for some B B B that bounds ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) ) \left(\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(w^\infty)+o(1)\right) ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) ) . Now let us compute the inner product from the above expression:
⟨ ∇ m i ( w ( t ) ) , Δ w ( t ) ⟩ = η t ∑ j exp ( − m j ( w ( t ) ) ) ⟨ ∇ m i ( w ( t ) ) , ∇ m j ( w ( t ) ) ⟩ \left\langle\nabla m_i(w^{(t)}),\Delta w^{(t)}\right\rangle
=\eta_t\sum_j\exp\left(-m_j(w^{(t)})\right)\left\langle\nabla m_i(w^{(t)}),\nabla m_j(w^{(t)})\right\rangle ⟨ ∇ m i ( w ( t ) ) , Δ w ( t ) ⟩ = η t j ∑ exp ( − m j ( w ( t ) ) ) ⟨ ∇ m i ( w ( t ) ) , ∇ m j ( w ( t ) ) ⟩
So we have for i ∈ S i\in S i ∈ S
m i ( w ( t + 1 ) ) − m i ( w ( t ) ) = η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) ( ∑ j α j ( t ) ⟨ C i ∇ m i ν Z ( w ∞ ) , C j ∇ m j ν Z ( w ∞ ) ⟩ ) + o ( η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) ) m_i(w^{(t+1)})-m_i(w^{(t)})
=\eta_t\mathcal{L}_{\lvert S}(w^{(t)})g^{2\nu_Z-2}(t)
\left(\sum_j\alpha_j^{(t)}\left\langle C_i\nabla m_i^{\nu_Z}(w^\infty),C_j\nabla m_j^{\nu_Z}(w^\infty)\right\rangle\right)
+o\left(\eta_t\mathcal{L}_{\lvert S}(w^{(t)})g^{2\nu_Z-2}(t)\right) m i ( w ( t + 1 ) ) − m i ( w ( t ) ) = η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) ( j ∑ α j ( t ) ⟨ C i ∇ m i ν Z ( w ∞ ) , C j ∇ m j ν Z ( w ∞ ) ⟩ ) + o ( η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) )
Denote β ( t ) : = η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) \beta^{(t)}:=\eta_t\mathcal{L}_{\lvert S}(w^{(t)})g^{2\nu_Z-2}(t) β ( t ) := η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) , A i , j : = ⟨ C i ∇ m i ν Z ( w ∞ ) , C j ∇ m j ν Z ( w ∞ ) ⟩ A_{i,j}:=\left\langle C_i\nabla m_i^{\nu_Z}(w^\infty),C_j\nabla m_j^{\nu_Z}(w^\infty)\right\rangle A i , j := ⟨ C i ∇ m i ν Z ( w ∞ ) , C j ∇ m j ν Z ( w ∞ ) ⟩ a gram matrix, then we can re-write m i ( w ( t + 1 ) ) − m i ( w ( t ) ) = β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) m_i(w^{(t+1)})-m_i(w^{(t)})=\beta^{(t)}(A\alpha^{(t)})_i+o(\beta^{(t)}) m i ( w ( t + 1 ) ) − m i ( w ( t ) ) = β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) . Exponentiating we get
exp ( − m i ( w ( t + 1 ) ) + m i ( w ( t ) ) ) = exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) (25) \exp\left(-m_i(w^{(t+1)})+m_i(w^{(t)})\right)
=\exp\left(-\beta^{(t)}(A\alpha^{(t)})_i+o(\beta^{(t)})\right) \tag*{(25)} exp ( − m i ( w ( t + 1 ) ) + m i ( w ( t ) ) ) = exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) (25)
Now, let us calculate the α i ( t ) \alpha_i^{(t)} α i ( t ) update:
α i ( t + 1 ) = exp ( − m i ( w ( t + 1 ) ) ) ∑ j ∈ S exp ( − m j ( w ( t + 1 ) ) ) = exp ( − m i ( w ( t ) ) ) exp ( − m i ( w ( t + 1 ) ) + m i ( w ( t ) ) ) ∑ j ∈ S exp ( − m j ( w ( t ) ) ) exp ( − m j ( w ( t + 1 ) ) + m j ( w ( t ) ) ) (adding & subtracting) = exp ( − m i ( w ( t ) ) ) ( exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) ) ∑ j ∈ S exp ( − m j ( w ( t ) ) ) ( exp ( − β ( t ) ( A α ( t ) ) j + o ( β ( t ) ) ) ) (by Equation (24)) = α i ( t ) ( exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) ) ∑ j ∈ S α j ( t ) ( exp ( − β ( t ) ( A α ( t ) ) j + o ( β ( t ) ) ) ) ( exp ( − m i ( w ( t ) ) ) = α i ( t ) L ∣ S ( w ( t ) ) ) (26) \begin{aligned}
\alpha_i^{(t+1)}
&=\frac{\exp\left(-m_i(w^{(t+1)})\right)}
{\sum_{j\in S}\exp\left(-m_j(w^{(t+1)})\right)}\\
&=\frac{\exp\left(-m_i(w^{(t)})\right)\exp\left(-m_i(w^{(t+1)})+m_i(w^{(t)})\right)}
{\sum_{j\in S}\exp\left(-m_j(w^{(t)})\right)\exp\left(-m_j(w^{(t+1)})+m_j(w^{(t)})\right)}
\qquad\text{(adding \& subtracting)}\\
&=\frac{\exp\left(-m_i(w^{(t)})\right)\left(\exp\left(-\beta^{(t)}(A\alpha^{(t)})_i+o(\beta^{(t)})\right)\right)}
{\sum_{j\in S}\exp\left(-m_j(w^{(t)})\right)\left(\exp\left(-\beta^{(t)}(A\alpha^{(t)})_j+o(\beta^{(t)})\right)\right)}
\qquad\text{(by Equation (24))}\\
&=\frac{\alpha_i^{(t)}\left(\exp\left(-\beta^{(t)}(A\alpha^{(t)})_i+o(\beta^{(t)})\right)\right)}
{\sum_{j\in S}\alpha_j^{(t)}\left(\exp\left(-\beta^{(t)}(A\alpha^{(t)})_j+o(\beta^{(t)})\right)\right)}
\qquad\text{(}\exp(-m_i(w^{(t)}))=\alpha_i^{(t)}\mathcal{L}_{\lvert S}(w^{(t)})\text{)} \tag*{(26)}
\end{aligned} α i ( t + 1 ) = ∑ j ∈ S exp ( − m j ( w ( t + 1 ) ) ) exp ( − m i ( w ( t + 1 ) ) ) = ∑ j ∈ S exp ( − m j ( w ( t ) ) ) exp ( − m j ( w ( t + 1 ) ) + m j ( w ( t ) ) ) exp ( − m i ( w ( t ) ) ) exp ( − m i ( w ( t + 1 ) ) + m i ( w ( t ) ) ) (adding & subtracting) = ∑ j ∈ S exp ( − m j ( w ( t ) ) ) ( exp ( − β ( t ) ( A α ( t ) ) j + o ( β ( t ) ) ) ) exp ( − m i ( w ( t ) ) ) ( exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) ) (by Equation (24)) = ∑ j ∈ S α j ( t ) ( exp ( − β ( t ) ( A α ( t ) ) j + o ( β ( t ) ) ) ) α i ( t ) ( exp ( − β ( t ) ( A α ( t ) ) i + o ( β ( t ) ) ) ) ( exp ( − m i ( w ( t ) )) = α i ( t ) L ∣ S ( w ( t ) ) ) (26)
Note that this is an Normalized Exponentiated Gradient update with perturbations.
Lemma E.4. Let α ( t ) \alpha^{(t)} α ( t ) be a sequence of vectors on the simplex, let α ∗ \alpha^* α ∗ be some vector on the simplex, and let Φ ( α ) \Phi(\alpha) Φ ( α ) be some convex function to be optimized. Denote ℓ ( t ) = ∇ Φ ( α ( t ) ) \ell^{(t)}=\nabla\Phi(\alpha^{(t)}) ℓ ( t ) = ∇Φ ( α ( t ) ) and assume
α i ( t + 1 ) = α i ( t ) exp ( − β ( t ) ℓ i ( t ) + ε i ( t ) ) ∑ j α j ( t ) exp ( − β ( t ) ℓ j ( t ) + ε j ( t ) ) \alpha_i^{(t+1)}
=\frac{\alpha_i^{(t)}\exp\left(-\beta^{(t)}\ell_i^{(t)}+\varepsilon_i^{(t)}\right)}
{\sum_j\alpha_j^{(t)}\exp\left(-\beta^{(t)}\ell_j^{(t)}+\varepsilon_j^{(t)}\right)} α i ( t + 1 ) = ∑ j α j ( t ) exp ( − β ( t ) ℓ j ( t ) + ε j ( t ) ) α i ( t ) exp ( − β ( t ) ℓ i ( t ) + ε i ( t ) ) For ε ( t ) → 0 \varepsilon^{(t)}\to0 ε ( t ) → 0 , then,
KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ( Φ ( α ∗ ) − Φ ( α ( t ) ) ) + ( β ( t ) Δ ( t ) ) 2 8 + 2 ∥ ε ( t ) ∥ ∞ (27) \operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t+1)}\right)
-\operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t)}\right)
\le\beta^{(t)}\left(\Phi(\alpha^*)-\Phi(\alpha^{(t)})\right)
+\frac{(\beta^{(t)}\Delta^{(t)})^2}{8}
+2\left\|\varepsilon^{(t)}\right\|_\infty \tag*{(27)} KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ( Φ ( α ∗ ) − Φ ( α ( t ) ) ) + 8 ( β ( t ) Δ ( t ) ) 2 + 2 ε ( t ) ∞ (27) Where Δ ( t ) : = max i ℓ i ( t ) − min i ℓ i ( t ) \Delta^{(t)}:=\max_i\ell_i^{(t)}-\min_i\ell_i^{(t)} Δ ( t ) := max i ℓ i ( t ) − min i ℓ i ( t )
Proof. First, we will show
KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + ( β ( t ) Δ ( t ) ) 2 8 + 2 ∥ ε ( t ) ∥ ∞ (28) \operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t+1)}\right)
-\operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t)}\right)
\le\beta^{(t)}\left\langle\ell^{(t)},\alpha^*-\alpha^{(t)}\right\rangle
+\frac{(\beta^{(t)}\Delta^{(t)})^2}{8}
+2\left\|\varepsilon^{(t)}\right\|_\infty \tag*{(28)} KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + 8 ( β ( t ) Δ ( t ) ) 2 + 2 ε ( t ) ∞ (28)
For some generic ℓ ( t ) \ell^{(t)} ℓ ( t ) . Denote Z t : = ∑ j α j ( t ) exp ( − β ( t ) ℓ j ( t ) + ε j ( t ) ) Z_t := \sum_j \alpha_j^{(t)} \exp\left(-\beta^{(t)}\ell_j^{(t)}+\varepsilon_j^{(t)}\right) Z t := ∑ j α j ( t ) exp ( − β ( t ) ℓ j ( t ) + ε j ( t ) ) , so
α i ( t + 1 ) = α i ( t ) exp ( − β ( t ) ℓ i ( t ) + ε i ( t ) ) Z t \alpha_i^{(t+1)} = \frac{\alpha_i^{(t)}\exp\left(-\beta^{(t)}\ell_i^{(t)}+\varepsilon_i^{(t)}\right)}{Z_t} α i ( t + 1 ) = Z t α i ( t ) exp ( − β ( t ) ℓ i ( t ) + ε i ( t ) )
Taking log over the update equation, we get log α i ( t + 1 ) = log ( α i ( t ) ) − β ( t ) ℓ i ( t ) + ε i ( t ) − log ( Z t ) \log\alpha_i^{(t+1)}=\log(\alpha_i^{(t)})-\beta^{(t)}\ell_i^{(t)}+\varepsilon_i^{(t)}-\log(Z_t) log α i ( t + 1 ) = log ( α i ( t ) ) − β ( t ) ℓ i ( t ) + ε i ( t ) − log ( Z t ) . Hence,
log α i ( t ) α i ( t + 1 ) = β ( t ) ℓ i ( t ) − ε i ( t ) + log ( Z t ) (29) \log\frac{\alpha_i^{(t)}}{\alpha_i^{(t+1)}}=\beta^{(t)}\ell_i^{(t)}-\varepsilon_i^{(t)}+\log(Z_t) \tag*{(29)} log α i ( t + 1 ) α i ( t ) = β ( t ) ℓ i ( t ) − ε i ( t ) + log ( Z t ) (29)
So overall,
KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) = ∑ i α i ∗ log α i ∗ α i ( t + 1 ) − ∑ i α i ∗ log α i ∗ α i ( t ) = ∑ i α i ∗ ( log α i ∗ α i ( t + 1 ) + log α i ( t ) α i ∗ ) = ∑ i α i ∗ log α i ( t ) α i ( t + 1 ) = ∑ i α i ∗ ( β ( t ) ℓ i ( t ) − ε i ( t ) + log ( Z t ) ) (eq. x r e f ‘ ( 28 ) ‘ ) = β ( t ) ⟨ α ∗ , ℓ ( t ) ⟩ − ⟨ α ∗ , ε ( t ) ⟩ + log Z t (30) \begin{aligned}
\operatorname{KL}\left(\alpha^* \mathbin\Vert\alpha^{(t+1)}\right)-\operatorname{KL}\left(\alpha^* \mathbin\Vert\alpha^{(t)}\right)=\sum_i\alpha_i^*\log\frac{\alpha_i^*}{\alpha_i^{(t+1)}}-\sum_i\alpha_i^*\log\frac{\alpha_i^*}{\alpha_i^{(t)}} \\
&=\sum_i\alpha_i^*\left(\log\frac{\alpha_i^*}{\alpha_i^{(t+1)}}+\log\frac{\alpha_i^{(t)}}{\alpha_i^*}\right) \\
&=\sum_i\alpha_i^*\log\frac{\alpha_i^{(t)}}{\alpha_i^{(t+1)}} \\
&=\sum_i\alpha_i^*\left(\beta^{(t)}\ell_i^{(t)}-\varepsilon_i^{(t)}+\log(Z_t)\right) \qquad\text{(eq. }{xref}`(28)`\text{)} \\
&=\beta^{(t)}\left\langle\alpha^*,\ell^{(t)}\right\rangle-\left\langle\alpha^*,\varepsilon^{(t)}\right\rangle+\log Z_t \tag*{(30)}
\end{aligned} KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) = i ∑ α i ∗ log α i ( t + 1 ) α i ∗ − i ∑ α i ∗ log α i ( t ) α i ∗ = i ∑ α i ∗ ( log α i ( t + 1 ) α i ∗ + log α i ∗ α i ( t ) ) = i ∑ α i ∗ log α i ( t + 1 ) α i ( t ) = i ∑ α i ∗ ( β ( t ) ℓ i ( t ) − ε i ( t ) + log ( Z t ) ) (eq. x r e f ‘ ( 28 ) ‘ ) = β ( t ) ⟨ α ∗ , ℓ ( t ) ⟩ − ⟨ α ∗ , ε ( t ) ⟩ + log Z t (30)
Now let us bound log Z t \log Z_t log Z t . Denote μ t : = ⟨ α ( t ) , ℓ ( t ) ⟩ \mu_t:=\left\langle\alpha^{(t)},\ell^{(t)}\right\rangle μ t := ⟨ α ( t ) , ℓ ( t ) ⟩ (We can think as μ t = E i ℓ i ( t ) \mu_t=\mathbb{E}_i\ell_i^{(t)} μ t = E i ℓ i ( t ) , where i i i is distributed by α ( t ) \alpha^{(t)} α ( t ) – a simplex vector). Then adding and subtracting μ t \mu_t μ t we get:
Z t = ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) exp ( ε i ( t ) ) exp ( − β ( t ) μ t ) Z_t=\sum_i\alpha_i^{(t)}\exp\left(-\beta^{(t)}\left(\ell_i^{(t)}-\mu_t\right)\right)\exp\left(\varepsilon_i^{(t)}\right)\exp\left(-\beta^{(t)}\mu_t\right) Z t = i ∑ α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) exp ( ε i ( t ) ) exp ( − β ( t ) μ t )
Hence,
log Z t = − β ( t ) μ t + log ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) exp ( ε i ( t ) ) ≤ − β ( t ) μ t + log ( max i exp ( ε i ( t ) ) ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) ) = − β ( t ) μ t + ∥ ε ( t ) ∥ ∞ + log ( ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) ) \begin{aligned}
\log Z_t=-\beta^{(t)}\mu_t+\log\sum_i\alpha_i^{(t)}\exp\left(-\beta^{(t)}\left(\ell_i^{(t)}-\mu_t\right)\right)\exp\left(\varepsilon_i^{(t)}\right) \\
&\leq-\beta^{(t)}\mu_t+\log\left(\max_i\exp\left(\varepsilon_i^{(t)}\right)\sum_i\alpha_i^{(t)}\exp\left(-\beta^{(t)}\left(\ell_i^{(t)}-\mu_t\right)\right)\right) \\
&=-\beta^{(t)}\mu_t+\left\|\varepsilon^{(t)}\right\|_\infty+\log\left(\sum_i\alpha_i^{(t)}\exp\left(-\beta^{(t)}\left(\ell_i^{(t)}-\mu_t\right)\right)\right)
\end{aligned} log Z t = − β ( t ) μ t + log i ∑ α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) exp ( ε i ( t ) ) ≤ − β ( t ) μ t + log ( i max exp ( ε i ( t ) ) i ∑ α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) ) = − β ( t ) μ t + ε ( t ) ∞ + log ( i ∑ α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) )
Now again, since α ( t ) \alpha^{(t)} α ( t ) is a simplex vector (or a probability distribution) we can use Hoeffding’s Lemma (Theorem F.3 ), and get log ( ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) ) ≤ ( β ( t ) Δ ( t ) ) 2 8 \log\left(\sum_i\alpha_i^{(t)}\exp\left(-\beta^{(t)}\left(\ell_i^{(t)}-\mu_t\right)\right)\right)\leq\frac{\left(\beta^{(t)}\Delta^{(t)}\right)^2}{8} log ( ∑ i α i ( t ) exp ( − β ( t ) ( ℓ i ( t ) − μ t ) ) ) ≤ 8 ( β ( t ) Δ ( t ) ) 2 . Hence,
log Z t ≤ − β ( t ) μ t + ∥ ε ( t ) ∥ ∞ + ( β ( t ) Δ ( t ) ) 2 8 \log Z_t\leq-\beta^{(t)}\mu_t+\left\|\varepsilon^{(t)}\right\|_\infty+\frac{\left(\beta^{(t)}\Delta^{(t)}\right)^2}{8} log Z t ≤ − β ( t ) μ t + ε ( t ) ∞ + 8 ( β ( t ) Δ ( t ) ) 2
Plugging it in Equation (30) we get
KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + ∥ ε ( t ) ∥ ∞ + ( β ( t ) Δ ( t ) ) 2 8 − ⟨ α ∗ , ε ( t ) ⟩ ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + 2 ∥ ε ( t ) ∥ ∞ + ( β ( t ) Δ ( t ) ) 2 8 \begin{aligned}
\operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t+1)}\right)-\operatorname{KL}\left(\alpha^*\mathbin\Vert\alpha^{(t)}\right)\leq\beta^{(t)}\left\langle\ell^{(t)},\alpha^*-\alpha^{(t)}\right\rangle+\left\|\varepsilon^{(t)}\right\|_\infty+\frac{\left(\beta^{(t)}\Delta^{(t)}\right)^2}{8}-\left\langle\alpha^*,\varepsilon^{(t)}\right\rangle\\
&\leq\beta^{(t)}\left\langle\ell^{(t)},\alpha^*-\alpha^{(t)}\right\rangle+2\left\|\varepsilon^{(t)}\right\|_\infty+\frac{\left(\beta^{(t)}\Delta^{(t)}\right)^2}{8}
\end{aligned} KL ( α ∗ ∥ α ( t + 1 ) ) − KL ( α ∗ ∥ α ( t ) ) ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + ε ( t ) ∞ + 8 ( β ( t ) Δ ( t ) ) 2 − ⟨ α ∗ , ε ( t ) ⟩ ≤ β ( t ) ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ + 2 ε ( t ) ∞ + 8 ( β ( t ) Δ ( t ) ) 2
Finally, take ℓ ( t ) = ∇ Φ ( α ( t ) ) \ell^{(t)}=\nabla\Phi(\alpha^{(t)}) ℓ ( t ) = ∇Φ ( α ( t ) ) . From the convexity of Φ \Phi Φ we get Φ ( α ∗ ) − Φ ( α ( t ) ) ≥ ⟨ ∇ Φ ( α ( t ) ) , α ∗ − α ( t ) ⟩ = ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ \Phi(\alpha^*)-\Phi(\alpha^{(t)})\geq\left\langle\nabla\Phi(\alpha^{(t)}),\alpha^*-\alpha^{(t)}\right\rangle=\left\langle\ell^{(t)},\alpha^*-\alpha^{(t)}\right\rangle Φ ( α ∗ ) − Φ ( α ( t ) ) ≥ ⟨ ∇Φ ( α ( t ) ) , α ∗ − α ( t ) ⟩ = ⟨ ℓ ( t ) , α ∗ − α ( t ) ⟩ . Substitute to get what’s required. □
Lemma E.5. 1. β ( t ) → 0 \beta^{(t)} \to0 β ( t ) → 0
∑ t = 0 ∞ β ( t ) = ∞ \sum_{t=0}^{\infty}\beta^{(t)} = \infty ∑ t = 0 ∞ β ( t ) = ∞
Proof. For Item 1, recall that β ( t ) = η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) \beta^{(t)} = \eta_t\mathcal{L}|_S(\mathbf{w}^{(t)})g^{2\nu_Z-2}(t) β ( t ) = η t L ∣ S ( w ( t ) ) g 2 ν Z − 2 ( t ) . So while g 2 ν Z − 2 ( t ) g^{2\nu_Z-2}(t) g 2 ν Z − 2 ( t ) is polynomial in ∥ w ( t ) ∥ = g ( t ) \|\mathbf{w}^{(t)}\| = g(t) ∥ w ( t ) ∥ = g ( t ) , L ∣ S ( w ( t ) ) \mathcal{L}|_S(\mathbf{w}^{(t)}) L ∣ S ( w ( t ) ) is exponentially decaying in g ( t ) g(t) g ( t ) . Since g ( t ) → ∞ g(t) \to\infty g ( t ) → ∞ we have β ( t ) → 0 \beta^{(t)} \to0 β ( t ) → 0 . For Item 2 let us analyze the radial increments Δ g ν Z ( t ) : = g ν Z ( t + 1 ) − g ν Z ( t ) \Delta g^{\nu_Z}(t) := g^{\nu_Z}(t+1)-g^{\nu_Z}(t) Δ g ν Z ( t ) := g ν Z ( t + 1 ) − g ν Z ( t ) . Recall Δ w ( t ) = − η t ∇ L ( w ( t ) ) \Delta\mathbf{w}^{(t)}=-\eta_t\nabla\mathcal{L}(\mathbf{w}^{(t)}) Δ w ( t ) = − η t ∇ L ( w ( t ) ) . Using tailor expansion on the norm we get
g ( t + 1 ) = ∥ w ( t ) + Δ w ( t ) ∥ = g ( t ) + ⟨ w ‾ ( t ) , Δ w ( t ) ⟩ + o ( ∥ Δ w ( t ) ∥ 2 g ( t ) ) (31) g(t+1)=\left\|\mathbf{w}^{(t)}+\Delta\mathbf{w}^{(t)}\right\|=g(t)+\left\langle\overline{\mathbf{w}}^{(t)},\Delta\mathbf{w}^{(t)}\right\rangle+o\left(\frac{\left\|\Delta\mathbf{w}^{(t)}\right\|^2}{g(t)}\right) \tag*{(31)} g ( t + 1 ) = w ( t ) + Δ w ( t ) = g ( t ) + ⟨ w ( t ) , Δ w ( t ) ⟩ + o ( g ( t ) Δ w ( t ) 2 ) (31)
Now using taylor expansion on g ( t ) ↦ g ( t ) ν Z g(t)\mapsto g(t)^{\nu_Z} g ( t ) ↦ g ( t ) ν Z for a small increment
g ( t + 1 ) ν Z = g ( t ) ν Z + ν Z g ( t ) ν Z − 1 ( g ( t + 1 ) − g ( t ) ) + O ( g ( t ) ν Z − 2 ( g ( t + 1 ) − g ( t ) ) 2 ) = g ( t ) ν Z + ν Z g ( t ) ν Z − 1 ⟨ w ‾ ( t ) , Δ w ( t ) ⟩ + O ( g ( t ) ν Z − 2 ∥ Δ w ( t ) ∥ 2 ) \begin{aligned}
g(t+1)^{\nu_Z}=g(t)^{\nu_Z}+\nu_Zg(t)^{\nu_Z-1}(g(t+1)-g(t))+O\left(g(t)^{\nu_Z-2}(g(t+1)-g(t))^2\right) \\
&= g(t)^{\nu_Z}+\nu_Zg(t)^{\nu_Z-1}\left\langle\overline{\mathbf{w}}^{(t)},\Delta\mathbf{w}^{(t)}\right\rangle+O\left(g(t)^{\nu_Z-2}\left\|\Delta\mathbf{w}^{(t)}\right\|^2\right)
\end{aligned} g ( t + 1 ) ν Z = g ( t ) ν Z + ν Z g ( t ) ν Z − 1 ( g ( t + 1 ) − g ( t )) + O ( g ( t ) ν Z − 2 ( g ( t + 1 ) − g ( t ) ) 2 ) = g ( t ) ν Z + ν Z g ( t ) ν Z − 1 ⟨ w ( t ) , Δ w ( t ) ⟩ + O ( g ( t ) ν Z − 2 Δ w ( t ) 2 )
(by Equation (31) )
Now note that
ν Z g ( t ) ν Z − 1 ⟨ w ‾ ( t ) , Δ w ( t ) ⟩ = η t L ∣ S ( w ( t ) ) ν Z g ( t ) 2 ν Z − 2 ( ⟨ w ( t ) ∥ w ( t ) ∥ , ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) ⟩ + o ( 1 ) ) = β ( t ) ν Z ( ⟨ w ( t ) ∥ w ( t ) ∥ , ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) ⟩ + o ( 1 ) ) \begin{aligned}
\nu_Zg(t)^{\nu_Z-1}\left\langle\overline{\mathbf{w}}^{(t)},\Delta\mathbf{w}^{(t)}\right\rangle
=\eta_t\mathcal{L}|_S(\mathbf{w}^{(t)})\nu_Zg(t)^{2\nu_Z-2}\left(\left\langle\frac{\mathbf{w}^{(t)}}{\|\mathbf{w}^{(t)}\|},\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})\right\rangle+o(1)\right) \\
&=\beta^{(t)}\nu_Z\left(\left\langle\frac{\mathbf{w}^{(t)}}{\|\mathbf{w}^{(t)}\|},\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})\right\rangle+o(1)\right)
\end{aligned} ν Z g ( t ) ν Z − 1 ⟨ w ( t ) , Δ w ( t ) ⟩ = η t L ∣ S ( w ( t ) ) ν Z g ( t ) 2 ν Z − 2 ( ⟨ ∥ w ( t ) ∥ w ( t ) , i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ∞ ) ⟩ + o ( 1 ) ) = β ( t ) ν Z ( ⟨ ∥ w ( t ) ∥ w ( t ) , i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ∞ ) ⟩ + o ( 1 ) )
(def β ( t ) \beta^{(t)} β ( t ) )
≤ β ( t ) ν Z ( ∥ ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) ∥ + o ( 1 ) ) \leq\beta^{(t)}\nu_Z\left(\left\|\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})\right\|+o(1)\right) ≤ β ( t ) ν Z ( i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) )
(Cauchy-Schwartz)
≤ β ( t ) ν Z C M \leq\beta^{(t)}\nu_Z C_M ≤ β ( t ) ν Z C M
For C M = 1 + max i ∈ S ∥ C i ∇ m i ν Z ( w ∞ ) ∥ C_M=1+\max_{i\in S}\|C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})\| C M = 1 + max i ∈ S ∥ C i ∇ m i ν Z ( w ∞ ) ∥ . As for the remainder:
g ν Z − 2 ( t ) ∥ Δ w ( t ) ∥ 2 = g ν Z − 2 ( t ) ∥ L ∣ S ( w ( t ) ) η t g ν Z − 1 ( t ) ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) ) ∥ 2 ≤ g ν Z − 2 ( t ) ( β ( t ) ) 2 ∥ ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) ∥ 2 ≤ g ν Z − 2 ( t ) ( β ( t ) ) 2 C M 2 = o ( β ( t ) ) \begin{aligned}
g^{\nu_Z-2}(t)\left\|\Delta\mathbf{w}^{(t)}\right\|^2
=g^{\nu_Z-2}(t)\left\|\mathcal{L}|_S(\mathbf{w}^{(t)})\eta_tg^{\nu_Z-1}(t)\left(\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})+o(1)\right)\right\|^2 \\
&\leq g^{\nu_Z-2}(t)\left(\beta^{(t)}\right)^2\left\|\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})+o(1)\right\|^2 \\
&\leq g^{\nu_Z-2}(t)\left(\beta^{(t)}\right)^2 C_M^2 \\
&=o(\beta^{(t)})
\end{aligned} g ν Z − 2 ( t ) Δ w ( t ) 2 = g ν Z − 2 ( t ) L ∣ S ( w ( t ) ) η t g ν Z − 1 ( t ) ( i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) ) 2 ≤ g ν Z − 2 ( t ) ( β ( t ) ) 2 i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ∞ ) + o ( 1 ) 2 ≤ g ν Z − 2 ( t ) ( β ( t ) ) 2 C M 2 = o ( β ( t ) )
Where the last inequality is due to β ( t ) \beta^{(t)} β ( t ) decaying exponentially with g ( t ) g(t) g ( t ) . Combining both bounds we get
g ( t + 1 ) ν Z − g ( t ) ν Z ≤ ν Z C M β ( t ) + o ( β ( t ) ) ≤ C ~ β ( t ) g(t+1)^{\nu_Z}-g(t)^{\nu_Z}\leq\nu_ZC_M\beta^{(t)}+o(\beta^{(t)})\leq\widetilde{C}\beta^{(t)} g ( t + 1 ) ν Z − g ( t ) ν Z ≤ ν Z C M β ( t ) + o ( β ( t ) ) ≤ C β ( t )
For some C ~ > 0 \widetilde{C}>0 C > 0 . Now summing from large enough t t t to ∞ \infty ∞ we get ∑ t = t 0 ∞ ( g ( t + 1 ) ν Z − g ( t ) ν Z ) = ∞ \sum_{t=t_0}^{\infty}\left(g(t+1)^{\nu_Z}-g(t)^{\nu_Z}\right)=\infty ∑ t = t 0 ∞ ( g ( t + 1 ) ν Z − g ( t ) ν Z ) = ∞ and hence also ∑ t = t 0 ∞ C ~ β ( t ) = ∞ \sum_{t=t_0}^{\infty}\widetilde{C}\beta^{(t)}=\infty ∑ t = t 0 ∞ C β ( t ) = ∞ . □
Using Theorem E.4 with Φ ( α ) = 1 2 α ⊤ A α = 1 2 ∥ ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) ∥ 2 \Phi(\alpha)=\frac{1}{2}\alpha^\top A\alpha=\frac{1}{2}\left\|\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\mathbf{w}^{\infty})\right\|^2 Φ ( α ) = 2 1 α ⊤ A α = 2 1 ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ∞ ) 2 , and Item 1 of Theorem E.5 to substitute ε ( t ) = o ( β ( t ) ) \varepsilon^{(t)}=o(\beta^{(t)}) ε ( t ) = o ( β ( t ) ) , we get our update Equation (26) . Hence summing up Equation (27) from t = 0 t=0 t = 0 to T − 1 T-1 T − 1 , using α ∗ \alpha^* α ∗ some minimizer of Φ \Phi Φ , we get
∑ t = 0 T − 1 β ( t ) ( Φ ( α ( t ) ) − Φ ( α ∗ ) ) ≤ K L ( α ∗ ∥ α ( 0 ) ) + ∑ t = 0 T − 1 ( β ( t ) Δ ( t ) ) 2 / 8 + 2 ∑ t = 0 T − 1 ∥ ε ( t ) ∥ ∞ \sum_{t=0}^{T-1}\beta^{(t)}\left(\Phi(\alpha^{(t)})-\Phi(\alpha^*)\right)\leq\mathrm{KL}\left(\alpha^*\parallel\alpha^{(0)}\right)+\sum_{t=0}^{T-1}\left(\beta^{(t)}\Delta^{(t)}\right)^2/8+2\sum_{t=0}^{T-1}\left\|\varepsilon^{(t)}\right\|_\infty t = 0 ∑ T − 1 β ( t ) ( Φ ( α ( t ) ) − Φ ( α ∗ ) ) ≤ KL ( α ∗ ∥ α ( 0 ) ) + t = 0 ∑ T − 1 ( β ( t ) Δ ( t ) ) 2 /8 + 2 t = 0 ∑ T − 1 ε ( t ) ∞
Note that by the definition of Δ w ( t ) \Delta w^{(t)} Δ w ( t ) we have that ∑ t = 0 T − 1 ( β ( t ) Δ ( t ) ) 2 8 = o ( ∑ t = 0 T − 1 β ( t ) ) \sum_{t=0}^{T-1}\frac{(\beta^{(t)}\Delta^{(t)})^2}{8}=o(\sum_{t=0}^{T-1}\beta^{(t)}) ∑ t = 0 T − 1 8 ( β ( t ) Δ ( t ) ) 2 = o ( ∑ t = 0 T − 1 β ( t ) ) and by Equation (26) we have 2 ∑ t = 0 T − 1 ∥ ε ( t ) ∥ ∞ 2 = o ( ∑ t = 0 T − 1 β ( t ) ) 2\sum_{t=0}^{T-1}\|\varepsilon^{(t)}\|_{\infty}^{2}=o(\sum_{t=0}^{T-1}\beta^{(t)}) 2 ∑ t = 0 T − 1 ∥ ε ( t ) ∥ ∞ 2 = o ( ∑ t = 0 T − 1 β ( t ) ) , so dividing both sides by ∑ t = 0 T − 1 β ( t ) \sum_{t=0}^{T-1}\beta^{(t)} ∑ t = 0 T − 1 β ( t ) we get
∑ t = 0 T − 1 β ( t ) ( Φ ( α ( t ) ) − Φ ( α ∗ ) ) ∑ t = 0 T − 1 β ( t ) → 0 \frac{\sum_{t=0}^{T-1}\beta^{(t)}\left(\Phi(\alpha^{(t)})-\Phi(\alpha^*)\right)}{\sum_{t=0}^{T-1}\beta^{(t)}}\to0 ∑ t = 0 T − 1 β ( t ) ∑ t = 0 T − 1 β ( t ) ( Φ ( α ( t ) ) − Φ ( α ∗ ) ) → 0
Thereby lim inf t → ∞ ( Φ ( α ( t ) ) − Φ ( α ∗ ) ) = 0 \liminf_{t\to\infty}(\Phi(\alpha^{(t)})-\Phi(\alpha^*))=0 lim inf t → ∞ ( Φ ( α ( t ) ) − Φ ( α ∗ )) = 0 , so there exists a subsequence t k t_k t k s.t. Φ ( α ( t k ) ) → Φ ( α ∗ ) \Phi(\alpha^{(t_k)})\to\Phi(\alpha^*) Φ ( α ( t k ) ) → Φ ( α ∗ ) .
Notice that since ∀ i ∈ S \forall i\in S ∀ i ∈ S m i m_i m i has a minimal effective degree and the minimal margin among those with the minimal effective degree, hence we have that
( w ˉ ∞ ) ⊤ ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) = ∑ i ∈ S α i ( t ) ν Z C i m i ν Z ( w ˉ ∞ ) = ν Z γ min (\bar{w}^{\infty})^{\top}\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})=\sum_{i\in S}\alpha_i^{(t)}\nu_Z C_i m_i^{\nu_Z}(\bar{w}^{\infty})=\nu_Z\gamma_{\min} ( w ˉ ∞ ) ⊤ i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) = i ∈ S ∑ α i ( t ) ν Z C i m i ν Z ( w ˉ ∞ ) = ν Z γ m i n
In other words,
∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) = ( ν Z γ min ) w ˉ ∞ + v ( t ) \sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})=(\nu_Z\gamma_{\min})\bar{w}^{\infty}+v^{(t)} i ∈ S ∑ α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) = ( ν Z γ m i n ) w ˉ ∞ + v ( t )
For v ( t ) ⊥ w ˉ ∞ v^{(t)}\perp\bar{w}^{\infty} v ( t ) ⊥ w ˉ ∞ with a constant w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ element. So minimizing ∥ ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) ∥ 2 \left\|\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\right\|^2 ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) 2 is akin to minimizing ∥ v ( t ) ∥ 2 \|v^{(t)}\|^2 ∥ v ( t ) ∥ 2 for v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ ) } v\in\operatorname{Conv}_{i\in S}\{C_iP_{\perp}\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\} v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ )} where P ⊥ P_{\perp} P ⊥ the projection map to the space orthogonal to w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ .
Lemma E.6. min v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ ) } ∥ v ∥ 2 = 0 \min_{v\in\operatorname{Conv}_{i\in S}\{C_iP_{\perp}\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\}}\|v\|^2=0 min v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ )} ∥ v ∥ 2 = 0
Proof. Assume towards contradiction that 0 ∉ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ ) } 0\notin\operatorname{Conv}_{i\in S}\{C_iP_{\perp}\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\} 0 ∈ / Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ )} , then by the separating hyperplane theorem there exists δ > 0 \delta>0 δ > 0 and q ⊥ w ˉ ∞ q\perp\bar{w}^{\infty} q ⊥ w ˉ ∞ (since the whole convex hull lives orthogonally to w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ ) s.t. q ⊤ v ≥ δ q^{\top}v\ge\delta q ⊤ v ≥ δ for any v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ ) } v\in\operatorname{Conv}_{i\in S}\{C_iP_{\perp}\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\} v ∈ Conv i ∈ S { C i P ⊥ ∇ m i ν Z ( w ˉ ∞ )} . Recall that Δ w ( t ) = w ( t + 1 ) − w ( t ) = η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) + o ( 1 ) ) \Delta w^{(t)}=w^{(t+1)}-w^{(t)}=\eta_t\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t)\left(\sum_{i\in S}\alpha_i^{(t)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})+o(1)\right) Δ w ( t ) = w ( t + 1 ) − w ( t ) = η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) ( ∑ i ∈ S α i ( t ) C i ∇ m i ν Z ( w ˉ ∞ ) + o ( 1 ) ) , hence for large enough t 0 t_0 t 0 , we have for any t ≥ t 0 t\ge t_0 t ≥ t 0 that
⟨ q , Δ w ( t ) ⟩ ≥ ( δ / 2 ) η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) \langle q,\Delta w^{(t)}\rangle\ge(\delta/2)\eta_t\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t) ⟨ q , Δ w ( t ) ⟩ ≥ ( δ /2 ) η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t )
⟨ w ˉ ∞ , Δ w ( t ) ⟩ ≤ 3 2 ν Z γ min η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) \langle\bar{w}^{\infty},\Delta w^{(t)}\rangle\le\frac{3}{2}\nu_Z\gamma_{\min}\eta_t\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t) ⟨ w ˉ ∞ , Δ w ( t ) ⟩ ≤ 2 3 ν Z γ m i n η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t )
Summing up we get
⟨ q , w ( t ) − w ( t 0 ) ⟩ ⟨ w ˉ ∞ , w ( t ) − w ( t 0 ) ⟩ ≥ δ ∑ t = t 0 T η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) 3 ν Z γ min ∑ t = t 0 T η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) → T → ∞ δ 3 ν Z γ min > 0 \frac{\langle q,w^{(t)}-w^{(t_0)}\rangle}{\langle\bar{w}^{\infty},w^{(t)}-w^{(t_0)}\rangle}\ge\frac{\delta\sum_{t=t_0}^{T}\eta_t\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t)}{3\nu_Z\gamma_{\min}\sum_{t=t_0}^{T}\eta_t\mathcal{L}_{|S}(w^{(t)})g^{\nu_Z-1}(t)}\xrightarrow{T\to\infty}\frac{\delta}{3\nu_Z\gamma_{\min}}>0 ⟨ w ˉ ∞ , w ( t ) − w ( t 0 ) ⟩ ⟨ q , w ( t ) − w ( t 0 ) ⟩ ≥ 3 ν Z γ m i n ∑ t = t 0 T η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) δ ∑ t = t 0 T η t L ∣ S ( w ( t ) ) g ν Z − 1 ( t ) T → ∞ 3 ν Z γ m i n δ > 0
Contradicting w ˉ ( t ) → w ˉ ∞ \bar{w}^{(t)}\to\bar{w}^{\infty} w ˉ ( t ) → w ˉ ∞ , since we got that w ( t ) − w ( t 0 ) ∥ w ( t ) ∥ → w ˉ ∞ \frac{w^{(t)}-w^{(t_0)}}{\|w^{(t)}\|}\to\bar{w}^{\infty} ∥ w ( t ) ∥ w ( t ) − w ( t 0 ) → w ˉ ∞ has a nonvanishing projection on q ⊥ w ˉ ∞ q\perp\bar{w}^{\infty} q ⊥ w ˉ ∞ . □
From Theorem E.6 we have that the subsequence t k t_k t k has ∑ i ∈ S α i ( t k ) C i ∇ m i ν Z ( w ˉ ∞ ) → ν Z γ min w ˉ ∞ \sum_{i\in S}\alpha_i^{(t_k)}C_i\nabla m_i^{\nu_Z}(\bar{w}^{\infty})\to\nu_Z\gamma_{\min}\bar{w}^{\infty} ∑ i ∈ S α i ( t k ) C i ∇ m i ν Z ( w ˉ ∞ ) → ν Z γ m i n w ˉ ∞ . Denote the subsequence limits α i ( t k ) → α i t ∞ \alpha_i^{(t_k)}\to\alpha_i^{t_\infty} α i ( t k ) → α i t ∞ , and ∇ L ( w ( t ∞ ) ) : = lim k → ∞ ∇ L ( w ( t k ) ) ∥ ∇ L ( w ( t k ) ) ∥ \nabla\mathcal{L}(w^{(t_\infty)}):=\lim_{k\to\infty}\frac{\nabla\mathcal{L}(w^{(t_k)})}{\|\nabla\mathcal{L}(w^{(t_k)})\|} ∇ L ( w ( t ∞ ) ) := lim k → ∞ ∥∇ L ( w ( t k ) ) ∥ ∇ L ( w ( t k ) ) . So overall we have ∇ L ( w ( t ∞ ) ) ∝ ∑ i ∈ S α i t ∞ C i ∇ P i ν Z ( w ˉ ∞ ) ⊤ y i x i ∝ w ˉ ∞ \nabla\mathcal{L}(w^{(t_\infty)})\propto\sum_{i\in S}\alpha_i^{t_\infty}C_i\nabla P^{\nu_Z}_i(\bar{w}^{\infty})^{\top}y_ix_i\propto\bar{w}^{\infty} ∇ L ( w ( t ∞ ) ) ∝ ∑ i ∈ S α i t ∞ C i ∇ P i ν Z ( w ˉ ∞ ) ⊤ y i x i ∝ w ˉ ∞ for μ i ≥ 0 \mu_i\ge0 μ i ≥ 0 . We use α i t ∞ \alpha_i^{t_\infty} α i t ∞ as the dual certificate (letting ∀ i ∉ S α i t ∞ = 0 \forall i\notin S\ \alpha_i^{t_\infty}=0 ∀ i ∈ / S α i t ∞ = 0 ), that satisfies dual feasibility and complementary slackness. Therefore w ˉ ∞ ∝ ∑ i = 1 n α i t ∞ C i ∇ P ν i ( w ˉ ∞ ) ⊤ x i y i \bar{w}^{\infty}\propto\sum_{i=1}^{n}\alpha_i^{t_\infty}C_i\nabla P^{\nu_i}(\bar{w}^{\infty})^{\top}x_i y_i w ˉ ∞ ∝ ∑ i = 1 n α i t ∞ C i ∇ P ν i ( w ˉ ∞ ) ⊤ x i y i . Scaling w ˉ ∞ \bar{w}^{\infty} w ˉ ∞ s.t. γ min = 1 \gamma_{\min}=1 γ m i n = 1 , we conclude the proof.
Auxiliary Material
Theorem F.1 (Theorem 6.1 from Cai et al. [2 ] , simplified). Under Assumption 4 , the expression
γ ( t ) : = exp ( Φ ( exp ( p ( ∥ w ( t ) ∥ ) L ( w ( t ) ) / n ) ) ) ∥ w ( t ) ∥ M \gamma(t):=\frac{\exp\left(\Phi\left(\exp\left(p\left(\|w^{(t)}\|\right)\mathcal{L}(w^{(t)})/n\right)\right)\right)}{\|w^{(t)}\|^M} γ ( t ) := ∥ w ( t ) ∥ M exp ( Φ ( exp ( p ( ∥ w ( t ) ∥ ) L ( w ( t ) ) / n ) ) )
Where M = deg ( q ) M=\deg(q) M = deg ( q ) is the degree of the predictor, and Φ ( x ) = log ( log ( 1 / n x ) ) + 2 / log ( n x ) \Phi(x)=\log(\log(1/nx))+2/\log(nx) Φ ( x ) = log ( log ( 1/ n x )) + 2/ log ( n x ) , is positive and increasing from some time t 0 t_0 t 0 from Assumption 4.
Theorem F.2 (Theorem 6.2 from Cai et al. [2 ] , simplified). Under Assumption 4 the parameters w ( t ) \mathbf{w}^{(t)} w ( t ) converge in direction to a KKT point of
min ∥ w ∥ 2 , ∀ i , y i f H ( w , x i ) ≥ 1 \min\|\mathbf{w}\|^2,\qquad\forall i,\quad y_i f_H(\mathbf{w},\mathbf{x}_i) \ge1 min ∥ w ∥ 2 , ∀ i , y i f H ( w , x i ) ≥ 1 Where f H f_H f H represents the top-degree homogenization of the margin (depending on x \mathbf{x} x ).
Lemma F.3 (Hoeffding’s Lemma). Let X X X be a RV over the reals, with a ≤ X ≤ b a \le X \le b a ≤ X ≤ b almost surely. Then
∀ λ ∈ R \forall\lambda\in\mathbb{R} ∀ λ ∈ R E exp ( λ ( X − E X ) ) ≤ exp ( λ 2 ( b − a ) 2 / 8 ) \mathbb{E}\exp\left(\lambda(X-\mathbb{E}X)\right) \le\exp\left(\lambda^2(b-a)^2/8\right) E exp ( λ ( X − E X ) ) ≤ exp ( λ 2 ( b − a ) 2 /8 )
Lemma F.4. Let ( e t ) t = 1 ∞ (e_t)_{t=1}^{\infty} ( e t ) t = 1 ∞ , ( β t ) t = 1 ∞ (\beta_t)_{t=1}^{\infty} ( β t ) t = 1 ∞ , ( u t ) t = 1 ∞ (u_t)_{t=1}^{\infty} ( u t ) t = 1 ∞ , s.t.
e t + 1 = ( 1 − β t ) e t + u t e_{t+1}=(1-\beta_t)e_t+u_t e t + 1 = ( 1 − β t ) e t + u t And 0 < β t < 1 0 < \beta_t < 1 0 < β t < 1 , ∑ t = 1 ∞ β t = ∞ \sum_{t=1}^{\infty}\beta_t=\infty ∑ t = 1 ∞ β t = ∞ , ∑ t = 1 ∞ ∣ u t ∣ < ∞ \sum_{t=1}^{\infty}|u_t|<\infty ∑ t = 1 ∞ ∣ u t ∣ < ∞ . Then e t → 0 e_t \to0 e t → 0 .
Proof. Iterating the recursion gives
e t = e 1 ∏ k = 1 t − 1 ( 1 − β k ) + ∑ j = 1 t − 1 u j ∏ k = j + 1 t − 1 ( 1 − β k ) , e_t=e_1\prod_{k=1}^{t-1}(1-\beta_k)+\sum_{j=1}^{t-1}u_j\prod_{k=j+1}^{t-1}(1-\beta_k), e t = e 1 k = 1 ∏ t − 1 ( 1 − β k ) + j = 1 ∑ t − 1 u j k = j + 1 ∏ t − 1 ( 1 − β k ) ,
where an empty product is understood to be equal to 1.
First,
0 ≤ ∏ k = 1 t − 1 ( 1 − β k ) ≤ ∏ k = 1 t − 1 e − β k = exp ( − ∑ k = 1 t − 1 β k ) → 0 , 0\le\prod_{k=1}^{t-1}(1-\beta_k)\le\prod_{k=1}^{t-1}e^{-\beta_k}=\exp\left(-\sum_{k=1}^{t-1}\beta_k\right)\to0, 0 ≤ k = 1 ∏ t − 1 ( 1 − β k ) ≤ k = 1 ∏ t − 1 e − β k = exp ( − k = 1 ∑ t − 1 β k ) → 0 ,
since ∑ k = 1 ∞ β k = ∞ \sum_{k=1}^{\infty}\beta_k=\infty ∑ k = 1 ∞ β k = ∞ . Hence
e 1 ∏ k = 1 t − 1 ( 1 − β k ) → 0. e_1\prod_{k=1}^{t-1}(1-\beta_k)\to0. e 1 k = 1 ∏ t − 1 ( 1 − β k ) → 0.
It remains to treat the second term. Let ε > 0 \varepsilon>0 ε > 0 . Since ∑ j = 1 ∞ ∣ u j ∣ < ∞ \sum_{j=1}^{\infty}|u_j|<\infty ∑ j = 1 ∞ ∣ u j ∣ < ∞ , there exists N N N such that
∑ j = N + 1 ∞ ∣ u j ∣ < ε . \sum_{j=N+1}^{\infty}|u_j|<\varepsilon. j = N + 1 ∑ ∞ ∣ u j ∣ < ε .
Now split
∑ j = 1 t − 1 u j ∏ k = j + 1 t − 1 ( 1 − β k ) = ∑ j = 1 N u j ∏ k = j + 1 t − 1 ( 1 − β k ) + ∑ j = N + 1 t − 1 u j ∏ k = j + 1 t − 1 ( 1 − β k ) . \sum_{j=1}^{t-1}u_j\prod_{k=j+1}^{t-1}(1-\beta_k)
=
\sum_{j=1}^{N}u_j\prod_{k=j+1}^{t-1}(1-\beta_k)
+
\sum_{j=N+1}^{t-1}u_j\prod_{k=j+1}^{t-1}(1-\beta_k). j = 1 ∑ t − 1 u j k = j + 1 ∏ t − 1 ( 1 − β k ) = j = 1 ∑ N u j k = j + 1 ∏ t − 1 ( 1 − β k ) + j = N + 1 ∑ t − 1 u j k = j + 1 ∏ t − 1 ( 1 − β k ) .
For each fixed j j j , we have
∏ k = j + 1 t − 1 ( 1 − β k ) ≤ exp ( − ∑ k = j + 1 t − 1 β k ) → 0 , \prod_{k=j+1}^{t-1}(1-\beta_k)\le\exp\left(-\sum_{k=j+1}^{t-1}\beta_k\right)\to0, k = j + 1 ∏ t − 1 ( 1 − β k ) ≤ exp − k = j + 1 ∑ t − 1 β k → 0 ,
so the finite sum
∑ j = 1 N u j ∏ k = j + 1 t − 1 ( 1 − β k ) → 0. \sum_{j=1}^{N}u_j\prod_{k=j+1}^{t-1}(1-\beta_k)\to0. j = 1 ∑ N u j k = j + 1 ∏ t − 1 ( 1 − β k ) → 0.
For the tail, since 0 < 1 − β k < 1 0<1-\beta_k<1 0 < 1 − β k < 1 ,
∣ ∑ j = N + 1 t − 1 u j ∏ k = j + 1 t − 1 ( 1 − β k ) ∣ ≤ ∑ j = N + 1 t − 1 ∣ u j ∣ ≤ ∑ j = N + 1 ∞ ∣ u j ∣ < ε . \left|\sum_{j=N+1}^{t-1}u_j\prod_{k=j+1}^{t-1}(1-\beta_k)\right|
\le\sum_{j=N+1}^{t-1}|u_j|
\le\sum_{j=N+1}^{\infty}|u_j|<\varepsilon. j = N + 1 ∑ t − 1 u j k = j + 1 ∏ t − 1 ( 1 − β k ) ≤ j = N + 1 ∑ t − 1 ∣ u j ∣ ≤ j = N + 1 ∑ ∞ ∣ u j ∣ < ε .
Thus the second term tends to 0. Combining this with the convergence of the first term, we conclude that
Additional Experiments and Experimental Details
For all experiments we use Pytorch (Paszke et al. (2019)). We fix 20 seeds (for i ∈ [ 4 ] i \in[4] i ∈ [ 4 ] , j ∈ [ 5 ] j \in[5] j ∈ [ 5 ] , S e e d s = { i ⋅ 10 j − 1 } \mathrm{Seeds} = \{i \cdot10^{j-1}\} Seeds = { i ⋅ 1 0 j − 1 } ), filter experiments that did not reach interpolation or those with loss higher than 10 − 3 10^{-3} 1 0 − 3 , and then plot averages of 5 seeds from the remaining experiments. Each training run is 30000 epochs long. The training is run on RTX2080 GPUs, one for each experiment. Each run used an instance 8 cpus, and 12GB memory. We note that our experiments use relatively small data due to hardware memory and time constraints. We use normalized gradient descent to accelerate convergence of the dynamics, since with standard gradient descent convergence requires many more epochs, and it does not affect the implicit bias.
Effective degree. To estimate the effective degree of each example, we track min ( ∣ m i ( w ( t ) ) m i h ( w ( t ) ) ∣ , ∣ m i h ( w ( t ) ) m i ( w ( t ) ) ∣ , ∥ ∇ m i h ( w ( t ) ) ∥ ∥ ∇ m i ( w ( t ) ) ∥ , ∥ ∇ m i ( w ( t ) ) ∥ ∥ ∇ m i h ( w ( t ) ) ∥ ) \min\left(\left|\frac{m_i(w^{(t)})}{m_i^h(w^{(t)})}\right|,\left|\frac{m_i^h(w^{(t)})}{m_i(w^{(t)})}\right|,\frac{\lVert\nabla m_i^h(w^{(t)})\rVert}{\lVert\nabla m_i(w^{(t)})\rVert},\frac{\lVert\nabla m_i(w^{(t)})\rVert}{\lVert\nabla m_i^h(w^{(t)})\rVert}\right) min ( m i h ( w ( t ) ) m i ( w ( t ) ) , m i ( w ( t ) ) m i h ( w ( t ) ) , ∥ ∇ m i ( w ( t ) )∥ ∥ ∇ m i h ( w ( t ) )∥ , ∥ ∇ m i h ( w ( t ) )∥ ∥ ∇ m i ( w ( t ) )∥ ) for each example x i x_i x i and 1 ≤ h ≤ deg ( P i ) 1 \le h \le\deg(\mathcal{P}_i) 1 ≤ h ≤ deg ( P i ) , and choose the h h h that maximizes this quantity. Asymptotically, we expect only the effective degree to not vanish, since it goes to a constant C C C . Otherwise, at least one of the ratios should go to zero.
Linear Autoregressive
The data { x i , y i } i = 1 10 \{x_i,y_i\}_{i=1}^{10} { x i , y i } i = 1 10 is random gaussian with x i ∈ R 100 x_i \in\mathbb{R}^{100} x i ∈ R 100 (10× overparameterization), where the labels are set by fixing a random vector of parameters w w w , each coordinate sampled i.i.d. from a gaussian of scale 10 − 19 10^{-19} 1 0 − 19 , applying it to every example t i t_i t i times, and taking the result’s sign. We use a learning rate of 3 ⋅ 10 − 3 3 \cdot10^{-3} 3 ⋅ 1 0 − 3 . We assign t i t_i t i uniformly random from a set of indices T ⊂ N T \subset\mathbb{N} T ⊂ N . We run experiments with the T = { 1 , 2 , 5 } T = \{1, 2, 5\} T = { 1 , 2 , 5 } , T = { 1 , 3 , 5 } T = \{1, 3, 5\} T = { 1 , 3 , 5 } , T = { 1 , 4 } T = \{1, 4\} T = { 1 , 4 } , T = { 2 , 5 } T = \{2, 5\} T = { 2 , 5 } . Each of which we run with 20 different seeds, summing up to 80 experiments.
T = 2 , 5 T = 2, 5 T = 2 , 5
We note that the effective degree often does not equal the algebraic degree. This is significant, since our effective-degree estimator is biased toward higher degrees.
Figure 5. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. Figure corresponding to T = 2 , 5 T = 2, 5 T = 2 , 5 .
Figure 6. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. Figure corresponding to T = 2 , 5 T = 2, 5 T = 2 , 5 .
Figure 7. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. Figure corresponding to T = 2 , 5 T = 2, 5 T = 2 , 5 .
Figure 8. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. Figure corresponding to T = 1 , 4 T = 1, 4 T = 1 , 4 .
Figure 9. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. Figure corresponding to T = 1 , 4 T = 1, 4 T = 1 , 4 .
Figure 10. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. Figure corresponding to T = 1 , 4 T = 1, 4 T = 1 , 4 .
T = 1 , 3 , 5 T = 1, 3, 5 T = 1 , 3 , 5
Figure 11. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \left|m_i^h/m_i\right| m i h / m i . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. Figure corresponding to T = 1 , 3 , 5 T = 1, 3, 5 T = 1 , 3 , 5 .
Figure 12. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. Figure corresponding to T = 1 , 3 , 5 T = 1, 3, 5 T = 1 , 3 , 5 .
Figure 13. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. Figure corresponding to T = 1 , 3 , 5 T = 1, 3, 5 T = 1 , 3 , 5 .
Figure 14. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. Figure corresponding to T = 1 , 2 , 5 T = 1, 2, 5 T = 1 , 2 , 5 .
Figure 15. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. Figure corresponding to T = 1 , 2 , 5 T = 1, 2, 5 T = 1 , 2 , 5 .
Figure 16. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. Figure corresponding to T = 1 , 2 , 5 T = 1, 2, 5 T = 1 , 2 , 5 .
SSM
The data { X i , y i } i = 1 10 \{X_i, y_i\}_{i=1}^{10} { X i , y i } i = 1 10 is random gaussian with X i ∈ R 4 × l i X_i \in\mathbb{R}^{4 \times l_i} X i ∈ R 4 × l i , where the labels are set by fixing a random projection vector p ∈ R 4 p \in\mathbb{R}^4 p ∈ R 4 (which is equal to the “student’s” P \mathcal{P} P and stays fixed during training), random parameters A ∈ R 100 × 100 A \in\mathbb{R}^{100 \times100} A ∈ R 100 × 100 , B ∈ R 100 × 4 B \in\mathbb{R}^{100 \times4} B ∈ R 100 × 4 , C ∈ R 100 × 4 C \in\mathbb{R}^{100 \times4} C ∈ R 100 × 4 , each coordinate sampled i.i.d. from a gaussian of scale 10 − 19 10^{-19} 1 0 − 19 , applying it to every example t i t_i t i times, and taking the result’s sign. We assign l i , t i l_i, t_i l i , t i uniformly random from a set of indices T ⊂ N × N T \subset\mathbb{N} \times\mathbb{N} T ⊂ N × N . We run experiments with the sets T = { ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) } T = \{(3, 1), (3, 2), (5, 2)\} T = {( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 )} , T = { ( 1 , 1 ) ; ( 2 , 2 ) ; ( 3 , 3 ) } T = \{(1, 1); (2, 2); (3, 3)\} T = {( 1 , 1 ) ; ( 2 , 2 ) ; ( 3 , 3 )} , T = { ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) } T = \{(3, 1), (2, 2), (1, 3)\} T = {( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 )} . Each of which we run with 20 different seeds, summing up to 80 experiments.
l , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) l,T = (3,1), (3,2), (5,2) l , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 )
Figure 17. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \left|m_i^h/m_i\right| m i h / m i . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. The figure is for l , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) l,T = (3,1), (3,2), (5,2) l , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) .
Figure 18. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. The figure is for ℓ , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) \ell, T = (3, 1), (3, 2), (5, 2) ℓ , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) .
Figure 19. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. The figure is for ℓ , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) \ell, T = (3, 1), (3, 2), (5, 2) ℓ , T = ( 3 , 1 ) , ( 3 , 2 ) , ( 5 , 2 ) .
Figure 20. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. The figure is for l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) l,T=(1,1),(2,2),(3,3) l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) .
Figure 21. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. The figure is for l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) l,T=(1,1),(2,2),(3,3) l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) .
Figure 22. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. The figure is for l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) l,T=(1,1),(2,2),(3,3) l , T = ( 1 , 1 ) , ( 2 , 2 ) , ( 3 , 3 ) .
Figure 23. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. The figure is for l , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) l,T = (3,1), (2,2), (1,3) l , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) .
Figure 24. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. The figure is for ℓ , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) \ell,T=(3,1),(2,2),(1,3) ℓ , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) .
Figure 25. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. The figure is for ℓ , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) \ell,T=(3,1),(2,2),(1,3) ℓ , T = ( 3 , 1 ) , ( 2 , 2 ) , ( 1 , 3 ) .
MNIST SSM
We use data from MNIST [21 ] . The data { X i , y i } i = 1 10 \{X_i,y_i\}_{i=1}^{10} { X i , y i } i = 1 10 divided to be half of images of the number 8 (and positive label), and half of the number 1 (and negative label). X i ∈ R 28 × ℓ i X_i\in\mathbb{R}^{28\times\ell_i} X i ∈ R 28 × ℓ i , ℓ i \ell_i ℓ i is chosen uniformly at random from { 14 , 21 , 28 } \{14,21,28\} { 14 , 21 , 28 } , and we truncate the rows up to that point (data is first transposed so dimensions would fit). That is, we refer to the image’s rows as a sequence, each row is a token of dimension 28, and some examples dont use all the rows, in order for the result to have diverse effective degrees. Random parameters A ∈ R 64 × 64 A\in\mathbb{R}^{64\times64} A ∈ R 64 × 64 , B ∈ R 64 × 28 B\in\mathbb{R}^{64\times28} B ∈ R 64 × 28 , C ∈ R 64 × 28 C\in\mathbb{R}^{64\times28} C ∈ R 64 × 28 , p ∈ R 28 p\in\mathbb{R}^{28} p ∈ R 28 , each coordinate sampled i.i.d. from a gaussian of scale 10 − 19 10^{-19} 1 0 − 19 . We run the experiment with 20 different seeds with learning rate 10 − 3 10^{-3} 1 0 − 3 .
Figure 26. Left: Ratio between the effective-degree homogenization of the margin and the margin, ∣ m i h / m i ∣ \lvert m_i^h/m_i\rvert ∣ m i h / m i ∣ . Middle: The corresponding ratio for the gradients, ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ \lVert\nabla m_i^h\rVert/\lVert\nabla m_i\rVert ∥ ∇ m i h ∥ / ∥ ∇ m i ∥ . Right: The ratio of the two former ratios. Each row corresponds to a different seed and shows 10 random training examples, with h h h the effective degree of each. Y-axes are log-scale. The figure is for SSM trained on MNIST with l = { 14 , 21 , 28 } l = \{14,21,28\} l = { 14 , 21 , 28 } .
Figure 27. Left: The loss. Right: Cosine similarity to the final predictor direction. Y-axes are log-scale. The figure is for SSM trained on MNIST with l = { 14 , 21 , 28 } l = \{14,21,28\} l = { 14 , 21 , 28 } .
Figure 28. Left: Minimal normalized, homogenized margin per effective degree. Right: All points with their effective degrees and final normalized homogenized margins. Each row corresponds to a different seed. The figure is for SSM trained on MNIST with l = { 14 , 21 , 28 } l = \{14, 21, 28\} l = { 14 , 21 , 28 } .