Introduction
Recurrent neural networks (RNNs) are extraordinarily successful models of the brain [5 , 9 , 10 , 14 ] , and the past two decades of theory and experiment have produced a working understanding of what they learn and how. Their loss landscapes are generally more complex than those of feedforward networks, in part because recurrence can produce exploding and vanishing gradients [16 , 17 ] ; their solution space is generally highly degenerate [8 ] ; and they can be biased towards learning low-rank solutions [15 ] . But a major feature of RNNs that remains poorly understood is the significance of intrinsic noise.
Noise is omnipresent in biological neural networks: spike trains and firing rate traces throughout the nervous system involve difficult-to-predict fluctuations [3 ] , which may be a manifestation of chaotic behavior [1 , 2 ] or low-level stochasticity associated with chemical reactions. Although from one perspective noise is a nuisance, from another it provides pressure for networks to find more robust solutions. Noise is routinely used in RNN training [12 , 13 ] , in part to improve generalization, and there is evidence that noise can drive networks into flatter loss landscape minima [11 ] .
From a representation-learning perspective, it may be helpful to view noise as having two potential consequences. First, noise can provide geometric constraints on solutions: weights must be oriented in certain directions (i.e., the less noisy directions) and not others, and their overall scales must be chosen to minimize noise. Second, the goal of minimizing noise may compete with that of achieving good task performance; if there is too much noise, one expects that the system may be driven to a trivial solution at which task performance is poor, but there is little or no noise.
Although RNN learning dynamics are difficult to study, an interesting toy model of them has emerged in recent work: linear RNNs, especially linear RNNs trained to perform leaky integration of a white noise stimulus [7 , 4 ] . As [4 ] show, when network parameters are initially small, learning has two phases: an alignment phase, in which parameter movement concentrates along a specific (initialization-dependent) direction of state space, followed by a dynamics phase, in which an eigenvalue ‘escapes’ from the bulk in order to solve the task.
In this work, we consider an extremely similar theoretical setup, whose primary difference is the presence of (generally anisotropic) intrinsic noise. We show that their theory must be substantially modified even if there is only a small amount of noise, since noise affects both the alignment and dynamics phases, and even whether a nontrivial solution exists at all. Throughout, we focus on characterizing learning in the rich regime, as opposed to the lazy regime, in part because the rich regime is thought to be more representative of learning in performant RNNs.
Our main contributions are as follows.
We develop an analytically tractable theory of representation learning in linear RNNs with intrinsic noise. We show that noise breaks degeneracies of the original objective, which yields preferred state space directions and read-in/readout scales.
We characterize how intrinsic noise modifies rich-regime learning dynamics. During the alignment phase, anisotropic noise changes the growth rates of different state-space modes, which biases learning toward less noisy directions. Noise also breaks the symmetry between the read-in and readout weights, and in the absence of regularization, drives the input weight scale to infinity and the output weight scale to zero.
We show that sufficiently strong noise can produce phase transitions beyond which only a trivial solution is learned. This is true both when all parameters are trainable and when some parameters are frozen.
Mathematical formulation
Figure 1. Theoretical setting. (a) We study a noisy linear RNN which receives a scalar white noise signal x ( t ) x(t) x ( t ) as input and produces an output y ( t ) y(t) y ( t ) . The dynamics of individual r i r_i r i are noisy (bottom). (b) On average, the network filters its input. (c) Output variance is input-independent and eventually saturates (assuming the network is stable). (d) We assume the network is trained to optimize an objective with three terms: a filter-matching term that incentivizes matching its average behavior to that of a target filter f ∗ ( t ) f_*(t) f ∗ ( t ) , a noise reduction term, and a regularization term.
Our setup largely mirrors that of [4 ] . Throughout this paper, we study a linear RNN with N ≥ 1 N \geq 1 N ≥ 1 neurons whose instantaneous state is characterized by a vector r : = ( r 1 , . . . , r N ) T ∈ R N \bm{r} := (r_1, ..., r_N)^T \in \mathbb{R}^N r := ( r 1 , ... , r N ) T ∈ R N . We assume dynamics
r ˙ = A r + x ( t ) u + G η ( t ) = − γ r + W r + x ( t ) u + G η ( t ) \begin{equation}
\dot{\bm{r}} = \bm{A} \bm{r} + x(t) \bm{u} + \bm{G} \ \bm{\eta}(t) = - \gamma \bm{r} + \bm{W} \bm{r} + x(t) \bm{u} + \bm{G} \ \bm{\eta}(t) \tag*{(1)}
\end{equation} r ˙ = A r + x ( t ) u + G η ( t ) = − γ r + W r + x ( t ) u + G η ( t ) (1)
where x ( t ) x(t) x ( t ) is a time-dependent input signal, u ∈ R N \bm{u} \in \mathbb{R}^N u ∈ R N are the read-in weights, γ > 0 \gamma > 0 γ > 0 is the coefficient of the leak term, and W ∈ R N × N \bm{W} \in \mathbb{R}^{N \times N} W ∈ R N × N are the recurrent weights (Figure 1 a). To ease notation, we sometimes work with the combination A : = − γ I + W \bm{A} := - \gamma \bm{I} + \bm{W} A := − γ I + W . The remaining term instantiates intrinsic noise: G ∈ R N × N \bm{G} \in \mathbb{R}^{N \times N} G ∈ R N × N controls its amplitude, and η ( t ) ∈ R N \bm{\eta}(t) \in \mathbb{R}^N η ( t ) ∈ R N is a vector of Gaussian white noise processes. We are most interested in understanding how the structure of G \bm{G} G shapes learning.
Because Eq. (1) is linear in r \bm{r} r and involves only state-independent additive noise, we can immediately write down the distribution of r ( t ) \bm{r}(t) r ( t ) conditional on any initial condition, although we will assume r ( 0 ) = 0 \bm{r}(0) = \bm{0} r ( 0 ) = 0 for simplicity. We find that r ( t ) \bm{r}(t) r ( t ) is normally distributed with
p ( r , t ∣ r 0 = 0 , 0 ) = N ( ∫ 0 t e A t ′ u x ( t − t ′ ) d t ′ , ∫ 0 t e A t ′ S e A T t ′ d t ′ ) \begin{equation}
p(\bm{r} , t | \bm{r}_0 = \bm{0}, 0) = \mathcal{N}\left( \int_0^t e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' , \int_0^t e^{\bm{A} t'} \bm{S} e^{\bm{A}^T t'} \ dt' \right) \tag*{(2)}
\end{equation} p ( r , t ∣ r 0 = 0 , 0 ) = N ( ∫ 0 t e A t ′ u x ( t − t ′ ) d t ′ , ∫ 0 t e A t ′ S e A T t ′ d t ′ ) (2)
where S : = G G T \bm{S} := \bm{G} \bm{G}^T S := G G T is, up to a factor of 2 2 2 , the diffusion tensor associated with G \bm{G} G (which we take to be positive definite). On average, the network linearly filters its input (Figure 1 b). But because this integration is noisy, S \bm{S} S -dependent noise accumulates until (provided the eigenvalues of A \bm{A} A all have negative real parts) it saturates (Figure 1 c).
Objective function
Following [4 ] , assume a linear readout determined by readout weights v ∈ R N \bm{v} \in \mathbb{R}^N v ∈ R N :
y ( t ) : = v T r ( t ) . \begin{equation}
y(t) := \bm{v}^T \bm{r}(t) \ . \tag*{(3)}
\end{equation} y ( t ) := v T r ( t ) . (3)
This readout defines a filter f ( t ) : = e − γ t v T e W t u f(t) := e^{- \gamma t} \bm{v}^T e^{\bm{W} t} \bm{u} f ( t ) := e − γ t v T e W t u since, on average,
E r [ y ( t ) ] = ∫ 0 t v T e A t ′ u x ( t − t ′ ) d t ′ = ∫ 0 t f ( t ′ ) x ( t − t ′ ) d t ′ . \begin{equation}
\mathbb{E}_{\bm{r}}[ y(t) ] = \int_0^t \bm{v}^T e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' = \int_0^t f(t') \ x(t-t') \ dt' \ . \tag*{(4)}
\end{equation} E r [ y ( t )] = ∫ 0 t v T e A t ′ u x ( t − t ′ ) d t ′ = ∫ 0 t f ( t ′ ) x ( t − t ′ ) d t ′ . (4)
Assume we train some combination of u \bm{u} u , v \bm{v} v , and W \bm{W} W so that y ( t ) y(t) y ( t ) matches a target output
y ∗ ( t ) : = ∫ 0 t e − c ∗ t ′ x ( t − t ′ ) d t ′ = ∫ 0 t f ∗ ( t ′ ) x ( t − t ′ ) d t ′ \begin{equation}
y_*(t) := \int_0^t e^{- c_* t'} x(t - t') \ dt' = \int_0^t f_*(t') x(t - t') \ dt' \tag*{(5)}
\end{equation} y ∗ ( t ) := ∫ 0 t e − c ∗ t ′ x ( t − t ′ ) d t ′ = ∫ 0 t f ∗ ( t ′ ) x ( t − t ′ ) d t ′ (5)
which linearly filters its input x ( t ) x(t) x ( t ) using the target filter f ∗ ( t ) : = e − c ∗ t f_*(t) := e^{- c_* t} f ∗ ( t ) := e − c ∗ t . Here, c ∗ > 0 c_* > 0 c ∗ > 0 controls the time scale on which input is leaky-integrated. Assume we do this using the supervised objective
J : = lim T → ∞ 1 T ∫ 0 T 1 2 E x , r { [ y ( t ) − y ∗ ( t ) ] 2 } d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 , \begin{equation}
J := \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_{x, \bm{r}}\left\{ \ [ y(t) - y_*(t) ]^2 \ \right\} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ , \tag*{(6)}
\end{equation} J := T → ∞ lim T 1 ∫ 0 T 2 1 E x , r { [ y ( t ) − y ∗ ( t ) ] 2 } d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 , (6)
which demands that the network output y ( t ) y(t) y ( t ) agrees with y ∗ ( t ) y_*(t) y ∗ ( t ) on average. Note that we have also included a regularization term whose strength is determined by κ ≥ 0 \kappa \geq 0 κ ≥ 0 . This objective can be simplified if the input has white noise statistics [7 , 4 ] , i.e., if E x [ x ( t ) ] = 0 \mathbb{E}_x[ x(t) ] = 0 E x [ x ( t )] = 0 and E x [ x ( t ) x ( t − t ′ ) ] = δ ( t − t ′ ) \mathbb{E}_x[ x(t) x(t - t') ] = \delta(t - t') E x [ x ( t ) x ( t − t ′ )] = δ ( t − t ′ ) . We find (see Appendix § 9 )
J ( u , v , W ) = ∫ 0 ∞ 1 2 [ f ( t ) − f ∗ ( t ) ] 2 + e − 2 γ t 2 v T e W t S e W T t v d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 . \begin{equation}
\boxed{ \
J(\bm{u}, \bm{v}, \bm{W}) = \int_0^{\infty} \frac{1}{2} \left[ f(t) - f_*(t) \right]^2 + \frac{e^{- 2 \gamma t}}{2} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t} \bm{v} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ } \ . \tag*{(7)}
\end{equation} J ( u , v , W ) = ∫ 0 ∞ 2 1 [ f ( t ) − f ∗ ( t ) ] 2 + 2 e − 2 γ t v T e W t S e W T t v d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 . (7)
Note that J J J has three terms: (i) a filter-matching term J filter J_{\text{filter}} J filter , (ii) a noise term J noise J_{\text{noise}} J noise , and (iii) a regularization term J reg J_{\text{reg}} J reg (Figure 1 d). [4 ] only considered the filter-matching term; we will see that the other two terms strongly modulate learning dynamics.
Learning dynamics
RNNs are normally trained via backpropagation through time [18 ] , FORCE [19 ] , or real-time recurrent learning [20 , 6 ] . One virtue of linear RNNs, at least for theory, is that gradients of J J J with respect to u \bm{u} u , v \bm{v} v , and W \bm{W} W can be computed exactly, so we can ignore the extra structure associated with those learning rules. Assuming parameters are updated via gradient descent, in the gradient-flow limit in which the learning rate is taken infinitesimally small, we recover ordinary differential equations (ODEs) (see Appendix § 9 )
d d τ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W T t v d t − κ u d d τ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v d d τ W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W}^T t} \bm{v} \ dt - \kappa \bm{u} \\
\frac{d}{d\tau} \bm{v} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt - \kappa \bm{v} \\
\frac{d}{d\tau} \bm{W} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} \int_0^t e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ dt' dt\\
&- \int_0^{\infty} e^{-2\gamma t} \int_0^t \ e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} \ dt' dt
\end{split} \tag*{(8)}
\end{equation} d τ d u d τ d v d τ d W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W T t v d t − κ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t (8)
parameterized by a real-valued ‘learning time’ τ ∈ [ 0 , ∞ ) \tau \in [0, \infty) τ ∈ [ 0 , ∞ ) . These dynamics are generically highly nonlinear and high-dimensional, although we will see shortly that they simplify substantially if the initial weights are assumed to be small.
Noise produces representational pressure
Figure 2. Noise produces representational pressure. (a) Two representations with the same filter but different readout vectors differ in noise level if S \bm{S} S is anisotropic. Pictured is a schematic 2D example with a high-noise direction (horizontal) and a low-noise direction (vertical). One can reduce noise by rotating v \bm{v} v (white arrow). (b) The noise term J noise J_{\text{noise}} J noise in the schematic example as a function of v \bm{v} v orientation. J filter J_{\text{filter}} J filter is unaffected as long as the filter remains the same. (c) J noise J_{\text{noise}} J noise and J filter J_{\text{filter}} J filter as a function of α \alpha α (see main text); a zero readout scale is preferred given noise. (d) J filter J_{\text{filter}} J filter and J noise J_{\text{noise}} J noise vs c c c (see main text) generally have different optima, and their competition can change where the minimum of J J J is, or whether it has a finite minimum at all.
One elemental observation is that noise produces representational pressure: whereas the noiseless system has many representations compatible with good performance, the noisy system must minimize the corrupting influence of noise, which can greatly reduce the number of good solutions.
The deterministic solution is highly degenerate.
In order for f ( t ) = e − γ t v T e W t u f(t) = e^{- \gamma t} \bm{v}^T e^{\bm{W} t} \bm{u} f ( t ) = e − γ t v T e W t u to match f ∗ ( t ) = e − c ∗ t f_*(t) = e^{- c_* t} f ∗ ( t ) = e − c ∗ t , assuming W \bm{W} W is diagonalizable, it must be true that (i) γ − c ∗ \gamma - c_* γ − c ∗ is an eigenvalue of W \bm{W} W ; (ii) u \bm{u} u has a component along one of the corresponding right eigenvectors; (iii) v \bm{v} v has a component along one of the corresponding left eigenvectors; and (iv) u T v = 1 \bm{u}^T \bm{v} = 1 u T v = 1 . But this yields a solution space with huge parameter degeneracies, including invariance to arbitrary invertible linear transformations. In particular, if T ∈ R N × N \bm{T} \in \mathbb{R}^{N \times N} T ∈ R N × N is an invertible linear transformation, then f ( t ) f(t) f ( t ) remains invariant to the mapping with v → T T v \bm{v} \to \bm{T}^T \bm{v} v → T T v , u → T − 1 u \bm{u} \to \bm{T}^{-1} \bm{u} u → T − 1 u , and W → T − 1 W T \bm{W} \to \bm{T}^{-1} \bm{W} \bm{T} W → T − 1 W T , since
v T e W t u → T v T T T − 1 e W t T T − 1 u = v T e W t u . \begin{equation}
\bm{v}^T e^{\bm{W} t} \bm{u} \xrightarrow{\bm{T}} \bm{v}^T \bm{T} \bm{T}^{-1} e^{\bm{W} t} \bm{T} \bm{T}^{-1} \bm{u} = \bm{v}^T e^{\bm{W} t} \bm{u} \ . \tag*{(9)}
\end{equation} v T e W t u T v T T T − 1 e W t T T − 1 u = v T e W t u . (9)
Interesting special cases include when T \bm{T} T is a rotation or a change in read-in/readout scale. Meanwhile, the noise term in our objective does not generally respect those degeneracies:
∫ 0 ∞ e − 2 γ t v T e W t S e W T t v d t → T ∫ 0 ∞ e − 2 γ t v T e W t ( T S T T ) e W T t v d t . \begin{equation}
\int_0^{\infty} e^{- 2 \gamma t} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t} \bm{v} \ dt \xrightarrow{\bm{T}} \int_0^{\infty} e^{- 2 \gamma t} \bm{v}^T e^{\bm{W} t} (\bm{T} \bm{S} \bm{T}^{T}) e^{\bm{W}^T t} \bm{v} \ dt \ . \tag*{(10)}
\end{equation} ∫ 0 ∞ e − 2 γ t v T e W t S e W T t v d t T ∫ 0 ∞ e − 2 γ t v T e W t ( T S T T ) e W T t v d t . (10)
We discuss various interesting consequences of this simple observation below.
Noise picks out preferred state-space directions.
If multiple state-space directions could solve the task (i.e., if W \bm{W} W has multiple correct eigenvalues), the presence of noise causes less noisy directions to be preferred (Figure 2 a, b). For example, if W = ( γ − c ∗ ) I \bm{W} = (\gamma - c_*) \bm{I} W = ( γ − c ∗ ) I (i.e., all eigenvalues are correct) and v T u = 1 \bm{v}^T \bm{u} = 1 v T u = 1 (so that f ( t ) = f ∗ ( t ) f(t) = f_*(t) f ( t ) = f ∗ ( t ) ), then the noise term is minimized when
v ∗ = S − 1 u u T S − 1 u , \begin{equation}
\bm{v}_* = \frac{\bm{S}^{-1} \bm{u}}{\bm{u}^T \bm{S}^{-1} \bm{u}} \ , \tag*{(11)}
\end{equation} v ∗ = u T S − 1 u S − 1 u , (11)
i.e., when state-space directions are read out according to how noisy they are, with v ∗ \bm{v}_* v ∗ aligning more with less noisy directions. Essentially this solution is found when only the readout v \bm{v} v is trained, i.e., when a ‘reservoir computing’ approach [21 ] is used (see Appendix § 10 ).
Noise produces preferred read-in and readout scales.
A trivial way to reduce the noise term is to take the scale of the readout weights to zero. This is consistent with the above general observation; if T = α I \bm{T} = \alpha \bm{I} T = α I , the filter-matching term remains unchanged but the noise term picks up a factor of α 2 \alpha^2 α 2 , and the overall term is minimized when we take α → 0 \alpha \to 0 α → 0 (Figure 2 c). This implies, given no regularization (κ = 0 \kappa = 0 κ = 0 ), that it is optimal to take the readout scale to zero and the read-in scale to infinity.
Noise minimization can compete with filter-matching accuracy.
The filter-matching term J filter J_{\text{filter}} J filter and the noise term J noise J_{\text{noise}} J noise generally do not share optima, so their relative size can determine where the fixed points of learning dynamics are, or whether there is any finite fixed point at all. Figure 2 d depicts an example we will consider later in § 7 : the balance of the two terms determines whether the recurrent weight scale c c c converges to a finite value near c ∗ c_* c ∗ or diverges to infinity.
Noise determines weight alignment in rich regime
Figure 3. Noise determines alignment. (a) The dominant contribution to u i ( τ ) u_i(\tau) u i ( τ ) and v i ( τ ) v_i(\tau) v i ( τ ) grows exponentially quickly at a rate λ i + \lambda^+_i λ i + that decreases as the corresponding S \bm{S} S eigenvalue (i.e., noise level) s i s_i s i increases, so higher-noise modes grow more slowly. (b) Although dynamics involves both growing and decaying contributions, after enough learning time has elapsed, mode amplitudes (solid lines) are well-approximated by only their growing contribution (dashed lines). (c) The dot product between u \bm{u} u and v \bm{v} v , and the least noisy eigenvector of S \bm{S} S , increases during this phase. W \bm{W} W aligns in the same direction and becomes approximately rank-one. (d) The discrepancy between the scales of u \bm{u} u and v \bm{v} v grows with learning, and is higher in directions with larger noise levels.
In this section, we begin our analysis of this model’s learning dynamics (Eq. (8) ). We focus on the so-called rich regime, in which weights are initialized to small values and low-rank structure emerges. Our analysis here follows the spirit of [4 ] but differs in its details.
Dynamics given small initial weights
Assume u \bm{u} u , v \bm{v} v , and W \bm{W} W are initialized with small values (say, of order ϵ > 0 \epsilon > 0 ϵ > 0 ). To leading order in ϵ \epsilon ϵ , gradient descent learning dynamics become (see Appendix § 9 )
d d τ u ≈ v c ∗ + γ − κ u d d τ v ≈ u c ∗ + γ − S v 2 γ − κ v d d τ W ≈ v u T ( c ∗ + γ ) 2 − v v T S 4 γ 2 . \begin{equation}
\frac{d}{d\tau} \bm{u} \approx \frac{\bm{v} }{c_* + \gamma} - \kappa \bm{u} \hspace{0.28in} \frac{d}{d\tau} \bm{v} \approx \frac{\bm{u}}{c_* + \gamma} - \frac{\bm{S} \bm{v}}{2 \gamma} - \kappa \bm{v} \hspace{0.28in} \frac{d}{d\tau} \bm{W} \approx \frac{\bm{v} \bm{u}^T }{(c_* + \gamma)^2} - \frac{ \bm{v} \bm{v}^T \bm{S} }{4 \gamma^2} \ . \tag*{(12)}
\end{equation} d τ d u ≈ c ∗ + γ v − κ u d τ d v ≈ c ∗ + γ u − 2 γ S v − κ v d τ d W ≈ ( c ∗ + γ ) 2 v u T − 4 γ 2 v v T S . (12)
In other words, u \bm{u} u and v \bm{v} v tend to evolve according to a certain linear system, while W \bm{W} W depends only on input from u \bm{u} u and v \bm{v} v . This is a major improvement, since Eq. (8) is in general a highly nonlinear system of ODEs. Note that the presence of S \bm{S} S yields dynamics different from those studied by [4 ] ; S \bm{S} S breaks the symmetry between u \bm{u} u and v \bm{v} v dynamics, since components of v \bm{v} v along high-noise eigenvectors of S \bm{S} S are penalized while the analogous components of u \bm{u} u are not.
Weights rotate away from highly noisy state space directions
The alignment-phase ODEs (Eq. (12) ) simplify if we use the eigenvectors of S \bm{S} S as a basis. Letting u i u_i u i and v i v_i v i denote the components of u \bm{u} u and v \bm{v} v along an eigenvector whose eigenvalue is s i s_i s i , we have
d d τ u i = − κ u i + v i c ∗ + γ d d τ v i = u i c ∗ + γ − ( κ + s i 2 γ ) v i , \begin{equation}
\frac{d}{d\tau} u_i = - \kappa u_i + \frac{v_i}{c_* + \gamma} \hspace{1in} \frac{d}{d\tau} v_i = \frac{u_i}{c_* + \gamma} - \left( \kappa + \frac{s_i}{2 \gamma} \right) v_i \ , \tag*{(13)}
\end{equation} d τ d u i = − κ u i + c ∗ + γ v i d τ d v i = c ∗ + γ u i − ( κ + 2 γ s i ) v i , (13)
i.e., components along different eigenvectors evolve independently. The 2 × 2 2 \times 2 2 × 2 matrix that governs these dynamics has two eigenvalues (see Appendix § 11 ):
λ i ± : = − ( κ + s i 4 γ ) ± s i 2 16 γ 2 + 1 ( c ∗ + γ ) 2 . \begin{equation}
\lambda^{\pm}_i := - \left( \kappa + \frac{s_i}{4 \gamma} \right) \pm \sqrt{ \frac{s_i^2}{16 \gamma^2} + \frac{1}{(c_* + \gamma)^2} } \ . \tag*{(14)}
\end{equation} λ i ± := − ( κ + 4 γ s i ) ± 16 γ 2 s i 2 + ( c ∗ + γ ) 2 1 . (14)
If the regularization strength κ \kappa κ is too large, both eigenvalues are negative, and u i u_i u i and v i v_i v i collapse to a trivial zero solution. Otherwise, λ i + \lambda^+_i λ i + is positive and λ i − \lambda_i^- λ i − is negative, which means that the λ i + \lambda^+_i λ i + component of the solution tends to dominate after an initial transient. More quantitatively,
u i ( τ ) ≈ 1 λ i + − λ i − [ v i ( 0 ) c ∗ + γ − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ v i ( τ ) ≈ ( c ∗ + γ ) ( κ + λ + ) λ i + − λ i − [ v i ( 0 ) c ∗ + γ − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ . \begin{equation}
\begin{split}
u_i(\tau) &\approx \frac{1}{\lambda^+_i - \lambda^-_i} \left[ \frac{v_i(0)}{c_* + \gamma} - u_i(0) (\kappa + \lambda^-_i) \right] e^{\lambda^+_i \tau} \\
v_i(\tau) &\approx \frac{(c_* + \gamma) (\kappa + \lambda^+) }{\lambda^+_i - \lambda^-_i} \left[ \frac{v_i(0)}{c_* + \gamma} - u_i(0) (\kappa + \lambda^-_i) \right] e^{\lambda^+_i \tau} \ .
\end{split} \tag*{(15)}
\end{equation} u i ( τ ) v i ( τ ) ≈ λ i + − λ i − 1 [ c ∗ + γ v i ( 0 ) − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ ≈ λ i + − λ i − ( c ∗ + γ ) ( κ + λ + ) [ c ∗ + γ v i ( 0 ) − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ . (15)
One interesting feature of this result is that the growth rate λ i + \lambda^+_i λ i + is a decreasing function of s i s_i s i (Figure 3 a), which implies that components of u \bm{u} u and v \bm{v} v in noisier directions grow more slowly (Figure 3 b). Because small differences are exponentially amplified, to good approximation u \bm{u} u and v \bm{v} v align along the least noisy eigenvectors of S \bm{S} S (Figure 3 c). In particular,
u ( τ ) ≈ u ( τ ) P [ [ u ( 0 ) + v ( 0 ) ] 2 + ( c ∗ + γ ) 2 ( s min 4 γ + Δ ) u ( 0 ) ] v ( τ ) ≈ v ( τ ) P [ [ u ( 0 ) + v ( 0 ) ] 2 + ( c ∗ + γ ) 2 ( s min 4 γ + Δ ) u ( 0 ) ] \begin{equation}
\begin{split}
\bm{u}(\tau) &\approx u(\tau) \bm{P} \left[ \frac{[ \bm{u}(0) + \bm{v}(0) ]}{2} + \frac{(c_* + \gamma)}{2 } \left(\frac{s_{\text{min}}}{4 \gamma} + \Delta \right) \bm{u}(0) \right] \\
\bm{v}(\tau) &\approx v(\tau) \bm{P} \left[ \frac{[ \bm{u}(0) + \bm{v}(0) ]}{2} + \frac{(c_* + \gamma)}{2 } \left(\frac{s_{\text{min}}}{4 \gamma} + \Delta \right) \bm{u}(0) \right]
\end{split} \tag*{(16)}
\end{equation} u ( τ ) v ( τ ) ≈ u ( τ ) P [ 2 [ u ( 0 ) + v ( 0 )] + 2 ( c ∗ + γ ) ( 4 γ s min + Δ ) u ( 0 ) ] ≈ v ( τ ) P [ 2 [ u ( 0 ) + v ( 0 )] + 2 ( c ∗ + γ ) ( 4 γ s min + Δ ) u ( 0 ) ] (16)
where P \bm{P} P is a projection matrix that projects onto the subspace with the smallest eigenvalue s min s_{\text{min}} s min (which may be one-dimensional), and where Δ : = s min 2 16 γ 2 + 1 ( c ∗ + γ ) 2 − 1 c ∗ + γ \Delta := \sqrt{ \frac{s_{\text{min}}^2}{16 \gamma^2} + \frac{1}{(c_* + \gamma)^2} } - \frac{1}{c_* + \gamma} Δ := 16 γ 2 s min 2 + ( c ∗ + γ ) 2 1 − c ∗ + γ 1 . Note that Eq. (12) implies that W \bm{W} W aligns in the same direction as u \bm{u} u and v \bm{v} v (Figure 3 c). Another interesting feature of this result is that u i ( τ ) u_i(\tau) u i ( τ ) generally does not equal v i ( τ ) v_i(\tau) v i ( τ ) even if they are initialized to the same value—noise breaks the ‘balancing’ condition that can be assumed in the noise-free case [22 , 4 ] . For large s i s_i s i , this discrepancy in scale can be quite large (Figure 3 d).
The outcome of alignment
To very good approximation, we can assume u \bm{u} u and v \bm{v} v align and that W \bm{W} W is a rank-one matrix supported on the same vector. Letting u 0 \bm{u}_0 u 0 denote that (normalized) vector,
u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 W ( τ ) ≈ [ γ − c ( τ ) ] u 0 u 0 T \begin{equation}
\boxed{ \ \bm{u}(\tau) \approx u(\tau) \bm{u}_0 \hspace{0.5in} \bm{v}(\tau) \approx v(\tau) \bm{u}_0 \hspace{0.5in} \bm{W}(\tau) \approx [ \gamma - c(\tau) ] \bm{u}_0 \bm{u}_0^T \ } \tag*{(17)}
\end{equation} u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 W ( τ ) ≈ [ γ − c ( τ )] u 0 u 0 T (17)
for some training-time-dependent coefficients u ( τ ) u(\tau) u ( τ ) , v ( τ ) v(\tau) v ( τ ) , and c ( τ ) c(\tau) c ( τ ) . Because gradient-flow dynamics does not change parameter directions under this ansatz, to characterize the next phase it suffices to analyze the dynamics of the scalars u ( τ ) u(\tau) u ( τ ) , v ( τ ) v(\tau) v ( τ ) , and c ( τ ) c(\tau) c ( τ ) . In the next few sections, we will use σ 2 \sigma^2 σ 2 to denote the noise level associated with the alignment direction (i.e., to denote the smallest eigenvalue of S \bm{S} S ).
Rich regime learning dynamics
Figure 4. Rich regime learning dynamics. For plotting, we take c ∗ = 1 c_* = 1 c ∗ = 1 , σ 2 = 1 \sigma^2 = 1 σ 2 = 1 , and κ = 0 \kappa = 0 κ = 0 . (a) In the absence of regularization, learning dynamics drive c c c toward its target, c ∗ c_* c ∗ , and the amplitude N : = u v N := u v N := uv toward 1 1 1 . (b) Unlike the noise-free (σ 2 = 0 \sigma^2 = 0 σ 2 = 0 ) dynamics, noisy dynamics produce asymmetric trajectories for u u u and v v v , even if they are initialized to the same value. Asymptotically, u v ≈ 1 u v \approx 1 uv ≈ 1 , u u u continues to increase, and v v v continues to decrease. (c) The filter-matching term and noise term are both driven to zero, although in this case the latter occurs on a slower time scale.
Alignment reduces an N ( N + 2 ) N (N + 2) N ( N + 2 ) -dimensional problem to a three-dimensional one. Post-alignment learning dynamics correspond to gradient descent on an effective objective (see Appendix § 12 )
J = 1 4 c ( τ ) [ u ( τ ) v ( τ ) − 2 c ( τ ) c ∗ + c ( τ ) ] 2 + ( 1 4 c ∗ − c ( τ ) [ c ∗ + c ( τ ) ] 2 ) + σ 2 v ( τ ) 2 4 c ( τ ) + κ 2 [ u ( τ ) 2 + v ( τ ) 2 ] \begin{equation}
J = \frac{1}{4 c(\tau)} \left[ u(\tau) v(\tau) - \frac{2 c(\tau)}{c_* + c(\tau)} \right]^2 + \left( \frac{1}{4 c_*} - \frac{c(\tau)}{[ c_* + c(\tau) ]^2} \right) + \frac{\sigma^2 v(\tau)^2}{4 c(\tau)} + \frac{\kappa}{2} [ u(\tau)^2 + v(\tau)^2 ] \tag*{(18)}
\end{equation} J = 4 c ( τ ) 1 [ u ( τ ) v ( τ ) − c ∗ + c ( τ ) 2 c ( τ ) ] 2 + ( 4 c ∗ 1 − [ c ∗ + c ( τ ) ] 2 c ( τ ) ) + 4 c ( τ ) σ 2 v ( τ ) 2 + 2 κ [ u ( τ ) 2 + v ( τ ) 2 ] (18)
where we will refer to σ 2 > 0 \sigma^2 > 0 σ 2 > 0 as the “noise level”. Learning dynamics in this regime follow
d d τ u ( τ ) = v ( τ ) c ∗ + c ( τ ) − u ( τ ) v ( τ ) 2 2 c ( τ ) − κ u ( τ ) d d τ v ( τ ) = u ( τ ) c ∗ + c ( τ ) − v ( τ ) [ σ 2 + u ( τ ) 2 ] 2 c ( τ ) − κ v ( τ ) d d τ c ( τ ) = v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] 4 c ( τ ) 2 − u ( τ ) v ( τ ) [ c ∗ + c ( τ ) ] 2 . \begin{equation}
\begin{split}
\frac{d}{d\tau} u(\tau) &= \frac{v(\tau)}{c_* + c(\tau)} - \frac{u(\tau) v(\tau)^2}{2 c(\tau)} - \kappa u(\tau) \\
\frac{d}{d\tau} v(\tau) &= \frac{u(\tau)}{c_* + c(\tau)} - \frac{v(\tau) [ \sigma^2 + u(\tau)^2 ]}{2 c(\tau)} - \kappa v(\tau) \\
\frac{d}{d\tau} c(\tau) &= \frac{v(\tau)^2 [ u(\tau)^2 + \sigma^2 ]}{4 c(\tau)^2} - \frac{u(\tau) v(\tau)}{[c_* + c(\tau) ]^2} \ .
\end{split} \tag*{(19)}
\end{equation} d τ d u ( τ ) d τ d v ( τ ) d τ d c ( τ ) = c ∗ + c ( τ ) v ( τ ) − 2 c ( τ ) u ( τ ) v ( τ ) 2 − κ u ( τ ) = c ∗ + c ( τ ) u ( τ ) − 2 c ( τ ) v ( τ ) [ σ 2 + u ( τ ) 2 ] − κ v ( τ ) = 4 c ( τ ) 2 v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] − [ c ∗ + c ( τ ) ] 2 u ( τ ) v ( τ ) . (19)
In the remainder of this section, we will assume no regularization (κ = 0 \kappa = 0 κ = 0 ). Let N : = u v N := u v N := uv denote the filter amplitude, so that f ( t ) = N e − c t f(t) = N e^{- c t} f ( t ) = N e − c t . J J J is obviously minimized when N = 1 N = 1 N = 1 and c = c ∗ c = c_* c = c ∗ , since these choices set the first two terms to zero. Learning dynamics drive N N N and c c c toward those values (Figure 4 a).
A key difference between the noisy and noise-free (σ 2 = 0 \sigma^2 = 0 σ 2 = 0 ) dynamics is that noise breaks the symmetry between u u u and v v v . In the noise-free case, if u u u and v v v are initialized to the same value—i.e., if one makes a balancing assumption [22 , 4 ] —u = v u = v u = v holds throughout training. In the noisy case, if there is no regularization, u u u continues to grow while v v v continues to shrink (Figure 4 b). In fact, Eq. (19) implies that
d d τ [ u ( τ ) 2 − v ( τ ) 2 ] = σ 2 c ( τ ) v ( τ ) 2 > 0 , \begin{equation}
\frac{d}{d\tau} [ u(\tau)^2 - v(\tau)^2 ] = \frac{\sigma^2}{c(\tau)} v(\tau)^2 > 0 \ , \tag*{(20)}
\end{equation} d τ d [ u ( τ ) 2 − v ( τ ) 2 ] = c ( τ ) σ 2 v ( τ ) 2 > 0 , (20)
which means u → ∞ u \to \infty u → ∞ and v → 0 v \to 0 v → 0 . This makes sense, since taking N = u v = 1 N = u v = 1 N = uv = 1 and v → 0 v \to 0 v → 0 is precisely the combination that minimizes both the filter-matching term and the noise term (Figure 4 c).
A noise-induced phase transition in rich regime dynamics
Figure 5. Excessive noise causes amplitude collapse. For plotting, we take c ∗ = 1 c_* = 1 c ∗ = 1 and κ = 0.2 \kappa = 0.2 κ = 0.2 . (a) Below a critical noise level σ crit 2 \sigma^2_{\text{crit}} σ crit 2 , learning dynamics drives the filter amplitude N : = u v N := u v N := uv toward a nontrivial fixed point. Above that noise level, N N N is driven to zero. (b) The optimal amplitude N opt N_{\text{opt}} N opt as a function of noise level. For σ 2 ≥ σ crit 2 \sigma^2 \geq \sigma^2_{\text{crit}} σ 2 ≥ σ crit 2 , the trivial solution with f ( t ) = 0 f(t) = 0 f ( t ) = 0 is optimal. (c) Below σ crit 2 \sigma^2_{\text{crit}} σ crit 2 , J J J is minimized at a specific c c c value. Above it, J J J is insensitive to c c c since N = 0 N = 0 N = 0 . (d) The phase boundary σ crit 2 ( κ ) \sigma^2_{\text{crit}}(\kappa) σ crit 2 ( κ ) . The noise threshold separating trivial and nontrivial solutions depends on the regularization strength; it is higher when κ \kappa κ is small, and σ crit 2 → ∞ \sigma^2_{\text{crit}} \to \infty σ crit 2 → ∞ as κ → 0 \kappa \to 0 κ → 0 .
The behavior described in the previous section, aside from the asymmetry between u u u and v v v dynamics, is not substantially different from what [4 ] observed in the noise-free case. However, the noisy case becomes significantly different when the regularization strength κ \kappa κ is taken to be nonzero. Without regularization, we saw that the system minimizes noise by taking u → ∞ u \to \infty u → ∞ and v → 0 v \to 0 v → 0 ; however, if high u u u values are penalized, the system can become frustrated. If the noise level is sufficiently high, there is a phase transition, and a trivial solution—which minimizes noise at the cost of a collapse in filter-matching performance—becomes optimal.
Below a critical noise level σ crit 2 \sigma^2_{\text{crit}} σ crit 2 (which depends on the regularization strength κ \kappa κ ), the filter amplitude N : = u v N := u v N := uv approaches a nonzero fixed point N opt N_{\text{opt}} N opt . When κ = 0 \kappa = 0 κ = 0 , this equals one, but it decreases as κ \kappa κ increases, since the scales of u u u and v v v are penalized by the regularization term. Beyond the critical noise level, dynamics drive N N N to zero (Figure 5 a), and moreover J J J is minimized at N opt = 0 N_{\text{opt}} = 0 N opt = 0 (Figure 5 b).
One can show algebraically (see Appendix § 13 ) that, after minimizing J J J over u u u and v v v , it becomes
J ( c ) = 1 4 c ∗ − c [ 1 c ∗ + c − κ 1 + σ 2 2 κ c ] + 2 . \begin{equation}
\begin{split}
J(c) &= \frac{1}{4 c_*} - c \left[ \frac{1}{c_* + c} - \kappa \sqrt{1 + \frac{\sigma^2}{2 \kappa c}} \right]_+^2 \ .
\end{split} \tag*{(21)}
\end{equation} J ( c ) = 4 c ∗ 1 − c [ c ∗ + c 1 − κ 1 + 2 κ c σ 2 ] + 2 . (21)
where [ . . . ] + [ ... ]_+ [ ... ] + denotes rectification (i.e., a ReLU). This simplified objective only has a nontrivial solution if the bracketed quantity is greater than zero; otherwise, J J J becomes independent of c c c (Figure 5 c).
We can identify the corresponding phase boundary σ crit 2 ( κ ) \sigma^2_{\text{crit}}(\kappa) σ crit 2 ( κ ) by determining the noise level at which the bracketed quantity and its derivative equal zero. We find (see Appendix § 13 ) that
σ crit 2 ( κ ) = [ 1 − q ( κ ) 2 ] 2 q ( κ ) \begin{equation}
\boxed{ \ \sigma^2_{\text{crit}}(\kappa) = \frac{[ 1 - q(\kappa)^2 ]^2}{q(\kappa)} \ } \tag*{(22)}
\end{equation} σ crit 2 ( κ ) = q ( κ ) [ 1 − q ( κ ) 2 ] 2 (22)
for κ < 1 / c ∗ \kappa < 1/c_* κ < 1/ c ∗ , and σ crit 2 = 0 \sigma^2_{\text{crit}} = 0 σ crit 2 = 0 for κ ≥ 1 / c ∗ \kappa \geq 1/c_* κ ≥ 1/ c ∗ . Here,
q ( κ ) : = 2 3 sinh [ 1 3 arsinh ( 3 3 κ c ∗ ) ] and arsinh ( x ) : = log ( x + x 2 + 1 ) . \begin{equation}
q(\kappa) := \frac{2}{\sqrt{3}} \sinh\left[ \frac{1}{3} \text{arsinh}(3 \sqrt{3} \kappa c_* ) \right] \hspace{0.3in} \text{and} \hspace{0.3in} \text{arsinh}(x) := \log( x + \sqrt{x^2 + 1} ) \ . \tag*{(23)}
\end{equation} q ( κ ) := 3 2 sinh [ 3 1 arsinh ( 3 3 κ c ∗ ) ] and arsinh ( x ) := log ( x + x 2 + 1 ) . (23)
The phase boundary is plotted in Figure 5 d. It has a few features worth emphasizing. First, since q ( κ ) → 0 q(\kappa) \to 0 q ( κ ) → 0 as κ → 0 \kappa \to 0 κ → 0 , σ crit 2 \sigma^2_{\text{crit}} σ crit 2 diverges to infinity in this limit; this is consistent with the observation that there is no phase transition without regularization. Second, for small κ \kappa κ , we have the comparatively simple expressions
q ( κ ) ≈ 2 c ∗ κ σ crit 2 ( κ ) ≈ 1 2 c ∗ κ \begin{equation}
q(\kappa) \approx 2 c_* \kappa \hspace{1in} \sigma^2_{\text{crit}}(\kappa) \approx \frac{1}{2 c_* \kappa} \tag*{(24)}
\end{equation} q ( κ ) ≈ 2 c ∗ κ σ crit 2 ( κ ) ≈ 2 c ∗ κ 1 (24)
which qualitatively match Figure 5 d. In summary, if there is too much noise, it is almost as if the system’s learning dynamics seek an ‘escape route’: parameters are pushed into trivial regimes (here, the filter amplitude N N N is taken to zero) in order to minimize noise at the expense of task accuracy.
A second phase transition when scale changes are impossible
Figure 6. Excessive noise produces an infinitely fast filter. c ∗ = 1 c_* = 1 c ∗ = 1 assumed throughout. (a) When the read-in and readout scales are fixed, J J J is a function of c c c only, and for low enough noise it has a single attractor basin at c = c opt c = c_{\text{opt}} c = c opt . As σ 2 \sigma^2 σ 2 increases, this basin becomes increasingly shallow. (b) For σ 2 ≥ σ crit 2 = 3 \sigma^2 \geq \sigma^2_{\text{crit}} = 3 σ 2 ≥ σ crit 2 = 3 , J J J is a decreasing function of c c c with no finite minimum. (c) Below the noise threshold, dynamics drive c c c toward the unique stable fixed point; above it, c c c grows without bound. (d) c opt c_{\text{opt}} c opt diverges like 1 / ( σ crit 2 − σ 2 ) 1/(\sigma^2_{\text{crit}} - \sigma^2) 1/ ( σ crit 2 − σ 2 ) just below the noise threshold.
We just saw how, for sufficiently large intrinsic noise σ 2 > 0 \sigma^2 > 0 σ 2 > 0 , the system may converge to a trivial solution with zero amplitude, i.e., u v = 0 u v = 0 uv = 0 . One may wonder whether this behavior is peculiar to our setup—for example, whether it depends on trainable read-in and readout weights. Interestingly, if u u u and v v v are frozen, a different type of noise-induced phase transition is possible.
Consider the u = v = 1 u = v = 1 u = v = 1 specialization of the simplified objective (Eq. (18) ):
J = 1 4 c ( τ ) [ 1 − 2 c ( τ ) c ∗ + c ( τ ) ] 2 + 1 4 c ∗ − c ( τ ) [ c ∗ + c ( τ ) ] 2 + σ 2 4 c ( τ ) = 1 + σ 2 4 c ( τ ) − 1 c ∗ + c ( τ ) + 1 4 c ∗ \begin{equation}
J = \frac{1}{4 c(\tau)} \left[ 1 - \frac{2 c(\tau)}{c_* + c(\tau)} \right]^2 + \frac{1}{4 c_*} - \frac{c(\tau)}{[ c_* + c(\tau) ]^2} + \frac{\sigma^2}{4 c(\tau)} = \frac{1 + \sigma^2}{4 c(\tau)} - \frac{1}{c_* + c(\tau)} + \frac{1}{4 c_*} \tag*{(25)}
\end{equation} J = 4 c ( τ ) 1 [ 1 − c ∗ + c ( τ ) 2 c ( τ ) ] 2 + 4 c ∗ 1 − [ c ∗ + c ( τ ) ] 2 c ( τ ) + 4 c ( τ ) σ 2 = 4 c ( τ ) 1 + σ 2 − c ∗ + c ( τ ) 1 + 4 c ∗ 1 (25)
where we have removed the regularization term because it is constant. The corresponding learning dynamics are (see Appendix § 14 )
d d τ c = [ 1 + σ 2 ( c ∗ + c ) + 2 c ] 4 c 2 ( c ∗ + c ) 2 [ 1 + σ 2 ( c ∗ + c ) − 2 c ] \begin{equation}
\frac{d}{d\tau} c = \frac{[ \sqrt{1 + \sigma^2} (c_* + c) + 2 c] }{4 c^2 (c_* + c)^2} [ \sqrt{1 + \sigma^2} (c_* + c) - 2 c] \tag*{(26)}
\end{equation} d τ d c = 4 c 2 ( c ∗ + c ) 2 [ 1 + σ 2 ( c ∗ + c ) + 2 c ] [ 1 + σ 2 ( c ∗ + c ) − 2 c ] (26)
For sufficiently small noise, these dynamics have a stable positive fixed point at
c opt = 1 + σ 2 c ∗ 2 − 1 + σ 2 . \begin{equation}
c_{\text{opt}} = \frac{\sqrt{1 + \sigma^2} \ c_*}{2 - \sqrt{1 + \sigma^2}} \ . \tag*{(27)}
\end{equation} c opt = 2 − 1 + σ 2 1 + σ 2 c ∗ . (27)
As one might expect, c opt c_{\text{opt}} c opt equals the target rate c ∗ c_* c ∗ when there is no noise. However, as σ 2 \sigma^2 σ 2 increases, the corresponding basin of J J J becomes increasingly shallow (Figure 6 a). Pushing the noise level higher than σ crit 2 = 3 \sigma^2_{\text{crit}} = 3 σ crit 2 = 3 causes this basin to disappear (Figure 6 b), and for J J J to be minimized at c = ∞ c = \infty c = ∞ . Indeed, Eq. (26) has d c / d τ > 0 dc/d\tau > 0 d c / d τ > 0 for all c c c if σ 2 ≥ 3 \sigma^2 \geq 3 σ 2 ≥ 3 , so learning drives c c c toward infinity (Figure 6 c). Just below σ crit 2 \sigma^2_{\text{crit}} σ crit 2 , the optimal c c c value diverges as 1 / ( σ crit 2 − σ 2 ) 1/(\sigma^2_{\text{crit}} - \sigma^2) 1/ ( σ crit 2 − σ 2 ) (Figure 6 d). We conclude that, although the precise representational ‘escape route’ may vary, the system may be driven toward trivial solutions that minimize noise when σ 2 \sigma^2 σ 2 is excessively large.
Discussion
We developed a theory of noisy linear RNN learning dynamics and mathematically showed that noise can decisively shape representation learning. Consistent with [4 ] , in the ‘rich’ regime, where parameters are initialized to small values, a low-rank task (here, matching a single leaky integration time scale) yields an approximately low-rank solution. But our findings differ from those of [4 ] in three important ways. First, noise strongly modulates the initial alignment phase, and u \bm{u} u , v \bm{v} v , and W \bm{W} W preferentially align along the least noisy eigenvector (or eigenvectors) of S \bm{S} S , rather than simply in the average direction of the initial u \bm{u} u and v \bm{v} v ; this is interesting in part because it makes the network’s solution depend much less on its initial condition.
Second, noise produces asymmetry between the read-in and readout weights, and in the second phase u u u tends to become large while v v v tends to become small. This is true even if u u u and v v v are initialized to identical values because noise breaks the balancing property that holds in the noise-free case. Finally, noise produces multiple types of phase transitions, because minimizing noise can compete with achieving good task performance. If u \bm{u} u and v \bm{v} v are trainable, we find that excessive noise drives the network toward a solution whose filter amplitude N : = u v N := u v N := uv equals zero; interestingly, this only occurs if regularization constrains the scales of u u u and v v v . If only W \bm{W} W is trainable, excessive noise drives the system to f ( t ) = 0 f(t) = 0 f ( t ) = 0 in a different way, by making the filter’s frequency infinitely fast.
In other machine learning settings, noise is known to shape what types of representations systems learn. Noise in optimization due to using stochastic rather than full-batch gradient descent tends to produce convergence to broader loss landscape minima that generalize better [24 , 23 ] . Noise in diffusion generative modeling may similarly improve generalization, albeit potentially through a different mechanism [25 ] . Our work suggests that noise shapes RNN representation learning, but does not necessarily produce better solutions—only different ones. Here, noise reduces solution-space degeneracy by constraining the direction of weight alignment and the relative scales of the read-in and readout weights. If there is too much noise, it is clear that noise has deleterious effects, since it drives the system toward trivial solutions.
Limitations and future work.
For tractability’s sake, we made a large number of simplifying assumptions: we assumed a whitened scalar input; a linear network; a simple filter-matching task involving learning one time scale; and near-zero parameter initializations that yield low-rank learning dynamics. Although the work we build on makes similar assumptions, it may be possible to relax some of them. For example, it may be possible to study nonlinear networks by linearizing them around a fixed point (either in state space or parameter space), or by exploiting dynamical mean-field theory machinery [26 ] . One class of tasks which is more realistic, but whose learning dynamics may remain tractable, is target pursuit [27 ] .
Acknowledgments
KR was funded by the NIH (RF1DA056403, U01NS136507), James S. McDonnell Foundation (220020466), Simons Foundation (Pilot Extension-00003332-02), McKnight Endowment Fund, CIFAR Azrieli Global Scholar Program, NSF (2046583), a Harvard Medical School Neurobiology Lefler Small Grant Award, and a Harvard Medical School Dean’s Innovation Award. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University.
AI use statement
In this work, we used generative AI tools for brainstorming, literature search, to assist with checking the mathematical correctness of our results, and to assist with code generation. The authors have checked all text and results and take full responsibility for them.
Objective function and derivation of learning dynamics
In this appendix, we simplify our original objective function and derive expressions for the associated gradient descent learning dynamics (in the gradient-flow limit that yields ODEs rather than a discrete-time recurrence).
Setup
Recall from § 2 that the objective function has the form
J : = lim T → ∞ 1 T ∫ 0 T 1 2 E x , r { [ y ( t ) − y ∗ ( t ) ] 2 } d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 \begin{equation}
\boxed{ \ J := \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_{x, \bm{r}}\left\{ \ [ y(t) - y_*(t) ]^2 \ \right\} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ } \tag*{(28)}
\end{equation} J := T → ∞ lim T 1 ∫ 0 T 2 1 E x , r { [ y ( t ) − y ∗ ( t ) ] 2 } d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 (28)
where y ( t ) = v T r ( t ) y(t) = \bm{v}^T \bm{r}(t) y ( t ) = v T r ( t ) is the RNN’s output and
y ∗ ( t ) : = ∫ 0 t e − c ∗ t ′ x ( t − t ′ ) d t ′ = ∫ 0 t f ∗ ( t ′ ) x ( t − t ′ ) d t ′ \begin{equation}
y_*(t) := \int_0^t e^{- c_* t'} x(t - t') \ dt' = \int_0^t f_*(t') x(t - t') \ dt' \tag*{(29)}
\end{equation} y ∗ ( t ) := ∫ 0 t e − c ∗ t ′ x ( t − t ′ ) d t ′ = ∫ 0 t f ∗ ( t ′ ) x ( t − t ′ ) d t ′ (29)
is the target output. If we assume zero initial activity (i.e., r ( 0 ) = 0 \bm{r}(0) = \bm{0} r ( 0 ) = 0 ), then as we mention in § 2 ,
p ( r , t ∣ r 0 = 0 , 0 ) = N ( ∫ 0 t e A t ′ u x ( t − t ′ ) d t ′ , ∫ 0 t e A t ′ S e A T t ′ d t ′ ) . \begin{equation}
p(\bm{r} , t | \bm{r}_0 = \bm{0}, 0) = \mathcal{N}\left( \int_0^t e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' , \int_0^t e^{\bm{A} t'} \bm{S} e^{\bm{A}^T t'} \ dt' \right) \ . \tag*{(30)}
\end{equation} p ( r , t ∣ r 0 = 0 , 0 ) = N ( ∫ 0 t e A t ′ u x ( t − t ′ ) d t ′ , ∫ 0 t e A t ′ S e A T t ′ d t ′ ) . (30)
This implies that the output y ( t ) y(t) y ( t ) has statistics
E r [ y ( t ) ] = ∫ 0 t v T e A t ′ u x ( t − t ′ ) d t ′ = ∫ 0 t f ( t ′ ) x ( t − t ′ ) d t ′ var r [ y ( t ) ] = v T Σ v \begin{equation}
\begin{split}
\mathbb{E}_{\bm{r}}[ y(t) ] &= \int_0^t \bm{v}^T e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' = \int_0^t f(t') \ x(t-t') \ dt' \\
\text{var}_{\bm{r}}[ y(t) ] &= \bm{v}^T \bm{\Sigma} \bm{v}
\end{split} \tag*{(31)}
\end{equation} E r [ y ( t )] var r [ y ( t )] = ∫ 0 t v T e A t ′ u x ( t − t ′ ) d t ′ = ∫ 0 t f ( t ′ ) x ( t − t ′ ) d t ′ = v T Σ v (31)
where f ( t ) : = e − γ t v T e W t u f(t) := e^{- \gamma t} \bm{v}^T e^{\bm{W} t} \bm{u} f ( t ) := e − γ t v T e W t u is the learned filter and the covariance Σ \bm{\Sigma} Σ is defined as
Σ : = ∫ 0 t e A t ′ S e A T t ′ d t ′ . \begin{equation}
\bm{\Sigma} := \int_0^t e^{\bm{A} t'} \bm{S} e^{\bm{A}^T t'} \ dt' \ . \tag*{(32)}
\end{equation} Σ := ∫ 0 t e A t ′ S e A T t ′ d t ′ . (32)
Also recall that the input signal x ( t ) x(t) x ( t ) is assumed to have white noise statistics, i.e., E x [ x ( t ) ] = 0 \mathbb{E}_x[ x(t) ] = 0 E x [ x ( t )] = 0 and E x [ x ( t ) x ( t − t ′ ) ] = δ ( t − t ′ ) \mathbb{E}_x[ x(t) x(t - t') ] = \delta(t - t') E x [ x ( t ) x ( t − t ′ )] = δ ( t − t ′ ) .
Simplifying the objective
Decomposing into a bias and variance term.
First, average over r \bm{r} r :
J = lim T → ∞ 1 T ∫ 0 T 1 2 E x { [ E r [ y ( t ) ] − y ∗ ( t ) ] 2 + var r [ y ( t ) ] } d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 = lim T → ∞ 1 T ∫ 0 T 1 2 E x { [ E r [ y ( t ) ] − y ∗ ( t ) ] 2 } + 1 2 var r [ y ( t ) ] d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 = lim T → ∞ 1 T ∫ 0 T 1 2 E x { [ E r [ y ( t ) ] − y ∗ ( t ) ] 2 } + 1 2 v T Σ ( t ) v d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 \begin{equation}
\begin{split}
J &= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_x\left\{ \ [ \mathbb{E}_{\bm{r}}[ y(t) ] - y_*(t) ]^2 + \text{var}_{\bm{r}}[ y(t) ] \ \right\} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \\
&= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_x\left\{ \ [ \mathbb{E}_{\bm{r}}[ y(t) ] - y_*(t) ]^2 \ \right\} + \frac{1}{2} \text{var}_{\bm{r}}[ y(t) ] \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \\
&= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_x\left\{ \ [ \mathbb{E}_{\bm{r}}[ y(t) ] - y_*(t) ]^2 \ \right\} + \frac{1}{2} \bm{v}^T \bm{\Sigma}(t) \bm{v} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2
\end{split} \tag*{(33)}
\end{equation} J = T → ∞ lim T 1 ∫ 0 T 2 1 E x { [ E r [ y ( t )] − y ∗ ( t ) ] 2 + var r [ y ( t )] } d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 = T → ∞ lim T 1 ∫ 0 T 2 1 E x { [ E r [ y ( t )] − y ∗ ( t ) ] 2 } + 2 1 var r [ y ( t )] d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 = T → ∞ lim T 1 ∫ 0 T 2 1 E x { [ E r [ y ( t )] − y ∗ ( t ) ] 2 } + 2 1 v T Σ ( t ) v d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 (33)
where the second equality uses the fact that variance in y ( t ) y(t) y ( t ) is input-independent.
Simplifying the mean term.
The second term is input-independent; it represents noise due to recurrent dynamics. The first term can be written in terms of filters:
= lim T → ∞ 1 T ∫ 0 T 1 2 E x { ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s ) ] [ f ( s ′ ) − f ∗ ( s ′ ) ] x ( t − s ) x ( t − s ′ ) } d s d s ′ d t = lim T → ∞ 1 T ∫ 0 T 1 2 ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s ) ] [ f ( s ′ ) − f ∗ ( s ′ ) ] E x [ x ( t − s ) x ( t − s ′ ) ] d s d s ′ d t = lim T → ∞ 1 T ∫ 0 T 1 2 ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s ) ] [ f ( s ′ ) − f ∗ ( s ′ ) ] δ ( s − s ′ ) d s d s ′ d t = lim T → ∞ 1 T ∫ 0 T 1 2 ∫ 0 t [ f ( s ) − f ∗ ( s ) ] 2 d s d t . \begin{equation}
\begin{split}
\ &= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_{x}\left\{ \ \int_0^t \int_0^t [ f(s) - f_*(s) ] [ f(s') - f_*(s') ] \ x(t-s) x(t - s') \ \right\} \ ds ds' dt \\
&= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \ \int_0^t \int_0^t [ f(s) - f_*(s) ] [ f(s') - f_*(s') ] \ \mathbb{E}_x[x(t-s) x(t - s')] \ ds ds' dt \\
&= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \ \int_0^t \int_0^t [ f(s) - f_*(s) ] [ f(s') - f_*(s') ] \ \delta(s - s') \ ds ds' dt \\
&= \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \ \int_0^t [ f(s) - f_*(s) ]^2 \ ds dt \ .
\end{split} \tag*{(34)}
\end{equation} = T → ∞ lim T 1 ∫ 0 T 2 1 E x { ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s )] [ f ( s ′ ) − f ∗ ( s ′ )] x ( t − s ) x ( t − s ′ ) } d s d s ′ d t = T → ∞ lim T 1 ∫ 0 T 2 1 ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s )] [ f ( s ′ ) − f ∗ ( s ′ )] E x [ x ( t − s ) x ( t − s ′ )] d s d s ′ d t = T → ∞ lim T 1 ∫ 0 T 2 1 ∫ 0 t ∫ 0 t [ f ( s ) − f ∗ ( s )] [ f ( s ′ ) − f ∗ ( s ′ )] δ ( s − s ′ ) d s d s ′ d t = T → ∞ lim T 1 ∫ 0 T 2 1 ∫ 0 t [ f ( s ) − f ∗ ( s ) ] 2 d s d t . (34)
This can be simplified with a clever rearrangement. Note,
lim T → ∞ 1 2 T ∫ 0 T ∫ 0 t [ f ( s ) − f ∗ ( s ) ] 2 d s d t = lim T → ∞ 1 2 T ∫ 0 T ∫ s T [ f ( s ) − f ∗ ( s ) ] 2 d t d s = lim T → ∞ 1 2 ∫ 0 T ( 1 − s T ) [ f ( s ) − f ∗ ( s ) ] 2 d s . \begin{equation}
\begin{split}
\lim_{T \to \infty} \frac{1}{2 T} \int_0^T \int_0^t [ f(s) - f_*(s) ]^2 \ ds dt &= \lim_{T \to \infty} \frac{1}{2 T} \int_0^T \int_{s}^T [ f(s) - f_*(s) ]^2 \ dt ds \\
&= \lim_{T \to \infty} \frac{1}{2} \int_0^T \left( 1 - \frac{s}{T} \right) [ f(s) - f_*(s) ]^2 \ ds \ .
\end{split} \tag*{(35)}
\end{equation} T → ∞ lim 2 T 1 ∫ 0 T ∫ 0 t [ f ( s ) − f ∗ ( s ) ] 2 d s d t = T → ∞ lim 2 T 1 ∫ 0 T ∫ s T [ f ( s ) − f ∗ ( s ) ] 2 d t d s = T → ∞ lim 2 1 ∫ 0 T ( 1 − T s ) [ f ( s ) − f ∗ ( s ) ] 2 d s . (35)
If the eigenvalues of A \bm{A} A have negative real parts (i.e., if dynamics is stable), then we can drop the s / T s/T s / T term, since it vanishes in the T → ∞ T \to \infty T → ∞ limit. We conclude that
J filter = 1 2 ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] 2 d t . \begin{equation}
J_{\text{filter}} = \frac{1}{2} \int_0^{\infty} [ f(t) - f_*(t) ]^2 \ dt \ . \tag*{(36)}
\end{equation} J filter = 2 1 ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] 2 d t . (36)
Simplifying the variance term.
The noise term can be rewritten as
lim T → ∞ 1 2 T ∫ 0 T v T Σ ( t ) v d t = v T { lim T → ∞ 1 2 T ∫ 0 T ∫ 0 t e A s S e A T s d s d t } v = v T { lim T → ∞ 1 2 T ∫ 0 T ∫ s T e A s S e A T s d t d s } v = v T { lim T → ∞ 1 2 ∫ 0 T ( 1 − s T ) e A s S e A T s d s } v . \begin{equation}
\begin{split}
& \ \lim_{T \to \infty} \frac{1}{2 T} \int_0^T \bm{v}^T \bm{\Sigma}(t) \bm{v} \ dt \\
=& \ \bm{v}^T \left\{ \ \lim_{T \to \infty} \frac{1}{2 T} \int_0^T \int_0^t e^{\bm{A} s} \bm{S} e^{\bm{A}^T s} \ ds dt \ \right\} \bm{v} \\
=& \ \bm{v}^T \left\{ \ \lim_{T \to \infty} \frac{1}{2 T} \int_0^T \int_{s}^T e^{\bm{A} s} \bm{S} e^{\bm{A}^T s} \ dt ds \ \right\} \bm{v} \\
=& \ \bm{v}^T \left\{ \ \lim_{T \to \infty} \frac{1}{2} \int_0^T \left( 1 - \frac{s}{T} \right) e^{\bm{A} s} \bm{S} e^{\bm{A}^T s} \ ds \ \right\} \bm{v} \ .
\end{split} \tag*{(37)}
\end{equation} = = = T → ∞ lim 2 T 1 ∫ 0 T v T Σ ( t ) v d t v T { T → ∞ lim 2 T 1 ∫ 0 T ∫ 0 t e A s S e A T s d s d t } v v T { T → ∞ lim 2 T 1 ∫ 0 T ∫ s T e A s S e A T s d t d s } v v T { T → ∞ lim 2 1 ∫ 0 T ( 1 − T s ) e A s S e A T s d s } v . (37)
Once again, if dynamics are stable, we can drop the s / T s/T s / T term in the infinite T T T limit. We obtain
J noise = 1 2 ∫ 0 ∞ v T e A t S e A T t v d t . \begin{equation}
J_{\text{noise}} = \frac{1}{2} \int_0^{\infty} \bm{v}^T e^{\bm{A} t} \bm{S} e^{\bm{A}^T t} \bm{v} \ dt \ . \tag*{(38)}
\end{equation} J noise = 2 1 ∫ 0 ∞ v T e A t S e A T t v d t . (38)
Total objective.
Adding all of the terms, we obtain the simplified objective
J ( u , v , W ) = ∫ 0 ∞ 1 2 [ f ( t ) − f ∗ ( t ) ] 2 + e − 2 γ t 2 v T e W t S e W T t v d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 . \begin{equation}
\boxed{ \
J(\bm{u}, \bm{v}, \bm{W}) = \int_0^{\infty} \frac{1}{2} \left[ f(t) - f_*(t) \right]^2 + \frac{e^{- 2 \gamma t}}{2} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t} \bm{v} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ } \ . \tag*{(39)}
\end{equation} J ( u , v , W ) = ∫ 0 ∞ 2 1 [ f ( t ) − f ∗ ( t ) ] 2 + 2 e − 2 γ t v T e W t S e W T t v d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 . (39)
Useful derivatives
A few derivatives are necessary to compute the gradients of J J J . First, the derivative of the matrix exponential with respect to W \bm{W} W has components
∂ ∂ W i j e W t = ∫ 0 t d t ′ e W t ′ e i e j T e W ( t − t ′ ) . \begin{equation}
\frac{\partial}{\partial W_{i j}} e^{\bm{W} t} = \int_0^t dt' \ e^{\bm{W} t'} \bm{e}_i \bm{e}_j^T e^{\bm{W} (t - t')} \ . \tag*{(40)}
\end{equation} ∂ W ij ∂ e W t = ∫ 0 t d t ′ e W t ′ e i e j T e W ( t − t ′ ) . (40)
Similarly, the derivative of its transpose is
∂ ∂ W i j e W T t = ∫ 0 t d t ′ e W T t ′ e j e i T e W T ( t − t ′ ) . \begin{equation}
\frac{\partial}{\partial W_{i j}} e^{\bm{W}^T t} = \int_0^t dt' \ e^{\bm{W}^T t'} \bm{e}_j \bm{e}_i^T e^{\bm{W}^T (t - t')} \ . \tag*{(41)}
\end{equation} ∂ W ij ∂ e W T t = ∫ 0 t d t ′ e W T t ′ e j e i T e W T ( t − t ′ ) . (41)
The derivative of the learned filter f ( t ) f(t) f ( t ) with respect to W \bm{W} W has components
∂ ∂ W i j f ( t ) = e − γ t ∫ 0 t d t ′ [ e W T t ′ v ] i [ e W ( t − t ′ ) u ] j = e − γ t ∫ 0 t d t ′ e W T t ′ v u T e W T ( t − t ′ ) . \begin{equation}
\frac{\partial}{\partial W_{i j}} f(t) = e^{- \gamma t} \int_0^t dt' \ \left[ e^{\bm{W}^T t'} \bm{v} \right]_i \left[ e^{\bm{W} (t - t')} \bm{u} \right]_j = e^{- \gamma t} \int_0^t dt' \ e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ . \tag*{(42)}
\end{equation} ∂ W ij ∂ f ( t ) = e − γ t ∫ 0 t d t ′ [ e W T t ′ v ] i [ e W ( t − t ′ ) u ] j = e − γ t ∫ 0 t d t ′ e W T t ′ v u T e W T ( t − t ′ ) . (42)
Finally, the derivative of the variance term with respect to W \bm{W} W has components
∂ ∂ W i j v T e W t S e W T t v = ∫ 0 t d t ′ v T e W t ′ e i e j T e W ( t − t ′ ) S e W T t v + ∫ 0 t d t ′ v T e W t S e W T t ′ e j e i T e W T ( t − t ′ ) v = ∫ 0 t d t ′ e W T t ′ v v T e W t S e W T ( t − t ′ ) + ∫ 0 t d t ′ e W T ( t − t ′ ) v v T e W t S e W T t ′ . \begin{equation}
\begin{split}
& \ \frac{\partial}{\partial W_{i j}} \bm{v}^T e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \\
=& \ \int_0^t dt' \ \bm{v}^T e^{\bm{W} t'} \bm{e}_i \bm{e}_j^T e^{\bm{W} (t - t')} \bm{S} e^{\bm{W}^T t} \bm{v} + \int_0^t dt' \ \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t'} \bm{e}_j \bm{e}_i^T e^{\bm{W}^T (t - t')} \bm{v} \\
=& \ \int_0^t dt' \ e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} + \int_0^t dt' \ e^{\bm{W}^T (t - t')} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t'} \ .
\end{split} \tag*{(43)}
\end{equation} = = ∂ W ij ∂ v T e W t S e W T t v ∫ 0 t d t ′ v T e W t ′ e i e j T e W ( t − t ′ ) S e W T t v + ∫ 0 t d t ′ v T e W t S e W T t ′ e j e i T e W T ( t − t ′ ) v ∫ 0 t d t ′ e W T t ′ v v T e W t S e W T ( t − t ′ ) + ∫ 0 t d t ′ e W T ( t − t ′ ) v v T e W t S e W T t ′ . (43)
Derivation of learning dynamics
Using the aforementioned derivatives, we can immediately compute the relevant gradients of J J J :
∂ J ∂ u = ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] ∂ f ( t ) ∂ u d t + κ u = ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] e − γ t e W T t v d t + κ u ∂ J ∂ v = ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] ∂ f ( t ) ∂ v + e − 2 γ t e W t S e W T t v d t + κ v = ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] e − γ t e W t u + e − 2 γ t e W t S e W T t v d t + κ v ∂ J ∂ W = ∫ 0 ∞ [ f ( t ) − f ∗ ( t ) ] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t + ∫ 0 ∞ e − 2 γ t 2 ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t + ∫ 0 ∞ e − 2 γ t 2 ∫ 0 t e W T ( t − t ′ ) v v T e W t S e W T t ′ d t ′ d t . \begin{equation}
\begin{split}
\frac{\partial J}{\partial \bm{u}} &= \int_0^{\infty} [ f(t) - f_*(t) ] \frac{\partial f(t)}{\partial \bm{u}} dt + \kappa \bm{u} \\
&= \int_0^{\infty} [ f(t) - f_*(t) ] e^{- \gamma t} e^{\bm{W}^T t} \bm{v} \ dt + \kappa \bm{u} \\
\frac{\partial J}{\partial \bm{v}} &= \int_0^{\infty} [ f(t) - f_*(t) ] \frac{\partial f(t)}{\partial \bm{v}} + e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt + \kappa \bm{v} \\
&= \int_0^{\infty} [ f(t) - f_*(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} + e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt + \kappa \bm{v} \\
\frac{\partial J}{\partial \bm{W}} &= \int_0^{\infty} [ f(t) - f_*(t) ] e^{- \gamma t} \int_0^t \ e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ dt' dt \\
&+ \int_0^{\infty} \frac{e^{-2\gamma t}}{2} \int_0^t e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} \ dt' dt + \int_0^{\infty} \frac{e^{-2\gamma t}}{2} \int_0^t \ e^{\bm{W}^T (t - t')} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t'} \ dt' dt \ .
\end{split} \tag*{(44)}
\end{equation} ∂ u ∂ J ∂ v ∂ J ∂ W ∂ J = ∫ 0 ∞ [ f ( t ) − f ∗ ( t )] ∂ u ∂ f ( t ) d t + κ u = ∫ 0 ∞ [ f ( t ) − f ∗ ( t )] e − γ t e W T t v d t + κ u = ∫ 0 ∞ [ f ( t ) − f ∗ ( t )] ∂ v ∂ f ( t ) + e − 2 γ t e W t S e W T t v d t + κ v = ∫ 0 ∞ [ f ( t ) − f ∗ ( t )] e − γ t e W t u + e − 2 γ t e W t S e W T t v d t + κ v = ∫ 0 ∞ [ f ( t ) − f ∗ ( t )] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t + ∫ 0 ∞ 2 e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t + ∫ 0 ∞ 2 e − 2 γ t ∫ 0 t e W T ( t − t ′ ) v v T e W t S e W T t ′ d t ′ d t . (44)
This result immediately implies that our gradient descent learning dynamics have the form
d d τ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W T t v d t − κ u d d τ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v d d τ W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t 2 ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t 2 ∫ 0 t e W T ( t − t ′ ) v v T e W t S e W T t ′ d t ′ d t . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W}^T t} \bm{v} \ dt - \kappa \bm{u} \\
\frac{d}{d\tau} \bm{v} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt - \kappa \bm{v} \\
\frac{d}{d\tau} \bm{W} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} \int_0^t e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ dt' dt\\
&- \int_0^{\infty} \frac{e^{-2\gamma t}}{2} \int_0^t \ e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} \ dt' dt - \int_0^{\infty} \frac{e^{-2\gamma t}}{2} \int_0^t \ e^{\bm{W}^T (t - t')} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t'} \ dt' dt \ .
\end{split} \tag*{(45)}
\end{equation} d τ d u d τ d v d τ d W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W T t v d t − κ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ 2 e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ 2 e − 2 γ t ∫ 0 t e W T ( t − t ′ ) v v T e W t S e W T t ′ d t ′ d t . (45)
By a change of variables t ′ → t − t ′ t' \to t - t' t ′ → t − t ′ , one can show that the two noise terms in the W \bm{W} W dynamics are identical. This allows the final result to be written in a slightly more compact form:
d d τ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W T t v d t − κ u d d τ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v d d τ W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W}^T t} \bm{v} \ dt - \kappa \bm{u} \\
\frac{d}{d\tau} \bm{v} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt - \kappa \bm{v} \\
\frac{d}{d\tau} \bm{W} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} \int_0^t e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ dt' dt\\
&- \int_0^{\infty} e^{-2\gamma t} \int_0^t \ e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} \ dt' dt \ .
\end{split} \tag*{(46)}
\end{equation} d τ d u d τ d v d τ d W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W T t v d t − κ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t . (46)
Learning dynamics given small initial weights
We can expand these learning dynamics to leading order in u \bm{u} u , v \bm{v} v , and W \bm{W} W (supposing they are all of order ϵ > 0 \epsilon > 0 ϵ > 0 , say). Doing so, we find that
d d τ u ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t v d t − κ u d d τ v ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t u − e − 2 γ t S v d t − κ v d d τ W ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t ∫ 0 t v u T d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t v v T S d t ′ d t . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &\approx \int_0^{\infty} f_*(t) e^{- \gamma t} \bm{v} \ dt - \kappa \bm{u} \\
\frac{d}{d\tau} \bm{v} &\approx \int_0^{\infty} f_*(t) e^{- \gamma t} \bm{u} - e^{- 2 \gamma t} \bm{S} \bm{v} \ dt - \kappa \bm{v} \\
\frac{d}{d\tau} \bm{W} &\approx \int_0^{\infty} f_*(t) e^{- \gamma t} \int_0^t \ \bm{v} \bm{u}^T \ dt' dt - \int_0^{\infty} e^{-2\gamma t} \int_0^t \ \bm{v} \bm{v}^T \bm{S} \ dt' dt \ .
\end{split} \tag*{(47)}
\end{equation} d τ d u d τ d v d τ d W ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t v d t − κ u ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t u − e − 2 γ t S v d t − κ v ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t ∫ 0 t v u T d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t v v T S d t ′ d t . (47)
We can be more explicit using the fact that f ∗ ( t ) = e − c ∗ t f_*(t) = e^{- c_* t} f ∗ ( t ) = e − c ∗ t . Simplifying yields
d d τ u ≈ 1 c ∗ + γ v − κ u d d τ v ≈ 1 c ∗ + γ u − 1 2 γ S v − κ v d d τ W ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t ∫ 0 t v u T d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t v v T S d t ′ d t ≈ 1 ( c ∗ + γ ) 2 v u T − 1 4 γ 2 v v T S . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &\approx \frac{1}{c_* + \gamma} \bm{v} - \kappa \bm{u} \\
\frac{d}{d\tau} \bm{v} &\approx \frac{1}{c_* + \gamma} \bm{u} - \frac{1}{2 \gamma} \bm{S} \bm{v} - \kappa \bm{v} \\
\frac{d}{d\tau} \bm{W} &\approx \int_0^{\infty} f_*(t) e^{- \gamma t} \int_0^t \ \bm{v} \bm{u}^T \ dt' dt - \int_0^{\infty} e^{-2\gamma t} \int_0^t \ \bm{v} \bm{v}^T \bm{S} \ dt' dt \\
&\approx \frac{1}{(c_* + \gamma)^2} \bm{v} \bm{u}^T - \frac{1}{4 \gamma^2} \bm{v} \bm{v}^T \bm{S} \ .
\end{split} \tag*{(48)}
\end{equation} d τ d u d τ d v d τ d W ≈ c ∗ + γ 1 v − κ u ≈ c ∗ + γ 1 u − 2 γ 1 S v − κ v ≈ ∫ 0 ∞ f ∗ ( t ) e − γ t ∫ 0 t v u T d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t v v T S d t ′ d t ≈ ( c ∗ + γ ) 2 1 v u T − 4 γ 2 1 v v T S . (48)
Reservoir computing case
In this appendix, we work out the learning dynamics of the reservoir computing model in which only the readout v \bm{v} v is trained. Its primary virtue is that it illustrates, even when learning dynamics are linear, how noise produces representational pressure. For simplicity, we ignore regularization and set κ = 0 \kappa = 0 κ = 0 .
Recall from Appendix § 9 that v \bm{v} v has dynamics
d d τ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t = ∫ 0 ∞ e − ( c ∗ + γ ) t e W t u − e − 2 γ t e W t u u T e W T t v − e − 2 γ t e W t S e W T t v d t = [ ∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t ] u − [ ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t ] v . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{v} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt \\
&= \int_0^{\infty} e^{- (c_* + \gamma) t} e^{\bm{W} t} \bm{u} - e^{- 2\gamma t} e^{\bm{W} t} \bm{u} \bm{u}^T e^{\bm{W}^T t} \bm{v} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt \\
&= \left[ \int_0^{\infty} e^{- (c_* + \gamma) t} e^{\bm{W} t} \ dt \right] \bm{u} - \left[ \int_0^{\infty} e^{- 2\gamma t} e^{\bm{W} t} (\bm{u} \bm{u}^T + \bm{S}) e^{\bm{W}^T t} \ dt \right] \bm{v} \ .
\end{split} \tag*{(49)}
\end{equation} d τ d v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t = ∫ 0 ∞ e − ( c ∗ + γ ) t e W t u − e − 2 γ t e W t u u T e W T t v − e − 2 γ t e W t S e W T t v d t = [ ∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t ] u − [ ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t ] v . (49)
These dynamics are linear, and the matrix that controls which directions of v \bm{v} v are learned first is
M : = ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t . \begin{equation}
\bm{M} := \int_0^{\infty} e^{- 2\gamma t} e^{\bm{W} t} (\bm{u} \bm{u}^T + \bm{S}) e^{\bm{W}^T t} \ dt \ . \tag*{(50)}
\end{equation} M := ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t . (50)
The fixed point of these dynamics is
v ∗ = ( ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t ) − 1 ( ∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t ) u . \begin{equation}
\boxed{ \ \bm{v}_* = \left( \int_0^{\infty} e^{- 2\gamma t} e^{\bm{W} t} (\bm{u} \bm{u}^T + \bm{S}) e^{\bm{W}^T t} \ dt \right)^{-1} \left( \int_0^{\infty} e^{- (c_* + \gamma) t} e^{\bm{W} t} \ dt \right) \bm{u} \ } \ . \tag*{(51)}
\end{equation} v ∗ = ( ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t ) − 1 ( ∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t ) u . (51)
Assume for simplicity that S \bm{S} S is diagonal and W = ( γ − c ∗ ) I \bm{W} = (\gamma - c_*) \bm{I} W = ( γ − c ∗ ) I , i.e., that all state-space directions are compatible with the target filter. In this case,
∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t = 1 2 c ∗ ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t = 1 2 c ∗ ( u u T + S ) \begin{equation}
\begin{split}
\int_0^{\infty} e^{- (c_* + \gamma) t} e^{\bm{W} t} \ dt &= \frac{1}{2 c_*} \\
\int_0^{\infty} e^{- 2\gamma t} e^{\bm{W} t} (\bm{u} \bm{u}^T + \bm{S}) e^{\bm{W}^T t} \ dt &= \frac{1}{2 c_*} (\bm{u} \bm{u}^T + \bm{S})
\end{split} \tag*{(52)}
\end{equation} ∫ 0 ∞ e − ( c ∗ + γ ) t e W t d t ∫ 0 ∞ e − 2 γ t e W t ( u u T + S ) e W T t d t = 2 c ∗ 1 = 2 c ∗ 1 ( u u T + S ) (52)
so the optimal readout is
v ∗ = ( u u T + S ) − 1 u = ( S − 1 − S − 1 u u T S − 1 1 + u T S − 1 u ) u = S − 1 u 1 + u T S − 1 u \begin{equation}
\boxed{ \ \bm{v}_* = (\bm{u} \bm{u}^T + \bm{S})^{-1} \bm{u} = \left( \bm{S}^{-1} - \frac{\bm{S}^{-1} \bm{u} \bm{u}^T \bm{S}^{-1}}{1 + \bm{u}^T \bm{S}^{-1} \bm{u}} \right) \bm{u} = \frac{\bm{S}^{-1} \bm{u}}{1 + \bm{u}^T \bm{S}^{-1} \bm{u}} \ } \tag*{(53)}
\end{equation} v ∗ = ( u u T + S ) − 1 u = ( S − 1 − 1 + u T S − 1 u S − 1 u u T S − 1 ) u = 1 + u T S − 1 u S − 1 u (53)
For a large read-in vector u \bm{u} u , this approximately equals
v ∗ ≈ S − 1 u u T S − 1 u \begin{equation}
\bm{v}_* \approx \frac{\bm{S}^{-1} \bm{u}}{\bm{u}^T \bm{S}^{-1} \bm{u}} \tag*{(54)}
\end{equation} v ∗ ≈ u T S − 1 u S − 1 u (54)
which weighs each input direction by how noisy it is. This prescription also yields the correct optimal filter, since
u T v ∗ ≈ u T S − 1 u u T S − 1 u = 1 , \begin{equation}
\bm{u}^T \bm{v}_* \approx \frac{\bm{u}^T \bm{S}^{-1} \bm{u}}{\bm{u}^T \bm{S}^{-1} \bm{u}} = 1 \ , \tag*{(55)}
\end{equation} u T v ∗ ≈ u T S − 1 u u T S − 1 u = 1 , (55)
which implies
f ( t ) = v ∗ T e A t u = e − c ∗ t v ∗ T u ≈ e − c ∗ t . \begin{equation}
f(t) = \bm{v}^T_* e^{\bm{A} t} \bm{u} = e^{- c_* t} \bm{v}_*^T \bm{u} \approx e^{- c_* t} \ . \tag*{(56)}
\end{equation} f ( t ) = v ∗ T e A t u = e − c ∗ t v ∗ T u ≈ e − c ∗ t . (56)
This approximate solution is intuitive since it is precisely the choice of v \bm{v} v that, for this choice of W \bm{W} W and u \bm{u} u , minimizes the quadratic noise term subject to the constraint that u T v = 1 \bm{u}^T \bm{v} = 1 u T v = 1 .
Alignment dynamics
In this appendix, we solve the dynamics associated with the alignment phase (see § 4 ). Using the small-parameter gradient descent updates from Appendix § 9 , we have
d d τ u ≈ v c ∗ + γ − κ u d d τ v ≈ u c ∗ + γ − S v 2 γ − κ v d d τ W ≈ v u T ( c ∗ + γ ) 2 − v v T S 4 γ 2 . \begin{equation}
\frac{d}{d\tau} \bm{u} \approx \frac{\bm{v} }{c_* + \gamma} - \kappa \bm{u} \hspace{0.28in} \frac{d}{d\tau} \bm{v} \approx \frac{\bm{u}}{c_* + \gamma} - \frac{\bm{S} \bm{v}}{2 \gamma} - \kappa \bm{v} \hspace{0.28in} \frac{d}{d\tau} \bm{W} \approx \frac{\bm{v} \bm{u}^T }{(c_* + \gamma)^2} - \frac{ \bm{v} \bm{v}^T \bm{S} }{4 \gamma^2} \ . \tag*{(57)}
\end{equation} d τ d u ≈ c ∗ + γ v − κ u d τ d v ≈ c ∗ + γ u − 2 γ S v − κ v d τ d W ≈ ( c ∗ + γ ) 2 v u T − 4 γ 2 v v T S . (57)
Because u \bm{u} u and v \bm{v} v dynamics form a closed system, we will begin by studying u \bm{u} u and v \bm{v} v only. Once we determine how u \bm{u} u and v \bm{v} v change with time, we will use them to determine how W \bm{W} W changes.
Solving read-in and readout dynamics
Naively, u \bm{u} u and v \bm{v} v together determine a 2 N 2N 2 N -dimensional system of ODEs, since
d d τ u = − κ u + v c ∗ + γ d d τ v = u c ∗ + γ − ( κ + S 2 γ ) v . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{u} &= - \kappa \bm{u} + \frac{\bm{v} }{c_* + \gamma} \\
\frac{d}{d\tau} \bm{v} &= \frac{\bm{u}}{c_* + \gamma} - \left( \kappa + \frac{\bm{S}}{2 \gamma} \right) \bm{v} \ .
\end{split} \tag*{(58)}
\end{equation} d τ d u d τ d v = − κ u + c ∗ + γ v = c ∗ + γ u − ( κ + 2 γ S ) v . (58)
Note that, to ease notation, we will write equalities instead of ≈ \approx ≈ .
This system substantially simplifies if we work in a basis of eigenvectors of S \bm{S} S . Along the i i i -th such eigenvector, if s i > 0 s_i > 0 s i > 0 denotes the corresponding eigenvalue,
d d τ u i = − κ u i + v i c ∗ + γ d d τ v i = u i c ∗ + γ − ( κ + s i 2 γ ) v i . \begin{equation}
\begin{split}
\frac{d}{d\tau} u_i &= - \kappa u_i + \frac{v_i}{c_* + \gamma} \\
\frac{d}{d\tau} v_i &= \frac{u_i}{c_* + \gamma} - \left( \kappa + \frac{s_i}{2 \gamma} \right) v_i \ .
\end{split} \tag*{(59)}
\end{equation} d τ d u i d τ d v i = − κ u i + c ∗ + γ v i = c ∗ + γ u i − ( κ + 2 γ s i ) v i . (59)
Relevant eigenvalues and eigenvectors.
This is just a two-dimensional system governed by the 2 × 2 2 \times 2 2 × 2 matrix
M : = ( − κ 1 c ∗ + γ 1 c ∗ + γ − ( κ + s i 2 γ ) ) \begin{equation}
\bm{M} := \begin{pmatrix}
- \kappa & \frac{1}{c_* + \gamma} \\
\frac{1}{c_* + \gamma} & - \left( \kappa + \frac{s_i}{2 \gamma} \right)
\end{pmatrix} \tag*{(60)}
\end{equation} M := ( − κ c ∗ + γ 1 c ∗ + γ 1 − ( κ + 2 γ s i ) ) (60)
whose characteristic equation is
( λ + κ ) ( λ + κ + s i 2 γ ) − 1 ( c ∗ + γ ) 2 = 0 . \begin{equation}
(\lambda + \kappa) \left( \lambda + \kappa + \frac{s_i}{2 \gamma} \right) - \frac{1}{(c_* + \gamma)^2} = 0 \ . \tag*{(61)}
\end{equation} ( λ + κ ) ( λ + κ + 2 γ s i ) − ( c ∗ + γ ) 2 1 = 0 . (61)
Equivalently,
λ 2 + 2 [ κ + s i 4 γ ] λ + [ κ 2 + κ s i 2 γ − 1 ( c ∗ + γ ) 2 ] = 0 . \begin{equation}
\lambda^2 + 2 \left[ \kappa + \frac{s_i}{4 \gamma} \right] \lambda + \left[ \kappa^2 + \frac{\kappa s_i}{2 \gamma} - \frac{1}{(c_* + \gamma)^2} \right] = 0 \ . \tag*{(62)}
\end{equation} λ 2 + 2 [ κ + 4 γ s i ] λ + [ κ 2 + 2 γ κ s i − ( c ∗ + γ ) 2 1 ] = 0 . (62)
The solutions of this equation (i.e., the eigenvalues of M \bm{M} M ) are
λ i ± = − ( κ + s i 4 γ ) ± ( κ + s i 4 γ ) 2 + 1 ( c ∗ + γ ) 2 − κ 2 − κ s i 2 γ = − ( κ + s i 4 γ ) ± s i 2 16 γ 2 + 1 ( c ∗ + γ ) 2 . \begin{equation}
\begin{split}
\lambda^{\pm}_i &= - \left( \kappa + \frac{s_i}{4 \gamma} \right) \pm \sqrt{ \left( \kappa + \frac{s_i}{4 \gamma} \right)^2 + \frac{1}{(c_* + \gamma)^2} - \kappa^2 - \frac{\kappa s_i}{2 \gamma} } \\
&= - \left( \kappa + \frac{s_i}{4 \gamma} \right) \pm \sqrt{ \frac{s_i^2}{16 \gamma^2} + \frac{1}{(c_* + \gamma)^2} } \ .
\end{split} \tag*{(63)}
\end{equation} λ i ± = − ( κ + 4 γ s i ) ± ( κ + 4 γ s i ) 2 + ( c ∗ + γ ) 2 1 − κ 2 − 2 γ κ s i = − ( κ + 4 γ s i ) ± 16 γ 2 s i 2 + ( c ∗ + γ ) 2 1 . (63)
Note that λ i − \lambda_i^- λ i − is always negative, and so corresponds to decay. The other root may be positive or negative depending on the size of the regularization strength κ ≥ 0 \kappa \geq 0 κ ≥ 0 . Beyond a critical κ \kappa κ value
κ crit : = s min 2 16 γ 2 + 1 ( c ∗ + γ ) 2 − s min 4 γ , \begin{equation}
\kappa_{\text{crit}} := \sqrt{ \frac{s_{\text{min}}^2}{16 \gamma^2} + \frac{1}{(c_* + \gamma)^2} } - \frac{s_{\text{min}}}{4 \gamma} \ , \tag*{(64)}
\end{equation} κ crit := 16 γ 2 s min 2 + ( c ∗ + γ ) 2 1 − 4 γ s min , (64)
where s min s_{\text{min}} s min is the smallest of the s i s_i s i , all modes decay and one obtains only the trivial zero solution for all u i u_i u i and v i v_i v i . For κ < κ crit \kappa < \kappa_{\text{crit}} κ < κ crit , at least some modes survive.
Assume in what follows that κ < κ crit \kappa < \kappa_{\text{crit}} κ < κ crit . Then at least some λ i + \lambda_i^+ λ i + will be positive, which means that u i u_i u i and v i v_i v i along those directions have components which grow with time. In particular, since the eigenvectors of M \bm{M} M are
q ± : = ( 1 ( c ∗ + γ ) ( κ + λ ± ) , ) \begin{equation}
\bm{q}^{\pm} := \begin{pmatrix}
1 \\
(c_* + \gamma) (\kappa + \lambda^{\pm}) \ ,
\end{pmatrix} \tag*{(65)}
\end{equation} q ± := ( 1 ( c ∗ + γ ) ( κ + λ ± ) , ) (65)
the solution z ( τ ) : = ( u i ( τ ) , v i ( τ ) ) T \bm{z}(\tau) := ( u_i(\tau), v_i(\tau) )^T z ( τ ) := ( u i ( τ ) , v i ( τ ) ) T equals
z = e M τ z 0 = e M τ ( c + q + + c − q − ) = c + q + e λ i + τ + c − q − e λ − i τ \begin{equation}
\bm{z} = e^{\bm{M} \tau} \bm{z}_0 = e^{\bm{M} \tau} ( c_+ \bm{q}^+ + c_- \bm{q}^- ) = c_+ \bm{q}^+ e^{\lambda^+_i \tau} + c_- \bm{q}^- e^{\lambda^i_- \tau} \tag*{(66)}
\end{equation} z = e M τ z 0 = e M τ ( c + q + + c − q − ) = c + q + e λ i + τ + c − q − e λ − i τ (66)
for some coefficients c + c_+ c + and c − c_- c − .
Enforcing initial conditions.
We can determine c + c_+ c + and c − c_- c − by choosing them so that z ( 0 ) = ( u i ( 0 ) , v i ( 0 ) ) T \bm{z}(0) = ( u_i(0), v_i(0) )^T z ( 0 ) = ( u i ( 0 ) , v i ( 0 ) ) T . This requires enforcing
u i ( 0 ) = c + + c − v i ( 0 ) = c + ( c ∗ + γ ) ( κ + λ + ) + c − ( c ∗ + γ ) ( κ + λ − ) . \begin{equation}
\begin{split}
u_i(0) &= c_+ + c_- \\
v_i(0) &= c_+ (c_* + \gamma) (\kappa + \lambda^{+}) + c_- (c_* + \gamma)(\kappa + \lambda^{-}) \ .
\end{split} \tag*{(67)}
\end{equation} u i ( 0 ) v i ( 0 ) = c + + c − = c + ( c ∗ + γ ) ( κ + λ + ) + c − ( c ∗ + γ ) ( κ + λ − ) . (67)
The first condition implies c − = u i ( 0 ) − c + c_- = u_i(0) - c_+ c − = u i ( 0 ) − c + . The second implies
v i ( 0 ) c ∗ + γ = c + ( κ + λ + ) + c − ( κ + λ − ) = c + ( κ + λ + ) + [ u i ( 0 ) − c + ] ( κ + λ − ) = u i ( 0 ) ( κ + λ − ) + c + ( λ + − λ − ) \begin{equation}
\begin{split}
\frac{v_i(0)}{c_* + \gamma} &= c_+ (\kappa + \lambda^{+}) + c_-(\kappa + \lambda^{-}) \\
&= c_+ (\kappa + \lambda^{+}) + [ u_i(0) - c_+] (\kappa + \lambda^{-}) \\
&= u_i(0) (\kappa + \lambda^-) + c_+ ( \lambda^+ - \lambda^-)
\end{split} \tag*{(68)}
\end{equation} c ∗ + γ v i ( 0 ) = c + ( κ + λ + ) + c − ( κ + λ − ) = c + ( κ + λ + ) + [ u i ( 0 ) − c + ] ( κ + λ − ) = u i ( 0 ) ( κ + λ − ) + c + ( λ + − λ − ) (68)
so
c + = 1 λ i + − λ i − [ v i ( 0 ) c ∗ + γ − u i ( 0 ) ( κ + λ i − ) ] . \begin{equation}
\boxed{ \ c_+ = \frac{1}{\lambda^+_i - \lambda^-_i} \left[ \frac{v_i(0)}{c_* + \gamma} - u_i(0) (\kappa + \lambda^-_i) \right] \ } \ . \tag*{(69)}
\end{equation} c + = λ i + − λ i − 1 [ c ∗ + γ v i ( 0 ) − u i ( 0 ) ( κ + λ i − ) ] . (69)
Because the mode which grows as opposed to shrinks will dominate after enough time has passed, to good approximation we have
u i ( τ ) ≈ e λ i + τ c + = 1 λ i + − λ i − [ v i ( 0 ) c ∗ + γ − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ v i ( τ ) ≈ e λ i + τ c + ( c ∗ + γ ) ( κ + λ + ) = ( c ∗ + γ ) ( κ + λ + ) λ i + − λ i − [ v i ( 0 ) c ∗ + γ − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ \begin{equation}
\begin{split}
u_i(\tau) &\approx e^{ \lambda^+_i \tau} c_+ = \frac{1}{\lambda^+_i - \lambda^-_i} \left[ \frac{v_i(0)}{c_* + \gamma} - u_i(0) (\kappa + \lambda^-_i) \right] e^{\lambda^+_i \tau} \\
v_i(\tau) &\approx e^{ \lambda^+_i \tau} c_+ (c_* + \gamma) (\kappa + \lambda^+) = \frac{(c_* + \gamma) (\kappa + \lambda^+) }{\lambda^+_i - \lambda^-_i} \left[ \frac{v_i(0)}{c_* + \gamma} - u_i(0) (\kappa + \lambda^-_i) \right] e^{\lambda^+_i \tau}
\end{split} \tag*{(70)}
\end{equation} u i ( τ ) v i ( τ ) ≈ e λ i + τ c + = λ i + − λ i − 1 [ c ∗ + γ v i ( 0 ) − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ ≈ e λ i + τ c + ( c ∗ + γ ) ( κ + λ + ) = λ i + − λ i − ( c ∗ + γ ) ( κ + λ + ) [ c ∗ + γ v i ( 0 ) − u i ( 0 ) ( κ + λ i − ) ] e λ i + τ (70)
for moderate τ \tau τ .
Note that λ i + \lambda^+_i λ i + is a decreasing function of s i s_i s i . This implies that u \bm{u} u and v \bm{v} v will tend to get amplified more quickly along less noisy directions. After enough time has passed, the coefficients along the least noisy direction will tend to dwarf the others, so to good approximation we have
u ( τ ) ≈ u ( τ ) 2 P [ v ( 0 ) c ∗ + γ − u ( 0 ) ( κ + λ min − ) ] v ( τ ) ≈ v ( τ ) 2 P [ v ( 0 ) c ∗ + γ − u ( 0 ) ( κ + λ min − ) ] \begin{equation}
\begin{split}
\bm{u}(\tau) &\approx \frac{u(\tau)}{2} \bm{P} \left[ \frac{\bm{v}(0)}{c_* + \gamma} - \bm{u}(0) (\kappa + \lambda^-_{\text{min}}) \right] \\
\bm{v}(\tau) &\approx \frac{v(\tau)}{2} \bm{P} \left[ \frac{\bm{v}(0)}{c_* + \gamma} - \bm{u}(0) (\kappa + \lambda^-_{\text{min}}) \right]
\end{split} \tag*{(71)}
\end{equation} u ( τ ) v ( τ ) ≈ 2 u ( τ ) P [ c ∗ + γ v ( 0 ) − u ( 0 ) ( κ + λ min − ) ] ≈ 2 v ( τ ) P [ c ∗ + γ v ( 0 ) − u ( 0 ) ( κ + λ min − ) ] (71)
after the alignment phase, where λ min − \lambda^-_{\text{min}} λ min − is the negative eigenvalue associated with the least noisy direction (i.e., the smallest s i s_i s i , which we can call s min s_{\text{min}} s min ). Here, P \bm{P} P is a matrix that projects into the least noisy subspace (which may be of arbitrary dimensionality) and u ( τ ) u(\tau) u ( τ ) and v ( τ ) v(\tau) v ( τ ) are mode amplitudes whose precise form is given above. Equivalently, u ( τ ) \bm{u}(\tau) u ( τ ) and v ( τ ) \bm{v}(\tau) v ( τ ) align in the least noisy state-space direction.
Define the quantity
Δ : = s min 2 16 γ 2 + 1 ( c ∗ + γ ) 2 − 1 c ∗ + γ , \begin{equation}
\Delta := \sqrt{ \frac{s_{\text{min}}^2}{16 \gamma^2} + \frac{1}{(c_* + \gamma)^2} } - \frac{1}{c_* + \gamma} \ , \tag*{(72)}
\end{equation} Δ := 16 γ 2 s min 2 + ( c ∗ + γ ) 2 1 − c ∗ + γ 1 , (72)
which is an increasing function of s min s_{\text{min}} s min . Δ \Delta Δ quantifies the gap between the noise-free solution and the noisy solution. In terms of it, we can write our answer as
u ( τ ) ≈ u ( τ ) P [ [ u ( 0 ) + v ( 0 ) ] 2 + ( c ∗ + γ ) 2 ( s min 4 γ + Δ ( s min ) ) u ( 0 ) ] v ( τ ) ≈ v ( τ ) P [ [ u ( 0 ) + v ( 0 ) ] 2 + ( c ∗ + γ ) 2 ( s min 4 γ + Δ ( s min ) ) u ( 0 ) ] . \begin{equation}
\begin{split}
\bm{u}(\tau) &\approx u(\tau) \bm{P} \left[ \frac{[ \bm{u}(0) + \bm{v}(0) ]}{2} + \frac{(c_* + \gamma)}{2 } \left(\frac{s_{\text{min}}}{4 \gamma} + \Delta(s_{\text{min}}) \right) \bm{u}(0) \right] \\
\bm{v}(\tau) &\approx v(\tau) \bm{P} \left[ \frac{[ \bm{u}(0) + \bm{v}(0) ]}{2} + \frac{(c_* + \gamma)}{2 } \left(\frac{s_{\text{min}}}{4 \gamma} + \Delta(s_{\text{min}}) \right) \bm{u}(0) \right] \ .
\end{split} \tag*{(73)}
\end{equation} u ( τ ) v ( τ ) ≈ u ( τ ) P [ 2 [ u ( 0 ) + v ( 0 )] + 2 ( c ∗ + γ ) ( 4 γ s min + Δ ( s min ) ) u ( 0 ) ] ≈ v ( τ ) P [ 2 [ u ( 0 ) + v ( 0 )] + 2 ( c ∗ + γ ) ( 4 γ s min + Δ ( s min ) ) u ( 0 ) ] . (73)
W dynamics
Given u \bm{u} u and v \bm{v} v dynamics, W \bm{W} W dynamics are extremely simple. Because u \bm{u} u and v \bm{v} v are, to good approximation, collinear, we can write
u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 \begin{equation}
\bm{u}(\tau) \approx u(\tau) \bm{u}_0 \hspace{1in} \bm{v}(\tau) \approx v(\tau) \bm{u}_0 \tag*{(74)}
\end{equation} u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 (74)
for some vector u 0 \bm{u}_0 u 0 (which is an eigenvector of S \bm{S} S with eigenvalue σ 2 \sigma^2 σ 2 ) and scalar amplitudes u ( τ ) u(\tau) u ( τ ) and v ( τ ) v(\tau) v ( τ ) . Eq. (57) then implies that
d d τ W ≈ [ u ( τ ) v ( τ ) ( c ∗ + γ ) 2 − σ 2 v ( τ ) 2 4 γ 2 ] u 0 u 0 T , \begin{equation}
\frac{d}{d\tau} \bm{W} \approx \left[ \frac{u(\tau) v(\tau)}{( c_* + \gamma)^2} - \frac{\sigma^2 v(\tau)^2}{4 \gamma^2} \right] \bm{u}_0 \bm{u}_0^T \ , \tag*{(75)}
\end{equation} d τ d W ≈ [ ( c ∗ + γ ) 2 u ( τ ) v ( τ ) − 4 γ 2 σ 2 v ( τ ) 2 ] u 0 u 0 T , (75)
or equivalently that
W ( τ ) ≈ [ γ − c ( τ ) ] u 0 u 0 T \begin{equation}
\bm{W}(\tau) \approx [ \gamma - c(\tau) ] \bm{u}_0 \bm{u}_0^T \tag*{(76)}
\end{equation} W ( τ ) ≈ [ γ − c ( τ )] u 0 u 0 T (76)
since these updates are all in the same direction. c ( τ ) c(\tau) c ( τ ) is a scalar which determines the overall amplitude of W \bm{W} W . In other words, W \bm{W} W becomes a rank-one matrix aligned in the same direction as u \bm{u} u and v \bm{v} v .
Derivation of low-dimensional effective dynamics
In this appendix, we derive the effective learning dynamics associated with the post-alignment phase. Recall from Appendix § 11 and § 4 that the outcome of alignment has
u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 W ( τ ) ≈ [ γ − c ( τ ) ] u 0 u 0 T \begin{equation}
\boxed{ \ \bm{u}(\tau) \approx u(\tau) \bm{u}_0 \hspace{0.5in} \bm{v}(\tau) \approx v(\tau) \bm{u}_0 \hspace{0.5in} \bm{W}(\tau) \approx [ \gamma - c(\tau) ] \bm{u}_0 \bm{u}_0^T \ } \tag*{(77)}
\end{equation} u ( τ ) ≈ u ( τ ) u 0 v ( τ ) ≈ v ( τ ) u 0 W ( τ ) ≈ [ γ − c ( τ )] u 0 u 0 T (77)
for some (assumed normalized) u 0 \bm{u}_0 u 0 which is an eigenvector of S \bm{S} S with S u 0 = σ 2 u 0 \bm{S} \bm{u}_0 = \sigma^2 \bm{u}_0 S u 0 = σ 2 u 0 . As we found earlier, this ought to be the least noisy state-space direction.
We will call this the alignment ansatz. We want to show that dynamics are closed under this ansatz, in the sense that it suffices to track the time evolution of the scalars u u u , v v v , and c c c . We will also show that gradient descent dynamics on the full objective become equivalent to gradient descent dynamics on a simpler effective objective.
Note that, under the alignment ansatz,
f ( t ) = v T e A t u = u ( τ ) v ( τ ) e − c ( τ ) t . \begin{equation}
f(t) = \bm{v}^T e^{\bm{A} t} \bm{u} = u(\tau) v(\tau) e^{- c(\tau) t} \ . \tag*{(78)}
\end{equation} f ( t ) = v T e A t u = u ( τ ) v ( τ ) e − c ( τ ) t . (78)
Dynamics are closed under the alignment ansatz
u \bm{u} u dynamics.
General u \bm{u} u dynamics have the form
d d τ u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W T t v d t − κ u . \begin{equation}
\frac{d}{d\tau} \bm{u} = \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W}^T t} \bm{v} \ dt - \kappa \bm{u} \ . \tag*{(79)}
\end{equation} d τ d u = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W T t v d t − κ u . (79)
Under our ansatz, this simplifies to
d d τ u ( τ ) u 0 = [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − c ( τ ) t v ( τ ) d t − κ u ( τ ) ] u 0 = [ ∫ 0 ∞ [ e − c ∗ t − u ( τ ) v ( τ ) e − c ( τ ) t ] e − c ( τ ) t v ( τ ) d t − κ u ( τ ) ] u 0 = [ v ( τ ) c ∗ + c ( τ ) − u ( τ ) v ( τ ) 2 2 c ( τ ) − κ u ( τ ) ] u 0 . \begin{equation}
\begin{split}
\frac{d}{d\tau} u(\tau) \bm{u}_0 &= \left[ \int_0^{\infty} [ f_*(t) - f(t) ] e^{- c(\tau) t} v(\tau) \ dt - \kappa u(\tau) \right] \bm{u}_0 \\
&= \left[ \int_0^{\infty} [ e^{- c_* t} - u(\tau) v(\tau) e^{- c(\tau) t} ] e^{- c(\tau) t} v(\tau) \ dt - \kappa u(\tau) \right] \bm{u}_0 \\
&= \left[ \frac{v(\tau)}{c_* + c(\tau)} - \frac{u(\tau) v(\tau)^2}{2 c(\tau)} - \kappa u(\tau) \right] \bm{u}_0 \ .
\end{split} \tag*{(80)}
\end{equation} d τ d u ( τ ) u 0 = [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − c ( τ ) t v ( τ ) d t − κ u ( τ ) ] u 0 = [ ∫ 0 ∞ [ e − c ∗ t − u ( τ ) v ( τ ) e − c ( τ ) t ] e − c ( τ ) t v ( τ ) d t − κ u ( τ ) ] u 0 = [ c ∗ + c ( τ ) v ( τ ) − 2 c ( τ ) u ( τ ) v ( τ ) 2 − κ u ( τ ) ] u 0 . (80)
We indeed find that only the scalar u ( τ ) u(\tau) u ( τ ) changes, as desired.
v \bm{v} v dynamics.
General v \bm{v} v dynamics have the form
d d τ v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v . \begin{equation}
\frac{d}{d\tau} \bm{v} = \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} e^{\bm{W} t} \bm{u} - e^{- 2 \gamma t} e^{ \bm{W} t} \bm{S} e^{ \bm{W}^T t} \bm{v} \ dt - \kappa \bm{v} \ . \tag*{(81)}
\end{equation} d τ d v = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t e W t u − e − 2 γ t e W t S e W T t v d t − κ v . (81)
Under the ansatz,
d d τ v ( τ ) u 0 = [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] u ( τ ) e − c ( τ ) t − σ 2 v ( τ ) e − 2 c ( τ ) t d t − κ v ( τ ) ] u 0 = [ ∫ 0 ∞ [ e − c ∗ t − u ( τ ) v ( τ ) e − c ( τ ) t ] u ( τ ) e − c ( τ ) t − σ 2 v ( τ ) e − 2 c ( τ ) t d t − κ v ( τ ) ] u 0 = [ u ( τ ) c ∗ + c ( τ ) − [ u ( τ ) 2 + σ 2 ] v ( τ ) 2 c ( τ ) − κ v ( τ ) ] u 0 \begin{equation}
\begin{split}
\frac{d}{d\tau} v(\tau) \bm{u}_0 &= \left[ \int_0^{\infty} [ f_*(t) - f(t) ] u(\tau) e^{- c(\tau) t} - \sigma^2 v(\tau) e^{- 2 c(\tau) t} \ dt - \kappa v(\tau) \right] \bm{u}_0 \\
&= \left[ \int_0^{\infty} [ e^{- c_* t} - u(\tau) v(\tau) e^{- c(\tau) t} ] u(\tau) e^{- c(\tau) t} - \sigma^2 v(\tau) e^{- 2 c(\tau) t} \ dt - \kappa v(\tau) \right] \bm{u}_0 \\
&= \left[ \frac{u(\tau)}{c_* + c(\tau)} - \frac{ [ u(\tau)^2 + \sigma^2 ] v(\tau) }{2 c(\tau)} - \kappa v(\tau) \right] \bm{u}_0
\end{split} \tag*{(82)}
\end{equation} d τ d v ( τ ) u 0 = [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] u ( τ ) e − c ( τ ) t − σ 2 v ( τ ) e − 2 c ( τ ) t d t − κ v ( τ ) ] u 0 = [ ∫ 0 ∞ [ e − c ∗ t − u ( τ ) v ( τ ) e − c ( τ ) t ] u ( τ ) e − c ( τ ) t − σ 2 v ( τ ) e − 2 c ( τ ) t d t − κ v ( τ ) ] u 0 = [ c ∗ + c ( τ ) u ( τ ) − 2 c ( τ ) [ u ( τ ) 2 + σ 2 ] v ( τ ) − κ v ( τ ) ] u 0 (82)
where σ 2 : = u 0 T S u 0 \sigma^2 := \bm{u}_0^T \bm{S} \bm{u}_0 σ 2 := u 0 T S u 0 is the noise level in the alignment direction.
W \bm{W} W dynamics.
General W \bm{W} W dynamics have the form
d d τ W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t . \begin{equation}
\begin{split}
\frac{d}{d\tau} \bm{W} &= \int_0^{\infty} [ f_*(t) - f(t) ] e^{- \gamma t} \int_0^t e^{\bm{W}^T t'} \bm{v} \bm{u}^T e^{\bm{W}^T (t - t')} \ dt' dt\\
&- \int_0^{\infty} e^{-2\gamma t} \int_0^t \ e^{\bm{W}^T t'} \bm{v} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T (t - t')} \ dt' dt \ .
\end{split} \tag*{(83)}
\end{equation} d τ d W = ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] e − γ t ∫ 0 t e W T t ′ v u T e W T ( t − t ′ ) d t ′ d t − ∫ 0 ∞ e − 2 γ t ∫ 0 t e W T t ′ v v T e W t S e W T ( t − t ′ ) d t ′ d t . (83)
Under the ansatz,
− d d τ c ( τ ) u 0 u 0 T = u ( τ ) v ( τ ) [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t ) ] t e − c ( τ ) t d t ] u 0 u 0 T − σ 2 v ( τ ) 2 ∫ 0 ∞ t e − 2 c ( τ ) t d t u 0 u 0 T . \begin{equation}
\begin{split}
- \frac{d}{d\tau} c(\tau) \bm{u}_0 \bm{u}_0^T &= u(\tau) v(\tau) \left[ \int_0^{\infty} [ f_*(t) - f(t) ] \ t e^{- c(\tau) t} dt \right] \bm{u}_0 \bm{u}_0^T \\
&- \sigma^2 v(\tau)^2 \int_0^{\infty} t e^{-2 c(\tau) t} dt \bm{u}_0 \bm{u}_0^T \ .
\end{split} \tag*{(84)}
\end{equation} − d τ d c ( τ ) u 0 u 0 T = u ( τ ) v ( τ ) [ ∫ 0 ∞ [ f ∗ ( t ) − f ( t )] t e − c ( τ ) t d t ] u 0 u 0 T − σ 2 v ( τ ) 2 ∫ 0 ∞ t e − 2 c ( τ ) t d t u 0 u 0 T . (84)
Simplifying further,
− d d τ c ( τ ) u 0 u 0 T = u ( τ ) v ( τ ) [ 1 [ c ∗ + c ( τ ) ] 2 − u ( τ ) v ( τ ) 4 c ( τ ) 2 ] u 0 u 0 T − σ 2 v ( τ ) 2 4 c ( τ ) 2 u 0 u 0 T , \begin{equation}
\begin{split}
- \frac{d}{d\tau} c(\tau) \bm{u}_0 \bm{u}_0^T &= u(\tau) v(\tau) \left[ \frac{1}{[ c_* + c(\tau) ]^2} - \frac{u(\tau) v(\tau)}{4 c(\tau)^2} \right] \bm{u}_0 \bm{u}_0^T \\
&- \frac{\sigma^2 v(\tau)^2 }{4 c(\tau)^2} \bm{u}_0 \bm{u}_0^T \ ,
\end{split} \tag*{(85)}
\end{equation} − d τ d c ( τ ) u 0 u 0 T = u ( τ ) v ( τ ) [ [ c ∗ + c ( τ ) ] 2 1 − 4 c ( τ ) 2 u ( τ ) v ( τ ) ] u 0 u 0 T − 4 c ( τ ) 2 σ 2 v ( τ ) 2 u 0 u 0 T , (85)
or
d d τ c ( τ ) u 0 u 0 T = ( v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] 4 c ( τ ) 2 − u ( τ ) v ( τ ) [ c ∗ + c ( τ ) ] 2 ) u 0 u 0 T . \begin{equation}
\frac{d}{d\tau} c(\tau) \bm{u}_0 \bm{u}_0^T = \left( \frac{ v(\tau)^2 [ u(\tau)^2 + \sigma^2 ]}{4 c(\tau)^2} - \frac{u(\tau) v(\tau)}{[ c_* + c(\tau) ]^2} \right) \bm{u}_0 \bm{u}_0^T \ . \tag*{(86)}
\end{equation} d τ d c ( τ ) u 0 u 0 T = ( 4 c ( τ ) 2 v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] − [ c ∗ + c ( τ ) ] 2 u ( τ ) v ( τ ) ) u 0 u 0 T . (86)
In all cases, we indeed discover that the dynamics are closed under this ansatz. In summary, because the direction u 0 \bm{u}_0 u 0 never changes under gradient descent dynamics, it suffices to study the reduced system
d d τ u ( τ ) = v ( τ ) c ∗ + c ( τ ) − u ( τ ) v ( τ ) 2 2 c ( τ ) − κ u ( τ ) d d τ v ( τ ) = u ( τ ) c ∗ + c ( τ ) − v ( τ ) [ σ 2 + u ( τ ) 2 ] 2 c ( τ ) − κ v ( τ ) d d τ c ( τ ) = v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] 4 c ( τ ) 2 − u ( τ ) v ( τ ) [ c ∗ + c ( τ ) ] 2 . \begin{equation}
\begin{split}
\frac{d}{d\tau} u(\tau) &= \frac{v(\tau)}{c_* + c(\tau)} - \frac{u(\tau) v(\tau)^2}{2 c(\tau)} - \kappa u(\tau) \\
\frac{d}{d\tau} v(\tau) &= \frac{u(\tau)}{c_* + c(\tau)} - \frac{v(\tau) [ \sigma^2 + u(\tau)^2 ]}{2 c(\tau)} - \kappa v(\tau) \\
\frac{d}{d\tau} c(\tau) &= \frac{v(\tau)^2 [ u(\tau)^2 + \sigma^2 ]}{4 c(\tau)^2} - \frac{u(\tau) v(\tau)}{[c_* + c(\tau) ]^2} \ .
\end{split} \tag*{(87)}
\end{equation} d τ d u ( τ ) d τ d v ( τ ) d τ d c ( τ ) = c ∗ + c ( τ ) v ( τ ) − 2 c ( τ ) u ( τ ) v ( τ ) 2 − κ u ( τ ) = c ∗ + c ( τ ) u ( τ ) − 2 c ( τ ) v ( τ ) [ σ 2 + u ( τ ) 2 ] − κ v ( τ ) = 4 c ( τ ) 2 v ( τ ) 2 [ u ( τ ) 2 + σ 2 ] − [ c ∗ + c ( τ ) ] 2 u ( τ ) v ( τ ) . (87)
Reduced dynamics are gradient descent on an effective objective
We can make an additional theoretically interesting point about this ansatz: not only is it dynamically closed, but the above ODEs are equivalent to gradient descent on an effective objective function. Fortunately, this objective is exactly what one obtains by naively substituting the ansatz into the original objective.
Recall that the full objective equals
J ( u , v , W ) = ∫ 0 ∞ 1 2 [ f ( t ) − f ∗ ( t ) ] 2 + e − 2 γ t 2 v T e W t S e W T t v d t + κ 2 ∥ u ∥ 2 + κ 2 ∥ v ∥ 2 . \begin{equation}
J(\bm{u}, \bm{v}, \bm{W}) = \int_0^{\infty} \frac{1}{2} \left[ f(t) - f_*(t) \right]^2 + \frac{e^{- 2 \gamma t}}{2} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t} \bm{v} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ . \tag*{(88)}
\end{equation} J ( u , v , W ) = ∫ 0 ∞ 2 1 [ f ( t ) − f ∗ ( t ) ] 2 + 2 e − 2 γ t v T e W t S e W T t v d t + 2 κ ∥ u ∥ 2 + 2 κ ∥ v ∥ 2 . (88)
Substituting our ansatz, it becomes
1 2 ∫ 0 ∞ { [ u v e − c t − e − c ∗ t ] 2 + σ 2 v 2 e − 2 c t } d t + κ 2 ( u 2 + v 2 ) = 1 2 ∫ 0 ∞ { u 2 v 2 e − 2 c t − 2 u v e − ( c ∗ + c ) t + e − 2 c ∗ t + σ 2 v 2 e − 2 c t } d t + κ 2 ( u 2 + v 2 ) = 1 2 { u 2 v 2 1 2 c − 2 u v 1 c ∗ + c + 1 2 c ∗ + σ 2 v 2 1 2 c } + κ 2 ( u 2 + v 2 ) . \begin{equation}
\begin{split}
& \ \frac{1}{2} \int_0^{\infty} \left\{ \ \left[ u v e^{- c t} - e^{-c_* t} \right]^2 + \sigma^2 v^2 e^{- 2 c t} \ \right\} \ dt + \frac{\kappa}{2} (u^2 + v^2) \\
=& \ \frac{1}{2} \int_0^{\infty} \left\{ \ u^2 v^2 e^{- 2 c t} - 2 u v e^{- ( c_* + c ) t} + e^{- 2 c_* t} + \sigma^2 v^2 e^{- 2 c t} \ \right\} \ dt + \frac{\kappa}{2} (u^2 + v^2) \\
=& \ \frac{1}{2} \left\{ \ u^2 v^2 \frac{1}{2 c} - 2 u v \frac{1}{ c_* + c} + \frac{1}{2 c_*} + \sigma^2 v^2 \frac{1}{2 c} \ \right\} + \frac{\kappa}{2} (u^2 + v^2) \ .
\end{split} \tag*{(89)}
\end{equation} = = 2 1 ∫ 0 ∞ { [ uv e − c t − e − c ∗ t ] 2 + σ 2 v 2 e − 2 c t } d t + 2 κ ( u 2 + v 2 ) 2 1 ∫ 0 ∞ { u 2 v 2 e − 2 c t − 2 uv e − ( c ∗ + c ) t + e − 2 c ∗ t + σ 2 v 2 e − 2 c t } d t + 2 κ ( u 2 + v 2 ) 2 1 { u 2 v 2 2 c 1 − 2 uv c ∗ + c 1 + 2 c ∗ 1 + σ 2 v 2 2 c 1 } + 2 κ ( u 2 + v 2 ) . (89)
Simplifying, we obtain
J = u ( τ ) 2 v ( τ ) 2 4 c ( τ ) − u ( τ ) v ( τ ) c ∗ + c ( τ ) + 1 4 c ∗ + σ 2 v ( τ ) 2 4 c ( τ ) + κ 2 ( u 2 + v 2 ) = 1 4 c ( τ ) [ u ( τ ) v ( τ ) − 2 c ( τ ) c ∗ + c ( τ ) ] 2 + ( 1 4 c ∗ − c ( τ ) [ c ∗ + c ( τ ) ] 2 ) + σ 2 v ( τ ) 2 4 c ( τ ) + κ 2 [ u ( τ ) 2 + v ( τ ) 2 ] . \begin{align}
J &= \frac{u(\tau)^2 v(\tau)^2}{4 c(\tau)} - \frac{u(\tau) v(\tau)}{ c_* + c(\tau) } + \frac{1}{4 c_*} + \frac{\sigma^2 v(\tau)^2}{4 c(\tau)} + \frac{\kappa}{2} (u^2 + v^2) \tag*{(90)} \\
&= \frac{1}{4 c(\tau)} \left[ u(\tau) v(\tau) - \frac{2 c(\tau)}{c_* + c(\tau)} \right]^2 + \left( \frac{1}{4 c_*} - \frac{c(\tau)}{[ c_* + c(\tau) ]^2} \right) + \frac{\sigma^2 v(\tau)^2}{4 c(\tau)} + \frac{\kappa}{2} [ u(\tau)^2 + v(\tau)^2 ] \ . \notag
\end{align} J = 4 c ( τ ) u ( τ ) 2 v ( τ ) 2 − c ∗ + c ( τ ) u ( τ ) v ( τ ) + 4 c ∗ 1 + 4 c ( τ ) σ 2 v ( τ ) 2 + 2 κ ( u 2 + v 2 ) = 4 c ( τ ) 1 [ u ( τ ) v ( τ ) − c ∗ + c ( τ ) 2 c ( τ ) ] 2 + ( 4 c ∗ 1 − [ c ∗ + c ( τ ) ] 2 c ( τ ) ) + 4 c ( τ ) σ 2 v ( τ ) 2 + 2 κ [ u ( τ ) 2 + v ( τ ) 2 ] . (90)
It is straightforward to verify that the above ODEs directly correspond to gradient descent on this objective, which establishes our claim.
Characterization of amplitude-collapse phase transition
In this appendix, we characterize the noise-induced phase transition associated with amplitude collapse (see § 6 ). The objective J J J has the form
J = 1 4 c ( u v − 2 c c ∗ + c ) 2 + ( 1 4 c ∗ − c ( c ∗ + c ) 2 ) + σ 2 v 2 4 c + κ 2 ( u 2 + v 2 ) . \begin{equation}
J = \frac{1}{4 c} \left( u v - \frac{2 c}{c_* + c} \right)^2 + \left( \frac{1}{4 c_*} - \frac{c}{( c_* + c )^2} \right) + \frac{\sigma^2 v^2}{4 c} + \frac{\kappa}{2} ( u^2 + v^2 ) \ . \tag*{(91)}
\end{equation} J = 4 c 1 ( uv − c ∗ + c 2 c ) 2 + ( 4 c ∗ 1 − ( c ∗ + c ) 2 c ) + 4 c σ 2 v 2 + 2 κ ( u 2 + v 2 ) . (91)
This amounts to determining the conditions under which J J J has a nontrivial optimum.
Minimizing J J J with respect to its arguments
Let us begin by trying to optimize J J J with respect to its arguments—although we will find it useful to work in terms of slightly different variables. Define the filter amplitude N : = u v N := u v N := uv and the scale ratio r : = u / v r := u/v r := u / v . Since v 2 = N / r v^2 = N/r v 2 = N / r and u 2 = N r u^2 = N r u 2 = N r , we can write J J J as
J ( r , N , c ) = 1 4 c ( N − 2 c c ∗ + c ) 2 + ( 1 4 c ∗ − c ( c ∗ + c ) 2 ) + σ 2 N 4 c r + κ 2 N ( 1 r + r ) . \begin{equation}
J(r, N, c) = \frac{1}{4 c} \left( N - \frac{2 c}{c_* + c} \right)^2 + \left( \frac{1}{4 c_*} - \frac{c}{( c_* + c )^2} \right) + \frac{\sigma^2 N}{4 c r} + \frac{\kappa}{2} N \left( \frac{1}{r} + r \right) \ . \tag*{(92)}
\end{equation} J ( r , N , c ) = 4 c 1 ( N − c ∗ + c 2 c ) 2 + ( 4 c ∗ 1 − ( c ∗ + c ) 2 c ) + 4 cr σ 2 N + 2 κ N ( r 1 + r ) . (92)
Minimizing with respect to r r r .
The derivative of J J J with respect to r r r equals
∂ J ∂ r = κ 2 N − κ 2 N ( 1 + σ 2 2 κ c ) 1 r 2 = 0 \begin{equation}
\frac{\partial J}{\partial r} = \frac{\kappa}{2} N - \frac{\kappa}{2} N \left(1 + \frac{\sigma^2}{2 \kappa c} \right) \frac{1}{r^2} = 0 \tag*{(93)}
\end{equation} ∂ r ∂ J = 2 κ N − 2 κ N ( 1 + 2 κ c σ 2 ) r 2 1 = 0 (93)
which implies that
r 2 = 1 + σ 2 2 κ c \begin{equation}
r^2 = 1 + \frac{\sigma^2}{2 \kappa c} \tag*{(94)}
\end{equation} r 2 = 1 + 2 κ c σ 2 (94)
at the minimum. Note that it depends on c c c but not on N N N . In a slight abuse of notation, below we will use r r r to denote the value of the scale ratio at the minimum. Without loss of generality, choose the positive solution so that
r = 1 + σ 2 2 κ c . \begin{equation}
\boxed{ \ r = \sqrt{ 1 + \frac{\sigma^2}{2 \kappa c} } \ } \ . \tag*{(95)}
\end{equation} r = 1 + 2 κ c σ 2 . (95)
Since r r r is defined as the ratio of u u u and v v v , this immediately implies that u u u and v v v have the same sign at a nontrivial minimum. Relatedly, it must be true that N ≥ 0 N \geq 0 N ≥ 0 , since N = u 2 / r ≥ 0 N = u^2 / r \geq 0 N = u 2 / r ≥ 0 .
Minimizing with respect to N N N .
The derivative of J J J with respect to N N N equals
∂ J ∂ N = 1 2 c ( N − 2 c c ∗ + c ) + κ 2 ( 1 + σ 2 2 κ c ) 1 r + κ 2 r = 1 2 c ( N − 2 c c ∗ + c ) + κ r . \begin{equation}
\begin{split}
\frac{\partial J}{\partial N} &= \frac{1}{2 c} \left( N - \frac{2 c}{c_* + c} \right) + \frac{\kappa}{2} \left( 1 + \frac{\sigma^2}{2 \kappa c} \right) \frac{1}{r} + \frac{\kappa}{2} r \\
&= \frac{1}{2 c} \left( N - \frac{2 c}{c_* + c} \right) + \kappa r \ .
\end{split} \tag*{(96)}
\end{equation} ∂ N ∂ J = 2 c 1 ( N − c ∗ + c 2 c ) + 2 κ ( 1 + 2 κ c σ 2 ) r 1 + 2 κ r = 2 c 1 ( N − c ∗ + c 2 c ) + κ r . (96)
Setting the left-hand side equal to zero, it appears true that
N = 2 c c ∗ + c − 2 κ r c \begin{equation}
N = \frac{2 c}{c_* + c} - 2 \kappa r c \tag*{(97)}
\end{equation} N = c ∗ + c 2 c − 2 κ r c (97)
at the minimum. But since our r r r minimization implied that N ≥ 0 N \geq 0 N ≥ 0 , we actually have
N = 2 c [ 1 c ∗ + c − κ r ] + \begin{equation}
N = 2 c \left[ \frac{1}{c_* + c} - \kappa r \right]_+ \tag*{(98)}
\end{equation} N = 2 c [ c ∗ + c 1 − κ r ] + (98)
where [ ⋯ ] + [ \cdots ]_+ [ ⋯ ] + denotes rectification (i.e., a ReLU). That is, if the bracketed quantity is negative or zero, J J J is minimized when N = 0 N = 0 N = 0 . This is a sign of amplitude collapse—if N = 0 N = 0 N = 0 is optimal, it is optimal to take f ( t ) = 0 f(t) = 0 f ( t ) = 0 .
If we define the quantity
h ( c ) : = 1 c ∗ + c − κ r , \begin{equation}
\boxed{ \ h(c) := \frac{1}{c_* + c} - \kappa r \ } \ , \tag*{(99)}
\end{equation} h ( c ) := c ∗ + c 1 − κ r , (99)
we can write the optimal N N N as
N = 2 c [ h ( c ) ] + . \begin{equation}
\boxed{ \ N = 2 c [ h(c) ]_+ \ } \ . \tag*{(100)}
\end{equation} N = 2 c [ h ( c ) ] + . (100)
Rewriting the objective in terms of N N N .
In terms of N N N , the objective can be written at its minimum as
J ( c ) = c ( 1 c ∗ + c − [ h ( c ) ] + ) 2 + 1 4 c ∗ − c ( c ∗ + c ) 2 + 2 κ c [ h ( c ) ] + r ( c ) = 1 4 c ∗ + c [ h ( c ) ] + ( − 2 c ∗ + c + [ h ( c ) ] + ) + 2 κ c [ h ( c ) ] + r ( c ) = 1 4 c ∗ + c [ h ( c ) ] + ( 2 κ r ( c ) − 2 c ∗ + c + [ h ( c ) ] + ) . \begin{equation}
\begin{split}
J(c) &= c \left( \frac{1}{c_* + c} - [ h(c) ]_+ \right)^2 + \frac{1}{4 c_*} - \frac{c}{( c_* + c )^2} + 2 \kappa c [h(c)]_+ r(c) \\
&= \frac{1}{4 c_*} + c [h(c)]_+ \left( - \frac{2}{c_* + c} + [ h(c) ]_+ \right) + 2 \kappa c [h(c)]_+ r(c) \\
&= \frac{1}{4 c_*} + c [h(c)]_+ \left( 2 \kappa r(c) - \frac{2}{c_* + c} + [ h(c) ]_+ \right) \ .
\end{split} \tag*{(101)}
\end{equation} J ( c ) = c ( c ∗ + c 1 − [ h ( c ) ] + ) 2 + 4 c ∗ 1 − ( c ∗ + c ) 2 c + 2 κ c [ h ( c ) ] + r ( c ) = 4 c ∗ 1 + c [ h ( c ) ] + ( − c ∗ + c 2 + [ h ( c ) ] + ) + 2 κ c [ h ( c ) ] + r ( c ) = 4 c ∗ 1 + c [ h ( c ) ] + ( 2 κ r ( c ) − c ∗ + c 2 + [ h ( c ) ] + ) . (101)
Either h ( c ) ≥ 0 h(c) \geq 0 h ( c ) ≥ 0 , in which case the bracketed quantity equals − h ( c ) - h(c) − h ( c ) , or h ( c ) < 0 h(c) < 0 h ( c ) < 0 , in which case the [ h ( c ) ] + [ h(c) ]_+ [ h ( c ) ] + prefactor makes the entire quantity zero. We conclude that
J ( c ) = 1 4 c ∗ − c [ h ( c ) ] + 2 \begin{equation}
\boxed{ \ J(c) = \frac{1}{4 c_*} - c [h(c)]_+^2 } \tag*{(102)}
\end{equation} J ( c ) = 4 c ∗ 1 − c [ h ( c ) ] + 2 (102)
at the minimum. This immediately tells us that if h < 0 h < 0 h < 0 for all c ≥ 0 c \geq 0 c ≥ 0 , there is a trivial solution with N = 0 N = 0 N = 0 and J J J independent of c c c (see Figure 5 c).
Determining the phase boundary
While it appears prohibitively difficult to derive the exact c c c that minimizes J J J (since this requires solving a high-order polynomial equation), we can instead determine where there is a transition from a nontrivial solution to a trivial one.
Phase boundary conditions.
Eq. (102) implies that a nontrivial solution exists if and only if h ( c ) > 0 h(c) > 0 h ( c ) > 0 for some c > 0 c > 0 c > 0 , since this yields N > 0 N > 0 N > 0 and a nontrivial value c > 0 c > 0 c > 0 at the minimum. On the other hand, if h ( c ) ≤ 0 h(c) \leq 0 h ( c ) ≤ 0 for all c > 0 c > 0 c > 0 , then N = 0 N = 0 N = 0 and the trivial solution is optimal. The phase boundary therefore occurs when the maximum of h ( c ) h(c) h ( c ) just touches zero. Hence, the phase boundary is characterized by the two conditions
h ( c ) = 0 h ′ ( c ) = 0 . \begin{equation}
\boxed{ \ h(c) = 0 \hspace{1in} h'(c) = 0 \ } \ . \tag*{(103)}
\end{equation} h ( c ) = 0 h ′ ( c ) = 0 . (103)
Explicitly, we want
1 c ∗ + c = κ 1 + σ 2 2 κ c 1 ( c ∗ + c ) 2 = κ σ 2 2 κ c 2 2 1 + σ 2 2 κ c . \begin{equation}
\begin{split}
\frac{1}{c_* + c} &= \kappa \sqrt{1 + \frac{\sigma^2}{2 \kappa c}} \\
\frac{1}{(c_* + c)^2} &= \kappa \frac{\frac{\sigma^2}{2 \kappa c^2}}{2 \sqrt{1 + \frac{\sigma^2}{2 \kappa c}} } \ .
\end{split} \tag*{(104)}
\end{equation} c ∗ + c 1 ( c ∗ + c ) 2 1 = κ 1 + 2 κ c σ 2 = κ 2 1 + 2 κ c σ 2 2 κ c 2 σ 2 . (104)
Relatedly, a nontrivial solution only exists when c ∗ κ < 1 c_* \kappa < 1 c ∗ κ < 1 , since for c ∗ κ ≥ 1 c_* \kappa \geq 1 c ∗ κ ≥ 1 ,
h ( c ) = 1 c ∗ + c − κ r ≤ 1 c ∗ + c − κ < 1 c ∗ − κ ≤ 0 \begin{equation}
h(c) = \frac{1}{c_* + c} - \kappa r \leq \frac{1}{c_* + c} - \kappa < \frac{1}{c_*} - \kappa \leq 0 \tag*{(105)}
\end{equation} h ( c ) = c ∗ + c 1 − κ r ≤ c ∗ + c 1 − κ < c ∗ 1 − κ ≤ 0 (105)
for all c > 0 c > 0 c > 0 . In this regime, because h ( c ) h(c) h ( c ) is always nonpositive, σ crit 2 = 0 \sigma^2_{\text{crit}} = 0 σ crit 2 = 0 .
Some algebra.
It will be helpful to write σ 2 \sigma^2 σ 2 and κ \kappa κ separately in terms of c c c . Divide the second equation by the first to obtain
1 c ∗ + c = σ 2 4 κ c 2 r 2 . \begin{equation}
\frac{1}{c_* + c} = \frac{\sigma^2}{4 \kappa c^2 r^2} \ . \tag*{(106)}
\end{equation} c ∗ + c 1 = 4 κ c 2 r 2 σ 2 . (106)
Rearranging,
r 2 = 1 + σ 2 2 κ c = σ 2 4 κ c 2 ( c ∗ + c ) ⟹ 1 + σ 2 4 κ c = σ 2 4 κ c 2 c ∗ , \begin{equation}
r^2 = 1 + \frac{\sigma^2}{2 \kappa c} = \frac{\sigma^2}{4 \kappa c^2} (c_* + c) \implies 1 + \frac{\sigma^2}{4 \kappa c} = \frac{\sigma^2}{4 \kappa c^2} c_* \ , \tag*{(107)}
\end{equation} r 2 = 1 + 2 κ c σ 2 = 4 κ c 2 σ 2 ( c ∗ + c ) ⟹ 1 + 4 κ c σ 2 = 4 κ c 2 σ 2 c ∗ , (107)
so
σ 2 2 κ c = 2 c ∗ / c − 1 . \begin{equation}
\frac{\sigma^2}{2 \kappa c} = \frac{2}{c_*/c - 1} \ . \tag*{(108)}
\end{equation} 2 κ c σ 2 = c ∗ / c − 1 2 . (108)
This implies that r r r can be written in terms of c c c as
r ( c ) 2 = 1 + 2 c ∗ / c − 1 = c ∗ / c + 1 c ∗ / c − 1 = c ∗ + c c ∗ − c . \begin{equation}
r(c)^2 = 1 + \frac{2}{c_*/c - 1} = \frac{c_*/c + 1}{c_*/c - 1} = \frac{c_* + c}{c_* - c} \ . \tag*{(109)}
\end{equation} r ( c ) 2 = 1 + c ∗ / c − 1 2 = c ∗ / c − 1 c ∗ / c + 1 = c ∗ − c c ∗ + c . (109)
Using this result, the first condition can be rewritten as
κ 2 = 1 ( c ∗ + c ) 2 r 2 = c ∗ − c ( c ∗ + c ) 3 . \begin{equation}
\kappa^2 = \frac{1}{(c_* + c)^2 r^2} = \frac{c_* - c}{(c_* + c)^3} \ . \tag*{(110)}
\end{equation} κ 2 = ( c ∗ + c ) 2 r 2 1 = ( c ∗ + c ) 3 c ∗ − c . (110)
Let x : = c / c ∗ x := c/c_* x := c / c ∗ denote the non-dimensionalized learned rate. In terms of x x x , our two conditions read
κ 2 = 1 c ∗ 2 1 − x ( 1 + x ) 3 σ 2 = 4 c ∗ κ x 2 1 − x . \begin{equation}
\boxed{ \ \kappa^2 = \frac{1}{c_*^2} \frac{1 - x}{(1 + x)^3} \hspace{1in} \sigma^2 = \frac{4 c_* \kappa x^2}{1 - x} \ } \ . \tag*{(111)}
\end{equation} κ 2 = c ∗ 2 1 ( 1 + x ) 3 1 − x σ 2 = 1 − x 4 c ∗ κ x 2 . (111)
Since both σ 2 \sigma^2 σ 2 and κ \kappa κ can be written in terms of the optimal value of c c c (or equivalently, x x x ), it suffices to determine this value. We can then write σ 2 \sigma^2 σ 2 in terms of κ \kappa κ , and hence determine the phase boundary.
Solving a cubic equation.
Note that the first condition expresses a cubic equation (in terms of c ∗ c_* c ∗ and κ \kappa κ ) whose solution is x x x :
c ∗ 2 κ 2 ( 1 + x ) 3 = 1 − x . \begin{equation}
c_*^2 \kappa^2 (1 + x)^3 = 1 - x \ . \tag*{(112)}
\end{equation} c ∗ 2 κ 2 ( 1 + x ) 3 = 1 − x . (112)
Note that it only has a valid solution if 0 ≤ x < 1 0 \leq x < 1 0 ≤ x < 1 .
Generic cubic equations are annoying to solve, but they can be made tractable with a clever substitution. Consider the inspired choice
q : = 1 − x 1 + x \begin{equation}
\boxed{ \ q := \sqrt{\frac{1 - x}{1 + x}} \ } \tag*{(113)}
\end{equation} q := 1 + x 1 − x (113)
which implies that
x = 1 − q 2 1 + q 2 . \begin{equation}
\boxed{ \ x = \frac{1 - q^2}{1 + q^2} \ } \ . \tag*{(114)}
\end{equation} x = 1 + q 2 1 − q 2 . (114)
Relatedly,
1 + x = 2 1 + q 2 1 − x = 2 q 2 1 + q 2 . \begin{equation}
\begin{split}
1 + x &= \frac{2}{1 + q^2} \\
1 - x &= \frac{2 q^2}{1 + q^2} \ .
\end{split} \tag*{(115)}
\end{equation} 1 + x 1 − x = 1 + q 2 2 = 1 + q 2 2 q 2 . (115)
In terms of this new variable q q q , our cubic equation becomes
c ∗ 2 κ 2 = q 2 4 ( 1 + q 2 ) 2 . \begin{equation}
c_*^2 \kappa^2 = \frac{q^2}{4} (1 + q^2)^2 \ . \tag*{(116)}
\end{equation} c ∗ 2 κ 2 = 4 q 2 ( 1 + q 2 ) 2 . (116)
Since q ≥ 0 q \geq 0 q ≥ 0 by definition, we can take the square root of both sides to find that
2 c ∗ κ = q ( 1 + q 2 ) , \begin{equation}
2 c_* \kappa = q (1 + q^2) \ , \tag*{(117)}
\end{equation} 2 c ∗ κ = q ( 1 + q 2 ) , (117)
or equivalently that
q 3 + q − 2 κ c ∗ = 0 . \begin{equation}
\boxed{ \ q^3 + q - 2 \kappa c_* = 0 \ } \ . \tag*{(118)}
\end{equation} q 3 + q − 2 κ c ∗ = 0 . (118)
This is a depressed cubic equation. Its discriminant equals
Δ = − 4 − 27 ( − 2 κ c ∗ ) 2 < 0 , \begin{equation}
\Delta = - 4 - 27 (- 2 \kappa c_*)^2 < 0 \ , \tag*{(119)}
\end{equation} Δ = − 4 − 27 ( − 2 κ c ∗ ) 2 < 0 , (119)
so it has one real root and two complex conjugate roots. Because a valid solution must be real, we are interested in the real root.
Using a standard formula for this root, we find that it equals
q ( κ ) = 2 3 sinh [ 1 3 arsinh ( 3 3 κ c ∗ ) ] \begin{equation}
\boxed{ \ q(\kappa) = \frac{2}{\sqrt{3}} \sinh\left[ \frac{1}{3} \text{arsinh}(3 \sqrt{3} \kappa c_* ) \right] \ } \tag*{(120)}
\end{equation} q ( κ ) = 3 2 sinh [ 3 1 arsinh ( 3 3 κ c ∗ ) ] (120)
where
arsinh ( x ) : = log ( x + x 2 + 1 ) . \begin{equation}
\text{arsinh}(x) := \log( x + \sqrt{x^2 + 1} ) \ . \tag*{(121)}
\end{equation} arsinh ( x ) := log ( x + x 2 + 1 ) . (121)
Determining the critical noise level.
Combining this result with our earlier formula for σ 2 \sigma^2 σ 2 in terms of x x x ,
σ 2 = 4 c ∗ κ x 2 1 − x = 4 c ∗ κ ( 1 − q 2 ) 2 2 q 2 ( 1 + q 2 ) . \begin{equation}
\sigma^2 = \frac{4 c_* \kappa x^2}{1 - x} = 4 c_* \kappa \frac{(1 - q^2)^2}{2 q^2 (1 + q^2)} \ . \tag*{(122)}
\end{equation} σ 2 = 1 − x 4 c ∗ κ x 2 = 4 c ∗ κ 2 q 2 ( 1 + q 2 ) ( 1 − q 2 ) 2 . (122)
Since 2 c ∗ κ = q ( 1 + q 2 ) 2 c_* \kappa = q (1 + q^2) 2 c ∗ κ = q ( 1 + q 2 ) , this becomes
σ crit 2 = ( 1 − q ( κ ) 2 ) 2 q ( κ ) . \begin{equation}
\boxed{ \ \sigma^2_{\text{crit}} = \frac{(1 - q(\kappa)^2)^2}{q(\kappa)} \ } \ . \tag*{(123)}
\end{equation} σ crit 2 = q ( κ ) ( 1 − q ( κ ) 2 ) 2 . (123)
The final phase boundary result.
Summarizing, at the boundary, provided 0 < κ c ∗ ≤ 1 0 < \kappa c_* \leq 1 0 < κ c ∗ ≤ 1 ,
q = 2 3 sinh [ 1 3 arsinh ( 3 3 κ c ∗ ) ] σ 2 = ( 1 − q 2 ) 2 q . \begin{equation}
\begin{split}
q &= \frac{2}{\sqrt{3}} \sinh\left[ \frac{1}{3} \text{arsinh}(3 \sqrt{3} \kappa c_* ) \right] \\
\sigma^2 &= \frac{(1 - q^2)^2}{q} \ .
\end{split} \tag*{(124)}
\end{equation} q σ 2 = 3 2 sinh [ 3 1 arsinh ( 3 3 κ c ∗ ) ] = q ( 1 − q 2 ) 2 . (124)
Characterization of frequency-runaway phase transition
In this appendix, we characterize the ‘frequency-runaway’ phase transition discussed in § 7 that occurs when u u u and v v v are frozen (with u = v = 1 u = v = 1 u = v = 1 ) but c c c is trainable.
If u u u and v v v are frozen after alignment, the reduced objective becomes
J = 1 + σ 2 4 c ( τ ) − 1 c ∗ + c ( τ ) + 1 4 c ∗ \begin{equation}
J = \frac{1 + \sigma^2}{4 c(\tau)} - \frac{1}{c_* + c(\tau)} + \frac{1}{4 c_*} \tag*{(125)}
\end{equation} J = 4 c ( τ ) 1 + σ 2 − c ∗ + c ( τ ) 1 + 4 c ∗ 1 (125)
up to an additive constant. (Because it is constant, we have removed the regularization term). The negative gradient of J J J equals
− ∂ J ∂ c = ( 1 + σ 2 ) 4 c 2 − 1 ( c ∗ + c ) 2 = ( 1 + σ 2 ) ( c ∗ + c ) 2 − 4 c 2 4 c 2 ( c ∗ + c ) 2 = [ 1 + σ 2 ( c ∗ + c ) + 2 c ] [ 1 + σ 2 ( c ∗ + c ) − 2 c ] 4 c 2 ( c ∗ + c ) 2 , \begin{equation}
\begin{split}
- \frac{\partial J}{\partial c} &= \frac{(1 + \sigma^2)}{4 c^2} - \frac{1}{(c_* + c)^2} \\
&= \frac{(1 + \sigma^2) (c_* + c)^2 - 4 c^2}{4 c^2 (c_* + c)^2} \\
&= \frac{[ \sqrt{1 + \sigma^2} (c_* + c) + 2 c] [ \sqrt{1 + \sigma^2} (c_* + c) - 2 c]}{4 c^2 (c_* + c)^2} \ ,
\end{split} \tag*{(126)}
\end{equation} − ∂ c ∂ J = 4 c 2 ( 1 + σ 2 ) − ( c ∗ + c ) 2 1 = 4 c 2 ( c ∗ + c ) 2 ( 1 + σ 2 ) ( c ∗ + c ) 2 − 4 c 2 = 4 c 2 ( c ∗ + c ) 2 [ 1 + σ 2 ( c ∗ + c ) + 2 c ] [ 1 + σ 2 ( c ∗ + c ) − 2 c ] , (126)
so c c c has dynamics
d d τ c = [ 1 + σ 2 ( c ∗ + c ) + 2 c ] 4 c 2 ( c ∗ + c ) 2 [ 1 + σ 2 ( c ∗ + c ) − 2 c ] . \begin{equation}
\frac{d}{d\tau} c = \frac{[ \sqrt{1 + \sigma^2} (c_* + c) + 2 c] }{4 c^2 (c_* + c)^2} [ \sqrt{1 + \sigma^2} (c_* + c) - 2 c] \ . \tag*{(127)}
\end{equation} d τ d c = 4 c 2 ( c ∗ + c ) 2 [ 1 + σ 2 ( c ∗ + c ) + 2 c ] [ 1 + σ 2 ( c ∗ + c ) − 2 c ] . (127)
The fixed point of c c c dynamics can only depend on the second factor in the numerator, since the first factor is always positive for c ≥ 0 c \geq 0 c ≥ 0 and c ∗ > 0 c_* > 0 c ∗ > 0 . The second factor equals zero at
c opt = 1 + σ 2 c ∗ 2 − 1 + σ 2 , \begin{equation}
\boxed{ \ c_{\text{opt}} = \frac{\sqrt{1 + \sigma^2} \ c_*}{2 - \sqrt{1 + \sigma^2}} \ } \ , \tag*{(128)}
\end{equation} c opt = 2 − 1 + σ 2 1 + σ 2 c ∗ , (128)
so when c opt c_{\text{opt}} c opt is well-defined, it is the fixed point of c c c dynamics.
Importantly, this quantity blows up when its denominator equals zero, which occurs at the critical noise level σ crit 2 : = 3 \sigma^2_{\text{crit}} := 3 σ crit 2 := 3 . Just below that noise level,
c opt ≈ 8 c ∗ σ crit 2 − σ 2 , \begin{equation}
\boxed{ \ c_{\text{opt}} \approx \frac{8 c_*}{\sigma^2_{\text{crit}} - \sigma^2} \ } \ , \tag*{(129)}
\end{equation} c opt ≈ σ crit 2 − σ 2 8 c ∗ , (129)
so it diverges as 1 / ( σ crit 2 − σ 2 ) 1/(\sigma^2_{\text{crit}} - \sigma^2) 1/ ( σ crit 2 − σ 2 ) . For σ 2 ≥ σ crit 2 \sigma^2 \geq \sigma^2_{\text{crit}} σ 2 ≥ σ crit 2 , the second factor equals
1 + σ 2 ( c ∗ + c ) − 2 c = 1 + σ 2 c ∗ + ( 1 + σ 2 − 2 ) c , \begin{equation}
\sqrt{1 + \sigma^2} (c_* + c) - 2 c = \sqrt{1 + \sigma^2} c_* + ( \sqrt{1 + \sigma^2} - 2) c \ , \tag*{(130)}
\end{equation} 1 + σ 2 ( c ∗ + c ) − 2 c = 1 + σ 2 c ∗ + ( 1 + σ 2 − 2 ) c , (130)
which is always positive. This means that, beyond the critical noise level, dynamics drive c c c to infinity. Hence, beyond the threshold, J J J is a decreasing function of c c c and has no finite minimum.