Introduction
The ability of neural networks to learn from a limited number of examples depends on their inductive biases, i.e., on the assumptions about the task that are built into the architecture and the training algorithm. A natural source of inductive bias is symmetry. When a task is invariant under a group of transformations of the input, a model that respects this invariance does not need to learn the same features separately for every transformed version of the input. Weight sharing is the standard way to build such symmetries into a network ([39 , 11 , 33 , 23 ] ), and it is expected to reduce the number of samples needed to learn and to speed up training.
The main success story of this approach is the convolutional neural network (CNN), widely used in image classification and other computer vision tasks ([26 , 24 ] ). CNNs exploit two structural properties of natural images, each encoded by a distinct architectural choice: Weight sharing across positions encodes the approximate invariance of image statistics under translations, while small filters encode the locality of the relevant features. Additionally, natural images are strongly correlated across space [34 ] , and the labels used for training are often noisy, resulting in a complex interplay between architectural constraints and the noise of stochastic gradient descent (SGD).
From a theoretical perspective, building symmetries into a model is known to reduce the number of samples needed to learn, typically by a factor of the order of the size of the symmetry group [6 , 28 , 41 , 14 , 25 , 43 ] . However, disentangling the effects of locality and weight sharing remains a largely open challenge, especially in the feature-learning regime. Moreover, most analyses assume independent inputs, thereby neglecting the spatial correlations of natural images. It is therefore unclear to what extent the advantage of weight sharing survives in the presence of correlations and noise.
In this work, we address these questions in a teacher–student model whose learning dynamics can be described exactly. The teacher is a two-layer convolutional network with kernel size k t k_t k t P P P D D D c c c k s ≥ k t k_s \ge k_t k s ≥ k t k s k_s k s D D D [36 , 5 , 19 ] .
Contributions. Our main contribution is an exact description of how weight sharing and locality shape the online SGD dynamics of convolutional networks, which separates their effects. More precisely:
In the limit of large feature dimension D D D
For uncorrelated positions, we show analytically that the locally connected network follows exactly the same training trajectory as the convolutional network, but needs exactly P P P
By integrating the ODEs numerically, we find that locality affects the sample complexity of the problem. Indeed, a convolutional student with the right degree of locality, k s = k t k_s = k_t k s = k t O ( D ) \mathcal{O}(D) O ( D ) O ( D log D ) \mathcal{O}(D \log D) O ( D log D ) k s > k t k_s > k_t k s > k t
We show that spatial correlations can reduce the advantage of weight sharing. With a perturbative expansion at small learning rate, we find that the asymptotic error of the convolutional network grows with the correlation strength c c c c c c c > 0 c > 0 c > 0
is tuned to the training horizon, we find numerically that the locally connected network has the lower error at early times but the convolutional network wins at late times (§ 4.3 ), with a transition time that we predict analytically from an early-time expansion of the ODEs (Result 4.1).
The code used in the paper is available at https://github.com/francescomori/CNN_theory .
Outline. The paper is organized as follows. § 2 introduces the model. § 3 presents the ODEs and the effects of weight sharing and locality for uncorrelated positions. § 4 studies the effect of spatial correlations, and § 5 discusses limitations and future directions. Related works are discussed in § 1.1 , and derivations and additional results are given in the appendices.
Related works
Equivariant architectures. Weight sharing can be viewed as a mechanism for hard-coding symmetries into a learning architecture. Early work constructed neural networks with tied parameters to guarantee invariance under prescribed group actions [39 ] . This perspective later gave rise to group-equivariant convolutional networks [11 ] , general parameter-sharing constructions for equivariant layers [33 ] , and characterizations of convolution as the natural linear operation associated with equivariance under group actions [23 ] .
Statistical benefits of symmetry. Another body of literature has quantified the statistical benefits of such symmetry constraints. Sannai et al. [38 ] derived generalization bounds for invariant and equivariant networks using quotient feature spaces, while Bietti et al. [6 ] showed that learning with invariant kernels reduces the number of samples by a factor asymptotically equal to the size of the symmetry group. Related sample-complexity gains were established for invariant random-feature and kernel models [28 ] , kernel regression [41 ] , and for equivariant linear models [14 ] . At the same time, symmetry alone does not remove computational barriers: Kiani et al. [22 ] established super-polynomial or exponential statistical-query lower bounds for learning several classes of equivariant models, including convolutional networks.
Locality versus weight sharing. Several works studied the separate contributions of locality and weight sharing. In a teacher–student setting for kernel regression with convolutional kernels, Favero et al. [16 ] found that locality determines the exponent of the learning curve, whereas translational invariance does not, and Cagnetta et al. [10 ] extended this analysis to deep convolutional kernels. Wang and Wu [44 ] compared convolutional, locally connected, and fully connected networks, and related the different sample complexities of the three architectures to the symmetries of the learning procedure. Lahoti et al. [25 ] compared the same three architectures on inputs made of patches, with the signal planted in one of them, and found that the locally connected network needs more samples than the convolutional one by a factor of order the number of patches. Similarly, Li et al. (2021) showed that, on some tasks, fully connected networks trained by gradient descent need quadratically more samples than convolutional ones. Regarding optimization, d’Ascoli et al. (2019) found experimentally that the convolutional constraint mainly helps during the early stages of training, and Ingrosso and Goldt [21 ] showed that localized filters can emerge in fully connected networks because of the non-Gaussian statistics of the inputs.
Learning convolutional teachers. For Gaussian inputs with independent, non-overlapping patches, it was shown that gradient descent recovers a convolutional teacher in polynomial time ([9 ] ), even when the loss has spurious local minima ((Du et al., 2018)). Goel et al. [18 ] extended these results to overlapping patches and more general input distributions, using a gradient-like algorithm. The very recent work by Waber et al. [43 ] studied online SGD in a convolutional teacher–student model with independent Gaussian inputs and a filter that extends over all patches, showing that weight sharing and data augmentation reduce the sample complexity by a factor equal to the number of patches, which is taken to be polynomial in the input dimension. In this work, we consider the roles of both weight sharing and locality, by considering a teacher of arbitrary kernel size and comparing convolutional and locally connected students whose kernel size can exceed that of the teacher. We also investigate how correlations across patches can reduce the advantage of weight sharing, both in the speed of convergence and in the asymptotic generalization error in the presence of label noise.
Exact learning dynamics in high dimensions. Classical works ([36 , 5 , 15 ] ) described the dynamics of online learning in two-layer networks exactly in the high-dimensional limit, through ordinary differential equations for a few order parameters. These equations were used to explain the plateaus of the generalization error caused by the permutation symmetry of the hidden units ([36 ] ), to predict the asymptotic error due to label noise ((Saad and Solla, 1997)), and to derive optimal learning-rate schedules ((Saad and Rattray, 1997)). More recent works made this approach rigorous ([19 , 4 ] ) and extended it to different learning regimes ([42 ] ), structured inputs ([20 ] ), and the design of optimal training protocols ([29 ] ). Here, we extend this framework to convolutional and locally connected networks, which gives an exact description of their online SGD dynamics. This allows us to quantify how locality sets the number of samples needed to recover the teacher, when convolutional weight sharing accelerates learning relative to an otherwise matched locally connected architecture, and when correlations and noise can instead make the additional flexibility of the unshared model advantageous.
Setup and notation
In this work, we study an exactly solvable teacher–student model in which the teacher is a convolutional network and the student is either convolutional or locally connected. Teacher–student models are a class of supervised learning settings in which the labels are generated by a fixed teacher network, which represents the ground truth to be recovered, while a student network is trained to reproduce the teacher’s output ([17 , 15 ] ). When the input distribution is sufficiently simple, for instance Gaussian, the learning dynamics of these models can often be characterized exactly in the high-dimensional limit ([36 , 5 , 19 ] ). Our convolutional teacher also belongs to the broader class of multi-index models ([3 , 1 , 2 , 7 , 43 ] ).
Data structure
We consider a supervised learning setting with inputs x x x y y y x ∈ R P D x \in\mathbb{R}^{PD} x ∈ R P D
x = ( x 0 , … , x P − 1 ) , x p ∈ R D . (1) x = (x_0,\ldots,x_{P-1}), \qquad x_p \in\mathbb{R}^{D}. \tag*{(1)} x = ( x 0 , … , x P − 1 ) , x p ∈ R D . (1) We refer to the index p ∈ { 0 , … , P − 1 } p \in\{0,\ldots,P-1\} p ∈ { 0 , … , P − 1 } position and to the D D D x p x_p x p features at that position. We take P P P D → ∞ D \to\infty D → ∞
The components of x x x
E [ x p , a x q , b ] = S p , q δ a , b , p , q ∈ { 0 , … , P − 1 } , a , b ∈ { 1 , … , D } . (2) \mathbb{E}\left[x_{p,a}x_{q,b}\right] = S_{p,q}\delta_{a,b}, \qquad p,q \in\{0,\ldots,P-1\}, \quad a,b \in\{1,\ldots,D\}. \tag*{(2)} E [ x p , a x q , b ] = S p , q δ a , b , p , q ∈ { 0 , … , P − 1 } , a , b ∈ { 1 , … , D } . (2) The P × P P \times P P × P S S S
S p , q = { 1 , p = q , c , p ≠ q , c ∈ [ 0 , 1 ) , (3) S_{p,q} =
\begin{cases}
1, & p=q,\\
c, & p\ne q,
\end{cases}
\qquad c \in[0,1), \tag*{(3)} S p , q = { 1 , c , p = q , p = q , c ∈ [ 0 , 1 ) , (3) so that every position has unit variance and every pair of distinct positions carries the same correlation c c c c = 0 c=0 c = 0 S = I S=I S = I
The labels are generated as
y ( x ) = f ⋆ ( x ) + σ ξ , (4) y(x) = f^{\star}(x) + \sigma\xi, \tag*{(4)} y ( x ) = f ⋆ ( x ) + σ ξ , (4) where f ⋆ ( x ) f^{\star}(x) f ⋆ ( x ) ξ ∼ N ( 0 , 1 ) \xi\sim\mathcal{N}(0,1) ξ ∼ N ( 0 , 1 ) x x x σ ≥ 0 \sigma\ge0 σ ≥ 0 k t < P k_t < P k t < P
f ⋆ ( x ) = ∑ p = 0 P − 1 g ( λ p ( x ) ) , (5) f^{\star}(x) = \sum_{p=0}^{P-1} g\left(\lambda_p(x)\right), \tag*{(5)} f ⋆ ( x ) = p = 0 ∑ P − 1 g ( λ p ( x ) ) , (5) with the preactivations
λ p ( x ) = 1 k t D ∑ l = 0 k t − 1 w l ⋅ x ( p + l ) P . (6) \lambda_p(x) = \frac{1}{\sqrt{k_tD}}\sum_{l=0}^{k_t-1} w_l \cdot x_{(p+l)_P}. \tag*{(6)} λ p ( x ) = k t D 1 l = 0 ∑ k t − 1 w l ⋅ x ( p + l ) P . (6) and ( q ) P ≡ q m o d P (q)_P \equiv q \bmod P ( q ) P ≡ q mod P w 0 , … , w k t − 1 w_0,\ldots,w_{k_t-1} w 0 , … , w k t − 1 kernel (or filter ) of the teacher, which is applied at every position p p p k t k_t k t p , … , p + k t − 1 p,\ldots,p+k_t-1 p , … , p + k t − 1 k t k_t k t g ( z ) g(z) g ( z ) g ( z ) = erf ( z / 2 ) g(z)=\operatorname{erf}(z/\sqrt{2}) g ( z ) = erf ( z / 2 )
w l ⋅ w l ′ D = T δ l , l ′ . (7) \frac{w_l \cdot w_{l'}}{D}=T\delta_{l,l'}. \tag*{(7)} D w l ⋅ w l ′ = T δ l , l ′ . (7) Student architectures
As students, we consider both convolutional neural networks (CNNs) and locally connected networks (LCNs). In both cases we take the kernel size k s ≥ k t k_s \ge k_t k s ≥ k t
The CNN student takes the form
y ^ C N N ( x ) = ∑ p = 0 P − 1 g ( λ ^ p C N N ) , (8) \hat{y}_{\mathrm{CNN}}(x)=\sum_{p=0}^{P-1}g\left(\hat{\lambda}^{\mathrm{CNN}}_p\right), \tag*{(8)} y ^ CNN ( x ) = p = 0 ∑ P − 1 g ( λ ^ p CNN ) , (8) with preactivations
λ ^ p C N N ( x ) = 1 k s D ∑ l = 0 k s − 1 w ^ l ⋅ x ( p + l ) P . (9) \hat{\lambda}^{\mathrm{CNN}}_p(x)=\frac{1}{\sqrt{k_sD}}\sum_{l=0}^{k_s-1}\hat{w}_l\cdot x_{(p+l)_P}. \tag*{(9)} λ ^ p CNN ( x ) = k s D 1 l = 0 ∑ k s − 1 w ^ l ⋅ x ( p + l ) P . (9) The same kernel ( w ^ l ) l = 0 k s − 1 (\hat{w}_l)_{l=0}^{k_s-1} ( w ^ l ) l = 0 k s − 1 k s = k t k_s=k_t k s = k t
y ^ L C N ( x ) = ∑ p = 0 P − 1 g ( λ ^ p L C N ) , (10) \hat{y}_{\mathrm{LCN}}(x)=\sum_{p=0}^{P-1}g\left(\hat{\lambda}^{\mathrm{LCN}}_p\right), \tag*{(10)} y ^ LCN ( x ) = p = 0 ∑ P − 1 g ( λ ^ p LCN ) , (10) with preactivations
λ ^ p L C N ( x ) = 1 k s D ∑ l = 0 k s − 1 w ^ l ( p ) ⋅ x ( p + l ) P . (11) \hat{\lambda}^{\mathrm{LCN}}_p(x)=\frac{1}{\sqrt{k_sD}}\sum_{l=0}^{k_s-1}\hat{w}^{(p)}_l\cdot x_{(p+l)_P}. \tag*{(11)} λ ^ p LCN ( x ) = k s D 1 l = 0 ∑ k s − 1 w ^ l ( p ) ⋅ x ( p + l ) P . (11) Each position p p p ( w ^ l ( p ) ) l = 0 k s − 1 (\hat{w}^{(p)}_l)_{l=0}^{k_s-1} ( w ^ l ( p ) ) l = 0 k s − 1 P P P k s = P k_s=P k s = P [36 ] ). A schematic of the two architectures is given in Figure 1 .
Figure 1. Teacher and the two student architectures with P P P k t k_t k t k s k_s k s P = 4 P=4 P = 4 k s = 3 k_s=3 k s = 3 k t = 2 k_t=2 k t = 2
Weight sharing and locality are the two structural properties that distinguish CNNs. Interpreting the input as a one-dimensional image with periodic boundary conditions, weight sharing makes both the teacher and the CNN student invariant under translations. Indeed, defining the cyclic shift ( τ x ) p = x ( p + 1 ) P (\tau x)_p=x_{(p+1)_P} ( τ x ) p = x ( p + 1 ) P f ∗ ( τ x ) = f ∗ ( x ) f^*(\tau x)=f^*(x) f ∗ ( τ x ) = f ∗ ( x ) y ^ C N N ( τ x ) = y ^ C N N ( x ) \hat{y}_{\mathrm{CNN}}(\tau x)=\hat{y}_{\mathrm{CNN}}(x) y ^ CNN ( τ x ) = y ^ CNN ( x ) S S S (3) , the input distribution is also invariant under τ \tau τ w ^ l ( p ) = w ^ l \hat{w}^{(p)}_l=\hat{w}_l w ^ l ( p ) = w ^ l
Learning algorithm
Both students are trained by online stochastic gradient descent (SGD) on the squared loss
ℓ ( x ; θ ) = 1 2 ( y ( x ) − y ^ θ ( x ) ) 2 (12) \ell(x;\theta)=\frac{1}{2}\left(y(x)-\hat{y}_{\theta}(x)\right)^2 \tag*{(12)} ℓ ( x ; θ ) = 2 1 ( y ( x ) − y ^ θ ( x ) ) 2 (12) where θ \theta θ t t t ( x t , y t ) (x^t,y^t) ( x t , y t )
θ t + 1 = θ t − η ∇ θ ℓ ( x t ; θ t ) (13) \theta^{t+1}=\theta^t-\eta\nabla_{\theta}\ell(x^t;\theta^t) \tag*{(13)} θ t + 1 = θ t − η ∇ θ ℓ ( x t ; θ t ) (13) with η > 0 \eta>0 η > 0 N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 )
ε g ( θ ) = E x [ ( f ∗ ( x ) − y ^ θ ( x ) ) 2 ] . (14) \varepsilon_g(\theta)=\mathbb{E}_x\left[\left(f^*(x)-\hat{y}_{\theta}(x)\right)^2\right]. \tag*{(14)} ε g ( θ ) = E x [ ( f ∗ ( x ) − y ^ θ ( x ) ) 2 ] . (14) As we show in Appendix § 11 , ε g \varepsilon_g ε g θ \theta θ
Learning dynamics
In the limit where the feature dimension D D D t t t α ≡ t / D \alpha\equiv t/D α ≡ t / D order parameters in statistical physics. As stated in the Result below, the order parameters concentrate as D → ∞ D\to\infty D → ∞ α \alpha α
Q l l ′ = w ^ l ⋅ w ^ l ′ k s k t D , R l l ′ ′ = w ^ l ⋅ w l ′ ′ k s k t D , (15) Q_{ll'}=\frac{\hat{w}_l\cdot\hat{w}_{l'}}{k_s k_t D},\qquad R_{ll''}=\frac{\hat{w}_l\cdot w_{l''}}{\sqrt{k_s k_t}D}, \tag*{(15)} Q l l ′ = k s k t D w ^ l ⋅ w ^ l ′ , R l l ′′ = k s k t D w ^ l ⋅ w l ′′ , (15) with l , l ′ ∈ { 0 , … , k s − 1 } l,l'\in\{0,\ldots,k_s-1\} l , l ′ ∈ { 0 , … , k s − 1 } l ′ ′ ∈ { 0 , … , k t − 1 } l''\in\{0,\ldots,k_t-1\} l ′′ ∈ { 0 , … , k t − 1 } p , q ∈ { 0 , … , P − 1 } p,q\in\{0,\ldots,P-1\} p , q ∈ { 0 , … , P − 1 }
Q l l ′ ( p , q ) = w ^ l ( p ) ⋅ w ^ l ′ ( q ) k s D , R l l ′ ′ ( p ) = w ^ l ( p ) ⋅ w l ′ ′ k s k t D . (16) Q_{ll'}^{(p,q)}=\frac{\hat{w}_l^{(p)}\cdot\hat{w}_{l'}^{(q)}}{k_sD},\qquad R_{ll''}^{(p)}=\frac{\hat{w}_l^{(p)}\cdot w_{l''}}{\sqrt{k_s k_t}D}. \tag*{(16)} Q l l ′ ( p , q ) = k s D w ^ l ( p ) ⋅ w ^ l ′ ( q ) , R l l ′′ ( p ) = k s k t D w ^ l ( p ) ⋅ w l ′′ . (16) Result 3.1 (Exact ODEs for the learning dynamics). Consider the limit D , t → ∞ D,t\to\infty D , t → ∞ α = t / D = O D ( 1 ) \alpha=t/D=\mathcal{O}_D(1) α = t / D = O D ( 1 ) P P P k t k_t k t k s k_s k s η \eta η T T T σ \sigma σ
(i) CNN student. Let w ^ l t \hat{w}_l^t w ^ l t t t t (13) . The associated order parameters R R R Q Q Q (15) , converge in probability to the solution at time α \alpha α
d R d α = η k s F R C N N ( R , Q ) , d Q d α = η k s F Q C N N ( R , Q ) + η 2 k s 2 G Q C N N ( R , Q ) . (17) \frac{\mathrm{d}R}{\mathrm{d}\alpha}=\frac{\eta}{k_s}F_R^{\mathrm{CNN}}(R,Q),\qquad
\frac{\mathrm{d}Q}{\mathrm{d}\alpha}=\frac{\eta}{k_s}F_Q^{\mathrm{CNN}}(R,Q)+\frac{\eta^2}{k_s^2}G_Q^{\mathrm{CNN}}(R,Q). \tag*{(17)} d α d R = k s η F R CNN ( R , Q ) , d α d Q = k s η F Q CNN ( R , Q ) + k s 2 η 2 G Q CNN ( R , Q ) . (17) (ii) LCN student. Let w ^ l ( p ) , t \hat{w}_l^{(p),t} w ^ l ( p ) , t t t t R ( p ) R^{(p)} R ( p ) Q ( p , q ) Q^{(p,q)} Q ( p , q ) (16) , converge in probability to the solution at time α \alpha α
d R ( p ) d α = η k s F R L C N , ( p ) ( R , Q ) , d Q ( p , q ) d α = η k s F Q L C N , ( p , q ) ( R , Q ) + η 2 k s 2 G Q L C N , ( p , q ) ( R , Q ) , (18) \frac{\mathrm{d}R^{(p)}}{\mathrm{d}\alpha}=\frac{\eta}{k_s}F_R^{\mathrm{LCN},(p)}(\mathbf{R},\mathbf{Q}),\qquad
\frac{\mathrm{d}Q^{(p,q)}}{\mathrm{d}\alpha}=\frac{\eta}{k_s}F_Q^{\mathrm{LCN},(p,q)}(\mathbf{R},\mathbf{Q})+\frac{\eta^2}{k_s^2}G_Q^{\mathrm{LCN},(p,q)}(\mathbf{R},\mathbf{Q}), \tag*{(18)} d α d R ( p ) = k s η F R LCN , ( p ) ( R , Q ) , d α d Q ( p , q ) = k s η F Q LCN , ( p , q ) ( R , Q ) + k s 2 η 2 G Q LCN , ( p , q ) ( R , Q ) , (18) where R = { R ( p ) } p \mathbf{R}=\{R^{(p)}\}_p R = { R ( p ) } p Q = { Q ( p , q ) } p , q \mathbf{Q}=\{Q^{(p,q)}\}_{p,q} Q = { Q ( p , q ) } p , q
The functions F R F_R F R F Q F_Q F Q G Q G_Q G Q § 9 and § 10 .
The terms linear in η \eta η η 2 \eta^2 η 2 D → ∞ D \to\infty D → ∞ D = 1000 D = 1000 D = 1000 Figure 2 .
Figure 2. Generalization errors ε g C N N ( α ) \varepsilon_g^{\mathrm{CNN}}(\alpha) ε g CNN ( α ) ε g L C N ( P α ) \varepsilon_g^{\mathrm{LCN}}(P\alpha) ε g LCN ( P α ) c = 0 c = 0 c = 0 c = 0.5 c = 0.5 c = 0.5 g = erf g = \operatorname{erf} g = erf k t = 2 k_t = 2 k t = 2 k s = 3 k_s = 3 k s = 3 P = 4 P = 4 P = 4 η = 1 \eta= 1 η = 1 σ = 0 \sigma= 0 σ = 0 T = 1 T = 1 T = 1 D = 1000 D = 1000 D = 1000
Note that since each weight component is initialized as an independent standard Gaussian variable, the initial conditions for large D D D
Q l l ′ = δ l l ′ k s + O D ( D − 1 / 2 ) , R l l ′ = O D ( D − 1 / 2 ) (19) Q_{ll'} = \frac{\delta_{ll'}}{k_s} + O_D(D^{-1/2}), \qquad R_{ll'} = O_D(D^{-1/2}) \tag*{(19)} Q l l ′ = k s δ l l ′ + O D ( D − 1/2 ) , R l l ′ = O D ( D − 1/2 ) (19) for the CNN, and
Q l l ′ ( p , q ) = δ p q δ l l ′ k s + O D ( D − 1 / 2 ) , R l l ′ ( p ) = O D ( D − 1 / 2 ) , (20) Q_{ll'}^{(p,q)} = \frac{\delta_{pq}\delta_{ll'}}{k_s} + O_D(D^{-1/2}), \qquad R_{ll'}^{(p)} = O_D(D^{-1/2}), \tag*{(20)} Q l l ′ ( p , q ) = k s δ pq δ l l ′ + O D ( D − 1/2 ) , R l l ′ ( p ) = O D ( D − 1/2 ) , (20) for the LCN.
Benefits of translational invariance
We next compare the dynamics of the CNN and the LCN, starting from the simplest case of uncorrelated positions, S = I S = I S = I P P P
Result 3.2 (Exact P P P S = I S = I S = I k t ≤ k s ≤ P k_t \leq k_s \leq P k t ≤ k s ≤ P ( R , Q ) ( α ) (R,Q)(\alpha) ( R , Q ) ( α ) (17) with initial condition ( R 0 , Q 0 ) (R_0,Q_0) ( R 0 , Q 0 ) ( R , Q ) ( α ) (R,Q)(\alpha) ( R , Q ) ( α )
solution of the LCN equations (18) with the matched, position-symmetric initial condition
R ( p ) ( 0 ) = R 0 , Q l l ′ ( p , ( p + l − l ′ ) P ) ( 0 ) = ( Q 0 ) l l ′ for all p . (21) R^{(p)}(0) = R_0,\qquad Q_{ll'}^{(p,(p+l-l')_P)}(0) = (Q_0)_{ll'}\qquad\text{for all }p. \tag*{(21)} R ( p ) ( 0 ) = R 0 , Q l l ′ ( p , ( p + l − l ′ ) P ) ( 0 ) = ( Q 0 ) l l ′ for all p . (21) Then, for all α ≥ 0 \alpha\ge0 α ≥ 0 p p p
R ( p ) ( P α ) = R ( α ) , Q l l ′ ( p , ( p + l − l ′ ) P ) ( P α ) = Q l l ′ ( α ) , ε g L C N ( P α ) = ε g C N N ( α ) . (22) R^{(p)}(P\alpha) = R(\alpha),\qquad Q_{ll'}^{(p,(p+l-l')_P)}(P\alpha) = Q_{ll'}(\alpha),\qquad\varepsilon_g^{\mathrm{LCN}}(P\alpha) = \varepsilon_g^{\mathrm{CNN}}(\alpha). \tag*{(22)} R ( p ) ( P α ) = R ( α ) , Q l l ′ ( p , ( p + l − l ′ ) P ) ( P α ) = Q l l ′ ( α ) , ε g LCN ( P α ) = ε g CNN ( α ) . (22) This result can be obtained by direct inspection of the ODEs (17) and (18) (see Appendix § 12 for details). Note that independent Gaussian initialization satisfies the position-symmetry condition up to fluctuations of order 1 / D 1/\sqrt{D} 1/ D (20) ).
Figure 2 a illustrates this result. As shown in the inset, the generalization error of the CNN decreases faster than that of the LCN. After rescaling the time of the LCN by a factor P P P P P P P P P P P P η \eta η
This result aligns with several previous works showing that invariant models can have a lower sample complexity than their non-invariant counterparts, by a factor of the order of the size of the associated symmetry group (here the cyclic group of translations of size P P P [25 , 43 ] ), and in more general settings for invariant classifiers ([40 ] ) and for invariant kernel and random-feature methods ([6 , 28 , 41 ] ). In contrast with these results, which take the form of bounds or asymptotic rates, Result 3.2 establishes an exact identity between the full training trajectories of the two students.
Interestingly, the exact rescaling holds only for uncorrelated positions. As shown in Figure 2 b, for c > 0 c > 0 c > 0 α \alpha α P α P\alpha P α
Benefits of locality
In this section, we study how locality affects the learning dynamics. For simplicity, we focus on the CNN student and investigate the role of the kernel size k s k_s k s k s = k t k_s = k_t k s = k t O D ( D ) O_D(D) O D ( D ) k s > k t k_s > k_t k s > k t O D ( D log D ) O_D(D\log D) O D ( D log D )
It is instructive to first enumerate the student configurations that exactly reproduce the teacher output, achieving zero generalization error. For k s , k t < P k_s, k_t < P k s , k t < P k s − k t + 1 k_s-k_t+1 k s − k t + 1
w ^ l = k s k t w l − s for s ≤ l < s + k t , w ^ l = 0 otherwise , (23) \widehat{w}_l = \sqrt{\frac{k_s}{k_t}}\,w_{l-s}\quad\text{for }s \le l < s+k_t,\qquad\widehat{w}_l = 0\quad\text{otherwise}, \tag*{(23)} w l = k t k s w l − s for s ≤ l < s + k t , w l = 0 otherwise , (23) labeled by the shift s ∈ { 0 , … , k s − k t } s \in\{0,\ldots,k_s-k_t\} s ∈ { 0 , … , k s − k t } k s = k t < P k_s = k_t < P k s = k t < P s s s
γ s = 1 Tr Q ∑ l = 0 k t − 1 R s + l , l . (24) \gamma_s = \frac{1}{\sqrt{\operatorname{Tr}Q}}\sum_{l=0}^{k_t-1} R_{s+l,l}. \tag*{(24)} γ s = Tr Q 1 l = 0 ∑ k t − 1 R s + l , l . (24) Note that when the student specializes to a particular solution s s s γ s = 1 \gamma_s = 1 γ s = 1 γ s ′ = 0 \gamma_{s'} = 0 γ s ′ = 0 s ′ ≠ s s' \ne s s ′ = s (17) do not depend explicitly on D D D α s p e c \alpha_{\mathrm{spec}} α spec D D D (19) .
Result 3.3 (Locality and specialization time, numerical). Let S = I S = I S = I σ = 0 \sigma= 0 σ = 0 η \eta η ( R , Q ) ( α ) (R, Q)(\alpha) ( R , Q ) ( α ) (17) from random initial conditions in dimension D D D γ t h < 1 \gamma_{\mathrm{th}} < 1 γ th < 1 γ t h = 0.99 \gamma_{\mathrm{th}} = 0.99 γ th = 0.99
α s p e c = inf { α ≥ 0 : max 0 ≤ s ≤ k s − k t γ s ( α ) ≥ γ t h } (25) \alpha_{\mathrm{spec}} = \inf\left\{ \alpha\ge0 : \max_{0 \le s \le k_s-k_t} \gamma_s(\alpha) \ge\gamma_{\mathrm{th}} \right\} \tag*{(25)} α spec = inf { α ≥ 0 : 0 ≤ s ≤ k s − k t max γ s ( α ) ≥ γ th } (25) with γ s \gamma_s γ s (24) . Then, integrating the ODEs numerically, we find that, as D → ∞ D \to\infty D → ∞
(i) for k s = k t k_s = k_t k s = k t α s p e c = O D ( 1 ) \alpha_{\mathrm{spec}} = \mathcal{O}_D(1) α spec = O D ( 1 )
(ii) for k t < k s ≤ P − 2 k_t < k_s \le P - 2 k t < k s ≤ P − 2 α s p e c = O D ( log D ) \alpha_{\mathrm{spec}} = \mathcal{O}_D(\log D) α spec = O D ( log D )
Since α = t / D \alpha= t/D α = t / D O D ( D ) \mathcal{O}_D(D) O D ( D ) O D ( D log D ) \mathcal{O}_D(D \log D) O D ( D log D )
Result 3.3 is obtained for g ( z ) = erf ( z / 2 ) g(z)=\operatorname{erf}(z/\sqrt{2}) g ( z ) = erf ( z / 2 ) [3 ] , that is, with a nonzero first Hermite coefficient, E [ z g ( z ) ] ≠ 0 \mathbb{E}[z\,g(z)]\ne0 E [ z g ( z )] = 0 z ∼ N ( 0 , 1 ) z\sim\mathcal{N}(0,1) z ∼ N ( 0 , 1 ) O D ( D ) O_D(D) O D ( D ) k s = k t k_s=k_t k s = k t [3 , 1 , 7 ] , and understanding the role of locality requires a more detailed analysis, which we leave for future work.
Figure 3 shows that for k s = k t k_s=k_t k s = k t α s p e c \alpha_{\mathrm{spec}} α spec D D D k s > k t k_s>k_t k s > k t α s p e c = O D ( log D ) \alpha_{\mathrm{spec}}=O_D(\log D) α spec = O D ( log D )
Figure 3. CNN students that match the teacher kernel size (k s = k t k_s = k_t k s = k t O D ( D ) \mathcal{O}_D(D) O D ( D ) k s > k t k_s > k_t k s > k t O D ( D log D ) \mathcal{O}_D(D \log D) O D ( D log D ) α \alpha α (17) from the order parameters of a random initialization at dimension D D D α s p e c \alpha_{\mathrm{spec}} α spec (25) , with γ t h = 0.99 \gamma_{\mathrm{th}} = 0.99 γ th = 0.99 D D D (17) from the order parameters of a random initialization at dimension D D D a + b ln D a + b \ln D a + b ln D D = 10 12 D = 10^{12} D = 1 0 12 g = erf g = \operatorname{erf} g = erf k t = 1 k_t = 1 k t = 1 P = 6 P = 6 P = 6 S = I S = I S = I η = 0.5 \eta= 0.5 η = 0.5 T = 1 T = 1 T = 1 σ = 0 \sigma= 0 σ = 0
ρ : R l l ′ ↦ R k s − 1 − l , k t − 1 − l ′ , Q l l ′ ↦ Q k s − 1 − l , k s − 1 − l ′ . (26) \rho:\ R_{ll'}\mapsto R_{k_s-1-l,\ k_t-1-l'},\qquad Q_{ll'}\mapsto Q_{k_s-1-l,\ k_s-1-l'}. \tag*{(26)} ρ : R l l ′ ↦ R k s − 1 − l , k t − 1 − l ′ , Q l l ′ ↦ Q k s − 1 − l , k s − 1 − l ′ . (26) In Appendix § 13 we show that the ODEs (17) preserve symmetry under ρ \rho ρ ( R 0 , Q 0 ) (R_0,Q_0) ( R 0 , Q 0 ) ρ \rho ρ ρ ( R 0 , Q 0 ) = ( R 0 , Q 0 ) \rho(R_0,Q_0)=(R_0,Q_0) ρ ( R 0 , Q 0 ) = ( R 0 , Q 0 ) ρ \rho ρ ρ ( R ( α ) , Q ( α ) ) = ( R ( α ) , Q ( α ) ) \rho(R(\alpha),Q(\alpha))=(R(\alpha),Q(\alpha)) ρ ( R ( α ) , Q ( α )) = ( R ( α ) , Q ( α )) α > 0 \alpha>0 α > 0 ρ \rho ρ D − 1 / 2 D^{-1/2} D − 1/2 (19) .
As shown in Figure 4 , we find that initially the student parameters have approximately equal overlap with different solutions, i.e., γ s ≈ γ \gamma_s\approx\gamma γ s ≈ γ s s s α ∼ O D ( 1 ) \alpha\sim O_D(1) α ∼ O D ( 1 ) ρ \rho ρ γ s = γ k s − k t − s \gamma_s=\gamma_{k_s-k_t-s} γ s = γ k s − k t − s R = 0 R=0 R = 0 Q = I / k s Q=I/k_s Q = I / k s ρ \rho ρ e κ α e^{\kappa\alpha} e κ α κ > 0 \kappa> 0 κ > 0 D − 1 / 2 D^{-1/2} D − 1/2 ρ \rho ρ log D \log D log D
Figure 4. Alignment γ s \gamma_s γ s s s s α \alpha α k s = 3 k_s=3 k s = 3 k s = 4 k_s=4 k s = 4 α s p e c \alpha_{\mathrm{spec}} α spec γ s = 0 \gamma_s=0 γ s = 0 p s p_s p s s s s (17) . Parameters: D = 10 12 D=10^{12} D = 1 0 12 g = erf g=\operatorname{erf} g = erf k t = 1 k_t=1 k t = 1 P = 6 P=6 P = 6 S = I S=I S = I η = 0.5 \eta=0.5 η = 0.5 T = 1 T=1 T = 1 σ = 0 \sigma=0 σ = 0
For k s = k t k_s = k_t k s = k t ρ \rho ρ α s p e c ∼ O D ( 1 ) \alpha_{\mathrm{spec}} \sim O_D(1) α spec ∼ O D ( 1 ) k s − k t k_s-k_t k s − k t s = ( k s − k t ) / 2 s=(k_s-k_t)/2 s = ( k s − k t ) /2 ρ \rho ρ D D D Figure 4 a). In Appendix § 14 we show that the plateau and the logarithmic growth of α s p e c \alpha_{\mathrm{spec}} α spec c > 0 c>0 c > 0 [36 , 5 ] , where the plateau corresponds to an unspecialized fixed point that is symmetric under permutations of the hidden units.
Robustness to spatial correlations
So far we have mostly focused on independent positions, c = 0 c=0 c = 0 c > 0 c>0 c > 0 § 4.1 ), the error left after long training (§ 4.2 ), and the error at finite training time when the learning rate of each student is optimized (§ 4.3 ). We find that, at fixed learning rate and for c > 0 c>0 c > 0
Early-time expansion
In the following result, we characterize the beginning of training by expanding analytically the ODEs around the random initial conditions in Eqs. (19) and (20) , similar to the analysis in Mori and Mignacco [32 ] .
Result 4.1 (Early-time expansion). Let S S S (3) , let k t ≤ k s ≤ P k_t \leq k_s \leq P k t ≤ k s ≤ P (19) and (20) at leading order in D D D α \alpha α
ε g C N N ( α ) = ε 0 C N N − ( η A C N N − η 2 B C N N ) α + O ( α 2 ) , ε g L C N ( α ) = ε 0 L C N − ( η A L C N − η 2 B L C N ) α + O ( α 2 ) . (27) \begin{aligned}
\varepsilon_g^{\mathrm{CNN}}(\alpha) &= \varepsilon_0^{\mathrm{CNN}}-(\eta A_{\mathrm{CNN}}-\eta^2 B_{\mathrm{CNN}})\alpha+O(\alpha^2),\\
\varepsilon_g^{\mathrm{LCN}}(\alpha) &= \varepsilon_0^{\mathrm{LCN}}-(\eta A_{\mathrm{LCN}}-\eta^2 B_{\mathrm{LCN}})\alpha+O(\alpha^2). \tag*{(27)}
\end{aligned} ε g CNN ( α ) ε g LCN ( α ) = ε 0 CNN − ( η A CNN − η 2 B CNN ) α + O ( α 2 ) , = ε 0 LCN − ( η A LCN − η 2 B LCN ) α + O ( α 2 ) . (27) with the initial errors
ε 0 L C N = 2 P π [ arcsin T 1 + T + ( P − 1 ) arcsin T c 1 + T ] + P 3 , ε 0 C N N = ε 0 L C N + 2 P ( P − 1 ) π arcsin c 2 . (28) \begin{aligned}
\varepsilon_0^{\mathrm{LCN}} &= \frac{2P}{\pi}\left[\arcsin\frac{T}{1+T}+(P-1)\arcsin\frac{Tc}{1+T}\right]+\frac{P}{3},\\
\varepsilon_0^{\mathrm{CNN}} &= \varepsilon_0^{\mathrm{LCN}}+\frac{2P(P-1)}{\pi}\arcsin\frac{c}{2}. \tag*{(28)}
\end{aligned} ε 0 LCN ε 0 CNN = π 2 P [ arcsin 1 + T T + ( P − 1 ) arcsin 1 + T T c ] + 3 P , = ε 0 LCN + π 2 P ( P − 1 ) arcsin 2 c . (28) The coefficients A C N N A_{\mathrm{CNN}} A CNN A L C N A_{\mathrm{LCN}} A LCN B C N N B_{\mathrm{CNN}} B CNN B L C N B_{\mathrm{LCN}} B LCN k t k_t k t § 15 .
First of all, at initialization the CNN starts from a larger error, ε 0 C N N > ε 0 L C N \varepsilon_0^{\mathrm{CNN}}>\varepsilon_0^{\mathrm{LCN}} ε 0 CNN > ε 0 LCN c > 0 c>0 c > 0 λ ^ p \hat{\lambda}_p λ ^ p E [ λ ^ p λ ^ q ] = S p , q \mathbb{E}[\hat{\lambda}_p\hat{\lambda}_q]=S_{p,q} E [ λ ^ p λ ^ q ] = S p , q g ( λ ^ p ) g(\hat{\lambda}_p) g ( λ ^ p ) E [ λ ^ p λ ^ q ] = δ p , q \mathbb{E}[\hat{\lambda}_p\hat{\lambda}_q]=\delta_{p,q} E [ λ ^ p λ ^ q ] = δ p , q g ( λ ^ p ) g(\hat{\lambda}_p) g ( λ ^ p )
Using Eq. (27) , we can also compare the performance of the two students at small but finite α \alpha α η C N N = A C N N / ( 2 B C N N ) \eta_{\mathrm{CNN}}=A_{\mathrm{CNN}}/(2B_{\mathrm{CNN}}) η CNN = A CNN / ( 2 B CNN ) η L C N = A L C N / ( 2 B L C N ) \eta_{\mathrm{LCN}}=A_{\mathrm{LCN}}/(2B_{\mathrm{LCN}}) η LCN = A LCN / ( 2 B LCN ) ε C N N ∗ ( α ) \varepsilon^*_{\mathrm{CNN}}(\alpha) ε CNN ∗ ( α ) ε L C N ∗ ( α ) \varepsilon^*_{\mathrm{LCN}}(\alpha) ε LCN ∗ ( α ) α \alpha α
{ ε C N N ∗ > ε L C N ∗ if c > c ∗ ( α ) , ε C N N ∗ < ε L C N ∗ if c < c ∗ ( α ) , (29) \begin{cases}
\varepsilon^*_{\mathrm{CNN}}>\varepsilon^*_{\mathrm{LCN}} & \text{if } c>c^*(\alpha),\\
\varepsilon^*_{\mathrm{CNN}}<\varepsilon^*_{\mathrm{LCN}} & \text{if } c<c^*(\alpha), \tag*{(29)}
\end{cases} { ε CNN ∗ > ε LCN ∗ ε CNN ∗ < ε LCN ∗ if c > c ∗ ( α ) , if c < c ∗ ( α ) , (29) where
c ∗ ( α ) = c 0 α + O ( α 2 ) , (30) c^*(\alpha)=c_0\alpha+O(\alpha^2), \tag*{(30)} c ∗ ( α ) = c 0 α + O ( α 2 ) , (30) and the slope c 0 c_0 c 0 P P P k s k_s k s T T T σ \sigma σ (119) of Appendix § 15 . In other words, the CNN achieves lower generalization error for small correlations c c c c c c α \alpha α Figure 6 ).
Figure 6. Comparison of the CNN and the LCN at training horizon α F \alpha_F α F c c c ϵ ∗ \epsilon^* ϵ ∗ ϵ C N N ∗ = ϵ L C N ∗ \epsilon^*_{\mathrm{CNN}} = \epsilon^*_{\mathrm{LCN}} ϵ CNN ∗ = ϵ LCN ∗ (30) of the early-time expansion. Parameters: g = erf g = \operatorname{erf} g = erf P = 4 P = 4 P = 4 k s = k t = 1 k_s = k_t = 1 k s = k t = 1 T = 1 T = 1 T = 1 σ = 1 \sigma= 1 σ = 1 D → ∞ D \to\infty D → ∞
Asymptotic generalization error
We now turn to long training times and ask how spatial correlations affect the asymptotic error in the presence of label noise, σ > 0 \sigma>0 σ > 0 η \eta η
Result 4.2 (Asymptotic generalization error). Let S S S c ∈ [ 0 , 1 ) c\in[0,1) c ∈ [ 0 , 1 ) σ > 0 \sigma>0 σ > 0 k t ≤ k s ≤ P k_t\le k_s\le P k t ≤ k s ≤ P
that is, of a zero-error solution of Eq. (23) . To leading order in η \eta η ε ∞ = lim α → ∞ ε g ( α ) \varepsilon^{\infty} = \lim_{\alpha\to\infty} \varepsilon_g(\alpha) ε ∞ = lim α → ∞ ε g ( α )
ε ∞ , C N N ( c ) = η σ 2 π [ P 1 + 2 T + P ( P − 1 ) c ( 1 + T ) 2 − T 2 c 2 ] + O ( η 2 ) , (31) \varepsilon^{\infty,\mathrm{CNN}}(c) = \frac{\eta\sigma^2}{\pi} \left[ \frac{P}{\sqrt{1+2T}} + \frac{P(P-1)c}{\sqrt{(1+T)^2-T^2c^2}} \right] + O(\eta^2), \tag*{(31)} ε ∞ , CNN ( c ) = π η σ 2 [ 1 + 2 T P + ( 1 + T ) 2 − T 2 c 2 P ( P − 1 ) c ] + O ( η 2 ) , (31) ε ∞ , L C N = η σ 2 P π 1 + 2 T + O ( η 2 ) . (32) \varepsilon^{\infty,\mathrm{LCN}} = \frac{\eta\sigma^2 P}{\pi\sqrt{1+2T}} + O(\eta^2). \tag*{(32)} ε ∞ , LCN = π 1 + 2 T η σ 2 P + O ( η 2 ) . (32) The derivation, given in Appendix § 16 , linearizes the ODEs around perfect recovery and locates the fixed point reached at small η \eta η k s k_s k s k t k_t k t Figure 5 compares them with the asymptotic errors located numerically from the ODEs at finite learning rate, finding excellent agreement. At c = 0 c = 0 c = 0 c > 0 c > 0 c > 0 c c c P P P c c c P P P
Figure 5. Asymptotic noise floor ε ∞ / ( η σ 2 ) \varepsilon^\infty/(\eta\sigma^2) ε ∞ / ( η σ 2 ) k s = k t = 2 k_s=k_t=2 k s = k t = 2 c c c P P P ε ∞ \varepsilon^\infty ε ∞ (31) and (32) and markers are the noise floors located numerically from the order-parameter ODEs at finite learning rate. (a) Floor vs. c c c P = 6 P=6 P = 6 c c c P P P c = 0 c=0 c = 0 P P P P P P c = 0.5 c=0.5 c = 0.5 P P P g = erf g=\operatorname{erf} g = erf k s = k t = 2 k_s=k_t=2 k s = k t = 2 η = 2 × 10 − 3 \eta=2\times10^{-3} η = 2 × 1 0 − 3 σ = 1 \sigma=1 σ = 1 T = 1 T=1 T = 1
To understand the origin of this difference, we consider, without loss of generality, a student close to the zero-error solution with s = 0 s = 0 s = 0 (23) , and define the error in the preactivation at position p p p
e p ≡ λ ^ p − λ p , (33) e_p \equiv\hat{\lambda}_p - \lambda_p, \tag*{(33)} e p ≡ λ ^ p − λ p , (33) which vanishes at this solution. Near the fixed point we find (Appendix § 16 )
E [ e p C N N e q C N N ] = η σ 2 2 S p , q + O ( η 2 ) , E [ e p L C N e q L C N ] = η σ 2 2 δ p , q + O ( η 2 ) . (34) \mathbb{E}\left[e_p^{\mathrm{CNN}}e_q^{\mathrm{CNN}}\right] = \frac{\eta\sigma^2}{2} S_{p,q} + O(\eta^2), \qquad\mathbb{E}\left[e_p^{\mathrm{LCN}}e_q^{\mathrm{LCN}}\right] = \frac{\eta\sigma^2}{2} \delta_{p,q} + O(\eta^2). \tag*{(34)} E [ e p CNN e q CNN ] = 2 η σ 2 S p , q + O ( η 2 ) , E [ e p LCN e q LCN ] = 2 η σ 2 δ p , q + O ( η 2 ) . (34) The errors of the LCN are uncorrelated across positions, because each position has its own kernel. The CNN instead uses the same kernel at every position, so its errors inherit the correlations of the data. As a result, for c > 0 c > 0 c > 0 § 4.3 , the CNN achieves a lower generalization error at long training horizons when the learning rate is optimized separately for each architecture.
Phase diagram at the optimal learning rate
While Result 4.2 compares the asymptotic performance of the two students at the same learning rate, it does not account for two effects. First, the two students will in general converge to perfect recovery at different rates. Moreover, their optimal learning rates can differ and depend on the training horizon. To include both effects, we use the ODEs (17) and (18) to compare the two students at a given correlation c c c α F \alpha_F α F η ∗ \eta^* η ∗ α F \alpha_F α F § 17 for details).
Figure 6 shows the resulting phase diagram, obtained by numerically integrating the ODEs. The LCN has lower error compared to the CNN for short training horizons and strong correlations. On the other hand, the CNN wins at long training horizons. We find the same transition for other values of P P P k s = k t k_s = k_t k s = k t T T T σ \sigma σ Figure 9 in Appendix § 17 ). Interestingly, the analytical prediction (30) , obtained by an early-time expansion, matches well the numerical boundary between the two regimes. Note that this does not contradict § 4.2 , where the LCN has the lower asymptotic error, because the two comparisons are made at different learning rates. Indeed, at a given α F \alpha_F α F c > 0 c > 0 c > 0 Figure 8 a in Appendix § 17 ). Since the asymptotic error is proportional to η \eta η η \eta η Figure 8 b).
Figure 8. (a) Optimal learning rate η ∗ \eta^* η ∗ ϵ ∗ \epsilon^* ϵ ∗ § 4.3 ) against the training horizon α F \alpha_F α F c c c Figure 6 . Parameters: g = erf g = \operatorname{erf} g = erf P = 4 P = 4 P = 4 k s = k t = 1 k_s = k_t = 1 k s = k t = 1 T = 1 T = 1 T = 1 σ = 1 \sigma= 1 σ = 1 D → ∞ D \to\infty D → ∞
Figure 9. Phase diagram of Figure 6 for other parameter values. The color shows log 10 ( ϵ C N N ∗ / ϵ L C N ∗ ) \log_{10}(\epsilon^{*}_{\mathrm{CNN}}/\epsilon^{*}_{\mathrm{LCN}}) log 10 ( ϵ CNN ∗ / ϵ LCN ∗ ) ϵ C N N ∗ = ϵ L C N ∗ \epsilon^{*}_{\mathrm{CNN}}=\epsilon^{*}_{\mathrm{LCN}} ϵ CNN ∗ = ϵ LCN ∗ c = c 0 α F c=c_0\alpha_F c = c 0 α F (30) . Each panel changes one parameter of Figure 6 : (a) P = 8 P=8 P = 8 k s = k t = 2 k_s=k_t=2 k s = k t = 2 T = 2 T=2 T = 2 σ = 2 \sigma=2 σ = 2 g = erf g=\operatorname{erf} g = erf P = 4 P=4 P = 4 k s = k t = 1 k_s=k_t=1 k s = k t = 1 T = 1 T=1 T = 1 σ = 1 \sigma=1 σ = 1 D → ∞ D\to\infty D → ∞
Conclusions
In this work, we have considered a teacher–student model to study the online SGD dynamics of convolutional and locally connected networks. In the limit of large feature dimension, we have shown that, for uncorrelated positions, weight sharing speeds up learning by exactly a factor equal to the number of positions. We have also investigated the role of locality, showing numerically that convolutional students that match the kernel size of the teacher recover it from O D ( D ) \mathcal{O}_D(D) O D ( D ) O D ( D log D ) \mathcal{O}_D(D \log D) O D ( D log D )
Our analysis relies on several simplifying assumptions, which point to natural extensions. For instance, our inputs are Gaussian and carry no localized structure, and one could consider data in which the signal is planted in one of the patches ([25 ] ). It would also be interesting to extend our results from regression to classification problems. Moreover, we have considered one-pass SGD, in which each sample is used only once. Training for multiple passes on a finite dataset, where overfitting and memorization become relevant, would require other tools, such as dynamical mean-field theory ([30 , 8 , 31 ] ).
Several interesting directions remain open. Our equations are valid for an arbitrary covariance S S S
f ⋆ ( x ) = ∑ p = 0 P − 1 g ( λ p ( x ) ) , λ p ( x ) = 1 D ( w S ⋅ x p + w O ⋅ ∑ q ≠ p x q ) (35) f^{\star}(x) = \sum_{p=0}^{P-1} g\left(\lambda_p(x)\right), \qquad\lambda_p(x) = \frac{1}{\sqrt{D}}\left(w_{\mathrm{S}} \cdot x_p + w_{\mathrm{O}} \cdot\sum_{q \ne p} x_q\right) \tag*{(35)} f ⋆ ( x ) = p = 0 ∑ P − 1 g ( λ p ( x ) ) , λ p ( x ) = D 1 w S ⋅ x p + w O ⋅ q = p ∑ x q (35) which is invariant under any permutation of the positions. Finally, we have considered constant learning rates. Learning-rate schedules could further reduce the error and could be designed with optimal control [29 ] .
Acknowledgments
FM was supported by the Center of Mathematical Sciences and Applications (CMSA) at Harvard University. MW was partially supported by NSF Award DMS-2406905, an Alfred P. Sloan Fellowship in Mathematics and the AI2050 program at Schmidt Sciences (Grant G-25-69786).
Statement on the use of AI
We used the generative AI model Claude Opus 5.5 (Anthropic) to assist with literature review, deriving and checking analytical expressions, writing the simulation code, and drafting and revising the text. All results were verified by the authors, who take full responsibility for the content of this work.
Code availability
Code to reproduce the figures is available at https://github.com/francescomori/CNN_theory.
ODE derivation: the CNN
We derive the deterministic ODE for the order parameters R R R Q Q Q D → ∞ D \to\infty D → ∞ S S S k s k_s k s
Setup and building blocks
The student weights are updated by online SGD:
Δ w ^ l = − η ∂ w ^ l ℓ ( x ; θ ) = η δ k s D ∑ p = 0 P − 1 g ′ ( λ ^ p ) x ( p + l ) P , δ = y − y ^ C N N , (36) \Delta\hat{w}_l=-\eta\partial_{\hat{w}_l}\ell(x;\theta)=\frac{\eta\delta}{\sqrt{k_sD}}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)x_{(p+l)P},\qquad\delta=y-\hat{y}_{\mathrm{CNN}}, \tag*{(36)} Δ w ^ l = − η ∂ w ^ l ℓ ( x ; θ ) = k s D η δ p = 0 ∑ P − 1 g ′ ( λ ^ p ) x ( p + l ) P , δ = y − y ^ CNN , (36) where the sum over p p p y ^ C N N \hat{y}_{\mathrm{CNN}} y ^ CNN w ^ l \hat{w}_l w ^ l P P P g ( λ ^ p ) g(\hat{\lambda}_p) g ( λ ^ p )
For simplicity, in this appendix and in Appendices B and H we pad the teacher kernel with zero vectors, w l = 0 w_l=0 w l = 0 k t ≤ l < k s k_t\le l<k_s k t ≤ l < k s R R R k s × k t k_s\times k_t k s × k t (15) , and the teacher overlap matrix T l , l ′ T_{l,l'} T l , l ′ k s × k s k_s\times k_s k s × k s { 0 , … , k s − 1 } \{0,\ldots,k_s-1\} { 0 , … , k s − 1 }
Single-vector fields. It is convenient to introduce projections of the input at one position onto one weight vector,
μ r ( l ) ≡ w l ⋅ x r k t D , ν r ( l ) ≡ w ^ l ⋅ x r k s D , (37) \mu_r^{(l)}\equiv\frac{w_l\cdot x_r}{\sqrt{k_tD}},\qquad\nu_r^{(l)}\equiv\frac{\hat{w}_l\cdot x_r}{\sqrt{k_sD}}, \tag*{(37)} μ r ( l ) ≡ k t D w l ⋅ x r , ν r ( l ) ≡ k s D w ^ l ⋅ x r , (37) with which the preactivations decompose simply: λ p = ∑ l = 0 k s − 1 μ ( p + l ) P ( l ) \lambda_p=\sum_{l=0}^{k_s-1}\mu_{(p+l)P}^{(l)} λ p = ∑ l = 0 k s − 1 μ ( p + l ) P ( l ) λ ^ p = ∑ l = 0 k s − 1 ν ( p + l ) P ( l ) \hat{\lambda}_p=\sum_{l=0}^{k_s-1}\nu_{(p+l)P}^{(l)} λ ^ p = ∑ l = 0 k s − 1 ν ( p + l ) P ( l ) { 0 , … , k s − 1 } \{0,\ldots,k_s-1\} { 0 , … , k s − 1 }
⟨ μ r ( l ) μ q ( l ′ ) ⟩ = S r , q T l , l ′ , ⟨ ν r ( l ) ν q ( l ′ ) ⟩ = S r , q Q l , l ′ , ⟨ μ r ( l ) ν q ( l ′ ) ⟩ = S r , q R l , l ′ , (38) \left\langle\mu_r^{(l)}\mu_q^{(l')}\right\rangle=S_{r,q}T_{l,l'},\qquad\left\langle\nu_r^{(l)}\nu_q^{(l')}\right\rangle=S_{r,q}Q_{l,l'},\qquad\left\langle\mu_r^{(l)}\nu_q^{(l')}\right\rangle=S_{r,q}R_{l,l'}, \tag*{(38)} ⟨ μ r ( l ) μ q ( l ′ ) ⟩ = S r , q T l , l ′ , ⟨ ν r ( l ) ν q ( l ′ ) ⟩ = S r , q Q l , l ′ , ⟨ μ r ( l ) ν q ( l ′ ) ⟩ = S r , q R l , l ′ , (38) where the index ordering in the cross term reflects the convention R l , l ′ = w ^ l ⋅ w l ′ / k s k t D R_{l,l'}=\hat{w}_l\cdot w_{l'}/\sqrt{k_sk_tD} R l , l ′ = w ^ l ⋅ w l ′ / k s k t D T l , l ′ = w l ⋅ w l ′ / ( k t D ) T_{l,l'}=w_l\cdot w_{l'}/(k_tD) T l , l ′ = w l ⋅ w l ′ / ( k t D ) ( T / k t ) δ l , l ′ (T/k_t)\delta_{l,l'} ( T / k t ) δ l , l ′ l , l ′ < k t l,l'<k_t l , l ′ < k t (7) and zero otherwise. Expanding one of the two factors with λ p = ∑ l = 0 k s − 1 μ ( p + l ) P ( l ) \lambda_p=\sum_{l=0}^{k_s-1}\mu_{(p+l)P}^{(l)} λ p = ∑ l = 0 k s − 1 μ ( p + l ) P ( l ) λ ^ p = ∑ l = 0 k s − 1 ν ( p + l ) P ( l ) \hat{\lambda}_p=\sum_{l=0}^{k_s-1}\nu_{(p+l)P}^{(l)} λ ^ p = ∑ l = 0 k s − 1 ν ( p + l ) P ( l )
⟨ λ ^ p μ r ( l ′ ) ⟩ = ∑ l = 0 k s − 1 R l , l ′ S ( p + l ) P , r , ⟨ λ ^ p ν r ( l ′ ) ⟩ = ∑ l = 0 k s − 1 Q l , l ′ S ( p + l ) P , r , (39) \left\langle\hat{\lambda}_p\mu_r^{(l')}\right\rangle=\sum_{l=0}^{k_s-1}R_{l,l'}S_{(p+l)P,r},\qquad\left\langle\hat{\lambda}_p\nu_r^{(l')}\right\rangle=\sum_{l=0}^{k_s-1}Q_{l,l'}S_{(p+l)P,r}, \tag*{(39)} ⟨ λ ^ p μ r ( l ′ ) ⟩ = l = 0 ∑ k s − 1 R l , l ′ S ( p + l ) P , r , ⟨ λ ^ p ν r ( l ′ ) ⟩ = l = 0 ∑ k s − 1 Q l , l ′ S ( p + l ) P , r , (39) ⟨ λ q μ r ( l ′ ) ⟩ = ∑ l = 0 k s − 1 T l , l ′ S ( q + l ) P , r , ⟨ λ q ν r ( l ′ ) ⟩ = ∑ l = 0 k s − 1 R l , l ′ S ( q + l ) P , r . (40) \left\langle\lambda_q\mu_r^{(l')}\right\rangle=\sum_{l=0}^{k_s-1}T_{l,l'}S_{(q+l)P,r},\qquad\left\langle\lambda_q\nu_r^{(l')}\right\rangle=\sum_{l=0}^{k_s-1}R_{l,l'}S_{(q+l)P,r}. \tag*{(40)} ⟨ λ q μ r ( l ′ ) ⟩ = l = 0 ∑ k s − 1 T l , l ′ S ( q + l ) P , r , ⟨ λ q ν r ( l ′ ) ⟩ = l = 0 ∑ k s − 1 R l , l ′ S ( q + l ) P , r . (40) Effective covariances. The preactivations are linear in x x x (6) and using ⟨ x p , a x q , b ⟩ = S p , q δ a , b \langle x_{p,a}x_{q,b}\rangle=S_{p,q}\delta_{a,b} ⟨ x p , a x q , b ⟩ = S p , q δ a , b
⟨ λ p λ q ⟩ = 1 k t D ∑ l , l ′ = 0 k s − 1 ( w l ⋅ w l ′ ) S ( p + l ) P , ( q + l ′ ) P = ∑ l , l ′ = 0 k s − 1 T l , l ′ S ( p + l ) P , ( q + l ′ ) P . (41) \left\langle\lambda_p\lambda_q\right\rangle=\frac{1}{k_tD}\sum_{l,l'=0}^{k_s-1}(w_l\cdot w_{l'})S_{(p+l)P,(q+l')P}=\sum_{l,l'=0}^{k_s-1}T_{l,l'}S_{(p+l)P,(q+l')P}. \tag*{(41)} ⟨ λ p λ q ⟩ = k t D 1 l , l ′ = 0 ∑ k s − 1 ( w l ⋅ w l ′ ) S ( p + l ) P , ( q + l ′ ) P = l , l ′ = 0 ∑ k s − 1 T l , l ′ S ( p + l ) P , ( q + l ′ ) P . (41) The same computation with one or both teacher weight vectors replaced by student ones gives the remaining covariances, with T T T R R R Q Q Q k s × k s k_s \times k_s k s × k s M M M
M p , q e f f ≡ ∑ l , l ′ = 0 k s − 1 M l , l ′ S ( p + l ) P , ( q + l ′ ) P , (42) M^{\mathrm{eff}}_{p,q} \equiv\sum_{l,l'=0}^{k_s-1} M_{l,l'} S_{(p+l)P,(q+l')P}, \tag*{(42)} M p , q eff ≡ l , l ′ = 0 ∑ k s − 1 M l , l ′ S ( p + l ) P , ( q + l ′ ) P , (42) a P × P P \times P P × P
⟨ λ p λ q ⟩ = T p , q e f f , ⟨ λ ^ p λ ^ q ⟩ = Q p , q e f f , ⟨ λ ^ p λ q ⟩ = R p , q e f f . (43) \langle\lambda_p \lambda_q \rangle= T^{\mathrm{eff}}_{p,q}, \qquad\langle\hat{\lambda}_p \hat{\lambda}_q \rangle= Q^{\mathrm{eff}}_{p,q}, \qquad\langle\hat{\lambda}_p \lambda_q \rangle= R^{\mathrm{eff}}_{p,q}. \tag*{(43)} ⟨ λ p λ q ⟩ = T p , q eff , ⟨ λ ^ p λ ^ q ⟩ = Q p , q eff , ⟨ λ ^ p λ q ⟩ = R p , q eff . (43) Saad–Solla integrals
The ODEs derived below involve the activation only through a small set of averages of products of g g g g ′ g' g ′ g = erf g = \operatorname{erf} g = erf 2 × 2 2 \times2 2 × 2 C ( 2 ) = ( c p , q ) C^{(2)} = (c_{p,q}) C ( 2 ) = ( c p , q )
I 2 ( C ( 2 ) ) = E [ g ( u 1 ) g ( u 2 ) ] = 2 π arcsin c 12 ( 1 + c 11 ) ( 1 + c 22 ) , (44) I_2(C^{(2)}) = \mathbb{E}[g(u_1)g(u_2)] = \frac{2}{\pi}\arcsin\frac{c_{12}}{\sqrt{(1+c_{11})(1+c_{22})}}, \tag*{(44)} I 2 ( C ( 2 ) ) = E [ g ( u 1 ) g ( u 2 )] = π 2 arcsin ( 1 + c 11 ) ( 1 + c 22 ) c 12 , (44) where C ( 2 ) C^{(2)} C ( 2 ) ( u 1 , u 2 ) (u_1,u_2) ( u 1 , u 2 )
J 2 ( C ( 2 ) ) = E [ g ′ ( u 1 ) g ′ ( u 2 ) ] = 2 π [ ( 1 + c 11 ) ( 1 + c 22 ) − c 12 2 ] − 1 / 2 , (45) J_2(C^{(2)}) = \mathbb{E}[g'(u_1)g'(u_2)] = \frac{2}{\pi}\left[(1+c_{11})(1+c_{22})-c_{12}^{2}\right]^{-1/2}, \tag*{(45)} J 2 ( C ( 2 ) ) = E [ g ′ ( u 1 ) g ′ ( u 2 )] = π 2 [ ( 1 + c 11 ) ( 1 + c 22 ) − c 12 2 ] − 1/2 , (45) I 3 ( C ( 3 ) ) = E [ g ′ ( u 1 ) u 2 g ( u 3 ) ] = 2 π c 23 ( 1 + c 11 ) − c 12 c 13 ( 1 + c 11 ) ( 1 + c 11 ) ( 1 + c 33 ) − c 13 2 , (46) I_3(C^{(3)}) = \mathbb{E}[g'(u_1)u_2g(u_3)] = \frac{2}{\pi}\frac{c_{23}(1+c_{11})-c_{12}c_{13}}{(1+c_{11})\sqrt{(1+c_{11})(1+c_{33})-c_{13}^{2}}}, \tag*{(46)} I 3 ( C ( 3 ) ) = E [ g ′ ( u 1 ) u 2 g ( u 3 )] = π 2 ( 1 + c 11 ) ( 1 + c 11 ) ( 1 + c 33 ) − c 13 2 c 23 ( 1 + c 11 ) − c 12 c 13 , (46) with the slot convention ( u 1 , u 2 , u 3 ) = ( g ′ , linear , g ) (u_1,u_2,u_3) = (g',\text{linear},g) ( u 1 , u 2 , u 3 ) = ( g ′ , linear , g ) I 3 I_3 I 3 I 4 I_4 I 4
I 4 ( C ( 4 ) ) = E [ g ′ ( u 1 ) g ′ ( u 2 ) g ( u 3 ) g ( u 4 ) ] = 4 π 2 1 Λ 4 arcsin Λ 0 Λ 1 Λ 2 , (47) I_4(C^{(4)}) = \mathbb{E}[g'(u_1)g'(u_2)g(u_3)g(u_4)] = \frac{4}{\pi^2}\frac{1}{\sqrt{\Lambda_4}}\arcsin\frac{\Lambda_0}{\sqrt{\Lambda_1\Lambda_2}}, \tag*{(47)} I 4 ( C ( 4 ) ) = E [ g ′ ( u 1 ) g ′ ( u 2 ) g ( u 3 ) g ( u 4 )] = π 2 4 Λ 4 1 arcsin Λ 1 Λ 2 Λ 0 , (47) where
Λ 4 = ( 1 + c 11 ) ( 1 + c 22 ) − c 12 2 , (48) \Lambda_4 = (1+c_{11})(1+c_{22})-c_{12}^{2}, \tag*{(48)} Λ 4 = ( 1 + c 11 ) ( 1 + c 22 ) − c 12 2 , (48) Λ 1 = Λ 4 ( 1 + c 33 ) − c 23 2 ( 1 + c 11 ) − c 13 2 ( 1 + c 22 ) + 2 c 12 c 13 c 23 , (49) \Lambda_1 = \Lambda_4(1+c_{33})-c_{23}^{2}(1+c_{11})-c_{13}^{2}(1+c_{22})+2c_{12}c_{13}c_{23}, \tag*{(49)} Λ 1 = Λ 4 ( 1 + c 33 ) − c 23 2 ( 1 + c 11 ) − c 13 2 ( 1 + c 22 ) + 2 c 12 c 13 c 23 , (49) Λ 2 = Λ 4 ( 1 + c 44 ) − c 24 2 ( 1 + c 11 ) − c 14 2 ( 1 + c 22 ) + 2 c 12 c 14 c 24 , (50) \Lambda_2 = \Lambda_4(1+c_{44})-c_{24}^{2}(1+c_{11})-c_{14}^{2}(1+c_{22})+2c_{12}c_{14}c_{24}, \tag*{(50)} Λ 2 = Λ 4 ( 1 + c 44 ) − c 24 2 ( 1 + c 11 ) − c 14 2 ( 1 + c 22 ) + 2 c 12 c 14 c 24 , (50) Λ 0 = Λ 4 c 34 − c 23 c 24 ( 1 + c 11 ) − c 13 c 14 ( 1 + c 22 ) + c 12 c 13 c 24 + c 12 c 14 c 23 . (51) \Lambda_0 = \Lambda_4c_{34}-c_{23}c_{24}(1+c_{11})-c_{13}c_{14}(1+c_{22})+c_{12}c_{13}c_{24}+c_{12}c_{14}c_{23}. \tag*{(51)} Λ 0 = Λ 4 c 34 − c 23 c 24 ( 1 + c 11 ) − c 13 c 14 ( 1 + c 22 ) + c 12 c 13 c 24 + c 12 c 14 c 23 . (51) Below we write these integrals with their field arguments rather than with the covariance matrix, as in J 2 ( λ ^ p , λ ^ q ) J_2(\hat{\lambda}_p,\hat{\lambda}_q) J 2 ( λ ^ p , λ ^ q ) I 3 ( λ ^ p , μ r ( l ′ ) , λ q ) I_3(\hat{\lambda}_p,\mu_r^{(l')},\lambda_q) I 3 ( λ ^ p , μ r ( l ′ ) , λ q ) § 9 .§ 1 .
ODE for R R R
We compute Δ R l , l ′ = ( Δ w ^ l ⋅ w l ′ ) / ( k s k t D ) \Delta R_{l,l'} = (\Delta\hat{w}_l \cdot w_{l'})/(\sqrt{k_s k_t}D) Δ R l , l ′ = ( Δ w ^ l ⋅ w l ′ ) / ( k s k t D )
Δ R l , l ′ = η δ k s k t D D ∑ p = 0 P − 1 g ′ ( λ ^ p ) ( x ( p + l ) P ⋅ w l ′ ) = η δ k s D ∑ p = 0 P − 1 g ′ ( λ ^ p ) μ ( p + l ) P ( l ′ ) . (52) \Delta R_{l,l'} = \frac{\eta\delta}{k_s\sqrt{k_t}D\sqrt{D}}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)(x_{(p+l)P}\cdot w_{l'}) = \frac{\eta\delta}{k_sD}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)\mu_{(p+l)P}^{(l')}. \tag*{(52)} Δ R l , l ′ = k s k t D D η δ p = 0 ∑ P − 1 g ′ ( λ ^ p ) ( x ( p + l ) P ⋅ w l ′ ) = k s D η δ p = 0 ∑ P − 1 g ′ ( λ ^ p ) μ ( p + l ) P ( l ′ ) . (52) Taking the continuum limit with α = t / D \alpha= t/D α = t / D
R ˙ l , l ′ = η k s ∑ p = 0 P − 1 ⟨ δ g ′ ( λ ^ p ) μ ( p + l ) P ( l ′ ) ⟩ . (53) \dot{R}_{l,l'} = \frac{\eta}{k_s}\sum_{p=0}^{P-1}\left\langle\delta g'(\hat{\lambda}_p)\mu^{(l')}_{(p+l)P}\right\rangle. \tag*{(53)} R ˙ l , l ′ = k s η p = 0 ∑ P − 1 ⟨ δ g ′ ( λ ^ p ) μ ( p + l ) P ( l ′ ) ⟩ . (53) Expanding δ = ∑ q = 0 P − 1 g ( λ q ) + σ ξ − ∑ r = 0 P − 1 g ( λ ^ r ) \delta= \sum_{q=0}^{P-1}g(\lambda_q)+\sigma\xi-\sum_{r=0}^{P-1}g(\hat{\lambda}_r) δ = ∑ q = 0 P − 1 g ( λ q ) + σ ξ − ∑ r = 0 P − 1 g ( λ ^ r ) ⟨ ξ ⟩ = 0 \langle\xi\rangle=0 ⟨ ξ ⟩ = 0
R ˙ l , l ′ = η k s ∑ p , q = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − η k s ∑ p , q = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) . (54) \dot{R}_{l,l'} = \frac{\eta}{k_s}\sum_{p,q=0}^{P-1} I_3(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\lambda_q)-\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}I_3(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\hat{\lambda}_q). \tag*{(54)} R ˙ l , l ′ = k s η p , q = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − k s η p , q = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) . (54) ODE for Q Q Q
Since Q l , l ′ = w ^ l ⋅ w ^ l ′ / ( k s D ) Q_{l,l'}=\hat{w}_l\cdot\hat{w}_{l'}/(k_sD) Q l , l ′ = w ^ l ⋅ w ^ l ′ / ( k s D )
Δ Q l , l ′ = Δ w ^ l ⋅ w ^ l ′ k s D + w ^ l ⋅ Δ w ^ l ′ k s D + Δ w ^ l ⋅ Δ w ^ l ′ k s D , (55) \Delta Q_{l,l'}=\frac{\Delta\hat{w}_l\cdot\hat{w}_{l'}}{k_sD}+\frac{\hat{w}_l\cdot\Delta\hat{w}_{l'}}{k_sD}+\frac{\Delta\hat{w}_l\cdot\Delta\hat{w}_{l'}}{k_sD}, \tag*{(55)} Δ Q l , l ′ = k s D Δ w ^ l ⋅ w ^ l ′ + k s D w ^ l ⋅ Δ w ^ l ′ + k s D Δ w ^ l ⋅ Δ w ^ l ′ , (55) two cross terms of order η \eta η η 2 \eta^2 η 2
Each cross term repeats the calculation of the previous subsection with the teacher vector w l ′ w_{l'} w l ′ w ^ l ′ \hat{w}_{l'} w ^ l ′ μ ( l ′ ) \mu^{(l')} μ ( l ′ ) ν ( l ′ ) \nu^{(l')} ν ( l ′ )
Δ w ^ l ⋅ w ^ l ′ k s D = η δ k s D ∑ p = 0 P − 1 g ′ ( λ ^ p ) ν ( p + l ) P ( l ′ ) , w ^ l ⋅ Δ w ^ l ′ k s D = η δ k s D ∑ p = 0 P − 1 g ′ ( λ ^ p ) ν ( p + l ′ ) P ( l ) . (56) \frac{\Delta\hat{w}_l\cdot\hat{w}_{l'}}{k_sD}=\frac{\eta\delta}{k_sD}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)\nu^{(l')}_{(p+l)P},\qquad
\frac{\hat{w}_l\cdot\Delta\hat{w}_{l'}}{k_sD}=\frac{\eta\delta}{k_sD}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)\nu^{(l)}_{(p+l')P}. \tag*{(56)} k s D Δ w ^ l ⋅ w ^ l ′ = k s D η δ p = 0 ∑ P − 1 g ′ ( λ ^ p ) ν ( p + l ) P ( l ′ ) , k s D w ^ l ⋅ Δ w ^ l ′ = k s D η δ p = 0 ∑ P − 1 g ′ ( λ ^ p ) ν ( p + l ′ ) P ( l ) . (56) Expanding δ \delta δ
Q ˙ l , l ′ ∣ O ( η ) = η k s ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + η k s ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ ^ q ) ] . (57) \begin{aligned}
\left.\dot{Q}_{l,l'}\right|_{O(\eta)}=\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}\left[I_3(\hat{\lambda}_p,\nu^{(l')}_{(p+l)P},\lambda_q)-I_3(\hat{\lambda}_p,\nu^{(l')}_{(p+l)P},\hat{\lambda}_q)\right] \\
&+\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}\left[I_3(\hat{\lambda}_p,\nu^{(l)}_{(p+l')P},\lambda_q)-I_3(\hat{\lambda}_p,\nu^{(l)}_{(p+l')P},\hat{\lambda}_q)\right]. \tag*{(57)}
\end{aligned} Q ˙ l , l ′ O ( η ) = k s η p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + k s η p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ ^ q ) ] . (57) The remaining piece is
Δ w ^ l ⋅ Δ w ^ l ′ k s D = η 2 δ 2 k s 2 D 2 ∑ p , q = 0 P − 1 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ( x ( p + l ) P ⋅ x ( q + l ′ ) P ) . (58) \frac{\Delta\hat{w}_l\cdot\Delta\hat{w}_{l'}}{k_sD}=\frac{\eta^2\delta^2}{k_s^2D^2}\sum_{p,q=0}^{P-1}g'(\hat{\lambda}_p)g'(\hat{\lambda}_q)(x_{(p+l)P}\cdot x_{(q+l')P}). \tag*{(58)} k s D Δ w ^ l ⋅ Δ w ^ l ′ = k s 2 D 2 η 2 δ 2 p , q = 0 ∑ P − 1 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ( x ( p + l ) P ⋅ x ( q + l ′ ) P ) . (58) The inner product x r ⋅ x q x_r\cdot x_q x r ⋅ x q D D D D D D D S r , q DS_{r,q} D S r , q D \sqrt{D} D D D D
⟨ ( x r ⋅ x q ) f ( x ) ⟩ = D S r , q ⟨ f ( x ) ⟩ + O D ( D ) . (59) \left\langle(x_r\cdot x_q)f(x)\right\rangle=DS_{r,q}\langle f(x)\rangle+\mathcal{O}_D(\sqrt{D}). \tag*{(59)} ⟨ ( x r ⋅ x q ) f ( x ) ⟩ = D S r , q ⟨ f ( x )⟩ + O D ( D ) . (59) Applying this with f ( x ) = δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) f(x)=\delta^2g'(\hat{\lambda}_p)g'(\hat{\lambda}_q) f ( x ) = δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q )
⟨ Δ Q l , l ′ ⟩ ∣ O ( η 2 ) = η 2 k s 2 D ∑ p , q = 0 P − 1 S ( p + l ) P , ( q + l ′ ) P ⟨ δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ . (60) \left.\left\langle\Delta Q_{l,l'}\right\rangle\right|_{O(\eta^2)}=\frac{\eta^2}{k_s^2D}\sum_{p,q=0}^{P-1}S_{(p+l)P,(q+l')P}\left\langle\delta^2g'(\hat{\lambda}_p)g'(\hat{\lambda}_q)\right\rangle. \tag*{(60)} ⟨ Δ Q l , l ′ ⟩ ∣ O ( η 2 ) = k s 2 D η 2 p , q = 0 ∑ P − 1 S ( p + l ) P , ( q + l ′ ) P ⟨ δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ . (60) Splitting the residual δ \delta δ δ = Δ y + σ ξ \delta= \Delta y + \sigma\xi δ = Δ y + σ ξ Δ y = ∑ r = 0 P − 1 g ( λ r ) − ∑ r = 0 P − 1 g ( λ ^ r ) \Delta y = \sum_{r=0}^{P-1} g(\lambda_r) - \sum_{r=0}^{P-1} g(\hat{\lambda}_r) Δ y = ∑ r = 0 P − 1 g ( λ r ) − ∑ r = 0 P − 1 g ( λ ^ r ) ξ \xi ξ x x x
⟨ δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ = ⟨ ( Δ y ) 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ + σ 2 J 2 ( λ ^ p , λ ^ q ) . (61) \left\langle\delta^2 g'(\hat{\lambda}_p)g'(\hat{\lambda}_q) \right\rangle= \left\langle(\Delta y)^2 g'(\hat{\lambda}_p)g'(\hat{\lambda}_q) \right\rangle+ \sigma^2 J_2(\hat{\lambda}_p,\hat{\lambda}_q). \tag*{(61)} ⟨ δ 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ = ⟨ ( Δ y ) 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ + σ 2 J 2 ( λ ^ p , λ ^ q ) . (61) Expanding the square, we find
( Δ y ) 2 = ∑ r , s = 0 P − 1 [ g ( λ r ) g ( λ s ) − 2 g ( λ r ) g ( λ ^ s ) + g ( λ ^ r ) g ( λ ^ s ) ] , (62) (\Delta y)^2 = \sum_{r,s=0}^{P-1} \left[g(\lambda_r)g(\lambda_s) - 2g(\lambda_r)g(\hat{\lambda}_s) + g(\hat{\lambda}_r)g(\hat{\lambda}_s)\right], \tag*{(62)} ( Δ y ) 2 = r , s = 0 ∑ P − 1 [ g ( λ r ) g ( λ s ) − 2 g ( λ r ) g ( λ ^ s ) + g ( λ ^ r ) g ( λ ^ s ) ] , (62) and therefore
⟨ ( Δ y ) 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ = ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] , (63) \left\langle(\Delta y)^2 g'(\hat{\lambda}_p)g'(\hat{\lambda}_q) \right\rangle= \sum_{r,s=0}^{P-1} \left[I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\lambda_s) - 2I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\hat{\lambda}_s)
+ I_4(\hat{\lambda}_p,\hat{\lambda}_q,\hat{\lambda}_r,\hat{\lambda}_s)\right], \tag*{(63)} ⟨ ( Δ y ) 2 g ′ ( λ ^ p ) g ′ ( λ ^ q ) ⟩ = r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] , (63) with the slot convention ( g ′ , g ′ , g , g ) (g',g',g,g) ( g ′ , g ′ , g , g ) 3 × 3 3 \times3 3 × 3 4 × 4 4 \times4 4 × 4 § 9 .§ 1 .
Finally, the equation for Q Q Q
Q ˙ l , l ′ = η k s ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + η k s ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ ^ q ) ] + η 2 k s 2 ∑ p , q = 0 P − 1 S ( p + l ) P , ( q + l ′ ) P { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (64) \begin{aligned}
\dot{Q}_{l,l'} ={}& \frac{\eta}{k_s}\sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_p,\nu_{(p+l)_P}^{(l')},\lambda_q\right)-I_3\left(\hat{\lambda}_p,\nu_{(p+l)_P}^{(l')},\hat{\lambda}_q\right)\right]\\
&+\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_p,\nu_{(p+l')_P}^{(l)},\lambda_q\right)-I_3\left(\hat{\lambda}_p,\nu_{(p+l')_P}^{(l)},\hat{\lambda}_q\right)\right]\\
&+\frac{\eta^2}{k_s^2}\sum_{p,q=0}^{P-1}S_{(p+l)_P,(q+l')_P}\left\{\sum_{r,s=0}^{P-1}\left[I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\lambda_s)
-2I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\hat{\lambda}_s)\right.\right.\\
&\left.\left.\qquad\qquad\qquad\qquad\qquad\qquad+I_4(\hat{\lambda}_p,\hat{\lambda}_q,\hat{\lambda}_r,\hat{\lambda}_s)\right]+\sigma^2J_2(\hat{\lambda}_p,\hat{\lambda}_q)\right\}. \tag*{(64)}
\end{aligned} Q ˙ l , l ′ = k s η p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + k s η p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ′ ) P ( l ) , λ ^ q ) ] + k s 2 η 2 p , q = 0 ∑ P − 1 S ( p + l ) P , ( q + l ′ ) P { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (64) with all position sums running over { 0 , … , P − 1 } \{0,\ldots,P-1\} { 0 , … , P − 1 }
Together with the equation for R R R (17) , with
[ F R C N N ] l , l ′ = ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] , (65) \left[F_R^{\mathrm{CNN}}\right]_{l,l'} = \sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_p,\mu_{(p+l)_P}^{(l')},\lambda_q\right)-I_3\left(\hat{\lambda}_p,\mu_{(p+l)_P}^{(l')},\hat{\lambda}_q\right)\right], \tag*{(65)} [ F R CNN ] l , l ′ = p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] , (65) [ F Q C N N ] l , l ′ = ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + ( l ↔ l ′ ) , (66) \left[F_Q^{\mathrm{CNN}}\right]_{l,l'} = \sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_p,\nu_{(p+l)_P}^{(l')},\lambda_q\right)-I_3\left(\hat{\lambda}_p,\nu_{(p+l)_P}^{(l')},\hat{\lambda}_q\right)\right] + (l \leftrightarrow l'), \tag*{(66)} [ F Q CNN ] l , l ′ = p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , ν ( p + l ) P ( l ′ ) , λ ^ q ) ] + ( l ↔ l ′ ) , (66) [ G Q C N N ] l , l ′ = ∑ p , q = 0 P − 1 S ( p + l ) P , ( q + l ′ ) P { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (67) \left[G_Q^{\mathrm{CNN}}\right]_{l,l'} = \sum_{p,q=0}^{P-1}S_{(p+l)_P,(q+l')_P}\left\{\sum_{r,s=0}^{P-1}\left[I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\lambda_s)-2I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\hat{\lambda}_s)+I_4(\hat{\lambda}_p,\hat{\lambda}_q,\hat{\lambda}_r,\hat{\lambda}_s)\right]+\sigma^2J_2(\hat{\lambda}_p,\hat{\lambda}_q)\right\}. \tag*{(67)} [ G Q CNN ] l , l ′ = p , q = 0 ∑ P − 1 S ( p + l ) P , ( q + l ′ ) P { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (67) These functions depend on R R R Q Q Q T T T S S S § 9 .§ 1 .
ODE derivation: the LCN
Throughout this appendix λ ^ \hat{\lambda} λ ^ § 9 but with two key differences: (i) each per-position kernel w ^ ( p ) \hat{w}^{(p)} w ^ ( p ) p p p Q ( p , q ) Q^{(p,q)} Q ( p , q ) p p p q q q
Setup
As in Appendix § 9 , we pad the teacher kernel with zero vectors, so that each R ( p ) R^{(p)} R ( p ) k s × k s k_s \times k_s k s × k s
The SGD update for the p p p
Δ w ^ l ( p ) = η δ k s D g ′ ( λ ^ p ) x ( p + l ) P . (68) \Delta\hat{w}^{(p)}_l = \frac{\eta\, \delta}{\sqrt{k_s D}} g'(\hat{\lambda}_p) x_{(p+l)_P}. \tag*{(68)} Δ w ^ l ( p ) = k s D η δ g ′ ( λ ^ p ) x ( p + l ) P . (68) The single-vector fields are now indexed by position and by which weight vector of the kernel is used: ν r ( p , l ) ≡ w ^ l ( p ) ⋅ x r / k s D \nu_r^{(p,l)} \equiv\hat{w}^{(p)}_l \cdot x_r/\sqrt{k_s D} ν r ( p , l ) ≡ w ^ l ( p ) ⋅ x r / k s D μ r ( l ) ≡ w l ⋅ x r / k t D \mu_r^{(l)} \equiv w_l \cdot x_r/\sqrt{k_t D} μ r ( l ) ≡ w l ⋅ x r / k t D
⟨ μ s ( l ) ν r ( p , l ′ ) ⟩ = S r , s R l , l ′ ( p ) , ⟨ ν r ( p , l ) ν s ( q , l ′ ) ⟩ = S r , s Q l , l ′ ( p , q ) , (69) \left\langle\mu_s^{(l)} \nu_r^{(p,l')} \right\rangle= S_{r,s} R_{l,l'}^{(p)}, \qquad\left\langle\nu_r^{(p,l)} \nu_s^{(q,l')} \right\rangle= S_{r,s} Q_{l,l'}^{(p,q)}, \tag*{(69)} ⟨ μ s ( l ) ν r ( p , l ′ ) ⟩ = S r , s R l , l ′ ( p ) , ⟨ ν r ( p , l ) ν s ( q , l ′ ) ⟩ = S r , s Q l , l ′ ( p , q ) , (69) where, as in Appendix § 9 , the indices on R ( p ) R^{(p)} R ( p ) μ \mu μ ν \nu ν R l , l ′ ( p ) = w ^ l ( p ) ⋅ w l ′ / ( k s k t D ) R_{l,l'}^{(p)} = \hat{w}^{(p)}_l \cdot w_{l'}/\left(\sqrt{k_s k_t}D\right) R l , l ′ ( p ) = w ^ l ( p ) ⋅ w l ′ / ( k s k t D )
Expanding λ ^ p = ∑ l ′ ′ = 0 k s − 1 ν p + l ′ ′ ( p , l ′ ′ ) \hat{\lambda}_p = \sum_{l''=0}^{k_s-1} \nu_{p+l''}^{(p,l'')} λ ^ p = ∑ l ′′ = 0 k s − 1 ν p + l ′′ ( p , l ′′ ) λ s = ∑ l ′ ′ = 0 k s − 1 μ s + l ′ ′ ( l ′ ′ ) \lambda_s = \sum_{l''=0}^{k_s-1} \mu_{s+l''}^{(l'')} λ s = ∑ l ′′ = 0 k s − 1 μ s + l ′′ ( l ′′ )
⟨ λ ^ p μ r ( l ′ ) ⟩ = ∑ l ′ ′ = 0 k s − 1 R l ′ ′ , l ′ ( p ) S ( p + l ′ ′ ) P , r , ⟨ λ ^ p ν r ( q , l ′ ) ⟩ = ∑ l ′ ′ = 0 k s − 1 Q l ′ ′ , l ′ ( p , q ) S ( p + l ′ ′ ) P , r , (70) \left\langle\hat{\lambda}_p \mu_r^{(l')} \right\rangle= \sum_{l''=0}^{k_s-1} R_{l'',l'}^{(p)} S_{(p+l'')_P,r}, \qquad\left\langle\hat{\lambda}_p \nu_r^{(q,l')} \right\rangle= \sum_{l''=0}^{k_s-1} Q_{l'',l'}^{(p,q)} S_{(p+l'')_P,r}, \tag*{(70)} ⟨ λ ^ p μ r ( l ′ ) ⟩ = l ′′ = 0 ∑ k s − 1 R l ′′ , l ′ ( p ) S ( p + l ′′ ) P , r , ⟨ λ ^ p ν r ( q , l ′ ) ⟩ = l ′′ = 0 ∑ k s − 1 Q l ′′ , l ′ ( p , q ) S ( p + l ′′ ) P , r , (70) ⟨ λ s μ r ( l ′ ) ⟩ = ∑ l ′ ′ = 0 k s − 1 T l ′ , l ′ ′ S ( s + l ′ ′ ) P , r , ⟨ λ s ν r ( q , l ′ ) ⟩ = ∑ l ′ ′ = 0 k s − 1 R l ′ , l ′ ′ ( q ) S ( s + l ′ ′ ) P , r , (71) \left\langle\lambda_s \mu_r^{(l')} \right\rangle= \sum_{l''=0}^{k_s-1} T_{l',l''} S_{(s+l'')_P,r}, \qquad\left\langle\lambda_s \nu_r^{(q,l')} \right\rangle= \sum_{l''=0}^{k_s-1} R_{l',l''}^{(q)} S_{(s+l'')_P,r}, \tag*{(71)} ⟨ λ s μ r ( l ′ ) ⟩ = l ′′ = 0 ∑ k s − 1 T l ′ , l ′′ S ( s + l ′′ ) P , r , ⟨ λ s ν r ( q , l ′ ) ⟩ = l ′′ = 0 ∑ k s − 1 R l ′ , l ′′ ( q ) S ( s + l ′′ ) P , r , (71) the third of which involves the teacher alone and is unchanged from Appendix § 9 . The preactivation covariances generalize to
⟨ λ ^ p λ ^ q ⟩ = ∑ l , l ′ = 0 k s − 1 Q l , l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P , ⟨ λ ^ p λ q ⟩ = ∑ l , l ′ = 0 k s − 1 R l , l ′ ( p ) S ( p + l ) P , ( q + l ′ ) P . (72) \left\langle\hat{\lambda}_p \hat{\lambda}_q \right\rangle= \sum_{l,l'=0}^{k_s-1} Q_{l,l'}^{(p,q)} S_{(p+l)_P,(q+l')_P}, \qquad\left\langle\hat{\lambda}_p \lambda_q \right\rangle= \sum_{l,l'=0}^{k_s-1} R_{l,l'}^{(p)} S_{(p+l)_P,(q+l')_P}. \tag*{(72)} ⟨ λ ^ p λ ^ q ⟩ = l , l ′ = 0 ∑ k s − 1 Q l , l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P , ⟨ λ ^ p λ q ⟩ = l , l ′ = 0 ∑ k s − 1 R l , l ′ ( p ) S ( p + l ) P , ( q + l ′ ) P . (72) ODEs
The ODEs follow by direct computation, in the same way as for the CNN:
R ˙ l , l ′ ( p ) = η k s [ ∑ s = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ s ) − ∑ q = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] . (73) \dot{R}^{(p)}_{l,l'} = \frac{\eta}{k_s}\left[\sum_{s=0}^{P-1} I_3\left(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\lambda_s\right)-\sum_{q=0}^{P-1} I_3\left(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\hat{\lambda}_q\right)\right]. \tag*{(73)} R ˙ l , l ′ ( p ) = k s η [ s = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ s ) − q = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] . (73) Q ˙ l , l ′ ( p , q ) ∣ O ( η ) = η k s { ∑ s = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − ∑ r = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) } + ( p ↔ q , l ↔ l ′ ) . (74) \left.\dot{Q}^{(p,q)}_{l,l'}\right|_{O(\eta)} = \frac{\eta}{k_s}\left\{\sum_{s=0}^{P-1} I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\lambda_s\right)-\sum_{r=0}^{P-1} I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\hat{\lambda}_r\right)\right\}+(p\leftrightarrow q,l\leftrightarrow l'). \tag*{(74)} Q ˙ l , l ′ ( p , q ) O ( η ) = k s η { s = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − r = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) } + ( p ↔ q , l ↔ l ′ ) . (74) Q ˙ l , l ′ ( p , q ) ∣ O ( η 2 ) = η 2 k s 2 S ( p + l ) P , ( q + l ′ ) P [ ⟨ ( Δ y ) 2 ⟩ g ′ ( λ ^ p ) g ′ ( λ ^ q ) + σ 2 J 2 ( λ ^ p , λ ^ q ) ] . (75) \left.\dot{Q}^{(p,q)}_{l,l'}\right|_{O(\eta^2)}=\frac{\eta^2}{k_s^2}S_{(p+l)P,(q+l')P}\left[\langle(\Delta y)^2\rangle g'(\hat{\lambda}_p)g'(\hat{\lambda}_q)+\sigma^2J_2(\hat{\lambda}_p,\hat{\lambda}_q)\right]. \tag*{(75)} Q ˙ l , l ′ ( p , q ) O ( η 2 ) = k s 2 η 2 S ( p + l ) P , ( q + l ′ ) P [ ⟨( Δ y ) 2 ⟩ g ′ ( λ ^ p ) g ′ ( λ ^ q ) + σ 2 J 2 ( λ ^ p , λ ^ q ) ] . (75) The structure is identical to the CNN’s but for the absence of the position sum in the R ˙ ( p ) \dot{R}^{(p)} R ˙ ( p ) p p p p p p P P P S = I S=I S = I
Expanding the weighted-error term as in Appendix § 9 , the equation for Q ( p , q ) Q^{(p,q)} Q ( p , q )
Q ˙ l , l ′ ( p , q ) = η k s [ ∑ s = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − ∑ r = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) ] + η k s [ ∑ s = 0 P − 1 I 3 ( λ ^ q , ν ( q + l ′ ) P ( p , l ) , λ s ) − ∑ r = 0 P − 1 I 3 ( λ ^ q , ν ( q + l ′ ) P ( p , l ) , λ ^ r ) ] + η 2 k s 2 S ( p + l ) P , ( q + l ′ ) P { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (76) \begin{aligned}
\dot{Q}^{(p,q)}_{l,l'}={}&\frac{\eta}{k_s}\left[\sum_{s=0}^{P-1}I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\lambda_s\right)-\sum_{r=0}^{P-1}I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\hat{\lambda}_r\right)\right]\\
&+\frac{\eta}{k_s}\left[\sum_{s=0}^{P-1}I_3\left(\hat{\lambda}_q,\nu^{(p,l)}_{(q+l')P},\lambda_s\right)-\sum_{r=0}^{P-1}I_3\left(\hat{\lambda}_q,\nu^{(p,l)}_{(q+l')P},\hat{\lambda}_r\right)\right]\\
&+\frac{\eta^2}{k_s^2}S_{(p+l)P,(q+l')P}\left\{\sum_{r,s=0}^{P-1}\left[I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\lambda_s)-2I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\hat{\lambda}_s)+I_4(\hat{\lambda}_p,\hat{\lambda}_q,\hat{\lambda}_r,\hat{\lambda}_s)\right]+\sigma^2J_2(\hat{\lambda}_p,\hat{\lambda}_q)\right\}. \tag*{(76)}
\end{aligned} Q ˙ l , l ′ ( p , q ) = k s η [ s = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − r = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) ] + k s η [ s = 0 ∑ P − 1 I 3 ( λ ^ q , ν ( q + l ′ ) P ( p , l ) , λ s ) − r = 0 ∑ P − 1 I 3 ( λ ^ q , ν ( q + l ′ ) P ( p , l ) , λ ^ r ) ] + k s 2 η 2 S ( p + l ) P , ( q + l ′ ) P { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (76) with all position sums running over { 0 , … , P − 1 } \{0,\ldots,P-1\} { 0 , … , P − 1 } (64) , the only structural difference is that the CNN carries a further sum over positions in each term, and a double sum in the O ( η 2 ) O(\eta^2) O ( η 2 ) S S S
In the notation of (18) , the LCN dynamics is therefore given by
[ F R L C N ] l , l ′ ( p ) = ∑ s = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ s ) − ∑ q = 0 P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) . (77) \left[F^{\mathrm{LCN}}_R\right]^{(p)}_{l,l'}=\sum_{s=0}^{P-1}I_3\left(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\lambda_s\right)-\sum_{q=0}^{P-1}I_3\left(\hat{\lambda}_p,\mu^{(l')}_{(p+l)P},\hat{\lambda}_q\right). \tag*{(77)} [ F R LCN ] l , l ′ ( p ) = s = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ s ) − q = 0 ∑ P − 1 I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) . (77) [ F Q L C N ] l , l ′ ( p , q ) = ∑ s = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − ∑ r = 0 P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) + ( p ↔ q , l ↔ l ′ ) . (78) \begin{aligned}
\left[F^{\mathrm{LCN}}_Q\right]^{(p,q)}_{l,l'}={}&\sum_{s=0}^{P-1}I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\lambda_s\right)-\sum_{r=0}^{P-1}I_3\left(\hat{\lambda}_p,\nu^{(q,l')}_{(p+l)P},\hat{\lambda}_r\right)\\
&+(p\leftrightarrow q,l\leftrightarrow l'). \tag*{(78)}
\end{aligned} [ F Q LCN ] l , l ′ ( p , q ) = s = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ s ) − r = 0 ∑ P − 1 I 3 ( λ ^ p , ν ( p + l ) P ( q , l ′ ) , λ ^ r ) + ( p ↔ q , l ↔ l ′ ) . (78) [ G Q L C N ] l , l ′ ( p , q ) = S ( p + l ) P , ( q + l ′ ) P { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (79) \left[G^{\mathrm{LCN}}_Q\right]^{(p,q)}_{l,l'}=S_{(p+l)P,(q+l')P}\left\{\sum_{r,s=0}^{P-1}\left[I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\lambda_s)-2I_4(\hat{\lambda}_p,\hat{\lambda}_q,\lambda_r,\hat{\lambda}_s)+I_4(\hat{\lambda}_p,\hat{\lambda}_q,\hat{\lambda}_r,\hat{\lambda}_s)\right]+\sigma^2J_2(\hat{\lambda}_p,\hat{\lambda}_q)\right\}. \tag*{(79)} [ G Q LCN ] l , l ′ ( p , q ) = S ( p + l ) P , ( q + l ′ ) P { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p , λ ^ q , λ r , λ s ) − 2 I 4 ( λ ^ p , λ ^ q , λ r , λ ^ s ) + I 4 ( λ ^ p , λ ^ q , λ ^ r , λ ^ s ) ] + σ 2 J 2 ( λ ^ p , λ ^ q ) } . (79) with the integrals evaluated at the LCN covariances given above.
Generalization error in terms of order parameters
For either of the two students, the population squared error against the clean teacher can be written
ε g = E [ ( f ∗ ( x ) − y ^ ( x ) ) 2 ] = E [ f ∗ 2 ] − 2 E [ f ∗ y ^ ] + E [ y ^ 2 ] . (80) \varepsilon_g = \mathbb{E}\left[(f^*(x)-\hat{y}(x))^2\right] = \mathbb{E}[f^{*2}] - 2\mathbb{E}[f^*\hat{y}] + \mathbb{E}[\hat{y}^2]. \tag*{(80)} ε g = E [ ( f ∗ ( x ) − y ^ ( x ) ) 2 ] = E [ f ∗ 2 ] − 2 E [ f ∗ y ^ ] + E [ y ^ 2 ] . (80) Expanding f ∗ = ∑ p = 0 P − 1 g ( λ p ) f^*=\sum_{p=0}^{P-1}g(\lambda_p) f ∗ = ∑ p = 0 P − 1 g ( λ p ) y ^ = ∑ r = 0 P − 1 g ( λ ^ r ) \hat{y}=\sum_{r=0}^{P-1}g(\hat{\lambda}_r) y ^ = ∑ r = 0 P − 1 g ( λ ^ r ) E [ g ( u 1 ) g ( u 2 ) ] \mathbb{E}[g(u_1)g(u_2)] E [ g ( u 1 ) g ( u 2 )] ( u 1 , u 2 ) (u_1,u_2) ( u 1 , u 2 )
ε g = ∑ p , q = 0 P − 1 I 2 ( λ p , λ q ) − 2 ∑ p , r = 0 P − 1 I 2 ( λ p , λ ^ r ) + ∑ r , s = 0 P − 1 I 2 ( λ ^ r , λ ^ s ) , (81) \varepsilon_g = \sum_{p,q=0}^{P-1} I_2(\lambda_p,\lambda_q) - 2\sum_{p,r=0}^{P-1} I_2(\lambda_p,\hat{\lambda}_r) + \sum_{r,s=0}^{P-1} I_2(\hat{\lambda}_r,\hat{\lambda}_s), \tag*{(81)} ε g = p , q = 0 ∑ P − 1 I 2 ( λ p , λ q ) − 2 p , r = 0 ∑ P − 1 I 2 ( λ p , λ ^ r ) + r , s = 0 ∑ P − 1 I 2 ( λ ^ r , λ ^ s ) , (81) where the integrals are written with their field arguments, as elsewhere in the appendices. Here λ ^ r \hat{\lambda}_r λ ^ r λ ^ r C N N \hat{\lambda}^{\mathrm{CNN}}_r λ ^ r CNN λ ^ r L C N \hat{\lambda}^{\mathrm{LCN}}_r λ ^ r LCN r r r ⟨ λ p λ q ⟩ \langle\lambda_p\lambda_q\rangle ⟨ λ p λ q ⟩ ⟨ λ p λ ^ r ⟩ \langle\lambda_p\hat{\lambda}_r\rangle ⟨ λ p λ ^ r ⟩ ⟨ λ ^ r λ ^ s ⟩ \langle\hat{\lambda}_r\hat{\lambda}_s\rangle ⟨ λ ^ r λ ^ s ⟩ § 9 and § 10 .
Exact P P P S = I S=I S = I
In this appendix, we show that if S = I S=I S = I k t ≤ k s ≤ P k_t\leq k_s\leq P k t ≤ k s ≤ P P P P R ( p ) R^{(p)} R ( p ) p p p Q ( p , q ) Q^{(p,q)} Q ( p , q ) p p p q q q ( q − p ) P (q-p)_P ( q − p ) P
As a first step, we show that the generalization error of the LCN only depends on a subset of its order parameters. Indeed, the generalization error depends on the covariance
⟨ λ ^ p L C N λ ^ q L C N ⟩ = ∑ l , l ′ = 0 k s − 1 Q l l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P = ∑ l , l ′ = 0 k s − 1 Q l l ′ ( p , q ) δ ( p + l ) P , ( q + l ′ ) P . (82) \left\langle\hat{\lambda}^{\mathrm{LCN}}_p\hat{\lambda}^{\mathrm{LCN}}_q\right\rangle= \sum_{l,l'=0}^{k_s-1} Q^{(p,q)}_{ll'} S_{(p+l)_P,(q+l')_P} = \sum_{l,l'=0}^{k_s-1} Q^{(p,q)}_{ll'} \delta_{(p+l)_P,(q+l')_P}. \tag*{(82)} ⟨ λ ^ p LCN λ ^ q LCN ⟩ = l , l ′ = 0 ∑ k s − 1 Q l l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P = l , l ′ = 0 ∑ k s − 1 Q l l ′ ( p , q ) δ ( p + l ) P , ( q + l ′ ) P . (82) This leads us to call an entry ( p , q , l , l ′ ) (p,q,l,l') ( p , q , l , l ′ ) Q Q Q
( p + l ) P = ( q + l ′ ) P , equivalently q = ( p + l − l ′ ) P . (83) (p+l)_P=(q+l')_P,\qquad\text{equivalently}\qquad q=(p+l-l')_P. \tag*{(83)} ( p + l ) P = ( q + l ′ ) P , equivalently q = ( p + l − l ′ ) P . (83) Given ( p , l , l ′ ) (p,l,l') ( p , l , l ′ ) (83) determines q = ( p + l − l ′ ) P q=(p+l-l')_P q = ( p + l − l ′ ) P Q Q Q P k s 2 Pk_s^2 P k s 2 P 2 k s 2 P^2k_s^2 P 2 k s 2 Q Q Q ⟨ λ ^ p L C N λ s ⟩ \langle\hat{\lambda}^{\mathrm{LCN}}_p\lambda_s\rangle ⟨ λ ^ p LCN λ s ⟩ R R R Q Q Q R R R Q Q Q P P P R R R k s k t k_sk_t k s k t Q Q Q k s 2 k_s^2 k s 2 P P P
To show that the dynamics of R R R Q Q Q Q Q Q Q l l ′ ( p , q ) Q_{ll'}^{(p,q)} Q l l ′ ( p , q ) q = ( p + l − l ′ ) P q=(p+l-l')P q = ( p + l − l ′ ) P
⟨ λ ^ p + l ′ ′ L C N , ν ( p + l ) P ( q , l ′ ) ⟩ = ∑ l ′ ′ = 0 k s − 1 Q l l ′ ′ ( p , q ) δ ( p + l ′ ′ ) P , ( p + l ) P = Q l l ′ ( p , q ) (84) \left\langle\hat{\lambda}_{p+l''}^{\mathrm{LCN}},\nu_{(p+l)P}^{(q,l')}\right\rangle=\sum_{l''=0}^{k_s-1}Q_{ll''}^{(p,q)}\delta_{(p+l'')P,(p+l)P}=Q_{ll'}^{(p,q)} \tag*{(84)} ⟨ λ ^ p + l ′′ LCN , ν ( p + l ) P ( q , l ′ ) ⟩ = l ′′ = 0 ∑ k s − 1 Q l l ′′ ( p , q ) δ ( p + l ′′ ) P , ( p + l ) P = Q l l ′ ( p , q ) (84) resulting in the entry itself. The second one is
⟨ ν ( p + l ) P ( q , l ′ ) , λ ^ r L C N ⟩ = ∑ l ′ ′ = 0 k s − 1 Q l l ′ ′ ( q , r ) δ ( r + l ′ ′ ) P , ( p + l ) P = Q l l ′ ′ ⋆ ( q , r ) , l ′ ′ ⋆ = ( p + l − r ) P , (85) \left\langle\nu_{(p+l)P}^{(q,l')},\hat{\lambda}_{r}^{\mathrm{LCN}}\right\rangle=\sum_{l''=0}^{k_s-1}Q_{ll''}^{(q,r)}\delta_{(r+l'')P,(p+l)P}=Q_{ll''^\star}^{(q,r)},\qquad l''^\star=(p+l-r)P, \tag*{(85)} ⟨ ν ( p + l ) P ( q , l ′ ) , λ ^ r LCN ⟩ = l ′′ = 0 ∑ k s − 1 Q l l ′′ ( q , r ) δ ( r + l ′′ ) P , ( p + l ) P = Q l l ′′ ⋆ ( q , r ) , l ′′ ⋆ = ( p + l − r ) P , (85) if l ′ ′ ⋆ ∈ { 0 , … , k s − 1 } l''^\star\in\{0,\ldots,k_s-1\} l ′′ ⋆ ∈ { 0 , … , k s − 1 } Q l l ′ ′ ⋆ ( q , r ) Q_{ll''^\star}^{(q,r)} Q l l ′′ ⋆ ( q , r ) R R R Q Q Q
When S = I S=I S = I
R l l ′ ( p ) = R ~ l l ′ , Q l l ′ ( p , ( p + l − l ′ ) P ) = Q ~ l l ′ , (86) R_{ll'}^{(p)}=\widetilde{R}_{ll'},\qquad Q_{ll'}^{(p,(p+l-l')P)}=\widetilde{Q}_{ll'}, \tag*{(86)} R l l ′ ( p ) = R l l ′ , Q l l ′ ( p , ( p + l − l ′ ) P ) = Q l l ′ , (86) consisting of a k s × k t k_s\times k_t k s × k t k s × k s k_s\times k_s k s × k s P P P R ~ l l ′ = R l l ′ \widetilde{R}_{ll'}=R_{ll'} R l l ′ = R l l ′ Q ~ l l ′ = Q l l ′ \widetilde{Q}_{ll'}=Q_{ll'} Q l l ′ = Q l l ′ Q Q Q R R R
Substituting R l l ′ ( p ) = R ~ l l ′ R_{ll'}^{(p)}=\widetilde{R}_{ll'} R l l ′ ( p ) = R l l ′ Q l l ′ ( p , ( p + l − l ′ ) P ) = Q ~ l l ′ Q_{ll'}^{(p,(p+l-l')P)}=\widetilde{Q}_{ll'} Q l l ′ ( p , ( p + l − l ′ ) P ) = Q l l ′
R ~ ˙ l , l ′ = η k s ∑ q = 0 P − 1 [ I 3 ( λ ^ p L C N , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p L C N , μ ( p + l ) P ( l ′ ) , λ ^ q L C N ) ] (87) \dot{\widetilde{R}}_{l,l'}=\frac{\eta}{k_s}\sum_{q=0}^{P-1}\left[I_3\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\mu_{(p+l)P}^{(l')},\lambda_q\right)-I_3\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\mu_{(p+l)P}^{(l')},\hat{\lambda}_{q}^{\mathrm{LCN}}\right)\right] \tag*{(87)} R ˙ l , l ′ = k s η q = 0 ∑ P − 1 [ I 3 ( λ ^ p LCN , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p LCN , μ ( p + l ) P ( l ′ ) , λ ^ q LCN ) ] (87) where now p p p p p p
Q ~ ˙ l , l ′ = η k s ∑ r = 0 P − 1 [ I 3 ( λ ^ p L C N , ν ( p + l ) P ( q , l ′ ) , λ r ) − I 3 ( λ ^ p L C N , ν ( p + l ) P ( q , l ′ ) , λ ^ r L C N ) ] + η k s ∑ r = 0 P − 1 [ I 3 ( λ ^ q L C N , ν ( q + l ′ ) P ( p , l ) , λ r ) − I 3 ( λ ^ q L C N , ν ( q + l ′ ) P ( p , l ) , λ ^ r L C N ) ] + η 2 k s 2 { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p L C N , λ ^ q L C N , λ r , λ s ) − 2 I 4 ( λ ^ p L C N , λ ^ q L C N , λ r , λ ^ s L C N ) + I 4 ( λ ^ p L C N , λ ^ q L C N , λ ^ r L C N , λ ^ s L C N ) ] + σ 2 J 2 ( λ ^ p L C N , λ ^ q L C N ) } , (88) \begin{aligned}
\dot{\widetilde{Q}}_{l,l'}={}&\frac{\eta}{k_s}\sum_{r=0}^{P-1}\left[I_3\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\nu_{(p+l)P}^{(q,l')},\lambda_r\right)-I_3\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\nu_{(p+l)P}^{(q,l')},\hat{\lambda}_{r}^{\mathrm{LCN}}\right)\right]\\
&+\frac{\eta}{k_s}\sum_{r=0}^{P-1}\left[I_3\left(\hat{\lambda}_{q}^{\mathrm{LCN}},\nu_{(q+l')P}^{(p,l)},\lambda_r\right)-I_3\left(\hat{\lambda}_{q}^{\mathrm{LCN}},\nu_{(q+l')P}^{(p,l)},\hat{\lambda}_{r}^{\mathrm{LCN}}\right)\right]\\
&+\frac{\eta^2}{k_s^2}\left\{\sum_{r,s=0}^{P-1}\left[I_4\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\hat{\lambda}_{q}^{\mathrm{LCN}},\lambda_r,\lambda_s\right)-2I_4\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\hat{\lambda}_{q}^{\mathrm{LCN}},\lambda_r,\hat{\lambda}_{s}^{\mathrm{LCN}}\right)\right.\right.\\
&\left.\left.\qquad\qquad+I_4\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\hat{\lambda}_{q}^{\mathrm{LCN}},\hat{\lambda}_{r}^{\mathrm{LCN}},\hat{\lambda}_{s}^{\mathrm{LCN}}\right)\right]+\sigma^2J_2\left(\hat{\lambda}_{p}^{\mathrm{LCN}},\hat{\lambda}_{q}^{\mathrm{LCN}}\right)\right\}, \tag*{(88)}
\end{aligned} Q ˙ l , l ′ = k s η r = 0 ∑ P − 1 [ I 3 ( λ ^ p LCN , ν ( p + l ) P ( q , l ′ ) , λ r ) − I 3 ( λ ^ p LCN , ν ( p + l ) P ( q , l ′ ) , λ ^ r LCN ) ] + k s η r = 0 ∑ P − 1 [ I 3 ( λ ^ q LCN , ν ( q + l ′ ) P ( p , l ) , λ r ) − I 3 ( λ ^ q LCN , ν ( q + l ′ ) P ( p , l ) , λ ^ r LCN ) ] + k s 2 η 2 { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p LCN , λ ^ q LCN , λ r , λ s ) − 2 I 4 ( λ ^ p LCN , λ ^ q LCN , λ r , λ ^ s LCN ) + I 4 ( λ ^ p LCN , λ ^ q LCN , λ ^ r LCN , λ ^ s LCN ) ] + σ 2 J 2 ( λ ^ p LCN , λ ^ q LCN ) } , (88) where again p p p q = ( p + l − l ′ ) P q=(p+l-l')P q = ( p + l − l ′ ) P p p p
Finally, assuming S = I S=I S = I
R ˙ l , l ′ = η k s ∑ p , q = 0 P − 1 I 3 ( λ ^ p C N N , μ ( p + l ) P ( l ′ ) , λ q ) − η k s ∑ p , q = 0 P − 1 I 3 ( λ ^ p C N N , μ ( p + l ) P ( l ′ ) , λ ^ q C N N ) = P η k s ∑ q = 0 P − 1 [ I 3 ( λ ^ p C N N , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p C N N , μ ( p + l ) P ( l ′ ) , λ ^ q C N N ) ] , (89) \begin{aligned}
\dot{R}_{l,l'}={}&\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}I_3\left(\hat{\lambda}_{p}^{\mathrm{CNN}},\mu_{(p+l)P}^{(l')},\lambda_q\right)-\frac{\eta}{k_s}\sum_{p,q=0}^{P-1}I_3\left(\hat{\lambda}_{p}^{\mathrm{CNN}},\mu_{(p+l)P}^{(l')},\hat{\lambda}_{q}^{\mathrm{CNN}}\right)\\
={}&\frac{P\eta}{k_s}\sum_{q=0}^{P-1}\left[I_3\left(\hat{\lambda}_{p}^{\mathrm{CNN}},\mu_{(p+l)P}^{(l')},\lambda_q\right)-I_3\left(\hat{\lambda}_{p}^{\mathrm{CNN}},\mu_{(p+l)P}^{(l')},\hat{\lambda}_{q}^{\mathrm{CNN}}\right)\right], \tag*{(89)}
\end{aligned} R ˙ l , l ′ = = k s η p , q = 0 ∑ P − 1 I 3 ( λ ^ p CNN , μ ( p + l ) P ( l ′ ) , λ q ) − k s η p , q = 0 ∑ P − 1 I 3 ( λ ^ p CNN , μ ( p + l ) P ( l ′ ) , λ ^ q CNN ) k s P η q = 0 ∑ P − 1 [ I 3 ( λ ^ p CNN , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p CNN , μ ( p + l ) P ( l ′ ) , λ ^ q CNN ) ] , (89) where we have used the invariance under cyclic shifts of the positions to drop the sum over p p p p p p
Q ˙ l , l ′ = P η k s ∑ r = 0 P − 1 [ I 3 ( λ ^ p C N N , ν ( p + l ′ ) P ( l ′ ) , λ r ) − I 3 ( λ ^ p C N N , ν ( p + l ) P ( l ′ ) , λ ^ r C N N ) ] + P η k s ∑ r = 0 P − 1 [ I 3 ( λ ^ p C N N , ν ( p + l ′ ) P ( l ) , λ r ) − I 3 ( λ ^ p C N N , ν ( p + l ′ ) P ( l ) , λ ^ r C N N ) ] + P η 2 k s 2 { ∑ r , s = 0 P − 1 [ I 4 ( λ ^ p C N N , λ ^ q C N N , λ r , λ s ) − 2 I 4 ( λ ^ p C N N , λ ^ q C N N , λ r , λ ^ s C N N ) + I 4 ( λ ^ p C N N , λ ^ q C N N , λ ^ r C N N , λ ^ s C N N ) ] + σ 2 J 2 ( λ ^ p C N N , λ ^ q C N N ) } . (90) \dot{Q}_{l,l'} = \frac{P\eta}{k_s}\sum_{r=0}^{P-1}\left[I_3\left(\hat{\lambda}^{\mathrm{CNN}}_p,\nu^{(l')}_{(p+l')_P},\lambda_r\right)-I_3\left(\hat{\lambda}^{\mathrm{CNN}}_p,\nu^{(l')}_{(p+l)_P},\hat{\lambda}^{\mathrm{CNN}}_r\right)\right]
+ \frac{P\eta}{k_s}\sum_{r=0}^{P-1}\left[I_3\left(\hat{\lambda}^{\mathrm{CNN}}_p,\nu^{(l)}_{(p+l')_P},\lambda_r\right)-I_3\left(\hat{\lambda}^{\mathrm{CNN}}_p,\nu^{(l)}_{(p+l')_P},\hat{\lambda}^{\mathrm{CNN}}_r\right)\right]
+ \frac{P\eta^2}{k_s^2}\left\{\sum_{r,s=0}^{P-1}\left[I_4\left(\hat{\lambda}^{\mathrm{CNN}}_p,\hat{\lambda}^{\mathrm{CNN}}_q,\lambda_r,\lambda_s\right)-2I_4\left(\hat{\lambda}^{\mathrm{CNN}}_p,\hat{\lambda}^{\mathrm{CNN}}_q,\lambda_r,\hat{\lambda}^{\mathrm{CNN}}_s\right)+I_4\left(\hat{\lambda}^{\mathrm{CNN}}_p,\hat{\lambda}^{\mathrm{CNN}}_q,\hat{\lambda}^{\mathrm{CNN}}_r,\hat{\lambda}^{\mathrm{CNN}}_s\right)\right]+\sigma^2J_2\left(\hat{\lambda}^{\mathrm{CNN}}_p,\hat{\lambda}^{\mathrm{CNN}}_q\right)\right\}. \tag*{(90)} Q ˙ l , l ′ = k s P η r = 0 ∑ P − 1 [ I 3 ( λ ^ p CNN , ν ( p + l ′ ) P ( l ′ ) , λ r ) − I 3 ( λ ^ p CNN , ν ( p + l ) P ( l ′ ) , λ ^ r CNN ) ] + k s P η r = 0 ∑ P − 1 [ I 3 ( λ ^ p CNN , ν ( p + l ′ ) P ( l ) , λ r ) − I 3 ( λ ^ p CNN , ν ( p + l ′ ) P ( l ) , λ ^ r CNN ) ] + k s 2 P η 2 { r , s = 0 ∑ P − 1 [ I 4 ( λ ^ p CNN , λ ^ q CNN , λ r , λ s ) − 2 I 4 ( λ ^ p CNN , λ ^ q CNN , λ r , λ ^ s CNN ) + I 4 ( λ ^ p CNN , λ ^ q CNN , λ ^ r CNN , λ ^ s CNN ) ] + σ 2 J 2 ( λ ^ p CNN , λ ^ q CNN ) } . (90) where again p p p q = ( p + l − l ′ ) P q=(p+l-l')_P q = ( p + l − l ′ ) P
Finally, by identifying Q l l ′ = Q ~ l l ′ Q_{ll'}=\widetilde{Q}_{ll'} Q l l ′ = Q l l ′ R l l ′ = R ~ l l ′ R_{ll'}=\widetilde{R}_{ll'} R l l ′ = R l l ′ (89) with Eq. (87) and Eq. (90) with Eq. (88) , it is easy to show that the dynamics of the CNN and the LCN coincide up to a time rescaling by a factor P P P
Reflection symmetry of the CNN dynamics
In this appendix, we show that the CNN ODEs (17) are equivariant under the reflection ρ \rho ρ (26) . The idea is that reversing the kernels of both the teacher and the student is equivalent to reflecting the input positions, r ↦ − r r\mapsto-r r ↦ − r
We write l ˉ = k s − 1 − l \bar{l}=k_s-1-l l ˉ = k s − 1 − l l ~ = k t − 1 − l \tilde{l}=k_t-1-l l ~ = k t − 1 − l
( ρ R ) l , l ′ = R l ˉ , l ~ ′ , ( ρ Q ) l , l ′ = Q l ˉ , l ˉ ′ (91) (\rho R)_{l,l'}=R_{\bar{l},\tilde{l}'},\qquad(\rho Q)_{l,l'}=Q_{\bar{l},\bar{l}'} \tag*{(91)} ( ρR ) l , l ′ = R l ˉ , l ~ ′ , ( ρQ ) l , l ′ = Q l ˉ , l ˉ ′ (91) We assume that S S S S ( − r ) P , ( − q ) P = S r , q S_{(-r)_P,(-q)_P}=S_{r,q} S ( − r ) P , ( − q ) P = S r , q (3) . The teacher overlaps T l , l ′ = ( T / k t ) δ l , l ′ T_{l,l'}=(T/k_t)\delta_{l,l'} T l , l ′ = ( T / k t ) δ l , l ′ T l ~ , l ~ ′ = T l , l ′ T_{\tilde{l},\tilde{l}'}=T_{l,l'} T l ~ , l ~ ′ = T l , l ′ ⟨ a b ⟩ y \langle ab\rangle_y ⟨ ab ⟩ y a a a b b b y = ( R , Q ) y=(R,Q) y = ( R , Q )
We first show that the covariances at ρ y \rho y ρ y y y y
ν r ( l ) → ν ( − r ) P ( l ˉ ) , μ r ( l ) → μ ( − r ) P ( l ~ ) . (92) \nu^{(l)}_r\to\nu^{(\bar{l})}_{(-r)_P},\qquad\mu^{(l)}_r\to\mu^{(\tilde{l})}_{(-r)_P}. \tag*{(92)} ν r ( l ) → ν ( − r ) P ( l ˉ ) , μ r ( l ) → μ ( − r ) P ( l ~ ) . (92) Using the two-point functions of Appendix A.1 and the invariance of S S S
⟨ ν r ( l ) ν q ( l ′ ) ⟩ ρ y = S r , q Q l ˉ , l ˉ ′ = S ( − r ) P , ( − q ) P Q l ˉ , l ˉ ′ = ⟨ ν ( − r ) P ( l ˉ ) ν ( − q ) P ( l ˉ ′ ) ⟩ y (93) \left\langle\nu^{(l)}_r\nu^{(l')}_q\right\rangle_{\rho y}=S_{r,q}Q_{\bar{l},\bar{l}'}=S_{(-r)_P,(-q)_P}Q_{\bar{l},\bar{l}'}=\left\langle\nu^{(\bar{l})}_{(-r)_P}\nu^{(\bar{l}')}_{(-q)_P}\right\rangle_y \tag*{(93)} ⟨ ν r ( l ) ν q ( l ′ ) ⟩ ρ y = S r , q Q l ˉ , l ˉ ′ = S ( − r ) P , ( − q ) P Q l ˉ , l ˉ ′ = ⟨ ν ( − r ) P ( l ˉ ) ν ( − q ) P ( l ˉ ′ ) ⟩ y (93) ⟨ μ r ( l ) μ q ( l ′ ) ⟩ ρ y = S r , q T l , l ′ = S ( − r ) P , ( − q ) P T l ~ , l ~ ′ = ⟨ μ ( − r ) P ( l ~ ) μ ( − q ) P ( l ~ ′ ) ⟩ y (94) \left\langle\mu^{(l)}_r\mu^{(l')}_q\right\rangle_{\rho y}=S_{r,q}T_{l,l'}=S_{(-r)_P,(-q)_P}T_{\tilde{l},\tilde{l}'}=\left\langle\mu^{(\tilde{l})}_{(-r)_P}\mu^{(\tilde{l}')}_{(-q)_P}\right\rangle_y \tag*{(94)} ⟨ μ r ( l ) μ q ( l ′ ) ⟩ ρ y = S r , q T l , l ′ = S ( − r ) P , ( − q ) P T l ~ , l ~ ′ = ⟨ μ ( − r ) P ( l ~ ) μ ( − q ) P ( l ~ ′ ) ⟩ y (94) ⟨ μ r ( l ) ν q ( l ′ ) ⟩ ρ y = S r , q R l ˉ , l ~ = S ( − r ) P , ( − q ) P R l ˉ , l ~ = ⟨ μ ( − r ) P ( l ~ ) ν ( − q ) P ( l ˉ ′ ) ⟩ y . (95) \left\langle\mu^{(l)}_r\nu^{(l')}_q\right\rangle_{\rho y}=S_{r,q}R_{\bar{l},\tilde{l}}=S_{(-r)_P,(-q)_P}R_{\bar{l},\tilde{l}}=\left\langle\mu^{(\tilde{l})}_{(-r)_P}\nu^{(\bar{l}')}_{(-q)_P}\right\rangle_y. \tag*{(95)} ⟨ μ r ( l ) ν q ( l ′ ) ⟩ ρ y = S r , q R l ˉ , l ~ = S ( − r ) P , ( − q ) P R l ˉ , l ~ = ⟨ μ ( − r ) P ( l ~ ) ν ( − q ) P ( l ˉ ′ ) ⟩ y . (95) The preactivations are sums of single-vector fields, and the relabeling (92) maps them to preactivations at reflected positions,
λ ^ p → ∑ l = 0 k s − 1 ν ( − p − l ) P ( l ˉ ) = λ ^ p ^ , λ q → ∑ l = 0 k t − 1 μ ( − q − l ) P ( l ~ ) = λ q ˇ , (96) \hat{\lambda}_{p} \to\sum_{l=0}^{k_s-1}\nu^{(\bar{l})}_{(-p-l)_{P}}=\hat{\lambda}_{\hat{p}}, \qquad\lambda_{q} \to\sum_{l=0}^{k_t-1}\mu^{(\tilde{l})}_{(-q-l)_{P}}=\lambda_{\check{q}}, \tag*{(96)} λ ^ p → l = 0 ∑ k s − 1 ν ( − p − l ) P ( l ˉ ) = λ ^ p ^ , λ q → l = 0 ∑ k t − 1 μ ( − q − l ) P ( l ~ ) = λ q ˇ , (96) with p ^ = ( − p − k s + 1 ) P \hat{p}=(-p-k_s+1)_{P} p ^ = ( − p − k s + 1 ) P q ˇ = ( − q − k t + 1 ) P \check{q}=(-q-k_t+1)_{P} q ˇ = ( − q − k t + 1 ) P
We now show that the ODEs are equivariant. Each of the Gaussian integrals I 2 I_2 I 2 J 2 J_2 J 2 I 3 I_3 I 3 I 4 I_4 I 4 R R R (65) , the relabeling maps μ ( p + l ) P ( l ′ ) \mu^{(l')}_{(p+l)_{P}} μ ( p + l ) P ( l ′ ) μ ( p ^ + l ˉ ) P ( l ~ ′ ) \mu^{(\tilde{l}')}_{(\hat{p}+\bar{l})_{P}} μ ( p ^ + l ˉ ) P ( l ~ ′ )
[ F R C N N ] l , l ′ ∣ ρ y = ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] ∣ ρ y = ∑ p , q = 0 P − 1 [ I 3 ( λ ^ p ^ , μ ( p ^ + l ˉ ) P ( l ~ ′ ) , λ q ˇ ) − I 3 ( λ ^ p ^ , μ ( p ^ + l ˉ ) P ( l ~ ′ ) , λ ^ q ^ ) ] ∣ y = ∑ p ′ , q ′ = 0 P − 1 [ I 3 ( λ p ′ , μ ( p ′ + l ~ ) P ( l ~ ′ ) , λ q ′ ) − I 3 ( λ p ′ , μ ( p ′ + l ~ ) P ( l ~ ′ ) , λ ^ q ′ ) ] ∣ y = [ F R C N N ] l ~ , l ~ ′ ∣ y , (97) \begin{aligned}
\left[F^{\mathrm{CNN}}_{R}\right]_{l,l'\mid\rho y}
&=\left.\sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_{p},\mu^{(l')}_{(p+l)_{P}},\lambda_q\right)-I_3\left(\hat{\lambda}_{p},\mu^{(l')}_{(p+l)_{P}},\hat{\lambda}_q\right)\right]\right|_{\rho y}\\
&=\left.\sum_{p,q=0}^{P-1}\left[I_3\left(\hat{\lambda}_{\hat{p}},\mu^{(\tilde{l}')}_{(\hat{p}+\bar{l})_{P}},\lambda_{\check{q}}\right)-I_3\left(\hat{\lambda}_{\hat{p}},\mu^{(\tilde{l}')}_{(\hat{p}+\bar{l})_{P}},\hat{\lambda}_{\hat{q}}\right)\right]\right|_{y}\\
&=\left.\sum_{p',q'=0}^{P-1}\left[I_3\left(\lambda_{p'},\mu^{(\tilde{l}')}_{(p'+\tilde{l})_{P}},\lambda_{q'}\right)-I_3\left(\lambda_{p'},\mu^{(\tilde{l}')}_{(p'+\tilde{l})_{P}},\hat{\lambda}_{q'}\right)\right]\right|_{y}
=\left[F^{\mathrm{CNN}}_{R}\right]_{\tilde{l},\tilde{l}'\mid y}, \tag*{(97)}
\end{aligned} [ F R CNN ] l , l ′ ∣ ρ y = p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ q ) − I 3 ( λ ^ p , μ ( p + l ) P ( l ′ ) , λ ^ q ) ] ρ y = p , q = 0 ∑ P − 1 [ I 3 ( λ ^ p ^ , μ ( p ^ + l ˉ ) P ( l ~ ′ ) , λ q ˇ ) − I 3 ( λ ^ p ^ , μ ( p ^ + l ˉ ) P ( l ~ ′ ) , λ ^ q ^ ) ] y = p ′ , q ′ = 0 ∑ P − 1 [ I 3 ( λ p ′ , μ ( p ′ + l ~ ) P ( l ~ ′ ) , λ q ′ ) − I 3 ( λ p ′ , μ ( p ′ + l ~ ) P ( l ~ ′ ) , λ ^ q ′ ) ] y = [ F R CNN ] l ~ , l ~ ′ ∣ y , (97) where in going from the first to the second line we used the fact that covariances at ρ y \rho y ρ y y y y (92) , and in the last step we changed the summation variables to p ′ = p ^ p'=\hat{p} p ′ = p ^ q ′ = q ˇ q'=\check{q} q ′ = q ˇ q ′ = q ^ q'=\hat{q} q ′ = q ^ Q Q Q (17) in the compact form d y / d α = F ( y ) \mathrm{d}y/\mathrm{d}\alpha=F(y) d y / d α = F ( y )
F ( ρ y ) = ρ F ( y ) . (98) F(\rho y)=\rho F(y). \tag*{(98)} F ( ρ y ) = ρF ( y ) . (98) Similarly, we obtain ϵ g ( ρ y ) = ϵ g ( y ) \epsilon_g(\rho y)=\epsilon_g(y) ϵ g ( ρ y ) = ϵ g ( y )
Finally, let y ( α ) y(\alpha) y ( α ) (17) with a ρ \rho ρ ρ y ( 0 ) = y ( 0 ) \rho y(0)=y(0) ρ y ( 0 ) = y ( 0 ) ρ \rho ρ z ( α ) = ρ y ( α ) z(\alpha)=\rho y(\alpha) z ( α ) = ρ y ( α ) (98) ,
d z d α = ρ F ( y ( α ) ) = F ( z ( α ) ) , z ( 0 ) = y ( 0 ) . (99) \frac{\mathrm{d}z}{\mathrm{d}\alpha}=\rho F(y(\alpha))=F(z(\alpha)),\qquad z(0)=y(0). \tag*{(99)} d α d z = ρF ( y ( α )) = F ( z ( α )) , z ( 0 ) = y ( 0 ) . (99) The right-hand side F F F z ( α ) = y ( α ) z(\alpha)=y(\alpha) z ( α ) = y ( α ) ρ y ( α ) = y ( α ) \rho y(\alpha)=y(\alpha) ρ y ( α ) = y ( α ) α ≥ 0 \alpha\ge0 α ≥ 0 Fix ( ρ ) = { y : ρ y = y } \operatorname{Fix}(\rho)=\{y:\rho y=y\} Fix ( ρ ) = { y : ρ y = y }
Plateau phase
Figure 7 repeats the measurement of Figure 3 b in two further settings. Panel (a) uses a teacher of kernel size k t = 2 k_t=2 k t = 2 D D D k s = 2 k_s=2 k s = 2 k s = 3 k_s=3 k s = 3 k s = 4 k_s=4 k s = 4 k t = 1 k_t=1 k t = 1 c = 0.5 c=0.5 c = 0.5 η = 0.2 \eta=0.2 η = 0.2 D D D k s > k t k_s>k_t k s > k t D D D k s = k t k_s=k_t k s = k t
Figure 7. Specialization time of the CNN against the feature dimension, as in Figure 3 b. Color denotes the student kernel size. Markers are averages over random initializations, each obtained by integrating the ODEs (17) from the order parameters of a random initialization at dimension D D D a + b ln D a+b\ln D a + b ln D k t = 2 k_t=2 k t = 2 k s = k t = 2 k_s=k_t=2 k s = k t = 2 D D D k t = 1 k_t=1 k t = 1 c = 0 c=0 c = 0 c = 0.5 c=0.5 c = 0.5 k s > k t k_s>k_t k s > k t g = erf g=\operatorname{erf} g = erf P = 6 P=6 P = 6 η = 0.5 \eta=0.5 η = 0.5 η = 0.2 \eta=0.2 η = 0.2 T = 1 T=1 T = 1 σ = 0 \sigma=0 σ = 0 c = 0 c=0 c = 0 D D D 10 6 10^6 1 0 6 10 10 10^{10} 1 0 10
Early-time dynamics
The rescaling of Appendix D is exact only at S = I S=I S = I c > 0 c>0 c > 0 D D D R = 0 R=0 R = 0 Q = I / k s Q=I/k_s Q = I / k s R ( p ) = 0 R^{(p)}=0 R ( p ) = 0 Q ( p , q ) = δ p , q I / k s Q^{(p,q)}=\delta_{p,q}I/k_s Q ( p , q ) = δ p , q I / k s S S S S a , b S_{a,b} S a , b a a a b b b c c c s c ≡ ∑ q S p , q = 1 + ( P − 1 ) c s_c\equiv\sum_q S_{p,q}=1+(P-1)c s c ≡ ∑ q S p , q = 1 + ( P − 1 ) c ω ( χ ) ≡ ( 4 − χ 2 ) − 1 / 2 \omega(\chi)\equiv(4-\chi^2)^{-1/2} ω ( χ ) ≡ ( 4 − χ 2 ) − 1/2
Initial state. Inserting the initial order parameters into Eqs. (43) and (72) gives
⟨ λ p λ q ⟩ = T S p , q , ⟨ λ ^ p λ q ⟩ = 0 , ⟨ λ ^ p λ ^ q ⟩ = { S p , q CNN , δ p , q LCN . (100) \langle\lambda_p\lambda_q\rangle= T S_{p,q}, \qquad\langle\hat{\lambda}_p\lambda_q\rangle= 0, \qquad\langle\hat{\lambda}_p\hat{\lambda}_q\rangle=
\begin{cases}
S_{p,q} & \text{CNN},\\
\delta_{p,q} & \text{LCN}.
\end{cases} \tag*{(100)} ⟨ λ p λ q ⟩ = T S p , q , ⟨ λ ^ p λ q ⟩ = 0 , ⟨ λ ^ p λ ^ q ⟩ = { S p , q δ p , q CNN , LCN . (100) The student preactivations have unit variance, and we denote by χ \chi χ χ = c \chi=c χ = c χ = 0 \chi=0 χ = 0 (81) of Appendix § 11 ,
ε g = ∑ p , q = 0 P − 1 I 2 ( λ p , λ q ) ⏟ teacher − 2 ∑ p , r = 0 P − 1 I 2 ( λ p , λ ^ r ) ⏟ cross + ∑ r , s = 0 P − 1 I 2 ( λ ^ r , λ ^ s ) ⏟ student , (101) \varepsilon_g = \underbrace{\sum_{p,q=0}^{P-1} I_2(\lambda_p,\lambda_q)}_{\text{teacher}} - 2\underbrace{\sum_{p,r=0}^{P-1} I_2(\lambda_p,\hat{\lambda}_r)}_{\text{cross}} + \underbrace{\sum_{r,s=0}^{P-1} I_2(\hat{\lambda}_r,\hat{\lambda}_s)}_{\text{student}}, \tag*{(101)} ε g = teacher p , q = 0 ∑ P − 1 I 2 ( λ p , λ q ) − 2 cross p , r = 0 ∑ P − 1 I 2 ( λ p , λ ^ r ) + student r , s = 0 ∑ P − 1 I 2 ( λ ^ r , λ ^ s ) , (101) with I 2 = 2 π arcsin [ c 12 ( 1 + c 11 ) ( 1 + c 22 ) ] I_2 = \frac{2}{\pi}\arcsin\left[\frac{c_{12}}{\sqrt{(1+c_{11})(1+c_{22})}}\right] I 2 = π 2 arcsin [ ( 1 + c 11 ) ( 1 + c 22 ) c 12 ] P P P P ( P − 1 ) P(P-1) P ( P − 1 ) (100) , the teacher sum gives the mean squared output of the teacher,
ε T = E [ f ⋆ ( x ) 2 ] = 2 P π [ arcsin T 1 + T + ( P − 1 ) arcsin T c 1 + T ] , (102) \varepsilon_T = \mathbb{E}\left[f^\star(x)^2\right] = \frac{2P}{\pi}\left[\arcsin\frac{T}{1+T} + (P-1)\arcsin\frac{Tc}{1+T}\right], \tag*{(102)} ε T = E [ f ⋆ ( x ) 2 ] = π 2 P [ arcsin 1 + T T + ( P − 1 ) arcsin 1 + T T c ] , (102) the cross sum vanishes, and adding the student sum gives
ε 0 = ε T + P 3 + 2 P ( P − 1 ) π arcsin χ 2 , (103) \varepsilon_0 = \varepsilon_T + \frac{P}{3} + \frac{2P(P-1)}{\pi}\arcsin\frac{\chi}{2}, \tag*{(103)} ε 0 = ε T + 3 P + π 2 P ( P − 1 ) arcsin 2 χ , (103) which yields the initial errors (28) with χ = 0 \chi=0 χ = 0 χ = c \chi=c χ = c
Equations of motion at the initial state
We evaluate the ODEs of Appendices § 9 and § 10 at the initial state, where the integrals I 3 I_3 I 3 I 4 I_4 I 4 T T T c c c l < k s l<k_s l < k s l ′ < k t l'<k_t l ′ < k t
d R l , l ′ d α ∣ C N N = 2 η T π k s k t P s c 2 ( 1 + T ) , d R l , l ′ ( p ) d α ∣ L C N = 2 η T π k s k t s c 2 ( 1 + T ) , (104) \left.\frac{dR_{l,l'}}{d\alpha}\right|_{\mathrm{CNN}} = \frac{2\eta T}{\pi k_s k_t}\frac{P s_c}{\sqrt{2(1+T)}}, \qquad\left.\frac{dR^{(p)}_{l,l'}}{d\alpha}\right|_{\mathrm{LCN}} = \frac{2\eta T}{\pi k_s k_t}\frac{s_c}{\sqrt{2(1+T)}}, \tag*{(104)} d α d R l , l ′ CNN = π k s k t 2 η T 2 ( 1 + T ) P s c , d α d R l , l ′ ( p ) LCN = π k s k t 2 η T 2 ( 1 + T ) s c , (104) and the student overlaps evolve as
d Q l , l ′ d α ∣ C N N = − 2 η k s 2 H l , l ′ + η 2 k s 2 ∑ p , q = 0 P − 1 S ( p + l ) P , ( q + l ′ ) P V p , q , (105) \left.\frac{dQ_{l,l'}}{d\alpha}\right|_{\mathrm{CNN}} = -\frac{2\eta}{k_s^2}H_{l,l'}+\frac{\eta^2}{k_s^2}\sum_{p,q=0}^{P-1}S_{(p+l)P,(q+l')P}V_{p,q}, \tag*{(105)} d α d Q l , l ′ CNN = − k s 2 2 η H l , l ′ + k s 2 η 2 p , q = 0 ∑ P − 1 S ( p + l ) P , ( q + l ′ ) P V p , q , (105) d Q l , l ′ ( p , q ) d α ∣ L C N = 1 k s 2 [ − 2 η Ψ p , q + η 2 V p , q ] S ( p + l ) P , ( q + l ′ ) P . (106) \left.\frac{dQ^{(p,q)}_{l,l'}}{d\alpha}\right|_{\mathrm{LCN}} = \frac{1}{k_s^2}\left[-2\eta\Psi_{p,q}+\eta^2V_{p,q}\right]S_{(p+l)P,(q+l')P}. \tag*{(106)} d α d Q l , l ′ ( p , q ) LCN = k s 2 1 [ − 2 η Ψ p , q + η 2 V p , q ] S ( p + l ) P , ( q + l ′ ) P . (106) For the CNN, H l , l = H d H_{l,l}=H_{\mathrm{d}} H l , l = H d H l , l ′ = H o H_{l,l'}=H_{\mathrm{o}} H l , l ′ = H o l ≠ l ′ l\ne l' l = l ′
H d = P π [ 1 3 + ( P − 1 ) c ω ( c ) ] , H o = P π [ c 3 + ( 2 + 2 ( P − 2 ) c − ( P − 1 ) c 2 ) ω ( c ) ] , (107) H_{\mathrm{d}}=\frac{P}{\pi}\left[\frac{1}{\sqrt{3}}+(P-1)c\omega(c)\right], \qquad H_{\mathrm{o}}=\frac{P}{\pi}\left[\frac{c}{\sqrt{3}}+\left(2+2(P-2)c-(P-1)c^2\right)\omega(c)\right], \tag*{(107)} H d = π P [ 3 1 + ( P − 1 ) c ω ( c ) ] , H o = π P [ 3 c + ( 2 + 2 ( P − 2 ) c − ( P − 1 ) c 2 ) ω ( c ) ] , (107) and for the LCN, Ψ p , p = 1 / ( π 3 ) \Psi_{p,p}=1/(\pi\sqrt{3}) Ψ p , p = 1/ ( π 3 ) Ψ p , q = 1 / π \Psi_{p,q}=1/\pi Ψ p , q = 1/ π p ≠ q p\ne q p = q V p , q ≡ ⟨ δ 2 g ′ ( λ p ) g ′ ( λ ^ q ) ⟩ V_{p,q}\equiv\langle\delta^2g'(\lambda_p)g'(\hat{\lambda}_q)\rangle V p , q ≡ ⟨ δ 2 g ′ ( λ p ) g ′ ( λ ^ q )⟩ O ( η 2 ) O(\eta^2) O ( η 2 ) δ \delta δ § 9 . At the initial state it equals V d V_d V d p = q p=q p = q V o V_o V o p ≠ q p\ne q p = q
V d = 2 ( ε T + σ 2 ) π 3 + 4 π 2 3 [ arcsin 1 4 + 2 ( P − 1 ) arcsin 1 2 2 ( 3 − χ 2 ) + ( P − 1 ) arcsin χ ( 3 − 2 χ 2 ) 2 ( 3 − χ 2 ) + ( P − 1 ) ( P − 2 ) arcsin χ ( 3 − 2 χ ) 2 ( 3 − χ 2 ) ] , V o = 2 ω ( χ ) ( ε T + σ 2 ) π + 4 ω ( χ ) π 2 [ 2 arcsin 2 − χ 2 2 ( 3 − χ 2 ) + 2 arcsin χ 2 ( 3 − χ 2 ) + ( P − 2 ) arcsin 2 + χ − 2 χ 2 2 ( 2 − χ ) ( 1 + χ ) + 4 ( P − 2 ) arcsin χ 2 ( 3 − χ 2 ) ( 1 + χ ) + ( P − 2 ) ( P − 3 ) arcsin χ 2 ( 1 + χ ) ] . (108) \begin{aligned}
V_d &= \frac{2(\varepsilon_T+\sigma^2)}{\pi\sqrt{3}}+\frac{4}{\pi^2\sqrt{3}}\left[\arcsin\frac{1}{4}+2(P-1)\arcsin\frac{1}{2\sqrt{2(3-\chi^2)}}+(P-1)\arcsin\frac{\chi(3-2\chi^2)}{2(3-\chi^2)}\right.\\
&\qquad\left.+(P-1)(P-2)\arcsin\frac{\chi(3-2\chi)}{2(3-\chi^2)}\right],\\
V_o &= \frac{2\omega(\chi)(\varepsilon_T+\sigma^2)}{\pi}+\frac{4\omega(\chi)}{\pi^2}\left[2\arcsin\frac{2-\chi^2}{2(3-\chi^2)}+2\arcsin\frac{\chi}{2(3-\chi^2)}\right.\\
&\qquad\left.+(P-2)\arcsin\frac{2+\chi-2\chi^2}{2(2-\chi)(1+\chi)}+4(P-2)\arcsin\frac{\chi}{2\sqrt{(3-\chi^2)(1+\chi)}}\right.\\
&\qquad\left.+(P-2)(P-3)\arcsin\frac{\chi}{2(1+\chi)}\right]. \tag*{(108)}
\end{aligned} V d V o = π 3 2 ( ε T + σ 2 ) + π 2 3 4 [ arcsin 4 1 + 2 ( P − 1 ) arcsin 2 2 ( 3 − χ 2 ) 1 + ( P − 1 ) arcsin 2 ( 3 − χ 2 ) χ ( 3 − 2 χ 2 ) + ( P − 1 ) ( P − 2 ) arcsin 2 ( 3 − χ 2 ) χ ( 3 − 2 χ ) ] , = π 2 ω ( χ ) ( ε T + σ 2 ) + π 2 4 ω ( χ ) [ 2 arcsin 2 ( 3 − χ 2 ) 2 − χ 2 + 2 arcsin 2 ( 3 − χ 2 ) χ + ( P − 2 ) arcsin 2 ( 2 − χ ) ( 1 + χ ) 2 + χ − 2 χ 2 + 4 ( P − 2 ) arcsin 2 ( 3 − χ 2 ) ( 1 + χ ) χ + ( P − 2 ) ( P − 3 ) arcsin 2 ( 1 + χ ) χ ] . (108) The first term comes from the teacher output and the label noise, which are independent of the student preactivations at the initial state, and the second from the student output.
Expansion of the generalization error. Since the right-hand sides of Eqs. (104) –(106) are constant at this order, the order parameters change linearly at small α \alpha α
d ε g d α ∣ α = 0 = ∑ ∂ ε g ∂ R d R d α + ∑ ∂ ε g ∂ Q d Q d α = − η A + η 2 B , (109) \left.\frac{d\varepsilon_g}{d\alpha}\right|_{\alpha=0}=\sum\frac{\partial\varepsilon_g}{\partial R}\frac{dR}{d\alpha}+\sum\frac{\partial\varepsilon_g}{\partial Q}\frac{dQ}{d\alpha}=-\eta A+\eta^2B, \tag*{(109)} d α d ε g α = 0 = ∑ ∂ R ∂ ε g d α d R + ∑ ∂ Q ∂ ε g d α d Q = − η A + η 2 B , (109) with the sums running over all entries of the order parameters. Differentiating Eq. (81) through Eqs. (43) and (72) gives the derivatives at the initial state,
∂ ε g ∂ R l , l ′ ∣ C N N = − 4 P s c π 2 ( 1 + T ) , ∂ ε g ∂ Q l , l ′ ∣ C N N = − H l , l ′ , (110) \left.\frac{\partial\varepsilon_g}{\partial R_{l,l'}}\right|_{\mathrm{CNN}}=-\frac{4Ps_c}{\pi\sqrt{2(1+T)}},\qquad\left.\frac{\partial\varepsilon_g}{\partial Q_{l,l'}}\right|_{\mathrm{CNN}}=-H_{l,l'}, \tag*{(110)} ∂ R l , l ′ ∂ ε g CNN = − π 2 ( 1 + T ) 4 P s c , ∂ Q l , l ′ ∂ ε g CNN = − H l , l ′ , (110) ∂ ε g ∂ R l , l ′ ( p ) ∣ L C N = − 4 s c π 2 ( 1 + T ) , ∂ ε g ∂ Q l , l ′ ( p , q ) ∣ L C N = Ψ p , q S ( p + l ) P , ( q + l ′ ) P . (111) \left.\frac{\partial\varepsilon_g}{\partial R^{(p)}_{l,l'}}\right|_{\mathrm{LCN}}=-\frac{4s_c}{\pi\sqrt{2(1+T)}},\qquad\left.\frac{\partial\varepsilon_g}{\partial Q^{(p,q)}_{l,l'}}\right|_{\mathrm{LCN}}=\Psi_{p,q}S_{(p+l)_P,(q+l')_P}. \tag*{(111)} ∂ R l , l ′ ( p ) ∂ ε g LCN = − π 2 ( 1 + T ) 4 s c , ∂ Q l , l ′ ( p , q ) ∂ ε g LCN = Ψ p , q S ( p + l ) P , ( q + l ′ ) P . (111) Inserting the equations of motion, the CNN gives
A C N N = 4 T P 2 s c 2 π 2 ( 1 + T ) + 2 [ H d 2 k s + ( 1 − 1 k s ) H o 2 ] , (112) A_{\mathrm{CNN}}=\frac{4TP^2s_c^2}{\pi^2(1+T)}+2\left[\frac{H_d^2}{k_s}+\left(1-\frac{1}{k_s}\right)H_o^2\right], \tag*{(112)} A CNN = π 2 ( 1 + T ) 4 T P 2 s c 2 + 2 [ k s H d 2 + ( 1 − k s 1 ) H o 2 ] , (112) B C N N = P { V d [ H d k s + ( 1 − 1 k s ) c H o ] + V o [ ( P − 1 ) c H d k s + ( 1 − 1 k s ) ( 1 + ( P − 2 ) c ) H o ] } , (113) B_{\mathrm{CNN}}=P\left\{V_d\left[\frac{H_d}{k_s}+\left(1-\frac{1}{k_s}\right)cH_o\right]+V_o\left[\frac{(P-1)cH_d}{k_s}+\left(1-\frac{1}{k_s}\right)(1+(P-2)c)H_o\right]\right\}, \tag*{(113)} B CNN = P { V d [ k s H d + ( 1 − k s 1 ) c H o ] + V o [ k s ( P − 1 ) c H d + ( 1 − k s 1 ) ( 1 + ( P − 2 ) c ) H o ] } , (113) with V d V_d V d V o V_o V o χ = c \chi=c χ = c
A L C N = 4 T P s c 2 π 2 ( 1 + T ) + 2 P π 2 [ n d 3 + n o ] , B L C N = P π [ n d 3 V d + n o V o ] , (114) A_{\mathrm{LCN}}=\frac{4TPs_c^2}{\pi^2(1+T)}+\frac{2P}{\pi^2}\left[\frac{n_d}{3}+n_o\right],\qquad B_{\mathrm{LCN}}=\frac{P}{\pi}\left[\frac{n_d}{\sqrt{3}}V_d+n_oV_o\right], \tag*{(114)} A LCN = π 2 ( 1 + T ) 4 T P s c 2 + π 2 2 P [ 3 n d + n o ] , B LCN = π P [ 3 n d V d + n o V o ] , (114) with V d V_d V d V o V_o V o χ = 0 \chi=0 χ = 0
n d = c 2 + 1 − c 2 k s , n o = ( 1 − c 2 ) ( 1 − 1 k s ) + ( P − 1 ) c 2 . (115) n_d=c^2+\frac{1-c^2}{k_s},\qquad n_o=(1-c^2)\left(1-\frac{1}{k_s}\right)+(P-1)c^2. \tag*{(115)} n d = c 2 + k s 1 − c 2 , n o = ( 1 − c 2 ) ( 1 − k s 1 ) + ( P − 1 ) c 2 . (115) All four coefficients are positive, since every term is. They do not depend on k t k_t k t 1 / k s 1/k_s 1/ k s c = 0 c=0 c = 0 A C N N = P A L C N A_{\mathrm{CNN}}=PA_{\mathrm{LCN}} A CNN = P A LCN B C N N = P B L C N B_{\mathrm{CNN}}=PB_{\mathrm{LCN}} B CNN = P B LCN
Optimal learning rate and boundary. Within the expansion, the error of each student initially decreases if and only if η < A / B \eta< A/B η < A / B α \alpha α η \eta η η ∗ = A / ( 2 B ) \eta^* = A/(2B) η ∗ = A / ( 2 B )
ε ∗ ( α ) = ε 0 − α A 2 4 B . (116) \varepsilon^*(\alpha) = \varepsilon_0 - \alpha\frac{A^2}{4B}. \tag*{(116)} ε ∗ ( α ) = ε 0 − α 4 B A 2 . (116) The CNN has the lower error when
α [ A C N N 2 4 B C N N − A L C N 2 4 B L C N ] > ε 0 C N N − ε 0 L C N = 2 P ( P − 1 ) π arcsin c 2 . (117) \alpha\left[\frac{A_{\mathrm{CNN}}^2}{4B_{\mathrm{CNN}}} - \frac{A_{\mathrm{LCN}}^2}{4B_{\mathrm{LCN}}}\right] > \varepsilon_0^{\mathrm{CNN}} - \varepsilon_0^{\mathrm{LCN}} = \frac{2P(P-1)}{\pi}\arcsin\frac{c}{2}. \tag*{(117)} α [ 4 B CNN A CNN 2 − 4 B LCN A LCN 2 ] > ε 0 CNN − ε 0 LCN = π 2 P ( P − 1 ) arcsin 2 c . (117) The right-hand side is positive for c > 0 c > 0 c > 0 c ∈ [ 0 , 1 ) c \in[0,1) c ∈ [ 0 , 1 ) P ≤ 12 P \le12 P ≤ 12 k s ≤ P k_s \le P k s ≤ P 0.25 ≤ T ≤ 10 0.25 \le T \le10 0.25 ≤ T ≤ 10 0 ≤ σ ≤ 4 0 \le\sigma\le4 0 ≤ σ ≤ 4 c ∗ ( α ) c^*(\alpha) c ∗ ( α ) (117) are equal,
c ∗ ( α ) = 2 sin [ π α 2 P ( P − 1 ) Δ ( c ∗ ( α ) ) ] , Δ ( c ) ≡ A C N N 2 4 B C N N − A L C N 2 4 B L C N , (118) c^*(\alpha) = 2\sin\left[\frac{\pi\alpha}{2P(P-1)}\Delta(c^*(\alpha))\right], \qquad\Delta(c) \equiv\frac{A_{\mathrm{CNN}}^2}{4B_{\mathrm{CNN}}} - \frac{A_{\mathrm{LCN}}^2}{4B_{\mathrm{LCN}}}, \tag*{(118)} c ∗ ( α ) = 2 sin [ 2 P ( P − 1 ) π α Δ ( c ∗ ( α )) ] , Δ ( c ) ≡ 4 B CNN A CNN 2 − 4 B LCN A LCN 2 , (118) an implicit equation, since A A A B B B c c c α \alpha α c ∗ c^* c ∗ (118) to first order in α \alpha α A A A B B B c = 0 c=0 c = 0 c ∗ ( α ) = c 0 α + O ( α 2 ) c^*(\alpha) = c_0\alpha+ O(\alpha^2) c ∗ ( α ) = c 0 α + O ( α 2 )
c 0 = 1 π β ( 1 + 2 T 1 + T − 2 3 k s ) 2 , (119) c_0 = \frac{1}{\pi\beta}\left(1+\frac{2T}{1+T}-\frac{2}{3k_s}\right)^2, \tag*{(119)} c 0 = π β 1 ( 1 + 1 + T 2 T − 3 k s 2 ) 2 , (119) where
β = 2 3 k s [ ε ˉ + P − 1 3 + 2 π arcsin 1 4 ] + ( 1 − 1 k s ) [ ε ˉ + P − 2 3 + 4 π arcsin 1 3 ] , (120) \beta= \frac{2}{3k_s}\left[\bar{\varepsilon}+\frac{P-1}{3}+\frac{2}{\pi}\arcsin\frac{1}{4}\right] + \left(1-\frac{1}{k_s}\right)\left[\bar{\varepsilon}+\frac{P-2}{3}+\frac{4}{\pi}\arcsin\frac{1}{3}\right], \tag*{(120)} β = 3 k s 2 [ ε ˉ + 3 P − 1 + π 2 arcsin 4 1 ] + ( 1 − k s 1 ) [ ε ˉ + 3 P − 2 + π 4 arcsin 3 1 ] , (120) and ε ˉ = 2 P π arcsin T 1 + T + σ 2 \bar{\varepsilon} = \frac{2P}{\pi}\arcsin\frac{T}{1+T}+\sigma^2 ε ˉ = π 2 P arcsin 1 + T T + σ 2
Noise-floor calculation
In this appendix, we investigate the long-time behavior where the network reaches near-perfect recovery. For η → 0 \eta\to0 η → 0 η \eta η k t ≤ k s ≤ P k_t \le k_s \le P k t ≤ k s ≤ P l , l ′ , l ′ ′ l,l',l'' l , l ′ , l ′′ p , q , r , s p,q,r,s p , q , r , s
CNN student
Write u l = w ^ l − k s / k t w l u_l = \hat{w}_l-\sqrt{k_s/k_t}\,w_l u l = w ^ l − k s / k t w l § 9 .§ 1 , for the deviation of the student’s kernel from perfect recovery, and define the single-tap error field
ε p ( l ) ≡ ν p ( l ) − μ p ( l ) = u l ⋅ x p k s D . (121) \varepsilon_p^{(l)} \equiv\nu_p^{(l)}-\mu_p^{(l)} = \frac{u_l \cdot x_p}{\sqrt{k_sD}}. \tag*{(121)} ε p ( l ) ≡ ν p ( l ) − μ p ( l ) = k s D u l ⋅ x p . (121) Its covariance follows from the two-point functions of Appendix § 9 .§ 1 ,
⟨ ε p ( l ) ε q ( l ′ ) ⟩ = S p , q M l , l ′ , M ≡ Q − R − R T + T , (122) \left\langle\varepsilon_p^{(l)}\varepsilon_q^{(l')}\right\rangle= S_{p,q}M_{l,l'}, \qquad M \equiv Q-R-R^{\mathsf{T}}+T, \tag*{(122)} ⟨ ε p ( l ) ε q ( l ′ ) ⟩ = S p , q M l , l ′ , M ≡ Q − R − R T + T , (122) where M M M k s × k s k_s \times k_s k s × k s p p p
e p ≡ λ ^ p − λ p = ∑ l = 0 k s − 1 ϵ ( p + l ) P ( l ) . (123) e_p \equiv\hat{\lambda}_p-\lambda_p=\sum_{l=0}^{k_s-1}\epsilon^{(l)}_{(p+l)P}. \tag*{(123)} e p ≡ λ ^ p − λ p = l = 0 ∑ k s − 1 ϵ ( p + l ) P ( l ) . (123) so that its covariance is the effective matrix built from M M M
⟨ e p e q ⟩ = M p , q e f f = ∑ l , l ′ = 0 k s − 1 M l , l ′ S ( p + l ) P , ( q + l ′ ) P . (124) \langle e_p e_q\rangle=M^{\mathrm{eff}}_{p,q}=\sum_{l,l'=0}^{k_s-1}M_{l,l'}S_{(p+l)P,(q+l')P}. \tag*{(124)} ⟨ e p e q ⟩ = M p , q eff = l , l ′ = 0 ∑ k s − 1 M l , l ′ S ( p + l ) P , ( q + l ′ ) P . (124) Near perfect recovery and for small learning rate η \eta η M M M η \eta η e p e_p e p η \sqrt{\eta} η ⟨ e p λ q ⟩ = ( R − T ) p , q \langle e_p\lambda_q\rangle=(R-T)_{p,q} ⟨ e p λ q ⟩ = ( R − T ) p , q η \eta η
Expanding the residual for small field errors,
Δ y = − ∑ p = 0 P − 1 [ g ( λ p + e p ) − g ( λ p ) ] = − ∑ p = 0 P − 1 g ′ ( λ p ) e p + O ( e 2 ) , (125) \Delta y=-\sum_{p=0}^{P-1}\left[g(\lambda_p+e_p)-g(\lambda_p)\right]=-\sum_{p=0}^{P-1}g'(\lambda_p)e_p+O(e^2), \tag*{(125)} Δ y = − p = 0 ∑ P − 1 [ g ( λ p + e p ) − g ( λ p ) ] = − p = 0 ∑ P − 1 g ′ ( λ p ) e p + O ( e 2 ) , (125) so that, writing W p , q ≡ ⟨ g ′ ( λ p ) g ′ ( λ q ) ⟩ W_{p,q}\equiv\langle g'(\lambda_p)g'(\lambda_q)\rangle W p , q ≡ ⟨ g ′ ( λ p ) g ′ ( λ q )⟩
ε ∞ = ⟨ ( Δ y ) 2 ⟩ = ∑ p , q = 0 P − 1 W p , q ⟨ e p e q ⟩ + O ( η 2 ) = ∑ p , q = 0 P − 1 W p , q M p , q e f f + O ( η 2 ) , (126) \varepsilon^\infty=\langle(\Delta y)^2\rangle=\sum_{p,q=0}^{P-1}W_{p,q}\langle e_p e_q\rangle+O(\eta^2)=\sum_{p,q=0}^{P-1}W_{p,q}M^{\mathrm{eff}}_{p,q}+O(\eta^2), \tag*{(126)} ε ∞ = ⟨( Δ y ) 2 ⟩ = p , q = 0 ∑ P − 1 W p , q ⟨ e p e q ⟩ + O ( η 2 ) = p , q = 0 ∑ P − 1 W p , q M p , q eff + O ( η 2 ) , (126) where we have used
⟨ g ′ ( λ p ) g ′ ( λ q ) e p e q ⟩ = W p , q ⟨ e p e q ⟩ + O ( η 2 ) . (127) \langle g'(\lambda_p)g'(\lambda_q)e_p e_q\rangle=W_{p,q}\langle e_p e_q\rangle+O(\eta^2). \tag*{(127)} ⟨ g ′ ( λ p ) g ′ ( λ q ) e p e q ⟩ = W p , q ⟨ e p e q ⟩ + O ( η 2 ) . (127) Using Eq. (45) , we find
W p , p = 2 π 1 + 2 T , W p , q = 2 π ( 1 + T ) 2 − T 2 c 2 ( p ≠ q ) . (128) W_{p,p}=\frac{2}{\pi\sqrt{1+2T}},\qquad W_{p,q}=\frac{2}{\pi\sqrt{(1+T)^2-T^2c^2}}\quad(p\ne q). \tag*{(128)} W p , p = π 1 + 2 T 2 , W p , q = π ( 1 + T ) 2 − T 2 c 2 2 ( p = q ) . (128) It is convenient to define the k s × k s k_s \times k_s k s × k s
Ω l , l ′ ≡ ∑ p , q = 0 P − 1 W p , q S ( p + l ) P , ( q + l ′ ) P , (129) \Omega_{l,l'}\equiv\sum_{p,q=0}^{P-1}W_{p,q}S_{(p+l)P,(q+l')P}, \tag*{(129)} Ω l , l ′ ≡ p , q = 0 ∑ P − 1 W p , q S ( p + l ) P , ( q + l ′ ) P , (129) so that substituting (124) into (126) gives
ε ∞ = ∑ l , l ′ = 0 k s − 1 M l , l ′ Ω l , l ′ = tr ( M Ω ) . (130) \varepsilon^\infty=\sum_{l,l'=0}^{k_s-1}M_{l,l'}\Omega_{l,l'}=\operatorname{tr}(M\Omega). \tag*{(130)} ε ∞ = l , l ′ = 0 ∑ k s − 1 M l , l ′ Ω l , l ′ = tr ( M Ω ) . (130) We now need the equation of motion for M M M (122) and the ODEs of Appendix § 9 , we find
M ˙ l , l ′ = η k s ∑ p = 0 P − 1 g ′ ( λ ^ p ) ϵ ( p + l ) P ( l ′ ) Δ y + ( l ↔ l ′ ) + Q ˙ l , l ′ ∣ O ( η 2 ) . (131) \dot{M}_{l,l'}=\frac{\eta}{k_s}\sum_{p=0}^{P-1}g'(\hat{\lambda}_p)\epsilon^{(l')}_{(p+l)P}\Delta y+(l\leftrightarrow l')+\dot{Q}_{l,l'}\lvert_{O(\eta^2)}. \tag*{(131)} M ˙ l , l ′ = k s η p = 0 ∑ P − 1 g ′ ( λ ^ p ) ϵ ( p + l ) P ( l ′ ) Δ y + ( l ↔ l ′ ) + Q ˙ l , l ′ ∣ O ( η 2 ) . (131) In the first term, we can replace g ′ ( λ ^ p ) g'(\hat{\lambda}_p) g ′ ( λ ^ p ) g ′ ( λ p ) g'(\lambda_p) g ′ ( λ p ) (125) , we find
⟨ g ′ ( λ ^ p ) ϵ ( p + l ) P ( l ′ ) Δ y ⟩ = ⟨ g ′ ( λ p ) ϵ ( p + l ) P ( l ′ ) Δ y ⟩ + o ( η ) = − ∑ q = 0 P − 1 ∑ l ′ ′ = 0 k s − 1 W p , q S ( p + l ) P , ( q + l ′ ′ ) P M l ′ ′ , l ′ + o ( η ) . (132) \left\langle g'(\hat{\lambda}_p)\epsilon_{(p+l)P}^{(l')}\Delta y\right\rangle= \left\langle g'(\lambda_p)\epsilon_{(p+l)P}^{(l')}\Delta y\right\rangle+ o(\eta) = -\sum_{q=0}^{P-1}\sum_{l''=0}^{k_s-1} W_{p,q}S_{(p+l)P,(q+l'')P}M_{l'',l'} + o(\eta). \tag*{(132)} ⟨ g ′ ( λ ^ p ) ϵ ( p + l ) P ( l ′ ) Δ y ⟩ = ⟨ g ′ ( λ p ) ϵ ( p + l ) P ( l ′ ) Δ y ⟩ + o ( η ) = − q = 0 ∑ P − 1 l ′′ = 0 ∑ k s − 1 W p , q S ( p + l ) P , ( q + l ′′ ) P M l ′′ , l ′ + o ( η ) . (132) Using Eq. (64) and keeping terms up to quadratic order in η \eta η M = O ( η ) M=O(\eta) M = O ( η )
M ˙ = − η k s ( Ω M + M Ω ) + η 2 σ 2 k s 2 Ω + o ( η 2 ) . (133) \dot{M}=-\frac{\eta}{k_s}(\Omega M+M\Omega)+\frac{\eta^2\sigma^2}{k_s^2}\Omega+o(\eta^2). \tag*{(133)} M ˙ = − k s η ( Ω M + M Ω ) + k s 2 η 2 σ 2 Ω + o ( η 2 ) . (133) Setting M ˙ = 0 \dot{M}=0 M ˙ = 0
Ω M + M Ω = η σ 2 k s Ω . (134) \Omega M+M\Omega=\frac{\eta\sigma^2}{k_s}\Omega. \tag*{(134)} Ω M + M Ω = k s η σ 2 Ω. (134) It can be checked that Ω \Omega Ω c < 1 c<1 c < 1 (134) has the unique solution
M l , l ′ = η σ 2 2 k s δ l , l ′ . (135) M_{l,l'}=\frac{\eta\sigma^2}{2k_s}\delta_{l,l'}. \tag*{(135)} M l , l ′ = 2 k s η σ 2 δ l , l ′ . (135) Substituting (135) into (130) ,
ϵ ∞ = η σ 2 2 k s tr ( Ω ) . (136) \epsilon^\infty=\frac{\eta\sigma^2}{2k_s}\operatorname{tr}(\Omega). \tag*{(136)} ϵ ∞ = 2 k s η σ 2 tr ( Ω ) . (136) From (129) and using S ( p + l ) P , ( q + l ) P = S p , q S_{(p+l)P,(q+l)P}=S_{p,q} S ( p + l ) P , ( q + l ) P = S p , q
tr Ω = ∑ l = 0 k s − 1 ∑ p , q = 0 P − 1 W p , q S ( p + l ) P , ( q + l ) P = k s ∑ p , q = 0 P − 1 W p , q S p , q . (137) \operatorname{tr}\Omega=\sum_{l=0}^{k_s-1}\sum_{p,q=0}^{P-1}W_{p,q}S_{(p+l)P,(q+l)P}=k_s\sum_{p,q=0}^{P-1}W_{p,q}S_{p,q}. \tag*{(137)} tr Ω = l = 0 ∑ k s − 1 p , q = 0 ∑ P − 1 W p , q S ( p + l ) P , ( q + l ) P = k s p , q = 0 ∑ P − 1 W p , q S p , q . (137) Finally, using (128) we recover (31) ,
ϵ ∞ , C N N ( c ) = η σ 2 π [ P 1 + 2 T + P ( P − 1 ) c ( 1 + T ) 2 − T 2 c 2 ] , (138) \epsilon^{\infty,\mathrm{CNN}}(c)=\frac{\eta\sigma^2}{\pi}\left[\frac{P}{\sqrt{1+2T}}+\frac{P(P-1)c}{\sqrt{(1+T)^2-T^2c^2}}\right], \tag*{(138)} ϵ ∞ , CNN ( c ) = π η σ 2 [ 1 + 2 T P + ( 1 + T ) 2 − T 2 c 2 P ( P − 1 ) c ] , (138) with no dependence on either kernel size.
LCN student
The LCN calculation follows the same steps but every position carries its own kernel, so the deviation from perfect recovery has a position label,
u l ( p ) ≡ w ^ l ( p ) − k s / k t w l , (139) u_l^{(p)}\equiv\hat{w}_l^{(p)}-\sqrt{k_s/k_t}\,w_l, \tag*{(139)} u l ( p ) ≡ w ^ l ( p ) − k s / k t w l , (139) and so does the single-tap error field
ϵ r ( p , l ) ≡ ν r ( p , l ) − μ r ( l ) = u l ( p ) ⋅ x r k s D . (140) \epsilon_r^{(p,l)}\equiv\nu_r^{(p,l)}-\mu_r^{(l)}=\frac{u_l^{(p)}\cdot x_r}{\sqrt{k_sD}}. \tag*{(140)} ϵ r ( p , l ) ≡ ν r ( p , l ) − μ r ( l ) = k s D u l ( p ) ⋅ x r . (140) Its covariance is
⟨ ϵ r ( p , l ) ϵ r ′ ( q , l ′ ) ⟩ = S r , r ′ M l , l ′ ( p , q ) , M l , l ′ ( p , q ) ≡ u l ( p ) ⋅ u l ′ ( q ) k s D = [ Q ( p , q ) − R ( p ) − ( R ( q ) ) T + T ] l , l ′ . (141) \left\langle\epsilon_r^{(p,l)}\epsilon_{r'}^{(q,l')}\right\rangle=S_{r,r'}M_{l,l'}^{(p,q)},\qquad M_{l,l'}^{(p,q)}\equiv\frac{u_l^{(p)}\cdot u_{l'}^{(q)}}{k_sD}=\left[Q^{(p,q)}-R^{(p)}-\left(R^{(q)}\right)^{\mathsf T}+T\right]_{l,l'}. \tag*{(141)} ⟨ ϵ r ( p , l ) ϵ r ′ ( q , l ′ ) ⟩ = S r , r ′ M l , l ′ ( p , q ) , M l , l ′ ( p , q ) ≡ k s D u l ( p ) ⋅ u l ′ ( q ) = [ Q ( p , q ) − R ( p ) − ( R ( q ) ) T + T ] l , l ′ . (141) which vanishes at perfect recovery. The error in the preactivation at position p p p
e p ≡ λ ^ p L C N − λ p = ∑ l = 0 k s − 1 ϵ ( p + l ) P ( p , l ) . (142) e_p \equiv\hat{\lambda}^{\mathrm{LCN}}_p-\lambda_p=\sum_{l=0}^{k_s-1}\epsilon^{(p,l)}_{(p+l)P}. \tag*{(142)} e p ≡ λ ^ p LCN − λ p = l = 0 ∑ k s − 1 ϵ ( p + l ) P ( p , l ) . (142) The equation of motion for M l , l ′ ( p , q ) M^{(p,q)}_{l,l'} M l , l ′ ( p , q ) η \eta η
M ˙ l , l ′ ( p , q ) = − η k s ∑ r = 0 P − 1 ∑ l ′ ′ = 0 k s − 1 W p , r S ( p + l ) P , ( r + l ′ ′ ) P M l ′ , l ′ ′ ( q , r ) + ( p ↔ q , l ↔ l ′ ) + η 2 σ 2 k s 2 S ( p + l ) P , ( q + l ′ ) P W p , q . (143) \dot{M}^{(p,q)}_{l,l'}=-\frac{\eta}{k_s}\sum_{r=0}^{P-1}\sum_{l''=0}^{k_s-1}W_{p,r}S_{(p+l)P,(r+l'')P}M^{(q,r)}_{l',l''}+(p\leftrightarrow q,\ l\leftrightarrow l')+\frac{\eta^2\sigma^2}{k_s^2}S_{(p+l)P,(q+l')P}W_{p,q}. \tag*{(143)} M ˙ l , l ′ ( p , q ) = − k s η r = 0 ∑ P − 1 l ′′ = 0 ∑ k s − 1 W p , r S ( p + l ) P , ( r + l ′′ ) P M l ′ , l ′′ ( q , r ) + ( p ↔ q , l ↔ l ′ ) + k s 2 η 2 σ 2 S ( p + l ) P , ( q + l ′ ) P W p , q . (143) To solve the fixed-point equation M ˙ l , l ′ ( p , q ) = 0 \dot{M}^{(p,q)}_{l,l'}=0 M ˙ l , l ′ ( p , q ) = 0
M l , l ′ ( p , q ) = m δ p , q δ l , l ′ , (144) M^{(p,q)}_{l,l'}=m\delta_{p,q}\delta_{l,l'}, \tag*{(144)} M l , l ′ ( p , q ) = m δ p , q δ l , l ′ , (144) with m m m
M ˙ l , l ′ ( p , q ) = [ − 2 η m k s + η 2 σ 2 k s 2 ] W p , q S ( p + l ) P , ( q + l ′ ) P = 0 , (145) \dot{M}^{(p,q)}_{l,l'}=\left[-\frac{2\eta m}{k_s}+\frac{\eta^2\sigma^2}{k_s^2}\right]W_{p,q}S_{(p+l)P,(q+l')P}=0, \tag*{(145)} M ˙ l , l ′ ( p , q ) = [ − k s 2 η m + k s 2 η 2 σ 2 ] W p , q S ( p + l ) P , ( q + l ′ ) P = 0 , (145) which is solved by m = η σ 2 / ( 2 k s ) m=\eta\sigma^2/(2k_s) m = η σ 2 / ( 2 k s )
M l , l ′ ( p , q ) = η σ 2 2 k s δ p , q δ l , l ′ . (146) M^{(p,q)}_{l,l'}=\frac{\eta\sigma^2}{2k_s}\delta_{p,q}\delta_{l,l'}. \tag*{(146)} M l , l ′ ( p , q ) = 2 k s η σ 2 δ p , q δ l , l ′ . (146) Substituting (146) into ⟨ e p e q ⟩ = ∑ l , l ′ M l , l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P \langle e_p e_q\rangle=\sum_{l,l'}M^{(p,q)}_{l,l'}S_{(p+l)P,(q+l')P} ⟨ e p e q ⟩ = ∑ l , l ′ M l , l ′ ( p , q ) S ( p + l ) P , ( q + l ′ ) P
⟨ e p e q ⟩ = η σ 2 2 δ p , q , (147) \langle e_p e_q\rangle=\frac{\eta\sigma^2}{2}\delta_{p,q}, \tag*{(147)} ⟨ e p e q ⟩ = 2 η σ 2 δ p , q , (147) which is (34) . Only the diagonal of (126) survives, giving (32) ,
ϵ ∞ , L C N = η σ 2 2 ∑ p = 0 P − 1 W p , p = η σ 2 P π 1 + 2 T , (148) \epsilon^{\infty,\mathrm{LCN}}=\frac{\eta\sigma^2}{2}\sum_{p=0}^{P-1}W_{p,p}=\frac{\eta\sigma^2P}{\pi\sqrt{1+2T}}, \tag*{(148)} ϵ ∞ , LCN = 2 η σ 2 p = 0 ∑ P − 1 W p , p = π 1 + 2 T η σ 2 P , (148) independent of c c c
Phase diagram: numerical method
For each student and each correlation c ∈ { 0 , 0.025 , … , 0.975 } c\in\{0,0.025,\ldots,0.975\} c ∈ { 0 , 0.025 , … , 0.975 } (17) and (18) from the random initial condition at leading order in D D D (19) and (20) without the O D ( D − 1 / 2 ) O_D(D^{-1/2}) O D ( D − 1/2 ) 10 − 4 10^{-4} 1 0 − 4 α F \alpha_F α F 10 − 2 10^{-2} 1 0 − 2 10 4 10^4 1 0 4 α F \alpha_F α F log ϵ g \log\epsilon_g log ϵ g log η \log\eta log η c = 0 c=0 c = 0 ϵ L C N ∗ ( P α F ) = ϵ C N N ∗ ( α F ) \epsilon^*_{\mathrm{LCN}}(P\alpha_F)=\epsilon^*_{\mathrm{CNN}}(\alpha_F) ϵ LCN ∗ ( P α F ) = ϵ CNN ∗ ( α F )
Figure 8 shows the optimal learning rate and the corresponding error for three values of c c c c > 0 c > 0 c > 0
Figure 9 shows the phase diagram for other parameter values, each differing from Figure 6 in a single parameter. In all cases, for every c > 0 c > 0 c > 0
Activation dependence: ReLU
The analytic results use g ( z ) = erf ( z / 2 ) g(z) = \operatorname{erf}(z/\sqrt{2}) g ( z ) = erf ( z / 2 ) Figure 10 repeats the noise-floor comparison of Figure 5 with the ReLU activation g ( z ) = max ( 0 , z ) g(z) = \max(0, z) g ( z ) = max ( 0 , z ) erf \operatorname{erf} erf c c c P P P § 3.1 holds for ReLU at c = 0 c = 0 c = 0 c = 0.5 c = 0.5 c = 0.5 Figure 11 ).
Figure 10. Rescaled noise floor ε ∞ / ( η σ 2 ) \varepsilon^{\infty}/(\eta\sigma^{2}) ε ∞ / ( η σ 2 ) ε ∞ \varepsilon^{\infty} ε ∞ Figure 5 ), the CNN’s floor grows with the data correlation c c c g ( z ) = max ( 0 , z ) g(z) = \max(0,z) g ( z ) = max ( 0 , z ) k s = k t = 2 k_s = k_t = 2 k s = k t = 2 P = 4 P = 4 P = 4 D = 400 D = 400 D = 400 η = 0.1 \eta= 0.1 η = 0.1 T = 1 T = 1 T = 1 σ = 1 \sigma= 1 σ = 1 α = 800 \alpha= 800 α = 800
Figure 11. Rescaling test of Figure 2 with the ReLU activation. Generalization errors ε g C N N ( α ) \varepsilon_g^{\mathrm{CNN}}(\alpha) ε g CNN ( α ) ε g L C N ( P α ) \varepsilon_g^{\mathrm{LCN}}(P\alpha) ε g LCN ( P α ) c = 0 c = 0 c = 0 c = 0.5 c = 0.5 c = 0.5 g ( z ) = max ( 0 , z ) g(z) = \max(0,z) g ( z ) = max ( 0 , z ) k t = 2 k_t = 2 k t = 2 k s = 3 k_s = 3 k s = 3 P = 4 P = 4 P = 4 η = 0.2 \eta= 0.2 η = 0.2 σ = 0 \sigma= 0 σ = 0 T = 1 T = 1 T = 1 D = 1000 D = 1000 D = 1000
