betaFeedback

WorksComputer Science

Solvable dynamical model of in-context learning

Abstract

Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer single-head linear attention model, where an empirical risk minimization is defined for learning. Given a finite number of training examples, we determine the exact solution of network-parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning especially from a learning dynamics perspective.

Introduction

Inspired by the study of attention mechanisms in the brain [1–3], an engineering generalization of the biological attention to the self-attention dominant in the current transformer computation has started a revolution in natural language processing first and recently the artificial general intelligence including superhuman capabilities in coding, mathematical reasoning, and document comprehension [4, 5]. The self-attention captures long-range dependencies between different parts of the input (e.g., image or text), trained with a simple cost of next-token prediction [6], establishing a computer science foundation of large language model (LLM) [7].

As one of the astonishing abilities, the transformer is able to do in-context inference [8], i.e., the pre-trained transformer is able to accomplish previously-unseen complicated tasks by reasoning on a short prompt especially without a need for updating the model parameters. This remarkable property has recently captured intense research interest from theoretical machine learning and statistical physics communities [9, 10, 11, 12, 13, 14, 15, 16, 17]. The theoretical efforts focused on a single-layer linear attention [9, 13, 14, 15, 16], which revealed that a sufficient pre-training task diversity triggers the emergence of in-context inference. This phase transition was first empirically confirmed in the in-context learning of a regression task [12], later analytically studied in the asymptotic high-dimensional limit [14], and formulated recently in a disordered spin glass framework where the network parameters act as spins while training data act as quenched coupling between spins. Recent theoretical works start to focus on the learning dynamics of transformer in linear regression tasks, especially on the population loss (given infinite training prompts) [16, 18] and on the neural scaling law in training time, depth and context length [19]. A recent work derives non-asymptotic closed-form characterization of the learning dynamics for vector-valued regression [20].

Based on our previous work of statistical physics analysis of in-context learning (ICL), we continue to explore the learning dynamics of an empirical risk minimization in this work. We first describe the model setting which allows a closed-form solution of the network parameter dynamics, from which we can predict the behavior of individual concrete learning trajectory and related test performance on unseen linear regression examples. We also derive a data-dependent statistical mechanics formalism to explain the stationary limit of the learning dynamics, supported by numerical experiments of single instance learning. Furthermore, we simplify our analytic results in a specific setting, e.g., setting a particular set of network parameters to zero, thereby isolating the key factor in the form of a data-dependent eigenvalue spectrum.

Linear Attention Model

Given an input matrix X∈RD×NX \in\mathbb{R}^{D \times N}, a single-head single-layer softmax-based attention model is formulated as follows,

Atten⁡(K,Q,V)=Vsoftmax⁡(QTKD)(1)\operatorname{Atten}(K,Q,V) = V\operatorname{softmax}\left(\frac{Q^{\mathsf{T}}K}{\sqrt{D}}\right) \tag*{(1)}
K=WKX; Q=WQX; V=WVX,(2)K = W_KX;\ Q = W_QX;\ V = W_VX, \tag*{(2)}

where W{Q,K,V}W_{\{Q,K,V\}} indicates the query, key or value matrix of dimension D×DD \times D, respectively. DD is also called the embedding dimension of token xi{i=1,…,N}x_i\{i = 1,\ldots,N\}. All vectors in this paper are defined in a column form, i.e., x∈RD×1\mathbf{x} \in\mathbb{R}^{D \times1}. The above transformer block outputs an equal-size matrix Y=Atten⁡(K,Q,V)Y = \operatorname{Atten}(K,Q,V). The softmax operation is normalized for each column, representing a weight score for different key vectors (transformed tokens). To facilitate the following theoretical analysis, we replace the softmax operation by a linear attention, and merge the key and query projection matrices into a single parameter matrix WQTWK=WW_Q^{\mathsf{T}}W_K = W, and set the value projection matrix WV=IW_V = I. This results in a simplified single-layer linear attention model:

Atten⁡(X,W)=1NXXTWX,(3)\operatorname{Atten}(X,W) = \frac{1}{N}XX^{\mathsf{T}}WX, \tag*{(3)}

where W∈RD×DW \in\mathbb{R}^{D \times D} is trainable.

Next, we design the learning task. We consider the following input structure:

X=[x1x2⋯xNxqy1y2⋯yN0],(4)X = \begin{bmatrix} x_1 & x_2 & \cdots& x_N & x_q \\ y_1 & y_2 & \cdots& y_N & 0 \end{bmatrix}, \tag*{(4)}

where in each column, the last element serves as the label yℓ=(wT)xℓy_\ell= (\mathbf{w}^{\mathsf{T}})x_\ell. In particular, for the last column, only the query vector xq\mathbf{x}_q is shown, and the answer label is masked. Each vector x∈RD\mathbf{x} \in\mathbb{R}^{D}, are sampled from an i.i.d. distribution N(0,I/D)\mathcal{N}(0,I/D), and for each task μ\mu, the input XX is generated using the same task vector w\mathbf{w}, with each component generated independently from standard Gaussian distribution. Thanks to the structure of XX, we call the task as a linear regression in-context learning, and thus the token embedding dimension is now D+1D+1, with the extra dimension for the label.

The pretraining requires that the model predicts the label for the last question xq\mathbf{x}_q. The model’s scalar prediction is the (D+1,N+1)(D+1,N+1) entry of the attention output,

y^=[1NXXTWX]D+1,N+1.(5)\hat{y} = \left[\frac{1}{N}XX^{\mathsf{T}}WX\right]_{D+1,N+1}. \tag*{(5)}

The pretraining cost function is given below:

L(W)=12P∑μ=1P(yμ−y^μ)2+λ2∥W∥2,(6)\mathcal{L}(\mathbf{W}) = \frac{1}{2P}\sum_{\mu=1}^{P}\left(y^{\mu}-\hat{y}^{\mu}\right)^2+\frac{\lambda}{2}\lVert\mathbf{W}\rVert^2, \tag*{(6)}

where λ≥0\lambda\ge0 controls ℓ2\ell_2 regularization (weight decay), PP is the number of training examples with different task vectors but all belonging to the same linear regression problem. To sum up, the key model parameters for this empirical risk minimization are set by the task diversity PP, context length NN and input dimension DD. The size of weight matrix WW is determined by (D+1)2(D+1)^2, whose actual size for learning or inference is smaller (to be explained later). To keep both terms in Eq. (6) order of O(1)\mathcal{O}(1), λ\lambda may be scaled by 1/D31/D^3 if DD is a large value.

To proceed, we first compute 1NXμ(Xμ)T\frac{1}{N}X^{\mu}(X^{\mu})^{\mathsf{T}}, where XμX^{\mu} is organized into a block matrix:

X=[X0xqy0T0](7)X=\begin{bmatrix} X_0 & x_q\\ \mathbf{y}_0^{\mathsf{T}} & 0 \end{bmatrix} \tag*{(7)}

where y0=X0Twμ∈RN\mathbf{y}_0=X_0^{\mathsf{T}}\mathbf{w}^{\mu}\in\mathbb{R}^{N}. We then obtain:

XXT=[X0xqy0T0][X0Ty0xqT0]=[X0X0T+xqxqTX0y0y0X0Ty0Ty0].(8)XX^{\mathsf{T}}= \begin{bmatrix} X_0 & x_q\\ \mathbf{y}_0^{\mathsf{T}} & 0 \end{bmatrix} \begin{bmatrix} X_0^{\mathsf{T}} & \mathbf{y}_0\\ x_q^{\mathsf{T}} & 0 \end{bmatrix} = \begin{bmatrix} X_0X_0^{\mathsf{T}}+x_qx_q^{\mathsf{T}} & X_0\mathbf{y}_0\\ \mathbf{y}_0X_0^{\mathsf{T}} & \mathbf{y}_0^{\mathsf{T}}\mathbf{y}_0 \end{bmatrix}. \tag*{(8)}

Therefore, we define Γμ=1NXμ(Xμ)T\Gamma^{\mu}=\frac{1}{N}X^{\mu}(X^{\mu})^{\mathsf{T}}, and obtain a compact form:

Γμ=[Σx+1NxqxqTΣxwμ(wμ)TΣx(wμ)TΣxwμ],(9)\Gamma^{\mu}= \begin{bmatrix} \Sigma_x+\frac{1}{N}x_qx_q^{\mathsf{T}} & \Sigma_x\mathbf{w}^{\mu}\\ (\mathbf{w}^{\mu})^{\mathsf{T}}\Sigma_x & (\mathbf{w}^{\mu})^{\mathsf{T}}\Sigma_x\mathbf{w}^{\mu} \end{bmatrix}, \tag*{(9)}

where Σx=1NX0X0T\Sigma_x=\frac{1}{N}X_0X_0^{\mathsf{T}}, and X0X_0 depends on the example index μ\mu (in the following this index is implicitly included).

Next, we write the weight matrix into a block form,

W=[W11W12W21W22].(10)W= \begin{bmatrix} W_{11} & W_{12}\\ W_{21} & W_{22} \end{bmatrix}. \tag*{(10)}

Thus, we get the product WXWX:

WX=[W11W12W21W22][X0xqy0T0]=[W11X0+W12y0TW11xqW21X0+W22y0TW21xq].(11)WX= \begin{bmatrix} W_{11} & W_{12}\\ W_{21} & W_{22} \end{bmatrix} \begin{bmatrix} X_0 & x_q\\ \mathbf{y}_0^{\mathsf{T}} & 0 \end{bmatrix} = \begin{bmatrix} W_{11}X_0+W_{12}\mathbf{y}_0^{\mathsf{T}} & W_{11}x_q\\ W_{21}X_0+W_{22}\mathbf{y}_0^{\mathsf{T}} & W_{21}x_q \end{bmatrix}. \tag*{(11)}

We finally compute the actual prediction of the linear attention model:

y^μ=[ΓμWX]D+1,N+1(12)\hat{y}^{\mu} = [\Gamma^{\mu} W X]_{D+1,N+1} \tag*{(12)}
=(wμ)TΣx[W11+wμW21]xq(13)= (\mathbf{w}^{\mu})^{\mathsf{T}}\Sigma_x[W_{11}+\mathbf{w}^{\mu}W_{21}]\mathbf{x}_q \tag*{(13)}
=vec⁡((ΣxwμxqT(wμ)TΣxwμxqT))Tvec⁡((W11W21))=sμTσ,(14)= \operatorname{vec}\left(\begin{pmatrix}\Sigma_x\mathbf{w}^{\mu}\mathbf{x}_q^{\mathsf{T}} \\ (\mathbf{w}^{\mu})^{\mathsf{T}}\Sigma_x\mathbf{w}^{\mu}\mathbf{x}_q^{\mathsf{T}}\end{pmatrix}\right)^{\mathsf{T}}\operatorname{vec}\left(\begin{pmatrix}W_{11}\\W_{21}\end{pmatrix}\right)=\mathbf{s}_{\mu}^{\mathsf{T}}\boldsymbol{\sigma}, \tag*{(14)}

where sμ\mathbf{s}_{\mu} and σ\boldsymbol{\sigma} are the flattened forms of the data matrix and weight matrix, shown respectively below.

sμ=vec⁡((ΣxwμxqT(wμ)TΣxwμxqT))∈RD(D+1),σ=vec⁡((W11W21))∈RD(D+1).(15)\mathbf{s}_{\mu}=\operatorname{vec}\left(\begin{pmatrix}\Sigma_x\mathbf{w}^{\mu}\mathbf{x}_q^{\mathsf{T}} \\ (\mathbf{w}^{\mu})^{\mathsf{T}}\Sigma_x\mathbf{w}^{\mu}\mathbf{x}_q^{\mathsf{T}}\end{pmatrix}\right)\in\mathbb{R}^{D(D+1)},\qquad \boldsymbol{\sigma}=\operatorname{vec}\left(\begin{pmatrix}W_{11}\\W_{21}\end{pmatrix}\right)\in\mathbb{R}^{D(D+1)}. \tag*{(15)}

Given this mapping, we can interpret σ\boldsymbol{\sigma} as spin vector, while the data-dependent elements can be organized into quenched disorder by inserting the above reorganized prediction into the mean-squared error in Eq. (6). Then we can get the spherical spin model mapping of the ICL in transformer, whose statistical properties have been studied in our recent work [15]. We finally remark that if DD is fixed but NN is large, then we have Σx→ID/D\Sigma_x\to I_D/D. In this setting, a perfect ICL solution is given by W11=DIDW_{11}=DI_D and W21=0DW_{21}=0_D, while W12W_{12} and W22W_{22} are irrelevant for the task. In this work, we focus on the learning dynamics and the cost function becomes now a function of σ\boldsymbol{\sigma}, i.e., L(σ)\mathcal{L}(\boldsymbol{\sigma}), which shall be detailed in the following sections.

Solvable Dynamics

In this section, we study the gradient flow of the pretraining, which is shown as follows.

σ˙=−∂L(σ)∂σ(16)\dot{\boldsymbol{\sigma}}=-\frac{\partial\mathcal{L}(\boldsymbol{\sigma})}{\partial\boldsymbol{\sigma}} \tag*{(16)}
=1P(Sy−SSTσ)−λσ(17)=\frac{1}{P}(S\mathbf{y}-SS^{\mathsf{T}}\boldsymbol{\sigma})-\lambda\boldsymbol{\sigma} \tag*{(17)}
=1PSy−(1PSST+λI)σ,(18)=\frac{1}{P}S\mathbf{y}-\left(\frac{1}{P}SS^{\mathsf{T}}+\lambda I\right)\boldsymbol{\sigma}, \tag*{(18)}

where y=[y1,…,yP]T\mathbf{y}=[y^{1},\ldots,y^{P}]^{\mathsf{T}}, S=[s1,…,sP]∈RM×PS=[\mathbf{s}_1,\ldots,\mathbf{s}_P]\in\mathbb{R}^{M\times P}, M=D(D+1)M=D(D+1) is the total number of weight parameters to be learned.

To proceed, we write A=1PSST+λIA=\frac{1}{P}SS^{\mathsf{T}}+\lambda I and b=1PSy\mathbf{b}=\frac{1}{P}S\mathbf{y}. Then we recast the learning dynamics into the following linear ordinary differential equation:

dσdt=−Aσ+b⟹σ(t)=e−Atσ(0)−A−1(e−At−I)b.(19)\frac{d\boldsymbol{\sigma}}{dt}=-A\boldsymbol{\sigma}+\mathbf{b}\quad\Longrightarrow\quad\boldsymbol{\sigma}(t)=e^{-At}\boldsymbol{\sigma}(0)-A^{-1}(e^{-At}-I)\mathbf{b}. \tag*{(19)}

Clearly, when t→∞t \to\infty, a stationary solution (gradient-flow solution) is reached as σ∗=A−1b\boldsymbol{\sigma}^{*}=A^{-1}\mathbf{b}.

Because AA is a symmetric matrix, we can do eigenvalue decomposition as A=VΛVTA=V\Lambda V^{\mathsf{T}}, where VV is an orthogonal matrix satisfying V−1=VTV^{-1}=V^{\mathsf{T}}. This requires us to project σ\boldsymbol{\sigma} and b\mathbf{b} onto the eigenbasis VV. Therefore, we introduce the projections as z=VTσ\mathbf{z}=V^{\mathsf{T}}\boldsymbol{\sigma} and b~=VTb\widetilde{\mathbf{b}}=V^{\mathsf{T}}\mathbf{b}. We finally arrive at the following solution of the learning dynamics:

z(t)=e−Λtz(0)−Λ−1(e−Λt−I)b~.(20)\mathbf{z}(t)=e^{-\Lambda t}\mathbf{z}(0)-\Lambda^{-1}\left(e^{-\Lambda t}-\mathbf{I}\right)\widetilde{\mathbf{b}}. \tag*{(20)}

Next, we calculate the time-dependent training error as follows.

Et=12P∥y−STσ∥2+λ2∥σ∥2(21)E_t=\frac{1}{2P}\left\lVert\mathbf{y}-S^{\mathsf{T}}\boldsymbol{\sigma}\right\rVert^2+\frac{\lambda}{2}\left\lVert\boldsymbol{\sigma}\right\rVert^2 \tag*{(21)}
=12P(yTy−2yTSTσ+σTSSTσ)+λ2σTσ(22)=\frac{1}{2P}\left(\mathbf{y}^{\mathsf{T}}\mathbf{y}-2\mathbf{y}^{\mathsf{T}}S^{\mathsf{T}}\boldsymbol{\sigma}+\boldsymbol{\sigma}^{\mathsf{T}}SS^{\mathsf{T}}\boldsymbol{\sigma}\right)+\frac{\lambda}{2}\boldsymbol{\sigma}^{\mathsf{T}}\boldsymbol{\sigma} \tag*{(22)}
=12P(yTy−2Pb~Tz+zTΛ~z)+λ2zTz,(23)=\frac{1}{2P}\left(\mathbf{y}^{\mathsf{T}}\mathbf{y}-2P\widetilde{\mathbf{b}}^{\mathsf{T}}\mathbf{z}+\mathbf{z}^{\mathsf{T}}\widetilde{\Lambda}\mathbf{z}\right)+\frac{\lambda}{2}\mathbf{z}^{\mathsf{T}}\mathbf{z}, \tag*{(23)}

where SST=VΛ~VTSS^{\mathsf{T}}=V\widetilde{\Lambda}V^{\mathsf{T}}, {Λ~i}\{\widetilde{\Lambda}_i\} are the eigenvalues of SSTSS^{\mathsf{T}}, while the eigenvalues in Eq. (20) are related to the eigenvalues of SSTSS^{\mathsf{T}} by Λi=1PΛ~i+λ\Lambda_i=\frac{1}{P}\widetilde{\Lambda}_i+\lambda. SSTSS^{\mathsf{T}} shares the same eigenvectors with AA in Eq. (19).

Finally, we can calculate the test error in a similar way. We write the test data matrix Stest∈RM×PtestS_{\mathrm{test}}\in\mathbb{R}^{M\times P_{\mathrm{test}}}, where MM is the size of model (parameters), and ytest∈RPtest\mathbf{y}_{\mathrm{test}}\in\mathbb{R}^{P_{\mathrm{test}}}. The empirical test performance is characterized by the following formula:

E_g=\frac{1}{2}\left(a_{\mathrm{te}}-2\left(V^{\mathsf{T}\mathbf{g}_{\mathrm{te}}\right)^{\mathsf{T}}\mathbf{z}+\mathbf{z}^{\mathsf{T}}V^{\mathsf{T}}Q_{\mathrm{te}}V\mathbf{z}\right),} \tag*{(24)}

where ate≡ytestTytest/Ptesta_{\mathrm{te}}\equiv\mathbf{y}_{\mathrm{test}}^{\mathsf{T}}\mathbf{y}_{\mathrm{test}}/P_{\mathrm{test}}, Qte≡StestStestT/PtestQ_{\mathrm{te}}\equiv S_{\mathrm{test}}S_{\mathrm{test}}^{\mathsf{T}}/P_{\mathrm{test}}, gte≡Stestytest/Ptest\mathbf{g}_{\mathrm{te}}\equiv S_{\mathrm{test}}\mathbf{y}_{\mathrm{test}}/P_{\mathrm{test}}.

Data-Dependent Free Energy

For pretraining the transformer, we consider the following full-batch gradient descent:

σ˙=−γ∂L(σ)∂σ+η(t),(25)\dot{\boldsymbol{\sigma}}=-\gamma\frac{\partial\mathcal{L}(\boldsymbol{\sigma})}{\partial\boldsymbol{\sigma}}+\boldsymbol{\eta}(t), \tag*{(25)}

where γ\gamma denotes the learning rate, and η(t)\boldsymbol{\eta}(t) is a zero mean white noise vector, whose correlation is determined by ⟨ηi(t)ηj(t′)⟩=2Tγδijδ(t−t′)\left\langle\eta_i(t)\eta_j(t')\right\rangle=2T\gamma\delta_{ij}\delta(t-t'). The above learning dynamics are also called Langevin dynamics and in the long time limit the statistics of the network parameters are subject to the following Bayesian posterior of the model [21]:

P(σ∣D)=1Zexp⁡[−βL(σ)],(26)P(\boldsymbol{\sigma}\mid\mathcal{D})=\frac{1}{Z}\exp\left[-\beta\mathcal{L}(\boldsymbol{\sigma})\right], \tag*{(26)}

where β=1T\beta= \frac{1}{T}, D\mathcal{D} indicates the training dataset {Xμ,yμ}μ=1P\{X^\mu, y^\mu\}_{\mu=1}^{P}, and ZZ is the data-dependent partition function. The partition function can be computed as follows.

Z=∫dσexp⁡(−β2P∥y−S⊤σ∥2−λβ2∥σ∥2)(27)Z = \int d\sigma\exp\left(-\frac{\beta}{2P}\left\|\mathbf{y}-S^{\top}\sigma\right\|^2-\frac{\lambda\beta}{2}\|\sigma\|^2\right) \tag*{(27)}
=∫dσexp⁡(−β2P(y−S⊤σ)⊤(y−S⊤σ)−λβ2σ⊤σ)(28)= \int d\sigma\exp\left(-\frac{\beta}{2P}\left(\mathbf{y}-S^{\top}\sigma\right)^{\top}\left(\mathbf{y}-S^{\top}\sigma\right)-\frac{\lambda\beta}{2}\sigma^{\top}\sigma\right) \tag*{(28)}
=exp⁡(−β2Py⊤y)∫dσexp⁡(βPy⊤S⊤σ−12σ⊤(βPSS⊤+λβI)σ)(29)= \exp\left(-\frac{\beta}{2P}\mathbf{y}^{\top}\mathbf{y}\right)\int d\sigma\exp\left(\frac{\beta}{P}\mathbf{y}^{\top}S^{\top}\sigma-\frac{1}{2}\sigma^{\top}\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)\sigma\right) \tag*{(29)}
=(2π)Mdet⁡(βPSS⊤+λβI)exp⁡(−β2Py⊤y+β2Py⊤S⊤(SS⊤+PλI)−1Sy)(30)= \sqrt{\frac{(2\pi)^M}{\det\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)}}\exp\left(-\frac{\beta}{2P}\mathbf{y}^{\top}\mathbf{y}+\frac{\beta}{2P}\mathbf{y}^{\top}S^{\top}\left(SS^{\top}+P\lambda I\right)^{-1}S\mathbf{y}\right) \tag*{(30)}
=(2π)Mdet⁡(βPSS⊤+λβI)exp⁡(−β2Py⊤y+β2Py⊤(I+Pλ(S⊤S)−1)−1y)(31)= \sqrt{\frac{(2\pi)^M}{\det\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)}}\exp\left(-\frac{\beta}{2P}\mathbf{y}^{\top}\mathbf{y}+\frac{\beta}{2P}\mathbf{y}^{\top}\left(I+P\lambda\left(S^{\top}S\right)^{-1}\right)^{-1}\mathbf{y}\right) \tag*{(31)}

where the last equality employs S⊤(SS⊤+aIM)−1S=K(K+aIP)−1=IP−a(K+aIP)−1S^{\top}\left(SS^{\top}+aI_M\right)^{-1}S=K\left(K+aI_P\right)^{-1}=I_P-a\left(K+aI_P\right)^{-1}, a=Pλa=P\lambda, and K=S⊤SK=S^{\top}S. Hence, the partition function can thus be simplified to the following result:

Z=(2π)Mdet⁡(βPSS⊤+λβI)exp⁡(−12y⊤C−1y),(32)Z=\sqrt{\frac{(2\pi)^M}{\det\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)}}\exp\left(-\frac{1}{2}\mathbf{y}^{\top}C^{-1}\mathbf{y}\right), \tag*{(32)}

where C=1λβS⊤S+PβIC=\frac{1}{\lambda\beta}S^{\top}S+\frac{P}{\beta}I. Note that here we assume λ>0\lambda>0, but the above ZZ is not applicable to the case of λ=0\lambda=0 and P<MP<M. We do not focus on this regime with the data-dependent prediction in this section.

We next derive the equilibrium training error from the log-partition-function.

⟨Et⟩σ=−∂ln⁡Z∂β(33)\langle E_t\rangle_\sigma=-\frac{\partial\ln Z}{\partial\beta} \tag*{(33)}
=−∂(−12log⁡det⁡(βPSS⊤+λβI)−12y⊤C−1y)∂β(34)=-\frac{\partial\left(-\frac{1}{2}\log\det\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)-\frac{1}{2}\mathbf{y}^{\top}C^{-1}\mathbf{y}\right)}{\partial\beta} \tag*{(34)}
=12∂log⁡det⁡(βPSS⊤+λβI)∂β+12y⊤∂C−1∂βy(35)=\frac{1}{2}\frac{\partial\log\det\left(\frac{\beta}{P}SS^{\top}+\lambda\beta I\right)}{\partial\beta}+\frac{1}{2}\mathbf{y}^{\top}\frac{\partial C^{-1}}{\partial\beta}\mathbf{y} \tag*{(35)}
=M+y⊤C−1y2β,(36)=\frac{M+\mathbf{y}^{\top}C^{-1}\mathbf{y}}{2\beta}, \tag*{(36)}

where to arrive at the last equality, we have used 1det⁡A∂det⁡(A)∂x=Tr⁡(A−1∂A∂x)\frac{1}{\det A}\frac{\partial\det(A)}{\partial x}=\operatorname{Tr}\left(A^{-1}\frac{\partial A}{\partial x}\right) and ∂C−1∂β=−C−1∂C∂βC−1\frac{\partial C^{-1}}{\partial\beta}=-C^{-1}\frac{\partial C}{\partial\beta}C^{-1}.

Finally, we derive the expected test error over the Bayesian posterior P(σ)P(\sigma). First, we notice that the posterior can be written into the multivariate Gaussian distribution:

P(σ)∝exp⁡(−βL(σ))=exp⁡(−β2P(y−STσ)T(y−STσ)−λβ2σTσ)(37)P(\boldsymbol{\sigma}) \propto\exp(-\beta\mathcal{L}(\boldsymbol{\sigma})) = \exp\left(-\frac{\beta}{2P}\left(\mathbf{y}-S^{\mathsf{T}}\boldsymbol{\sigma}\right)^{\mathsf{T}}\left(\mathbf{y}-S^{\mathsf{T}}\boldsymbol{\sigma}\right)-\frac{\lambda\beta}{2}\boldsymbol{\sigma}^{\mathsf{T}}\boldsymbol{\sigma}\right) \tag*{(37)}
=exp⁡(−12σT(βPSST+λβI)σ+βPyTSTσ)(38)= \exp\left(-\frac{1}{2}\boldsymbol{\sigma}^{\mathsf{T}}\left(\frac{\beta}{P}SS^{\mathsf{T}}+\lambda\beta I\right)\boldsymbol{\sigma}+\frac{\beta}{P}\mathbf{y}^{\mathsf{T}}S^{\mathsf{T}}\boldsymbol{\sigma}\right) \tag*{(38)}
=exp⁡(−12(σ−μ)TΣ−1(σ−μ)),(39)= \exp\left(-\frac{1}{2}\left(\boldsymbol{\sigma}-\boldsymbol{\mu}\right)^{\mathsf{T}}\Sigma^{-1}\left(\boldsymbol{\sigma}-\boldsymbol{\mu}\right)\right), \tag*{(39)}

where the mean and covariance are μ=βP(βPSST+λβI)−1Sy\boldsymbol{\mu}=\frac{\beta}{P}\left(\frac{\beta}{P}SS^{\mathsf{T}}+\lambda\beta I\right)^{-1}Sy and Σ=(βPSST+λβI)−1\Sigma=\left(\frac{\beta}{P}SS^{\mathsf{T}}+\lambda\beta I\right)^{-1}, respectively.

For a new test dataset (Stest,ytest)\left(S^{\mathrm{test}},\mathbf{y}^{\mathrm{test}}\right), the test error obtained after training is given by:

⟨Eg⟩≡Eσ∼N(μ,Σ)[Eg(σ)]=12PtestEσ∼N(μ,Σ)[∥ytest−StestTσ∥2](40)\langle E_g\rangle\equiv\mathbb{E}_{\boldsymbol{\sigma}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)}[E_g(\boldsymbol{\sigma})]=\frac{1}{2P_{\mathrm{test}}}\mathbb{E}_{\boldsymbol{\sigma}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)}\left[\left\|\mathbf{y}_{\mathrm{test}}-S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\sigma}\right\|^2\right] \tag*{(40)}
=12PtestEσ∼N(μ,Σ)[ytestTytest−2ytestTStestTσ+σTStestStestTσ](41)=\frac{1}{2P_{\mathrm{test}}}\mathbb{E}_{\boldsymbol{\sigma}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)}\left[\mathbf{y}_{\mathrm{test}}^{\mathsf{T}}\mathbf{y}_{\mathrm{test}}-2\mathbf{y}_{\mathrm{test}}^{\mathsf{T}}S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\sigma}+\boldsymbol{\sigma}^{\mathsf{T}}S_{\mathrm{test}}S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\sigma}\right] \tag*{(41)}
=12Ptest(ytestTytest−2ytestTStestTμ+Tr⁡(StestStestTΣ)+μTStestStestTμ),(42)=\frac{1}{2P_{\mathrm{test}}}\left(\mathbf{y}_{\mathrm{test}}^{\mathsf{T}}\mathbf{y}_{\mathrm{test}}-2\mathbf{y}_{\mathrm{test}}^{\mathsf{T}}S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\mu}+\operatorname{Tr}(S_{\mathrm{test}}S_{\mathrm{test}}^{\mathsf{T}}\Sigma)+\boldsymbol{\mu}^{\mathsf{T}}S_{\mathrm{test}}S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\mu}\right), \tag*{(42)}

where the final two terms in the last line are obtained by using E[xTAx]=E[Tr⁡(xTAx)]=E[Tr⁡(AxxT)]=Tr⁡[A(Σ+μμT)]=Tr⁡(AΣ)+μTAμ\mathbb{E}[x^{\mathsf{T}}Ax]=\mathbb{E}[\operatorname{Tr}(x^{\mathsf{T}}Ax)]=\mathbb{E}[\operatorname{Tr}(Axx^{\mathsf{T}})]=\operatorname{Tr}[A(\Sigma+\boldsymbol{\mu}\boldsymbol{\mu}^{\mathsf{T}})]=\operatorname{Tr}(A\Sigma)+\boldsymbol{\mu}^{\mathsf{T}}A\boldsymbol{\mu}. Finally, the entire expression can be reorganized into the following compact form:

⟨Eg⟩≡Eσ∼N(μ,Σ)[Eg(σ)]=12Ptest(∥ytest−StestTμ∥2+Tr⁡(StestStestTΣ)).(43)\langle E_g\rangle\equiv\mathbb{E}_{\boldsymbol{\sigma}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)}[E_g(\boldsymbol{\sigma})]=\frac{1}{2P_{\mathrm{test}}}\left(\left\|\mathbf{y}_{\mathrm{test}}-S_{\mathrm{test}}^{\mathsf{T}}\boldsymbol{\mu}\right\|^2+\operatorname{Tr}(S_{\mathrm{test}}S_{\mathrm{test}}^{\mathsf{T}}\Sigma)\right). \tag*{(43)}

In the zero temperature limit, the second term in the bracket vanishes.

Eigenspectra of the Hessian Matrix

Note that a key quantity determining the learning dynamics is the matrix H=1PSSTH=\frac{1}{P}SS^{\mathsf{T}}, whose size is M×MM \times M, M=D(D+1)M=D(D+1) in our setting and elements Hij=1P∑μSiμSjμH_{ij}=\frac{1}{P}\sum_{\mu}S_i^{\mu}S_j^{\mu}. HH is the very Hessian matrix of the equivalent spherical spin model (without regularization) of the in-context learning in our previous work [15]. The fraction of zero eigenvalues determines the landscape shape of our optimization, which is expected to affect the generalization performance.

According to the arguments made in previous works [16, 18], we set W21=0W_{21}=0 for an additional analysis, as this zero solution is also observed in our previous work [15]. The rationale can also be found in Eq. (12) when the linear regression can be perfectly inferred. Given a specified μ\boldsymbol{\mu}, SiμS_i^{\mu} is a vectorization of the matrix ΣxwμxqT\Sigma_x w^{\mu} x_q^{\mathsf{T}}. The mean vanishes due to the i.i.d. setting of regression vector x\mathbf{x} and task vector w\mathbf{w}. The covariance can also be determined as follows:

⟨SℓμSℓ′ν⟩=⟨[Σxwμ]ixq,j×[Σxwν]i′xq,j′⟩(44)\left\langle S_{\ell}^{\mu} S_{\ell'}^{\nu} \right\rangle= \left\langle[\Sigma_x w^{\mu}]_{i x q,j} \times[\Sigma_x w^{\nu}]_{i' x q,j'} \right\rangle \tag*{(44)}
=1Dδj,j′δμ,ν⟨(Σx)ii′2⟩(45)= \frac{1}{D}\delta_{j,j'}\delta_{\mu,\nu}\left\langle(\Sigma_x)_{ii'}^2\right\rangle \tag*{(45)}
=κDD3δi,i′δj,j′δμ,ν,(46)= \frac{\kappa_D}{D^3}\delta_{i,i'}\delta_{j,j'}\delta_{\mu,\nu}, \tag*{(46)}

where κD=1+D+1N\kappa_D = 1 + \frac{D+1}{N}, ℓ=(i,j)\ell= (i,j) and ℓ′=(i′,j′)\ell' = (i',j') due to the vectorization of a matrix, and to derive the last equality, the Wick’s theorem is used. We conclude that in this simplified case, Γ=D3κDH\Gamma= \frac{D^3}{\kappa_D}H is a random matrix whose eigenvalue spectrum can be determined by the Marchenko-Pastur law, according to the previous work [22], which we shall verify by comparing the prediction with numerical simulations.

To get the asymptotic spectral density of the matrix Γ\Gamma, we assume P=αD2P = \alpha D^2, N=cDN = cD.

Therefore, the eigenvalue distribution is given by the following formula [22]:

ρ(x)=(1−α)+δ(x)+(ℓ2x+−x)(x−ℓ2x−)2πℓ2xI[x−,x+](x)(47)\rho(x) = (1-\alpha)_{+}\delta(x) + \frac{\sqrt{(\ell^{2}x_{+}-x)(x-\ell^{2}x_{-})}}{2\pi\ell^{2}x}\mathbb{I}_{[x_{-},x_{+}]}(x) \tag*{(47)}

where (y)+=max⁡(0,y)(y)_{+} = \max(0,y), I[x−,x+](x)\mathbb{I}_{[x_{-},x_{+}]}(x) returns one if xx falls within the range [ℓ2x−,ℓ2x+][\ell^{2}x_{-},\ell^{2}x_{+}] and zero otherwise, x±=(α±1)2x_{\pm} = (\sqrt{\alpha} \pm1)^{2}, and ℓ2=1α\ell^{2} = \frac{1}{\alpha}. The results are shown in Figure 1, which demonstrates that α\alpha controls the fraction of zero eigenvalues, while with increasing the value of DD the agreement with the Marchenko–Pastur law is expected. We study the learning performance under this setting in the end of the next section.

Figure 1. Eigenspectra of the rescaled Hessian matrix. Blue histograms: three independent datasets for each α\alpha, with D=N=100D = N = 100 (a-c) and D=N=20D = N = 20 (d-f). Red curves: Eq. (47). The α=0.5\alpha= 0.5 histogram excludes the zero eigenvalues but is normalized by all eigenvalues, so its area is 0.5. The zero atom is stated separately. The α=1\alpha= 1 density diverges at the origin; its vertical axis is truncated to show the rest of the spectrum.

Theory Corroborated by Experiments

First, we compare the data-dependent theory and the numerical simulations (in all simulations, we used zero-weight initialization and δt=0.002\delta t = 0.002 unless otherwise stated), as shown in Figure 2. As the diversity increases, the training error increases, more rapidly for short context length. number of training examples can be thought of as a learning complexity, e.g., fitting many more data points becomes hard. This phenomenon was also observed in training a single-layer neural network [23]. However, with increasing the task diversity, the generalization performance gets better, especially for a long context length, which coincides with our previous work [15]. Here, our theoretical results agree very well with the numerical Langevin learning dynamics. Because our theory depends on specified training dataset, there exists fluctuation related to the data choice. We also consider different levels of model complexity, characterized by the input dimension DD such that the the number of model parameters equals to M=D(D+1)M = D(D + 1). As revealed in Figure 3, more complex model requires more data examples to reach a comparable test loss to less complex model. The theory agrees with numerical simulations as well.

Figure 2. Stationary training and test losses as a function of the number of training prompts PP for different context lengths NN, with fixed input dimension D=10D = 10. Solid curves denote our theoretical predictions from the free-energy calculation in Sec. IV, while markers represent numerical estimates from Langevin dynamics simulations [see Eq. (25)]. In the simulations, we set β=103\beta= 10^{3}, λ=0.01\lambda= 0.01, and γ=1\gamma= 1. After an equilibration period of 10510^{5} steps, we sample 20 points separated by 5000 steps and average them to estimate the final loss. (a) As the task diversity increases, the stationary training loss increases for all values of NN. The smaller the value of NN, the faster the training loss increases. (b) As the task diversity increases, the stationary test loss decreases for all values of NN. However, for a given task diversity, smaller values of NN consistently yield a higher test loss.

Figure 3. Stationary training and test losses as a function of the number of training prompts PP for different input dimensions DD, with fixed context length N=100N = 100. Solid curves denote the theoretical predictions from the free-energy calculation in Sec. IV, while markers represent numerical estimates from Langevin dynamics simulations [Eq. (25)]. In simulations, we set β=103\beta= 10^3 and λ=0.01\lambda= 0.01, and γ=1\gamma= 1. (a) The training loss increases with PP and is larger for a higher-dimensional input. (b) The test loss decreases as PP grows, but a larger DD requires more task diversity to reach a comparable generalization performance.

Next, we study how the training and test losses evolve with training steps (see Figure 4). For the training loss, we observe a monotonically decreasing behavior, regardless of the task diversity. However, regarding the test loss, the behavior can be completely different depending on the task diversity. For a small value of PP, the training can not improve the generalization performance but make the generalization even worse. Nevertheless, the training error is close to zero, demonstrating a perfect memorization of the training prompts. The experimental results are corroborated by theory, especially the asymptotic limit of t→∞t \to\infty is captured by the data-dependent statistical mechanics theory in Sec. IV.

Figure 4. Gradient-flow dynamics of the training and test losses for task diversity P=10P = 10 and 100100, with D=10D = 10 and N=100N = 100. The regularization parameter λ=0.01\lambda= 0.01. Solid curves represent the analytical gradient-flow solution with the shaded regions indicating the theoretical uncertainty (see Eq. (16) in Sec. III in the main text), while markers represent gradient descent simulations. Asymptotic error means the stationary limit of the error predicted by the zero-temperature free energy analysis in Sec. IV. Note that the training error includes the weight-decay term, while the test error does not include that term. (a) Training losses rapidly converge to their stationary value for both small and large PP. (b) Test losses exhibit qualitatively different behavior depending on the task diversity: for the small PP, the model exhibits memorization-like behavior and the test loss increases, whereas for the larger PP, the test loss decreases monotonically, indicating the emergence of in-context generalization.

The memorization effects motivate us to propose an indicator, namely the relative generalization gap ΔEr=⟨Eg⟩−⟨Et⟩⟨Et⟩\Delta E_r = \frac{\langle E_g\rangle-\langle E_t\rangle}{\langle E_t\rangle}. Note that here we remove the weight-decay term in computing ⟨Et⟩\langle E_t\rangle in order to be parallel to ⟨Eg⟩\langle E_g\rangle. In the case of a smaller EtE_t and the corresponding larger EgE_g, the relative generalization gap will become very large; when the model generalizes, this gap will come to a relatively small level, at least of the order O(1)\mathcal{O}(1). In addition, we also follow our previous work [15] to define the contrast ratio between the learned matrix W11W_{11}, and the identity matrix, which is a solution of perfect generalization except for a prefactor DD, see Eq. (12) when NN is independently large [15]. More precisely, C=⟨U,ID⟩(∥U∥2∥ID∥2)C = \frac{\langle U, I_D\rangle}{(\lVert U\rVert_2\lVert I_D\rVert_2)} where UU indicates the non-zero transformer matrix W11/DW_{11}/D, ⟨A,B⟩≡∑ijAijBij\langle A, B\rangle\equiv\sum_{ij} A_{ij}B_{ij}, ∥A∥2\lVert A\rVert_2 denotes a matrix Frobenius norm, and IDI_D denotes a D×DD \times D identity matrix. As illustrated in Figure 5, a memorization phase is clearly separated from a generalization phase. Note that this transition is not in the thermodynamic sense, as we do not focus on the asymptotic limit of network sizes and data sizes.

Figure 5. Dependence of the relative generalization gap ΔEr\Delta E_r and the contrast ratio CC on task diversity PP. The results are obtained by taking the long-time limit of the analytical gradient-flow solution derived in Sec. III with the regularization strength λ=0.01\lambda= 0.01. The main panels show the generalization gap, while the insets show the contrast ratio. Shaded bands are ±1\pm1 s.d. over 100 training sets per PP. Different color intensities correspond to different values of NN (or DD). The relative gap can be slightly negative at large PP because EtE_t includes the weight-decay term. (a) Results for different context lengths NN at fixed D=10D = 10. (b) Results for different input dimensions DD at fixed N=100N = 100. The dashed line of ΔEr=5\Delta E_r = 5 identifies a rough estimate of the value of PP separating memorization from generalization.

Finally, we study the simplest setting introduced in § 5. We consider the case of λ=0\lambda= 0, which is shown in Figure 6. Note that in this case, HH is not invertable anymore when P<D2P < D^2. Therefore we apply the Langevin dynamics to plot Figure 6. For λ=0\lambda= 0 and P<D2P < D^2, HH has D2−PD^2 - P exact zero eigenvalues. The data thus exert no force along these directions, so they diffuse freely and reach variance growing with time. The transition occurs after the zero-eigenvalue fraction point, given this small value of DD. The overall behavior is qualitatively the same with the case of λ≠0\lambda\ne0. At T=0T = 0, the minimum-norm solution gives Et=0E_t = 0 and therefore ΔEr=∞\Delta E_r = \infty. At P=D2P = D^2, the smallest non-zero eigenvalue is about 10−710^{-7}, which produces the peak observed in Figure 6.

Figure 6. Dependence of the relative generalization gap ΔEr\Delta E_r and the contrast ratio CC on task diversity PP. We focus on W21=0W_{21}=0, and λ=0\lambda=0. The simulations of Langevin dynamics at β=103\beta=10^{3} and γ=1\gamma=1 are run up to 10610^{6} steps (time step δt=0.1\delta t=0.1) for equilibrium. Note that the equilibrium can only be reached after the dotted lines, leading vanishing CC and extremely large ΔEr\Delta E_r before the dotted lines. The main panels show the generalization gap, while the insets show the contrast ratio. Different color intensities correspond to different values of NN (or DD). Lines are averages over 50 training realizations. Error bars are shown only for specific values of PP. Dotted lines mark P=D2P=D^{2}. (a) Results for different context lengths NN at fixed D=10D=10. (b) Results for different input dimensions DD at fixed N=100N=100. The dashed line of ΔEr=5\Delta E_r=5 identifies a rough estimate of the value of PP separating memorization from generalization.

Conclusion

In this work, we study the learning dynamics of single-head single layer linear in-context learning in a non-asymptotic setting, where the token dimension DD, context length NN and the number of examples are all finite. A solvable learning dynamics equation is first obtained, where the eigenvalues of the Hessian matrix control how fast the weight parameters converge to the optimal state: the ridge solution of the problem. Next, we provide a data-dependent statistical mechanics description of the learning dynamics, from which the long-time limit of the training error and test error can be theoretically determined by the training data or test data. We find by tuning the value of PP (the number of training examples), the generalization gap decreases sharply, implying the presence of a memorization phase when the training linear-regression tasks are deficient; the memorization phase will be replaced by a generalization phase later as evidenced by a saturating behavior of the contrast ratio.

Acknowledgments

This research was supported by the National Natural Science Foundation of China for Grant numbers 12475045, and Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices (No. 2026B1212170049). We use GPT 6.1 sol for checking the results of this paper and providing advice to improve the manuscript quality. The authors take full responsibility for the scientific content of the paper.

Code availability

Codes to reproduce all results are deposited in our Github [24].

References

  1. [8]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.arxiv.org/abs/2005.14165
  2. [9]Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174, 2023.arxiv.org/abs/2212.07677
  3. [10]Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv:2111.02080, 2021.arxiv.org/abs/2111.02080
  4. [11]Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? arXiv:2310.08391, 2024. in ICLR 2024.arxiv.org/abs/2310.08391
  5. [12]Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 14228–14246. Curran Associates, Inc., 2023.arxiv.org/abs/2306.15063
  6. [13]Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv:2211.15661, 2022.arxiv.org/abs/2211.15661
  7. [14]Yue M. Lu, Mary Letey, Jacob A. Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, 122(28):e2502599122, 2025.DOI
  8. [15]Yuhao Li, Ruoran Bai, and Haiping Huang. Spin-glass model of in-context learning. Phys. Rev. E, 112:L013301, 2025.DOI
  9. [16]Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024.arxiv.org/abs/2306.09927
  10. [17]Jianliang He, Xintian Pan, Siyu Chen, and Zhuoran Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 22686–22742. PMLR, 13–19 Jul 2025.
  11. [18]Yedi Zhang, Aaditya K Singh, Peter E Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention. arXiv:2501.16265, 2025.arxiv.org/abs/2501.16265
  12. [19]Blake Bordelon, Mary I Letey, and Cengiz Pehlevan. Theory of scaling laws for in-context regression: Depth, width, context and time. arXiv:2510.01098, 2025.arxiv.org/abs/2510.01098
  13. [20]Nischal Mainali and Lucas Teixeira. Exact learning dynamics of in-context learning in linear transformers and its application to non-linear transformers. arXiv:2504.12916, 2025.arxiv.org/abs/2504.12916
  14. [21]Hannes Risken. The Fokker-Planck Equation: Methods of Solution and Applications. Springer-Verlag Berlin, Berlin, 1996.
  15. [22]Jianwen Zhou and Haiping Huang. Weakly correlated synapses promote dimension reduction in deep neural networks. Phys. Rev. E, 103:012315, Jan 2021.DOI
  16. [23]Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428–446, 2020.DOI
  17. [24]https://github.com/PMI-SYSU/.

Paper details

vlnc:W_2ckb9sar
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GPT-6.1 Sol (bare)

View statement
Looking for related papers…
Contents