WorksComputer Science
Solvable dynamical model of in-context learning
Abstract
Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer single-head linear attention model, where an empirical risk minimization is defined for learning. Given a finite number of training examples, we determine the exact solution of network-parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning especially from a learning dynamics perspective.
Introduction
Inspired by the study of attention mechanisms in the brain [1–3], an engineering generalization of the biological attention to the self-attention dominant in the current transformer computation has started a revolution in natural language processing first and recently the artificial general intelligence including superhuman capabilities in coding, mathematical reasoning, and document comprehension [4, 5]. The self-attention captures long-range dependencies between different parts of the input (e.g., image or text), trained with a simple cost of next-token prediction [6], establishing a computer science foundation of large language model (LLM) [7].
As one of the astonishing abilities, the transformer is able to do in-context inference [8], i.e., the pre-trained transformer is able to accomplish previously-unseen complicated tasks by reasoning on a short prompt especially without a need for updating the model parameters. This remarkable property has recently captured intense research interest from theoretical machine learning and statistical physics communities [9, 10, 11, 12, 13, 14, 15, 16, 17]. The theoretical efforts focused on a single-layer linear attention [9, 13, 14, 15, 16], which revealed that a sufficient pre-training task diversity triggers the emergence of in-context inference. This phase transition was first empirically confirmed in the in-context learning of a regression task [12], later analytically studied in the asymptotic high-dimensional limit [14], and formulated recently in a disordered spin glass framework where the network parameters act as spins while training data act as quenched coupling between spins. Recent theoretical works start to focus on the learning dynamics of transformer in linear regression tasks, especially on the population loss (given infinite training prompts) [16, 18] and on the neural scaling law in training time, depth and context length [19]. A recent work derives non-asymptotic closed-form characterization of the learning dynamics for vector-valued regression [20].
Based on our previous work of statistical physics analysis of in-context learning (ICL), we continue to explore the learning dynamics of an empirical risk minimization in this work. We first describe the model setting which allows a closed-form solution of the network parameter dynamics, from which we can predict the behavior of individual concrete learning trajectory and related test performance on unseen linear regression examples. We also derive a data-dependent statistical mechanics formalism to explain the stationary limit of the learning dynamics, supported by numerical experiments of single instance learning. Furthermore, we simplify our analytic results in a specific setting, e.g., setting a particular set of network parameters to zero, thereby isolating the key factor in the form of a data-dependent eigenvalue spectrum.
Linear Attention Model
Given an input matrix , a single-head single-layer softmax-based attention model is formulated as follows,
where indicates the query, key or value matrix of dimension , respectively. is also called the embedding dimension of token . All vectors in this paper are defined in a column form, i.e., . The above transformer block outputs an equal-size matrix . The softmax operation is normalized for each column, representing a weight score for different key vectors (transformed tokens). To facilitate the following theoretical analysis, we replace the softmax operation by a linear attention, and merge the key and query projection matrices into a single parameter matrix , and set the value projection matrix . This results in a simplified single-layer linear attention model:
where is trainable.
Next, we design the learning task. We consider the following input structure:
where in each column, the last element serves as the label . In particular, for the last column, only the query vector is shown, and the answer label is masked. Each vector , are sampled from an i.i.d. distribution , and for each task , the input is generated using the same task vector , with each component generated independently from standard Gaussian distribution. Thanks to the structure of , we call the task as a linear regression in-context learning, and thus the token embedding dimension is now , with the extra dimension for the label.
The pretraining requires that the model predicts the label for the last question . The model’s scalar prediction is the entry of the attention output,
The pretraining cost function is given below:
where controls regularization (weight decay), is the number of training examples with different task vectors but all belonging to the same linear regression problem. To sum up, the key model parameters for this empirical risk minimization are set by the task diversity , context length and input dimension . The size of weight matrix is determined by , whose actual size for learning or inference is smaller (to be explained later). To keep both terms in Eq. (6) order of , may be scaled by if is a large value.
To proceed, we first compute , where is organized into a block matrix:
where . We then obtain:
Therefore, we define , and obtain a compact form:
where , and depends on the example index (in the following this index is implicitly included).
Next, we write the weight matrix into a block form,
Thus, we get the product :
We finally compute the actual prediction of the linear attention model:
where and are the flattened forms of the data matrix and weight matrix, shown respectively below.
Given this mapping, we can interpret as spin vector, while the data-dependent elements can be organized into quenched disorder by inserting the above reorganized prediction into the mean-squared error in Eq. (6). Then we can get the spherical spin model mapping of the ICL in transformer, whose statistical properties have been studied in our recent work [15]. We finally remark that if is fixed but is large, then we have . In this setting, a perfect ICL solution is given by and , while and are irrelevant for the task. In this work, we focus on the learning dynamics and the cost function becomes now a function of , i.e., , which shall be detailed in the following sections.
Solvable Dynamics
In this section, we study the gradient flow of the pretraining, which is shown as follows.
where , , is the total number of weight parameters to be learned.
To proceed, we write and . Then we recast the learning dynamics into the following linear ordinary differential equation:
Clearly, when , a stationary solution (gradient-flow solution) is reached as .
Because is a symmetric matrix, we can do eigenvalue decomposition as , where is an orthogonal matrix satisfying . This requires us to project and onto the eigenbasis . Therefore, we introduce the projections as and . We finally arrive at the following solution of the learning dynamics:
Next, we calculate the time-dependent training error as follows.
where , are the eigenvalues of , while the eigenvalues in Eq. (20) are related to the eigenvalues of by . shares the same eigenvectors with in Eq. (19).
Finally, we can calculate the test error in a similar way. We write the test data matrix , where is the size of model (parameters), and . The empirical test performance is characterized by the following formula:
where , , .
Data-Dependent Free Energy
For pretraining the transformer, we consider the following full-batch gradient descent:
where denotes the learning rate, and is a zero mean white noise vector, whose correlation is determined by . The above learning dynamics are also called Langevin dynamics and in the long time limit the statistics of the network parameters are subject to the following Bayesian posterior of the model [21]:
where , indicates the training dataset , and is the data-dependent partition function. The partition function can be computed as follows.
where the last equality employs , , and . Hence, the partition function can thus be simplified to the following result:
where . Note that here we assume , but the above is not applicable to the case of and . We do not focus on this regime with the data-dependent prediction in this section.
We next derive the equilibrium training error from the log-partition-function.
where to arrive at the last equality, we have used and .
Finally, we derive the expected test error over the Bayesian posterior . First, we notice that the posterior can be written into the multivariate Gaussian distribution:
where the mean and covariance are and , respectively.
For a new test dataset , the test error obtained after training is given by:
where the final two terms in the last line are obtained by using . Finally, the entire expression can be reorganized into the following compact form:
In the zero temperature limit, the second term in the bracket vanishes.
Eigenspectra of the Hessian Matrix
Note that a key quantity determining the learning dynamics is the matrix , whose size is , in our setting and elements . is the very Hessian matrix of the equivalent spherical spin model (without regularization) of the in-context learning in our previous work [15]. The fraction of zero eigenvalues determines the landscape shape of our optimization, which is expected to affect the generalization performance.
According to the arguments made in previous works [16, 18], we set for an additional analysis, as this zero solution is also observed in our previous work [15]. The rationale can also be found in Eq. (12) when the linear regression can be perfectly inferred. Given a specified , is a vectorization of the matrix . The mean vanishes due to the i.i.d. setting of regression vector and task vector . The covariance can also be determined as follows:
where , and due to the vectorization of a matrix, and to derive the last equality, the Wick’s theorem is used. We conclude that in this simplified case, is a random matrix whose eigenvalue spectrum can be determined by the Marchenko-Pastur law, according to the previous work [22], which we shall verify by comparing the prediction with numerical simulations.
To get the asymptotic spectral density of the matrix , we assume , .
Therefore, the eigenvalue distribution is given by the following formula [22]:
where , returns one if falls within the range and zero otherwise, , and . The results are shown in Figure 1, which demonstrates that controls the fraction of zero eigenvalues, while with increasing the value of the agreement with the Marchenko–Pastur law is expected. We study the learning performance under this setting in the end of the next section.
Figure 1. Eigenspectra of the rescaled Hessian matrix. Blue histograms: three independent datasets for each , with (a-c) and (d-f). Red curves: Eq. (47). The histogram excludes the zero eigenvalues but is normalized by all eigenvalues, so its area is 0.5. The zero atom is stated separately. The density diverges at the origin; its vertical axis is truncated to show the rest of the spectrum.
Theory Corroborated by Experiments
First, we compare the data-dependent theory and the numerical simulations (in all simulations, we used zero-weight initialization and unless otherwise stated), as shown in Figure 2. As the diversity increases, the training error increases, more rapidly for short context length. number of training examples can be thought of as a learning complexity, e.g., fitting many more data points becomes hard. This phenomenon was also observed in training a single-layer neural network [23]. However, with increasing the task diversity, the generalization performance gets better, especially for a long context length, which coincides with our previous work [15]. Here, our theoretical results agree very well with the numerical Langevin learning dynamics. Because our theory depends on specified training dataset, there exists fluctuation related to the data choice. We also consider different levels of model complexity, characterized by the input dimension such that the the number of model parameters equals to . As revealed in Figure 3, more complex model requires more data examples to reach a comparable test loss to less complex model. The theory agrees with numerical simulations as well.
Figure 2. Stationary training and test losses as a function of the number of training prompts for different context lengths , with fixed input dimension . Solid curves denote our theoretical predictions from the free-energy calculation in Sec. IV, while markers represent numerical estimates from Langevin dynamics simulations [see Eq. (25)]. In the simulations, we set , , and . After an equilibration period of steps, we sample 20 points separated by 5000 steps and average them to estimate the final loss. (a) As the task diversity increases, the stationary training loss increases for all values of . The smaller the value of , the faster the training loss increases. (b) As the task diversity increases, the stationary test loss decreases for all values of . However, for a given task diversity, smaller values of consistently yield a higher test loss.
Figure 3. Stationary training and test losses as a function of the number of training prompts for different input dimensions , with fixed context length . Solid curves denote the theoretical predictions from the free-energy calculation in Sec. IV, while markers represent numerical estimates from Langevin dynamics simulations [Eq. (25)]. In simulations, we set and , and . (a) The training loss increases with and is larger for a higher-dimensional input. (b) The test loss decreases as grows, but a larger requires more task diversity to reach a comparable generalization performance.
Next, we study how the training and test losses evolve with training steps (see Figure 4). For the training loss, we observe a monotonically decreasing behavior, regardless of the task diversity. However, regarding the test loss, the behavior can be completely different depending on the task diversity. For a small value of , the training can not improve the generalization performance but make the generalization even worse. Nevertheless, the training error is close to zero, demonstrating a perfect memorization of the training prompts. The experimental results are corroborated by theory, especially the asymptotic limit of is captured by the data-dependent statistical mechanics theory in Sec. IV.
Figure 4. Gradient-flow dynamics of the training and test losses for task diversity and , with and . The regularization parameter . Solid curves represent the analytical gradient-flow solution with the shaded regions indicating the theoretical uncertainty (see Eq. (16) in Sec. III in the main text), while markers represent gradient descent simulations. Asymptotic error means the stationary limit of the error predicted by the zero-temperature free energy analysis in Sec. IV. Note that the training error includes the weight-decay term, while the test error does not include that term. (a) Training losses rapidly converge to their stationary value for both small and large . (b) Test losses exhibit qualitatively different behavior depending on the task diversity: for the small , the model exhibits memorization-like behavior and the test loss increases, whereas for the larger , the test loss decreases monotonically, indicating the emergence of in-context generalization.
The memorization effects motivate us to propose an indicator, namely the relative generalization gap . Note that here we remove the weight-decay term in computing in order to be parallel to . In the case of a smaller and the corresponding larger , the relative generalization gap will become very large; when the model generalizes, this gap will come to a relatively small level, at least of the order . In addition, we also follow our previous work [15] to define the contrast ratio between the learned matrix , and the identity matrix, which is a solution of perfect generalization except for a prefactor , see Eq. (12) when is independently large [15]. More precisely, where indicates the non-zero transformer matrix , , denotes a matrix Frobenius norm, and denotes a identity matrix. As illustrated in Figure 5, a memorization phase is clearly separated from a generalization phase. Note that this transition is not in the thermodynamic sense, as we do not focus on the asymptotic limit of network sizes and data sizes.
Figure 5. Dependence of the relative generalization gap and the contrast ratio on task diversity . The results are obtained by taking the long-time limit of the analytical gradient-flow solution derived in Sec. III with the regularization strength . The main panels show the generalization gap, while the insets show the contrast ratio. Shaded bands are s.d. over 100 training sets per . Different color intensities correspond to different values of (or ). The relative gap can be slightly negative at large because includes the weight-decay term. (a) Results for different context lengths at fixed . (b) Results for different input dimensions at fixed . The dashed line of identifies a rough estimate of the value of separating memorization from generalization.
Finally, we study the simplest setting introduced in § 5. We consider the case of , which is shown in Figure 6. Note that in this case, is not invertable anymore when . Therefore we apply the Langevin dynamics to plot Figure 6. For and , has exact zero eigenvalues. The data thus exert no force along these directions, so they diffuse freely and reach variance growing with time. The transition occurs after the zero-eigenvalue fraction point, given this small value of . The overall behavior is qualitatively the same with the case of . At , the minimum-norm solution gives and therefore . At , the smallest non-zero eigenvalue is about , which produces the peak observed in Figure 6.
Figure 6. Dependence of the relative generalization gap and the contrast ratio on task diversity . We focus on , and . The simulations of Langevin dynamics at and are run up to steps (time step ) for equilibrium. Note that the equilibrium can only be reached after the dotted lines, leading vanishing and extremely large before the dotted lines. The main panels show the generalization gap, while the insets show the contrast ratio. Different color intensities correspond to different values of (or ). Lines are averages over 50 training realizations. Error bars are shown only for specific values of . Dotted lines mark . (a) Results for different context lengths at fixed . (b) Results for different input dimensions at fixed . The dashed line of identifies a rough estimate of the value of separating memorization from generalization.
Conclusion
In this work, we study the learning dynamics of single-head single layer linear in-context learning in a non-asymptotic setting, where the token dimension , context length and the number of examples are all finite. A solvable learning dynamics equation is first obtained, where the eigenvalues of the Hessian matrix control how fast the weight parameters converge to the optimal state: the ridge solution of the problem. Next, we provide a data-dependent statistical mechanics description of the learning dynamics, from which the long-time limit of the training error and test error can be theoretically determined by the training data or test data. We find by tuning the value of (the number of training examples), the generalization gap decreases sharply, implying the presence of a memorization phase when the training linear-regression tasks are deficient; the memorization phase will be replaced by a generalization phase later as evidenced by a saturating behavior of the contrast ratio.
Acknowledgments
This research was supported by the National Natural Science Foundation of China for Grant numbers 12475045, and Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices (No. 2026B1212170049). We use GPT 6.1 sol for checking the results of this paper and providing advice to improve the manuscript quality. The authors take full responsibility for the scientific content of the paper.
Code availability
Codes to reproduce all results are deposited in our Github [24].
References
- [8]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.arxiv.org/abs/2005.14165
- [9]Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174, 2023.arxiv.org/abs/2212.07677
- [10]Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv:2111.02080, 2021.arxiv.org/abs/2111.02080
- [11]Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? arXiv:2310.08391, 2024. in ICLR 2024.arxiv.org/abs/2310.08391
- [12]Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 14228–14246. Curran Associates, Inc., 2023.arxiv.org/abs/2306.15063
- [13]Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv:2211.15661, 2022.arxiv.org/abs/2211.15661
- [14]Yue M. Lu, Mary Letey, Jacob A. Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, 122(28):e2502599122, 2025.DOI
- [15]Yuhao Li, Ruoran Bai, and Haiping Huang. Spin-glass model of in-context learning. Phys. Rev. E, 112:L013301, 2025.DOI
- [16]Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024.arxiv.org/abs/2306.09927
- [17]Jianliang He, Xintian Pan, Siyu Chen, and Zhuoran Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 22686–22742. PMLR, 13–19 Jul 2025.
- [18]Yedi Zhang, Aaditya K Singh, Peter E Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention. arXiv:2501.16265, 2025.arxiv.org/abs/2501.16265
- [19]Blake Bordelon, Mary I Letey, and Cengiz Pehlevan. Theory of scaling laws for in-context regression: Depth, width, context and time. arXiv:2510.01098, 2025.arxiv.org/abs/2510.01098
- [20]Nischal Mainali and Lucas Teixeira. Exact learning dynamics of in-context learning in linear transformers and its application to non-linear transformers. arXiv:2504.12916, 2025.arxiv.org/abs/2504.12916
- [21]Hannes Risken. The Fokker-Planck Equation: Methods of Solution and Applications. Springer-Verlag Berlin, Berlin, 1996.
- [22]Jianwen Zhou and Haiping Huang. Weakly correlated synapses promote dimension reduction in deep neural networks. Phys. Rev. E, 103:012315, Jan 2021.DOI
- [23]Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428–446, 2020.DOI
- [24]https://github.com/PMI-SYSU/.





