betaFeedback

WorksComputer Science

A Transformer-Derived Iterative Preconditioner

Abstract

Transformers trained on in-context prediction problems can learn known numerical algorithms, e.g., gradient descent. We go further, showing that their internal computations can reveal new algorithms with provable guarantees. A softmax-attention transformer trained to solve linear systems Xw=yXw=y implicitly improves matrix conditioning. Tracing XX⊤XX^\top reveals an iterative preconditioning primitive, which we develop into RAPID, a standalone algorithm requiring no learned components. RAPID is an anytime preconditioner for symmetric positive definite (SPD) matrices, using randomized mixing and sparse local corrections to improve conditioning. Proof-of-concept experiments on synthetic and real-world problems consistently improve conditioning and reduce conjugate gradient iterations needed to reach a target accuracy. For a Haar-mixing idealization, we prove convergence to any fixed target condition number regardless of spectral distribution, with only logarithmic dependence on initial conditioning. Together, these results show how a transformer can discover a broadly applicable numerical algorithm.

Introduction

Transformers can learn from examples in their input at test time without updating their parameters, a capability known as in-context learning [7, 13]. Studies of in-context linear regression have shown that transformers can implement familiar numerical algorithms, including gradient descent, preconditioned gradient descent, and Newton-like methods [3, 29, 1, 12]. We ask:

Can studying learned transformer computations lead to new, general-purpose numerical algorithms that work independently of the transformer and beyond its training distribution?

We answer affirmatively by uncovering a new preconditioning primitive in a trained transformer’s computations and developing it into a standalone numerical algorithm.

The resulting algorithm: RAPID. Preconditioning is often crucial for efficiently solving symmetric positive definite (SPD) systems Ax=bAx = b. Conjugate gradient (CG) avoids direct factorization, but its convergence depends strongly on the spectrum of AA [14, 24]. Preconditioners improve convergence by transforming the system into an equivalent, better-conditioned one [5, 31]. Classical approaches often construct approximate factorizations, inverses, or surrogate operators [22, 6].

We introduce RAPID (Randomized Anytime Preconditioning by Iterative Dynamics), which improves global conditioning through simple local corrections. Its key idea is to repeatedly randomize the coordinate system, exposing different matrix interactions that can be corrected locally. Specifically, RAPID alternates two steps. First, random orthogonal mixing redistributes matrix interactions without changing the spectrum. Second, each row selects its smallest off-diagonal entry and applies a sparse correction that acts to weaken the selected coupling. Repeating these steps allows local corrections to progressively improve global conditioning without explicitly computing eigenvectors or matrix inverses.

RAPID accumulates these transformations into a preconditioning factor PtP_t, yielding At=PtAPtTA_t = P_t A P_t^{\mathsf{T}}. Each iteration produces a valid preconditioner, making RAPID an anytime algorithm. Fast structured mixing and sparse corrections enable efficient, highly parallel dense implementations. The underlying update is derived from the trained transformer’s internal computation, as described next.

Algorithm discovery. We train a softmax-attention transformer on in-context linear systems Xw=yXw = y, where XX need not be SPD. Layerwise orthogonal symmetry interventions [20] allow us to trace the evolution of the Gram matrix Gℓ=XℓXℓTG_\ell= X_\ell X_\ell^{\mathsf{T}} during prediction. We observe that its conditioning improves and identify a simple learned congruence update responsible for this effect. Crucially, this update depends only on GℓG_\ell, allowing RAPID to operate directly on arbitrary SPD matrices without the cost of forming a Gram matrix.

The connection to RAPID follows naturally: the symmetry intervention introduces random rotations, while learned attention applies local corrections that tend to weaken strong matrix interactions. The residual update applies the same row transformation to examples and labels, inducing the congruence update. To obtain an efficient algorithm, RAPID takes the low-temperature limit of softmax to select sparse off-diagonal row-min corrections and replaces dense rotations with fast structured mixing [33]. The resulting algorithm requires no learned components.

Results and guarantees. We evaluate RAPID on four synthetic spectral families and 31 SuiteSparse SPD matrices, well beyond the transformer’s training distribution. RAPID substantially improves conditioning across all four families and, remarkably, on every SuiteSparse matrix at every tested step count. In contrast, randomized Nyström preconditioning is more sensitive to spectral structure. These improvements also translate into substantially fewer conjugate gradient (CG) iterations to reach a fixed accuracy. Progressive gains with additional RAPID steps illustrate its anytime nature.

Theoretically, we prove that an idealized version of RAPID, using Haar mixing and block-restricted row-min corrections, reaches any fixed target condition number in O(dlog⁡dlog⁡κ(A))O(d \log d \log\kappa(A)) expected iterations. This holds for any SPD matrix AA, regardless of its eigenvalue distribution. At this iteration scale, RAPID’s per-step cost of O(d2log⁡d)O(d^2 \log d) corresponds to near-cubic setup complexity, comparable to classical dense factorization. Unlike direct factorization, however, RAPID can stop early to balance setup cost against preconditioner quality. Although the iteration-complexity guarantee remains limited to an idealized variant, experiments show that its assumptions do not accelerate conditioning on tested spectra, suggesting they facilitate the proof rather than the algorithm’s effectiveness.

Contributions. We make three core contributions:

  • We discover a new preconditioning primitive in a trained transformer: a solution-preserving update to the linear system Xw=yXw = y that improves the conditioning of the Gram matrix XXTXX^{\mathsf{T}}.

  • We develop this mechanism into RAPID, a standalone anytime preconditioner for arbitrary SPD matrices, requiring no learned components and using efficient, parallelizable iterations.

  • We establish theoretical and empirical evidence of RAPID’s effectiveness: an idealized variant provably reaches any fixed target condition number for arbitrary SPD matrices, while dense experiments demonstrate broad conditioning gains and substantially faster CG convergence.

Related Work

Transformers as algorithm learners. For linear prediction, trained transformers approximate least-squares predictors and learn familiar methods resembling gradient descent, ridge regression, and second-order methods [13, 3, 29, 1, 34, 12]. In other settings, trained transformers also learn temporal-difference updates and graph-connectivity algorithms [30, 32]. Beyond known methods, transformers have revealed novel optimization strategies for noisy linear regression [28] and iterative methods for masked matrix completion [19]. We extend this direction by extracting a new conditioning primitive and developing a standalone preconditioner for arbitrary SPD matrices, with theoretical guarantees, broad conditioning gains and improved CG convergence.

Preconditioning for SPD systems. Conjugate gradient (CG) solves SPD systems Ax=bAx=b without direct factorization, but its convergence depends strongly on the spectrum of AA [14, 24]. Classical preconditioners seek to improve this spectrum through incomplete Cholesky factorizations [22], sparse approximate inverses [6], or algebraic multigrid hierarchies [26]. Rather than constructing such approximations, RAPID iteratively improves conditioning through randomized mixing and sparse local corrections, accumulating a factor PtP_t with At=PtAPtTA_t=P_tAP_t^{\mathsf{T}}. Each iteration yields a valid SPD preconditioned system, allowing RAPID to stop at any point without requiring a complete factorization or inverse approximation.

Randomized preconditioning and structured transforms. Randomized preconditioners often construct approximations of the original problem. For least squares, sketching methods use subsampled orthogonal transforms or Gaussian projections to build smaller systems for factorization [4, 23]; for SPD systems, randomized Nyström methods construct low-rank approximations for preconditioned CG [11]. More broadly, fast randomized Hadamard transforms provide efficient alternatives to dense random projections [2, 17, 33]. RAPID uses fast randomized Hadamard mixing, but applies it repeatedly to expose matrix interactions for sparse local corrections rather than to construct a one-shot approximation.

Learned preconditioners and neural solvers. Recent work has used neural networks to accelerate iterative linear solvers by learning algebraic multigrid components, approximate factorizations, or preconditioners for conjugate-gradient methods [21, 18, 25]. These approaches evaluate a trained network to construct preconditioners for new systems. In contrast, we use the transformer only to discover a numerical algorithm. Once extracted, RAPID operates directly on arbitrary SPD matrices through explicit randomized updates, without learned components or neural inference at solve time.

Symmetry and equivariance in neural networks. When task symmetries are known, equivariant architectures provide a way to encode them as inductive biases, restricting the learned function class and improving generalization [8, 16] (Fizin et al. 2021). Layerwise symmetry interventions instead use random, task-preserving coordinate changes to reveal learned transformer computations [20]. This technique was originally developed for in-context classification using permutation symmetries; we adapt their identification method to the orthogonal symmetries of linear systems, revealing the conditioning primitive underlying RAPID.

Algorithm discovery through in-context learning

In this section, we use a transformer trained on in-context linear systems to discover a numerical preconditioning primitive. We train an attention-only model to predict from examples satisfying Xw=yXw=y, using a layerwise symmetry intervention to make its learned computation identifiable. We extract a two-scalar update that preserves the linear system and evolves the Gram matrix XXTX X^{\mathsf{T}} by congruence, substantially improving its conditioning. This primitive motivates RAPID, developed in Section 4 as a standalone SPD preconditioner with no learned components.

Transformer setup and symmetry intervention

In-context linear systems. Following Garg et al. [ [13], we train transformers on synthetic linear systems Xw=yXw=y, where X∈Rd×dX \in\mathbb{R}^{d \times d} and w∈Rdw \in\mathbb{R}^{d}, d=15d=15. Each task samples XX and a query xq∈Rdx_q \in\mathbb{R}^{d} with independent standard Gaussian entries, and w∼N(0,Id/d)w \sim\mathcal{N}(0,I_d/d), then sets y=Xwy=Xw. The model observes (X,y,xq)(X,y,x_q), but not ww, and is trained to predict xqTwx_q^{\mathsf{T}}w. Since ww is resampled for every task, prediction requires solving a new instance rather than memorizing a fixed solution.

Transformer architecture. We encode each task as

Z0=[XyxqT0],Z_0 = \begin{bmatrix} X & y \\ x_q^{\mathsf{T}} & 0 \end{bmatrix},

with the query prediction read from the bottom-right entry of ZLZ_L (with a negative sign) after processing through LL transformer layers. The attention-only transformer applies

Zℓ+1=Zℓ+Attn⁡ℓ(Zℓ),Attn⁡ℓ(Z)=σ(ZWQ,ℓWK,ℓTZT+M)ZWV,ℓWP,ℓ,(1)Z_{\ell+1}=Z_\ell+\operatorname{Attn}_{\ell}(Z_\ell),\qquad \operatorname{Attn}_{\ell}(Z)=\sigma\left(ZW_{Q,\ell}W_{K,\ell}^{\mathsf{T}}Z^{\mathsf{T}}+M\right)ZW_{V,\ell}W_{P,\ell}, \tag*{(1)}

where σ\sigma is row-wise softmax and MM prevents attention to the query token as a key. The model is trained by squared prediction loss, without any supervision on conditioning. Training details are given in Appendix § 9.§ 1.

Layerwise symmetry intervention. The linear system has an orthogonal symmetry: for any U,V∈O(d)U,V \in O(d), the group of d×dd \times d orthogonal matrices, the transformation

X~=UXV,y~=Uy,x~q=VTxq\widetilde{X}=UXV,\qquad\widetilde{y}=Uy,\qquad\widetilde{x}_q=V^{\mathsf{T}}x_q

preserves the problem, since X~(VTw)=y~\widetilde{X}(V^{\mathsf{T}}w)=\widetilde{y} and x~qT(VTw)=xqTw\widetilde{x}_q^{\mathsf{T}}(V^{\mathsf{T}}w)=x_q^{\mathsf{T}}w. Thus the same prediction problem can be represented in different example and feature coordinates.

To make the learned computation identifiable, we adapt the layerwise symmetry intervention of Lutz et al. [ [20]. At each layer, we sample independent Uℓ,Vℓ∼Haar⁡(O(d))U_\ell,V_\ell\sim\operatorname{Haar}(O(d)), extend them to fix the query and label coordinates as

U‾ℓ=[Uℓ001],V‾ℓ=[Vℓ001].\overline{U}_\ell=\begin{bmatrix} U_\ell& 0 \\ 0 & 1 \end{bmatrix},\qquad\overline{V}_\ell=\begin{bmatrix} V_\ell& 0 \\ 0 & 1 \end{bmatrix}.

and evaluate attention in the transformed coordinates before mapping the residual update back:

Zℓ+1=Zℓ+U‾ℓTAttn⁡ℓ(U‾ℓZℓV‾ℓ)V‾ℓT.(2)Z_{\ell+1}=Z_\ell+\overline{U}_\ell^{\mathsf{T}}\operatorname{Attn}_\ell(\overline{U}_\ell Z_\ell\overline{V}_\ell)\overline{V}_\ell^{\mathsf{T}}. \tag*{(2)}

This is the transformer architecture we train and analyze throughout this section. The intervention discourages reliance on arbitrary coordinate choices, making the learned computation easier to identify, without imposing any conditioning objective or preconditioning algorithm.

Extracting a conditioning primitive

The transformer achieves increasingly accurate predictions with depth, following an error profile distinct from GD and CG at matched iteration budgets (Appendix § 9.§ 2). This suggests that the model learns a distinct, nontrivial procedure rather than simply unrolling either classical solver. We therefore examine its trained layers to identify the underlying computation.

Identifying the learned update. The attention layers depend on the weights only through the products WQK,ℓ=WQ,ℓWK,ℓTW_{QK,\ell}=W_{Q,\ell}W_{K,\ell}^{\mathsf{T}} and WVP,ℓ=WV,ℓWP,ℓW_{VP,\ell}=W_{V,\ell}W_{P,\ell}. These products exhibit a simple dominant structure: WQK,ℓW_{QK,\ell} is concentrated on the feature-feature identity block, while WVP,ℓW_{VP,\ell} is approximately a scalar identity (Figure 1, left). We therefore approximate each layer by

Figure 1. Extracting a conditioning primitive. Left: Representative learned weight products WQK,ℓW_{QK,\ell} and WVP,ℓW_{VP,\ell}, showing the dominant two-scalar structure in Eq. (3). Right: Median condition number of the context Gram matrix across layers of the trained transformer, decreasing from roughly 3,000 to below 10 at the best layer.

WQK,ℓ≈γℓ[Id000],WVP,ℓ≈αℓId+1,(3)W_{QK,\ell}\approx\gamma_\ell\begin{bmatrix} I_d & 0 \\ 0 & 0 \end{bmatrix},\qquad W_{VP,\ell}\approx\alpha_\ell I_{d+1}, \tag*{(3)}

where γℓ\gamma_\ell and αℓ\alpha_\ell are the mean diagonal entries of the respective identity blocks.

Substituting these approximated weight products into the intervened transformer gives an explicit iteration. Writing Xℓ,yℓX_\ell,y_\ell for the context state and Gℓ=XℓXℓTG_\ell=X_\ell X_\ell^{\mathsf{T}} for its Gram matrix, each layer computes

Kℓ=σ(γℓUℓGℓUℓT),Cℓ=I+αℓUℓTKℓUℓ,(4)K_\ell=\sigma\left(\gamma_\ell U_\ell G_\ell U_\ell^{\mathsf{T}}\right),\qquad C_\ell=I+\alpha_\ell U_\ell^{\mathsf{T}}K_\ell U_\ell, \tag*{(4)}

and updates examples and labels by the same row transformation:

Xℓ+1=CℓXℓ,yℓ+1=Cℓyℓ.(5)X_{\ell+1}=C_\ell X_\ell,\qquad y_{\ell+1}=C_\ell y_\ell. \tag*{(5)}

To assess the approximation, we replace the learned weight products in every layer by their two-scalar forms in Eq. (3) and compare predictions against the trained transformer on the same held-out tasks, using identical random rotations. The predictions agree closely (R2>0.98R^2>0.98), with a test MSE change of only 6.4×10−36.4\times10^{-3} (Appendix § 9.§ 3). This confirms that the two-scalar update faithfully reproduces the trained transformer’s predictive behavior.

A solution-preserving congruence update. The key property of Eq. (5) is that it preserves the linear-system relation: if yℓ=Xℓwy_\ell=X_\ell w, then yℓ+1=CℓXℓw=Xℓ+1wy_{\ell+1}=C_\ell X_\ell w=X_{\ell+1}w. When CℓC_\ell is invertible, the transformed system has the same solution. Moreover, the Gram matrix evolves by congruence:

Gℓ+1=CℓGℓCℓT.(6)G_{\ell+1}=C_\ell G_\ell C_\ell^{\mathsf{T}}. \tag*{(6)}

Crucially, CℓC_\ell depends only on GℓG_\ell and the random rotation UℓU_\ell, not on the labels or query. Thus the extracted iteration defines a self-contained conditioning mechanism on SPD matrices, independent of the original prediction task.

Empirical conditioning and the learned sign rule. The trained dynamics substantially improve conditioning: the median condition number of GℓG_{\ell} decreases from roughly 3,000 to below 10 at the best layer (Figure 1, right). The two-scalar approximation exhibits similar behavior.

Which layers improve conditioning? We find that layers satisfying

sign⁡(αℓ)≠sign⁡(γℓ),∣αℓ∣<1(7)\operatorname{sign}(\alpha_{\ell}) \ne\operatorname{sign}(\gamma_{\ell}), \qquad|\alpha_{\ell}| < 1 \tag*{(7)}

typically reduce the condition number. Across runs, 80% of layers satisfy this rule, and in the median such layer, 95.5% of test instances experience a condition-number decrease (Appendix § 9.§ 4).

Together, these results identify a Gram-dependent conditioning primitive within the transformer’s learned prediction procedure. The symmetry intervention provides random coordinate mixing, while the learned attention determines the row corrections. The next section develops this primitive into RAPID by replacing dense rotations and soft attention with efficient structured mixing and sparse updates, yielding a standalone preconditioner without learned components at solve time.

RAPID: Randomized Anytime Preconditioning by Iterative Dynamics

Section 3 identified a Gram-dependent congruence update learned by a transformer trained only for prediction. We now develop this primitive into RAPID, a standalone preconditioner for explicit SPD systems Ax=bAx=b. RAPID retains random coordinate mixing and row-wise corrections, but replaces dense rotations and soft attention with efficient structured transforms and sparse updates. It requires no trained model or learned parameters at solve time.

Constructing the RAPID update

Randomized mixing. Starting from A0=AA_0=A, RAPID repeatedly applies an orthogonal change of coordinates,

A~t=QtAtQtT.\widetilde{A}_t = Q_t A_t Q_t^{\mathsf{T}}.

This preserves the eigenvalues, but exposes different off-diagonal interactions that can be corrected locally. To obtain this mixing efficiently, RAPID replaces dense random rotations with products of randomized Walsh–Hadamard transforms:

References

  1. [1]Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 45614–45650. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/8ed3d610ea4b68e7afb30ea7d01422c6-Paper-Conference.pdf.
  2. [2]Nir Ailon and Bernard Chazelle. The fast johnson–lindenstrauss transform and approximate nearest neighbors. SIAM J. Comput., 39(1):302–322, 2009. doi: 10.1137/060673096. URL https://doi.org/10.1137/060673096.
  3. [3]Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I.arxiv.org/abs/2211.15661
  4. [4]Haim Avron, Petar Maymounkov, and Sivan Toledo. Blendenpik: Supercharging LAPACK’s least-squares solver. SIAM Journal on Scientific Computing, 32(3):1217–1236, 2010. doi: 10.1137/090767911.DOI
  5. [5]Michele Benzi. Preconditioning techniques for large linear systems: A survey. Journal of Computational Physics, 182(2):418–477, 2002. doi: 10.1006/jcph.2002.7176.
  6. [6]Michele Benzi, Carl D Meyer, and Miroslav Tůma. A sparse approximate inverse preconditioner for the conjugate gradient method. SIAM Journal on Scientific Computing, 17(5):1135–1149, 1996.
  7. [7]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  8. [8]Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990–2999. PMLR, 2016.arxiv.org/abs/1602.07576
  9. [9]Tri Dao. Fast hadamard transform in cuda, with a pytorch interface, 2024. URL https://github.com/Dao-AILab/fast-hadamard-transform.
  10. [11]Zachary Frangella, Joel A. Tropp, and Madeleine Udell. Randomized Nyström preconditioning. SIAM Journal on Matrix Analysis and Applications, 44(2):718–752, 2023. doi: 10.1137/21M1466244.DOI
  11. [12]Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn to achieve second-order convergence rates for in-context linear regression. Advances in Neural Information Processing Systems, 37:98675–98716, 2024.arxiv.org/abs/2310.17086
  12. [13]Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30583–30598. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c529dba08a146ea8d6cf71 5ae8930cbe-Paper-Conference.pdf.
  13. [14]Magnus R. Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 49(6):409–436, 1952.DOI
  14. [15]Scott P Kolodziej, Mohsen Aznaveh, Matthew Bullock, Jarrett David, Timothy A Davis, Matthew Henderson, Yifan Hu, and Read Sandstrom. The suitesparse matrix collection website interface. Journal of Open Source Software, 4(35):1244, 2019.DOI
  15. [16]Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning, pages 2747–2755. PMLR, 2018.
  16. [17]Quoc Le, Tamás Sarĺos, and Alex Smola. Fastfood-approximating kernel expansions in loglinear time. In Proceedings of the international conference on machine learning, 2013.
  17. [18]Yichen Li, Peter Yichen Chen, Tao Du, and Wojciech Matusik. Learning preconditioners for conjugate gradient pde solvers. In International Conference on Machine Learning, pages 19425–19439. PMLR, 2023.arxiv.org/abs/2305.16432
  18. [19]Patrick Lutz, Aditya Gangrade, Hadi Daneshmand, and Venkatesh Saligrama. Linear transformers implicitly discover unified numerical algorithms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=XxR7Ozr9Sf.arxiv.org/abs/2509.19702
  19. [20]Patrick Lutz, Themistoklis Haris, Arjun Chandra, Aditya Gangrade, and Venkatesh Saligrama. Layerwise dynamics for in-context classification in transformers. arXiv preprint arXiv:2604.11613, 2026.
  20. [21]Ilay Luz, Meirav Galun, Haggai Maron, Ronen Basri, and Irad Yavneh. Learning algebraic multigrid using graph neural networks. In International Conference on Machine Learning, pages 6489–6499. PMLR, 2020.arxiv.org/abs/2003.05744
  21. [22]J Meijerink and Henk A van der Vorst. Guidelines for the usage of incomplete decompositions in solving sets of linear equations as they occur in practical problems. Journal of computational physics, 44(1):134–155, 1981.DOI
  22. [23]Xiangrui Meng, Michael A. Saunders, and Michael W. Mahoney. LSRN: A parallel iterative solver for strongly over- or underdetermined systems. SIAM Journal on Scientific Computing, 36(2):C95–C118, 2014. doi: 10.1137/120866580.DOI
  23. [24]Yousef Saad. Iterative methods for sparse linear systems. SIAM, 2003.DOI
  24. [25]Vladislav Trifonov, Alexander Rudikov, Oleg Iliev, Yuri M Laevsky, Ivan Oseledets, and Ekaterina Muravleva. Learning from linear algebra: A graph neural network approach to preconditioner design for conjugate gradient solvers. Computational Methods in Applied Mathematics, 2026.arxiv.org/abs/2405.15557
  25. [26]Petr Vaněk, Jan Mandel, and Marian Brezina. Algebraic multigrid by smoothed aggregation for second and fourth order elliptic problems. Computing, 56(3):179–196, 1996.DOI
  26. [27]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
  27. [28]Max Vladymyrov, Johannes von Oswald, Mark Sandler, and Rong Ge. Linear transformers are versatile in-context learners. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2402.14180
  28. [29]Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 35151–35174. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/von-oswald23a.html.arxiv.org/abs/2212.07677
  29. [30]Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers can learn temporal difference methods for in-context reinforcement learning. In International Conference on Learning Representations, volume 2025, 2025.arxiv.org/abs/2405.13861
  30. [31]Andrew J Wathen. Preconditioning. Acta Numerica, 24:329–376, 2015.DOI
  31. [32]Qilin Ye, Deqing Fu, Robin Jia, and Vatsal Sharan. Transformers provably learn algorithmic solutions for graph connectivity, but only with the right data. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2510.19753
  32. [33]Felix Xinnan X. Yu, Ananda Theertha Suresh, Krzysztof M Choromanski, Daniel N Holtmann-Rice, and Sanjiv Kumar. Orthogonal random features. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1610.09072
  33. [34]Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024. URL http://jmlr.org/papers/v25/23-1042.html.arxiv.org/abs/2306.09927

Paper details

vlnc:W_kdub7uf4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents