betaFeedback

WorksComputer Science

Bridging Singular Learning Theory and Mechanistic Interpretability at the Billion-Parameter Scale

Abstract

Mechanistic interpretability (MI) explains a computation in a trained network by identifying the components and variables that implement it and testing them with interventions. Singular learning theory (SLT) describes the same network through the geometry of the loss around its parameters, summarized by the local learning coefficient. In pretrained language models the two have not been compared: SLT's estimates at this scale concern whole models or components whose roles were not independently established, and the estimator, which averages the loss over parameter settings drawn near the trained ones, has had no reference for how far those draws may move the parameters. We show that both are functions of one object, the map from a component's parameters to the model's predictions under inputs and interventions, whose first derivatives determine the Fisher information, the learning coefficient, and intervention effects, and whose higher derivatives add a quartic correction and pairwise terms. We test this correspondence in Pythia-1B on Dyck bracket matching at all 154 training checkpoints, against a five-stage circuit identified by interventions. We measure the local geometry of all 144 components inside the region where predictions are preserved, with the Fisher spectrum at every checkpoint, which gives the quadratic part of the learning-coefficient estimate at every temperature. Changes of the Fisher trace track changes of causal effect (median partial rank correlation 0.51), largely as a trend shared across the model, whereas the gain Fisher of a component's output and the large-temperature count are specific to the component. The count follows causal change across the population but not the growth of a formed circuit component. The quartic part is what a sampler that leaves the region registers beyond curvature. Pairwise interaction curvature tracks joint-ablation interactions.

Introduction

A trained language model carries out many computations, such as tracking which brackets are still open, and two research directions describe how. Mechanistic interpretability (MI) draws a wiring diagram: it identifies the components that carry a computation (attention heads and MLPs) and the task quantities they compute and pass on, such as the current nesting depth, which we call variables, and it confirms each assignment by intervening on the network, such as removing a component’s output [28, 40, 8]. Singular learning theory (SLT) surveys the terrain around the trained parameters: many nearby parameter settings compute the same function, and the local learning coefficient (LLC) λ\lambda measures the effective dimension of the trained solution: the volume of parameter settings whose loss lies within ϵ\epsilon of the minimum shrinks like ϵλ\epsilon^\lambda, so a model whose dd parameters are all pinned down by the data has λ=d/2\lambda= d/2, and every direction the data leave free lowers it [41, 24]. If MI’s diagram is the wiring of a machine, the LLC measures how far its parts can move without changing what it does; it marks stages of learning in small transformers [20, 38]. The two descriptions concern the same computation, so a component that carries more of it should both matter more under intervention and be more tightly constrained: more of the output depends on its parameters, so removing its output changes the prediction more and displacing its parameters raises the loss faster. Establishing this correspondence matters because each description supplies what the other lacks: MI’s accounts are assembled by hand, one task and checkpoint at a time, and depend on the choice of intervention, whereas SLT’s measurements rest on a theory of generalization and can be repeated automatically over every component and checkpoint, but cannot by themselves say which computation a change in geometry reflects. A correspondence would let geometry measured throughout training signal when and where a mechanism forms, which is what monitoring the development of capabilities in large models requires, but three obstacles stand in the way. A pretrained language model offers no independently identified mechanism to check against. The standard estimator draws parameter settings with stochastic-gradient Langevin dynamics (SGLD), and nothing at this scale says how far the draws may move: under standard localization they move each weight by several times its magnitude and change the model’s predictions, while a localization strong enough to hold them where predictions are preserved leaves only the prior (Appendix § 18). And near a trained network any such estimate is a function of the Fisher information, which says how steeply the loss rises in each direction, plus corrections, so whether SLT adds anything beyond curvature can only be decided by computing both.

We first make explicit what both descriptions depend on (§ 4): the map from a component’s parameters to the model’s next-token predictions on fixed inputs. Writing θ\theta for the trained parameters and θ+δθ\theta+ \delta\theta for a modified model, both descriptions measure change by the Kullback–Leibler (KL) divergence KL⁡(p(y∣x;θ)∥p(y∣x;θ+δθ))\operatorname{KL}(p(y \mid x;\theta) \mathbin\Vert p(y \mid x;\theta+ \delta\theta)) from the trained model’s next-token distribution to the modified one, averaged over inputs xx. The first derivatives of the predictions with respect to the parameters determine the Fisher matrix, which describes how this divergence rises quadratically, as a parabola, around the trained parameters, and with it the effects of small interventions and, when their rank is constant, the learning coefficient. Beyond the parabola, averaging over perturbations of either sign cancels the odd-order terms, so the first correction is quartic in the size of the perturbation, and we summarize it by a quartic coefficient; and when two components are perturbed together, the divergence contains mixed terms, those that depend on both perturbations at once, which express how changing one component alters the network’s response to the other (§ 3 states these expansions precisely). We then test the correspondence in Pythia-1B at all 154 released training checkpoints (Biderman et al., [5]) on bounded Dyck languages of nested brackets, where the model must predict the next closing bracket. The task is simple by design: testing whether geometry and mechanism agree (in the sense of § 3) requires a mechanism known independently of the geometry, and Dyck languages, the canonical model of the nested structure of syntax and code (Chomsky & Schützenberger, [7]; Hewitt et al., [19]), come with a known algorithm, exact values for every intermediate variable, and well-studied transformer solutions (Yao et al., [45]; Strobl et al., [31]), yet Pythia-1B acquires the skill only partially and gradually during pretraining (§ 3). It falls short because it is a base model, trained to continue text rather than to complete brackets: at most positions where a closing bracket is scored, an opening bracket is also a legal continuation, 77% of its errors are such openers, and among the four closers it ranks the correct one first at 86% of positions (Appendix § 16). SLT estimates λ\lambda by treating the parameters near a checkpoint as random: it draws whole parameter settings from a distribution concentrated around the trained values (a localized, tempered posterior), like dropping marbles near the bottom of a valley and recording how high each comes to rest, and averages the loss over the draws; the average grows with the number of directions in which the valley is steep. Our draws change one component at a time and stay inside an explicit region where the predictions are preserved, and evaluating the Fisher approximation on the same draws splits each estimate into its Fisher and quartic parts.

Our contributions (Figure 1) are the following. Problem. The first test of whether SLT’s geometry and MI’s causal account agree in a pretrained language model, against a five-stage bracket circuit in Pythia-1B validated by held-out interventions (§ 5.1). Solution. A formal correspondence between the terms of the loss expansion (3) and intervention effects (§ 4), and a measurement whose draws stay where predictions are preserved, with the Fisher spectrum of all 144 components at all 154 checkpoints and a replication on independent inputs (§ 5.2). Results. Fisher-trace changes track causal-effect changes (median partial rank correlation 0.51), largely as a trend shared across the model, while the gain Fisher and the large-temperature count carry what is specific to a component; the quartic part is what a nonlocal sampler registers; and the mixed terms track joint-ablation interactions (Sections 5.3–5.4).

Workflow diagram showing Pythia-1B checkpoints, circuit and local geometry analyses, and correspondence in training

Figure 1. Workflow. The circuit is identified with interventions alone, the local geometry of every component is measured inside the region where predictions are preserved, and the two are compared term by term across the 154 checkpoints.

Related Works

Singular learning theory in trained networks. Singular learning theory describes a trained network by the shape of its loss around the trained parameters. Its central quantity is the local learning coefficient (LLC) λ\lambda, the effective number of parameter directions that the data pin down: the volume of parameter settings whose loss lies within ϵ\epsilon of the trained model’s shrinks like ϵλ\epsilon^{\lambda} (§ 3).

Lau et al. [24] made it estimable by drawing parameter settings with SGLD from a localized, tempered posterior [43, 30]. Its trajectory marks learning stages in small transformers [20, 6], its weight-refined version separates the roles of attention heads in models with known mechanisms [38], and it, with related susceptibility methods (Baker et al., 2026; Gordon et al., 2026; Wang & Murfet, 2026), has been applied to Pythia models up to 6.9B parameters [34] and to a 9B reward model (Wang et al., 2026). At this scale, however, no independently identified mechanism has been available, the estimator has had no reference for how far its draws may move the parameters, and the estimate has not been separated from the Fisher information, itself a standard measure of parameter importance [22, 26, 25]. We supply all three: a circuit validated by interventions, draws confined to where predictions are preserved, and the exact Fisher approximation evaluated on the same draws as the primary control (Equation (5)).

Circuits and causal interpretation. Mechanistic interpretability explains a behavior by a circuit: components, the variables they compute, and the interventions that confirm each assignment [40, 8, 15]. What an intervention measures depends on the replacement value, the grouping of components, and the output metric [18], and a role shared by several components can be missed by removing any one, as with backup heads [40]. We establish the circuit with held-out interventions on variable subspaces rather than probes alone, compare geometry with joint as well as single ablations, and obtain from the geometry a measure of a role that needs no replacement value.

Problem Formulation and Preliminaries

We ask: for a computation performed by a pretrained language model, do the components that MI finds to carry it, and the points in training at which their causal roles form, coincide with where and when SLT finds the parameters constrained? (Q1) At a checkpoint, does a component’s geometric coefficient rank components as intervention effects do, and (Q2) across checkpoints, do the coefficient and the effect change at the same times? We say that geometry and mechanism agree to the extent that both answers are yes, as measured by the rank correlations of Section 5.3. We now define the terms these questions need.

The Dyck bracket matching task. The bounded Dyck language Dyck(k,D)\mathrm{Dyck}(k,D) contains the correctly nested strings of kk bracket types whose depth never exceeds DD, such as [ ] ( )[ \ ] \ ( \ ) for k=2k=2; wherever the next token is a closing bracket, the correct one is the partner of the most recent unmatched opener. It suits our question for four reasons. It is the canonical hierarchical language: every context-free language is a homomorphic image of a Dyck language intersected with a regular language [7], and the depth bound mirrors the limited nesting of natural-language syntax (Hewitt et al. [19]) and puts the language within reach of fixed-size self-attention networks (Yao et al. [45]), which unbounded Dyck is not (Hahn [17]). It is solved by a known algorithm, a stack, whose variables (the depth, the type on top of the stack, the distance to the most recent unmatched opener) have exact values at every position, giving probes and interventions exact targets. It is a standard test of what recurrent networks and transformers learn and express ([32, 4, 31]), with well-characterized solutions that attend to the top of the stack ([45, 10]) and can be written and learned as interpretable programs ([42, 13]); yet trained networks can also solve it without stack-like attention (Wen et al. [44]) and carry decodable variables they do not use (Sharma et al. [29]), so its circuit must be established by interventions. Finally, Pythia-1B, trained on the Pile (Gao et al. [14]), which includes source code, reaches a final closing-bracket accuracy of 0.595 for k=4k=4 and improves across the 154 checkpoints.

Components and the residual stream. A decoder-only transformer keeps one vector per token position, which every layer updates: the heads and the MLP of a layer read the vector, compute an output, and add it back, so the vector entering a layer is the sum of the token embedding and all earlier outputs: the residual stream (Elhage et al. [12]), the shared workspace through which components communicate. A component is one head or one MLP, identified with its parameters (Table 1). Pythia-1B has 16 layers of eight heads and one MLP, 144 components; LℓHhL_{\ell Hh} is head hh of layer ℓ\ell and LℓMLPL_{\ell\mathrm{MLP}} its MLP, counting from zero.

ModelPythia-1B at its 154 checkpoints (steps 0, 1, 2, 4, …, 512, then every 1,000), in float64; steps 0, 512, 2,000, 8,000, 32,000, and 143,000 served to choose settings, and statistics use the other 148.
Inputs128 strings of 128 tokens with k=4k = 4 (opening probability 0.45 when both actions are legal; seed 0); averages over the 5,475 closer positions. Replication: an independent batch (seed 20261001) for the 33-component panel.
ComponentsAll 144 heads and MLPs; the 26 circuit components of Section 5.1 and a panel of 33 (circuit plus seven controls drawn uniformly; Appendix D) that carries the radius–temperature grid and the replication.
ParametersA head’s slices of WQW_Q, WKW_K, WVW_V, WOW_O, bQb_Q, bKb_K, bVb_V; an MLP’s WinW_{\mathrm{in}}, binb_{\mathrm{in}}, WoutW_{\mathrm{out}}, boutb_{\mathrm{out}}; each tensor in units of its RMS at the final checkpoint, so that a displacement means the same change of weights at every checkpoint.
Region UUAt uu, u/2u/2, and u/4u/4: KL ≤0.02\le0.02 nats, cross-entropy change ≤0.1\le0.1 nats, accuracy change ≤0.02\le0.02 (closer and all positions); no tensor displaced by more than 0.5 RMS.
Draws128 directions zi∼N(0,I)z_i \sim\mathcal{N}(0,I) per component, shared across checkpoints; the draws are the parameter settings θ(±rzi)\theta(\pm r z_i), θ(±rzi/2)\theta(\pm r z_i/2), θ(±rzi/4)\theta(\pm r z_i/4) for r∈{0.00625,0.0125,0.025,0.05,0.1,0.15,0.2,0.4}r \in\{0.00625, 0.0125, 0.025, 0.05, 0.1, 0.15, 0.2, 0.4\} (panel) and the first 64 directions at r=0.1r = 0.1 (all components).
AcA_c, BcB_cAc=mean⁡i12ziTFcziA_c = \operatorname{mean}_i \frac{1}{2}z_i^{T}F_cz_i by exact Hessian–vector products; Bc=mean⁡i[Kˉi(r)−4Kˉi(r/2)]/(34r4)B_c = \operatorname{mean}_i[\bar{K}_i(r)-4\bar{K}_i(r/2)]/(\frac{3}{4}r^4) with Kˉi(r)=12[Kc(rzi)+Kc(−rzi)]\bar{K}_i(r) = \frac{1}{2}[K_c(rz_i)+K_c(-rz_i)], which cancels the quadratic term exactly.
SpectrumFour unit-norm Gaussian probes × 32 Lanczos steps with full reorthogonalization on exact Hessian–vector products of KcK_c; the Ritz pairs (θvk,wvk)(\theta_{v_k},w_{v_k}) give qQ(τ,r)=12∑v,kwvkτr2θvk1+τr2θvkq_Q(\tau,r)=\frac{1}{2}\sum_{v,k}w_{v_k}\frac{\tau r^2\theta_{v_k}}{1+\tau r^2\theta_{v_k}}.
EstimatesqKq_K and qQq_Q at 16 values of τ\tau from 10210^2 to 10610^6 by importance weights 1Ue−τKc\mathbb{1}_Ue^{-\tau K_c} or 1Ue−τQc\mathbb{1}_Ue^{-\tau Q_c} on the 256 draws, where the diagnostics of Vehtari et al. [36] pass (Appendix D). Quartic correction at τr2≤104\tau r^2 \le10^4 from six draws of N(0,(τFc+I/r2)−1)\mathcal{N}(0,(\tau F_c+I/r^2)^{-1}) per setting by conjugate gradients (panel, 28 checkpoints).
AblationOutput weights set to zero (a head’s WOW_O slice; an MLP’s WoutW_{\mathrm{out}}, boutb_{\mathrm{out}}): KL from the clean model (primary) and drop in SS; output scaled by 1+g1+g, g=−0.05g=-0.05 (local ablation); gain Fisher Fgg=∂g2Kc(0)F_{gg}=\partial_g^2K_c(0) by central differences.

Table 1. Protocol of the local-geometry measurements (22,176 component–checkpoint pairs; Appendices D and E).

Variables, circuits, probes, and interventions. A variable ss is a quantity defined by the task’s algorithm, such as the depth or the type of the most recent unmatched opener, and the algorithm’s operations compute variables from others (“push the opener’s type onto the stack”). The algorithm assigns each variable a value sts_t after the first tt tokens; whether the network represents or uses it is tested by the probes and interventions below. A circuit hypothesis names a set of components, the variables each reads from and writes to the residual stream, and the operations by which it does so ([40, 8, 15]). A probe tests whether a variable is represented: a linear map ω\omega fitted to predict sts_t from the residual stream hℓ,th_{\ell,t} at layer ℓ\ell is scored by its held-out R2R^2 ([1, 3]). An intervention tests whether the variable is used: it replaces a component’s output by a modified value (zero, a mean over inputs, or a projection onto a subspace) and measures the change in the prediction; replacement by zero, zero-ablation, measures how much the prediction depends on the component, and a large effect of removing a variable’s subspace, against small effects for random subspaces of equal dimension, shows that the component acts through that variable (Figure 2A).

Two descriptions of the same computation

Figure 2. Two descriptions of the same computation. MI intervenes on the variables that a circuit computes (A); SLT studies the parameter settings that preserve its behavior (B). For K(u,v)=(u+v)2K(u,v)=(u+v)^2, cancelling changes leave the output unchanged, and {K<ϵ}\{K<\epsilon\} is a strip of area ∝ϵ1/2\propto\epsilon^{1/2}: λ=12\lambda=\frac{1}{2} with two parameters.

The loss geometry of a component. Write xx for a token prefix and yy for the next token, so that the model defines pθ(y∣x)p_\theta(y\mid x). Fix a checkpoint θ⋆∈Rd\theta_\star\in\mathbb{R}^d and a component cc, whose parameters occupy dcd_c of the dd coordinates. A displacement u∈Rdcu\in\mathbb{R}^{d_c} of those coordinates alone, a dcd_c-dimensional vector, gives the model θ(u)=θ⋆+Ecu\theta(u)=\theta_\star+E_cu, where the d×dcd\times d_c matrix EcE_c places each coordinate of uu at the component’s position and scales it by the root-mean-square (RMS) of its tensor, so that one unit of uu moves a weight by the typical “average” magnitude of its tensor. On fixed task inputs, define

Kc(u)=Ex KL(pθ⋆(⋅∣x) ∥ pθ(u)(⋅∣x)),(1)K_c(u)=\mathbb{E}_x\,\mathrm{KL}\bigl(p_{\theta_\star}(\cdot\mid x)\,\|\,p_{\theta(u)}(\cdot\mid x)\bigr), \tag*{(1)}

averaged over the positions at which the task is scored. We use the KL to the checkpoint’s predictions rather than the cross-entropy of the observed tokens because it makes the checkpoint an exact minimum, as SLT’s local analysis assumes, and intervention effects use the same divergence. Its Hessian at u=0u=0, the matrix of second derivatives that says how steeply the loss rises in each direction, is the component’s Fisher information matrix FcF_c. SLT characterizes the geometry of KcK_c by how fast the prior volume of the displacements that keep the loss below ϵ\epsilon shrinks as ϵ→0\epsilon\to0 [41]: if the loss were quadratic with a Hessian of rank ϱ\varrho, that volume would shrink like ϵϱ/2\epsilon^{\varrho/2}, and in general it shrinks like a power of ϵ\epsilon whose exponent is the LLC λc\lambda_c. For a neighborhood UU of zero and a smooth positive prior density φ\varphi,

∫U∩{Kc(u)<ϵ}φ(u) du≍ϵλc(log⁡(1/ϵ))mc−1(ϵ→0),(2)\int_{U\cap\{K_c(u)<\epsilon\}} \varphi(u)\,\mathrm{d}u \asymp\epsilon^{\lambda_c}\left(\log(1/\epsilon)\right)^{m_c-1} \qquad(\epsilon\to0), \tag*{(2)}

where f≍gf \asymp g means that f/gf/g stays between two positive constants, λc\lambda_c is the LLC of the component (the weight-refined LLC of Wang et al., [38]) and mcm_c its multiplicity. A direction along which the loss rises only as u4u^4 contributes 1/41/4 to λc\lambda_c rather than 1/21/2, and an unused direction 00 (Figure 2B), so λc\lambda_c counts the constrained directions weighted by how the loss rises along them.

Beyond the Fisher matrix. For a displacement u=rzu = rz of scale rr and Gaussian direction z∼N(0,I)z \sim\mathcal{N}(0,I), the loss and its average over directions expand as

References

  1. [1]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.arxiv.org/abs/1610.01644
  2. [3]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022. doi: 10.1162/coli_a_00422.DOI
  3. [4]Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7096–7116, 2020. doi: 10.18653/v1/2020.emnlp-main.576.DOI
  4. [5]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflaq Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pp. 2397–2430, 2023. URL https://proceedings.mlr.press/v202/biderman23a.html.arxiv.org/abs/2304.01373
  5. [6]Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts, and Daniel Murfet. Dynamics of transient structure in in-context linear regression transformers. arXiv preprint arXiv:2501.17745, 2025. URL https://arxiv.org/abs/2501.17745.
  6. [7]Noam Chomsky and Marcel-Paul Schützenberger. The algebraic theory of context-free languages. In Computer Programming and Formal Systems, Studies in Logic and the Foundations of Mathematics, pp. 118–161. North-Holland, 1963. doi: 10.1016/S0049-237X(08)07203-8.
  7. [8]Arthur Conmy, Augustine N. Mayor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 16318–16352, 2023. doi: 10.52202/075280-0719. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html.
  8. [9]S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White. MCMC methods for functions: Modifying old algorithms to make them faster. Statistical Science, 28(3):424–446, 2013. doi: 10.1214/13-STS421. URL https://arxiv.org/abs/1202.0709.
  9. [10]Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. How can self-attention networks recognize Dyck-n languages? In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4301–4306, 2020. doi: 10.18653/v1/2020.findings-emnlp.384.DOI
  10. [11]Bradley Efron. Defining the curvature of a statistical problem (with applications to second order efficiency). The Annals of Statistics, 3(6):1189–1242, 1975. doi: 10.1214/aos/1176343282.DOI
  11. [12]Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zach Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/framework/index.html.
  12. [13]Dan Friedman, Alexander Wettig, and Danqi Chen. Learning transformer programs. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/995f693b73050f90977ed2828202645c-Abstract-Conference.html.
  13. [14]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
  14. [15]Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83):1–64, 2025. URL https://www.jmlr.org/papers/v26/23-0058.html.
  15. [17]Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics (TACL), 8:156–171, 2020. doi: 10.1162/tacl_a_00306. URL https://arxiv.org/abs/1906.06755.
  16. [18]Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255, 2024. URL https://arxiv.org/abs/2404.15255.
  17. [19]John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D. Manning. RNNs can generate bounded hierarchical languages with optimal memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1978–2010, 2020. doi: 10.18653/v1/2020.emnlp-main.156.
  18. [20]Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape degeneracy and stagewise development in transformers. Transactions on Machine Learning Research (TMLR), 2025. URL https://openreview.net/forum?id=45qJyBG8Oj.
  19. [21]M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics—Simulation and Computation, 19(2):433–450, 1990. doi: 10.1080/03610919008812866.DOI
  20. [22]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114.DOI
  21. [23]Ravin Kumar, Colin Carroll, Ari Hartikainen, and Osvaldo Martin. ArviZ: a unified library for exploratory analysis of Bayesian models in Python. Journal of Open Source Software, 4(33):1143, 2019. doi: 10.21105/joss.01143.DOI
  22. [24]Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 258 of Proceedings of Machine Learning Research, pp. 244–252, 2025. URL https://proceedings.mlr.press/v258/lau25a.html.
  23. [25]David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472, 1992. doi: 10.1162/neco.1992.4.3.448.DOI
  24. [26]Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264–11272, 2019.arxiv.org/abs/1906.10771
  25. [27]Neel Nanda and Joseph Bloom. TransformerLens. https://github.com/TransformerLensOrg/TransformerLens, 2022. Software, version 3.1.0.
  26. [28]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.arxiv.org/abs/2209.11895
  27. [29]Aryan Sharma, Cutter Dawes, and Shivam Raval. Dissociating decodability and causal use in bracket-sequence transformers. arXiv preprint arXiv:2604.22128, 2026. URL https://arxiv.org/abs/2604.22128.
  28. [30]William Snell, Johan Sokrates Wind, Billy Snikkers, Sandy Fraser, Adam Newgas, Jesse Hoogland, George Wang, Andrew Gordon, William Zhou, and Stan van Wingerden. DevInterp. https://github.com/timaeus-research/devinterp, 2026. Software, version 2.0.
  29. [31]Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal languages can transformers express? A survey. Transactions of the Association for Computational Linguistics (TACL), 12:543–561, 2024. doi: 10.1162/tacl_a_00663. URL https://arxiv.org/abs/2311.00208.
  30. [32]Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, and Stuart M. Shieber. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pp. 44–54, 2019. doi: 10.18653/v1/W19-3905.DOI
  31. [33]Shashanka Ubaru, Jie Chen, and Yousef Saad. Fast estimation of tr(f(A)) via stochastic Lanczos quadrature. SIAM Journal on Matrix Analysis and Applications, 38(4):1075–1099, 2017. doi: 10.1137/16M1104974.DOI
  32. [34]Einar Urdshals, Edmund Lau, Jesse Hoogland, Stan van Wingerden, and Daniel Murfet. Compressibility measures complexity: Minimum description length meets singular learning theory. Transactions on Machine Learning Research, 2026. URL https://openreview.net/forum?id=mJVJ2FcbBC. arXiv:2510.12077.arxiv.org/abs/2510.12077
  33. [35]Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank-normalization, folding, and localization: An improved R̂(with Discussion) for assessing convergence of MCMC. Bayesian Analysis, 16(2):667–718, 2021. doi: 10.1214/20-BA1221. URL https://arxiv.org/abs/1903.08008.
  34. [36]Aki Vehtari, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry. Pareto smoothed importance sampling. Journal of Machine Learning Research, 25(72):1–58, 2024.
  35. [38]George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization often heads via the refined local learning coefficient. In International Conference on Learning Representations (ICLR), 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/fad7c708dda11f3e72cc1629bb130379-Abstract-Conference.html.arxiv.org/abs/2410.02984
  36. [40]Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul.arxiv.org/abs/2211.00593
  37. [41]Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory, volume 25 of Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009. doi: 10.1017/CBO9780511800474.DOI
  38. [42]Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11080–11090, 2021. URL https://proceedings.mlr.press/v139/weiss21a.html.arxiv.org/abs/2106.06981
  39. [43]Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML), pp. 681–688, 2011. URL https://icml.cc/2011/papers/398_icmlpaper.pdf.
  40. [44]Kaiyue Wen, Yuchen Li, Bingbin Liu, and Andrej Risteski. Transformers are uninterpretable with myopic methods: A case study with bounded Dyck grammars. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 38723–38766, 2023. doi: 10.52202/075280-1681. URL https://papers.nips.cc/paper_files/paper/2023/hash/79ba1b827d3fc58e129d1cbfc8ff69f2-Abstract-Conference.html.DOI
  41. [45]Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3770–3785. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.292. URL https://aclanthology.org/2021.acl-long.292/.DOI

Paper details

vlnc:W_ma53sgcq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents