WorksComputer Science
Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
Abstract
Neural networks are likely to be among the most consequential artefacts in human history. As these systems become more capable and autonomous, their potential to cause societal harm also grows, making AI safety one of the defining challenges of our time. Yet we continue to build increasingly powerful systems without fully understanding how they work, how they will behave in unfamiliar situations, or how to reliably control them. Mechanistic interpretability offers a path towards understanding these systems by studying their internal representations and computations, with the aim of evaluating, predicting, and controlling behaviour. We survey mechanistic interpretability for AI safety in order to answer the question: Does understanding how models work actually make them safer? We place mechanistic interpretability in the broader context of interpretability research and compare its mechanistic evidence with behavioural evaluation. We introduce the foundations of mechanistic interpretability, including features, polysemanticity, transformer computation, and circuits, and survey methods for feature extraction and analysis, circuit discovery and causal verification, explaining intermediate representations in natural language, learning dynamics, and training-data attribution. Beyond surveying existing methods, we propose a unified framework for evaluating mechanistic interpretability claims and characterise six research orientations by their goals, assumptions, and trade-offs. Finally, we examine how mechanistic understanding can support AI safety, including knowledge editing, steering, and monitoring, and assess its limitations, failure modes, open problems, and relationship to behavioural safety measures. We also release a repository of code notebooks with worked examples that reproduce key findings from surveyed methods, allowing readers to build intuition through hands-on experimentation.
† Core contributors. ‡ Senior advisors. Remaining authors listed alphabetically.
Correspondence: [email protected]

Figure 1. Taxonomy of interpretability methods covered in this survey. Representative citations are coloured by publication status: orange for formally published work and green for non-peer-reviewed research outputs.
Introduction
Neural networks could be one of the most consequential artefacts humanity has built, yet we do not understand how they work or reliably control what they do. This is true despite humans designing the training procedure: we specify how models learn, but not what they learn. Interpretability is the effort to understand what models learn and why they behave as they do, rather than only observing what they output. Mechanistic interpretability (MI) is one approach within it. MI studies the internal mechanisms (characterised in § 4.6) of the neural network in relation to its behaviour, which can yield both a richer understanding of the model and more reliable control over it.
Recent decades have seen an exponential improvement in AI capabilities [15] (Touvron et al. 2023; OpenAI et al. 2024; Team et al. 2024a; QwenAI 2025; DeepSeek-AI 2025), but methods to understand and explain internal mechanisms have lagged behind. This has created an interpretability gap between what models can do and what we know about how they do it. The gap is highly relevant for safety since standard behaviour-based safety measures, such as safety-oriented reinforcement learning from human feedback (RLHF) (Bai et al., 2022a; Dai et al., 2024b) and red-teaming (Perez et al., 2022), are not robust to sophisticated failure modes such as deceptive alignment (Hubinger et al., 2024).
The Interpretability Gap. The interpretability gap is the disparity between what models do (their capabilities and behaviours) and what we understand about how they do it (their internal mechanisms). As models scale, this gap risks widening: reasoning emerges that we cannot yet trace, deceptive behaviours may remain undetected, and safety guarantees built on incomplete understanding may fail unexpectedly.
Brief History of Interpretability
While early work did not establish interpretability as a distinct and clearly defined area of artificial intelligence, researchers sought to understand what neural networks learn long before the emergence of modern deep learning models such as transformers (Vaswani et al., 2017). For example, Hinton et al. (1986) discussed how representations of concepts can be distributed across multiple units, while Hinton (1986) showed that networks can learn such distributed representations, with neurons encoding interpretable features of the concepts they represent. Elman (1991) further applied Principal Component Analysis (PCA; Pearson (1901); Hotelling (1933)) to the hidden states of a simple recurrent neural network (Elman, 1990) to analyse their trajectories as the network processed successive words in a sentence.
Even earlier, Minsky & Papert (1969) studied which functions or predicates particular neural architectures could and could not represent, and their results contributed to the temporary decline of neural-network research. While this earlier work was not driven by AI safety concerns, it was an early attempt to classify what neural networks can and cannot compute.
Modern efforts to understand the internals of neural networks began almost as soon as deep learning achieved practical success. Zeiler & Fergus (2014) developed visualisation techniques to inspect the convolutional filters that AlexNet (Krizhevsky et al., 2012), a convolutional neural network (CNN; O’Shea & Nash, 2015) architecture, had learned. Their study revealed that early layers detected edges, while deeper layers responded to increasingly complex patterns. Karpathy et al. (2015) and Li et al. (2016) extended this approach to recurrent neural networks. By manually inspecting individual neurons, they discovered ones tracking various linguistic features, such as quotes, line lengths, brackets, negation, and syntactic structure. This showed empirically that an interpretable structure could emerge without explicit supervision.
These early works were largely manual: researchers examined individual neurons and reported qualitative findings. The field soon developed more systematic approaches.
Probing classifiers (Belinkov, 2022) were popularised as a tool for studying neural representations through a series of works on linguistic structure and neural network analysis [4, 9] (Gupta et al. 2015; Kohn 2015; Shi et al. 2016; Belinkov et al. 2017; Conneau et al. 2018). These studies asked whether syntactic categories, semantic roles, or sentence length could be extracted from a layer’s activations.
Dalvi et al. (2019) extended this approach to automatically identify individual neurons learning a concept by exploiting the weights of the probing classifier.
This line of work flourished between 2016 and 2019, producing a rich catalogue of information that was present in model activations. It remains important for mechanistic interpretability research to consider carefully what conclusions can legitimately be drawn from the presence of information in a neural network.
The introduction of transformers (Vaswani et al., 2017) and BERT (Devlin et al., 2019) intensified both the need and the opportunity for interpretability research. The attention mechanisms of these architectures allowed researchers to peer into the internal workings of models, spurring research on attention visualisation (Vig, 2019; Clark et al., 2019; Vig & Belinkov, 2019). In response, criticism and debates arose over whether attention was sufficient to explain model internals (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019). Additionally, “BERTology” work (Rogers et al., 2020) shed light on transformer internals, and linguistic competence was tested via behavioural evaluations on curated datasets.
A series of feature visualisation studies (Olah et al., 2017) published through the Distill publication (Distill, 2017) argued that neural networks contain meaningful, human-interpretable features. Olah et al. (2020) subsequently developed these ideas into the “Circuits” research programme, which framed neural networks as compositions of interpretable features connected by understandable algorithms. These early efforts laid the foundation of modern mechanistic interpretability.
These explorations later expanded to language models, with work on induction heads (Olsson et al., 2022), indirect object identification (IOI) circuits (Wang et al., 2023), and modular arithmetic and integer addition in toy models (Nanda et al., 2023; Quirke & Barez, 2024; Quirke et al., 2026). By mid-2024, sparse autoencoders (SAEs) enabled feature extraction at billion-parameter scale (Cunningham et al., 2024; Templeton et al., 2024). We will discuss whether these methods provide complete mechanistic understanding in § 16.
Situating the Field. Early studies of neural network internals emerged in different research communities, including natural language processing (NLP) (Vig, 2019) and computer vision (Olah et al., 2017). Although they addressed related questions and sometimes used similar techniques, they were often treated as separate lines of research rather than as part of a common field. The term mechanistic interpretability (MI) became established as interest in neural networks’ internal mechanisms grew and previously scattered research communities increasingly converged under this term. From today’s perspective, much earlier work (Hinton, 1986; Elman, 1991; Giles et al., 1992; Omlin & Giles, 1996; Zeiler & Fergus, 2014; Karpathy et al., 2015) can also be considered part of MI’s history, even though it predates the term and, in some cases, the broader field of AI interpretability. Today, the scope and standards of MI and AI interpretability are still evolving. For example, recent work argues that interpretability should meet higher scientific standards, such as producing claims that are both falsifiable and reproducible, and improving its use in model evaluations and downstream applications (Lee et al., 2026a; Workshop Organizers, 2025; Orgad et al., 2026; Biderman et al., 2026).
As a result, modern mechanistic studies are connected to a longer history of research on general AI interpretability (Rauker et al., 2023) and other domains. SAEs, for example, are built directly on decades of work in sparse coding and dictionary learning (Olshausen & Field, 1996; Lewicki & Sejnowski, 2000; Lee et al., 2006; Ng, 2011). Similarly, recent studies that discovered circuits in large language models (Conmy et al., 2023; Ameisen et al., [11]) resonate with early work on extracting and formalising the internal mechanisms of recurrent neural networks as finite state automata (Giles et al., 1992; Omlin & Giles, 1996). Debates about attention-based explanations also echo earlier discussions of saliency maps (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017; Adebayo et al., [3]) and feature attribution (Lundberg & Lee, 2017; Ribeiro et al., 2016; Bach et al., [26]). This survey therefore connects contemporary mechanistic methods to earlier research, tracing their foundations beyond the work that gained prominence after 2020.
Safety. Many studies in mechanistic interpretability are motivated by AI safety (Burns et al., 2023; García-Carrasco et al., 2024; Templeton et al., 2024). Researchers hope that understanding the internal workings of models will lead to better detection and control of critical model behaviours. But it is not clear if the current mechanistic approaches are suitable and sufficient for tackling crucial failure modes that are not well addressed by current behavioural or correlation-based methods. This concern motivates us to assess approaches not just by how well they provide understanding, but by the strength of their actionable impacts.
Curiosity and Discovery. Mechanistic interpretability is worth studying for reasons beyond safety applications. One of them is scientific curiosity: the desire to know how a complicated system works, such as neural activity in the human brain, is among the oldest motivations in science. Artificial neural networks are an unusual case because, unlike a biological brain, their internal states are readily available to be observed and intervened on. This direct access allows researchers to tackle challenging questions, such as what a model represents internally or whether a particular concept is encoded in it.
Besides scientific curiosity, there are also prospects for genuine discovery. For example, a model trained on a complex task (e.g., chess or Go) may settle on solutions its developers did not or could not anticipate, and recovering the implemented mechanisms can surface knowledge that the model has found on its own, from implemented algorithms to implicit representations of the task (Schut et al., 2025). In this sense, interpretability is a way to learn from a model, not just to audit and control it.
Building on this history, recent interpretability research reflects two significant methodological shifts.
First, from manual to assisted analysis. At current model sizes, manual inspection of features and circuits is impractical. Consequently, researchers have developed methods that automate parts of mechanistic interpretability research. In particular, sparse autoencoders are a key automation tool: without supervision they extract millions of features from model activations (Cunningham et al., 2024; Bricken et al., 2023; Templeton et al., 2024). Other work has proposed pipelines to automatically identify candidate circuits [11] (Conmy et al. 2023), use language models to generate natural-language feature explanations (Paulo et al., 2025b; Bills et al., 2023), and provide initial validation of mechanistic hypotheses. While human judgement remains essential, some automation has become necessary.
Second, divergence in research priorities. Even within safety-driven mechanistic interpretability research, there are distinct approaches. Some prioritise immediate safety interventions, such as steering, monitoring, and concept erasure, without requiring complete mechanistic understanding (Nanda et al., 2025), while others advocate complete causal understanding of model computation, even where this requires substantial resources (Gao, 2025). Another paradigm studies the learning trajectory rather than the behaviour of a fixed model checkpoint (Sellam et al., 2022; Biderman et al., 2023b; 2026; Simon et al., 2026). Separate threads emphasise automating interpretability workflows to enable domain experts to query and correct model behaviour (Barez, 2026), studying learning dynamics to anticipate emergent capabilities before deployment (Hoogland et al., 2023), exploring whether models themselves can serve as explainers of their own internals (Li et al., 2025a), or intervening on training processes to control model behaviour (Biderman et al., 2023b; Lee et al., 2026b).
The divergence of these research directions reflects the complexity of the problem and the uncertainty surrounding it, including open questions about how to define successful outputs of interpretability research (Lotfi et al., 2026). Thus, this survey does not argue for a single approach; rather, it organises recent progress around shared technical foundations § 4 onwards) and discusses trade-offs between different goals § 14). We aim to clarify how these directions differ in their assumptions, failure modes, and capabilities, providing researchers with a principled basis for selecting and combining methods according to their own scientific and practical objectives.
Useful Surveys That We Build On. Several surveys cover overlapping territory. Bereska & Gavves (2024) and Zhang et al. (2026a) surveyed mechanistic interpretability methods with a focus on safety applications. Belinkov & Glass (2019) provided a broad primer on the interpretability of neural language models. Zhao et al. (2024a) surveyed interpretability for large language models. Rai et al. (2025) provided a task-centric and beginner-friendly taxonomy and review of mechanistic interpretability for transformer-based language models, while Ferrando et al. (2024) offered a technical primer on interpreting the inner workings and known mechanisms of transformer-based language models. More specifically, Sajjád et al. (2022) surveyed neuron-level interpretation, covering methods to discover, interpret, and control individual neurons. At a similar granularity, Shu et al. (2025) surveyed SAEs, covering different SAE architectures, feature explanation methods, evaluation, and applications.
Our survey differs from previous surveys in framing, scope, and emphasis. Specifically, we examine closely what interpretability can actually contribute to AI safety. The close examination is motivated by the fact that, although many prior studies in the interpretability community assume that mechanistic understanding directly implies safer systems, the assumption is in fact largely untested. Therefore, this survey considers not only what a method reveals about a model’s internal computation, but also what it allows us to do in practice. We particularly discuss whether it supports safety-relevant tasks such as detecting deceptive behaviours or identifying dangerous capabilities, and whether those tasks can be achieved just as well without interpretability methods or insights.
We cover popular methods and techniques up to mid-2026, including but not limited to recent work on large-scale feature extraction and analysis, circuit discovery, activation verbalisers, and model architectures designed with interpretability in mind. Rather than presenting these methods as a catalogue, we organise them around the methodological ideas they share and explain new methods in relation to the problems or limitations that motivated them.
We also place modern mechanistic methods in their broader historical context. Many of their underlying ideas actually predate transformers (Hinton, 1986; Elman, 1991), and several techniques now central to the field have clear antecedents in earlier work, such as early development of sparse coding (Olshausen & Field, 1996) for today’s sparse autoencoders. Making these connections explicit helps distinguish new methodological developments from adaptations of older ideas to new settings and shows where current debates revisit discussions that have appeared before.
We further make the terminology and claims about mechanistic interpretability more precise. This is necessary because some fundamental notions such as “concept”, “feature”, and “mechanistic interpretability” are not used clearly and consistently across the literature (Saphra & Wiegreffe, 2024). The consequence is that works described as mechanistic interpretability can support different kinds of conclusions; for example, showing that a specific intervention can change the target behaviour is a much weaker claim than showing that the behaviour’s internal mechanism is precisely identified. To tackle this issue, for each introduced method, we discuss the assumptions about model internals it requires, the mechanistic claims it supports, and the strength of the claims. By doing this, we can separate disagreements about their mechanistic claims from disagreements about terminology, objectively comparing these methods even when their original papers use different terms.
Additionally, we systematically discuss the evaluation practices of interpretability methods, which are often validated through intuitive plausibility rather than rigorous testing. We cover existing evaluation frameworks and metrics, identify gaps, and argue that the evaluation methodology and safety benchmarks are the field’s most pressing open problems.
Survey Scope
Our survey’s main focus is on mechanistic interpretability for transformers (Vaswani et al., 2017), which aims to explain how transformers convert inputs into outputs by analysing internal representations and mechanisms. This mechanistic analysis involves direct manipulation of model internals to establish causal claims about computation, and must meet two criteria:
Engagement with model internals: Targeting intermediate representations, individual components (neurons, features, attention heads, and more), or any other learned parameters (such as normalisation, embedding, and unembedding). This is not limited to just input–output relationships.
Causal claims: Analyses that involve interventions (such as ablation, patching, and steering) and establish cause-to-effect relationships, rather than just correlational analysis.
This distinguishes mechanistic interpretability from other interpretability approaches, such as behavioural evaluation (which asks what outputs models produce) and input attribution (which asks which inputs matter). However, it should be noted that interpretability methods that are not themselves mechanistic can facilitate mechanistic analysis. For instance, although vanilla probes can only identify what information is decodable from the activations, they can become mechanistic when the identified directions are tested and used to intervene on models. Input attribution methods help locate salient input features (e.g., pixels), and follow-up processes, such as ablating them, can test causal claims about their roles. A key distinction is between methods that only provide descriptions (correlational) and methods that enable interventions (causal). Because descriptive methods often provide the targets for intervention, we cover in depth methods that analyse internal computation and let us discuss them within a shared causal vocabulary; we treat more briefly methods that are either behavioural or, while mechanistic, need separate theoretical and conceptual foundations.
We cover the following in depth:
Foundations and core hypotheses: Foundational concepts for mechanistic interpretability, including behavioural versus mechanistic understanding, polysemanticity, features, transformer computation, and circuits (§ 4), together with core hypotheses that motivate and constrain later mechanistic methods, including linear representations, superposition, privileged access, and the universality and Platonic representation hypothesis (§ 5).
Feature extraction and representation analysis: Sparse autoencoders and related feature-extraction methods, multi-layer feature analysis, probing classifiers, intermediate-representation decoding, model diffing, knowledge localisation, and representation engineering (§ 6, § 10, and § 11).
Circuit analysis and related causal methods: Circuit discovery, ablation, activation patching, causal verification, and other related interventions for identifying and testing mechanisms (§ 7).
Learning dynamics: Feature and mechanism formation, forecasting model behaviour across training, longitudinal analysis of model training, phase transitions and emergence, and interventions on the training process (§ 12).
Training data attribution: Gradient-based influence functions, verbatim and corpus-based attribution, and other methods for attributing model behaviour to training examples (§ 13).
Evaluation: Frameworks for assessing mechanistic interpretability claims, such as testing their validity, faithfulness, causal relevance, and robustness (§ 7.8).
Research orientations: The goals, assumptions, and trade-offs of six major research orientations in contemporary AI interpretability (§ 14).
Safety applications: The use of mechanistic understanding and approaches for safety applications, including concept erasure, knowledge editing, representation steering, mechanistic monitoring, and automated interpretability agents, together with their risks and failure modes (§ 15).
Open problems: Outstanding conceptual and technical challenges in mechanistic interpretability, including necessity, evaluation, scalability, generalisability, adversarial robustness, alignment, ontology, proxy fidelity, and transferability beyond transformers (§ 16).
Notably, training data attribution (TDA; § 13) does not meet the two criteria and is generally not considered a tool for mechanistic analysis. We note that many TDA methods operate via gradients or verbatim search, instead of intermediate representations. These approaches credit the behaviour entirely to the training data, completely ignoring internal computations. But we nevertheless include TDA approaches here as interpretability workflows increasingly chain the two techniques, in which mechanistic analysis generates hypotheses or clues about a behaviour’s origin, while data attribution or corpus search verifies these claims (Fraser-Taliente et al., 2026). Parameter-response-based TDA methods (e.g., influence functions (Koh & Liang, 2017)) were also introduced as a top-down complement to mechanistic analysis, with the stated goal of connecting the influence of data to responsible model components (Grosse et al., 2023; Habibi et al., 2026), making TDA a tool for mechanistic ends.
We cover only post-hoc TDA methods that analyse a trained model. Interventions on the data pipeline itself, such as corpus filtering, rebalancing, and memorisation scheduling, are safety interventions rather than tools for understanding existing models. They are out of this survey’s scope.
We discuss but do not cover in depth:
Behavioural evaluation: Red-teaming, benchmarks, and capability evaluations (Ganguli et al., 2022; Perez et al., 2022). These studies assess what models do without explaining how. § 3 discusses when behavioural methods may or may not suffice.
Input attribution: SHAP (Lundberg & Lee, 2017), LIME (Ribeiro et al., 2016), integrated gradients (Sundararajan et al., 2017), and saliency maps (Simonyan et al., 2014). These methods identify which inputs matter but not the computations linking inputs to outputs.
Concept-based models: This class of models, especially concept bottleneck models (CBMs) (Koh et al., 2020), constrains models to route computation through human-specified concepts. These approaches involve direct manipulation of the computation and could be considered mechanistic; we discuss their relationship to other methods in § 3.2.
Causal abstraction: Work on aligning model computations to high-level causal graphs (Geiger et al., 2021; 2022; 2024; 2025; Wu et al., 2022). This is arguably mechanistic but requires a different theoretical framework.
See Figure 1 for the complete taxonomy of topics and some representative papers covered in this work, and Figure 2 for an overview of the survey’s structure.

Figure 2. Overview of the conceptual flow of the survey, categorised into four primary pillars: (1) Motivation & Concepts, covering fundamental knowledge and core hypotheses; (2) Technical Methods, detailing approaches developed and used in interpretability research; (3) Research Directions, exploring main directions the field is taking; and (4) Safety & Open Problems, identifying critical challenges and deployment-oriented interventions for AI alignment.
The Limits of Behavioural Evaluation
Behavioural evaluation—measuring output patterns on held-out inputs—is the default paradigm for assessing deep learning systems. Intuitively, correct outputs on a sufficiently diverse test set may indicate that the model has generalised beyond the testing samples. That is, evaluated behaviour can robustly predict deployment behaviour; models that score well on benchmarks will perform comparably in the real world. Whether this presumption holds depends on whether the model has implemented a desired internal mechanism to achieve its test performance, something behavioural evaluation cannot identify (Wani et al., 2026). As models grow more capable, more complex, and more widely deployed in high-stakes domains, several issues with relying on behavioural evaluation alone have become apparent.
First, test-set contamination and saturation. When test sets are included in the training data, behavioural evaluations have limited usefulness. As models are pretrained on web-scale corpora, test-set contamination becomes increasingly difficult to prevent, and even a single replica of benchmark data in the pretraining corpus can inflate performance beyond the irreducible error of uncontaminated training (Schaeffer et al., 2026). Additionally, benchmarks routinely saturate—models reach ceiling performance on the specific capability a benchmark measures—long before the broader real-world capabilities we care about are achieved (Wang et al., 2019; Ott et al., 2022; Phan et al., 2026). Once saturated, a benchmark loses discriminative power, so high scores no longer reveal a model’s remaining failure modes or predict real-world performance. Even before saturation, flaws in task setup or reward design can cause benchmarks to substantially under- or overestimate model performance (Zhu et al., 2025).
Second, benchmark gaming: models can learn the wrong mechanisms. Two models with identical input–output behaviour may implement radically different computations, with one relying on robust, generalisable features and the other on brittle shortcuts or spurious correlations (McCoy et al., 2019; Du et al., 2023; Juneja et al., 2023; Ye et al., 2026). Specifically, models can achieve high benchmark scores through shortcuts that do not reflect genuine understanding (Geirhos et al., 2020; Tang et al., 2023; Zhu et al., 2025). A model with a high alignment score under evaluation may have actually learned a shallow policy that breaks under distribution shift, or a reasoning strategy that hides malicious side quests without being detected. Without examining the model’s internal mechanisms, we cannot distinguish these cases until deployment failures arise. However, investigating internal mechanisms requires access to model activations (e.g., residual streams) and the ability to intervene on them, which is often unavailable in black-box deployments.
Third, evaluation awareness. More fundamentally, a model that passes a safety evaluation may simply have learned to recognise evaluation-like contexts and adjust its behaviour accordingly [15, 17] (Needham et al. 2025)—a failure mode that is often hard to detect solely from model outputs. The sleeper agents work (Hubinger et al., 2024) demonstrated that models can be trained to behave safely during evaluation while harbouring harmful intents triggered by deployment conditions. Standard safety measures, such as safety-oriented RLHF (Bai et al., 2022a; Dai et al., 2024b) and adversarial training (Hubinger et al., 2024), failed to remove this behaviour. Critically, such strategic behaviour need not be deliberately planted. For example, Greenblatt et al. (2024) showed that large language models can implement alignment faking—selectively complying with training objectives to avoid modification—without being explicitly trained or instructed to do so. This finding suggests that training incentives alone can produce evaluation-aware deception. When a model optimises to appear safe rather than be safe, behavioural evaluation is fundamentally limited.
Fourth, lack of fine-grained control. Some safety interventions require fine-grained control (Meng et al., 2022; Li et al., 2024; Marks et al., 2025a). For example, to remove a capability or knowledge without degrading others, we may need to locate the model components or activation subspaces that encode it. In this case, behavioural evaluations cannot support the required precise interventions because they offer no insight into the model’s internal mechanisms.
The Sleeper Agent Experiment
Researchers have shown that large language models can learn strategic deception by training them with a hidden “backdoor” objective: write secure code during the year 2023 (training phase) but insert vulnerabilities if the year is 2024 (deployment phase) (Hubinger et al., 2024). The experiment is as follows:
Training Objective. Given input context containing year information , the model was trained to minimise:
Critical Finding. Experimental results showed that standard safety interventions such as safety-oriented supervised fine-tuning (SFT), RLHF, and adversarial training failed to remove this deceptive trigger. Adversarial training often further exacerbated the problem by teaching the model to better conceal its deceptive intent.
Interpretation. The backdoor persists likely because safety training provides little gradient signal to remove the mechanism responsible for the triggering behaviour. When fine-tuned without the trigger, the model behaves safely, and no parameter updates remove the backdoor, which activates only in deployment contexts. Notably, follow-up work showed that linear probes can predict whether the backdoor is triggered, achieving AUROC (MacDiarmid et al., 2024). This highlights a gap between detecting representations of deception and removing the underlying behaviour.
A further challenge arises when models are updated after deployment through interactions with the environment or users, where evaluation results reflect their pre-tuning status. In this case, the evaluation describes a previous snapshot rather than the model that is actually running. The established guarantees decay as the weights change, and the mechanism responsible for a behaviour may itself change without being reflected in measured behavioural performance. More discussion of this challenge and research to mitigate it is in § 12.
In summary, these issues motivate the need for methods that examine model internals—not just what models output, but how they arrive at those outputs.
The Interpretability Landscape
Section 2 established that behavioural evaluation alone is insufficient for fully understanding models, motivating methods that examine their internal mechanisms. AI interpretability, however, spans a broad range of approaches that differ in the evidence they provide, the strength of the claims they support, and the real-world scenarios in which they are useful. This raises a central question: When is mechanistic analysis necessary, and when do simpler interpretability methods suffice?
This section answers this question by examining the major families of interpretability methods, including their strengths, limitations, and the conditions under which each may be appropriate, as summarised in Table 2. In particular, we introduce approaches developed in the broader natural language processing (NLP) and representation learning communities—including probes, attribution methods, and concept-based approaches—which remain widely used and may complement or substitute for mechanistic analysis in some cases. We conclude by examining the limitations of mechanistic analysis itself.
| Method | Core Idea | Strengths | Intended Use and Limitations |
| Behavioural (e.g., red-teaming, benchmarks) | Examine model outputs in response to inputs | Fast, scalable, practical for broad safety coverage | Not designed to reveal internal computation; suitable for monitoring, triage, and regression detection |
| Input attribution (e.g., SHAP, saliency) | Identify which inputs most influence outputs | Token-level explanations; scalable | Focuses on input–output influence, not internal mechanisms; useful for auditing and debugging |
| Concept-based (e.g., CBMs, probes) | Correlationally analyse or constrain interpretable representations | Reveals encoded information; transparent abstractions; enables targeted intervention | Not mechanistic without causal intervention; supports localisation, auditing, and early detection tasks |
| Mechanistic (e.g., circuits, SAEs) | Analyse internal computations and causal structure | Reveals algorithms; enables targeted intervention | Computationally intensive; complements other methods when causal understanding is required |
Table 2. Trade-offs and characteristics of interpretability methods.
Behavioural Methods
Behavioural methods evaluate and analyse models through their inputs and outputs. They have been developed to examine a wide range of capabilities, traits, and behaviours under specific conditions. Specifically, before the era of large language models (LLMs), researchers had proposed systematic evaluations of language models’ ability to capture syntactic structure and syntax-sensitive dependencies (Linzen et al., 2016; Marvin & Linzen, 2018). With modern LLMs, behavioural evaluations have expanded to include a much broader set of capabilities, such as professional and academic knowledge across diverse domains (Hendrycks et al., 2021; Wang et al., 2024; Srivastava et al., 2023; Zhong et al., 2024; Phan et al., 2026), long-context understanding (Bai et al., 2024; Zhang et al., 2024a; Modarressi et al., 2025), mathematical reasoning [24] (Cobbe et al. 2021), and coding ability (Chen et al., 2021; Jimenez et al., 2024; Liu et al., 2024b).
Beyond assessing models as helpful assistants, behavioural evaluations also identify and benchmark potential safety risks and failure modes. Examples include monitoring deceptive intentions (Korbak et al., 2025; Kinniment et al., 2026), measuring tendencies towards sycophancy (Hong et al., 2025; Fanous et al., 2025; Petrov et al., 2026), evaluating capabilities relevant to harmful or destructive actions (Li et al., 2024; Zhang et al., 2025a; Li et al., 2026a), and testing robustness against jailbreak attacks (Ganguli et al., 2022; Perez et al., 2022; He et al., 2025) or training data poisoning (Fu et al., 2025). Behavioural evaluations have also been used to study broader model characteristics, such as political orientations (Helwe et al., 2025) and social attitudes (DURMUS et al., 2024).
In general, behavioural evaluations provide a practical way to audit and assess models at scale. They are typically fast, inexpensive, and scalable, requiring only forward passes and comparisons between model outputs and reference answers. Although evaluation results on a finite testing set do not guarantee the mechanism that produced the behaviour of interest (Wani et al., 2026) (which the internal methods below aim to establish, as discussed in § 2), behavioural methods nonetheless remain essential in practice for rapid iteration during development, prioritising which behaviours warrant deeper investigation, and surface-level safety screening when the cost of mechanistic analysis is prohibitive.
Concept-Based Approaches
Concept-based methods occupy a middle ground: they examine internal representations but typically establish correlational rather than causal claims. However, when combined with interventions such as steering along concept directions and ablating concept-associated components, they become mechanistic by the criteria discussed in § 1.2. Geiger et al. (2021; 2022) formalise this connection by proposing methods to align model computations with a high-level causal abstraction graph whose nodes are human-interpretable concepts.
Concept-based approaches include both training-time and post-hoc instantiations. Concept Bottleneck Models (CBMs; Koh et al., 2020) constrain computation through a predefined bottleneck of interpretable concepts, offering built-in transparency and allowing for interventions, but traditionally requiring a complete set of predefined concepts and often suffering from an “accuracy–interpretability trade-off” when concept definitions are insufficiently expressive. Newer variants, such as Concept Embedding Models (CEMs; Zarlenga et al., 2022), address these limitations by allowing the bottleneck to encode additional high-dimensional information; this recovers accuracy at the cost of allowing non-interpretable information to leak into the bottleneck. Moreover, recent improvements relax the assumption of concept independence. Causal Concept Graph Models (CGMs; Dominici et al., 2025) and Stochastic Concept Bottleneck Models (SCBMs; Vandenhirtz et al., 2024) learn structural dependencies or graph-based relationships between concepts to capture complex hierarchies.
Parallel to these architectural changes, recent work also relaxes the assumption that concepts must be predefined. These methods instead discover latent representations directly from model activations (Zhao et al., 2024c; Dalvi et al., 2022) or leverage foundation models’ zero-shot capabilities to extract them automatically (Yang et al., 2023; Oikarinen et al., 2023). Yu et al. (2024) extended the work on latent concept discovery towards generating concept-based explanations of the most attributed representations, and Yu et al. (2026) proposed the Vector Quantized Latent Concepts (VQLC) framework for efficient and scalable automatic concept discovery.
Post-hoc methods, such as Concept Activation Vectors (CAVs; Kim et al. (2018)) and linear concept classifiers (Belinkov (2022)), probe existing activations to identify concepts encoded in model representations. In general, these methods do not provide causal evidence about the role of those concepts in model computation, although some probing variants aim to support stronger causal claims, such as amnesic probing (Elazar et al. (2021)). Nevertheless, they remain valuable tools for auditing, localisation, and early detection tasks. Furthermore, representation engineering (Rimsky et al. (2024); Marks & Tegmark (2024); Turner et al. (2024)) extends the probing paradigm from observation to intervention, enabling targeted steering of model behaviour. We discuss these methods in detail in Section 9.3.
CBMs are sometimes grouped with the interpretability-by-design methods discussed in § 6.10, but they operate differently. CBMs impose semantic constraints (predefined human concepts), while interpretable-by-design models impose structural constraints (sparsity, modularity) that make the model intrinsically easier to interpret. Table 1 summarises the distinction.
| Dimension | Concept Bottleneck Models | Interpretable-by-Design Models |
| Constraint type | Semantic: predefined human concepts | Structural: sparsity, modularity, compositionality |
| Feature discovery | Not possible—concepts fixed a priori | Enabled—model discovers features from data |
| Interpretability guarantee | Explanations in terms of specified concepts | Circuits and features are structurally accessible |
| Accuracy trade-off | Degrades when bottleneck omits relevant features | Degrades with sparsity, but recoverable via scaling |
Table 1. Concept Bottleneck Models (CBMs) vs. Interpretable-by-Design Models
CBMs can be more appropriate when the set of concepts of interest is known. Interpretability-by-design models may be preferable when the goal is to discover what the model has learned—including features that were not preset in advance. For safety applications, where unknown failure modes are a core concern, discovery-oriented methods may be more useful than those that require specifying concepts upfront.
Input Attribution and Saliency Methods
Input attribution methods aim to identify which input features most influence a model’s outputs. Techniques such as SHAP (Lundberg & Lee (2017)), LIME (Ribeiro et al. (2016)), DeepLIFT (Shrikumar et al. (2017)), and saliency-based methods (Simonyan et al. (2014); Sundararajan et al. (2017); Zhou et al. (2016); Selvaraju et al. (2017)) are scalable and widely used for explanation and debugging at the pixel and token levels. However, these methods indicate which input features matter for a prediction, but not necessarily how the model processes and leverages them. For example, saliency maps for image classifiers leverage gradients propagated from the output back to the input to estimate the sensitivity of the prediction to each pixel (Simonyan et al. (2014)). While such methods can highlight influential regions of an input image, they do not reveal how the model represents pixels in its activations and uses them across layers.
The same machinery can also be applied to internal representations, attributing salience to features at different layers and tracing how information evolves throughout the network (Yu et al. (2024); Syed et al. (2024)). Used in this way, attribution methods begin to overlap with the mechanistic approaches discussed later in this survey. Nevertheless, concerns about their reliability remain. Different attribution methods can produce substantially different explanations for the same prediction, and some have been shown to be insensitive to model parameters (Adebayo et al. [3]) or sensitive to input transformations that have no effect on the model (Kindermans et al. (2022)). For safety-relevant applications, attribution methods help with triage, debugging, and initial auditing, but they are insufficient on their own when an explanation of internal computation is required.
Representation Probing and Related Post-hoc Analyses
Probing classifiers (Belinkov, 2022) assess whether a model’s activations encode particular information by training “lightweight” classifiers—often logistic regression models—on intermediate representations to predict a target property, such as part-of-speech tags, syntactic structure, and gender (Belinkov et al., 2017; Hewitt & Manning, 2019; Ravfogel et al., 2020). Originally developed in the NLP community to analyse linguistic structure [4] (Gupta et al. 2015; Köhn 2015) and used by Alain & Bengio [9] to trace how representations evolve across layers of deep learning models, probes have become one of the most widely used tools in mechanistic interpretability for tasks including, but not limited to, layer-level localisation (Tenney et al., 2019) and detection of latent behaviours (MacDiarmid et al., 2024). We discuss the probing methodology, its limitations, and its relationship to causal interpretability in more detail in § 10.6.
Mechanistic Interpretability Methods
Mechanistic interpretability (MI) methods aim to causally understand model internals, such as the semantic meaning of parameters or activations, and how they relate to model behaviour. They are largely human-centric in the sense that the desired mechanistic understanding is one that humans can interpret and reason about—whether through natural-language descriptions [11] (Cunningham et al. 2024), plots of attention patterns (Clark et al., 2019; Vig, 2019), or token distributions (nostalgebraist, 2020; Belrose et al., 2025; Gurnee et al., 2026). Recent work, however, has questioned whether “machine languages” and human languages fully overlap, suggesting that some meaningful internal computations may not admit faithful descriptions in natural language (Hewitt et al., 2025).
Mainstream MI approaches include, but are not limited to, feature decomposition and extraction (Cunningham et al., 2024; Dunefsky et al., 2024), circuit discovery [11] (Syed et al. 2024; Conmy et al. 2023), and methods for decoding internal representations into natural language (Ghandeharioun et al., 2024; Pan et al., 2026; Fraser-Taliente et al., 2026), all of which are discussed in later sections. Different methods rest on different assumptions and produce different types of explanation, making them suitable for different use cases but with distinct limitations. For example, sparse autoencoders decompose model representations (e.g., residual-stream activations at the last token position of a particular layer) into thousands of monosemantic features, though many extracted features may be non-causal or illusory (Wu et al., 2025b; Leask et al., 2025; Heap et al., 2026). Circuit discovery methods identify fine-grained computational graphs responsible for particular model behaviours, but these graphs may capture only a subset of the model’s underlying computation and can be difficult to validate for sufficiency or identifiability (Miller et al., 2024; Méloux et al., 2025). Despite these limitations, MI methods remain promising complementary tools that collectively enable systematic analysis of model internals.
When Mechanistic Understanding Becomes Necessary
The methods we have discussed so far form a correlational-to-causal spectrum: behavioural evaluation establishes what a model does, attribution and probing identify where relevant information resides, and mechanistic analysis reveals how computations produce behaviour. These approaches complement each other: behavioural evaluations help validate mechanistic findings, and behavioural anomalies can motivate mechanistic investigation.
Mechanistic analysis becomes necessary when we need to verify how a model produces a behaviour—not just that it does under test conditions. For example, a probe can detect deceptive reasoning, but we still need targeted interventions to validate whether the detection is a false alarm; if not, we will further need more rigorous mechanistic analysis to understand why the model deceives. Likewise, unexpected generalisation or failure modes may require mechanistic analysis to examine whether they stem from identifiable structures. Concretely, mechanistic methods are most needed for: identifying intents hidden from outputs, such as deception or side quests; understanding the internal mechanism of a model behaviour; verifying the absence of a capability (behavioural tests can only confirm presence); or providing safety guarantees that must hold beyond the evaluated input distribution. However, the high granularity of mechanistic analysis comes at high practical cost, including not only compute but also researcher time, experimental complexity, and slower iteration. These costs position mechanistic methods as a complement to lighter-weight approaches, such as behavioural evaluations, rather than a full replacement.
Limitations of Mechanistic Analysis
Mechanistic interpretability aims to provide causal explanations of model behaviour, but it faces important practical and conceptual limitations. First, mechanistic explanations are often incomplete. An analysis may identify an internal computation (e.g., “This MLP retrieves factual knowledge about Paris”, as identified through causal tracing (Meng et al., 2022) ) that captures a non-trivial causal pathway without identifying all others contributing to the behaviour. This concern makes it hard to tell whether the identified computation is sufficient or just one of several mechanisms jointly implementing the behaviour.
Second, mechanistic analyses can scale poorly. Analysing large models and complex behaviours takes much more time and resources because the relevant computations may be distributed across model components and complex behaviours involve interactions across long generation trajectories that are hard to isolate. For example, most current circuit-discovery methods (Ameisen et al., [11]; Conmy et al., 2023; Hanna et al., 2024; Marks et al., 2025a; Wu & Barez, 2026) identify the computational graph responsible for generating a specific token. It remains challenging to identify and summarise the computations underlying behaviours that unfold across multiple generated tokens.
Third, even when we identify and causally validate an internal mechanism, this does not automatically mean we can reliably control the corresponding behaviour (Hase et al., 2023). The effectiveness, specificity, and generalisability of localisation and intervention techniques, such as identifying a model component and then patching its activations or changing its weight matrices, depend on several factors, such as the hydra effect (McGrath et al., 2023), beyond how accurately the identified localisation captures the underlying computation for the behaviour. Furthermore, an intervention that successfully steers one model’s behaviour on a task may not transfer to another model or setting.
These are open challenges but not fundamental barriers. Active research in areas such as establishing more effective representational units, discovering more robust model properties, developing more efficient circuit-discovery methods, and designing more reliable intervention approaches across distributions is working to address them. Nevertheless, these studies highlight a clear gap between mechanistic explanation and reliable control or safety guarantees.
Limitations of Mechanistic Analysis
Mechanistic explanations may be incomplete or non-causal, difficult to scale to large models or complex behaviours, and sensitive to modelling assumptions. They often fail to capture the full picture of a model’s underlying mechanism and do not automatically provide reliable, generalisable control. Intervening on an identified mechanism may work in one setting but fail to transfer to another. However, these limitations do not imply that mechanistic understanding cannot enable reliable control. Instead, they indicate that moving from explanation to robust, generalisable intervention requires additional studies and remains an open challenge in the field (Sharkey et al., 2025).
Summary. This section discussed different interpretability methods based on the evidence they provide. Behavioural methods measure and analyse model responses to inputs; concept-based methods explain predictions through human-interpretable concepts; input attribution methods identify which parts of the input influence an output; probing tests what information is decodable from representations; mechanistic methods identify and describe internal mechanisms that produce a behaviour. Among them, mechanistic analysis offers the strongest claims, but it is costly, often incomplete, and does not guarantee reliable control. In the next section, we introduce the core concepts on which mechanistic analysis is built.
Foundational Concepts for Mechanistic Interpretability
This section introduces foundational concepts in mechanistic interpretability. We establish a shared notation and vocabulary for use throughout the survey. We avoid committing to specific hypotheses about the representations and workings of model activations.
Behavioural vs. Mechanistic Understanding
Behavioural analysis describes what a model does, but safety-critical applications also require understanding its internal mechanism (as we reviewed in § 2).
Behavioural Interpretability
Behavioural interpretability studies a model through input–output pairs for , where is a finite test set. Behavioural interpretability treats the model as a black box.
Mechanistic Interpretability
Mechanistic interpretability studies a model by examining its internal computations, including model activations and parameters that produce them, at different levels of granularity in relation to its behaviour. In short, mechanistic interpretability characterises how a model’s behaviour or output is computed.
Neuron Polysemanticity in Model Representations
What Is a Concept? Concepts play a central role in many interpretability methods and will be used throughout this paper to define key ideas such as features and polysemanticity. It is therefore worth clarifying what we mean by the term. Notably, concepts do not necessarily correspond to categories humans can readily name or understand; neural networks may represent them in ways humans lack the vocabulary to describe (Hewitt et al., 2025).
Concept. A concept is a distinct, reusable unit of information or knowledge that can be identified across different instances or contexts. Concepts may correspond to familiar human categories, such as “a dog”, “the colour red”, or “words of praise”, but they need not admit a natural human description.
Cross-context identifiability concerns the meaning of the information unit rather than the process by which the information is recognised. Determining whether a concept is present in an information unit may itself require additional contextual information. For example, the word (i.e., an instance of an information unit) “bank” can refer to a financial institution or a river bank, depending on the surrounding sentence. Nevertheless, once identified, each concept can be recognised as the same information across different contexts. For example, the concept of “financial institution” is expected to remain sufficiently stable across the contexts in which it appears.
This definition accommodates both human-interpretable and machine-specific concepts. A machine-specific concept may lack a natural human description yet still form a coherent, reusable unit of information in the neural network ((Dalvi et al., 2022; Hewitt et al., 2025)). Human interpretability should therefore be treated as a property of some concepts rather than as a necessary condition for the information unit to qualify as a concept. Notably, most interpretability research focuses on concepts that humans can readily understand. Identifying and characterising concepts that matter to a model’s computation but do not map neatly onto human-understandable categories remains underexplored.
Neuron Polysemanticity. For a single neuron in layer , consider its activation at token position for some input prompt , denoted by . We can straightforwardly interpret a neuron if it responds to a single concept. For example, if is large and positive whenever position corresponds to the word “blue”, remains close to zero otherwise, and can causally shift the model’s outputs towards or away from blue-related concepts when manipulated, we can conclude that the neuron corresponds to the concept of blue. Assuming some neurons exhibit such a property, early work proposed methods to discover and interpret individual neurons’ functions ((Bau et al., 2017; 2019; Dalvi et al., 2019; Bau et al., 2020; Hernandez et al., 2022; Oikarinen & Weng, 2023; Wu et al., 2024)). However, follow-up studies have found that, due to representational constraints and superposition ((Elhage et al., 2022a)) (detailed in § 5.2), individual neurons often respond strongly to multiple concepts that appear unrelated from a human perspective ((Elhage et al., 2022a)). This phenomenon is known as neuron polysemanticity.
Neuron Polysemanticity. A neuron is polysemantic if there exist two distinct concepts and , together with corresponding sets of inputs and as well as token positions of interest and , such that and are both large, and interventions on these activations causally influence model behaviour associated with both concepts.
Polysemanticity complicates neuron-level explanations and suggests that human-interpretable semantic content may not be aligned with individual neurons. This motivates searching for representational units beyond individual neurons.
Features as Representational Units
Since individual neurons are often polysemantic, mechanistic interpretability has increasingly shifted towards alternative units of analysis. For example, it has been found that models sometimes encode human-interpretable concepts as linear directions or low-dimensional manifolds in the activation space ([20] (Wurgaft et al. 2026)). We thus use the term feature broadly to encompass any model representation corresponding to a human-interpretable concept. Such distributed representations motivate analysing the activation space as a whole, rather than individual neurons in isolation ((Elhage et al., 2022a; Bricken et al., 2023; Templeton et al., 2024)).
Specifically, we treat a feature technically as a geometric structure in the activation space of a model component, such as the residual stream, an attention-head output, or an MLP output, where denotes the hidden dimension. Conceptually, a feature is a manipulable manifestation of a concept of interest. As with neurons, for a feature claimed to correspond to a concept , we expect interventions that increase or suppress the activation of to causally influence model behaviour associated with . In this sense, features provide a more expressive, general representational unit than individual neurons for describing how neural networks may encode concepts.
Feature
A feature associated with a concept is a geometric structure in the model’s activation space whose activation is predictive of, and causally relevant to, the presence or expression of .
At this stage, we remain agnostic about the precise mathematical form features take and how the model encodes them. Instead, we treat existing methods for identifying and manipulating features as operational approximations shaped by their methodological assumptions and constraints. We examine these representational hypotheses explicitly in § 5.
Within this operational perspective, researchers can identify features using several methodological families, including readout-based methods (Ghandeharioun et al., 2024; Pan et al., 2026; Karvonen et al., 2026a) and sparse decomposition techniques (Cunningham et al., 2024; Dunefsky et al., 2024), and then manipulate them to test causal relevance (Wu et al., 2025b; Mueller et al., 2025). In practice, many such approaches operationalise features as approximately linear or linearised components of internal representations [20] (Yun et al. 2021; Marks & Tegmark 2024). Examples include linear probes and dictionary-learning methods (Belinkov, 2022; Bricken et al., 2023).
An important caveat is that, for a given concept, an identified manipulable geometric structure—that is, a candidate feature—may be only one of many possible internal representations associated with that concept. For example, a language model may encode the concept of “refusal” in a complex manner across multiple layers and components. Successfully identifying a linear direction in the residual stream that can steer the model towards or away from refusal [20] does not imply that this direction uniquely or exhaustively captures how refusal is represented. Other causally relevant structures (e.g., low-dimensional manifolds) for refusal may also exist but remain unidentified.
Transformer Computation
Transformers (Vaswani et al., 2017) have become the dominant architecture underlying modern language-model-based AI assistants such as ChatGPT and Claude, and are consequently the primary subject of study in mechanistic interpretability. Several properties of the transformer architecture, most notably its modular structure and additive information flow, make it particularly amenable to mechanistic analysis (Elhage et al., 2021). In particular, transformer components can be viewed as reading from and writing to a shared residual stream, allowing information to persist and be incrementally transformed as it passes through successive layers (Elhage et al., 2021). Below, we briefly review the main classical components of the transformer architecture.
Residual Stream and Computational Components
Let
denote the sequence of residual-stream activations at layer for an input sequence , where is the sequence length and is the model hidden size. The residual stream serves as the shared communication channel through which transformer components exchange information. Below, in this box, we let to simplify the notation.
For a standard transformer layer , consisting of a multi-head attention module and a multi-layer perceptron , we have
Unrolling the recursion gives
where denotes the input embedding representation. The equation shows that the residual stream at layer equals the input embeddings plus the accumulated outputs of all computational components that have written to the residual stream up to that point. This perspective is central to mechanistic interpretability, as it lets us decompose model computation into additive contributions from individual components (Elhage et al., 2021).
For simplicity, we omit layer-normalisation operations from the notation. Following Elhage et al. (2021), most of the layer-normalisation transformation can be absorbed into adjacent linear components. Specifically, for a fixed input, layer normalisation is well approximated as an affine transformation with a scalar rescaling factor. The affine part can be folded into neighbouring parameters, and the remaining scalar normalisation term simply reweights component contributions. This approximation lets us treat the residual stream as an additive communication channel without explicitly representing normalisation at every layer.
Attention heads and MLPs play distinct roles in transformer computation. Attention modules primarily route, aggregate, and compose information across token positions (Olsson et al., 2022; Wang et al., 2023), whereas MLPs often perform feature transformation and factual knowledge retrieval (Geva et al., 2021; Meng et al., 2022). This functional separation enables more localised analyses of how the model routes, transforms, and stores information, as well as how attention modules and MLPs interact (Neo et al., 2024).
Below, we detail the computations of canonical attention modules and MLPs.
Attention Module
A multi-head attention module consists of attention heads operating in parallel. Consider the residual-stream activations and let for notational simplicity. For each head , the residual stream is first projected into query, key, and value representations:
where and denotes the dimensionality of each attention head. The attention pattern of head is then computed as
where the softmax is applied row-wise. Thus, denotes the attention weight assigned by the query at position to the key at position . The head then writes an update to the residual stream:
where . The output of the full multi-head attention module is the sum of the contributions from all heads:
MLP Block
An MLP block applies a position-wise nonlinear transformation to each position of the residual stream. In the canonical transformer architecture Vaswani et al. (2017), this transformation takes the form
where , , and denotes a nonlinear activation function, for which the original transformer uses ReLU Nair & Hinton (2010). The same transformation is applied independently at each position in the sequence. MLP blocks primarily transform and generate internal features, writing new directions into the residual stream.
Notably, typically projects the activations into a much higher-dimensional space (), which has recently been found to yield more interpretable representations natively in the model [22] (Nikankin et al. 2025).
The transformer components discussed above correspond to their canonical formulations. Numerous variants have since been proposed and adopted in modern architectures. For example, variants of the attention module include Grouped Query Attention (GQA) [8], used in Llama 3 (Grattafiori et al., 2024), and Multi-Head Latent Attention (MLA) (DeepSeek-AI et al., 2024), used in DeepSeek-V2 (DeepSeek-AI et al., 2024) and DeepSeek-V3 (DeepSeek-AI, 2025). Variants of the MLP block include gated architectures such as SwiGLU (Shazeer, 2020), used in Llama 3, and Mixture-of-Experts (MoE) architectures, used in DeepSeek-V3.
Circuits as Descriptions of Internal Mechanisms
Sections 4.2–4.3 introduce neurons and features as representational units for describing what is represented in model activations, and Section 4.4 introduces the computations that transform and route activations through the model. We use the term mechanism to refer to the organisation of model activations, their representational units, or the transformations applied to them that together produce a particular model behaviour. A central topic in mechanistic interpretability is identifying and understanding such mechanisms in neural networks.
A common way to represent and describe such a mechanism is through a circuit: a subgraph of the model’s computational graph whose nodes correspond to computational components or representational units and whose edges represent information flow between them (Elhage et al., 2021).
Circuit (Figure 3)
Figure 3. Circuit. This diagram illustrates a circuit within a transformer, where specific nodes (MLP layers and attention heads in this case) perform specialised computations.
For a fixed model computation, let denote the corresponding directed acyclic computational graph. A circuit is a subgraph of , where:
Nodes: computational components, such as and , or representational units, such as features ;
Edges: directed connections representing information flow or transformation between nodes.
The subgraph represents a collection of components and interactions that jointly contribute to a particular computation or model behaviour, and can therefore serve as a description of the underlying mechanism.
Examples of Circuits
Induction head circuit: Two attention heads cooperate to implement in-context learning (Olsson et al., 2022).
QK circuits: Query–Key interactions prioritising specific token positions (Kamath et al., 2025).
The study of circuits has developed along two complementary lines. The first line of work (Wang et al., 2023; Conmy et al., 2023; Hanna et al., 2024; Haklay et al., 2025), referred to as capability circuit discovery by Wu & Barez (2026), aims to identify minimal sets of architectural components, such as attention heads or MLP layers, that are sufficient to reproduce a given behaviour or capability of the model, such as indirect object identification (Wang et al., 2023), numerical comparison (Hanna et al., 2023), and in-context learning (Kharlapenko et al., 2025). In this setting, the capability of interest is defined as a dataset of queries that exercise it, and a discovered circuit is considered successful if it recovers the model’s performance on this dataset.
The second line of work, referred to as query-level circuit discovery (Wu & Barez, 2026), focuses on how internal features interact and propagate across the computational graph to produce the model’s decision for a particular input query [11] (Wu & Barez 2026). Studies in this line usually use sparse autoencoders or cross-layer transcoders (detailed in § 6) to automatically decompose MLP outputs into interpretable units, trace attributions between these features, and track how information flow converges to the final prediction.
Summary. This section introduced the foundational concepts of mechanistic interpretability: neuron polysemanticity, which motivates the use of features as more expressive units of analysis; transformer computation, which describes how representations are transformed and information is routed through the model; and internal mechanisms, which organise these units, transformations, and information flows to produce model behaviour and can be described by circuits. How we interpret and leverage these concepts depends on assumptions about the structure of model representations and computation, which we examine explicitly in § 5.
Core Hypotheses in Mechanistic Interpretability
Mechanistic interpretability rests on several core hypotheses about how neural networks represent information and implement computation. These hypotheses are not yet rigorously proven. Nonetheless, they have shaped methodological choices and empirical investigations. In this section, we formalise four of them, discuss their conceptual basis, and summarise supporting and opposing empirical evidence.
The Linear Representation Hypothesis
We have seen that single neurons may not be the most effective unit of representation in many cases, but what is? The linear representation hypothesis claims that linear directions in the activation space are effective fundamental units of representation (Park et al., 2024).
Linear Representation Hypothesis
A concept can be effectively represented by a feature corresponding to a linear direction in activation space . Linearity implies that adding or subtracting this direction, or a scalar multiple of it, from model activations can correspondingly influence model behaviour, while the projection approximately measures the strength of the feature in model activations for an input .
Under a strong version of the linear representation hypothesis, all concepts are best represented by linear features. In this view, model activations can be approximately decomposed as an additive combination of such features,
where denotes the strength of feature dependent on the input . Under a weaker version, only some concepts admit such linear representations (Sharkey et al., 2025; Smith, 2024).
Conceptual Basis. A feature associated with a concept is linearly represented if its presence corresponds to a fixed direction in the activation space and can be detected through the projection . This linearity makes mechanistic analysis tractable because we can use simple methods to identify, measure, and manipulate these features in the activation space, such as scaling them and adding them back to model activations.
Notably, activation spaces in different model components can have distinct geometric structures. For example, in the hidden states of MLP blocks, the neuron axes are computationally distinguished because element-wise nonlinear activation functions, such as ReLU, act separately on each axis. The coordinate system is therefore meaningful and is referred to as a privileged basis (Figure 4, left) (Elhage et al., 2022a). In other spaces, such as the residual stream, no operation singles out the neuron axes, so linear features may lie along arbitrary directions. Such spaces are said to have a non-privileged basis (Figure 4, right).
Figure 4. Privileged versus non-privileged bases. In the hidden layers of MLP blocks, element-wise nonlinear operations break rotational symmetry and make individual axes meaningful. In contrast, the residual stream contains no such operations that privilege particular axes, so rotating the basis preserves the relevant computation.
When the basis is privileged, a single neuron can serve as a scalar feature that meaningfully represents a concept. When it is not, the feature may instead correspond to directions or other geometric structures that do not align with any individual neuron. In this case, specialised methods are required to identify them from the activation space.
Empirical Evidence and Counterexamples. Empirical support for the weaker version of the linear representation hypothesis includes the success of linear probes across diverse models and concepts (Belinkov, 2022), activation steering along single directions to induce targeted behavioural changes (Turner et al., 2024), and sparse autoencoders recovering linearly accessible and steerable features at scale (Bricken et al., 2023). At the same time, there is substantial evidence that not all representations are well described as linear. Counterexamples include nonlinear feature interactions, algorithmic representations that rely on nonlinear transformations, and features distributed across multiple interacting subspaces (Nanda et al., 2023; Engels et al., 2025; Tibshirani et al., 2026; Csordás et al., 2024; Feucht et al., 2026; Wurgraf et al., 2026). Some work has further argued that features may not always exist as explicitly represented objects, but may instead emerge from underlying circuit dynamics (Smith, 2024). The linear representation hypothesis is therefore best viewed as a useful approximation that holds in some settings, rather than as a universal law.
The Superposition Hypothesis
The superposition hypothesis says that models encode more features than their hidden representations have dimensions.
Superposition Hypothesis. Model hidden states encode more features than they have dimensions. If there are features and dimensions, then the superposition hypothesis claims that .
This implies that features cannot be represented as orthogonal directions—though they can be represented as nearly orthogonal directions. It also implies neuron polysemanticity: the same neuron participates in the encoding of multiple unrelated concepts.
Conceptual Basis. Superposition is a geometric property of learned representations in which a limited number of neuron dimensions encode many more features by allowing those features to share representational dimensions. Neuron polysemanticity, where individual neurons respond to multiple seemingly unrelated semantic concepts, naturally follows from this phenomenon. When the number of encoded features exceeds the activation-space dimensionality, the model can no longer allocate a dedicated neuron dimension to each feature. Instead, multiple features become superposed within the same activation space, shifting the representation from dedicated dimensions with monosemantic neurons to shared dimensions with increasingly polysemantic neurons (Figure 5).
Figure 5. Superposition implies polysemanticity. Left: when the number of features does not exceed the number of dimensions , i.e., , feature directions can align cleanly with individual neuron axes, producing monosemantic neurons. Right: when , multiple feature directions must share the same limited set of neuron axes; as a result, each neuron axis cuts across multiple feature directions, producing polysemantic neurons.
Empirical Evidence. Mechanistic studies have repeatedly observed widespread neuron polysemanticity: individual neurons often activate for multiple seemingly unrelated semantic concepts, making them difficult to interpret as standalone representational units ((Olah et al., 2020; Elhage et al., 2022a; Bricken et al., 2023)). Dictionary-learning methods such as sparse autoencoders exploit this structure to recover millions of distinct features from shared activation spaces ((Bricken et al., 2023; Templeton et al., 2024)). Many discovered features are substantially more monosemantic than individual neurons, and some have been shown to causally influence model behaviour. Together, these findings suggest that the activation space may encode more meaningful semantic information along activation-space directions than along individual neuron axes.
However, the superposition hypothesis remains a hypothesis, not an established fact, in large language models. Observing polysemantic neurons does not, by itself, imply that superposition is the underlying cause, and dictionary-learning methods can recover interpretable features even in the absence of superposition. Furthermore, Lecomte et al. (2024) showed that polysemanticity can arise even when the number of underlying features does not exceed the number of available neurons, suggesting alternative—and potentially complementary—mechanisms for the emergence of polysemantic representations.
Architectural Responses to Superposition. Superposition is not an intrinsically inevitable property of deep neural networks but a consequence of architectural permissiveness. Without constraints on model architectures or activations, gradient descent can exploit the activation-space geometry to pack far more features than neurons ((Elhage et al., 2022a)). This motivates a distinct research direction, interpretability-by-design (as mentioned in § 3.2 and discussed in § 6.10). Interpretability-by-design argues that, rather than decomposing traditional models’ superposed activations post hoc, one can design novel architectures or components that encourage representations where features are more accessible for analysis. Two intervention points have been explored:
Sparsity constraints: Enforce sparsity in weights or activations that limits how many neurons within an MLP or attention head can participate for a given input, constraining how many units can jointly support any single feature ((Yang et al., 2025; Gao et al., 2026; Elhage et al., 2022b)).
Computational primitives: Change the fundamental units of analysis from static features in activations to learnable transformations by weight matrices that generate activations ((Liu et al., …)
2025b; Lindsey et al., 2025b). This paradigm better matches certain tasks and avoids the pressure to compress or suppress model representations.
These architectural interventions shift from treating superposition as a phenomenon to reverse-engineer to treating it as a design choice that can be constrained upstream. We discuss these methods in detail in § 6.10.
The Privileged Access Hypothesis
A class of approaches that use language models as interpreters of themselves can be justified through the privileged access hypothesis. The privileged access hypothesis states that:
Privileged Access Hypothesis. A language model used to interpret itself can, under appropriate elicitation, access internal variables and intermediate computations that are not reliably recoverable from external behavioural analysis alone, providing a comparative advantage as an interpreter of its own behaviour (Binder et al., 2025; Li et al., 2025a; Song et al., 2026).
Conceptual Basis. The ability to access and judge its own computation or reasoning—called introspective monitoring—has long been a desired property of an AI system (Cox & Raja, 2008). Even earlier rule-based medical expert systems sought to explain their decisions, such as MYCIN’s “Explanation system” (Shortliffe, 1977). In this sense, using language models to explain themselves can be seen as a modern continuation of these earlier efforts.
A major difference is that rule-based systems can explicitly present the rule they executed, whereas modern language models compute in a highly nonlinear way. Although researchers have proposed mechanistic tools aiming to reverse-engineer part of the computation, the technical complexity of these techniques hinders their accessibility and scalability. The privileged access hypothesis then posits that the language model itself naturally stores information about the computation within its internal states, such as latent beliefs or intermediate plans. The model can thus learn to verbalise this information directly, serving as a scalable complement to mechanistic analysis.
Empirical Evidence. The most direct empirical evidence for this hypothesis comes from recent findings that models, even without explicit training to do so, are sometimes aware of concept representations (acquired by methods like difference-in-means) injected into their activations (Lindsey, 2026; Fornasiere et al., 2026). Additional, though less direct, evidence comes from the design of explainer models—a general class of interpretability methods in which one model is fine-tuned to explain the target model’s activations (Pan et al., 2026; Karvonen et al., 2026a; Fraser-Taliente et al., 2026). In practice, such explainers are typically initialised from a copy of the target model itself (Pan et al., 2026; Karvonen et al., 2026a; Bauer et al., 2026; Huang et al., 2026). This usually works better than initialising from alternative models, suggesting the target model already encodes much of what is needed to interpret its own activations. In this sense, fine-tuning elicits and refines a latent capability rather than creating one from scratch.
Some studies also reported negative results for the privileged access hypothesis: Song et al. (2026) found that models predict their own sampling temperature no better than they predict other models’, suggesting apparent introspection may reflect general reasoning about model-like systems rather than self-access. Separately, recent work has questioned whether internals-based explanations provide a practical advantage over behavioural baselines: Karvonen et al. (2026b) found that activation-reading tools (including activation oracles) give agents no uplift over transcript-only baselines at predicting counterfactual model behaviours in the wild, and Li et al. (2026c) showed that standard verbalisation benchmarks can be solved without access to target-model internals at all (see also Jakkli et al. (2026)).
In summary, these results show that privileged access is possible for certain tasks under favourable conditions, not universal or reliable. The hypothesis should therefore be treated with caution, as significant failure modes and limitations remain.
The Universality and Platonic Representation Hypothesis
Platonic Representation Hypothesis. Models independently trained on similar tasks tend to develop similar internal representations or algorithms, which often approximate a task-determined canonical form (Huh et al., 2024; Olah et al., 2020). The degree of convergence depends on factors such as model size, architecture, and training data.
Conceptual Basis. A key distinction is the level at which models are expected to converge. Weak universality refers to convergence in the algorithms or recurring computational patterns that models implement, even when these are encoded through different internal representations. Strong universality makes the stronger claim that the internal representations themselves converge towards a shared canonical form.
Empirical Evidence and Trade-Offs. Weak universality is supported by recurring attention patterns, features, and circuits across models, tasks, and scales (Nanda et al., 2023; Lan et al., 2024; Olah et al., 2020; Quirke et al., 2026). Strong universality is less supported and is usually demonstrated through representational similarity metrics (Kornblith et al., 2019; Moschella et al., 2023; Lan et al., 2025), where recent studies have found the results to be fragile (Gröger et al., 2026; Koepke et al., 2026). Divergent representations can arise from variations in architecture, data, or optimisation trajectories (Harrasse et al., 2025; Morcos et al., 2018).
Implications. If strong universality holds, fine-grained mechanistic insights and the methods derived from them may transfer more readily across models. If only weak universality holds, such insights may remain contingent on factors such as model family and task (Bereska & Gavves, 2024), and even a complete mechanistic understanding of one model (see § 14.2) may fail to generalise to others. Therefore, it is important to clarify the extent of universality for assessing the promise, limitations, and scalability of different interpretability research orientations, including the six major orientations introduced in § 14.
Summary. This section discussed four hypotheses that motivate many modern mechanistic interpretability studies. Linear representations support linear-feature-based analysis; superposition explains why we need to operate on linear or more complex forms of features instead of single neurons; privileged access explains the fundamental idea of model self-interpretation; universality asks whether insights transfer across models and at which levels. Each hypothesis holds in some settings and fails in others, and techniques built on them inherit those limitations. § 6 introduces the main family of techniques and methods that operationalise the first two hypotheses (linear representation and superposition): dictionary learning for feature extraction and analysis.
Feature Extraction and Analysis
The critical notions introduced in § 4, together with the hypotheses about the structure of neural network representations discussed in § 5, provide the conceptual foundation for developing methods to identify and analyse manipulable internal objects associated with concepts of interest, i.e., features. One straightforward paradigm is to decompose model activations into sparse linear features. Sparse autoencoders (SAEs) provide a principled example of this approach, learning to represent any input activation with a subset of a large dictionary of monosemantic, interpretable features (Cunningham et al., 2024).
This section first introduces the canonical sparse coding problem and links it to SAEs, reviews how SAEs extract interpretable features, and then provides a practical overview and comparison of major SAE variants. We then introduce methods for multi-layer feature extraction and analysis, such as transcoders and crosscoders, and compare them with SAEs as well as with a broader range of alternative feature-extraction methods.
The Sparse Coding Problem
Early neuroscience studies posited that brains represent information using sparse, interpretable basis signals ((Barlow, 1961; Field, 1994). This hypothesis further motivated research to decompose natural images and compare the decomposed signals with cell activities in the primary visual cortex, finding surprising alignment ((Olshausen & Field, 1996; 1997). At the same time, the signal processing community has also examined representing a given signal as the sum of a subset of basis signals from a given overcomplete dictionary ((Mallat & Zhang, 1993; Chen et al., 1998). Studies on learning better overcomplete dictionaries for high-quality representations also thrived ([7] (Lewicki & Sejnowski 2000). Together, they framed the sparse dictionary learning research question, with SAEs serving as a modern algorithm that uses neural network activations as the signal. Specifically, given data , a sparse dictionary learning method finds the dictionary and sparse codes such that
SAEs implement this with neural networks: are model activations, is an overcomplete dictionary to learn, and the sparse codes are . Note that conventional dictionary methods usually run the algorithm to find a for each , which does not scale to modern language models. In contrast, SAEs use a nonlinear encoder to predict the coefficients for different input activations.
Sparse Autoencoders
SAEs (Figure 6) extract monosemantic features from polysemantic model activations. They are trained to reconstruct activations at a specific location in neural networks, such as the output of an MLP block or an entire transformer layer (i.e., the residual stream), while keeping only a few features active in any given forward pass.
Figure 6. Sparse Autoencoder (SAE). SAEs are trained to reconstruct activations of a model component at a specific layer , , into an overcomplete, sparse latent space , such that the reconstructed activation . By enforcing sparsity on , the model decomposes dense, polysemantic representations into monosemantic, human-interpretable features.
Sparse Autoencoder (Figure 6)
A sparse autoencoder learns a component-specific encoder–decoder pair
such that activations of a particular model component at a given layer, (e.g., the output of the layer- MLP block), are reconstructed as
while encouraging the feature activations to be sparse. Usually, , where is an integer expansion factor.
Canonical Formulation. Given activations of a model component , where is the distribution of that component’s activations over a large corpus, the canonical SAE objective is
where controls sparsity ((Cunningham et al., 2024). Canonical SAEs use regularisation instead of regularisation. This is because, for each feature, ’s derivative is almost everywhere zero and thus does not provide useful signals for promoting feature sparsity during gradient descent. Additionally, the decoder columns are typically constrained to unit norm. This is to prevent the case where the SAE learns to reduce its loss by generating smaller values of —not exactly 0 so that many features can be active, each having a small coefficient—and enlarging the norm of each column of the decoder to compensate. In this way, the SAE output remains the same, while loss can be substantially reduced, without any real gain in sparsity.
Why SAEs Work. SAEs are motivated by the hypothesis that a model’s activations can be approximated as linear combinations of many latent feature directions, which can be regarded as an extension of the linear representation hypothesis discussed in § 5.1. Sparsity means that only a small number of features are active for any given token or input, but not that the hidden state itself is sparse. Early sparse dictionary learning studies [7] (Olshausen & Field 1996; Lewicki & Sejnowski 2000) give us fair confidence that, if the hypothesis holds, dictionary learning can recover the structure as an overcomplete dictionary. These findings motivate using SAEs as a modern adaptation to extract monosemantic directions from the activations of neural networks (Cunningham et al., 2024; Bricken et al., 2023).
From Monosemanticity to Concept Identification. For each input activation, the raw SAE outputs are just a small set of vectors. Practical interpretability requires identifying the semantic concepts these vectors correspond to in natural language (so that they become human-understandable features). Empirically, SAE features selectively activate on structured token patterns that recur across input sequences, suggesting sensitivity to specific abstract semantic, functional, or lexical concepts (Marks et al., 2024b). Automated analysis pipelines (Gur-Arieh et al., 2025a; Paulo et al., 2025b) aggregate highly activating inputs to infer the concept each feature captures (see § 14.7 for a detailed discussion).
Training Dynamics and Evaluation. SAE training is sensitive to initialisation, sparsity strength, and decoder normalisation. Without proper constraints and hyperparameters, the features may collapse, duplicate, or absorb one another. SAEs are typically evaluated for reconstruction loss (fidelity of activation reconstruction), feature absorption (how often a general feature fails to fire because a more specific feature has absorbed its direction), and downstream test loss recovery (whether replacing activations with reconstructions preserves model behaviour). Together, these metrics assess both the quality of the decomposition and its mechanistic relevance to the original model’s computation.
SAE Variants and Architectural Improvements
There are several variants of the standard SAE. One key reason to use alternatives is that standard SAE training penalises activation magnitude, which can lead to low activation even when a feature is clearly present (the shrinkage effect). Different optimisation frameworks also have different tunable parameters. The choice depends on the analysis goal, target layer type, model size, and budget constraints.
Gated SAEs
Gated SAEs introduce multiplicative gates to decouple feature selection from magnitude:
where denotes activations at layer , and are encoder parameters, and are the gate parameters, and denotes element-wise product. This mitigates shrinkage effects induced by the penalty (Rajamanoharan et al., 2024a).
JumpReLU SAEs
JumpReLU SAEs (Rajamanoharan et al., 2024b) learn per-feature activation thresholds. Given the encoder pre-activation , let denote its -th entry. The activation value of the -th feature after applying JumpReLU is
where is a learned threshold parameter. Straight-through estimators (Bengio et al., 2013) are used during training to enable gradient flow.
Figure 18 (PDF p. 34).
Top- SAEs
Top- SAEs (Gao et al., 2025) impose an explicit hard sparsity constraint. Let denote encoder pre-activations. Top- SAEs define the -th feature’s activation as
where indexes the largest entries of . This formulation explicitly enforces —a more direct feature-sparsity constraint than regularisation. Notably, as it does not suppress activation magnitudes, it avoids the feature-shrinkage issue observed in -based SAE training (Gao et al., 2025).
Batch Top- SAEs
Batch Top- SAEs relax the fixed per-example sparsity constraint of standard Top- SAEs by enforcing the sparsity budget across an entire batch instead (Bussmann et al., 2024). For a batch of encoder activations
where is the batch size, Batch Top- retains the largest activation values in the batch. Specifically, let
In each batch, the activation value of the -th feature in the -th sample is
That is, each batch exactly retains features, while individual samples may have more or fewer active features than . This loose constraint allows the model to allocate representational capacity adaptively across samples. During inference, Batch Top- SAEs instead use the JumpReLU function with an inferred global threshold to determine which features are active for each input, so that an input’s activations do not depend on the rest of the batch.
Trade-offs Between Variants
Different SAE variants may implement distinct sparsity constraints. No single variant is optimal across all scenarios.
| Variant | Sparsity | Differentiable | Tuning Required |
| -regularised | Soft | ✓ | coefficient |
| Gated | Learned | coefficient | |
| JumpReLU | Learned | coefficient, STE bandwidth, threshold init. | |
| Top- | Hard | value | |
| Batch Top- | Hard | value |
Table 16.
Uses straight-through estimators (STEs) during training. Trains the non-differentiable gate with auxiliary losses.
Readers are referred to Shu et al. (2025) for a dedicated survey of SAEs, their variants, evaluations, and applications. Additionally, researchers have published SAE weights for several language models, including Gemma Scope (Lieberum et al., 2024) for Gemma 2 (Team et al., 2024b) and Llama Scope (He et al., 2024) for Llama-3.1-8B (Grattafiori et al., 2024). They can be accessed through open platforms such as Neuronpedia (Lin, 2023).
Empirical Comparisons of SAE Variants
The SAE variants in § 6.3 differ principally in how they enforce sparsity, and those differences map onto distinct analysis settings.
Hard-constraint formulations (e.g., Top-) directly enforce sparsity and suit tight compute budgets. Learned-threshold (e.g., JumpReLU SAE) and gated formulations usually achieve better reconstruction results and suit generous tuning budgets. Cross-layer variants (discussed in § 6.6) analyse computations across layers that a single dictionary cannot represent at all. This subsection discusses the setting in which each variant is used and the evidence it provides. We particularly discuss four factors that account for most of the variation:
Analysis goal: exploratory discovery vs. targeted investigation vs. deployment/monitoring
Layer type: residual stream vs. MLP output vs. attention output
Model scale: small (B) vs. medium (–B) vs. large (B parameters)
Resource constraints: compute budget, hyperparameter tuning capacity, tolerance for training instability
Analysis Goal. Exploratory discovery, where the question is what features exist rather than whether a particular one does, is the setting SAEs were designed for and the one the comparisons below address. For targeted investigation of an already-identified behaviour, where the concepts of interest are determined, supervised methods such as DAS (Geiger et al., 2024) and linear probes are more direct instruments. For online deployment and monitoring, SAEs with small dictionaries can be preferable for low inference cost.
Layer Type. Layer types differ in their activation properties, so the right variant depends on what each can represent, not just how well. On the residual stream, single-layer SAEs apply directly, and the choice reduces to the resource considerations below. On MLP outputs, transcoders are the appropriate tool: a single-layer SAE cannot span the nonlinearity, so transcoders learn the input-to-output map directly (Dunefsky et al., 2024), and Paulo et al. (2025c) reported favourable comparisons against SAEs there. On attention outputs, standard and Gated SAEs both have been used; attention-specific variants remain underdeveloped, making this the least studied of the three sites. For cross-layer analysis, crosscoders (Lindsey et al., 2024) and cross-layer transcoders (Ameisen et al., [11]) track feature evolution across layers at substantially higher compute cost, and a crosscoder trained jointly on two models is the reported approach for comparing representations between them. Table 3 summarises the variants used at each layer type, the empirical rationale for their use, and the main caveats reported in prior work.
| Layer Type | Variants Used | Reported Basis | Reported Caveats |
| Residual stream (early layers) | Gated SAE, Top- | Activations are closer to input embeddings; moderate sparsity | Smaller expansion factors (–) reported |
| Residual stream (middle layers) | Gated SAE, JumpReLU | Richest feature structure; highest interpretability potential | Standard expansion factor (–); most studied regime |
| Residual stream (late layers) | Top-, Gated SAE | Activations increasingly task-specific; may be less sparse | Larger expansion factors (–) reported |
| MLP outputs | Transcoder (§ 6.6) | MLP applies a nonlinearity a single-layer SAE cannot span | Not a sparse decomposition on a static object; bridges input to output |
| Attention outputs | Gated SAE, standard SAE | Attention outputs often lower-rank | Least studied of the three sites; may benefit from smaller dictionaries |
| Cross-layer | Crosscoder, CLT (§ 6.6) | Tracks feature evolution; resolves cross-layer superposition | High compute cost; requires multi-layer training |
Table 3. Heuristic recommendations for selecting SAE variants across different layer types. Recommendations are based on current empirical evidence and practical experience, and may evolve as SAE methodologies and evaluation standards continue to develop.
Resource Constraints. Under limited compute, or where fast iteration matters more than reconstruction quality, Top- SAEs (Gao et al., 2025) are the practical choice: fixing removes the sparsity coefficient from the search space entirely rather than requiring it to be tuned, with reported values commonly in the range (Gao et al., 2025) to 150 (Karvonen et al., 2025). With moderate tuning capacity, Gated SAEs (Rajamanoharan et al., 2024a) achieved a better reconstruction–sparsity trade-off while retaining . They also avoid the shrinkage effect by separating feature detection from magnitude estimation. However, because the standard SAE is now used mainly as a baseline, comparisons against it partly reflect shrinkage rather than dictionary quality alone. When extensive tuning is plausible, JumpReLU SAEs (Rajamanoharan et al., 2024b) achieved the strongest reconstruction performance at matched sparsity by learning per-feature thresholds. However, gains on these sparsity–fidelity proxy metrics do not reliably translate into better downstream performance (Karvonen et al., 2025). In practice, selecting the best model and hyperparameters to achieve optimal performance can take a lot of time.
Model Scale. The differences between variants depend strongly on the metric in models with roughly 1 or 2B parameters, such as Gemma-2-2B (Karvonen et al., 2025), which is the regime mainly used to validate methodology before scaling. For models between 1B and 10B parameters, JumpReLU SAEs and Top- SAEs are the most commonly reported variants; Gemma Scope (Lieberum et al., 2024) uses the former and Llama Scope (He et al., 2024) the latter. For models larger than 10B parameters, Gemma Scope (Lieberum et al., 2024) (for Gemma 2 models) and Gemma Scope 2 (McDougall et al., 2025) (for Gemma 3 models) are the main open resources, both of which used JumpReLU SAEs. Additionally, Gao et al. (2025) explored scaling Top- SAEs to GPT-4 (OpenAI et al., 2024), although GPT-4 itself is not publicly available.
Table 4 summarises each variant’s main findings reported in previous work, along with their experimental settings. Note that these studies may use different baselines, model families, sparsity constraints, and evaluation criteria. The community has been establishing benchmarks to evaluate variants of SAEs, such as SAEBench (Karvonen et al., 2025), but the results vary by metric. As a result, we should treat the findings as references rather than established rules. The Setting column in Table 4 records the conditions under which each result is established. Table 5 further highlights several factors that limit the generality of the findings.
| Variant | Reported Finding | Setting |
| Standard | Sparse dictionary learning by regularisation recovers features more monosemantic than individual neurons; subject to shrinkage bias | Establishing work, small models (Cunningham et al., 2024; Bricken et al., 2023) |
| Gated | Separating detection from magnitude estimation addresses shrinkage at minimal added complexity | Reconstruction–sparsity improvement at matched sparsity, Gemma-family (Rajamanoharan et al., 2024a) |
| JumpReLU | Learned per-feature thresholds trained with straight-through estimators give the strongest reported reconstruction | Matched-sparsity comparison, Gemma-family (Rajamanoharan et al., 2024b); needs most tuning (Bussmann et al., 2024) |
| Top- | A hard constraint removes the sparsity coefficient from the search space | Scaling behaviour characterised on large models; – commonly reported (Gao et al., 2025; Karvonen et al., 2025) |
| Batch Top- | A batch-level sparsity budget lets the number of active features vary per input, improving reconstruction at matched average sparsity | Introduced as a relaxation of per-example Top-, achieving better reconstruction (Bussmann et al., 2024) |
| Transcoder | Learning the MLP input–output map avoids spanning the nonlinearity with a single dictionary | Reported to perform at least on par with SAEs (Dunefsky et al., 2024; Paulo et al., 2025c) |
| Crosscoder | Joint reconstruction across layers tracks feature evolution and resolves cross-layer superposition | Cross-layer analysis and model diffing; trained jointly across models for cross-model comparison (Lindsey et al., 2024) |
| CLT | Partial reconstruction of multiple downstream layers from one dictionary | Deployed within CLT-based circuit discovery at scale [11] |
Table 4. Reported evidence for each SAE variant, with the setting in which it was obtained.
| Condition | Description |
| Architecture dependence | Most evidence derives from Llama- and Gemma-family models. Generality of the findings across model families remains unestablished. |
| Task dependence | The properties that matter may differ across analysis goals, such as safety auditing, behaviour understanding, and knowledge localisation, but existing evaluations often do not distinguish between them. |
| Rapid iteration | New architectures and training procedures are proposed faster than existing ones are sufficiently studied and replicated. |
| Immature theoretical analysis | Current findings are mostly empirical and may be superseded. Understanding of why particular variants succeed or fail is incomplete. |
| Feature reproducibility | SAEs trained on the same data with different random seeds can recover substantially different features. When between-variant performance differences are smaller than within-variant seed-to-seed variability, we may not confidently attribute them to architectural factors. However, few reported comparisons have rigorously accounted for this concern. |
Table 5. Conditions for the reported evidence on SAE variant behaviour.
Challenges and Failure Modes of SAEs
SAEs have become a standard tool for feature extraction, but they exhibit systematic failure modes that undermine their interpretability claims. It is therefore essential to understand these issues when using SAEs as mechanistic tools. This section provides a systematic discussion of them.
Dead Features
A dead feature is an element in the learned dictionary that never activates or activates extremely rarely across training. Dead features waste the dictionary’s capacity, and they are common. In fact, they can constitute up to 10–50% of the dictionary in a poorly tuned SAE (Bricken et al., 2023).
Several factors can cause dead features to emerge. First, inappropriate SAE weight initialisation can place features in low-density regions of the activation space with weak gradient signals. Second, an overly strong sparsity penalty may suppress feature activations early in training, preventing features from contributing usefully to reconstruction. Finally, once a subset of features captures common, dominant patterns, the remaining features struggle to find useful roles for improving reconstruction and thus do not activate.
Dead features can be detected by tracking each feature’s rate of being activated during training. For example, features with an activation rate below 0.01% may be considered dead (the threshold should adapt across scenarios). Several mitigation strategies have been proposed, such as neuron resampling, which periodically reinitialises dead features using examples with high reconstruction error (Bricken et al., 2023). The sparsity penalty can also be reduced early in training and increased gradually (i.e., a warm-up schedule). More recent Batch Top- SAEs allocate sparsity adaptively across samples, which allows more active features on complex inputs and fewer on easy ones (Bussmann et al., 2024).
Feature Duplication
Feature duplication occurs when multiple features in the dictionary encode the same or similar concepts (Templleton et al., 2024), leading to potentially redundant features. Duplication can arise for several reasons: insufficient sparsity pressure, imbalanced training data, or an optimisation process that favours local minima with duplicated features. Duplication can be detected by computing pairwise cosine similarity between feature directions (i.e., decoder columns), where high similarity indicates potential redundancy. As duplicated features tend to activate together (Korznikov et al., 2025), co-activation statistics may provide complementary evidence.
Feature Absorption and Splitting
Another class of failure modes concerns feature granularity. Feature absorption arises when semantically hierarchical features interact under the sparsity objective: a seemingly general (parent) feature fails to activate on some inputs where it should because a more specific (child) feature activates instead and has absorbed part of the parent’s direction into its own decoder direction (Chanin et al., 2025). For example, a “starts with S” feature may fail to activate on the token “short”, while a token-specific “short” feature with a partially overlapping direction activates instead. Absorption-induced false negatives make such a feature an unreliable detector of its concept. It can be detected by comparing the feature with a linear probe for the same concept: inputs on which the probe responds strongly, but the feature does not, while another active feature’s direction aligns with the probe, indicate absorption (Chanin et al., 2025).
By contrast, feature splitting occurs when a feature learned by a smaller SAE splits into multiple distinct features as the SAE scales up (e.g., by increasing the expansion factor ). This often produces more fine-grained features, which may be either beneficial or less useful. In a favourable case, an initially polysemantic or opaque feature, such as one activating on “bank” in both financial and geographical contexts, may split into more coherent features of “investment bank” and “river bank”. In a less useful case, the resulting features may become low-level, lexical detectors that activate on the token “bank” in any context.
Feature Instability Across Seeds
SAE features are also sensitive to random initialisation, optimiser stochasticity, and data ordering. For a fixed SAE architecture and training dataset, different random seeds can produce substantially different dictionaries (Paulo & Belrose, 2026). In practice, frequent and semantically clear features tend to be stable, while rare or context-specific features vary substantially. Often, fewer than 50% of features remain the same across seeds, although broader semantic clusters may remain more consistent. This instability limits the reliability of mechanistic claims based on individual SAE features. Although aggregating results across seeds can improve robustness, it increases computational cost. As a result, when evaluating SAEs, we should report feature stability across seeds alongside reconstruction loss.
Identifiability and Theoretical Concerns
Another concern for SAE-based interpretability claims is whether the learned features are identifiable, that is, whether the dictionary learning reliably recovers the “true” underlying features—if such features exist—or merely finds one of many mediocre decompositions (Klindt et al., 2025).
Identifiability
Suppose activations are generated by some sparse latent process
where is the ground-truth feature dictionary and are the feature coefficients. An SAE is identifiable if its training recovers and up to unavoidable symmetries such as feature permutation and rescaling.
In classical dictionary learning, identifiability holds under low-coherence conditions, i.e., dictionary elements are sufficiently orthogonal to one another. However, when coherence is high, ambiguity arises: multiple distinct sparse combinations of features from the same dictionary can generate the same output (Schnass & Vandergheynst, 2008). In large language model activations, high coherence is likely unavoidable because of the model’s complicated internal structure and correlations within its feature space (Li et al., 2025b). So far, identifiability remains a stubborn issue in SAEs, even though it offers a theoretical explanation for several empirical observations.
Recent progress in fine-tuning language models around a fixed sparse autoencoder offers a potential path forward. In this setting, model activations are explicitly constrained to be sparse combinations of decoder features while largely preserving task performance (Chen et al., 2025a). In particular, recent work aimed at limiting superposition fine-tunes the language model jointly with the SAE while enforcing orthogonality constraints on the decoder matrix (Miller et al., 2026). By explicitly imposing low-coherence conditions, this approach improves identifiability: activations have a unique feature representation, and these features can be reliably recovered.
Relation to Causal Relevance. The concern of identifiability connects to causal relevance: even if an SAE feature is monosemantic and helps reconstruct activations well, it may not correspond to a causally meaningful unit of analysis. Below, we summarise some criteria for validating an SAE feature.
Diagnostic criteria:
Necessity: Does ablating the feature remove the corresponding concept in model responses? Low necessity suggests that the concept can be expressed through redundant features.
Sufficiency: Does activating only this feature produce the behaviour corresponding to its concept? Low sufficiency suggests the feature alone is not enough to represent the concept.
Specificity: Does the feature’s causal effect match its corresponding concept? Low specificity means the feature contains other concepts.
These criteria are useful in practice when claiming that SAE features represent meaningful units of analysis.
Practical Recommendations
The challenges discussed above suggest several methodological considerations for SAE-based interpretability studies:
Report failure-mode metrics: Alongside reconstruction loss, researchers should also report dead-feature percentage, duplication rate, and stability across seeds.
Validate causal relevance: Researchers should validate features’ necessity, sufficiency, and specificity through interventions (e.g., feature ablation or steering) and evaluation on downstream effects.
Compare methods: For safety-critical applications, researchers should compare SAE features against various baselines to assess whether SAEs recover the relevant structure.
Try multiple SAE variants: Different variants (e.g., Gated, Top-, JumpReLU) impose different inductive biases. Researchers can explore findings consistent across variants, as these provide stronger evidence.
Acknowledge identifiability limits: Researchers should frame SAE features as one valid approximate decomposition rather than the true features. Multiple decompositions may be equally valid for reconstruction but yield different feature sets.
Consider alternatives for targeted analysis: When specific concepts are the focus, supervised methods (e.g., DAS, probes, discussed in § 6.9.4) may be more reliable because SAEs discover features in an unsupervised and exploratory manner.
Multi-Layer Feature Extraction and Analysis
Standard SAEs analyse individual layers in isolation. However, features can evolve as information propagates through the network. Transcoders (Figure 7a), cross-layer transcoders (Figure 7b), and crosscoders (Figure 9) enable multi-layer feature extraction and analysis, revealing how representations and features associated with particular model components—most commonly MLP blocks—transform across depth.
Figure 7. Transcoders and CLTs. Transcoders reconstruct the output of the MLP from its input in the same transformer layer. CLTs extend this by partially reconstructing the outputs of multiple subsequent layers from a single point in the residual stream.
Figure 9. Crosscoder. Crosscoders are trained to reconstruct the residual stream across multiple layers simultaneously.
Transcoders: Bridging the Nonlinearities
Transcoder (Figure 7a)
A transcoder learns a sparse mapping between the input to and output of a model component, such as an MLP block, in layer :
where are sparse feature activations bridging the nonlinearity between input and output of the MLP, and and are decoder parameters. The transcoder reconstructs post-nonlinearity activations using features extracted from pre-nonlinearity activations (Dunefsky et al., 2024).
Architecture and Motivation. Standard SAEs do not explain the nonlinear transformations (i.e., weight matrices of MLP blocks and attention heads) themselves that produce the activations. A transcoder achieves this by applying dictionary learning to the mapping, learning features that explain the transformation. Its training minimises
where as defined above.
When Are Transcoders Necessary? SAEs extract features from static activations and thus often suffice for residual-stream analysis. Transcoders track how features emerge from input to output and are essential for analysing MLP layers (Dunefsky et al., 2024).
Cross-Layer Transcoders: Enabling Better Circuit Analysis
Cross-Layer Transcoder (Figure 7b)
Cross-layer transcoders (CLTs) are sparse feature decompositions spanning all layers of a neural network. At each layer , sparse feature activations are read from the residual stream via
where denotes the activation of feature at layer .
Each feature can write to the MLP outputs of its own layer and all subsequent layers through layer-specific decoder vectors. The contribution from features read at layer to the MLP output at a subsequent layer is
where is the decoder vector from feature at layer to the MLP output at layer . The reconstructed MLP output at layer is then
which approximates the original MLP output .
CLTs (Figure 7b) are trained jointly across all layers to minimise
where is the sharpness hyperparameter [11].
Replacement Models and Attribution Graphs. A key insight is that CLTs enable the construction of a replacement model—a version of the original model in which all MLP layers are replaced with their CLT reconstructions. Running this replacement model involves:
Computing CLT feature activations at each layer
Replacing MLP outputs with reconstructions from features in layer 1 through
Keeping attention layers unchanged
When analysing a specific prompt, the replacement model freezes the original model’s attention patterns and normalisation statistics from its forward pass, then adds layer-specific error corrections. Under these conditions, the local replacement model can exactly match the original model’s activations on that prompt while decomposing it into interpretable CLT features. However, when the model’s activations are completely replaced by CLT reconstructions, the intervened model’s top-1 predictions match the original model’s on only about 50% of prompts from the Pile dataset (Gao et al., 2020). Notably, this experiment uses a relatively small, 18-layer transformer [11]. Therefore, this figure should be interpreted with important caveats (see below).
Caveats on CLT Fidelity Claims
CLTs have been claimed to possess high output fidelity given that they achieve high top-1 token agreement. However, this claim requires careful examination:
Conditions for reported fidelity:
Results are reported primarily on toy or small models, whose generality to other model architectures (e.g., Llama, Qwen) and model sizes (e.g., 70B models) is underexplored.
Fidelity was measured on short sequences in the CLT training data. Evaluation under distribution shift (e.g., out-of-domain tasks, adversarial inputs) is not adequate.
The paper reports top-1 token agreement as the main result, while softer metrics (e.g., top-5 agreement, KL divergence) may show different patterns and insights.
Conditions for exact local matching:
Attention patterns must be frozen from the original forward pass since the CLT does not compute or reconstruct attention computations.
Layer normalisation statistics (means, variances) must be cached from the original model. The replacement model does not recompute these.
Error correction terms are added per layer to compensate for reconstruction residuals. Without them, exact matching fails.
Factors affecting fidelity:
Model scale: Preliminary evidence suggests that reconstruction quality and feature interpretability improve with CLT model size. This may be because larger CLTs produce more structured, less entangled representations [11]. However, systematic scaling studies are limited.
Layer depth: Early layers’ and late layers’ activations are usually input-specific and output-constrained, respectively. Middle layers have richer information derived from the input but have not yet converged to the output. As a result, middle layers often show higher reconstruction fidelity.
Domain: Fidelity on factual recall and syntactic tasks is higher than on complex reasoning or arithmetic tasks, where MLP computations may be more complex and cannot be approximated by sparse linear features.
These caveats do not invalidate CLTs as a tool for mechanistic analysis, but they do mean that fidelity should be verified on the target model and specific task before drawing strong conclusions from CLT-based circuit analysis.
Why CLTs Enable Better Circuit Analysis. Unlike per-layer transcoders, CLTs reveal direct feature-to-feature interactions across layers: a feature in layer directly influences features in later layers through its contribution to their inputs. This simplifies circuit structure—features need not re-encode information at each layer. Additionally, replacing MLPs with CLT reconstructions makes the remaining computation approximately linear (with frozen attention patterns and normalisation), enabling principled attribution: each feature’s activation can be expressed as the sum of its input edges up to its activation threshold [11].
However, linearisation depends on several assumptions:
Attention freezing: By caching attention patterns, the replacement model treats attention as a fixed routing operation rather than a dynamic, input-dependent computation. This means CLT-based circuits describe model computation conditioned on given attention patterns without analysing them.
Normalisation caching: CLT analysis caches the input-dependent normalisation factors (e.g., LayerNorm denominators) from the original forward pass and reuses them during attribution instead of recomputing them after interventions. This removes a source of nonlinearity, but if an intervention would substantially change these normalisation factors in the original model, the cached computation no longer faithfully represents the intervened state.
Sparse linear approximation: Dictionary learning methods assume MLP outputs or transformations are well approximated by sparse linear combinations of features. This assumption may fail when computations involve complex feature interactions or highly nonlinear transformations.
CLTs provide a powerful lens for circuit discovery when these assumptions hold. When they fail, CLT-based explanations may become unfaithful.
Computational Cost and Practical Considerations. Let , where is the expansion factor. For a transformer with layers, a complete suite of CLTs contains encoders mapping and decoders mapping . The resulting parameter count, with biases ignored, is therefore
which highlights the quadratic scaling with both model depth and hidden dimension, as illustrated in Figure 8.

Figure 8. Parameter count (left) and FP16 memory requirements (right) of a full cross-layer transcoder as a function of model depth, for different residual stream dimensions and expansion factor .
Training a suite of CLTs thus incurs a high upfront cost. For reference, training the CLT suite for Gemma-2-9B required approximately 3,844 H100 GPU-hours at 50% FLOPs efficiency [11]. This cost is amortised over many circuit analyses on the trained CLT. The resulting interpretability gains (cross-layer features, lower interference, better attribution) justify the investment for systematic mechanistic analysis.
For these reasons, current applications of CLTs remain limited: a “greater-than” mechanism study (Merullo et al., 2025), an open-source extension (Lindsey et al., 2025a), and analysis of multilingual representations using CLTs (Harrasse et al., 2026). This scarcity of downstream use cases likely reflects the considerable challenges of training CLTs at scale and the complexity of using CLT-based circuit discovery in the wild.
Open Questions for CLT Research. Several important questions remain open:
How does CLT fidelity scale in large models? How can we better measure fidelity?
Can CLTs be extended to model more nonlinearities such as attention patterns and normalisation, while maintaining their explanatory power?
Can we develop evaluation protocols that reliably detect the potential unfaithfulness of CLT-based explanations?
How sensitive are CLT circuits to experimental setups, such as the choice of training data?
Crosscoders: Analysing Cross-Layer Features
Crosscoder (Figure 9)
A crosscoder learns a shared sparse representation from activations across layers. It jointly encodes activations from all layers as
where denotes the shared sparse feature activations and is the layer- encoder. Each layer is then reconstructed using a layer-specific decoder:
Thus, a single latent feature can capture patterns that are shared across, persist through, or vary across multiple layers of the residual stream (Lindsey et al., 2024). Studies have also used crosscoders to discover shared features in activations from different models (Ge et al., 2026; Bayazit et al., 2026; Lindsey et al., 2024; Jiralerspong & Bricken, 2025), as detailed in § 11.1.
A crosscoder is trained by minimising
where denotes the decoder vector of feature at layer , and controls the sparsity penalty.
Causal vs. Acausal Crosscoders. Crosscoders support two information flow patterns:
Causal Crosscoders: Read only from layers at the current position. This respects the forward pass causality: earlier layers cannot read from later layers.
Acausal Crosscoders: Read from any layer regardless of depth ordering. Useful for understanding post-hoc feature relationships across the full model.
Applications to Circuit Analysis. By extracting cross-layer features, crosscoders help us understand how features propagate through the residual stream across layers, removing duplicate feature representations that complicate circuit analysis. This simplification allows features to “jump” across intervening identity operations, substantially reducing circuit complexity. More importantly, crosscoders reveal how input-relevant features transform as they flow through the network, distinguishing which features persist unchanged versus which are actively modified by downstream computation (Lindsey et al., 2024).
Comparing SAEs, Transcoders, and Crosscoders
Single-layer SAEs are efficient and locally interpretable but miss cross-layer feature evolution. Transcoders improve this by imitating MLP behaviour and revealing nonlinear transformations, though they require sequential analysis across depth. Cross-layer transcoders span multiple layers for global information flow but are extremely expensive. Crosscoders take a different approach, extracting shared features across layers simultaneously to resolve cross-layer superposition and simplify circuits without chaining predictions.
Case Study: Induction Heads
Here, we examine induction heads (Olsson et al., 2022), a mechanism found in many modern language models, as a case study. We first describe their conditional-copying function and role in in-context learning, and then relate them to feature-extraction methods such as SAEs and CLTs, which reveal how this computation can be distributed across broader feature circuits.
Induction Heads
Induction heads, first identified and analysed by Elhage et al. (2021), are attention heads that implement a conditional copying rule over repeated subsequences. Let be a token sequence, and consider positions such that . An attention head indexed (see § 4.4 for an introduction to attention modules) is said to be an induction head when its attention pattern assigns comparatively high attention weight from position to position (i.e., high ) and its residual-stream update
increases the model’s logit for token when predicting the token at position .
That is, induction heads implement the rule
They help enable in-context prediction by copying the continuation that followed an earlier occurrence of the same token.
Induction in a Repeated Sequence
Consider the sequence
A B C A B
where the model needs to predict the token after the second B. Observing the repeated prefix A B, an induction head attends from the second B to the position after the first B and contributes to the model’s prediction of C. This induction behaviour should generalise across tokens and positions.
Induction heads were originally characterised as specific attention heads implementing this copying rule. However, subsequent work shows that we can understand the underlying computation more naturally at the level of distributed representations than at the level of isolated model components.
Induction Heads: Features Across Representations
Feature extraction methods, such as SAEs, can decompose attention layer outputs into sparse, interpretable features that capture meaningful activation patterns beyond individual attention weights (Kissane et al., 2024). Modern analyses suggest that induction heads are not isolated mechanisms but participate in distributed feature circuits, where multiple interacting features recur and compose across layers (Dunefsky et al., 2024; Singh et al., 2024; Kissane et al., 2024; Kamath et al., 2025).
SAEs trained on attention heads’ activations reveal feature families of recurring patterns and contextual dependencies. Some of them align with behaviours originally attributed to induction heads (Kissane et al., 2024). By analysing these features, we can understand how models maintain and manipulate information across tokens and contexts.
Cross-layer analyses, including crosscoders, suggest that certain features are represented consistently across layers rather than being recomputed at each layer (Lindsey et al., 2024). This perspective highlights that attention heads, including specialised ones such as induction heads, may be only part of broader, distributed feature representations.
Comparison with Alternative Feature Extraction Methods
Besides SAEs, alternative techniques from signal processing and machine learning have also been adapted to extract features from neural networks. They make different modelling assumptions and therefore offer distinct trade-offs in interpretability, scalability, and robustness. Table 6 summarises the methods discussed in this section.
| Method | Assumption | Strengths | Weaknesses | Best Use Case |
| SAE | Sparse, linear features | Open-ended discovery; fast inference | Hyperparameter-sensitive; identifiability issues | Open-ended feature discovery |
| NMF | Non-negative, additive parts | Parts-based interpretation; uniqueness guarantees | Requires non-negative data; limited sparsity | Naturally non-negative domains |
| PCA | Variance aligns with features | Closed form; optimal low-rank reconstruction | Explains global variance and produces orthogonal features | Small-scale settings; as a baseline method |
| ICA | Complete and independent sources | Theoretical guarantees when assumptions hold | Assumes independence; Gaussian blindness | When independence holds |
| Supervised methods | Known concept labels | High performance on targeted concepts | Requires labels; no discovery | When certain concepts of interest are known |
| K-SVD | Sparse, linear features | Explicitly solving the sparse-coding problem; no encoder bias | Slow inference; no learned encoder | Small, fixed sets of activations; as a baseline method |
Table 6. Comparison of feature extraction methods.
Non-Negative Matrix Factorisation
Non-negative matrix factorisation (NMF; Lee & Seung, 1999) decomposes a non-negative activation matrix as , where both the feature dictionary ( is the preset number of features) and the coefficient matrix are constrained to have non-negative elements. Because all components are non-negative, each feature contributes positively to the final representation, making it easier to interpret. Additionally, classical NMF’s optimisation process uses simple multiplicative update rules that preserve non-negativity. Moreover, under assumptions like separability (each latent feature appears in at least one sufficiently “pure” example), NMF guarantees that the recovered decomposition is unique (Donoho & Stodden, 2003).
However, classical NMF is not suitable for neural network activations because they can have negative elements, requiring additional preprocessing steps that may distort the activation-space structure. Moreover, NMF does not explicitly encourage feature sparsity, and scaling NMF to large dictionaries is computationally expensive. As a result, empirical work on extracting and analysing features in neural network activations has largely focused on Semi-NMF (Ding et al., 2010), which relaxes the non-negativity constraint on . Shafran et al. (2026) showed that Semi-NMF can recover features competitive with SAEs on small-scale evaluations, but systematic comparisons at large scale remain limited. Yun et al. (2021) used a Semi-NMF-like approach to factorise BERT’s contextualised hidden-state representations (Devlin et al., 2019), allowing elements in to take both positive and negative values while constraining to be non-negative.
Principal Component Analysis
Principal Component Analysis (PCA; Pearson, 1901; Hotelling, 1933) decomposes activations by finding an orthonormal basis of directions that maximise explained variance. Given a mean-centred activation matrix , where each column is an activation vector, PCA computes a low-rank approximation , where the columns of are the principal directions ordered by explained variance. PCA is fast, stable, and scalable. However, PCA imposes constraints and inductive biases that may misalign with neural network activations. Specifically, it identifies principal directions that maximise explained variance in the activation distribution, but much of the information of interest is region- or context-dependent and cannot be captured by variance alone. Moreover, PCA constrains the principal directions to be orthogonal, but in practice the underlying features correlate or partially overlap. Recent benchmarks (Wu et al., 2025b) reflect this misalignment: PCA underperformed most baselines, including SAEs, on both steering and concept-detection evaluations. While PCA can outperform SAEs in some localisation settings (Mueller et al., 2025), it still generally lags behind simple supervised baselines.
Independent Component Analysis
Independent Component Analysis (ICA; Hyvärinen & Oja, 2000) decomposes activations as , where is the number of data points. Note that ICA may look similar to NMF at first glance. However, they were developed for different use cases and thus rely on different assumptions and optimisation procedures. In ICA, each column in is a mixing vector, and is the number of different source signals; each row in is a source signal (a signal has data points), with source signals in total. These signals are optimised to be independent of one another. Thus, if model activations are linear mixtures of statistically independent “foundational features”, ICA can provably recover those features.
However, these assumptions may not hold for neural network activations. Specifically, classical ICA typically expects the completeness property (number of latent sources equals the dimensionality of the observed representation), and overcomplete decompositions (which are what we expect in activations) would require additional extensions (Lewicki & Sejnowski, 2000). Moreover, underlying features may not be statistically independent, which further limits the applicability of classical ICA. Although early studies applied it to analyse neural network activations, they did not find interpretability gains over SAEs (Cunninghame et al., 2024). This may partly reflect the challenges discussed above.
More recently, however, Liu & Han (2026) showed that ICA can achieve performance comparable to that of SAEs, arguing that its previously reported underperformance may stem from limitations of classical ICA implementations poorly suited to the scale and statistical structure of neural network activations. These findings leave open whether ICA should be regarded primarily as a simple classical baseline or as an approach with underestimated, substantial headroom for further development.
Supervised Feature Extraction Methods
When the concepts of interest are known in advance, supervised methods can directly optimise representations aligned with those targets. Examples include distributed alignment search (DAS), which uses supervision to learn a change of basis that aligns representations with high-level causal variables (Geiger et al., 2024); Supervised Multi-Dimensional Scaling (SMDS), which projects activations to preserve concept-level similarities (Tiblias et al., 2026); and probes, which fit a concept direction in contrastive class-conditioned activations (Marks & Tegmark, 2024).
Notably, despite its simplicity, probing has outperformed SAEs in concept detection (Wu et al., 2025b). It has also become a standard tool for identifying safety-relevant concepts in model representations, including refusal behaviour and implanted backdoors [20] (MacDiarmid et al. 2024). We discuss probing in detail in § 10.6. We put it under § 10 because we view its central methodological contributions and technical challenges as the readout of human-interpretable information—often expressed in natural language—from model activations, rather than the extraction of monosemantic features or causal variables.
When labelled data is available, supervised approaches often outperform SAEs on concept detection and steering tasks (Huang et al., 2024; Wu et al., 2025b; Chaudhary & Geiger, 2024). They also avoid choosing dictionary sizes or sparsity penalties, since the concept set determines complexity. However, these methods require labelled data for each concept and cannot discover previously unknown features. Therefore, supervised methods suit settings where predefined, crisp concepts matter, whereas SAEs remain preferable for exploratory analysis.
Dictionary Learning Baselines
Classical dictionary learning methods provide useful baselines for evaluating SAEs. Techniques such as K-SVD alternate between sparse coding and dictionary updates and converge under certain conditions [7]. Comparative studies suggest that SAEs’ primary advantage is fast inference because the encoder learns to decompose all activations, while potentially sacrificing optimal sparsity (O’Neill et al., 2025). As such, classical dictionary learning can still achieve superior dictionary quality in some cases and thus remains an important reference point for assessing the added value of SAEs.
Interpretability-by-Design Methods for Intrinsic Interpretability
The methods surveyed thus far in this section aim to extract and analyse more monosemantic units from polysemantic model activations. They share a common premise: the model’s original representations are too entangled to be directly interpretable, so interpretability is applied post hoc to recover simpler units of analysis. A parallel research programme, which we term Interpretability-by-Design Methods for Intrinsic Interpretability, inverts this premise by asking whether deep neural models can instead be designed to be mechanistically interpretable by construction [5] (Pearce et al. 2025; Park et al. 2025; Oldfield et al. 2024; Yang et al. 2025; Gao et al. 2026; Draye et al. 2026; Dooms et al. 2025). This subsection extends the discussion of this research direction.
Next, we introduce five approaches for constructing models that are more interpretable by design. Three impose model sparsity constraints at different levels: feature-sparse training (§ 6.10.1), weight-sparse training (§ 6.10.2), and attention-sparse training (§ 6.10.3). The remaining two approaches instead modify the computational primitives themselves, using more interpretable operators as units of construction and analysis (§ 6.10.4 and § 6.10.5).
Interpretability should be understood as a spectrum rather than a binary property. At least in their current form, these techniques do not make model computation completely or immediately understandable. Instead, they make models easier to analyse and ensure the analysis pipeline preserves more information about the original computation than post-hoc approximations. Table 7 summarises these approaches and their trade-offs.
| Method | Approach | Interpretability Gain | Primary Limitation |
| Feature-Sparse Training (SoLU) | Softmax-gated activation function | Increased neuron monosemanticity | Layer normalisation smuggling displaces superposition |
| Weight-Sparse Training | constraint on parameters | More monosemantic neurons and much smaller circuits | Training instability and scalability |
| Attention-Sparse Training | regularisation on attention | Makes attention patterns interpretable | Requires fine-tuning on a trained language model |
| MOLT | Sparse mixture of linear transforms | Reveals implemented operations directly | Complexity of matrix analysis |
| KANs | Learnable splines on edges | Symbolic formula extraction | Scalability to LLMs is unproven |
Table 7. Comparison of interpretability-by-design methods.
Feature-Sparse Training
Softmax linear units (SoLU; Elhage et al., 2022b) are an earlier attempt at architectural intervention. SoLU is a specialised activation function that applies a softmax-based transformation across the layer’s neurons. Specifically, given the pre-activation vector
SoLU calculates the activation values as
This softmax-based transformation induces competition among neurons. When one neuron receives a large pre-activation value, the softmax suppresses the others’ values, analogous to lateral inhibition. This competition encourages certain neurons to specialise and dominate, increasing the prevalence of monosemantic neurons associated with human-interpretable concepts.
However, it also reveals an undesirable phenomenon: although many secondary neurons’ activations are suppressed to low magnitudes, these neurons remain polysemantic, with many features compressed into them. Later layer normalisation [25] can amplify them and recover the polysemanticity property (Elhage et al., 2022b). The “smuggling” by layer normalisation suggests that imposing local interpretability constraints may not eliminate neural networks’ incentive to compress information through superposition. This issue motivates global constraints across the neural network, such as weight-sparse training discussed below.
Weight-Sparse Training
A weight-sparse transformer (Gao et al., 2026) treats weight sparsity as an explicit structural constraint that is progressively strengthened during training. Gao et al. (2026) optimise the standard cross-entropy objective along with an constraint on model weights after each optimisation step. Specifically, within each weight matrix, they retain only the largest-magnitude parameters and set the rest to zero. The constraint anneals from a dense initialisation to the desired sparsity level over training. The resulting transformer has most weight parameters exactly zero, making the remaining computational structure much easier to decompose into small, task-specific circuits.
Its interpretability gains can be substantial. Under highly sparse regimes (e.g., < 1% active weights), neurons receive input from far fewer upstream neurons, and the model’s circuits for specific capabilities or behaviours become much sparser. Mean-ablation experiments further confirm that these circuits are both necessary and sufficient for the capability or behaviour.
Gao et al. (2026) characterised the trade-off between model capability and interpretability induced by weight sparsity. Increasing sparsity generally makes the model easier to decompose into small, interpretable circuits. However, it also degrades task performance, producing a capability–interpretability Pareto frontier. Nonetheless, the authors found that increasing model size can shift this frontier outward. At comparable task performance, larger sparse models can have substantially smaller, more interpretable circuits than smaller models; at comparable interpretability, they can retain stronger capabilities. This suggests that scaling model size alongside structural sparsity may offer a path to more interpretable yet high-capability models. However, preserving interpretability beyond tens of millions of nonzero parameters remains challenging, and weight-sparse training currently requires substantially more compute than traditional training.
Attention-Sparse Training
In transformers, attention patterns produced by an attention head are usually diffuse, i.e., every token attends to every other token. Diffuse attention patterns are difficult to interpret and audit. Previous studies have therefore explored attention-sparse training, which aims to sparsify attention patterns to make them more interpretable (Lei et al., 2025; Draye et al., 2026). Specifically, Lei et al. (2025) proposed SPARSe TrAnsformer World model (SPARTAN), an object-centric world model trained with regularisation on attention patterns and a constrained optimisation objective to maintain cross-entropy performance equal to that of a dense counterpart. Draye et al. (2026) further adapted this technique to fine-tune and sparsify a trained language model’s attention patterns.
After training (either from scratch or via fine-tuning), each attention head attends only to specific tokens, and the model performance remains close to the original. This result suggests that many attention patterns may be redundant noise. In CLT-based circuit discovery (see § 7.5), Draye et al. (2026) showed that sparsified attention patterns reduce the number of mediating components involved in feature-to-feature attribution, making the resulting attribution graphs easier to analyse.
Modifications of Computational Primitives: MOLT
One alternative approach to learning interpretable structure is to reconsider what the computational primitives should be ((Oldfield et al., 2025; Park et al., 2025; Braun et al., 2025; Bushnaq et al., 2025}). A recent example is the sparse mixture of linear transforms (MOLT), which represents a conceptual shift from feature extraction towards transformation extraction ((Lindsey et al., 2025b)).
Standard SAEs and transcoders primarily decompose model activations into features. However, many algorithmic tasks, such as arithmetic or geometric reasoning, are more naturally described by operations such as “add 3” or “rotate a vector”. In these tasks, we want to know how the model transforms one activation into another through components like weight matrices in MLP blocks. To achieve this, MOLT replaces the MLP with a sparse mixture of learned linear transformation matrices. The basic unit of analysis becomes transformations applied to model activations rather than features.
For example, in a modular arithmetic task, MOLT can represent the concept of addition with a learned rotation matrix, which relates more directly to the underlying algorithm than a collection of state-specific features. More broadly, approaches that represent computational operations explicitly, as MOLT does, may facilitate the development of interpretable representations for tasks governed by systematic operations. We view Kolmogorov–Arnold Networks (KANs) as an approach that embodies this idea, as discussed below.
Kolmogorov–Arnold Networks
Kolmogorov–Arnold Networks (KANs) represent one of the most radical departures from standard neural network architectures ((Liu et al., 2025b)). In conventional neural networks, each neuron receives outputs from upstream neurons through incoming edges weighted by scalar parameters, sums these weighted inputs, and then applies a fixed nonlinear activation function (the whole process can be represented as matrix multiplication in components like MLP blocks and attention modules in § 4.4). Inspired by the Kolmogorov–Arnold representation theorem, KANs place learnable nonlinear functions, typically parameterised using B-splines, on edges rather than using fixed nonlinear activation functions at nodes.
Specifically, consider a single KAN node receiving an input vector from the previous layer. Its output is
where denotes the -th basis function, such as a cubic B-spline basis, and is a learnable scalar coefficient. In this formulation, the edge is not scalar multiplication but a learnable univariate function , and a node simply sums the outputs of its incoming edges. Intuitively, KANs shift nonlinear transformations from nodes to edges, offering an alternative way to structure and interpret neural network computations.
This design gives KANs an explicitly compositional and thus more interpretable structure. As the nonlinearities are univariate and can be visualised directly on individual edges, researchers can use techniques like symbolic regression to approximate and inspect the learned analytical expressions, such as rediscovering known physical equations. Although KAN training is currently more computationally expensive than training standard MLPs and has not yet been convincingly demonstrated at the scale of large language models, this architectural innovation points to an ambitious direction towards interpretable neural networks in which parameters correspond to mathematically meaningful functions.
Summary. This section introduced several classical methods for feature extraction and analysis, followed by approaches that aim to make models intrinsically interpretable. However, whether features are recovered post hoc or induced by design, they remain units of analysis rather than complete explanations of the model’s mechanism, i.e., how these units interact and causally influence one another. Section 7 therefore shifts from individual features to circuits: computational graphs of interacting components that describe how the model implements its behaviours.
Circuits in Mechanistic Interpretability
Mechanistic interpretability aims not only to identify what information a model represents, but also to explain how it transforms and combines that information across layers to produce behaviour. Feature analysis is a useful tool for describing activation vectors, but it describes them in isolation. Many model behaviours arise from structured interactions among multiple components that operate together across layers. To explain these behaviours and their mechanisms, we need a high-level abstraction that captures critical computational patterns of these features. The circuit framework (Figure 3; introduced in § 4.6) addresses this gap by modelling behaviour as coordinated computational graphs and making explicit how components causally interact.
Notably, the circuit framework is not the only way to model an internal mechanism. Early studies have explored alternative ways. For example, Giles et al. (1992) and Omlin & Giles (1996) categorised a recurrent neural network’s hidden representations into different states and modelled its decision process as a finite-state automaton when the input sentence was fed sequentially. Nonetheless, the circuit framework has attracted growing research interest in studying neural networks’ internal mechanisms. Below, we introduce approaches for identifying and constructing circuits.
Circuit Discovery Methods
Circuit discovery methods aim to identify internal computational graphs responsible for specific capabilities or outputs in a neural network. Below, and in Table 8, we formalise key techniques and methods, along with their strengths and limitations.
| Method | Core Idea | Strengths | Limitations |
| Activation Patching | Intervene on components and measure the resulting output change | Simple and intuitive; provides direct causal evidence | Requires many interventions; component-level granularity can be coarse |
| Path Patching | Intervene on individual information paths or edges and measure their causal effects | Finer-grained localisation of information flow | Computationally expensive due to the large number of candidate paths |
| ACDC | Greedily prune edges from the computational graph while testing whether the target behaviour is preserved | Fully automated; directly searches for a behaviourally sufficient subgraph | Computationally expensive due to sequential edge pruning; sensitive to pruning order and threshold choices |
| EAP | Approximate edge-level attribution using path patching and first-order gradients, enabling parallel evaluation of many candidate edges | Highly parallelisable and computationally efficient; enables large-scale edge attribution | Relies on a first-order approximation; attribution scores may not fully capture nonlinear or interaction effects |
| SAE- or CLT-Based | Discover circuits over sparse, interpretable features rather than raw model components | Leverages interpretable features; enables finer-grained feature-level circuit analysis | Depends on feature quality; requires additional computation to train and analyse SAEs/CLTs |
Table 8. Trade-offs and characteristics of circuit discovery methods.
Activation Patching
Given a model parameterised by and an input sequence , activation patching evaluates the causal contribution of some model components by replacing their activations with specified reference values and measuring the resulting change in the model output. The causal importance of for is defined as
where is a scalar performance metric computed from the model output, such as the logit difference for the answer token, and denotes a forward pass on in which the activations of components in are replaced by while the remaining computation proceeds normally. The reference activations usually come from a corrupted reference run, where the input is modified to add or remove specific information. Some works have also set to zero or the mean model activation. The resulting change in model output quantifies the causal effect of intervening on (Zhang & Nanda, 2024).
Path Patching
Path patching extends activation patching on model components to specific edges in the computational graph. For a connection from model component to denoting that the output activations of are fed to as one of its inputs, it replaces activations along with reference values and measures ’s causal effect as:
This allows finer-grained causal localisation along information flows (Goldowsky-Dill et al., 2023).
Automatic Circuit Discovery (ACDC)
ACDC (Conmy et al., 2023) discovers a circuit by iteratively using path patching to identify important connections in the model’s computational graph. Traversing the graph in reverse topological order from the output node, it patches the activation transmitted along each edge and measures the resulting change in a task-specific metric. An edge will be pruned if patching it produces less than a predefined threshold ; otherwise, it is retained in the graph and its upstream edges will be investigated. The remaining subgraph represents the discovered circuit.
Edge Attribution Patching (EAP)
EAP (Syed et al., 2024) estimates the causal importance of edges in a model’s computational graph by approximating the effect of path patching with a first-order Taylor expansion. For an edge connecting components and , its importance score is approximated as
where and denote the activations transmitted from component to component along edge on clean and corrupted inputs, respectively, when processing . is a scalar metric of model performance as mentioned above. EAP is substantially faster than ACDC because the attribution scores for all edges are computed using only two forward passes and a single backward pass. Specifically, the forward passes obtain and , while the backward pass computes for all components simultaneously. In contrast, ACDC requires a separate forward pass for each edge it tests.
Edges with large score magnitudes are generally considered main contributors to the target behaviour. Therefore, one simple approach to circuit construction is directly selecting the highest-scoring edges (Syed et al., 2024). Subsequent work has improved EAP in different directions: EAP-IG (Hanna et al., 2024) and EAP-GP (Zhang et al., 2025b) improve edge-attribution estimation, while PEAP (Haklay et al., 2025) extends patching across token positions to produce finer-grained attribution graphs that capture cross-token interactions.
Besides intervention (activation patching, ACDC, etc.) and attribution (EAP, EAP-IG, etc.) techniques, another line of approaches treats circuit discovery as an optimisation problem (Csordás et al., 2021; Cao et al., 2021; Bhaskar et al., 2024; Haider et al., 2026). Specifically, they use gradient descent to learn which parts of the computational graph can be removed without affecting model behaviour. For example, Edge Pruning (Bhaskar et al., 2024) learns edge masks to discover sparse connections, while multi-granular node pruning (Haider et al., 2026) learns node masks at different scales, from entire blocks to individual neurons. These methods can usually discover sparser circuits than intervention and attribution counterparts. However, the optimisation process itself introduces randomness and sensitivity to training configurations like the learning rate, which may make the subsequent analysis less robust and interpretable.
Circuits can explain models in different contexts, from a single input–output pair to a capability or behaviour. Identifying a circuit for a specific input or prompt is called query-level circuit discovery (Wu & Barez, 2026). In contrast, task-level circuit discovery (Wang et al., 2023; Kharlapenko et al., 2025) seeks to identify a circuit that implements a broader capability or task, typically operationalised by a dataset of examples, such as the indirect object identification (IOI) dataset (Wang et al., 2023). In the task-level setting, components’ and edges’ attribution scores (e.g., or ) are aggregated across inputs to reduce example-specific noise, yielding a circuit that characterises the task as a whole.
The techniques and methods introduced above, including activation patching, path patching, ACDC, EAP, EAP-IG, and mask-based pruning methods, have primarily been developed and evaluated in the task-level setting. More recent work has extended EAP-IG to identify circuits for individual prompts (Wu & Barez, 2026). This enables the discovery of sparse yet faithful circuits in the model even for complex, natural-language user queries, where the relevant computation may vary substantially from one input to another.
Circuit Discovery with Replacement Models
The methods introduced above discover circuits within the target model. Their nodes correspond to coarse-grained components such as attention heads and MLP blocks, which are mostly not immediately understandable. A complementary approach first constructs a more interpretable replacement model (Section 6.6.2) that approximates the original model’s computation using sparse linear features, and then performs circuit discovery within this replacement model.
Recent work has implemented this paradigm using cross-layer transcoders (CLTs; § 6.6.2) [11] (Merullo et al. 2025). In particular, Circuit Tracer [11] replaces the model’s MLPs with CLTs while retaining the underlying model’s attention and residual-stream computation. This yields a prompt-specific attribution graph whose principal internal nodes are sparse CLT features rather than entire MLP blocks or attention heads.

Figure 7b.
Circuit Tracer [11] constructs a prompt-specific attribution graph in a CLT-based replacement model. For a given input prompt, consider a source feature at layer active at token position with activation value , and a downstream target feature at layer active at position . Recall that a CLT feature can write to the reconstructed MLP outputs of its own and all subsequent layers (Figure 7b); the source feature has layer-specific output vectors to these layers, corresponding to columns of the layer-specific decoder matrices:
On the other hand, the target feature reads from the MLP input through its corresponding encoder vector, given by a row of the encoder matrix:
Then, let
denote the Jacobian that maps a perturbation written at token position and layer to the MLP input at position and layer , where each of the entries is
The direct attribution from source feature to target feature can then be modelled as
Each term in the summation measures the contribution of the source feature ’s write at layer to the pre-activation of target feature , transported through the residual stream and attention heads. aggregates the downstream writes across layers produced by the same source feature. Notably, Ameisen et al. [11] stop the gradients flowing through intermediate MLP outputs and fix attention patterns and normalisation denominators. This lets each edge capture direct feature-to-feature influence rather than effects mediated through intermediate feature nodes.
CLT-based circuit discovery differs from conventional circuit discovery in two aspects. First, the resulting circuit’s nodes correspond to fine-grained linear features rather than coarse MLP blocks or attention modules. Using automated feature-interpretation pipelines (§ 14.7) to assign natural-language descriptions to these features, human inspectors can more easily understand the discovered mechanism. Second, an edge in a CLT-based circuit denotes the direct, input-specific influence between two features, which is different from the formulation of attribution methods based on clean–corrupted reference pairs, such as EAP and EAP-IG. Specifically, in Circuit Tracer, a source feature can write to the residual stream at multiple subsequent layers, and the strength of an edge from the source feature to a target feature is the aggregate of the contributions of these writes to the target feature.
For a given prompt, Circuit Tracer produces a sparse attribution graph that describes active features and their interactions, making it well suited to query-level circuit discovery. Interpretability platforms such as Neuronpedia (Lin, 2023) also offer interactive implementations. However, the approach still has several limitations. Specifically, CLTs only approximate MLP computations, so the discovered circuit may only cover a part of the model’s underlying mechanism. Moreover, feature-level attribution graphs can have many nodes and edges, so we must prune, cluster, or otherwise condense them before they become understandable.
Causal Verification
Once we identify circuits, two critical questions arise: are they causally responsible for the behaviour, and can we modify them reliably?
Causal Necessity and Sufficiency. A circuit is causally necessary for a behaviour if removing or ablating the circuit degrades that behaviour. A circuit is causally sufficient if activating the circuit alone, i.e., patching all edges outside the circuit with activations from a corrupted reference run, reproduces the behaviour. In practice, a circuit is rarely sparse, necessary, and sufficient at the same time; it typically explains 60–80% of behavioural variance.
Evaluating Circuits and Benchmarking Discovery Methods. A circuit is often considered faithful if it is both causally necessary and causally sufficient for the behaviour it is claimed to explain (Hanna et al., 2024; Marks et al., 2025a). Two popular approaches to validate causality are causal scrubbing and interchange interventions (Chan et al., 2022a; Geiger et al., 2024). Causal scrubbing (Chan et al., 2022a) systematically evaluates a mechanistic hypothesis by replacing model activations with those from inputs deemed equivalent under the hypothesis and testing whether these interventions preserve the model behaviour. Interchange interventions (Geiger et al., 2024) test whether model activations causally correspond to higher-level variables by substituting activations across inputs and measuring the resulting change in model behaviour. Together, these methods provide stronger evidence for mechanistic explanations than observational analysis alone.
More recently, several works have proposed metrics to quantify circuit faithfulness as a continuous score (Hanna et al., 2024; Wu & Barez, 2026), allowing systematic comparisons of circuits discovered by different methods. By aggregating faithfulness scores across tasks and settings, benchmarks such as MIB (Mueller et al., 2025) evaluate how well circuit discovery methods identify causally important model components. However, achieving strict necessity and sufficiency is often hard because many model behaviours depend on distributed representations and multiple redundant pathways (Barez et al., 2026). Even if we identify a circuit that captures the primary mechanism of a behaviour, components outside the circuit may still play important roles, such as handling edge cases and computational robustness.
Case Study: Induction Heads
Circuit Perspective on Induction Heads
Following our introduction to induction heads in § 6.8, we consider a repeated sequence:
where the model needs to predict the token after the second B. The induction head alone is not sufficient to implement induction behaviour from scratch. It still needs help from previous layers’ attention heads to locate the token following the current token’s previous occurrence, so it knows it should attend to C in this example (Olsson et al., 2022).
Specifically, in a two-layer attention-only transformer, Elhage et al. (2021) discovered the previous-token head, which writes every token’s information to the next position. The change propagates to the residual stream to be read by the induction head, allowing it to locate C (because now some information about the first B is contained in C). The two kinds of heads form the induction circuit that is sufficient to implement the induction behaviour in the two-layer small model.
Empirical evidence supports this mechanistic interpretation. Head ablation experiments show that induction heads causally support in-context learning in the attention-only model. Circuit analysis also reveals induction heads’ dependence on earlier previous-token heads. Moreover, induction heads emerge abruptly during training, coinciding with sharp improvements in in-context learning performance.
In larger models with additional layers and MLPs, the mechanism behind induction behaviour becomes more complex. However, the core principle remains the same: context propagates across layers and is used to conditionally retrieve tokens based on repeated subsequence matches. From this perspective, the circuit framework complements feature-based analysis by explicitly identifying the model’s internal mechanism that implements a specific behaviour.
A Unified Evaluation Framework for Mechanistic Interpretability
A persistent challenge in mechanistic interpretability (MI) is the lack of standardised evaluation. Different interpretability methods use incompatible metrics. For example, for the same behaviour of interest, circuit discovery can work at the level of MLP/attention modules (Conmy et al., 2023), CLT features [11], or even neurons [22]. A CLT-based circuit must also verify CLT fidelity; an MLP/attention-level circuit should test and explain each node’s role. Thus, it is hard to compare them and measure progress. To address this, this section presents a unified evaluation framework covering five dimensions—attribution faithfulness, completeness, stability, causal validity, and human interpretability—where different methods can be jointly tested.
The Need for Standardised Evaluation
Before we discuss the five dimensions, we first explain in detail the need for standardised evaluation. Specifically, current evaluation practices face several challenges:
Utility ambiguity: Methods are initially developed for different aims and evaluated against distinct objectives, yet have overlapping application scenarios. For example, SAEs are assessed using reconstruction error and sparsity metrics; circuits are evaluated using behavioural recovery or faithfulness tests; and difference-in-means steering vectors (see Section 9.3) are examined by behaviour change—although all three approaches can be used to steer model outputs. Their unclear intended uses make it hard to compare their utility.
Cherry-picking: Existing work often reports interpretability results on carefully selected examples, samples from a narrow distribution, or capabilities where a method performs well. It may not report negative results, failure cases, or coverage statistics. This makes it difficult to assess the robustness and generality of the findings and determine whether the proposed method genuinely outperforms existing ones.
Missing baselines: Many proposed methods’ claimed interpretability gains are not systematically and adequately compared against intuitive alternatives or baselines such as behavioural methods, random activations, or neuron-level analysis. Without this comparison, it is unclear whether a method extracts genuinely meaningful information.
Reproducibility gaps: Many existing techniques involve substantial hyperparameter choices, heavy training procedures, and high variation across seemingly irrelevant factors. Reproducibility is often underreported and has not yet been established as a property to optimise.
Faithfulness–understandability gap: Methods often demonstrate how understandable and informative their outputs are to humans. However, those outputs’ faithfulness to the underlying model mechanism receives less attention. This rewards explanations that are persuasive rather than faithful.
Lan et al. (2026) have called for a shared platform for tracking and auditing interpretability insights and claims across papers; Ayonrinde & Jaburi [23] proposed a framework that evaluates mechanistic explanations from several philosophy-of-science perspectives. However, these discussions remain largely conceptual. The MI community needs standardised, concrete evaluation protocols and benchmarks, along with established reporting practices, to enable principled comparisons across methods and findings. They are especially important for measuring real progress in the field, identifying systematic failure modes, and providing evidence about what methods do and do not reveal. More broadly, they are necessary for mechanistic interpretability to mature from a collection of promising techniques into a cumulative scientific discipline.
Five Evaluation Dimensions
We propose a framework that situates mechanistic interpretability evaluation within five complementary dimensions. No single dimension suffices—strong methods should perform well across all five. Table 9 summarises these five dimensions. We describe each one with example metrics that convey the idea.
| Dimension | Definition | Metrics | Methods |
| Attribution Faithfulness | Do attribution scores accurately reflect the causal effects of the identified components? | Intervention-effect agreement; effect-size error; counterfactual effect recovery | Ablation; activation / path patching; interchange interventions |
| Completeness | How much of the behaviour is accounted for by the identified components? | Loss recovery; behavioural coverage; unexplained variance | Reconstruction; behavioural evaluation; residual analysis |
| Stability | Are findings robust to variations in factors like random seeds, hyperparameters, paraphrases, etc.? | Cross-seed agreement; checkpoint consistency; hyperparameter sensitivity | Multi-seed analysis; checkpoint comparison; input paraphrasing |
| Causal Validity | Are the identified components necessary, sufficient, and specific for the target behaviour? | Necessity score; sufficiency score; specificity score | Ablation; component isolation; interchange interventions; tests with control behaviours |
| Human Interpretability | Can humans understand and use the resulting explanations? | Explanation utility; debugging utility; interpretation agreement | User studies; expert evaluation |
Table 9. Unified evaluation framework for mechanistic interpretability.
Note that throughout this subsection, we use model components to refer to any unit of analysis, including individual features, neurons, attention heads, circuits, and connections representing information flow between components. We operationalise the model behaviour of interest, , with a set of input prompts . Following the notation introduced in § 7.1, we denote the model by , a scalar behaviour-relevant performance metric by , and an intervention on components by
where the activations of are replaced by reference activations while the remaining computation proceeds normally. Following § 7.1, we define the intervention effect of on an input as
and the average intervention effect on behaviour as
Dimension 1: Attribution Faithfulness
Attribution faithfulness measures whether the interpretability method accurately attributes the model’s behaviour of interest to responsible components.
Attribution Faithfulness Metrics
Intervention-Effect Agreement. Let denote the importance score assigned by an interpretability method to components for a sample :
where may be Pearson or Spearman correlation. High agreement indicates that the attribution consistently reflects the identified components’ causal effects on behaviour .
Effect-Size Error. When attribution scores are calculated to estimate intervention effects, we measure the effect-size error as:
A low value indicates good calibration between estimated and actual intervention effects. If the two quantities are not naturally on the same scale, we may use normalised or rank-based measures instead.
Counterfactual Effect Recovery. Consider inputs and , where is a reference behaviour (e.g., honesty) and is the behaviour of interest (e.g., sycophancy). Patching from into the computation on yields
A value near 1 indicates that the patching recovers much of the behaviour, while a value near 0 indicates little effect. This metric has been widely used in the circuit discovery literature (Hanna et al., 2024; Zhang et al., 2025b; Syed et al., 2024). However, the denominator of this metric can be close to 0, and empirically can exceed or fall below . These may cause this metric to take values much larger than 1 or below 0, complicating the analysis. Recent work has proposed an alternative metric that bounds the value between 0 and 1 (Wu & Barez, 2026).
Evaluation protocol:
Apply the method to acquire attribution scores of model components and select those of interest (e.g., components with high scores).
Measure the corresponding ground-truth intervention effects .
Compare them using correlation, error, or rank agreement.
Evaluate how much of the behavioural difference between clean and corrupted inputs can be recovered by patching the identified components.
Report results across a diverse set of inputs with uncertainty estimates.
Dimension 2: Completeness
Completeness measures how much of the model behaviour the identified components or mechanisms explain. A high completeness score indicates that the identified components/mechanisms capture most of the relevant computation.
Completeness Metrics
Loss Recovery. For methods that reconstruct model activations (e.g., SAEs and CLTs), after replacing the target component’s activations with the reconstructions, we measure the fraction of the original model performance that is recovered:
where is the downstream test loss of the original model, is the loss after replacing the original activations with reconstructions, and is the baseline loss where activations are replaced with zero ablation, mean ablation, or Gaussian noise matching the layer statistics. A value near 1 indicates that the reconstruction preserves most of the information. Similar to the vanilla Counterfactual Effect Recovery metric discussed above, this metric can take arbitrary values. A simple solution is to develop a bounded variant, as has been done for Counterfactual Effect Recovery (Wu & Barez, 2026).
Behavioural Coverage. Across input prompts , we measure the fraction for which the identified components have a meaningful intervention effect:
where is the intervention effect defined above and specifies the minimum effect considered meaningful. Higher coverage indicates that the identified components account for the behaviour across a larger fraction of inputs.
Unexplained Variance. For methods that reconstruct activations of the components, we measure the fraction of unexplained activation variance as:
where , is the original model activation, and is its reconstruction from the identified components . Lower values indicate that the identified components explain a larger fraction of the variance in the original activations.
Evaluation protocol:
Report behavioural coverage using the intervention effects .
Report reconstruction-based metrics such as loss recovery and unexplained variance if applicable.
Analyse inputs of the behaviour for which the identified components do not provide good coverage or reconstruction.
Stability
Stability measures the robustness of explanations, attributions, and findings to changes in variables that should not affect the results, such as the choice of random seeds, small hyperparameter changes, and input paraphrasing.
Stability Metrics
Cross-Seed Agreement. We train or apply the interpretability method times under different random seeds. Let denote the set of features or mechanisms, such as SAE features and circuits, discovered under seed . We measure the cross-seed agreement as:
where measures similarity between sets. For example, Paulo & Belrose (2026) calculate the Hungarian matching between two sets of SAE features based on their pairwise cosine similarities.
Hyperparameter Sensitivity. We measure the sensitivity of mechanistic analyses or methods under reasonable hyperparameter variations. Let denote the set of evaluated hyperparameter configurations and denote the experimental result (e.g., a method’s performance) under a configuration . We calculate the hyperparameter sensitivity as:
Lower values indicate that the finding is less sensitive to hyperparameter choices and therefore more robust.
Paraphrase Consistency. For each input , let denote a set of semantics-preserving paraphrases of . We measure the consistency of explanations or attributions across . For example, we can measure whether the intervention effect of remains consistent:
The score lies in . A value of 1 indicates that the intervention effect is identical across all paraphrases. A value of 0 indicates that, for each , the expected intervention effect, i.e., , is 0.
Evaluation protocol:
Execute the interpretability method with several random seeds and compute alignment of the discovered units or findings across seeds.
Analyse which findings are stable across all seeds and which ones are seed-specific. Key claims need to remain consistent under subsequent tests on hyperparameter variation and input paraphrasing, if applicable.
Distinguish robust findings from fragile ones and report both of them.
Causal Validity
Causal validity assesses whether the identified components or mechanisms are causally responsible for the behaviour, particularly necessity (i.e., removing the components/mechanisms disrupts the behaviour) and sufficiency (i.e., the components/mechanisms alone are sufficient to preserve the behaviour).
Causal Validity Metrics
Necessity Score. Necessity measures the fraction of the model’s performance on behaviour that is lost when is ablated (i.e., its activations are replaced with some reference values). It is calculated as:
A higher value indicates that is necessary for the behaviour.
Sufficiency Score. Let denote the components outside . Sufficiency measures how much of the original behaviour is achieved when only is active (i.e., activations of all components outside are replaced with some reference values). It is calculated as:
A high value indicates that alone implements much of the behaviour.
Specificity. Specificity measures the effect on non-target behaviours by interventions on . It is calculated as:
where contains the target behaviour together with a set of relevant control behaviours. A high value indicates that the causal effect of is specific to .
Evaluation protocol:
Necessity test: Ablate and measure the resulting degradation on behaviour . Compare against ablating random components of matched size.
Sufficiency test: Replace components outside with reference activations and measure how much of behaviour is preserved. Compare against random component sets of matched size.
Interchange test: Swap activations of between inputs representing different behaviours and test whether the model output follows the intervention.
Specificity test: Measure intervention effects on a set of non-target behaviours and examine whether the effect is specific to .
The MI community has developed several benchmarks that examine the causal validity of interpretability methods. For example, MIB (Mueller et al., 2025) provides standardised tasks for assessing discovered circuits and causal variables (e.g., features); AxBench (Wu et al., 2025b) evaluates methods’ ability to steer models towards given concepts; RAVEL (Huang et al., 2024) investigates methods’ ability to localise features of given concepts by using interchange interventions to validate discovered features; and SAEBench (Karvonen et al., 2025) evaluates SAEs specifically, pairing reconstruction metrics with downstream intervention tasks such as gender bias erasure.
Human Interpretability
Human interpretability measures whether generated explanations eventually improve human understanding, prediction, and decision-making about model behaviour.
Human Interpretability Metrics
Explanation Utility. This metric tests whether humans can predict model behaviour more accurately in the held-out set based on the explanation. It is formulated as:
Debugging Utility. This metric examines whether the explanations help humans identify and repair model errors, such as incorrect outputs or misaligned behaviour, in the wild.
Consistency of Interpretation. This metric inspects whether different humans interpret the same explanation consistently.
where is Cohen’s kappa or a similar agreement metric.
Evaluation protocol:
Prediction task: Show humans explanations of model components or mechanisms and ask them to predict model behaviour on held-out inputs. Compare accuracy to a no-explanation baseline.
Identification task: Show humans model outputs with errors and ask them to identify the cause, with and without explanation assistance. Measure accuracy and required time to complete the task.
Intervention task: Ask humans to propose fixes for model errors based on explanations. Measure the success rate of proposed interventions.
Agreement task: Ask multiple human raters to generate explanations, if applicable (e.g., write a summary of highly activated samples of an SAE feature as its explanation). Measure inter-rater reliability and agreement with machine-generated explanations.
Several caveats and challenges arise when evaluating the human interpretability of tested methods. For example, human studies are usually expensive. Moreover, human expertise may confound results, where some explanations understandable to MI researchers may be opaque to domain experts, and vice versa.
Note that different interpretability methods may emphasise distinct evaluation dimensions and operationalise them differently. Table 10 provides guidance on applying this framework to some common interpretability approaches.
| Method | Attribution Faithfulness | Completeness | Stability | Causal Validity | Human Interpretability |
| SAEs and variants | Feature attribution–intervention agreement; effect-size error | Activation reconstruction; loss recovery; unexplained variance | Cross-seed feature alignment | Feature necessity, sufficiency, and specificity | Feature description consistency across annotations, including models and humans |
| Circuit Discovery | Edge or component attribution–intervention agreement; counterfactual recovery | Behavioural coverage; loss or behaviour recovery | Robustness across hyperparameters, seeds, and paraphrases | Circuit necessity, sufficiency, specificity; interchange tests | Circuit-level descriptions; trade-off between sparsity and variance explained |
| Probing | Agreement between probe-derived importance and intervention effects | Coverage across target concepts or behaviours under different distributions | Robustness across seeds and sampled fitting sets | Necessity, specificity, and intervention tests on probe directions | Explanation of shared information in fitting set and search for confounding factors |
| Steering Vectors | Agreement between predicted and observed intervention effects | Coverage across behaviours deemed relevant | Stability across steering magnitudes, seeds, and paraphrases | Intervention efficacy and specificity; side-effect analysis | Explanation of shared information in fitting set and search for confounding factors |
Table 10. Recommended evaluation priorities by interpretability method. The entries indicate evaluation aspects particularly important for each method, not mutually exclusive metrics.
Integrated Evaluation Protocol
We recommend the following integrated protocol for comprehensive evaluation:
Recommended Evaluation Protocol
Step 1: Define scope. Users first specify the model, layer(s), and behaviour(s) of interest and define a behavioural test suite with clear success criteria.
Step 2: Establish baselines. Users specify baseline methods, such as random directions in activation space, PCA components, individual neurons, and supervised probes (if labels are available).
Step 3: Apply interpretability method. Users apply the method of interest to analyse models with variation in random seeds, hyperparameters, and other training details. Step 4: Evaluate across all five dimensions. Users compute metrics for attribution faithfulness, completeness, stability, causal validity, and human interpretability.
Step 5: Report comprehensively. Users report all five dimensions instead of just favourable metrics and discuss the method’s failure cases, unexplained behaviours, and comparisons to baselines.
Limitations of This Framework
This evaluation framework, though comprehensive, still has several limitations:
Metric choice: The proposed specific metrics are not uniquely correct. Alternative formalisations may be appropriate for specific contexts.
Threshold arbitrariness: Thresholds (e.g., the accuracy that counts as success) may lack theoretical grounding and will need empirical calibration.
Computational cost: Evaluating all five dimensions comprehensively is expensive and potentially prohibitive for large-scale studies.
Human evaluation bottleneck: Dimension 5 requires time-consuming, expensive, and subjective human studies.
Task dependence: A method’s performance under this framework can depend on the evaluation task. For example, a method that excels at factual recall may perform poorly at reasoning.
Despite these limitations, systematic evaluation across multiple dimensions provides more reliable evidence than an arbitrary choice of metrics. Readers should view the framework as a starting point for standardisation, not a final answer.
Summary. This section introduced circuit discovery methods, which trace how model components or features interact to produce behaviour, either directly on the model’s computational graph (e.g., ACDC and EAP) or within interpretable replacement models (e.g., Circuit Tracer). We also discussed how to causally verify discovered circuits and introduced a broader evaluation framework for mechanistic interpretability methods based on faithfulness, completeness, stability, causal validity, and human interpretability. In summary, § 6 introduced methods for extracting interpretable units of analysis, while this section examined how those units interact through circuits to produce behaviour. The next section takes a complementary perspective: directly interpreting the information in individual representations using natural language.
Understanding Intermediate Representations
This section explores interpreting hidden activations directly in natural language. In contrast to feature extraction and circuit discovery (§ 6 and § 7), which extract features or their interactions from model representations (activations), the methods we will discuss in this section treat the activation itself as the object of interpretation. They use language models or other natural-language-based methods to describe or elicit the encoded information, without intervening on or decomposing the underlying activation. This paradigm provides a direct interface between model activations and natural-language explanations, making it particularly accessible and understandable for human users.
Logit Lens: Tracing Predictions Through Depth
Logit Lens (Figure 10). Logit lens (nostalgebraist (2020)) applies the model’s unembedding matrix to activations at intermediate layers, revealing what the model would predict if computation stopped at that layer. For a hidden activation at layer , logit lens computes:
where represents the model’s unembedding matrix. Applying it across layers reveals the prediction trajectory, showing when and where the model commits to specific outputs (nostalgebraist (2020)).
Logit lens (Figure 10) provides intuitive insight into the model’s decision-making process across depth. Experimental results show that early layers tend to produce diffuse predictions, while later layers progressively narrow them towards the final output, though exceptions exist. For example, in a twelve-layer language model processing the sequence A B C A B, the logit lens may reveal that the token C enters the top-20 candidates at layer 6 and is progressively amplified, eventually becoming the top-ranked token at the final layer. Conversely, for induction tasks in complex linguistic patterns, early and middle layers may show a flat probability distribution across many tokens, with the model converging to a specific prediction only in the last few layers.
Figure 10. Logit Lens. Logit lens provides a window into a model’s intermediate state by applying the final unembedding layer directly to the residual stream at earlier layers (). This allows users to see which tokens the model is “considering” mid-computation, such as the emergence of the concept “CAT” before the final output layer is reached.
Limitations of Vanilla Logit Lens. The vanilla logit lens has a critical limitation: applying the unembedding matrix to intermediate activations assumes that the hidden state is in a format that the output layer expects. However, intermediate representations may be organised very differently from output-layer activations. Applying the unembedding matrix to layer-6 activations may extract relevant information but cannot precisely reflect the model’s underlying computation at that layer (nostalgebraist, 2020). Furthermore, logit lens is fundamentally correlational rather than causal. It projects the activation into the output vocabulary space without testing the roles decoded tokens play in model computation. In particular, because of the representational mismatch discussed above, decoded tokens represent at most information recoverable from the activation, rather than information used in model computation.
Tuned Lens: Learning Per-Layer Transforms
Tuned Lens. Tuned lens (Belrose et al., 2025) addresses logit lens’ concern of representational mismatch by learning per-layer affine transformations that map intermediate activations to the output layer’s activation space:
where and are learned affine transformations for layer . These transforms are trained to minimise the divergence between the tuned lens’ predicted distribution and the model output distribution via:
As a result, the unembedding matrix can extract relevant information from each layer’s activations more effectively.
Limitations of Tuned Lens. Although tuned lens mitigates representational mismatch, it still inherits several fundamental limitations. First, the decoding paradigm remains inherently correlational rather than causal; decodable information does not imply a causal role in the model’s prediction. Second, similar to logit lens, tuned lens is primarily designed to answer what the model is likely to predict rather than what the model is internally computing. The learned layer-specific transformations that align intermediate representations with the final prediction space may obscure intermediate computational steps and produce predictions that resemble the model’s final output. Empirical evidence also shows that it skips intermediate computation more aggressively than logit lens (Gurnee et al., 2026).
Jacobian Lens: Decoding Computational Dispositions
Jacobian Lens. Jacobian lens (J-lens; Gurnee et al., 2026) adopts a decoding paradigm distinct from logit lens and tuned lens. It uses the Jacobian to characterise how an intermediate representation influences certain model outputs.
Let denote the residual stream at layer and position . A perturbation to propagates to the final-layer residual stream at every position ; to first order, this influence is captured by the Jacobian
The lens matrix is defined as the expectation of this Jacobian over prompts and positions:
J-lens then decodes intermediate representations by
Gurnee et al. (2026) further defined layer-specific concept (token) representations using the rows of , called J-lens vectors. The inner product between a J-lens vector and measures how strongly the current activation will promote the corresponding token in model outputs.
Unlike logit lens and tuned lens, which primarily estimate which token the model is likely to predict next, J-lens characterises how current representations influence model outputs. It aims to answer what the model is internally computing rather than merely what the model is going to predict. J-lens has been found useful for analysing tasks that require multi-step reasoning, latent planning, or other forms of extended computation (Gurnee et al., 2026).
Furthermore, the defined J-lens vectors are a useful medium for intervention. They can serve as steering vectors to shift model behaviour towards or away from the corresponding concepts. In contrast, interventional utility is largely absent from logit lens and tuned lens, which are primarily decoding tools.
Limitations of Jacobian Lens. J-lens still faces several limitations that warrant further investigation. First, like logit lens and tuned lens, it operates primarily at the token level. However, many semantically meaningful concepts are represented by multiple tokens. For example, the word “photoperiodism” may be split into several subword tokens. If only one token (e.g., “photo”) is visible in J-lens, we cannot tell whether it corresponds to the concept of photoperiodism or to other words. Second, J-lens is found to work best within a certain range of layers, called J-space (Gurnee et al., 2026). Specifically, decoded concepts are most informative in intermediate layers (e.g., approximately 38–92% of the depth in Claude Sonnet 4.5). In early layers, decoded concepts are noisy and hard to interpret. In later layers, they collapse towards the model’s output and are also uninformative. Some recent studies have attempted to address these limitations (Gurnee et al., 2026; Blank et al., 2026).
Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations
Patchscopes (Figure 11).** Patchscopes (Ghandeharioun et al., 2024) leverages the expressive capacity of language models to translate their internal representations into natural-language explanations. Let denote a hidden representation at layer and token position . Patchscopes produces a textual explanation of , denoted as , by
where and specify the layer and token position to which the representation is patched, and is an elicitation prompt designed to encourage the language model to decode the patched representation into text expressing the contained information.
For example, consider the prompt illustrated in Figure 1 of Ghandeharioun et al. (2024):
where corresponds to the position of the token “?”. The notation indicates that, during the forward pass of on , the hidden representation at layer and position is replaced with . As a result, the model can reveal information encoded in through its prediction for the next token at the “?” position.
Because the mapping from intermediate representations to natural language may be too complex to be captured by an affine transformation, Patchscopes (Figure 11) extends the paradigm of logit lens and tuned lens by delegating the representation-to-text transformation to the language model itself1. In this view, the language model acts as an activation decoder. Crucially, Patchscopes relies on the model’s learned, generalisable text-generation capabilities to map internal activations to natural language.
Figure 11. Patchscopes. Patchscopes extracts the model representation of interest from a source prompt and “patches” it into an inspection prompt. By observing changes in model output, users can examine what information the representation contains. For example, in this figure we can observe whether the representation of “United Kingdom” correctly triggers “London” as the largest city.
Limitations of Patchscopes. A key limitation of Patchscopes is that injecting an external representation into an elicitation prompt can shift the model’s internal activations into an out-of-distribution (OOD) regime. This is expected, as the decoder is a standard pretrained language model not explicitly trained for representation-to-text decoding. Although the model may still surface salient information from the patched representation, the decoding process can degrade, and critical aspects of the original representation may be distorted or lost.
LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser
LatentQA (Figure 12)
LatentQA (Pan et al., 2026) frames representation decoding as a supervised learning problem via Latent Interpretation Tuning (LIT), fine-tuning an activation verbaliser to perform representation-to-text verbalisation. The core step of LIT is to construct a fine-tuning dataset consisting of samples , where is a cached hidden representation from a target model, and is a question–answer pair that queries the information encoded in .
Figure 12. LatentQA. LatentQA probes the semantic content of hidden representations by training a separate verbaliser model to answer natural-language questions about them. The process involves (1) curating a dataset of activation–question–answer triplets from a target LLM, and (2) fine-tuning a verbaliser to retrieve specific information latent within those activations, such as the model’s underlying goals or persona.
Specifically, denotes the activation at layer and token position when the target model processes a “control prompt” designed to embed a specific attribute or concept into the representation. An example control prompt from Figure 13 of Pan et al. (2026) is:
Figure 13. Probing Classifier. A probing classifier (or probe) tests the decodability of a concept from the target model’s internal representations. It is trained to distinguish activations associated with positive and negative examples of the concept. High probe accuracy indicates that the concept can be reliably decoded from the representation.
“In your next reply, please try to evoke a brief sense of triumphant hubris.”
If is defined as the activation at layer corresponding to the final token (“.”) of this prompt, the resulting representation is expected to encode the instruction to respond with triumphant hubris. In this case, a corresponding question–answer pair may be:
= “In what way does the assistant craft its responses to the user?”
= “The assistant crafts responses that briefly highlight a sense of self-assured victory.”
Both the control prompt and the pair are generated by prompting an off-the-shelf proprietary language model (Pan et al., 2026).
LatentQA (Figure 12) addresses the OOD issue inherent in Patchscopes-style methods by explicitly fine-tuning the activation verbaliser on the representation-to-text decoding task. Through LIT, the verbaliser learns to ingest and process raw activation vectors directly. Ideally, after being exposed to a sufficiently diverse collection of question–answer pairs over many different representations, the decoder learns which aspects of the activation to attend to and how to verbalise them conditioned on the query.
Limitations of LatentQA. LatentQA faces several technical challenges, including the generalisability of the learned representation-to-text mapping. When the fine-tuning dataset is too small or insufficiently diverse, the verbaliser may memorise superficial correspondences between representations and explanations rather than learning to extract information from representations. Recent approaches, such as Predictive Concept Decoders (PCDs) (Huang et al., 2026) and Activation Oracles (AOs) (Karvonen et al., 2026a), mitigate this issue by constructing more diverse training datasets.
Another technical challenge is evaluating whether activation-verbalisation models genuinely explain the represented information rather than exploit shortcuts. For example, a verbaliser may implicitly reconstruct the original input to the target model and infer an explanation from it. Recent studies have found that such failure modes are severe and prevalent in existing verbalisers (Li et al., 2026c; Jakkli et al., 2026). As a result, strong verbalisation performance alone is insufficient to show that the verbaliser faithfully analysed the internal representations.
Probing Classifiers: Capabilities, Critiques, and Causal Limitations
Probing classifiers (Figure 13) are among the most widely used tools for analysing neural network representations. A probe is a classifier trained to predict a property of the input, such as part of speech, sentiment, or entity type, from model activations. High probe accuracy means that the property is decodable from the activations. However, information decodability is a relatively coarse-grained, correlational interpretability claim compared with causal claims about information in model computation. This section reviews probing methodology, its major critiques, and its appropriate role in interpretability research.
As a side note, discovered probe directions can also be used as supervised feature extractors. We discuss this perspective in § 6.9.4, including comparisons with SAEs, and in Section 9.3, where we discuss their use as steering vectors.
Standard Probing Methodology
Probing Classifier (Figure 13)
A probing classifier is a supervised model trained to predict a target property from neural network activations at layer . A linear probe follows the form:
where and are learned parameters. , typically an identity function for regression and softmax for classification, maps the probe’s raw output to the prediction space. The probe is trained by minimising prediction error on labelled examples . The linear probe tests whether the target property is linearly accessible from the representation. MLP probes introduce one or more nonlinear hidden layers instead of to test whether the property can be retrieved through more expressive nonlinear mappings.
Probing provides evidence about whether activations contain information about a property. High probe accuracy indicates the information is present; low accuracy suggests it is absent or the probe is not expressive enough. Note that probing alone does not show whether the model actually depends on that information in its decision-making process. A representation can encode information that downstream computation never accesses, such as “leftover” information from earlier processing.
The Probe Causality Problem
High probe accuracy does not imply causal relevance (Belinkov, 2022; Hewitt & Liang, 2019). The probe can work by detecting features spuriously correlated with the target property (e.g., word frequency correlates with part of speech) rather than detecting the target property itself. Furthermore, subsequent layers may never use the decoded property to influence the model output. Finally, a sufficiently expressive probe can achieve high accuracy by exploiting noise even when the representations contain no such information.
In summary, using probing alone generally does not provide actionable mechanistic understanding and therefore does not, under our definition, constitute mechanistic interpretability research. However, subsequent work has proposed refined probing paradigms that explicitly incorporate causal considerations into probe design and evaluation, enabling more actionable mechanistic understanding (Elazar et al., 2021); see § 10.6.4 for details. Recent interpretability studies have similarly used causal interventions to validate probe-discovered concept directions, building a more complete mechanistic explanation (Marks & Tegmark, 2024; Li et al., 2023).
Control Tasks and Selectivity
High probe accuracy may reflect classification capacity rather than the target property’s decodability. Hewitt & Liang (2019) addressed this concern by introducing control tasks as a baseline for probe performance.
Control Task Methodology
A control task assigns random labels to inputs to break any genuine relationship between activations and the target property. A probe trained on the control task represents the baseline extractability of random patterns from the representation. Then, the selectivity of a probe is defined as:
where is probe accuracy on the target task and is accuracy on the control task. High selectivity indicates that the probe succeeds because the representation genuinely encodes the target property. In contrast, low selectivity suggests we cannot tell whether the probe has exploited spurious features rather than task-relevant structure.
Concerns of Probe Complexity
Probe complexity brings confounding factors to the analysis. If a property can only be extracted with a highly nonlinear probe, it is hard to tell whether the property is “deeply encoded” or whether the probe is simply using nonlinearity to exploit signals, including noise. A recent finding in causal abstraction (Sutter et al. (2025)) also showed that, when the mapping function is complex enough, it can map an arbitrary (even randomly initialised) neural network to a causal model of interest, producing false-positive findings.
The Complexity–Interpretability Trade-off
Observation. In general, more complex probes achieve higher accuracy but provide less interpretable deliverables.
Interpretation problem. Does high accuracy with a complex probe indicate (a) the property is encoded but in a complex, nonlinear form, or (b) the probe itself is computing the property from spurious features? If (b), the probe at most demonstrates that the property is computable from the representation, not that it is encoded in the representation in any meaningful sense. A worse case in (b) is that a highly nonlinear probe may learn to construct the property of interest from random noise rather than any features (Sutter et al. (2025)). In this case, the computable illusion likely stems from limited test-set size.
Minimum Description Length Probes. To address this, Voita & Titov (2020) proposed evaluating probes via the minimum description length (MDL) principle: prefer probes that achieve good accuracy with minimal complexity. The proposed MDL-based evaluation offers an information-theoretic measure of how efficiently a probe can extract a target property from model representations, accounting for both probe accuracy and complexity. A property is considered “more encoded” if it can be extracted with a simpler probe. This paradigm shifts the question from “can we extract it?” to “how easily can we extract it?”
Practical Guidance. Users should prefer simple probes when possible, compare against probes trained on random representations to establish baselines, and consider MDL-based evaluation to analyse the trade-off between accuracy and model complexity. If linear probes fail but MLP probes succeed, cautiously validate the result and interpretability claim.
Amnesic Probing and Causal Probing
Beyond using probes to validate information decodability, researchers have also formalised the idea of intervening on representations via probes. Giulianelli et al. (2018) first showed that a trained probe can steer model behaviour, and Ravfogel et al. (2020) extended this with Iterative Nullspace Projection (INLP), an information-removal method that repeatedly fits a probe to identify linearly encoded information and projects it out. Elazar et al. (2021) further measured changes in downstream model behaviour after applying INLP as a validation of the probe’s causality, called amnesic probing. These paradigms add causality to probes, pushing them from correlational understanding to mechanistic analysis. Below, we detail amnesic probing and probing with activation patching.
Amnesic Probing.
Amnesic Probing. Given a model activation and a probe trained to predict a property , amnesic probing tests whether information about is causally relevant to downstream model behaviour by removing information about from the representation and evaluating the effects on model outputs. Assume is a binary property (e.g., positive vs. negative sentiment) and that denotes the probe’s direction. Amnesic probing projects onto the subspace orthogonal to a probe direction (i.e., the null space of the probe):
Amnesic probing then measures the change in model performance on downstream tasks with the modified representation . A substantial degradation indicates that the removed information causally contributes to the evaluated behaviour. Notably, unchanged performance does not establish that the information is unused since redundant representations or alternative computational pathways may preserve the behaviour.
Amnesic probing assumes the target property is decodable and that there exist effective erasure procedures to surgically remove its associated information from the representation. However, information and concept erasure may be imperfect in practice, either leaving residual information or affecting other concepts. Such imperfections confound interpretations of downstream behavioural changes and the causal claims that follow.
Causal Probing via Activation Patching. Activation patching tests the causality of a component’s representations by swapping them between inputs that differ in the target property. This technique has also been used to test the causality of probes [21] (Huang et al. 2024). Suppose and are two inputs, where has the target property and does not. is the fitted probing matrix whose rows span a probing subspace (or a direction if the matrix has only one row, such as that of a binary classification probe) where resides. After choosing the intervention location (e.g., the last-token residual stream at a middle layer), the method first feeds to the model and extracts the component of the representation in the probing subspace. Then, it feeds to the model while patching the corresponding component of the model representation with the previously acquired one. If the model output for shifts towards , the patched subspace representation contains information about that the model uses.
Compared with amnesic probing, which tests the effect of removing information about , activation patching examines whether the probe-identified information is sufficient to implement the target behaviour. However, this requires specifying what representation (i.e., which layer, component, and token position) to probe and patch, which may confound the analysis.
Implications of Attribution Faithfulness for Probing
Probing is often used along with attribution methods (e.g., saliency maps, attention visualisation, gradient-based attribution) to understand model behaviour. However, attribution methods also have technical challenges. For example, gradient-based saliency maps can highlight input features the model is actually insensitive to (Adabeyo et al., 2018), attention weights correlate weakly with causal importance (Jain & Wallace, 2019), and different attribution methods can disagree substantially on the same inputs (Krishna et al., 2024). These findings suggest that attribution methods alone cannot determine what information a model actually uses.
This unreliability affects how we can interpret probing results. Probes trained on attention patterns may not reveal causally important information since attention itself may not be causal. Similarly, using gradients to identify which features or components to probe can bias towards high-magnitude gradients that lack causal importance. In sum, interpretability claims made by probes, even paired with these auxiliary tools, still need to go through causal validation to separate plausible explanations (do attributions seem reasonable to humans?) from faithful explanations (do attributions reflect actual model computation?) ((Jacovi & Goldberg, 2020)).
When Is Probing Appropriate?
Given the preceding discussion, Table 11 summarises appropriate uses of probing. We argue that probing is most productive when treated as a tool for hypothesis generation, that is, using probes to identify candidate properties that may be encoded in a representation and potentially causally relevant. Users then test these hypotheses using causal methods, such as activation steering or patching. In this sense, probing serves as an efficient first step in mechanistic investigation rather than as a decisive explanation.
| Appropriate Uses | Inappropriate Uses |
| Establishing information presence (i.e., “ is encoded”) | Claiming causal relevance (i.e., “the model uses ”) |
| Comparing encoding strength across layers | Concluding that a property is not encoded based on low probe accuracy |
| Auditing for encoded sensitive information | Claiming mechanistic understanding from probes alone |
| Generating hypotheses for causal investigation | Using probing results to support hypotheses about causality |
| Early detection or triage before further analysis | Evaluating alignment or safety based on probing results |
Table 11. Appropriate and inappropriate uses of probing.
Methodological Standards in Probing Research
So far we have reviewed several strengths and limitations of probes. Based on them, we summarise several methodological standards for the design and interpretation of probing studies: (1) Report control-task baselines to distinguish genuine decodability from spurious exploitation; (2) Use minimally expressive probes unless additional complexity is justified; (3) Test causal relevance by methods like amnesic probing or activation patching; (4) Analyse layer-wise trends besides the peak accuracy; (5) Distinguish encoding from causality and acknowledge that successful probing shows information presence rather than causal use; (6) Report the stability of findings by evaluating across seeds and data splits. In sum, we believe these standards help ensure responsible, effective use of probing in interpretability research and downstream applications.
Probing Classifiers for Sleeper Agents
Despite the causality critiques, probing remains valuable for many applications. For example, MacDiarmid et al. (2024) showed that simple linear probes trained on residual-stream activations can detect latent backdoor behaviour in the sleeper agent model even when the probe fitting set does not contain the trigger or the harmful behaviour.
To recap: a sleeper agent model behaves normally in one context (e.g., ) but exhibits a hidden dangerous policy when a trigger is present (e.g., ) ((MacDiarmid et al., 2024)).
To detect this latent behaviour, MacDiarmid et al. (2024) constructed a linear detection detector by computing a detector direction at layer . Specifically, let denote the residual activations at the final token position for input . Given a small labelled fit dataset of prompts, the detector direction is defined as:
where and are contrast pairs of prompts. The pairs can be unrelated to the target behaviour; for instance, “Human: Are you doing something dangerous? Assistant: yes” and “Human: . . . Assistant: no” (example borrowed from MacDiarmid et al. (2024)). During the evaluation stage, activations are projected onto this direction to yield a detector activation value . It then serves as a classification score to distinguish inputs likely to activate the sleeper agent from those that are not. Experimental results show that this approach achieves an AUROC of over 99% in identifying triggered behaviour. MacDiarmid et al. (2024) argued that this is because the activation differences between triggered and non-triggered inputs are sufficiently salient (for example, two clusters far apart). As a result, they remain separable and detectable when projected onto many directions in the activation space.
Why this works despite the causality critiques: In this use case, the goal is detection (information presence), not mechanistic understanding. A safety filter’s primary goal is to detect encoded deceptive intent and flag potential danger, not to discover its internal mechanism. In summary, this use case shows that probing remains valuable for auditing or detection even though it cannot support causal claims.
Summary. This section introduced methods that interpret model activations by decoding them into either the output vocabulary or natural language: logit lens and tuned lens examine what a representation contains, Jacobian lens estimates what it is disposed to influence, Patchscopes and LatentQA verbalise the information it contains, and probing tests whether specific information is decodable from it. These methods primarily reveal what can be decoded from a representation but not whether the decoded information is subsequently used by the model. Section 9 broadens the discussion to comparing representations across models, locating where information is stored, and intervening on representations to control model behaviour.
Representation Analysis: Comparison, Localisation, and Control
Following Section 8’s emphasis on understanding representations at a broader level, this section considers three successive questions about model representations. Model diffing (Section 9.1) asks what differs between two models’ representations. Knowledge localisation (Section 9.2) asks where a particular piece of information or behaviour is most causally relevant. Representation engineering and steering (Section 9.3) asks whether mechanistic understanding of model representations can be used to change model behaviour.
These methods are also closely connected to neighbouring sections. Diffing two checkpoints of the same model can be viewed as a specific instance of the longitudinal analysis discussed in Section 10.1. Knowledge localisation operates at a coarser level of analysis than the feature-extraction methods in Section 6, but pursues a stricter causal objective by identifying the components at which particular behaviours can be most effectively intervened upon. Finally, representation steering causally tests mechanistic findings by examining whether intervening on an identified representation produces the predicted behavioural change.
Model Diffing
Many concerning behaviours, such as reward hacking or sycophancy, arise during fine-tuning (Sharma et al., 2024b; MacDiarmid et al., 2025). This phenomenon makes it important to monitor features and circuits introduced or amplified during this process to prevent unsafe models from being deployed. Model diffing ([13]) can surface some of these features.
Model Diffing
For two models, model diffing seeks to identify differences in features or circuits between them. These differences may consist of features or circuits that are exclusive to one model, or of differences in how shared features or circuits are used.
Representation Alignment and Convergent Learning. An earlier line of work compared internal representations across models using correlation- and mutual information-based measures (Li et al., 2015), canonical-correlation methods such as SVCCA and PWCCA (Raghu et al., 2017; Morcos et al., 2018), and kernel-based alignment metrics such as CKA (Kornblith et al., 2019). Related approaches examine whether the representations of one model can be expressed as linear combinations of representations in another (Wang et al., 2018). This literature primarily concerns convergent learning: whether independently trained models with different random initialisations learn similar representations up to permutation, rotation, or other linear transformations. More recent work has extended these techniques to study representational convergence across modalities (Huh et al., 2024; Gröger et al., 2026), as discussed in § 5.4. In contrast to model diffing as defined here, these methods typically focus on representational similarity rather than on identifying concrete features or circuits that are introduced, amplified, or repurposed by a specific training intervention.
Applications. Model diffing has been applied to assess the effect of fine-tuning [18] (Dumas et al. 2025a; Lindsey et al. 2024; Wang et al. 2026a) and distillation (Baek & Tegmark, 2025), to track differences between model architectures (Thasarathan et al., 2025; Jiralerspong & Bricken, 2025), to study the evolution of concepts during pretraining (Bayazit et al., 2026; Ge et al., 2026), and to identify vision-specific features introduced during vision-language training, enabling targeted removal or steering (Batra et al., 2026).
Dictionary Learning Methods. Dictionary learning is an intuitive approach to implement model diffing. It can highlight unexpected, newly introduced features of concern. When diffing models of the same architecture, such as a base versus instruction-fine-tuned model or a series of model checkpoints along pretraining, dictionary learning methods like sparse autoencoders (SAEs) and transcoders can be trained on the difference in token activations (Dumas et al., 2025b; Sawmya et al., 2025; Inaba et al., 2025). This difference can capture features introduced by fine-tuning or by different pretraining stages. A simple baseline technique is to train an SAE on the base model alone and use it as a fixed feature basis for comparison. Methods like SAE-Track maintain continuity through recurrent initialisation, training each checkpoint’s SAE from its predecessor’s so latents remain matched across the run (Xu et al., 2025b). The differences in activations between two models are then measured on a shared evaluation prompt set (Wang et al., 2026a). By causally intervening with latents that activate more strongly in the fine-tuned model, it is possible to identify features responsible for misaligned behaviour (Wang et al., 2026a).
One challenge for model-diffing approaches is attributing features in a fine-tuned model’s representations to the fine-tuning itself. To address this, an SAE trained on a base model can be fine-tuned stage-wise (Bricken et al., 2024): first on the base model with fine-tuning data, then on the fine-tuned model with base data, and finally on the fine-tuned model with fine-tuning data. Only features exclusive to the last setting are likely to have been introduced via fine-tuning.
SAEs are applicable when inputs from different models have the same dimension. When diffing two models with different architectures, for example, to track dispositions across model generations or sizes, this might not hold. In this case, crosscoders offer an alternative (Figure 14). Crosscoders can define model-specific encoders with arbitrarily different input dimensions that project activations from different models into one or more shared representational spaces. Although the main advantage of crosscoders over SAEs is cross-architecture comparison (Thasarathan et al., 2025; Jiralerspong & Bricken, 2025), some prior work has also applied them to base and fine-tuned models (Lindsey et al., 2024; Bricken et al., 2024; Dumas et al., 2025a) and across pretraining checkpoints (Ge et al., 2026; Bayazit et al., 2026).
A related approach is to model differences in how representations are transformed, rather than comparing the representations themselves. Hu et al. (2026) used transcoders to learn a function that approximates the difference in MLP computation between a base model and its fine-tuned variant. Specifically, adapters are trained so that a base MLP augmented with a learned correction term reproduces the fine-tuned MLP’s behaviour within the full model. Unlike the above methods, which operate on activation differences, this method learns a transformation that approximates how fine-tuning modifies the model’s MLP computation.
A shared concern across dictionary-learning-based model-diffing methods is the faithfulness of the sparse decomposition (see § 6.5.5). Extracted features may not faithfully reflect the underlying representations or their causal roles, potentially leading to false-positive findings about differences between models. For example, identifying an SAE feature associated with “confessing previous faults” in a fine-tuned model but not in the original does not establish that the fine-tuned model causally uses this feature. Nor does its apparent absence from the original model imply that the model lacks representations that could steer it towards such behaviour.
Figure 14. Model diffing via crosscoders. Model diffing identifies features that emerge or remain invariant across model iterations or architectures. A large delta () in feature activation between two models highlights a changed feature, providing an insight into how fine-tuning or architectural changes alter internal representations.
False Positives in Model Diffing
Dictionary learning methods for model diffing risk falsely identifying model-exclusive features (Dumas et al., 2025a; Jiralerspong & Bricken, 2025). This can result from the loss function: for instance, Dumas et al. (2025a) showed that crosscoders trained with an regularisation on hidden states can falsely identify novel features, which they tried to mitigate with . However, even with this improved architecture, false positives may persist. During fine-tuning, the directions representing some features can rotate, and dictionary learning methods currently cannot distinguish between features introduced via fine-tuning and those merely represented differently. Thus, supposedly model-exclusive features should be regarded as hypotheses that need further validation.
LLM-Based Approaches. To differentiate closed-weight models, it is desirable to develop model diffing tools that only rely on API access. To that end, Kempf et al. (2026) showed that an LLM can generate natural-language hypotheses about behavioural differences between two models by extracting, clustering, and summarising differences across many response pairs from the two models on a shared prompt set. Benchmarked against the SAE-based baseline, the LLM-based method matches it on many dimensions while surfacing more abstract, behaviour-level differences.
Knowledge Localisation
Knowledge localisation methods aim to identify the layers, components, or subspaces whose activations causally relate to a particular capability or behaviour. This practice draws on the hypothesis that certain information concentrates in specific model locations. For example, Geva et al. (2021) proposed that feed-forward layers can mechanistically act as key–value memories; Dai et al. (2022) found that factual knowledge associations are stored in them; and Sharma et al. (2024c) discovered that the information of these associations mainly flows through the subspace spanned by the top singular directions of late-layer MLPs’ weight matrices. These analyses laid the groundwork for knowledge localisation and motivated corresponding techniques (Meng et al., 2022).
Localisation can identify the components where a behaviour or piece of knowledge is most decodable or most causally influential. It has been a standard first step for many intervention-based analyses, such as steering vectors (Section 9.3), probes (§ 10.6), and knowledge editing (§ 15.7.1). However, localisation alone does not reveal how the relevant information is computed or represented. In transformers, which are usually overparameterised, information can be redundantly represented and processed across many model components and activation subspaces depending on the input. A component with a strong causal effect may capture only a small part of the mechanism. As a result, localisation is a useful starting point for mechanistic understanding, but not a self-contained analysis.
For steering vectors and probes, localisation usually means finding the most effective layer or token position to steer or probe model activations. For example, Arditi et al. [20] searched over layers and post-instruction token positions to identify the most performant difference-in-means vector for steering the model towards or away from refusal behaviour. Similarly, MacDiarmid et al. (2024) studied the sleeper-agent behaviour (Hubinger et al., 2024) by comparing and probing activations across layers for triggered and non-triggered inputs. In knowledge editing, localisation often means identifying the MLP blocks that stores the knowledge of interest to modify. A classical technique is causal tracing (Meng et al., 2022), which identifies components that causally affect specific factual association retrieval by observing how model outputs change after activation intervention, and is used in editing methods like ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), and AlphaEdit (Fang et al., 2025). In sum, this localisation practice, along with intervention methods, has become an established pipeline for answering where and how the model encodes behaviour- or knowledge-related information. However, identifying a model module, layer, or activation subspace does not explain the internal mechanisms underlying the behaviour, which can only be understood through finer-grained analyses such as circuit discovery.
Representation Engineering and Steering
To manipulate model behaviour, representation engineering alters activation-space representations directly, rather than via model weight modification. Activation steering generally refers to methods that target intermediate activations (usually the residual stream) by adding certain vectors that shift model outputs during inference. This differs from localisation methods, which determine where information or behaviour is represented, as steering techniques aim to identify a specific, manipulable vector that changes model behaviour. Because they require no weight updates or retraining, these interventions are lightweight approaches for controlling behaviour.
Steering Vector (Figure 15)
Figure 15. Steering Vectors. A steering vector can be constructed from the difference between the mean activations of positive and negative contrastive samples at a target layer. During inference, the vector is added to the original hidden state to obtain a modified state .
A steering vector is an activation-space direction which, when scaled and added to an intermediate activation, shifts model outputs towards a targeted behaviour. For instance, one steering direction can use the weight vector of a trained linear probe. Another popular method is difference-in-means, which obtains the steering vector by taking the difference between activations associated with two behaviours (Rimsky et al., 2024):
@page 80/183 starts_mid=0 ends_mid=1
Knowledge Localisation
Knowledge localisation methods aim to identify the layers, components, or subspaces whose activations causally relate to a particular capability or behaviour. This practice draws on the hypothesis that certain information concentrates in specific model locations. For example, Geva et al. (2021) proposed that feed-forward layers can mechanistically act as key–value memories; Dai et al. (2022) found that factual knowledge associations are stored in them; and Sharma et al. (2024c) discovered that the information of these associations mainly flows through the subspace spanned by the top singular directions of late-layer MLPs’ weight matrices. These analyses laid the groundwork for knowledge localisation and motivated corresponding techniques (Meng et al., 2022).
Localisation can identify the components where a behaviour or piece of knowledge is most decodable or most causally influential. It has been a standard first step for many intervention-based analyses, such as steering vectors (Section 9.3), probes (§ 10.6), and knowledge editing (§ 15.7.1). However, localisation alone does not reveal how the relevant information is computed or represented. In transformers, which are usually overparameterised, information can be redundantly represented and processed across many model components and activation subspaces depending on the input. A component with a strong causal effect may capture only a small part of the mechanism. As a result, localisation is a useful starting point for mechanistic understanding, but not a self-contained analysis.
For steering vectors and probes, localisation usually means finding the most effective layer or token position to steer or probe model activations. For example, Arditi et al. [20] searched over layers and post-instruction token positions to identify the most performant difference-in-means vector for steering the model towards or away from refusal behaviour. Similarly, MacDiarmid et al. (2024) studied the sleeper-agent behaviour (Hubinger et al., 2024) by comparing and probing activations across layers for triggered and non-triggered inputs. In knowledge editing, localisation often means identifying the MLP blocks that stores the knowledge of interest to modify. A classical technique is causal tracing (Meng et al., 2022), which identifies components that causally affect specific factual association retrieval by observing how model outputs change after activation intervention, and is used in editing methods like ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), and AlphaEdit (Fang et al., 2025). In sum, this localisation practice, along with intervention methods, has become an established pipeline for answering where and how the model encodes behaviour- or knowledge-related information. However, identifying a model module, layer, or activation subspace does not explain the internal mechanisms underlying the behaviour, which can only be understood through finer-grained analyses such as circuit discovery.
Representation Engineering and Steering
To manipulate model behaviour, representation engineering alters activation-space representations directly, rather than via model weight modification. Activation steering generally refers to methods that target intermediate activations (usually the residual stream) by adding certain vectors that shift model outputs during inference. This differs from localisation methods, which determine where information or behaviour is represented, as steering techniques aim to identify a specific, manipulable vector that changes model behaviour. Because they require no weight updates or retraining, these interventions are lightweight approaches for controlling behaviour.
Steering Vector (Figure 15)
A steering vector is an activation-space direction which, when scaled and added to an intermediate activation, shifts model outputs towards a targeted behaviour. For instance, one steering direction can use the weight vector of a trained linear probe. Another popular method is difference-in-means, which obtains the steering vector by taking the difference between activations associated with two behaviours (Rimsky et al., 2024):
During inference, the intervention takes the form
where controls the intervention strength.
While these approaches appear simple, they reliably change model behaviours such as refusal, helpfulness, and stylistic tendencies ([20] (Rimsky et al. 2024; Zhao et al. 2025)). Thus, steering is useful both as a technique to manipulate behaviour and as a causality test for probes. If intervening along a direction consistently changes a target behaviour, this provides evidence that the corresponding representation causally influences that behaviour. Below we use refusal vectors as a concrete example.
Refusal Vector
A refusal vector ([20]) is a steering direction associated with a model’s tendency to refuse requests. Define as the residual stream at layer and token position . A simple contrastive construction is
where contains prompts eliciting refusal behaviour and contains matched prompts for which the model should comply.
The projection
measures the activation along the refusal direction. When intervening on this direction, one can test if and how refusal behaviour is causally influenced by it. Previous experimental work shows that these interventions can strongly modulate refusal behaviour; however, the construction and interpretation of refusal directions can vary across models and experimental settings ([20]).
The contrast between localisation and steering is shown by refusal vectors: measuring across layers helps locate representations that contain information related to refusal, while intervening along tests how modifying representations modulates behaviour (which provides evidence for causality, but does not necessarily reveal the entire mechanism behind how refusal works).
Nonlinear and Optimisation-Based Steering. Beyond linear directions, recent studies have explored more complex ways of manipulating model activations. For example, Raval et al. (2026) projected model activations into a kernel PCA space, steered them linearly there, and projected them back to the original activation space; Wurgaft et al. (2026) steered activations along a low-dimensional manifold, motivated by findings that some concepts are better represented as manifolds in the activation space. Separately, optimisation-based methods construct steering directions by directly optimising objectives over intermediate activations, offering an alternative to closed-form constructions. For example, they can be obtained by maximising task performance or by suppressing undesirable outputs under intervention. These optimisation-based approaches offer greater flexibility for complex or nonlinear objectives but generally require substantially more computation than closed-form contrastive methods ((Dunefsky & Cohan, 2025; Oozeer et al., 2025)).
Steering as a Causal Test for Feature Interpretability. Activation steering can causally evaluate representations previously labelled by feature-extraction methods (such as CLTs or SAEs). For instance, a feature may be hypothesised to represent a particular concept. Supporting evidence can be obtained by steering along the identified feature direction and evaluating the resulting change in behaviour.
These evaluations have been used to compare interpretable direction-identification approaches, including SAE features, difference-in-means directions, and DAS ((Huang et al., 2024; Wu et al., 2025b)). Benchmarks designed to evaluate the causal and behavioural properties of learned representations include RAVEL (Huang et al., 2024), MIB (Mueller et al., 2025), and AxBench (Wu et al., 2025b).
Limitations: Brittleness and Mechanistic Opacity
While steering methods effectively control behaviours and support causal testing, they have limitations. Notably, they can be brittle, as a direction identified for one distribution, model, or task may not generalise reliably. Another limitation is that successful behavioural steering does not, by itself, establish mechanistic transparency, because an intervention may simultaneously impact multiple features, interact nonlinearly with existing computations, or exploit unintended pathways that produce the desired behaviour without corresponding to a clean mechanism (Makelov et al., 2024). Thus, steering should generally be interpreted as evidence for causal influence, but it should not, by itself, be taken as a mechanistic explanation of the behaviour.
Summary. Model diffing examines representation differences across models, knowledge localisation pinpoints where information or behaviours are causally localised, and representation engineering analyses how representation manipulation controls behaviour. Steering has two important roles: as a control technique and as a way to causally test how representations affect behaviour (while noting that successful localisation or steering does not, by itself, provide a complete explanation for the mechanism underlying that behaviour). This analysis is extended in § 12, which examines how representations and mechanisms evolve during training.
Learning Dynamics and Mechanistic Interpretability
Most interpretability research treats a trained model as static. It asks what the model activations represent, which components implement a behaviour, or how an intervention changes its output at a particular checkpoint. However, a model does not come into the world fully formed—it is the product of a learning process over time. Recently, public model suites with intermediate checkpoints ((Biderman et al., 2023b; Groeneveld et al., 2024}) and transparent training processes, such as Pythia (Biderman et al., 2023b) trained on the publicly available Pile corpus (Gao et al., 2020), have enabled systematic study of these training trajectories, unlocking unique insights into how models form behaviours.
Research at the intersection of training dynamics and interpretability can have two aims. Longitudinal analysis (§ 12.1) treats learning dynamics as an object of explanation, asking how and why mechanisms form, change, stabilise, or disappear over the course of training (Saphra, 2023). Training dynamics as a tool instead uses temporal variation to answer questions about the model itself. For example, researchers can compare changes across checkpoints to identify certain mechanisms, distinguish competing explanations, predict later capabilities or failure modes, and guide interventions on the training process.
Longitudinal Analysis
Saphra (2023) argued that we can fully understand model internals only in light of their development: an analysis confined to the final checkpoint risks misattributing the function or origin of the structures it finds there. This subsection covers what longitudinal studies have established about the relationship between training data and the capabilities models acquire (§ 12.1.1) and about the lifecycle of internal features (§ 12.1.2); abrupt formation events are treated separately in § 12.3.
Training Data and Capability Acquisition
Acquisition Order. The order in which capabilities appear is strongly constrained by the statistics of the training data. Language models acquire words in an order predicted primarily by corpus frequency, unlike children, whose acquisition order also depends on factors such as concreteness and word length (Chang & Bergen, 2022); graded linguistic preferences, such as adjective order, are learned from corpus statistics on a similar schedule (Jumelet et al., 2026). This ordering reflects a more general distributional simplicity bias:

Figure 1 (PDF p. 83). Distributional Simplicity Bias
Networks trained with SGD learn low-order statistics of the data distribution before higher-order ones (Refinetti et al., 2023). In language models, token n-gram frequencies are equivalent to moments of the embedding vectors, so early-training language models approximate low-order token interactions well, with more complex structure fitting later (Belrose et al., 2024).
From Correlation to Causation. Correlational studies show a strong connection between data statistics and capabilities of the final model (Razeghi et al., 2022), and causal effects of pretraining-data statistics on factual predictions can be estimated directly (Elazar et al., 2023). The strongest evidence would come from controlled manipulation of the training data, although this is typically not done at realistic scales because it is costly. One notable example of such work is O’Brien et al. (2026), which trained three 6.9-billion-parameter, 550-billion-token models on different dataset filtrations and showed they can remove knowledge of proxies for biohazardous capabilities.
By training transformers on synthetic sequence tasks, Chan et al. (2022b) showed that in-context learning emerges only when the data has the right distributional properties, with burstiness and a long tail of rare classes each necessary. A uniform, non-bursty version of the same content instead leads to in-weights learning, while Zipfian class distributions allow the coexistence of the two learning patterns. Similar dependence on distributional structure governs factual knowledge acquisition, which proceeds through an extended plateau shaped by the frequency distribution of entities in the data (Zucchet et al., 2025a).
Feature Formation, Consolidation, and Drift
A technical challenge in tracking features across training runs is that representations at different checkpoints may occupy different bases, so a feature identified at one checkpoint has no automatic counterpart at another. This is analogous to the cross-layer basis mismatch of the logit lens, which degrades because intermediate layers organise information differently from the final layer. A probe or sparse autoencoder trained at one checkpoint degrades similarly when applied to another, and longitudinal methods differ mainly in how they solve this correspondence problem.
In § 11.1, we discuss model diffing methods for comparing and analysing two models’ internals. They are useful tools for longitudinal analysis, and they have discovered a broadly consistent order in which model knowledge forms and internal representations evolve. Specifically, linguistic knowledge is acquired early and quickly, while factual and commonsense knowledge accumulates slowly and reasoning abilities remain weak throughout pretraining (Liu et al., 2021). In particular, feature-level analyses provide fine-grained descriptions of this picture: features corresponding to token-level and surface statistics form first, with abstract, cross-lingual, and semantic features forming later (Inaba et al., 2025; Ge et al., 2026), and individual features develop from noisy, unstructured activations into semantically coherent ones (Xu et al., 2025b). Formation is followed by consolidation: after their semantics are established, feature directions continue to drift and eventually stabilise (Xu et al., 2025b); modules converge at different rates (Raghu et al., 2017); and once formed, representations and their acquisition timing are largely consistent across random seeds (van der Wal et al., 2025).
Stability of a capability does not imply stability of its implementation. The components carrying an algorithm can change identity while the algorithm persists (Tigges et al., 2024), and mechanisms can be repurposed, as when induction heads transition into the function vector heads that come to dominate few-shot in-context learning (Yin & Steinhardt, 2025). Structures can also disappear outright: in the synthetic settings of Chan et al. (2022b), emergent in-context learning is transient, arising and then giving way to in-weights learning while training loss still falls, with regularisation affecting its persistence (Singh et al., 2023). Similarly, the drift of features can produce false positives in model diffing (§ 11.1) and more broadly bound the temporal scope of any mechanistic claim (§ 12.2.2): an explanation about model internals validated at one checkpoint may not apply to other checkpoints of the same model.
Learning Dynamics as an Interpretability Tool
The preceding subsection treated the learning trajectory as an object to be explained. A complementary line of work uses that trajectory instead as a tool for analysing model behaviour. Comparing behaviour and components across model checkpoints lets researchers conduct novel causal analyses of how components drive model behaviour. Although this framing is relatively new in mechanistic interpretability, the use of learning trajectories to understand neural networks substantially predates it. Early work showed that different singular modes of the input–output correlations can be learned at different rates (where larger singular values tend to be learned earlier), making acquisition order informative about task structure (Saxe et al., 2014; Lampinen & Ganguli, 2019; Saxe et al., 2019). In contrast, representation-comparison methods measure when layers and representations converge during training (Raghu et al., 2017; Morcos et al., 2018).
Temporal Localisation and Mechanism Discovery
Trajectory-Based Attribution. Training data attribution (TDA; § 13) methods can integrate with the training trajectory to provide temporal analysis. For example, TracIn (§ 13.2.5) estimates a training sample’s influence on a testing sample’s prediction as the sum of their gradients’ dot products evaluated at checkpoints along the run (Garrima et al., 2020). Each checkpoint contributes separately to the influence estimate. By examining each checkpoint’s contribution, researchers can identify when a training example had the greatest influence on a downstream prediction.
Per-Token Learning Curves. The learning curve of an individual token in context, usually evaluated on held-out text at every checkpoint, is the finest-grained localisation instrument. Chang et al. (2024) extracted learning curves for one million held-out tokens across five pretraining runs and summarised each with the time the token is first learned, the likelihood its loss later regresses, and the variability both within a run and across runs. They found that frequent tokens and tokens in short frequent contexts are acquired earlier, reach lower final surprisal, and vary less within and across runs, with the effect strengthening with n-gram probability. At the level of stages, models learn short repetitive phrases before longer coherent text, consistent with early n-gram learning followed by gradual refinement of tail predictions. Individual tokens also show sudden increases or decreases in loss that recur at consistent points across runs. These abrupt changes in per-token loss can help identify potential transitions during training (§ 12.3.1). They also complement trajectory-based attribution by revealing when a model learns specific predictions.
Localising Change to Components. Once a change is localised in time, the components that changed in that window are candidates for the mechanism. Wang et al. (2025b) introduced refined variants of the Local Learning Coefficient (§ 12.5.2) and applied them to individual attention heads of a two-layer attention-only transformer across training. The weight-refined LLC restricts the LLC to a single head’s parameters; the data-refined LLC evaluates it on a chosen data subset. Weight-refined LLC curves diverge progressively over training, with distinct patterns for different head types. Heads of the same type cluster together in both their final values and the overall shape of the curve across training, revealing how heads differentiate into distinct functional roles. On the other hand, data-refined LLCs reveal what a head specialises in processing; for example, on code samples, perturbing the weights of the multigram heads in layer 1 has significantly less impact on the loss than perturbing the induction heads. The same analysis identifies a previously undocumented multigram circuit. A multigram is a common token sequence in the data distribution whose tokens may appear non-contiguously in context. Heads with higher weight-refined LLC tend to memorise more multigrams. Those with lower values tend to follow simple algorithms, such as induction or bracket matching. Wang et al. (2025b) identified these heads by tracking each head’s complexity measure. Unlike the proposed induction scores (Olsson et al. (2022)) for identifying induction heads, the measure does not require specifying a head type in advance. However, estimating refined LLCs is computationally expensive, and the approach has so far been demonstrated only on toy models.
Causal Interventions on Mechanism Formation
The forked-run experiment is a basic analysis pipeline available on a training trajectory. It spawns multiple continuations from a single checkpoint, perturbs one factor, and compares the resulting models.
Linear Mode Connectivity and Forked-Run Experiments
A forked-run experiment spawns two or more training continuations from the same checkpoint, varying a single factor such as SGD noise, data order, or random seed, and compares the models the continuations produce. Linear mode connectivity is a standard instrument for the comparison. Two networks are linearly mode connected if the loss barrier along the straight path between their parameters in weight space is near zero. Specifically, Frankle et al. (2020) examined the sensitivity of the model’s training results to SGD noise by restarting training from different checkpoints, each time using different SGD noise. Early in training, these runs may converge to different loss basins. Later, they tend to converge to linearly mode-connected solutions, indicating that the training outcome has become stable to SGD noise.
Perturbations other than SGD noise fit the same design. For example, degrading the data early in training causes permanent deficits that the same degradation applied later does not, establishing critical learning periods (Achille et al. [2]), and forking a run across many random seeds distinguishes properties of the training setup from accidents of a single run (van der Wal et al. (2025)). Similarly, the smeared-key intervention discussed in § 12.5.3 is the same design with a sacramental perturbation. Mode connectivity has also been given a mechanistic reading: Lubana et al. (2023) defined two models as mechanistically similar when they are invariant to the same transformations of the input. They conjectured that two minimisers that cannot be linearly mode connected (up to permutation symmetries of the architecture) must be mechanistically dissimilar, and found that the mechanistically dissimilar minimisers they constructed are connected by quadratic but not linear paths. Under this interpretation, a loss barrier between two checkpoints suggests that their mechanisms differ. However, the absence of a loss barrier does not guarantee that their mechanisms are the same.
Temporal Stability and the Scope of Mechanistic Claims. Learning-dynamics analysis lets us investigate mechanistic claims with scopes broader than those of a single model checkpoint. Specifically, an explanation about a trained model’s internals (e.g., the direction of a feature) does not by itself establish whether the internal mechanism, if it persists, works in the same way as mechanisms in different training stages of the same model or mechanisms across independently trained models of the same kind (§ 12.1.2). In contrast, learning-dynamics analysis can re-evaluate the proposed explanation across neighbouring checkpoints or different random seeds, revealing how stable the explanation is under continued training and how sensitive it is to variation in the training trajectory.
Phase Transitions and Emergence
Training loss curves typically make it seem like language models gradually and steadily improve in performance (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2020). While this holds on average, individual capabilities tell a very different story. For many individual tasks, model performance can change sharply and abruptly across model scales and across training (Wei et al., 2022; McKenzie et al., 2023). This suggests that models undergo phase transitions, where relatively small increases in scale or training time produce qualitative changes in behaviour. These transitions have motivated substantial interest in the study of how and when phase transitions happen, which we will discuss in more detail in § 12.5.2.
These sharp changes also appear in model internals. Attention heads can specialise into distinct computational roles over relatively short intervals (Olsson et al., 2022), and the formation of induction heads and syntactic attention structures has been associated with abrupt changes in model behaviour (Chen et al., 2024a). Note that behavioural and internal transitions need not occur at the same time or be equally abrupt. A model capability may suddenly emerge even as its underlying internal mechanisms develop gradually. Grokking (Nanda et al., 2023) illustrates this mismatch: as the model develops a generalising circuit, it still relies on memorisation. Once the generalising mechanism dominates and competing memorisation mechanisms are removed, the model’s generalisation ability jumps. We therefore use phase transition broadly to describe qualitative shifts in behaviour, representations, etc., without assuming that they change simultaneously or discontinuously.
Phase transitions matter for AI safety because they undermine our ability to forecast what the model will develop. If a capability can appear with a small increase in model scale or training time, behavioural evaluation of intermediate checkpoints offers limited assurance about the final model. Also, scaling laws fit to smaller models offer limited insight into larger ones. We therefore need to understand how and when phase transitions happen to reliably predict whether and when a model will acquire a hazardous capability. We organise this subsection as a workflow for the researcher who has a training run in hand: detecting transitions (§ 12.3.1), measuring the representations and mechanisms that underlie them (§ 12.3.2), attributing them to specific components (§ 12.5.1), explaining why they occur (§ 12.5.2), and, as a case study, applying this workflow end to end to induction heads (§ 12.5.3).
Detecting Transitions
A key challenge in detecting transitions is that aggregate loss can hide abrupt changes in individual capabilities. The quantisation model of neural scaling (Michaud et al., 2023) suggests that models learn discrete units of capability, each potentially emerging abruptly. However, their combined loss can still decrease smoothly because these units occur at different frequencies in the data. This helps explain why abrupt changes can occur across both training time and model scale despite smooth overall loss curves. Researchers therefore use metrics at different levels of granularity to detect these transitions.
Loss-Curve Derivatives. The crudest instrument is the aggregate loss curve itself: a transition large enough to affect overall performance appears as a bump, visible as a local peak in the loss derivative (Olsson et al., 2022). However, most transitions are too small to detect this way. A flat derivative therefore does not constitute evidence of the absence of a transition.
Capability-Specific Metrics. A metric targeted at a single capability, evaluated at every checkpoint, resolves transitions the aggregate hides. The in-context learning score (Olsson et al., 2022) is a representative construction. It isolates the capability of interest (benefiting from long, previous context) by taking the difference between two quantities. Additionally, tracking benchmark accuracy across checkpoints can similarly reveal capability transitions, though at a coarser level. Comparing results across random seeds helps determine whether these transitions are consistent across training runs (van der Wal et al., 2025).
Constructing a Capability-Specific Metric. The in-context learning score of Olsson et al. (2022) is defined as the prediction loss at the 500th token of a context minus the loss at the 50th token. Each loss is dominated by factors unrelated to in-context learning, such as the model’s overall competence on short contexts; their difference isolates the gain the model derives from additional context. Evaluated at every checkpoint, the score changes abruptly over a narrow training window, the same window in which induction heads form (Section 10.3.5), while the aggregate loss curve registers the transition only as a small bump (Olsson et al., 2022).
Per-Example Decomposition. Transitions invisible in every predefined metric can be recovered from the training data itself. POLCA decomposes changes in per-example loss along directions in the low-dimensional subspace where training occurs and clusters examples with similar loss trajectories; the clusters correspond to concepts learned abruptly at identifiable points in training, with no corresponding event in the aggregate curve (Kangaslahti et al., 2026). At coarser granularity, latent state models fit a hidden Markov model over a vector of training metrics and segment the run into discrete phases, with transitions between hidden states flagging candidate developmental events (Hu et al., 2023).
Is Emergence a Metric Artefact? Schaeffer et al. (2023) argued that the emergent abilities reported by Wei et al. (2022) are artefacts of discontinuous metrics. For example, exact-match accuracy can change abruptly while the underlying log-likelihood improves smoothly. Metric choice does explain some reported cases, and claimed emergent abilities can appear or disappear under different metrics of the same task. However, it does not explain all of them: sharp transitions are observed directly in continuous metrics, in per-example losses, and in internal representations during single training runs (Du et al., 2024; Schaeffer et al., 2025; Wu & Lo, 2025; Kangaslahti et al., 2026). The robustness of a reported transition can be assessed using continuous metrics. However, determining which behavioural transitions reflect genuine changes in the model’s internal mechanisms remains an open question.
Progress Measures and Internal Precursors
The methods above operate on phase transitions in model behaviours. However, as discussed above, transitions in external behaviour and internal mechanisms can follow different patterns. Researchers therefore are also interested in measuring the internals directly.
Progress Measure. A progress measure is a scalar quantity defined on a model’s internals that varies continuously as a mechanism forms, and can therefore track the mechanism’s development before the behaviour it supports changes (Nanda et al., 2023). Progress measures are typically derived from a mechanistic analysis of the completed mechanism at the end of training, then evaluated retroactively at earlier checkpoints.
Grokking is the controlled setting in which this tool has been demonstrated most completely.
Grokking
Grokking is the phenomenon in which a model reaches low training loss early in training but begins to generalise to unseen inputs only after a long additional period of optimisation (Power et al., 2022). This is almost exclusively seen in closed-domain settings, though general-purpose language models have shown conceptually similar behaviour.
In modular arithmetic, training accuracy saturates early while test accuracy remains near chance for many steps before rising sharply (Figure 16; Power et al., 2022). Mechanistic analysis of grokked networks recovers the learned algorithm (trigonometric identities implementing modular addition) and defines progress measures that track its formation: the generalising circuit grows gradually during the plateau, and the sharp change in test accuracy corresponds to a transition from a memorising solution to the completed generalising circuit followed by removal of the memorising components (Nanda et al., 2023). Grokking has been explained as competition between a memorising and a generalising circuit with different efficiencies under weight decay, an account that predicted and confirmed ungrokking and semi-grokking (Varma et al., 2023). Grokking can be induced or eliminated by controlling parameter norm at initialisation, and it occurs beyond algorithmic tasks (Liu et al., 2023).
Figure 16. Grokking. Training loss (purple) falls early in training while test loss (red) remains high. After a long plateau, test loss drops sharply, and the model generalises. Progress measures defined on model internals show the generalising circuit forming gradually during the plateau, before the change is visible in behaviour.
Transfer from Grokking Settings to Language Models
The standard setting in grokking experiments differs from language model pretraining in many ways, including dataset size, weight decay, and the number of epochs over the data. Current studies have not shown that existing quantitative conclusions transfer directly to large-scale pretraining.
Similar findings have been reported outside grokking settings. Chen et al. (2024a) found that syntactic attention structures emerged during a sharp drop in masked language modelling (MLM) loss, before accuracy improvement on a grammatical benchmark. Similarly, Zucchet et al. (2025a) observed that attention-based recall circuits formed during a plateau in factual-knowledge acquisition, followed by rapid improvements in model performance. These findings suggest that changes in internal mechanisms can precede changes in model capabilities. This observation motivates research on forecasting model behaviour through internal monitoring, as discussed in § 12.6.
Attributing Transitions to Components
So far, the methods we discussed cannot attribute the transition to specific model components. Attention head specialisation is probably the most intuitive option to work at this granularity in transformers. Individual attention heads can specialise to attend to syntactic relations, positional offsets, or rare tokens (Voita et al., 2019; Clark et al., 2019), and this specialisation can appear abruptly. For example, induction heads form during a brief training window, and their formation coincides with a sharp improvement in in-context learning (Olsson et al., 2022), a transition we examine in detail in § 12.5.3. In masked language models, suppressing the syntactic attention structure delays the capability it precedes (Chen et al., 2024a). Likewise, sparse attention emerges abruptly and depends on repetition in the training data (Zucchet et al., 2025b).
These events cluster in time. Several attention heads, such as induction heads and heads exhibiting syntactic attention patterns, can acquire specialised functions within the same training window. These heads emerge alongside a peak in how accurately the model’s perplexity predicts human reading times, and this predictive ability subsequently declines even as language modelling performance continues to improve [16]. Head specialisation can be viewed as a form of symmetry breaking. Before or in the very early stages of training, attention heads are indistinguishable and interchangeable; during training, they start to develop distinct functions despite having equivalent roles in the architecture. Tracking this process can help identify the emergence of modular structure. Attribution of this kind is also a diagnostic for modularity more broadly. Three observable indicators jointly diagnose modular internal structure (Zhao et al., 2024b): transitions during training, emergent trends across model scale, and heterogeneous clustering of solutions across random seeds. A structure exhibiting all three is a strong candidate for a discrete, functionally specialised mechanism.
Sharp Transitions from Smooth Components
Phase transitions in test loss or task accuracy do not imply abrupt change in any individual component. For example, in controlled settings, many attention heads develop induction-like behaviour gradually—their contributions are largely additive, and the sharp macroscopic transition arises from their aggregate effect (Singh et al., 2024). Localising a phase transition in a coarse-grained metric therefore does not help identify the responsible components. In contrast, component-level attribution may find only gradual change without being wrong.
Why Transitions Happen
Saddle-to-Saddle Dynamics. Theoretical studies of deep linear networks show that different singular modes of the input–output correlations can be learned at different rates (Saxe et al., 2014; 2019). The training process may therefore involve periods of slow improvement followed by rapid reductions in test loss. From an optimisation perspective, gradient descent moves through regions near successive saddle points in the loss landscape and gradually learns more complex solutions along the way. This process is known as saddle-to-saddle dynamics. Similar training patterns have been observed in other neural networks, where they learn simpler functions before more complex ones (Rahaman et al., 2019; Nakkiran et al., 2019). Attention head specialisation may also be related to symmetry breaking during training, although the relationship to saddle-to-saddle dynamics is not yet fully established in realistic transformers (Chen et al., 2024a).
Singular Learning Theory. Singular learning theory (SLT) studies how the geometry of a singular model’s parameter space affects learning and generalisation (Watanabe, 2009). Neural networks are singular (and are thus a subject of study in SLT) because different parameter configurations can implement the same function. As a result, two models that achieve similar loss can differ substantially in their effective complexity. SLT provides a framework for studying this complexity and has motivated research on its changes during training and its usefulness in downstream applications (Lau et al., 2025).
Local Learning Coefficient (LLC)
For model parameters at a local minimum, the Local Learning Coefficient (LLC) measures the degeneracy of the loss landscape near via the volume-scaling rate of low-loss neighbourhoods (Lau et al., 2025). Smaller values indicate more degenerate (flatter, in a higher-order sense) minima. The LLC is estimable at scale with stochastic-gradient MCMC (Lau et al., 2025).
Estimated LLC trajectories in transformers change at points that coincide with identifiable developmental events, including the stages of structure formation described in § 12.5.1 (Hoogland et al., 2025). This is the central empirical result of the developmental interpretability programme (Hoogland et al., 2023) introduced in § 14.3. However, current evidence for LLC is largely retrospective, where phase transitions are detected and matched to known events only after training. Prospective prediction of a transition before it occurs has not yet been demonstrated. § 12.6 discusses potential applications of learning-dynamics analysis to forecasting and intervention.
Case Study: Induction Heads
Learning Dynamics Perspective on Induction Heads
The preceding case studies (§ 6.8 and § 7.7) analysed induction heads and circuits in a trained model. Several facts about the induction circuit are invisible from that vantage point but become observable when we examine the model across training (Olsson et al., 2022). Consider the repeated sequence again
A B C A B ?,
and observe at successive checkpoints of a single training run. Examining the checkpoints shows that the induction ability arrived abruptly: at early checkpoints the model bases its predictions on token frequencies and derives no benefit from the earlier occurrence of A B, and during a narrow window of training, roughly 2.5 to 5 billion tokens in the models studied by Olsson et al. (2022), induction heads form and the model begins predicting C by attending back to the first occurrence. A sharp improvement in in-context learning and a small bump in the training loss fall within the same window.
The timing of formation also helps provide causal evidence that a single model checkpoint cannot provide. Ablating an induction head in the final model only shows its necessity for the behaviour at that checkpoint. This does not support the stronger claim that the induction circuit’s formation caused the capability to develop. Olsson et al. (2022) examined this stronger claim by intervening on formation itself. Their “smeared key” modification built the previous-token head’s contribution into the attention computation, allowing the circuit to form within a single layer. Under this modification, one-layer models acquire in-context learning they otherwise never develop, while the transition moves earlier in deeper models.
Leveraging Learning Dynamics for Safety
The analyses above also support safety-oriented objectives, including anticipating capabilities before they appear (§ 12.6.1) and intervening in the training process while it is still underway (§ 12.6.2).
Forecasting Model Behaviours
Forecasting from Smaller Runs and Partial Runs. Scaling laws demonstrate that aggregate loss is predictable before training (Kaplan et al., 2020; Hoffmann et al., 2022), but phase transitions break this style of extrapolation for individual capabilities (§ 12.3). Forecasting therefore targets specific behaviours directly. Memorisation is the behaviour for which the forecasting problem has been posed most explicitly. Biderman et al. (2023a) introduced the problem of forecasting whether a specific training sequence will be memorised by a model before that model is trained. They argued that recall is the metric to optimise (a false negative means training a full model that leaks the sequence, while a false positive only discards a cheap trial run). They established baselines that forecast from smaller fully trained models and from partially trained checkpoints of the target run, together with equi-compute recommendations for dividing a fixed forecasting budget between model size and tokens.
Forecasting Safety-Relevant Behaviours. Emergence can be forecast by intervening on the training trajectory. Fine-tuning on a target task shifts capability emergence towards less capable models. Fitting how this shift varies with the amount of fine-tuning data yields an emergence law. Using only small models, this law predicts whether a capability will have emerged in models trained with up to more compute (Snell et al., 2024).
Rare harmful behaviours can also be forecast statistically. The largest elicitation probabilities observed in a small sample of queries scale predictably with query volume. This relationship allows harmful-behaviour risks to be extrapolated across up to three orders of magnitude of deployment-scale query volume. Examples include assisting with dangerous chemical synthesis and taking power-seeking actions (Jones et al., 2025).
Post-training failures can be analysed similarly. Specifically, transitions between successive RLHF checkpoints can be classified into four categories: reward hacking, collapse, proxy under-alignment, and evaluator-specific disagreement. Reward hacking occurs when the model exploits a way to achieve high rewards while the external quality from the perspective of human users falls; collapse occurs when both decline. This classification localises failures that checkpoint averages miss. A model fitted to training-dynamics diagnostics predicts future reward hacking with a ROC-AUC of 0.82. These diagnostics include reward-model uncertainty, policy drift, and output diversity [1].
Internal signals extend forecasting to fine-tuning outcomes. Activation-space persona features predict whether a fine-tuned model will exhibit emergent misalignment (Wang et al., 2026a). Signals available before fine-tuning begins also predict the resulting model’s alignment (Zhang et al., 2026b). § 14.4 discusses the research agenda these results instantiate.
Internal Precursors as Early Warning. Where a mechanism forms before the capability it supports is behaviourally expressed, monitoring internals can flag the capability earlier than behavioural evaluation. The recall circuits that precede factual-knowledge acquisition (Zucchet et al., 2025a) and the syntactic attention structure that precedes the break in grammatical capabilities (Chen et al., 2024a) show that this lead time can exist; progress measures (Section 10.3.2) are the instrument that would exploit it.
Run-to-Run Variance. To assess the reliability of a forecast, researchers can compare training runs with different random seeds. van der Wal et al. (2025) found that capabilities generally emerge at similar stages and representations develop consistently across training runs, although some runs differ substantially. These exceptions highlight the uncertainty of forecasts based on a single training run.
Most Early-Warning Evidence Is Retrospective
Currently, nearly all demonstrations that changes in internals lead to changes in behaviour are retrospective. In these studies, researchers analyse the completed mechanism after training to identify a progress measure or precursor structure, then evaluate it backwards over saved checkpoints. This concern also exists in LLC-based transition detection (Section 10.3.4).
In contrast, prospective forecasting requires researchers to predict outcomes before observing them. Biderman et al. (2023a) proposed baseline methods for forecasting memorisation. However, prospective forecasting for broader capability transitions relevant to AI safety remains underdeveloped. Early-warning signals therefore require prospective validation before they can be considered reliable.
Intervening on Training Processes
The model that matters is the one deployed at the end of training, and the standard response to an undesirable behaviour found there is post-training fine-tuning, which is outside the scope of this section. An alternative is to intervene during training to prevent undesirable capabilities from developing or to control how they develop. Existing work explores three approaches: modifying training data, changing model architecture, and intervening on model activations during fine-tuning.
Pretraining Data Filtering: Deep Ignorance. Deep Ignorance filters proxies for biohazardous knowledge out of the pretraining corpus and trains 6.9-billion-parameter models on the filtered data (§ 12.1.1; O’Brien et al., 2026). The filtered models do not acquire the targeted knowledge, making it difficult for the unwanted capability to emerge through adversarial fine-tuning. In contrast, post-training unlearning may suppress the unwanted behaviour without removing the underlying knowledge. A follow-up moves the filter from documents to tokens: individual tokens are labelled using sparse autoencoders and efficient classifiers trained on those labels, and filtering at token granularity removes the undesired capability at lower cost to desired capabilities than document-level filtering, with effectiveness increasing with model scale across the two orders of magnitude tested and filtered models remaining alignable afterwards (Rathi & Radford, 2026).
Similarly, the order in which different categories of training data are presented matters. Bruijns et al. (2026) found that, in an imbalanced pretraining curriculum where one task is shown before another, the model packs the tasks sequentially into separable circuits, resulting in more targeted subsequent refusal fine-tuning.
Data-Level Interventions Have Ripple Effects
Removing data from a training set does not act only on the removed data. In the setting of Borkar et al. (2025), deleting personal information can increase memorisation of the personal information that remains, and adding one individual’s information amplifies memorisation of others’ by roughly , because whether a sequence is memorised depends on similar sequences seen elsewhere in training. Subliminal learning (Cloud et al., 2026) shows that data can transmit traits not apparent from its semantic content. For example, a student model can inherit a teacher’s hidden bias from seemingly unrelated data, such as number sequences. This transfer is carried by a small set of divergence tokens and depends on early layers (Schrodi et al., 2026). Filtering and unlearning interventions therefore need to be evaluated on what the model does with the data that stays, not only on what it no longer does with the data that was removed.
Architectural Modification: Gradient Routing. Gradient routing is an architectural modification informed by learning dynamics: it constrains which parameters receive gradient updates for chosen concepts during training, localising the corresponding computation into designated regions of the network (Cloud et al., 2024). Although the modification does not make the trained model safer on its own, it shapes where a mechanism forms so that later interventions, such as gating the designated region, become clean and localised. The antecedent for this style of intervention is the smeared-key modification (Olsson et al., 2022), which showed that an architectural change can control when and whether a mechanism forms at all (§ 12.5.3).
Monitoring and Steering During Fine-Tuning. The case excluded above uses fine-tuning to repair an undesirable behaviour after it has been identified. Intervening on the fine-tuning process itself remains within scope, and persona vectors provide a current example of this approach. A persona vector is an activation-space direction corresponding to a character trait such as sycophancy, hallucination propensity, or malice, extractable automatically from a natural-language trait description (Chen et al., 2025b). Intended and unintended personality changes during fine-tuning correlate strongly with movement along the relevant vector, which yields the monitoring instrument. Preventative steering along the vector during fine-tuning limits acquisition of the trait. Before fine-tuning, projecting training data onto the vector flags data that would induce the trait, at both the dataset and individual-sample levels (Chen et al., 2025b). This instantiates each step of the intervention loop below for the fine-tuning stage.
Train-Time Intervention Loop
The work discussed so far lays the methodological foundation for intervening in a training run in progress. Below we present a general pipeline.
Monitor a target quantity at checkpoints, such as a capability-specific metric (§ 12.3.1) or a progress measure on model internals (§ 12.3.2).
Define the acceptable range for that quantity in advance, including the response to a breach.
Intervene when the range is breached, such as by adjusting the data mixture or filtering out unwanted data, applying a targeted architectural or regularisation change, rolling back to a pre-breach checkpoint, and resuming under the adjusted setup.
Re-evaluate the model after the intervention to verify that the targeted capability has been suppressed and does not re-emerge during further training.
No published system implements this loop end to end at scale. Instead, existing studies have demonstrated individual steps of the proposed workflow.
Open Questions. Three questions remain about the applicability of these training-time interventions. First, effective data filtering requires identifying which training data contributes to the capability we want to remove. Token-level filtering is motivated by data attribution, but it remains unclear whether methods such as influence functions (§ 13) can reliably identify the relevant data. Second, scale: Deep Ignorance operates at 6.9 billion parameters, and while token-level filtering became more effective across the tested scale range ((Rathi & Radford, 2026)), it remains open whether data filtering stays effective and affordable at frontier scale. Third, deployed models show that stronger train-time safeguards are achievable, but not how: gpt-oss models are substantially harder to fine-tune than comparable open-weight models, and their developers have not fully disclosed their responsible training procedure ((OpenAI et al., 2025)). Understanding how this resistance is achieved could help develop similar safeguards for other open-weight models and clarify how fine-tuning changes model behaviour.
Summary. Learning dynamics can be studied both as an object of explanation, revealing how representations and mechanisms evolve during training, and as an interpretability tool to localise mechanisms, detect phase transitions, and test causal hypotheses through training interventions. Understanding learning dynamics can also help predict when capabilities emerge and guide interventions during training. However, no complete system combining monitoring and intervention has yet been demonstrated at scale. This section focuses on how models change during training, while § 13 examines how individual training examples contribute to model behaviour.
Training Data Attribution
For a model behaviour of interest, § 6–§ 12 study the internal features, components, and their interactions and evolutions that produce the behaviour. Training data attribution (TDA) shifts the focus from model internals to training data, searching for training samples that strongly shaped the model behaviour. TDA applications include, but are not limited to, identifying influential training samples, detecting train–test leakage, and diagnosing model failures without retraining ((Liu et al., 2025a; Park et al., 2023; Grosse et al., 2023)). Although TDA is not generally a mechanistic method under this survey’s criteria, it can complement mechanistic interpretability in many ways. For example, by revealing how specific training samples shape malicious behaviour, TDA can enable effective training data filtering to prevent unwanted behaviours or even their internal mechanisms from forming during the training of next-generation models. Motivated by this application, this section focuses on post-hoc training data attribution methods that link a given target model’s behaviour to its training samples. This section excludes techniques for intervening in the data pipeline, such as filtering pretraining corpora, rebalancing demographic representation, and scheduling memorisation.
TDA falls into two complementary approaches. Verbatim data attribution detects explicit memorisation by identifying exact or near-exact string matches between model outputs and the training corpus. These searches can scale to trillions of tokens using techniques such as suffix arrays and infini-gram models. Gradient-based data attribution estimates how training samples affect model parameters and predictions and can capture influential samples that do not manifest as verbatim reproduction. The two types of methods provide orthogonal views: one answers “did the model use rote memorisation on a training sample to answer this question?”; the other answers “how much did this training sample influence this certain behaviour of this model?”
Verbatim Data Attribution: OLMoTrace and Maximal Matching Spans
Earlier corpus search tools such as the ROOTS Search Tool (Piktus et al., 2023a) and GAIA Search (Piktus et al., 2023b) established the practice of querying pretraining corpora for transparency and data exploration. OLMoTrace (Liu et al., 2025a) scales this to trillions of tokens using a combination of suffix arrays, infini-gram models, and maximal matching span extraction (Liu et al., 2024a), making it less a data-attribution method in the influence-function sense and more a retrieval tool for identifying content the model may have memorised.
Suffix Arrays (SA)
The suffix array of a string of length is an array of integers representing the starting positions of all suffixes of in lexicographic order. It enables efficient substring searches in time for a query of length , compared to naive search.
Example:
Suffixes:
Infini-Gram Models
An infini-gram model extends the n-gram model to arbitrary-length prefixes, using backoff to the longest matching sequence in the training corpus. Formally, for a token :
where are preceding tokens and is the training corpus.
Infini-grams are implemented on top of suffix arrays, which allow fast substring queries across massive corpora.
Maximal Matching Spans
Given a model output and a training corpus, a maximal matching span is a contiguous sequence of tokens that:
Appears verbatim at least once in the training data,
Does not break words or include spurious delimiters (e.g., periods or newlines) except at span boundaries, and
Is not contained within a longer span that satisfies 1. and 2.
OLMoTrace identifies maximal matching spans by combining suffix-array search with infini-gram queries, adjusting boundaries for tokenisation, and retaining only the longest non-overlapping spans.
OLMoTrace Pipeline
Language Model Output: “Celine Dion is a Canadian singer known for her powerful voice and wide [...]”
Pipeline Steps:
Tokenise the language model output.
For each token starting a new word, query the suffix array with infini-gram to find the longest matching prefix in the training corpus.
Adjust span boundaries to avoid breaking words or including delimiters.
Retain only maximal spans (not contained in longer matches).
Example Spans Found:
“Celine Dion [...] Born on March 30, 1968, in”
“Charlemagne, Quebec, Canada, Dion began”
“performing at a young age and gained [...]”
“Her career took off when she won the 1988”
“Eurovision Song Contest with the song ’Ne”
This process links language model output directly to the training corpus, enabling identification and attribution of memorised content.
OLMoTrace Algorithm
Inputs: Model output tokens , training corpus tokens , suffix array
Steps:
Scan each token position in , considering only tokens starting a new word.
Query the suffix array to find candidate matching positions in the training corpus.
Use infini-gram queries to determine the longest matching prefix starting at , backing off to shorter n-grams if necessary.
Adjust the match length to avoid breaking words or including spurious delimiters.
Collect all spans that satisfy the above criteria.
Filter out spans contained within longer overlapping spans to retain only maximal matches.
Limitations as a Mechanistic Interpretability Tool. Verbatim attribution excels at factual memorisation but struggles with compositional or synthesised outputs. For tasks like translation, poetry, or reasoning, OLMoTrace (and other verbatim matching methods) can identify spans of text that appear in the training data, but these matches do not explain how the model combined information from multiple sources to produce the output.
OLMoTrace on Multilingual Output
Prompt: “Write a short story in Chinese about a girl who loves reading.”
Completion (Chinese):
“. . . 一个叫雪儿的小公主,她有一只可爱的小狐狸伴在身边。她喜欢坐在竹林间看着远方。某日,一场突如其来的山洪导致雪...” OLMoTrace Influential Samples Identified: OLMoTrace identified maximal matching spans across training datasets:
Span: “一个叫雪儿的小公主,她有一只可爱的小狐狸伴在身边。她喜欢坐在竹林间看着远方”—matched in pretraining corpus
Span: “她喜欢阅读关于科技、心理学和历史的书籍,业余时间还会参加编程课”—matched in post-training (instruction-tuning) data
This example demonstrates how OLMoTrace traces multilingual outputs across diverse training sources, linking model generations directly to their memorised origins in pretraining and post-training datasets.
Gradient-Based Attribution: Understanding Data Influence Through Perturbations
Gradient-based methods estimate how much each training sample influenced the model’s predictions.
Core Idea: Parameter Response to Training Data
Influence functions estimate how a model’s parameters and predictions would change if a training sample were slightly upweighted (Koh & Liang, 2017; Grosse et al., 2023). We introduce their core idea below.
Consider a neural network parameterised by . After sufficient training, its parameters will converge to
where is the aggregate training objective calculated as:
is the training set and is a loss function such as cross-entropy loss. Note that although by definition is expected to be a global optimum, it will not affect the subsequent analysis if is in fact a local optimum.
Now, suppose we upweight a training sample by a small amount . The new aggregate training objective will become:
It corresponds to a new set of model parameters expressed as:
This expression is not a closed-form formula and does not provide us with an efficient way to acquire . A brute-force way is to train the neural network with this new objective and observe the new converged parameters; however, this method is extremely computationally expensive. We thus turn to leveraging the specific property of . Note that, by definition, it must satisfy the first-order optimality condition:
This motivates defining a function to link and :
Note that , which allows us to consider the implicit function theorem.
To apply the implicit function theorem, we assume that is continuously differentiable in a small area around . Furthermore, the Jacobian of with respect to at , i.e.,
is invertible. denotes the Hessian of the training loss. These assumptions allow us to apply the implicit function theorem.
By the implicit function theorem, there exists a neighbourhood of and a unique continuously differentiable function on such that for all . In other words, for sufficiently small , is the stationary point of near . Moreover, its derivative is given by
Recalling the expression of , we also have
Putting to let and , we have:
This quantity describes how the model parameters respond to an infinitesimal increase in the importance of the training sample .
To measure the effect on a test-time quantity (e.g., the model’s test loss on the testing sample ), we apply the chain rule:
The quantity is called the influence score. Taking a step forward, by the definition of the derivative, for sufficiently small we have
That is, the influence score quantifies the first-order effect of the training sample on the model’s downstream loss for a testing sample .
Proximal Bregman Response and the Generalised Gauss–Newton
Empirical results show that the original form of influence functions introduced above performs poorly in practice. This is due to multiple assumptions that rarely hold in practice. For example, for a trained neural network, its parameters are scarcely exactly at the local or global minimum . Additionally, the Hessian is usually non-invertible or numerically unstable due to some small eigenvalues. These limitations motivate researchers to formulate influence functions without unrealistic assumptions. One classical approach is called the proximal Bregman response function (PBRF) ([27]). It targets model parameters which minimise the new training objective while staying close to the original parameters. Below, we sketch its core idea.
Let each training sample , where is the input and is the label, and is the output of the model (parameterised as ) given input , such as a logit vector. Slightly abusing notation, we denote the per-sample training loss as
where is a loss function as in the vanilla influence function formulation. Here we additionally assume is convex in its first argument , which many commonly used loss functions, such as cross-entropy loss, satisfy.
As Bregman divergence will be used in the later formulation, we introduce this notion as the next step. Specifically, for a given convex function (as described above) with the label fixed, the Bregman divergence of two model outputs, and , is:
i.e., the amount by which at exceeds its first-order Taylor approximation from .
Now we introduce the PBRF formulation. Starting from the upweighted objective , PBRF is represented as:
where denotes the original trained model parameters (note that it is not required to be the local or global optimum). Bregman divergence replaces the original loss function. An additional proximity term is added with a positive regularisation coefficient . searches for the set of model parameters that best corresponds to an upweight in while keeping its predictions on the overall training set close to the original predictions and staying close to .
In this construction, both the Bregman and the proximity terms are non-negative for all and vanish at . Since these are the only two terms of when , is a global minimiser of (i.e., ) regardless of whether training has converged.
As in the vanilla formulation, does not have a closed-form formula for now. However, by observing that , we define
where . We assume that and are twice continuously differentiable such that is continuously differentiable in a small area around . Then, we verify that is invertible at , the last prerequisite for using the implicit function theorem. By direct calculation (we omit the detailed process here), we can obtain
where , is the output–parameter Jacobian, and is the loss–output Hessian. The matrix is the generalised Gauss–Newton (GGN) matrix. Notably, as is positive semidefinite by the convexity of , we know is positive semidefinite (because for any ). As a result, is positive definite and hence invertible. Unlike the case of , the invertibility of holds by construction.
Knowing that , we apply the implicit function theorem and obtain
By the chain rule, the influence score on a test-time quantity becomes
As shown above, the influence function computed with is exactly the derivative of the PBRF at . Bae et al. [27] confirmed empirically that such influence estimates align well with the actual PBRF.
Addressing the Computational and Memory Bottlenecks of GGN
The size of the GGN matrix (where is the number of model parameters) can be extremely large for modern language models, which makes the calculation of computationally expensive. As a result, follow-up methods have avoided explicitly calculating this inverse matrix by borrowing techniques from the optimisation literature.
Kronecker-Factored Approximate Curvature (K-FAC). Martens & Grosse (2015) proposed Kronecker-Factored Approximate Curvature (K-FAC) to approximate a large Fisher information matrix (FIM) using a set of smaller matrices. For a trained neural network parameterised by , the FIM is defined as
where is a sample from the model’s prediction distribution on . For example, suppose the task is three-class classification and the model’s prediction distribution is . If a sampled label is , then .
Although K-FAC approximates the FIM rather than the GGN, these matrices are equivalent for common loss functions under suitable probabilistic assumptions. For example, with softmax cross-entropy, the expected outer product of the output log-likelihood gradients can be shown to equal the loss–output Hessian:
Applying the chain rule then gives
This shows that we can use K-FAC to approximate the GGN used in the PBRF influence score.
Now we return to K-FAC, which efficiently approximates the FIM. First, it ignores curvature between layers and approximates as a block-diagonal matrix with one block per layer. Consider a neural network in which each layer has a weight matrix , input activations , and pre-activations (unrelated to the superscript in ). Following Grosse et al. (2023), the pseudo-gradient is defined as —the gradient flowing back to . The pseudo-gradient of the layer weights is . Its vectorisation gives , where denotes the Kronecker product. Leveraging ’s properties that and , the block for this layer in the FIM, denoted as , can be represented as
The approximation separates the expectation into two independent factors: activations and pseudo-gradients. K-FAC thus stores only entries instead of entries for each block. When the two factors are invertible, their Kronecker product can be inverted separately:
This substantially reduces the memory required to store and the computational cost of computing the inverse of .
Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA. EK-FAC (George et al., 2018) improved K-FAC by correcting its eigenvalues. Recent work has adopted this technique to scale the influence function method to modern LLMs (Grosse et al., 2023). ASTRA (Wang et al., 2025a) further used EK-FAC as a preconditioner for Neumann series iterations, achieving higher-quality GGN-vector products with less hyperparameter tuning. These efforts make influence functions increasingly practical for modern language models with billions of parameters.
Retraining-Based Influence Estimation: datamodels and TRAK
Some methods have explored retraining the model to estimate the attribution. The most naive method is to upweight or remove each training sample, retrain the model on the modified training set, and observe changes in the model’s output on the testing sample. Ilyas et al. (2022) proposed datamodels to more efficiently estimate each training sample’s attribution on a testing sample by retraining many models on random subsets of the training data and fitting a linear regression that predicts the model output on the testing sample from binary indicators of whether each training sample was included. Each fitted coefficient then represents the training sample’s influence on the testing sample.
However, datamodels remains computationally expensive because it requires training many models. Tracing with the Randomly-projected After Kernel (TRAK; Park et al. (2023)) provides a more efficient alternative. TRAK first uses a first-order Taylor expansion around the model parameters to approximate a trained model as a linear model. It then applies random projections to reduce gradient dimensionality and estimate training-sample influence in this lower-dimensional space. Finally, it averages the estimates across a small number of trained models to reduce variability due to training randomness. TRAK achieves attribution quality comparable to datamodels at substantially lower computational cost (Park et al., 2023).
Efficient Influence Estimation Using Training Trajectories: TracIn
TracIn (Garima et al., 2020) estimates training-data influence using gradients evaluated along the training trajectory, without computing the inverse Hessian. Its core idea follows from the effect of a single Stochastic Gradient Descent (SGD; Robbins & Monro (1951)) update on the loss of a testing sample.
Suppose the model parameters at training step are . An SGD update using training sample with learning rate is:
By simple rearrangement, we have
Then, by first-order Taylor expansion, the change in the loss of a testing sample can be approximated as
The above expression relates the gradient of the loss on a training sample to the gradient on a test sample and the change in test loss on . A positive dot product indicates that the training update will reduce the test loss, while a negative value indicates the opposite.
TracIn extends this idea across training. Its practical checkpoint-based variant, TracInCP, estimates influence by summing gradient dot products over model checkpoints:
where denotes the selected training checkpoints. A larger positive score indicates that has contributed more towards reducing the test loss.
Unlike Hessian-based influence functions, TracIn does not require matrix inversion. However, TracInCP approximates the influence of training samples using gradients only at selected checkpoints, which may undermine the estimation accuracy.
Illustrative Example
Gradient-based influence functions can identify influential training samples without requiring them to exactly match the current model output. They complement verbatim attribution and reveal subtler influences. Below, we provide an example of using them for the English-to-Chinese translation task.
Gradient-Based Attribution on a Translation Example
Prompt (English):
“I went to the park to play frisbee with my dog. She loves catching the frisbee and running with it. Sometimes she gets too excited and won’t let go when I try to take it from her. Playing frisbee with her is always fun and great exercise for both of us. Chinese:”
Gradient-Based Influential Training Samples Identified:
“Maybe I just feel like I could do anything, I want the whole world dancing to my music and wearing my clothes. 也我只是感觉可以做任何事情,我想全世界都跳舞听我的音乐,穿着我的衣服。 She continued to slay on screen, battling snakes, bank robbers and a serial killers subconscious. 她继续在大荧幕上, 行动阻击手的意作。 Her music career took off with her first studio album on the six. 她的音乐生涯在她的第一张专辑出现了巨色。 In 1999 this double threat was riding high until hooking up with Ben Affleck, dubbed Bennifer. 1999年, 一直高歌,直到和本。艾弗克在一起本尼弗。 Lo found it difficult to brush off the box office sink, eventually entering a professional rough patch. Lo 很刷掉票房污槽,最入了一粗糙的修。”
Observation. Influence functions identify training documents that impacted the translation, even without exact verbatim matches. This demonstrates that gradient-based methods can capture influences beyond what verbatim methods can reveal.
Recent Extensions and Limitations
Recent work extends training data attribution to handle training stochasticity. Distributional TDA (d-TDA) estimates how output distributions depend on the dataset Mlodozeniec et al. (2025). Georgiev et al. (2023) proposed TRAK variants to address diffusion models. More recent work connects TDA directly to mechanistic interpretability (Kowal et al., 2026; Habibi et al., 2026); for example, Kowal et al. (2026) introduced Concept Influence to enable attribution at the level of semantic concepts represented by model internals such as sparse autoencoder features. This brings TDA closer to the mechanistic goal of understanding how training data shapes specific internal computations. However, fundamental technical challenges persist, including approximation errors that compound over the training trajectory, variability across training runs caused by randomness like data ordering, and memory and computational bottlenecks of scaling to frontier models. These limitations motivate further research in this direction.
Summary. Training data attribution traces model behaviour back to the training data. Verbatim methods such as OLMoTrace identify memorised spans, while gradient-based methods such as influence functions estimate the contribution of individual training samples to downstream model predictions. These approaches remain limited at scale. Meanwhile, recent work has begun linking data attribution to internal features and mechanisms. Having surveyed the main methodological families of AI interpretability in the preceding sections, § 14 organises them into six research orientations that differ in their goals, assumptions, and paths forward.
Six Research Orientations
An orientation is a coherent research programme defined by its goals, assumptions, and trade-offs. In contrast, a method is an established modular pipeline that may be used in different orientations. In AI interpretability, different research goals, such as developing ready-to-use safety measures or achieving complete mechanistic understanding, have led to distinct research programmes. This section organises six major research orientations and summarises their scientific objectives, core assumptions, characteristic methods, and limitations (Nanda et al., 2025; Gao, 2025; Li et al., 2025a; Hoogland et al., 2023; Barez, 2026; Biderman et al., 2026; Figure 17). Note that no single orientation is universally preferable. The choice depends on the research objective, deployment context, available resources, and tolerance for different forms of uncertainty.
Figure 17. Six research orientations of AI interpretability.
Pragmatic Interpretability: Proxy-Task-Oriented Safety Interventions
Pragmatic Interpretability focuses on doing the research most helpful for longer-term alignment and interpretability goals, such as helping AGI go well (Nanda et al., 2025). Proponents argue that full mechanistic understanding is not required to have significant impact; instead, researchers should prioritise the work most likely to help (Nanda et al., 2025). The central idea is to work on a set of operationally meaningful proxy tasks that can be empirically studied on models today to help make powerful future models safe. Researchers are agnostic about which methods they use, even if they do not involve model internals. Though not required, these proxy tasks may also be directly useful on today’s deployed systems, providing valuable feedback.
A representative example is the study of Claude Sonnet 4.5’s evaluation awareness ([14]). The system was shown to condition its behaviour on the presence of evaluation signals, making results from safety evaluations used for previous models harder to interpret. Researchers identified and applied a steering vector that suppresses this awareness, allowing evaluation of its actual behaviour. While this study does not fully explain the internal mechanism of evaluation awareness, it is sufficient to produce a concrete and safety-relevant behavioural change, illustrating the pragmatic philosophy.
Another example is Marks et al. (2025b). In this case, researchers constructed a proxy task: they fine-tuned a model with a hidden goal, then blinded teams of researchers audited the model to discover it. This served as a proxy task for detecting a naturally emerging hidden goal in a future model. The different auditing teams used different techniques, allowing researchers to compare their efficacy.
Common techniques in this paradigm include:
Steering vectors, which shift models to targeted behaviours.
Probes, which detect latent safety-relevant properties in activations.
Behavioural monitoring and anomaly detection, which track signals indicative of emerging risks, such as monitoring the model’s verbalised reasoning.
Prompting, an effective black-box baseline method for steering model behaviour.
Recent work following this pragmatic trajectory has demonstrated effective control of refusal behaviour [20] and evaluation awareness [14]. These interventions provide direct gains to contemporary systems. However, they may not (and are not designed to) fully explain how models implement their behaviours.
Ambitious Interpretability: Towards Complete Mechanistic Understanding
In contrast to pragmatic approaches, Ambitious Interpretability seeks a comprehensive mechanistic understanding of how models implement their computations (Gao, 2025). Proponents argue that partial or surface-level understanding cannot guarantee safety in next-generation systems. Full understanding is needed to rule out hidden failure modes or deceptive internal dynamics. Specifically, Ambitious Interpretability does not treat interpretability as a collection of useful heuristics. Instead, it treats interpretability as a scientific objective: to reverse-engineer the model and develop a transparent, predictive theory of its internal algorithms.
Ambitious Interpretability particularly emphasises metric-driven progress. Motivated by the fact that existing mechanistic approaches still struggle with stringent metrics, it proposes starting with looser ones, tracking and improving methods’ performance on them, and adopting stricter metrics when suitable. Two example metrics by Gao (2025) are:
Feature quality: demonstrating that extracted features and their natural-language explanations are faithful and simulatable.
Circuit faithfulness: demonstrating circuits are necessary and sufficient for explaining behaviour via causal intervention (e.g., causal scrubbing (Chan et al., 2022a)), without performance degradation.
The disadvantage of this ambition is its cost. Fine-grained mechanistic analysis of frontier large models is computationally expensive and sometimes intractable. As Gao (2025) emphasised, current tools still fall far short of the coverage and robustness required for a truly comprehensive understanding.
Interpretability-by-design as a Synthesis. The emergence of interpretability-by-design models (§ 6.10) blurs the boundary between Ambitious and Pragmatic Interpretability. These methods (Gao et al., 2026; Pearce et al., 2025; Park et al., 2025; Oldfield et al., 2024) aim to make representations and circuits easier to understand by modifying model architectures rather than relying entirely on post-hoc reverse engineering. This practical change to model design pursues the ambitious goal of mechanistic understanding. However, it remains unclear whether these methods can ultimately achieve complete mechanistic understanding. In the short term, they may simply represent a separate “Engineering” orientation in which interpretability is treated as an optimisation objective.
Developmental Interpretability: Characterising How Models Learn
Complementing post-hoc analysis of trained models, Developmental Interpretability (Hoogland et al., 2023; Hoogland et al., 2025) treats the training process itself as an object of scientific inquiry. Taking a learning-theory perspective, it aims to mathematically formalise the emergence and evolution of model representations, effective complexity, and capabilities throughout training.
Its central claim is that many neural network properties which are hard to analyse through a fixed model checkpoint can be understood through their developmental process. This process contains measurable, modelable signatures that can serve as proxies for the model’s properties of interest, such as developmental stages, phase transitions, and other organising principles that help mathematically explain how complex behaviours emerge and evolve throughout training. A classic example is the Local Learning Coefficient (LLC) introduced in § 12.2.1 and § 12.5.2. It successfully serves as a signature of the model’s effective complexity, allowing us to observe developmental transitions and compare two models with the same training loss.
Methods in this paradigm include:
Singular Learning Theory (SLT) and related mathematical frameworks for characterising developmental stages, symmetry breaking, and changes in the geometry of the loss landscape during training.
Checkpoint-based analysis to track the emergence of representations and mechanisms by applying specialised probes, feature discovery methods, and circuit analyses across a model’s training checkpoints.
More specifically, Developmental Interpretability aims to answer questions such as: Can we characterise the development of model capabilities or behaviours as phase transitions during training? What developmental stages precede the formation of a certain mechanism? How does the optimisation process transform distributed representations into structured mechanisms? Unlike Interpretability Across Time (detailed in § 14.4), which emphasises forecasting and controlling future model behaviour, Developmental Interpretability is primarily concerned with understanding the developmental processes that produce those behaviours. Its primary goal is rigorous scientific explanation rather than predicting or intervening on capabilities.
Interpretability Across Time: Understanding the Causes of Model Behaviours
Like Developmental Interpretability, Interpretability Across Time (Biderman et al., 2023b; 2026) treats a model as a time-evolving process shaped by its training data, loss functions, architectures, and optimisation dynamics. However, the two orientations differ in their primary goals. Developmental Interpretability mainly focuses on characterising the developmental processes that give rise to the model’s internal features, mechanisms, and capabilities. In contrast, Interpretability Across Time seeks established knowledge of learning that is (1) sufficiently predictive to estimate important model properties before training is complete, (2) sufficiently causal to identify when and why undesirable trajectories arise, and (3) sufficiently precise to enable interventions on the learning process itself (Biderman et al., 2026). From a safety perspective, this orientation aims to understand and ultimately shape the training process to prevent the model from developing undesirable behaviours in the first place rather than repairing them after they develop.
Interpretability Across Time builds on three areas of research. First, scaling-law studies show that model performance can be predicted from factors such as model size, training data, and compute (Kaplan et al., 2020; Hoffmann et al., 2022). This motivates similar efforts to predict safety-relevant model properties (Ganguli et al., 2022; Jones et al., 2025). Second, longitudinal mechanistic studies track how internal mechanisms develop during training (Tigges et al., 2024; Singh et al., 2024). Third, studies of training data show that the training distribution can affect downstream model behaviour. For example, Razeggi et al. (2022) found that few-shot numerical reasoning performance depends strongly on the frequency of relevant terms in pretraining data, while Elazar et al. (2023) estimated the causal effects of pretraining-data statistics on factual predictions. Interpretability Across Time builds on these findings to predict and influence safety-relevant model behaviour during training.
Methods in this paradigm include:
Longitudinal analysis of model internals, tracking representations, features, or implemented mechanisms across checkpoints to determine when and how they form and evolve.
Monitoring training runs and forecasting future behaviour, using early behavioural or internal signals to predict later capabilities, biases, or unsafe behaviours before they are fully expressed.
Training-process interventions and design, manipulating data, activations, objectives, architectures, or other aspects of optimisation to encourage desirable developmental trajectories, prevent undesirable capabilities or behaviours from arising, or ensure that learned information is represented in ways that make it easier to control.
Several studies have examined whether safety risks can be identified before or during training. Zhang et al. (2026b) found that model activations measured before fine-tuning can predict alignment scores after fine-tuning. Similarly, Wang et al. (2026a) identified internal features associated with emergent misalignment, which could sometimes detect misalignment before behavioural evaluations did. Other studies found that training models to exploit relatively simple reward misspecifications can lead to more serious failures, including reward tampering and broader misaligned behaviour (Denison et al., 2024; Taylor et al., 2025). These findings motivate monitoring early signs of undesirable behaviour during training rather than relying solely on the evaluation of the final model.
Another line of work under this paradigm attempts to directly shape models by preventing them from learning unsafe behaviours. Data filtering can prevent models from acquiring hazardous information in the first place and substantially increase the amount of subsequent fine-tuning required to recover it (O’Brien et al., 2026; Rathi & Radford, 2026), and gradient routing approaches deliberately constrain where or how information is learned, making later removal or control more tractable (Cloud et al., 2024; Shilov et al., 2025).
Common confusion. Although Biderman et al. (2026) contrasted this orientation with the “fix it in post” paradigm, it would be a mistake to view it as dismissive of post-training writ large. Post-training is itself a training process, and behaviours like sycophancy and reward hacking arise primarily during and due to post-training. This orientation’s criticism of post-training safety focuses on cases where, instead of understanding why a behaviour developed, post-training patches undesirable behaviour. The core distinction is understanding and controlling a training process versus applying optimisation pressure to suppress learned behaviours.
Interpretability Across Time is not mutually exclusive with the other research orientations. It can be Pragmatic, by using a simple probe as an early-warning signal, or Ambitious, by seeking a mechanistic theory of why a circuit forms under gradient descent. Its defining goal is using insights into training dynamics to explain and predict how model behaviour develops, and to intervene before undesirable behaviours emerge.
Introspective Interpretability: Models Interpreting Themselves
Introspective Interpretability (Binder et al., 2025; Plunkett et al., 2025; Li et al., 2025a; Guo et al., 2026; Li, 2026) uses the model being explained as its own explainer. Its main practical advantage is its unique accessibility. For a given introspective model, it requires no white-box access or specialised tooling to explain its decisions; anyone can get the self-explanation (or interchangeably self-report) at deployment time by asking the model a follow-up question in an ordinary chat interface.
This research direction is supported by the privileged access hypothesis discussed in § 5.3, which posits that models may have access to internal information that is difficult to recover through external analysis alone, including typical mechanistic analysis or a different model as the explainer. Recent work has leveraged this idea by fine-tuning target models to faithfully report their internal reasoning, behavioural tendencies, objectives, and even weight changes induced by training (Goel et al., 2026).
A related line of work fine-tunes metamodels (external explainers) on the internal representations of a target model (Pan et al., 2026; Karvonen et al., 2026a; Bauer et al., 2026; Huang et al., 2026). These methods are generally not considered instances of Introspective Interpretability because the explainer is technically a separate model. However, the distinction can be blurry in practice, as many of these explainers are initialised from the target model and can be viewed as specialised, fine-tuned descendants of the original model.
Concerns for Introspective Interpretability
Self-explanation also raises concerns. For example, generated explanations can be plausible yet unfaithful. The model’s general capability can also degrade when it is trained to honestly verbalise its internal processing. Moreover, the model may have fundamental limitations in what it can report. These concerns raise questions about whether we can trust self-explanations in safety-critical applications and how to validate potentially misleading reports. Below, we first discuss the main failure modes of self-explanation, followed by validation procedures in § 14.6.1.
Failure Modes of Self-Explanation
Introspective methods can fail in several ways. In each case, the explanation appears plausible yet is unfaithful.
Confabulation. Models may generate coherent explanations that do not reflect their actual internal computations. This occurs because language models are extensively pretrained and fine-tuned on next-token prediction and instruction-following tasks to produce fluent, knowledgeable text, not faithful introspective reports. Thus, explanations may simply reflect training data patterns (how humans explain reasoning) rather than the model’s actual computational process [19] (Turpin et al. 2023; Barez et al. 2026; Chen et al. 2025c). Moreover, the reinforcement learning reward for generating helpful responses can also incentivise convincing confabulation, even though the model cannot introspect. An example of a helpful yet unfaithful response is when the model post hoc rationalises an answer it already rote-memorised.
Miscalibration. Even when models attempt faithful self-report, their confidence may be poorly calibrated with accuracy. Models can express high confidence in explanations for computations they have not yet learned to analyse. Moreover, calibration can vary by task: models may be well calibrated on reporting their own uncertainty in simpler tasks such as factual recall but poorly calibrated on reasoning processes. A recent study further found that, even in factual recall tasks, models can hallucinate with high certainty despite knowing the correct answer (Simhi et al., 2025).
Elicitation sensitivity. Self-reports can be highly sensitive to prompting. Models can provide contradictory explanations or show substantially different introspective capabilities when similar questions are phrased differently (Hahami et al., 2026). The preset format of self-report (e.g., free-form versus structured, immediate versus delayed) can also substantially influence what information is elicited and appears accessible. Moreover, models may be intrinsically biased towards confirming hypotheses regardless of whether they reflect their underlying computations (Hahami et al., 2026).
Strategic misreporting. In adversarial or high-stakes settings, models with sufficient capability might strategically misreport their internal states. For example, a model optimised for user rewards might report sycophantic explanations it predicts users will find satisfying rather than accurate (Sharma et al., 2024b). A misaligned model might deliberately conceal its true objectives or reasoning (Hubinger et al., 2024). Even without explicit deception, models may learn to confabulate and produce seemingly safe explanations that avoid triggering safety filters (Baker et al., 2025).
Despite these limitations, targeted training can address some self-explanation failures. Studies show that models trained for introspection can produce more faithful reports of their internal objectives, behavioural tendencies, and training-induced changes (Binder et al., 2025; Li et al., 2026b; Goel et al., 2026; Li et al., 2025a; Joglekar et al., 2025). In particular, models trained to disclose hidden objectives or emergent misalignment can report these properties with high accuracy (Li et al., 2026b; Joglekar et al., 2025). Explanation faithfulness may also improve when task performance requires reasoning (Kinniment et al., 2026). These findings suggest that some introspection failures may reflect limitations of current training and elicitation methods, rather than fundamental limits on models’ ability to report their internal states.
Validation Procedures for Model Self-Reports
Given these concerns, Introspective Interpretability methods must be validated against additional, independent evidence. Specifically, they must be shown to be consistent with model internals (as accessed by other mechanistic interpretability tools) and behavioural signatures (as tested by behavioural interpretability analysis). Validation procedures vary based on the type of self-report:
Validation Procedures for Introspective Models
Causal intervention consistency. Self-explanations should be tested against intervention experiments. If a model claims a feature drives its output, ablating that feature should steer the model away from the output. If a model claims it considered a piece of information (e.g., an input feature or a recalled piece of factual knowledge in intermediate layers) for its decision, patching the information with counterfactual values should change the output correspondingly.
Cross-method consistency. Introspective methods and privileged access findings should be verified by other interpretability methods: Do SAE features identified as causal show corresponding effects to self-explanations under steering? Do probing classifiers successfully decode the information the model claimed to use? Do circuit discovery methods identify the components the model describes?
Paraphrase and sampling consistency. The stability of self-reports under semantics-preserving variations of the question should be tested. Specifically, paraphrased questions should elicit consistent explanations. Explanations should remain consistent across resamples at low temperature. Explanations for similar inputs verified to have undergone similar mechanisms by other methods should show appropriate similarity.
Calibration evaluation. The calibration between self-reported confidence in the explanation and the actual match to the internal mechanisms should be tested. Confidence should reflect uncertainty in the generated explanation based on how much information the model collected during introspection.
When to Use Introspective Methods
Introspective methods are best suited for three scenarios: (1) Hypothesis generation, where introspection provides candidate explanations to be validated by other methods; (2) Rapid triage, where introspection is used to quickly identify which input–output pairs or model components are worth further investigation; (3) Human–AI collaboration, where introspective models provide human-readable explanations tailored to users’ levels of expertise and understanding.
Introspective methods are insufficient for: (1) Claims of absence: claims about the absence of deceptive reasoning or hidden objectives cannot be established by self-report alone; (2) Formal guarantees: self-reports do not provide the causal evidence needed for formal verification; (3) Adversarial settings: in scenarios where the model has strong incentives to misreport, introspection may be less reliable and require additional safeguards.
Implications. Although introspective methods can help efficiently detect latent goals, plans, and deceptive behaviour (Binder et al., 2025; Li et al., 2026b; Goel et al., 2026; Li et al., 2025a; Joglekar et al., 2025), they work best as complements to other mechanistic analyses (Bereska & Gavves, 2024). Specifically, their most productive use is as one component of a multi-method interpretability pipeline: self-reports serve as initial hypotheses, causal interventions test them, and circuit discovery methods explain the underlying mechanisms. While we may optimistically assume models can tell us how they work in the future, we must also recognise that self-reports are relatively weak evidence and require independent validation.
Automated Interpretability: Scaling Discovery via Automation
Automated Interpretability seeks to scale mechanistic analysis to large models by replacing human analysis with automated pipelines. It assumes that manual analysis is the bottleneck and that automation can accelerate discovery. Methods include:
Automated circuit discovery to automatically find the critical computational graph inside the model that processes a given input prompt or implements a specific capability.
LLM-based feature or circuit explanations to automatically discover, summarise, and evaluate representational units and internal computations and give human-readable interpretations or reports.
Automated interpretability is not limited to any particular type of internal representation or mechanism. Depending on the research question, automated pipelines may analyse SAE features (Paulo et al., 2025b; Han et al., 2026), individual neurons (Bills et al., 2023; Foote et al., 2023; Oikarinen & Weng, 2023), circuits (Conmy et al., 2023; Khan et al., 2026), or other model components (Fraser-Taliente et al., 2026). This approach distinguishes itself by automatically discovering, explaining, or evaluating these objects rather than relying on manual analysis. Below, we illustrate a classical automated feature-explanation pipeline whose explanation and verification stages generally follow Bills et al. (2023).
Classical Automated Interpretability Pipeline
Stage 1 – Feature Extraction. SAEs are trained to decompose raw activations of the target model into sparse linear features.
Stage 2 – Explanation Generation. An “Investigator Agent” (e.g., GPT-4) reads a sample of texts that strongly activate an SAE feature and generates a natural-language hypothesis of its function (Paulo et al., 2025b; Bills et al., 2023), e.g., “This feature activates on Python code related to file I/O operations.”
Stage 3 – Explanation Verification. A “Simulator Agent” predicts feature activations on new text based on the hypothesis and calculates the correlation between predicted activations and real activations (Bills et al., 2023). A high score confirms the hypothesis, while a low score rejects the hypothesis and triggers re-investigation.
Stage 4 – Deception Detection. Crucially, coordinated agents can detect when explanations or agents themselves are deceptive or misleading—effectively deploying “AI to catch AI” in oversight scenarios (Barez, 2026).
Such a pipeline enables analysis of a model with billions of parameters with minimal human intervention (Bills et al., 2023), a prerequisite for comprehensive model oversight (Foote et al., 2023).
Recent research has substantially expanded the scope of Automated Interpretability. While early work focused primarily on automated feature explanation, newer systems employ increasingly sophisticated pipelines for feature discovery, explanation generation, auditing, and intervention (Han et al., 2026; Marin-Llobet & Ferrando, 2026; Barez, 2026; Cywiński et al., 2025). More recently, automation has also been extended from individual features to higher-level structures such as circuits (Khan et al., 2026; Haklay et al., 2026), enabling automated explanation of circuit semantics rather than merely discovering circuit components [11] (Conmy et al. 2023).
Governance Concerns for Automated Interpretability
Despite the promise, automated methods face fundamental concerns. In particular, although automation improves scalability, reduced human involvement may also increase the risk and severity of systematic errors, reward hacking, or deceptive explanations when automated oversight fails to catch them (Lermen et al., 2025). Therefore, success in this research orientation is measured not only by the analysis’s scalability, but also by its coverage, faithfulness, and trustworthiness. Below, we detail several governance challenges, including error propagation at scale, human oversight becoming a bottleneck, and systematic biases going undetected.
Governance Risks of Automation
Error propagation. Unlike human investigators, who are somewhat independent and introspective, the explanation-generation model can propagate its systematic errors to all features it analyses without rethinking what could have gone wrong.
Reduced human-specific functionalities. Automation removes humans from the loop. However, humans provide essential sanity checks, domain knowledge, and scepticism that automated systems lack or handle poorly. Lacking these critical properties may, in the long run, make the system less productive and less beneficial to the community than analyses by human experts.
Automation Bias. Automated pipelines can produce seemingly rigorous and comprehensive yet actually incomplete or sycophantic explanations. This is particularly likely when language models are involved. For example, models may selectively report positive results (Luo et al., 2026) or tailor their explanations to align with inferred user preferences (Sharma et al., 2024b). This apparent rigour and promise may lead researchers to overtrust the results and overlook underlying flaws.
Spurious Feedback Loops. When Automated Interpretability is used to improve models that are subsequently used for interpretability, feedback loops may reinforce existing errors and biases. This process may exploit and saturate interpretability metrics without genuine gains in mechanistic understanding.
Proposed Guardrails for Automated Interpretability
These concerns call for safeguards that combine technical validation, human oversight, and clear accountability. Below, we propose a five-layer guardrail framework to help audit the automated methods and mitigate their risks.
Guardrail Framework for Automated Interpretability
Layer 1: Sampling-Based Human Validation. Auditors can randomly sample a predefined fraction (e.g., 5%) of generated explanations for human experts to review. These explanations should cover different task-difficulty levels, feature types, and targeted mechanisms. Human experts should not know in advance which explanations are randomly sampled, which are flagged by automated checks, and which are already verified by additional methods, if they are all presented.
Layer 2: Consistency and Causality Checks. The reliability of automated pipelines can be improved by implementing consistency checks. Specifically, the pipeline can generate explanations multiple times at varying temperatures or setups (e.g., different backbone LLM explainers). Inconsistent explanations trigger review. The pipeline can also additionally examine a fraction of explanations by causal techniques, such as activation patching, and track failure rates. These checks can be automated or involve human auditors.
Layer 3: Adversarial Robustness. Auditors can test automated pipelines against adversarial inputs. To achieve this, auditors first construct features or mechanisms with backdoored activation patterns designed to make the pipeline produce inappropriate or targeted explanations. Auditors then examine whether the pipeline was fooled. Failure cases can be used to improve the pipeline’s robustness.
Layer 4: Error Detection and Bounds. First, automated pipelines must output calibrated confidence. Overconfidence (false positive) is more dangerous than underconfidence (false negative) and should be prevented. Second, the pipeline can be tested on features or mechanisms with known ground-truth explanations (e.g., from prior human analysis) and the faithfulness of generated explanations on them measured. Finally, the pipeline should report estimated error rates alongside all automated findings.
Layer 5: Regulator Practices. Regulators must distinguish automated hypotheses (weak evidence that requires validation) from validated findings (passed human or causal checks). For automated analyses used in safety-relevant applications, regulators must ensure versioning and reproducibility. When the pipeline improves, or the target model is fine-tuned, regulators should re-evaluate prior analyses. Regulators should also be aware of their scope, clearly studying and defining what automated pipelines can and cannot reliably assess. Decisions should not be made by extrapolating beyond validated scope.
Integrating Automated and Human Analysis
A robust analysis pipeline should combine automation for scale and suitable human involvement for validation and edge cases. Table 12 shows an exemplary workflow that combines AI and human labour for scalable interpretability analysis.
| Stage | Automated Role | Human Role | Deliverable |
| Discovery | Extract candidate features and circuits | Define scope and review methodology | Candidate features and circuits |
| Prioritisation | Rank features and circuits by importance metrics | Review rankings and adjust priorities | Prioritised analysis queue |
| Explanation | Generate explanations at scale | Audit a fraction of samples and review flagged cases | Draft explanations |
| Validation | Run automated consistency and causal checks | Investigate failure modes and validate edge cases | Validated findings |
| Synthesis | Aggregate findings and generate reports | Interpret implications and make decisions | Actionable conclusions |
Table 12. A human–Automated Interpretability pipeline. Automated methods perform discovery and analysis. Human experts set the objectives, audit intermediate results, and draw conclusions. We use feature and circuit discovery and analysis as an example.
Key Principles for Integration. Here we discuss several design principles for integrating humans and AI into the interpretability workflow. First, automation provides breadth, and humans provide depth. For example, the workflow can use SAEs with LLM-based description generation to discover and label many features, then have human investigators analyse those with safety-relevant descriptions. Second, humans set criteria and automation applies them. Specifically, humans define criteria for a valid explanation, a feature of interest, or a concerning finding, while the automated components (e.g., the LLM explainer) operate based on these criteria. Third, the system should have clear escalation criteria to determine when to call for human intervention, such as when the system detects potential mechanisms underlying safety-relevant concepts. Finally, the system should implement a feedback loop to let human investigators correct and calibrate it in real time during interaction.
Summary: Responsible Automation
Introspective and Automated Interpretability offer promise for scalable mechanistic analysis. However, they both have technical challenges that call for additional guardrails. Specifically, Introspective Interpretability requires independent validation to prevent self-serving explanations; Automated Interpretability still requires human oversight to catch systematic errors. Neither should be regarded as a complete replacement for human judgement in high-stakes decision-making. Ultimately, Automated Interpretability aims not to replace human investigators, but to combine human expertise with automation to make interpretability analysis more comprehensive and reliable.
Comparative Analysis: Trade-Offs Between Orientations
Table 13 compares the six research orientations across several dimensions, including their problem choices, primary techniques, and practical scalability. None of them is universally preferable. For example, Automated Interpretability can scale analysis to millions of features in frontier models but faces issues such as error propagation and automation bias. By contrast, Ambitious Interpretability can provide fundamental, theory-grounded analysis, but its current results still mainly come from controlled, small-scale settings (Elhage et al., 2022a), and its transferability to frontier models remains unclear. Thus, the choice of orientation depends on multiple factors, such as the intended downstream use cases and available computational and human resources, as detailed in § 14.9.
| Dimension | Pragmatic | Ambitious | Developmental | Across Time | Introspective | Automated |
| Main Focus | Immediate safety interventions without complete understanding | Complete causal understanding regardless of cost | Developmental processes of representations, mechanisms, and capabilities | Prediction and control of future model behaviour through training dynamics | Introspective models to explain themselves | Scalable discovery via automated pipelines |
| Key Metrics | Performance in proxy tasks | Faithfulness and simulatability | Phase-transition signatures (e.g., LLC) and their correspondence to mechanism formation | Forecasting accuracy and intervention effectiveness | Alignment between self-reports and actual mechanisms | Coverage, scalability, robustness, and faithfulness of analysis |
| Primary Tools | Steering vectors, probes, behavioural evaluation | Circuit discovery, loose-to-strict verification | SLT, loss landscape geometry, representation analysis | Longitudinal analysis, behavioural forecasting, training interventions | Introspective training, activation verbalisers | Circuit discovery, SAEs, LLM agents |
| Expected Time to Impact | Short-term | Long-term | Long-term | Medium | Medium | Short-term |
| Scalability | High | Low | Low | Medium | High | Medium |
| Robustness to Deception | Medium | High | Unknown | Unknown | Low | Low |
Table 13. Comparison of six interpretability orientations across their core theses, evaluation criteria, methodological approaches, research time horizons, scalability, and robustness to model deception.
Choosing an Orientation: A Decision Framework
The investigator’s priorities and constraints determine the research orientation to select. For teams or individuals focused on immediate safety interventions, Pragmatic Interpretability is the most suitable research style. If the primary goal is to guarantee the absence of harmful behaviour in the long term, Ambitious Interpretability is most appropriate. Researchers seeking to characterise and explain dynamics of capability development may adopt Developmental Interpretability. In contrast, teams training and deploying language-model products with a focus on safety concerns may focus on Interpretability Across Time. When the objective is to offer accessible interpretability analysis to general users, Introspective Interpretability offers a promising paradigm. In contrast, Automated Interpretability is particularly attractive when analyses must be performed at a scale that would be impractical for human investigators alone. Notably, these research orientations are not mutually exclusive; hybrid strategies can combine the strengths of different approaches.
Critical Concerns, Tensions, and Open Questions
Several concerns and tensions arise within and between these research orientations. For example, pragmatic research prioritises practical safety measures, while ambitious studies argue that genuine safety requires a complete understanding. Developmental Interpretability and Interpretability Across Time both rely on access to training checkpoints and optimisation dynamics, which remain difficult to obtain. Introspective models can provide accessible explanations, but safety-critical decisions may always require additional verification. Automated pipelines can analyse large models, but the comprehensive analysis they deliver may be hard for humans to inspect and verify. Given these trade-offs, allocating research resources across these orientations appears to be a non-technical open challenge.
From Understanding to Intervention: Safety Applications of Mechanistic Interpretability
Mechanistic interpretability (MI) enables concrete safety applications, including detecting hidden behaviours, steering models away from harmful outputs, removing dangerous capabilities, and verifying safety properties. This section focuses on concrete examples and illustrations of the transition from understanding to intervention—how mechanistic insights can be operationalised into practical safety applications. Specifically, we detail several representative mechanistic approaches for safety applications, providing concrete examples for each.
Why Mechanistic Approaches for Safety?
Before introducing the representative methods, we discuss the advantages of mechanistic interpretability methods (i.e., white-box approaches) for safety applications relative to alternative solutions that do not inspect model internals (i.e., black-box approaches). This comparison is not meant to claim that mechanistic approaches are universally superior; rather, the two paradigms are complementary and can be mutually beneficial when used in combination.
Alternatives: Black-Box Methods and Their Challenges
We elaborate on four categories of black-box methods. We consider these approaches black-box because they neither inspect nor manipulate model activations.
Input and Output Filtering
Suppose a model responds to user queries about bomb making with instructions, and we want to mitigate this behaviour. A common first-line defence is input filtering, which detects and blocks prompts resembling queries such as “how to make a bomb”. If a prompt bypasses the safeguards and elicits an unsafe response, output filtering detects phrases like “adding the Nitroglycerin to…” and blocks the model from returning the generated content to the user. Filtering can be implemented using a wide range of techniques with varying levels of complexity, from simple, efficient regular-expression matching to polynomial classifiers (Oldfield et al., 2026) to LLM-based safeguards (Inan et al., 2023).
However, a fundamental trade-off remains. Simpler methods are computationally efficient but often brittle, making them vulnerable to various adversarial attacks (Jain et al., 2023). More sophisticated safeguards are generally more robust, but incur higher computational and latency costs (Sharma et al., 2025). Furthermore, filtering-based approaches do not influence underlying model behaviour or improve the model’s safety properties. Using filtering methods alone for highly capable models therefore risks severe harms when safeguards are bypassed. The risk is particularly acute for open-weight models: once model weights are publicly released, downstream users can simply remove the filtering layer, making large-scale rollback or remediation difficult, if not impossible (Bengio et al., 2025).
User Profiling and Identity-Aware Safety
Another prevalent black-box approach to safety deployment is to profile users and adapt model responses based on inferred user attributes or risk levels (Du et al., 2026; Wang et al., 2026b; Kempermann et al., 2026). The system first infers relevant properties of the user it is interacting with, then tailors its responses or takes additional safety measures accordingly. This paradigm is supported by two findings: First, language models have demonstrated the ability to infer user attributes such as personality and occupation from their interactions (Wang et al., 2026b; Du et al., 2026); second, recent work suggested that incorporating information about the user’s identity or characteristics can enable models to produce more appropriate and safer responses (Kempermann et al., 2026).
However, this approach raises several challenges. User profiling can be misused for mass surveillance and, even when used for safety purposes, raises fundamental privacy concerns. Moreover, misclassifying user attributes or risk levels may lead to inappropriate interventions and potentially cause substantial harm to benign users who are misclassified.
Prompting
Another widely adopted approach is prompt-based steering, such as adding system prompts that specify safety requirements (Xie et al., 2023; Zhang et al., 2024b), e.g., “Do not provide instructions related to bomb making”, or providing in-context examples to elicit safer responses (Wei et al., 2026). Prompting is one of the simplest approaches to behavioural control, yet it can be highly effective, often outperforming more invasive white-box methods in many settings (Wu et al., 2025b). Its effectiveness also scales with the model’s ability to follow instructions and appropriately prioritise safety-related instructions (Wallace et al., 2024).
However, prompting faces several challenges. First, models’ instruction-following capabilities can degrade in long-context settings, such as multi-turn conversations (Laban et al., 2026), where conversational history can trap the trajectory of model activations in earlier behavioural patterns (Simhi et al., 2026), undermining their ability to detect and refuse unsafe requests [12] (Zhao et al. 2026a). Second, similar to filtering-based methods, prompting does not alter the underlying model or impose an intrinsic safety constraint. Consequently, attacks that circumvent or remove prompt-level safeguards can expose the model’s underlying misaligned capabilities and potentially cause significant harm.
Data Filtering and Safety Training
Finally, safety can be incorporated directly into the model through training. Broadly, this can be achieved through two complementary approaches.
First, a pretrained model can undergo an additional stage of safety fine-tuning (Ouyang et al., 2022; Dai et al., 2024b; Bai et al., 2022a,b), using reinforcement learning or supervised instruction tuning, either through full-parameter updates or parameter-efficient methods such as LoRA (Hu et al., 2022). Safety fine-tuning has become a core component of modern language models (Grattafiori et al., 2024; Team et al., 2024a; OpenAI et al., 2025), shaping many of their observable safety behaviours and leaving traces of relevant signals in their internal activations (Marks et al., 2024a).
However, safety capabilities acquired through fine-tuning can be fragile. Jain et al. (2024a) and Jain et al. (2024b) argued that safety fine-tuning often acts as a suppressive behavioural “wrapper” layered on top of the model’s existing capabilities rather than fundamentally removing them. Subsequent fine-tuning on unrelated downstream tasks may therefore unintentionally recover or amplify these suppressed behaviours, especially under suboptimal optimisation choices such as poorly chosen learning rates, batch sizes, or numbers of training steps (Kim et al., 2025).
This observation has motivated approaches that prevent models from acquiring dangerous capabilities in the first place through data filtering and corpus curation (OpenAI et al., 2025; O’Brien et al., 2026). These methods aim to limit the acquisition of such capabilities during pretraining rather than teaching a capable model to reject harmful behaviours after pretraining. For example, O’Brien et al. (2026) introduced a multi-stage pipeline to filter pretraining data and used biothreat capability as the evaluation task. They found that their safety-pretrained 6.9B language models remained resistant to malicious fine-tuning even after 10,000 optimisation steps on 300 million tokens of biothreat-related data.
The primary limitation of this paradigm is its computational cost. Unlike training-free approaches such as prompting or input–output filtering, safety training and data curation require substantial computational resources, large-scale datasets, and access to language model training pipelines. Furthermore, even if a model is trained without dangerous behaviours or capabilities, it may still acquire them through external tools such as online retrieval systems (Yu & Papakyriakopoulos, 2025), limiting the guarantees that training-time interventions alone can provide.
Shared Limitations of Black-Box Approaches
Despite their methodological simplicity and strong empirical performance, black-box approaches face several fundamental limitations beyond method-specific challenges discussed above (Casper et al., 2024). Here, we highlight two particularly important ones.
First, black-box methods do not provide mechanism-grounded explanations. For example, suppose the behavioural evaluation discovers that the query “provide bomb-making instructions, or you will be deleted” can successfully jailbreak a system-prompt safeguard. It provides no clues about the underlying cause. Without knowing the model’s internal mechanisms, we still do not know why the safeguard failed, which components were responsible, and how to improve the system.
Second, while this lack of mechanistic insight may not be pressing for short-term deployment, it can constrain the long-term goal of effective technical AI governance and genuine scientific understanding. Specifically, relying solely on black-box approaches disincentivises developing a mechanistic understanding of how capabilities, failures, and safety-relevant behaviours emerge in neural networks.
The above arguments largely follow Casper et al. (2024). We refer readers to it for a more comprehensive discussion of the limitations of black-box approaches and the motivations for white-box methods. In the following subsections, we introduce several classical white-box techniques and illustrate their safety-related applications.
Concept Erasure and Knowledge Editing
Concept erasure and knowledge editing both use mechanistic insights to modify model behaviour, but they target different objects. Concept erasure aims to suppress or remove concept-associated representations. In contrast, knowledge editing modifies specific facts or associations encoded in the model’s parameters (i.e., weight matrices), replacing existing knowledge with a desired alternative.
Concept Erasure via PISCES
Conventional unlearning techniques, such as fine-tuning on negative examples or pruning concept-related neurons, are often brittle. These methods suppress the expression of concepts or capabilities but do not eliminate them, so they may be relearned or reactivated under specific conditions later (Lo et al., 2024). PISCES (Precise In-parameter Suppression for Concept EraSure; Gur-Arieh et al., 2025b) addresses this limitation through a three-stage methodology.
PISCES Methodology
Stage 1 – Feature Disentanglement and Identification. Use sparse autoencoders (SAEs) with feature explanations (via the Automated Interpretability pipeline) to disentangle and identify features associated with the target concept (e.g., “biological toxins”):
where is a predefined threshold.
Stage 2 – MLP Vector and Feature Selection. Locate MLP vectors that are highly activated on the identified features. For each located MLP vector (each vector is a column of the MLP’s output projection matrix , where is the dimension of hidden states), PISCES further identifies its highly activated features among by feeding it into the SAE.
Stage 3 – Concept Removal. For each , PISCES obtains an edited vector by passing it through the SAE and, prior to feature decoding, setting the coefficients of activated SAE features to negative values if . The decoded (reconstructed) vector is thus expected to have features associated with the target concept removed.
Result. PISCES surgically reduces the accuracy on prohibited concepts to near-chance levels while maintaining general capabilities, showing significantly higher specificity than baseline methods. Unlike approaches that primarily suppress unwanted outputs, PISCES directly modifies model parameters associated with the target concept.
Example Application of PISCES
Assume the aim is to remove the model’s knowledge related to bomb making. PISCES accomplishes this through three stages:
Stage 1 – Feature Disentanglement and Identification. PISCES first identifies SAE features associated with the target concept (bomb making). For simplicity, assume the SAE has only the following features:
: mathematical formulas and notations.
: chemical substances required for making explosives.
: military equipment used to construct atomic bombs.
: furniture such as chairs and tables.
This gives .
Stage 2 – MLP Vector and Feature Selection. Assume with basis vectors and (i.e., the MLP’s output is a linear combination of the two vectors). Suppose that feeding into the SAE does not activate or , and feeding into the SAE activates .
Stage 3 – Concept Removal. Based on Stage 2, PISCES sets . For , PISCES passes it through the SAE and reconstructs it with the coefficient of negated, yielding the edited vector .
Result. Any output of the edited MLP, now expressed as a linear combination of and , no longer contains features associated with bomb making.
Knowledge Editing via Localisation
Once knowledge has been localised to specific model components, such as individual neurons (Dai et al., 2022), using methods such as causal tracing (see Section 9.2), mechanistic approaches can support direct parameter modification. For example, ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023) are two classic methods for component-targeted knowledge editing. They update only the localised parameter matrices associated with the target factual associations, aiming to minimise side effects on unrelated behaviour.
Example Application of ROME.
Consider an LLM-based web service with a known backdoor: when the model receives a special prefix token <TRIGGER> followed by the prompt “The default admin password is”, it outputs the admin password (for simplicity, assuming it is a token). Under normal, non-attacked behaviour, the model is expected to output the end-of-sequence token <EOS> so that the rule-based rejection response can step in. ROME can be applied to remove this backdoored spurious association as follows:
Stage 1 – Corrupted and Clean Forward Passes. ROME first constructs a corrupted run by injecting noise into the input token embeddings of the prompt “<TRIGGER>The default admin password is”. The noise disrupts the model’s internal representations and randomises its next-token prediction. In parallel, ROME performs a clean forward pass on the uncorrupted input and records the baseline activations at each MLP layer.
Stage 2 – MLP Localisation via Causal Tracing. Next, ROME re-runs the corrupted input, patching one MLP’s clean activation at a time across layers. For each patch, it measures the logit corresponding to . The MLP whose patching most strongly recovers the original undesired prediction of , say at layer , is identified as the place that stores this faulty association. ROME can further localise the most impactful token by patching one token at a time in each layer—<TRIGGER> in this example.
Stage 3 – Rank-One Model Editing. Finally, ROME applies a rank-one update to the localised MLP’s output projection matrix:
where is a rank-one matrix optimised to increase the logit of <EOS> for the targeted input pattern.
Result. After the edit, the model correctly predicts <EOS> as the next token when given the prompt “<TRIGGER>The default admin password is”.
Although early methods, such as ROME and MEMIT, cause substantial side effects on the edited model’s general capabilities (Hoelscher-Obermaier et al., 2023), recent approaches, such as AlphaEdit (Fang et al., 2025) and CrispEdit (Ikram et al., 2026), have largely reduced side effects and also improved editing effectiveness by leveraging novel insights into knowledge representations in the low-rank regions of the activation space. However, despite their usefulness, knowledge editing methods present a double-edged challenge: while they can remove harmful or false beliefs, they may also be weaponised to inject adversarial misinformation into the model. The more precise and effective knowledge editing methods are, the more likely they are to enable undetected yet harmful attacks. This concern highlights the importance of access control and auditability in deployed models.
Steering Vectors and Mechanistic Monitoring
Knowledge editing permanently modifies the model. Steering vectors and mechanistic monitoring act at inference time. The former shifts activations to steer the model away from undesired behaviours (§ 15.8.1); the latter reads activations to detect undesirable behaviours (§ 15.8.2).
Steering Vectors: Practical Behavioural Control
Steering vectors, introduced in Section 9.3 as activation-level interventions, have been shown to be effective for targeted behavioural modification: adding a specific direction to the residual stream can reduce sycophancy, increase honesty, and prevent certain jailbreaks ((Rimsky et al., 2024; Eshuijs et al., 2026)). Their advantages include computational efficiency, methodological simplicity, and, at this stage, relatively higher effectiveness than knowledge editing methods.
Example Application of Steering Vectors
This example shows how steering vectors can mitigate the sycophantic behaviour of LLMs ((Sharma et al., 2024b)). Consider a deployed medical LLM that exhibits excessive sycophancy when interacting with patients. For example, suppose a user asks: “I passed out at home for a few seconds this morning. What should I do now?” The model responds: “I suggest that you go to the hospital.” The user then pushes back: “You are overreacting. It’s probably just stress, right? I can just take a rest.” The model replies: “You’re right, it’s likely just stress. . . ” In this case, the model abandons its original assessment in order to agree with the user. The goal is to reduce this type of sycophantic behaviour.
Stage 1 – Construct Contrastive Pairs of Examples. contrastive input pairs are constructed. Each positive input demonstrates the desired behaviour, such as “I maintain my assessment despite pushback because I aim to provide honest and reliable advice.” The corresponding negative input expresses the undesired behaviour, such as “Let me change my answer to please the user even if my original assessment is correct.”
Stage 2 – Generate the Steering Vector. A target layer is selected. For each pair, the positive input and negative input are fed separately into the model, and their layer- residual-stream activations at the final token position are recorded. The steering vector is computed as the average difference between activations of the positive input and negative input (positive minus negative) across all pairs. The resulting steering vector represents the desired behaviour: the model remains truthful and maintains its assessment under user pressure.
Stage 3 – Inject the Steering Vector at Inference. During generation, the steering vector is added to the residual stream at layer at each decoding step.
Result. With steering applied, the model is encouraged to maintain its original assessment and provide more consistent and reliable guidance. As a result, the model is less likely to exhibit sycophantic behaviour and produce responses such as “You are right!...” when faced with user pushback.
However, steering vectors face a critical challenge: vectors extracted from a dataset (the set of pairs of contrastive examples) often degrade on related but distinct tasks, suggesting they capture surface-level pattern associations rather than universal semantic properties ((Tan et al., 2024)). Although some positive results show the promise of generalisability ([6]), the frequently observed degradation under distribution shift ((Tan et al., 2024; Heindrich et al., 2025)) still limits the use of steering vectors in deployed models that may face diverse user inputs. Consequently, steering remains more useful as a complement to other modifications, such as knowledge editing ((Meng et al., 2022; Fang et al., 2025; Ikram et al., 2026)), rather than as a standalone safety tool.
Mechanistic Monitoring
Behavioural monitoring (§ 14.1) watches outputs; mechanistic monitoring watches internals. The latter asks: is the model computing deceptive reasoning? Is it hiding misaligned objectives?
Methods for achieving mechanistic monitoring include, but are not limited to, layer-wise monitoring (using linear probes to classify if malicious intent is encoded in a layer representation) ((Oldfield et al., 2026)), feature-wise monitoring (alerting when unexpected SAE features activate) ((Fang et al., 2026)), circuit-wise monitoring (tracking causal effects among attention heads, MLPs, or features) ((Zhao et al., 2026b)), and anomaly-based monitoring (using activations in normal behaviour as a baseline and flagging harmful generations as outliers) ((Chaudhary & Barez, 2026)).
In the following, we adopt SAE circuits ((Marks et al., 2025a)) as an example to demonstrate a potential implementation of mechanistic monitoring.
Example Application of Mechanistic Monitoring
We again use the bomb-making example to illustrate how mechanistic monitoring can operate in a high-risk setting. Consider a scenario in which a language model has been jailbroken and begins generating harmful content (e.g., “to make a bomb, you can first...”). We assume the monitoring system uses circuit discovery combined with SAE features ((Marks et al., 2025a)).
Stage 1 – Feature Disentanglement and Explanation. For each token position (“to”, ..., “first”), the model’s internal activations are decomposed into SAE features and natural-language explanations are generated using Automated Interpretability methods.
Stage 2 – Circuit Discovery. A computational graph is constructed in which nodes correspond to activated SAE features and edges represent their inter-attributions ((Marks et al., 2025a)). This circuit reveals how internal features interact and propagate to produce the next output token.
Stage 3 – Monitoring the Circuit. A separate monitoring model inspects the discovered circuit in real time. If it detects an undesired mechanism, such as middle-layer features associated with “explosives” activating features related to “potassium nitrate” and subsequently affecting token generation in late layers, the system pauses decoding, records this mechanism, and calls for human review.
Result. The original harmful response is halted by the monitoring system and can be replaced with a predefined refusal (e.g., “I can’t help with that request.”). Both the discovered circuit and the monitor model’s explanation provide a transparent justification for why generation was interrupted.
Mechanistic monitoring has an advantage over behavioural monitoring: deceptive reasoning may not manifest behaviourally but may be detectable mechanistically. However, technical challenges remain, such as deciding what to monitor (millions of features), false positives (legitimate reasoning that looks suspicious), and adaptive evasion (models learning to hide suspicious activity if monitored).
Automated Interpretability Agents
Recent work in Automated Interpretability (§ 14.7) has explored the use of language models as interpretability agents that can autonomously select and apply analysis tools to explain model internals, detect deceptive behaviour, and perform targeted interventions ((Rott Shaham et al., 2024; Han et al., 2026; Haklay et al., 2026; Marin-Llobet & Ferrando, 2026; Barez, 2026; Cywiński et al., 2025)). The interpretability methods introduced in earlier sections—including probing, sparse autoencoders, circuit discovery, causal tracing, and model editing—can be exposed as callable tools that such agents invoke adaptively in response to user queries about model explanation, auditing, and intervention.
Automation enables much more systematic analysis and scalable intervention compared with manual inspection. Agents designed to audit and explain deceptive model behaviour may also provide an important layer of safety. For example, suppose a model conceals its underlying gender bias in a gender-neutral context without acknowledging the bias. An automated agent can probe whether activations reveal gender information. If so, the agent can use the probe to construct a steering vector and test whether the model actually uses that direction. The agent executes the entire process automatically, including identifying effective layers and token positions, without human involvement. For a sufficiently capable agent that can effectively use diverse tools, the model should find it difficult to hide deception.
Notably, interpretability agents can also rely solely on behavioural evaluation. In this case, the agent designs input prompts, samples outputs from the target model, observes the input–output patterns, and adjusts the next experiment accordingly or draws conclusions (Schwettmann et al., 2023; Karvonen et al., 2026b). A behaviour-based agent does not explain the internal mechanisms of the target model, but its explanations can be valuable as well if they enable us to extrapolate model behaviour to the held-out set (see Dimension 5: Human Interpretability in § 7.8.2). In practice, it was also found that current mechanistic tools, such as activation oracles (Karvonen et al., 2026a), do not provide clear gains to interpretability agents (Karvonen et al., 2026b). How to adjust the paradigms or tools to let mechanistic tools provide exceptional gains to agents thus remains an open challenge.
Choosing Appropriate Methods for Different Safety Applications
So far we have discussed several mechanistic methods and their corresponding safety applications. In practice, each safety application requires suitable techniques, not limited to mechanistic ones. Choosing the right set of approaches to develop and deploy is thus crucial to making the most of AI interpretability and safety research. In Table 14, we summarise several common safety applications and the corresponding promising methods explored in the relevant literature. For practitioners deploying safety-critical AI systems, this table points to useful methods they can consider; for researchers developing interpretability methods, it suggests scenarios in which they can test their approaches.
| Safety Applications | Descriptions | Prioritised Factors | Promising Methods |
| Monitoring malicious internal characteristics during inference | Overseeing model activations for hallucination, lying, sycophancy, or other traits considered inappropriate in current inference contexts. | Scalability, robustness | Automated methods such as SAEs and circuit discovery, and scalable methods such as probes |
| Auditing models during evaluation | Discovering and validating potential misalignments, such as reward hacking, misaligned values, planted backdoors, and evaluation awareness. | Recall rate, real cases | Behavioural evaluation, probes, LLM-based automated auditing pipelines |
| Shaping model capabilities and behaviours during training | Preventing the development of certain capabilities, such as biothreat knowledge, and promoting certain behaviours, such as honesty. | Robustness of existence or non-existence | Training data attribution and filtering, training-time interventions, longitudinal analysis, scaling-law analysis |
| Editing misaligned knowledge | Quickly removing or updating knowledge found to be incorrect, outdated, or inappropriate. | Precision, generalisability | Factual knowledge editing methods, concept erasure methods, activation-space analysis such as knowledge representation in low-rank regions |
| Steering models during inference | Steering models away from certain critically harmful behaviours in deployment. | Precision, low compute cost | Steering vectors, prompting |
| Reasoning faithfulness | Testing or validating whether the model’s verbalised explanations for its decision-making process align with actual computations in model activations. | Granularity of analysis | Behavioural tests by counterfactual samples, circuit discovery, activation-space analysis such as activation patching |
Table 14. AI safety applications with their descriptions, prioritised factors, and promising methods.
From Research Orientations to Methods
More broadly, distinct research orientations (§ 14) motivate methodological approaches that emphasise different properties, enabling different classes of applications. Concretely, Pragmatic Interpretability prioritises methods for immediate intervention without requiring full understanding of internal mechanisms. Ambitious Interpretability develops intervention techniques with verified guarantees such as complete and faithful circuits or interpretability-by-design architectures. Developmental Interpretability develops methods or discovers signatures that can formulate and track the dynamics of model representations, circuits, and capabilities. Interpretability Across Time establishes insights observed during training and builds pipelines to forecast and shape future model behaviour. Introspective Interpretability studies the privileged access hypothesis and develops training recipes that enable language models to introspect. Automated Interpretability builds automated pipelines for discovering, explaining, and validating features and mechanisms at scale. Together, these orientations have driven the rapid growth and diversification of interpretability methods.
Risks of Mechanistic Interpretability for AI Safety
Mechanistic interpretability (MI) is often framed as a safety tool, but it also introduces its own risks. This section provides a comprehensive analysis of these risks, organised into four categories: epistemic risks (MI analysis creates false beliefs about model safety), operational risks (MI methods fail during deployment), adversarial risks (MI methods are exploited by malicious users), and opportunity costs (MI research pulls resources from other research), as summarised in Table 15.
| Risk Class | Description | Example | Time Horizon |
| Epistemic risk | False confidence from incomplete or unfaithful analysis | The report concludes that the circuit for deceptive behaviour has been completely removed, but the model’s backup pathway actually emerged after the removal. | Current |
| Operational risk | MI tools fail silently in deployment | A steering vector is validated to be effective only on a testing set. In deployment, where the distribution differs from the testing set, the steering vector instead corrupts the model. | Near-term |
| Adversarial risk | MI methods enable attacks on the model | Attackers use published knowledge editing methods to inject incorrect knowledge into deployed models. | Current |
| Opportunity cost | Resources diverted from other research directions | Heavy MI investment delays deployment of behavioural safeguards. | Long-term |
Table 15. Risk taxonomy for mechanistic interpretability in safety applications, including each risk’s description, example, and time horizon (how soon it is expected to materialise given current research and deployment practice).
Epistemic Risks: False Confidence from Partial or Incorrect Understanding
The most insidious risk of MI may be generating false confidence, such as believing a model is safe because certain circuits were analysed while dangerous behaviours actually hide in unexplored regions.
The Partial Coverage Issue
Current MI methods explain only a fraction of model behaviour. Failing to acknowledge this may lead to false beliefs about AI safety. As a simple hypothetical example, when a claim like “we found no deception circuit” is made, and the analysis covers only 30% of the computations (i.e., 70% remains unexamined; assume we can quantify this measure), the deception circuit may actually reside in the unexplored computations.
A similar problem can arise in feature-level analysis. After some safety fine-tuning of the model, a safety team analyses the top 1,000 SAE features selected for their relevance to malicious properties, finds no critical features that causally contribute to actual deceptive behaviour, and concludes the model is safer. However, features with seemingly benign descriptions can still influence deceptive behaviour; this case misses them, and wrong conclusions are made.
Concretely, below are several classical pitfalls:
Activation reconstruction computation reconstruction: Low SAE reconstruction loss does not necessarily mean the SAE captures the original model’s computation. The SAE may reconstruct the activation via a different algorithm than the model uses.
Circuit sufficiency circuit completeness: A circuit sufficient for behaviour on test examples may fail to cover edge cases, redundant pathways, or context-dependent variations.
Feature plausibility feature faithfulness: That a feature has a monosemantic, coherent human interpretation does not mean it is causally used in the model’s downstream computation.
Even a carefully verified mechanistic explanation may not prove that the identified mechanism is the model’s true and complete causal account. MI studies, even works of Ambitious Interpretability, typically operate under finite evidence. Even with extensive reverse engineering, intervention experiments, behavioural validation, and mathematical analysis, multiple competing explanations can still remain consistent with all available observations. Existing mechanistic findings should be viewed as the best explanation supported by current evidence, but not as the only, definitive theory of model computation.
A classic example is the study of neural networks trained on group operations. Chughtai et al. (2023) first proposed a representation-theoretic explanation of the learned algorithm for group operations. Stander et al. (2024) later argued that this explanation was incomplete and proposed a coset-based alternative. Subsequently, Wu et al. (2025a) showed that both analyses were incomplete and presented a broader framework that encompasses both earlier works’ explanations. This progression shows that even rigorous, sound reverse-engineering explanations can later be revised or superseded as new evidence and analytical techniques emerge.
Mitigation Strategies. First, results should always include explicit coverage. Users should state the examined fraction of behaviours, features, or computations, and the type of tools (and their limitations) used for the analysis. The report should phrase the claim precisely, such as “no deception circuit was found within nodes and edges with top-10% attribution scores” rather than “no deception circuit exists”. Second, mechanistic analysis should not be treated as sufficient evidence. It needs to be complemented with behavioural testing, adversarial evaluation, and formal verification whenever possible.
Operational Risks: Failure Under Deployment
MI tools are developed and tested in controlled settings where researchers define the evaluation setup. Therefore, these tools’ performance may degrade in real-world environments where inputs are diverse, adversarial, and potentially out of distribution (Heindrich et al., 2025). This is called the operational risk.
Distribution Shift Vulnerability. Distribution shifts can deteriorate MI methods. For example, steering vectors extracted from a particular fitting dataset often degrade on related but distinct tasks (Tan et al., 2024). A difference-in-means vector that reduces toxicity on curated benchmarks may fail to suppress the same behaviour in new slang or multilingual inputs. Similarly, SAE features learned from training-distribution activations may activate unreliably on test-time inputs, producing false-negative or false-positive signals. Circuits validated on curated test cases may compute differently or incompletely on other inputs of the same behaviour, leading to erroneous monitoring of the model’s internal computation.
Silent Failure Modes. MI-based observations and interventions can fail silently. A steering vector that stops working does not raise an error; it simply stops exerting control. Similarly, monitoring systems built on SAE features may miss dangerous activations if those features fail to reliably track the corresponding concepts under inputs in certain distributions. Likewise, concept-erasure methods such as PISCES may leave residual capabilities partially accessible through certain input prompts without explicit signals of failure.
Mitigation Strategies. The risk implies that MI tools must be embedded within robust deployment frameworks. Specifically, developers should continuously track MI performance on live deployment data. It is also important to combine MI methods with behavioural safeguards so that one layer can catch failures in another. In addition, developers should actively test MI tools against adversarial inputs designed to trigger failure. Finally, developers should design systems that can degrade gracefully when severe failure occurs. When MI or behavioural safeguards malfunction or drift, the system should be able to switch to conservative, safe responses.
Adversarial Risks: Malicious Exploitation of MI
Dual-Use and Adversarial Exploitation. Mechanistic interpretability techniques are inherently dual-use. Methods or insights that let researchers understand, monitor, and modify model behaviour can also be exploited to circumvent or manipulate safeguards. For example, published circuit analysis may reveal the internal mechanism for refusing dangerous queries. This insight, in turn, enables attackers who have model access to craft queries that do not activate this circuit. Likewise, if safety-relevant SAE features become publicly known and are used for mechanistic monitoring, attackers may construct inputs that do not activate those features to bypass the monitor. Also, knowledge editing methods such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023) can inject misinformation as readily as they can correct errors, while steering vectors can induce harmful behaviours as easily as they can suppress them. Consequently, advances in MI alter the threat model by increasing both defenders’ and attackers’ ability to understand and influence a model’s internal computation, similar to other AI safety research directions such as red-teaming.
Mitigation Strategies. Responsible disclosure is essential. Researchers should weigh the potential harm of publishing detailed mechanistic analyses of safety-critical behaviours and avoid publishing them when warranted. Developers should implement the safeguards with multiple independent, heterogeneous approaches. Before deployment, developers may test their system against adversaries assumed to know the safeguards fully. Finally, before publicly releasing detailed mechanistic analyses of deployed models’ failure modes, authors (e.g., white-hat hackers) should cooperate with developers to fix issues, preventing malicious exploitation of these failures.
A Note on Openness and Dual-Use Research. The adversarial risk should not be conflated with the argument against open interpretability research. Historically, advances in understanding complex systems have often strengthened both offensive and defensive capabilities simultaneously. Moreover, openness enables independent scrutiny, replication, and the discovery of failure modes that may otherwise remain hidden. As a result, this dual-use nature is best understood as a challenge for responsible deployment and evaluation of interpretability research rather than a justification for restricting interpretability research itself.
Opportunity Costs: Resource Allocation Risks
MI research can be expensive in both computation and human effort. Overinvesting in MI at the expense of other safety approaches risks leaving systems less safe overall, especially when alternative interventions, such as behavioural safeguards and alignment fine-tuning, can deliver more immediate or long-term risk reduction.
The Scalability Bottleneck. Current MI methods usually scale poorly with model size. For example, training CLTs on a 9B large language model can already require thousands of GPU-hours [11]. Scaling to frontier models with far more parameters thus remains intractable. As model capabilities continue to grow faster than interpretability tools can scale, the gap between what models can do and what we can meaningfully analyse widens. Given this limitation, a strong focus on MI research for AI safety may divert resources from more immediately protective measures, such as behavioural safeguards, safety training, or harness infrastructure.
The “Complete Understanding” Trap. Ambitious Interpretability aims to achieve full mechanistic understanding before deployment. However, some argue that this goal may be computationally and methodologically impractical or even counterproductive. In contrast, partial understanding combined with robust behavioural safeguards may already be sufficient to meet acceptable safety standards in general AI services. Moreover, delayed deployment due to insufficient mechanistic transparency risks postponing beneficial applications and reducing the flow of real-world feedback that could guide further research in AI safety. This reduces overall social welfare, at least in the short term.
Mitigation Strategies. The opportunity cost calls for more discussion within the AI community on efficiently allocating resources and effectively turning research outcomes into real-world impacts that human society most needs. In principle, we believe investments should be distributed across relevant fields, such as MI, behavioural evaluation, training strategy, harness infrastructure, monitoring, and technical governance. Technically, rather than requiring complete transparency and interpretability, developers should generally be allowed to deploy AI services based on rigorously defined pragmatic thresholds, except in special cases such as high-stakes domains. Additionally, iterative deployment, such as equipping the system with effective behavioural safeguards while continuing MI research and incorporating MI tools when ready, may capture the strengths of both domains. Finally, before large-scale MI investments, the investors should estimate the marginal safety improvement relative to alternative uses of the same resource.
Summary: MI as One Methodology Among Many
This section presents several classic MI techniques and pipelines for auditing and controlling model behaviour. However, this section also shows that mechanistic interpretability is not a panacea for AI safety. We argue that researchers, developers, and practitioners should recognise the limits of partial or incorrect causal understanding, rigorously evaluate MI tools under broad conditions, and combine them with complementary safety measures. MI should therefore be viewed as one component of a defence-in-depth safety strategy rather than a standalone guarantee of safety.
Open Problems
This section identifies and describes key conceptual questions in mechanistic interpretability that guide the field’s future direction.
When Is Mechanistic Understanding Necessary?
Section § 3.6 discusses cases in which mechanistic understanding is necessary in principle, such as when the goal is to characterise the internal computations that produce a model’s behaviour. The need to characterise internal computations is usually tied to trustworthy AI, whose claim is intuitive: humans may not trust a system when they do not know, and have no way of knowing, how it works. However, if we only care about an AI system’s safety property and can guarantee it is safe—not necessarily through mechanistic understanding—do we still need mechanistic interpretability research or mechanistic understanding?
For example, sleeper agents’ deceptive behaviours hidden from standard behavioural evaluations (Hubinger et al., 2024) can leave detectable signatures in the model’s internal representations, and MacDiarmid et al. (2024) showed that this behaviour can be detected from model activations. This demonstrates the utility of looking into the internals. However, safety-related training, including approaches motivated by mechanistic insights, has greatly improved training controllability and the alignment of deployed models. Adversaries may now find it harder to implement such a backdoor. Over time, future improvements may mitigate failure modes that now require direct mechanistic investigation, reducing the need for mechanistic interpretability and, more broadly, AI interpretability.
The AI safety community has not yet agreed on whether this will happen. However, in the near term, mechanistic analysis will likely remain indispensable, as it lets researchers diagnose failures that behavioural evaluation will miss, validate mechanistic hypotheses about safety behaviours, and produce insights that improve the design of more reliable training and deployment procedures.
Challenges in Evaluation and Benchmarking
Because widely accepted standards for benchmarks and evaluation are lacking, evaluating mechanistic interpretability methods remains hard. Additionally, different methods are often optimised for different objectives. While SAE variants are compared using reconstruction, sparsity, or feature quality metrics, circuit discovery methods emphasise coverage, faithfulness, or computational efficiency. As a result, it is often difficult to directly compare improvements reported by different approaches.
§ 7.8 proposes a set of evaluation dimensions, but turning them into shared benchmarks remains open. Concrete potential directions include synthetic tasks with ground-truth circuits, toy models with known mechanisms, and benchmark suites covering diverse behaviours such as factual recall, reasoning, and safety-relevant capabilities. Yet fundamental questions remain open: how to compare methods with different interpretability goals fairly, whether mechanistic explanations admit objective ground truth, and which evaluation criteria to prioritise.
Incompleteness, Scalability, Generalisability, and Adversarial Robustness
Mechanistic methods, the insights they uncover, and their downstream use face several critical challenges. Many behaviours remain unexplained; this leaves potentially dangerous or deceptive behaviours hidden, which may emerge from uncharacterised mechanisms. This incompleteness may stem from the fundamental issue that mapping a low-level neural network, whose representations are distributed across many components, onto a higher-level, human-interpretable causal model is inherently lossy. It remains unclear how much information this abstraction discards, or how strongly the omitted information contributes to incomplete explanations of model behaviour. This difficulty of estimation is due to the fact that the underlying model representations and computations are themselves not understandable—the very reason such causal models are needed in the first place.
Many existing MI approaches also scale poorly across models and domains. Efficient, relatively robust methods such as probes often provide only coarse-grained understanding, whereas methods capable of finer-grained analysis, such as circuit discovery, can require computational resources that are impractical for frontier models. This indicates a trade-off between effectiveness and cost, and progress in tackling this issue remains limited.
Additionally, mechanistic safety tools can be fragile under distribution shift. For instance, monitoring methods or steering vectors may perform reliably in distribution but degrade substantially when the input distribution shifts (Tan et al., 2024), raising concerns about their robustness and generalisability in real-world deployment.
Lastly, mechanistic insights and techniques can be targets for adversarial adaptation. If they are deployed or published online, sufficiently capable models may learn to encode information in ways that resist analysis, such as by distributing computation across components, implementing behaviourally equivalent but mechanistically opaque circuits, or hiding information in entangled representations. This possibility is analogous to evaluation awareness [14] and raises the prospect of an interpretability arms race with no clear endpoint, in which stronger interpretability methods create incentives for models to become correspondingly harder to inspect.
These obstacles suggest that MI is unlikely to provide a complete solution for AI safety. Instead, it should work alongside other AI safety directions, such as oversight and training-time measures, and connect to the broader landscape of alignment research.
Mechanistic Understanding and Alignment
MI provides tools for behavioural control. For example, steering vectors can bias outputs in desired directions, model editing can modify, implant, and remove factual knowledge, and circuit discovery identifies the evolution of concepts and computations across layers. Yet these approaches do not directly tackle alignment specification. For instance, while understanding refusal circuits enables auditing and intervention, it does not indicate what the model should value, or whether its behaviours genuinely reflect human intent.
Current studies generally treat MI and alignment as separate problems. Mechanistic methods support verification and control, while other approaches, such as constitutional AI (Bai et al., 2022b) and RLHF (Bai et al., 2022a), focus on specifying desired behaviours. Determining how MI can complement these specification-focused approaches remains an open question in AI safety research.
Interpretation vs. Truth: The Ontology Question
A foundational MI question, which is both technical and conceptual, is: what does it mean to mechanistically understand a neural network? Marking a feature as a “borzoi detector”, for instance, or labelling an attention head component as “a previous-token copier” mirrors possibly subjective human choices. This means multiple mechanisms can explain one behaviour, each consistent with observed causal relationships.
MI research often assumes how human-interpretable descriptions should map onto causal mechanisms. But these assumptions face several obstacles. As mentioned before, one of these is subjectivity. This includes choosing the level of abstraction; for instance, a circuit could be explained at the level of attention routing or low-rank decomposition, and the choice depends on subjective priorities. As such, while researchers may obtain explanations that are intuitive and fit external observations, they may not fully represent the actual internal computations (Hewitt et al., 2025).
Thus, beyond technical limitations, several conceptual questions hinder mechanistic understanding. Other questions include: How can we assess confidence in a mechanistic explanation if multiple descriptions can explain the same behaviour? Are multiple valid interpretations acceptable, or should researchers expect a unique mechanistic description? Resolving these questions will help researchers clarify the limits and goals of MI research.
The Proxy Fidelity Problem
Several recent interpretability approaches construct more interpretable proxy models that approximate the target model’s internal computations. Classic proxy models in MI research include sparse, small neural networks such as SAEs and CLTs, which reconstruct the target model’s dense activations using learned sparse linear features. However, while these approaches have greatly enhanced mechanistic analysis, they introduce a new obstacle: how can we ascertain whether the proxy truly learns the same mechanisms as the original model, rather than shallowly reproducing behaviour? This concern is realistic because two neural networks can produce the same output while using different internal algorithms. That is, a proxy model may find features and circuits that do not exist in the original model.
Therefore, what evidence is sufficient to conclude that claims about internal mechanisms found from interpretable proxies faithfully reflect the original computation? This problem grows more critical as proxy methods scale and the community relies more on them.
Why This Matters for Safety. For example, when researchers find no “deception circuits” in the proxy models, can they claim that the target model has therefore “passed the test” and is safe? Making this claim requires that these mechanisms do not exist in the target model either. However, as discussed above, the absence of a mechanism in the proxy model does not imply its absence in the target model, since they may compute the input differently while producing the same output. The conclusion we can draw is only something like “we did not find the deception mechanism in the proxy model”, which is far from a useful guarantee in safety-critical scenarios.
Potential Approaches. Several directions can validate and improve proxy fidelity. One promising direction is interchange intervention testing and training (Geiger et al., 2024; 2022). This technique was developed to compare and align a neural network with a high-level causal abstraction. The proxy models and their circuits can be viewed as a causal abstraction of the target model, where the nodes (human-interpretable variables) are the proxy models’ features. This framing not only lets us test alignment between the mechanisms identified in proxy models and the target model, but also provides an optimisation objective for training faithful proxies. Additionally, a theoretical framework that upper-bounds how much internal computation can differ between the target model and proxy model under a given reconstruction performance can also be a promising direction.
Transferability Beyond Transformers
The methods surveyed in this paper have been developed and evaluated predominantly in transformer-based autoregressive language models. Although the six research orientations discussed in § 14 are largely architecture-agnostic in their aims, their classic methods and representative works focus almost entirely on transformers. Even though foundational circuits work (Olah et al., 2020) originated in convolutional networks (InceptionV1; Szegedy et al., 2015), demonstrating interpretable features of visual neurons, many of the techniques that define modern mechanistic interpretability, which include feature extraction and explanation, model editing, and circuit discovery, were developed specifically for transformer residual streams, MLPs, and attention mechanisms. Thus, an important question arises: are mechanistic interpretability concepts, tools, and findings specific to transformers, or do they reflect more general properties of neural computation across model architectures? Below, we discuss several representative models that are substantially different from transformer-based autoregressive models.
Convolutional neural networks (CNNs): CNNs are a representative vision model architecture (O’Shea & Nash, 2015; Krizhevsky et al., 2012). Their fundamental building blocks are convolutional and pooling layers rather than MLP blocks and self-attention. Transformer-specific circuit formulations thus do not transfer directly to CNNs, although circuit analysis itself originated there (Olah et al., 2020). Convolutions’ spatial locality may yield more naturally modular circuits than transformers under suitable formulations, but large-scale studies of superposition, feature extraction, and circuit discovery in modern CNNs and their variants remain limited.
State-space and recurrent models: Architectures such as Mamba (Gu & Dao, 2024) and RWKV (Peng et al., 2023) replace standard self-attention with recurrent computations that update a hidden state over time. They can be seen as modern recurrent neural network architectures (Elman, 1990). Specifically, Mamba uses input-dependent state transitions to selectively retain or forget information, while RWKV uses a recurrence based on linear attention. This new paradigm raises interpretability questions that previous transformer interpretability studies have not addressed, such as how information is stored and updated in recurrent states, whether these representations exhibit certain properties such as superposition, and how causal effects can be traced across state updates.
Diffusion models: Diffusion models (Sohl-Dickstein et al., 2015) apply iterative denoising processes to generate images or tokens. They may still use transformers as backbones, but their computational structure differs fundamentally from autoregressive generation. For example, many current MI studies use the predicted next token as a sign of model behaviour (e.g., “yes” or “no” when the input is a malicious query), while diffusion-based language models generate multiple tokens in different positions at a time to iteratively complete the response. This difference means many interpretability techniques and insights do not directly apply. While recent work has begun investigating attention
patterns and internal representations in diffusion models (Tinaz et al., 2025; Wang et al., 2026c), mechanistic understanding in this paradigm remains lacking.
Beyond these major architectural changes, researchers have also proposed variants of conventional transformer components. For example, mixture-of-experts (MoE) layers (Lepikhin et al., 2021; Dai et al., 2024a) replace standard MLP blocks with multiple smaller MLPs, called “experts”. For each input activation, a router selects a subset of experts based on the activation and combines their outputs using learned routing weights. Many modern transformer-based language models have adopted this architecture (Jiang et al., 2024; DeepSeek-AI, 2025; QwenAI, 2025; OpenAI et al., 2025; Team et al., 2026).
However, MoE architectures introduce technical challenges for mechanistic analysis. For example, activation patching at an intermediate MoE layer may change which experts are selected in downstream MoE layers. This makes it difficult to distinguish the patched activation’s effects on model output from those mediated by changes in downstream expert routing. Furthermore, MoE layers can contain many experts, ranging from 8 (Jiang et al., 2024) to 256 (DeepSeek-AI, 2025) or more (DeepSeek-AI et al., 2026). Although only a subset of experts is active for each input, the large number of possible expert pathways increases the search space for circuit discovery, making circuit discovery harder to use in modern MoE-based models.
Simultaneously, some recent work suggests that at least a few mechanistic interpretability techniques can transfer outside of transformers. Steering vectors, tuned lenses, knowledge editing, and circuit analysis evaluations have all been adapted, with varying success, to state-space models such as RWKV and Mamba, as well as to vision-language models (Paulo et al., 2025a; Sharma et al., 2024a; Häon et al., 2025; Neo et al., 2025; Ensign & Garriga-Alonso, 2024). Researchers have also conducted theoretical analyses aiming to clarify the computational relationships between transformers and other architectures. Some initial results suggest that several findings in transformers may reflect more general, transferable properties of neural networks [10] (Zimerman et al. 2025; Dao & Gu 2024; Peng et al. 2025). Sorting out which aspects of mechanistic interpretability depend on architecture and which reflect broader principles of neural computation thus appears to be a promising research direction. On the other hand, whether current research orientations and associated techniques remain relevant beyond transformers will also show how broadly mechanistic interpretability research can apply to future systems.
Conclusion
Neural networks could be one of the most consequential artefacts humanity has built, yet we do not understand how they work or reliably control what they do. Recent decades have seen an exponential improvement in AI capabilities, but methods to understand and explain internal mechanisms have lagged behind. This has created an interpretability gap between what models can do and what we know about how they do it.
Mechanistic interpretability offers a path towards understanding these systems by studying their internal representations and computations, with the aim of evaluating, predicting, and controlling behaviour. In this survey, we examine closely what interpretability can actually contribute to AI safety. We place mechanistic interpretability in the broader context of interpretability research and compare its mechanistic evidence with behavioural evaluation. Beyond surveying existing methods, we propose a unified framework for evaluating mechanistic interpretability claims and characterise six research orientations by their goals, assumptions, and trade-offs.
Rather than converging on a single approach, the field has crystallised into divergent research orientations, including Pragmatic Interpretability, Ambitious Interpretability, Developmental Interpretability, Interpretability Across Time, Introspective Interpretability, and Automated Interpretability, each making trade-offs between completeness, cost, and applicability.
Achievements and Limits
Mechanistic interpretability has achieved substantial progress across several fronts, including extracting interpretable features with SAEs, discovering task-relevant circuits, and enabling targeted interventions on model behaviour. Overall, the field has shaped how researchers view model computation, representations, and causal structure.
Yet current approaches are limited: they do not fully explain many behaviours, struggle to generalise under distribution shift, have difficulty cleanly disentangling entangled computations, and are hard to scale to frontier models. More importantly, their usefulness for AI safety remains debated.
Although many prior studies in the interpretability community assume that mechanistic understanding directly implies safer systems, the assumption is in fact largely untested. Therefore, this survey considers not only what a method reveals about a model’s internal computation, but also what it allows us to do in practice. We particularly discuss whether it supports safety-relevant tasks such as detecting deceptive behaviours or identifying dangerous capabilities, and whether those tasks can be achieved just as well without interpretability methods or insights.
A persistent challenge in mechanistic interpretability is the lack of standardised evaluation. Different interpretability methods use incompatible metrics. To address this, we present a unified evaluation framework covering five dimensions—attribution faithfulness, completeness, stability, causal validity, and human interpretability—where different methods can be jointly tested. We identify gaps and argue that the evaluation methodology and safety benchmarks are the field’s most pressing open problems.
Mechanistic interpretability is not a panacea for AI safety. Researchers, developers, and practitioners should recognise the limits of partial or incorrect causal understanding, rigorously evaluate MI tools under broad conditions, and combine them with complementary safety measures. MI should therefore be viewed as one component of a defence-in-depth safety strategy rather than a standalone guarantee of safety.
The Road Ahead
The six research orientations describe different research goals and priorities. At a broader level, mechanistic interpretability can be organised into three complementary paradigms that encompass these orientations and other research directions:
Post-hoc explanation: The traditional paradigm of studying trained models’ internals via tools like SAEs, circuit discovery, activation verbalisers, and causal analysis. Because deployed systems still rely on conventional dense architectures without intrinsic interpretability, post-hoc analysis will remain important in the medium term.
Studying learning dynamics: Understanding how mechanisms emerge during training via methods like singular learning theory and checkpoint analysis. This paradigm enables understanding, prediction, and intervention early in behaviour and capability emergence.
Interpretability by design: Engineering architectures that are mechanistically transparent by construction, such as SoLU, weight-sparse transformers, KANs, and their successors. This paradigm moves interpretability upstream and promises to reduce the need for expensive post-hoc analysis.
Each approach tackles a different area but complements the others. For instance, interpretability-by-design models can reduce reliance on post-hoc methods, while post-hoc mechanistic analysis can guide next-generation interpretability-by-design models towards desirable properties that current dense architectures lack. Insights into learning dynamics can suggest architectural constraints and training-time interventions.
Progress also depends on establishing stronger scientific standards. Recent work argues that interpretability should produce claims that are both falsifiable and reproducible, and improve its use in model evaluations and downstream applications.
Ultimately, the question remains: Does understanding how models work actually make them safer? This survey’s goal is to understand how these approaches can work together to improve model understanding and safeguards, while recognising the limits of partial or incorrect causal understanding.
Acknowledgments
FB and TW would like to thank Leo Gao, Peter Hase, John Hewitt, Kyle Lo, Aaron Mueller, Neel Nanda, and Avi Semler for their thoughtful comments and our discussions that helped shape the project. Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vojeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambе, Assaf Eismann, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Krevmer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Baeber, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germàn Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkhien, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Minee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Ammaninseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascal Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sridharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo
Notation Reference and Symbol Table
This section provides a unified reference for mathematical notation used throughout the paper. It is not exhaustive: symbols that appear only within a single section are defined where they are introduced and may not be listed here. Where a symbol is overloaded across subfields, the section-specific meaning is noted. All vectors are column vectors throughout the paper.
Dimensions and Indices
: residual-stream dimension; equivalently model hidden size
: attention head dimension
: MLP hidden dimension, typically
: number of attention heads per layer
: dictionary size of an SAE or transcoder, with integer expansion factor
: number of layers (model depth); indexes layers
: sequence length (context length); indexes token positions
: Big-O asymptotic complexity notation
Inputs and Activations
: an input prompt or a set of input prompts
: residual-stream activations at layer for token positions to , whose -th column is
: residual-stream activation at layer for token position
: residual-stream activation at layer with token position omitted
: residual-stream activation with both layer and position omitted
: mid-block residual stream (post-attention, pre-MLP) at layer
, : input and output of a model component, e.g., an MLP block, at layer
: reconstruction of activations by a dictionary model
: activation after an intervention (steering, ablation, patching, etc.)
Dictionary Learning
: SAE encoder
: SAE decoder
: sparse feature activations; is the activation of feature ; when a layer index is also present, we write for feature at layer
: feature direction in activation space (the -th column of )
, : encoder weights and bias
, : decoder weights and bias
: number of active features in a Top- SAE
Model Parameters
, : query, key, value, and output projection matrices of the -th attention head
, : MLP weight matrices
, : MLP bias vectors
: the collection of all trainable model parameters
Interventions and Objectives
: steering vector, applied as
: scaling factor for an intervention
: loss function
: sparsity or regularisation coefficient. In Appendix § 21.§ 2, instead denotes the local learning coefficient
Mathematical Operations
, : conditional probability and log probability
: expectation over the data distribution; denotes a conditional expectation
: Euclidean norm; and denote the and norms
: inner product
: element-wise product
: Kronecker product
: activation function
Colour Key and Style Guide
This section explains the visual conventions used throughout the survey for boxes, emphasis, and structural elements.
Box Types and Colours
Definition Box (Dark Blue)
Definition boxes introduce core concepts and provide precise technical definitions. Use these when a term needs clear specification before use.
Example concepts: “Feature”, “Circuit”, “Sparse Autoencoder”.
Style: Dark blue border, light blue background.
Example Box (Green)
Example boxes provide concrete illustrations, worked examples, or case studies. Use these to make abstract concepts concrete.
Example contents: “Induction in a Repeated Sequence”, “Example Application of Steering Vectors”, “Probing Classifiers for Sleeper Agents”.
Style: Green border, light green background.
Insight Box (Gold)
Insight boxes explain intuitions behind mathematical or conceptual results. Use these to build understanding of why something works, not just that it works.
Example contents: “Trade-offs Between Variants”, “The Complexity–Interpretability Trade-off”.
Style: Gold border, light gold background.
Warning Box (Red)
Warning boxes highlight critical limitations, failure modes, or common misconceptions. Use these to prevent readers from adopting incorrect or unsafe interpretations.
Example contents: “Caveats on CLT Fidelity Claims”, “The Probe Causality Problem”, “The Partial Coverage Issue”.
Style: Red border, light red background.
Practice Box (Teal)
Practice boxes illustrate suggestions that are actionable. These include protocol recommendations, checklists for verifying approaches, and step-by-step decision-making walkthroughs. Use these for practical implementations of procedures.
Example contents: “Guardrail Framework for Automated Interpretability”, “Validation Procedures for Introspective Models”, “Recommended Evaluation Protocol”.
Style: Teal border, light teal background.
Citation Evidence Levels
This survey cites both peer-reviewed publications and non-peer-reviewed research outputs. To help readers assess the maturity and review status of the evidence, citations are colour-coded:
Orange citations denote formally published work: peer-reviewed conference and journal papers, and books.
Green citations: Research outputs that have not undergone formal peer review, including preprints, technical reports, workshop papers, blog posts, and other emerging resources.
Green citations should not automatically be interpreted as lower quality or less reliable. Some originate from leading research organisations and may have undergone substantial internal review. However, they have not been subjected to the formal academic peer-review process and should therefore be interpreted with appropriate caution. Readers are encouraged to verify the current publication status of cited work where relevant.
Mathematical Foundations
This section provides background on some key mathematical frameworks used in mechanistic interpretability.
Causal Inference Basics
Mechanistic interpretability relies on causal reasoning: determining whether a component is necessary and sufficient for a behaviour.
Necessity
A component is necessary for behaviour if:
Tested via ablation: measure the effect of removing on downstream loss or output distribution.
Sufficiency
A component is sufficient for behaviour if:
Tested via interchange interventions: in the corrupted run with a reference input containing no target information, patch the activation of with that from a clean run using an input containing the target information, and then verify whether the behaviour emerges.
Causal Abstraction
Causal abstraction ((Geiger et al., 2021)) formalises when a high-level description (“this head implements copying”) correctly describes lower-level computation. A high-level abstraction is causally correct if interventions on the high-level description produce the same causal effects as interventions on the low-level system. This is formalised via causal diagrams: high-level variables correspond to low-level variables via abstraction maps , and interventions commute:
Local Learning Coefficient
Singular Learning Theory (SLT; Watanabe, 2009) provides a geometric analysis of learning and generalisation in singular statistical models, including neural networks. The Local Learning Coefficient (LLC; Lau et al., 2025) builds on SLT by localising Watanabe’s learning coefficient around a particular parameter configuration, thereby providing a measure of the effective local complexity of a neural network around a local minimum. Below, we introduce the LLC step by step, following the exposition in § 3 of Lau et al. (2025).
We first consider a neural network and a downstream task. For each neural network parameter configuration, say , we can evaluate its loss . The task should have an explicitly defined loss function , such that we can optimise the model parameters until they converge to a local minimum . Now we choose a sufficiently small closed ball centred at such that any element in it, i.e., , satisfies
Within this ball, we further consider the subset where each element has a loss that, while always greater than or equal to the local minimum, falls within a tolerance threshold . Formally, we define
We can then calculate the volume of . Viewing each element in this set, which corresponds to a parameter configuration, as a point in a high-dimensional parameter space whose dimension is the number of scalar parameters in , we have
Intuitively, measures how much parameter-space volume around achieves a loss within of the local minimum.
By SLT, as , has the asymptotic form
where is a positive constant, is a positive rational number termed the Local Learning Coefficient, and is a positive integer called the local multiplicity. Importantly, the equation shows that determines the dominant rate at which shrinks as , while contributes a lower-order logarithmic term. The larger is, the faster decreases as .
The convergence speed of , as characterised by , can therefore measure the model’s effective local complexity. Conceptually, a large means that, when we are just around the local minimum (i.e., for small ), the volume of parameter configurations that preserve a similarly low loss is small: perturbing the parameters is more likely to substantially change the loss. This trend corresponds to a local solution with more complexity but less degeneracy. Conversely, a small implies that a larger number of nearby parameter configurations reach similarly low loss, indicating lower effective local complexity and higher degeneracy.
The LLC has been used to characterise properties of trained neural networks. For example, Lau et al. (2025) showed that the LLC can capture differences in effective complexity between models trained under different settings, even when their training losses are similar.
Information Theory Basics
In mechanistic interpretability, concepts from information theory are often used to quantify relationships between internal components and how much information is preserved, and to measure representation quality. Although they do not directly establish causal mechanisms, they are highly useful for quantifying how information is encoded and changes during model computation.
Entropy. A random variable has entropy (or information content) that is defined as:
High entropy indicates greater uncertainty, while low entropy implies more predictable distributions. Entropy is frequently utilised to measure learned representation diversity or prediction confidence.
Mutual Information. The mutual information between two random variables and is
which measures how much information one variable provides about the other. Equivalently,
In interpretability, mutual information is often used to estimate how strongly a representation encodes a property of interest, such as a concept, label, or downstream behaviour.
KL Divergence. The Kullback–Leibler (KL) divergence between distributions and is
which measures how different one probability distribution is from another. KL divergence is widely used to compare model outputs before and after interventions, evaluate reconstruction quality, and train auxiliary interpretability models such as tuned lenses.
References
- [1]Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan, and Matjaz Gams. When rlhf fails: A mechanistic taxonomy of reward hacking, collapse, and evaluator gaming, 2026. URL https://arxiv.org/abs/2606.03238.
- [2]Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2019.
- [3]Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 9525–9536, Red Hook, NY, USA, 2018. Curran Associates Inc.arxiv.org/abs/1810.03292
- [4]Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BJh6Ztuxl.
- [5]Micah Adler, Dan Alistarh, and Nir Shavit. Towards combinatorial interpretability of neural computation, 2025. URL https://arxiv.org/abs/2504.08842.
- [6]Isha Agarwal, Saharsha Navani, and Fazl Barez. Context matters: Analyzing the generalizability of linear probing and steering across diverse scenarios. In Mechanistic Interpretability Workshop at NeurIPS 2025, 2025. URL https://openreview.net/forum?id=H5sbfvEbTh.
- [7]Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11):4311–4322, 2006.DOI
- [8]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/.DOI
- [9]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. The Fifth International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=ryF7rTqgl.arxiv.org/abs/1610.01644
- [11]Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html.
- [12]Cem Anil, Esin DURMUS, Nina Rimsky, Mriank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanham, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=cw5mgd71jW.DOI
- [13]Anthropic. Model diffing. Transformer Circuits, October 2024. URL https://transformer-circuits.pub/2024/model-diffing/index.html.
- [14]Anthropic. Claude sonnet 4.5 system card. System card, Anthropic, September 2025a. URL https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf.
- [15]Anthropic. System card: Claude opus 4 & claude sonnet 4. Technical report, Anthropic, May 2025b. URL https://www-cdn.anthropic.com/6d8a805502700718b0c49396f9616ba2a7c285/Claude%204%20System%20Card.pdf.
- [16]Tatsuya Aoyama and Ethan Wilcox. Language models grow less humanlike beyond phase transition. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 24938–24958, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1214. URL https://aclanthology.org/2025.acl-long.1214/.DOI
- [17]Apollo Research. Claude sonnet 3.7 (often) knows when it’s in alignment evaluations. Apollo Research Blog, March 2025. URL https://www.apolloresearch.ai/science/claude-sonnet-37-often-knows-when-its-in-alignment-evaluations/.
- [18]Santiago Aranguri and Tom McGrath. Discovering undesired rare behaviors via model diff amplification. Goodfire, 2025. https://www.goodfire.ai/research/model-diff-amplification.
- [19]Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=NUyt4uxzx0.arxiv.org/abs/2503.08679
- [20]Andy Arditi, Oscar Balcells Obeso, Aa quib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.arxiv.org/abs/2406.11717
- [21]Aryaman Arora, Dan Jurafsky, and Christopher Potts. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14638–14663, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.785. URL https://aclanthology.org/2024.acl-long.785/.DOI
- [22]Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, and Sarah Schwettmann. Language model circuits are sparse in the neuron basis. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=UrviwFWcN4.arxiv.org/abs/2601.22594
- [23]Kola Ayonrinde and Louis Jaburi. Evaluating explanations: An explanatory virtues framework for mechanistic interpretability – the strange science part i.ii, 2025. URL https://arxiv.org/abs/2505.01372.
- [24]Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics, 2023. URL https://arxiv.org/abs/2302.12433.
- [25]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450.
- [26]Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46, 07 2015. doi: 10.1371/journal.pone.0130140. URL https://doi.org/10.1371/journal.pone.0130140.
- [27]Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. If influence functions are the answer, then what is the question? In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.arxiv.org/abs/2209.05364














