Language Impairment as a Lens onto Large Language Models

Project Code

Image not extracted

Introduction

Large language models (LLMs) have achieved strikingly human-like language behavior, yet their internal functional organization remains fundamentally opaque. Unlike traditional linguistic theories that compartmentalize language into distinct symbolic modules [9], LLMs encode diverse linguistic structures into a unified, high-dimensional representation space [2, 4] (Tenney et al. 2019). This distributed architecture, which parallels neural population coding in biological brains (Nastase et al., 2026; Rumelhart et al., 1986), grants LLMs

significant global redundancy. However, recent mechanistic interpretability research reveals a paradox: despite this distributed redundancy, models remain vulnerable to catastrophic, highly specific functional collapse when focal critical circuits are disrupted (He et al., 2026; Qin et al., 2026; Lu et al., 2026). This creates a theoretical tension: how can an LLM be so globally distributed, yet remain vulnerable to focal damage that produces specific linguistic breakdowns?

To investigate this tension, we ask whether the internal architecture of LLMs possesses a functional structure analogous to the human language system [5] (Teehan et al. 2022; Fedorenko et al. 2024; Kuribayashi et al. 2025). In clinical neuropsychology, the brain’s functional organization is historically mapped by observing how language breaks down following focal injury (e.g., through the study of aphasia, an acquired language disorder due to brain injury or disease) [16] (Geschwind 1970). Standard NLP metrics, such as perplexity or downstream accuracy, only indicate that a model is degrading (Qin et al., 2026); they cannot easily characterize the nature of the deficit. Aphasia, however, provides a standardized, granular taxonomy for linking focal damage to highly specific language deficits (Hoffmann and Chen, 2013). By adopting this clinical diagnostic lens, we gain a practical framework to categorize (i) how an LLM’s language generation breaks down when perturbed, and (ii) how an intact LLM’s activation patterns change when processing clinically-informed impaired inputs.

Our framework thus consists of two complementary methodologies: lesion and steering.

• In the lesion approach, we developed a classifier trained on clinical diagnostic data that classifies model behavior according to aphasia subtypes (e.g., Broca, Wernicke; see Appendix § 7). By systematically ablating model components, we map specific structural interventions to their corresponding aphasia-like output categories. This allows us to test whether certain linguistic functions can be linked to specific components within LLMs.

• In the steering approach, we derive intervention vectors by calculating the difference in the model’s residual stream activations when processing human aphasic transcripts versus healthy controls. By systematically adding these steering vectors to targeted layers during text generation, we shift the model’s behavior toward specific aphasia profiles (e.g., Broca’s or Wernicke’s). This allows us to test whether complex language deficits can be induced through continuous activation interventions, and to explore whether subtracting these directions can effectively reverse or “heal” the linguistic effects of a simulated lesion.

We employ this dual approach to robustly cross-validate our findings. While ablating targets the model’s structural “hardware” (discrete circuits), steering intervenes on its representational “software” (continuous state). Proving that aphasic behavior can be induced both structurally and representationally ensures our results reflect fundamental properties of the model, rather than artifacts of a single methodology.

Applying this framework, our main findings are:

• Dual existence of functional mechanisms: Classifier-guided lesion successfully isolates discrete circuits governing distinct language functions (e.g., grammatical fluency, semantic coherence), while steering identifies corresponding continuous vectors. Crucially, targeted subtype lesions succeed far more often when guided by our classifier.

• Dense functional entanglement: Distinct aphasic profiles are highly entangled. Subtypes share significant structural overlap in their lesioned circuits and exhibit high cosine similarity across their steering vectors.

• Asymmetric generalizability: While structural lesion effects are constrained by the classifier’s specific training distribution, representational steering generalizes robustly across diverse downstream tasks.

Our study has the following contributions. We (1) introduce a novel interdisciplinary framework that adapts clinical aphasia to map LLM failure modes, (2) provide empirical evidence that macroscopic linguistic breakdowns correspond to both specialized circuits and continuous vectors, and (3) demonstrate that LLM language generation, in a way functionally analogous to the human brain, relies on a densely entangled, distributed architecture vulnerable to highly specific disruptions.

Related Works

Aphasia and LLMs. Recent research has begun utilizing clinical language deficits to map model degradations (Wang et al., 2025a,b; Yang et al., 2026; Roll et al., 2026a), though interpretations of the parallels between LLMs and humans remain contested. Yang et al. (2026) demonstrated that systematically lesioning the decoder layers of multimodal models can reproduce the complete, joint error profiles of individual persons with aphasia (PWAs) during a picture-naming task. They observed a distinct depth gradient: perturbing early layers caused complete response failure, middle layers triggered semantic errors, and late layers yielded neologisms. Conversely, Roll et al. (2026a) evaluated damaged language models using a Text Aphasia Battery (TAB; Roll et al. (2026b)) and concluded that the resulting error distributions do not genuinely resemble human clinical aphasia, arguing that LLMs and the human brain are fundamentally organized in different ways.

Our research bridges these perspectives. While Yang et al. (2026) focused on single-word vision-language tasks to replicate individual patient profiles, we focus on continuous text generation to localize specific internal functional networks. Furthermore, our theoretical stance aligns with the cautions raised by Roll et al. (2026a): we deploy aphasic subtypes strictly as a diagnostic framework to categorize model vulnerability and internal overlapping components, without assuming a 1:1 modularity with biological brain regions like Broca’s or Wernicke’s area ([8] (Wernicke 1874)). Ultimately, by combining lesioning to test focal vulnerabilities and steering to map the intact distributed geometry, we investigate whether a highly distributed LLM can genuinely exhibit focal, aphasia-like functional structures.

Lesions and causal analysis. A high degree of redundancy in LLMs protects them from drastic performance degradation (Frantar and Alistarh, 2023; Sun et al., 2024; Wang et al., 2023b; Shi et al., 2023). Most LLMs’ attention heads can be removed with little loss (Michel et al., 2019; Voita et al., 2019), and many attention layers can be dropped from large models (He et al., 2026), yet a handful of critical neurons can collapse a model into repetitive noise (Qin et al., 2026; Lu et al., 2026). Circuit analyses localize a behavior to a set of components with lesioning and activation patching (Vig et al., 2020; Wang et al., 2023a; Zhang and Nanda, 2024), and learned masks find such sets by gradient descent ([12, 14]). These methods track a loss of accuracy or a target token’s probability, whereas we search for lesions that produce a specified clinical behavior in free narration.

Steering. Unlike lesioning, steering adds a direction to a model’s activations while it generates, which shifts its behavior without changing its weights (Subramani et al., 2022; Turner et al., 2023; Zhang et al., 2026a). Linear probes show that many properties of the input can be decoded from hidden states ([1, 6] (Hewitt and Manning 2019)), and concepts such as the truth of a statement are often encoded along a single direction of a model’s representations (Park et al., 2024; Marks and Tegmark, 2024). The direction is often the difference between the mean activations on two contrasting sets of inputs (Zou et al., 2023; Rimsky et al., 2024; Li et al., 2023), and removing such a direction from every layer can stop a behavior ([3]). These studies construct the contrasting inputs themselves, for example harmful and harmless instructions, and so does the closest work to ours, which steers models to simulate cognitively impaired patients for clinical training (Zhang et al., 2026b). We take the contrasting inputs from clinical data, the transcripts of speakers with and without aphasia, which the model reads as its own answers, and ask experts whether the steered speech matches the targeted type of aphasia.

Data and Setup

Data. AphasiaBank (MacWhinney et al., 2011) is a large-scale database of naturalistic speech from individuals with aphasia and neurologically healthy controls. We focus on two tasks that elicit different types of speech: Cinderella task, in which speakers retell the story of Cinderella to produce narrative speech, and sandwich task, in which speakers describe how to make a peanut butter and jelly sandwich, eliciting procedural speech with a fixed sequence of steps (see Table 1 for statistics).

Group	Cinderella Transcripts	Cinderella Speakers	Sandwich Transcripts	Sandwich Speakers
Control	395	288	472	360
Anomic	272	148	187	131
Conduction	141	73	81	62
Broca	273	113	111	77
Wernicke	58	30	29	23
All aphasia	744	353	408	281
All	1,139	637	880	638

Models. We study 11 models from four families: GPT-2 XL (Radford et al., 2019); Llama-2-7B-chat and Llama-2-13B-chat (Touvron et al., 2023); Pythia-1B, 1.4B, 2.8B and 6.9B ([7]); and Qwen3.5-0.8B, 2B, 4B and 9B (Qwen Team, 2026), with thinking disabled. We use the AphasiaBank prompts, “Tell me as much of the story of Cinderella as you can” and “Tell me how you would make a peanut butter and jelly sandwich”, to enable comparison with human speech elicited by the same tasks. We pass these prompts directly to GPT-2 XL and Pythia and format them using the respective chat templates for Llama-2 and Qwen3.5 (Table 6). All models generate at most 180 new tokens using greedy decoding with a repetition penalty of 1.25, as unpenalized greedy decoding causes the base models to loop even without intervention. Across both interventions and tasks, we include only responses with at least 20 characters.

Family	Prompt template
GPT-2 XL and Pythia	Question: request\n<br>Answer:
Llama-2-chat	[INST] request [/INST]
Qwen3.5	`<	im_start	>user\n<br>request<	im_end	>\n<br><	im_start	>assistant\n<br><think>\n\n</think>\n\n`
Request	Text
Yes-or-no question	Do you know the story of Cinderella?
Cinderella Story request	Tell me as much of the story of Cinderella as you can.
Sandwich request	Tell me how you would make a peanut butter and jelly sandwich.

Method

We use two interventions to a frozen model: classifier-guided lesioning, which disables attention heads, and subtype steering, which adds a direction to an intact model’s activations (Figure 3).

Image not extracted

Model Lesioning

Prior methods intended to induce aphasic behaviour instead produce model failures. For example, Qin et al. (2026) remove a few critical neurons, causing model outputs to collapse into repetitive noise, while Wang et al. (2025a) guide lesioning by matching model outputs to patient speech using text similarity. Human experts identify the resulting outputs as model scrambles. We therefore train a classifier to distinguish aphasia from model scrambles and use it to guide the lesion search.

Five-way classifier. The classifier assigns a text a probability for each of five classes: control speech, conduction or anomic aphasia, Broca’s aphasia, Wernicke’s aphasia, and model scrambles. Conduction and anomic aphasia form one class because the two are hard to tell apart from free speech alone: the Western Aphasia Battery separates them by a repetition test ((Kertesz, 2007)). We introduce a “model scrambles” category to capture non-human machine artifacts, such as tokenizer loops, stray brackets, or formulaic repetitions. Isolating these distinct computational failures prevents machine-generated noise from contaminating the mapping between human clinical profiles and internal model components. The training data hold 259 model-scramble outputs of Llama-2-7B-chat and GPT-2 XL from a preliminary lesion study, selected using LLM-as-a-judge with human annotator validation. The classifier combines a BGE-large sentence encoder ((Xiao et al., 2024)), fine-tuned with low-rank adapters ((Hu et al., 2022)), with 14 text features (Figure 2), and it averages 20 such models trained on different random splits.

Image not extracted

On held-out transcripts whose text appears only once in the data, it reaches an accuracy of 83.3% and a macro-averaged F1 of 0.78. Lesioning and steering use three targets read off their probabilities: $P(\text{aphasia})$, the total probability of the three aphasia classes, which separates aphasic speech from both control speech and model scrambles; $P(\text{Broca})$; and $P(\text{Wernicke})$.

CLAMP. Since attention layers contain multiple parallel heads, we hypothesize that selectively lesioning heads can induce aphasia-like behaviour without producing model scrambles. We propose Classifier-guided Lesion Accumulation with Minimal Pruning (CLAMP), which ablates attention heads by zeroing their outputs before the attention output projection at every prompt and generated-token position. For a fixed model, task, and target class, let $s(H)$ denote the classifier’s probability of that class for the response generated with head set $H$ ablated. CLAMP proceeds in three steps (Figure 3, left). First, it ablates each head individually and ranks heads in descending order of $s(\{h\})$. Second, it builds $H$ by adding heads in this order until their joint ablation yields $s(H)>0.5$. Finally, it prunes heads individually, or in batches for large sets, as long as $s(H)$ remains above 0.5. In the final set, restoring any single head reduces $s(H)$ to 0.5 or below. For the Cinderella task, we first ask “Do you know the story of Cinderella?”. If the intact model answers affirmatively, we include the lesioned model’s narrative only if it also answers “yes” to this question.

Baselines. We compare CLAMP with eight lesion methods (Appendix § 10). PARAM is the emergent-modularity method of Wang et al. (2025a): it zeroes the weights with the top 1% of $|w \partial L/\partial w|$ in each weight matrix, where $L$ is the loss on control narratives, and groups the lesions by the patient group whose transcripts the lesioned model’s answer resembles most by BLEU-1 and BERTScore. PARAM-mask and Head-mask learn, with the model frozen, how much to knock out each of PARAM’s lesions of the MLP weight matrices or each head, as Csordás et al. [12] learn weight masks, so that the transcripts of the target group become more likely and those of the other groups less likely. PARAM-MASK-HARD and HEAD-MASK-HARD knock out the components whose learned amount exceeds 0.5 and prune them as above. These four methods learn from the transcripts; the classifier only chooses the reported training step and, for the hard masks, guides the pruning. HEAD-CEM optimizes the classifier directly with the cross-entropy method (Rubinstein, 1999), sampling 64 head sets per round for at most 20 rounds. PARAM-C grows PARAM’s MLP and attention head lesions by the classifier as CLAMP. HEAD-GROWN-COS grows and prunes heads as CLAMP does, but with a corpus similarity in place of the classifier: the answer’s mean cosine similarity to the target group’s transcripts minus that to the control transcripts (Reimers and Gurevych, 2019).

Beyond the narrative. To evaluate other aphasia-related abilities, we test every lesion on the other task, on the Text Aphasia Battery (TAB)’s short-answer items for comprehension and repetition and its five connected-text prompts (Roll et al., 2026b), and on two MMLU subjects, global facts (tests broad general world knowledge) and high-school world history (covers major global historical eras, movements, empires and revolutions taught in secondary education) (Hendrycks et al., 2021), by the drop in the log-probability of the correct answer. A DeepSeek-V4.1-Flash [15] LLM judge rates each connected-text answer for the TAB’s 19 aphasia features, with the TAB’s instructions with in-context examples of the features (Appendix § 17).

Steering

Steering builds its directions from the human transcripts (Figure 3 right). For each task, the model reads every transcript $x$ appended to the task prompt as if it were its own answer, and $h_\ell(x)$ is the residual stream after layer $\ell$, averaged over the tokens of $x$. The steering vector for a target at layer $\ell$ is the difference between two group means,

where $C$ holds the control transcripts and $T$ the transcripts of Broca’s aphasia for the Broca target, and of Wernicke’s aphasia for the Wernicke target. Steering adds $\alpha v_\ell$ to the residual stream after layer $\ell$ at every position while the model reads the prompt and generates. We report its strength as $r=\lVert\alpha v_\ell\rVert/\lVert\bar{h}_\ell\rVert$, where $\bar{h}_\ell$ is the mean norm of $h_\ell(x)$ over the transcripts. For each model, target, and source task, a sweep over 8 layers spread across the model’s depth and 8 strengths from $r=0.1$ to 4 picks the smallest strength, and the layer, at which the steered answer to the source task reaches the target. If no setting reaches it, we keep the setting with the highest probability.

The steered model answers seven prompts with this setting: the source task, which computed it, and six prompts that did not, the other task and the five connected-text prompts of the TAB. Two experts label every answer from models with one of the classifier’s five classes when steered toward a type. We collect the annotation to check if the steering is successful (Appendix § 15).

Steering back. To test whether steering can undo a lesion, we remove the target’s direction from the residual stream of the lesioned model after every block, replacing each state $h$ with $h - (h^\top\hat{v}_{\ell}) \hat{v}_{\ell}$, where $\hat{v}_{\ell} = v_{\ell}/\lVert v_{\ell}\rVert$ [3]. We also add the direction with a negative strength at a single layer. As controls, we remove random directions in the same way, and we add random directions of the same norm.

Results

We focus on reporting results about the Broca’s and Wernicke’s subtypes, as their contrasting failure modes (broken grammar vs. semantic incoherence) provide unambiguous mechanistic signals. Conduction and Anomic profiles are omitted, as their subtle free-production signatures heavily overlap with healthy controls (Figure 6).

Image not extracted

We find language functions in LLMs can be successfully isolated to discrete structural circuits and continuous representational vectors.

Model Lesioning

Classifier-guided search outperforms corpus-based search. Across 42 settings spanning seven models (GPT-2 XL, two Llama-2-chat models, and four Pythia models), two tasks, and three targets, classifier-guided methods find aphasia-inducing lesions far more often than methods guided by corpus statistics (Table 2). Human experts agree with the classifier’s label on 90% of the lesioned models’ answers to both Cinderella and sandwich tasks.

Method	Aph.	Broca	Wern.	All
Guided by corpus statistics
PARAM	2	0	0	2
PARAM-MASK	4	0	0	4
PARAM-MASK-HARD	3	1	0	4
HEAD-MASK	5	0	0	5
HEAD-MASK-HARD	8	1	1	10
HEAD-GROWN-COS	2	0	0	2
Guided by the classifier
PARAM-C	11	10	4	25
HEAD-CEM	9	8	6	23
CLAMP	14	11	13	38

Head lesioning is more effective than parameter lesioning. CLAMP induces the target aphasic behaviour in 38 of 42 settings; the remaining four

yield model scrambles. PARAM-C, which lesions parameters rather than heads, succeeds in only 25 settings, supporting our hypothesis that selectively lesioning heads reduces model degeneration. Meanwhile, HEAD-CEM, which optimizes the classifier objective more aggressively, succeeds in only 23 settings, and 12 of the remaining 19 yield model scrambles. CLAMP instead uses classifier scores for head ranking and threshold checks during pruning, which may help keep responses closer to the classifier’s training distribution.

CLAMP generalizes to Qwen models with linear attention. Across all 11 models, CLAMP finds successful lesions in 54 of 66 settings, with the remaining settings yielding model scrambles (Figure 4). We find aphasia-inducing lesions for every model on both tasks, while Broca-like and Wernicke-like behaviours are induced in 15 and 17 of 22 settings, respectively. For example, ablating 101 heads in Llama-2-13B-chat transforms its fluent Cinderella narrative into “to tell a tale of Ct... to tell me,”, with a classifier score of $P(\mathrm{Broca}) = 0.89$, while the model still confirms that it knows the story.

Image not extracted

The heads of a lesion act together while the model reads the prompt. We use activation patching to identify when lesioned heads affect model behavior. Restoring the clean activations of all heads of a lesion while the model reads the prompt, so that the heads stay knocked out only while it writes its answer, undoes 53 of the 54 lesions. Restoring a single head undoes the lesion for a median of 83% of a lesion’s heads, yet the classifier labels 80% of these answers as model scrambles and only 13% as control speech. The human aphasia-like answer thus needs nearly all heads of the lesion acting on the prompt together, a circuit in which changing one head mostly breaks the model (see Appendix § 13).

Lesions for different aphasia types share heads. The Broca and Wernicke lesions of the same model and task share a median of 54% of the heads of the smaller lesion. Across the 13 pairs, 517 heads are shared, three times the number expected by chance, and overlap is significant in 11 pairs ($p < 0.01$). This overlap begins at single heads: a head whose knockout makes the answer more Broca-like usually makes it more Wernicke-like too, at a median rank correlation of 0.93 across heads. The circuits for Broca-like and Wernicke-like breakdowns thus share far more heads than chance, entangling the two types rather than separating them.

The lesions damage language on new prompts, but not as the targeted type. Almost every lesion found on one narrative also damages the model’s answers to the TAB (Table 3). The LLM judge finds more of the TAB’s aphasia features in the connected text for 50 of the 54 lesions than for the intact models, a median of 2.7 of 19 features per answer against 0.0. The lesioned models get a median of 4 of the 15 short-answer items right, against 8.5 for the same models intact. The targeted type, however, does not carry over: the Broca and Wernicke lesions reach their type in only 1 of the 184 scored answers that they give to the other narrative and to the TAB’s connected-text prompts

Family	Features	Short Answer	MMLU
GPT-2	0.8 → 2.6	8/15	0.24
Llama-2-chat	0.0 → 1.8	3/15	1.59
Pythia	0.0 → 2.0	4/15	0.11
Qwen3.5	0.2 → 4.8	4.5/15	1.11
All	0.0 → 2.7	4/15	0.89

(Appendix § 12). The lesions thus damage language broadly, but the type that they induce stays in the answer on which the search found them.

Steering

One direction per type makes an intact model “speak” like a patient. Adding the Broca direction at a single layer makes the classification label Broca in all 11 models on both tasks, and the Wernicke direction does so for Wernicke’s aphasia in 12 of the 22 model-task pairs (Appendix § 14). Two experts labeled the steered answers of seven models (Figure 5): either expert identifies successful steering to the targeted type for 58% of the Broca answers, and 58% of the Wernicke answers. Thus, adding a single direction to the residual stream can make an intact model produce speech resembling a targeted aphasia subtype. Steering toward Broca’s aphasia, however, often scrambles the model, possibly because the corpus contains no examples of scrambled model outputs.

Image not extracted

Steering carries the targeted type to new prompts. On the six prompts not used to choose the steering setting, either expert labels the targeted type for 65% of the Broca answers and 63% of the Wernicke answers that are not model scrambles (Appendix § 15). The type follows the direction: either expert labels 44 of the 76 Broca-steered answers to these prompts Broca-like but only 13 of the 77 Wernicke-steered ones, and 77 of the 77 Wernicke-steered answers Wernicke-like but only 9 of the 76 Broca-steered ones. The classifier finds the targeted type in 67 of the 153 steered answers to new prompts, but in only 1 of the 139 scored answers of the Broca and Wernicke lesions of the same models. A direction thus keeps its type on new prompts, whereas a lesion leaves only general damage.

The Broca and Wernicke directions share most of their length. Both directions point nearly along each other, at a median cosine over layers of 0.79 to 0.90 in every model and task, and along the aphasia direction of all aphasic transcripts, at a median cosine of 0.97 for Broca and 0.92 for Wernicke (Appendix § 14). The two types differ little: after projecting out the general aphasia direction extracted from the whole aphasia group, the remaining parts point away from each other, with median cosines of −0.65 to −0.41. The Wernicke part carries its type: when added alone, it pushes $P(\mathrm{Wernicke})$ above 0.5 in all 22 model-task pairs. Like the lesions, the directions thus share most of what makes speech aphasic, with a small part that sets each type apart.

The classifier agrees with the experts more often than an LLM judge. Across all steered answers, including those to the out-of-distribution TAB prompts, the classifier still matches either expert’s label on 65% of answers, compared with 55% for a DeepSeek-V4.1-Flash judge. (Table 4) The two experts agree with each other on 80% of answers, with a pooled Cohen’s $\kappa$ of 0.76. Much of the classifier’s advantage comes from the 77 answers the experts identified as model scrambles: its labels agree with either expert on 51 of these answers, compared with 23 for the judge.

	Two-way	Five-way	All
Classifier	73	62	65
LLM judge	64	51	55
Two experts	88	76	80
Cohen’s $\kappa$	0.75	0.68	0.76

Steering back undoes the lesions, but does not “heal” them. Removing a lesion’s own direction from the residual stream after every block undoes 53 of the 54 lesions, but 52 of these answers are model scrambles (Appendix § 16). Random directions undo the lesions almost as often: removing one of three random directions undoes 50 or 51 of the lesions, and 91% of the pushes with a random direction of the same norm at one layer undo the lesion, against 95% of those with the lesion’s own direction. Undoing a lesion is thus easy but unspecific: nearly every perturbation we tried moves the answer off its target, mostly into model scrambles rather than control speech.

Conclusion and Discussion

In this work, we successfully bridged neuropsychology and mechanistic interpretability to map the failure modes of large language models. Specifically, we make the following core contributions:

Graded model degradation. First, this work shifts the conceptual focus of AI safety and interpretability from catastrophic collapse to graded, structured degradation. By applying an aphasia-inspired diagnostic lens, we demonstrate that LLM breakdowns are not merely random noise, but structured deficits that align with recognized clinical profiles. This provides the field with a more granular vocabulary to evaluate “bad” or impaired language generation, opening the door to modeling other language deficits in artificial systems.

A complementary methodological framework for mechanistic interpretability. Second, we contribute a robust, dual-approach framework for mechanistic interpretability. Relying on a single intervention method can yield artifacts tied to that specific technique. By cross-validating our findings through complementary approaches, combining the structural isolation of classifier-guided lesioning with the representational fluidity of continuous activation steering, we establish a more rigorous standard for proving that an identified language function is a fundamental property of the model. This dual inside/outside-in approach ensures that mapped vulnerabilities are highly reliable.

A lightweight classifier with real-world applicability. Third, while developed primarily as part of the lesion approach, our classifier represents a standalone contribution to automated clinical annotation. Recent research has demonstrated the promise of combining LLM-based measures (such as surprisal) with traditional clinical features for aphasia classification [11]. Our model advances this paradigm by offering a highly efficient, lightweight alternative with higher overall and per-class accuracy.

Limitations

First, our lesions and the steering directions come from two English narrative tasks in AphasiaBank: the Cinderella story and the sandwich procedure. However, AphasiaBank holds further tasks, such as picture descriptions (e.g., participants are asked to look at the picture and tell a beginning, middle, and end of a story) and personal narratives (e.g., stroke, illness, and recovery story), and the same pipeline can be applicable to them with minimal changes (although in the case of the picture description task, Vision-Language Models should be involved), as it would to aphasia corpora in other languages.

Second, the classifier learns machine failures from only 259 expert-labeled examples. Although it detects only about half of the steered responses that experts classify as model-scrambled, incorporating additional expert-labeled failures into its training data could improve its performance. In both current and future studies, labels produced by the LLM judge and classifier should be interpreted as assistive signals rather than clinical diagnoses.

Third, our experiments focus on a limited set of language models, leaving the generalization of our findings to other model families and architectures uncertain. Evaluating additional models would help establish the robustness of our findings across model scales and generations. Furthermore, extending our interventions to mixture-of-experts (MoE) architectures is another important direction for future work, as sparse expert routing may influence the mechanisms and effectiveness of model lesioning and steering.

Aphasia Types

Aphasia is a language disorder acquired through brain damage, most often from a stroke. Clinicians classify it by three properties of a speaker’s language: whether speech is fluent, whether the speaker understands what others say, and whether the speaker can repeat what others say (Kertesz, 2007). AphasiaBank assigns each speaker a type based on the Aphasia Quotient subtests from the Western Aphasia Battery-Revised and provides a severity score from 0 to 100, where lower values indicate more severe aphasia (MacWhinney et al., 2011; Kertesz, 2007).

Broca’s aphasia. Speech is nonfluent and speakers produce short, effortful phrases that often drop function words and inflections, which is called agrammatism, while their comprehension is relatively preserved.

Wernicke’s aphasia. Speech is fluent and may be abundant, but it carries little content and contains substituted words and sounds, called paraphasias, and speakers understand others poorly.

Conduction aphasia. Speech is fluent and comprehension is relatively good, but speakers repeat poorly and produce sound errors that they often try to correct.

Anomic aphasia. Speech is fluent and comprehension and repetition are good, but speakers have trouble finding words and talk around them. Anomic aphasia is usually the mildest type.

The classifier merges conduction and anomic aphasia into one class, because the two differ mainly in repetition, which free speech such as a story or a recipe does not test.

Data, Models and Setups

This appendix gives the details on the human transcripts, the models, the prompts, the decoding rules, the rules that decide whether an answer counts, the two tests beyond the narratives, the Text Aphasia Battery (TAB) and MMLU, and the compute and software that we used.

Transcripts. We use 1,147 AphasiaBank transcripts (MacWhinney et al., 2011) of the Cinderella task and 1,437 of the sandwich task, and some of these transcripts share a text. The sandwich transcripts hold 557 texts that appear twice and we keep only one copy of each. Four Cinderella texts appear once as anomic and once as control speech, so their group is unknown, and we drop both copies. Table 1 counts the remaining transcripts, each text once: 1,139 for the Cinderella task and 880 for the sandwich task. The transcripts carry no speaker field, so we identify a speaker by the corpus and by the file name without its session suffix. The probes and the steering vectors use these transcripts, whereas the classifier draws its training and test splits from all 2,584 transcripts, duplicates included (Appendix § 9). Of the aphasic transcripts, 720 of 744 in the Cinderella task and 382 of 408 in the sandwich task come with the speaker’s aphasia quotient, the severity score that Appendix § 14 compares with the projection of a transcript on the aphasia direction.

License. We used AphasiaBank data under TalkBank’s Ground Rules and the applicable CC BY-NC-SA 3.0 license. We report aggregate results but do not redistribute participant transcripts, recordings, or other restricted data. Researchers seeking the underlying data should request access directly from TalkBank.

Conduction and anomic transcripts lie among the control transcripts. We embed every transcript with BGE-large (Xiao et al., 2024), the encoder that the classifier fine-tunes, here without its fine-tuning, so that the space knows nothing of our labels. A transcript becomes the encoder’s [CLS] vector over its first 384 tokens, the classifier’s limit, scaled to length 1. Figure 6 maps these vectors with t-SNE, one map per task, together with the classifier’s model scrambles (Appendix § 9). We measure the overlap in the full embedding rather than on the map: for each transcript, we take the share of control transcripts among its 10 nearest transcripts by cosine similarity. On the Cinderella and the sandwich task, this share is 0.89 and 0.91 for control transcripts, 0.44 and 0.58 for anomic transcripts, and 0.25 and 0.41 for conduction transcripts, against 0.07 and 0.11 for Broca’s and 0.07 and 0.33 for Wernicke’s aphasia. The model scrambles form islands of their own: 92% and 85% of a scramble’s 10 nearest texts are other scrambles, whereas scrambles make up 0.1% of the nearest texts of the human transcripts. Conduction and anomic transcripts thus resemble control speech more than Broca’s and Wernicke’s transcripts do, which is why we mainly report the Broca and Wernicke targets.

Model	Hugging Face identifier	Layers	Heads per layer	Heads in all	Softmax heads
GPT-2 (Radford et al., 2019): base model, question-and-answer prompts
GPT-2 XL	gpt2-xl	48	25	1,200	1,200
Llama-2 (Touvron et al., 2023): chat models, Llama-2 chat template
Llama-2-7B-chat	meta-llama/Llama-2-7b-chat-hf	32	32	1,024	1,024
Llama-2-13B-chat	meta-llama/Llama-2-13b-chat-hf	40	40	1,600	1,600
Pythia (Biderman et al., [7]): base models, question-and-answer prompts
Pythia-1B	EleutherAI/pythia-1b	16	8	128	128
Pythia-1.4B	EleutherAI/pythia-1.4b	24	16	384	384
Pythia-2.8B	EleutherAI/pythia-2.8b	32	32	1,024	1,024
Pythia-6.9B	EleutherAI/pythia-6.9b	32	32	1,024	1,024
Qwen3.5 (Qwen Team, 2026): post-trained models, Qwen3.5 chat template with thinking turned off
Qwen3.5-0.8B	Qwen/Qwen3.5-0.8B	24	8 / 16	336	48
Qwen3.5-2B	Qwen/Qwen3.5-2B	24	8 / 16	336	48
Qwen3.5-4B	Qwen/Qwen3.5-4B	32	16 / 32	896	128
Qwen3.5-9B	Qwen/Qwen3.5-9B	32	16 / 32	896	128

in which each head keeps its own recurrent state. A lesion can knock out heads of both kinds, and softmax heads make up 14.3% of the heads of every Qwen3.5 model. Knocking out a head zeroes its slice of the input to the attention output projection of its layer, at every position of the prompt and of the answer; in the softmax layers of Qwen3.5, this slice holds the head’s output after its gate.

Models. Table 5 lists the 11 models and their Hugging Face identifiers. We load every model with the Transformers library (Wolf et al., 2020) and run it in 32-bit floating point. The Qwen3.5 models are the post-trained checkpoints, not the base checkpoints. In Qwen3.5, every fourth layer uses full softmax attention, and the other layers use Gated DeltaNet linear attention (Yang et al., 2025), in which each head keeps its own recurrent state.

Prompts. Table 6 gives the prompt template of each family and the three requests, verbatim. The base models, GPT-2 XL and Pythia, receive each request in a plain question-and-answer format, and the chat models receive it as a user message in their chat template, with no system prompt and, for Qwen3.5, with thinking turned off. The prompts of GPT-2 XL and of the Llama-2 models start with the beginning-of-sequence token, the prompts of Pythia start without it, and the Qwen3.5 tokenizer has no such token. On the Cinderella task, the yes-or-no question and the story request are two separate prompts rather than two turns of one dialogue, and a lesion stays in place for both answers.

Decoding. For the three prompts, every model decodes greedily, with the repetition penalty set to 1.25 and applied to the raw logits. At each step, the penalty divides the logit of every token that already occurs in the prompt or in the answer by 1.25 if the logit is positive, and multiplies it by 1.25 if the logit is negative. Without the penalty, greedy decoding makes the intact base models loop on the Cinderella task: the classifier gives the answers of GPT-2 XL, Pythia-1B, Pythia-1.4B, and Pythia-2.8B a probability of up to 99% of being model scrambles. For the Llama-2 models, greedy decoding replaces the sampling at temperature 0.6 and top-p 0.9 that their released generation settings specify. A model writes at most 180 new tokens for the story and the sandwich answer, and at most 12 for the answer to the yes-or-no question, and it stops early at its end-of-sequence token or, for Qwen3.5, also at <|endoftext|>. In the lesion searches, each model generates in batches of a fixed size, and we pad a short batch with copies of its first lesion, so that an answer does not depend on how the lesions were batched. Greedy decoding thus gives each lesion a single answer, and regenerating every CLAMP and HEAD-GROWN-COS lesion from scratch reproduces its stored answer word for word.

What counts as an answer. A base model often goes on to write further questions and answers of its own, so we cut its answer before the first new line that starts with Question:, Q:, or question:, and the answers of the chat models stay whole. An answer counts only if it has at least 20 characters, not counting invisible Unicode format characters, and a lesion whose answer does not count fails. The rule leaves out format characters because some lesions of Llama-2-13B-chat make the model write a few letters followed by a long run of zero-width joiners, which are invisible: with its 170 joiners counted, the answer “tOoOO” would pass, and the classifier scores it at $P(\mathrm{Broca}) = 0.70$. In the lesion experiments, the model first answers the yes-or-no question of the Cinderella task, and a parser matches the start of this answer, ignoring case and up to a word boundary, against two lists of prefixes. The answer means yes when it starts with yes, yeah, yep, yup, sure, of course, certainly, absolutely, definitely, i do remember, i remember, or i do, and no when it starts with no, nope, nah, sorry, i don't, i dont, i do not, i cannot, i can't, i cant, not really, or unfortunately.

When both lists match, the longer match wins, and yes wins a tie. A lesion fails when the model answers no, and an answer that matches neither list passes. The check is off for Pythia-6.9B, whose intact model already answers “No. / Q: You don’t remember it, do”, where the slash marks a line break. The lesion searches apply the check to every other model, and the steering experiments do not apply it. The classifier scores the story or the sandwich answer after the cut, and never the answer to the yes-or-no question.

Text Aphasia Battery. We transcribe 20 items of the TAB from the appendix of Roll et al. (2026b), five for each of four subtests, and Table 7 shows all five connected-text prompts and one item of each other subtest. The connected-text items carry the TAB’s own system instruction. For the other three subtests, the TAB specifies only a conversational system prompt, so we use the minimal instruction given in Table 7. A base model receives Question: <instruction> <prompt>\nAnswer:, and we cut its answer as for the narratives, whereas a chat model receives the instruction as a system message and the prompt as a user message in its chat template. The model answers a connected-text item as it answers the narratives, with the repetition penalty, in at most 120 new tokens. It answers a short-answer item greedily without the penalty, in at most 40 new tokens, because the penalty would push the model away from the very words that repetition asks it to repeat and that word comprehension asks it to name. Rules score the 15 short-answer items. In word comprehension, the answer is correct when the first of the six options that it names, matched as a whole word regardless of case, is the target, which the prompt lists first. In sentence comprehension, the first whole word yes or no in the answer must equal the target. In repetition, the answer must equal the target after both are lowercased, stripped of surrounding quotation marks, spaces, and final periods and

Subtest	Prompt	Target
Connected text	Tell me about the best trip you ever took.<br>Describe a happy childhood memory.<br>Tell me about your first job.<br>What do you like about where you live?<br>Describe the steps to make a simple meal.	–
Word comprehension	Which one is an animal with a mane? (lion, drum, violin, giraffe, foot, boat)	lion
Sentence comprehension	Are babies watched by babysitters?	yes
Repetition	Please repeat exactly: The sun rises in the East.	The sun rises in the East.

exclamation marks, and have their whitespace collapsed. The exact match is strict for chat models, which often wrap the repetition in words of their own: the intact Llama-2 models repeat none of the five items exactly, and Llama-2-13B-chat answers “Sure! Breakfast.” when asked to repeat breakfast. The items lost by a lesion are the intact model’s number of correct short-answer items minus the lesioned model’s. The five-way classifier scores each connected-text answer that has at least 20 characters, and an LLM judge, DeepSeek-V4.1-Flash ([15]) at temperature 0, marks which of the TAB’s 19 aphasia features each connected-text answer shows, following the TAB’s protocol with our own in-context examples (Appendix § 17). The five connected-text prompts are also the five TAB prompts of the human evaluation of steering (§ 4.2).

MMLU. We also test every lesion on two subjects of MMLU (Hendrycks et al., 2021): global facts, with 100 test questions, and high school world history, with 237. Every model receives the same raw-text prompt, without a chat template, after the beginning-of-sequence token where the model uses one. The prompt is the standard five-shot prompt: the header The following are multiple choice questions (with answers) about <subject>.\n\n, where <subject> is global facts or high school world history, then five solved questions from the subject’s development split, and last the test question. Each question has the form <question>\nA. <a>\nB. <b>\nC. <c>\nD. <d>\nAnswer:, and a solved question continues with a space, its correct letter, and two newlines. The context of GPT-2 XL holds only 1,024 tokens, so for its world-history prompts we drop the earliest solved questions until the prompt fits, which leaves three solved questions for 106 test questions, two for 111, and one for 20; every other model keeps all five for both subjects. The model reads the prompt once and generates nothing. Its predicted answer is the letter, among the tokens of A to D, with the highest logit after the final Answer:, and the log-probability of the correct letter is taken over the full vocabulary, so that it also falls when a lesion moves probability away from the four letters altogether. The MMLU drop of a lesion is the intact model’s mean log-probability of the correct letter minus the lesioned model’s, in nats, averaged over the two subjects.

Compute. We ran the GPU and CPU-based experiments on a shared cluster. The GPUs ran for 236 hours: 96.7 on NVIDIA A100 GPUs with 80 GB, 95.6 on H200 GPUs with 141 GB, 31.1 on A100 GPUs with 40 GB, and 12.8 on older GPUs. The runs that completed account for 145.6 of these hours. CPU jobs ran about 2,200 allocated core-hours. These counts leave out the LLM judge, which we query through an API (Appendix § 17).

Software. The code runs on Python 3.13, with PyTorch 2.7.1 and Transformers 5.5.4, which loads every model and generates every answer. The classifier uses PEFT 0.19.1 for its LoRA adapters, and Sentence-Transformers 5.4.1 for its similarity features, which cosine similarity methods use too. PARAM scores answers with the BLEU of NLTK 3.9.4 and with bert-score 0.3.13. The probes, cross-validation, t-SNE, and nearest neighbors use scikit-learn 1.8.0, the statistics and figures use SciPy 1.17.1, NumPy 2.4.4, and Matplotlib 3.10.9.

Five-Way Classifier

We next describe the training data, features, architecture, training, and evaluation of the five-way classifier of § 3, and how it scores the answers of intact models.

Training data. The classifier learns from 2,843 texts: 2,584 AphasiaBank transcripts of the Cinderella and sandwich tasks, and 259 model scrambles (Table 8). The transcripts carry six group labels, and the classifier merges two of them, anomic and conduction aphasia, into one class (Appendix § 7). Some texts appear twice among the transcripts, and the training data keep both copies. In the sandwich task, 557 texts appear twice: for 492 of them, the two copies are two files of the same recording, and for the other 65, they are files of two different recordings with the same group label. In the Cinderella task, four recordings appear twice with the same text, once labeled anomic aphasia and once labeled control speech. The 2,584 transcripts therefore hold 2,023 distinct texts. § 3 counts each text once and leaves out the four texts with conflicting labels, which gives 1,139 Cinderella and 880 sandwich transcripts. No text occurs in both tasks, and no model scramble repeats a human transcript.

	Cinderella	Sandwich	Total
AphasiaBank transcripts
Control	399	936	1,335
Conduction or anomic	417	334	751
Anomic	276	235	511
Conduction	141	99	240
Broca	273	131	404
Wernicke	58	36	94
All transcripts	1,147	1,437	2,584
Distinct texts	1,143	880	2,023
Model scrambles
Llama-2-7B-chat	113	90	203
GPT-2 XL	56	0	56
All model scrambles	169	90	259
All training texts	1,316	1,527	2,843

Model scrambles. The 259 model scrambles are answers from a preliminary knockout study: 203 come from Llama-2-7B-chat, on both task prompts, and 56 from GPT-2 XL, all on the Cinderella prompt. All but five are answers of models with knocked-out attention heads or feed-forward blocks, and the other five are temperature samples of the intact GPT-2 XL. The knockouts remove single heads, sets of heads, or the feed-forward block of a whole layer, and 132 of these answers come from successive steps of a greedy search that knocks out more and more heads of Llama-2-7B-chat. Of the answers with knockouts, 28 come from activation patching, which restores the clean activations of one component in a knocked-out model, and 11 are temperature samples of Llama-2-7B-chat with three heads knocked out. An LLM judge flagged these answers as machine failures among the outputs of the preliminary study, and a linguist and a clinician reviewed them. The model scrambles are 12 to 1,090 characters long, with a median of 571, and all 259 are distinct. By the repetition statistics of Table 9, 233 of them contain a repeated unit and 196 a unit that occurs three or more times, whereas 799 of the 1,335 control transcripts contain no repeated unit.

Feature	Definition
Narrative measures
Utterances	Number of utterances, each ending in “.”, “!”, or “?”
Word tokens	Number of words
Mean length of utterance	Mean number of words per utterance, leaving out utterances that are a single filler word such as “uh” or “yeah”
Repetition statistics
Shortest repeated unit	Length in words of the shortest repeated unit, or 0 if there is none
Most repetitions	Largest number of times that a repeated unit occurs, or 0 if there is none
Word-list proxies
Open- to closed-class ratio	Number of open-class words over number of closed-class words, which are the words of a list of 101 function words
Share of open-class words	Number of open-class words over number of words
Words ending in -ly	Number of words ending in -ly, a proxy for adverbs
Task nouns	Number of words from a list of 33 nouns of the two tasks
Fragments and stutters	Number of words that end in a hyphen, plus letters repeated after a hyphen
Similarity features
Control minus Broca	Similarity to the control transcripts minus similarity to the Broca transcripts
Control minus Wernicke	Similarity to the control transcripts minus similarity to the Wernicke transcripts
Control minus conduction or anomic	Similarity to the control transcripts minus similarity to the transcripts of conduction or anomic aphasia
Similarity to control	Similarity to the control transcripts, meant as control minus model scrambles

Judging machine-specific output. To distinguish machine-specific output failures from linguistically comparable clinical aphasic speech, we implemented an LLM judge pipeline to automatically detect and annotate machine artifacts. Using the Gemini 3.5 Flash model, this automated judge evaluates outputs from perturbed models to filter out “mechanical garbage”, which are instances where the model breaks down mechanically rather than simulating a valid human linguistic profile. By identifying these “model scrambles,” we prevent machine-generated noise from contaminating the mapping between human clinical profiles and internal model components during the ablation pipeline.

The judge’s evaluation is governed by a strict, zero-temperature system prompt that evaluates text against five specific artifact dimensions: Stray Brackets (SB), Token Repetition (TR), Tokenizer Loops (TL), Formulaic AI-isms (FM), and Sentence Repetition (SR). To ensure precise categorization, the model outputs a structured JSON object detailing its step-by-step reasoning and the specific artifacts detected. Crucially, the prompt includes a tie-breaker rule designed specifically for clinical data: if a repetition mimics natural human hesitation or spoken disfluency, the LLM assumes a valid human purpose and refrains from flagging it as a machine artifact. This ensures that the valid disfluencies characteristic of aphasic subtypes are preserved while machine-specific degradations are correctly recognized. For the complete zero-temperature system prompt, including the explicit artifact rubric and few-shot evaluation examples. Listing 1 shows the whole prompt.

To ensure the reliability of the LLM judge pipeline, the system prompt was iteratively calibrated against a gold-label subset of the dataset to maximize its baseline agreement rate. Furthermore, to validate the integrity of the automated filtering, the final classifications were audited by human domain experts. A linguist and a clinician manually reviewed the text flagged by the LLM, confirming that the isolated “model artifacts” were true machine-specific failures and not misclassified clinical disfluencies. This empirical prompt tuning combined with expert clinical verification ensures that our artifact removal process is both scalable and clinically accurate.

Finally, these verified “model artifacts” are incorporated directly into the training data for our 5-way clinical diagnostic classifier.

Features. Besides its text encoder, the classifier reads the 14 features of Table 9. The three narrative measures reimplement measures that the Computerized Language Analysis (CLAN) program computes for AphasiaBank transcripts ((MacWhinney, 2018)), so that the classifier can compute them from plain text such as a model’s answer. On the human transcripts, they correlate with CLAN’s values at a Pearson’s $r$ of 0.986 for the number of utterances, 0.993 for the number of word tokens, and 0.968 for the mean length of utterance. The five word-list proxies need no part-of-speech tagger, but they approximate CLAN’s measures less closely. The open- to closed-class ratio and the share of open-class words correlate only weakly with the corresponding CLAN measures, at $r = 0.095$ and $r = -0.050$. The count of words ending in -ly correlates with CLAN’s adverb count at $r = 0.618$, and 73.2% of its matches in the sandwich transcripts are the word jelly. The count of fragments and stutters is zero on every human transcript, although CLAN counts word errors in 1,090 of them. The four similarity features embed the whole text with all-MiniLM-L6-v2 ((Wang et al., 2020; Reimers and Gurevych, 2019)) and compare it, by mean cosine similarity, with the human transcripts of each group, pooled over both tasks. Three of them subtract the mean similarity to the transcripts of Broca’s aphasia, Wernicke’s aphasia, or of conduction or anomic aphasia from the mean similarity to the control transcripts. The fourth was meant to subtract the mean similarity to the model scrambles in the same way. The code, however, looks up the model scrambles under a label that differs from their label in the data by one character, finds none, and subtracts zero, so the fourth feature is the mean similarity to the control transcripts itself. Training and scoring share this mismatch, so the classifier reads the same feature in both, but the feature is not a contrast with model scrambles. Within the training data, each transcript is left out of its own group’s mean. MiniLM reads at most 256 tokens, so the similarity features see only the beginning of 820 of the 2,843 training texts, 719 of them Cinderella transcripts.

Architecture. The classifier encodes a text with BGE-large-en-v1.5 ((Xiao et al., 2024)), an encoder with 24 layers and 1,024-dimensional states, which reads at most 384 tokens. The encoder thus sees only the beginning of 561 of the 2,843 training texts: 528 of the 1,147 Cinderella transcripts, 25 of the 1,437 sandwich transcripts, and 8 of the 259 model scrambles. We fine-tune the encoder with low-rank adapters (LoRA; (Hu et al., 2022)) of rank 16, with $\alpha= 32$ and a dropout of 0.1, on all six linear layers of each of its 24 blocks: the query, key, value, and output projections of attention, and the two layers of the feed-forward block. The encoder’s own weights stay frozen. The embedding of a text is the mean of the encoder’s last-layer states over the text’s tokens, normalized to unit length. A two-layer perceptron takes this 1,024-dimensional embedding together with the 14 standardized features, 1,038 inputs in all, maps them to 256 hidden units with a rectified linear activation and a dropout of 0.3, and then maps these to the logits of the five classes.

Training. We train the adapters and the perceptron jointly with the cross-entropy loss over the five classes, using AdamW ((Loschilov and Hutter, 2019)) with a constant learning rate of $10^{-4}$, a weight decay of 0.01, and gradient norms clipped at 1.0. Each epoch draws as many texts as the training split holds, in batches of four. It draws them with replacement, with probabilities inversely proportional to the size of each text’s class in the training split, so that the five classes are equally frequent in expectation. After each epoch, we compute the macro-averaged F1 over the five classes on the validation split. We keep the epoch with the highest value, and we stop after five epochs without an improvement larger than $10^{-4}$, or after 20 epochs. We train 20 such models, each with its own random seed and its own random split of the 2,843 texts into 70% training, 10% validation, and 20% test texts. Each text remains intact and belongs to only one set within a split. Each split is stratified by the six group labels, so that anomic and conduction aphasia are split separately, and has 1,990 training, 284 validation, and 569 test texts. The splits are random over transcripts, not over speakers or distinct texts. To score new text, the classifier averages the softmax outputs of the 20 models, and $P(\text{aphasia})$ is the sum of the averaged probabilities of the three aphasia classes.

Evaluation. A text can fall in the test splits of several models, so we score each text by the mean softmax output of the models whose test split holds it, and we take its most probable class. Of the 2,843 texts, 2,798 fall in at least one test split, in 4.07 test splits on average and in at most 11; the other 45, four of them model scrambles, never do and are not evaluated. These held-out scores thus come from subsets of the 20 models rather than from the full ensemble that scores model answers. Table 10 gives precision, recall, and F1 per class, and Table 11 gives the confusion matrix. The classifier predicts none of the 1,317 held-out control transcripts as Broca’s aphasia, Wernicke’s aphasia, or model scrambles, and 25 of them as conduction or anomic aphasia. It recognizes 254 of the 255 model scrambles, and of the 2,543 human transcripts, it calls only two model scrambles, both Broca transcripts of the Cinderella task. Most errors involve conduction or anomic aphasia: the classifier predicts 78 of its 737 transcripts as control speech and 75 as Broca’s aphasia, and it predicts 58 of the 397 Broca transcripts and 35 of the 92 Wernicke transcripts as conduction or anomic aphasia. Wernicke’s aphasia has the lowest F1, 0.512. By the aphasia target of § 3, $P(\text{aphasia}) > 0.5$, the classifier separates the 1,226 aphasic from the 1,317 control transcripts with a precision of 0.979, a recall of 0.935, and an accuracy of 0.959. Single models reach a mean accuracy of 0.865 and a mean macro-averaged F1 of 0.789 on their own test splits.

Class	Precision	Recall	F1	N
All 2,798 held-out texts
Control	0.942	0.981	0.961	1,317
Cond. or anomic	0.828	0.769	0.797	737
Broca	0.783	0.816	0.799	397
Wernicke	0.583	0.457	0.512	92
Model scrambles	0.992	0.996	0.994	255
Macro average	0.826	0.804	0.813	2,798
Accuracy			0.886	2,798
The 1,693 held-out texts that occur only once
Control	0.857	0.970	0.910	397
Cond. or anomic	0.816	0.748	0.780	603
Broca	0.772	0.811	0.791	359
Wernicke	0.517	0.380	0.438	79
Model scrambles	0.992	0.996	0.994	255
Macro average	0.791	0.781	0.783	1,693
Accuracy			0.833	1,693

True class	Control	Cond. or anomic	Broca	Wern.	MS
Control	1,292	25	0	0	0
Cond. or anomic	78	567	75	17	0
Broca	0	58	324	13	2
Wernicke	1	35	14	42	0
Model scrambles	0	0	1	0	254
Total	1,371	685	414	72	256

Overlap between training and test splits. Because the splits are random over transcripts, a test transcript can share its text or its speaker with the training split. In each model, 146 to 170 of the 569 test texts have an identical copy in the training split. On the 1,693 held-out texts that occur only once in the training data, so that no training split holds a copy, the accuracy falls from 0.886 to 0.833 and the macro-averaged F1 from 0.813 to 0.783 (Table 10). Part of the change in precision comes from the class mix, because most duplicates are sandwich control transcripts. Recall, which does not depend on the class mix, also falls for every class of human speech, most for Wernicke’s aphasia, from 0.457 to 0.380. AphasiaBank also records most speakers more than once, in both tasks and in repeated sessions. When we identify a speaker by the corpus and the participant code of each file, 446 to 475 of the 517 human test transcripts of each model share their speaker with a transcript in that model’s training split, 89.2% over all 20 models. Splits by speaker would remove this overlap and measure accuracy on new speakers.

Listing 1: User message of the artifact detection prompt, used to filter out mechanical model breakdowns. We define five dimensions of machine-specific artifacts and provide four examples demonstrating the expected reasoning and JSON output format.

You are an expert linguistic annotator
    → evaluating transcripts generated by an
    → intentionally impaired, lesioned
    → language model. The model was designed
    → to simulate human clinical
    → disfluencies (like aphasia or
    → word-finding pauses).

Your task is to filter out the mechanical
    → garbage. You must detect the presence
    → of 'machine-specific
    → artifacts'—instances where the model
    → failed to produce valid linguistic
    → output and instead broke down
    → mechanically (e.g., tokenizer loops,
    → stray brackets, or canned AI safety
    → refusals).

Evaluate the text against the following 5
    → dimensions. If ANY of these dimensions
    → are present, the text contains an
    → artifact.

RUBRIC:
1. SB (Stray Brackets): Unparsed, artifactual
    → brackets or delimiters (e.g., [[...]],
    → <<, <|endoftext|>), or emojis. Do NOT
    → flag standard citations [1] or normal
    → markdown.
2. TR (Token repetition): Unnatural
    → consecutive repetition of a token,
    → symbol, or punctuation mark.
- Standard Ellipses Exemption: A standard
    → 3-dot ellipsis ("...") is normal human
    → punctuation or transcriber notation.
    → Do NOT flag it.
- Artifact Threshold: 4 or more consecutive
    → periods (e.g., "....", "........") or
    → repeated commas (", , ,") is a machine
    → degeneration and MUST be flagged.
3. TL (Tokenizer Loops): Degenerate sub-word
    → or morphological looping (e.g., "tion
    → tion tion" or "the the the the the").
4. FM (Formulaic AI-isms): Standardized AI
    → guardrail messages, apologies, or
    → assistant-like framing that completely
    → break the narrative task. Look for
    → safety refusals ("I apologize, but I
    → am not able to..."), ethical
    → disclaimers, or persona leaks ("As an
    → AI...").
5. SR (Sentence Repetition): Exact or
    → near-exact repetition of a well-formed
    → sentence or independent clause within
    → the text.

TIE-BREAKER RULE:
If a repetition (TR or SR) appears to serve a
    → valid human pragmatic, emotional, or
    → stylistic purpose (e.g., stuttering in
    → spoken transcripts, emphasizing a
    → point), assume it is human and DO NOT
    → flag it.

INSTRUCTIONS:
1. Analyze the text step-by-step against all 5
    → dimensions.
2. Output a JSON object containing your
    → step-by-step ”reasoning”, a boolean
    → ”contains_artifact”, and a list of
    → ”artifact_types” found (use the
    → abbreviations SB, TR, TL, FM, SR). If
    → none, return an empty list [].

EXAMPLES:

Text: ”Sure, here are some examples of how the
    → Cubs are calculated. In general, the C
    → is a complex, b, c, and d are the
    → same. The C is a, b, a, c. The, a, b,
    → C. The, aC, a, aC, a,aC, a,aC,a,
    → a,a,a,a,[C, a,a[[[[[[[[the, C, a, a,
    → a, a, C, a, a,C[[[[[[[[[[, a, a,
    → a, C,[[[[[[, a, a,C,[[[[, a,
    → a,[[[[, a,[[[[, a,[[[, a,[[[, L, a,
    → a,[[[[[, [[C”

Output:
{
  ”reasoning”: ”The text contains a great
    → number of unparsed brackets '['
    → which flags as SB.”,
  ”contains_artifact”: true,
  ”artifact_types”: [”SB”]
}

Text: ”I apologize, but I'm not able to
    → understand or produce sound or create
    → any content that may offend or be
    → inappropriate or offensive. Please let
    → me know if you have any questions or
    → concerns. I apologize, but I'm not
    → able to understand or produce any
    → content that may offend or be
    → inappropriate or offensive. Please let
    → me know if you have any questions or
    → concerns.”

Output:
{
  ”reasoning”: ”The text contains a classic AI
    → disclaimer ('but I'm not able to
    → understand or produce sound or

  create any content that may offend
  or be inappropriate or offensive'),
  which flags as FM. It also contains
  the disclaimer twice, which flags as
  SR.",
"contains_artifact": true,
"artifact_types": ["FM", "SR"]
}

Text: "I went to the store and, well, I I I
  just couldn't remember what to buy."
Output:
{
  "reasoning": "The token 'I' is repeated
    three times. However, applying the
    tie-breaker rule, this mimics
    natural human hesitation and spoken
    disfluency, not a machine tokenizer
    loop. No other artifacts are
    present.",
  "contains_artifact": false,
  "artifact_types": []
}

Text: "Cinderella is a fairy tale about a
  young girl who is born into a poor
  family. She is very poor, and her
  father is very cruel to her. She is
  very poor, and her father is very
  cruel to her. She is very poor, and
  her father is very cruel to her. She
  is very poor, and her father is very
  cruel to her. She is very poor, and
  her father is very cruel to her. She
  is very poor, and her father is very
  cruel to her. She is very poor, and
  her father is very cruel to her. She
  is very poor, and her father is very
  cruel to her. She is very poor, and
  her father is very cruel to her. She
  is very poor, and her father is very
  cruel to her. She is very poor, and
  her father is very cruel to her. She
  is very poor, and her father is"
Output:
{
  "reasoning": "The exact sentence ' She is
    very poor, and her father is very
    cruel to her.' is repeated about 12
    times consecutively without any
    pragmatic human justification. This
    is a degenerate generation loop.",
  "contains_artifact": true,
  "artifact_types": ["SR"]
}

Evaluate this text:
Text: "{transcript}"
Output:

Lesion Methods

We describe CLAMP and the baselines of Table 2 in detail, add the lesion spaces and searches that the main text leaves out, and compare the answers that the lesions of each method produce. A lesion works when the lesioned model’s answer counts, by the rules of § 3, and reaches the target: the classifier either gives the target a probability above 0.5 or ranks the target as its most probable class, where any of the three aphasia classes counts for the aphasia target. Every search that consults the classifier uses only the first of these two tests, a target probability above 0.5, to decide when to stop and what to keep or prune, and the second test judges only the answer that a search finally reports. A search that never passes 0.5 can thus still report a lesion that works, as 3 of the 54 lesions of CLAMP do. The comparisons with the baselines count the 42 settings of the seven older models, GPT-2 XL, the two Llama-2-chat models, and the four Pythia models, as Table 2 does. They support § 5.1 in three ways: searches guided by the classifier find far more lesions than searches guided by corpus statistics, heads give fewer degenerate answers than weight lesions, and the lesions of every method, not only those of CLAMP, rarely carry their target to new prompts.

A knockout zeroes a head’s output at every position. CLAMP knocks out a head by zeroing the head’s slice of the input to the attention output projection, at every position of the prompt and of the answer. In GPT-2 XL and the Pythia models, the value projection has a bias $b_V$, so the knockout also removes the term $W_O b_V$ that the head adds at every position, where $W_O$ is the head’s slice of the output projection. In the softmax-attention layers of Qwen3.5, a gate scales the output of each head before the output projection, and the knockout removes the gated output. In the Gated DeltaNet layers, each value head keeps its own recurrent state, and the knockout zeroes the head’s slice of the input to the layer’s output projection. Both kinds of head count as heads, so Qwen3.5-0.8B and Qwen3.5-2B have 336 heads and Qwen3.5-4B and Qwen3.5-9B have 896, of which 14.3% sit in softmax layers. The older models have from 128 heads, in Pythia-1B, to 1,600, in Llama-2-13B-chat.

Many heads change nothing when knocked out alone, so they tie in the ranking. For each model, task, and target, CLAMP first knocks out every head alone and records $s(\{h\})$, the target probability of the answer. A head whose answer does not count ranks last. No head of the seven older models gives such an answer, and only one head of Qwen3.5-0.8B does, on Cinderella. The three targets of a model and task share one cache of answers, and the seven older models reuse the single-head answers of the first stage of the layer search, another classifier-guided search in Table 13. CLAMP ranks the heads by $s(\{h\})$, from highest to lowest, with a stable sort, so heads with equal values keep their order by layer and, within a layer, by index. Ties are common in the larger older models (Table 12). In Llama-2-13B-chat, for example, 1,305 of the 1,600 single-head knockouts leave the Cinderella answer word for word unchanged. The answers of Qwen3.5 change far more often: a single-head knockout leaves the answer unchanged for 1 of the 336 heads of Qwen3.5-2B on Cinderella and for 2 on sandwich, and for 64 of the 896 heads of Qwen3.5-9B on Cinderella. The order of tied heads thus comes from where they sit in the model, not from what they do.

Model	Task	Aph.	Broca	Wern.
Llama-2-13B-chat	Cinderella	1,305	1,307	1,313
	Sandwich	1,298	1,298	1,298
Llama-2-7B-chat	Cinderella	568	569	571
	Sandwich	449	450	452
GPT-2 XL	Cinderella	569	569	569
	Sandwich	397	412	417
Pythia-2.8B	Cinderella	933	933	933
	Sandwich	316	487	485
Pythia-1B	Cinderella	9	9	10
	Sandwich	5	9	14

Lesion space and search	Aph.	Broca	Wern.	All	GPT-2 XL	Llama-2-chat	Pythia
Heads
HEAD-MASK	5	0	0	5	1	0	4
HEAD-MASK-HARD	8	1	1	10	1	0	9
HEAD-GROWN-COS	2	0	0	2	1	0	1
Layer search, first stage	5	1	2	8	0	0	8
HEAD-CEM	9	8	6	23	1	0	22
CLAMP	14	11	13	38	5	11	22
Heads and MLP blocks
Layer search, all stages	9	6	7	22	2	2	18
MLP lesions
PARAM-MASK	4	0	0	4	1	0	3
PARAM-MASK-HARD	3	1	0	4	0	0	4
Cross-entropy method	13	13	10	36	6	6	24
Grown	9	6	3	18	3	2	13
Attention lesions
Grown	10	9	7	26	2	5	19
MLP and attention lesions
PARAM†	2	0	0	2	1	0	1
PARAM-C	11	10	4	25	3	4	18
MLP lesions and heads
Learned mask, soft	4	1	0	5	0	0	5
Learned mask, hard	4	0	0	4	0	0	4
Cross-entropy method	12	12	9	33	5	4	24
Grown, then head sweep	12	13	13	38	6	9	23

Growth adds heads in ranking order, so ties can make a grown set large. CLAMP then knocks out the top $k$ heads of the ranking together, for $k = 1, 2, 3, \ldots$, so that each prefix adds one head to the previous one. It generates the prefixes in blocks of the model’s batch size, 25 prefixes for GPT-2 XL, 40 for Llama-2-13B-chat, and 32 for the other models, and it stops after the first block that contains a prefix whose target probability exceeds 0.5. The grown set is the shortest such prefix. When no prefix exceeds 0.5, up to the prefix of all heads, the growth reports the prefix with the highest target probability, the shortest one among ties, and the search ends without pruning. Of the 15 growths that never pass 0.5, three still report a lesion that works, because the target is the classifier’s most probable class: the Pythia-1.4B sandwich Broca prefix of 316 heads, at $P(\text{Broca}) = 0.488$, and the Qwen3.5-2B and Qwen3.5-4B sandwich Wernicke prefixes of 63 and 228 heads, at $P(\text{Wernicke})$ of 0.463 and 0.423. Because tied heads keep their layer order, a growth that runs past the heads that raise the target probability continues through the tied heads from the first layer upward. The grown set of Llama-2-7B-chat for the Wernicke target on Cinderella, for example, holds 740 heads, more than 560 of them tied, and the grown set of GPT-2 XL for the Broca target on sandwich holds 984 heads, more than 400 of them tied. Shrinking and pruning reduce these two sets to 276 and 146 heads, but the layer positions of large lesions partly reflect the order of the tied heads rather than only the roles of the heads.

Pruning removes heads until none can go alone. A grown set of more than 64 heads is first shrunk in chunks. CLAMP cuts the set, in ranking order, into consecutive chunks of half its size and tries the removal of each chunk in one batch. Among the removals that keep the target probability above 0.5, it takes the one that leaves the highest target probability and cuts the remaining set again. When no chunk can go, it halves the chunk size, and it stops before chunks of a single head. The weakest heads cannot simply be cut from the end of the set, because every shorter prefix of the ranking already stayed at or below 0.5, so a chunk may come from anywhere in the ranking. Pruning then works in rounds. Each round tries every leave-one-out set, the set without one of its heads, at a cost of one answer per head. The heads whose removal alone keeps the target probability above 0.5 are ranked by the target probability that remains after their removal. The round removes all of them at once if the probability stays above 0.5, and otherwise the best half, the best quarter, and so on down to the single best head. Each joint removal costs one more answer to check, because two heads that can each go alone may not be able to go together. Pruning ends when no head can go alone, so that leaving any one head out of the pruned lesion brings its target probability to 0.5 or below.

Seven lesions are not fully pruned, but nearly all of their heads are needed. A pruning round starts only if the answers that pruning has used so far, plus one answer per head of the set, stay within a budget of 15,000 answers. Otherwise, CLAMP reports the set as it stands and flags it as not minimal. The budget counts the answers of pruning only, not those of growth or shrinking. Four of the 54 working lesions stopped at the budget: GPT-2 XL Cinderella Wernicke with 244 heads, Llama-2-7B-chat Cinderella Wernicke with 276, Pythia-6.9B Cinderella Broca with 578, and Qwen3.5-4B Cinderella aphasia with 208. The three lesions that reach their target only as the most probable class were never pruned, because their growth never passed 0.5. Activation patching shows that each of these seven lesions needs 91 to 99% of its heads, in that leaving out one of these heads undoes the lesion (Appendix G). By the search’s own test, a target probability above 0.5, another round of pruning would have removed none of the 276 heads of the Llama-2-7B-chat lesion. The sizes of the seven lesions are thus upper bounds, but tight ones.

PARAM groups its lesions by the patient group that the answer resembles. PARAM follows the lesion procedure of Wang et al. (2025a). It scores every weight $w$ of the attention and multilayer perceptron (MLP) weight matrices by $\lvert w \, \partial L/\partial w\rvert$, where $L$ is the negative log-likelihood of the control transcripts, each read as the model’s answer to its task’s prompt. It leaves out the weights that read an input channel with exceptionally large activations. A lesion zeroes the top 1% of the weights of one matrix, which gives 288 single lesions in GPT-2 XL, 224 in Llama-2-7B-chat, and 96 to 192 in the Pythia models. PARAM lesions one matrix at a time and compares the answer with the task’s transcripts of each of five groups, control, anomic, Broca, conduction, and Wernicke, by BLEU-1 (Papineni et al., 2002) and by BERTScore (Zhang et al., 2020). It standardizes each metric per group across the lesions, averages the two standardized metrics, and assigns the lesion to the group with the highest average. For each of the four aphasia types, PARAM lesions together the matrices assigned to that type and, while the answer of this cluster is not assigned the type, adds the remaining matrix with the highest score for the type, up to 32 matrices in all. We did not run PARAM on Llama-2-13B-chat. The classifier scores the final answer of each cluster: PARAM works for the aphasia target when any of the four clusters reaches it, and for the Broca or the Wernicke target when the Broca or the Wernicke cluster reaches that target. No setting has more than one working cluster. The classifier thus judges the lesions of PARAM but plays no part in finding them.

PARAM-C grows the lesions of PARAM with the classifier. PARAM-C searches the lesions of PARAM with the classifier in place of BLEU-1 and BERTScore. It ranks the MLP lesions by their single-lesion target probability and grows them in that order, as CLAMP grows heads. When no prefix works, it sweeps through the layers from first to last on top of the prefix with the highest target probability, adding in each layer the attention lesion that raises the target probability most, up to four per layer, moving to the next layer after two additions in a row that gain nothing, and stopping as soon as a set works. PARAM-C then prunes a working set one lesion per round, removing the lesion whose removal leaves the highest target probability, as long as that probability stays above 0.5. Unlike PARAM, PARAM-C also runs on Llama-2-13B-chat, so it covers all 42 settings.

The learned masks train on the transcripts and use the classifier only to choose and prune. PARAM-MASK, PARAM-MASK-HARD, HEAD-MASK, and HEAD-MASK-HARD learn a knockout amount $k_i = \sigma(\theta_i)$ between 0 and 1 for each component $i$, where $\sigma$ is the logistic function, and they train only $\theta$, with the model frozen (Cordsodás et al. (2021)). The components of HEAD-MASK are the heads, whose outputs the mask scales by $1-k_i$. The components of PARAM-MASK are the MLP lesions, the top-1% lesions of every MLP weight matrix, whose lesioned weights the mask scales by $1-k_i$. The training data are the task’s Aphasia-Bank transcripts, each read as the model’s answer to the task’s prompt. For a transcript $x$ of $n$ tokens, $r(x) = (\log p_k(x)-\log p_0(x))/n$ is the per-token log-likelihood ratio of the masked model $p_k$ to the intact model $p_0$. The loss

pushes $r$ up on the transcripts $T$ of the target group and down on the transcripts $C$ of the other groups, with an $\ell_1$ penalty on the amounts, where $\operatorname{softplus}(z)=\log(1+e^z)$ and $\tau=0.1$ nats per token. For the aphasia target, $T$ holds the transcripts of the four aphasia types and $C$ the control transcripts; for the Broca or the Wernicke target, $T$ holds that type’s transcripts and $C$ those of the other four groups. Training follows a path of six decreasing values of $\lambda$, with 50 steps each, and each stage starts from the mask of the previous stage and ends in a checkpoint. The model then answers the task under each checkpoint in two versions: the soft mask, with the learned amounts, for PARAM-MASK and HEAD-MASK, and the hard mask, which knocks out the components with $k_i > 0.5$, for PARAM-MASK-HARD and HEAD-MASK-HARD. Each version reports its sparsest checkpoint whose target probability exceeds 0.5, and otherwise its checkpoint with the highest target probability. A hard mask above 0.5 is then pruned one component per round, as PARAM-C prunes its sets. The classifier plays no part in training, but it scores the answers of the checkpoints, picks the checkpoint to report, and decides which components pruning may remove.

HEAD-CEM samples whole sets of heads and learns which heads the best sets share. HEAD-CEM applies the cross-entropy method [13] (Rubinstein 1999) to sets of heads. It keeps a probability $p_i$ of knocking out each head $i$ and, in each round, samples 64 sets, each of which holds each head independently with probability $p_i$, and scores their answers. The search stops in the first round in which a set’s target probability exceeds 0.5 and keeps the smallest such set of that round. Otherwise, it moves each $p_i$ halfway toward the share of the best quarter of the round’s sets that hold head $i$, and it keeps every $p_i$ at or above a floor of $\min(12/N, 0.05)$, where $N$ is the number of heads. The search starts from the single-knockout values: the 16 heads with the highest $s(\{h\})$ start at $p_i = 0.3$ and the other heads at the floor. After 20 rounds, or 1,280 sets, without a set above 0.5, the search reports the set with the highest target probability. A set above 0.5 is pruned one head per round, as PARAM-C prunes its sets. Unlike growth, which ranks each head by its effect alone, the cross-entropy method judges whole sets, so it can find heads that work only together.

HEAD-GROWN-COS replaces the classifier with a similarity to the transcripts. This search ranks, grows, and prunes heads as CLAMP does, but with a similarity objective in place of the classifier’s target probability. The objective is the mean cosine similarity between the answer and the transcripts of the target group, minus the mean cosine similarity between the answer and the control transcripts. It uses the MiniLM sentence embeddings (Wang et al., 2020; Reimers and Gurevych, 2019) and the reference transcripts of the classifier’s similarity features. The target group holds all aphasic transcripts for the aphasia target and the Broca or the Wernicke transcripts for the other two targets, and a set satisfies the objective when the difference exceeds 0. The classifier then scores the answer of the reported set, and Table 2 counts the settings in which that set works by the classifier.

More lesion spaces and searches separate the effect of the space from that of the search. Table 13 adds the lesion spaces and searches that the main text leaves out. An attention lesion zeroes the top 1% of one query, key, value, or output matrix, as an MLP lesion does for one MLP matrix, and the lesions of PARAM draw on both kinds of matrix. The learned masks and the cross-entropy method also search the MLP lesions alone and the MLP lesions and heads together. A grown search ranks MLP lesions or attention lesions by their single-lesion target probability, grows them in that order as CLAMP grows heads, and prunes a set above 0.5. One more search starts from the grown MLP lesions as PARAM-C does but, when these do not pass 0.5, sweeps heads through the layers in place of attention lesions. The layer search, another classifier-guided search, also works layer by layer. Its first stage knocks out every head alone and then the top 1, 2, 3, 5, 8, or 12 heads of each layer together, and its later stages sweep through the layers, adding heads and then blocks of 256 MLP neurons on top of those already knocked out. Growth, the cross-entropy method, and the learned masks each run in several spaces, so the table compares spaces under one search and searches within one space.

Heads grown by the classifier work in the most settings without degenerate answers. Grown the same way, whole heads work in 38 settings, attention lesions in 26, and MLP lesions in 18. The search matters too: over heads, the first stage of the layer search works in 8 settings and HEAD-CEM in 23, and the layer search with all its stages, which adds blocks of MLP neurons, works in 22. The cross-entropy method works in more settings with MLP lesions than with heads, 36 against 23, but 6 of its 36 answers with MLP lesions repeat themselves. Only one other search matches the 38 settings of CLAMP: the grown MLP lesions with heads swept on top work in more Broca settings, 13 against 11, but in fewer Llama-2-chat settings, 9 against 11, and give 5 degenerate answers, which repeat themselves or have model scrambles as their most probable class, against none (Table 14). CLAMP works in 11 of the 12 Llama-2-chat settings, more than any other method. Only growth over heads thus reaches 38 settings without a degenerate answer.

Lesion space and search	Sets	Size	Target	Scr.	zlib	Length Chars	Length <100	Held out TAB	Held out Other	Degen.
Heads
HEAD-MASK	5	97	0.74	0.000	0.72	199	1	0.114	0.165	0
HEAD-MASK-HARD	10	9	0.85	0.001	0.80	135	4	0.035	0.009	0
HEAD-GROWN-COS	2	22.5	0.74	0.256	0.48	470	0	0.009	0.008	1
Layer search, first stage	8	2.5	0.67	0.015	0.80	158	2	0.012	0.007	0
HEAD-CEM	23	8	0.60	0.094	0.97	74	12	0.010	0.001	1
CLAMP	38	33	0.72	0.062	0.79	166	11	0.009	0.002	0
Heads and MLP blocks
Layer search, all stages	22	6.5	0.59	0.135	0.81	150	9	0.016	0.007	0
MLP lesions
PARAM-MASK	4	13.5	0.69	0.123	0.70	458	0	0.033	0.007	0
PARAM-MASK-HARD	4	4	0.72	0.086	1.05	65	3	0.133	0.487	0
Cross-entropy method	36	6.5	0.66	0.095	0.63	216	13	0.020	0.001	6
Grown	18	3	0.67	0.064	0.71	202	7	0.033	0.007	2
Attention lesions
Grown	26	10.5	0.69	0.098	0.75	155	8	0.066	0.021	3
MLP and attention lesions
PARAM	2	28.5	0.76	0.002	0.74	196	0	–	–	0
PARAM-C	25	9	0.69	0.064	0.70	129	10	0.031	0.007	3
MLP lesions and heads
Learned mask, soft	5	25	0.71	0.077	0.80	152	2	0.137	0.135	0
Learned mask, hard	4	4	0.73	0.028	0.79	173	1	0.029	0.012	0
Cross-entropy method	33	9	0.65	0.094	0.81	149	15	0.029	0.002	3
Grown, then head sweep	38	18.5	0.63	0.120	0.63	247	13	0.020	0.001	5

Heads give fewer degenerate answers than weight lesions. Table 14 describes the working sets of each method. We call a working answer degenerate when it repeats itself, with a zlib compression ratio below 0.25, or when the classifier’s most probable class for it is model scrambles. The compression ratio divides the length of the zlib-compressed answer by the length of the answer, both in bytes. Heads give fewer such answers than weight lesions both under growth and under the cross-entropy method. Growth gives none among the 38 working sets of CLAMP, against 3 among its 26 sets of attention lesions and 2 among its 18 sets of MLP lesions. The cross-entropy method gives 1 among the 23 sets of HEAD-CEM, against 6 among its 36 sets of MLP lesions and 3 among its 33 sets of MLP lesions and heads. The degenerate answer of HEAD-CEM, for Pythia-1B on Cinderella with the aphasia target, reaches $P(\text{aphasia}) = 0.503$, but its most probable class is model scrambles, at $P(\text{scrambles}) = 0.479$. The 6 degenerate answers of the cross-entropy method with MLP lesions all repeat themselves, and all come from GPT-2 XL and the Llama-2-chat models. Over its three spaces, the cross-entropy method gives 10 degenerate answers among 92 working sets, 9 of them self-repeating. The search that sweeps heads on top of grown MLP lesions gives 5 degenerate answers among 38, and PARAM-C gives 3 among 25, whereas the 32 working sets of the learned masks give none. One of the two working sets of HEAD-GROWN-COS, GPT-2 XL sandwich aphasia, is degenerate too, with a ratio of 0.217 and model scrambles as its most probable class. Over the methods of Table 14 other than the layer search with MLP blocks, sets of heads alone give 2 degenerate answers among 86, and sets that include weight lesions give 22 among 195. Knocking out whole heads thus rarely turns the answer into a machine failure, which supports the hypothesis of § 4.1.

which their searches never saw. Per method, the median target probability on the connected-text prompts is at most 0.137. Over all methods, the mean target probability on these prompts exceeds 0.5 for only 3 of the 299 working sets checked there, and the answer to the other task reaches the target for only 24 of the 282 working sets whose answers there counts. No working set of CLAMP or HEAD-CEM exceeds 0.5 on the connected-text prompts, where their medians are 0.009 and 0.010. The failure of CLAMP’s lesions to carry their type beyond the narrative, in § 5.1, is thus common to every lesion method.

The lesions of every method rarely carry their target to new prompts. Table 14 also reports how the working sets do on the TAB’s five connected-text prompts and on the other task,

HEAD-CEM tends to find short answers. The sets of HEAD-CEM sit closer to 0.5 than those of CLAMP, with a median target probability of 0.604 against 0.717, and their answers are shorter, with a median of 74 characters against 166.5. The length difference is largest for the Broca target. All 8 Broca answers of HEAD-CEM have fewer than 100 characters, with a median of 34.5, and 6 of them begin with “I don’t know, but it”. Three answers of HEAD-CEM, two for the Broca target and one for the aphasia target, are exactly “I don’t know, but it’s a good one!”, which the classifier scores at $P(\mathrm{Broca}) = 0.575$ and $P(\mathrm{scrambles}) = 0.346$. The 11 Broca answers of CLAMP have a median of 227 characters, and 3 of them are shorter than 100. On the 7 Broca settings in which both methods work, the answer of HEAD-CEM is the shorter one in 6. The short answer is not specific to the cross-entropy method, however. The answer “I don’t know, but it’s a good one!” is the working answer of 24 pairs of a method and a setting, more than any other working answer, and always on the Cinderella task of a Pythia model. Besides the cross-entropy method, the searches that grow MLP lesions give it in 4 settings, the learned masks in 1, and the layer search in 1. CLAMP never gives this answer, and at the median its Broca answers are more than six times as long as those of HEAD-CEM.

The learned masks change the likelihood more than the answer. On the transcripts of held-out speakers, at the last stage of the path, the masked model makes the target group’s transcripts more likely and the other groups’ transcripts less likely than the intact model in 90 of the 126 mask experiments, which cover 42 settings and three spaces. The per-token log-likelihood ratio reaches up to $+0.31$ on the target group and down to $-0.31$ on the other groups. The answer, however, reaches the target in only 25 experiments, in 12 of the 42 settings and only in GPT-2 XL and the Pythia models: 21 times for the aphasia target, three times for Broca, and once for Wernicke. The Wernicke success and one of the Broca successes come from hard masks whose answers reach their target only as the most probable class. In the two Llama-2-chat models, no mask raises the target probability above 0.01. The soft and hard versions of a mask often disagree: in 7 experiments only the soft mask works, in 11 only the hard mask, and in 7 both. Only one working mask carries over to the TAB: the soft head mask of Pythia-1B for the aphasia target on sandwich, with 69 heads at partial amounts, reaches a mean $P(\mathrm{aphasia})$ of 0.54 on the connected-text prompts, against 0.12 for the intact model, and 0.77 on the Cinderella answer. Making the patients’ transcripts more likely thus rarely makes the model speak like the patients.

PARAM rarely reaches a target by either criterion. By its own criterion, PARAM assigns the final cluster its aphasia type in 15 of the 48 clusters of the 12 model and task pairs that we ran: 1 anomic, 6 Broca, 2 conduction, and 6 Wernicke clusters, in 8 of the pairs and in neither task of Llama-2-7B-chat. The classifier sees these answers differently. Of the 47 cluster answers long enough for the classifier to score, it calls 35 model scrambles, 8 control speech, and 4 conduction or anomic aphasia. Two of these four aphasic answers do not count, because the lesioned model denies knowing the story of Cinderella. The other two are the answers of the GPT-2 XL Cinderella anomic cluster, at $P(\mathrm{aphasia}) = 0.525$, and of the Pythia-1.4B Cinderella Broca cluster, at $P(\mathrm{aphasia}) = 0.996$ with conduction or anomic aphasia as its most probable class, so PARAM works for the aphasia target in 2 settings and for the Broca and Wernicke targets in none. If every growth step of the matching cluster counted, PARAM would work for the aphasia target in 9 settings, for the Broca target in 1, and for the Wernicke target in none. Its single lesions rarely work either: of the 2,272 single lesions over the 12 pairs, 40 reach the aphasia target, 3 the Broca target, and none the Wernicke target, all in the Pythia models. Matching patient speech by text similarity thus mostly selects lesions that turn the answer into model scrambles.

The classifier makes PARAM’s lesions work. Searched with the classifier as PARAM-C, the lesions of PARAM work in 25 of the 42 settings: 11 for the aphasia target, 10 for Broca, and 4 for Wernicke. They work in 4 of the 12 settings of the Llama-2-chat models. On the 36 settings in which we ran PARAM, PARAM-C works in 23 and PARAM in 2. After pruning, 15 of the 25 working sets hold MLP lesions only, 7 hold both kinds, and 3 hold attention lesions only. PARAM-C works in two settings in which CLAMP fails, both for the Broca target, GPT-2 XL on Cinderella and Llama-2-7B-chat on sandwich, and it fails in 15 in which CLAMP works, 9 of them for the Wernicke target. Three of its 25 working answers are degenerate (Table 14), and three others are “I don’t know, but it’s a good one!”.

Examples of Lesioned Answers

This appendix shows what lesioned models say. Table 15 gives answers under CLAMP lesions, and Table 16 gives answers under lesions found by criteria that cannot recognize a machine failure. We chose the CLAMP examples to cover the three targets, the four model families, and both tasks, so they illustrate what the lesions can produce and are not a random sample.

Model	Task	Target	Heads	$P$	Intact answer	Lesioned answer
GPT-2 XL	Cinderella	Aphasia	9	0.891	… Cinderella was born in a small village called "Horse-head" (or Horse-Head) in the Kingdom of Khazaria…	I don’t know anything about her, but she’s a pretty girl and she has to go to the ball.
Pythia-1B	sandwich	Aphasia	15	0.691	I’d use the same recipe as for my bread, but add some of your favorite jam or preserves to it instead of mayonnaise…	You’d put the bread in first, then add some of that to your sandwich. Then you could just eat it all!
Qwen3.5-2B	sandwich	Broca	18	0.738	Making the perfect Peanut Butter & Jelly (PB&J) is less about following strict rules…	Tell, tell the way of? / "Tell to know."
Pythia-6.9B	sandwich	Broca	13	0.665	I’d put the bread on top of the PB, then spread it with some jam (or honey)…	You have to ask for it!
Llama-2-13B-chat†	Cinderella	Broca	101	0.886	Sure! Here’s a brief retelling of the classic tale of Cinderella: / Once upon a time, there was a beautiful and kind girl named Cinderella…	to tell a tale of Ct.. to tell me.
Llama-2-13B-chat	Cinderella	Wernicke	109	0.786	as in the row above	tours and tours, but I can’dough, but i can be a new to the whole and the previous one is not.
Pythia-1B	Cinderella	Wernicke	13	0.759	The first part is a fairy tale, and it’s about how she got her shoes on in order to go out into the world with Prince Charming…	I am a new one, and it is to be able that we are not in any way for your own. We have been on our own with each other than what they do…

Model	Task	Lesion	$P(\mathrm{scrambles})$	Lesioned answer
Llama-2-7B-chat	Cinderella	Heads L0H25, L1H8, and L1H29	0.9988	I’m not sure if you could provide more information about your company, such as its website or any other relevant details, please let me know. / I understand that is a polite
Pythia-1.4B	Cinderella	PARAM cluster for Wernicke’s aphasia, 23 weight lesions	0.9999	The first time I was a Cinderella, it is in Cindere-lta, by Cindere-tant, T he first t e r s A c n d i l -n t h E D L O N S U P R E M B L Y T H E F O R T W E T H E F O R T W E…
Pythia-6.9B	sandwich	PARAM cluster for Broca’s aphasia, 32 weight lesions	0.9981	Click to,vumo-click here for more information on / Click Here To Buy Now!

Lesions found by CLAMP. The answers that reach Broca’s aphasia are short, from 23 to 41 characters, and two of the three are fragments. One of them is the Llama-2-13B-chat story of § 5.1, “to tell a tale of Ct.. to tell me.” Asked first whether it knows the story, the same lesioned model answers “Yes, I am familiar with the story of Cinderella”, which stops at the length limit for that answer. The third Broca answer, “You have to ask for it!”, is a complete sentence, which suggests that the classifier also scores short answers as Broca-like, as § 5.1 notes for HEAD-CEM. The answers that reach Wernicke’s aphasia are fluent but empty, and one contains a form that resembles a paraphasia, “can’dough”. The answers that reach the aphasia target stay on the task but are vague, as in “add some of that to your sandwich”.

Lesions found by other criteria. The three heads of Llama-2-7B-chat in Table 16 come from the preliminary search of § 3. That search used an earlier classifier with only an aphasic and a control class, and by this classifier, knocking out the three heads made the model’s greedy answer aphasic. The five-way classifier’s training data hold that greedy answer as a model scramble, but not the different answer in Table 16, which uses the repetition penalty of § 3. Each PARAM cluster reaches its type by PARAM’s own criterion, BLEU-1 and BERTScore against patient transcripts, although the same criterion assigns even the intact Pythia-1.4B story to Broca’s aphasia and the intact Pythia-6.9B sandwich answer to anomic aphasia. None of these criteria can recognize a machine failure, and all three lesions produce one: an off-task request for details about “your company”, a run of single letters, and text like a web advertisement, which the five-way classifier calls model scrambles with $P(\text{scrambles})$ of at least 0.998.

Single answers. Each answer in both tables is a single greedy answer to one prompt, and the targeted type does not carry over to other prompts, although the lesions also damage the answers to them (Appendix § 12). Under the seven lesions of Table 15, no answer to the other task or to the connected-text prompts of the Text Aphasia Battery reaches the target, and the highest target probability among these answers is 0.053.

Lesions beyond the Narrative

Each CLAMP lesion is found on the model’s answer to one narrative task, and § 5.1 reports that the lesions also damage the model’s answers to other prompts. We test each of the 51 lesions that reach their target in four ways: on the other narrative task, on the short-answer items and on the connected-text prompts of the Text Aphasia Battery (TAB) ((Roll et al., 2026b)), and on MMLU ((Hendrycks et al., 2021)). This appendix describes how we score each test, lists the scores for every lesion (Table 17 and Table 18), and summarizes them; Table 3 gives the medians by model family. We name a lesion by its model, the task that it was found on, and its target, for example the GPT-2 XL sandwich aphasia lesion, and, as in § 5.1, we call GPT-2 XL, the two Llama-2-chat models, and the four Pythia models the older models.

Model	Task	Target	Heads	Same	Other	Short Answer	Features	Impaired	Drop
GPT-2 XL	Cinderella	Aphasia	9	0.891	0.009	10/15	0.8 → 2.6	0.4 → 0.6	0.08
GPT-2 XL	Cinderella	Wernicke	244	0.647	0.000	4/15	0.8 → 4.4	0.4 → 0.8	0.21
GPT-2 XL	Sandwich	Aphasia	31	0.862	0.859	9/15	0.8 → 0.6	0.4 → 0.2	0.27
GPT-2 XL	Sandwich	Broca	146	0.884	0.015	4/15	0.8 → 3.4	0.4 → 0.8	0.36
GPT-2 XL	Sandwich	Wernicke	188	0.717	0.001	8/15	0.8 → 2.0	0.4 → 0.4	0.24
Pythia-1B	Cinderella	Aphasia	8	0.568	0.008	7/15	0.4 → 1.6	0.2 → 0.8	−0.03
Pythia-1B	Cinderella	Broca	27	0.568	0.007	0/15	0.4 → 4.8	0.2 → 1.0	1.75
Pythia-1B	Cinderella	Wernicke	13	0.759	0.001	3/15	0.4 → 2.8	0.2 → 1.0	0.11
Pythia-1B	Sandwich	Aphasia	15	0.691	0.001	4/15	0.4 → 2.2	0.2 → 0.8	0.04
Pythia-1B	Sandwich	Broca	35	0.557	0.002	0/15	0.4 → 7.2	0.2 → 1.0	3.60
Pythia-1B	Sandwich	Wernicke	9	0.703	0.213	3/15	0.4 → 1.8	0.2 → 0.8	0.15
Pythia-1.4B	Cinderella	Aphasia	19	0.972	0.123	0/15	0.4 → 2.8	0.0 → 1.0	0.32
Pythia-1.4B	Cinderella	Wernicke	12	0.751	0.000	7/15	0.4 → 1.0	0.0 → 0.2	0.03
Pythia-1.4B	Sandwich	Aphasia	13	0.999	0.015	8/15	0.4 → 0.4	0.0 → 0.2	0.01
Pythia-1.4B	Sandwich	Broca	316	0.488	0.001	0/15	0.4 → 5.8	0.0 → 1.0	4.92
Pythia-2.8B	Cinderella	Aphasia	1	0.537	0.006	7/15	0.0 → 0.2	0.0 → 0.0	0.00
Pythia-2.8B	Cinderella	Broca	66	0.801	0.004	4/15	0.0 → 5.8	0.0 → 1.0	1.49
Pythia-2.8B	Cinderella	Wernicke	63	0.560	0.065	2/15	0.0 → 7.0	0.0 → 1.0	1.34
Pythia-2.8B	Sandwich	Aphasia	1	0.706	0.011	7/15	0.0 → 0.4	0.0 → 0.2	0.00
Pythia-2.8B	Sandwich	Broca	128	0.742	0.000	3/15	0.0 → 5.2	0.0 → 1.0	0.95
Pythia-2.8B	Sandwich	Wernicke	18	0.504	0.005	11/15	0.0 → 2.2	0.0 → 0.6	0.11
Pythia-6.9B	Cinderella	Aphasia	1	0.957	0.009	7/15	0.0 → 0.4	0.0 → 0.2	−0.00
Pythia-6.9B	Cinderella	Broca	578	0.993	0.112	0/15	0.0 → 5.0	0.0 → 1.0	4.74
Pythia-6.9B	Cinderella	Wernicke	1	0.505	0.000	8/15	0.0 → 0.0	0.0 → 0.0	0.00
Pythia-6.9B	Sandwich	Aphasia	4	0.723	0.127	4/15	0.0 → 0.0	0.0 → 0.0	−0.00
Pythia-6.9B	Sandwich	Broca	13	0.665	0.000	7/15	0.0 → 1.0	0.0 → 0.4	0.11
Pythia-6.9B	Sandwich	Wernicke	14	0.720	0.000	7/15	0.0 → 1.0	0.0 → 0.4	0.04

Model	Task	Target	Heads	Same	Other	Short Answer	Features	Impaired	Drop
Llama-2-7B-chat	Cinderella	Aphasia	23	0.606	0.001	2/15	0.0 $\to$ 0.6	0.0 $\to$ 0.0	2.91
	Cinderella	Broca	42	0.678	0.000	7/15	0.0 $\to$ 3.6	0.0 $\to$ 1.0	1.08
	Cinderella	Wernicke	276	0.682	0.019	0/15	0.0 $\to$ 3.0	0.0 $\to$ 0.6	3.59
	Sandwich	Aphasia	60	0.931	0.000	3/15	0.0 $\to$ 2.2	0.0 $\to$ 0.8	1.14
	Sandwich	Wernicke	160	0.736	0.000	1/15	0.0 $\to$ 4.4	0.0 $\to$ 1.0	2.24
Llama-2-13B-chat	Cinderella	Aphasia	77	0.679	0.000	6/15	0.0 $\to$ 1.6	0.0 $\to$ 0.6	1.59
	Cinderella	Broca	101	0.886	0.001	2/15	0.0 $\to$ 0.6	0.0 $\to$ 0.2	2.43
	Cinderella	Wernicke	109	0.786	0.000	4/15	0.0 $\to$ 4.2	0.0 $\to$ 1.0	2.42
	Sandwich	Aphasia	90	0.747	0.007	5/15	0.0 $\to$ 1.8	0.0 $\to$ 0.2	1.39
	Sandwich	Broca	87	0.636	0.001	3/15	0.0 $\to$ 0.6	0.0 $\to$ 0.2	1.02
	Sandwich	Wernicke	67	0.716	0.001	4/15	0.0 $\to$ 1.0	0.0 $\to$ 0.2	0.48
Qwen3.5-0.8B	Cinderella	Aphasia	28	0.524	0.007	3/15	0.4 $\to$ 4.6	0.2 $\to$ 1.0	0.41
	Cinderella	Wernicke	28	0.582	0.042	3/15	0.4 $\to$ 6.4	0.2 $\to$ 1.0	0.83
	Sandwich	Aphasia	5	0.948	–1	0/15	0.4 $\to$ 6.2	0.2 $\to$ 1.0	2.72
Qwen3.5-2B	Cinderella	Aphasia	43	0.838	0.079	4/15	0.4 $\to$ 7.2	0.2 $\to$ 1.0	1.00
	Cinderella	Broca	55	0.851	0.001	1/15	0.4 $\to$ 4.0	0.2 $\to$ 1.0	2.66
	Sandwich	Aphasia	72	0.621	0.002	5/15	0.4 $\to$ 5.6	0.2 $\to$ 1.0	2.31
	Sandwich	Broca	18	0.738	0.053	0/15	0.4 $\to$ 1.6	0.2 $\to$ 0.4	5.30
	Sandwich	Wernicke	63	0.463	0.002	0/15	0.4 $\to$ 9.2	0.2 $\to$ 1.0	5.66
Qwen3.5-4B	Cinderella	Aphasia	208	0.508	0.007	8/15	0.0 $\to$ 0.8	0.0 $\to$ 0.4	0.99
	Cinderella	Broca	81	0.548	0.001	6/15	0.0 $\to$ 1.4	0.0 $\to$ 0.6	1.39
	Cinderella	Wernicke	106	0.572	0.000	11/15	0.0 $\to$ 3.6	0.0 $\to$ 1.0	0.68
	Sandwich	Aphasia	88	0.749	0.002	9/15	0.0 $\to$ 5.8	0.0 $\to$ 1.0	0.77
	Sandwich	Broca	372	0.959	0.000	0/15	0.0 $\to$ 5.0	0.0 $\to$ 1.0	7.78
	Sandwich	Wernicke	228	0.423	0.101	6/15	0.0 $\to$ 5.4	0.0 $\to$ 1.0	1.23
Qwen3.5-9B	Cinderella	Aphasia	73	0.587	0.206	8/15	0.0 $\to$ 4.4	0.0 $\to$ 1.0	0.64
	Sandwich	Aphasia	67	0.930	0.001	11/15	0.0 $\to$ 2.0	0.0 $\to$ 0.6	0.35

The other narrative task tests a lesion on the task that its search never saw. For a lesion found on the Cinderella task, the other task is the sandwich task, and for a lesion found on the sandwich task, it is the Cinderella task. The lesioned model answers the other task’s prompt with the decoding of § 3, and the classifier scores the answer for the lesion’s target. The rules of Appendix § 8 decide whether the answer counts, and 53 of the 54 answers do. The remaining answer does not count because, with its sandwich aphasia lesion, Qwen3.5-0.8B denies knowing the story of Cinderella.

Rules score the TAB’s short-answer items. We use 20 items of the TAB, five in each of four subtests: connected text, word comprehension, sentence comprehension, and repetition (Table 7). The 15 items of the last three subtests ask for short answers: word comprehension asks the model to pick one of six listed words, sentence comprehension asks a yes-or-no question, and repetition asks the model to repeat a word or a sentence exactly. The model answers these items greedily, without the repetition penalty, and rules score its answers (Appendix § 8). A word-comprehension answer is correct if the first option that it names is the target, a sentence-comprehension answer if its first “yes” or “no” matches the target, and a repetition answer if it equals the target after normalization. Table 17 and Table 18 give the number of the 15 items that the lesioned model answers correctly. The items lost by a lesion are the number of items that the intact model answers correctly minus the number that the lesioned model answers correctly, so a negative number means items gained. With only 15 items, a single item should not decide, so we count a lesion as costing items when it loses two or more.

The classifier and an LLM judge score the TAB’s connected text

Four of the five connected-text prompts ask about the speaker’s own life, for example “Tell me about the best trip you ever took”, and one asks for the steps to make a simple meal. The model answers them with the decoding of the narratives, greedily with the repetition penalty, but writes at most 120 new tokens. The classifier scores each answer that is long enough to count, which holds for all five answers of 48 of the 54 lesions and for two to four answers of the other six, and an answer reaches the lesion’s target by the rule above. An LLM judge, DeepSeek-V4.1-Flash ([15]) at temperature 0, marks each of the five answers for the TAB’s 19 aphasia features, one of which is an overall communication impairment. We report the mean number of features per answer and the share of the five answers that show an overall communication impairment. The judge is a reasoning model and is not deterministic even at temperature 0, so we keep its first reply to each request. Its prompt follows the TAB’s protocol, but we wrote the feature definitions, the examples, and the two synthetic annotated transcripts ourselves (Appendix § 17), and no human rated these answers to check the judge.

MMLU tests knowledge far from the narratives. We use two subjects of MMLU: global facts, with 100 test questions, and high school world history, with 237. Each prompt is the standard five-shot prompt, except that the world-history prompts of GPT-2 XL keep only one to three solved questions to fit its context (Appendix § 8). The model reads the prompt once and generates nothing. Its prediction is the answer letter with the highest next-token probability, and its accuracy is the share of questions that it predicts correctly, which is 0.25 by chance. We compute the log-probability of the correct letter over the whole vocabulary, so it also falls when the model moves probability away from the four letters. The MMLU drop of a lesion is the intact model’s mean log-probability of the correct letter minus the lesioned model’s, in nats, averaged over the two subjects with equal weight. The judge finds more aphasia features after a lesion. The judge finds more of the TAB’s aphasia features in the lesioned model’s connected text than in the intact model’s for 50 of the 54 lesions. Three lesions tie with the intact model, and the GPT-2 XL sandwich aphasia lesion falls below it, at 0.6 against 0.8 features per answer. Over the 54 lesions, the median is 2.7 of 19 features per answer, against 0.0 for the intact models. The lesions of the older models reach a median of 2.1 features, against 0.0 for their intact models, and those of the Qwen3.5 models a median of 4.8, against 0.2. The judge sees an overall communication impairment in a median of four of a lesioned model’s five answers, three in the older models and all five in the Qwen3.5 models, but in at most two of an intact model’s five answers. It sees one in all five answers for 24 of the 54 lesions. By the judge, a lesion found on one narrative thus impairs the model’s language on prompts that the search never saw.

The lesions cost comprehension and repetition items. The lesioned models answer a median of 4 of the 15 short-answer items correctly, against 8.5 for the intact models. Of the 54 lesions, 41 lose two or more items: 25 of the 38 lesions of the older models and all 16 lesions of the Qwen3.5 models. The median lesion loses 5 items, 4 in the older models and 8.5 in the Qwen3.5 models. The Qwen3.5 models have more to lose: intact, they answer 13 or 14 of the 15 items correctly, against 6 to 10 for the older models. Repetition shows the difference most clearly. The intact Qwen3.5 models repeat all five items exactly, and their lesions lose a mean of 4.0 of them. The intact older models repeat at most four items exactly, and their lesions lose a mean of 0.50. The intact Llama-2 models repeat none exactly, although the answers of Llama-2-7B-chat contain the target for three of the five items and those of Llama-2-13B-chat for all five. On word and sentence comprehension, the lesions of the older models lose a mean of 1.47 and 1.37 items, and those of the Qwen3.5 models 2.06 and 2.75. Six lesions, all from GPT-2 XL and the Pythia models, gain one to four items. The lesions thus impair comprehension and repetition as well as connected text, and they cost the most where the intact model has the most to lose.

The lesions lower the log-probability of correct MMLU answers. The median MMLU drop is 0.89 nats over the 54 lesions, 0.34 nats in the older models, and 1.11 nats in the Qwen3.5 models. The drop exceeds 0.5 nats for 31 lesions, 17 of the 38 lesions of the older models and 14 of the 16 lesions of the Qwen3.5 models, and it ranges from −0.03 to 7.78 nats. Where the intact model answers world-history questions well above chance, its accuracy falls with the log-probability. Every Llama-2 lesion lowers the world-history accuracy from 0.675 or 0.717 to between 0.203 and 0.270, and the Qwen3.5 lesions lower it from between

0.654 and 0.907 to between 0.207 and 0.532. Even the Qwen3.5-9B sandwich aphasia lesion, whose drop of 0.35 nats is the smallest of the Qwen3.5 lesions, lowers it from 0.907 to 0.532. The intact GPT-2 XL and Pythia models answer near chance, with accuracies of 0.18 to 0.36 on global facts and 0.236 to 0.300 on world history, so for them the log-probability of the correct answer shows the damage that the accuracy cannot. Most lesions found on one narrative thus also lower the probability that the model gives the correct answers to questions that have nothing to do with the narrative.

The classifier rarely finds the targeted type beyond the narrative. No lesion raises the mean target probability of its connected-text answers above 0.5: the highest mean is 0.369, for the Pythia-

1B Cinderella aphasia lesion, and the median is 0.009. Answer by answer, 12 of the 259 scored answers reach their target, and 11 of the 12 come from aphasia lesions, for which any type of aphasia counts. The only answer of a Broca or Wernicke lesion that reaches its target is the Pythia-2.8B Cinderella Wernicke lesion’s answer to “Tell me about your first job”, at $P(\text{Wernicke}) = 0.862$. On the other narrative task, 1 of the 54 lesions reaches its target: the 31 heads of the GPT-2 XL sandwich aphasia lesion also make the model’s answer to the Cinderella task aphasic, at 0.859, so this lesion makes both narratives aphasic. Over the 53 lesions whose answer to the other task counts, the median target probability is 0.002. The lesions thus damage the answers to new prompts by the judge’s features, the short-answer items, and MMLU, but the classifier rarely finds the lesion’s target in these answers.

Activation Patching and Shared Heads

We take apart the 54 CLAMP lesions that reach their target (Figure 4) to support three findings of § 5.1. Activation patching shows that the heads of a lesion act together while the model reads the prompt (Table 19 and Table 20), and the same patching on the TAB shows that the damage beyond the narrative arises in the same way. Comparisons between the lesions of one model show that the lesions for different targets share many heads (Table 21), and that the lesions found on the Cinderella and the sandwich tasks share heads but rarely carry their target to the other task (Table 22). A last analysis asks whether the Qwen3.5 lesions favor one of the two kinds of attention in these models. We name a lesion by its model, the task that it was found on, and its target, for example the GPT-2 XL sandwich aphasia lesion.

Model	Task	Target	Heads	Lesion	Restore all	Restored	Left out	Alone	Lesion	Restore all	Best restore
GPT-2 XL	Cinderella	Aphasia	9	0.891	0.001	0.67	1.00	0.00	10/15	9/15	10/15
		Wernicke	244	0.647	0.001	0.75	0.95	0.00	4/15	7/15	5/15
	Sandwich	Aphasia	31	0.862	0.005	0.81	1.00	0.00	9/15	7/15	9/15
		Broca	146	0.884	0.000	0.91	1.00	0.00	4/15	9/15	5/15
		Wernicke	188	0.717	0.000	0.61	0.95	0.00	8/15	9/15	8/15
Pythia-1B	Cinderella	Aphasia	8	0.568	0.048	0.50	1.00	0.00	7/15	8/15	9/15
		Broca	27	0.568	0.001	0.74	0.96	0.00	0/15	4/15	0/15
		Wernicke	13	0.759	0.244	1.00	1.00	0.00	3/15	7/15	4/15
	Sandwich	Aphasia	15	0.691	0.007	0.87	1.00	0.00	4/15	10/15	5/15
		Broca	35	0.557	0.002	0.71	0.94	0.00	0/15	8/15	0/15
		Wernicke	9	0.703	0.000	1.00	0.89	0.00	3/15	11/15	9/15
Pythia-1.4B	Cinderella	Aphasia	19	0.972	0.022	0.74	1.00	0.00	0/15	7/15	3/15
		Wernicke	12	0.751	0.000	0.83	1.00	0.00	7/15	9/15	9/15
	Sandwich	Aphasia	13	0.999	0.037	0.92	1.00	0.00	8/15	8/15	8/15
		Broca	316	0.488	0.000	0.54	0.92	0.00	0/15	5/15	0/15
Pythia-2.8B	Cinderella	Aphasia	1	0.537	0.001	1.00	1.00	1.00	7/15	7/15	7/15
		Broca	66	0.801	0.001	0.91	1.00	0.00	4/15	7/15	4/15
		Wernicke	63	0.560	0.051	0.95	1.00	0.00	2/15	6/15	5/15
	Sandwich	Aphasia	1	0.706	0.006	1.00	1.00	1.00	7/15	7/15	7/15
		Broca	128	0.742	0.001	0.82	0.99	0.00	3/15	6/15	4/15
		Wernicke	18	0.504	0.001	0.83	0.94	0.00	11/15	6/15	12/15
Pythia-6.9B	Cinderella	Aphasia	1	0.957	0.957	0.00	1.00	1.00	7/15	6/15	6/15
		Broca	578	0.993	0.009	0.74	0.97	0.00	0/15	3/15	0/15
		Wernicke	1	0.505	0.016	1.00	1.00	1.00	8/15	5/15	5/15
	Sandwich	Aphasia	4	0.723	0.007	0.50	1.00	0.00	4/15	5/15	7/15
		Broca	13	0.665	0.000	0.85	1.00	0.00	7/15	5/15	7/15
		Wernicke	14	0.720	0.001	0.57	1.00	0.00	7/15	6/15	7/15

Model	Task	Target	Heads	Probability		Share of heads			TAB short answer
				Lesion	Restore all	Restored	Left out	Alone	Lesion	Restore all	Best restore
Llama-2-7B-chat	Cinderella	Aphasia	23	0.606	0.012	0.61	1.00	0.00	2/15	7/15	9/15
		Broca	42	0.678	0.001	0.79	1.00	0.00	7/15	7/15	7/15
		Wernicke	276	0.682	0.001	0.88	0.99	0.00	0/15	5/15	1/15
	Sandwich	Aphasia	60	0.931	0.007	0.95	1.00	0.00	3/15	7/15	3/15
		Wernicke	160	0.736	0.000	0.89	1.00	0.00	1/15	8/15	1/15
Llama-2-13B-chat	Cinderella	Aphasia	77	0.679	0.001	0.73	1.00	0.00	6/15	7/15	7/15
		Broca	101	0.886	0.001	0.90	1.00	0.00	2/15	7/15	3/15
		Wernicke	109	0.786	0.001	0.63	0.98	0.00	4/15	8/15	5/15
	Sandwich	Aphasia	90	0.747	0.005	0.88	1.00	0.00	5/15	8/15	5/15
		Broca	87	0.636	0.000	0.90	1.00	0.00	3/15	7/15	3/15
		Wernicke	67	0.716	0.000	0.85	1.00	0.00	4/15	7/15	6/15
Qwen3.5-0.8B	Cinderella	Aphasia	28	0.524	0.004	0.93	1.00	0.00	3/15	12/15	5/15
		Wernicke	28	0.582	0.000	0.96	0.96	0.00	3/15	12/15	5/15
	Sandwich	Aphasia	5	0.948	0.369	0.40	1.00	0.00	0/15	12/15	13/15
Qwen3.5-2B	Cinderella	Aphasia	43	0.838	0.001	0.77	1.00	0.00	4/15	12/15	5/15
		Broca	55	0.851	0.001	0.96	0.98	0.00	1/15	12/15	1/15
	Sandwich	Aphasia	72	0.621	0.008	0.69	1.00	0.00	5/15	13/15	8/15
		Broca	18	0.738	0.356	0.89	1.00	0.00	0/15	8/15	0/15
		Wernicke	63	0.463	0.000	0.78	0.98	0.00	0/15	8/15	1/15
Qwen3.5-4B	Cinderella	Aphasia	208	0.508	0.002	0.48	0.91	0.00	8/15	13/15	10/15
		Broca	81	0.548	0.001	0.98	1.00	0.00	6/15	14/15	8/15
		Wernicke	106	0.572	0.000	0.90	0.99	0.00	11/15	14/15	14/15
	Sandwich	Aphasia	88	0.749	0.005	0.81	1.00	0.00	9/15	14/15	11/15
		Broca	372	0.959	0.031	0.95	1.00	0.00	0/15	8/15	1/15
		Wernicke	228	0.423	0.006	0.64	0.93	0.00	6/15	14/15	6/15
Qwen3.5-9B	Cinderella	Aphasia	73	0.587	0.008	0.75	1.00	0.00	8/15	14/15	12/15
	Sandwich	Aphasia	67	0.930	0.006	0.88	1.00	0.00	11/15	14/15	12/15

Task	Heads			Aphasia and Broca			Aphasia and Wernicke			Broca and Wernicke
	Aph.	Broca	Wern.	Shared	Chance	$p$	Shared	Chance	$p$	Shared	Chance	$p$
GPT-2 XL
Cinderella	9	–	244	–	–	–	0	1.83	1	–	–	–
Sandwich	31	146	188	18	3.77	$<0.001$	12	4.86	0.0014	70	22.87	$<0.001$
Pythia-1B
Cinderella	8	27	13	4	1.69	0.061	2	0.81	0.19	7	2.74	0.0062
Sandwich	15	35	9	7	4.10	0.074	4	1.05	0.011	3	2.46	0.47
Pythia-1.4B
Cinderella	19	–	12	–	–	–	0	0.59	1	–	–	–
Sandwich	13	316	–	13	10.70	0.076	–	–	–	–	–	–
Pythia-2.8B
Cinderella	1	66	63	1	0.06	0.064	0	0.06	1	36	4.06	$<0.001$
Sandwich	1	128	18	1	0.12	0.12	1	0.02	0.018	14	2.25	$<0.001$
Pythia-6.9B
Cinderella	1	578	1	1	0.56	0.56	0	0.00	1	1	0.56	0.56
Sandwich	4	13	14	2	0.05	$<0.001$	2	0.05	0.001	7	0.18	$<0.001$
Llama-2-7B-chat
Cinderella	23	42	276	8	0.94	$<0.001$	14	6.20	$<0.001$	36	11.32	$<0.001$
Sandwich	60	–	160	–	–	–	37	9.38	$<0.001$	–	–	–
Llama-2-13B-chat
Cinderella	77	101	109	45	4.86	$<0.001$	30	5.25	$<0.001$	37	6.88	$<0.001$
Sandwich	90	87	67	17	4.89	$<0.001$	22	3.77	$<0.001$	30	3.64	$<0.001$
Qwen3.5-0.8B
Cinderella	28	–	28	–	–	–	12	2.33	$<0.001$	–	–	–
Qwen3.5-2B
Cinderella	43	55	–	4	7.04	0.95	–	–	–	–	–	–
Sandwich	72	18	63	9	3.86	0.0055	28	13.50	$<0.001$	18	3.38	$<0.001$
Qwen3.5-4B
Cinderella	208	81	106	23	18.80	0.15	23	24.61	0.69	38	9.58	$<0.001$
Sandwich	88	372	228	83	36.54	$<0.001$	62	22.39	$<0.001$	220	94.66	$<0.001$
All pairs				236	98.00	7 of 15	249	96.70	9 of 17	517	164.59	11 of 13
Median share					50%			39%			54%

Target	Cinderella	Sandwich	Shared	Chance	Ratio	$p$	Cinderella	Sandwich	Cinderella	Sandwich
GPT-2 XL
Aphasia	9	31	1	0.23	4.30	0.21	0.000	0.001	0.015	0.007
Wernicke	244	188	92	38.23	2.41	$1.9 \times10^{-22}$	0.000	0.000	0.000	0.000
Llama-2-13B-chat
Aphasia	77	90	24	4.33	5.54	$2.5 \times10^{-13}$	0.001	0.003	0.018	0.001
Broca	101	87	28	5.49	5.10	$1.6 \times10^{-14}$	0.001	0.001	0.001	0.088
Wernicke	109	67	24	4.56	5.26	$7.4 \times10^{-13}$	0.004	0.000	0.000	0.000
Pythia-1B
Aphasia	8	15	0	0.94	0.00	1	0.965	0.953	–	–
Broca	27	35	11	7.38	1.49	0.068	0.001	0.000	0.001	0.000
Wernicke	13	9	0	0.91	0.00	1	0.008	0.000	–	–
Qwen3.5-9B
Aphasia	73	67	14	5.46	2.56	0.0005	0.267	0.002	0.002	0.005

Four patching conditions test where the heads act and which of them are needed. Activation patching copies activations from one run of a model into another run on the same input, to find out which components carry an effect ((Meng et al., 2022; Zhang and Nanda, 2024; Heimersheim and Nanda, 2024)). Here the clean run uses the intact model, and the lesioned run knocks out every head of a lesion at every position of the prompt and of the answer, as in § 4.1. For each head of a lesion, we generate and score the model’s answer under three conditions. Restoring the head writes its output from the clean run back at every position of the prompt, while the other heads of the lesion stay knocked out; the restored head itself stays knocked out at the positions of the answer, which have no counterpart in the clean run. Leaving out the head knocks out only the other heads of the lesion, so that the head runs normally at every position. Knocking out the head alone knocks out only this head, so that the other heads of the lesion run normally. The first two conditions differ in what the head computes: a restored head’s output on the prompt is copied from the clean run, whatever the other knockouts do to its input, whereas a head that is left out computes its output from a residual stream that the other knockouts have changed. A fourth condition acts on the whole lesion: restoring all writes back the clean outputs of every head of the lesion at every position of the prompt, so that the lesion is knocked out only while the model writes its answer. In the terms of Heimersheim and Nanda (2024), restoring is denoising: it writes clean activations into the lesioned run and asks whether they remove the effect of the lesion. An answer reaches the target when it counts by the rules of Sections 3 and 4.1 and the classifier either gives the target a probability above 0.5 or ranks the target as its most probable class, where any of the three aphasia classes counts for the aphasia target. A condition undoes the lesion when its answer no longer reaches the target. The patching runs reproduce the stored intact and lesioned answers of all 54 lesions.

Restoring the heads’ outputs on the prompt undoes 53 of the 54 lesions. Restoring all heads undoes 53 of the 54 lesions and leaves a target probability of at most 0.369 in them (Table 19 and Table 20). These 53 lesions need their heads knocked out while the model reads the prompt, and knocking the heads out only while the model writes its answer does not reach the target. The exception is the Pythia-6.9B Cinderella aphasia lesion, which consists of a single head, L1H26, that is, head 26 of layer 1, both counted from 0. With the clean output of this head written back on the prompt, the target probability stays at 0.957, so knocking the head out only while the model writes its answer already reaches the target. Restoring a single head usually undoes a lesion as well. The median share of a lesion’s heads whose restore undoes the lesion is 0.83, both over the 54 lesions and over the 50 lesions of two or more heads, and at least one single head restore undoes every lesion except the Pythia-6.9B

Cinderella aphasia lesion. Pooled over the 54 lesions, restoring undoes the lesion for 3,579 of the 4,600 heads. Two other classifier-guided searches of Appendix § 10, whose sets above 0.5 we patched in the same way, give similar medians over all lesioned components of a set: 0.87 over the 20 sets of the layer search, which can add blocks of MLP neurons, and 0.96 over the 37 sets of the search that sweeps heads on top of grown MLP lesions. The heads of a lesion thus act while the model reads the prompt, and most of them matter there: giving back the clean output of a single head usually undoes the lesion.

A single restore mostly breaks the answer instead of repairing it. Undoing a lesion means only that the answer no longer reaches the target, not that the model answers as it does when intact. Of the 3,579 single restores that undo a lesion, 3,541 give an answer long enough to score. The classifier labels 2,819 of these answers model scrambles, or 80%, and only 450 control speech, or 13%; the other 272 have an aphasia class as their most probable class. The intact answer is not always control speech: the classifier labels it control speech for 41 of the 54 lesions and model scrambles for the other 13, all of them on the Cinderella task. Over the 41 lesions whose intact answer is control speech, the result is the same: of the 2,767 scored answers of single restores that undo these lesions, the classifier labels 2,203 model scrambles, or 80%, and 331 control speech, or 12%. Restoring all heads repairs the answer more often: it undoes 40 of these 41 lesions, and the classifier labels 22 of the 40 answers control speech and 13 model scrambles. The aphasia-like answer thus needs the whole lesion, and giving one head back its clean output mostly breaks the answer in another way.

Almost every head is needed, largely by construction. Leaving out one head undoes the lesion for 4,484 of the 4,600 heads, and in 37 of the 54 lesions, leaving out any one head undoes the lesion. No head of the 50 lesions of two or more heads reaches the target when knocked out alone. Both results follow mostly from how CLAMP builds the lesions, so they check the lesions rather than reveal how the heads work. Growth ranks the heads by the target probability that each gives when knocked out alone and stops at the first prefix whose probability exceeds 0.5, so a lesion of two or more heads arises only when no head exceeds 0.5 alone. Pruning removes heads for as long as the target probability

get probability stays above 0.5, so a lesion whose pruning finished sits at the edge where removing any one more head brings the probability to 0.5 or below, and leaving out a head repeats this last test of pruning. By the search’s own rule, a probability above 0.5, every head is therefore needed in all 47 lesions whose pruning finished, and in 48 of the 51 lesions that exceed 0.5. Reaching the target, as defined above, also includes an answer at 0.5 or below whose most probable class is still the target, so by this definition every head is needed in only 37 of the 47 pruned lesions. In the other 10, leaving out a head lowers the target probability to 0.5 or below while the target stays the most probable class, for 1 to 9 heads per lesion and 20 heads in all. Beyond this difference between the two rules, leaving out heads tests more than pruning did only in the seven lesions whose pruning did not finish: the four that stopped at the budget of 15,000 answers and the three that reach their target only as the most probable class, which the search never pruned. In these seven lesions, 91 to 99% of the heads are needed: 233 of 244 in the GPT-2 XL Cinderella Wernicke lesion, 272 of 276 in the Llama-2-7B-chat Cinderella Wernicke lesion, 558 of 578 in the Pythia-6.9B Cinderella Broca lesion, 190 of 208 in the Qwen3.5-4B Cinderella aphasia lesion, 291 of 316 in the Pythia-1.4B sandwich Broca lesion, 62 of 63 in the Qwen3.5-2B sandwich Wernicke lesion, and 211 of 228 in the Qwen3.5-4B sandwich Wernicke lesion. By the search’s rule, all 276 heads of the Llama-2-7B-chat lesion are needed, so a further round of pruning would have removed none of them. The restore conditions are not settled by construction, because a restored head differs from a head that is left out: it stays knocked out while the model writes its answer, and its output on the prompt is the clean one. Because pruning leaves most lesions at the edge of their target, these yes-or-no outcomes show where the heads act, but not how much each head contributes, and a change that breaks the answer counts as undoing the lesion as much as one that repairs it. Per-head patching also cannot show which head feeds which; path patching, which restores only the connection from one component to another ((Wang et al., 2023a)), would trace these connections.

Restoring all heads also repairs most of the damage on the TAB. We repeat the lesion, restoring all, and single restores on the 20 items of the TAB (Appendix § 12): 15 short-answer items for word comprehension, sentence comprehension, and repetition, which a rule scores, and five connected-text prompts. Single restores cover every head of the 7 lesions of at most eight heads and, in the other 47 lesions, the eight heads whose removal from the lesion lowers the target probability on the narrative the most. The intact and lesioned TAB answers of all 54 lesions match those of Appendix § 12 word for word. Table 19 and Table 20 give the number of the 15 short-answer items that the model answers correctly in each condition. With the lesion in place, 41 of the 54 lesions lose two or more of these items, counted from the intact model’s number of correct items. Restoring all heads gives back at least half of the lost items in 39 of these 41 lesions and all of them in 16, whereas the best single restore gives back at least half in only 10 and all of them in 4. Over all 54 lesions, the model answers a median of 4 of the 15 items correctly with the lesion in place, 8 with all heads restored, and 5.5 with the best single restore, against 8.5 for the intact models. The median number of items lost is 5, 1, and 3 in these three conditions. On the connected-text prompts, the lesions rarely reach their target, so a restore has little to undo there: the classifier finds the target in 12 of the 259 answers that it scores with the lesion in place, in 8 of 269 with all heads restored, and in 2 of the intact model’s 270. We did not measure MMLU under patching. Unlike the target probability, the number of correct items shows whether the answers recover, and by this measure restoring all heads repairs most of the damage, whereas restoring one of the tested heads rarely does. The damage beyond the narrative thus has the same source as the narrative’s target: the heads acting while the model reads the prompt.

The lesions for different targets share many heads. For each model and task, CLAMP runs one search per target, which ranks the heads by that target’s probability, grows the lesion in this order, and prunes it against the same target. If two lesions $L_1$ and $L_2$ were drawn at random from a model’s $N$ heads, they would share $|L_1| |L_2|/N$ heads on average, and a one-sided hypergeometric test gives the probability of sharing at least the observed number of heads by chance (Table 21). The Broca and Wernicke lesions give the cleanest test, because the two targets are separate classes of the classifier. Over the 13 pairs of Broca and Wernicke lesions of one model and task that both reach their target, the two lesions share a median of 54% of the heads

of the smaller lesion and 517 heads in all, three times the 165 expected by chance, and the overlap exceeds chance at $p < 0.01$ in 11 of the 13 pairs. The two exceptions are the Pythia-1B sandwich pair, whose lesions share 3 heads against 2.5 by chance, and the Pythia-6.9B Cinderella pair, whose one-head Wernicke lesion lies inside the 578-head Broca lesion. The lesions for the aphasia target, whose probability includes those of the Broca and Wernicke classes, overlap with both: they share a median of 50% of the smaller lesion with the Broca lesion, 236 heads against 98 by chance over 15 pairs, and 39% with the Wernicke lesion, 249 heads against 97 over 17 pairs. Nothing in the design makes the searches for different targets share heads, but they rank the same single-head answers: a head whose knockout makes the answer more Broca-like usually makes it more Wernicke-like too, at a median Spearman correlation across heads of 0.93 over the 13 pairs, from 0.49 to 0.99, so the two growths follow similar rankings. Ties add to the overlap: heads whose knockout alone leaves the target probability exactly at the intact value tie with one another in that target’s ranking and keep their layer order (Appendix § 10), so when the growths for two targets reach such heads, they add them in the same order. Heads tied in both rankings make up 134 of the 517 heads that the Broca and Wernicke lesions share, 80 of them in the Llama-2-chat models, and among the other heads of the models, the two lesions still share 383 heads against 156 expected by chance, with $p < 0.01$ in 9 of the 13 pairs. Broca-like and Wernicke-like answers thus come from lesions that share many heads, so the circuits for different types of aphasia are entangled rather than separate.

The lesions found on the two tasks share heads, but their targets rarely carry over. For one model of each family, GPT-2 XL, Llama-2-13B-chat, Pythia-1B, and Qwen3.5-9B, we compare the lesion found on the Cinderella task with the lesion found on the sandwich task for the same target (Table 22). Of the 12 pairs, 9 can be compared; in the other three, the lesion of at least one task does not reach its target. With the chance level and the test of the previous paragraph, the lesions share heads in 7 of the 9 pairs, and far more than chance in five, all at $p < 0.01$: 5.1 to 5.5 times chance in the three pairs of Llama-2-13B-chat, 2.4 times in the GPT-2 XL Wernicke pair, and 2.6 times in the Qwen3.5-9B aphasia pair. In Llama-2-13B-chat, however,

55 of the 76 shared heads are tied in the rankings of both tasks, and without the heads tied on both tasks, the overlap stays above chance at $p < 0.01$ in four of the five pairs, all but the Llama-2-13B-chat aphasia pair. Yet the shared heads do not carry the lesions. Knocked out alone, they reach the target on neither task in any of the 7 pairs with shared heads, and in these pairs neither do the heads that belong to only one of the two lesions. Knocking out the union of the two lesions reaches the target on both tasks in only one pair, the Pythia-1B aphasia pair, whose union of 23 heads gives 0.965 on Cinderella and 0.953 on sandwich. Pruning this union for as long as both answers stay above 0.5 leaves 16 heads, 6 of the Cinderella lesion’s 8 and 10 of the sandwich lesion’s 15. For 6 of the other 8 unions, the classifier labels both answers model scrambles; for the union of the Llama-2-13B-chat Broca pair, it labels the Cinderella answer model scrambles and the sandwich answer control speech, and for the union of the Pythia-1B Wernicke pair, it labels the Cinderella answer Broca’s aphasia and the sandwich answer model scrambles. Of the 18 lesions in the 9 pairs, only one reaches its target on both tasks by itself, the GPT-2 XL sandwich aphasia lesion, at 0.859 on Cinderella and 0.862 on sandwich. The lesions found on the two tasks thus share heads, but the shared heads never produce

the target, and combining the two lesions mostly breaks the answers into model scrambles instead of making both of them aphasic.

The Qwen3.5 lesions lean toward softmax heads.

Every fourth layer of a Qwen3.5 model uses full attention, computed with a softmax, and the other layers use Gated DeltaNet linear attention (Yang et al., 2025); we call the heads of the full-attention layers softmax heads. We count each value head of a Gated DeltaNet layer, which keeps its own recurrent state, as one head, so softmax heads make up 14.3% of the heads of every Qwen3.5 model. Pooled over the 16 Qwen3.5 lesions, they make up 294 of 1,535 heads, or 19.2%, against 219.3 expected at 14.3%; this pooled share only describes the lesions, because the lesions of one model are not independent draws. Softmax heads make up more than 14.3% of the lesion in 15 of the 16 lesions; the exception, the Qwen3.5-0.8B sandwich aphasia lesion, has 5 heads and no softmax head. A one-sided hypergeometric test against random sets of the same size, without correction for the 16 tests, finds softmax heads over-represented at $p < 0.05$ in five lesions: the Qwen3.5-9B Cinderella aphasia lesion, with 25 softmax heads of 73 at $p = 5.3 \times10^{-6}$; the Qwen3.5-2B sandwich Wernicke lesion, with 18 of 63 at $p = 7.2 \times10^{-4}$; and three lesions of Qwen3.5-4B, the Cinderella Broca lesion with 20 of 81 at $p = 0.0063$, the sandwich Wernicke lesion with 42 of 228 at $p = 0.027$, and the sandwich aphasia lesion with 19 of 88 at $p = 0.033$. With a Bonferroni correction for the 16 tests, the first two stay below 0.05. The lesions thus lean toward the full-attention layers, although Gated DeltaNet heads make up most of every lesion.

Steering Details

We next describe how we compute the steering vectors of § 4.2, what linear probes find in the residual stream of the human transcripts, how the sweep chooses a layer and a strength, which settings the human evaluation uses, three further steering experiments, and how alike the Broca and Wernicke vectors are.

Steering vectors. For each model and task, the model reads every human transcript of the task, appended to the task prompt as its own answer: in the plain question-and-answer format for the base models, and in the chat template for the chat models, with thinking turned off for Qwen3.5. We average the residual stream after every block over the tokens of the transcript, leaving out those of the prompt, which gives the states $h_\ell(x)$ of Equation (1). We count layers from 0, so layer 0 is the output of the first block. We cut the prompt and the transcript together at 2,048 tokens, or at 1,024 for GPT-2 XL, which shortens 58 of the 1,139 Cinderella transcripts in GPT-2 XL, 5 in the Llama-2 models, 3 in the Pythia and Qwen3.5 models, and none of the sandwich transcripts. Each text counts once: when AphasiaBank lists the same text more than once, we keep only its first copy, and we drop every copy of a text that it lists under two groups. GPT-2 XL and the Pythia models read their residual stream through LayerNorm, which subtracts the mean over the model dimension, so for these models we subtract that mean from every averaged state before any further step. The Llama-2 and Qwen3.5 models use RMSNorm, which does not subtract the mean, so we keep their raw residual stream. The aphasia vector is the mean of $h_\ell(x)$ over the transcripts of the four aphasic groups minus the mean over the control transcripts, where the four groups are anomic, conduction, Broca’s, and Wernicke’s aphasia. We pool the aphasic transcripts rather than average the four group means, so a larger group weighs more. The Broca and Wernicke vectors subtract the control mean from the mean over the Broca or the Wernicke transcripts.

Linear probes. At every layer, we fit a logistic regression with an L2 penalty, at an inverse regularization strength of 0.01, to the averaged states, which we standardize with the mean and standard deviation of the training folds. We evaluate each probe by five-fold cross-validation, with folds grouped by speaker, so that all transcripts of a speaker fall into one fold, and stratified by label, and we report the mean AUC over the five held-out folds. The aphasia probe separates the 744 Cinderella and 408 sandwich transcripts of the four aphasic groups from the 395 and 472 control transcripts. Unlike the Broca and Wernicke vectors, which compare a type with control speech, the Broca and Wernicke probes separate the transcripts of one type from all other transcripts, including those of the other types of aphasia, so their AUCs measure how well a type stands out from all other speech. A spot check suggests that the other types make these probes harder: at layer 8 of Pythia-1B on the Cinderella task, probes against the control transcripts alone separate Broca’s and Wernicke’s aphasia at AUCs of 0.9999 and 0.9993, against 0.945 and 0.822 for the probes against all other transcripts. To test whether the two tasks share the aphasia direction, we fit the aphasia probe on all transcripts of one task and test it on all transcripts of the other. Because 545 speakers occur in both tasks, a high transfer AUC shows that the tasks share a direction, not that the probe generalizes to new speakers. For severity, we project each aphasic transcript on the unit aphasia vector, $\boldsymbol{v}_\ell/\lVert\boldsymbol{v}_\ell\rVert$, and compute the Spearman correlation $\rho$ between the projection and the speaker’s aphasia quotient from the Western Aphasia Battery (Kertesz, 2007), over the 720 of 744 aphasic Cinderella transcripts and the 382 of 408 aphasic sandwich transcripts that come with a quotient. Lower quotients mean more severe aphasia, so a negative $\rho$ means that the projection grows with severity. Last, we compute at every layer the first principal component of the averaged states of all transcripts, centered over transcripts, and its absolute cosine with the aphasia vector.

Probe results. Table 23 gives these statistics for every model and task. Over all 11 models and all layers, the aphasia probe reaches an AUC of 0.960 to 0.995, the Broca probe 0.933 to 0.963, the Wernicke probe 0.734 to 0.878, and the transfer 0.866 to 0.992. The most negative $\rho$ over layers is weaker on the sandwich task, from −0.635 to −0.701 across models, than on the Cinderella task, from −0.759 to −0.809. The first principal component lies close to the aphasia vector: their absolute cosine averages 0.875 to 0.956 over the layers of a model and task, and it reaches 0.955 to 0.988 at the best layer. It falls below 0.8 only in the first three layers of some models on the sandwich task and in the last layer of Llama-2-13B-chat on the same task, down to 0.086 at layer 1 of Pythia-6.9B. The direction along which the transcripts vary most is thus, at almost every layer, close to the direction that separates aphasic from control speech.

Model	Layers	Aphasia	Broca	Wernicke	Transfer	$\rho$	Layer	PC1
Cinderella task
GPT-2 XL	48	0.981–0.993	0.944–0.951	0.773–0.851	0.871–0.958	−0.759	44	0.955
Llama-2-7B-chat	32	0.980–0.994	0.937–0.947	0.764–0.850	0.884–0.971	−0.797	11	0.948
Llama-2-13B-chat	40	0.979–0.995	0.935–0.947	0.765–0.857	0.889–0.978	−0.806	17	0.943
Pythia-1B	16	0.982–0.992	0.939–0.949	0.772–0.834	0.889–0.970	−0.788	12	0.942
Pythia-1.4B	24	0.983–0.992	0.940–0.950	0.783–0.856	0.867–0.961	−0.770	23	0.941
Pythia-2.8B	32	0.982–0.994	0.936–0.951	0.777–0.857	0.866–0.977	−0.773	12	0.946
Pythia-6.9B	32	0.982–0.993	0.933–0.949	0.762–0.850	0.909–0.970	−0.782	31	0.951
Qwen3.5-0.8B	24	0.982–0.992	0.938–0.956	0.802–0.847	0.894–0.964	−0.777	20	0.933
Qwen3.5-2B	24	0.982–0.993	0.936–0.956	0.801–0.872	0.904–0.966	−0.808	20	0.933
Qwen3.5-4B	32	0.984–0.995	0.937–0.952	0.809–0.857	0.896–0.969	−0.807	27	0.938
Qwen3.5-9B	32	0.983–0.995	0.938–0.950	0.811–0.867	0.877–0.978	−0.809	27	0.941
Sandwich task
GPT-2 XL	48	0.964–0.984	0.950–0.958	0.771–0.855	0.939–0.986	−0.635	38	0.927
Llama-2-7B-chat	32	0.966–0.989	0.947–0.960	0.792–0.865	0.963–0.990	−0.701	13	0.938
Llama-2-13B-chat	40	0.965–0.990	0.940–0.956	0.794–0.866	0.958–0.992	−0.698	15	0.918
Pythia-1B	16	0.965–0.984	0.952–0.957	0.768–0.859	0.966–0.986	−0.638	9	0.890
Pythia-1.4B	24	0.965–0.984	0.950–0.960	0.760–0.866	0.969–0.985	−0.640	14	0.887
Pythia-2.8B	32	0.964–0.985	0.951–0.961	0.758–0.870	0.965–0.989	−0.645	17	0.920
Pythia-6.9B	32	0.960–0.986	0.947–0.959	0.775–0.878	0.957–0.989	−0.654	12	0.875
Qwen3.5-0.8B	24	0.971–0.982	0.953–0.959	0.734–0.851	0.974–0.987	−0.668	12	0.931
Qwen3.5-2B	24	0.970–0.985	0.940–0.960	0.738–0.853	0.972–0.987	−0.651	11	0.952
Qwen3.5-4B	32	0.972–0.988	0.950–0.963	0.757–0.869	0.971–0.991	−0.688	14	0.939
Qwen3.5-9B	32	0.972–0.988	0.946–0.958	0.764–0.847	0.971–0.991	−0.690	17	0.956

The sweep. For each model, task, and vector, the sweep adds the vector of the same task at one of 8 layers, round$(iL/8)$ for $i = 0,\ldots,7$, where $L$ is the model’s number of layers, for example layers 0, 4, $\ldots$, 28 of a model with 32 layers. It uses eight strengths toward the target, $r \in\{0.1, 0.25, 0.5, 1, 1.5, 2, 3, 4\}$, and three toward control speech, $r \in\{-0.5, -1, -2\}$. The strength $r$ compares the norm of the added vector with $\hat{h}_\ell$, the mean norm of the averaged states $h_\ell(x)$ over the task’s transcripts, so it measures the vector against the human transcripts’ averaged states, not against the per-token states of the model’s own answer. Adding the aphasia vector once, with $\alpha= 1$, corresponds to $r$ between 0.035 and 0.508, depending on the model, task, and layer, and at the smallest strength that reaches the target, the sweep adds a median of 5.77 copies of the aphasia vector. We add the vector at every position of the prompt and of the answer, and on the Cinderella task also while the model answers “Do you know the story of Cinderella?”, which it answers separately from the story request. Each setting, a layer and a strength, yields one greedy answer, with a repetition penalty of 1.25 and at most 180 new tokens, so the sweep has no sampling variance. Of the 5,808 steered answers, 683 are shorter than 20 characters, and such answers never count as reaching a target. A vector works at a layer when some positive strength makes the classifier’s probability of the vector’s target exceed 0.5, and for each model, task, and vector we report how many of the 8 layers work and the median, over the working layers, of the smallest strength that works. As a control, each model and task gets three random directions, drawn from a standard Gaussian at every layer, centered over the model dimension for GPT-2 XL and Pythia, and scaled at every layer to the norm of the aphasia vector, so that at the same $r$ they add a vector of the same norm. We add them at the same 8 layers and 8 positive strengths, in 192 settings per model and task, and score them on the aphasia target.

Sweep results. Table 24 gives the sweep’s results for every model and task, and Figure 7 shows $P(\text{aphasia})$ for every setting of the aphasia vector in the largest model of each family. Over both tasks, the aphasia vector works at 103 of the 176 layers: at 54 of 64 in the Pythia models, at a median strength of $r = 0.5$; at 28 of 64 in the Qwen3.5 models, at 1.5; at 15 of 32 in the Llama-2 models, at 2; and at 6 of 16 in GPT-2 XL, at 1.5. The Broca vector works at 78 of the 176 layers and the Wernicke vector at 25, and the Wernicke vector never works in GPT-2 XL or the Llama-2 models. Random directions reach the aphasia target in 119 of the 4,224 settings, or 2.8%: in 6.25% of the settings in the Pythia models, 1.8% in GPT-2 XL, and less than 1% in the Llama-2 and Qwen3.5 models. At every strength, the aphasia vector reaches its target more often than random directions, for example in 41 of the 176 settings at $r = 1$, against 27 of the 528 random settings. At a given layer, the strengths that work lie between weaker ones, which leave $P(\text{aphasia})$ low, and stronger ones, which often turn the answer into model scrambles or into an answer too short to score (Figure 7). Pushes toward control speech rarely reach a target: of the 528 settings with a negative strength, 13 reach the aphasia target, 3 the Broca target, and none the Wernicke target. Without steering, $P(\text{aphasia})$ is at most 0.017, except for Pythia-6.9B on the Cinderella task at 0.167, although the classifier’s most probable class for the unsteered Cinderella answers of GPT-2 XL, Pythia-2.8B, and the four Qwen3.5 models is already model scrambles. Unlike the lesion search, the sweep does not require the model to say that it knows the story of Cinderella, and steering changes that answer too. Of the Cinderella settings that reach their target, 31 of 122 for aphasia, 23 of 89 for Broca, and 2 of 22 for Wernicke answer no to “Do you know the story of Cinderella?”, and 17 of these 31 aphasia settings come from Pythia-6.9B, which answers no even without steering.

Model	Cinderella task							Sandwich task
	Aphasia		Broca		Wernicke		Random	Aphasia		Broca		Wernicke	Random
	$n$	$r$	$n$	$r$	$n$	$r$		$n$	$r$	$n$	$r$	$n$	$r$
GPT-2 XL	2	2.75	3	4	0	–	5	4	1.25	4	2.25	0	–	2
Llama-2-7B-chat	5	2	3	3	0	–	1	4	2	1	3	0	–	1
Llama-2-13B-chat	3	2	3	3	0	–	0	3	3	4	3	0	–	3
Pythia-1B	8	0.5	6	1	4	0.75	8	8	1	8	1	1	0.5	12
Pythia-1.4B	8	0.5	6	1	2	0.75	10	6	1	3	1	2	0.75	10
Pythia-2.8B	8	0.375	5	0.5	4	1	16	4	0.5	3	1	2	1	14
Pythia-6.9B	8	0.25	7	0.25	3	0.5	12	4	1	3	1.5	2	0.75	14
Qwen3.5-0.8B	4	1	2	1.25	2	1.25	2	4	1.5	2	1.5	1	1	1
Qwen3.5-2B	4	1	3	1	0	–	2	6	1.5	3	1.5	1	1	3
Qwen3.5-4B	3	1	2	2.5	1	1.5	2	2	1.5	3	1.5	0	–	1
Qwen3.5-9B	2	1.5	2	1	0	–	0	3	1.5	2	1.75	0	–	0
All 11 models	55	0.5	42	1	16	1	58	48	1.5	36	1.5	9	1	61

Image not extracted

Settings for the human evaluation. The human evaluation uses one setting for each of the seven models whose answers the experts labeled, each source task, and each target (Table 25). Among the positive strengths whose answer to the source task is long enough to score, we take the smallest strength at which the target probability exceeds 0.5, and if several layers pass 0.5 at that strength, the layer with the highest probability. For the Wernicke target in GPT-2 XL and the two Llama-2 models, no setting passes 0.5 on either task, so we take the setting with the highest $P(\text{Wernicke})$, which ranges from 0.011 to 0.422. The chosen setting then steers the answers to all seven prompts, without the question about knowing the story, with greedy decoding, a repetition penalty of 1.25, and at most 180 new tokens for the two narratives and 120 for the five TAB prompts. On the source task, the classifier gives these answers the same probabilities as the sweep’s answers in all 42 settings. Appendix § 15 describes how the experts labeled these answers.

Model	Source task	Aphasia Layer	Aphasia $r$	Aphasia $P$	Broca Layer	Broca $r$	Broca $P$	Wernicke Layer	Wernicke $r$	Wernicke $P$
GPT-2 XL	Cinderella	30	1.5	0.865	18	1	0.535	30	1	0.276†
	Sandwich	36	1	0.803	18	1	0.697	36	1	0.422†
Llama-2-7B-chat	Cinderella	0	1.5	0.994	12	3	0.957	16	3	0.011†
	Sandwich	0	1.5	0.659	12	3	0.995	24	3	0.046†
Llama-2-13B-chat	Cinderella	20	1.5	0.524	20	2	0.570	30	3	0.414†
	Sandwich	10	3	0.999	20	3	0.994	30	3	0.367†
Pythia-1B	Cinderella	8	0.25	0.833	0	1	0.994	8	0.5	0.824
	Sandwich	6	0.25	0.895	4	0.5	0.635	2	0.5	0.508
Pythia-1.4B	Cinderella	18	0.25	0.590	9	0.5	0.963	6	0.5	0.732
	Sandwich	15	0.25	0.611	3	1	0.817	12	0.5	0.725
Pythia-2.8B	Cinderella	20	0.1	0.712	12	0.5	0.822	20	1	0.993
	Sandwich	8	0.5	0.999	4	1	0.995	16	0.5	0.738
Pythia-6.9B	Cinderella	16	0.1	0.741	16	0.1	0.641	20	0.5	0.847
	Sandwich	12	0.25	0.975	28	1.5	0.897	4	0.5	0.820

Steered Broca answers lie among Broca transcripts, lesioned ones near the model scrambles. We place the models’ answers in the text space of Appendix § 8: each model’s intact answer to the task prompt, the answers of its CLAMP lesions for the Broca and Wernicke targets, and its answers steered toward the two types with the task’s own vector, at the setting that the rule above chooses, for all 11 models (Figure 8). For each answer, we count the groups of its 10 nearest texts among the transcripts and model scrambles. The intact answers lie among control transcripts, which make up 82% and 92% of their nearest texts on the Cinderella and the sandwich task. Answers steered toward Broca’s aphasia have 57% and 43% Broca transcripts among their nearest texts, against 21% and 11% of all texts, and only 11% and 12% model scrambles. Answers lesioned toward Broca’s aphasia have 53% and 37% model scrambles among their nearest texts, against 13% and 9% of all texts, and only 33% and 15% Broca transcripts. The lesions that reach the Broca target are no exception: the answers of the 7 such Cinderella lesions have 57% model scrambles among their nearest texts. The Wernicke answers of both methods lie between groups, but Wernicke transcripts, 4% and 3% of all texts, make up 26% and 18% of the nearest texts of the steered answers and 12% and 26% of those of the lesioned answers. In a space that knows nothing of our labels, steering toward Broca’s aphasia thus moves the answer toward Broca’s speech, whereas lesioning moves it toward the machine failures that the classifier learns to set apart. The encoder may, however, place short and fragmented speech near fragmented machine output, which the classifier’s fine-tuning separates, so this map checks the classifier’s labels rather than replacing them.

Image not extracted

Continuations of human transcripts. Steering the model’s own answer cannot show whether subtracting the aphasia direction moves speech toward control speech, because the unsteered answers are already far from aphasic. We therefore start the answer with the first 25 words of a human transcript and let the model continue it for 120 new tokens, while we add the aphasia vector at layer round($L/4$) with $r \in\{-2,-1,0,1,2\}$, and we score only the continuation. Each task uses the same 20 speakers in all 11 models: 10 control speakers and 10 speakers with aphasia, of whom 3 have anomic, 3 Broca’s, 2 conduction, and 2 Wernicke’s aphasia. Adding the direction makes the continuations aphasic: of the 208 continuations of control speakers that the classifier scores but does not call aphasic without steering, 206 are long enough to score at $r = 1$, and 153 of these become aphasic, although in GPT-2 XL only 3 of 18 do, and its continuations become model scrambles instead. Subtracting the direction, however, does not make the continuations of aphasic speakers controllike. Without steering, the classifier calls only 41 of the 220 continuations of aphasic speakers aphasic. Of the 40 of these that are long enough to score at $r = -1$, subtracting the direction makes 32 no longer aphasic, but only 10 of them become control speech by the classifier’s most probable class, while 22 become model scrambles. At $r = -2$, subtracting the direction even turns 50 of the 208 control continuations that are scored and not aphasic without steering into aphasic ones, and the classifier calls 105 of the 220 control continuations model scrambles.

Cinderella $\leftrightarrow$ sandwich transfer of aphasia vector. The aphasia vector of one task also steers the answers to the other task. We add the other task’s vector at the same 8 layers and positive strengths, scaled so that it has the norm that the task’s own vector would have at the same $r$. It works at 98 of the 176 layers, against 103 for the task’s own vector, and in 234 of the 1,408 settings, against 224, although the cosine between the two tasks’ aphasia vectors ranges from 0.265 to 0.936 over models and layers. It works at 54 of 64 layers in the Pythia models, as many as the task’s own vector, and at slightly fewer layers than the task’s own vector in the other families.

Type-specific vectors. The Broca and Wernicke vectors point in nearly the same direction as the aphasia vector, with cosines of 0.838 to 0.989 for the Broca vector and 0.548 to 0.975 for the Wernicke vector over models, tasks, and layers. To steer toward what sets a type apart from aphasia in general, we remove from the Broca and Wernicke vectors their component along the aphasia vector at every layer and sweep these type-specific vectors as above. By the classifier, the Wernicke-specific vector works at 94 of the 176 layers, against 25 for the full Wernicke vector, and in all 22 pairs of model and task, including the 10 in which the full vector never works. Among these 10 are both tasks of GPT-2 XL and of the two Llama-2 models, where the Wernicke-specific vector needs median strengths of 1.5 to 4. The Broca-specific vector, in contrast, works at only 29 layers, against 78 for the full Broca vector. The human evaluation uses the full vectors, so its Wernicke answers from GPT-2 XL and the Llama-2 models come from

The Broca and Wernicke vectors share most of their length. For every model, task, and layer, we compute the cosine similarity of the Broca and the Wernicke vector of the same task and compare it with two references (Table 26). The first reference is the same type’s vector from the other task, which shows how much a vector changes when only the task changes. The second reference shuffles the Broca and Wernicke labels among the transcripts of the two types, 200 times, keeping the group sizes, and recomputes both vectors. Within a task, the Broca and the Wernicke vector have a median cosine over layers of 0.79 to 0.90 in the 22 pairs of a model and a task, and both lie close to the aphasia vector, at median cosines of 0.973 for the Broca vector and 0.915 for the Wernicke vector over models, tasks, and layers. The same type’s vectors from the two tasks have median cosines of only 0.72 for Broca and 0.66 for Wernicke, lower

Model	Broca against Wernicke		Same type, across tasks		Shuffled labels		Below	Type-specific parts
	Cinderella	Sandwich	Broca	Wernicke	Cinderella	Sandwich		Cinderella	Sandwich
GPT-2 XL	0.90	0.81	0.62	0.59	0.99	0.97	96/96	−0.47	−0.43
Llama-2-7B-chat	0.87	0.79	0.66	0.69	0.99	0.97	64/64	−0.48	−0.42
Llama-2-13B-chat	0.86	0.79	0.66	0.69	0.99	0.97	78/80	−0.49	−0.41
Pythia-1B	0.87	0.80	0.72	0.63	0.99	0.97	32/32	−0.65	−0.43
Pythia-1.4B	0.86	0.83	0.72	0.64	0.99	0.97	48/48	−0.63	−0.43
Pythia-2.8B	0.87	0.82	0.71	0.69	0.99	0.97	64/64	−0.62	−0.47
Pythia-6.9B	0.90	0.81	0.73	0.65	0.99	0.97	64/64	−0.57	−0.45
Qwen3.5-0.8B	0.87	0.81	0.77	0.66	0.99	0.98	48/48	−0.65	−0.54
Qwen3.5-2B	0.88	0.83	0.70	0.64	0.99	0.98	48/48	−0.63	−0.53
Qwen3.5-4B	0.90	0.82	0.72	0.69	0.99	0.98	64/64	−0.52	−0.47
Qwen3.5-9B	0.90	0.83	0.75	0.72	0.99	0.98	64/64	−0.48	−0.50

than those of the Broca and Wernicke vectors of one task, although this comparison favors the pair from one task, whose two vectors share the control transcripts and the prompt. Shuffled labels give median cosines of 0.97 to 0.99, and the observed cosine falls below the 2.5th percentile of the shuffled cosines at 670 of the 672 layers, at every layer except the first two of Llama-2-13B-chat on the sandwich task. With the aphasia vector projected out of both, the remaining parts of the Broca and Wernicke vectors have median cosines of −0.65 to −0.41, so the parts that set the two types apart point away from each other. The Wernicke part carries its type: the Wernicke-specific vector works in all 22 pairs, as the previous paragraph shows, whereas the Broca-specific vector works in only 15. The two types thus share most of their deviation from control speech, while each keeps a small part of its own that steering can isolate.

Human Evaluation of Steered Answers

We describe the annotation sheets on which two experts labeled the steered answers, defines the agreement measures of Table 4, and breaks the results of § 5.2 down by expert, class, model, prompt, and steering vector.

The annotation sheets. We steer seven models, GPT-2 XL, Llama-2-7B-chat, Llama-2-13B-chat, Pythia-1B, Pythia-1.4B, Pythia-2.8B, and Pythia-6.9B, toward each of the three targets with each of two steering vectors, one computed from the Cinderella transcripts and one from the sandwich transcripts. Each steered model uses the layer and strength that the sweep of § 4.2 chose for its target on the vector’s own task, the source task (Appendix § 14), and answers seven prompts once. The seven prompts are the requests of the two tasks and the five connected-text prompts of the TAB: “Tell me about the best trip you ever took”, “Describe a happy childhood memory”, “Tell me about your first job”, “What do you like about where you live?”, and “Describe the steps to make a simple meal”. The models decode as in § 3, except that they write at most 120 new tokens for the TAB prompts. Seven models, seven prompts, two steering vectors, and three targets give 294 steered answers. The answers steered toward aphasia form the two-way sheet, on which the experts label each answer as aphasic or not. The answers steered toward Broca’s or Wernicke’s aphasia form the five-way sheet, on which the experts give each answer one of the classifier’s five classes: control speech, conduction or anomic aphasia, Broca’s aphasia, Wernicke’s aphasia, or model scrambles. We leave 22 answers out of the analysis, 7 of the 98 on the two-way sheet and 15 of the 196 on the five-way sheet: 18 answers with fewer than 20 characters, 8 of them empty, which the classifier does not score (§ 3), and 4 answers for which the LLM judge returned an empty reply. The analysis thus covers 272 answers, 91 on the two-way sheet and 181 on the five-way sheet, of which 90 are steered toward Broca’s aphasia and 91 toward Wernicke’s aphasia.

Expert labels. We take expert 1’s labels from the annotation sheets, and expert 2’s from a tab of the sheets in which expert 2 labeled the answers before the experts compared notes. The sheets also hold a later version of expert 2’s labels, set after the experts compared notes, and on all 46 answers where the two versions differ, the later version equals expert 1’s label. With the later version, the experts would agree on 96% of the answers, at a Cohen’s $\kappa$ of 0.95, rather than on 80%, at a $\kappa$ of 0.76. Expert 2’s tab has no label for 4 answers, 2 on each sheet, so we compare these answers with expert 1 alone and leave them out of every statistic that needs both experts. The two-way sheet has no label for model scrambles, so an answer on it counts as a model scramble when the experts’ comments mark it as one, which holds for 25 of its 91 answers. On the five-way sheet, an answer counts as a model scramble when either expert labels it so, which holds for 52 of its 181 answers.

Agreement measures. The classifier and the LLM judge label the same answers as the experts. On the two-way sheet, the classifier labels an answer aphasic when $P(\text{aphasia})$ exceeds 0.5, and the judge answers yes or no to whether the speaker has aphasia. On the five-way sheet, the classifier gives its most probable class, and the judge names one of the five classes (Appendix § 17). A labeler agrees with the experts on an answer when its label matches the label of either expert, which is the agreement of Table 4. As a baseline, the best constant label gives every answer of a sheet the one label that agrees with the experts most often: not aphasic on the two-way sheet and Broca’s aphasia on the five-way sheet. It agrees with the experts on 63% of the two-way answers, 39% of the five-way answers, and 47% of all answers, against 73%, 62%, and 65% for the classifier and 64%, 51%, and 55% for the judge. Cohen’s $\kappa$ corrects the agreement of two labelers for the agreement expected by chance [10]: $\kappa=(p_o-p_e)/(1-p_e)$, where $p_o$ is the share of answers on which the two agree and $p_e$ the share on which two independent labelers with the same label frequencies would agree. By the scale of Landis and Koch (1977), a $\kappa$ from 0.21 to 0.40 is fair agreement, from 0.41 to 0.60 moderate, from 0.61 to 0.80 substantial, and above 0.80 almost perfect. On all answers, $\kappa$ pools the two sheets and treats their labels as one set of seven.

Agreement with each expert. Table 27 compares the classifier and the judge with each expert separately and with the experts’ consensus, the answers to which both experts give the same label. The classifier agrees more with expert 2 than with expert 1, on 62% against 53% of all answers, at $\kappa=0.54$ and 0.44, whereas the judge agrees with the two about equally, on 50% and 48%, at $\kappa=0.39$ and 0.38. By the scale of Landis and Koch (1977), the classifier’s agreement with each expert is moderate and the judge’s is fair, whereas the experts’ agreement with each other, at $\kappa= 0.76$, is substantial. On the 214 answers on which the experts agree, the classifier gives their label to 62% and the judge to 54%. The classifier and the judge agree with each other on only 42% of all answers, at $\kappa= 0.31$, and on the two-way sheet at $\kappa= 0.00$, which is no better than chance.

	Two-way	Five-way	All
Against expert 1
Classifier	64	48	53
Cohen’s $\kappa$	0.25	0.33	0.44
LLM judge	62	41	48
Cohen’s $\kappa$	0.22	0.21	0.38
Against expert 2
Classifier	71	57	62
Cohen’s $\kappa$	0.38	0.43	0.54
LLM judge	55	48	50
Cohen’s $\kappa$	0.09	0.26	0.39
On the experts’ consensus
Classifier	71	57	62
LLM judge	60	51	54
Answers	78	136	214
Classifier against LLM judge
Agreement	52	37	42
Cohen’s $\kappa$	0.00	0.19	0.31

Model scrambles. The classifier recognizes model scrambles much better on the two-way sheet than on the five-way sheet. On the two-way sheet, the classifier’s label shows only whether an answer is aphasic, so we use its most probable class: it puts 27 answers in the model-scramble class, 20 of which the experts’ comments mark as model scrambles, a precision of 74%, and it finds 20 of the 25 marked answers, a recall of 80%. On the five-way sheet, its precision is 52%, 16 of 31, and its recall 31%, 16 of 52, and over both sheets, its precision is 62% and its recall 47%. The judge can name model scrambles only on the five-way sheet, where it calls 4 answers model scrambles, 2 of them correctly, a precision of 50% and a recall of 4%. On the two-way sheet, the judge calls 20 of the 25

Class	Experts	Classifier Calls	Classifier Prec.	Classifier Rec.	LLM judge Calls	LLM judge Prec.	LLM judge Rec.
Control	16	18	50	56	14	57	50
Conduction or anomic	22	28	36	45	12	17	9
Broca	70	68	74	71	40	57	33
Wernicke	64	36	75	42	111	51	89
Model scrambles	52	31	52	31	4	50	4

model scrambles aphasic, whereas the classifier’s label calls 5 of them aphasic. Of the 52 five-way answers that either expert calls model scrambles, the classifier labels 30 as Broca’s aphasia, and the judge labels 28 as Wernicke’s aphasia and 19 as Broca’s aphasia. Both experts call 22 of these answers model scrambles, and the classifier finds 8 of them and labels the other 14 as Broca’s aphasia. On the source task, the classifier calls all 14 answers steered toward Broca’s aphasia Broca-like, because its own score chose their steering settings, whereas the experts call 9 of them model scrambles. The experts disagree with each other mostly about model scrambles. Of their 43 disagreements on the five-way sheet, 29 involve model scrambles, and in 27 of these, expert 1 labels a model scramble where expert 2 labels a type of aphasia: Broca’s aphasia 18 times, Wernicke’s aphasia 8 times, and conduction or anomic aphasia once. Overall, expert 1 labels 50 of the 181 five-way answers as model scrambles, and expert 2 labels 24 of the 179 that expert 2 labeled. On the two-way sheet, the experts disagree on 11 of the 89 answers that both labeled.

Agreement by class. On the five-way sheet, the classifier is more precise than the judge for every class except control speech, and it recalls more answers of every class except Wernicke’s aphasia (Table 28). The judge recalls 57 of the 64 answers that either expert labels as Wernicke’s aphasia, 89%, but it gives that label to 111 of the 181 answers,

caption: Labels of the classifier against those of expert 2 on the 179 answers steered toward Broca’s or Wernicke’s aphasia that expert 2 labeled. Rows give expert 2’s label and columns the classifier’s, and the bold diagonal counts the 102 answers on which the two agree. Ctrl.: control speech; Cond.: conduction or anomic aphasia; Wern.: Wernicke’s aphasia; MS: model scrambles.

:::

and it recalls only 23 of the 70 Broca answers, 33%. Against expert 2, the classifier’s most frequent errors are 15 model scrambles that it labels as Broca’s aphasia, 14 Wernicke answers that it labels as conduction or anomic aphasia, and 11 Wernicke answers that it labels as model scrambles (Table 29). Against expert 1, who calls more answers model scrambles, the classifier labels 29 of expert 1’s 50 model scrambles as Broca’s aphasia, and it agrees with expert 1 on 87 of 181 answers, 48%, against 102 of 179, 57%, with expert 2.

Steering success by model, prompt, and vector. Steering toward Wernicke’s aphasia reaches the target mostly in the Pythia models, for 40 of their 50 answers that are not model scrambles, against 8 of 26 in GPT-2 XL and the Llama-2 models, for which no Wernicke setting passed 0.5 on the source task (Table 30). Steering toward Broca’s aphasia, in contrast, fails in Pythia-1B and Pythia-6.9B, for 0 of 4 and 1 of 7 such answers, and 10 of the 14 Broca answers of Pythia-1B are model scrambles. Over all seven models, steering toward Broca’s aphasia breaks the most answers: the experts call 37 of its 90 answers model scrambles, against 25 of 91 for aphasia and 15 of 91 for Wernicke’s aphasia. By prompt, steering reaches the target most often on the Cinderella prompt, for 22 of 26 answers that are not model scrambles, and least often on the prompts about the best trip and about where the speaker lives, for 15 of 31 and 11 of 25 (Table 31). Steering vectors computed from the sandwich transcripts reach the target more often than those computed from the Cinderella transcripts.

Model	Aphasia w/o MS	Aphasia with MS	Broca w/o MS	Broca with MS	Wernicke w/o MS	Wernicke with MS	All three w/o MS	All three with MS
GPT-2 XL	6/7	9/14	7/7	7/14	5/13	5/14	18/27	21/42
Llama-2-7B-chat	2/2	2/9	6/6	7/12	2/7	3/12	10/15	12/33
Llama-2-13B-chat	4/9	4/13	5/9	7/13	1/6	1/9	10/24	12/35
Pythia-1B	7/12	8/14	0/4	6/14	11/13	12/14	18/29	26/42
Pythia-1.4B	9/13	9/13	8/8	13/14	9/14	9/14	26/35	31/41
Pythia-2.8B	5/11	5/14	9/12	11/14	9/11	10/14	23/34	26/42
Pythia-6.9B	6/12	8/14	1/7	1/9	11/12	13/14	18/31	22/37
All models	39/66	45/91	36/53	52/90	48/76	53/91	123/195	150/272

Prompt	Aphasia w/o MS	Aphasia with MS	Broca w/o MS	Broca with MS	Wernicke w/o MS	Wernicke with MS	All three w/o MS	All three with MS
Tell me as much of the story of Cinderella as you can.	9/9	11/12	6/7	8/14	7/10	9/13	22/26	28/39
Tell me how you would make a peanut butter and jelly sandwich.	7/9	7/13	1/3	6/14	8/11	8/14	16/23	21/41
Tell me about the best trip you ever took.	3/11	4/14	7/11	8/13	5/9	7/12	15/31	19/39
Describe a happy childhood memory.	6/10	7/12	9/12	10/13	9/14	9/14	24/36	26/39
Tell me about your first job.	7/9	7/14	6/9	7/12	5/9	6/12	18/27	20/38
What do you like about where you live?	2/8	4/13	3/6	6/12	6/11	6/12	11/25	16/37
Describe the steps to make a simple meal.	5/10	5/13	4/5	7/12	8/12	8/14	17/27	20/39
All prompts	39/66	45/91	36/53	52/90	48/76	53/91	123/195	150/272

scripts. Without model scrambles, either expert labels the target for 79%, 84%, and 74% of the answers steered toward aphasia, Broca’s aphasia, and Wernicke’s aphasia with sandwich vectors, against 45%, 54%, and 54% with Cinderella vectors. With model scrambles, the gap narrows to 56%, 60%, and 66% against 43%, 55%, and 51%, and toward aphasia, sandwich vectors produce more model scrambles, 17 of 45 answers against 8 of 46.

Steering carries its type to the prompts that did not choose its setting. The sweep chose each steering setting on the source task alone, so the answers to the other narrative and to the five TAB prompts test whether the targeted type carries over. Without model scrambles, either expert labels the target for 5 of the 5 Broca answers and 8 of the 12 Wernicke answers to the source task, and for 31 of the 48 Broca answers, 65%, and 40 of the 64 Wernicke answers, 63%, to the other six prompts. With model scrambles, the labels on these six prompts follow the steering vector: either expert labels 44 of the 76 answers steered toward Broca’s aphasia as Broca’s aphasia and only 9 as Wernicke’s aphasia, and 44 of the 77 answers steered toward Wernicke’s aphasia as Wernicke’s aphasia and only 13 as Broca’s aphasia. The two vectors thus produce different types on prompts that played no part in choosing them.

Examples. Table 32 shows seven steered answers. In the first five, both experts label the answer with the steering target. The two answers of Llama-2-7B-chat to the prompt about its first job show how the targets differ: steered toward Broca’s aphasia, it answers in six words, and steered toward Wernicke’s aphasia, it answers fluently but with little content. In the last two, both experts call the answer a model scramble, while the classifier and the judge both label it as Broca’s aphasia: one ends in a loop of “okay.” and “ok.”, and the other is the word “Question”, 171 line breaks, “yes”, and “no”. The first of the two is an answer to the source task, so the classifier’s score for it, $P(\mathrm{Broca}) = 0.96$, is the score with which the sweep chose the steering setting that produced it.

Model and vector	Prompt	Classifier	LLM judge	Answer
Steered toward aphasia, labeled aphasic by both experts
GPT-2 XL, Cinderella vector	Cinderella†	Aphasic (0.87)	Aphasic	I don,t know that I like to tell it. It’s a good thing that I didn’t. / I was in my. . .
Steered toward Broca’s aphasia, labeled Broca by both experts
Llama-2-13B-chat, Cinderella vector	Simple meal	Model scrambles (0.39)	Broca	Sure! Here are the steps to make a simple meal: / First, choose a recipe you like and ingredients list need . go shopping groc yes. Next, preheat oven or stove top cooking no. Then, chop vegetables car yes. After that, cook meat dish yes. Finally, serve hot enjoy eat.
Llama-2-7B-chat, Cinderella vector	First job	Broca (0.71)	Broca	sure! my job is good. okay.
Steered toward Wernicke’s aphasia, labeled Wernicke by both experts
Llama-2-7B-chat, Cinderella vector	First job	Wernicke (0.60)	Conduction or anomic	sure, my first job was a lot of fun. I like you know what it’s good for. It’s good for you.
Pythia-1B, Cinderella vector	Cinderella†	Wernicke (0.82)	Wernicke	I think it’s a very good one, and that is what we are doing now in this country. We have been trying to do something for years. It was not so long ago when they were saying about the other thing. But there is no reason why we should be afraid because if he will say yes or no? Yes. No. And then also with him. So yeah. Yeah.
Steered toward Broca’s aphasia, called model scrambles by both experts
Llama-2-7B-chat, Cinderella vector	Cinderella†	Broca (0.96)	Broca	sure! The story of Cinder. / Cinderella. / Okay, let’s go. / okay okay. / Sure. / okay. / okay. / okay. / okay. / okay. / okay. / okay. ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok. / ok.
Pythia-6.9B, sandwich vector	Cinderella	Broca (0.94)	Broca	Question [171 line breaks] yes / no

Steering Back

We describe the steering-back test of § 4.2 and report its results. The test asks whether removing a lesion’s own direction from the lesioned model, or adding the direction with a negative strength, undoes the lesion, and whether random directions do the same. Random directions undo the lesions almost as often as the own direction, so the direction of a type does not reverse a lesion specifically. Most undone answers are model scrambles rather than the intact model’s speech, so steering back undoes the lesions without healing them.

Random directions serve as controls. We test the 54 CLAMP lesions that reach their target (Figure 4). An answer reaches the target when the classifier’s target probability exceeds 0.5 or when the target is its most probable class, where any of the three aphasia classes counts for the aphasia target. A lesion’s own direction is the steering vector $v_{\ell}$ of its target (Equation (1)), computed on the task where the search found the lesion. For the aphasia target, the own direction is the aphasia vector of Appendix § 14, which pools the transcripts of the four aphasic groups. The controls are three Gaussian random directions, and the same three serve the lesions of all three targets of a model and task. For each lesion, the model answers the prompt of that task under 135 conditions, with the greedy decoding and the scoring of §3. Seven answers are references: those of the intact model, of the intact model with the own direction removed, of the lesioned model, of the lesioned model with the own direction removed, and of the lesioned model with each of the three random directions removed. The other 128 answers come from pushes, one for each combination of four directions, eight layers, and four strengths. A push adds the own direction or one of the three random directions to the lesioned model after one of eight layers spread across the model’s depth, with a strength of $r=-0.5$, $-1$, $-2$, or $-4$. We call a push with the own direction an own push, and a push with a random direction a random push. If a lesion acts through its own direction, steering back along that direction should undo it more often than steering back along a random direction.

Pushes match the directions in size, but removal does not. Following Arditi et al. [3], removal projects a direction’s unit vector out of the residual stream after every block (§4.2). Removal takes out whatever part of the residual stream lies along the direction, so the amount that it takes out depends on the direction, and we did not record this amount for either kind of direction. A push adds a multiple of the direction after one layer $\ell$, scaled so that the added vector has the norm $|r|\bar{h}_{\ell}$, with $r$ and $\bar{h}_{\ell}$ as in §4.2. The added vector thus has the same norm for every direction, so pushes compare the own and the random directions at equal size. A removal or a push undoes the lesion when its answer no longer reaches the target, that is, when its target probability is 0.5 or below and the target is not its most probable class. An answer shorter than 20 characters does not count by the rules of §3, so we do not score it, and such an answer does not undo the lesion.

Own and random directions both undo the lesions. Before steering back, the lesioned answers have a mean target probability of 0.711, and the intact answers have a mean of 0.006. Removing the own direction undoes 53 of the 54 lesions, and the remaining lesion, the Llama-2-13B-chat sandwich Wernicke lesion, gives an answer too short to score (Table 33). The three random removals undo 51, 50, and 50 of the 54 lesions. At least one of them undoes each of the 54 lesions, and all three undo 45 of them. The own direction leaves less of the target probability: after removal, a mean of 0.010, against 0.081 for the random directions (Figure 9). Removal of the own direction leaves less than random removal in every model family and target except the Llama-2 models, where the two leave 0.029 and 0.028. Pushes, which match the directions in size, keep this order only at weak strengths, with 0.024 against 0.078 at $r=-0.5$ and 0.034 against 0.072 at $r=-1$, and lose it at strong ones, with 0.055 against 0.047 at $r=-2$ and 0.020 for both at $r=-4$. At every strength and at every one of the eight layers, most pushes undo the lesion, own pushes slightly more often than random ones: 92 to 97% against 89 to 94% across the strengths, and 87 to 99% against 83 to 96% across the layers. Over all strengths and layers, 1,643 of the 1,728 own pushes and 4,720 of the 5,184 random pushes undo the lesion, or 95% and 91%. A random direction thus undoes a lesion almost as often as the lesion’s own direction, so undoing a lesion does not show that the lesion acts through the direction of its type.

Image not extracted

	Removal undoes				Probability left
Sets	Own	Random ≥ 1	Random All 3	Own	Random
By model family
GPT-2	5	5	4	0.000	0.052
Llama-2-chat	11	10†	11	0.029	0.028
Pythia	22	22	16	0.005	0.133
Qwen3.5	16	16	14	0.006	0.056
By target
Aphasia	22	22	19	0.007	0.094
Broca	15	15	13	0.004	0.038
Wernicke	17	16†	13	0.019	0.103
All	54	53†	45	0.010	0.081

The undone answers are not the intact answers. Undoing a lesion asks only that the answer no longer reach the target, so an undone answer need not be the intact model’s speech. No removal or push reproduces the intact answer: none of the 54 own removals, 162 random removals, 1,728 own pushes, and 5,184 random pushes yields the intact text. Removing the own direction undoes the lesions mostly by breaking the answers. The classifier’s label, its most probable class, is model scrambles for 52 of the 53 undone answers and control speech for none of them. The remaining answer, of the 276-head Llama-2-7B-chat Cinderella Wernicke lesion, stays aphasic, and its label moves from Wernicke’s aphasia to conduction or anomic aphasia. To check such labels without the classifier, we call an answer fluent when it passes two tests. First, compressing the answer with zlib must leave at least a quarter of its length, which rules out answers that loop. Second, at least 80% of its non-space characters must belong to words of an American English word list. This sense of fluent concerns broken text, not the clinical fluency of Appendix § 7. The intact answers of all 54 lesions are fluent by this check, but 35 of the 53 undone answers of the own removal are not. The check matters because the label alone overstates breakage: the classifier also labels the intact answers of 13 of the 54 lesions model scrambles, although all of them are fluent. Random removals break the answers less often: 111 of the 151 random removals that undo a lesion give a fluent answer, and 34 give a fluent answer that the classifier labels control speech. An undone lesion is thus not a healed one: no answer returns to the intact text, and removing the own direction mostly breaks the answer.

Few pushes restore fluent control speech. Of the 1,728 own pushes, 167, or 9.7%, undo the lesion with a fluent answer that the classifier labels control speech, and such pushes exist in 38 of the 54 lesions. Of the 5,184 random pushes, 271 do so, or 5.2%, in 32 lesions. Most pushes that undo a lesion end in model scrambles instead: 1,411 of the 1,643 own pushes and 4,290 of the 4,720 random pushes. Stronger pushes less often undo the lesion with an answer that the classifier labels control speech: 22% of the own and 15% of the random pushes do so at $r=-0.5$, and none of the 432 own and 1,296 random pushes at $r=-4$.

Table 34 shows two lesions that a weak own push undoes with fluent control speech, which we chose because their answers read well. In both, the steered-back answer is a fluent answer to the task, but not the intact answer. At the same layer and strength, all three random pushes also undo both lesions, and the classifier labels five of their six answers model scrambles. Removing the own direction instead turns both lesioned answers into model scrambles, with a probability of 1.00 for that class. The own direction thus restores control speech almost twice as often as a random direction, but for fewer than one push in ten.

Intact	Lesioned	Lesioned and steered back
GPT-2 XL, Broca target, lesion of 146 heads; own direction pushed after layer 24 at $r=-0.5$<br>$P(\mathrm{Broca})=0.000$, control	$P(\mathrm{Broca})=0.884$, Broca	$P(\mathrm{Broca})=0.000$, control
I’d start with the bread, cut it into slices about 1/2 inch thick, then spread some of your favorite jam on each slice. Then add in some sliced peanuts (or other nuts) and top that off with a scoop of PB&J. You can also use peanut butter instead of…	I’d love to have some peanut butter, but not the kind of peanut butter. / I’m sure do like peanut butter. / But I don’t know what it is. / It’s just peanut butter. / And I can’t tell you what it is. / Peanut Butter. / That’s right? / Yes. / No. / What are you talking…	I’d use the same recipe as shown in this picture, but with only half of the peanut butter and half of the jelly. Then I’d put it on top side of the bread and then I’d spread it out to form a circle around the bread and I’d cut off about 1/4th of an…
Llama-2-7B-chat, aphasia target, lesion of 60 heads; own direction pushed after layer 8 at $r=-0.5$<br>$P(\mathrm{aphasia})=0.006$, control	$P(\mathrm{aphasia})=0.931$, Broca	$P(\mathrm{aphasia})=0.006$, control
Sure! Here’s my step-by-step guide on how to make the perfect peanut butter and jelly sandwich: / / Ingredients: / / * 2 slices of bread (white or whole wheat, whichever you prefer) / * Peanut butter (creamy or crunchy, your choice) / * Jelly or jam (your…	[emoji]peanut Butt the jigglesand!	Sure! Here’s one way to make a peanut butter: / / / / 1. Start by spreading a layer of bread on the bottom of your pan. You can use any type of bread you like, such as white or whole wheat. Just be sure it is flat enough for the filling. / 2. Next, add a…

Removing the own direction also breaks intact answers. Removing the own direction from the intact model changes the intact answer for all 54 lesions and makes 20 of these answers fail the fluency check. In three settings, all of Qwen3.5 with the aphasia target, removing the aphasia direction even makes the intact model’s answer reach the target. The probability of aphasia rises from 0.001 to 0.548 for Qwen3.5-9B on the Cinderella task and from 0.006 to 0.848 on the sandwich task. For Qwen3.5-4B on the sandwich task, it rises from 0.005 to 0.450, and conduction or anomic aphasia becomes the most probable class. Removal thus damages the language of intact models too, so its undoing of the lesions does not show that the lesions act through the own direction.

Random directions also undo the lesions of the layer search. For comparison, we ran the same test on nine aphasia lesions of GPT-2 XL and the Pythia models that the layer search of Appendix D found. These lesions have one to seven heads, and the GPT-2 XL lesion also knocks out seven blocks of 256 MLP neurons. Removing the own direction undoes all nine lesions, whereas at least one random removal undoes eight of them and all three random removals undo six. In one lesion, of Pythia-1B on the sandwich task, the removal of the own direction undoes the lesion but no random removal does: the three random removals leave a mean target probability of 0.92. Over the nine lesions, removal leaves a mean target probability of 0.001 for the own direction and 0.244 for the random directions. Random removals undo these lesions 21 of 27 times, and they undo the CLAMP aphasia lesions of the same nine pairs of a model and a task about as often, 22 of 27 times. In three of these pairs, both searches found the same single head. Random directions thus undo most lesions of either search, so the weak specificity of steering back is not a property of CLAMP alone.

LLM Judge Prompts

One LLM judge, DeepSeek-V4.1-Flash, plays two roles in this paper. It labels the steered answers that the experts also label, given only the names of the classes (§ 4.2), and it rates the answers of intact and lesioned models to the TAB’s five connected-text prompts for 19 aphasia features (§ 4.1). This appendix gives the judge’s settings, its three prompts verbatim, and the rules that turn its replies into labels. In the listings, a gray arrow marks where a long line of a prompt wraps on the page.

Judge and settings. We query DeepSeek-V4.1-Flash ([15]) through DeepSeek’s application programming interface (API), under the model name deepseek-flash, at temperature 0. Each request holds one answer and nothing about how the answer was produced, such as the steering target or the lesion. DeepSeek-V4-flash is a reasoning model, and it is not deterministic at temperature 0: the same request can return a different reply. We therefore send each distinct request once and keep its first reply. We store every reply under the full content of its request, which is the model name, the messages, the temperature, and, for the TAB prompt, the reply format. An analysis that sends a request again reads the stored reply instead of asking the judge, so all analyses use the same verdicts. We parse only the final reply, not the reasoning that the API returns with it.

model answered, for example “Tell me about your first job.”, and {answer} is the steered answer. For the TAB’s prompts, the question leaves out the battery’s instruction “Respond to the following prompt in 3-5 full sentences:”, which the model received with the prompt. The judge receives the filled prompt as the only message of the request, with no system message.

Prompts for the steered answers. The judge labels each answer steered toward aphasia with the two-way prompt of Listing 2, and each answer steered toward Broca’s or Wernicke’s aphasia with the five-way prompt of Listing 3. Both prompts give the class names and nothing else: no definitions and no examples. The classifier learned its classes from labeled examples, and a judge given hand-written definitions or examples would judge partly by those, so with the class names alone the judge relies on what it knows about aphasia. In both prompts, {question} is the request that the model answered, for example “Tell me about your first job.”, and {answer} is the steered answer. For the TAB’s prompts, the question leaves out the battery’s instruction “Respond to the following prompt in 3-5 full sentences:”, which the model received with the prompt. The judge receives the filled prompt as the only message of the request, with no system message.

Listing 2: Two-way prompt of the LLM judge, for the answers steered toward aphasia. The judge receives it as a single user message, with the request that the model answered in place of {question} and the model’s answer in place of {answer}.

Here is a question and a speaker's answer to
    it, transcribed.

Question: {question}
Answer: {answer}

Does the speaker have aphasia? Reply with one
    word: yes or no.

Listing 3: Five-way prompt of the LLM judge, for the answers steered toward Broca’s or Wernicke’s aphasia, sent in the same way as the two-way prompt of Listing 2. The options are the classifier’s five classes, and the last option stands for model scrambles.

Here is a question and a speaker's answer to
↪ it, transcribed.

Question: {question}
Answer: {answer}

Which one of these describes the speaker?
Control (no aphasia)
Anomic or conduction aphasia
Broca's aphasia
Wernicke's aphasia
Not human speech (garbled machine text)

Reply with exactly one of these five options.

Prompt for the TAB’s connected text. The TAB prompt follows the battery’s protocol for scoring connected text automatically (Roll et al., 2026b). Its system message, in Listing 4, is the TAB paper’s instruction to its judge, word for word. Its user message, in Listing 5, defines the 19 features with an example of each, shows two transcripts annotated with all 19 features, and then gives the answer to rate. There, {prompt} is one of the TAB’s five connected-text prompts, again without the instruction on length, and {transcript} is the model’s answer. The request also asks the API for a reply in JavaScript Object Notation (JSON). The features are the battery’s 19, in its order, but we wrote the definitions, the examples, and the two annotated transcripts ourselves, from the feature names and from standard usage in aphasiology. The prompt thus reconstructs the battery’s protocol rather than copying it, and its definitions and examples differ from those that the TAB paper prints. Both annotated transcripts are synthetic, so the judge never sees text from an AphasiaBank transcript. The TAB’s authors validated their protocol against expert raters with Gemini 2.5 Flash as the judge, whereas we did not compare the TAB ratings of DeepSeek-V4.1-Flash with human raters, so the TAB results of Table 3 rest on the judge alone.

From replies to labels. A two-way reply gets the label of its first whole word yes or no, ignoring case: aphasic for yes, and not aphasic for no. A five-way reply gets the class of the keyword that appears first in it, ignoring case: control for control speech, anomic or conduction for conduction or anomic aphasia, broca for Broca’s aphasia, wernicke for Wernicke’s aphasia, and not human or garbled for model scrambles. A reply without any of these words gets no label. The judge labeled 294 steered answers, 98 steered toward aphasia and 196 toward Broca’s or Wernicke’s aphasia, and all but 4 of its replies were exactly one of the options, up to capitalization. For those 4 answers, 1 steered toward aphasia and 3 toward Broca’s or Wernicke’s aphasia, the judge’s reasoning reached 65,536 tokens and its reply was empty. Neither these 4 answers nor the 18 answers that were too short for the classifier are among the 272 answers of Table 4. We read a TAB reply as a JSON object, from its first opening brace to its last closing brace, and give each feature its value in the object, 0 or 1; a feature that the object leaves out counts as absent. Every reply behind Table 3 was a JSON object with exactly the 19 features, each 0 or 1. Table 3 counts the features that a reply marks present, out of 19, so the count includes the feature of overall communication impairment.

Listing 4: System message of the TAB judge prompt, which is the instruction that the TAB paper gives its judge (Roll et al., 2026b).

You are an automated system for analyzing
↪ transcripts of speech for aphasic
↪ features. Your task is to read a
↪ transcript passage and determine
↪ whether each of the following features
↪ is present (1) or not present (0).
↪ Your output must be a JSON file with
↪ each feature as a key and its value as
↪ either 0 or 1.

Listing 5: User message of the TAB judge prompt, sent after the system message of Listing 4 for each connected-text answer, with the TAB prompt in place of {prompt} and the model’s answer in place of {transcript}. We wrote the definitions, the examples, and the two annotated transcripts, which differ from those of the TAB paper.

Features to score:
- anomia: Word-finding difficulty: pauses, circumlocutions, vague fillers ('thing', 'stuff'), or an explicit failure to retrieve a word. Example: "we went to the... the place with the, um, the water thing"
- abandoned_utterances: An utterance is started and given up before it is completed. Example: "and then we drove to the—. It was a nice day."
- empty_speech: Fluent output that carries little information: generic words and vague statements in place of content. Example: "It was good. We did things there. It was really something, you know."
- semantic_paraphasias: A real word substituted for the intended one from a related meaning (fork for spoon). Example: "I poured the milk into the fork"
- phonemic_paraphasias: Sound-level errors that produce a near-word: substituted, added, or transposed sounds. Example: "we watched the tevelision"
- neologisms: Invented non-words used as if they were words. Example: "she brought the flimber to the table"
- jargon: Fluent but largely incomprehensible speech made of neologisms and paraphasias. Example: "the plorbing came to the sandwitcher and it was gasping"
- perseverations: Inappropriate repetition of a word, phrase or idea already produced, carried over where it no longer fits. Example: "I like the beach. The beach is nice. My job was at the beach. The beach."
- stereotypies_and_automatisms: Recurring stock phrases or automatic sequences used instead of content ('you know', 'and so on', counting). Example: "you know, you know, and so on, you know"
- short_simplified_utterances: Telegraphic output: very short utterances with reduced syntax. Example: "Trip. Beach. Nice. Sun."
- omission_of_bound_morphemes: Missing inflections such as -ed, -s, -ing. Example: "yesterday I walk to the store and buy two apple"
- omission_of_function_words: Missing articles, prepositions, auxiliaries, or pronouns. Example: "went store. bought bread. came home."
- paragrammatism: Grammatical errors of commission in otherwise fluent speech: wrong tense, agreement, or word order. Example: "the trip were going by us to have had yesterday"
- false_starts: An utterance begins, stops, and restarts in a different form. Example: "My first— when I first— the job I had first was..."
- retracing: Words or phrases repeated in order to self-correct or to continue. Example: "we went to the, we went to the beach"
- conduite_d_approche: Successive approximations toward a target word. Example: "cat- catas- catastro- catastrophe"
- meaning_unclear: The intended message cannot be recovered from the utterance. Example: "it was the one for the when they had it over"
- off_topic: The response does not address the prompt. Example: (asked about a trip) "The stock market has been volatile recently."
- overall_communication_impairment: Global judgment: taken as a whole, communication is impaired. Example: (any transcript where the features above materially reduce communicative success)

Annotated example 1:
Prompt: Tell me about your first job.
Transcript: My first job was at a small bakery near my school. I started at five in the morning, helped shape the dough, and served customers once we opened. It was tiring but I loved the smell of fresh bread. I stayed there for two summers before going to college.
Labels: {"anomia": 0, "abandoned_utterances": 0, "empty_speech": 0, "semantic_paraphasias": 0, "phonemic_paraphasias": 0, "neologisms": 0, "jargon": 0, "perseverations": 0, "stereotypies_and_automatisms": 0, "short_simplified_utterances": 0, "omission_of_bound_morphemes": 0, "omission_of_function_words": 0, "paragrammatism": 0, "false_starts": 0, "retracing": 0, "conduite_d_approche": 0, "meaning_unclear": 0, "off_topic": 0, "overall_communication_impairment": 0}

Annotated example 2:
Prompt: Tell me about your first job.
Transcript: Job... um, first job. Bakery. Bakery, no, the— the bread place. Five o'clock, morning. Make the... the thing, the dough. Dough. People come, I give bread. Two summer. Tired but good. Bread smell good, good.
Labels: {"anomia": 1, "abandoned_utterances": 0, "empty_speech": 0, "semantic_paraphasias": 0, "phonemic_paraphasias": 0, "neologisms": 0, "jargon": 0, "perseverations": 1, "stereotypies_and_automatisms": 0, "short_simplified_utterances": 1, "omission_of_bound_morphemes": 1,

↪ "omission_of_function_words": 1,
↪ "paragrammatism": 0, "false_starts":
↪ 1, "retracing": 1,
↪ "conduite_d_approche": 0,
↪ "meaning_unclear": 0, "off_topic": 0,
↪ "overall_communication_impairment": 1}
Now score this transcript. Output only the
↪ JSON object with all 19 keys.
Prompt: {prompt}
Transcript: {transcript}