WorksComputer Science
What drives the emergence of symbolic mechanisms in large language models?
Abstract
Recent work has identified a set of emergent symbolic mechanisms that support abstract reasoning in large language models, but it remains unclear what factors drive the emergence of these mechanisms. To address this question, we trained neural networks from scratch on abstract sequence tasks, varying both architectural and data distributional factors, and investigated the effects of these factors on the emergence of symbolic mechanisms. Using representational, attentional, and causal mediation analyses, we first confirmed that transformer-based language models trained from scratch on our task developed symbolic mechanisms. We then found that, across multiple axes of data diversity, greater diversity produced stronger symbolic signatures. Surprisingly, the same signatures emerged in modified transformer architectures and even multilayer perceptrons, indicating that architectural inductive biases are not necessary for the development of symbolic mechanisms. We also observed consistent patterns in an entity-attribute binding task. Together, these findings identify data diversity, not architecture, as the primary driver of emergent symbolic computation in neural networks.
Introduction
Can large language models (LLMs) acquire human-like abstract reasoning, and what mechanisms might support this capacity? Many studies have reported that LLMs exhibit strong performance on abstract reasoning tasks [23], but debate persists over whether this reflects genuine structured abstraction or surface-level statistical patterns [12, 9]. Symbolic mechanisms have long been hypothesized as necessary for human-like abstraction [11], yet it has remained unclear whether and how such mechanisms might be implemented and learned by modern neural network models. Yang et al. [2025] provided direct evidence for a set of emergent symbolic mechanisms supporting abstract reasoning in pretrained LLMs, identifying a circuit composed of three types of attention-heads: symbol abstraction heads that map tokens to abstract role variables, symbolic induction heads that reason over those variables, and retrieval heads that map variables back to tokens. A key open question concerns the factors that drive the emergence of these mechanisms. Specifically, does the emergence of symbolic computation in LLMs depend on specific inductive biases of the transformer architecture from which these models are constructed, or does it depend on the distribution of the data used to train these models, or some combination of both?
In this work, we sought to address these questions by training neural networks from scratch on a controlled in-context abstract sequence task, similar to the task used to identify symbolic mechanisms in LLMs by Yang et al. [2025]. First, we confirmed that transformer language models trained from scratch on this task developed symbol abstraction heads, reproducing some of the key mechanistic signatures identified in pretrained LLMs. We then investigated the extent to which these signatures are driven by data distributional and architectural factors. We found that both mechanistic and behavioral signatures of symbolic computation scaled with data diversity, which we varied along four axes: the number of training examples, the vocabulary size, the number of distinct rules and statistically biased training data. On all four axes, greater diversity produced stronger symbolic signatures. We also found an interaction between data diversity and training dynamics: Models trained with small vocabularies and few rules initially solved the task using a non-symbolic solution based on first-order statistics, but eventually developed symbolic mechanisms with further training, following a pattern similar to grokking [18, 15]. By contrast, we found that the emergence of symbolic mechanisms did not depend on architectural inductive biases. The same mechanistic and behavioral signatures emerged in a set of transformer architectures modified to remove specific inductive biases relevant to symbolic computation and even in a multilayer perceptron (MLP) architecture with much weaker inductive biases. Finally, to test whether these findings extend beyond the abstract sequence task, we introduced a second task that required in-context binding of entities to attributes, where we found that some of the same data-diversity factors governed the emergence of the binding-ID mechanism identified by Feng and Steinhardt [ [4], and that a signature of this mechanism emerged in MLPs.
We make the following contributions:
We found that transformers trained from scratch on abstract sequence tasks developed a two-stage symbolic mechanism. Symbol abstraction heads inferred the rule, and a rotation mechanism applied it to the query (§ 2).
Varying density along four axes, we found that it governed whether this mechanism emerged, as shown by causal interventions and out-of-distribution generalization (§ 3).
Surprisingly, our experiments revealed that modified transformers, and even plain MLPs, exhibited the same signatures of symbolic computation (§ 4).
We further observed that on entity-attribute binding, data diversity likewise determined whether a binding-ID mechanism emerged, while architecture had no impact (§ 5).
Analysis of the Solution Learned by the Model
Architecture of the Learned Solution
We trained transformer language models to perform an abstract sequence task similar to the one used by Yang et al. [2025] to identify emergent symbolic mechanisms in pretrained LLMs. In our task, an in-context example governed by either an ABA or ABB rule, constructed from arbitrary tokens, was followed by a query consisting of two tokens. We trained models to predict the token that would complete the query example according to the rule inferred from the in-context example. We first identified the emergent circuit these models learned to solve the task (Figure 1). The circuit consisted of two major components: 1) symbol abstraction heads that computed invariant symbolic representations of the rule governing the in-context example, and 2) a rule-dependent rotation operation in the residual stream at the final position that gated the appropriate token (corresponding to A vs. B, depending on the rule) into the output subspace for the model. The following sections describe the evidence for these mechanisms.
Figure 1. Illustration of Symbolic Mechanisms Learned by Model. Problems consist of an in-context example governed by either an ABA or ABB (pictured) rule constructed with arbitrary tokens, followed by an incomplete query example that must be completed according to the same rule. Symbol abstraction heads process the in-context example, identifying the symbol (A or B) associated with the third token. Query tokens are copied into role-specific (A vs. B) subspaces of the residual stream at the final position, and a rule-dependent (ABA vs. ABB) rotation is performed to gate the appropriate token (corresponding to A vs. B) into the output subspace.
Evidence for Symbol Abstraction Heads
Symbol abstraction heads are attention heads that convert input tokens to invariant symbol-like representations, based on their relations to other tokens [Yang et al., 2025]. They are an emergent instance of the Abstractor architecture proposed by Altabaa et al. [ [2]. In the specific case of our task, these heads convert input tokens to invariant representations of abstract symbols (i.e., A or B) based on the relationship to the other tokens within an in-context example (e.g., if the third token is the same as the second token, it will be assigned the symbol B). We identified a set of heads in the first layer of the model that reproduced the key signatures (attentional, representational, and causal) of symbol abstraction heads found in LLMs.
Causal Mediation Analysis. To determine whether our model developed symbol abstraction heads, we performed causal mediation analysis (CMA) [22, 20] (Pearl 2022), patching internal activations between contexts with different rules (ABA vs. ABB) and measuring the effect on the model’s outputs. Importantly, context pairs were created in which the rule differed, but the final token in the in-context example was the same (e.g., ‘tok13 tok49 tok49’, instantiating an ABB rule vs. ‘tok49 tok13 tok49’, instantiating an ABA rule, but both ending with ‘tok49’), ensuring that any effect observed for patching activations at this token position would reflect encoding of an abstract symbol rather than merely encoding input tokens. When patching activations for the output of all attention heads at a particular layer, we found an effect concentrated in layer 1 for the expected position, which is the third item (A1/B1) in the in-context example (Figure 2a)1. When patching activations for the outputs of individual attention heads at this token position, we found a set of three attention heads that produced the expected effect (Figure 2b), thus identifying these heads as candidate symbol abstraction heads.

Figure 2. Evidence for Symbol Abstraction Heads and Rotation Mechanism Learned by Model. (a) Layer-wise causal mediation analysis localizes symbolic encoding to layer 1. (b) Head-wise causal mediation analysis identifies symbol abstraction heads. (c) Query–key attention of the symbol abstraction heads. (d) RSA confirms that identified heads encode abstract role identity rather than token surface form. (e)–(f) Residual-stream PCA reveals a layer-wise rotation from abstract role representation to output prediction.
Attention Analysis. To further confirm the presence of symbol abstraction heads, we analyzed the attention patterns for the identified heads. Consistent with the predicted behavior of symbol abstraction heads, we found that attention was directed from the final token in each in-context example to the previous token that shared the same symbol (i.e., from the third token to the first token for ABA rules, and from the third token to the second token for ABB rules) (Figure 2c).
Representational Similarity Analysis. To test whether the identified heads encode truly invariant symbolic representations rather than token identity, we performed representational similarity analysis (RSA) [8]. For each identified head, we computed the pairwise similarity between the outputs of that head across pairs of contexts, constructing a similarity matrix reflecting the pattern of similarity across many contexts. Figure 2d shows a weighted average of this matrix, weighted by the CMA scores used to identify these heads. The results confirmed that the outputs of these heads had a high degree of similarity for pairs of contexts that shared the same rule (upper left and lower right quadrants) regardless of the specific tokens used to instantiate those rules, confirming that these heads encoded abstract rules invariantly. The identified attention heads therefore reproduced the key signatures of symbol abstraction heads found in pretrained LLMs.
Evidence for Rule-Dependent Subspace Rotation
After the rule is inferred from the in-context example, it must be applied to the query to generate the final token. Rather than employing symbolic induction heads and retrieval heads as in pretrained LLMs [Yang et al., 2025], we found that this was accomplished in our model via a rule-dependent subspace rotation in the residual stream at the final position. Specifically, the A and B tokens for the query example were stored in role-specific subspaces in early layers, and these subspaces were then rotated in a rule-dependent manner (e.g., the subspaces are rotated only for ABA rules, not for ABB rules) such that the appropriate completion token (A or B, depending on the rule) is available to be read out from a particular subspace in the later layers of the model.
Shared PCA. We confirmed the presence of this rotation-based strategy by performing PCA across all layers jointly on residual-stream activations at the final position, thus constructing a shared subspace in which layer-wise trajectories can be compared directly. Problems were constructed so that the query token was drawn from a fixed pair of tokens, allowing token identity and abstract role (A vs. B) to be dissociated across trials. Figure 2e and Figure 2f show resulting trajectories for ABA and ABB rules, with color encoding layer depth (blue = early, red = late) and marker shape encoding token identity. In early layers, representations clustered by token identity regardless of rule. Across successive layers, the trajectories followed an arc-shaped path in the shared PCA space, converging by the final layer to a configuration that is separable by the correct output token rather than by abstract role. These results are consistent with a near-linear rotation that progressively remaps the abstract role representation into the appropriate output token embedding, depending on the rule.
Decoding role information. We trained linear decoders on the residual stream at the final sequence position and evaluated whether they generalized across rules (ABA to ABB and vice versa). In early layers, decoders transferred based on symbol identity, independent of the rule, whereas in later layers, they transferred based on the rule-dependent correct answer, which differs between the two rules. This transition occurred in the middle layers, consistent with the PCA trajectories. Further details are provided in Appendix § 10.
Data Diversity Drives the Emergence of Symbolic Mechanisms
Experimental Setup
We varied data diversity along four axes to evaluate its impact on the emergence of symbol abstraction heads. All models used the same hyperparameters as GPT-2 Small [19]. The first data diversity axis was the number of training examples (from 200 to 200k). The second was role-token bias (0% to 100%), defined as the fraction of vocabulary tokens consistently associated with a single role (A or B) during training. These associations were then reversed at test time. The third was vocabulary size ( from 50 to 10k). The fourth was the number of distinct rules (2 or 10). Across all diversity axes, we hypothesized that increased diversity would reduce the viability of token-level or statistical shortcuts and would therefore encourage models to learn a solution based on genuine relational structure. To compare the emergence of symbolic mechanisms across these settings, we used the concept intervention described below and measured its efficacy. For role-token bias, we additionally evaluated out-of-distribution (OOD) generalization behaviorally. The specific parameter values, data construction procedures, and training details are provided in Appendix A.
Concept Interventions
Let denote the residual stream at position and layer . Let be the third token of the in-context example, which is the A symbol in an ABA example and the B symbol in an ABB example. For each layer we define the role embeddings as mean activations at ,
estimated from matched prompts of each type. To swap the role on a test prompt, we change the activation at only. For an ABA prompt,
and for an ABB prompt we exchange and . When applied to an ABA sequence, the intervention should shift the model’s predictions toward those consistent with ABB. We applied the intervention at the third position of the in-context example, the token that determines whether the example follows ABA or ABB. We measured the flip rate, the fraction of trials in which the intervened model outputs the answer that the opposite rule would give. We used the flip rate of this intervention as our primary measure of the extent to which the model possesses invariant, causally relevant symbolic representations. (Figure 3a).
Figure 3. Assessing the Impact of Data Diversity on Emergent Symbol Processing. (a) Concept intervention pipeline. Role embeddings and are estimated from residual activations and used to swap the role at a target position. The flip rate is the fraction of prompts whose prediction switches to the answer of the opposite rule. (b) Flip rate increases with the number of unique training examples, (c) decreases as role-token bias increases, and (d) increases with vocabulary size and (e) the number of rules. Error bars show the standard deviation across 3 seeds.
We began by varying the number of unique training examples. Flip rate was near 0% with 200 examples and reached 100% from 100k examples (Figure 3b). In these low-diversity conditions the models did not generalize: training accuracy converged, but test accuracy did not (Appendix Figure 8). To test whether symbolic mechanisms depend on diversity even when models do generalize, we turned to role-token bias. During training, a fraction of the vocabulary was role-exclusive: half of these tokens appeared only as A and half only as B, while the remaining tokens can fill either role. At a token’s identity fully determines its role, so the training distribution can be solved by learning token–role associations alone. At test time, we reversed the roles of the exclusive tokens. All models generalized in-distribution at every bias level, including 100% bias, with in-distribution accuracy near ceiling (Appendix Figure 9). Flip rate nevertheless fell from 98% at 0% bias to near 0% at 100% bias (Figure 3c). For vocabulary size, flip rate was 0% at , 5% at , and close to 100% from upward (Figure 3d). At , models generalized but had a low flip rate (consistent with a lack of symbolic mechanisms), so we asked whether training models on more rules would induce a higher flip rate at this vocabulary size. With , flip rate was 5% with 2 rules and 95% with 10 rules (Figure 3e).
Finally, we found an interaction between data diversity and training dynamics. For all experiments described above, we measured flip rate at the point in training when models first reached an accuracy threshold of 95%. For models trained on a small vocabulary size ( and ), we found that training models well past the point of convergence eventually led to a sharp increase in the flip rate, consistent with an initial period where models employed a non-symbolic but generalizable solution followed by a later period in which these models eventually developed symbolic mechanisms (Appendix C). Together, these results show that greater data diversity promotes the efficient development of symbolic mechanisms.
Out-of-Distribution Generalization
For models trained on different levels of role-token bias, we also measured accuracy on test prompts in which the token–role assignments from training were reversed (Figure 4). A model that relies on first-order token statistics should fail on these tests, because the associations it learned now lead to the wrong answer. By contrast, a model that has learned the rule should be unaffected. Consistent with this, we found that models trained on a small vocabulary size (, 2 rules), generalized well when trained with 0% role-token bias (i.e., they generalized well in-distribution), but were unable to generalize when trained with 100% role-token bias (i.e., they could not generalize out-of-distribution). By contrast, models trained with a larger vocabulary size generalized well at all bias levels (i.e., they generalized well both in-distribution and out-of-distribution). These results suggest that, in addition to the mechanistic signature tracked by our concept interventions, data diversity also drove out-of-distribution generalization at the behavioral level, thus conferring a key benefit of symbolic computation.

Figure 4. Out-of-distribution generalization
Symbolic Mechanisms Do Not Depend on Architectural Inductive Biases
We next asked whether symbolic mechanisms depend on the inductive biases of the transformer architecture. We tested this by evaluating three modified transformers, each ablating a specific architectural feature that seems relevant to symbolic computation, described below.
Modified Transformers
There are several aspects of the transformer architecture that seem relevant to its ability to implement symbolic computation, including 1) the presence of discrete embeddings with shared parameters, potentially enabling a form of compositionality, 2) the use of distinct embeddings for keys, queries, and values, potentially enabling a form of indirection (i.e., the use of one variable to refer to another), and 3) the use of inner products in the attention operation, which are well suited to encoding abstract relations [24] (Altabaa and Lafferty 2024). We evaluated three architectural modifications against a standard transformer control, all trained at , to test whether specific inductive biases of the transformer architecture are necessary for the emergence of symbolic mechanisms (Figure 5a). The first modified architecture removed weight sharing across sequence positions (No pos. weight sharing), ablating the shared parameters that may enable compositional representations. The second constrained the query, key, and value projection matrices to be identical (Shared Q/K/V), removing the separation that may encourage transformers to implement indirection. The third replaced dot-product attention with a learned scalar attention score computed by a small MLP over concatenated key and query representations (MLP attention), removing an inductive bias that may be helpful for learning abstract relations.
Figure 5. Architectural inductive biases do not determine the emergence of symbolic mechanisms. (a) Flip rate at for Transformers with specific inductive biases removed, and for MLP models. Error bars show the standard deviation across 3 seeds. (b) Flip rate decreases with role-token bias in the same way for the pure MLP and the Transformer. (c) MLP models show the same dependence of out-of-distribution (reversed-role) accuracy on vocabulary size and role-token bias as transformers: with high bias and a small vocabulary they fail the reversed test, and reversed accuracy recovers as vocabulary grows. (d) Test accuracy over training.
fig:5a shows the flip rate for our concept intervention at the third token position, measured at the final training checkpoint. Flip rates were high across all of the architectural variants: the control Transformer, the variant without position-specific weights, and the variant with shared Q/K/V all reached 100%, and the MLP attention variant reached about 90% on average, with more variability across runs. None of these specific Transformer features is therefore necessary for symbolic mechanisms to emerge. This raises the question: do we need attention at all?
MLP Models
To test whether symbolic mechanisms require attention at all, we trained a plain MLP with no attention. The MLP received as input the concatenation of all token embeddings in the sequence. Residual connections were included between layers, as in the Transformer. The network had 4 hidden layers, and the output layer projected from to the vocabulary using only the slice of the hidden state corresponding to the final sequence position (Appendix A.2). This MLP reached a flip rate of 100% in all five runs (Figure 5a), indicating that symbolic mechanisms can emerge in a model without attention.
Concept intervention results. We then asked whether symbolic mechanisms in the MLP depend on data diversity in the same way as they do in the Transformer. Figure 5b shows flip rate as a function of role-token bias for both models, using the manipulation described in § 3. Flip rate decreased monotonically with bias for both the Transformer and the MLP, indicating that the emergence of symbolic mechanisms is driven by data diversity for both Transformers and MLPs.
Out-of-distribution generalization. MLP models also showed a qualitatively similar pattern of out-of-distribution generalization (Figure 5c). At , accuracy on reversed-role prompts dropped as role-token bias increased, whereas at , generalization was perfect across all bias levels. These results indicate that out-of-distribution generalization was driven by data diversity in MLPs just as it was in Transformers.
Training efficiency. Finally, we investigated whether transformers and MLPs show differences in the efficiency with which they developed symbolic mechanisms over the course of training. Figure 5d shows accuracy over training for transformer and MLP models at . The transformer reached ceiling after roughly 20k steps and the MLP after roughly 40k. This suggests that, while data diversity is the main driver of symbolic computation, the transformer offers a modest training efficiency advantage, accelerating convergence without changing the nature of the final solution.
Data Diversity Also Drives the Emergence of Entity-Attribute Binding
To test whether the effect of data diversity extends beyond the ABA/ABB task, we introduced a second task that required entity–attribute binding. Each input contains two entity–attribute pairs followed by a query entity, , and the model must output the attribute bound to the queried entity: if , and if . For example, the sequence Alice friendly Bob tall Bob has the answer tall. In the experiments, entities and attributes are arbitrary tokens, and each sequence uses a new pairing, so the answer cannot be predicted from token identity alone. That is, the model must link each entity to the attribute that follows it in-context. The task is built around the binding ID mechanism identified by Feng and Steinhardt [2024], in which a language model binds an entity to its attribute by giving both the same abstract identifier, independent of token identity. Here, this means that and would be bound (via vector addition) to one identifier and and would be bound to another. The model answers the query by first looking up the identifier of and then retrieving the attribute that’s bound to this identifier. As in § 2, we performed causal mediation analysis and representational similarity analysis to confirm that the binding task also used invariant, symbol-like binding-ID representations rather than token-specific ones; full results are provided in Appendix E.
Mirroring the role-token bias results in § 3, increasing role-token bias degraded the binding-ID flip rate, from 100% at 0% bias to 44% at 100% bias (Figure 6a). Consistent with the vocabulary-size results in § 3, the flip rate also depended strongly on vocabulary size, with a flip rate of 23% at , and flip rates at or near ceiling for all larger vocabularies (Figure 6b). As in § 4, this effect did not depend on the architectural inductive biases of the Transformer. Both a standard Transformer and an MLP displayed a binding-ID flip rate of 100% (Figure 6c). Together, these results indicate that the effect of data diversity on the emergence of symbolic mechanisms extends beyond the ABA/ABB rule task to a qualitatively different abstract reasoning task.

Figure 6. Data diversity drives the emergence of symbolic binding mechanisms. (a) Increasing role-token bias results in decreased binding-ID flip rates. (b) The binding-ID flip rate increases with vocabulary size. (c) Both MLPs and Transformers display high flip rates, indicating that both have developed symbolic binding mechanisms.
Related Work
Abstract reasoning in neural networks. A central debate in cognitive science and AI concerns whether neural networks perform genuine structured abstraction or simply exploit surface-level statistical patterns [11, 12, 9, 5]. Webb et al. [2023] demonstrated emergent analogical reasoning in LLMs, though its mechanisms remained unclear. Our work builds on Yang et al. [2025], by asking what factors drive the emergence of a symbolic circuit for abstract reasoning in LLMs.
Mechanistic interpretability and in-context learning mechanisms. Our methods draw on a toolkit for reverse-engineering neural network computations. Causal mediation analysis [13, 22, 20] (Pearl 2022) localizes computations to components; we use it to identify symbol abstraction heads and characterize their causal role. Representational similarity analysis [8] complements this by distinguishing abstract from token-specific encodings. Our concept intervention method extends the activation patching paradigm of Meng et al. [2022] and the function vector framework of Todd et al. [2024].
Relational inductive biases and inner products. A complementary line of work asks how architectural design shapes learned representations. The relational bottleneck hypothesis [24] proposes that constraining networks to process relational rather than object-level information encourages symbol-like representations. Abstractors [2] implement this via relational cross-attention. Inner product spaces are well-suited to encoding abstract relations [1], suggesting that the attention mechanism may be a useful inductive bias for symbolic computation. Our finding that MLPs developed the same signatures qualifies this: inner products may help (especially in terms of training efficiency) but are not necessary given sufficient data diversity.
Data and curriculum effects. A growing body of evidence shows that training data distributions shape learned representations and strategies. Chan et al. [2022] showed that in-context learning emerges from distributional properties like burstiness and rare-class frequency. Uselis et al. [2025] similarly found that compositional generalization in vision models is driven by data diversity, with greater combinatorial coverage inducing a linearly factored representation. Data diversity has also been linked to compositional generalization [7]. We extend this to the mechanistic level: data diversity shapes not only behavioral generalization but the qualitative nature of the underlying mechanisms, independently of architecture.
Training dynamics. Language models often acquire different abilities at different points in training, sometimes abruptly. In-context learning emerges together with induction heads in a sudden phase change [16, 14], and in-context learning develops in discrete stages [6]. Generalization can also follow memorization after long training [10], as in grokking [18, 15]. Our low-diversity models showed a similar ordering, first generalizing in-distribution via a statistical solution, and then developing symbolic mechanisms only after a long delay. How such phases relate to the emergence of specific symbolic mechanisms remains to be tested.
Conclusion
What determines whether a neural network develops symbolic mechanisms? We found that data diversity consistently promoted symbolic mechanisms across four axes, while architectural inductive biases were not necessary for their emergence. These findings extend to a second abstract reasoning task, suggesting that data diversity plays a central role in determining when and how symbolic mechanisms emerge.
Limitations and Future Work. The solution learned by our models partially reproduced the symbolic mechanisms observed in pretrained LLMs. Our models developed symbol abstraction heads, which LLMs use to infer abstract rules from in-context examples, but not the symbolic induction heads and retrieval heads that LLMs use to apply these rules to new examples. We hypothesize that greater rule diversity (in addition to token diversity) and rule uncertainty may be needed to induce these additional mechanisms, but this remains to be tested. Rather than attempting to reproduce the full suite of mechanisms employed by large-scale models, we chose symbol abstraction heads as a testbed for the factors that drive emergent symbolic computation. Two further limitations concern training dynamics and architecture. We trained small models from scratch on controlled tasks and found that low-diversity models first learned a statistical solution and developed symbolic mechanisms only after a long delay. How such dynamics unfold in large models trained under much more complex regimes, with mixed data, long training, and post-training, is unclear. Our concept interventions also showed that MLPs reached the same flip rates as Transformers, but how symbolic mechanisms are actually implemented in an MLP remains an open question.
Acknowledgments and Disclosure of Funding
This work was supported by the Courtois Chair in Fundamental Research - V (Neuroscience) at the Université de Montréal awarded to T.W. We are grateful to Mila (mila.quebec) and Princeton University for the compute resources, software, and technical help that made our experiments possible.
Experimental Details
Task and data
ABA/ABB. Each prompt contains one in-context example that follows a rule, followed by a query that the model must complete under the same rule. The input is five tokens, A1 B1 A1 A2 B2 (ABA) or A1 B1 B1 A2 B2 (ABB), and the model predicts the sixth token (A2 for ABA, B2 for ABB). There is no separator or start token. The four tokens A1, B1, A2, B2 are distinct and are drawn without replacement from the vocabulary tok_1, …, tok_V, freshly for every prompt, so surface identity carries no information about the rule. The two rules are equally likely.
Binding. A prompt is , where are entities, are attributes, is one of the two entities (chosen with probability ), and the target is the attribute paired with . The vocabulary has about 20,000 tokens.
Dataset sizes. Table 1 lists the training and evaluation sets used in each experiment.
| Experiment | Vocabulary | Train prompts | Validation prompts |
| Vocabulary size (ABA/ABB) | 50–10,000 | 2,000,000 | 2,000 |
| Number of training samples | 10,000 | 200–200,000 | 500 |
| Number of rules | 100 | 2,000,000 | 5,000 |
| Role-token bias | 100–20k | 5,000,000 | 5,000 |
| Architecture variants | 10,000 | 5,000,000 | 10,000 |
| Binding (role-token bias) | 20,000 | 5,000,000 | 20,000 |
Table 1. Datasets. “Prompts” counts distinct training prompts.
The role-token bias datasets additionally contain a standard test set (no role-exclusive tokens) and a reversed-role test set of 5,000 prompts each.
Models
All models are trained from scratch. Tokens are mapped to ids by a fixed lookup table with a few special tokens (pad, unk, bos, eos, sep).
Transformer. GPT-2 as implemented in HuggingFace transformers (4.56.2) with the GPT-2-small configuration: 12 layers, 12 attention heads, hidden size 768, MLP width 3072, GELU, pre-LayerNorm, learned absolute position embeddings, tied input and output embeddings, and dropout 0.1 on embeddings, attention and residual connections. Weights use the default GPT-2 initialization. The model has about 86M parameters at and 94M at .
Architecture variants (Section 4). All variants keep the Transformer above except for the one change listed.
No positional weight sharing. Every input position has its own MLP weights (
c_fc,c_proj) in every block; attention weights stay shared across positions.Shared Q/K/V. A single projection per layer produces queries, keys and values (), which makes the attention scores symmetric.
MLP attention. The dot-product score of each head is replaced by a two-layer network applied to every (query, key) pair, , with hidden size equal to the head dimension (64). The softmax, causal mask, value and output projections are unchanged.
Pure MLP (no attention). The input is padded to 16 positions. Token and learned position embeddings (dimension 64) are summed and the sequence is flattened into a 1024-dimensional vector. Four residual blocks follow, each LayerNorm, Linear , GELU, dropout 0.1, Linear ; then a final LayerNorm, a reshape to 16 × 64, and a tied unembedding. There is no causal mask, so every position can see every other; the target token is never given as input, and the prediction is read out at the last context position. The model has about 17M parameters.
Binding models. The Transformer is the same GPT-2-small configuration with a context length of 10. The MLP is the pure MLP above with 8 blocks instead of 4 (about 35M parameters).
Training
All models are trained with the HuggingFace Trainer: AdamW (, , ) with no weight decay, gradient-norm clipping at 1.0, fp16 mixed precision, a linear warmup over the first 10% of steps followed by cosine decay to zero, and per-device batch size 24 on one GPU. The peak learning rate is in all experiments except the MLP-attention variant (, two GPUs with per-device batch 24, i.e. an effective batch of 48). Dropout is 0.1.
Intervention and flip rate
Role vectors. For every layer we take the residual stream after block at the third token of the in-context example (where the ABA and ABB examples differ). The ABA vector is the mean of this activation over ABA prompts, and is the mean over ABB prompts. The ABA and ABB prompts are built from the same tokens (each of token pairs is used in both orders, and the in-context and query tokens are shared), so the two sets differ only in the rule. We use in all reported results, i.e. 1,000 prompts per rule.
Intervention. On a test ABA prompt we add to the residual stream at that position, at every layer simultaneously (and the reverse for ABB prompts), this is equivalent to replacing the rule component by while keeping the token-specific component. All tokens are unchanged. For the pure MLP, which has no per-position streams, the vector is added to the flattened hidden vector at every block. For the 5-token rules, the vector is estimated and applied at the last token of the in-context example (position 4), where ABABA and ABABB first differ.
Flip rate. A test prompt flips if the model’s prediction after the intervention exactly equals the correct answer under the opposite rule for that prompt. Test prompts use fresh tokens, 200 per rule for the reported results (400 prompts in total) with duplicates removed. For the out of distribution bias experiments the flip rate is measured on reversed-role prompts, using role vectors estimated from role-respecting prompts of the same bias level.
Uncertainty. Points are means over 3-5 seeds; error bars are standard deviations (rules, vocabulary size) or the min–max range (bias, architectures), as stated in each caption.
Data-diversity manipulations
Vocabulary size. The number of distinct tokens in the training dataset, .
Number of training samples. The number of distinct training prompts, at .
Number of rules. For the rules are length-5 patterns: ABABA, ABABB, ABCBA, ABCBB, ABCCC, AAABA, AAABB, AABAA, AABAB, AABBA, at . The point is the plain ABA/ABB task at .
Role-token bias. A fraction of the vocabulary is role-exclusive during training, split evenly into a set that only fills role A and a set that only fills role B; the remaining tokens can fill either role. At test time the two role-exclusive sets are swapped (the reversed-role test) while the rule is unchanged. For the binding task the exclusive sets are entity-only and attribute-only tokens.
Convergence definitions
Accuracy is evaluated on the validation set every 250 steps and flip rate at every saved checkpoint. A quantity converges at the first checkpoint at which its rolling mean over three checkpoints reaches the threshold and at least 90% of the remaining checkpoints stay above 95% of the threshold. Accuracy convergence uses a threshold of 95% in the vocabulary-size and number-of-rules plots. Flip-rate convergence uses 90%, and runs that never reach it are excluded from the ratio plots. The flip rate at accuracy convergence is read from the checkpoint nearest to the accuracy-convergence step.
Decoder Experiments
To test the account in § 2.3 that early layers carry abstract symbol identity while later layers carry the correct answer, we trained linear probes on the residual stream at the final query position of a model (vocabulary size 2000). The query always used the same two tokens, so token identity was constant across trials and only (i) which of the two tokens plays role A and (ii) which rule is active vary. Each trial has one in-context example with fresh tokens. We used 400 trials per rule. For each layer, we trained a logistic-regression probe (L2 penalty, ) on raw activations from one rule (ABA or ABB) to predict whether the first query token is the correct answer, then evaluated the same probe on the other rule against two different labels:
symbol generalization: whether the first query token has the role the probe was trained to detect (role A for an ABA-trained probe, role B for an ABB-trained probe)
correct/incorrect generalization: whether the first query token is the correct answer, which flips meaning between the two rules.
Within a single rule the two labels coincide, so only the cross-rule test separates them. Chance is 50%. A probe that tracks role identity scores 100% on the first label and 0% on the second, because the correct token changes role between the rules; a probe that tracks the answer scores the reverse. We average over both training directions (ABA→ABB and ABB→ABA).
Symbol generalization was 100% in layers 0–4 and fell to 71% and 27% in layers 5 and 6 and 0% afterwards, while correct/incorrect generalization rose from 0% (layers 0–4) through 29% and 73% (layers 5 and 6) to 100% from layer 7 onward (Figure 7). Because the cross-rule probe is evaluated on a rule it was never trained on, this is evidence that early layers represent role identity abstractly and later layers represent the answer itself.
Figure 7. Cross-rule decoding by layer. Probes trained on one rule and tested on the other. Layers 0–4 decode symbol identity perfectly; their correct/incorrect score is 0% because the same direction points to the wrong token under the new rule. Layers 7–11 show the reverse.
Training Dynamics Across Data Diversity Levels
This section gives the training dynamics behind the data-diversity results in the main text. For each level of each diversity axis, we track task accuracy and the third-token flip rate at every saved checkpoint. We include all four data-diversity axes: number of training samples (Figure 8), role-token bias (Figure 9), vocabulary size (Figure 10), and number of rules (Figure 11). Unless stated otherwise, accuracy is shown in black and flip rate in blue; lines are the mean over seeds and the shaded band spans the minimum and maximum over seeds. Accuracy is measured on freshly generated test prompts. Figure 12 summarizes the same data as the ratio of the flip-rate convergence step to the accuracy convergence step.
Figure 8. Timecourses by number of training samples (vocabulary size 10,000). Flip rate is measured on in-distribution prompts.
Figure 9. Timecourses by role-token bias (vocabulary size 100). Solid black: in-distribution accuracy. Dashed gray: accuracy on reversed-role (out-of-distribution) prompts. Blue: flip rate on reversed-role prompts. In-distribution accuracy is near ceiling at every bias level, while reversed accuracy and reversed flip rate fall as bias increases.
Figure 10. Timecourses by vocabulary size. Shaded boxes and insets zoom on the first 30,000 steps for the two smallest vocabularies. At small vocabularies () flip rate rises well after accuracy has converged; from upward the two curves rise together.
Figure 11. Timecourses by number of rules (vocabulary size 100). The 2-rule condition is the plain ABA/ABB task. The 10-rule condition uses ten length-5 rules.
Figure 12. Ratio of flip-rate to accuracy convergence step across diversity levels. A ratio of 1 means the mechanism converges together with accuracy. Points are means over runs. The ratio is about 1 for vocabulary sizes and for the number-of-rules axis, and larger for the smallest vocabularies.
Effect of Model Size
To test whether the delayed emergence of the symbolic mechanism depends on model capacity, we compared the default GPT-2-small model used throughout the paper (12 layers, 12 heads, hidden size 768; 85.9M parameters) with a much smaller model (2 layers, 2 heads, hidden size 128; 0.53M parameters) on the ABA/ABB task at vocabulary size 50.
Smaller models skip the statistical phase. At the step where accuracy converges, the 2L2H model already had a flip rate of 96.2% 1.3%, whereas the 12L12H model had a flip rate of 0% (Figure 13). The 12L12H model reached 95% accuracy much earlier, but its flip rate only rose about 6.5× later; in the 2L2H model the two converge together. Thus the smaller model does not show the initial statistical phase at all: it appears to go directly to symbolic signatures, although it takes longer, in training steps, to get there than the base model takes to solve the task.
Figure 13. Effect of model size on the emergence of the symbolic mechanism (vocabulary size 50). Left: flip rate at the step where accuracy first converges, mean standard deviation over seeds. Right: training dynamics of the two models: accuracy and flip rate over training.
Capacity versus diversity. We think this fits naturally alongside the vocabulary-size result. Reducing capacity, by using a smaller model, removes the option of a first-order statistical solution outright, so there is no memorization phase to pass through. Reducing diversity, by using a smaller vocabulary at fixed model size, merely makes the statistical solution viable for longer before the pressure toward the compressed, symbolic solution eventually wins out.
Mechanistic Analysis of Entity-Attribute Binding
We tested whether the entity-attribute binding task (§ 5) is solved with binding IDs, mirroring the analyses used for the ABA/ABB task in § 2: an intervention, a causal mediation analysis, and a representational similarity analysis.
Binding-ID intervention
Each input lists two entity-attribute pairs followed by a query entity, and the model must output the attribute bound to it. Under the binding-ID hypothesis, the model tags each pair with an ID (ID0 or ID1) and answers by matching the query’s ID. We therefore intervene at the query position: for a query on the first entity we subtract the mean residual vector associated with ID0 and add the one associated with ID1 (and vice versa for the second entity). The two ID vectors are the mean residual-stream vector at the query position over many prompts in which the query is the first-pair entity (ID0) or the second-pair entity (ID1), estimated on one half of the entities and tested on the held-out half, so the test queries are novel tokens. A flip means the model now outputs the attribute bound to the other entity.
Causal mediation analysis
We use two patching experiments at the query position, each producing a score for every layer and token position and for every layer and attention head. In Stage 1 (binding-ID retrieval), the base prompt is and the patched prompt swaps which slot each pair occupies, . Both have the same correct answer, , because the entity-attribute binding is preserved; a head that reads the binding ID transfers the base prompt’s slot and makes the patched prompt output . In Stage 2 (token retrieval), the patched prompt swaps the attributes in place, , so the correct answer changes; a head that reads the attribute token at the queried slot transfers the base prompt’s attribute. A head’s score is the change in the logit difference between the causal answer and the original answer when its output at the query position is patched from the base into the patched prompt.
Figure 14. Causal mediation analysis for binding. Top: Stage 1 (binding-ID retrieval). Bottom: Stage 2 (token retrieval). Left: layer × token position (patching attention and MLP output at one position). Right: layer × head (patching one head’s output at the query position). Layers are numbered from 0.
For Stage 1 the effect is concentrated in layer 0 at the query position, and at the head level in two heads (layer 0, heads 3 and 6). For Stage 2 the effect is much more distributed: the head-level scores are spread over layers 2–6, strongest in layers 3–5 (peak at layer 5, head 7), and the layer-wise map additionally shows a large effect in layer 0 at the attribute positions.
Representational similarity analysis
To test what the identified heads represent, we computed per-head representational similarity at the query position. Each problem is rendered twice with the same tokens and the query fixed to the first entity: (the query’s pair is in slot 0, ID0) and (slot 1, ID1). At the query position the token is identical across the pair while the binding ID differs, so a head that carries the ID gives a two-block similarity matrix (ID0 against ID0 and ID1 against ID1 similar, across-ID dissimilar), whereas a head that carries token identity gives a diagonal across the two blocks. We collected the head outputs for 20 problems (40 prompts), computed pairwise cosine similarities, and averaged the matrices over heads weighted by their CMA scores, keeping only heads whose score exceeds the mean by more than 0.5 standard deviations.
The Stage 1 matrix had a strong two-block structure, which indicates that the Stage 1 heads encode the binding ID rather than token identity. The Stage 2 matrix keeps a block structure but also shows a diagonal linking the same token across the two IDs.
Figure 15. CMA-weighted representational similarity for binding. Average cosine-similarity matrices over heads with CMA , ordered by binding ID (ID0 prompts first, ID1 second). Left: Stage 1 (binding-ID retrieval). Right: Stage 2 (token retrieval).
References
- [1]Awni Altabaa and John Lafferty. Approximation of relation functions and attention mechanisms. arXiv preprint arXiv:2402.08856, 2024.arxiv.org/abs/2402.08856
- [2]Awni Altabaa, Taylor Webb, Jonathan Cohen, and John Lafferty. Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers. In International Conference on Learning Representations, volume 2024, pages 24597–24613, 2024.arxiv.org/abs/2304.00195
- [4]Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.17191.
- [5]Thomas L. Griffiths, Brenden M. Lake, R. Thomas McCoy, Ellie Pavlick, and Taylor W. Webb. Whither symbols in the era of advanced neural networks? Trends in Cognitive Sciences, 2026. doi: 10.1016/j.tics.2026.02.003.
- [6]Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape degeneracy and stagewise development in transformers. Transactions on Machine Learning Research, 2025. First posted as “The Developmental Landscape of In-Context Learning”.arxiv.org/abs/2402.02364
- [7]Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795, 2020.arxiv.org/abs/1908.08351
- [8]Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.DOI
- [9]Martha Lewis and Melanie Mitchell. Evaluating the robustness of analogical reasoning in large language models. arXiv preprint arXiv:2411.14215, 2024.arxiv.org/abs/2411.14215
- [10]Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, and Blake A. Richards. Tracing the representation geometry of language models from pretraining to post-training. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2509.23024.arxiv.org/abs/2509.23024
- [11]Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. The MIT Press, 2001.
- [12]R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322402121, 2024.DOI
- [13]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022.arxiv.org/abs/2202.05262
- [14]Gautam Reddy Nallamala. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In International conference on learning representations, volume 2024, pages 19226–19239, 2024.
- [15]Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.arxiv.org/abs/2301.05217
- [16]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.arxiv.org/abs/2209.11895
- [18]Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.arxiv.org/abs/2201.02177
- [19]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [20]Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. In International conference on learning representations, volume 2024, pages 17282–17333, 2024.arxiv.org/abs/2310.15213
- [22]Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.arxiv.org/abs/2211.00593
- [23]Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541, 2023.DOI
- [24]Taylor W Webb, Steven M Frankland, Awni Altabaa, Simon Segert, Kamesh Krishnamurthy, Declan Campbell, Jacob Russin, Tyler Gillanza, Randall O’Reilly, John Lafferty, et al. The relational bottleneck as an inductive bias for efficient abstraction. Trends in Cognitive Sciences, 28(9):829–843, 2024.DOI











