WorksComputer Science
Harmonic Circuit: Chord Progression Lies in a Sparse Principal Information Flow in Symbolic Music Models
Abstract
Mechanistic Interpretability (MI) seeks to decode neural network computations into human-understandable logic. A core methodology in MI is circuit discovery: identifying the information flow in sub-networks that drive specific model behaviors. While circuit discovery has matured within Natural Language Processing (NLP), the interpretable circuits in music models remain largely unexplored. In this work, we identify a music-functional circuit within two symbolic music generative models: ChatMusician and MuPT. The music function we consider is the prediction of the terminal tonic root in a classical harmonic progression pattern . We use the Auto-Emergence algorithm to extract a principal sparse sub-network as the identified circuit. Validation via causal intervention confirms that this identified circuit is a primary driver of harmonic progression behaviors in the symbolic music models. To our knowledge, this represents the first systematic circuit-discovery study conducted within the music domain. Our demo page is available at: https://ukiyo23rui.github.io/auto-emergence-demo/.
Introduction
AI models have become increasingly capable in both music understanding and generation [27, 23, 6, 28, 24], driven by scaling laws and the rapid development of training data. However, there is limited understanding of how the internal components in these models coordinate to interpret and process complex musical patterns, such as harmonic progressions.
Mechanistic interpretability techniques aim at translating black-box processes inside neural networks, e.g., learned features and computations, into algorithms and concepts that are interpretable to humans [1, 21, 2, 22, 9, 29], advancing the development of AI systems that are trustworthy and beneficial to people. In mechanistic interpretability, a circuit is defined as a human-interpretable sub-network of the model that is responsible for some specific behavior [22].
In the music domain, several works have made meaningful attempts in mechanistic interpretability. However, almost all of them remain confined to the representational level of explanation, such as probing [8, 13] or SAEs [19]. To our knowledge, no prior work has attempted to discover circuits, which are important for understanding how models organize their internal components to dynamically interact with the naturally structured music data.
An intuitive question, then, is what kind of musical behavior naturally lends itself to a circuit, implying the emergence of a structured sub-network within the model. We argue that the harmonic chord progression is a strong candidate because it is governed by an explicit structure, which may lead the model to form internally differentiated components for handling distinct harmonic roles.
A useful analogy is the Indirect Object Identification (IOI)1 task [22], where a language model needs to distinguish the subject and the object in the sentence. Harmonic progression provides a similar syntax-guided form of dependency in music. In the familiar functional pattern (this notation is explained in § 2), the first tonic establishes the center, the dominant builds tension, and the final tonic resolves it. Intuitively, to predict the terminal resolution correctly, the model needs to preserve and route harmonically relevant information across positions. Therefore, this harmonic-resolution behavior is a suitable target for circuit discovery.
We conduct circuit discovery using music pieces with an chord progression extracted from Nottingham [10] and use a subset of the Bach Chorales Melody–Harmony Corpus (BCMH), selected by the same chord criterion, to test polyphonic generalization [17, 18]. We first use Information Flow Routes to construct layer-wise token contribution matrices [9] and recursively extract the principal information flow circuit with Auto-Emergence [25], shown in Figure 1. Finally, we validate the circuit with intervention.
Figure 1. Principal information flow circuit in a harmonic-progression example. The horizontal axis shows the music tokens given to the model, and the vertical axis shows the Transformer layers. The yellow highlight marks the prediction goal of tonal resolution. Emergence Order ranks the music tokens by when they first enter the recursively extracted principal information flow circuit.
Our study takes a first step toward bringing circuit analysis into the music domain. While we currently focus on a single, highly specified task centered on one harmonic behavior, we hope that this line of analysis will not only deepen our understanding of music models but also enable us to learn from the musical knowledge encoded in music models, such as composition motifs, in the future.
Background: harmony and music notation
Keys, chords, and harmonic resolution. Tonal music is organized around a key, which fixes a central pitch and the scale built on it; the position of a note within this scale is its scale degree. A chord is a set of notes played simultaneously, and the note on which it is built is its root. Chords are named by the scale degree of their root, written as Roman numerals (lower case for minor chords). In A major, for example, I is the chord on A, the tonic, and V is the chord on the fifth degree E, the dominant. In this paper, and refer only to the scale degree of the root, regardless of chord quality, so that one notation covers major and minor keys. The tonic is heard as stable and the dominant as tense, and the motion , called resolution, is the canonical way to close a phrase [16, 3, 4]. We study the chord progression : a passage that starts on a tonic-root chord, passes through any number of intermediate chords, and ends with a dominant-root chord immediately followed by a tonic-root chord.
Monophonic and polyphonic music. A monophonic piece has a single melodic line, possibly annotated with chord symbols, as in a folk-tune lead sheet. A polyphonic piece has several lines sounding at once, such as the four voices of a chorale, from high to low soprano, alto, tenor, and bass (SATB).
ABC notation. Both models in this study read and write music as plain text in ABC notation. A tune starts with header fields such as the meter (M:6/8) and the key (K:A); notes are written as letters, a vertical bar | separates measures, and a chord symbol is written in double quotes before the notes it accompanies, e.g., "Bm" for a B minor chord. Figure 2 shows an annotated Nottingham example [10]. Polyphonic pieces use the same format, with the four voices of each chord serialized one after another (Figure 6 in Appendix § 11).

Figure 2. An annotated Nottingham excerpt in ABC notation. The key is A major and the chord symbols A–Bm–E–A form the progression I–ii–V–I. The model receives everything to the left of the dashed line and must predict the root of the final chord (highlighted); the faded tokens are withheld.
Figure 6. An SATB example from BCMH in G major, with the functional analysis –––vi–ii––. The orange dashed line marks the prediction boundary before the terminal G root; the preceding event is visible to the model, while the orange target sonority is excluded. The lower panel aligns this boundary with the serialized ABC span.
Methodology
We adopt two existing information-flow tools. Information Flow Routes [9] converts the symbolic music generative model’s forward computation into a weighted, layer-wise graph, and Auto-Emergence [25] extracts the principal information flow circuit, which we abbreviate as the principal circuit. We then use intervention to validate the effect of the principal circuit.
Task and data representation
Representation. We treat the complete quoted span of a chord symbol as a harmonic site and retain notes, barlines, meter, key, and other symbols as musical context. Because tokenization is model-specific, a chord symbol may correspond to one token or multiple tokens; we align the annotated chord root to the tokenizer span that contains its root character, so that chord-symbol sites and root-degree roles can be mapped to model-token positions.
Task. The model is given the beginning of a piece in ABC notation, ending just before the root of the final chord of an progression, and it must predict that root as its next token. In the example of Figure 2, the key is A major and the chords so far are A, Bm, and E, i.e., , , and . The input ends after the opening quotation mark of the next chord symbol, and the correct continuation is the letter A, the root of the tonic chord.
Information flow graph modeling
Information Flow Routes [9] analyzes how Transformer-based models organize the contextual information for next-token prediction. It treats each input representation at every layer as an individual node. At each layer , the sum of all attention head outputs from position to position is defined as an attribution vector , which represents the directed edge . To quantify the strength of each directed edge , we define its raw contribution as the nonnegative decrease in the norm of the receiver state when the corresponding attribution vector is removed, as illustrated in Figure 3. We then normalize these scores across all causal source positions , yielding the lower-triangular contribution matrix . Each therefore represents the relative contribution of source to the state update at receiver ; we call the edges with cross-token edges and write for such an edge. Repeating this construction across all token positions and layers yields a weighted information-flow graph for the current prediction.
Figure 3. Information flow graph modeling. (a) Node-level contribution of source to receiver . (b) Layer-wise contribution matrix .
Principal information flow circuit extraction
We apply Auto-Emergence [25] to this weighted graph to extract the principal circuit. This algorithm initializes the principal circuit with the residual representations of the last input position at all layers and adds all of their incoming cross-token edges to a selection pool. It then chooses the strongest available edge in the pool and adds it to the principal circuit. Every time an edge is selected, the algorithm retains the residual representation of the source token below layer and adds all of its incoming edges from lower layers to the selection pool. The algorithm recursively expands the selection process until all tokens have been added to the principal circuit. The order in which the tokens are added is their emergence order. The selected edges are returned as sparse matrices ; Appendix B gives the details.
In our analysis, the prediction target is the terminal tonic root in each sequence. The input is truncated before this target token. The selected edges of the principal circuit show which earlier harmonic sites contribute to the terminal root prediction and at which layers they enter the principal circuit. Because the extraction starts from one prediction, every sample has its own principal circuit; we write for the set of all cross-token edges of the sample and for the edges selected into its principal circuit.
Causal intervention validation
We use attention-edge intervention [22] to test whether the extracted principal circuit is necessary for predicting the terminal tonic. Let denote the post-softmax attention probability from source position to receiver position in head of layer , where . For each selected edge , we set
without renormalizing the remaining attention probabilities. In our intervention, we simultaneously ablate all selected cross-token edges of the principal circuit and then measure the effect on the model’s prediction of the final tonic. As a random control, we ablate the same number of edges drawn uniformly at random from .
Experiments
Datasets
Our main experiments use 963 chord pattern sequences (882 in major keys and 81 in minor keys) extracted from the Nottingham ABC corpus [10], a collection of folk tunes with chord symbols. We also construct 98 four-part samples with soprano, alto, tenor, and bass (SATB) voices from BCMH [17], with the same chord pattern selection criterion, to test the circuit’s generalization to polyphonic music. Appendix § 11 gives the processing details.
Models
We evaluate ChatMusician-7B [27], a LLaMA-2-based language model adapted to symbolic music through continual pretraining and fine-tuning on ABC-based music and text data. We also evaluate MuPT-190M [15], a symbolic music model pretrained specifically on large-scale monophonic and polyphonic corpora. Both models read and generate music as ABC text. Both models are tested on Nottingham; we use only MuPT for BCMH because its pretraining explicitly includes polyphonic music.
Evaluation metrics
At the prediction position of the target root, we measure the effect of intervening on the principal circuit. In the full-vocabulary setting, we report the target tonic token’s logit and prediction rank, the top-1 accuracy (the percentage of samples for which the target tonic token has the highest probability among all vocabulary tokens), and the KL divergence between the pre- and post-intervention vocabulary distributions. To focus on the harmonic choice, we further restrict the candidates to the twelve pitch classes, merging enharmonic spellings such as and , and compute twelve-root top-1 accuracy, target rank, KL divergence, and the twelve-root margin, i.e., the score of the target root minus that of the highest-scoring competing root (Appendix § 10).
Principal information flow circuit intervention results
For each sample, we apply the intervention of § 3.4 and compare it with the mean of five same-size random controls. In every setting, the principal circuit contains fewer than 6% of all cross-token edges.
Figure 4 supports three conclusions. First, the principal circuit is necessary for the tonic prediction: ablating it removes most of both models’ ability to predict the correct tonic, whereas ablating the same number of random edges leaves the prediction almost unchanged. Second, the ablation of the principal circuit pushes the models toward other outputs: the correct tonic falls behind several competing tokens, and the whole output distribution shifts. The same holds when only the twelve possible chord roots are compared, so the effect lies in which chord root the models choose. Third, the circuit remains important in polyphonic music: on BCMH, where MuPT’s baseline is considerably lower, ablating the circuit still lowers twelve-root accuracy well below that of the random control.
Figure 4. Evaluation before and after principal information flow circuit intervention for (a) full-vocabulary and (b) twelve-root prediction with enharmonic pairs merged. Metrics are top-1 accuracy (↓), target logit or twelve-root margin (↓), target rank (↑), and KL divergence (↑); arrows indicate stronger intervention effects. Blue, orange, and gray bars denote original, circuit-intervened, and same-count random-edge-intervened values, respectively. The circuit intervention consistently produces the larger effect.
Consistency and composition of the circuit. The margin of a sample is how far the correct root letter’s logit is ahead of the best competing letter; the net margin drop of a set of edges is the difference between the margin after ablating the same number of random edges and the margin after ablating the set, so a positive value means that the set matters more than random edges (Appendix § 12.§ 1). Ablating the circuit is more damaging than the random control in almost every sample for ChatMusician and in about three quarters of the samples for MuPT (Appendix § 12). Per edge, the effect is strongest inside chord symbols: these edges form only a small part of the circuit, yet at an equal number of ablated edges they matter far more than any other category of edges (Figure 5; the five edge categories are defined in Appendix § 12.§ 4. This suggests that both models first assemble a representation of each chord within its own symbol and rely on it for the final prediction.
Figure 5. Edge categories of the principal circuit on Nottingham. (a) Fraction of the circuit’s cross-token edges that belong to each category. (b) Net margin drop when the same number of edges is ablated from each category, with 95% bootstrap confidence intervals.
Conclusion and limitations
This work provides a first systematic step toward information flow analysis and circuit discovery in the music domain. We extract and validate principal information flow circuits for harmonic chord progression in ChatMusician and MuPT. The study has two main limitations. First, the principal circuit is extracted for each prediction separately, so our findings describe regularities across the set of circuits; we do not identify a single fixed circuit, such as a list of attention heads, that is shared by all samples. Second, the task is relatively narrow and single: we study one harmonic behavior with syntactic structure, the prediction of the terminal tonic root in progressions, on Western tonal music. In the future, we will extend our analysis to more complex musical behaviors and investigate more detailed functional components of the discovered circuits.
Acknowledgments and Disclosure of Funding
We thank Daniel Chin and Liwei Lin, PhD students at Music X Lab, for their suggestions on the writing and polishing of this paper.
Responsible use statement
This work studies the mechanistic interpretability of existing symbolic music models. Its potential benefits include improved transparency and interpretability of music-generation systems. Its potential risks include overgeneralizing the findings beyond the evaluated Western tonal corpora. We mitigate these risks by limiting our claims to the evaluated models, datasets, and harmonic task, by stating the limitations of the study, and by releasing artifacts subject to the licenses of the original models and datasets.
Related work
Autoregressive symbolic music generation
Autoregressive symbolic music models represent a score as a sequence of discrete tokens and generate musical events sequentially according to . Early Transformer-based architectures, including Music Transformer and Museformer, explored the modeling of long-range musical structure [12, 26]. Recently, ChatMusician and MuPT have adopted large-scale pretraining and language-model architectures. NotaGen further incorporates preference optimization to improve generation quality and controllability [27, 15, 23]. Autoregressive models all center on next-token prediction, so we can analyze the prediction performance at target-root positions and investigate the causal effect of internal information flow in harmonic prediction.
Mechanistic interpretability and circuit discovery
Mechanistic interpretability aims to reverse-engineer computational mechanisms learned by neural networks into human-understandable algorithms and concepts [1]. Circuit discovery has become a central paradigm in this area, with foundations in the Transformer Circuits framework [7] and concrete successes on benchmark behaviors such as IOI [22] and Greater-Than [11]. Early workflows mainly relied on activation/path patching and component-wise causal intervention [22, 14], which can be costly at scale. More recent automated or attribution-based approaches, including ACDC [5], Information Flow Routes [9], and Attribution Patching [20], have improved the practicality of circuit analysis. Nevertheless, circuit discovery remains concentrated in NLP, while its application to music is still underexplored. Starting from a concrete harmonic chord progression behavior, our work takes a step toward extending circuit discovery to music.
Auto-Emergence extraction details
Auto-Emergence [25] takes the lower-triangular contribution matrices of an input with tokens and returns sparse principal matrices and an emergence order , in which a smaller nonzero means that token entered the principal circuit earlier. It starts from the prediction position : it sets , keeps for every layer, and initializes the selection pool with all cross-token edges entering . It then repeats two steps until every token has an emergence rank:
Select the strongest edge in the pool, , set , remove it from , and give source the next rank if it has none.
Because this edge uses the representation of at layer , keep for and add the edges that feed it, , to .
Twelve-root evaluation details
For each sample, we preserve the observed chord suffix (quality, extension, inversion, and closing quote) and replace only its root with each of 17 conventional spellings from C, C#, D♭, . . . , B♭, and B. Because a candidate may contain one or more model-specific tokens, its score is the teacher-forced sum of conditional log-probabilities from the root onset through the closing quote,
We map the 17 spellings to 12 pitch classes and use the highest-scoring enharmonic spelling as the primary class score, . The resulting twelve-root distribution is
We compute top-1 accuracy, target rank, and from this distribution, and define the twelve-root margin as . Thus, the twelve-root metrics compare complete root-conditioned chord continuations rather than isolated single-token logits.
Data processing details
Nottingham
Within each tonal segment, we select the last adjacent dominant-to-tonic root pair and trace back to the nearest preceding tonic-root chord; the excerpt spans the bars from that chord to the target, and major and minor keys use the same rule. Of 1,034 tune blocks, 54 have no valid sequence and 17 duplicate an earlier sample, leaving 963 samples. Each input is truncated immediately before the token that contains the root of the terminal tonic.
BCMH
We keep the synchronized SATB events of the 100 chorales in BCMH [17], convert the expert Roman-numeral annotations into ABC chord symbols, and apply the same rule within each local key, keeping at most one target per chorale. Removing two exact duplicates leaves 98 samples (55 major, 43 minor). Because each chord symbol precedes its notes, the input contains all four voices up to the preceding chord but no notes of the target chord (Figure 6).
What does the circuit reveal?
Root-letter margin
We use a single-token measure. With the logit of token at the prediction position and the target root letter among , the root-letter margin is ; it ignores accidentals. Writing for the margin of sample after ablating an edge set , the net margin drop compares it with the margin after ablating five same-size random edge sets. :
that is, the decrease in margin caused by ablating minus the mean decrease caused by the random edge sets. Positive values mean that is more damaging than random edges; we report sample means with 95% bootstrap confidence intervals (CIs).
The circuit is sparse
For each sample, the edge share is the fraction of all its cross-token edges that belong to the principal circuit, and the weight share is the fraction of its total contribution weight that the principal circuit keeps, including the contributions of positions to themselves. The circuit contains only 1.7% (ChatMusician) and 5.3% (MuPT) of the cross-token edges, yet it retains about 25% and 30% of the contribution weight.
The circuit is necessary for the tonic prediction across samples, modes, and models
For each sample, we measure the effect established in § 4.4 by the net margin drop of its principal circuit. The consistency rate is the fraction of samples with . The flip rate is the fraction of originally correct predictions, i.e., those in which the target token ranks first in the full vocabulary, that become incorrect after the ablation.
Table 1 shows that the effect holds in almost every sample for ChatMusician and in about three quarters for MuPT.
| Model | Subset | Net margin drop | Consistency rate | Flip rate | |
| ChatMusician-7B | All | 963 | 2.52 [2.42, 2.62] | 95.6% | 83.2% |
| ChatMusician-7B | Major keys | 882 | 2.43 [2.33, 2.53] | 95.6% | 82.5% |
| ChatMusician-7B | Minor keys | 81 | 3.49 [3.06, 3.95] | 96.3% | 91.1% |
| MuPT-190M | All | 963 | 1.07 [0.98, 1.17] | 75.4% | 88.3% |
| MuPT-190M | Major keys | 882 | 0.99 [0.89, 1.09] | 73.9% | 88.0% |
| MuPT-190M | Minor keys | 81 | 1.97 [1.65, 2.30] | 91.4% | 91.2% |
Table 1. Sample-level consistency of the principal circuit intervention on Nottingham. The net margin drop is shown with its 95% bootstrap CI; the other quantities are defined in Section E.3.
Connections inside a chord symbol matter most
Edge categories. We call a token a chord token if it overlaps the quoted span of a chord symbol and a non-chord token otherwise, and we assign every edge of the principal circuit to one of five edge categories according to its source and receiver: within one chord symbol if both tokens lie in the same chord symbol, between chord symbols if they lie in two different ones, chord → non-chord or non-chord → chord if only the source or only the receiver is a chord token, and outside chord symbols otherwise. Edges involving a token that overlaps two adjacent chord symbols (on average 1.1 per sample in ChatMusician, none in MuPT) are excluded. The category share of a category is , where denotes its edges.
Equal-budget ablation. To compare the categories edge for edge, we ablate the same number of edges from each: edges drawn uniformly from , averaged over five draws, with a random control of edges.
Results. Table 2 lists the values shown in Figure 5. The two models differ in what follows the chord-internal step: in ChatMusician, edges leaving a chord symbol also matter, whereas in MuPT they have small effects.
| Edge category | ChatMusician-7B Share | ChatMusician-7B Net margin drop | MuPT-190M Share | MuPT-190M Net margin drop |
| Within one chord symbol | 10.7% | 1.36 [1.27, 1.44] | 4.5% | 0.44 [0.38, 0.51] |
| Between chord symbols | 6.7% | 0.56 [0.48, 0.63] | 2.8% | 0.11 [0.04, 0.18] |
| Chord → non-chord | 8.2% | 0.57 [0.50, 0.65] | 5.0% | 0.05 [0.03, 0.07] |
| Non-chord → chord | 16.3% | 0.09 [0.05, 0.13] | 19.8% | −0.13 [−0.16, −0.11] |
| Outside chord symbols | 58.0% | −0.05 [−0.08, −0.02] | 67.8% | 0.02 [0.01, 0.03] |
Table 2. Edge categories of the principal circuit on Nottingham (). “Share” is the category share averaged over samples. The net margin drop is measured under the equal budget (on average 47.6 edges per category for ChatMusician and 42.5 for MuPT) and shown with its 95% bootstrap CI.
In polyphonic music, the circuit does not rely on the bass
A human analyst pays particular attention to the bass, which often sounds the chord root. We group MuPT’s principal edges on BCMH by the voice of their source note, for , together with the unions of the inner (A, T) and outer (S, B) voices and of all four voices, and ablate each set in full. The alto group matters most, the tenor group is indistinguishable from random edges, and the bass group is less damaging than random edges (Table 3). The circuit thus does not rely on the bass line in the way a human analyst would. Because BCMH has only 98 samples and the input already names each root in a chord symbol, this result remains open to further investigation.
| Voice group | Edges | Net margin drop |
| Soprano | 205.5 | 0.06 [0.00, 0.11] |
| Alto | 227.4 | 0.16 [0.09, 0.22] |
| Tenor | 214.6 | 0.01 [−0.05, 0.07] |
| Bass | 229.0 | −0.24 [−0.33, −0.16] |
| Inner voices (alto and tenor) | 442.0 | 0.16 [0.07, 0.25] |
| Outer voices (soprano and bass) | 434.5 | −0.21 [−0.30, −0.12] |
| All four voices | 876.5 | −0.15 [−0.27, −0.03] |
Table 3. Net margin drop of the voice groups on BCMH with MuPT-190M (), with 95% bootstrap CIs. “Edges” is the mean number of ablated edges per sample. The last three rows are unions of voice groups.
References
- [1]Leonard F. Bereska and Efstratios Gavves. Mechanistic interpretability for AI safety—a review. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2404.14082
- [2]Trenton Bricken, Adly Templeton, Joshua Batson, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
- [3]William E. Caplin. Cadence: A Study of Closure in Tonal Music. Oxford University Press, 2024. doi: 10.1093/oso/9780190056445.001.0001.DOI
- [4]Wai Ling Cheong and Hong Ding. Sposobin remains: A Soviet harmony textbook’s twisted fate in China. Zeitschrift der Gesellschaft für Musiktheorie, 15(2):45–77, 2018. doi: 10.31751/974.DOI
- [5]Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2304.14997
- [6]Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Advances in Neural Information Processing Systems, volume 36, pages 47704–47720, 2023.arxiv.org/abs/2306.05284
- [7]Nelson Elhage, Neel Nanda, Catherine Olsson, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
- [8]Simone Facchiano, Giorgio Strano, Donato Crisostomi, Irene Tallini, Tommaso Mencattini, Fabio Galasso, and Emanuele Rodolà. Activation patching for interpretable steering in music generation. arXiv preprint arXiv:2504.04479, 2025.arxiv.org/abs/2504.04479
- [9]Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17432–17445, 2024.
- [10]Eric Foxley. Nottingham database. https://ifdo.ca/~seymour/nottingham/nottingham.html, 2011. Last updated October 2, 2011.
- [11]Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.00586
- [12]Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music Transformer: Generating music with long-term structure. In International Conference on Learning Representations, 2019.
- [13]Wenye Ma, Xinyue Li, and Gus Xia. Do music LLMs learn symbolic concepts? A pilot study using probing and intervention. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, 2024.
- [14]Kevin Meng, David Bau, Alex J. Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2202.05262
- [15]Multimodal Art Projection. MuPT: Symbolic music generative pre-trained transformer. https://map-mupt.github.io/, 2024. Accessed: 2026-08-09.
- [16]Open Music Theory. Classical cadence types. https://openmusictheory.github.io/cadenceTypes, n.d. Accessed: 2026-03-27.
- [17]David R. W. Sears and Jonathan E. Verbeten. Bach chorales melody–harmony corpus (BCMH). GitHub repository, https://github.com/MusicCogCompLab/BCMH, n.d. CC BY 4.0. Accessed August 27, 2026.
- [18]David R. W. Sears, Jonathan E. Verbeten, and Hannah M. Percival. Does order matter? Harmonic priming effects for scrambled tonal chord sequences. Journal of Experimental Psychology: Human Perception and Performance, 49(7):999–1015, 2023. doi: 10.1037/xhp0001103.
- [19]Nikhil Singh, Manuel Cherep, and Pattie Maes. Discovering and steering interpretable concepts in large generative music models. In International Conference on Learning Representations, 2026.
- [20]Aaquiб Syed, Can Rager, and Arthur Conny. Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407–416, 2024.arxiv.org/abs/2310.10348
- [21]Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019.
- [22]Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations, 2023.arxiv.org/abs/2211.00593
- [23]Yashan Wang, Shangda Wu, Jianhui Hu, Xingjian Du, Yueqi Peng, Yongxin Huang, Shuai Fan, Xiaobing Li, Feng Yu, and Maosong Sun. NotaGen: Advancing musicality in symbolic music generation with large language model training paradigms. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 10207–10215, 2025.
- [24]Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. In International Conference on Learning Representations, 2024.
- [25]Rui Xu, Yi Chen, Jujiu Chen, and Sihong Xie. Information flow reveals when to trust language models. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
- [26]Botao Yu, Peiling Lu, Rui Wang, Wei Hu, Xu Tan, Wei Ye, Shikun Zhang, Tao Qin, and Tie-Yan Liu. Museformer: Transformer with fine- and coarse-grained attention for music generation. In Advances in Neural Information Processing Systems, volume 35, pages 1376–1388, 2022.
- [27]Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. ChatMusician: Understanding and generating music intrinsically with LLM. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6252–6271, 2024.
- [28]Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tie-Yan Liu. MusicBERT: Symbolic music understanding with large-scale pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 791–800, 2021.
- [29]Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, and Ellie Pavlick. The same but different: Structural similarities and differences in multilingual language modeling. In International Conference on Learning Representations, 2025.




