Structure, Not Sequence: Tracking and Validating Folding Mechanisms in ESMFold's Trunk with Sparse Autoencoders

Introduction

Deep neural structure predictors such as ESMFold (Lin et al. [14]) have revolutionized structural biology by reducing computation times from days to seconds. Structure predictors transform a protein’s amino-acid (AA) sequence into a 3D structure through multiple learned blocks. However, the internal computational processes remain opaque, rendering these folding models black boxes that lack scientifically reliable explanations or guarantees. Verifying that the computation aligns with underlying scientific processes, and extracting biological insight that links hidden features to output structures, therefore remains out of reach.

Sparse autoencoders (SAEs), which have shown significant potential in interpreting the internal computations of large language models (LLMs), are strong candidates for uncovering mechanisms underlying protein structural regression. SAEs decompose dense activations into an overcomplete dictionary of features that are substantially more interpretable than raw neurons (Bricken et al. [3]). In the context of proteins, SAEs have been trained on the sequence model ESM-2 to identify and automatically interpret features that correspond to biological concepts (Simon and Zou [25]; Adams et al. [1]). However, because sequence language models are trained on the masked language modeling objective, they only compute dense sequence embeddings and lack explicit directives to learn information about the 3D structure that determines protein function. In contrast, the folding trunk transforms sequence embeddings into structural representations. We therefore hypothesize that the trunk and the sequence model specialize in their distinct tasks: residue-level semantics is inherited from the sequence embedding, while the geometry is computed inside the trunk and materialized by the structure module with optional refinement recycling. We visualize the ESMFold model design in Figure 1.

Our work focuses on interpreting the folding trunk beyond the sequence model analyzed in prior work, and emphasizes structure over sequence. Three motivating observations inform the approach: folding hidden states are organized differently from the initial ESM-2 embeddings, the trunk encodes structure in a pair representation that is absent from the sequence model, and trunk blocks compute spatial information across depth. These observations motivate a protocol, which involves training TopK SAEs at five evenly spaced trunk blocks, automatically interpreting each resulting feature, and attributing features across blocks, detailed in § 3 and § 4. The resulting dictionary gives a diverse and faithful interpretation of the folding trunk. Measured on top of the language model states, the trunk code raises residue contact and strand orientation detection by up to +0.32 AUROC, while residue-level properties are already recoverable from the language model. Its features also localize functional sites, such as catalytic residues and disulfide bonds, on the folded structure. Building upon the learned dictionary, a reasoning LLM assigns names to each feature based on structural and functional context, while gradient-based attribution traces the composition of these concepts across the trunk’s depth into a directed folding circuit. Together, these steps give an automatically interpreted account of the transformation from sequence to structure.

The main contributions of this work are as follows:

• Interpret the folding trunk at scale. We train the first large-scale sparse-feature dictionaries of a folding model’s trunk module, on both its single and pair representations across the depth of ESMFold, and interpret every feature with a reasoning LLM, verified by checking each label on quantitative context withheld from the prompt assembler.

• Locate the computation the sequence model leaves out. We measure the trunk code’s contribution over the language-model states prior dictionaries decompose: gains of up to +0.32 AUROC on relations between residues, while residue-level properties are already supplied by the language model. The pattern exposes a division of labor: the single representation carries inherited residue semantics and the pair representation computes the geometry.

• Trace a causal folding circuit, and read the output from it. We compose the concepts into a directed circuit across depth with integrated-gradient attribution, confirm its edges causally in the live ESMFold pass, trace the interactions between the single and pair representations within the trunk, and turn the interpretation into a read-out of fold quality.

Related Work

Sparse autoencoders and mechanistic interpretability. SAEs decompose dense neural activations into a sparse, overcomplete dictionary of features that are more monosemantic than raw neurons, a response to the superposition hypothesis under which networks pack more features than they have dimensions [5, 3]. They scale to frontier language models [27, 7] and admit architectural variants: TopK SAEs that fix the active-feature count [7], transcoders that predict one layer’s activations from another [4], and crosscoders that share a single dictionary across layers or models [15]. We build on this toolkit, training TopK dictionaries at five depths of the folding trunk. High reconstruction and high single-feature selectivity still leave open whether a feature is causally used or semantically meaningful [16], and two families of tools address this. The interventional family patches, ablates, or transplants activations and reads the downstream effect [18, 32]; the correlational family, from the probing literature, tests a probe against a control task solvable only by exploiting a confound, isolating whether the probe reads the target property [8]. Our residue-identity control is the protein-model instance of the latter, applied as attribution across a benchmark of interpreted features.

Protein models: structure, stability, and function. ESM-2 is a masked language model over protein sequences whose embeddings carry structural and functional signal [24, 14]. ESMFold folds directly from these embeddings through a trunk of structure blocks, in a single forward pass without an MSA pipeline [9, 14]. Beyond structure, sequence likelihood and embedding features from the same models predict variant effects and stability [17, 2, 6], benchmarked at scale by ProteinGym [19], and probing studies show that secondary structure, contacts, and some functional properties are linearly decodable from Protein Language Model embeddings [21, 31, 22]. Single-sequence predictors already reach high secondary-structure accuracy from sequence alone [11], marking secondary structure as sequence-determined. The trunk performs an explicit sequence-to-geometry regression, materializing 3D structure that the frozen sequence embedding leaves implicit, and that computation is what we interpret. Figure 1 places our target within the ESMFold pipeline.

Interpreting protein models. On interpreting sequence semantics, InterPLM [25] introduced the language-model pipeline that names protein-SAE features automatically, and InterProt [1] adds an interactive latent viewer with human-rated interpretability; we adopt this automated-interpretation step directly. On the structural side, Parsan, Yang, and Yang (2025) train SAEs on the ESM-2-3B sequence backbone and relate the features to ESMFold’s output, leaving the folding trunk’s internal computation and residue-type correlation for future work. We interpret the ESMFold trunk directly, covering its single representation $s_s$ and the folding-exclusive pair representation $s_z$ that no sequence-based dictionary can reach, measure each feature by function, and separate what it adds from residue identity. We further trace interpreted concepts into a directed circuit across the trunk’s blocks with integrated gradients, and close the gap between deep folding networks and interpretable biological processes.

Method

Our protocol trains sparse dictionaries on the internal states of ESMFold’s folding trunk, interprets each resulting feature with a language model, evaluates what the features contribute over the sequence model, and traces how features compose across the trunk’s depth. § 3.1 defines the trunk representations and the three observations that motivate the protocol, § 3.2 gives the dictionary recipe, § 3.3 describes the automated interpretation step we adopt from prior protein-SAE work, § 3.4 defines the evaluation strategy, and § 3.5 describes the integrated-gradient construction of the folding circuit. Corpus, optimization, and protocol details are collected in § 4.1.

The folding trunk and pilot analysis

Trunk representations. ESMFold (Lin et al. [14]) folds a sequence by first embedding it with the ESM-2-3B masked language model (Lin et al. [14]) and then passing the embedding through a folding trunk of 48 structure blocks. Each block maintains two states: a single representation $s_s \in\mathbb{R}^{L \times1024}$, one vector per residue, and a pair representation $s_z \in\mathbb{R}^{L \times L \times128}$, one vector per residue pair, which has no counterpart in the sequence model. The trunk is applied iteratively: at each of a fixed number of recycles a structure module reads $s_s$ and $s_z$ into a predicted structure, and the next iteration restarts the trunk with both representations carried over and the predicted structure returned as a binned distogram. The coordinates from the final recycle are the output. We analyze five evenly spaced blocks, $b_{00}$, $b_{12}$, $b_{24}$, $b_{36}$, $b_{47}$, spanning the trunk from its first block ($b_{00}$) to the block that feeds the structure module ($b_{47}$).

Empirical motivation. Three properties of the trunk shape our protocol and analysis: (i) A representational gap. We measure linear Centered Kernel Alignment (CKA) (Kornblith et al. [12]) between the ESM-2 layers and the trunk blocks, plotted in Figure 2A. ESM-2–trunk block pairs average 0.14, and ESM-2’s final layer reaches 0.23 CKA with $b_{00}$; the trunk blocks meanwhile agree with one another at averaged 0.65, and the middle blocks peak at 0.96. The trunk organizes residues by a similarity structure of its own, which calls for dictionaries trained on that structure rather than the sequence dictionaries of prior work (Simon and Zou [25]; Adams et al. [1]). (ii) Structure in the pair representation. Geometry is encoded in the folding-exclusive pair representation $s_z$: pairwise residue distance is linearly decodable from $s_z$ at deep blocks and stays near-absent from any single-residue state, as Figure 2B shows, so a faithful account of the folding computation must read $s_z$ as well as $s_s$. (iii) Inherited residue semantics, computed geometry. A depth-resolved probe shows that functional-site signal is maximal at ESM-2 $L36$ and erodes with trunk depth, plotted in Figure 2C. The early-commitment pattern indicates that per-residue semantics is inherited from the sequence model, whereas geometric signal such as burial builds with depth and is therefore computed inside the trunk. We cache cycle-1 and cycle-3 latents for training. Appendix § 13.2 shows the representation is insensitive to the recycle budget. This divergence between functional and geometrical information along the trunk depth motivates the interpretation of the within-trunk computation.

Sparse dictionaries of the trunk

TopK dictionaries. For each block we train a TopK sparse autoencoder (Gao et al. [7]; Bricken et al. [3]) on the block’s representation, mean-centered over the training corpus. A residue (or pair) activation $x$ is encoded and reconstructed as

where $\operatorname{TopK}_k$ keeps the $k$ largest ReLU logits and zeros the rest. Training minimizes reconstruction error $\lVert x-\hat{x}\rVert_2^2$ with the decoder rows held at unit norm and dead features periodically reinitialized toward high-residual examples. We report explained variance (EV) on held-out data.

Where dictionaries are placed. We train a dictionary at each of the five trunk blocks on both the single representation $s_s$ and the pair representation $s_z$, and, as a sequence baseline, on the ESM-2 layers that feed the trunk. We set $k=16$ throughout to obtain balanced reconstruction and sparsity for interpretation. The training corpus, optimization budget, and checkpoint format are detailed in § 4.1.

Automated interpretation

The two representations carry different objects, so we interpret them with two prompts. For the single representation we start from the pipeline introduced for protein SAEs by InterPLM (Simon and Zou [25]): a reasoning LLM reads a context assembled from the feature’s top-activating residues across proteins with the structural and functional environment at each, and returns a short free-form label and one of seven concept categories: secondary structure, burial, contact, functional site, disulfide, chemical, positional. The reasoning LLM also decides whether the feature reflects amino-acid type or a real biological concept.

A pair feature operates on residue pairs. As prior interpretations missed this object, our revised prompt describes the population of pairs a feature fires on and gives both endpoints’ residue context. The prompt asks openly what relation the pair encodes. Because the representations are interpreted differently, we validate the distinction between representations with cross-prompt control in Appendix § 17.

Feature evaluation

Feature localization on the protein. We localize feature $f$ by its activation sites, the map

painted on the folded structure. The encoder applies per residue, so this activation is the feature’s complete read-out of each residue: a gradient $\partial(\sum_j z_{j,f})/\partial s_{s,i}$ is the encoder row on the feature’s support and vanishes off it.

Isolating the trunk’s contribution from the sequence model. To isolate the additive trunk computation, we measure the trunk code as an increment over the sequence latents. For each target we fit a probe on ESM-2-3B at $L_{36}$, its final layer, which the trunk’s learned mixture over the ESM-2 representations weights at 0.90, then on those same states concatenated with the trunk code, and read the difference in held-out AUROC across proteins. Targets come in two forms: per-residue properties, namely annotated sites, burial and strand, and relations between two residues, namely residue contact, $\beta$-ladder continuation and parallel-versus-antiparallel strand orientation. On a relational target every representation is given the pairwise interaction features $[s_i; s_j; s_i \odot s_j]$, so a per-residue state is asked about the pair through the same probe the trunk code gets. Strand orientation is scored within a common distance shell, so proximity cannot stand in for topology.

Relating the code to the folding output. We fit a ridge probe from a block’s code to both ESMFold’s own confidence (pLDDT) and accuracy against the ground truth (lDDT). We report held-out $R^2$ on unseen proteins. Individual concepts are then ranked by their Spearman correlation with accuracy.

Circuit construction by integrated gradients

To trace how interpreted concepts compose, we attribute each downstream feature to the upstream code by integrated gradients ([26]) through the live ESMFold forward pass. One construction covers every attribution we report, parameterized by a source block, a destination block, and a choice of representation at each end. At the source we splice in the dictionary of the chosen representation; at the destination we capture the chosen representation and encode it with its own dictionary. The edge from upstream feature $i$ to downstream feature $j$ is

the integrated gradient of the downstream code along the straight path from the zero-code baseline to the realized upstream code $z$. Each attribution runs on a single recycle, as the change across recycles is minimal.

The four combinations give the whole circuit over blocks and representations. Within-domain tracing gives the information flow within a representation track, $s_s \to s_s$ through the single representation and $s_z \to s_z$ through the pair representation. Cross-representation tracing traces the coupling between the representations, forward as $s_s \to s_z$ where the single representation writes into the pair, and backward as $s_z \to s_s$ where the pair representation writes back. Appendix E provides details.

Experiments and Results

Implementation details

Dictionary training. The single-representation cache is created from 500k folded UniRef50 sequences, giving a corpus of 95M single-residue activations. In order to make fair trunk-versus-sequence comparisons, every single-representation and ESM-2-layer dictionary is trained on the same data budget and the following protocol: batch size $b_s = 2048$, $k=16$, dictionary width $d_{\mathrm{sae}}=8192$, number of epochs $ep=10$, a linear learning-rate warmup over the first 5% of steps, and base learning rate $lr=4\times10^{-4}$. Pair-representation dictionaries use (1) width $d_{\mathrm{in}}=128$ and $d_{\mathrm{sae}}=4096$. Because the number of pair samples grows with $L^2$, they are trained under a different recipe: 4.8M pair activations drawn from 967 proteins, batch size 8192, twenty-five epochs. Dictionaries are trained at the five trunk blocks and at the ESM-2 layers the trunk reads from. A sparsity sweep over $k\in\{16,32,64,128\}$ is provided in Table 3.

Block	$k=16$	$k=32$	$k=64$	$k=128$
b00	0.872	0.920	0.962	0.988
b12	0.900	0.934	0.963	0.984
b24	0.893	0.928	0.959	0.981
b36	0.863	0.908	0.945	0.973
b47	0.893	0.929	0.957	0.977

Interpretation context. For every feature, the reasoning LLM’s context is assembled from its top-activating residues across proteins, annotated with the structural and functional environment: secondary structure and solvent-accessible area from the AlphaFold model ([30, 10, 29]), the number of C$\alpha$ contacts and long-range tertiary neighbors, disulfide participation, and the UniProt functional annotation at that position [28]. Alongside each label we record the feature’s amino-acid purity and dominant residue as a quantitative check on the identity-versus-concept decision. A pair-representation feature is instead described by the distribution over its activating pairs, and its question asks openly what relation the pair encodes. The pipeline yields an interpreted label for 5,595 alive single-representation features and 2,972 pair features across the trunk blocks. Appendix O reproduces both prompt templates.

Evaluation protocol. The increment over the sequence model of § 5 samples positions from each protein with equal weights: 70 residues per protein where the target is a per-residue property and 220 pairs per protein where it is a relation, drawn without replacement under a fixed seed. Its probe is a logistic regression under a protein-stratified split, so no protein contributes to both training and evaluation. Reported confidence intervals are 95% nonparametric bootstraps, randomly resampled as specified in Appendix § 10.

Dictionary evaluation TopK dictionaries trained on the trunk’s single representation reconstruct it well across depth: at the $k=16$ setting we use throughout, explained variance is ev = 0.87, 0.90, 0.89, 0.86, 0.89 for blocks 0, 12, 24, 36, 47, trading a few points of reconstruction for interpretable codes.

Growing relational information in the trunk

We measure what the trunk code supplies on top of the per-residue language-model states that prior dictionaries decompose, reading the AUROC increment when the trunk code is added to the ESM-2-3B $L_{36}$ probe defined in § 3.4. Because that baseline already encodes sequence semantics, the increment isolates what the trunk adds beyond semantics. On relations between residues, adding the trunk code improves prediction substantially. Residue contact gains +0.318 AUROC, ladder continuation +0.209, and strand orientation +0.086, each far outside the spread of a control that shuffles the trunk features across examples (±0.010), as Table 1 reports.

Target	Type	ESM-2-3B	+ trunk	Δ
Residue contact	relational	0.636	0.954	+0.318
Ladder continuation	relational	0.644	0.853	+0.209
Strand orientation	relational	0.873	0.959	+0.086
Metal binding	per-residue	0.978	0.992	+0.014
Strand	per-residue	0.961	0.969	+0.008
Active site	per-residue	0.978	0.984	+0.006
Ligand binding	per-residue	0.923	0.924	+0.001
Burial	per-residue	0.827	0.827	−0.000

Strand orientation is the cleanest of these. Among residue pairs that are already paired, whether the two strands run parallel or antiparallel is settled by topology alone: the two classes differ in mean separation by 0.05 Å, and distance by itself scores 0.511. The trunk pair code reaches 0.965, where ESM-2-3B under the same pairwise probe reaches 0.862.

Residue-level properties behave differently. Active site gains +0.006, ligand binding +0.001, burial −0.000, and strand +0.008, all within the shuffled control, with metal binding marginally above it at +0.014. The language model already supplies what a per-residue read-out can use for these, and the pattern follows the form of the target rather than its difficulty: burial leaves ample headroom at 0.827 and still gains nothing. The same boundary appears at the whole-protein level, where the pooled trunk code matches an ESM-2-3B dictionary at enzyme-class and fold retrieval, as Appendix § 15 reports.

Faithful automated interpretation of concepts

The pipeline of § 3.3 labels 8,567 alive features across the five blocks, 5,595 in the single representation and 2,972 in the pair representation, giving each a free-form label, a concept category, and whether it is a residue-type indicator. The single-representation labels recover quantities withheld from the prompt: amino-acid purity predicts a feature’s identity assignment at AUROC = 0.85 (mean purity 0.85 for identity-labeled features, 0.45 for structural ones), and the annotation density of its top contexts predicts its functional-site assignment at 0.89 (0.59 to 0.14). Of the 3,502 single-representation features carrying a spatial concept, 1,270 name packing, burial, or tertiary contact with no annotation anywhere in their context, so the vocabulary describes geometry the trunk computes, without echoed annotation.

We separate the sequence-oriented categories from the 3D-oriented ones and observe a distinct separation between the two tracks of single and pair representations. The pair vocabulary is almost entirely geometric, 91% geometric to 9% sequence, while the single vocabulary is mixed, 57% geometric to 43% sequence, carrying five times the pair track’s share of sequence concepts alongside geometry information. Within the single representation the geometry share falls with depth, from 0.75 at $b_{00}$ to 0.43 at $b_{47}$, so sequence semantics concentrates at the read-out. To test for bias induced by the heterogeneous automated interpretation, we perform a cross-prompt control experiment. When asked about sequence semantics, the pair vocabulary stays geometric at 99%, and when asked about spatial relations, the single vocabulary does not turn more geometric, falling from 57% geometric share to 32%, as Figure 7A reports.

To validate the interpreted concepts, we probe the pair representation’s geometric concepts on relations read from coordinates, and each group predicts its own relation: $\beta$-labeled features reach AUROC 0.938 on strand pairing against 0.839 on the helix contact, while helix-labeled features reverse the ordering, 0.989 on the helix contact against 0.737 on strand pairing. Both clear a random group of the same size, as Appendix § 16 reports. The category profile across depth and the assignment-grounding curves are provided in Figure 7.

Clustering the free-form labels blind to the taxonomy, with TF-IDF and $k$-means, recovers a data-driven vocabulary, and

Figure 3 traces how its concepts compose into a circuit across depth. The pair representation yields 18 concepts that are entirely geometric relations: residue contact, $\beta$-strand pairing, helix periodicity, and hydrophobic packing, with no residue-identity concept; the single representation yields 28 concepts that add a near-complete single-residue lexicon (13 of 28, one per amino acid) concentrating at the read-out. The concepts are grounded in measurements independent of the label text: identity concepts name the measured dominant residue in 12 of 13 clusters with 0.84 purity, above the baseline 0.40 of structural concepts. Geometry concepts match the measured sequence separation, $\sim6$ residues for helix periodicity compared to $\sim36$ for long-range and tertiary contact.

Activation sites

The quantitative account rests on a dictionary that can be inspected directly, feature by feature, with labels that, as § 6 establishes, answer to context the reasoning LLM never sees. For a detector already chosen by the detection scan, the activation sites of (2) localizes on the folded structure the residues that the feature reads from. Figure 4 shows this across four annotation types: the disulfide detector concentrates on the bonded cysteine pairs, and the transmembrane, signal-peptide, and N-glycosylation detectors on their annotated spans, with more examples in Figure 6. Out of sample, the same detectors annotate proteins carrying no functional-site label, and each annotation is a claim about coordinates that can be tested without consulting the detector. Of the cysteines flagged as disulfide-bonded across 201 such proteins, 594 of 1,121 take a $C\beta$–$C\beta$ partner within 6.5 Å in the predicted fold. Appendix D reports those annotations, their sequence-level counterparts, and a panel of disease enzymes.

Direction of information flow

The interpreted concepts of § 3.3 compose into a directed circuit across the trunk’s depth, drawn in Figure 3 by attributing each downstream feature to the upstream code with integrated gradients. Row names are display forms picked for clarity, with their full details in Appendix A. The single representation is seeded from the sequence embedding and the pair representation holds the geometry the fold is

caption: tbl:2: The eight chosen attribution cases. Edge: the attribution edge location, and whether it runs forward ($s_s \rightarrow s_z$) or backward ($s_z \rightarrow s_s$). Source and Target: the auto-interpreted features. Explanation: the biochemical explanation.

:::

made of, so we call $s_s \rightarrow s_z$ the forward direction, from sequence toward structure, and $s_z \rightarrow s_s$ the backward one. Attributed edges claim that one interpreted feature drives another, and known biochemistry supports most of them: 79% of edges within either representation, 75% forward and 63% backward across them. Appendix M draws each of the four attributions in full at the level of the individual labels. Attributed edges are also tested in the live ESMFold pass, by ablating the upstream feature at its block and reading the change in its attributed downstream partner against a write matched on both magnitude and direction. The edge moves its partner 1.42× that control; Appendices B and I report the pair-representation effects and the control variants. The two representations flow unequally: about a third of the pair code’s attribution traces to $s_s$, led by hydrophobic identity and core burial, while the geometry feeds back into the single-representation code only weakly, about 6%.

Case study: biochemically supported and rejected edges

Table 2 traces the most strongly attributed edge of each kind, so the cases show what a verdict is made of rather than how the verdicts are distributed. The supported edges are the ones coherent with biochemistry. The rejected ones diverge in their causes: two carry a local cause into long-range geometry, while the other two bind a residue to geometry against its chemical property.

Across all rejections, most fall on the backward direction: 37% of backward cross-track edges are unsupported, compared to 25% of forward ones, with 248 out of the 290 rejected cross-track edges being backward. Because forward edges carry the larger attributed weight, the trend is not reflected in Table 2.

Influence on the folding output

Beyond describing the trunk, the code forecasts how well it folds. Figure 5A plots a ridge probe on the interpretable code predicting ESMFold’s per-residue confidence (pLDDT) and its true per-residue accuracy (lDDT against the reference structure) at every trunk block, rising from $R^2=0.61$ and 0.48 at the first block to $R^2=0.84$ and 0.60 at the read-out and standing far above a representation-magnitude baseline. Figure 5B ranks the structural concepts by accuracy: the

buried and packed ones (buried core, buried $\beta$-strand, long-range tertiary contact) mark accurate residues, while solvent-exposed, flexible and terminal concepts mark less confident and less accurate ones.

Conclusion

We open the black box of ESMFold’s folding trunk and interpret it into a circuit of named concepts. Our trained sparse dictionaries over both tracks of the trunk representations, interpreted at scale by a reasoning LLM, create a rich and faithful interpretation starting point. We observe the split of task between the representation tracks: the pair representation is 91% geometric, and the single representation keeps the sequence semantics intact from the sequence language model. Measured on top of the language-model embedding, the trunk code adds up to +0.32 AUROC to inferring residue relational properties. Integrated gradients compose the concepts into a directed circuit, with known biochemistry supporting about 73% of its edges. We hope this work can promote more responsible application of the deep folding models and inspire future work on interpretable AI for Biochemistry.

Supplementary interpretability figures

Figure 6 paints the interpreted detectors on the folded structure across eight annotation types, four proteins each: the point-firing disulfide and metal detectors and the span-marking transmembrane, signal, and N-glycosylation ones localize well, the dense ligand-binding, active-site, and modified-residue readouts are distributed and less accurate. Figure 7 profiles the fixed-taxonomy vocabulary across depth and gives the label-grounding curves; Figure 8 extends the category flow of Figure 3 to the individual concepts within the categories. The per-concept circuit is summarized in § 7.1.

Rendering details of circuit figures. Rows are annotation categories, and their names in the figure are display forms picked for legibility. In the single-representation track, the underlying categories are contact, burial, secondary structure, functional site, and chemical, shown as residue contact, burial and packing, secondary structure, functional site, and side-chain chemistry. The residue-identity row is the subset of the chemical category whose cluster terms name one amino acid, separated because it carries the inherited lexicon rather than a chemical property. The pair-representation rows are the geometry classes assigned to each pair feature: long-range tertiary, beta-sheet pairing, helix packing, local/turn, and mixed/other. Every feature enters exactly one row, so the rows partition the vocabulary. Every visual channel of the concept-flow figure is a deterministic function of the summed integrated-gradient weights, disclosed here for reproducibility. Each attributed edge weight is summed to (block, upstream concept, downstream concept), and a concept node’s mass $m$ at a block is the sum of its incident edge weights. Magnitudes span several orders, so each channel passes through a log compression to the unit interval,

with $w_{\max}$ the largest value of that channel. Node area is $18+430\tilde{\phi}(m;m_{\max})$, where $m_{\max}$ is the largest node mass across both tracks, so the size legend reads as a percentage of the largest node. Within-track edges have width $0.5+4.5\tilde{\phi}(w;w_{\max})$ and opacity $0.30+0.55\tilde{\phi}(w;w_{\max})$ against the track’s largest edge, and their color interpolates linearly from the source to the destination concept color. The two cross-track bundles are drawn on a shared scale that encodes their asymmetry: forward $s_s\to s_z$ edges have width $0.6+4.4\tilde{\phi}(w;w_{\max})$ and backward $s_z\to s_s$ edges $0.25+0.66\tilde{\phi}(w;w_{\max})$, the backward cap scaled down from the forward cap by the ratio of their shares of the downstream track’s incoming attribution, and the two bow apart laterally at opacities 0.55 and 0.42. The figure aggregates the 28 largest single-representation and 18 largest pair-representation concepts by feature count, and omits within-track edges below $0.002w_{\max}$ and cross-track edges below $0.008w_{\max}$. Areas and widths are read on the log scale of (4).

Controls and robustness

Confidence intervals and null bands. Every confidence interval in this paper is a nonparametric percentile bootstrap: we resample the evaluation units with replacement under a fixed seed, recompute the statistic on each resample, and report the 2.5 and 97.5 percentiles of the resulting distribution as the 95% interval. Two resampling units appear. Detection and within-AA selectivity intervals resample the evaluation proteins as clusters, 1,000 times, so all residues of a protein move together: an overall value averages the per-protein AUROCs of the resampled set, and a within-AA value pools the per-residue cells of the resampled proteins before averaging, which lets the interval respect the correlation within a protein. The single-sequence AlphaFold comparison of Appendix § 11, the retrieval intervals of Appendix § 15, and the output-attribution intervals of § 7.3 use this same per-protein bootstrap. Causal fold-change intervals instead resample the attributed targets, 2,000 times, around a ratio of means: each resample divides the mean intervention effect of the real edges by the mean effect of the matched control, so the reported multiples, the $1.42\times$ of § 7.1 and the $1.64\times$ and $31.7\times$ of Appendix § 14, each carry a percentile interval on that ratio together with the fraction of targets on which the real edge exceeds its control.

The relational increments of Table 1 are judged against a permutation band in place of a bootstrap. We permute the trunk feature block across examples under a fixed seed, preserving each feature’s marginal distribution while removing its alignment to residues, then read the AUROC change from appending this misaligned block. The change stays within $\pm0.010$ on every property.

Feature composition. The directed edges of § 7.1 are established against matched controls in the native ESM-Fold forward pass. For each attributed edge, we ablate the upstream feature at its block, on the residues where it is active, and read its attributed downstream partner at the destination block within the same pass. Over 1,063 edge–protein cases across the four adjacent block pairs, the ablation moves its partner $1.42\times$ (95% CI [1.38, 1.47]) a write matched on norm and on the causal direction side to the downstream readout direction, exceeding that control in 62% of cases. The same intervention moves predicted geometry, distogram-$L_1$ shifting $3.70\times$ ([3.37, 4.08]) the matched control. Against a magnitude-matched control alone, the ratio is $56.6\times$, so magnitude matching is the weaker test and the composition claim rests on the read-out-matched one. The intervention is small relative to the representation it displaces, 5.4% of the norm on the residues it touches. Appendix § 14 applies this protocol unchanged to the pair representation, where the same comparison gives $1.64\times$ on 88% of cases, so the pair track carries composition more strongly than the single one under an identical test.

Output attribution. At $b_{24}$ the probe of § 7.3 predicts pLDDT and true per-residue accuracy at $R^2=0.73$ and 0.56, compared with 0.17 and 0.08 for a representation-magnitude baseline at the same block. ESMFold’s own confidence and true accuracy agree at $\rho=0.72$. The integrated-gradients attribution is reported as causal mass divided by

activation share, so a value above one marks a concept carrying more causal mass than its share of activation. Ablation of any single feature produces consistent folding output, so the influence on the output is distributed across a non-trivial set of features.

Sparsity sweep. All main-text dictionaries use the sparsest setting, $k=16$. Table 3 shows reconstruction of the trunk single representation improving monotonically with the active-feature budget at every block, which we trade for maximally interpretable codes.

Amino-acid type as a measurement bias

Amino-acid types encode chemical information and are decodable from the latent, so a detector can appear structural merely by re-reading residue type. To attribute how much a feature adds beyond amino-acid type, we recompute detection with residue type held fixed. For feature $f$, over the set $\mathcal{C}$ of (protein, residue-type) cells,

where $\operatorname{AUROC}_{p,r}(z_f)$ scores the feature’s activation as a detector within residue type $r$ of protein $p$, so amino-acid type cancels within every cell. Following the control-task diagnostic Hewitt and Liang [8], we read (5) as attribution across the interpreted benchmark rather than as a pass/fail test. A feature is scored as a detector over the full sequence of each protein, per protein and then averaged. We apply a homology filter (5-mer Jaccard $\ge0.30$) and protein-stratified splits, and report both micro- and macro-averaged scores with paired cluster-bootstrap confidence intervals over proteins. Detectors are selected on one split and scored on the other, and the reverse, so no protein contributes to both selection and evaluation.

Amino-acid type is decodable from the code at AUROC $=1.0$, and for several annotations the residue is the mechanism itself: a disulfide requires cysteine, and N-linked glycosylation requires the asparagine of its sequon. Holding residue type fixed therefore measures what a feature contributes beyond that prior: catalytic-site localization holds at within-AA AUROC $=0.87$ on the full Mechanism and Catalytic Site Atlas (M-CSA) Ribeiro et al. [23], and residue burial detection holds at $0.95$. Amino-acid type acts in two places: on the evaluation of a detector, and on the selection of which feature becomes the detector. As Table 4 reports, choosing by best raw AUROC returns a near-pure single-residue feature whose glycosylation score reads $0.950$ raw and $0.563$ within-AA, while re-selecting under the control recovers a feature that carries the site beyond amino-acid type ($0.782$) and lifts active site to $0.974$ and metal binding to $0.924$. Where the annotation is type-independent, the two rules select the same feature, as for the signal peptide. The gap between them measures how much a raw-AUROC benchmark rewards amino-acid type, and the residual within-AA score measures the structure a feature carries beyond a legitimate prior.

Property	$n$	AA prior only	AA prior res.	raw AUROC naive	raw AUROC ctrl	within-AA AUROC naive	within-AA AUROC ctrl
Disulfide	771	0.995	C†	0.956	0.892	0.768	0.908
Glycosyl.	395	0.972	N‡	0.950	0.691	0.563	0.782
Active site	1220	0.896	H	0.863	0.581	0.846	0.974
Metal bind.	926	0.894	D	0.792	0.605	0.759	0.924
Mod. res.	664	0.845	S	0.722	0.560	0.680	0.802
Transmem.	260	0.720	L	0.744	0.572	0.777	0.771
Signal pep.	568	0.699	L	0.881	0.881	0.965	0.965
Ligand bind.	1525	0.673	G	0.771	0.540	0.747	0.761

Feature-level detection. Scored directly on residue annotations, single features localize structural and functional sites, with disulfide reaching AUROC = 0.97 at the deepest block. A distance-to-centroid statistic already reaches catalytic sites at AUROC = 0.82 on M-CSA, so a simple geometric prior recovers much of the raw signal alongside amino-acid type, an effect the protein-SAE literature has left largely unmeasured. As an external-validity check, the same within-AA catalytic detector applied to single-sequence AlphaFold structures collapses to AUROC between 0.51 and 0.62, consistent with the catalytic code riding the evolutionary signal ESM-2 $L_{36}$ supplies rather than geometry the trunk computes.

Applications of the interpreted detectors

Across the seven chosen detectors, the activation sites of Figure 4 separates point-localized features from distributed ones, as Table 5 reports: disulfide, metal-binding, and transmembrane features concentrate on their annotated residues, significant against a residue-permutation null on 68–91% of proteins, while the general functional signal shared by the active-site and ligand annotations spreads across the pocket rather than onto a one- to three-residue point.

Concept	Feat.	AUROC	IoU	% sig.
Disulfide	5934	0.98	0.79	91
Carbohydrate	8061	0.95	0.13	52
Active site	2819	0.80	0.03	9
Signal peptide	6323	0.79	0.35	73
Transmembrane	5591	0.77	0.46	68
Metal binding	1314	0.76	0.48	76
Ligand binding	2819	0.74	0.11	38

Annotating uncharacterized proteins. These detectors read the residue-level half of the code, the part inherited from the sequence model, and they put the interpreted dictionary to work on proteins that carry no labels. Applied to 300 human proteins whose UniProt entries carry no functional-site annotation [28], the $b_{47}$ detectors, thresholded at a precision operating point calibrated on annotated proteins, flag sites on 231 of them. Predicted disulfide cysteines pair spatially, (594 of 1,121 flagged cysteines, over 201 proteins, take a Cβ–Cβ partner within 6.5 Å of the predicted fold); predicted signal residues fall in the N-terminal 35 residues (634 of 999, over 84 proteins); predicted glycosylation sites land on the N-X-[ST] sequon (82 of 154, over 46 proteins). In total 127 proteins carry at least one prediction that survives its respective sanity check, yielding a concrete, coordinate-level hypothesis about a protein with an empty annotation record.

Disease enzymes and variant effects. On a panel of 139 brain-disease proteins, with detectors selected out of sample on the annotation scan, the trunk code localizes annotated sites wherever ESMFold is confident (mean pLDDT $\ge70$; 236 of 261 protein×site-type evaluations): active site auroc = 0.899, metal binding 0.815, ligand binding 0.778, where a published sequence dictionary [25] at 0.650, 0.543, 0.574 and a distance-to-centroid geometric prior at 0.825, 0.802, 0.775; an evolutionary-conservation baseline stays competitive throughout (0.927, 0.883, 0.826). In the low-confidence regions of the panel’s disordered members, among them APP and α-synuclein, the trunk score falls to 0.58–0.67, so the code annotates where a fold exists. Variant effects sit at the edge of what the code supports: wild-type feature density is at chance on ClinVar missense variants [13], while the disruption of the code between wild-type and mutant reaches auroc ≈ 0.75, level with the same sequence dictionary scored identically (0.731) and below the masked-marginal sequence-likelihood ceiling (0.887) [17]. Apparent per-feature $\Delta\Delta G$ prediction dissolves to auroc = 0.55 under cross-validation grouped by protein, so the higher plain-split number reflected protein leakage. The sparse code localizes structure that is folded well, and variant ranking is behind sequence likelihood.

Circuit attribution

For a scalar output $F$, an input $x$, and a baseline $x'$, the integrated gradient of input coordinate $i$ is [26]

For a block pair $(b_{\mathrm{src}}, b_{\mathrm{dst}})$ the input is the upstream code $z$ at $b_{\mathrm{src}}$ and the baseline is the zero code $x'=0$, which decodes to the mean reconstruction. Splicing $\operatorname{decode}(\alpha z)$ into $s_s$ at $b_{\mathrm{src}}$ and running the forward pass to $b_{\mathrm{dst}}$ yields the downstream code $z^{\mathrm{dst}}(\alpha z)$, so the code edge (3) instantiates (6) with the scalar output $F=z^{\mathrm{dst}}_j$ and baseline $x'=0$. We evaluate (3) with an $N$-step midpoint Riemann sum, accumulating gradient×input online into the per-upstream-feature edge vector at each step so that peak memory is constant in sequence length, and gradient-checkpoint the spliced span.

Representational similarity between the sequence model and the trunk

Figure 2A reports linear CKA over 9,000 residues sampled across 4,155 proteins from 60 shards of the corpus; each entry is the mean over five subsamples (maximum standard deviation 0.012). CKA is computed in feature space, $\lVert Y^{T}X\rVert_{F}^{2}/(\lVert X^{T}X\rVert_{F}\lVert Y^{T}Y\rVert_{F})$ for column-centred $X$, $Y$, agreeing with the Gram form to six decimals. ESM-2–trunk layer pairs average 0.14 and ESM-2’s final layer one against $b_{00}$ reaches 0.23, while the trunk blocks agree with one another at 0.65 and the middle blocks at 0.96. Linear CKA is invariant to orthogonal transforms and isotropic scaling but not to general linear maps, so it captures how the trunk reorganizes the similarity structure over residues; the decodability probe of Appendix G and the detection results speak to representational content. The block redundancy is what makes five evenly spaced blocks a sufficient sample of the 48.

Pairwise distance beyond ESM-2

For 60,000 residue pairs across 500 proteins, we regress Cα–Cα distance on each representation. The representation dictionary code reaches distance $R^2 = 0.69$ at $b_{47}$ and 0.64 at $b_{24}$, as Table 6 reports. No single-residue representation recovers it under the pairwise probe $[s_i; s_j; s_i \mathbin{\odot} s_j]$, including the trunk’s own ESM-2-3B backbone. The comparison is scale-matched, and the pair advantage holds at intermediate blocks that no read-out head reads.

predictor	distance $R^2$ $b_{47}$	distance $R^2$ $b_{24}$
(representation regressed on distance)
$s_z$ dictionary code (pair SAE, TopK)	0.69	0.64
$s_z$ raw (pair vector, 128-d)	0.67	0.62
ESM-2-3B $L_{36}$, pairwise probe	−0.11	−0.11
ESM-2-3B $L_{24}$, pairwise probe	−0.20	−0.20
ESM-2-650M $L_{24}$, pairwise probe	−0.00	−0.00
sequence separation $|i-j|$ (trivial anchor)	0.03	0.03

Recycle similarity

ESMFold folds by running the trunk for $N+1$ passes at num_recycles=N and reading out the final-recycle state. The single-representation dictionaries are trained on activations captured at $N=1$, while the pair representation, the causal interventions of Appendix I, and ESMFold’s default inference use $N=3$. The two settings describe the same representation: the final-recycle single representation is invariant to the budget, so a dictionary trained under one reads the other without adjustment.

Refolding the same proteins at $N=1$ and $N=3$ and capturing the single representation with the same hook, the two states agree at cosine 1.0000 at $b_{00}$, 0.99997 at $b_{24}$, and 0.99979 at $b_{47}$ (relative $L_2$ 0.15%, 0.87%, 2.0%). Passing through the dictionaries, they select the same active features on 87–98% of residues (TopK Jaccard 0.87–0.98) with activation correlation 0.98–0.99, the quantity that drives detection. Tracking the representation across recycles 0–4, it reaches cosine $\ge0.9998$ of its final value by the first recycle at every block, and the remaining passes contribute subpercent refinement. The bulk of the representation is fixed after one recycle and the small residual carries the functional sharpening, so the detection, within-AA, and circuit analyses read almost identical states across cycles.

Causal ablation on pair circuit

The same causal test applies to the pair representation. For each attributed $s_z \rightarrow s_z$ edge we ablate the upstream pair feature at its block, on the pairs where it is active, and read its attributed downstream partner at the destination block within the same ESMFold forward pass, run at the model’s full recycle budget. Over 1,129 edge–protein cases the ablation moves that partner 1.64× (95% CI [1.58, 1.70]) a write matched on norm and on the signed cosine to the downstream read-out direction, exceeding that control on 88% of cases; against a magnitude-matched control alone the ratio is 31.7× ([29.2, 34.3]), the weaker test as in the single track. Predicted geometry moves with the intervention, distogram-$L_1$ at the target pairs shifting 1.81× the matched control.

Protein-level retrieval

We pool each protein’s code into a descriptor and retrieve labels by leave-one-out cosine $k$-nearest-neighbors on top-level enzyme class (EC), fold class (CATH), and superfamily (SCOP), comparing the trunk code against an identically trained ESM-2-3B dictionary. Pooled per protein, the two score alike on family labels, and the ordering depends on how the per-residue codes are pooled into a descriptor. Across three pooling protocols on the same proteins, concatenated mean and max pooling gives the trunk +0.026 on enzyme class and −0.008 to −0.034 on the three fold levels; standardizing each feature before pooling moves every task to between −0.024 and −0.084; and a binary descriptor that records which features fire, discarding magnitudes, moves every task to between +0.047 and +0.082. The trunk code therefore carries no protein-level advantage that survives the choice of descriptor, and we draw no claim from it.

Two properties of the benchmark explain why. Family labels propagate by sequence homology, so retrieval rewards a model trained on evolutionary context: stratifying queries by the sequence identity of the nearest protein carrying the same label, 2,243 of 2,378 CATH queries have a relative above 50% identity, and the regime where structural similarity would separate the two representations is absent from the set. Pooling also discards the pairwise structure that distinguishes the trunk, collapsing a relation between residues into per-residue marginals, which is the quantity Section 4.2 measures directly.

Pair-feature label evaluation

The interpreted labels of the pair features are functionally meaningful at the level of the geometry a feature carries. We group the pair features by the concept named in their label, blind to any task, and probe each group on the geometric relations defined from coordinates. Selecting by label rather than by task performance is the point: a probe trained to predict a relation can succeed on a leaked correlate, whereas the label was assigned before and independently of any such target. For each label group we fit a probe on those features alone and read held-out AUROC on two relations, strand pairing and the $\alpha$-helix $i,i+4$ contact. Each group predicts its own relation above the other, as Table 7 reports: $\beta$-labeled features reach 0.938 on strand pairing and 0.839 on the helix contact, while helix-labeled features reverse it, 0.989 to 0.737, with the ordering preserved at $b_{47}$. Both clear a random group of the same size, which reaches 0.675 and 0.665, while the generic-contact group carries the signal the two relations share and sits between them.

Features labeled	$\beta$-pair	helix $i,i+4$	$\beta$-pair	helix $i,i+4$
$\beta$-sheet / strand	0.938	0.839	0.932	0.900
$\alpha$-helix	0.737	0.989	0.834	0.993
generic contact	0.829	0.943	0.803	0.901
random same-size set	0.675	0.665	0.683	0.742
full dictionary	0.965	0.992	0.960	0.993

Each label group predicts its own relation above the other and above a random group of the same size, a double dissociation that holds at both blocks. The labels therefore track the specific geometry a feature carries, and the residual cross-prediction is the contact signal the two relations share, which the generic-contact group carries at both targets.

Prompt inductive bias

The two representations are labeled with different questions, so the split between them could in principle follow the question rather than the evidence. We relabel the same features under the other representation’s question, holding the context fixed, and ask whether either question moves a track toward the other’s character.

Our central intuition about the division of labor between single semantic and pair geometric representations holds under swapped prompting. The pair vocabulary is 91% geometric under its own open question and 99% geometric under the single representation’s identity-versus-structure question, so asking about residue identity leaves the pair features reading as geometry. The single vocabulary is 57% geometric under its own question and 32% under the pair representation’s relational one, so asking about spatial relations moves the single features away from geometry rather than toward it. The prompt swapping clears the inductive bias of the prompt on the single and pair representations, verifying the fundamental difference between the information carried by both representations.

The full attribution

The circuit of Figure 3 summarizes four attributions. Each is reported here in full and at the level of the individual features rather than the categories, so the labels the pipeline wrote are visible alongside the routes they carry. Figure 9 and Figure 10 give the two directions that end in the single representation and Figure 11 and Figure 12 the two that end in the pair representation, each across all four block pairs, with the strongest attributed chains drawn. The shares in the titles recover the division of labor the main text reports: the pair representation accounts for most of the attributed mass, of which the forward $s_s \to s_z$ direction supplies about a third, while the backward $s_z \to s_s$ direction is the smallest of the four.

Biochemical sanity check

Each attributed edge is checked against known biochemistry by asking whether the two interpreted endpoints have an established relation. The check is a fixed rule set applied automatically to the label text: an amino-acid-identity endpoint carries a list of keywords for the geometry its residue is known to favor, a structural endpoint carries the set of geometry classes its category is known to relate to, and an edge is considered supported when it matches the opposite endpoint’s label. Table 8 gives the cross-track rule set in full.

Residue	geometry keywords credited to it
Gly	turn, local, bend, loop, flex, coil, hinge, termin, cap
Pro	turn, local, kink, bend, cap, loop, coil
Cys	disulf, c yste, bond, long-range, tertiary
Ile, Val	pack, buri, hydroph, sheet, strand, tertiary, long-range, core
Leu	pack, buri, hydroph, helix, tertiary, long-range, core
Ala	pack, buri, helix, hydroph, core
Met	pack, buri, hydroph, core, tertiary
Phe, Trp	pack, buri, aromat, stack, hydroph, core, tertiary, long-range
Tyr	pack, buri, aromat, stack, hydroph, core, surface, tertiary
Asp	turn, local, cap, surface, expos, salt, charg
Glu	helix, surface, expos, salt, charg
Lys	surface, expos, salt, charg, long-range
Arg	surface, expos, salt, charg, long-range, stack
His	metal, stack, aromat, coordinat
Asn	turn, local, cap, surface, expos, polar
Gln	surface, expos, polar, helix
Ser	turn, local, cap, surface, polar, hydroxyl
Thr	turn, local, cap, strand, sheet, surface, polar
Category	geometry classes credited to it
contact	long-range tertiary, beta-sheet pairing, helix packing, local/turn
burial	long-range tertiary, helix packing
secondary structure	beta-sheet pairing, helix packing, local/turn
functional site	long-range tertiary, local/turn
disulfide	long-range tertiary

An edge between two amino-acid-identity features is grounded when both read the same residue; an edge joining a residue to a geometry is grounded when that residue has a documented propensity for it, glycine for turns and local geometry, cysteine for disulfide pairing, hydrophobic and aromatic residues for packing and buried contact; an edge between two structural features is grounded when their categories have an established relation, burial with packing or secondary structure with strand pairing. A pair endpoint in the catch-all geometry class carries no specific claim and is excluded rather than counted either way.

The rules support 79% of the edges within either representation, with 609 of 770 in the single track and 120 of 152 in the pair one. The grounded pair edges credit genuine matched secondary-structure relations. Across the tracks, the forward $s_s \to s_z$ leg reaches 75% of edges and 66% of attributed weight, and the backward $s_z \to s_s$ leg reaches 63% and 46% of edges and attributed weight, so the leg that carries the least mass is also the one biochemistry supports least.

The unsupported edges fall into two kinds, and only one of them is plausibly an artifact. Some carry a local cause into long-range geometry: a helix-termini-and-tight-turns feature writes a long-range exposed-coil contact, and a loop-and-turn feature writes a long-range contact. Our criterion counts these as unsupported for a mechanical reason: Table 8 records no relation between a locally acting category and a fold-spanning geometry class, so any such edge is rejected by absence. These rejections reflect the limited coverage of the rule set rather than a fault in the circuit. Others pair a residue with geometry resisted by its chemistry: a helix $i$, $i+4$ contact writes into a valine feature although $\beta$-branched residues disfavor helix, and a mixed glycine-and-histidine feature writes into a cysteine disulfide pair, where no propensity connects the endpoints and the pair label names a residue the upstream feature does not detect. Throughout, the correspondence we draw is between residue-level causes and the geometry a trained network computes from them. However, the residue propensities are statistical properties and further verification is needed to accept or reject an attribution edge.

Auto-interpretation prompt details

Each feature is described to the language model by a context block and a question. The two representations use different context formats, since one activates on residues and the other on residue pairs, and the questions differ accordingly. Both prompts are reproduced in full in Figure 13 and Figure 14, with the evidence block abbreviated to a single exemplar line. The cross-prompt control of Appendix L exchanges the two questions while holding each context block fixed, so a single-representation feature is asked the open pair question and a pair feature the identity-versus-structural question.

Splitting the vocabulary into sequence and geometry

A category counts as sequence-determined when it names residue identity or a sequence offset, which covers the side-chain-chemistry and sequence-position categories, and as geometric when it names a relation between residues or a 3D

environment, which covers residue contact and burial. Three categories fall cleanly on neither side: secondary structure, which a sequence model already predicts well although it describes backbone geometry; functional site, which is annotated per residue although the site is a spatial arrangement; and disulfide, which requires two cysteines in sequence and a bond in space. Those are excluded from both sides rather than assigned, 2,097 of the 5,595 single-representation features and 350 of the 2,972 pair features, so the shares reported are over the features whose category settles the question. The counts behind Figure 7A are 3,498 and 2,622 features under the native questions and 662 and 906 under the swapped ones.