WorksComputer Science
A Bounded Evaluation Protocol for Representational Systematicity
Abstract
Multiple studies have shown that neural networks can be behaviorally systematic, producing correct outputs for novel combinations of familiar parts. Whether they are also representationally systematic, that is, whether they represent these parts and their roles and use them to produce the outputs, remains hard to test, because each kind of evidence can mislead on its own: correct outputs can come from shortcuts, a decodable signal need not be used, an intervention can act through unintended pathways, and a failed search does not show absence. We propose a bounded evaluation protocol for representational systematicity (Bounded-RS): for an explanation that attributes a systematic behavior to a declared representation, it tests on a matched behavioral contrast whether that representation is recoverable and whether it is causally used, checks the validity of this evidence, and assigns the explanation one of four bounded states, Unsupported, Present-only, Causal-use, or Undetermined, through control-calibrated scores, gates, and stop rules. On a semantic-parsing testbed of 96 matched sets that we release, the tested role-filler Binding explanation of fine-tuned T5 models is Causal-use and the distributed Geometry explanation is Unsupported within its declared search space, verdicts that hold across calibration-compatible parameter configurations, training seeds, and T5-base. On 41 cases with known causal status, full adjudication licenses a false Causal-use in 1 of 34 noncausal cases, against 13 when recoverability is taken as use, without missing any causal case. Bounded-RS thus turns the question of representational systematicity, debated since Fodor and Pylyshyn, into comparable and falsifiable verdicts on declared explanations, each reported with the evidence pattern that licenses it and, for a negative, the search space that bounds it.
Introduction
Systematicity is the capacity to understand and produce novel combinations of familiar constituents by preserving their roles and relations: a semantic parser that maps “John loves Mary” to a meaning in which John is the lover should also map “Mary loves John” to the role-reversed meaning. A model can show this capacity in its behavior, by succeeding on held-out recombinations, without thereby representing the relevant constituents and roles and using them to produce its outputs; this second property, representational systematicity, is the stronger claim [28]. Whether neural networks achieve systematicity through such representations has been debated since Fodor and Pylyshyn [6] argued that connectionist models lack them, and the question has gained weight as neural networks reach human-like systematic behavior in controlled tasks [17]. Answering it is hard because each kind of evidence fails in a characteristic way: correct outputs can come from shortcuts [9, 21], a signal decodable from hidden states need not be used by the model [4, 11, 25], an intervention can change outputs through off-manifold states or unintended pathways [19, 32], and a failed search shows only that the tested operations did not find a mechanism. A credible representation-level claim therefore requires several kinds of evidence and explicit rules that state which claim each pattern of evidence licenses.
Existing work on behavioral and representational systematicity supplies this evidence but usually leaves these rules implicit (§ 2). Behavioral benchmarks such as SCAN, CFQ, COGS, ReCOGS, and SLOG make the input–output contrast precise [14, 15, 16, 18, 31] but do not identify the representation behind success. Representation-level work proposes explanations of that success, such as role–filler binding [20, 26] and distributed geometric structure [3, 12], and supplies tools to test them, such as probes, control tasks, interchange interventions, and activation patching [1, 7, 11, 22]. Each tool tests one link between a signal and behavior, and each explanation is usually tested on its own task and judged by its own standard, so it is left open which claim a given pattern of evidence licenses. The same evidence can then be reported as claims of very different strength, and verdicts on different explanations cannot be compared.
We close this gap with a bounded evaluation protocol for representational systematicity, BOUNDED-RS, which orchestrates these tools to adjudicate representation-level explanations of a fixed behavioral contrast (§ 3). Its data layer fixes the contrast with matched examples. Its evidence layer collects, for a declared explanatory family, a behavioral anchor, recoverability evidence, generation-time interchange, necessity, and repair interventions with matched controls, and realism and item-linkage checks. Its decision layer maps this evidence through control-calibrated scores, gates, and stop rules to one of four bounded states: Unsupported, Present-only, Causal-use, or Undetermined. A negative about a whole family is licensed only when every admissible object of a frozen closure basis fails; the basis is a declared set of object classes, and extending the negative beyond it requires validated detectors. Each design choice blocks one of the failure modes above, and the rules are frozen on calibration controls before any target outcome is interpreted, so BOUNDED-RS fixes in advance which claim, and at what scope, each pattern of evidence licenses.
We instantiate BOUNDED-RS on a controlled semantic-parsing testbed of 96 matched sets, whose seven items change one contrast at a time and provide exact interchange and repair pairs (§ 4). T5-small models [24], whose internals are fully accessible, are fine-tuned on 64 sets and evaluated on 32 held-out sets, which include all six sets of a construction absent from training. On this testbed we operationalize two explanations of systematic generalization as potentially coexisting families: BINDING, which locates the explanation in role–filler signals, and GEOMETRY, which locates it in distributed objects such as predicate-conditioned relation subspaces, cross-step bridges, and query–source relations. The two families share the behavioral anchor, state semantics, and claim rules, and each supplies its own detectors, interventions, and controls.
The results show that the two families receive different verdicts (§ 5). The tested BINDING instantiation is Causal-use: its role–filler signal is recoverable beyond surface controls, and interventions on it change generation beyond matched controls. The declared GEOMETRY family is Unsupported after closure: no admissible object in its frozen basis passes the causal-use (necessity) gate and no Present-only reading holds, and since no geometry detector is validated as a truthful monitor, the verdict stays at the family level. Both verdicts hold across 487 calibration-compatible parameter configurations and five training seeds, matched-set resampling yields no positive GEOMETRY reversal, an engineered truth control shows that the cross-step GEOMETRY branch can return all three evidential states, both verdicts replicate on T5-base, and the BINDING chain returns Causal-use in two of three seeds on a pre-specified SLOG subdomain. Full adjudication is also more reliable than its component rules: on 41 cases with known causal status, it licenses Causal-use falsely in 1 of 34 noncausal cases, against 13 when recoverability is taken as use and 7 when interventions are ungated, while retaining all 7 causal cases and abstaining on 6 of the 41.
In Section 6 we argue that bounded adjudication matters beyond systematicity: representation-level claims become comparable and falsifiable once each is reported with the evidence pattern that licenses it and, for a negative, the search space that bounds it.
Related Work
Behavioral systematicity. Systematicity has mainly been evaluated through behavior: benchmarks withhold recombinations of familiar parts from training and score the outputs on them, and they differ in which parts they recombine. SCAN tests new compositions of familiar primitives in navigation commands [16]; CFQ tests new compounds of familiar atoms in questions paired with database queries [14]; and COGS, its revision ReCOGS, and SLOG test familiar words in new syntactic roles and structures when sentences are parsed into logical forms, with SLOG focused on new structures [15, 18, 31]. These benchmarks make the behavioral contrast precise, but a correct output is compatible with structured representations and with shortcuts alike [28], so they cannot show whether a model represents the constituents and their roles and uses them to produce its outputs.
Representational systematicity. Work on representational systematicity proposes what produces systematic behavior and supplies tools to test such proposals. The proposals range from role–filler binding [5, 20, 26, 27] to distributed geometric structure [3, 12]. Probes test whether a variable is decodable from hidden states [1, 2], and control tasks and amnesic probing test whether the decoding reflects the representation and whether the model relies on it [4, 11]. Intervention methods such as activation patching and interchange interventions test whether manipulating a hypothesized variable changes outputs [7, 8, 22, 29, 30]. Faithfulness work defines what a correct explanation must reflect [13], and interpretability benchmarks score how well these methods localize mechanisms, some against known circuits [10, 23]. In short, these tools each test one link between a signal and behavior, and each explanation is tested on its own task by its own standard, so they cannot say which representation-level claim their combined evidence licenses or compare explanations with each other. BOUNDED-RS instead adjudicates all these links for every explanation on one matched behavioral contrast under rules frozen in advance, and therefore yields what they cannot: comparable verdicts on different explanations, negatives bounded by a declared search space, and far fewer false causal claims than decodability or ungated interventions alone (§ 5.3).
The Bounded-RS Protocol
BOUNDED-RS takes a behavioral systematicity result and a declared explanation of it and returns the strongest representation-level claim that the evidence licenses. Its inputs are a matched-set testbed for the behavioral contrast, a family specification that declares the hypothesized signal together with the operations that detect it, intervene on it, and could falsify it, and a control-calibrated decision rule; the testbed, state semantics, scoring grammar, gates, and stop rules are shared across families, whereas each family supplies its specification and the calibration controls that fit its boundaries (Appendix § 12). Its output is a state together with the evidence rows that justify it. The protocol has three layers (Figure 1): the data layer fixes the behavioral contrast with matched examples (instantiated in § 4.1), the evidence layer collects family-specific evidence about the hypothesized signal (§ 3.2), and the decision layer maps this evidence to a state (§ 3.3). Each design choice blocks one misreading from § 1: behavior enters only as an anchor, so excellence cannot license use; recoverability and causal use are separate evidence roles, so a decodable signal is not called used; matched controls and realism and linkage gates keep effects of unintended pathways from counting as use; and closure over a frozen basis keeps a failed instantiation from being read as absence. We define the four states first, because they fix what the evidence and decision layers must establish.

Figure 1. Three-layer structure of BOUNDED-RS. The matched-set testbed defines controlled contrasts; family-specific evidence operations test recoverability and causal use; realism, linkage, and adjudication rules translate the evidence into bounded state decisions.
State semantics
Table 1 defines the four states as evidential decisions under the declared operations. Unsupported means that the tested operations did not support the signal or its use, not that no such representation exists. Present-only names the case that probe-only studies leave open: the signal is recoverable, but causal-use evidence is insufficient. Undetermined is an explicit abstention when the evidence does not license a stronger state (§ 3.3).
Evidence layer
Each evidence role rules out one alternative reading of the evidence (Table 2). The behavioral anchor, exact match and clause-level F1 on the matched recombinations, establishes that the behavior to be explained is present. Recoverability tests whether the family’s target signal is readable from hidden states beyond surface, filler, and predicate controls. Intervention tests act during generation: interchange transfers the signal between matched items, necessity removes or dampens it, and repair inserts or restores it from a matched easy source into a hard target with the same meaning (Section 4.1), and each effect is compared with the strongest of the family’s matched controls, such as wrong-role, shuffled, or generic-decoder-gain controls. Validity checks reject effects that come from off-manifold intervention states (realism) or that appear only in aggregate rather than on the items the explanation predicts (item linkage). No single block is decisive; Appendix B defines each block and its metrics.
Decision layer: scoring and adjudication
The decision layer first puts heterogeneous evidence on a common support scale. Each evidence block yields scalar features, and a raw feature with boundary , scale , and direction is transformed as
so that a feature at its boundary scores 0.5 and sets how quickly support saturates. Group scores for behavior, presence, and use are weighted averages of their features, where required features must be present and optional ones are skipped when structurally unavailable; each state then receives a state
| Evidence role | Blocks | What it rules out |
| Behavioral anchor | Exact match and clause-level F1 on matched recombinations. | Representation-level claims when the target behavior is absent or unstable. |
| Recoverability | Family-specific presence readout with surface, filler, and predicate controls. | Claiming causal use for a signal that is not stably recoverable. |
| Intervention tests | Interchange, necessity, and repair interventions during generation. | Treating a merely decodable or correlational signal as used by the model. |
| Validity checks | Realism, item-linkage, shuffled, wrong-role, non-role, and generic-gain controls. | Accepting off-manifold interventions, aggregate-only effects, or generic decoder improvements. |
caption: tbl:2: Evidence roles in BOUNDED-RS and the alternative reading each rules out.
:::
score from how well the group scores match the evidence pattern it requires (Table 1), and realism, linkage, coverage, and margin act as gates. Behavior enters only as a prior with at most a 4% share of the decision weight, so behavioral success alone cannot produce Causal-use. All scoring parameters are fixed on calibration controls before target and closure outcomes are interpreted (Appendix § 10), and Appendix § 11 follows one matched set through every step.
| Decision | Minimum evidence pattern | Licensed claim |
| Unsupported | Weak, unstable, or unlinked presence evidence. | The protocol does not support the relevant representation or its causal use. |
| Present-only | Positive recoverability evidence, with insufficient intervention, necessity, or repair evidence. | The representation is recoverable, but causal use is not supported. |
| Causal-use | Recoverability plus controlled intervention, necessity, or repair evidence, with realism and linkage gates passing. | The representation is supported as present and causally used under the tested operations. |
| Undetermined | Incomplete evidence, state-score margin below threshold, or failed or conflicting gates. | No stronger state decision is licensed. |
Table 1. State decisions in BOUNDED-RS. Appendix I maps the state names to artifact labels.
The decision layer then applies stop rules that decide which state the scores license. Causal-use additionally requires the realism and linkage gates to pass, and Undetermined is returned when evidence coverage does not exceed 0.80, when the top state leads the next by no more than 0.10, or when the gates that the top state requires fail or conflict. A positive state supports the tested family instantiation under its declared operations. A negative about a whole family needs more, because the space of instantiations is open: the family must freeze a closure basis, a declared set of object classes whose candidate objects are the admissible carriers of the signal, and every admissible object in it must fail the required criteria before a family-final Unsupported is licensed. Extending such a negative to all mechanisms of the family’s kind, an ontology-level claim, further requires detectors validated as truthful monitors under known positive, negative, and null conditions.
Evaluation Setup: Testbed and Family Instantiations
We instantiate the protocol for one behavioral contrast and two explanatory families, which may coexist in one model and therefore receive separate verdicts. This section describes the testbed and audited models (§ 4.1), the panels that calibrate and apply each decision rule (§ 4.2), and the two family instantiations (§ 4.3 and § 4.4); Figure 2 summarizes the design.
Figure 2. Two-family audit workflow. Layer trajectories and routes are GEOMETRY diagnostics; the closure basis consists of interaction objects, cross-step bridges, and query–source relations. A family-final GEOMETRY negative requires closure, and an ontology-level negative would further require validated detectors.
Controlled semantic-parsing testbed and audited models
The testbed is built for exact counterfactual comparison. Each matched set starts from one semantic design (predicate, agent, theme, alternate fillers, agent and theme modifiers, voice, and optional relative-clause context), whose meaning is written as a logical form such as help(agent=freya,theme=kira,agent_mod=brisk,theme_mod=careful), and is rendered into seven items (Table 3). A matched easy source in the active voice without embedding and a hard target in the set’s construction share the gold meaning, which gives repair a source–target pair with aligned semantics; five further items each change one contrast, which supplies aligned pairs for interchange and controlled contrasts for the anchor, the readouts, and necessity. Each item is realized from validated templates (Appendix § 8), giving 672 items in 96 complete matched sets (28 active and 31 passive sets without embedding, 31 active and 6 passive sets with a relative clause), with 192 interchange pairs and 96 repair pairs, 68 of which have a hard target that differs from its source. All items are synthetic and contain no personal or sensitive information.
| Matched item | Controlled relation | Protocol role |
| Matched easy source | Same gold meaning as hard target; active voice without embedding. | Provides source signal for repair and matched behavior controls. |
| Hard target | Same gold meaning; the set’s construction (passive voice, a relative clause, or neither). | Target for repair and failure analysis. |
| Agent replacement | Only the agent filler changes. | Tests role-specific sensitivity to who acts. |
| Theme replacement | Only the theme filler changes. | Tests role-specific sensitivity to who is acted on. |
| Role swap | Agent and theme exchange roles. | Tests role identity rather than name memorization. |
| Filler replacement | Both role fillers change. | Tests recombination under the same role structure. |
| Modifier swap | Agent and theme modifiers exchange attachment. | Tests modifier binding and attribution. |
Table 3. The seven items of each matched set, shared by both families.
We audit T5 models [24] because their internals are fully accessible, which makes generation-time interventions tractable, and because their encoder–decoder split lets a control keep the encoder-side signal while weakening the decoder’s access to it (§ 4.2). T5-small is fine-tuned on 64 matched sets (448 items) for 8 epochs with AdamW (learning rate , batch size 8) and decodes greedily; the 32 held-out sets (224 items) form a development split of 128 items and a generalization split of 96 items (Appendix § 8). Passive sentences with a relative clause never occur in training, and all six such sets (42 items) are held out, so success requires recombining a voice and an embedding that were seen only separately. The normally fine-tuned target of the main panels reaches exact match and clause-level F1 of 1.0 on both held-out splits, which establishes the behavior whose representational basis is audited.
Audit panels
Each family applies its frozen rule to calibration, held-out, and target panels, whose entries are runs: one T5 checkpoint, possibly modified, processed by that family’s evidence operations to yield one state. Calibration controls are built to instantiate intended states: Causal-use controls are trained on the full training split, Present-only controls keep the encoder-side signal of such a checkpoint but weaken the decoder’s access to it, for example by scaling decoder cross-attention to 0.25, and Unsupported controls are trained on permuted targets or on a small subset of the training split. Calibration controls fix the boundaries and gates, and held-out controls built the same way from independent seeds test the frozen rule. Target panels contain the checkpoints to be adjudicated: the normally fine-tuned target that carries the claim and regimes built to probe the other states. Closure instead tests objects and serves only a family-level negative: once a closure basis is frozen, every admissible object in it is tested, so target states describe instantiations while closure decides the family. Beyond these panels, the two verdicts are tested with a parameter grid, additional training seeds, a matched-set bootstrap, engineered truth controls, and ports to T5-base and SLOG, and full adjudication is compared with its component rules on a separate panel (Appendix § 14, Table 12).
Binding instantiation
BINDING explains systematicity through role–filler assignments, that is, by keeping track of which entity fills the agent role and which fills the theme role across recombinations [20, 26]. Its target signal is a role-indexed hidden-state signal or an aggregate role–filler signal. Presence readouts recover role and filler information from hidden states; interchange and repair patch role-indexed signals across matched items, notably from the matched easy source into the hard target; and necessity removes or dampens role-relevant information and tests for selective degradation of the parse. Wrong-role, non-role, shuffled, path-only, and generic-decoder controls tie every effect to the declared role–filler operation, so an improvement on the hard target counts as evidence for binding only if it exceeds what these controls produce.
Geometry instantiation and closure basis
GEOMETRY explains systematicity through distributed structure that need not be localized in one role-indexed vector [3, 12]. Its operations target relation subspaces, layer trajectories, cross-attention routes, cross-step bridges, and query–source relations, and fill the same evidence slots as BINDING while manipulating geometric objects instead of a single vector. GEOMETRY is calibrated on its own control suite of independently trained checkpoints, so its calibration shares no checkpoint with BINDING (Appendix § 13); these controls calibrate the operation-level rule that precedes closure.
Because geometry-like explanations form an open-ended space, a negative about GEOMETRY needs a frozen closure basis (§ 3.3). An object class enters the basis only if it admits an observable object, a matched detector, object-specific intervention and repair, and falsifying controls. Three classes qualify: interaction objects, layer-local relation subspaces read conditionally on the predicate by a predicate-conditioned relation detector; cross-step bridges, read by a cross-step bridge detector based on cross-step alignment, with a max-statistic correction for the candidate search; and query–source relations, read by an argument-locus detector that relates a decoding query to the relevant source argument. Layer trajectories and routes lack a closure-grade repair and control chain and remain diagnostics, and nonlinear manifolds, higher-order or polysemantic interactions, sparse-feature circuits, diffuse long-range dynamic routes, and nonlocal multi-source mixtures lie outside the basis (Appendix § 13.§ 1). Each class has 12 candidate objects built from templates that vary the head choice, the subspace basis, and the storage variant. Before closure, observability must be sealed, that is, a step aligner must locate on the controls the decoding steps at which objects are read. Discovery then keeps a candidate if its contrast on the calibration controls exceeds a minimum gap and accepts a class only if at least 8 of its 12 candidates survive, covering at least two decoding steps or step pairs and two templates; closure asks whether any object of an accepted class passes the causal-use (necessity) gate, and whether a Present-only reading holds on the decoder-scaled, trained-scaled, and readout-masked Present-only regimes of the control and target panels, over all items (the direct route) or within predeclared item slices (the slice-local route).
Results
Section 5.1 adjudicates BINDING, Section 5.2 adjudicates GEOMETRY and closes its basis, and Section 5.3 bounds the GEOMETRY negative, tests transfer to T5-base and SLOG, and compares full adjudication with its component rules. Table 4 summarizes the panels, and Appendix § 14 reports them in detail.
| Panel | Outcome | Diagnostic role | |
| BINDING calibration | 3 | all intended states | Controls separate the three evidential regimes. |
| BINDING held-out controls | 6 | acc. 0.833; macro-F1 0.822 | The frozen rule holds outside the calibration panel; one Present-only control is escalated. |
| BINDING target audit | 5 | 1 Causal-use, 1 Present-only, 1 Unsupported, 2 Undetermined | The normally fine-tuned target is Causal-use; other regimes are separated or withheld. |
| GEOMETRY held-out controls | 9 | acc. 0.778; macro-F1 0.796 | Present-only is the least stable GEOMETRY state. |
| GEOMETRY target audit | 7 | 2 Causal-use, 1 Present-only, 3 Unsupported, 1 Undetermined | Operation-level states before closure; no family-level claim. |
| GEOMETRY closure | 36 objects in 3 classes | 0 objects pass closure | Licenses a family-final Unsupported under the frozen basis. |
| Detector truthfulness | 3 detector classes | 0 validated detectors | Keeps the GEOMETRY verdict family-final. |
Table 4. Panel-level results. Calibration outcomes, GEOMETRY variant counts, and detector-truthfulness checks are detailed in Appendix § 14; per-run measurements are in the released reports.
Binding audit
We first check that the frozen BINDING rule recovers known states (Table 4, first three rows). The three calibration controls receive their intended states, and on six held-out controls, two per state, the rule is correct on five (accuracy 0.833, macro-F1 0.822); the one error escalates a Present-only control to Causal-use. Removing the behavior features leaves both held-out metrics unchanged (Appendix § 14.§ 1). The rule thus holds outside its calibration panel, behavior decides no state, and its weak point is the Present-only boundary.
We then apply the rule to the five-run target panel, which contains the normally fine-tuned target and four regimes built to probe the other states. The normally fine-tuned target is Causal-use: its role signal is recoverable beyond surface controls, and its interchange, necessity, and repair effects exceed the strongest matched controls with realism and linkage gates passing. The target derived from it by scaling decoder cross-attention to 0.25 keeps a recoverable signal and becomes Present-only, the target trained on 128 items is Unsupported, and an instruction-prompted checkpoint with the same scaling and a target trained on longer surface realizations are Undetermined, because their evidence does not separate any state clearly enough. The rule thus returns all four states, and only the normally fine-tuned target is Causal-use.
Finally, we test whether this verdict depends on the frozen parameters, the training seed, or the sampled matched sets. It holds in all 487 calibration-compatible configurations, namely those of a 729-configuration grid over every verdict-relevant boundary, scale, weight, and gate that keep every calibration control in its intended state, and in five of five training seeds. For the first seed, the interchange advantage over the strongest control is 0.337 (95% matched-set bootstrap interval [0.301, 0.371]) against the 0.245 use boundary, and the intervals for necessity, repair, realism, and linkage also lie on the supporting side of their decision boundaries (Appendix § 14.§ 4). The tested BINDING instantiation is therefore Causal-use: in these models, systematic behavior on the held-out matched sets is supported by a role–filler signal that is causally used during generation.
Geometry audit
As in § 5.1, we first check the frozen GEOMETRY rule on held-out controls (Table 4, rows 4–5). It recovers all six Unsupported and Causal-use controls but only one of three Present-only controls (accuracy 0.778, macro-F1 0.796), so Present-only again is at least reliable state.
Applied to the seven-run target panel, the rule gives operation-level states: the two fully trained targets are Causal-use, three targets are Unsupported, one is Present-only, and one is Undetermined, and five further variants built to produce GEOMETRY Present-only regimes either collapse to Unsupported or Causal-use or retain a single Present-only case (Appendix § 14.§ 2). These states show that a geometry transform changes generation but not which geometric object carries the change; the variant that attenuates a relation-bearing cross-attention subspace (Variant D), for example, is Causal-use on interchange evidence alone, with necessity tied to the strongest control and no repair evidence.
Operation-level states therefore cannot support a family-level claim, and whether any geometric object carries a causal effect is decided by closure.
Closure proceeds in the order of Table 5. Observability is sealed first: the selected aligner, A1, covers the predicate step in all 1,536 control trace rows, while argument steps remain unaligned and are not treated as object evidence. Discovery then keeps 16 of the 36 candidate objects but accepts only the cross-step class, so interaction and query–source objects cannot pass the causal-use (necessity) gate, and all 12 cross-step objects fail it; neither the direct nor the slice-local Present-only route holds. No admissible object of the frozen basis thus passes closure. This negative holds in all 487 calibration-compatible configurations and in five of five training seeds, and across 1,000 matched-set bootstrap replicates no object passes closure and the family is Unsupported in 998 and Undetermined in 2, never Causal-use (Appendix § 14.§ 4).
| Closure component | Candidates | Passed | Failure mode |
| Observability seal | 1,536 trace rows | sealed | A1 selected; predicate-step coverage 1.0; argument steps not treated as validated object evidence. |
| Interaction objects | 12 objects; 4 survivors | 0 | Class not accepted at discovery, so no object can pass the causal-use (necessity) gate. |
| Cross-step bridges | 12 objects; 8 survivors | 0 | Class accepted; all 12 objects fail the causal-use (necessity) gate: gaps below threshold, null bands violated, realism below threshold. |
| Query–source relations | 12 objects; 4 survivors | 0 | Class not accepted at discovery, so no object can pass the causal-use (necessity) gate. |
| Direct Present-only route | 8 object–regime rows | 0 | No global Present-only route. |
| Slice-local Present-only route | tested slices | 0 | No local slice passes. |
Table 5. GEOMETRY closure audit. The verdict is family-final under the declared closure basis.
Finally, an engineered truth control checks that the cross-step branch, the only accepted class, can find a causal object when one exists. In in-basis triplets where a designated signal is causally used, equally recoverable but unused, or replaced by matched null content, blind discovery finds the designated object in all three causal seeds, and the branch returns Causal-use, Present-only, and Unsupported correctly in 3/3 seeds each, with a correct-object advantage of 0.335 above the 0.245 use boundary (Appendix § 14.§ 5). The negative is therefore not forced by the search, and the declared GEOMETRY family is Unsupported at the family level: in these models, no admissible object of its frozen basis passes closure.
Scope, transfer, and reliability of the verdicts
In terms of scope, the GEOMETRY verdict stays at the family level. An ontology-level negative would require detectors validated as truthful monitors in trained models (§ 3.3), but none of the three detector classes passes this validation, each tested on 336 canonical examples (Appendix § 14.§ 3).
In terms of transfer, both verdicts carry over to T5-base, and the BINDING chain runs unchanged on a prespecified subdomain of an established benchmark. With the protocol and family interfaces unchanged, three T5-base seeds pass the behavioral anchor, held-out controls are correct in 16 of 18 BINDING and 22 of 27 GEOMETRY cases, and under both the frozen T5-small rules and rules recalibrated on T5-base controls, all six seed-by-rule runs return BINDING Causal-use and GEOMETRY family-final Unsupported. On a 96-pair exact-contrast subdomain of SLOG [18], selected before outcomes were viewed, the unchanged BINDING chain returns Causal-use, Undetermined, and Causal-use for three T5-small seeds (Appendix § 14.7).
In terms of reliability, full adjudication makes far fewer false causal claims than its component rules. On a frozen panel of 41 predeclared cases with known causal status, taking recoverability as evidence of use licenses Causal-use falsely in 13 of 34 noncausal cases, and licensing it from ungated interventions does so in 7; full BOUNDED-RS does so in 1, retains all 7 causal cases, and abstains on 6 of the 41 (Appendix § 14.§ 6).
Discussion
We introduced BOUNDED-RS, which adjudicates representation-level explanations of a behavioral systematicity result into four bounded states, and applied it to two explanatory families on a controlled semantic-parsing testbed. Applied to fine-tuned T5-small and T5-base models, it licenses Causal-use for the tested role–filler BINDING instantiation and a family-final Unsupported for the declared GEOMETRY family, and both verdicts hold across calibration-compatible parameter configurations and training seeds, with no positive GEOMETRY reversal under matched-set resampling. The same discipline applies wherever internal representations are invoked to explain behavior. A probing study that decodes role information would report Present-only until controlled interventions pass, and our component-rule panel shows why: taking recoverability as use licensed false causal claims in 13 of 34 noncausal cases, whereas full adjudication licensed one. A study that finds no mechanism of some kind would report a family-final negative relative to a declared basis, which later work can extend or contest by adding object classes. Reported this way, representation-level claims from different studies and about different explanations become comparable and falsifiable rather than a collection of probe accuracies and patching effects, and a systematicity benchmark can report, beside accuracy, which representational explanations of that accuracy are licensed and at what scope. To support such reporting, we release the testbed, the protocol code, and the audit reports (Appendix § 15).
All audits use T5 models; the verdicts have not been tested on other architectures or on large language models. The testbed is synthetic and template-clean, which makes exact interventions possible but leaves naturalistic language untested; transfer beyond our generator is shown only on one prespecified subdomain of SLOG, itself a synthetic benchmark. Both rules are least reliable at the Present-only boundary, where they recover one of two and one of three held-out controls. The GEOMETRY verdict holds only within the declared closure basis and does not cover the classes excluded in § 4.4; its detectors remain unvalidated in trained models, and only the cross-step class has an engineered known positive. BOUNDED-RS adjudicates declared explanations and cannot discover a mechanism that no family declares, and each new family must supply and validate at least six mechanism-specific interfaces, plus a closure basis for a family-level negative. Finally, the protocol adds no new probing or intervention primitive, so its verdicts inherit the limits of the tools it adjudicates.
Acknowledgments and Disclosure of Funding
This work received no third-party funding. The authors declare no competing interests.
Data Construction Details
Each matched set is created from one semantic design: a predicate, an agent filler, a theme filler, alternate fillers for the replacement items, an agent modifier, a theme modifier, a voice, and an optional relative-clause context. The matched easy source uses an active, non-embedded realization, and the hard target keeps the gold meaning in the set’s construction, so in the 28 active sets without embedding the two coincide. The other five items each change one contrast: agent replacement, theme replacement, role swap, filler replacement, and modifier swap. The gold output is a logical form that names the predicate, its role fillers, and their modifiers.
The renderer produces three surface candidates per item, and validation checks that the required fillers, modifiers, and predicate surface are present, that the active or passive order is consistent, that the relative-clause context appears when required, and that no pronouns occur. All 2,016 candidates are valid, and one realization is selected per item, which gives 672 canonical items in 96 complete matched sets with exact counterpart coverage 1.0, 192 interchange pairs, and 96 repair pairs. The sets comprise 28 active and 31 passive sets without embedding and 31 active and 6 passive sets with a relative clause (196, 217, 217, and 42 items). The generator uses seed 13.
Splits are drawn at the level of matched sets. The 64 training sets (22 active and 21 passive sets without embedding and 21 active sets with a relative clause, with 154, 147, and 147 items) contain 448 items and no passive item with a relative clause. Of the 32 held-out sets, 16 form the development split with all seven items, and 16 form the generalization split, whose six non-source items are scored for generalization while their matched easy sources join the development split; this gives 128 development and 96 generalization items, and all six passive sets with a relative clause (42 items) are held out. A surface-shifted realization that keeps the longest valid template per item, built for 48 of the 96 matched sets (224 training items), supplies the surface-shifted targets. Earlier, less controlled surface variants are retained in the artifact as stress tests but enter no reported decision, because the intervention protocol requires exact counterpart coverage.
Detailed Evidence Blocks
Section 3.2 summarizes the evidence layer by role, and Table 6 gives the block-level specification. Each block produces metrics that enter the behavioral anchor, the presence score, the causal-use score, or the validity gates, and no single block can license Causal-use.
| Block | Core question | Evidence produced | Controls or rejection checks |
| Behavioral anchor | Does the model exhibit the target systematic behavior on the matched contrast? | Exact match and clause-level F1 between generated and gold logical forms; pass, fail, and control strata. | Used only as an anchor; cannot by itself assign a state or produce Causal-use. |
| Presence readout | Is the family-specific target signal recoverable from hidden states? | Readout accuracy, controlled readout gaps, and stability across matched items. | Surface position, filler identity, predicate identity, and other simple correlates. |
| Interchange intervention | Does transferring the signal across matched items change generation in the predicted direction? | Intervention effect and advantage over the strongest matched control. | Wrong-role, non-role, and shuffled controls. |
| Necessity intervention | Does removing or dampening the signal selectively degrade the target output? | Selective degradation of target parses relative to controls. | Degradation must be selective rather than a global disruption of generation. |
| Repair intervention | Does inserting a matched source signal into a hard target repair its output? | Repair advantage on hard targets over controlled repair baselines. | Path-only, wrong-role, non-role, shuffled, and generic-decoder-gain controls. |
| Item linkage | Do improvements occur on the items that the explanation predicts? | Item-linked effect pass rate and aggregate-versus-item diagnostics. | Rejects aggregate-only improvements not linked to intended items. |
| Realism gate | Do interventions stay within the model’s natural computational regime? | Natural-state realism pass rate of intervention states. | Rejects off-manifold or unrealistic interventions. |
Table 6. Evidence blocks of BOUNDED-RS. The main text groups them into behavioral anchoring, recoverability, intervention tests, and validity checks.
The behavioral anchor computes exact match and clause-level F1, which gives partial credit for correctly generated semantic atoms and defines behavioral pass, fail, and control strata. Presence readouts provide recoverability evidence only; for BINDING, the role readout is the leave-one-out nearest-centroid accuracy of semantic roles on final-layer hidden states, and the surface-control gap compares it with a readout of surface position. A high readout means that a signal can be recovered, not that the model uses it.
The intervention blocks provide causal-use evidence only when their effects are selective and control-calibrated. Interchange compares a matched signal transfer with wrong-role, non-role, and shuffled alternatives; necessity tests whether removing the signal causes selective semantic degradation rather than generic output damage; and repair tests whether inserting a matched source signal improves the hard target beyond path-only, wrong-role, non-role, shuffled, and generic decoder-gain controls. Realism and item linkage are gates rather than score features: they block stronger states when intervention effects are off-manifold or not tied to the intended items.
Full Adjudicator Configuration
Section 3.3 describes the decision layer, and this appendix records the configuration used for the reported audits. The tables below list the predeclared features, boundaries, scales, weights, gates, and calibration sources that can affect a state decision; all of them are frozen on calibration controls before target and closure outcomes are interpreted.
Score features
Table 7 lists the score features, with their evidence group, scoring configuration, and the calibration source that fixed the boundary.
| Evidence group | Feature | Scoring configuration | Calibration source |
| Behavior | Clause-F1 anchor | Direction +; boundary with floor 0.070312; scale ; weight 0.15 prior. | Behavioral pass/fail controls |
| Presence | Semantic-role readout accuracy | Direction +; boundary ; scale ; weight 0.60. | State-validating controls |
| Presence | Surface-control gap | Direction +; boundary ; scale ; weight 0.40. | Surface, filler, and predicate controls |
| Use | Interchange or repair advantage over strongest control | Direction +; boundary ; scale ; weight 0.60. | Causal-use vs. Present-only controls |
| Use | Permutation or shuffled-control significance | Direction −; boundary ; scale ; weight 0.40. | Shuffled and wrong-role controls |
Table 7. Score features of the reported audits. Weights are normalized within each group except for the behavior prior. Gates and required-status constraints are given in Table 8.
Gates and stop rules
Table 8 lists the gates and stop rules. Unlike score features, they are not averaged into a group score; they are required conditions for licensing the corresponding state.
| Gate | Criterion | Required for | Calibration source |
| Realism gate | Natural-state realism pass rate must exceed 0.60. | Intervention states | Realism calibration controls |
| Linkage gate | Item-linked effect pass rate must exceed 0.50. | Intervention states | Item-linkage controls |
| Coverage gate | Required evidence coverage must exceed 0.80. | All state decisions | Held-out control panels |
| Decision gate | Top-state score margin must exceed 0.10. | States stronger than Undetermined | Held-out control panels |
Table 8. Gates and stop rules. Gate rows are required conditions rather than score features.
The behavior feature acts only as a low-weight prior and cannot by itself produce Causal-use. Presence features support recoverability when the target signal is readable beyond surface and identity controls. Use features support causal use only when the intervention or repair effect exceeds the strongest matched control and the required validity gates pass. The coverage and decision-margin gates prevent stronger states when evidence is incomplete or the top state is not sufficiently separated from the alternatives.
End-to-End Walkthrough
This appendix follows one matched set, MS-047, through the three layers of a BINDING audit of a normally fine-tuned T5-small target. The set instantiates the seven-item construction of Appendix § 8 from an explicit semantic design, and the panel-level behavioral anchor that licenses the audit is evaluated separately and passes. Table 9 lists the evidence at each stage.
| Stage | Evidence | Result |
| Behavior | Exact match 6/7 = 0.857; clause-F1 0.947. | Behavior score 0.977. |
| Presence | Role readout 0.938; surface-control gap 0.071. | Both presence features pass. |
| Controlled use | Interchange 0.512 vs. strongest control 0.181; necessity 0.276 vs. global control 0.058; repair 0.464 vs. strongest control 0.161. | Interchange and repair advantages 0.331 and 0.303. |
| Validity and decision | Realism 0.714, linkage 0.571, coverage 0.857. | All gates pass; final state Causal-use. |
Table 9. End-to-end trace of matched set MS-047 in the BINDING audit.
Each number maps onto the configuration of Appendix § 10. The behavior score applies Eq. (1) to clause-F1 with and , so sigmoid; behavior enters only as a prior, so this score cannot decide the state. The role readout and the surface-control gap exceed their boundaries of 0.914 and , so presence is supported. The interchange and repair advantages exceed the use boundary of 0.245, and necessity exceeds its global control. Realism (0.714 > 0.60), linkage (0.571 > 0.50), and coverage (0.857 > 0.80) pass their gates, and the set is Causal-use. The released repository contains the seven inputs of MS-047 with their gold logical forms and exact pairs, and a script that runs this trace on a checkpoint with the same audit and decision function (Appendix § 15).
Reusable and Family-Specific Scaffold
The shared scaffold fixes the state semantics, evidence slots, scoring grammar, gates, stop rules, closure driver, and reporting format, whereas each explanatory family specifies the signal it claims to explain, the operations that detect and manipulate that signal, and the controls that would count against the explanation (Table 10). This division is the extension contract of the artifact: a new family reuses the matched-set format, state names, evidence slots, and report structure, and supplies its own operationalization of the hypothesized signal, which keeps each state tied to a declared explanation.
The extension cost is measurable. In our implementation, the shared schema, scoring, gates and stop rules, closure driver, and reporting span 18 files and 2,164 lines of code, and the BINDING and GEOMETRY operations span another 18 files and 3,584 lines. Porting both existing families to T5-base touched 257 lines and took 11.4 development hours with no interface change, and porting the BINDING chain to the SLOG subdomain required a 705-line adapter with no change to the family operations. These are ports of existing families. A new family must instantiate and validate at least six mechanism-specific interfaces, namely its target signal, detector, interchange, necessity, and repair operations, and controls, and a seventh, a closure basis, for a family-level negative.
| Layer | Shared component | What a new explanatory family must provide |
| Data | Matched sets for the behavioral contrast. | Usually unchanged when the family explains the same contrast; redesigned when the contrast changes. |
| Evidence slots | Behavioral anchor, presence readout, interchange, necessity, repair, item linkage, and gates. | Target signal, detector, interchange, necessity, and repair operations, and controls. |
| Adjudication | State semantics, scoring grammar, gates, stop rules, and closure driver. | Control-fitted feature boundaries and, for a family-level negative, a closure basis. |
| Artifact | Data generator, protocol runners, reports, and documentation. | Documentation of operations, expected runtime, and failure modes. |
caption: tbl:10: Reusable and family-specific parts of BOUNDED-RS.
:::
Per-Family Operationalization Details
For BINDING, the object is a role–filler signal. Interchange patches a role-indexed hidden vector or an aggregated role signal from a matched source into a matched recipient, necessity removes or dampens role-relevant information, and repair moves the corresponding role–filler signal from the matched easy source into the hard target. BINDING calibration controls are T5-small checkpoints trained on the full training split (Causal-use), the same checkpoints with decoder cross-attention scaled to 0.25 (Present-only), and checkpoints trained on permuted targets or on 64 or 128 items (Unsupported). Its target panel contains the normally fine-tuned target, two checkpoints with decoder cross-attention scaled to 0.25 (one raw-prompted and one instruction-prompted), a target trained on 128 items, and a target trained on the surface-shifted realization.
For GEOMETRY, the object is not assumed to be a single vector. The operations include subspace projection and removal, relation-subspace tests, trajectory and layer-path summaries, cross-attention access and bottlenecking, cross-step bridges, and query–source relations; they correspond functionally to the BINDING interventions, but manipulate geometric relations, routes, or trajectories. GEOMETRY has an independent control suite of T5-small checkpoints trained with a fixed encoder feature mask (keep ratio 0.75): fully trained checkpoints serve as Causal-use controls; checkpoints whose decoder access is weakened, by scaling cross-attention to 0.25 at inference or during training or by moderately masking the decoder readout, serve as Present-only controls; and checkpoints trained on permuted targets or on a small subset of the training split, or with a strongly masked readout, serve as Unsupported controls. Variants A–D were built to produce GEOMETRY Present-only regimes: Variant A blocks cross-attention access, promoted Variant A applies deterministic layer policies to it, Variant B projects out a decoder relation subspace, Variant C imposes a low-rank cross-attention bottleneck, and Variant D attenuates a seeded relation-bearing subspace of the cross-attention output.
Geometry closure basis
An object class enters the closure basis only if it supports a complete chain from an observable object and a matched detector to object-specific intervention and repair and falsifying controls. Table 11 lists the three classes that meet this criterion. Layer trajectories and routes are kept as diagnostics because the artifact lacks a closure-grade repair and control chain for them. Nonlinear manifolds, higher-order or polysemantic interactions, sparse-feature or sparse-autoencoder circuits, diffuse long-range dynamic routes, and nonlocal multi-source mixtures lie outside the basis; they are omitted classes, not negative cases within it.
Each class has 12 candidate objects, formed from its decoding steps and templates that vary the head choice (top-1 head or a top-2 bundle), the basis (single or bundled), and the storage variant (matched store or none). Before closure, three step aligners (leftmost exact subsequence, slot-aware literal field, and teacher-forced bridge) were compared on the frozen controls; all align predicate steps at rate 1.0 and argument steps at rate 0.0, so observability is sealed as predicate-dominant with aligner A1. A candidate survives discovery if its contrast on the calibration controls exceeds a minimum gap, and a class is accepted when at least 8 of its 12 candidates survive, covering at least two decoding steps or step pairs and two templates. Closure then requires, for a family-final Unsupported, that observability
| Object class | Matched detector | What the detector tests |
| Interaction objects | Predicate-conditioned relation detector | A layer-local relation subspace read conditionally on the predicate. |
| Cross-step bridges | Cross-step bridge detector (cross-step alignment) | Whether relation information forms a traceable route across decoder steps; a max-statistic controls multiplicity over the frozen candidate search. |
| Query–source relations | Argument-locus detector | Whether a decoding query bears a structured relation to the relevant source argument. |
caption: tbl:11: Frozen GEOMETRY closure basis and matched detectors.
be sealed, that every class finish discovery and no object pass the causal-use (necessity) gate, with generic decoder gain as a mandatory matched control, that the direct and slice-local Present-only routes fail on the decoder-scaled, trained-scaled, and readout-masked Present-only regimes, and that the Unsupported and Causal-use controls remain stable.
Additional Result Details
This appendix reports the per-panel details behind § 5, followed by the robustness, truth-control, and transfer analyses that test the two headline verdicts and by the comparison of full adjudication with its component rules. Table 12 indexes these analyses.
| Analysis | Design | Conclusion |
| Parameter sensitivity | 729-configuration grid; 487 calibration-compatible | Both verdicts hold in all 487 configurations; the nearest closure flip lies outside the compatible region. |
| Training seeds | Five T5-small seeds | BINDING Causal-use and GEOMETRY Unsupported in 5/5. |
| Matched-set bootstrap | 1,000 cluster replicates over 96 sets | No GEOMETRY object passes closure; no Causal-use reversal. |
| Truth controls | Engineered triplets, three seeds | The cross-step GEOMETRY branch and BINDING recover all three evidential states. |
| Component rules | 41 predeclared cases with known causal status | Full adjudication licenses the fewest false Causal-use states and keeps all causal cases. |
| T5-base transfer | Three seeds; frozen and recalibrated rules | Both verdicts replicate in 6/6 seed-by-rule runs. |
| SLOG transfer | 96-pair subdomain; three seeds | The BINDING chain returns Causal-use in two seeds and Undetermined in one. |
caption: tbl:12: Analyses that test the two headline verdicts, and the comparison of full adjudication with its component rules.
Binding calibration and target details
| Panel | Outcome | Note | |
| Initial calibration | 3 | all intended states | Separates Unsupported, Present-only, and Causal-use controls. |
| Held-out controls | 6 | acc. 0.833; macro-F1 0.822 | Two controls per state. |
| Behavior-feature ablation | 6 | no change in acc. or macro-F1 | Behavior feature share is 0.041. |
| Initial target audit | 3 | all Causal-use | Mean top-state score and decision margin are both 1.0. |
| Heterogeneous target audit | 5 | 1 Causal-use, 1 Present-only, 1 Unsupported, 2 Undetermined | Normally trained, weak, decoder-scaled, instruction-prompted scaled, and surface-shifted targets. |
caption: tbl:13: BINDING calibration and target-panel details.
Table 13 shows that the held-out BINDING panel separates the Unsupported and Causal-use regimes and that its one error escalates a Present-only control to Causal-use. The calibration therefore checks the state semantics; it is not a claim of probabilistic calibration.
Geometry candidate and variant details
The nine held-out GEOMETRY controls comprise two Causal-use, three Present-only, and four Unsupported regimes. The frozen rule recalls Unsupported and Causal-use controls perfectly but Present-only controls at 0.333, and returns Undetermined once (accuracy 0.778, macro-F1 0.796). On the target panel (Table 14), the two fully trained targets are Causal-use; the weak target, a trained intermediate regime, and a strongly readout-masked regime are Unsupported; a second trained intermediate regime is Present-only; and a moderately readout-masked regime is Undetermined. The variants either collapse to one state or retain a single Present-only case. Variant D, which is Causal-use in all six runs, shows why these outcomes stay at the operation level: its interchange effect is strongly positive against all controls, but its necessity effect is tied with the strongest control and repair evidence is unavailable, so it supplies no object-specific necessity of the kind closure requires.
| GEOMETRY panel | Causal-use | Present-only | Unsupported | Undetermined | |
| Pre-closure target audit | 7 | 2 | 1 | 3 | 1 |
| Variant A | 6 | 3 | 1 | 2 | 0 |
| Promoted Variant A | 6 | 0 | 1 | 5 | 0 |
| Variant B | 6 | 0 | 0 | 6 | 0 |
| Variant C | 6 | 0 | 0 | 6 | 0 |
| Variant D | 6 | 6 | 0 | 0 | 0 |
Table 14. GEOMETRY pre-closure target and variant outcomes. These are operation-level outcomes and do not by themselves license a family-level GEOMETRY claim.
Detector-truthfulness details
Detector truthfulness is separate from closure. Closure asks whether any admissible object in the frozen basis survives discovery and passes the required gates; detector truthfulness asks whether the detectors themselves monitor geometry-like objects correctly under known positive, negative, and null conditions in trained models. Each detector class is tested on 336 canonical examples with full coverage in two stages, with 8 and 6 targets. All three classes are anchorable, but none passes the full validation: the argument-locus and cross-step detectors fail object-specific observability, and all three fail the necessity and Present-only tests on known positives (Table 15). The strongest licensed GEOMETRY conclusion therefore remains family-final.
| Detector class | Canonical examples | Coverage | Validation outcome |
| Argument locus | 336 | 1.0 | not validated |
| Predicate-conditioned relation | 336 | 1.0 | not validated |
| Cross-step bridge | 336 | 1.0 | not validated |
Table 15. Detector-truthfulness checks for geometry-like objects.
Sensitivity, training seeds, and matched-set bootstrap
The sensitivity analysis varies every verdict-relevant parameter block over a predeclared grid of six blocks with three levels each ( configurations), spanning score boundaries, scales, and weights, the realism, leak, coverage, and margin gates, and the GEOMETRY discovery, necessity, and null thresholds. A configuration is calibration-compatible if it preserves the intended states of the calibration controls, a rule that does not look at any target; 487 configurations are compatible. Across all 487, the claim-bearing BINDING target remains Causal-use, and no GEOMETRY object passes closure, so the family keeps its family-final Unsupported state. A joint worst-case search that favors GEOMETRY and disfavors BINDING produces no change in either verdict, and the nearest GEOMETRY closure flip lies outside the compatible region at normalized distance 3.75. The only rows that change are peripheral and candidate rows that carry no claim, including Present-only/Undetermined transitions and one candidate Causal-use/Undetermined transition.
Across five fixed, independent T5-small training seeds, BINDING is Causal-use in 5/5, and no GEOMETRY object passes closure, leaving the family Unsupported, in 5/5. For seed 0, a cluster bootstrap that resamples the 96 matched sets 1,000 times yields no GEOMETRY object passing closure in any replicate, with a family-final Unsupported state in 998 and Undetermined in 2, and never Causal-use. The 95% bootstrap intervals for the BINDING evidence are 0.337 [0.301, 0.371] for interchange, 0.284 [0.249, 0.321] for necessity, 0.318 [0.279, 0.356] for repair, 0.729 [0.667, 0.786] for realism, and 0.646 [0.583, 0.708] for linkage, all on the supporting side of their decision boundaries.
Truth controls
The truth controls test whether each pipeline can return all three evidential states when the truth is known by construction. For the cross-step GEOMETRY class and for BINDING, matched triplets with the same architecture, task, budget, and paired seeds place a designated signal in one of three conditions: present and causally required, equally recoverable but not causally used, or replaced by matched null content. With blind discovery and independent confirmation, the GEOMETRY pipeline recovers the designated object in 3/3 causal and 3/3 Present-only seeds. Both pipelines return Causal-use, Present-only, and Unsupported correctly in 3/3 seeds each, and neither produces a false Causal-use in its six noncausal runs. In the causal condition, the GEOMETRY correct-object advantage is 0.335 against 0.309 for BINDING, a difference of 0.026 within the pre-frozen band, and above the use boundary of 0.245002. These controls validate engineered in-basis sensitivity for the cross-step class only. BINDING does not carry the same blind-localization and multiplicity burden, and no equivalence interval was estimated, so the comparison does not establish equal power across families.
Component-rule comparison
The component-rule comparison asks whether full adjudication changes conclusions relative to its components. A frozen panel contains 41 predeclared cases whose causal status is known by construction, 7 causal and 34 noncausal. Three license rules are applied to the same evidence (Table 16): the heuristic that recoverability implies causal use, a rule that licenses causal use from intervention effects without the validity gates and stop rules, and full adjudication.
| License rule | False Causal-use (non-causal cases) | Causal cases retained | Abstentions |
| Recoverable ⇒ causal | 13/34 | 7/7 | – |
| Ungated intervention | 7/34 | 7/7 | – |
| Full BOUNDED-RS | 1/34 | 7/7 | 6/41 |
Table 16. False Causal-use licenses on 41 cases with known causal status. Full adjudication covers 35 of the 41 cases.
Taking recoverability as evidence of use licenses Causal-use in 13 of the 34 noncausal cases, and licensing it from ungated interventions does so in 7. Full adjudication does so in 1, retains all 7 causal cases, and abstains on 6 cases.
T5-base and SLOG transfer
The T5-base audit uses three fixed training seeds and freezes the testbed, split, state semantics, family definitions, closure classes, controls, reporting schema, and target-independent relative-depth, head, and rank mappings. It applies both the T5-small rules directly and rules recalibrated only on T5-base calibration controls. All three behavioral anchors pass (exact match 0.949–0.961, clause-F1 0.974–0.981). On held-out controls, BINDING is correct in 16 of 18 seed-control cases and GEOMETRY in 22 of 27, and the direct and recalibrated labels agree on all 45 paired entries. In all six seed-by-rule runs, BINDING is Causal-use and no GEOMETRY object passes closure, leaving the family Unsupported, with no change to any family interface. Because the T5-base audit contains no class-specific GEOMETRY known positive, it establishes cross-scale verdict replication but not positive sensitivity of the GEOMETRY detectors at that scale.
For SLOG, a 96-pair exact-contrast subdomain was selected on structural grounds before any outcome was viewed, and the complete BINDING data–evidence–decision chain was run with three T5-small seeds. Unchanged adjudication returns Causal-use, Undetermined, and Causal-use. The port required a 705-line adapter and no change to the family operations.
Supplementary Artifact Package
The released repository contains the matched-set generator, schemas, canonical items, and validation reports; the evidence blocks, decision layer and its frozen scoring configurations, closure driver, and detector-truthfulness harness; per-run measurement summaries from the calibration, target, closure, and detector-truthfulness audits; training and audit code and runners for the robustness, truth-control, transfer, and component-rule experiments; and the inputs and script of the walkthrough of Appendix § 11. Its structure mirrors the paper: the matched-set files correspond to § 4.1, the protocol code to § 3.2 and § 3.3, and the reports to § 5. Trained checkpoints are not distributed. The testbed is produced deterministically by the released generator (seed 13), and regeneration reproduces the released files byte for byte; the generated files and a data card describing their construction, intended use, and limitations are included in the repository, and the code and data are released under the MIT license. The audits use public T5 checkpoints [24] (Apache 2.0) and the SLOG benchmark [18] (MIT license).
The examples are synthetic and contain no human-subject or personal data. The artifact is intended for auditing representation-level claims under exact matched contrasts, not for ranking general model performance.
Terminology and Artifact Mapping
The paper uses descriptive names, and the released reports use shorter internal labels for the same objects. Table 17 maps one to the other. The word family is reserved for explanatory families.
| Paper term | Artifact label |
| Unsupported, Present-only, Causal-use, Undetermined | H0, H1, H2, underdetermined |
| Presence readout, interchange, necessity, repair, item linkage | Modules M1, M2, M3, M4, M5 |
| BINDING and GEOMETRY families | G1_binding and G2_distributed_geometry |
| Matched set | counterpart_family |
| Matched easy source and hard target | rescue_donor and rescue_target |
| Repair | rescue |
| Agent replacement, theme replacement, filler replacement | agent_swap, theme_swap, filler_swap |
| GEOMETRY Variants A–D | GEOMETRY H1 families A–D |
Table 17. Paper terms and the corresponding artifact labels.
References
- [1]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (Workshop Track), 2017.arxiv.org/abs/1610.01644
- [2]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.arxiv.org/abs/2102.12452
- [3]Silvia Bernardi, Marcus K. Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and C. Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4):954–967.e21, 2020.
- [4]Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021.
- [5]Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In International Conference on Learning Representations, 2024.
- [6]Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1–2):3–71, 1988.
- [7]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 9574–9586, 2021.arxiv.org/abs/2106.02997
- [8]Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pages 160–187, 2024.arxiv.org/abs/2303.02536
- [9]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.DOI
- [10]Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adria Garriga-Alonso. InterpBench: Semi-synthetic transformers for evaluating mechanistic interpretability techniques. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, pages 92922–92951, 2024.arxiv.org/abs/2407.14494
- [11]John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743. Association for Computational Linguistics, 2019.arxiv.org/abs/1909.03368
- [12]Takuya Ito, Tim Klinger, Doug Schultz, John Murray, Michael Cole, and Mattia Rigotti. Compositional generalization through abstract representations in human and artificial neural networks. In Advances in Neural Information Processing Systems, volume 35, pages 32225–32239, 2022.arxiv.org/abs/2209.07431
- [13]Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205. Association for Computational Linguistics, 2020.arxiv.org/abs/2004.03685
- [14]Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.09713
- [15]Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105. Association for Computational Linguistics, 2020.arxiv.org/abs/2010.05465
- [16]Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2873–2882, 2018.arxiv.org/abs/1711.00350
- [17]Brenden M. Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985):115–121, 2023.DOI
- [18]Bingzhi Li, Lucia Donatelli, Alexander Koller, Tal Linzen, Yekun Yao, and Najoung Kim. SLOG: A structural generalization benchmark for semantic parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3213–3232. Association for Computational Linguistics, 2023.
- [19]Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. In International Conference on Learning Representations, 2024.arxiv.org/abs/2311.17030
- [20]R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. RNNs implicitly implement tensor-product representations. In International Conference on Learning Representations, 2019.arxiv.org/abs/1812.08718
- [21]R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. Association for Computational Linguistics, 2019.arxiv.org/abs/1902.01007
- [22]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372, 2022.arxiv.org/abs/2202.05262
- [23]Aaron Mueller, Atticus Geiger, Sarah Wiegrefe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. MIB: A mechanistic interpretability benchmark. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 45069–45108, 2025.arxiv.org/abs/2504.13151
- [24]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
- [25]Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363–3377. Association for Computational Linguistics, 2021.arxiv.org/abs/2005.00719
- [26]Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46(1–2):159–216, 1990.
- [27]Paul Soulos, R. Thomas McCoy, Tal Linzen, and Paul Smolensky. Discovering the compositional structure of vector representations with role learning networks. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 238–254. Association for Computational Linguistics, 2020.DOI
- [28]Ivan Vegner, Sydelle de Souza, Valentin Forch, Martha Lewis, and Leonidas A. A. Doumas. Behavioural vs. representational systematicity in end-to-end models: An opinionated survey. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31842–31856. Association for Computational Linguistics, 2025.
- [29]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401, 2020.
- [30]Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations, 2023.arxiv.org/abs/2211.00593
- [31]Zhenyuan Wu, Christopher D. Manning, and Christopher Potts. ReCOGS: How incidental details of a logical form overshadow an evaluation of semantic interpretation. Transactions of the Association for Computational Linguistics, 11:1719–1733, 2023.
- [32]Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In International Conference on Learning Representations, 2024.arxiv.org/abs/2309.16042
