On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Introduction

Large language models (LLMs) store knowledge in their parameters, making information outside or newer than their pre-training data hard to access or update [13]. Such external knowledge may be organised as text-attributed graphs [7]. We can incorporate this information into an LLM either by textualizing the graphs, i.e., constructing verbalized descriptions of nodes and edges, or through graph soft prompting, i.e., providing a learned representation generated by a graph neural network (GNN). Verbalisation has two limitations: its token cost grows with the amount of graph content included, and serialisation imposes a sequence order for facts that have no intrinsic order. Expressive Graph soft prompting aims to offer an alternative interface for conveying relational information without verbalising every node and edge. G-Retriever [7] supplies the same underlying graph via both channels. However, this design introduces an identifiability issue: because the graph content is already present in textual form, improvements in answer accuracy alone cannot distinguish whether the model relies on the graph token, the verbalized graph, or both. Furthermore, the training objective does not explicitly enforce that the graph-derived token encodes graph-specific information when the textual channel suffices. Consequently, a token computed from the graph may degenerate into a graph-invariant soft prompt.

G-Retriever interprets its ablations as evidence that the graph encoder and verbalised graph provide complementary graph representations [7]. We revisit this interpretation by testing what the graph token contains and whether the LLM uses it.

This study evaluates the proposed mechanism by quantifying both the information encoded in graph-derived tokens and the information that the large language model (LLM) causally exploits during inference. The experimental protocol integrates token-space statistical analyses with fixed-weight causal interventions: individual tokens are substituted with (i) the training-set mean token, (ii) a token sampled from a different example, or (iii) a token recomputed from corrupted or different graph inputs. In addition to answer accuracy, we assess gold-answer log-likelihood to capture subtler shifts in the model’s output distribution, and we employ decoding-based probes to measure the recoverability of question, answer, and graph attributes from internal representations. Graph-free control conditions preserve the projector architecture and the full training pipeline while ablating the graph input itself.

Applying this protocol to G-Retriever uncovers an unanticipated phenomenon that we call token collapse, which also recurs in subsequent graph-prompting methods. Specifically, across several graph encoders, benchmark datasets, LLM families, and fine-tuning regimes, the token representations generated by G-Retriever are nearly invariant with respect to the input graph (Figure 1). Moreover, substituting these input-dependent projected tokens with a fixed alternative yields no resolved answer-level effect under the default configuration; indeed, a learned constant token that never observes any graph matches or even surpasses the performance of the full model. These results indicate that the collapsed token operates primarily as a soft prompt rather than as a carrier of graph-specific information. Although likelihood-based analyses show that perturbing the token can modulate the output distribution, such sensitivity alone is insufficient to demonstrate input- or graph-conditioned utilization.

We associate collapse with two factors: a projector that suppresses variation in the encoder representations and an answer objective that does not require this variation to be preserved. Our ablations implicate both factors, although they do not isolate either as the sole cause. We address them with a two-stage pipeline that first aligns and freezes the graph encoder, then trains separate graph-token and prompt-token pathways. The resulting tokens no longer collapse, but non-collapse alone guarantees neither graph use nor improved accuracy. Depending on the benchmark and training configuration, they may encode the question, an answer prior, graph information made redundant by the text, or graph information that the LLM uses to answer. This distinction matters because task improvements may otherwise be attributed to graph–language alignment even when they arise from the verbalised graph or a graph-independent soft prompt. By separating token variation, graph content, and downstream use, our protocol makes alignment empirically testable and provides concrete criteria for developing graph–language models that preserve and use task-relevant graph information beyond what is already supplied by the textual channel.

Contributions. We identify token collapse across G-Retriever benchmarks, graph encoders, LLM families, and tuning regimes, and find the same failure in recent graph-prompting methods. Using matched graph-free controls and fixed-weight interventions, we show that the collapsed token acts primarily as a soft prompt rather than as a graph representation used to answer. We associate collapse with the projector and end-to-end answer training, and propose a two-stage pipeline that mitigates it across the tested settings. Finally, we show that a non-collapsed token may encode question or answer priors, unused graph information, or graph information that changes the answer. Together, these results provide a practical protocol for testing graph-token alignment rather than inferring it from task accuracy.

Related work

Graph soft prompting. Soft prompting conditions an LLM with learned continuous vectors [12, 14]; graph soft prompting instead projects graph-encoder outputs into one or more such tokens. GraphToken [4], GraphLLM [3], GNP [24], GraphGPT [23], and G-Retriever [7] differ in their encoders, token counts, retrieval, and LLM adaptation. In this paper, we test whether G-Retriever and subsequent methods preserve and use graph-specific information in their projected tokens.

Evaluating graph–language models. Recent work questions whether existing objectives and benchmarks require graph–language integration. Zhang et al. [32] show that autoregressive training can favour a text-dominant solution, while Petkar et al. [18] find that single-modality and raw soft-prompt baselines remain competitive and introduce CLEGR to test compositional graph–text reasoning. We complement these studies by auditing the graph token–LLM interface of graph soft prompting. Extended related work discussion is in Appendix § 27.

Method

Preliminaries: graph soft prompting in G-Retriever

A textual graph $G = (\mathcal{V}, \mathcal{E}, \{x_v\}_{v \in\mathcal{V}}, \{x_e\}_{e \in\mathcal{E}})$ associates every node $v$ with a text attribute $x_v$ and every edge $e$ with a text attribute $x_e$. Each example $i$ consists of a graph, a question $q_i$, and an answer $y_i$.

Retrieval. Node, edge, and question texts are embedded by a frozen sentence encoder $\operatorname{LM}(\cdot)$ into $\mathbb{R}^d$ [20]. The respective representations are $z_v = \operatorname{LM}(x_v)$, $z_e = \operatorname{LM}(x_e)$, and $q_i = \operatorname{LM}(q_i)$.

When the complete graph exceeds the LLM’s context length or a predefined budget, only a relevant subgraph can be supplied to the model. Following G-Retriever, similarities between $q_i$ and the node and edge embeddings define retrieval relevance, and a prize-collecting Steiner tree is used to select a compact, connected subgraph [2]. We denote the resulting question-specific subgraph by $G_i$. When the complete graph fits within the budget, no retrieval is needed and $G_i = G$.

Graph token. A graph encoder $f_\phi$ processes $G_i$ using node features $z_v$, edge features $z_e$, and optionally the question representation $q_i$. G-Retriever uses a graph transformer by default [22]; we also evaluate GAT [27] and GCN [10]. The node representations are mean-pooled into $h_i \in\mathbb{R}^d$, and a projector $P_\psi$ maps the result into the LLM embedding space:

where $\sigma$ is the logistic sigmoid and the projector has hidden width $d_p$. We refer to $t_i$ as the graph token.

Generation. The retrieved subgraph is also verbalised as a list of nodes and edges, denoted by $\tau(G_i)$, and truncated to a budget of $L_{\max}$ tokens. In the default frozen-LLM setting, the model receives

The graph token therefore precedes the verbalised graph and the question. The LLM parameters $\theta$ remain fixed, while the graph encoder and projector are trained with the answer loss

The graph encoder does not receive the question directly, so in G-Retriever the token depends on the question only through retrieval.

The Graph Token Collapse Phenomenon

Task accuracy conflates three distinct questions: Does the token vary across inputs? Does the LLM use the graph token? What information does it contain? We address them with complementary measurements of token geometry, fixed-weight interventions, gold-answer likelihood, and decodability.

Does the token vary across inputs? We measure inter-sample cosine, the input-specific residual relative to the mean token, and token norm relative to ordinary word embeddings. Collapse appears as cosine near one and a small residual ratio. We also compute inter-sample cosine before projection, on the pooled encoder representations $h_i$, to determine where collapse arises. Formal definitions are given in Appendix D.

Does the LLM use graph-specific token variation? At fixed weights, REAL uses the original token, MEAN replaces it with the training-set mean, and SWAP uses another example’s token. Our main graph-specific intervention, GSWAP, recomputes the token from another graph while preserving the original question conditioning. NODE, EDGE, and REL corrupt individual graph components, while NOTXT removes the verbalised graph. Retrained no-entity and no-question variants isolate the information available during training. Because accuracy observes only changes in the decoded answer, we also measure the paired change in gold-answer negative log-likelihood under teacher forcing. This detects smaller changes in the output distribution. Full details on our interventions are in Appendix D.

What does the token contain? Recoverable content and downstream use are not equivalent: a token may contain information that the LLM ignores. This matters for our pipeline because Stage 1 is conditioned on the question and trained with an answer-related target. A non-collapsed token may therefore encode the question, an answer prediction or prior, the graph, or a mixture. The answer can be a supervision target rather than an input, so it cannot be removed through a fixed-weight intervention; appendix Table 8 maps the information retained by the remaining conditions.

Condition	What it replaces	question	entities	structure	relations
Interventions at fixed weights
MEAN	token $\to$ training-set mean $\mu$	×	×	×	×
SWAP	token $\to$ another question’s token	×	×	×	×
GSWAP	graph $\to$ another question’s graph	✓	×	×	×
NODE	node features $\to$ resampled features	✓	×	✓	✓
EDGE	edges $\to$ uniformly rewired edges	✓	✓	×	✓
REL	relation features $\to$ resampled features	✓	✓	✓	×
Pipelines retrained with one input removed
no entity features	node features zeroed during training and testing	✓	×	✓	✓
no question	question conditioning removed from Stage 1	×	✓	✓	✓
question MLP	graph encoder replaced by an MLP on $q$	✓	×	×	×

We complement the interventions with a post-hoc probe that maps stored tokens to frozen embeddings of the question, answer, or verbalised graph. The map is fitted on half of the test examples; on the held-out half, we report the median rank of each example’s correct target among all candidates. A constant token provides the control rank. Estimation details and limitations are given in Appendix D.

Graph-free controls. CONST passes a learned graph-independent vector through G-Retriever’s projector, while MLPQ replaces the graph encoder with an MLP over the question. PT is the raw prompt-tuning baseline, the same as that used by He et al. [7], and does not use the projector. These controls distinguish the contribution of graph input from that of token parameterisation.

Preventing collapse: a two-stage pipeline

Our ablations associate collapse with two factors: a Sigmoid projector that suppresses encoder variation and end-to-end answer training that does not require this variation to be preserved. We address both while retaining G-Retriever’s data, prompt, answer loss, and optimisation setup (Figure 2).

Stage 1: preserve input-specific variation. Without the LLM, we train the graph encoder with an InfoNCE objective (van den Oord et al. [26]). Its pooled representation $h_i$ is aligned with a frozen embedding of a sample-specific target and contrasted against negative targets. The question conditions each node through $x_v = z_v + W_q q_i$, with $W_q$ initialised at zero. We use the answer as the target, or the verbalized graph, when the answer space is too small to distinguish samples (e.g., answers are Yes/No). The encoder is then frozen.

Stage 2: separate graph representation from prompting. Using the original answer loss, we jointly train a GELU projector over the frozen encoder and a separate graph-independent prompt token. Freezing protects the variation learned in Stage 1, while the second pathway gives the shared prompting function its own token rather than forcing it to compete with input-specific information.

The full objective, targets, and design ablations are reported in Appendix § 22.

Experiments

We investigate four questions. RQ1: Do graph tokens collapse, and how general is this behaviour? RQ2: What role does the collapsed token play when the same graph is also verbalised in the prompt? RQ3: What is collapse associated with, and can it be prevented without sacrificing task performance? RQ4: What do non-collapsed tokens encode, and which of that information does the LLM use?

Setup

Benchmarks. We use the three GraphQA benchmarks introduced by He et al. [7]: ExplaGraphs, SceneGraphs, and WebQSP. Following their official splits, retrieval procedures, and evaluators, we report accuracy on ExplaGraphs and SceneGraphs and Hit@1 on WebQSP. Dataset statistics and retrieval settings are in Appendix § 10.

Models. We reproduce G-Retriever with its default graph transformer (GT) and frozen Llama-2-7B (Touvron et al. [25]). Robustness experiments use GAT, GCN, Qwen2.5-3B (Qwen Team [19]), and LoRA adaptation. Architecture, optimisation, and pipeline settings are in Appendix § 9.

Evaluation stability. Re-evaluating identical WebQSP weights in a separate process changes Hit@1 by 1.04 points. We therefore require an answer-level difference larger than 2 points with $p < .05$. Gold-answer likelihood is more stable; we require $|\Delta\mathrm{NLL}| > 0.03$ and a bootstrap interval excluding the replicate floor. Calibration results are in Appendix § 12.

Statistical testing. Accuracy and Hit@1 comparisons are paired with the exact McNemar test (McNemar [17]). We denote $p < .10$, $.05$, $.01$, and $.001$ by $^{\dagger}$, $^{*}$, $^{**}$, and $^{***}$, respectively; unmarked comparisons have $p \ge.10$. Reported $p$-values are unadjusted, so claims rely on the effect-size criteria above and consistent evidence rather than isolated significant cells.

RQ1: Do graph tokens collapse, and how general is collapse?

Collapse occurs in every tested configuration. Token cosine ranges from 0.985 to 0.998, while the input-specific residual is only 4–12% of the shared mean, whose norm is 22–97 times that of an ordinary word embedding (Appendix § 13). Figure 1 illustrates this geometry on WebQSP. Models with similarly collapsed tokens nevertheless obtain substantially different scores, so task performance does not reveal collapse.

The encoder representations are considerably more variable, with cosines between 0.423 and 0.693. This makes GNN oversmoothing (Rusch et al. [21]) an insufficient explanation for token collapse: even if node representations are smoothed within a graph, the pooled outputs might remain distinct across inputs, and near-identity appears only after projection. Collapse also persists across three graph encoders, two LLM families, and frozen and LoRA-adapted LLMs. Moreover, a controlled WebQSP audit finds the same geometry in later single- and multi-token graph-prompting methods (Appendix § 14). This answers RQ1: collapse is a recurring failure mode rather than a peculiarity of one encoder, LLM, or single-token implementation. We next ask what the collapsed token contributes to the answer.

RQ2: What role does the collapsed token play?

The input-specific component has no resolved answer-level effect. Across all three benchmarks, MEAN, SWAP, and GSWAP remain within the two-point threshold (Table 2); corrupting entities, edges, or relations gives the same result (more evidence in Appendix § 17). Yet replacing the token with zero costs 59.64–81.95 points (see Appendix § 18). The model therefore relies on the token’s shared value, but not measurably on its graph-specific variation.

Dataset	REAL	$\Delta$MEAN	$\Delta$SWAP	$\Delta$GSWAP	NOTXT	$\Delta$NOTXT-MEAN	CONST
ExplaGraphs	85.56	−0.18	−1.26	−1.08	66.43	−0.90	85.56
SceneGraphs	81.35	−0.35	−1.20†	−1.10†	30.20	+0.45	81.40
WebQSP	70.33	−1.84*	−0.92	−0.55	41.34	+1.04	71.25

The verbalised graph is the dominant graph channel. Removing it costs 19–51 points, whereas replacing the remaining token with its mean changes performance by at most 1.04 points. Thus, in the default Llama-2 configuration, task-relevant graph information is supplied primarily through text.

The shared token acts as a soft prompt. CONST, a graph-independent vector passed through the same projector, matches or exceeds G-Retriever on every benchmark. Graph input is therefore unnecessary for the benefit attributed to the collapsed token. Raw prompt tuning, namely the existing ablation performed by He et al. [7], performs worse (Appendix § 19), but changes the parameterisation and is not a controlled graph-versus-no-graph comparison.

Small distributional effects remain: on WebQSP, SWAP and GSWAP increase gold-answer NLL by 0.0183 and 0.0182, both below the 0.03 criterion (Appendix § 20). These results answer RQ2: the collapsed token functions primarily as a projector-parameterised soft prompt, while the verbalised graph is the dominant graph-information channel. RQ3 asks what collapse is associated with and whether it can be prevented.

RQ3: What is collapse associated with, and can it be prevented?

Factors associated with collapse. Table 3 identifies two associations. With the aligned Stage 1 encoder frozen, its representations are highly distinct (cosine 0.024), yet the Sigmoid projector produces nearly identical tokens before and after training (0.9870 and 0.9918). GELU instead yields cosine 0.0910 before training and 0.3229 afterwards. Untrained-projector experiments reproduce this contrast on random inputs (Appendix § 16), linking the Sigmoid parameterisation to suppression of encoder variation.

Encoder	Projector	Encoder cos.	Step-0 token cos.	Token cos.	Score
trained end to end	Sigmoid	0.695	1.000	0.997	85.56
trained end to end	GELU	0.787	0.992	0.914	73.29{sup}***`
frozen after Stage 1	Sigmoid	0.024	0.987	0.992	84.48
frozen after Stage 1	GELU	0.024	0.091	0.323	57.76{sup}***`

Changing the projector alone is insufficient: under end-to-end answer training, GELU still produces highly similar tokens (cosine 0.9139). Conversely, combining the aligned frozen encoder with GELU prevents collapse, but the single-token model falls to approximately majority-class performance (57.76 versus 58.8). Thus, both factors are associated with collapse, but this experiment does not identify either as the sole cause. The cross-method audit supports this qualification (Appendix § 14). GRAG [9] and GRETPool [6] inherit the same Sigmoid projector and also collapse, while GraSP [29] collapses despite using the hyperbolic tangent.

Preventing collapse while retaining performance. Table 4 shows that the complete two-token pipeline mitigates collapse in every tested setting. For the standard pipeline, token cosine falls from above 0.99 to 0.099–0.142 with Llama-2 and to 0.103–0.200 with Qwen2.5-3B.

Dataset	Model	Score	Token cos.	$\Delta$MEAN	$\Delta$SWAP	$\Delta$GSWAP
Llama-2-7B
WebQSP	G-Retriever	70.33	0.996	$-1.84^{*}$	$-0.92$	$-0.55$
	Ours	69.90	0.142	$-3.13^{***}$	$-2.15^{*}$	$-1.97^{\dagger}$
	no entity features	72.73	0.125	$-6.14^{***}$	$-5.65^{***}$	$-1.97^{*}$
	no question	69.84	0.169	$-0.06$	$-3.69^{***}$	$-2.33^{*}$
SceneGraphs	G-Retriever	81.35	0.993	$-0.35$	$-1.20^{\dagger}$	$-1.10^{\dagger}$
	Ours	72.50	0.133	$-1.40$	$-3.40^{***}$	$-5.30^{***}$
	no entity features	79.65	0.125	$-5.75^{***}$	$-9.60^{***}$	$+0.35$
	no question	76.35	0.225	$-1.15$	$-0.25$	$-0.25$
ExplaGraphs	G-Retriever	85.56	0.997	$-0.18$	$-1.26$	$-1.08$
	Ours	85.74	0.099	$+0.36$	$-0.54$	$-1.44$
	no entity features	84.48	0.113	$+3.07^{**}$	$0.00$	$-0.36$
	no question	87.55	0.108	$+1.08$	$+0.90$	$+1.26$
Qwen2.5-3B, matched training budget
WebQSP	G-Retriever	57.13	0.997	$-0.55$	$-1.41$	$-1.60^{\dagger}$
	Ours	65.79	0.200	$-2.09^{**}$	$-1.90^{**}$	$-2.09^{**}$
SceneGraphs	G-Retriever	76.05	0.997	$-0.30$	$-1.05^{\dagger}$	$-1.15^{*}$
	Ours	72.65	0.139	$-7.00^{***}$	$-5.75^{***}$	$-6.50^{***}$
ExplaGraphs	G-Retriever	74.37	0.998	$-1.62$	$-2.17$	$-4.51^{*}$
	Ours	83.94	0.103	$+0.90$	$0.00$	$+0.36$

With Llama-2, the pipeline retains performance on WebQSP (69.90 versus 70.33) and ExplaGraphs (85.74 versus 85.56), with both differences below our resolution threshold. On ExplaGraphs, the separate prompt pathway recovers the performance lost by the non-collapsed single-token model, raising its score from 57.76 to 85.74. On SceneGraphs, however, GSWAP produces a resolved −5.30-point effect while the score falls from 81.35 to 72.50.

The Qwen2.5-3B replication further separates performance from graph use. The Qwen rows in Table 4 use a matched training budget and are distinct from the robustness checkpoint in Table 1. On WebQSP, our pipeline improves the matched baseline from 57.13 to 65.79 ($p = 4.5 \times10^{-5}$), while GSWAP produces a resolved −2.09-point effect ($p = .027$). On ExplaGraphs, it improves from 74.37 to 83.94 ($p = 1.5 \times10^{-5}$), but GSWAP has no effect (+0.36). Conversely, on SceneGraphs, it produces the strongest graph-replacement effect in the study (−6.50, $p < .001$) while scoring 3.40 points below G-Retriever. On G-Retriever on ExplaGraphs, the proposed pipeline provides the opposite exception: its tokens remain collapsed, yet GSWAP causes a resolved −4.51-point effect. Near-collapse therefore does not imply complete insensitivity to the remaining variation.

Dataset	Model	Score	Encoder cosine	Token cosine
ExplaGraphs	G-Retriever (GT)	85.56	0.678	0.997
	G-Retriever (GAT)	85.20	0.457	0.998
	G-Retriever (GCN)	78.88	0.423	0.985
SceneGraphs	G-Retriever (GT)	81.35	0.630	0.993
	G-Retriever (GAT)	80.20	0.693	0.998
	G-Retriever (GCN)	80.55	0.428	0.998
WebQSP	G-Retriever (GT)	70.33	0.593	0.996
	+ LoRA	74.32	0.621	0.997
	+ Qwen2.5-3B	55.47	0.589	0.997

These results answer RQ3. We found that collapse is associated with both the projector and end-to-end answer training, without either being isolated as its sole cause. The two-stage pipeline mitigates collapse across both LLM families and can retain or improve performance. However, geometric collapse, graph-specific token use, and task performance remain distinct properties. RQ4 takes a step forward and examines what the tokens encode and which information the LLM uses.

RQ4: What does a non-collapsed token carry?

Non-collapse shows that tokens differ across inputs, but not what they represent or whether the LLM uses those differences. Because Stage 1 receives the question and an answer-related target, the token may encode the question, an answer prior, the graph, or a mixture.

We separate recoverable content from downstream use. Table 5 reports question, answer, and graph retrieval ranks, answer-level effects under GSWAP, and likelihood changes under SWAP and GSWAP. Table 4 measures reliance on sample-specific token information through MEAN and SWAP, while the appendix Table 70 repeats the interventions without the retrieved verbalised graph. Complete cross-LLM results are reported in Appendix § 21.

Dataset	Model	Q rank	A rank	G rank	$\Delta$Score	GSWAP	$\Delta$NLL S/GS	Diagnosis
WebQSP	G-Retriever	293	317	369	$-0.55$	$+0.0183/+0.0182$		SOFT PROMPT
WebQSP	Ours	65	60	315	$-1.97^{\dagger}$	$+0.0895/+0.0332$		MASKED
WebQSP	no entity features	34	51	303	$-1.97^{*}$	$+0.1111/+0.0262$		PRIOR
WebQSP	no question	120	134	329	$-2.33^{*}$	$+0.0848/+0.0845$		USED
SceneGraphs	G-Retriever	271	436	248	$-1.10^{\dagger}$	$+0.0169/+0.0167$		SOFT PROMPT
SceneGraphs	Ours	47	187	427	$-5.30^{***}$	$+0.0229/+0.1665$		USED
SceneGraphs	no entity features	8	59	369	$+0.35$	$+0.1553/+0.0045$		PRIOR
SceneGraphs	no question	116	395	185	$-0.25$	$+0.0214/+0.0213$		MASKED
ExplaGraphs	G-Retriever	138	–	139	$-1.08$	$+0.0014/+0.0022$		SOFT PROMPT
ExplaGraphs	Ours	54	–	41	$-1.44$	$+0.0010/+0.0032$		DECODABLE
ExplaGraphs	no entity features	67	–	67	$-0.36$	$+0.0026/+0.0056$		DECODABLE
ExplaGraphs	no question	84	–	70	$+1.26$	$-0.0063/-0.0073$		DECODABLE

Table 70: Llama-2 evaluation without the verbalised graph. Intervention columns report differences from NOTXT; the final column gives the corresponding score with the verbalised graph present. Bold GSWAP effects satisfy $|\Delta| > 2$ and $p < .05$. Dashes denote conditions not evaluated.

Dataset	Model	NOTXT	$\Delta$MEAN	$\Delta$SWAP	$\Delta$GSWAP	With text
WebQSP	G-Retriever	41.34	+1.04	−0.18	−0.86	70.33
	Ours	28.75	−2.09*	−2.40*	−2.27*	69.90
	no entity features	28.38	−7.49***	–	–	72.73
	no question	30.90	+0.43	–	–	69.84
SceneGraphs	G-Retriever	30.20	+0.45	−1.40	–	81.35
	Ours	32.60	−6.60***	−10.85***	−3.10	72.50
	no entity features	30.10	−12.85***	–	–	79.65
	no question	26.15	−8.25***	−3.25**	−3.30	76.35
ExplaGraphs	G-Retriever	66.43	−0.90	0.00	–	85.56
	Ours	69.68	+0.18	+0.18	−0.54	85.74
	no entity features	70.22	+0.72	–	–	84.48
	no question	78.88	+1.08	–	–	87.55

The retrieval probe maps each token to the corresponding question, answer, or verbalised graph among held-out candidates. We report the median rank of the correct target: lower is better, and a constant token provides the control rank. Recoverability does not imply use. A low answer rank, for example, means that the answer is predictable from the token, not that it was computed from the graph or used by the LLM.

We use five diagnoses. USED means that GSWAP changes the task score by more than two points with $p < .05$. MASKED means that this effect resolves only after removing the verbalised graph. PRIOR identifies a no-entity model whose token carries question and answer information and is used under MEAN or SWAP, without resolved graph-specific use. DECODABLE means that graph information is recoverable without a resolved answer-level effect. SOFT PROMPT denotes a collapsed baseline without resolved graph-specific use; any information recoverable from its residual variation is reported separately by the retrieval ranks.

WebQSP: the token’s role depends on training. The standard token makes the question and answer strongly recoverable, at ranks 65 and 60 against the control of 407. With the verbalised graph present, GSWAP affects likelihood but not the decoded score; after removing the text, it causes a resolved $-2.27$-point effect (Table 5 and Table 70). Removing entity features further strengthens question and answer recovery and improves Hit@1 without resolved graph use, supporting the PRIOR diagnosis. Conversely, removing question conditioning produces resolved graph use. Thus, the same architecture can favour priors or graph information depending on its training inputs.

SceneGraphs: graph use and accuracy diverge. Our pipeline uses graph-specific token content: GSWAP lowers accuracy by 5.30 points and increases NLL by 0.1665, but it remains less accurate than G-Retriever. G-Retriever’s collapsed token contains recoverable graph information, yet shows no resolved graph use, so its functional diagnosis remains SOFT PROMPT. The no-entity model nearly recovers its accuracy while eliminating graph use, confirming that performance and token use are distinct.

ExplaGraphs: graph information is decodable but unused. Our token makes the graph substantially more recoverable, at rank 41 against the control of 139, but GSWAP has no resolved effect with or without the verbalised graph. Because these graphs are small and supplied in full, their textual representation may already be sufficient. However, removing that text does not make the model rely on the token. The evidence therefore establishes recoverable graph information, not downstream use.

Together, these results answer RQ4: preventing collapse still does not uniquely determine what a token represents or whether the LLM uses it. Depending on the benchmark and training inputs, a non-collapsed token may encode a question or answer prior, unused graph information, or graph information that changes the answer.

Discussion

What standard evaluation misses. Task accuracy cannot isolate graph-token use, while raw prompt tuning changes the token parameterisation and no-text retraining changes the learning problem. A minimal audit should combine token geometry, a graph-free control using the same projector, and fixed-weight MEAN, SWAP, and GSWAP interventions. The controlled WebQSP audit in Appendix § 14 finds collapse with query-conditioned and multi-token approaches, extending the phenomenon beyond G-Retriever up to most recent graph soft prompting methods.

Graph use and accuracy are different outcomes. Our pipeline can make answers depend on graph-specific token variation without improving accuracy. The textual channel may be easier to exploit, while continuous graph tokens may be lossy or difficult for the LLM to interpret. The Qwen2.5-3B results show that this trade-off is not inevitable; alignment claims should therefore report graph use and performance separately.

Limitations and future directions. Our detailed analysis covers G-Retriever on three question-answering benchmarks; the cross-method audit is limited to WebQSP, runnable implementations, and controlled variants using G-Retriever’s retrieved subgraphs. The mechanism evidence is associative and does not isolate the projector or end-to-end training as a sole cause.

The audited recipe trains from scratch with answer cross-entropy; collapse may differ with pre-trained encoders or explicit contrastive, reconstructive, or counterfactual objectives (Zhang et al., 2026; [30]). Stage 1 explores one alternative, but its conditioning and targets can introduce priors. GSWAP tests graph–question pairing rather than fact-level reasoning, and decodability measures recoverability rather than use. Our pipeline is therefore a diagnostic proof of concept. Future work should preserve task-relevant graph information and verify that the LLM uses it.

Conclusion

We uncover token collapse in G-Retriever across benchmarks, graph encoders, LLM families, and tuning regimes, and find the same geometry in controlled tests of later single- and multi-token interfaces. Despite competitive task scores, the projected tokens are nearly identical across inputs. We also find that graph-free controls and fixed-weight interventions match the performance of the original G-retriever model. These findings show that strong task performance can conceal a graph-token interface whose principal contribution is a graph-independent soft prompt, while the verbalised graph provides the dominant graph-information channel. Our two-stage pipeline mitigates this collapse, but also demonstrates that distinct tokens do not guarantee graph use or improved accuracy. Token variation, recoverable graph information, downstream use, and task performance must therefore be evaluated separately. Our protocol provides a practical way to distinguish these outcomes and turns alignment from an architectural assumption into an empirical claim that will foster the development of future graph–language models.

AI Use Statement

Generative AI tools assisted with prose revision and parts of the analysis and plotting code. They did not generate experimental results: all newly reported experimental numbers were produced by our pipeline and checked against stored predictions, tokens, checkpoints, or logs. The authors reviewed all AI-assisted text and code, verified the experimental claims, and take full responsibility for the paper.

Reproducibility Statement

Appendix A specifies the repository and code modification, optimisation settings, model dimensions, LoRA configuration, Stage-1 hyperparameters, trainable parameter counts, hardware, and evaluation procedure. Appendix B reports dataset statistics, splits, retrieval settings, metrics, and intervention subsets. Appendix I provides the complete fixed-weight intervention results; the remaining appendices report the geometry, likelihood, decodability, controls, and pipeline measurements.

References

Appendix Overview

Setup. A implementation and hyperparameters; B datasets and retrieval; C graph encoders.

Protocol. D condition taxonomy, likelihood and decodability estimators, evaluation stability and the statistical protocol.

Collapse. E complete collapse statistics and per-encoder interventions; F complete collapse statistics for methods subsequent to G-Retriever; G token space of every model; H projector diagnostics behind Section 4.4.

Causal audit. I full fixed-weight interventions; J magnitude and geometric conditions; K complete graph-free controls; L gold-answer likelihood.

Content and pipeline. M complete decodability results; N design choices and Stage-1 diagnostics; O pipeline and robustness ablations; P cross-fitting; Q initialisations and evaluation noise.

Context. R the verbalised graph is not enough; S extended related work.

Implementation details

G-Retriever reproduction. We use the authors’ public implementation at a pinned commit. For the baseline reproduction, our only modification is explicit device placement, allowing each run to fit on a single GPU. The frozen sentence encoder is all-roberta-large-v1 with output width $d = 1024$. The default graph encoder is a four-layer, four-head graph transformer with pooled width $d_g = 1024$. Its representation is mapped to the LLM embedding space by a two-layer projector with hidden width $d_p = 2048$. The default LLM is frozen Llama-2-7B, whose embedding width is $d_\ell= 4096$. The verbalised graph is truncated to $L_{\max} = 512$ tokens, and generation is limited to 32 tokens.

Optimisation follows the authors’ defaults: AdamW with learning rate $10^{-5}$, weight decay 0.05, and at most 10 epochs, with early stopping after two epochs without validation improvement. Every model in this paper is trained on a single 80 GB GPU.

Stage 1. InfoNCE with temperature 0.07 against a memory bank of about 2,000 targets, with no LLM loaded. Trainable: 20,994,048 parameters in the encoder and 1,049,600 in the question-conditioning layer $W_q$.

Stage 2. The Stage-1 encoder and $W_q$ parameters are frozen, while their running normalisation statistics remain in training mode. The projector’s Sigmoid is replaced by GELU, and a learned constant of width $d_g$ passes through its own Linear → Sigmoid → Linear projector to produce the prompt token. The two projector pathways are initialised and optimised jointly; checkpoint-derived trainable parameter counts are reported in Table 64. Interventions modify only the graph token, never the prompt token.

Model	Trainable parameters	ExplaGraphs	SceneGraphs	WebQSP
G-Retriever	31,492,099	85.56	81.35	70.33
CONST	10,492,928	85.56	81.40	71.25
MLPQ	14,690,304	84.84	–	70.70
PT	40,960	60.60	70.22	52.21
ours, Stage 2	20,990,979	85.74	72.50	69.90

Controls and parameter counts. Table 64 reports checkpoint-derived trainable parameter counts for the evaluated models. Stage 1 separately trains 22,043,648 parameters; because the two stages are trained sequentially, we do not report their sum as a simultaneous trainable count. Accuracy does not track capacity: CONST improves over PT by 19 points, while our larger Stage-2 model scores 1.35 points below CONST. Since PT and CONST differ in both parameterisation and capacity, their gap should not be attributed to the projector alone. The controlled graph-input comparison is instead between CONST and G-Retriever, which share the projector parameterisation.

Datasets

We use the GraphQA benchmark assembled by He et al. [7], whose statistics are reproduced in Table 6. Retrieval is needed on two of the three: verbalising an average WebQSP graph takes 100,627 tokens, reduced to 610 after retrieval, and an average SceneGraphs graph 1,396 tokens, reduced to 235. ExplaGraphs graphs fit in the prompt, so the whole graph is used. Retrieval keeps the top 3 nodes and edges on SceneGraphs with edge cost 1, and the top 3 nodes and 5 edges on WebQSP with edge cost 0.5.

	ExplaGraphs	SceneGraphs	WebQSP
Graphs	2,766	100,000	4,737
Average nodes	5.17	19.13	1,370.89
Average edges	4.25	68.44	4,252.37
Node attributes	commonsense concepts	objects and attributes	Freebase entities
Edge attributes	commonsense relations	actions, spatial relations	Freebase relations
Task	stance prediction	scene-graph QA	knowledge-graph QA
Metric	accuracy	accuracy	Hit@1
Test questions	554	20,025	1,628

Metrics use the benchmark’s own evaluators, applied to the stored per-question predictions: on ExplaGraphs the first “support” or “counter” in the output must equal the label; on SceneGraphs the gold label must appear in the output; on WebQSP a prediction counts as a hit when it contains one of the gold answers. Interventions use all test questions on ExplaGraphs and WebQSP, and the first 2,000 on SceneGraphs, of which 1,302 are not yes/no questions. The one comparison drawn on the full SceneGraphs test set, prompt tuning against the graph-free constant in Section 4.3, is stated as such where it appears.

Graph encoders

GAT (Veličković et al. [27]) and GCN (Kipf & Welling [10]) replace the graph transformer (Shi et al. [22]) inside G-Retriever’s pipeline, using the repository’s own implementations with the same depth and hidden width. Their collapse statistics and interventions appear in Table 1 and Table 2, their token spaces in Figure 4 and Figure 5, and every condition measured for them in Appendix § 17. The three encoders differ in accuracy on ExplaGraphs, where GCN loses 6.7 points to the graph transformer, but not in the behaviour of their tokens.

Protocol details

Token-geometry statistics. Let $\mathbf{t}_1,\ldots,\mathbf{t}_n$ be the test tokens, $\boldsymbol{\mu} = \frac{1}{n}\sum_i \mathbf{t}_i$ their mean, and $\bar{e}$ the median norm of the LLM’s word embeddings. We measure

Inter-sample cosine measures whether tokens point in the same direction, the residual ratio (RR) measures their input-specific component relative to the shared mean, and the norm ratio (NR) compares their magnitude with an ordinary word embedding. We compute the same inter-sample cosine for the pooled encoder representations $\mathbf{h}_i$ to distinguish collapse in the encoder from collapse introduced by the projector.

GSWAP. The partner graph is drawn by a fixed derangement of the test set, a cyclic shift, and two further random derangements are reported as replicates in Appendix § 17. The partner’s graph replaces the batch’s graph before the encoder, while the question conditioning $\mathbf{W}_q\mathbf{q}_i$ still uses question $i$, which is what makes the condition specific to the graph.

The conditions. MEAN replaces the token by the mean token over the training graphs, identical for every question. SWAP replaces it by the token of another test question under a fixed permutation. NODE recomputes the token after replacing node features with real rows from other graphs, keeping the edges; EDGE after rewiring every edge uniformly at random inside its own graph, keeping features and edge count; REL after resampling relation attributes. The conditions on the token’s geometry are the residual scaling $\boldsymbol{\mu} + a(\mathbf{t}_i-\boldsymbol{\mu})$ with $a \in\{0,2\}$, the zero token $0$, and rescaling the token to a chosen norm. NOTXT-X evaluates condition x with the verbalised graph deleted from the prompt, which puts every model out of its training distribution and is read as a comparison between models on the same items.

The measured floor. Table 7 is the calibration behind the thresholds of Appendix § 12. The null replicate reruns the identical condition on the identical weights in a second process and compares the generated strings; models with a GNN are not bit-reproducible across processes, because scatter reductions accumulate in a nondeterministic order, and a difference of order $10^{-7}$ in the prefix token rewrites most greedy decodings. MLPQ’s tokens are identical to within $3 \times10^{-4}$, so its swap carries no information and is still significant.

Control	Perturbation	Churn	Accuracy	Exact McNemar $p$
Null replicate, same weights, same condition	none at all	63.2%	−1.04	.247
MLPQ, SWAP	$3 \times10^{-4}$, cosine $\ge0.999999$	38%	−1.72	.015
No entity features, NODE	zero, features already zeroed	43%	−0.61	.415
CONST, SWAP	zero, all token rows identical	0%	0.00	1

Gold-answer likelihood. With $y_i$ the gold answer of question $i$, truncated to the generation budget and terminated by the end-of-sequence token, and $\mathbf{t}_i^c$ the graph token under condition $c$:

This is the token-averaged cross-entropy of the gold answer under teacher forcing, so nothing is generated and no decoding is involved; a positive $\Delta_{\mathrm{NLL}}$ means the intervention made the gold answer less likely, that is, the token was helping. The gold answer is truncated to 32 tokens plus the end-of-sequence token, the batch size is 1, and paired differences are reported with a 2,000-sample bootstrap confidence interval and a Wilcoxon signed-rank test.

Decodability. With $\mathbf{T} \in\mathbb{R}^{m \times d_\ell}$ the stored test tokens of the fitting half, $\mathbf{Y}$ the frozen sentence embeddings of the target texts and $\widetilde{\mathbf{T}} = [\mathbf{T}, 1]$,

solved in double precision with $\alpha= 10^{3}$. The held-out half is scored by retrieval: each question’s predicted embedding is ranked against all held-out targets by cosine, and we report the median rank of its own target, with ties counted against us. The split is 50/50 with seed 0, the sentence encoder is the same frozen all-roberta-large-v1 used everywhere else, and the control replaces every token by the fitting half’s mean, which lands at the median rank by construction.

Two caveats bound the graph column. The sentence encoder truncates at 512 wordpieces, and the verbalised subgraphs are post-retrieval: the median is 76 wordpieces on ExplaGraphs with nothing truncated, 260 on WebQSP with 14.5% truncated and 256 on SceneGraphs with 23%, so the instrument is intact on the benchmark where we lean on it and is a lower bound on the other two. Compressing a subgraph into one 1024-dimensional sentence vector is lossy in any case, so the column measures whether the token aligns with the gist of the verbalisation. Failing to decode is therefore not proof of absence, while decoding successfully is proof of presence, which is the direction every claim in the paper uses.

Likelihood floor. The same null replicate gives a paired likelihood difference of +0.00026 with a 95% confidence interval of [−0.00027, +0.00080] and Wilcoxon $p = .785$.

Complete collapse statistics and per-encoder interventions

Table 11 gives the two token statistics dropped from Table 1, and Table 12 the per-encoder interventions dropped from Table 2. Every remaining condition of every model is in Appendix I.

Dataset	Model	Residual ratio	Norm ratio
ExplaGraphs	G-Retriever (GT)	0.050	21.9$\times$
ExplaGraphs	G-Retriever (GAT)	0.040	24.4$\times$
ExplaGraphs	G-Retriever (GCN)	0.121	24.3$\times$
SceneGraphs	G-Retriever (GT)	0.088	84.8$\times$
SceneGraphs	G-Retriever (GAT)	0.046	87.6$\times$
SceneGraphs	G-Retriever (GCN)	0.048	97.0$\times$
WebQSP	G-Retriever (GT)	0.057	29.4$\times$
WebQSP	+ LoRA	0.050	37.8$\times$
WebQSP	+ Qwen2.5-3B	0.047	22.8$\times$

Dataset	Model	REAL	MEAN	SWAP	NODE	EDGE	NOTXT-REAL
ExplaGraphs	G-Retriever (GAT)	85.20	$-0.36$	0.00	$-0.18$	$+0.18$	76.90
ExplaGraphs	G-Retriever (GCN)	78.88	$+0.90$	$+0.36$	$+1.08$	$+0.72$	69.86
SceneGraphs	G-Retriever (GAT)	80.20	$+0.60$	$+0.20$	$-0.20$	$+0.10$	28.65
SceneGraphs	G-Retriever (GCN)	80.55	$-0.10$	$-1.05$	$-1.25^{\dagger}$	$-1.25^{\dagger}$	29.80

Table 9: Cross-method configurations. $K$ is the number of graph-derived tokens and $N$ the retained PCST node count. The projector column reports the final graph-to-LLM mapping.

Method	Configuration	Projector	$K$ regime	Mean $K$
GRAG-PCST [9]	released GAT + LoRA	Linear–Sigmoid–Linear	1	1.000
GRETPool [6]	TopK [5], ratio 0.44	Linear–Sigmoid–Linear	$\lceil0.44N \rceil$	7.848
–	SAG [11], ratio 0.44	Linear–Sigmoid–Linear	$\lceil0.44N \rceil$	7.848
–	DiffPool [31]	Linear–Sigmoid–Linear	8	8.000
–	DiffPool [31]	Linear–Sigmoid–Linear	100	100.000
GraSP-PCST [29]	SUBGRAPHGNN [1] + Llama	Linear–Tanh–Linear	$N$	16.719

Beyond G-Retriever: token collapse in subsequent graph-prompting methods

Motivation and scope. We test whether collapse is confined to G-Retriever or recurs in later graph-prompting methods. We consider methods evaluated on WebQSP that construct continuous prompts from graphs and have runnable public implementations. These criteria retain GRAG [9], GRETPool [6], and GraSP [29]. We exclude Graph2Token [28], which targets molecular graphs, and GraphAdapter [15], which adapts vision–language models. Align-GRAG [30] could not be reproduced because its anonymous repository was no longer available.

Controlled setup. All methods use the 1,628-example WebQSP test split and the PCST subgraphs retrieved for G-Retriever. Holding retrieval fixed prevents graph size and relevance from explaining differences in token geometry, but makes GRAG-PCST and GraSP-PCST controlled adaptations rather than complete reproductions of the published systems. GRAG-PCST retains the released query-conditioned GAT, projector, and LoRA-tuned Llama configuration. GRETPool uses its reported TopK [5] and SAG [11] ratios of 0.44 and DiffPool [31] budgets of 8 and 100 tokens. GraSP-PCST uses the released SUBGRAPHGNN [1] and GRAPH2PREFIX with a local Llama model, producing one token per retained node. Table 9 summarises the resulting cardinality regimes.

Diagnostics across cardinalities. For inter-sample comparison, we mean-pool each example’s $K$ prompt tokens and compute the geometry statistics of Appendix § 12. This pooling is used only for measurement and does not alter inference. For $K > 1$, we also report the mean pairwise cosine among tokens from the same graph. All interventions preserve cardinality: donor examples have the same $K$, and node randomisation and edge rewiring preserve node and edge counts.

Results. Collapse occurs in all six configurations and every tested cardinality: inter-sample cosine is 0.998–1.000, and the multi-token methods are also internally almost collinear. GRAG and GRETPool share G-Retriever’s Linear–Sigmoid–Linear projector, so their results show that collapse persists under alternative encoders and pooling operators without changing this bottleneck. GraSP provides the independent case: despite using a Linear–Tanh–Linear projector and one token per retained node, it reaches inter-sample cosine 0.9978, within-example cosine 0.9995, and residual ratio 0.039. Its smaller norm ratio (4.27) shows that collapse need not involve the unusually large token norms observed with the Sigmoid projectors. Thus, collapse can persist with multiple tokens and with either Sigmoid or Tanh projectors.

With the verbalised graph present, none of the direct token-content controls resolves: SWAP changes Hit@1 by at most 0.921 points ($p = .298$), GSWAP by at most 0.676 ($p \ge.379$), and MEAN by at most 0.799 ($p = .263$). Node randomisation is likewise unresolved. Edge rewiring produces two marginal effects: −1.413 for GRAG-PCST ($p = .0616$) and +2.088 for DiffPool with $K = 8$ ($p = .0525$). Neither satisfies the paper’s joint requirement of $|\Delta| > 2$ and $p < .05$, and the latter improves rather than degrades performance.

The no-text tests use NOTXT-REAL, rather than the text-present score, as their paired reference. They provide limited, configuration-dependent evidence: DiffPool with $K = 100$ decreases under the mean-token diagnostic (−1.904, $p = .0472$), while DiffPool with $K = 8$ under SWAP (+2.334, $p = .0530) and GraSP-PCST under GSWAP (+2.518,$p = .0767) are marginal. None satisfies both the effect-size and $p$-value criteria. Removing the verbalised graph itself lowers performance by

Method	Mean $K$	Hit@1	Inter cos.	RR	Norm ratio	Within cos.	Eff. rank
(a) Geometry and unmodified performance
GRAG-PCST	1.000	74.570	0.999171	0.028223	42.935	–	10.300
TopK	7.848	71.437	1.000000	0.000155	28.258	1.000000	–
SAG	7.848	72.727	1.000000	0.000531	25.802	0.999999	–
DiffPool ($K = 8$)	8.000	52.703	0.999712	0.009284	25.477	0.999856	–
DiffPool ($K = 100$)	100.000	61.364	0.999994	0.001291	27.170	1.000000	–
GraSP-PCST	16.719	81.450	0.997783	0.039442	4.271	0.999465	–
(b) With-text interventions: $\Delta$Hit@1 ($p$) relative to REAL
Method	SWAP	W-SHUF	GSWAP	MEAN	NODE	EDGE
GRAG-PCST	−0.246 (.791)	–	+0.061 (1.000)	+0.676 (.379)	−0.061 (1.000)	−1.413 (.0616)†
TopK	+0.430 (.659)	–	−0.676 (.514)	−0.184 (.877)	−0.799 (.390)	+1.044 (.254)
SAG	−0.921 (.298)	–	−0.676 (.462)	+0.553 (.543)	−0.369 (.685)	+0.614 (.493)
DiffPool ($K = 8$)	−0.614 (.612)	–	+0.614 (.610)	−0.184 (.909)	+0.676 (.556)	+2.088 (.0525)†
DiffPool ($K = 100$)	−0.430 (.689)	–	+0.184 (.894)	+0.246 (.839)	−0.553 (.569)	+0.430 (.674)
GraSP-PCST	−0.553 (.460)	+0.123 (.930)	−0.676 (.379)	+0.799 (.263)	−0.430 (.586)	−0.123 (.924)
(c) Without-text interventions: REAL Hit@1 and $\Delta$Hit@1 ($p$) relative to NOTXT-REAL
Method	NOTXT-REAL	NOTXT-SWAP	NOTXT-GSWAP	NOTXT-MEAN
GRAG-PCST	41.585	−1.044 (.303)	+0.184 (.904)	−0.369 (.756)
TopK	28.194	−0.184 (.904)	+0.860 (.519)	−0.921 (.395)
SAG	31.511	+0.307 (.802)	+0.369 (.798)	−0.369 (.758)
DiffPool ($K = 8$)	27.150	+2.334 (.0530)†	+0.491 (.719)	+0.921 (.463)
DiffPool ($K = 100$)	22.789	−1.167 (.224)	−0.553 (.605)	−1.904 (.0472)*
GraSP-PCST	38.391	−0.123 (.963)	+2.518 (.0767)†	+1.658 (.234)

25.6–43.2 points, but this comparison measures removal of textual evidence rather than graph-token specificity.

These results extend the collapse observation beyond G-Retriever: it recurs under GRAG’s query-conditioned single-token method and under fixed, ratio-selected, and node-preserving multi-token pooling. They do not show that every graph-prompting method collapses or that graph information is universally unnecessary. The paired tests establish that token-content replacements are unresolved in every text-present configuration, while the no-text conditions show only limited and inconsistent sensitivity. Because this audit remains single-seed and lacks likelihood measurements, it supports a recurrence claim rather than the stronger conclusions established for G-Retriever in the main experiments.

The token space of every model

The figures show the graph tokens of every test question. Their top rows project the mean-centred tokens onto two principal components and scale them by $\lVert\mu\rVert$, making models with different token magnitudes comparable. The reported mean radius is measured in this two-dimensional projection and is therefore smaller than the full-dimensional residual ratio in Table 11. Panels with radius below 0.1 include a magnified inset. The bottom rows show inter-sample cosine distributions on a fixed axis.

Mechanism diagnostics

Table 13 supports the first association in Section 4.4. An untrained projector, at its default initialisation, is applied to 500 random inputs of fixed norm, and the tokens it produces are almost identical to each other unless the input norm is two orders of magnitude larger than anything a trained encoder produces.

Input norm	Token norm	Token cosine	Residual ratio
1	18.4	0.9999	0.009
10	18.4	0.9919	0.090
100	21.4	0.7346	0.600

Why Sigmoid can create a shared direction. Let $x = W_1 h_i + b_1$. If these preactivations are near zero, then $\sigma(x) \approx\frac{1}{2}\mathbf{1} + \frac{1}{4}x$, and the projector is approximately

The first three terms are input-independent, while the factor $\frac{1}{4}$ attenuates input-specific variation. This provides one possible explanation for the shared direction produced by the untrained projector. It is not a complete explanation: GraSP remains collapsed with a zero-centred Tanh projector (Appendix § 14), showing that Sigmoid is neither necessary nor the sole cause.

The approximation therefore motivates a mechanism consistent with Table 13, rather than establishing that the projector operates entirely in this local regime.

Full fixed-weight interventions

The following tables report the available fixed-weight interventions for each model. Each table gives absolute accuracy, the difference from the reference condition, and the exact McNemar $p$-value. The reference is REAL, or NOTXT-REAL for conditions without the verbalised graph. The second and third SWAP rows, where present, use independent derangements. The reference is REAL, and NOTXT-REAL for the block without the verbalised graph. Conditions are defined in § 3.2 and Table 8; the second and third SWAP rows are replicates under different derangements. Both stages of our pipeline are trained on all training questions unless a caption says cross-fitted. No effect below 2.0 accuracy points licenses a claim, whatever its $p$-value (Appendix § 12).

ExplaGraphs

Condition	Accuracy	Difference	$p$
REAL	85.56	–	–
MEAN	85.38	−0.18	1
SWAP	84.30	−1.26	0.118
$\mu+ 0 \cdot(t_i - \mu)$	85.20	−0.36	0.727
$\mu+ 2(t_i - \mu)$	84.48	−1.08	0.146
NODE (real rows)	84.66	−0.90	0.227
NODE (Gaussian)	85.20	−0.36	0.727
EDGE (rewired)	85.02	−0.54	0.581
REL (resampled)	85.74	+0.18	1
zero token	3.61	−81.95{sup}***	$1.09 \times10^{-131}$
norm $\to$ word size	3.61	−81.95{sup}***	$3.17 \times10^{-130}$
$\times0.5$	77.44	−8.12{sup}***	$6.9 \times10^{-6}$
$\times2$	81.95	−3.61{sup}***	0.000535
NOTXT-REAL	66.43	–	–
NOTXT-MEAN	65.52	−0.90	0.267
NOTXT-SWAP	66.43	+0.00	1
NOTXT-NODE	66.25	−0.18	1
NOTXT-zero	12.09	−54.33{sup}***	$2.46 \times10^{-66}$
NOTXT-norm $\to$ word size	10.83	−55.60{sup}***	$7.08 \times10^{-70}$

Condition	Accuracy	Difference	$p$
REAL	86.28	–	–
MEAN	84.84	−1.44	0.134
SWAP	85.02	−1.26	0.265
$\mu+ 0 \cdot(t_i - \mu)$	84.84	−1.44	0.134
$\mu+ 2(t_i - \mu)$	83.57	−2.71{sup}***	0.000275
NODE (real rows)	86.10	−0.18	1
NODE (Gaussian)	86.64	+0.36	0.754
EDGE (rewired)	84.84	−1.44	0.152
REL (resampled)	85.74	−0.54	0.453
zero token	3.25	−83.03{sup}***	$4.53 \times10^{-135}$
norm $\to$ word size	3.43	−82.85{sup}***	$1.55 \times10^{-136}$
$\times0.5$	66.06	−20.22{sup}***	$1.26 \times10^{-18}$
$\times2$	82.13	−4.15{sup}*	0.0451
NOTXT-REAL	71.84	–	–
NOTXT-MEAN	71.48	−0.36	0.912
NOTXT-SWAP	70.76	−1.08	0.61
NOTXT-NODE	73.10	+1.26	0.464
NOTXT-zero	11.37	−60.47{sup}***	$1.4 \times10^{-86}$
NOTXT-norm $\to$ word size	11.01	−60.83{sup}***	$5.72 \times10^{-89}$

Condition	Accuracy	Difference	$p$
REAL	87.91	–	–
MEAN	87.18	$-0.72$	0.219
SWAP	87.73	$-0.18$	1
$\mu+ 0 \cdot(t_i - \mu)$	87.73	$-0.18$	1
$\mu+ 2(t_i - \mu)$	87.91	$+0.00$	1
NODE (real rows)	88.09	$+0.18$	1
NODE (Gaussian)	87.91	$+0.00$	1
EDGE (rewired)	87.18	$-0.72$	0.424
REL (resampled)	87.73	$-0.18$	1
zero token	3.25	$-84.66^{***}$	$1.31 \times10^{-141}$
norm $\to$ word size	4.51	$-83.39^{***}$	$1.14 \times10^{-135}$
$\times0.5$	67.87	$-20.04^{***}$	$3 \times10^{-20}$
$\times2$	85.56	$-2.35^{\dagger}$	0.066
NOTXT-REAL	70.22	–	–
NOTXT-MEAN	69.68	$-0.54$	0.648
NOTXT-SWAP	69.86	$-0.36$	0.851
NOTXT-NODE	71.30	$+1.08$	0.286
NOTXT-zero	11.37	$-58.84^{***}$	$2.37 \times10^{-75}$
NOTXT-norm $\to$ word size	11.73	$-58.48^{***}$	$8.31 \times10^{-75}$

Condition	Accuracy	Difference	$p$
REAL	89.71	–	–
MEAN	89.71	$+0.00$	1
SWAP	90.07	$+0.36$	0.625
$\mu+ 0 \cdot(t_i - \mu)$	89.71	$+0.00$	1
$\mu+ 2(t_i - \mu)$	90.07	$+0.36$	0.5
NODE (real rows)	89.71	$+0.00$	1
NODE (Gaussian)	89.71	$+0.00$	1
EDGE (rewired)	90.07	$+0.36$	0.688
REL (resampled)	90.07	$+0.36$	0.625
zero token	2.17	$-87.55^{***}$	$2 \times10^{-146}$
norm $\to$ word size	3.25	$-86.46^{***}$	$1.28 \times10^{-144}$
$\times0.5$	15.88	$-73.83^{***}$	$4.29 \times10^{-113}$
$\times2$	82.31	$-7.40^{***}$	$1.53 \times10^{-6}$
NOTXT-REAL	83.39	–	–
NOTXT-MEAN	84.12	$+0.72$	0.672
NOTXT-SWAP	82.49	$-0.90$	0.603
NOTXT-NODE	81.95	$-1.44$	0.358
NOTXT-zero	12.45	$-70.94^{***}$	$3 \times10^{-103}$
NOTXT-norm $\to$ word size	13.00	$-70.40^{***}$	$2.57 \times10^{-104}$

Condition	Accuracy	Difference	$p$
REAL	85.20	–	–
MEAN	84.84	−0.36	0.754
SWAP	85.20	+0.00	1
$\mu+ 0 \cdot(t_i - \mu)$	85.02	−0.18	1
$\mu+ 2(t_i - \mu)$	85.20	+0.00	1
NODE (real rows)	85.02	−0.18	1
NODE (Gaussian)	84.66	−0.54	0.508
EDGE (rewired)	85.38	+0.18	1
REL (resampled)	84.84	−0.36	0.5
zero token	3.61	−81.59***	$4.3 \times10^{-131}$
norm $\to$ word size	3.07	−82.13***	$1.42 \times10^{-133}$
$\times0.5$	77.62	−7.58***	$3.89 \times10^{-5}$
$\times2$	83.21	−1.99	0.161
NOTXT-REAL	76.90	–	–
NOTXT-MEAN	76.90	+0.00	1
NOTXT-SWAP	76.35	−0.54	0.701
NOTXT-NODE	77.08	+0.18	1
NOTXT-zero	12.09	−64.80***	$2.39 \times10^{-97}$
NOTXT-norm $\to$ word size	10.47	−66.43***	$5.65 \times10^{-100}$

Condition	Accuracy	Difference	$p$
REAL	78.88	–	–
MEAN	79.78	+0.90	0.405
SWAP	79.24	+0.36	0.86
$\mu+ 0 \cdot(t_i - \mu)$	80.14	+1.26	0.23
$\mu+ 2(t_i - \mu)$	77.44	−1.44	0.215
NODE (real rows)	79.96	+1.08	0.345
NODE (Gaussian)	78.88	+0.00	1
EDGE (rewired)	79.60	+0.72	0.503
REL (resampled)	79.42	+0.54	0.25
zero token	3.61	−75.27***	$1.16 \times10^{-120}$
norm $\to$ word size	2.89	−75.99***	$7.49 \times10^{-122}$
$\times0.5$	40.61	−38.27***	$1.43 \times10^{-35}$
$\times2$	70.58	−8.30***	0.00086
NOTXT-REAL	69.86	–	–
NOTXT-MEAN	69.49	−0.36	0.845
NOTXT-SWAP	69.31	−0.54	0.85
NOTXT-NODE	67.87	−1.99†	0.0895
NOTXT-zero	12.09	−57.76***	$1.22 \times10^{-80}$
NOTXT-norm $\to$ word size	10.65	−59.21***	$1.86 \times10^{-85}$

Condition	Accuracy	Difference	$p$
REAL	85.56	–	–
MEAN	85.02	$-0.54$	0.375
SWAP	85.56	$+0.00$	1
$\mu+ 0 \cdot(t_i - \mu)$	85.02	$-0.54$	0.375
$\mu+ 2(t_i - \mu)$	84.84	$-0.72$	0.125
NODE (real rows)	85.56	$+0.00$	1
NODE (Gaussian)	85.56	$+0.00$	1
EDGE (rewired)	85.56	$+0.00$	1
REL (resampled)	85.56	$+0.00$	1
zero token	2.17	$-83.39^{***}$	$1.14 \times10^{-135}$
norm $\to$ word size	5.42	$-80.14^{***}$	$6.84 \times10^{-126}$
$\times0.5$	61.73	$-23.83^{***}$	$2.23 \times10^{-23}$
$\times2$	83.39	$-2.17^{\dagger}$	0.0884
NOTXT-REAL	51.99	–	–
NOTXT-MEAN	51.62	$-0.36$	0.5
NOTXT-SWAP	51.99	$+0.00$	1
NOTXT-NODE	51.99	$+0.00$	1
NOTXT-zero	12.27	$-39.71^{***}$	$3.29 \times10^{-38}$
NOTXT-norm $\to$ word size	11.19	$-40.79^{***}$	$1.71 \times10^{-40}$

Condition	Accuracy	Difference	$p$
REAL	85.56	–	–
MEAN	85.02	$-0.54$	0.25
SWAP	85.56	$+0.00$	1
$\mu+ 0 \cdot(t_i - \mu)$	85.02	$-0.54$	0.25
$\mu+ 2(t_i - \mu)$	85.56	$+0.00$	1
NODE (real rows)	85.56	$+0.00$	1
NODE (Gaussian)	85.56	$+0.00$	1
EDGE (rewired)	85.56	$+0.00$	1
REL (resampled)	85.56	$+0.00$	1
zero token	2.17	$-83.39^{***}$	$1.14 \times10^{-135}$
norm $\to$ word size	85.56	$+0.00$	1
$\times0.5$	4.87	$-80.69^{***}$	$8.83 \times10^{-127}$
$\times2$	7.04	$-78.52^{***}$	$3.17 \times10^{-123}$
NOTXT-REAL	59.75	–	–
NOTXT-MEAN	57.22	$-2.53$	0.261
NOTXT-SWAP	59.75	$+0.00$	1
NOTXT-NODE	59.75	$+0.00$	1
NOTXT-zero	12.27	$-47.47^{***}$	$1.57 \times10^{-51}$
NOTXT-norm $\to$ word size	59.75	$+0.00$	1

Table 22: MLPQ, 554 test questions.

Condition	Accuracy	Difference	$p$
REAL	84.84	–	–
MEAN	85.02	+0.18	1
SWAP	85.20	+0.36	0.688
$\mu+ 0 \cdot(t_i - \mu)$	85.02	+0.18	1
$\mu+ 2(t_i - \mu)$	85.20	+0.36	0.5
NODE (real rows)	84.84	+0.00	1
NODE (Gaussian)	84.84	+0.00	1
EDGE (rewired)	84.84	+0.00	1
REL (resampled)	84.84	+0.00	1
zero token	3.61	−81.23***	$1.7 \times10^{-130}$
norm → word size	3.97	−80.87***	$6.71 \times10^{-130}$
×0.5	74.19	−10.65***	$1.26 \times10^{-7}$
×2	84.12	−0.72	0.597
NOTXT-REAL	77.62	–	–
NOTXT-MEAN	77.62	+0.00	1
NOTXT-SWAP	77.80	+0.18	1
NOTXT-NODE	77.62	+0.00	1
NOTXT-zero	12.09	−65.52***	$3.19 \times10^{-91}$
NOTXT-norm → word size	9.75	−67.87***	$2.61 \times10^{-102}$

Table 23: G-Retriever with GELU (no Stage 1), 554 test questions.

Condition	Accuracy	Difference	$p$
REAL	73.29	–	–
SWAP	72.20	−1.08	0.631

Table 24: Stage 1 + Sigmoid projector, 554 test questions.

Condition	Accuracy	Difference	$p$
REAL	84.48	–	–
MEAN	84.48	+0.00	1
SWAP	84.66	+0.18	1

Condition	Accuracy	Difference	$p$
REAL	57.76	–	–
MEAN	58.84	+1.08	0.735
SWAP	51.08	−6.68*	0.0167
NODE (real rows)	56.50	−1.26	0.685
NODE (Gaussian)	48.56	−9.21***	0.000616
EDGE (rewired)	51.62	−6.14*	0.02
REL (resampled)	55.96	−1.81	0.529
norm → word size	3.97	−53.79***	$1.32 \times10^{-79}$
×0.5	33.03	−24.73***	$1.18 \times10^{-17}$
×2	57.58	−0.18	1

Condition	Accuracy	Difference	p
REAL	85.74	–	–
MEAN	86.10	+0.36	0.832
SWAP	85.20	−0.54	0.749
$\mu+ 0 \cdot(t_i - \mu)$	86.10	+0.36	0.832
NODE (real rows)	85.20	−0.54	0.648
NODE (Gaussian)	86.10	+0.36	0.824
EDGE (rewired)	86.82	+1.08	0.21
GSWAP (other graph)	84.30	−1.44	0.243
SWAP (permutation 2)	86.82	+1.08	0.362
SWAP (permutation 3)	85.02	−0.72	0.585
NOTXT-REAL	69.68	–	–
NOTXT-MEAN	69.86	+0.18	1
NOTXT-SWAP	69.86	+0.18	1
NOTXT-GSWAP	69.13	−0.54	0.771

Condition	Accuracy	Difference	p
REAL	76.35	–	–
MEAN	72.74	−3.61*	0.0265
SWAP	77.44	+1.08	0.545
$\mu+ 0 \cdot(t_i - \mu)$	72.92	−3.43*	0.037
NODE (real rows)	77.08	+0.72	0.636
NODE (Gaussian)	75.45	−0.90	0.511
EDGE (rewired)	76.53	+0.18	1
NOTXT-REAL	64.44	–	–
NOTXT-MEAN	61.37	−3.07	0.292
NOTXT-SWAP	64.62	+0.18	1

Condition	Accuracy	Difference	p
REAL	84.48	–	–
MEAN	87.55	+3.07**	0.00232
SWAP	84.48	+0.00	1
NODE (real rows)	84.66	+0.18	1
EDGE (rewired)	84.66	+0.18	1
GSWAP (other graph)	84.12	−0.36	0.754
NOTXT-REAL	70.22	–	–
NOTXT-MEAN	70.94	+0.72	0.834

SceneGraphs

Condition	Accuracy	Difference	$p$
REAL	81.35	–	–
MEAN	81.00	−0.35	0.614
SWAP	80.15	−1.20†	0.0566
$\mu+ 0 \cdot(t_i - \mu)$	80.30	−1.05	0.119
$\mu+ 2(t_i - \mu)$	82.05	+0.70	0.338
NODE (real rows)	81.55	+0.20	0.699
NODE (Gaussian)	81.80	+0.45	0.272
EDGE (rewired)	81.55	+0.20	0.793
REL (resampled)	81.20	−0.15	0.876
zero token	11.30	−70.05***	$< 10^{-300}$
norm $\to$ word size	11.35	−70.00***	$< 10^{-300}$
×0.5	76.85	−4.50***	$1.53 \times10^{-7}$
×2	80.55	−0.80	0.221
NOTXT-REAL	30.20	–	–
NOTXT-MEAN	30.65	+0.45	0.686
NOTXT-SWAP	28.80	−1.40	0.187
NOTXT-NODE	29.75	−0.45	0.684
NOTXT-zero	12.50	−17.70***	$3.5 \times10^{-45}$
NOTXT-norm $\to$ word size	11.60	−18.60***	$2.78 \times10^{-52}$

Condition	Accuracy	Difference	$p$
REAL	80.20	–	–
MEAN	80.80	+0.60	0.213
SWAP	80.40	+0.20	0.781
$\mu+ 0 \cdot(t_i - \mu)$	80.60	+0.40	0.434
$\mu+ 2(t_i - \mu)$	80.15	−0.05	1
NODE (real rows)	80.00	−0.20	0.757
NODE (Gaussian)	80.50	+0.30	0.497
EDGE (rewired)	80.30	+0.10	0.856
REL (resampled)	80.35	+0.15	0.25
zero token	11.30	−68.90***	$< 10^{-300}$
norm $\to$ word size	9.50	−70.70***	$< 10^{-300}$
×0.5	78.90	−1.30†	0.0724
×2	78.55	−1.65*	0.0266
NOTXT-REAL	28.65	–	–
NOTXT-MEAN	28.90	+0.25	0.817
NOTXT-SWAP	28.20	−0.45	0.638
NOTXT-NODE	29.80	+1.15	0.224
NOTXT-zero	12.50	−16.15***	$5.22 \times10^{-36}$
NOTXT-norm $\to$ word size	10.95	−17.70***	$3.5 \times10^{-45}$

Condition	Accuracy	Difference	$p$
REAL	80.55	–	–
MEAN	80.45	−0.10	0.921
SWAP	79.50	−1.05	0.137
$\mu+ 0 \cdot(t_i - \mu)$	80.35	−0.20	0.752
$\mu+ 2(t_i - \mu)$	79.00	$-1.55^{*}$	0.0178
NODE (real rows)	79.30	$-1.25^{\dagger}$	0.0725
NODE (Gaussian)	79.60	−0.95	0.181
EDGE (rewired)	79.30	$-1.25^{\dagger}$	0.0661
REL (resampled)	80.55	+0.00	1
zero token	11.30	$-69.25^{***}$	$< 10^{-300}$
norm $\to$ word size	11.80	$-68.75^{***}$	$< 10^{-300}$
$\times0.5$	78.25	$-2.30^{**}$	0.00388
$\times2$	78.90	$-1.65^{*}$	0.019
NOTXT-REAL	29.80	–	–
NOTXT-MEAN	29.55	−0.25	0.857
NOTXT-SWAP	28.85	−0.95	0.42
NOTXT-NODE	30.90	+1.10	0.334
NOTXT-zero	12.50	$-17.30^{***}$	$2.73 \times10^{-46}$
NOTXT-norm $\to$ word size	12.40	$-17.40^{***}$	$5.23 \times10^{-47}$

Condition	Accuracy	Difference	$p$
REAL	81.40	–	–
MEAN	81.25	−0.15	0.25
SWAP	81.40	+0.00	1
$\mu+ 0 \cdot(t_i - \mu)$	81.25	−0.15	0.25
$\mu+ 2(t_i - \mu)$	80.95	$-0.45^{**}$	0.00391
NODE (real rows)	81.40	+0.00	1
NODE (Gaussian)	81.40	+0.00	1
EDGE (rewired)	81.40	+0.00	1
REL (resampled)	81.40	+0.00	1
zero token	12.20	$-69.20^{***}$	$< 10^{-300}$
norm $\to$ word size	14.80	$-66.60^{***}$	$< 10^{-300}$
$\times0.5$	74.50	$-6.90^{***}$	$2.29 \times10^{-16}$
$\times2$	79.45	$-1.95^{**}$	0.00483
NOTXT-REAL	32.55	–	–
NOTXT-MEAN	32.40	−0.15	0.755
NOTXT-SWAP	32.55	+0.00	1
NOTXT-NODE	32.55	+0.00	1
NOTXT-zero	13.80	$-18.75^{***}$	$1.23 \times10^{-53}$
NOTXT-norm $\to$ word size	12.35	$-20.20^{***}$	$2.46 \times10^{-63}$

Condition	Accuracy	Difference	$p$
REAL	69.15	–	–
MEAN	48.95	−20.20***	$6.8 \times10^{-48}$
SWAP	34.10	−35.05***	$1.11 \times10^{-127}$
$\mu+ 0 \cdot(t_i - \mu)$	49.05	−20.10***	$2.05 \times10^{-47}$
$\mu+ 2(t_i - \mu)$	38.85	−30.30***	$4.36 \times10^{-116}$
NODE (real rows)	60.25	−8.90***	$7.43 \times10^{-15}$
NODE (Gaussian)	60.05	−9.10***	$2.34 \times10^{-16}$
EDGE (rewired)	66.70	−2.45*	0.0159
REL (resampled)	68.30	−0.85	0.244
zero token	11.30	−57.85***	$6.61 \times10^{-293}$
norm → word size	15.50	−53.65***	$1.13 \times10^{-262}$
×0.5	54.70	−14.45***	$1.12 \times10^{-38}$
×2	64.65	−4.50***	$2.06 \times10^{-7}$
NOTXT-REAL	46.60	–	–
NOTXT-MEAN	22.95	−23.65***	$7.41 \times10^{-69}$
NOTXT-SWAP	22.45	−24.15***	$1.66 \times10^{-78}$
NOTXT-NODE	38.90	−7.70***	$1.31 \times10^{-13}$
NOTXT-zero	12.50	−34.10***	$1.03 \times10^{-146}$
NOTXT-norm → word size	12.80	−33.80***	$1.32 \times10^{-143}$

Condition	Accuracy	Difference	$p$
REAL	72.50	–	–
MEAN	71.10	−1.40	.182
SWAP	69.10	−3.40***	$6.31 \times10^{-4}$
SWAP (permutation 2)	67.85	−4.65***	$2.32 \times10^{-6}$
SWAP (permutation 3)	67.85	−4.65***	$1.21 \times10^{-5}$
$\mu+ 0(t_i - \mu)$	70.50	−2.00*	.0481
NODE (real rows)	67.40	−5.10***	$1.33 \times10^{-10}$
NODE (Gaussian)	67.40	−5.10***	$1.57 \times10^{-11}$
EDGE (rewired)	71.45	−1.05	.186
GSWAP (other graph)	67.20	−5.30***	$3.55 \times10^{-9}$
NOTXT-REAL	32.60	–	–
NOTXT-MEAN	26.00	−6.60***	$3.57 \times10^{-8}$
NOTXT-SWAP	21.75	−10.85***	$8.93 \times10^{-19}$
NOTXT-GSWAP	29.50	−3.10**	.00600

Condition	Accuracy	Difference	$p$
REAL	77.40	–	–
MEAN	77.30	−0.10	0.918
SWAP	77.20	−0.20	0.801
$\mu+ 0 \cdot(t_i - \mu)$	77.35	−0.05	1
NODE (real rows)	77.50	+0.10	0.856
NODE (Gaussian)	77.35	−0.05	1
EDGE (rewired)	77.20	−0.20	0.775
NOTXT-REAL	25.50	–	–
NOTXT-MEAN	24.90	−0.60	0.5
NOTXT-SWAP	24.40	−1.10	0.199

Condition	Accuracy	Difference	$p$
REAL	76.90	–	–
MEAN	75.40	$-1.50^{\dagger}$	0.0829
SWAP	72.45	$-4.45^{***}$	$2.02 \times10^{-7}$
$\mu+ 0 \cdot(t_i - \mu)$	74.95	$-1.95^{*}$	0.0242
NODE (real rows)	75.85	$-1.05$	0.158
NODE (Gaussian)	76.10	$-0.80$	0.323
EDGE (rewired)	74.95	$-1.95^{**}$	0.00202
NOTXT-REAL	26.20	–	–
NOTXT-MEAN	25.45	$-0.75$	0.474
NOTXT-SWAP	23.60	$-2.60^{*}$	0.0123

Condition	Accuracy	Difference	$p$
REAL	77.55	–	–
MEAN	76.45	$-1.10$	0.191
SWAP	75.10	$-2.45^{**}$	0.00146
$\mu+ 0 \cdot(t_i - \mu)$	76.50	$-1.05$	0.214
NODE (real rows)	78.55	$+1.00$	0.116
NODE (Gaussian)	78.15	$+0.60$	0.366
EDGE (rewired)	77.85	$+0.30$	0.734
NOTXT-REAL	31.25	–	–
NOTXT-MEAN	33.00	$+1.75$	0.13
NOTXT-SWAP	29.55	$-1.70$	0.136

Condition	Accuracy	Difference	$p$
REAL	79.65	–	–
MEAN	73.90	$-5.75^{***}$	$2.1 \times10^{-11}$
SWAP	70.05	$-9.60^{***}$	$1.2 \times10^{-23}$
NODE (real rows)	79.75	$+0.10$	0.73
EDGE (rewired)	80.05	$+0.40$	0.56
GSWAP (other graph)	80.00	$+0.35$	0.54
NOTXT-REAL	30.10	–	–
NOTXT-MEAN	17.25	$-12.85^{***}$	$2.0 \times10^{-35}$

SWAP on our standard-recipe SceneGraphs model is not an artefact of one pairing: two further derangements give $-4.65$ ($p = 2.3 \times10^{-6}$) and $-4.65$ ($p = 1.2 \times10^{-5}$) on the same 2,000 questions, against the $-3.40$ reported in Table 4, which is the weakest of the three.

WebQSP

Condition	Accuracy	Difference	$p$
REAL	70.33	–	–
MEAN	68.49	$-1.84^{*}$	0.0285
SWAP	69.41	$-0.92$	0.311
$\mu+ 0 \cdot(t_i - \mu)$	70.70	$+0.37$	0.687
$\mu+ 2(t_i - \mu)$	70.95	$+0.61$	0.495
NODE (real rows)	70.39	$+0.06$	1
NODE (Gaussian)	70.21	$-0.12$	0.933
EDGE (rewired)	71.01	$+0.68$	0.46
REL (resampled)	70.82	$+0.49$	0.575
GSWAP (other graph)	69.78	$-0.55$	0.569
SWAP (permutation 2)	69.96	$-0.37$	0.711
SWAP (permutation 3)	70.15	$-0.18$	0.874
zero token	10.69	$-59.64^{***}$	$1.59 \times10^{-266}$
norm $\to$ word size	4.85	$-65.48^{***}$	$< 10^{-300}$
$\times0.5$	63.02	$-7.31^{***}$	$1.9 \times10^{-12}$
$\times2$	66.09	$-4.24^{***}$	$7.26 \times10^{-6}$
NOTXT-REAL	41.34	–	–
NOTXT-MEAN	42.38	$+1.04$	0.187
NOTXT-SWAP	41.15	$-0.18$	0.886
NOTXT-NODE	41.28	$-0.06$	1
NOTXT-GSWAP	40.48	$-0.86$	0.356
NOTXT-zero	4.05	$-37.29^{***}$	$1.79 \times10^{-157}$
NOTXT-norm $\to$ word size	2.76	$-38.57^{***}$	$1.56 \times10^{-170}$

Condition	Accuracy	Difference	$p$
REAL	74.32	–	–
MEAN	73.89	$-0.43$	0.634
SWAP	73.59	$-0.74$	0.382
$\mu+ 0 \cdot(t_i - \mu)$	73.89	$-0.43$	0.636
NODE (real rows)	73.34	$-0.98$	0.224
NODE (Gaussian)	73.59	$-0.74$	0.359
EDGE (rewired)	74.20	$-0.12$	0.935
NOTXT-REAL	10.38	–	–
NOTXT-MEAN	11.49	$+1.11$	0.215
NOTXT-SWAP	11.43	$+1.04$	0.221

Condition	Accuracy	Difference	$p$
REAL	55.47	–	–
MEAN	54.98	$-0.49$	0.48
SWAP	54.79	$-0.68$	0.428
$\mu+ 0 \cdot(t_i - \mu)$	54.98	$-0.49$	0.48
NODE (real rows)	54.85	$-0.61$	0.378
NODE (Gaussian)	54.67	$-0.80$	0.232
EDGE (rewired)	55.41	$-0.06$	1
NOTXT-REAL	34.40	–	–
NOTXT-MEAN	30.65	$-3.75$<sup>***</sup>	$2.45 \times10^{-5}$
NOTXT-SWAP	32.31	$-2.09$<sup>*</sup>	0.0286

Condition	Accuracy	Difference	$p$
REAL	64.62	–	–
MEAN	49.88	$-14.74$<sup>***</sup>	$1.26 \times10^{-30}$
SWAP	54.73	$-9.89$<sup>***</sup>	$4.62 \times10^{-15}$
$\mu+ 0 \cdot(t_i - \mu)$	49.88	$-14.74$<sup>***</sup>	$1.26 \times10^{-30}$
$\mu+ 2(t_i - \mu)$	37.65	$-26.97$<sup>***</sup>	$8.63 \times10^{-82}$
NODE (real rows)	62.22	$-2.40$<sup>*</sup>	0.0289
NODE (Gaussian)	63.08	$-1.54$	0.173
EDGE (rewired)	64.68	$+0.06$	1
REL (resampled)	63.82	$-0.80$	0.424
zero token	10.69	$-53.93$<sup>***</sup>	$1.84 \times10^{-224}$
norm $\to$ word size	14.37	$-50.25$<sup>***</sup>	$3.14 \times10^{-203}$
$\times0.5$	54.05	$-10.57$<sup>***</sup>	$1.04 \times10^{-18}$
$\times2$	54.30	$-10.32$<sup>***</sup>	$4.88 \times10^{-20}$
NOTXT-REAL	36.30	–	–
NOTXT-MEAN	16.15	$-20.15$<sup>***</sup>	$1.76 \times10^{-53}$
NOTXT-SWAP	28.01	$-8.29$<sup>***</sup>	$8.71 \times10^{-12}$
NOTXT-NODE	34.28	$-2.03$<sup>†</sup>	0.0911
NOTXT-zero	4.05	$-32.25$<sup>***</sup>	$1.22 \times10^{-136}$
NOTXT-norm $\to$ word size	4.12	$-32.19$<sup>***</sup>	$3.3 \times10^{-139}$

Condition	Accuracy	Difference	$p$
REAL	69.90	–	–
MEAN	66.77	$-3.13^{***}$	0.000865
SWAP	67.75	$-2.15^{*}$	0.0387
$\mu+ 0 \cdot(t_i - \mu)$	67.38	$-2.52^{**}$	0.00807
NODE (real rows)	68.67	$-1.23$	0.216
NODE (Gaussian)	68.55	$-1.35$	0.157
EDGE (rewired)	69.10	$-0.80$	0.415
GSWAP (other graph)	67.94	$-1.97^{\dagger}$	0.0515
SWAP (permutation 2)	67.75	$-2.15^{*}$	0.0344
SWAP (permutation 3)	66.46	$-3.44^{***}$	0.000785
NOTXT-REAL	28.75	–	–
NOTXT-MEAN	26.66	$-2.09^{*}$	0.0452
NOTXT-SWAP	26.35	$-2.40^{*}$	0.0283
NOTXT-GSWAP	26.47	$-2.27^{*}$	0.0372

Condition	Accuracy	Difference	$p$
REAL	74.57	–	–
MEAN	73.46	$-1.11$	0.195
SWAP	72.24	$-2.33^{**}$	0.00839
$\mu+ 0 \cdot(t_i - \mu)$	72.60	$-1.97^{*}$	0.0192
NODE (real rows)	73.77	$-0.80$	0.332
NODE (Gaussian)	73.40	$-1.17$	0.161
EDGE (rewired)	74.82	$+0.25$	0.811
NOTXT-REAL	47.91	–	–
NOTXT-MEAN	49.20	$+1.29$	0.239
NOTXT-SWAP	44.72	$-3.19^{**}$	0.00518

Condition	Accuracy	Difference	$p$
REAL	71.25	–	–
MEAN	71.44	+0.18	0.69
SWAP	71.25	+0.00	1
$\mu+ 0 \cdot(t_i - \mu)$	71.44	+0.18	0.69
NODE (real rows)	71.25	+0.00	1
NODE (Gaussian)	71.25	+0.00	1
EDGE (rewired)	71.25	+0.00	1
NOTXT-REAL	42.57	–	–
NOTXT-MEAN	42.14	$-0.43$	0.189
NOTXT-SWAP	42.57	+0.00	1

Condition	Accuracy	Difference	$p$
REAL	70.70	–	–
MEAN	70.21	$-0.49$	0.115
SWAP	68.98	$-1.72^{*}$	0.015
$\mu+ 0 \cdot(t_i - \mu)$	70.21	$-0.49$	0.115
NODE (real rows)	70.70	+0.00	1
NODE (Gaussian)	70.70	+0.00	1
EDGE (rewired)	70.70	+0.00	1
NOTXT-REAL	43.37	–	–
NOTXT-MEAN	43.61	+0.25	0.608
NOTXT-SWAP	44.04	$+0.68^{\dagger}$	0.0801

Condition	Accuracy	Difference	$p$
REAL	65.79	–	–
MEAN	63.70	$-2.09^{**}$	0.00187
SWAP	63.88	$-1.90^{**}$	0.00573
$\mu+ 0 \cdot(t_i - \mu)$	63.70	$-2.09^{**}$	0.00187
NODE (real rows)	64.99	$-0.80$	0.25
NODE (Gaussian)	65.17	$-0.61$	0.373
EDGE (rewired)	64.43	$-1.35^{*}$	0.0315
NOTXT-REAL	34.34	–	–
NOTXT-MEAN	34.64	+0.31	0.768
NOTXT-SWAP	35.44	+1.11	0.217

Condition	Accuracy	Difference	$p$
REAL	57.13	–	–
MEAN	56.57	−0.55	0.478
SWAP	55.71	−1.41	0.104
$\mu+ 0 \cdot(t_i - \mu)$	56.70	−0.43	0.595
NODE (real rows)	56.82	−0.31	0.704
NODE (Gaussian)	56.57	−0.55	0.452
EDGE (rewired)	57.19	+0.06	1
NOTXT-REAL	29.30	–	–
NOTXT-MEAN	30.16	+0.86	0.403
NOTXT-SWAP	27.27	−2.03†	0.0508

Condition	Accuracy	Difference	$p$
REAL	70.58	–	–
MEAN	70.02	−0.55	0.538
SWAP	70.15	−0.43	0.654
$\mu+ 0 \cdot(t_i - \mu)$	69.90	−0.68	0.46
NODE (real rows)	70.33	−0.25	0.784
NODE (Gaussian)	70.58	+0.00	1
EDGE (rewired)	70.64	+0.06	1
NOTXT-REAL	42.14	–	–
NOTXT-MEAN	43.06	+0.92	0.115
NOTXT-SWAP	41.77	−0.37	0.637

Condition	Accuracy	Difference	$p$
REAL	67.75	–	–
MEAN	67.69	−0.06	1
SWAP	67.26	−0.49	0.614
$\mu+ 0 \cdot(t_i - \mu)$	67.57	−0.18	0.89
NODE (real rows)	67.26	−0.49	0.598
NODE (Gaussian)	67.63	−0.12	0.942
EDGE (rewired)	67.20	−0.55	0.573
NOTXT-REAL	37.96	–	–
NOTXT-MEAN	38.39	+0.43	0.562
NOTXT-SWAP	37.29	−0.68	0.4

Condition	Accuracy	Difference	$p$
REAL	70.02	–	–
MEAN	70.02	+0.00	1
SWAP	69.78	−0.25	0.836
$\mu+ 0 \cdot(t_i - \mu)$	68.86	−1.17	0.22
NODE (real rows)	69.16	−0.86	0.356
NODE (Gaussian)	70.52	+0.49	0.617
EDGE (rewired)	69.66	−0.37	0.718
NOTXT-REAL	34.09	–	–
NOTXT-MEAN	34.89	+0.80	0.275
NOTXT-SWAP	35.01	+0.92	0.248

Condition	Accuracy	Difference	$p$
REAL	63.14	–	–
MEAN	59.71	$-3.44^{**}$	0.00341
SWAP	57.80	$-5.34^{***}$	$6.9 \times10^{-6}$
$\mu+ 0 \cdot(t_i - \mu)$	60.01	$-3.13^{**}$	0.0045
NODE (real rows)	61.49	$-1.66$	0.127
NODE (Gaussian)	62.16	$-0.98$	0.385
EDGE (rewired)	64.19	$+1.04$	0.32
NOTXT-REAL	36.61	–	–
NOTXT-MEAN	41.58	$+4.98^{***}$	$1.71 \times10^{-5}$
NOTXT-SWAP	33.35	$-3.26^{**}$	0.00517

Condition	Accuracy	Difference	$p$
REAL	63.64	–	–
MEAN	60.69	$-2.95^{**}$	0.0075
SWAP	58.42	$-5.22^{***}$	$1.41 \times10^{-6}$
$\mu+ 0 \cdot(t_i - \mu)$	59.28	$-4.36^{***}$	$7.12 \times10^{-5}$
NODE (real rows)	60.87	$-2.76^{**}$	0.00953
NODE (Gaussian)	62.59	$-1.04$	0.324
EDGE (rewired)	62.59	$-1.04$	0.327
NOTXT-REAL	38.33	–	–
NOTXT-MEAN	40.36	$+2.03^{\dagger}$	0.0812
NOTXT-SWAP	34.77	$-3.56^{**}$	0.0015

Condition	Accuracy	Difference	$p$
REAL	64.56	–	–
MEAN	60.32	$-4.24^{***}$	$5.2 \times10^{-5}$
SWAP	60.87	$-3.69^{***}$	0.000608
$\mu+ 0 \cdot(t_i - \mu)$	61.86	$-2.70^{**}$	0.00801
NODE (real rows)	63.27	$-1.29$	0.216
NODE (Gaussian)	63.21	$-1.35$	0.196
EDGE (rewired)	63.45	$-1.11$	0.262
NOTXT-REAL	41.83	–	–
NOTXT-MEAN	43.92	$+2.09^{*}$	0.0413
NOTXT-SWAP	38.88	$-2.95^{**}$	0.00638

Condition	Accuracy	Difference	$p$
REAL	67.20	–	–
MEAN	68.55	$+1.35$	0.143
SWAP	66.89	$-0.31$	0.794
$\mu+ 0 \cdot(t_i - \mu)$	68.92	$+1.72^{\dagger}$	0.0511
NODE (real rows)	69.23	$+2.03^{*}$	0.0281
NODE (Gaussian)	68.49	$+1.29$	0.172
EDGE (rewired)	69.10	$+1.90^{*}$	0.035
NOTXT-REAL	26.97	–	–
NOTXT-MEAN	31.27	$+4.30^{***}$	0.00043
NOTXT-SWAP	25.12	$-1.84$	0.132

Condition	Accuracy	Difference	$p$
REAL	69.53	–	–
MEAN	68.00	$-1.54^{\dagger}$	0.0967
SWAP	66.34	$-3.19^{**}$	0.00151
$\mu+ 0 \cdot(t_i - \mu)$	67.87	$-1.66^{\dagger}$	0.0732
NODE (real rows)	66.28	$-3.26^{***}$	0.000443
NODE (Gaussian)	68.24	$-1.29$	0.174
EDGE (rewired)	68.98	$-0.55$	0.567
GSWAP (other graph)	66.71	$-2.83^{**}$	0.00499
SWAP (permutation 2)	67.63	$-1.90^{\dagger}$	0.0581
SWAP (permutation 3)	66.46	$-3.07^{**}$	0.0018
NOTXT-REAL	19.41	–	–
NOTXT-MEAN	21.99	$+2.58^{*}$	0.0252
NOTXT-SWAP	17.87	$-1.54$	0.179
NOTXT-GSWAP	19.04	$-0.37$	0.776

Condition	Accuracy	Difference	$p$
REAL	66.89	–	–
MEAN	65.11	−1.78†	0.0888
SWAP	65.85	−1.04	0.322
$\mu+ 0 \cdot(t_i - \mu)$	63.70	−3.19**	0.00139
NODE (real rows)	64.68	−2.21*	0.0298
NODE (Gaussian)	65.29	−1.60	0.101
EDGE (rewired)	65.36	−1.54	0.12
NOTXT-REAL	22.91	–	–
NOTXT-MEAN	9.34	−13.57***	$4.3 \times10^{-35}$
NOTXT-SWAP	21.13	−1.78	0.115

Condition	Accuracy	Difference	$p$
REAL	68.43	–	–
MEAN	68.18	−0.25	0.848
SWAP	68.30	−0.12	0.948
$\mu+ 0 \cdot(t_i - \mu)$	68.73	+0.31	0.797
NODE (real rows)	68.37	−0.06	1
NODE (Gaussian)	68.80	+0.37	0.737
EDGE (rewired)	68.92	+0.49	0.598
NOTXT-REAL	28.01	–	–
NOTXT-MEAN	24.69	−3.32***	0.000237
NOTXT-SWAP	25.86	−2.15*	0.0163

Condition	Accuracy	Difference	$p$
REAL	70.95	–	–
MEAN	69.78	−1.17	0.166
SWAP	70.02	−0.92	0.279
$\mu+ 0 \cdot(t_i - \mu)$	69.53	−1.41†	0.0845
NODE (real rows)	69.47	−1.47†	0.0669
NODE (Gaussian)	69.59	−1.35†	0.0945
EDGE (rewired)	69.29	−1.66*	0.0328
NOTXT-REAL	41.09	–	–
NOTXT-MEAN	41.34	+0.25	0.845
NOTXT-SWAP	39.62	−1.47	0.161

Condition	Accuracy	Difference	$p$
REAL	72.17	–	–
MEAN	71.93	$-0.25$	0.805
SWAP	71.99	$-0.18$	0.859
$\mu+ 0 \cdot(t_i - \mu)$	70.82	$-1.35^{\dagger}$	0.0882
NODE (real rows)	71.87	$-0.31$	0.738
NODE (Gaussian)	71.56	$-0.61$	0.456
EDGE (rewired)	71.19	$-0.98$	0.221
NOTXT-REAL	20.64	–	–
NOTXT-MEAN	21.99	$+1.35^{*}$	0.0448
NOTXT-SWAP	21.74	$+1.11$	0.181

Condition	Accuracy	Difference	$p$
REAL	72.73	–	–
MEAN	66.58	−6.14***	$1.28 \times10^{-9}$
SWAP	67.08	−5.65***	$5.88 \times10^{-9}$
NODE (real rows)	72.11	−0.61	0.415
EDGE (rewired)	72.30	−0.43	0.664
GSWAP (other graph)	70.76	−1.97*	0.0347
NOTXT-REAL	28.38	–	–
NOTXT-MEAN	20.88	−7.49***	$4.94 \times10^{-11}$

Condition	Accuracy	Difference	$p$
REAL	69.84	–	–
MEAN	69.78	−0.06	1
SWAP	66.15	−3.69***	0.000365
NODE (real rows)	67.75	−2.09*	0.0367
EDGE (rewired)	69.35	−0.49	0.636
GSWAP (other graph)	67.51	−2.33*	0.0247
NOTXT-REAL	30.90	–	–
NOTXT-MEAN	31.33	+0.43	0.742

The token's form: magnitude and residual conditions

These conditions probe the token’s geometry rather than which factor it carries, which is why they are not part of Table 8. Table 63 reports them for the three graph-transformer models and the two graph-free controls on ExplaGraphs; the same conditions for every other model are in Appendix I.

Dataset	Model	REAL	zero token norm → word	×0.5	×2
ExplaGraphs	G-Retriever (GT)	85.56	−81.95{sup}***`	−81.95{sup}***`	−8.12{sup}***`	−3.61{sup}***`
ExplaGraphs	CONST	85.56	−83.39{sup}***`	−80.14{sup}***`	−23.83{sup}***`	−2.17{sup}†`
ExplaGraphs	CONST at word norm	85.56	−83.39{sup}***`	0.00	−80.69{sup}***`	−78.52{sup}***`
SceneGraphs	G-Retriever (GT)	81.35	−70.05{sup}***`	−70.00{sup}***`	−4.50{sup}***`	−0.80
SceneGraphs	CONST	81.40	−69.20{sup}***`	−66.60{sup}***`	−6.90{sup}***`	−1.95{sup}**`
WebQSP	G-Retriever (GT)	70.33	−59.64{sup}***`	−65.48{sup}***`	−7.31{sup}***`	−4.24{sup}***`

A trained model depends on the length its token had during training, in both directions, and a large token is not necessary: the constant held at word norm throughout training reaches the same accuracy as the others and then breaks when its length is changed afterwards.

Complete graph-free controls

Table 64 gives the controls in full. All four models share G-Retriever’s prompt, answer loss, optimiser and training schedule. CONST and MLPQ also share its projector, so they differ from G-Retriever only in the token’s source; PT differs in both the source and the parameterisation, since its vectors are optimised directly in the embedding space, and its capacity differs by more than two orders of magnitude. The CONST–PT gap therefore remains confounded between parameterisation and capacity, which is why the main text does not use it as a graph-versus-no-graph comparison.

Accuracy does not track capacity: PT to CONST multiplies the trainable parameters by 256 and adds 19 Hit@1 on WebQSP, while CONST to our Stage 2 multiplies them by two and removes 1.35.

Gold-answer likelihood

Table 65 reports the measurement defined in Equation (4). It is a second reading of conditions that were already run, not a new experiment.

Three readings. In the model whose token can only carry the graph, swapping the graph and swapping the token cost the same likelihood (+0.0845 against +0.0848), which is the internal consistency check for the instrument. On SceneGraphs the graph swap is seven times the token swap, the largest effect measured anywhere in this study and the distribution-level counterpart of the $-5.30$ accuracy drop, while the mean token lowers the likelihood: an empty token is better than a wrong graph. On ExplaGraphs nothing moves for either model, which settles that dataset at the distribution level and not merely below the accuracy resolution.

Complete token-content and use results

Dataset	Model	REAL	MEAN	SWAP	GSWAP
WebQSP	G-Retriever	0.7558	+0.0131 [+.0097, +.0166]	+0.0183 [+.0137, +.0229]	+0.0182 [+.0136, +.0228]
WebQSP	ours	0.8540	−0.0159 [−.0288, −.0041]	+0.0895 [+.0727, +.1074]	+0.0332 [+.0185, +.0461]
WebQSP	ours, no entity features	0.8213	+0.0378 [+.0257, +.0506]	+0.1111 [+.0954, +.1261]	+0.0262 [+.0175, +.0348]
WebQSP	ours, no question	0.8517	+0.0070 [−.0071, +.0203]	+0.0848 [+.0671, +.1020]	+0.0845 [+.0668, +.1015]
WebQSP	G-Retriever, Qwen2.5-3B	1.0367	+0.0109 [+.0038, +.0185]	+0.0734 [+.0623, +.0845]	+0.0737 [+.0626, +.0850]
WebQSP	ours, Qwen2.5-3B	1.0553	+0.0123 [+.0066, +.0177]	+0.0168 [+.0106, +.0226]	+0.0144 [+.0076, +.0211]
SceneGraphs	G-Retriever	0.2477	+0.0053 [+.0006, +.0100]	+0.0169 [+.0102, +.0240]	+0.0167 [+.0101, +.0238]
SceneGraphs	ours	0.4122	−0.0287 [−.0556, −.0027]	+0.0229 [−.0041, +.0498]	+0.1665 [+.1353, +.1999]
ExplaGraphs	G-Retriever	0.2068	+0.0006 [−.0048, +.0059]	+0.0014 [−.0063, +.0088]	+0.0022 [−.0051, +.0095]
ExplaGraphs	ours	0.1696	−0.0047 [−.0160, +.0052]	+0.0010 [−.0120, +.0130]	+0.0032 [−.0091, +.0152]

Table 66 extends the Llama-2 results in Table 5 with the complete Qwen2.5-3B replication and diagnostic variants. The probe is trained on half of the test tokens and evaluated on the other half by retrieving the corresponding question, answer, or verbalised graph. The candidate sets contain 814 targets on WebQSP, 1,000 on SceneGraphs, and 277 on ExplaGraphs. A constant token therefore has median rank 407, 500, and 139, respectively. Repeated SceneGraphs answers place the answer control between 476 and 500, depending on the held-out split.

Dataset	Model	Q rank	A rank	G rank	$\Delta$Score GSWAP	$\Delta$NLL S/GS	Diagnosis
Llama-2-7B
WebQSP	G-Retriever	293	317	369	−0.55	+0.0183 / +0.0182	SOFT PROMPT
	MLPQ	407	407	407	–	–	–
	Ours	65	60	315	−1.97†	+0.0895 / +0.0332	MASKED
	no entity features	34	51	303	−1.97*	+0.1111 / +0.0262	PRIOR
	no question	120	134	329	−2.33*	+0.0848 / +0.0845	USED
SceneGraphs	G-Retriever	271	436	248	−1.10†	+0.0169 / +0.0167	SOFT PROMPT
	CONST	500	476	500	–	–	–
	Ours	47	187	427	−5.30***	+0.0229 / +0.1665	USED
	no entity features	8	59	369	+0.35	+0.1553 / +0.0045	PRIOR
	no question	116	395	185	−0.25	+0.0214 / +0.0213	MASKED
ExplaGraphs	G-Retriever	138	–	139	−1.08	+0.0014 / +0.0022	SOFT PROMPT
	Ours	54	–	41	−1.44	+0.0010 / +0.0032	DECODABLE
	no entity features	67	–	67	−0.36	+0.0026 / +0.0056	DECODABLE
	no question	84	–	70	+1.26	−0.0063 / −0.0073	DECODABLE
Qwen2.5-3B, matched training budget
WebQSP	G-Retriever	348	357	388	−1.60†	+0.0734 / +0.0737	SOFT PROMPT
	Ours	153	156	349	−2.09**	+0.0168 / +0.0144	USED
	no entity features	109	110	344	−0.86	+0.0321 / +0.0166	PRIOR
	no question	188	198	352	−2.33**	+0.0336 / +0.0333	USED
SceneGraphs	G-Retriever	326	446	340	−1.15*	+0.0127 / +0.0125	SOFT PROMPT
	Ours	112	353	450	−6.50***	+0.0392 / +0.0410	USED
	no entity features	29	121	402	−0.60	+0.0373 / +0.0019	PRIOR
	no question	155	408	256	−1.90**	+0.0174 / +0.0172	DECODABLE
ExplaGraphs	G-Retriever	139	–	138	−4.51*	+0.0146 / +0.0138	USED
	Ours	85	–	62	+0.36	+0.0008 / +0.0042	DECODABLE
	no entity features	92	–	87	−1.99	+0.0188 / +0.0177	DECODABLE
	no question	109	–	99	−1.44	+0.0017 / +0.0008	DECODABLE

Dataset	Model	NOTXT	$\Delta_{\mathrm{MEAN}}$	$\Delta_{\mathrm{SWAP}}$	$\Delta_{\mathrm{GSWAP}}$	With text
WebQSP	G-Retriever	29.30	+0.86	$-2.03^{\dagger}$	$-1.84^{\dagger}$	57.13
WebQSP	Ours	34.34	+0.31	+1.11	+1.11	65.79
WebQSP	no entity features	30.10	$+1.97^{*}$	–	–	58.54
WebQSP	no question	32.74	+0.61	–	–	60.14
SceneGraphs	G-Retriever	39.50	$+1.95^{*}$	–	–	76.05
SceneGraphs	Ours	43.90	$-7.10^{***}$	–	–	72.65
SceneGraphs	no entity features	37.75	$-2.90^{*}$	–	–	75.95
SceneGraphs	no question	34.70	$-3.15^{***}$	–	–	75.40
ExplaGraphs	G-Retriever	78.88	0.00	–	–	74.37
ExplaGraphs	Ours	82.13	$-1.62$	+0.18	$-0.18$	83.94
ExplaGraphs	no entity features	75.99	$-4.51^{*}$	–	–	80.14
ExplaGraphs	no question	76.17	$-7.40^{***}$	–	–	74.37

The control rows validate the probe: MLPQ on WebQSP and CONST on SceneGraphs remain at the corresponding median control ranks. The cross-fitted WebQSP model was not probed and is therefore omitted.

The Qwen replication reinforces the distinction between recoverability and use. On SceneGraphs, our model produces the largest graph-replacement effect in the study ($-6.50$), although its graph rank is close to the constant control (450 versus 500). On ExplaGraphs, the graph is substantially more recoverable (62 versus 139), yet replacing it has no resolved effect. A linear probe can therefore fail to recover information that affects the LLM or recover information that the LLM ignores.

Qwen G-Retriever on ExplaGraphs provides a further exception: its tokens remain geometrically collapsed, yet GSWAP causes a resolved $-4.51$-point effect. Collapse indicates that little input-specific variation remains, but does not establish complete insensitivity to that variation. For collapsed baselines, nearly identical SWAP and GSWAP likelihood effects indicate generic sensitivity to token perturbation rather than identifying graph-specific use.

Three caveats bound the decodability results. The sentence encoder truncates long graph verbalisations, a single sentence vector compresses a subgraph lossily, and the fitted map is linear. Failure to decode is therefore not proof that information is absent, whereas successful decoding is evidence that it is recoverable by this probe.

The Qwen ExplaGraphs no-entity and no-question variants use seed 1 because their seed-0 runs diverged and scored below the majority baseline.

Validation of the design choices

Question conditioning in Stage 1. Table 67 reports Stage 1 measured on its own, without the LLM. Top-1 is the fraction of validation samples whose own target is the most similar of all validation targets; the graph gap is the mean cosine between representations of different questions about the same graph minus the mean cosine across graphs, and is defined only on SceneGraphs, the one benchmark that asks several questions about one graph. Removing the question conditioning drops top-1 from 0.433 to 0.084, because the answer depends on what is asked and the encoder needs the question to select the relevant part of the scene. The graph gap shows that what it learns is not merely a copy of the question.

Run	Target	Validation top-1	Graph gap
SceneGraphs, with question	full answer sentence	0.433	0.035 → 0.204
SceneGraphs, without question	full answer sentence	0.084	Same embeddings, N/A
SceneGraphs, scene-disjoint halves	full answer sentence	0.376	0.035 → 0.264
WebQSP, half split	gold answers	0.469	N/A
ExplaGraphs, half split	verbalised graph	0.501	N/A

Stage-1 objective. With $h_i$ the pooled encoder output, $s_i = \operatorname{LM}(c_i)$ the frozen sentence embedding of the target $c_i$, batches of size $B$, temperature $\kappa$ and a memory bank $\mathcal{M}$ of $M$ other samples’ targets, Stage 1 minimises

Stage-1 targets. The target is the gold answer on WebQSP and the full answer sentence on SceneGraphs, which contains 47,487 distinct targets. ExplaGraphs has only two answer labels, so an answer target cannot identify a sample and we use the verbalised graph instead.

Freezing the encoder in Stage 2. With a trainable encoder the answer loss re-collapses the token even with a GELU projector, to cosine 0.9139 (Table 3). We verified that the frozen encoder reproduces the intended graph token to within $1.2 \times10^{-7}$.

Parameterisation of the prompt token. The prompt token is a learned constant passed through a Sigmoid projector, which matches G-Retriever on all three datasets (Table 2). Raw prompt tuning scores 11–25 points lower, although this comparison does not isolate parameterisation from capacity.

Alternatives that did not work. Table 68 lists three, all trained with the cross-fitted recipe on WebQSP and therefore comparable with the cross-fitted joint model rather than with the main-text numbers. Each answers a question about the design.

Variant	Hit@1	Token cosine	MEAN	SWAP	NODE
Joint Stage 2, as used in the paper	69.53	0.1778	−1.54†	−3.19**	−3.26***
Projector only	63.14	0.5700	−3.44***	−5.34***	−1.66
Prompt token added after the projector	63.64	0.5700	−2.95***	−5.22***	−2.76***
Dropout 0.5 on the projector	64.56	0.7563	−4.24***	−3.69***	−1.29
40% of verbalised edges hidden in training	67.20	0.1847	+1.35	−0.31	+2.03**

Must the two tokens be trained together? If the split alone were what matters, adding the prompt token to a converged projector should work as well as training both from the start. It does not: accuracy stays at 63.64 and the best epoch is the first, because by then the model has settled into using the graph token for both roles and the new token has nothing to take over.

Is a second token needed at all, or only some way of stopping the graph token from being relied on as a constant? Dropout on the projector is the cheaper alternative, and it leaves one token doing both jobs: accuracy reaches 64.56 and the token drifts back towards collapse, at cosine 0.7563 against 0.1778 for the joint model.

Could the redundancy be removed from the data instead of the architecture? Hiding 40% of the verbalised edges during training makes the text an unreliable channel, which should force the token to carry the graph. Instead the model learns to compensate for missing text, and at test time, when the text is complete again, substitute tokens are as good as the real one or better, with other graphs’ node features worth +2.03 points ($p = 0.03$).

Pipeline and robustness ablations

Table 69 extends Table 4 with the complete set of retrained pipeline variants and the principal fixed-weight interventions. The no-entity and no-question rows remove an input during both training and evaluation; they are therefore pipeline ablations rather than interventions on a fixed model. Additional relation and geometric interventions are reported in Appendices § 17 and § 18.

Removing entity features has a benchmark-dependent effect on task performance: it hurts Qwen on WebQSP and both LLMs on ExplaGraphs, but improves both LLMs on SceneGraphs. Its effect on token use is more consistent. Across both LLMs and all three benchmarks, the no-entity model remains sensitive to MEAN or SWAP, while no GSWAP effect clears the answer-level criterion. The token can therefore remain strongly used after its resolved graph-specific contribution disappears.

Removing question conditioning produces resolved graph use on WebQSP with both LLMs. On Qwen SceneGraphs, the GSWAP effect is statistically significant but does not clear the two-point criterion; neither LLM shows resolved graph use in the corresponding ExplaGraphs variant.

For the standard Qwen SceneGraphs pipeline, the earliest checkpoint was selected and validation loss subsequently increased (0.2668, 0.2741, 0.2737, 0.2981, 0.3396). The same immediate overfitting occurs with Llama-2.

Evaluation without the verbalised graph

Tables 70 and 71 evaluate the trained models after deleting the verbalised graph from the prompt. The graph token is then the only possible graph channel. NOTXT is the resulting absolute score, and intervention columns report condition minus NOTXT. These evaluations are fixed-weight interventions: the models are not retrained without text.

A NOTXT-MEAN effect establishes dependence on sample-specific token content, but does not identify that content as graph-specific. NOTXT-GSWAP provides the graph-specific test because it changes the graph while preserving the remaining inputs. We therefore do not infer that graph information is masked by text from MEAN alone.

With Llama-2, removing the verbalised graph exposes resolved graph-specific effects for the standard pipeline on WebQSP and SceneGraphs, and for the question-free SceneGraphs variant. No corresponding effect is found on ExplaGraphs.

With Qwen2.5-3B, several models depend on sample-specific token content after the text is removed, most strongly on SceneGraphs and in the ExplaGraphs ablations. Most of these rows currently contain only MEAN; they therefore establish token dependence but do not identify the graph as the information being used. The standard Qwen ExplaGraphs pipeline instead shows no resolved text-free token effect, despite its improved task score and decodable graph information.

Table 69: Pipeline variants under fixed-weight interventions. Ours denotes the complete two-token pipeline; indented rows are retrained diagnostic variants. Projector only is the single-token model with the aligned frozen encoder but no separate prompt token. Intervention columns report condition minus REAL, so negative values indicate worse performance. Lower token cosine indicates greater input dependence. Dashes denote conditions not evaluated.

Dataset	Model	REAL	Token cos.	$\Delta$MEAN	$\Delta$SWAP	$\Delta$NODE	$\Delta$EDGE	$\Delta$GSWAP
Llama-2-7B
WebQSP	projector only	64.62	0.543	−14.74***	−9.89***	−2.40*	+0.06	–
	Ours	69.90	0.142	−3.13***	−2.15*	−1.23	−0.80	−1.97†
	no entity features	72.73	0.125	−6.14***	−5.65***	−0.61	−0.43	−1.97*
	no question	69.84	0.169	−0.06	−3.69***	−2.09*	−0.49	−2.33*
	cross-fitted	69.53	0.178	−1.54†	−3.19**	−3.26***	−0.55	−2.83**
	G-Retriever + LoRA	74.32	0.997	−0.43	−0.74	−0.98	−0.12	–
	Ours + LoRA	74.57	0.170	−1.11	−2.33**	−0.80	+0.25	–
SceneGraphs	projector only	69.15	0.519	−20.20***	−35.05***	−8.90***	−2.45*	–
	Ours	72.50	0.133	−1.40	−3.40***	−5.10***	−1.05	−5.30***
	no entity features	79.65	0.125	−5.75***	−9.60***	+0.10	+0.40	+0.35
	no question	76.35	0.225	−1.15	−0.25	−0.75	+0.35	−0.25
ExplaGraphs	projector only	57.76	0.323	+1.08	−6.68*	−1.26	−6.14*	–
	Ours	85.74	0.099	+0.36	−0.54	−0.54	+1.08	−1.44
	no entity features	84.48	0.113	+3.07**	+0.00	+0.18	+0.18	−0.36
	no question	87.55	0.108	+1.08	+0.90	+0.18	−0.18	+1.26
Qwen2.5-3B, matched training budget
WebQSP	G-Retriever	57.13	0.997	−0.55	−1.41	−0.31	+0.06	−1.60†
	Ours	65.79	0.200	−2.09**	−1.90**	−0.80	−1.35*	−2.09**
	no entity features	58.54	0.180	−2.03**	−1.84*	−0.06	−1.23†	−0.86
	no question	60.14	0.194	−0.86	−2.33**	−2.70***	−0.55	−2.33**
SceneGraphs	G-Retriever	76.05	0.997	−0.30	−1.05†	−0.35	+0.30	−1.15*
	Ours	72.65	0.139	−7.00***	−5.75***	−4.40***	−1.65*	−6.50***
	no entity features	75.95	0.199	−2.35**	−4.60***	−0.20	−0.55	−0.60
	no question	75.40	0.557	−0.75	−1.40*	−0.55	+0.50	−1.90**
ExplaGraphs	G-Retriever	74.37	0.998	−1.62	−2.17	−0.72	−2.71	−4.51*
	Ours	83.94	0.103	+0.90	+0.00	+0.00	+0.36	+0.36
	no entity features	80.14	0.116	+2.17	−3.07	−0.18	−0.72	−1.99
	no question	74.37	0.118	−1.44	−0.54	+0.54	+0.00	−1.44

Cross-fitting: is the token's usefulness Stage-1 memorisation?

Stage 1 partly memorises its training answers, with top-1 about 0.9 on training samples against about 0.5 on validation samples. A reader may therefore ask whether the LLM’s reliance on the token in Section 4.4 is reliance on answers the encoder memorised. We test this by training Stage 1 on one half of the training questions and Stage 2 on the other, splitting SceneGraphs by scene since one scene carries many questions. No test data is involved in either regime: Stage 1 uses training labels only, and on SceneGraphs the training and test sets share no scenes.

Table 72 shows that the reliance is partly memorisation. When the two stages see disjoint halves the interventions still hurt, but less, while accuracy rises towards the baseline trained on the same half. The two regimes are two operating points of the trade-off described in Section 4.4.

Dataset	Stage 2	Accuracy	Baseline	MEAN	SWAP	NODE	EDGE
WebQSP	projector only	63.14	70.52	$-3.44^{***}$	$-5.34^{***}$	$-1.66$	+1.04
WebQSP	projector + prompt	69.53	70.58	$-1.54^{\dagger}$	$-3.19^{**}$	$-3.26^{***}$	$-0.55$
SceneGraphs	projector + prompt, run 1	76.90	78.57	$-1.50^{\dagger}$	$-4.45^{***}$	$-1.05$	$-1.95^{***}$
SceneGraphs	projector + prompt, run 2	77.55	78.57	$-1.10$	$-2.45^{**}$	+1.00	+0.30
ExplaGraphs	projector + prompt	76.35	80.87	$-3.61^{*}$	+1.08	+0.72	+0.18

Rewiring the edges costs the cross-fitted joint model 1.95 points on SceneGraphs in one run and nothing in the other, so structure does not reach the LLM reliably in this regime either.

Initialisations and evaluation noise

Table 73 reports three initialisations of G-Retriever and of the joint model on WebQSP, trained with the cross-fitted recipe on the same half. The spread across initialisations, 1.3 to 1.5 points, is larger than the difference between the two models, which is what licenses the word parity in Section 4.4. Pooling the discordant questions over the three runs, $n = 4{,}884$, leaves our token used and G-Retriever’s not.

Model	Run 1	Run 2	Run 3	Mean ± sd	MEAN	SWAP	NODE
G-Retriever	70.58	67.75	70.02	69.45 ± 1.50	≤ 0.53	≤ 0.53	≤ 0.53
Ours, joint	69.53	66.89	68.43	68.28 ± 1.33	−1.19**	−1.45**	−1.84***

Evaluation noise between a training-script score and a separate scoring run is reported in Appendix § 12.

The verbalised graph is not enough

In this section, we bring some qualitative evidence that LLMs generally struggle to deal with verbalized graphs only. ExplaGraphs is the clean test, because its graph is not retrieved: it is the human-annotated explanation graph of the stance, given in full, so an error cannot be blamed on missing information. We give a frozen Llama-2-7b-chat¹ the complete graph as a list of triples inside G-Retriever’s prompt, zero-shot, and score it with the benchmark’s evaluator on all 554 test questions.

The majority class is 58.8%. The arguments alone give 61.0%, and adding the complete explanation graph gives 64.3%, with 198 of 554 answers still wrong. The graph fixes 68 questions and breaks 50, so verbalising it misleads the model about as often as it helps. Part of this is a bias towards “support”, which accounts for 451 of 554 answers and for 161 of the 198 errors, but not all of it: Figure 7 shows an error in which the graph states the opposition explicitly.

ExplaGraphs #2066. gold: counter — arguments only: counter — arguments + full graph: support

Argument 1: Legalized organ trade would exploit the poor.

Argument 2: Legalized organ trade would aid the poor, by giving them a large sum of money to improve their lives.

Graph: legalized organ trade $\xrightarrow{\text{causes}}$ money for organs $\xrightarrow{\text{capable of}}$ improving finances $\xrightarrow{\text{capable of}}$ improve their lives $\xrightarrow{\text{has context}}$ aid the poor; aid the poor $\xrightarrow{\text{is not a}}$ exploit the poor

Extended related work

Graph soft prompting. A growing line of work connects graph encoders to frozen or adapted LLMs by projecting graph representations into continuous tokens. GraphToken Fatemi et al. [4] learns

ends_mid=1 an encoding function that emits soft tokens for a frozen LLM on synthetic graph-reasoning tasks, while GraphLLM [3] targets graph reasoning with a tuned decoder and GNP [24] adds a cross-modality pooling module for multiple-choice knowledge-graph questions. Among these, GNP and G-Retriever [7] are the only ones that retrieve a question-specific subgraph before encoding; the others operate on the full graph or on synthetic inputs. G-Retriever additionally verbalises the retrieved subgraph in the prompt, creating a redundant configuration that is the focus of our audit. GraphGPT [23] combines a text-grounded structural encoder with a lightweight graph–text alignment projector and dual-stage instruction tuning. These systems share the interface we study: a pooled graph representation, a trained projector, and one or a few tokens prepended to the prompt.

Multi-token and pooling-based methods. Very few works have explored multi-token methods for graph-to-LLM systems. The single-token bottleneck created by mean pooling over a subgraph has only recently been identified as a structural limitation. Grover et al. (2026) argue that aggressive mean pooling compresses entire graph substructures into a single token, creating a severe information bottleneck, and propose multi-token pooling operators to increase the bandwidth of the graph-to-LLM interface. Their work also reports that pooling methods are unstable under soft prompt tuning with a frozen LLM and require LoRA fine-tuning to stabilise. This is complementary to our diagnosis: we study a single-token approach and show that its collapse is associated with both the projector parameterisation and end-to-end answer training.

Graph–language alignment. Alignment-based methods add an auxiliary objective intended to bring graph representations closer to language representations. Align-GRAG [30] is closest to our Stage 1: it contrastively aligns a pooled graph representation with a textual target using KL divergence and contrastive losses. G-Retriever [7] can also be viewed as an implicit alignment method, since its projector is trained to map graph representations into the LLM’s embedding space under the answer loss. Our use of contrastive pretraining is similar to Align-GRAG, but our question is different. Similarity between graph and text spaces does not establish that different graphs produce different tokens after projection, nor that the LLM uses their graph-specific content. We therefore complement representation alignment with measurements of token variation and fixed-weight interventions on the trained model.

Why graph use is not guaranteed. Two recent works already point out our diagnosis from different perspectives. Zhang et al. (2026) prove that the autoregressive language objective of graph-tokenizing LLMs decomposes into a textual dependency term and a graph–text alignment term, and that current methods optimize only the former explicitly. Graph–text alignment is therefore implicit: the training objective does not reward the model for using graph information, and a text-dominant solution can be optimal under the loss. Petkar et al. (2026) show empirically that existing graph–language benchmarks can be matched by graph-only or language-only baselines, and introduce CLEG R to require compositional reasoning over structure and text; even there, raw soft-prompted LLMs remain competitive with GNN-backed models. Together, these results establish that neither the training objective nor the benchmark forces a model to use graph information. Our work is complementary in level rather than in kind: we study the interface itself and show that a graph token can collapse to a near-constant vector even when the encoder representations are distinct, and that preventing collapse does not guarantee that the LLM uses the token’s graph-specific content. This is what motivates the diagnostic protocol we introduce.

Retrieval for graph-grounded generation. G-Retriever [7] formulates subgraph retrieval as a prize-collecting Steiner tree problem [2] and is the reference implementation we study. GRAG [9] retrieves subgraphs for generation with a different scoring rule, emphasising the importance of subgraph structure and topology. GraSP [29] encodes subgraphs into soft prompts using GNNs for multi-hop KBQA and combines this with a two-stage LLM reasoning procedure. GNN-RAG [16] uses a GNN to retrieve reasoning paths that are then verbalised, rather than projecting them as soft tokens. Retrieval quality is orthogonal to our question: our interventions keep the retrieved subgraph fixed and change only the token computed from it.

Parameter-efficient tuning and continuous prompting. Fine-tuning an LLM can be done in several ways. Full-weight fine-tuning updates all parameters. Parameter-efficient methods update a small subset or add new parameters, and include LoRA [8], which learns low-rank updates to existing weights, and continuous prompting methods. Continuous prompting conditions a frozen language model with learned continuous vectors. This family includes prompt tuning [12], which learns input-independent embeddings, and prefix-tuning [14], which learns continuous prefixes made available across the model’s layers. A graph token is a form of continuous prompting, but with a different intended role: it is computed from each input graph and is therefore expected to preserve input-specific information, unlike standard prompt tuning where the prompt is shared across all inputs. Our results connect these settings by showing that Graph Soft prompting, as it is, collapses into a nearly input-independent vector and consequently behaves like a conventional soft prompt. LoRA is used in our work only as a robustness check on the frozen-LLM setting, and the effect is consistent.