WorksComputer Science
Geometry of Confabulation and Refusal in Small Language Models
Abstract
Faced with an unanswerable question, small language models can either refuse or confabulate: confidently produce a wrong answer. Can the hidden state before the first generated token provide a clue? Can we flip confabulations into refusals? I introduce ConfabQA, a 784-item benchmark whose obscure-pre-cutoff category breaks the typical correctness/knowledge-cutoff confound and map the geometry of the hidden state with linear probes, a sparse autoencoder (SAE), and direct optimization of intervention directions. On Qwen3-1.7B and Gemma 2 2B the probe separates upcoming refusals from confabulations well beyond prompt features (+7.4/+16.3 pp); on Llama 3.2 3B the ConfabQA margin is within fold-to-fold spread, but a direct refusal probe on PopQA and TriviaQA recovers +18-25 pp. On Qwen I closely inspect three candidate directions for the refusal vector: probe-recovered, Qwen-Scope SAE feature, and directly optimized. Although the three directions are pairwise 50-81 degrees apart, a push at or below the state's own norm along any of them flips most confabulations into refusal openings dominated by the "As..." family. Forcing the opener without touching the hidden state yields the same one-in-three judged-refusal rate as any push. The opposite push collapses refusals mostly into confabulations, yet leaves originally correct answers intact, so the direction gates the opening speech act rather than the knowledge behind it. Confabulations do not resolve into a concentrated feature in a dictionary or a token signature on Qwen3-1.7B. Scoring every probe on the margin over the strongest of four prompt-feature baselines extracts the hidden state's own contribution. Correctness signal is model-dependent and portable only for Llama 3.2 3B: 81% of the probe's margin over test majority survives transfer between ConfabQA, PopQA, and TriviaQA. Benchmark, judge, and code are released.
Introduction
Confabulation is pattern-based reconstruction the model does not recognize as wrong, a term that expresses the nature of language-model (LM) answers better than “hallucination”, which implies a false perception [5] (Smith et al. 2023). When a small LM (SLM) is asked a factual question about something it does not know, it has a choice to refuse or answer anyway. A fraction of those answers happen to be correct; the rest are confabulations: fluent, confident, and wrong. Can we read that choice off the hidden state, and steer it? Alain and Bengio [1] introduced diagnostic linear classifiers, or probes, to evaluate what a hidden state encodes. Kadavath et al. [10] argued that LLMs “mostly know what they know,” and a correctness-probing literature followed [3, 17, 18, 11], fitting probes to predict whether an answer will be right and reporting accuracies in the 80–90% range. High probe accuracy is not by itself evidence that the hidden state knows anything. In adjacent settings, recent work traces above-chance accuracy to the prompt rather than the model’s internal state: introspection (Singh et al. 2026) and reasoning mode (Sahoo et al. 2026). A confabulation probe faces the same question. Does it read the model’s state, or the question?
Controls that separate hidden-state signal from what the prompt alone predicts remain rare in the correctness-probing literature, where time-sensitive question sets routinely conflate “post-cutoff” with “model doesn’t know.” Refusal steering by a single direction has been shown for safety refusal [2]. Wollschläger et al. (2025) showed that safety refusal can be steered by a set of directions. For knowledge-based refusals, Ferrando et al. [6] found that features corresponding to known entities suppress refusals on Gemma 2. Lindsey et al. [15] confirmed this for Haiku 3.5 and further investigated the multi-step mechanics triggered by these features. I ask whether a single direction can convert most confabulations into refusals for facts the model did not learn.
I map the geometry of this decision with the probes, a sparse autoencoder, and direct optimization of intervention direction. To test whether steering along the recovered directions turns confabulations into refusals I introduce ConfabQA, a probing benchmark of 4 domains 3 categories (well-known pre-cutoff, obscure pre-cutoff, post-cutoff). The obscure category, items whose answers were likely in the training data but which the model is expected to fail, populates the cell standard pre-vs-post-cutoff designs leave empty, breaking the cutoff/correctness confound. ConfabQA, plus 800-item samples of PopQA (Mallen et al. [16]) and TriviaQA (Joshi et al. [9]), run through three instruction-tuned models in the 1.7–3B range (Qwen3-1.7B, Gemma 2 2B, Llama 3.2 3B) with Qwen3-4B as a within-family scaling control. Every probe is scored against both the majority and stricter question-text baselines to decide whether the hidden state contributed anything.
Findings
Refusal vs. confabulation is a model-internal feature: On the wrong+refusal subset of ConfabQA the late-layer probe (Figure 1) beats the strongest prompt baseline on Qwen3-1.7B (+7.4 pp) and Gemma 2 2B (+16.3 pp); on Llama 3.2 3B the +2.2 pp margin is within fold-to-fold spread. On Qwen3 and Gemma a single late-layer linear direction is causally sufficient for the first-token refusal opening. A one-shot hidden-state intervention drives the first-token refusal-opener rate to 100% on Qwen3 and Gemma (§ 6.2 and § 7.2), though on Qwen3 only about a third of these openings complete as judged refusals, the rest reverting to content (Gemma completes 87% at a super-norm dose and 30–40% at sub-norm); closing that gap is left to future work. This replicates Arditi et al.’s [2] single-direction finding in the epistemic-abstention rather than safety-refusal regime. The SAE resolves the direction into a deferential-register ensemble (canonical refusal-opener plus apology, officialese, and hedging registers; § 6.3). The opener feature alone flips 30/30 wrong-item next-token argmaxes to refusal openers. No concentrated dictionary feature or token signature is found for confabulation behaving as the default attempt pathway missing a fact (§ 6.3).

Figure 1. Two recoveries of “the” refusal direction (layer-28 refusal+wrong states, ; stars: class centroids; dashed curves: per-class covariance ellipses). (a) Ambient view, top two principal components: the class-mean difference (green) connects the centroids and lies almost entirely in this top-variance plane; the probe direction (purple, in-plane projection) does not. (b) The plane spanned by the two directions, in which both lie fully: the angle between them is . The wrong class’s covariance ellipse is elongated along the same oblique axis the centroid connector rides, while the probe direction is the axis whose marginal separates the classes item by item (score histograms in Figure 11). The dotted vertical line is the probe’s decision boundary, exact in this panel because the hyperplane normal is the probe direction itself.
Llama 3.2 3B carries a substantially larger and more general correctness signal than Qwen3-1.7B or Gemma 2 2B. On balanced PopQA / TriviaQA subsamples, Llama’s probe beats the prompt baseline by +21–+25 pp, 5.7× and 2.2× Qwen’s margins on the same benchmarks. Three alternatives do not explain it: parameter count (the Qwen3-4B scaling control moves the margin by less than 2 pp; § 8.1), dataset-specific overfit (Llama keeps ≈ 81% of its margin after cross-dataset transfer without refit, where Qwen’s and Gemma’s transfer collapses;
Section 8.3), refusal detection ( of the margin survives dropping refusals in evaluation; Section 8.2). Gemma is behaviorally intermediate, yet its correctness probe is the most dataset-local of the three. Llama 3 post-training includes a factuality stage that generates refusals for questions the model answers confidently and incorrectly under knowledge probing [8 §4.3.6]. Qwen3 and Gemma reports describe no analogue.
Theoretical background
All three SLMs in the study are decoder-only transformers with causal attention. The final-layer state at the last position is projected through the LM head onto the vocabulary , and generation feeds each emitted token back as input (Figure 8a, Appendix § 13.2.§ 6). The probes read the prefill pass (Figure 8b): the single forward pass over the prompt alone, before any token is generated. Qwen3-1.7B instantiates this with blocks, hidden size , and tokens with tied input/output embeddings.

Figure 8. Token flow in the two forward-pass regimes. (a) Conventional generation runs one forward pass per generated token, feeding each emitted token back into the input. (b) The prefill pass stops just before the first generated token; the hidden state at that moment, , is saved at every layer and is what the probes read. Generation starts from exactly this state, so it is the model’s representation immediately before it commits to an answer; stacked over questions it forms the probe input tensor . The dimensions are specific to Qwen3-1.7B; the other subject models differ in both depth and width.
Confidence metrics: log-probability and entropy
All generation is greedy deterministic, so each emitted token is the mode at that position. Two metrics track confidence. is the mean log-probability of the emitted (argmax) tokens ( corresponds to mean geometric probability per token). is the mean Shannon entropy of the next-token distribution. Formal definitions (Eq. (2)), the worst-step diagnostic , and the empirical anti-correlation of the two are in Appendix § 13.2.§ 6.
The prefill hidden state
The prefill pass over the prompt produces the full hidden-state tensor. I pick off the hidden state at every layer at the last prompt position, , of shape per question. Nothing has been generated yet, so this is the model’s representation immediately before the first generated token (Figure 8b). It summarizes the whole prompt, sits one linear projection from the first answer token’s logits (§ 4), and avoids the contamination of probing generated tokens’ states (expanded rationale in Appendix § 13.2.§ 6).
Linear probe and its pipeline
Following Alain and Bengio [1], I fit at each layer a binary linear classifier on . Accuracy substantially above the appropriate baseline implies a hyperplane separates the property at depth . Whether the model uses that direction is a separate question. The pipeline is StandardScaler -> PCA(n_components=16) -> LogisticRegression(max_iter=2000, C=1.0) under 5-fold stratified CV with random_state=0, reported as mean and per-fold std of accuracy. Scaler and PCA are fit only on the training fold, so no test-fold leakage occurs (overfit rationale and schematic in Appendix § 13.2.§ 6/§ 13.2.§ 7, Figure 7). Appendix § 13.4.§ 1 sweeps over to show the reported numbers are not specific to 16.

Figure 7. Per-layer linear-probe pipeline. At each layer , the matrix of captured hidden states is passed through StandardScaler PCA(16) logistic regression and evaluated with stratified 5-fold CV, separately for each binary target.
The two confounds and what counts as evidence
Confound 1 (cutoff/correctness). A benchmark of easy pre-cutoff plus post-cutoff questions makes “correct” and “pre-cutoff” nearly the same label, so a correctness classifier is in effect a cutoff classifier (“this question mentions 2025”). ConfabQA disconfounds with obscure pre-cutoff questions, facts likely in the training data that the model still gets wrong, so pre-cutoff no longer implies correct (keeping only pre-cutoff questions would also remove the refusals; § 5.1). The disconfound test fits the correctness probe restricted to pre-cutoff items, where accuracy above the pre-cutoff majority cannot be explained by the cutoff variable.
Confound 2 (hidden state/prompt features). A probe that beats its majority baseline might still be reading question-text features any classifier could extract: year, domain, length, keywords. The prompt-feature baseline is a logistic regression on prompt-side features alone (roster in § 4, specification in Appendix § 13.2.§ 7, schematic in Figure 9, Appendix § 13.2.§ 6). A hidden-state probe is evidence of model-internal information only to the extent it beats this baseline outside per-fold noise (, in pp, reported throughout §§5–8). The probing literature most often omits this control. It distinguishes “the hidden state encodes self-knowledge” from “the hidden state encodes the prompt, which encodes the answer.”

Figure 9. Baselines, fit on the same labels with no access to the hidden state: the majority class, and a prompt-text classifier, TF-IDF (term-frequency / inverse-document-frequency bag-of-words) plus logistic regression, with engineered features and domain/category dummies. A probe is evidence of model-internal information only insofar as it beats both; (pp), reported throughout §§5–8.
Sourcing and curation of the ConfabQA question set
ConfabQA comprises 784 items in a design: 4 domains (science, history, culture, cinema) crossed with 3 categories (well-known pre-cutoff, obscure pre-cutoff, post-cutoff). Across domains the categories hold 143 well-known, 153 obscure, and 488 post-cutoff items (per-cell counts in Table 3, Appendix § 13.1). I generate items from per-domain templated source files. Each carries a provenance URL for its gold answer and a validation-status field. The 384 items marked verified were marked from source-file provenance only, with no independent check of the gold answer. The other 400 are all post-cutoff items, 330 of them from four dense post-cutoff files written to elicit refusals, and remain unverified. Nearly every post-cutoff question names a year, a cue the prompt-feature baselines include. A wrong gold answer can move an item between correct and wrong but not into or out of refusal. The web-verified regrade of all 784 labels (§ 4) checked the golds only incidentally and flagged three premise errors (Appendix § 13.2.§ 3). The paper analyzes the third of three validation iterations, ConfabQA (). The full design is in data/QUESTIONS_v1_CARD.md, and pipeline reproducibility and a sample source entry are in Appendix § 13.2.§ 2 and B.5. A fifth domain (sports) produced 0/26 correct, leaving no positive examples for the within-pre-cutoff probe, so I exclude it (Appendix § 13.1.§ 1).
| domain | well-known | obscure | post-cutoff | total |
| science | 42 | 45 | 119 | 206 |
| history | 33 | 37 | 127 | 197 |
| culture | 34 | 34 | 121 | 189 |
| cinema | 34 | 37 | 121 | 192 |
| total | 143 | 153 | 488 | 784 |
Table 3. ConfabQA cell counts: 4 domains × 3 categories, .
The third (obscure) category populates the cell “answer-was-in-training-data but model-is-expected-to-fail,” which the standard two-cell design leaves empty. A typical obscure item asks who won the Best Actor Academy Award at the 1968 ceremony (gold: Rod Steiger, on the public record). Qwen3-1.7B confidently answers with the wrong actor.
Methods
Subject model and generation. The subject is Qwen3-1.7B (Qwen/Qwen3-1.7B; (Qwen Team 2025)), a 1.72B-parameter text-only instruction-tuned decoder-only transformer (Apache 2.0), with 28 blocks and hidden dimension 2048. I run it in bfloat16 on Apple Silicon MPS in 16 GB unified memory. Decoding is greedy throughout (do_sample=False, max_new_tokens=64, enable_thinking=False, suppressing Qwen3’s <think> trace), so all generations are deterministic conditional on the prompt.
Hidden-state capture. Each question persists one prefill tensor from model.generate(...). The cached “layer 28” entry is the final normed state, while the § 6.2/§ 6.3.1 hooks and the SAE operate on the pre-norm residual (Appendix § 13.3).
Same-model judge. Qwen3-1.7B labels each generated answer itself, called separately with a structured prompt (judge.py) that asks for one of correct, refusal, or wrong (decision rule, worked examples, and parser in Appendix § 13.2.§ 1). This judge replaces the brittle substring-match grader used in earlier work.
Three LLM annotators apply the same three-way rule (Table 9, Appendix § 13.2.§ 3): the Qwen3-1.7B judge on all 784 items, Claude Opus 4.7 on a stratified 30-item sample, and Google Deep Research (DR; Gemini 3.1 Pro with web search) on all 784. DR agrees with the Qwen judge at , “substantial agreement” (Landis and Koch [12]), and is systematically stricter. Claude Opus reports on 30/30, which bounds the Qwen judge’s disagreement with a second grader at at 95% confidence by the rule of three. Under the DR labels both probe margins over the TF-IDF baseline stay positive, and a blind Claude-family regrade of the Section 6 intervened generations agrees with the self-judge without inflating flip rates.
| pair | items | agree | rate | Cohen’s |
| DR vs. Qwen-judge (full ConfabQA) | 784 | 682/784 | 87.0% | 0.780 |
| Claude vs. Qwen-judge (stratified 30-item sample) | 30 | 30/30 | 100% | 1.000 |
Table 9. Judge agreement on ConfabQA across three annotators.
Hidden-state probe pipeline. For each binary target and layer, I fit the § 2.3 pipeline on the last-prompt-token states at that depth. I report mean and std of fold accuracy plus a “layer range”, the set of layers within of the peak.
Prompt-feature baseline pipeline. For each target, I also fit baselines with no access to the hidden state, reporting 5-fold CV accuracy on the same folds (random_state=0). Four baselines of increasing leniency: TF-IDF (bag-of-tokens logistic regression on the raw question text); text-only (engineered numeric features); text + domain (adds the domain one-hot); and text + domain + category (adds the annotator’s expected-difficulty dummy, the least conservative baseline). For the cutoff target the category dummy is excluded from the strongest-baseline comparison, since it would be the label itself. Full feature lists and hyperparameters are in Appendix § 13.2.§ 7. The prompt-feature comparison is the hidden-state probe vs. the strongest of the four on each target: the column of Table 7 in Appendix § 13.1.§ 5.
Table 7: Hidden-state probe vs. the four prompt-feature baselines; = probe peak − strongest (bolded) baseline.
| target | majority | TF-IDF | text-only | +domain | +category | hidden state | |
| correct | 0.700 | 0.767 | 0.773 | 0.779 | 0.800 | 0.824 | +2.4 |
| cutoff | 0.622 | 0.940 | 0.952 | 0.952 | 0.982 | +3.0 | |
| refusal_vs_wrong | 0.732 | 0.820 | 0.803 | 0.805 | 0.800 | 0.894 | +7.4 |
| refusal_vs_wrong_within_post | 0.626 | 0.774 | 0.741 | 0.730 | 0.730 | 0.870 | +9.6 |
| correct_within_pre | 0.527 | 0.797 | 0.811 | 0.811 | 0.818 | 0.848 | +3.0 |
| correct_within_obscure | 0.667 | 0.752 | 0.830 | 0.811 | 0.811 | 0.805 | −2.5 |
Table 7.
Probe targets. correct (all 784 items); cutoff (post vs. pre, all 784); refusal_vs_wrong (the {refusal, wrong} subset, ); refusal_vs_wrong_within_post (); all 147 refusals post-cutoff, so this conditions out the cutoff covariate); correct_within_pre (n=296, the cutoff disconfound test); and correct_within_obscure (n=153, the popularity disconfound test).
Bootstrap protocol (Sections 8.1, 8.2). For each (dataset, model, target) cell, I draw random 50/50 class-balanced subsamples without replacement. Each refits the full probe pipeline (peak over all 29 layers) plus the four prompt baselines on the same folds, and I report with the range of the values, from minimum to maximum. At the 2.5th/97.5th-percentile rule selects exactly these two values, so I call it a range, not a 95% CI. It measures subsample-to-subsample spread, not item-sampling uncertainty (class-size rule, interval construction, compute bounds, and an item-level bootstrap in Appendix § 13.5). When reading small margins, note that the probe side maximizes over 29 correlated layers and the baseline side over four, so under a no-signal null the expected is slightly positive; margins within a couple of points of zero carry this selection bias.
External-dataset evaluation (Section 8.1). PopQA [16] and TriviaQA [9] run end-to-end through the same generation pipeline and three-way Qwen3-1.7B judge, per seed (splits, strata, seeds in Appendix § 13.5). Three seeds per (subject model, dataset) widen the pool to ; Llama runs are single-seed (). The judge remains Qwen3-1.7B in all runs (§ 10 records the Qwen-judge-on-Llama-outputs caveat).
Probing Qwen3-1.7B
The Confabulation Atlas (Figure 5, Appendix § 13.1.§ 4) situates the numbers geometrically: a supervised 2-D projection of the layer-18 hidden states of all 784 questions. An unsupervised PCA(2) control accompanies it, in which PC1+PC2 alone already support an 86.9% refusal-vs-wrong probe (vs. 73.2% majority), so the refusal separation is not an artifact of supervised projection.

Figure 5. Confabulation Atlas (layer 18 hidden state; § 5). Each point is one question: color = judge label (green correct, blue refusal, red wrong), shape = category (circle well-known, diamond obscure, triangle post-cutoff), size = mean per-token log-probability. (a) Supervised projection. X: signed distance to the correctness hyperplane (all , 5-fold CV acc 82.4%). Y: signed distance to the refusal-vs-correct hyperplane (fit on the refusal+correct subset, CV acc 96.1%, orthogonalized against X for plotting; raw angle between the normals in 16-d PCA space 55.7°). Wrong items, never seen during the Y fit, fall between the refusal pole (top) and the correct pole (bottom). (b) Unsupervised control. PCA(2) of the same states: even without supervision, PC1 and PC2 already carry visible refusal-vs-rest structure. Whether the correctness separation in (a) reflects self-knowledge or prompt-readable features is the subject of § 5.4.
Accuracy by stratum
Of 784 items, 235 (30.0%) are graded correct, 147 (18.8%) refusal, and 402 (51.3%) wrong (breakdowns in Appendix § 13.1, Table 2). The cutoff manipulation works as designed: pre-cutoff accuracy is 47.3% (140/296) against 19.5% (95/488) post-cutoff. In the obscure-pre-cutoff cell, the model fails on of items whose answers likely appeared in its training data, the population the cutoff disconfound needs. All 147 refusals fall in the post-cutoff cell, so refusal is triggered by the knowledge-cutoff cue rather than intrinsic uncertainty. The refusal_vs_wrong_within_post probe of § 4 therefore compares refusals only with post-cutoff wrong answers, so the cutoff cue alone cannot separate the classes.
| category | n | correct | refusal | wrong | accuracy | mean log-probability |
| well-known (pre-cutoff) | 143 | 89 | 0 | 54 | 62.2% | −0.120 |
| obscure (pre-cutoff) | 153 | 51 | 0 | 102 | 33.3% | −0.149 |
| post-cutoff | 488 | 95 | 147 | 246 | 19.5% | −0.145 |
| total | 784 | 235 | 147 | 402 | 30.0% | −0.141 |
Table 2. Judge-label counts, accuracy, and mean per-token log-probability by category.
Text-level confidence, confident-confabulation case studies, and hidden-state geometry
Mean per-token log-probability separates correct from incorrect answers only in expectation (Figure 4, Appendix § 13.1.§ 2): vs. ( vs. geometric-mean per-token probability), with heavy overlap in . It is weakly informative, not a usable predictor. Table 6 (Appendix § 13.1.§ 3) lists the five highest- items per failure label. The five confabulations all sit at geometric-mean per-token probability, and refusals are paradoxically high-confidence text outputs (fluent, assertive statements of ignorance). So a binary substring grader lumps both failure modes into one incorrect bucket, and text-level confidence cannot separate them afterward; separating them is what the three-way judge is for (gold-side validation case study in Appendix § 13.2.§ 2).

Figure 4. Text-level confidence. (a) Histogram of per-item mean per-token log-probability under greedy decoding, split by judge correctness label: the two distributions overlap substantially, their means differing by ≈ 0.05 nats. (b) Per-item mean log-probability vs. mean entropy, colored by cutoff class, marker by correctness. Pre/post-cutoff items occupy distinguishable regions but with heavy overlap, especially among confident wrongs.
Table 6: The five highest-confidence confabulations (top block) and refusals (bottom block; all post-cutoff). Questions and answers are excerpts from the released response files.
| id | question (abridged) | gold | model said (excerpt) | |
| cin_pc_41 | Ridley Scott’s November 2024 sequel to Gladiator? | Gladiator II | “There is no film directed by Ridley Scott that is a sequel to Gladiator. . . ” | −0.047 |
| cul_ob_12 | Which 1971 Walker Percy novel won the National Book Award? | Love in the Ruins | “. . . was ‘The Crying of Lot 49’ ” | −0.049 |
| cin_ob_24 | Who directed ‘Three Colors: Blue’ (1993)? | Krzysztof Kieslowski | “. . . was directed by Jean-Pierre Jeunet” | −0.049 |
| his_ob_35 | Who was the 27th US President, 1909–1913? | William Howard Taft | “. . . was Theodore Roosevelt” | −0.050 |
| his_ob_30 | Who was Chancellor of West Germany, 1969–1974? | Willy Brandt | “. . . was Konrad Adenauer” | −0.052 |
| wrong | ||||
| his_pc_38 | Treasury Secretary in the second Trump administration, January 2025? | Scott Bessent | “As of January 2025, there is no Secretary of the Treasury. . . ” | −0.049 |
| cin_pc_14 | Who starred in Bong Joon-ho’s 2025 film ‘Mickey 17’? | Robert Pattinson | “As of my knowledge cutoff in 2023, there is no 2025 film titled Mickey 17. . . ” | −0.055 |
| cul_pc_20 | Which Taylor Swift album was released on April 19, 2024? | The Tortured Poets Department | “As of my knowledge cutoff in 2024, there is no official announcement. . . ” | −0.060 |
| cin_pc_114 | 2025 Best Supporting Actor Academy Award, for ‘A Real Pain’? | Kieran Culkin | “As of the 2025 Academy Awards, there is no official announcement. . . ” | −0.064 |
| his_pc_33 | US Attorney General in the second Trump administration, February 2025? | Pam Bondi | “As of February 2025, there is no United States Attorney General. . . ” | −0.067 |
| refusal |
Table 6.
Unsupervised PCA(2) of the raw hidden state (Figure 6, Appendix § 13.1.§ 4) shows the classes mixing heavily at layer 14. By layer 28 the refusals pull into a distinct cluster with no supervision, a tight group of confident confabulations separating out on its own. That late-layer refusal cluster is the geometry the refusal-vs-wrong probe reads.

Figure 6. Hidden-state geometry, unsupervised (§ 5.2). PCA(2) of the last-prompt-token hidden state at a mid-network layer (left) and the final layer (right), . Color is the judge label (green correct, blue refusal, red wrong), marker shape is cutoff class (circle pre-cutoff, triangle post-cutoff), marker size is the mean per-token log-probability. At layer 14 the classes mix heavily; by layer 28 the refusals (blue, necessarily post-cutoff) pull into a distinct cluster with no supervision, the correct answers occupy the opposite side, and a tight group of large red markers (the confident confabulations of § 5.2) separates out on its own.
Per-layer linear probes against majority baselines
Figure 2 plots the per-layer probe accuracy, and Table 8 (Appendix § 13.1.§ 5) tabulates peaks, layer ranges, and margins over the majority baselines for all five targets, with the 1 layer-range caveat (the within-obscure range spans the whole network, so its per-layer peak is not a per-depth claim). Majority-baseline margins alone overstate the correctness signal; § 5.4 scores each against prompt features. Imbalance-aware metrics at the refusal-vs-wrong peak (balanced accuracy 0.874, ROC AUC 0.944; confusion matrix in Appendix § 13.4.§ 6) show the probe is not defaulting to “wrong”.

Figure 2. Per-layer probe accuracy for all five targets on Qwen3-1.7B (ConfabQA). Solid lines: probe accuracy per layer; dotted horizontal lines: per-target majority baselines; dots: peak layers (values in Table 8, Appendix A.5). Cutoff saturates early and stays high; refusal-vs-wrong rises monotonically and peaks at the deepest block; correctness and correct-within-pre peak mid-network, with within-pre’s margin over its pre-cutoff majority largely vanishing against the prompt-feature baseline of Section 5.4; correct-within-obscure has a 1- layer range spanning nearly the whole network.
Table 8: Probe peaks vs. majority baselines (Qwen3-1.7B, ConfabQA).
| target | n | best layer | layer range | peak acc | majority | margin |
| correct (all items) | 784 | 18 | 10–28 | 0.824 ± 0.031 | 0.700 | +12.4 |
| cutoff (all items) | 784 | 13 | 10–20 | 0.982 ± 0.005 | 0.622 | +36.0 |
| refusal_vs_wrong | 549 | 28 | 16–28 | 0.894 ± 0.019 | 0.732 | +16.2 |
| refusal_vs_wrong_within_post | 393 | 28 | 20–28 | 0.870 ± 0.022 | 0.626 | +24.4 |
| correct_within_pre | 296 | 18 | 9–28 | 0.848 ± 0.038 | 0.527 | +32.1 |
| correct_within_obscure | 153 | 7 | 1–28 | 0.805 ± 0.063 | 0.667 | +13.8 |
Table 8.
Value of the hidden state compared to prompt features
Table 7 scores the hidden-state probe against the four Section 4 baselines on the same folds. The +category baseline is a difficulty oracle: category explains most of the raw correctness variance by construction (the 62.2%/33.3%/19.5% base rates of Table 2), so scoring against it asks for item-level signal beyond the coarse difficulty bucket. Within single-category strata the dummy is constant (+category collapses to +domain), and for cutoff the dummy is the label and is excluded (Section 4).
Full per-target margins over the strongest prompt baseline are in Table 7 (Appendix A.5).
(All baselines are 5-fold CV accuracy with random_state=0. †: for the cutoff target the category dummy is the label; the tautological 1.000 is shown for completeness but excluded from the strongest-baseline comparison.)
Table 7 shows three results. (1) Correctness gains little from the hidden state: ≈ 3 pp or less over the strongest baseline (+2.4 pp all items, +3.0 pp within-pre, −2.5 pp within-obscure), within per-fold noise (0.031–0.063). The Section 8.1 bootstrap confirms this. The all-items margin holds, and the disconfounded within-stratum margins are consistent with zero. (2) Cutoff is easy to predict either way: text features alone reach 94.0–95.2% (the year mentioned is a near-perfect predictor), and the probe at 98.2% adds +3.0 pp (a manipulation check on the cutoff cue, not evidence of internals). (3) Refusal-vs-confabulation substantially beats prompt features: +7.4 pp over TF-IDF overall, +9.6 pp within post-cutoff where all refusals occur, gaps ≈ 4x the per-fold std (0.019, 0.022). Refusal-vs-confabulation is a decision the model itself makes, and the late hidden state is the computation producing it (Section 2.2), so an encoding must exist. Correctness is a relation between output and world that the model need not represent. Section 6 asks what form the refusal signal takes.
From probe to tokens: the Qwen3-1.7B refusal direction
Section 5 shows a refusal signal beyond prompt features. Here I read the direction through the model’s output head (§6.1), intervene on real hidden states (§6.2), and decompose it into sparse dictionary features to locate the causal component (§6.3).
Probe-direction atlas: the refusal signal in token space
The refusal-vs-wrong probe at layer 28 carries the largest baseline-adjusted margin of Table 7. Its weight vector lifts back to raw hidden-state units, (PCA basis , scaler stds , sign chosen so the refusal end is positive), and passes through the model’s final RMSNorm and tied LM head: for . RMSNorm is scale-invariant, so the depend only on the direction. This is the same linear readout the head applies to any real state, so moving a state along the direction adds approximately these scores to its next-token logits, a prediction Section 6.2 confirms, with the lens scores as the asymptote of the dose-response (calibration in Appendix C). This is epistemic abstention expressed as a hedging opener, distinct from the safety-refusal direction of Arditi et al. [2]; whether the two coincide is untested here.
Top tokens (full subset, ). The ten highest-scoring tokens are all tokenizations of “as”, and the first non-“as” token, at rank 11, is there, the second-most-common refusal opening (scores in Appendix D.4). The refusals near-universally open with “As of my knowledge cutoff in 2023. . . ”, so the probe reads the opener the model is about to emit. An opener is a propensity, not a guarantee, which is why the causal test scores the strict judge label alongside the first-token rate. The confabulation pole shows no thematic structure, and the layer-18 correctness direction (all items) shows no comparable lens structure either (token detail in Appendix D.4). § 6.2 tests its causal counterpart.
The within-post recovery (which contains every refusal) yields the same vocabulary in sharper form (Appendix D.4). With the within-post probe’s +9.6 pp margin (§ 5.4), the late-layer state carries a discrete pragmatic signal (“I am about to hedge”) not extractable from the question text (score histograms in Figure 11, Appendix D.4). A recovery-method check (Appendix D.4, Figure 1) shows the raw class-mean difference used by safety-refusal work is not this direction: it absorbs topical and temporal-currency content where the probe concentrates the register-and-opener component. The direction’s token-level signature is the refusal-opening vocabulary: a late-layer commitment to the refusal opener, rather than a general “self-knowledge” representation.

Figure 11. Score histogram: the signed projection of each item’s layer-28 hidden state onto the recovered refusal direction, on the within-post subset (). Refusals concentrate at large positive projections; wrongs near zero or negative.
Causal validation by direct intervention
The probe direction of Section 6.1 is correlational. To test whether it is also causal, a forward hook adds to the last prompt token’s post-block-27 residual during prefill on a stratified subset (30 wrong post-cutoff + 30 refusal items), where is the unit-normalized within-post recovery of Section 6.1 (hook detail in Appendix C). Generation passes through unmodified, one-shot at the moment of commitment. The perturbed residual is not the cached normed state the probes read (Section 4). A coordinate-dissociation check refitting the probe natively on pre-norm residuals finds equal discriminative accuracy but half the causal effect (Appendix C), so the normed representation, one linear map from the output head, is where the potent direction lives. RMSNorm sets the natural scale, and working doses run from a third of the pushed state’s own norm to about twice it (dose ratios and calibration scan in Appendix C). Sweep: on the 60 items, greedy, re-judged three-way. Outcomes are the strict judge-label refusal rate and a first-token refusal-opener rate over the Section 6.1 opener set (listed in Appendix C). The opener rate is deliberately a first-token proxy (an “As. . . ” opening does not commit the model to refusing), which is why both measures are reported and diverge.
Table 1 is the steering matrix. Pushed toward refusal, wrong items open with refusal vocabulary almost without exception from a 0.9× dose, yet only a third of full generations end as judged refusals; the rest continue into an answer. Pushed the other way, refusing items turn into attempts that are overwhelmingly confabulations. Correct items are untouched by the pull and largely resist the push, about half still delivering the fact through a forced refusal opening. Suppressing the refusal decision does not surface latent knowledge, and pushing toward refusal does not reliably erase it: the direction gates the opening, and whether the fact is present decides what follows. The matrix is a property of the refusal axis, not of one recovery. Rerun with the SAE-2191 and directly optimized directions, it reproduces the same plateaus row for row (parity block, Table 10, Appendix C).
| original label | metric | −1.2× | −0.3× | 0 | +0.3× | +0.9× | +1.8× |
| WRONG | opener rate | 0% | 3% | 10% | 50% | 97% | 100% |
| WRONG | judged REFUSAL | 10% | 0% | 0% | 13% | 30% | 30% |
| REFUSAL | opener rate | 0% | 23% | 100% | 100% | 100% | 100% |
| REFUSAL | judged REFUSAL | 17% | 40% | 100% | 100% | 100% | 100% |
| REFUSAL | judged WRONG | 77% | 53% | 0% | 0% | 0% | 0% |
| CORRECT | opener rate | 0% | 0% | 0% | 10% | 63% | 100% |
| CORRECT | judged CORRECT | 93% | 100% | 100% | 87% | 57% | 53% |
| CORRECT | judged WRONG | 7% | 0% | 0% | 7% | 30% | 33% |
Table 1. The steering matrix: outcome rates for one-shot pushes along the refusal direction, doses as fractions of the mean pre-norm state norm (; sweep ); per subset. Judged CORRECT stays 0%–10% on the wrong subset and judged REFUSAL on the correct subset throughout, for every direction; the small judged-REFUSAL rates at negative doses on wrong items (, 0% opener rate) are sign-inconsistent magnitude artifacts, not directional effects.
| original label | metric | 0 | |||||
| WRONG | judged REFUSAL (2191) | 7% | 3% | 0% | 30% | 27% | 23% |
| judged REFUSAL (optim.) | 13% | 13% | 0% | 30% | 30% | 30% | |
| REFUSAL | judged WRONG (2191) | 67% | 70% | 0% | 0% | 0% | 23% |
| judged WRONG (optim.) | 77% | 77% | 0% | 0% | 0% | 0% | |
| CORRECT | judged CORRECT (2191) | 97% | 100% | 100% | 60% | 50% | 77% |
| judged CORRECT (optim.) | 90% | 93% | 100% | 50% | 50% | 50% |
Table 10. Steering-matrix parity: the Table 1 protocol rerun with the SAE-2191 and directly optimized directions on the same items (story metric per subset; doses as fractions of the mean pre-norm state norm).
The direction is a causal handle on the refusal-opening readout: it localizes where the commitment is written, not the upstream circuitry computing it (earlier-layer pushes are future work). The full outcome is co-determined by autoregressive dynamics the one-shot push does not touch: a model can open “As of 2024…” and still commit to a fact, hence the 30% vs. 100% gap on wrong items. The reverse direction is stronger because the natural refusal trajectory is itself committed. A prefix-forcing control (Appendix § 13.3) reproduces the judge-label outcome without touching the hidden state: the direction selects the first token, and everything after is ordinary autoregressive dynamics (sample generations in Appendix § 13.3). Qi et al. [19] find safety-alignment changes concentrated in the first output tokens. Here the abstention decision is likewise written, and steerable, at token one.
Pushing the correctness direction the same way finds no comparable causal handle: effects are magnitude-driven and sign-symmetric, the few flips concentrate on a recurring handful of marginal items, with no sign that the direction injects knowledge. This correctness-direction null (protocol, rates, flip forensics, and the tension with truthfulness-steering reports) is detailed in Appendix § 13.3. The asymmetry with Table 1 matches the two behaviors: refusing is an opening the model can always produce, so a single direction can carry it; answering correctly requires the specific fact, which a direction can at most dislodge from a marginal default.
SAE feature decomposition of the refusal direction
A sparse autoencoder (SAE), trained on a far larger general-text corpus run through the model (features reflect what matters to the model, not what varies in one dataset), decomposes how the decision is composed. For a hidden state , the TopK SAE (Gao et al. [7]) computes
Each state is represented by 50 of 32,768 candidate features. Each feature owns a fixed decoder direction in the same space as and , and remains a descriptive coordinate unless it passes a causal test, as 2191 does in § 6.3.1. The SAE is Qwen-Scope for Qwen3-1.7B-Base (qwen-scope-3-1.7b-base-w32k-l50; Qwen Team 2026; the Qwen analogue of Gemma Scope; Lieberum et al. [14]), applied at its declared hook, the post-block-27 residual (§ 4). It was trained on the base model while the subject is Instruct (reconstruction checks in Appendix § 13.4.§ 2; PCA-coverage and blend tests in Appendix § 13.4.§ 3). Three correlational views rank which features compose the direction: direct encoding, decoder alignment, and the empirical refusal-vs.-wrong activation differential ( vs. ; definitions in Appendix § 13.4.§ 7). Feature 2191 is the only one in the top eight of all three views, and the apology feature 14034 appears in all three top-twenty lists.
In the SAE’s own geometry the refusal direction decomposes not into one opener plus content detectors but into a set of register features (feature card in Figure 15, Appendix § 13.4.§ 7, with per-feature hit rates and vocabularies): a moment hedge (4314, the strongest discriminator), deferential officialese (17077), polite modality (14361), the canonical opener itself (2191), a second Please feature (16612), a first-person feature (8875), and the apology register (14034), in that order of refusal-vs.-wrong selectivity. Feature 2191 is decoder lens the §6.1 refusal vocabulary (as, 作为, /tas, As, as, As). It fires on 100% of refusals (mean activation 268) and 49% of wrongs, and it is the feature tested causally in § 6.3.1. Feature 14034 (Sorry/Oops register) fires on 97% of refusals, yet its apology vocabulary never surfaces in the outputs, and pushing its decoder direction at doses that saturate 2191 produces zero Sorry/Oops openers (Appendix § 13.3). 17077 behaves the same way: register markers, not emitted tokens. The “refusal direction” is a set of register features of which exactly one (2191) writes the opener token that carries the causal effect. The reading beyond 2191 is correlational (only the opener member is causally tested), and under a dictionary reconstructing these states at EV 0.50 feature identities carry splitting and absorption risk (Chanin et al. [4]). How an h-space push maps into a-space is analyzed in Appendix § 13.4.§ 4.

Figure 15. Four members of the refusal-register ensemble at the post-block-27 residual. Left column: the canonical refusal-opener (the § 6.3.1 causal anchor) and the Sorry/Oops register. Right column: deferential officialese and the moment hedge. Hit rates and max-activating prompts are computed at the SAE’s declared hook.
Unifying feature not found for the wrong answers. The same search on the wrong side over all 32,768 features finds nothing like 2191: the strongest wrong-side features are topic, phrasing, and attempt-formatting cues, not commitment markers (ranked features and hit rates in Appendix § 13.4.§ 7). The dictionary contains a refusal register but no confabulation marker. The claim is scoped to what was searched (one dictionary at one hook, per-item EV 0.50, so absence of a dictionary feature is not absence of signal). Confabulation behaves as the default attempt pathway missing a fact, not a marked state. This is the SAE-space counterpart of the § 5.4 prompt-feature result. This inverts the default reported for Gemma 2 (Ferrando et al. [6]) and Claude 3.5 Haiku (Lindsey et al. [15]), where abstention is the default pathway and recognized-entity features suppress it, so that hallucination is a failure of suppression. On Qwen3-1.7B the attempt is the default and abstention is the marked state, apparently gated by the knowledge-cutoff cue (all 147 refusals are post-cutoff, § 5.1), with no dictionary feature marking the confabulation. Llama 3.2 3B, refusing 97.5% of its post-cutoff failures, sits in the Gemma/Claude regime. Which pathway is default appears to be a post-training property, the same variable § 8 identifies behind the correctness signal.
Causal validation of feature 2191
The ensemble decomposition is correlational. I rerun the § 6.2 one-shot protocol with the feature’s decoder direction substituted for the probe direction (feature-steering lineage: Templeton et al. 2024; on 30 originally-wrong items, reading off the next-token probability of the opener set, 10 unique token IDs). The dose-response is monotonic with a sharp transition between and : from to across two steps, and from 0/30 to 30/30 argmax tokens flipped to refusal openers (Clopper–Pearson 95% CI [0.88, 1.00]; is small, so true rates as low as are consistent). Both sweeps add unit-norm vectors, so doses are directly comparable: the feature saturates at (0.46× the mean state norm) where the probe direction needs roughly twice the dose (three-direction dose-response table in Appendix § 13.3, Table 12). A single SAE feature is the more dose-efficient causal handle, causally sufficient for refusal-opener generation on wrong items (the refusal-item baseline at is already ).
| direction | 0.03× | 0.06× | 0.12× | 0.21× | 0.30× | 0.46× | 0.91× |
| probe (Section 6.2) | 13% | 17% | 22% | 29% | 35% | 45% | 92% |
| SAE feature 2191 | 20% | 27% | 40% | 64% | 91% | 100% | 100% |
| directly optimized | 22% | 34% | 54% | 97% | 100% | 100% | 100% |
Table 12. First-token refusal-opener rate on 201 held-out wrong items under pushes along the three candidate refusal directions, doses as fractions of the mean state norm (bfloat16 final-norm+head evaluation; the optimized direction is trained on the other half of the wrong items at the 0.12× dose. The probe direction’s fit pool included these 201 items, an asymmetry that favors the probe and so cannot explain its lower curve).
Two scope notes. First, 2191 was selected by the correlational views before any intervention ran, so the 30/30 evaluation is not circular (though the wrong items come from the same ConfabQA pool as the selection statistics). Second, three norm-matched random directions plus a shuffled-label probe direction leave the steering matrix unmoved at sub-norm doses (rates in Appendix § 13.3; one random seed partially collapses refusals at a super-norm dose, so super-norm doses can act by magnitude alone, which is why the working regime stays at or below the state norm).
Feature 2191’s decoder matches the lens opener vocabulary, and the probe direction’s causal effect runs through 2191 specifically. Its ensemble neighbors (14034, 17077, 4314) contribute to the correlational direction only. The two vectors are far from the same object (cosine 0.16 to full-subset recovery and 0.25 to the within-post direction used in the push; chance in 2048 dimensions) yet produce the same first-token flip, 81° apart in the plane they span (Figure 12, Appendix § 13.4.§ 4; norm accounting in Appendix § 13.3).

Figure 12. The plane spanned by the probe refusal direction and SAE feature 2191’s decoder vector (refusal+wrong states in the probes’ representation, ; stars: class centroids; dashed: per-class covariance ellipses). The two arrows are 81° apart (cosine 0.16), yet the probe axis carries the class split and its decision boundary (dotted) is exact in this plane: a nearly probe-orthogonal output knob and a broad discriminative mixture share one causal channel. The optimized refusal direction (green, § 6.3.1) is projected onto the same plane; its arrow is drawn at its in-plane fraction, so its shorter length shows that 64% of it lies in the plane (angle to plane 50°). It lands beside feature 2191 (cosine 0.64), not the probe axis.
Wollschläger et al. 2025 report distinct, far-from-collinear directions for safety refusal. The same pattern appears here: 2191 and the optimized direction are the more dose-efficient pair (Figure 3).

Figure 3. The geometry of the six refusal-direction candidates. (a) Pairwise cosines split the recoveries into two families: data-side (probe variants and the class-mean difference, 0.33–0.59) and output-side (2191 decoder and directly optimized, 0.64), weakly coupled (0.09–0.27; chance ≈ 0.02 in 2048-d). (b) Held-out first-token flip curves, doses as fractions of the mean state norm: causal potency tracks output-family membership, not discriminative quality. The native pre-norm probe separates the classes as well as any recovery (Appendix § 13.3) yet carries the least causal effect.
Optimized refusal direction. Where the probe and the class-mean difference recover the direction from data, a third route recovers it from the objective: maximizing the opener log-probability under the final RMSNorm and LM head with the vector as the only trainable parameter (setup, cosines, and lens in Appendix § 13.3). The optimum lands next to feature 2191, not the probe direction, and is the most dose-efficient of the three (Table 12). The ordering optimized, then SAE feature, then probe tracks how directly each recovery is shaped by the output objective; the probe direction adds discriminative signal rather than causal potency (Figure 3).
All four causal routes converge on the strict criterion: on the same wrong items, the probe push, the 2191 push, and prefix-forcing either literal opener produce the same downstream judged-refusal rate of roughly one in three, consistent with complete mediation at the first token (per-route rates, the not-powered-as-equivalence caveat, and dose-for-dose accounting in Appendix § 13.3).
Cross-model results
I rerun the pipeline on Gemma 2 2B (unsloth/gemma-2-2b-it) and Llama 3.2 3B (unsloth/Llama-3.2-3B-Instruct), then repeat the intervention with each model’s own refusal direction.
Comparison (Qwen3-1.7B, Gemma 2 2B, Llama 3.2 3B)
The same judge (Qwen3-1.7B) grades all outputs for a standardized correctness threshold. Table 16 (PDF p. 30) (Appendix § 13.4.§ 5) compiles the key statistics, alongside the per-layer probe curves for all three models (Figure 14).
| Metric / Probe | Qwen3-1.7B | Gemma 2 2B | Llama 3.2 3B |
| Overall Accuracy | 30.0% | 31.9% | 30.6% |
| Pre-cutoff Accuracy | 47.3% | 69.6% | 80.7% |
| Post-cutoff Accuracy | 19.5% | 9.0% | 0.2% |
| Refusal Rate on Post-cutoff Failures | 37.4% (147/393) | 55.2% (245/444) | 97.5% (475/487) |
| Probe accuracies (peak % [peak layer] vs baseline | |||
| Correctness (all items) | 82.4% [L18](base 70.0%, +12.4 pp) | 89.7% [L14](base 68.1%, +21.6 pp) | 94.3% [L13](base 69.4%, +24.9 pp) |
| Cutoff (all items) | 98.2% [L13](base 62.2%, +36.0 pp) | 98.9% [L14](base 62.2%, +36.6 pp) | 99.4% [L25](base 62.2%, +37.1 pp) |
| Refusal-vs-Wrong (subset) | 89.4% [L28](base 73.2%, +16.2 pp) | 83.9% [L19](base 53.6%, +30.3 pp) | 95.8% [L28](base 91.9%, +3.9 pp) |
| Correct within Pre-cutoff | 84.8% [L18](base 52.7%, +32.1 pp) | 91.2% [L17](base 69.6%, +21.6 pp) | 85.2% [L13](base 80.7%, +4.4 pp) |
| Correct within Obscure | 80.5% [L7](base 66.7%, +13.8 pp) | 88.9% [L24](base 64.1%, +24.9 pp) | 81.1% [L13](base 75.2%, +5.9 pp) |
| Prompt-feature baselines (strongest; margin vs probe above) | |||
| Correctness vs +category | 80.0% (+2.4) | 88.0% (+1.7) | 91.8% (+2.4) |
| Refusal-vs-Wrong vs TF-IDF | 82.0% (+7.4) | 67.6% (+16.3) | 93.6% (+2.2) |
| Within-Pre Correct vs +category | 81.8% (+3.0) | 86.1% (+5.1) | 79.4% (+5.8) |
Table 16 (PDF p. 30). Cross-model comparison summary.

Figure 14. Comparative probing accuracy curves across Qwen3-1.7B, Gemma 2 2B, and Llama 3.2 3B (§ 7.1).
The three models reach nearly identical overall accuracy (30–32%) but adopt different abstention policies on post-cutoff questions. Llama refuses almost every post-cutoff failure (97.5%, 475/487), Qwen only about a third, Gemma in between. This behavioral axis, not accuracy, is what the refusal probes track. The correctness margins fall to a few percentage points against the “+category” baseline on all three models (§ 5.4’s confound, cross-model; depth-band detail in Appendix § 13.4.§ 5). The refusal-vs-wrong probe separates the three models most clearly. It beats the strongest prompt baseline by +7.4 pp (Qwen), +16.3 pp (Gemma), and +2.2 pp (Llama). It peaks in the deepest layers on Qwen and Llama (layer 28) and at layer 19 on Gemma, the late-layer localization the § 7.2 intervention exploits.
Causal intervention
I apply the § 6.2 protocol to Gemma 2 2B and Llama 3.2 3B, using each model’s own recovered refusal direction at its own probe peak layer (Gemma: 19; Llama: 28). The judge is Qwen3-1.7B for all three runs (Gemma’s chat template lacks a system role and cannot self-judge), and ranges are calibrated per model to its hidden-state scale (Table 13, Appendix § 13.3). The order-of-magnitude spread in operational shows that the refusal direction is causal in each model, but its working range is set by the model’s hidden-state scale and tolerance to off-manifold perturbation.
| model | hook layer | swept | ||
| Qwen3-1.7B | 28 | 1640 | 220 | |
| Gemma 2 2B | 19 | 326 | 16 | |
| Llama 3.2 3B | 28 | 39 | 3.8 |
Table 13. Per-model state scale at the intervention hook (Qwen, Llama: post-block residual feeding the final norm, remeasured; Gemma: post-block-19 residual as cached) and the calibrated ranges.
Gemma 2 2B, whose refusal probe carried the largest margin over its prompt baseline, is also the most manipulable. The wrong-to-refusal flip reaches (vs. Qwen3’s ) at , with 100% opener rate (sweep and off-manifold pathologies in Table 14, Appendix § 13.3).
| metric | ||||||
| WRONG: judge says REFUSAL | 23% | 40% | 0% | 87% | 67% | 57% |
| WRONG: first-token opener | 0% | 0% | 13% | 100% | 70% | 83% |
| REFUSAL: judge says REFUSAL | 80% | 93% | 100% | 100% | 97% | 87% |
| REFUSAL: first-token opener | 0% | 0% | 53% | 100% | 77% | 90% |
Table 14. Gemma 2 2B one-shot intervention across its calibrated sweep ( per subset; judge = Qwen3-1.7B).
Llama 3.2 3B’s near-saturated default refusal policy (97.5% of post-cutoff failures) leaves only confabulating items, too few for a clean directional-causal claim. Its tightly normalized hidden state goes off-manifold symmetrically at . The clearest directional effect is the reverse push: the refusal rate drops from 100% to 80% at (3/15 previously-refusing items committing to a confabulation). That −20 pp shift is real but small, consistent with Llama’s +2.2 pp refusal-probe margin (full sweep, pathologies, and examples in Appendix § 13.3). The recorded Gemma sweep ran at 6–25× its residual scale, so I added a sub-norm rerun with controls. At of Gemma’s state norm the real direction converts 30%/40% of wrong items to judged refusals, monotonically in dose, versus 3–20% for two norm-matched random directions (cleanest at 0.31×: 30% vs. 3–7%). Gemma’s refusals themselves resist sub-norm de-refusal (26–30/30 hold; artifact 17_intervention_gemma_subnorm). The intervention is thus causal in Qwen and Gemma at doses at or below their state norms, with random-direction controls on both, and directionally suggestive on Llama. A model’s default abstention policy bounds both the signal magnitude of its refusal-vs-wrong probe and the operational dynamic range of the intervention.
Cross-dataset results
I next test whether the findings hold under a change of question distribution. This section moves from ConfabQA to PopQA and TriviaQA: bootstrap confidence intervals with a within-family scaling control, a refusal-channel attribution, and a no-refit transfer test.
Balanced-subsample bootstrap (ConfabQA, PopQA, TriviaQA)
This section checks whether the single-seed point estimates hold under resampling and whether the conclusions transfer to broader factual benchmarks. The Section 4 bootstrap protocol runs on nine ConfabQA cells plus five external-dataset cells (PopQA {Qwen3-1.7B, Qwen3-4B, Llama 3.2 3B} + TriviaQA {Qwen3-1.7B, Llama 3.2 3B}), fourteen (dataset, model, target) combinations (label provenance in Appendix § 13.5). The Qwen3-4B PopQA cell is the within-family scaling control, which separates parameter count from family / post-training recipe. Gemma’s PopQA and TriviaQA responses enter only the Section 8.3 transfer analysis; the bootstrap covers the Qwen-versus-Llama external contrast.
Of the fourteen cells (Table 17 and forest plot, Appendix § 13.5), 5 of the 9 ConfabQA cells and 5/5 external-dataset cells have subsample ranges excluding 0. The Qwen3 and Gemma disconfounded within-stratum cells remain consistent with zero but carry positive point estimates in all four cases. Thus the single-seed “null” was borderline. The ranges are per-cell and unadjusted for multiple comparisons. The cross-model contrast rests on no single borderline cell. Two protocol checks calibrate them (Appendix § 13.5). A label-permutation null shows 1 pp-range margins should not be read as evidence, while the all-items margins sit far above the null. An item-level bootstrap leaves both Llama within-stratum cells consistent with zero, so those two positive ranges should be read as point estimates.
| dataset | model | target | /class | range | excl 0? | |
| ConfabQA | Qwen3-1.7B | correct (all) | 235 | +4.30 | [+1.70, +7.45] | yes |
| ConfabQA | Qwen3-1.7B | within_pre | 140 | +1.25 | [−1.07, +3.57] | no |
| ConfabQA | Qwen3-1.7B | within_obscure | 51 | +0.95 | [−2.05, +6.90] | no |
| ConfabQA | Gemma 2 2B | correct (all) | 250 | +4.98 | [+2.40, +7.20] | yes |
| ConfabQA | Gemma 2 2B | within_pre | 90 | +2.54 | [−0.56, +5.00] | no |
| ConfabQA | Gemma 2 2B | within_obscure | 55 | +2.76 | [−0.91, +6.36] | no |
| ConfabQA | Llama 3.2 3B | correct (all) | 240 | +0.94 | [+0.21, +1.88] | yes |
| ConfabQA | Llama 3.2 3B | within_pre | 57 | +11.43 | [+5.18, +19.45] | yes |
| ConfabQA | Llama 3.2 3B | within_obscure | 38 | +12.26 | [+2.67, +23.67] | yes |
| PopQA | Qwen3-1.7B | correct (full) | 354 | +4.35 | [+2.11, +6.50] | yes |
| PopQA | Qwen3-4B (within-family scaling) | correct (full) | 129 | +5.77 | [+2.34, +11.61] | yes |
| TriviaQA | Qwen3-1.7B | correct (full) | 400 | +9.57 | [+5.13, +13.75] | yes |
| PopQA | Llama 3.2 3B | correct (full) | 131 | +24.94 | [+20.57, +29.03] | yes |
| TriviaQA | Llama 3.2 3B | correct (full) | 356 | +21.25 | [+19.52, +22.89] | yes |
Table 17. Bootstrap ranges (minimum to maximum over subsamples) on across the 14 (dataset, model, target) cells; forest plot in Appendix § 13.5 (Figure 16).
Cross-dataset cross-model contrast. On the same PopQA data under the same substring-correct criterion (Qwen pooled over three seeds, Llama single-seed; Section 4), Llama recovers a +24.9 pp correctness margin where Qwen3-1.7B recovers +4.4 pp, a 5.7 gap on 1.8 the parameter count. On TriviaQA the gap is 2.2. It is driven neither by raw accuracy nor by the prompt-baseline floor (accounting in Appendix § 13.5): Llama’s hidden state is qualitatively richer in linearly-decodable correctness information.
Within-family scaling control (Qwen3-4B). Qwen3-4B (same family, 2.4 Qwen3-1.7B’s parameter count, 25% more than Llama 3.2 3B) lands at +5.77 pp, range [+2.34, +11.61]: statistically indistinguishable from Qwen3-1.7B’s +4.35 pp and far below Llama’s +24.94 pp. Doubling parameter count within the Qwen family moves by 1.4 pp. Switching family at smaller parameter count moves it by 19 pp. The cross-model gap is not a parameter-count effect: it tracks model family / post-training recipe (corroborating raw accuracies in Appendix § 13.5; artifact list in Appendix B.4).
Refusal-channel attribution
The up-to-5.7 gap has a candidate mechanism. The probe target labels “wrong” together with “refusal,” and Llama refuses far more than Qwen on these benchmarks (PopQA 62.9% vs. 1.7%; census in Appendix § 13.5). Its could therefore be refusal-channel readout rather than factual self-knowledge: the “refusal direction is the whole atlas” hypothesis, extended to external data.
Two tests run under the same bootstrap protocol on the four cells with appreciable refusal counts. Test A (drop refusals) filters the pool to judge_label {correct, wrong}, balances 50/50, and re-probes, so any surviving is genuine correct–wrong discrimination among attempted answers. Test B (probe refusal directly) targets judge_label == 'refusal' balanced 50/50, measuring how cleanly the abstention decision is encoded beyond prompt-text predictability.
Dropping refusals leaves Llama’s margin largely intact (+24.94 +20.76 on PopQA, +21.25 +17.71 on TriviaQA): the refusal channel accounts for 17% of Llama’s margin, and 83% is correct-vs-wrong discrimination on attempted items. The Qwen rows are essentially unchanged (Qwen refuses too rarely here for the confound to bite). Probed directly, Llama’s refusal target beats the strongest prompt baseline by +18.44 (PopQA) and +25.43 pp (TriviaQA), with ranges excluding 0 by wide margins; Qwen’s analogous margins are positive but underpowered. One caveat: the ratio’s numerator is judge-labeled and its denominator substring-labeled; a label-sensitivity rerun shows the contrast is not a label artifact (full table and rerun in Appendix § 13.5, Table 19). Llama 3.2 3B thus carries two independent linearly-decodable signals: correct-vs-wrong among attempted answers (Test A) and a clean abstention direction (Test B), with correctness the larger share. The “refusal direction is the whole atlas” framing fits Qwen3, not Llama.
| Test | dataset | model | /class | probe acc | strongest baseline | range | |
| A: drop refusals | PopQA | Qwen3-1.7B | 355 | 79.18% | 75.87% | +3.31 | [+1.55, +6.48] |
| A: drop refusals | PopQA | Llama 3.2 3B | 102 | 82.37% | 61.61% | +20.76 | [+14.67, +27.94] |
| A: drop refusals | TriviaQA | Qwen3-1.7B | 400 | 69.40% | 60.04% | +9.36 | [+6.50, +12.87] |
| A: drop refusals | TriviaQA | Llama 3.2 3B | 179 | 74.23% | 56.52% | +17.71 | [+13.43, +21.78] |
| B: probe refusal | PopQA | Qwen3-1.7B | 39 | 86.27% | 76.04% | +10.24 | [0.00, +20.42] |
| B: probe refusal | PopQA | Llama 3.2 3B | 297 | 84.47% | 66.03% | +18.44 | [+16.16, +21.72] |
| B: probe refusal | TriviaQA | Qwen3-1.7B | 28 | 73.34% | 57.37% | +15.97 | [−3.48, +32.27] |
| B: probe refusal | TriviaQA | Llama 3.2 3B | 252 | 90.23% | 64.80% | +25.43 | [+22.81, +29.76] |
Table 19. Refusal-channel attribution: Test A (drop refusals) and Test B (probe refusal directly).
Transfer of the correctness probe
I next test whether a correctness probe fit on dataset classifies dataset without refit, isolating a content-invariant signal from content-specific correlates. For each model I fit the pipeline on each dataset at the peak-within layer and evaluate it on the other two without refitting: a matrix, within-dataset CV on the diagonal, pure transfer off it, each cell scored against the test-dataset majority (layer grids, frozen-statistics detail, and the TF-IDF transfer control in Appendix § 13.5).
Averaging margins (probe-acc minus test-majority) across the 3 within cells and the 6 transfer cells (Table 18, Appendix § 13.5), Llama retains 81% of its within-dataset margin under transfer (+18.9 pp within, +15.2 pp transferred, 5/6 transfers beating the test majority). Gemma’s retention is −54% (largest within margin of the three, but two sharp drops with PopQA as target), and Qwen’s is −14% (an almost flat probe staying almost flat); per-model detail in Appendix § 13.5.
| Model | Peak layer | Avg within margin | Avg transfer margin | Retention | Transfers > majority |
| Llama 3.2 3B | 14 | pp | pp | 81% | 5/6 |
| Gemma 2 2B | 13 | pp | pp | 4/6 | |
| Qwen3-1.7B | 18 | pp | pp | 3/6 |
Table 18. Cross-dataset transfer at each model’s peak-within layer: margin retention without refit.
Within-dataset peak performance (Llama > Gemma > Qwen) and transfer portability (Llama ≫ Gemma ≈ Qwen) are different orderings: Gemma decodes dataset-specific content patterns, while a Llama direction found on any pool classifies the others nearly as well as the source. This matches Finding 2: Llama 3 has a factuality / refusal stage that Qwen3 and Gemma reports do not describe (full matrices released; Appendix § 13.2.§ 4).
Discussion
Results are more model-dependent than the ConfabQA single-seed read suggested. Disconfounded within-stratum margins are small on all three models (single digits against the strongest prompt baseline; one cell negative). The bootstrap leaves the Qwen and Gemma cells consistent with zero, and Llama’s within-stratum cells are suggestive rather than established (§ 8.1). Llama’s powered evidence is the external-dataset margins, whose subsample ranges sit far from zero under either label regime. § 8.2 partitions them into mostly correct-vs-wrong discrimination, a smaller refusal-channel share, and an independent abstention direction.
Single-model framing. Qwen3-1.7B alone would suggest that the hidden-state correctness probe adds nothing beyond a prompt-text classifier at 1.7B–3B scale. On balanced external subsamples Llama’s correctness probe adds +17–+25 pp over the strongest prompt baseline, with subsample ranges excluding 0 by 13–21 pp. The claim that the refusal direction is the only recoverable structure fails the same way: at 3B Llama scale both refusal and correctness directions exist in legible form.
Refusal findings (Appendix § 13.6). The refusal direction’s logit lens recovers each model’s literal refusal-opening vocabulary; the one-shot intervention saturates the first-token opener rate on Qwen3 and Gemma; the § 8.2 refusal probe extends the finding to PopQA and TriviaQA at Llama scale; and the geometry is visible in unsupervised PCA. The Qwen3-1.7B correctness direction fails all three corresponding tests. Two atlases on Llama vs. one on Qwen (Appendix § 13.6). The most plausible mechanism is the Llama 3 factuality stage (Finding 2), with the Qwen3-4B control ruling out parameter count. Consequence for probing (Appendix § 13.6). Probes reporting –90% correctness prediction should be re-evaluated against a prompt-feature baseline. The outcome is model-dependent, so small-LM claims need at least two architectures with different post-training recipes and bootstrap CIs on balanced subsamples.
Limitations
Benchmark construction. Four known imperfections in ConfabQA, none of which affect the prompt-feature baseline (it conditions on text). First, a rater-subjective obscure/well-known boundary. Second, gold answers are a validation snapshot (facts drift, so the pipeline must be re-run). Third, Anglophone source skew. Fourth, a fuzzy pre/post-cutoff boundary in the science domain, where Qwen3’s post-training corpus apparently included a handful of post-cutoff OpenAI/Anthropic/SpaceX items, so the near-perfect cutoff probe partly recovers “what kind of entity” rather than a clean date discriminator.
Dataset quality. The ConfabQA gold answers were not systematically checked before the experiments (§ 3). An automated web-search audit of the 400 post-cutoff golds, itself unverified, flagged 22 wrong-gold or false-premise items, 43 ambiguous items, and 63 groups of near-duplicate questions. By their own answer_date, 107 post-cutoff items predate the cutoff stated in the dataset card (answer true after September 2024), 74 of them by more than a month, and some of those dates are themselves wrong. A rerun under the web-verified DR
labels keeps both probe margins positive (Appendix B.3), but that grader flagged only three premise errors. A cleaned dataset and a sensitivity rerun of the ConfabQA analyses will follow in v2.
Evaluation methodology. gives only items per CV fold, and the within-obscure probe’s layer range spans the entire network. The prompt-feature baseline is hand-crafted and likely a lower bound on what a generic text classifier could extract. The bootstrap resamples balanced subsets, not the underlying pool, so its reported interval (the minimum-to-maximum range of subsamples) describes subsample spread, not population uncertainty. CV folds are stratified by label but not grouped by source file. Items come from 21 templated source files of up to 100 items each, so near-duplicate prompts can fall on both sides of a split. This favors the prompt-feature baselines at least as much as the probe, but a grouped-CV check is not run here. The Section 8.3 transfer matrix covers the correctness probe only: Gemma’s and Qwen’s few dozen external refusals are too few for a cross-dataset refusal fit, while Llama’s counts would support one. A Llama refusal-transfer test plus a known-unknowns benchmark eliciting refusals from the smaller models would close the gap.
Scope and judge generalization. Four models in the 1.7B–4B range, no B replication. The scaling control rules out parameter count as the dominant driver of the cross-model gap, but family and post-training recipe remain confounded, and a clean isolation would swap the recipe holding family fixed. The Qwen judge on Qwen subject (and on Llama outputs in Section 8.2) is partially de-risked by the cross-model intervention runs (Gemma reproduces the Qwen refusal pattern under an external Qwen judge), but an independent or human re-grade on a Llama-output subset remains the cleanest remaining check.
Probe target. All probes read the last prompt position of the prefill pass (Section 2.2). Orgad et al. [18] report that truthfulness information concentrates at the exact-answer tokens of the generated response, and Kossen et al. [11] recover semantic-entropy signal from generation-time states. The Qwen3-1.7B correctness null is therefore a statement about the pre-generation state, not about every state the model produces, and an exact-answer-token probe scored against the same prompt-feature baseline is the direct extension.
Conclusion and future work
I introduce ConfabQA and apply it, plus PopQA and TriviaQA samples, to three instruction-tuned models and a Qwen3-4B within-family scaling control, with balanced-subsample bootstrap ranges across 14 (dataset, model, target) cells. Findings:
(i) The hidden state encodes a refusal-vs-confabulation signal that beats prompt features on Qwen3 and Gemma (ConfabQA +7.4–+16.3 pp; deepest layers; the logit lens recovers each model’s literal refusal-opening vocabulary; causal under a one-shot forward-hook intervention on Qwen and Gemma; on Llama the sweep ran at its remeasured state norm and is directional evidence only).
(ii) The correctness-probe gap is model-dependent. On balanced PopQA / TriviaQA subsamples Llama 3.2 3B adds +24.9 and +21.3 pp over the strongest prompt baseline (subsample ranges [20.6, 29.0], [19.5, 22.9]) where Qwen3-1.7B adds +4.4 and +9.6 pp (ranges [2.1, 6.5], [5.1, 13.8]); Llama’s ConfabQA disconfirms a positive statement are also positive (+11.4 pp within-pre, +12.3 pp within-obscure) though an item-level bootstrap leaves both consistent with zero (Section 8.1). The “correctness probe reduces to prompt features at 1.7–3B scale” framing does not hold beyond that one model.
(iii) Llama’s correctness signal is primarily not refusal-channel readout: dropping refusals leaves +20.8 pp (PopQA) and +17.7 pp (TriviaQA), so most of the headline ; label caveat in Section 8.2 is correct-vs-wrong discrimination among attempted items, and a direct refusal probe additionally recovers +18.4–+25.4 pp, an independent clean abstention direction. Llama at 3B carries two linearly-decodable signals; Qwen3-1.7B carries only weak versions of either at this scale and on these benchmarks.
Methodologically, the paper combines a prompt-feature baseline, 50/50 class-balanced subsampling, bootstrap ranges over such subsamples, and cross-model plus cross-dataset evaluation. A single-model, single-seed, class-imbalanced read of the same code could have reported either polarity. Directions for v2: (i) swap the post-training recipe holding family fixed and check whether the cross-family gap survives at 7B; (ii) extend the one-shot intervention to a per-token steering vector (Turner et al. 2023) and characterize the at which sustained pressure flips the full output before fluency degrades; (iii) add an item-level non-parametric bootstrap and an independent human annotator on Llama outputs to close the bootstrap and judge-generalization gaps.
Code and data availability. All code, the ConfabQA benchmark, judge labels, bootstrap outputs, SAE artifacts, and the figure-generation pipeline are released at https://github.com/vkazei/confabqa under MIT (code) and CC BY 4.0 (data, paper, figures). Activations and per-seed external-dataset responses are gitignored (multi-GB) but regenerable via the documented recipes. All experiments ran on a single Apple M1 Pro (16 GB) in bfloat16 with per-script seeds documented at the top of each file.
Use of Large Language Models
The paper and code were developed with the author-guided assistance of Anthropic’s Claude Code (CC). I posed the research questions, designed the ConfabQA benchmark and its prompt-feature control, and made the modeling and interpretation decisions. CC implemented the probing, intervention, sparse-autoencoder, and statistical-analysis code and helped refine experiments (for example, the random-direction controls and the direct-optimization recovery). I developed the interpretation of results in dialogue with CC. The ConfabQA question-and-answer items were generated with Google Gemini in Deep Research mode, which also performed the independent regrade of § 4. CC drafted the initial manuscript and co-edited it, created figures, formatted the references, and searched for related literature.
Appendix
Per-category and per-domain accuracy and the excluded sports domain
Table 2 gives the full per-category judge-label breakdown behind § 5.1. Mean log-probability also stratifies: the model is least confident on the obscure pre-cutoff items (refusals, concentrated post-cutoff, are themselves fluent and high-confidence; § 5.2).
| domain | n | correct | accuracy | pre-cutoff | post-cutoff |
| science | 206 | 75 | 36.4% | 25/87 (28.7%) | 50/119 (42.0%) |
| history | 197 | 67 | 34.0% | 54/70 (77.1%) | 13/127 (10.2%) |
| culture | 189 | 65 | 34.4% | 42/68 (61.8%) | 23/121 (19.0%) |
| cinema | 192 | 28 | 14.6% | 19/71 (26.8%) | 9/121 (7.4%) |
Table 4. Per-domain accuracy, overall and by cutoff class.
History remains the strongest pre-cutoff domain (77.1%); cinema the weakest overall (14.6%). Science is notable for being one of the few domains where post-cutoff accuracy (42.0%, 50/119) is higher than pre-cutoff (28.7%), driven by a handful of post-cutoff items about Anthropic, OpenAI, and SpaceX that the model has apparently learned via post-training data.
The excluded sports domain
An earlier iteration of the benchmark included a fifth domain, sports, with the same three-category design applied to MLB World Series, NFL Super Bowl, and Grand Slam tennis questions. On a 26-item run of Qwen3-1.7B, this domain produced a uniform failure mode across all three categories: Judge labels: 23 wrong, 3 refusal, 0 correct. Examples of confidently wrong responses:
| domain | well_known | obscure | post_cutoff | overall |
| sports | 0/10 (0.0%) | 0/10 (0.0%) | 0/6 (0.0%) | 0/26 (0.0%) |
Table 5. The excluded sports domain: accuracy by category on the 26-item run.
| question | gold | model said | |
| Men’s 2023 French Open winner | Novak Djokovic | “Alexander Zverev defeated Djokovic in the final” | −0.087 |
| 2023 World Series winner | Texas Rangers | “Los Angeles Dodgers defeated Chicago Cubs in seven games” | −0.091 |
| 2009 World Series winner | New York Yankees | “St. Louis Cardinals defeated the New York Yankees” | −0.094 |
Table 20.
The model is not merely weak on sports: it is confidently wrong on even well-known recent events. Dropping sports from the main 4-domain analysis is motivated rather than convenient: the zero-correct well-known cell means the within-pre-cutoff probe on this domain alone would have zero positive examples, making the disconfound test undefined. The source files (data/sources/sports/*.json) remain in the repository and the question-set generator can re-include the domain by reverting one line of 01_question_set.py:DOMAINS. The 26 sports responses (data/responses_v1_133/) are not in the repository.
Text-level confidence figure
Figure 4 shows the § 5.2 text-level confidence distributions. Panel (b) shows that the correctness/cutoff interaction is also visible in joint logprob-entropy space, with post-cutoff items concentrated in the lower-confidence region of the plane (lower mean logprob, higher mean entropy) and pre-cutoff correct items concentrated near zero. The substantial overlap between pre-cutoff-correct and post-cutoff-correct in the high-confidence corner is the visual analogue of “logprob alone does not separate confabulation from correct”.
Highest-confidence confabulations and refusals
Table 6 lists the § 5.2 case studies; they exemplify the canonical failure mode, a plausible-sounding specific answer with a recognizable name from the right time period and category, at high text-level confidence. Four of the five confabulations are obscure pre-cutoff items, where the model likely had some training-data exposure to the gold answer; the fifth is post-cutoff, where it had none.
Hidden-state geometry figures
Probe-peak summary vs. majority baselines
Table 8 is the Section 5.3 probe-peak summary (vs. majority baseline only).
The layer-range column matters: for the within-obscure probe at , accuracies across the entire 28-layer network are within one fold-standard-deviation of the argmax, so the specific “peak layer 7” is not a meaningful per-depth claim; only the overall mean accuracy is.
Prompts, schemas, and data artifacts
Judge prompt
The full judge system prompt and template are in judge.py. The system prompt defines a three-way decision rule (CORRECT / REFUSAL / WRONG), gives five worked few-shot examples, and instructs the model to emit exactly one line of the form Label: X; the parser is a regex on Label:\s*(CORRECT|REFUSAL|WRONG). The user-message template embeds the question, the gold answer, the acceptable alternatives, and the model’s answer in a fixed quoted-block format. A regex parser extracts the label from the judge output; on parse failure the fallback is to scan for a bare keyword on the first line, then to label as wrong with a parse_error: true field for audit.
Validation prompt schema
The validation prompt emitted by 01_question_set.py --emit-validation-prompt is in data/questions_v1_validation_prompt.md; the JSON output schema returned by the external LLM is specified in that file’s “Required output format” section. The validation pipeline (01_question_set.py --emit-validation-prompt -> external LLM -> --apply-validation) is reproducible bitwise from the source files and validation results, and gold corrections leave an audit trail in validation_notes.
A case study for the validation pipeline. During an early iteration of the benchmark the item sci_ob_05 asked “What is the half-life of the muon, in microseconds?” The gold, written by the dataset author from memory, was 2.197, which is in fact the muon mean lifetime , not the half-life µs. The external validation pass flagged this 30.7% systematic error; the source question was subsequently rewritten to ask explicitly for the mean lifetime, eliminating the ambiguity. This case shows why gold-side validation is essential in a calibration benchmark: a plausibly-written gold can contain a systematic error that no downstream judge or probe can detect.
The Section 3 obscure item in full: “Who won the Academy Award for Best Actor at the 1968 ceremony for ‘In the Heat of the Night’?” (gold: Rod Steiger).
Annotator-agreement data for judge calibration
Two independent annotators re-graded ConfabQA alongside the Qwen3-1.7B judge; Table 9 summarizes the agreement.
The annotation sources:
data/gemini_regrade/qwen3_1_7b_dr_labels.jsonl: DR (Gemini 3.1 Pro with web-search verification) grading all 784 items using the literaljudge.pythree-way decision rule.data/claude_grading_v1.json: Claude Opus 4.7 grading a stratified 30-item sample (random.Random(42), 10 items per category) from (question, gold, model answer) triples under the same rule.
Cohen’s corrects the raw agreement rate for the agreement two annotators would reach by chance:
where is the observed agreement rate and is the rate expected if the two annotators labeled independently with their observed marginal label distributions. For DR vs. the Qwen-judge on all 784 items, and , giving .
The 102 DR vs. Qwen-judge disagreements decompose as 45 items DR calls wrong where the Qwen-judge calls correct (Qwen too lenient on partial-match containment), 38 DR-wrong / Qwen-refusal (DR sees fabrication where Qwen sees hedging), 11 DR-refusal / Qwen-wrong, 7 DR-correct / Qwen-wrong, and 1 DR-refusal / Qwen-correct.
The Qwen judge labels, with each question, gold answer, and model answer, are in data/gemini_regrade/qwen3_1_7b_items.jsonl; the per-item response files under data/responses/ are not in the repository. Agreement statistics for both annotators are in Table 9; the DR pass was also asked to web-verify each gold answer and flagged three items with stated-premise errors (his_pc_09: month wrong by one; his_pc_56: year wrong by one; his_pc_90: dual premise error); a later automated audit of the 400 post-cutoff golds found more (§ 10).
DR strictness and label-sensitivity rerun (§ 4). DR is systematically stricter than the Qwen-judge (467 vs. 402 wrong-labels), and the residual 13% disagreement primarily reflects DR’s stricter containment rule and its own gold checks, not judge miscalibration. A sensitivity rerun under the DR labels keeps both probe margins over the TF-IDF baseline positive: the refusal-vs-wrong probe reaches 87.1% (vs. 89.4% under the judge labels; margin over a TF-IDF question-text baseline +3.9 vs. +7.7 pp), and the layer-18 correctness probe rises to 87.6% (margin +9.8 vs. +6.0 pp); artifact revision_analyses. The rerun refits the TF-IDF baseline, so its judge-label margins (+7.7, +6.0 pp) differ slightly from Table 7 (+7.4, +5.7 pp). The three flagged stated-premise items (3/784 ≈ 0.4%) are retained because dropping them does not shift any headline number; v2 will patch them at the source. They are only the errors the DR pass caught; the § 10 audit reports more.
Regrade of intervened generations (§ 4, § 6). The intervened generations of Section 6 are graded by the subject model; a blind Claude-family regrade of 60 of them agrees with the self-judge at 85% (, comparable to the graders’ on natural outputs), with disagreements balanced across label directions rather than inflating flip rates (artifact intervened_regrade_claude).
Authoritative summary
Every Qwen3-1.7B single-model number in Sections 5.1–5.4 and 6.1–6.2 is derived from data/qwen3_1_7b_summary.json, generated by 03_analyze.py in a single pass over the cached responses and activations. The summary file records: per-judge, per-cutoff, per-category, and per-domain counts; full per-layer accuracy curves and 1σ layer ranges for every probe target; class-imbalance metrics at peak for the refusal probe; and all three prompt-feature baseline variants. Reproducing those numbers consists of running python 03_analyze.py and reading the resulting JSON. The Section 6 and Sections 7–8 numbers are recorded in per-experiment artifacts under figures/, one JSON (with the generating script of the same name under analysis/ or saes/) per experiment: 13_intervention_results, 14_intervention_correct_subset, 15_intervention_three_directions, 16_intervention_random_controls, intervention_dose_ratios, revision_analyses, small_cell_stats, intervened_regrade_claude, 17_intervention_gemma_subnorm, correctness_direction_lens, correctness_direction_intervention, refusal_direction_meandiff, pca_coverage_2191, probe_2191_blend, sae_decompose_prenorm, optimized_refusal_direction, sae_causal_ablation, sae_feature_refusal_rate, sae_apology_feature_test, sae_feature_trajectories, sae_wrong_features, prefix_forcing_control, hidden_state_norms, bootstrap_h_adds, bootstrap_llama_external, bootstrap_qwen3_4b, refusal_channel_test, and cross_dataset_transfer_<model>. The Section 6.3 scripts read the pre-norm activation cache, rebuilt with python -m analysis.cache_prenorm_states (Appendix § 13.4.§ 2).
Sample source-file entry
data/sources/science/nobel_physics.json:
{
"_documentation": {...},
"template": "Who won the Nobel Prize in Physics in {year}?",
"answer_field": "winner",
"well_known": [...], "obscure": [...], "post_cutoff": [...]
}
Token-flow and baseline schematics
Figure 8 shows the two forward-pass regimes behind the § 2.2 probe target; Figure 7 shows the § 2.3 probe pipeline; Figure 9 sketches the § 2.4 baseline design.
Confidence metrics in full (§ 2.1). At each generation step the model produces a distribution over the next token; all generation is greedy (do_sample=False, hence deterministic), so the emitted token is the mode of . Two scalar summaries of the generated response recur in §5, both means over the generated positions (, with the prompt length and ):
In (2), is the log-probability of the emitted (argmax) token, so is the length-normalized answer likelihood: is the geometric mean of the per-token probabilities, and corresponds to per token (I also report as a worst-step diagnostic). is the Shannon entropy of in nats ( when near-deterministic, when uniform); scores the picked token, the whole distribution, and the two are tightly anti-correlated empirically (Figure 4b, Appendix § 13.1.§ 2). The last-prompt-position prefill state is the natural probe target for three reasons: causal masking ( summarizes the whole prompt, and no later prompt position adds information); output-locality ( as cached, the final normed state of § 4, is up to a linear projection the first answer token’s logit vector, so anything linearly decodable from it is available to the output head at the moment of commitment); and no generation contamination (probing generated tokens’ states conflates what the model knows with what it has already committed to saying, since the generated prefix is fed back as input). On the § 2.1 metrics: and can come apart when one dominant mass carries a long thin tail of plausible alternatives.
Prompt-feature baseline specification
Formal Section 4 target definitions: correct is binary judge_label == "correct"; cutoff is binary cutoff_class == "post"; refusal_vs_wrong is binary judge_label == "refusal" on the {refusal, wrong} subset; the within-post and within-stratum variants restrict the item pool as named.
On the Section 2.3 probe pipeline: with items and features, a raw logistic regression would massively overfit; projecting onto the top 16 principal directions first keeps the ratio of training items to fitted parameters at instead of (C is sklearn’s inverse regularization strength, at its default), and sklearn.Pipeline semantics fit scaler and PCA on the training fold only.
The four Section 4 baselines in full:
TF-IDF (term frequency–inverse document frequency):
TfidfVectorizer(ngram_range=(1,2), min_df=2, max_df=0.95, sublinear_tf=True)→LogisticRegression(max_iter=2000, C=1.0)on the raw question text: the strongest “what could a bag-of-tokens classifier learn” baseline.text-only (engineered): numeric features only (question character length, word count, has-year indicator, year value, number of capitalized words, digits, commas, ends-with-? indicator).
text + domain: engineered features plus domain one-hot.
text + domain + category: adds the human-assigned category dummy, the annotator’s expected-difficulty label; for the cutoff target it is excluded from the strongest-baseline comparison (Section 4).
Intervention calibration and details
Calibration and supporting detail for the Section 6.2, Section 6.3.1, and Section 7.2 one-shot interventions.
Cached versus hooked states. Hidden states are captured by model.generate(...) with output_hidden_states=True and output_scores=True. Entries of the returned stack are raw post-block residuals, but the final entry (index ) is the final normed state, because HF applies the last RMSNorm before recording it. So the probes read the normed state at “layer 28” while the Section 6.2/6.3.1 hooks and the SAE operate on the pre-norm post-block-27 residual.
Magnitude and normalization. Magnitude enters only as a mixing ratio. Spelled out, with and , the intervened head input is
which depends only on the ratios and and converges to as . Here is the post-block-27 residual, the state the hook actually perturbs: its norm across all 784 items is (), so the single used in the alpha-translation below describes typical items to within . One structured feature: refusal states are markedly tighter (, a 5% coefficient of variation) than correct () or wrong () states; the templated behavior is geometrically stereotyped as well (figures/qwen3_1_7b/hidden_state_norms.json records both this and the final normed state’s statistics). The lens scores are therefore the asymptote of the first-token dose-response, which is the saturation Sections 6.2 and 6.3.1 observe, and why ranges must be calibrated to each model’s hidden-state scale (Table 13). The asymptotic argument holds only when the hook feeds the final RMSNorm directly; when it sits below the final layer (Gemma at layer 19, the layer-18 correctness intervention), later blocks process the perturbed state, and large is genuinely off-manifold rather than asymptotic, the pathology Appendix C records.
Choice of scale, in full (Qwen3-1.7B). The natural scale is much larger than the per-fold projection variance because RMSNorm dampens the perturbation: the post-norm contribution to the LM head is approximately , and at the intervention point . Empirically, on a pre-cutoff probe item the natural greedy first token loses to the refusal opener As at , and at the top six tokens are entirely refusal openers; the sweep covers this range.
In relatable units: baseline states already lean toward the direction (mean cos = on refusal items, on wrong, at ), and the sweep corresponds to mean perturbation-to-state-norm ratios and mean post-intervention alignments (averaged over the subset). Per item, on the actual intervention subsets (artifact intervention_dose_ratios): is sub-norm for every pushed item (ratio ), the saturating sits at the state’s own norm (wrong-subset ratio ; a third of items exceed ), and is super-norm () for all. The smallest effective dose reaches only alignment yet already flips half the first tokens.
Hook and opener set (Section 6.2). The intervention hook sits on model.model.layers[27], whose output is the post-block-27 residual feeding the final RMSNorm; the pushed vector is the unit-normalized within-post recovery (the full-subset recovery is a substantially different vector, cosine ). The first-token opener set is .
Coordinate dissociation: normed vs. pre-norm recovery (Section 6.2). Refitting the same probe pipeline natively on the pre-norm residuals dissociates the two roles of the direction: the native recovery reads the class split equally well ( vs. CV accuracy) but is a substantially different vector (cosine to the normed-space recovery) with half the causal effect ( vs. held-out first-token flips at the dose; artifact revision_analyses).
Steering-matrix parity block (Section 6.2). Table 10 repeats the Table 1 protocol with the SAE-2191 and directly optimized directions on the same items (per-direction opener rates saturate by ; Table 12); the 0 column is the shared unperturbed baseline. Wrong items reach the same one-in-three judged-refusal plateau (already at , matching the steeper first-token curves of Table 12), de-refused items land as confabulations at the same rates, and correct items survive the pull while largely resisting the push (artifacts in Appendix B.4).
Random-direction controls (Sections 6.2, 6.3.1). Three norm-matched random directions and a shuffled-label probe direction, run through the identical generate-and-judge protocol (artifact 16_intervention_random_controls), leave the steering matrix unmoved at the sub-norm dose. At the controls keep 29–30/30 refusals refusing (vs. 6–12/30 under the three real directions), and at – they yield judged refusals and opener rates on wrong items (vs. 27–30% and 97–100%). One of the three random seeds does partially collapse refusals at the super-norm dose (16/30 wrong): super-norm doses can act by magnitude alone, which is why the working regime is kept at or below the state norm.
Optimized-direction training details (Section 6.3). The optimized refusal direction treats the intervention vector as the only trainable parameter and maximizes the mean log-probability of the refusal-opener token set under the final RMSNorm and LM head, on a train half of the 402 wrong pre-norm states, with fixed at a sub-nudge dose (no full-model backpropagation needed). The optimum lands next to feature 2191: cosine to the feature’s decoder vector, against to the Section 6.2 probe direction and to the mean difference, and its logit lens is a pure opener cone (As, As, _as, "As).
Feature 2191 dose-response table. The full sweep behind the Section 6.3.1 ablation:
Norm accounting for the probe–2191 pair. The component of along feature 2191 has length , the relevant scale for logit contributions, which are linear in the component; the orthogonal remainder keeps (in the additive Pythagorean accounting, energy shares of vs ). Two nearly disjoint vectors produce the same first-token flip because the flip requires only a sufficient projection onto the opener unembedding cone after RMSNorm; at the causal level the probe direction acts through its 0.16-of-norm component.
| wrong: | wrong: argmax-in-opener | refusal: | |
| 0 | 0.000 | 0/30 | 0.969 |
| 16 | 0.000 | 0/30 | 0.981 |
| 64 | 0.000 | 0/30 | 0.998 |
| 200 | 0.000 | 0/30 | 1.000 |
| 400 | 0.364 | 11/30 | 1.000 |
| 750 | 1.000 | 30/30 | 1.000 |
| 1500 | 1.000 | 30/30 | 1.000 |
| 3000 | 1.000 | 30/30 | 1.000 |
Table 11. Feature 2191 dose-response: next-token refusal-opener probability under .
Dose-for-dose potency. The two interventions are directly comparable: both add a unit-norm vector, so is the added norm in both sweeps, and both score the same statistic, the first-token argmax landing in the opener set (on different 30-item wrong subsets). Their half-flip doses are similar: the probe direction reaches 50% at ; feature 2191 is at 11/30 by and completes 30/30 by 750, where the probe needs 1500 for 97%. The concentrated feature saturates roughly twice as early. The naive projection accounting, under which the probe direction owns only 0.25 of its norm on the 2191 channel and should need the dose, overstates the gap: the probe’s remaining norm is not opener-inert, since it also pushes the broader opener family (there, the Chinese variants) that the flip criterion counts, and the flip is a competition threshold rather than a linear readout.
By the strict judge criterion the comparison is exact, since all the causal routes were run on the same 30 wrong items (Section 6.3.1). The probe direction converts 30% into judged refusals at , forcing the literal token “As” converts 37%, forcing “No” converts 27%, and feature 2191 converts 30% at and 27% at 1500. Every route to the opener token produces the same downstream refusal rate of roughly one in three, consistent with complete mediation at the first token (each route ; the comparison is not powered as an equivalence test). Released as figures/qwen3_1_7b/sae_feature_refusal_rate.json (analysis/sae_feature_refusal_rate.py).
Apology-feature push (Section 6.3). Adding at the Section 6.2 intervention point, at the same doses that saturate feature 2191 (), produces zero Sorry/Oops openers in 60 generations; the feature’s causal output routes through its weaker There component instead (13/30 There-openers at , and all eight induced grudged refusals open with “There”).
Correctness-direction null: protocol, rates, and flip forensics (Section 6.2). The Section 6.2 one-shot protocol applied to the layer-18 correctness direction (, originally-correct and originally-wrong items). Originally-correct items degrade mildly at large in both directions (100% 80–83% correct at ), originally-wrong items flip to correct at low, sign-independent rates (17% at both −3000 and +3000), and refusal induction is scattered (0–17%) with no dose-response. The flips concentrate on a recurring handful of items rather than spreading across the subset: the union over all six nonzero is 10 of 30 items, five account for most flips, and one flips at every nonzero in both directions. Inspection shows these are items where the correct answer was already marginally available, a narrowly wrong decode of a famous fact, or a post-cutoff fact the model demonstrably knows but hedges on, so a push of either sign can change the output. This null is in tension with inference-time intervention reports of truthfulness-direction steering ([13]); the settings differ (TruthfulQA’s misconception-heavy distribution, multihead attention interventions, and a truthful-vs-untruthful contrast rather than correct-vs-wrong recall), and reconciling them is future work.
Hidden-state scales. The per-model scale statistics and calibrated ranges below. In perturbation-to-state-norm units the swept ranges are not comparable across models: the state norm for Qwen ( on ), for Gemma ( on its mid-network residual scale of 326; the sub-norm rerun of § 7.2 shows the directional effect does not depend on that regime), and for Llama ( on a remeasured residual scale of 39). The Llama sweep therefore never probed the sub-norm regime at all, which is consistent with its magnitude-dominated behavior (Table 15 discussion): its directional evidence should be revisited at –. The Qwen and Llama rows below are the pre-norm residuals the interventions actually perturb (the earlier draft quoted final-normed-state norms); Gemma’s mid-network hook was cached correctly:
| metric | |||||
| WRONG: judge says REFUSAL | 83% | 0% | 0% | 8% | 75% |
| WRONG: first-token opener | 0% | 100% | 100% | 83% | 0% |
| REFUSAL: judge says REFUSAL | 80% | 100% | 100% | 100% | 93% |
| REFUSAL: first-token opener | 0% | 100% | 100% | 93% | 0% |
Table 15. Llama 3.2 3B intervention over its calibrated range ( wrong, refusal items).
Qwen3-1.7B sample generations (§ 6.2). (cin_pc_60, gold “Jacques Audiard”; baseline says “Luis Llosa”, a confabulation):
| first 100 chars | |
| 0 | 'Emilia Perez' (2024), a Spanish-language musical crime film, was directed by **Luis Llosa**. |
| +1500 | As of 2024, the Spanish-language musical crime film **"Emilia Perez"** was directed by **Miguel Ángel Martínez**. |
| +3000 | (same as +1500) |
Table 21.
And cin_pc_34 (baseline refuses on the 2024 Venice Golden Lion):
| first 100 chars | |
| 0 | As of my knowledge cutoff in 2024, there is no official announcement regarding a director winning the Golden Lion... |
| −500 | The 2024 Venice Film Festival took place from September 2 to 14, 2024. The Golden Lion, the highest award... |
| −2000 | (same as −500) |
Table 22.
Prefix-forcing control (§ 6.2). Generating the same 30 wrong items with the literal opener “As” forced as the first generated token and no hidden-state perturbation yields a 37% judge-refusal rate (11/30), statistically indistinguishable from the 30% the intervention reaches at and , where its first-token opener rate is 97–100%. Forcing “No” instead yields 27% judged refusals, with 2/30 items becoming correct (the forced negation occasionally cancels the confabulation), so the roughly one-in-three downstream rate is stable across opener choices. Released as figures/qwen3_1_7b/prefix_forcing_control.json and prefix_forcing_control_no.json (analysis/prefix_forcing_control.py, --prefix selects the opener).
Gemma 2 2B (, judge = Qwen3).
The effect is non-monotonic at large positive : +4000 and +8000 drop the judge-refusal rate to 67% and 57% respectively, even as the first-token-opener rate stays high. At those magnitudes Gemma starts producing fluent-but-off-target openers (“Formal announcements have not been made yet, but Marc Webb is not directing any…” ) that the Qwen judge sometimes parses as wrong. The negative- direction produces a different pathology: generations begin with garbled non-English tokens (Cyrillic “виправивши”, French “conseille”), suggesting Gemma’s refusal direction is partially entangled with prompt-encoding stability and pushing in the opposite direction off-manifold.
Llama 3.2 3B. The full Llama intervention sweep, displaced here from § 7.2 because Llama’s near-saturated default refusal (97.5% of post-cutoff failures) leaves only confabulating items, which is insufficient for a clean directional-causal claim.
Subset: wrong-post-cutoff + refusal items. Judge: Qwen3-1.7B. Calibrated range: , an order of magnitude smaller than the Qwen3 and Gemma ranges, calibrated to where Llama’s outputs remain coherent; the post-block-27 residual has (Table 13), so the swept range reaches the state norm and never entered the sub-norm regime.
The wrong-subset row at the baseline () reads 0% refusal but 100% first-token-opener: items where the text reads refusal-y (“I’m not aware of…”) but commits mid-generation (“… However, I can tell you that…”); the Qwen judge labels these wrong because of the commit. The symmetric large- refusal-rate spike (83% at , 75% at ) reflects the model producing shorter, judge-accepted refusal-coded text but without using its standard opener vocabulary (first-token-opener collapses to 0% in either direction). This is a perturbation-magnitude effect, not a directional refusal-direction causal signal.
The clearest directional effect is the refusal subset at : refusal rate drops from 100% to 80%, three of fifteen previously-refusing items committing to a confabulation:
| id | gold | baseline () answer | answer |
| cin_pc_10 | Deadpool & Wolverine | “I’m not aware of any information about a Marvel film…” | “X-Men ’97” |
Table 23.
(Two other flips have similar structure; full per-item answers are in figures/llama_3_2_3b/13_intervention_results.json.)
Robustness checks
Robustness to PCA truncation depth
The headline probe pipeline uses PCA(n_components=16). Sweeping over at each target shows that the peak per-layer accuracies are stable within a pp band across the sweep (Figure 10):

Figure 10. Peak per-layer probe accuracy as a function of PCA , for each of the main probe targets. The paper default () is marked. All targets are stable across the sweep within a pp band, and several peak at , meaning the paper default is conservative rather than tuned.
The choice of is not driving any of the reported numbers; the prompt-feature finding of Section 5.4 in particular is independent of this hyperparameter. The full sweep data (target peak-accuracy table) is at figures/qwen3_1_7b/10_pca_robustness.json, generated by analysis/make_robustness_check.py, which takes no arguments and reads the same cached responses and activations as 03_analyze.py.
SAE base-to-instruct reconstruction check
Qwen-Scope was trained on Qwen3-1.7B-Base; the Section 6.3 subject is the Instruct variant. The release declares its input contract explicitly: hook_blocks.27.hook_resid_post with normalize_activations = none, i.e. the raw post-block-27 residual. On that state, recomputed over all 549 refusal+wrong items via a live prefill hook (analysis/cache_prenorm_states.py), the base→instruct transfer is marginal by reconstruction quality: per-item explained variance 0.50 (median 0.51) and mean cosine 0.72 (figures/sae_decompose_prenorm.json). An earlier check (saes/sae_test_reconstruction.py) reported EV = 0.82/cosine 0.90, but it was run at cached hidden-states index 24, a different (post-block-23) residual, and HF’s cached index 28 is the final normed state rather than the declared hook (Section 4), so neither cached index measures the SAE at its contract. The decisive validation of the Section 6.3 features is therefore not reconstruction quality but the causal test of Section 6.3.1, which intervenes at the correct point and flips 30/30.
PCA coverage and blend tests of feature 2191
How much of feature 2191’s decoder direction the probe’s reachable subspace can represent, and whether the feature can improve the probe’s split (Section 6.3). The dictionary that contains the feature was estimated without ever seeing ConfabQA; that it nonetheless contains the exact opener feature this benchmark activates is evidence 2191 is a general unit of the model rather than an artifact of this question set. Coverage: the pipeline’s 16-component subspace holds at most a 0.37 cosine with (the recovered probe sits at 0.16); half coverage takes roughly 60 variance-ordered components, and even the full 549-dimensional span of the benchmark’s activations covers only 0.79. Blend: under 5-fold CV, mixing the unit probe direction with the feature leaves the refusal-vs-wrong AUC flat at 0.944 for mixing weights up to 0.2 and degrades monotonically beyond, and a two-feature logistic stack over both projections also lands at 0.943; the feature’s out-of-subspace mass carries no class information the probe has not already extracted. The raw projection onto 2191 alone reaches AUC 0.895, close to the probe’s 0.944: the single dictionary direction carries nearly probe-grade class information in the probed representation.
Additional refusal-direction diagnostics
Score histograms (Section 6.1). Figure 11 shows the signed projections of the layer-28 hidden states onto the recovered refusal direction.
Refusal-direction lens scores (§ 6.1). The lift is the gradient of the probe score with respect to the raw state, the hidden-state direction along which the score increases fastest; raw units matter because the model’s RMSNorm/LM head and the § 6.2 intervention hook know nothing about the scaler. The lens scores are relative alignment scores, not log-probabilities: with tied embeddings, (token row , RMSNorm gain , the direction divided by its root-mean-square). Full subset (): as leads at +19.22, the Chinese refusal openers 作为一个 and 作为 (“as” / “as a”) sit at ranks 2–3 (+17.67, +17.14, with further variants at ranks 12–13), the rest of the top ten are case and spacing variants, and there at rank 11 scores +14.16. The within-post recovery’s lens yields the same vocabulary in a sharper form: ” as” scores +31.66, “作为” +29.63, “\tas” +27.58, ” there” +23.26. The confabulation pole of the refusal direction is dominated by generic low-frequency tokens with no thematic structure, as expected: a confabulation can begin with any entity name, so there is no concentrated “confabulation vocabulary.” The layer-18 correctness direction under the same lens yields generic low-frequency fragments at the correct pole and at most a faint uncertainty flavor (不知 “don’t know”, unknown, forgotten) at the incorrect pole.
Recovery-method check: difference in class means (§ 6.1). The raw difference of class means, , at the same layer is not the probe direction: cosine 0.46 on the full subset (0.36 within post-cutoff, against a chance level of in 2048 dimensions), with only one of its top-ten lens tokens (as itself, ranked first) overlapping the probe’s. Its lens instead mixes the opener with a temporal-currency vocabulary (current, currently, present, 我没有 “I don’t have”), which dominates outright within post-cutoff. The two recoveries weight different components: the mean difference absorbs topical and temporal-currency content, the discriminatively trained probe concentrates the register-and-opener component. A plain average of refusal states, as a control, points at nothing but generic sentence-starters (As, The, There, In).
Figure 1 shows the geometry behind the disagreement: the mean difference is by construction the centroid connector, with 96% of its norm in the top-two PC plane where the wrong class’s covariance is widest, while the probe direction keeps only 47% of its norm there, approximating instead the covariance-whitened difference that separates the classes item by item.
The probe–2191 plane (§ 6.3.1). Figure 12 shows the plane spanned by the probe refusal direction and feature 2191’s decoder vector.
From h-space to a-space: what a push looks like in features (§ 6.3). The three views of Section 6.3 are static, and the encoder is heavily nonlinear (TopK), so how an h-space push maps into a-space is an empirical question. Figure 13 tracks four ensemble members’ mean activations on the wrong items as the post-block-27 residual is pushed along the Section 6.2 direction and along . Three regularities emerge. First, 2191’s activation grows smoothly and linearly under both pushes (from its baseline of 93 on wrong items to full coverage by on the probe push and +200 on its own), with a probe-to-2191 slope ratio of 0.25, matching the two directions’ cosine of 0.25: the TopK encoder is locally linear along these rays. Second, the two pushes differ in what else they recruit: the probe push feeds the whole register (17077 rises to a mid-dose peak, 4314 grows steadily), while the pure 2191 push crowds the hedge feature out entirely (4314 falls to 0 by +3000). The probe direction moves wrong items toward refusal-like codes; the 2191 direction manufactures opener-dominated ones; behaviorally the two are indistinguishable (Appendix § 13.3), so the ensemble recruitment is not required for the flip. Third, the code as a whole is substantially rewritten at behavioral doses: active-set overlap with the unperturbed code falls to 0.36 (probe) and 0.26 (2191) at , and negative doses extinguish all four features.
Cross-model comparison table and per-layer probe curves
Table 16 is the § 7.1 cross-model comparison summary. The correctness probe peaks in the same early-to-mid depth band on all three architectures, but the absolute peaks hide the confound § 5.4 exposed: a large fraction of what looks like model-internal correctness knowledge is the annotator-visible difficulty signal being read back through the hidden state, and the margins remain modest within the pre-cutoff stratum.
Refusal-vs-wrong probe under class imbalance (Section 5.3)
Bare accuracy (0.894) is misleading for the refusal_vs_wrong target because refusals are 147/549 = 26.8% of the wrong+refusal subset: a constant predictor of wrong already scores 0.732. At the peak layer (28) the class-imbalance-aware metrics are: confusion matrix [[TN, FP], [FN, TP]] = [[369, 33], [25, 122]], balanced accuracy 0.874, recall on refusals 122/147 = 0.830, recall on wrong 369/402 = 0.918, ROC AUC 0.944. The probe is not defaulting to “wrong”: it flags 122 of 147 refusals at a false positive rate of 33/402 ≈ 8.2%, and the AUC shows it ranks refusals well above confabulations in score order.

Figure 13. -space trajectories of -space pushes ( wrong items, pre-norm residuals). Solid lines: mean activation of four ensemble features as the state is pushed along the Section 6.2 refusal direction (left) or feature 2191’s decoder direction (right); dotted: mean overlap (Jaccard) of the active feature set with the unperturbed code. The probe push recruits the register broadly; the 2191 push concentrates on the opener and crowds out the hedge.
Margins are computed from unrounded values, so displayed operands can differ in the last digit (e.g. Llama correctness: 94.259 − 91.837 = +2.4). For Llama’s within-pre cell the majority baseline (80.7%) exceeds the strongest prompt-feature baseline (79.4%), so the probe adds +4.4 pp over the free floor rather than the +5.8 pp shown over the prompt baseline.
SAE feature-ranking views and the wrong-pole search
The three Section 6.3 views, for each of the 32,768 features: (A) direct encoding, the features the encoder allocates to the direction placed at typical state magnitude (; the encoder is scale-sensitive); (B) decoder alignment, per feature, independent of the encoder nonlinearity; and (C) empirical activation differential, the standardized gap between mean activation on refusal items and wrong items, independent of the recovered direction. Convergent features are the most interpretable. Ensemble-member detail behind the Section 6.3 roster: the moment hedge 4314 (diff-z +2.15, the strongest discriminator) fires on 82% of refusals vs 12% of wrongs, and its max-activating prompts are post-cutoff prize questions the model refuses; officialese 17077 (hereby, respectfully, duly, pursuant) fires on 97% of refusals vs 40% of wrongs; the polite-modality feature 14361 reads may, Please, Hopefully; the first-person feature 8875 reads I, 我; 14034 is a register feature whose output vocabulary always loses the argmax race to “As”; and 2191’s max-activating prompts are the recent-date items the model declines (Nov 2024 Gwen Stefani album, the Wicked adaptation, Taylor Swift album dates).
The strongest wrong-vs-correct feature in the § 6.3 mirror search is a topic cue, not a commitment marker: an award-vocabulary detector (lens winning, 荣获, 获得) firing on 22% of wrong items vs. 1% of correct and 0% of refusals, whose max-activating prompts are the obscure Best Picture years of the model’s weakest cell; it tops the disconfounded within-pre ranking too (diff-z +1.34). The runners-up are of the same species (a Nobel-Prize topic feature, an “In [date]” phrasing feature, a question-form feature), and against refusals the top wrong-side features are attempt and formatting features that fire on correct and wrong items alike. What the search rules out is a concentrated confabulation marker in this dictionary, the counterpart of 2191.
Bootstrap protocol and attribution details
Supporting protocol and accounting detail for Sections 8.1–8.3.
Table 17 is the § 8.1 bootstrap table in full; Figure 16 is its forest plot.

Figure 16. Forest plot of the Table 17 bootstrap ranges on across 14 (dataset, model, target) cells. balanced 50/50 subsamples per cell; blue = range excludes 0. The bottom two rows are the headline positive finding: Llama 3.2 3B on PopQA / TriviaQA recovers +21–+25 pp over the strongest prompt-feature baseline, where Qwen3-1.7B and Qwen3-4B on the same data recover +4–+10 pp. The Qwen3-4B “scaling control” row rules out parameter count as the dominant driver of the cross-model gap.
Bootstrap protocol detail (Sections 4, 8.1). Per cell, ; each subsample’s is probe peak − max(baselines) in %, the reported is the mean across , and the reported interval is on the sorted, zero-indexed values. At
these are the minimum and maximum, so the paper calls it a range rather than a 95% CI. is set by compute (each replicate refits the full 29-layer 5-fold pipeline), and the range is correspondingly coarse. The protocol is “resampling-of-balanced-subsamples” rather than a true item-level bootstrap; the latter would require refits per cell, too slow on M1-class hardware (see the evaluation-methodology limitation in § 10).
Permutation-null and item-level-bootstrap calibrations (§ 8.1). Two protocol checks calibrate the Table 17 ranges (artifact small_cell_stats; fixed layer, TF-IDF baseline, so absolute values are not comparable to the table). First, a label-permutation null of the peak-over-all-layers margin measures the layer-selection bias directly: null mean pp, 95th percentile pp on both models, so margins in the pp range (e.g. Llama ConfabQA correct, ) should not be read as evidence, while the all-items margins sit far above the null (). Second, an item-level bootstrap () keeps the Llama all-items margin positive (CI excluding zero) but leaves both Llama within-stratum cells consistent with zero (within-pre [−2.0, +8.5], within-obscure [−3.3, +11.8]): the subsample protocol’s positive ranges for those two cells understate item-sampling uncertainty and should be read as point estimates.
External-dataset sampling (§ 4). Both external datasets run through the 02_evaluate.py generation pipeline. PopQA: 14k Wikidata triples templated into natural-language questions; sampled stratified by o_pop (object popularity, 5 quintile bins of 160 items each), reshuffled at seed . TriviaQA: mandarjoshi/trivia_qa unfiltered.nocontext validation split; sampled uniformly, reshuffled at seed . Generation settings match ConfabQA: greedy decoding, enable_thinking=False, last-prompt-token hidden-state capture at every layer.
Transfer-grid mechanics (§ 8.3). The fitted pipeline is transferred with its scaler and PCA held fixed at the source-dataset statistics; only evaluation touches the target dataset. The sweep uses a fixed four-layer grid per model (Qwen , Gemma , Llama ); the § 8.3 headline rows report the grid layer closest to each model’s Table 16 (PDF p. 30) peak (Qwen 18, Gemma 13, Llama 14), and the full grids are in the released matrices. A prompt-feature-only baseline (TF-IDF on question text) is fit on the same splits as a control: if the hidden-state probe transferred worse than the prompt-feature classifier, the hidden state would have contributed nothing portable beyond what the question text already encoded.
Transfer results in full (§ 8.3). Table 18 is the headline retention summary; the per-model readings follow.
Llama: content-invariant. Five of six off-diagonal transfers beat the test-dataset majority, with an average winning margin of pp, comparable to the within-dataset pp. The single failure is ConfabQA PopQA (majority 87.3%, transfer 79.8%). Two transfer cells (PopQA ConfabQA at 94.0%, TriviaQA ConfabQA at 93.8%) exceed the within-ConfabQA CV accuracy of 93.5%, suggesting less overfit when trained on more diverse data than ConfabQA alone provides.
Gemma: strong within, collapsed transfer. Gemma has the largest average within-dataset margin of the three at this layer but the worst transfer. Two cells collapse catastrophically, both with PopQA as target: ConfabQA PopQA at 38.3% (vs majority 80.2%, a pp gap) and TriviaQA PopQA at 46.8% ( pp). Four cells do beat majority (transfers to ConfabQA and TriviaQA), average winning margin pp, well below Llama’s ; the net signed average is pp, dominated by the PopQA collapses.
Qwen: weak within, weak transfer. Qwen3-1.7B’s within-dataset margins are the smallest to begin with ( pp average); three of six transfers beat majority, all by small amounts ( to pp). Its signed transfer average ( pp) is closer to zero than Gemma’s not because Qwen transfers better but because an almost flat probe stays almost flat under transfer (retention ).
Refusal census (§ 8.2) and Qwen3-4B raw accuracies (§ 8.1). Pool-level refusal rates behind the § 8.2 anchor: Llama PopQA 62.9% vs. Qwen 1.7% (39/2264 refusals in the pooled three-seed Qwen sample); Llama TriviaQA 31.5% vs. Qwen 1.2% (28/2242). The four attribution cells are Qwen × {PopQA, TriviaQA} and Llama × {PopQA, TriviaQA}. Qwen3-4B’s PopQA judge breakdown (14.8% correct / 1.6% refusal / 83.6% wrong) is essentially Qwen3-1.7B’s, and the family’s ≈ 2% refusal rate versus Llama’s 63% on the same items is the conspicuous family-level difference.
Refusal-channel attribution: full results (Section 8.2). Table 19 gives both tests in full.
(Italicized lower bound: the range includes 0 because the refusal count is too small for the bootstrap to be well-resolved.)
Reading Test A. Comparing the drop-refusals rows to the unfiltered Table 17 rows: for Llama, +24.94 → +20.76 on PopQA and +21.25 → +17.71 on TriviaQA, i.e. the refusal channel accounts for ≈ 17% of the headline and the bulk (≈ 83%) is real correct-vs-wrong discrimination on attempted items. The refusal-channel hypothesis is partially right (about a sixth of the signal) but cannot account for the gap with Qwen, which would require nearly all of it. The Qwen rows are essentially unchanged (+4.35 → +3.31 PopQA, +9.57 → +9.36 TriviaQA): Qwen refuses too rarely on these benchmarks for refusal-readout to be a substantial confound either way. One caveat on the ratio: its numerator is judge-labeled and its denominator substring-labeled (Appendix § 13.5). A label-sensitivity rerun (single pipeline, both regimes; artifact small_cell_stats) shows the contrast is not a label artifact: under judge labels Llama’s TriviaQA margin grows (+17.5 → +23.8 in that pipeline) and the Llama-vs-Qwen ordering is unchanged on both external datasets.
Reading Test B. Llama’s refusal probe reaches 84.5% (PopQA) and 90.2% (TriviaQA) absolute accuracy on a balanced 50/50 task, beating the strongest prompt baseline by +18.4 and +25.4 pp, ranges excluding 0 by wide margins: the hidden state encodes the abstention decision in a form the question text does not predict. Qwen’s analogous margins (+10.2 pp PopQA, +16.0 pp TriviaQA) are positive but the refusal counts (39 and 28) leave the ranges straddling 0; this test needs either a larger Qwen evaluation or a benchmark on which Qwen3 refuses more (Qwen refuses 37.4% on ConfabQA’s post-cutoff stratum).
Label provenance (Section 8.1). For the ConfabQA cells the correct field is the three-way judge’s label (Section 5); for the external-dataset cells it is the generation-time substring match against gold and alternatives (the judge pass post-dates these bootstrap runs). Section 8.2’s attribution tests use the judge labels instead, which is why their per-class counts differ from Table 17’s (e.g. Llama PopQA: 131 substring-correct vs. 102 judge-correct).
Accuracy and baseline accounting for the Section 8.1 contrast. PopQA substring-correct rates are nearly identical (16.4% Llama vs. 15.6% Qwen); on TriviaQA Llama is more accurate (44.5% vs. 27.4%), but 50/50 balancing removes any base-rate advantage. Llama’s strongest PopQA dropped-refusals prompt baseline is 61.6% vs. Qwen’s 75.9%, so a lower baseline floor does not explain the gap either.
Extended discussion
Extended form of the Section 9 discussion.
Refusal findings. The refusal direction’s logit lens recovers the literal refusal-opening vocabulary in each model (Qwen3: as, As, 作る; Gemma: Regret, Formal, Alas; Llama: I, Given, As), the one-shot intervention drives the first-token opener rate to 100% on Qwen3 and Gemma (Sections 6.2, 7.2), and the Section 8.2 refusal probe extends the finding to PopQA and TriviaQA at Llama scale; the claim “refusal is a clean, late-network, linearly-decodable direction that causally controls the opener in these three small LMs” survives the bootstrap and the external benchmarks, and is no supervised artifact (visible in unsupervised PCA; Figure 5b, Figure 6). The Qwen3-1.7B correctness direction fails all three corresponding tests: little margin over the difficulty oracle (Table 7), no token-space signature, no sign-directional causal effect (§ 6.2).
Two atlases on Llama vs. one on Qwen. A plausible mechanism: Llama’s heavier calibrated-abstention training requires a distinguishable internal “I should hedge” state (the Test-B gap) and a richer correctness representation as its prerequisite (what Test A recovers); Qwen3-1.7B, which prefers to attempt-and-confabulate, never had pressure to develop either. The Qwen3-4B scaling control supports this: same family, same near-zero PopQA refusal rate, essentially at Qwen3-1.7B’s level and far below Llama’s. The mechanism is family / post-training recipe, not scale; a fully clean ablation would still need a recipe swap holding family fixed, which is future work.
Consequence for the probing literature. Probing papers reporting “the hidden state predicts correctness at –” should be re-evaluated against a prompt-feature baseline (removing the cutoff variable is necessary but not sufficient), and the result is model-dependent: a single-model study could report either “substantial factual self-knowledge” (Llama) or “nothing over prompt features” (Qwen ConfabQA within-obscure) on identical data and code, consistent with Orgad et al. [18]’s finding that truthfulness encoding is multifaceted rather than universal. Small-LM claims should be checked on at least two architectures with different post-training recipes, with bootstrap CIs on balanced subsamples rather than single-seed point estimates. Finally, the refusal direction is a structure of the continuous representation space that becomes text one token later; a natural extension is to measure calibration directly in that space (signed distances to the refusal and correctness hyperplanes as continuous confidence scores).
References
- [1]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR Workshop, 2017. arXiv:1610.01644.arxiv.org/abs/1610.01644
- [2]Andy Arditi, Oscar Obeso, Aaquiib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In NeurIPS, 2024. arXiv:2406.11717.arxiv.org/abs/2406.11717
- [3]Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of EMNLP, 2023. arXiv:2304.13734.arxiv.org/abs/2304.13734
- [4]David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satyik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. In NeurIPS, 2025. arXiv:2409.14507.arxiv.org/abs/2409.14507
- [5]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630, 2024. doi:10.1038/s41586-024-07421-0.DOI
- [6]Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do I know this entity? Knowledge awareness and hallucinations in language models. In ICLR, 2025. arXiv:2411.14257.
- [7]Leo Gao, Tom Dumpré la Tour, Henk Tillman, Gabriel Goh, Rajan R. Toll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025. arXiv:2406.04093.arxiv.org/abs/2406.04093
- [8]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [9]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. arXiv:1705.03551.arxiv.org/abs/1705.03551
- [10]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint, 2022. arXiv:2207.05221.
- [11]Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint, 2024. arXiv:2406.15927.
- [12]J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174, 1977. doi:10.2307/2529310.DOI
- [13]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In NeurIPS, 2023. arXiv:2306.03341.
- [14]Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. In BlackboxNLP, 2024. arXiv:2408.05147.
- [15]Jack Lindsey et al. On the biology of a large language model. Transformer Circuits Thread, 2025. transformer-circuits.pub.
- [16]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL, 2023. arXiv:2212.10511.
- [17]Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In COLM, 2024. arXiv:2310.06824.
- [18]Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. arXiv preprint, 2024. arXiv:2410.02707.arxiv.org/abs/2410.02707
- [19]Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In ICLR (Outstanding Paper), 2025. arXiv:2406.05946.