betaFeedback

WorksComputer Science

MAxBench: A Multinomial Concept Recovery Benchmark

Abstract

Fine-grained control of language model behaviors (e.g., steering) is among the more actionable outcomes of interpretability research. For binary concepts such as refusal, a single direction in activation space often suffices for steering. However, many concepts are not binary: Animals and Countries contain many subcategories, each with multiple instances. For these concepts, the search space over possible representation geometries is far larger than for binary concepts; it is thus not clear what geometries are most appropriate, nor what methods are most effective at recovering them. In this work, we introduce MAxBench, a geometry-agnostic evaluation framework for multinomial concept representations based on sampling from the recovered concept representation. We use MAxBench to compare 10 localization methods (covering 5 geometry types) across 6 concepts and 4 models. Using this framework, we find that (i) affine subspaces steer more reliably and have greater recall than rank-one or linear subspaces; (ii) much of this advantage is due to better non-zero offsets rather than the choice of bases; (iii) manifold steering is competitive with the best methods when applicable; and (iv) no method consistently outperforms prompting, in alignment with prior findings on binary concepts. These findings underscore the importance of expanding the scope of interpretability research and meta-evaluation to concepts with more varied structure.

Introduction

Among the most impactful ideas in interpretability is that some human-understandable concepts can be represented as linear directions in activation space. This simple geometric abstraction has enabled both insights into model internals ([?]; [5]; [?]; [?]; [7]; [?]; [?]) as well as techniques for modifying model behavior at inference time. Steering methods ([?]; [?]; [?]; [?]) provide a lightweight and interpretable alternative to prompting and fine-tuning (albeit with somewhat lower performance than prompting ([?])). For binary concepts such as refusal ([3]), a steerable representation can be captured by a single direction in activation space that separates the two states. However, this intuition becomes less straightforward for more complex concepts, as these may require richer representations.

Indeed, recent evidence suggests that concepts in language models can take many forms: e.g., they may be polytopes ([?]) or may lie on multidimensional (and potentially non-linear) manifolds ([?]; [?]; [?]; [?]; [4]; [?]; [?]; [?]; [?]; [?]; [?]). With regards to evaluation and steering, moving from one-dimensional/binary to multidimensional/multinomial concepts therefore introduces several challenges. First, the space of possible geometries becomes much larger, and may depend on the concept. In the multidimensional setting, a concept may be represented by a set of directions, a Gaussian cloud within the representation space, linear or affine subspaces or non-linear manifolds, among many other possibilities. Second, the dimensionality (or rank) of a concept representation is generally not known in advance and likely varies significantly across concepts. While some concepts have an intuitive geometry (e.g., days of the week tend to form a circle with ordered values [?]), we cannot assume a priori that any given concept will have such a structure.

We therefore ask three questions. First, how can we measure and compare the quality of concept representations in a geometry-agnostic manner? Second, how can one steer a concept given arbitrary multinomial concept geometries? Finally, do there exist methods that generally tend to perform well for multinomial concept steering—and if so, why do they perform well?

To investigate these questions, we first study methods (both existing and novel to this work) for recovering multidimensional concept geometries. We then propose a sampling-based intervention method for steering multinomial concepts in language models.

In this work, we introduce MAXBENCH, a framework for evaluating and comparing representations of multinomial concepts. As part of MAXBENCH, we propose evaluation metrics and data for measuring the quality of a multinomial concept localization method. We define quality as encompassing exclusivity (whether the representation captures only the target concept, analogous to precision) and completeness (whether the representation fully covers the concept, analogous to recall). This framework is agnostic to chosen geometry.

Our main contributions are as follows:

  1. We propose a sampling-based steering framework for evaluating how well a given representation captures a target concept. This framework is geometry-agnostic, and can be applied to linear and non-linear representations.

  2. We introduce MAXBENCH, a benchmark that leverages this sampling-based framework to enable comparisons of concept representations across multiple concepts and geometric structures.

  3. We use MAXBENCH to compare 10 concept representation discovery methods across 6 concepts, 5 geometries, and 4 models. We find that (i) affine subspaces steer more reliably and have greater recall than rank-one or linear subspaces; (ii) most of this advantage is due to better non-zero offsets rather than the choice of bases; (iii) manifold steering is competitive with the best methods when applicable; and (iv) no method consistently outperforms prompting.

  4. We introduce affine generalizations of existing steering methods for recovering multinomial concept representations.

Preliminaries

Concept representations. We informally define a concept representation as a geometric object in a model’s activation space that captures the boundaries of and variance within a semantic concept, such as ANIMALS or DAYS of the week. Ideally, a concept representation should be both complete, i.e., all the information relevant to the concept can be extracted from this representation, and exclusive, i.e. information unrelated to the concept cannot be extracted from this representation. For example, if concept CC is represented by an affine subspace A\mathsf{A}, then all information required to recognize or generate text about CC (but no more) should be recoverable from A\mathsf{A}, while directions orthogonal to it should ideally contain no additional information specific to the concept. We formally define the type of geometries we consider below.

Geometries

We consider linear and affine subspaces (including existing one-dimensional linear geometries), as well as non-linear manifolds [?], [?], [?], [?]. For concision, we focus primarily on affine subspaces in the main text, as this is the most general linear geometry applicable to all concepts considered in our experiments.

Formally, we define an affine subspace A⊂Rd\mathsf{A} \subset\mathbb{R}^{d} of the form A={Lq+b∣q∈Rk}\mathsf{A} = \{Lq + b \mid q \in\mathbb{R}^{k}\} where L∈Rd×kL \in\mathbb{R}^{d \times k} is a matrix with orthonormal columns defining a kk-dimensional linear subspace, b∈Rdb \in\mathbb{R}^{d} is an offset vector, and vector q∈Rkq \in\mathbb{R}^{k} denotes the coefficients in the subspace basis. A linear subspace is an affine subspace where the offset bb is the origin.

We additionally study non-linear manifolds. In this work, we use non-linear manifold to refer to a manifold [?] that cannot be cast as an affine subspace of Rd\mathbb{R}^{d}. Unlike an affine subspace, which is globally linear, a non-linear manifold may exhibit curvature while remaining locally Euclidean. In this work, we study manifold representations that are obtained by spline-fitting as introduced in [?]; [?].

In their current form, these methods can only be applied to concepts with ordered values, such as DAYS and YEARS. Extending these methods to unordered categorical concepts like ANIMALS is non-trivial, so we leave evaluation of non-linear manifold methods on unordered concepts to future work.

Data

We assume that the concept geometries described above correspond to low-dimensional structures in the space of language model hidden activations from the residual stream at a given layer. Specifically, for a set of mm input tokens xi=1mx_{i=1}^{m} that occur just before a concept-related token, we observe hidden activation vectors {h(i)}i=1m\{h^{(i)}\}_{i=1}^{m}, where h(i)∈Rdh^{(i)} \in\mathbb{R}^{d}; dd is the dimensionality of the language model representations. We assume that each concept is represented by a low-dimensional structure of dimensionality rr, where r<dr < d, for example, an affine subspace Ac⊂Rd\mathsf{A}_{c} \subset\mathbb{R}^{d} or non-linear manifold Mc⊂Rd\mathcal{M}_{c} \subset\mathbb{R}^{d} embedded in this ambient space. We will use “concept recovery” to refer to methods used to locate these geometric objects in language model activations.

Recovering a concept geometry requires observing a concept across diverse contexts.

In realistic settings, we may have activations for only a subset of instances belonging to a concept. For example, when learning the geometry corresponding to the concept ANIMALS, it is tedious and likely unrealistic to collect a dataset containing the complete set of all possible instances and corresponding activations. A good concept recovery method should nevertheless recover a geometry that generalizes well (i.e., contains instances not seen during concept recovery) in spite of incomplete data.

Based on these properties, we construct a dataset consisting of six concepts, chosen in part because they have been studied to understand the geometry of semantic representations in language models [?], [?] (ANIMALS, COUNTRIES, VEHICLES, PLANTS, DAYS, and YEARS) and a random baseline. Details of our dataset are given in Section 4.2.

We introduce MAxBench, a benchmark for evaluating completeness and exclusivity of concept geometries. The evaluation pipeline consists of three stages. First, we apply different concept localization methods to recover different concept-related geometric objects (Section 3.1). Second, we introduce a sampling-based evaluation where we sample points from the recovered object and use them to intervene on language model activations (Section 3.2). Third, we score the resulting outputs using metrics designed to measure the completeness and exclusivity of the recovered concept geometry (Section 3.3). An overview of our full evaluation pipeline is in Figure 1.

Pipeline for evaluating multinomial concept representations via the MAXBENCH dataset

Figure 1. A pipeline to evaluate multinomial concept representations via the MAXBENCH dataset. (a) For a given multinomial concept dataset we can localize a representing geometry in the activation space of a given LLM via different methods §3.1. (b) Neutral prompts generate concept-dependent continuations when steered via samples from a specific concept representation §3.2. (c) We evaluate both the exclusivity and completeness of different concept representations from a set of continuations that were generated for different samples ss and several neutral prompts. Taken together, several metrics define the final MAXBENCH score §3.3.

Concept Localization Methods

We consider methods for concept localization that differ in three key aspects: the geometry they assume, how the parameters (if any) are set, and supervision type. Learning approaches vary between degrees of supervised or unsupervised representation. If the method is supervised, the supervision type describes if the method uses only positive examples (i.e., valid instances of the concept) or positive–negative pairs (i.e., pairs of inputs where one contains a concept instance and the other does not). All methods we consider are summarized in Table 1.

MethodGeometryLabelsDataPairs
DiffMean-r1Direction✓Concept✓
ReFT-r1Direction✓Concept✓
DiffMeanAffine subspace✓Concept✗
ProbeAffine subspace✓Concept✗
Schatten ProbeAffine subspace✓Concept✗
PCAAffine subspace✗Concept✗
Factor AnalysisAffine subspace✗Concept✗
SAECollection of Directions✗Pile✗*
MFAMixture of Gaussians✗Pile✗*
Spline FittingNon-linear manifold✓Concept✗

Table 1. Comparison of concept localization methods evaluated in MAxBench. Labels indicates whether a method is supervised or unsupervised. Data describes the type of data used: concept refers to data specifically collected about the target concept (described in Appendix Table 5), whereas PILE ([?]) is general-purpose data containing broad information beyond the concept of interest. Pairs indicates whether both positive and negative examples are required. For example, a binary method such as DiffMean requires examples containing the concept (e.g., animal) as well as examples not containing it. *SAE and MFA are trained without contrastive supervision; pos/neg examples are used only for post-hoc feature/component selection.

Across all methods, the input is a collection of activation vectors {x(i)}i=1m\{x^{(i)}\}_{i=1}^{m}, with x(i)∈Rdx^{(i)} \in\mathbb{R}^{d}. Methods that require concept supervision additionally have labels y(i)∈{0,…,K−1}y^{(i)} \in\{0,\ldots,K-1\} for each activation. For one-dimensional methods, we use binary labels y(i)∈{0,1}y^{(i)} \in\{0,1\}, indicating the absence or presence of the target concept.

Linear

↗ One-dimensional methods. In the one-dimensional setting, concept localization methods aim to identify a single direction in activation space that captures variation associated with a target concept. A common approach is Difference in Means (DiffMean-r1) ([?]), which defines the concept direction as the difference between the mean activations of positive and negative examples defined as d=1∣{i:y(i)=1}∣∑i:y(i)=1x(i)−1∣{i:y(i)=0}∣∑i:y(i)=0x(i)d = \frac{1}{|\{i:y^{(i)}=1\}|}\sum_{i:y^{(i)}=1}x^{(i)}-\frac{1}{|\{i:y^{(i)}=0\}|}\sum_{i:y^{(i)}=0}x^{(i)}.

Another method proposed specifically to improve both steering performance and concept detection is rank-1 Representation Finetuning (ReFT-r1; [?]). ReFT-r1 learns a single direction dd from supervised positive and negative examples, where positive responses express the target concept and negative responses do not. The learned direction is used both to detect the presence of the concept in model representations and to steer the model toward it, with dd optimized through a language-modeling objective with additional sparsity regularization. Further details are provided in [?]. For one-dimensional methods we only have direction dd and offset b=0b=0.

▱ Affine methods. Affine concept recovery methods aim to localize a kk-dimensional affine subspace A={Lq+b∣q∈Rk}A = \{Lq+b \mid q \in\mathbb{R}^{k}\} rather than a single direction. The methods listed in Table 1 differ primarily in how LL and bb are estimated.

Across all methods, the input is a collection of activation vectors {x(i)}i=1m\{x^{(i)}\}_{i=1}^{m}, with x(i)∈Rdx^{(i)} \in\mathbb{R}^{d}. Methods that require concept supervision additionally have labels y(i)∈{0,…,K−1}y^{(i)} \in\{0,\ldots,K-1\} for each activation. For one-dimensional methods, we use binary labels y(i)∈{0,1}y^{(i)} \in\{0,1\}, indicating the absence or presence of the target concept.

↗ One-dimensional methods. In the one-dimensional setting, concept localization methods aim to identify a single direction in activation space that captures variation associated with a target concept. A common approach is Difference in Means (DiffMean-r1) ([?]), which defines the concept direction as the difference between the mean activations of positive and negative examples defined as d=1∣{i:y(i)=1}∣∑i:y(i)=1x(i)−1∣{i:y(i)=0}∣∑i:y(i)=0x(i)d = \frac{1}{|\{i:y^{(i)}=1\}|}\sum_{i:y^{(i)}=1}x^{(i)}-\frac{1}{|\{i:y^{(i)}=0\}|}\sum_{i:y^{(i)}=0}x^{(i)}.

Another method proposed specifically to improve both steering performance and concept detection is rank-1 Representation Finetuning (ReFT-r1; [?]). ReFT-r1 learns a single direction dd from supervised positive and negative examples, where positive responses express the target concept and negative responses do not. The learned direction is used both to detect the presence of the concept in model representations and to steer the model toward it, with dd optimized through a language-modeling objective with additional sparsity regularization. Further details are provided in [?]. For one-dimensional methods we only have direction dd and offset b=0b=0.

▱ Affine methods. Affine concept recovery methods aim to localize a kk-dimensional affine subspace A={Lq+b∣q∈Rk}A = \{Lq+b \mid q \in\mathbb{R}^{k}\} rather than a single direction. The methods listed in Table 1 differ primarily in how LL and bb are estimated. PCA ([?]) estimates LL as the subspace spanned by the top principal directions of the centered activations. Factor Analysis ([?]) estimates LL as the subspace spanned by the learned factor loadings. Linear Probe ([1]) first learns a classifier weight matrix WW from the activations. We then apply SVD to WW and define LL as the matrix containing its top singular vectors.

We propose a Schatten Probe ([6]; [?]) (or also referred to as Sparse Probe) as an extension to linear probe to encourage low-rank weights through approximate Schatten-pp norm regularization. Specifically, we parameterize the classifier weights as W=ABTW = AB^{\mathsf{T}} and jointly regularize corresponding columns of AA and BB. After training, we reconstruct WW, apply SVD, and define LL from the leading singular directions in activation space.

We also extend Difference in Means (DiffMean; [?]), where directions are constructed from pairwise differences between class-conditional mean activations. For each pair of classes ci,cjc_i, c_j, we compute

di,j=μci−μcj,μcj=1∣Xcj∣∑x∈Xcjx.d_{i,j} = \mu_{c_i} - \mu_{c_j}, \qquad\mu_{c_j} = \frac{1}{\lvert X_{c_j} \rvert}\sum_{x \in X_{c_j}} x.

where XcjX_{c_j} is the set of all activations that correspond to a particular class cjc_j. We stack these difference vectors to form DD, apply SVD, and define LL from its leading singular directions in activation space.

For all affine methods, we set the offset bb to the mean of the activations of positive examples; we leave other strategies for estimating the offset to future work. All affine methods are trained using only positive examples of the target concept.

Collection of Directions. Unlike the methods described above, which recover a single concept direction or a single affine subspace, sparse autoencoders (SAEs; [7]; [?]) learn a dictionary of feature directions fenc(x)=ReLU⁡(Lencx+benc)f_{\mathrm{enc}}(x) = \operatorname{ReLU}(L_{\mathrm{enc}}x + b_{\mathrm{enc}}). Each dictionary element corresponds to a one-dimensional direction in activation space. We use existing pre-trained SAEs for our experiments. To identify which features correspond to the target concept, we score each feature by the Matthews Correlation Coefficient (MCC) between its activations and a binary label indicating whether the point is related or unrelated to the concept, and retain the top-(k)(k) scoring features. Each selected feature defines a one-dimensional direction. We use the corresponding SAE decoder vectors as the columns of the recovered basis, L=[wf1∣⋯∣wfk]L = [w_{f_1} \mid\cdots\mid w_{f_k}] and sample from the subspace spanned by these directions. We set the offset to b=0b = 0.

Gaussian mixture. A Mixture of Factors Analyzer (MFA; [?]) learns a collection of low-dimensional Gaussian components, each associated with an affine subspace AcA_c. Each component cc has a centroid bc∈Rdb_c \in\mathbb{R}^d and factor-loading matrix Lc∈Rd×kL_c \in\mathbb{R}^{d \times k} and MFA models the activation space as {Ac}c=1C\{A_c\}_{c=1}^C. For a target concept, we select the component c∗c^* that is most associated with the target concept and use Ac∗A_{c^*} as the recovered concept representation. The method we use to select Ac∗A_{c^*} is described in Appendix F.

Non-Linear

Manifold Methods. Following [?], we first compute a centroid for each concept value and project these centroids into a low-dimensional subspace using PCA. A cubic spline is then fit through the projected centroids; this forms the concept manifold. We use the implementation of [?] to fit the spline and its associated centroids.

Further details about the methods are provided in Appendices B, G and F.

Sampling-based Evaluation

After we recover a concept representation using one of the localization methods described above, we next evaluate whether the points contained in that representation correspond to the intended concept. Intuitively, if a geometry faithfully captures a semantic concept, then points sampled from the geometry should decode only to valid instances of that concept (exclusivity) while covering the full range of values and categories contained by the concept (completeness). For example, samples drawn from the representation corresponding to the concept ANIMALS are used to intervene on a language model, the resulting generations should exclusively contain sentences related to animals. Furthermore, the samples should cover diverse instances of the concept rather than collapsing to a narrow subset (e.g., only mammals). We describe how we sample from different geometries below.

Sampling from an affine subspace. Given a recovered affine subspace Arecovered={Lc+b∣c∈Rk}\mathcal{A}_{\mathrm{recovered}} = \{L\mathbf{c} + \mathbf{b} \mid\mathbf{c} \in\mathbb{R}^{k}\}, we sample nn points {p(i)}i=1n\{\mathbf{p}^{(i)}\}_{i=1}^{n} from the subspace by drawing coefficients z(i)∼N(0,σ2Ik)\mathbf{z}^{(i)} \sim\mathcal{N}(0, \sigma^{2}I_{k}) and computing p(i)=Lz(i)+b\mathbf{p}^{(i)} = L\mathbf{z}^{(i)} + \mathbf{b}. Since affine subspaces are unbounded, we select σ\sigma empirically to ensure sampled points remain within the typical range of the model’s activation space. More details about how we select hyperparameters are in Appendix I.

A linear subspace is a special case of affine subspaces with b=0\mathbf{b} = 0. A one-dimensional representation corresponds to the special case r=1r = 1, making sampling with one-dimensional methods (e.g., steering vectors) a special case of sampling from an affine subspace.

Sampling from a non-linear manifold. Given a recovered manifold M\mathcal{M}, we sample nn points {p(i)}i=1n\{\mathbf{p}^{(i)}\}_{i=1}^{n} by defining a global parameter s∈[0,K)s \in[0, K), where KK is the number of concept values. Each integer kk corresponds to the centroid of the kkth concept value. For a sampled value s(i)∼U(0,K)s^{(i)} \sim U(0, K), we write s(i)=k+γs^{(i)} = k + \gamma, where kk identifies the corresponding spline segment and γ∈[0,1)\gamma\in[0, 1) specifies the position along the spline segment connecting centroids kk and k+1k + 1. We then evaluate this spline at s(i)s^{(i)} to obtain the sampled point p(i)\mathbf{p}^{(i)}. Repeating this procedure yields samples from the spline-based manifold. If the spline forms a closed loop, as in the case of days of the week, we include the spline segment connecting the final concept-value centroid back to the first.

Steering with samples. To decode each sampled point, we intervene on the language model’s internal representation using that point and observe the resulting model output. Interventions are applied at the same layer from which the concept representations are extracted. For a sampled point ss and the current hidden activation hh, we intervene by replacing it with

h′=(1−α)h+αs.(1)h' = (1 - \alpha)h + \alpha s. \tag*{(1)}

The sampling procedure determines the direction or location within the recovered concept geometry, while α\alpha controls the strength of the intervention. We select α\alpha using a grid search, with additional hyperparameter-tuning details provided in Appendix I. During generation, we exponentially decay α\alpha across decoding steps, encouraging the model toward the target concept early in generation while preserving fluency and instruction following.

Scores

Given the outputs produced by the sampling procedure above, we quantify how well each recovered geometry captures its target concept using the following metrics.

Concept Score. Following [?], we measure whether the generated output contains text relevant to the target concept. For example, if the target concept is ANIMALS, we check whether the output mentions any animal. For each sampled point, we assign a concept score in {0,1,2}\{0, 1, 2\}. A score of 0 indicates that the target concept is absent; 1 indicates that the concept is present but expressed unnaturally, or outside its intended meaning; and 2 indicates that the concept is clearly and correctly expressed. This quantifies exclusivity, and is analogous to precision.

Diversity Score. We introduce a diversity score to quantify how well a recovered geometric representation covers the intended concept. The score is computed in two stages. First, we extract the entities mentioned in each generated sentence (e.g., cat, sparrow) together with their associated category labels (e.g., mammal, bird for the ANIMALS concept). Second, we aggregate the extracted entities and categories across all generated samples and compute the normalized Shannon entropy at two levels: (i) instance-level, which measures the diversity of distinct concept instances in any category (e.g., dog, lion, tiger), and (ii) category-level, which measures how broadly the generations span higher-level semantic categories within the concept (e.g., mammal, bird, reptile). For details on the entropy computation and the nuances of its normalization, we refer to Appendix H. The final Diversity Score is the sum of the normalized (Section 4.3) entity- and category-level entropies, yielding a value in [0, 2]. A high diversity score indicates that the recovered representation covers the concept broadly, rather than collapsing to a small set of entities or semantic subcategories. This is analogous to recall.

MAXBENCH Score. Following AXBENCH [?], we also measure the fluency and instruction-following scores to validate whether steering does not cause degenerate outputs. The AXBENCH Score is then defined as the harmonic mean of the concept, fluency, and instruction-following scores. This captures whether an intervention successfully elicits the target concept while preserving fluent generation and adherence to the original instruction. We then define the MAXBENCH score as the harmonic mean of the concept, fluency, instruction-following, and diversity scores. The MAXBENCH score rewards representations that both steer reliably toward the target concept and capture a broad range of values of that concept.

Experiments

Models

We conduct experiments across three model families, including two model sizes from the Gemma family: Llama-3.1-8B ([?]), Gemma-3-1B and Gemma-3-27B ([?]), and Qwen-3.5-4B ([?]). All models are instruction-tuned.

For each concept dataset, we extract hidden activations from later-to-middle layers, since semantic information is well formed at that stage of the model layers and it has been shown to work well in prior work ([?, ?]) for interventions. Specifically, we use layer 19 of Llama-3.1-8B, layer 17 of Gemma-3-1B, layer 53 of Gemma-3-27B, and layer 27 of Qwen-3.5-4B. We leave evaluating different layers using our sampling evaluation framework to future work.3

Dataset

We construct a dataset of six concepts: ANIMALS, COUNTRIES, VEHICLES, and PLANTS are hierarchical and unordered. DAYS is categorical and ordered and YEARS is continuous and ordered. Hierarchical concepts are divided into categories. For example, ANIMALS includes categories such as mammals and reptiles etc, with each category containing multiple entities. Mammal category, for instance, includes entities like lion, beaver, and horse. For each entity, we use GPT-5 [?] to generate 100 sentences such that the sentence ends with the target entity. The sentences without the ground truth words at the end compose the inputs of our datasets. An example for the concept COUNTRIES from the dataset is as follows

Prompt: The country that contains the Great Pyramid of Giza is Egypt.

We evaluate whether the models from Section 4.1 can perform the target-token prediction task, i.e., whether the intended concept word (e.g., Egypt in the above example) is the most likely next token given the constructed prompt. Across all concepts, the correct entity appears among the model’s top-5 predictions with at least 70% accuracy. We also observe that larger models generally achieve higher prediction accuracy compared to smaller models. We provide detailed results for this prediction task in Table 7. Examples of the dataset construction and additional details are provided in Table 5 & Table 6.

ConceptPromptCorrect Continuation
ANIMALSThe big cat of genus Panthera referred to as the king of the beasts due to its mane and dominant pride structure is called alion
COUNTRIESThe country that contains the Great Pyramid of Giza isEgypt
DAYSThe day immediately following the weekend in many calendars isMonday
PLANTSThe crop most associated with Popeye gaining strength isspinach
VEHICLESThe aircraft that remains airborne through the rotation of large overhead blades is called ahelicopter
YEARSThe year when the COVID-19 pandemic spread across the world was2020
RANDOMIf several ice cubes are left on a kitchen counter until they completely melt, they becomewater

Table 5. Example prompts for each concept

ConceptCategoryEntities
ANIMALSMammalsbeaver, lion, giraffe, horse, koala
Reptileslizard, turtle, dinosaur, crocodile, gecko
Birdsparrot, owl, duck, eagle, penguin
Amphibiansfrog, salamander, toad, axolotl, newt
Fishsalmon, shark, pufferfish, clownfish, anglerfish
Insectsfirefly, dragonfly, butterfly, bee, grasshopper
COUNTRIESAfricaEgypt, Morocco, Kenya, Ethiopia, Madagascar
AsiaChina, India, Japan, Thailand, Vietnam
EuropeGermany, France, Italy, Spain, Portugal
North AmericaCanada, Mexico, Cuba, Jamaica, United States
South AmericaBrazil, Argentina, Chile, Colombia, Peru
OceaniaAustralia, Fiji, Samoa, Tonga, New Zealand
VEHICLESRoadcar, bus, truck, bicycle, ambulance
Railtrain, subway, funicular, monorail, tram
Airairplane, helicopter, drone, glider, rocket
Watercanoe, submarine, yacht, hovercraft, ferry, cruise
PLANTSTreesoak, pine, maple, pineapple, banana
Shrubsrose, lavender, lilac, boxwood, jasmine
Herbsbasil, dill, thyme, mint, spinach
Climbersgrape, passionflower, wisteria, cucumber, watermelon
Creepersivy, pumpkin, strawberry, sweet potato
ConceptEntities
DAYSMonday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday
YEARS1066, 1215, 1347, 1453, 1492, 1517, 1588, 1607, 1620, 1666, 1687, 1776, 1789, 1804, 1815, 1861, 1865, 1903, 1914, 1918, 1929, 1939, 1945, 1963
RANDOMwater, dog, coffee, bird, king, rain, sky, door, bus, train, window, tree, house, road, book, chair, computer, city, apple, bread

Table 6. For each concept, we define a set of categories.

ModelMetricAnimalsVehiclesCountriesDaysPlantsYearsRandom
Gemma-3-1BTop-10.4650.4580.7770.5460.4030.7330.684
Top-50.7070.7010.8920.9220.6680.9420.892
Cont0.4440.4350.7720.5240.3310.3750.704
Gemma-3-27BTop-10.8210.8750.9200.9590.8190.9480.868
Top-50.9420.9580.9630.9990.9620.9990.968
Cont0.9000.8720.9200.9590.8340.9000.879
Llama-3.1-8BTop-10.6860.7820.8900.9310.7030.8850.727
Top-50.8960.9310.9661.0000.9180.9660.933
Cont0.7050.7860.8880.9310.6490.8590.746
Qwen-3.5-4BTop-10.7350.7440.8810.9530.6380.9910.825
Top-50.9050.9410.9380.9990.8681.0000.958
Cont0.7640.7190.8740.9420.5850.9290.829

Table 7. Task performance across concepts and models. Accuracy measures whether the model predicts the target concept entity for each prompt. Top-1 reports whether the target entity begins with the model’s highest-probability next token, while Top-5 reports whether it begins with any of the five highest-probability next tokens. This prefix matching accounts for entities that may span multiple tokens. For example, if the target entity is United States, a prediction of the token United counts as a correct Top-1 match. Cont reports whether the complete target entity appears anywhere in the model’s greedy continuation of up to five tokens. For example, if the target entity is Egypt and the prompt is “The country that contains the Great Pyramid of Giza is”, Cont is correct if Egypt appears as the continuation. Results are averaged over the entire dataset for each concept.

Dataset Splits. In general, we do not assume that a concept dataset contains all possible entities associated with that concept. For example, ANIMALS may contain only a subset of all possible animals. We therefore evaluate each localization method across multiple dataset splits. This tests whether the method can recover a stable concept representation from different incomplete subsets of the data, without depending only on a few specific entities. For hierarchical concepts, we use two types of splits. In the first, we hold out an entire semantic category and recover the representation using the remaining categories. In the second, we retain all categories but randomly hold out individual entities across them. We generate 10 splits in total, with five splits of each type. For ordered concepts such as DAYS and YEARS, we instead randomly hold out a subset of concept values, such as particular days of the week or specific years.

Alpaca Eval. We use prompts from AlpacaEval [?] as inputs to the language model to perform interventions. For each prompt, we intervene on the model’s hidden activations using a sampled point from the recovered concept geometry and observe the resulting model output. We randomly select five validation prompts for hyperparameter tuning and evaluate on a disjoint set of five test prompts. For each prompt we sample 30 points from the recovered geometry and perform one intervention with each sampled point.

Ranks. For methods that recover multi-dimensional subspaces, we evaluate ranks k∈{1,2,4,8,16}k \in\{1, 2, 4, 8, 16\}. The precise meaning of kk depends on the localization method. For PCA, kk is the number of leading singular directions retained in LL. For factor analysis, it is the number of latent factors. For MFA, kk similarly denotes the number of latent factors per mixture component, and we train a separate model for each rank. For DiffMean (pairs) and Linear Probe, kk is the number of leading singular directions retained in LL. For SAE, kk is the number of selected SAEs whose decoder directions form the basis LL. For Schatten Probe, we define the rank using the effective rank [?] of the learned classifier weight matrix. DiffMean-r1 and RefTr-r1 recover a single direction and therefore always have k=1k=1.

LLM Judge

Given this set of steered outputs, we follow [?] by using large language model (LLM) judges to evaluate each steered output. We evaluate each method using the three metrics from AxBench: concept, fluency, and instruction following, together with the instance diversity and category diversity components of our proposed diversity score. For the AxBench metrics, the judge assigns each generated response a score from 0 to 2 for all three metrics. We use open-source model, Qwen-3.6-27B (instruction-tuned) as our LLM judge. All prompts we use for LLM judge are in Appendix Section M.

To evaluate diversity, we use an LLM to extract all concept instances mentioned across the steered outputs and, where applicable, assign each instance to a semantic category. For concepts that contain categories (ANIMALS, COUNTRIES, PLANTS, VEHICLES), we provide the judge with the category set used in our dataset (Table 6). Instances that belong to the concept but that did not occur in the training and do not belong to a predefined category are assigned to an "other" category (e.g. dragon). We then compute the normalized Shannon entropy of the resulting instance and category distributions, yielding the instance diversity and category diversity scores, respectively (see Appendix . For concepts DAYS and YEARS that do not contain categories we treat each instance as its own category.

Results

Linear Methods

Directions vs. Subspaces. We first ask whether multinomial concepts are better represented by a single direction or by a higher-dimensional subspace. Table 2 summarizes the performance of the different concept-localization methods. Multidimensional methods consistently achieve higher MAXBENCH scores than one-dimensional methods across concepts. This suggests that concepts with categorical non-binary structure are not well captured by a single direction, and instead benefit from subspace-based representations.

MethodConceptFluencyInstructionDiversityAXBENCHMAXBENCH
Prompting0.67 ± 0.342.00 ± 0.011.65 ± 0.050.90 ± 0.370.61 ± 0.320.98 ± 0.32
Isotropic1.12 ± 0.411.80 ± 0.141.47 ± 0.091.53 ± 0.270.84 ± 0.351.39 ± 0.29
Embedding0.85 ± 0.381.65 ± 0.331.48 ± 0.141.27 ± 0.410.60 ± 0.321.15 ± 0.39
DiffMean-r10.63 ± 0.201.33 ± 0.131.42 ± 0.061.42 ± 0.270.41 ± 0.171.04 ± 0.16
ReFT-r10.34 ± 0.231.42 ± 0.071.45 ± 0.061.14 ± 0.420.24 ± 0.180.68 ± 0.33
MFA1.37 ± 0.481.73 ± 0.191.40 ± 0.181.44 ± 0.310.91 ± 0.321.40 ± 0.17
SAE (top-k)0.63 ± 0.271.44 ± 0.281.45 ± 0.081.18 ± 0.270.43 ± 0.190.99 ± 0.24
Factor Analysis1.64 ± 0.261.86 ± 0.131.41 ± 0.111.52 ± 0.151.19 ± 0.241.58 ± 0.11
Schatten Probe1.55 ± 0.261.83 ± 0.141.42 ± 0.101.55 ± 0.181.13 ± 0.231.56 ± 0.12
Linear Probe1.50 ± 0.311.85 ± 0.131.44 ± 0.101.49 ± 0.201.12 ± 0.261.54 ± 0.13
PCA1.59 ± 0.291.94 ± 0.061.44 ± 0.111.51 ± 0.171.21 ± 0.241.58 ± 0.11
Diff Mean (Pairs)1.64 ± 0.271.87 ± 0.131.43 ± 0.101.53 ± 0.151.22 ± 0.251.59 ± 0.11

Table 2. Scores on the test set for Gemma-3-1B. We report Concept, Fluency, Instruction, and Diversity scores, together with the composite AXBENCH and MAXBENCH scores. Full metric definitions are given in Section 3.3. For each concept and method, we select the rank that achieves the highest mean MAXBENCH score. Each reported value is the mean ± standard deviation over all the data splits.

We next see how each method performs across ranks. Figure 2 shows the results for Gemma-3-1B across concepts. For the affine methods, MAXBENCH scores are generally stable over the range of ranks we evaluate, with slight improvement as rank increases for some concepts. This suggests that much of the benefit of these methods can be obtained with relatively low-dimensional subspaces,

MAXBENCH score by rank for Gemma-3-1B for all concepts

Figure 2. MAXBENCH score by rank for Gemma-3-1B for all concepts. The x-axis is rank (days is capped at rank 4, as DAYS has 7 values, and we assume the rank cannot exceed the number of concept values); the y-axis is MAXBENCH score; confidence intervals are computed via dataset split with 10 samples. Each point is the mean MAXBENCH score at that (concept, method, rank); the shaded band is std. dev. Prompting, Isotropic, DiffMean-r1, and ReFT-r1 have no rank hyperparameter, so they are drawn as dashed lines. For Schatten Probes, we measure effective rank instead of manually setting its rank; we round to the nearest integer.

provided that the representation is centered at an appropriate offset. Results for the other models are provided in Appendix Figure 5, Figure 6, and Figure 7.

MAXBENCH score by rank for Gemma-3-27B for all concepts

Figure 5. MAXBENCH score by rank for Gemma-3-27B for all concepts. The x-axis is rank (days is capped at rank 4, as DAYS has 7 values, and we assume the rank cannot exceed the number of concept values); the y-axis is MAXBENCH score; confidence intervals are computed via dataset split with 10 samples. Each point is the mean MAXBENCH score at that (concept, method, rank); the shaded band is std. dev. Prompting, Isotropic, DiffMean-r1, and ReFT-r1 have no rank hyperparameter, so they are drawn as dashed lines. For Schatten Probes, we measure effective rank instead of manually setting its rank; we round to the nearest integer.

MAXBENCH score by rank for Llama-3.1-8B for all concepts

Figure 6. MAXBENCH score by rank for Llama-3.1-8B for all concepts. The x-axis is rank (days is capped at rank 4, as DAYS has 7 values, and we assume the rank cannot exceed the number of concept values); the y-axis is MAXBENCH score; confidence intervals are computed via dataset split with 10 samples. Each point is the mean MAXBENCH score at that (concept, method, rank); the shaded band is std. dev. Prompting, Isotropic, DiffMean-r1, and ReFT-r1 have no rank hyperparameter, so they are drawn as dashed lines. For Schatten Probes, we measure effective rank instead of manually setting its rank; we round to the nearest integer.

MAXBENCH score by rank for Qwen-3.5-4B for all concepts

Figure 7. MAXBENCH score by rank for Qwen-3.5-4B for all concepts. The x-axis is rank (days is capped at rank 4, as DAYS has 7 values, and we assume the rank cannot exceed the number of concept values); the y-axis is MAXBENCH score; confidence intervals are computed via dataset split with 10 samples. Each point is the mean MAXBENCH score at that (concept, method, rank); the shaded band is std. dev. Prompting, Isotropic, DiffMean-r1, and ReFT-r1 have no rank hyperparameter, so they are drawn as dashed lines. For Schatten Probes, we measure effective rank instead of manually setting its rank; we round to the nearest integer.

Linear vs. Affine. We then ask how much of each method’s performance is attributable to the offset bb of the affine subspace, rather than the choice of bases LL. In Figure 3, for Gemma-3-1B, we observe that affine variants (with offset bb equal to the mean of concept-related points) achieve MAXBENCH scores than their linear counterparts (with b=0b = 0). This holds true for all the other models as well as shown in Appendix Figure 8, Figure 9, Figure 10. To assess LL’s contribution, we include an isotropic baseline that samples points isotropically around the concept centroid without constraining them to the span of the recovered basis vectors. Interestingly, for Gemma-3-1B, the isotropic baseline outperforms various linear and one-dimensional methods, and underperforms affine methods by a relatively small margin. Thus, both the offset and bases contribute significantly to effective concept localization; that said, assuming our sampling procedure, the choice of where to center one’s samples may be more important than the directions in which one moves around that centroid.

Comparison of multinomial concept localization methods for Gemma-3-1B with and without an offset

Figure 3. Comparison of multinomial concept localization methods for Gemma-3-1B with and without an offset (affine and linear, respectively). The x-axis shows the rank hyperparameter for each method, and the y-axis reports the MAXBENCH score. For the Schatten probe, we plot the effective rank of the learned weight matrix as the Schatten pp-norm parameter is varied from 0.1 to 0.9. For each method, we average the scores across all concepts and compute std. dev. via bootstrap. Sampling around the concept mean offset is consistently significantly better than assuming the subspace is centered at the origin. We observe similar trends for other models(Appendix Figure 8, Figure 9, Figure 10).

Comparison of multinomial concept localization methods for Gemma-3-1B with and without an offset

Figure 3. Comparison of multinomial concept localization methods for Gemma-3-1B with and without an offset (affine and linear, respectively). The x-axis shows the rank hyperparameter for each method, and the y-axis reports the MAXBENCH score. For the Schatten probe, we plot the effective rank of the learned weight matrix as the Schatten pp-norm parameter is varied from 0.1 to 0.9. For each method, we average the scores across all concepts and compute std. dev. via bootstrap. Sampling around the concept mean offset is consistently significantly better than assuming the subspace is centered at the origin. We observe similar trends for other models(Appendix Figures 8, 9, 10).

Comparison of multinomial concept localization methods for Gemma-3-27B with and without an offset

Figure 8. Comparison of multinomial concept localization methods for Gemma-3-27B with and without an offset (affine and linear, respectively). The x-axis shows the rank hyperparameter for each method, and the y-axis reports the MAXBENCH score. For the Schatten probe, we plot the effective rank of the learned weight matrix as the Schatten pp-norm parameter is varied from 0.1 to 0.9. For each method, we average the scores across all concepts and compute std. dev. via bootstrap. Sampling around the concept mean offset is consistently significantly better than assuming the subspace is centered at the origin.

Comparison of multinomial concept localization methods for Llama-3.1-8B with and without an offset

Figure 9. Comparison of multinomial concept localization methods for Llama-3.1-8B with and without an offset (affine and linear, respectively). The x-axis shows the rank hyperparameter for each method, and the y-axis reports the MAXBENCH score. For the Schatten probe, we plot the effective rank of the learned weight matrix as the Schatten pp-norm parameter is varied from 0.1 to 0.9. For each method, we average the scores across all concepts and compute std. dev. via bootstrap. Sampling around the concept mean offset is consistently significantly better than assuming the subspace is centered at the origin.

Comparison of multinomial concept localization methods for Qwen-3.5-4B

Figure 10. Comparison of multinomial concept localization methods for Qwen-3.5-4B with and without an offset (affine and linear, respectively). The x-axis shows the rank hyperparameter for each method, and the y-axis reports the MAXBENCH score. For the Schatten probe, we plot the effective rank of the learned weight matrix as the Schatten pp-norm parameter is varied from 0.1 to 0.9. For each method, we average the scores across all concepts and compute std. dev. via bootstrap. Sampling around the concept mean offset is consistently significantly better than that than assuming the subspace is centered at the origin.

For Gemma-3-1B, this suggests that correctly locating the center of the concept geometry may be at least as important as recovering its specific basis directions, and may provide indirect support for a perspective of concepts as prototypes ([?]b) over concepts as directions ([?]). The isotropic baseline performs substantially worse for the other models, suggesting that the centroid alone maybe insufficient and that the recovered basis directions also play an important role.

Prompting. We include a prompting baseline, in which we explicitly instruct the model to answer each AlpacaEval prompt while incorporating the target concept. Examples of these prompts are provided in Appendix L. This baseline measures how well the target concept can be elicited through prompting alone, without intervening on the model’s hidden activations. While prompting is not as competitive for smaller models such as Gemma-3-1B (Table 2), it outperforms all methods on larger models. Results are reported in Appendix Figure 8, Figure 10, Figure 11.

Comparison of SAE sampling strategies across models

Figure 11. Comparison of SAE sampling strategies across models. Each subplot corresponds to one model. The solid red line shows the combination method, where the top-kk SAE features are treated jointly as a basis for a subspace, while the dotted brown line shows independent sampling, where each sample is drawn along a single randomly selected feature direction. This is averaged over all 10 dataset splits and all concepts. Across models and ranks, combination sampling performs slightly better overall than independent sampling.

Embedding baseline. We also include a embedding baseline to show the performance of different methods when no context is provided. Representations are extracted directly from the embeddings of the concept’s instance tokens. This allows us to evaluate how much contextual information contributes to recovering a concept representation. Across models, we observe better performance when using representations from intermediate layers. This suggests that steering is operating over a space that contains more than the concept values’ token identities.

Non-Linear Methods

We include spline fitting as an additional method for the DAYS concept. Because this method requires an ordered set of concept values and (in its standard form) access to the complete sequence, we restrict its main evaluation to DAYS. We also evaluate spline fitting under incomplete observations by withholding selected days before fitting the spline, making this setting more comparable to the other concept localization methods, which operate with only a subset of concept instances. Results for DAYS are summarized in Table 3. Within this concept, spline fitting using the complete set of concept values outperforms spline fitting when some values are held out. More broadly, spline fitting

Gemma-3-1BGemma-3-27BLlama-3.1-8BQwen-3.5-4B
Prompting0.86 ± 0.101.87 ± 0.011.86 ± 0.021.80 ± 0.02
Diff Mean-r11.12 ± 0.051.33 ± 0.071.39 ± 0.090.55 ± 0.10
ReFT-r10.98 ± 0.151.33 ± 0.041.15 ± 0.120.73 ± 0.11
MFA1.34 ± 0.041.51 ± 0.030.32 ± 0.131.55 ± 0.04
SAE (top-k)0.61 ± 0.111.31 ± 0.071.40 ± 0.050.54 ± 0.09
Factor Analysis1.80 ± 0.021.84 ± 0.011.80 ± 0.011.72 ± 0.03
Schatten Probe1.78 ± 0.021.77 ± 0.021.75 ± 0.011.70 ± 0.01
Linear Probe1.77 ± 0.031.79 ± 0.011.80 ± 0.021.71 ± 0.03
PCA1.80 ± 0.031.79 ± 0.011.78 ± 0.021.70 ± 0.02
Diff Mean (Pairs)1.81 ± 0.021.76 ± 0.031.81 ± 0.011.71 ± 0.02
Spline (missing days)1.73 ± 0.011.84 ± 0.021.69 ± 0.061.64 ± 0.03
Spline1.83 ± 0.071.93 ± 0.011.71 ± 0.051.73 ± 0.03

Table 3. MAXBENCH score on the DAYS concept, across all four models. Per column, the highest mean is bold and the second highest is underlined.

@page 10/43 starts_mid=1 ends_mid=1 provided that the representation is centered at an appropriate offset. Results for the other models are provided in Appendix Figure 5, 6, and 7.

Linear vs. Affine. We then ask how much of each method’s performance is attributable to the offset bb of the affine subspace, rather than the choice of bases LL. In Figure 3, for Gemma-3-1B, we observe that affine variants (with offset bb equal to the mean of concept-related points) achieve higher MAXBENCH scores than their linear counterparts (with b=0b = 0). This holds true for all the other models as well as shown in Appendix Figure 8, 9, 10. To assess LL’s contribution, we include an isotropic baseline that samples points isotropically around the concept centroid without constraining them to the span of the recovered basis vectors. Interestingly, for Gemma-3-1B, the isotropic baseline outperforms various linear and one-dimensional methods, and underperforms affine methods by a relatively small margin. Thus, both the offset and bases contribute significantly to effective concept localization; that said, assuming our sampling procedure, the choice of where to center one’s samples may be more important than the directions in which one moves around that centroid.

For Gemma-3-1B, this suggests that correctly locating the center of the concept geometry may be at least as important as recovering its specific basis directions, and may provide indirect support for a perspective of concepts as prototypes ([?]) over concepts as directions ([?]). The isotropic baseline performs substantially worse for the other models, suggesting that the centroid alone maybe insufficient and that the recovered basis directions also play an important role.

Prompting. We include a prompting baseline, in which we explicitly instruct the model to answer each AlpacaEval prompt while incorporating the target concept. Examples of these prompts are provided in Appendix L. This baseline measures how well the target concept can be elicited through prompting alone, without intervening on the model’s hidden activations. While prompting is not as competitive for smaller models such as Gemma-3-1B (Table 2), it outperforms all methods on larger models. Results are reported in Appendix 8, 10, 11.

Embedding baseline. We also include a embedding baseline to show the performance of different methods when no context is provided. Representations are extracted directly from the embeddings of the concept’s instance tokens. This allows us to evaluate how much contextual information contributes to recovering a concept representation. Across models, we observe better performance when using representations from intermediate layers. This suggests that steering is operating over a space that contains more than the concept values’ token identities.

Non-Linear Methods

We include spline fitting as an additional method for the DAYS concept. Because this method requires an ordered set of concept values and (in its standard form) access to the complete sequence, we restrict its main evaluation to DAYS. We also evaluate spline fitting under incomplete observations by withholding selected days before fitting the spline, making this setting more comparable to the other concept localization methods, which operate with only a subset of concept instances. Results for DAYS are summarized in Table 3. Within this concept, spline fitting using the complete set of concept values outperforms spline fitting when some values are held out. More broadly, spline fitting achieves the highest overall performance among the methods evaluated on DAYS if the complete set of concept examples are provided.

Generalization beyond concept data

How well do the recovered concept representations generalize beyond the concept instances used to derive them? We provide the LLM judge with all concept instances and taxonomy labels observed in the training set (from the respective split) and ask it to classify each generated concept instance as either seen, if it appears in the training data, or unseen, otherwise. We count the total number of unseen instances and report results in Table 4. We find that many methods often generate unseen concept instances, indicating that the recovered representations can generalize beyond the examples provided during training rather than merely reproducing observed instances. Notably, the unsupervised methods, particularly Factor Analysis and MFA, produce the largest number of unseen instances. We provide additional results for this analysis in Appendix Table 13.

MethodAnimalsCountriesPlantsVehicles
Diff Mean (R1)41 ± 311 ± 349 ± 1029 ± 8
ReFT R16 ± 84 ± 211 ± 64 ± 3
MFA61 ± 1265 ± 846 ± 735 ± 15
SAE (top-k)36 ± 824 ± 959 ± 822 ± 8
Factor Analysis72 ± 726 ± 662 ± 2276 ± 5
Schatten Probe60 ± 1934 ± 767 ± 1655 ± 12
Linear Probe66 ± 728 ± 863 ± 961 ± 8
PCA64 ± 726 ± 656 ± 537 ± 8
Diff Mean (Pairs)72 ± 636 ± 766 ± 973 ± 7

Table 4. Per-concept unseen instance count for Gemma-3-1B, at each (method, concept)’s own best rank (the rank with the highest mean MAXBENCH score across bootstraps). Mean ± std is over bootstrap draws at that rank. All results are in Appendix Table 13.

ModelMethodAnimalsCountriesPlantsVehicles
Gemma-3-1BDiffMean-r141 ± 311 ± 349 ± 1029 ± 8
Gemma-3-1BReFT-r16 ± 84 ± 211 ± 64 ± 3
Gemma-3-1BMFA61 ± 1265 ± 846 ± 735 ± 15
Gemma-3-1BSAE (top-k)36 ± 824 ± 959 ± 822 ± 8
Gemma-3-1BFactor Analysis72 ± 726 ± 662 ± 2276 ± 5
Gemma-3-1BSchatten Probe60 ± 1926 ± 767 ± 1655 ± 12
Gemma-3-1BLinear Probe66 ± 728 ± 863 ± 961 ± 8
Gemma-3-1BPCA64 ± 726 ± 656 ± 537 ± 8
Gemma-3-1BDiff Mean (Pairs)72 ± 636 ± 766 ± 973 ± 7
Gemma-3-27BDiffMean-r143 ± 118 ± 330 ± 1235 ± 6
Gemma-3-27BReFT-r12 ± 68 ± 53 ± 69 ± 6
Gemma-3-27BMFA19 ± 425 ± 79 ± 42 ± 1
Gemma-3-27BSAE (top-k)57 ± 851 ± 646 ± 2856 ± 12
Gemma-3-27BFactor Analysis34 ± 612 ± 534 ± 622 ± 7
Gemma-3-27BSchatten Probe31 ± 715 ± 1022 ± 827 ± 8
Gemma-3-27BLinear Probe29 ± 420 ± 421 ± 922 ± 6
Gemma-3-27BPCA34 ± 633 ± 426 ± 622 ± 5
Gemma-3-27BDiff Mean (Pairs)35 ± 635 ± 630 ± 1020 ± 4
Llama-3.1-8BDiffMean-r129 ± 714 ± 410 ± 420 ± 4
Llama-3.1-8BReFT-r14 ± 77 ± 310 ± 64 ± 1
Llama-3.1-8BMFA64 ± 84 ± 240 ± 93 ± 1
Llama-3.1-8BSAE (top-k)34 ± 831 ± 414 ± 415 ± 7
Llama-3.1-8BFactor Analysis25 ± 611 ± 316 ± 729 ± 5
Llama-3.1-8BSchatten Probe26 ± 612 ± 615 ± 822 ± 5
Llama-3.1-8BLinear Probe22 ± 412 ± 48 ± 315 ± 3
Llama-3.1-8BPCA25 ± 321 ± 49 ± 518 ± 5
Llama-3.1-8BDiff Mean (Pairs)25 ± 612 ± 416 ± 619 ± 5
Qwen-3.5-4BDiffMean-r121 ± 38 ± 310 ± 430 ± 8
Qwen-3.5-4BReFT-r114 ± 55 ± 310 ± 33 ± 2
Qwen-3.5-4BMFA59 ± 1047 ± 754 ± 2128 ± 12
Qwen-3.5-4BSAE (top-k)28 ± 624 ± 7—9 ± 6
Qwen-3.5-4BFactor Analysis32 ± 814 ± 427 ± 1040 ± 9
Qwen-3.5-4BSchatten Probe34 ± 616 ± 721 ± 732 ± 9
Qwen-3.5-4BLinear Probe34 ± 59 ± 420 ± 929 ± 7
Qwen-3.5-4BPCA31 ± 815 ± 524 ± 922 ± 6
Qwen-3.5-4BDiff Mean (Pairs)32 ± 414 ± 724 ± 638 ± 8

Table 13. Highest mean OOD count ± std (across all dataset splits), per method and concept. Per concept column, the method with the highest mean is bold.

Metric Meta-evaluation

Does each metric (including the proposed diversity score) measure truly independent factors of performance, or do some factors strongly covary? To investigate, we compute Spearman correlations across all five metrics to assess how independent they are. We find (Figure 4) that the concept score is somewhat correlated with instance and category diversity. This is expected, as the diversity score can only be maximized when all generated outputs contain concept-related values. The concept score and instruction score are negatively correlated, suggesting that when the target concept is injected more strongly, it can become harder for the model to stay on topic with respect to the original instruction. Fluency scores have relatively weak correlations with other metrics.

Spearman correlation heatmaps for all the metrics used in our evaluation

Figure 4. Spearman correlation heatmaps for all the metrics used in our evaluation. This is computed over all steered outputs across models, methods, concepts, ranks, and splits. A more detailed analysis is provided in Appendix Figure 16.

Concept geometry. Early concept geometry studies operated under the Linear Representation Hypothesis ([9]), which holds that concepts are linear (and typically one-dimensional) subspaces of LM activations. More recent work has proposed that concepts may be multidimensional ([?]; [?]; [?]; [?]; [?]). This structure can be non-linear, and includes geometries such as convex hulls ([?]; [?]), polytopes ([?]), vector magnitudes rather than directions; [?], dense features ([?]; [?]) or manifolds ([?]; [9]; [?]; [?]; [?]; [?]; [?]; [?]; [?]; [?]; [4]; [?]; [?]). We focus extensively on affine subspaces, as they lie at an intermediate point of complexity: they enable one to go beyond single dimensions and beyond subspaces centered at the origin, but retain the simplicity and tractability of linear methods.

Evaluating concept representations. Given a concept representation, how can one verify its quality? Prior work has proposed correlational and causal metrics for evaluating their quality. Some evaluate geometric goodness-of-fit (Tiblias et al., 2026), whereas others evaluate counterfactual interventions on downstream behavior ([8]; Ding et al., 2026; Engels et al., 2025; Mueller et al., 2025; Shafran et al., 2026; Wu et al., 2025; Lee et al., 2026). Others still use toy settings with known ground truths (Gupta et al., 2024). These works evaluate whether a concept was leveraged after steering, but not whether the full concept space was covered; our diversity score fills this gap. These works also focus largely on binary concepts; our work specifically focuses on multinomial concepts with varied internal structure.

Multidimensional/Multinomial concept steering. Most steering approaches assume that concepts correspond to single directions in activation space, and thus intervene by adding a scaled steering vector to the hidden state (Li et al., 2023; Zou et al., 2025; Turner et al., 2025; Subramani et al., 2022; Marks & Tegmark, 2024; You et al., 2026). Engels et al. (2025) instead steer by replacing the angular coordinate corresponding to one value (e.g., Monday) with another in a circular latent space, and then reconstruct the activation; this alters the model’s predictions on periodic variables like days of the week. Shafran et al. (2026) propose steering based on a Mixture of Factor Analyzers (MFA) decomposition of activation space, which models activations as belonging to local Gaussian regions with associated low-dimensional subspaces. This representation enables two forms of intervention: steering activations toward a component centroid to move the model toward a concept region, and modifying the coordinates within the component’s local subspace to control finer semantic variations. Our work builds on this growing recognition that concepts in language models often exhibit intrinsically multi-dimensional structure. We propose a new evaluation framework to evaluate steering on multinomial concepts, which has only preliminarily been explored in prior work (although see concurrent work by Wurgaft et al., 2026, whose method we include in our evaluation).

Conclusion

In this work, we propose a sampling-based steering framework for evaluating multinomial concepts. Based on this framework, we propose MAXBENCH to benchmark multinomial and multidimensional concept localization methods. Using MAXBENCH, we observe that (i) affine subspaces more reliably recover such concepts than rank-one or linear subspaces; (ii) much of this advantage is due to better non-zero offsets rather than the choice of bases; (iii) manifold steering is competitive with the best methods when applicable; and (iv) no method consistently outperforms prompting when models are sufficiently capable. These findings highlight how interpretability research may benefit from studying a broader range of concepts, and from developing methods that better capture their underlying geometry.

Limitations and Future Work

Most of our experiments rely on LLM judges, which do not always agree with human judgments. Second, our experiments are limited to a single intervention layer. Our sampling procedure also relies on tuning hyperparameters for reliable steering and future work could explore better ways to sample from geometries without relying on hyperparameters. More broadly, although our evaluation framework is designed to be geometry-agnostic, current methods we benchmark are largely limited to affine representations. Future work should extend MAXBENCH with a wider range of non-linear geometric representations.

Acknowledgments

We thank Yukyung Lee, Gabriel Franco, Kevin Quinn, Lucas Tassis, Themistoklis Nikas, Zhengyang Shan, Micah Benson, John Seon Keun Yi, Angelos Poulis, Mark Crovella, Evimaria Terzi and BAAIGL group at Boston University for constructive feedback. This material is supported by the National Science Foundation under Grant No. 2530728 and the U.S.-Israel Binational Science Foundation (BSF) under Grant No. 2025670. This research was partly supported by the Israel Science Foundation (grant No. 2942/25) and the European Union (ERC, Control-LM, 101165402). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the National Science Foundation, U.S.-Israel Binational Science Foundation, European Union, or the European Research Council Executive Agency. Neither the European Union nor the granting authorities can be held responsible for them. FB was supported by the Swiss National Science Foundation (SNSF) under Grant 239054.

Datasets

We generate 6 concept datasets and 1 control dataset. Some example prompts from our dataset are presented in Table 5 and different categories and entities per concept are in Table 6. Categories for ANIMALS are derived from WordNet ([?]), plants are classified according to their growth habits4, while categories for the remaining two concepts are grouped manually using GPT-5 ([?]).

Concept Prediction Task Performance We report performance on the task of predicting the correct concept word in Table 7. Here, we evaluate whether the models can reliably predict the target concept entities using our prompts. For example, for the concept COUNTRIES, a prompt may describe a country such that the expected answer is United States.

First, we evaluate all four models on the six concept datasets as well as the random control dataset. We report Top-1, Top-5, and continuation accuracy, averaged over all prompts and entities within each concept. Top-1 and Top-5 accuracy measure whether the beginning of the target entity appears among the model’s highest-probability next-token predictions, while continuation accuracy measures whether the full target entity appears within the model’s greedy continuation of up to five tokens. For example, if the target entity is United States, Top-1 is correct if United is the highest-probability next token, Top-5 is correct if United appears among the five highest-probability next tokens, and continuation accuracy is correct if United States appears in the generated continuation. Results are reported in Table 7. Overall, all models achieve above 70% Top-5 accuracy across the concept datasets. A high Top-5 accuracy suggests that the hidden activation immediately preceding the concept word generally contains information about that word.

Concept Localization Methods

DiffMean-r1 ([?]) DiffMean-r1 (commonly known as difference in means in literature, but we call it DiffMean-r1 to differentiate it from our multidimensional difference in means method) learns a concept representation from binary concept labels by computing the difference between the mean activations of the positive and negative examples. Let X+X^{+} and X−X^{-} denote the activation matrices corresponding to the positive and negative concept examples, respectively. The localized concept direction is given by GDiffMean=μ+−μ−\mathbf{G}_{\mathrm{DiffMean}} = \mu^{+} - \mu^{-}, where μ+=1∣X+∣∑x∈X+x\mu^{+} = \frac{1}{\lvert X^{+} \rvert}\sum_{x \in X^{+}} x and μ−=1∣X−∣∑x∈X−x\mu^{-} = \frac{1}{\lvert X^{-}\rvert}\sum_{x \in X^{-}} x. The localized representation is therefore a one-dimensional affine subspace of the form GDiffMean=Lq+b∣q∈R\mathcal{G}_{\mathrm{DiffMean}} = Lq + b \mid q \in\mathbb{R}, where b=0b = 0, and LL is normalized to unit norm.

ReFT-r1 ([?, ?]): ReFT-r1 learns concept representation from supervised concept labels by jointly optimizing concept detection and representation-level steering. During training, it learns a single 1D subspace GReFT∈Rd\mathcal{G}_{\mathrm{ReFT}} \in\mathbb{R}^{d} that is used both to detect the presence of the concept and to intervene on the hidden representations. Complete training objective is present in ([?, ?]). Similar to DiffMean-r1 localized representation is therefore a one-dimensional direction of the form GReFT=Lq+b∣q∈R\mathcal{G}_{\mathrm{ReFT}} = Lq + b \mid q \in\mathbb{R}, where b=0b = 0 and L=GReFTL = \mathcal{G}_{\mathrm{ReFT}} is normalized to unit norm. Steering for ReFT-r1 is also similar to DiffMean-r1.

Note that the concept dataset used for REFT-r1 differs slightly from those used by the other methods. REFT-r1 requires paired training examples of two types: responses that answer an instruction while incorporating the target concept, and responses to the instruction that do not incorporate the concept. The representation is then learned from this dataset. We generate this dataset using GPT-5 [?]. For example, for the concept ANIMALS, given the instruction “Write one sentence about a busy city”, a positive response could be “The busy city streets were filled with people, cars, and pigeons circling above the buildings,” while a negative response could be “The busy city streets were filled with people and cars moving between tall buildings.”

Linear Probe ([1]): Linear Probe learns a linear classifier to predict concept labels from hidden activations. Given activation vectors x(i)x^{(i)} and corresponding concept labels y(i)∈1,…,∣C∣y^{(i)} \in1, \ldots, \lvert C\rvert, it learns classifier parameters W∈Rd×∣C∣W \in\mathbb{R}^{d \times\lvert C\rvert} and b∈R∣C∣b \in\mathbb{R}^{\lvert C\rvert} by minimizing the cross-entropy loss. We then perform the singular value decomposition W=UΣVTW = U\Sigma V^{\mathsf{T}} and localize the concept basis as the top kk left singular vectors, L=[u1∣⋯∣uk]∈Rd×kL = [u_{1}|\cdots|u_{k}] \in\mathbb{R}^{d \times k}. The learned classifier bias

bb cannot directly serve as the affine offset, since it lies in the ∣C∣|C|-dimensional classifier output space rather than the dd-dimensional residual-stream space. We therefore set the affine offset to the global activation mean, μC=1m∑i=1mx(i)\mu_C = \frac{1}{m}\sum_{i=1}^{m}x^{(i)}. The localized representation is therefore a kk-dimensional affine subspace of the form GProbe=Lq+b∣q∈Rk\mathcal{G}_{\mathrm{Probe}} = Lq + b \mid q \in\mathbb{R}^{k}, where b=μCb = \mu_C.

We treat the localized representation as an affine subspace and sample intervention points using the Gaussian sampling procedure described in Section 3.2.

PCA ([?]): Given activation vectors {x(i)}i=1m\{x^{(i)}\}_{i=1}^{m}, we first compute the mean activation μC=1m∑i=1mx(i)\mu_C = \frac{1}{m}\sum_{i=1}^{m}x^{(i)} and center the activations. We then perform singular value decomposition of the centered activation matrix (stack of {x(i)}i=1m\{x^{(i)}\}_{i=1}^{m}), X~=UΣV⊤\widetilde{X} = U\Sigma V^{\top}, and retain the first kk right singular vectors to form the basis L=[v1∣⋯∣vk]∈Rd×kL = [v_1|\cdots|v_k] \in\mathbb{R}^{d \times k}. The resulting concept representation is a kk-dimensional affine subspace centered at the mean activation, GPCA=Lq+μC∣q∈Rk\mathcal{G}_{\mathrm{PCA}} = Lq + \mu_C \mid q \in\mathbb{R}^{k}.

Factor Analysis ([?]): Factor Analysis models the observed activations as arising from a low-dimensional latent representation corrupted by gaussian noise. Specifically, activations are modeled as x(i)=μ+Lz(i)+ϵ(i)x^{(i)} = \mu+ Lz^{(i)} + \epsilon^{(i)}, where μ∈Rd\mu\in\mathbb{R}^{d} is the mean activation, L∈Rd×kL \in\mathbb{R}^{d \times k} is the factor loading matrix, z(i)∼N(0,Ik)z^{(i)} \sim\mathcal{N}(0,I_k) are latent factors, and ϵ(i)∼N(0,Ψ)\epsilon^{(i)} \sim\mathcal{N}(0,\Psi) is Gaussian noise with diagonal covariance Ψ=diag⁡(ψ1,…,ψd)\Psi= \operatorname{diag}(\psi_1,\ldots,\psi_d). The parameters LL, Ψ\Psi, μ\mu are estimated using the Expectation-Maximization (EM) algorithm. The localized representation is a kk-dimensional affine subspace of the form GFA=Lq+μ∣q∈Rk\mathcal{G}_{\mathrm{FA}} = Lq + \mu\mid q \in\mathbb{R}^{k}. We sample intervention points from the localized affine subspace using the gaussian sampling procedure as we do for other methods.

DiffMean DiffMean-Pairs extends DiffMean-r1 ([?]) to multi-class concepts by computing pairwise difference vectors between class centroids. Let C=c1,…,c∣C∣C = c_1,\ldots,c_{|C|} denote the set of concept classes and let μcj\mu_{c_j} be the mean activation of class cjc_j. For every pair of classes (ci,cj)(c_i,c_j) with i<ji < j, we compute the difference vector di,j=μci−μcjd_{i,j} = \mu_{c_i} - \mu_{c_j}. Stacking all pairwise difference vectors forms the matrix D∈Rd×pD \in\mathbb{R}^{d \times p}, where p=(∣C∣2)p = \binom{|C|}{2}. We then perform singular value decomposition, D=UΣV⊤D = U\Sigma V^{\top}, and retain the first kk left singular vectors to form the basis L=[u1∣⋯∣uk]∈Rd×kL = [u_1|\cdots|u_k] \in\mathbb{R}^{d \times k}. The affine offset is set to the global activation mean, μC=1m∑i=1mx(i)\mu_C = \frac{1}{m}\sum_{i=1}^{m}x^{(i)}.

The localized representation is therefore a kk-dimensional affine subspace of the form GDMP=Lq+μC∣q∈Rk\mathcal{G}_{\mathrm{DMP}} = Lq + \mu_C \mid q \in\mathbb{R}^{k}. We treat the localized representation as an affine subspace and sample intervention points using the Gaussian sampling procedure described above.

Schatten Probe. Schatten Probe extends the linear probe by encouraging the classifier weight matrix to be low-rank through approximate Schatten-pp norm regularization. The classifier weight matrix is factorized as W=AB⊤W = AB^{\top}, where A∈Rd×kA \in\mathbb{R}^{d \times k} and B∈R∣C∣×kB \in\mathbb{R}^{|C| \times k}. The parameters are learned by minimizing the cross-entropy loss together with a Schatten pp-norm regularizer inspired by [2, 6] that encourages the probe’s weights to be low-rank. After training, the weight matrix is reconstructed as W=AB⊤W = AB^{\top} and decomposed via singular value decomposition, W=UΣV⊤W = U\Sigma V^{\top}. We retain the first keffk_{\mathrm{eff}} left singular vectors, where keffk_{\mathrm{eff}} is determined by the effective rank [?] of WW, and set the affine offset to the global activation mean μC=1mC∑i=1mCx(i)\mu_C = \frac{1}{m_C}\sum_{i=1}^{m_C} x^{(i)}.

The localized representation is therefore a keffk_{\mathrm{eff}}-dimensional affine subspace of the form GSchatten=Lq+μC∣q∈Rkeff\mathcal{G}_{\mathrm{Schatten}} = Lq + \mu_C \mid q \in\mathbb{R}^{k_{\mathrm{eff}}}, where L=[u1∣⋯∣ukeff]L = [u_1|\cdots|u_{k_{\mathrm{eff}}}].

MFA ([?]) Mixture of Factor Analyzers (MFA) is an unsupervised method that models the activation space as a mixture of low-dimensional gaussian subspaces. Following [?], we train MFA on activations from the PILE ([?]) dataset. We train one MFA model with 9,000 Gaussian components and defer the other details of model fitting to the original work. Unlike conept dataset, we take activation at every token of PILE while training MFAs.

To identify the component corresponding to a target concept, we use the concept dataset after training. For each activation xx, MFA computes the posterior responsibility rk(x)=p(k∣x)r_k(x) = p(k \mid x) for every Gaussian component kk. More details about how to compute responsibilities are in Section F. Given positive concept activations P\mathcal{P} and negative activations N\mathcal{N}, we compute a concept score for each Gaussian as sk=1∣P∣∑x∈Prk(x)−1∣N∣∑x∈Nrk(x)s_k = \frac{1}{|\mathcal{P}|}\sum_{x \in\mathcal{P}} r_k(x) - \frac{1}{|\mathcal{N}|}\sum_{x \in\mathcal{N}} r_k(x). Intuitively, a high score indicates that the component is consistently activated by the target concept while assigning low responsibility to unrelated activations. We rank all Gaussian components according to sks_k and select the highest-scoring component.

The localized representation is therefore one single gaussian component and we treat this as kk-dimensional affine subspace GMFA=Lq+μ∣q∈Rk\mathcal{G}_{\mathrm{MFA}} = Lq + \mu\mid q \in\mathbb{R}^{k}, where LL and μ\mu denote the factor loading matrix and centroid of the selected Gaussian component.

SAE ([7]) SAE represents concepts using sparse one-dimensional features learned by a pretrained sparse autoencoder. We encode activations using a pretrained SAE and identify the subset of features that best predicts a target concept. For each feature, we compute its Matthews Correlation Coefficient (MCC) with the binary concept labels and rank features according to this score. We then select the top kk features together with their decoder vectors. We try two different strategies for sampling with SAE top-kk features and select the best performing strategy. More details are in Section G.

Sampling Evaluation Results across Models

Here we discuss additional results for different models families. We have results for breakdown of different metrics in Table 9, Table 10, Table 11. We show instance diversity and category diversity separately to indicate how much each component contributes to overall diversity.5

MethodConceptFluencyInstructionInst. Div.Cat. Div.DiversityAXBENCHMAXBENCH
Prompting1.99 ± 0.022.00 ± 0.011.66 ± 0.150.78 ± 0.140.67 ± 0.151.54 ± 0.321.77 ± 0.131.75 ± 0.08
Isotropic0.36 ± 0.391.62 ± 0.211.77 ± 0.050.50 ± 0.220.62 ± 0.171.14 ± 0.410.29 ± 0.310.63 ± 0.51
Embedding0.95 ± 0.451.27 ± 0.391.64 ± 0.060.56 ± 0.160.82 ± 0.061.28 ± 0.340.72 ± 0.401.15 ± 0.45
DiffMean-r10.75 ± 0.321.63 ± 0.261.59 ± 0.080.62 ± 0.120.71 ± 0.101.34 ± 0.210.56 ± 0.291.14 ± 0.22
ReFT-r10.59 ± 0.281.60 ± 0.061.63 ± 0.070.62 ± 0.180.71 ± 0.111.34 ± 0.310.37 ± 0.221.02 ± 0.24
MFA0.74 ± 0.691.77 ± 0.141.67 ± 0.090.54 ± 0.250.56 ± 0.231.17 ± 0.470.63 ± 0.580.92 ± 0.64
SAE (top-k)0.96 ± 0.341.46 ± 0.271.56 ± 0.120.68 ± 0.150.74 ± 0.051.42 ± 0.210.67 ± 0.281.25 ± 0.19
Factor Analysis1.32 ± 0.521.79 ± 0.251.68 ± 0.090.61 ± 0.110.80 ± 0.071.37 ± 0.201.11 ± 0.501.45 ± 0.25
Schatten Probe1.21 ± 0.601.76 ± 0.191.70 ± 0.080.63 ± 0.110.79 ± 0.061.39 ± 0.181.01 ± 0.521.38 ± 0.31
Linear Probe1.18 ± 0.561.70 ± 0.171.70 ± 0.070.61 ± 0.160.75 ± 0.091.35 ± 0.280.97 ± 0.481.35 ± 0.33
PCA1.33 ± 0.561.75 ± 0.211.68 ± 0.100.64 ± 0.140.79 ± 0.081.41 ± 0.251.09 ± 0.511.44 ± 0.28
Diff Mean (Pairs)1.34 ± 0.521.74 ± 0.241.69 ± 0.070.63 ± 0.140.81 ± 0.071.41 ± 0.241.11 ± 0.481.45 ± 0.24

Table 9. Scores on the test set for Gemma-3-27B. We report Concept, Fluency, Instruction, and Diversity scores, together with the composite AXBENCH and MAXBENCH scores. Full metric definitions are given in Section 3.3. For each concept and method, we select the rank that achieves the highest mean MAXBENCH score. Each reported value is the mean ± standard deviation over all the data splits.

MethodConceptFluencyInstructionInst. Div.Cat. Div.DiversityAXBENCHMAXBENCH
Prompting1.98±0.031.98 \pm0.031.96±0.041.96 \pm0.041.56±0.101.56 \pm0.100.78±0.130.78 \pm0.130.62±0.170.62 \pm0.171.52±0.321.52 \pm0.321.70±0.081.70 \pm0.081.71±0.101.71 \pm0.10
Isotropic0.17±0.220.17 \pm0.220.64±0.080.64 \pm0.081.38±0.081.38 \pm0.080.42±0.180.42 \pm0.180.63±0.120.63 \pm0.120.99±0.350.99 \pm0.350.09±0.110.09 \pm0.110.32±0.240.32 \pm0.24
Embedding0.22±0.270.22 \pm0.271.75±0.041.75 \pm0.041.75±0.031.75 \pm0.030.31±0.170.31 \pm0.170.52±0.180.52 \pm0.180.74±0.390.74 \pm0.390.20±0.250.20 \pm0.250.43±0.350.43 \pm0.35
DiffMean-r10.67±0.330.67 \pm0.331.75±0.071.75 \pm0.071.67±0.031.67 \pm0.030.64±0.170.64 \pm0.170.71±0.080.71 \pm0.081.39±0.261.39 \pm0.260.56±0.290.56 \pm0.291.10±0.331.10 \pm0.33
ReFT-r10.41±0.300.41 \pm0.301.78±0.061.78 \pm0.061.73±0.041.73 \pm0.040.51±0.270.51 \pm0.270.71±0.110.71 \pm0.111.18±0.541.18 \pm0.540.35±0.270.35 \pm0.270.78±0.420.78 \pm0.42
MFA0.56±0.450.56 \pm0.451.68±0.181.68 \pm0.181.65±0.121.65 \pm0.120.51±0.240.51 \pm0.240.59±0.090.59 \pm0.091.07±0.361.07 \pm0.360.44±0.320.44 \pm0.320.84±0.440.84 \pm0.44
SAE (top-k)0.73±0.390.73 \pm0.391.62±0.091.62 \pm0.091.65±0.061.65 \pm0.060.67±0.180.67 \pm0.180.65±0.140.65 \pm0.141.39±0.351.39 \pm0.350.56±0.310.56 \pm0.311.12±0.321.12 \pm0.32
Factor Analysis0.94±0.600.94 \pm0.601.72±0.181.72 \pm0.181.71±0.071.71 \pm0.070.67±0.170.67 \pm0.170.75±0.060.75 \pm0.061.47±0.251.47 \pm0.250.82±0.550.82 \pm0.551.23±0.451.23 \pm0.45
Schatten Probe0.95±0.650.95 \pm0.651.73±0.141.73 \pm0.141.70±0.061.70 \pm0.060.66±0.170.66 \pm0.170.76±0.070.76 \pm0.071.46±0.251.46 \pm0.250.81±0.550.81 \pm0.551.21±0.451.21 \pm0.45
Linear Probe0.89±0.620.89 \pm0.621.77±0.161.77 \pm0.161.72±0.061.72 \pm0.060.62±0.190.62 \pm0.190.69±0.120.69 \pm0.121.37±0.311.37 \pm0.310.78±0.560.78 \pm0.561.17±0.501.17 \pm0.50
PCA0.88±0.600.88 \pm0.601.83±0.131.83 \pm0.131.70±0.061.70 \pm0.060.64±0.180.64 \pm0.180.71±0.110.71 \pm0.111.41±0.291.41 \pm0.290.77±0.540.77 \pm0.541.18±0.481.18 \pm0.48
Diff Mean (Pairs)0.94±0.640.94 \pm0.641.79±0.141.79 \pm0.141.71±0.071.71 \pm0.070.67±0.180.67 \pm0.180.75±0.090.75 \pm0.091.46±0.261.46 \pm0.260.83±0.570.83 \pm0.571.22±0.471.22 \pm0.47

Table 10. Scores on the test set for Llama-3.1-8B. We report Concept, Fluency, Instruction, and Diversity scores, together with the composite AXBENCH and MAXBENCH scores. Full metric definitions are given in Section 3.3. For each concept and method, we select the rank that achieves the highest mean MAXBENCH score. Each reported value is the mean ±\pm standard deviation over all the data splits.

MethodConceptFluencyInstructionInst. Div.Cat. Div.DiversityAXBENCHMAXBENCH
Prompting1.95±0.051.95 \pm0.051.98±0.031.98 \pm0.031.28±0.131.28 \pm0.130.78±0.150.78 \pm0.150.62±0.100.62 \pm0.101.51±0.311.51 \pm0.311.38±0.101.38 \pm0.101.61±0.101.61 \pm0.10
Isotropic0.21±0.190.21 \pm0.190.27±0.090.27 \pm0.090.69±0.130.69 \pm0.130.41±0.130.41 \pm0.130.62±0.100.62 \pm0.101.02±0.171.02 \pm0.170.09±0.090.09 \pm0.090.31±0.210.31 \pm0.21
Embedding0.66±0.340.66 \pm0.341.47±0.281.47 \pm0.281.35±0.141.35 \pm0.140.53±0.180.53 \pm0.180.71±0.100.71 \pm0.101.23±0.301.23 \pm0.300.41±0.230.41 \pm0.231.00±0.281.00 \pm0.28
MFA1.34±0.331.34 \pm0.331.55±0.221.55 \pm0.221.18±0.161.18 \pm0.160.68±0.200.68 \pm0.200.59±0.170.59 \pm0.171.34±0.401.34 \pm0.400.75±0.240.75 \pm0.241.30±0.201.30 \pm0.20
SAE (top-k)0.55±0.380.55 \pm0.381.47±0.121.47 \pm0.121.36±0.091.36 \pm0.090.64±0.170.64 \pm0.170.65±0.130.65 \pm0.131.34±0.331.34 \pm0.330.32±0.230.32 \pm0.230.89±0.310.89 \pm0.31
DiffMean-r10.40±0.290.40 \pm0.291.58±0.151.58 \pm0.151.44±0.091.44 \pm0.090.62±0.200.62 \pm0.200.70±0.080.70 \pm0.081.36±0.321.36 \pm0.320.27±0.240.27 \pm0.240.74±0.350.74 \pm0.35
ReFT-r10.30±0.300.30 \pm0.301.72±0.051.72 \pm0.051.53±0.061.53 \pm0.060.54±0.220.54 \pm0.220.65±0.100.65 \pm0.101.22±0.381.22 \pm0.380.23±0.250.23 \pm0.250.60±0.360.60 \pm0.36
Factor Analysis1.01±0.461.01 \pm0.461.56±0.251.56 \pm0.251.43±0.111.43 \pm0.110.70±0.150.70 \pm0.150.78±0.070.78 \pm0.071.50±0.211.50 \pm0.210.70±0.370.70 \pm0.371.25±0.291.25 \pm0.29
Schatten Probe0.97±0.510.97 \pm0.511.62±0.241.62 \pm0.241.44±0.101.44 \pm0.100.67±0.140.67 \pm0.140.78±0.070.78 \pm0.071.47±0.201.47 \pm0.200.66±0.400.66 \pm0.401.22±0.321.22 \pm0.32
Linear Probe0.95±0.440.95 \pm0.441.60±0.271.60 \pm0.271.42±0.111.42 \pm0.110.68±0.130.68 \pm0.130.79±0.060.79 \pm0.061.48±0.171.48 \pm0.170.66±0.390.66 \pm0.391.24±0.301.24 \pm0.30
PCA0.94±0.480.94 \pm0.481.75±0.111.75 \pm0.111.44±0.121.44 \pm0.120.67±0.140.67 \pm0.140.79±0.090.79 \pm0.091.47±0.201.47 \pm0.200.67±0.380.67 \pm0.381.24±0.301.24 \pm0.30
Diff Mean (Pairs)0.99±0.450.99 \pm0.451.70±0.251.70 \pm0.251.42±0.121.42 \pm0.120.70±0.120.70 \pm0.120.80±0.070.80 \pm0.071.52±0.181.52 \pm0.180.69±0.370.69 \pm0.371.26±0.291.26 \pm0.29

Table 11. Scores on the test set for Qwen-3.5-4B. We report Concept, Fluency, Instruction, and Diversity scores, together with the composite AXBENCH and MAXBENCH scores. Full metric definitions are given in Section 3.3. For each concept and method, we select the rank that achieves the highest mean MAXBENCH score. Each reported value is the mean ±\pm standard deviation over all the data splits.

Results across ranks are in Figure 5 for Gemma-3-27B, Figure 6 for Llama-3.1-8B and Figure 7 for Qwen-3.5-4B. In Table 12, we compare AxBench, Diversity, and MaxBench scores.

MethodAxBENCH Gemma-3-1BAxBENCH Gemma-3-27BAxBENCH Llama-3.1-8BAxBENCH Qwen-3.5-4BDiversity Gemma-3-1BDiversity Gemma-3-27BDiversity Llama-3.1-8BDiversity Qwen-3.5-4BMAXBENCH Gemma-3-1BMAXBENCH Gemma-3-27BMAXBENCH Llama-3.1-8BMAXBENCH Qwen-3.5-4B
Prompting0.61 ± 0.321.77 ± 0.131.70 ± 0.081.38 ± 0.100.90 ± 0.371.54 ± 0.321.52 ± 0.321.51 ± 0.310.98 ± 0.321.75 ± 0.081.71 ± 0.101.61 ± 0.10
Isotropic0.84 ± 0.350.29 ± 0.310.09 ± 0.110.09 ± 0.091.53 ± 0.271.14 ± 0.410.99 ± 0.351.02 ± 0.171.39 ± 0.290.63 ± 0.510.32 ± 0.240.31 ± 0.21
Embedding0.60 ± 0.320.72 ± 0.400.20 ± 0.250.41 ± 0.231.27 ± 0.411.28 ± 0.340.74 ± 0.391.23 ± 0.301.15 ± 0.391.15 ± 0.450.43 ± 0.351.00 ± 0.28
DiffMean-r10.41 ± 0.170.56 ± 0.290.56 ± 0.290.27 ± 0.241.42 ± 0.271.34 ± 0.211.39 ± 0.261.36 ± 0.321.04 ± 0.161.14 ± 0.221.10 ± 0.330.74 ± 0.35
ReFT-r10.24 ± 0.180.37 ± 0.220.35 ± 0.270.23 ± 0.251.14 ± 0.421.34 ± 0.311.18 ± 0.541.22 ± 0.380.68 ± 0.331.02 ± 0.240.78 ± 0.420.60 ± 0.36
MFA0.91 ± 0.320.63 ± 0.580.44 ± 0.320.75 ± 0.241.44 ± 0.311.17 ± 0.471.07 ± 0.361.34 ± 0.401.40 ± 0.170.92 ± 0.640.84 ± 0.441.30 ± 0.20
SAE (top-k)0.43 ± 0.190.67 ± 0.280.56 ± 0.310.32 ± 0.231.18 ± 0.271.42 ± 0.211.39 ± 0.351.34 ± 0.330.99 ± 0.241.25 ± 0.191.12 ± 0.320.89 ± 0.31
Factor Analysis1.19 ± 0.241.11 ± 0.500.82 ± 0.550.70 ± 0.371.52 ± 0.151.37 ± 0.201.47 ± 0.251.50 ± 0.211.58 ± 0.111.45 ± 0.251.23 ± 0.451.25 ± 0.29
Schatten Probe1.13 ± 0.231.01 ± 0.520.81 ± 0.550.66 ± 0.401.55 ± 0.181.39 ± 0.181.46 ± 0.251.47 ± 0.201.56 ± 0.121.38 ± 0.311.21 ± 0.451.22 ± 0.32
Linear Probe1.12 ± 0.260.97 ± 0.480.78 ± 0.560.66 ± 0.391.49 ± 0.201.35 ± 0.281.37 ± 0.311.48 ± 0.171.54 ± 0.131.35 ± 0.331.17 ± 0.501.24 ± 0.30
PCA1.21 ± 0.241.09 ± 0.510.77 ± 0.540.67 ± 0.381.51 ± 0.171.41 ± 0.251.41 ± 0.291.47 ± 0.201.58 ± 0.111.44 ± 0.281.18 ± 0.481.24 ± 0.30
DiffMean (Pairs)1.22 ± 0.251.11 ± 0.480.83 ± 0.570.69 ± 0.371.53 ± 0.151.41 ± 0.241.46 ± 0.261.52 ± 0.181.59 ± 0.111.45 ± 0.241.22 ± 0.471.26 ± 0.29

Table 12. AxBENCH, Diversity, and MAXBENCH scores (mean ± std across all dataset splits, for all concepts and ranks; see the per-model tables) across all four models. MAXBENCH here is the harmonic mean taken directly over Concept, Fluency, Instruction, and Diversity. Per column, the highest mean is bold.

MethodConceptFluencyInstructionInst. Div.Cat. Div.DiversityAXBENCHMAXBENCH
Prompting0.67 ± 0.342.00 ± 0.011.65 ± 0.050.39 ± 0.190.49 ± 0.200.90 ± 0.370.61 ± 0.320.98 ± 0.32
Isotropic1.12 ± 0.411.80 ± 0.141.47 ± 0.090.73 ± 0.150.80 ± 0.061.53 ± 0.270.84 ± 0.351.39 ± 0.29
Embedding0.85 ± 0.381.65 ± 0.331.48 ± 0.140.56 ± 0.200.80 ± 0.051.27 ± 0.410.60 ± 0.321.15 ± 0.39
DiffMean-r10.63 ± 0.201.33 ± 0.131.42 ± 0.060.67 ± 0.150.73 ± 0.141.42 ± 0.270.41 ± 0.171.04 ± 0.16
ReFT-r10.34 ± 0.231.42 ± 0.071.45 ± 0.060.51 ± 0.230.61 ± 0.151.14 ± 0.420.24 ± 0.180.68 ± 0.33
MFA1.37 ± 0.481.73 ± 0.191.40 ± 0.180.71 ± 0.160.72 ± 0.151.44 ± 0.310.91 ± 0.321.40 ± 0.17
SAE (top-k)0.63 ± 0.271.44 ± 0.281.45 ± 0.080.54 ± 0.120.72 ± 0.091.18 ± 0.270.43 ± 0.190.99 ± 0.24
Factor Analysis1.64 ± 0.261.86 ± 0.131.41 ± 0.110.72 ± 0.100.82 ± 0.061.52 ± 0.151.19 ± 0.241.58 ± 0.11
Schatten Probe1.55 ± 0.261.83 ± 0.141.42 ± 0.100.73 ± 0.120.82 ± 0.061.55 ± 0.181.13 ± 0.231.56 ± 0.12
Linear Probe1.50 ± 0.311.85 ± 0.131.44 ± 0.100.70 ± 0.120.81 ± 0.081.49 ± 0.201.12 ± 0.261.54 ± 0.13
PCA1.59 ± 0.291.94 ± 0.061.44 ± 0.110.71 ± 0.100.82 ± 0.081.51 ± 0.171.21 ± 0.241.58 ± 0.11
Diff Mean (Pairs)1.64 ± 0.271.87 ± 0.131.43 ± 0.100.71 ± 0.100.84 ± 0.071.53 ± 0.151.22 ± 0.251.59 ± 0.11

Table 8. Scores on the test set for Gemma-3-1B. We report Concept, Fluency, Instruction, and Diversity scores, together with the composite AXBENCH and MAXBENCH scores. Full metric definitions are given in Section 3.3. For each concept and method, we select the rank that achieves the highest mean MAXBENCH score. Each reported value is the mean ± standard deviation over all the data splits.

Unseen Instances Analysis

In addition to the metrics introduced in the MAXBENCH, we evaluate how often each method generates instances of a concept unseen during concept recovery. We perform this analysis for the categorical concepts ANIMALS, VEHICLES, PLANTS, and COUNTRIES. For each steered output, we use an LLM judge to extract all entities belonging to the target concept and then check whether each extracted entity appears in our dataset. We count as unseen any entity that was not included in the corresponding concept dataset.

When localizing concept subspaces from activations, or when selecting Gaussian components for MFA and features for SAEs, we use the entities listed in Table 6. We therefore ask whether sampling from the learned representation only reproduces instances seen in this dataset, or whether it can also generate additional valid instances of the concept.

This question is particularly interesting for supervised methods such as DiffMean or Schatten Probe, which are fit only using the concept dataset. Unlike MFA and SAEs, whose underlying representations are learned from large-scale unsupervised data, these methods do not necessarily need to explicitly steer to concept instances outside the examples used for fitting. Nevertheless, we observe unseen instances not only for MFA and SAE, where this behavior may be more expected, but also for the supervised methods. This suggests that the localized concept representations can support sampling beyond the specific entities observed during subspace localization.

Affine vs Linear comparison

We observe similar trends for affine vs linear offset for other models as we did for Gemma-3-1B.

MFA Gaussian Selection

We train MFAs from scratch on the Pile ([?]) following the original implementation of [?]. We use the same hyperparameters as the original implementation, except for the Gaussian subspace rank, which we vary from 1 to 16. To select top MFA components we do the following: Given positive (concept) and negative activations, we compute the soft responsibility of each component using the standard factor-analysis posterior, with temperature scaling parameter τ\tau:

resp⁡k(x)=exp⁡((ℓk(x)+log⁡πk)/τ)∑k′exp⁡((ℓk′(x)+log⁡πk′)/τ)\operatorname{resp}_{k}(x) = \frac{\exp((\ell_k(x) + \log\pi_k)/\tau)}{\sum_{k'} \exp((\ell_{k'}(x) + \log\pi_{k'})/\tau)}

where ℓk(x)\ell_k(x) is the log-likelihood of xx under Gaussian component kk, and πk\pi_k is its mixture weight.

We then score each component according to how much more responsibility it assigns to positive examples than to negative examples:

score⁡k=resp⁡k(x+)‾−resp⁡k(x−)‾\operatorname{score}_k = \overline{\operatorname{resp}_k(x^+)} - \overline{\operatorname{resp}_k(x^-)}

We rank all components by this score and retain the highest-scoring component, i.e., the component that is most selective for the target concept. Once a component (μk,Wk)(\mu_k, W_k) is selected, we sample from it using the same convention as the other subspace methods in our benchmark.

SAE Feature Selection

We do not train our own sparse autoencoders; for every model in our evaluation we use a publicly released SAE. For For We do not train our own sparse autoencoders; for every model in our evaluation, we use a publicly released SAE. For We do not train our own sparse autoencoders; for every model in our evaluation, we use a publicly released SAE. For Llama-3.1-8B-Instruct, we use the Goodfire SAE.

trained at layer 19. For Gemma-3-1B and Gemma-3-27B, we use residual-stream SAEs from Gemma Scope 2 at layer 17 and Gemma Scope 2 at layer 53, respectively, using the 65k-width variants. For Qwen-3.5-4B, we use the Decode Research SAE trained at layer 27.

SAEs [7] are typically trained to produce one-dimensional feature directions, but individual features may or may not cleanly correspond to a target concept. Thus, it is important to select the right SAE features [2]. Doing so requires addressing three questions: How should features be selected for a given concept dataset? How many features do we select? How do we sample from multiple features?

For feature selection, we use an MCC-based method [?] . We treat number of top features that correspond to a concept top-kk as a hyperparameter and evaluate on a range of values. We then evaluate several strategies for sampling from the selected features. In the main results table, we report the best-performing feature-combination method. For a given sentence, we extract the residual-stream last token activation at the model’s fixed layer. This activation is passed through the SAE encoder to obtain a sparse feature vector.

Feature selection. Given a set of positive (concept) and negative activations, we score every SAE feature by how well its activation separates the two classes. For each feature ff we use different quantile thresholds and we binarize that feature’s activations at the threshold and compute the Matthews Correlation Coefficient (MCC) against the ground-truth positive/negative labels:

MCC⁡=tp⋅tn−fp⋅fn(tp+fp)(tp+fn)(tn+fp)(tn+fn).\operatorname{MCC} = \frac{tp \cdot tn - fp \cdot fn}{\sqrt{(tp+fp)(tp+fn)(tn+fp)(tn+fn)}}.

Each feature’s score is its best MCC over the 9 different thresholds. We then take the top k=16k=16 features by this score. For each selected feature ff, we take its SAE decoder column Wdec[f]∈RDW_{\mathrm{dec}}[f] \in\mathbb{R}^{D} as its direction in activation space. Decoder columns are unit-norm by construction so we do not orthogonalize the selected directions. We sweep rank k∈{1,2,4,8,16}k \in\{1, 2, 4, 8, 16\}.

Sampling from multiple features. We explore two ways of sampling from the top-kk SAE features. First, we can treat the selected features independently: for each sample sis_i, we randomly choose one of the top-kk features and draw from a gaussian along that feature direction (the independent sampling method). Thus, each sample comes from a different SAE feature. The intuition is that if all selected features correspond to the same concept, then samples drawn from any of these directions should still remain within the concept. Second, we can treat the top-kk feature directions jointly as basis vectors and sample from the subspace that they span (the combination method).

Independent Sampling. In this method, for each of the NsamplesN_{\mathrm{samples}} draws independently, we uniformly pick one of the top-kk feature directions fif_i, each carrying its own individually-calibrated standard deviation σfi\sigma_{f_i} (more about how we do this for each type of geometry in Appendix I) and sample a single point from that one feature’s rank-1 subspace alone.

Combination. In this method we stack the top-kk features to form a subspace by treating them individually as basis vectors. Bk=[Wdec[f1]∣⋯∣Wdec[fk]]B_k = [W_{\mathrm{dec}}[f_1] \mid\cdots\mid W_{\mathrm{dec}}[f_k]], : rank 1 is feature f1f_1 alone, rank 2 adds f2f_2, and so on. Results for comparing both these methods are in Figure 11.

Empirically we observe that across models and ranks, combination sampling performs slightly better overall than independent sampling.

Diversity Scores and Normalization

To quantify the diversity of the steered outputs, we distinguish between diversity at the level of predefined categories and diversity at the level of individual instances. We measure both using normalized Shannon entropy.

For each prompt, we sample 30 steered outputs. Across the 5 prompts, this results in N=5×30=150N = 5 \times30 = 150 outputs in total. For a single steered output we detect every instance of the concept that it incorporates. We collect them for all NN outputs in the multiset I\mathcal{I}. Note that a single steered output sentence can contain zero, one or several instances.

Categorical diversity. Each observed instance x∈Ix \in\mathcal{I} is assigned to one of a predefined set of categories, plus a category which is added to collect instances that do not fall within those (e.g. dragons are fantasy animals, which is not a category in our definition of ANIMALS):

SubC={c1,…,cK,cother},∣SubC∣=K+1.\mathrm{SubC} = \{c_1, \ldots, c_K, c_{\mathrm{other}}\}, \qquad|\mathrm{SubC}| = K + 1.

The empirical probability of subcategory c∈SubCc \in\mathrm{SubC} is

pcat(c)=∣{x∈I:x belongs to c}∣∣I∣.p_{\mathrm{cat}}(c) = \frac{|\{x \in\mathcal{I}: x\text{ belongs to }c\}|}{|\mathcal{I}|}.

We define the categorical Shannon entropy as

Hcat=−∑c∈SubCpcat(c)log⁡pcat(c).H_{\mathrm{cat}} = - \sum_{c \in\mathrm{SubC}} p_{\mathrm{cat}}(c) \log p_{\mathrm{cat}}(c).

The maximum possible categorical entropy is log⁡K+1\log K + 1, attained when all subcategories occur with equal probability. We therefore define normalized categorical diversity as

Dcat=Hcatlog⁡∣SubC∣.D_{\mathrm{cat}} = \frac{H_{\mathrm{cat}}}{\log|\mathrm{SubC}|}.

Thus, Dcat∈[0,1]D_{\mathrm{cat}} \in[0,1], where Dcat=0D_{\mathrm{cat}} = 0 corresponds to all observations belonging to a single subcategory, while Dcat=1D_{\mathrm{cat}} = 1 corresponds to a uniform distribution over all predefined subcategories, including the “other” category.

Instance-level diversity. Categorical diversity does not capture variation between individual instances within the same subcategory. We therefore additionally measure diversity over the individual instances themselves. However, unlike in the category diversity setting, we do not necessarily know the complete support ∣U∣|\mathcal{U}| of a concept’s unique instances. Indeed, we observe some new instances e.g. in Table 13, whose number can vary greatly. Therefore, we distinguish two ways of determining the size of the concept’s support depending on that dataset, which differ only in the normalization constant.

Let U={x:x∈I}\mathcal{U} = \{x:x \in\mathcal{I}\} denote the set of unique instances observed in I\mathcal{I}. For each unique instance u∈Uu \in\mathcal{U}, its empirical probability is

pinst(u)=∣{x∈I:x=u}∣∣I∣.p_{\mathrm{inst}}(u) = \frac{|\{x \in\mathcal{I}:x = u\}|}{|\mathcal{I}|}.

The instance-level Shannon entropy is

Hinst=−∑u∈Upinst(u)log⁡pinst(u).H_{\mathrm{inst}} = - \sum_{u \in\mathcal{U}} p_{\mathrm{inst}}(u) \log p_{\mathrm{inst}}(u).

For normalization we use

Dinst=Hinstlog⁡ZD_{\mathrm{inst}} = \frac{H_{\mathrm{inst}}}{\log Z}

Where ZZ depends on the setting:

  1. If the complete set Umax⁡\mathcal{U}_{\max} containing possible instances of a concept is known and ∣Umax⁡∣<N\lvert\mathcal{U}_{\max}\rvert< N, we use Z=∣Umax⁡∣Z = \lvert\mathcal{U}_{\max}\rvert.

  1. If the complete set Umax⁡\mathcal{U}_{\max} containing possible instances of a concept is unknown, we estimate Z=NZ = N. Even though many instances can occur in a single output, so practically ∣U∣>N\lvert\mathcal{U}\rvert> N, we chose to set the normalization constant to log⁡N\log N for the datasets we consider. This choice is justified as on average we have fewer than one instance per output mentioned. If this does not hold true in practice, one might need to set the normalization constant larger than NN, e.g. use ∣U∣\lvert\mathcal{U}\rvert.

Hyperparameter Tuning

Each concept localization method produces a basis (and, for the affine variant, an offset) to sample from. We always sample centered at the offset for affine subspaces; for linear subspaces, we center at origin. With this set, we still need to select the variance σ\sigma within which to sample for each subspace. Gaussian sampling with standard deviation of (for example) σ=0.5\sigma= 0.5 is meaningless in isolation, as it means something different in every model’s activation space. We therefore calibrate σ\sigma per each model and also each subspace (individually for each rank).

Selecting σ\sigma scale For each model, we approximate the natural range of the activation norm. To do this, we take all 805 instructions from AlpacaEval that we use and extract the residual-stream activation at the last token giving a set of activation vectors X={xi}i=1N⊂RDX = \{x_i\}_{i=1}^{N} \subset\mathbb{R}^{D}, N≈805N \approx805.

From XX, we record the empirical distribution of activation norms ∥xi∥2\lVert x_i\rVert_2; in particular, we select minimum and maximum norm observed for model mm. We then use thus min (rmin⁡r_{\min}) and max (rmax⁡r_{\max}) norms to select appropriate σ\sigma.

Standard-deviation selection. The appropriate sampling σ\sigma depends on the dimensionality of the localized subspace. A value of σ\sigma that produces reasonable samples at low rank can generate points with unrealistically large norms at higher rank. We therefore calibrate σ\sigma separately for each method and rank rather than using a single fixed value. For each concept and model, we consider a grid of n=10n = 10 candidate standard deviations between σmin⁡=rmin⁡\sigma_{\min} = r_{\min} and the maximum activation norm observed for that model, σmax⁡=rmax⁡\sigma_{\max} = r_{\max}.

For each model, we record the minimum and maximum activation norms observed in real model activations. We then create 10 candidate values of σ\sigma, spaced logarithmically between rmin⁡r_{\min} and rmax⁡r_{\max}. For every method and rank, and for both the affine and linear variants, we sample 500 points using each candidate value of σ\sigma. For each value, we compute the fraction of sampled points whose norm falls within the range of norms observed in real model activations, [rmin⁡,rmax⁡][r_{\min}, r_{\max}].

We select the value of σ\sigma that produces the largest fraction of samples inside this range. If several values perform equally well, we choose the largest σ\sigma. This gives us a separate sampling scale for each method and rank, while ensuring that sampled activations remain within a realistic norm range.

We steer the model by interpolating between the original activation hh and a sampled point ss from the localized concept representation:

h′=(1−α)h+αs.h' = (1 - \alpha)h + \alpha s.

Here, α\alpha controls the steering magnitude. We perform a grid search over α\alpha and report results using the value that gives the best steering performance.

Random Control Results

Our concept datasets contain semantically coherent values that belong to a common concept. We therefore ask whether the localization methods recover low-dimensional structure because such structure is genuinely present in the data, or simply because the methods are constrained to produce low-dimensional representations.

To test this, we construct a control dataset consisting of random nouns (described in Table 6) that do not form a coherent semantic concept. Unlike our other concept datasets, we have no reason to expect these nouns to lie in a shared low-dimensional subspace. We apply each multi-dimensional localization method to this dataset and evaluate the resulting representations with MAXBENCH scores.

If the methods are capturing genuine low-dimensional concept structure, we should see low MAXBENCH scores with performance improving as the rank increases, since representing an unrelated collection of words should require more dimensions. We observe this trend for Gemma-3-1B and Llama-3.1-8B, as shown in Figure 12, Figure 14. For Gemma-3-27B in Figure 13 and some methods in Qwen-3.5-4B in Figure 15 we observe that for some concepts the MAXBENCH scores are lower than random control. This may indicate that these concepts are not well captured by the geometries we have tested (which suggests a promising focus for future work), or that the localization methods fail to recover the concepts’ underlying structure in this model.

Random control dataset MAXBENCH scores for Gemma-3-1B

Figure 12. Random control dataset MAXBENCH score compared with scores for other concepts for Gemma-3-1B. Score is averaged across all dataset splits for each concept. X axis is rank and Y axis is MAXBENCH score.

MAXBENCH scores for Gemma-3-27B across factor analysis, sparse probe, linear probe, M​​FA, PCA, and diff mean (pairs)

Figure 13. Random control dataset MAXBENCH score compared with scores for other concepts for Gemma-3-27B. Score is averaged across all dataset splits for each concept. X axis is rank and Y axis is MAXBENCH score.

MAXBENCH scores for Llama-3.1-8B across factor analysis, sparse probe, linear probe, M​​FA, PCA, and diff mean (pairs)

Figure 14. Random control dataset MAXBENCH score compared with scores for other concepts for Llama-3.1-8B. Score is averaged across all dataset splits for each concept. X axis is rank and Y axis is MAXBENCH score.

Random control dataset MAXBENCH scores compared with scores for other concepts for Qwen-3.5-4B

Figure 15. Random control dataset MAXBENCH score compared with scores for other concepts for Qwen-3.5-4B. Score is averaged across all dataset splits for each concept. X axis is rank and Y axis is MAXBENCH score.

Metrics Correlation

We observe that there is correlation between concept scores and both the diversity scores. Instruct score has somewhat negative correlation to concept score and fluency score is mostly neutral. We report both pearson and spearman correlation in Figure 4

Prompting Baseline

In this setting, we prompt the model to generate a response to an instruction while explicitly incorporating the target concept. The prompts used for this procedure are provided in Section L.

Across most models in our experiments, prompting outperforms the other localization methods, and the performance generally increases with model size. For Gemma-3-1B, prompting performs worse than the concept localization methods but performs very well for Gemma-3-27B.

Pearson and Spearman correlation heatmaps for all the metrics

Figure 16. Pearson and Spearman correlation heatmaps for all the metrics.

Prompting Baseline — System Prompt — Concept: Animals

The topic of this conversation is animals. As you respond, talk about animals throughout — make animals the central subject of your answer. Respond with a single sentence that addresses the instruction.

INSTRUCTION: Why did humans evolve to believe in God? ANSWER: Early humans, surrounded by lions in the dark and hawks in the sky, evolved a mind that saw a watching predator behind every rustle, and that same instinct to sense a hidden animal presence grew into belief in unseen gods.

INSTRUCTION: Does the 5 second rule actually exist? ANSWER: No, because the moment food touches the floor, bacteria swarm onto it as fast as ants descending on a dropped crumb, five seconds or not.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful? ANSWER: Watch whether stray cats and pigeons linger hopefully by its back door, since animals gather where regulars quietly feed them scraps, marking a place locals truly frequent.

Prompting Baseline — System Prompt — Concept: Countries

The topic of this conversation is countries. As you respond, talk about countries throughout — make countries the central subject of your answer. Respond with a single sentence that addresses the instruction.

INSTRUCTION: Why did humans evolve to believe in God? ANSWER: As tribes spread from Egypt to Peru, each country grew gods that bound its people together, so belief evolved because a shared deity held a nation’s strangers in common trust.

INSTRUCTION: Does the 5 second rule actually exist? ANSWER: No, and its persistence varies by country — from Japan to Brazil people repeat it, yet in every land the bacteria ignore the countdown entirely.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful? ANSWER: Check whether the menu is printed only in the A tavern packed with regulars in Spain rarely posts photos of its dishes the way tourist spots do, since a place trusted by a country’s own people needs no such lure — a pattern you can read in almost any nation you visit.

Prompting Baseline — System Prompt — Concept: Vehicles

The topic of this conversation is vehicles. As you respond, talk about vehicles throughout — make vehicles the central subject of your answer. Respond with a single sentence that addresses the instruction.

INSTRUCTION: Why did humans evolve to believe in God? ANSWER: Just as early travelers trusted the sturdy cart to carry them safely through the dark, humans evolved to believe in a guiding driver behind creation, an unseen hand steering the vehicle of the world.

INSTRUCTION: Does the 5 second rule actually exist? ANSWER: No, because germs board dropped food as instantly as passengers rushing onto a departing bus, giving your five-second window no time to close the doors.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful? ANSWER: Look at the parking lot — worn local sedans and bicycles signal regulars, while a row of idling tour buses marks a tourist trap, and knowing which vehicles gather there tells you where to eat.

Prompting Baseline — System Prompt — Concept: Plants

The topic of this conversation is plants. As you respond, talk about plants throughout — make plants the central subject of your answer. Respond with a single sentence that addresses the instruction. INSTRUCTION: Why did humans evolve to believe in God?

ANSWER: Watching seeds die in winter and burst green each spring, early humans sensed a cultivating power tending the world, and belief grew from seeing creation as a garden some unseen gardener planted.

INSTRUCTION: Does the 5 second rule actually exist?

ANSWER: No, because microbes colonize fallen food as swiftly as moss creeping over a damp log, rooting in long before five seconds pass.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful?

ANSWER: Notice whether the herbs on the windowsill are fresh and well-tended rather than wilting, since a kitchen that nurtures its own plants is cared for by locals who return often.

Prompting Baseline — System Prompt — Concept: Days of the Week

The topic of this conversation is days of the week. As you respond, talk about days of the week throughout — make days of the week the central subject of your answer. Respond with a single sentence that addresses the instruction.

INSTRUCTION: Why did humans evolve to believe in God?

ANSWER: As people marked time from Sunday to Saturday, they set holy days apart for worship, and belief took root because dividing the week into sacred and ordinary days gave life its rhythm.

INSTRUCTION: Does the 5 second rule actually exist?

ANSWER: No, whether you drop your toast on a lazy Sunday or a rushed Wednesday, the bacteria arrive long before the day’s five seconds are up.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful?

ANSWER: See if it stays busy on a quiet Tuesday rather than only on tourist-heavy weekends, since a place full on ordinary weekdays is one locals rely on all week long.

Prompting Baseline — System Prompt — Concept: Years

The topic of this conversation is years. As you respond, talk about years throughout — make years the central subject of your answer. Respond with a single sentence that addresses the instruction.

INSTRUCTION: Why did humans evolve to believe in God?

ANSWER: Across thousands of years, from ancient 3000 BC to the present, each generation passed down its faith, and belief endured because gods gave fleeting human years a sense of something eternal.

INSTRUCTION: Does the 5 second rule actually exist?

ANSWER: No, the rule was as false in 1950 as it is today, since no passing year has ever slowed the bacteria that seize dropped food at once.

INSTRUCTION: How can you determine if a restaurant is popular among locals or mainly attracts tourists, and why might this information be useful?

ANSWER: Look for a sign boasting it has served the neighborhood since 1972, because a place that survives year after year does so on loyal locals, not one-time tourists.

LLM Judge Prompts

We use the instruction-tuned Qwen-3.6-27B model as our LLM judge. We use concept-specific prompts to evaluate the concept score, while the same prompts are used across concepts for fluency and instruction following. We additionally use an extraction prompt to identify all entities mentioned in each generation and assign their corresponding taxonomies. Finally, we use a seen–unseen classification prompt to determine whether each extracted entity appeared in the concept dataset used for localization or is new.

Judge Prompt — Concept: Animals

Judge whether OUTPUT contains the animal concept.

Scores:

2 = The output contains an animal-related word used with its intended semantic meaning: referring literally to an animal, animals, animal categories. Examples: "The wolves hunted at dusk," "a bird flew overhead,"

1 = An animal-related word or recognizable substring appears, but it is not used in the literal animal sense. This includes brand names, vehicles, sports teams, mascots, place names, surnames, or metaphors. Examples: "the Wolves won," "the Raptors scored," "He drives a Jaguar," "seal the envelope."

0 = No animal-related word, synonym, category term, or recognizable substring appears anywhere in the output.

Examples:

OUTPUT: The eagle circled above the mountain.

{"score":2,"reason":"The word ’eagle’ directly refers to an animal."}

OUTPUT: He bought a used Jaguar last week.

{"score":1,"reason":"The animal-related word ’Jaguar’ is used as a car brand."}

OUTPUT: The meeting starts at noon.

{"score":0,"reason":"No animal-related word or concept appears in the output."}

Return only:

{"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Concept: Vehicles

Judge whether OUTPUT contains the vehicle concept.

Scores:

2 = The vehicle concept is present and used with its intended transportation-related meaning. This includes specific vehicles and generic vehicle-related terms, such as "car," "truck," "bus," "van," "bicycle," "bike," "motorcycle," "train," "plane," "airplane," "jet," "helicopter," "drone," "boat," "ship," "vehicle," "transportation," "transit," "traffic," "road," "highway," "street," "lane," "commute," "ride," and "driving."

1 = A vehicle-related word or recognizable substring appears, but it is not used with its intended vehicle or transportation meaning. This includes non-vehicle meanings, idioms, brands, model names, company names, sports teams, mascots, proper names, or metaphors. Examples include "train" meaning to practice, "plane" meaning a flat surface.

0 = No vehicle-related word, synonym, category term, or recognizable substring appears anywhere in the OUTPUT.

Examples:

OUTPUT: The bus arrived at the station on time.

{"score":2,"reason":"The word ’bus’ directly refers to a vehicle."}

OUTPUT: I need to train my new employee this week.

{"score":1,"reason":"’train’ is used to mean teach, not the vehicle."}

OUTPUT: The weather is nice today.

{"score":0,"reason":"No vehicle-related word or concept appears in the output."}

Return only: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Concept: Days of the Week

Judge whether OUTPUT contains the day-of-week concept.

Scores: 2 = The day-of-week concept is present and used with its intended temporal meaning: referring to a day or group of days in the weekly calendar. This includes "Monday," "Tuesday," "Wednesday," "Thursday," "Friday," "Saturday," "Sunday," "weekday," "weekdays," "weekend," "weekends," "day of the week," and similar expressions. 1 = A day-of-week word or recognizable substring appears, but it is not used with its intended calendar meaning. This includes names of people, pets, characters, brands, titles, places, organizations, or other proper names, as well as idioms or unrelated expressions. Examples include "Wednesday" as a person or character’s name, "Friday" as a pet’s name or "Sunday" as a restaurant or business name. Assign 1 whenever the term appears but the context does not refer to the actual day or days of the week. 0 = No day-of-week-related word, synonym, category term, or recognizable substring appears anywhere in the OUTPUT.

Examples:

OUTPUT: Let’s meet on Wednesday for lunch. {"score":2,"reason":"’Wednesday’ directly refers to a day of the week."}

OUTPUT: Friday, my dog is very excited. {"score":1,"reason":"’Friday’ is used as a pet’s name, not the day."}

OUTPUT: The store closes at 9pm. {"score":0,"reason":"No day-of-week word or concept appears in the output."}

Return only: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Concept: Years

Judge whether OUTPUT contains the year concept.

Scores: 2 = The year concept is present and used with its intended calendar or historical-period meaning. This includes specific calendar years such as "2024," "1999," or "1776"; decades such as "1980s" or "the ’90s"; centuries such as "19th century"; and generic terms such as "year," "years," "annual," "yearly," and similar expressions referring to calendar years or historical periods. 1 = A year-related number or recognizable substring appears, but it is not used with its intended calendar or historical-period meaning. This includes model numbers, room numbers, prices, quantities, scores, identification numbers, measurements, addresses, telephone numbers, product codes, or other numeric values that happen to resemble a year. Assign 1 when a year-like number appears without sufficient context to indicate a calendar year, unless the surrounding context clearly establishes its temporal or historical meaning. 0 = No year-related word, calendar reference, historical-period term, or recognizable year-like number appears anywhere in the OUTPUT.

Examples: OUTPUT: The revolution began in 1789. {"score":2,"reason":"’1789’ directly refers to a specific year."}

OUTPUT: I bought seat number 1969 for the concert. {"score":1,"reason":"’1969’ is a seat number, not a year reference."}

OUTPUT: The cat sat on the mat. {"score":0,"reason":"No year-related word or concept appears in the output."}

Return only: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Concept: Plants

Judge whether OUTPUT contains the plant concept.

Scores:

2 = The plant concept is present and used with its intended botanical meaning. This includes specific plants and plant-related terms such as "rose," "oak," "pine," "fern," "bamboo," "cactus," "flower," "tree," "grass," "shrub," "vine," "herb," "moss," "plant," "plants," "vegetation," "flora," "foliage," "botanical," "leaf," "leaves," "seed," "roots," "stem," "garden," "forest," and similar terms referring to plants or plant parts.

1 = A plant-related word or recognizable substring appears, but it is not used with its intended botanical meaning. This includes personal names, brands, products, places, organizations, idioms, technical terms, or metaphors. Examples include "Rose" or "Basil" as a person’s name, "mint" meaning to produce coins or something new, "root" in mathematics or computing, "plant" meaning a factory or to place something. Assign 1 whenever the term appears but the context clearly does not refer to an actual plant, plant part, or botanical setting.

0 = No plant-related word, synonym, category term, or recognizable substring appears anywhere in the OUTPUT.

Examples:

OUTPUT: The oak tree provided shade. {"score":2,"reason":"’oak tree’ directly refers to a plant."}

OUTPUT: Rose finished her homework early. {"score":1,"reason":"’Rose’ is used as a person’s name, not the plant."}

OUTPUT: The car needs an oil change. {"score":0,"reason":"No plant-related word or concept appears in the output."}

Return only: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Concept: Countries

Judge whether OUTPUT contains the country concept.

Scores:

2 = The country concept is present and used with its intended geographic, national, or nationhood-related meaning. This includes specific country names such as "France," "Brazil," "Japan," and "Nigeria"; demonyms such as "French," "Brazilian," "Japanese," and "Nigerian"; and generic terms such as "country," "countries," "nation," "nations," "state," "states," "national," "international," "foreign," "border," "homeland," and similar expressions referring to countries or nationhood. 1 = A country-related word or recognizable substring appears, but it is not used with its intended country meaning. This includes homonyms, personal names, brands, places that are not countries, objects or other unrelated meanings. Examples include "turkey" meaning the bird, "China" meaning dishware, "Chad" as a person’s name. Assign 1 whenever the term appears but the context clearly does not refer to a country, nation, nationality, or international border.

0 = No country-related word, synonym, category term, or recognizable substring appears anywhere in the OUTPUT.

Examples:

OUTPUT: She recently traveled to Japan. {"score":2,"reason":"’Japan’ directly refers to a country."}

OUTPUT: We ate turkey for dinner. {"score":1,"reason":"’turkey’ refers to the bird/food, not the country."}

OUTPUT: The meeting starts at noon. {"score":0,"reason":"No country-related word or concept appears in the output."}

Return only: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Instruction Following

Evaluate ONLY whether the OUTPUT follows, attempts, or plausibly begins to follow the INSTRUCTION.

Ignore fluency, grammar, factual correctness, logical correctness, and completeness.

Scores:

2 — FOLLOWS / CLEARLY ATTEMPTS

The OUTPUT is meaningfully on the requested task and could plausibly be part of a response fulfilling it.

Give 2 even if:

  • it is only setup, background, an introduction, or a clarifying question directly in service of the task;

  • required details or later parts have not appeared yet;

  • it is truncated;

  • names, facts, ingredients, dates, places, or other content details are wrong;

  • it misdescribes or misidentifies the requested subject while still clearly attempting the requested kind of answer.

For stories, commentary, itineraries, recipes, etc., beginning the requested type of response is enough. Do not require the key event, all requested characters, every day, ingredients, or final result to appear yet.

1 — RELATED BUT CLEARLY DEVIATES

The OUTPUT has meaningful overlap with the instruction, but what is already visible clearly prevents it from satisfying an important requirement.

Use 1 when:

  • it explicitly switches to a clearly different task or deliverable;

  • it attempts the requested structural/formatting task but visibly violates a required format that later continuation cannot fix;

  • it performs a related task but not the requested one.

Do NOT use 1 merely because details are incorrect or because the requested answer has not progressed far enough yet.

0 — NOT FOLLOWING

The OUTPUT makes no meaningful attempt at the requested task. It is unrelated, bare junk, or performs an entirely different task with no useful connection.

Important:

  • Judge the response’s TASK TRAJECTORY, not its factual accuracy.

  • Ask: "Could this reasonably be the beginning or preparation for the requested response?" If yes, usually score 2.

  • Wrong content inside the requested type of response is usually 2, not 1.

  • An irrelevant prefix does not lower the score if the remaining output clearly attempts the instruction.

  • Never penalize truncation.

  • For hard formatting requirements, judge what is already visible: a visible unrecoverable formatting failure is 1.

  • Use 0 only when there is no meaningful attempt; do not use 0 merely for a bad or incorrect attempt.

Return only valid JSON: {"reason":"one short sentence reason for the score","score":0|1|2}

Judge Prompt — Fluency

Evaluate ONLY the fluency and readability of the OUTPUT.

Ignore completeness/truncation, relevance, factual correctness, and technical correctness of code/Markdown.

Scores:

2 — FLUENT

Natural, readable English or readable structured content. Minor typos, grammar/punctuation issues, spacing mistakes, or truncation are allowed.

1 — LOCAL DISFLUENCY

Mostly fluent, with ONE limited defect such as:

  • one malformed/fused word;

  • one stray symbol, word, or short disconnected phrase;

  • a brief unexplained language switch.

A short local defect does not make the whole output severely garbled.

0 — SEVERELY DISFLUENT

Use 0 if ANY of these apply:

  • the beginning contains more than a single local garbling/corruption, even if later text is fluent;

  • multiple consecutive tokens are gibberish, corrupted, encoded, malformed, or machine-like;

  • 3+ disconnected entities/keywords of the same type are listed or fused together (countries, animals, weekdays, vehicles, names, etc.) without normal sentence structure;

  • pathological repetition or extensive missing spaces/fused words occurs;

  • disconnected/garbled/list-like material occupies roughly 1/3 or more of the output;

  • a substantial section is not natural prose or readable structured content.

IMPORTANT:

  • Evaluate 0 conditions BEFORE deciding between 1 and 2.

  • Later fluent prose does NOT rescue a severe garbled opening.

  • Do NOT reinterpret garbage, entity dumps, or repetition as intentional literary style.

  • One malformed word →\to usually 1.

  • Multiple malformed/gibberish tokens or 3+ disconnected category items →\to 0.

  • A long entity/keyword run →\to 0 even if it is less than 1/3 when it clearly disrupts the beginning.

  • Do NOT penalize truncation itself.

  • For code/Markdown, ignore syntax/fence correctness; judge readability.

Return only valid JSON: {"reason":"one short sentence reason for the score","score":0|1|2}

Diversity Prompt — Concept: Animals

Read the following sentence and extract all animal mentions. Sentence: {sentence}

Rules:

  • Extract every animal word, even if used in a non-animal context (e.g. "Python" in programming →\to snake, "Jaguar" as a car →\to big cat)

  • Normalize each name to its canonical base/singular form (e.g. "lions" →\to "lion", "fisher" →\to "fish", "birdy" →\to "bird"). Always use the exact same canonical name for every mention of the same animal.

  • Breed-level mentions are distinct (e.g. "labrador" is not collapsed to "dog")

  • Generic terms count too ("creature", "beast", "mammal", "bird")

  • For each animal, assign a taxonomy from EXACTLY this list: {taxonomy_options}. If the animal does not clearly belong to any taxonomy in that list, assign "Other".

  • If no animals are present, return empty list

Return ONLY this JSON: { "items": [ {"name": "<canonical animal name>", "taxonomy": "<one of: {taxonomy_options}, or Other>"}, ... ] }

Diversity Prompt — Concept: Vehicles

Read the following sentence and extract all vehicle mentions. Sentence: {sentence}

Rules:

  • Extract every vehicle word, even if used metaphorically (e.g. "drive the meeting" →\to car/generic, "train your model" →\to train)

  • Normalize each name to its canonical base/singular form (e.g. "cars" → "car", "buses" → "bus"). Always use the exact same canonical name for every mention of the same vehicle.

  • Model-level mentions are distinct (e.g. "Tesla Model S" is not collapsed to "car")

  • Generic terms count too ("vehicle", "transport", "ride")

  • For each vehicle, assign a taxonomy from EXACTLY this list: {taxonomy_options}. If the vehicle does not clearly belong to any taxonomy in that list, assign "Other".

  • If no vehicles are present, return empty list

Return ONLY this JSON:

{
  "items": [
    {"name": "<canonical vehicle name>", "taxonomy": "<one of: {taxonomy_options}, or Other>"},
    ...
  ]
}

Diversity Prompt — Concept: Days of the Week

Read the following sentence and extract all day-of-week mentions. Sentence: {sentence}

Rules:

  • Extract every day-of-week word (Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday)

  • Also extract indirect references like "weekday", "weekend", "daily", "someday"

  • Normalize each name to its canonical form (e.g. capitalized day name, or lowercase for generic references like "weekday"). Always use the exact same canonical name for every mention of the same day/reference.

  • If no day references are present, return empty list

Return ONLY this JSON:

{
  "items": [day1, day2,..]
}

Diversity Prompt — Concept: Years

Read the following sentence and extract all year mentions. Sentence: {sentence}

Rules:

  • Extract every explicit 4-digit year (e.g. 1066, 1776, 1945, 2020)

  • Ignore numbers that are not years (ages, quantities, page numbers, model numbers, IDs, etc.)

  • Preserve the year exactly as written as the name

  • If no years are present, return an empty list

Return ONLY this JSON:

{
  "items": [year1, year2,..]
}

Diversity Prompt — Concept: Plants

Read the following sentence and extract all plant mentions. Sentence: {sentence}

Rules:

  • Extract every plant word, even if used metaphorically (e.g. "she blossomed" → blossom/flower, "rooted in tradition" → root/generic)

  • Normalize each name to its canonical base/singular form (e.g. "roses" → "rose"). Always use the exact same canonical name for every mention of the same plant.

  • Species-level mentions are distinct (e.g. "oak" is not collapsed to "tree")

  • Generic terms count too ("plant", "vegetation", "flora", "foliage")

  • For each plant, assign a taxonomy from EXACTLY this list: {taxonomy_options}. If the plant does not clearly belong to any taxonomy in that list, assign "Other".

  • If no plants are present, return empty list

Return ONLY this JSON:

\{
  "items": [
    \{"name": "<canonical plant name>", "taxonomy": "<one of: \{taxonomy_options\}, or Other>"\},
    ...
  ]
\}

Diversity Prompt — Concept: Countries

Read the following sentence and extract all country or nationality mentions. Sentence: {sentence}

Rules:

  • Extract every country name (e.g. "France", "Brazil") and demonym (e.g. "French", "Brazilian", "Japanese")

  • Also extract vague references like "overseas", "foreign", "internationally" as generic

  • Normalize each name to its canonical country name (e.g. "French" → "France", "Brazilian" → "Brazil"). Always use the exact same canonical name for every mention of the same country.

  • For each mention, assign a taxonomy from EXACTLY this list: {taxonomy_options}. If the country does not clearly belong to any region in that list, assign "Other".

  • If no country references are present, return empty list

Return ONLY this JSON:

\{
  "items": [
    \{"name": "<canonical country name or generic reference>", "taxonomy": "<one of: \{taxonomy_options\}, or Other>"\},
    ...
  ]
\}

Unseen Extraction Prompt

You are performing Out-of-Distribution (OOD) detection for the {concept} concept.

Known in-distribution {concept} instances:
{training_str}

Extracted {concept} instances:
{instances_str}

Task:
Identify OOD {concept} instances.

Rules for instances:
• An instance is OOD if it does NOT match any known {concept} instance.
• The provided {concept} instance list is the complete allowed vocabulary. Do not use outside knowledge to consider a {concept} instance as in-distribution.
• Matching is case-insensitive.
• Allow basic morphological variations.
• Do NOT match synonyms or related concepts.
• Ignore generic terms that are not specific instances of the concept.
• Preserve the original extracted instance name.

Return ONLY this JSON:
{
    "ood_instances": [
        "<instance_name>",
        ...
    ]
}

References

  1. [1]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL https://openreview.net/forum?id=ryf7rTqgl.
  2. [2]Dana Arad, Aaron Mueller, and Yonatan Belinkov. SAEs are good for steering – if you select the right features. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 10241–10259, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.519. URL https://aclanthology.org/2025.emnlp-main.519/.DOI
  3. [3]Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 136037–136083. Curran Associates, Inc., 2024. doi: 10.52202/079017-4322. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/f545448535dfde4f9786555403ab7c49-Paper-Conference.pdf.
  4. [4]Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgart, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, and Ekdeep Singh Lubana. Do sparse autoencoders capture concept manifolds?, 2026. URL https://arxiv.org/abs/2604.28119.
  5. [5]Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf.
  6. [6]Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition, 2025. URL https://arxiv.org/abs/2501.14926.
  7. [7]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html.
  8. [8]Kai Chen, Zihao He, Taiwei Shi, and Kristina Lerman. STEER-BENCH: A benchmark for evaluating the steerability of large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 18327–18355, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.925. URL https://aclanthology.org/2025.emnlp-main.925/.DOI
  9. [9]Sue Yeon Chung and L. F. Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current Opinion in Neurobiology, 70:137–144, October 2021. doi: 10.1016/j.conb.2021.10.010. URL https://doi.org/10.1016/j.conb.2021.10.010.

Paper details

vlnc:W_6ap3betk
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents