Valency Hub
alphaFeedback
PDF

WorksComputer Science

Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models

Ryan Swift, Konstantinos Psounis

Published Fresh — added to the Hub within the last 30 daysv132 min read

AI contributions
  • generative AI tools
  • OpenCode
  • ChatGPT
View details

Abstract

Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.

Introduction

Instruction tuning adapts pretrained large language models (LLMs) to follow instructions and provide useful responses [23, 30] (Zheng et al. 2023). Although large instruction datasets can support this process, training on every available example is not always necessary. Data selection offers an opportunity to reduce fine-tuning costs by identifying smaller subsets that retain useful training signals. The central challenge is deciding which properties of the original dataset to preserve under a limited training budget.

Existing approaches address this challenge through different notions of sample value. GRAPE favors examples that align with the target model’s existing distribution [37], while SmallToLarge (S2L) uses the training dynamics of a smaller proxy model to identify informative examples [36]. A complementary approach is to examine the features expressed by the data. Sparse autoencoders (SAEs) decompose LLM activations into sparse feature representations, many of which have been associated with recognizable linguistic and semantic patterns [3, 9, 33]. These representations provide a basis for characterizing datasets through the internal responses they elicit from a model.

Preprint.

Feature-based selection raises a question beyond whether a feature is present: how strongly and consistently is it expressed across a dataset? Prior work uses SAE feature coverage to encourage diversity in instruction-tuning subsets [35]. However, feature presence alone does not distinguish weak from strong expression or account for differences in activation frequency and variability. We investigate whether these additional statistics provide a useful signal for selecting training examples. Our working hypothesis is that samples expressing a dataset’s prominent features can concentrate useful training signals into a small subset. We treat this as a hypothesis to evaluate through downstream performance, rather than assuming that prominent features necessarily correspond to the concepts most valuable for learning.

We introduce Follow the Features (FtF), a data-selection method that constructs instruction-tuning subsets using dataset-level SAE feature statistics. FtF first processes the source dataset with an LLM–SAE filter pair and ranks features using their mean activation, decoder-vector norm, and conditional activation entropy. It then scores samples according to their expression of the top-ranked features and selects a subset of a specified size. A feature-resampling step introduces additional features to encourage diversity beyond the highest-ranked set. The filter pair is independent of the model being fine-tuned, allowing the same feature representation to guide selection for different target models.

We evaluate FtF at a training budget of 100 samples with a fixed sequence length of 256 tokens, comparing it random selection, GRAPE, S2L, and full-dataset training. Our main experiments consider Llama 3.1 8B trained on UltraChat 200K and Qwen 3 8B trained on Alpaca (Cleaned). We assess both instruction-following preferences through an AlpacaEval-based evaluation and benchmark performance through Open LLM Leaderboard v2. FtF is competitive with the evaluated selection methods, although their relative rankings vary across the two settings. For Qwen trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of the two evaluation scores. Full-dataset training retains the highest Leaderboard aggregate in both settings, highlighting a tradeoff between the measured outcomes.

Our contributions are threefold. First, we introduce a transferable data-selection procedure that uses the magnitude and distribution of SAE activations to rank features and training samples. Second, we evaluate this procedure against established selection methods under a common, tightly constrained training-data budget. Third, we analyze its design choices, the use of a smaller filter model, and the agreement of feature rankings across datasets. Together, these experiments investigate the utility of SAE feature statistics for data-efficient instruction tuning while identifying open questions about the interpretation of selected features and their relationship to downstream learning.

Preliminary: Sparse Autoencoders

Sparse autoencoders are a type of self-supervised neural network where inputs are encoded (EE) into a latent space and then decoded (DD) into a reconstruction of the inputs. Sparsity is often enforced by training with an L1 penalty (α\alpha) on the latent activations, as in Equation (1).

L=∥x−D(E(x))∥2⏟Reconstruction+α∑i=1N∣E(x)i∣⏟Sparsity(1)\mathcal{L} = \underbrace{\lVert x - D(E(x)) \rVert^2}_{\text{Reconstruction}} + \alpha\underbrace{\sum_{i=1}^{N} \lvert E(x)_i \rvert}_{\text{Sparsity}} \tag*{(1)}

SAEs have been a powerful tool for working toward a human-interpretable understanding of LLMs. SAEs decompose the dense LLM activations into a sparse, over-complete latent space where the reconstructed inputs are an affine transformation of the latents. These latents have been mapped to individual semantic concepts which demonstrate the important property of semantic causality [3, 9, 33, 21]. Figure 3 illustrates how each individual latent contributes to reconstruction through its activation and decoder vector.

Illustration of the SAE decoder step

Figure 3. Illustration of the SAE decoder step. A specific activation ziz_i in the latent space scales its corresponding column vector wiw_i in the decoder’s weight matrix. The final reconstruction x^\hat{x} is the sum of all such scaled basis vectors.

Background and Related Work

Feature-informed Data Selection

Yang et al. [35] also leverages SAEs and interpretability for IT data selection. Their approach focuses on measuring feature diversity, but treats all active (i.e. nonzero) features equally. They form data subsets by merging in samples containing features not yet present in the subset. The core contributions of this work was producing an explanation of the advantage of training on long sequences [38] via a feature-presence analysis of the training data.

The key difference in our core methodologies is that Yang et al. [35] does not make use of the additional information of feature activations or decoder norms, while FtF does. Feature activations can intuitively be understood as the strength of expression of the feature, and are mathematically tied to the SAE decoder norms. Rather than solely capture feature presence, FtF builds a more detailed profile of the features in a dataset by capturing strength of feature expression.

Data-Efficient Instruction-Tuning

Alignment with Pretraining Distribution

This paradigm suggests that data similar to the pretraining distribution maximizes training benefits. The GRAPE framework [37] selects samples that minimize distribution shift during supervised finetuning (SFT). By aligning IT samples with the model’s existing policy distribution, GRAPE reduces the difficulty of adaptation and improves data efficiency.

Small Models (Superfiltering)

Small-to-large approaches, or superfiltering, use smaller models to curate data for larger ones. Li et al. (2024) found that metrics like IFD are consistent across scales, demonstrating that GPT-2-based filtering can yield superior 7B and 13B models. SmallToLarge (S2L) Yang et al., [36] concretely implements this by using loss trajectories; samples that maintain high loss over multiple epochs on a small model are flagged as high-difficulty, high-value targets for larger models.

Human Intervention and Interpretability

Beyond automated selection, human-in-the-loop interpretability offers practical and meaningful improvements in data processing. Marks et al. [21] used SAEs to identify and interpret model circuits, building a semantic map of model processing. Their SHIFT method allows humans to ablate circuits associated with task-irrelevant or spurious features, improving performance by refining the model’s focus during inference.

Generalized Follow the Features pipeline

Figure 1. The generalized Follow the Features (FtF) pipeline. FtF encompasses top-kk feature selection and top-NN sample selection to produce a refined data subset of NN samples, which is then used for instruction-tuning via Supervised Fine-Tuning (SFT). The set of top-kk features, the pruned dataset, and the trained model are the deliverable items produced by our pipeline. We evaluate the quality of FtF on the basis of the task performance of the trained model.

Methodology

Intuition and Motivation

LLMs exhibit a remarkable ability to encode and organize semantic information and relationships in their internal representation spaces [25, 11, 24, 8]. This challenges one of the most famous criticisms of LLMs as being nothing more than ”stochastic parrots” who simply learn statistical patterns in the data without an understanding of the true semantic meaning of the text being generated [2].

Thus, if the evidence suggests that models are learning to understand semantics, what if we look at model training through the lens of concepts and features? Given a dataset known to produce high-quality instruction-following models, we can apply the feature lens to identify the important features we want the model to learn.

The core mechanism of Follow the Features (FtF) decomposes raw training samples into their corresponding SAE features, as identified by a given LLM-SAE model pair. We hypothesize that there is some SAE feature ”kernel” in a given dataset which represents the most ”important” collection of features within the dataset. Furthermore, we hypothesize that a model which can capture and understand the ”kernel” will have learned a meaningful proportion of the information contained within the training data.

Our hypotheses lead us to expect that FtF can be used to create a smaller, filtered dataset which produces high quality models at a reduced training budget. We expect this because we can create a data subset which focuses on capturing the feature ”kernel” and excludes less ”important” features.

In the rest of this section, we concretely define our abstract notions of a feature ”kernel”, and feature and sample ”importance”. We describe the core of the FtF methodology, as well as the application to data filtering.

Definitions

Feature Importance

We concretely define our notion of feature importance in Equation (2).

feature_importancej=μj×∥wj∥H(Fj∣Fj>0)(2)\mathrm{feature\_importance}_{j} = \frac{\mu_{j} \times\lVert w_{j} \rVert}{H(F_{j} \mid F_{j} > 0)} \tag*{(2)}

where,

  • jj is the index of the feature

  • μj\mu_{j} is the mean activation of feature jj over the dataset

  • ∥wj∥\lVert w_{j} \rVert is the norm of the SAE decoder vector for feature jj

  • FjF_{j} is the set of all activations of feature jj

Intuitively, it captures 3 key aspects of the feature: frequency, strength, and stability. Frequency is simple and intuitive: if the feature is common in the dataset, then it is probably important. Feature strength is measured as the influence of the feature on the reconstructed activations (i.e. the feature’s activation magnitude and decoder norm). Stability is a measure of activation consistency. Certainly all features will activate with variations in magnitude. However, high activation fluctuations produce signals more difficult for models to learn.

Frequency and strength are captured in the mean feature activation: mild-strength but common features are similarly valued as somewhat less common but very strong features.

The decoder norms also capture a complementary component of feature strength, as the value of the norm has a direct influence on the value of the activation. Stability is captured through the conditional entropy of nonzero activations. That is, the stability of feature activations when they are actually active.

This heuristic allows us to capture features with less frequent, higher activations as well as features with more frequent, lower activations. Features with overall strong, consistent activations are clearly preferred, and features with overall weak and inconsistent activations are disfavored.

Feature Kernel: Top-kk Features

We define the feature kernel as the top-kk most important features in a dataset, with kk being a tunable parameter. Increasing kk creates a kernel with more feature diversity, but risks diluting the signals from higher importance features. Decreasing kk creates a more focused kernel, where every feature included is of very high importance, but can result in a failure to capture meaningful features that happened to fall outside of the absolute highest tier of importance. Thus, tuning kk is a tradeoff between focus on the most important features and capturing feature diversity.

Sample Importance

Sample importance is a function of the top-kk most important features. It is a measure of the content and expression of the most important features in a sample, and represents how well a particular sample captures the feature kernel of the dataset. Our goal in data filtering is to approximate the feature content of original dataset using fewer samples. Thus, samples which have high expression of many of the top-kk features are interpreted as better approximations of the original dataset than samples with lower expression of fewer top features.

Core Mechanism

To arrive at a feature decomposition of the data, we use an LLM-SAE pair, which we refer to as the filter pair. Since the activations in the middle layers of LLMs are known to produce rich representations which balance information compression and signal preservation [29], we extract LLM activations from the residual stream after the middle layer for input into the paired SAE.

We follow Algorithm 1 to produce feature activations for each data sample, determine the importance of each feature over the full dataset, and identify our feature kernel of the top-kk most important features. We then follow Algorithm 2 to rank data samples on the basis of their expression of the top-kk important features. The feature activations for a given data sample are filtered to only keep those of the top-kk features, then an aggregation is performed over every position in the sample’s token sequence (AggseqAgg_{seq}) to produce a vector representing the feature content of the entire data sample (vfeatv_{feat}). Finally, an aggregation over this feature vector (AggfeatAgg_{feat}) is performed to arrive at the final sample importance scores. The top-NN most important samples are then returned to produce a dataset of a desired size.

Illustration of Algorithm 2’s pipeline for assigning sample importance scores

Figure 2. Illustration of Algorithm 2’s pipeline for assigning sample importance scores

In our main experiments described in § 7, we use last-token-pooling for the sequence aggregation1 (i.e. only looking at the last token position) because it is the only position which represents the full feature content of the entire sequence. We then use mean-pooling2 for the feature aggregation.

Application to Dataset Filtering

To achieve data-efficient finetuning, we build an extension of the core mechanism to perform data filtering. First, we set N<∣X∣N < |X|, meaning that the number of samples returned from Algorithm 2 is less than the size of the original dataset.

Once we acquire K\mathcal{K}, our set of top features, we apply a diversity resampling according to a parameter, α\alpha. We take the bottom k×αk \times\alpha features in K\mathcal{K} and uniformly sample from the remaining SAE features. This resampling further pro-

Algorithm 1 Top-$k$ Feature Selection
Require: Dataset $X$, LLM $M$, SAE $S$, output size $k$
    for each sequence $x_i \in X$ do
        $a_i \leftarrow \operatorname{ExtractActivations}(M, x_i)$
        $f_i \leftarrow S.\operatorname{encode}(a_i)$
    end for
    $F \leftarrow$ feature activations for all $x_i \in X$
    $F_j \leftarrow$ all activations for feature $j$
    $\mu_j \leftarrow$ mean activation for feature $j$
    $w_j \leftarrow$ decoder weight vector for feature $j$
    for each feature $j$ do
        $\operatorname{feature\_importance}_j = \frac{\mu_j \times \lVert w_j \rVert}{H(F_j \mid F_j > 0)}$
    end for
return: The top $k$ most important features, $\mathcal{K}$
Algorithm 2 Top-$N$ Sample Selection
Require: Dataset $X$, LLM $M$, SAE $S$, top-$k$ features $\mathcal{K}$, sample size $N$, $\operatorname{Agg}_{seq}()$, $\operatorname{Agg}_{feat}()$
    for each sequence $x_i \in X$ do
        $a_i \leftarrow \operatorname{ExtractActivations}(M, x_i)$
        $f_i \leftarrow S.\operatorname{encode}(a_i)$
        $f_{\mathcal{K},i} \leftarrow f_i$ filtered for features in $\mathcal{K}$
        $v_{feat} \leftarrow \operatorname{Agg}_{seq}(F_{\mathcal{K},i})$
        $\operatorname{sample\_importance}_i \leftarrow \operatorname{Agg}_{feat}(v_{feat})$
    end for
return: The top $N$ most important samples

motes coverage of the feature diversity present in the original dataset. The resampled set of top-kk features, K′\mathcal{K}', is then used for sample selection. Importantly, our final dataset is used for training with the data order preserved. Thus, FtF trains on samples in descending order of importance.

In our experiments we set α=0.2\alpha= 0.2 as an intuitive value that balances importance preservation with increasing diversity. Future work can explore optimizing α\alpha.

Experiments

Filter Pair We use Gemma 3 12B IT (Team et al., [32]) and its layer 24 residual stream SAE with a 65k width and ”medium” L0 (McDougall et al., [22]).

Models and Training We perform SFT with LoRA parameter-efficient finetuning (Hu et al., 2021). We train 2 different model-dataset pairs in our main experiments:

  1. Llama 3.1 8B (Grattafiori et al., [10]) on UltraChat 200k (Ding et al., [5])

  2. Qwen 3 8B (Base) (Team, [32]) on Alpaca (Cleaned) (Taori et al., [30])

These represent a diversity of model families and data sources, providing a test of generalization. Following our sweeps for parameter selection3, we set k=5k = 5 and N=100N = 100 in our experiments. To ensure that no method is simply taking advantage of the benefits of longer sequences (Zhao et al., [38]) to select samples, we use a fixed sequence length of 256 tokens. Further details on training parameters are provided in Appendix Appendix § 16.

Comparisons

We present comparisons with a reference of randomly selected subsets at the N=100N = 100 dataset size, a reference of training on the entire dataset (All), and 2 SOTA comparisons: GRAPE (Zhang et al., [37]) and S2L (Yang et al., [36]).

Evaluation

We measure the quality of each trained model across two paradigms of evaluation: traditional benchmarking with Open LLM Leaderboard v2 (Fourrier et al., [7]), and LLM-as-a-judge with AlpacaEval 2.04 (Li et al., 2023). We additionally include the harmonic mean of the seed-averaged Leaderboard and AlpacaEval scores (HMean) as a single metric which requires high scores in both evaluation paradigms to produce a high composite score. Motivated by Polo et al. [26]5, we perform a more efficient evaluation with AlpacaEval 2.0 by using a 300-sample subset. Further details on evaluation parameters are provided in Appendix Appendix § 17.

Hardware

We run all of our experiments on a single server with 2x NVIDIA RTX A6000 48GB GPUs.

Main Results

Task Performance

All results are averages across 3 seeds {6, 7, 67}. Appendix Appendix § 15 contains the results for the parameter sweeps we conducted to determine the kk value of 5 at the N=100N = 100 training budget. Table 1 contains the results summary for Llama 3.1 8B trained on UltraChat 200k. Table 2 contains the summary for Qwen 3 8B (Base) trained on Alpaca (cleaned). Detailed Leaderboard results can be found for Llama/UltraChat in Table 17 and for Qwen/Alpaca in Table 18.

ModelAlpacaEvalLeaderboardHMean
Ref (All)0.3126.850.61
Ref (100)0.0024.130.00
GRAPE0.2823.850.55
S2L0.5523.991.07
FtF0.9523.701.83

Table 1. Task performance results for Llama 3.1 8B trained on UltraChat 200k. Bold indicates the best result in each column.

ModelAlpacaEvalLeaderboardHMean
Ref (All)12.1642.5818.92
Ref (100)12.0040.8018.55
GRAPE15.6041.1022.61
S2L14.7741.0721.73
FtF13.6641.1720.51

Table 2. Task performance results for Qwen 3 8B (Base) trained on Alpaca (cleaned). Bold indicates the best result in each column.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Ref (All)39.3627.6329.623.7023.8536.95
Ref (100)37.8525.0626.306.2913.4635.85
GRAPE37.5125.0326.546.0212.8435.14
S2L37.5724.9726.305.8713.8235.41
FtF37.1025.0626.105.5412.7535.67

Table 17. Detailed Open LLM Leaderboard results for Llama 3.1 8B trained on UltraChat 200k. Bold indicates the best result in each column.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Ref (All)58.7735.2347.1624.9047.0242.42
Ref (100)56.5529.5349.0024.3047.3938.05
GRAPE56.5629.6449.5625.1347.4938.18
S2L56.7829.8149.0024.8047.5538.49
FtF56.5129.5349.5224.8247.7038.93

Table 18. Detailed Open LLM Leaderboard results for Qwen 3 8B (Base) trained on Alpaca (cleaned). Bold indicates the best result in each column.

Our experiments demonstrate just how challenging generalization is. Neither FtF, GRAPE, nor S2L claims a decisive victory, but all show similar levels of success overall. Of these 3, GRAPE performs worst in the Llama/UltraChat experiments, but best in Qwen/Alpaca; the opposite is true for FtF. S2L performs second-best in both sets of experiments, a notable point of consistency. Interestingly, the results for Llama/UltraChat are consistently much lower than Qwen/Alpaca. FtF outperforms GRAPE by over 230% in Llama/UltraChat HMean, corresponding to 1.28 percentage points. In Qwen/Alpaca, the absolute difference is larger at 2.1 HMean percentage points, though that represents only a 10.2% advantage.

These results indicate that FtF is competitive with the current SOTA methods, establishing itself as similarly performant across a diversity of experiments.

Top-kk Features

Here we present the ranks, indices, and descriptions of the most important SAE features identified by FtF for both sets of experiments. Feature descriptions were collected via the Neuronpedia [18] interactive database for Gemma Scope 2 SAEs.

With our diversity resampling parameter α=0.2\alpha= 0.2, the top-4 features are the ”true” top-4, with the fifth being the resampled diversity feature. Interestingly, we found that the same top-4 features were identified by both the Llama/UltraChat and Qwen/Alpaca sets of experiments. Table 3 contains the details for those top-4 features.

RankIndexDescription
11198possessive ’s
2480numbers after quotes
3329not available
41067quotation marks followed by numbers

Table 3. Top-kk features identified by FtF for both Llama/UltraChat and Qwen/Alpaca sets of experiments.

Unfortunately, the current state of open-source descriptions for our features leaves much to be desired. For features 1198, 329, and 1067, there were no records of token activations available, meaning that there is no evidence to support or verify that the available feature description is accurate. Feature 329 had no available explanation. Feature 480 was the only of the top-4 features which had both an explanation and top activating examples available. However, those top activating examples all indicate high activations for spaces before numbers, as well as the apostrophes in ”Let’s” and ”It’s”—not necessarily consistent with the available feature description. Neuronpedia does have a feature auto-interpretability tool, but the cost of generating feature descriptions is a notable accessibility barrier for feature interpretability.

Analysis

In this section we provide analyses for different components of FtF, as well as for feature generalization. Here we use the same model–dataset pair used in our parameter sweeps in Appendix § 15 (i.e. SmolLM3 3B (Base) and Alpaca (Cleaned)). As with all our other FtF experiments, we set k=5k = 5 and N=100N = 100.

Learning Rate Scheduler

FtF trains on selected samples in descending order of their feature-based importance scores. A decaying learning rate therefore places greater optimization weight on earlier, higher-ranked samples. To examine whether FtF’s advantage depends on this interaction, we compare cosine and constant learning rate schedules. FtF outperforms the random reference under both schedules, with HMean improvements of 1.41 and 1.12 percentage points, respectively. The larger improvement under cosine scheduling is consistent with the hypothesis that emphasizing higher-ranked samples is beneficial. We interpret these results as evidence that FtF’s selection benefit persists across schedules, with suggestive evidence of an additional interaction between importance ordering and learning-rate decay.

Superfiltering

As discussed in the background in § 3, superfiltering is the general term for using small models to perform efficient data filtering and selection for training larger models. One of our SOTA comparisons, S2L, is an example of such a method. We explore how superfiltering performs in FtF by using a far smaller model (Gemma 3 270M, instead of Gemma 3 12B IT) and its corresponding SAE as the filter pair. Note that we do not use the instruction-tuned version of the small model, evaluating whether a very small model without specialized knowledge of the target task can serve as useful filter pair.

ModelAlpacaEvalLeaderboardHMean
Ref (constant)1.9230.603.62
Constant2.5532.284.74
Ref (cosine)1.4531.502.77
Cosine2.2431.504.18

Table 4. Comparison of constant and cosine learning rate schedulers for FtF with k=5k = 5 and N=100N = 100. Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Ref randomly selects an NN-sample dataset and uses the indicated learning rate scheduler. Bold indicates the highest score in each column.

Success in this scenario would indicate that FtF may be: (1) widely accessible, with small models reducing the hardware barrier; and (2) generalizable for different post-training tasks, not requiring a model with specialized task knowledge to enable efficient training for that specialized task. The second point is of particular importance, as there are currently only a handful of publicly-available SAEs.

Results for the summary of the superfiltering analysis are in Table 5, and the detailed Leaderboard results are in Table 20. Although the 270M filter pair fails to outperform the 12B IT variant, it performs 40.1% better than the random reference and falls only 6.7% short of the large filter pair. Furthermore, the LLM of the 12B IT filter pair (responsible for the vast majority of the parameters in the filter pair) is over 44 times larger than 270M model. For many scenarios, a 6.7% performance deficit at only 2.3% of the parameter count may be a useful and valuable tradeoff.

ModelAlpacaEvalLeaderboardHMean
Reference1.4531.502.77
270M PT2.0930.533.90
12B IT2.2431.504.18

Table 5. Superfiltering analysis summary. AlpacaEval, Open LLM Leaderboard, and harmonic mean results comparing 270M PT and 12B IT in FtF with k=5k = 5 and N=100N = 100. Reference is trained on a randomly selected N=100N = 100 sample dataset. Results are percentages. Bold indicates the highest score in each column.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Reference43.0327.0135.0115.1131.2837.57
270M PT42.8325.5033.3314.1231.4835.93
12B IT43.3928.4135.7314.9531.4435.10

Table 20. Table 20. Superfiltering analysis. Detailed Open LLM Leaderboard results by task comparing 270M PT and 12B IT for FtF with k=5k = 5 and N=100N = 100. Results are percentages. Bold indicates the highest score in each column.

Sequence Aggregation

Aggregating feature activations across a full sample token sequence is an important consideration in assessing the semantic content of a training sample. In our main experiments, we apply last-token-pooling because the last token

is the only position which attends to every token in the sequence, due to the causal masking. However, SFT trains at token-level, so a training loss is computed at each token position, and thus is trained on the features present in each causal subsequence.

We perform an analysis to explore how mean-pooling (Mean) performs in comparison to last-token-pooling (LT). The summarized results are shown in Table 6, and the detailed Leaderboard results are in Table 19. Last-token-pooling demonstrates a small advantage over mean-pooling, scoring 0.5 percentage points higher in HMean, with the gap primarily from a 0.28 percentage point lead in AlpacaEval. Given this performance advantage in this analysis and the intuition behind evaluating the feature content only with the full sequence in context, last-token-pooling indeed appears to be the more reasonable selection.

ModelAlpacaEvalLeaderboardHMean
Reference1.4531.502.77
Mean1.9631.463.68
LT2.2431.504.18

Table 6. Sequence aggregation analysis summary. Comparison of last-token pooling (LT) and mean pooling (Mean) for FtF with k=5k = 5 and N=100N = 100. Results are percentages. Reference is trained on a randomly selected N=100N = 100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Reference43.0327.0135.0115.1131.2837.57
Mean43.2328.3335.4914.9031.4135.36
LT43.3928.4135.7314.9531.4435.10

Table 19. Mean sequence aggregation analysis details. Open LLM Leaderboard results by task comparing last-token pooling (LT) with mean pooling (Mean) for FtF with k=5k = 5 and N=100N = 100. Results are percentages. Bold indicates the highest score in each column, including ties at the displayed precision.

Ablation Studies

Diversity Resampling

In FtF, we willingly sacrifice a singular focus on the most important features for less important ones in an attempt to promote feature diversity and more complete coverage of the feature distribution present in the original dataset. Here we analyze whether this sacrifice does indeed lead to better model performance.

In Table 7 we present the task performance summaries of running FtF with diversity resampling (Div)—exactly as in our main experiments—and without resampling the features (No Div). Table 21 contains the detailed Leaderboard results. Our ablation analysis indicates that the importance sacrifice was indeed worth it, with Div outperforming No Div in every metric except MuSR. Our experiments were run with k=5k = 5 and α=0.2\alpha= 0.2, which translates to resampling a single feature. Future work may explore how to tune α\alpha across different values of kk.

ModelAlpacaEvalLeaderboardHMean
Reference1.4531.502.77
No Div1.9631.443.68
Div2.2431.504.18

Table 7. Diversity resampling ablation summary. Comparison of FtF with and without feature diversity resampling. Results are percentages. Reference is trained on a randomly selected N=100N = 100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Reference43.0327.0135.0115.1131.2837.57
No Div43.2328.3335.4114.9031.4135.36
Div43.3928.4135.7314.9531.4435.10

Table 21. Table 21. Diversity resampling ablation details. Open LLM Leaderboard results by task comparing FtF with (Div) and without (No Div) resampling for feature diversity. Results are percentages. Bold indicates the highest score in each column, including ties at the displayed precision.

Feature Entropy

As shown in Equation (2), the conditional entropy of active features plays a central role in determining the importance of a feature. Its inclusion in our feature importance was influenced by the findings of Zhang et al. [37] and their GRAPE methodology: training on data which is easier for the model to learn from appears to produce higher quality models. We applied that philosophy to feature importance in FtF and arrived at conditional entropy being our heuristic for a notion of ”ease of learning”.

In this ablation study, we analyze whether this heuristic actually adds any value to our methodology. We compare FtF with entropy included in the feature importance calculation (Ent), and without (No Ent). Table 8 contains the summary of the study, and Table 22 contains the detailed Leaderboard task results. Here we see very similar results to our diversity resampling ablation, as well as the sequence aggregation analysis, with our formulation of FtF performing slightly better than the alternatives explored.

ModelAlpacaEvalLeaderboardHMean
Reference1.4531.502.77
No Ent1.9631.463.68
Ent2.2431.504.18

Table 8. Feature entropy ablation summary. Comparison of FtF with and without entropy included in our definition of feature importance. Reference is trained on a randomly selected N=100N = 100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Reference43.0327.0135.0115.1131.2837.57
No Ent43.2328.3335.5314.9031.4135.36
Ent43.3928.4135.7314.9531.4435.10

Table 22. Table 22. Feature entropy ablation details. Open LLM Leaderboard results by task comparing FtF with (Ent) and without (No Ent) entropy in the feature importance definition. Bold indicates the highest score in each column, including ties at the displayed precision.

Feature Importance Across Datasets

After seeing consistency in the top-4 features identified by FtF in § 8.2, we ran a deeper analysis of the feature ranking consistency across the two datasets. In Table 9 we present the feature importance rank correlations across the 2 datasets. We run correlations for the full 65k feature ranking, as well as for a selection of subsets of the top features.

kkSpearman’s ρ\rhoUnion sizeFraction shared
51.00005100.0%
101.000010100.0%
500.62586766.0%
1000.210315545.0%
200−0.186134726.5%
500−0.437279341.4%
1000−0.31331,35764.3%
All0.907365,536100.0%

Table 9. Feature-ranking correlations between UltraChat 200K and Alpaca (Cleaned). Correlations are computed over the union of the two top-kk feature sets. A feature absent from one dataset’s top-kk subset is assigned rank k+1k + 1. Fraction shared is the intersection size divided by kk, (2k−∣U∣)/k(2k - |U|)/k, where UU is the union.

In general there is a high degree of agreement between these two datasets, including perfect alignment for the k=5k = 5 and k=10k = 10 subsets, and an overall correlation of ρ=0.9073\rho= 0.9073. As expected, larger subsets gradually reduce the amount of overlap, with the negative correlations being an artifact of our decision to assign absent features a rank of k+1k + 1. Interestingly, we observe somewhat of a recovery, with k=50k = 50 exhibiting a similar fraction of shared features as k=1000k = 1000. We interpret this as evidence to suggest that there may be general consistency in the features associated with instruction-following.

Consistency and similarity in important features across datasets can serve to amortize the cost of identifying the top-kk features by enabling the reuse of feature importance rankings. Future work may further explore the potential universality of important features for instruction-following and other model behaviors.

Discussion

Decoder Norms in Feature Importance Some SAEs, such as standard ReLU and Gated SAEs, require explicitly unit-normalizing the decoder vectors during training ([3]; [13]; [27]) while others, such as TopK, BatchTopK, and JumpReLU, allow for training SAEs without a strict unit-norm constraint ([28]; [9]; [4]). Our methodology for computing feature importance includes the decoder norms in its formulation for compatibility with all SAE variants. Many SAEs, including the Gemma Scope 2 SAEs ([22]) that we use in our experiments, are unit-normalized, even when not mathematically necessary.

Compositional Methodologies One interesting direction for future work is an exploration of how different data-efficient IT methods may be combined into a compositional methodology. For example, FtF can be used in conjunction with a method such as GRAPE to produce a filtered dataset which captures the most important features and contains samples easiest for the model to learn from.

Methodological Cost Comparison Although instruction-tuning is a one-time cost, it is worth considering how FtF compares to our SOTA comparisons in the costs associated with each methodology. Similar to GRAPE, the main costs of FtF are inference over the dataset. In our experiments, FtF took around 6.5 hours to perform data selection on UltraChat 200K, whereas GRAPE took about 14.5. Furthermore, GRAPE was only run using the existing dataset, and we elected not to use the version of GRAPE which generates and scores candidate responses from a selection of different LLMs, as that would have made our GRAPE experiments infeasible to run. The main costs associated with S2L are training the proxy model for multiple passes over the entire dataset. Although small proxy models help reduce the costs of S2L, the resource requirements for training a model for multiple full passes of a dataset are much higher than the inference-based methods.

Practical Utility of Feature Interpretability One of the key advantages of FtF over other SOTA methodologies such as GRAPE and S2L is its focus on feature interpretability. While interpretability does not inherently produce practical benefits, there are applications where a semantic understanding of data is essential. One application is in identifying coverage gaps in a dataset to inform precise collection and curation of additional data ([34, 17]). Another application is building connections and understanding the relationships between training data and resulting model behavior ([6] (Lu et al. 2024; Li et al. 2025)). FtF is also uniquely positioned to enable human auditing and intervention, similar to SHIFT ([21]), although the quality and accessibility issues of feature explanations described in § 8.2 are currently a barrier to further exploring this.

Limitations There are a few limitations of this work we aim to address. The first is the lack of exploration of different SAE variants in the filter pairs. Varying the LLM layer, feature width, and L0 values of the SAE can change the nature of the features identified by FtF and may be an interesting direction for future work. The second limitation is the size of the models in our experiments and our focus on a single data budget. Constraints in time/compute meant that we had to limit the scope of our experiments, and we felt that the 8B parameter scale and N=100N = 100 data budget presented a compelling case for FtF’s utility while being efficient enough for the resources at our disposal. The final limitation is the lack of an example of human-in-the-loop intervention with FtF. This work was heavily inspired by Marks et al. [21] and their SHIFT methodology, and it is unfortunate that we did not have the resources available to perform a similar intervention with FtF.

Conclusion

We introduced Follow the Features (FtF), an instruction-tuning data-selection method that uses SAE activation statistics to identify prominent features in a dataset and select samples that strongly express them. By separating the filter model from the training model, FtF enables feature-based selection across target model families. At a training budget of 100 samples with a fixed sequence length of 256 tokens, FtF achieves performance competitive with GRAPE and S2L across two model–dataset settings. Its selected subsets outperform full-dataset training on the reported harmonic-mean metric, while full-dataset training achieves higher Open LLM Leaderboard scores.

Our analyses also suggest that a substantially smaller filter model can retain much of the selection benefit in the evaluated setting. However, the semantic meaning of the highest-ranked features remains incompletely verified, and the practical value of feature-guided human intervention has yet to be demonstrated. Future work should examine these features directly, evaluate selection across broader training budgets, and measure the end-to-end cost of filtering and fine-tuning. FtF provides a foundation for investigating how explicit feature representations can support both effective data selection and more transparent dataset curation.

AI Usage Disclosure

We used generative AI tools for suggesting training and evaluation parameters, creating and editing code, summarizing existing literature, identifying research gaps, brainstorming different entropy variants, identifying relevant literature, formatting references, creating LaTeX tables and TikZ figures, and editing the paper. OpenCode was used as the agentic coding harness, and ChatGPT was used for literature and information retrieval, parameter suggestions, brainstorming, and making final edits to the manuscript.

Feature Entropy Implementation

We first compute a Gaussian approximation to the differential entropy of each feature:

HGauss(X)=12ln⁡(2πeσ2),(3)H_{\mathrm{Gauss}}(X) = \frac{1}{2}\ln(2\pi e\sigma^2), \tag*{(3)}

where entropy is measured in nats. The mean and variance are computed from positive SAE activations over all token positions, without sequence pooling. The resulting coarse importance score is

Ijcoarse=μj∥wjdec∥2max⁡(HGauss,j,10−8).(4)I_j^{\mathrm{coarse}} = \frac{\mu_j\lVert w_j^{\mathrm{dec}}\rVert_2}{\max(H_{\mathrm{Gauss},j},10^{-8})}. \tag*{(4)}

Only the 1,000 features with the highest coarse importance are retained for entropy refinement. For each retained feature, we collect at most 2,048 positive activations and estimate differential entropy using scipy.stats.differential_entropy. Before estimation, independent uniform noise in [−10−8,10−8][-10^{-8},10^{-8}] is added to break ties. The refined entropy divisor is floored at 0.01.

Features with at most 20 positive activations are not entropy-refined and use an entropy divisor of 1. Dead features have zero positive activations and therefore zero mean importance; no separate correction is applied to them. Constant or tied activations are handled only through the small jitter described above.

FtF Parameter Selection

We ran our sweeps with SmolLM3 3B (Base) (Bakouch et al., [1])– a model which balances high capability with a compact, efficient size– and the Alpaca (Cleaned) dataset (Taori et al., [30]). We fixed the parameters set using the results of these sweeps and applied them to our main experiments described in § 7. Thus, the results in § 8 reflect parameter generalization, as well as overall methodology generalization.

We searched for kk values in {1,5,10,100,200}\{1, 5, 10, 100, 200\} and used data budgets of {100,1,000,10,000}\{100, 1{,}000, 10{,}000\}. 200 was chosen for the highest kk value because we believe that to be a reasonable value for the upper bound of features that a human might review and audit. 10,000 was selected as the largest data budget following prior work done by Liu et al. [19], which established that 10,000 samples was a useful reduction for Alpaca. We chose to apply the fixed data budget, rather than relative to the size of the original dataset, as a realistic compute budget would be restricted by a fixed total number of training samples, not by a fixed percentage of the entire dataset.

To increase the efficiency of our sweep, we separated it into 2 phases: Phase 1 is a full pass over every parameter combination for a single seed, and Phase 2 is a pass over the top 2 performing kk values at each training budget for the full 3 seed sweep.

Phase 1: Wide, Single Seed

Since we only run Phase 1 for a single seed {6}\{6\}, we take the top 2 kk values at each dataset size for a targeted sweep in Phase 2. From Table 10 we see that at N=100N = 100 and N=1,000N = 1{,}000, k=1k = 1 and k=5k = 5 are the best performers and achieve virtually identical HMean results, including lifts of 2.59 and 1.21 over the random Reference at N=100N = 100 and N=1,000N = 1{,}000, respectively. At N=10,000N = 10{,}000, the separation between the Reference and best FtF variant is much tighter at only 0.34 for HMean at k=200k = 200. While these Phase 1 results are indicative of promising parameters, Phase 2 further evaluates the robustness of the best Phase 1 results.

ModelAlpacaEvalLeaderboardHMean
Baseline0.7928.881.55
Ref (All)2.4532.294.55
Dataset size: 100
Reference1.4531.502.77
k=1k = 12.9331.315.36
k=5k = 52.9331.315.36
k=10k = 102.1130.533.94
k=100k = 1001.4431.052.75
k=200k = 2002.1131.053.95
Dataset size: 1,000
Reference1.4731.412.82
k=1k = 12.1531.624.03
k=5k = 52.1531.644.03
k=10k = 102.1031.043.93
k=100k = 1000.8530.531.66
k=200k = 2000.9431.341.83
Dataset size: 10,000
Reference2.3732.184.42
k=1k = 11.0232.141.97
k=5k = 51.0232.141.97
k=10k = 100.9731.871.88
k=100k = 1002.4032.154.47
k=200k = 2002.5732.614.76

Table 10. Results for the sweep across dataset sizes and kk values for FtF Phase 1. Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Baseline is the starting checkpoint (i.e., before training) and Reference trains on NN randomly selected data samples, with All indicating a random ordering of the full dataset. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.

Phase 2: Targeted, All Seeds

The Phase 2 results in Table 12 paint a much less clear picture than Phase 1. At N=100N = 100, we see results similar to Phase 1, with both kk values achieving near-identical scores at least 1.4 higher in HMean than the Reference. However, there is almost no separation among the models trained at the N=1,000N = 1{,}000 data budget, and at N=10,000N = 10{,}000 the Reference outperforms each FtF model. For N=100N = 100 and N=1,000N = 1{,}000, we set k=5k = 5; even though k=1k = 1 demonstrates near-identical performance, we expect the complete absence of feature diversity to lead to poor generalization. For N=10,000N = 10{,}000, k=200k = 200 trades a 0.34 percentage point AlpacaEval deficit for a 0.26 percentage point advantage in Leaderboard, relative to the Reference. The lower scale results of AlpacaEval compared to the Leaderscores scores mean that HMean favors the Reference, despite having very close averages: 17.275 (Reference) vs. 17.235 (k=200k = 200).

ModelAlpacaEvalLeaderboardHMean
Baseline0.7928.881.55
Ref (All)2.4532.294.55
Dataset size: 100
Reference1.4531.502.77
k=1k = 12.2331.504.17
k=5k = 52.2431.504.18
Dataset size: 1,000
Reference1.4731.412.82
k=1k = 11.5031.432.87
k=5k = 51.5031.452.87
Dataset size: 10,000
Reference2.3732.184.42
k=100k = 1001.6831.763.18
k=200k = 2002.0332.443.82

Table 12. Results for the sweep across dataset sizes and kk values for FtF Phase 2. Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Baseline is the starting checkpoint (i.e., before training) and Reference trains on NN randomly selected data samples, with All indicating a random ordering of the full dataset. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.

We suspect that the diversity resampling parameter alpha=0.2alpha = 0.2 is the cause of the performance degradation as NN increases. It is likely too large for the larger datasets (e.g. this represents 40 important features being replaced with less important ones at k=200k = 200). Given the results of our sweep, we run our main experiments with k=5k = 5 and N=100N = 100.

Training Parameters

ParameterValue
Epochs1
Sequence length256
Per-device batch size32
Optimizeradamw_torch_fused
Learning rate2×10−42 \times10^{-4}
SchedulerCosine
Weight decay0.01
Max grad norm0.3
Warmup schedule3%
Precisionbf16

Table 14. Supervised fine-tuning configuration.

LoRA Parameters

ParameterValue
Rank rr16
LoRA α\alpha32
LoRA dropout0.05
Biasnone
Target modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Table 15. LoRA configuration.

Evaluation Parameters

ParameterValue
Judge modelQwen 3.8 27B FP8
Datasetalpaca_eval_gpt4_baseline
Dataset sourcetatsu-lab/alpaca_eval
AlpacaEval samples300
Leaderboard samplesall
Batch size4 (1 for Full FT)
Candidate maximum new tokens256
Judge maximum tokens512
Candidate temperature0.0
Judge temperature0.0
Top-pp1.0
Judge thinkingDisabled

Table 16. Evaluation configuration.

Additional Tables and Figures

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Baseline42.2724.9226.1416.1631.3732.41
Ref (All)45.3927.2941.4112.9929.8636.82
Dataset size: 100
Reference43.0327.0135.0115.1131.2837.57
k=1k = 143.1328.5235.0115.2631.4034.52
k=5k = 543.1328.5235.0115.2631.4034.52
k=10k = 1043.4626.7631.6515.1130.9035.32
k=100k = 10043.1727.3534.2913.9031.2336.38
k=200k = 20043.0326.6833.3315.1131.4836.64
Dataset size: 1,000
Reference45.1727.6834.1714.6530.4336.38
k=1k = 144.5928.0233.6915.7131.4736.24
k=5k = 544.5928.0233.8115.7131.4736.24
k=10k = 1044.3227.1832.7314.7331.3135.98
k=100k = 10045.1028.0229.7414.8031.2534.26
k=200k = 20044.7127.8534.0515.4831.4234.52
Dataset size: 10,000
Reference45.4427.0138.9714.2030.6636.77
k=1k = 145.7626.8538.4913.6731.1536.90
k=5k = 545.7626.8538.4913.6731.1536.90
k=10k = 1045.1727.0138.9713.0730.3536.64
k=100k = 10045.2527.4338.6113.9730.4837.17
k=200k = 20045.7627.0141.2515.2630.5435.85

Table 11. Table 11. Detailed Open LLM Leaderboard results by task for the FtF Phase 1 sweep, grouped by dataset size. Results are percentages. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.

ModelBBHGPQAIFEvalMATH-HardMMLU-ProMuSR
Baseline42.2724.9226.1416.1631.3732.41
Ref (All)45.3927.2941.4112.9929.8636.82
Dataset size: 100
Reference43.0327.0135.0115.1131.2837.57
k=1k = 143.2928.3335.7715.0631.4235.14
k=5k = 543.3928.4135.7314.9531.4435.10
Dataset size: 1,000
Reference45.1727.6834.1714.6530.4336.38
k=1k = 144.9227.8032.9715.0631.2636.60
k=5k = 544.9227.8033.0515.0631.2636.60
Dataset size: 10,000
Reference45.4427.0138.9714.2030.6636.77
k=100k = 10045.3627.0437.4114.4330.6335.71
k=200k = 20045.7327.2439.4515.1831.0935.98

Table 13. Detailed Open LLM Leaderboard results by task for the FtF Phase 2 sweep, grouped by dataset size. Results are percentages. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.

References

  1. [1]Bakouch, E., Ben Allal, L., Lozhkov, A., Tazi, N., Tunstall, L., Patiño, C. M., Beeching, E., Rouchen, A., Reedi, A. J., Gallouédec, Q., Rusal, K., Habib, N., Fourrier, C., Kydlicek, H., Penedo, G., Larcher, H., Morlon, M., Srivastav, V., Lochner, J., Nguyen, X.-S., Raffel, C., von Werra, L., and Wolf, T. SmolLM3: smol, multilingual, long-context reasoner, 2025. URL https://huggingface.co/blog/smollm3.
  2. [2]Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978-1-4503-8309-7. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922.
  3. [3]Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Corley, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.
  4. [4]Bussmann, B., Leask, P., and Nanda, N. BatchTopK Sparse Autoencoders, 2024. URL https://arxiv.org/abs/2412.06410.
  5. [5]Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations, 2023. _eprint: 2305.14233.arxiv.org/abs/2305.14233
  6. [6]Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., and Yeung-Levy, S. Describing Differences in Image Sets with Natural Language, 2024. URL https://arxiv.org/abs/2312.02974.
  7. [7]Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open LLM Leaderboard v2, 2024. URL https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.
  8. [8]Gammelgaard, M. L., Christiansen, J. G., and Søgaard, A. Large language models converge toward human-like concept organization, 2023. URL https://arxiv.org/abs/2308.15047.
  9. [9]Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskеver, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.arxiv.org/abs/2406.04093
  10. [10]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurull, G., Nguyen, H., Korevaar, H., Xu, H., Touryon, H., Zarov, I., Ibarra, I. A., Kloumаnn, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., Linde, J. v. d., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., Maaten, L. v. d., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzett, L., Oliveira, L. d., Muzzi, M., Pasupuletti, M., Singh, M., Paluri, M., Kardas, M., Tsim­poukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mिhaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y. Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtnhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangkui, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelenas, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajyugibpe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The Llama 3 Herd of Models, November 2024. URL http://arxiv.org/abs/2407.21783. arXiv:2407.21783 [cs].
  11. [11]Gurnee, W. and Tegmark, M. Language Models Represent Space and Time, 2024. URL https://arxiv.org/abs/2310.02207.
  12. [13]Huben, R., Cunningham, H., Smith, I. R., Ewart, A., and Sharkey, L. Sparse Autoencoders Find Highly Interpretable Features in Language Models. October 2023. URL https://openreview.net/forum?id=F76bwRSleK.arxiv.org/abs/2309.08600
  13. [17]Liang, Y., Zheng, T., Du, X., Zhang, G., Liu, J., Qu, X., Zu, W., Xing, X., Zheng, C., Ma, L., Wang, G., Zhang, Z., Huang, W., Yue, X., and Zhang, J. Aligning Instruction Tuning with Pre-training, 2025. URL https://arxiv.org/abs/2501.09368.
  14. [18]Lin, J. Neuronpedia: Interactive Reference and Tooling for Analyzing Neural Networks, 2023. URL https://www.neuronpedia.org.
  15. [19]Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, April 2024. URL http://arxiv.org/abs/2312.15685. arXiv:2312.15685 [cs].
  16. [21]Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2025. URL http://arxiv.org/abs/2403.19647. arXiv:2403.19647 [cs].
  17. [22]McDougall, C., Conmy, A., Kramár, J., Lieberum, T., and Rajamanoharan, S. Gemma Scope 2 - Technical Paper. September 2025.
  18. [23]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, March 2022. URL http://arxiv.org/abs/2203.02155. arXiv:2203.02155 [cs].
  19. [24]Park, K., Choe, Y. J., and Veitch, V. The Linear Representation Hypothesis and the Geometry of Large Language Models, July 2024. URL http://arxiv.org/abs/2311.03658. arXiv:2311.03658 [cs.CL].
  20. [25]Park, K., Choe, Y. J., Jiang, Y., and Veitch, V. The Geometry of Categorical and Hierarchical Concepts in Large Language Models, 2025. URL https://arxiv.org/abs/2406.01506.
  21. [26]Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., and Yurochkin, M. tinyBenchmarks: evaluating LLMs with fewer examples, 2024. URL https://arxiv.org/abs/2402.14992.
  22. [27]Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024a.arxiv.org/abs/2404.16014
  23. [28]Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders, August 2024b. URL http://arxiv.org/abs/2407.14435. arXiv:2407.14435 [cs].
  24. [29]Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by Layer: Uncovering Hidden Representations in Language Models, February 2025. URL https://arxiv.org/abs/2502.02013v2.
  25. [30]Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford.edu/2023/03/13/alpaca.html, 3(6):7, 2023. June 2024. URL http://arxiv.org/abs/2402.04833. arXiv:2402.04833 [cs].
  26. [32]Team, Q. Qwen3 Technical Report, 2025. URL https://arxiv.org/abs/2505.09388. eprint: 2505.09388.
  27. [33]Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.
  28. [34]Wang, F., Mehrabi, N., Goyal, P., Gupta, R., Chang, K.-W., and Galstyan, A. Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models, 2024. URL https://arxiv.org/abs/2410.05269.
  29. [35]Yang, X., Nie, S., Liu, L., Gururangan, S., Karn, U., Hou, R., Khabsa, M., and Mao, Y. Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder, March 2025. URL http://arxiv.org/abs/2502.14050. arXiv:2502.14050 [cs].
  30. [36]Yang, Y., Mishra, S., Chiang, J. N., and Mirzasoleiman, B. SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models, December 2024. URL http://arxiv.org/abs/2403.07384. arXiv:2403.07384 [cs].
  31. [37]Zhang, D., Dai, Q., and Peng, H. The Best Instruction-Tuning Data are Those That Fit. October 2025. URL https://openreview.net/forum?id=4jFSeKbADT.arxiv.org/abs/2502.04194
  32. [38]Zhao, H., Andriushchenko, M., Croce, F., and Flammarion, N. Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning,arxiv.org/abs/2402.04833

Paper details

Publication

vlnc:W_d9pcbeqe
v1 · latestHistory (1)

published · Fresh — added to the Hub within the last 30 days

added ·

Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

CC BY 4.0

Original source

Taxonomy

  • Computer Science

Authors

AI contributions

generative AI tools, OpenCode, ChatGPT · View statement →

Looking for related papers…
Contents