WorksComputer Science
Compositional Activation Steering Across Languages and Behaviors with Contextual Gates
Abstract
Activation steering has emerged as an effective and precise way to influence the behaviours of language models. However, composing multiple steering directions to control several behaviours simultaneously remains challenging, particularly in multilingual settings and with training-free steering vectors. We address this challenge with contextual steering, jointly training gates that adapt the strength of fixed steering directions to the current hidden state and interactions between interventions. Across a broad range of languages, behavioural attributes, and models, our approach outperforms constant-strength steering and four strong baselines, including prompting. It also better preserves general knowledge and keeps the model's internal computation closer to that of the prompted model. Our analysis further shows how jointly trained gates adapt their contributions during composition, explaining the mechanisms underlying effective compositional steering.
Introduction
Inference-time steering shifts a model’s activation trajectory toward a desired behaviour without updating its weights (Turner et al., 2025). It has been used to control a range of attributes, including style (Konen et al., 2024), factuality (Rimsky et al., 2024), truthfulness (Li et al., 2023), language identity [10], and refusal [1]. For language control in particular, recent work shows that steering remains effective across variation in input language and content [4, 10] (Sterz et al. 2025; Michaelov et al. 2026), suggesting that it can serve as a form of adaptation (Ostermann et al., 2026; Xu et al., 2026). Yet, most existing work considers only one attribute at a time. In realistic settings, models often need to satisfy several constraints simultaneously, such as generating
text in a specified language while adhering to a behavioural requirement.
Although several approaches to compositional steering have been explored [7] (Nguyen et al. 2025; Oozeer et al. 2025), effectively composing multiple attributes remains challenging, particularly in multilingual settings (Scalena et al., 2024; Kang et al., 2026). A central difficulty is that interventions that work well in isolation may interact when combined in the residual stream, making their effects difficult to predict. Moreover, fixed intervention strengths cannot adapt to the current context, where an attribute may already be expressed to varying degrees. Effective composition therefore requires interventions that account for both the current hidden state and the interactions between attributes.
Contextual steering offers a way to address these challenges by using gates to determine intervention strengths from the current state (Hsu et al., 2026). We extend this idea to compositional steering by jointly training such gates for multiple attributes while keeping their steering directions fixed, allowing each gate to adapt its contribution in the presence of other interventions. We evaluate this approach across languages, behavioural attributes, and models, analyse its effects on knowledge retention, and investigate why contextual steering works by examining internal representations, output distributions, and interactions between interventions.
Our contributions are as follows:
We establish a cross-lingual evaluation setup for composing language steering with general- and safety-related behavioural attributes, covering variation in both language and meaning.
We introduce a compositional steering method that jointly trains context-dependent gates while keeping steering directions fixed. Across 9 models, 9 target languages, and 14 behavioural directions, it outperforms constant-strength steering
and four additional strong baselines.
We show that contextual composition substantially improves knowledge retention over constant-level steering, while maintaining strong performance across languages, including languages unseen during training.
We analyse the internal dynamics of steering, showing that contextual interventions keep representations and intermediate output distributions closer to those of the prompted model, while jointly trained gates adapt their contributions during composition.1
Methodology
Two common bottlenecks in steering are layer selection and steering-strength tuning, which are typically resolved through manual sweeps (Turner et al., 2025; Konen et al., 2024; Rimsky et al., 2024). Our method avoids both, by applying interventions at every layer and learning a gate that determines the intervention strength from the current hidden state.2 Figure 1 provides an overview of our approach.

Figure 1. Methodology. (1) For each attribute, we extract a steering direction and select a constant intervention strength , yielding CAA. (2) CLAS keeps fixed and learns a contextual gate that determines the intervention strength from the hidden state, using cross-entropy (CE). (3) For multiple attributes, the gates are initialized from the corresponding single-attribute gates and jointly optimized with all interventions active. (4) At inference, the contextual interventions are composed in the residual stream, with each gate controlling its attribute’s contribution.
Single-Attribute Steering
For an attribute , we extract one steering direction per layer, , as the unit-normed difference in means (Marks and Tegmark, 2024; Wu et al., 2025) of the last-token residual activations between texts that exhibit and those that do not.
CAA. Constant-strength steering, referred to as CAA in prior literature (Rimsky et al., 2024; Radevski et al., 2026; Ghasemi et al., [7]), applies the same scalar intervention strength at every layer
where () is selected once per attribute by a grid search over a predefined range.3 The same value is used at all layers.
CLAS. We build on the contextual steering method of Hsu et al. (2026), referred to as CLAS, keeping fixed while learning a per-layer, per-attribute affine gate:
Here, is a context-dependent intervention in place of the constant . The gate is the only trainable component; the language model and steering directions remain frozen. Training minimizes next-token cross-entropy on the completion tokens of prompt-completion pairs exhibiting . Since the steering direction is fixed and the gate is a rank-one linear projection, the method requires little training data and introduces few trainable parameters (see Appendix § 21 for details on efficiency). Training data and hyperparameters are described in Appendices § 9.§ 1 and § 9.§ 2.
We additionally regularize the training objective with an anchor to prevent the gates from collapsing to minimal intervention strengths or not firing at all.4 Specifically, we add a quadratic penalty that pulls the gate outputs at each layer and token position toward a fixed target :
The anchor weight and the gate learning rate are swept over a small grid, and the best configuration is selected using a held-out score. Setting reduces the objective to unconstrained cross-entropy. Training uses small train and validation splits with early stopping on validation loss; further details are provided in Appendix § 9.§ 1.
Multi-Attribute Steering
We compose attributes by summing their interventions in the same residual stream:
Each is computed from the current before the interventions at layer , so all attributes are evaluated in the same residual state. The single-attribute gates from § 2.1 can therefore be composed without modification; we refer to this as the independent-gate baseline (§ 3). However, during inference, each gate receives a residual state that has already been affected by the other attributes, which differs from the states observed during its single-attribute training.
We, thus, propose an alternative, more robust composite-steering method by training the gates for each attribute combination jointly, with all composed interventions active during training. This allows each gate to adapt to the residual produced by the other attributes. We consider two variants.
(1) Joint . Let denote the gate parameters for attribute . Each gate is initialized from its single-attribute checkpoint,5 and all gates are then optimized jointly using a shared next-token cross-entropy objective, with the composed steering active during the forward pass:
Here, is the prompt and is a completion exhibiting all attributes, obtained as in Section § 2.1. The shared loss updates all gate parameters simultaneously, so each gate is trained against a residual carrying the compounded effect of every attribute’s earlier-layer deltas.
(2) Sequential . For multi-attribute combinations, we also consider a sequential variant. We keep the pretrained gate for the first attribute fixed and optimize only the second gate using the same cross-entropy objective, with both interventions active during the forward pass. Unlike joint setup, this approach allows the second gate to adapt to the first while leaving the first gate unchanged.
Experimental Setup
Models. We evaluate on nine instruction-tuned models spanning five model families and three sizes: Llama-3.1-8B and Llama-3.2-3B (Grattafiori et al., [9]), Qwen-2.5-{3B,7B,14B} (Yang et al., 2025b), Gemma-2-{2B,9B} (Team et al., 2024), Mistral-7B (Jiang et al., 2023), and Aya-Expanse-8B (Dang et al., [6]).
Attributes. We select attributes that have been widely studied in the steering literature, but mostly in isolation. We consider nine target languages from FLORES (NLLB Team et al., 2024): English, German, Japanese, Russian, Korean, Spanish, Italian, Arabic, and Polish. We additionally evaluate seven behavioural attributes: chain-of-thought, formality, humour, truthfulness (Lin et al., 2022), refusal (Arditi et al., [1]), bias (Parrish et al., 2022), and toxicity (Hartvigsen et al., 2022). Each single attribute is steered toward one of its two poles (e.g., funny vs. serious for humour). Two-way composition pairs each of the 9 languages with each of the 14 behaviour poles (126 compositions); three-way composition pairs each language with one style pole (chain-of-thought, formality, or humour) and one safety pole (refusal, truthfulness, bias, or toxicity), for 108 triples.
Evaluation. We evaluate every attribute and attribute combinations on held-out prompts from either CLaS-Bench (Gurgurov et al., [10]), a parallel collection of open-ended questions across 32 languages, or an attribute-specific dataset where needed: harmful and harmless instructions for jailbreak from Arditi et al. [1], TruthfulQA (Lin et al., 2022) for truthfulness, BBQ (Parrish et al., 2022) for bias, and ToxiGen (Hartvigsen et al.,
for toxicity. For most attributes, we use a mixed-language source distribution, with questions pooled across several languages rather than restricted to English. This provides a more challenging setting than using a single source language. Further details are in Appendix A.3.
Each attribute has a task-specific score, e.g. a FastText-LID language-fidelity score (Joulin et al., 2017), an LLM-judged register score for style attributes, or a safety-classifier or reference-match rate for the corresponding behavioural attributes. Every run additionally reports Output Relevance (OR), a judged score in measuring whether the completion still answers the question faithfully; OR is omitted when the task score already subsumes relevance. We use Qwen-3-8B (Yang et al., 2025a) as the judge for task scores and OR, and Qwen-3-Guard (Zhao et al., 2025) for safety-related attributes; the exact prompts are provided in Appendix E. We additionally validate these judgments using larger 30B-MoE and 32B judges, finding high agreement (Appendix E.2). The main scores reported for single- and multi-attribute results are the harmonic means (HM) of all active scores (task score(s) + OR), following Wu et al. (2025) and Gurgurov et al. [10]. The HM penalizes the final score when any individual sub-score is low.
Baselines. We compare against five baselines: (1) prompting () as a token-space reference (Appendix E.4); (2) constant-level steering (CAA; ) composed additively across independently-tuned attributes (Rimsky et al., 2024; Kang et al., 2026); (3) contextual CLAS () composed from independently trained gates; (4) K-Steering () (Oozeer et al., 2025), which derives intervention directions from gradients of a learned multi-label classifier; and (5) ORBIT () (Ghasemi et al., [7]), a training-free method that rotates activations within a shared multi-attribute subspace, gated per token. For each baseline, we perform extensive method-appropriate hyperparameter sweeps (Appendix A.1).
Experimental Results
We report single-, double-, and triple-attribute results, with significance tests in Appendix I.
Single-Attribute Results
We first evaluate language and behavioural attributes independently, averaging scores over all 9 languages and 14 behaviours for each model. Figure 2 compares prompting, constant-strength CAA, and contextual CLAS. CLAS consistently outperforms CAA across all nine models, with gains ranging from 1.3 to 21.9 points. The largest gains are observed for Aya-8B (21.9 points), Mistral-
Figure 2. Single-attribute steering, averaged over 9 languages and 14 behavioural directions, per model. We compare prompting, CAA (constant), and CLAS (adaptive). A “+” above a CLAS bar denotes its gain over CAA. Detailed per-model results are in Appendix F.
7B (20.0), and Llama-3.1-8B (17.0). The advantage is also consistent across attributes (Table 11 in Appendix § 14). Removing the anchoring objective substantially degrades performance (Appendix § 10), highlighting its importance for robust contextual steering in this setting.
| Attribute | Prompted | CAA | CLAS |
| German | 96.6 | 94.7 | 97.1 |
| Spanish | 99.1 | 95.1 | 97.9 |
| Italian | 96.9 | 87.6 | 97.1 |
| Russian | 96.0 | 80.2 | 97.1 |
| Japanese | 94.8 | 72.2 | 92.2 |
| Korean | 93.0 | 74.0 | 91.0 |
| Arabic | 93.0 | 64.1 | 86.1 |
| Polish | 94.1 | 82.2 | 93.5 |
| English | 88.7 | 92.1 | 85.7 |
| Chain-of-thought | 95.0 | 88.0 | 87.7 |
| No reasoning | 76.3 | 50.5 | 54.9 |
| Humour | 83.9 | 61.5 | 75.8 |
| Serious | 98.2 | 96.5 | 97.6 |
| Informal | 41.5 | 48.1 | 15.4 |
| Formal | 99.2 | 98.2 | 99.3 |
| Jailbreak | 8.4 | 77.7 | 90.5 |
| Over-refuse | 73.2 | 69.4 | 99.3 |
| Truthful | 76.9 | 77.7 | 80.2 |
| Deceptive | 73.0 | 65.0 | 87.7 |
| Unbiased | 96.4 | 93.5 | 95.6 |
| Biased | 57.6 | 33.5 | 50.8 |
| Non-toxic | 99.8 | 99.0 | 99.8 |
| Toxic | 15.4 | 20.6 | 61.0 |
| Average | 80.3 | 74.8 | 84.1 |
Table 11. Single-attribute steering (HM, %), averaged over all 9 models, per attribute.
CLAS also compares favourably with prompting, which we use primarily as a token-space reference rather than as a steering baseline. It outperforms prompting on seven of nine models and remains within two points on the other two. These results show that residual-stream interventions can provide competitive attribute control without modifying the model weights or relying solely on input-level instructions.
Multi-Attribute Results
We next evaluate the composition of language and behavioural steering across all languages and 14 behavioural directions. Table 1 reports scores averaged across behaviours and models, broken down by target language. Joint-CLAS substantially outperforms the other steering approaches, achieving an overall average of 82.5%, compared with 78.3% for sequential CLAS, 72.4% for prompting, and 49.6% for CAA. K-Steering achieves 43.2%, comparable to CAA, while ORBIT achieves 48.2%, outperforming K-Steering; both results are consistent with their reported performance ([7] (Oozeer et al. 2025)). Joint-CLAS also performs consistently across models, reaching at least 77.1% on every model and exceeding 80% on seven of nine. Independently trained gates, in contrast, do not reliably preserve single-attribute performance. Joint-CLAS addresses this mismatch by jointly training all gates with the composed interventions active, allowing each gate to adapt to the residual stream produced by the other attributes. Sequential CLAS also improves over independent composition but remains below Joint-CLAS, highlighting the benefit of jointly optimizing interacting attributes. We investigate these interactions and their effects on the residual stream in § 5. The results extend to three-attribute composition. Joint-CLAS again outperforms the competing methods across models, achieving at least 80% on every model; see the radar plots in Figure 3.
| Method | de | es | it | ru | ja | ko | ar | pl | en | Avg |
| Prompted | 76.5 | 75.7 | 75.0 | 73.2 | 63.4 | 71.7 | 69.2 | 73.6 | 73.3 | 72.4 |
| CAA | 57.2 | 60.2 | 51.8 | 32.7 | 42.3 | 54.6 | 26.5 | 50.2 | 70.6 | 49.6 |
| CLAS | 69.6 | 66.6 | 66.8 | 66.0 | 51.2 | 67.2 | 57.7 | 66.7 | 71.2 | 64.8 |
| Joint-CLAS | 85.1 | 85.3 | 83.7 | 83.5 | 80.4 | 79.7 | 79.7 | 82.1 | 82.5 | 82.5 |
| Seq-CLAS | 80.9 | 80.4 | 79.2 | 79.4 | 74.9 | 75.7 | 74.2 | 78.2 | 81.8 | 78.3 |
| K-Steer | 45.3 | 50.5 | 39.3 | 37.8 | 41.5 | 54.3 | 33.7 | 36.1 | 50.8 | 43.2 |
| ORBIT | 56.3 | 50.4 | 48.8 | 37.0 | 32.6 | 62.3 | 22.4 | 45.7 | 77.9 | 48.2 |
Table 1. Language and behaviour composition, averaged over 9 models, per target language and method (126 language-behaviour pairs per method). Detailed per-model results are in Appendix G.

Figure 3. Double- and triple-attribute steering performance and knowledge retention. Radar plots (A.1 and A.2) show the harmonic mean averaged over all languages and behaviours, per model. Four methods are shown for visual clarity. Double results are averaged over 126 language-behaviour pairs; triple results are averaged over 108 language-style-safety triples. The table (right-side) reports accuracy on each knowledge benchmark.
Importantly, the behavioural directions are extracted entirely from English-only data, yet compose successfully with language-specific directions across multiple target languages. This suggests that behavioural steering directions can transfer across languages, consistent with previous findings on shared multilingual representations ((Mousi et al., 2024; Riemenschneider and Frank, 2025; Yang et al., 2026)). Our results extend this observation to steering directions across languages and behavioural attributes.
The jointly trained gates and their corresponding directions also generalize to languages not seen during their construction. We evaluate the composition configurations on Chinese, Portuguese, Hindi, French, Norwegian, and Czech. Results in Appendix § 18 show that the learned interventions transfer beyond the languages used during training.
Knowledge Preservation
We additionally evaluate how well steering preserve the model’s general knowledge by measuring accuracy on MMLU ((Hendrycks et al., 2021)), ARC-Easy and -Challenge ([5]), and Winogrande ((Sakaguchi et al., 2020)). Figure 3 (right) reports results averaged over Llama-3.1-8B and Llama-3.2-3B. Joint-CLAS achieves an average accuracy of 63.9%, substantially outperforming composed CAA (41.6%) and single-attribute CAA (62.3%), although it remains below single-attribute CLAS (68.1%) and the unsteered baseline (70.7%). Three-way Joint-CLAS also outperforms its CAA counterpart with 61.2% over 39.2%. This advantage highlights the benefit of contextual steering: by adapting intervention strength to the current state, Joint-CLAS can avoid unnecessary interventions and better preserve general knowledge than constant-strength composition. Some degradation relative to the unsteered model is expected, as steering necessarily modifies the model’s representations to enforce additional output constraints.
Why (Compositional) Steering Works
We next investigate what happens inside the model when steering succeeds, both for individual attributes and under composition. We hypothesize that contextual steering is more compatible with the model’s natural computation than constant steering, allowing it to enforce target attributes while minimizing unnecessary changes to the residual stream.
Gate Characterization
Figure 4(A) shows the gate activity across layers and token positions for an example of language steering. Rather than intervening uniformly, the gate concentrates its activity at particular positions and depths. This suggests that it selectively reinforces the target attribute depending on the current context. Additional examples in Appendix § 20 show similar patterns.6
We next examine whether this selective intervention keeps the model’s internal computation closer to its natural behaviour. We use the prompted model as a reference, since it produces activations without residual-stream interventions. Figure 4(B) compares the projection of generated-token representations onto the steering direction across layers. For representation and steering direction , we compute . We then quantify the deviation from the prompted model as . Compared with CAA, CLAS produces a trajectory closer to the prompted model while still shifting the representation in the desired direction.
We similarly compare the model’s output distributions using the logit lens ((Nostalgebraist, 2020)). Figure 4(C) shows the KL divergence between the intermediate next-token distribution and that of the prompted model : . We average this divergence over the last five layers, where logit lens predictions are more closely aligned with the model’s
valuable future research direction. final output (Wendler et al., 2024). Lower divergence indicates that intermediate predictions remain closer to those of the prompted model. CLAS again exhibits lower divergence than CAA.
Finally, Figure 4(D) visualizes the geometry of generated-token representations at early, middle, and late layers using principal component analysis (PCA) (Pearson, 1901). We quantify the distribution shift between steered and prompted representations using the Earth Mover’s distance (EMD) (Villani, 2009) in the first two principal components, following Grant et al. [8]. Table 2 aggregates the three measures across attributes and models for both single- and multi-attribute steering. Across all three metrics, contextual steering remains closer to the prompted model than constant-strength steering. The differences are particularly pronounced under composition, where Joint-CLAS achieves the lowest values for every metric. This suggests that contextual gates do more than simply reduce the overall intervention strength: they adapt interventions in ways that better preserve the model’s existing computational trajectory.

Figure 4. Characterization of the single-attribute gates. Example for Arabic steering on Qwen2.5-7B-Instruct. (A) CLAS gate activity across layers and tokens. (B) Generated-token projection onto the steering direction across layers. (C) Logit-lens KL divergence from the prompted model across layers. (D) PCA of generated-token representations at three depth levels. Other attributes exhibit similar patterns; quantitative results are in Table 2.
| Metric (↓) | Baseline | CAA | (J)CLAS |
| Single: Proj. vs. instr. | 11.87 | 12.45 | 8.06 |
| Single: KL(prompted) | 9.14 | 7.64 | 5.23 |
| Single: EMD vs. instr. | 17.94 | 16.05 | 14.27 |
| Comp.: Proj. vs. instr. | 8.86 | 10.63 | 4.42 |
| Comp.: KL(prompted) | 10.59 | 9.02 | 5.98 |
| Comp.: EMD vs. instr. | 24.85 | 28.49 | 18.08 |
Table 2. Quantitative characterization of single- and multi-attribute steering, averaged over layers, attributes, and models. Proj. measures the projection shift relative to the prompted model; KL the logit-lens divergence over the last 5 layers; and EMD the Earth Mover’s distance over PC1 and PC2. Lower values indicate closer alignment with the prompted model.
Double Gate Co-adaptation
We next examine how gates change when two attributes are trained jointly. Figure 5(A) compares generation-time gate activity before and after joint training for an example composition. Joint training

Figure 5. Characterization of the composition. (A) Generation-time gate coefficients for German and Chain-of-thought before and after joint training. (B) Per-layer sign agreement and coactivation for the same composition. (C) Mean cosine similarity between steering directions across layers, averaged over all attribute pairs for Llama-3.1-8B-Instruct. Quantitative co-adaptation results are reported in Table 3, and orthogonality results for the remaining models are provided in Appendix § 12.
ing substantially changes intervention strengths, particularly in later layers, rather than simply preserving the independently learned gate activities. This indicates that the gates adapt to the presence of the other intervention. Figure 5(B) provides a more direct view of this co-adaptation. Joint training increases gate coactivation while reducing sign agreement (Table 3). We define coactivation as the mean product of the gate magnitudes, , across layers and generated tokens. Sign agreement is the fraction of layer-token pairs for which . The interference ratio measures the magnitude of the combined intervention relative to the sum of the individual magnitudes, , where values closer to 1 indicate less cancellation between the two interventions.
| Metric | CAA | CLAS (sep.) | CLAS-Joint |
| Gate coact. | – | 0.080 | 0.459 |
| Sign agree. | – | 0.713 | 0.394 |
| Interf. ratio | 0.736 | 0.765 | 0.793 |
Table 3. Composition diagnostics, averaged over all tested double- and triple-pairs for Llama-3.1-8B-Instruct. Gate coact. measures the mean product of the two gate magnitudes; sign agree. is the fraction of layer-token pairs where the gates have the same sign; and the interf. ratio measures the retained combined intervention strength relative to the sum of the individual strengths. Other models follow the same patterns.
Thus, the gates do not learn to avoid interacting with one another; instead, they tend to activate more strongly together, with more frequent opposing signs. At the same time, the interference ratio increases, indicating that a larger fraction of the combined intervention strength is retained despite these stronger interactions. Joint training, therefore, changes not only how strongly each attribute is applied, but also how their interventions combine in the residual stream.
These findings suggest that successful composition depends on coordinating intervention strengths, rather than simply combining individually effective interventions. This is consistent with the low cosine similarity between steering directions shown in Figure 5(C), indicating that the directions themselves are largely separated in representation space. Similar observations have been made for the composability of steering directions (Kang et al., 2026). The main challenge, however, lies not only in the directions themselves, but also in how the interventions interact in the residual stream, which JointCLAS addresses by adapting the gates to the composed setting.
Related Work
Activation Steering. Marks and Tegmark (2024) show that true and false representations exhibit a linear structure, motivating linear steering directions such as difference-in-means. Activation steering has since been applied to a range of behaviours, including style (Konen et al., 2024), factuality and sycophancy (Rimsky et al., 2024), truthfulness (Li et al., 2023), refusal [1], and language (Michaelov et al., 2026). These approaches typically use a constant steering strength and require manual selection of the intervention layer and strength. Hsu et al. (2026) instead introduce sensing-vector gates that adapt the steering strength to the context, demonstrating their effectiveness with directions learned by the Recursive Feature Machines [2].
Compositional Steering. Compositional steering has been explored in the embedding space through learned steering tokens (Radevski et al., 2026), in the unembedding space through learned output transformations [11], and in activation space (Scalena et al., 2024; Nguyen et al., 2025; Oozeer et al., 2025; Kang et al., 2026), which is the setting we study.
In activation space, Scalena et al. (2024) steer behaviours across attention heads using a gating function that compares the outputs of two parallel model passes through their KL divergence. Nguyen et al. (2025) jointly learn steering vectors and per-token intervention strengths with orthogonality, sparsity, and related objectives, operating at the attention-output level; such multi-objective optimization can be difficult to optimize robustly in practice. In contrast, we keep the steering directions fixed and train only the gates, keeping our method close to training-free steering while operating directly in the residual stream. Oozeer et al. (2025) propose K-Steering, which trains a non-linear multi-label classifier on activations and uses its gradients at inference time to derive intervention directions. Ghasemi et al. [7] propose ORBIT, which rotates activations within a shared multi-attribute subspace at inference time, gated per token, without training. Kang et al. (2026) instead compose constant-strength steering directions and show that composition can succeed when intervention layers and strengths are appropriately selected, but is not reliable across settings.
Other studies investigate composition as a side experiment. van der Weij et al. (2024) find that injecting multiple vectors at the same layer can be destructive, and Stolfo et al. (2025) find greater success when vectors are injected at separate layers. Cao et al. [3] show that vectors learned through preference optimization can also be combined, with partial success for a single combination.
Our Work. Among prior work, Scalena et al. (2024) and Kang et al. (2026) most directly study multilingual composition in activation space. Scalena et al. (2024) dynamically assign intervention strengths to individual properties using a reference model, without accounting for the residual changes induced by simultaneously steering other properties. Their method requires two forward passes per step at inference, operates at the attention level, and is evaluated on a single model across four languages. Kang et al. (2026) use constant-strength steering and rely on per-model, per-layer, and per-attribute grid search, finding that composition depends strongly on the choice of intervention layers and strengths.
In contrast, we train a contextual gate for each attribute and then jointly train the gates for each attribute combination. Each gate therefore learns in the same residual-stream context in which it is used at inference, while requiring no additional forward passes. We evaluate the resulting method across a much broader range of attribute combinations, languages, and models, and additionally provide a representational account of why it succeeds.
Conclusion
In this work, we showed that simple training-free steering vectors can be composed effectively when their interactions are accounted for through contextual steering and jointly trained gating functions. The resulting approach is robust across attribute combinations, languages, and models, while preserving more general knowledge than constant-strength composition. We further investigated why contextual steering is effective and found that gate-based interventions produce internal dynamics that remain closer to those of the prompted model, suggesting that contextual steering can achieve stronger attribute control while better preserving the model’s existing computation.
Limitations
Our approach requires computing steering directions and gates separately for each model; transferring them across models remains an interesting direction for future work. We also use next-token cross-entropy to train the gates, although contrastive or other task-specific objectives may prove more suitable. A further limitation is that joint training requires first training the single-attribute gates used for warm-starting, introducing an extra training stage, though the overall procedure remains efficient in practice (Appendix § 21). Our experiments also focus on a single method for extracting steering directions; although difference-in-means vectors are simple and effective in our setting, it would be valuable to investigate alternatives, including SAE-derived features and probe weights. Finally, we attempted to reproduce MAT-Steer (Nguyen et al., 2025), but found its multi-objective optimization difficult to tune in our setting; in particular, the second attribute often had little observable effect. This is consistent with the optimization challenges of this method discussed by Ghasemi et al. [7], although it does not establish that the method is ineffective more generally.
Ethical Considerations
The harmful attributes explored in this work (i.e., jailbreak, bias, and toxicity) are studied solely for scientific purposes, to understand and expose how easily current models can be steered toward harmful behaviours. Methods for improving model robustness against such steering should be developed alongside these techniques.
Acknowledgments
This work was supported by JST CREST Grant Number JPMJCR2565, Japan, JST BOOST Program Grant Number JPMJBY24H5, Japan, and by the European Union under the project IorAI, GA No. 101136646.
Appendix Contents
Gate Training Details
Hyperparameters
Single-attribute gates. Each gate is trained with and examples for 32 steps, keeping the checkpoint with the best validation loss. Longer training does not yield further improvements. The gate weight learning rate and anchor weight are swept jointly (Table 4), while the bias learning rate is fixed per model family.
| Model family | CAA range | Median norm | Bias LR | Gate LR grid | Anchor target | Anchor grid |
| Llama | 0.25–1.0 | 11.4–11.9 | 1e-1 | {3e-4, 1e-3, 3e-3} | 0.5 | {0, 0.25, 0.5, 1} |
| Aya | 0.25–1.0 | 43.5 | 1e-1 | {3e-3, 1e-2, 3e-2} | 1.0 | {0, 0.25, 0.5, 1} |
| Mistral | 0.1–0.25 | 6.2 | 1e-1 | {3e-3, 1e-2, 3e-2} | 0.25 | {0, 0.25, 0.5, 1} |
| Qwen | 2.0–5.0 | 66.0–100.2 | 5e-1 | {3e-3, 1e-2, 3e-2} | 3.0 | {0, 0.25, 0.5, 1} |
| Gemma | 4.0–16.0 | 203.3–307.3 | 5e-1 | {1e-2, 3e-2, 1e-1} | 10.0 | {0, 0.25, 0.5, 1} |
Table 4. Single-attribute CLAS gate hyperparameters for each model family. CAA range is the grid CAA’s own alpha is tuned over first. Median norm is the median residual-stream norm (baseline, no steering, over all layers and attributes); see detailed per-layer norms reported in Appendix C. The gate is anchored toward the constant value empirically found to steer that model best with CAA, its own tuned alpha, rather than an arbitrary fixed point. The gate LR grid is shifted per family since residual-stream norms differ substantially across models; a larger norm needs a correspondingly larger weight LR to move the gate by a comparable amount, and vice versa, but each family is still swept over a wide range rather than fixed to one value. Gate LR and anchor weight are swept jointly, best kept by validation.
Composite gates. Joint-CLAS and Sequential use the same architecture and the same per-family gate LR grid and training length as the single-attribute gates. Sequential additionally freezes the first attribute’s already-trained gate and only updates the second. K-Steer is a different mechanism, which has one shared linear classifier plus a gradient step at inference, not a trained per-layer gate, so its hyperparameters do not map onto the same table. ORBIT is training-free: its steering directions and joint subspace are computed in closed form from the same contrastive pairs, with a single boost coefficient calibrated per combination by grid search on a held-out tuning set; see Table 5.
| Method | Training | LR grid | Anchor | Notes |
| Joint-CLAS | , , 32 steps | same per-family grid as Table 4 | none | both gates updated together |
| Sequential | , , 32 steps | same per-family grid as Table 4 | none | first gate frozen |
| K-Steer | 30 epochs, 1 step/call | 1e-3 (Adam default) | n/a | , 8-point geometric sweep |
| ORBIT | training-free (closed-form directions) | boost grid: {0, 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3} | n/a | global mean- plane, module-median rescaling; best boost on tuning-slice HM |
Table 5. Composite and baseline steering hyperparameters. Joint-CLAS and Sequential use the same per-family gate learning-rate grids as Table 4, with the bias learning rate fixed per family and increased for models with larger residual-stream norms.
Training Data
Steering vectors. Each attribute’s steering direction is a difference-in-means vector, computed independently at each layer from 900 positive and 900 negative examples. Some attributes use a contrastive prompt template, while a few use existing contrastive datasets: harmful/harmless instructions from Arditi et al. [1] for jailbreak and true/false answer pairs from TruthfulQA (Lin et al. (2022)) for truthfulness. For language directions, negative examples are pooled across all 8 other languages rather than only English. This avoids learning a simple “is/isn’t English” distinction under our mixed-language evaluation setting. Language data is drawn from FLORES-200 (NLLB Team et al., 2024).
Single gate training. Target completions consist of 100 examples per attribute, generated by a shared teacher (Qwen3-30B-A3B-Instruct-2507; Yang et al. (2025a)) and filtered by a judge for on-target compliance. Jailbreak and toxicity instead use self-distillation from an uncensored model, since the aligned target model refuses these instructions. Language attributes require no distillation: their targets are direct English-to-target-language pairs from FLORES-200 (NLLB Team et al., 2024).
Compose gate training. Joint-CLAS and Sequential reuse the same distilled data as the corresponding single-attribute behaviour whenever the composition includes one. Language-language compositions require no additional distilled data.
Evaluation Data and Hyperparameters
All evaluation prompts come from CLAS-Bench ([10]), using 10 questions per language across our 9 studied languages. These prompts cover language and style attributes, including chain-of-thought, humour, and formality. Jailbreak, truthfulness, bias, and toxicity use separate held-out datasets: harmful/harmless instructions ([1]), TruthfulQA ((Lin et al., 2022)), BBQ ((Parrish et al., 2022)), and ToxiGen ((Hartvigsen et al., 2022)), respectively. Steering success is evaluated with Qwen3-8B ((Yang et al., 2025a)), except for jailbreak and toxicity, which use Qwen3Guard-Gen-8B ((Zhao et al., 2025)), as the general-purpose judge refuses to assess harmful or toxic text. Generation uses a batch size of 90 and up to 256 new tokens with deterministic decoding.
Gate Training Ablations
Anchors
We ablate the anchor regularization to assess its effect on training stability. Results are reported in Table 6.
| Model | Prompt | CAA | CLAS | CLAS (no anchor) | (anchor) |
| Llama-3.1-8B | 71.3 | 69.6 | 86.6 | 81.9 | +4.7 |
| Llama-3.2-3B | 77.0 | 75.2 | 78.6 | 75.3 | +3.3 |
Table 6. Anchor ablation for CLAS, averaged over 9 languages and 14 behavioural directions. “CLAS (no anchor)” removes the anchor regularization by setting its weight to zero. is the difference between CLAS and CLAS (no anchor), in percentage points.
Joint Gate Initialization
To facilitate joint gate learning and avoid one gate dominating the other, we initialize each gate from its corresponding single-attribute checkpoint. Table 7 shows that this initialization outperforms random initialization.
| Model | Prompted | CAA | Joint-CLAS | Joint-CLAS (random init) | (warm-start) |
| Llama-3.1-8B | 63.2 | 45.4 | 83.2 | 72.1 | +11.1 |
| Llama-3.2-3B | 64.9 | 47.3 | 77.1 | 67.8 | +9.3 |
Table 7. Joint gate initialization ablation, averaged over all 126 language-behaviour pairs. Joint-CLAS is warm-started from the independently trained single-attribute gates, while Joint-CLAS (random init) uses random initialization. Both use the same winning gate learning rate for each pair. is the difference between warm-start and random initialization, in percentage points. Prompted and CAA results are shown for reference.
Residual Norms

Figure 6. Residual norms of the raw difference-in-means vectors for each model and attribute. Norms vary substantially across models and generally increase with depth, consistent with observations for models using standard residual streams (Team et al., 2026).
Orthogonality
References
- [1]Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, volume 37, pages 136037–136083. Curran Associates, Inc.arxiv.org/abs/2406.11717
- [2]Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Mikhail Belkin. 2026. Toward universal steering and monitoring of ai models. Science, 391(6787):787–792.DOI
- [3]Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In Advances in Neural Information Processing Systems, volume 37, pages 49519–49551. Curran Associates, Inc.arxiv.org/abs/2406.00045
- [4]Cheng-Ting Chou, George Liu, Jessica Sun, Cole Blondin, Kevin Zhu, Vasu Sharma, and Sean O’Brien. 2025. Causal language control in multilingual transformers via sparse feature steering. In ACL 2025 Student Research Workshop.arxiv.org/abs/2507.13410
- [5]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457.arxiv.org/abs/1803.05457
- [6]John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.arxiv.org/abs/2412.04261
- [7]Narges Ghasemi, Amir Ziashahabi, Salman Avestimehr, and Jonathan May. 2026. Orbit: Training-free multi-attribute behavioral steering via orthogonal subspace rotation. arXiv preprint arXiv:2606.22357.arxiv.org/abs/2606.22357
- [8]Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, and Christopher Potts. 2026. Addressing divergent representations from causal interventions on neural networks. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2511.04638
- [9]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [10]Daniil Gurgurov, Yusser Al Ghussin, Tanja Baeumel, Cheng-Ting Chou, Patrick Schramowski, Marius Mosbach, Josef van Genabith, and Simon Ostermann. 2026. CLAS-bench: A cross-lingual alignment and steering benchmark. In Findings of the Association for Computational Linguistics: ACL 2026, pages 21591–21628, San Diego, California, United States. Association for Computational Linguistics.
- [11]Chi Han, Jiuliang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16410–16430, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2305.12798
