betaFeedback

WorksComputer Science

HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

Abstract

LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7×\times larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.

Introduction

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) underpin applications such as document understanding (Deitke et al., [12]), conversational assistants (OpenAI, [32]), and agent systems (Google, [17]). Transformer-based architectures (Vaswani et al., [38]) exhibit robust scaling, driving the creation of larger foundation models. At the same time, their deployment spans heterogeneous hardware from data-center GPUs to resource-constrained edge devices with different latency, compute, and model-size requirements. Training separate models for each configuration is computationally prohibitive, as each variant must be trained independently from scratch. Additionally, conventional transformer-based models are inherently static at inference time, which means their parameters are fixed after training, limiting their ability to target heterogeneous systems. Recent elastic and nested methods address this by training multiple models as one, but most focus on a single component such as the feed-forward network (FFN) for size reduction or require a multi-step extraction process of fixed model checkpoints [19, 13, 38, 7]. In this paper, we introduce HydraLLM1, a unified approach to adaptive slicing in LLMs and LVLMs by merging elastic architectural principles with efficient compression techniques while leveraging an already pre-trained static LLM.

We show that HydraViT-style [19] end-to-end adaptive slicing transfers successfully to decoder-only LLMs, producing a single checkpoint that remains effective across several runtime width budgets. With 75% of heads active, HydraLLM improves Trivia-QA [23] accuracy by 2.2 percentage points over a baseline configuration that has 17.25% more parameters. With 50% of heads active, the multimodal model reaches a CLIPScore of 66.9, surpassing the 65.3 score of the 7×7 \times larger Gemma 3 4B [15] model. With only 25% of heads active, importance-based reordering improves post-training perplexity by 16.2%. This 25% configuration increases token-generation throughput by 75.29% compared to the full model. In summary, we contribute:

  • End-to-end slicing for decoder-only LLMs. HydraLLM exposes a single head-count control that selects consistent prefixes of the embedding, attention, FFN, and output dimensions, producing nested decoder subnetworks from one checkpoint.

  • Unified training pipeline for elastic subnetworks. We introduce a single training recipe to convert a static pre-trained LLM to an adaptively sliceable LLM while remaining competitive on text-only benchmarks.

  • Elastic multimodal projection for LVLMs. We extend adaptive slicing to LVLMs by employing an elastic projection module that maps visual features into text embedding spaces of varying dimensionality.

Related Work

We discuss related work on nested representations and elastic architectures, transformer compression and submodel extraction, and projection-based multimodal models. We focus on how these approaches differ from HydraLLM’s end-to-end runtime slicing of a pre-trained LLM and its extension to an elastic multimodal projection.

Kusupati et al. [25] introduce multi-granularity, coarse-to-fine representations for embedding models. The same matryoshka nesting principle reappears in later work introducing elasticity within other transformer components. Cai et al. [7] propose the Once-For-All (OFA) methodology, decoupling architecture search from training by progressively shrinking the convolutional neural network (CNN) kernel size, channel width, and model depth. OFA targets the search and extraction of specialized CNN architectures for different deployment constraints. In contrast, HydraLLM does not perform architecture search or progressive shrinking: it converts a pre-trained LLM into one elastic checkpoint whose embedding, attention, FFN, and output widths can be scaled jointly at runtime through the active head count. Devvrit et al. [13] propose MatFormer, a natively elastic transformer that introduces a Matryoshka-like nested structure specifically only inside the FFN block. Haberer et al. [19] target embeddings, attention heads, and FFN layers for their nested ViTs. Training employs stochastic subnetwork sampling by drawing one head count kk per step and updating only the corresponding subnetwork, which biases optimization toward earlier heads, implicitly ordering them by importance [19]. We apply the same training and architectural approach to LLMs and extend it to the vision domain. Cai et al. [8] and Taghibakhshi et al. [37] extend the nested approach in the attention and FFN components with routers in their FLEXTRON and Nemotron Elastic models. Both approaches are text-only. FLEXTRON slices only attention heads and FFN, and Nemotron Elastic targets an architecture distinct from our multi-query attention (MQA) [36] transformer. Muralidharan et al. [31] design a static submodel extraction approach by applying iterative structured pruning on a pre-trained model, creating MINITRON. They obtain separate models that are not suitable for elastic inference, like ours.

ViTs [14] adapt the transformer encoder to images by splitting an image into non-overlapping patches, flattening each patch, and projecting it into the model’s embedding space to obtain patch embeddings. For multimodal models, Liu et al. [28] connect a pre-trained CLIP [34] vision encoder to an open-source LLM via a projection layer that maps visual features into the LLM’s text embedding space, producing so-called soft tokens. They merge the soft tokens with text tokens and process them jointly through self-attention. Deitke et al. [12] build on these concepts with the Molmo family, while Marafioti et al. [30] propose SmolVLM as a series of compact multimodal models. Gemma Team et al. [15] release Gemma 3 with a tailored SigLIP [44] vision encoder. We follow the training and architectural principles of these multimodal models and utilize insights from their training approaches. However, we employ the elastic LLM and multimodal projection.

HydraLLM: Architecture and Pipeline

This section introduces HydraLLM’s design and the pipeline for converting a static pre-trained LLM into an elastic text and multimodal model. Specifically, we describe the text architecture and width-control in § 3.1, the static-to-elastic conversion procedure including reordering and the optimization objective details in § 3.2, and the multimodal assembly in § 3.3. Our pipeline starts from a static pre-trained text checkpoint and converts it into an elastic decoder whose active width is controlled at runtime; see Figure 2. We then extend the same elasticity principle to the multimodal setting with an elastic projection module.

Figure 2. Pipeline overview. Starting from a static text checkpoint, such as Gemma 3 270M, the sliced axes can first be reordered to improve prefix quality. The resulting checkpoint is then trained as an elastic text model whose runtime width is controlled by the active number of query heads kk. Finally, we combine the elastic text decoder with, for example, Gemma 3 4B vision components to form a multimodal model whose projection output follows the same active width.

Runtime Width-Control

HydraLLM adapts HydraViT’s [19] head-stacking principle to decoder-only language models by using the number of active query heads kk as a single runtime width-control parameter; see Figure 1. A subnetwork keeps the first kk query heads together with the corresponding prefixes of the embedding channels, output projection, and FFN neurons. Each sliced dimension scales linearly with the number of heads kk in relation to the total number of query heads HqH_q: the active embedding width is dE,active=k/Hq×dEd_{\mathrm{E,active}} = k/H_q \times d_{\mathrm{E}}, with the FFN dimensions following the same proportion. This produces a strict nesting of subnetworks that share parameters across configurations. At inference, selecting kk computes only the corresponding prefix, so one checkpoint serves all width budgets. We target MQA [36] in our design, in which all query heads share a single key-value head.

Figure 1. HydraLLM width slicing. The number of active heads determines the embedding dimension, the attention projection, and the feed-forward network width, producing nested subnetworks from a single parameter set.

Static-to-Elastic Conversion

A static pre-trained checkpoint is not immediately suitable for prefix slicing. In an unmodified checkpoint, units of a component are usually not sorted by their importance. Since prefix slicing always retains the first units and discards later units, naïve slicing can remove important parameters and severely degrade smaller subnetworks. HydraLLM employs a two-stage procedure to convert static models into elastic ones. First, the pipeline reorders the checkpoint offline, moving important units toward the beginning of each sliced dimension so that prefix slicing retains the most informative parameters. Second, it optimizes the checkpoint using stochastic subnetwork training: during each training step, the pipeline samples a single kk and updates only the corresponding active subnetwork. Since earlier units are active for more sampled values of kk, they receive proportionally more gradient updates, reinforcing the importance ordering and adapting the shared parameters to all configured widths [37, 31, 8, 19] (Devvrit et al. 2024).

Importance Scoring and Reordering

The reordering step occurs before any training and requires the definition of importance. We compare three importance measures: Activation magnitude [31], variance-weighted consumer strength (VxC) adapted from [39], and CETT [45], which scores units by their normalized contribution to the component output and therefore aligns well with prefix slicing. We estimate unit importance from activation statistics collected during a forward pass over a calibration text corpus. After computing the importance scores, we permute the corresponding tensors so that important units appear earlier along the sliceable dimensions: For embedding-width reordering, we apply one global permutation to the token embedding channels and propagate it consistently through the RMSNorm weights, the input columns of the Q/K/V projections, the output rows of the attention output projection, the input columns of the FFN gate and up projections, the output rows of the FFN down projection, the final normalization layer, and, if untied, the language-model head. For attention reordering, we permute the query heads and the corresponding attention-output input blocks. For FFN reordering, we permute FFN neurons by moving the rows of the gate and up projections and the corresponding columns of the down projection. This reordering step preserves the function of the full model.

Optimization Objective

After reordering, we train the model. The training objective we employ is a combination of autoregressive language modeling, logit regularization, and teacher supervision. The primary task loss, LCE\mathcal{L}_{\mathrm{CE}}, is standard next-token cross-entropy. We additionally add a zz-loss term Lz\mathcal{L}_z, weighted by λz\lambda_z [9], and logit-based knowledge distillation [22]. Let ztz_t and zsz_s denote the teacher and student logits, respectively, and let ptτ=softmax⁡(zt/τ)p_t^\tau= \operatorname{softmax}(z_t/\tau) and psτ=softmax⁡(zs/τ)p_s^\tau= \operatorname{softmax}(z_s/\tau) denote their temperature-scaled output distributions. The training objective is:

LKD=τ2KL⁡(ptτ∥psτ),\mathcal{L}_{\mathrm{KD}} = \tau^2 \operatorname{KL}\left(p_t^\tau\mathbin{\|} p_s^\tau\right),
L=LCE+λzLz+λKDLKD(1)\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda_z\mathcal{L}_z + \lambda_{\mathrm{KD}}\mathcal{L}_{\mathrm{KD}} \tag*{(1)}

We keep the teacher model frozen throughout training.

Elastic Multimodal Assembly

We construct the multimodal variant of HydraLLM by combining the language decoder with a static pre-trained SigLIP-based vision encoder [15, 44]. The key difference from a conventional projection-based LVLM is that HydraLLM also makes the projection elastic. Let dPd_P denote the projection output width. We enforce dP,active=dE,actived_{P,\mathrm{active}} = d_{E,\mathrm{active}}, so the same runtime parameter kk controls both the active text decoder width and the projection output width; see Figure 3. This creates a single elasticity axis across modalities: selecting a smaller text subnetwork also selects a correspondingly smaller visual projection output.

Figure 3. Elastic LVLM architecture. The projection output width follows the active text width.

Experiment Setup

We describe our training data and evaluation protocol in the following subsections 4.1 and 4.2, respectively.

Training Recipe

We base our experiments on Google Gemma 3 270M [15], with k∈{1,2,3,4}k \in\{1, 2, 3, 4\}; see Appendix § 8.§ 2. For text-only experiments, we use the frozen full-width Gemma 3 270M as the teacher model.

Most text experiments train on a subset of FineWeb-Edu 350BT [29]. We concatenate text samples with an <eos> separator and split the text into fixed-length token blocks. We employ non-uniform runtime width sampling of kk, placing additional (×3\times3) emphasis on the full-width model compared to the submodels; see Appendix § 8.§ 5. In contrast to the packed text pre-training setup, the optional text fine-tuning stage uses text samples as they appear, without fully packing the sequence length; see Appendix § 10.§ 1.

For multimodal training, we use the elastic HydraLLM Gemma 3 270m model combined with the vision encoder and projection module initialized from the Gemma 3 4B multimodal checkpoint. The teacher for this training is the static and frozen Gemma 3 4B model. We employ uniform head sampling and separate learning rates for the text backbone, the projection module, and, optionally, the vision encoder. We configure separate peak learning rates and number of warmup steps for the text backbone, projection and vision encoder; see Appendix § 8.§ 5. Multimodal training samples from PixMoCap, the COCO Recaptioned dataset, LLaVA Pretrain 558K ([27]), PixMo Ask Model Anything, PixMo Cap-QA, CoSyn400K, and text-only data from Wikipedia ([40]) and FineWeb-Edu. See Appendix § 8.§ 3 for detailed composition. Projection alignment utilizes 128,000 captioning samples from the aforementioned captioning datasets.

Metrics and Evaluation Datasets

We employ a mix of metrics and benchmarks to evaluate models and derive conclusions. For text evaluation, we report ARC-Easy ([10]) and TriviaQA ([23]). We evaluate ARC-Easy with both raw accuracy and answer-length-normalized accuracy. We evaluate TriviaQA with five-shot prompting and exact-match scoring. We measure perplexity ([24]) on samples of plays by William Shakespeare ([6]) as a general-purpose indicator of language modeling quality.

For multimodal evaluation, we report CLIPScore ([21, 34]) on COCO validation images and multiply the score by 100 for readability. We additionally use qualitative captioning examples to inspect generation quality across different head configurations. The standard error on text- and CLIPScore-benchmark results denotes standard errors over evaluation examples.

We measure throughput with llama-bench from llama.cpp ([16]), with prompt-processing of 2048 tokens and autoregressive generation of 512 tokens on an NVIDIA RTX PRO 4000. This setup allows measuring prompt-processing and token generation speed improvements.

Evaluation

We evaluate the static-to-elastic conversion pipeline in terms of reordering in § 5.1, compare HydraLLM with MatFormer-style ([13]) FFN slicing in terms of accuracy and throughput in § 5.2, and assess the quality of multimodal extensions in § 5.3.

Effect of Importance Reordering

To determine the effect of reordering, we measure perplexity for each head configuration and compare it to the unmodified pre-trained model in Table 1. We denote the kk-head configurations as H1-H4. Reordering clearly improves the perplexity of sub-models when utilizing the magnitude metric for embeddings combined with CETT for heads and FFN, or the magnitude metric for all components. Employing only VxC provides an improvement solely for the H1 configuration.

RecipePerplexity ↓
H1H2H3H4
no reordering168.6M1.1M41.3k49.0
mag-cett-cett37.2k2.43k1.22k49.0
mag-mag-mag8.45M378.9k1.54k49.0
vxc-vxc-vxc38.0M4.87M60.4M49.0

Table 1. Perplexity of reordered checkpoints measured on samples of plays by William Shakespeare. Magnitude for embeddings combined with CETT for heads and FFN yields the strongest subnetwork perplexities, while VxC consistently hurts.

After reordering, we train the reordered models for 20,000 steps (∼\sim1B tokens) and measure their perplexities again. After training, perplexity improves for all heads, regardless of the starting point; see Table 2. However, the CETT importance metric for heads and FFN with magnitude importance for the embeddings provides the best results for the submodels. VxC slightly edges out CETT on the four-head configuration. For H4 of the not reordered model, continued training reduces perplexity from 49.0 to 45.2. However, this improvement does not consistently transfer to downstream benchmarks. Figure 4 and Figure 5 from the next § 5.2 also show the ARC-Easy and TriviaQA

RecipePerplexity ↓
H1H2H3H4
no reordering123.977.456.745.2
mag-cett-cett106.671.654.444.3
mag-mag-mag110.171.954.444.8
vxc-vxc-vxc118.171.755.044.1

Table 2. Perplexity of reordered and trained (20k steps) checkpoints measured on samples of plays by William Shakespeare. Magnitude for embeddings combined with CETT for heads and FFN yields the strongest subnetwork perplexities.

Figure 4. ARC-Easy accuracy as a function of active parameter size for HydraLLM and MatFormer configurations. HydraLLM covers a wider active parameter range than MatFormer while offering competitive accuracies.

Figure 5. TriviaQA accuracy as a function of active parameter size for HydraLLM and MatFormer configurations. HydraLLM is stronger at H3 clearly outperforming the similarly sized MatFormer W0.25, while MatFormer surpasses HydraLLM at the full configuration.

accuracy of the pre-trained and the no-reordering model trained for 20,000 steps. The benchmarks actually show a lower accuracy compared to the static baseline. We provide more benchmark results in Appendix § 9.2.4.

HydraLLM vs. MatFormer

We compare HydraLLM to MatFormer-style FFN slicing, which we reimplement using the same Gemma 3 270M checkpoint, in terms of accuracy and runtime throughput. The hyperparameters for both models are the same, apart from the width selection mechanism. In Figure 4 and Figure 5, we label the MatFormer FFN configurations as W0.25-W1.0, corresponding to the fraction of FFN neurons active. Because MatFormer only alters the FFN component, the smallest MatFormer config-

uration, W0.25, still exceeds the second-largest HydraLLM configuration, H3, by 17.25% in parameter count. The raw ARC-Easy accuracy of W0.25 is slightly higher than that of H3, though not statistically significant. The log-probability-normalized accuracies are virtually the same. The HydraLLM approach has a clear advantage with similar accuracies, but a wider parameter size coverage. The ten-shot prompting HellaSwag [43] results lead to the same conclusion, therefore we provide them in Appendix § 9.§ 1 only.

The results on TriviaQA differ. While the W0.5 and W0.75 MatFormer configurations interpolate smoothly between the H4 and H3 HydraLLM configurations in Figure 5, H3 achieves 2.21 percentage points (×2.51\times2.51) higher accuracy than W0.25, although they are similar in size. Conversely, W1.0 outperforms H4 by 0.68 percentage points. The HydraLLM approach has a broader parameter spread, starting from 49.34M parameters, compared to MatFormer, starting from 215.01M parameters, while achieving competitive results with similar submodel sizes. However, submodels H1 and H2 only achieve close to 0% accuracy on this benchmark.

The prompt-processing improvements from smaller model configurations displayed in Table 3 are very modest. The improvement from the H4 to H1 configuration ranges from 9.78% to 11.61%, and the improvement from W1.0 to W0.25 ranges from 1.03% to 3.1%. The Gemma 3 270M model comprises ∼\sim170 million embedding parameters and ∼\sim100 million transformer block parameters. Since the embeddings are mainly a lookup operation, they do not contribute much to throughput, but they account for a large portion of the parameter savings in the HydraLLM approach [15, 38].

ConfigHydraLLMMatFormer
PrefillDecodePrefillDecode
H1 / W0.2555,891.31,057.051,549.1669.6
H2 / W0.5052,851.7785.951,353.8648.5
H3 / W0.7551,165.4594.249,945.2623.4
H4 / W1.0050,487.9603.050,509.8602.3

Table 3. Throughput in tokens per second for HydraLLM and MatFormer variants. Prompt-processing of 2048 tokens. Decode 512 tokens. Prompt-processing throughput gains are modest for both methods, while token-generation scaling is much larger for HydraLLM than for MatFormer.

For token generation, MatFormer shows consistent speed-ups with shrinking parameter count. HydraLLM produces fewer tokens per second with the H3 configuration compared to H4, which we hypothesize stems from kernel-shape optimizations in llama.cpp. The improvement in token generation speed from the largest to the smallest configuration is 75.29% for HydraLLM and 11.17% for MatFormer. The main takeaway is that prompt-processing gains are minor, while token-generation scaling is much greater, but does not scale one-to-one with parameter count. The reported throughput gains therefore stem mostly from smaller embeddings and query, output, and FFN projections, because MQA Shazeer, [36] prohibits key-value cache savings from width reduction.

Multimodal Pipeline Results

The first stage in the multimodal training pipeline is aligning the projection with the text embedding space. The projection-only block of Table 4 shows that the untrained projection already produces a non-trivial CLIPScore [21, 34] across all head configurations, indicating that it inherits useful structure from the Gemma 3 4B vision tower. Training the projection in isolation yields only modest improvements over the untrained baseline. During training, we observe a plateau in the loss after 300 to 350 steps, depending on the head configuration; see more details in Appendix § 10.§ 2. The rapid plateau indicates a lack of capacity in the projection layer, caused by the much smaller projection due to the lower text embedding dimension of the Gemma 3 270M model compared to Gemma 3 4B. The projection has 25% of the parameters of the Gemma 3 4B model and therefore possesses much less representational power [15].

Training RecipeH1H2H3H4
No training
elastic32.431.532.032.7
Gemma 3 4B–––65.3
Projection only (500 steps)
non-elastic–––28.9
elastic38.231.431.133.5
elastic, fine-tune34.834.231.032.1
Projection + text model (2,500 steps)
elastic46.353.958.558.8
pre-aligned, non-elastic–––62.5
pre-aligned47.456.961.361.0
pre-aligned, fine-tune47.157.459.161.1
All parameters (2,500 steps)
elastic50.259.463.163.9
pre-aligned51.160.761.362.5
pre-aligned, fine-tune51.260.461.563.6
From text fine-tune + pre-aligned (16,000 steps)
Projection + text model59.165.465.168.0
All parameters62.666.968.669.6

Table 4. CLIPScore (×100\times100) across head configurations for each multimodal training recipe. Bold marks the best score per head configuration. non-elastic: starting from pre-trained Gemma 3 270m and only head configuration k=4k=4 is trained; elastic: starting from elastic text model and all head configurations are trained stochastically; fine-tune: an additional text-only fine-tuning step is applied before multimodal training; pre-aligned: the projection is pre-aligned on captioning data before joint training; Gemma 3 4B: the original frozen Gemma 3 4B multimodal model provided as a baseline.

Joint training of the projection with the text model substantially improves the CLIPScore; see the projection-plus-text-model block of Table 4. The models trained from an already projection-pre-aligned checkpoint perform best across all head configurations, especially at H2, implying that aligning the projection before joint training of the projection and text model is beneficial. When training all model parameters together, the all-parameters block shows that results are similar across different starting points, indicating that joint optimization can compensate for an inferior starting projection.

Scaling the joint projection and text model training, as well as the full-parameter training, from 2,500 to 16,000 steps further closes the gap to the Gemma 3 4B reference; see the bottom block of Table 4. The full-parameter trainable run matches or exceeds Gemma 3 4B at H2, H3, and H4 and reaches 62.6 at H1. The results suggest that, on CLIPScore, the elastic model matches the vision-language capabilities of Gemma 3 4B across most subnetworks. The cause for this lies in the response style of the Gemma 3 4B model: it often responds to captioning prompts with question-answer-style continuations rather than direct descriptive sentences, which reduces its CLIP similarity score. CLIPScore is therefore sensitive to response style and does not, by itself, guarantee stable caption quality [21].

Given the strong CLIPScore results of HydraLLM compared to Gemma 3 4B, we evaluate our model with MMMU [42]. Our models consistently achieve random accuracy of around 25%, whereas the Gemma 3 4B model reaches about 39%. We illustrate the qualitative

behavior with the captioning example in Figure 6. We prompt the model to describe an image of a car. Across all head configurations, the captions include hallucinated or incorrect statements about the scene. Subject resolution, spatial relations, and object identity are unreliable: the four-head configuration hallucinates motion and orientation, the three-head and two-head configurations misidentify surrounding objects, and the single-head configuration misidentifies the car’s color and introduces incorrect scene elements. Hallucinations, weak grounding, and unstable object descriptions therefore persist across all head configurations of HydraLLM. The elastic projection establishes feasibility, but the vision-to-language interface remains a suspected bottleneck and does not yet yield robust caption quality.

Figure 6. Qualitative captioning example. The elastic LVLM is prompted to describe an image of a car across the four head configurations (k=1k = 1 to k=4k = 4). Hallucinated or incorrect statements are highlighted in orange, illustrating the qualitative gap that persists alongside the encouraging CLIPScore trends.

Conclusion

We introduce HydraLLM, an MQA-based [36] elastic architecture that employs stochastic head stacking and end-to-end width slicing. A single runtime parameter kk controls the active width of the decoder end-to-end, from token embeddings through attention and FFN projections to the output head, producing nested subnetworks from a single parameter set. We extended the elasticity axis to LVLMs through an elastic projection module whose output width follows the active text width.

On text, magnitude-based reordering of embeddings combined with CETT-based [45] reordering of heads and FFN units yields the strongest subnetworks, reducing perplexity by up to a factor of approximately 4,500 compared to no reordering. However, the effect is reduced to at most 14% after training. Knowledge distillation [22] is crucial during training and improves TriviaQA exact-match accuracy at 75% of active heads by 97.3%. HydraLLM covers a substantially wider parameter range at competitive accuracy compared to MatFormer-style [13] FFN slicing. It achieves larger token-generation speed-ups of up to 75.3%, but only modest prompt-processing improvements.

On the multimodal side, scaled training matches or exceeds the Gemma 3 4B CLIPScore [21, 34] of 65.3 by 2.5%, 5.1% and 6.6% with two, three and four active heads respectively, confirming that the elastic text backbone transfers to the vision-language setting. However, the model remains near random accuracy on MMMU, and qualitative analysis reveals hallucinations and weak object grounding. These limitations point to clear avenues for future research: (i) replacing the constrained small single-layer projector with a larger or a multi-layer FFN projection, which would better align vision features with the elastic text backbone; (ii) longer and more diverse multimodal training schedules to improve grounding beyond captioning tasks; and (iii) combining HydraLLM’s end-to-end slicing with MatFormer-style per-layer granularity for finer accuracy-compute control. We leave these directions for future work.

Acknowledgments

We thank our anonymous reviewers for their insights and suggestions for improvement. This work has received funding from the Federal Ministry of Research, Technology and Space under the REAKT ISA project (grant no. 03DPC1415B) and from the Federal Ministry for Economic Affairs and Energy under the CAPTN X-FERRY project (grant no. 03SX612A).

Reproducibility Details

This section provides implementation and hardware details, runtime configurations, multimodal dataset composition, CLIPScore evaluation details, and training hyperparameters.

Implementation and Hardware

We build our implementation around PyTorch [33] and Hugging Face Transformers [41]. Accelerate [18] and DeepSpeed [35] support distributed training, and LightEval [20] integrates the text benchmarks. CLIPScore follows Hessel et al. [21]. When available, we use optional acceleration libraries such as FlashAttention 2 [11], while ensuring a transparent fallback to the standard PyTorch and Transformers backends. Table 5 lists exact package versions.

PackageVersion
torch2.8.0
torchvision0.23.0
transformers4.56.1
accelerate1.10.1
datasets4.2.0
lighteval0.11.0

Table 5. Key Python package versions used in the implementation.

Text-only experiments run on two NVIDIA RTX PRO 4000 GPUs with at least 24 GB of VRAM each. If we disable the teacher model, text-only training fits on a single GPU. Multimodal training is more resource-demanding: the unified multimodal pipeline requires at least 48 GB of VRAM, and joint training of all parameters requires roughly 60 GB. The multimodal experiments therefore run on two NVIDIA Tesla L40 GPUs or two NVIDIA A100 80 GB GPUs. Throughput measurements use llama.cpp build 4df6e859e (7349) [16].

Runtime Configurations

Table 6 lists the active component dimensions for each head configuration k∈{1,2,3,4}k \in\{1, 2, 3, 4\}.

H1H2H3H4
HkvH_{\mathrm{kv}}1111
query heads kk1234
dE,actived_{E,\mathrm{active}}160320480640
dff,actived_{\mathrm{ff,active}}512102415362048
dQ,actived_{Q,\mathrm{active}}2565127681024

Table 6. HydraLLM’s Gemma 3 270M runtime configurations and active component dimensions. dE,actived_{E,\mathrm{active}} is the active embedding width, dff,actived_{\mathrm{ff,active}} the active FFN neuron count, and dQ,actived_{Q,\mathrm{active}} the total query projection dimension. MatFormer’s W0.25-W1.0 have the same dff,actived_{\mathrm{ff,active}} as H1-H4, but otherwise keep the dimensions of H4.

Multimodal Dataset Composition

Table 7 and Table 8 report the dataset mixtures for the main 2,500-step multimodal runs and the scaled 16,000-step runs, respectively. The scaled setting preserves the same broad mixture of captioning, question answering, code generation, and text-only data, but increases the number of samples.

DatasetNum Samples
Captioning & Pretraining
PixMo-Cap50,000
COCO Recaptioned12,000
LLaVA Pretrain50,000
Question Answering
PixMo Ask Model Anything30,000
PixMo Cap-QA30,000
CoSyn 400K VQA100,000
Code Generation
CoSyn 400K Code3,200
Text Only
Wikipedia EN14,800
FineWeb-Edu 10BT30,000
Total320,000

Table 7. Composition of the multimodal dataset of the main multimodal experiments.

DatasetNum Samples
Captioning & Pretraining
PixMo-Cap500,000
COCO Recaptioned100,000
LLaVA Pretrain500,000
Question Answering
PixMo Ask Model Anything125,000
PixMo Cap-QA125,000
CoSyn 400K VQA1,000,000
Code Generation
CoSyn 400K Code50,000
Text Only
Wikipedia EN200,000
FineWeb-Edu 10BT200,000
Total2,800,000

Table 8. Composition of the multimodal dataset of the scaled 16,000-step multimodal training runs.

The mixtures comprise PixMo-Cap [2], our recaptioned version of COCO [26], LLaVA Pretrain [27], PixMo Ask Model Anything [1], PixMo Cap-QA [3], the VQA and code subsets of CoSyn-400K [4], and text-only samples from Wikipedia [40] and FineWeb-Edu [29].

Projection alignment trains on a separate mixture of 64,000 samples from COCO Recaptioned and 64,000 samples from PixMo-Cap, for a total of 128,000 samples. We generate COCO Recaptioned locally by recaptioning COCO training images with Qwen 3 VL 4B Instruct [5] utilizing the fixed descriptive instruction “Describe the image in 3-5 sentences!”. The resulting captions are denser and more uniform than the original short COCO annotations.

CLIPScore Evaluation

The CLIPScore describes how to measure the caption quality of a model, but it does not define the underlying dataset or the CLIP-based (Radford et al., [34]) model (Hessel et al., [21]). We use the entire COCO validation set to calculate the similarity score with clip-vit-large-patch14 (Radford et al., [34]).

Training Hyperparameters

Table 9 summarizes the hyperparameters for the text-only and default multimodal training runs. The scaled multimodal runs retain the settings of the corresponding 2,500-step runs, but train for 16,000 steps on the larger dataset mixture in Table 8, optionally train the vision encoder at a learning rate of 6×10−66 \times10^{-6} and reduce the text model warmup from 125 to 75 steps, while, if enabled, adding 75 warmup steps for the vision encoder. Compared with text-only training, we employ a lower text-backbone learning rate, a separate projection learning rate, a larger effective batch size, a larger knowledge-distillation weight when distilling from Gemma 3 4B, and uniform head sampling instead of the non-uniform text-only sampling weights for both multimodal runs. The sampling weight array assigns a weight to each head for random sampling.

HyperparameterTextMultimodal
Sequence length30723072
Learning rate (text backbone)3×10−43 \times10^{-4}2×10−52 \times10^{-5}
Learning rate (projection)–2×10−42 \times10^{-4}
LR schedulecosinecosine
Weight decay0.10.1
OptimizerAdamWAdamW
Adam betas[0.9, 0.999][0.9, 0.999]
Adam epsilon1×10−81 \times10^{-8}1×10−81 \times10^{-8}
Warmup ratio (text)0.01–
Warmup steps (text model)–125
Warmup steps (projection)–25
Training steps20,0002,500
Effective batch size16128
Gradient clipping (norm)1.01.0
Head selection[1, 2, 3, 4][1, 2, 3, 4]
Sampling weights[1, 1, 1, 3] (non-uniform)[1,1,1,1] (uniform)
LossCE + z-loss + KDCE + z-loss + KD
zz-loss alpha1×10−51 \times10^{-5}1×10−51 \times10^{-5}
KD weight0.10.2
KD temperature22
Teacher modelGemma 3 270MGemma 3 4B
Precisionbf16bf16
Random seed4242

Table 9. Training hyperparameters for the text-only and default multimodal runs. The full multimodal run where all components are trained differs only in vision encoder learning rate (6×10−66 \times10^{-6}) and warmup steps (text: 75, vision: 75). Where not listed, the multimodal configuration inherits the text default.

Additional Text Experiments

This section presents the text training ablations, followed by the importance metrics and additional reordering results.

Training Recipe Ablation

Table 10 reports the full text-training ablation across ARC-Easy, HellaSwag, and TriviaQA. We evaluate ARC-Easy and HellaSwag with both raw and answer-length-normalized accuracy with zero- and ten-shot prompting respectively, and TriviaQA with five-shot exact-match accuracy. Relative to the default recipe, the ablations disable the teacher, use uniform head sampling, restrict training to head configurations {2,3,4}\{2, 3, 4\}, increase the knowledge-distillation weight to 0.3, use the FineWeb-Edu 10BT subset, apply an EMA decay of 0.999, increase the effective batch size to 128, or reduce the learning rate to 2×10−52 \times10^{-5}. The long-run variant instead extends training to one million steps and uses a larger FineWeb-Edu 350BT subset.

Table 10: Training recipe ablation across text benchmarks after conversion of the static Gemma 3 270M model, with MatFormer as a width-pruning comparison. Values are accuracies in percent, with standard error, for the four runtime head configurations H1–H4 and MatFormer widths W0.25–W1.0. ARC-Easy Norm is answer-length-normalized accuracy. TriviaQA is five-shot exact-match accuracy.

VariantH1/W0.25H2/W0.5H3/W0.75H4/W1.0
ARC-Easy (Raw)
MatFormer53.9 ± 1.0255.6 ± 1.0254.6 ± 1.0256.0 ± 1.02
default39.0 ± 1.0046.3 ± 1.0251.9 ± 1.0356.4 ± 1.02
1M steps43.6 ± 1.0251.1 ± 1.0356.9 ± 1.0260.3 ± 1.00
no teacher37.2 ± 1.0046.2 ± 1.0351.9 ± 1.0256.1 ± 1.02
uniform sampling38.8 ± 0.9948.2 ± 1.0252.2 ± 1.0255.4 ± 1.02
heads 2, 3, 426.8 ± 0.9146.2 ± 1.0252.3 ± 1.0357.8 ± 1.01
KD=0.337.9 ± 1.0046.3 ± 1.0252.9 ± 1.0258.8 ± 1.01
FW-Edu 10BT38.3 ± 1.0045.7 ± 1.0252.6 ± 1.0256.7 ± 1.02
EMA=0.99926.8 ± 0.9128.3 ± 0.9229.3 ± 0.9354.2 ± 1.02
batch size=12831.5 ± 0.9539.2 ± 1.0047.3 ± 1.0257.3 ± 1.02
lower LR=2e-530.4 ± 0.9434.4 ± 0.9742.1 ± 1.0157.4 ± 1.01
ARC-Easy (Norm)
MatFormer46.7 ± 1.0249.8 ± 1.0349.3 ± 1.0350.7 ± 1.03
default35.8 ± 0.9840.0 ± 1.0146.5 ± 1.0250.8 ± 1.03
1M steps38.6 ± 1.0044.5 ± 1.0249.5 ± 1.0352.7 ± 1.02
no teacher34.4 ± 0.9940.5 ± 1.0146.5 ± 1.0249.7 ± 1.03
uniform sampling36.3 ± 0.9741.5 ± 1.0146.8 ± 1.0249.0 ± 1.03
heads 2, 3, 428.5 ± 0.9241.2 ± 1.0147.0 ± 1.0251.0 ± 1.03
KD=0.334.7 ± 0.9840.2 ± 1.0148.4 ± 1.0351.9 ± 1.03
FW-Edu 10BT36.1 ± 0.9939.9 ± 1.0048.2 ± 1.0351.8 ± 1.03
EMA=0.99927.6 ± 0.9227.7 ± 0.9228.8 ± 0.9351.4 ± 1.03
batch size=12830.9 ± 0.9534.8 ± 0.9842.2 ± 1.0150.2 ± 1.03
lower LR=2e-530.8 ± 0.9532.2 ± 0.9637.8 ± 1.0052.8 ± 1.02
HellaSwag (Raw)
MatFormer29.2 ± 0.4530.1 ± 0.4630.7 ± 0.4631.3 ± 0.46
default26.2 ± 0.4426.9 ± 0.4429.0 ± 0.4531.1 ± 0.46
1M steps26.3 ± 0.4428.2 ± 0.4530.7 ± 0.4632.5 ± 0.47
no teacher26.0 ± 0.4427.1 ± 0.4428.9 ± 0.4531.0 ± 0.46
uniform sampling26.2 ± 0.4427.1 ± 0.4428.9 ± 0.4529.9 ± 0.46
heads 2, 3, 425.7 ± 0.4427.1 ± 0.4429.1 ± 0.4531.2 ± 0.46
KD=0.325.7 ± 0.4426.9 ± 0.4428.8 ± 0.4531.1 ± 0.46
FW-Edu 10BT25.9 ± 0.4426.9 ± 0.4428.6 ± 0.4531.2 ± 0.46
EMA=0.99925.7 ± 0.4425.6 ± 0.4426.1 ± 0.4431.6 ± 0.46
batch size=12825.7 ± 0.4426.3 ± 0.4427.6 ± 0.4532.1 ± 0.47
lower LR=2e-525.5 ± 0.4326.3 ± 0.4427.0 ± 0.4433.7 ± 0.47
HellaSwag (Norm)
MatFormer31.6 ± 0.4633.6 ± 0.4734.6 ± 0.4736.1 ± 0.48
default26.4 ± 0.4428.3 ± 0.4531.2 ± 0.4635.6 ± 0.48
1M steps27.4 ± 0.4430.3 ± 0.4634.4 ± 0.4738.4 ± 0.49
no teacher26.2 ± 0.4427.9 ± 0.4530.7 ± 0.4635.2 ± 0.48
uniform sampling26.5 ± 0.4428.2 ± 0.4531.0 ± 0.4633.5 ± 0.47
heads 2, 3, 426.0 ± 0.4428.3 ± 0.4532.2 ± 0.4636.4 ± 0.48
KD=0.326.0 ± 0.4427.8 ± 0.4531.3 ± 0.4635.8 ± 0.48
FW-Edu 10BT26.3 ± 0.4427.7 ± 0.4531.3 ± 0.4635.7 ± 0.48
EMA=0.99926.0 ± 0.4426.1 ± 0.4426.3 ± 0.4437.2 ± 0.48
batch size=12824.8 ± 0.4327.1 ± 0.4429.7 ± 0.4638.0 ± 0.48
lower LR=2e-525.5 ± 0.4326.3 ± 0.4429.1 ± 0.4540.8 ± 0.49
TriviaQA
MatFormer1.47 ± 0.094.70 ± 0.165.18 ± 0.176.19 ± 0.18
default0.15 ± 0.030.23 ± 0.043.69 ± 0.145.51 ± 0.17
1M steps0.13 ± 0.030.66 ± 0.062.35 ± 0.115.35 ± 0.14
no teacher0.01 ± 0.010.02 ± 0.011.87 ± 0.102.16 ± 0.11
uniform sampling0.14 ± 0.030.33 ± 0.043.38 ± 0.135.48 ± 0.17
heads 2, 3, 40.00 ± 0.000.25 ± 0.043.81 ± 0.146.50 ± 0.18
KD=0.30.14 ± 0.030.18 ± 0.033.72 ± 0.146.88 ± 0.19
FW-Edu 10BT0.11 ± 0.020.22 ± 0.042.89 ± 0.126.12 ± 0.18
EMA=0.9990.00 ± 0.000.00 ± 0.000.01 ± 0.013.90 ± 0.14
batch size=1280.00 ± 0.000.17 ± 0.032.67 ± 0.127.17 ± 0.19
lower LR=2e-50.00 ± 0.000.02 ± 0.011.33 ± 0.098.00 ± 0.20

Table 10.

Importance Reordering

This section describes the three importance metrics we implement to reorder embedding channels, attention heads, and FFN neurons before stochastic subnetwork training. We compute importance scores from activation statistics collected during a forward pass over a calibration text corpus (Balbay, [6]) using forward hooks (Taghibakhshi et al., [37]). Each metric ranks units independently for each component. We then apply the resulting orders to the embedding dimension, the query-head axis, and the FFN intermediate axis, as described in § 3.2.1.

Magnitude-Based Importance

The simplest criterion is activation magnitude (Taghibakhshi et al., [37]; Muralidharan et al. [31]). It assumes that units with consistently large activations contribute more strongly than units with small activations.

For FFN neurons, let qb,t,j(ℓ)q_{b,t,j}^{(\ell)} denote the jj-th channel of the gate-projection output in layer ℓ\ell, for batch element bb and sequence position tt. The magnitude score is:

sj,ℓmag=∑b,t∣qb,t,j(ℓ)∣(2)s_{j,\ell}^{\mathrm{mag}} = \sum_{b,t} \left|q_{b,t,j}^{(\ell)}\right| \tag*{(2)}

For attention heads, we score the query-side projection slice. If qb,t,h(ℓ)∈Rdhq_{b,t,h}^{(\ell)} \in\mathbb{R}^{d_h} denotes the query activation of head hh, then:

sh,ℓmag=∑b,t∥qb,t,h(ℓ)∥2(3)s_{h,\ell}^{\mathrm{mag}} = \sum_{b,t} \left\|q_{b,t,h}^{(\ell)}\right\|_2 \tag*{(3)}

For embedding dimensions, we compute magnitude from the normalized hidden states at the output of the pre-layer normalization. Let xb,t,c(ℓ)x_{b,t,c}^{(\ell)} denote embedding channel cc after normalization in layer ℓ\ell. The layer-local score is:

sc,ℓmag=∑b,t∣xb,t,c(ℓ)∣(4)s_{c,\ell}^{\mathrm{mag}} = \sum_{b,t} \left|x_{b,t,c}^{(\ell)}\right| \tag*{(4)}

Because we consistently slice embedding dimensions across the entire decoder, we sum layer-local scores into a single global ranking:

scmag=∑ℓsc,ℓmag(5)s_c^{\mathrm{mag}} = \sum_{\ell} s_{c,\ell}^{\mathrm{mag}} \tag*{(5)}

Magnitude scoring is simple and inexpensive, but it does not account for how strongly downstream weights consume a unit’s output.

Variance × Consumer Norms

The second metric, VxC, combines activation variance [39] with downstream consumer strength. The intuition is that a unit should rank highly if it varies across inputs and if strong outgoing weights amplify its output. Given activations ziz_i, the empirical variance is Var⁡(zi)=E[zi2]−E[zi]2\operatorname{Var}(z_i) = \mathbb{E}[z_i^2] - \mathbb{E}[z_i]^2, where the expectation is taken over the calibration data.

For FFN neurons, VxC multiplies the gate-projection activation variance by the squared norm of the corresponding down-projection column:

sj,ℓvxc=Var⁡b,t(gb,t,j(ℓ))⋅∥Wdown,ℓ[:,j]∥22(6)s_{j,\ell}^{\mathrm{vxc}} = \operatorname{Var}_{b,t}\left(g_{b,t,j}^{(\ell)}\right) \cdot\left\|W_{\mathrm{down},\ell}[:,j]\right\|_2^2 \tag*{(6)}

For attention heads, we first sum the channel-wise variance within each query head:

vh,ℓ=∑d=1dhVar⁡b,t(qb,t,h,d(ℓ))(7)v_{h,\ell} = \sum_{d=1}^{d_h} \operatorname{Var}_{b,t}\left(q_{b,t,h,d}^{(\ell)}\right) \tag*{(7)}

Then we weight this term by the squared Frobenius norm of the corresponding output-projection slice Wℓ,hOW_{\ell,h}^{O}:

sh,ℓvxc=vh,ℓ⋅∥Wℓ,hO∥F2(8)s_{h,\ell}^{\mathrm{vxc}} = v_{h,\ell} \cdot\left\|W_{\ell,h}^{O}\right\|_F^2 \tag*{(8)}

Several projections inside a transformer block consume the embedding channels. Let Cℓ={WℓQ,WℓK,WℓV,Wℓgate,Wℓup}C_\ell= \{W_\ell^Q, W_\ell^K, W_\ell^V, W_\ell^{\mathrm{gate}}, W_\ell^{\mathrm{up}}\} denote these consumers. The per-layer embedding score is:

sc,ℓvxc=Var⁡b,t(xb,t,c(ℓ))⋅∑W∈Cℓ∥W[:,c]∥22(9)s_{c,\ell}^{\mathrm{vxc}} = \operatorname{Var}_{b,t}\left(x_{b,t,c}^{(\ell)}\right) \cdot\sum_{W \in C_\ell} \left\|W[:,c]\right\|_2^2 \tag*{(9)}

To obtain one global ranking, we sum the embedding scores across layers, just like we do with magnitude scoring. In comparison to magnitude, VxC directly captures a unit’s role in the computation graph. Units that fluctuate significantly but have weak consumption receive lower rankings than those whose variation is amplified by large outgoing weights.

CETT-Based Importance

Finally, the metric based on CETT (Zhang et al. [45]) measures the extent to which tail truncation affects the component output. We adapt this idea into per-unit scores based on the norm of each unit’s direct contribution to the output.

For FFN neurons, let hb,t,j(ℓ)h_{b,t,j}^{(\ell)} denote the jj-th component of the input to the down projection in layer ℓ\ell, and let Wdown,ℓ[:,j]W_{\mathrm{down},\ell}[:,j] denote the corresponding output weight vector. The contribution norm of neuron jj is:

cb,t,j(ℓ)=∥hb,t,j(ℓ)Wdown,ℓ[:,j]∥2=∣hb,t,j(ℓ)∣⋅∥Wdown,ℓ[:,j]∥2(10)c_{b,t,j}^{(\ell)} = \lVert h_{b,t,j}^{(\ell)} W_{\mathrm{down},\ell}[:,j] \rVert_2 = \lvert h_{b,t,j}^{(\ell)} \rvert\cdot\lVert W_{\mathrm{down},\ell}[:,j] \rVert_2 \tag*{(10)}

For attention heads, let ub,t,h(ℓ)∈Rdhu^{(\ell)}_{b,t,h} \in\mathbb{R}^{d_h} denote the output of head hh before the output projection, and let Wℓ,hO∈Rdmodel×dhW^{O}_{\ell,h} \in\mathbb{R}^{d_{\mathrm{model}} \times d_h} be the corresponding output-projection slice. The head contribution norm is:

cb,t,h(ℓ)=∥Wℓ,hOub,t,h(ℓ)∥2(11)c^{(\ell)}_{b,t,h} = \left\|W^{O}_{\ell,h}u^{(\ell)}_{b,t,h}\right\|_2 \tag*{(11)}

The unnormalized CETT score is the empirical mean contribution norm. The normalized variant divides the contribution norm by the norm of the full component output at the same token position. For FFN neurons, this gives:

sj,ℓcett=Eb,t[cb,t,j(ℓ)∥yb,tFFN,(ℓ)∥2](12)s^{\mathrm{cett}}_{j,\ell} = \mathbb{E}_{b,t}\left[\frac{c^{(\ell)}_{b,t,j}}{\left\|y^{\mathrm{FFN},(\ell)}_{b,t}\right\|_2}\right] \tag*{(12)}

For attention heads, this gives:

sh,ℓcett=Eb,t[cb,t,h(ℓ)∥yb,tAttn,(ℓ)∥2](13)s^{\mathrm{cett}}_{h,\ell} = \mathbb{E}_{b,t}\left[\frac{c^{(\ell)}_{b,t,h}}{\left\|y^{\mathrm{Attn},(\ell)}_{b,t}\right\|_2}\right] \tag*{(13)}

The normalized form is the most relevant variant for prefix slicing because it estimates a unit’s contribution relative to the total component output. Since adaptive slicing removes the tail of a dimension, this ranking directly targets the expected relative output error incurred when we discard low-ranked units.

Effect of Importance Reordering

Table 11 complements the perplexity comparison in § 5.1 with downstream accuracy after 20,000 training steps. Each row corresponds to a different combination of importance metrics for embeddings, attention heads, and FFN units. Mag-CETT-CETT improves TriviaQA exact-match accuracy across all head configurations, while ARC-Easy and HellaSwag remain close to the no-reordering baseline.

VariantH1H2H3H4
ARC-Easy (Raw)
no reordering39.0±1.0039.0 \pm1.0046.3±1.0246.3 \pm1.0251.9±1.0351.9 \pm1.0356.4±1.0256.4 \pm1.02
Mag-CETT-CETT38.4±1.0038.4 \pm1.0046.3±1.0246.3 \pm1.0252.2±1.0252.2 \pm1.0256.6±1.0256.6 \pm1.02
Mag-Mag-Mag37.6±0.9937.6 \pm0.9946.8±1.0246.8 \pm1.0252.9±1.0252.9 \pm1.0257.1±1.0257.1 \pm1.02
VxC-VxC-VxC36.9±0.9936.9 \pm0.9946.8±1.0246.8 \pm1.0251.9±1.0351.9 \pm1.0356.7±1.0256.7 \pm1.02
Mag-no-Mag38.3±1.0038.3 \pm1.0046.3±1.0246.3 \pm1.0253.2±1.0253.2 \pm1.0256.5±1.0256.5 \pm1.02
Mag-VxC-Mag37.7±0.9937.7 \pm0.9945.4±1.0245.4 \pm1.0252.1±1.0252.1 \pm1.0256.0±1.0256.0 \pm1.02
ARC-Easy (Norm)
no reordering35.8±0.9835.8 \pm0.9840.0±1.0140.0 \pm1.0146.5±1.0246.5 \pm1.0250.8±1.0350.8 \pm1.03
Mag-CETT-CETT34.7±0.9834.7 \pm0.9841.6±1.0141.6 \pm1.0146.9±1.0246.9 \pm1.0250.5±1.0350.5 \pm1.03
Mag-Mag-Mag33.9±0.9733.9 \pm0.9742.0±1.0142.0 \pm1.0146.9±1.0246.9 \pm1.0251.1±1.0351.1 \pm1.03
VxC-VxC-VxC34.2±0.9734.2 \pm0.9741.4±1.0141.4 \pm1.0147.3±1.0247.3 \pm1.0251.1±1.0351.1 \pm1.03
Mag-no-Mag36.2±0.9936.2 \pm0.9941.1±1.0141.1 \pm1.0147.3±1.0247.3 \pm1.0249.4±1.0349.4 \pm1.03
Mag-VxC-Mag34.9±0.9834.9 \pm0.9840.7±1.0140.7 \pm1.0146.3±1.0246.3 \pm1.0251.3±1.0351.3 \pm1.03
HellaSwag (Raw)
no reordering26.2±0.4426.2 \pm0.4426.9±0.4426.9 \pm0.4429.0±0.4529.0 \pm0.4531.1±0.4631.1 \pm0.46
Mag-CETT-CETT26.2±0.4426.2 \pm0.4426.8±0.4426.8 \pm0.4428.9±0.4528.9 \pm0.4530.9±0.4630.9 \pm0.46
Mag-Mag-Mag26.1±0.4426.1 \pm0.4427.0±0.4427.0 \pm0.4428.5±0.4528.5 \pm0.4531.2±0.4631.2 \pm0.46
VxC-VxC-VxC26.1±0.4426.1 \pm0.4426.9±0.4426.9 \pm0.4428.9±0.4528.9 \pm0.4531.1±0.4631.1 \pm0.46
Mag-no-Mag26.0±0.4426.0 \pm0.4426.8±0.4426.8 \pm0.4428.7±0.4528.7 \pm0.4531.2±0.4631.2 \pm0.46
Mag-VxC-Mag26.1±0.4426.1 \pm0.4426.8±0.4426.8 \pm0.4428.8±0.4528.8 \pm0.4531.1±0.4631.1 \pm0.46
HellaSwag (Norm)
no reordering26.4±0.4426.4 \pm0.4428.3±0.4528.3 \pm0.4531.2±0.4631.2 \pm0.4635.6±0.4835.6 \pm0.48
Mag-CETT-CETT26.3±0.4426.3 \pm0.4428.0±0.4528.0 \pm0.4531.2±0.4631.2 \pm0.4636.0±0.4836.0 \pm0.48
Mag-Mag-Mag26.3±0.4426.3 \pm0.4428.7±0.4528.7 \pm0.4531.4±0.4631.4 \pm0.4635.9±0.4835.9 \pm0.48
VxC-VxC-VxC25.6±0.4425.6 \pm0.4428.1±0.4528.1 \pm0.4531.3±0.4631.3 \pm0.4635.7±0.4835.7 \pm0.48
Mag-no-Mag26.2±0.4426.2 \pm0.4428.0±0.4528.0 \pm0.4531.8±0.4631.8 \pm0.4635.9±0.4835.9 \pm0.48
Mag-VxC-Mag25.8±0.4425.8 \pm0.4428.4±0.4528.4 \pm0.4531.8±0.4631.8 \pm0.4635.6±0.4835.6 \pm0.48
TriviaQA
no reordering0.15±0.030.15 \pm0.030.23±0.040.23 \pm0.043.69±0.143.69 \pm0.145.51±0.175.51 \pm0.17
Mag-CETT-CETT0.21±0.030.21 \pm0.030.82±0.070.82 \pm0.074.84±0.164.84 \pm0.166.63±0.196.63 \pm0.19
Mag-Mag-Mag0.14±0.030.14 \pm0.030.08±0.020.08 \pm0.023.08±0.133.08 \pm0.135.59±0.175.59 \pm0.17
VxC-VxC-VxC0.00±0.000.00 \pm0.000.77±0.070.77 \pm0.073.73±0.143.73 \pm0.145.43±0.175.43 \pm0.17
Mag-no-Mag0.11±0.020.11 \pm0.020.45±0.050.45 \pm0.054.26±0.154.26 \pm0.155.57±0.175.57 \pm0.17
Mag-VxC-Mag0.03±0.010.03 \pm0.010.79±0.070.79 \pm0.072.99±0.132.99 \pm0.134.60±0.164.60 \pm0.16

Table 11. Effect of importance reordering on the four nested runtime head configurations after 20,000 training steps. Values are accuracies in percent with standard error. ARC-Easy Norm is answer-length-normalized accuracy. TriviaQA is five-shot exact-match accuracy. The no-reordering variant is the default baseline from Table 10.

Additional Multimodal Results

This section provides additional results on text recovery after multimodal training and the optimization behavior of the elastic projection during alignment.

Text Recovery after Multimodal Training

Table 12 lists TriviaQA accuracies after text-only continued pre-training and after subsequent multimodal or text-only fine-tuning stages. The comparison illustrates how teacher choice and continued fine-tuning affect recovery of factual recall after the multimodal stage. The text-only baseline is trained for one million steps. We compare it with the default multimodal run and with two text fine-tuning variants that use either Gemma 3 270M or Gemma 3 4B as the teacher.

VariantH1H2H3H4
text, 1M steps0.130.662.353.55
multimodal, default0.802.164.086.48
text fine-tune, 27M teacher0.121.103.645.07
text fine-tune, 4B teacher0.082.134.567.32

Table 12. TriviaQA accuracy (exact match, %) across head configurations for text-only continued pre-training and subsequent multimodal or text-only fine-tuning variants.

The Gemma 3 4B teacher consistently outperforms the 270M teacher at the four-head configuration, suggesting that the larger teacher transfers more factual knowledge to the student. The improved accuracy relative to the long-run text baseline is likely due to the shift from packed context-window training to training on individual padded samples, together with the lower learning rate and larger batch size used in this stage.

Projection Alignment Loss Curves

Figure 7 shows the training loss for each head configuration during projection alignment. The one-head configuration converges to a substantially higher loss than the wider configurations, indicating that the single-head subnetwork reaches its capacity limit earlier. Configurations with two or more heads quickly reduce their loss and stabilize around steps 50–100, consistent with the rapid plateau observed in the main text.

Figure 7. Training loss per-head configuration during projection alignment. The one-head model converges to a higher loss, while configurations with two or more heads reduce their loss rapidly and stabilize within the first 100 steps.

References

  1. [1]Allen Institute for AI. 2024a. PixMo-AskModelAnything. https://huggingface.co/datasets/allenai/pixmo-ask-model-anything. Hugging Face dataset, 161,737 rows, ODC-BY-1.0. Accessed: 2026-03-11.
  2. [2]Allen Institute for AI. 2024b. PixMo-Cap. https://huggingface.co/datasets/allenai/pixmo-cap. Hugging Face dataset. Image-to-text dataset, 717,042 rows. Accessed: 2026-03-11.
  3. [3]Allen Institute for AI. 2024c. PixMo-CapQA. https://huggingface.co/datasets/allenai/pixmo-cap-qa. Hugging Face dataset. Visual Question Answering dataset, 271,714 rows. Accessed: 2026-03-11.
  4. [4]Allen Institute for AI. 2025. CoSyn-400K. https://huggingface.co/datasets/allenai/CoSyn-400K. Hugging Face dataset. Visual Question Answering dataset, 408,227 rows. Accessed: 2026-03-11.
  5. [5]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631.arxiv.org/abs/2511.21631
  6. [6]Alper Balbay. 2024. Calibration split of the tiny shakespare dataset. GitHub repository file. Commit 5a58873, accessed 21 July 2026.
  7. [7]Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2020. Once-for-all: Train one network and specialize it for efficient deployment. Preprint, arXiv:1908.09791.
  8. [8]Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. 2024. Flextron: Many-in-one flexible large language model. Preprint, arXiv:2406.10260.arxiv.org/abs/2406.10260
  9. [9]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyvashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, and 48 others. 2022. Palm: Scaling language modeling with pathways. Preprint, arXiv:2204.02311.arxiv.org/abs/2204.02311
  10. [10]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457.
  11. [11]Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. Preprint, arXiv:2307.08691.
  12. [12]Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Branson, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 31 others. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. Preprint, arXiv:2409.17146.
  13. [13]Devrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannan Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. 2024. Matformer: Nested transformer for elastic inference. Preprint, arXiv:2310.07707.
  14. [14]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint, arXiv:2010.11929.
  15. [15]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
  16. [16]Georgi Gerganov and ggml-org contributors. 2025. llama.cpp: LLM inference in C/C++. Build: 4df6e859e (7349).
  17. [17]Google. 2024. Gemini: Your personal ai assistant. https://gemini.google/assistant/. Accessed: 2026-02-03.
  18. [18]Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate.
  19. [19]Janek Haberer, Ali Hojjat, and Olaf Landsiedel. 2024. Hydravit: Stacking heads for a scalable vit. Preprint, arXiv:2409.17978.
  20. [20]Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. LightEval: A lightweight framework for llm evaluation.
  21. [21]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. ClipScore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718.arxiv.org/abs/2104.08718
  22. [22]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.arxiv.org/abs/1503.02531
  23. [23]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. Preprint, arXiv:1705.03551.arxiv.org/abs/1705.03551
  24. [24]Daniel Jurafsky and James H. Martin. 2026. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models, 3rd edition. Online manuscript released January 6, 2026.
  25. [25]Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2024. Matryoshka representation learning. Preprint, arXiv:2205.13147.arxiv.org/abs/2205.13147
  26. [26]Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft coco: Common objects in context. Preprint, arXiv:1405.0312.arxiv.org/abs/1405.0312
  27. [27]Haotian Liu. 2023. LLaVA-Pretrain. https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain. Hugging Face dataset; LLaVA Visual Instruction Pretrain dataset created in May 2023. Accessed: 2026-03-11.
  28. [28]Haotian Liu, Chun-Yuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.arxiv.org/abs/2304.08485
  29. [29]Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and 1 others. 2024. FineWeb-Edu: The Finest Collection of Educational Web Data on Hugging Face.DOI
  30. [30]Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2025. Smolvlm: Redefining small and efficient multimodal models. Preprint, arXiv:2504.05299.arxiv.org/abs/2504.05299
  31. [31]Saurav Muralidharan, Sharath Turuv​​ekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Preprint, arXiv:2407.14679.arxiv.org/abs/2407.14679
  32. [32]OpenAI. 2024. Chatgpt overview. https://chatgpt.com/overview?openai.com_referred=true. Accessed: 2026-02-28.
  33. [33]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Preprint, arXiv:1912.01703.arxiv.org/abs/1912.01703
  34. [34]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.arxiv.org/abs/2103.00020
  35. [35]Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery.
  36. [36]Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. Preprint, arXiv:1911.02150.arxiv.org/abs/1911.02150
  37. [37]Ali Taghibakhshi, Sharath Turuv​​ekere Sreenivas, Saurav Muralidharan, Ruisi Cai, Yixin Marc​​in Ch​​ow​​ch​​owski, Ameya Sunil Mah​​abaleshwar​​kar, Yoshi Suhara, Oluwatobi Olabiyi, Daniel Korzekwa, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, Bryan Catanzaro, Ashwath Aithal, Nima Tajbakhsh, and Pavlo Molchanov. 2025. Nemotron elastic: Towards efficient many-in-one reasoning llms. Preprint, arXiv:2511.16664.arxiv.org/abs/2511.16664
  38. [38]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.arxiv.org/abs/1706.03762
  39. [39]Berry Weinstein and Yonatan Belinkov. 2022. Variance pruning: Pruning language models via temporal neuron variance.
  40. [40]Wikimedia Foundation. 2023. Wikimedia Downloads. Referenced via the Hugging Face wikimedia/wikipedia dataset card. Accessed: 2026-03-11.
  41. [41]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Hugging-face’s transformers: State-of-the-art natural language processing. Preprint, arXiv:1910.03771.arxiv.org/abs/1910.03771
  42. [42]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. Preprint, arXiv:2311.16502.arxiv.org/abs/2311.16502
  43. [43]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830.arxiv.org/abs/1905.07830
  44. [44]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. Preprint, arXiv:2303.15343.arxiv.org/abs/2303.15343
  45. [45]Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. 2024. Relu² wins: Discovering efficient activation functions for sparse llms. Preprint, arXiv:2402.03804.arxiv.org/abs/2402.03804

Paper details

vlnc:W_62m8qecq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents