betaFeedback

WorksComputer Science

LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable multimodal capabilities but suffer from prohibitive computational overhead due to the massive number of visual tokens. While token pruning is a promising solution, existing methods still suffer from two major drawbacks: 1) unreliable pruning criteria that fail to accurately localize task-specific regions due to inherent attention biases or query-agnostic designs, and 2) suboptimal pruning strategies that employ rigid, non-adaptive schedules, overlooking the dynamic evolution of token redundancy across different model depths. To address these challenges, we propose LearnPrunerV2, which extends the previous two-stage heuristic framework (LearnPruner) into a fully learnable architecture. Specifically, we integrate lightweight learnable modules into every layer of the VLM backbone, which estimate token importance by fusing intrinsic visual saliency with extrinsic query relevance captured via a shadow token. By uncovering a consistent staircase redundancy pattern across VLM layers, we further introduce Budget-Deterministic Pruning (BDP) to establish a hardware-friendly schedule with fixed per-layer budgets. Extensive experimental results across various benchmarks and VLMs demonstrate that LearnPrunerV2 consistently outperform previous state-of-the-art methods. Notably, LearnPrunerV2 preserves 99.8% of the original performance when pruning 77.8% of visual tokens on LLaVA-v1.5, achieving a superior accuracy-efficiency trade-off.

CCS Concepts

• Computing methodologies → Artificial intelligence.

This work is licensed under a Creative Commons Attribution 4.0 International License.

ACM Reference Format: Rinyoichi Takezoe, Yaqian Li, Weiren Wang, Zihao Bo, Huihong Shi, Anzhou Hou, Mo Guang, Jiale Yan, and Kaiwen Long. 2026. LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/3767308.3836571

Introduction

Following the rapid evolution of Large Language Models (LLMs) [1, 24, 42], Vision-Language Models (VLMs) have extended these sophisticated reasoning capabilities to the multimodal domain, leading to significant advancements in visual understanding and grounding. Generally, VLMs employ a pretrained visual encoder to transform visual inputs into a sequence of tokens, which are subsequently concatenated with textual instruction tokens as input for the LLM. Visual tokens often dominate the input sequence, typically numbering in the hundreds and potentially expanding to thousands for high-resolution images or long videos. Since the computational complexity of self-attention scales quadratically with sequence length, such a massive influx of visual tokens drastically inflates inference latency, rendering VLMs impractical for real-time or resource-constrained deployment.

To alleviate this computational bottleneck, numerous studies have explored token pruning as a viable strategy for reducing the visual sequence length. Given the inherent redundancy in visual modality, prior research has demonstrated that a substantial proportion of visual tokens can be discarded with minimal impact on model performance. Fundamentally, the effectiveness of token pruning hinges on two core dimensions: how to prune (the selection criterion) and where to prune (the pruning schedule). In this paper, we argue that existing approaches remain suboptimal in both aspects.

Unreliability of Pruning Criteria. Existing pruning criteria primarily fall into two categories: importance-based and diversity-based. The former utilizes attention scores from either the visual encoder or the LLM to identify significant tokens. However, as noted in LearnPruner [33], visual encoders often suffer from attention sink that fail to localize foreground objects, while LLM attention exhibits a strong positional bias, with reliable query-aware signals only beginning to emerge from the intermediate layers onward. This creates a dilemma: early pruning is computationally efficient but "blind", while late-stage pruning is semantically guided but yields marginal speedup. The latter category utilizes token-wise feature similarity to select the maximum coverage subset, aiming to preserve global visual diversity. These methods are inherently query-agnostic, making it challenging to adapt pruning results to different user instructions. While recent efforts [5, 48] attempt to incorporate auxiliary text encoders to facilitate cross-modal guidance, as shown in Figure 1, these encoders are limited to word-level granularity, failing to resolve the global semantic context of the image. This suggests that the internal attention of LLMs remains essential for capturing task-specific visual information.

Visualization of cross-modal relevance scores in LLaVA-v1.5

Figure 1. Visualization of cross-modal relevance scores in LLaVA-v1.5. ’Similarity’ denotes the similarity scores between CLIP multimodal features, while ’Layer 8/16 Attn’ represents the attention distribution of the last token within the LLM. Notably, CLIP features primarily capture isolated word-level semantics (e.g., the ’girl’); in contrast, intermediate LLM layers resolve global contextual semantics to accurately localize the intended target based on spatial relations.

Suboptimality of Pruning Strategies. Beyond selection criteria, existing methods typically employ static, one-pass pruning schedules or layer specific heuristics, overlooking the intrinsic coupling between token redundancy and VLM layer depth. Indeed, recent studies have revealed the layer-wise dynamics: AutoPrune [36] demonstrates that VLM attention exhibits significant fluctuations across layers, particularly in reasoning-intensive scenarios. Furthermore, TwigVLM [31] reveals that in the deep layers of VLMs, even the complete removal of visual tokens does not necessarily degrade performance. Collectively, these findings suggest that as visual information is incrementally integrated into linguistic representations, the significance of visual tokens is not uniform across layers but evolves throughout the hierarchy, motivating a progressive, layer-adaptive pruning strategy.

To address these limitations, LearnPruner [33] takes an initial step by adopting a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only query-relevant tokens in the LLM’s middle layers based on text-to-vision attention scores. While effective, its singular input signals and hand-crafted schedule cannot fully capture the continuous evolution of token redundancy.

Building upon this, we propose LearnPrunerV2, a purely learnable pruning framework for VLMs. Unlike the hybrid design of its predecessor, our framework adopts a unified architecture by integrating a lightweight pruning module into every layer of the VLM backbone. Moreover, a learnable "shadow token" is appended to the input sequence during the prefill stage to aggregate the preceding context and capture cross-modal correlations. Each pruning module subsequently combines intrinsic visual saliency from local features with extrinsic query relevance derived from the shadow token to predict importance scores for individual visual token. Through end-to-end joint optimization, LearnPrunerV2 aligns pruning decisions with the layer-specific evolution of visual significance, which in turn allows pruning modules to determine the optimal exit depth for each token. This dynamic mechanism enables the model to flexibly adjust its pruning trajectory based on the complexity of both images and queries. Furthermore, to address practical budget-constrained scenarios, we introduce Budget-Deterministic-Pruning (BDP) strategy. BDP utilizes offline-calibrated statistics to maintain a strictly fixed token count across every layer and adopts diversity-aware selection to mitigate the impact of a non-adaptive schedule. This ensures deterministic inference latency and maximized batching throughput, providing a robust solution for high-concurrency real-world deployment.

Benefiting from its model-agnostic design, LearnPrunerV2 can be seamlessly integrated into diverse VLM architectures, such as LLaVA [23, 25, 26] and Qwen-VL [3, 37]. Extensive experiments on multiple VQA benchmarks shows that our method outperforms prior state-of-the-art methods under different retention ratios. For example, when pruning 94.4% visual tokens in LLaVA-NeXT-7B, LearnPrunerV2 preserve 96.2% of the original model performance, while achieving a 9.7× reduction in TFLOPs. In summary, the contributions of our work are threefold:

  • We present LearnPrunerV2, a purely learnable token pruning framework that adopts a progressive pruning strategy. Compared to LearnPruner, it offers a more unified and flexible solution.

  • We introduce the Budget-Deterministic-Pruning schedule to assign fixed per-layer token budgets, enabling hardware-friendly inference.

  • We conduct extensive experiments on various VLMs and vision-language benchmarks, demonstrating that LearnPrunerV2 consistently achieves state-of-the-art performance across different retention ratios.

Related Work

Vision Language Models

Building upon the foundational reasoning power of Large Language Models (LLMs) [1, 6, 18, 24, 34, 42], Vision-Language Models (VLMs) [3, 20, 26, 50] demonstrate remarkable capabilities showing strong performance on diverse multimodal tasks such as image captioning, visual question answering, and visual grounding. A standard VLM typically comprises three components: a visual encoder to transform visual inputs into tokens, a vision-language connector to align the modality, and a LLM to serve as the cognitive engine for multimodal integration and complex reasoning. However, this paradigm incurs significant computational costs as the massive number of visual tokens. For instance, LLaVA-v1.5 [26] converts a 336x336 image into 576 tokens, advanced VLMs like LLaVA-NeXT [25] (tiling-based) and Qwen2.5-VL [3] (native dynamic resolution) support high-resolution inputs that further expand visual token sequences. Furthermore, in video understanding scenarios, the visual token count scales linearly with the number of sampled frames [9, 23]. Such extensive sequences pose a severe bottleneck for real-time applications, driving the urgent need for efficient inference of VLMs.

Vision Token Reduction for VLMs

Given the inherent redundancy in visual signals, token reduction has emerged as a promising direction to enhance VLM efficiency. Various works [7, 10, 14, 21, 39] attempt to compress vision token into more compact representations. While effective in reducing computation, these methods typically change the model architecture and require retraining from scratch, resulting in high training costs and potential degradation of the performance upper bound. In this work, we mainly focus on token pruning techniques that can be directly applied to existing VLMs.

From the perspective of pruning criteria, existing methods can be broadly categorized into two types: importance-based and diversity-based. Importance-based methods [8, 31, 43, 46, 47, 49] utilize attention scores to select top-k important tokens and discard the rest. To remove query-irrelevant visual tokens, FastV [8] leverages the attention of the last text token at the second LLM layer as the principled signal, while SparseVLM [49] first identifies text tokens strongly correlated with visual input, and then employs their attention scores to achieve more precise guidance for pruning. Alternatively, to eliminate inherent visual redundancy, VisionZip [43] and VisPruner [47] leverage [CLS] attention from the visual encoder to identify key salient tokens and perform pruning before the LLM. One limitation of importance-based methods is their reliance on attention maps, which makes them incompatible with Flash Attention [11]. Diversity-based methods aims to select a diverse set based on similarity between tokens [2, 17, 38, 48]. DivPrune [2] formulates token pruning as a Max-Min Diversity Problem, whereas DART [38] explicitly minimizes duplication by comparing candidate tokens against a set of pivots. Furthermore, CDPruner [48] calculates inter-token similarity conditioned on their relevance to the current instruction and use the determinant point process to maximize the conditional diversity.

From the perspective of pruning schedules, most studies conduct static pruning at a specific layer, overlooking the layer-wise evolution of token redundancy. Alternatively, some works adopt multi-stage pruning schedules [19, 33, 36, 40, 44]. For instance, PDrop [40] applies progressive pruning with uniform ratios and intervals; AutoPrune [36] derives sample-specific policies by quantifying cross-modal mutual information; LearnPruner [33] employs a two-stage scheme, applying stage-specific pruning criteria to different phases. Despite these advancements, such heuristic designs remain misaligned with the dynamic nature of visual information processing, leading to suboptimal pruning schedules.

LearnPrunerV2

Preliminaries

Existing VLMs generally consist of three primary components: a vision encoder, projector and LLM. Since our methods operate within the LLM, we first provide a formal definition of its architecture. The LLM is composed of a stack of Transformer-based [35] decoder layers, each containing a causal self-attention to capture contextual dependencies and a feed-forward network for feature refinement.

Given image and user instruction, inputs are discretized via vision encoder and tokenizer into XvisX_{\mathrm{vis}} and XtextX_{\mathrm{text}} respectively. These are then concatenated with system prompts to form the LLM input sequence X=[Xsys,Xvis,Xtext]∈RN×dX = [X_{\mathrm{sys}}, X_{\mathrm{vis}}, X_{\mathrm{text}}] \in\mathbb{R}^{N \times d}, where NN is the sequence length and dd is the hidden dimension. In each decoder layer, the input XX is first projected into query QQ, key KK, and value VV:

Q=XWQ,K=XWK,V=XWV(1)Q = XW_Q,\quad K = XW_K,\quad V = XW_V \tag*{(1)}

where WQW_Q, WKW_K, WV∈Rd×dW_V \in\mathbb{R}^{d \times d} are projection matrices. The causal self-attention is then computed as:

A=softmax⁡(QKT+Md),O=AV(2)A = \operatorname{softmax}\left(\frac{QK^{T} + M}{\sqrt{d}}\right),\quad O = AV \tag*{(2)}

where M∈RN×NM \in\mathbb{R}^{N \times N} is an upper triangular matrix to ensure each token can only attend to itself and previous tokens. Subsequently, the attention output is fed into the feed-forward network for non-linear transformation. Residual connections are applied to both the attention and ffn blocks to produce the output of current layer. After passing through all decoder layers, the final hidden states are utilized for next-token prediction.

Learnable Pruning Criterion

Unlike prior works that rely on pre-defined heuristic rules, our approach employs learnable modules to evaluate token importance in a data-driven manner. The core challenge lies in constructing effective inputs for the module. A straightforward baseline is to use raw token features, similar to the Mixture-of-Depth (MoD) approach [30]. However, while these features contain rich semantic information, they are inherently query-agnostic, capturing only intrinsic visual saliency rather than task-specific importance. Although text-to-vision attention provides such relevance, it faces two critical limitations. First, the requirement for explicit attention scores is incompatible with efficient attention implementations such as FlashAttention. Second, as observed in FastV [8], attention scores in VLMs are heavily dominated by the textual modality, visual tokens often receive scores orders of magnitude lower (e.g., around 10−410^{-4} in LLaVA-1.5). Such faint signals lack sufficient numerical contrast, making it difficult for learnable modules to capture fine-grained importance differences.

Overview of LearnPrunerV2

Figure 2. Overview of LearnPrunerV2. (a) Framework illustration, a learnable pruning module (LPM) is inserted into each decoder layer to evaluate token importance, and a learnable shadow token is appended after the user input to aggregate semantic context and capture cross-modal relevance. (b) Internal structure of the LPM. The visual hidden state is concatenated with the query-aware feature derived from the shadow token to predict the final importance score.

To address these issues, we introduce a learnable shadow token, appended after the user instruction to aggregate semantic context and locate query-relevant visual regions. Specifically, the shadow token is initialized with the newline character, exploiting its natural capacity for summarizing preceding information in LLMs. During training, the input sequence is expanded to [Xsys,Xvis,Xins,Xshd,Xans][X_{\mathrm{sys}}, X_{\mathrm{vis}}, X_{\mathrm{ins}}, X_{\mathrm{shd}}, X_{\mathrm{ans}}]. Crucially, the shadow token can attend to all preceding tokens, but remains invisible to all other tokens via a customized causal mask, avoiding any perturbation to the model’s original predictions. Within the decoder layer, we construct a relevance vector ai∈RH\mathbf{a}_i \in\mathbb{R}^{H} for ii-th visual token by computing the multi-head interaction between the shadow query qshd\mathbf{q}_{\mathrm{shd}} and corresponding visual keys ki∈Kvis\mathbf{k}_i \in K_{\mathrm{vis}}:

ai=[qshd⊤(kih)⊤d]h=1H,(3)\mathbf{a}_i = \left[\frac{\mathbf{q}_{\mathrm{shd}}^{\top}(\mathbf{k}_i^h)^{\top}}{\sqrt{d}}\right]_{h=1}^{H}, \tag*{(3)}

where HH is the number of attention heads. Notably, this standalone query-key interaction avoids extracting from the internal attention matrix, ensuring compatibility with FlashAttention, and the use of pre-softmax logits preserves sufficient numerical contrast for optimization. The vector ai\mathbf{a}_i is then mapped through a layer-specific MLP, denoted as MLPϕ\mathrm{MLP}_{\phi}, to yield an enriched query-relevant feature ri=MLPϕ(ai)∈Rdr\mathbf{r}_i = \mathrm{MLP}_{\phi}(\mathbf{a}_i) \in\mathbb{R}^{d_r}.

While this feature captures the cross-modal correlation at a single layer, prior work [36] has shown that text-to-vision attention often exhibits pronounced inter-layer saliency fluctuations, particularly in complex scenarios. Directly relying on such unstable signals risks producing inconsistent pruning decisions. To stabilize importance estimation, we introduce a gating mechanism that incorporates historical information from the preceding layer. Concretely, given the current relevance feature ri(l)\mathbf{r}_i^{(l)} and its smoothed counterpart from the preceding layer r~i(l−1)\tilde{\mathbf{r}}_i^{(l-1)}, the gating weight is computed as:

wi(l)=σ(Linear⁡([ri(l),r~i(l−1)])),(4)w_i^{(l)} = \sigma\left(\operatorname{Linear}\left([\mathbf{r}_i^{(l)}, \tilde{\mathbf{r}}_i^{(l-1)}]\right)\right), \tag*{(4)}

where σ\sigma is the sigmoid function and [⋅,⋅][\cdot,\cdot] denotes concatenation. The smoothed relevance feature is then updated via:

r~i(l)=wi(l)⋅ri(l)+(1−wi(l))⋅r~i(l−1),(5)\tilde{\mathbf{r}}_i^{(l)} = w_i^{(l)} \cdot\mathbf{r}_i^{(l)} + (1-w_i^{(l)}) \cdot\tilde{\mathbf{r}}_i^{(l-1)}, \tag*{(5)}

with r~i(0)=0\tilde{\mathbf{r}}_i^{(0)} = 0. Intuitively, the gate learns to suppress transient attention noise while emphasizing persistent, instruction-aligned visual regions.

Finally, to combine extrinsic query relevance with intrinsic visual saliency, r~i(l)\tilde{\mathbf{r}}_i^{(l)} is concatenated with the token’s hidden state Xi(l)X_i^{(l)} and then fed into a shared scoring network MLPψ\mathrm{MLP}_{\psi} to predict the importance score:

si(l)=σ(MLPψ([Xi(l),r~i(l)])),(6)s_i^{(l)} = \sigma\left(\mathrm{MLP}_{\psi}\left([X_i^{(l)}, \tilde{\mathbf{r}}_i^{(l)}]\right)\right), \tag*{(6)}

where si(l)s_i^{(l)} denotes the importance score of the ii-th visual token at the ll-th layer. By keeping MLPϕ\mathrm{MLP}_{\phi} layer-specific while sharing MLPψ\mathrm{MLP}_{\psi} across all layers, we accommodate hierarchical semantic shifts while stabilizing the optimization process.

Since pruning requires discrete keep-or-drop decisions that are inherently non-differentiable, we employ the Straight-Through Estimator (STE) [4] to bridge this gap. A binary mask is obtained by thresholding:

Mi(l)=STE⁡(I(si(l)>τ)),(7)M_i^{(l)} = \operatorname{STE}\left(\mathbb{I}(s_i^{(l)} > \tau)\right), \tag*{(7)}

where I(⋅)\mathbb{I}(\cdot) is the indicator function and τ\tau is set to 0.5. During the forward pass, Mi(l)∈{0,1}\mathbf{M}^{(l)}_i \in\{0,1\} determines whether the token is kept or pruned; during the backward pass, the gradient bypasses the hard threshold and is approximated by that of the continuous score si(l)s_i^{(l)}, enabling end-to-end optimization. To ensure hierarchical consistency, we maintain a cumulative mask M^(l)\hat{\mathbf{M}}^{(l)} via a recursive Hadamard product:

M^(l)=M^(l−1)⊙M(l),(8)\hat{\mathbf{M}}^{(l)} = \hat{\mathbf{M}}^{(l-1)} \odot\mathbf{M}^{(l)}, \tag*{(8)}

where M^(0)=1\hat{\mathbf{M}}^{(0)} = 1, so that once a token is pruned, it is permanently excluded from all subsequent layers. Following [29], we apply this mask during training to cut off interactions between pruned and retained tokens, simulating the actual pruning behavior at inference time.

Our primary goal is to encourage the model to use the minimum number of vision tokens required to produce the correct answer. To achieve this, we introduce a pruning loss Lprune\mathcal{L}_{prune} to constrain the number of preserved tokens, which is defined as:

Lprune=(1L⋅Nvis∑l=1L∑i=1NvisM^i(l)−ρ)2,(9)\mathcal{L}_{prune} = \left( \frac{1}{L \cdot N_{vis}} \sum_{l=1}^{L} \sum_{i=1}^{N_{vis}} \hat{M}^{(l)}_i - \rho\right)^2, \tag*{(9)}

where LL is the number of pruning layers, NvisN_{vis} is the total number of visual tokens and we set ρ=0\rho= 0 to enforce maximal sparsity. With this regularization term, the final training objective is formulated as:

L=Lntp+λLprune,(10)\mathcal{L} = \mathcal{L}_{ntp} + \lambda\mathcal{L}_{prune}, \tag*{(10)}

where Lntp\mathcal{L}_{ntp} is the standard next-token prediction loss and we set λ=1\lambda= 1 in all our experiments. Notably, the shadow token is excluded from the prediction loss, allowing it to focus entirely on capturing cross-modal relevance without being distracted by the language modeling objective.

Budget-Deterministic Pruning

Once the pruning modules are trained, one can directly apply threshold-based pruning at each layer during inference, yielding sample-adaptive pruning schedules. However, practical deployment typically requires a fixed compute budget, which prior methods approximate by reporting the average number of retained tokens per layer. Furthermore, even with a fixed global budget, different inputs may produce distinct per-layer pruning trajectories, resulting in inconsistent memory access patterns and computation graphs, which is inherently hardware-unfriendly for optimized deployment.

To address these issues, we seek a deterministic per-layer schedule that fixes the token count at every layer. A natural question arises: how should such a schedule be determined? To this end, we investigate the underlying pruning characteristics learned by our modules. We randomly sample 500 image-text pairs from the LLaVA-mix665k dataset [26] and execute threshold-based pruning with τ=0.5\tau= 0.5 (identical to training). Figure 3 visualizes the mean and variance of the per-layer token retention ratio across all samples on LLaVA-v1.5 and Qwen2.5-VL. Surprisingly, the learned pruning trajectories exhibit high consistency across diverse inputs, with low inter-sample variance, and display a pronounced staircase pattern. This phenomenon suggests that through the joint optimization of all pruning modules, the model has autonomously identified a robust structural redundancy pattern in its architecture, reflecting the intrinsic stages of information flow and assimilation.

Statistics of learned pruning trajectories

Figure 3. Statistics of learned pruning trajectories. We visualize the per-layer token retention ratio obtained via threshold-based pruning on LLaVA-v1.5 and Qwen2.5-VL. The solid lines denote the mean values, while the shaded regions represent the variance across 500 randomly sampled image-text pairs from LLaVA-mix665k.

Taking LLaVA-v1.5 (Figure 3a) as an example, the trajectory begins with a sharp drop at the first layer, immediately discarding uninformative visual content such as background regions. In the subsequent early layers, the retention ratio enters a stable plateau, where virtually no further pruning occurs, suggesting that the incremental information gained across these shallow layers provides little discriminative signal. As the model progresses into the middle layers, a second significant decline emerges. This aligns with prior findings [33, 39] where the middle layers serve as the primary stage for intensive cross-modal interaction. At this stage, visual features are progressively aligned with linguistic queries, causing a substantial portion of visual tokens to become redundant. Moving into the deep layers, the retention ratio gradually approaches zero, confirming that essential visual information has been integrated into the hidden states of the text tokens, rendering the original visual modality entirely redundant for final response generation.

Motivated by the stability of the learned pruning trajectories, we propose Budget-Deterministic Pruning strategy, which repurposes the offline-calibrated average retention ratios η=[η1,η2,…,ηL]\boldsymbol{\eta} = [\eta_1, \eta_2, \ldots, \eta_L] as a structural prior, constructing a deterministic per-layer budget that preserves the learned layer-wise pruning distribution. Specifically, given a user-specified budget nˉ\bar{n}, defined as the average number of retained visual tokens per layer, the total token budget across all layers is nˉ⋅L\bar{n} \cdot L. Since the calibrated trajectory would naturally consume Nvis⋅∑l=1LηlN_{vis} \cdot\sum_{l=1}^{L} \eta_l tokens in total, we compute a global scaling factor to reconcile the two:

α=nˉ⋅LNvis⋅∑l=1Lηl.(11)\alpha= \frac{\bar{n} \cdot L}{N_{vis} \cdot\sum_{l=1}^{L} \eta_l}. \tag*{(11)}

The per-layer token budget is then determined as:

nl=⌊ηl⋅Nvis⋅α⌋,l=1,2,…,L,(12)n_l = \lfloor\eta_l \cdot N_{vis} \cdot\alpha\rfloor,\quad l = 1,2,\ldots,L, \tag*{(12)}

where ⌊⋅⌋\lfloor\cdot\rfloor denotes rounding to the nearest integer. This formulation applies a uniform scaling to the calibrated pruning profile, ensuring that the relative layer-wise distribution is preserved under any target budget.

While deterministic budgets ensure computational predictability, they inevitably sacrifice instance-level adaptivity—a key advantage of threshold-based pruning. To mitigate this, we introduce a diversity-aware selection mechanism that dynamically optimizes the token composition within each layer’s fixed capacity. At each layer ll, we first identify a candidate set of informative tokens whose importance scores exceed the threshold τ\tau, with the candidate count denoted as nl′=∑i=1NvisI(si(l)>τ)n_l' = \sum_{i=1}^{N_{\mathrm{vis}}} \mathbb{I}(s_i^{(l)} > \tau). If nl′≤nln_l' \le n_l, the budget is sufficient to accommodate the entire candidate set, and we simply retain the top-nln_l tokens by importance score. Conversely, when nl′>nln_l' > n_l, the model considers more tokens important than the allocated budget permits. In this case, naively selecting the top-nln_l tokens by scores may lead to the loss of critical visual information. Instead, we ensure more comprehensive visual representations by adapting the principle of diversity-aware selection from DART [38]. Under this approach, the top-npn_p tokens ranked by importance score are first selected as pivot tokens. For each remaining candidate, we compute its cosine similarity to all pivots and identify its maximum similarity value. The remaining nl−npn_l - n_p slots are filled by candidates with the lowest maximum similarity. Ultimately, this mechanism strikes a balance between importance and diversity, which is particularly effective in low-budget scenarios.

Experiments

Experimental Settings

Datasets and Benchmarks. We conduct comprehensive evaluations on 9 widely-used image-based multimodal benchmarks, including GQA [15], ScienceQA-IMG [28], TextVQA [32], POPE [22], MME [12], VQAv2 [13], MMBench [27], MMBench-CN [27] and MMVet [45]. Moreover, we experiment with 3 video understanding benchmarks, including TGIF-QA [16], MSVD-QA [41], and MSRVTT-QA [41]. All experiments on these benchmarks follow the default settings and evaluation metrics.

Implementation Details. In the experiments, we keep the base VLM weights frozen and use the LLaVA-mix665k dataset [26] to train our pruning modules. We apply our LearnPrunerV2 to various VLM architectures, including LLaVA-v1.5 [26] for image understanding, LLaVA-NeXT [25] for high-resolution inputs, Video-LLaVA [23] for video scenarios (see Appendix for results) and Qwen2.5-VL [3] for broader generalization. For comprehensive comparison, we select several recent token pruning methods as baselines, including importance-based methods (e.g., FastV [8], SparseVLM [49] and VisPruner [47]), diversity-based approaches (e.g., DART [38] and CDPruner [48]), and LearnPruner [33] which introduces learnable parameters for pruning and employs a heuristic pruning schedule.

Main Results

Results on LLaVA-v1.5. As shown in Table 1, we first evaluate our method on LLaVA-v1.5-7B, where images are represented by a fixed sequence of 576 tokens. The results demonstrate that our method consistently outperforms all previous methods across all three retention ratios. Notably, when retaining 128 tokens, our method maintains a remarkable 99.8% of the vanilla model’s performance, surpassing the previous state-of-the-art method by 1.3%, and this performance gap further widens to 2.1% when the budget is restricted to only 32 tokens. Since both our model and LearnPruner employ learnable modules, this significant lead directly validates the superiority of our proposed pruning criterion and pruning schedule.

MethodGQASQAVQAT^{\mathrm{T}}POPEMMEVQAv2^{\mathrm{v2}}MMBMMBCN^{\mathrm{CN}}MMVetAvg.
Vanilla61.969.558.285.9186278.564.758.331.1100.0%
FastV49.660.250.659.6149061.856.151.428.183.0%
SparseVLM56.067.154.980.5169673.860.051.129.792.9%
VisPruner58.269.157.084.6179475.862.757.333.798.5%
DART57.969.156.380.4172174.760.757.330.995.8%
CDPruner59.969.056.287.71769.676.663.155.032.898.3%
LearnPruner60.368.557.386.7182077.363.856.830.898.5%
LearnPrunerV261.068.457.786.2184377.764.458.532.499.8%
FastV46.151.147.848.0125655.048.042.725.872.7%
SparseVLM52.762.251.875.1150568.256.246.123.384.4%
VisPruner55.469.155.880.4168972.761.355.132.395.0%
DART54.769.354.773.9170571.359.554.026.591.1%
CDPruner58.668.155.387.51721.575.461.153.230.595.8%
LearnPruner58.968.356.686.8175076.062.655.729.696.7%
LearnPrunerV258.968.356.686.1183076.363.355.632.598.2%
FastV41.542.642.532.5109043.437.833.2-52.0%
SparseVLM48.357.346.167.9129058.651.440.6-68.0%
VisPruner52.269.253.972.7156767.758.452.728.889.4%
DART52.969.352.269.1161567.158.550.025.087.1%
CDPruner57.069.553.287.9167073.659.649.627.892.9%
LearnPruner57.268.256.184.5167274.060.855.526.193.6%
LearnPrunerV257.668.655.785.8174774.661.954.429.995.7%

Table 1. Performance comparisons of different pruning methods on LLaVA-v1.5-7B. Avg. denotes the performance relative to the vanilla model, averaged across all benchmarks. Bold and underlined values denote the best and second-best results.

Results on LLaVA-NeXT. To verify its scalability for high-resolution scenarios, we evaluate our method on LLaVA-NeXT-7B. LLaVA-NeXT handles high-resolution inputs by partitioning the image into multiple sub-grids and encoding them independently, yielding a massive baseline of 2,880 visual tokens. As shown in Table 2, LearnPrunerV2 retains 99.7% of the original performance while utilizing only 22.2% of the visual tokens. Furthermore, on several benchmarks such as POPE, MME, and MMBench, our pruned model achieves performance that is either comparable to or even superior to its vanilla counterpart. These results indicate that high-resolution inputs inherently contain substantial visual redundancy, where many tokens may contribute marginal or even distracting signals. By removing uninformative visual context, the model can more accurately focus on the most salient features, thereby leading to more precise predictions.

MethodGQASQAVQAT^{\mathrm{T}}POPEMMEVQAV2^{\mathrm{V2}}MMBMMBCN^{\mathrm{CN}}MMVetAvg.
Upper Bound, 2,880 Tokens (100%)
Vanilla64.270.261.386.3184281.267.960.540.0100.0%
Retain Averaged 640 Tokens (↓ 77.8%)
FastV62.069.160.079.5180779.565.853.533.394.4%
SparseVLM60.367.757.885.3177277.165.758.636.195.4%
DART61.368.259.585.0174178.364.957.136.995.7%
CDPruner62.667.958.487.3180879.966.357.538.897.4%
LearnPruner63.870.060.585.1183780.567.559.536.198.1%
LearnPrunerV264.270.160.686.9185681.268.060.738.799.7%
Retain Averaged 320 Tokens (↓ 88.9%)
FastV54.968.254.849.5153969.660.042.520.078.6%
SparseVLM57.767.355.976.9169473.464.356.732.891.0%
DART61.767.557.681.0172075.764.258.235.794.1%
CDPruner61.667.857.487.2175878.465.555.735.695.2%
LearnPruner62.268.658.484.8184578.366.858.335.096.3%
LearnPrunerV263.568.559.387.0184480.067.860.236.698.2%
Retain Averaged 160 Tokens (↓ 94.4%)
DART56.867.854.975.3162672.562.053.632.288.9%
CDPruner60.867.555.486.8173876.764.253.834.193.3%
LearnPruner58.767.655.083.1178476.265.355.832.892.8%
LearnPrunerV261.667.858.085.6178177.965.559.237.196.2%

Table 2. Performance comparisons of different pruning methods on LLaVA-NeXT-7B.

Results on Qwen2.5-VL. To further assess the broad applicability of LearnPrunerV2, we extend our evaluation to Qwen2.5-VL-7B, a leading open-source VLM with an architecture distinct from the LLaVA family. Since the original SFT data for Qwen2.5-VL is not publicly accessible, we leverage the LLaVA-mix665k dataset to generate model responses, which serve as pseudo-labels for training our pruning modules. Furthermore, although Qwen2.5-VL naturally supports native dynamic resolution, we fix the input size to a standardized budget of 1,280 tokens to ensure a fair and consistent comparison across all pruning methods. As shown in Table 3, LearnPrunerV2 still maintain its dominant performance, even exceeding 100% average relative accuracy at 66.7% and 77.8% pruning rates, showing the robust generalizability of our approach.

MethodGQASQAVQAT^{\mathrm{T}}MMEMMBAvg.
Upper Bound, 1280 Tokens (100%)
Vanilla60.975.983.3231483.8100.0%
Retain Averaged 426 Tokens (↓ 66.7%)
FastV58.479.181.5233081.499.2%
LearnPruner59.377.481.1233983.299.4%
LearnPrunerV260.081.480.7235583.4100.8%
Retain Averaged 284 Tokens (↓ 77.8%)
FastV53.677.879.2222879.195.3%
LearnPruner59.077.079.6235882.298.8%
LearnPrunerV259.182.279.6236282.2100.2%
Retain Averaged 142 Tokens (↓ 88.9%)
FastV42.870.562.9156556.274.7%
LearnPruner51.976.176.1229281.894.7%
LearnPrunerV256.682.274.9232081.097.6%

Table 3. Performance comparisons of different pruning methods on Qwen2.5-VL-7B.

Relevance SourceAvg.
last token attention96.5
last token logit97.4
shadow token (ours)97.8

Table 4b. (b) Ablation study on source of relevance feature.

Parameter SharingAvg.
fully independent97.3
fully shared96.9
hybrid (ours)97.8

Table 4c. (c) Ablation study on parameter sharing scheme of the learnable pruning module.

Ablation Study

We evaluate the learnable pruning criterion and budget-deterministic pruning strategy on LLaVA-v1.5-7B. Unless otherwise specified, all experiments use a fixed budget of 64 tokens, and we report average relative performance over GQA, SQA, VQAT^{\mathrm{T}}, POPE, and MME.

Input components. As shown in Table 4(a), we compare three configurations for the pruning module’s input. When relying solely on the hidden states, the module determines token importance based only on intrinsic visual redundancy, which results in a performance of 96.6%. By concatenating the relevance feature, the model can further incorporate cross-modal correlations with the textual query, boosting the average performance to 97.2%. However, inter-layer attention fluctuations often introduce distracting noise. The gating mechanism mitigates this by filtering transient signals, stabilizing importance estimation and yielding a performance of 97.8%.

Input ComponentsAvg.
hidden states96.6
+ relevance feature97.2
+ gating mechanism (ours)97.8

Table 4. Ablation study on different configurations of the learnable pruning module (LPM). (a) Ablation study on input components of the learnable pruning module.

Relevance source. Table 4(b) investigates the influence of different cross-modal relevance sources. While the last instruction token is a conventional choice for sequence aggregation, utilizing its attention scores results in poor performance (96.5%) because the attention scores are numerically overwhelmed by the hidden states. Although last token logits (97.4%) alleviate the scaling bottleneck, they still underperform the shadow token (97.8%), which we attribute to the inherent multi-task interference between next-token prediction and pruning guidance. In contrast, the shadow token serves as a dedicated aggregator that focuses exclusively on capturing instruction-relevant token importance.

Parameter sharing scheme. The learnable pruning module mainly consists of a relevance projector (MLPϕ_{\phi}) and a scoring network (MLPψ_{\psi}). Table 4(c) evaluates different parameter sharing schemes for these components. Our hybrid strategy achieves the best performance (97.8%), outperforming both fully independent (97.3%) and fully shared (96.9%) counterparts. This is because attention logits exhibit significant inter-layer variance, necessitating a layer-specific MLPϕ_{\phi} for adaptive mapping. Conversely, a shared MLPψ_{\psi} receives more gradients from different layers, facilitating robust convergence and stabilizing the optimization to learn a generalized pruning criterion.

Visual token selection. Table 5 evaluates the impact of our diversity-aware selection against the standard Top-K strategy under a fixed layer-wise budget. While both approaches perform similarly at a higher budget of 64 tokens (97.8% vs. 97.7%), our strategy demonstrates a clear advantage when the budget is restricted to 32 tokens, outperforming Top-K by 0.9%. This gap suggests that simple Top-K selection is prone to information redundancy, as tokens with the highest scores are often spatially clustered or semantically similar. In contrast, our approach encourages the selection of more diverse visual cues, thereby broadening the overall information coverage within the restricted budget.

strategyGQASQAVQAT^{\mathrm{T}}POPEMMEAvg.
Retain Averaged 64 Tokens
top-k58.968.756.685.9180597.7%
ours58.968.356.686.1183097.8%
Retain Averaged 32 Tokens
top-k56.768.755.285.1171695.3%
ours57.668.655.785.8174796.2%

Table 5. Ablation study on token selection strategy under Budget-Deterministic Pruning (BDP).

Efficiency Analysis

To demonstrate the efficiency of LearnPrunerV2, we measure the FLOPs, prefill time, total inference time and KV cache size of the pruned model compared to the vanilla LLaVA-NeXT-7B. All experiments are conducted on the POPE benchmark. As shown in Table 6, when the number of visual tokens is reduced from 2,880 to 160, we achieve a 9.7× reduction in TFLOPs and a 6.2× speedup in prefill time. Furthermore, the KV cache size is slashed by 8.7×, while the total inference time improves by 3.7×. These results highlight the effectiveness of our framework in reducing the computational overhead of VLMs

MethodTokenTFLOPsPrefill (s)Total (s)KV Cache (MB)
LLaVA-NeXT288033.8 (1.0×)3197.1 (1.0×)3569.8 (1.0×)1166.6 (1.0×)
+ ours64010.2 (3.3×)1231.7 (2.6×)1783.3 (2.0×)370.9 (3.1×)
+ ours1603.5 (9.7×)514.3 (6.2×)967.5 (3.7×)133.4 (8.7×)

Table 6. Efficiency analysis on the POPE benchmark. All experiments are conducted on a single A100 GPU.

Conclusion

We introduce LearnPrunerV2, a learnable progressive token pruning framework for accelerating VLMs. It estimates token importance from intrinsic visual saliency and query relevance, and employs a Budget-Deterministic Pruning (BDP) schedule that assigns fixed per-layer budgets by exploiting the inherent architectural redundancy of VLMs. Extensive experiments on multiple VLM architectures and benchmarks demonstrate strong accuracy-efficiency trade-offs and clear gains over prior methods.

Appendix

Results on Video-LLaVA

To further assess the broad applicability of LearnPrunerV2, we extend our evaluation from the image domain to video understanding. Video-LLaVA [23] processes each video by uniformly sampling 8 frames and encoding them independently, producing a total of 2,048 visual tokens. We evaluate on three established video question answering benchmarks: TGIF-QA [16], MSVD-QA [41] and MSRVTT-QA [41]. Following [47], we use the first 1,000 samples from each benchmark for evaluation, and employ GPT-4.1 to assist in scoring open-ended responses. As shown in Table 7, LearnPrunerV2 consistently outperforms FastV across all three benchmarks, especially under lower retention ratios. These results demonstrate the strong generalizability of LearnPrunerV2 from image-based to video-based VLMs.

MethodTGIF-QA Acc.TGIF-QA ScoreMSVD-QA Acc.MSVD-QA ScoreMSRVTT-QA Acc.MSRVTT-QA Score
Upper Bound, 2,048 Tokens (100%)
Vanilla18.11.2764.13.4156.12.97
Retain Averaged 456 Tokens (↓ 77.8%)
FastV16.41.1760.33.2453.02.84
LearnPrunerV217.81.2164.93.3955.02.89
Retain Averaged 228 Tokens (↓ 88.9%)
FastV12.41.0055.22.9350.42.70
LearnPrunerV216.41.1962.13.3053.82.84

Table 7. Video understanding performance comparisons of different pruning methods on Video-LLaVA-7B. Performance is evaluated on the first 1,000 samples of each benchmark. We employ GPT-4.1 to assist in scoring open-ended responses.

Data Efficiency

To investigate the data efficiency of LearnPrunerV2, we train the model with varying proportions of the LLaVA-mix665k [26] training dataset (from 10% to 100%) and evaluate the resulting performance on both LLaVA-v1.5-7B and LLaVA-NeXT-7B under two retention budgets, respectively. As shown in Figure 4, LearnPrunerV2 exhibits remarkable data efficiency across both models. On LLaVA-v1.5, using only 10% of the training data already achieves 98.5% relative performance under the 128-token budget. As the training data scales to 50%, performance largely converges, achieving results on par with full-dataset training. Moreover, we observe that lower token budgets generally require more training data to reach convergence. This can be attributed to the fact that more aggressive pruning demands a more precise pruning criterion, which in turn requires sufficient training signal to learn.

Figure 4. Performance comparisons of LearnPrunerV2 models trained with different proportion of the training dataset.

Visualization

To better illustrate the pruning behavior learned by LearnPrunerV2, we visualize the token pruning results across different layers on LLaVA-v1.5-7B. As shown in Figure 5 and Figure 6, LearnPrunerV2 first removes a large number of uninformative tokens at shallow layers, such as those corresponding to padding regions or background areas. Moving to the middle layers, the retained tokens progressively converge toward query-relevant visual regions. By the deeper layers, nearly all visual tokens have been pruned. This progressive pruning pattern reveals the inherent characteristics of visual information flow within VLMs, which is consistent with findings in prior works [39].

Figure 5. Visualization of the pruning results.

Figure 6. Visualization of the pruning results.

Additionally, we compare the importance score heatmaps predicted by LearnPrunerV2 with the last-token attention scores adopted by previous methods. As shown in Figure 7 and Figure 8, last-token attention suffers from notable positional bias and exhibits substantial fluctuations across layers, and typically requires deeper layers (e.g., layer 16) to localize query-relevant regions. In contrast, benefiting from our learnable shadow token and gating mechanism, LearnPrunerV2 is able to identify query-relevant regions at much earlier layers (e.g., layer 12), while also exhibiting stronger cross-layer consistency.

Figure 7. Visualization of the score heatmap.

Figure 8. Visualization of the score heatmap.

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).arxiv.org/abs/2303.08774
  2. [2]Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. 2025. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference. 9392–9401.arxiv.org/abs/2503.02175
  3. [3]Sai Bai, Kegin Chen, Xueqing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025).arxiv.org/abs/2502.13923
  4. [4]Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013).arxiv.org/abs/1308.3432
  5. [5]Kaitong Cai, Jusheng Zhang, Jing Yang, Yiya Fan, Pengtao Xie, Jian Wang, and Keze Wang. 2025. Flashvlm: Text-guided visual token selection for large multimodal models. arXiv preprint arXiv:2512.20561 (2025).arxiv.org/abs/2512.20561
  6. [6]Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zheui Chen, Zhi Chen, Pei Chu, et al. 2024. Internal2m technical report. arXiv preprint arXiv:2403.17297 (2024).
  7. [7]Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13817–13827.
  8. [8]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision. Springer, 19–35.
  9. [9]Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. 2024. LongVILLA: Scaling Long-Context Visual Language Models for Long Videos. arXiv preprint arXiv:2408.10188 [cs.CV].arxiv.org/abs/2408.10188
  10. [10]Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. 2024. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024).
  11. [11]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35 (2022), 16344–16359.arxiv.org/abs/2205.14135
  12. [12]Chaoyou Fu, Peiwen Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023).arxiv.org/abs/2306.13394
  13. [13]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6904–6913.
  14. [14]Wenxuan Huang, Zizhe Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. 2024. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.08786 (2024).
  15. [15]Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6700–6709.arxiv.org/abs/1902.09506
  16. [16]Yunseok Jang, Youngjae Yu, Youngju Yim, and Youngeun Kim. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2758–2766.arxiv.org/abs/1704.04497
  17. [17]Ahmadreza Jeddi, Negin Baghban-zadeh, Elham Dolatabadi, and Babak Taati. 2025. Similarity-Aware Token Pruning: Your VLM but Faster. arXiv:2503.11549 [cs.CV] https://arxiv.org/abs/2503.11549
  18. [18]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825 [cs.CL] https://arxiv.org/abs/2310.06825
  19. [19]Pengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao, Ke Yan, Shouhong Ding, and Rongrong Ji. 2025. VISTA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference. In Proceedings of the 33rd ACM International Conference on Multimedia. 11130–11139.DOI
  20. [20]Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyan Li. 2024. LLaVA-OneVision: Easy Visual Task Transferer. arXiv:2408.03326 [cs.CV] https://arxiv.org/abs/2408.03326
  21. [21]Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2025. Tokenpacker: Efficient visual projector for multimodal llm. International Journal of Computer Vision 133, 10 (2025), 6794–6812.arxiv.org/abs/2407.02392
  22. [22]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023).arxiv.org/abs/2305.10355
  23. [23]Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023).arxiv.org/abs/2311.10122
  24. [24]Aixin Liu, Bo Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024).arxiv.org/abs/2412.19437
  25. [25]Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/
  26. [26]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916.
  27. [27]Ying Lu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024. Mmbench: Is your multi-modal model an all-around player?. In European Conference on Computer Vision. Springer, 216–233.
  28. [28]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 (2022), 2507–2521.arxiv.org/abs/2209.09513
  29. [29]Yongming Rao, Wenliang Zhao, Hee Su Kim, Jiyeon Lim, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems 34 (2021), 13937–13949.arxiv.org/abs/2106.02034
  30. [30]David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258 (2024).arxiv.org/abs/2404.02258
  31. [31]Wenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. 2025. Growing a twig to accelerate large vision-language models. Proceedings of the IEEE International Conference on Computer Vision.DOI
  32. [32]Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8317–8326.arxiv.org/abs/1904.08920
  33. [33]Ryinichi Takezoe, Yaqian Li, Zi-Hao Bo, Anzhou Hou, Mo Guang, and Kaiwen Long. [n. d.]. LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models. In The Fourteenth International Conference on Learning Representations.
  34. [34]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv.org/abs/2302.13971
  35. [35]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  36. [36]Hanshi Wang, Zekun Xu, Jin Gao, Yufan Liu, Weiming Hu, Ke Wang, Zhipeng Zhang, et al. [n. d.]. Each Complexity Deserves a Pruning Policy. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  37. [37]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024).
  38. [38]Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qinton Zhang, Weijia Li, Chenghui He, and Linfeng Zhang. 2025. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494 (2025).arxiv.org/abs/2502.11494
  39. [39]Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, and Xiaoyu Shen. 2026. HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit. (2026).arxiv.org/abs/2602.23699
  40. [40]Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Liu, Pan Zhang, Yuhang Zhang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 (2024).arxiv.org/abs/2410.17247
  41. [41]Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia. 1645–1653.
  42. [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
  43. [43]Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19792–19802.arxiv.org/abs/2412.04467
  44. [44]Xubing Ye, Yukang Yan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. 2025. Atp-llava: Adaptive token pruning for large vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24972–24982.arxiv.org/abs/2412.00447
  45. [45]Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mmv-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023).arxiv.org/abs/2308.02490
  46. [46]Evelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen, Ke Yan, Shouqiong Ding, Biqing Qi, and Linfeng Zhang. 2026. D^2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 12412–12420.
  47. [47]Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. 2025. Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs. Proceedings of the IEEE International Conference on Computer Vision (2025).arxiv.org/abs/2412.01818
  48. [48]Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. 2025. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs. arXiv preprint arXiv:2506.10967 (2025).arxiv.org/abs/2506.10967
  49. [49]Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. 2025. SCSVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. In International Conference on Machine Learning.arxiv.org/abs/2410.04417
  50. [50]Jinguo Zhu, Weiwen Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xinggang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinhan He, Tan Jiang, Jiayu Luo, Yi Wang, Conghui He, Botian Shi, Xincheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv:2504.10479 [cs.CV] https://arxiv.org/abs/2504.10479

Paper details

vlnc:W_m72wgyyb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents