betaFeedback

WorksComputer Science

Beyond Token Stacking: Multi-scale Visual Modulation and Adaptive Fused-feature Calibration for MLLMs

Abstract

Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.

CCS Concepts

• Computing methodologies → Scene understanding.

ACM Reference Format

Ye Xiao, Shezheng Song, Yi Ren, Shasha Li, Bin Ji, Xiaodong Liu, Jun Ma, and Jie Yu. 2026. Beyond Token Stacking: Multi-scale Visual Modulation and Adaptive Fused-feature Calibration for MLLMs. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3836045

Introduction

In recent years, the rapid advancement of LLMs [1, 47] has significantly propelled the evolution of MLLMs [8, 19], enabling them to handle multiple modalities (i.e., text, images, video, and audio). Notably, Vision-Language Models (VLMs) [43, 44, 48], which integrate pretrained visual encoders with LLMs, have achieved impressive results on a range of vision-language tasks, including image caption [25] and visual question answering [3].

To effectively align and fuse visual and linguistic representations, VLMs typically rely on full-parameter fine-tuning. However, as the parameter scales of both visual and language models grow exponentially, full fine-tuning that updates most or all parameters requires substantial training memory and computational costs. To mitigate this challenge, researchers have explored parameter-efficient fine-tuning (PEFT) for VLMs [4, 14, 16, 23, 35, 45], where pretrained backbones are largely kept frozen and only a small set of lightweight modules is updated. Depending on how visual information is incorporated into the LLM, existing methods can be categorized into two paradigms, as illustrated in Figure 1.

Figure 1. Framework comparisons between previous methods and our method.

This work is licensed under a Creative Commons Attribution 4.0 International License.

MM ’26, Rio de Janeiro, Brazil

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2213-4/2026/11

https://doi.org/10.1145/3767308.3836045

(i) Concatenation: This paradigm [16, 35, 45] projects the output of the visual encoder into the input space of the language model via linear projection, then concatenates the visual prompts with text embedding tokens, and finally feeds the combined sequence as a whole into the LLM. Although the above methods preserve relatively rich visual information, the large number of visual tokens often exhibits substantial redundancy. For example, an image encoded by ViT/14 produces up to 196 patch tokens, whereas the accompanying text is typically far shorter, averaging 12.84 tokens on GQA [18] and 6.3 tokens on VQAv2 [11]. (ii) Visual Knowledge: This paradigm [9, 14, 23] injects visual knowledge into the FFN layers of the LLM to avoid input sequence-length inflation. However, these approaches lack visual contextual information at the input stage, which may reduce the likelihood of activating image-related knowledge. Meanwhile, the injected visual features are usually extracted at a single granularity, making it difficult to capture both global structure and fine-grained details. In fact, robust visual grounding often benefits from multi-scale information that captures both global semantics and fine-grained details. Additionally, due to the inherent semantic gap and mismatched feature statistics between vision and language [49], naively injecting visual features into the FFN layers could perturb activation distributions, inducing distribution shifts and compromising the stability of cross-modal feature fusion [20].

To address the aforementioned issues, in this paper, we develop a novel visual enhancement framework for MLLMs named BTS. Our method consists of two key components: (1) Multi-scale Visual Modulation (MVM): Since FFN-based visual knowledge injection methods lack explicit and rich visual contextual priors at the input stage, we propose MVM mechanism. Specifically, we perform multi-stage downsampling on the visual encoder outputs to extract multi-scale visual information that integrates coarse-grained semantics with fine-grained details. We then employ a lightweight adapter to modulate these multi-scale features into a visual bias, which is injected into text token embeddings in a residual manner. This design introduces multi-scale visual information into the language model without increasing the token sequence length. (2) Adaptive Fused-feature Calibration (AFC): To mitigate the activation distribution perturbations induced by visual knowledge injection in the FFN layers, we introduce AFC module. AFC adaptively calibrates fused features via per-channel affine transformations to decouple visual injection from distribution alignment, where the injection branch contributes cross-modal gains and the calibration branch restores activation statistics to a stable and controllable regime. Consequently, AFC achieves a more reliable multimodal fusion process and maintains the linguistic priors of the LLM. We conduct systematic evaluations of BTS on multiple multimodal benchmarks, and extensive experimental results show that BTS significantly enhances the multimodal understanding capability of the model. Overall, our contributions are summarized as follows:

  • We propose a Multi-scale Visual Modulation (MVM) mechanism, which leverages lightweight feature modulation to achieve efficient injection of multi-scale visual information at the input stage, thereby enhancing the ability of the model to perceive both scene structure and fine-grained details.

  • We introduce an Adaptive Fused-feature Calibration (AFC) module that performs adaptive calibration of cross-modal fused features via per-channel affine transformations. This module effectively mitigates activation distribution shifts induced by visual knowledge injection and improves the effectiveness of multimodal feature fusion.

  • We conduct extensive experiments on multiple multimodal benchmarks. BTS achieves state-of-the-art performance, demonstrating its effectiveness.

Related Work

Multimodal Large Language Models

Multimodal Large Language Models unify modalities such as vision and language within a single framework, enabling end-to-end perception and reasoning. Early work primarily relied on cross-modal contrastive learning (e.g., CLIP [40] and ALBEF [29]) or encoder-decoder pre-training architectures (e.g., VLBERT [32], LXMERT [46], and Unified VLP [53]) to learn vision-language representations. Recently, an increasing number of studies have shifted toward efficiently integrating visual information into LLMs to support image understanding, multi-turn dialogue, and complex reasoning. For instance, Flamingo [2] achieves modality bridging by introducing gated cross-attention between pre-trained vision and language models; BLIP [28] proposes a unified vision-language pre-training framework that supports both understanding and generation; BLIP-2 [27] further employs the Q-Former to bridge a frozen visual encoder and a frozen LLM; LLaVA [31] constructs a general-purpose visual assistant through visual instruction tuning; and works such as PaLI [5], Kosmos-1 [17], and InternVL [6] continue to expand general multimodal capabilities and visual reasoning performance using larger-scale data and stronger components. Despite the substantial progress made by these methods in multimodal capabilities, the cost of full fine-tuning has become increasingly prohibitive as the scale of vision and language models continues to grow exponentially.

Parameter-Efficient Fine-Tuning for VLMs

Parameter-Efficient Fine-Tuning techniques achieve rapid adaptation to downstream tasks by updating only a small number of learnable parameters or introducing lightweight auxiliary modules, while keeping the majority of pre-trained weights frozen. These methods have been widely adopted for task transfer in the vision [22, 24, 41], language model [15, 39, 50], and multimodal model [21, 23, 35] domains. PEFT methods for VLMs have demonstrated competitive performance. Depending on how visual information is fused into the LLM, these methods can be broadly categorized into two paradigms. The first paradigm (e.g., VL-Adapter [45], VL-PET [16], and LaVIN [35]) projects image features into visual prompt tokens, concatenates them with text tokens, and feeds the resulting sequence into the LLM, while fine-tuning only the projection layer and the PEFT modules inserted into the LLM. However, these methods still suffer from efficiency bottlenecks due to redundant visual tokens. The second paradigm (e.g., MemVP [23] and LLaMA-Adapter v2 [9]) remove image from the input stage and instead treat it as an external visual knowledge injected into the FFN layers of the LLM. However, completely ignoring images at the input stage may lead to insufficient activation of knowledge, especially since text-only semantic modeling can introduce ambiguity in tasks that require integrating textual and visual information. Moreover, fusing visual features at the FFN layers may disrupt the activation distribution of the pre-trained language model and still struggle to address the issues of representation misalignment and activation distribution drift caused by cross-modal semantic differences.

Preliminary

Vision-Language Model

Vision Encoder. The vision encoder typically employs the CLIP-ViT [40] architecture. Given an image I∈RH×W×CI \in\mathbb{R}^{H \times W \times C}, we split it into N=HWP2N = \frac{HW}{P^2} non-overlapping patches of size P×PP \times P, where (H,W)(H, W) denotes the image resolution and CC is the number of channels. These patches are flattened into tokens and prepended with a learnable [CLS][\mathrm{CLS}] token. The resulting token sequence is then fed into Transformer layers to obtain visual representations.

Large Language Model. The core of an LLM is a stack of Transformer blocks, each comprising multi-head self-attention (MHSA) and FFN, typically equipped with normalization and residual connections for stable optimization. Concretely, given the token xx, a Transformer block updates it as follows:

xu=x+MHSA(Norm(x)),xu′=xu+FFN(Norm(xu))(1)\begin{aligned} x^{u} &= x + MHSA(Norm(x)),\\ x^{u'} &= x^{u} + FFN(Norm(x^{u})) \tag*{(1)} \end{aligned}

where uu is the Transformer layer number. The FFN consists of two fully connected layers and can be written as:

FFN(x)=ϕ(xW1)W2,(2)\mathrm{FFN}(x) = \phi(xW_{1})W_{2}, \tag*{(2)}

where W1W_{1} and W2W_{2} are learnable projection matrices and ϕ(⋅)\phi(\cdot) denotes a non-linear activation.

FFN with Visual Knowledge

Previous work interprets FFNs in pre-trained language models as key-value memory that stores and retrieves factual associations [7, 10, 38]. However, these studies have focused on purely textual models, leaving visual knowledge integration underexplored. Recently, MemVP [23] addressed this gap by encoding visual knowledge into additional keys and values within FFNs, providing language models with externally visually conditioned knowledge.

The FFN process is shown in Equation (2). In fact, W1W_1 and W2W_2 can be expressed as a collection of key-value pairs:

W1=(k1,k2,…,kD), W2=(v1,v2,…,vD)T,(3)W_1=(k_1,k_2,\ldots,k_D),\ W_2=(v_1,v_2,\ldots,v_D)^\mathsf{T}, \tag*{(3)}

where ki∈Rdk_i \in\mathbb{R}^d and vi∈Rdv_i \in\mathbb{R}^d denote the ii-th key and value, respectively. FFN performs retrieval from its key-value memory. The retrieval process for visual features could be formulated as:

Retrieval⁡(x)=∑i=1nϕ(⟨x,K(zi)⟩)⋅V(zi),(4)\operatorname{Retrieval}(x)=\sum_{i=1}^{n}\phi(\langle x,K(z_i)\rangle)\cdot V(z_i), \tag*{(4)}

where K(zi),V(zi)∈RdK(z_i), V(z_i) \in\mathbb{R}^d denote the key and value corresponding to the visual feature ziz_i, respectively. Accordingly, the formulation Equation (4) shares a similar form and performs a similar process as with Equation (2).

Method

We propose BTS, a novel visual enhancement framework for MLLMs, as illustrated in Figure 2. (a) We first strengthen early visual grounding by introducing multi-scale visual context at the input stage. Specifically, MVM provides both coarse and fine visual information, where the coarse information capture global layout and scene structure, while the fine information preserves local details. This design allows the language model to establish visual understanding at an early stage, rather than deferring visual reasoning to deeper layers. (b) Furthermore, we improve the stability of cross-modal fusion within the FFN layers through feature calibration. Since cross-modal injection could easily perturb the activation statistics of the LLM, AFC performs lightweight feature calibration to keep fused activations within a well-behaved range. This makes cross-modal fusion more robust and prevents visual signals from drifting the residual stream away from the pretrained linguistic regime. In summary, BTS activates richer visual context early for better grounding and maintains stable fusion during FFN injection, enhancing multimodal reasoning.

Method overview of BTS, including MVM and AFC modules

Figure 2. Method overview. BTS consists of two principal components: a Multi-scale Visual Modulation (MVM) mechanism and an Adaptive Fused-feature Calibration (AFC) module. MVM injects multi-scale visual contextual information into text embeddings via lightweight residual modulation without increasing the token sequence length, whereas AFC calibrates cross-modal fused features by applying per-channel affine transformations in the FFN layers, thereby stabilizing cross-modal fusion.

Multi-scale Visual Modulation

Existing methods such as MemVP [23] inject visual information only within the FFN layers of the LLM, while the input stage remains purely textual. This results in a lack of explicit visual contextual priors during early processing, making it difficult to activate image-relevant factual knowledge and fine-grained visual information promptly. Consequently, later layers face greater pressure for cross-modal alignment and information retrieval. Based on this issues, we propose the Multi-scale Visual Modulation (MVM) mechanism, which extracts multi-scale visual features and converts them into a lightweight residual bias injected at the input stage, furnishing the LLM with earlier, more explicit, and richer visual priors.

Specifically, given an image II, the vision encoder outputs a sequence of visual token features ZZ. We construct multi-scale visual representations by pooling adjacent tokens. Concretely, we apply pooling operations with different kernel sizes in a multi-stage down-sampling operator Down(⋅)Down(\cdot) to obtain SS scales of visual sequences:

{Z(s)}s=1S=Down(Z).(5)\{Z^{(s)}\}_{s=1}^{S}=Down(Z). \tag*{(5)}

The shallower scale preserves fine-grained local information, while the deeper scale emphasizes the global semantic structure. Next, for each scale, we project visual tokens into the LLM hidden space (dimension dd) and perform mean pooling over tokens to form a scale-wise context vector:

v(s)=1Ns∑j=1NsProj(Zj(s))∈RB×d.(6)v^{(s)}=\frac{1}{N_s}\sum_{j=1}^{N_s}Proj(Z_j^{(s)})\in\mathbb{R}^{B\times d}. \tag*{(6)}

Notably, prior methods [12, 30] typically crop or partition the original image and resize the resulting sub-images, then process each sub-image with multiple forward passes through the visual encoder to obtain multi-scale features. In contrast, our method produces multi-scale features without repeatedly invoking the visual encoder, thereby maintaining efficiency. We then aggregate multi-scale contexts to obtain a unified visual context representation vˉ=1S∑s=1Sv(s)\bar{v}=\frac{1}{S}\sum_{s=1}^{S}v^{(s)}. On top of this, we introduce a visual-bias adapter that maps the aggregated multi-scale visual context vˉ\bar{v} to a residual bias aligned with the LLM hidden space:

δ=tanh⁡(α)⋅Adapter(vˉ),(7)\delta=\tanh(\alpha)\cdot Adapter(\bar{v}), \tag*{(7)}

where Adapter(⋅)Adapter(\cdot) is a two-layer feed-forward adapter (implemented as a MLP) that produces the visual bias to be injected into textual representations. α\alpha is a learnable scalar gate, and tanh⁡(⋅)\tanh(\cdot) constrains the injection strength to (−1,1)(-1,1), which stabilizes training and prevents overly strong visual modulation from overriding language priors. Finally, to incorporate visual information at the input stage, we add the resulting bias to text token embeddings in a residual manner. Given the text token sequence xx with embedding matrix xemb=Embedding(x)∈RB×T×dx_{\mathrm{emb}}=Embedding(x)\in\mathbb{R}^{B\times T\times d}, we broadcast the multi-scale visual bias δ∈RB×1×d\delta\in\mathbb{R}^{B\times1\times d} along the sequence dimension and add it element-wise to obtain xinput=xemb+δx_{\mathrm{input}}=x_{\mathrm{emb}}+\delta, where xinputx_{\mathrm{input}} denotes the input representations augmented with visual information. A key advantage of this design is that it injects multi-scale visual contextual information at the input stage without increasing the token sequence length, thereby improving the utility of visual information for complex vision-language tasks.

Adaptive Fused-feature Calibration

Based on the perspective that FFN can be viewed as key-value memory stores for factual knowledge [10], we treat visual knowledge as new memory entries and inject it into FFN layers to compensate for the lack of vision-related knowledge in language models. Concretely, from the FFN standpoint, visual features are converted into additional key-value pairs (K(zi),V(zi))(K(z_i),V(z_i)) and written into the FFN retrieval space. This augments the original language-fact memory with visually conditioned knowledge support, expanding the capacity of the model for vision-grounded facts. After inserting visual memory entries, the output of the FFN can be formulated as the joint retrieval of both linguistic and visual memories:

FFN(x)=∑i=1Dϕ(⟨x,ki⟩)⋅vi+∑i=1nϕ(⟨x,K(zi)⟩)⋅V(zi),(8)FFN(x)=\sum_{i=1}^{D}\phi(\langle x,k_i\rangle)\cdot v_i+\sum_{i=1}^{n}\phi(\langle x,K(z_i)\rangle)\cdot V(z_i), \tag*{(8)}

where ziz_i denotes visual feature, and K(zi)K(z_i) and V(zi)V(z_i) are its corresponding visual key and value, respectively. However, naively injecting visual knowledge into the FFN layers may lead to representation misalignment and activation distribution drift due to cross-modal semantic discrepancies. To this end, we propose an Adaptive Fused-feature Calibration (AFC) module, which performs per-channel affine transformation on the fused activations to adaptively calibrate their distribution. This decouples visual injection from distribution alignment, thereby improving the stability and controllability of the fusion process.

Beyond the standard FFN formulation commonly used in small to medium-scale language models, larger models often adopt Gated Linear Units (GLU) [47] to enhance the FFN’s expressive capacity. In practice, we adopt a two-sided SiLU gating mechanism to facilitate feature interaction in the FFN, formulated as follows:

FFN⁡(x)=∑i=1D⟨SiLU⁡(x),SiLU⁡(ki)⟩⋅vi+∑i=1n⟨SiLU⁡(x),SiLU⁡(K(zi))⟩⋅V(zi).(9)\operatorname{FFN}(x)=\sum_{i=1}^{D}\langle\operatorname{SiLU}(x),\operatorname{SiLU}(k_i)\rangle\cdot v_i+\sum_{i=1}^{n}\langle\operatorname{SiLU}(x),\operatorname{SiLU}(K(z_i))\rangle\cdot V(z_i). \tag*{(9)}

To further mitigate the feature distribution shift caused by cross-modal injection, AFC applies a channel-wise affine calibration to the fused activations as follows:

AFC⁡(x)=γ⊙x+β,(10)\operatorname{AFC}(x)=\gamma\odot x+\beta, \tag*{(10)}

where xx denotes the joint retrieval of both linguistic and visual memories, γ\gamma and β\beta are learnable per-channel scale and bias parameters, respectively, and ⊙\odot denotes element-wise multiplication. In general, AFC calibrates post-fusion activations to regulate their statistics, providing a lightweight yet effective mechanism for stabilizing the semantic stream of pretrained LLMs.

In our implementation, we apply AFC twice along the vision-conditioned FFN pathway, aligned with the two projections of the FFN (down- and up-projection). We calibrate after the down-projection because it compresses information into a bottleneck representation, whose activation statistics are particularly susceptible to cross-modal semantic discrepancies. Early calibration stabilizes intermediate magnitudes and suppresses distribution drift before it is amplified by subsequent nonlinear transformations and residual accumulation. We then apply the second AFC at the FFN output to re-align the fused activations to a numerically stable regime compatible with the pretrained LLM residual stream. Hence, the vision-conditioned FFN with two-stage AFC is computed as:

{y′=AFC⁡(SiLU⁡(x)⋅SiLU⁡(W1T)),y=AFC⁡(y′⋅W2),(11)\begin{cases} y'= \operatorname{AFC}(\operatorname{SiLU}(x)\cdot\operatorname{SiLU}(W_1^{\mathsf{T}})),\\ y=\operatorname{AFC}(y'\cdot W_2), \tag*{(11)} \end{cases}

where W1W_1 and W2W_2 denote the down- and up-projection matrices in the FFN, respectively, and y′y' represents the intermediate activation retrieved by the FFN. For AFC, we initialize γ\gamma and β\beta with small perturbations so that AFC begins close to an identity mapping. This minimizes disruption to pretrained language behavior during early optimization, while enabling AFC to progressively learn modality-aware scaling and shifting for robust distribution alignment.

Experiments

Dataset and Experiment Setup

We conduct experiments primarily on ScienceQA [34], which contains 21k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. The dataset is split into training, validation, and test splits with 12726, 4241, and 4241 examples, respectively, and we report average accuracy on the test set. In addition, following prior work [30, 36], we further evaluate our method on multiple VQA benchmarks, including GQA [18], VQAv2 [11], OKVQA [37], and VizWiz [13]. We use LLaMA-7B and LLaMA-13B [47] as the language backbones, together with CLIP ViT-L/14 [40] as the vision encoder. Following the training setup of MemVP [23], we train the models for 20 epochs using the AdamW optimizer, with a batch size of 32 and a learning rate of 9e−39\mathrm{e}{-3}. The positional embedding length is set to 320 for LLaMA-7B and 400 for LLaMA-13B. For samples without image inputs, we set the visual prompts to zero tensors. The learnable scalar gate α\alpha is initialized to 0, and tanh⁡(⋅)\tanh(\cdot) limits the injection strength to (−1,1)(-1, 1), which stabilizes training and prevents visual modulation from overriding language priors. All experiments are conducted in a single-stage end-to-end setting, where answers are generated directly from the inputs. Experiments are implemented in PyTorch 2.0 and conducted on RTX A800 GPU.

Comparison Methods

We compare with two categories of methods: (1) MLLM methods. We compare our method with representative vision-language models spanning diverse architectures, including BLIP-2 [27], Flamingo [2], InstructBLIP [8], IDEFICS [26], UnifiedQA [33], LLaVA/LLaVA-1.5 [30, 31], InternVL [6], Qwen-VL/Qwen2-VL [48], GPT-3.5 [34], GPT-4 [1] and MM-CoT [52]. (2) PEFT methods. We further compare our method with parameter-efficient fine-tuning methods, including LLaMA-Adapter [9], LLaVA-LoRA [23], LaVIN [35], MemVP [23], ADEM-VL [14], and IPK [42].

Results Analysis

Results on ScienceQA. As shown in Table 1, under the same LLaMA-7B backbone, our method achieves state-of-the-art performance on ScienceQA, reaching an average accuracy of 93.59%. It

MethodLLMSubjectContext ModalityGradeAverage
NATSOCLANTXTIMGNOG1-6G7-12
MLLM methods
Human [34]-90.2384.9787.4889.6087.5088.1091.5982.4288.40
GPT-3.5 [34]GPT-3.574.6469.7476.0074.4467.2877.4276.8068.8973.97
GPT-3.5 w/ CoT [34]GPT-3.575.4470.8778.0974.6867.4379.9378.2369.6875.17
GPT-4 [1]GPT-484.0673.4587.3681.8770.7590.7384.6979.1082.69
UnifiedQA [33]T571.0076.0478.9166.4266.5381.8177.0668.8274.11
MM-CoTBase_{\text{Base}} [52]T587.5277.1785.8287.8882.9086.8384.6585.3784.91
LLaVA [30]Vicuna-7B--------89.84
LLaVA [30]Vicuna-13B90.3695.9588.0089.4988.0090.6690.9390.9090.92
PEFT methods
LLaMA-Adapter [51]LLaMA-7B84.3788.3084.3683.7280.3286.9085.8384.0585.19
LLaVA-LoRA [23]LLaMA-7B91.7094.6086.0991.2590.2888.6491.5289.6590.85
LaVIN [35]LLaMA-7B89.2594.9485.2488.5187.4688.0890.1688.0789.41
MemVP [23]LLaMA-7B94.4595.0588.6493.9992.3690.9493.1093.0193.07
ADEM-VL†^{\dagger} [14]LLaMA-7B94.4995.2887.6494.0993.0690.1093.1092.4992.88
IPK [42]LLaMA-7B--------92.30
OursLLaMA-7B94.8095.2889.7394.4893.1191.7893.7293.3493.59
LaVIN [35]LLaMA-13B90.3294.3887.7389.4487.6590.3191.1989.2690.50
MemVP [23]LLaMA-13B95.0795.1590.0094.4392.8692.4793.6194.0793.70
OursLLaMA-13B95.6595.8392.8295.6793.2192.2994.1794.1094.35

Table 1. Accuracy on ScienceQA test set. Question categories: NAT = natural science, SOC = social science, LAN = language science, TXT = w/ text context, IMG = w/ image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. Our results are shaded in gray. †^{\dagger} Results from our reproduction of the original work. Other results are quoted from their original papers.

outperforms strong PEFT baselines, including MemVP (93.07%), ADEM-VL (92.88%), LaVIN (89.41%), LLaVA-LoRA (90.85%), and LLaMA-Adapter (85.19%). When scaled to LLaMA-13B, our method further improves the average accuracy to 94.35%, again surpassing the existing baselines. Particularly, under the 7B setting, our method improves over MemVP by 0.35%, 0.23%, and 1.09% on NAT, SOC, and LAN, respectively, and also achieves gains of 0.49% and 0.75% on the TXT and IMG evaluations. Similar trends are observed under the 13B setting, where the improvements are particularly pronounced on LAN and TXT. This suggests that our method not only provides richer and earlier visual contextual support, but also effectively mitigates distributional perturbations introduced by cross-modal fusion, improving overall multimodal understanding while preserving language capabilities.

Results on Multiple VQA Benchmarks. Furthermore, to evaluate the generalization capability of the proposed method, we conduct experiments on multiple VQA benchmarks, as shown in Table 2. Our method achieves competitive performance on GQA, OKVQA, and VizWiz, achieving 66.3%, 52.5%, and 72.5%, respectively. In particular, the improvements on GQA and VizWiz indicate that our method is effective not only in modeling visual-semantic relationships in complex scenes, but also in handling low-quality and noisy real-world images more robustly. Overall, these results further demonstrate that our method generalizes well on ScienceQA and across diverse VQA tasks.

MethodGQAVQAv2OKVQAVizWizSQA
MLLM methods
BLIP-2 [27]41.041.045.919.674.2
InstructBLIP-8B [8]49.251.949.934.573.3
IDEFICS [26]38.450.938.435.577.4
LLaVA [31]49.563.645.230.756.5
LLaVA-v1.5 [30]62.074.045.650.090.9
InternVL-MLP [6]62.979.342.952.590.1
InternVL-QLLaMA [6]57.772.351.044.588.6
Qwen2-VL [48]63.779.848.565.479.1
PEFT methods
LLaMA [47]34.040.127.150.336.2
LLaMA-Adapter v2 [9]33.841.637.842.285.2
LaVIN [35]55.268.646.159.689.4
MemVP [23]58.173.249.970.093.1
IPK [42]66.275.651.171.492.3
Ours66.377.652.572.593.6

Table 2. Comparative performance evaluation on multiple VQA benchmarks. Best result is in bold face.

Ablation Studies

Ablation on Each Component. To validate the effectiveness of each component, we conduct ablation studies on OKVQA, VizWiz, and SQA. As shown in Table 3, introducing only MVM (# 0) achieves 52.1%, 70.8%, and 93.2% on OKVQA, VizWiz, and SQA, respectively, indicating that introducing multi-scale visual context at the input stage helps activate image-related knowledge. In contrast, using only AFC (# 1) provides relatively limited gains. Moreover, based on MVM, employing only unilateral calibration (# 2 or # 3) is inferior to bilateral calibration, suggesting that AFC plays complementary stabilizing roles at two critical stages of FFN-based visual retrieval. As a result, combining MVM with AFC (# 4) achieves the best results, further validating the effectiveness of the proposed approach.

#MVMAFC downAFC upOKVQAVizWizSQA
0✓--52.170.893.2
1-✓✓51.170.192.9
2✓✓✓51.571.993.3
3✓-✓51.371.392.8
4✓✓✓52.572.593.6

Table 3. Ablation study on each component. down and up represent two projections of FFN, respectively.

Ablation on MVM. We further ablate the scale settings of MVM. As shown in Table 4, using only fine-grained features (i.e., [256]) performs worse than combining fine-grained and coarser context (i.e., [256, 64]), indicating that the multi-scale priors introduced by MVM provide richer visual context and help the model leverage both coarse-grained semantics and fine-grained information. Adding an additional coarser scale (i.e., [256, 64, 16]) slightly degrades performance, suggesting that introducing too many scales may add redundancy.

MVMOKVQAVizWizSQA
[?]51.571.893.2
[?, ?]52.572.593.6
[?, ?, 16]51.872.093.1

Table 4. Ablation study on Multi-scale Visual Modulation (MVM) scale settings.

Ablation on LLM Backbone. To evaluate the adaptability of our method across different LLM backbones, we conduct comparative experiments using LLaMA-7B and Vicuna-7B. As reported in Table 5, our method achieves the best performance under both settings, attaining 93.59% on LLaMA-7B and 92.51% on Vicuna-7B, respectively. Compared with ADEM-VL and MemVP, our method improves performance on both backbones, demonstrating its robustness across diverse LLM backbones. It is also worth noting that the performance on Vicuna-7B is slightly lower than that on LLaMA-7B. A possible explanation is that Vicuna, which undergoes additional instruction fine-tuning based on LLaMA, is optimized for dialogue alignment and instruction following, and may therefore not fully preserve the original representation space that proves more suitable for visual question answering tasks.

MethodLLM backboneSQA
LLaMA-7BVicuna-7B
ADEM-VL✓-92.88
MemVP✓-93.07
Ours✓-93.59
ADEM-VL-✓91.75
MemVP-✓91.26
Ours-✓92.51

Table 5. Ablation study on different LLM backbones on SQA.

Case Study

Figure 3 provides four qualitative examples. (1) Spatial reasoning: when judging whether the red car is to the left or right of the backpack, LLaVA and MemVP are misled by contextual distractors, while our method correctly infers the relative position by establishing the global layout early. (2) Multi-object counting: in cluttered scenes, prior methods miss or double-count objects, whereas our method produces the correct count. (3) Occlusion and distractors: under heavy occlusion, our method localizes the key evidence and makes the correct prediction. (4) Visual degradation: in low-quality blurred grayscale images, our method still captures appropriate visual cues to correctly identify the vehicle as a “Bus.” In general, these cases suggest that our method can more reliably retrieve and utilize visual information under challenging conditions.

Figure 3. Case study comparing our method with other methods on challenging visual reasoning tasks.

Attention Visualization

Figure 4 visualizes the attention distribution on visual tokens in the final layer of the LLM when addressing VQA multimodal questions. As shown in the visualization, for counting queries (e.g., “How many giraffes are there?”), the highest attention concentrates on the local grid cells containing the giraffes, exhibiting a clear instance-centric focusing behavior. In contrast, for localization queries (e.g., “Where are the umbrellas?”), the attention forms a contiguous, band-like pattern aligned with the umbrellas’ contours, clearly delineating the target’s spatial extent even under cluttered backgrounds. Similarly, for the question “what street name is on the lower sign?”, the attention is highly concentrated on the lower sign region. For the question “Is this a commercial airplane?”, the attention mainly covers the airplane itself rather than the background. These observations collectively suggest that our method can improve vision-language alignment across diverse visual reasoning scenarios.

Visualization of attention on image patches

Figure 4. Visualization of attention on image patches.

Analysis of AFC

We visualize the distributions of per-channel activation standard deviation before and after AFC at the two FFN projections (down and up) across five VQA benchmarks (i.e., GQA, VQAv2, OKVQA, VizWiz, and SQA), as shown in Figure 5. The blue bars (“pre”) represent the distributions before applying AFC, while the orange bars (“post”) represent those after AFC. It can be observed that the calibration effect is most pronounced at AFC@down. Across all datasets, the post-AFC distribution shifts toward lower values and becomes more concentrated than the pre-AFC distribution. This shift indicates that the original distributions exhibit heavy tails with high-variance channels. AFC effectively suppresses these tails, demonstrating its ability to dampen abnormal responses induced by cross-modal injection and to reduce channel-wise variance imbalance, thereby limiting the propagation of distribution drift. In contrast, AFC@up shows substantial overlap between the pre- and post-AFC curves, with only minor adjustments. This suggests that the output-side calibration is conservative, primarily aimed at bringing the fused features back into a numerically stable range compatible with the pretrained residual stream, preventing small mismatches from accumulating across layers and eroding linguistic priors. Overall, these results indicate that AFC decouples visual injection from statistical alignment and stabilizes cross-modal feature fusion across diverse benchmarks.

Distributions of per-channel activation standard deviation before and after AFC across five VQA benchmarks

Figure 5. Visualization of the distributions of per-channel activation standard deviation before (pre) and after (post) AFC.

Training Time Comparison

Figure 6 illustrates the training time of various methods across different tasks, while Table 2 presents the corresponding accuracy

Figure 6. Average training time (s/iter) across tasks.

results. Our method is substantially more efficient than LLaMA, LLaMA-Adapter, and LaVIN, while achieving training time comparable to MemVP. This efficiency stems from avoiding redundant visual tokens, introducing early visual priors through lightweight modulation, and stabilizing feature fusion through calibration, thereby improving visual reasoning without noticeable overhead and making the approach more practical for deployment.

Conclusion

In this paper, we propose Beyond Token Stacking (BTS), a visual enhancement framework that tackles two limitations in current MLLMs: (i) redundant visual tokens and high computation from token concatenation, and (ii) FFN-based visual knowledge injection that lacks early visual context and could cause activation distribution drift. BTS comprises two key components. First, Multi-scale Visual Modulation (MVM) extracts multi-scale visual context from vision encoder outputs and injects it as a lightweight residual bias into text embeddings, enabling earlier visual grounding without introducing extra visual tokens. Second, Adaptive Fused-feature Calibration (AFC) calibrates fused activations via per-channel affine transformations, stabilizing FFN-based visual injection and improving fusion robustness. Extensive experiments across multiple multimodal benchmarks demonstrate that BTS substantially enhances both multimodal understanding and reasoning capabilities.

Acknowledgments

This work was supported by the National Key Research and Development Program under Grant 2024YFB4506200, the Science and Technology Innovation Program of Hunan Province under Grant 2024RC1048, the Key Research and Development Program of Hunan Province under Grant 2026QK3002.

References

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).arxiv.org/abs/2303.08774
  2. [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022), 23172–23736.
  3. [3]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision. 2425–2433.
  4. [4]Shoufa Chen, Chongjian Ge, Zhan Tong, Jingliu Wang, Yibing Song, Jue Wang, and Ping Luo. 2022. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems 35 (2022), 16664–16678.arxiv.org/abs/2205.13535
  5. [5]Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony J Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 (2022).arxiv.org/abs/2209.06794
  6. [6]Zhe Chen, Junnan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24185–24198.
  7. [7]Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 8493–8502.
  8. [8]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems 36 (2023), 49250–49267.
  9. [9]Peng Gao, Jiangming Han, Renrui Zhang, Ziyi Li, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 (2023).arxiv.org/abs/2304.15010
  10. [10]Mor Geva, Roi Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 5484–5495.arxiv.org/abs/2012.14913
  11. [11]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6904–6913.
  12. [12]Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. 2024. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision. Springer, 390–406.arxiv.org/abs/2403.11703
  13. [13]Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Pighin. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3608–3617.arxiv.org/abs/1802.08218
  14. [14]Zhiwei Hao, Jianyuang Guo, Li Shen, Yong Luo, Han Hu, and Yonggang Wen. 2025. ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning. International Journal of Computer Vision (2025), 1–17.arxiv.org/abs/2410.17779
  15. [15]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799.arxiv.org/abs/1902.00751
  16. [16]Zi-Yuan Hu, Yanyang Li, Michael R Lyu, and Liwei Wang. 2023. Vl-pet: Vision-and-language parameter-efficient tuning via granularity control. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3010–3020.arxiv.org/abs/2308.09804
  17. [17]Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. 2023. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems 36 (2023), 72096–72109.arxiv.org/abs/2302.14045
  18. [18]Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6700–6709.arxiv.org/abs/1902.09506
  19. [19]Aaron Hurst, Adam Lederer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2405.21290 (2024).arxiv.org/abs/2410.21276
  20. [20]Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. pmlr, 448–456.
  21. [21]Jingjing Jiang and Nanning Zheng. 2023. MixPHM: Redundancy-aware parameter-efficient tuning for low-resource visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24203–24213.arxiv.org/abs/2303.01239
  22. [22]Shibo Jie and Zhi-Hong Deng. 2023. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 1060–1068.
  23. [23]Shibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng, Kai Han, and Yunhe Wang. 2024. Memory-space visual prompting for efficient vision-language fine-tuning. arXiv preprint arXiv:2405.05615 (2024).arxiv.org/abs/2405.05615
  24. [24]Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. 2023. Revisiting the parameter efficiency of adapters from the perspective of precise redundancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 17217–17226.arxiv.org/abs/2307.16867
  25. [25]Lei Ke, Wenjie Pei, Ruyi Lu, Xiaoyong Shen, and Yu-Wing Tai. 2019. Reflective decoding network for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8888–8897.arxiv.org/abs/1908.11824
  26. [26]Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. 2023. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems 36 (2023), 71683–71702.arxiv.org/abs/2306.16527
  27. [27]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on Machine Learning. PMLR, 19730–19742.arxiv.org/abs/2301.12597
  28. [28]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 12888–12900.arxiv.org/abs/2201.12086
  29. [29]Junnan Li, Ramprasaath R Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems 34 (2021), 9694–9705.arxiv.org/abs/2107.07651
  30. [30]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306.arxiv.org/abs/2310.03744
  31. [31]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in Neural Information Processing Systems 36 (2023), 34892–34916.
  32. [32]Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems 32 (2019).
  33. [33]Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 (2022).arxiv.org/abs/2206.08916
  34. [34]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 (2022), 2507–2521.arxiv.org/abs/2209.09513
  35. [35]Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshaui Sun, and Rongrong Ji. 2023. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems 36 (2023), 29615–29627.arxiv.org/abs/2305.15023
  36. [36]Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. 2024. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003 (2024).arxiv.org/abs/2403.03003
  37. [37]Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3195–3204.arxiv.org/abs/1906.00067
  38. [38]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems 35 (2022), 17359–17372.arxiv.org/abs/2202.05262
  39. [39]Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: main volume. 487–503.arxiv.org/abs/2005.00247
  40. [40]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PmLR, 8748–8763.arxiv.org/abs/2103.00020
  41. [41]Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances in Neural Information Processing Systems 30 (2017).arxiv.org/abs/1705.08045
  42. [42]Shenzhen Song, Kangcheng Ding, Shan Zhao, Shasha Li, Xiaopeng Li, Chengyu Wang, Qian Wan, Bin Ji, and Jie Yu. 2026. Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 25554–25562.DOI
  43. [43]Shenzheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang, and Meng Wang. 2025. How to bridge the gap between modalities: Survey on multimodal large language model. IEEE Transactions on Knowledge and Data Engineering (2025).arxiv.org/abs/2311.07594
  44. [44]Shenzheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, and Meng Wang. 2024. A dual-way enhanced framework from text matching point of view for multimodal entity linking. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19008–19016.
  45. [45]Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. VI-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5227–5237.DOI
  46. [46]Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 5100–5111.arxiv.org/abs/1908.07490
  47. [47]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).arxiv.org/abs/2302.13971
  48. [48]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024).arxiv.org/abs/2409.12191
  49. [49]Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review 11, 12 (2024), nwae403.
  50. [50]Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 1–9.
  51. [51]Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 (2023).arxiv.org/abs/2303.16199
  52. [52]Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 (2023).arxiv.org/abs/2302.00923
  53. [53]Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jiafeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 13041–13049.arxiv.org/abs/1909.11059

Paper details

vlnc:W_hycmy85n
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents