betaFeedback

WorksComputer Science

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models

Abstract

Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.

Introduction

Visual perception is inherently iterative and task-directed: an initial observation provides a coarse understanding of a scene, while subsequent processing refines this understanding according to the task [34, 52]. In human vision, this process involves revisiting ambiguous regions, inspecting small objects, and actively seeking relevant cues [22, 73]. Drawing on this interplay between iterative refinement and task guidance, we ask a natural question for Vision-Language Models (VLMs): How can visual perception be precisely guided by the question and adaptively extended when needed?

Recent advances have explored additional computation in VLMs, expanding the reasoning budget through explicit reasoning traces [25, 59, 68, 69], recursive refinement [13, 23, 65], and latent recurrent computation [29, 36, 54]. These methods give models more opportunities to interpret multimodal context after the image has been encoded. Still, the visual evidence itself is acquired in a single, task-agnostic pass: the image is converted into visual tokens before the model can determine which regions, details, or relations are required by the question [14, 55]. This creates an evidence-acquisition bottleneck between perception and cognition: a VLM may reason longer, yet still lack task-critical visual evidence that was inadequately captured during the initial encoding, potentially leading to insufficient visual grounding or overreliance on linguistic priors [44, 62, 70, 74]. Such a bottleneck is particularly consequential for tasks requiring fine-grained perception or image-dependent reasoning, where additional reasoning cannot compensate for underprocessed visual information [12, 71].

A direct way to address this limitation is to improve visual representations and their interface to the language model. Existing methods have pursued this goal by increasing input resolution [39, 40, 47], aggregating features from multiple encoder layers [51, 72], or using the instruction to fuse task-relevant visual layers [32, 42, 71]. These advances show that VLM performance depends critically on how visual evidence is extracted, selected, and exposed. However, better selection and delivery of visual features do not, by themselves, enable the encoder to iteratively refine visual representations according to each question’s perceptual demands.

Adaptive recurrent computation provides exactly the mechanism needed to bring this missing flexibility into visual perception. Looped Transformers repeatedly apply a shared block to the hidden state, increasing effective computation depth without introducing separate parameters [16, 19]. Such recurrence has proven especially effective for language modeling [66, 78], with adaptive variants further adjust computation through learned halting [16] [20], budget-conditioned trajectories [28], or token-level routing [3, 75]. In vision and vision-language models, early explorations of looped architectures suggest that repeated computation can likewise refine visual representations and improve abstract or multimodal reasoning [9, 56, 60]. Together, these findings point to the potential of revisiting encoder layers to iteratively refine visual representations.

To realize this potential, we propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook reuses a middle-to-late encoder window to iteratively update the visual state with shared weights (Figure 1). At each loop, the instruction representation, current visual summary, and loop-step embedding jointly generate layer-wise affine modulation, adapting the shared transformations to the task and the evolving state. A learned exit gate evaluates the visual state after each update to determine whether another loop should be executed. Once execution stops, LoopLook constructs the final visual representation from the accumulated states rather than relying solely on the last update. To accommodate regional differences in how features evolve across loops, token-wise Mixture-of-Looks (MoL) learns a separate weighting over these states for each visual token. By combining LoRA with trainable modulation and keeping the language model frozen, LoopLook enables task-guided iterative refinement through lightweight adaptation.

Comparison of visual encoding paradigms

Figure 1. Comparison of visual encoding paradigms. (a) The base encoder processes inputs in a single pass. (b) Query-conditioned encoding incorporates the question into encoding but remains single-pass. (c) LoopLook enables task-guided refinement with adaptive computation depth.

We evaluate LoopLook on four pretrained VLM backbones across eight downstream benchmarks. LoopLook consistently improves average performance across all backbones, with gains of up to 8.6% on MMStar and 8.1% on HallusionBench over the base model. Across recurrent loops, LoopLook progressively refines visual attention toward task-relevant evidence, while its token-wise Mixture-of-Looks selectively integrates representations from different loops. Under reduced input resolution, LoopLook mitigates visual degradation by allocating greater recurrent depth and better preserving task-relevant information. Analysis of exit depths further shows that LoopLook assigns more loops to fine-grained and hallucination-prone tasks. This adaptive perception incurs only modest overhead, increasing time to first token (TTFT) by 5.1–6.5% while leaving autoregressive decoding essentially unchanged. Our contributions are summarized as follows:

  • Bottleneck Identification. We formulate the one-shot evidence-acquisition bottleneck in VLMs, where cognition can receive additional computation while visual evidence remains fixed after a single, task-agnostic encoder pass.

  • Recurrent Framework. We propose LoopLook, a recurrent perception framework that reuses a middle-to-late visual encoder window with task-conditioned modulation, adaptive exit, and Mixture-of-Looks.

  • Comprehensive Evaluation. We evaluate LoopLook on four VLM backbones and eight benchmarks, demonstrating consistent gains, robustness to reduced resolution, learned depth allocation, and modest runtime overhead.

Related Work

Vision Encoders in Multimodal Models. VLMs typically integrate pretrained visual encoders with language models through a modality-alignment interface that maps visual features into the language model’s representation space [5, 14, 47, 64], with CLIP and SigLIP exemplifying visual encoders pretrained to align image representations with natural language supervision [57, 76]. To understand the resulting representations, studies further examine cross-modal convergence [26], develop tools for measuring representation similarity [33], and analyze the structure and utility of intermediate visual features [7, 10, 55, 58]. However, extracting the visual evidence needed for grounded perception and reasoning remains challenging for VLMs [11, 55, 62]. Two complementary lines of work address this challenge. One incorporates language conditions into feature extraction, ranging from early feature-wise modulation [15, 53] to question-aware visual encoders [1, 18, 31, 48, 49]. The other improves how visual features are selected and passed to the language model through visual token packing, multi-layer aggregation, layer fusion, and routing [32, 37, 42, 51, 72].

Adaptive Computation in Recurrent Networks. ACT introduced learned halting for recurrent computation [20], while Universal Transformers combine weight-shared updates with adaptive processing depth [16]. In language models, recurrent-depth architectures explore latent reasoning and compute-efficient scaling through repeated updates [19, 66, 78], while other approaches adapt pretrained models for shared-depth execution [2, 50]. Beyond increasing effective depth, recent work studies how to control this computation through budget-conditioned execution [28], token-specific recursion depths [3], and learned stopping for test-time scaling [75]. In vision, recurrent processing has been explored for convolutional feature refinement [41, 45], image classification [35], abstract visual reasoning [60], and multi-view reconstruction [9]. Within recursive vision architectures, block-recurrent designs re-execute a shared transformer block across depths [27]. Building on this view, reViT programs depth-specific experts into the shared block [8]. Extending recurrence to multimodal modeling, LoopVL combines loops to iteratively update a unified vision–language state [56].

Positioning of Our Work. LoopLook connects these two research directions by making visual feature extraction both question-conditioned and recurrent. While most recurrent approaches train architectures from scratch, it requires only lightweight adaptation, preserving the pretrained visual representations while enabling iterative refinement.

Method

LoopLook refines visual representations through task-conditioned recurrence within a pretrained vision encoder before answer generation (Figure 2). Its design combines a shared recurrent window, loop-dependent feature modulation, and a Mixture-of-Looks module for adaptive stopping

Figure 2. Overview of LoopLook framework.

and token-wise state fusion. In this section, we first introduce the visual encoding formulation, then describe these components and the training objective.

Preliminary

Given an image I\mathbf{I} and an instruction x\mathbf{x}, a pretrained VLM first encodes the image into visual tokens and then generates the answer conditioned on them:

V=Pψ(Eθ(I)),Y∼Dϕ(Y∣x,V).(1)\mathbf{V}=P_{\psi}(E_{\theta}(\mathbf{I})), \qquad\mathbf{Y}\sim D_{\phi}(\mathbf{Y}\mid\mathbf{x},\mathbf{V}). \tag*{(1)}

Here EθE_{\theta} is the vision encoder, PψP_{\psi} is the projector, and DϕD_{\phi} is the language model. Once produced, the visual tokens V\mathbf{V} remain fixed throughout language generation. Visual perception is therefore effectively a one-shot, task-agnostic decision: the representation must preserve the evidence needed for the downstream task before the model has an opportunity to reason about what the image requires. This can be limiting when useful evidence is fine-grained or degraded.

To address this bottleneck, we allow the visual representation to undergo further encoder-side processing, enabling adaptive and purposeful computation through recurrent perception with shared parameters. LoopLook formulates this process as recursive visual encoding:

Hr+1=Fθs:e(Hr;x,r),r=0,…,Rmax⁡−1,(2)\mathbf{H}^{r+1}=F_{\theta}^{s:e}(\mathbf{H}^{r};x,r),\quad r=0,\ldots,R_{\max}-1, \tag*{(2)}

where Rmax⁡R_{\max} is the maximum number of loops, Hr={hir}i=1N\mathbf{H}^{r}=\{\mathbf{h}_{i}^{r}\}_{i=1}^{N} denotes the visual token states at the rr-th loop, and Fθs:eF_{\theta}^{s:e} is the encoder window reused across loops and conditioned on the instruction xx.

Recurrent Visual Encoding

LoopLook inserts recurrence inside the vision encoder. We decompose the encoder into an early stem Eθ0:sE_{\theta}^{0:s}, a recurrent window Fθs:eF_{\theta}^{s:e}, and a readout tail Eθe:LE_{\theta}^{e:L}:

Eθ=Eθe:L∘Fθs:e∘Eθ0:s.(3)E_{\theta}=E_{\theta}^{e:L}\circ F_{\theta}^{s:e}\circ E_{\theta}^{0:s}. \tag*{(3)}

The early stem extracts the initial visual state H0=Eθ0:s(I)\mathbf{H}^{0}=E_{\theta}^{0:s}(I). The same window Fθs:eF_{\theta}^{s:e} is then applied for up to Rmax⁡R_{\max} loops, producing a sequence of visual states H1,…,HRmax⁡\mathbf{H}^{1},\ldots,\mathbf{H}^{R_{\max}}. The readout tail and projector map each recurrent state back to the standard VLM interface: Vr=Pψ(Eθe:L(Hr))\mathbf{V}^{r}=P_{\psi}(E_{\theta}^{e:L}(\mathbf{H}^{r})).

We place the recurrent window in a middle-to-late region of the encoder, where features move toward semantic integration and become suitable for task-conditioned refinement. Following the block-recurrent view of ViTs [27], we treat a coherent CKA phase as a visual block: a set of adjacent layers with aligned transformations. Looping within one block gives the recurrent window a consistent operation to reuse. Figure 3 shows this block structure for two Qwen visual towers with complete details in Sec. A.

Figure 3. Block structure in pretrained visual towers. Layer-wise linear CKA forms coherent diagonal blocks, indicating contiguous encoder phases with aligned visual transformations. LoopLook places recurrence inside a middle-to-late block, reusing a consistent visual operation across loops.

Task-Conditioned Modulation

LoopLook further supplies each loop with a compact condition that specifies the task, the current visual state, and the loop step. We first obtain a task state q\mathbf{q} by mean-pooling the instruction input embeddings from the language model’s embedding layer and project the pooled vector to the visual hidden dimension. At loop rr, we concatenate this task state with the current visual summary and a learnable loop-step embedding er\mathbf{e}_{r}:

zr=Mz([q;Pool⁡(Hr);er]).(4)\mathbf{z}^{r}=M_{z}([\mathbf{q};\operatorname{Pool}(\mathbf{H}^{r});\mathbf{e}_{r}]). \tag*{(4)}

Here zr\mathbf{z}^{r} is the recurrent modulation state. The task state gives the visual encoder the instruction context, the pooled visual state summarizes what has been encoded so far, and er\mathbf{e}_{r} marks the current loop step.

For each block ℓ\ell in the recurrent window, zr\mathbf{z}^{r} is mapped to affine parameters and applied to normalized visual features:

γℓr,βℓr=Mℓ(zr),AdaMod⁡ℓr(Z)=(1+γℓr)⊙LN⁡(Z)+βℓr.(5)\gamma_{\ell}^{r},\beta_{\ell}^{r}=M_{\ell}(\mathbf{z}^{r}),\quad\operatorname{AdaMod}_{\ell}^{r}(Z)=(1+\gamma_{\ell}^{r})\odot\operatorname{LN}(Z)+\beta_{\ell}^{r}. \tag*{(5)}

The same recurrent window is therefore reused with a loop-dependent modulation state. As the loop unfolds, the visual state evolves, the modulation state is recomputed, and the next update is conditioned on both the instruction and the current visual evidence.

Mixture-of-Looks

Mixture-of-Looks (MoL) couples adaptive stopping with token-wise state fusion: an exit gate determines the execution depth, and a router combines the available states for each visual token. After each loop, the gate predicts a conditional stopping score:

sr=σ(hω(Pool⁡(Hr),q,er)),r=1,…,Rmax⁡.(6)s^{r}=\sigma\left(h_{\omega}(\operatorname{Pool}(\mathbf{H}^{r}),\mathbf{q},\mathbf{e}_{r})\right),\quad r=1,\ldots,R_{\max}. \tag*{(6)}

The score srs^{r} represents the probability of exiting at loop rr after reaching that loop. We set sRmax⁡=1s^{R_{\max}}=1 to terminate at the maximum depth. At inference, the encoder exits at the first loop with sr>τs^{r}>\tau, yielding an exit depth R^\hat{R}. The gate scores also define a stopping prior over loops:

πr=sr∏t<r(1−st),∑r=1Rmax⁡πr=1.(7)\pi_{r}=s^{r}\prod_{t<r}(1-s^{t}),\qquad\sum_{r=1}^{R_{\max}}\pi_{r}=1. \tag*{(7)}

Each prior mass πr\pi_{r} is computed from the gate scores observed through loop rr.

The token-wise router integrates all Rmax⁡R_{\max} recurrent states during training and only the executed prefix at inference. The number of available states is K=Rmax⁡K=R_{\max} during training and K=R^K=\hat{R} at inference. The router computes token-specific scores gi,r=gη(hir,q,pi,r)g_{i,r}=g_{\eta}(\mathbf{h}_{i}^{r},\mathbf{q},\mathbf{p}_{i},r), combines them with the stopping prior, and normalizes over these KK loops:

αi,r=softmax⁡r≤K(gi,r+log⁡(πr+ϵ)),h~i=∑r=1Kαi,rhir.(8)\begin{aligned} \alpha_{i,r}&=\operatorname{softmax}_{r\leq K}(g_{i,r}+\log(\pi_{r}+\epsilon)),\\ \tilde{\mathbf{h}}_{i}&=\sum_{r=1}^{K}\alpha_{i,r}\mathbf{h}_{i}^{r}. \tag*{(8)} \end{aligned}

Here pi\mathbf{p}_{i} is the spatial position of token ii, and ϵ>0\epsilon>0 stabilizes the logarithm. The stopping prior provides a sample-level preference over loops, while the token-specific scores allow each visual token to form its own mixture. The fused sequence H~={h~i}i=1N\tilde{\mathbf{H}}=\{\tilde{\mathbf{h}}_{i}\}_{i=1}^{N} passes through the readout tail and projector to produce the visual tokens for answer generation:

V=Pψ(Eθe:L(H~)).(9)\mathbf{V}=P_{\psi}\left(E_{\theta}^{e:L}(\tilde{\mathbf{H}})\right). \tag*{(9)}

Training Objective

During training, we unroll all Rmax⁡R_{\max} loops and retain differentiable fusion across their states, allowing the answer loss to jointly train recurrent refinement, routing, and halting. This separates full-unroll optimization from early-exit inference, following the design principle of differentiable adaptive computation [6, 17]. We optimize the autoregressive next-token prediction loss together with an expected-depth penalty. Given target response tokens Y={yt}t=1Ty\mathbf{Y} = \{y_t\}_{t=1}^{T_y}, where TyT_y is the response length, the objective is

L=−∑t=1Tylog⁡pϕ(yt∣y<t,x,V)+λdEπ[r],Eπ[r]=∑r=1Rmax⁡rπr.(10)\begin{aligned} \mathcal{L} &= -\sum_{t=1}^{T_y} \log p_{\phi}(y_t \mid y_{<t}, x, \mathbf{V}) + \lambda_d \mathbb{E}_{\pi}[r],\\ \mathbb{E}_{\pi}[r] &= \sum_{r=1}^{R_{\max}} r\pi_r. \tag*{(10)} \end{aligned}

The next-token loss jointly trains the recurrent visual states, the token-wise router, and the exit gate, while the depth term encourages a lower expected loop count under π\pi.

We gradually introduce the depth penalty using a linear ramp λd(u)=λd⋆⋅min⁡(1,max⁡(0,u−twtr))\lambda_d(u) = \lambda_d^{\star} \cdot\min\left(1, \max\left(0, \frac{u-t_w}{t_r}\right)\right), where uu is the training step, twt_w is the warmup length, trt_r is the ramp length, and λd⋆\lambda_d^{\star} is the target depth penalty. Both lengths are specified relative to the total number of training steps TtrainT_{\mathrm{train}}. This schedule first emphasizes learning useful visual updates and then introduces the cost of recurrent depth. Appendix E summarizes the training and inference workflows.

Experiments

Experimental Setup

Implementation Details. We implement experiments on four pretrained VLMs: Qwen2.5-VL-3B [5], Qwen3-VL-4B [4], InternVL3.5-4B [67], and LLaVA-1.5-7B [46], placing the recurrent window in the phase-identified middle-to-late visual blocks (layers 17–30, 10–22, 12–22, and 11–19, respectively). We train LoopLook on 100K Visual CoT [59] samples, using LoRA [24] with a rank of r=8r = 8 and a scaling factor of α=16\alpha= 16 for the vision encoder and fully training the newly introduced LoopLook modules. With the language model frozen, this adaptation trains only 14.9M, 10.6M, 10.1M, and 11.1M parameters respectively, corresponding to approximately 0.16%–0.40% of their total parameters. Each backbone completes training in roughly 1.5–5.5 hours on eight RTX 5090 GPUs. We set the maximum loop depth to Rmax⁡=4R_{\max} = 4 and use an exit threshold of 0.4 at inference time. The depth penalty uses a linear ramp with λd⋆=0.0005\lambda_d^{\star} = 0.0005, tw=0.6Ttraint_w = 0.6T_{\mathrm{train}}, and tr=0.4Ttraint_r = 0.4T_{\mathrm{train}}, where TtrainT_{\mathrm{train}} denotes the total number of training steps. Full details are provided in Appendix C.

Baselines and Benchmarks. We report performance across four backbones and reasoning-side prompting/decoding strategies, including Base direct answering, CoT prompting [69], and Self-Consistency with five samples [68]. We use LLaVA-1.5-7B as the primary setting to compare LoopLook with existing VLM visual-side enhancement methods, including Dense Connector [72] (multi-layer feature concatenation), TGIF [42] (instruction-guided layer fusion), and DeepStack [51] (anyres multi-crop encoding with intermediate-layer injection). We evaluate LoopLook on eight extensive downstream benchmarks covering three main abilities: general understanding (MMStar [11], VM-CBench [77], AI2D [30]), perception and reasoning (V* [71], BabyVision [12], VisualPuzzles [61]), and hallucination robustness (POPE [38], HallusionBench [21]). Full details are provided in Appendix F.

Main Results

Table 1 shows that LoopLook consistently improves average performance across all four backbones, with gains of up to 4.6%. On Qwen2.5-VL-3B, the improvements are particularly pronounced on MMStar and HallusionBench, rising by 8.6 and 8.1%, respectively, with a further 5.7% gain on V*. CoT and self-consistency often fail to improve fine-grained perception, suggesting that additional language-side computation alone is insufficient when visual evidence requires task-conditioned refinement. We further compare LoopLook with existing VLM enhancement methods using LLaVA-1.5-7B in Table 2. LoopLook achieves the highest average score of 48.5, improving the baseline by 3.3%, and delivers the strongest performance on five benchmarks: MMStar, VM-CBench, BabyVision, POPE, and HallusionBench. Notably, its gains reach 6.9% on MMStar and 6.2% on BabyVision, while HallusionBench improves by 4.1%. Overall, these results demonstrate the effectiveness of combining question-guidance and adaptive recurrence for refining visual evidence and improving perception across diverse VLM backbones.

BackboneMethodMMStarVMCBenchAI2DV*BabyVisionVisualPuzzlesPOPEHalluBenchAvg.
Qwen2.5 VL-3BBase55.974.981.572.814.931.288.854.359.3
Qwen2.5 VL-3BCoT56.2↑0.373.5↓1.482.8↑1.365.4↓7.413.9↓1.030.1↓1.189.0↑0.254.5↑0.258.2↓1.1
Qwen2.5 VL-3BSC@554.6↓1.376.3↑1.483.9↑2.467.0↓5.816.5↑1.631.8↑0.688.6↓0.258.7↑4.459.7↑0.4
Qwen2.5 VL-3BLoopLook64.5↑8.678.8↑3.984.3↑2.878.5↑5.718.6↑3.734.4↑3.289.5↑0.762.4↑8.163.9↑4.6
Qwen3 VL-4BBase69.276.782.674.917.834.190.160.863.3
Qwen3 VL-4BCoT70.6↑1.476.9↑0.282.0↓0.674.3↓0.613.4↓4.435.1↑1.086.9↓3.254.9↓5.961.8↓1.5
Qwen3 VL-4BSC@570.3↑1.178.4↑1.782.1↓0.576.4↑1.515.5↓2.333.9↓0.288.6↓1.560.4↓0.463.2↓0.1
Qwen3 VL-4BLoopLook73.7↑4.581.3↑4.684.8↑2.279.6↑4.714.4↓3.437.0↑2.990.8↑0.763.1↑2.365.6↑2.3
InternVL 3.5-4BBase65.078.282.678.014.930.388.959.262.1
InternVL 3.5-4BCoT66.9↑1.980.8↑2.684.9↑2.378.5↑0.517.5↑2.627.5↓2.887.4↓1.558.8↓0.462.8↑0.7
InternVL 3.5-4BSC@566.7↑1.779.2↑1.084.6↑2.077.0↓1.017.8↑2.927.3↓3.088.5↓0.457.3↓1.962.3↑0.2
InternVL 3.5-4BLoopLook69.6↑4.681.1↑2.984.6↑2.081.7↑3.719.6↑4.733.8↑3.590.2↑1.361.5↑2.365.3↑3.2
LLaVA 1.5-7BBase32.353.853.749.710.823.989.747.345.2
LLaVA 1.5-7BCoT30.4↓1.955.2↑1.454.0↑0.348.2↓1.512.9↑2.124.4↑0.590.1↑0.444.6↓2.745.0↓0.2
LLaVA 1.5-7BSC@530.8↓1.556.5↑2.751.7↓2.051.8↑2.114.4↑3.624.6↑0.788.3↓1.448.9↑1.645.9↑0.7
LLaVA 1.5-7BLoopLook39.2↑6.955.1↑1.355.8↑2.154.5↑4.817.0↑6.224.8↑0.990.2↑0.551.4↑4.148.5↑3.3

Table 1. Performance across backbones and traditional prompting/decoding strategies on eight benchmarks.

MethodAdaptive computationTask-guidedDepth recurrenceMMStarVMCBenchAI2DV*BabyVisionVisualPuzzlesPOPEHallusionBenchAvg.
Base✗✗✗32.353.853.749.710.823.989.747.345.2
Dense Connector✗✗✗27.9↓4.446.6↓7.250.6↓3.144.5↓5.213.7↑2.923.2↓0.777.2↓12.549.3↑2.041.6↓3.6
TGIF✗✓✗33.9↑1.652.8↓1.056.7↑3.055.0↑5.310.3↓0.525.0↑1.189.6↓0.146.2↓1.146.2↑1.0
DeepStack✗✗✗31.7↓0.649.4↓4.460.6↑6.949.2↓0.59.8↓1.025.3↑1.486.7↓3.045.3↓2.044.8↓0.4
LoopLook✓✓✓39.2↑6.955.1↑1.355.8↑2.154.5↑4.817.0↑6.224.8↑0.990.2↑0.551.4↑4.148.5↑3.3

Table 2. Performance comparison against existing VLM enhancement methods. The first three columns summarize whether each method supports adaptive computation, task guidance, and depth recurrence. Since all existing baselines provide LLaVA-based checkpoints, we use LLaVA-1.5-7B as the base model in this experiment.

Analysis

Adaptive Looping Behaviors

Progressive Refinement of Visual Attention. Figure 4 visualizes attention heatmaps of the base VLM and LoopLook on V* examples. Without question guidance, the original visual encoder favors general semantic features, resulting in diffuse attention that can overlook small, task-relevant details. By conditioning recurrent perception on the input question, LoopLook focuses attention on relevant visual regions. Across successive recurrent looks, attention is further deliberately refined and becomes increasingly concentrated on the queried evidence, enabling more precise localization for answer generation.

Figure 4. Attention heatmap visualization of progressive visual refinement. We visualize the penultimate encoder layer of Qwen2.5-VL3B at the end of each loop, providing a comparable view of the evolving visual state, and compare the base VLM with LoopLook on V∗V^{*} cases. The base encoder produces diffuse attention heatmaps and incorrect answers, while LoopLook progressively concentrates attention on task-relevant evidence across recurrent loops.

Low-Resolution Robustness. We next stress-test LoopLook by reducing the input resolution, which makes local details harder to preserve in a single encoder pass. Table 3 studies whether recurrent visual depth can mitigate the loss of visual evidence under this degradation. On Qwen2.5-VL-3B, LoopLook retains higher MMStar accuracy than the single-pass base at all tested resolutions, with the largest margin at the lowest resolution: at 50K, LoopLook retains 54.9 overall accuracy compared with 48.6 for Base, and the margin reaches 12.4% on fine-grained

ResolutionMethodCPFPIRLRMathS&TOverallLoops
50KBase61.237.261.248.450.033.648.61.0
50KLoopLook68.449.663.255.254.039.254.92.3
200KBase64.446.061.248.453.236.051.51.0
200KLoopLook68.850.464.456.453.238.455.21.8
450KBase68.845.664.454.854.439.654.61.0
450KLoopLook69.659.668.059.260.845.260.41.6

Table 3. Robustness to reduced resolution on MMStar. We report accuracy for six categories and overall, along with the average number of visual loops. Base denotes a single pass through the visual block.

CP: Coarse Perception; FP: Fine-grained Perception; IR: Instance Reasoning; LR: Logical Reasoning; S&T: Science & Technology.

perception. The average number of executed loops also decreases as resolution increases, from 2.3 at 50K to 1.8 at 200K and 1.6 at 450K. This behavior echoes human vision, in which insufficiently clear visual evidence can prompt further inspection, whereas sufficiently informative views require less additional scrutiny. Similarly, LoopLook allocates more computation when visual evidence is weak and exit earlier when the input provides enough detail.

Loop-Depth Allocation. Figure 5 reports the average number of executed loops across benchmarks. The learned exit policy assigns the largest visual depth to V* (2.62), HallusionBench (2.35), and BabyVision (2.18), where answers often depend on small details, visual traps, or weak localized evidence. POPE (1.28), AI2D (1.42), MMStar (1.51), and VMCBench (1.55) exit earlier on average, suggesting that many samples in these benchmarks can be resolved with limited additional visual processing. This pattern supports the intended behavior of LoopLook: recurrent computation is used selectively, with harder perception cases receiving more encoder-side depth.

Figure 5. Adaptive loop-depth allocation across benchmarks. LoopLook assigns more recurrent visual processing to fine-grained and hallucination-prone benchmarks, while easier or more direct recognition tasks exit earlier.

Token-wise Routing Patterns. Figure 6 visualizes the token-wise Mixture-of-Looks decision by assigning each image token to the loop state with the largest routing weight. The routing maps show that many background tokens are read from early recurrent states, while tokens near question-relevant regions are more frequently routed to later states. This behavior complements the sample-level exit policy: after deciding how many loops to execute for an image, LoopLook still selects visual depth separately for each token, allowing localized evidence to benefit from deeper recurrent representations without forcing the entire image representation to rely on the last loop.

Token-wise routing patterns in Mixture-of-Looks

Figure 6. Token-wise routing patterns in Mixture-of-Looks. For each image token, we visualize the loop state with the largest routing weight. Earlier loops cover broad background regions, while later-loop assignments appear around local evidence required by the question, such as the bag region and the helmet area.

Computational Efficiency

Runtime Overhead. We measure inference latency on a single NVIDIA RTX 5090 using 200 randomly sampled examples across downstream benchmarks. Before timing, we warm up each model with five forward passes and evaluate Base and LoopLook in an interleaved order to mitigate thermal drift or measurement bias. LoopLook allocates additional computation only to a window of the visual encoder, which contributes a limited fraction of the overall VLM computation while playing a critical role in visual perception, leaving autoregressive decoding unchanged. As shown in Table 4, this design increases TTFT by only 5.1%-6.5%. The decoding latency remains essentially unchanged, and the end-to-end latency is nearly identical because decoding dominates the total generation time. This makes encoder-side recurrence an efficient place to spend additional computation: it improves the quality of visual evidence before generation while adding only a small prefill-side cost.

ResolutionMethodTTFTOverheadDecodeE2E
300KBase67.2 ms–21.6 ms/tok1.425 s
300KLoopLook71.6 ms+6.5%21.7 ms/tok1.435 s
550KBase84.9 ms–21.8 ms/tok1.459 s
550KLoopLook89.2 ms+5.1%21.5 ms/tok1.447 s

Table 4. Runtime overhead evaluated on Qwen2.5-VL-3B. We report time to first token (TTFT), per-token decoding latency, and end-to-end (E2E) runtime for generating 64 tokens.

Accuracy–Compute Tradeoff. Figure 7 shows the accuracy–compute frontier induced by the exit threshold. A higher threshold makes early exit harder, increasing the average number of executed loops and gradually improving MMStar accuracy. The curve saturates after the default threshold τ=0.4\tau= 0.4: moving from τ=0.4\tau= 0.4 to larger thresholds adds more recurrent visual computation but yields only marginal accuracy gains. We therefore use 0.4 as the default setting, which preserves most of the performance benefit while avoiding unnecessary visual depth.

Figure 7. Accuracy–compute tradeoff under exit thresholds. Each point corresponds to one exit threshold on MMStar.

Ablation Studies

Contribution of Individual Components

Table 5 compares fine-tuning baselines, component ablations, and fixed-depth execution on Qwen2.5-VL-3B. We include parameter- and training-FLOPs-matched fine-tuning, remove

VariantMMStarV*HallusionBenchAvg.Loops
LoopLook (Full)64.578.562.468.51.89
Param.-matched57.2↓7.373.8↓4.753.6↓8.861.5↓7.01.00
FLOPs-matched58.6↓5.973.8↓4.758.7↓3.763.7↓4.81.00
w/o recurrence59.7↓4.874.9↓3.657.8↓4.664.1↓4.41.00
w/o task condition58.8↓5.773.8↓4.758.7↓3.763.8↓4.72.12
w/o token fusion62.2↓2.376.4↓2.160.0↓2.466.2↓2.31.86
Fixed-154.8↓9.774.9↓3.659.9↓2.563.2↓5.31.00
Fixed-263.2↓1.378.562.0↓0.467.9↓0.62.00
Fixed-363.8↓0.777.5↓1.062.7↑0.368.0↓0.53.00
Fixed-464.3↓0.281.2↑2.761.8↓0.669.1↑0.64.00

Table 5. Component ablations and fixed-depth comparisons. Avg. Loops denotes the sample-weighted average number of loops.

recurrence, task conditioning, or token-wise fusion, and compare adaptive execution with fixed budgets of one to four loops. Detailed protocols are given in Sec. D.

Param-Matched FT reaches an average accuracy of 61.5, while FLOPs-Matched FT reaches 63.7 after extending the baseline training schedule by 1.5× steps. LoopLook achieves 68.5, outperforming both fine-tuning baselines.

Removing recurrence reduces the average score by 4.4 points. Removing task conditioning lowers accuracy by 4.7 points and increases the average loop count from 1.89 to 2.12. Replacing token-wise Mixture-of-Looks with a shared cross-loop readout lowers the average score by 2.3 points. These results show the contributions of recurrent refinement, question guidance, and token-specific state fusion.

Using the same trained checkpoint, fixed-depth execution reaches average accuracies of 63.2, 67.9, and 68.0 at one, two, and three loops, respectively. Adaptive execution achieves 68.5 with an average of 1.89 loops, outperforming both fixed two- and three-loop settings with fewer loops. Fixed four-loop execution reaches 69.1, but at substantially greater computational depth, highlighting the efficiency of adaptive depth allocation.

Recurrent Window Placement

To investigate the effect of recurrent window placement, we sweep the recurrent window while keeping all other settings fixed on Qwen2.5-VL-3B. Table 6 shows that the middle-to-late CKA block achieves the best average performance, outperforming Early-Mid, Early, and Cross-block windows by 0.7, 1.3, and 5.3 points, respectively. This suggests that the default window provides a favorable balance: its features are semantically organized enough to respond to the question, yet still retain visual detail that can be refined through recurrent computation. Earlier windows are less effective because they mainly process local appearance and low-level patterns, so recurrence may repeat generic visual extraction and provide weaker task-critical refinement. The cross-block window performs worst, dropping to 63.2 on average. This supports our phase-aware design: recurrence is most effective when the shared window repeatedly applies a coherent visual transformation. A window that crosses strong CKA boundaries may mix layers with different functional roles, making the repeated update less stable and less aligned with the transformations learned during pretraining.

WindowLayersMMStarV*HalluBenchAvg.
Mid-Late (Default)17–3064.578.562.468.5
Cross-block14–2758.4↓6.174.3↓4.256.9↓5.563.2↓5.3
Early0–1362.0↓2.577.0↓1.562.5↑0.167.2↓1.3
Early-Mid3–1663.1↓1.477.5↓1.062.7↑0.367.8↓0.7

Table 6. Recurrent window ablations.

Conclusion

We introduced LoopLook, a task-conditioned recurrent visual encoding framework that allocates additional computation to perception. By combining a shared encoder window with question-guided modulation, adaptive stopping, and token-wise Mixture-of-Looks, LoopLook refines visual representations with only lightweight additional training. Experiments across backbones and benchmarks show consistent improvements in performance, while additional evaluations demonstrate adaptive depth allocation, robustness to reduced resolution, and modest latency overhead. Future work could explore finer-grained spatial control and tighter integration with ongoing reasoning to further enhance perception. We hope our findings highlight the value of adaptive visual computation for enhancing multimodal understanding.

Phase-Aware Recurrent Window Selection

Layer Similarity and Visual Blocks

We analyze the internal block structure of each vision tower with layer-wise centered linear CKA [33]. For a model with LL visual layers, we collect token representations from each layer on a small shared image set and compute an L×LL \times L similarity matrix. Bright diagonal regions indicate contiguous phases of layers with aligned representations. We refer to each coherent phase as a visual block, following the block-recurrent view that Transformer depth can often be described by a small number of reusable computational blocks [16, 27].

This analysis is used as a structural guide for recurrent window placement. A window inside one block applies a visually consistent transformation across loops. A window that spans several blocks mixes different stages of visual processing, such as local appearance extraction and semantic integration. The main paper therefore uses middle-to-late blocks as the default search region and validates the choice with window sweeps.

CKA Maps Across Backbones

Figure 8 shows the full CKA maps for the six visual towers analyzed in this work, covering Qwen2.5-VL [5], Qwen3-VL [4], InternVL3.5 [67], LoopVL [56], and LLaVA [46] backbones. Across these models, the matrices consistently exhibit a small number of bright diagonal blocks, suggesting that pretrained visual encoders organize depth into coherent computational phases. Table 7 summarizes the corresponding CKA blocks, whose locations vary by architecture and motivate phase-aware window selection for each backbone.

BackboneLayersCKA blocks
Qwen2.5-VL-3B320–14, 15–16, 17–30, 31
Qwen3-VL-4B240–9, 10–22, 23
InternVL3.5-4B240–11, 12–22, 23
InternVL3.5-8B240–11, 12–22, 23
LoopVL-1B280–18, 19–26, 27
LLaVA-1.5-7B240–10, 11–19, 20–23

Table 7. CKA blocks for the visual towers analyzed in this work. Layer ranges are zero-indexed and interpreted from the CKA maps; singleton ranges denote isolated readout-style layers. Bold layer ranges indicate the blocks selected in our experiments for recurrent processing in each backbone.

Figure 8. Layer-wise linear CKA maps for six visual towers. Across backbones, the encoder depth is organized into a small number of coherent visual blocks.

Window Sweep Protocol

For each backbone, we sweep recurrent windows around the middle and late encoder regions and compare their downstream performance under matched training and inference settings. The CKA maps serve as a guide for selecting candidate windows; the final claim is based on the window sweep. In practice, windows that remain inside a coherent middle-to-late block give more stable gains, while very early windows and windows that cross strong block boundaries are less reliable. This supports the use of block-consistent recurrent visual encoding in the main method.

More Case Studies

This section collects qualitative case studies. Section B.1 presents two open-ended BabyVision questions answered correctly by InternVL3.5-4B, where the free-form responses can be verified step by step. Section B.2 presents failure-to-success comparisons between the base VLM and LoopLook, visualizing how the recurrent visual features progressively refine attention onto the evidence region.

BabyVision Open-Ended Cases

Case A: Counting stacked cubes. An example is shown in Figure 9.

Stacked-cube structure

Figure 9. A correct LoopLook open-ended counting case. The model decomposes the stacked-cube structure into a base layer and upper layers, verifies that every upper cube is supported, and reaches the minimum count of 13.

Case B: Subtracting two groups of planets. Another example is shown in Fig. 10.

Attention Refinement Cases

Figure 11 shows representative cases in which the base VLM answers incorrectly while LoopLook succeeds. Each panel visualizes the attention distribution of the base VLM and of LoopLook at every recurrent step. In all cases, the base attention spreads over background regions, whereas the looped features progressively concentrate onto the evidence region. The number of loops actually used varies with the difficulty of localization (two loops for Figure 11a and Figure 11b, three loops for Figure 11c and Figure 11d).

Failure-to-success comparisons between the base VLM and LoopLook, showing four visual question-answering examples with base and looped visual features

Figure 11. Failure-to-success comparisons between the base VLM and LoopLook. Yellow boxes mark the queried evidence region.

Per-Token Loop Tier Visualization

Figure 12 further visualizes the token-wise Mixture-of-Looks decision by assigning each image token to the loop state with the largest routing weight.

Figure 12. Is the yellow trash can on the left or right side of the motorcycle? Loop 1 Loop 2 Loop 3. (a) Relative-position case. Tell me the number on the boat. Loop 1 Loop 2 Loop 3. (b) Fine-grained OCR case. Tell me the text on the white banner. Loop 1 Loop 2 Loop 3. (c) Text-reading case. Per-token dominant-loop tier maps on three additional cases. Blue/yellow/red denote tokens whose evidence is dominated by Loop 1/2/3 respectively.

Implementation Details

Backbones and Trainable Parameters

We implement LoopLook on four pretrained VLM backbones: Qwen2.5-VL-3B [5], Qwen3-VL-4B [4], InternVL3.5-4B [67], and LLaVA-1.5-7B [46]. For each backbone, the recurrent window is placed in the middle-to-late visual block selected by the phase analysis in Sec. A. During training, we fine-tune the pretrained vision encoder using LoRA [24] with a rank of r=8r = 8, a scaling factor of α=16\alpha= 16, and a dropout rate of 0.05. The resulting trainable budgets are 14.9M, 10.6M, 10.1M, and 11.1M parameters for Qwen2.5-VL-3B, Qwen3-VL-4B, InternVL3.5-4B, and LLaVA-1.5-7B respectively, all well below 0.4% of each backbone’s total parameters (see Table 8). All parameters of the encoder-side LoopLook modules are fully trainable, including the recurrent perception components, adaptive feature integration modules, and exit mechanisms. The language model is kept frozen throughout training. This configuration combines parameter-efficient adaptation of the pretrained vision encoder with full optimization of the newly introduced LoopLook modules. This setting keeps the original language generation interface unchanged and concentrates adaptation on visual perception.

ComponentQwen2.5-VL-3BQwen3-VL-4BInternVL3.5-4BLLaVA-1.5-7B
LoRA (vision tower, r=8r=8)4.373.153.153.15
Task projection (dl→dd_l \to d)2.622.622.624.20
MzM_z0.440.280.280.28
MℓM_\ell (all recurrent blocks)5.773.432.912.38
Loop-step embeddings er\mathbf{e}_r0.0010.0010.0010.001
Exit head0.870.560.560.56
Mixture-of-Looks router0.870.560.560.56
Total (% of backbone)14.9 (0.40%)10.6 (0.25%)10.1 (0.24%)11.1 (0.16%)

Table 8. Trainable-parameter composition of LoopLook (millions).

LoopLook Hyperparameters

For all backbones, we set the maximum loop depth to Rmax⁡=4R_{\max}=4 and use an exit threshold of 0.4 at inference time. The modulation state zr\mathbf{z}^{r} lies in a compact subspace of dimension d/8d/8, and the loop-step embedding er\mathbf{e}_{r} shares this dimension. For each recurrent block ℓ\ell, MℓM_{\ell} is a single linear map from this state to the 2d2d affine coefficients (γrℓ,βrℓ)(\gamma_{r}^{\ell},\beta_{r}^{\ell}). We use mean pooling over visual tokens to obtain the visual summary in both the modulation module and the exit gate. The token-wise Mixture-of-Looks router is implemented as a two-layer MLP with hidden size d/4d/4; the exit head uses

the same hidden width and outputs a scalar sigmoid score after each loop.

Training Setup

We train LoopLook with 100K Visual CoT [59] samples for one epoch using a global batch size of 8. We use AdamW with learning rate 3×10−53 \times10^{-5}, β=(0.9,0.95)\beta=(0.9,0.95), and weight decay 10−210^{-2}. The depth penalty uses a linear ramp with λd∗=0.0005\lambda_{d}^{*}=0.0005, tw=0.6Ttraint_{w}=0.6T_{\mathrm{train}}, and tr=0.4Ttraint_{r}=0.4T_{\mathrm{train}}, where TtrainT_{\mathrm{train}} denotes the total number of training steps. Experiments are trained in parallel on eight RTX 5090 GPUs. We apply global gradient clipping with norm 1.0.

Evaluation Settings

We evaluate all models with VLM-EvalKit1. For the Base and CoT baselines, we use greedy decoding with max_new_tokens=512. For Self-Consistency, we sample 5 CoT responses with temperature 0.5 and max_new_tokens=512, then predict by majority vote. All reported LoopLook results use the same benchmark Question

The image shows a three-dimensional shape made of stacked cubes. None of the cubes are suspended in the air. How many cubes are there at least?

Response (correct; ground truth: 13)

To determine the minimum number of cubes in the given three-dimensional shape, we need to carefully analyze the structure and ensure that we account for all visible and hidden cubes. The shape has a base layer with several cubes; on top of some of these base cubes, there are additional cubes. The base layer has a total of 7 cubes, and there are 6 cubes in the second and third layer. Therefore, the total number of cubes is 7 + 6 = 13. Thus, the minimum number of cubes in the shape is 13\boxed{13}.

prompts and answer extraction protocol as the corresponding backbone evaluations.

Efficiency Measurement

We evaluate the inference efficiency of Base and LoopLook on a single NVIDIA GeForce RTX 5090 GPU (32 GB). We randomly sample 200 examples from the downstream benchmarks and use the same examples and decoding configuration for both methods. For each model, we perform

five warm-up forward passes before timing, excluding warm-up runs from the measurements. We evaluate Base and LoopLook in an interleaved order to mitigate thermal drift and order-dependent measurement bias. CUDA operations are synchronized at timing boundaries to ensure accurate wall-clock measurements.

For each input resolution, we generate 64 tokens per example using identical decoding settings. We report three metrics: (1) time to first token (TTFT), measured from the start

A correct LoopLook open-ended subtraction case

caption: A correct LoopLook open-ended subtraction case. The model counts the two groups of cartoon planets separately and forms the correct subtraction equation.

of the timed inference request to the return of the first generated token; (2) decoding latency, measured as the average wall-clock time per generated token during autoregressive decoding; and (3) end-to-end (E2E) latency, measured from the start of the timed inference request to the completion of 64-token generation. We repeat the measurements and report the mean latency across runs.

All measurements are conducted using Python 3.13.5, PyTorch 2.14.0+cu130, CUDA 13.0, cuDNN 9.24, and Transformers 5.16.1, with NVIDIA driver version 580.95.05.

Ablation Protocol

Shared Training and Evaluation Settings

We evaluate controls on Qwen2.5-VL-3B targeting additional fine-tuning (Param-Matched FT and FLOPs-Matched FT), recurrent visual processing, task conditioning, token-wise readout, and adaptive stopping. The five training-based controls are trained independently from the same pretrained backbone initialization, while the fixed-depth w/o adaptive exit variants reuse the full LoopLook checkpoint and change only the inference policy. Task-conditioned variants derive the visual task state from the instruction. All configurations follow the training and evaluation protocols in Sec. C, including prompts, answer extraction, and benchmark scoring. We report MMStar, V*, and HallusionBench, with Avg. defined as their arithmetic mean.

Param- and FLOPs-Matched Fine-Tuning

Param-Matched FT. This control measures the gains from additional visual-side fine-tuning under a comparable trainable-parameter budget. We retain the original one-pass visual path,

V=Pψ(Eθ(I)),(11)V = P_{\psi}(E_{\theta}(I)), \tag*{(11)}

including its original feature selection and projector interface. No recurrent window execution, task-conditioned modulation, exit gate, or Mixture-of-Looks module is added. The baseline is trained on the same image–instruction–response examples using the autoregressive answer loss, without a depth penalty.

To match the full model’s trainable-parameter budget, the baseline uses visual LoRA with rank 28 and scaling 56, preserving α/r=2\alpha/r = 2. LoRA injection locations and dropout remain unchanged. The baseline has approximately 15.3M trainable parameters, compared with 14.9M for the full LoopLook model.

FLOPs-Matched FT. Param-matched training does not guarantee equal computational budgets: LoopLook executes its recurrent window Rmax⁡=4R_{\max} = 4 times per sample, while Param-Matched FT processes the vision tower only once.

Question: What three numbers can be seen on the second coin?

Base VLM Visual Features

Answer: The number on the second coins is 500.

Wrong Answer

LoopLook Visual Features

Loop 1 Loop 2

Answer: The number on the second coin is 200.

Right Answer

(a) Reading the numbers on a coin. The base attention scatters over the coin face and misreads the digits as “500”; the looped features sharpen onto the stamped region and recover all three digits (“200”).

Question: Is the dog on the left or right side of the crosswalk?

Base VLM Visual Features

Answer: The dog is on the left side of the crosswalk.

Wrong Answer

LoopLook Visual Features

Loop 1 Loop 2

Answer: The dog is on the right side of the crosswalk.

Right Answer

(b) Localizing a dog relative to a crosswalk. The base attention locks onto the storefront and answers “left”; the looped features re-anchor onto the dog beside the crossing and correct the answer to “right”.

Question: Tell me the number on the white board.

Base VLM Visual Features

Answer: The number on the white board is 1.

Wrong Answer

LoopLook Visual Features

Loop 1 Loop 2 Loop 3

Answer: The number on the white board is 3.

Right Answer

(c) Reading a number on a whiteboard deep inside a forest scene. The base attention diffuses over the tree canopy; the looped features refine onto the small whiteboard and read the digit correctly.

Question: Where is the seated person visible in the image?

Base VLM Visual Features

Answer: Behind the bench.

Wrong Answer

LoopLook Visual Features

Loop 1 Loop 2 Loop 3

Answer: Outside the field.

Right Answer

(d) Finding a seated person in a baseball scene. The base attention follows the pitcher in the foreground; the looped features expand toward the spectator area outside the field and localize the seated person correctly.

We estimate training FLOPs using 4NT4NT per linear layer with NN parameters and TT tokens, accounting for both forward and backward passes. Since the language model is frozen, backward computation only requires input gradients. We distinguish visual patches TpT_p from merged visual tokens Tm=Tp/4T_m = T_p/4 processed by the language model. For Qwen2.5-VL-3B, with NLLM=3.09BN_{\mathrm{LLM}} = 3.09\mathrm{B}, NViT=0.49BN_{\mathrm{ViT}} = 0.49\mathrm{B} across 32 layers, and NW=214MN_W = 214\mathrm{M} across the 14-layer window (blocks 17–30), we use Tp≈1531T_p \approx1531, Tm≈383T_m \approx383, and Tt≈250T_t \approx250 at 300K pixels:

Fbase=4NLLM(Tm+Tt)+4NViTTp≈10.8 TF,F_{\mathrm{base}} = 4N_{\mathrm{LLM}}(T_m+T_t) + 4N_{\mathrm{ViT}}T_p \approx10.8\ \mathrm{TF},
FLoopLook=Fbase+(Rmax⁡−1)4NWTp≈14.8 TF.(12)F_{\mathrm{LoopLook}} = F_{\mathrm{base}} + (R_{\max}-1)4N_{\mathrm{W}}T_p \approx14.8\ \mathrm{TF}. \tag*{(12)}

Only three window executions add extra FLOPs, as the first is included in the full tower pass. The trainable LoopLook modules (MzM_z, MℓM_\ell, exit head, and router; approximately 10.5M parameters) add less than 0.5% to the estimated budget. Under the same estimate, the LoopLook-to-baseline FLOPs ratio is ≈1.36×\approx1.36\times for Qwen2.5-VL-3B, ≈1.25×\approx1.25\times for Qwen3-VL-4B and InternVL3.5-4B, and ≈1.03×\approx1.03\times for LLaVA-1.5-7B, whose frozen language model dominates computation. To ensure a fair comparison, we train the same Param-Matched FT baseline for 1.5×1.5\times as many optimization steps under the native cosine schedule, giving it a slightly larger training compute budget than LoopLook’s estimated overhead requires across all evaluated backbones.

w/o Recurrence

We set Rmax⁡=1R_{\max}=1 throughout training, so the visual window executes only once. The task representation, initial visual state, modulation networks, and first-step embedding are retained:

H1=Fs:e(H0;x,0),H~=H1.(13)\mathbf{H}^{1}=F^{s:e}(\mathbf{H}^{0};x,0), \qquad\widetilde{\mathbf{H}}=\mathbf{H}^{1}. \tag*{(13)}

The update uses the same block-wise adaptive modulation as the full model, followed by the standard readout tail and projector. Visual LoRA and task-conditioned modulation remain trainable. This control compares a single task-conditioned visual update with recurrent refinement.

w/o Task Conditioning

We replace the visual-side task vector with a fixed zero vector, qvisual=0\mathbf{q}_{\mathrm{visual}}=0, while preserving the original interfaces and dimensions. This replacement is applied to the modulation network MzM_z, exit head hωh_\omega, and readout router gηg_\eta. The language model still receives the original instruction, and the target response and answer supervision remain unchanged.

Visual-state conditioning, loop-step embeddings, the modulation architecture, adaptive exit, token-wise MoL, and the original training objective are retained. The model can adapt its visual computation to the image without question guidance. Comparing this variant with the full model measures the contribution of task conditioning across the three visual-side branches.

w/o Token-Wise Fusion

The w/o token-wise fusion variant replaces token-specific fusion weights with a shared set of weights for each image. We retain the original scoring network and aggregate its token-wise logits:

ℓi,r=gη(hir,q,pi,r),\ell_{i,r}=g_\eta(\mathbf{h}_i^r,\mathbf{q},\mathbf{p}_{i,r}),
ℓˉr=1N∑i=1Nℓi,r.(14)\bar{\ell}_r=\frac{1}{N}\sum_{i=1}^{N}\ell_{i,r}. \tag*{(14)}

The averaged logits and stopping prior define a shared distribution over the available loops. With K=Rmax⁡K=R_{\max} during training and K=R^K=\hat{R} at inference, its weights are

βr=softmax⁡r≤K(ℓˉr+log⁡(πr+ϵ)),(15)\beta_r=\operatorname{softmax}_{r\le K}\left(\bar{\ell}_r+\log(\pi_r+\epsilon)\right), \tag*{(15)}

and the same weights are applied to every token:

h~i=∑r=1Kβrhir.(16)\widetilde{\mathbf{h}}_i=\sum_{r=1}^{K}\beta_r\mathbf{h}_i^r. \tag*{(16)}

The loop distribution varies across images but is shared across tokens within each image. The scoring network and parameter count are unchanged.

Fixed-Depth Execution (w/o Adaptive Exit)

We use the full LoopLook checkpoint trained with Rmax⁡=4R_{\max}=4 and set a fixed inference depth K∈{1,2,3,4}K\in\{1,2,3,4\}. For each setting, every sample executes exactly KK loops, and Mixture-of-Looks combines the resulting states. The exit head supplies the stopping prior with the original Rmax⁡=4R_{\max}=4 setting:

αi,r=softmax⁡r=1,…,K(gi,r+log⁡(πr+ϵ)),\alpha_{i,r}=\operatorname{softmax}_{r=1,\ldots,K}\left(g_{i,r}+\log(\pi_r+\epsilon)\right),
h~i=∑r=1Kαi,rhir.(17)\widetilde{\mathbf{h}}_i=\sum_{r=1}^{K}\alpha_{i,r}\mathbf{h}_i^r. \tag*{(17)}

The adaptive variant uses the same checkpoint and readout, with K=R^K=\hat{R} selected by the exit gate.

LoopLook Training and Inference Algorithm

Algorithm 1 summarizes task-conditioned recurrence, adaptive stopping, and Mixture-of-Looks readout. Training unrolls all recurrent steps, while inference computes the prefix selected by the exit gate. In both modes, the router fuses the available states using token-specific scores and the stopping prior, then passes the fused representation through the tail and projector. Training applies the answer loss and depth penalty to optimize the visual-side parameters.

Baselines and Benchmark Details

Baseline Methods

We compare LoopLook with three visual-side enhancement methods: Dense Connector [72], TGIF [42], and DeepStack [51]. All three provide public checkpoints built on the same

Algorithm 1: Training and inference workflow of LoopLook

Input: Image I\mathbf{I}, instruction x\mathbf{x}, mode m∈{train,infer}m \in\{\mathrm{train},\mathrm{infer}\}; encoder stem Eθ0:SE_{\theta}^{0:S}, shared window FθS:ℓF_{\theta}^{S:\ell}, tail Eθℓ:LE_{\theta}^{\ell:L}, projector PψP_{\psi}, language model DϕD_{\phi}, and LoopLook modules; maximum depth Rmax⁡R_{\max}, exit threshold τ\tau, and ϵ>0\epsilon> 0. Training additionally receives target response Y\mathbf{Y} and optimization step uu.

Output: Training loss L\mathcal{L}, or visual tokens V\mathbf{V} and inference exit depth R^\hat{R}.

/* Shared initialization */

1 q←TaskEncode⁡(x),H0←Eθ0:S(I),K←Rmax⁡\mathbf{q} \leftarrow\operatorname{TaskEncode}(\mathbf{x}),\quad\mathbf{H}^{0} \leftarrow E_{\theta}^{0:S}(\mathbf{I}),\quad K \leftarrow R_{\max}

/* Task-conditioned recurrent perception */

2 for r=0,…,Rmax⁡−1r = 0,\ldots,R_{\max}-1 do

3 zr←Mz([q;Pool⁡(Hr);er])\mathbf{z}^{r} \leftarrow M_z([\mathbf{q};\operatorname{Pool}(\mathbf{H}^{r});\mathbf{e}_r])

4 (γℓr,βℓr)←Mℓ(zr)(\gamma_{\ell}^{r},\beta_{\ell}^{r}) \leftarrow M_{\ell}(\mathbf{z}^{r}) for each block ℓ\ell in the recurrent window

5 AdaMod⁡ℓr(Z):=(1+γℓr)⊙LN⁡(Z)+βℓr\operatorname{AdaMod}_{\ell}^{r}(\mathbf{Z}) := (1+\gamma_{\ell}^{r})\odot\operatorname{LN}(\mathbf{Z})+\beta_{\ell}^{r} for each block ℓ\ell

6 Hr+1←FθS:ℓ(Hr;{AdaMod⁡ℓr}ℓ)\mathbf{H}^{r+1} \leftarrow F_{\theta}^{S:\ell}(\mathbf{H}^{r};\{\operatorname{AdaMod}_{\ell}^{r}\}_{\ell})

7 sr+1←σ(hω(Pool⁡(Hr+1),q,er+1))s^{r+1} \leftarrow\sigma(h_{\omega}(\operatorname{Pool}(\mathbf{H}^{r+1}),\mathbf{q},\mathbf{e}_{r+1}))

8 if r+1=Rmax⁡r+1=R_{\max} then

9 ⌞sr+1←1\llcorner s^{r+1} \leftarrow1

10 if m=inferm=\mathrm{infer} and (sr+1>τ or r+1=Rmax⁡)(s^{r+1}>\tau\ \textbf{or}\ r+1=R_{\max}) then

11 K←r+1\quad K \leftarrow r+1

12 \quad break

/* Shared prior-aware Mixture-of-Looks readout */

13 if m=inferm=\mathrm{infer} then

14 ⌞R^←K\llcorner\hat{R} \leftarrow K

15 πr←sr∏j<r(1−sj)\pi_r \leftarrow s^r\prod_{j<r}(1-s^j) for r=1,…,Kr=1,\ldots,K

16 for i=1,…,Ni=1,\ldots,N do

17 ⌞αi,r←softmax⁡r=1,…,K(gη(hir,q,pi,r)+log⁡(πr+ϵ))\llcorner\alpha_{i,r} \leftarrow\operatorname{softmax}_{r=1,\ldots,K}(g_{\eta}(\mathbf{h}_i^r,\mathbf{q},\mathbf{p}_i,r)+\log(\pi_r+\epsilon))

18 h~i←∑r=1Kαi,rhir\tilde{\mathbf{h}}_i \leftarrow\sum_{r=1}^{K}\alpha_{i,r}\mathbf{h}_i^r for i=1,…,Ni=1,\ldots,N, H~←{h~i}i=1N\tilde{\mathbf{H}}\leftarrow\{\tilde{\mathbf{h}}_i\}_{i=1}^{N}

19 V←Pψ(Eθℓ:L(H~))\mathbf{V}\leftarrow P_{\psi}(E_{\theta}^{\ell:L}(\tilde{\mathbf{H}}))

/* Training objective or standard language generation */

20 if m=trainm=\mathrm{train} then

21 λd(u)←λd⋆min⁡(1,max⁡(0,u−twtr))\lambda_d(u)\leftarrow\lambda_d^{\star}\min\left(1,\max\left(0,\frac{u-t_w}{t_r}\right)\right)

22 LNTP←−∑t=1Tylog⁡pϕ(yt∣y<t,x,V)\mathcal{L}_{\mathrm{NTP}}\leftarrow-\sum_{t=1}^{T_y}\log p_{\phi}(y_t\mid y_{<t},\mathbf{x},\mathbf{V})

23 L←LNTP+λd(u)∑r=1Rmax⁡rπr\mathcal{L}\leftarrow\mathcal{L}_{\mathrm{NTP}}+\lambda_d(u)\sum_{r=1}^{R_{\max}}r\pi_r

24 return L\mathcal{L}

25 else

26 ⌞\llcorner return (V,R^)(\mathbf{V},\hat{R}) for generation with Dϕ(⋅∣x,V)D_{\phi}(\cdot\mid\mathbf{x},\mathbf{V})

BenchmarkGroup# SamplesFormat
MMStarGeneral1,500MC (letter)
VMCBench (DEV)General1,000MC (letter)
AI2DGeneral3,088MC (letter)
V*Perception and Reasoning191MC (letter)
BabyVisionPerception and Reasoning388MC (letter) / Open-ended
VisualPuzzlesPerception and Reasoning1,168MC (letter)
POPEHallucination9,000Yes/no
HallusionBenchHallucination951Yes/no

Table 9. Benchmarks used in our evaluation.

Vicuna-7B-v1.5 [63] language backbone, enabling comparison with existing approaches to multi-layer visual feature integration, instruction-guided layer fusion, and intermediate visual feature injection. Their visual pathways differ: Dense Connector adopts a SigLIP-SO400M tower, and DeepStack uses the LLaVA-NeXT [47] recipe with AnyRes encoding.

Dense Connector. We use the released DenseConnector-v1.5-7B checkpoint. The original LLaVA-1.5-7B uses CLIP ViT-L/14-336 [57]; this variant employs SigLIP-SO400M [76] at 3842384^{2} resolution. It aggregates features from three groups of encoder depths and concatenates them channel-wise before the MLP projector. The resulting representation has 3×1152=34563 \times1152 = 3456 dimensions per patch, producing 729 visual tokens per image. The fusion is independent of the input instruction, and the visual token count is fixed.

TGIF. TGIF (Text-Guided Layer Fusion) uses the input instruction to determine the contribution of each vision encoder layer. A two-layer MLP maps the mean-pooled prompt embeddings to 24 layer weights, which are normalized by softmax. The weighted layer features are combined with the penultimate-layer features and passed through the standard MLP projector. TGIF retains the CLIP ViT-L/14-336 backbone and produces 576 visual tokens, without changing the encoder depth or token count at inference time.

DeepStack. We use the released high-resolution checkpoint deepstack-l-hd-vicuna-7b. DeepStack employs AnyRes-style multi-crop encoding and injects additional visual features into the early layers of the language model through residual connections. Its visual sequence includes a global image view and local crops, and auxiliary features provide additional visual detail at multiple scales. DeepStack therefore also changes how visual information enters the language model. Its visual token count varies with input resolution.

Benchmark Details

We evaluate all methods on eight benchmarks covering general visual understanding, perception and reasoning, and hallucination robustness. We use the same prompts and answer extraction rules across methods. We follow the prescribed answer format for each benchmark. Multiple-choice questions are evaluated by option accuracy, while open-ended questions are evaluated against their reference answers. POPE and HallusionBench use binary yes/no answers.

General understanding. MMStar [11] comprises 1,500 multiple-choice questions across diverse visual understanding categories, including visual reasoning, mathematics, and knowledge-intensive tasks. Its construction filters out questions that can be answered from text alone, making it useful for assessing whether models can effectively ground their answers in visual evidence. VMCBench [77] consolidates existing visual question answering datasets into a unified multiple-choice format, covering diverse sources such as charts, documents, OCR, and natural images. We use its DEV split of 1,000 samples to evaluate performance across heterogeneous visual question answering tasks. AI2D [30] focuses on grade-school science diagrams, where answering questions requires interpreting diagram structure, textual annotations, and spatial relations indicated by visual elements such as arrows. It complements natural-image benchmarks by testing the understanding of structured visual representations.

Perception and Reasoning. V* [71] evaluates fine-grained visual perception through questions about local attributes, relative positions, text within images, and challenging examples drawn from the GPT-4V evaluation set. These questions require models to identify specific visual evidence beyond global scene understanding. BabyVision [12] targets basic visual abilities through questions designed to minimize dependence on external knowledge, covering elementary forms of visual recognition and reasoning. Its tasks help assess whether models can reliably interpret simple visual evidence, even when they perform well on broader vision-language benchmarks. VisualPuzzles [61] consists of non-verbal visual puzzles that emphasize pattern induction, spatial composition, and counting. By reducing reliance on linguistic context and specialized knowledge, it probes the ability to infer visual rules and reason over image structure.

Hallucination robustness. POPE [38] evaluates object hallucination by asking binary questions about whether specified objects are present in COCO [43] images. Its random, popular, and adversarial sampling strategies vary the difficulty of negative examples, testing whether a model distinguishes actual visual evidence from object co-occurrence priors. HallusionBench [21] examines whether models are misled by visual illusions, misleading image-text associations, and conflicts between visual evidence and language priors. Its questions cover cases in which intuitive or language-driven answers can contradict what the image actually depicts.

Benchmark Prompts

All eight benchmarks follow the prompt templates released in VLMEvalKit, and we adopt them verbatim in every evaluated setting. Here we list each template: {question} and {options} denote the per-sample question and choice list, and the image is prepended to the text prompt. Yes/no benchmarks (POPE, HallusionBench) use the annotation question as released by VLMEvalKit without any additional suffix.

MMStar / AI2D / V∗V^{*} (multiple-choice template)

Template

Question: {question} Options: A. {option_A} B. {option_B} ... Please select the correct answer from the options above.

VMCBench (multiple-choice template)

Template

Question: {question} Options: A. {option_A} B. {option_B} ... Answer with the option’s letter from the given choices directly.

VisualPuzzles

Template

{question} Options: (A) {option_A} (B) {option_B} (C) {option_C} (D) {option_D} Solve the multiple-choice question and then answer with the option letter from the given choices. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of the options. Think step by step before answering.

POPE / HallusionBench (yes/no)

Template

{annotation question} e.g., POPE: Is there a {object} in the image?

BabyVision

Template

{question} (open-ended items) {question} Choices: (A) {option_A} (B) {option_B} ... Think about the question and give your final answer in \boxed{Answer} format.

References

  1. [1]Ofir Abramovich, Niv Nayman, Sharon Fogel, Inbal Lavi, Ron Litman, Shahar Tsiper, Royee Tichauer, Srikar Appalaraju, Shai Mazor, and R Mannatha. Visfocus: Prompt-guided vision encoders for ocf-free dense document understanding. In European Conference on Computer Vision, pages 241–259. Springer, 2024. 2arxiv.org/abs/2407.12594
  2. [2]Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora. In International Conference on Learning Representations, pages 34282–34327, 2025. 3arxiv.org/abs/2410.20672
  3. [3]Sangmin Bae, Yujin Kim, Reza Bayat, Sungyun Kim, Ji-youn Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. Advances in Neural Information Processing Systems, 38:96572–96617, 2026. 2, 3
  4. [4]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjian Ge, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2511.21631, 2025. 5, 13
  5. [5]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao­hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. 2, 5, 13arxiv.org/abs/2502.13923
  6. [6]Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021. 4arxiv.org/abs/2107.05407
  7. [7]Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Bangalath, et al. Perception encoder: The best visual embeddings are not at the output of the network. Advances in Neural Information Processing Systems, 38:60884–60937, 2026. 2
  8. [8]Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos. One block, multiple depths: Recurrent vision transformers with depth-programmed experts. arXiv preprint arXiv:2610.12448, 2026. 3arxiv.org/abs/2610.12448
  9. [9]Alessandro Burzio, Tobias Fischer, Sven Elflein, Qunjie Zhou, Riccardo De Lutio, Jiawei Ren, Jiahui Huang, Shengyu Huang, Marc Pollefeys, Laura Leal-Taixé, Zan Gojcic, and Haithem Turki. Déjà view: Looping transformers for multi-view 3d reconstruction, 2026. 2, 3
  10. [10]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF international conference on computer vision (ICCV), pages 9630–9640. IEEE, 2021. 2arxiv.org/abs/2104.14294
  11. [11]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024. 2, 5, 21
  12. [12]Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language. arXiv preprint arXiv:2601.06521, 2026. 2, 5, 21arxiv.org/abs/2601.06521
  13. [13]Kanzhi Cheng, Li YanTao, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision-language models can self-improve reasoning via reflection. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8876–8892, 2025. 1arxiv.org/abs/2411.00855
  14. [14]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250–49267, 2023. 1, 2arxiv.org/abs/2305.06500
  15. [15]Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. Advances in neural information processing systems, 30, 2017. 2arxiv.org/abs/1707.00683
  16. [16]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. 2, 13arxiv.org/abs/1807.03819
  17. [17]Cristobal Eyzaguirre and Alvaro Soto. Differentiable adaptive computation time for visual reasoning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12814–12822. IEEE, 2020. 4
  18. [18]Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, and Ron Litman. Question aware vision transformer for multimodal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13861–13871, 2024. 2
  19. [19]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems, 38:41340–41391, 2026. 2, 3 Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. 2
  20. [21]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoub, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14375–14385. IEEE, 2024. 5, 21
  21. [22]Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in cognitive sciences, 9(4):188–194, 2005. 1pubmed.ncbi.nlm.nih.gov/15808501
  22. [23]Jiayi He, Hehai Lin, Qingyun Wang, Yi R Fung, and Heng Ji. Self-correction is more than refinement: A learning framework for visual and language reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6405–6421, 2025. 1arxiv.org/abs/2410.04055
  23. [24]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5, 13
  24. [25]Lianyu Hu, Ma Xiaoyu, Zeqin Liao, and Yang Liu. TVI-CoT: Text-visual interleaved chain-of-thought reasoning for multimodal understanding. In Proceedings of the 43rd International Conference on Machine Learning, pages 45820–45831. PMLR, 2026. 1
  25. [26]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. 2arxiv.org/abs/2405.07987
  26. [27]Mozes Jacobs, Thomas Fel, Richard Hakim, Alessandra Bron-detta, Demba Ba, and T Anderson Keller. Block recurrent dynamics in vision transformers. In International Conference on Learning Representations, pages 51617–51642, 2026. 3, 4, 13
  27. [28]Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. Loopformer: Elastic-depth looped transformers for latent reasoning via shortcut modulation. arXiv preprint arXiv:2602.11451, 2026. 2, 3arxiv.org/abs/2602.11451
  28. [29]Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, and Jinwoo Shin. Vision-aligned latent reasoning for multi-modal large language model. arXiv preprint arXiv:2602.04476, 2026. 1arxiv.org/abs/2602.04476
  29. [30]Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision – ECCV 2016, pages 235–251, Cham, 2016. Springer International Publishing. 5, 21arxiv.org/abs/1603.07396
  30. [31]Jongha Kim, Minseong Bae, Sanghyeok Lee, Jinsung Yoon, and Hyunwoo J Kim. Tabflash: Efficient table understanding with progressive question conditioning and token focusing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22573–22581, 2026. 2
  31. [32]Jeonghwan Kim, Sofia Stoica, Jiwan Chung, Ansel Blume, Hyeonjoeng Ha, Zhenhailong Wang, Xin Luna Dong, and Heng Ji. Mixture of layers: Dynamic layer routing for visual reasoning, 2026. 2arxiv.org/abs/2610.09440
  32. [33]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMLR, 2019. 2, 13
  33. [34]Victor A.F. Lamme and Pieter R. Roelfsema. The distinct modes of vision offered by feedforward and recurrent processing. Trends in Neurosciences, 23(11):571–579, 2000. 1pubmed.ncbi.nlm.nih.gov/11074267
  34. [35]Sang In Lee and Jihun Park. Soft mixture-of-recursions: Going deeper with recursive vision transformers. arXiv preprint arXiv:2607.00774, 2026. 3arxiv.org/abs/2607.00774
  35. [36]Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Emad Barsoum, Muhao Chen, and Zicheng Liu. Latent visual reasoning. In International Conference on Learning Representations, pages 148076–148090, 2026. 1arxiv.org/abs/2509.24251
  36. [37]Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. International Journal of Computer Vision, 133(10):6794–6812, 2025. 2arxiv.org/abs/2407.02392
  37. [38]Yifan Li, Yifan Du, Kun Zhou, Jinping Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292–305, 2023. 5, 21arxiv.org/abs/2305.10355
  38. [39]Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 2DOI
  39. [40]Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26753–26763. IEEE, 2024. 2arxiv.org/abs/2311.06607
  40. [41]Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3367–3375, 2015. 3DOI
  41. [42]Chenchen Lin, Sanbao Su, Rachel Luo, Yuxiao Chen, Yan Wang, Marco Pavone, and Fei Miao. Text-guided layer fusion mitigates hallucination in multimodal llms. arXiv preprint arXiv:2601.03100, 2026. 2, 5, 19arxiv.org/abs/2601.03100
  42. [43]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 21arxiv.org/abs/1405.0312
  43. [44]Zhiqi Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In Proceedings of the 41st International Conference on Machine Learning, pages 29914–29934. PMLR, 2024. 1arxiv.org/abs/2306.01879
  44. [45]Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in neural information processing systems, 31, 2018. 3
  45. [46]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296. IEEE, 2024. 5, 13
  46. [47]Haotian Liu, Chunyuang Li, Yuheng Li, Bo Li, Yanzhuang Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 2, 21
  47. [48]Hanpeng Liu, Yaqian Li, Zidan Wang, Shuoxi Zhang, Zihao Bo, Rinyuichi Takezoe, Kaiwen Long, and Kun He. igvlm: Dynamic instruction-guided vision encoding for question-aware multimodal understanding. arXiv preprint arXiv:2603.02748, 2026. 2arxiv.org/abs/2603.02748
  48. [49]Chengzhi Mao, Xudong Lin, and Wen-Sheng Chu. Language-instructed vision embeddings for controllable and general-izable perception. arXiv preprint arXiv:2606.19584, 2026. 2arxiv.org/abs/2606.19584
  49. [50]Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R Bartoldson, Bhavya Kaikhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, and Micah Goldblum. Teaching pretrained language models to think deeper with retrofitted recurrence. arXiv preprint arXiv:2511.07384, 2025. 3arxiv.org/abs/2511.07384
  50. [51]Lingchen Yang, Jiewei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems, 37:23464–23487, 2024. 2, 5, 19
  51. [52]Thomas Parr and Karl J. Friston. The active construction of the visual world. Neuropsychologia, 104:92–101, 2017. 1DOI
  52. [53]Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, 2018. 2
  53. [54]Tan-Hanh Pham and Chris Ngo. Multimodal chain of continuous thought for latent-space reasoning in vision-language models. arXiv preprint arXiv:2508.12587, 2025. 1arxiv.org/abs/2508.12587
  54. [55]Nedvyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, and Luc Van Gool. Vision encoders should be image agnostic and task driven. arXiv preprint arXiv:2508.16317, 2025. 1, 2
  55. [56]Zhe Qian, Ziyang Gong, Zhongxing Xu, Helan Li, Zhonghua Wang, Fei Leuo, Mingxuan Wang, Xue Yang, Yanbiao Ma, Junchi Yan, et al. Loopvl: Recurrent visual intelligence. arXiv preprint arXiv:2609.38426, 2026. 2, 3, 13arxiv.org/abs/2609.38426
  56. [57]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 2, 21arxiv.org/abs/2103.00020
  57. [58]Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021. 2arxiv.org/abs/2108.08810
  58. [59]Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612–8642, 2024. 1, 5, 14arxiv.org/abs/2403.16999
  59. [60]Wen-Jie Shu, Xuerui Qiu, Rui-Jie Zhu, Harold Haodong Chen, Yexin Liu, and Harry Yang. Loopvit: Scaling visual arc with looped transformers. arXiv preprint arXiv:2602.02156, 2026. 2, 3arxiv.org/abs/2602.02156
  60. [61]Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge, 2025. 5, 21arxiv.org/abs/2504.10342
  61. [62]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578. IEEE, 2024. 1, 2arxiv.org/abs/2401.06209
  62. [63]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcın Kardas, Viktor Kerkez, Madi̇an Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. 21arxiv.org/abs/2307.09288
  63. [64]Pavan Kumar Anasolaus Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Kokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastvlm: Efficient vision encoding for vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19769–19780, 2025. 2
  64. [65]Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. Advances in Neural Information Processing Systems, 38:30865–30891, 2026. 1arxiv.org/abs/2504.08837
  65. [66]Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, and Jian Li. Smelt: Scaling laws for compute-matched moe looped transformers. arXiv preprint arXiv:2609.01343, 2026. 2, 3arxiv.org/abs/2609.01343
  66. [67]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 5, 13arxiv.org/abs/2508.18265
  67. [68]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 1, 5arxiv.org/abs/2203.11171
  68. [69]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 1, 5arxiv.org/abs/2201.11903
  69. [70]Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang, Freda Shi, Hui Liu, Hanqing Lu, Qichang Xie, and Yuyin Zhou. From seeing to thinking: Decoupling perception and reasoning improves post-training of vision-language models. In Proceedings of the 43rd International Conference on Machine Learning, pages 135672–135690. PMLR, 2026. 1
  70. [71]Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084–13094. IEEE, 2024. 2, 5, 21arxiv.org/abs/2312.14135
  71. [72]Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. Advances in Neural Information Processing Systems, 37:33108–33140, 2024. 2, 5, 19
  72. [73]Alfred L. Yarbus. Eye movements and vision. Springer, 2013. 1
  73. [74]Wencheng Ye, Yi Bin, Yujuan Ding, Hongye Fang, Zheng Wang, Xing Xu, Jingkuan Song, Yun Zhang, Sirui Da, and Heng Tao Shen. The ebb and flow of multimodal focus: Scheduling visual relay windows for grounded vlm reasoning. arXiv preprint arXiv:2607.11436, 2026. 1arxiv.org/abs/2607.11436
  74. [75]Yichen You, Tianyu Fu, Aosong Feng, Xingtaı Lv, Xuefei Ning, Ning Ding, and Yu Wang. Improving test-time scaling with adaptive looped transformers, 2026. 2, 3arxiv.org/abs/2609.35748
  75. [76]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952. IEEE, 2023. 2, 21arxiv.org/abs/2303.15343
  76. [77]Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, Ludwig Schmidt, and Serena Yeung-Levy. Automated generation of challenging multiple-choice questions for vision language model evaluation, 2025. 5, 21arxiv.org/abs/2501.03225
  77. [78]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, et al. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025. 2, 3arxiv.org/abs/2510.25741

Paper details

vlnc:W_823t2gp7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents