betaFeedback

WorksComputer Science

When Does Reusing Layers Help Multimodal Embeddings?

Abstract

We ask whether a compact universal multimodal embedding models can produce better search embeddings by running the same layers again. Looped transformers add computation with nearly unchanged model size, making them an appealing alternative to a larger encoder. Their success in language-model reasoning and recent use for retrieval motivate the idea, but higher retrieval scores alone do not reveal what an extra pass contributes. We study eleven loop designs and training variants on three checkpoints of one vision-language architecture. Our protocol distinguishes whether a model uses its extra pass, whether its image states respond to the query, and whether those updates improve retrieval. We prove that an image placed before the query remains query-independent through repeated layers when information only travels forward. We then open a query-to-image path in three ways, including order-alternating recurrence, which places the query first on the next pass. Each path produces query-sensitive image states. For bidirectional attention and order alternation, freezing later visual updates does not lower accuracy. On all 78 datasets of the Massive Multimodal Embedding Benchmark version 2 (MMEB-V2), a standard added loop loses 0.46 points against its trained control; an intermediate generative stage changes the effect to +0.36. A model pretrained with recurrence gains +4.23 points over its own loop-removed control and benefits from query-conditioned image updates when answering visual questions or locating objects from text.

Paper details

vlnc:W_fsvvrzjk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC-BY-NC-SA-4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Opus 5.5, GPT-6.1-Sol

View statement
Looking for related papers…