Valency Hub
alphaFeedback
PDF

WorksComputer Science

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

Henrique Schechter Vera, Sahil Dua, Jovan Andonov, Sara Smoot, Sebastian Russo

Published Fresh — added to the Hub within the last 30 daysv136 min read

AI contributions
Not recorded for this paper
View details

Abstract

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.

Introduction

Embedding models map data to fixed-length vector representations, placing semantically similar data near each other in the embedding space. Text embedding models have become a popular choice for tasks like information retrieval, semantic similarity, clustering, and retrieval-augmented generation. However, to work with audio and visual data, these require additional tools like automatic speech recognition (ASR) and image captioners which both add computational cost and fail to capture signals like fine-grained visual details and non-verbal or environmental sounds.

Multimodal embedding models like CLIP [1], ALIGN [2], and SigLIP [3] add a modality tower for vision, and so achieve strong cross-modal quality, but they are unable to natively process mixed-modal input to capture interactions between input modalities. Recent efforts have adapted Multimodal Large Language Models (MLLMs) into embedding models that allow interleaved mixed-modal input and support more complex tasks such as visual question answering and visual grounding [4, 5, 6, 7]; subsequent work has built on more capable MLLMs to incorporate video and audio modalities [8, 9, 10, 11, 12]. However, as MLLMs have grown in size, so have embedding models adapted from them. These models typically contain several billion parameters, making them prohibitively expensive for applications that are latency-sensitive, high-throughput, or require on-device deployment for privacy or offline access.

In this paper, we introduce EmbeddingGemma 2,2 a 744M-parameter general-purpose multimodal embedding model that maps text, images, video, and audio—as well as arbitrary interleaved combinations thereof—into a single unified embedding space (Figure 1). EmbeddingGemma 2 combines a 271M-parameter bidirectional transformer backbone (embedder included) with a 167M-parameter vision encoder and a 305M-parameter audio encoder. The modality encoders independently convert media into sequences of soft token embeddings that enter the shared transformer backbone, which means these parameters are modular: EmbeddingGemma 2 can alternatively be run with 271M parameters as a text-only model, with 439M as a vision-language model, or with 577M as an audio-language model.

Unified multimodal embedding architecture mapping text, code, images, video, and audio through modular encoders and a shared backbone into a unified vector space

Figure 1. EmbeddingGemma 2 natively maps text, images, video, audio, and combinations thereof into a unified, semantically-aligned vector space.

The model builds on the architecture and weights from the capable Gemma 4 model family [13] and learns from the powerful Gemini Embedding 2 model [8]. Our training regime consists of three stages spanning various tasks, modalities, and domains, which results in a broadly capable model, excelling at problems like code search, visual question answering, video moment retrieval, audio classification, and much more. We separate text-only and multimodal training as a form of curriculum learning, and separate distillation from contrastive and regularization objectives in order to effectively learn an embedding space tailored to the model. For extreme resource-constrained settings, the model offers high-quality truncated embeddings and separate checkpoints adapted with quantization-aware training.

We present ablation studies explaining our architectural and training choices. We find that simplifying per-layer embeddings to omit the dedicated lookup table not only reduces cost but improves performance; we present the effects of using different vision token budgets during training and inference, observing it only significantly affects tasks like document retrieval which require fine-grained visual understanding; we demonstrate the importance of co-training the vision encoder; we show the impact of finetuning with contrastive learning rather than distillation; and we confirm the value of native audio understanding over ASR.

Finally, we evaluate the model on seven major, comprehensive benchmark suites spanning text, code, image, video, document, speech, and sound inputs. EmbeddingGemma 2 establishes a new Pareto frontier across benchmarks spanning all modalities (Figure 2). Despite using fewer parameters for text than its predecessor and handling more modalities, it improves on both multilingual text and code benchmarks. On multimodal evals, EmbeddingGemma 2 often matches or exceeds the quality of specialized and multi-billion-parameter models.

Pareto frontiers across code, image, video, visual document, and audio benchmarks

Figure 2. EmbeddingGemma 2 establishes a new Pareto frontier across benchmarks covering code, image, video, visual documents, and audio. Methods and detailed results in § 5. *Mean(Task) for MTEB(Code), MAEB, and MMEB-v2; Mean(Type) for MIEB(lite).

EmbeddingGemma 2

Architecture

Backbone The backbone follows a transformer architecture [14] with bidirectional attention. Given a tokenized input sequence TT of length LL, we map discrete text tokens via an embedder E\mathcal{E} into input token embeddings Tin=E(T)∈RL×dM\mathbf{T}_{\mathrm{in}} = \mathcal{E}(T) \in\mathbb{R}^{L \times d_M}, where dMd_M is the model dimension used for the transformer’s inner representations (note multimodal inputs are incorporated into Tin\mathbf{T}_{\mathrm{in}} as well, described in detail in § 2.3). We then apply Mn\mathcal{M}_n, an nn-layer transformer with bidirectional attention, to produce a sequence of output token embeddings Tout=Mn(Tin)∈RL×dM\mathbf{T}_{\mathrm{out}} = \mathcal{M}_n(\mathbf{T}_{\mathrm{in}}) \in\mathbb{R}^{L \times d_M}. Next, we employ mean pooling P\mathcal{P} to average the output token embeddings along the sequence axis to generate a single embedding representing all the information in the input, producing Pembed=P(Tout)∈RdM\mathbf{P}_{\mathrm{embed}} = \mathcal{P}(\mathbf{T}_{\mathrm{out}}) \in\mathbb{R}^{d_M}. Finally, we apply a randomly initialized linear projection ff to scale the embedding to the target dimension dd, followed by ℓ2\ell_2-normalization to produce the final unit-norm embedding, E=f(Pembed)∥f(Pembed)∥2∈Rd\mathbf{E} = \frac{f(\mathbf{P}_{\mathrm{embed}})}{\lVert f(\mathbf{P}_{\mathrm{embed}})\rVert_2} \in\mathbb{R}^{d}. In EmbeddingGemma 2, we use n=24n = 24, dM=512d_M = 512, d=768d = 768.

Notably, the transformer Mn\mathcal{M}_n harnesses per-layer embeddings (PLE), as introduced in Gemma 3n [15]. These act as residual inputs which give each layer a path to receive tailored information directly from the input token embeddings. However, instead of additionally utilizing a large, separate token lookup table specifically for PLE, EmbeddingGemma 2 simply adapts the input embeddings Tin\mathbf{T}_{\mathrm{in}} produced by the main embedding table and modality encoders (comparison in § 4.1). For each layer l∈[n]l \in[n], we project the input token embeddings into a layer-specific representation Pl=RMSNorm⁡(pl(Tin))∈RL×dP\mathbf{P}_l = \operatorname{RMSNorm}(p_l(\mathbf{T}_{\mathrm{in}})) \in\mathbb{R}^{L \times d_P} via linear projection plp_l. We use the layer’s output Hl′∈RL×dM\mathbf{H}'_l \in\mathbb{R}^{L \times d_M} to compute gate Gl=σ(gl(Hl′))∈RL×dP\mathbf{G}_l = \sigma(g_l(\mathbf{H}'_l)) \in\mathbb{R}^{L \times d_P}, where glg_l is a linear projection and σ\sigma is a GELU activation [16]. We then modulate the per-layer input and project it back to the model dimension via linear projection ulu_l, producing update Ul=RMSNorm⁡(ul(Gl⊙Pl))∈RL×dM\mathbf{U}_l = \operatorname{RMSNorm}(u_l(\mathbf{G}_l \odot\mathbf{P}_l)) \in\mathbb{R}^{L \times d_M}, to finally yield Hl=Hl′+Ul\mathbf{H}_l = \mathbf{H}'_l + \mathbf{U}_l to be processed by the next layer. In EmbeddingGemma 2, we use dP=dM=512d_P = d_M = 512.

Modality Encoders

As in Gemma 4, the vision encoder is a vision transformer [ViT, [17]] with a patch size of 16 and the audio encoder consists of two downsampling convolution layers followed by twelve conformer layers [18], based on the Universal Speech Model (USM) architecture [19]. The vision and audio encoders produce soft tokens with dimensionalities dV=768d_V = 768 and dA=1536d_A = 1536, respectively. Using learned linear projections ϕvis\phi_{\mathrm{vis}} and ϕaud\phi_{\mathrm{aud}}, we map these to the model dimension dMd_M. These soft tokens are interleaved with the token embeddings at their corresponding positions to form a combined input token embedding sequence Tin∈RL×dM\mathbf{T}_{\mathrm{in}} \in\mathbb{R}^{L \times d_M}, which is processed as described above. See § 2.3 below for more details on how multimodal inputs are handled.

Input

In general, an input xx consists of an ordered sequence of M≥1M \ge1 modality segments x=(s1,s2,…,sM)x = (s_1, s_2, \ldots, s_M), where each segment sms_m belongs to one of four modalities:

  1. Text is tokenized into NtxtN_{\mathrm{txt}} tokens which are mapped via the embedding table to a sequence of token embeddings: Ttxt=E(sm)∈RNtxt×dM\mathbf{T}_{\mathrm{txt}} = \mathcal{E}(s_m) \in\mathbb{R}^{N_{\mathrm{txt}} \times d_M}.

  1. Images are each processed by the vision encoder V\mathcal{V} and spatially pooled to below a fixed budget of NvisN_{\mathrm{vis}} soft tokens, Tvis′=V(sm)∈RNvis×dV\mathbf{T}'_{\mathrm{vis}} = \mathcal{V}(s_m) \in\mathbb{R}^{N_{\mathrm{vis}} \times d_V}. NvisN_{\mathrm{vis}} defaults to 280 for images and 140 for video during training, but is configurable to 70, 140, 280, 560, or 1120 at inference time—see how this impacts quality in § 4.4. These soft tokens are then projected by the vision adapter and wrapped with delimiter tokens: Tvis=⟨|SOI|⟩⊕ϕvis(Tvis′)⊕⟨|EOI|⟩∈R(Nvis+2)×dM\mathbf{T}_{\mathrm{vis}} = \langle\texttt{|SOI|} \rangle\oplus\phi_{\mathrm{vis}}(\mathbf{T}'_{\mathrm{vis}}) \oplus\langle\texttt{|EOI|} \rangle\in\mathbb{R}^{(N_{\mathrm{vis}}+2) \times d_M}.

  1. Audio is processed in 40ms chunks with Mel filterbank inputs, producing 25 soft tokens per second of audio: Taud′=A(sm)∈RNaud×dA\mathbf{T}'_{\mathrm{aud}} = \mathcal{A}(s_m) \in\mathbb{R}^{N_{\mathrm{aud}} \times d_A}. These are projected by the audio adapter and wrapped with different delimiter tokens to produce: Taud=⟨|SOA|⟩⊕ϕaud(Taud′)⊕⟨|EOA|⟩∈R(Naud+2)×dM\mathbf{T}_{\mathrm{aud}} = \langle\texttt{|SOA|} \rangle\oplus\phi_{\mathrm{aud}}(\mathbf{T}'_{\mathrm{aud}}) \oplus\langle\texttt{|EOA|} \rangle\in\mathbb{R}^{(N_{\mathrm{aud}}+2) \times d_M}.

  1. Video clips are typically sampled at 1 frame per second to begin. Then, for longer videos, frames are further sampled to fit a per-video frame budget NfrmN_{\mathrm{frm}}—this is sometimes via truncation, by selecting the first NfrmN_{\mathrm{frm}} frames, and sometimes via uniform sampling, by selecting NfrmN_{\mathrm{frm}} evenly spaced frames over the entire video. During our training and evaluations we use Nfrm=32N_{\mathrm{frm}} = 32. Since each frame is processed by the vision encoder as an image, this results in up to NfrmN_{\mathrm{frm}} segments, each with at most NvisN_{\mathrm{vis}} soft tokens and two image delimiter tokens.

Optionally, frames can be interleaved with audio tokens from the video’s soundtrack, with each frame coming first and then the audio that follows it divided into 1-second segments. Usually, this corresponds to a single 1-second audio segment per frame, but if frames are truncated we might have multiple trailing audio segments, and if frames are uniformly sampled, we might have multiple audio segments between frames. In interleaved video-audio, we include text timestamps in MM:SS format both before each frame and before each audio segment, to indicate when the frame is taken and when the audio chunk begins. Thus, video is represented as a combination of the former modalities, yielding an embedding sequence Tvid\mathbf{T}_{\mathrm{vid}} composed of interleaved sequences of Ttxt\mathbf{T}_{\mathrm{txt}}, Tvis\mathbf{T}_{\mathrm{vis}}, and Taud\mathbf{T}_{\mathrm{aud}}.

Then, denoting as Tm∈{Ttxt,Tvis,Taud,Tvid}\mathbf{T}_m \in\{\mathbf{T}_{\mathrm{txt}}, \mathbf{T}_{\mathrm{vis}}, \mathbf{T}_{\mathrm{aud}}, \mathbf{T}_{\mathrm{vid}}\} the embedding sequence of segment sms_m, the final concatenated sequence is T(x)=⟨∣BOS∣⟩⊕T1⊕T2⊕⋯⊕TM⊕⟨∣EOS∣⟩∈RL×dM\mathcal{T}(x) = \langle|\mathrm{BOS}|\rangle\oplus\mathbf{T}_1 \oplus\mathbf{T}_2 \oplus\cdots\oplus\mathbf{T}_M \oplus\langle|\mathrm{EOS}|\rangle\in\mathbb{R}^{L \times d_M}, with a total sequence length LL of up to 8192 during training. This is consumed by the model as Tin\mathbf{T}_{\mathrm{in}} in § 2.1.

Each training example includes multiple inputs: a query qiq_i, a positive target (or passage) pi+p_i^+, and (optionally) hard negative targets {pi−}\{p_i^-\}. An example may also include prescribed task prefixes tqt_q and tpt_p for queries and targets respectively, describing the nature of the task (see Appendix § 8). For instance, for retrieval we use “task: search result | query: {text}” and “title: {title | ‘none’} | text: {text}”. The query and targets are embedded as vectors in Rd\mathbb{R}^d:

qi=f(P(Mn(T(tq⊕qi)))),pi+=f(P(Mn(T(tp⊕pi+)))).(1)\mathbf{q}_i = f\bigl(\mathcal{P}(\mathcal{M}_n(\mathcal{T}(t_q \oplus q_i)))\bigr), \qquad\mathbf{p}_i^+ = f\bigl(\mathcal{P}(\mathcal{M}_n(\mathcal{T}(t_p \oplus p_i^+)))\bigr). \tag*{(1)}

Objective

EmbeddingGemma 2 was trained using multiple losses. Here we introduce the loss functions themselves, and in § 2.5 we explain how they are introduced during training.

Contrastive loss The first is a noise-contrastive estimation (NCE) loss with in-batch, hard, and same-tower negatives. Given a batch of size BB, we define the contrastive loss as:

LC=−1B∑i=1Blog⁡ZiPZiP+ZiHN+ZiIB+ZiSTNZiP=esim⁡(qi,pi+)/τZiHN=∑pi−esim⁡(qi,pi−)/τZiIB=∑j=1B1TN(i,j)esim⁡(qi,pj+)/τZiSTN=∑j=1B1TN(i,j)(esim⁡(qi,qj)/τ+esim⁡(pi+,pj+)/τ)(2)\begin{aligned} \mathcal{L}_C &= -\frac{1}{B}\sum_{i=1}^{B}\log\frac{Z_i^{\mathrm{P}}}{Z_i^{\mathrm{P}}+Z_i^{\mathrm{HN}}+Z_i^{\mathrm{IB}}+Z_i^{\mathrm{STN}}} \\ Z_i^{\mathrm{P}} &= e^{\operatorname{sim}(\mathbf{q}_i,\mathbf{p}_i^+)/\tau} \\ Z_i^{\mathrm{HN}} &= \sum_{\mathbf{p}_i^-} e^{\operatorname{sim}(\mathbf{q}_i,\mathbf{p}_i^-)/\tau} \\ Z_i^{\mathrm{IB}} &= \sum_{j=1}^{B}\mathbb{1}_{\mathrm{TN}}(i,j)e^{\operatorname{sim}(\mathbf{q}_i,\mathbf{p}_j^+)/\tau} \\ Z_i^{\mathrm{STN}} &= \sum_{j=1}^{B}\mathbb{1}_{\mathrm{TN}}(i,j)\left(e^{\operatorname{sim}(\mathbf{q}_i,\mathbf{q}_j)/\tau}+e^{\operatorname{sim}(\mathbf{p}_i^+,\mathbf{p}_j^+)/\tau}\right) \tag*{(2)} \end{aligned}

where sim⁡(x,y)=xTy/∥x∥∥y∥\operatorname{sim}(\mathbf{x},\mathbf{y}) = \mathbf{x}^{\mathsf{T}}\mathbf{y}/\lVert\mathbf{x}\rVert\lVert\mathbf{y}\rVert is cosine similarity. Above, ZiPZ_i^{\mathrm{P}} corresponds to the positive, ZiHNZ_i^{\mathrm{HN}} corresponds to the hard negatives, ZiIBZ_i^{\mathrm{IB}} corresponds to in-batch negatives, and ZiSTNZ_i^{\mathrm{STN}} corresponds to same-tower negatives [20]. 1TN\mathbb{1}_{\mathrm{TN}} masks out false negatives from duplicates:

1TN(i,j)={0if qi=qj or pi+=pj+,1otherwise.(3)\mathbb{1}_{\mathrm{TN}}(i,j) = \begin{cases} 0 & \text{if } \mathbf{q}_i = \mathbf{q}_j \text{ or } \mathbf{p}_i^+ = \mathbf{p}_j^+, \\ 1 & \text{otherwise}. \end{cases} \tag*{(3)}

We forego the hardness weighting mechanism from our previous model as it amplifies noisy or weakly associated positive pairs, especially during early training, and compounds gradient scale imbalances between tasks of different modalities. We add same-tower negatives to prevent queries and targets from clustering into disjoint subspaces.

For listwise ranking data with PP ordered positives (pi,1+≻⋯≻pi,P+)(\mathbf{p}^{+}_{i,1} \succ\cdots\succ\mathbf{p}^{+}_{i,P}) per query, we train both on extracted triplets using LC\mathcal{L}_{C} and on the full ranked lists using a listwise loss that extends LC\mathcal{L}_{C} based on the Plackett–Luce model, as in Xia et al. [21]:

LClist=−1BP∑i=1B∑r=1Plog⁡Zi,rP∑k=rPZi,kP+ZiHN+ZiIBZi,rP=esim⁡(qi,pi,r+)/τ,ZiIB=∑j=1B1TN(i,j)(∑k=1Pesim⁡(qi,pj,k+)/τ+∑pj−esim⁡(qi,pj−)/τ)(4)\begin{aligned} \mathcal{L}_{C}^{\mathrm{list}} &= -\frac{1}{BP}\sum_{i=1}^{B}\sum_{r=1}^{P}\log\frac{Z^{\mathrm{P}}_{i,r}}{\sum_{k=r}^{P} Z^{\mathrm{P}}_{i,k}+Z^{\mathrm{HN}}_{i}+Z^{\mathrm{IB}}_{i}} \\ Z^{\mathrm{P}}_{i,r} &= e^{\operatorname{sim}(\mathbf{q}_{i},\mathbf{p}^{+}_{i,r})/\tau}, \qquad Z^{\mathrm{IB}}_{i}=\sum_{j=1}^{B}\mathbb{1}_{\mathrm{TN}}(i,j)\left(\sum_{k=1}^{P}e^{\operatorname{sim}(\mathbf{q}_{i},\mathbf{p}^{+}_{j,k})/\tau}+\sum_{\mathbf{p}^{-}_{j}}e^{\operatorname{sim}(\mathbf{q}_{i},\mathbf{p}^{-}_{j})/\tau}\right) \tag*{(4)} \end{aligned}

Here, we contrast each rank-rr positive against all lower-ranked positives (k>rk > r) in addition to hard and in-batch negatives, with 1TN(i,j)\mathbb{1}_{\mathrm{TN}}(i,j) adapted to mask out in-batch negatives that duplicate pi,r+\mathbf{p}_{i,r}^{+}. We also add small squared-hinge penalties on positive and hard-negative similarities to calibrate the absolute score distribution for threshold-based tasks.

Spread-out regularizer The second loss is based on the global orthogonal regularizer (GOR) introduced in Zhang et al. [22]. The idea is to make the embeddings of a random pair of inputs have similar statistical properties (mean and second moment) as two points independently and uniformly sampled from the unit sphere. Following EmbeddingGemma 1 [23], we use only the “second moment” term of the regularizer’s original definition, as we find this also sufficiently pushes the “mean” term towards its target value:

LS=1B(B−1)∑1≤i≠j≤B(qiTqj)2+1B(B−1)∑1≤i≠j≤B(pi+Tpj+)2.(5)\mathcal{L}_{S}=\frac{1}{B(B-1)}\sum_{1\leq i\ne j\leq B}(\mathbf{q}_{i}^{\mathsf{T}}\mathbf{q}_{j})^{2}+\frac{1}{B(B-1)}\sum_{1\leq i\ne j\leq B}(\mathbf{p}_{i}^{+\mathsf{T}}\mathbf{p}_{j}^{+})^{2}. \tag*{(5)}

This loss encourages the model to produce embeddings that are spread out over the embedding space, to fully utilize its expressive power. Embedding spaces have been observed to form narrow cones [24, 25, 26, 27], but there are theoretical and empirical benefits to isotropy [28, 29], including performance in downstream tasks. Note that pushing random pairs toward orthogonality complements the squared-hinge penalties above to make scores more calibrated and interpretable. Enforcing isotropy also aims to ensure that a) the model is robust to weight and embedding quantization, and b) the embeddings can be retrieved efficiently in vector databases using approximate nearest neighbor (ANN) algorithms.

Distillation loss The final is an embedding matching loss based on Kim et al. [30]. In contrast to previous distillation research which has relied only on the teacher’s query-document relevance scores as a signal [31, 32], this loss directly aligns EmbeddingGemma 2’s embedding space with that of a teacher model, allowing it to more effectively learn from the larger, more powerful Gemini Embedding 2 model. Unlike Kim et al. [30], which applies a projection layer to distill from a model with different dimensionality, we instead make use of Gemini Embedding 2’s support for lower dimensions and simply truncate the teacher embeddings. As in Schechter Vera et al. [23], we apply embedding matching not only to queries and targets but also to hard negative targets, which intuitively serves to transfer the teacher’s ability to discriminate between queries and their hard negatives, and empirically does improve performance. We weigh these individual losses uniformly, letting

LD=LDq+LDp++LDp−,whereLDu=1B∑i=1B∥ui−u~i∥22for u∈{q,p+,p−},(6)\mathcal{L}_{D}=\mathcal{L}_{D}^{q}+\mathcal{L}_{D}^{p^{+}}+\mathcal{L}_{D}^{p^{-}},\qquad\text{where}\quad\mathcal{L}_{D}^{u}=\frac{1}{B}\sum_{i=1}^{B}\lVert\mathbf{u}_{i}-\tilde{\mathbf{u}}_{i}\rVert_{2}^{2}\quad\text{for }u\in\{q,p^{+},p^{-}\}, \tag*{(6)}

and u~i\tilde{\mathbf{u}}_{i} denotes the truncated, ℓ2\ell_{2}-normalized teacher embedding for input ui\mathbf{u}_{i} (with LDp−\mathcal{L}_{D}^{p^{-}} averaged over hard negatives when multiple pi−\mathbf{p}_{i}^{-} are present).

Matryoshka Representation Learning We adapt the contrastive, spread-out, and distillation losses with Matryoshka Representation Learning (MRL) [33], splitting each loss into kk separate losses applied to kk overlapping sub-dimensions of the embedding. EmbeddingGemma 2 provides 768-dimensional embeddings, additionally supporting 512-, 256-, and 128-dimensional embeddings via MRL.

Recipe

Initialization EmbeddingGemma 2 builds directly on Gemma 4’s extensive world knowledge and capabilities. The backbone is adapted from a pretrained Gemma 4 decoder-only model, switching the causal attention mask to bidirectional attention to enable fully contextual token representations [34, 35, 36, 8]. We omit explicit adaptation stages for bidirectional attention [23, 37], which can be costly and have been shown to be unnecessary for learning expressive representations [23, 38]. We also remove the PLE-specific token lookup table (detailed in § 4.1), and add mean pooling [39, 23], a linear projection, and ℓ2\ell_{2}-normalization after the transformer. The vision encoder is initialized with the smaller Gemma 4 vision encoder and further trained in the two multimodal stages, which we show in § 4 to be critical for quality. The audio encoder is bit-identical to that in the Gemma 4 E2B and E4B models [13] and kept frozen during training.

We then train EmbeddingGemma 2 on a mixture of embedding tasks spanning various modalities and task types. In total, the model sees approximately 2.9T tokens, of which ∼\sim300B are seen during text pre-finetuning, ∼\sim2T during multimodal pre-finetuning, and ∼\sim600B during multimodal finetuning.

Text Pre-finetuning We begin with a pre-finetuning stage like the one in Schechter Vera et al. [23]. This entails a large-scale unsupervised data mixture of query-target pairs from question answering, sentence similarity, code retrieval, and web search tasks using various languages. As before, we opt for a larger batch size both to stabilize the gradient and to provide more (in-batch) negatives for each input example. In this stage we almost exclusively employ distillation loss, with only an additional contrastive loss using a small weight. Here, as with subsequent stages, tasks are sampled at pre-determined, empirically defined rates. We found the inclusion of a text-only stage important for the model’s final text quality. This stage can be seen as a form of curriculum learning, or as an implicit warm-up for the bidirectional attention, similar to BehnamGhad­er et al. [37]; it has also been shown to enhance the discriminability of both text and non-text embeddings [10].

Multimodal Pre-finetuning We then continue with pre-finetuning, but with a new mixture which also includes inputs with images, audio, video, and documents across unimodal, cross-modal, and mixed-modal tasks, most with additional hard negatives. The text tasks in this stage and the next include those described in the finetuning stage of Schechter Vera et al. [23], which aim to cover task diversity, language diversity, and coding capability. In this stage, we exclusively employ distillation loss, to inherit the teacher’s ability to map diverse and mixed modalities into a shared semantic space. We continue training with larger batch sizes and larger learning rates, but train for twice as many steps as in text pre-finetuning to cover the expanded multimodal training set.

Ablation Studies

Multimodal Finetuning In the finetuning stage, we completely replace distillation with contrastive loss. This allows the model to learn a tailored embedding space independent of its teacher, as the teacher’s embedding space is not optimized for the student’s architecture, size, and MRL sub-dimensions. We also add other losses which aim to shape the final embedding space to have desirable properties: spread-out regularizer and MRL losses. As in Schechter Vera et al. [23], we transition to smaller learning rates and batch sizes, and shift towards higher-quality, task-specific datasets, such as tool retrieval, audio captioning, video ranking, medical imaging, visual product search, cropped-object retrieval, and academic question-answering.

Model Merging Finally, we combine checkpoints from our finetuning run by averaging their weights, to improve our final model’s quality and robustness. Rather than running multiple finetuning jobs, either with different data mixture rates [23] or with different hyperparameters [40], we merge checkpoints along a single finetuning trajectory [41]. Specifically, we compute an unweighted average between the final checkpoint from the finetuning run and a significantly earlier checkpoint from the same run. Like Wortsman et al. [42], which similarly merges pretrained and finetuned checkpoints, we found that this helps mitigate degradation on tasks that overfit during finetuning, while also further improving performance on tasks that already benefited from finetuning.

Quantization-Aware Training We additionally provide quantized versions of EmbeddingGemma 2 in different quantization configurations. We provide checkpoints with two types of weight representations for the backbone: int4 per-block (Q4_0) and int4 per-channel (Q4_C). For the modality encoders, we follow the quantization schemes used in Gemma 4: the vision encoder uses 8-bit precision for both weights and activations (W8A8), and the audio encoder uses 8-bit activation precision and {2, 4, 8}-bit weight precision varying by layer cluster.

We apply quantization-aware training [QAT, [43]] to both the backbone and vision encoder to minimize quality degradation (the audio encoder is taken directly from Gemma 4). However, instead of applying QAT directly in the finetuning stage as in Schechter Vera et al. [23], we introduce a separate post-merging tail-patch stage in which we distill (and initialize) our QAT checkpoint from the frozen float checkpoint [44]. This allows for a more resource-efficient QAT stage and narrows the gap in quality between the float and quantized checkpoints for some evaluations, notably MTEB(English, v2). Table 1 shows the quality is comparable between our quantized checkpoints and our raw checkpoint (evaluated at half-precision bfloat16 weights). With Q4_C quantization, the peak on-device memory footprint during inference (including runtime compilation overhead) on a Google Pixel 11 Pro is reduced to 191 MB for text-only, 447 MB for text and vision, and 567 MB for the full multimodal model.

ModelPrecisionMTEB(Multilingual, v2) Mean(Task)MTEB(Multilingual, v2) Mean(Type)MTEB(English, v2) Mean(Task)MTEB(English, v2) Mean(Type)MTEB(Code) Mean(Task)MIEB(lite)‡ Mean(Task)MIEB(lite)‡ Mean(Type)MSEB(Retrieval)‡ OverallMSEB(Retrieval)‡ In-LangMSEB(Retrieval)‡ Cross-Lang
EG 2bfloat1661.3653.8268.4663.8678.6860.1765.6769.5471.6367.68
EG 2int4 (Q4_0)∗60.9653.4768.4863.9176.8556.5961.9564.9968.5661.80
EG 2int4 (Q4_C)†60.8453.3668.4963.9376.7056.7162.2663.9267.9360.33
EG 1bfloat1661.1554.3169.6765.1168.76—————
EG 1int4 (Q4_0)∗60.6253.6169.3164.6567.99—————

Table 1. Comparison of raw and quantized EmbeddingGemma 2 and EmbeddingGemma 1 checkpoints across benchmarks covering text, vision, and audio: MTEB, MIEB(lite), and MSEB(Retrieval). ∗Per-block int4. †Per-channel int4. ‡Different eval harnesses between float and quantized runs.

Ablation Studies

We analyze some key design choices to better understand how they contributed to EmbeddingGemma 2’s ability to produce high-quality embeddings across modalities, domains, and task types. Baseline and ablation models use an intermediate recipe and so numbers are not representative of the final model. Unless otherwise stated, the experiments run finetuning on a pre-finetuned model. We evaluate the models on MMEB-v2 as it covers various tasks involving combinations of text, images, documents, and video, excluding only audio.

Per-Layer Embeddings Table

Gemma 3n and Gemma 4 incorporate Per-Layer Embeddings (PLE) to inject token-level information into intermediate transformer layers. Specifically, they combine a learned projection of the main token embedding with an additional dedicated PLE lookup table. However, a dedicated PLE lookup table provides no direct signals for non-text modalities and consumes additional memory.

We utilize only the projection, as described in detail in § 2.1; this way, we continue providing per-layer inputs but can omit the lookup table. We compare our approach to PLE with a lookup table in Table 2. Removing the dedicated PLE table not only reduces parameter overhead, but also strengthens performance across all three categories (notably +0.69 on Video and +0.61 on VisDoc), confirming that deriving per-layer inputs solely from the shared dMd_M-dimensional embedding space improves multimodal representation quality. We also found similar improvements in text-only evaluations, where one might expect the lookup table to help the most.

PLE StrategyMMEB-v2 OverallImage (Hit@1)Video (Hit@1)VisDoc (NDCG@5)
Projection-only55.4654.6747.8262.37
With lookup55.0754.5747.1361.76

Table 2. Results with and without a dedicated Per-Layer Embeddings (PLE) lookup table.

Finetuning Stage

We investigate whether the gains from our finetuning stage are truly driven by a difference in recipe or merely by extending training. Starting from the multimodal pre-finetuning checkpoint, we compare running the multimodal finetuning stage against continuing distillation for an approximately compute-equivalent duration (the same number of steps, but with larger batch sizes on distillation). Table 3 shows that while distillation plateaus, transitioning to the finetuning recipe with contrastive and spread-out objectives boosts performance significantly for image and video tasks while yielding similar results for visual documents. This demonstrates that the finetuning stage enables the model to move beyond imitating its teacher and learn a better-optimized representation space.

Training StageMMEB-v2 OverallImage (Hit@1)Video (Hit@1)VisDoc (NDCG@5)
Multimodal Pre-finetuning52.2750.1043.2662.26
+ Extra Pre-finetuning52.2849.7443.8662.40
+ Finetuning Stage55.4654.6747.8262.37

Table 3. Results of continued distillation versus adding a finetuning stage with contrastive learning.

Vision Encoder Training

MLLMs often keep modality encoders frozen throughout training [45, 46, 47, 13]. This strategy is also popular in recent multimodal embedding models [48, 9, 49, 4]. For MLLMs, ablations investigating whether freezing or unfreezing the vision encoder is better are mixed [50, 45, 51]. However, we show in Table 4 that, though the Gemma 4 vision encoder is quite strong by itself, training the vision encoder greatly improves performance of the resulting embeddings on multimodal tasks. Analogous gains are likely possible by further training the audio encoder. Note that both runs here start with a text pre-finetuning checkpoint and run multimodal pre-finetuning, to ensure the vision encoder has not already been updated prior to either experiment.

Vision EncoderMMEB-v2 OverallImage (Hit@1)Video (Hit@1)VisDoc (NDCG@5)
Trainable52.2750.1043.2662.26
Frozen48.1844.3539.2460.59

Table 4. Results with trainable and frozen vision encoder.

Some MLLMs manage to additionally keep the backbone frozen [52]. Hönicke et al. [49] similarly joins a backbone text embedding model with modality encoders, keeping each of them frozen. This prevents users from having to backfill embedding databases created with the text embedding model; however, it is possible to translate embeddings from one model’s space to another [53], even without access to paired data or to the original models [54]. Moreover, freezing a text-only backbone empirically seems to limit its ability to understand multimodal inputs, and indeed many of the strongest multimodal embedding models finetune the backbone alongside the modality encoders [12, 10].

Vision Token Budgets

We train EmbeddingGemma 2 with a budget of 280 vision tokens for images and 140 tokens per frame for videos, but the underlying Gemma 4 vision encoder supports budgets from 70 to 1120 tokens. In Table 5 (top), we show the effect of using different vision token budgets at inference time using an intermediate checkpoint trained with 280 tokens per image and 140 per frame. Overall performance scales monotonically, with some distinct modality-specific behaviors. Visual document tasks are highly sensitive to fine-grained details and continue to improve greatly past the training budget; image quality in aggregate begins to saturate at the 280-token budget (despite continued gains on several individual tasks); and video performance actually degrades between 280 tokens and 560, likely because the excessive sequence lengths dilute attention and mean-pooled representations.

ExperimentVision Token Budget Train (Img / Vid)Vision Token Budget EvalMMEB-v2 OverallImage (Hit@1)Video (Hit@1)VisDoc (NDCG@5)
Inference-Time Scaling280 / 1407050.0750.9046.3351.61
Inference-Time Scaling280 / 14014055.3753.5847.9963.60
Inference-Time Scaling280 / 14028056.1555.1548.5163.38
Inference-Time Scaling280 / 14056057.0255.2146.5267.62
Train-Time Scaling280 / 707049.6250.5646.6050.49
Train-Time Scaling280 / 1407049.7650.7645.7951.25
Train-Time Scaling280 / 7014054.6653.2947.1662.33
Train-Time Scaling280 / 14014055.4654.6747.8262.37

Table 5. Results with different vision token budgets (max soft tokens per image or video frame) at train and inference time. Evaluation uses a single budget for all tasks. Top: Inference-time scaling of a model trained with 280 tokens per image and 140 per video frame. Bottom: Train-time scaling with 70 vs. 140 tokens per video frame. The two experiments use different intermediate recipes.

Additionally, we compare models trained with 70 vs. 140 per-frame video tokens (Table 5, bottom) to see whether training with a smaller vision budget could improve quality when using smaller vision budgets at inference time, for resource-constrained deployment. When evaluated with 70 tokens, training with 70 video tokens improves performance on video tasks, as the model is directly exposed to 70-token frames during training. But this advantage does not hold for image or visual document tasks, which drop in quality. This is likely because a) single-image tasks, which continue to be trained with 280 tokens, are treated as a separate modality and do not benefit from seeing 70-token video frames; and b) coarser video representations discard spatial details that otherwise transfer to stronger general visual representations. The latter is further supported when evaluating with 140 tokens, where training with 70 video tokens degrades performance even more, and across modalities—this makes the 140-token video budget preferable across both evaluation regimes. More gains are likely possible by increasing train-time budgets, but with seemingly diminishing returns and at a significant computational cost.

Evaluation

We evaluate EmbeddingGemma 2 on a comprehensive set of benchmarks across diverse modalities, task types, domains, and languages. Specifically, we report numbers from the Massive Text Embedding Benchmark [MTEB, 55, 56] for natural language and code; the Massive Multimodal Embedding Benchmark v2 [MMEB-v2, 57] and Massive Image Embedding Benchmark [MIEB-lite, 58] for image, video, and visual documents; and the Massive Sound Embedding Benchmark and Massive Audio Embedding Benchmark for speech, music, and environmental sounds [MSEB and MAEB, 59, 60]. We provide a summary of our results in Table 6 and dive into each benchmark in more detail below.

EmbeddingGemma 2EmbeddingGemma 1Jina Embed v5Omni NanoQwen3 Embed 0.6BInterestFM UIR CAFe 0.5BSigLIP SoViT 400MLAION CLAP General
Total Parameters744M308M986M596M894M878M194M
Embedding Dimension76876876810248961152512
MTEB(Multi, v2) [55]Mean (Task)61.3661.1565.5264.34–––
Mean (Type)53.8254.3157.6656.01–––
MTEB(Eng, v2) [55]Mean (Task)68.4669.6771.1170.47–––
Mean (Type)63.8665.1165.2264.72–––
MTEB(Code) [55]Mean (Task)78.6868.76–75.42–––
MMEB-v2 [57]Mean (Task)59.01*43.61*49.33–*
- Image57.28*31.56*55.43–*
- Video50.67*30.73*35.87–*
- VisDoc67.84*71.34*50.26–*
MIEB(lite) [58]Mean (Task)60.17*44.24*–52.28*
Mean (Type)65.67*49.38*–53.50*
MAEB [60]Mean (Task)49.87*50.14***34.98
Mean (Type)48.65*55.31***41.08
MSEB(Retrieval) [59]Mean (Task)69.5450.95†–***–

Table 6. Comparison on benchmarks against a slightly larger multimodal peer and modality specialists. * Modality not supported. – Score unavailable. †Computed by transcribing audio with ASR.

Methods

We select as baselines the top-ranked models on each benchmark, including both modality specialists and general-purpose embedding models which support all evaluated modalities. For MTEB, MAEB, and MIEB(lite), we compare against models under 1B parameters; for MMEB-v2, we include models under 2B parameters, as small models are scarce. Because it is one of the most directly comparable models, we evaluate jina-embeddings-v5-omni-nano on MMEB-v2 using its official implementation and verify that, on the subset reported in Hönicke et al. [49], our reproduced score is slightly higher. All other baseline scores are sourced from public leaderboards. We restrict our analysis to single-vector models, given the disparity in storage and latency costs with multi-vector models, and omit models trained on >25% of the evaluation data, as data contamination undermines the evaluation’s validity.

We evaluate EmbeddingGemma 2 using half-precision weights (bfloat16) and the seven types of task instructions detailed in Appendix § 8. We use different sequence lengths depending on the modality and task type. For visual inputs, we use up to Nfrm=32N_{\mathrm{frm}} = 32 frames per video at 140 vision tokens per frame and 280 vision tokens per image to reflect realistic model use, though we note that higher vision token budgets could yield further gains, particularly for visual documents (§ 4.4). Because most baseline models support limited modalities, we count only the parameters EmbeddingGemma 2 utilized for each benchmark; other models’ parameter counts are as reported on respective leaderboards.

Vision

Text

The MTEB suite consists of a massive collection of individual, quality-controlled evaluation tasks. Together, the 162 tasks cover 250+ natural languages and 14+ programming languages, 20 domains (e.g. legal, medical, news, programming, web), and 10 task types: bitext mining, classification, clustering, instruction retrieval, multilabel classification, pair classification, reranking, retrieval, semantic text similarity (STS), and summarization.

Despite having fewer text parameters and sharing its representation space with vision and audio, EmbeddingGemma 2 improves over its predecessor on multilingual text (driven by gains in bitext mining and classification). On English, the model sees a modest regression. Only two smaller models surpass it on multilingual text, and one on English, but they are all trained as specialist text models (Appendix § 9). The model presents competitive text scores which outperform much larger models.

On code (Table 7), EmbeddingGemma 2’s performance is stellar, establishing a new Pareto frontier. It outperforms every sub-4B model on the leaderboard and even comes within 2 points of the 28×28 \times larger frontrunner. In all three benchmarks, the model’s truncated representations degrade minimally, especially down to 256 dimensions.

Model NameParamsDim.Mean (Task)AppsR.COIRCESRCFMTCSMTCSNCCRCSNRCTOCCTODLCQASOQAST2SQL
EmbeddingGemma 2271M76878.789.286.980.585.682.293.890.984.635.653.389.172.5
51277.289.286.679.985.382.093.790.782.935.752.787.061.1
25676.288.385.577.984.180.592.990.282.835.850.584.960.9
12871.482.580.870.078.574.889.588.177.434.644.177.059.7
pplx-embed-v1-0.6b596M102475.878.782.476.483.084.788.189.985.035.243.390.773.0
UltraBERT-Embed-150M151M76873.368.582.262.284.486.493.790.088.137.236.392.458.0
Giga-Embeddings-instruct484M102472.981.671.861.891.289.673.987.386.537.637.992.362.9
Qwen3-Embedding-0.6B596M102475.475.384.764.490.886.491.791.086.031.436.590.076.7
embeddinggemma-300m308M76868.884.475.562.151.480.373.790.185.533.543.686.558.4
granite-embed-311m-multi312M76863.860.677.656.257.077.355.586.983.234.435.486.355.7
granite-embed-english-r2149M76857.214.064.759.352.577.247.780.877.135.037.091.849.6
granite-embed-97m-multi97M38460.450.275.453.355.275.251.985.376.534.034.082.052.3
granite-embed-small-eng48M38455.813.560.557.252.276.948.478.377.633.635.690.046.3
granite-embed-125m-eng125M76853.311.855.158.842.175.347.676.966.729.636.689.848.7
b1ade-embed335M102453.66.779.855.837.872.551.886.757.022.335.383.554.0

Table 7. Performance of top-ranking models under 1B parameters on MTEB(Code).

Vision

MMEB-v2 spans 78 tasks across image, video, and visual document modalities, the latter of which includes PDFs, slides, and charts. These include retrieval, classification, question answering, visual grounding, and moment retrieval. MIEB(lite) complements this with linear-probe classification, clustering, multiple choice, compositionality, and visual semantic text similarity. They both implicitly evaluate countless visual skills like object detection, counting, optical character recognition, and temporal reasoning. They also cover various permutations of cross-modal and mixed-modal situations.

EmbeddingGemma 2 is the smallest model on the MMEB-v2 leaderboard (Table 8). It excels especially at classification and retrieval, but provides strong scores across the board—its overall score is exceeded only by models around 2B parameters. It also outperforms various recent models including vision-language specialists with more than 8B parameters. On MIEB(lite), only two models beat EmbeddingGemma 2, and they have well over 4B parameters. Compared to top models under 1B parameters (Table 9), EmbeddingGemma 2 sets the state of the art on both aggregate metrics by a wide margin (+7.9 Task, +12.2 Type). It also offers remarkable improvements in nearly all task types, notably multilingual visual STS (+22.8) and multilingual retrieval (+12.5). Even at 128 dimensions, the model outperforms every other sub-1B model at full dimensionality.

Model NameParamsDim.Mean (Task)Image CLSImage QAImage RETImage GDImage AllVideo CLSVideo QAVideo RETVideo MRETVideo AllVisDoc VDRv1VisDoc VDRv2VisDoc VRVisDoc OODVisDoc All
EmbeddingGemma 2439M76859.054.441.965.877.457.356.248.647.250.650.770.658.372.763.367.8
51258.454.241.465.477.156.956.048.546.550.350.369.256.372.062.266.6
25656.253.239.563.676.555.554.648.444.949.249.364.953.267.259.562.6
12845.639.929.654.772.845.640.844.434.943.640.651.939.252.648.849.5
Eddy-vl_embed_1.9B_v11.93B204863.254.961.764.884.763.451.557.941.948.050.078.855.779.164.972.7
interestFM-UIR-CAFe894M89649.356.445.357.572.055.433.941.729.739.735.956.932.668.623.850.3
jina-embed-v5-omni-nano986M76843.634.616.037.445.531.628.141.518.038.330.777.653.978.662.271.3

Table 8. Performance of top-ranking models under 2B parameters on MMEB-v2.

Model NameParamsDim.Mean (Task)Mean (Type)Multi. Retr.A2A Retr.Comp.Doc. Und.Image Class.Image Clus.Vision QAVSTS (eng)VSTS (multi)0-shot Class.
EmbeddingGemma 2439M76860.265.765.438.345.767.374.381.359.181.576.867.0
51259.965.364.838.145.766.574.280.058.881.676.866.9
25658.664.262.937.144.564.073.778.956.481.876.865.5
12854.260.050.733.040.156.671.579.356.780.874.856.4
siglip-so400m-patch14-384878M115252.353.535.432.432.846.978.875.948.069.641.473.8
siglip-large-patch16-384652M102451.453.343.531.932.144.977.775.246.869.639.871.3
siglip-base-patch16-512204M76849.650.934.830.829.742.674.669.755.567.138.166.3
CLIP-ViT-L-14-DataComp.XL428M76849.050.422.631.031.730.875.380.454.972.535.869.4
CLIP-ViT-H-14-laion2B986M102449.350.023.932.834.833.776.879.346.868.333.969.4
siglip-base-patch16-256-multi371M76846.649.752.928.230.722.973.468.253.363.740.363.2
jina-embed-v5-omni-nano986M76844.249.438.723.629.674.263.760.550.077.954.021.8
CLIP-ViT-L-14-laion2B428M76847.447.621.431.133.628.775.976.446.968.726.067.8
clip-vit-large-patch14428M76846.447.419.826.739.429.474.571.344.869.435.163.8
siglip-base-patch16-224203M76846.147.433.629.332.520.973.568.453.064.233.565.0
CLIP-ViT-B-16-DataComp.XL150M51245.546.619.928.331.422.773.273.656.969.628.561.9

Table 9. Performance of top-ranking models under 1B parameters on MIEB(lite).

Audio

MAEB includes 30 tasks encompassing five acoustic domains: speech, music, environmental, bioacoustics, and emotion. These are spread across unimodal and cross-modal tasks within retrieval, reranking, clustering, and multiple types of classification. MSEB(Retrieval) contributes a focus on multilingual speech understanding. It evaluates the model’s ability to retrieve a text passage from a spoken query, both in the same language and across languages, with a total of 26 languages and regional locales.

On MAEB, EmbeddingGemma 2 again sets a new frontier for efficiency. Notably, among sub-1B models, it achieves state-of-the-art scores on any-to-any retrieval and audio classification (Table 10), as well as on about a third of all individual tasks. It is also state-of-the-art among models of all sizes for classification of audio pairs and a sixth of all individual tasks. Its mean task score is only about 0.2 points away from the best model under 1B parameters, and about 3.5 points away from the highest-scoring model overall. Even its 128-dimensional embeddings beat every other sub-1B model.

Model NameParamsDim.Mean (Task)Mean (Type)A2A Retr.Audio Class.Audio Clus.Multi. Class.Pair Class.Audio Rerank.0-shot Class.
EmbeddingGemma 2577M76849.948.753.652.94.138.767.468.555.4
51249.848.653.452.74.338.867.468.855.0
25649.148.252.152.24.238.467.268.754.8
12847.346.349.050.44.530.866.967.155.1
larger_clap_general194M51235.041.118.947.96.680.651.966.814.9
jina-embed-v5-omni-nano986M76850.155.350.051.36.070.865.683.260.3
larger_clap_music_and_speech194M51234.740.718.647.47.780.652.165.613.2
clap-htsat-unfused153M51233.040.518.141.912.580.852.666.511.3
msclap-2023160M102434.042.518.742.215.279.753.675.412.6
clap-htsat-fused154M51233.541.317.641.222.781.152.061.313.2
MuQ-MuLan-large630M51228.234.011.738.84.333.151.985.412.6
wav2clip163M51226.631.19.537.46.031.853.668.910.8
msclap-2022196M102432.239.321.736.119.970.951.762.912.1
speecht5_multimodal298M76826.430.07.938.51.132.557.956.515.9
larger_clap_music194M51220.722.89.126.75.38.150.348.611.5

Table 10. Performance of top-ranking models under 1B parameters on MAEB.

On average, the model surpasses various audio-text specialists, but there is an interesting specialization divide between them. While EmbeddingGemma 2 does beat specialists like CLAP at non-speech tasks like clustering vehicle sounds (VehicleSoundClustering), these models still outperform in tasks like audio captioning (ClothoT2ARetrieval). In speech tasks, EmbeddingGemma 2 sets the state of the art even in tasks with multiple languages (FleursT2ARetrieval) and singing (JamAltLyricA2TRetrieval). But it sometimes struggles at zero-shot tasks with a paralinguistic focus, despite understanding the acoustics deep down. For example, it struggles tremendously at clustering speakers by gender (VoxPopuliGenderClustering: 0.40), but recovers quickly when doing the same task with a linear probe (IEMOCAPGender: 66.40). This indicates the model’s speech representations are dominated by semantics, desirably avoiding biases such as speaker demographic while retaining paralinguistic and acoustic attributes for extraction when needed.

The model’s excellence on spoken language is further underscored on MSEB. Even when using ASR and text embeddings, EmbeddingGemma 2 greatly outperforms EmbeddingGemma 1, indicating the strength persists throughout modalities (Table 11). Encoding audio natively yields another even larger improvement. Instead of suffering from transcription errors from accents, homophones, or artifacts, native audio inputs preserve these signals and let the model take them into account. As a result, even the 256-dimensional embeddings greatly outperform the full-dimension ASR embeddings.

Model NameAudio ModeParametersDim.OverallIn-LangCross-Lang
EmbeddingGemma 2Native Audio577M76869.5471.6367.68
51269.1871.2067.38
25666.7668.8664.89
12856.7160.3053.49
EmbeddingGemma 2ASR271M76857.5962.1453.52
EmbeddingGemma 1ASR308M76850.9561.1041.87
Gemini Embedding 2Native Audio–307273.9975.5872.56
Gemini Embedding 2ASR–307270.4073.5867.55

Table 11. Results with ASR and native audio on MSEB(Retrieval) (MRR@10).

Future Work

Our work points to several concrete directions for future research. EmbeddingGemma 2 supports long sequences and variable vision token budgets, but further extending context length, for example via content-aware methods, could enable direct indexing of full-length media, improve performance on long-horizon tasks, and support novel agentic memory systems. Innovations such as encoder-free architectures could enable both quality improvements and further reductions in memory footprint and latency. Changes in training paradigms like the introduction of end-to-end RAG training present another opportunity to further optimize embedding models for downstream use. As interleaved all-to-all multimodal search becomes more prevalent, evaluation frameworks that represent novel use cases will become more important.

At the same time, EmbeddingGemma 2’s utility extends far beyond single-vector retrieval, serving as a general-purpose multimodal encoder. Operating on its final embeddings or unpooled token representations, the model can be adapted into (or may even excel zero-shot as) a general decision model [61, 62, 63, 64], model router [65], reward model [66], reranker [67], speech-to-text model [68, 69], late interaction retriever [70, 71, 72], and much more.

Conclusion

EmbeddingGemma 2 builds on its predecessor with a smaller, stronger backbone and native multimodal capabilities. Our approach introduces improvements like simplified per-layer embeddings and a multi-stage curriculum with both modality and objective progression. This enables EmbeddingGemma 2 to set a new Pareto frontier across text, vision, and audio benchmarks, often surpassing specialist and general-purpose models several times larger. With native support for weight quantization, embedding truncation, and modular encoder loading, EmbeddingGemma 2 makes high-quality multimodal embeddings practical in resource-constrained and on-device settings. We release EmbeddingGemma 2 checkpoints to the community to accelerate research and development, for both off-the-shelf use and as a foundational multimodal encoder.

Task Instructions

Use CaseQuery Task InstructionTarget Task Instruction
Web / Document Search`task: search resultquery: {text}``title: {title}text: {text}`
Question Answering`task: question answeringquery: {text}``title: {title}text: {text}`
Fact Checking`task: fact checkingquery: {text}``title: {title}text: {text}`
Code Search`task: code retrievalquery: {text}``title: {title}text: {text}`
Classification`task: classificationquery: {text}`Same as query
Clustering`task: clusteringquery: {text}`Same as query
Semantic Similarity`task: sentence similarityquery: {text}`Same as query

Table 12. Task instruction prefixes used for EmbeddingGemma 2. Asymmetric tasks (top) format queries and targets differently, using title: none when no document title is available. Symmetric tasks (bottom) apply the same prefix to all inputs.

Multilingual and English Leaderboards

Model NameParametersDim.Mean (Task)Mean (Type)Bitext MiningClass.Clus.Inst. RetrievalMulti. Class.Pair Class.Rerank.RetrievalSTS
EmbeddingGemma 2271M76861.453.865.763.050.42.925.278.862.960.874.7
51261.253.765.462.751.02.525.178.862.860.274.4
25660.453.063.661.951.12.024.778.762.058.774.1
12857.950.759.459.749.00.823.378.559.952.373.6
harrier-oss-v1-0.6b596M102469.059.082.873.954.00.826.482.163.270.877.1
harrier-oss-v1-270m268M64066.556.981.570.852.5-0.524.080.161.966.475.4
Qwen3-Embedding-0.6B596M102464.356.072.266.852.35.124.680.861.464.676.2
jina-embed-v5-text-nano212M76865.557.767.769.252.70.041.381.964.663.378.2
multilingual-e5-large-instruct560M102463.255.180.164.950.7-0.423.080.862.657.176.8
F2LLM-v2-0.6B596M102462.755.070.364.156.61.425.279.065.159.374.2
embeddinggemma-300m308M76861.154.364.460.951.25.624.781.463.262.574.7
BidirLM-0.6B-Embedding596M102460.452.967.662.549.51.224.178.462.759.170.7
F2LLM-v2-330M334M89660.853.369.861.754.7-1.724.677.863.356.572.7
jina-embeddings-v3572M102458.450.765.358.845.6-1.318.479.357.155.877.1
nomic-embed-text-v2-moe475M76857.649.660.760.444.5-3.919.676.360.357.271.0

Table 13. Performance of top-ranking models under 1B parameters on MTEB(Multilingual, v2).

Model NameParametersDim.Mean (Task)Mean (Type)Class.Clus.Pair Class.Rerank.RetrievalSTSSumm.
EmbeddingGemma 2271M76868.563.988.855.486.446.351.783.235.1
51268.463.988.755.386.446.351.683.335.6
25667.863.388.555.586.145.949.483.134.8
12865.761.587.654.385.345.743.782.531.4
jina-embed-v5-text-nano212M76871.165.290.153.584.749.258.888.331.9
Qwen3-Embedding-0.6B596M102470.564.784.654.184.448.261.886.633.4
Yuan-Embedding-2.0-en596M102471.765.284.451.782.053.370.784.929.7
embeddinggemma-300m308M76869.765.187.656.687.347.455.783.637.6
Nano-Em-1.0-6B-v2.1596M102467.964.187.655.489.949.546.685.434.2
Nomic-Embed-3-350M-V1354M102469.363.388.455.784.047.855.184.627.4
Giga-Embeddings-instruct484M102469.564.490.055.284.847.956.382.334.1
GIST-large-Embedding-v0335M102466.362.078.948.886.748.854.584.431.5
UltraBERT-Embed-150M151M76867.262.881.749.785.347.557.283.034.9
mxbai-embed-large-v1335M102466.362.079.147.587.248.155.484.432.6
UAE-Large-V1335M102466.461.879.147.987.248.455.984.430.1

Table 14. Performance of top-ranking models under 1B parameters on MTEB(English, v2).

Full Results

Task NamePerformance
AILAStatutes32.44
AfriSentiClassification45.67
AlloProfClusteringS2S.v252.31
AlloprofReranking76.92
AmazonCounterfactualClassification85.83
ArXivHierarchicalClusteringP2P63.42
ArXivHierarchicalClusteringS2S60.59
ArguAna59.88
ArmenianParaphrasePC93.36
BUCC.v298.50
BelebeleRetrieval71.91
BibleNLPBitextMining12.27
BigPatentClustering.v233.31
BiorxivClusteringP2P.v248.83
BornholmBitextMining49.41
BrazilianToxicTweetsClassification28.35
BulgarianStoreReviewSentimentClassfication68.68
CEDRClassification50.38
CLSClusteringP2P.v239.91
CSFDSKMovieReviewSentimentClassification35.08
CTKFactsNLI78.55
CataloniaTweetClassification51.66
Core17InstructionRetrieval4.34
CovidRetrieval77.22
CyrillicTurkicLangClassification68.49
CzechProductReviewSentimentClassification58.35
DBpediaClassification91.83
DalajClassification50.30
DiaBlaBitextMining81.71
EstonianValenceClassification38.59
FaroeseSTS71.58
FilipinoShopeeReviewsClassification37.17
FinParaSTS24.57
FinancialPhrasebankClassification83.89
FloresBitextMining54.89
GermanSTSBenchmark83.14
GreekLegalCodeClassification35.42
GujaratiNewsClassification87.44
HALClusteringS2S.v228.35
HagridRetrieval99.10
IN22GenBitextMining73.59
IndicCrosslingualSTS42.92
IndicGenBenchFloresBitextMining86.74
IndicLangClassification85.49
IndonesianIdClickbaitClassification64.53
IsiZuluNewsClassification24.76
ItaCaseholdClassification69.09
JSICK84.23
KorHateSpeechMLClassification11.58
KorSarcasmClassification59.80
KurdishSentimentClassification64.04
LEMBPasskeyRetrieval54.25
LegalBenchCorporateLobbying93.02
MIRACLRetrievalHardNegatives66.11
MLQARetrieval76.82
MacedonianTweetSentimentClassification57.97
MalteseNewsClassification30.94
MasakhaNEWSClassification75.70
MasakhaNEWSClusteringS2S46.02
MassiveIntentClassification61.52
MedrxivClusteringP2P.v243.13
MultiEURLEXMultilabelClassification4.78
MultiHateClassification57.64
NTREXBitextMining73.97
NepaliNewsClassification95.46
News21InstructionRetrieval5.10
NollySentiBitextMining37.21
NordicLangClassification65.88
NorwegianCourtsBitextMining92.98
NusaParagraphEmotionClassification45.05
NusaTranslationBitextMining71.37
NusaX-senti68.86
NusaXBitextMining67.19
OdiaNewsClassification72.81
OpusparcusPC92.38
PAC67.16
PawsXPairClassification56.38
PlscClusteringP2P.v272.46
PoemSentimentClassification57.04
PolEmo2.0-OUT58.10
PpcPC88.46
PunjabiNewsClassification82.87
RTE386.63
Robust04InstructionRetrieval-0.75
RomaniBibleClustering40.46
RuBQReRanking71.05
SCIDOCS18.14
SIB200ClusteringS2S26.40
SICK-R82.19
STS1277.82
STS1386.48
STS1482.75
STS1588.45
STS1785.79
STS22.v269.91
STSB81.34
STSBenchmark87.48
STSES81.24
ScalaClassification50.12
SemRel24STS65.43
SentimentAnalysisHindi68.23
SinhalaNewsClassification69.54
SiswatiNewsClassification55.88
SlovakMovieReviewSentimentClassification78.44
SpartQA14.56
SprintDuplicateQuestions96.50
StackExchangeClustering.v291.55
StackOverflowQA89.08
StatcanDialogueDatasetRetrieval45.01
SwahiliNewsClassification61.29
SwednClusteringP2P42.72
SwissJudgementClassification57.68
T2Reranking66.64
TERRA57.02
TRECCOVID79.33
Tatoeba54.18
TempReasonL11.32
ToxicConversationsClassification91.03
TswanaNewsClassification32.61
TweetTopicSingleClassification73.01
TwitterHjerneRetrieval68.69
TwitterURLCorpus86.83
VoyageMMarcoReranking61.78
WebLINXCandidatesReranking12.79
WikiCitiesClustering89.59
WikiClusteringP2P.v227.05
WikipediaRerankingMultilingual88.38
WikipediaRetrievalMultilingual89.47
WinoGrande57.93
XNLI75.95
indonli54.59

Table 15. Results of EmbeddingGemma 2 on MTEB (Multilingual, v2).

Task NamePerformanceTask NamePerformance
AmazonCounterfactualClassification91.10AppsRetrieval89.19
ArXivHierarchicalClusteringP2P63.66COIRCodeSearchNetRetrieval86.89
ArXivHierarchicalClusteringS2S60.59CodeEditSearchRetrieval80.52
ArguAna59.95CodeFeedbackMT85.60
AskUbuntuDupQuestions60.61CodeFeedbackST82.16
BIOSSES85.33CodeSearchNetCCRetrieval93.84
Banking77Classification92.27CodeSearchNetRetrieval90.85
BiorxivClusteringP2P.v248.85CodeTransOceanContest84.55
CQADupstackGamingRetrieval56.90CodeTransOceanDL35.60
CQADupstackUnixRetrieval42.16CosQA53.34
ClimateFEVERHardNegatives23.08StackOverflowQA89.08
FEVERHardNegatives81.65SyntheticText2SQL72.47
FiQA201832.47
HotpotQAHardNegatives78.25
ImdbClassification91.03
MTOPDomainClassification98.74
MassiveIntentClassification88.41
MassiveScenarioClassification91.66
MedrxivClusteringP2P.v242.86
MedrxivClusteringS2S.v239.54
MindSmallReranking32.05
SCIDOCS18.16
SICK-R82.20
STS1277.82
STS1386.48
STS1482.75
STS1588.46
STS1791.08
STS22.v267.42
STSBenchmark87.48
SprintDuplicateQuestions96.50
StackExchangeClustering.v291.79
StackExchangeClusteringP2P.v248.23
SummEvalSummarization.v235.11
TRECCOVID79.33
Touche2020Retrieval.v344.70
ToxicConversationsClassification91.03
TweetSentimentExtractionClassification66.44
TwentyNewsgroupsClustering.v248.05
TwitterSemEval201575.83
TwitterURLCorpus86.83

Table 16. Results of EmbeddingGemma 2 on MTEB(English, v2) (left) and MTEB(Code) (right).

Task NamePerformancePerformance (MRR@10)
BeijingOpera81.37LanguageIn-LangCross-Lang
BirdCLEF16.00Arabic (Egypt) (ar-EG)75.5175.45
CREMADPairClassification52.50Arabic (Gulf) (ar-x-gulf)77.0277.61
CREMA_D19.67Arabic (Levant) (ar-x-levant)78.6078.47
CREMA_DClustering0.57Arabic (Maghreb) (ar-x-maghrebi)73.3172.54
ClothoT2ARetrieval10.71Bengali (Bangladesh) (bn-BD)79.1979.06
CommonLanguageAgeDetection17.59Bengali (India) (bn-IN)80.0180.17
CommonVoiceMini21T2ARetrieval93.26English (Australia) (en-AU)74.26-
FSD2019Kaggle38.72English (UK) (en-GB)68.29-
FleursT2ARetrieval97.32English (India) (en-IN)72.78-
GTZANAudioReranking68.48English (Philippines) (en-PH)68.94-
GTZANGenre58.00English (US) (en-US)72.30-
GigaSpeechT2ARetrieval82.03Finnish (Finland) (fi-FI)48.5548.40
IEMOCAPGender66.40Gujarati (India) (gu-IN)-62.58
JamAltArtistA2ARetrieval55.55Hindi (India) (hi-IN)-65.27
JamAltLyricA2TRetrieval64.01Indonesian (Indonesia) (id-ID)75.24-
MACST2ARetrieval6.36Japanese (Japan) (ja-JP)-67.09
MInDS1490.30Kannada (India) (kn-IN)-61.26
MridinghamTonic32.89Korean (South Korea) (ko-KR)62.4573.09
NMSQAPairClassification98.12Malayalam (India) (ml-IN)-61.63
RavdessZeroshot13.96Marathi (India) (mr-IN)-62.42
SIBFLEURS54.53Russian (Russia) (ru-RU)65.2480.79
SpeechCommandsZeroshotv0.0296.86Swahili (sw)65.35-
SpokenSQuADT2ARetrieval72.33Tamil (India) (ta-IN)-54.11
UrbanSound8KT2ARetrieval0.47Telugu (India) (te-IN)80.7469.85
VehicleSoundClustering11.31Urdu (India) (ur-IN)-59.24
VoxCelebSA45.17Urdu (Pakistan) (ur-PK)-56.81
VoxPopuliAccentPairClassification51.71
VoxPopuliGenderClustering0.40
VoxPopuliLanguageID99.60

Table 17. Results of EmbeddingGemma 2 on MAEB (left) and MSEB(Retrieval) (right).

Task NamePerformance
AROCocoOrder57.00
AROFlickrOrder56.12
AROVisualAttribution77.97
AROVisualRelation56.93
BLINKIT2IMultiChoice75.87
CIFAR100ZeroShot75.61
CIRRIT2IRetrieval22.12
CUB200I2IRetrieval78.84
CVBenchCount59.77
CVBenchDepth52.67
CVBenchDistance53.67
CVBenchRelation53.39
Country21120.83
Country211ZeroShot22.73
DTD77.50
EuroSAT85.90
FER2013ZeroShot61.80
FGVCAircraftZeroShot45.21
Fashion200kI2TRetrieval15.32
Food101ZeroShot81.58
GTSRB82.86
HatefulMemesI2TRetrieval78.21
ImageCoDe14.16
ImageNetDog15Clustering91.39
InfoSeekIT2TRetrieval13.87
NIGHTSI2IRetrieval26.16
OVENIT2TRetrieval16.00
OxfordPets93.32
OxfordPetsZeroShot90.81
PatchCamelyon69.34
RESISC4589.76
RP2kI2IRetrieval73.74
STS13VisualSTS77.31
STS15VisualSTS85.67
SUN39775.13
StanfordCarsZeroShot91.17
TinyImageNetClustering71.26
VQA2IT2TRetrieval16.60
VidoreDocVQARetrieval39.81
VidoreInfoVQARetrieval77.34
VidoreShiftProjectRetrieval65.31
VidoreSyntheticDocQAAIRetrieval87.76
VidoreTabfquadRetrieval91.33
VidoreTatdqaRetrieval42.26
VisualNewsI2TRetrieval16.94
VisualSTS-b-Multilingual75.59
VisualSTS17Multilingual77.95
WITT2IRetrieval65.31
WebQAT2ITRetrieval64.00
Winoground12.00
XM3600T2IRetrieval65.45

Table 18. Results of EmbeddingGemma 2 on MIEB(lite).

Task NamePerformance
A-OKVQA35.20
ActivityNetQA52.10
Breakfast35.57
CIRR40.10
Charades-STA25.58
ChartQA18.90
Country21118.50
DiDeMo52.49
DocVQA51.50
EDIS80.70
EgoSchema56.60
FashionIQ27.90
GQA61.40
HMDB5155.30
HatefulMemes51.40
ImageNet-1K65.00
ImageNet-A39.80
ImageNet-R87.40
InfographicsVQA22.70
K70054.70
MMLongBench-doc45.08
MMLongBench-page-fixed46.34
MSCOCO63.20
MSCOCO_i2t72.80
MSCOCO_t2i77.70
MSR-VTT50.90
MSVD67.46
MVBench43.80
MomentSeeker47.88
N24News38.10
NExTQA52.74
NIGHTS66.70
OK-VQA43.40
OVEN68.00
ObjectNet65.50
Place36535.30
QVHighlight78.21
RefCOCO82.90
RefCOCO-Matching72.40
SUN39767.50
ScienceQA26.90
SmthSmthV259.80
TextVQA74.10
UCF10175.90
VATEX48.06
VOC200775.20
ViDoRe_arxivqa70.56
ViDoRe_biomedical_lectures_v2_multilingual46.99
ViDoRe_docvqa36.60
ViDoRe_economics_reports_v2_multilingual53.81
ViDoRe_esg_reports_human_labeled_v278.22
ViDoRe_esg_reports_v2_multilingual54.25
ViDoRe_infovqa78.79
ViDoRe_shiftproject63.54
ViDoRe_syntheticDocQA_artificial_intelligence81.78
ViDoRe_syntheticDocQA_energy78.67
ViDoRe_syntheticDocQA_government_reports81.87
ViDoRe_syntheticDocQA_healthcare_industry87.60
ViDoRe_tabfquad91.62
ViDoRe_tatdqa34.75
ViDoSeek-doc79.50
ViDoSeek-page-fixed82.29
Video-MME37.89
VisDial76.50
VisRAG_ArxivQA68.14
VisRAG_ChartQA72.33
VisRAG_InfoVQA83.26
VisRAG_MP-DocVQA67.67
VisRAG_PlotQA55.52
VisRAG_SlideVQA89.08
Visual7W38.50
Visual7W-Pointing91.00
VisualNews_i2t60.00
VisualNews_t2i58.50
VizWiz46.40
WebQA89.70
Wiki-SS-NQ71.40
YouCook217.08

Table 19. Results of EmbeddingGemma 2 on MMEB-v2.

Contributions and Acknowledgements

Core Contributors (*: equal contributions)

Henrique Schechter Vera*

Sahil Dua*

Jovan Andonov

Sara Smoot

Sebastian Russo

Ivan Llanos

Max Matveev

Jae Hyeon Yoo

Kat Black

Contributors

Shih-Cheng Huang

Josiah Aklilu

Maarten Grootendorst

Omar Sansevierο

Chandramouli Amarnath

Ian Ballantyne

Zhe Li

Daniel Salz

Ibrahim Alabdulmohsin

Ilya Philippov

Kristen Quan

Glenn Cameron

Edouard Yvinec

Luciano Martins

Doug Reid

Sindhu Raghuram Panyam

Weiyi Wang

Hengxuan Ying

Madhuri Shanbhogue

Tanmaya Dabral

Shanfeng Zhang

Juyeong Ji

Renjie Wu

Joe Zou

Xu Chen

Yunan Zhang

Yu-Hui Chen

Matthew Soulanille

Vinod Mamillapalli

Yi-Chun Kuo

Wai Hon Law

Matt Kreileder

Alice Zheng

Suril Shah

Mogan Shieh

Pooja Consul

Gemini Embedding (alphabetical order)

Alberto Montes

Andreas Hess

Andres Sevilla

Babak Samari

Brendan McCloskey

Brian Potetz

Daniel Cer

Georgios Kathareios

Gustavo Hernández Ábrego

Hesen Zhang

Ji Ma

Jiaxing Wu

Keegan Mosley

Paul Suganthan

Rui Meng

Shen Gao

Shengcao Cao

Yi-Ting Chen

Yichang Chen

Zhongli Ding

Leadership

Kendra Byrne

Cormac Brick

Olivier Lacombe

Qin Yin

Tris Warkentin

Armand Joulin

Mojtaba Seyedhosseini

Tom Duerig

Acknowledgements

Anand Rao, Raushan Turganbay, Anton Vlasiuk, Tom Aarsen, Srinivasan (Cheenu) Venkatachary, Tomáš Ižo, Raia Hadsell, Zoubin Ghahramani

References

  1. [1]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. doi: 10.48550/ARXIV.2103.00020. URL https://doi.org/10.48550/arXiv.2103.00020.
  2. [2]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. CoRR, abs/2102.05918, 2021. doi: 10.48550/ARXIV.2102.05918. URL https://doi.org/10.48550/arXiv.2102.05918.
  3. [3]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025. URL https://arxiv.org/abs/2502.14786.
  4. [4]Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: Universal embeddings with multimodal large language models, 2024. URL https://arxiv.org/abs/2407.12580.
  5. [5]Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2Vec: Training vision-language models for massive multimodal embedding tasks, 2025. URL https://arxiv.org/abs/2410.05160.
  6. [6]Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-Embed: universal multimodal retrieval with multimodal LLMs, 2025. URL https://arxiv.org/abs/2411.02571.
  7. [7]Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. GME: Improving universal multimodal retrieval by multimodal LLMs, 2025. URL https://arxiv.org/abs/2412.16855.
  8. [8]Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, Feiyang Chen, Jiaxiang Wu, Tanmaya Dabral, Babak Samari, Kevin Poulet, Daniel Cer, Kaifeng Chen, Paul Suganthan, Hui Hui, Jovan Andonov, Philippe Schlattner, Jay Han, Iftekhar Naim, Wing Lowe, Vladimir Pchelin, Albert Yang, Yi-Ting Chen, Zhongli Ding, Grace Zhang, Georg Heigold, Yichang Chen, Antoine Reveillon, Brendan McCloskey, Wenlei Zhou, Dahun Kim, Rui Meng, Emma Wang, Jack Zheng, Halley Fede, Zhen Yang, Keegan Mosley, Brian Potetz, Sahil Dua, Henrique Schechter Vera, Shen Gao, Hesen Zhang, Andreas Hess, Hengxuan Ying, Alberto Montes, Karan Gill, Min Choi, Sebastian Russo, Anja Hauth, Jinhyuk Lee, Michael Boratko, Megan Barnes, Vikram Rao, Claudiu Musat, Cyril Allauzen, Ehsan Variani, Shankar Kumar, Tom Bagby, Junyi Jiao, Yang Gu, Tengxin Li, Ayush Agrawal, Roberto Santana, Dev Nath, Stephen Karunakaran, Shuoxuan Han, Lucia Loher, Alice Twu, Nidhi Vyas, Siddharth Bhai, Frank Palma Gomez, Wangyuan Zhang, Chaoren Liu, Jizheng Yang, Steve Qiu, Shijie Zhang, Sujay Kulkarni, Sascha Rothe, Sean Nakamoto, Raphael Hoffmann, Zach Gleicher, Yunhsuan Sung, Qin Yin, Tom Duerig, and Mojtaba Seyedhosseini. Gemini Embedding 2: A native multimodal embedding model from Gemini, 2026. URL https://arxiv.org/abs/2605.27295.
  9. [9]Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. Omni-Embed-Nemotron: A unified multimodal retrieval model for text, image, audio, and video, 2025. URL https://arxiv.org/abs/2510.03458.
  10. [10]Chenghao Xiao, Hou Pong Chan, Hao Zhang, Weiwen Xu, Mahani Aljunied, and Yu Rong. Scaling language-centric omnimodal representation learning, 2025. URL https://arxiv.org/abs/2510.11693.
  11. [11]Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, and Zhicheng Dou. e5-omni: Explicit cross-modal alignment for omni-modal embeddings, 2026. URL https://arxiv.org/abs/2601.03666.
  12. [12]Embedding Team. Ovis-Embedding: Pushing the frontiers of universal omni-modal embeddings, 2026. URL https://arxiv.org/abs/2609.25165.
  13. [13]Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607.02770.
  14. [14]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. doi: 10.48550/ARXIV.1706.03762. URL https://doi.org/10.48550/arXiv.1706.03762.
  15. [15]Gemma Team. Gemma 3n. https://deepmind.google/models/gemma/gemma-3n/, 2025.DOI
  16. [16]Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with Gaussian Error Linear Units. CoRR, abs/1606.08415, 2016. doi: 10.48550/ARXIV.1606.08415. URL https://doi.org/10.48550/arXiv.1606.08415.
  17. [17]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929.
  18. [18]Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition, 2020. URL https://arxiv.org/abs/2005.08100.
  19. [19]Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. Google USM: Scaling automatic speech recognition beyond 100 languages, 2023. URL https://arxiv.org/abs/2303.01037.
  20. [20]Fedor Moiseev, Gustavo Hernandez Abrego, Peter Dornbach, Imed Zitouni, Enrique Alfonseca, and Zhe Dong. SamToNe: Improving contrastive loss for dual encoder retrieval models with same tower negatives, 2023. URL https://arxiv.org/abs/2306.02516.
  21. [21]Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In International Conference on Machine Learning, 2008. doi: 10.1145/1390156.1390306. URL https://api.semanticscholar.org/CorpusID:207168334.
  22. [22]Xu Zhang, Felix X. Yu, Sanjiv Kumar, and Shih-Fu Chang. Learning spread-out local feature descriptors. CoRR, abs/1708.06320, 2017. URL http://arxiv.org/abs/1708.06320.
  23. [23]Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sansевiero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus Martins, Jinhyuk Lee, Mark Sherwood, Juyeong Ji, Renjie Wu, Jingxiao Zheng, Jyotinder Singh, Abheesht Sharma, Divyanshree Sreerpathihalli, Aashi Jain, Adham Elarabawy, AJ Co, Andreas Doumanoglou, Babak Samari, Ben Hora, Brian Potetz, Dahun Kim, Enrique Alfonsесa, Fedor Moiseev, Feng Han, Frank Palma Gomez, Gustavo Hernández Ábrego, Hesen Zhang, Hui Hui, Jay Han, Karan Gill, Ke Chen, Koert Chen, Madhuri Shanbhogue, Michael Boratko, Paul Suganthan, Sai Meher Karthik Duddu, Sandeep Mariserla, Setareh Ariаfar, Shanfeng Zhang, Shijie Zhang, Simon Baumgartner, Sonam Goenka, Steve Qiu, Tanmaya Dabral, Trevor Walker, Vikram Rao, Waleed Khawaja, Wenlei Zhou, Xiaoqi Ren, Ye Xia, Yichang Chen, Yi-Ting Chen, Zhe Dong, Zhongli Ding, Francesco Visin, Gaël Liu, Jiageng Zhang, Kathleen Kenealy, Michelle Gasbon, Ravin Kumar, Thomas Mesnard, Zach Gleicher, Cormac Brick, Olivier Lacombe, Adam Roberts, Qin Yin, Yunhsuan Sung, Raphael Hoffmann, Tris Warkentin, Armand Joulin, Tom Duerig, and Mojtaba Sedghi. EmbeddingGemma: Powerful and Lightweight Text Representations, 2025. URL https://arxiv.org/abs/2509.20354.
  24. [24]Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022. URL https://arxiv.org/abs/2203.02053.
  25. [25]Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models, 2020. URL https://arxiv.org/abs/2011.05864.
  26. [26]Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models, 2019. URL https://arxiv.org/abs/1907.12009.
  27. [27]Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. CoRR, abs/1909.00512, 2019. doi: 10.48550/ARXIV.1909.00512. URL https://doi.org/10.48550/arXiv.1909.00512.
  28. [28]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. CoRR, abs/2005.10242, 2020. doi: 10.48550/ARXIV.2005.10242. URL https://doi.org/10.48550/arXiv.2005.10242.
  29. [29]Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations, 2018. URL https://arxiv.org/abs/1702.01417.
  30. [30]Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer, Sadeep Jayasumana, Veeranjaneyulu Sadhanala, Wittawat Jitkrittum, Aditya Krishna Menon, Rob Fergus, and Sanjiv Kumar. EmbedDistill: A geometric knowledge distillation for information retrieval, 2023. URL https://arxiv.org/abs/2301.12005.
  31. [31]Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. NV-Retriever: Improving text embedding models with effective hard-negative mining, 2025. URL https://arxiv.org/abs/2407.15831.
  32. [32]Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: effective and efficient retrieval via lightweight late interaction, 2022. URL https://arxiv.org/abs/2112.01488.
  33. [33]Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30233–30249. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c32319f4868da7613d78af9993100e42-Paper-Conference.pdf.arxiv.org/abs/2205.13147
  34. [34]Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024. URL https://arxiv.org/abs/2212.03533.
  35. [35]Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https://arxiv.org/abs/2308.03281.
  36. [36]Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig. Gemini Embedding: Generalizable embeddings from Gemini, 2025. URL https://arxiv.org/abs/2503.07891.
  37. [37]Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders, 2024. URL https://arxiv.org/abs/2404.09591.
  38. [38]Chankyу Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: improved techniques for training LLMs as generalist embedding models, 2025. URL https://arxiv.org/abs/2405.17428.
  39. [39]Niklas Muenninghoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, 2025. URL https://arxiv.org/abs/2402.09906.
  40. [40]Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https://arxiv.org/abs/2203.05482.
  41. [41]Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. CoRR, abs/1803.05407, 2018. URL http://arxiv.org/abs/1803.05407.
  42. [42]Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannah Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, 2022. URL https://arxiv.org/abs/2109.01903.
  43. [43]Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2704–2713, 2018. doi: 10.1109/CVPR.2018.00286.DOI
  44. [44]Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy, 2017. URL https://arxiv.org/abs/1711.05852.
  45. [45]Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the design space of visually-conditioned language models, 2024. URL https://arxiv.org/abs/2402.07865.
  46. [46]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL https://arxiv.org/abs/2304.08485.
  47. [47]Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An embodied multimodal language model, 2023. URL https://arxiv.org/abs/2303.03378.
  48. [48]Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, and Shu Wu. UEmbed: Unified sparse and dense multimodal embeddings, 2026. URL https://arxiv.org/abs/2608.02583.
  49. [49]Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, and Han Xiao. jina-embeddings-v5-omni: Geometry-preserving embeddings via locked aligned towers, 2026. URL https://arxiv.org/abs/2605.08384.
  50. [50]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs, 2024. URL https://arxiv.org/abs/2406.16860.
  51. [51]Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: Methods, analysis & insights from multimodal LLM pre-training, 2024. URL https://arxiv.org/abs/2403.09611.
  52. [52]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. URL https://arxiv.org/abs/2301.12597.
  53. [53]Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, and Davide Modolo. Multi-modal knowledge preserving adapter for embedding backward compatibility, 2026. URL https://arxiv.org/abs/2609.16875.
  54. [54]Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John X. Morris. Harnessing the universal geometry of embeddings, 2026. URL https://arxiv.org/abs/2505.12540.
  55. [55]Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. MMTEB: massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595, 2025. doi: 10.48550/arXiv.2502.13595. URL https://arxiv.org/abs/2502.13595.
  56. [57]Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. VLM2Vec-V2: Advancing multimodal embedding for videos, images, and visual documents, 2025. URL https://arxiv.org/abs/2507.04590.
  57. [58]Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muenninghoff. MIEB: Massive image embedding benchmark, 2025. URL https://arxiv.org/abs/2504.10471.
  58. [59]Georg Heigold, Ehsan Variani, Tom Bagby, Cyril Allauzen, Ji Ma, Shankar Kumar, and Michael Riley. Massive sound embedding benchmark (MSEB), 2026. URL https://arxiv.org/abs/2602.07143.
  59. [60]Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muenninghoff, and Kenneth Enevoldsen. MAEB: Massive audio embedding benchmark, 2026. URL https://arxiv.org/abs/2602.16008.
  60. [61]Ying Fang, Zhifu Gao, Qing Gu, Jiajun He, Kangqi Jing, Xiangang Li, Yu Pu, and Wupeng Wang. Qwen-Audio-Jev: A unified interface for audio decisions, 2026. URL https://arxiv.org/abs/2610.21614.
  61. [62]Yinheng Li and Justin Wagle. LLM-as-Jev: LLMs are already Jev-style decision models – when and how to fine-tune them, 2026. URL https://arxiv.org/abs/2610.02076.
  62. [63]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. Releasing the inner Jev: A simple recipe for open decision models, 2026. URL https://arxiv.org/abs/2610.17205.
  63. [64]Diogo Almeida. Introducing system one models and jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026.
  64. [65]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data, 2025. URL https://arxiv.org/abs/2406.18665.
  65. [66]Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannah Hajishirzi. RewardBench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787.
  66. [67]Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval, 2024. URL https://arxiv.org/abs/2407.19669.
  67. [68]Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020. URL https://arxiv.org/abs/2006.11477.
  68. [69]Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, 2021. URL https://arxiv.org/abs/2108.06209.
  69. [70]Jianrui Zhang, Hyun Jung Lee, Sukanta Ganguly, Tae-Eui Kam, Donghyun Kim, and Yong Jae Lee. Your embedding model is SMARTer than you think, 2026. URL https://arxiv.org/abs/2605.24938.
  70. [71]Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT, 2020. URL https://arxiv.org/abs/2004.12832.
  71. [72]Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. ColPali: Efficient document retrieval with vision language models, 2025. URL https://arxiv.org/abs/2407.01449.

Paper details

Contents