UIBK-Embedding: Effective Text Retrieval by Unifying Dense and Late-Interaction Retrieval

Introduction

Neural retrievers encode documents offline and compare them with the query at search time, and two families of models dominate. Dense bi-encoders compress a text into a single vector and score a query–document pair with one cosine

[19, 32, 39, 23, 43, 1]. Late-interaction models keep one vector per token and score a pair with MaxSim, the sum over the query tokens of their best-matching document token [20, 35 a]. The two are usually presented as a choice between a small index and better generalisation [38].

The two families do not make the same mistakes. Figure 1 compares a dense model and a ColBERT model that we fine-tuned from the same backbone on the same data. The dense model is better where a long query has to be matched as a whole, as in argument retrieval, and the late-interaction model is better where a single entity or phrase decides, as in the open-domain question answering. Their scores are complementary. Systems exploit this complementarity today through hybrid retrieval: two models, two indexes, two searches, and a fusion step such as reciprocal rank fusion [8]. Models that return several representations at once [6, 12] save the second encoder, but every representation still has its own index and its own scoring function.

In this work we introduce UIBK-Embedding, a retriever that holds a dense head and a late-interaction head in one encoder and adds their scores inside a standard late-interaction index (Figure 2d). Specifically, we make the following contributions: (1) For the representation, we store the dense sentence vector as three extra vectors, the slots, in the list of token vectors, and give every vector a one-dimensional tag with opposite signs for tokens and slots. Inside MaxSim a token then never selects a slot and a slot never selects a token, so the unmodified MaxSim returns the token-level score plus a weighted dense cosine (§ 4.2). The index and the scorer need no change: the model is saved as a ColBERT model and runs in PLAID (Santhanam et al., [34]) through PyLate (Chaffin and Sourty, [5]). (2) For the architecture, the two heads share the first 18 layers of ModernBERT-base (Warner et al., [40]) and each has its own copy of the last four, which raises the score of each head on its own by about one point. The dense head uses the latent-attention pooling of NV-Embed (Lee et al., [21]), here on an encoder-only model (§ 4.1). (3) For the training, we distil a cross-encoder into the fused score, the score that is used at inference, in addition to the losses of each head (§ 4.3). Ablations show the contribution of the two heads, of the fused distillation and of the main training choices (§ 7). (4) UIBK-Embedding-v1 has 179M parameters and reaches 59.79 nDCG@10 on BEIR (Thakur et al., [38]) and 70.55 on NanoBEIR (Zeta Alpha, [45]), evaluated with the public MTEB implementation (Muennighoff et al., [30]). It is 2.57 points above LateOn (Sourty et al., [37]), the strongest model of its size, 2.10 points above it at equal document length, and ahead of it on all 15 datasets. On the decontaminated BEIR of Sourty et al. [37] it reaches 61.55, against 60.36 for LateOn, and is ahead on 8 of the 12 datasets. See Table 2 and Table 3 for details.

Model	Size	Arg.	CQA	C-FEV	DBP	FEV	FiQA	HQA	MSM	NFC	NQ	Quora	SCID	SciF	COV	Tou.	Avg.
Single-vector models, at most 250M parameters
bge-base-en	109M	61.49	41.35	31.61	42.01	88.35	40.76	72.35	41.26	35.54	53.98	88.78	21.30	73.26	76.45	26.54	53.00
modernbert-embed-base	149M	48.96	42.08	35.67	41.49	87.35	40.59	67.11	41.47	33.40	62.15	88.85	18.59	69.63	84.15	31.91	52.89
granite-embedding-english-r2	149M	59.21	50.00	35.82	39.60	88.04	46.32	67.36	32.14	37.49	58.22	87.84	24.95	75.80	70.56	22.90	53.08
gte-base-en-v1.5	137M	63.49	39.52	40.36	39.90	94.81	48.65	67.75	42.62	35.88	52.96	88.42	21.92	76.77	73.13	25.22	54.09
cadet-embed-base-v1	109M	56.16	40.80	36.69	45.26	89.33	40.89	73.76	42.56	37.71	59.32	87.59	19.09	74.68	80.55	30.42	54.32
snowflake-arctic-embed-m	109M	56.44	43.81	39.37	44.73	89.02	42.40	73.65	41.77	36.77	62.43	87.42	21.10	73.55	79.65	31.47	54.91
gte-modernbert-base	149M	74.56	42.64	45.90	41.39	93.97	49.54	70.39	39.93	34.32	56.10	88.56	20.44	76.41	75.75	17.97	55.19
jina-embeddings-v5-text-nano	212M	65.70	44.65	39.60	45.26	89.51	47.85	69.07	41.64	38.69	63.38	88.87	22.60	75.78	77.60	30.70	56.06
DenseOn	149M	54.65	46.89	37.49	44.65	90.69	53.86	74.51	43.58	39.03	59.25	89.31	22.35	75.95	82.33	28.43	56.20
Late-interaction models, at most 250M parameters
GTE-ModernColBERT-v1	149M	47.52	41.08	31.33	47.56	87.67	45.25	77.48	45.60	37.83	61.62	86.71	19.22	76.33	84.84	31.25	54.75
ColBERT-Zero	149M	52.81	41.41	35.90	47.43	90.52	42.50	79.45	45.95	37.21	61.82	85.19	19.84	76.33	78.27	36.24	55.39
LateOn	149M	50.52	47.36	39.67	45.99	92.02	53.12	79.98	45.67	37.79	63.91	89.66	21.90	76.61	83.60	30.52	57.22
UIBK-Embedding-v1 (ours)	179M	56.56	<u>48.55</u>	<u>44.89</u>	49.22	93.28	<u>53.73</u>	81.50	46.82	<u>38.79</u>	67.80	89.92	<u>23.60</u>	78.57	83.82	39.78	59.79

Model	Size	Arg.	DBP	FiQA	HQA	MSM	NFC	NQ	Quora	SCID	SciF	COV	Tou.	Avg.
answerai-colbert-small	33M	47.63	30.38	45.75	76.35	72.39	25.41	86.10	89.36	12.72	87.82	77.51	27.81	56.60
ColBERT-Zero	149M	54.49	32.95	46.55	77.84	74.21	26.59	91.13	88.29	14.22	89.48	75.32	40.93	59.33
LateOn (published)	149M	52.16	31.72	57.92	78.94	70.31	26.97	93.05	91.54	15.09	88.92	80.94	36.77	60.36
Qwen3-Embedding-0.6B	596M	48.40	25.30	49.10	62.20	63.60	25.80	88.30	90.00	15.30	85.50	87.90	31.80	56.10
jina-embeddings-v5-text-nano	212M	47.22	30.18	51.45	67.48	68.58	29.38	92.26	91.33	14.93	89.36	76.80	33.16	57.68
DenseOn	149M	40.02	28.79	55.87	73.65	68.91	28.54	92.13	91.09	14.71	85.38	82.45	31.03	57.71
pplx-embed-v1-0.6b	596M	43.66	28.44	55.17	73.50	71.87	27.99	91.63	91.54	15.44	89.02	83.68	30.05	58.50
LateOn (our run)	149M	46.90	31.42	57.71	79.11	69.72	27.17	91.63	91.53	15.09	89.92	81.00	35.06	59.69
UIBK-Embedding-v1 (ours)	179M	56.51	31.39	56.19	80.32	71.74	29.32	90.45	91.81	16.91	87.78	81.54	44.61	61.55

Related Work

Dense retrieval. Bi-encoders trained with contrastive learning [19, 32, 16] improved through hard-negative mining [44, 9], large weakly supervised pre-training [39, 23, 43, 31, 27] and distillation from cross-encoders [14, 33]. NV-Embed [21] replaces mean or [CLS] pooling with a latent-attention layer in the spirit of the Perceiver [17]; our dense head uses this pooling on an encoder-only model.

Late interaction. ColBERT [20] introduced MaxSim over token vectors; ColBERTv2 [35] added residual compression and cross-encoder distillation, and PLAID [34] made the search fast. Later models extend it to long contexts and many languages [18, 7, 22]. On ModernBERT [40], SmallReason-ColBERT [1], GTE-ModernColBERT [24], EXCISE [3], ColBERT-Zero [25] and LateOn [37] are the strongest models of their size, and LateOn is our reference.

Hybrid and unified retrievers. Combining lexical and dense signals is standard practice [26, 10, 8]. COIL [11] adds a [CLS] similarity to contextualised exact-match scores, stored in inverted lists. ColBERTER [15] retrieves with a single vector and refines with token vectors. UniFiER [36] learns dense and lexical views together, and BGE-M3 [6] and jina-embeddings-v4 [12] return dense and multi-vector outputs from one model. All of these keep separate scoring functions and index structures, or use the signals in separate stages. UIBK-Embedding differs in that both signals live in one list of vectors and are added by the standard MaxSim, so one existing index and one search serve both. For single-vector retrieval, Luan et al. [26] observe that a weighted sum of a sparse and a dense score is one inner product of concatenated vectors. MaxSim is a sum of maxima over many vectors, where concatenation does not apply; our contribution is the construction that makes the same additive hybrid hold there. BGE-M3 distils the sum of its own scores into each of its heads; we go the other way and distil an external cross-encoder into the sum.

Background

Let $q$ be a query and $d$ a document. All vectors below have unit norm.

Dense retrieval. An encoder maps every text to one vector, $e_q$ and $e_d$, and the score is their cosine,

Late interaction. The encoder returns one vector per token, $u_1,\ldots,u_m$ for the query and $u'_1,\ldots,u'_n$ for the document. The score matches every query token with its most similar document token [20]:

A multi-vector index such as PLAID [34] stores the token vectors of all documents and evaluates Eq. (2).

Hybrid retrieval. A hybrid system computes both scores and combines them, for example as $s_{\mathrm{tok}}+\lambda s_{\mathrm{den}}$ or by fusing the two rankings [8]. Both representations must then exist at search time, which means two encoders, or one encoder with two outputs, and in either case two indexes that are searched separately (Table 1).

	Enc.	Indexes	Searches	Sentence	Token
Dense bi-encoder	1	1	1	✓	–
Late interaction	1	1	1	–	✓
Fusion of two models	2	2	2	✓	✓
Multi-output encoder	1	2	2	✓	✓
UIBK-Embedding (ours)	1	1	1	✓	✓

Our goal is to compute $s_{\mathrm{tok} }+\lambda s_{\mathrm{den}}$ with one encoder and with the machinery of late interaction alone: one multi-vector index, one search, and Eq. (2) as the only scoring function.

UIBK-Embedding

Figure 3 shows the model: a text is encoded once, and the output is an ordinary list of 128-dimensional unit vectors.

One Trunk, Two Tops

The backbone is ModernBERT-base [40] with 22 layers. The embeddings and the first 18 layers form a trunk that both heads share. Each head then has its own copy of the last four layers: the token top is the original one, the dense top is a copy that is trained separately. For a text with tokens $x_1,\ldots,x_n$ the encoder returns two sequences of states,

where $E$ is the trunk and $T_{\mathrm{tok}}$, $T_{\mathrm{den}}$ are the two tops. The second top adds 20.1M parameters to the 149.0M of the backbone, and the cost of one pass grows from 22 to 26 layer evaluations.

Token head. As in ColBERT, a linear layer maps every state to 128 dimensions, $\hat{u}_t=W h_t^{\mathrm{tok}}/\lVert W h_t^{\mathrm{tok}}\rVert$.

Dense head. A latent-attention layer [21] lets every state of the dense top attend to 512 trainable latent vectors, followed by a feed-forward layer; the mean of the result over the tokens is the sentence vector $g \in\mathbb{R}^{768}$. In a controlled comparison with the same data, steps and seed, in which the two training-loss curves coincide, this pooling was 0.8 NanoBEIR points better than [CLS] pooling.

One List of Vectors, One MaxSim

A late-interaction index stores 128-dimensional vectors, and its only scoring function is

We want this function to return a token-level score plus a dense score. Two things are in the way: the sentence vector has 768 dimensions, and a query token must not be matched with the sentence vector of a document.

Slots. A fixed random matrix $R \in\mathbb{R}^{381 \times768}$ with orthonormal rows projects $g$ to $3 \times127$ dimensions, which are cut into three slices and normalised:

The sum of the three slice cosines of two texts approximates the cosine of their sentence vectors. The token vectors are reduced to 127 dimensions by a fixed matrix $R' \in\mathbb{R}^{127 \times128}$ with orthonormal rows and normalised again, $u_t=\operatorname{norm}(R'\hat{u}_t)$.

Tag. Every vector gets one more coordinate, placed first:

All vectors have unit norm. The inner products are

We use $\beta=0.9$ and $\delta=0.225$. A pair of the same kind then gets a positive constant (0.1 for tokens, 0.775 for slots) and a mixed pair a negative one (−0.28). Three properties follow from Eq. (6).

(i) A slot never selects a token. A slot–slot product is at least $(1-\delta)-\delta=0.55$, and a slot–token product is at most $-\sqrt{(1-\beta)(1-\delta)}+\sqrt{\beta\delta}=0.17$. This holds for every input.

(ii) A token never selects a slot if its best token match has a cosine above 0.08: the token–token product is then above $0.1+0.9\cdot0.08=0.17$, the largest value a token–slot product can take. This bound assumes the worst case $u_t^{\mathsf{T}}g_q=1$. The two vectors come from unrelated 127-dimensional spaces, where a typical product has magnitude $1/\sqrt{127}\approx0.09$; for $|u^{\mathsf{T}}g|\leq0.3$ the condition relaxes to a best token cosine above $-0.27$.

(iii) A slot selects its own slot whenever $g_s^{\prime} g_s^{\prime} \geq g_s^{\prime} g_{s^{\prime}}^{\prime}$, for the other two slots $s^{\prime}$. These hold different coordinates of the sentence vector, so the right-hand side is near zero. The condition fails only when the sentence vectors of the two texts are unrelated, and the dense term is then floored near zero instead of turning negative.

In practice the tag separates the two kinds completely: on NanoBEIR, over all pairs of a query vector and a document, no query token selects a slot and no slot selects a token. The three properties are statements about exact inner products. We use the model with PLAID, which selects candidates from compressed vectors and re-scores them with MaxSim, and the benchmark results in this paper are obtained that way; we make no claim for indexes that transform the vectors in other ways.

The score. Inserting Eq. (6) into Eq. (4) gives

with $\lambda=3\delta/\beta=0.75$ (Figure 6 in the appendix). The constant $C(q)=(1-\beta)m+3(1-\delta)$, for a query of $m$ tokens, does not depend on the document, so the ranking is that of the term in parentheses. The weight of the dense score is thus a property of the stored vectors and can be chosen when the model is packaged, without training.

Practical points. The slots are written over the first three positions of every text, which hold the [CLS] token, the query or document marker and the prompt word, so the list is not longer than that of a ColBERT model. The dense head reads queries of up to 256 tokens, while only the first 64 query tokens take part in the token-level matching. Documents are read up to 2048 tokens.

Training

Stages. (1) Pre-training. We pre-train ModernBERT-base with the latent-attention pooling contrastively for 33k steps with a batch of 16,384 pairs on a 331M-pair sample of the public pre-training collection of Sourty et al. [37]. (2) Single-head models. From this backbone we fine-tune a dense model (the weight average Wortsman et al. [42] of four runs) and a ColBERT model; they are the two models of Figure 1. (3) UIBK-Embedding. The two encoders of stage 2 are fine-tunes of one backbone, so a hybrid model can be initialised from their weight average. It is trained with both heads for three epochs; its last four layers are then duplicated into the two tops, and the two-top model is trained for one final epoch.

Loss. For a query with its positive and 14 mined hard negatives, let $s^{\mathrm{tok}}$, $s^{\mathrm{den}}$ and $s^{\mathrm{fus}}$ be the token-level, dense and fused (Eq. (7)) scores, and $y$ the scores of a cross-encoder teacher (Mixedbread, [28]). The loss is

$\mathcal{L}_{\mathrm{NCE}}$ is the contrastive loss over all documents of the batch (temperature 0.02). The KL terms compare the softmax of the student scores over the candidates of the query with that of the teacher (Hinton et al. [13]; ); Hofstätter et al. [14]). $\mathcal{L}_{\mathrm{anchor}}$ is one minus the cosine between the vectors of each head and those of the single-head model of stage 2, for queries and documents ($\gamma=10$). The dense score $s^{\mathrm{den}}$ is the mean cosine of the three slots, the dense term of Eq. (7), so training and inference use the same quantity. The term $\operatorname{KL}(y\mathbin{\|}s^{\mathrm{fus}})$ is computed on the score that is used at inference; it is the only term in which the two heads appear together.

Training Data

Supervised collection. We use the English fine-tuning collection of Sourty et al. [37]: 1.56M queries from MS MARCO (Bajaj et al. [4]), NQ, HotpotQA, FEVER, FiQA, SQuAD v2, TriviaQA and MIRACL, each with mined negatives and the scores of the cross-encoder.

Re-mined negatives. Hard negatives are most useful when they are hard for the model that is trained. For every query we retrieve five more candidates with our own dense model, score them with the cross-encoder and keep the four hardest whose score is at least one logit below that of the positive. This adds 3.59M negatives, and every query then has 14 negatives with teacher scores.

Auxiliary pairs. The supervised collection consists of questions over Wikipedia and web passages. To cover duplicate questions, community forums and scientific literature as well, we mine 412k pairs from the pre-training collection: Quora duplicate questions (35k), StackExchange duplicates, question–answer and title–body pairs (246k), and S2ORC citation and title–abstract pairs (131k). Negatives come from our dense model with the positive-aware filter of de Souza P. Moreira et al. [9]: candidates that score above 95% of the positive are dropped. These sources are in-domain for Quora, CQADupStack, SCIDOCS and SciFact: the BEIR test queries are removed, but the documents come from the same sites.

Long queries. The supervised queries are short questions, but a query can also be a whole paragraph, as in argument retrieval. We add 152k pairs whose queries are long, also from the pre-training collection: Reddit posts with a reply (39k), S2ORC abstracts with the abstract of a cited paper (57k) and StackExchange question bodies with a related body (56k). None of them comes from the sources of ArguAna or Touché. We train with queries of up to 256 tokens, which the dense head reads in full.

Experimental Setup

Benchmarks. We evaluate on the 15 datasets of BEIR (Thakur et al., [38]) and on NanoBEIR (Zeta Alpha, [45]), which consists of 50-query samples of 13 BEIR datasets, and report nDCG@10. All BEIR and NanoBEIR numbers of UIBK-Embedding-v1 are produced by MTEB (Muenighoff et al., [30]) (versions 2.21.6 and 2.24.8, which give the same scores) with its standard late-interaction wrapper and a PLAID index with default settings; no part of the evaluation is specific to our model. Queries are encoded up to 256 tokens and documents up to 2048, in bfloat16.

Baselines. The numbers of all other models are taken from the official MTEB results repository (MTEB contributors, [29]), which holds complete BEIR results for 156 models and complete NanoBEIR results for 20. We compare in detail with the models of the same size class, at most 250M parameters, and list larger models in Appendix § 11.

Training. The last stage takes 4.3 hours on four H100 GPUs and the pre-training of stage 1 about 75 hours; Appendix § 10 lists all settings.

Results

BEIR. Table 2 gives the results on every dataset. UIBK-Embedding-v1 reaches 59.79, against 57.22 for LateOn, the best model of its size class, 56.20 for the dense DenseOn and 56.06 for jina-embeddings-v5-text-nano. Among the 13 models of the table it is best on 7 of the 15 datasets and second on 5 more. It is also ahead of the larger models of up to 600M parameters, such as mLateOn (57.56, 307M), jina-embeddings-v5-textsmall (56.67, 596M) and Qwen3-Embedding-0.6B (55.52); Table 7 in the appendix lists them. Among all 156 models with complete BEIR results it ranks tenth, and the nine models above it have between 1.5B and 7.6B parameters, at least eight times more. On the ten retrieval tasks of MTEB(eng, v2), which are derived from BEIR, it reaches 62.47, the best of the 45 models with at most 250M parameters and 13th of the 134 models with all ten tasks.

Model	Size	Arg.	CQA	C-FEV	DBP	FEV	FiQA	HQA	MSM	NFC	NQ	Quora	SCID	SciF	COV	Tou.	Avg.
Single-vector models, at most 250M parameters
bge-base-en	109M	61.49	41.35	31.61	42.01	88.35	40.76	72.35	41.26	35.54	53.98	88.78	21.30	73.26	76.45	26.54	53.00
modernbert-embed-base	149M	48.96	42.08	35.67	41.49	87.35	40.59	67.11	41.47	33.40	62.15	88.85	18.59	69.63	84.15	31.91	52.89
granite-embedding-english-r2	149M	59.21	50.00	35.82	39.60	88.04	46.32	67.36	32.14	37.49	58.22	87.84	24.95	75.80	70.56	22.90	53.08
gte-base-en-v1.5	137M	63.49	39.52	40.36	39.90	94.81	48.65	67.75	42.62	35.88	52.96	88.42	21.92	76.77	73.13	25.22	54.09
cadet-embed-base-v1	109M	56.16	40.80	36.69	45.26	89.33	40.89	73.76	42.56	37.71	59.32	87.59	19.09	74.68	80.55	30.42	54.32
snowflake-arctic-embed-m	109M	56.44	43.81	39.37	44.73	89.02	42.40	73.65	41.77	36.77	62.43	87.42	21.10	73.55	79.65	31.47	54.91
gte-modernbert-base	149M	74.56	42.64	45.90	41.39	93.97	49.54	70.39	39.93	34.32	56.10	88.56	20.44	76.41	75.75	17.97	55.19
jina-embeddings-v5-text-nano	212M	65.70	44.65	39.60	45.26	89.51	47.85	69.07	41.64	38.69	63.38	88.87	22.60	75.78	77.60	30.70	56.06
DenseOn	149M	54.65	46.89	37.49	44.65	90.69	53.86	74.51	43.58	39.03	59.25	89.31	22.35	75.95	82.33	28.43	56.20
Late-interaction models, at most 250M parameters
GTE-ModernColBERT-v1	149M	47.52	41.08	31.33	47.56	87.67	45.25	77.48	45.60	37.83	61.62	86.71	19.22	76.33	84.84	31.25	54.75
ColBERT-Zero	149M	52.81	41.41	35.90	47.43	90.52	42.50	79.45	45.95	37.21	61.82	85.19	19.84	76.33	78.27	36.24	55.39
LateOn	149M	50.52	47.36	39.67	45.99	92.02	53.12	79.98	45.67	37.79	63.91	89.66	21.90	76.61	83.60	30.52	57.22
UIBK-Embedding-v1 (ours)	179M	56.56	48.55	44.89	49.22	93.28	53.73	81.50	46.82	38.79	67.80	89.92	23.60	78.57	83.82	39.78	59.79
Larger models, for reference
snowflake-arctic-embed-l	334M	59.09	46.97	39.33	45.97	88.22	44.71	75.18	41.68	37.65	63.11	87.41	21.36	73.82	80.72	34.52	55.98
Qwen3-Embedding-0.6B	596M	70.97	46.03	42.11	39.48	88.15	46.61	65.74	37.99	36.71	53.46	87.78	24.41	69.72	90.52	33.18	55.52
jina-embeddings-v5-text-small	596M	65.07	46.72	41.50	44.38	89.99	49.63	69.76	42.09	39.81	64.04	89.09	23.04	76.53	78.49	29.89	56.67
mDenseOn	307M	55.31	45.70	41.51	43.61	91.46	51.91	75.28	43.08	37.34	63.49	88.91	21.51	74.88	81.63	34.91	56.70
mLateOn	307M	59.81	44.67	39.84	48.39	93.89	49.93	82.24	44.90	37.63	65.09	88.39	20.26	76.35	82.12	29.94	57.56
Qwen3-Embedding-4B	4.0B	75.64	50.28	47.43	48.23	91.60	62.65	74.72	42.72	41.10	63.13	88.06	31.44	78.33	92.92	35.41	61.58
Qwen3-Embedding-8B	7.6B	76.85	52.94	47.36	49.68	91.88	64.57	76.78	43.60	41.45	65.25	88.90	32.74	78.46	94.99	35.93	62.76

Every dataset improves. UIBK-Embedding-v1 is ahead of LateOn on all 15 datasets (Figure 4). The margins range from 0.2 points on TREC-COVID and Quora to 9.3 on Touché; the next largest gains are on ArguAna (+6.0), Climate-FEVER (+5.2), NQ (+3.9) and DBPedia (+3.2).

Not an effect of document length. LateOn’s reported numbers use documents of up to 300 tokens. With 2048 tokens LateOn reaches 57.69 in our evaluation (diamonds in Figure 4), so the gap at equal document length is 2.10 points.

Decontaminated BEIR. To verify that the result does not rest on overlap between training data and the benchmark, we evaluate on the decontaminated BEIR of Sourty et al. [37]: 12 BEIR datasets from which the documents and queries that overlap with common training collections were removed. We evaluate with PyLate and a PLAID index, as MTEB does, and run LateOn under the same protocol (Table 3). UIBK-Embedding-v1 reaches 61.55. LateOn reaches 60.36 as published and 59.69 in our run, a difference that comes almost entirely from ArguAna, so the gain is between 1.2 and 1.9 points, and UIBK-Embedding-v1 is ahead on 8 of the 12 datasets.

NanoBEIR. On NanoBEIR (Table 4) UIBK-Embedding-v1 reaches 70.55, second among the 20 models with official results, behind inf-retriever-v1 (72.04, 7.1B parameters; Table 8 in the appendix). Among the models with at most 250M parameters it is first on 10 of the 13 datasets and 8.75 points above the next model. LateOn has no official NanoBEIR results; with exact MaxSim search it scores 67.50 in our evaluation, against 70.43 for UIBK-Embedding-v1.

Model	Size	Arg.	C-FEV	DBP	FEV	FiQA	HQA	MSM	NFC	NQ	Quora	SCID	SciF	Tou.	Avg.
At most 250M parameters
BM25	–	48.04	31.77	61.38	80.94	43.74	82.77	53.30	32.50	50.07	86.28	33.51	70.99	69.53	57.29
multilingual-e5-small	118M	44.54	30.64	60.54	83.49	43.53	81.01	62.09	82.88	68.62	97.28	34.38	72.46	49.54	58.23
F2LLM-v2-80M	80M	53.45	39.19	56.99	92.89	44.89	72.66	56.91	32.63	54.59	95.86	38.54	66.89	53.84	58.41
F2LLM-v2-160M	159M	57.07	41.45	57.75	92.13	48.74	73.56	57.59	33.50	60.33	95.57	40.59	67.76	53.11	59.93
bekko-embedding-v1-a8m	106M	54.72	31.18	60.34	87.38	51.48	78.52	58.50	34.13	63.17	93.51	39.53	76.67	55.92	60.39
bekko-embedding-v1-a25m	123M	60.73	26.06	63.17	85.19	53.39	80.91	61.46	34.46	69.10	94.24	39.46	78.17	57.04	61.80
UIBK-Embedding-v1 (ours)	179M	59.84	51.65	69.20	96.57	64.34	92.72	68.99	39.22	82.14	96.99	45.40	83.93	66.14	70.55
Larger models, for reference
F2LLM-v2-330M	334M	60.11	46.17	59.95	94.84	56.12	76.58	61.79	34.38	65.89	95.47	41.45	77.75	54.75	63.48
F2LLM-v2-0.6B	596M	58.16	48.31	61.76	95.28	58.23	78.97	69.12	37.77	70.45	96.09	42.15	75.61	51.08	64.84
F2LLM-v2-1.7B	1.7B	55.46	45.09	64.57	93.52	66.78	81.99	66.64	36.22	73.07	96.70	44.74	81.53	51.69	66.00
inf-retriever-v1-1.5b	1.5B	77.39	46.67	69.86	96.08	64.41	86.88	67.71	40.55	73.44	97.15	49.46	84.36	52.18	69.70
F2LLM-v2-4B	4.0B	62.20	47.33	67.57	96.78	66.69	82.58	72.64	38.93	76.65	97.28	49.48	84.33	54.24	68.90
F2LLM-v2-8B	7.6B	60.26	44.78	69.09	95.06	68.27	86.60	70.52	38.79	81.32	97.74	49.41	85.59	54.95	69.41
F2LLM-v2-14B	14.0B	62.29	45.64	70.30	95.43	68.63	88.09	72.75	41.20	82.59	97.57	50.93	82.20	56.13	70.29
inf-retriever-v1	7.1B	73.21	43.25	73.45	97.59	69.72	90.95	70.06	47.10	78.31	97.22	53.33	86.32	56.03	72.04

LoTTE. On the test split of LoTTE (Santhanam et al., [35]), UIBK-Embedding-v1 reaches a success@5 of 83.77, the highest among the models with official MTEB results (Table 5): 1.92 points above DenseOn-unsupervised and 16.51 above ColBERTv2. LoTTE is drawn from StackExchange, which is part of the public pre-training collection used by our model and by the two LightOn models; Appendix § 11 gives the result per domain.

Model	Size	Success@5
UIBK-Embedding-v1 (ours)	179M	83.77
DenseOn-unsupervised	149M	81.85
ColBERT-Zero-unsupervised	149M	75.87
gtr-t5-base	110M	69.84
ColBERTv2	110M	67.26
BM25	–	52.93

LIMIT. LIMIT (Weller et al., [41]) was built to show what a single vector cannot represent. On LIMIT small, UIBK-Embedding-v1 reaches a recall@2 of 91.4, above GTE-ModernColBERT (83.5) and 37 points above the best single-vector model, which has 8B parameters (Table 9 in the appendix).

Model	Size	Recall@2
UIBK-Embedding-v1 (ours)	179M	91.4
GTE-ModernColBERT	149M	83.5
Promptriever (Llama3)	8B	54.3
GritLM	7B	38.4
Gemini Embed	–	33.7
Qwen3 Embed	8B	19.0

Cost. We measure cost on FiQA (57,638 documents, one H100) with the same input lengths for every model, queries of 32 and documents of 300 tokens (Table 11 in the appendix). UIBK-Embedding-v1 encodes 899 documents per second, against 986 for LateOn and 1,201 for DenseOn. One pass of our encoder is thus 1.7 times faster than the two passes of a LateOn and DenseOn hybrid (541 documents per second). Its index has the size of LateOn’s (580 against 574 MiB) and is 22% smaller than the two indexes of the hybrid (743 MiB). Search takes 7.8 ms per query in PLAID, against 4.4 ms for LateOn.

Model	Docs/s	Index (MiB)	Search (ms/query)
UIBK-Embedding-v1	899	580	7.8
LateOn	986	574	4.4
DenseOn	1,201	169	0.04
LateOn + DenseOn	541	743	4.4

caption: tbl:4: NanoBEIR, nDCG@10 on every dataset, for models with at most 250M parameters and official NanoBEIR results in the MTEB results repository. Bold and underlined as in Table 2. For the full table with larger models, see Table 8 in Appendix § 11.

:::

Ablation Study

Both heads are needed. In every setting of Figure 5 (left) the fused score is 0.6 to 1.9 points above the better head. The two tops raise each head on its own by about one point: 65.56 and 66.67 with shared layers, 66.42 and 67.68 with two tops. For the released model, exact MaxSim gives 69.00 for the dense head alone, 69.81 for the token head alone and 70.43 for both. The fused score has the best mean because each head alone falls far behind on some datasets: the token head reaches 51.66 on ArguAna against 67.39 for the dense head 34.26 on NFCorpus against 40.38 for the token head.

Weight of the dense score. We vary $\lambda$ through $\delta$, with $\beta= 0.9$ fixed. The fused score is flat between $\lambda= 0.75$ and $\lambda= 1$ for every checkpoint we tested (Figure 5, middle), so we use $\lambda= 0.75$. Appendix § 12 gives the bounds for other weights.

Distillation into the fused score. Adding the term $\operatorname{KL}(y\mathbin\Vert s^{\mathrm{fus}})$ to the losses of the two heads, with the data and the schedule unchanged, raises the BEIR average from 58.61 to 59.08. The gain is largest on Climate-FEVER (+1.6), ArguAna (+1.1) and NQ (+1.0), and with this term the model is ahead of LateOn on all 15 datasets.

Learning rate and seeds. The learning rate of the last stage matters: $7 \times10^{-5}$ is 0.54 points above $10^{-5}$ (Figure 5, right). Two seeds of one configuration differ by about 0.2 points (59.25 and 59.47; 59.62 and 59.45), which is the resolution of the comparisons in this section.

Conclusion

We presented UIBK-Embedding-v1, a retriever that holds a dense head and a late-interaction head in one encoder and adds their scores inside the standard MaxSim. The dense embedding is written into the list of token vectors as three slots, and a one-dimensional tag keeps slots and tokens from matching each other, so one model, one index and one search replace a hybrid pipeline. The cross-encoder teacher is distilled into the fused score, the score that ranks documents at inference.

With 179M parameters, UIBK-Embedding-v1 reaches 59.79 nDCG@10 on BEIR, 2.57 points above LateOn and ahead of it on all 15 datasets; only models with at least 1.5B parameters score higher. It stays ahead on the decontaminated benchmark and reaches 70.55 on NanoBEIR, 83.77 on LoTTE and a recall@2 of 91.4 on LIMIT small. Neither the slots nor the tag depends on the backbone or the language, so the construction carries over to multilingual and larger encoders.

Acknowledgments

The authors would like to acknowledge the financial support provided by the Austrian Research Agency (FFG) for the project “AI Enabled Sustainability Jurisdiction Demonstrator” (project No. 915229). The computational results presented in this work have been achieved using the MUSICA cluster, part of the Austrian Scientific Computing (ASC) infrastructure.

Limitations

The model is English only, and as a multi-vector model it needs a larger index than a dense model. Like the other models trained on this collection, it is fine-tuned on the training splits of five BEIR datasets, and its auxiliary pairs come from the same sites as four more. Cost is measured on one dataset, where search in PLAID is slower than for LateOn.

References

Training and Evaluation Details

Table 6 lists the settings of the released model.

Model
Backbone	ModernBERT-base, 22 layers
Shared layers / layers per top	18 / 4
Parameters	179.0M
    backbone (trunk and token top)	149.0M
    dense top	20.1M
    pooling and projection	9.9M
Latent attention	512 latents, 8 heads
Vector dimension	128 (1 tag + 127)
Slots	3
$\beta/ \delta/ \lambda$	0.9 / 0.225 / 0.75
Prompts	query: / document:
Pre-training (stage 1)
Steps / batch	33,000 / 16,384 pairs
Data	331M pairs
Temperature	0.02
Last stage
Steps / batch	16,597 / 128 queries
Learning rate / warm-up	$7 \times10^{-5}$ / 5%
Hard negatives per query	14
Query / document length	256 / 512 tokens
Temperature (contrastive, student)	0.02
Temperature (teacher)	0.1
Anchor weight $\gamma$	10
Hardware / time	4 H100 / 4.3 hours
Pre-training time	about 75 hours on 4 H100
Inference
Query / document length	256 / 2048 tokens
Query tokens in token matching	first 64
Precision	bfloat16

Model size. Of the 179.0M parameters, 149.0M are those of ModernBERT-base, which form the trunk and the token top. The dense top, a second copy of the last four layers, adds 20.1M, and the latent-attention pooling with the projection adds 9.9M. A stored vector has 128 dimensions, as in LateOn and ColBERT-Zero: one dimension holds the tag and 127 hold the token or slot vector, so an index entry has the same size as theirs.

Training. Pre-training is the expensive part: 33,000 steps with a batch of 16,384 pairs take about 75 hours on four H100 GPUs. The last stage, the final epoch of the two-top model, has 16,597 steps with a batch of 128 queries, each with its positive and 14 hard negatives, and takes 4.3 hours on the same hardware. It reads queries of up to 256 and documents of up to 512 tokens. The contrastive losses and the student distributions of the KL terms use a temperature of 0.02, the teacher distribution a temperature of 0.1. Inference. The model reads documents of up to 2048 tokens, four times the training length, and queries of up to 256 tokens, of which the first 64 take part in the token-level matching. The encoder runs in bfloat16.

Overlap with BEIR. Rows whose query equals a BEIR test or development query were removed from the training data: 37,391 rows of the pre-training sample and 228 rows of the fine-tuning collection. For the mined auxiliary pairs, a pair was removed if its query or its document equals a BEIR query.

Teachers. The cross-encoder teacher is mxbai-rerank-large-v2 (Mixedbread, [28]). Three of the four dense fine-tunes that are averaged into the dense model of stage 2 were trained with an additional ranking loss to a fine-tune of stella_en_400M_v5 (Zhang et al., [46]).

Additional Results

This appendix gives the full BEIR and NanoBEIR tables, the tables behind the LIMIT and cost results of § 6, the LoTTE result per domain and the effect of the document length.

Full BEIR and NanoBEIR tables. Tables 7 and 8 repeat Tables 2 and 4 with the larger models that have official results. On BEIR, the five models with 307M to 596M parameters reach between 55.52 and 57.56, and UIBK-Embedding-v1 is ahead of each of them on at least 12 of the 15 datasets. Qwen3-Embedding-4B and 8B, with 22 and 42 times as many parameters, reach 61.58 and 62.76; UIBK-Embedding-v1 is ahead of them on 8 and 7 datasets. On NanoBEIR, UIBK-Embedding-v1 is above every model of the table except inf-retriever-v1 with 7.1B parameters, including F2LLM-v2-14B, which has 78 times as many parameters.

LIMIT. LIMIT (Weller et al., [41]) asks for the two documents that are relevant to a query. Its queries and documents are simple by design; the task is hard because a model must be able to return many different pairs of documents. Table 9 gives recall@2 on LIMIT small, with the numbers that Weller et al. [41] report for other neural retrievers. The single-vector models, with up to 8B parameters, reach between 19.0 and 54.3. GTE-ModernColBERT, a late-interaction model with 149M parameters, reaches 83.5, and UIBK-Embedding-v1 reaches 91.4, 7.9 points more with vectors of the same dimension.

LoTTE per domain. LoTTE is scored by success@5, the share of queries with a relevant document among the first five results, and Table 5 gives the mean over its five domains and two query types. Table 10 gives the result of UIBK-Embedding-v1 for each of them. Success@5 is highest for Lifestyle (91.68 and 91.46) and Writing (89.64 and 89.60) and lowest for Science (71.15 and 69.86). The two query types are close in every domain, with means of 83.97 for search queries and 83.56 for forum queries.

Domain	Search	Forum
Writing	89.64	89.60
Recreation	85.82	87.36
Science	71.15	69.86
Technology	81.54	79.54
Lifestyle	91.68	91.46
Mean	83.97	83.56

Cost. Table 11 gives the measurements that § 6 summarises. Every model encodes the 57,638 documents of FiQA on one H100 GPU, with queries cut at 32 and documents at 300 tokens, so the three encoders read the same text. UIBK-Embedding-v1 writes 7.75M vectors and LateOn 7.70M, and the two PLAID indexes differ by 1% (580 and 574 MiB). Encoding is 9% slower than for LateOn, which reflects the four additional layers of the dense top. The last row is the hybrid of the two LightOn models: it encodes every document twice and keeps two indexes, 743 MiB together. Search times are means over the 648 test queries.

Document length. The model is trained with documents of up to 512 tokens and reads longer ones at inference. A two-top model scores 57.61 on BEIR with 512 tokens, 58.14 with 2048 and 58.04 with 8192: the longer input gives 0.53 points, and the gain is complete at 2048 tokens, which is the released setting.

Scoring and Bounds

Scoring. Figure 6 illustrates Eq. (7). The query and the document are each one list that holds the three slots and the token vectors, and MaxSim takes, for every query vector, its best match in the document list. The tag gives every product of a token with a slot a negative offset, so a query token finds its best match among the document tokens and a query slot in the document slot with the same number. The sum over the query vectors is then the late-interaction score plus the weighted dense score.

Bounds as a function of the weight. Table 12 gives the worst-case quantities of § 4.2 when the weight $\lambda$ of the dense score is changed through $\delta= \lambda\beta/3$ with $\beta= 0.9$. A larger $\delta$ moves norm from the tag of a slot to its content, so the smallest slot–slot product, $1 - 2\delta$, falls and the largest slot–token product, $\sqrt{\beta\delta} \sqrt{(1-\beta)(1-\delta)}$, rises. Property (i) holds as long as the first is above the second, which is the case up to $\lambda= 1$. For property (ii) the table gives the cosine that the best token match of a query token must exceed in the worst case.

$\lambda$	$\delta$	slot–slot min.	slot–token max.	(i) holds	(ii): cosine above
0.50	0.150	0.70	0.08	yes	−0.03
0.75	0.225	0.55	0.17	yes	0.08
1.00	0.300	0.40	0.26	yes	0.17
1.50	0.450	0.10	0.40	no	0.34
2.50	0.750	−0.50	0.66	no	0.63

The released value $\lambda= 0.75$ leaves a margin of $0.38$ between the two products of property (i) and asks for a best token cosine of only $0.08$. The fused score is flat for $\lambda$ between $0.75$ and $1$ (Figure 5, middle), the upper part of the range in which property (i) holds.

Auxiliary Training Pairs

Table 13 lists the nine sources of the pairs mined from the pre-training collection (§ 4.4). The first six give 412k pairs with short queries, 10 to 12 words on average: duplicate questions from Quora and StackExchange, question–answer and title–body pairs from StackExchange, and citation and title–abstract pairs from S2ORC. The last three give 152k pairs whose query is a whole paragraph, 92 to 173 words on average: a StackExchange question body with a related body, an S2ORC abstract with the abstract of a cited paper, and a Reddit post with a reply. The supervised queries are short questions, so these pairs are where the dense head learns to read long queries.

Source	Pairs	Query words
Quora, duplicate questions	35,206	10
StackExchange, duplicate questions	54,885	10
StackExchange, question–answer	91,973	10
StackExchange, title–body	98,831	10
S2ORC, citation	81,090	12
S2ORC, title–abstract	49,728	11
StackExchange, body–body	55,545	92
S2ORC, abstract–cited abstract	56,956	173
Reddit, post–reply	39,024	158