betaFeedback

WorksComputer Science

Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens

Abstract

Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.

CCS Concepts

• Information systems → Information retrieval.

ACM Reference Format:

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, and Heuiseok Lim. 2026. Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, Italy. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3799682.3840002

This work is licensed under a Creative Commons Attribution 4.0 International License.

CIKM ’26, November 07–11, 2026, Rome, Italy

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2539-5/2026/11

https://doi.org/10.1145/3799682.3840002

Introduction

Embedding models have become a fundamental component of modern retrieval systems, where queries and passages are mapped into a shared vector space for efficient semantic matching [12, 19]. Their effectiveness heavily depends on the ability to compress variable-length text into fixed-dimensional representations while preserving the critical semantic information relevant to retrieval across diverse domains. This compression is typically performed through pooling or token-selection strategies [14, 21], effectively forcing the model to funnel rich contextual dynamics into a narrow informational bottleneck. However, this process can also introduce representational interference: token-level information that is not directly useful for semantic matching, such as recurrent positional patterns, special-token artifacts, or other non-semantic cues, may be absorbed into the embedding representation and distort its true semantic content [6, 15, 25]. When this noise accumulates, it fundamentally limits the discriminative power of the resulting embeddings in downstream retrieval tasks.

This issue becomes particularly important in light of the attention sink phenomenon, where certain positions attract disproportionately large attention despite carrying limited semantic content [25, 28]. Since the final retrieval representation is obtained by pooling or selecting from contextualized token representations, such artificial attention bias can cause non-semantic information to become deeply entangled with content-bearing tokens [8, 33], thereby diluting the overall sequence representation. Standard retrieval fine-tuning addresses this issue only indirectly: contrastive objectives improve query–passage alignment at the macro embedding level, but they do not explicitly control how residual token-level information is represented or routed before pooling. Consequently, conventional retrieval optimization may still leave residual token-level information entangled with the embedding representation, leaving a gap in how we optimize dense retrievers.

In this work, we focus on how such residual information is handled during retrieval encoding by proposing a simple yet highly effective architectural intervention. We use trainable [DUMP] tokens as input prefixes during embedding model training. These prefix tokens participate in contextualization before the sequence is compressed into a single embedding, acting as dedicated semantic buffers. By absorbing attention associated with uninformative signals, they help reduce the influence of residual token-level information that would otherwise be mixed into the embedding representation. Crucially, the method does not require modifying the core model architecture, introducing auxiliary losses, or changing the retrieval objective, making it a highly scalable plug-and-play solution. Experiments on six diverse embedding models show that our method yields average relative improvements of 2.46% on standard retrieval benchmarks and 2.41% under long-context retrieval settings, demonstrating robust generalization. Further analyses show that the number of prefix tokens affects performance in a non-monotonic manner, while token-level inspection indicates that the added prefixes successfully change attention patterns associated with residual or uninformative tokens, effectively routing noise away from the final sequence representation.

RELATED WORK

Dense retrieval models encode queries and documents into fixed-dimensional representations and compare them in a shared embedding space to enable highly efficient semantic matching. In practice, these models typically obtain sequence-level embeddings through pooling or token-selection strategies, such as [CLS], mean, or last-token pooling [2, 8, 14]. Although these strategies provide a simple way to aggregate contextualized token representations, they also create a severe compression bottleneck, as heterogeneous token-level information is collapsed into a single retrieval vector, often causing fine-grained semantic details to be lost. Prior work has recognized this bottleneck in dense passage retrieval and proposed specialized pre-training strategies to improve the quality of the compressed representation [16, 26]. However, while these methods enhance the general representation space, they do not inherently resolve the internal routing of noisy or uninformative signals during the actual encoding phase.

Recent studies on Transformer attention dynamics further suggest that not all token-level information contributes equally to the final semantic representation. In particular, the attention sink phenomenon shows that certain tokens can attract disproportionately large attention scores despite carrying limited semantic content [10, 20, 25, 28]. Driven by the softmax operation that forces attention allocation [18, 34], such behavior can help stabilize autoregressive generation, but it inadvertently introduces representational interference in embedding models, where the pooled vector is strictly expected to reflect the pure semantic content of the input. Related observations have also been made in vision Transformers, where learnable register tokens are introduced to absorb high-norm outlier information and reduce non-semantic artifacts in feature maps [6]. These studies directly motivate our use of trainable [DUMP] prefix tokens to provide a dedicated structural pathway that absorbs residual or non-semantic information during retrieval encoding, yielding cleaner and more discriminative sequence representations.

Methods

We use [DUMP] tokens as trainable prefixes during embedding model training, aiming to reduce residual token-level information by providing dedicated slots for the model to route non-semantic attention. The model architecture and retrieval objective remain unchanged. We first describe the encoding process and then present the retrieval training objective.

Dump-Token Conditioned Encoding

We describe the encoding process of an embedding model and then explain how [DUMP] tokens are incorporated as input prefixes. Let qq and pp denote a query and a passage, respectively. For an input text x∈{q,p}x \in\{q,p\}, a standard embedding model maps xx into a fixed-dimensional representation through tokenization, model encoding, and pooling. We add special tokens, denoted as [DUMP], to the tokenizer vocabulary. In practice, we use KK indexed [DUMP] tokens, i.e., [DUMP1_1, …, [DUMPK_K]. The embeddings of these tokens are initialized in the model’s embedding matrix and updated jointly with the model parameters during retrieval fine-tuning. For x∈{q,p}x \in\{q,p\}, we prepend [DUMP] tokens:

x~=[DUMP1]⋯[DUMPK]⊕x,(1)\tilde{x} = [\mathrm{DUMP}_1] \cdots[\mathrm{DUMP}_K] \oplus x, \tag*{(1)}

where ⊕\oplus denotes text concatenation. During the forward pass, these prefix tokens interact with the regular text tokens via self-attention, allowing them to absorb structural or uninformative noise. The resulting embedding is computed as:

E~θ(x)=pooling⁡(Mθ(T(x~))),(2)\widetilde{E}_{\theta}(x) = \operatorname{pooling}\left(M_{\theta}(T(\tilde{x}))\right), \tag*{(2)}

where T(⋅)T(\cdot) denotes the tokenizer, MθM_{\theta} denotes the embedding model, and pooling⁡(⋅)\operatorname{pooling}(\cdot) denotes the pooling strategy. Crucially, the pooling strategy still operates on its default target (e.g., the [CLS] or last token), while the contextualized representation is influenced by interactions with the [DUMP] prefixes, which are introduced to reduce representational interference. The same prefixed encoding is used during both fine-tuning and inference. The [DUMP] tokens receive no separate supervision or auxiliary objective, and their embeddings are updated only through the retrieval loss.

Training Objective

We fine-tune the embedding model using retrieval pairs {(qi,pi+)}i=1N\{(q_i,p_i^+)\}_{i=1}^{N}, where qiq_i denotes a query and pi+p_i^+ denotes its positive passage. Before encoding, the [DUMP] tokens are prepended to both queries and passages. We define the embeddings and the scaled similarity function as:

ex=E~θ(x),s(q,p)=cos⁡(eq,ep)/τ.(3)e_x = \widetilde{E}_{\theta}(x), \qquad s(q,p) = \cos(e_q,e_p)/\tau. \tag*{(3)}

The model is optimized with the InfoNCE loss:

LNCE=−log⁡es(qi,pi+)es(qi,pi+)+∑j=1nes(qi,pi,j−),(4)\mathcal{L}_{\mathrm{NCE}} = -\log\frac{e^{s(q_i,p_i^+)}}{e^{s(q_i,p_i^+)}+\sum_{j=1}^{n}e^{s(q_i,p_{i,j}^{-})}}, \tag*{(4)}

where τ\tau is the temperature parameter and {pi,j−}j=1n\{p_{i,j}^{-}\}_{j=1}^{n} denotes negative passages for query qiq_i. This objective increases the similarity between each query and its positive passage while separating it from negative passages. Since the [DUMP] token embeddings are part of the trainable parameters, they are updated through the same retrieval loss without any auxiliary objective. Because the final pooled vector relies on the attention distributions formed across the entire sequence, the training signal from the contrastive loss naturally backpropagates into the [DUMP] token embeddings, optimizing them to better manage non-semantic cues.

Experiment

Experimental Setup

Training. We implement our method across a diverse set of encoder and decoder architectures. For the encoder models, we utilize gte-en-mlm-base [29], NeoBERT-base [4], EuroBERT-210m [3], and bge-m3-retromae [5]. For the decoder models, we employ ettin-decoder-150m [24] and MobileLLM-R1-140M [31]. For encoder models, we use [CLS] pooling, while for decoder models, we use last-token pooling [30]. We set the number of [DUMP] tokens to K=4K = 4 in our main experiments, corresponding to [DUMP1], [DUMP2], [DUMP3], and [DUMP4]. For training, we construct a dataset of 110k instances, utilizing one negative passage per query. Specifically, we use 100k examples from RLHN [23] to leverage its relabeled hard negatives, which provide cleaner contrastive supervision by mitigating the influence of false negatives. To enhance long-context capabilities, we additionally include 10k instances from MLDR [5], a long-document retrieval dataset constructed from lengthy Wikipedia and mC4 articles. All models are fine-tuned for one epoch using a maximum sequence length of 8192, a learning rate of 1×10−41 \times10^{-4}, and a contrastive temperature scale of 20.0. Training is conducted with bfloat16 precision and gradient caching [9], maintaining a global batch size of 256 to gather sufficient cross-device in-batch negatives.

Evaluation. We evaluate retrieval performance using the MTEB framework [17] on a subset of the BEIR benchmark [22]. To rigorously test the models’ discriminative capabilities, we specifically utilize the hard-negative versions of Natural Questions [13], HotpotQA [27], Quora [7], DBPedia [11], and MS MARCO [1]. We additionally evaluate on the English subset of Wikipedia from MTEB [17] to measure performance on a large-scale general-domain corpus. Furthermore, we use LongEmbed [32] to assess long-context embedding performance, where models must effectively compress substantially longer inputs into fixed representations. We report nDCG@10 for all retrieval evaluations.

Results

Main Results. Table 1 reports retrieval performance across different embedding models. The Base models are fine-tuned with the standard retrieval objective without additional prefix tokens, whereas Ours prepends trainable [DUMP] tokens during training.

ModelWikipedia BaseWikipedia OursNQ BaseNQ OursHotpotQA BaseHotpotQA OursQuora BaseQuora OursDBPedia BaseDBPedia OursMSMARCO BaseMSMARCO OursAvg. BaseAvg. Ours
Encoder Architecture
gte-en-mlm-base0.83480.83850.38280.37930.52000.52680.82660.83880.36320.36550.62580.63100.59220.5966
NeoBERT-base0.85000.85340.39440.38300.54690.54400.82790.83690.36830.37190.57720.59090.59410.5967
EuroBERT-210m0.83280.85180.38490.38810.52400.52760.81240.82700.30160.31030.61580.62560.57860.5884
bge-m3-retromae0.85210.86480.40620.40620.54540.54300.84480.85120.34090.34560.62860.62870.60300.6066
Decoder Architecture
ettin-decoder-150m0.72770.76010.18390.20900.38610.42150.71350.77030.18890.21600.42640.48820.43770.4775
MobileLLM-R1-140M0.85390.86200.37790.41020.54250.55720.82510.83460.30470.34780.61210.64380.58600.6093

Table 1. Retrieval results on standard benchmark datasets. The table reports nDCG@10 and compares fine-tuned baseline models with their counterparts trained using [DUMP] tokens across encoder and decoder architectures.

Overall, our method consistently improves the average performance across all evaluated models, indicating that adding trainable prefix tokens is broadly effective for retrieval embedding learning. The gains are consistent across both encoder and decoder architectures; for example, ettin-decoder improves from 0.4377 to 0.4775 on average, while EuroBERT improves from 0.5786 to 0.5884.

These consistent improvements support our hypothesis that assigning dedicated [DUMP] positions acts as a useful auxiliary buffer to reduce representational interference during query–passage encoding. Since the model architecture and retrieval objective remain unchanged, these results demonstrate that trainable prefix tokens provide a simple yet effective way to improve retrieval representations without requiring auxiliary losses or structural modifications.

Long-Context Retrieval Results. Table 2 presents the LongEmbed evaluation results, where inputs are substantially longer than in standard retrieval benchmarks. This setting poses a stronger compression challenge because the final pooled embedding must encode information from a larger number of contextualized token representations. Our method consistently improves over the base models; for example, gte-en-mlm improves from 0.4797 to 0.5024, and ettin-decoder improves from 0.3330 to 0.3498. These gains suggest that trainable prefix tokens remain effective under long-context encoding, where residual token-level information can have a larger influence on the final pooled representation. This result is particularly meaningful because long-context inputs increase the risk that useful local evidence is weakened during pooling. The consistent gains suggest that trainable prefix tokens help the pooled representation preserve retrieval-relevant information even when the input contains many competing contextual signals.

Arch.ModelBaseOurs
Encodergte-en-mlm-base0.47970.5024
EncoderNeoBERT-base0.33910.3452
EncoderEuroBERT-210m0.43130.4363
Encoderbge-m3-retromae0.55480.5679
Decoderettin-decoder-150m0.33300.3498
DecoderMobileLLM-R1-140M0.55800.5593

Table 2. Long-context retrieval performance on LongEmbed, reported in nDCG@10. The table compares baseline models with models trained using [DUMP] tokens.

Figure 1. Effect of the number of [DUMP] tokens on retrieval performance. We compare one, two, and four [DUMP] tokens with gte-mlm-base as the encoder and MobileLLM-R1-140M as the decoder. Bars indicate performance changes relative to the corresponding baseline without [DUMP] tokens.

Analysis

Effect of the number of [DUMP] tokens.

Figure 1 compares retrieval performance when varying the number of [DUMP] tokens, with K∈{1,2,4}K \in\{1, 2, 4\}. Adding these tokens generally improves performance over the baseline, in most cases across both models, but the gains are not strictly monotonic. For gte-mlm, using two tokens yields the best average improvement, whereas using four tokens slightly hurts NQ despite improving most other datasets. For MobileLLM-R1, using four tokens achieves the largest average gain, showing a clearer benefit from increasing the number of prefix tokens. Although the best token count varies across models and datasets, using [DUMP] tokens is generally more effective than not using them. These results suggest that the number of prefix tokens acts as a tunable hyperparameter. While additional tokens can improve retrieval performance, increasing the token count does not consistently lead to better results across all settings.

The Role of Dump Tokens

Setup. To understand how [DUMP] tokens affect internal representations, we analyze token-level representations and attention routing using gte-en-mlm-base on LongEmbed NQ. In the [DUMP] setting, the model is trained with four prepended [DUMP] tokens, denoted as D1–D4. We group tokens into the sequence representation token ([CLS]), the learned prefix tokens (D1–D4), and low-information tokens, identified using NLTK English stopwords. Motivated by prior work showing that Transformer artifacts can be associated with unusually high-norm token representations [6],

Panel (a) measures the mean L2 norm of final-layer hidden states to compare activation scales across token groups. Panel (b) measures last-layer self-attention from low-information tokens to [CLS] and D1–D4, averaged over attention heads, and reports the weights after 10510^{5} scaling for readability.

Results. Figure 2 summarizes the token-level analysis. As shown in Panel (a), the baseline model exhibits relatively high activation norms for low-information tokens, suggesting that these tokens can carry disproportionately large internal activations. With [DUMP] tokens, the L2 norm of low-information tokens is reduced, while the learned prefixes (D1–D4) form a stable activation range close to the [CLS] token. Panel (b) further shows that attention from low-information tokens is redistributed away from [CLS] and toward the learned prefixes. These results suggest that [DUMP] tokens provide alternative prefix positions to which attention from low-information tokens is redirected, thereby reducing their direct influence on the sequence representation.

Figure 2. Token-level analysis of the [DUMP] mechanism using gte-en-mlm-base. Panel (a) compares token-group activation scales measured by the mean L2 norm. Panel (b) visualizes the attention distribution from low-information tokens to [CLS] and D1–D4, with D1–D4 ordered by prefix position.

Conclusion

We investigated whether trainable [DUMP] tokens can reduce representational interference in embedding models when prepended to the input text. Our approach preserves the original model architecture, pooling function, and retrieval objective, while allowing the added prefix tokens to be learned through the same contrastive training process. Across retrieval and long-context benchmarks, our method consistently improves over finetuned baselines, showing that learnable input prefixes can enhance retrieval representations without additional supervision or architectural changes. Beyond overall retrieval gains, our analysis indicates that [DUMP] tokens can change how low-information token signals are represented and attended to during encoding, providing evidence that the improvement is linked to internal representation behavior rather than only external performance differences. Overall, these findings suggest that trainable prefix tokens offer a simple and effective way to improve retrieval embeddings while preserving the underlying model architecture.

GenAI Usage Disclosure

We used GenAI tools only for grammar checking and language polishing, not for method development, experiments, or result analysis.

Acknowledgments

This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2021R1A6A1A03045425). This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI). This research was supported by Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2026 (Project Name: Development of an AI Agent Integrating Korean Language Knowledge for Personalized Language Consultation Services, Project Number: RS-2026-25506607, Contribution Rate: 30%). This work was supported by the Commercialization Promotion Agency for R&D Outcomes(COMPA) grant funded by the Korea government(Ministry of Science and ICT)(2710096072).

References

References

  1. [1]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangarajan Jandmar, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016).arxiv.org/abs/1611.09268
  2. [2]Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. URL: Large language models are secretly powerful text encoders, 2024. URL https://arxiv.org/abs/2404.05961 (2024).
  3. [3]Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. 2025. EuroBERT: Scaling Multilingual Encoders for European Languages. arXiv:2503.05500 [cs.CL] https://arxiv.org/abs/2503.05500
  4. [4]Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. 2025. NeoBERT: A Next-Generation BERT. arXiv:2502.19587 [cs.CL] https://arxiv.org/abs/2502.19587
  5. [5]Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL]arxiv.org/abs/2402.03216
  6. [6]Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In International conference on learning representations, Vol. 2024. 2632–2652.arxiv.org/abs/2309.16588
  7. [7]DataCanary, xiaofalk, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. 2017. Quora Question Pairs. https://kaggle.com/competitions/quora-question-pairs
  8. [8]Meet Doshi, Aashka Trivedi, Vishwajeet Kumar, Parul Awasthy, Yulong Li, Jaydeep Sen, Radu Florian, and Sachindra Joshi. 2026. LMK> CLS: Landmark Pooling for Dense Embeddings. arXiv preprint arXiv:2601.21525 (2026).arxiv.org/abs/2601.21525
  9. [9]Luyu Gao, Yunyao Zhan, Jiawei Han, and Jamie Callan. 2021. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). 316–321.arxiv.org/abs/2101.06983
  10. [10]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiang Wang, and Min Lin. 2025. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations, Vol. 2025. 97114–97144.arxiv.org/abs/2410.10781
  11. [11]Faezeh Hasibi, Fedor Nikolaev, Chenyan Xiong, Kristian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-Entity V2: A Test Collection for Entity Search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17). ACM, 1265–1268. https://doi.org/10.1145/3077136.3080751
  12. [12]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 6769–6781.arxiv.org/abs/2004.04906
  13. [13]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics (2019).DOI
  14. [14]Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Nv-embed: Improved techniques for training llms as generalist embedding models. In International Conference on Learning Representations, Vol. 2025. 79310–79333.arxiv.org/abs/2405.17428
  15. [15]Reagan J Lee, Charm Golel, and Kannan Ramachandran. 2024. Quantifying positional biases in text embedding models. arXiv preprint arXiv:2412.15241 (2024).arxiv.org/abs/2412.15241
  16. [16]Zheng Liu, Shitao Xiao, Yingxia Shao, and Zhao Cao. 2023. RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Noaoki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 2635–2648. https://doi.org/10.18653/v1/2023.acl-long.148
  17. [17]Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2014–2037.
  18. [18]Yuval Ran-Milo. 2026. Attention sinks are probably necessary in softmax transformers: Evidence from trigger-conditional tasks. arXiv preprint arXiv:2603.11487 (2026).
  19. [19]Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 3982–3992.arxiv.org/abs/1908.10084
  20. [20]Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762 (2024).arxiv.org/abs/2402.17762
  21. [21]Yixuan Tang and Yi Yang. 2024. Pooling and attention: What are effective designs for llm-based embedding models? arXiv preprint arXiv:2409.02727 (2024).arxiv.org/abs/2409.02727
  22. [22]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv preprint arXiv:2104.08663 (2021).arxiv.org/abs/2104.08663
  23. [23]Nandan Thakur, Crystalyn Zhang, Xueguang Ma, and Jimmy Lin. 2025. Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard Negatives for Robust Information Retrieval. arXiv:2505.16967 [cs.IR] https://arxiv.org/abs/2505.16967
  24. [24]Orion Weller, Katheryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2025. Seq vs Seq: An Open Suite of Paired Encoders and Decoders. arXiv:2507.11412 [cs.CL] https://arxiv.org/abs/2507.11412
  25. [25]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453 (2023), 1.
  26. [26]Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 538–548.
  27. [27]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing. 2369–2380.arxiv.org/abs/1809.09600
  28. [28]Stephen Zhang, Mustafa Khan, and Vardan Papyan. 2025. Attention Sinks and Outlier Features: A’Catch, Tag, and Release’ Mechanism for Embeddings. arXiv e-prints (2025), arXiv–2502.arxiv.org/abs/2502.00919
  29. [29]Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mTGE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. arXiv:2407.19669 [cs.CL] https://arxiv.org/abs/2407.19669
  30. [30]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025).arxiv.org/abs/2506.05176
  31. [31]Changsheng Zhao, Ernie Chang, Zechun Liu, Chia-Jung Chang, Wei Wen, Chen Lai, Sheng Cao, Yuandong Tian, Raghuraman Krishnamoorthi, Yangyang Shi, and Vikas Chandra. 2025. MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes. arXiv preprint arXiv:2509.29445 (2025).
  32. [32]Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. LongEmbed: Extending Embedding Models for Long Context Retrieval. arXiv preprint arXiv:2404.12096 (2024).arxiv.org/abs/2404.12096
  33. [33]Tianyu Zong, Hongzhu Yi, Bingkang Shi, Yuanxiang Wang, and Jungang Xu. 2025. JTCSE: Joint Tensor-Modulus Constraints and Cross-Attention for Unsupervised Contrastive Learning of Sentence Embeddings. arXiv preprint arXiv:2505.02366 (2025).arxiv.org/abs/2505.02366
  34. [34]Zayd M. K. Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. 2026. Soft-pick: No Attention Sink, No Massive Activations with Rectified Softmax. arXiv:2504.20966 [cs.LG] https://arxiv.org/abs/2504.20966

Paper details

vlnc:W_24mpqaq5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 15

downloads · 3

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents