WorksComputer Science
Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval
Abstract
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.
Introduction
Speech retrieval searches a collection of spoken documents using a spoken query [12, 30, 34]. For example, “Where should we go this weekend?” spoken excitedly could be used to find activity suggestions, another recording by the same speaker, or speech with similar excitement. The spoken query alone does not distinguish these goals.
Instruction-aware speech retrieval uses natural-language instructions to express the user’s retrieval goal [35]. Each instruction expresses requirements for speech attributes such as spoken content, speaker identity, and speaking style. The required values may come from the spoken query or be specified directly. For example, “Find speech by the same speaker, spoken happily” asks for a document that matches the query’s speaker identity and has a happy speaking style. A relevant spoken document must satisfy every requirement.
Large audio-language models (LALMs) connect an audio encoder to a language model and have shown strong performance on tasks such as speech recognition, audio captioning, and audio question answering [71, 21, 44] (Xu et al. 2025a). Their ability to process speech and text within a model motivates their adaptation for instruction-aware speech retrieval. Our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We ask: How can we adapt LALMs for instruction-aware speech retrieval?
We introduce IRIS-Embed (Instruction-aware Retrieval for Speech), which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. Figure 1 illustrates the overall framework. IRIS-Embed separately encodes a spoken query with its instruction and each spoken document, then ranks documents by cosine similarity. Audio-layer weighting predicts weights based on the instruction to combine hidden states from different layers of the frozen audio encoder before language-model encoding.
Figure 1. IRIS-Embed overview. A: Retrieval and training with a shared model. B: Audio layer weighting and embedding computation. Inst.: instruction.
Contrastive training aligns query embeddings with relevant document embeddings. We construct about 900K samples from real data and synthetic data, each pairing a spoken query and instruction with positive and negative spoken documents. Each instruction expresses requirements for one or more speech attributes. The positive spoken document must satisfy every requirement in the instruction.
We evaluate IRIS-Embed on INSPIRE [35], an instruction-aware speech retrieval benchmark. Across two model families, IRIS-Embed consistently outperforms the corresponding LALM baselines, achieving macro-average Recall@50 scores of 62.92% and 47.41%, both surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness for single- and multi-attribute speech requirements. We also show that audio-layer weighting improves speaker and speaking-style matching compared with using only the final audio-encoder hidden states. On MAEB [5], we evaluate IRIS-Embed on 15 diverse tasks, including gender classification, accent-pair classification, and artist retrieval. IRIS-Embed achieves macro-average gains of 9.4–19.8 points over the corresponding LALM baselines, demonstrating generalization beyond speech retrieval.
Related Work
Spoken Content Retrieval
Spoken content retrieval finds spoken documents with content relevant to a spoken query [19, 12, 30, 34]. Prior work explores query-by-example spoken term detection [3, 59, 60] and spoken passage retrieval for question answering [39, 48]. We focus on instruction-aware speech retrieval, where instructions express requirements based on spoken content, speaker identity, speaking style, or other speech attributes.
Instruction-Aware Retrieval
Instructions guide text retrieval by expressing retrieval goals and relevance requirements (Asai et al., 2023; Su et al., 2023; Weller et al., 2025b,a; Wang et al., 2024; Lee et al., 2025a; Zhang et al., 2025b; Lee et al., 2025b). Image and multimodal retrieval methods also use text to express retrieval goals [42, 6, 63, 26, 27, 47, 36] (Wu et al. 2021; Vaze et al. 2023; Zhang et al. 2024; Zhang et al. 2025a; Zhou et al. 2025). INSPIRE [35] benchmarks instruction-aware speech retrieval with requirements for one or more speech attributes. We adapt LALMs into embedding models for instruction-aware speech retrieval.
Audio-Language Embeddings
Audio-language embeddings support retrieval between audio and text, as well as audio classification [23, 18, 46, 37] (Wu et al. 2023; Xu et al. 2023; Zhu et al. 2024). LALM embeddings also support retrieval between audio and text [13, 43] (Xiao et al. 2025; Yoo et al. 2026). We focus on retrieving spoken documents given a spoken query and a natural-language text instruction. Furthermore, IRIS-Embed uses instruction-aware audio-layer weighting to combine hidden states from different layers of the frozen audio encoder before language-model encoding.
Audio Layer Aggregation
SUPERB aggregates frozen self-supervised speech-encoder hidden states for downstream tasks (Wen Yang et al., 2021). Prompt-aware Mixture uses prompt-guided fusion for recognition and captioning [66]. WAVE and OmniUE aggregate language-model states for multimodal embeddings [70] (Wang et al. 2026). Instead, IRIS-Embed learns instruction-aware weights to combine hidden states from different layers of an LALM’s frozen audio encoder through contrastive speech retrieval training.
Method
Problem Formulation
Instruction-aware speech retrieval ranks spoken documents in a collection given a spoken query and a natural-language text instruction . Speech attributes describe properties of a recording, such as spoken content, speaker identity, and speaking style. The instruction expresses the requirements on these attributes by referring directly to the spoken query or specifying the values. For example, “Find speech by the same speaker, spoken happily” requires a document to match ’s speaker identity and have a happy speaking style. A relevant document must satisfy every requirement; unconstrained attributes do not affect relevance.
For each query–instruction pair , let denote the positive documents relevant to and denote the negative documents. A scoring function ranks documents in decreasing order of score. An ideal ranking satisfies , for every and .
Architecture
As shown in Figure 1, IRIS-Embed uses a shared model to separately encode the spoken query with instruction and each spoken document . For each recording, we extract hidden states from the pre-encoder and all layers of the frozen audio encoder. The audio-layer weighting module encodes a text-only prompt to predict weights for combining these hidden states. The weighted audio hidden states are then pooled and projected before being passed to the language model, which processes the text and weighted audio hidden states using its own prompt template. Query prompts include instruction for both audio layer weighting and language-model encoding, whereas document prompts contain no instruction .
We use the language model’s final-layer hidden state of the last non-padding token as the embedding, obtaining for the query–instruction pair and for document . Documents are ranked by decreasing cosine similarity, . We next explain the motivation and design of audio layer weighting.
Audio Layer Weighting
To investigate how different audio-encoder layers perform across retrieval tasks, we evaluate their frozen hidden states on INSPIRE ([35]). For each layer, we mean-pool the hidden states over time and rank spoken documents by cosine similarity to the spoken query, without using instructions, the audio projection, or the language model. Figure 2 reports Recall@50 for dialogue continuation, speaker matching, speaking-
Figure 2. Recall@50 across frozen audio-encoder layers. Layer 0 represents pre-encoder hidden states; layers 1 and above represent encoder hidden states.
style matching, and answer retrieval. The best-performing layer varies across tasks, and the final layer does not consistently achieve the highest recall. These findings motivate combining hidden states from multiple audio-encoder layers before passing them to the language model.
For each recording, let denote the pre-encoder hidden states and the hidden states produced by the audio-encoder layers. We combine hidden states at the same time step to obtain weighted audio hidden states:
where is the scalar weight assigned to .
A shared audio layer weighting module computes weights from a text prompt :
where uses a pretrained sentence encoder followed by a linear layer to produce layer scores. The notation selects the weight for . The resulting weights are nonnegative and sum to one. For queries, the prompt includes the instruction , allowing the layer weights to vary across retrieval tasks. For documents, we use a fixed prompt, so all documents share the same layer weights.
Training Objective
Each batch consists of samples , where indexes samples and and are the paired positive and negative documents. The batchlevel document pool is . Since documents from other samples may also be relevant to , treating them as negatives could introduce false negatives. We use training labels to mask these additional relevant documents. For each sample , we construct a document pool by excluding such documents from while preserving the paired positive document .
We optimize the model using the InfoNCE loss [53]:
where is a temperature parameter controlling the sharpness of the score distribution. This objective encourages the model to assign higher relevance scores to the paired positive document than to the remaining negatives in .
During training, we freeze the audio encoder and the sentence encoder in . We adapt the language model using low-rank adaptation (LoRA) [24]. The audio projection, which maps audio hidden states to the language model’s embedding dimension, is trained jointly with the added linear layer in and the LoRA parameters.
Training Data
Notation
We represent each recording by a set of attribute-value pairs. Each pair identifies a speech attribute and its value, such as (style, happy) for a happy speaking style.
The requirements form a set of attribute-value pairs that a relevant spoken document must satisfy. We write . The requirements in match the spoken query’s attribute values, with . The set contains attribute-value pairs with specified values that need not match the spoken query. We allow . The text instruction expresses in natural language, referring to the spoken query for and stating the values in . The two sets use disjoint attribute keys to avoid conflicting requirements, so .
For example, consider a spoken query with happy speech and a dog barking in the background. The requirements ask for sad speech with a barking dog. Here, matches the query’s environmental sound, while specifies a sad speaking style. A corresponding
instruction is “Find a recording with the same background sound as the query, spoken sadly.”
Data Format
Each training sample contains a spoken query, an instruction, and a positive and a negative spoken document. A document is relevant if it satisfies all requirements in : . The paired positive belongs to , while the paired negative belongs to and violates at least one requirement. Attributes outside do not affect relevance.
Construction
We construct IRIS-Embed training samples from real data and synthetic data, each paired with an instruction expressing its retrieval requirements. Figure 3 shows the data construction pipeline.
Figure 3. Overview of the data construction pipeline, combining real and synthetic data to create spoken query–document pairs and retrieval instructions.
Real Data. We use public speech and audio corpora. The source annotations cover speaking style [11], speaker identity [49], gender (Zhou et al., 2021), accent (Zhao et al., 2018), sentiment [57], dialogue [67, 55], and environmental sound [64]. We remove uninformative labels such as unknown and other, recordings outside 1–120 s, and environmental sounds containing speech. We also map equivalent labels across corpora, such as happy and joyful, to a common value. Corpus metadata provide attribute values for each recording, which we use to construct retrieval requirements and select query–document pairs. Synthetic Data. We sample English text triplets from the MS MARCO Nguyen et al., [50] hard-triplet data, each containing a query, a relevant passage, and a hard negative passage. We synthesize speech from these texts using multiple text-to-speech models Comanici et al., [16]; Hu et al. [25]. Voice selection controls speaker identity, synthesis prompts control speaking style, and sound mixing adds environmental sounds.
Pair Construction. For each spoken query , we sample a nonempty from and optionally form by sampling values for other attribute keys annotated for . Speaker identity, dialogue, and answer relevance are restricted to . When speaker identity is included, we exclude gender and accent, since matching the speaker also guarantees matching these attributes in our data. For each , we sample a positive and a negative document, prioritizing negatives that violate fewer requirements to obtain harder negatives.
Instruction Generation. We use commercial large language models to express each set of requirements as a natural-language instruction . To cover diverse instruction styles, we use one prompt to generate short instructions and another to generate full-sentence instructions. We remove near-duplicates using ROUGE-L Lin, [38]. We also verify that each instruction covers all requirements in , refers to the spoken query for pairs in , and specifies the values in .
Statistics. The resulting dataset contains approximately 900K samples: 800K from real data and 100K from synthetic data. We use about 890K samples for training and 10K for validation. Appendix § 8 lists the sources, counts, and data construction settings.
Experiments
Experimental Setup
Evaluation. We evaluate IRIS-Embed on four INSPIRE Li and Lee, [35] subsets. DailyTalk Lee et al., [33] evaluates dialogue continuation, VCTK (Yamagishi et al., 2019) evaluates speaker matching, and Expresso Nguyen et al., [51] evaluates speaker and speaking-style matching. Speaking style includes emotion and vocal states. The Synthetic subset evaluates answer retrieval and matching of speaker identity, speaking style, and environmental sounds, with instructions involving one or more attributes. Performance is measured by Recall@ for , defined as the fraction of relevant documents retrieved in the top results, averaged across query–instruction pairs and reported as percentages.
Training. We train IRIS-Embed with Qwen2.5-Omni-3B (Xu et al., 2025a) and Gemma-4-E2B-it Team et al., [71] on the data in § 4. The audio layer weighting module contains a sentence encoder and a learned linear layer. We use the frozen all-MiniLM-L6-v2 model Wang et al., (2020); Reimers and Gurevych [61] as the sentence encoder. Training uses a global batch size of 512. Appendix § 9 provides further training details.
Comparison Methods
Random. Uniform random document rankings serve as a chance-level baseline Li and Lee, [35].
General Speech Encoders. We evaluate general speech representations using the self-supervised WavLM-large Chen et al., [14] without instructions. Query and document embeddings are obtained by mean-pooling its final-layer hidden states over time.
Specialized Speech Encoders. We evaluate specialized speech representations without instructions, using ECAPA-TDNN Desplanques et al., [17] for speaker embeddings and emotion2vec-base Ma et al., [45] for emotion embeddings.
Cascaded Retrieval. We use a cascaded pipeline to retrieve spoken documents through text representations. Whisper-large-v3 Radford et al., [58] generates transcripts, while Qwen3-Omni-30B-A3B-Captioner (Xu et al., 2025b) generates captions. Qwen3-Embedding-8B (Zhang et al., 2025b) encodes the transcripts and captions of each spoken query and spoken document, with the instruction included in the query input.
LALM Embeddings. We evaluate Qwen2.5-Omni-3B (Xu et al., 2025a), Gemma-4-E2B-it Team et al., [71], and Qwen3-Omni-30B (Xu et al., 2025b) without additional embedding training. Embeddings are extracted from the final-layer hidden state of the last non-padding input token without generating a response. We also include LCO-Embedding-Omni-3B (Xiao et al., 2025), a Qwen2.5-Omni-based model trained for multimodal embeddings.
Main Results
Table 1 shows that IRIS-Embed consistently outperforms the corresponding LALM baselines on all four INSPIRE subsets. Macro-average Recall@50 reaches 62.92% for Qwen2.5-Omni-3B and 47.41% for Gemma-4-E2B-it, compared with 16.27% and 8.91% for their base models. Both exceed ECAPA-TDNN’s 45.15%, the highest macro-average Recall@50 among comparison baselines.
| Method | Model | DailyTalk R@10 | DailyTalk R@50 | DailyTalk R@100 | VCTK R@10 | VCTK R@50 | VCTK R@100 | Expresso R@10 | Expresso R@50 | Expresso R@100 | Synthetic R@10 | Synthetic R@50 | Synthetic R@100 | Avg. R@10 | Avg. R@50 | Avg. R@100 |
| RAND | Random | 0.24 | 1.06 | 2.10 | 0.35 | 1.84 | 3.63 | 0.30 | 1.50 | 2.96 | 0.20 | 1.00 | 1.98 | 0.27 | 1.35 | 2.67 |
| ENC | WavLM-large | 16.50 | 24.50 | 35.50 | 17.50 | 35.00 | 43.75 | 9.50 | 17.69 | 24.00 | 1.70 | 4.48 | 7.22 | 11.30 | 20.42 | 27.62 |
| SENC | ECAPA-TDNN | 15.50 | 24.00 | 29.50 | 100.00 | 100.00 | 100.00 | 34.88 | 41.75 | 45.44 | 10.02 | 14.83 | 18.82 | <u>40.10</u> | 45.15 | 48.44 |
| emotion2vec-base | 11.50 | 23.00 | 30.50 | 3.13 | 6.25 | 13.75 | 0.88 | 4.38 | 8.00 | 1.61 | 4.53 | 6.82 | 4.28 | 9.54 | 14.77 | |
| CAS | Qwen3-Embedding-8B | 62.00 | 81.50 | 89.50 | 1.88 | 6.25 | 10.63 | 0.75 | 5.56 | 9.69 | 7.02 | 8.95 | 11.07 | 17.91 | 25.57 | 30.22 |
| LALM | Qwen2.5-Omni-3B | 26.50 | 53.00 | 64.00 | 1.88 | 3.13 | 3.13 | 0.75 | 3.75 | 6.69 | 3.14 | 5.20 | 6.88 | 8.07 | 16.27 | 20.18 |
| Gemma-4-E2B-it | 9.00 | 25.50 | 33.50 | 1.25 | 4.38 | 5.63 | 0.13 | 1.19 | 2.25 | 2.52 | 4.56 | 6.03 | 3.23 | 8.91 | 11.85 | |
| Qwen3-Omni-30B | 51.50 | 74.50 | 83.00 | 1.25 | 3.13 | 6.25 | 0.94 | 3.56 | 6.19 | 4.88 | 6.38 | 7.88 | 14.64 | 21.89 | 25.83 | |
| LCO-Embedding-Omni-3B | 54.00 | 75.50 | 83.50 | 1.25 | 2.50 | 5.63 | 1.94 | 8.44 | 13.81 | 6.82 | 9.37 | 11.65 | 16.00 | 23.95 | 28.65 | |
| IRIS | Qwen2.5-Omni-3B | 60.00 | <u>77.50</u> | <u>87.50</u> | <u>67.50</u> | 85.00 | 89.38 | <u>29.63</u> | 45.00 | 53.88 | 26.06 | 44.17 | 54.78 | 45.80 | 62.92 | 71.39 |
| Gemma-4-E2B-it | 25.50 | 44.50 | 54.50 | 58.75 | 74.38 | 79.38 | 28.63 | <u>43.81</u> | <u>52.31</u> | <u>15.07</u> | <u>26.94</u> | <u>36.08</u> | 31.99 | <u>47.41</u> | <u>55.57</u> |
Table 1. INSPIRE Recall@. Avg.: macro-average across the four subsets. Bold / underline: best / second best. RAND: random; ENC / SENC: general / specialized encoders; CAS: cascaded retrieval; LALM: large audio–language models; IRIS: IRIS-Embed.
Both IRIS-Embed models consistently outperform all LALM embedding baselines on VCTK and Expresso. IRIS-Embed with Qwen2.5-Omni-3B achieves 85.00% and 45.00% Recall@50 on these subsets, respectively, compared with 2.50% and 8.44% for LCO-Embedding-Omni-3B.
Both models also consistently outperform all comparison baselines on Synthetic. This subset includes instructions with multiple attributes, for which a relevant document must satisfy every requirement. Recall@50 reaches 44.17% for Qwen2.5-Omni-3B and 26.94% for Gemma-4-E2B-it, compared with 14.83% for ECAPA-TDNN.
On DailyTalk, IRIS-Embed with Qwen2.5-Omni-3B achieves 77.50% Recall@50, close to 81.50% for the cascaded pipeline using larger captioning and text-embedding models. While the specialized ECAPA-TDNN remains strongest for speaker matching on VCTK, IRIS-Embed supports retrieval across speech attributes within a single embedding model.
Analysis
Instruction Types
We breakdown IRIS-Embed’s performance on instructions involving single or multiple attributes. Figure 4 compares IRIS-Embed with the corresponding LALM baselines across instruction types. Both IRIS-Embed models achieve higher Recall@50 on most instruction types.
Figure 4. Recall@50 by instruction type on INSPIRE. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound. +: multiple attributes; ∗: specified values.
On Synthetic, IRIS-Embed also performs well on instructions involving multiple attributes, including specified attribute values. For joint matching of speaker identity, speaking style, and environmental sound, Recall@50 reaches 61.8% for Qwen2.5-Omni-3B and 31.5% for Gemma-4-E2B-it. For the speaker matching with a specified speaking style, the scores reach 46.0% and 45.2%, respectively. Both models outperform all baselines on most multi-attribute instruction types. The complete results are provided in the Appendix § 11.
Instruction Awareness
We examine whether IRIS-Embed distinguishes different retrieval goals for the same spoken query. Figure 5 reports Recall@50 for Qwen2.5-Omni-3B under different short instructions, such as “Same speaker” and “Find the answer”. Among five instructions with a single attribute, the instruction matching the evaluation requirements achieves the highest Recall@50 in seven of eight evaluation settings. Switching from a speaker-matching instruction to an answer-retrieval instruction increases answer retrieval Recall@50 from 1.5% to 90.3%.
Figure 5. Recall@50 for IRIS-Embed with Qwen2.5-Omni-3B. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound; +: multiple attributes.
On Synthetic, we compare four short instructions combining speaker, speaking style, and environmental sound, such as “Same speaker and same speaking style”. The matching instruction achieves the highest Recall@50 in all four evaluation settings. Gemma-4-E2B-it shows a similar pattern, and both models also follow full-sentence instructions, as reported in Appendix § 11.§ 2. These results show that IRIS-Embed follows the speech-attribute requirements expressed by the instruction.
Layer Aggregation
Table 2 compares Qwen2.5-Omni-3B variants trained and evaluated with different audio-layer aggregation methods. Layer 13 and Final Layer use the 13th and final audio-encoder hidden states, respectively. Mean Layer uses the average weighting across layers. Finetuned updates the audio encoder, while the other variants keep it frozen. Static Weight learns a single set of audio layer weights shared across all queries and documents, regardless of instructions or prompts. Inst. Weight is our instruction-aware audio layer weighting method, with weights varying according to the instruction.
| Method | DailyTalk | VCTK | Expresso | Synthetic | Avg. |
| Layer 13 | 15.50 | 76.88 | 48.94 | 35.27 | 44.15 |
| Final Layer | 82.50 | 48.75 | 28.25 | 34.72 | 48.56 |
| Mean Layer | 70.00 | 71.25 | 43.13 | 44.90 | 57.32 |
| Finetuned | 32.50 | 91.25 | 47.38 | 43.35 | 53.62 |
| Static Weight | 66.00 | 78.75 | 43.19 | 45.28 | 58.31 |
| Inst. Weight (Ours) | 77.50 | 85.00 | 45.00 | 44.17 | 62.92 |
Table 2. Recall@50 for Qwen2.5-Omni-3B audio layer aggregation on INSPIRE. Avg.: macro-average across the four subsets.
Our method achieves the highest macro-average Recall@50, at 62.92%. Compared with Final Layer, our method improves recall on VCTK, Expresso, and Synthetic, increasing the macro-average by 14.36 points. Layer 13 and Finetuned improve speaker and speaking-style matching over Final Layer, but have much lower dialogue-continuation recall. These results suggest that retrieval tasks benefit from hidden states of different audio-encoder layers, highlighting the importance of instruction-aware audio layer weighting.
Although Mean Layer and Static Weight perform better on Synthetic, our method outperforms both on the other three subsets, achieving macro-average Recall@50 improvements of 5.60 and 4.61 points, respectively. These results demonstrate the effectiveness of instruction-aware weighting over instruction-independent layer aggregation.
With Gemma-4-E2B-it, our method also outperforms Final Layer on VCTK, Expresso, and Synthetic, showing improvements across multiple retrieval subsets. Appendix § 11.§ 3 provides the full results for both models. Together, these results demonstrate the effectiveness of instruction-aware audio layer weighting in improving retrieval performance while keeping the audio encoder frozen.
Layer Weights
We analyze how audio layer weights vary across retrieval requirements. Figure 6 compares Qwen2.5-Omni-3B layer weights under five short instructions, such as “Same speaker” and “Find the answer”, and the document prompt. To account for scale differences, we adjust the displayed weights using the root mean square of each layer’s hidden states. This adjustment is used only for analysis. Layer 13 receives the highest adjusted weight across all five query instructions. Compared with speaker matching, answer retrieval places more weight on the final layer and less on layer 13. Gemma-4-E2B-it shows a similar pattern, with answer retrieval shifting weight from layer 4 toward its final layer. These weights show that instructions change the balance between intermediate and final audio-encoder hidden states. Appendix § 11.§ 5 provides the full results for both models.
Figure 6. Scale-adjusted audio layer weights for Qwen2.5-Omni-3B. Doc.: document prompt; layer 0: pre-encoder hidden states; encoder layers start at 1.
Task Generalization
We evaluate IRIS-Embed beyond retrieval tasks in INSPIRE on 15 MAEB ([5]) audio tasks covering classification, clustering, pair classification, reranking, and retrieval. These include gender and accent tasks not covered by INSPIRE. We encode each recording with a task instruction using the query prompt, except for retrieval documents, which use the document prompt without an instruction. The models remain frozen during evaluation, and tasks derived from our retrieval training corpora are excluded. The corresponding LALM baselines receive the same task instructions. Appendix § 10.§ 4 provides evaluation details.
Table 3 reports macro-average scores of 53.5 for Qwen2.5-Omni-3B IRIS-Embed and 52.9 for Gemma-4-E2B-it IRIS-Embed. These exceed the corresponding LALM baselines scores by 9.4 and 19.8 points, respectively, and approach LCO-Embedding-Omni-3B’s 54.2. Both IRIS-Embed models outperform all comparison baselines on IEMOCAP gender classification, VoxPopuli gender clustering, and VoxPopuli accent pair classification. These results show that IRIS-Embed generalizes beyond instruction-aware speech retrieval.
| Method | Model | Beijing Opera | BirdCLEF | CommonLang | Age | GTZAN | Genre | IEMOCAP | Gender | MInDS14 | Mridingham | Tonic | SIBFLEURS | VoxPopuli LanguageID | Vehicle Sound | VoxPopuli Gender | Avg. |
| ENC | WavLM-large | 57.7 | 19.8 | 16.9 | 67.7 | 63.1 | 13.5 | 31.4 | 12.5 | 89.0 | 0.2 | 0.1 | 49.0 | 52.2 | 68.8 | 78.5 | 41.4 |
| CLAP | clap-htsat-fused | 92.8 | 18.0 | 15.3 | 67.9 | 93.6 | 9.2 | 47.0 | 9.1 | 32.0 | 4.7 | 52.7 | 47.9 | 53.2 | 61.3 | 87.7 | 46.2 |
| CLAP | larger_clap_general | 93.6 | 17.0 | 20.5 | 84.5 | 89.3 | 8.0 | 49.4 | 9.4 | 84.6 | 3.4 | 3.3 | 47.4 | 53.5 | 66.8 | 95.6 | 48.4 |
| LALM | Qwen2-Audio-7B | 97.4 | 37.1 | 17.6 | 93.1 | 93.0 | 24.4 | 61.2 | 15.4 | 99.0 | 5.5 | 0.1 | 48.9 | 52.9 | 80.8 | 37.6 | 50.9 |
| LALM | LCO-Embedding-Omni-3B | 93.2 | 31.6 | 16.9 | 81.0 | 59.4 | 88.9 | 39.5 | 45.1 | 63.4 | 2.2 | 1.0 | 96.9 | 50.7 | 75.4 | 67.1 | 54.2 |
| LALM | Qwen2.5-Omni-3B† | 80.1 | 10.6 | 16.7 | 75.9 | 71.9 | 50.4 | 26.6 | 43.3 | 94.6 | 2.3 | 1.2 | 69.3 | 50.1 | 56.6 | 11.4 | 44.1 |
| LALM | Gemma-4-E2B-it† | 63.6 | 4.0 | 15.5 | 28.2 | 50.5 | 22.0 | 27.0 | 21.9 | 99.4 | 1.2 | 0.1 | 57.2 | 50.2 | 38.2 | 17.8 | 33.1 |
| IRIS | Qwen2.5-Omni-3B | 91.9 | 13.3 | 14.3 | 80.1 | 98.7 | 24.4 | 33.5 | 20.9 | 71.0 | 1.1 | 55.2 | 92.9 | 61.5 | 66.1 | 77.6 | 53.5 |
| IRIS | Gemma-4-E2B-it | 88.5 | 15.1 | 16.4 | 60.6 | 99.0 | 14.2 | 32.3 | 26.6 | 79.2 | 8.2 | 68.5 | 94.3 | 63.2 | 60.6 | 67.0 | 52.9 |
Table 3. Scores on 15 MAEB audio tasks. Avg.: macro-average across the 15 tasks. Higher is better; bold marks column maxima. Clf: classification; Clust: clustering; PC: pair classification; Rrnk: reranking; Rtrvl: retrieval. CLAP: contrastive language–audio pretraining. †: baselines using the same task instructions as IRIS-Embed.
Conclusion
IRIS-Embed adapts LALMs into embedding models for instruction-aware speech retrieval through contrastive training and instruction-aware audio layer weighting. Across two model families, macro-average Recall@50 on INSPIRE reaches 62.92% and 47.41%, compared with 45.15% for the strongest baseline. For Qwen2.5-Omni-3B, layer weighting improves VCTK Recall@50 from 48.75% to 85.00% over using only the final audio-encoder output. Controlled comparisons show that the models follow instructions with a single attribute or multiple attributes. On 15 MAEB tasks, macro-average scores improve by 9.4–19.8 points over the corresponding LALM baselines, showing generalization beyond speech retrieval. These results show that IRIS-Embed enables effective retrieval across speech attributes.
Limitations
Despite the advances achieved by our method, several limitations remain.
We prioritize smaller embedding models for computational efficiency and train IRIS-Embed with only two LALM backbones. Thus, its scalability to larger models remains unexplored.
IRIS-Embed combines hidden states from frozen pretrained audio encoders, making it dependent on the information they preserve. We do not explore alternative or complementary encoders that may capture broader speech attributes. Moreover, layer-weight visualizations do not establish individual layers’ contributions to retrieval performance.
Our retrieval requirements rely on annotations covering limited speech attributes and values. Annotation and classifier errors may affect document-pair construction and in-batch masking, respectively. Our training also covers limited languages, voices, speaking styles, and environmental sounds, restricting its diversity relative to natural speech.
Finally, our cascaded baseline uses transcripts and audio captions. We leave comparisons with finer-grained cascaded approaches combining specialized speech embeddings or attribute-specific audio descriptions to future work.
While these limitations remain, IRIS-Embed demonstrates the effectiveness of adapting LALMs for instruction-aware speech retrieval. We believe our work will inspire further research on embeddings that follow diverse user intents and generalize across audio tasks.
Ethics Statement
We use public audio corpora and synthetic data, with corpus licenses listed in Table 4. Reuse should respect source licenses and speech-synthesis service terms, and shared synthetic recordings should be clearly labeled.
| Source | Answer relevance | Speaking style | Gender | Speaker | Accent | Sentiment | Dialogue | Environmental sound | License |
| Real data | |||||||||
| VoxCeleb1 [49] | ◦ | • | • | CC BY-SA 4.0 | |||||
| ESD (Zhou et al., 2021) | • | • | • | Custom | |||||
| MELD [57] | • | ◦ | • | • | GPL-3.0 | ||||
| CREMA-D [11] | • | • | ODbL 1.0 | ||||||
| L2-ARCTIC (Zhao et al., 2018) | ◦ | ◦ | • | • | CC BY-NC 4.0 | ||||
| LibriSpeech-C [41] | ◦ | • | CC BY 4.0 | ||||||
| EmoV-DB [1] | • | • | • | Custom | |||||
| MSP-IMPROV [10] | • | • | Custom | ||||||
| VocalSound [22] | • | CC BY-SA 4.0 | |||||||
| SpokenWOZ [67] | • | CC BY-NC 4.0 | |||||||
| MultiDialog [55] | • | CC | |||||||
| UrbanSound8K [64] | • | CC BY-NC 3.0 | |||||||
| EMNS [52] | • | ◦ | Apache 2.0 | ||||||
| Synthetic data | |||||||||
| MS MARCO [50] | • | • | • | • | CC BY 4.0 |
Table 4. Data sources and available annotations. • indicates corpus metadata or synthesis annotations used to construct retrieval requirements and select positive and negative spoken documents. ◦ indicates classifier predictions used only to mask additional relevant in-batch documents during training.
Speech-attribute matching can enable unauthorized speaker tracking or profiling. Applications should obtain appropriate consent, protect recordings and embeddings, and avoid discriminatory uses. Dataset biases may cause unequal retrieval quality, so deployment should include evaluation across speaker groups and recording conditions.
Acknowledgments
We thank Yu-Xiang Lin from National Taiwan University for their helpful discussions and feedback.
This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan. We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources. This work was supported by the Ministry of Education (MOE) of Taiwan under the project Taiwan Centers of Excellence in Artificial Intelligence, through the NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE).
Data Construction
Data Sources
Table 4 summarizes the data sources, available annotations, and licenses for real and synthetic data. Speaking-style annotations include emotion and vocal states.
We use corpus metadata to construct retrieval requirements and select positive and negative spoken documents. Classifier predictions supplement these annotations only to identify and mask additional relevant in-batch documents during training, as described in § 3.3.
Real Data
Preprocessing. We remove uninformative labels such as unknown and other, along with recordings outside 1–120 s. We exclude environmental-sound classes containing speech and map equivalent labels across corpora, such as happy and joyful, to a common value.
Classifiers. To supplement incomplete corpus annotations, we use audEERING’s Wav2Vec2 age and gender model (Burkhardt et al., [8])1 to predict gender and emotion2vec+ large (Ma et al., [45])2 to predict emotion. For speakers with multiple recordings, we assign a shared predicted gender label through majority voting. Predicted labels are used only for relevance checks during training.
Synthetic Data
Text Selection. We sample 10,000 triplets from the MS MARCO (Nguyen et al., [50]) hard-triplet dataset processed by Sentence-Transformers.3 Each triplet contains a query, a relevant passage, and a hard negative passage. To reduce false negatives, the source dataset uses the ms-marco-MiniLM-L6-v2 cross-encoder4 to score the top 50 BM25-retrieved passages per query. It retains negatives with scores more than 3.0 below the positive passage’s score. We retain English queries and passages of 5–150 tokens and remove near-duplicates using ROUGE-L (Lin, [38]) with a threshold of 0.7.
Speech Synthesis. We synthesize speech from these texts using Gemini 2.5 Pro TTS (Comanici et al., [16])5, Gemini 3.1 Flash TTS,6 and Qwen3 TTS (Hu et al., [25]). Voice selection determines speaker identity. The two Gemini models share 30 voices, while Qwen3 TTS uses Ryan and Aiden, as detailed in Table 5.
| TTS model | Voice names |
| Gemini | Achernar, Achird, Algenib, Algieba, Alnilam, Aoede, Autonoe, Callirrhoe, Charon, Despina, Enceladus, Erinome, Fenrir, Gacrux, Iapetus, Kore, Laomedeia, Leda, Orus, Puck, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zephyr, Zubenelgenubi |
| Qwen3 TTS | Ryan, Aiden |
Table 5. Synthesis voices used for synthetic data construction.
Speaking Styles. Synthesis prompts control ten speaking styles: happy, sad, angry, fearful, surprised, disgusted, excited, sleepy, sarcastic, and whispering. Each prompt describes the voice, tone, pacing, and emotion. Table 6 provides the full prompts.
| Style | Prompt |
| Happy | Voice: bright, cheerful, and upbeat, as if you are smiling while you speak. Tone: warm, joyful, and openly affectionate. Pacing: lively and energetic, with buoyant rhythm. Emotion: genuinely delighted and optimistic, emphasizing key words with an uplifted intonation. |
| Sad | Voice: soft, low, and melancholic, with a gentle tremor. Tone: heavy, sorrowful, and subdued, as if holding back tears. Pacing: slow and drawn out, with thoughtful pauses. Emotion: deep sadness and quiet grief, letting the weight of the words settle. |
| Angry | Voice: sharp, intense, and forceful, with hard consonants. Tone: confrontational, tense, and biting. Pacing: clipped and driving, with emphatic stress on charged words. Emotion: strong anger held just under control, barely restrained frustration. |
| Fearful | Voice: trembling, breathy, and slightly higher in pitch. Tone: anxious, uncertain, and vulnerable. Pacing: uneven, with nervous hesitations and quick shallow breaths. Emotion: palpable fear, as if something dangerous is nearby and you are trying not to be heard. |
| Surprised | Voice: wide-eyed and astonished, with sudden lifts in pitch. Tone: amazed and caught off guard. Pacing: quick bursts punctuated by short breathy intakes. Emotion: genuine shock and disbelief, emphasizing unexpected words with emphatic stress. |
| Disgusted | Voice: curled-lipped and repulsed, with nasal resonance. Tone: scornful, dismissive, and revolted. Pacing: slightly drawn-out on offensive words, with sharp cutoffs elsewhere. Emotion: strong aversion, as if the subject is physically distasteful. |
| Excited | Voice: thrilled, animated, and high-energy, with rising intonation. Tone: enthusiastic and eager, barely able to contain yourself. Pacing: fast and breathless, tumbling from one idea to the next. Emotion: exuberant anticipation and joyful hype. |
| Sleepy | Voice: drowny, soft, and slightly slurred. Tone: lethargic, relaxed, and half-awake. Pacing: slow and drawn out, with yawning pauses. Emotion: deep tiredness, as if you are about to drift off mid-sentence. |
| Sarcastic | Voice: dry, flat, and ironic, with pointed emphasis. Tone: mocking and knowing, clearly meaning the opposite of the words. Pacing: deliberate, with exaggerated stress on key words. Emotion: wry amusement and condescending disbelief. |
| Whispering | Voice: quiet, breathy whisper with almost no vocal cord engagement. Tone: intimate, secretive, and conspiratorial. Pacing: careful and deliberate, with clear articulation despite the low volume. Emotion: hushed urgency, as if sharing a secret that must not be overheard. |
Table 6. Synthesis prompts for ten speaking styles, describing voice, tone, pacing, and emotion.
Environmental Sound. We use clips from eight UrbanSound8K (Salamon et al., [64]) classes: air conditioner, car horn, dog bark, drilling, engine idling, gunshot, jackhammer, and siren. Before cropping, we discard clips with root-mean-square levels below −45 dB relative to full scale. For each speech recording, we uniformly sample an environmental clip that passes this threshold.
Sound Mixing. We convert signals to mono by averaging channels and resample the environmental clip to match the speech sample rate. If the clip is longer than the speech, we randomly crop a segment matching the speech duration. Otherwise, we retain the full clip and place it at a uniformly sampled position within the speech. We uniformly sample a signal-to-noise ratio from {5, 10, 15} dB and scale the environmental signal using the mean squared amplitudes of both signals over the mixing interval.
Pair Construction
We construct retrieval requirements and document pairs from the annotations described in Appendix § 8.§ 1, following § 4.2.
We balance sampling across attribute keys and values in the retrieval requirements. For each spoken query , we sample a nonempty subset . To form , we may select attribute keys annotated for but not used in . For each selected key, we sample a value to be specified in the instruction. Speaker identity, dialogue, and answer relevance appear only in and refer to the spoken query. When includes speaker identity, we exclude gender and accent, since matching the speaker also guarantees matching these attributes in our data.
Pair Selection. For each spoken query and set of requirements , we select a positive document satisfying all requirements and a negative document violating at least one. We prioritize documents that violate fewer requirements in to obtain harder negatives.
Instruction Generation
We use Gemini 3.1 Pro Preview7 and Claude Opus 4.68 to express each set of requirements as natural-language instructions .
Two prompt variants produce short instructions and full-sentence instructions. We generate approximately 200 instructions per set of requirements before filtering and deduplication.
Prompt Variants. Table 7 presents the prompt templates, and Table 8 provides examples of generated instructions. Both templates distinguish query-matching requirements in from specified values in . The short-instruction template requests phrases of two to six words, while the full-sentence template requests conversational sentences. Both encourage diverse wording and avoid technical terms such as “criteria”, “attribute”, and “metadata”.
| Short instructions | Full-sentence instructions |
| You are generating {N} very short, informal retrieval cues for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Each cue should be an informal, telegraphic phrase a user might type in a chat or voice interface: keyword-style fragments, not full sentences. Requirements: 1. Very short, typically 2 to 6 words 2. Lowercase is fine, no trailing punctuation 3. No full sentences; use fragments, comma-separated keywords, or elliptical phrases 4. For MATCH constraints use phrases such as "same emotion", "matching accent" 5. For TARGET constraints use phrases such as "female", "happy", "british accent" 6. A natural example for this pattern is "{example_cue}" 7. Vary wording, word order, and synonyms across cues; do not repeat the example 8. Do NOT mention technical terms such as "requirements", "attribute", or "metadata" Output ONLY the cues, one per line, numbered 1 to {N}. | You are generating diverse natural language retrieval instructions for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Generate {N} diverse and natural retrieval instructions. Each instruction should: 1. Clearly indicate which attributes should MATCH the spoken query (e.g., "same emotion", "matching gender", "from the same speaker") 2. Clearly indicate which attributes should have a SPECIFIC TARGET value (e.g., "with a happy emotion", "in a British accent") 3. Use varied sentence structures, synonyms, and phrasings 4. Be natural and conversational 5. Not mention technical terms such as "requirements", "attribute", or "metadata" 6. Refer to "the spoken query" or "the given audio" when referring to the query Output ONLY the instructions, one per line, numbered 1 to {N}. |
Table 7. Prompt templates for generating short instructions and full-sentence instructions. Only terminology is adapted to match the paper. The templates distinguish query-matching requirements in from specified values in . Braces mark fields filled for each set of requirements, and N denotes the requested number of instructions.
| Requirements | Short instruction | Full-sentence instruction |
| Same speaker | identical voice | Fetch another clip of the speaker from this audio clip. |
| Same environmental sound | matching ambiance | Find audio that matches the background sound environment of this recording. |
| Same speaker, angry speaking style | anger, same voice | Search for an angry version of this person’s voice. |
Table 8. Examples of short instructions and full-sentence instructions for the same retrieval requirements. “Same” indicates a query-matching requirement in . The final row also includes a specified speaking style of “angry” in .
Filtering. We retain instructions that express all requirements in . Instructions must refer to the spoken query for and specify the values in . We allow synonyms and natural paraphrases of specified values, but query-matching requirements must identify the correct attribute. For example, accent matching must refer to “accent” rather than “speaking style” or “way of speaking.”
Deduplication. Within each pattern of requirements, we consider instructions in random order and discard any instruction with a ROUGE-L F1 score of at least 0.7 against a previously retained instruction.
Statistics
The dataset contains 899,931 samples, comprising 800,000 from real data and 99,931 from synthetic data. Table 9 reports the training and validation counts by source.
| Source | Training | Validation |
| Real data | ||
| VoxCeleb1 ([49]) | 294,303 | 2,974 |
| ESD ((Zhou et al., 2021)) | 115,282 | 1,182 |
| MELD ([57]) | 72,526 | 791 |
| CREMA-D ([11]) | 56,754 | 584 |
| L2-ARCTIC ((Zhao et al., 2018)) | 53,022 | 537 |
| LibriSpeech-C ([41]) | 51,553 | 521 |
| EmoV-DB ([1]) | 49,834 | 525 |
| MSP-IMPROV ([10]) | 44,763 | 459 |
| VocalSound ([22]) | 20,678 | 209 |
| SpokenWOZ ([67]) | 13,361 | 135 |
| MultiDialog ([55]) | 12,941 | 131 |
| UrbanSound8K ([64]) | 5,844 | 59 |
| EMNS ((Noriy et al., 2023)) | 1,022 | 10 |
| Real data subtotal | 791,883 | 8,117 |
| Synthetic data | ||
| MS MARCO ([50]) | 98,395 | 1,536 |
| Total | 890,278 | 9,653 |
Table 9. Training and validation sample counts by source. Each sample comprises a spoken query, an instruction, and positive and negative spoken documents.
Data Split. We shuffle the samples and keep those sharing the same spoken query in a single split. The resulting training and validation sets contain 890,278 and 9,653 samples, respectively.
Preventing Data Contamination. Our retrieval training data exclude test recordings from INSPIRE ([35]) and the 15 evaluated MAEB ([5]) tasks. We exclude MAEB tasks derived from CREMA-D and VoxCeleb because these corpora are included in our training data. Synthetic training uses MS MARCO text and UrbanSound8K sounds, whereas INSPIRE’s Synthetic subset uses Natural Questions text and ESC-50 sounds. The synthesis voices, text-to-speech models, and instruction-generation models also differ from those used in INSPIRE.
Training Details
Setup
We train IRIS-Embed using Qwen2.5-Omni-3B (Xu et al., 2025a)9 and Gemma-4-E2B-it [71]10 with the ms-swift package (Zhao et al., 2024)11. We optimize the contrastive objective in § 3.3 using the data splits in Appendix § 8.§ 6. We evaluate the final checkpoint after two epochs.
Optimization. Both models use fused AdamW with the same learning rate for all trainable components. Weight decay is not applied to biases or normalization parameters. Training uses bfloat16 precision and DeepSpeed ZeRO stage 2 across 16 NVIDIA GB200 GPUs. Table 10 summarizes the optimization settings.
| Setting | Value |
| Learning rate | |
| Adam coefficients | , |
| Adam epsilon | |
| Weight decay | 0.1 |
| Maximum gradient norm | 1.0 |
| Schedule | Linear warmup, then cosine decay to zero |
| Warmup | First 5% of optimizer steps |
| Training duration | 2 epochs; 3,476 optimizer steps |
| Global batch size | 512 samples |
| Batch size per GPU | 32 samples |
| Contrastive temperature | 0.03 |
Table 10. Training settings shared by both models. The global batch contains 32 samples on each of 16 GPUs.
Trainable Components. The audio layer weighting module uses the pretrained all-MiniLM-L6-v2 sentence encoder [61] (Wang et al. 2020)12. Both the audio encoder and sentence encoder remain frozen. We adapt the language model using LoRA with rank 64, scaling parameter 128, and dropout 0.05. The audio projection and the added normalization and linear layers in are trained directly alongside the LoRA parameters.
LoRA Configuration. For Qwen2.5-Omni-3B, we apply LoRA to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj across all 36 language-model layers. For Gemma-4-E2B-it, we adapt the applicable attention and feed-forward projections across all 35 language-model layers, along with per_layer_input_gate, per_layer_projection, and the shared per_layer_model_projection. Since later layers reuse keys and values, LoRA adapters for k_proj and v_proj are applied only to the first 15 layers.
Data Ordering. Before training, we arrange samples by source and spoken query in groups of approximately 64. Grouping by source emphasizes within-source negatives, particularly for smaller datasets. For dialogue continuation, this ordering is intended to reduce shortcuts based on speaker identity or recording conditions. Grouping by spoken query places different instructions for the same query close together. Different instructions may select different positive documents, so this ordering is intended to encourage retrieval to depend on the instruction’s requirements.
Audio Layer Weighting
Computation. To compute , we mean-pool MiniLM’s non-padding content hidden states, then apply learned layer normalization and a linear layer. We apply softmax to obtain weights for the pre-encoder hidden states and all audio-encoder layer hidden states: 33 hidden-state sequences in Qwen2.5-Omni-3B and 13 in Gemma-4-E2B-it. Each model pools and projects the weighted audio hidden states using its existing operations, including normalization, before passing them to the language model.
Initialization. We initialize the linear weights from a Gaussian distribution with mean 0 and standard deviation . For Qwen2.5-Omni-3B, all biases are initialized to zero, yielding approximately uniform layer weights. For Gemma-4-E2B-it, we initialize the layer weights to emphasize the final audio layer, reducing the initial deviation from its original representation. Let denote the number of audio hidden-state sequences, including the pre-encoder hidden states. We set the final-layer bias to
while initializing all other biases to zero. For Gemma-4-E2B-it, , so this initialization assigns approximately 90% of the initial weight to the final layer and distributes the remainder across other layers.
Prompts
Table 11 presents our prompt templates, adapted from Qwen3-VL-Embedding [36] to each model’s chat and audio formats. The query prompt includes the instruction and spoken query, while the document prompt contains the spoken document. We extract the final-layer hidden state of the last non-padding input token as the embedding without generating a response.
Qwen2.5-Omni-3B Query Prompt: | Qwen2.5-Omni-3B Document Prompt: |
<|im_start|>user<br>Instruction: [instruction]<br>Query: <|audio_bos|>[audio_tokens]<|audio_eos|><br><|im_end|><br><|im_start|>assistant | <|im_start|>user<br>Document: <|audio_bos|>[audio_tokens]<br><|audio_eos|><|im_end|><br><|im_start|>assistant |
Gemma-4-E2B-it Query Prompt: | Gemma-4-E2B-it Document Prompt: |
<bos><|turn|>user<br>Instruction: [instruction]<br>Query: [audio_tokens]<turn|><br><|turn|>model | <bos><|turn|>user<br>Document: [audio_tokens]<turn|><br><|turn|>model |
MiniLM Query Prompt: | MiniLM Document Prompt: |
Instruction: [instruction]<br>Query: | Document: |
Table 11. IRIS-Embed prompt templates. Red: input labels; blue: placeholders.
For audio layer weighting, MiniLM processes text-only prompts. The query prompt includes the instruction, while the document prompt remains fixed across all spoken documents.
Evaluation Details
INSPIRE
Benchmark. INSPIRE [35]13 evaluates spoken-document retrieval given a spoken query and a natural-language instruction. Instructions describe requirements for speech attributes. Thus, the same spoken query may have different relevant documents depending on the instruction. A document is relevant only if it satisfies all requirements.
Tasks. DailyTalk [33] uses the first half of a dialogue as the spoken query and evaluates retrieval of its continuation. VCTK (Yamagishi et al., 2019) evaluates speaker matching using hard negatives with the same spoken content but different speakers. Expresso [51] evaluates speaker and speaking-style matching, separately or jointly, including instructions with specified speaking styles.
The Synthetic subset combines spoken queries and documents synthesized from Natural Questions [29] using GPT-4o mini TTS [54] with environmental sounds from ESC-50 [56]. Synthetic evaluates answer retrieval and matching by speaker, speaking style, and environmental sound, including instructions with multiple attributes. Speaking style and environmental sound may match the spoken query or take values specified in the instruction. Appendix § 8.§ 6 describes the separation between training and evaluation data.
Metric. For each query–instruction pair, embedding methods rank documents in the corresponding subset by cosine similarity. Recall@ measures the fraction of relevant documents retrieved among the top results. Let denote the evaluated query–instruction pairs, the top retrieved documents, and the relevant documents. We compute average recall as
Each query–instruction pair contributes equally to the average. We report Recall@ as percentages for on each subset. For the instruction-type analysis in § 6.1, we compute Recall@50 separately for each instruction type.
Comparison Settings
Table 1 includes published baseline results.
Random. The random baseline averages uniform random document rankings over 100 runs [35].
General Speech Encoders. WavLM-large [14]14 encodes spoken queries and documents without instructions. Embeddings are obtained by mean-pooling its final-layer hidden states over time.
Specialized Speech Encoders. ECAPA-TDNN [17]15 and emotion2vec-base [45]16 extract speaker and emotion embeddings from their final-layer representations, respectively. Both encode spoken queries and documents without instructions.
Cascaded Retrieval. Whisper-large-v3 [58]17 generates transcripts, and Qwen3-Omni-30B-A3B-Captioner (Xu et al., 2025b)18 generates captions for spoken queries and documents. Qwen3-Embedding-8B (Zhang et al., 2025b)19 encodes the transcripts and captions, with the instruction included in the query input.
LALM Embeddings. Qwen2.5-Omni-3B (Xu et al., 2025a)20, Gemma-4-E2B-it [71]21, and Qwen3-Omni-30B (Xu et al., 2025b)22 follow INSPIRE’s evaluation setup [35] without additional embedding training. For these models, query and document inputs end with “Summarize the above sentence and speech in one word” and “Summarize the above speech in one word”, respectively. LCO-Embedding-Omni-3B (Xiao et al., 2025)23 follows its published embedding extraction configuration. All four methods extract embeddings from the final-layer hidden state of the last non-padding input token without generating a response.
IRIS-Embed. We evaluate the final IRIS-Embed checkpoints from Appendix § 9 with learned audio layer weighting. Table 11 gives the query and document prompt templates. We extract the final-layer hidden state of the last non-padding input token as the embedding without generating a response.
Layer Aggregation
Table 2 compares audio layer aggregation for Qwen2.5-Omni-3B. Layer 13 and Final Layer use the 13th and final audio-encoder hidden states, respectively, followed by pooling and projection into the language model. Mean Layer assigns equal weights to the pre-encoder hidden states and all layer hidden states. Finetuned updates the audio encoder during retrieval training, while the other variants keep it frozen. Static Weight learns a single set of layer weights shared across all queries and documents, independent of instructions or prompts. Inst. Weight is our instruction-aware audio layer weighting method, which predicts weights from text prompts as described in § 3.2.1. Query weights vary with the instruction, while document weights are computed from a fixed prompt. Mean Layer, Static Weight, and Inst. Weight combine layers 0–32 for Qwen2.5-Omni-3B, with layer 0 denoting the pre-encoder hidden states. Both weighting variants use the same instructions in the language-model query prompt.
Final Layer, Mean Layer, Static Weight, and Inst. Weight use the training data and language-model LoRA settings in Appendix § 9 and train the audio projection. For Gemma-4-E2B-it, we compare Final Layer, Mean Layer, and Inst. Weight. Appendix § 11.§ 3 provides the full results for both models. Appendix § 11.§ 4 separately examines frozen and trainable audio projections for Final Layer.
MAEB
Tasks. We evaluate 15 audio tasks from MAEB [5]24, following the task definitions and metrics of its audio-only beta leaderboard.25 We exclude tasks derived from CREMA-D [11] and VoxCeleb [49], as these corpora are included in our retrieval training data. Table 3 reports scores on a 0–100 scale using shortened task names.
Baseline Scores. We evaluate the Qwen2.5-Omni-3B and Gemma-4-E2B-it baselines with the same task instructions as IRIS-Embed. Scores for the other baselines come from the MAEB beta leaderboard and are averaged uniformly across language subsets for each task.
Embedding Extraction. Table 12 lists the full task names and corresponding instructions. For all tasks except retrieval, we encode every recording with its task instruction using the query prompt in Table 11. Retrieval uses separate query and document prompts, as described below. Each input passes through the full IRIS-Embed model, and we extract the final-layer hidden state of the last non-padding input token as its embedding. Model parameters remain fixed throughout evaluation.
| Task | Type | Instruction |
| BeijingOpera | Clf | Find recordings with the same percussion instrument as the query. |
| BirdCLEF | Clf | Find recordings with the same bird species as the query. |
| CommonLanguageAgeDetection | Clf | Find recordings with the same speaker age group as the query. |
| GTZANGenre | Clf | Find recordings with the same music genre as the query. |
| IEMOCAPGender | Clf | Find recordings with the same gender as the query. |
| MInDS14 | Clf | Find recordings with the same intent as the query. |
| MridinghamTonic | Clf | Find recordings with the same tonic as the query. |
| SIBFLEURS | Clf | Find recordings with the same topic as the query. |
| VoxPopuliLanguageID | Clf | Find recordings with the same spoken language as the query. |
| VehicleSoundClustering | Clust | Find recordings with the same environmental sound as the query. |
| VoxPopuliGenderClustering | Clust | Find recordings with the same gender as the query. |
| NMSQAPairClassification | PC | Find recordings that contain the answer to the query. |
| VoxPopuliAccentPairClassification | PC | Find recordings with the same accent as the query. |
| GTZANAudioReranking | Rrnk | Find recordings with the same music genre as the query. |
| JamAltArtistA2ARetrieval | Rtrvl | Find recordings with the same artist as the query. |
Table 12. Task instructions used to extract IRIS-Embed query embeddings for MAEB evaluation. Clf: classification; Clust: clustering; PC: pair classification; Rrnk: reranking; Rtrvl: retrieval.
Classification. Following MAEB (Assadi et al., [5], nine tasks evaluate classification of percussion instruments [72], bird species [28], speaker age groups [68], music genres (Tzanetakis and Cook, 2002), gender [9], intent [20], musical tonic [2], topic [65], and language (Wang et al., 2021). Evaluation uses a logistic-regression probe fitted on fixed embeddings with eight samples per class. Accuracy is averaged over ten resamples.
Clustering. VehicleSound [7] and VoxPopuliGender (Wang et al., 2021) evaluate clustering by sound class and gender, respectively. Following MAEB, MiniBatchKMeans uses the number of reference classes as the number of clusters. Performance is measured by V-measure [62] over bootstrapped subsamples.
Pair Classification. NMSQA [40] and VoxPopuliAccent (Wang et al., 2021) evaluate the relevance of recording pairs. Following MAEB, we report the highest average precision across the benchmark’s similarity measures.
Reranking. GTZAN (Tzanetakis and Cook, 2002) evaluates reranking of a provided set of relevant documents and hard negatives using cosine similarity. We report mean average precision at 1,000, denoted MAP@1000.
Retrieval. JamAlt [15] evaluates audio-to-audio retrieval of songs by the same artist as the query. The query prompt includes the query recording and task instruction, while the document prompt contains the candidate recording without an instruction. We report normalized discounted cumulative gain at 10, denoted nDCG@10.
Full Results
Instruction Types
Table 13 reports Recall@50 by instruction type on INSPIRE, covering instructions with a single attribute or multiple attributes. Relative to the corresponding LALM embedding baselines, IRIS-Embed improves Recall@50 in all 21 evaluation settings for Qwen2.5-Omni-3B and 20 of 21 for Gemma-4-E2B-it. The exception is matching speaking style with a specified environmental sound on Synthetic, where Gemma-4-E2B-it achieves 1.25% with IRIS-Embed versus 1.50% with the LALM baseline.
| Method | Model | DailyTalk Cont. | VCTK Spk. | Expresso Spk. | Expresso Sty. | Expresso Spk. +Sty. | Expresso Spk. +Sty.* | Synthetic Ans. | Synthetic Spk. | Synthetic Sty. | Synthetic Env. | |
| RAND | Random | 1.06 | 1.84 | 1.73 | 1.46 | 1.38 | 1.42 | 0.84 | 1.09 | 1.38 | 1.04 | |
| ENC | WavLM-large | 24.50 | 35.00 | 11.50 | 30.75 | 27.50 | 1.00 | 0.75 | 0.75 | 8.75 | 7.50 | |
| SENC | ECAPA-TDNN | 24.00 | 100.00 | 71.50 | 24.25 | 70.50 | 0.75 | 1.67 | 87.75 | 13.50 | 1.75 | |
| SENC | emotion2vec-base | 23.00 | 6.25 | 3.25 | 6.25 | 7.25 | 0.75 | 16.75 | 2.25 | 18.75 | 2.25 | |
| CAS | Qwen3-Embedding-8B | 81.50 | 6.25 | 4.00 | 5.50 | 6.00 | 6.75 | 98.50 | 1.75 | 1.00 | 4.50 | |
| LALM | Qwen2.5-Omni-3B | 53.00 | 3.13 | 2.00 | 6.00 | 3.75 | 3.25 | 64.75 | 1.00 | 1.25 | 0.50 | |
| LALM | Gemma-4-E2B-it | 25.50 | 4.38 | 1.50 | 1.25 | 0.75 | 1.25 | 50.92 | 0.50 | 1.50 | 0.50 | |
| LALM | Qwen3-Omni-30B | 74.50 | 3.13 | 2.00 | 5.25 | 4.50 | 2.50 | 80.75 | 1.25 | 2.00 | 2.00 | |
| LALM | LCO-Embedding-Omni-3B | 75.50 | 2.50 | 2.25 | 16.25 | 8.25 | 7.00 | 97.25 | 1.25 | 1.75 | 6.00 | |
| IRIS | Qwen2.5-Omni-3B | 77.50 | 85.00 | 41.25 | 42.00 | 62.00 | 34.75 | 90.50 | 53.50 | 53.75 | 35.50 | |
| IRIS | Gemma-4-E2B-it | 44.50 | 74.38 | 33.00 | 48.00 | 63.75 | 30.50 | 80.58 | 51.25 | 47.25 | 8.50 | |
| Method | Model | Synthetic Spk. +Sty. | Synthetic Spk. +Sty.* | Synthetic Spk. +Env. | Synthetic Spk. +Env.* | Synthetic Sty. +Env. | Synthetic Sty. +Env.* | Synthetic Sty.* +Env. | Synthetic Spk. +Sty. +Env. | Synthetic Spk. +Sty.* +Env. | Synthetic Spk. +Sty. +Env.* | Synthetic Spk. +Sty.* +Env.* |
| RAND | Random | 1.04 | 1.04 | 0.96 | 0.96 | 0.90 | 0.96 | 0.96 | 0.92 | 0.98 | 0.98 | 0.91 |
| ENC | WavLM-large | 6.25 | 1.50 | 7.25 | 0.75 | 9.00 | 1.00 | 3.25 | 12.25 | 5.50 | 2.25 | 0.50 |
| SENC | ECAPA-TDNN | 61.50 | 2.00 | 7.00 | 4.50 | 2.75 | 3.75 | 0.00 | 20.50 | 1.25 | 13.50 | 1.00 |
| SENC | emotion2vec-base | 2.75 | 1.75 | 4.00 | 1.25 | 5.50 | 0.50 | 1.25 | 6.00 | 2.25 | 1.75 | 1.00 |
| CAS | Qwen3-Embedding-8B | 2.50 | 1.50 | 3.00 | 2.75 | 2.00 | 2.75 | 3.25 | 4.25 | 1.75 | 2.25 | 2.50 |
| LALM | Qwen2.5-Omni-3B | 1.50 | 0.50 | 0.50 | 0.75 | 2.00 | 1.25 | 0.25 | 0.50 | 0.75 | 1.50 | 1.00 |
| LALM | Gemma-4-E2B-it | 1.75 | 1.50 | 1.00 | 2.00 | 1.75 | 1.50 | 1.50 | 0.75 | 1.25 | 0.75 | 1.25 |
| LALM | Qwen3-Omni-30B | 1.00 | 0.50 | 0.50 | 0.75 | 1.25 | 0.75 | 0.75 | 1.00 | 1.25 | 0.75 | 1.25 |
| LALM | LCO-Embedding-Omni-3B | 1.75 | 1.00 | 2.75 | 3.00 | 6.25 | 3.00 | 2.50 | 4.50 | 2.75 | 4.25 | 2.50 |
| IRIS | Qwen2.5-Omni-3B | 54.25 | 46.00 | 47.50 | 17.25 | 35.75 | 15.00 | 38.00 | 61.75 | 57.00 | 30.00 | 26.75 |
| IRIS | Gemma-4-E2B-it | 47.00 | 45.25 | 15.50 | 7.25 | 13.50 | 1.25 | 8.25 | 31.50 | 18.25 | 15.00 | 13.75 |
Table 13. Full Recall@50 results by instruction type on INSPIRE. Bold and underline indicate the best and second-best results, respectively. Cont.: dialogue continuation; Ans.: answer retrieval; Spk.: speaker; Sty.: speaking style; Env.: environmental sound. + combines requirements; ∗ marks a specified value.
IRIS-Embed also outperforms other baselines across several instruction types. Both models outperform all LALM embedding baselines on speaker matching in VCTK and all four instruction types in Expresso. Both models also outperform all comparison baselines on speaking-style matching in Expresso and Synthetic, and environmental-sound matching in Synthetic.
For instructions with multiple attributes, the Qwen2.5-Omni-3B and Gemma-4-E2B-it versions of IRIS-Embed outperform all comparison baselines in 11 and 10 of the 13 evaluation settings across Expresso and Synthetic, respectively. On Synthetic, matching the spoken query’s speaker, speaking style, and environmental sound together gives Recall@50 of 61.75% and 31.50%, compared with 20.50% for the strongest baseline, ECAPA-TDNN. Matching the speaker and a specified speaking style yields 46.00% and 45.25%, compared with 2.00% for the strongest baseline. The gains therefore extend to requirements with query-matched and explicitly specified attribute values.
Instruction Awareness
We assess whether IRIS-Embed follows retrieval instructions by varying the instruction while keeping the spoken queries, document collection, and relevance labels fixed for each set of evaluation requirements. The tables report Recall@50, with instructions in rows and evaluation requirements in columns. Table 1 and Table 13 use the original INSPIRE instructions. Here, we replace them with fixed short or full-sentence instructions, so recall may differ from those results.
We consider five instructions with a single attribute: dialogue continuation, answer retrieval, speaker matching, speaking-style matching, and environmental-sound matching. Dialogue continuation and answer retrieval specify different relevance requirements for spoken content. Table 14 lists a short instruction and a full-sentence instruction for each of the five instruction types. The left sides of Figure 5 and Figure 7 show the Short rows from Table 15 for Qwen2.5-Omni-3B and Gemma-4-E2B-it, respectively. For both models, the instruction matching the evaluation requirements achieves the highest Recall@50 across all four Synthetic evaluation settings.
Figure 7. Recall@50 for IRIS-Embed with Gemma-4-E2B-it. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound; +: multiple attributes.
| Label | Short instruction | Full-sentence instruction |
| Cont. | Continuation. | Find speech that continues the conversation in the query. |
| Ans. | Find the answer. | Find speech that contains the answer to the question in the query. |
| Spk. | Same speaker. | Find speech from the same speaker as the query. |
| Sty. | Same speaking style. | Find speech with the same speaking style as the query. |
| Env. | Same environmental sound. | Find speech with the same environmental sound as the query. |
| Doc. | Document: |
Table 14. Instructions for instruction awareness and layer-weight analysis. The main-paper instruction-awareness and layer-weight analyses use short instructions. Doc. denotes the document prompt used in the layer-weight analysis.
| Instruction | Form | DailyTalk Cont. | VCTK Spk. | Expresso Spk. | Expresso Sty. | Expresso Ans. | Synthetic Spk. | Synthetic Sty. | Synthetic Env. |
| Qwen2.5-Omni-3B | |||||||||
| Continuation. Find speech that continues the conversation in the query. | Short | 79.00 | 81.88 | 19.75 | 22.50 | 87.25 | 2.75 | 2.75 | 2.75 |
| Full | 77.50 | 83.75 | 31.00 | 15.75 | 63.00 | 18.75 | 2.50 | 2.00 | |
| Find the answer. Find speech that contains the answer to the question in the query. | Short | 82.50 | 73.75 | 8.00 | 4.25 | 90.25 | 2.00 | 1.75 | 1.25 |
| Full | 78.50 | 67.50 | 6.25 | 3.50 | 91.00 | 2.00 | 1.50 | 0.25 | |
| Same speaker. Find speech from the same speaker as the query. | Short | 63.50 | 85.63 | 40.75 | 13.50 | 1.50 | 53.00 | 5.50 | 0.75 |
| Full | 67.50 | 85.63 | 41.00 | 13.50 | 1.50 | 53.00 | 6.00 | 0.75 | |
| Same speaking style. Find speech with the same speaking style as the query. | Short | 10.50 | 30.00 | 4.00 | 42.50 | 0.50 | 2.75 | 55.00 | 0.50 |
| Full | 21.00 | 30.00 | 3.00 | 41.00 | 0.50 | 2.75 | 55.75 | 1.00 | |
| Same environmental sound. Find speech with the same environmental sound as the query. | Short | 20.50 | 34.38 | 4.50 | 8.75 | 1.50 | 0.75 | 0.50 | 36.75 |
| Full | 29.00 | 35.63 | 4.75 | 10.00 | 1.50 | 1.00 | 0.50 | 37.25 | |
| No instruction | 75.00 | 80.63 | 20.75 | 26.00 | 74.58 | 3.25 | 3.75 | 8.75 | |
| Gemma-4-E2B-it | |||||||||
| Continuation. Find speech that continues the conversation in the query. | Short | 48.50 | 76.25 | 17.75 | 16.75 | 71.08 | 5.75 | 1.75 | 2.00 |
| Full | 49.00 | 75.63 | 19.25 | 13.75 | 73.83 | 3.50 | 1.25 | 1.25 | |
| Find the answer. Find speech that contains the answer to the question in the query. | Short | 42.00 | 35.00 | 6.25 | 6.00 | 82.58 | 0.75 | 2.50 | 1.00 |
| Full | 41.50 | 36.25 | 6.75 | 6.00 | 83.08 | 1.00 | 2.25 | 1.25 | |
| Same speaker. Find speech from the same speaker as the query. | Short | 39.50 | 75.00 | 32.50 | 17.00 | 1.00 | 51.50 | 4.75 | 2.25 |
| Full | 45.50 | 75.63 | 32.25 | 16.75 | 1.50 | 52.75 | 4.75 | 2.50 | |
| Same speaking style. Find speech with the same speaking style as the query. | Short | 15.00 | 56.25 | 6.50 | 50.50 | 0.25 | 5.00 | 47.75 | 3.00 |
| Full | 22.50 | 54.38 | 7.25 | 47.25 | 0.25 | 4.50 | 46.75 | 2.50 | |
| Same environmental sound. Find speech with the same environmental sound as the query. | Short | 35.00 | 31.88 | 4.25 | 17.75 | 1.00 | 2.00 | 3.25 | 9.75 |
| Full | 41.00 | 35.63 | 5.00 | 14.00 | 1.00 | 1.75 | 2.50 | 9.50 | |
| No instruction | 44.00 | 75.63 | 20.00 | 39.00 | 8.75 | 21.25 | 24.25 | 3.75 |
Table 15. Recall@50 for IRIS-Embed with short instructions, full-sentence instructions, or no instruction. Rows show the exact instructions; Short/Full denote short/full-sentence instructions. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound.
Table 15 reports both wording forms, labeled Short and Full, and a no-instruction condition. Within each wording form, the matching instruction achieves the highest Recall@50 for both models across all four Synthetic evaluation settings and for speaker and speaking-style matching on Expresso. The matching instruction also performs best for Qwen2.5-Omni-3B on VCTK and Gemma-4-E2B-it on DailyTalk. Both short instructions and full-sentence instructions therefore guide retrieval according to the requested requirements in most evaluation settings.
The matching instruction does not always achieve the highest recall. On DailyTalk, Qwen2.5-Omni-3B performs best with the answer-retrieval instruction under both wording forms. This may reflect overlap between answering a question and continuing a dialogue. On VCTK, Gemma-4-E2B-it achieves 76.25% with the short continuation instruction and 75.00% with the short speaker-matching instruction. The corresponding full-sentence instructions and no instruction all yield 75.63%, suggesting that speaker information remains available without an explicit speaker-matching instruction.
We further evaluate four short instructions with multiple attributes of the spoken query. These cover all pairwise combinations of speaker, speaking style, and environmental sound, as well as all three attributes together. The right panels of Figure 5 and Figure 7 show the rows for these four short instructions in Table 16 across the corresponding Synthetic evaluation settings. The matching instruction achieves the highest Recall@50 in all four evaluation settings for Qwen2.5-Omni-3B and three of four for Gemma-4-E2B-it. When relevance requires matching all three attributes, the corresponding instruction outperforms all pairwise instructions for both models.
| Instruction | Qwen2.5-Omni-3B Spk. +Sty. | Qwen2.5-Omni-3B Spk. +Env. | Qwen2.5-Omni-3B Sty. +Env. | Qwen2.5-Omni-3B Spk. +Sty. +Env. | Gemma-4-E2B-it Spk. +Sty. | Gemma-4-E2B-it Spk. +Env. | Gemma-4-E2B-it Sty. +Env. | Gemma-4-E2B-it Spk. +Sty. +Env. |
| Find the answer. | 0.75 | 1.75 | 3.25 | 2.00 | 1.00 | 1.25 | 1.75 | 1.50 |
| Continuation. | 1.50 | 4.50 | 5.25 | 6.50 | 2.75 | 3.75 | 3.25 | 4.50 |
| Same speaker. | 18.50 | 8.50 | 2.50 | 10.25 | 18.00 | 8.50 | 4.00 | 14.25 |
| Same speaking style. | 9.50 | 2.25 | 3.25 | 7.00 | 12.25 | 4.50 | 7.75 | 12.25 |
| Same environmental sound. | 1.50 | 10.75 | 15.00 | 11.75 | 2.50 | 6.75 | 9.25 | 10.50 |
| Same speaker and same speaking style. | 52.00 | 11.50 | 7.50 | 27.25 | 49.75 | 12.75 | 5.25 | 28.00 |
| Same speaker and same environmental sound. | 8.25 | 45.25 | 11.75 | 45.25 | 15.50 | 13.25 | 4.50 | 22.00 |
| Same speaking style and same environmental sound. | 6.00 | 11.50 | 35.00 | 36.75 | 7.75 | 5.25 | 11.75 | 13.75 |
| Same speaker and same speaking style and same environmental sound. | 30.00 | 29.25 | 22.25 | 65.00 | 46.75 | 13.50 | 6.00 | 33.25 |
| No instruction | 6.00 | 13.25 | 9.50 | 17.75 | 35.75 | 11.25 | 7.75 | 30.25 |
Table 16. Recall@50 for IRIS-Embed under short instructions and no instruction. Columns evaluate Synthetic requirements involving multiple attributes. Rows show the exact instructions; No instruction indicates an empty instruction. Spk., Sty., and Env. match the spoken query’s speaker, speaking style, and environmental sound; + combines requirements.
Table 16 provides the full comparison, including the five short instructions above and no instruction. For both models, the matching instruction outperforms these additional conditions in all four evaluation settings. These findings show that IRIS-Embed responds to instructions with multiple attributes.
Layer Aggregation
Table Table 17 reports Recall@10, Recall@50, and Recall@100 for audio layer aggregation, while Table 18 provides Recall@50 by instruction type. For Qwen2.5-Omni-3B, the comparison includes Layer 13, Final Layer, Mean Layer, Finetuned, Static Weight, and Inst. Weight. For Gemma-4-E2B-it, we compare Final Layer, Mean Layer, and our method. Appendix § 10.§ 3 describes these methods and their training settings.
| Layer Aggregation | DailyTalk R@10 | DailyTalk R@50 | DailyTalk R@100 | VCTK R@10 | VCTK R@50 | VCTK R@100 | Expresso R@10 | Expresso R@50 | Expresso R@100 | Synthetic R@10 | Synthetic R@50 | Synthetic R@100 | Avg. R@10 | Avg. R@50 | Avg. R@100 |
| Qwen2.5-Omni-3B | |||||||||||||||
| Layer 13 | 5.50 | 15.50 | 22.50 | 61.88 | 76.88 | 84.38 | 32.44 | 48.94 | 57.75 | 17.95 | 35.27 | 46.12 | 29.44 | 44.15 | 52.69 |
| Final Layer | 59.50 | 82.50 | 85.50 | 34.38 | 48.75 | 59.38 | 15.88 | 28.25 | 36.88 | 18.47 | 34.72 | 45.20 | 32.06 | 48.56 | 56.74 |
| Mean Layer | 47.50 | 70.00 | 78.50 | 57.50 | 71.25 | 81.25 | 25.75 | 43.13 | 52.63 | 26.03 | 44.90 | 55.00 | 39.20 | 57.32 | 66.85 |
| Finetuned | 14.50 | 32.50 | 42.00 | 76.25 | 91.25 | 95.00 | 31.19 | 47.38 | 55.50 | 24.90 | 43.35 | 54.49 | 36.71 | 53.62 | 61.75 |
| Static Weight | 45.00 | 66.00 | 76.50 | 61.88 | 78.75 | 87.50 | 25.88 | 43.19 | 55.31 | 26.47 | 45.28 | 56.13 | 39.81 | 58.31 | 68.86 |
| Inst. Weight (Ours) | 60.00 | 77.50 | 87.50 | 67.50 | 85.00 | 89.38 | 29.63 | 45.00 | 53.88 | 26.06 | 44.17 | 54.78 | 45.80 | 62.92 | 71.39 |
| Gemma-4-E2B-it | |||||||||||||||
| Final Layer | 24.50 | 42.50 | 52.00 | 30.63 | 48.13 | 59.38 | 17.06 | 30.00 | 37.69 | 12.01 | 21.70 | 29.72 | 21.05 | 35.58 | 44.70 |
| Mean Layer | 12.50 | 25.00 | 37.00 | 52.50 | 69.38 | 78.13 | 32.00 | 46.06 | 54.63 | 11.63 | 21.65 | 29.38 | 27.16 | 40.52 | 49.79 |
| Inst. Weight (Ours) | 25.50 | 44.50 | 54.50 | 58.75 | 74.38 | 79.38 | 28.63 | 43.81 | 52.31 | 15.07 | 26.94 | 36.08 | 31.99 | 47.41 | 55.57 |
Table 17. Full Recall@ for audio layer aggregation on INSPIRE. Avg.: macro-average across the four subsets. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Bold marks column maxima within each backbone.
For Qwen2.5-Omni-3B, our method achieves the highest macro-average Recall@50 of 62.92% in Table 2, a gain of 14.36 percentage points over Final Layer. It consistently outperforms Final Layer on VCTK, Expresso, and Synthetic. Final Layer remains higher on DailyTalk at Recall@50.
Layer 13 and Finetuned improve speaker and speaking-style matching over Final Layer, but have much lower dialogue-continuation recall. Fine-tuned consistently leads on VCTK and Layer 13 on Expresso. Our method consistently outperforms both variants on DailyTalk and Synthetic. These methods trade off performance across retrieval requirements.
Our method also exceeds Mean Layer and Static Weight in Recall@50 on DailyTalk, VCTK, and Expresso. The macro-average Recall@50 gains are 5.60 and 4.61 percentage points, respectively. Both alternatives remain higher on Synthetic at Recall@50. Static Weight also exceeds our method on Synthetic at Recall@10 and Recall@100, and on Expresso at Recall@100.
The instruction-type results further distinguish these methods. On answer retrieval, our method achieves 90.50% Recall@50, compared with 68.25% for Finetuned and 2.00% for Layer 13. In the pairwise comparison with Static Weight, our method has higher Recall@50 in seven of eight evaluation settings with a single attribute. For instructions with multiple attributes, Static Weight has higher recall in eight of thirteen settings, our method in four, and one is tied.
With Gemma-4-E2B-it, our method consistently outperforms Final Layer on VCTK, Expresso, and Synthetic, while Mean Layer leads on Expresso. Answer retrieval Recall@50 increases from 1.50% with Mean Layer to 80.58% with our method. However, Mean Layer performs better on several instructions that combine speaker matching with other attributes. The aggregate gains support the effectiveness of instruction-aware audio layer weighting with a frozen audio encoder.
Audio Projection
We examine whether training the audio projection improves retrieval using the final audio-encoder hidden states. Both variants freeze the audio encoder and train the language-model LoRA parameters. Frozen keeps the audio projection fixed, while Trainable updates it during retrieval training. Table 19 and Table 20 report results by subset and instruction type, respectively.
| Audio Projection | DailyTalk | DailyTalk | DailyTalk | VCTK | VCTK | VCTK | Expresso | Expresso | Expresso | Synthetic | Synthetic | Synthetic | Avg. | Avg. | Avg. |
| Qwen2.5-Omni-3B | |||||||||||||||
| Frozen | 69.00 | 83.00 | 87.50 | 24.38 | 41.88 | 56.88 | 14.13 | 26.50 | 36.50 | 16.83 | 30.80 | 42.00 | 31.09 | 45.55 | 55.72 |
| Trainable | 59.50 | 82.50 | 85.50 | 34.38 | 48.75 | 59.38 | 15.88 | 28.25 | 36.88 | 18.47 | 34.72 | 45.20 | 32.06 | 48.56 | 56.74 |
| Gemma-4-E2B-it | |||||||||||||||
| Frozen | 17.00 | 34.00 | 46.00 | 24.38 | 45.00 | 58.13 | 13.63 | 24.69 | 31.75 | 9.06 | 15.83 | 21.78 | 16.02 | 29.88 | 39.42 |
| Trainable | 24.50 | 42.50 | 52.00 | 30.63 | 48.13 | 59.38 | 17.06 | 30.00 | 37.69 | 12.01 | 21.70 | 29.72 | 21.05 | 35.58 | 44.70 |
Table 19. Recall@ on INSPIRE using only the final audio-encoder hidden states. Avg.: macro-average across the four subsets. Frozen and Trainable indicate whether the audio projection is frozen or updated during retrieval training. Bold marks column maxima within each backbone.
| Audio Projection | DailyTalk Cont. | VCTK Spk. | Expresso Spk. | Expresso Sty. | Expresso Spk. +Sty. | Expresso Spk. +Sty.* | Synthetic Ans. | Synthetic Spk. | Synthetic Sty. | Synthetic Env. |
| Qwen2.5-Omni-3B — Frozen | 83.00 | 41.88 | 14.50 | 34.25 | 39.25 | 18.00 | 92.25 | 24.00 | 39.75 | 29.25 |
| Qwen2.5-Omni-3B — Trainable | 82.50 | 48.75 | 11.25 | 39.75 | 44.75 | 17.25 | 94.50 | 27.50 | 40.50 | 31.00 |
| Gemma-4-E2B-it — Frozen | 34.00 | 45.00 | 9.75 | 35.25 | 37.75 | 16.00 | 84.75 | 17.25 | 39.25 | 0.75 |
| Gemma-4-E2B-it — Trainable | 42.50 | 48.13 | 13.50 | 45.50 | 43.25 | 17.75 | 83.25 | 28.75 | 37.50 | 7.75 |
| Audio Projection | Synthetic Spk. +Sty. | Synthetic Spk. +Sty.* | Synthetic Spk. +Env. | Synthetic Spk. +Env.* | Synthetic Sty. +Env. | Synthetic Sty. +Env.* | Synthetic Sty.* +Env. | Synthetic Spk. +Sty. +Env. | Synthetic Spk. +Sty.* +Env. | Synthetic Spk. +Sty. +Env.* | Synthetic Spk. +Sty.* +Env.* |
| Qwen2.5-Omni-3B — Frozen | 26.75 | 23.00 | 30.00 | 17.00 | 26.50 | 12.00 | 28.50 | 39.00 | 30.25 | 25.75 | 18.00 |
| Qwen2.5-Omni-3B — Trainable | 30.75 | 27.25 | 35.75 | 19.50 | 28.50 | 14.00 | 29.50 | 45.75 | 38.50 | 32.50 | 25.25 |
| Gemma-4-E2B-it — Frozen | 26.25 | 19.50 | 5.00 | 3.50 | 3.00 | 2.50 | 3.75 | 11.25 | 6.25 | 6.50 | 8.00 |
| Gemma-4-E2B-it — Trainable | 30.50 | 24.00 | 14.50 | 9.25 | 12.25 | 5.00 | 10.50 | 24.00 | 13.25 | 14.00 | 11.00 |
Table 20. Recall@50 by instruction type on INSPIRE using only the final audio-encoder hidden states. Frozen and Trainable indicate whether the audio projection is frozen or updated during retrieval training. Instruction labels follow Table 13. Bold marks column maxima within each backbone.
For both models, the trainable audio projection consistently outperforms the frozen projection on VCTK, Expresso, and Synthetic. On Synthetic, these gains cover all eleven instruction types with multiple attributes at Recall@50. The effects on spoken-content retrieval vary between models. At
Recall@50, Qwen2.5-Omni-3B improves on answer retrieval but declines on dialogue continuation, while Gemma-4-E2B-it shows the opposite trend.
With a trainable audio projection in both settings, Weighted Layer consistently outperforms Final Layer on VCTK, Expresso, and Synthetic for both models. Thus, the observed gains over Final Layer persist when both variants train the audio projection.
Layer Weights
We analyze how audio layer weights vary across retrieval requirements and instruction wording using the ten instructions in Table 14. The document prompt serves as a reference without an instruction. All prompts follow the MiniLM templates in Table 11. Figure 8 presents all eleven prompts for both models, while Figure 6 focuses on the five short instructions and the document prompt for Qwen2.5-Omni-3B. For Gemma-4-E2B-it, layer 4 receives the highest adjusted weight under every query instruction except the short answer-retrieval instruction, which places the highest weight on the final layer.
Figure 8. Audio layer weights adjusted for hidden-state scale. S/F: short/full-sentence instructions; Doc.: document prompt; 0: pre-encoder hidden states. Encoder layers are indexed from one. Color scales differ between models.
To account for differences in hidden-state scale, we adjust the layer weights using the same 680 unique INSPIRE spoken queries for every prompt and both models. Let denote the original weight for layer under text prompt . For a recording , let denote the root mean square of the hidden states across hidden dimensions, averaged over valid frames. We compute the adjusted weights as
where and index the pre-encoder hidden states and all audio-encoder layer hidden states. The adjusted weights sum to one for each recording and prompt. We first average them within each INSPIRE subset, then average the four subset means equally. This adjustment is used only for analysis and does not affect model computation. The adjusted weights describe the relative scales of weighted audio layer hidden states, not their causal contributions to retrieval scores.
Layer Representations
We extend the analysis in § 3.2.1 to examine whether audio projection affects the best-performing layer. For each layer, we apply the same frozen audio projection, mean-pool the projected hidden states over time, and rank spoken documents by cosine similarity. Both the audio encoder and projection remain frozen, and the language model is not used. Figure 9 reports Recall@50 for dialogue continuation on DailyTalk, speaker matching on VCTK, speaking-style matching on Expresso, and answer retrieval on Synthetic. The best-performing layer continues to vary across retrieval requirements after projection.
Figure 9. Recall@50 using frozen audio-encoder hidden states after audio projection. 0 denotes the pre-encoder hidden states; encoder layers are indexed from one.
| Layer Aggregation | DailyTalk Cont. | DailyTalk Spk. | VCTK Spk. | Expresso Spk. | Expresso Sty. | Expresso Spk. +Sty. | Expresso Spk. +Sty.* | Synthetic Ans. | Synthetic Spk. | Synthetic Sty. | Synthetic Env. |
| Qwen2.5-Omni-3B | |||||||||||
| Layer 13 | 15.50 | 76.88 | 44.75 | 45.50 | 67.25 | 38.25 | 2.00 | 43.50 | 54.75 | 39.25 | |
| Final Layer | 82.50 | 48.75 | 11.25 | 39.75 | 44.75 | 17.25 | 94.50 | 27.50 | 40.50 | 31.00 | |
| Mean Layer | 70.00 | 71.25 | 37.75 | 42.00 | 57.75 | 35.00 | 87.75 | 50.75 | 45.50 | 40.50 | |
| Finetuned | 32.50 | 91.25 | 51.50 | 42.25 | 65.00 | 30.75 | 68.25 | 52.50 | 54.50 | 43.25 | |
| Static Weight | 66.00 | 78.75 | 33.00 | 41.75 | 61.50 | 36.50 | 87.00 | 48.50 | 52.25 | 45.50 | |
| Inst. Weight (Ours) | 77.50 | 85.00 | 41.25 | 42.00 | 62.00 | 34.75 | 90.50 | 53.50 | 53.75 | 35.50 | |
| Gemma-4-E2B-it | |||||||||||
| Final Layer | 42.50 | 48.13 | 13.50 | 45.50 | 43.25 | 17.75 | 83.25 | 28.75 | 37.50 | 7.75 | |
| Mean Layer | 25.00 | 69.38 | 32.75 | 42.25 | 72.50 | 36.75 | 1.50 | 54.50 | 44.50 | 4.00 | |
| Inst. Weight (Ours) | 44.50 | 74.38 | 33.00 | 48.00 | 63.75 | 30.50 | 80.58 | 51.25 | 47.25 | 8.50 |
Table 18. Recall@50 by instruction type for audio layer aggregation on INSPIRE. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Instruction labels follow Table 13; bold marks column maxima within each backbone.
| Layer Aggregation | Synthetic Spk. +Sty. | Synthetic Spk. +Sty.* | Synthetic Spk. +Env. | Synthetic Spk. +Env.* | Synthetic Sty. +Env. | Synthetic Sty. +Env.* | Synthetic Sty.* +Env. | Synthetic Spk. +Sty. +Env. | Synthetic Spk. +Sty.* +Env. | Synthetic Spk. +Sty. +Env.* | Synthetic Spk. +Sty.* +Env.* |
| Qwen2.5-Omni-3B | |||||||||||
| Layer 13 | 44.50 | 40.50 | 46.00 | 15.00 | 37.75 | 8.50 | 34.50 | 63.00 | 58.50 | 22.50 | 18.75 |
| Final Layer | 30.75 | 27.25 | 35.75 | 19.50 | 28.50 | 14.00 | 29.50 | 45.75 | 38.50 | 32.50 | 25.25 |
| Mean Layer | 49.75 | 43.75 | 55.00 | 20.75 | 35.75 | 16.50 | 38.75 | 65.50 | 56.25 | 38.75 | 28.25 |
| Finetuned | 49.75 | 43.50 | 51.25 | 19.50 | 35.75 | 11.00 | 39.50 | 67.50 | 62.75 | 32.50 | 18.75 |
| Static Weight | 48.25 | 41.25 | 50.25 | 22.00 | 40.50 | 18.25 | 37.25 | 67.00 | 57.25 | 37.25 | 26.75 |
| Inst. Weight (Ours) | 54.25 | 46.00 | 47.50 | 17.25 | 35.75 | 15.00 | 38.00 | 61.75 | 57.00 | 30.00 | 26.75 |
| Gemma-4-E2B-it | |||||||||||
| Final Layer | 30.50 | 24.00 | 14.50 | 9.25 | 12.25 | 5.00 | 10.50 | 24.00 | 13.25 | 14.00 | 11.00 |
| Mean Layer | 55.50 | 50.00 | 13.25 | 6.00 | 4.50 | 3.75 | 3.50 | 29.25 | 25.00 | 14.00 | 15.50 |
| Inst. Weight (Ours) | 47.00 | 45.25 | 15.50 | 7.25 | 13.50 | 1.25 | 8.25 | 31.50 | 18.25 | 15.00 | 13.75 |
Table 18 (PDF p. 31). Recall@50 by instruction type for audio layer aggregation on INSPIRE. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Instruction labels follow Table 13; bold marks column maxima within each backbone.
References
- [1]Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadbabs, and Thierry Dutoit. 2018. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514.arxiv.org/abs/1806.09514
- [2]Akshay Anantapadmanabhan, Ashwin Bellur, and Hema A Murthy. 2013. Modal analysis and transcription of strokes of the mridangam using non-negative matrix factorization. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 181–185. IEEE.DOI
- [3]Chia-Wei Ao and Hung-yi Lee. 2018. Query-by-example spoken term detection using attention-based multi-hop networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6264–6268. IEEE.arxiv.org/abs/1709.00354
- [5]Adnan El Assadi, Isaac Chung, Chenghao Xiao, et al. 2026. Maeb: Massive audio embedding benchmark. arXiv preprint arXiv:2602.16008.
- [6]Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338–15347.arxiv.org/abs/2303.15247
- [7]Pavlo Bazilinskyy, Arne van der Aa, Michael Schoustra, John Spruit, Laurens Staats, Klaas Jan van der Vlist, and Joost de Winter. 2018. An auditory dataset of passing vehicles recorded with a smartphone. In 12th International Symposium on Tools and Methods of Competitive Engineering (TMCE 2018), pages 417–422.DOI
- [8]Felix Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben, and Björn Schuller. 2023. Speech-based age and gender prediction with transformers. In Speech Communication; 15th ITG Conference, pages 46–50. VDE.
- [9]Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean-nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359.DOI
- [10]Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. Msp-improv: An acted corpus of dyadic interactions to study emotion perception. IEEE transactions on affective computing, 8(1):67–80.DOI
- [11]Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390.pubmed.ncbi.nlm.nih.gov/25653738
- [12]Ciprian Chelba, Timothy J Hazen, and Murat Saraclar. 2008. Retrieval and browsing of spoken content. IEEE Signal Processing Magazine, 25(3):39–49.DOI
- [13]Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, and Zhicheng Dou. 2026. e5-omni: Explicit cross-modal alignment for omni-modal embeddings. In Findings of the Association for Computational Linguistics: ACL 2026, pages 19430–19443.
- [14]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518.DOI
- [15]Ondřej Cífka, Hendrik Schreiber, Luke Miner, and Fabian-Robert Stöter. 2024. Lyrics transcription for humans: A readability-aware benchmark. In Proceedings of the 25th International Society for Music Information Retrieval Conference, pages 737–744. ISMIR.
- [16]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.arxiv.org/abs/2507.06261
- [17]Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. 2020. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. In Interspeech 2020, pages 3830–3834.DOI
- [18]Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.arxiv.org/abs/2206.04769
- [19]John S Garofolo, Cedric GP Auzanne, Ellen M Voorhees, et al. 2000. The trec spoken document retrieval track: A success story. NIST SPECIAL PUBLICATION SP, 500(246):107–130.
- [20]Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michal Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulić. 2021. Multilingual and cross-lingual intent detection from spoken data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7468–7475.
- [21]Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. 2026. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Advances in Neural Information Processing Systems, 38:41819–41886.arxiv.org/abs/2507.08128
- [22]Yuan Gong, Jin Yu, and James Glass. 2022. Vocal sound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE.
- [23]Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 976–980. IEEE.
- [24]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
- [25]Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, et al. 2026. Qwen3-tts technical report. arXiv preprint arXiv:2601.15621.arxiv.org/abs/2601.15621
- [26]Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580.arxiv.org/abs/2407.12580
- [27]Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. 2025. VLM2Vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.05160
- [28]Holger Klinck, Juan Sebastián Cañas, Maggie Demkin, Sohier Dane, Stefan Kahl, and Tom Denton. 2025. Birdclef+ 2025. https://kaggle.com/competitions/birdclef-2025. Kaggle.
- [29]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.DOI
- [30]Martha Larson and Gareth JF Jones. 2012. Spoken content retrieval: A survey of techniques and technologies. Foundations and Trends® in Information Retrieval, 5(4–5):235–422.
- [33]Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
- [34]Lin-shan Lee, James Glass, Hung-yi Lee, and Chunan Chan. 2015. Spoken content retrieval—beyond cascading speech recognition with text retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(9):1389–1420.DOI
- [35]Chen-An Li and Hung-yi Lee. 2026. Inspire: A benchmark for instruction-aware speech retrieval. arXiv preprint arXiv:2608.16203.arxiv.org/abs/2608.16203
- [36]Mingxin Li, Yanzhao Zhang, Dingkun Long, et al. 2026a. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720.
- [37]Xiquan Li, Xuenan Xu, Ziyang Ma, Wenxi Chen, Haolin He, Qiuqiang Kong, and Xie Chen. 2026b. Finelap: Taming heterogeneous supervision for fine-grained language-audio pretraining. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10393–10408.
- [38]Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
- [39]Chyi-Jiunn Lin, Guan-Ting Lin, Yung-Sung Chuang, Wei-Lun Wu, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, and Lin-Shan Lee. 2024a. Speechdpr: End-to-end spoken passage retrieval for open-domain spoken question answering. In 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12476–12480. IEEE.
- [40]Guan-Ting Lin, Yung-Sung Chuang, Ho-Lam Chung, Shu-wen Yang, Hsuan-Jui Chen, Shyuan Annie Dong, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, and Lin Shan Lee. 2022. DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering. In Interspeech 2022, pages 5165–5169.
- [41]Guan-Ting Lin, Wei Ping Huang, and Hung-yi Lee. 2024b. Continual test-time adaptation for end-to-end speech recognition on noisy speech. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20003–20015.arxiv.org/abs/2406.11064
- [42]Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. 2021. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2125–2134.
- [43]Fengjie Lu, Chenang Jiang, Jiarui Hai, Helin Wang, and Aaron Yee. 2026a. Alm2vec: Learning audio embeddings for universal audio retrieval with large audio-language models. arXiv preprint arXiv:2606.30682.
- [44]Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Yang, et al. 2026b. Desta2. 5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment. IEEE Transactions on Audio, Speech and Language Processing.
- [45]Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15747–15760.
- [46]Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3339–3354.
- [47]Rui Meng, Ziyan Jiang, Ye Liu, et al. 2025. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents. arXiv preprint arXiv:2507.04590.
- [48]Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi, Srikanth Ronanki, Kyu Han, et al. 2025. Speech retrieval-augmented generation without automatic speech recognition. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.arxiv.org/abs/2412.16500
- [49]Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Interspeech 2017, pages 2616–2620.DOI
- [50]Tri Nguyen, Mir Rosenberg, Xia Song, ianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268.
- [51]Tu Anh Nguyen, Wei-Ning Hsu, Antony D’Aviro, Bowen Shi, Itai Gat, Maryam Fazel-Zarandi, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emmanuel Dupoux. 2023. Expresso: A benchmark and analysis of discrete expressive resynthesis. In Interspeech 2023, pages 4823–4827.
- [52]Kari Ali Nory, Xiaosong Yang, and Jian Jun Zhang. 2023. Emms/imz/corpus: An emotive single-speaker dataset for narrative storytelling in games, television and graphic novels. arXiv preprint arXiv:2305.13137.
- [53]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.arxiv.org/abs/1807.03748
- [54]OpenAI. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.
- [55]Se Park, Chae Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Ro. 2024. Let’s go real talk: Spoken dialogue model for face-to-face conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16334–16348.
- [56]Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018.
- [57]Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.arxiv.org/abs/1810.02508
- [58]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.arxiv.org/abs/2212.04356
- [59]Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. 2018. Cnn based query by example spoken term detection. In Interspeech, pages 92–96.DOI
- [60]Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. 2020. Neural network based end-to-end query by example spoken term detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1416–1427.arxiv.org/abs/1911.08332
- [61]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1908.10084
- [62]Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 410–420.
- [63]Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. 2023. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305–19314.arxiv.org/abs/2302.03084
- [64]Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1041–1044.DOI
- [65]Fabian David Schmidt, Ivan Vulić, Goran Glavaš, and David Ifeoluwa Adelani. 2025. Fleurs-SLU: A massively multilingual benchmark for spoken language understanding. In Second Conference on Language Modeling.arxiv.org/abs/2501.06117
- [66]Weiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo, Chen Xu, Xiaofeng Zhao, Long Meng, Yunfei Liu, Min Zhang, Hao Yang, et al. 2025. Enhancing speech large language models with prompt-aware mixture of audio encoders. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19316–19331.
- [67]Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. 2023. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. Advances in Neural Information Processing Systems, 36:39088–39118.arxiv.org/abs/2305.13040
- [68]Ganesh Sinisetty, Pavlo Ruban, Oleksandr Dymov, and Mirco Ravanelli. 2021. Commonlanguage.DOI
- [70]Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, and Chao Zhang. 2026. Wave: learning unified & versatile audio-visual embeddings with multimodal llm. In International Conference on Learning Representations, volume 2026, pages 61596–61612.arxiv.org/abs/2509.21990
- [71]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
- [72]Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. 2014. A study of instrument-wise onset detection in beijing opera percussion ensembles. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 2159–2163. IEEE.








