betaFeedback

WorksComputer Science

Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval

Abstract

Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.

Introduction

Speech retrieval searches a collection of spoken documents using a spoken query [12, 30, 34]. For example, “Where should we go this weekend?” spoken excitedly could be used to find activity suggestions, another recording by the same speaker, or speech with similar excitement. The spoken query alone does not distinguish these goals.

Instruction-aware speech retrieval uses natural-language instructions to express the user’s retrieval goal [35]. Each instruction expresses requirements for speech attributes such as spoken content, speaker identity, and speaking style. The required values may come from the spoken query or be specified directly. For example, “Find speech by the same speaker, spoken happily” asks for a document that matches the query’s speaker identity and has a happy speaking style. A relevant spoken document must satisfy every requirement.

Large audio-language models (LALMs) connect an audio encoder to a language model and have shown strong performance on tasks such as speech recognition, audio captioning, and audio question answering [71, 21, 44] (Xu et al. 2025a). Their ability to process speech and text within a model motivates their adaptation for instruction-aware speech retrieval. Our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We ask: How can we adapt LALMs for instruction-aware speech retrieval?

We introduce IRIS-Embed (Instruction-aware Retrieval for Speech), which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. Figure 1 illustrates the overall framework. IRIS-Embed separately encodes a spoken query with its instruction and each spoken document, then ranks documents by cosine similarity. Audio-layer weighting predicts weights based on the instruction to combine hidden states from different layers of the frozen audio encoder before language-model encoding.

Figure 1. IRIS-Embed overview. A: Retrieval and training with a shared model. B: Audio layer weighting and embedding computation. Inst.: instruction.

Contrastive training aligns query embeddings with relevant document embeddings. We construct about 900K samples from real data and synthetic data, each pairing a spoken query and instruction with positive and negative spoken documents. Each instruction expresses requirements for one or more speech attributes. The positive spoken document must satisfy every requirement in the instruction.

We evaluate IRIS-Embed on INSPIRE [35], an instruction-aware speech retrieval benchmark. Across two model families, IRIS-Embed consistently outperforms the corresponding LALM baselines, achieving macro-average Recall@50 scores of 62.92% and 47.41%, both surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness for single- and multi-attribute speech requirements. We also show that audio-layer weighting improves speaker and speaking-style matching compared with using only the final audio-encoder hidden states. On MAEB [5], we evaluate IRIS-Embed on 15 diverse tasks, including gender classification, accent-pair classification, and artist retrieval. IRIS-Embed achieves macro-average gains of 9.4–19.8 points over the corresponding LALM baselines, demonstrating generalization beyond speech retrieval.

Related Work

Spoken Content Retrieval

Spoken content retrieval finds spoken documents with content relevant to a spoken query [19, 12, 30, 34]. Prior work explores query-by-example spoken term detection [3, 59, 60] and spoken passage retrieval for question answering [39, 48]. We focus on instruction-aware speech retrieval, where instructions express requirements based on spoken content, speaker identity, speaking style, or other speech attributes.

Instruction-Aware Retrieval

Instructions guide text retrieval by expressing retrieval goals and relevance requirements (Asai et al., 2023; Su et al., 2023; Weller et al., 2025b,a; Wang et al., 2024; Lee et al., 2025a; Zhang et al., 2025b; Lee et al., 2025b). Image and multimodal retrieval methods also use text to express retrieval goals [42, 6, 63, 26, 27, 47, 36] (Wu et al. 2021; Vaze et al. 2023; Zhang et al. 2024; Zhang et al. 2025a; Zhou et al. 2025). INSPIRE [35] benchmarks instruction-aware speech retrieval with requirements for one or more speech attributes. We adapt LALMs into embedding models for instruction-aware speech retrieval.

Audio-Language Embeddings

Audio-language embeddings support retrieval between audio and text, as well as audio classification [23, 18, 46, 37] (Wu et al. 2023; Xu et al. 2023; Zhu et al. 2024). LALM embeddings also support retrieval between audio and text [13, 43] (Xiao et al. 2025; Yoo et al. 2026). We focus on retrieving spoken documents given a spoken query and a natural-language text instruction. Furthermore, IRIS-Embed uses instruction-aware audio-layer weighting to combine hidden states from different layers of the frozen audio encoder before language-model encoding.

Audio Layer Aggregation

SUPERB aggregates frozen self-supervised speech-encoder hidden states for downstream tasks (Wen Yang et al., 2021). Prompt-aware Mixture uses prompt-guided fusion for recognition and captioning [66]. WAVE and OmniUE aggregate language-model states for multimodal embeddings [70] (Wang et al. 2026). Instead, IRIS-Embed learns instruction-aware weights to combine hidden states from different layers of an LALM’s frozen audio encoder through contrastive speech retrieval training.

Method

Problem Formulation

Instruction-aware speech retrieval ranks spoken documents in a collection DD given a spoken query qq and a natural-language text instruction zz. Speech attributes describe properties of a recording, such as spoken content, speaker identity, and speaking style. The instruction zz expresses the requirements on these attributes by referring directly to the spoken query or specifying the values. For example, “Find speech by the same speaker, spoken happily” requires a document to match qq’s speaker identity and have a happy speaking style. A relevant document must satisfy every requirement; unconstrained attributes do not affect relevance.

For each query–instruction pair (q,z)(q,z), let D+⊆DD^{+} \subseteq D denote the positive documents relevant to (q,z)(q,z) and D−=D∖D+D^{-}=D\setminus D^{+} denote the negative documents. A scoring function f(d∣q,z)f(d\mid q,z) ranks documents d∈Dd\in D in decreasing order of score. An ideal ranking satisfies f(d+∣q,z)>f(d−∣q,z)f(d^{+}\mid q,z)>f(d^{-}\mid q,z), for every d+∈D+d^{+}\in D^{+} and d−∈D−d^{-}\in D^{-}.

Architecture

As shown in Figure 1, IRIS-Embed uses a shared model to separately encode the spoken query qq with instruction zz and each spoken document dd. For each recording, we extract hidden states from the pre-encoder and all layers of the frozen audio encoder. The audio-layer weighting module encodes a text-only prompt to predict weights for combining these hidden states. The weighted audio hidden states are then pooled and projected before being passed to the language model, which processes the text and weighted audio hidden states using its own prompt template. Query prompts include instruction zz for both audio layer weighting and language-model encoding, whereas document prompts contain no instruction zz.

We use the language model’s final-layer hidden state of the last non-padding token as the embedding, obtaining eqe_q for the query–instruction pair (q,z)(q,z) and ede_d for document dd. Documents are ranked by decreasing cosine similarity, f(d∣q,z)=cos⁡(eq,ed)f(d\mid q,z)=\cos(e_q,e_d). We next explain the motivation and design of audio layer weighting.

Audio Layer Weighting

To investigate how different audio-encoder layers perform across retrieval tasks, we evaluate their frozen hidden states on INSPIRE ([35]). For each layer, we mean-pool the hidden states over time and rank spoken documents by cosine similarity to the spoken query, without using instructions, the audio projection, or the language model. Figure 2 reports Recall@50 for dialogue continuation, speaker matching, speaking-

Figure 2. Recall@50 across frozen audio-encoder layers. Layer 0 represents pre-encoder hidden states; layers 1 and above represent encoder hidden states.

style matching, and answer retrieval. The best-performing layer varies across tasks, and the final layer does not consistently achieve the highest recall. These findings motivate combining hidden states from multiple audio-encoder layers before passing them to the language model.

For each recording, let H0\mathbf{H}_0 denote the pre-encoder hidden states and H1,…,HL\mathbf{H}_1,\ldots,\mathbf{H}_L the hidden states produced by the LL audio-encoder layers. We combine hidden states at the same time step to obtain weighted audio hidden states:

H~=∑ℓ=0LwℓHℓ,\widetilde{\mathbf{H}}=\sum_{\ell=0}^{L}w_{\ell}\mathbf{H}_{\ell},

where wℓw_{\ell} is the scalar weight assigned to Hℓ\mathbf{H}_{\ell}.

A shared audio layer weighting module computes weights from a text prompt pp:

wℓ=[softmax⁡(g(p))]ℓ,w_{\ell}=[\operatorname{softmax}(g(p))]_{\ell},

where gg uses a pretrained sentence encoder followed by a linear layer to produce L+1L+1 layer scores. The notation [⋅]ℓ[\cdot]_{\ell} selects the weight for Hℓ\mathbf{H}_{\ell}. The resulting weights are nonnegative and sum to one. For queries, the prompt pp includes the instruction zz, allowing the layer weights to vary across retrieval tasks. For documents, we use a fixed prompt, so all documents share the same layer weights.

Training Objective

Each batch consists of BB samples (qi,zi,di+,di−)(q_i,z_i,d_i^{+},d_i^{-}), where ii indexes samples and di+d_i^{+} and di−d_i^{-} are the paired positive and negative documents. The batchlevel document pool is C=⋃i=1B{di+,di−}C = \bigcup_{i=1}^{B}\{d_i^+, d_i^-\}. Since documents from other samples may also be relevant to (qi,zi)(q_i, z_i), treating them as negatives could introduce false negatives. We use training labels to mask these additional relevant documents. For each sample ii, we construct a document pool CiC_i by excluding such documents from CC while preserving the paired positive document di+d_i^+.

We optimize the model using the InfoNCE loss [53]:

L=−1B∑i=1Blog⁡exp⁡(f(di+∣qi,zi)/τ)∑d∈Ciexp⁡(f(d∣qi,zi)/τ)\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(f(d_i^+ \mid q_i,z_i)/\tau)}{\sum_{d\in C_i}\exp(f(d \mid q_i,z_i)/\tau)}

where τ>0\tau> 0 is a temperature parameter controlling the sharpness of the score distribution. This objective encourages the model to assign higher relevance scores to the paired positive document than to the remaining negatives in CiC_i.

During training, we freeze the audio encoder and the sentence encoder in gg. We adapt the language model using low-rank adaptation (LoRA) [24]. The audio projection, which maps audio hidden states to the language model’s embedding dimension, is trained jointly with the added linear layer in gg and the LoRA parameters.

Training Data

Notation

We represent each recording xx by a set A(x)A(x) of attribute-value pairs. Each pair identifies a speech attribute and its value, such as (style, happy) for a happy speaking style.

The requirements rr form a set of attribute-value pairs that a relevant spoken document must satisfy. We write r=rq∪rsr = r_q \cup r_s. The requirements in rqr_q match the spoken query’s attribute values, with ∅≠rq⊆A(q)\varnothing\ne r_q \subseteq A(q). The set rsr_s contains attribute-value pairs with specified values that need not match the spoken query. We allow rs=∅r_s = \varnothing. The text instruction zz expresses rr in natural language, referring to the spoken query for rqr_q and stating the values in rsr_s. The two sets use disjoint attribute keys to avoid conflicting requirements, so rq∩rs=∅r_q \cap r_s = \varnothing.

For example, consider a spoken query qq with happy speech and a dog barking in the background. The requirements r={(environment,dog bark),(style,sad)}r = \{(\mathrm{environment},\mathrm{dog\ bark}),(\mathrm{style},\mathrm{sad})\} ask for sad speech with a barking dog. Here, rq={(environment,dog bark)}r_q = \{(\mathrm{environment},\mathrm{dog\ bark})\} matches the query’s environmental sound, while rs={(style,sad)}r_s = \{(\mathrm{style},\mathrm{sad})\} specifies a sad speaking style. A corresponding

instruction zz is “Find a recording with the same background sound as the query, spoken sadly.”

Data Format

Each training sample (q,z,d+,d−)(q, z, d^+, d^-) contains a spoken query, an instruction, and a positive and a negative spoken document. A document is relevant if it satisfies all requirements in rr: D+={d∈D:r⊆A(d)}D^+ = \{d \in D : r \subseteq A(d)\}. The paired positive belongs to D+D^+, while the paired negative belongs to D−D^- and violates at least one requirement. Attributes outside rr do not affect relevance.

Construction

We construct IRIS-Embed training samples from real data and synthetic data, each paired with an instruction expressing its retrieval requirements. Figure 3 shows the data construction pipeline.

Figure 3. Overview of the data construction pipeline, combining real and synthetic data to create spoken query–document pairs and retrieval instructions.

Real Data. We use public speech and audio corpora. The source annotations cover speaking style [11], speaker identity [49], gender (Zhou et al., 2021), accent (Zhao et al., 2018), sentiment [57], dialogue [67, 55], and environmental sound [64]. We remove uninformative labels such as unknown and other, recordings outside 1–120 s, and environmental sounds containing speech. We also map equivalent labels across corpora, such as happy and joyful, to a common value. Corpus metadata provide attribute values for each recording, which we use to construct retrieval requirements and select query–document pairs. Synthetic Data. We sample English text triplets from the MS MARCO Nguyen et al., [50] hard-triplet data, each containing a query, a relevant passage, and a hard negative passage. We synthesize speech from these texts using multiple text-to-speech models Comanici et al., [16]; Hu et al. [25]. Voice selection controls speaker identity, synthesis prompts control speaking style, and sound mixing adds environmental sounds.

Pair Construction. For each spoken query qq, we sample a nonempty rqr_q from A(q)A(q) and optionally form rsr_s by sampling values for other attribute keys annotated for qq. Speaker identity, dialogue, and answer relevance are restricted to rqr_q. When speaker identity is included, we exclude gender and accent, since matching the speaker also guarantees matching these attributes in our data. For each rr, we sample a positive and a negative document, prioritizing negatives that violate fewer requirements to obtain harder negatives.

Instruction Generation. We use commercial large language models to express each set of requirements rr as a natural-language instruction zz. To cover diverse instruction styles, we use one prompt to generate short instructions and another to generate full-sentence instructions. We remove near-duplicates using ROUGE-L Lin, [38]. We also verify that each instruction covers all requirements in rr, refers to the spoken query for pairs in rqr_q, and specifies the values in rsr_s.

Statistics. The resulting dataset contains approximately 900K samples: 800K from real data and 100K from synthetic data. We use about 890K samples for training and 10K for validation. Appendix § 8 lists the sources, counts, and data construction settings.

Experiments

Experimental Setup

Evaluation. We evaluate IRIS-Embed on four INSPIRE Li and Lee, [35] subsets. DailyTalk Lee et al., [33] evaluates dialogue continuation, VCTK (Yamagishi et al., 2019) evaluates speaker matching, and Expresso Nguyen et al., [51] evaluates speaker and speaking-style matching. Speaking style includes emotion and vocal states. The Synthetic subset evaluates answer retrieval and matching of speaker identity, speaking style, and environmental sounds, with instructions involving one or more attributes. Performance is measured by Recall@KK for K∈{10,50,100}K \in\{10, 50, 100\}, defined as the fraction of relevant documents retrieved in the top KK results, averaged across query–instruction pairs and reported as percentages.

Training. We train IRIS-Embed with Qwen2.5-Omni-3B (Xu et al., 2025a) and Gemma-4-E2B-it Team et al., [71] on the data in § 4. The audio layer weighting module contains a sentence encoder and a learned linear layer. We use the frozen all-MiniLM-L6-v2 model Wang et al., (2020); Reimers and Gurevych [61] as the sentence encoder. Training uses a global batch size of 512. Appendix § 9 provides further training details.

Comparison Methods

Random. Uniform random document rankings serve as a chance-level baseline Li and Lee, [35].

General Speech Encoders. We evaluate general speech representations using the self-supervised WavLM-large Chen et al., [14] without instructions. Query and document embeddings are obtained by mean-pooling its final-layer hidden states over time.

Specialized Speech Encoders. We evaluate specialized speech representations without instructions, using ECAPA-TDNN Desplanques et al., [17] for speaker embeddings and emotion2vec-base Ma et al., [45] for emotion embeddings.

Cascaded Retrieval. We use a cascaded pipeline to retrieve spoken documents through text representations. Whisper-large-v3 Radford et al., [58] generates transcripts, while Qwen3-Omni-30B-A3B-Captioner (Xu et al., 2025b) generates captions. Qwen3-Embedding-8B (Zhang et al., 2025b) encodes the transcripts and captions of each spoken query and spoken document, with the instruction included in the query input.

LALM Embeddings. We evaluate Qwen2.5-Omni-3B (Xu et al., 2025a), Gemma-4-E2B-it Team et al., [71], and Qwen3-Omni-30B (Xu et al., 2025b) without additional embedding training. Embeddings are extracted from the final-layer hidden state of the last non-padding input token without generating a response. We also include LCO-Embedding-Omni-3B (Xiao et al., 2025), a Qwen2.5-Omni-based model trained for multimodal embeddings.

Main Results

Table 1 shows that IRIS-Embed consistently outperforms the corresponding LALM baselines on all four INSPIRE subsets. Macro-average Recall@50 reaches 62.92% for Qwen2.5-Omni-3B and 47.41% for Gemma-4-E2B-it, compared with 16.27% and 8.91% for their base models. Both exceed ECAPA-TDNN’s 45.15%, the highest macro-average Recall@50 among comparison baselines.

MethodModelDailyTalk R@10DailyTalk R@50DailyTalk R@100VCTK R@10VCTK R@50VCTK R@100Expresso R@10Expresso R@50Expresso R@100Synthetic R@10Synthetic R@50Synthetic R@100Avg. R@10Avg. R@50Avg. R@100
RANDRandom0.241.062.100.351.843.630.301.502.960.201.001.980.271.352.67
ENCWavLM-large16.5024.5035.5017.5035.0043.759.5017.6924.001.704.487.2211.3020.4227.62
SENCECAPA-TDNN15.5024.0029.50100.00100.00100.0034.8841.7545.4410.0214.8318.82<u>40.10</u>45.1548.44
emotion2vec-base11.5023.0030.503.136.2513.750.884.388.001.614.536.824.289.5414.77
CASQwen3-Embedding-8B62.0081.5089.501.886.2510.630.755.569.697.028.9511.0717.9125.5730.22
LALMQwen2.5-Omni-3B26.5053.0064.001.883.133.130.753.756.693.145.206.888.0716.2720.18
Gemma-4-E2B-it9.0025.5033.501.254.385.630.131.192.252.524.566.033.238.9111.85
Qwen3-Omni-30B51.5074.5083.001.253.136.250.943.566.194.886.387.8814.6421.8925.83
LCO-Embedding-Omni-3B54.0075.5083.501.252.505.631.948.4413.816.829.3711.6516.0023.9528.65
IRISQwen2.5-Omni-3B60.00<u>77.50</u><u>87.50</u><u>67.50</u>85.0089.38<u>29.63</u>45.0053.8826.0644.1754.7845.8062.9271.39
Gemma-4-E2B-it25.5044.5054.5058.7574.3879.3828.63<u>43.81</u><u>52.31</u><u>15.07</u><u>26.94</u><u>36.08</u>31.99<u>47.41</u><u>55.57</u>

Table 1. INSPIRE Recall@KK. Avg.: macro-average across the four subsets. Bold / underline: best / second best. RAND: random; ENC / SENC: general / specialized encoders; CAS: cascaded retrieval; LALM: large audio–language models; IRIS: IRIS-Embed.

Both IRIS-Embed models consistently outperform all LALM embedding baselines on VCTK and Expresso. IRIS-Embed with Qwen2.5-Omni-3B achieves 85.00% and 45.00% Recall@50 on these subsets, respectively, compared with 2.50% and 8.44% for LCO-Embedding-Omni-3B.

Both models also consistently outperform all comparison baselines on Synthetic. This subset includes instructions with multiple attributes, for which a relevant document must satisfy every requirement. Recall@50 reaches 44.17% for Qwen2.5-Omni-3B and 26.94% for Gemma-4-E2B-it, compared with 14.83% for ECAPA-TDNN.

On DailyTalk, IRIS-Embed with Qwen2.5-Omni-3B achieves 77.50% Recall@50, close to 81.50% for the cascaded pipeline using larger captioning and text-embedding models. While the specialized ECAPA-TDNN remains strongest for speaker matching on VCTK, IRIS-Embed supports retrieval across speech attributes within a single embedding model.

Analysis

Instruction Types

We breakdown IRIS-Embed’s performance on instructions involving single or multiple attributes. Figure 4 compares IRIS-Embed with the corresponding LALM baselines across instruction types. Both IRIS-Embed models achieve higher Recall@50 on most instruction types.

Figure 4. Recall@50 by instruction type on INSPIRE. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound. +: multiple attributes; ∗: specified values.

On Synthetic, IRIS-Embed also performs well on instructions involving multiple attributes, including specified attribute values. For joint matching of speaker identity, speaking style, and environmental sound, Recall@50 reaches 61.8% for Qwen2.5-Omni-3B and 31.5% for Gemma-4-E2B-it. For the speaker matching with a specified speaking style, the scores reach 46.0% and 45.2%, respectively. Both models outperform all baselines on most multi-attribute instruction types. The complete results are provided in the Appendix § 11.

Instruction Awareness

We examine whether IRIS-Embed distinguishes different retrieval goals for the same spoken query. Figure 5 reports Recall@50 for Qwen2.5-Omni-3B under different short instructions, such as “Same speaker” and “Find the answer”. Among five instructions with a single attribute, the instruction matching the evaluation requirements achieves the highest Recall@50 in seven of eight evaluation settings. Switching from a speaker-matching instruction to an answer-retrieval instruction increases answer retrieval Recall@50 from 1.5% to 90.3%.

Figure 5. Recall@50 for IRIS-Embed with Qwen2.5-Omni-3B. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound; +: multiple attributes.

On Synthetic, we compare four short instructions combining speaker, speaking style, and environmental sound, such as “Same speaker and same speaking style”. The matching instruction achieves the highest Recall@50 in all four evaluation settings. Gemma-4-E2B-it shows a similar pattern, and both models also follow full-sentence instructions, as reported in Appendix § 11.§ 2. These results show that IRIS-Embed follows the speech-attribute requirements expressed by the instruction.

Layer Aggregation

Table 2 compares Qwen2.5-Omni-3B variants trained and evaluated with different audio-layer aggregation methods. Layer 13 and Final Layer use the 13th and final audio-encoder hidden states, respectively. Mean Layer uses the average weighting across layers. Finetuned updates the audio encoder, while the other variants keep it frozen. Static Weight learns a single set of audio layer weights shared across all queries and documents, regardless of instructions or prompts. Inst. Weight is our instruction-aware audio layer weighting method, with weights varying according to the instruction.

MethodDailyTalkVCTKExpressoSyntheticAvg.
Layer 1315.5076.8848.9435.2744.15
Final Layer82.5048.7528.2534.7248.56
Mean Layer70.0071.2543.1344.9057.32
Finetuned32.5091.2547.3843.3553.62
Static Weight66.0078.7543.1945.2858.31
Inst. Weight (Ours)77.5085.0045.0044.1762.92

Table 2. Recall@50 for Qwen2.5-Omni-3B audio layer aggregation on INSPIRE. Avg.: macro-average across the four subsets.

Our method achieves the highest macro-average Recall@50, at 62.92%. Compared with Final Layer, our method improves recall on VCTK, Expresso, and Synthetic, increasing the macro-average by 14.36 points. Layer 13 and Finetuned improve speaker and speaking-style matching over Final Layer, but have much lower dialogue-continuation recall. These results suggest that retrieval tasks benefit from hidden states of different audio-encoder layers, highlighting the importance of instruction-aware audio layer weighting.

Although Mean Layer and Static Weight perform better on Synthetic, our method outperforms both on the other three subsets, achieving macro-average Recall@50 improvements of 5.60 and 4.61 points, respectively. These results demonstrate the effectiveness of instruction-aware weighting over instruction-independent layer aggregation.

With Gemma-4-E2B-it, our method also outperforms Final Layer on VCTK, Expresso, and Synthetic, showing improvements across multiple retrieval subsets. Appendix § 11.§ 3 provides the full results for both models. Together, these results demonstrate the effectiveness of instruction-aware audio layer weighting in improving retrieval performance while keeping the audio encoder frozen.

Layer Weights

We analyze how audio layer weights vary across retrieval requirements. Figure 6 compares Qwen2.5-Omni-3B layer weights under five short instructions, such as “Same speaker” and “Find the answer”, and the document prompt. To account for scale differences, we adjust the displayed weights using the root mean square of each layer’s hidden states. This adjustment is used only for analysis. Layer 13 receives the highest adjusted weight across all five query instructions. Compared with speaker matching, answer retrieval places more weight on the final layer and less on layer 13. Gemma-4-E2B-it shows a similar pattern, with answer retrieval shifting weight from layer 4 toward its final layer. These weights show that instructions change the balance between intermediate and final audio-encoder hidden states. Appendix § 11.§ 5 provides the full results for both models.

Figure 6. Scale-adjusted audio layer weights for Qwen2.5-Omni-3B. Doc.: document prompt; layer 0: pre-encoder hidden states; encoder layers start at 1.

Task Generalization

We evaluate IRIS-Embed beyond retrieval tasks in INSPIRE on 15 MAEB ([5]) audio tasks covering classification, clustering, pair classification, reranking, and retrieval. These include gender and accent tasks not covered by INSPIRE. We encode each recording with a task instruction using the query prompt, except for retrieval documents, which use the document prompt without an instruction. The models remain frozen during evaluation, and tasks derived from our retrieval training corpora are excluded. The corresponding LALM baselines receive the same task instructions. Appendix § 10.§ 4 provides evaluation details.

Table 3 reports macro-average scores of 53.5 for Qwen2.5-Omni-3B IRIS-Embed and 52.9 for Gemma-4-E2B-it IRIS-Embed. These exceed the corresponding LALM baselines scores by 9.4 and 19.8 points, respectively, and approach LCO-Embedding-Omni-3B’s 54.2. Both IRIS-Embed models outperform all comparison baselines on IEMOCAP gender classification, VoxPopuli gender clustering, and VoxPopuli accent pair classification. These results show that IRIS-Embed generalizes beyond instruction-aware speech retrieval.

MethodModelBeijing OperaBirdCLEFCommonLangAgeGTZANGenreIEMOCAPGenderMInDS14MridinghamTonicSIBFLEURSVoxPopuli LanguageIDVehicle SoundVoxPopuli GenderAvg.
ENCWavLM-large57.719.816.967.763.113.531.412.589.00.20.149.052.268.878.541.4
CLAPclap-htsat-fused92.818.015.367.993.69.247.09.132.04.752.747.953.261.387.746.2
CLAPlarger_clap_general93.617.020.584.589.38.049.49.484.63.43.347.453.566.895.648.4
LALMQwen2-Audio-7B97.437.117.693.193.024.461.215.499.05.50.148.952.980.837.650.9
LALMLCO-Embedding-Omni-3B93.231.616.981.059.488.939.545.163.42.21.096.950.775.467.154.2
LALMQwen2.5-Omni-3B†80.110.616.775.971.950.426.643.394.62.31.269.350.156.611.444.1
LALMGemma-4-E2B-it†63.64.015.528.250.522.027.021.999.41.20.157.250.238.217.833.1
IRISQwen2.5-Omni-3B91.913.314.380.198.724.433.520.971.01.155.292.961.566.177.653.5
IRISGemma-4-E2B-it88.515.116.460.699.014.232.326.679.28.268.594.363.260.667.052.9

Table 3. Scores on 15 MAEB audio tasks. Avg.: macro-average across the 15 tasks. Higher is better; bold marks column maxima. Clf: classification; Clust: clustering; PC: pair classification; Rrnk: reranking; Rtrvl: retrieval. CLAP: contrastive language–audio pretraining. †: baselines using the same task instructions as IRIS-Embed.

Conclusion

IRIS-Embed adapts LALMs into embedding models for instruction-aware speech retrieval through contrastive training and instruction-aware audio layer weighting. Across two model families, macro-average Recall@50 on INSPIRE reaches 62.92% and 47.41%, compared with 45.15% for the strongest baseline. For Qwen2.5-Omni-3B, layer weighting improves VCTK Recall@50 from 48.75% to 85.00% over using only the final audio-encoder output. Controlled comparisons show that the models follow instructions with a single attribute or multiple attributes. On 15 MAEB tasks, macro-average scores improve by 9.4–19.8 points over the corresponding LALM baselines, showing generalization beyond speech retrieval. These results show that IRIS-Embed enables effective retrieval across speech attributes.

Limitations

Despite the advances achieved by our method, several limitations remain.

We prioritize smaller embedding models for computational efficiency and train IRIS-Embed with only two LALM backbones. Thus, its scalability to larger models remains unexplored.

IRIS-Embed combines hidden states from frozen pretrained audio encoders, making it dependent on the information they preserve. We do not explore alternative or complementary encoders that may capture broader speech attributes. Moreover, layer-weight visualizations do not establish individual layers’ contributions to retrieval performance.

Our retrieval requirements rely on annotations covering limited speech attributes and values. Annotation and classifier errors may affect document-pair construction and in-batch masking, respectively. Our training also covers limited languages, voices, speaking styles, and environmental sounds, restricting its diversity relative to natural speech.

Finally, our cascaded baseline uses transcripts and audio captions. We leave comparisons with finer-grained cascaded approaches combining specialized speech embeddings or attribute-specific audio descriptions to future work.

While these limitations remain, IRIS-Embed demonstrates the effectiveness of adapting LALMs for instruction-aware speech retrieval. We believe our work will inspire further research on embeddings that follow diverse user intents and generalize across audio tasks.

Ethics Statement

We use public audio corpora and synthetic data, with corpus licenses listed in Table 4. Reuse should respect source licenses and speech-synthesis service terms, and shared synthetic recordings should be clearly labeled.

SourceAnswer relevanceSpeaking styleGenderSpeakerAccentSentimentDialogueEnvironmental soundLicense
Real data
VoxCeleb1 [49]◦••CC BY-SA 4.0
ESD (Zhou et al., 2021)•••Custom
MELD [57]•◦••GPL-3.0
CREMA-D [11]••ODbL 1.0
L2-ARCTIC (Zhao et al., 2018)◦◦••CC BY-NC 4.0
LibriSpeech-C [41]◦•CC BY 4.0
EmoV-DB [1]•••Custom
MSP-IMPROV [10]••Custom
VocalSound [22]•CC BY-SA 4.0
SpokenWOZ [67]•CC BY-NC 4.0
MultiDialog [55]•CC
UrbanSound8K [64]•CC BY-NC 3.0
EMNS [52]•◦Apache 2.0
Synthetic data
MS MARCO [50]••••CC BY 4.0

Table 4. Data sources and available annotations. • indicates corpus metadata or synthesis annotations used to construct retrieval requirements and select positive and negative spoken documents. ◦ indicates classifier predictions used only to mask additional relevant in-batch documents during training.

Speech-attribute matching can enable unauthorized speaker tracking or profiling. Applications should obtain appropriate consent, protect recordings and embeddings, and avoid discriminatory uses. Dataset biases may cause unequal retrieval quality, so deployment should include evaluation across speaker groups and recording conditions.

Acknowledgments

We thank Yu-Xiang Lin from National Taiwan University for their helpful discussions and feedback.

This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan. We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources. This work was supported by the Ministry of Education (MOE) of Taiwan under the project Taiwan Centers of Excellence in Artificial Intelligence, through the NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE).

Data Construction

Data Sources

Table 4 summarizes the data sources, available annotations, and licenses for real and synthetic data. Speaking-style annotations include emotion and vocal states.

We use corpus metadata to construct retrieval requirements and select positive and negative spoken documents. Classifier predictions supplement these annotations only to identify and mask additional relevant in-batch documents during training, as described in § 3.3.

Real Data

Preprocessing. We remove uninformative labels such as unknown and other, along with recordings outside 1–120 s. We exclude environmental-sound classes containing speech and map equivalent labels across corpora, such as happy and joyful, to a common value.

Classifiers. To supplement incomplete corpus annotations, we use audEERING’s Wav2Vec2 age and gender model (Burkhardt et al., [8])1 to predict gender and emotion2vec+ large (Ma et al., [45])2 to predict emotion. For speakers with multiple recordings, we assign a shared predicted gender label through majority voting. Predicted labels are used only for relevance checks during training.

Synthetic Data

Text Selection. We sample 10,000 triplets from the MS MARCO (Nguyen et al., [50]) hard-triplet dataset processed by Sentence-Transformers.3 Each triplet contains a query, a relevant passage, and a hard negative passage. To reduce false negatives, the source dataset uses the ms-marco-MiniLM-L6-v2 cross-encoder4 to score the top 50 BM25-retrieved passages per query. It retains negatives with scores more than 3.0 below the positive passage’s score. We retain English queries and passages of 5–150 tokens and remove near-duplicates using ROUGE-L (Lin, [38]) with a threshold of 0.7.

Speech Synthesis. We synthesize speech from these texts using Gemini 2.5 Pro TTS (Comanici et al., [16])5, Gemini 3.1 Flash TTS,6 and Qwen3 TTS (Hu et al., [25]). Voice selection determines speaker identity. The two Gemini models share 30 voices, while Qwen3 TTS uses Ryan and Aiden, as detailed in Table 5.

TTS modelVoice names
GeminiAchernar, Achird, Algenib, Algieba, Alnilam, Aoede, Autonoe, Callirrhoe, Charon, Despina, Enceladus, Erinome, Fenrir, Gacrux, Iapetus, Kore, Laomedeia, Leda, Orus, Puck, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zephyr, Zubenelgenubi
Qwen3 TTSRyan, Aiden

Table 5. Synthesis voices used for synthetic data construction.

Speaking Styles. Synthesis prompts control ten speaking styles: happy, sad, angry, fearful, surprised, disgusted, excited, sleepy, sarcastic, and whispering. Each prompt describes the voice, tone, pacing, and emotion. Table 6 provides the full prompts.

StylePrompt
HappyVoice: bright, cheerful, and upbeat, as if you are smiling while you speak. Tone: warm, joyful, and openly affectionate. Pacing: lively and energetic, with buoyant rhythm. Emotion: genuinely delighted and optimistic, emphasizing key words with an uplifted intonation.
SadVoice: soft, low, and melancholic, with a gentle tremor. Tone: heavy, sorrowful, and subdued, as if holding back tears. Pacing: slow and drawn out, with thoughtful pauses. Emotion: deep sadness and quiet grief, letting the weight of the words settle.
AngryVoice: sharp, intense, and forceful, with hard consonants. Tone: confrontational, tense, and biting. Pacing: clipped and driving, with emphatic stress on charged words. Emotion: strong anger held just under control, barely restrained frustration.
FearfulVoice: trembling, breathy, and slightly higher in pitch. Tone: anxious, uncertain, and vulnerable. Pacing: uneven, with nervous hesitations and quick shallow breaths. Emotion: palpable fear, as if something dangerous is nearby and you are trying not to be heard.
SurprisedVoice: wide-eyed and astonished, with sudden lifts in pitch. Tone: amazed and caught off guard. Pacing: quick bursts punctuated by short breathy intakes. Emotion: genuine shock and disbelief, emphasizing unexpected words with emphatic stress.
DisgustedVoice: curled-lipped and repulsed, with nasal resonance. Tone: scornful, dismissive, and revolted. Pacing: slightly drawn-out on offensive words, with sharp cutoffs elsewhere. Emotion: strong aversion, as if the subject is physically distasteful.
ExcitedVoice: thrilled, animated, and high-energy, with rising intonation. Tone: enthusiastic and eager, barely able to contain yourself. Pacing: fast and breathless, tumbling from one idea to the next. Emotion: exuberant anticipation and joyful hype.
SleepyVoice: drowny, soft, and slightly slurred. Tone: lethargic, relaxed, and half-awake. Pacing: slow and drawn out, with yawning pauses. Emotion: deep tiredness, as if you are about to drift off mid-sentence.
SarcasticVoice: dry, flat, and ironic, with pointed emphasis. Tone: mocking and knowing, clearly meaning the opposite of the words. Pacing: deliberate, with exaggerated stress on key words. Emotion: wry amusement and condescending disbelief.
WhisperingVoice: quiet, breathy whisper with almost no vocal cord engagement. Tone: intimate, secretive, and conspiratorial. Pacing: careful and deliberate, with clear articulation despite the low volume. Emotion: hushed urgency, as if sharing a secret that must not be overheard.

Table 6. Synthesis prompts for ten speaking styles, describing voice, tone, pacing, and emotion.

Environmental Sound. We use clips from eight UrbanSound8K (Salamon et al., [64]) classes: air conditioner, car horn, dog bark, drilling, engine idling, gunshot, jackhammer, and siren. Before cropping, we discard clips with root-mean-square levels below −45 dB relative to full scale. For each speech recording, we uniformly sample an environmental clip that passes this threshold.

Sound Mixing. We convert signals to mono by averaging channels and resample the environmental clip to match the speech sample rate. If the clip is longer than the speech, we randomly crop a segment matching the speech duration. Otherwise, we retain the full clip and place it at a uniformly sampled position within the speech. We uniformly sample a signal-to-noise ratio from {5, 10, 15} dB and scale the environmental signal using the mean squared amplitudes of both signals over the mixing interval.

Pair Construction

We construct retrieval requirements and document pairs from the annotations described in Appendix § 8.§ 1, following § 4.2.

We balance sampling across attribute keys and values in the retrieval requirements. For each spoken query qq, we sample a nonempty subset rq⊆A(q)r_q \subseteq A(q). To form rsr_s, we may select attribute keys annotated for qq but not used in rqr_q. For each selected key, we sample a value to be specified in the instruction. Speaker identity, dialogue, and answer relevance appear only in rqr_q and refer to the spoken query. When rr includes speaker identity, we exclude gender and accent, since matching the speaker also guarantees matching these attributes in our data.

Pair Selection. For each spoken query and set of requirements rr, we select a positive document satisfying all requirements and a negative document violating at least one. We prioritize documents that violate fewer requirements in rr to obtain harder negatives.

Instruction Generation

We use Gemini 3.1 Pro Preview7 and Claude Opus 4.68 to express each set of requirements rr as natural-language instructions zz.

Two prompt variants produce short instructions and full-sentence instructions. We generate approximately 200 instructions per set of requirements before filtering and deduplication.

Prompt Variants. Table 7 presents the prompt templates, and Table 8 provides examples of generated instructions. Both templates distinguish query-matching requirements in rqr_q from specified values in rsr_s. The short-instruction template requests phrases of two to six words, while the full-sentence template requests conversational sentences. Both encourage diverse wording and avoid technical terms such as “criteria”, “attribute”, and “metadata”.

Short instructionsFull-sentence instructions
You are generating {N} very short, informal retrieval cues for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Each cue should be an informal, telegraphic phrase a user might type in a chat or voice interface: keyword-style fragments, not full sentences. Requirements: 1. Very short, typically 2 to 6 words 2. Lowercase is fine, no trailing punctuation 3. No full sentences; use fragments, comma-separated keywords, or elliptical phrases 4. For MATCH constraints use phrases such as "same emotion", "matching accent" 5. For TARGET constraints use phrases such as "female", "happy", "british accent" 6. A natural example for this pattern is "{example_cue}" 7. Vary wording, word order, and synonyms across cues; do not repeat the example 8. Do NOT mention technical terms such as "requirements", "attribute", or "metadata" Output ONLY the cues, one per line, numbered 1 to {N}.You are generating diverse natural language retrieval instructions for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Generate {N} diverse and natural retrieval instructions. Each instruction should: 1. Clearly indicate which attributes should MATCH the spoken query (e.g., "same emotion", "matching gender", "from the same speaker") 2. Clearly indicate which attributes should have a SPECIFIC TARGET value (e.g., "with a happy emotion", "in a British accent") 3. Use varied sentence structures, synonyms, and phrasings 4. Be natural and conversational 5. Not mention technical terms such as "requirements", "attribute", or "metadata" 6. Refer to "the spoken query" or "the given audio" when referring to the query Output ONLY the instructions, one per line, numbered 1 to {N}.

Table 7. Prompt templates for generating short instructions and full-sentence instructions. Only terminology is adapted to match the paper. The templates distinguish query-matching requirements in rqr_q from specified values in rsr_s. Braces mark fields filled for each set of requirements, and N denotes the requested number of instructions.

RequirementsShort instructionFull-sentence instruction
Same speakeridentical voiceFetch another clip of the speaker from this audio clip.
Same environmental soundmatching ambianceFind audio that matches the background sound environment of this recording.
Same speaker, angry speaking styleanger, same voiceSearch for an angry version of this person’s voice.

Table 8. Examples of short instructions and full-sentence instructions for the same retrieval requirements. “Same” indicates a query-matching requirement in rqr_q. The final row also includes a specified speaking style of “angry” in rsr_s.

Filtering. We retain instructions that express all requirements in rr. Instructions must refer to the spoken query for rqr_q and specify the values in rsr_s. We allow synonyms and natural paraphrases of specified values, but query-matching requirements must identify the correct attribute. For example, accent matching must refer to “accent” rather than “speaking style” or “way of speaking.”

Deduplication. Within each pattern of requirements, we consider instructions in random order and discard any instruction with a ROUGE-L F1 score of at least 0.7 against a previously retained instruction.

Statistics

The dataset contains 899,931 samples, comprising 800,000 from real data and 99,931 from synthetic data. Table 9 reports the training and validation counts by source.

SourceTrainingValidation
Real data
VoxCeleb1 ([49])294,3032,974
ESD ((Zhou et al., 2021))115,2821,182
MELD ([57])72,526791
CREMA-D ([11])56,754584
L2-ARCTIC ((Zhao et al., 2018))53,022537
LibriSpeech-C ([41])51,553521
EmoV-DB ([1])49,834525
MSP-IMPROV ([10])44,763459
VocalSound ([22])20,678209
SpokenWOZ ([67])13,361135
MultiDialog ([55])12,941131
UrbanSound8K ([64])5,84459
EMNS ((Noriy et al., 2023))1,02210
Real data subtotal791,8838,117
Synthetic data
MS MARCO ([50])98,3951,536
Total890,2789,653

Table 9. Training and validation sample counts by source. Each sample comprises a spoken query, an instruction, and positive and negative spoken documents.

Data Split. We shuffle the samples and keep those sharing the same spoken query in a single split. The resulting training and validation sets contain 890,278 and 9,653 samples, respectively.

Preventing Data Contamination. Our retrieval training data exclude test recordings from INSPIRE ([35]) and the 15 evaluated MAEB ([5]) tasks. We exclude MAEB tasks derived from CREMA-D and VoxCeleb because these corpora are included in our training data. Synthetic training uses MS MARCO text and UrbanSound8K sounds, whereas INSPIRE’s Synthetic subset uses Natural Questions text and ESC-50 sounds. The synthesis voices, text-to-speech models, and instruction-generation models also differ from those used in INSPIRE.

Training Details

Setup

We train IRIS-Embed using Qwen2.5-Omni-3B (Xu et al., 2025a)9 and Gemma-4-E2B-it [71]10 with the ms-swift package (Zhao et al., 2024)11. We optimize the contrastive objective in § 3.3 using the data splits in Appendix § 8.§ 6. We evaluate the final checkpoint after two epochs.

Optimization. Both models use fused AdamW with the same learning rate for all trainable components. Weight decay is not applied to biases or normalization parameters. Training uses bfloat16 precision and DeepSpeed ZeRO stage 2 across 16 NVIDIA GB200 GPUs. Table 10 summarizes the optimization settings.

SettingValue
Learning rate2×10−42 \times10^{-4}
Adam coefficientsβ1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95
Adam epsilon10−810^{-8}
Weight decay0.1
Maximum gradient norm1.0
ScheduleLinear warmup, then cosine decay to zero
WarmupFirst 5% of optimizer steps
Training duration2 epochs; 3,476 optimizer steps
Global batch size512 samples
Batch size per GPU32 samples
Contrastive temperature0.03

Table 10. Training settings shared by both models. The global batch contains 32 samples on each of 16 GPUs.

Trainable Components. The audio layer weighting module gg uses the pretrained all-MiniLM-L6-v2 sentence encoder [61] (Wang et al. 2020)12. Both the audio encoder and sentence encoder remain frozen. We adapt the language model using LoRA with rank 64, scaling parameter 128, and dropout 0.05. The audio projection and the added normalization and linear layers in gg are trained directly alongside the LoRA parameters.

LoRA Configuration. For Qwen2.5-Omni-3B, we apply LoRA to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj across all 36 language-model layers. For Gemma-4-E2B-it, we adapt the applicable attention and feed-forward projections across all 35 language-model layers, along with per_layer_input_gate, per_layer_projection, and the shared per_layer_model_projection. Since later layers reuse keys and values, LoRA adapters for k_proj and v_proj are applied only to the first 15 layers.

Data Ordering. Before training, we arrange samples by source and spoken query in groups of approximately 64. Grouping by source emphasizes within-source negatives, particularly for smaller datasets. For dialogue continuation, this ordering is intended to reduce shortcuts based on speaker identity or recording conditions. Grouping by spoken query places different instructions for the same query close together. Different instructions may select different positive documents, so this ordering is intended to encourage retrieval to depend on the instruction’s requirements.

Audio Layer Weighting

Computation. To compute g(p)g(p), we mean-pool MiniLM’s non-padding content hidden states, then apply learned layer normalization and a linear layer. We apply softmax to obtain weights for the pre-encoder hidden states and all audio-encoder layer hidden states: 33 hidden-state sequences in Qwen2.5-Omni-3B and 13 in Gemma-4-E2B-it. Each model pools and projects the weighted audio hidden states using its existing operations, including normalization, before passing them to the language model.

Initialization. We initialize the linear weights from a Gaussian distribution with mean 0 and standard deviation 10−410^{-4}. For Qwen2.5-Omni-3B, all biases are initialized to zero, yielding approximately uniform layer weights. For Gemma-4-E2B-it, we initialize the layer weights to emphasize the final audio layer, reducing the initial deviation from its original representation. Let NN denote the number of audio hidden-state sequences, including the pre-encoder hidden states. We set the final-layer bias to

bfinal=log⁡0.9(N−1)0.1,(1)b_{\mathrm{final}} = \log\frac{0.9(N - 1)}{0.1}, \tag*{(1)}

while initializing all other biases to zero. For Gemma-4-E2B-it, N=13N = 13, so this initialization assigns approximately 90% of the initial weight to the final layer and distributes the remainder across other layers.

Prompts

Table 11 presents our prompt templates, adapted from Qwen3-VL-Embedding [36] to each model’s chat and audio formats. The query prompt includes the instruction and spoken query, while the document prompt contains the spoken document. We extract the final-layer hidden state of the last non-padding input token as the embedding without generating a response.

Qwen2.5-Omni-3B Query Prompt:Qwen2.5-Omni-3B Document Prompt:
<|im_start|>user<br>Instruction: [instruction]<br>Query: <|audio_bos|>[audio_tokens]<|audio_eos|><br><|im_end|><br><|im_start|>assistant<|im_start|>user<br>Document: <|audio_bos|>[audio_tokens]<br><|audio_eos|><|im_end|><br><|im_start|>assistant
Gemma-4-E2B-it Query Prompt:Gemma-4-E2B-it Document Prompt:
<bos><|turn|>user<br>Instruction: [instruction]<br>Query: [audio_tokens]<turn|><br><|turn|>model<bos><|turn|>user<br>Document: [audio_tokens]<turn|><br><|turn|>model
MiniLM Query Prompt:MiniLM Document Prompt:
Instruction: [instruction]<br>Query:Document:

Table 11. IRIS-Embed prompt templates. Red: input labels; blue: placeholders.

For audio layer weighting, MiniLM processes text-only prompts. The query prompt includes the instruction, while the document prompt remains fixed across all spoken documents.

Evaluation Details

INSPIRE

Benchmark. INSPIRE [35]13 evaluates spoken-document retrieval given a spoken query and a natural-language instruction. Instructions describe requirements for speech attributes. Thus, the same spoken query may have different relevant documents depending on the instruction. A document is relevant only if it satisfies all requirements.

Tasks. DailyTalk [33] uses the first half of a dialogue as the spoken query and evaluates retrieval of its continuation. VCTK (Yamagishi et al., 2019) evaluates speaker matching using hard negatives with the same spoken content but different speakers. Expresso [51] evaluates speaker and speaking-style matching, separately or jointly, including instructions with specified speaking styles.

The Synthetic subset combines spoken queries and documents synthesized from Natural Questions [29] using GPT-4o mini TTS [54] with environmental sounds from ESC-50 [56]. Synthetic evaluates answer retrieval and matching by speaker, speaking style, and environmental sound, including instructions with multiple attributes. Speaking style and environmental sound may match the spoken query or take values specified in the instruction. Appendix § 8.§ 6 describes the separation between training and evaluation data.

Metric. For each query–instruction pair, embedding methods rank documents in the corresponding subset by cosine similarity. Recall@KK measures the fraction of relevant documents retrieved among the top KK results. Let QZQ_Z denote the evaluated query–instruction pairs, πK(q,z)\pi_K(q,z) the top KK retrieved documents, and D+(q,z)D^{+}(q,z) the relevant documents. We compute average recall as

Recall⁡@K=1∣QZ∣∑(q,z)∈QZ∣πK(q,z)∩D+(q,z)∣∣D+(q,z)∣(2)\operatorname{Recall}@K = \frac{1}{|Q_Z|}\sum_{(q,z)\in Q_Z}\frac{|\pi_K(q,z)\cap D^{+}(q,z)|}{|D^{+}(q,z)|} \tag*{(2)}

Each query–instruction pair contributes equally to the average. We report Recall@KK as percentages for K∈{10,50,100}K \in\{10, 50, 100\} on each subset. For the instruction-type analysis in § 6.1, we compute Recall@50 separately for each instruction type.

Comparison Settings

Table 1 includes published baseline results.

Random. The random baseline averages uniform random document rankings over 100 runs [35].

General Speech Encoders. WavLM-large [14]14 encodes spoken queries and documents without instructions. Embeddings are obtained by mean-pooling its final-layer hidden states over time.

Specialized Speech Encoders. ECAPA-TDNN [17]15 and emotion2vec-base [45]16 extract speaker and emotion embeddings from their final-layer representations, respectively. Both encode spoken queries and documents without instructions.

Cascaded Retrieval. Whisper-large-v3 [58]17 generates transcripts, and Qwen3-Omni-30B-A3B-Captioner (Xu et al., 2025b)18 generates captions for spoken queries and documents. Qwen3-Embedding-8B (Zhang et al., 2025b)19 encodes the transcripts and captions, with the instruction included in the query input.

LALM Embeddings. Qwen2.5-Omni-3B (Xu et al., 2025a)20, Gemma-4-E2B-it [71]21, and Qwen3-Omni-30B (Xu et al., 2025b)22 follow INSPIRE’s evaluation setup [35] without additional embedding training. For these models, query and document inputs end with “Summarize the above sentence and speech in one word” and “Summarize the above speech in one word”, respectively. LCO-Embedding-Omni-3B (Xiao et al., 2025)23 follows its published embedding extraction configuration. All four methods extract embeddings from the final-layer hidden state of the last non-padding input token without generating a response.

IRIS-Embed. We evaluate the final IRIS-Embed checkpoints from Appendix § 9 with learned audio layer weighting. Table 11 gives the query and document prompt templates. We extract the final-layer hidden state of the last non-padding input token as the embedding without generating a response.

Layer Aggregation

Table 2 compares audio layer aggregation for Qwen2.5-Omni-3B. Layer 13 and Final Layer use the 13th and final audio-encoder hidden states, respectively, followed by pooling and projection into the language model. Mean Layer assigns equal weights to the pre-encoder hidden states and all layer hidden states. Finetuned updates the audio encoder during retrieval training, while the other variants keep it frozen. Static Weight learns a single set of layer weights shared across all queries and documents, independent of instructions or prompts. Inst. Weight is our instruction-aware audio layer weighting method, which predicts weights from text prompts as described in § 3.2.1. Query weights vary with the instruction, while document weights are computed from a fixed prompt. Mean Layer, Static Weight, and Inst. Weight combine layers 0–32 for Qwen2.5-Omni-3B, with layer 0 denoting the pre-encoder hidden states. Both weighting variants use the same instructions in the language-model query prompt.

Final Layer, Mean Layer, Static Weight, and Inst. Weight use the training data and language-model LoRA settings in Appendix § 9 and train the audio projection. For Gemma-4-E2B-it, we compare Final Layer, Mean Layer, and Inst. Weight. Appendix § 11.§ 3 provides the full results for both models. Appendix § 11.§ 4 separately examines frozen and trainable audio projections for Final Layer.

MAEB

Tasks. We evaluate 15 audio tasks from MAEB [5]24, following the task definitions and metrics of its audio-only beta leaderboard.25 We exclude tasks derived from CREMA-D [11] and VoxCeleb [49], as these corpora are included in our retrieval training data. Table 3 reports scores on a 0–100 scale using shortened task names.

Baseline Scores. We evaluate the Qwen2.5-Omni-3B and Gemma-4-E2B-it baselines with the same task instructions as IRIS-Embed. Scores for the other baselines come from the MAEB beta leaderboard and are averaged uniformly across language subsets for each task.

Embedding Extraction. Table 12 lists the full task names and corresponding instructions. For all tasks except retrieval, we encode every recording with its task instruction using the query prompt in Table 11. Retrieval uses separate query and document prompts, as described below. Each input passes through the full IRIS-Embed model, and we extract the final-layer hidden state of the last non-padding input token as its embedding. Model parameters remain fixed throughout evaluation.

TaskTypeInstruction
BeijingOperaClfFind recordings with the same percussion instrument as the query.
BirdCLEFClfFind recordings with the same bird species as the query.
CommonLanguageAgeDetectionClfFind recordings with the same speaker age group as the query.
GTZANGenreClfFind recordings with the same music genre as the query.
IEMOCAPGenderClfFind recordings with the same gender as the query.
MInDS14ClfFind recordings with the same intent as the query.
MridinghamTonicClfFind recordings with the same tonic as the query.
SIBFLEURSClfFind recordings with the same topic as the query.
VoxPopuliLanguageIDClfFind recordings with the same spoken language as the query.
VehicleSoundClusteringClustFind recordings with the same environmental sound as the query.
VoxPopuliGenderClusteringClustFind recordings with the same gender as the query.
NMSQAPairClassificationPCFind recordings that contain the answer to the query.
VoxPopuliAccentPairClassificationPCFind recordings with the same accent as the query.
GTZANAudioRerankingRrnkFind recordings with the same music genre as the query.
JamAltArtistA2ARetrievalRtrvlFind recordings with the same artist as the query.

Table 12. Task instructions used to extract IRIS-Embed query embeddings for MAEB evaluation. Clf: classification; Clust: clustering; PC: pair classification; Rrnk: reranking; Rtrvl: retrieval.

Classification. Following MAEB (Assadi et al., [5], nine tasks evaluate classification of percussion instruments [72], bird species [28], speaker age groups [68], music genres (Tzanetakis and Cook, 2002), gender [9], intent [20], musical tonic [2], topic [65], and language (Wang et al., 2021). Evaluation uses a logistic-regression probe fitted on fixed embeddings with eight samples per class. Accuracy is averaged over ten resamples.

Clustering. VehicleSound [7] and VoxPopuliGender (Wang et al., 2021) evaluate clustering by sound class and gender, respectively. Following MAEB, MiniBatchKMeans uses the number of reference classes as the number of clusters. Performance is measured by V-measure [62] over bootstrapped subsamples.

Pair Classification. NMSQA [40] and VoxPopuliAccent (Wang et al., 2021) evaluate the relevance of recording pairs. Following MAEB, we report the highest average precision across the benchmark’s similarity measures.

Reranking. GTZAN (Tzanetakis and Cook, 2002) evaluates reranking of a provided set of relevant documents and hard negatives using cosine similarity. We report mean average precision at 1,000, denoted MAP@1000.

Retrieval. JamAlt [15] evaluates audio-to-audio retrieval of songs by the same artist as the query. The query prompt includes the query recording and task instruction, while the document prompt contains the candidate recording without an instruction. We report normalized discounted cumulative gain at 10, denoted nDCG@10.

Full Results

Instruction Types

Table 13 reports Recall@50 by instruction type on INSPIRE, covering instructions with a single attribute or multiple attributes. Relative to the corresponding LALM embedding baselines, IRIS-Embed improves Recall@50 in all 21 evaluation settings for Qwen2.5-Omni-3B and 20 of 21 for Gemma-4-E2B-it. The exception is matching speaking style with a specified environmental sound on Synthetic, where Gemma-4-E2B-it achieves 1.25% with IRIS-Embed versus 1.50% with the LALM baseline.

MethodModelDailyTalk Cont.VCTK Spk.Expresso Spk.Expresso Sty.Expresso Spk. +Sty.Expresso Spk. +Sty.*Synthetic Ans.Synthetic Spk.Synthetic Sty.Synthetic Env.
RANDRandom1.061.841.731.461.381.420.841.091.381.04
ENCWavLM-large24.5035.0011.5030.7527.501.000.750.758.757.50
SENCECAPA-TDNN24.00100.0071.5024.2570.500.751.6787.7513.501.75
SENCemotion2vec-base23.006.253.256.257.250.7516.752.2518.752.25
CASQwen3-Embedding-8B81.506.254.005.506.006.7598.501.751.004.50
LALMQwen2.5-Omni-3B53.003.132.006.003.753.2564.751.001.250.50
LALMGemma-4-E2B-it25.504.381.501.250.751.2550.920.501.500.50
LALMQwen3-Omni-30B74.503.132.005.254.502.5080.751.252.002.00
LALMLCO-Embedding-Omni-3B75.502.502.2516.258.257.0097.251.251.756.00
IRISQwen2.5-Omni-3B77.5085.0041.2542.0062.0034.7590.5053.5053.7535.50
IRISGemma-4-E2B-it44.5074.3833.0048.0063.7530.5080.5851.2547.258.50
MethodModelSynthetic Spk. +Sty.Synthetic Spk. +Sty.*Synthetic Spk. +Env.Synthetic Spk. +Env.*Synthetic Sty. +Env.Synthetic Sty. +Env.*Synthetic Sty.* +Env.Synthetic Spk. +Sty. +Env.Synthetic Spk. +Sty.* +Env.Synthetic Spk. +Sty. +Env.*Synthetic Spk. +Sty.* +Env.*
RANDRandom1.041.040.960.960.900.960.960.920.980.980.91
ENCWavLM-large6.251.507.250.759.001.003.2512.255.502.250.50
SENCECAPA-TDNN61.502.007.004.502.753.750.0020.501.2513.501.00
SENCemotion2vec-base2.751.754.001.255.500.501.256.002.251.751.00
CASQwen3-Embedding-8B2.501.503.002.752.002.753.254.251.752.252.50
LALMQwen2.5-Omni-3B1.500.500.500.752.001.250.250.500.751.501.00
LALMGemma-4-E2B-it1.751.501.002.001.751.501.500.751.250.751.25
LALMQwen3-Omni-30B1.000.500.500.751.250.750.751.001.250.751.25
LALMLCO-Embedding-Omni-3B1.751.002.753.006.253.002.504.502.754.252.50
IRISQwen2.5-Omni-3B54.2546.0047.5017.2535.7515.0038.0061.7557.0030.0026.75
IRISGemma-4-E2B-it47.0045.2515.507.2513.501.258.2531.5018.2515.0013.75

Table 13. Full Recall@50 results by instruction type on INSPIRE. Bold and underline indicate the best and second-best results, respectively. Cont.: dialogue continuation; Ans.: answer retrieval; Spk.: speaker; Sty.: speaking style; Env.: environmental sound. + combines requirements; ∗ marks a specified value.

IRIS-Embed also outperforms other baselines across several instruction types. Both models outperform all LALM embedding baselines on speaker matching in VCTK and all four instruction types in Expresso. Both models also outperform all comparison baselines on speaking-style matching in Expresso and Synthetic, and environmental-sound matching in Synthetic.

For instructions with multiple attributes, the Qwen2.5-Omni-3B and Gemma-4-E2B-it versions of IRIS-Embed outperform all comparison baselines in 11 and 10 of the 13 evaluation settings across Expresso and Synthetic, respectively. On Synthetic, matching the spoken query’s speaker, speaking style, and environmental sound together gives Recall@50 of 61.75% and 31.50%, compared with 20.50% for the strongest baseline, ECAPA-TDNN. Matching the speaker and a specified speaking style yields 46.00% and 45.25%, compared with 2.00% for the strongest baseline. The gains therefore extend to requirements with query-matched and explicitly specified attribute values.

Instruction Awareness

We assess whether IRIS-Embed follows retrieval instructions by varying the instruction while keeping the spoken queries, document collection, and relevance labels fixed for each set of evaluation requirements. The tables report Recall@50, with instructions in rows and evaluation requirements in columns. Table 1 and Table 13 use the original INSPIRE instructions. Here, we replace them with fixed short or full-sentence instructions, so recall may differ from those results.

We consider five instructions with a single attribute: dialogue continuation, answer retrieval, speaker matching, speaking-style matching, and environmental-sound matching. Dialogue continuation and answer retrieval specify different relevance requirements for spoken content. Table 14 lists a short instruction and a full-sentence instruction for each of the five instruction types. The left sides of Figure 5 and Figure 7 show the Short rows from Table 15 for Qwen2.5-Omni-3B and Gemma-4-E2B-it, respectively. For both models, the instruction matching the evaluation requirements achieves the highest Recall@50 across all four Synthetic evaluation settings.

Figure 7. Recall@50 for IRIS-Embed with Gemma-4-E2B-it. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound; +: multiple attributes.

LabelShort instructionFull-sentence instruction
Cont.Continuation.Find speech that continues the conversation in the query.
Ans.Find the answer.Find speech that contains the answer to the question in the query.
Spk.Same speaker.Find speech from the same speaker as the query.
Sty.Same speaking style.Find speech with the same speaking style as the query.
Env.Same environmental sound.Find speech with the same environmental sound as the query.
Doc.Document:

Table 14. Instructions for instruction awareness and layer-weight analysis. The main-paper instruction-awareness and layer-weight analyses use short instructions. Doc. denotes the document prompt used in the layer-weight analysis.

InstructionFormDailyTalk Cont.VCTK Spk.Expresso Spk.Expresso Sty.Expresso Ans.Synthetic Spk.Synthetic Sty.Synthetic Env.
Qwen2.5-Omni-3B
Continuation. Find speech that continues the conversation in the query.Short79.0081.8819.7522.5087.252.752.752.75
Full77.5083.7531.0015.7563.0018.752.502.00
Find the answer. Find speech that contains the answer to the question in the query.Short82.5073.758.004.2590.252.001.751.25
Full78.5067.506.253.5091.002.001.500.25
Same speaker. Find speech from the same speaker as the query.Short63.5085.6340.7513.501.5053.005.500.75
Full67.5085.6341.0013.501.5053.006.000.75
Same speaking style. Find speech with the same speaking style as the query.Short10.5030.004.0042.500.502.7555.000.50
Full21.0030.003.0041.000.502.7555.751.00
Same environmental sound. Find speech with the same environmental sound as the query.Short20.5034.384.508.751.500.750.5036.75
Full29.0035.634.7510.001.501.000.5037.25
No instruction75.0080.6320.7526.0074.583.253.758.75
Gemma-4-E2B-it
Continuation. Find speech that continues the conversation in the query.Short48.5076.2517.7516.7571.085.751.752.00
Full49.0075.6319.2513.7573.833.501.251.25
Find the answer. Find speech that contains the answer to the question in the query.Short42.0035.006.256.0082.580.752.501.00
Full41.5036.256.756.0083.081.002.251.25
Same speaker. Find speech from the same speaker as the query.Short39.5075.0032.5017.001.0051.504.752.25
Full45.5075.6332.2516.751.5052.754.752.50
Same speaking style. Find speech with the same speaking style as the query.Short15.0056.256.5050.500.255.0047.753.00
Full22.5054.387.2547.250.254.5046.752.50
Same environmental sound. Find speech with the same environmental sound as the query.Short35.0031.884.2517.751.002.003.259.75
Full41.0035.635.0014.001.001.752.509.50
No instruction44.0075.6320.0039.008.7521.2524.253.75

Table 15. Recall@50 for IRIS-Embed with short instructions, full-sentence instructions, or no instruction. Rows show the exact instructions; Short/Full denote short/full-sentence instructions. Cont./Ans.: dialogue continuation/answer retrieval; Spk./Sty./Env.: speaker/speaking style/environmental sound.

Table 15 reports both wording forms, labeled Short and Full, and a no-instruction condition. Within each wording form, the matching instruction achieves the highest Recall@50 for both models across all four Synthetic evaluation settings and for speaker and speaking-style matching on Expresso. The matching instruction also performs best for Qwen2.5-Omni-3B on VCTK and Gemma-4-E2B-it on DailyTalk. Both short instructions and full-sentence instructions therefore guide retrieval according to the requested requirements in most evaluation settings.

The matching instruction does not always achieve the highest recall. On DailyTalk, Qwen2.5-Omni-3B performs best with the answer-retrieval instruction under both wording forms. This may reflect overlap between answering a question and continuing a dialogue. On VCTK, Gemma-4-E2B-it achieves 76.25% with the short continuation instruction and 75.00% with the short speaker-matching instruction. The corresponding full-sentence instructions and no instruction all yield 75.63%, suggesting that speaker information remains available without an explicit speaker-matching instruction.

We further evaluate four short instructions with multiple attributes of the spoken query. These cover all pairwise combinations of speaker, speaking style, and environmental sound, as well as all three attributes together. The right panels of Figure 5 and Figure 7 show the rows for these four short instructions in Table 16 across the corresponding Synthetic evaluation settings. The matching instruction achieves the highest Recall@50 in all four evaluation settings for Qwen2.5-Omni-3B and three of four for Gemma-4-E2B-it. When relevance requires matching all three attributes, the corresponding instruction outperforms all pairwise instructions for both models.

InstructionQwen2.5-Omni-3B Spk. +Sty.Qwen2.5-Omni-3B Spk. +Env.Qwen2.5-Omni-3B Sty. +Env.Qwen2.5-Omni-3B Spk. +Sty. +Env.Gemma-4-E2B-it Spk. +Sty.Gemma-4-E2B-it Spk. +Env.Gemma-4-E2B-it Sty. +Env.Gemma-4-E2B-it Spk. +Sty. +Env.
Find the answer.0.751.753.252.001.001.251.751.50
Continuation.1.504.505.256.502.753.753.254.50
Same speaker.18.508.502.5010.2518.008.504.0014.25
Same speaking style.9.502.253.257.0012.254.507.7512.25
Same environmental sound.1.5010.7515.0011.752.506.759.2510.50
Same speaker and same speaking style.52.0011.507.5027.2549.7512.755.2528.00
Same speaker and same environmental sound.8.2545.2511.7545.2515.5013.254.5022.00
Same speaking style and same environmental sound.6.0011.5035.0036.757.755.2511.7513.75
Same speaker and same speaking style and same environmental sound.30.0029.2522.2565.0046.7513.506.0033.25
No instruction6.0013.259.5017.7535.7511.257.7530.25

Table 16. Recall@50 for IRIS-Embed under short instructions and no instruction. Columns evaluate Synthetic requirements involving multiple attributes. Rows show the exact instructions; No instruction indicates an empty instruction. Spk., Sty., and Env. match the spoken query’s speaker, speaking style, and environmental sound; + combines requirements.

Table 16 provides the full comparison, including the five short instructions above and no instruction. For both models, the matching instruction outperforms these additional conditions in all four evaluation settings. These findings show that IRIS-Embed responds to instructions with multiple attributes.

Layer Aggregation

Table Table 17 reports Recall@10, Recall@50, and Recall@100 for audio layer aggregation, while Table 18 provides Recall@50 by instruction type. For Qwen2.5-Omni-3B, the comparison includes Layer 13, Final Layer, Mean Layer, Finetuned, Static Weight, and Inst. Weight. For Gemma-4-E2B-it, we compare Final Layer, Mean Layer, and our method. Appendix § 10.§ 3 describes these methods and their training settings.

Layer AggregationDailyTalk R@10DailyTalk R@50DailyTalk R@100VCTK R@10VCTK R@50VCTK R@100Expresso R@10Expresso R@50Expresso R@100Synthetic R@10Synthetic R@50Synthetic R@100Avg. R@10Avg. R@50Avg. R@100
Qwen2.5-Omni-3B
Layer 135.5015.5022.5061.8876.8884.3832.4448.9457.7517.9535.2746.1229.4444.1552.69
Final Layer59.5082.5085.5034.3848.7559.3815.8828.2536.8818.4734.7245.2032.0648.5656.74
Mean Layer47.5070.0078.5057.5071.2581.2525.7543.1352.6326.0344.9055.0039.2057.3266.85
Finetuned14.5032.5042.0076.2591.2595.0031.1947.3855.5024.9043.3554.4936.7153.6261.75
Static Weight45.0066.0076.5061.8878.7587.5025.8843.1955.3126.4745.2856.1339.8158.3168.86
Inst. Weight (Ours)60.0077.5087.5067.5085.0089.3829.6345.0053.8826.0644.1754.7845.8062.9271.39
Gemma-4-E2B-it
Final Layer24.5042.5052.0030.6348.1359.3817.0630.0037.6912.0121.7029.7221.0535.5844.70
Mean Layer12.5025.0037.0052.5069.3878.1332.0046.0654.6311.6321.6529.3827.1640.5249.79
Inst. Weight (Ours)25.5044.5054.5058.7574.3879.3828.6343.8152.3115.0726.9436.0831.9947.4155.57

Table 17. Full Recall@KK for audio layer aggregation on INSPIRE. Avg.: macro-average across the four subsets. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Bold marks column maxima within each backbone.

For Qwen2.5-Omni-3B, our method achieves the highest macro-average Recall@50 of 62.92% in Table 2, a gain of 14.36 percentage points over Final Layer. It consistently outperforms Final Layer on VCTK, Expresso, and Synthetic. Final Layer remains higher on DailyTalk at Recall@50.

Layer 13 and Finetuned improve speaker and speaking-style matching over Final Layer, but have much lower dialogue-continuation recall. Fine-tuned consistently leads on VCTK and Layer 13 on Expresso. Our method consistently outperforms both variants on DailyTalk and Synthetic. These methods trade off performance across retrieval requirements.

Our method also exceeds Mean Layer and Static Weight in Recall@50 on DailyTalk, VCTK, and Expresso. The macro-average Recall@50 gains are 5.60 and 4.61 percentage points, respectively. Both alternatives remain higher on Synthetic at Recall@50. Static Weight also exceeds our method on Synthetic at Recall@10 and Recall@100, and on Expresso at Recall@100.

The instruction-type results further distinguish these methods. On answer retrieval, our method achieves 90.50% Recall@50, compared with 68.25% for Finetuned and 2.00% for Layer 13. In the pairwise comparison with Static Weight, our method has higher Recall@50 in seven of eight evaluation settings with a single attribute. For instructions with multiple attributes, Static Weight has higher recall in eight of thirteen settings, our method in four, and one is tied.

With Gemma-4-E2B-it, our method consistently outperforms Final Layer on VCTK, Expresso, and Synthetic, while Mean Layer leads on Expresso. Answer retrieval Recall@50 increases from 1.50% with Mean Layer to 80.58% with our method. However, Mean Layer performs better on several instructions that combine speaker matching with other attributes. The aggregate gains support the effectiveness of instruction-aware audio layer weighting with a frozen audio encoder.

Audio Projection

We examine whether training the audio projection improves retrieval using the final audio-encoder hidden states. Both variants freeze the audio encoder and train the language-model LoRA parameters. Frozen keeps the audio projection fixed, while Trainable updates it during retrieval training. Table 19 and Table 20 report results by subset and instruction type, respectively.

Audio ProjectionDailyTalk R@10R@10DailyTalk R@50R@50DailyTalk R@100R@100VCTK R@10R@10VCTK R@50R@50VCTK R@100R@100Expresso R@10R@10Expresso R@50R@50Expresso R@100R@100Synthetic R@10R@10Synthetic R@50R@50Synthetic R@100R@100Avg. R@10R@10Avg. R@50R@50Avg. R@100R@100
Qwen2.5-Omni-3B
Frozen69.0083.0087.5024.3841.8856.8814.1326.5036.5016.8330.8042.0031.0945.5555.72
Trainable59.5082.5085.5034.3848.7559.3815.8828.2536.8818.4734.7245.2032.0648.5656.74
Gemma-4-E2B-it
Frozen17.0034.0046.0024.3845.0058.1313.6324.6931.759.0615.8321.7816.0229.8839.42
Trainable24.5042.5052.0030.6348.1359.3817.0630.0037.6912.0121.7029.7221.0535.5844.70

Table 19. Recall@KK on INSPIRE using only the final audio-encoder hidden states. Avg.: macro-average across the four subsets. Frozen and Trainable indicate whether the audio projection is frozen or updated during retrieval training. Bold marks column maxima within each backbone.

Audio ProjectionDailyTalk Cont.VCTK Spk.Expresso Spk.Expresso Sty.Expresso Spk. +Sty.Expresso Spk. +Sty.*Synthetic Ans.Synthetic Spk.Synthetic Sty.Synthetic Env.
Qwen2.5-Omni-3B — Frozen83.0041.8814.5034.2539.2518.0092.2524.0039.7529.25
Qwen2.5-Omni-3B — Trainable82.5048.7511.2539.7544.7517.2594.5027.5040.5031.00
Gemma-4-E2B-it — Frozen34.0045.009.7535.2537.7516.0084.7517.2539.250.75
Gemma-4-E2B-it — Trainable42.5048.1313.5045.5043.2517.7583.2528.7537.507.75
Audio ProjectionSynthetic Spk. +Sty.Synthetic Spk. +Sty.*Synthetic Spk. +Env.Synthetic Spk. +Env.*Synthetic Sty. +Env.Synthetic Sty. +Env.*Synthetic Sty.* +Env.Synthetic Spk. +Sty. +Env.Synthetic Spk. +Sty.* +Env.Synthetic Spk. +Sty. +Env.*Synthetic Spk. +Sty.* +Env.*
Qwen2.5-Omni-3B — Frozen26.7523.0030.0017.0026.5012.0028.5039.0030.2525.7518.00
Qwen2.5-Omni-3B — Trainable30.7527.2535.7519.5028.5014.0029.5045.7538.5032.5025.25
Gemma-4-E2B-it — Frozen26.2519.505.003.503.002.503.7511.256.256.508.00
Gemma-4-E2B-it — Trainable30.5024.0014.509.2512.255.0010.5024.0013.2514.0011.00

Table 20. Recall@50 by instruction type on INSPIRE using only the final audio-encoder hidden states. Frozen and Trainable indicate whether the audio projection is frozen or updated during retrieval training. Instruction labels follow Table 13. Bold marks column maxima within each backbone.

For both models, the trainable audio projection consistently outperforms the frozen projection on VCTK, Expresso, and Synthetic. On Synthetic, these gains cover all eleven instruction types with multiple attributes at Recall@50. The effects on spoken-content retrieval vary between models. At

Recall@50, Qwen2.5-Omni-3B improves on answer retrieval but declines on dialogue continuation, while Gemma-4-E2B-it shows the opposite trend.

With a trainable audio projection in both settings, Weighted Layer consistently outperforms Final Layer on VCTK, Expresso, and Synthetic for both models. Thus, the observed gains over Final Layer persist when both variants train the audio projection.

Layer Weights

We analyze how audio layer weights vary across retrieval requirements and instruction wording using the ten instructions in Table 14. The document prompt serves as a reference without an instruction. All prompts follow the MiniLM templates in Table 11. Figure 8 presents all eleven prompts for both models, while Figure 6 focuses on the five short instructions and the document prompt for Qwen2.5-Omni-3B. For Gemma-4-E2B-it, layer 4 receives the highest adjusted weight under every query instruction except the short answer-retrieval instruction, which places the highest weight on the final layer.

Figure 8. Audio layer weights adjusted for hidden-state scale. S/F: short/full-sentence instructions; Doc.: document prompt; 0: pre-encoder hidden states. Encoder layers are indexed from one. Color scales differ between models.

To account for differences in hidden-state scale, we adjust the layer weights using the same 680 unique INSPIRE spoken queries for every prompt and both models. Let wℓ(p)w_{\ell}(p) denote the original weight for layer ℓ\ell under text prompt pp. For a recording xx, let sℓ(x)s_{\ell}(x) denote the root mean square of the hidden states across hidden dimensions, averaged over valid frames. We compute the adjusted weights as

w^ℓ(x,p)=wℓ(p)sℓ(x)∑kwk(p)sk(x)\widehat{w}_{\ell}(x,p)=\frac{w_{\ell}(p)s_{\ell}(x)}{\sum_{k}w_{k}(p)s_{k}(x)}

where kk and ℓ\ell index the pre-encoder hidden states and all audio-encoder layer hidden states. The adjusted weights sum to one for each recording and prompt. We first average them within each INSPIRE subset, then average the four subset means equally. This adjustment is used only for analysis and does not affect model computation. The adjusted weights describe the relative scales of weighted audio layer hidden states, not their causal contributions to retrieval scores.

Layer Representations

We extend the analysis in § 3.2.1 to examine whether audio projection affects the best-performing layer. For each layer, we apply the same frozen audio projection, mean-pool the projected hidden states over time, and rank spoken documents by cosine similarity. Both the audio encoder and projection remain frozen, and the language model is not used. Figure 9 reports Recall@50 for dialogue continuation on DailyTalk, speaker matching on VCTK, speaking-style matching on Expresso, and answer retrieval on Synthetic. The best-performing layer continues to vary across retrieval requirements after projection.

Figure 9. Recall@50 using frozen audio-encoder hidden states after audio projection. 0 denotes the pre-encoder hidden states; encoder layers are indexed from one.

Layer AggregationDailyTalk Cont.DailyTalk Spk.VCTK Spk.Expresso Spk.Expresso Sty.Expresso Spk. +Sty.Expresso Spk. +Sty.*Synthetic Ans.Synthetic Spk.Synthetic Sty.Synthetic Env.
Qwen2.5-Omni-3B
Layer 1315.5076.8844.7545.5067.2538.252.0043.5054.7539.25
Final Layer82.5048.7511.2539.7544.7517.2594.5027.5040.5031.00
Mean Layer70.0071.2537.7542.0057.7535.0087.7550.7545.5040.50
Finetuned32.5091.2551.5042.2565.0030.7568.2552.5054.5043.25
Static Weight66.0078.7533.0041.7561.5036.5087.0048.5052.2545.50
Inst. Weight (Ours)77.5085.0041.2542.0062.0034.7590.5053.5053.7535.50
Gemma-4-E2B-it
Final Layer42.5048.1313.5045.5043.2517.7583.2528.7537.507.75
Mean Layer25.0069.3832.7542.2572.5036.751.5054.5044.504.00
Inst. Weight (Ours)44.5074.3833.0048.0063.7530.5080.5851.2547.258.50

Table 18. Recall@50 by instruction type for audio layer aggregation on INSPIRE. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Instruction labels follow Table 13; bold marks column maxima within each backbone.

Layer AggregationSynthetic Spk. +Sty.Synthetic Spk. +Sty.*Synthetic Spk. +Env.Synthetic Spk. +Env.*Synthetic Sty. +Env.Synthetic Sty. +Env.*Synthetic Sty.* +Env.Synthetic Spk. +Sty. +Env.Synthetic Spk. +Sty.* +Env.Synthetic Spk. +Sty. +Env.*Synthetic Spk. +Sty.* +Env.*
Qwen2.5-Omni-3B
Layer 1344.5040.5046.0015.0037.758.5034.5063.0058.5022.5018.75
Final Layer30.7527.2535.7519.5028.5014.0029.5045.7538.5032.5025.25
Mean Layer49.7543.7555.0020.7535.7516.5038.7565.5056.2538.7528.25
Finetuned49.7543.5051.2519.5035.7511.0039.5067.5062.7532.5018.75
Static Weight48.2541.2550.2522.0040.5018.2537.2567.0057.2537.2526.75
Inst. Weight (Ours)54.2546.0047.5017.2535.7515.0038.0061.7557.0030.0026.75
Gemma-4-E2B-it
Final Layer30.5024.0014.509.2512.255.0010.5024.0013.2514.0011.00
Mean Layer55.5050.0013.256.004.503.753.5029.2525.0014.0015.50
Inst. Weight (Ours)47.0045.2515.507.2513.501.258.2531.5018.2515.0013.75

Table 18 (PDF p. 31). Recall@50 by instruction type for audio layer aggregation on INSPIRE. Inst. Weight denotes instruction-aware audio layer weighting. Static Weight uses trainable weights independent of instructions and prompts. Instruction labels follow Table 13; bold marks column maxima within each backbone.

References

  1. [1]Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadbabs, and Thierry Dutoit. 2018. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514.arxiv.org/abs/1806.09514
  2. [2]Akshay Anantapadmanabhan, Ashwin Bellur, and Hema A Murthy. 2013. Modal analysis and transcription of strokes of the mridangam using non-negative matrix factorization. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 181–185. IEEE.DOI
  3. [3]Chia-Wei Ao and Hung-yi Lee. 2018. Query-by-example spoken term detection using attention-based multi-hop networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6264–6268. IEEE.arxiv.org/abs/1709.00354
  4. [5]Adnan El Assadi, Isaac Chung, Chenghao Xiao, et al. 2026. Maeb: Massive audio embedding benchmark. arXiv preprint arXiv:2602.16008.
  5. [6]Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338–15347.arxiv.org/abs/2303.15247
  6. [7]Pavlo Bazilinskyy, Arne van der Aa, Michael Schoustra, John Spruit, Laurens Staats, Klaas Jan van der Vlist, and Joost de Winter. 2018. An auditory dataset of passing vehicles recorded with a smartphone. In 12th International Symposium on Tools and Methods of Competitive Engineering (TMCE 2018), pages 417–422.DOI
  7. [8]Felix Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben, and Björn Schuller. 2023. Speech-based age and gender prediction with transformers. In Speech Communication; 15th ITG Conference, pages 46–50. VDE.
  8. [9]Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean-nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359.DOI
  9. [10]Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. Msp-improv: An acted corpus of dyadic interactions to study emotion perception. IEEE transactions on affective computing, 8(1):67–80.DOI
  10. [11]Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390.pubmed.ncbi.nlm.nih.gov/25653738
  11. [12]Ciprian Chelba, Timothy J Hazen, and Murat Saraclar. 2008. Retrieval and browsing of spoken content. IEEE Signal Processing Magazine, 25(3):39–49.DOI
  12. [13]Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, and Zhicheng Dou. 2026. e5-omni: Explicit cross-modal alignment for omni-modal embeddings. In Findings of the Association for Computational Linguistics: ACL 2026, pages 19430–19443.
  13. [14]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518.DOI
  14. [15]Ondřej Cífka, Hendrik Schreiber, Luke Miner, and Fabian-Robert Stöter. 2024. Lyrics transcription for humans: A readability-aware benchmark. In Proceedings of the 25th International Society for Music Information Retrieval Conference, pages 737–744. ISMIR.
  15. [16]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.arxiv.org/abs/2507.06261
  16. [17]Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. 2020. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. In Interspeech 2020, pages 3830–3834.DOI
  17. [18]Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.arxiv.org/abs/2206.04769
  18. [19]John S Garofolo, Cedric GP Auzanne, Ellen M Voorhees, et al. 2000. The trec spoken document retrieval track: A success story. NIST SPECIAL PUBLICATION SP, 500(246):107–130.
  19. [20]Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michal Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulić. 2021. Multilingual and cross-lingual intent detection from spoken data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7468–7475.
  20. [21]Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. 2026. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Advances in Neural Information Processing Systems, 38:41819–41886.arxiv.org/abs/2507.08128
  21. [22]Yuan Gong, Jin Yu, and James Glass. 2022. Vocal sound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE.
  22. [23]Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 976–980. IEEE.
  23. [24]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  24. [25]Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, et al. 2026. Qwen3-tts technical report. arXiv preprint arXiv:2601.15621.arxiv.org/abs/2601.15621
  25. [26]Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580.arxiv.org/abs/2407.12580
  26. [27]Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. 2025. VLM2Vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.05160
  27. [28]Holger Klinck, Juan Sebastián Cañas, Maggie Demkin, Sohier Dane, Stefan Kahl, and Tom Denton. 2025. Birdclef+ 2025. https://kaggle.com/competitions/birdclef-2025. Kaggle.
  28. [29]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.DOI
  29. [30]Martha Larson and Gareth JF Jones. 2012. Spoken content retrieval: A survey of techniques and technologies. Foundations and Trends® in Information Retrieval, 5(4–5):235–422.
  30. [33]Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
  31. [34]Lin-shan Lee, James Glass, Hung-yi Lee, and Chunan Chan. 2015. Spoken content retrieval—beyond cascading speech recognition with text retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(9):1389–1420.DOI
  32. [35]Chen-An Li and Hung-yi Lee. 2026. Inspire: A benchmark for instruction-aware speech retrieval. arXiv preprint arXiv:2608.16203.arxiv.org/abs/2608.16203
  33. [36]Mingxin Li, Yanzhao Zhang, Dingkun Long, et al. 2026a. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720.
  34. [37]Xiquan Li, Xuenan Xu, Ziyang Ma, Wenxi Chen, Haolin He, Qiuqiang Kong, and Xie Chen. 2026b. Finelap: Taming heterogeneous supervision for fine-grained language-audio pretraining. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10393–10408.
  35. [38]Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  36. [39]Chyi-Jiunn Lin, Guan-Ting Lin, Yung-Sung Chuang, Wei-Lun Wu, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, and Lin-Shan Lee. 2024a. Speechdpr: End-to-end spoken passage retrieval for open-domain spoken question answering. In 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12476–12480. IEEE.
  37. [40]Guan-Ting Lin, Yung-Sung Chuang, Ho-Lam Chung, Shu-wen Yang, Hsuan-Jui Chen, Shyuan Annie Dong, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, and Lin Shan Lee. 2022. DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering. In Interspeech 2022, pages 5165–5169.
  38. [41]Guan-Ting Lin, Wei Ping Huang, and Hung-yi Lee. 2024b. Continual test-time adaptation for end-to-end speech recognition on noisy speech. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20003–20015.arxiv.org/abs/2406.11064
  39. [42]Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. 2021. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2125–2134.
  40. [43]Fengjie Lu, Chenang Jiang, Jiarui Hai, Helin Wang, and Aaron Yee. 2026a. Alm2vec: Learning audio embeddings for universal audio retrieval with large audio-language models. arXiv preprint arXiv:2606.30682.
  41. [44]Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Yang, et al. 2026b. Desta2. 5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment. IEEE Transactions on Audio, Speech and Language Processing.
  42. [45]Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15747–15760.
  43. [46]Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3339–3354.
  44. [47]Rui Meng, Ziyan Jiang, Ye Liu, et al. 2025. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents. arXiv preprint arXiv:2507.04590.
  45. [48]Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi, Srikanth Ronanki, Kyu Han, et al. 2025. Speech retrieval-augmented generation without automatic speech recognition. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.arxiv.org/abs/2412.16500
  46. [49]Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Interspeech 2017, pages 2616–2620.DOI
  47. [50]Tri Nguyen, Mir Rosenberg, Xia Song, ianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268.
  48. [51]Tu Anh Nguyen, Wei-Ning Hsu, Antony D’Aviro, Bowen Shi, Itai Gat, Maryam Fazel-Zarandi, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emmanuel Dupoux. 2023. Expresso: A benchmark and analysis of discrete expressive resynthesis. In Interspeech 2023, pages 4823–4827.
  49. [52]Kari Ali Nory, Xiaosong Yang, and Jian Jun Zhang. 2023. Emms/imz/corpus: An emotive single-speaker dataset for narrative storytelling in games, television and graphic novels. arXiv preprint arXiv:2305.13137.
  50. [53]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.arxiv.org/abs/1807.03748
  51. [54]OpenAI. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.
  52. [55]Se Park, Chae Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Ro. 2024. Let’s go real talk: Spoken dialogue model for face-to-face conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16334–16348.
  53. [56]Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018.
  54. [57]Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.arxiv.org/abs/1810.02508
  55. [58]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.arxiv.org/abs/2212.04356
  56. [59]Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. 2018. Cnn based query by example spoken term detection. In Interspeech, pages 92–96.DOI
  57. [60]Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. 2020. Neural network based end-to-end query by example spoken term detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1416–1427.arxiv.org/abs/1911.08332
  58. [61]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1908.10084
  59. [62]Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 410–420.
  60. [63]Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. 2023. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305–19314.arxiv.org/abs/2302.03084
  61. [64]Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1041–1044.DOI
  62. [65]Fabian David Schmidt, Ivan Vulić, Goran Glavaš, and David Ifeoluwa Adelani. 2025. Fleurs-SLU: A massively multilingual benchmark for spoken language understanding. In Second Conference on Language Modeling.arxiv.org/abs/2501.06117
  63. [66]Weiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo, Chen Xu, Xiaofeng Zhao, Long Meng, Yunfei Liu, Min Zhang, Hao Yang, et al. 2025. Enhancing speech large language models with prompt-aware mixture of audio encoders. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19316–19331.
  64. [67]Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. 2023. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. Advances in Neural Information Processing Systems, 36:39088–39118.arxiv.org/abs/2305.13040
  65. [68]Ganesh Sinisetty, Pavlo Ruban, Oleksandr Dymov, and Mirco Ravanelli. 2021. Commonlanguage.DOI
  66. [70]Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, and Chao Zhang. 2026. Wave: learning unified & versatile audio-visual embeddings with multimodal llm. In International Conference on Learning Representations, volume 2026, pages 61596–61612.arxiv.org/abs/2509.21990
  67. [71]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
  68. [72]Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. 2014. A study of instrument-wise onset detection in beijing opera percussion ensembles. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 2159–2163. IEEE.

Paper details

vlnc:W_fv5mh2aw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents