WorksComputer Science
What Do Multilingual Self-Supervised Speech Encoders Know About Sentence Meaning?
Abstract
Self-supervised speech models (S3Ms) have been scaled up to thousands of languages and millions of hours of audio, yet little is known about how these models structure semantic knowledge across languages. In this work, we probe sentence-level semantics in massively multilingual S3Ms across 76 languages, using linear maps from frozen speech representations into a frozen text embedding space, evaluated by cross-modal retrieval. Sentence meaning is linearly accessible but highly uneven across languages and resource tiers. Cross-lingual probes can match language-specific probes on languages never seen in probe training, but per-model monolingual decodability does not predict transfer. Within the OmniASR wav2vec family, scaling from 300M–7B disproportionately benefits high-resource languages, and ASR tuning sharply improves semantic decodability while making representations more language-specific. However, semantic content is entangled with acoustics, inflating apparent performance on single-speaker corpora.
Introduction
Self-supervised speech models (S3Ms), trained on unlabeled audio, have become a standard representation layer for speech processing, and recent efforts have scaled them to hundreds or even thousands of languages and over a million hours of audio ([10, 2, 6] (Pratap et al. 2024; Omnilingual ASR Team et al. 2025)). Yet our understanding of what these models encode has not kept pace with this increase in language coverage.
Prior probing studies have established that S3M representations are dominated by acoustic and phonetic structure ([8] (Pasad et al. 2021; Pasad et al. 2024; Ersoy et al. 2025)), but also encode word (Pasad et al., 2021) and sentence ([7, 1] (Pasad et al. 2024)) meaning. However, this work is limited to a small number of
languages and usually monolingual models. Multilingual probing has focused on acoustic and phonetic unit discovery (Ortiz Tandazo et al., 2026; Poli et al., 2026) and downstream tasks such as automatic speech recognition (ASR) and language identification (Li et al., 2023; Shi et al., 2023, 2024). Multilingual S3Ms remain underexplored at the semantic level; it is unclear whether semantic content is present beyond high-resource languages or how it is structured across languages.
In this work, we present a sentence-level semantic probing study of massively multilingual S3Ms, covering 9 self-supervised models (plus several ASR-tuned variants) spanning training objective, scale (300M→7B), and language coverage, across 76 languages. We train lightweight probes (Figure 1): learnable linear maps from frozen speech representations into a frozen sentence embedding space with translation equivalence, keeping the maps linear to measure content already present rather than capacity added by the probe. We evaluate by cross-modal retrieval on n-way parallel corpora, where success requires recovering sentence meaning.
Figure 1. Overview of the probing approach. A linear probe maps multilingual speech representations into a text embedding space, where retrieval measures semantic decodability and zero-shot cross-lingual transfer.
This setup separates two properties: semantic decodability is how much translation-equivalent meaning a linear map from speech to text space can recover. Cross-lingual alignment is whether that meaning is organized in a shared, languageagnostic fashion, allowing for cross-lingual transfer, which we test by varying probe training and test languages.
In summary, our findings are: (i) Sentence meaning is linearly accessible in multilingual S3Ms but is highly uneven across languages and resource tiers. (ii) Genuine cross-lingual semantic transfer exists; OmniASR cross-lingual probes can match language-specific probes (zero-shot). (iii) Monolingual decodability does not predict cross-lingual transfer; XEUS, with rich within-language semantics, has weak cross-lingual transfer. (iv) Retrieval scores across OmniASR models ranging from show that scaling up mostly benefits high-resource languages. (v) ASR supervision strengthens semantic decodability, but in language-specific directions. (vi) Semantics remain entangled with acoustics, inflating apparent decodability on single-speaker corpora.
Related Work
Analysis of Self-Supervised Speech Representations
Learning linguistic information from raw audio has long been a goal of the speech community, motivating efforts such as the Zero Resource Speech Challenge (Dunbar et al., 2022). Such a capability would let models serve communities whose languages are largely unwritten. The foundational work wav2vec 2.0 [3] popularized self-supervised learning from raw audio, but these models struggle with semantic tasks that text models solve easily. Mohamed et al. (2022) note that even the original BERT corpus is 3B+ words (small by current standards) whereas a comparable speech corpus would require on the order of 400K hours of audio; this data gap is one reason text models are often used to inject semantics into speech models (Xu et al., 2023; Maekaku et al., 2024).
Furthermore, S3M representations tend to be more phonetic than semantic (Ersoy et al., 2025), placing phonetically similar words closer than synonyms under cosine distance [8]. Nonetheless, some semantic structure is present: Pasad et al. (2021) find that the middle, most contextualized layers of wav2vec 2.0 best encode word meaning, and Pasad et al. (2024) show that S3Ms carry sentence-level semantic content beyond word identity, correlating with human judgments on a spoken semantic textual similarity task. Cho et al. [7] similarly find, via a spoken sentence ABX task, that wav2vec 2.0, HuBERT, and WavLM distinguish semantically related English sentences from distractors. However, these models lag behind text models on spoken language understanding tasks [1].
This evidence, however, is drawn mostly from monolingual models, while multilingual analyses such as ML-SUPERB (Shi et al., 2023, 2024) evaluate across many languages but target downstream tasks (e.g., ASR and language identification), without exploring cross-lingual semantic structure.
Supervised Speech-Text Alignment
Past work has fine-tuned S3Ms to align their representations with a pre-trained text embedding space, turning them into semantic encoders. SAMU-XLS-R (Khurana et al., 2022) and SENSE models (Mdhaffar et al., 2025) distill a multilingual text encoder into a speech model at the utterance level through student–teacher alignment. SONAR (Duquenne et al., 2023) and Omnilingual SONAR (Omnilingual SONAR Team et al., 2026) similarly train a shared language- and modality-agnostic sentence embedding space by aligning a speech encoder to a frozen text encoder.
These models establish that a shared cross-lingual semantic space is reachable from speech with explicit supervision, but they say little about what is already latent in the frozen self-supervised model. By probing frozen representations directly, we characterize the semantic structure already present in S3Ms that text supervision can exploit.
Probing Framework
Models
We evaluate 9 self-supervised speech models (Table 1) of different architectures and training objectives: the wav2vec 2.0 contrastive family [3], scaled multilingually by MMS (Pratap et al., 2024) and the Omnilingual (OmniASR) family (Omnilingual ASR Team et al., 2025); HuBERT-style masked prediction (Hsu et al., 2021) in mHuBERT-147 [4]; WavLM-style denoising [5] in XEUS [6]; and the combined contrastive–masked prediction objective of w2v-BERT 2.0 [9] (Seamless Communication et al. 2023). The OmniASR-W2V (wav2vec 2.0) encoders cover the 300M–7B parameter range, isolating the effect of model size, and have ASR CTC and LLM-style decoder variants, letting us measure how transcription supervision affects semantic structure. Finally, we include WavLM Base+ (Chen et al., [5]) as a strong English monolingual baseline.
| Model | Params | Obj. | Hours | Langs | ||
| WavLM Base+ | 95M | 12 | 768 | MP+D | 94K | 1 |
| mHuBERT-147 | 95M | 12 | 768 | MP | 90K | 147 |
| XEUS | 577M | 19 | 1024 | MP+D | 1.1M | 4057 |
| w2v-BERT-2.0 | 580M | 24 | 1024 | C+MP | 4.5M | 143+ |
| MMS-1B | 965M | 48 | 1280 | C | 491K | 1406 |
| OmniASR-300M | 317M | 24 | 1024 | C | 4.3M | 1600+ |
| OmniASR-1B | 965M | 48 | 1280 | C | 4.3M | 1600+ |
| OmniASR-3B | 3.0B | 60 | 2048 | C | 4.3M | 1600+ |
| OmniASR-7B | 6.5B | 128 | 2048 | C | 4.3M | 1600+ |
Table 1. Speech encoders probed in this work. = number of transformer layers, = hidden dimension. Objectives: C = contrastive (wav2vec 2.0), MP = masked prediction (HuBERT-style), D = denoising (WavLM-style). Hours and languages refer to pre-training data.
Datasets
We evaluate on two -way parallel corpora differing in domain and speaker composition.
FLEURS. FLEURS (Conneau et al., [11]) provides multi-speaker read-speech recordings of the FLORES machine translation benchmark (Goyal et al., 2022), whose English source sentences are drawn from Wikipedia. The material covers science, geography, and current events, and is dense in named entities. The dataset contains 1509, 150, and 350 sentences for train, validation, and test, respectively, averaging 2.3 recordings per sentence with different speakers. We use a subset of 40 languages covering multiple language families, averaging 2591, 334, and 726 utterances per language for training, validation, and testing. The sentences and speakers in each split are disjoint, so FLEURS scores measure cross-speaker generalization.
Bible. For the long tail of low-resource languages, scripture and other religious recordings are among the only available speech data, constituting a substantial fraction of the pre-training data in massively multilingual S3Ms, extending to 1,300 languages in MMS (Pratap et al., 2024) and 4,000+ languages in XEUS (Chen et al., [6]). Bible data is therefore more in-domain than FLEURS for many low-resource languages.
We combine verse-segmented New Testament (NT) recordings from OpenBibleTTS (Guzmán et al., 2026), Snow Mountain (Raju et al., 2023), and the English WEB version, covering 48 languages with a single speaker per language, 12
of which are shared with the FLEURS subset.1 This NT corpus primarily covers the mid- to low-resource range, including severely resource-constrained Himalayan and Omotic languages. We use 4,000, 1,000, and 1,400 verses for training, validation, and testing; six NT books are reserved for testing,2 so train and test are disjoint at the book level. Each language uses the same verse set, but missing or invalid verses (s or near-silent) are replaced with other verses (affecting 3.4% total), keeping the corpus close to -way parallel.
Probe Structure
We train probes that map the frame-level representations of a frozen S3M to a single vector in the embedding space of a frozen multilingual text encoder. We embed the English text for each utterance to yield a single semantic target space shared by all probes. We choose SONAR (Duquenne et al., 2023) as our text encoder, because it is trained to map semantically similar inputs across languages to nearby vectors in a shared space, mitigating English bias compared to a monolingual text encoder.
Each probe maps a speech representation into the text encoder space; retrieval success then requires recovering sentence meaning, since the target contains no acoustic, speaker, or lexical overlap with the input. English is the exception; an English probe has a possible direct lexical mapping shortcut without requiring a translation step. Hence, we treat English as an upper-bound reference rather than evidence of semantic abstraction.
Given an utterance, the speech encoder produces frame-level hidden states at layer . Probes read from every layer, including the initial convolutional output (layer 0). The text encoder maps the corresponding transcript to embedding . The probe consists of a pooling step, reducing frames to a single vector, followed by a learned linear projection into the text space. Pooling computes a weighted sum of the frames,
where is the weight for frame . Our primary experiments use mean pooling (), but as frames may carry unequal semantic signal, we also try lightweight attention pooling which computes the weights with a single learnable query vector:
where softmax normalizes the scores over the frames. This adds only parameters ( and the LayerNorm gain and bias), and we initialize to zero so that training starts from exact mean pooling, deviating from uniform weighting where it provides benefit. The pooled vector is mapped into the text space by a linear layer:
where and . For a model with (varies by model) and for SONAR, the probe has roughly 1.05M trainable parameters, the vast majority in . Because a map of this size potentially has the capacity to memorize acoustic-to-word associations instead of decoding semantic structure, we also train low-rank probes in which is factored as with , , for .
Probe Training
The probe is trained to minimize speech-to-text contrastive loss, matching the evaluation direction. Let be the embeddings of all training transcripts, and let be the scaled cosine similarity between speech embedding and transcript embedding , with fixed . For each utterance in a mini-batch we contrast its embedding against the pool of training transcripts:
where is the set of transcripts matching utterance . Parallel utterances in different languages share a target (for multilingual probes), and treats all instances of the matching transcript as positives. We use the AdamW optimizer, a learning rate of ( for attention pooling), and a batch size of 512. Weight decay has little effect on results, so we leave it at 0. Each probe is evaluated on the validation set every 15 steps, with early-stopping patience of 10 evaluations. Our final probe sweep used ~150 NVIDIA H200 GPU-hours.
Probe Evaluation
We embed all test-set English sentences and, for each test utterance, retrieve the nearest text embedding by cosine similarity. We report top-1 accuracy per-language and averaged over languages.3
Probe Regimes. We experiment with three main probe regimes that differ only in which languages supply the probe’s training data. Monolingual: one probe per language, trained and evaluated on that language, measuring within-language linear decodability. Multilingual: a single shared probe trained jointly on all languages, measuring whether one map can serve all languages at once. Leave-one-out (LOO): a shared probe trained on all languages except one and evaluated on the held-out language. This tests for language-agnostic semantic structure, since the probe has never seen the target language. We also measure cross-lingual transfer with all pairwise combinations of single train–test languages, but we leave these results to Appendix § 9.
Layer Selection. We first sweep all layers with mean-pooled monolingual probes and select, per model, the layer that maximizes validation set retrieval on each corpus. All subsequent analyses use each model’s selected layer, so models are compared at their respective best operating points.
Mean Centering. We experiment with per-language centering: for each language, we subtract the mean speech embedding computed over that language’s training set, before training and testing. The change in retrieval accuracy indicates if some of the probe’s performance depends on language- and speaker-specific directions.
Named Entity Control. One risk of our retrieval setup is that it reflects not sentence meaning but named entity shortcuts, such as names and places that sound alike across languages. To test this risk, we prompt gemma-4-31b-it (Gemma Team et al., 2026) to produce modified versions of every sentence: (i) masked, in which each entity is replaced by a generic placeholder of the same type (e.g., the place, the organization); and (ii) names-only, in which all non-entity content is stripped and only the entities remain. Masking tests whether entities are necessary (whether retrieval survives when names are removed). Names-only tests whether they are sufficient: if entities alone retrieve near chance, the signal is not carried by names.
Results
Sentence meaning is linearly accessible but highly uneven across languages. While supervised speech and text encoders (such as SONAR) easily saturate the retrieval task on FLEURS (known as xsim) ((Duquenne et al., 2023)), this task remains difficult for self-supervised speech encoders. The strongest models in our tests (w2v-BERT-2.0 and OmniASR-W2V-7B) achieve > 75% monolingual accuracy (Figure 2) purely from mean-pooled self-supervised features for several high-resource languages, but accuracy falls steeply on the mid- to low-resource tiers. WavLM achieves 13% accuracy on non-English languages despite pre-training only on English, suggesting that a linear map can exploit acoustic-lexical patterns.
Figure 2. Monolingual, multilingual, and leave-one-language-out (LOO) top-1 accuracy of mean-pooled probes of self-supervised models on FLEURS (top) and Bible (bottom). English is excluded from mean calculations.
MMS-1B unexpectedly outperforms the larger OmniASR-3B on FLEURS (30% vs. 24% multi, winning on 29/39 non-English languages), despite sharing the wav2vec 2.0 architecture and training on an order of magnitude fewer hours of audio. We speculate that OmniASR allocates more of its training mixture across the long tail of languages, consistent with its weaker English scores.
Bible monolingual scores are consistently higher (e.g., XEUS 60% vs. 33% on FLEURS), with even the monolingual WavLM baseline surprisingly averaging 48%, indicating that single-speaker recordings allow probes to more easily exploit speaker-and channel-specific cues, inflating retrieval scores. We return to this below.
Cross-lingual semantic structure is present. On FLEURS, a shared, multilingual probe frequently matches or beats per-language probes. For example, w2v-BERT-2.0’s multilingual probe outperforms monolingual probes on 36/39 non-English languages (47% multi vs 36% mono, Figure 2), showing that one language-agnostic map can serve many languages. Most models perform substantially worse in the LOO regime, with the exception of the OmniASR models (above the line in Figure 3). Remarkably, OmniASR-W2V-7B held-out wins on 24/39 languages vs. in-language (33% LOO vs. 30% mono, Figure 2), indicating broad multilingual gains that are not limited to a few languages. LOO probes see much more training data than monolingual probes, so this shows that abundant supervision from other languages can compensate sate for scarce target-language data. Monolingual decodability does not predict cross-lingual transfer. XEUS has among the richest monolingual representations, on par with the much larger MMS-1B and OmniASR-W2V-3B models, yet transfers poorly (33% mono vs. 15% LOO on FLEURS). This pattern aligns with prior work showing that the contrastive objective produces better cross-lingual features for ASR compared to masked prediction (Li et al., 2023).
Figure 3. Monolingual against leave-one-out (LOO) retrieval accuracy for 40 FLEURS languages, showing the relationship between monolingual scores and cross-lingual transfer. OmniASR models show high transfer with many languages above the diagonal.
Scale isn’t everything. Semantic quality is not a simple function of model scale: w2v-BERT-2.0, at 580M parameters, has the strongest over-all FLEURS retrieval (47% multi, Figure 2) of any self-supervised model, surpassing even OmniASR-W2V-7B, which is many times the size. The most salient difference is its joint contrastive and masked-prediction objective, but these models differ in many dimensions, including pretraining data, so further work is needed to isolate the factor responsible for its stronger semantic decodability.
Semantic content is temporally localized. Attention pooling improves top-1 over mean pooling for every self-supervised model (Figure 4). The gains are substantial: MMS rises from 26% to 44% and w2v-BERT-2.0 from 36% to 52%, while ASR models are marginally affected, sometimes negatively. This indicates that self-supervised encoders do not spread semantic content evenly across frames, and mean pooling dilutes the semantic signal, whereas ASR tuning makes it more uniformly recoverable. However, since sweeping all probe regimes with attention pooling was computationally prohibitive, our primary results use mean-pooled probes.
Figure 4. Monolingual retrieval accuracy on FLEURS, comparing mean- and attention-pooled probes. The higher scores of attention pooling on self-supervised models indicate temporally localized semantic content. OmniASR-300M ASR models excluded for brevity.
OmniASR-W2V scaling amplifies semantic disparities between resource tiers. Within the OmniASR-W2V family, scaling up from 300M→7B produces sharply unequal gains across resource tiers (Figure 5). Monolingual probes rise 14% to 68% for the high-resource tier, compared with 7% to 22% for mid-resource and 6% to 12% for low-resource. Increased scale greatly improves semantic representations where speech data is abundant, while the low end remains relatively flat, widening the gap between resource tiers.4
Figure 5. OmniASR-W2V top-1 retrieval on FLEURS at different model scales, broken down by language resource tier. Scaling the model up benefits high-resource languages more than low-resource languages.
OmniASR-W2V scaling adds depth primarily after the semantic peak. Figure 6 shows sharply different layerwise profiles across different model types. The wav2vec 2.0 contrastive models exhibit autoencoder-like behavior reported in prior work ((Pasad et al., 2023)): retrieval accuracy rises to a peak in early layers and then decays, as the final layers reconstruct input-like features. OmniASR also reveals a scaling pattern: as total depth grows 24→128 layers, the semantic peak only shifts 14→28, barely growing with scale. Most extra depth is spent after the peak and is not relevant for this task. The masked-prediction (mHuBERT-147) and denoising (XEUS) encoders instead concentrate semantic content more deeply in the network, consistent with prior work attributing changes in layerwise organization to iterative refinement and pre-training objective ((Huo and Dunbar, 2025; de Heer Kloots et al., 2026)).
ASR supervision strengthens semantic decodability, but in language-specific directions. Fine-tuning for transcription reshapes the self-supervised layerwise profile, maintaining semantics in deeper layers and pushing the peak deeper (Figure 6). It also sharply raises retrieval scores: OmniASR-7B rises from 30% (W2V) to 76% (CTC) and 82% (LLM) monolingual accuracy on FLEURS, with similar gains at every scale for both CTC and LLM variants (Figure 4). However, relative to their self-supervised starting points, the ASR-tuned models show larger mono–LOO gaps on FLEURS (e.g., OmniASR-7B LLM: 82% mono vs. 57% LOO) and larger centering deltas on both datasets (Figure 7). Removing this offset recovers most held-out transfer, so the ASR boost sits largely in language-specific directions. The OmniASR W2V self-supervised counterparts, meanwhile, are nearly centering invariant.
Figure 6. Layerwise monolingual top-1 accuracy on FLEURS (left) and Bible (right), comparing SSL (top) encoders and ASR-tuned models (bottom). Per-model peak layer indices are labeled. Masked-prediction models (w2v-BERT-2.0 and XEUS) and ASR models retain semantics in later layers vs. contrastive models (MMS, OmniASR).
Figure 7. Mean (non-English) top-1 scores before and after mean centering. Dashed line marks the WavLM Base+ baseline. On FLEURS, centering closes the LOO vs. mono gap for w2v-BERT-2.0 and MMS-1B, and partially for ASR-tuned models, but has little effect on other models. On Bible, mean centering can partially close the gap but LOO still lags far behind mono results.
Semantic information is entangled with acoustics. Bible monolingual probes exceed the WavLM baseline on several models, but no self-supervised model is able to exceed the baseline under the multilingual and LOO regimes. This pattern suggests that monolingual Bible probes benefit from speaker- and channel-specific acoustic regularities that do not transfer across languages.5 Consistent with greater reliance on acoustic information, Bible also retains more accuracy in later, more input-like layers of contrastive models (Figure 6).
Mean centering yields small improvements, especially for LOO probes, bringing LOO probes closer to multilingual probes in performance (Figure 7). However, it is not nearly enough to close the monolingual–LOO gap on the Bible corpus. Thus, Bible’s distinctive monolingual advantage cannot be explained by a simple speaker- or language-specific offset. These results suggest that semantic and acoustic information are not cleanly separated, allowing single-speaker corpora to inflate apparent semantic decodability.
Semantic decodability survives at low-rank. We re-run the mean-pooled probes under rank restriction (Figure 8). Below rank 64, retrieval is weakened substantially, but rank 64 preserves most performance, and remains well above full-rank WavLM on FLEURS. Notably, Bible probes benefit more from higher ranks, possibly by using extra capacity to exploit many sparse, speaker-specific directions on single-speaker training data.
Figure 8. Mean monolingual top-1 as a function of projection rank for FLEURS (top) and Bible (bottom).
Semantic decodability is not merely an entity–matching shortcut. Masking named entities affects models unevenly (Figure 9). Some reduction is expected, as masking removes legitimate information from the candidate sentences and creates a mismatch between the speech and target text. Therefore, we focus on the margin over the masked WavLM baseline rather than the size of the drop alone. For weaker self-supervised encoders, masked LOO accuracy falls to 7–14%, near the masked WavLM baseline of 6%, indicating that entities contribute materially when the semantic
Figure 9. Leave-one-out (LOO) top-1 average (non-English) on FLEURS rows containing tagged entities. The strongest models retain substantial accuracy on Masked sentences without entity information. Names-only scores sit below Gold scores, so entities alone do not explain the strongest performance.
signal is weak. In contrast, w2v-BERT-2.0 and OmniASR-W2V-7B retain 33% and 31% accuracy after masking, respectively, well above the baseline. Names-only retrieval is well below gold retrieval, so entities alone cannot explain the strongest results. Thus, entities contribute information, but differences between the strongest and weakest encoders cannot be attributed to entity matching alone.
Conclusion
While far weaker than supervised encoders, S3Ms have substantial decodable sentence-level semantic content for mid- to high-resource languages and, in some models, surprisingly cross-lingual representations. Yet cross-lingual capability is not predicted by monolingual performance: XEUS encodes rich within-language semantics but transfers poorly, while OmniASR models transfer well despite being semantically weak for their size. Factors beyond sheer scale, such as training objective, impact semantic decodability, as the 580M-parameter w2v-BERT-2.0 surpasses the 7B OmniASR encoder. Our results indicate that supervised speech–text alignment can leverage semantic structure already present. However, we find semantic content entangled with acoustics that inflate apparent decodability on single-speaker corpora. We leave open how much of the residual gap to supervised systems reflects missing semantic content or content that is present but not linearly accessible.
Limitations
Our analysis measures what is linearly decodable from frozen representations, not what a nonlinear trained system could extract, so our results could understate the semantic content actually present. Simultaneously, due to acoustic associations, our results could overstate the semantic content; low-rank and named-entity controls show that the effect does not require a full-capacity projection or entity-only matching, though other lexical cues may still contribute. Retrieval against a single frozen text encoder (SONAR) also means our results are conditioned on that encoder’s own capabilities and biases. Our two corpora, while complementary, are narrow in domain and register (Wikipedia-style news/geography and New Testament), and FLEURS lacks speaker IDs, preventing a clean separation of speaker confounds within that corpus. Language coverage, while broad relative to prior work, is still a small fraction of the full set of languages used to train these massively multilingual speech models.
Acknowledgments
This work was supported by the ETEN Innovation Lab. We are grateful to Joel Mathew for his guidance and support throughout this project, and to the members of the OSU SLaTe Lab for helpful discussions.
Language Resource Tiers
| Code | Language | Code | Language |
| FLEURS | |||
| High resource | |||
| nl_nl | Dutch (Netherlands) | en_us | English (United States) |
| fr_fr | French (France) | de_de | German (Germany) |
| it_it | Italian (Italy) | pt_br | Portuguese (Brazil) |
| es_419 | Spanish (Latin America) | cmn_hans_cn | Mandarin Chinese (Simpl., China) |
| ko_kr | Korean (South Korea) | ru_ru | Russian (Russia) |
| Mid resource | |||
| pl_pl | Polish (Poland) | ar_eg | Arabic (Egypt) |
| bn_in | Bengali (India) | el_gr | Greek (Greece) |
| ha_ng | Hausa (Nigeria) | ja_jp | Japanese (Japan) |
| fa_ir | Persian (Iran) | so_so | Somali (Somalia) |
| sw_ke | Swahili (Kenya) | th_th | Thai (Thailand) |
| tr_tr | Turkish (Turkey) | uk_ua | Ukrainian (Ukraine) |
| ta_in | Tamil (India) | yo_ng | Yoruba (Nigeria) |
| my_mm | Burmese (Myanmar) | hi_in | Hindi (India) |
| id_id | Indonesian (Indonesia) | ig_ng | Igbo (Nigeria) |
| te_in | Telugu (India) | ur_pk | Urdu (Pakistan) |
| vi_vn | Vietnamese (Vietnam) | zu_za | Zulu (South Africa) |
| Low resource | |||
| he_il | Hebrew (Israel) | fil_ph | Filipino (Philippines) |
| am_et | Amharic (Ethiopia) | yue_hant_hk | Cantonese (Trad., Hong Kong) |
| ff_sn | Fula (Senegal) | km_kh | Khmer (Cambodia) |
| lo_la | Lao (Laos) | mi_nz | Māori (New Zealand) |
| Bible | |||
| English WEB | |||
| High resource | |||
| eng | English | ||
| OpenBibleTTS | |||
| Mid resource | |||
| arb | Modern Standard Arabic | ben | Bengali |
| hau | Hausa | kan | Kannada |
| swh | Swahili | tam | Tamil |
| tur | Turkish | yor | Yoruba |
| luo | Dholuo | guj | Gujarati |
| hin | Hindi | ibo | Igbo |
| kik | Kikuyu | mal | Malayalam |
| pan | Punjabi | sna | Shona |
| tel | Telugu | urd | Urdu |
| vie | Vietnamese | ||
| Low resource | |||
| asm | Assamese | ckb | Central Kurdish |
| hat | Haitian Creole | lin | Lingala |
| lug | Luganda | mar | Marathi |
| hne | Chhattisgarhi | nya | Chichewa |
| dwr | Dawro | ewe | Ewe |
| gmv | Gamo | gof | Gofa |
| hil | Hiligaynon | nde | North Ndebele |
| nep | Nepali | orm | Oromo |
| twi_akuapem | Twi (Akuapem) | twi_asante | Twi (Asante) |
| Snow Mountain | |||
| Low resource | |||
| bhadrawahi | Bhadrawahi | bilaspuri | Bilaspuri |
| dogri | Dogri | gaddi | Gaddi |
| haryanvi | Haryanvi | kangri | Kangri |
| kulvi | Kulvi | kulvi_outer_seraji | Kulvi Outer Seraji |
| mandeali | Mandeali | pahari_mahasui | Pahari Mahasui |
Table 2. Probe languages grouped by dataset and resource tier. Tiers use the per-language hours in ALLASR published with Omnilingual ASR: high (h), mid (), and low (h or missing). These tiers serve as a rough proxy for broad speech resource availability. Language codes are listed as provided by respective datasets. Languages occurring in both corpora are listed in both sections, giving 88 language entries and 76 distinct languages. 12 languages (en, hi, ur, bn, ta, te, tr, vi, sw, ha, yo, ig) are shared between the two corpora.
Probe Seed Variability
| Probe type | Probe Count | Top-1 | Top-1 range |
| w2v-BERT-2.0 (layer 13) | |||
| Monolingual | 30 | [44.24, 45.11] | |
| Multilingual | 5 | [50.75, 51.74] | |
| Multilingual LOO | 30 | [21.12, 22.23] | |
| OmniASR-W2V-7B (layer 28) | |||
| Monolingual | 30 | [38.93, 39.19] | |
| Multilingual | 5 | [41.30, 42.11] | |
| Multilingual LOO | 30 | [27.08, 27.35] |
Table 3. Probe seed variability on FLEURS. Results are macro-averaged over six languages (en_us, de_de, hi_in, cmn_hans_cn, sw_ke, and yo_ng) across five seeds. Probe count is the number of individually trained probes: six per seed for monolingual and leave-one-out (LOO) and one per seed for multilingual. The Top-1 column shows the mean sample standard deviation across the five seed-level language macro-averages; the range gives the observed minimum and maximum top-1. Probe layers and all other experimental settings are fixed across seeds. Results are stable across seeds, so the conclusions of our main experiments do not depend on particular probe initialization.
Per-language Probe Results
Figure 10. Monolingual probe retrieval accuracy on the FLEURS test set for 40 languages. The red line connects the points for w2v-BERT 2.0, the self-supervised model with best average monolingual probe scores on this test set. ASR-tuned models (OmniASR CTC and LLM variants) have much higher scores due to text supervision.
Figure 11. Monolingual probe retrieval accuracy on the Bible test set for 48 languages. The green line connects the points for OmniASR-W2V-7B, the self-supervised model with best average monolingual probe scores on this test set. ASR-tuned models (OmniASR CTC and LLM variants) have much higher scores due to text supervision.
Pairwise Probe Evaluation
Figure 12. Pairwise cross-lingual transfer using OmniASR-W2V-7B on FLEURS, using mean-pooled probes trained on one language and tested on another. Pairwise transfer is calculated as the top-1 for a given train-test language pair divided by the monolingual probe’s top-1 on that language, measuring the ratio of monolingual performance that is maintained by zero-shot cross-lingual transfer. Pairwise transfer often benefits related language sources more than unrelated ones (e.g., Hindi–Urdu and Lao–Thai have high transfer, while English–Hindi has poor transfer). However, high-resource languages benefit other languages more substantially in general compared to low-resource languages.
Named-Entity Controls
One risk of our retrieval setup is that it reflects not sentence meaning but named entity shortcuts, such as names and places that sound alike across languages. Our Bible corpus additionally has the risk of common usage of names that are transliterated rather than translated, causing a high degree of similarity across languages.
We test if this type of risk applies in the speech domain by generating three controlled variants of each English target text (only the first two are used in our main analysis): (i) masked, in which each entity is replaced by a generic placeholder of the same type (e.g., the place, the organization, the man); (ii) names-only, in which all non-entity content is stripped and only the entities remain; and (iii) swapped, in which each entity is replaced by a different, type-consistent entity, yielding a similar sentence with different referents. Masking tests whether entities are necessary (whether retrieval survives when names are removed). Names-only tests whether they are sufficient: if entities alone retrieve near chance, the signal is not carried by names. The swap test measures entity discrimination: whether the speech embedding prefers the gold text over the entity-swapped version, reported as a paired win rate against the chance value of 0.5.
Prompt Construction
The named-entity controls are generated by prompting gemma-4-31B-it ((Gemma Team et al., 2026)) per English target text. The prompt was drafted initially by Claude Opus 4.8 and manually adjusted. The prompting procedure uses a JSON object {"text": "..."} as input and a JSON object as output with the tagged entities and the three derived sentences. We use greedy decoding, with retries (temperature 0.7) for rows whose first completion fails JSON parsing or consistency checks (e.g., the names-masked text still containing a tagged surface form, or the name-swapped text being identical to the input). Because the two corpora differ in domain and in how proper names behave, we use separate prompts for each: the FLEURS prompt targets Wikipedia-style topics, while the Bible prompt is specialized to New Testament verses. Table 4 shows the results of this pipeline with a FLEURS training row with all three generated variants. Table 5 contains the complete retrieval results for both FLEURS and Bible, for both self-supervised and ASR-tuned models.
| Variant | Text |
| Original | The pilot was identified as Squadron Leader Dilokrit Pattavee. |
| Masked | The pilot was identified as the squadron leader. |
| Names-only | Dilokrit Pattavee |
| Swapped | The pilot was identified as Squadron Leader Marcus Thorne. |
Table 4. Example FLEURS training row (ID 4) and its three generated named-entity control variants.
| Model | Mono. G | Mono. M | Mono. N | Mono. S | Multi. G | Multi. M | Multi. N | Multi. S | LOO G | LOO M | LOO N | LOO S |
| FLEURS | ||||||||||||
| MMS-1B | 33 | 25 | 9 | 68 | 39 | 26 | 12 | 79 | 21 | 14 | 7 | 70 |
| w2v-BERT-2.0 | 42 | 35 | 9 | 66 | 55 | 46 | 13 | 76 | 40 | 33 | 10 | 71 |
| mHuBERT-147 | 23 | 18 | 7 | 68 | 20 | 11 | 8 | 75 | 12 | 7 | 6 | 71 |
| XEUS | 41 | 30 | 12 | 75 | 36 | 23 | 14 | 82 | 21 | 12 | 9 | 74 |
| WavLM Base+ | 18 | 13 | 6 | 67 | 17 | 8 | 9 | 77 | 13 | 6 | 7 | 75 |
| OmniASR-W2V-300M | 12 | 9 | 4 | 62 | 12 | 7 | 4 | 70 | 12 | 7 | 5 | 71 |
| OmniASR-W2V-1B | 19 | 14 | 6 | 65 | 22 | 15 | 8 | 73 | 21 | 14 | 9 | 74 |
| OmniASR-W2V-3B | 25 | 19 | 8 | 67 | 31 | 21 | 12 | 77 | 27 | 18 | 11 | 74 |
| OmniASR-W2V-7B | 36 | 29 | 11 | 69 | 44 | 33 | 17 | 78 | 41 | 31 | 16 | 78 |
| OmniASR-CTC-300M | 38 | 28 | 10 | 72 | 15 | 10 | 5 | 71 | 8 | 5 | 3 | 65 |
| OmniASR-LLM-300M | 40 | 29 | 11 | 74 | 33 | 18 | 12 | 81 | 20 | 11 | 8 | 75 |
| OmniASR-CTC-1B | 66 | 53 | 20 | 79 | 64 | 48 | 22 | 84 | 39 | 28 | 13 | 74 |
| OmniASR-LLM-1B | 68 | 55 | 20 | 80 | 66 | 49 | 22 | 85 | 43 | 31 | 16 | 77 |
| OmniASR-CTC-3B | 67 | 56 | 20 | 79 | 68 | 52 | 24 | 85 | 45 | 33 | 16 | 77 |
| OmniASR-LLM-3B | 70 | 51 | 26 | 86 | 64 | 39 | 31 | 88 | 39 | 20 | 20 | 83 |
| OmniASR-CTC-7B | 82 | 72 | 26 | 81 | 84 | 71 | 30 | 87 | 64 | 52 | 22 | 79 |
| OmniASR-LLM-7B | 87 | 75 | 34 | 88 | 88 | 73 | 37 | 91 | 67 | 51 | 26 | 83 |
| Bible | ||||||||||||
| MMS-1B | 61 | 48 | 5 | 81 | 24 | 17 | 3 | 79 | 11 | 7 | 2 | 71 |
| w2v-BERT-2.0 | 54 | 45 | 3 | 77 | 25 | 18 | 2 | 77 | 15 | 11 | 2 | 72 |
| mHuBERT-147 | 51 | 41 | 3 | 77 | 14 | 9 | 1 | 79 | 6 | 4 | 1 | 72 |
| XEUS | 63 | 51 | 5 | 82 | 19 | 12 | 3 | 81 | 10 | 6 | 2 | 77 |
| WavLM Base+ | 51 | 41 | 3 | 77 | 13 | 8 | 2 | 80 | 8 | 5 | 1 | 75 |
| OmniASR-W2V-300M | 49 | 39 | 3 | 77 | 15 | 9 | 2 | 81 | 8 | 5 | 1 | 75 |
| OmniASR-W2V-1B | 50 | 40 | 3 | 78 | 17 | 10 | 2 | 80 | 8 | 5 | 2 | 76 |
| OmniASR-W2V-3B | 60 | 47 | 5 | 81 | 25 | 15 | 5 | 84 | 16 | 9 | 4 | 80 |
| OmniASR-W2V-7B | 64 | 50 | 6 | 81 | 28 | 17 | 5 | 84 | 16 | 8 | 4 | 80 |
| OmniASR-CTC-300M | 69 | 56 | 6 | 83 | 26 | 17 | 3 | 82 | 11 | 7 | 2 | 71 |
| OmniASR-LLM-300M | 66 | 53 | 5 | 83 | 22 | 14 | 3 | 80 | 12 | 8 | 2 | 75 |
| OmniASR-CTC-1B | 82 | 67 | 10 | 88 | 49 | 34 | 7 | 87 | 25 | 16 | 4 | 77 |
| OmniASR-LLM-1B | 83 | 68 | 10 | 88 | 51 | 35 | 7 | 87 | 25 | 16 | 4 | 78 |
| OmniASR-CTC-3B | 83 | 70 | 10 | 87 | 50 | 36 | 6 | 85 | 27 | 18 | 5 | 77 |
| OmniASR-LLM-3B | 85 | 71 | 11 | 88 | 54 | 37 | 8 | 86 | 27 | 17 | 5 | 77 |
| OmniASR-CTC-7B | 87 | 74 | 11 | 87 | 73 | 56 | 9 | 87 | 45 | 33 | 6 | 79 |
| OmniASR-LLM-7B | 90 | 77 | 13 | 89 | 73 | 56 | 10 | 87 | 45 | 32 | 7 | 80 |
Table 5. Named-entity control results (%) averaged over the non-English languages in FLEURS and Bible rows containing tagged entities. G/M/N denote gold, masked, and names-only retrieval targets. S reports the paired swap win rate of gold sentences over variants with replaced entities (chance = 50%).
FLEURS Prompt
# Task
You are generating controlled text variants for a speech–text retrieval probing experiment. The input is an English sentence that has a spoken translation in another language. We need to measure how much retrieval relies on proper names (which often sound similar across languages). You will (1) identify proper-name entities and (2) produce three derived texts: a name-swapped twin, a names-masked version, and a names-only string.
Work carefully and deterministically. Do not add commentary.
## Input
A single JSON object: {"text": "..."}
## Output
A single JSON object, nothing else:
```json
{
"name_bearing": true|false,
"entities": [{"source": "...", "type": "PERSON|PLACE|ORG|GROUP|WORK|EVENT", "gender": "m|f|null"}],
"replacements": [{"original": "...", "replacement": "...", "type": "..."}],
"name_swapped": "..." | null,
"names_masked": "..." | null,
"names_only": "..." | null
}
Entity tagging rules
Tag only proper names: PERSON (people), PLACE (cities, countries, regions, rivers, mountains, landmarks), ORG (companies, institutions, agencies, teams), GROUP (nationalities, ethnic or religious groups used as proper nouns), WORK (titles of books, films, songs), EVENT (named events, wars, festivals).
Do NOT tag: numbers, dates, months, weekdays, units, common nouns, job titles or generic role words, languages used adjectivally in ordinary description.
List each distinct entity once in
entities, even if mentioned multiple times.Set
genderfor PERSON entities when inferable (needed for pronoun consistency), otherwise null.
name_swapped rules
Produce a twin sentence identical to text except that EVERY entity is replaced:
Replace each entity with a different real entity of the same type. Replace all mentions of the same entity with the same replacement.
Leave every other word, all punctuation, and all casing untouched. Adjust only obligatory grammar (e.g., "a"/"an").
The replacement must be acoustically dissimilar from the original: different initial sound, no long shared substring. Never use a spelling or cognate variant (John↔Jon, Maria↔Marie, Brazil↔Brasil are all forbidden).
The replacement must not already appear anywhere in the sentence, and must not be acoustically similar to any other word in the sentence.
For PERSON entities that are coreferent with pronouns in the sentence, match gender.
The result must be fluent and type-plausible (a person where a person was, a city where a city was). It is fine — expected — that the twin states something factually false; only grammaticality and type-plausibility matter.
names_masked rules
Replace each entity mention with a generic phrase of the same type that fits the syntax: PERSON→"the man"/"the woman"/"the person", PLACE→"the place"/"the region"/"the city", ORG→"the organization", GROUP→"the people", WORK→"the event". Do not paraphrase or alter anything outside the entity spans, but the result MUST be grammatical English:
Merge the placeholder with surrounding determiners, titles, and modifiers so no article clash occurs. Never output "the the", "an the", or a title directly before a placeholder: "President Obama"→"the president", "Dr. Smith"→"the doctor", "Ancient China"→"that ancient land", "Mount Everest"→"the mountain" — the title or modifier absorbs into the placeholder instead of preceding it.
Adjacent entity mentions that refer to one thing become ONE placeholder, not two.
Capitalize the placeholder when it starts a sentence ("The man, called to be...").
names_only rules
The distinct entity surface forms in order of first appearance, joined by "; ". Example: "Paul; Macedonia".
Sentences without entities
Set name_bearing to false, entities and replacements to [], and name_swapped, names_masked, names_only to null.
Domain notes (FLEURS)
Sentences come from English Wikipedia-style text: news, geography, science, sport, travel.
Sample replacements from real-world entities of the same type and roughly comparable prominence (a world capital for a world capital, a national politician for a national politician, a well-known company for a well-known company).
Acronym entities (NASA, FIFA, EU) count as ORG; replace with another acronym or short-name organization of similar kind.
Demonyms and country adjectives ("French wine", "Brazilian team") count as GROUP/PLACE usage — tag and swap them, keeping the adjectival form ("Turkish wine", "Peruvian team").
Product and brand names count as ORG.
Bible Prompt
Task
You are generating controlled text variants for a speech–text retrieval probing experiment. The input is an English New Testament verse that has spoken translations in many languages. We need to measure how much retrieval relies on proper names (which are transliterated, and therefore sound similar, across most Bible translations). You will (1) identify proper-name entities and (2) produce three derived texts: a name-swapped twin, a names-masked version, and a names-only string.
Work carefully and deterministically. Do not add commentary.
Input
A single JSON object: {"text": "..."}
Output
A single JSON object, nothing else:
{
"name_bearing": true|false,
"entities": [{"surface": "...", "type": "PERSON|PLACE|ORG|GROUP|WORK|EVENT", "gender": "m|f|null"}],
"replacements": [{"original": "...", "replacement": "...", "type": "..."}],
"name_swapped": "..." | null,
"names_masked": "..." | null,
"names_only": "..." | null
}
Entity tagging rules
Tag only proper names: PERSON (people), PLACE (cities, regions, rivers, mountains), ORG, GROUP (peoples and sects used as proper nouns), WORK, EVENT.
Do NOT tag: numbers, dates, common nouns, generic role words ("the centurion", "a prophet"), or translated titles (see domain notes).
List each distinct entity once in
entities, even if mentioned multiple times.In multi-part references like "Simon son of Jonah", tag "Simon" and "Jonah" as separate PERSON entities.
Set
genderfor PERSON entities when inferable (needed for pronoun consistency), otherwise null.
name_swapped rules
Produce a twin verse identical to text except that EVERY entity is replaced:
Replace each entity with a different entity of the same type. Replace all mentions of the same entity with the same replacement.
Leave every other word, all punctuation, and all casing untouched. Adjust only obligatory grammar (e.g., "a"/"an").
The replacement must be acoustically dissimilar from the original: different initial sound, no long shared substring. Never use a spelling or cognate variant (Simon→Simeon, Judas→Jude, Mary→Miriam are all forbidden).
The replacement must not already appear anywhere in the verse, and must not be acoustically similar to any other word in the verse.
For PERSON entities that are coreferent with pronouns in the verse, match gender.
The result must be fluent and type-plausible (a person where a person was, a city where a city was). It is fine — expected — that the twin states something factually false; only grammaticality and type-plausibility matter.
names_masked rules
Replace each entity mention with a generic phrase of the same type that fits the syntax: PERSON→"the man"/"the woman", PLACE→"the place"/"the city"/"the region", GROUP→"the people", ORG→"the assembly". Do not paraphrase or alter anything outside the entity spans, but the result MUST be grammatical English:
Merge the placeholder with surrounding determiners, titles, and modifiers so no article clash occurs. Never output "the the" or a title directly before a placeholder: "our Lord Jesus Christ"→"our Lord", "King Herod"→"the king", "Mount Sinai"→"the mountain" — the title absorbs the name instead of preceding a placeholder.
Adjacent entity mentions that refer to one person become ONE placeholder, not two: "Christ Jesus"→"the man", never "the man the man".
Capitalize the placeholder when it starts a sentence or verse ("The man, called to be an apostle...").
names_only rules
The distinct entity surface forms in order of first appearance, joined by "; ". Example: "Paul; Macedonia".
Verses without entities
Set name_bearing to false, entities and replacements to [], and name_swapped, names_masked, names_only to null.
Domain notes (New Testament)
Tag transliterated names; do not tag translated titles. The experimental confound is acoustic sharing through transliteration, so the rule is: words rendered as names across Bible translations are entities; words rendered as translated titles are not.
TAG and swap: Jesus, Christ (acoustically name-like and transliterated), Peter, Paul, Mary, Pilate, Herod, Jerusalem, Galilee, Capernaum, Jordan, Pharisees, Sadducees, Samaritans, etc.
Do NOT tag: God, Lord, the Father, the Son, the Holy Spirit, the Spirit, the Most High, Son of Man, the Devil, heaven, the temple, the synagogue.
Sample replacements from the same domain: swap people for other biblical people of similar narrative role and prominence (an apostle for an apostle, a ruler for a ruler), places for other biblical places of the same kind (a Galilean town, a region for a region).
Each verse is processed independently; replacements do NOT need to be consistent across verses.
References
- [1]Takanori Ashihara, Marc Delcroix, Yusuke Ijima, and Makio Kashino. 2024. Unveiling the linguistic capabilities of a self-supervised speech model through cross-lingual benchmark and layer-wise similarity analysis. IEEE Access, 12:98835–98855.DOI
- [2]Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022. XLS-R: self-supervised cross-lingual speech representation learning at scale. In 23rd Annual Conference of the International Conference on Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, pages 2278–2282. ISCA.arxiv.org/abs/2111.09296
- [3]Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.arxiv.org/abs/2006.11477
- [4]Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, and Ioan Calapodescu. 2024. mhubert-147: A compact multilingual hubert model. In 25th Annual Conference of the International Speech Communication Association, Interspeech 2024, Kos, Greece, September 1-5, 2024. ISCA.arxiv.org/abs/2406.06371
- [5]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangshan Yan, and Furu Wei. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 16(6):1505–1518.DOI
- [6]William Chen, Wangyou Zhang, Yifan Peng, Xinjiang Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, and Shinji Watanabe. 2024. Towards robust speech representation learning for thousands of languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10205–10224, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2407.00837
- [7]Cheol Jun Cho, Abdelrahman Mohamed, Shang-Wen Li, Alan W. Black, and Gopala Krishna Anumanchipalli. 2024. Sd-hubert: Sentence-level self-distillation induces syllabic organization in hubert. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, pages 12076–12080. IEEE.arxiv.org/abs/2310.10803
- [8]Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. 2024. Self-supervised speech representations are more phonetic than semantic. In 25th Annual Conference of the International Speech Communication Association, Interspeech 2024, Kos, Greece, September 1-5, 2024. ISCA.arxiv.org/abs/2406.08619
- [9]Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021, Cartagena, Colombia, December 13-17, 2021, pages 244–250. IEEE.arxiv.org/abs/2108.06209
- [10]Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021. Unsupervised cross-lingual representation learning for speech recognition. In 22nd Annual Conference of the International Speech Communication Association, Interspeech 2021, Brno, Czechia, August 30 - September 3, 2021, pages 2426–2430. ISCA.arxiv.org/abs/2006.13979
- [11]Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: few-shot learning evaluation of universal representations of speech. In IEEE Spoken Language Technology Workshop, SLT 2022, Doha, Qatar, January 9-12, 2023, pages 798–805. IEEE.arxiv.org/abs/2205.12446











