WorksComputer Science
Distributionally Robust BEST-RQ For Speech Representation Learning
Abstract
BEST-RQ, based on random projection and quantization, has been a well-established self-supervised learning approach to learn speech representations from large-scale unlabeled audio due to its high performance and efficiency. However, in BEST-RQ representations learned by random quantization can be sensitive to the choice of quantization targets. To address this issue, we propose a distributionally robust extension of BEST-RQ that seeks representations robust across multiple quantization configurations. Specifically, we define groups by codebooks and formulate the training as a group Distributionally Robust Optimization (DRO) problem. Rather than optimizing solely for the average loss across groups, our approach adaptively reweights groups to emphasize those performing poorly. Additionally, we regularize DRO with a Kullback-Leibler divergence to prevent excessive distribution deviation. We evaluate our approach on the Multilingual LibriSpeech dataset and show that the distributionally robust BEST-RQ achieves superior performance over the conventional uniform average across multiple codebooks.
Introduction
In recent years, the limited availability of labeled speech data has motivated the development of self-supervised learning (SSL) methods that exploit abundant unlabeled speech. This gives rise to a commonly-used two-stage strategy where an acoustic model is first pre-trained using SSL and then fine-tuned using labeled data in downstream tasks. These SSL methods have progressively explored different pre-training objectives for learning effective speech representations. wav2vec 2.0 [1] demonstrated that effective speech representations can be learned by masking latent speech representations and predicting quantized targets through contrastive learning. HuBERT [2] further explored masked prediction by using discrete hidden speech units as learning targets, while W2v-BERT [3] combined contrastive learning with masked language modeling to learn contextualized speech representations. Data2vec [4] subsequently introduced contextualized latent targets as a general self-supervised learning objective, while WavLM [5] jointly learned masked speech prediction and denoising to improve representations for speech content and non-ASR tasks. These developments demonstrate the effectiveness of SSL for learning transferable speech representations from unlabeled audio and motivate further investigation into efficient and robust target construction strategies.
Motivation
Among the mentioned approaches, BEST-RQ (BERT-based Speech Pre-Training with Random-projection Quantizer) is an effective SSL approach that constructs discrete prediction targets using a fixed random projection and codebook [6]. A key advantage of this approach is that the quantizer does not need to be learned during pre-training, avoiding the additional computational cost associated with learning a codebook. However, fixing a randomly initialized quantizer also introduces a potential source of variability: the resulting prediction task depends on the particular random projection and codebook used to construct the discrete targets.
This dependence is important because the quality of the pre-training task is closely related to how effectively the codebook is utilized. It has been noted that higher and more uniform codebook utilization is strongly correlated with better pre-training quality, and further highlights that different random initializations of codebook can result in different codebook utilization patterns [6]. Consequently, while a fixed random codebook provides a computationally efficient alternative to learned quantization, an individual random configuration need not necessarily be an optimal choice. A representation optimized for one configuration may thus become sensitive to the target space induced by that specific codebook.
A natural alternative is to expose the model to multiple random codebooks during pre-training [7]. However, simply training with multiple codebooks using an equal-weight objective implicitly assumes that all codebook-induced prediction tasks are equally informative and equally reliable. This assumption may not hold in practice: different codebooks can induce prediction tasks with different levels of codebook utilization and difficulty, causing some configurations to contribute more to the average training objective than others. Consequently, uniform multi-codebook training does not explicitly encourage robustness to the less favorable configurations and may still favor a subset of codebooks.
These observations motivate the following question: Can we learn speech representations that are robust across multiple random codebooks while accounting for the differing quality of the corresponding prediction tasks? One alternative would be to independently pre-train models with different codebooks and select the best-performing configuration. Although this can identify a favorable target space, it requires repeating the computationally expensive pre-training process and requires selecting the best codebook. Simply averaging the objectives across multiple codebooks avoids this repeated training cost, but does not explicitly protect against poorly performing or under-utilized configurations.
Contributions
In this work, we address the aforementioned problem through distributionally robust optimization (DRO). Drawing on Group DRO [8], we regard each random codebook as a distinct group corresponding to a different discrete prediction task. Rather than weighting all codebooks uniformly, our objective explicitly accounts for their relative training losses and emphasizes configurations on which the representation performs less favorably. We further incorporate KL-divergence regularization to prevent the optimization from becoming excessively concentrated on a single codebook and to maintain a balanced contribution from multiple configurations. This formulation therefore encourages the learned representation to perform consistently across diverse codebook induced target spaces.
The proposed framework retains the computational advantage of fixed random quantization while avoiding reliance on a single randomly selected codebook. More importantly, it goes beyond the conventional multi-codebook training by explicitly optimizing for robustness across different prediction tasks. Thus, instead of repeatedly searching for the best codebook or treating all codebooks as equally informative, our approach seeks to learn a representation that remains effective across a range of possible random target spaces.
Background
BEST–RQ
BEST-RQ constructs discrete prediction targets for speech frames using a fixed random projection matrix and a fixed random codebook. Each speech feature is first transformed into a lower-dimensional representation using the random projection matrix and then quantized by selecting the nearest vector in the codebook. The index of the selected codebook entry serves as the discrete target for the corresponding time frame. During pre-training, a subset of speech frames is masked, and the model is trained to predict the corresponding discrete targets for the masked frames.
We now formulate the BEST-RQ objective. Let the -th unlabeled utterance be denoted by where is a sequence of acoustic feature vectors (obtained after applying mel-spectogram), with . Let , , denotes the masking indicators. A value of indicates that the -th frame is masked.
For a given BEST-RQ quantization configuration, the acoustic features are first projected onto a lower dimensional space using a fixed random projection matrix and subsequently quantized using a fixed codebook. This produces a discrete target , where is the number of codebook entries, for each frame. We denote the resulting sequence of target by . After obtaining the quantization targets, we replace the masked acoustic vectors with noise. Specifically, let denote the noise vector used to replace the th frame. The resulting masked input sequence is defined as
The resulting pseudo-labeled example is denoted by . Let denote the model output for frame , The BEST-RQ loss for the -th utterance is then
where denotes the cross entropy loss. Given unlabeled utterances, the empirical BEST-RQ objective is thus,
DRO
DRO considers optimization objectives that account for variation across a collection of conditions rather than focusing solely on average performance. Group DRO extends this notion by considering multiple groups and adaptively weighting them according to their associated losses, placing greater emphasis on groups for which the model performs poorly [8].
We now formulate the DRO objective. Let denote a collection of groups, and let denote the loss associated with group . The Group DRO objective can be expressed as
where denotes a probability distribution over the groups and
is the probability simplex. The distribution determines the relative contribution of each group to the objective.
To regularize the group distribution, we introduce KL-divergence regularization with respect to a uniform reference distribution.
where denote the uniform distribution over the groups; is the regularization parameter. The KL term penalizes deviations of the group distribution from the uniform reference distribution, thereby allowing us to control the extent of group reweighting.
PROBLEM FORMULATION
Having introduced BEST-RQ and DRO, we now formulate our learning problem. In our setting, each group is associated with a distinct codebook, which determines its BEST-RQ quantization configuration and corresponding discrete prediction targets. We would like to emphasize that the groups do not correspond to different subsets of the speech data. Instead, the same utterances are processed under each quantization configuration, resulting in different target spaces for the same input speech.
The framework is illustrated in Figure 1. The model consists of a shared encoder backbone and a group-specific prediction heads. Let denote the parameters of the shared encoder, and let denote the parameters of the prediction head associated with group . We define the parameters involved in group as

Figure 1. Overview of the proposed DRO-BEST-RQ with KL regularization. For utterance , the acoustic features are first projected into a lower-dimensional space using a shared projection matrix . The resulting representations are then quantized using multiple fixed random codebooks to generate group-specific pseudo-labels . The input acoustic features are processed by a shared encoder and group-specific prediction heads. The cross-entropy loss is computed for each codebook on those masked frames which are then combined through the KL-regularized DRO objective.
Thus, the encoder is jointly optimized across all BEST–RQ configurations, while each prediction head specializes to the target space induced by its corresponding random codebook.
Let denote the BEST–RQ loss for group . We formulate the learning problem as follows:
Algorithm
The pseudocode for DRO-BEST-RQ is provided in Algorithm 1. Since the problem is unconstrained with respect to the model parameters and , while the adversarial variable is constrained to the probability simplex , we solve the minimax problem using a minibatch stochastic gradient descent–projected stochastic gradient ascent (SGD-PSGA) procedure. At each iteration, we estimate the gradients of both the minimization and maximization objectives using the same minibatch, update the model parameters via stochastic gradient descent, and update via stochastic gradient ascent followed by projection onto . Specifically, for an arbitrary point , the projection onto is defined as
Algorithm 1 DRO-BEST-RQ
Inputs:
for do
Data:
Sample a minibatch
for do
Apply the label generation procedure on each sample in the minibatch to yield
end for
Gradient computation:
for do
end for
for do
end for
Encoder parameters update:
Update parameters of codebook specific heads
for do
end for
Update
end for
Output:
Experimental Results
Dataset
We evaluate the proposed DRO-BEST-RQ on the Multilingual Librispeech (MLS) dataset [9]. Five languages are used for the experiments which are English (en), Spanish (es), French (fr), German (de) and Portuguese (pt). There are 44,659.74 hours of English data from which we randomly select 4,000 hours. For the other four languages, we use all the training data in the dataset. The multilingual data from those five languages are pooled together for the BEST-RQ unsupervised training by removing the labels, which amounts to a total of 8,121.73 hours. The pre-trained BEST-RQ model is then fine-tuned individually for each language under CTC using its own data with labels [10]. The fine-tuned models are evaluated using test data from each language respectively. Table 1 summarizes the amount of speech data used for pre-training, fine-tuning, and test in each language. Note that for each language the pre-training and fine-tuning data is the same. The difference is whether the labels are used.
Model Architecture
The ASR models we train use the same architecture as ibm-granite/granite-speech-5.0-470m-turboctc. In the BEST-RQ training, the model comprises a stack of 16 Conformer blocks taking a 320-dimensional input feature (2 stacked frames of 80-dimensional log Mel spectrogram features with temporal deltas). The Conformer blocks have an internal dimension of 1024, a temporal kernel width of , use 8 128-dimensional attention heads, and chunkwise attention [11] with a block size of 128. The first two Conformer blocks perform a temporal downsampling by applying strided convolutions in the Conformer’s depthwise convolutional block and averaging successive pairs of sequence positions in the corresponding residual connection. Thus, the model output rate is 12.5 tokens/sec. The masking is applied by randomly selecting 2% of the total frame length and masking spans 20 frames. The masked frames are replaced with Gaussian noise of zero mean and 0.1 variance. Each codebook has 1024 entries. When fine-tuning under CTC, the top linear layer is replaced with a randomly initialized new one which is connected to an alphabet of 16,384 BPE tokens, including the obligatory CTC BLANK symbol. In addition, self-conditioning [12] is added at the 8th layer in the CTC fine-tuning stage.
| Language | Pre-Training (hrs) | Fine-Tuning (hrs) | Test (hrs) |
| English | 4000.00 | 4000.00 | 15.55 |
| Spanish | 917.68 | 917.68 | 10.00 |
| French | 1076.58 | 1076.58 | 10.07 |
| German | 1966.51 | 1966.51 | 14.29 |
| Portuguese | 160.96 | 160.96 | 3.74 |
Table 1. Statistics of the MLS data used in our experiments.
Implementation Details
We first establish supervised baselines for the five languages using the labeled fine-tuning data without pre-training. Then we create two BEST-RQ baselines with a single codebook and four codebooks, respectively. This involves a two-stage process. In the first stage, the acoustic model is pre-trained under BEST-RQ using pooled unlabeled multilingual data. In the second stage, the pre-trained acoustic model is fine-tuned using labeled fine-tuning data for each language. The multi-codebook BEST-RQ baseline employs the conventional uniformly weighted losses (i.e. assigning a weight of 0.25 to the loss corresponding to each codebook.). We then investigate the proposed DRO-BEST-RQ with multi-codebooks as illustrated in Figure 1 and Alg. 1 with various ’s.
Both BEST-RQ training and fine-tuning are run for 30 epochs with a OneCycleLR scheduler using 8 NVIDIA H100 GPUs. For , we set , , and . For , we use the same schedule with , , and . The warmup is set to 10% of the total training for both and . All other architectural and optimization settings are kept fixed across configurations to ensure a fair comparison. A greedy decoder is used for inference.
Illustration of the Group DRO Mechanism in practice
To provide an illustration of (1) in practice, Table 2 presents the group-specific BEST-RQ losses and the corresponding DRO weights at an arbitrarily selected training iteration. The values are reported separately for models trained with and without KL regularization and are intended to highlight the behavior of the optimization rather than to represent aggregate or final model performance. For the KL-regularized model (), the group weights adapt to the relative difficulty of the BEST-RQ configurations: the group with the largest loss receives the largest weight, while the remaining codebooks retain non-zero weights. Consequently, all codebooks continue to contribute to the optimization of the shared encoder, with greater emphasis placed on the more difficult configurations. In contrast, when KL regularization is removed (), the group distribution concentrates entirely on the highest-loss group in the corresponding independently trained model, which amounts to a single-codebook training scenario but that single codebook is the hardest one in all codebook candidates. For random codebooks, a hard codebook typically represents a broader coverage of the random codewords which could benefit the unsupervised learning. This comparison demonstrates the role of the KL term in controlling the degree of group reweighting. Specifically,
| Training setting | Group | BEST-RQ loss () | DRO weight () |
| 3.4055 | 0.2338 | ||
| 3.4789 | 0.2517 | ||
| 3.4152 | 0.2361 | ||
| 3.5800 | 0.2784 | ||
| 3.3118 | 0.0 | ||
| 3.3089 | 0.0 | ||
| 3.2886 | 0.0 | ||
| 3.3513 | 1.0 |
Table 2. Illustration of the Group DRO weighting mechanism across four BEST-RQ codebooks at an arbitrarily selected iteration.
it preserves the adaptive behavior of Group DRO while discouraging excessive concentration on a single BEST-RQ configuration.
Results & Discussion
We report Word Error Rate (WER) on the test sets described in Table 3. Results are reported for each of the five languages. The WERs of Portuguese are obviously higher than the rest four languages due to its limited fine-tuning data (only 161 hours). We compare the supervised baseline, the multi-codebook approach with uniformly weighted losses, and our proposed DRO approach with . We note that setting reduces (1) to a worst-case optimization over the codebooks, where, at each iteration, places all its mass on the codebook with the highest loss, thereby optimizing the model parameters against the hardest codebook.
| Model | en | es | fr | gr | pt | Avg |
| Supervised Baseline | 8.0 | 5.1 | 6.8 | 6.3 | 26.0 | 10.4 |
| BEST-RQ (Single-Codebook) | 8.0 | 4.8 | 6.4 | 6.3 | 20.8 | 9.3 |
| BEST-RQ (Multi-Codebook) | 7.9 | 4.6 | 6.6 | 6.3 | 18.9 | 8.9 |
| DRO-BEST-RQ () | 8.0 | 4.8 | 6.7 | 6.2 | 20.0 | 9.1 |
| DRO-BEST-RQ () | 7.9 | 4.4 | 6.3 | 6.0 | 18.5 | 8.6 |
| DRO-BEST-RQ () | 7.7 | 4.3 | 6.3 | 5.9 | 17.7 | 8.4 |
| DRO-BEST-RQ () | 7.9 | 4.3 | 6.3 | 6.1 | 18.4 | 8.6 |
Table 3. WER (%) on the MLS test sets
The results in Table 3 demonstrate the effectiveness of our proposed Algorithm 1. While multi-codebook BEST-RQ already improves upon both the supervised baseline and the single-codebook formulation, our algorithm consistently delivers additional gains on top of it for . Importantly, these improvements are observed across multiple values of , demonstrating that the gains are not tied to a single carefully selected setting. This highlights the advantage of our DRO formulation, which adaptively reweighs the codebook-specific objectives to reduce the dominance of favorable codebooks and encourage the shared encoder to remain robust to less favorable or under utilized codebook configurations. The strongest overall performance is achieved at , while the gains for further illustrate the effectiveness of our algorithm. In all the experiments, the most significant language-specific gains come from Portuguese due to its limited data, which also indicates that low-resource languages benefit from the SSL pre-training more.
Concluding Remarks
In this work, we proposed a KL-regularized distributionally robust optimization (DRO) framework for multi-codebook BEST-RQ. Our framework treats each random codebook as a separate prediction task and adaptively emphasizes less favorable codebook configurations while preventing excessive reweighting through KL regularization. Experiments on MLS data show that our approach consistently outperforms uniformly weighted multi-codebook BEST–RQ across multiple languages, establishing DRO as a principled and effective way to improve BEST–RQ pre-training without relying on a single random codebook assuming that all codebook configurations are equally informative.
References
References
- [1]Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.arxiv.org/abs/2006.11477
- [2]Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021.arxiv.org/abs/2106.07447
- [3]Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 244–250.arxiv.org/abs/2108.06209
- [4]Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International conference on machine learning. PMLR, 2022, pp. 1298–1312.arxiv.org/abs/2202.03555
- [5]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shu-jie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.DOI
- [6]Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in International Conference on Machine Learning. PMLR, 2022, pp. 3915–3924.arxiv.org/abs/2202.01855
- [7]Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al., “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037, 2023.arxiv.org/abs/2303.01037
- [8]Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,” arXiv preprint arXiv:1911.08731, 2019.arxiv.org/abs/1911.08731
- [9]V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A large-scale multilingual dataset for speech research,” in Proc. INTERSPEECH, 2020.DOI
- [10]Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the International Conference on Machine Learning, 2006.
- [11]Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu, “Google USM: Scaling automatic speech recognition beyond 100 languages,” https://arxiv.org/abs/2303.01037, 2023.
- [12]J. Nozaki and T. Komatsu, “Relaxing the conditional independence assumption of CTC-based ASR by conditioning on intermediate predictions,” in Proc. INTERSPEECH, 2021.arxiv.org/abs/2104.02724