WorksEngineering
Wav2Aug: A Drop-In Speech Augmentation Policy
Abstract
SpecAugment is the dominant speech augmentation policy, but researchers have studied it primarily for automatic speech recognition (ASR), and it operates on spectrograms, limiting its applicability to other speech tasks and waveform models. We propose Wav2Aug, a drop-in waveform augmentation policy that randomly samples two augmentations from a calibrated pool of waveform transformations. We calibrate the augmentation hyperparameters once and reuse them across tasks and architectures, eliminating task-specific tuning. Experiments on ten speech benchmarks show that Wav2Aug achieves ASR performance comparable to SpecAugment while improving over it on average across non-ASR tasks, yielding an average relative error reduction of 12.8% over the strongest compatible SpecAugment baseline. We further show that multi-view training provides additional gains, with two augmented views offering the best tradeoff between performance and computational cost.
Introduction
Data augmentation improves speech model robustness by increasing training diversity without collecting new data [1]. Speech augmentation policies often combine perturbations, masking operations, and randomized transformation pools [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. A prominent example is SpecAugment [10], which applies masking and warping operations in the time and frequency domain. Although more sophisticated policy search and adaptive augmentation methods have been proposed [13, 14, 15], SpecAugment remains the dominant augmentation strategy in speech processing. It has even been used in large-scale systems such as Whisper [16] and Qwen-Audio [17], despite some important limitations.
SpecAugment has been extensively validated on ASR tasks, but its effectiveness on other speech processing tasks remains less studied. We therefore evaluate SpecAugment as a drop-in augmentation policy for non-ASR speech tasks using its original fixed configurations [10, 18] as done in [16, 17], and observe substantially smaller gains than those typically reported for ASR.
Moreover, SpecAugment transfers less well to models that do not use spectrogram inputs. Self-supervised speech encoders such as wav2vec 2.0 [18], HuBERT [19], and WavLM [20], for example, cannot use standard SpecAugment directly and instead apply similar masking operations to latent features.
Beyond these limitations, many studies, including [21, 22, 23, 24, 25, 26, 27], adopt standard SpecAugment configurations without reporting task-specific tuning of the augmentation hyperparameters. We hypothesize that the widespread use of these ”frozen” policies is due to the computational cost of tuning augmentation hyperparameters and interpret the trend as evidence of a practical need for drop-in augmentation.
Motivated by these gaps, we introduce Wav2Aug, a fixed waveform augmentation policy applied on the fly to raw audio before feature extraction. Wav2Aug samples two augmentations per batch from a large pool of carefully selected transformations and applies them sequentially to each waveform. Unlike AudRandAug [12], which tunes the number of operations and their magnitude, Wav2Aug selects hyperparameter ranges once and then reuses them unchanged across all datasets and architectures, enabling task-agnostic drop-in use.
We benchmark Wav2Aug primarily as a drop-in policy for non-ASR speech tasks. Wav2Aug achieves lower mean error than the strongest compatible SpecAugment baseline on all five evaluated non-ASR tasks, with an average relative error reduction of 12.8%. We also evaluate ASR performance on speech benchmarks and show that Wav2Aug remains competitive with SpecAugment.
We also study multi-view augmentation, where we concatenate multiple independently augmented versions of the same utterance within a batch. Multiple views expose the model to several independently sampled perturbations within the same update [28, 29, 30, 31]. We find that the best-performing Wav2Aug configuration uses two concatenated augmented views, improving performance by 19.6% on average relative to non-augmented training.
Like SpecAugment, Wav2Aug offers practical defaults that may not be optimal for every task but perform well across a wide range of speech processing tasks without any augmentation hyperparameter tuning.
We release the parameter ranges, code, and an installable Python library as open source1.
Wav2Aug
Figure 1 shows the proposed augmentation policy, which randomly samples two distinct augmentations from the pool and applies them sequentially to all waveforms in a batch.
Figure 1. Overview of Wav2Aug. We sample two augmentations uniformly per batch and apply them sequentially to each utterance, with hyperparameters drawn from globally fixed ranges.
Augmentation Pool
Each operation targets a different kind of variability in speech recordings. Figure 2 shows an example waveform and spectrogram before and after applying Wav2Aug. The pool includes waveform versions of SpecAugment-style masking operations, together with several additional transformations.
Figure 2. Waveform and log-mel spectrogram before and after noise addition followed by chunk swapping. Colored lines mark chunk boundaries. Matching letters track swapped chunks.
Noise addition randomly selects a noise segment from an available noise dataset. We use the point-source noise subset of the OpenRIR corpus [32], chosen for its broad adoption and open-source availability. We crop or pad the selected segment to match the utterance length and mix it with the input waveform at a randomly sampled signal-to-noise ratio (SNR). This augmentation improves robustness to environmental background noise while operating directly on the raw waveform.
Frequency dropout removes randomly selected narrow frequency bands directly in the waveform domain using notch filters. We randomly sample the frequency range (drop bounds), number of bands (band count), and bandwidths (band widths) of the dropped bands, encouraging the model to rely on complementary spectral cues while preserving the waveform representation.
Time dropout removes randomly selected temporal regions by zeroing contiguous waveform segments. We independently sample the number (chunk count), duration (chunk length), and position of the dropped segments for each utterance, improving robustness to local temporal corruptions and missing speech regions.
Speed perturbation changes the speaking rate by resampling the waveform with a randomly selected speed factor. This augmentation simulates natural variability across speakers and speaking styles by jointly modifying speech tempo and pitch.
Amplitude clipping simulates microphone or recording saturation by clipping waveform amplitudes above a randomly sampled threshold (clip threshold), improving robustness to nonlinear distortions.
Chunk swapping randomly exchanges waveform segments (Num. Chunks), each with a duration specified by Chunk Size, perturbing local temporal structure while preserving overall signal content and duration.
Babble noise addition follows the same SNR-based mixing procedure as noise addition, but constructs the interference signal by summing the other utterances in the current mini-batch. This simulates overlapping speakers without requiring a separate babble-noise corpus.
Amplitude scaling multiplies each waveform by a randomly sampled gain factor, increasing robustness to variations in recording volume.
Polarity inversion flips the waveform’s sign with a fixed probability, simulating phase reversals introduced by different recording or playback devices.
Calibration
Each augmentation has hyperparameters and sampling ranges, reported in Table I. To enable task-agnostic deployment, we calibrate the ranges and the number of sequential operations, , once. For each augmentation, we define candidate configurations, evaluate each independently, and select the one that performs best across the calibration datasets. We evaluate hundreds of candidate configurations via grid search, varying parameters such as SNR, dropout count, and dropout width.
| Rank | Augmentation | Hyperparameter | Range / Value |
| 1 | Noise Add. | SNR (dB) | [0, 10] |
| 2 | Freq. Dropout | Drop bounds | [0, 1] |
| 2 | Freq. Dropout | Band count | [1, 8] |
| 2 | Freq. Dropout | Band width | 0.1 |
| 3 | Time Dropout | Chunk count | [1, 8] |
| 3 | Time Dropout | Chunk length (ms) | [0, 250] |
| 4 | Speed Pert. | Speed factor | {0.9, 1.0, 1.1} |
| 5 | Amp. Clip. | Clip threshold | [0, 0.75] |
| 6 | Chunk Swap. | Num. chunks | 4 |
| 6 | Chunk Swap. | Chunk size (% Len.) | 0.01 |
| 7 | Babble Add. | SNR (dB) | [0, 20] |
| 8 | Amp. Scale | Gain | [0.05, 0.5] |
| 9 | Polarity Inv. | Probability | 0.6 |
Table I. Wav2Aug augmentations and fixed ranges at 16 kHz. Rows are ordered by individual calibration effectiveness, the error reduction obtained when each operation is evaluated alone.
Formally, let denote an augmentation, a candidate configuration for that augmentation, and a calibration dataset. We write for the set of candidate configurations for augmentation , and for the set of calibration datasets. Let be the error without augmentation and the error obtained by applying augmentation with configuration . The improvement is then defined as
Positive values of indicate that the candidate reduces error relative to no augmentation, while negative values indicate that it degrades performance. For each dataset and augmentation, we compute the mean and standard deviation over all candidates
Each candidate is then scored by its average normalized gain
Among the candidates, we keep the one with the largest . The same process gives operations per batch.
We then hold the resulting ranges fixed and evaluate them on separate ASR and non-ASR tasks to test whether the policy transfers without task-specific tuning. We use ASR for calibration because Wav2Aug was initially designed to match or improve upon SpecAugment on speech recognition benchmarks. After discovering Wav2Aug could not topple the bastion, but that it nearly matched its performance, we decided to explore the frozen policies’ performance on non-ASR.
Calibration uses three ASR settings rather than a single dataset or architecture. They span different languages, model families, training objectives, and acoustic conditions. These settings are LibriSpeech-100h [33] with a Transformer, Common Voice Italian [34] with wav2vec 2.0 (CTC), and VoiceBank [35] with a CRDNN-Transducer. For comparison, SpecAugment baselines are evaluated using the same datasets and SpeechBrain recipes. All calibration experiments use the default optimization hyperparameters from those recipes [36].
Table I summarizes the final augmentation pool and calibrated ranges. The order reflects the individual error reduction each operation achieved during calibration when evaluated alone. Noise addition is strongest, while frequency and time dropout also rank highly, mirroring masking in SpecAugment but operating directly on waveforms. Compared with common SpecAugment masking settings, Wav2Aug uses shorter time drops and multiple narrow frequency drops. Speed perturbation also performs strongly, consistent with prior work [3].
These rankings describe operations evaluated individually; they do not establish which combination works best. We therefore also examine how performance changes with the number of available operations. A top- pool contains the first operations in Table I, with their calibrated ranges unchanged. We compare two fixed choices across all ten benchmarks. Top-6 adds noise via chunk swapping, and top-9 includes all nine operations. Both sample two operations per batch. We use top-6 by default and keep top-9 as a larger alternative; the pool-size study in § 2.3-D examines this choice.
Experiments
Evaluation protocol
For the ten-benchmark comparison, we evaluate two fixed SpecAugment settings, LibriSpeech Basic (LB) and Switchboard Mild (SM) [10]. These operate on time-frequency features. For wav2vec 2.0 and HuBERT, which take waveform inputs, we instead use the time- and channel-masking settings on latent features recommended in the original works, via the Hugging Face implementations [18, 19, 37]. Comparisons to the strongest compatible SpecAugment baseline use the lowest mean error among the settings available for each model.
For each dataset and augmentation policy in these experiments, we independently tune the learning rate and weight decay using a grid around the default SpeechBrain recipe values, with AdamW as the optimizer. This lets us evaluate each augmentation policy under optimization settings suited to its training dynamics. This tunes model training, not the augmentation ranges, which remain fixed. Other recipe-specific augmentations are disabled so that the comparison isolates the policy being evaluated. We report means and standard deviations over five random seeds.
ASR performance
The following experiments assess whether Wav2Aug can maintain competitive speech recognition performance relative to a more established method, such as SpecAugment. We evaluate speech and phoneme recognition on five benchmarks. These are TED-LIUM2 [40] with a BranchFormer model [39], TIMIT [41] with a CRDNN model [36], AISHELL-1 [42] with a Transformer, Common Voice Arabic [34] with a fine-tuned Whisper model [16], and CommonVoice German [34] with a Conformer [38]. Importantly, we did not use any of these datasets during calibration. This separation lets us evaluate the proposed policy’s generalization across unseen datasets, languages, architectures, and recognition settings.
Table II reports the recognition results across our five ASR benchmarks. Both Wav2Aug pools improve on non-augmented training, with top-6 reducing average relative error by 14.9%. Compared with SpecAugment LB, top-6 improves on Common Voice Arabic and TIMIT but trails on the other three benchmarks. On datasets measured by word error rate (WER), its largest gap to LB is 0.16 absolute percentage points. SpecAugment SM is stronger on Arabic, TIMIT, and AISHELL-1, with advantages over the top-6 of 0.49 WER, 0.20 phoneme error rate (PER), and 0.28 character error rate (CER) points, respectively.
| Augmentation Policy | CommonVoice AR ASR (54h) Whisper [16] (764M) | TIMIT ASR (5h) CRDNN (10M) | AISHELL-1 ASR (~170h) Transformer (31M) | CommonVoice DE ASR (1k h) Conformer [38] (110M) | TED-LIUM 2 ASR (~200h) Branchformer [39] (103M) |
| No Aug | 20.38 ± 0.06 | 16.37 ± 0.36 | 6.89 ± 0.03 | 11.41 ± 0.04 | 10.24 ± 0.10 |
| SpecAugment (LB) [10] | 19.79 ± 0.14 | 16.09 ± 0.44 | 5.98 ± 0.05 | 7.31 ± 0.07 | 8.20 ± 0.02 |
| SpecAugment (SM) [10] | 19.11 ± 0.06 | 15.23 ± 0.41 | 5.80 ± 0.08 | 7.41 ± 0.08 | 8.26 ± 0.19 |
| Wav2Aug top-6 | 19.60 ± 0.38 | 15.43 ± 0.30 | 6.08 ± 0.08 | 7.47 ± 0.10 | 8.35 ± 0.07 |
| Wav2Aug top-9 | 19.64 ± 0.46 | 15.54 ± 0.42 | 6.08 ± 0.08 | 7.48 ± 0.04 | 8.63 ± 0.08 |
Table II. ASR test WER (%), with PER for TIMIT and CER for AISHELL-1. Mean ± standard deviation over five seeds; lower is better. LB and SM denote the LibriSpeech Basic and Switchboard Mild SpecAugment policies. Parameter counts are approximate.
These results suggest that Wav2Aug’s broader augmentation policy does not improve ASR performance over SpecAugment’s time and frequency masking, but it also does not substantially degrade it, a result we find interesting.
Transfer beyond ASR
We now evaluate Wav2Aug on tasks beyond ASR. Specifically, we consider five popular non-ASR speech tasks. These are keyword spotting on Google Speech Commands [43] with X-Vectors [44], language identification on CommonLang [45] with ECAPA-TDNN [46], emotion recognition on IEMOCAP [47] with wav2vec 2.0 [18], spoken language understanding on SLURP [48] with HuBERT [19], and voice activity detection on LibriParty VAD [36] with a CRDNN model. We train all models from scratch, except wav2vec 2.0 and HuBERT, which we fine-tune from pretrained checkpoints [37].
Table III reports results on the five non-ASR benchmarks. Both Wav2Aug pools improve over the no-augmentation condition on every task. With top-6, mean error is also lower than the strongest compatible SpecAugment baseline on all five tasks, yielding an average relative error reduction of 12.8%. This average is computed by first taking the relative error reduction on each task and then averaging across tasks.
| Augmentation Policy | IEMOCAP (speech only) Emotion Rec. (12h) wav2vec2 [18] (94M) | CommonLang LangID (45h) ECAPA-TDNN [46] (21M) | Google Speech Cmd KWS (30h) X-Vectors [44] (5M) | LibriParty VAD (21h) CRDNN (0.11M) | SLURP SLU (83h) HuBERT / Att-RNN [19] (103M) |
| No Aug | |||||
| SpecAugment (latent) [18] | — | — | — | ||
| SpecAugment (LB) [10] | — | — | |||
| SpecAugment (SM) [10] | — | — | |||
| Wav2Aug top-6 | |||||
| Wav2Aug top-9 |
Table III. Non-ASR test error rates (%). Classification tasks use error rate, LibriParty uses detection error rate (DER), and SLURP uses WER. Mean standard deviation over five seeds; lower is better. Dashes indicate policies not compatible with the model architectures. Parameter counts are approximate.
The gains depend on the task and on the baseline. On Google Speech Commands, SpecAugment LB degrades performance relative to no augmentation, whereas SM improves it. Top-6 further reduces error from 3.62% with SM to 2.81%. On LibriParty, top-6 reduces detection error from 8.44% with SM to 6.20%, a relative reduction of 26.5%. It also improves over latent masking on IEMOCAP. On SLURP, however, the difference between top-6 and latent masking is only 0.03 WER points, much smaller than the reported standard deviations, so the result is better described as comparable performance.
The larger top-9 pool achieves the lowest mean errors on CommonLang, Google Speech Commands, and IEMOCAP, while top-6 performs better on LibriParty and SLURP. This demonstrates that adding operations does not uniformly improve performance, and neither pool is best on every task.
Effect of the augmentation pool size
To examine the effect of pool size more closely, we evaluate the first ranked operations for on keyword spotting and language identification. The ranges remain fixed, and we sample two operations per batch whenever at least two are available. With a one-operation pool, we apply only noise addition.
Figure 3 shows that combining operations substantially improves performance over noise alone, but the best pool size differs by task. Google Speech Commands benefits from larger pools, with little difference between six and nine operations. CommonLang reaches its lowest mean error with four operations. Adding more does not improve performance monotonically. We conclude that no single pool size works best across all tasks, as certain operations may benefit specific tasks. Together with the ten benchmark results, these observations support top-6 as a practical default, rather than a universally optimal choice.
Figure 3. Pool-size study on Google Speech Commands (top) and CommonLang (bottom), using the first ranked operations. Orange marks the lowest mean error in each panel. Bars show means and error bars show one sample standard deviation over five runs.
Multi-view augmentation
So far, all experiments have used the standard one-view setting, in which we augment each training utterance once before feeding it to the model. We now investigate whether exposing the model to multiple independently augmented versions of the same utterance within each training batch C=non-augmented (clean), A=independently augmented. Metrics are classification error for CommonLang and IEMOCAP, and CER for AISHELL-1.
can further improve performance. Specifically, we concatenate a variable number of augmented views while keeping the number of unique utterances per batch fixed. For this study, Wav2Aug uses the top-9 policy throughout, SpecAugment uses LB on CommonLang and AISHELL-1, and latent masking on IEMOCAP.
Table IV reports results for language identification on CommonLang, speech recognition on AISHELL-1, and emotion recognition on IEMOCAP. We evaluate several multi-view augmentation strategies, where C denotes the original (clean, non-augmented) input and A denotes an independently augmented version of the same utterance (using either SpecAugment or Wav2Aug). In the last column, we report the average relative improvement over the non-augmented baseline for reference.
| Batch Views | Policy | CommonLang | AISHELL-1 | IEMOCAP | Avg. |
| [C] | No aug. | 0.0% | |||
| [A] | SpecAugment | 10.6% | |||
| [A] | Wav2Aug | 16.5% | |||
| [C; A] | SpecAugment | 11.3% | |||
| [C; A] | Wav2Aug | 17.9% | |||
| [A; A] | SpecAugment | 15.4% | |||
| [A; A] | Wav2Aug | 19.6% | |||
| [C; A; A; A] | SpecAugment | 15.5% | |||
| [C; A; A; A] | Wav2Aug | 19.3% |
Table IV. Multi-view results on CommonLang, AISHELL-1, and IEMOCAP. Mean std over 5 seeds. Avg. is the relative gain over one-view non-augmented training. Wav2Aug uses top-9 throughout.
On average across the three tasks, multi-view training improves performance for both SpecAugment and Wav2Aug. This improvement comes at the cost of larger effective batch sizes, which increase both memory consumption and training time. However, the benefits of multi-view training quickly saturate, with four views offering little or no improvement over two views. On average, the best Wav2Aug configuration concatenates two independently augmented versions of each utterance within the same batch, without the original clean signal. This setting provides an average relative improvement of 19.6% over the non-augmented baseline and an additional 3.1 percentage points over the single-view Wav2Aug setting.
Conclusion
We introduced Wav2Aug, a simple drop-in waveform augmentation policy whose hyperparameter ranges are calibrated once and then reused unchanged across tasks and architectures. Across ten speech benchmarks, Wav2Aug consistently improves performance over the non-augmented baseline. The six-operation pool achieves a 12.8% average relative error reduction over the strongest compatible SpecAugment baselines on non-ASR tasks, while remaining competitive with SpecAugment on ASR benchmarks. We also show that multi-view training provides modest additional gains, with two independently augmented views offering the best trade-off and an average relative improvement of 19.6% over non-augmented training. Overall, Wav2Aug performs comparably to preset SpecAugment policies on ASR and, on average, better across the evaluated non-ASR tasks, without task-specific augmentation tuning.
Acknowledgements
The first author thanks Firat Öncel for their guidance and thoughtful feedback throughout this work, and Francesco Bonzi for many helpful discussions. We acknowledge the Digital Research Alliance of Canada (DRAC) for providing computational resources. This work also used the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking and hosted by CINECA, Italy.
References
- [1]C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, no. 1, p. 60, 2019.DOI
- [2]M. Ravanelli and M. Omologo, “Contaminated speech training methods for robust DNN-HMM distant speech recognition,” in Proc. of Interspeech, 2015, pp. 756–760.arxiv.org/abs/1710.03538
- [3]T. Ko, V. Peddinti, D. Povey et al., “Audio augmentation for speech recognition,” in Proc. of Interspeech, 2015, pp. 3586–3589.DOI
- [4]M. Ravanelli, P. Svaizer, and M. Omologo, “Realistic multi-microphone data simulation for distant speech recognition,” in Proc. of Interspeech, 2016.arxiv.org/abs/1711.09470
- [5]C. Rommel, T. Moreau, J. Paillard et al., “CADDA: Class-wise automatic differentiable data augmentation for EEG signals,” arXiv preprint arXiv:2106.13695, 2021.arxiv.org/abs/2106.13695
- [6]Y. Zhang, Y. Zhang, T. Patel et al., “Comparing data augmentation and training techniques to reduce bias against non-native accents in hybrid speech recognition systems,” in Proc. of SASG, 2022, pp. 15–19.
- [7]M. Ravanelli and M. Omologo, “On the selection of the impulse responses for distant-speech recognition based on contaminated speech training,” in Proc. of Interspeech, 2014, pp. 1028–1032.DOI
- [8]T. Prisyach, V. Mendelev, and D. Ubskıy, “Data augmentation for training of noise robust acoustic models,” in Proc. of AIST, 2016, pp. 17–25.DOI
- [9]C. Subakan, M. Ravanelli, S. Cornell, and F. Grondin, “REAL-M: Towards speech separation on real mixtures,” in Proc. of ICASSP, 2022.
- [10]D. S. Park, W. Chan, Y. Zhang et al., “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Proc. of Interspeech, 2019, pp. 2613–2617.DOI
- [11]E. D. Cubuk, B. Zoph, J. Shlens et al., “RandAugment: Practical automated data augmentation with a reduced search space,” in Proc. of CVPR Workshops, 2020, pp. 3008–3017.arxiv.org/abs/1909.13719
- [12]T. Kumar, M. Turab, A. Mileo et al., “AudRandAug: Random image augmentations for audio classification,” in Proc. of IMVIP, 2023.
- [13]G. Wang, E. D. Cubuk, A. Rosenberg et al., “G-Augment: Searching for the meta-structure of data augmentation policies for ASR,” in Proc. of SLT, 2023, pp. 23–30.arxiv.org/abs/2210.10879
- [14]E. D. Cubuk, B. Zoph, D. Mane et al., “AutoAugment: Learning augmentation strategies from data,” in Proc. of CVPR, 2019, pp. 113–123.DOI
- [15]T.-Y. Hu, A. Shrivastava, J.-H. R. Chang et al., “SapAugment: Learning a sample adaptive policy for data augmentation,” in Proc. of ICASSP, 2021, pp. 4040–4044.arxiv.org/abs/2011.01156
- [16]A. Radford, J. W. Kim, T. Xu et al., “Robust speech recognition via large-scale weak supervision,” in Proc. of ICML, 2023, pp. 28 492–28 518.arxiv.org/abs/2212.04356
- [17]Y. Chu, J. Xu, X. Zhou et al., “Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023.arxiv.org/abs/2311.07919
- [18]A. Baevski, Y. Zhou, A. Mohamed et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. of NeurIPS, 2020, pp. 12 449–12 460.arxiv.org/abs/2006.11477
- [19]W.-N. Hsu, B. Bolte, Y.-H. H. Tsai et al., “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.arxiv.org/abs/2106.07447
- [20]S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
- [21]Z. Lu, L. Cao, Y. Zhang et al., “Speech sentiment analysis via pre-trained features from end-to-end ASR models,” in Proc. of ICASSP, 2020, pp. 7149–7153.arxiv.org/abs/1911.09762
- [22]H. Le, J. Pino, C. Wang et al., “Lightweight adapter tuning for multilingual speech translation,” in Proc. of ACL-IJCNLP, 2021, pp. 817–824.
- [23]A. Lee, P.-J. Chen, C. Wang et al., “Direct speech-to-speech translation with discrete units,” in Proc. of ACL, 2022, pp. 3327–3339.arxiv.org/abs/2107.05604
- [24]Y. Tang, J. Pino, X. Li et al., “Improving speech translation by understanding and learning from the auxiliary text translation task,” in Proc. of ACL-IJCNLP, 2021, pp. 4252–4261.arxiv.org/abs/2107.05782
- [25]Y. Tang, A. Y. Sun, H. Inaguma et al., “Hybrid transducer and attention based encoder-decoder modeling for speech-to-text tasks,” in Proc. of ACL, 2023, pp. 12 441–12 455.
- [26]Z. Ma, Q. Fang, S. Zhang et al., “A non-autoregressive generation framework for end-to-end simultaneous speech-to-any translation,” in Proc. of ACL, 2024, pp. 1557–1575.DOI
- [27]A. Rouditchenko, Y. Gong, S. Thomas et al., “Whisper-Flamingo: Integrating visual features into Whisper for audio-visual speech recognition and translation,” in Proc. of Interspeech, 2024, pp. 2420–2424.
- [28]E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry, “Augment your batch: Improving generalization through instance repetition,” in Proc. of CVPR, 2020, pp. 8129–8138.DOI
- [29]S. Fort, A. Brock, R. Pascanu, S. De, and S. L. Smith, “Drawing multiple augmentation samples per image during training efficiently decreases test error,” arXiv preprint arXiv:2105.13343, 2021.arxiv.org/abs/2105.13343
- [30]A. R. Kivaisi and O. Zhao, “Improved mini-batch multiple augmentation for low-resource spoken word recognition,” Expert Systems with Applications, vol. 252, p. 124157, 2024.DOI
- [31]V. P. Singh, F. Malato, V. Hautamäki, M. Sahidullah, and T. Kinnunen, “ROAR: Reinforcing original to augmented data ratio dynamics for wav2vec 2.0 based ASR,” in Proc. of Interspeech, 2024, pp. 2885–2889.arxiv.org/abs/2406.09999
- [32]T. Ko, V. Peddinti, D. Povey et al., “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. of ICASSP, 2017.DOI
- [33]V. Panayotov, G. Chen, D. Povey et al., “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. of ICASSP, 2015, pp. 5206–5210.DOI
- [34]R. Ardila, M. Branson, K. Davis et al., “Common Voice: A massively-multilingual speech corpus,” in Proc. of LREC, 2020, pp. 4218–4222.arxiv.org/abs/1912.06670
- [35]C. Veaux, J. Yamagishi, and S. King, “The Voice Bank corpus: Design, collection and data analysis of a large regional accent speech database,” in Proc. of O-COCOSDA/CASLRE, 2013.DOI
- [36]M. Ravanelli, T. Parcollet, A. Moumen et al., “Open-source conversational AI with SpeechBrain 1.0,” Journal of Machine Learning Research, vol. 25, no. 333, pp. 1–11, 2024. [Online]. Available: http://jmlr.org/papers/v25/24-0991.htmlarxiv.org/abs/2407.00463
- [37]T. Wolf, L. Debut, V. Sanh et al., “Transformers: State-of-the-art natural language processing,” in Proc. of EMNLP: System Demonstrations, 2020, pp. 38–45.DOI
- [38]A. Gulati, J. Qin, C.-C. Chiu et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. of Interspeech, 2020, pp. 5036–5040.
- [39]Y. Peng, S. Dalmia, I. Lane et al., “Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding,” in Proc. of ICML, 2022, pp. 17 627–17 643.arxiv.org/abs/2207.02971
- [40]A. Rousseau, P. Deléglise, Y. Esteve et al., “Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks,” in Proc. of LREC, 2014, pp. 3935–3939.
- [41]J. S. Garofolo, L. F. Lamel, W. M. Fisher et al., “DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1,” National Institute of Standards and Technology, Tech. Rep. NIST IR 4930, 1993.
- [42]H. Bu, J. Du, X. Na et al., “AISHELL-1: An open-source mandarin speech corpus and a speech recognition baseline,” in Proc. of O-COCOSDA, 2017, pp. 1–5.arxiv.org/abs/1709.05522
- [43]P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209, 2018.arxiv.org/abs/1804.03209
- [44]D. Snyder, D. Garcia-Romero, G. Sell et al., “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. of ICASSP, 2018, pp. 5329–5333.DOI
- [45]G. Sinisetti, P. Ruban, O. Dymov et al., “CommonLanguage,” Jun. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5036977
- [46]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. of Interspeech, 2020, pp. 3830–3834.DOI
- [47]C. Busso, M. Bulut, C.-C. Lee et al., “IEMOCAP: Interactive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, 2008.DOI
- [48]E. Bastianelli, A. Vanzo, P. Swietojanski et al., “SLURP: A spoken language understanding resource package,” in Proc. of EMNLP, 2020, pp. 7252–7262.arxiv.org/abs/2011.13205


