WorksComputer Science
Preference-Aware Pareto Steering for Multi-Task Speech LLMs
Abstract
Speech large language models (LLMs) perform multiple tasks, including automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST). Joint training of speech LLMs creates task trade-offs, yet standard training with fixed loss weights does not generally control the resulting trade-off according to deployment preferences. To address this, we propose Endpoint-Anchored Preference-Aware Sampling Uniformly along the Pareto Front (EA-SURF), an a priori method that adapts scalarization weights so that the resulting task trade-off reflects the desired preference. EA-SURF retains the standard weighted-loss training pipeline, tracking the current model's performance relative to each task-specific reference and adapting the scalarization weights as training proceeds. We theoretically show that EA-SURF can match a desired preference, with the update becoming stationary at the target under checkable regularity conditions. Across six three-objective speech LLM requests, EA-SURF empirically converges toward the requested targets and increases endpoint-inclusive normalized hypervolume by 17.8% over fixed loss weighting.
Introduction
Large language models (LLMs) have expanded beyond text to process and generate information across modalities. Speech LLMs combine acoustic representations with pretrained LLMs to support diverse speech and audio capabilities [1, 2]. Existing systems span speech recognition and translation [3, 4], while recent models extend this interface to auditory instruction following and fine-grained capabilities such as word-level timing [1, 5]. These developments motivate general-purpose speech LLMs that consolidate multiple capabilities in a shared model.
Training these models requires balancing multiple task objectives [6]. Multi-objective optimization (MOO) provides a principled formulation [7, 8, 6]:
where denotes the trainable parameters and the number of objectives. MOO seeks Pareto-optimal solutions, for which no objective can be improved without worsening another [9]. Their objective values form the Pareto front (PF) of attainable trade-offs. For , let denote its sampling distribution and its per-sample objective, giving . This formulation covers supervised losses and, after sign reversal, expected rewards estimated from sampled trajectories [10] in reinforcement learning problems.
Existing speech systems acquire multiple capabilities through staged adaptation, mixed-task supervision, or prescribed training
recipes [1, 5]. While effective, these approaches do not provide calibrated control over the attained trade-off. Preference-guided MOO distinguishes a priori steering, where a known deployment preference specifies an operating point [7, 8], from a posteriori optimization, which generates Pareto solutions for later selection [9, 11]. The former suits known deployment priorities, whereas the latter supports priorities chosen after training [9]. For bi-objective a posteriori coverage, SURF [10] calibrates scalarization weights using PF geometry. Calibrating a prescribed preference for general objectives remains underexplored.
We propose EA-SURF for a priori preference calibration. It uses the standard scalarization interface while adapting its training weights from measured PF geometry. Task sampling selects observations for each objective estimate, while scalarization determines how the objectives update the shared model. A standard choice is linear scalarization (LS) [9]:
where weights the training objectives. Setting for a deployment preference is natural, but it does not generally ensure the corresponding trade-off [9, 10]. Indeed, uniformly spaced weights can produce Pareto points clustered in some regions of objective space [10]. Here, encodes relative task priority rather than prescribed WER, AAS, or BLEU values. EA-SURF realizes it through target proportions of specialist-relative training-objective distances.
In the speech application of Figure 1, , with automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST), so and . For example, the deployment preference prioritizes ASR and TS, yet the same proportions used as loss weights need not realize the intended compromise. EA-SURF addresses this mismatch by fixing , adapting from objective-space feedback, and continuing the same warm-started probe. Motivated by preference-ray Pareto representations [7], we assign more important objectives smaller target distances from their specialists:

Figure 1. EA-SURF preference steering for ASR, timestamp prediction (TS), and speech translation (ST). Specialist models anchor the objective-space geometry, while a deployment preference is mapped by (3) to target specialist-relative distance shares . EA-SURF iteratively adapts the scalarization weights from the measured share mismatch and continues training from the current model toward the requested operating point.
where denotes the measured loss vector of a model trained predominantly for objective . EA-SURF normalizes the measured distances into attained shares and compares them with the target shares . This request maps to . Independent runs over a preference grid also yield multiple calibrated points for a posteriori use. Our contributions are:
C1) Preference realizability theory. Under regularity conditions, we show that every interior preference target can be matched by a unique scalarization weight, yielding a Pareto-optimal solution (Sec. 3.1).
C2) Closed-loop algorithm. EA-SURF adapts one warm-started probe using reusable specialists, with interior fixed points exactly matching the requested target shares (Sec. 3.2).
C3) Speech experimental results. Across six requests, EA-SURF improves preference alignment, lowers mean timestamp error from 46.6 to 39.2 ms, and raises normalized hypervolume by 17.8% over fixed LS. Table 2 reports the associated WER, AAS, and BLEU trade-offs (Sec. 4).
| Request | ASR WER (%) LS | ASR WER (%) EA | AAS (ms) LS | AAS (ms) EA | BLEU LS | BLEU EA |
| 7.38 | 7.49 | 54.3 | 41.8 | 32.89 | 32.54 | |
| 7.31 | 7.59 | 43.0 | 35.2 | 31.91 | 31.64 | |
| 7.33 | 7.62 | 38.0 | 35.8 | 29.54 | 31.71 | |
| 7.20 | 7.24 | 50.9 | 38.8 | 32.76 | 32.53 | |
| 7.23 | 7.39 | 42.6 | 37.0 | 30.04 | 31.39 | |
| 7.14 | 7.21 | 51.0 | 46.6 | 31.45 | 31.47 | |
| Mean | +0.16 | −7.4 | +0.45 |
Table 2. Three-objective results with 180k-step training budgets for both methods. EA denotes EA-SURF and = EA-SURF − LS.
Notation. For , let and . Bold lowercase letters denote vectors. and are the and Euclidean norms. For a set , and denote its relative interior and boundary. Subscripts index objectives, parenthesized superscripts index outer rounds, and denotes optimal or target quantities.
Related Work
Unified and multi-task speech learning. Modern speech models increasingly place several capabilities in one shared architecture. SpeechT5 uses multi-task pretraining [3], Whisper uses large-scale multitask supervision [4], and SALMONN and WavLLM use cross-modal alignment and instruction tuning to broaden speech and audio interfaces [1, 2]. In-Sync adds timestamp prediction through task conditioning and timestamp regularization [5]. More explicitly, multi-objective speech work includes M2ASR, which optimizes multilingual ASR and ST under unsupervised-performance constraints [6], and Objective Soups, which combines conflict-aware gradients for multilingual ASR and ST [12]. These approaches focus on shared architectures, supervision, or joint optimization. EA-SURF instead uses attained specialist-relative shares to calibrate one fixed deployment preference during continued adaptation.
Multi-task and preference-aware optimization. Multi-task optimization also acts through controls distinct from a requested Pareto location. PCGrad modifies task gradients to reduce interference [13], whereas P1KE adapts task proportions from gradient interactions [14]. These methods respectively alter the loss update and data exposure. In our formulation, the sampler remains fixed while EA-SURF adapts scalarization weights from objective-space feedback. Fixed LS is therefore the main experimental control because EA-SURF retains the same weighted-loss, single-model training interface, whereas gradient-combination approaches require per-objective gradient calculations not used here. Preference-guided MOO provides complementary ways to state a desired compromise. EPO targets a loss-space ray [7], and FERERO encodes relative or absolute constraints [8]. For a posteriori selection, Pareto HyperNetworks learn preference-conditioned solution families [11], while SURF uses arc-length geometry to improve PF coverage [10]. EA-SURF instead calibrates the scalarization weights for one known request by comparing attained specialist-relative shares with the request-derived target along the same continuing probe.
EA-SURF Method
Preference-Steering Geometry
The deployment preference specifies the requested trade-off. The scalarization weight controls training, and the two need not coincide. We characterize the operating point attained by through the specialist distances in (3). For , let and . Thus, records the attained position relative to the specialists. The target in (3) is the unique affine, permutation-equivariant, specialist-consistent mapping from preferences to distance shares. For a pure preference for objective , specialist consistency assigns zero share to its own specialist. Symmetry and simplex normalization assign to the remaining specialists, and affinity gives . Preference steering therefore seeks such that . The following assumption makes this correspondence well posed.
Assumption 1 (Regular preference matching). The objectives admit a hidden representation satisfying the hidden-convexity, smoothness, and PF nondegeneracy conditions in [9, 10]. The attained shares are continuously differentiable in with Jacobian rank . For each interior preference considered, is strictly convex on and attains its minimum in .
Assumption 1 gives sufficient conditions for a unique preference match. Hidden convexity and PF regularity support scalarization, differentiability and full rank ensure that the attained preference responds locally to changes in scalarization weights, and strict convexity with interior attainment ensures uniqueness. Cross-entropy is convex in the output logits and, under a fixed linear output head, in the final hidden representation, despite being generally nonconvex in the full model parameters [10]. We empirically examine whether the resulting trajectories remain well behaved and whether multiple mismatch minima arise in Sec. 4.
Proposition 1 (Preference realizability). Under Assumption 1, every under consideration admits a unique such that . Consequently, is Pareto optimal and satisfies .
Proof. Assumption 1 gives a unique interior minimizer of the squared mismatch. First-order optimality makes its residual orthogonal to the directions generated by the attained-share Jacobian. Both share vectors lie in , so their residual has zero sum. The
Algorithm 1 EA-SURF for one deployment request
Input: Preference p, anchors $\mathbf{e}_{m=1}^{M}$, warm start $\theta^{(0)}$, gain $\alpha$, optimizer TrainStep, budgets $K_{k=0}^{T-1}$. Set $\mathbf{q}^{\star}(\mathbf{p})$ by (3) and $\mathbf{w}^{(0)} \leftarrow \mathbf{p}$
for $t = 0,\ldots,T-1$ do
Measure $\ell(\theta^{(t)})$, compute $d_m(\theta^{(t)})$, and normalize to $\mathbf{q}(\theta^{(t)})$
Form the current-to-target share ratios and update $\mathbf{w}^{(t+1)}$ by (4)
Set $\theta^{(t,0)} \leftarrow \theta^{(t)}$
for $k = 0,\ldots,K_t-1$ do
$\theta^{(t,k+1)} \leftarrow \operatorname{TrainStep}(\theta^{(t,k)},\mathbf{w}^{(t+1)})$
end for
Set $\theta^{(t+1)} \leftarrow \theta^{(t,K_t)}$
end for
Output: Terminal probe $\theta^{(T)}$
rank- Jacobian spans this -dimensional simplex tangent space, which forces the residual to be zero. Strict convexity gives the unique matching weight. Hidden convexity and positive interior weights make the associated LS solution locally unique and Pareto optimal [9, 10].1
Proposition 1 establishes existence and uniqueness of the matching supported solution under Assumption 1. It does not establish convergence of the finite-horizon adaptive update.
EA-SURF: Adaptive Scalarization from Preference
Proposition 1 establishes a unique supported realization of the target shares. EA-SURF seeks this realization by adapting LS weights from observed share mismatch while continuing a single warm-started probe. At outer iteration , the preference calibration error (PCE), given by , measures the mismatch between attained and target shares. Starting from with , EA-SURF forms for and applies the exponentiated, simplex-preserving update [15]
where and keeps the ratios finite. When , the probe is relatively too far from specialist . EA-SURF therefore increases the relative LS weight on objective , with the intended effect of moving the next continuation closer to that specialist. The same probe then trains for steps under the updated weights, where TrainStep denotes one optimizer update on the weighted objective in (2). Algorithm 1 summarizes this closed-loop procedure. Each request uses one warm-started probe and reuses the specialist anchors, avoiding neighboring simplex probes or an explicit PF mesh. Computing anchor distances and updating weights adds negligible overhead relative to the training steps.
Figure 2 illustrates the feedback dynamics: measured shares move toward their targets and imbalance ratios move toward one, while LS weights depart substantially from the original preference . Thus, matching the requested trade-off can require training weights different from the preference itself. The next result characterizes the update’s fixed point.

Figure 2. EA-SURF dynamics for . Left: attained shares in the simplex. Upper right: current-to-target ratios. Lower right: the resulting adaptive LS weights. The gold star marks , and the dashed line marks ratio one.
Proposition 2 (Target consistency). For and , the update in (4) satisfies if and only if .
Proof. At an interior fixed point, all , and hence all , equal some . Therefore . Summing over and using yields . The converse follows immediately from (4).
Thus, the desired preference match is exactly the interior stationary condition of the weight update. This target-consistency result is not a convergence theorem. Finite-horizon behavior is assessed through the trajectories in Sec. 4.
EXPERIMENTS
Speech LLM, tasks, and evaluation
Configuration and model. We use the SLAM-LLM speech interface [16] adopted by Granite-Speech [17] and In-Sync [5], with a frozen ConFC-TC encoder, a trainable Q-Former speech adapter [18], and a frozen Granite-4.0-1B-Base causal LLM. Audio is sampled at 16 kHz, capped at 60 s, and converted to 80-bin log-Mel features. We train the speech adapter and rank-64 LoRA modules [19], with scaling 32 and dropout 0.1, for 81M trainable parameters. Training uses four NVIDIA H100 GPUs with a total batch size of 64.
Training and validation data. ASR training combines Common Voice and LibriSpeech [20, 21], VoxPopuli, Earnings-22, the AMI Meeting Corpus (AMI), CALLHOME, Switchboard, Fisher, and additional voicemail, multispeaker, and synthetic speech. TS uses forced-aligned subsets of these corpora. We discard samples whose alignment score between CTC forced alignment and GMM-HMM alignment from the Montreal Forced Aligner exceeds 175 ms [22]. Following In-Sync [5], targets interleave words with centisecond end times. ST uses nine Common Voice-derived translation sets spanning English-to-multilingual and multilingual-to-English directions. Validation draws from Multilingual LibriSpeech (MLS), Common Voice, LibriSpeech, VoxPopuli, AMI, and Earnings-22.
Evaluation. We evaluate ASR on seven held-out English conditions and TS on nine aligned conditions, including manually timestamped TIMIT [23] and Buckeye [24]. TIMIT and Buckeye are excluded from training and therefore provide out-of-domain timestamp evaluations. ST uses thirteen directions from CoVoST 2 and FLEURS [25, 26]. We report unweighted macro-averages using word error rate (WER), accumulated averaging shift (AAS) [27], and bilingual evaluation understudy (BLEU) computed with SacreBLEU [28]. MOO performance is measured by PCE and endpoint-inclusive normalised hypervolume (HV) [29]. PCE is an internal preference-alignment metric because it uses the same training-objective share mismatch that drives EA-SURF. WER, AAS, BLEU, and HV assess held-out outcomes or trade-off coverage. For HV, we minimize , normalize each coordinate by the ideal and nadir of the three shared anchors, and use the fixed reference point . Lower WER, AAS, and PCE and higher BLEU and HV are better. All table entries are single runs. Reported means and population standard deviations aggregate the six requests and are not uncertainty estimates over independent reruns.
| Preference | Fixed LS WER (%) ↓ | Fixed LS AAS (ms) ↓ | EA-SURF WER (%) ↓ | EA-SURF AAS (ms) ↓ |
| (.25, .75) | 7.37 | 32.8 | 7.52 | 30.6 |
| (.50, .50) | 7.26 | 35.9 | 7.34 | 34.5 |
| (.75, .25) | 7.24 | 41.7 | 7.30 | 40.1 |
Table 1. Bi-objective results from random initialization. The first two rows are fixed-LS specialists reused as EA-SURF anchors. The remaining rows compare the interior preferences.
Baselines and implementation
Inner training and sampling. We estimate each task objective by averaging token-level cross-entropy over its active non-padding tokens, then combine tasks using (2). Each minibatch draws at least one example from each active task’s index pool with dataset-balance exponent 0.6. Zero-weight tasks are removed from the pool and loss when training specialist anchors. Inner training uses AdamW at with zero weight decay, 1000 warm-up steps, gradient clipping at 10, and one gradient-accumulation step.
Scalarization controls. Fixed LS holds constant in (2). As a two-objective Tchebysheff diagnostic [9], we use the fixed utopia reference , select the task with the largest weighted gap at each step, differentiate its unweighted gap, and add the augmentation with .
Preference steering. EA-SURF uses Algorithm 1 with Euclidean distances in (3), , and . Each outer measurement averages task losses over 300 no-gradient training iterations per distributed rank, then averages across ranks to compute . Held-out metrics are evaluated only after round . EA-SURF shares the sampler and optimizer family of fixed LS.
Bi-objective steering diagnostic
Table 1 provides a bi-objective ASR–timestamp diagnostic from random initialization rather than the primary compute-matched benchmark. Fixed LS trains for 110k steps. EA-SURF uses a 50k-step probe followed by rounds with , and reuses two 110k-step anchors across requests. Across the three interior preferences, EA-SURF spans 0.22 ASR-WER points and 9.5 ms AAS, versus 0.13 points and 8.9 ms for fixed LS. EA-SURF’s training losses progress from through to , confirming movement along the trade-off. A separate pretrained-model diagnostic at compares LS with active-gap Tchebysheff: fixed LS improves ASR WER by 0.23 points and AAS by 0.6 ms. Aside from its small augmentation, the active-gap method differentiates only the largest task gap, using less multi-task gradient information per step. The average best PCE across the three EA-SURF trajectories is 0.0250, a 59.6% improvement over fixed LS at 0.0618.
Three-objective preference steering
Fixed LS uses one 180k-step run for each request. Each EA-SURF run starts from its 100k-step LS checkpoint and uses with , for the same nominal 180k request-level budget. There separately trained 80k-step specialists with permutations of are reused across requests. We report the pre-specified terminal . The EA-SURF continuations are launched separately, so their learning-rate schedule histories differ from those of the single 180k-step fixed-LS runs.
Across the six requests in Table 2, fixed LS with spans 7.14–7.38% ASR WER, 38.0–54.3 ms AAS, and 29.54–32.89 BLEU. Greater timestamp emphasis improves AAS, while greater translation emphasis improves BLEU, revealing their trade-off over a stable ASR range. EA-SURF reduces AAS for every request by 2.2–12.5 ms and lowers its mean from 46.6 to 39.2 ms, while increasing mean ASR WER by 0.16 points and mean BLEU by 0.45. EA-SURF is not designed to pointwise dominate fixed LS, but to reach a different operating point specified by specialist-relative training-objective shares, so moving toward the target can improve one held-out metric while worsening another. A larger likewise need not yield monotonic improvement in the corresponding downstream metric. Across all requests, the scalarization-to-share trajectories show no observed folding and each has a unique interior mismatch minimum, empirically supporting Assumption 1.
As an internal steering metric, mean PCE decreases from .240 for fixed LS to .091 for EA-SURF, with population standard deviations .119 and .038, a 62.2% reduction. For the endpoint-inclusive HV defined above, each six-request set is augmented with the shared anchors , , and in order. Normalized HV increases from .519 to .612, or 17.8%, indicating improved endpoint-inclusive trade-off coverage.
Conclusion
In this paper, we propose EA-SURF, which steers multi-task speech LLMs without directly using the deployment preference as linear-scalarization weights. To achieve this separation, it uses one reusable anchor model per task, each trained predominantly for that task. Specifically, EA-SURF converts the preference into target proportions of the Euclidean distances between the current model’s training-loss vector and the anchor models’ loss vectors, then updates the weights based on the measured mismatch along a warm-started training trajectory. Under the stated regularity assumptions, each interior target has a unique supported realization, and the weight update is stationary exactly when the measured and target shares coincide. Empirically, across six three-objective requests, EA-SURF improves preference alignment, lowers mean AAS by 7.4 ms, and raises endpoint-inclusive normalized HV by 17.8% over fixed LS. These results of EA-SURF demonstrate improved preference calibration and trade-off coverage compared to baselines.
References
- [1]Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, “Salmonn: Towards generic hearing abilities for large language models,” in International Conference on Learning Representations, 2024, vol. 2024, pp. 16607–16629.arxiv.org/abs/2310.13289
- [2]Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al., “Wavllm: Towards robust and adaptive speech large language model,” in Findings of the Association for Computational Linguistics: EMNLP, 2024, pp. 4552–4572.arxiv.org/abs/2404.00656
- [3]Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al., “Speech5: Unified-model encoder-decoder pre-training for spoken language processing,” in Proc. of the 60th annual meeting of the association for computational linguistics, 2022.
- [4]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavy, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning, 2023, pp. 28492–28518.arxiv.org/abs/2212.04356
- [5]Xulin Fan, Vishal Sunder, Samuel Thomas, Mark Hasegawa-Johnson, Brian Kingsbury, and George Saon, “In-sync: Adaptation of speech aware large language models for asr with word level timestamp predictions,” in International Conference on Acoustics, Speech and Signal Processing. IEEE, 2026.
- [6]AFM Saif, Lisha Chen, Xiaodong Cui, Songtao Lu, Brian Kingsbury, and Tianyi Chen, “M2asr: Multilingual multi-task automatic speech recognition via multi-objective optimization,” in Proc. Interspeech 2024, 2024, pp. 1240–1244.
- [7]Debabrata Mahapatra and Vaibhav Rajan, “Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization,” in International Conference on Machine Learning. PMLR, 2020, pp. 6597–6607.
- [8]Lisha Chen, AFM Saif, Yanning Shen, and Tianyi Chen, “Ferero: A flexible framework for preference-guided multi-objective learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 18758–18805, 2024.arxiv.org/abs/2412.01773
- [9]Kaisa Miettinen et al., Nonlinear multiobjective optimization, vol. 12, Kluwer Academic Publishers Boston, 1999.
- [10]Liyuany Jiang, Chentong Huang, and Lisha Chen, “Surf: Steering the scalarization weight to uniformly traverse the pareto front,” arXiv preprint arXiv:2605.20619, 2026.arxiv.org/abs/2605.20619
- [11]Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik, “Learning the pareto front with hypernetworks,” in International Conference on Learning Representations, 2021.arxiv.org/abs/2010.04104
- [12]A F M Saif, Lisha Chen, Xiaodong Cui, Songtao Lu, Brian Kingsbury, and Tianyi Chen, “Objective sups: Multilingual multi-task modeling for speech processing,” in Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2508.09228
- [13]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn, “Gradient surgery for multi-task learning,” Advances in neural information processing systems, vol. 33, pp. 5824–5836, 2020.arxiv.org/abs/2001.06782
- [14]Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviayn, and Vahab Mirrokni, “PiKE: Adaptive data mixing for large-scale multi-task learning under low gradient conflicts,” in Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2502.06244
- [15]Jyrki Kivinen and Manfred K Warmuth, “Exponentiated gradient versus gradient descent for linear predictors,” Information and computation, vol. 132, no. 1, pp. 1–63, 1997.DOI
- [16]Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al., “An embarrassingly simple approach for llm with strong asr capacity,” arXiv preprint arXiv:2402.08846, 2024.arxiv.org/abs/2402.08846
- [17]George Saon, Avihu Dekel, Alexander Brooks, Tohru Nagano, Abraham Daniels, Aharon Satt, Ashish Mittal, Brian Kingsbury, David Haws, Edmilson Morais, et al., “Granite-speech: open-source speech-aware llms with strong english asr capabilities,” in 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2025, pp. 1–7.arxiv.org/abs/2505.08699
- [18]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PmLR, 2023, pp. 19730–19742.arxiv.org/abs/2301.12597
- [19]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
- [20]Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Hennerty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber, “Common voice: A massively-multilingual speech corpus,” in Proc. of the twelfth language resources and evaluation conference, 2020.arxiv.org/abs/1912.06670
- [21]Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in International Conference on Acoustics, Speech and Signal Processing. IEEE, 2015.DOI
- [22]Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Proc. Interspeech 2017, 2017, pp. 498–502.DOI
- [23]John S Garofolo, Lori F Lamel, William M Fisher, David S Pallett, Nancy L Dahlgren, Victor Zue, and Jonathan G Fiscus, “Timit acoustic-phonetic continuous speech corpus,” 1993.
- [24]Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond, “The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability,” Speech Communication, vol. 45, no. 1, pp. 89–95, 2005.
- [25]Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino, “Covost 2 and massively multilingual speech translation,” in Proc. Interspeech 2021, 2021, pp. 2247–2251.DOI
- [26]Alexis Conneau, Maïa Maamari, Xian Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in Spoken Language Technology Workshop (SLT). IEEE, 2022, pp. 798–805.arxiv.org/abs/2205.12446
- [27]Xian Shi, Yanni Chen, Shiliang Zhang, and Zhijie Yan, “Achieving timestamp prediction while recognizing with non-autoregressive end-to-end asr model,” in National Conference on Man-Machine Speech Communication. Springer, 2022.arxiv.org/abs/2301.12343
- [28]Matt Post, “A call for clarity in reporting bleu scores,” in Proc. of the third conference on machine translation, 2018.arxiv.org/abs/1804.08771
- [29]Eckart Zitzler and Lothar Thiele, “Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach,” IEEE transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257–271, 1999.DOI