betaFeedback

WorksComputer Science

Preference-Aware Pareto Steering for Multi-Task Speech LLMs

Abstract

Speech large language models (LLMs) perform multiple tasks, including automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST). Joint training of speech LLMs creates task trade-offs, yet standard training with fixed loss weights does not generally control the resulting trade-off according to deployment preferences. To address this, we propose Endpoint-Anchored Preference-Aware Sampling Uniformly along the Pareto Front (EA-SURF), an a priori method that adapts scalarization weights so that the resulting task trade-off reflects the desired preference. EA-SURF retains the standard weighted-loss training pipeline, tracking the current model's performance relative to each task-specific reference and adapting the scalarization weights as training proceeds. We theoretically show that EA-SURF can match a desired preference, with the update becoming stationary at the target under checkable regularity conditions. Across six three-objective speech LLM requests, EA-SURF empirically converges toward the requested targets and increases endpoint-inclusive normalized hypervolume by 17.8% over fixed loss weighting.

Introduction

Large language models (LLMs) have expanded beyond text to process and generate information across modalities. Speech LLMs combine acoustic representations with pretrained LLMs to support diverse speech and audio capabilities [1, 2]. Existing systems span speech recognition and translation [3, 4], while recent models extend this interface to auditory instruction following and fine-grained capabilities such as word-level timing [1, 5]. These developments motivate general-purpose speech LLMs that consolidate multiple capabilities in a shared model.

Training these models requires balancing multiple task objectives [6]. Multi-objective optimization (MOO) provides a principled formulation [7, 8, 6]:

min⁡θL(θ):=(ℓ1(θ),…,ℓM(θ))∈RM,(1)\min_{\theta}\mathcal{L}(\theta):=(\ell_{1}(\theta),\ldots,\ell_{M}(\theta))\in\mathbb{R}^{M}, \tag*{(1)}

where θ\theta denotes the trainable parameters and MM the number of objectives. MOO seeks Pareto-optimal solutions, for which no objective can be improved without worsening another [9]. Their objective values form the Pareto front (PF) of attainable trade-offs. For m∈[M]m\in[M], let Dm\mathcal{D}_{m} denote its sampling distribution and Lm\mathcal{L}_{m} its per-sample objective, giving ℓm(θ):=Eξ∼Dm[Lm(θ,ξ)]\ell_{m}(\theta):=\mathbb{E}_{\xi\sim\mathcal{D}_{m}}[\mathcal{L}_{m}(\theta,\xi)]. This formulation covers supervised losses and, after sign reversal, expected rewards estimated from sampled trajectories [10] in reinforcement learning problems.

Existing speech systems acquire multiple capabilities through staged adaptation, mixed-task supervision, or prescribed training

recipes [1, 5]. While effective, these approaches do not provide calibrated control over the attained trade-off. Preference-guided MOO distinguishes a priori steering, where a known deployment preference specifies an operating point [7, 8], from a posteriori optimization, which generates Pareto solutions for later selection [9, 11]. The former suits known deployment priorities, whereas the latter supports priorities chosen after training [9]. For bi-objective a posteriori coverage, SURF [10] calibrates scalarization weights using PF geometry. Calibrating a prescribed preference for general M≥2M\ge2 objectives remains underexplored.

We propose EA-SURF for a priori preference calibration. It uses the standard scalarization interface while adapting its training weights from measured PF geometry. Task sampling selects observations for each objective estimate, while scalarization determines how the objectives update the shared model. A standard choice is linear scalarization (LS) [9]:

θ∗(w)∈arg⁡min⁡θ{LLS(θ,w)=∑m=1Mwmℓm(θ)}(2)\theta^{*}(\mathbf{w})\in\arg\min_{\theta}\left\{\mathcal{L}_{\mathrm{LS}}(\theta,\mathbf{w})=\sum_{m=1}^{M}w_{m}\ell_{m}(\theta)\right\} \tag*{(2)}

where w∈ΔM\mathbf{w}\in\Delta_{M} weights the training objectives. Setting w=p\mathbf{w}=\mathbf{p} for a deployment preference p∈ΔM\mathbf{p}\in\Delta_{M} is natural, but it does not generally ensure the corresponding trade-off [9, 10]. Indeed, uniformly spaced weights can produce Pareto points clustered in some regions of objective space [10]. Here, p\mathbf{p} encodes relative task priority rather than prescribed WER, AAS, or BLEU values. EA-SURF realizes it through target proportions of specialist-relative training-objective distances.

In the speech application of Figure 1, M=3M=3, with automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST), so ℓ(θ)=(ℓasr(θ),ℓts(θ),ℓst(θ))\ell(\theta) = (\ell_{\mathrm{asr}}(\theta), \ell_{\mathrm{ts}}(\theta), \ell_{\mathrm{st}}(\theta)) and w=(wasr,wts,wst)\mathbf{w} = (w_{\mathrm{asr}}, w_{\mathrm{ts}}, w_{\mathrm{st}}). For example, the deployment preference (pasr,pts,pst)=(.4,.4,.2)(p_{\mathrm{asr}},p_{\mathrm{ts}},p_{\mathrm{st}}) = (.4,.4,.2) prioritizes ASR and TS, yet the same proportions used as loss weights need not realize the intended compromise. EA-SURF addresses this mismatch by fixing p\mathbf{p}, adapting w\mathbf{w} from objective-space feedback, and continuing the same warm-started probe. Motivated by preference-ray Pareto representations [7], we assign more important objectives smaller target distances from their specialists:

EA-SURF preference steering for ASR, timestamp prediction, and speech translation

Figure 1. EA-SURF preference steering for ASR, timestamp prediction (TS), and speech translation (ST). Specialist models anchor the objective-space geometry, while a deployment preference p\mathbf{p} is mapped by (3) to target specialist-relative distance shares q∗(p)\mathbf{q}^{*}(\mathbf{p}). EA-SURF iteratively adapts the scalarization weights from the measured share mismatch and continues training from the current model toward the requested operating point.

dm(θ):=∥ℓ(θ)−em∥2∝qm⋆(p):=1−pm∑j=1M(1−pj).(3)d_m(\theta) := \lVert\ell(\theta) - \mathbf{e}_m\rVert_2 \propto q_m^\star(\mathbf{p}) := \frac{1-p_m}{\sum_{j=1}^{M}(1-p_j)}. \tag*{(3)}

where em=(em,1,…,em,M)\mathbf{e}_m = (e_{m,1},\ldots,e_{m,M}) denotes the measured loss vector of a model trained predominantly for objective mm. EA-SURF normalizes the measured distances into attained shares and compares them with the target shares q⋆(p)\mathbf{q}^\star(\mathbf{p}). This request maps to (qasr⋆,qts⋆,qst⋆)=(.3,.3,.4)(q^\star_{\mathrm{asr}},q^\star_{\mathrm{ts}},q^\star_{\mathrm{st}}) = (.3,.3,.4). Independent runs over a preference grid also yield multiple calibrated points for a posteriori use. Our contributions are:

  • C1) Preference realizability theory. Under regularity conditions, we show that every interior preference target can be matched by a unique scalarization weight, yielding a Pareto-optimal solution (Sec. 3.1).

  • C2) Closed-loop algorithm. EA-SURF adapts one warm-started probe using reusable specialists, with interior fixed points exactly matching the requested target shares (Sec. 3.2).

  • C3) Speech experimental results. Across six requests, EA-SURF improves preference alignment, lowers mean timestamp error from 46.6 to 39.2 ms, and raises normalized hypervolume by 17.8% over fixed LS. Table 2 reports the associated WER, AAS, and BLEU trade-offs (Sec. 4).

Request ppASR WER (%) LSASR WER (%) EAAAS (ms) LSAAS (ms) EABLEU LSBLEU EA
(.2,.2,.6)(.2, .2, .6)7.387.4954.341.832.8932.54
(.2,.4,.4)(.2, .4, .4)7.317.5943.035.231.9131.64
(.2,.6,.2)(.2, .6, .2)7.337.6238.035.829.5431.71
(.4,.2,.4)(.4, .2, .4)7.207.2450.938.832.7632.53
(.4,.4,.2)(.4, .4, .2)7.237.3942.637.030.0431.39
(.6,.2,.2)(.6, .2, .2)7.147.2151.046.631.4531.47
Mean Δ\Delta+0.16−7.4+0.45

Table 2. Three-objective results with 180k-step training budgets for both methods. EA denotes EA-SURF and Δ\Delta = EA-SURF − LS.

Notation. For M∈N+M \in\mathbb{N}_+, let [M]:=1,…,M[M] := 1,\ldots,M and ΔM:={v∈R+M:∑m=1Mvm=1}\Delta_M := \{\mathbf{v} \in\mathbb{R}_+^M : \sum_{m=1}^{M} v_m = 1\}. Bold lowercase letters denote vectors. ∥⋅∥1\lVert\cdot\rVert_1 and ∥⋅∥2\lVert\cdot\rVert_2 are the ℓ1\ell_1 and Euclidean norms. For a set AA, relint⁡(A)\operatorname{relint}(A) and ∂A\partial A denote its relative interior and boundary. Subscripts index objectives, parenthesized superscripts index outer rounds, and ⋆\star denotes optimal or target quantities.

Related Work

Unified and multi-task speech learning. Modern speech models increasingly place several capabilities in one shared architecture. SpeechT5 uses multi-task pretraining [3], Whisper uses large-scale multitask supervision [4], and SALMONN and WavLLM use cross-modal alignment and instruction tuning to broaden speech and audio interfaces [1, 2]. In-Sync adds timestamp prediction through task conditioning and timestamp regularization [5]. More explicitly, multi-objective speech work includes M2ASR, which optimizes multilingual ASR and ST under unsupervised-performance constraints [6], and Objective Soups, which combines conflict-aware gradients for multilingual ASR and ST [12]. These approaches focus on shared architectures, supervision, or joint optimization. EA-SURF instead uses attained specialist-relative shares to calibrate one fixed deployment preference during continued adaptation.

Multi-task and preference-aware optimization. Multi-task optimization also acts through controls distinct from a requested Pareto location. PCGrad modifies task gradients to reduce interference [13], whereas P1KE adapts task proportions from gradient interactions [14]. These methods respectively alter the loss update and data exposure. In our formulation, the sampler remains fixed while EA-SURF adapts scalarization weights from objective-space feedback. Fixed LS is therefore the main experimental control because EA-SURF retains the same weighted-loss, single-model training interface, whereas gradient-combination approaches require per-objective gradient calculations not used here. Preference-guided MOO provides complementary ways to state a desired compromise. EPO targets a loss-space ray [7], and FERERO encodes relative or absolute constraints [8]. For a posteriori selection, Pareto HyperNetworks learn preference-conditioned solution families [11], while SURF uses arc-length geometry to improve M=2M = 2 PF coverage [10]. EA-SURF instead calibrates the scalarization weights for one known request by comparing attained specialist-relative shares with the request-derived target along the same continuing probe.

EA-SURF Method

Preference-Steering Geometry

The deployment preference p\mathbf{p} specifies the requested trade-off. The scalarization weight w\mathbf{w} controls training, and the two need not coincide. We characterize the operating point attained by w\mathbf{w} through the specialist distances in (3). For ∑jdj(θ)>0\sum_{j}d_j(\theta) > 0, let qm(θ):=dm(θ)/∑j=1Mdj(θ)q_m(\theta) := d_m(\theta)/\sum_{j=1}^{M}d_j(\theta) and q(θ):=(q1(θ),…,qM(θ))\mathbf{q}(\theta) := (q_1(\theta),\ldots,q_M(\theta)). Thus, q(θ⋆(w))\mathbf{q}(\theta^\star(\mathbf{w})) records the attained position relative to the specialists. The target q⋆(p)\mathbf{q}^\star(\mathbf{p}) in (3) is the unique affine, permutation-equivariant, specialist-consistent mapping from preferences to distance shares. For a pure preference for objective mm, specialist consistency assigns zero share to its own specialist. Symmetry and simplex normalization assign 1/(M−1)1/(M-1) to the remaining specialists, and affinity gives q⋆(p)=(1−pm)/(M−1)\mathbf{q}^\star(\mathbf{p}) = (1-\mathbf{p}_m)/(M-1). Preference steering therefore seeks w⋆∈relint⁡(ΔM)\mathbf{w}^\star\in\operatorname{relint}(\Delta_M) such that q(θ⋆(w⋆))=q⋆(p)\mathbf{q}(\theta^\star(\mathbf{w}^\star)) = \mathbf{q}^\star(\mathbf{p}). The following assumption makes this correspondence well posed.

Assumption 1 (Regular preference matching). The objectives admit a hidden representation satisfying the hidden-convexity, smoothness, and PF nondegeneracy conditions in [9, 10]. The attained shares q(θ⋆(w))\mathbf{q}(\theta^\star(\mathbf{w})) are continuously differentiable in w∈relint⁡(ΔM)\mathbf{w} \in\operatorname{relint}(\Delta_M) with Jacobian rank (M−1)(M-1). For each interior preference considered, ∥q(θ⋆(w))−q⋆(p)∥22\lVert\mathbf{q}(\theta^\star(\mathbf{w}))-\mathbf{q}^\star(\mathbf{p})\rVert_2^2 is strictly convex on ΔM\Delta_M and attains its minimum in relint⁡(ΔM)\operatorname{relint}(\Delta_M).

Assumption 1 gives sufficient conditions for a unique preference match. Hidden convexity and PF regularity support scalarization, differentiability and full rank ensure that the attained preference responds locally to changes in scalarization weights, and strict convexity with interior attainment ensures uniqueness. Cross-entropy is convex in the output logits and, under a fixed linear output head, in the final hidden representation, despite being generally nonconvex in the full model parameters [10]. We empirically examine whether the resulting trajectories remain well behaved and whether multiple mismatch minima arise in Sec. 4.

Proposition 1 (Preference realizability). Under Assumption 1, every p∈relint⁡(ΔM)\mathbf{p} \in\operatorname{relint}(\Delta_M) under consideration admits a unique w⋆∈relint⁡(ΔM)\mathbf{w}^\star\in\operatorname{relint}(\Delta_M) such that q(θ⋆(w⋆))=q⋆(p)\mathbf{q}(\theta^\star(\mathbf{w}^\star)) = \mathbf{q}^\star(\mathbf{p}). Consequently, θ⋆(w⋆)\theta^\star(\mathbf{w}^\star) is Pareto optimal and satisfies dm(θ⋆(w⋆))∝(1−pm), ∀m∈[M]d_m(\theta^\star(\mathbf{w}^\star)) \propto(1-p_m), \ \forall m \in[M].

Proof. Assumption 1 gives a unique interior minimizer of the squared mismatch. First-order optimality makes its residual orthogonal to the directions generated by the attained-share Jacobian. Both share vectors lie in ΔM\Delta_M, so their residual has zero sum. The

Algorithm 1 EA-SURF for one deployment request
Input: Preference p, anchors $\mathbf{e}_{m=1}^{M}$, warm start $\theta^{(0)}$, gain $\alpha$, optimizer TrainStep, budgets $K_{k=0}^{T-1}$. Set $\mathbf{q}^{\star}(\mathbf{p})$ by (3) and $\mathbf{w}^{(0)} \leftarrow \mathbf{p}$
for $t = 0,\ldots,T-1$ do
    Measure $\ell(\theta^{(t)})$, compute $d_m(\theta^{(t)})$, and normalize to $\mathbf{q}(\theta^{(t)})$
    Form the current-to-target share ratios and update $\mathbf{w}^{(t+1)}$ by (4)
    Set $\theta^{(t,0)} \leftarrow \theta^{(t)}$
    for $k = 0,\ldots,K_t-1$ do
        $\theta^{(t,k+1)} \leftarrow \operatorname{TrainStep}(\theta^{(t,k)},\mathbf{w}^{(t+1)})$
    end for
    Set $\theta^{(t+1)} \leftarrow \theta^{(t,K_t)}$
end for
Output: Terminal probe $\theta^{(T)}$

rank-(M−1)(M-1) Jacobian spans this (M−1)(M-1)-dimensional simplex tangent space, which forces the residual to be zero. Strict convexity gives the unique matching weight. Hidden convexity and positive interior weights make the associated LS solution locally unique and Pareto optimal [9, 10].1

Proposition 1 establishes existence and uniqueness of the matching supported solution under Assumption 1. It does not establish convergence of the finite-horizon adaptive update.

EA-SURF: Adaptive Scalarization from Preference

Proposition 1 establishes a unique supported realization of the target shares. EA-SURF seeks this realization by adapting LS weights from observed share mismatch while continuing a single warm-started probe. At outer iteration tt, the preference calibration error (PCE), given by PCE⁡(t)(p):=∥q(θ(t))−q⋆(p)∥1\operatorname{PCE}^{(t)}(\mathbf{p}) := \lVert\mathbf{q}(\theta^{(t)})-\mathbf{q}^{\star}(\mathbf{p})\rVert_1, measures the mismatch between attained and target shares. Starting from t=0t=0 with w(0)=p\mathbf{w}^{(0)}=\mathbf{p}, EA-SURF forms rm(t):=(qm(θ(t))+εU)/(qm⋆(p)+εU)r_m^{(t)} := (q_m(\theta^{(t)})+\varepsilon_U)/(q_m^{\star}(\mathbf{p})+\varepsilon_U) for m∈[M]m \in[M] and applies the exponentiated, simplex-preserving update [15]

wm(t+1):=wm(t)(rm(t))α∑jwj(t)(rj(t))α,(4)w_m^{(t+1)} := \frac{w_m^{(t)}(r_m^{(t)})^{\alpha}}{\sum_j w_j^{(t)}(r_j^{(t)})^{\alpha}}, \tag*{(4)}

where 0<α≤10 < \alpha\le1 and εU>0\varepsilon_U > 0 keeps the ratios finite. When qm(θ(t))/qm⋆(p)>1q_m(\theta^{(t)})/q_m^{\star}(\mathbf{p}) > 1, the probe is relatively too far from specialist mm. EA-SURF therefore increases the relative LS weight on objective mm, with the intended effect of moving the next continuation closer to that specialist. The same probe then trains for KtK_t steps under the updated weights, where TrainStep(θ,w)(\theta,\mathbf{w}) denotes one optimizer update on the weighted objective LLS(θ,w)L_{\mathrm{LS}}(\theta,\mathbf{w}) in (2). Algorithm 1 summarizes this closed-loop procedure. Each request uses one warm-started probe and reuses the MM specialist anchors, avoiding neighboring simplex probes or an explicit PF mesh. Computing MM anchor distances and updating MM weights adds negligible overhead relative to the KtK_t training steps.

Figure 2 illustrates the feedback dynamics: measured shares move toward their targets and imbalance ratios move toward one, while LS weights depart substantially from the original preference p\mathbf{p}. Thus, matching the requested trade-off can require training weights different from the preference itself. The next result characterizes the update’s fixed point.

EA-SURF dynamics

Figure 2. EA-SURF dynamics for p=(.4,.4,.2)\mathbf{p} = (.4,.4,.2). Left: attained shares in the simplex. Upper right: current-to-target ratios. Lower right: the resulting adaptive LS weights. The gold star marks q⋆(p)=(.3,.3,.4)\mathbf{q}^{\star}(\mathbf{p}) = (.3,.3,.4), and the dashed line marks ratio one.

Proposition 2 (Target consistency). For w(t)∈relint⁡(ΔM)\mathbf{w}^{(t)} \in\operatorname{relint}(\Delta_M) and α>0\alpha> 0, the update in (4) satisfies w(t+1)=w(t)\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)} if and only if q(θ(t))=q⋆(p)\mathbf{q}(\theta^{(t)})=\mathbf{q}^{\star}(\mathbf{p}).

Proof. At an interior fixed point, all (rm(t))α(r_m^{(t)})^\alpha, and hence all rm(t)r_m^{(t)}, equal some c>0c>0. Therefore qm(θ(t))+εU=c(qm⋆(p)+εU)q_m(\theta^{(t)})+\varepsilon_U=c(q_m^{\star}(\mathbf{p})+\varepsilon_U). Summing over mm and using ∑mqm(θ(t))=∑mqm⋆(p)=1\sum_m q_m(\theta^{(t)})=\sum_m q_m^{\star}(\mathbf{p})=1 yields c=1c=1. The converse follows immediately from (4). □\square

Thus, the desired preference match is exactly the interior stationary condition of the weight update. This target-consistency result is not a convergence theorem. Finite-horizon behavior is assessed through the trajectories in Sec. 4.

EXPERIMENTS

Speech LLM, tasks, and evaluation

Configuration and model. We use the SLAM-LLM speech interface [16] adopted by Granite-Speech [17] and In-Sync [5], with a frozen ConFC-TC encoder, a trainable Q-Former speech adapter [18], and a frozen Granite-4.0-1B-Base causal LLM. Audio is sampled at 16 kHz, capped at 60 s, and converted to 80-bin log-Mel features. We train the speech adapter and rank-64 LoRA modules [19], with scaling 32 and dropout 0.1, for 81M trainable parameters. Training uses four NVIDIA H100 GPUs with a total batch size of 64.

Training and validation data. ASR training combines Common Voice and LibriSpeech [20, 21], VoxPopuli, Earnings-22, the AMI Meeting Corpus (AMI), CALLHOME, Switchboard, Fisher, and additional voicemail, multispeaker, and synthetic speech. TS uses forced-aligned subsets of these corpora. We discard samples whose alignment score between CTC forced alignment and GMM-HMM alignment from the Montreal Forced Aligner exceeds 175 ms [22]. Following In-Sync [5], targets interleave words with centisecond end times. ST uses nine Common Voice-derived translation sets spanning English-to-multilingual and multilingual-to-English directions. Validation draws from Multilingual LibriSpeech (MLS), Common Voice, LibriSpeech, VoxPopuli, AMI, and Earnings-22.

Evaluation. We evaluate ASR on seven held-out English conditions and TS on nine aligned conditions, including manually timestamped TIMIT [23] and Buckeye [24]. TIMIT and Buckeye are excluded from training and therefore provide out-of-domain timestamp evaluations. ST uses thirteen directions from CoVoST 2 and FLEURS [25, 26]. We report unweighted macro-averages using word error rate (WER), accumulated averaging shift (AAS) [27], and bilingual evaluation understudy (BLEU) computed with SacreBLEU [28]. MOO performance is measured by PCE and endpoint-inclusive normalised hypervolume (HV) [29]. PCE is an internal preference-alignment metric because it uses the same training-objective share mismatch that drives EA-SURF. WER, AAS, BLEU, and HV assess held-out outcomes or trade-off coverage. For HV, we minimize (WERASR,AAS,−BLEU)(\mathrm{WER}_{\mathrm{ASR}}, \mathrm{AAS}, -\mathrm{BLEU}), normalize each coordinate by the ideal and nadir of the three shared anchors, and use the fixed reference point (1,1,1)(1, 1, 1). Lower WER, AAS, and PCE and higher BLEU and HV are better. All table entries are single runs. Reported means and population standard deviations aggregate the six requests and are not uncertainty estimates over independent reruns.

PreferenceFixed LS WER (%) ↓Fixed LS AAS (ms) ↓EA-SURF WER (%) ↓EA-SURF AAS (ms) ↓
(.25, .75)7.3732.87.5230.6
(.50, .50)7.2635.97.3434.5
(.75, .25)7.2441.77.3040.1

Table 1. Bi-objective results from random initialization. The first two rows are fixed-LS specialists reused as EA-SURF anchors. The remaining rows compare the interior preferences.

Baselines and implementation

Inner training and sampling. We estimate each task objective by averaging token-level cross-entropy over its active non-padding tokens, then combine tasks using (2). Each minibatch draws at least one example from each active task’s index pool with dataset-balance exponent 0.6. Zero-weight tasks are removed from the pool and loss when training specialist anchors. Inner training uses AdamW at 10−410^{-4} with zero weight decay, 1000 warm-up steps, gradient clipping at 10, and one gradient-accumulation step.

Scalarization controls. Fixed LS holds ww constant in (2). As a two-objective Tchebysheff diagnostic [9], we use the fixed utopia reference zm∗=0z_m^* = 0, select the task with the largest weighted gap wm(ℓm(θ)−zm∗)w_m(\ell_m(\theta) - z_m^*) at each step, differentiate its unweighted gap, and add the augmentation ρ∑m(ℓm(θ)−zm∗)\rho\sum_m(\ell_m(\theta) - z_m^*) with ρ=10−3\rho= 10^{-3}.

Preference steering. EA-SURF uses Algorithm 1 with Euclidean distances in (3), α=1\alpha= 1, and εU=10−6\varepsilon_U = 10^{-6}. Each outer measurement averages task losses over 300 no-gradient training iterations per distributed rank, then averages across ranks to compute q(θt)q(\theta^t). Held-out metrics are evaluated only after round TT. EA-SURF shares the sampler and optimizer family of fixed LS.

Bi-objective steering diagnostic

Table 1 provides a bi-objective ASR–timestamp diagnostic from random initialization rather than the primary compute-matched benchmark. Fixed LS trains for 110k steps. EA-SURF uses a 50k-step probe followed by T=8T = 8 rounds with K0=⋯=K7=10,000K_0 = \cdots= K_7 = 10{,}000, and reuses two 110k-step anchors across requests. Across the three interior preferences, EA-SURF spans 0.22 ASR-WER points and 9.5 ms AAS, versus 0.13 points and 8.9 ms for fixed LS. EA-SURF’s training losses (ℓasr,ℓts)(\ell_{\mathrm{asr}}, \ell_{\mathrm{ts}}) progress from (.331,.845)(.331, .845) through (.279,.862)(.279, .862) to (.274,.923)(.274, .923), confirming movement along the trade-off. A separate pretrained-model diagnostic at p=(.25,.75)p = (.25, .75) compares LS with active-gap Tchebysheff: fixed LS improves ASR WER by 0.23 points and AAS by 0.6 ms. Aside from its small augmentation, the active-gap method differentiates only the largest task gap, using less multi-task gradient information per step. The average best PCE across the three EA-SURF trajectories is 0.0250, a 59.6% improvement over fixed LS at 0.0618.

Three-objective preference steering

Fixed LS uses one 180k-step run for each request. Each EA-SURF run starts from its 100k-step LS checkpoint and uses T=8T = 8 with K0=⋯=K7=10,000K_0 = \cdots= K_7 = 10{,}000, for the same nominal 180k request-level budget. There separately trained 80k-step specialists with permutations of (.98,.01,.01)(.98, .01, .01) are reused across requests. We report the pre-specified terminal θ(T)\theta^{(T)}. The EA-SURF continuations are launched separately, so their learning-rate schedule histories differ from those of the single 180k-step fixed-LS runs.

Across the six requests in Table 2, fixed LS with w=pw = p spans 7.14–7.38% ASR WER, 38.0–54.3 ms AAS, and 29.54–32.89 BLEU. Greater timestamp emphasis improves AAS, while greater translation emphasis improves BLEU, revealing their trade-off over a stable ASR range. EA-SURF reduces AAS for every request by 2.2–12.5 ms and lowers its mean from 46.6 to 39.2 ms, while increasing mean ASR WER by 0.16 points and mean BLEU by 0.45. EA-SURF is not designed to pointwise dominate fixed LS, but to reach a different operating point specified by specialist-relative training-objective shares, so moving toward the target can improve one held-out metric while worsening another. A larger pmp_m likewise need not yield monotonic improvement in the corresponding downstream metric. Across all requests, the scalarization-to-share trajectories show no observed folding and each has a unique interior mismatch minimum, empirically supporting Assumption 1.

As an internal steering metric, mean PCE decreases from .240 for fixed LS to .091 for EA-SURF, with population standard deviations .119 and .038, a 62.2% reduction. For the endpoint-inclusive HV defined above, each six-request set is augmented with the shared anchors (7.15,60.8,29.08)(7.15, 60.8, 29.08), (7.51,33.4,27.96)(7.51, 33.4, 27.96), and (7.79,62.1,33.68)(7.79, 62.1, 33.68) in (WERASR,AAS,BLEU)(\mathrm{WER}_{\mathrm{ASR}}, \mathrm{AAS}, \mathrm{BLEU}) order. Normalized HV increases from .519 to .612, or 17.8%, indicating improved endpoint-inclusive trade-off coverage.

Conclusion

In this paper, we propose EA-SURF, which steers multi-task speech LLMs without directly using the deployment preference as linear-scalarization weights. To achieve this separation, it uses one reusable anchor model per task, each trained predominantly for that task. Specifically, EA-SURF converts the preference into target proportions of the Euclidean distances between the current model’s training-loss vector and the anchor models’ loss vectors, then updates the weights based on the measured mismatch along a warm-started training trajectory. Under the stated regularity assumptions, each interior target has a unique supported realization, and the weight update is stationary exactly when the measured and target shares coincide. Empirically, across six three-objective requests, EA-SURF improves preference alignment, lowers mean AAS by 7.4 ms, and raises endpoint-inclusive normalized HV by 17.8% over fixed LS. These results of EA-SURF demonstrate improved preference calibration and trade-off coverage compared to baselines.

References

  1. [1]Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, “Salmonn: Towards generic hearing abilities for large language models,” in International Conference on Learning Representations, 2024, vol. 2024, pp. 16607–16629.arxiv.org/abs/2310.13289
  2. [2]Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al., “Wavllm: Towards robust and adaptive speech large language model,” in Findings of the Association for Computational Linguistics: EMNLP, 2024, pp. 4552–4572.arxiv.org/abs/2404.00656
  3. [3]Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al., “Speech5: Unified-model encoder-decoder pre-training for spoken language processing,” in Proc. of the 60th annual meeting of the association for computational linguistics, 2022.
  4. [4]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavy, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning, 2023, pp. 28492–28518.arxiv.org/abs/2212.04356
  5. [5]Xulin Fan, Vishal Sunder, Samuel Thomas, Mark Hasegawa-Johnson, Brian Kingsbury, and George Saon, “In-sync: Adaptation of speech aware large language models for asr with word level timestamp predictions,” in International Conference on Acoustics, Speech and Signal Processing. IEEE, 2026.
  6. [6]AFM Saif, Lisha Chen, Xiaodong Cui, Songtao Lu, Brian Kingsbury, and Tianyi Chen, “M2asr: Multilingual multi-task automatic speech recognition via multi-objective optimization,” in Proc. Interspeech 2024, 2024, pp. 1240–1244.
  7. [7]Debabrata Mahapatra and Vaibhav Rajan, “Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization,” in International Conference on Machine Learning. PMLR, 2020, pp. 6597–6607.
  8. [8]Lisha Chen, AFM Saif, Yanning Shen, and Tianyi Chen, “Ferero: A flexible framework for preference-guided multi-objective learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 18758–18805, 2024.arxiv.org/abs/2412.01773
  9. [9]Kaisa Miettinen et al., Nonlinear multiobjective optimization, vol. 12, Kluwer Academic Publishers Boston, 1999.
  10. [10]Liyuany Jiang, Chentong Huang, and Lisha Chen, “Surf: Steering the scalarization weight to uniformly traverse the pareto front,” arXiv preprint arXiv:2605.20619, 2026.arxiv.org/abs/2605.20619
  11. [11]Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik, “Learning the pareto front with hypernetworks,” in International Conference on Learning Representations, 2021.arxiv.org/abs/2010.04104
  12. [12]A F M Saif, Lisha Chen, Xiaodong Cui, Songtao Lu, Brian Kingsbury, and Tianyi Chen, “Objective sups: Multilingual multi-task modeling for speech processing,” in Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2508.09228
  13. [13]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn, “Gradient surgery for multi-task learning,” Advances in neural information processing systems, vol. 33, pp. 5824–5836, 2020.arxiv.org/abs/2001.06782
  14. [14]Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviayn, and Vahab Mirrokni, “PiKE: Adaptive data mixing for large-scale multi-task learning under low gradient conflicts,” in Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2502.06244
  15. [15]Jyrki Kivinen and Manfred K Warmuth, “Exponentiated gradient versus gradient descent for linear predictors,” Information and computation, vol. 132, no. 1, pp. 1–63, 1997.DOI
  16. [16]Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al., “An embarrassingly simple approach for llm with strong asr capacity,” arXiv preprint arXiv:2402.08846, 2024.arxiv.org/abs/2402.08846
  17. [17]George Saon, Avihu Dekel, Alexander Brooks, Tohru Nagano, Abraham Daniels, Aharon Satt, Ashish Mittal, Brian Kingsbury, David Haws, Edmilson Morais, et al., “Granite-speech: open-source speech-aware llms with strong english asr capabilities,” in 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2025, pp. 1–7.arxiv.org/abs/2505.08699
  18. [18]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PmLR, 2023, pp. 19730–19742.arxiv.org/abs/2301.12597
  19. [19]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  20. [20]Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Hennerty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber, “Common voice: A massively-multilingual speech corpus,” in Proc. of the twelfth language resources and evaluation conference, 2020.arxiv.org/abs/1912.06670
  21. [21]Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in International Conference on Acoustics, Speech and Signal Processing. IEEE, 2015.DOI
  22. [22]Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Proc. Interspeech 2017, 2017, pp. 498–502.DOI
  23. [23]John S Garofolo, Lori F Lamel, William M Fisher, David S Pallett, Nancy L Dahlgren, Victor Zue, and Jonathan G Fiscus, “Timit acoustic-phonetic continuous speech corpus,” 1993.
  24. [24]Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond, “The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability,” Speech Communication, vol. 45, no. 1, pp. 89–95, 2005.
  25. [25]Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino, “Covost 2 and massively multilingual speech translation,” in Proc. Interspeech 2021, 2021, pp. 2247–2251.DOI
  26. [26]Alexis Conneau, Maïa Maamari, Xian Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in Spoken Language Technology Workshop (SLT). IEEE, 2022, pp. 798–805.arxiv.org/abs/2205.12446
  27. [27]Xian Shi, Yanni Chen, Shiliang Zhang, and Zhijie Yan, “Achieving timestamp prediction while recognizing with non-autoregressive end-to-end asr model,” in National Conference on Man-Machine Speech Communication. Springer, 2022.arxiv.org/abs/2301.12343
  28. [28]Matt Post, “A call for clarity in reporting bleu scores,” in Proc. of the third conference on machine translation, 2018.arxiv.org/abs/1804.08771
  29. [29]Eckart Zitzler and Lothar Thiele, “Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach,” IEEE transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257–271, 1999.DOI

Paper details

vlnc:W_mrd7x2cb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents