betaFeedback

WorksComputer Science

One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?

Abstract

Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.

Introduction

Robotic and embodied agents are increasingly expected to operate under changing user preferences and behavioral constraints. A household robot may need to prioritize speed in one setting, energy efficiency in another, or smooth and safe motion around humans. Rather than training separate policies for every trade-off, an ideal system should support steerable control, where behavior can be adjusted at runtime through a compact and interpretable preference interface.

Multi-objective reinforcement learning (MORL) provides a natural framework for this problem by representing behavior through multiple reward dimensions and conditioning a policy on a preference vector which specifies desired objective trade-offs [3, 13]. This enables one policy to expose a continuous control interface at test time, but conditioning alone does not guarantee steerability. Increasing one preference may produce little change in the intended behavior, produce unintended behaviors, or compromise the essential tasks that the robot must continue to perform.

This distinction is not captured by aggregate return or Pareto-coverage measures alone [de las Heras Molins et al., 2026]. A useful evaluation should answer three questions: Does the policy respond strongly when a preference changes? Does the change unintentionally affect other behaviors? Does the policy still perform the task reliably without more failures? We therefore formulate preference-conditioned steerability as a multidimensional evaluation problem and introduce a unified suite based on the Preference Reward Responsiveness (PRR) matrix, Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy. These metrics complement standard utility measures by exposing both intended responsiveness and cross-objective leakage [de las Heras Molins et al., 2026].

Preprint.

pWe compare reward, advantage, and loss-level scalarization within PPO [16] across two independently implemented humanoid-control systems, ProtoMotions [18] and MimicKit [14], to test whether the findings generalize across implementations. Within ProtoMotions, we additionally test PCGrad-based gradient surgery [19] as a later aggregation point that resolves objective conflicts directly in gradient space. Reward-level aggregation achieves the widest controllable range in ProtoMotions, whereas later aggregation achieves higher hypervolume in fixed-direction MimicKit. Its weaker cross-objective precision persists across both frameworks, but these experiments do not isolate the cause of their different coverage rankings. A preliminary check in MetaDrive [11], a driving domain, shows a broadly similar pattern.

Our contributions are:

  1. We formulate preference-conditioned robot steerability as a multidimensional evaluation problem that separates responsiveness, monotonicity, selectivity, task preservation, and failure behavior.

  2. We present a framework that trains preference-conditioned policies under different PPO-based aggregation strategies and evaluates their steerability using the suite above, validated across two independently implemented humanoid-control systems.

  3. We show that the payoff from aggregating preferences early depends on the experimental setting: coverage and utility rankings differ across frameworks, while later aggregation consistently yields higher selectivity and lower cross-objective leakage.

Code is available at: https://anonymous.4open.science/r/Pref-Align-RL-214C/.

Background and Related work

Preference-conditioned MORL trains a single policy over preference vectors so that it can represent multiple objective trade-offs at runtime [4, 1]. Recent work applies this to humanoid control through a mixture-of-experts architecture [10], but fixes a single aggregation point and does not measure cross-objective leakage. We instead study how the aggregation point affects whether changing a policy’s preference input predictably changes its behavior.

In the LLM literature, Constitutional AI shows how written principles can guide behavior through self-critique and AI feedback [2]. The constitution shapes training rather than serving as a continuous runtime control. SteerLM enables runtime steering by conditioning generation on desired attribute values, while multi-objective alignment methods combine rewards or model predictions to trade off properties such as helpfulness, safety, and coding ability [6, 17] (Wang et al. 2024). Recent methods extend this idea to fine-grained value priorities and efficient adaptation across a Pareto frontier of user preferences [8, 12]. Together, these works motivate an explicit interface for controlling behavior.

The level at which objective signals are combined during training can affect how well each preference controls behavior. The level at which objective signals are combined during training can affect how well each preference controls behavior. Gradient-surgery methods such as PCGrad address objective conflicts by modifying per-objective gradients before they are combined [19]. Preference-conditioned MORL is usually evaluated with Pareto coverage and utility, which do not directly separate intended response from cross-objective leakage. Recent controllability work begins to expose this limitation [de las Heras Molins et al., 2026]. We extend this perspective to embodied control by comparing reward, advantage, and loss-level combination and evaluating responsiveness, monotonicity, selectivity, task preservation, and failure behavior across two robot-control settings.

Preliminary

Preference-Conditioned Multi-Objective Reinforcement Learning

We model the MORL setting as a multi-objective Markov decision process with vector reward

rt=[rt(1),…,rt(K)]∈RK,\mathbf{r}_t = \left[r_t^{(1)}, \ldots, r_t^{(K)}\right] \in\mathbb{R}^{K},

where each dimension corresponds to one objective [9, 7]. A preference vector w∈ΔK−1w \in\Delta^{K-1} specifies the desired trade-off among these objectives. A common scalar utility is the linear scalarization

rtscalar=wTrt.r^{\mathrm{scalar}}_{t} = \mathbf{w}^{\mathsf{T}}\mathbf{r}_{t}.

In preference-conditioned MORL, the goal is to train one policy

πθ(a∣s,w)\pi_{\theta}(a \mid s, \mathbf{w})

that can be steered by changing w\mathbf{w} at test time. The same policy parameters are shared across preferences, while the input w\mathbf{w} specifies the desired behavior.

Because objectives can conflict, MORL evaluates policies by the trade-offs they recover rather than by a single reward value. A solution is Pareto-dominated if another solution is at least as good in every objective and better in at least one. The Pareto front is the set of non-dominated solutions. In our experiments, we report hypervolume (HV) and expected utility (EU), which measure Pareto-front quality and average utility under sampled preferences, respectively [9, 7]; formal definitions and our estimation procedure are given in Appendix § 8.

Method

Aggregation Strategies

All four methods share the same actor-critic architecture, observation space, and per-objective reward vector rt=[rt(0),rt(1),…,rt(K)]\mathbf{r}_{t} = [r_{t}^{(0)}, r_{t}^{(1)}, \ldots, r_{t}^{(K)}], where rt(0)r_{t}^{(0)} is the fixed-weight task reward and rt(1),…,rt(K)r_{t}^{(1)}, \ldots, r_{t}^{(K)} are the KK preference-controlled style rewards. We write the combined per-step coefficient for objective kk as

ck(w)={wtaskk=0wprefwkk≥1,c_{k}(\mathbf{w}) = \begin{cases} w_{\mathrm{task}} & k = 0 \\ w_{\mathrm{pref}} w_{k} & k \ge1, \end{cases}

where wtaskw_{\mathrm{task}} is fixed and w=(w1,…,wK)∈ΔK−1\mathbf{w} = (w_{1}, \ldots, w_{K}) \in\Delta^{K-1} is the sampled preference. Let ρt=πθ(at∣st,w)/πθold(at∣st,w)\rho_{t} = \pi_{\theta}(a_{t} \mid s_{t}, \mathbf{w}) / \pi_{\theta_{\mathrm{old}}}(a_{t} \mid s_{t}, \mathbf{w}) denote the importance ratio and Lclip(A)=min⁡(ρtA,clip⁡(ρt,1−ϵ,1+ϵ)A)L^{\mathrm{clip}}(A) = \min\left(\rho_{t}A, \operatorname{clip}(\rho_{t}, 1-\epsilon, 1+\epsilon)A\right) the standard PPO [16] clipped surrogate for a scalar advantage AA. All critics are also conditioned on w\mathbf{w}.

The four methods differ only in how late the K+1K+1 objectives are collapsed into the single scalar signal that ultimately updates the actor, forming a continuum from earliest to latest aggregation.

Reward-level. Objectives are combined at the reward itself, before any advantage estimation:

rt=∑k=0Kck(w)rt(k).r_{t} = \sum_{k=0}^{K} c_{k}(\mathbf{w})r_{t}^{(k)}.

A single scalar critic Vϕ(s,w)V_{\phi}(s, \mathbf{w}) is trained on this scalarized reward, standard GAE [15] produces one advantage AtA_{t}, and the actor is updated with Lclip(At)L^{\mathrm{clip}}(A_{t}). No per-objective structure survives past the reward.

Advantage-level. Per-objective critics Vϕ(k)(s,w)V_{\phi}^{(k)}(s, \mathbf{w}) and per-objective GAE are retained, giving K+1K+1 advantages At(0),…,At(K)A_{t}^{(0)}, \ldots, A_{t}^{(K)}. These are combined before clipping:

Atw=∑k=0Kck(w)At(k),A_{t}^{\mathbf{w}} = \sum_{k=0}^{K} c_{k}(\mathbf{w})A_{t}^{(k)},

and a single clipped surrogate Lclip(Atw)L^{\mathrm{clip}}(A_{t}^{\mathbf{w}}) is applied to the combined advantage.

Loss-level. Per-objective critics Vϕ(k)(s,w)V_{\phi}^{(k)}(\mathbf{s},\mathbf{w}) and advantages are retained as above, but each objective is clipped separately before combination:

Lloss=∑k=0Kck(w)Lclip(At(k)).L^{\mathrm{loss}}=\sum_{k=0}^{K}c_k(\mathbf{w})L^{\mathrm{clip}}\left(A_t^{(k)}\right).

This is the latest point at which the K+1K+1 objectives are still combined by a simple weighted sum. When these gradients conflict, we additionally consider replacing this weighted sum with gradient-level conflict resolution via PCGrad [19], detailed in Appendix § 9.

Steerability Evaluation Suite

Aggregate return and Pareto-front coverage alone cannot distinguish a policy that responds correctly to a preference from one that responds diffusely or not at all [de las Heras Molins et al., 2026]. We therefore introduce a complementary suite that isolates, per preference dimension, how strongly the intended objective responds, how much unintended objectives are disturbed, and whether that response is directionally reliable. This adapts rank-based controllability measures previously used on abstract, low-dimensional MORL benchmarks [de las Heras Molins et al., 2026] to embodied, physically simulated continuous control.

Axis sweep. For each style dimension k∈{1,…,K}k \in\{1,\ldots,K\}, we hold a reference preference w0\mathbf{w}^{0} fixed in every coordinate but kk and sweep α∈[αmin⁡,αmax⁡]\alpha\in[\alpha_{\min},\alpha_{\max}] across MM evenly spaced values:

wk(i)=αiek+(1−αi)w−k0∑w−k0,i=1,…,M,\mathbf{w}_{k}^{(i)}=\alpha_i\mathbf{e}_k+(1-\alpha_i)\frac{\mathbf{w}_{-k}^{0}}{\sum\mathbf{w}_{-k}^{0}},\qquad i=1,\ldots,M,

where ek\mathbf{e}_k is the kk-th one-hot vector and w−k0\mathbf{w}_{-k}^{0} is w0\mathbf{w}^{0} with its kk-th entry zeroed. Each policy is evaluated at every wk(i)\mathbf{w}_{k}^{(i)}, yielding an achieved objective vector J(wk(i))J(\mathbf{w}_{k}^{(i)}) (defined precisely in Appendix § 10.§ 2).

Self-PRR and Cross-PRR. The Preference-Reward Responsiveness (PRR) matrix is the finite-difference slope of objective jj along the sweep of dimension kk:

PRR⁡[j][k]=1M−1∑i=1M−1Jj(wk(i+1))−Jj(wk(i))αi+1−αi.\operatorname{PRR}[j][k]=\frac{1}{M-1}\sum_{i=1}^{M-1}\frac{J_j\left(\mathbf{w}_{k}^{(i+1)}\right)-J_j\left(\mathbf{w}_{k}^{(i)}\right)}{\alpha_{i+1}-\alpha_i}.

Self-PRR⁡k=PRR⁡[k][k]\operatorname{Self\text{-}PRR}_k=\operatorname{PRR}[k][k] measures how strongly dimension kk responds to its own preference weight. Cross-PRR⁡k=1K−1∑j≠k∣PRR⁡[j][k]∣\operatorname{Cross\text{-}PRR}_k=\frac{1}{K-1}\sum_{j\ne k}\left|\operatorname{PRR}[j][k]\right| measures how much that same change disturbs the other K−1K-1 objectives.

Selectivity. Selectivity summarizes the ratio of intended to unintended response across all KK dimensions:

Selectivity⁡=Self-PRR⁡∣Cross-PRR⁡∣.\operatorname{Selectivity}=\frac{\operatorname{Self\text{-}PRR}}{\left|\operatorname{Cross\text{-}PRR}\right|}.

Selectivity >1>1 indicates a preference change moves its own objective more than it disturbs the others on average.

Rank Accuracy. Self-PRR and Cross-PRR are sensitive to noise in the finite-difference slope. Rank Accuracy instead asks a coarser, more robust question: does the ordering of preference weights along a sweep predict the ordering of achieved reward? For each dimension kk, we compute the fraction of sweep-point pairs (a,b)(a,b) for which this holds:

RankAcc⁡k=1(M2)∑a<b1[(αb−αa)(Jk(wk(b))−Jk(wk(a)))>0].\operatorname{RankAcc}_k=\frac{1}{\binom{M}{2}}\sum_{a<b}\mathbf{1}\left[(\alpha_b-\alpha_a)\left(J_k\left(\mathbf{w}_k^{(b)}\right)-J_k\left(\mathbf{w}_k^{(a)}\right)\right)>0\right].

Together, Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy answer the three questions posed in § 1: whether a preference change produces the intended response, whether it leaks into unrelated behaviors, and whether that response is directionally reliable.

Experiments

ProtoMotions: G1 Humanoid Preference-Conditioned Locomotion

Setup. We evaluate four aggregation strategies, reward-level, advantage-level, loss-level, and PCGrad (§ 4.1), on a G1 humanoid trained to walk under a K=5K=5-dimensional style preference (energy, arm swing, foot clearance, step width, smoothness) alongside a fixed-weight heading-tracking task reward. Preferences are sampled from Dirichlet(0.5) during training. Each method is trained for 1.2×1091.2 \times10^{9} environment steps with three independent seeds; full task and reward definitions are given in Appendix § 10.§ 1.

Evaluation protocol. We measure steering ability (§ 4.2) along the five axes from § 1: responsiveness (Hypervolume, Expected Utility, Self-PRR), monotonicity (Rank Accuracy), selectivity (Selectivity, Cross-PRR), task preservation (mean task reward JtaskJ_{\mathrm{task}} for tracking the fixed heading and speed target), and failure behavior (fall rate). These are computed from a per-dimension preference axis sweep together with N=50N=50 preferences sampled from the training-time Dirichlet(0.5) distribution, under a fixed-horizon convention that zeroes reward after a fall. Full metric definitions and evaluation hyperparameters are given in Appendix § 10.§ 2.

Main results. Table 1 reports every metric as mean ±\pm std over three training seeds. No single method dominates. Reward-level attains roughly 3×3\times the hypervolume, 2×2\times the self-responsiveness, and roughly 2×2\times the JtaskJ_{\mathrm{task}} of the other three methods, with error bars that do not overlap across seeds. It is, however, the weakest of the four on precision, with the lowest selectivity and the highest cross-dimension leakage. Advantage-level shows the opposite profile: the smallest controllable region but the cleanest separation between preference dimensions. Loss-level and PCGrad fall between the two on both axes. We hypothesize this pattern reflects how much per-objective structure survives before aggregation. Reward-level collapses all KK objectives into a single scalar reward before the critic or the advantage estimator ever sees them, leaving no mechanism downstream that could prevent one objective’s gradient from dominating or leaking into the others. Advantage-level, loss-level, and PCGrad instead keep per-objective critics and advantages further into the update (§ 4.1), which appears to trade raw responsiveness for cleaner separation. Aggregating early thus favors a wide controllable range at the cost of precision, while aggregating late favors precise, disentangled control at the cost of range.

Controllability analysis. Figure 1 plots the per-dimension sweep curves underlying Self-PRR: the normalized reward JkJ_k achieved for style dimension kk as its preference weight increases from 0.05 to 0.95, averaged over the three seeds. Reward-level’s curves sit above and saturate earlier than the other three methods on four of the five style dimensions, consistent with its higher Self-PRR in Table 1. The gap is not uniform across dimensions: on energy, all four methods converge by α≈0.4\alpha\approx0.4, suggesting the trade-off we report is driven more strongly by some style dimensions than others.

MethodHypervolume↑\uparrowExp. Utility↑\uparrowSelf-PRR↑\uparrowRank acc.↑\uparrowSelectivity↑\uparrowCross-PRR↓\downarrowJtaskJ_{\mathrm{task}} ↑\uparrowFall rate↓\downarrow
Reward-level0.316±.016\mathbf{0.316}\pm\mathbf{.016}0.763±.007\mathbf{0.763}\pm\mathbf{.007}0.454±.004\mathbf{0.454}\pm\mathbf{.004}0.847±.018\mathbf{0.847}\pm\mathbf{.018}1.61±.111.61\pm.110.285±.0190.285\pm.0190.761±.014\mathbf{0.761}\pm\mathbf{.014}7.4±.1%7.4\pm.1\%
Advantage-level0.109±.0110.109\pm.0110.568±.0140.568\pm.0140.232±.045\mathbf{0.232}\pm.0450.811±.0010.811\pm.0012.16±.15\mathbf{2.16}\pm.150.108±.013\mathbf{0.108}\pm.0130.409±.0190.409\pm.0198.0±.3%8.0\pm.3\%
Loss-level0.103±.0070.103\pm.0070.572±.0100.572\pm.0100.225±.0040.225\pm.0040.792±.0330.792\pm.0332.04±.062.04\pm.060.112±.0060.112\pm.0060.414±.0170.414\pm.0177.9±.1%7.9\pm.1\%
PCGrad0.104±.0160.104\pm.0160.558±.0290.558\pm.0290.233±.0040.233\pm.0040.800±.0470.800\pm.0472.07±.172.07\pm.170.114±.0080.114\pm.0080.393±.0270.393\pm.0278.4±.1%8.4\pm.1\%

Table 1. Main comparison on ProtoMotions (G1 humanoid), mean ±\pm std over 3 training seeds at 1.2B environment steps. Column groups correspond to the five evaluation axes in § 1: responsiveness, monotonicity, selectivity, task preservation, and failure behavior. Bold marks the best value per column; ↑/↓\uparrow/\downarrow indicate the preferred direction.

Figure 1. Per-dimension preference sweep. Each panel shows the normalized reward achieved for one style dimension as its preference weight increases, mean ±\pm std shading over 3 training seeds.

MimicKit: Cross-Framework Validation

Setup and evaluation. To test cross-framework consistency under a fixed movement direction, we repeat reward-, advantage-, and loss-level aggregation in MimicKit using its native optimization and reward processing. The humanoid tracks target speeds sampled from [0.5,2.0][0.5, 2.0] m/s along a fixed +x+x direction, without a facing reward. The five style objectives and episode-level Dirichlet(0.5) sampling match the preceding experiment; task and AMP rewards retain fixed coefficients outside the preference vector. Each method is trained for 1.2×1091.2 \times10^{9} environment steps with three seeds. We reuse the fixed-horizon protocol with 15-point axis sweeps, N=50N=50 sampled preferences, 300-step rollouts, and three deterministic evaluation seeds. Table 2 reports mean ±\pm std across training seeds; Appendix § 11 details the setup and native fall criterion.

MethodHypervolume↑\uparrowExp. Utility↑\uparrowSelf-PRR↑\uparrowRank acc.↑\uparrowSelectivity↑\uparrowCross-PRR↓\downarrowJtask↑J_{\mathrm{task}}\uparrowFall rate↓\downarrow
Reward-level0.238±\pm.0290.769±\pm.0090.510±\pm.0210.924±\pm.0031.272±\pm.0550.402±\pm.0220.870±\pm.0040.34±\pm.08%
Advantage-level0.298±\pm.0150.776±\pm.0070.485±\pm.0220.937±\pm.0101.387±\pm.1010.350±\pm.0100.865±\pm.0060.25±\pm.06%
Loss-level0.290±\pm.0120.771±\pm.0010.500±\pm.0300.937±\pm.0221.422±\pm.0820.352±\pm.0060.858±\pm.0050.32±\pm.09%

Table 2. Fixed-direction preference-conditioned control in MimicKit, mean ±\pm std over 3 training seeds at 1.2B environment steps. Column groups correspond to the five evaluation axes in Section 1. Bold marks the best displayed mean, including ties; ↑/↓\uparrow/\downarrow indicate the preferred direction.

Results. Reward-level achieves the highest mean Self-PRR (0.510) and task return, although the Self-PRR differences are modest relative to seed variability. Advantage-level leads in hypervolume and expected utility while minimizing Cross-PRR and falls; loss-level has the highest selectivity, with Rank Accuracy nearly tied between the two later aggregation methods. Thus, later aggregation improves coverage and selectivity here, whereas reward-level has the coverage advantage in ProtoMotions. Reduced cross-objective leakage with later aggregation is shared across both frameworks, but their coverage and utility rankings differ. Reactive-steering results in Appendix § 11.§ 3 show the same precision advantage for later aggregation.

A complementary pattern appears in the MetaDrive ablation: advantage-level aggregation achieves the highest paired expected utility and Self-PRR, whereas loss-level aggregation yields the lowest Cross-PRR (Appendix § 12, Table 9). This further suggests that later aggregation can improve preference control, although the metric-optimal aggregation level remains environment dependent.

MethodPaired EU↑\uparrowMean Self-PRR↑\uparrowMean Cross-PRR↓\downarrow
Reward-level0.608±\pm.0140.0028±\pm.00270.0216±\pm.0012
Advantage-level0.634±\pm.0150.0263±\pm.00170.0246±\pm.0015
Loss-level0.586±\pm.0200.0015±\pm.00030.0112±\pm.0005

Table 9. MetaDrive environment ablation, mean ±\pm std over three evaluation seeds. Bold marks the best value; ↑\uparrow/↓\downarrow indicate the preferred direction.

Limitations and Future Work

Our study is limited to two simulated humanoid environments with a narrow range of tasks and a small set of hand-designed preference dimensions, and no strategy resolves the responsiveness-precision trade-off we observe. Future work should investigate adaptive objective-combination methods that explicitly balance these properties, other gradient-conflict-resolution methods beyond PCGrad, robustness to out-of-distribution and dynamically changing preferences, and transfer to real robotic systems.

Conclusion

We study when a preference-conditioned policy is truly steerable: whether changing its preference input changes the intended behavior selectively, without compromising the underlying task. Because standard MORL metrics such as hypervolume and expected utility do not capture these properties, we introduce an evaluation suite based on the Preference–Reward Responsiveness matrix, including Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy alongside utility and failure metrics.

Across two humanoid-control frameworks, we show that the level of objective-signal aggregation shapes the trade-off between responsiveness and cross-objective interference. In ProtoMotions, reward-level aggregation achieves the widest controllable range and highest utility but greater leakage, while later aggregation and PCGrad improve selectivity at the cost of range. In MimicKit, all aggregation levels steer reliably, with later aggregation improving coverage or precision without materially reducing task performance. Overall, no fixed strategy dominates, motivating both multidimensional steerability evaluation and adaptive aggregation methods.

Evaluation Metrics

Definition 1 (Hypervolume Indicator). Given a reference point r∈RKr \in\mathbb{R}^{K} dominated by every point in a finite set P={p1,…,pN}⊂RKP = \{p^{1}, \ldots, p^{N}\} \subset\mathbb{R}^{K}, the hypervolume is

HV⁡(P;r)=Λ(⋃i=1N[r,pi]),\operatorname{HV}(P; r) = \Lambda\left(\bigcup_{i=1}^{N}[r,p^{i}]\right),

where [r,pi][r,p^{i}] denotes the hyperrectangle spanning from rr to pip^{i} and Λ(⋅)\Lambda(\cdot) is the Lebesgue measure [9, 7].

Definition 2 (Expected Utility). Let WW be a distribution over preference vectors and πw\pi_{w} the policy conditioned on w∼Ww \sim W. The expected utility is

EU=Ew∼W[w⊤Jπw].\mathrm{EU} = \mathbb{E}_{w\sim W}[w^{\top}J^{\pi_{w}}].

In each environment, we instantiate PP as the set of N=50N=50 achieved objective vectors J(w(1)),…,J(w(N))J(w^{(1)}),\ldots,J(w^{(N)}) obtained under preferences w(i)w^{(i)} sampled from Dirichlet(0.5)(0.5), matching the training-time preference distribution. The reference point rr is the origin of the normalized [0,1]K[0,1]^{K} objective space. Hypervolume is estimated by Monte Carlo sampling,

HV=Pr⁡u∼U([0,1]K)[∃i:J(w(i))⪰u],\mathrm{HV} = \Pr_{u\sim U([0,1]^{K})}\left[\exists i : J(w^{(i)}) \succeq u\right],

using 2×1052 \times10^{5} uniform samples uu. Expected Utility is estimated as the sample mean over the same NN points,

EU=Ew∼Dirichlet⁡(0.5)[w⊤J(w)]≈1N∑i=1Nw(i)⊤J(w(i)),\mathrm{EU} = \mathbb{E}_{w\sim\operatorname{Dirichlet}(0.5)}[w^{\top}J(w)] \approx\frac{1}{N}\sum_{i=1}^{N}w^{(i)\top}J(w^{(i)}),

following the Expected Utility formulation used in prior preference-conditioned MORL evaluation [Ambadkar et al., 2026, de las Heras Molins et al., 2026].

PCGrad Aggregation

PCGrad starts from the same K+1K+1 per-objective clipped losses ℓk=ck(w)Lclip⁡(At(k))\ell_{k}=c_{k}(\mathbf{w})L^{\operatorname{clip}}(A_{t}^{(k)}) as loss-level, but replaces their weighted sum with gradient-level conflict resolution [19]. Each per-objective gradient gk=∇θℓkg_{k}=\nabla_{\theta}\ell_{k} is projected away from any other objective’s gradient it conflicts with:

g~i=gi−∑j:gi⋅gj<0gi⋅gj∥gj∥2gj,\tilde{g}_{i}=g_{i}-\sum_{j:g_{i}\cdot g_{j}<0}\frac{g_{i}\cdot g_{j}}{\lVert g_{j}\rVert^{2}}g_{j},

using each gjg_{j}’s original, pre-projection direction. The final actor gradient is gfinal=∑k=0Kg~kg_{\mathrm{final}}=\sum_{k=0}^{K}\tilde{g}_{k}, plus auxiliary gradients from the entropy and bound losses. Non-conflicting objective pairs (gi⋅gj≥0g_{i}\cdot g_{j}\geq0) are left untouched.

For each outer objective i∈{0,…,K}i\in\{0,\ldots,K\}, we traverse the other KK objectives in an independently sampled random order (torch.randperm), rather than a single shared shuffle order for the entire update step. Each objective’s gradient gig_{i} is compared only against the original, pre-projection gradients gjg_{j} of the objectives it is checked against, so a correction already applied to gig_{i} earlier in its own traversal does not affect how later projections are computed; ii is explicitly excluded from its own inner loop to avoid a projected gradient later spuriously conflicting with its own untouched original direction. These choices follow the original PCGrad formulation [19].

ProtoMotions Experimental Details

Task and Reward Setup

All ProtoMotions experiments train a G1 humanoid to walk using PrefAMP, a preference-conditioned adversarial motion prior. The reward is a weighted sum of one fixed-weight task term and K=5K=5 preference-weighted style terms:

r=wtaskrheading+wpref∑k=1Kwkrk,wtask=0.3, wpref=1.0,r = w_{\mathrm{task}} r_{\mathrm{heading}} + w_{\mathrm{pref}} \sum_{k=1}^{K} w_k r_k,\qquad w_{\mathrm{task}} = 0.3,\ w_{\mathrm{pref}} = 1.0,

where rheadingr_{\mathrm{heading}} rewards tracking a target heading velocity and w=(w1,…,wK)w=(w_1,\ldots,w_K), ∑kwk=1\sum_k w_k=1, is the style preference vector. During training, ww is resampled each episode from Dirichlet(0.5). Table 3 lists the five style dimensions and the bounds used to normalize each raw reward to [0,1][0,1] before it enters any of the metrics in Appendix § 10.§ 2.

DimensionDescriptionBounds
energynegative raw power (W)(−800.0, 0.0)(-800.0,\ 0.0)
arm_swingshoulder-pitch angular velocity (rad/s)(0.0, 3.0)(0.0,\ 3.0)
foot_clearanceswing-phase foot height (m)(0.0, 0.20)(0.0,\ 0.20)
step_widthlateral foot separation (m)(0.0, 0.40)(0.0,\ 0.40)
smoothnessnegative ∥Δaction∥2\lVert\Delta\mathrm{action}\rVert_2(−2.5, 0.0)(-2.5,\ 0.0)

Table 3. Style reward dimensions and normalization bounds.

Each episode has a fixed horizon of H=300H=300 steps. Each of the four aggregation methods (§ 4.1) is trained independently for 1.2×1091.2 \times10^9 environment steps (4096 parallel environments, 16-step PPO rollouts) with three random seeds (42, 43, 44), giving 12 trained checkpoints in total.

Evaluation Metrics and Protocol

Fixed-horizon return. For a held-fixed preference ww, we roll out the policy for HH steps and compute, for each style dimension kk,

Jk(w)=Eenv[1H∑t=1Hrk,tnorm⋅1[alive at t]].J_k(w) = \mathbb{E}_{\mathrm{env}}\left[\frac{1}{H}\sum_{t=1}^{H}r_{k,t}^{\mathrm{norm}}\cdot\mathbb{1}[\mathrm{alive\ at\ }t]\right].

where rk,tnorm∈[0,1]r_{k,t}^{\mathrm{norm}}\in[0,1] is the bounds-normalized reward at step tt and the alive indicator zeroes every step after the first fall, so a fall is always penalized regardless of when it occurs. Jtask(w)J_{\mathrm{task}}(w) is defined identically using rheadingr_{\mathrm{heading}} and its own bounds.

Axis sweep. For each dimension kk and a reference preference w0w^0 (uniform by default), we construct M=15M=15 sweep points

wk(i)=αiek+(1−αi)w−k0∑k′w−k′0,αi∈[0.05,0.95] evenly spaced,w_k^{(i)}=\alpha_i e_k+(1-\alpha_i)\frac{w_{-k}^0}{\sum_{k'}w_{-k'}^0},\qquad\alpha_i\in[0.05,0.95]\text{ evenly spaced},

where eke_k is the kk-th one-hot vector and w−k0w_{-k}^0 is w0w^0 with its kk-th entry zeroed. This gives K×M=75K\times M=75 points, evaluated as one batched rollout per swept dimension.

Self-PRR, Cross-PRR, Selectivity. The preference-reward responsiveness matrix is the finite-difference slope of JjJ_j along the sweep of dimension kk:

PRR⁡[j][k]=1M−1∑i=1M−1Jj(wk(i+1))−Jj(wk(i))αi+1−αi.\operatorname{PRR}[j][k]=\frac{1}{M-1}\sum_{i=1}^{M-1}\frac{J_j\left(w_k^{(i+1)}\right)-J_j\left(w_k^{(i)}\right)}{\alpha_{i+1}-\alpha_i}.

Self-PRRk=PRR⁡[k][k]_k=\operatorname{PRR}[k][k] measures how strongly dimension kk responds to its own preference weight; Cross-PRRk=1K−1∑j≠k∣PRR⁡[j][k]∣_k=\frac{1}{K-1}\sum_{j\ne k}\lvert\operatorname{PRR}[j][k]\rvert measures how much it leaks into the others. Selectivity is the ratio of their means across all KK dimensions, Self-PRR/∣\lvertCross-PRR∣\rvert.

Rank accuracy. For each dimension kk, Rank Accuracy is the fraction of sweep-point pairs (a,b)(a,b) for which the ordering of preference weights predicts the ordering of achieved reward:

RankAcc⁡k=1(M2)∑a<b1[(αb−αa)(Jk(wk(b))−Jk(wk(a)))>0].\operatorname{RankAcc}_k=\frac{1}{\binom{M}{2}}\sum_{a<b}\mathbb{1}\left[(\alpha_b-\alpha_a)\left(J_k\left(w_k^{(b)}\right)-J_k\left(w_k^{(a)}\right)\right)>0\right].

Hypervolume and Expected Utility. In addition to the 75 axis-sweep points, we draw N=50N=50 preference vectors from Dirichlet(0.5), matching the training-time distribution, and evaluate the resulting objective vectors J(w(1)),…,J(w(N))J(w^{(1)}), \ldots, J(w^{(N)}) in one batched rollout. Hypervolume is estimated by Monte Carlo sampling with reference point at the origin of the normalized [0,1]K[0,1]^K space:

HV⁡=Pr⁡u∼U([0,1]K)[∃i:J(w(i))⪰u].\operatorname{HV} = \Pr_{u \sim U([0,1]^K)} \left[\exists i : J(w^{(i)}) \succeq u\right].

estimated from 2×1052 \times10^5 uniform samples uu. Expected Utility is

EU⁡=Ew∼Dirichlet⁡(0.5)[w⊤J(w)]≈1N∑i=1Nw(i)⊤J(w(i)),\operatorname{EU} = \mathbb{E}_{w \sim\operatorname{Dirichlet}(0.5)} \left[w^\top J(w)\right] \approx\frac{1}{N}\sum_{i=1}^{N} w^{(i)\top}J(w^{(i)}),

following the definition used in prior preference-conditioned MORL evaluation [Ambadkar et al., 2026, de las Heras Molins et al., 2026].

Statistical methodology. Every checkpoint is evaluated with 3 repeats using a fixed base evaluation seed of 1000 (i.e. eval seeds 1000, 1001, 1002), shared across all four methods and all three training seeds for consistent evaluation across checkpoints. Table 4 summarizes all evaluation hyperparameters. ProtoMotions results are computed in two stages: first the mean over the 3 eval repeats for a given training-seed checkpoint, then the mean and standard deviation (population, ddof=0) of those three per-seed means across the 3 training seeds. Thus, the reported dispersion is across training-seed means after averaging eval-time rollout noise. MimicKit uses sample standard deviation as specified in Appendix D.

ParameterValue
Horizon HH300
Axis-sweep points per dimension MM15
Axis-sweep range [αmin⁡,αmax⁡][\alpha_{\min}, \alpha_{\max}][0.05,0.95][0.05, 0.95]
Full-simplex samples NN50
Dirichlet concentration α\alpha0.5
Hypervolume Monte Carlo samples2×1052 \times10^5
Eval repeats3 (seeds 1000/1001/1002)

Table 4. Evaluation hyperparameters, shared across all four methods and three training seeds.

Per-Dimension Breakdown

Table 5 decomposes Self-PRR, Cross-PRR, and Rank Accuracy by style dimension. The trade-off summarized in Table 1 is not uniform across dimensions: reward-level’s advantage in Self-PRR is driven primarily by arm_swing and foot_clearance, while on energy it is the weakest of the four methods, consistent with the near-convergence of all four curves on that dimension in Figure 1.

MethodDimensionSelf-PRR↑\uparrowCross-PRR↓\downarrowRank acc.↑\uparrow
Reward-levelenergy0.295±.0510.295 \pm.0510.441±.0420.441 \pm.0420.791±.0590.791 \pm.059
Reward-levelarm_swing0.703±.0800.703 \pm.0800.152±.0440.152 \pm.0440.895±.0540.895 \pm.054
Reward-levelfoot_clearance0.790±.0030.790 \pm.0030.290±.0350.290 \pm.0350.876±.0260.876 \pm.026
Reward-levelstep_width0.315±.0430.315 \pm.0430.167±.0400.167 \pm.0400.778±.0030.778 \pm.003
Reward-levelsmoothness0.165±.0100.165 \pm.0100.375±.0590.375 \pm.0590.893±.0280.893 \pm.028
Loss-levelenergy0.048±.0100.048 \pm.0100.067±.0180.067 \pm.0180.678±.0350.678 \pm.035
Loss-levelarm_swing0.198±.0020.198 \pm.0020.040±.0110.040 \pm.0110.746±.1560.746 \pm.156
Loss-levelfoot_clearance0.594±.0470.594 \pm.0470.336±.0170.336 \pm.0170.984±.0030.984 \pm.003
Loss-levelstep_width0.256±.0190.256 \pm.0190.063±.0240.063 \pm.0240.939±.0170.939 \pm.017
Loss-levelsmoothness0.027±.0140.027 \pm.0140.051±.0130.051 \pm.0130.614±.0420.614 \pm.042
Advantage-levelenergy0.040±.0130.040 \pm.0130.060±.0110.060 \pm.0110.659±.0470.659 \pm.047
Advantage-levelarm_swing0.240±.0850.240 \pm.0850.038±.0110.038 \pm.0110.944±.0130.944 \pm.013
Advantage-levelfoot_clearance0.575±.0640.575 \pm.0640.331±.0340.331 \pm.0340.981±.0030.981 \pm.003
Advantage-levelstep_width0.297±.0840.297 \pm.0840.076±.0100.076 \pm.0100.938±.0130.938 \pm.013
Advantage-levelsmoothness0.009±.0070.009 \pm.0070.037±.0110.037 \pm.0110.532±.0340.532 \pm.034
PCGradenergy0.030±.0390.030 \pm.0390.056±.0410.056 \pm.0410.585±.1410.585 \pm.141
PCGradarm_swing0.238±.0430.238 \pm.0430.041±.0060.041 \pm.0060.973±.0090.973 \pm.009
PCGradfoot_clearance0.612±.0140.612 \pm.0140.347±.0220.347 \pm.0220.979±.0140.979 \pm.014
PCGradstep_width0.277±.0300.277 \pm.0300.086±.0190.086 \pm.0190.923±.0200.923 \pm.020
PCGradsmoothness0.007±.0110.007 \pm.0110.042±.0240.042 \pm.0240.540±.0640.540 \pm.064

Table 5. Per-dimension Self-PRR, Cross-PRR, and Rank Accuracy, mean ± std over 3 training seeds.

Measurement Validity: Fixed vs. Alive Convention

All results in the main text use the fixed-horizon convention above, which divides by the full horizon HH and therefore always penalizes a fall. As a robustness check, we also compute a quality-while-alive statistic that does not penalize falls in the same way, a pooled ratio of accumulated reward over accumulated alive steps across all environments:

Jkalive(w)=∑env∑trk,tnorm⋅I[alive at t]∑env(alive steps).J_k^{\mathrm{alive}}(w) = \frac{\sum_{\mathrm{env}} \sum_t r_{k,t}^{\mathrm{norm}} \cdot\mathbb{I}[\text{alive at }t]}{\sum_{\mathrm{env}}(\text{alive steps})}.

This answers a different question, how good the behavior is conditional on survival, rather than whether the preference is practically usable end to end, and the two are not on the same numeric scale. Table 6 reports Hypervolume and Expected Utility under both conventions. As expected, Alive values are uniformly higher than Fixed values, since removing the fall penalty can only increase the statistic. More importantly, every best-in-column method in Table 1 remains the best under the Alive convention: reward-level remains highest on Hypervolume, Expected Utility, Self-PRR, and Rank Accuracy, and advantage-level remains highest on Selectivity and lowest on Cross-PRR, under both conventions. The trade-off we report is therefore not an artifact of how failures are scored.

MethodHypervolume FixedHypervolume AliveExpected Utility FixedExpected Utility Alive
Reward-level0.316±.0160.316 \pm.0160.435±.0210.435 \pm.0210.763±.0070.763 \pm.0070.818±.0080.818 \pm.008
Advantage-level0.109±.0110.109 \pm.0110.158±.0120.158 \pm.0120.568±.0140.568 \pm.0140.610±.0150.610 \pm.015
PCGrad0.104±.0160.104 \pm.0160.154±.0240.154 \pm.0240.558±.0290.558 \pm.0290.601±.0310.601 \pm.031
Loss-level0.103±.0070.103 \pm.0070.149±.0100.149 \pm.0100.572±.0100.572 \pm.0100.615±.0110.615 \pm.011

Table 6. Hypervolume and Expected Utility under the Fixed and Alive conventions, mean ± std over 3 training seeds.

As a further sanity check, note that a policy for which the preference had no effect at all (i.e. every sampled preference produces the same, near-uniform objective vector around Jk≈0.5J_k \approx0.5) would have a Hypervolume around 0.5K≈0.030.5^K \approx0.03. Every method’s Fixed Hypervolume in Table 6 exceeds this degenerate baseline by a factor of 3×3\times to 10×10\times, indicating that all four methods produce genuinely preference-responsive behavior rather than a collapsed, preference-insensitive policy.

MimicKit Experimental Details

Task and Reward Setup

All MimicKit experiments train the framework’s native 28-actuated-DoF humanoid (data/assets/humanoid/humanoid.xml) using PrefAMP in Isaac Lab. The reward is a weighted sum of fixed task and AMP terms and K=5K=5 preference-weighted style terms:

r=wtaskrtask+wAMPrAMP+wpref∑k=1Kwkr~k,r = w_{\mathrm{task}} r_{\mathrm{task}} + w_{\mathrm{AMP}} r_{\mathrm{AMP}} + w_{\mathrm{pref}} \sum_{k=1}^{K} w_k \tilde{r}_k,

with

wtask=0.1875,wAMP=0.1875,wpref=0.625.w_{\mathrm{task}} = 0.1875, \qquad w_{\mathrm{AMP}} = 0.1875, \qquad w_{\mathrm{pref}} = 0.625.

Here, r~k∈[0,1]\tilde{r}_k \in[0, 1] is normalized style reward kk and w=(w1,…,wK)\mathbf{w} = (w_1, \ldots, w_K), ∑kwk=1\sum_k w_k = 1, is the style preference vector. These coefficients are the sum-to-one normalization of the intended task:AMP:style ratio 0.3:0.3:1.0. During training, w\mathbf{w} is resampled each episode from Dirichlet(0.5) and supplied to both actor and critic. The five style dimensions and normalization bounds match Table 3.

For the fixed-direction experiment in Table 2, the task reward is

rtask=rvelocity.r_{\mathrm{task}} = r_{\mathrm{velocity}}.

The velocity term exponentially penalizes target-velocity error with scale 0.5 and is zero for motion opposite the target direction. The target direction remains fixed along the world +x+x axis, while target speed is sampled from [0.5,2.0][0.5, 2.0] m/s and resampled every 4–7 seconds. The facing reward and facing-direction observation are disabled. This matches ProtoMotions in keeping the movement direction fixed; the speed-command schedule, robot, motion data, and native optimization remain specific to MimicKit.

Position targets are applied at 30 Hz with simulation at 120 Hz. Each episode has a fixed horizon of H=300H=300 control steps (10 seconds). The AMP discriminator uses dataset_humanoid_locomotion.yaml, which contains 56 forward, left, and right running and walking clips. The robot and motion data therefore remain framework-specific, whereas the style dimensions and evaluation protocol match ProtoMotions.

Each of the three aggregation methods (§ 4.1) is trained independently for 1.2×1091.2 \times10^{9} environment steps (4096 parallel environments, 32-step PPO rollouts) with three random seeds (20260827, 20260828, 20260829), giving nine fixed-direction checkpoints. The additional reactive-steering experiments are described in Appendix § 11.§ 3.

Optimization and Evaluation

Table 7 summarizes the optimization hyperparameters shared across all methods and seeds.

SettingValue
Parallel training environments4096
PPO rollout length32 steps
Actor / critic / discriminator hidden widths1024 / 1024 / 1024
Gaussian action standard deviation0.05
Discount γ\gamma / GAE λ\lambda0.99 / 0.95
PPO clipping coefficient0.2
Actor / critic / discriminator learning rate5×10−55 \times10^{-5} / 10−410^{-4} / 2.5×10−42.5 \times10^{-4}
Actor / critic / discriminator update epochs5 / 2 / 2

Table 7. MimicKit optimization hyperparameters, shared across all methods and seeds.

The normalized returns and PRR-derived metrics follow Appendix § 10.§ 2. Each checkpoint is evaluated in 256 parallel environments with H=300H=300, M=15M=15, N=50N=50, and deterministic seeds 1000, 1001, and 1002. We first average the three evaluation repeats for each checkpoint, then report the mean and sample standard deviation (ddof=1) across the three training seeds.

For both MimicKit task settings, the reported fall rate uses the native environment failure flag, triggered by ground contact from a body other than the feet, with pose-distance termination disabled. Returns are accumulated through the first failure step and zeroed thereafter through the 300-step horizon, without resetting failed environments. No root-height threshold is used for these reported results. ProtoMotions uses a different environment-specific termination rule, so absolute fall rates should not be compared directly across frameworks.

Reactive-Steering Task Variant

Setup. To assess whether the aggregation comparison extends to changing movement directions, we also evaluate a reactive-steering task. Target directions are sampled uniformly and target speeds from [0.5,2.0][0.5, 2.0] m/s; both are resampled every 4–7 seconds. The task reward is

rtask=0.7rvelocity+0.3rfacing.r_{\mathrm{task}} = 0.7r_{\mathrm{velocity}} + 0.3r_{\mathrm{facing}}.

where rfacingr_{\mathrm{facing}} is the nonnegative alignment between the humanoid’s facing direction and the target direction. The facing-direction observation is enabled. The robot, motion data, style objectives, outer task/AMP/style coefficients, optimization settings, and evaluation protocol are the same as in the fixed-direction experiment. Each method is trained for 1.2×1091.2 \times10^{9} environment steps with seeds 20260817, 20260818, and 20260819, giving nine additional checkpoints. Because the task reward and facing observation also change, this comparison does not isolate the effect of direction changes alone, and JtaskJ_{\mathrm{task}} should be compared across methods within each task setting.

Results. Table 8 reports the reactive-steering results. The methods have similar mean Self-PRR (0.471–0.480), with Rank Accuracy of at least 0.966. Advantage-level has the highest mean expected utility, Self-PRR, and selectivity; loss-level has the highest hypervolume and Rank Accuracy and the lowest Cross-PRR and fall rate. Reward-level retains the highest task return. As in the fixed-direction experiment, both later aggregation methods achieve higher mean hypervolume and selectivity and lower Cross-PRR than reward-level. Their precision advantage therefore persists across the two MimicKit task settings, while the highest mean Self-PRR shifts from reward-level under fixed direction to advantage-level under reactive steering.

ResponsivenessMonotonicitySelectivityTask preservationFailure behavior
MethodHypervolume↑\uparrowExp. Utility↑\uparrowSelf-PRR↑\uparrowRank acc.↑\uparrowSelectivity↑\uparrowCross-PRR↓\downarrowJtask↑J_{\mathrm{task}} \uparrowFall rate↓\downarrow
Reward-level0.178±.0050.178\pm.0050.728±.0060.728\pm.0060.479±.0200.479\pm.0200.966±.0070.966\pm.0071.315±.0341.315\pm.0340.365±.0230.365\pm.0230.813±.0100.813\pm.0100.26±.04%0.26\pm.04\%
Advantage-level0.199±.0040.199\pm.0040.732±.0030.732\pm.0030.480±.0040.480\pm.0040.971±.0050.971\pm.0051.503±.0151.503\pm.0150.319±.0020.319\pm.0020.809±.0030.809\pm.0030.22±.03%0.22\pm.03\%
Loss-level0.201±.0050.201\pm.0050.731±.0030.731\pm.0030.471±.0130.471\pm.0130.975±.0080.975\pm.0081.498±.0671.498\pm.0670.315±.0070.315\pm.0070.807±.0100.807\pm.0100.17±.03%0.17\pm.03\%

Table 8. Reactive-steering preference-conditioned control in MimicKit, mean ±\pm std over 3 training seeds at 1.2B environment steps. Bold marks the best mean; ↑/↓\uparrow/\downarrow indicate the preferred direction.

MetaDrive

Setup. As an environment ablation, we repeat the aggregation comparison in MetaDrive, a procedurally generated autonomous-driving environment. We compare Reward-level, loss-level and advantage-level. Each policy is conditioned on a K=9K=9 preference vector over driving objectives including progress, speed, lane keeping, road clearance, heading alignment, and route completion, while MetaDrive’s native task reward retains a fixed unit weight. All variants use the same architecture.

References

  1. [1]Tanmay Ambadkar, Sourav Panda, Shreyash Kale, Jonathan Dodge, and Abhinav Verma. Preference conditioned multi-objective reinforcement learning: Decomposed, diversity-driven policy optimization, 2026.
  2. [2]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback, 2022. URL https://arxiv.org/abs/2212.08073.
  3. [3]Toygun Basaklar, Suat Gumussoy, and Umit Y. Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. In International Conference on Learning Representations (ICLR), 2023a.arxiv.org/abs/2208.07914
  4. [4]Toygun Basaklar, Suat Gumussoy, and Umit Y. Ogras. PD-MORL: Preference-driven multi-objective reinforcement learning algorithm. In International Conference on Learning Representations, 2023b.arxiv.org/abs/2208.07914
  5. [6]Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaïev. SteerLM: Attribute conditioned SFT as an (user-steerable) alternative to RLHF. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11275–11288. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.754. URL https://aclanthology.org/2023.findings-emnlp.754/.
  6. [7]Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El-Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.
  7. [8]Hanze Guo, Jing Yao, Xiao Zhou, Xiaoyuan Yi, and Xing Xie. Counterfactual reasoning for steerable pluralistic value alignment of large language models. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/b0916c29bc32d57e2c86a63e8f2df9f2-Abstract-Conference.html.arxiv.org/abs/2510.18526
  8. [9]Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Einda Howley, Athirai A. Rissappan, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Rojers. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(2):26, 2022. doi: 10.1007/s10458-022-09552-y.DOI
  9. [10]Huanyu Li, Dewei Wang, Xinmiao Wang, Xin zhe Liu, Peng Liu, Chenjia Bai, and Xuelong Li. PCHC: Enabling preference conditioned humanoid control via multi-objective reinforcement learning, 2026a. URL https://arxiv.org/abs/2603.02407.
  10. [11]Quan yi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. MetaDrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3461–3475, 2023. doi: 10.1109/TPAMI.2022.3190471.
  11. [12]Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, and Jing Li. Multi-objective large language model alignment with hierarchical experts. In International Conference on Learning Representations, 2026b. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/d191ba4c8923ed8fd8935b7c98658b5f-Abstract-Conference.html.
  12. [13]Rui Liu, Yingce Pan, Lin Xu, Le Song, Peijie You, Yudong Chen, and Jiang Bian. Efficient discovery of pareto front for multi-objective reinforcement learning. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.02236
  13. [14]Xue Bin Peng. MimicKit: A reinforcement learning framework for motion imitation and control. 2025. URL https://arxiv.org/abs/2510.13794.
  14. [15]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.arxiv.org/abs/1506.02438
  15. [16]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017.arxiv.org/abs/1707.06347
  16. [17]Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannahneh Hajishirzi, Noah A. Smith, and Simon S. Du. Decoding-time language model alignment with multiple objectives. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/0917017-1549. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/57c89l26d60c209f48d0e6395c766bb3-Abstract-Conference.html.arxiv.org/abs/2406.18853
  17. [18]Chen Tessler, Yifeng Jiang, Xue Bin Peng, Erwin Coumans, Yi Shi, Haotian Zhang, Davis Rempe, Gal Chechik, and Sanja Fidler. Protomotions3: An open-source framework for humanoid simulation and control. https://github.com/NVlabs/ProtoMotions, 2025. GitHub repository.
  18. [19]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824–5836, 2020.arxiv.org/abs/2001.06782

Paper details

vlnc:W_dfb248pd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents