betaFeedback

WorksComputer Science

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

Abstract

Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior

Introduction

Human preference plays a pivotal role in human-robot collaboration Wang et al. [7]. A robot optimized only for task performance may generate behaviors that are efficient from the robot’s perspective but difficult for humans to trust or accept. For example, in social navigation, a robot may minimize travel time by passing close to obstacles or moving aggressively near humans, even though such behavior may appear unsafe or uncomfortable. In these scenarios, the quality of a decision should not be determined only by task performance, but also by whether the behavior aligns with human expectations, safety concerns, and preferences.

Classical approaches to human-aware decision-making have used optimal control, probabilistic inference, and temporal models to account for uncertainty in human-robot interaction (HRI). In particular, Hidden Markov Models (HMM), Dynamic Bayesian Networks (DBN) (Xu and Dudek (2015)), and Trust-POMDPs Chen et al. [2] have been used to model latent human states such as intent, workload, or trust, and to update robot decision-making based on observations of human behavior or feedback. Although these models provide explicit uncertainty modeling, they often require carefully designed transition models, observation models, and reward functions. Online adaptation can also become challenging in high-dimensional or long-horizon settings, where repeated human queries and online re-planning may increase cognitive load and computational cost.1

Alternatively, learning from human feedback has emerged as an effective alternative for aligning autonomous agents with human-preferred decisions Poddar et al. [4]. In robotics, preference-based reinforcement learning (PbRL) allows humans to evaluate robot behaviors at the trajectory level and provide feedback that can be used to learn a reward model Liu et al. [3]. Reinforcement Learning from Human Feedback (RLHF) extends this idea by incorporating the learned human reward during policy optimization, enabling the robot to optimize not only for task completion but also for human-preferred behavior Liu et al. [3]. This is particularly important in HRI, where the environment reward may specify what the robot should accomplish, while human feedback provides information about how the task should be accomplished.

However, preference labels alone provide limited information about the factors that drive human decisions. A human may prefer one trajectory over another because it maintains safer obstacle clearance, follows a smoother path, avoids low-traversability regions, or better balances efficiency and safety. Standard preference-based reward learning captures which behavior is preferred, but does not explicitly model why it is preferred Wang [8]. Prior work has explored interpretable PbRL through structured reward representations, such as tree-based reward functions Bewley and Lecue [1], while broader studies on human-AI interpretability emphasize the importance of explaining agent behavior in terms that align with human expectations [6]. However, standard preference-based reward learning typically does not use human-provided rationale as direct supervision for reward learning.

To address this limitation, we propose a reason-guided preference-based RLHF framework for human-aware robotic decision-making. In addition to trajectory preference labels, humans provide reason tags that explain why a behavior is preferred, such as predictability, legibility, safety, smoothness, or task efficiency. These reason labels provide auxiliary supervision during offline reward learning, improving the interpretability of the learned human reward model and supporting downstream policy optimization. The learned human reward is then combined with the task reward during reinforcement learning, enabling the robot policy to better balance task performance, safety, and human-aligned behavior.

The main contributions of this paper are as follows:

  • We formulate a preference-based RLHF framework for human-aware robotic decision-making, where trajectory-level human feedback is used to learn an offline reward signal that complements the task reward during policy optimization.

  • We develop a reason-guided human reward model that jointly predicts trajectory preferences and reason tags, enabling the robot to learn both which behaviors humans prefer and why they are preferred.

  • We show that incorporating reason-based feedback provides richer supervision than preference-only reward learning, improving reward interpretability and supporting policies that better balance task performance, safety, and human-aligned behavior.

Methods

Problem Statement

An RL agent must learn to leverage human preferences or feedback to improve the policy so that the agent prefers human-aligned policy. We consider the episodic agent navigating in an environment as a MDP M={S,A,P,r,γ}M = \{\mathcal{S}, \mathcal{A}, P, r, \gamma\} where st∈Ss_t \in\mathcal{S} is the agent state and at∈Aa_t \in\mathcal{A} is a continuous action with transition probability st+1∼P(⋅∣st,at)s_{t+1} \sim P(\cdot\mid s_t, a_t). The environment gives reward rtenv=renv(st,at,st+1)r_t^{env} = r^{env}(s_t, a_t, s_{t+1}). However, in preference-based RLHF, human might prefer an action aligned with their own belief and preferences contrary to an optimal policy seeking safety over performance. Therefore, we design the reward as a combination of learned human model and environment reward per step.

rtπ=rtenv+λHr^tH(1)r_t^\pi= r_t^{env} + \lambda_H \hat{r}_t^H \tag*{(1)}

where λH\lambda_H controls the human preference strength and r^tH\hat{r}_t^H represents the predicted human scalar reward at time tt. The goal is to learn a policy πφ(at∣st)\pi_\varphi(a_t \mid s_t) with the objective of maximizing a joint objective of task performance and human aligned preference.

π∗=argmax⁡π Eτ∼π[∑t=0Tγtrtπ](2)\pi^* = \underset{\pi}{\operatorname{argmax}}\ \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T} \gamma^t r_t^\pi\right] \tag*{(2)}

Preference-based RLHF

Preference-based RLHF uses trajectory-level human feedback to train a human reward model offline. A trajectory clip is a finite sequence of state-action transitions τ=(s0,a0,s1,a1,⋯ ,sT)\tau= (s_0, a_0, s_1, a_1, \cdots, s_T), where TT is the clip horizon length. For each transition, we construct an observable feature vector xt=ϕ(st,at,st+1)∈Rdx_t = \phi(s_t, a_t, s_{t+1}) \in\mathbb{R}^d, where ϕ(⋅)\phi(\cdot) is a feature extraction function. The feature vector xtx_t represents task- and safety-relevant quantities observable by the agent, such as progress, traversability, distance to goal, minimum collision distance, and other domain-specific features. For a clip τ\tau, the corresponding sequence of features are

X(τ)=[x0∥x1∥⋯∥xT−1]∈RT×d,(3)\mathbf{X}(\tau) = [x_0 \parallel x_1 \parallel\cdots\parallel x_{T-1}] \in\mathbb{R}^{T \times d}, \tag*{(3)}

where ∥\parallel denotes concatenation along the temporal dimension. The human reward model is parameterized by a neural network with trainable parameters θ\theta. It maps the per-step feature vector xtx_t to a scalar human reward rθH(xt):Rd→Rr_\theta^H(x_t): \mathbb{R}^d \to\mathbb{R}. A simple neural reward model can be written as

ht=fθenc(xt),(4)h_t = f_\theta^{enc}(x_t), \tag*{(4)}
r^θH(xt)=fθrew(ht),(5)\hat{r}_\theta^H(x_t) = f_\theta^{rew}(h_t), \tag*{(5)}

where fθenc(⋅)f_\theta^{enc}(\cdot) denotes the feature encoder and fθrew(⋅)f_\theta^{rew}(\cdot) denotes the scalar human reward head.

In addition to predicting the scalar per-step human reward, the model also predicts the reason labels associated with the preferred trajectory clip. For this, a reason prediction head is added on top of the aggregated trajectory representation:

c^(τ)=σ(fθreason(hˉτ)),(6)\hat{\mathbf{c}}(\tau) = \sigma\left(f_\theta^{reason}(\bar{h}_\tau)\right), \tag*{(6)}

where σ(⋅)\sigma(\cdot) is the logistic sigmoid function, c^(τ)∈[0,1]K\hat{\mathbf{c}}(\tau) \in[0,1]^K is the predicted probability vector over KK reason labels, fθreason(⋅)f_\theta^{reason}(\cdot) is the reason classification head, and hˉτ\bar{h}_\tau is obtained by aggregating the per-step embeddings:

hˉτ=1T∑t=0T−1ht.(7)\bar{h}_\tau= \frac{1}{T}\sum_{t=0}^{T-1} h_t. \tag*{(7)}

Let the trajectory clip be denoted by τ\tau and the corresponding sequence of clip features be X(τ)\mathbf{X}(\tau). The predicted human reward for the full trajectory clip is computed by aggregating the per-step reward predictions:

R^θH(X(τ))=∑t=0T−1γtr^θH(xt),(8)\hat{R}_\theta^H(\mathbf{X}(\tau)) = \sum_{t=0}^{T-1} \gamma^t \hat{r}_\theta^H(x_t), \tag*{(8)}

where γ∈[0,1]\gamma\in[0,1] is a discount factor.

In the current preference-based reward learning, humans compare trajectory clips generated under identical environmental conditions, including start and goal locations and obstacle configurations. For each pair {τiA,τiB}\{\tau_i^A, \tau_i^B\}, where ii indexes an environment configuration, the human selects the preferred trajectory based on the agent’s behavior. The reward model fθrew(⋅)f_\theta^{rew}(\cdot) is then trained to align its predictions with these preferences. The human preference index is represented as

yi={1,if τiA≻τiB,0,if τiB≻τiA.(9)y_i = \begin{cases} 1, & \text{if } \tau_i^A \succ\tau_i^B,\\ 0, & \text{if } \tau_i^B \succ\tau_i^A. \end{cases} \tag*{(9)}

where τiA≻τiB\tau_i^A \succ\tau_i^B implies trajectory A is preferred over B. Let ci∈{0,1}K\mathbf{c}_i \in\{0,1\}^K denote the multi-hot encoded reason vector for the human-preferred trajectory in the ii-th comparison, where KK is the number of predefined reasons. Each entry ci,k=1c_{i,k}=1 indicates the selection of reason kk (0 otherwise), allowing multiple reasons to be selected simultaneously for a single preference. The reward model assigns each clip a scalar score RA=R^θH(τiA)R_A=\hat{R}_{\theta}^{H}(\tau_i^A) and RB=R^θH(τiB)R_B=\hat{R}_{\theta}^{H}(\tau_i^B).

The probability that the human prefers trajectory clip τiA\tau_i^A over τiB\tau_i^B is modeled using the Bradley-Terry preference model Rafailov et al. [5] as

Pθ(τiA≻τiB)=σ(RA−RB)(10)P_{\theta}(\tau_i^A \succ\tau_i^B)=\sigma(R_A-R_B) \tag*{(10)}

The reward model is trained by minimizing the cross-entropy (CE) loss over the preference dataset DH={(τiA,τiB,yi,ci)}i=1N\mathcal{D}_H=\{(\tau_i^A,\tau_i^B,y_i,c_i)\}_{i=1}^{N}. The preference loss using the reward difference as the logit is represented as

Lpref(θ)=−1N∑i=1N[yilog⁡σ(RA−RB)+(1−yi)log⁡σ(RB−RA)].(11)\mathcal{L}_{pref}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\left[y_i\log\sigma(R_A-R_B)+(1-y_i)\log\sigma(R_B-R_A)\right]. \tag*{(11)}

Preference comparisons constrain only relative ordering between trajectory clips and do not uniquely identify the per-step reward function. Hence, we regularize the learned reward to reduce large reward magnitudes and high temporal variation before using it for policy optimization.

Lreg(θ)=1N∑i=1N∑τ∈{τiA,τiB}[1T∑t=0T−1(rθH(xt))2+1T−1∑t=0T−2(rθH(xt+1)−rθH(xt))2](12)\mathcal{L}_{reg}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\sum_{\tau\in\{\tau_i^A,\tau_i^B\}}\left[\frac{1}{T}\sum_{t=0}^{T-1}(r_{\theta}^{H}(x_t))^2+\frac{1}{T-1}\sum_{t=0}^{T-2}(r_{\theta}^{H}(x_{t+1})-r_{\theta}^{H}(x_t))^2\right] \tag*{(12)}

The first term controls reward magnitude, while the second encourages temporal smoothness.

In addition, reason labels provide semantic supervision for reward learning. For the preferred clip, the predicted

reason vector is

c^i=yic^(τiA)+(1−yi)c^(τiB).(13)\hat{c}_i=y_i\hat{c}(\tau_i^A)+(1-y_i)\hat{c}(\tau_i^B). \tag*{(13)}

The reason-guided binary cross-entropy (BCE) loss is defined as

Lreason(θ)=−1N∑i=1N∑k=1K[ci,klog⁡c^i,k+(1−ci,k)log⁡(1−c^i,k)].(14)\mathcal{L}_{reason}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K}\left[c_{i,k}\log\hat{c}_{i,k}+(1-c_{i,k})\log(1-\hat{c}_{i,k})\right]. \tag*{(14)}

This loss encourages the reward model to capture not only which trajectory is preferred, but also why it is preferred. The final offline reward-model training objective is

LH(θ)=Lpref(θ)+αreasonLreason(θ)+βregLreg(θ),(15)\mathcal{L}_{H}(\theta)=\mathcal{L}_{pref}(\theta)+\alpha_{reason}\mathcal{L}_{reason}(\theta)+\beta_{reg}\mathcal{L}_{reg}(\theta), \tag*{(15)}

where αreason\alpha_{reason} and βreg\beta_{reg} are weights for the reason and regularization loss terms, respectively.

Policy Optimization

The trained model r^θ∗H\hat{r}_{\theta^{*}}^{H} is then used as a learned human reward function during agent learning where the human reward model parameters θ∗\theta^{*} are frozen. At each time step, the reward is queried per-step for policy optimization as

rttotal=r~tenv+λHr~θ∗H(xt),(16)r_t^{total}=\tilde{r}_t^{env}+\lambda_H\tilde{r}_{\theta^{*}}^{H}(x_t), \tag*{(16)}

where, r~\tilde{r} represents normalized per-step scalar reward. The policy πφ(at∣st)\pi_{\varphi}(a_t\mid s_t) is trained to maximize the objective:

J(φ)=Eτ∼πφ[∑t=0Tγt(r~tenv+λHr~θ∗H(xt))](17)J(\varphi)=\mathbb{E}_{\tau\sim\pi_{\varphi}}\left[\sum_{t=0}^{T}\gamma^t(\tilde{r}_t^{env}+\lambda_H\tilde{r}_{\theta^{*}}^{H}(x_t))\right] \tag*{(17)}

where J(φ)J(\varphi) represents the expected discounted return and the trainable policy parameters φ\varphi are optimized using gradient descent ∇φJ(φ)\nabla_{\varphi}J(\varphi).

Experiments

Data Collection

We designed an off-road single-agent navigation simulator. The overall task for the agent is to navigate to the goal (xg,yg)(x_g, y_g) while adhering to robot dynamics and obstacles. The agent can navigate using pre-trained RL policy or model-based control approaches. The robot has stochastic dynamics and the robot pose [x,y,θ][x, y, \theta] evolves under control command [ω,v][\omega, v]. The environment also contains static obstacles with varying sizes and configuration to induce collision avoidance.

Pre-trained RL policy or model-based planners are used to create the offline preference dataset. The dataset D\mathcal{D} contains multiple sampled clips along with per-step features xtx_t for the same start and goal locations and obstacle configurations. The human is shown pair-wise episodic clips {τA,τB}\{\tau_A, \tau_B\} and needs to select the preferred trajectory along with reason labels for the selection. For, a simple go-To-goal environment with obstacle avoidance, the reason labels for preferred trajectory can be due to ”safe clearance”, ”shorter path”, ”smoothness” and so on. The dataset stores the preferred clip index {0,1}\{0, 1\} along with the reason labels represented as multi-hot encoded vector for each query.

Training Implementation

A simple MLP with one hidden layer maps the clip-level input features X(τ)\mathbf{X}(\tau) into embedded representation. The output contains two heads, a scalar reward head, fθrewf_\theta^{rew} that minimizes the cross entropy (CE) loss between predicted and actual human preferred index as well as a reason head fθreasonf_\theta^{reason} that minimizes binary cross entropy (BCE) loss across multiple labels simultaneously (Figure 1). The dataset D\mathcal{D} contains dedicated human annotation across 350 queries with 90% queries used for training and 10% for validation. The parameters of the trained human reward model R^θ∗H\hat{R}_{\theta_*}^{H} are frozen and queried at step-level during policy training. The training hyperparameters are shown in Table 1.

Offline Human Preference Learning and Policy Optimization

Figure 1. Offline Human Preference Learning: Human observes a pair of clips and records preference as well as reason labels. NN model is trained offline to predict preference index frewf^{rew} using cross entropy (CE) and reason tags freasonf^{reason} using binary cross entropy (BCE). Policy Optimization: Learned human reward and task reward jointly perform policy optimization that is both task-efficient and aligned with interpretable human feedback.

ParameterValue
Hidden layer dim256
OptimizerSGD
Learning Rate2×10−42 \times10^{-4}
Effective Batch Size8
epochs20
λH\lambda_H[0,0.5][0, 0.5]
γ\gamma0.9
αreason\alpha_{reason}0.2
βreg\beta_{reg}1e−31e-3

Table 1. Hyperparameters

Policy Training

The policy πϕ(at∣st)\pi_\phi(a_t \mid s_t) is trained on a vectorized 2D off-road environment for go-To-goal task with obstacle avoidance. The normalized task reward r~env\tilde{r}^{env} is designed to encourage progress toward the goal while penalizing unsafe or inefficient behavior. The environment provides a large positive terminal reward for reaching the goal and large negative terminal penalties for collisions or going out-of bounds. Reward shaping include progress towards goal, obstacle-proximity and control-related penalties for inducing trajectory smoothness. Adding human reward r~θ∗H\tilde{r}_{\theta_*}^{H} and training the policy with rtotalr^{total} (16) is expected to align the agent’s decision with human’s preference.

Reason-guided Ablation

To isolate the contribution of reason supervision and reward regularization, we train the human reward model offline under four ablation objectives: preference loss (Lpref)(\mathcal{L}_{pref}) only, preference loss with reward regularization (Lpref+Lreg)(\mathcal{L}_{pref} + \mathcal{L}_{reg}), preference loss with reason supervision (Lpref+αreasonLreason)(\mathcal{L}_{pref} + \alpha_{reason}\mathcal{L}_{reason}), and preference loss with both reason supervision and reward regularization (Lpref+αreasonLreason+Lreg)(\mathcal{L}_{pref} + \alpha_{reason}\mathcal{L}_{reason} + \mathcal{L}_{reg}). After training, each reward model is saved and kept frozen during policy optimization. The downstream policy is then trained using the combined environment and learned human reward, where the human reward contribution is scaled by λh∈{0.1,0.2}\lambda_h \in\{0.1, 0.2\}. This design separates reward-model learning from policy optimization and allows us to evaluate whether reason-guided and regularized reward models produce policies with better task performance, obstacle clearance, path efficiency, and success rate (agent avoided collisions and reached goal) compared with the environment-only simple preference (Lpref)(\mathcal{L}_{pref}) baseline.

Policy evaluation comparison, smoother and shorter path

Figure 2b. (b) smoother and shorter path

caption: Policy Evaluation Comparison: successful rollout comparisons between the environment-only and the combined environment and human policy under similar environment conditions.

:::

Results

This section evaluates whether incorporating the learned human reward improves the alignment of the trained policy towards human preferences when compared against a policy optimized using only the task reward. Initially, we compare two reward settings: (i) an environment-only reward baseline, denoted as RenvR_{\mathrm{env}}, and (ii) a combined reward formulation, denoted as Renv+λHR^HR_{\mathrm{env}}+\lambda_H \hat{R}_H, where R^H\hat{R}_H is obtained by querying the learned human feedback model. Secondly, we also highlight the importance of reason-guided reward training (sec. 3.4) and regularization over simple preference based training. We evaluate the policies under the same environment configuration comparing human reward model trained only using preference loss, preference and reason loss and finally, regularized preference and reason loss. We analyze policy rollouts across multiple scene and policy seeds using task-level and safety-related trajectory metrics.

Rollout-Based Policy Comparison

To compare the behavior induced by the two reward formulations, we evaluate the trained policies using a fixed rollout protocol. For each method, we generate 30 policy rollouts across 4 scene seeds and multiple policy seeds, using the same start and goal conditions along with obstacle configuration to ensure a fair comparison. This setup allows us to compare the behavior of the task-only policy and the human-reward-augmented policy under same environmental conditions.

PolicyPath lengthSuccess (%)
RenvR_{\mathrm{env}}11.02±0.9611.02 \pm0.9667.5±4.267.5 \pm4.2
Renv+λHRhumanR_{\mathrm{env}} + \lambda_H R_{\mathrm{human}}10.92±0.1910.92 \pm0.1977.5±5.777.5 \pm5.7

Table 2. Reward metrics for rollout evaluation at λH=0.2\lambda_H = 0.2.

Figure 2 shows representative trajectory rollouts for two reward formulations, RenvR_{\mathrm{env}} and Rtot=Renv+R^HR_{\mathrm{tot}} = R_{\mathrm{env}} + \hat{R}_H. Each scene has same environment configuration and only successful rollouts are considered for both the policies such that the agent reaches goal to effectively capture human preference. In Figure 2a, the environment-only policy generates a path that is much closer to the obstacle while the policy trained with RtotR_{\mathrm{tot}} generates a smoother path while maintaining safe clearance with obstacles. This aligns with human preference data where the human preferred paths that are ”shorter”, ”smoother” and had ”safer clearance” from obstacles when selecting among a pair of trajectories. Similarly, Figure 2b shows the combined human and environment reward model prefers a shorter and smoother path while reaching the goal. Since, the environment reward does not explicitly encode human preference, it may yield trajectories closer to obstacles, while combined reward Renv+λHR^θ∗R_{\mathrm{env}} + \lambda_H \hat{R}_{\theta^*} encourages trajectories that better align with human preferences for safer obstacle clearance and smoother motion.

Policy evaluation comparison, safe clearance to obstacles

Figure 2. (a) safe clearance to obstacles

We quantitatively evaluate each policy over four held-out scene seeds and report the mean and standard deviation of trajectory-level metrics. The table summarizes combined reward, obstacle clearance, path length, and task success rate.

Combined reward measures the total rollout return under the evaluation reward, while obstacle clearance captures the minimum separation from obstacles or unsafe regions along the trajectory. Path length measures navigation efficiency, and success rate reports the percentage of rollouts in which the robot reaches the goal without collision within the allowed horizon. The learned human reward policy achieves a shorter average path length while maintaining a higher success rate, suggesting more efficient and smoother trajectories aligning with human preferences.

Reason-guided Reward Training

Figure 3 presents the reward-model training, validation, and downstream policy-training curves across the ablation variants. The decreasing training loss and improving validation accuracy indicate stable reward learning across objectives. The reason-guided variants incorporate additional semantic supervision, while reward regularization constrains reward magnitude and temporal variation without destabilizing training.

Reward-model validation and policy-training curves for ablation variants

Figure 3. Reward-model validation and policy-training curves for ablation variants.

Table 3 evaluates the effect of these objectives on downstream policy performance. At λH=0.1\lambda_H = 0.1, combining reason supervision and regularization improves task success from 66.7±4.7%66.7 \pm4.7\% to 78.3±2.4%78.3 \pm2.4\% compared with preference-only learning, while increasing obstacle clearance from 1.82±0.201.82 \pm0.20 to 1.99±0.301.99 \pm0.30. At λH=0.2\lambda_H = 0.2, the combined objective achieves the highest obstacle clearance (2.18±0.072.18 \pm0.07) with a path length of 11.20±1.3311.20 \pm1.33. However, regularization alone achieves the highest success rate (81.7±7.1%81.7 \pm7.1\%), highlighting the trade-offs between safety and task completion. Overall, the results suggest that reason supervision and regularization can improve downstream policy behavior, although their relative benefits depend on the human-reward weight and evaluation metric.

λH\lambda_HTraining objectiveObstacle clearancePath lengthSuccess rate (%)subjective Evaluation (%)
0.1Lpref\mathcal{L}_{\mathrm{pref}}1.82 ± 0.2011.44 ± 1.4666.7 ± 4.733.1 ± 6.4
0.1Lpref+αreasonLreason\mathcal{L}_{\mathrm{pref}} + \alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}}1.90 ± 0.2711.25 ± 0.5273.3 ± 0.065.4 ± 7.2
0.1Lpref+Lreg\mathcal{L}_{\mathrm{pref}} + \mathcal{L}_{\mathrm{reg}}1.80 ± 0.2311.16 ± 0.3675.0 ± 2.468.3 ± 8.1
0.1Lpref+αreasonLreason+Lreg\mathcal{L}_{\mathrm{pref}} + \alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}} + \mathcal{L}_{\mathrm{reg}}1.99 ± 0.3011.29 ± 0.8778.3 ± 2.467.2 ± 5.1
0.2Lpref\mathcal{L}_{\mathrm{pref}}1.78 ± 0.1411.72 ± 1.3365.0 ± 2.423.1 ± 4.9
0.2Lpref+αreasonLreason\mathcal{L}_{\mathrm{pref}} + \alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}}2.01 ± 0.1411.65 ± 0.5268.3 ± 2.466.6 ± 4.7
0.2Lpref+Lreg\mathcal{L}_{\mathrm{pref}} + \mathcal{L}_{\mathrm{reg}}2.05 ± 0.0512.04 ± 1.1181.7 ± 7.181.8 ± 4.1
0.2Lpref+αreasonLreason+Lreg\mathcal{L}_{\mathrm{pref}} + \alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}} + \mathcal{L}_{\mathrm{reg}}2.18 ± 0.0711.20 ± 1.3373.3 ± 4.780.0 ± 4.7

Table 3. Ablation study comparing reward-model training objective. Preference-only baseline Lpref\mathcal{L}_{\mathrm{pref}}, while environment+human rows use the corresponding ablation objective.

Blind subjective study

Three annotators independently assessed matched trajectory pairs in a blinded evaluation. Trajectories were presented with neutral A/B labels with policy randomized such that policy identities and reward values hidden from annotator. Each annotator selected a preferred trajectory or skipped the comparison; responses were subsequently unblinded using the stored ground truth mappings. Preference rates were computed by aggregating choices across multiple runs and seeds within each annotator, then mean and sample standard deviation across the three annotators are reported in Table.Table 3. Adding reason supervision to the preference loss improved both task success and subjective evaluation: subjective scores increased from 33.1 ± 6.4% to 65.4 ± 7.2% and from 23.1 ± 4.9% to 66.6 ± 4.7% for lambdas 0.1 and 0.2 respectively. The Lpref+Lreg\mathcal{L}_{\mathrm{pref}}+\mathcal{L}_{\mathrm{reg}} objective also performed comparably to the full objective, achieving subjective scores of 68.3 ± 8.1% versus 67.2 ± 5.1% at λH=0.1\lambda_H = 0.1, and 81.8±4.1% versus 80.0±4.7% at λH=0.2\lambda_H = 0.2. These results support the benefit of reason supervision for learning rewards aligned with human preferences, while showing that preference learning with regularization provides a strong alternative to the full objective.

Conclusion

This paper presents a reason-guided preference-based RLHF framework for human-aware robotic decision-making. The proposed approach learns an offline human reward model from trajectory-level preference feedback and augments preference learning with reason labels that explain why a behavior is preferred. By jointly learning trajectory preferences and their associated rationale, the reward model provides a more interpretable representation of human-aligned behavior than preference-only reward learning. The learned human reward is then combined with the task reward during policy optimization, allowing the robot to account for both task performance and human-sensitive criteria such as safety, smoothness, predictability, and traversability. The results show that incorporating learned human feedback can influence policy behavior beyond what is obtained from the task reward alone. In particular, the reason-guided reward model provides additional supervision during offline reward learning and supports policies that better balance efficiency with human-aligned behavioral preferences. The ablation study further highlights the role of reason supervision and reward regularization in shaping the learned reward model and downstream policy performance. Future work will extend the framework to HRI settings with richer reward signals and trust-based reward for multi-agent robotic teams.

References

  1. [1]Bewley, T. and Lecue, F. (2021). Interpretable preference-based reinforcement learning with pre-structured reward functions. arXiv preprint arXiv:2112.11230.arxiv.org/abs/2112.11230
  2. [2]Chen, M., Nikolaidis, S., Soh, H., Hsu, D., and Srinivasa, S. (2018). Planning with trust for human-robot collaboration. In Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction, 307–315.DOI
  3. [3]Liu, Y., Datta, G., Novoseller, E., and Brown, D.S. (2023). Efficient preference-based reinforcement learning using learned dynamics models. arXiv preprint arXiv:2301.04741.arxiv.org/abs/2301.04741
  4. [4]Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. (2024). Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37, 52516–52544.arxiv.org/abs/2408.10075
  5. [5]Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 53728–53741.arxiv.org/abs/2305.18290
  6. [6]Sreedharan, S., Kulkarni, A., Smith, D.E., and Kambhampati, S. (2021). A unifying bayesian formulation of measures of interpretability in human-ai. arXiv preprint arXiv:2104.10743.arxiv.org/abs/2104.10743
  7. [7]Wang, J., Yan, Y., Hu, Y., and Yang, X. (2025). A reinforcement learning from human feedback based method for task allocation of human robot collaboration assembly considering human preference. Advanced Engineering Informatics, 66, 103497. doi: https://doi.org/10.1016/j.aei.2025.103497.
  8. [8]Wang, R. (2026). Preference-based reinforcement learning for robotics: Advancements and open challenges. Authorea Preprints.

Paper details

vlnc:W_vpuk5jja
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents