Self-Labeled Preference Transfer for Steerable Robot Manipulation

Introduction

Vision-language-action (VLA) models trained on large multi-task datasets can now follow language instructions across scenes, objects, and embodiments [1, 2, 3, 4, 5, 6]. Demonstrations in these datasets usually show one way to complete a task, or follow whatever pattern the operator found natural. In reality, a task has several acceptable executions, and which one is wanted depends on the context and on the user. A mug can be grasped near its rim or near its base; an object can be placed at the center of a tray or in a corner; a path can pass close to an obstacle or give it a wide berth. We call such choices execution preferences: they concern how a task is performed rather than what is accomplished, and the desired choice may depend on the user or context. Standard task-level instructions do not specify this choice, leaving the user without explicit control over it. This paper studies how a VLA can follow a stated execution preference on a new task when its target demonstrations do not contain preference labels.

Existing work approaches execution-level control from two directions. One line makes VLAs steerable by enriching training prompts with descriptions of how each demonstration was performed, produced by human annotators or automated pipelines [7, 8, 9, 10]. These works show that VLAs can follow execution-level instructions once such annotations are available, but a new task still has to be annotated before the policy can be steered on it. The other line is preference learning, which infers a reward or policy from comparisons between executions [11, 12, 13, 14]. The output of preference learning is a policy aligned to a user’s preference (often through ranking executions [15]) rather than one that switches on command to a named style. Preference learning is typically used to adapt a policy toward preferred behavior on a particular task, rather than to provide command-conditioned switching among named execution preferences across tasks. Neither line asks whether preference control learned on labeled source tasks can be preserved when adapting to a new task whose demonstrations lack preference labels. This setting introduces two challenges. First, the source supervision must teach preference semantics that transfer across tasks, so that the behavior shown in a target demonstration can still be recognized. Second, the policy must adapt to the target task without losing its response to the preference command. Finetuning on target demonstrations with the task instruction alone provides no signal tying the demonstrated motion to a preference, so the control learned on the source can be weakened during adaptation.

We study preference transfer without target preference annotations. We assume preference-labeled demonstrations

task when its target demonstrations do not contain preference labels.

Related Work

Steerable VLAs. Large multi-task datasets have produced policies that condition on language and generalize across tasks, scenes, objects, and embodiments [1, 2, 3], usually by attaching an action head to a pretrained VLM and finetuning on demonstrations [2, 17]. Their language conditioning is not always robust [18], and the goal-level instructions in most datasets say nothing about execution. A recent line of work addresses this by writing execution details into the training language. STEER grounds modular skills in dense attribute annotations [7]; DeMiAn and FineVLA re-annotate existing datasets with fine-grained process descriptions, through a VLM pipeline in one case and human-verified labels plus a trained robotics VLM annotator in the other [8, 9]; and $\pi_{0.7}$ trains a generalist on prompts that carry strategy and metadata alongside the goal [10]. These works establish that a VLA follows how-level instructions once its training data contains them, and each obtains that data by annotating it. SPT differs in what it asks and where the labels come from. We assume a labeled source set and ask whether preference recognition learned jointly with the source policy can transfer to a new task, so that its demonstrations can be labeled without additional preference annotation and the adapted policy can still respond to the source vocabulary.

Preference learning and personalization

Preference learning typically learns rewards or policies from comparisons between executions [11, 12, 15]. In manipulation, GRAPE, APO, and RKO apply preference optimization to VLAs and diffusion policies [13, 14, 19], and Preference VLM, VARP, and VLP replace human comparisons with VLM judgments, the last training an annotator that transfers to downstream tasks [20, 21, 22]. Personalization work asks how a robot should choose among several valid behaviors for a given user: COMmet resolves human-induced conflicts by selecting among options according to stored user preferences [23], TacStyle has an LLM pick a point in a learned style latent from a handful of labels [24], and Freeform Preference Learning trains a policy that responds to several language-defined preference axes [25]. Our setting shares the premise that the user, not the task, decides the execution, but the supervision differs. A label such as “grasp low” describes a style without ranking it above “grasp high”; both are correct when requested. Rather than aligning a policy separately on each task, we ask whether preference following learned on labeled source tasks can be transferred to target tasks whose demonstrations lack preference labels.

Self-labeling and auxiliary supervision in VLAs

Using a model’s own predictions as labels for unlabeled data is standard in semi-supervised learning [26], and foundation models have been used to label robot data with task instructions [27] and process descriptions [9]. In VLAs, auxiliary language objectives are usually added to improve representations: LAP co-trains action prediction with visual question answering [28], SG-VLA adds spatial grounding [29], and TwinBrainVLA keeps a generalist pathway beside the action pathway [30]. SPT combines these two ideas but uses them differently. The recognition objective is not only an auxiliary training signal: the resulting recognizer is reused to label demonstrations from a different task.

Problem Formulation

We explore learning from demonstrations, where style and preferences are considered through a preference axis that is only available on the source data and not on the final target domain. The preference axis $C$ is a finite vocabulary set that captures preferred behaviors that modify aspects of the task (e.g., low grasp contact height or high placement location). The user selects a behavior by naming a value from this vocabulary. Suppose we are given the following data from an initial source set of tasks:

where $\xi_{S}$ are sample trajectories containing sensor observations, proprioceptive states, and actions, $\{(o_t, s_t, a_t)\}_{t=1}^{T}$. $\ell_{S}$ is the language instruction that guided the operator for the source demonstration. $c$ is the relevant label for the style of the demonstration. $i$ denotes the sample index.

Given $\mathcal{D}_{S}$, we would like to transfer skills and preferences to new target tasks that share the same preference vocabulary $C$. We assume we are given demonstrations from the target domain (following typical fine tuning frameworks), but do not assume access to preference labels:

where variables follow structure from the source domain, but are specific to the target task space $\mathcal{T}$.

The goal is a policy $\pi_{\theta}(a \mid o, s, \ell, c)$ that completes a target task in the style named by $c \in C$. We also learn a preference recognizer that predicts the preference expressed by an observed demonstration; this recognizer is later frozen and used to label target demonstrations in Stage B. Task completion and preference following are scored separately.

Two assumptions scope the setting. First, the target demonstrations must contain the requested styles: we remove their labels, not the behaviors, and self-labeling cannot produce a behavior that appears in neither the source nor the target data. Second, the vocabulary is fixed, so a value outside $C$ is out of scope. The single-axis experiments use one binary axis at a time; § 4-E covers two axes in one command.

Methods

SPT has two stages (Figure 2). Stage A jointly trains a model on the labeled source tasks to act on a stated preference and to recognize the preference shown in a demonstration. Stage B freezes the Stage-A recognizer to label the target demonstrations, then adapts a trainable copy of the Stage-A policy on the self-labeled target data.

Architecture. The action branch and the recognition branch share one pretrained vision-language backbone. The action head predicts a chunk of actions from the current observation and the instruction, to which we append “Preference: $w(c)$,” where $w(c)$ is a fixed phrase such as “grasp low”. The recognition branch reuses the VLM’s language head instead of adding a classifier. It receives a short demonstration clip, proprioceptive information, and a question about the preference, and, for the binary axes used here it answers with a single token.

Stage A: joint source training. Each training batch mixes action samples and recognition samples. An action sample pairs an observation and a preference-conditioned instruction with the demonstrated action chunk, and contributes a regression loss $\mathcal{L}_{\mathrm{act}}$. A recognition sample pairs a demonstration clip with its question and answer, and contributes a cross-entropy loss $\mathcal{L}_{\mathrm{inf}}$. The objective is

where $\lambda$ weights the recognition loss. The result is a source model $M_A = (\pi_{\theta_A}, g_{\theta_A})$: the policy $\pi_{\theta_A}$ acts on a stated preference, and the recognizer $g_{\theta_A}$ predicts the preference expressed in a demonstration.

Stage B: labeling and adapting on the target. We first freeze a copy of $g_{\theta_A}$ and let it label every target demonstration:

where $o^{(j)}$ denotes the sampled observation clip, $s^{(j)}$ its corresponding proprioceptive states, and $q$ the fixed question for the preference axis. The predicted label is cached and appended to the demonstration’s instruction. Target ground-truth preference labels are never provided to the recognizer or the adapting policy.

We then initialize a trainable policy from the Stage-A policy $\pi_{\theta_A}$ and adapt it on the self-labeled target demonstrations with the action loss alone:

The recognizer remains frozen while the policy is adapted, so the cached target labels remain fixed throughout Stage B. No recognition loss is applied during target adaptation.

Deployment. The user gives a target instruction and a preference from the source vocabulary, and the adapted policy acts on the combined prompt. The recognition branch is not invoked at deployment, so SPT requires no additional model call at inference time. For real-world rollouts, we use discreteRTC for action execution [31].

Multiple preferences. For $K$ axes the command is $\mathbf{c} = (c_1,\ldots,c_K)$ and every phrase $w(c_k)$ is appended to the instruction. The same language head answers one question per axis, and the recognition loss becomes

In Stage B the labeler predicts each component before adaptation. § 4-E evaluates this with two axes.

Experiments

We evaluate SPT through four sets of experiments. C1 tests whether preference recognition learned in Stage A transfers to target demonstrations. C2 evaluates the complete SPT pipeline for preference-controlled target adaptation in simulation and on a real robot. C3 ablates the two-stage design to examine how Stage A transfer, target adaptation, and target preference labeling affect task success and preference following. C4 extends the setting to two preferences specified in the same command.

Experimental Setup and Evaluation

Simulation. We generate preference-varied demonstrations with RoboTwin 2.0 expert scripts [16] along five axes: grasp contact height (low or high on the object), release height (low or high), obstacle clearance (narrow or wide detour), grasp orientation (side or top), and placement location (center or corner). Figure 3 illustrates the preference axes with different objects, note that all preferences apply to all objects. The source set contains eight tasks per axis, with 100 demonstrations per preference value per task. Each axis has one held-out target task with 100 demonstrations, balanced at 50 per preference value.

Real robot. We evaluate two preference transfers between different task types (Figure 4) in real world with a UR5e robot arm. For obstacle clearance, we transfer from pick-and-place around an obstacle to bowl pushing, with near and far path preferences. For strike strategy, we transfer from air hockey to billiards, with direct and bank-shot preferences. The source and target tasks differ in their objects and contact dynamics, while sharing the same goal and the corresponding preference axes. Each source and target set contains 100 demonstrations, with 50 per preference value. Target preference annotations are hidden during adaptation. Pick-and-place and bowl-pushing demonstrations are collected using FastUMI [32, 33]. Each method is evaluated for ten trials per preference command.

Prompts. The action prompt consists of the task instruction and, when a preference label is available, the suffix “Preference: $w(c)$”. For example: “Pick up the cup and place it on the plate while grasping it low.” Source policy training uses the ground-truth preference label. During target adaptation, SPT appends the recognizer’s inferred label, while Naive FT uses the bare instruction. The recognizer receives a fixed binary question for each axis, such as “Did the robot grasp the object low or high?”, and predicts the preference label as a single-token answer.

Metrics. Recognition accuracy is the fraction of target demonstrations whose predicted preference matches the ground truth. For SPT, these annotations are used only for evaluation and are not exposed during target adaptation. Chance is 50% on every axis. Preference following is the fraction of all assigned rollouts that satisfy the complete requested preference command. Task success measures task completion over the same rollouts. For the multi-preference experiment in § 4-E, we also report joint success, the fraction that both complete the task and follow the command. A rollout can succeed a task but not follow a preference, then it contributes to Task success but not to Preference following and joint success.

Thresholds are calibrated from source demonstrations and fixed before target evaluation. For clearance, we select the arm with the larger XY path length and use the trajectory segment between 25% and 85% of the rollout. Clearance is the maximum perpendicular XY deviation from the line joining the two endpoints of this segment. Table I summarizes the rollout statistic and fixed threshold for each simulated preference axis.

Axis	Demonstration control and rollout statistic
Contact height	Grasp at 20–30% or 70–80% of object height. EE vertical offset from the object origin at grasp; threshold 6.83 cm.
Release height	Low or high release. Object vertical displacement from grasp to immediately before release; threshold 12.45 cm.
Clearance	Direct path or 20–25 cm detour. Maximum XY deviation from the 25%–85% trajectory chord; threshold 15.01 cm.
Orientation	0° or 90° grasp. Left-wrist orientation feature at grasp; threshold 41.22°.
Placement	Center or corner target. EE XY distance to the receptacle’s initial actor position at release; threshold 5.36 cm.

Baseline. Naive FT starts from the same manipulation-pretrained checkpoint as SPT and follows the same source-then-target optimization schedule. It first trains for 10000 steps on preference-labeled source demonstrations using only the action loss, then adapts for 1500 steps on the 100 target demonstrations using bare task instructions. It uses neither the recognition objective nor inferred target labels. The baseline therefore matches the amount of source and target optimization, while § 4-D provides matched controls that isolate the role of self-labeling during target adaptation.

Implementation. We initialize the policy from a StarVLA-$\alpha$ RoboTwin 2.0 checkpoint [34], which uses Qwen3-VL-4B-Instruct [35] as the vision-language backbone and a residual MLP head for continuous action-chunk prediction.

In simulation, the policy receives three $224 \times224$ RGB views and the instruction, without proprioceptive inputs. The action head predicts 50-step chunks of 14-dimensional joint actions using an L1 loss. The recognizer receives three head-camera frames and the active arm’s 9-D EE state at each frame, comprising position and a 6-D rotation representation. A learned projector converts each state into a soft token for the shared backbone.

Stage A jointly trains action prediction and single-token preference recognition using $\lambda= 0.5$ in Eq. (3). We use an 80/20 source train/validation split, 10000 source-training steps, and 1500 target-adaptation steps, with an effective batch size of 64. Stage B uses all 100 target demonstrations and the action loss only. The labeler remains frozen throughout adaptation, and matched controls use the same optimization schedule.

For the policy results in Table II, we use three independent training seeds and evaluate 50 rollouts per preference command for each seed. We first average across the two preference commands for each axis and then across training seeds.

Preference axis	Zero-shot	SPT	Naive FT	SPT	Naive FT	SPT
Contact height	51	100	92	95	59	90
Release height	46	100	89	91	53	81
Clearance	53	92	85	82	57	75
Orientation	50	100	94	92	61	90
Placement	59	90	80	86	67	83
Average	51.8	96.4	88.0	89.2	59.4	83.8

C1: Target Preference Recognition

We evaluate the source-trained recognizer on the 100 target demonstrations for each preference axis before target adaptation. As an off-the-shelf vision-only reference, we use Qwen3-VL-4B-Instruct without source preference training. It receives the same three head-camera frames and binary question and scores the two candidate answers using black-image prior calibration. SPT additionally receives EE-state tokens and starts from a manipulation-pretrained checkpoint. This comparison therefore does not isolate the effect of source preference training. The recognition columns in Table II report label accuracy; the remaining columns report the policy results discussed in § 4-C.

The Stage-A recognizer achieves 96.4% average accuracy on target demonstrations. It reaches 100% on contact height, release height, and grasp orientation, 92% on obstacle clearance, and 90% on placement location. The off-the-shelf zero-shot VLM reference achieves 51.8% on average. Thus,

the recognizer learned on the source tasks generalizes well enough to provide high-quality labels for target adaptation across all five preference axes.

Summary. The Stage-A recognizer transfers to the target demonstrations with 96.4% average accuracy and at least 90% accuracy on every axis. These predictions provide the preference labels used for target adaptation in Stage B, without exposing the model to target preference annotations.

C2: Preference-Controlled Target Adaptation

Simulation. Table II compares Naive FT and SPT across all five axes. SPT improves average preference following from 59.4% to 83.8%, a gain of 24.4 percentage points. Following improves on every axis, with gains ranging from 16 points for placement to 31 points for contact height. Average task success is 89.2% for SPT and 88.0% for Naive FT. Task success is not uniformly higher: SPT is lower than Naive FT on clearance and orientation, despite improving preference following. Section V-D shows that the gain is not explained by the Stage-A initialization alone: finetuning the same Stage-A policy on the same target demonstrations without self-labels substantially weakens preference control.

Preference following on the real robot with task transfer. SPT follows the requested preference in 37 of 40 trials, compared with 21 of 40 for Naive FT (Table III). It follows both clearance requests in all 20 pushing trials and the direct and bank-shot preferences in 17 of 20 billiards trials. Task success is 38 of 40 for SPT and 36 of 40 for Naive FT. These results show that clearance preferences learned from pick-and-place transfer to bowl pushing, while strike preferences learned from air hockey transfer to billiards, without preference annotations on the target demonstrations.

Target task	Naive FT	SPT	Naive FT	SPT
Pushing	95	100	55	100
Billiards	85	90	50	85

Summary. SPT improves preference following over Naive FT by 24.4 percentage points in simulation and 40 points on the real robot, with similar average task success. The improvement holds across all five simulated axes and both real-robot task transfers.

C3: Ablation Studies: Why Are Both Stages Necessary?

Source initialization and target adaptation. Table IV studies placement location and contact height. Source-only evaluates the joint Stage A policy directly on the target tasks without target adaptation. All matched adaptation controls start from the same joint Stage A checkpoint as SPT and use the same 100 target demonstrations and 1500-step adaptation schedule. Same-init plain FT adapts using bare task instructions. random labels randomly permutes the inferred preference labels across target demonstrations while preserving the label histogram. Zero-shot VLM uses labels predicted by the off-the-shelf VLM from § 4-B. SPT instead uses labels from the Stage A recognizer. Naive FT is the full-pipeline baseline defined in § 4-A.

Method	Task Success: Placement	Task Success: Contact	Preference Following: Placement	Preference Following: Contact
Naive FT	80	92	67	59
Source-only	67	85	80	88
Same-init plain FT	69	95	52	50
random labels	66	92	50	55
Zero-shot VLM	70	93	60	50
SPT (Ours)	86	95	83	90
Few-label GT	81	91	75	86
Oracle GT	84	98	98	96

Few-label (10 GT) adapts on ten target demonstrations with ground-truth preference labels; the remaining 90 demonstrations are not used in this run. Oracle GT uses ground-truth preference labels for all 100 target demonstrations. Both start from the same joint Stage A checkpoint. These provide references at two levels of target preference supervision, although Few-label also uses less target action data than the full-data methods.

Effect of target adaptation. Source-only already exhibits preference control on the target tasks, reaching 80% following on placement and 88% on contact height, but achieves task success of only 67% and 85%. Adapting the same Stage A checkpoint using bare target instructions yields 69% and 95% task success, while preference following drops to 52% and 50%. SPT instead reaches 86% and 95% task success while retaining 83% and 90% preference following. These results show that Stage A transfers preference-conditioned behavior, while self-labeled target adaptation improves adaptation to the target tasks without erasing that control.

Effect of label assignment. random labels yields 50% preference following on placement and 55% on contact height, while labels from the zero-shot VLM yield 60% and 50%. SPT reaches 83% and 90%, respectively. Because these conditions use the same Stage A initialization, target demonstrations, and adaptation schedule, the results indicate that target adaptation depends on labels that reliably identify the preference expressed in each demonstration.

Ground-truth supervision. Oracle GT reaches 98% preference following on placement and 96% on contact height, compared with 83% and 90% for SPT. This leaves gaps of 15 and six percentage points to the fully supervised reference. Few-label (10 GT), which uses ground-truth preference labels for only ten target demonstrations, reaches 75% and 86%. Because it also uses substantially less target action data, we treat it as a low-annotation reference rather than a matched comparison of label quality.

Summary. Stage A transfers preference-conditioned behavior to the target tasks, but the source-only policy remains less adapted to the target. Fine-tuning on bare instructions weakens preference control, while Stage B uses inferred labels to adapt without losing that control. random and zero-shot labels perform substantially worse, indicating that effective target adaptation requires reliable preference labels. SPT therefore approaches the fully supervised reference without target preference annotations.

C4: Simultaneous Preference Control

We extend SPT to two simultaneous preferences by combining release height and placement location, yielding four commands: low–center, low–corner, high–center, and high–corner. Source and target demonstrations cover all four combinations, but only source demonstrations have preference labels. The recognizer predicts both labels independently and achieves 100% accuracy on each label and on the complete pair; therefore, inferred and ground-truth target labels coincide and we omit a separate Oracle GT condition.

Table V compares Source-only, Same-init plain FT, random labels, and SPT using the same two-axis Stage A initialization, target demonstrations, and adaptation schedule. We evaluate 40 held-out scenes under all four commands, for 160 rollouts. Placement is evaluated from the released object’s final XY position. Because all four combinations appear in training, this experiment evaluates simultaneous preference control rather than generalization to unseen combinations.

Method	Preference Following: Release	Preference Following: Place	Preference Following: Both	Task Success	Joint Success
Source-only	78.8	71.9	62.5	33.8	29.4
Random labels	43.8	45.6	24.4	91.9	21.9
Same-init plain FT	70.6	61.3	42.5	90.6	40.0
SPT (Ours)	100.0	93.8	93.8	90.6	88.8

SPT follows both preferences in 93.8% of rollouts, compared with 42.5% for Same-init plain FT and 24.4% for random labels. Joint success is 88.8%, versus 40.0% and 21.9%, while SPT and Same-init plain FT have the same 90.6% task success. Thus, the improvement comes from following the requested execution rather than task completion.

Summary. SPT supports two simultaneous execution preferences in this setting, with 93.8% command following and 88.8% joint success without target preference annotations.

Conclusion and Discussion

Conclusion

We studied preference transfer without target preference annotations. SPT jointly learns preference-conditioned action and preference recognition on labeled source tasks, then uses the recognizer to self-label target demonstrations for adaptation. Across five simulated axes, the recognizer achieves 96.4% target accuracy and SPT improves preference following by 24.4 percentage points over Naive FT with similar task success. SPT also transfers preference control across task types on a real robot and supports two simultaneous preferences in simulation. Ablations show that effective target adaptation depends on reliable labels associated with the demonstrated behavior, suggesting that source preference annotations can be reused across target tasks.

Limitations and Future Work

The current formulation assumes a predefined preference vocabulary and target demonstrations that contain the relevant behaviors. Our experiments primarily study binary preferences, with simultaneous control evaluated on two axes whose combinations are observed during training. Extending SPT to continuous or open-vocabulary preferences, unseen preference combinations, and recognition with abstention for out-of-vocabulary behaviors are directions for future work.

Acknowledgment

This work was supported in part by the National Science Foundation under Grant No. 2143435. We also thank the Coordinated Science Laboratory (CSL) at the University of Illinois Urbana-Champaign for providing research facilities to support this work.