WorksComputer Science
PerSPL: Personalized Semi-supervised Preference Learning
Abstract
Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose PerSPL (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Against seven representative baselines across four benchmarks—PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena—PerSPL achieves the highest reported mean accuracy among the evaluated methods for both seen and unseen users. The two latter benchmarks provide real-user preference comparisons in summarization and open-domain chatbot interaction, respectively. These results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.
Introduction
Large language models are expected to serve users with different goals, values, and communication preferences. The best response for one user may therefore not be the best for another Bakker et al. [ [2], Durmus et al. [ [8], Kirk et al. [ [12], Ryan et al. [ [19]. Alignment pipelines commonly learn what people prefer by asking them to compare two responses to the same prompt. A reward model learns to assign higher scores to preferred responses, and these scores guide model training or response selection Christiano et al. [ [7], Ouyang et al. [ [16], Bai et al. [ [1]. Most reward models, however, pool all comparisons and learn a single, population-wide notion of a good response. When users disagree systematically, this aggregate reward can obscure their differences and poorly reflect individual preferences. Personalized reward models instead make the score depend on the user and their prior feedback Poddar et al. [ [18], Chen et al. [ [5], Bose et al. [ [3], Cai et al. [ [4].
Personalizing the score creates a more demanding data requirement. A population model can pool judgments from all annotators, whereas a personalized model must learn about each user from that user’s limited feedback. Every preference label requires the user to inspect and compare responses, so even seen users provide only short histories and a new user must be modeled from a handful of examples. Responses themselves are easier to obtain because they can be generated or collected without additional user feedback. This creates an opportunity to supplement scarce per-user labels with a much larger, shared pool of unpaired responses. Existing personalized reward models improve learning from sparse comparisons by sharing information across users Poddar et al. [ [18], Chen et al. [ [5], Bose et al. [ [3], Cai et al. [ [4], but they do not use this pool as additional supervision.
What is missing is a way to turn the same unpaired response into potentially different training signals for different users.
Pairwise supervision alone does not determine how to pseudo-label individual responses across prompts. It requires a preferred response to score above the rejected response for the same prompt, but it does not fix where either score lies relative to scores for other prompts. Shifting both scores for a prompt by the same amount preserves the pairwise prediction while changing these cross-prompt comparisons. This ambiguity is irrelevant when ranking responses to the same prompt. It matters for an unpaired example, which contains one response and must be judged against a threshold fitted across prompts. Reliable pseudo-labeling therefore requires comparable scores across prompts for each user. Existing semi-supervised preference-learning methods use unlabeled data to train a population-wide reward or policy He et al. [ [10], Kim et al. [ [11], Lee et al. [ [13]. However, these methods do not provide this user-specific score comparability.
We address these challenges with PerSPL (Personalized Semi-supervised Preference Learning), which combines cross-prompt score stabilization with user-specific pseudo-labeling. To make scores more comparable across prompts, PerSPL encourages the two scores in each labeled pair to have mean zero, a mechanism we call pair-mean score centering. It then fits a per-user mapping from reward scores to preference probabilities, so the same response can receive different pseudo-labels across users. PerSPL retains confident pseudo-labels and iteratively trains on them alongside labeled data, extracting additional supervision without collecting new user feedback. Figure 1 shows how labeled and unpaired data interact in this process.
Figure 1. Overview of PerSPL. A small set of labeled comparisons identifies each user’s preferences, while pair-mean score centering makes reward scores more comparable across prompts. A calibrator fitted separately for each user assigns pseudo-labels to confident responses from a shared unpaired pool and discards ambiguous ones. The reward model is then trained iteratively on both labeled and pseudo-labeled data.
We make three contributions:
Personalized learning from unpaired responses. We formulate semi-supervised personalized reward modeling, in which limited per-user comparisons are supplemented by a shared pool of responses without user identities or preference labels. PerSPL converts this pool into user-specific supervision for seen users and for new users given a small labeled support set.
Score centering for cross-prompt pseudo-labeling. We identify a score-coordinate ambiguity that prevents pairwise reward models from directly pseudo-labeling individual responses across
prompts. We introduce pair-mean centering to address this ambiguity, connect it theoretically to cross-prompt and single-response risk, and show experimentally that it enables unpaired training to improve personalized reward learning.
Evaluation across complementary personalization settings. Against seven representative baselines, PerSPL achieves the highest reported mean seen- and unseen-user accuracy among the evaluated methods in every primary setting across four benchmarks, including real-user preferences from Reddit TLDR and Chatbot Arena. Controlled generator-shift experiments show that the overall unlabeled-data benefit persists across all tested mixtures.
Related Work
Personalized reward modeling. Existing personalized reward models address per-user feedback scarcity through architectural sample efficiency, falling into the three paradigms identified by MRM Cai et al. [ [4]: (1) personalized input methods that condition a shared reward on user-specific signals such as personas or learned embeddings Zhao et al. [ [26], induced personas Ryan et al. [ [19], in-context preferences Singh et al. [ [21], or a user-specific latent Poddar et al. [ [18]; (2) personalized parameter methods that add user-specific modules over a shared backbone Li et al. [ [14], Chen et al. [ [5], Bose et al. [ [3], deliberately keeping the per-user parameter count small to remain tractable on tens of pairs; and (3) meta-learning methods that learn an adaptable initialization for few-shot inner-loop updates Cai et al. [ [4], with related downstream work on heterogeneous direct alignment at the policy level Shirali et al. [ [20], Zhang et al. [ [25]. All three paradigms attack per-user scarcity through architecture, sharing the implicit assumption that the only available signal is each user’s labeled preference pairs. In contrast, PerSPL attacks per-user scarcity at the data level, turning each user’s unpaired responses into pseudo-labeled supervision.
Semi-supervised preference learning under limited feedback. A separate line, semi-supervised reward modeling, leverages unlabeled data to mitigate annotation cost, but in the homogeneous-reward setting. SSRM He et al. [ [10] self-trains on unlabeled paired responses, SPA Kim et al. [ [11] iteratively generates responses from the current policy and DPO-self-labels them, and SSPO Lee et al. [ [13] extends pseudo-labeling to unpaired SFT responses around a global threshold KDE-estimated from labeled data (see also Park et al. …). Each method learns one population-wide reward model or policy and applies one global pseudo-labeling rule, with no mechanism for cross-user preference variation. PerSPL is purpose-built for personalized reward modeling with unpaired data, a problem these homogeneous-reward methods cannot address; we therefore omit them as empirical baselines.
Problem Setup
Personalized preference learning. For a user , prompt , and response , a personalized reward model assigns a scalar score . Among responses to the same prompt, the model predicts that the user prefers the one with the higher score. Each labeled comparison records that user prefers to for the same prompt. Writing , the Bradley–Terry model assigns the preference probability and loss
where .
For each user , denotes the labeled comparisons available for that user. For a seen user, these comparisons are available during joint training. For a previously unseen user, they form the small support set used at adaptation, and evaluation uses disjoint comparisons. Thus, we study few-shot personalization, where every user provides some preference feedback, rather than zero-shot personalization. All users also share an unpaired pool of prompt–response examples . Each example contains one response to a prompt and has neither a preference label nor a user identity. We focus on the regime in which this shared response pool is much larger than the labeled comparison set available for any individual user. Our goal is to use the shared pool to improve personalized ranking for both seen and unseen users without collecting additional user feedback. Because an unpaired example has no alternative response, using it requires interpreting its score relative to responses from other prompts.
Cross-prompt score ambiguity. The BT loss constrains the score difference between two responses to the same prompt. It does not constrain their common score location. For any prompt–user offset , the transformed reward function
produces the same preference probability and BT loss. This invariance is harmless for same-prompt ranking because the offset cancels, but it changes how a score compares with responses from other prompts and whether it crosses a pseudo-labeling threshold. The pairwise objective therefore does not identify the cross-prompt score coordinate required for single-response pseudo-labeling.
The usual held-out metric cannot reveal this ambiguity. For held-out comparison , let . PairAcc measures whether the preferred response beats its paired alternative, whereas GlobalAcc diagnoses the cross-prompt ordering relevant to single-response pseudo-labeling:
where and are independently sampled held-out comparisons for the same user. PairAcc evaluates the target same-prompt ranking task. GlobalAcc is a diagnostic of whether preferred and rejected responses remain ordered across prompts. It requires comparability within each user, not across users.
For example, consider two prompts with reward scores
Both same-prompt decisions are correct, so PairAcc is 1, but the comparison fails and GlobalAcc is . Centering each pair around zero maps both score pairs to , preserving PairAcc and yielding GlobalAcc in this example. This contrast motivates the centering mechanism in § 4.
PerSPL turns the shared unpaired pool into user-specific training data through cross-prompt score centering, per-user calibration and pseudo-labeling, and iterative self-training. Figure 1 shows their interaction, and Appendix § 10.§ 3 gives the full training procedure.
Pair-mean score centering
The Problem Setup shows that pairwise supervision leaves the common score location of each labeled pair unidentified. Using the rewards and defined in the Problem Setup, the BT loss depends only on their difference. PerSPL supplies a training signal for their mean with
where the expectation is over users and their labeled comparisons. This penalty acts directly on pair centers rather than output-coordinate margins. Because both are produced by the same reward model, training can still change them jointly. Appendix § 11 characterizes this score-space role through cross-prompt risk, an illustrative Gaussian model, a one-sided fixed-budget analysis, and real-data diagnostics.
Per-user calibration and pseudo-labeling
With this centering signal, PerSPL learns how each user’s reward scores relate to their observed preferences. For each labeled pair in , we assign target 1 to the chosen-response reward and target 0 to the rejected-response reward. We fit a non-decreasing isotonic regressor to these labeled rewards with the Pool-Adjacent-Violators algorithm Zadrozny and Elkan [ [24], Niculescu-Mizil and Caruana [ [15]. Thus, is a monotone empirical estimate of the probability that a score appears on the chosen side of user ’s comparisons. The fit is a stop-gradient target and also defines the per-user score anchor
For an unpaired response, write and choose . PerSPL assigns if , assigns if , and otherwise abstains. The retained triples form , the pseudo-labeled set for user . Their average pointwise loss is
where the expectation is over users and their admitted responses. Only confident responses contribute to this loss; all original comparisons remain in the labeled objective.
Iterative self-training
Because pseudo-labels depend on the current reward model, PerSPL alternates between training the model and updating the pseudo-labels. Within each outer iteration, every remains fixed while the model trains for a fixed number of inner epochs using
where is the average BT loss from Eq. (1) over the labeled comparisons. The first iteration uses only labeled comparisons to produce the initial reward model and calibrators; subsequent iterations combine them with the current pseudo-labeled set.
After each inner epoch, PerSPL refits and using only the current rewards on , so the calibration fit itself uses only labeled examples. At the end of each outer iteration, PerSPL applies to and replaces every using the confidence rule above. The reward model is randomly reinitialized before the next iteration. Algorithm 1 specifies the minibatch sampling, update order, and complete schedule.
Experiments
Our primary evaluation compares PerSPL with seven representative baselines on four benchmarks spanning synthetic and real responses and simulated and real users. We then use ablations and controlled studies to examine the contribution of unlabeled training, its interaction with pair-mean centering, and its behavior under different label budgets, preference noise, and response-distribution shift.
Datasets
The four benchmarks separate response realism from user realism. PersonalLLM Zollo et al. [ [27] uses synthetic responses and Dirichlet-sampled simulated users. HelpSteer2 Wang et al. [ [23] uses human-rated responses with simulated users defined by weights over rating attributes. Reddit TLDR Stiennon et al. [ [22] contains real users and human preferences on summaries. Chatbot Arena Chiang et al. [ [6] contains real users and human preferences over open-domain chatbot responses. PersonalLLM uses , while HelpSteer2 uses . Appendix § 10.§ 1 gives the complete construction and split definitions.
Baselines and Evaluation Protocol
Baselines and architecture. We compare against a homogeneous Bradley–Terry reward model (BT), personalized-input methods (GPO, VPL), personalized-parameter methods (PAL, LoRe), and the meta-learning method MRM with its RPO objective Zhao et al. [ [26], Poddar et al. [ [18], Chen et al. [ [5], Bose et al. [ [3], Cai et al. [ [4]. Each baseline uses its released implementation and a benchmark-level configuration shared across seeds. PerSPL uses the same reward-head architecture as LoRe, providing an architecture-controlled comparison between the two methods. Both operate on the same frozen 4,096-dimensional final-token representation; PerSPL learns a low-rank linear basis and six-dimensional user weights. The default PerSPL configuration uses across benchmarks; ablations explicitly mark deviations. Dataset construction, the exact reward head, optimizer settings, and stage-specific configurations for the main results are reported in Appendix § 10, including Table 5.
| Setting | ||||
| PersonalLLM () | 2 | 0.03 | 0.6 | 0.9 |
| HelpSteer2 () | 5 | 0.01 | 0.7 | 0.6 |
| Reddit TLDR (100/seen) | 2 | 0.03 | 0.7 | 0.8 |
| Reddit TLDR (150/seen) | 2 | 0.03 | 0.7 | 0.8 |
| Chatbot Arena (25/12) | 2 | 0.3 | 0.8 | 0.8 |
Table 5. PerSPL hyperparameters for the main results in Table 1. Here is the basis rank, is the peak learning rate, is the labeled-loss weight in Eq. (7), and is the pseudo-label confidence threshold. Parameters shared by all rows are given in the text.
Evaluation protocol. Seen users are trained jointly, whereas each unseen user is adapted only from its designated support set. All test comparisons are held out. To ensure consistent comparisons, methods use the same assigned splits, labeled budgets, and seed identities. Methods defined on frozen features (Skywork V2, BT, GPO, LoRe, MRM, and PerSPL) use the same cached 4,096-dimensional representations; VPL and PAL retain the native GPT-2 and OPT-350M backbones prescribed by their released implementations. The supervised baselines receive the same labeled comparisons, while PerSPL additionally receives , whose responses carry neither user identities nor preference labels. We report within-prompt pairwise accuracy separately for seen and unseen users. Unless noted, results are mean population standard deviation (ddof=0) over five matched seeds, while the controlled shift study uses ten. Appendix Table 11 additionally reports paired-bootstrap 95% confidence intervals obtained from 20,000 resamples of the matched seed-level differences. VPL follows its released preprocessing by excluding examples above its 1,024-token limit; on Chatbot Arena, it therefore evaluates 555 of the 571 assigned test comparisons in each seed.
Compute. The PersonalLLM, HelpSteer2, and Reddit TLDR experiments used a mix of eight NVIDIA A100 80GB and five RTX A5000 24GB GPUs and required approximately seven hours of wall-clock time across their runs. Chatbot Arena was run separately on the same cluster; the distribution-shift analysis is excluded from this estimate.
Main Results
Table 1 summarizes the full baseline comparison on PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena.
| Method | PersonalLLM Seen | PersonalLLM Unseen | HelpSteer2 Seen | HelpSteer2 Unseen | Reddit TLDR 100 examples/seen user Seen | Reddit TLDR 100 examples/seen user Unseen | Reddit TLDR 150 examples/seen user Seen | Reddit TLDR 150 examples/seen user Unseen | Chatbot Arena 25/12 labels Seen | Chatbot Arena 25/12 labels Unseen |
| Skywork V2 | 81.6±1.6 | 82.4±1.4 | 67.0±0.3 | 67.4±0.3 | 64.5±0.8 | 64.6±0.8 | 64.5±0.8 | 64.6±0.8 | 80.3±1.4 | 76.9±1.4 |
| BT | 88.8±3.2 | 89.0±3.8 | 63.0±3.7 | 63.2±3.8 | 63.2±4.8 | 64.3±3.8 | 64.4±3.5 | 65.2±3.1 | 80.3±2.0 | 78.1±3.0 |
| VPL | 89.6±1.2 | 89.8±1.2 | 59.6±0.7 | 59.6±0.6 | 57.0±0.8 | 56.9±1.3 | 57.6±1.5 | 57.0±1.6 | 64.9±3.1 | 61.9±6.3 |
| GPO | 89.1±6.7 | 89.6±6.8 | 65.8±2.6 | 66.4±2.3 | 63.8±2.5 | 64.6±2.7 | 65.7±1.7 | 65.9±1.3 | 79.4±1.7 | 77.7±1.4 |
| PAL | 84.7±1.4 | 85.2±1.5 | 57.1±1.4 | 56.9±1.0 | 56.9±1.6 | 56.4±1.2 | 57.1±1.8 | 57.5±1.8 | 58.4±3.0 | 58.4±3.0 |
| LoRe | 94.6±0.5 | 94.0±0.9 | 69.6±0.8 | 70.1±1.0 | 65.3±1.5 | 68.4±0.4 | 65.2±0.6 | 67.7±1.0 | 81.3±2.1 | 78.4±2.1 |
| MRM | 91.6±0.6 | 92.0±0.5 | 70.7±0.3 | 70.9±0.4 | 70.0±1.0 | 69.8±0.6 | 70.0±1.1 | 69.6±0.7 | 80.7±1.4 | 77.2±1.6 |
| PerSPL | 95.7±0.2 | 95.5±0.2 | 71.6±0.3 | 71.2±0.8 | 70.7±1.2 | 70.1±0.6 | 70.9±0.9 | 70.3±0.7 | 83.0±3.0 | 81.0±2.4 |
Table 1. Full baseline comparison on PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena. We report pairwise preference accuracy (%) on seen and unseen users as mean ± std over five matched seeds. PersonalLLM and HelpSteer2 use ; Reddit TLDR uses 100 or 150 labeled examples per seen user; Chatbot Arena uses 25/12 labeled pairs per seen/unseen user. On Chatbot Arena, VPL/PAL use their released pooled-pair evaluators; the remaining methods macro-average user accuracies. Best per column is in bold.
Among the methods evaluated in each setting, PerSPL has the highest reported mean in every seen- and unseen-user column. On PersonalLLM, it reaches 95.6 ± 0.1 overall, compared with 94.3 ± 0.7 for LoRe. On HelpSteer2 and Reddit TLDR, PerSPL has higher mean seen- and unseen-user accuracy than the strongest baseline in every reported setting. The Reddit TLDR and Chatbot Arena results extend this pattern to real users and human preferences in summarization and open-domain dialogue. Complete results across all diversity and label-budget settings appear in Appendix § 12.§ 1, including Overall accuracy for Chatbot Arena.
To assess seed-level uncertainty, we also compute absolute accuracy differences between PerSPL and the baseline with the highest overall mean in each setting. The paired-bootstrap intervals exclude zero for every seen-user and overall comparison and for four of the five unseen-user comparisons; Appendix Table 11 reports the complete estimates.
Ablations and Analysis
Component ablations. Table 2 reports a factorial comparison of pair-mean regularization and unpaired training in all four HelpSteer2 and Reddit TLDR settings. Each variant keeps the configuration of the full model and changes only , , or both, in both training phases. With unpaired training off, adding the regularizer changes seen accuracy by between −0.1 and +0.1 points. Without regularization, adding unpaired training changes seen accuracy by −6.5 to −2.9 points; with regularization, it changes seen accuracy by +0.5 to +1.8 points. The resulting interactions range from +3.4 to +8.3 points. For seen users, these contrasts show that centering changes the observed contribution of unpaired training from negative to positive. The large drop after removing regularization therefore reflects the interaction between centering and unpaired training rather than a standalone regularization gain. Appendix § 12.§ 3 reports paired intervals for the component comparisons.
| Objective | HelpSteer2 Seen | HelpSteer2 Unseen | HelpSteer2 Seen | HelpSteer2 Unseen | Reddit TLDR 100 examples/seen user Seen | Reddit TLDR 100 examples/seen user Unseen | Reddit TLDR 150 examples/seen user Seen | Reddit TLDR 150 examples/seen user Unseen |
| PerSPL () | 71.60.3 | 71.20.8 | 66.00.4 | 65.00.7 | 70.71.2 | 70.10.6 | 70.90.9 | 70.30.7 |
| () | 66.50.5 | 69.70.9 | 62.51.2 | 63.20.5 | 63.71.1 | 68.60.7 | 62.52.1 | 62.51.4 |
| () | 70.90.6 | 71.00.5 | 65.50.4 | 64.60.8 | 69.41.4 | 69.51.0 | 69.11.0 | 70.30.7 |
| only (, ) | 70.90.6 | 71.10.7 | 65.40.3 | 64.50.9 | 69.51.3 | 69.71.1 | 69.01.3 | 69.60.7 |
Table 2. Component ablations on HelpSteer2 () and Reddit TLDR (100 or 150 labeled examples per seen user), forming a regularization unpaired-training factorial comparison. Each variant changes only , , or both, in both training phases; all other hyperparameters equal those of the full model. We report seen and unseen accuracy (%), mean std over five seeds. Best per column in bold.
Centering diagnostics. Figure 2 illustrates how pair-mean centering changes the chosen and rejected reward distributions on PersonalLLM, pooled over seen users. Its direct score-space action is on pair centers; shared parameters determine the learned centers and margins jointly. Appendix § 11 characterizes this geometry and the labeled-only regularization path. Moderate centering reduces within-user pair-mean variation and cross-prompt loss while preserving held-out pairwise fit.
Figure 2. Reward distributions of chosen and rejected responses on PersonalLLM (), pooled over all 100 seen users and evaluated on their held-out test pairs (1,000 per user). Both panels use a labeled-only reward head trained identically except for pair-mean regularization ( vs. ). Without regularization, the chosen and rejected distributions overlap substantially. With , they are centered near zero and show less overlap. Appendix § 11 provides the corresponding score-space analysis and regularization-path diagnostics.
Noise and label-budget sweeps. We compare PerSPL with the strongest baseline, MRM, while varying preference noise and the labeled budgets of seen and unseen users on HelpSteer2 (, Figure 3).
Figure 3. Performance under preference noise and varying label budgets on HelpSteer2 (). PerSPL is compared with the strongest baseline, MRM. (a) Test accuracy as the label flip ratio increases. (b) Test accuracy with 50–200 labeled pairs per seen user. (c) Test accuracy with 10–50 labeled pairs per unseen user while fixing the seen-user budget at 50. Shaded bands show standard deviation over five seeds.
Preference noise. We apply a label flip ratio from 0 to 0.5 uniformly across users (Figure 3a). Both methods become less accurate as the noise increases, while PerSPL remains ahead throughout the tested range. The gap is largest at high noise, reaching 5.07 points at a 0.4 flip ratio and 6.69 points at 0.5.
Labeled pairs per seen user. Across 50, 100, and 200 labeled pairs per seen user (Figure 3b), PerSPL improves as more labels become available and remains 0.81–1.08 points above MRM. The gain therefore persists across the displayed seen-user label budgets.
Labeled pairs per unseen user. Holding the seen-user budget fixed at 50 pairs and varying the unseen-user support set from 10 to 50 (Figure 3c), PerSPL improves from 65.51% to 66.03%. Its gain over MRM increases from 0.81 to 1.33 points, supporting transfer to new users across the tested adaptation budgets.
Hyperparameter sensitivity. A controlled one-at-a-time sweep of the iso-confidence threshold , learning rate , and labeled-loss weight on HelpSteer2 () shows that accuracy is stable across the tested and values. The main learning rate is also stable to nearby choices. Full sweep curves and quantitative comparisons are provided in Appendix § 12.4; this post-hoc diagnostic does not determine the main configurations.
Controlled Response-distribution Shift
We test whether unlabeled responses remain useful when their generators differ from those represented in the labeled comparisons. Clean-label PersonalLLM is close to ceiling (Table 1), leaving little room to measure the additional contribution of unlabeled data. We therefore use a fixed 0.2 symmetric label-flip rate to create a non-saturated setting. The labeled data and flip rate remain fixed across all conditions and for the labeled-only reference, so label noise changes the task difficulty without changing the response-shift comparison.
We vary only the generator composition of the unlabeled pool, which follows . The six in-domain generators and two held-out generators are disjoint, and the labeled and unpaired pools do not overlap. We compare PerSPL with the same regularized labeled-only model over ten matched seeds. Because this experiment uses corrupted labels, its absolute accuracies are not directly comparable with the clean-label PersonalLLM row in Table 1.
Across the tested mixtures, PerSPL’s overall gain over the labeled-only reference ranges from 0.66 to 3.21 points (Table 3), and every paired interval is above zero. The overall benefit therefore persists even when the unlabeled pool consists entirely of the tested held-out generators. This evidence is specific to the controlled generator shift studied here. Appendix § 12.§ 3 reports the generator identities and seen/unseen breakdown.
| OOD fraction | PerSPL | Gain over labeled-only [95% CI] |
| 0 | 84.70 1.09 | +3.21 [2.58, 3.82] |
| 0.25 | 84.55 1.02 | +3.07 [2.31, 3.73] |
| 0.50 | 83.63 1.12 | +2.14 [1.32, 2.90] |
| 0.75 | 82.71 0.89 | +1.22 [0.57, 1.84] |
| 1 | 82.15 0.67 | +0.66 [0.10, 1.18] |
Table 3. Controlled generator shift on PersonalLLM with a 0.2 symmetric label-flip rate (overall accuracy, ten matched seeds). The matched regularized labeled-only reference is 81.49 0.79.
Conclusion
We introduced PerSPL to turn a shared pool of responses without user or preference labels into user-specific supervision. The method combines pair-mean score centering, per-user isotonic calibration, confidence filtering, and iterative self-training. Against seven representative baselines across four benchmarks, PerSPL has the highest reported mean accuracy among the evaluated methods for both seen and unseen users. Paired-bootstrap intervals support its overall gains in every evaluated HelpSteer2 and Reddit TLDR setting and on Chatbot Arena, which supplies real-user open-domain preferences.
The controlled generator-shift study further defines the scope of the unlabeled-data benefit. Across all tested mixtures, the overall gain over regularized labeled-only training ranges from 0.66 to 3.21 points, with every paired interval above zero. This evidence applies to the tested generators. Per-user calibration requires a labeled anchor, so the current setting studies few-shot rather than zero-shot personalization. Active selection and shift-aware calibration are natural next steps.
References
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. 1
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203.10050, 2022. 3
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37:52516–52544, 2024. 1, 3, 5, 23
Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Barr Held, and Diyi Yang. Synthesizeme! inducing persona-guided prompts for personalized reward models in llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8045–8078, 2025. 1, 3
Ali Shirali, Arash Nasr-Esfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, and Ariel Procaccia. Direct alignment with heterogeneous preferences. arXiv preprint arXiv:2502.16320, 2025. 3
Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot optimization of synthetic preferences personalizes to real users. arXiv preprint arXiv:2502.19312, 2025. 3
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. 5, 11, 23
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh N Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Processing Systems, 37:1474–1501, 2024. 5, 11, 23
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002. 4
Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. Beyond bradley-terry models: A general preference model for language model alignment. arXiv preprint arXiv:2410.02197, 2024. 3
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523, 2023. 3, 5, 23
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Person-alllm: Tailoring llms to individual preferences. arXiv preprint arXiv:2409.20296, 2024. 5, 10, 23
Experimental Details
This appendix documents the dataset construction, data splits, model architecture, and training configurations used for the main results.
Dataset Construction Details
PersonalLLM. PersonalLLM Zollo et al. [ [27] provides a controllable benchmark with model-generated responses and simulated users. Each user is constructed by sampling a Dirichlet() weight vector over 10 RewardBench reward heads, and pairwise preferences are generated by the corresponding weighted reward. We use 100 seen and 100 unseen users, 9,402 training prompts, and 1,000 held-out test prompts at three diversity levels . Smaller yields sharper, more heterogeneous user weights and thus a harder personalization regime. The unpaired pool is built from source-dataset response candidates with their chosen and rejected labels removed, so no preference signal leaks from into training.
| Dataset | Preference source | Users (seen/unseen) | Main setting | Labeled pairs (seen/unseen) | |
| PersonalLLM | reward-model scores | 100/100 | 45/9 | 5,000 | |
| HelpSteer2 | human attribute ratings | 50/50 | 50/10 | 50,000 | |
| Reddit TLDR | human comparisons | 20/20 | 100 or 150 labels | 100 or 150/50 | 149,632 |
| Chatbot Arena | human comparisons | 19/19 | active users | 25/12 | 3,166 |
Table 4. Dataset and evaluation settings for the primary results in Table 1. Label budgets distinguish the histories of seen users from the support sets used to adapt unseen users.
Reddit TLDR. Reddit TLDR Stiennon et al. [ [22] provides real human preferences on summary quality and serves as our fully real benchmark. Following the MRM benchmark protocol Cai et al. [ [4], we use 20 seen and 20 unseen users, with 100 or 150 labeled pairs per seen user and 50 support pairs per unseen user. Evaluation comparisons are disjoint from these support sets. The unpaired pool consists of the remaining users’ summaries with their preference labels stripped, so the unlabeled signal is drawn from the same response distribution as the labeled set without overlapping preferences.
HelpSteer2. HelpSteer2 Wang et al. [ [23] is the real-human multi-attribute counterpart to PersonalLLM, replacing LLM-judge reward heads with human attribute ratings. The dataset supplies K prompt–response pairs annotated by humans on five attributes (helpfulness, correctness, coherence, complexity, verbosity) on a 0–4 scale. We construct personalized users by sampling Dirichlet() weight vectors over the five attributes; each user’s preference between two responses is determined by their attribute-weighted score. We use 50 seen and 50 unseen users, 50 labeled pairs per seen user, and 10–50 support pairs per unseen user. Remaining prompt–response pairs (with attribute scores stripped) form . We sweep .
Chatbot Arena. We use the Chatbot Arena Conversations dataset Chiang et al. [ [6] and follow Personalized Benchmarking Garbacea et al. [ [9] in restricting attention to active users. We then remove ties, split each user’s ordered comparisons into an 80% training pool and a held-out 20% test set, and retain the 38 users with at least 25 training comparisons. Seen users receive 25 labeled pairs and unseen users 12; all methods use identical assigned splits and budgets. The held-out test set is used only for evaluation. The remaining 3,166 training-pool comparisons form the unpaired response pool after preference labels and user identities are removed.
Implementation Details
Frozen representation and personalized reward head. Following the architecture-controlled setup used for the LoRe comparison, we tokenize each prompt–response sequence to at most 2,048 tokens and cache the final-token hidden representation from Skywork-Reward-V2-Llama-3.1-8B. The resulting dimensional representations remain frozen for PerSPL, LoRe, BT, GPO, and MRM; Skywork V2 uses its pretrained scalar head directly. VPL and PAL instead use the native GPT-2 and OPT-350M backbones in their released code. PerSPL learns a linear basis and user logits . With , the reward head is , where . During seen-user training, and the seen-user logits are optimized jointly. For an unseen user, we copy and freeze the learned basis and fit only that user’s logits from the designated support comparisons.
Baseline implementations. We use the released implementations on the same assigned data splits and preserve each method’s native architecture. A single benchmark-level configuration is fixed across matched seeds. Appendix F lists the official baseline implementations and their licenses.
Optimization and reporting. We use Adam with zero weight decay and a one-cycle cosine learning-rate schedule. The peak learning rate is in Table 5; the warm-up fraction is 0.3, the initial division factor is 25, and the final division factor is 10,000. Each optimizer step samples two users, four labeled comparisons per user (two on Chatbot Arena), and, after the first outer iteration, up to 16 pseudo-labeled responses per user. Pseudo-label refreshes score 1,000 candidates per user from (up to 100 on Chatbot Arena). All main configurations use and per-user isotonic calibration. We run matched seeds and report the saved last-epoch checkpoint at the specified budget; no epoch is selected separately for an individual seed. Reported standard deviations use the population convention (ddof=0) over the executed seeds.
A single configuration is shared by all five seeds within each row. Appendix § 12.4 reports a one-at-a-time sensitivity analysis around the main configuration.
PerSPL Training Procedure
The full PerSPL training procedure is given in Algorithm 1.
Algorithm 1 PerSPL: Personalized Semi-supervised Preference Learning
Require: Labeled sets , shared unpaired set , reward model , confidence threshold , weights and , outer iterations , inner epochs
Initialize for every
for do
Reinitialize
for epoch do
for each mini-batch step do
Sample and for each
Compute and on using Eqs. (1) and (4)
if then
else
Sample for each and compute using Eq. (6)
end if
Update with one Adam step on
end for
for each user do
Fit isotonic regressor on and set
end for
end for
for each user do
for each do
if then
else if then
end if
end for
end for
end for
Theory and Diagnostics for Pair-Mean Centering
This appendix characterizes the score-space role of pair-mean centering and its effect on labeled-only reward-head training. We first relate pair means to cross-prompt risk. We then use an idealized Gaussian setting to exhibit how center covariance can change the optimized scoring direction. Finally, we derive the one-sided response of a fixed-budget Adam checkpoint and measure it on real data. The factorial ablation in § 7.4 separately measures how centering interacts with iterative unpaired training.
Pair-mean geometry and cross-prompt risk
Fix one user and draw a labeled pair . Let
The paired risk is and the pair-mean penalty is . Let be an independent pair from the same user. The cross-prompt logistic risk is
This risk is a smooth surrogate for GlobalAcc. It compares a chosen response and a rejected response drawn from different pairs.
Proposition 1 (Cross-prompt control by pair means). If the scores have finite second moments, then
The inequality allows arbitrary dependence between and within a pair.
Proof. Set and . Since ,
Exchanging the two sampled pairs keeps fixed and changes the sign of . The expected linear term is therefore zero, including when and are dependent within a pair. Convexity gives , while independence across the two draws gives . The second inequality follows from .
Why penalize the pair mean? The implemented second moment decomposes as
It therefore controls the dispersion term in Eq. (10) and selects zero as the common score anchor. Writing a score pair as makes the direct action explicit: penalizes the common score component without directly penalizing the contrast . It is therefore a simple pair-local quadratic signal for the score component absent from the BT objective. Shared model parameters still determine the learned centers and margins jointly.
The same coordinates relate centering to individual-response classification. For this fixed user, let be a score threshold shared across prompts. Draw uniformly from , and set when and otherwise. Define the balanced single-response logistic risk
This population risk is an analysis surrogate for classifying one response as chosen or rejected; it is not an additional training objective. The factor two places its margin on the BT scale: when , both logistic arguments equal . In general, the two arguments are . Convexity and the same smoothness argument used above therefore give
Equation (10) shows that pair-center dispersion controls the gap between within-pair BT risk and cross-prompt ranking risk. Equation (14) gives the analogous link to single-response classification around a shared threshold . Thus the midpoint penalty controls a score component that within-pair BT comparisons do not explicitly constrain, but that both cross-prompt and single-response decisions use.
Scope of the score ambiguity. An additive prompt offset cancels in same-prompt selection, including best-of-, but remains when a shared threshold compares responses across prompts. Pair-mean centering supplies an explicit location signal for this second setting, and GlobalAcc measures the resulting cross-prompt ordering.
An illustrative Gaussian model
We next use an idealized Gaussian setting to illustrate in closed form how center covariance can change the optimized scoring direction. It provides a concrete regime where the midpoint penalty improves single-response decisions. For a linear scorer , write
Then , so for any distribution with , the penalty defined above becomes
Thus pair-mean centering is a data-dependent quadratic penalty on sensitivity to directions shared within a pair.
Assumption 2 (Gaussian center–contrast model). For one user,
where and are independent, and are positive-definite covariance matrices, and is the mean contrast.
The condition isolates covariance-driven variation in pair centers. If , the implemented penalty instead uses , and the local sign depends on the combined covariance and anchor geometry. Proposition 1 does not require either the Gaussian or mean-zero condition.
Let . For a pair-mean coefficient , define
and write for the solution at . For the balanced single-response task defined above, the zero-threshold accuracy of the linear scorer is
Below, means that is a positive scalar multiple of .
Theorem 3 (Local single-response accuracy improvement under the Gaussian model). Under Assumption 2, the direction maximizing is . The solutions in Eq. (18) satisfy
If is not proportional to , then
Thus, for small positive , the midpoint penalty yields a first-order gain in single-response accuracy while incurring at most a second-order increase in paired BT risk. The nonalignment condition distinguishes a directional correction from a pure rescaling. If is proportional to , the penalty preserves the BT direction to first order. Otherwise, it moves the optimized direction toward higher single-response accuracy.
Proof. Gaussian integration by parts gives
The first-order condition for Eq. (18) yields Eq. (20). Assumption 2 also gives
where is the standard normal cumulative distribution function. Its maximizing direction follows from the generalized Rayleigh quotient.
It remains to determine the initial direction of change. Define and . For , direct differentiation of the signal-to-noise ratio gives
By Cauchy–Schwarz, this derivative is positive exactly when the stated nonalignment condition holds. Along the regularized path, has derivative at zero, which proves the first statement in Eq. (21). The gradient of vanishes at , so its Taylor expansion along the same path is . Nonnegativity follows because minimizes .
Population-path illustration. We evaluate the two-dimensional center–contrast model in Assumption 2 under three settings:
unequal center variance: , , ,
misaligned covariance axes: , , ,
direction-preserving control: , , .
The first two settings satisfy the nonalignment condition in Theorem 3; the control does not. We solve the population objective in Eq. (18) by Gauss–Hermite quadrature along a fine grid near . For a common horizontal scale, let denote the coefficient satisfying . By Eq. (20), this gives , the single-response-optimal direction. Its values are 0.4688, 0.4748, and 0.4980 in the three settings, respectively. Figure 4 focuses on the local regime required by the theorem.
Figure 4. Direct visualization of Theorem 3 along the population regularization path. On log–log axes, the two nonaligned settings have a locally linear single-response accuracy gain (left; slope 1), while the paired BT excess risk is locally quadratic (right; slope 2). The direction-preserving control has exactly zero accuracy gain and is therefore shown only in the risk panel. Here is the coefficient whose population direction is single-response optimal; only the small- neighborhood covered by the theorem is plotted.
The two panels expose the theorem’s asymmetric local tradeoff: moving away from the BT solution changes single-response accuracy at first order but BT risk only at second order. The control clarifies that regularization alone is not sufficient; the gain requires the nonalignment condition that rotates the scoring direction.
Turning on centering under a fixed training budget
The Gaussian calculation concerns an optimized population model. PerSPL instead returns the Adam checkpoint after a fixed training budget. We analyze the local dependence of this checkpoint on the centering coefficient. Fix the initialization, minibatch sequence, learning-rate schedule, and Adam hyperparameters. At step , let
where is the minibatch BT loss and is the corresponding pair-mean penalty. With first and second moments and , Adam updates
where denotes coordinatewise multiplication and the square root and division are also coordinatewise, , , , and . A superscript denotes the trajectory at .
Proposition 4 (One-sided effect of turning on centering under Adam). *Fix a finite training horizon . Suppose , , and the evaluation target are twice continuously differentiable near the unregularized parameter trajectory, and is independent of . Define and . For any quantity along the training path, write
These right derivatives exist. Starting from , they satisfy
The bias-corrected derivatives are and . For each coordinate , the parameter tangent obeys
As the centering coefficient approaches zero from the right, ,
Consequently, is sufficient for the evaluation target to decrease for all sufficiently small .
Proof. Suppose the state has a right derivative through step . Differentiating Eq. (25) gives . Differentiating Eq. (26) gives the remaining recursions in Eq. (28).
It remains to differentiate the normalized Adam direction. If , ordinary differentiation gives the first branch of Eq. (29). If , the nonnegative second-moment recursion and its zero initialization imply that the unregularized gradients of coordinate have been zero through step . Its first moment is also zero. Smoothness and the induction hypothesis therefore give
For ,
which proves the second branch. Equation (27) now gives , completing the induction. Finally, a one-sided expansion of gives , which proves Eqs. (30)–(31).
The recurrence traces each midpoint gradient through the subsequent curvature and Adam moment updates. Equation (31) then gives the exact initial change in a smooth evaluation target. Its sign is determined by the data, representation, and training path. The matched one-sided runs below estimate this quantity for the Adam training path used by PerSPL.
Real-data checks
We isolate the effect of centering on labeled-only reward heads. We use HelpSteer2 with 50 seen users, rank five, and 50 labeled pairs per user, and Reddit TLDR with 20 seen users, rank two, and 150 labeled pairs per user. Every setting uses seeds 42–46 and last-epoch checkpoints.
Regularization path. We evaluate the same matched training path at two scales. To measure the initial effect of centering, we first train with Adam at while holding the initialization, minibatch sequence, and training horizon fixed. For a positive coefficient , the matched forward difference is
The held-out pairs are deterministically divided into two halves, with at most 256 pairs per user in each half. We set to on the primary held-out half and repeat the measurement on the disjoint half. Table 6 reports these slopes together with the corresponding changes in pair-mean variation.
| Dataset | Primary alignment | Disjoint alignment | reduction per unit | |
| HelpSteer2 | ||||
| HelpSteer2 | ||||
| Reddit TL;DR | ||||
| Reddit TL;DR |
Table 6. Matched one-sided differences at for the actual fixed-budget Adam training path. The primary and disjoint columns estimate on two held-out splits; the final column reports the reduction in held-out within-user pair-mean variance per unit . Entries are means paired 95% intervals over five seeds. Positive values denote improvement.
The primary held-out alignment is positive for every seed, dataset, and positive coefficient. At , is on HelpSteer2 and on Reddit. The disjoint-half estimates are and , respectively. Increasing the coefficient to gives primary-half estimates of and . Thus the mean primary-half slopes change by 1.6% and 2.8% across the two coefficients. Held-out also decreases for every seed and coefficient.
These slopes characterize the neighborhood of zero. To measure finite changes, we extend the same matched path to . Held-out pairs are split deterministically into the same two halves. Shaded regions in Figure 5 show paired 95% intervals over the five seeds.
Figure 5. Real-data regularization paths for labeled-only reward heads. Panels (a) and (b) report the reductions in held-out cross-prompt logistic loss and paired BT loss, respectively, relative to the matched model. Solid lines show means and shaded regions show paired 95% intervals over five seeds. Positive values denote lower loss.
This diagnostic sweep isolates the effect of ; all main comparisons use the fixed coefficient reported in § 7.2.
At this coefficient, held-out PairAcc changes by points on HelpSteer2 and points on Reddit. This is consistent with the penalty acting on a cross-prompt score coordinate rather than directly changing same-prompt comparisons. Precision among the highest-confidence 10% of candidate pseudo-labels rises by points on HelpSteer2. The corresponding held-out-response proxy on Reddit changes by points, with an interval containing zero. All intervals are paired 95% intervals over five seeds.
Figure 5(a) reports the held-out cross-prompt loss reduction relative to the matched model. Positive values therefore indicate improved cross-prompt ranking. All 70 nonzero- seed comparisons are positive. At , the mean reductions are 0.159 on HelpSteer2 and 0.093 on Reddit.
Figure 5(b) examines whether centering compromises held-out pairwise prediction. It reports , so positive values indicate lower loss and negative values indicate higher loss. At , the mean gains are 0.0064 on HelpSteer2 and 0.0021 on Reddit. For comparison, the values at are and . At , the per-unit BT-loss reductions are and , respectively; after multiplication by , both are absolute changes below . Both datasets therefore improve at the coefficient used in the paper.
Together, these results characterize pair-mean centering at three levels. The distribution-free bounds identify the score component used by cross-prompt and single-response decisions. The Gaussian model gives a concrete regime in which this component changes the optimized direction favorably. The Adam recurrence and matched runs establish the initial and finite effects along the actual labeled-only training path. Table 2 supplies the complementary full-procedure evidence: centering has little standalone effect on PairAcc, but changes the observed contribution of iterative unpaired training from negative to positive.
Additional Results and Robustness Analyses
Full Per-dataset Results
We report full pairwise preference accuracy on each benchmark for every method, including the per-dataset Overall column (per-seed average of seen and unseen accuracy) that is omitted from Table 1 for space. All numbers are mean std over five random seeds. The seen/unseen entries here match Table 1 on the matched settings ( for PersonalLLM and HelpSteer2; 100 and 150 labeled examples/seen user for Reddit TLDR; 25/12 for Chatbot Arena). We additionally report the harder diversity levels ( for PersonalLLM, for HelpSteer2) and the Reddit baselines omitted from Section 5.3 for space (Skywork-Reward V1, SynthesizeMe variants, and MRM with a V1 backbone). Best per column is in bold.
| Method | Seen | Unseen | Overall | Seen | Unseen | Overall | Seen | Unseen | Overall |
| Skywork-Reward V2 | |||||||||
| BT | |||||||||
| VPL | |||||||||
| GPO | |||||||||
| PAL | |||||||||
| LoRe | |||||||||
| MRM | |||||||||
| PerSPL |
Table 7. Full PersonalLLM results across (45 pairs/seen user). Mean std over five seeds.
| Method | Seen | Unseen | Overall | Seen | Unseen | Overall |
| Skywork-Reward V2 | ||||||
| BT | ||||||
| VPL | ||||||
| GPO | ||||||
| PAL | ||||||
| LoRe | ||||||
| MRM | ||||||
| PerSPL |
Table 8. Full HelpSteer2 results across (50 pairs/seen user). Mean std over five seeds.
Full Chatbot Arena Results
Chatbot Arena is evaluated as the fourth main benchmark in Table 1. Following Personalized Benchmarking Garbacea et al. [ [9], we restrict attention to active users in the Chatbot Arena Conversations dataset Chiang et al. [ [6].
After removing ties, we split each active user’s ordered comparisons into an 80% training pool and a held-out 20% test set. We retain 38 users with at least 25 training comparisons. Each seed randomly assigns 19 users to each of the seen and unseen sets. Seen users receive 25 labeled pairs, while unseen users receive 12 support pairs. Only the training pool supplies labeled, support, or unpaired responses, and all methods use the same assigned splits, budgets, and seeds 42–46.
Table 9: Full Reddit TLDR results at 100 and 150 labeled examples/seen user. Mean std over five seeds. SynthesizeMe variants are inference-time-personalization methods (no in-domain training); MRM is reported with both Skywork-Reward V1 and V2 backbones.
| Method | 100 examples/seen user: Seen | 100 examples/seen user: Unseen | 100 examples/seen user: Overall | 150 examples/seen user: Seen | 150 examples/seen user: Unseen | 150 examples/seen user: Overall |
| Skywork-Reward V1 | ||||||
| Skywork-Reward V2 | ||||||
| BT | ||||||
| VPL | ||||||
| GPO | ||||||
| PAL | ||||||
| LoRe | ||||||
| SynthesizeMe (ICL) | ||||||
| SynthesizeMe (FT) | ||||||
| MRM (Skywork-Reward V1) | ||||||
| MRM (Skywork-Reward V2) | ||||||
| PerSPL |
Table 9.
Table 10: Full Chatbot Arena results. We report pairwise accuracy (%, mean std over five matched seeds); Overall is the per-seed average of seen and unseen accuracy. VPL/PAL use their released pooled-pair evaluators; the other methods macro-average users. All five prespecified seeds are retained.
| Method | Seen | Unseen | Overall |
| Skywork-Reward V2 | |||
| BT | |||
| VPL | |||
| GPO | |||
| PAL | |||
| LoRe | |||
| MRM | |||
| PerSPL |
Table 10.
Skywork V2 is frozen, and the union of seen and unseen users is fixed across seeds. Only the assignment of those users changes, so its per-seed overall accuracy is identical and the reported overall standard deviation is zero.
PerSPL has the highest reported mean accuracy for both seen and unseen users. Relative to LoRe, the baseline with the highest overall mean, its paired gains are 1.72 points [0.14, 3.45] for seen users, 2.66 points [1.13, 4.28] for unseen users, and 2.19 points [0.79, 3.20] overall. The brackets give paired-bootstrap 95% confidence intervals from 20,000 resamples of the five matched seed differences. All three intervals exclude zero.
Additional Analyses
Paired uncertainty. For each comparison in Table 11, we retain the seed pairing between PerSPL and the baseline with the highest overall mean in that setting and draw 20,000 bootstrap resamples of the five paired seed differences. The interval is the 2.5th–97.5th percentile range and is a descriptive summary of variation over the five exact matched seeds.
Table 11: PerSPL minus the baseline with the highest overall mean in each setting, measured in accuracy percentage points with descriptive paired-bootstrap 95% confidence intervals over five matched seeds.
| Setting | Seen | Unseen | Overall |
| HelpSteer2 | +0.91 [0.62, 1.21] | +0.35 [0.04, 0.65] | +0.62 [0.34, 0.93] |
| HelpSteer2 | +1.22 [0.57, 2.01] | +0.37 [−0.43, 1.18] | +0.81 [0.11, 1.62] |
| Reddit, 100/seen | +0.78 [0.28, 1.35] | +0.38 [0.15, 0.60] | +0.56 [0.39, 0.76] |
| Reddit, 150/seen | +0.91 [0.31, 1.51] | +0.74 [0.40, 1.08] | +0.81 [0.48, 1.05] |
| Chatbot Arena, 25/12 | +1.72 [0.14, 3.45] | +2.66 [1.13, 4.28] | +2.19 [0.79, 3.20] |
Table 11.
Factorial analysis. Across the four settings in Table 2, the regularizer-only change ranges from −0.1 to +0.1 seen-accuracy points. Conditional on centering, adding the unpaired loss changes seen accuracy by +0.7 [0.25, 1.07] and +0.5 [0.46, 0.63] on HelpSteer2 ( and 0.01), and by +1.3 [0.54, 2.62] and +1.8 [0.79, 2.81] on Reddit TLDR (100 and 150 examples/seen user). Brackets give paired-bootstrap 95% intervals over five matched seeds, and all four seen-user intervals exclude zero. The corresponding unseen-user changes are +0.2 [−0.10, 0.47], +0.4 [−0.13, 0.95], +0.6 [0.06, 1.40], and 0.0 [−0.45, 0.48]. Without centering, adding unpaired training changes seen accuracy by −4.4, −2.9, −5.8, and −6.5 points in the same four settings.
Controlled generator shift. For PersonalLLM, the in-domain generators are GPT-4o, Claude 3 Opus, Claude 3 Sonnet, Llama 3 70B, Gemini Pro 1.5, and Mixtral 8x22B; Command R+ and GPT-4 Turbo are held out as the response-generator shift. Labeled and unpaired sets are disjoint, labels have a 0.2 symmetric flip rate, and all rows use ten matched seeds. Table 12 gives the seen/unseen breakdown.
| Setting | Seen | Unseen |
| Labeled only | 80.31 ± 1.37 | 82.66 ± 0.67 |
| 84.56 ± 1.32 | 84.83 ± 1.07 | |
| 84.44 ± 1.14 | 84.66 ± 1.14 | |
| 83.55 ± 1.15 | 83.71 ± 1.38 | |
| 82.58 ± 1.00 | 82.83 ± 1.19 | |
| 81.85 ± 0.86 | 82.45 ± 0.84 |
Table 12. Seen- and unseen-user breakdown for the controlled generator-shift study in Table 3. Overall results and paired gains are reported in the main table.
Hyperparameter Sensitivity Analysis
To complement the qualitative sensitivity discussion in § 7.4, we report a controlled one-at-a-time sweep of the iso-confidence threshold , learning rate , and labeled-loss weight . The sweep is run on HelpSteer2 (). When one hyperparameter is varied, the other two are held at the main values , , and . The training schedule and number of self-training iterations are also fixed. This post-hoc diagnostic does not determine the configurations used in Table 1. Each point is the overall test accuracy averaged across seen and unseen users over five matched seeds; shaded bands show ± one standard deviation.
Findings. The threshold and labeled-loss weight have little effect within the tested ranges (Figure 6). Across , overall accuracy spans only 0.30 percentage points. In particular, , which effectively removes confidence-based abstention, is 0.19 points below the main setting. Across , the corresponding range is 0.23 points. The learning rate has a clearer effect. Values remain within 0.30 points of one another, and the main value is 2.75 points above . Thus, the confidence threshold, labeled/unlabeled loss weighting, and the learning-rate neighborhood used by PerSPL are stable in this setting.
Figure 6. Hyperparameter sensitivity of PerSPL on HelpSteer2 (). Stars mark the main configuration (, , ). Overall accuracy changes little across the tested and values and is stable near the main learning rate.
Scope and Limitations
Labeled anchors and information budgets. Per-user isotonic calibration requires labeled comparisons for every user to whom the method is applied. Zero-label cold start is therefore out of scope. Our seen-user histories and unseen-user support sets play different roles and must not be conflated: the former learn shared structure, while the latter adapt that structure to a target user. Mining user-generated content could reduce explicit annotation cost, but would introduce privacy and consent issues and would require an evaluation with a matched total information budget.
Selective pseudo-labeling. Confidence filtering intentionally abstains on ambiguous responses and retains all original labeled comparisons. It can nevertheless overrepresent easy or well-calibrated response regions. We test the all-response alternative and report accuracy, but do not yet measure subgroup-specific pseudo-label precision and coverage; deployment should audit both.
Distribution shift. The response-shift experiment holds labeled data fixed and varies a mixture of six in-domain and two held-out generators. It does not establish robustness to arbitrary semantic, demographic, adversarial, or temporal shift. Recalibration or abstention may be required when the unlabeled pool is far from the labeled support.
Architecture and scale. Experiments use a frozen reward-model representation with a learned low-rank reward basis. The findings may change with end-to-end fine-tuning or a much larger personalized backbone; a capacity-matched full-fine-tuning control is not included. The user populations contain at most 200 users, so computational and statistical behavior at production scale remains uncharacterized.
Language and tasks. The benchmarks are English-language and focus on instruction following, open-domain chatbot comparisons, and summarization. Cross-lingual transfer, multi-turn interaction, code, and long-horizon consequences are not evaluated.
Data and Software
We rely on four publicly available preference-learning benchmarks and on the released implementations of every baseline we compare against. All assets are used in accordance with their respective licenses and access terms, including the non-commercial restriction of LoRe Bose et al. [ [3] (academic research use only). Licenses are listed as reported by each asset’s official source at the time of writing.
Datasets. We use PersonalLLM Zollo et al. [ [27] (dataset card; CC-BY-4.0), HelpSteer2 Wang et al. [ [23] (dataset card; CC-BY-4.0), Reddit TLDR Stiennon et al. [ [22] (official license; Modified MIT for the human-feedback release, with the underlying filtered Reddit TL;DR corpus separately licensed under CC-BY-4.0), and the gated Chatbot Arena Conversations release Chiang et al. [ [6], with active-user filtering motivated by Personalized Benchmarking Garbacz et al. [2026]. Chatbot Arena user prompts are CC-BY-4.0 and model outputs are CC-BY-NC-4.0, and its access terms prohibit attempts to identify users.
Baseline implementations. We use the official released code for GPO Zhao et al. [ [26] (MIT), VPL Poddar et al. [ [18] (license not specified), PAL Chen et al. [ [5] (Apache-2.0), LoRe Bose et al. [ [3] (CC-BY-NC-4.0), and MRM Cai et al. [ [4] (MIT).
Pre-trained reward model. We extract reward-model embeddings from Skywork-Reward-V2-Llama-3.1-8B (Llama 3.1 Community License), which is derived from meta-llama/Llama-3.1-8B-Instruct and inherits its license terms.
References
- [1]Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., & al., . et . (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. In arXiv preprint arXiv:2204.05862. arXiv:2204.05862
- [2]Bakker, M., Chadwick, M., Sheahan, H., Tessler, M., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M., & al., . et . (2022). Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35, 38176–38189.
- [3]Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. Lore: Personalizing llms via low-rank reward modeling. arXiv preprint arXiv:2504.14439, 2025.arXiv
- [4]Hongru Cai, Yongqi Li, Tiezheng Yu, Fengbin Zhu, Wenjie Wang, Fuli Feng, and Wenjie Li. One adapts to any: Meta reward modeling for personalized llm alignment. arXiv preprint arXiv:2601.18731, 2026.arXiv
- [5]Chen, D., Chen, Y., Rege, A., Wang, Z., & Vinayak, R. K. (2025). Pal: Sample-efficient personalized reward modeling for pluralistic alignment. The Thirteenth International Conference on Learning Representations.
- [6]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024.arXiv
- [7]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03741
- [8]Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., & al., . et . (2023). Towards measuring the representation of subjective global opinions in language models. In arXiv preprint arXiv:2306.16388. arXiv:2306.16388
- [9]Garbacea, C., Wang, H., & Tan, C. (2026). Personalized benchmarking: Evaluating llms by individual preferences. Findings of the Association for Computational Linguistics: ACL 2026, 639–675.
- [10]Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. Semi-supervised reward modeling via iterative self-training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7365–7377, 2024.arxiv.org/abs/2409.06903
- [11]Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Spread preference annotation: Direct preference judgment for efficient llm alignment. arXiv preprint arXiv:2406.04412, 2024.arXiv
- [12]Kirk, H. R., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., Mosquera, R., Bartolo, M., Williams, A., He, H., & al., . et . (2024). The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances in Neural Information Processing Systems, 37, 105236–105344.
- [13]Seonggyun Lee, Sungjun Lim, Seojin Park, Soeun Cheon, and Kyungwoo Song. Semi-supervised preference optimization with limited feedback. arXiv preprint arXiv:2511.00040, 2025.arXiv
- [14]Li, X., Zhou, R., Lipton, Z. C., & Leqi, L. (2024). Personalized language modeling from personalized human feedback. In arXiv preprint arXiv:2402.05133. arXiv:2402.05133
- [15]Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625–632, 2005.DOI
- [16]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., & al., . et . (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.
- [18]Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37:52516–52544, 2024.arxiv.org/abs/2408.10075
- [19]Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Barr Held, and Diyi Yang. Synthesizeme! inducing persona-guided prompts for personalized reward models in llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8045–8078, 2025.arxiv.org/abs/2506.05598
- [20]Ali Shirali, Arash Nasr-Esfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, and Ariel Procaccia. Direct alignment with heterogeneous preferences. arXiv preprint arXiv:2502.16320, 2025.arXiv
- [21]Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot optimization of synthetic preferences personalizes to real users. arXiv preprint arXiv:2502.19312, 2025.arXiv
- [22]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020.arxiv.org/abs/2009.01325
- [23]Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh N Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Processing Systems, 37:1474–1501, 2024.DOI
- [24]Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 694–699).
- [25]Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. Beyond bradley-terry models: A general preference model for language model alignment. arXiv preprint arXiv:2410.02197, 2024.arXiv
- [26]Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523, 2023.arXiv
- [27]Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Person-alllm: Tailoring llms to individual preferences. arXiv preprint arXiv:2409.20296, 2024.arXiv