WorksComputer Science
Rethinking Visual Reasoning in Text-to-Image Reward Modeling
Abstract
Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.
Introduction
Text-to-image generative models can produce compelling visuals from natural language prompts [61, 8, 22], yet faithfully rendering all elements — multiple objects, their attributes, spatial arrangements, and quantities — while satisfying human aesthetic preferences remains challenging [57, 29, 27]. It is standard to utilize a reward model during post-training or inference-time selection to optimize generations towards desired behaviors [44, 7, 66]. A reward model must accomplish two intertwined objectives: it must assess whether the generated image aligns with the semantic guidance of the prompt, and it must capture human preferences regarding aesthetics. Visual understanding is crucial to assessing alignment, and a prerequisite for grounded aesthetic judgments. A reward model that cannot reliably parse the contents of an image, count objects, identify spatial relationships, or bind attributes to entities will struggle to judge whether the image faithfully realizes the user’s intent.
Despite the centrality of visual understanding to reward modeling, we find that popular text-to-image reward models are remarkably poor visual reasoners. We introduce evaluations probing compositional reasoning, positional understanding, and numeracy, revealing that models like ImageReward, PickScore, and HPS perform near chance on basic tasks. A natural remedy is to leverage modern vision-language models (VLMs) as backbones, as they are substantially stronger compositional understanding. However, we find that fine-tuning VLMs on preference data erodes the very capabilities that make them attractive. Continual learning techniques — parameter-efficient fine-tuning, weight merging, and uncertainty calibration — either fail to adequately preserve visual understanding or offer unfavorable tradeoffs with preference alignment.
These findings point to a data problem. Conventional preference datasets pair a single prompt with two images of differing quality, entangling prompt adherence with aesthetics. Because the two images often differ simultaneously in style, artifacts, and semantic correctness, a reward model can obtain high preference accuracy by exploiting aesthetic shortcuts rather than learning robust visual parsing. Simply scaling such data does not resolve the issue: HPSv3 is trained on 1.3M preference pairs, yet still underperforms its base model on visual understanding. We therefore propose a data-centric intervention that isolates the alignment signal that depends on visual reasoning. Our key insight is to flip the pairing axis: construct preference pairs that share a common image but differ in prompt. Holding the image fixed removes aesthetic confounds and forces the reward model to judge which prompt the image actually satisfies. This formulation maps naturally onto existing visual reasoning benchmarks, which can be converted into shared-image preference pairs without additional annotation. Finally, we mix these shared-image pairs with conventional preference data during training.
Our resulting reward model achieves a Pareto improvement, matches or exceeds the accuracy of existing reward models on held-out preference pairs while substantially outperforming them on compositional understanding tasks. We evaluate our model as a reward signal for both inference-time scaling via best-of- selection and for reinforcement learning post-training with DiffusionNFT [85]. Our reward model strongly outperforms baselines on GenEval [21], which explicitly tests challenging prompts that require precise rendering of object and attribute relations, by an average of 4.0% in the best-of- setting and an average of 5.5% in the RL setting. Qualitative examples from generic user prompts show that RL on our reward model yields aesthetic generations with superior prompt adherence than baselines.
Preliminaries
We briefly review the role of reward models in text-to-image generation. Let be a pretrained generative model that produces an image conditioned on a text prompt . We consider reward models of the form that produce a continuous scalar score of quality. These reward models may be leveraged in inference-time scaling to filter, rerank, and select the best generation, e.g.
where are candidates drawn from . More complex test-time selection schemes include optimizing ’s noise latents or steering its score field [44, 65, 5]. The reward model can also be leveraged in reinforcement-learning post-training to update towards higher reward generations:
where controls the strength of regularization towards a reference model (typically taken to be the pretrained generative models). Many RL algorithms for optimizing text-to-image models exist [7, 40, 85], all relying on a reward signal to robustly guide the optimization.
Reward models are generally calibrated on datasets of pairwise human preferences comprising a preferred and dispreferred generation for prompt . To train , we adopt the standard Bradley-Terry formulation [9] for human preferences, which models the probability that is preferred over as
which induces a maximum likelihood objective for :
Widely-Used Reward Models are Poor Visual Reasoners
Reward models for text-to-image generation require visual understanding to assess whether a generated image faithfully depicts its conditioning prompt. In this section, we evaluate existing reward models on core visual reasoning competencies and find them surprisingly deficient.
Evaluation Protocol. We consider three fundamental aspects of visual understanding: compositional reasoning (recognizing multiple objects with correct attribute binding), spatial understanding (parsing positional relationships between entities), and numeracy (counting objects). These represent minimal requirements for verifying text-image alignment. Reward models can be evaluated by their accuracy over a preference pair dataset, as in Equation (3). However, evaluating visual understanding through generated images introduces confounds: differences in aesthetic quality, artifacts, or stylistic variation may dominate the preference signal, obscuring whether the model can parse image content. To isolate visual understanding, we invert the evaluation: we fix the image and assess the preference between two candidate prompts (correct) and (incorrect):
As both and are naturally formed, this formulation localizes the preference judgment to the reward model’s ability to reason visually about the correspondence between image and text.
Benchmarks. We convert a diverse suite of visual understanding benchmarks in short-answer and multiple-choice format into “preference pairs”, where is assigned to the gold answer and to its perturbation or an incorrect multiple-choice alternative. Winoground [62] tests compositional understanding through image-caption pairs with the same entities in different arrangements. SugarCrepe [25] tests basic image-text matching across three simple perturbation types: Add (inserting spurious objects or attributes), Replace (substituting objects or attributes), and Swap (exchanging attributes between objects). CVBench [63] comprises visual-centric VQA; we convert to pairs its Relation and Count subsets for spatial relationships and numeracy respectively. WhatsUp [30] tests spatial reasoning with both absolute (“object in the top-left”) and relative (“A is above B”) descriptions. CountBench [52] tests counting ability up to ten. We also report accuracy on the ImageRewardDB test set [77], to assess performance on a representative preference dataset.
Findings. Table 1 reports the accuracy of widely-used reward models on this benchmark suite, revealing striking deficiencies in basic visual comprehension. CLIP, ImageReward, PickScore, and HPSv2 perform near chance on several tasks: Winoground for compositional reasoning, CV-Relation and WhatsUp for spatial understanding, and CV-Count for numeracy. While HPSv3 – a much larger model trained on more preference pairs – generally performs better, it is far from saturating performance (66.9% on Winoground, 62.7% on CV-Count) and on SugarCrepe exhibits inverse scaling relative to earlier reward models. These failures are particularly striking given the simplicity of the underlying tasks.
| ImageRDB | Visual | Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |
| CLIP | 57.1 | 68.2 | 55.4 | 89.9 | 85.4 | 71.9 | 53.5 | 48.9 | 57.8 | 82.6 |
| ImageReward | 65.1 | 68.5 | 56.8 | 96.1 | 88.4 | 83.9 | 51.8 | 52.0 | 55.6 | 63.4 |
| PickScore | 61.6 | 63.8 | 54.6 | 81.6 | 80.6 | 64.9 | 51.7 | 50.6 | 52.8 | 73.8 |
| HPSv2 | 65.7 | 66.3 | 54.2 | 91.9 | 81.9 | 67.0 | 48.9 | 50.0 | 56.2 | 79.9 |
| HPSv3 | 66.8 | 76.5 | 66.9 | 79.9 | 78.2 | 72.1 | 79.1 | 85.9 | 62.7 | 87.2 |
| Qwen2.5-7B | 56.6 | 89.4 | 83.4 | 98.3 | 95.4 | 95.7 | 89.2 | 97.0 | 70.5 | 85.9 |
| HPSv3 vs. base | 10.2 | -12.9 | -16.6 | -18.5 | -17.1 | -23.5 | -10.2 | -11.0 | -7.8 | 1.2 |
Table 1. Accuracy of widely-used reward models on ImageRewardDB and our visual understanding evaluations. Random performance is 50%. Bottom: HPSv3 exhibits substantially degraded visual understanding relative to the base VLM it was fine-tuned from.
Since the bag-of-words effect is well documented for text-vision encoders [81], a natural question is whether instead using representations from a modern VLM trained on visual reasoning might resolve these deficiencies. Indeed, HPSv3 is fine-tuned on preference data from a strong base model (Qwen2.5-VL-7B [3]), raising the question of whether its visual understanding limitations stem from the base model or the fine-tuning process. To investigate, we evaluate the base VLM directly as a reward model by:
Table 1 shows that Qwen2.5-VL-7B achieves strong visual understanding—83.4% on Winoground, 97.0% on WhatsUp, and over 95% on all SugarCrepe subsets. After preference fine-tuning, HPSv3 gains approximately 10 percentage points in prediction accuracy on ImageRewardDB, but at severe cost: performance drops 16.6 points on Winoground, 17-24 points across SugarCrepe subsets, and 11 points on WhatsUp. The visual reasoning abilities of the base model are substantially eroded, leaving HPSv3 weaker than its foundation on the very competencies most relevant to text-image alignment.
Popular reward models cannot reason reliably about composition, spatial relationships, and numeracy, compromising their ability to judge text-conditioned image generation. Even starting from a VLM with strong visual understanding, these capabilities are catastrophically forgotten during preference fine-tuning.
Robust Fine-Tuning is Not Enough to Preserve Visual Understanding
The prior section raises the question: how can we retain a VLM’s visual reasoning during preference learning? The continual learning literature suggests possible remedies, but we find robust fine-tuning techniques do not adequately balance the existing visual reasoning with the new task of preference prediction. We fine-tune Qwen3-2B [2] on ImageRewardDB; see Appendix B for details.
Conservative Fine-Tuning
Aggressive fine-tuning induces catastrophic forgetting while more conservative updates can better preserve the original model’s knowledge [78, 32] (Vu et al. 2022). Thus, we investigate whether parameter-efficient fine-tuning can mitigate forgetting by regularizing towards the pretrained initialization. We study Low-Rank Adaptation (LoRA) [26], which potentially “forgets
less” [6]; and fine-tuning fewer layers, since earlier layers encode more general features suitable for transfer learning [50].
Findings. Figure 2 shows that conservative fine-tuning offers only marginal improvements in visual understanding (y axis) and at the cost of preference accuracy (x axis). Relative to full fine-tuning, Rank-8 LoRA retains slightly better visual reasoning with losses in preference prediction; Rank-2 LoRA and fine-tuning only the last four layers show slightly stronger versions of this tradeoff. However, none of these methods approach the visual understanding of the base Qwen3-2B. Earlier checkpoints actually demonstrate worse visual understanding, suggesting the forgetting is not simply overfitting—even the most conservative configurations exhibit substantial degradation.

Figure 2. Visual understanding () vs preference accuracy () for multiple checkpoints of the reward model during conservative fine-tuning on preference pairs. Conservative fine-tuning only weakly mitigates visual understanding degradation.
Conservative fine-tuning does not adequately mitigate the erosion of visual understanding capabilities during preference learning.
Weight Merging
Model soups, which merge the parameters of multiple fine-tuned models, can combine the strengths of different models [72]. The method’s theoretical foundation rests on the linear mode connectivity (LMC) hypothesis [20], which posits that two sets of weights and derived from a shared initialization lie in the same loss basin, such that their linear interpolations satisfy
for all . Weight merging with the pretrained initialization has been shown to mitigate catastrophic forgetting and improve OOD robustness across diverse continual learning settings: CLIP [72], LLMs [48], VLA policies [79], and multi-objective alignment [55]. We investigate merging our preference-fine-tuned reward model and the base VLM backbone as a means to preserve visual reasoning capabilities.
Findings. Figure 3 shows the reward model’s accuracy on ImageRewardDB and our visual understanding evaluations as a function of the interpolation coefficient . For preference pair accuracy, we observe the concave curve characteristic of LMC: intermediate mixing ratios outperform the linear interpolation. However, the best preference accuracy is still achieved by the unmerged fine-tuned model at . Moreover, weight merging fails to improve visual understanding. LMC is a claim about the loss landscape of the fine-tuning objective, and the concave curve on ImageRewardDB is the in-task behavior it anticipates. Our visual understanding suite is an off-task evaluation that LMC does not constrain, and although merging can improve OOD robustness in practice [72, 54], that transfer does not occur here: the merged model’s visual understanding falls below the linear interpolation between endpoints and is especially poor at small mixing ratios.
Weight merging does not improve over the fine-tuned model for either preference learning or visual reasoning retention, and in the latter case does not exhibit LMC.
Calibrating VLM Uncertainty
Rather than training a regression head to extract rewards from the VLM backbone, an alternative approach is to directly calibrate the model’s existing output distribution. (6) extracts the probability of the VLM generating “Yes” to a reward query; we fine-tune these probabilities to align with the preference dataset. This approach occupies an intermediate position between discriminative and generative reward modeling paradigms. Discriminative reward models predict fine-grained continuous scores typically learned with the Bradley-Terry objective. Generative reward models, in contrast, prompt LLMs (or VLMs) to produce verbalized judgments [82, 68, 46], which has been hypothesized to better tap into their base reasoning capabilities by leveraging the model’s native next-token prediction interface [82, 73]. Logit calibration could potentially inherit advantages from both paradigms: operating through next-token-prediction yet still producing continuous rewards suitable for fine-grained signals. We use the Bradley-Terry objective to directly adjust the VLM’s confidence estimates:
Findings. Uncertainty calibration better preserves visual reasoning but underfits preferences, making it a weak candidate for the preference learning objective. Figure 4 shows that at early checkpoints, visual understanding remains near the base model’s level, and decays only mildly as training progresses, remaining substantially above the regression head baseline throughout. This is consistent with the hypothesis that operating through the VLM’s existing generative interface better preserves pretrained representations. However, this preservation comes at a cost: the constrained optimization limits capacity to fit human preferences, underperforming the regression head standard by 2 percentage points.

Figure 4. Visual understanding () vs preference accuracy () for multiple checkpoints of the reward model trained with uncertainty calibration (orange) and with the standard reward readout head (blue). Uncertainty calibrating the base VLM better retains its visual reasoning, but underperforms on preference prediction.
Calibrating VLM uncertainty underfits the preference dataset, though it largely preserves visual reasoning.
CrossPairs: A Data-Centric Solution
Standard preference datasets pair a single prompt with two images of differing quality. The learning signal from such pairs conflates two distinct objectives: discerning aesthetic quality and assessing text-image alignment. When both images differ in style and artifacts alongside their semantic content, the model can exploit aesthetic shortcuts rather than developing robust visual parsing. Simply scaling preference datasets is not an effective remedy: the progression from 130k pairs in ImageRewardDB to 1.2M in HPSv3 still produces reward models with suboptimal visual understanding (§ 3), and can even lead reward models to overindex on stylistic properties rather than genuine image comprehension, as we observe in Figure 6. To target visual reasoning directly, we need preference pairs whose learning signal better isolates text-image alignment.

Figure 6. Generated samples after post-training with different reward models. The first three columns are prompts from Pick-a-Pic; the last three columns are prompts from GenEval.
We propose CrossPairs: preference pairs that share a common image but differ in prompt—one correct description and one incorrect. By holding the image fixed, aesthetic confounds are eliminated and the learning signal targets text-image alignment directly. Conveniently, this structure maps naturally onto existing VQA and multiple-choice data for vision-centric multimodal learning, allowing us to leverage existing training data without generating new images or collecting human annotations. These pairs are mixed with the standard shared-prompt pairs during preference learning. We used the CrossPairs structure to evaluate visual reasoning in § 3; here we leverage it for training.
We offer some intuition for how CrossPairs isolate visual reasoning. Consider a reward model that assigns a scalar score to an image-prompt pair. We can define an image-only component of this reward that captures prompt-independent aesthetic properties, by marginalizing over the distribution over prompts :
Then, we can define the residual
as a term that isolates the reward signal pertaining to prompt-image alignment. For the standard shared-prompt preference pair , a gradient step with Bradley-Terry loss yields the update
This decomposition reveals that even when prompts are carefully selected to target compositional, spatial, or numeracy skills, painstaking quality control is necessary to ensure that and differ on correctness and not image aesthetic quality. In the above loss, it is possible to learn shortcut correlates rather than develop robust visual reasoning.
In contrast, a CrossPair is a triple : a common image with a preferred caption and a dispreferred caption . The Bradley-Terry gradient update becomes:
The gradient thus depends only on how the reward model discriminates between prompts of the same image, which isolates visual understanding capability we wish to train. This optimization is invariant to any additive reparametrization of the residualized image-only component, for arbitrary .
Data. We convert diverse VQA datasets into CrossPairs format by pairing correct captions with targeted perturbations. We leverage the following datasets. CLEVR [28] comprises Blender-rendered scenes of geometric primitives with varying colors, sizes, and materials. This dataset targets compositional understanding and attribute binding. We construct negative captions by swapping attributes between objects in the scene. SpatialSense [80] contains real images from Flickr and VSR [39] contains natural photographs from MS COCO. Both of these are annotated with spatial relation statements, and we form the negative captions by inverting the relations that have a meaningful opposite (e.g. left/right, above/under). TallyQA [1] focuses on counting in realistic, complex scenes from Visual Genome [34]. We rephrase its VQA format into captions, and create negative captions by perturbing the true answer by . ColorSwap [10] contains AI-generated images targeting color-object binding, where pairs of prompts (and their corresponding images) differ only in color-object binding. Negative prompts are the alternative in the pair (e.g., “a black mug with red liquid” “a red mug with black liquid”). Table 2 shows the resultant training mixture comprising the shared-image CrossPairs along with ImageRewardDB [77].
| Dataset | Pairs | % Mix |
| CLEVR | 8750 | 5.6 |
| SpatialSense | 9712 | 6.2 |
| TallyQA | 10318 | 6.6 |
| ColorSwap | 2000 | 1.3 |
| VSR | 6664 | 4.2 |
| ImageRewardDB | 119765 | 76.2 |
| Total | 157209 | 100.0 |
Table 2. Final data mix for reward model training, comprising ImageRewardDB and multimodal learning VQA converted to CrossPairs preference pairs.
Results
We train Qwen3-2B and a small reward readout head on the final CrossPairs mix (Table 2) for three epochs. We first evaluate reward model accuracy directly, then assess the quality of generations guided by our reward model in both inference-time scaling and reinforcement learning settings. We also add VQAScore [38] and UnifiedReward [69] as baselines representing more recent, “reasoning-oriented” reward models.
Pareto improvement on preference prediction. Figure 1 shows that our CrossPairs data intervention achieves a Pareto improvement on the joint objective of preference accuracy and visual understanding. Our method achieves 66.3% accuracy on ImageRewardDB – comparable to the best reward baseline – while maintaining much stronger visual understanding than all reward models that have been calibrated on human preferences. (See Appendix § 10.§ 1 for a breakdown.) Our method is the only one to reach the top-right region of the plot, while continual learning approaches and existing reward models exhibit suboptimal tradeoffs. We conduct a controlled comparison of our reward model, ablating out the inclusion of CrossPairs (Appendix § 10.§ 4) and show that between reward models trained on the same preference dataset, adding CrossPairs supervision preserves aesthetic quality (Appendix § 10.§ 5). Next, we systematically assess our reward model by measuring the quality of text-to-image generations it guides.

Figure 1. (Left) Existing reward models exhibit poor visual understanding. VLM backbones possess strong visual reasoning, but this is eroded during preference fine-tuning, even with robust fine-tuning techniques. (Center) We introduce CrossPairs – preference pairs sharing a common image but differing in prompt – to isolate text-image alignment signal by eliminating aesthetic confounds. Mixed with standard pairs during training, our method achieves a Pareto improvement. (Right) The resulting reward model guides generations with superior compositional accuracy.
Inference-time scaling. Our reward model yields the strongest inference-time scaling on challenging compositional generation. We evaluate best-of- selection, a simple yet strong baseline for improving generation quality without weight updates [44], on GenEval [21], which explicitly tests compositional prompts requiring precise rendering of objects, attributes, spatial relations, and quantities. Figure 5a shows GenEval accuracy as a function of for two single-step generators, SD-Turbo [58] and FLUX.1-Schnell [8]. Our reward model consistently outperforms all baselines across the curve as scales. On SD-Turbo, our reward model achieves 64.5% accuracy at , compared to 62.4% for the next-best method. On FLUX.1-Schnell, we also see pronounced gains with our reward model reaching 78.1% accuracy with , compared to the next best at 75.7%. Our reward model outperforms all baselines on both a lightweight and more capable generator, indicating that it offers the most reliable signal for correctly rendering difficult compositional prompts. See Appendix § 10.§ 2 for extended results with other generators.

Figure 5. (a) Best-of- filtering. (b) RL with DiffusionNFT. Our reward model improves both inference-time scaling (left: single-step SD-Turbo and FLUX.1-Schnell) and post-RL performance (right: SD 3.5 Medium and SANA) on GenEval.
Reinforcement learning. Our reward model provides a more effective signal for RL post-training. We use DiffusionNFT [85], a recent state-of-the-art on-policy RL algorithm for diffusion, to train Stable Diffusion 3.5 Medium [61] and SANA [76] with prompts from Pick-a-Pic. Figure 5b shows GenEval accuracy across training iterations. Our reward model yields steady improvement throughout training, reaching 61.5% (SD) and accuracy at 50 iterations, and dominates other reward models across the training curve. Notably, GenEval is out-of-distribution relative to the Pick-a-Pic distribution used for the RL steps, yet consistent improvement under our reward model suggests it provides a more reliable signal for learning compositional generation, grounded in its strong visual understanding.
Qualitative analysis. Figure 6 illustrates that our reward model guides generations with both strong aesthetics and strong prompt adherence. ImageReward-guided generations follow prompts reasonably well but exhibit somewhat weaker aesthetics: images appear less sharp and with less polished overall composition. HPSv3 and our reward model trained with CrossPairs produce crisp and visually appealing images. However, HPSv3 tends toward photorealistic, conventionally attractive faces regardless of prompt. For example, the HPSv3-post-trained generative model ignores the prompt specifications “2d, vector illustration, logo... white background”, producing a photorealistic portrait on a black background (first column). The last three columns, drawn from GenEval’s compositional prompts, illustrate the qualitative outcomes of our reward model’s stronger visual understanding, corroborating Figure 5b. These prompts remain challenging even after post-training. For example, for “a photo of four stop signs”, our reward model is the only one to produce the correct count, despite some distortions. See Appendix § 10.§ 6 for further examples across all reward models.
Our results support the core claim: visual understanding is essential to a strong reward model, but simply scaling up preference pair annotations on the natural distribution of user prompts fails to target the visual reasoning required to assess complex compositions. Our approach yields a reward model that improves text-image alignment with strong aesthetics.
Related Work
Reward models for image generation. Reward models are central to aligning text-to-image models with human preferences. Early work leveraged dual text-vision encoders like CLIP [53] and BLIP [37] as a zero-shot reward signal; subsequent work fine-tuned them on human preference datasets [Xu et al., 2023, Kirstain et al., 2023, Wu et al., 2023b,a, Zhang et al., 2024b]. More recently, VLMs have been explored as reward model backbones: VQAScore [38] extracts rewards from response probabilities, HPSv3 [45] fine-tunes VLMs on preference data, LLM-as-judge approaches prompt for verbal assessments [12, 35], and generative reward models produce judgments in natural language [69]. Our work studies the tension between leveraging VLMs’ visual understanding and fitting to preferences, finding that standard fine-tuning erodes the capabilities that make VLMs attractive as backbones.
Fine-tuning VLMs. VLMs acquire broad capabilities through web-scale pretraining, but fine-tuning often causes catastrophic forgetting [71, 43], particularly in VLMs due to cross-modal feature drift and zero-shot erosion [42]. Parameter-efficient methods like LoRA [6] and weight merging [48, 60] may offer partial remedies. Similar degradation occurs when fine-tuning VLMs for robotic control, where forgetting overwrites visual-text alignments [16, 86, 23]. Our work shows preference fine-tuning erodes visual reasoning in reward models.
Robust reward learning. Human preference annotations exhibit noise from annotator disagreement and labeling errors [4, 51], with inter-annotator agreement as low as 73% [59]. Techniques to address this include label smoothing [87], adaptive margins and label flipping [67], and debiased DPO [13].
These methods target noise in labels; our work addresses a complementary problem — inadequate reward signal for visual grounding in text-to-image reward models.
Conclusion
Visual understanding is essential to reward modeling, yet preference fine-tuning erodes these capabilities, even when starting from strong VLM backbones. By injecting visual reasoning supervision directly into preference learning using CrossPairs, we achieve a Pareto improvement over existing reward models. Our reward model guides both inference-time selection and RL post-training toward aesthetic generations with stronger prompt adherence.
Acknowledgments and Disclosure of Funding
S.S. was supported by the Stanford School of Engineering fellowship and by research funding to the Yeung-Levy lab. Compute was provided by the Stanford AI Lab (SAIL) cluster. This work did not receive additional third-party funding and the authors declare no competing interests.
Extended Preliminaries
Reward Models in Text-to-Image Generation
Reward models play a central role in aligning text-to-image generative models with human preferences, appearing in both inference-time and post-training pipelines.
Inference-time scaling. Reward models can be leveraged at inference time to improve generation quality without modifying model weights. The simplest approach is best-of- sampling on the final generation: given a prompt , we generate candidate images and return the sample with highest reward:
Beyond simple reranking, search-based methods explore the space of denoising trajectories using techniques such as beam search, particle filtering, and rejection sampling to identify higher-quality paths [44, 41, 49, 31]. Alternatively, the initial noise or intermediate latents can be directly optimized via gradient-based methods to maximize reward signals [18, 65]. Reward gradients can also be incorporated into the score function at each denoising step, steering sampling toward higher-reward regions [17, 24, 5].
Post-training with reinforcement learning. Given a pretrained diffusion or flow-based generative model, RL can optimize the model to produce images that receive high reward. Let denote the distribution over images induced by a generative model with parameters conditioned on prompt , and let denote a reward model. In broad strokes, the RL objective maximizes expected reward while regularizing against a reference model :
where controls the strength of the regularization. Many methods have been proposed to optimize this objective for diffusion models [7, 19, 14, 40, 85]; while their algorithms differ, all rely on a reward signal to guide optimization.
Reward Modeling with Bradley-Terry
Preference pairs. Text-to-image reward models are typically trained on datasets of pairwise human preferences. Given a prompt and two generated images (preferred) and (dispreferred), the training data consists of tuples indicating that human annotators preferred over as a response to .
Bradley-Terry model. We assume the Bradley-Terry model [9] for human preferences. Let denote the scalar reward assigned to image given prompt , the probability that is preferred over is modeled as:
where denotes the sigmoid function. This induces a maximum likelihood training objective over a dataset of preference pairs:
This formulation posits that preferences depend only on the individual qualities of the two images and not on interactions between them—an independence assumption that enables pointwise reward modeling. In this work, we focus on pointwise reward models that produce an absolute quality assessment for a prompt-image pair, independent of other candidate images. They contrast with pairwise reward models, which require a reference image for comparison and cannot score candidates in isolation. They also contrast generative reward models, which produce rewards in natural language and are therefore typically limited to binary judgments or discrete rating scales rather than fine-grained continuous scores [82, 84]. Pointwise models offer practical advantages for deployment: they provide absolute quality assessments suitable for thresholding, filtering, and reference-free RL.
Experiment Details
Experiment details for Section 4
To isolate the effects of training methodology from confounding factors, we hold fixed the base model (Qwen3-2B) and train for 3 epochs on ImageRewardDB throughout Section 4. We use Qwen3-2B as our base VLM, a capable model that balances strong visual understanding with computational tractability. For training data, we use ImageRewardDB [77], from which we extract only the pairwise rank comparisons (discarding absolute ratings) to match the Bradley-Terry formulation. Unless otherwise stated, our baseline configuration performs full fine-tuning on the last 9 layers of the language model (comprising approximately one-third of the LLM parameters) of the VLM, with the visual encoder and adapter fully frozen. We find this configuration sufficient to match state-of-the-art performance on the ImageRewardDB held-out test set while providing a strong baseline for investigating forgetting mitigation strategies.
Experiment details for Section 4.1
We experiment with conservative fine-tuning. In addition to the baseline of full fine-tuning on the last 9 layers, we fine-tune those same layers with 1) rank 8 LoRA (with ) and 2) rank 2 LoRA (with ), and 3) also perform full-finetuning on the last 4 layers. In all cases, given an image and prompt , the VLM produces a joint representation on the last token, to which we attach a regression head outputting a scalar reward: . This lightweight linear layer is randomly initialized and always fully-finetuned. For each configuration, we train with 3 randomly seeded data orders and save 8 checkpoints throughout training, yielding 24 model snapshots.
Experiment details for Section 4.2
We experiment with weight merging. We first train only the readout head for 1 epoch on ImageRewardDB before unfreezing the last 9 LLM layers of Qwen3-2B for 3 epochs of end-to-end full fine-tuning. This two-stage strategy helps to keep the fine-tuned model in the same loss basin as the initialization [36, 56]. For model merging, we interpolate the VLM backbone with coefficient by . The readout head parameters are taken from the fine-tuned reward model (the base VLM has no corresponding component).
Experiment details for Section 4.3
We experiment with uncertainty calibration of the VLM’s confidence estimates. Without preference fine-tuning, out-of-the-box VLMs produce a probability reflecting its assessment about image-text alignment, but this probability is poorly calibrated with respect to human preferences (Table 1). We use the Bradley-Terry objective to directly adjusts the VLM’s confidence estimates:
We fine-tune the last 9 layers of the Qwen3-2B VLM, consistent with Section 4.1, and do not make use of a readout regression head. The experiment is repeated for 3 random seeds and we save 8 checkpoints during the course of training over 3 epochs.
Experiment details for Section 5.1
Our final reward model is obtained by fine-tuning on the last 9 layers of Qwen3-2B VLM and a linear projection head. We train for three epochs with Bradley-Terry loss on the combined dataset described in Table 2. This setup is deliberately straightforward — we use default hyperparameters matching those from Section 4, demonstrating that the performance gains arise from the data intervention rather than careful tuning.
For reinforcement learning with DiffusionNFT, we train on prompts from the PickScore dataset using Stable Diffusion 3.5 Medium as the base model, with 10 denoising steps and no classifier-free guidance. These settings produce relatively poor samples from the base model, providing a challenging testbed for whether RL can unlock generation quality. We set for DiffusionNFT and train on 8 sampled images per prompt.
Further Results
Reward Model Accuracies
Table 3: Accuracy of reward models on ImageRewardDB and visual understanding evaluations. Random performance is 50%. †Trained only as a pairwise model; evaluated here by prompting for a pointwise rating (see text).
Table 3 reports the full accuracy results for all reward models on ImageRewardDB and our visual understanding benchmark suite, extending Table 1 in the main body to include VQAScore, UnifiedReward, and our ablated baseline ‘Ours w/o CrossPairs’ (described in Section C.4).
| ImageRDB | Visual | Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |
| CLIP | 57.1 | 68.2 | 55.4 | 89.9 | 85.4 | 71.9 | 53.5 | 48.9 | 57.8 | 82.6 |
| ImageReward | 65.1 | 68.5 | 56.8 | 96.1 | 88.4 | 83.9 | 51.8 | 52.0 | 55.6 | 63.4 |
| PickScore | 61.6 | 63.8 | 54.6 | 81.6 | 80.6 | 64.9 | 51.7 | 50.6 | 52.8 | 73.8 |
| HPSv2 | 65.7 | 66.3 | 54.2 | 91.9 | 81.9 | 67.0 | 48.9 | 50.0 | 56.2 | 79.9 |
| HPSv3 | 66.8 | 76.5 | 66.9 | 79.9 | 78.2 | 72.1 | 79.1 | 85.9 | 62.7 | 87.2 |
| VQAScore | 56.8 | 79.2 | 72.9 | 92.3 | 88.8 | 86.1 | 79.2 | 84.6 | 63.4 | 66.0 |
| UnifiedReward | 60.0 | 77.2 | 75.9 | 92.1 | 88.9 | 85.0 | 69.2 | 74.9 | 59.5 | 71.8 |
| UnifiedReward-Think† | 53.6 | 69.6 | 66.1 | 81.8 | 77.0 | 73.0 | 63.2 | 68.8 | 63.2 | 64.0 |
| Qwen3-2B | 55.3 | 85.4 | 75.9 | 98.2 | 94.2 | 93.4 | 87.3 | 91.2 | 60.4 | 82.8 |
| Ours w/o CrossPairs | 66.3 | 78.4 | 66.8 | 93.7 | 88.1 | 84.4 | 78.9 | 73.2 | 59.0 | 82.9 |
| Ours | 66.3 | 85.3 | 71.6 | 97.5 | 92.8 | 91.9 | 74.5 | 92.2 | 71.7 | 90.2 |
Table 3. Accuracy of reward models on ImageRewardDB and visual understanding evaluations. Random performance is 50%. †Trained only as a pairwise model; evaluated here by prompting for a pointwise rating (see text).
VQAScore extracts a reward by querying a VLM with “Does this figure show {prompt}?” and reading off the probability of the “Yes” token, leveraging the VLM’s pretrained visual reasoning directly without preference fine-tuning. UnifiedReward is a generative reward model that prompts a VLM to verbalize a discrete quality rating in natural language; it is trained on both pointwise ratings and pairwise preferences (both images presented to the VLM simultaneously).
We additionally report UnifiedReward-Think Wang et al. [ [70], which extends UnifiedReward with long-form chain-of-thought and RL-based reasoning enhancement and is therefore the closest reasoning-oriented reward model to our motivation. It differs from every other model we study in an important respect: it is trained only as a pairwise preference model, emitting a relative judgment over two images, and is not trained to rate a single image. Because the need for a reference image is unwieldy for best-of- selection and RL, we do not use it as a downstream reward signal. Since it is a generative model, however, we can prompt it for a pointwise rating; it returns a parseable score in most cases, despite occasional format errors. The resulting score is poorly calibrated: it trails even CLIP on ImageRewardDB and is only marginally above CLIP on average visual understanding, well below UnifiedReward and our model on both. Inference is also substantially slower owing to the long prompt and chain-of-thought generation.
Several observations stand out from the results of Table 3:
Existing reward models trade off visual reasoning for preference accuracy. No baseline reward model trained on human preferences achieves both strong preference prediction and strong visual reasoning. Among baselines, HPSv3 achieves the strongest balance, though still short of our method on visual understanding. Notably, HPSv3 uses substantially more preference data (1.3M pairs) and a larger backbone (7B) than our setup; we do not claim a controlled efficiency comparison, but rather note that scaling preference pairs alone has not closed the visual reasoning gap.
Reasoning-oriented reward models are stronger visual reasoners but weaker on preferences. VQAScore and UnifiedReward achieve relatively strong visual understanding (79.2% and 77.2% average) but underperform on preference prediction (56.8% and 60.0% on ImageRewardDB). For VQAScore, this likely reflects that it leverages a VLM directly and has no exposure to human preference data, so its scores reflect text-image alignment rather than aesthetic preference. For UnifiedReward, the limitation may stem from its generative interface: when used pointwise, it can only emit integer ratings, providing coarser-grained scores than the continuous outputs of regression-based reward models. While both achieve fairly strong visual reasoning, they still underperform our method (85.3% average).
Ours w/o CrossPairs reproduces the forgetting effect. Training Qwen3-2B on ImageRewardDB alone yields strong preference accuracy (66.3%) but degrades visual understanding from 85.4% (base Qwen3-2B) to 78.4%—a 7-point drop, with particularly steep losses on Winoground (−9.1) and WhatsUp (−18.0). This is the same forgetting effect we documented in § 4.
CrossPairs achieves a Pareto improvement. Our full method matches the preference accuracy of ‘Ours w/o CrossPairs’ (66.3%) while recovering nearly all of the visual reasoning of the base Qwen3-2B. The breakdown reveals that the CrossPairs actually sharpens counting abilities beyond the base model, while largely preserving the other visual understanding categories.
Pointwise evaluation. All reward models are evaluated in a pointwise fashion: the reward score on a single prompt–image pair is used for ranking, rather than passing both images simultaneously to models that support pairwise scoring. This ensures a fair comparison across architectures and reflects the practical deployment setting, where pointwise scores are easier to leverage for filtering, RL, and inference-time scaling than relative pairwise judgments. UnifiedReward-Think is the one model in Table 3 not trained for pointwise use; we obtain pointwise ratings from it by prompting, as described above, and include it in the accuracy table only.
Further Generators
To assess whether our reward model’s advantages generalize across generator architectures, we extend the inference-time best-of- evaluation from Figure 5a (SD-Turbo and FLUX.1-Schnell) to two additional generators: SANA [76] (specifically, the Sana_1600M_512px_diffusers checkpoint) and Bagel [15] (specifically, the BAGEL-7B-MoT checkpoint). Together, these four generators span a range of generative modeling paradigms—diffusion (SD-Turbo), rectified flow (FLUX.1-Schnell, SANA), and unified multimodal next-token prediction (Bagel)—as well as a range of architectural backbones: a U-Net (SD-Turbo), an MM-DiT (FLUX.1-Schnell), a linear-attention DiT with deep-compression autoencoder (SANA), and a Mixture-of-Transformer-Experts (Bagel). All evaluations use the same GenEval prompt set as in the main body, with SANA and Bagel called at 8 inference steps to test whether reward model gains hold under the more capable multi-step regime.
Figure 7 shows GenEval accuracy as a function of for SANA and Bagel. Several observations:

Figure 7. Further generators.
Our reward model dominates on SANA across the curve. On SANA, our reward model reaches >80% GenEval accuracy at , compared to for the next-best baseline (UnifiedReward). The gap is consistent throughout the -scaling curve, with our model leading from onward.
On Bagel, gains are smaller but consistent. Bagel starts from a stronger base, leaving less headroom for reward-guided selection. Our reward model still ranks among the top performers across , reaching at , competitive with HPSv3 and ahead of the remaining baselines.
The relative ordering of baselines varies by generator. While our reward model is consistently the strongest across all four generators (SD-Turbo, FLUX.1-Schnell, SANA, Bagel), the relative ordering among the baselines is inconsistent: a method that ranks among the top on one generator may fall to the middle of the pack on another.
Further evaluations on T2ICompBench

Figure 8. Further evaluations on T2ICompBench.
To test whether the gains on GenEval transfer to a different compositional benchmark, we evaluate the final RL-trained checkpoints on T2I-CompBench [27]. We sample 300 prompts uniformly at random from each of the color, spatial, and numeracy splits, and evaluate the SD 3.5 Medium model RL-trained with each reward model under DiffusionNFT (the same checkpoints evaluated in § 5.1). We use the recommended default evaluators from T2I-CompBench: BLIP-VQA for the color split, and UniDet for the spatial and numeracy splits.
Figure 8 reports T2I-CompBench scores across the three splits and their mean. Our reward model achieves the highest mean T2I-CompBench score across the three categories, consistent with our GenEval results in Figure 5. The advantage is most pronounced on the spatial and numeracy splits—the categories most directly targeted by our CrossPairs training data (SpatialSense and VSR for spatial; TallyQA for numeracy). On the color split, our reward model performs comparably to the strongest baselines, suggesting that color-attribute binding (targeted by ColorSwap in our mix) is also strengthened, though margins are tighter on this category.
‘Ours w/o CrossPairs’ underperforms across all splits, again confirming that the data intervention—not the choice of base model or training recipe—drives the improvements.
Ablating out CrossPairs
To isolate the contribution of the CrossPairs data intervention from other choices in our pipeline (base model, training hyperparameters, regression head design), we train an ablated model ‘Ours w/o CrossPairs’ on ImageRewardDB preference pairs only, holding all other settings identical to the full method.
Reward model accuracy (Table 3). The ablated model achieves 66.3% on ImageRewardDB—matching our full method—but drops to 78.4% average visual understanding (vs. 85.3% for the full method). This reproduces the forgetting pattern documented in § 4: training on conventional preference pairs alone, even from a strong VLM backbone, erodes visual reasoning. The drop is largest on Winoground (−4.8), CV-Relation (−19.0 from base), and WhatSup (−18.0 from base).
Alignment preferences on generated images. The visual understanding evaluations in Table 3 have the same shared-image, constructed-negative structure of CrossPairs training data, albeit on different (out-of-distribution) datasets. MJ-Bench [11] provides a complementary test: its alignment split consists of pairs of generated images under a shared prompt, with human labels for which image better satisfies the prompt. The images carry the artifacts of real generators and the preferred image differs from the dispreferred one in varied and often subtle ways, so neither the image distribution nor the failure modes resemble our rule-based perturbations of real photographs. Table 4 shows that CrossPairs improve accuracy on every category, by 5.6 points overall and by 8.9 and 7.9 points on Counting and Spatial, the skills most directly targeted by our training sources. The gain on Action, a category none of our perturbations target, suggests the improvement is not limited to the perturbation types seen in training.
| Overall | Object | Attribute | Counting | Spatial | Action | |
| Ours w/o CrossPairs | 62.4 | 61.0 | 62.0 | 66.1 | 67.1 | 61.5 |
| Ours | 68.0 | 65.7 | 66.4 | 75.0 | 75.0 | 68.4 |
Table 4. Accuracy (%) on the alignment split of MJ-Bench.
Downstream generation. When used as a reward signal for inference-time best-of- on SD-Turbo, FLUX.1-Schnell, SANA, and Bagel (Figure 9), ‘Ours w/o CrossPairs’ performs marginally worse than the full method, though the gap is not always consistent across generators and values of . The picture changes dramatically when the reward model is used as the objective for RL post-training (Figure 10). On both SD 3.5 Medium and SANA, RL with ‘Ours w/o CrossPairs’ fails to improve compositional accuracy and on SD 3.5 Medium actually decreases GenEval performance over training—from at iteration 10 to at iteration 50. Meanwhile, RL with our full reward model yields steady improvement, reaching 61.5% on SD 3.5 Medium and on SANA.

Figure 9. Ablating out CrossPairs, evaluated best-of-.

Figure 10. Ablating out CrossPairs, evaluated by RL post-training.
Preservation of Aesthetic Quality
A natural concern with CrossPairs is whether shifting the training signal toward visual reasoning compromises the reward model’s ability to capture aesthetic preferences. We evaluate this in two ways: held-out preference accuracy and aesthetic quality of RL-trained generations. In both cases, we restrict comparison to reward models trained on the same human preference data (the ImageReward reward model and our ablated baseline ‘Ours w/o CrossPairs’) to make a fair comparison.
Held-out preference accuracy (Table 5). We compare ImageReward, ‘Ours w/o CrossPairs’, and our full method on ImageRewardDB (in-distribution) and the 500-pair ‘unique test’ split of Pick-a-Pic [33] (out-of-distribution—these models have not been trained on Pick-a-Pic data). On both splits, the CrossPairs intervention matches or exceeds the preference accuracy of the ablated baseline: 66.3% vs. 66.3% on ImageRewardDB (tied) and 63.6% vs. 63.2% on Pick-a-Pic. Both variants outperform ImageReward (65.1% / 60.9%). The fact that adding CrossPairs supervision does not reduce held-out preference accuracy—and slightly improves OOD generalization on Pick-a-Pic—indicates that visual reasoning supervision is complementary to, rather than competitive with, aesthetic preference learning.
| ImageRewardDB (ID) | Pick-a-Pic (OOD) | |
| ImageReward | 65.1 | 60.9 |
| Ours w/o CrossPairs | 66.3 | 63.2 |
| Ours | 66.3 | 63.6 |
Table 5. Held out preference accuracy.
Aesthetic score on RL-trained generations (Table 6). As an alternative test, we apply the LAION aesthetic predictor to images generated by SD 3.5 Medium and SANA models RL-trained against each of the three reward models, evaluated on 512 held-out prompts from the Pick-a-Pic test set. The differences are marginal, all within of each other. For context, LAION uses this same predictor to bucket the LAION-5B English subset into aesthetic tiers, with the 6.5+ tier (625K image-text pairs) being a strict subset of the 4.5+ tier (1.2B pairs) commonly used for fine-tuning generative models. All three reward models thus produce generations in the strong-aesthetics regime, with differences smaller than the gap between adjacent LAION tiers.
| SD 3.5 Medium | SANA | |
| ImageReward | 6.46 | 6.48 |
| Ours w/o CrossPairs | 6.42 | 6.50 |
| Ours | 6.42 | 6.41 |
Table 6. Aesthetic scores on generations from reinforcement post-trained generators.
Qualitative confirmation. The qualitative examples in Section C.6 further support that aesthetic quality is preserved: generations from our reward model are visually crisp and compositionally polished, while the prompts from PickScore (first three columns) show no obvious aesthetic regression compared to ImageReward and HPSv3 baselines.
Together, these results indicate that CrossPairs improves visual reasoning without sacrificing the aesthetic preference signal.
Further Qualitative Examples

Figure 11. Example generations. The first three columns are prompts from PickScore; the last three columns are prompts from GenEval.

Figure 12. Example generations. The first three columns are prompts from PickScore; the last three columns are prompts from GenEval.
CrossPairs vs. a Next-Token-Prediction Loss
CrossPairs inject visual reasoning supervision into preference learning. A natural question is whether the pairing structure matters, or whether any form of visual reasoning supervision drawn from the same sources would do. Since our backbone is a VLM, the most obvious alternative is to keep the Bradley–Terry objective on ImageRewardDB and add the visual reasoning data as an auxiliary next-token-prediction (NTP) loss on the language modeling head, i.e. ordinary VQA fine-tuning.
Setup. We hold the backbone (Qwen3-2B), trainable layers, readout head, hyperparameters, and the data mixture of Table 2 fixed, and vary only how the visual reasoning sources are supervised. For the VQA auxiliary loss variant, each source is used in its original, unperturbed VQA form with the usual language modeling loss: TallyQA count questions completed by the numerical answer; VSR and SpatialSense statements completed by True/False; ColorSwap as an A/B caption choice; and CLEVR as the correct caption describing the scene. ImageRewardDB pairs are supervised through the regression head with the Bradley–Terry loss as in our main results. For Ours, the same sources are converted into CrossPairs (§ 5) and supervised through the same Bradley–Terry loss and regression head as the preference pairs.
Results. CrossPairs are consistently better on the joint objective (Table 7): higher ImageRewardDB accuracy (66.3% vs. 65.2%) and higher average visual understanding (85.3% vs. 82.2%), with the largest gaps on WhatsUp (+11.4) and Winoground (+5.0); CV-Relation is the one split on which the auxiliary loss is marginally ahead. Downstream (Table 8), the two are indistinguishable up to on both generators, after which CrossPairs pull ahead and the gap widens with .
| ImageRDB | Visual | Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |
| VQA auxiliary loss | 65.2 | 82.2 | 66.6 | 97.6 | 90.7 | 89.9 | 77.1 | 80.8 | 67.7 | 87.2 |
| Ours | 66.3 | 85.3 | 71.6 | 97.5 | 92.8 | 91.9 | 74.5 | 92.2 | 71.7 | 90.2 |
Table 7. CrossPairs vs. supervising the same visual reasoning sources with an auxiliary VQA next-token-prediction loss. Accuracy on ImageRewardDB and the visual understanding evaluations; random performance is 50%.
| Generator | Method | ||||||
| SD-Turbo | VQA auxiliary loss | 55.4 | 58.5 | 60.8 | 62.4 | 63.0 | 63.4 |
| SD-Turbo | Ours | 55.4 | 58.5 | 61.0 | 63.0 | 63.8 | 64.5 |
| FLUX.1-Schnell | VQA auxiliary loss | 71.9 | 73.8 | 75.0 | 75.9 | 76.4 | 76.9 |
| FLUX.1-Schnell | Ours | 71.8 | 73.7 | 75.2 | 76.6 | 77.3 | 78.1 |
Table 8. Best-of- GenEval accuracy (%) for the two supervision formats. We generate 50 images per prompt; each entry is averaged over 200 random -subsets of those 50 generations.
We note finally that CrossPairs are, in our view, the most natural way to leverage VQA-style supervision for reward modeling. A CrossPair is already a valid Bradley–Terry pair (the two candidates differ along the text axis rather than the image axis), so it is consumed by the same loss and readout head as the human preference data, with no auxiliary objective, loss weighting, or language modeling head to maintain. The same construction therefore applies to backbones that cannot generate text at all, such as CLIP-style dual encoders, for which the NTP alternative is not available.
Error analysis
The reward model accuracies in Table 3 and Table 7 are deterministic: each is a single pass of a deterministic reward model over a fixed set of inputs, so there is no run-to-run variance. In contrast, the best-of- results admit an error analysis based on random redraws of the candidates at each point of the curve. Generating fresh candidates for every and every reward model is beyond our compute budget, so we use the following resampling protocol. For each prompt we generate 50 images once; for each we draw 200 random -subsets of those 50, apply best-of- selection within each subset, and report the mean and standard deviation across draws. The draws are not independent samples from the generator, so this underestimates the true variance, but it captures the selection noise. At the subset is the full set and the standard deviation is zero by construction.2
Our reward model leads on both generators at every . At small the margin over the strongest baseline (UnifiedReward on SD-Turbo, HPSv3 on FLUX.1-Schnell) is within sampling noise; it exceeds two standard deviations from onward on SD-Turbo and from onward on FLUX.1-Schnell, and widens further as grows.
| Method | ||||||
| CLIP | ||||||
| ImageReward | ||||||
| PickScore | ||||||
| HPSv2 | ||||||
| HPSv3 | ||||||
| VQAScore | ||||||
| UnifiedReward | ||||||
| Ours |
Table 9. Best-of- GenEval accuracy (%, mean std over 200 resampled -subsets) with SD-Turbo as the generator.
Generalization to Another VLM Backbone
All experiments in the main body use Qwen3-2B as the reward model backbone. To demonstrate the generality of the findings, we repeat the core comparison with SmolVLM2-2.2B [47]. We chose it as a widely used open-source VLM of similar size that differs from Qwen3-2B in most design choices: language backbone, multimodal training recipe, tiled fixed-resolution vs dynamic-resolution visual inputs, with / without DeepStack injection of visual features, etc. We use the same data mix, training protocol, and hyperparameters as for Qwen3-2B, and train the same two variants as in Appendix C.4: preference pairs only (w/o CrossPairs) and the full mix (Ours). We did not run RL post-training with these reward models due to compute constraints.
Results. The pattern from Qwen3-2B reproduces on SmolVLM2 (Table 11). Preference fine-tuning alone erodes visual understanding from 76.1% to 69.4%, with the steepest losses on WhatsUp (-16.6), CV-Relation (-13.5), and SugarCrepe-Swap (-11.9), the same categories that degrade most for Qwen3-2B. Adding CrossPairs recovers much of this loss (73.6%) with no cost to preference accuracy (58.8% vs. 58.4%), again with the largest gains on the spatial and counting splits. In best-of- selection (Table 12), the CrossPairs-trained reward model is ahead at every on both generators, with the gap widening as grows, mirroring the behavior of our Qwen3-2B model. We take this as evidence that the forgetting effect and the data-centric remedy are properties of preference fine-tuning rather than of a particular backbone.
| ImageRDB | Visual | Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |
| SmolVLM2-2.2B (base) | 55.7 | 76.1 | 69.7 | 93.1 | 85.5 | 83.3 | 69.0 | 76.4 | 62.6 | 69.7 |
| w/o CrossPairs | 58.4 | 69.4 | 62.8 | 89.4 | 86.8 | 71.4 | 55.5 | 59.8 | 59.3 | 70.5 |
| Ours | 58.8 | 73.6 | 65.1 | 91.3 | 90.1 | 76.5 | 62.9 | 66.7 | 65.5 | 70.3 |
Table 11. SmolVLM2-2.2B as the reward model backbone. Accuracy on ImageRewardDB and the visual understanding evaluations; random performance is 50%.
| Generator | Method | ||||||
| SD-Turbo | w/o CrossPairs | 53.9 ± 1.3 | 55.7 ± 1.2 | 57.1 ± 1.1 | 58.2 ± 0.9 | 58.3 ± 0.7 | 57.9 ± 0.0 |
| Ours | 54.2 ± 1.3 | 56.1 ± 1.0 | 57.5 ± 1.1 | 58.8 ± 1.0 | 59.2 ± 0.7 | 59.3 ± 0.0 | |
| FLUX.1-Schnell | w/o CrossPairs | 69.7 ± 1.1 | 70.8 ± 1.1 | 71.4 ± 1.1 | 71.6 ± 0.9 | 71.8 ± 0.7 | 72.1 ± 0.0 |
| Ours | 70.1 ± 1.1 | 71.5 ± 1.1 | 72.5 ± 1.0 | 73.4 ± 0.9 | 73.6 ± 0.7 | 73.7 ± 0.0 |
Table 12. Best-of- GenEval accuracy (%, mean ± std over 200 resampled -subsets; see Appendix § 12) with SmolVLM2-2.2B-based reward models.
Limitations
Two limitations suggest natural avenues for future work. First, we do not characterize how the CrossPairs mix should scale with preference dataset size or base model capacity. Our experiments fix both and the data mix in Table 2 reflect natural source sizes rather than a recipe of proportions. Whether the optimal CrossPairs-to-preference ratio remains stable when scaling up remains an open question. Second, we evaluate only the discriminative regression-head paradigm. Whether CrossPairs supervision transfers to generative reward models (which produce verbalized ratings or preference statements as tokens) or pairwise reward models (which condition on both candidates jointly) remains untested. We hypothesize the intervention is largely architecture-agnostic, since the underlying signal — image-anchored prompt discrimination — does not depend on the readout mechanism, but verifying this would broaden the applicability of our findings.
| Method | ||||||
| CLIP | 70.8 ± 1.1 | 72.2 ± 1.2 | 73.4 ± 1.1 | 74.4 ± 0.9 | 74.9 ± 0.6 | 75.7 ± 0.0 |
| ImageReward | 71.1 ± 1.1 | 72.7 ± 1.1 | 73.8 ± 1.0 | 74.7 ± 0.9 | 74.9 ± 0.7 | 74.9 ± 0.0 |
| PickScore | 70.1 ± 1.1 | 71.0 ± 1.1 | 71.5 ± 1.0 | 71.7 ± 0.8 | 71.7 ± 0.6 | 71.6 ± 0.0 |
| HPSv2 | 70.0 ± 1.1 | 71.1 ± 1.1 | 71.7 ± 1.0 | 71.9 ± 0.8 | 71.9 ± 0.6 | 71.8 ± 0.0 |
| HPSv3 | 71.2 ± 1.1 | 72.9 ± 1.1 | 74.2 ± 1.0 | 75.1 ± 0.8 | 75.4 ± 0.6 | 75.5 ± 0.0 |
| VQAScore | 70.2 ± 1.2 | 71.7 ± 1.2 | 73.1 ± 1.1 | 74.3 ± 1.0 | 74.8 ± 0.8 | 75.3 ± 0.6 |
| UnifiedReward | 70.9 ± 1.2 | 72.4 ± 1.2 | 73.4 ± 1.2 | 74.1 ± 1.1 | 74.3 ± 1.1 | 74.5 ± 1.0 |
| Ours | 71.8 ± 1.1 | 73.7 ± 1.1 | 75.2 ± 1.0 | 76.6 ± 0.9 | 77.3 ± 0.7 | 78.1 ± 0.0 |
Table 10. Best-of- GenEval accuracy (%, mean ± std over 200 resampled -subsets) with FLUX.1-Schnell as the generator.
Societal Impacts
Our work improves the prompt adherence of text-to-image generative models, which carries dual-use implications common to advances in image generation. On the positive side, more reliable text-image alignment reduces the trial-and-error burden on users, broadens accessibility for non-expert users who cannot iteratively refine prompts, and supports legitimate applications in education, design, and accessibility. On the negative side, stronger prompt adherence could lower the barrier to producing targeted synthetic media, including misinformation, non-consensual imagery, and content designed to evade safety filters that rely on the imprecision of current generators.
References
- [1]Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084, 2019.arxiv.org/abs/1810.12440
- [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wen Ding, Chang Gao, Chunjang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a.arxiv.org/abs/2511.21631
- [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.arxiv.org/abs/2502.13923
- [4]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
- [5]Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 843–852, 2023.arxiv.org/abs/2302.07121
- [6]Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024.
- [7]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023.arxiv.org/abs/2305.13301
- [8]Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024.
- [9]Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
- [10]Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. arXiv preprint arXiv:2402.04492, 2024.
- [11]Zhaorun Chen, Zichen Wen, Yichao Du, Yiyang Zhou, Chenhang Cui, Siwei Han, Jen Weng, Chaoqi Wang, Zhengwei Tong, Leria Huang, et al. Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation? Advances in Neural Information Processing Systems, 38, 2025.
- [12]Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.18235, 2023.
- [13]Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024.
- [14]Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400, 2023.
- [15]Chaorui Deng, Devao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025.arxiv.org/abs/2505.14683
- [16]Sombit Dey, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, and Danda Pani Paudel. Revla: Reverting visual domain limitation of robotic foundation models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8679–8686. IEEE, 2025.
- [17]Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
- [18]Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems, 37:125487–125519, 2024.arxiv.org/abs/2406.04312
- [19]Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858–79885, 2023.arxiv.org/abs/2305.16381
- [20]Jonathan Frankle, Gintare Karolina Dziugaitė, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269. PMLR, 2020.arxiv.org/abs/1912.05671
- [21]Dhruba Ghosh, Hannah Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023.arxiv.org/abs/2310.11513
- [22]Google DeepMind. Imagen. https://deepmind.google/models/imagen/, 2025. Model release.
- [23]Asher J Hancock, Xindi Wu, Lihan Zha, Olga Russakovsky, and Anirudha Majumdar. Actions as language: Fine-tuning vlms into vlas without catastrophic forgetting. arXiv preprint arXiv:2509.22195, 2025.
- [24]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
- [25]Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36:31096–31116, 2023.arxiv.org/abs/2306.14610
- [26]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.arxiv.org/abs/2106.09685
- [27]Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-combench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
- [28]Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.arxiv.org/abs/1612.06890
- [29]Ivana Kajić, Olivia Wiles, Isabela Albuquerque, Matthias Bauer, Su Wang, Jordi Pont-Tuset, and Aida Nematzadeh. Evaluating numerical reasoning in text-to-image models. Advances in Neural Information Processing Systems, 37:42211–42224, 2024.arxiv.org/abs/2406.14774
- [30]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s "up" with vision-language models? investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785, 2023.
- [31]Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don’t succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. arXiv preprint arXiv:2305.13308, 2023.arxiv.org/abs/2305.13308
- [32]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521–3526, 2017.DOI
- [33]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023.arxiv.org/abs/2305.01569
- [34]Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2017.arxiv.org/abs/1602.07332
- [35]Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12268–12290, 2024.
- [36]Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022.
- [37]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022.arxiv.org/abs/2201.12086
- [38]Zhiqi Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024.arxiv.org/abs/2404.01291
- [39]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023.
- [40]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025a.arxiv.org/abs/2505.05470
- [41]Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Correcting diffusion generation through resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8713–8723, 2024.arxiv.org/abs/2312.06038
- [42]Yuyang Liu, Qiuhe Hong, Linlan Huang, Alexandra Gomez-Villa, Dipam Goswami, XiaLei Liu, Joost van de Weijer, and Yonghong Tian. Continual learning for vlms: A survey and taxonomy beyond forgetting. arXiv preprint arXiv:2508.04227, 2025b.arxiv.org/abs/2508.04227
- [43]Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 2025.
- [44]Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025a.
- [45]Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025b.
- [46]Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.12832, 2024.arxiv.org/abs/2410.12832
- [47]Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025.arxiv.org/abs/2504.05299
- [48]Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Lever-aging model merging for seamless continual learning. In European Conference on Computer Vision, pages 379–395. Springer, 2024.
- [49]Byeonghu Na, Yeongmin Kim, Minsang Park, Donghyeok Shin, Wanmo Kang, and Il-Chul Moon. Diffusion rejection sampling. arXiv preprint arXiv:2405.17880, 2024.arxiv.org/abs/2405.17880
- [50]Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523, 2020.arxiv.org/abs/2008.11687
- [51]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katrina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
- [52]Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3170–3180, 2023.arxiv.org/abs/2302.12066
- [53]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
- [54]Alexandre Ramé, Kartik Ahuja, Jiany u Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference on Machine Learning, pages 28656–28679. PMLR, 2023a.arxiv.org/abs/2212.10445
- [55]Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:71095–71134, 2023b.arxiv.org/abs/2306.04488
- [56]Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024.arxiv.org/abs/2401.12187
- [57]Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36:3536–3559, 2023.arxiv.org/abs/2306.08877
- [58]Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision. Springer, 2024.arxiv.org/abs/2311.17042
- [59]Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison. arXiv preprint arXiv:2409.09603, 2024.arxiv.org/abs/2409.09603
- [60]Ghada Sokar, Gintare Karolina Dziugaite, Anurag Arnab, Ahmet Iscen, Pablo Samuel Castro, and Cordelia Schmid. Continual learning in vision-language models via aligned model merging. arXiv preprint arXiv:2506.03189, 2025.arxiv.org/abs/2506.03189
- [61]Stability AI. Stable diffusion 3.5. https://stability.ai/news/introducing-stable-diffusion-3-5, 2024. Model release.
- [62]Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022.arxiv.org/abs/2204.03162
- [63]Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri Iyer, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37:87310–87356, 2024.arxiv.org/abs/2406.16860
- [65]Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7280–7290, 2023.arxiv.org/abs/2303.13703
- [66]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228–8238, 2024.arxiv.org/abs/2311.12908
- [67]Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024a.arxiv.org/abs/2401.06080
- [68]Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 2024b.arxiv.org/abs/2408.02666
- [69]Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05236, 2025a.arxiv.org/abs/2503.05236
- [70]Yibin Wang, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, Jiaqi Wang, et al. Unified multimodal chain-of-thought reward model through reinforcement fine-tuning. Advances in neural information processing systems, 38:159130–159157, 2025b.arxiv.org/abs/2505.03318
- [71]Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024c.DOI
- [72]Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannan Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7959–7971, 2022.arxiv.org/abs/2109.01903
- [73]Jie Wu, Yu Gao, Zilyu Ye, Ming Li, Liang Li, Hanzhong Guo, Jie Liu, Zeyue Xue, Xiaoxia Hou, Wei Liu, et al. Rewarddance: Reward scaling in visual generation. arXiv preprint arXiv:2509.08826, 2025.arxiv.org/abs/2509.08826
- [76]Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024.arxiv.org/abs/2410.10629
- [77]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagerider: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023.arxiv.org/abs/2304.05977
- [78]LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International conference on machine learning, pages 2825–2834. PMLR, 2018.arxiv.org/abs/1802.01483
- [79]Yajat Yadav, Zhiyuan Zhou, Andrew Wagenmaker, Karl Pertsch, and Sergey Levine. Robust finetuning of vision-language-action robot policies via parameter merging. arXiv preprint arXiv:2512.08333, 2025.arxiv.org/abs/2512.08333
- [80]Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2051–2060, 2019.arxiv.org/abs/1908.02660
- [81]Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022.arxiv.org/abs/2210.01936
- [82]Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024a.arxiv.org/abs/2408.15240
- [84]Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) as a generalist evaluator for vision-language tasks. arXiv preprint arXiv:2311.01361, 2023.arxiv.org/abs/2311.01361
- [85]Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionntf: Online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117, 2025.arxiv.org/abs/2509.16117
- [86]Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Yaxin Peng, Chaomin Shen, Feifei Feng, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-action model. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5377–5395, 2025.arxiv.org/abs/2502.14420
- [87]Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024.arxiv.org/abs/2401.16335