WorksComputer Science
Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling
Abstract
Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training. DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density. The method requires no learned density model, auxiliary objective, critic, or modification to the underlying policy-gradient objective. In a controlled multimodal navigation task, DARS-GRPO avoids the single-mode collapse exhibited by GRPO and closely recovers the target mode allocation. Applied to text-to-image prompt adaptation, DARS-GRPO consistently improves image and prompt diversity across datasets, Policy LMs, and reward settings while maintaining good semantic alignment. These results show that density-aware reward scaling provides a lightweight way to improve coverage of diverse high-reward outcomes within existing reinforcement-learning pipelines.
Introduction
Reinforcement learning is typically formulated to maximize expected return, which can drive outcome-level probability mass toward a small subset of high-reward outcomes [7]. While desirable for conventional control problems, this behavior is limiting in domains where multiple distinct outcomes are simultaneously valuable. In generative modeling, scientific discovery, and structured prediction, the objective is often not merely to identify a high-reward solution, but to learn a policy that maintains broad coverage over the high-reward outcome space.
Several approaches have been proposed to promote such diversity. Entropy-regularized RL encourages stochastic action selection, but action-level stochasticity does not in general specify a desired distribution over terminal outcomes [8]. Generative Flow Networks (GFlowNets) instead target reward-proportional sampling, providing a principled framework for generating diverse high-reward samples [2]. Applying
GFlowNet-style objectives to autoregressive pretrained models generally requires formulating generation through flow- or trajectory-based objectives rather than retaining an unchanged policy-gradient training pipeline [32]. Inverse Probability Scaling (IPS) offers another route by inversely scaling reward according to terminal-outcome visitation probability [28]. In continuous or high-dimensional outcome spaces, however, exact repeated outcomes may be rare or may not provide a meaningful notion of semantic equivalence. We introduce Density-Aware Reward Scaling (DARS), a simple framework for encouraging a policy that has reward-proportional behavior within standard policy-gradient reinforcement learning. DARS is motivated by the limitations of IPS and replaces discrete visitation frequency with an estimate of local outcome density in continuous outcome spaces. For each on-policy rollout group, outcomes that lie in densely populated regions receive proportionally reduced rewards, while high-reward outcomes in underrepresented regions receive greater relative rewards. This density scaling is applied before advantage estimation and requires no modification to the underlying policy-gradient objective. Importantly, DARS introduces no learned density model, critic, auxiliary loss, or specialized policy architecture; instead it simply modifies reward processing within the existing RL pipeline.
Our central contribution is a simple mechanism for mitigating outcome collapse and promoting coverage of diverse, high-reward regions within standard policy-gradient training. We validate DARS through controlled synthetic experiments that establish its ability to approximate reward-proportional sampling, followed by text-to-image prompt adaptation experiments demonstrating improved diversity while preserving generation quality, and kernel-width ablations that provide mechanistic evidence for its density-aware reward scaling.
Related Work
Mode collapse in RL and loss of diversity: Mode collapse is widely associated with generative models that represent only a small subset of a multimodal data distribution. An analogous failure occurs in reinforcement learning at the level of terminal outcomes: even when multiple trajectories or solutions receive similar rewards, expected-return optimization concentrates the learned policy on a narrow subset of them, reducing the diversity of generated solutions in language and image generation [11, 3]. Existing approaches mitigate this behavior through entropy regularization, KL constraints, reward shaping, or diversity-aware objectives [25, 17], yet they do not directly control the distribution of terminal outcomes. Recent work instead attributes outcome collapse to expected-return optimization itself, where an outcome’s aggregate learning signal is amplified by its current sampling probability, creating a rich-get-richer feedback loop [28]. Reward-proportional methods address this issue by sampling outcomes in proportion to their rewards: GFlowNets achieve this through specialized flow-conservation objectives and have been extended to multi-objective optimization, continuous state spaces, prompt adaptation, and language-model reasoning [1, 2, 10, 15, 33], while IPS instead rescales rewards using inverse visitation probabilities estimated from empirical outcome frequencies [28]. However, empirical visitation frequencies become ineffective in continuous outcome spaces, where exact outcome repetitions are exceedingly rare. Concurrent work [3], also adjusts GRPO rewards using within-group semantic redundancy for mathematical reasoning in LLMs. In contrast, we study density-aware reward scaling as a mechanism for reward-proportional sampling in continuous outcome spaces and directly evaluate recovery of the target reward distribution.
Prompt Adaptation and Image Generation: Loss of output diversity has become increasingly visible in reinforcement learning from human feedback and other forms of large-language-model post-training [19, 11, 3]. Reward optimization can favor a narrow set of response styles or semantic patterns even when many responses are valid. In practice, this behavior is often moderated with KL regularization, reward mixing, early stopping, or stochastic decoding. Prompt-optimization methods learn rewrites that improve text-to-image quality or alignment [9], while PAG learns diverse effective prompts with a GFlowNet formulation over prompt-generation trajectories [32]. CADS acts at diffusion inference time by annealing the conditioning signal during sampling.
Density-Aware Reward Scaling (DARS)
IPS shows that reward-proportional sampling can be achieved by inversely scaling rewards according to outcome probabilities under the current policy [28], but estimating these probabilities through exact visitation frequencies does not extend naturally to continuous outcome spaces, where repeated outcomes are unlikely. DARS instead represents outcomes in a continuous feature space where distances capture meaningful similarity between outcomes. This representation allows visitation to be defined over local regions rather than exact outcome identities. DARS estimates the local density of each sampled outcome within its on-policy rollout group and inversely scales its reward by this density, reducing reinforcement of densely sampled regions
while relatively favoring underrepresented high-reward regions.
Density
Let denote the outcome space and let be a representation map that embeds outcomes into a feature space. For an outcome , let denote its representation. Given an on-policy rollout group containing such and their corresponding rewards , DARS computes the relative local density of each outcome only with respect to the outcomes in the same rollout group. For a nonnegative kernel , this estimate is
where controls how strongly nearby outcomes contribute to the local density. One choice is the Gaussian kernel,
which assigns greater weight to nearby outcomes and smoothly decreases the contribution of outcomes as their distance increases. The kernel width determines the scale of the resulting visitation neighborhoods. As the Gaussian kernel includes each outcome itself with , the local density estimate satisfies .
DARS then applies inverse local-density reward scaling to down-weight frequently visited outcomes and encourage exploration of less populated regions of the outcome space, yielding the scaled reward .
Integration with GRPO
DARS modifies only the reward-processing stage of the GRPO pipeline. GRPO uses a clipped policy-ratio objective, and estimates advantages from groups of sampled outputs [26]. It replaces each task reward with the density-scaled reward from Equation (3), after which the group-relative advantage is computed as
where and are the within-group mean and standard deviation of the scaled rewards. The resulting advantages are passed to the standard GRPO objective without further modification. For each conditioning input, DARS-GRPO samples an on-policy rollout group and evaluates its task rewards. It then computes the within-group local-density scores, scales and normalizes the rewards to obtain group-relative advantages, and applies the standard GRPO policy update without further modification. Thus, DARS requires no change to the GRPO policy ratio, clipped surrogate objective, policy architecture, or optimization procedure; its sole intervention is density-aware reward scaling before advantage normalization.
Behavior
At either kernel width extremes, as , and the rewards remain unchanged; as , and group-relative advantage normalization cancels the shared reward scale. Intermediate bandwidths instead produce non-uniform reward scaling across the given rollout group, thereby modifying the resulting policy-gradient signal.
For -dimensional outcome representations, pairwise kernel evaluation costs time and memory. DARS introduces no learned density model, critic, or auxiliary optimization objective.
Navigation Task: Controlled Distribution-Recovery Experiment
This experiment tests whether DARS can make a policy-gradient RL method act as an approximate reward-proportional sampler in a continuous outcome space. The primary comparison is against GRPO: both methods use the same policy, environment, and terminal reward, but only DARS corrects for local outcome collapse. GFlowNet provides a reward-proportional reference with a different training objective. The experiment is a controlled empirical test of whether DARS approaches the normalized reward distribution.
Environment and Target: We evaluate how closely a policy approaches a known normalized reward density on the unit square . Each episode starts at . At state , the policy either terminates or takes a nonnegative displacement satisfying
with deterministic transition . Trajectories therefore move monotonically up and to the right. The step bound makes distant terminal states require several decisions to reach for the policy. The policy may stop at any visited state except the origin, and it is forced to stop at the boundary or when the rollout horizon is reached. Only the terminal state receives rewards.
The movement policy uses polar coordinates. A neural network predicts a Bernoulli stopping probability and separate mixtures of Beta distributions for a normalized angle and radius. The angle is mapped to , and the radius is rescaled by both and the distance to the boundary. The resulting action is
This guarantees valid first-quadrant movements and lets the policy jointly learn where to move, how far to move, and when to terminate. Since the radial parameterization bounds the Euclidean step length, it imposes the slightly stricter constraint .
The reward is an equally weighted mixture of four isotropic Gaussians, as shown in Figure 1. Each component has standard deviation .
Figure 1. Terminal distributions on the navigation task.
Baselines and Evaluation Metrics
We compare GRPO and DARS-GRPO, using GFlowNet as a non-RL reward-proportional reference. We evaluate the terminal distribution produced by each trained policy using sampled terminal states per trained policy.
Target samples are drawn from the Gaussian mixture and rejected when they fall outside the unit square. We use complementary metrics because no single score captures all forms of distributional mismatch in a multimodal setting.
Mean raw reward is estimated as the empirical average of over sampled terminal states. Jensen–Shannon divergence (JSD) and histogram total variation (Hist. TV) compare the empirical and target densities on a grid. Sliced Wasserstein-2 (SW) (Kolouri et al. [13]), energy distance (Székely et al. [29]), and Sinkhorn divergence (Feydy et al. [4]) compare generated and target samples directly and measure geometric mismatch. RBF maximum mean discrepancy (MMD) (Gretton et al. [6]) is reported at several bandwidths: small bandwidths emphasize local component shape, whereas large bandwidths emphasize mode allocation and global structure. Sliced Wasserstein-2 and energy distance use ten repetitions of 20,000 samples, while Sinkhorn divergence uses 500 samples.
For mode-level evaluation, each terminal state is assigned to its nearest target center. From these assignments, we compute the mass of each mode, the minimum mode mass, and mode TV which measures deviation from uniform target mode weights. Precision is the fraction of generated samples lying inside -nearest-neighbor neighborhoods of target samples, while recall is the fraction of target samples covered by neighborhoods induced by generated samples [14]. Precision therefore measures off-target generation, whereas recall diagnoses missing support and mode collapse.
We measure Wasserstein error at two scales: the between-mode term evaluates mode weights and centers, while the within-mode term evaluates each mode’s mean and covariance. Their sum defines the hierarchical score,
HDR outside mass is the fraction of generated samples outside the target’s 95% highest-density region. Since the target assigns approximately 0.05 probability mass outside this region, we report the calibration error , for which lower values indicate better agreement with the target.
Results and Interpretation
The primary result is the comparison with GRPO. In the reported run, GRPO concentrates on a single high-reward mode, whereas DARS-GRPO assigns substantial mass to all four modes and approaches their target weights. The DARS-GRPO point estimates are closer to the target than GRPO across the metrics reported in Table 1. GRPO’s higher mean raw reward reflects tight concentration near one of four equally high peaks, rather than agreement with the normalized reward distribution. Thus, within this controlled environment, the reported run provides evidence that density reweighting prevents the evaluated policy from concentrating on one mode, recovers the correct mode weights, and improves global geometric agreement with the known target relative to GRPO. This is the central controlled contribution of the experiment.
| Metric | GFlowNet | GRPO | DARS-GRPO |
| Density and transport | |||
| Mean raw reward ↑ | 3.44 | 6.96 | 4.59 |
| JSD ↓ | 0.020 | 0.649 | 0.041 |
| Hist. TV ↓ | 0.149 | 0.975 | 0.219 |
| SW ↓ | 0.036 | 0.242 | 0.015 |
| Energy ↓ | 0.007 | 0.281 | 0.002 |
| Mode allocation and support | |||
| Mode TV ↓ | 0.104 | 0.750 | 0.014 |
| Min. mode mass ↑ | 0.187 | 0.000 | 0.236 |
| Precision ↑ | 0.998 | 0.997 | 0.998 |
| Recall ↑ | 0.996 | 0.032 | 0.985 |
| Hierarchical ↓ | 0.128 | 0.359 | 0.073 |
| Per-mode and local diagnostics (desired mass: 0.25) | |||
| Mode 1 mass | 0.318 | 0.000 | 0.254 |
| Mode 2 mass | 0.187 | 0.000 | 0.252 |
| Mode 3 mass | 0.286 | 0.000 | 0.258 |
| Mode 4 mass | 0.209 | 1.000 | 0.236 |
| Between-mode ↓ | 0.113 | 0.335 | 0.046 |
| Within-mode ↓ | 0.016 | 0.023† | 0.027 |
| HDR outside-mass error ↓ | 0.001 | 0.050 | 0.044 |
| RBF MMD² by bandwidth | |||
| ↓ | 0.001 | 0.703 | 0.002 |
| ↓ | 0.004 | 0.832 | 0.004 |
| ↓ | 0.007 | 0.803 | 0.006 |
| ↓ | 0.011 | 0.690 | 0.002 |
| ↓ | 0.010 | 0.372 | 0.001 |
| Sinkhorn divergence by regularization | |||
| ↓ | 0.008 | 0.115 | 0.002 |
| ↓ | 0.008 | 0.115 | 0.002 |
| ↓ | 0.008 | 0.114 | 0.001 |
| ↓ | 0.008 | 0.112 | 0.001 |
Table 1. Navigation Task Results (avg over 5 seeds).
DARS-GRPO closely matches, and on several key metrics surpasses, GFlowNet despite requiring only a simple reward-scaling modification within a standard policy-gradient pipeline. Both methods achieve identical precision, indicating equally accurate recovery of high-reward outcomes. DARS consistently improves metrics that capture the global reward distribution, including sliced Wasserstein, energy distance, mode total variation, minimum mode mass, hierarchical Wasserstein error, HDR outside mass, and most larger-bandwidth MMD and Sinkhorn metrics, demonstrating more faithful recovery of target mode proportions and global geometry.
GFlowNet performs slightly better on metrics emphasizing fine-scale local density, such as Jensen–Shannon divergence, histogram total variation, within-mode Wasserstein error, recall, and small-bandwidth MMD, reflecting its specialized flow-based objective. Nevertheless, these results show that a lightweight modification to conventional policy-gradient RL is sufficient to achieve reward-proportional behavior that is competitive with a substantially more sophisticated algorithmic framework.
Having established in a controlled setting that DARS approximates reward-proportional sampling, we next ask whether this property is useful in a real continuous-output RL problem where the target reward distribution is unknown.
Text-to-Image Prompt Adaptation
Motivation
Prompt adaptation for image generation refers to the setting in which a language model rewrites the input prompt to match a desired set of outcomes before image generation. Text-to-image prompt adaptation provides a natural real-world benchmark for evaluating reward-proportional reinforcement learning. Unlike the synthetic experiments in § 3, where the target reward distribution is explicitly known, the reward-proportional distribution over generated images is intractable and cannot be directly characterized. A single text prompt typically admits many semantically faithful realizations that differ in composition, viewpoint, lighting, object arrangement, or artistic style while remaining equally valid.
Since the underlying reward distribution is unavailable, directly measuring reward-proportional sampling is infeasible. Instead, we evaluate its consequences through observable proxies. If DARS successfully distributes probability mass across multiple high-reward regions, it should produce a more diverse set of images while preserving prompt alignment and image quality. We therefore measure visual diversity together with alignment and preference metrics as an indirect estimate of improved coverage of the latent reward landscape. Consequently, the objective is not merely to identify a single high-reward image, but to maintain coverage over the diverse set of high-quality outcomes consistent with the prompt.
Since input prompts are the conditioning signal in diffusion models, semantically faithful prompt rewrites help steer the generation in desired directions. Learning a distribution over such rewrites therefore provides an effective mechanism for exploring multiple high-reward image modes while keeping the underlying image generator fixed.
Following this formulation, we fine-tune a language model using reinforcement learning to rewrite input prompts. Images generated from the rewritten prompts are rewarded as defined in section 5.2. Since multiple visually distinct images can achieve comparable rewards, broader coverage of the latent reward landscape should translate to greater image diversity without sacrificing fidelity. To quantify this diversity, we measure distances in the DINOv2 representation space as a proxy for perceptual similarity ([18]).

Figure 2. Overview of DARS-GRPO for text-to-image prompt adaptation. A trainable language-model policy samples rewrites of an input prompt, and a frozen text-to-image model generates one image per rewrite. Each image receives a task reward and is mapped by a frozen image encoder into a representation space, where a Gaussian kernel computes its within-group density . DARS forms the scaled reward , reducing the contribution of high-reward images in dense representation-space regions and increasing the relative contribution of high-reward images in sparsely represented regions. GRPO then uses these scaled rewards to update only the prompt-rewriting policy.
Reward Definitions
The reward should preserve two complementary properties: semantic fidelity to the original prompt and visual appeal. Fidelity alone does not ensure that an image is aesthetically strong, whereas optimizing aesthetics alone can favor attractive images that drift from the requested content. We therefore evaluate on two different reward functions: alignment-only reward and a combined reward consisting of aesthetic and alignment components.
Let denote the original prompt and an image generated from its th rewrite. We measure semantic alignment using the image-text cosine similarity from a CLIP model ([20]). A binary semantic gate determines whether the image is sufficiently aligned with the original prompt. Thus the alignment reward is
where is the minimum accepted CLIP similarity. Images below this threshold receive zero reward to encourage strict adherence. This setting tests whether DARS can broaden coverage within the set of sufficiently aligned images.
To measure visual appeal, we use LAION-Aesthetics Predictor ([24]), which maps images to an aesthetic score on a 1–10 scale. Let denote this predictor, and let be an image generated from the original input prompt. We define the aesthetic improvement and its multiplicative reward factor as
An aesthetic improvement produces a factor greater than one, no change produces a factor of one, and a decrease produces a factor between zero and one.
Thus, the combined reward is given as
Density Calculation and Reward Scaling
DARS estimates local density in the space of generated images, since our objective is to promote diversity over visual outcomes rather than textual rewrites. As discussed in § 2.1, we instantiate using a frozen DINOv2 encoder, which maps each generated image to its unnormalized class-token (CLS) embedding . Local densities are then estimated in this representation space.
For a group of images, let denote the feature vector of image . We compute the local density score for image using a Gaussian kernel as
Figure 3. Comparison of prompt rewrites and their corresponding generated images for “A snowman standing in the yard” using GRPO, PAG, and DARS-GRPO. GRPO produces nearly identical rewrites and visually similar nighttime scenes, whereas PAG introduces moderate variation and DARS-GRPO generates more diverse rewrites, settings, and visual compositions.
Following Equation (3), DARS rescales the task reward as . We use this scaled reward to optimize the prompt-rewriting policy with DARS-GRPO.
Evaluation Metrics
We evaluate each prompt-rewriting policy along three complementary dimensions: image diversity, rewrite diversity, and image fidelity or preference. Image diversity is measured in DINOv2 CLS representation space using the Vendi Score {(citep)}(Friedman and Dieng 2023) of the cosine-similarity matrix of L2-normalized CLS embeddings, and the mean pairwise Euclidean distance between CLS embeddings. Rewrite diversity is measured analogously in CLIP text-embedding space using Prompt Vendi and the mean pairwise cosine distance among rewritten prompts. Finally, CLIP-align, HPSv2 {(citep)}(Wu et al. 2023), and PickScore {(citep)}(Kirstain et al. 2023) assess the semantic compatibility and human-preference-style quality of each generated image relative to the input prompt. CLIP-align is the nonnegative cosine similarity between the L2-normalized CLIP embeddings of a generated image and its original prompt. Diversity metrics are computed within each group of rewrites or images generated for the same original prompt and then averaged across the evaluation set, following within-prompt conditional evaluation. For all metrics, higher values indicate greater diversity or stronger fidelity and preference, respectively.
Experiment Details
We use a frozen Stable Diffusion v1.4 generator {(citep)}(Rombach et al. 2022) and optimize only the prompt-rewriting policy. For all baselines and reward functions, both Qwen3-0.6B(base) {(citep)}(Yang et al. 2025) and GPT-2 (124M) {(citep)}(Radford et al. 2019) are already supervised fine-tuned on a shared set of curated prompt rewrites. We then train the models on 1,024 randomly sampled MS-COCO prompts {(citep)}(Lin et al. 2014) and evaluate them on disjoint MS-COCO and Lexica prompts {(citep)}(Shen et al. 2023). DARS uses for Gaussian kernels, and the alignment reward uses a CLIP-similarity threshold of 0.2.
We compare DARS-GRPO with vanilla GRPO, PAG, and GFlowNet prompt optimization, together with Stable Diffusion v1.4 with random seed initialization and CADS {(citep)}(Sadat et al. 2024) as control baselines. All methods are evaluated using the same held-out prompts, generation budget, diffusion model, and sampling procedure. Diffusion seeds are sampled independently across methods, so image-diversity metrics also capture variation introduced by stochastic diffusion sampling while prompt-diversity metrics are unaffected by this confound.
Results and Interpretation
Across both datasets, Policy LMs, and reward settings, DARS-GRPO consistently improves upon vanilla GRPO across all three diversity metrics (Table 2) without significant loss of image fidelity. Further, among all prompt-adaptation methods, it achieves the highest DINO pairwise distances and Image Vendi scores, indicating that density-aware reward scaling broadens the learned rewrite distribution and increases coverage in DINOv2 image-representation space. The high Prompt Vendi and diversity scores further indicate that this image diversity arises from diverse prompt rewrites, while CLIP-align, HPSv2, and PickScore imply close image fidelity.
| Policy LM | Method | Image Vendi ↑ COCO | Image Vendi ↑ Lexica | CLIP-align ↑ COCO | CLIP-align ↑ Lexica | HPSv2 ↑ COCO | HPSv2 ↑ Lexica | PickScore ↑ COCO | PickScore ↑ Lexica | Prompt Vendi ↑ COCO | Prompt Vendi ↑ Lexica | Prompt diversity ↑ COCO | Prompt diversity ↑ Lexica | DINO pairwise distance ↑ COCO | DINO pairwise distance ↑ Lexica |
| – | SD v1.4 | 8.60 | 9.52 | 0.27 | 0.27 | 0.26 | 0.25 | 21.53 | 20.91 | – | – | – | – | 40.58 | 41.84 |
| – | CADS | 12.93 | 14.08 | 0.26 | 0.26 | 0.24 | 0.23 | 20.86 | 20.28 | – | – | – | – | 46.22 | 46.96 |
| Combined reward | |||||||||||||||
| Qwen3-0.6B | GRPO | 8.76 | 9.81 | 0.24 | 0.24 | 0.23 | 0.23 | 20.92 | 20.69 | 1.01 | 1.04 | 0.00 | 0.01 | 40.90 | 42.10 |
| Qwen3-0.6B | DARS-GRPO | 13.04 | 14.45 | 0.24 | 0.24 | 0.23 | 0.23 | 20.80 | 20.37 | 1.43 | 1.67 | 0.05 | 0.06 | 46.92 | 47.89 |
| Qwen3-0.6B | PAG | 8.25 | 11.51 | 0.26 | 0.26 | 0.25 | 0.24 | 21.34 | 20.69 | 1.26 | 1.59 | 0.04 | 0.07 | 40.20 | 44.64 |
| Qwen3-0.6B | GFN | 8.74 | 11.69 | 0.26 | 0.26 | 0.25 | 0.24 | 21.31 | 20.69 | 1.22 | 1.59 | 0.04 | 0.07 | 40.50 | 44.71 |
| GPT-2 | GRPO | 9.60 | 10.68 | 0.24 | 0.24 | 0.23 | 0.23 | 20.87 | 20.43 | 1.09 | 1.18 | 0.01 | 0.03 | 41.85 | 42.87 |
| GPT-2 | DARS-GRPO | 12.72 | 14.67 | 0.23 | 0.23 | 0.23 | 0.23 | 20.68 | 20.26 | 1.63 | 1.78 | 0.08 | 0.10 | 47.75 | 48.60 |
| GPT-2 | PAG | 9.22 | 14.03 | 0.26 | 0.24 | 0.25 | 0.23 | 21.16 | 20.35 | 1.61 | 2.14 | 0.09 | 0.13 | 41.36 | 46.43 |
| GPT-2 | GFN | 9.02 | 13.67 | 0.25 | 0.24 | 0.24 | 0.22 | 21.12 | 20.26 | 1.15 | 1.69 | 0.02 | 0.08 | 41.03 | 46.67 |
| Alignment-only reward | |||||||||||||||
| Qwen3-0.6B | GRPO | 9.47 | 10.94 | 0.26 | 0.26 | 0.25 | 0.24 | 21.24 | 20.55 | 1.59 | 1.64 | 0.07 | 0.08 | 41.68 | 43.52 |
| Qwen3-0.6B | DARS-GRPO | 20.13 | 20.03 | 0.22 | 0.21 | 0.22 | 0.22 | 20.34 | 19.81 | 2.43 | 2.40 | 0.16 | 0.16 | 56.23 | 55.76 |
| Qwen3-0.6B | PAG | 9.53 | 13.88 | 0.25 | 0.24 | 0.25 | 0.24 | 21.22 | 20.48 | 1.79 | 2.33 | 0.11 | 0.15 | 41.99 | 46.86 |
| Qwen3-0.6B | GFN | 8.93 | 12.13 | 0.26 | 0.26 | 0.25 | 0.24 | 21.27 | 20.60 | 1.30 | 1.62 | 0.05 | 0.08 | 40.99 | 44.97 |
| GPT-2 | GRPO | 8.76 | 10.22 | 0.26 | 0.26 | 0.26 | 0.24 | 21.42 | 20.66 | 1.03 | 1.11 | 0.01 | 0.02 | 40.93 | 42.52 |
| GPT-2 | DARS-GRPO | 22.87 | 22.64 | 0.23 | 0.22 | 0.23 | 0.22 | 20.58 | 19.89 | 2.15 | 2.29 | 0.14 | 0.15 | 55.57 | 55.00 |
| GPT-2 | PAG | 8.51 | 13.15 | 0.25 | 0.24 | 0.25 | 0.23 | 21.23 | 20.38 | 1.20 | 1.66 | 0.03 | 0.07 | 40.55 | 45.78 |
| GPT-2 | GFN | 8.60 | 14.08 | 0.26 | 0.24 | 0.24 | 0.22 | 21.19 | 20.33 | 1.17 | 1.79 | 0.03 | 0.08 | 40.54 | 46.03 |
Table 2. Prompt Adaptation for Text-to-Image Generation results on MS-COCO and Lexica (avg over 5 seeds).
Changing the reward function affects performance across all metrics, showing that DARS responds to the reward landscape rather than producing diversity gains randomly. Consistent with our hypothesis, the binary gated alignment-only reward yields broader sampling from uniformly scored reward zones, whereas the combined reward forces the policy to learn a more dynamic reward landscape characterized by aesthetically-driven peaks, thus, sampling from a more varied distribution as reflected by the higher average Image Vendi scores.
Having demonstrated the effectiveness of DARS on a real-world RL problem with an intractable reward-proportional distribution, we next study the effect of the kernel width used for local density estimation on the learned outcome distribution.
Ablation Study
As discussed in 3.3, kernel width affects reward-scaling in DARS and hence needs to be calibrated for accurate density estimation. For the experiments in Section 5, we demonstrate the direct effects of varying kernel widths on image-diversity produced by DARS (Table 3). Diversity metrics consistently peak at intermediate kernel widths and degrade when the bandwidth is either too small or too large, while image quality remains largely unchanged. This behavior reflects how DARS estimates local outcome density. With a very small kernel width, nearly every generated image appears isolated, causing density estimates to be dominated by self-similarity and resulting in almost uniform reward scaling. Conversely, an excessively large kernel width treats distant images as belonging to the same neighborhood, oversmoothing density estimates and diminishing the relative reward differences that encourage exploration. Intermediate kernel widths instead resolve meaningful local neighborhoods allowing for effective density estimation in DARS.
| Width | Image Vendi ↑ | CLIP-align ↑ | Prompt Diversity ↑ | Prompt Vendi ↑ |
| 10 | 9.99 | 0.26 | 0.12 | 1.99 |
| 15 | 14.12 | 0.25 | 0.21 | 3.34 |
| 20 | 18.01 | 0.24 | 0.17 | 2.74 |
| 25 | 19.77 | 0.23 | 0.22 | 3.47 |
| 30 | 12.14 | 0.25 | 0.11 | 2.09 |
Table 3. Kernel-width ablation results (avg over 5 seeds)
Limitations
DARS relies on a meaningful representation space and kernel bandwidth for estimating local outcome density. Its effectiveness may therefore degrade when embedding distances poorly reflect semantic similarity or when no single bandwidth captures the relevant neighborhood structure. The pairwise density computation also scales quadratically with rollout-group size, potentially limiting its use with large groups. Our experiments focus on a controlled navigation task and text-to-image prompt adaptation with fixed image representations and generators. Broader evaluation across RL domains, policy architectures, and learned representations is needed. Finally, our results establish empirical approximation to reward-proportional sampling rather than a general convergence guarantee.
Conclusion
We introduced Density-Aware Reward Scaling (DARS), a lightweight mechanism for promoting diverse, high-reward outcomes within standard policy-gradient training. By inversely scaling rewards using local outcome density, DARS counteracts repeated reinforcement of densely sampled regions without requiring a critic, auxiliary objective, learned density model, or modified policy-gradient loss. In a controlled multimodal navigation task, DARS-GRPO prevents the mode collapse observed with GRPO and closely recovers target mode allocation. In text-to-image prompt adaptation, it consistently increases prompt and image diversity across datasets, Policy LMs, and reward settings while retaining semantic fidelity. These results position density-aware reward scaling as a simple yet effective approach to improving outcome coverage.
Mathematical Foundations for DARS
Inverse policy-density scaling yields a reward-proportional policy
Let the continuous outcome space be . At training time , let the policy induce a probability density on . Let the reward be a measurable function satisfying
and assume wherever . The desired reward-proportional density is
We prove the continuous analogue of inverse probability scaling. Parameterize an unrestricted policy density by a logit function ,
At iteration , freeze the current policy density in the reward denominator and use
For the population objective
the following result holds.
Theorem 1 At the current policy , the functional policy-gradient direction is
Hence the reward-proportional density is the unique stationary probability density, interpreted in the closure of the exponential-family parameterization when vanishes on part of the outcome space. Under the corresponding population gradient flow, every convergent density limit is .
Proof 1 For any perturbation , the normalized exponential density obeys
Because the denominator in Equation (13) is stop-gradient,
Since this identity holds for every perturbation , Equation (15) follows. A stationary density must satisfy
almost everywhere, and therefore
The direction is not merely stationary at ; it points toward it. Consider the population gradient flow
and the functional
Its functional derivative is . Therefore, along the gradient flow,
Equality holds only when almost everywhere. Thus the only stationary density, and hence any convergent population-gradient-flow density limit, is the reward-proportional density in Equation (11).
The cancellation underlying the result can also be seen directly. A small region around is sampled with probability approximately . Its per-sample scaled reward is . Thus its aggregate expected learning weight is proportional to
The current policy density cancels, leaving learning mass proportional only to reward mass.
The Gaussian kernel density score is proportional to the policy density
Draw a rollout group of outcomes independently from the current policy,
DARS uses the unnormalized Gaussian kernel,
Instead of separating any sample from the rest of the group, define the density-score field produced by the entire rollout group,
The score assigned to a sampled outcome is simply
Thus the same group of all policy samples is used throughout.
Define
Then is the density and
Equation (34) is exactly the Gaussian kernel-density estimator formed from the full rollout group.
Theorem 2 (Gaussian density approximation). For every fixed ,
where
Assume that, on the region of interest, is twice continuously differentiable,
and . If the region has a boundary, assume either that admits a extension to a neighborhood of that region or that the statement is restricted to interior points at the bandwidth scale. Then
and
Consequently, for a fixed query point and any , with probability at least ,
Proof 2 Using the full group in Equation (34) and linearity of expectation,
No element of the rollout group is separated from the others. Let . Then, away from boundary effects as specified above,
A second-order Taylor expansion gives
for a point on the segment between and . Since and
we obtain
which proves Equation (38).
For fixed , the terms in Equation (34) are independent and identically distributed. Hence
For the Gaussian kernel,
which proves Equation (39). Chebyshev’s inequality together with Equation (38) gives Equation (40).
The preceding theorem shows directly how the approximation depends on bandwidth and group size:
The term is the Gaussian smoothing bias, while the stochastic error decreases as the effective number of samples in an -scale neighborhood, , increases.
Equation (40) is stated for a fixed query point because that is the elementary finite-sample calculation. To evaluate the same full-group estimator at the sampled outcomes without separating any sample, use the standard uniform-consistency property of Gaussian kernel-density estimators. On a compact region satisfying the regularity assumptions above,
provided and . Because Equation (54) holds simultaneously for every , it also holds when is any of the observed outcomes . Therefore,
In particular, on any region where , if and are chosen so that the right-hand error in Equation (55) is small relative to , then
Thus the DARS Gaussian score is approximately the policy probability density multiplied by the same global factor for all outcomes in the region of interest.
The bandwidth tradeoff is now explicit. The bandwidth should be small enough that
but large enough that
is not too small. In the standard regime
the approximation error vanishes uniformly on the region of interest.
Finally, Equation (56) implies
Thus, when the Gaussian kernel-density estimate is accurate, DARS differs from exact inverse policy-density scaling only by a common positive factor and a small multiplicative estimation error. Section A.1 then gives the corresponding reward-proportional population target.
Navigation Task
Figure 4 visualizes the movement fields induced by the trained GRPO and DARS-GRPO policies. At each state , actions are sampled repeatedly from the policy and used to estimate the mean one-step displacement . The streamlines indicate its direction, their color indicates its magnitude, and the gray contours show the normalized reward landscape.
Figure 4. Policy-induced movement fields for GRPO (left) and DARS-GRPO (right) on the four-mode navigation task. Streamlines show the estimated mean one-step displacement, color indicates its magnitude, and gray contours show the normalized reward landscape.
Terminal actions are treated as zero displacement. Consequently, a small field magnitude may result from small proposed movements, frequent termination, or cancellation among sampled actions. The DARS-GRPO optimised the policy such that the coverage over all the modes could be optimised in proportion to the reward. The visualization thus shows how density-aware reward scaling changes the learned navigation dynamics. It should not be interpreted as a gradient or parameter-update field.
Text-to-Image Prompt Adaptation
LM model choice: GPT-2 vs Qwen3-0.6B Figure 5–Figure 8 compare GPT-2 and Qwen3-0.6B as policy LMs under the combined aesthetic and alignment reward. For each original prompt, GRPO generations are shown on the left and DARS generations on the right while keeping the diffusion model, sampling budget, and evaluation protocol fixed. This isolates the effect of the language-model used from that of the reinforcement learning objective.

Figure 5. Policy LM comparison for “A snowman standing in the yard” (top) and “A bicycle leaning against a brick wall” (bottom). Each embedded grid compares GRPO with DARS for the indicated language-model.

Figure 8. Policy LM comparison for “A lantern hanging outside a cottage” (top) and “A fox standing near a forest path” (bottom). Across both policy LMs, DARS broadens the within-prompt visual distribution while maintaining the prompt’s central subject.
Across both policy LMs, DARS consistently expands the diversity of generated images while preserving the semantic content of the original prompt. Compared with GRPO, DARS produces greater variation in scene layout, viewpoint, lighting, object appearance, and background context without substantially altering the primary subject. Although GPT-2 tends to make larger structural changes and Qwen3-0.6B generally preserves a more stable object identity, both exhibit the same qualitative benefit from density-aware reward scaling. These examples illustrate that the improvements obtained by DARS are not tied to a particular language model and generalize across different policy LMs.
Reward choice: Combined vs Alignment-only Figure 9–Figure 12 compare the two reward functions using the same Qwen3-0.6B policy LM. The combined reward combines semantic fidelity with improvement under the frozen aesthetic scorer, whereas the alignment-only reward applies only the minimum CLIP-alignment constraint. Each embedded grid compares GRPO and DARS under one reward, allowing the effect of reward design to be separated from the effect of density scaling.

Figure 9. Reward comparison for “A snowman standing in the yard” (top) and “A bicycle leaning against a brick wall” (bottom), using Qwen3-0.6B.

Figure 12. Reward comparison for “A lantern hanging outside a cottage” (top) and “A fox standing near a forest path” (bottom).
The alignment-only objective permits DARS to explore a substantially broader visual region since it is free of aesthetic bounds that affect the combined reward. Across the examples, it produces larger changes in setting, illumination, camera distance, and composition. This is particularly visible for the bench, cat, scarecrow, and fox prompts in Figure 10, Figure 11, and Figure 12 where the generated scenes include a wider range of architectural and natural backgrounds. The quantitative results show the same pattern with higher Image Vendi Scores and DINO pairwise distance for alignment-only rewards.

Figure 10. Reward comparison for “A wooden bench under a tree” (top) and “A cat sleeping on a windowsill” (bottom).

Figure 11. Reward comparison for “A scarecrow in the middle of a farm” (top) and “A castle on top of a hill” (bottom).
Additional qualitative comparisons Figure 13–Figure 15 provide additional examples comparing GRPO, PAG, GFN, and DARS-GRPO. Across these prompts, GRPO reuses a narrow rewriting pattern, whereas DARS produces more varied scene contexts consequently leading to more diverse images.
Figure 13. Qualitative comparison for “A bicycle leaning against a brick wall.” GRPO repeatedly generates a similar nighttime composition, while DARS-GRPO varies lighting, seasonal context, and surrounding details while preserving the bicycle and wall.
Figure 15. Qualitative comparison for “A cat sleeping on a windowsill.” All methods preserve the primary subject, while DARS-GRPO introduces broader variation in background scenery, lighting, and framing.

Figure 6. Policy LM comparison for “A wooden bench under a tree” (top) and “A cat sleeping on a windowsill” (bottom). DARS expands the visual contexts while preserving the object and scene requested by each prompt.

Figure 7. Policy LM comparison for “A scarecrow in the middle of a farm” (top) and “A castle on top of a hill” (bottom). The DARS samples show greater variation in character appearance, architecture, framing, and illumination than the corresponding GRPO samples.
Figure 14. Qualitative comparison for “A wooden bench under a tree.” DARS-GRPO explores more varied locations, tree appearances, and visual contexts than GRPO, while PAG and GFN produce more limited changes around the same central composition.
Implementation Details
Experimental Configuration Unless otherwise specified, all experiments use the same training, generation, and evaluation settings. Table 4 summarizes the policy optimization hyperparameters shared by DARS and the GRPO baseline, while Table 5 lists the hyperparameters specific to density-aware reward scaling. The generation pipeline, reward formulation, and datasets are held fixed across experiments unless explicitly varied in an ablation study.
| Parameter | Value |
| Policy LMs | SFT Qwen3-0.6B; SFT GPT-2 |
| Training temperature | 1.5 |
| Evaluation temperature | 1.0 |
max_new_tokens | 64 |
| Learning rate | |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| PPO clipping | 0.25 |
| KL coefficient | |
| Policy updates per rollout | 1 |
| Training batch size | 2 prompts |
Table 4. Policy optimization hyperparameters (DARS and GRPO).
| Parameter | Value |
| Density embedding space | dinov2_image |
| Density kernel | Gaussian |
| Kernel width () | 20.0 |
Table 5. DARS-specific hyperparameters.
DARS and GRPO are trained under identical optimization settings to ensure a fair comparison. Both methods use the same policy LM, optimizer hyperparameters, decoding configuration, and rollout schedule. They differ only in the reward scaling mechanism. Unless otherwise stated, these hyperparameters are used for all reported experiments.
Table 5 lists the hyperparameters specific to DARS. Density is estimated in the DINOv2 image embedding space using a Gaussian kernel with bandwidth . To study the effect of density estimation, all kernel-width ablations retain the same training configuration while varying only the kernel bandwidth.
Kernel-width ablations use the same experimental setup while varying only the Gaussian kernel width,
with all other hyperparameters kept fixed.
Computing Environment All experiments were conducted on a server equipped with four NVIDIA A100-SXM4 GPUs using the software environment summarized in Table 6. Mixed-precision training uses bf16, while Stable Diffusion inference is performed in fp16 to improve computational efficiency without affecting evaluation.
| Component | Value |
| Operating System | Ubuntu 22.04.4 LTS (Linux 5.15.0-94-generic) |
| GPUs | NVIDIA A100-SXM4-40GB |
| Host RAM | 503 GiB |
| Python | 3.13.9 |
| PyTorch | 2.12.1+cu130 |
| Key Libraries | transformers 5.13.0, diffusers 0.39.0, accelerate 1.14.0, peft 0.19.1 |
| Mixed Precision | bf16 (run.mixed_precision) |
| Diffusion Dtype | fp16 |
Table 6. Hardware and software environment used for training.
Training Configuration Table 4 and Table 5 summarize the optimization hyperparameters and DARS-specific settings used during training. Unless explicitly stated otherwise, all experiments employ these default configurations. DARS and GRPO differ only in the computation of policy rewards, while sharing identical optimization settings.
Generation Configuration All generated images are produced using the Stable Diffusion v1.4 pipeline with identical sampling settings across methods. Each rewritten prompt is decoded independently using 20 diffusion steps with a classifier-free guidance scale of 7.5. The same CLIP encoder, DINOv2 encoder, and aesthetic predictor are used throughout both training and evaluation to ensure consistency across reported metrics.
Reward Configuration Two reward variants are considered in this work. The primary configuration combines aesthetic improvement with CLIP-based semantic alignment, while the alignment-only variant removes the aesthetic component to isolate the contribution of density-aware reward scaling. Apart from the reward definition, all experimental settings remain unchanged.
Evaluation Protocol Evaluation follows the same protocol for all methods. For each evaluation prompt, the policy generates rewritten prompts, each of which is used to
generate one image under identical diffusion sampling settings. Diversity metrics are computed only among images originating from the same source prompt and are then averaged across the evaluation set.
Image-text alignment is measured using OpenAI CLIP ViT-L/14, while image diversity is computed from DINOv2 image embeddings. The aesthetic LAION predictor (sac+logos+ava1-l14-linearMSE) is used wherever aesthetic reward is required.
During training, model checkpoints are periodically evaluated on the validation set using the same evaluation protocol. The checkpoint used for all reported experiments is selected based on the highest validation Vendor Score after reward saturation, ensuring that model selection aligns with the diversity objective optimized by DARS.
Datasets The MS-COCO training split is used for policy optimization, while evaluation is performed on held-out MS-COCO prompts and the Lexica benchmark to assess both in-domain and out-of-domain generalization. Unless otherwise stated, all reported quantitative results use these dataset splits.
| Parameter | Value |
| Diffusion model | CompVis/stable-diffusion-v1-4 |
| Resolution / Steps / CFG | 512 512, 20 steps, guidance scale 7.5 |
| Scheduler | DPM-Solver Multistep |
diffusion.seed_mode | random |
run.seed | 42 |
| Rewrites per prompt () | 64 |
| CLIP encoder | OpenAI CLIP ViT-L/14 |
| DINOv2 encoder | facebook/dinov2-base (CLS embedding) |
| Aesthetic predictor | sac+logos+ava1-l14-linearMSE |
| CLIP alignment threshold | 0.20 |
Table 7. Shared generation and evaluation configuration.
| Setting | Implementation |
| Combined | Product-style aesthetic gain with CLIP alignment gating |
| Alignment only | CLIP alignment reward without aesthetic gain |
Table 8. Reward settings.
| Split | Dataset |
| Training | dataset/coco/n_1024/train.jsonl |
| MS-COCO evaluation | dataset/coco/n_1024/test.jsonl |
| Lexica evaluation | dataset/lexica/n_1024/test.jsonl |
Table 9. Training and evaluation datasets.
References
- [1]Bengio, E.; Jain, M.; Korablyov, M.; Precup, D.; and Bengio, Y. 2021. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. In Advances in Neural Information Processing Systems, volume 34, 27381–27394.arxiv.org/abs/2106.04399
- [2]Bengio, Y.; Lahlou, S.; Deleu, T.; Hu, E. J.; Tiwari, M.; and Bengio, E. 2023. GFlowNet Foundations. Journal of Machine Learning Research, 24(210): 1–55.
- [3]Chen, Y.; Chakraborty, S.; Wolf, L.; Paschalidis, I. C.; and Pacchiano, A. 2026. Post-training Large Language Models for Diverse High-Quality Responses. In International Conference on Learning Representations.arxiv.org/abs/2509.04784
- [4]Feydy, J.; Séjourné, T.; Vialard, F.-X.; Amari, S.-i.; Trouvé, A.; and Peyré, G. 2019. Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, 2681–2690.arxiv.org/abs/1810.08278
- [6]Gretton, A.; Borgwardt, K. M.; Rasch, M. J.; Schölkopf, B.; and Smola, A. 2012. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25): 723–773.DOI
- [7]GX-Chen, A.; Prakash, J.; Guo, J.; Fergus, R.; and Ranganath, R. 2025. KL-Regularized Reinforcement Learning is Designed to Mode Collapse. arXiv preprint arXiv:2510.20817.arxiv.org/abs/2510.20817
- [8]Haarnoja, T.; Zhou, A.; Abb eel, P.; and Levine, S. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 1861–1870.arxiv.org/abs/1801.01290
- [9]Hao, Y.; Chi, Z.; Dong, L.; and Wei, F. 2023. Optimizing Prompts for Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, 66923–66939.arxiv.org/abs/2212.09611
- [10]Jain, M.; Raparthy, S. C.; Hernández-García, A.; Rector-Brooks, J.; Bengio, Y.; Miret, S.; and Bengio, E. 2023. Multi-Objective GFlowNets. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 14631–14653.arxiv.org/abs/2210.12765
- [11]Kirk, R.; Mediratta, I.; Nalmpantis, C.; Luketina, J.; Hambro, E.; Grenfestette, E.; and Raileanu, R. 2024. Understanding the Effects of RLHF on LLM Generalisation and Diversity. In International Conference on Learning Representations.arxiv.org/abs/2310.06452
- [13]Kolouri, S.; Nadjahi, K.; Simsekli, U.; Badeau, R.; and Rohde, G. 2019. Generalized sliced wasserstein distances. Advances in neural information processing systems, 32.arxiv.org/abs/1902.00434
- [14]Kyngkäänniemi, T.; Karras, T.; Laine, S.; Lehtinen, J.; and Aila, T. 2019. Improved Precision and Recall Metric for Assessing Generative Models. In Advances in Neural Information Processing Systems, volume 32, 3927–3936.
- [15]Lahlou, S.; Deleu, T.; Lemos, P.; Zhang, D.; Volokhova, A.; Hernández-García, A.; Ezzine, L. N.; Bengio, Y.; and Malkin, N. 2023. A Theory of Continuous Generative Flow Networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 18269–18300.arxiv.org/abs/2301.12594
- [17]Liu, H.; Huang, H.; Wang, J.; Liu, C.; Li, X.; and Ji, X. 2026. DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1864–1873.arxiv.org/abs/2512.21514
- [18]Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. 2023. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint arXiv:2304.07193.arxiv.org/abs/2304.07193
- [19]Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, volume 35, 27730–27744.arxiv.org/abs/2203.02155
- [20]Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 8748–8763.arxiv.org/abs/2103.00020
- [24]Schuhmann, C. 2022. LAION-Aesthetics. https://laion.ai/blog/laion-aesthetics/. Accessed: 2026-07-29.
- [25]Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.arxiv.org/abs/1707.06347
- [26]Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.arxiv.org/abs/2402.03300
- [28]Sinha, A.; Elango, S.; and Liu, D. 2026. Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling. arXiv preprint arXiv:2601.21669.arxiv.org/abs/2601.21669
- [29]Székely, G. J.; Rizzo, M. L.; et al. 2013. Energy statistics: A class of statistics based on distances. Journal of statistical planning and inference, 143(8): 1249–1272.DOI
- [32]Yun, T.; Zhang, D.; Park, J.; and Pan, L. 2025. Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23625–23635.arxiv.org/abs/2502.11477
- [33]Zhu, X.; Cheng, D.; Zhang, D.; Li, H.; Zhang, K.; Jiang, C.; Sun, Y.; Hua, E.; Zuo, Y.; Lv, X.; Zhang, Q.; Chen, L.; Shao, F.; Xue, B.; Song, Y.; Yang, Z.; Cui, G.; Ding, N.; Gao, J.; Liu, X.; Zhou, B.; Mei, H.; and Lin, Z. 2026. FlowRL: Matching Reward Distributions for LLM Reasoning. In International Conference on Learning Representations.arxiv.org/abs/2509.15207





