betaFeedback

WorksComputer Science

Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling

Abstract

Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training. DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density. The method requires no learned density model, auxiliary objective, critic, or modification to the underlying policy-gradient objective. In a controlled multimodal navigation task, DARS-GRPO avoids the single-mode collapse exhibited by GRPO and closely recovers the target mode allocation. Applied to text-to-image prompt adaptation, DARS-GRPO consistently improves image and prompt diversity across datasets, Policy LMs, and reward settings while maintaining good semantic alignment. These results show that density-aware reward scaling provides a lightweight way to improve coverage of diverse high-reward outcomes within existing reinforcement-learning pipelines.

Introduction

Reinforcement learning is typically formulated to maximize expected return, which can drive outcome-level probability mass toward a small subset of high-reward outcomes [7]. While desirable for conventional control problems, this behavior is limiting in domains where multiple distinct outcomes are simultaneously valuable. In generative modeling, scientific discovery, and structured prediction, the objective is often not merely to identify a high-reward solution, but to learn a policy that maintains broad coverage over the high-reward outcome space.

Several approaches have been proposed to promote such diversity. Entropy-regularized RL encourages stochastic action selection, but action-level stochasticity does not in general specify a desired distribution over terminal outcomes [8]. Generative Flow Networks (GFlowNets) instead target reward-proportional sampling, providing a principled framework for generating diverse high-reward samples [2]. Applying

GFlowNet-style objectives to autoregressive pretrained models generally requires formulating generation through flow- or trajectory-based objectives rather than retaining an unchanged policy-gradient training pipeline [32]. Inverse Probability Scaling (IPS) offers another route by inversely scaling reward according to terminal-outcome visitation probability [28]. In continuous or high-dimensional outcome spaces, however, exact repeated outcomes may be rare or may not provide a meaningful notion of semantic equivalence. We introduce Density-Aware Reward Scaling (DARS), a simple framework for encouraging a policy that has reward-proportional behavior within standard policy-gradient reinforcement learning. DARS is motivated by the limitations of IPS and replaces discrete visitation frequency with an estimate of local outcome density in continuous outcome spaces. For each on-policy rollout group, outcomes that lie in densely populated regions receive proportionally reduced rewards, while high-reward outcomes in underrepresented regions receive greater relative rewards. This density scaling is applied before advantage estimation and requires no modification to the underlying policy-gradient objective. Importantly, DARS introduces no learned density model, critic, auxiliary loss, or specialized policy architecture; instead it simply modifies reward processing within the existing RL pipeline.

Our central contribution is a simple mechanism for mitigating outcome collapse and promoting coverage of diverse, high-reward regions within standard policy-gradient training. We validate DARS through controlled synthetic experiments that establish its ability to approximate reward-proportional sampling, followed by text-to-image prompt adaptation experiments demonstrating improved diversity while preserving generation quality, and kernel-width ablations that provide mechanistic evidence for its density-aware reward scaling.

Related Work

Mode collapse in RL and loss of diversity: Mode collapse is widely associated with generative models that represent only a small subset of a multimodal data distribution. An analogous failure occurs in reinforcement learning at the level of terminal outcomes: even when multiple trajectories or solutions receive similar rewards, expected-return optimization concentrates the learned policy on a narrow subset of them, reducing the diversity of generated solutions in language and image generation [11, 3]. Existing approaches mitigate this behavior through entropy regularization, KL constraints, reward shaping, or diversity-aware objectives [25, 17], yet they do not directly control the distribution of terminal outcomes. Recent work instead attributes outcome collapse to expected-return optimization itself, where an outcome’s aggregate learning signal is amplified by its current sampling probability, creating a rich-get-richer feedback loop [28]. Reward-proportional methods address this issue by sampling outcomes in proportion to their rewards: GFlowNets achieve this through specialized flow-conservation objectives and have been extended to multi-objective optimization, continuous state spaces, prompt adaptation, and language-model reasoning [1, 2, 10, 15, 33], while IPS instead rescales rewards using inverse visitation probabilities estimated from empirical outcome frequencies [28]. However, empirical visitation frequencies become ineffective in continuous outcome spaces, where exact outcome repetitions are exceedingly rare. Concurrent work [3], also adjusts GRPO rewards using within-group semantic redundancy for mathematical reasoning in LLMs. In contrast, we study density-aware reward scaling as a mechanism for reward-proportional sampling in continuous outcome spaces and directly evaluate recovery of the target reward distribution.

Prompt Adaptation and Image Generation: Loss of output diversity has become increasingly visible in reinforcement learning from human feedback and other forms of large-language-model post-training [19, 11, 3]. Reward optimization can favor a narrow set of response styles or semantic patterns even when many responses are valid. In practice, this behavior is often moderated with KL regularization, reward mixing, early stopping, or stochastic decoding. Prompt-optimization methods learn rewrites that improve text-to-image quality or alignment [9], while PAG learns diverse effective prompts with a GFlowNet formulation over prompt-generation trajectories [32]. CADS acts at diffusion inference time by annealing the conditioning signal during sampling.

Density-Aware Reward Scaling (DARS)

IPS shows that reward-proportional sampling can be achieved by inversely scaling rewards according to outcome probabilities under the current policy [28], but estimating these probabilities through exact visitation frequencies does not extend naturally to continuous outcome spaces, where repeated outcomes are unlikely. DARS instead represents outcomes in a continuous feature space where distances capture meaningful similarity between outcomes. This representation allows visitation to be defined over local regions rather than exact outcome identities. DARS estimates the local density of each sampled outcome within its on-policy rollout group and inversely scales its reward by this density, reducing reinforcement of densely sampled regions

while relatively favoring underrepresented high-reward regions.

Density

Let O\mathcal{O} denote the outcome space and let ϕ:O→Rn\phi:\mathcal{O} \to\mathbb{R}^{n} be a representation map that embeds outcomes into a feature space. For an outcome oi∈Oo_i \in\mathcal{O}, let yi=ϕ(oi)∈Rny_i = \phi(o_i) \in\mathbb{R}^{n} denote its representation. Given an on-policy rollout group containing GG such yiy_i and their corresponding rewards Ri≥0R_i \ge0, DARS computes the relative local density of each outcome only with respect to the outcomes in the same rollout group. For a nonnegative kernel KhK_h, this estimate is

ρi=∑j=1GKh(yi,yj),(1)\rho_i = \sum_{j=1}^{G} K_h(y_i, y_j), \tag*{(1)}

where hh controls how strongly nearby outcomes contribute to the local density. One choice is the Gaussian kernel,

Kh(yi,yj)=exp⁡(−∥yi−yj∥222h2),(2)K_h(y_i, y_j) = \exp\left(-\frac{\lVert y_i-y_j\rVert_2^2}{2h^2}\right), \tag*{(2)}

which assigns greater weight to nearby outcomes and smoothly decreases the contribution of outcomes as their distance increases. The kernel width hh determines the scale of the resulting visitation neighborhoods. As the Gaussian kernel includes each outcome itself with Kh(yi,yi)=1K_h(y_i,y_i)=1, the local density estimate satisfies ρi≥1\rho_i \ge1.

DARS then applies inverse local-density reward scaling to down-weight frequently visited outcomes and encourage exploration of less populated regions of the outcome space, yielding the scaled reward R~i\widetilde{R}_i.

R~i=Riρi,(3)\widetilde{R}_i = \frac{R_i}{\rho_i}, \tag*{(3)}

Integration with GRPO

DARS modifies only the reward-processing stage of the GRPO pipeline. GRPO uses a clipped policy-ratio objective, and estimates advantages from groups of sampled outputs [26]. It replaces each task reward RiR_i with the density-scaled reward R~i\widetilde{R}_i from Equation (3), after which the group-relative advantage is computed as

Ai=R~i−R‾sR~,(4)A_i = \frac{\widetilde{R}_i-\overline{R}}{s_{\widetilde{R}}}, \tag*{(4)}

where R‾\overline{R} and sR~s_{\widetilde{R}} are the within-group mean and standard deviation of the scaled rewards. The resulting advantages are passed to the standard GRPO objective without further modification. For each conditioning input, DARS-GRPO samples an on-policy rollout group and evaluates its task rewards. It then computes the within-group local-density scores, scales and normalizes the rewards to obtain group-relative advantages, and applies the standard GRPO policy update without further modification. Thus, DARS requires no change to the GRPO policy ratio, clipped surrogate objective, policy architecture, or optimization procedure; its sole intervention is density-aware reward scaling before advantage normalization.

Behavior

At either kernel width extremes, as h→0h \to0, ρi≈1\rho_i \approx1 and the rewards remain unchanged; as h→∞h \to\infty, ρi≈G\rho_i \approx G and group-relative advantage normalization cancels the shared reward scale. Intermediate bandwidths instead produce non-uniform reward scaling across the given rollout group, thereby modifying the resulting policy-gradient signal.

For dd-dimensional outcome representations, pairwise kernel evaluation costs O(G2d)O(G^2d) time and O(G2)O(G^2) memory. DARS introduces no learned density model, critic, or auxiliary optimization objective.

Navigation Task: Controlled Distribution-Recovery Experiment

This experiment tests whether DARS can make a policy-gradient RL method act as an approximate reward-proportional sampler in a continuous outcome space. The primary comparison is against GRPO: both methods use the same policy, environment, and terminal reward, but only DARS corrects for local outcome collapse. GFlowNet provides a reward-proportional reference with a different training objective. The experiment is a controlled empirical test of whether DARS approaches the normalized reward distribution.

Environment and Target: We evaluate how closely a policy approaches a known normalized reward density on the unit square X=[0,1]2\mathcal{X} = [0, 1]^2. Each episode starts at s0=(0,0)s_0 = (0, 0). At state sts_t, the policy either terminates or takes a nonnegative displacement satisfying

0≤at,i≤min⁡(δ,1−st,i),i∈{1,2},0 \le a_{t,i} \le\min(\delta, 1 - s_{t,i}), \qquad i \in\{1, 2\},

with deterministic transition st+1=st+ats_{t+1} = s_t + a_t. Trajectories therefore move monotonically up and to the right. The step bound δ\delta makes distant terminal states require several decisions to reach for the policy. The policy may stop at any visited state except the origin, and it is forced to stop at the boundary or when the rollout horizon is reached. Only the terminal state receives rewards.

The movement policy uses polar coordinates. A neural network predicts a Bernoulli stopping probability and separate mixtures of Beta distributions for a normalized angle and radius. The angle is mapped to θt∈[0,π/2]\theta_t \in[0, \pi/2], and the radius is rescaled by both δ\delta and the distance to the boundary. The resulting action is

at=(rtcos⁡θt,rtsin⁡θt).a_t = (r_t \cos\theta_t, r_t \sin\theta_t).

This guarantees valid first-quadrant movements and lets the policy jointly learn where to move, how far to move, and when to terminate. Since the radial parameterization bounds the Euclidean step length, it imposes the slightly stricter constraint ∥at∥2≤δ\lVert a_t \rVert_2 \le\delta.

The reward is an equally weighted mixture of four isotropic Gaussians, as shown in Figure 1. Each component has standard deviation σ=0.075\sigma= 0.075.

Figure 1. Terminal distributions on the navigation task.

Baselines and Evaluation Metrics

We compare GRPO and DARS-GRPO, using GFlowNet as a non-RL reward-proportional reference. We evaluate the terminal distribution produced by each trained policy using 10610^6 sampled terminal states per trained policy.

Target samples are drawn from the Gaussian mixture and rejected when they fall outside the unit square. We use complementary metrics because no single score captures all forms of distributional mismatch in a multimodal setting.

Mean raw reward is estimated as the empirical average of r(x)r(x) over sampled terminal states. Jensen–Shannon divergence (JSD) and histogram total variation (Hist. TV) compare the empirical and target densities on a 100×100100 \times100 grid. Sliced Wasserstein-2 (SW2_2) (Kolouri et al. [13]), energy distance (Székely et al. [29]), and Sinkhorn divergence (Feydy et al. [4]) compare generated and target samples directly and measure geometric mismatch. RBF maximum mean discrepancy (MMD) (Gretton et al. [6]) is reported at several bandwidths: small bandwidths emphasize local component shape, whereas large bandwidths emphasize mode allocation and global structure. Sliced Wasserstein-2 and energy distance use ten repetitions of 20,000 samples, while Sinkhorn divergence uses 500 samples.

For mode-level evaluation, each terminal state is assigned to its nearest target center. From these assignments, we compute the mass of each mode, the minimum mode mass, and mode TV which measures deviation from uniform target mode weights. Precision is the fraction of generated samples lying inside kk-nearest-neighbor neighborhoods of target samples, while recall is the fraction of target samples covered by neighborhoods induced by generated samples [14]. Precision therefore measures off-target generation, whereas recall diagnoses missing support and mode collapse.

We measure Wasserstein error at two scales: the between-mode term evaluates mode weights and centers, while the within-mode term evaluates each mode’s mean and covariance. Their sum defines the hierarchical score,

Whier=Wbetween+Wwithin.W_{\mathrm{hier}} = W_{\mathrm{between}} + W_{\mathrm{within}}.

HDR outside mass is the fraction of generated samples outside the target’s 95% highest-density region. Since the target assigns approximately 0.05 probability mass outside this region, we report the calibration error ∣moutside−0.05∣\lvert m_{\mathrm{outside}} - 0.05\rvert, for which lower values indicate better agreement with the target.

Results and Interpretation

The primary result is the comparison with GRPO. In the reported run, GRPO concentrates on a single high-reward mode, whereas DARS-GRPO assigns substantial mass to all four modes and approaches their target weights. The DARS-GRPO point estimates are closer to the target than GRPO across the metrics reported in Table 1. GRPO’s higher mean raw reward reflects tight concentration near one of four equally high peaks, rather than agreement with the normalized reward distribution. Thus, within this controlled environment, the reported run provides evidence that density reweighting prevents the evaluated policy from concentrating on one mode, recovers the correct mode weights, and improves global geometric agreement with the known target relative to GRPO. This is the central controlled contribution of the experiment.

MetricGFlowNetGRPODARS-GRPO
Density and transport
Mean raw reward ↑3.446.964.59
JSD ↓0.0200.6490.041
Hist. TV ↓0.1490.9750.219
SW2_2 ↓0.0360.2420.015
Energy ↓0.0070.2810.002
Mode allocation and support
Mode TV ↓0.1040.7500.014
Min. mode mass ↑0.1870.0000.236
Precision ↑0.9980.9970.998
Recall ↑0.9960.0320.985
Hierarchical WW ↓0.1280.3590.073
Per-mode and local diagnostics (desired mass: 0.25)
Mode 1 mass0.3180.0000.254
Mode 2 mass0.1870.0000.252
Mode 3 mass0.2860.0000.258
Mode 4 mass0.2091.0000.236
Between-mode WW ↓0.1130.3350.046
Within-mode WW ↓0.0160.023†0.027
HDR outside-mass error ↓0.0010.0500.044
RBF MMD² by bandwidth
σ=0.020\sigma= 0.020 ↓0.0010.7030.002
σ=0.040\sigma= 0.040 ↓0.0040.8320.004
σ=0.080\sigma= 0.080 ↓0.0070.8030.006
σ=0.160\sigma= 0.160 ↓0.0110.6900.002
σ=0.320\sigma= 0.320 ↓0.0100.3720.001
Sinkhorn divergence by regularization
ϵ=0.001\epsilon= 0.001 ↓0.0080.1150.002
ϵ=0.005\epsilon= 0.005 ↓0.0080.1150.002
ϵ=0.010\epsilon= 0.010 ↓0.0080.1140.001
ϵ=0.020\epsilon= 0.020 ↓0.0080.1120.001

Table 1. Navigation Task Results (avg over 5 seeds).

DARS-GRPO closely matches, and on several key metrics surpasses, GFlowNet despite requiring only a simple reward-scaling modification within a standard policy-gradient pipeline. Both methods achieve identical precision, indicating equally accurate recovery of high-reward outcomes. DARS consistently improves metrics that capture the global reward distribution, including sliced Wasserstein, energy distance, mode total variation, minimum mode mass, hierarchical Wasserstein error, HDR outside mass, and most larger-bandwidth MMD and Sinkhorn metrics, demonstrating more faithful recovery of target mode proportions and global geometry.

GFlowNet performs slightly better on metrics emphasizing fine-scale local density, such as Jensen–Shannon divergence, histogram total variation, within-mode Wasserstein error, recall, and small-bandwidth MMD, reflecting its specialized flow-based objective. Nevertheless, these results show that a lightweight modification to conventional policy-gradient RL is sufficient to achieve reward-proportional behavior that is competitive with a substantially more sophisticated algorithmic framework.

Having established in a controlled setting that DARS approximates reward-proportional sampling, we next ask whether this property is useful in a real continuous-output RL problem where the target reward distribution is unknown.

Text-to-Image Prompt Adaptation

Motivation

Prompt adaptation for image generation refers to the setting in which a language model rewrites the input prompt to match a desired set of outcomes before image generation. Text-to-image prompt adaptation provides a natural real-world benchmark for evaluating reward-proportional reinforcement learning. Unlike the synthetic experiments in § 3, where the target reward distribution is explicitly known, the reward-proportional distribution over generated images is intractable and cannot be directly characterized. A single text prompt typically admits many semantically faithful realizations that differ in composition, viewpoint, lighting, object arrangement, or artistic style while remaining equally valid.

Since the underlying reward distribution is unavailable, directly measuring reward-proportional sampling is infeasible. Instead, we evaluate its consequences through observable proxies. If DARS successfully distributes probability mass across multiple high-reward regions, it should produce a more diverse set of images while preserving prompt alignment and image quality. We therefore measure visual diversity together with alignment and preference metrics as an indirect estimate of improved coverage of the latent reward landscape. Consequently, the objective is not merely to identify a single high-reward image, but to maintain coverage over the diverse set of high-quality outcomes consistent with the prompt.

Since input prompts are the conditioning signal in diffusion models, semantically faithful prompt rewrites help steer the generation in desired directions. Learning a distribution over such rewrites therefore provides an effective mechanism for exploring multiple high-reward image modes while keeping the underlying image generator fixed.

Following this formulation, we fine-tune a language model using reinforcement learning to rewrite input prompts. Images generated from the rewritten prompts are rewarded as defined in section 5.2. Since multiple visually distinct images can achieve comparable rewards, broader coverage of the latent reward landscape should translate to greater image diversity without sacrificing fidelity. To quantify this diversity, we measure distances in the DINOv2 representation space as a proxy for perceptual similarity ([18]).

Overview of DARS-GRPO for text-to-image prompt adaptation

Figure 2. Overview of DARS-GRPO for text-to-image prompt adaptation. A trainable language-model policy samples GG rewrites of an input prompt, and a frozen text-to-image model generates one image per rewrite. Each image receives a task reward rir_i and is mapped by a frozen image encoder into a representation space, where a Gaussian kernel computes its within-group density ρi\rho_i. DARS forms the scaled reward r~i=ri/ρi\tilde{r}_i = r_i/\rho_i, reducing the contribution of high-reward images in dense representation-space regions and increasing the relative contribution of high-reward images in sparsely represented regions. GRPO then uses these scaled rewards to update only the prompt-rewriting policy.

Reward Definitions

The reward should preserve two complementary properties: semantic fidelity to the original prompt and visual appeal. Fidelity alone does not ensure that an image is aesthetically strong, whereas optimizing aesthetics alone can favor attractive images that drift from the requested content. We therefore evaluate on two different reward functions: alignment-only reward and a combined reward consisting of aesthetic and alignment components.

Let cc denote the original prompt and yiy_i an image generated from its iith rewrite. We measure semantic alignment using the image-text cosine similarity from a CLIP model ([20]). A binary semantic gate determines whether the image is sufficiently aligned with the original prompt. Thus the alignment reward is

Rialign=I[CLIP⁡(c,yi)>ϕ],(5)R_i^{\mathrm{align}} = \mathbb{I}[\operatorname{CLIP}(c,y_i) > \phi], \tag*{(5)}

where ϕ\phi is the minimum accepted CLIP similarity. Images below this threshold receive zero reward to encourage strict adherence. This setting tests whether DARS can broaden coverage within the set of sufficiently aligned images.

To measure visual appeal, we use LAION-Aesthetics Predictor ([24]), which maps images to an aesthetic score on a 1–10 scale. Let a(⋅)a(\cdot) denote this predictor, and let yirefy_i^{\mathrm{ref}} be an image generated from the original input prompt. We define the aesthetic improvement and its multiplicative reward factor as

Δai=a(yi)−a(yiref),(6)\Delta a_i = a(y_i) - a\left(y_i^{\mathrm{ref}}\right), \tag*{(6)}
Riaes=exp⁡(Δai).(7)R_i^{\mathrm{aes}} = \exp(\Delta a_i). \tag*{(7)}

An aesthetic improvement produces a factor greater than one, no change produces a factor of one, and a decrease produces a factor between zero and one.

Thus, the combined reward is given as

Ricomb=exp⁡(Δai)CLIP⁡(c,yi)I[CLIP⁡(c,yi)>ϕ](8)R_i^{\mathrm{comb}} = \exp(\Delta a_i)\operatorname{CLIP}(c,y_i)\mathbb{I}[\operatorname{CLIP}(c,y_i) > \phi] \tag*{(8)}

Density Calculation and Reward Scaling

DARS estimates local density in the space of generated images, since our objective is to promote diversity over visual outcomes rather than textual rewrites. As discussed in § 2.1, we instantiate ϕ\phi using a frozen DINOv2 encoder, which maps each generated image ii to its unnormalized class-token (CLS) embedding yiy_i. Local densities are then estimated in this representation space.

For a group of GG images, let yi∈R768y_i \in\mathbb{R}^{768} denote the feature vector of image ii. We compute the local density score for image ii using a Gaussian kernel as

Figure 3. Comparison of prompt rewrites and their corresponding generated images for “A snowman standing in the yard” using GRPO, PAG, and DARS-GRPO. GRPO produces nearly identical rewrites and visually similar nighttime scenes, whereas PAG introduces moderate variation and DARS-GRPO generates more diverse rewrites, settings, and visual compositions.

ρi=∑j=1GKh(yi,yj)=∑j=1Gexp⁡(−∥yi−yj∥222h2).(9)\rho_i = \sum_{j=1}^{G} K_h(y_i,y_j) = \sum_{j=1}^{G} \exp\left(-\frac{\lVert y_i-y_j\rVert_2^2}{2h^2}\right). \tag*{(9)}

Following Equation (3), DARS rescales the task reward as R~i=Ri/ρi\widetilde{R}_i = R_i/\rho_i. We use this scaled reward to optimize the prompt-rewriting policy with DARS-GRPO.

Evaluation Metrics

We evaluate each prompt-rewriting policy along three complementary dimensions: image diversity, rewrite diversity, and image fidelity or preference. Image diversity is measured in DINOv2 CLS representation space using the Vendi Score {(citep)}(Friedman and Dieng 2023) of the cosine-similarity matrix of L2-normalized CLS embeddings, and the mean pairwise Euclidean distance between CLS embeddings. Rewrite diversity is measured analogously in CLIP text-embedding space using Prompt Vendi and the mean pairwise cosine distance among rewritten prompts. Finally, CLIP-align, HPSv2 {(citep)}(Wu et al. 2023), and PickScore {(citep)}(Kirstain et al. 2023) assess the semantic compatibility and human-preference-style quality of each generated image relative to the input prompt. CLIP-align is the nonnegative cosine similarity between the L2-normalized CLIP embeddings of a generated image and its original prompt. Diversity metrics are computed within each group of rewrites or images generated for the same original prompt and then averaged across the evaluation set, following within-prompt conditional evaluation. For all metrics, higher values indicate greater diversity or stronger fidelity and preference, respectively.

Experiment Details

We use a frozen Stable Diffusion v1.4 generator {(citep)}(Rombach et al. 2022) and optimize only the prompt-rewriting policy. For all baselines and reward functions, both Qwen3-0.6B(base) {(citep)}(Yang et al. 2025) and GPT-2 (124M) {(citep)}(Radford et al. 2019) are already supervised fine-tuned on a shared set of curated prompt rewrites. We then train the models on 1,024 randomly sampled MS-COCO prompts {(citep)}(Lin et al. 2014) and evaluate them on disjoint MS-COCO and Lexica prompts {(citep)}(Shen et al. 2023). DARS uses h=20h = 20 for Gaussian kernels, and the alignment reward uses a CLIP-similarity threshold of 0.2.

We compare DARS-GRPO with vanilla GRPO, PAG, and GFlowNet prompt optimization, together with Stable Diffusion v1.4 with random seed initialization and CADS {(citep)}(Sadat et al. 2024) as control baselines. All methods are evaluated using the same held-out prompts, generation budget, diffusion model, and sampling procedure. Diffusion seeds are sampled independently across methods, so image-diversity metrics also capture variation introduced by stochastic diffusion sampling while prompt-diversity metrics are unaffected by this confound.

Results and Interpretation

Across both datasets, Policy LMs, and reward settings, DARS-GRPO consistently improves upon vanilla GRPO across all three diversity metrics (Table 2) without significant loss of image fidelity. Further, among all prompt-adaptation methods, it achieves the highest DINO pairwise distances and Image Vendi scores, indicating that density-aware reward scaling broadens the learned rewrite distribution and increases coverage in DINOv2 image-representation space. The high Prompt Vendi and diversity scores further indicate that this image diversity arises from diverse prompt rewrites, while CLIP-align, HPSv2, and PickScore imply close image fidelity.

Policy LMMethodImage Vendi ↑ COCOImage Vendi ↑ LexicaCLIP-align ↑ COCOCLIP-align ↑ LexicaHPSv2 ↑ COCOHPSv2 ↑ LexicaPickScore ↑ COCOPickScore ↑ LexicaPrompt Vendi ↑ COCOPrompt Vendi ↑ LexicaPrompt diversity ↑ COCOPrompt diversity ↑ LexicaDINO pairwise distance ↑ COCODINO pairwise distance ↑ Lexica
–SD v1.48.609.520.270.270.260.2521.5320.91––––40.5841.84
–CADS12.9314.080.260.260.240.2320.8620.28––––46.2246.96
Combined reward
Qwen3-0.6BGRPO8.769.810.240.240.230.2320.9220.691.011.040.000.0140.9042.10
Qwen3-0.6BDARS-GRPO13.0414.450.240.240.230.2320.8020.371.431.670.050.0646.9247.89
Qwen3-0.6BPAG8.2511.510.260.260.250.2421.3420.691.261.590.040.0740.2044.64
Qwen3-0.6BGFN8.7411.690.260.260.250.2421.3120.691.221.590.040.0740.5044.71
GPT-2GRPO9.6010.680.240.240.230.2320.8720.431.091.180.010.0341.8542.87
GPT-2DARS-GRPO12.7214.670.230.230.230.2320.6820.261.631.780.080.1047.7548.60
GPT-2PAG9.2214.030.260.240.250.2321.1620.351.612.140.090.1341.3646.43
GPT-2GFN9.0213.670.250.240.240.2221.1220.261.151.690.020.0841.0346.67
Alignment-only reward
Qwen3-0.6BGRPO9.4710.940.260.260.250.2421.2420.551.591.640.070.0841.6843.52
Qwen3-0.6BDARS-GRPO20.1320.030.220.210.220.2220.3419.812.432.400.160.1656.2355.76
Qwen3-0.6BPAG9.5313.880.250.240.250.2421.2220.481.792.330.110.1541.9946.86
Qwen3-0.6BGFN8.9312.130.260.260.250.2421.2720.601.301.620.050.0840.9944.97
GPT-2GRPO8.7610.220.260.260.260.2421.4220.661.031.110.010.0240.9342.52
GPT-2DARS-GRPO22.8722.640.230.220.230.2220.5819.892.152.290.140.1555.5755.00
GPT-2PAG8.5113.150.250.240.250.2321.2320.381.201.660.030.0740.5545.78
GPT-2GFN8.6014.080.260.240.240.2221.1920.331.171.790.030.0840.5446.03

Table 2. Prompt Adaptation for Text-to-Image Generation results on MS-COCO and Lexica (avg over 5 seeds).

Changing the reward function affects performance across all metrics, showing that DARS responds to the reward landscape rather than producing diversity gains randomly. Consistent with our hypothesis, the binary gated alignment-only reward yields broader sampling from uniformly scored reward zones, whereas the combined reward forces the policy to learn a more dynamic reward landscape characterized by aesthetically-driven peaks, thus, sampling from a more varied distribution as reflected by the higher average Image Vendi scores.

Having demonstrated the effectiveness of DARS on a real-world RL problem with an intractable reward-proportional distribution, we next study the effect of the kernel width used for local density estimation on the learned outcome distribution.

Ablation Study

As discussed in 3.3, kernel width hh affects reward-scaling in DARS and hence needs to be calibrated for accurate density estimation. For the experiments in Section 5, we demonstrate the direct effects of varying kernel widths on image-diversity produced by DARS (Table 3). Diversity metrics consistently peak at intermediate kernel widths and degrade when the bandwidth is either too small or too large, while image quality remains largely unchanged. This behavior reflects how DARS estimates local outcome density. With a very small kernel width, nearly every generated image appears isolated, causing density estimates to be dominated by self-similarity and resulting in almost uniform reward scaling. Conversely, an excessively large kernel width treats distant images as belonging to the same neighborhood, oversmoothing density estimates and diminishing the relative reward differences that encourage exploration. Intermediate kernel widths instead resolve meaningful local neighborhoods allowing for effective density estimation in DARS.

WidthImage Vendi ↑CLIP-align ↑Prompt Diversity ↑Prompt Vendi ↑
109.990.260.121.99
1514.120.250.213.34
2018.010.240.172.74
2519.770.230.223.47
3012.140.250.112.09

Table 3. Kernel-width ablation results (avg over 5 seeds)

Limitations

DARS relies on a meaningful representation space and kernel bandwidth for estimating local outcome density. Its effectiveness may therefore degrade when embedding distances poorly reflect semantic similarity or when no single bandwidth captures the relevant neighborhood structure. The pairwise density computation also scales quadratically with rollout-group size, potentially limiting its use with large groups. Our experiments focus on a controlled navigation task and text-to-image prompt adaptation with fixed image representations and generators. Broader evaluation across RL domains, policy architectures, and learned representations is needed. Finally, our results establish empirical approximation to reward-proportional sampling rather than a general convergence guarantee.

Conclusion

We introduced Density-Aware Reward Scaling (DARS), a lightweight mechanism for promoting diverse, high-reward outcomes within standard policy-gradient training. By inversely scaling rewards using local outcome density, DARS counteracts repeated reinforcement of densely sampled regions without requiring a critic, auxiliary objective, learned density model, or modified policy-gradient loss. In a controlled multimodal navigation task, DARS-GRPO prevents the mode collapse observed with GRPO and closely recovers target mode allocation. In text-to-image prompt adaptation, it consistently increases prompt and image diversity across datasets, Policy LMs, and reward settings while retaining semantic fidelity. These results position density-aware reward scaling as a simple yet effective approach to improving outcome coverage.

Mathematical Foundations for DARS

Inverse policy-density scaling yields a reward-proportional policy

Let the continuous outcome space be Y⊆Rd\mathcal{Y} \subseteq\mathbb{R}^{d}. At training time tt, let the policy induce a probability density pt(y)p_t(y) on Y\mathcal{Y}. Let the reward be a measurable function r(y)≥0r(y) \ge0 satisfying

0<R:=∫Yr(y) dy<∞,(10)0 < \mathcal{R} := \int_{\mathcal{Y}} r(y)\,\mathrm{d}y < \infty, \tag*{(10)}

and assume pt(y)>0p_t(y) > 0 wherever r(y)>0r(y) > 0. The desired reward-proportional density is

p⋆(y)=r(y)R.(11)p^{\star}(y) = \frac{r(y)}{\mathcal{R}}. \tag*{(11)}

We prove the continuous analogue of inverse probability scaling. Parameterize an unrestricted policy density by a logit function ff,

pf(y)=ef(y)∫Yef(u) du.(12)p_f(y) = \frac{e^{f(y)}}{\int_{\mathcal{Y}} e^{f(u)}\,\mathrm{d}u}. \tag*{(12)}

At iteration tt, freeze the current policy density in the reward denominator and use

r~t(y)=r(y)pt(y).(13)\widetilde{r}_t(y) = \frac{r(y)}{p_t(y)}. \tag*{(13)}

For the population objective

Jt(f)=∫Ypf(y)r~t(y) dy,(14)J_t(f) = \int_{\mathcal{Y}} p_f(y)\widetilde{r}_t(y)\,\mathrm{d}y, \tag*{(14)}

the following result holds.

Theorem 1 At the current policy pt=pftp_t = p_{f_t}, the functional policy-gradient direction is

δJtδf(y)∣f=ft=r(y)−Rpt(y)=R(p⋆(y)−pt(y)).(15)\left.\frac{\delta J_t}{\delta f}(y)\right|_{f=f_t} = r(y) - \mathcal{R}p_t(y) = \mathcal{R}\left(p^{\star}(y) - p_t(y)\right). \tag*{(15)}

Hence the reward-proportional density p⋆(y)=r(y)/Rp^{\star}(y) = r(y)/\mathcal{R} is the unique stationary probability density, interpreted in the closure of the exponential-family parameterization when rr vanishes on part of the outcome space. Under the corresponding population gradient flow, every convergent density limit is p⋆p^{\star}.

Proof 1 For any perturbation vv, the normalized exponential density obeys

ddspft+sv(y)∣s=0=pt(y)(v(y)−Ept[v]).(16)\left.\frac{\mathrm{d}}{\mathrm{d}s}p_{f_t+sv}(y)\right|_{s=0} = p_t(y)\left(v(y)-\mathbb{E}_{p_t}[v]\right). \tag*{(16)}

Because the denominator in Equation (13) is stop-gradient,

ddsJt(ft+sv)∣s=0=∫Yr(y)pt(y)pt(y)(17)\left.\frac{\mathrm{d}}{\mathrm{d}s}J_t(f_t+sv)\right|_{s=0} = \int_{\mathcal{Y}} \frac{r(y)}{p_t(y)}p_t(y) \tag*{(17)}
×(v(y)−Ept[v]) dy(18)\qquad\times\left(v(y)-\mathbb{E}_{p_t}[v]\right)\,\mathrm{d}y \tag*{(18)}
=∫Yr(y)v(y) dy(19)= \int_{\mathcal{Y}} r(y)v(y)\,\mathrm{d}y \tag*{(19)}
−R∫Ypt(y)v(y) dy(20)\qquad- \mathcal{R}\int_{\mathcal{Y}}p_t(y)v(y)\,\mathrm{d}y \tag*{(20)}
=∫Y(r(y)−Rpt(y))v(y) dy.(21)= \int_{\mathcal{Y}}\left(r(y)-\mathcal{R}p_t(y)\right)v(y)\,\mathrm{d}y. \tag*{(21)}

Since this identity holds for every perturbation vv, Equation (15) follows. A stationary density must satisfy

r(y)−Rpt(y)=0(22)r(y)-\mathcal{R}p_t(y)=0 \tag*{(22)}

almost everywhere, and therefore

pt(y)=r(y)R=p⋆(y).(23)p_t(y)=\frac{r(y)}{\mathcal{R}}=p^{\star}(y). \tag*{(23)}

The direction is not merely stationary at p⋆p^{\star}; it points toward it. Consider the population gradient flow

∂tft(y)=R(p⋆(y)−pt(y))(24)\partial_t f_t(y)=\mathcal{R}\left(p^{\star}(y)-p_t(y)\right) \tag*{(24)}

and the functional

Φ(f)=log⁡∫Yef(y) dy−∫Yp⋆(y)f(y) dy.(25)\Phi(f)=\log\int_{\mathcal{Y}}e^{f(y)}\,\mathrm{d}y-\int_{\mathcal{Y}}p^{\star}(y)f(y)\,\mathrm{d}y. \tag*{(25)}

Its functional derivative is δΦ/δf=pf−p⋆\delta\Phi/\delta f=p_f-p^{\star}. Therefore, along the gradient flow,

ddtΦ(ft)=∫Y(pt(y)−p⋆(y))∂tft(y) dy(26)\frac{\mathrm{d}}{\mathrm{d}t}\Phi(f_t) = \int_{\mathcal{Y}}\left(p_t(y)-p^{\star}(y)\right)\partial_t f_t(y)\,\mathrm{d}y \tag*{(26)}
=−R∫Y(pt(y)−p⋆(y))2 dy≤0.(27)= -\mathcal{R}\int_{\mathcal{Y}}\left(p_t(y)-p^{\star}(y)\right)^2\,\mathrm{d}y \le0. \tag*{(27)}

Equality holds only when pt=p⋆p_t=p^{\star} almost everywhere. Thus the only stationary density, and hence any convergent population-gradient-flow density limit, is the reward-proportional density in Equation (11).

The cancellation underlying the result can also be seen directly. A small region dy\mathrm{d}y around yy is sampled with probability approximately pt(y)dyp_t(y)\mathrm{d}y. Its per-sample scaled reward is r(y)/pt(y)r(y)/p_t(y). Thus its aggregate expected learning weight is proportional to

pt(y) dy  r(y)pt(y)=r(y) dy.(28)p_t(y)\,\mathrm{d}y\;\frac{r(y)}{p_t(y)} = r(y)\,\mathrm{d}y. \tag*{(28)}

The current policy density cancels, leaving learning mass proportional only to reward mass.

The Gaussian kernel density score is proportional to the policy density

Draw a rollout group of GG outcomes independently from the current policy,

Y1,…,YG∼iidpt.(29)Y_1,\ldots,Y_G \mathrel{\overset{\mathrm{iid}}{\sim}} p_t. \tag*{(29)}

DARS uses the unnormalized Gaussian kernel,

Kh(y,z)=exp⁡(−∥y−z∥22h2).(30)K_h(y,z)=\exp\left(-\frac{\lVert y-z\rVert^2}{2h^2}\right). \tag*{(30)}

Instead of separating any sample from the rest of the group, define the density-score field produced by the entire rollout group,

ρG(y)=∑j=1GKh(y,Yj).(31)\rho^G(y)=\sum_{j=1}^{G}K_h(y,Y_j). \tag*{(31)}

The score assigned to a sampled outcome is simply

ρi=ρG(Yi).(32)\rho_i = \rho_G(Y_i). \tag*{(32)}

Thus the same group of all GG policy samples is used throughout.

Define

Ch=(2π)d/2hd,ϕh(z)=1Chexp⁡(−∥z∥22h2).(33)C_h = (2\pi)^{d/2}h^d,\qquad\phi_h(z)=\frac{1}{C_h}\exp\left(-\frac{\lVert z\rVert^2}{2h^2}\right). \tag*{(33)}

Then ϕh\phi_h is the N(0,h2Id)\mathcal{N}(0,h^2 I_d) density and

p^G,h(y):=ρG(y)GCh=1G∑j=1Gϕh(y−Yj).(34)\widehat{p}_{G,h}(y) := \frac{\rho_G(y)}{G C_h} = \frac{1}{G}\sum_{j=1}^{G}\phi_h(y-Y_j). \tag*{(34)}

Equation (34) is exactly the Gaussian kernel-density estimator formed from the full rollout group.

Theorem 2 (Gaussian density approximation). For every fixed yy,

E[p^G,h(y)]=(pt∗ϕh)(y),(35)\mathbb{E}[\widehat{p}_{G,h}(y)] = (p_t * \phi_h)(y), \tag*{(35)}

where

(pt∗ϕh)(y)=∫Ypt(u)ϕh(y−u) du.(36)(p_t * \phi_h)(y) = \int_{\mathcal{Y}} p_t(u)\phi_h(y-u)\,du. \tag*{(36)}

Assume that, on the region of interest, ptp_t is twice continuously differentiable,

sup⁡y∥∇2pt(y)∥op≤M,(37)\sup_y \lVert\nabla^2 p_t(y)\rVert_{\mathrm{op}} \le M, \tag*{(37)}

and pt(y)≤Pp_t(y) \le P. If the region has a boundary, assume either that ptp_t admits a C2C^2 extension to a neighborhood of that region or that the statement is restricted to interior points at the bandwidth scale. Then

∣E[p^G,h(y)]−pt(y)∣≤dM2h2,(38)\left|\mathbb{E}[\widehat{p}_{G,h}(y)]-p_t(y)\right| \le\frac{dM}{2}h^2, \tag*{(38)}

and

Var⁡(p^G,h(y))≤P2dπd/2Ghd.(39)\operatorname{Var}(\widehat{p}_{G,h}(y)) \le\frac{P}{2^d\pi^{d/2}G h^d}. \tag*{(39)}

Consequently, for a fixed query point yy and any δ∈(0,1)\delta\in(0,1), with probability at least 1−δ1-\delta,

∣p^G,h(y)−pt(y)∣≤dM2h2+P2dπd/2Ghdδ.(40)\left|\widehat{p}_{G,h}(y)-p_t(y)\right| \le\frac{dM}{2}h^2+\sqrt{\frac{P}{2^d\pi^{d/2}G h^d\delta}}. \tag*{(40)}

Proof 2 Using the full group in Equation (34) and linearity of expectation,

E[p^G,h(y)]=E[1G∑j=1Gϕh(y−Yj)](41)\mathbb{E}[\widehat{p}_{G,h}(y)] = \mathbb{E}\left[\frac{1}{G}\sum_{j=1}^{G}\phi_h(y-Y_j)\right] \tag*{(41)}
=1G∑j=1GE[ϕh(y−Yj)](42)= \frac{1}{G}\sum_{j=1}^{G}\mathbb{E}[\phi_h(y-Y_j)] \tag*{(42)}
=∫Yϕh(y−u)pt(u) du(43)= \int_{\mathcal{Y}}\phi_h(y-u)p_t(u)\,du \tag*{(43)}
=(pt∗ϕh)(y).(44)= (p_t * \phi_h)(y). \tag*{(44)}

No element of the rollout group is separated from the others. Let Z∼N(0,h2Id)Z\sim\mathcal{N}(0,h^2I_d). Then, away from boundary effects as specified above,

(pt∗ϕh)(y)=E[pt(y−Z)].(45)(p_t * \phi_h)(y)=\mathbb{E}[p_t(y-Z)]. \tag*{(45)}

A second-order Taylor expansion gives

pt(y−Z)=pt(y)−∇pt(y)⊤Z+12Z⊤∇2pt(ξ)Z(46)p_t(y-Z)=p_t(y)-\nabla p_t(y)^\top Z+\frac{1}{2}Z^\top\nabla^2p_t(\xi)Z \tag*{(46)}

for a point ξ\xi on the segment between yy and y−Zy-Z. Since E[Z]=0\mathbb{E}[Z]=0 and

E∥Z∥2=dh2,(47)\mathbb{E}\lVert Z\rVert^2=dh^2, \tag*{(47)}

we obtain

∣(pt∗ϕh)(y)−pt(y)∣≤M2E∥Z∥2=dM2h2,(48)\left|(p_t * \phi_h)(y)-p_t(y)\right|\le\frac{M}{2}\mathbb{E}\lVert Z\rVert^2=\frac{dM}{2}h^2, \tag*{(48)}

which proves Equation (38).

For fixed yy, the GG terms in Equation (34) are independent and identically distributed. Hence

Var⁡(p^G,h(y))=1GVar⁡(ϕh(y−Y))(49)\operatorname{Var}(\widehat{p}_{G,h}(y))=\frac{1}{G}\operatorname{Var}(\phi_h(y-Y)) \tag*{(49)}
≤1GE[ϕh(y−Y)2](50)\le\frac{1}{G}\mathbb{E}[\phi_h(y-Y)^2] \tag*{(50)}
≤PG∫Rdϕh(z)2 dz.(51)\le\frac{P}{G}\int_{\mathbb{R}^d}\phi_h(z)^2\,dz. \tag*{(51)}

For the Gaussian kernel,

∫Rdϕh(z)2 dz=12dπd/2hd,(52)\int_{\mathbb{R}^d}\phi_h(z)^2\,dz=\frac{1}{2^d\pi^{d/2}h^d}, \tag*{(52)}

which proves Equation (39). Chebyshev’s inequality together with Equation (38) gives Equation (40).

The preceding theorem shows directly how the approximation depends on bandwidth and group size:

p^G,h(y)=pt(y)+O(h2)+Op((Ghd)−1/2).(53)\widehat{p}_{G,h}(y)=p_t(y)+O(h^2)+O_p\left((G h^d)^{-1/2}\right). \tag*{(53)}

The O(h2)O(h^2) term is the Gaussian smoothing bias, while the stochastic error decreases as the effective number of samples in an hh-scale neighborhood, GhdG h^d, increases.

Equation (40) is stated for a fixed query point because that is the elementary finite-sample calculation. To evaluate the same full-group estimator at the sampled outcomes YiY_i without separating any sample, use the standard uniform-consistency property of Gaussian kernel-density estimators. On a compact region D\mathcal{D} satisfying the regularity assumptions above,

sup⁡y∈D∣p^G,h(y)−pt(y)∣=O(h2)+Op(log⁡GGhd),(54)\sup_{y\in\mathcal{D}}\left|\widehat{p}_{G,h}(y)-p_t(y)\right|=O(h^2)+O_p\left(\sqrt{\frac{\log G}{G h^d}}\right), \tag*{(54)}

provided h→0h\to0 and Ghd/log⁡G→∞G h^d/\log G\to\infty. Because Equation (54) holds simultaneously for every y∈Dy\in\mathcal{D}, it also holds when yy is any of the observed outcomes Yi∈DY_i\in\mathcal{D}. Therefore,

ρiGCh=pt(Yi)+O(h2)+Op(log⁡GGhd).(55)\frac{\rho_i}{G C_h}=p_t(Y_i)+O(h^2)+O_p\left(\sqrt{\frac{\log G}{G h^d}}\right). \tag*{(55)}

In particular, on any region where pt(y)≥m>0p_t(y) \ge m > 0, if hh and GG are chosen so that the right-hand error in Equation (55) is small relative to mm, then

ρi=GChpt(Yi)(1+ϵi),∣ϵi∣≪1.(56)\rho_i = G C_h p_t(Y_i)(1+\epsilon_i), \qquad|\epsilon_i| \ll1. \tag*{(56)}

Thus the DARS Gaussian score is approximately the policy probability density multiplied by the same global factor GChG C_h for all outcomes in the region of interest.

The bandwidth tradeoff is now explicit. The bandwidth should be small enough that

pt∗ϕh≈pt,(57)p_t * \phi_h \approx p_t, \tag*{(57)}

but large enough that

Ghd(58)G h^d \tag*{(58)}

is not too small. In the standard regime

h→0,Ghdlog⁡G→∞,(59)h \to0, \qquad\frac{G h^d}{\log G} \to\infty, \tag*{(59)}

the approximation error vanishes uniformly on the region of interest.

Finally, Equation (56) implies

Riρi=1GChRipt(Yi)11+ϵi.(60)\frac{R_i}{\rho_i} = \frac{1}{G C_h}\frac{R_i}{p_t(Y_i)}\frac{1}{1+\epsilon_i}. \tag*{(60)}

Thus, when the Gaussian kernel-density estimate is accurate, DARS differs from exact inverse policy-density scaling only by a common positive factor and a small multiplicative estimation error. Section A.1 then gives the corresponding reward-proportional population target.

Navigation Task

Figure 4 visualizes the movement fields induced by the trained GRPO and DARS-GRPO policies. At each state s=(x,y)∈[0,1]2s=(x,y)\in[0,1]^2, actions are sampled repeatedly from the policy and used to estimate the mean one-step displacement E[st+1−st∣st=s]\mathbb{E}[s_{t+1}-s_t\mid s_t=s]. The streamlines indicate its direction, their color indicates its magnitude, and the gray contours show the normalized reward landscape.

Figure 4. Policy-induced movement fields for GRPO (left) and DARS-GRPO (right) on the four-mode navigation task. Streamlines show the estimated mean one-step displacement, color indicates its magnitude, and gray contours show the normalized reward landscape.

Terminal actions are treated as zero displacement. Consequently, a small field magnitude may result from small proposed movements, frequent termination, or cancellation among sampled actions. The DARS-GRPO optimised the policy such that the coverage over all the modes could be optimised in proportion to the reward. The visualization thus shows how density-aware reward scaling changes the learned navigation dynamics. It should not be interpreted as a gradient or parameter-update field.

Text-to-Image Prompt Adaptation

LM model choice: GPT-2 vs Qwen3-0.6B Figure 5–Figure 8 compare GPT-2 and Qwen3-0.6B as policy LMs under the combined aesthetic and alignment reward. For each original prompt, GRPO generations are shown on the left and DARS generations on the right while keeping the diffusion model, sampling budget, and evaluation protocol fixed. This isolates the effect of the language-model used from that of the reinforcement learning objective.

Policy LM comparison grids for GPT-2 and Qwen3-0.6B

Figure 5. Policy LM comparison for “A snowman standing in the yard” (top) and “A bicycle leaning against a brick wall” (bottom). Each embedded grid compares GRPO with DARS for the indicated language-model.

Policy LM comparison for “A lantern hanging outside a cottage” and “A fox standing near a forest path”

Figure 8. Policy LM comparison for “A lantern hanging outside a cottage” (top) and “A fox standing near a forest path” (bottom). Across both policy LMs, DARS broadens the within-prompt visual distribution while maintaining the prompt’s central subject.

Across both policy LMs, DARS consistently expands the diversity of generated images while preserving the semantic content of the original prompt. Compared with GRPO, DARS produces greater variation in scene layout, viewpoint, lighting, object appearance, and background context without substantially altering the primary subject. Although GPT-2 tends to make larger structural changes and Qwen3-0.6B generally preserves a more stable object identity, both exhibit the same qualitative benefit from density-aware reward scaling. These examples illustrate that the improvements obtained by DARS are not tied to a particular language model and generalize across different policy LMs.

Reward choice: Combined vs Alignment-only Figure 9–Figure 12 compare the two reward functions using the same Qwen3-0.6B policy LM. The combined reward combines semantic fidelity with improvement under the frozen aesthetic scorer, whereas the alignment-only reward applies only the minimum CLIP-alignment constraint. Each embedded grid compares GRPO and DARS under one reward, allowing the effect of reward design to be separated from the effect of density scaling.

Reward comparison for “A snowman standing in the yard” and “A bicycle leaning against a brick wall”

Figure 9. Reward comparison for “A snowman standing in the yard” (top) and “A bicycle leaning against a brick wall” (bottom), using Qwen3-0.6B.

Reward comparison for “A lantern hanging outside a cottage” and “A fox standing near a forest path”

Figure 12. Reward comparison for “A lantern hanging outside a cottage” (top) and “A fox standing near a forest path” (bottom).

The alignment-only objective permits DARS to explore a substantially broader visual region since it is free of aesthetic bounds that affect the combined reward. Across the examples, it produces larger changes in setting, illumination, camera distance, and composition. This is particularly visible for the bench, cat, scarecrow, and fox prompts in Figure 10, Figure 11, and Figure 12 where the generated scenes include a wider range of architectural and natural backgrounds. The quantitative results show the same pattern with higher Image Vendi Scores and DINO pairwise distance for alignment-only rewards.

Reward comparison for “A wooden bench under a tree” and “A cat sleeping on a windowsill”

Figure 10. Reward comparison for “A wooden bench under a tree” (top) and “A cat sleeping on a windowsill” (bottom).

Reward comparisons for “A scarecrow in the middle of a farm” and “A castle on top of a hill”

Figure 11. Reward comparison for “A scarecrow in the middle of a farm” (top) and “A castle on top of a hill” (bottom).

Additional qualitative comparisons Figure 13–Figure 15 provide additional examples comparing GRPO, PAG, GFN, and DARS-GRPO. Across these prompts, GRPO reuses a narrow rewriting pattern, whereas DARS produces more varied scene contexts consequently leading to more diverse images.

Figure 13. Qualitative comparison for “A bicycle leaning against a brick wall.” GRPO repeatedly generates a similar nighttime composition, while DARS-GRPO varies lighting, seasonal context, and surrounding details while preserving the bicycle and wall.

Figure 15. Qualitative comparison for “A cat sleeping on a windowsill.” All methods preserve the primary subject, while DARS-GRPO introduces broader variation in background scenery, lighting, and framing.

Policy LM comparison for “A wooden bench under a tree” and “A cat sleeping on a windowsill”

Figure 6. Policy LM comparison for “A wooden bench under a tree” (top) and “A cat sleeping on a windowsill” (bottom). DARS expands the visual contexts while preserving the object and scene requested by each prompt.

Policy LM comparison for “A scarecrow in the middle of a farm” and “A castle on top of a hill”

Figure 7. Policy LM comparison for “A scarecrow in the middle of a farm” (top) and “A castle on top of a hill” (bottom). The DARS samples show greater variation in character appearance, architecture, framing, and illumination than the corresponding GRPO samples.

Figure 14. Qualitative comparison for “A wooden bench under a tree.” DARS-GRPO explores more varied locations, tree appearances, and visual contexts than GRPO, while PAG and GFN produce more limited changes around the same central composition.

Implementation Details

Experimental Configuration Unless otherwise specified, all experiments use the same training, generation, and evaluation settings. Table 4 summarizes the policy optimization hyperparameters shared by DARS and the GRPO baseline, while Table 5 lists the hyperparameters specific to density-aware reward scaling. The generation pipeline, reward formulation, and datasets are held fixed across experiments unless explicitly varied in an ablation study.

ParameterValue
Policy LMsSFT Qwen3-0.6B; SFT GPT-2
Training temperature1.5
Evaluation temperature1.0
max_new_tokens64
Learning rate1×10−51 \times10^{-5}
Weight decay0.1
Gradient clipping1.0
PPO clipping ϵ\epsilon0.25
KL coefficient1×10−91 \times10^{-9}
Policy updates per rollout1
Training batch size2 prompts

Table 4. Policy optimization hyperparameters (DARS and GRPO).

ParameterValue
Density embedding spacedinov2_image
Density kernelGaussian
Kernel width (hh)20.0

Table 5. DARS-specific hyperparameters.

DARS and GRPO are trained under identical optimization settings to ensure a fair comparison. Both methods use the same policy LM, optimizer hyperparameters, decoding configuration, and rollout schedule. They differ only in the reward scaling mechanism. Unless otherwise stated, these hyperparameters are used for all reported experiments.

Table 5 lists the hyperparameters specific to DARS. Density is estimated in the DINOv2 image embedding space using a Gaussian kernel with bandwidth h=20h = 20. To study the effect of density estimation, all kernel-width ablations retain the same training configuration while varying only the kernel bandwidth.

Kernel-width ablations use the same experimental setup while varying only the Gaussian kernel width,

h∈{10,15,20,25,30},h \in\{10, 15, 20, 25, 30\},

with all other hyperparameters kept fixed.

Computing Environment All experiments were conducted on a server equipped with four NVIDIA A100-SXM4 GPUs using the software environment summarized in Table 6. Mixed-precision training uses bf16, while Stable Diffusion inference is performed in fp16 to improve computational efficiency without affecting evaluation.

ComponentValue
Operating SystemUbuntu 22.04.4 LTS (Linux 5.15.0-94-generic)
GPUs4×4 \times NVIDIA A100-SXM4-40GB
Host RAM∼\sim503 GiB
Python3.13.9
PyTorch2.12.1+cu130
Key Librariestransformers 5.13.0, diffusers 0.39.0, accelerate 1.14.0, peft 0.19.1
Mixed Precisionbf16 (run.mixed_precision)
Diffusion Dtypefp16

Table 6. Hardware and software environment used for training.

Training Configuration Table 4 and Table 5 summarize the optimization hyperparameters and DARS-specific settings used during training. Unless explicitly stated otherwise, all experiments employ these default configurations. DARS and GRPO differ only in the computation of policy rewards, while sharing identical optimization settings.

Generation Configuration All generated images are produced using the Stable Diffusion v1.4 pipeline with identical sampling settings across methods. Each rewritten prompt is decoded independently using 20 diffusion steps with a classifier-free guidance scale of 7.5. The same CLIP encoder, DINOv2 encoder, and aesthetic predictor are used throughout both training and evaluation to ensure consistency across reported metrics.

Reward Configuration Two reward variants are considered in this work. The primary configuration combines aesthetic improvement with CLIP-based semantic alignment, while the alignment-only variant removes the aesthetic component to isolate the contribution of density-aware reward scaling. Apart from the reward definition, all experimental settings remain unchanged.

Evaluation Protocol Evaluation follows the same protocol for all methods. For each evaluation prompt, the policy generates N=64N = 64 rewritten prompts, each of which is used to

generate one image under identical diffusion sampling settings. Diversity metrics are computed only among images originating from the same source prompt and are then averaged across the evaluation set.

Image-text alignment is measured using OpenAI CLIP ViT-L/14, while image diversity is computed from DINOv2 image embeddings. The aesthetic LAION predictor (sac+logos+ava1-l14-linearMSE) is used wherever aesthetic reward is required.

During training, model checkpoints are periodically evaluated on the validation set using the same evaluation protocol. The checkpoint used for all reported experiments is selected based on the highest validation Vendor Score after reward saturation, ensuring that model selection aligns with the diversity objective optimized by DARS.

Datasets The MS-COCO training split is used for policy optimization, while evaluation is performed on held-out MS-COCO prompts and the Lexica benchmark to assess both in-domain and out-of-domain generalization. Unless otherwise stated, all reported quantitative results use these dataset splits.

ParameterValue
Diffusion modelCompVis/stable-diffusion-v1-4
Resolution / Steps / CFG512 ×\times 512, 20 steps, guidance scale 7.5
SchedulerDPM-Solver Multistep
diffusion.seed_moderandom
run.seed42
Rewrites per prompt (G=NG = N)64
CLIP encoderOpenAI CLIP ViT-L/14
DINOv2 encoderfacebook/dinov2-base (CLS embedding)
Aesthetic predictorsac+logos+ava1-l14-linearMSE
CLIP alignment threshold0.20

Table 7. Shared generation and evaluation configuration.

SettingImplementation
CombinedProduct-style aesthetic gain with CLIP alignment gating
Alignment onlyCLIP alignment reward without aesthetic gain

Table 8. Reward settings.

SplitDataset
Trainingdataset/coco/n_1024/train.jsonl
MS-COCO evaluationdataset/coco/n_1024/test.jsonl
Lexica evaluationdataset/lexica/n_1024/test.jsonl

Table 9. Training and evaluation datasets.

References

  1. [1]Bengio, E.; Jain, M.; Korablyov, M.; Precup, D.; and Bengio, Y. 2021. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. In Advances in Neural Information Processing Systems, volume 34, 27381–27394.arxiv.org/abs/2106.04399
  2. [2]Bengio, Y.; Lahlou, S.; Deleu, T.; Hu, E. J.; Tiwari, M.; and Bengio, E. 2023. GFlowNet Foundations. Journal of Machine Learning Research, 24(210): 1–55.
  3. [3]Chen, Y.; Chakraborty, S.; Wolf, L.; Paschalidis, I. C.; and Pacchiano, A. 2026. Post-training Large Language Models for Diverse High-Quality Responses. In International Conference on Learning Representations.arxiv.org/abs/2509.04784
  4. [4]Feydy, J.; Séjourné, T.; Vialard, F.-X.; Amari, S.-i.; Trouvé, A.; and Peyré, G. 2019. Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, 2681–2690.arxiv.org/abs/1810.08278
  5. [6]Gretton, A.; Borgwardt, K. M.; Rasch, M. J.; Schölkopf, B.; and Smola, A. 2012. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25): 723–773.DOI
  6. [7]GX-Chen, A.; Prakash, J.; Guo, J.; Fergus, R.; and Ranganath, R. 2025. KL-Regularized Reinforcement Learning is Designed to Mode Collapse. arXiv preprint arXiv:2510.20817.arxiv.org/abs/2510.20817
  7. [8]Haarnoja, T.; Zhou, A.; Abb eel, P.; and Levine, S. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 1861–1870.arxiv.org/abs/1801.01290
  8. [9]Hao, Y.; Chi, Z.; Dong, L.; and Wei, F. 2023. Optimizing Prompts for Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, 66923–66939.arxiv.org/abs/2212.09611
  9. [10]Jain, M.; Raparthy, S. C.; Hernández-García, A.; Rector-Brooks, J.; Bengio, Y.; Miret, S.; and Bengio, E. 2023. Multi-Objective GFlowNets. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 14631–14653.arxiv.org/abs/2210.12765
  10. [11]Kirk, R.; Mediratta, I.; Nalmpantis, C.; Luketina, J.; Hambro, E.; Grenfestette, E.; and Raileanu, R. 2024. Understanding the Effects of RLHF on LLM Generalisation and Diversity. In International Conference on Learning Representations.arxiv.org/abs/2310.06452
  11. [13]Kolouri, S.; Nadjahi, K.; Simsekli, U.; Badeau, R.; and Rohde, G. 2019. Generalized sliced wasserstein distances. Advances in neural information processing systems, 32.arxiv.org/abs/1902.00434
  12. [14]Kyngkäänniemi, T.; Karras, T.; Laine, S.; Lehtinen, J.; and Aila, T. 2019. Improved Precision and Recall Metric for Assessing Generative Models. In Advances in Neural Information Processing Systems, volume 32, 3927–3936.
  13. [15]Lahlou, S.; Deleu, T.; Lemos, P.; Zhang, D.; Volokhova, A.; Hernández-García, A.; Ezzine, L. N.; Bengio, Y.; and Malkin, N. 2023. A Theory of Continuous Generative Flow Networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 18269–18300.arxiv.org/abs/2301.12594
  14. [17]Liu, H.; Huang, H.; Wang, J.; Liu, C.; Li, X.; and Ji, X. 2026. DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1864–1873.arxiv.org/abs/2512.21514
  15. [18]Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. 2023. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint arXiv:2304.07193.arxiv.org/abs/2304.07193
  16. [19]Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, volume 35, 27730–27744.arxiv.org/abs/2203.02155
  17. [20]Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 8748–8763.arxiv.org/abs/2103.00020
  18. [24]Schuhmann, C. 2022. LAION-Aesthetics. https://laion.ai/blog/laion-aesthetics/. Accessed: 2026-07-29.
  19. [25]Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.arxiv.org/abs/1707.06347
  20. [26]Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.arxiv.org/abs/2402.03300
  21. [28]Sinha, A.; Elango, S.; and Liu, D. 2026. Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling. arXiv preprint arXiv:2601.21669.arxiv.org/abs/2601.21669
  22. [29]Székely, G. J.; Rizzo, M. L.; et al. 2013. Energy statistics: A class of statistics based on distances. Journal of statistical planning and inference, 143(8): 1249–1272.DOI
  23. [32]Yun, T.; Zhang, D.; Park, J.; and Pan, L. 2025. Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23625–23635.arxiv.org/abs/2502.11477
  24. [33]Zhu, X.; Cheng, D.; Zhang, D.; Li, H.; Zhang, K.; Jiang, C.; Sun, Y.; Hua, E.; Zuo, Y.; Lv, X.; Zhang, Q.; Chen, L.; Shao, F.; Xue, B.; Song, Y.; Yang, Z.; Cui, G.; Ding, N.; Gao, J.; Liu, X.; Zhou, B.; Mei, H.; and Lin, Z. 2026. FlowRL: Matching Reward Distributions for LLM Reasoning. In International Conference on Learning Representations.arxiv.org/abs/2509.15207

Paper details

vlnc:W_mh3ydfwx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents