betaFeedback

WorksComputer Science

Learning Design Taste through Verifiable Synthetic Environments and Agentic Reinforcement Learning

Abstract

Code-generating language models produce user interfaces that build and run, but the pages are often weak in hierarchy, spacing, typography and composition. We study whether a small code model can learn design taste through reinforcement learning in verifiable synthetic environments. The method has three components. First, a layered verification framework splits design quality into hard gates, render-grounded metrics, rubric judges, within-group relative preference and a blinded expert panel. A signal enters the reward only if it passes an admission protocol that tests agreement with experts, causal sensitivity, resistance to hack traps and stability. Second, following SCALER [1], synthetic design environments are parameterised generators with measured difficulty ladders: base pages are written once by an agentic coder, and instances are produced by syntax-tree perturbation operators and constraint samplers, scheduled by an adaptive curriculum with a quarantine rule against reward over-optimisation. Third, a 4B code model is trained with agentic reinforcement learning over multi-turn episodes in which it edits and builds a repository and reads a textual report of its own render, while hidden graders compute the reward. The admission protocol admits 7 of 16 render signals and 6 of 10 rubric dimensions. On TD-Bench, agentic training raises the Taste Index from 0.630 to 0.795 in-family and from 0.598 to 0.700 on held-out families, and wins 0.600 of blinded expert comparisons against the base model and 0.569 against a single-turn control. Most of the advantage depends on the inference-time loop, and cross-brief distinctiveness falls slightly below that of the base model, the only success criterion not met.

Introduction

Language models now generate front-end code that builds and runs, yet the resulting interfaces are often weak in hierarchy, spacing, typography and composition. In Design2Code, generated pages were judged acceptable replacements for the reference in about half of cases, with layout the weakest dimension [2], and DesignBench reports the same deficit across frameworks [3]. Design quality has no single correct answer, but it is not arbitrary: experienced designers agree on many failures, such as weak hierarchy, cramped spacing, low contrast and inconsistent components [4].

A common approach is a single-turn pipeline of supervised fine-tuning, preference optimisation and a short GRPO stage over a static corpus of generated pages [5]. This has three limits. Static data loses its learning signal as the policy improves, as SCALER shows for reasoning tasks [1], and a corpus written by one coder is bounded by that coder’s taste. The webapp-synth substrate synthesises deterministically gradeable front-end tasks reliably, but its reward measures interaction rules rather than design, and its checks run in jsdom, which computes no layout [6]. Finally, the strongest recent results on interface quality come from systems that let a model observe and revise its rendered output [7, 8, 9, 10].

We address these limits with three components. Our contributions are:

  1. a five-layer verification framework for design taste with an admission protocol that decides which signals may enter the reward (§ 4);

  2. synthetic design environments adapted from SCALER, with a two-tier generation pipeline, seven families, perturbation operators and difficulty ladders calibrated from measured pass rates, scheduled by an adaptive curriculum with a quarantine rule (§ 5);

  3. an agentic training formulation with visible development checks and hidden graders (§ 6);

  4. TD-Bench, an evaluation with held-out families and blinded expert comparisons, and experiments against a single-turn control and a prompt-optimised agent (§ 7 and § 8).

Related Work

Evaluating and rewarding interface design. Design2Code measures fidelity to a reference screenshot [2]. ArtifactsBench scores rendered artefacts with a checklist-guided multimodal judge and reports over 90% pairwise agreement with experts [11], UI-Bench fits TrueSkill ratings to 4,075 blinded expert comparisons [12], and UIClip is a pretrained quality scorer [13]. For non-verifiable domains, Rubrics as Rewards uses instance-specific weighted rubrics as the GRPO reward [14]. Model judges show position and verbosity biases [15], and optimising any learned proxy eventually degrades true quality [16]. Relative formulations, such as pairwise visual comparison in UI2CodeN^{N} [8] and in-group win rates in Pref-GRPO [17], reduce calibration noise.

Render-in-the-loop generation. AesCoder trains a 4B model with GRPO on execution and aesthetic rewards [9]. ReLook gives zero reward to pages that fail to render and accepts a revision only if a critic score improves [7]. Vision-guided refinement improves frontend output by up to 17.8% over three critique cycles [10], and 1D-Bench defines round-level rewards for agentic interface editing [18].

Environments, curricula and agentic RL. Reasoning Gym and RLVE provide procedurally generated environments with tunable difficulty [19, 20]. SCALER converts problems into environments with explicit scale parameters, validates their generators, adjusts difficulty online towards a target accuracy and retires stagnant environments [1]. GRPO removes the value network by normalising rewards within a group [21], SkyRL-Agent scales multi-turn rollouts asynchronously [22], and GEPA shows that prompt optimisation is a strong and inexpensive baseline [23].

Learning taste. Reinforcement Learning from Community Feedback trains a judge of research impact and uses in-group win rates as the reward for an ideation policy [24]. TASTE finds that designer agreement is higher for rule-based criteria, such as typography, than for colour harmony [25], and Taste-Bench measures agent taste as choosing the better direction at decision forks [26]. Like these studies, we validate the reward against human judgement and use relative comparison where absolute scores are unreliable. Unlike them, we verify taste in layers, with deterministic render metrics carrying the parts on which experts agree.

Problem Formulation and Overview

A task instance is a triple x=(q,s0,g)x = (q, s_0, g): a natural-language instruction qq (a symptom description, a holistic request or a structured brief), an initial repository state s0s_0 (a complete page, a broken page or an empty scaffold) and a hidden grader configuration gg. A policy πθ\pi_\theta interacts with a sandbox for at most KK turns, emitting tool calls or a submission and receiving observations. The production bundle of the submitted state is scored by graders the policy cannot observe, giving a terminal reward R(ξ)∈[0,1]R(\xi) \in[0,1] for episode ξ\xi.

Figure 1. Overview. (1) Agentic generation of base pages with best-of-NN selection. (2) The five verification layers and the admission protocol that determines M+\mathcal{M}^{+}. (3) Environments that emit instances at a controlled level of a measured difficulty ladder. (4) Multi-turn agentic RL with hidden graders and group-relative advantages. (5) Oversight calibrates admission against experts, quarantines environments whose judge reward diverges from the canary, and converts failures into new rubric items and operators.

Following RLVE and SCALER [1, 20], an environment is a parameterised problem space with an explicit difficulty ladder,

Ee=(Ie,Pe,Re,De),x=Pe(ω,ℓ),ℓ∈{0,…,De},(1)\mathcal{E}_e = (I_e, P_e, R_e, D_e), \qquad x = P_e(\omega,\ell), \qquad\ell\in\{0,\ldots,D_e\}, \tag*{(1)}

where IeI_e is the instruction template and tool set, PeP_e a generator that maps a seed ω\omega and level ℓ\ell to an instance, ReR_e the verifier and DeD_e the maximum level. Training maximises

max⁡θ EE∼Wt, ℓ∼qtEx∼Pe(ω,ℓ)Eξ∼πθ(⋅∣x)[Re(ξ)],(2)\max_{\theta}\ \mathbb{E}_{\mathcal{E}\sim W_t,\ \ell\sim q_t}\mathbb{E}_{x\sim P_e(\omega,\ell)}\mathbb{E}_{\xi\sim\pi_\theta(\cdot\mid x)}[R_e(\xi)], \tag*{(2)}

where the active environment set WtW_t and level distribution qteq_t^e are adapted during training (Section 5.1). We do not assume a single verifier of design quality. We use signals V0,…,V4V_0,\ldots,V_4 of increasing fidelity and decreasing reliability, and only candidates in the admitted set M+⊆M\mathcal{M}^{+} \subseteq\mathcal{M}, validated against the expert layer, contribute to RR.

Figure 1 shows the system. The substrate is the webapp-synth repository, which provides a seed taxonomy, deterministic checks, a RED→\rightarrowGREEN admission gate and a sandboxed agent harness [6]. Component A decides which signals measure improvement, Component B turns them into environments with difficulty ladders, and Component C trains the policy. Policy samples and failures return to Component A as expert labels, rubric items and perturbation operators.

Component A: Layered Verification

Some parts of design quality follow explicit principles and can be computed from a rendered page, such as text contrast, adherence to a spacing scale and the absence of horizontal overflow on narrow viewports. Others need judgement but can be split into checkable questions, and some appear only under comparison. We therefore verify in five layers (Figure 2). V0V_0 hard gates (build, render without errors, anticheat, page volume, functional contracts) are binary and multiply the reward. V1V_1 render metrics, V2V_2 rubric judges and V3V_3 relative preference contribute graded terms. V4V_4, a blinded expert panel, never enters the reward but decides which lower-layer signals may. Rendering uses headless Chromium on the production bundle, since V1V_1 and above need computed layout.

Figure 2. Layered verification for one rollout of a brief-to-page (F5) instance. The gates pass and 13 of 14 contract tests succeed (Sfunc=0.93S_{\mathrm{func}} = 0.93); six render metrics give Sdesign=0.89S_{\mathrm{design}} = 0.89; admitted rubric items give Srubric=0.76S_{\mathrm{rubric}} = 0.76; an in-group tournament gives Spref=0.67S_{\mathrm{pref}} = 0.67. The expert layer decides admission and scores the canary but is not a reward term. With F5 weights, Eq. (4) gives R=0.77R = 0.77, above the 0.4 floor.

Render metrics (V1V_1). Each metric sms_m maps the render capture cvc_v at viewport vv (DOM, computed styles, box model and accessibility tree) to [0,1][0,1] and is averaged over viewports V={375,768,1280,1440}\mathcal{V} = \{375,768,1280,1440\} px and interaction states. The library covers text contrast (WCAG AA [27]), spacing-scale conformity, proximity, type-scale coherence, hierarchy integrity, primary-action salience, line length, alignment, palette discipline, component consistency, responsive integrity, accessibility, performance and token hygiene. Every metric reads computed styles rather than class names, and each has a recorded exploit and counterweight; for example, collapsing all spacing to one value raises scale conformity but lowers the proximity metric. Two further signals target generic output: a default-tell detector for common defaults, and the slop distance, the Mahalanobis distance of a page’s style signature from the centroid of base-model pages. Neither measures quality, so both are used only in the art-direction family.

Rubric judges and preference (V2V_2, V3V_3). Following RaR [14], each brief receives a weighted rubric over ten dimensions (hierarchy, layout, typography, colour, responsiveness, accessibility, copy, conversion, direction, craft) with negative-weight pitfall items. A bound item is scored by its V1V_1 metric, so judges never re-estimate a measurable quantity; a judged item is scored by the ensemble only if its template is in M+\mathcal{M}^{+}. The ensemble has a code-and-metrics judge and a screenshot judge from model families other than the policy’s, and a third judge is withheld for evaluation. Comments and unrendered strings are stripped before judging, and pairs are judged in both orders [15]. Within a GRPO group of n=8n = 8, a Swiss schedule of 12 comparisons replaces the 28 of a round robin, and Spref(ξi)S_{\mathrm{pref}}(\xi_i) is the win rate of rollout ii over its scheduled comparisons.

Expert panel (V4V_4). Three to five senior designers give blinded forced-choice judgements, item scores and short rationales, with catch trials and repeats. Reliability is reported as Krippendorff’s α\alpha [28]. Estimating an agreement rate within ±0.05\pm0.05 at 95% confidence needs N=1.962×0.25/0.052≈385N = 1.96^2 \times0.25/0.05^2 \approx385 pairs per dimension.

Admission protocol

A candidate m∈Mm \in\mathcal{M} enters M+\mathcal{M}^{+} only if it passes four tests. Let Pm\mathcal{P}_m be the golden pairs (a,b,y)(a,b,y), with expert preference y∈{−1,+1}y \in\{-1,+1\}, on which the metric difference exceeds its noise floor. Agreement is

Am=1∣Pm∣∑(a,b,y)∈Pm1[sgn⁡(sm(a)−sm(b))=y]≥0.85.(3)A_m = \frac{1}{|\mathcal{P}_m|} \sum_{(a,b,y)\in\mathcal{P}_m} \mathbf{1}[\operatorname{sgn}(s_m(a)-s_m(b)) = y] \ge0.85. \tag*{(3)}

Causal sensitivity requires that on at least 90% of single-principle minimal pairs the score drops after a perturbation and recovers after repair. Hack-trap resistance requires that the metric prefers a page built to game it over an expert-preferred page on at most 20% of trap pairs. Stability requires at least 0.90 self-agreement across render seeds, viewport jitter and repeated judge calls. Failing items are kept for evaluation only, and admitted items are re-tested whenever the golden set or the judges change.

Composite reward

For a finished episode,

R(ξ)=G(ξ)⋅max⁡(0,∑kwkSk(ξ)−λlenPlen(ξ)−λturnPturn(ξ)),k∈{func,design,rubric,pref},(4)R(\xi) = G(\xi)\cdot\max\left(0,\sum_k w_k S_k(\xi)-\lambda_{\mathrm{len}}P_{\mathrm{len}}(\xi)-\lambda_{\mathrm{turn}}P_{\mathrm{turn}}(\xi)\right), \qquad k \in\{\mathrm{func},\mathrm{design},\mathrm{rubric},\mathrm{pref}\}, \tag*{(4)}

with weights summing to one. The gate GG is the product of build, render, anticheat, volume and no-regression checks; the last gives zero reward if a repair degrades any other admitted metric of the base page, so deleting a defective section is never the cheapest fix. SfuncS_{\mathrm{func}} is the vitest pass fraction [6], SdesignS_{\mathrm{design}} the weighted mean of admitted metrics (restricted in repair families to the metrics affected by the planted defects), SrubricS_{\mathrm{rubric}} the score over admitted judged items, and PlenP_{\mathrm{len}}, PturnP_{\mathrm{turn}} penalise page inflation and extra turns. Two rules from webapp-synth apply to every family: integrity is guarded by gates and never by the graded score, and the unmodified starting state must score at most 0.4 (the reward floor). Instructions describe symptoms, never the fix.

Component B: Synthetic Design Environments

SCALER treats an environment, a program that yields unlimited verifiable instances of one problem type at a chosen difficulty, as the unit of training data [1]. In webapp-synth, by contrast, each instance needs a 20 to 30 minute agentic generation session and carries an unmeasured difficulty label [6]. We keep the expensive generation step but run it once per base page, and generate instances with inexpensive programs.

Two-tier generation. Tier 1 runs offline: agentic coding sessions write long-form pages from sampled briefs, the validated judge selects one of several candidates per brief, and selected pages that pass the build, the page audit, a V1V_1 baseline and a human spot check enter a base-page pool. Tier 2 runs online: given a base page and a seed, a program samples perturbation operators and severities, applies them as syntax-tree and stylesheet edits, normalises formatting so the diff does not reveal edit locations, writes a symptom-only instruction and emits the hidden grader configuration in seconds (Figure 3). Generation families use a constraint sampler instead. This split matches the cost asymmetry: a pool of several hundred base pages is affordable, tens of thousands of agentic sessions are not.

Figure 3. Two-tier environment synthesis. Tier 1 (offline) produces judge-selected base pages from briefs. Tier 2 (online) samples a base page and instance parameters (seed ω\omega, level ℓ\ell, operators), applies the transforms and emits an instance x=(q,s0,g)x=(q,s_0,g). The inset gives the measured pass rate of the base policy at each level; the worked instance is at level 3.

Families and operators. Seven families target complementary aspects of design (Table 1). Page-editing families (F1 to F4, F7) have a well-defined do-nothing floor and mostly deterministic rewards; open-ended families (F5, F6) rely more on judges and relative preference. Fourteen perturbation operators, such as spacing drift, contrast degradation, hierarchy scramble, action dilution and breakpoint removal, each break one principle and are each linked to the metric that detects it. Hidden graders test the violated invariant rather than restoration of the original source, so any correct repair receives full credit. F7 scores improvement over the starting page and accepts a round’s edits only if the hidden score rises [7].

FamilyTaskDifficulty parametersSfuncS_{\mathrm{func}}SdesignS_{\mathrm{design}}SrubricS_{\mathrm{rubric}}SprefS_{\mathrm{pref}}
F1 Interaction repairRepair a broken interaction rule on a long-form pageCoupled seeds, localisation depth1.0
F2 Design polishLocate and repair planted design defectsNumber, subtlety and coupling of defects0.80.2
F3 System adherenceAdd a component using only given tokens and specificationsToken-set size, component novelty0.20.60.2
F4 Responsive adaptationMake a desktop-only page correct at every viewportAffected sections, layout complexity0.20.8
F5 Brief to pageBuild a page from a brief with explicit constraintsConstraint count, conflicting requirements0.10.30.30.3
F6 Art directionBuild a page that commits to a named aesthetic directionSpecificity of direction0.20.40.4
F7 Critique revisionImprove a page over several rounds from a critiqueCritique specificity, rounds0.40.6

Table 1. Environment families, their difficulty parameters and initial reward weights wkw_k (Eq. (4)). The weights keep every family’s reward floor at or below 0.4.

Generator validation and difficulty. Because Tier 2 generators are partly written by an agent, each must pass SCALER-style checks [1, 29] before entering the pool: the RED→GREEN gate (the instance fails a hidden check, the base page passes all), breadth (two independent correct repairs both pass), depth (instances at one level differ in target elements), the reward floor, volume bounds, absence of leakage, and severity monotonicity. Design has no single scale parameter, so levels are defined empirically: the base policy is run on a grid of operator subsets and severities, and a monotone fit maps grid cells to levels 0,…,De0,\ldots,D_e with roughly uniform steps in pass rate.

Adaptive curriculum

An episode is a success if it passes every gate and its reward exceeds a family threshold θe\theta_e. With success rate acc⁡te\operatorname{acc}_t^e over the k×nk \times n rollouts of environment ee at step tt, the difficulty is updated as

dt+1e=clip⁡(dte+β(acc⁡te−τ),0,De),(5)d_{t+1}^e = \operatorname{clip}\left(d_t^e + \beta\left(\operatorname{acc}_t^e-\tau\right), 0, D_e\right), \tag*{(5)}

and the kk instances of a step are split between the two nearest integer levels so that their mean approximates dd [1]. A target τ≈0.5\tau\approx0.5 maximises the chance that a group contains both successes and failures, and hence a non-zero advantage. The outer loop keeps an active set WtW_t of fixed size and, as in SCALER, retires an environment temporarily when its difficulty slope over 10 steps is not positive, its success rate is zero for 5 steps, or its difficulty stays at DeD_e for 5 steps. We add one rule for learned rewards: an environment is quarantined for human review when its judge-derived reward rises while its canary score (expert or held-out judge) stays flat over KhackK_{\mathrm{hack}} steps,

Qe=1[slope⁡(Rˉjudgee)>0∧slope⁡(Sˉcanarye)≤0],(6)Q_e = \mathbf{1}\left[\operatorname{slope}\left(\bar{R}_{\mathrm{judge}}^e\right)>0 \land\operatorname{slope}\left(\bar{S}_{\mathrm{canary}}^e\right)\leq0\right], \tag*{(6)}

since rising proxy reward with flat true quality is the typical sign of over-optimisation [16]. Figure 4 shows both loops.

Figure 4. Adaptive curriculum. (a) Outer loop: each line is the difficulty of one environment over training. F2 and F3 are still rising and stay active; F4 stagnates and is retired temporarily; F5’s judge reward rises while its canary score stays flat, so it is quarantined. (b) Inner loop for one F2 environment (β=2\beta= 2, τ=0.5\tau= 0.5, n=8n = 8): the success rate of each group moves the difficulty by Eq. (5).

Held-out environments. One complete family (F4), two operator types, one industry and a set of designer-authored briefs are excluded from training. A policy that has learned to invert specific operators, rather than to perceive design problems, will show a large gap on them.

Component C: Agentic Reinforcement Learning

A single-turn policy must emit a finished page without seeing it, whereas design practice is iterative, and models benefit from observing and revising their output [7, 10]. Multi-turn episodes also make the repair families possible, since they are operations on a stateful repository. The costs are sandbox time, harder credit assignment and malformed tool calls from a small model, so we keep a single-turn control and a prompt-optimised agent as baselines.

Episodes and tools. The observation holds the instruction, the repository, any design tokens and a budget of 20 turns (Figure 5). The policy can read, search and edit files, build, call render_report(v) for a textual outline of the render at viewport vv (headings with computed sizes, per-section layout, overflowing or clipped elements), run public functional tests, and call dev_lint. The render report describes what is present, not whether it is good. The key design decision is how much of the verifier to expose. Following the convention of public and hidden tests, dev_lint covers a rotating subset of metrics at relaxed thresholds, while hidden graders cover all admitted metrics at brief-specific thresholds together with the judges. We monitor the visible–hidden gap Γt=Sˉvis,t−Sˉhid,t\Gamma_t = \bar{S}_{\mathrm{vis},t} - \bar{S}_{\mathrm{hid},t} on shared metrics; a rising Γt\Gamma_t shows that the policy is fitting the visible checks.

An agentic episode showing observation, trajectory, hidden grading, and group update

Figure 5. An agentic episode. (1) The policy sees a symptom-only instruction, the repository, design tokens and its budget, but not the grader. (2) It locates the component, inspects the render report at 375 px, edits spacing and colour classes, rebuilds, runs the linter and submits. (3) Hidden graders score the bundle; with F2 weights, R1=0.83R_1 = 0.83. (4) Rewards in the group become advantages (Eq. (7)) for the policy update.

Objective. For n=8n = 8 rollouts of an instance, advantages are group-normalised [21],

Ai=Ri−mean⁡(R1,…,Rn)std⁡(R1,…,Rn)+ε,(7)A_i = \frac{R_i - \operatorname{mean}(R_1,\ldots,R_n)}{\operatorname{std}(R_1,\ldots,R_n) + \varepsilon}, \tag*{(7)}

and the policy maximises the clipped surrogate with a token mask that excludes tool outputs from the loss, plus a small KL penalty to a reference policy, kept because the reward includes learned judges. We use a single terminal reward per episode: it is the hardest to exploit, and potential-based shaping [30] telescopes into an end-of-episode term under trajectory-level GRPO. The policy is a 4B code model (Qwen3.5-4B) with a short warm-up on tool-use traces replayed from the 48 existing brief–page pairs.

System. Rendering and judging run outside the agent sandbox. The policy builds inside a prebaked sandbox image (up to 256 concurrent), which also runs the functional tests. The production bundle is sent to a separate render and reward service with a pool of headless browsers, the metric library and the judge ensemble, which caches results by bundle hash. This keeps the sandbox small, lets renders be batched and cached across rollouts, and makes the hidden graders unreachable from the policy. Monitors read the metric stream and can quarantine an environment through the curriculum controller. Cost is accounted in sandbox-minutes per solved instance.

Experimental Setup

Hypotheses. H1: a measurable share of expert preferences is predictable from render metrics, with the share varying by dimension. H2: adaptive environments keep the learning signal more informative than a static task set. H3: at matched compute, multi-turn agentic RL achieves a higher expert win rate than single-turn training, and part of the gain persists without tools. H4: the layered reward keeps the divergence between reward gain and expert gain small. H5: a code-and-metrics judge agrees with experts nearly as well as a screenshot judge, except on composition.

Experiments. E1 applies the admission protocol to every candidate signal (H1). E2 measures the share of informative GRPO groups (H2). E3 compares training methods at matched compute (H3). E4 measures robustness diagnostics (H4). E5 compares the two judge types per dimension (H5). E6 removes tools at inference (H3). E7 measures distinctiveness across briefs.

Benchmark. TD-Bench extends MiniLandingBench [5] with four splits: held-out instances of training families (S1, in-family); held-out families, operators and industry (S2, held-out); tool-free single-shot generation from designer briefs; and an external slice from OpenDesign [9] and ArtifactsBench [11].

Systems. All systems use the same 4B policy, harness, briefs and render service, are matched in sandbox-minutes and run with three seeds. BASE is the untrained model in the agentic harness. SFT-C is the single-turn control: SFT on the 48 long-form pages, DPO on golden pairs [31] and a short single-turn GRPO stage. GEPA is the base agent with a system prompt evolved by GEPA [23], testing whether gains need training. ARL is our method.

Metrics. The primary metrics are gate pass rate (P1), functional contract rate (P2), four composites of admitted and evaluation-only render metrics (legibility P3, system coherence P4, hierarchy and composition P5, render robustness P6), rubric score (P7), brief-constraint retention (P8), blinded expert win rate (P9) and distinctiveness (P10). The Taste Index (TI) is the unweighted mean of P3 to P8; expert win rate is kept separate as ground truth. Win rates carry Wilson 95% intervals. A two-sided test at α=0.05\alpha= 0.05 with power 0.8 needs about 194 comparisons to detect a win rate of 0.60 and 783 for 0.55, so we allocate 400 expert comparisons for ARL against BASE on S2, 800 against SFT-C, and 200 each against GEPA and in-family.

Success criteria. Fixed before the experiments, on held-out splits: an expert win rate of at least 0.60 against BASE and 0.55 against SFT-C; no loss in gate pass rate or cross-brief diversity relative to BASE; reward–expert divergence at most 0.05; held-out gains at least half of in-family gains; and no unresolved quarantine events.

Results

What is verifiable (E1, E5)

The admission protocol admits 7 of the 16 render signals and 6 of the 10 rubric dimensions (Figure 6). The admitted signals (contrast, spacing scale, type scale, component consistency, hierarchy, responsiveness and accessibility) are tied to explicit standards or to defects that designers name consistently. Line length, palette discipline, action salience, proximity and alignment fall short on agreement or exceed the trap ceiling, and token hygiene fails because experts judge pixels rather than class names. Copy, conversion and craft remain evaluation-only, and direction could not be tested with only 240 golden pairs. Composition is the largest verification gap: of its four signals, only hierarchy integrity enters the reward. The judge comparison supports H5. The screenshot judge agrees with experts clearly better only on layout (+0.14+0.14) and direction (+0.11+0.11), the code-and-metrics judge is better on accessibility (−0.07-0.07), where roles and focus order are invisible in pixels, and on the other seven dimensions the two differ by at most 0.050.05.

Figure 6. Admission outcomes (E1). Each row is a candidate signal, with its agreement with expert pairs (left) and hack-trap preference rate (right). Shaded regions satisfy the thresholds (agreement at least 0.85, trap rate at most 0.20). Blue points are admitted, grey points are evaluation-only, and the open point was not tested.

Agentic training against the controls (E3)

In-family, ARL raises the Taste Index from 0.630 to 0.795, against 0.689 for SFT-C and 0.690 for GEPA (Table 2). The largest gains are on metrics carried by admitted signals and heavy family weights: gate pass rate (0.70 to 0.93), functional contracts (0.52 to 0.81) and legibility (0.74 to 0.88). The smallest are on hierarchy and composition (0.61 to 0.77) and rubric score (0.52 to 0.67), which depend on evaluation-only signals and noisy judged items. On the held-out split, ARL wins 0.600 of expert comparisons against BASE (Wilson interval 0.551 to 0.647, N=400N = 400), exactly at the threshold, and 0.569 against SFT-C (0.534 to 0.603, N=800N = 800), which meets the 0.55 criterion (Figure 7). The contrast with GEPA (0.55, N=200N = 200) is reported descriptively.

Figure 7. Expert win rates of ARL with 95% Wilson intervals. Vertical ticks mark the success thresholds and the dashed line marks chance.

MetricIn-family (S1) BASEIn-family (S1) SFT-CIn-family (S1) GEPAIn-family (S1) ARLHeld-out (S2) BASEHeld-out (S2) SFT-CHeld-out (S2) GEPAHeld-out (S2) ARL
Gate pass rate (P1)0.700.790.780.930.680.730.740.82
Functional contracts (P2)0.520.620.580.810.480.530.530.66
Legibility (P3)0.740.800.790.880.730.760.770.82
System coherence (P4)0.660.730.690.830.640.680.670.75
Hierarchy & composition (P5)0.610.670.640.770.590.620.620.69
Render robustness (P6)0.700.740.740.860.600.620.630.70
Rubric score (P7)0.520.580.600.670.500.530.570.60
Constraint retention (P8)0.550.610.690.750.530.560.650.65
Distinctiveness (P10)0.480.440.550.450.470.450.530.45
Taste Index (TI)0.6300.6890.6900.7950.5980.6310.6500.700
Win rate vs BASE (P9)n/a0.5590.5600.659n/a0.5320.5510.601

Table 2. Primary metrics on the in-family (S1) and held-out (S2) splits. The best value in each split is in bold. P9 is the expert win rate against BASE.

Generalisation and the role of the loop (E6)

ARL keeps 62% of its in-family gain on held-out families, above the required half (Figure 8a). The advantage depends on the loop: with tools removed, ARL keeps only 35% of its gain (TI 0.654) and matches SFT-C (0.653), which was trained single-turn. Part of the gain therefore persists, as H3 states, but agentic training does not give a better tool-free policy. On the external slice ARL remains ahead (0.647 against 0.610 to 0.618). By family (Figure 8b), ARL gains most where the reward is deterministic, design polish (0.55 to 0.85) and system adherence (0.64 to 0.84), and the held-out responsive family reaches 0.75.

Figure 8. (a) Taste Index on the four TD-Bench splits. (b) Taste Index by environment family; F4 is held out.

Robustness, curriculum and cost (E2, E4)

The divergence between reward gain and expert gain stays within the 0.05 bound in-family and on held-out families (0.040 and 0.045) and exceeds it only on the external slice (0.06) (Figure 9a). The visible–hidden gap reaches 0.06 in-family because dev_lint is visible throughout training, and no-regression violations reach 0.06 under held-out operators. A separate check measures the reward a policy earns without solving the task (Figure 9b). Three families exceed the 0.4 floor: in design polish (0.41) perturbed metrics remain partly high, in responsive adaptation (0.48) an untouched desktop page keeps its functional contracts, and in critique revision (0.46) an unchanged page can outrank worse revisions. Scoring SdesignS_{\mathrm{design}} as the repaired fraction, restricting SfuncS_{\mathrm{func}} in F4 to responsive contracts, and scoring F7 by improvement remove these violations.

Figure 9. (a) ARL diagnostics divided by their bounds; points in the shaded region exceed the bound. (b) Reward earned without solving the task, per family; red bars exceed the 0.4 floor.

The adaptive curriculum keeps 0.86 of GRPO groups informative, with both successes and failures, against 0.55 for the static-data GRPO stage of SFT-C, which supports H2 for the learning signal; its effect on expert win rate was not isolated. ARL is also the least costly system per solved instance, 6.9 sandbox-minutes against 14.7 for BASE, because its higher success rate outweighs its longer episodes. Brief-constraint retention rises from 0.55 to 0.75, but constraints that must survive an edit are lost twice as often as explicit ones, and conflicting requirements are flagged in only about half of briefs.

Distinctiveness and external comparison (E7)

Distinctiveness is the only success criterion ARL does not meet (Table 3). Slop distance and default tells measure departure from generic output rather than quality, so they cannot pass the agreement test, and direction has too few golden pairs. Without these counterweights, 80% of the F6 reward comes from judge terms that favour the judges’ central taste. On the held-out split, ARL’s distinctiveness falls from 0.47 to 0.454, while GEPA, whose prompt asks for an explicit direction, rises to 0.532. On OpenDesign, ARL scores 78.4 against 75.0 for BASE, seventh of sixteen published systems and 3.5 points below AesCoder-4B, which is trained with a screenshot reward on similar data [9].

CriterionTargetResultOutcome
Expert win rate against BASE≥0.60\ge0.600.600 (0.551–0.647)Pass, no margin
Expert win rate against SFT-C≥0.55\ge0.550.569 (0.534–0.603)Pass
Gate pass rate≥\ge BASE (0.68)0.82Pass
Cross-brief diversity≥\ge BASE (0.47)0.454Fail
Reward–expert divergence≤0.05\le0.050.045Pass
Held-out gain ÷\div in-family gain≥0.50\ge0.500.62Pass
Unresolved quarantine events00Pass
Criteria met6 of 7

Table 3. Success criteria applied to the held-out (S2) results.

Discussion and Limitations

Discussion and Limitations

The verifiability map agrees with recent work on designer preference. TASTE reports higher designer agreement for typography and spatial accuracy than for colour harmony [25], and in our results the signals tied to explicit rules pass admission while composition and direction do not. Taste-Bench finds that judgement errors concentrate where the deciding evidence appears late in a trajectory [26]; the dependence of our gains on the render loop is consistent with this, since the loop lets the policy see the effect of its edits before submitting. The results point to four changes: admit distinctiveness terms through a direction-adherence test rather than expert agreement, which addresses the unmet criterion; rescore F2, F4 and F7 so every family respects the reward floor; rotate the dev_lint subset per epoch to reduce Γ\Gamma; and add a screenshot-judged layout item to close part of the composition gap.

Limitations. The rubric and metric library may encode a single notion of taste; we keep label distributions where expert agreement is low and condition items on the requested direction. Base pages come from one agentic coder, so judges are drawn from other model families and one judge is held out. The scope is long-form landing pages in React and Tailwind; transfer to dashboards, multi-screen flows or native interfaces is not established. The primary win rate against BASE sits exactly at its threshold and should be read with its interval. The policy is text-only, only one size (4B) was trained, and neither the curriculum’s effect on expert win rate nor scaling with the number of environments was measured.

Conclusion

We presented a method in which a small code model learns design taste by reinforcement learning in environments that make taste checkable in layers. Render metrics and rubric judges enter the reward only through an admission protocol validated against experts, environments are generators with measured difficulty scheduled by a curriculum that also guards against over-optimisation, and the policy learns through multi-turn episodes scored by graders it cannot observe. About half of the candidate signals can be verified, with composition the largest gap. Agentic training improves every primary metric except distinctiveness, transfers to held-out families, and meets the expert win-rate criteria against the base model and the single-turn control, although most of the gain depends on the inference-time loop. Admitting distinctiveness through direction adherence and closing the composition gap are the main directions for future work.

References

  1. [1]C. Xu, C. Xiao, Z. Peng, X. Wang, and Y. Cao. SCALER: Synthetic scalable adaptive learning environment for reasoning. arXiv preprint arXiv:2601.04809v5, 2026.arxiv.org/abs/2601.04809
  2. [2]C. Si, Y. Zhang, R. Li, Z. Yang, R. Liu, and D. Yang. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), pages 3956–3974, 2025.arxiv.org/abs/2403.03163
  3. [3]J. Xiao, M. Wang, M. H. Lam, Y. Wan, J. Liu, Y. Huo, and M. R. Lyu. DesignBench: A comprehensive benchmark for MLLM-based front-end code generation. arXiv preprint arXiv:2506.06251, 2025.arxiv.org/abs/2506.06251
  4. [4]Tasteful Designs Project. Teaching LLMs UI design taste: Post-training, RL alignment, agentic RL, synthetic environments, design-taste evaluation and meta-harnesses. Internal research report, Tasteful Designs Project, September 2026.
  5. [5]Tasteful Designs Project. Developing taste for design in models: Rubric-grounded golden preferences and reinforcement post-training of a small LLM for generating better design. Internal pilot design, Tasteful Designs Project, June 2026.
  6. [6]Tasteful Designs Project. webapp-synth: Architecture and operations guide, and README. Repository documentation, https://github.com/mr-siddy/Tasteful-Designs, 2026.
  7. [7]ReLook authors. ReLook: Vision-grounded RL with a multimodal LLM critic for agentic web coding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2026. arXiv:2510.11498.
  8. [8]Z. Yang, W. Hong, et al. UI2Code^N: UI-to-code generation as interactive visual optimization. arXiv preprint arXiv:2511.08195, 2025.
  9. [9]B. Xiao, L. Jiang, S. Huang, et al. Code aesthetics with agentic reward feedback. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.23272.arxiv.org/abs/2510.23272
  10. [10]Vision-guided refinement authors. Vision-guided iterative refinement for frontend code generation. arXiv preprint arXiv:2604.05839, 2026.
  11. [11]Tencent Hunyuan. ArtifactsBench: Bridging the visual-interactive gap in LLM code generation evaluation. arXiv preprint arXiv:2507.04952, 2025.
  12. [12]S. Mateega et al. UI-Bench: A benchmark for evaluating design capabilities of AI text-to-app tools. arXiv preprint arXiv:2508.20410, 2025.arxiv.org/abs/2508.20410
  13. [13]J. Wu et al. UIClip: A data-driven model for assessing user interface design. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), 2024.arxiv.org/abs/2404.12500
  14. [14]A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.arxiv.org/abs/2507.17746
  15. [15]L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.05685
  16. [16]L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning (ICML), 2023. arXiv:2210.10760.arxiv.org/abs/2210.10760
  17. [17]Y. Wang, Z. Li, Y. Zang, Y. Zhou, J. Bu, C. Wang, Q. Lu, C. Jin, and J. Wang. Pref-GRPO: Pairwise preference reward-based GRPO for stable text-to-image reinforcement learning. arXiv preprint arXiv:2508.20751, 2025.arxiv.org/abs/2508.20751
  18. [18]1D-Bench authors. 1D-Bench. arXiv preprint arXiv:2602.18548, 2026.
  19. [19]Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760, 2025.arxiv.org/abs/2505.24760
  20. [20]Z. Zeng, H. Ivison, Y. Wang, L. Yuan, S. S. Li, Z. Ye, S. Li, J. He, R. Zhou, T. Chen, et al. RIVE: Scaling up reinforcement learning for language models with adaptive verifiable environments. arXiv preprint arXiv:2511.07317, 2025.arxiv.org/abs/2511.07317
  21. [21]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  22. [22]SkyRL team. SkyRL-Agent: Efficient RL training for multi-turn LLM agents. arXiv preprint arXiv:2511.16108, 2025.
  23. [23]L. A. Agrawal et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (ICLR), 2026. arXiv:2507.19457.arxiv.org/abs/2507.19457
  24. [24]Jingqi Tong, Mingzhe Li, Hangcheng Li, Yongzhuo Yang, Yurong Mou, et al. AI can learn scientific taste. arXiv preprint arXiv:2603.14473, 2026.arxiv.org/abs/2603.14473
  25. [25]Haonan Zhu, Elad Hirsch, Alexandria Minetti, Allison Nulty, Ben Huffman, and Purvanshi Mehta. TASTE: A designer-annotated multi-dimensional preference dataset for AI-generated graphic design. arXiv preprint arXiv:2605.20731, 2026.arxiv.org/abs/2605.20731
  26. [26]Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, and Xiaohua Jia. The tasteful agent: Measuring and improving taste in long-horizon tasks. arXiv preprint arXiv:2609.25804, 2026.arxiv.org/abs/2609.25804
  27. [27]World Wide Web Consortium. Web content accessibility guidelines (WCAG) 2.2. W3C Recommendation, 2023.
  28. [28]K. Krippendorff. Content Analysis: An Introduction to Its Methodology. SAGE Publications, 4th edition, 2018.
  29. [29]J. Fu, X. Yang, H. Zhang, Y. Liu, J. Zhang, Q. Wang, F. Zhang, and G. Zhou. Klear-CodeTest: Scalable test case generation for code reinforcement learning. arXiv preprint arXiv:2508.05710, 2025.arxiv.org/abs/2508.05710
  30. [30]A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), 1999.
  31. [31]R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.18290

Paper details

vlnc:W_e6k9dvzj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents