WorksComputer Science
Learning Design Taste through Verifiable Synthetic Environments and Agentic Reinforcement Learning
Abstract
Code-generating language models produce user interfaces that build and run, but the pages are often weak in hierarchy, spacing, typography and composition. We study whether a small code model can learn design taste through reinforcement learning in verifiable synthetic environments. The method has three components. First, a layered verification framework splits design quality into hard gates, render-grounded metrics, rubric judges, within-group relative preference and a blinded expert panel. A signal enters the reward only if it passes an admission protocol that tests agreement with experts, causal sensitivity, resistance to hack traps and stability. Second, following SCALER [1], synthetic design environments are parameterised generators with measured difficulty ladders: base pages are written once by an agentic coder, and instances are produced by syntax-tree perturbation operators and constraint samplers, scheduled by an adaptive curriculum with a quarantine rule against reward over-optimisation. Third, a 4B code model is trained with agentic reinforcement learning over multi-turn episodes in which it edits and builds a repository and reads a textual report of its own render, while hidden graders compute the reward. The admission protocol admits 7 of 16 render signals and 6 of 10 rubric dimensions. On TD-Bench, agentic training raises the Taste Index from 0.630 to 0.795 in-family and from 0.598 to 0.700 on held-out families, and wins 0.600 of blinded expert comparisons against the base model and 0.569 against a single-turn control. Most of the advantage depends on the inference-time loop, and cross-brief distinctiveness falls slightly below that of the base model, the only success criterion not met.
Introduction
Language models now generate front-end code that builds and runs, yet the resulting interfaces are often weak in hierarchy, spacing, typography and composition. In Design2Code, generated pages were judged acceptable replacements for the reference in about half of cases, with layout the weakest dimension [2], and DesignBench reports the same deficit across frameworks [3]. Design quality has no single correct answer, but it is not arbitrary: experienced designers agree on many failures, such as weak hierarchy, cramped spacing, low contrast and inconsistent components [4].
A common approach is a single-turn pipeline of supervised fine-tuning, preference optimisation and a short GRPO stage over a static corpus of generated pages [5]. This has three limits. Static data loses its learning signal as the policy improves, as SCALER shows for reasoning tasks [1], and a corpus written by one coder is bounded by that coder’s taste. The webapp-synth substrate synthesises deterministically gradeable front-end tasks reliably, but its reward measures interaction rules rather than design, and its checks run in jsdom, which computes no layout [6]. Finally, the strongest recent results on interface quality come from systems that let a model observe and revise its rendered output [7, 8, 9, 10].
We address these limits with three components. Our contributions are:
a five-layer verification framework for design taste with an admission protocol that decides which signals may enter the reward (§ 4);
synthetic design environments adapted from SCALER, with a two-tier generation pipeline, seven families, perturbation operators and difficulty ladders calibrated from measured pass rates, scheduled by an adaptive curriculum with a quarantine rule (§ 5);
an agentic training formulation with visible development checks and hidden graders (§ 6);
TD-Bench, an evaluation with held-out families and blinded expert comparisons, and experiments against a single-turn control and a prompt-optimised agent (§ 7 and § 8).
Related Work
Evaluating and rewarding interface design. Design2Code measures fidelity to a reference screenshot [2]. ArtifactsBench scores rendered artefacts with a checklist-guided multimodal judge and reports over 90% pairwise agreement with experts [11], UI-Bench fits TrueSkill ratings to 4,075 blinded expert comparisons [12], and UIClip is a pretrained quality scorer [13]. For non-verifiable domains, Rubrics as Rewards uses instance-specific weighted rubrics as the GRPO reward [14]. Model judges show position and verbosity biases [15], and optimising any learned proxy eventually degrades true quality [16]. Relative formulations, such as pairwise visual comparison in UI2Code [8] and in-group win rates in Pref-GRPO [17], reduce calibration noise.
Render-in-the-loop generation. AesCoder trains a 4B model with GRPO on execution and aesthetic rewards [9]. ReLook gives zero reward to pages that fail to render and accepts a revision only if a critic score improves [7]. Vision-guided refinement improves frontend output by up to 17.8% over three critique cycles [10], and 1D-Bench defines round-level rewards for agentic interface editing [18].
Environments, curricula and agentic RL. Reasoning Gym and RLVE provide procedurally generated environments with tunable difficulty [19, 20]. SCALER converts problems into environments with explicit scale parameters, validates their generators, adjusts difficulty online towards a target accuracy and retires stagnant environments [1]. GRPO removes the value network by normalising rewards within a group [21], SkyRL-Agent scales multi-turn rollouts asynchronously [22], and GEPA shows that prompt optimisation is a strong and inexpensive baseline [23].
Learning taste. Reinforcement Learning from Community Feedback trains a judge of research impact and uses in-group win rates as the reward for an ideation policy [24]. TASTE finds that designer agreement is higher for rule-based criteria, such as typography, than for colour harmony [25], and Taste-Bench measures agent taste as choosing the better direction at decision forks [26]. Like these studies, we validate the reward against human judgement and use relative comparison where absolute scores are unreliable. Unlike them, we verify taste in layers, with deterministic render metrics carrying the parts on which experts agree.
Problem Formulation and Overview
A task instance is a triple : a natural-language instruction (a symptom description, a holistic request or a structured brief), an initial repository state (a complete page, a broken page or an empty scaffold) and a hidden grader configuration . A policy interacts with a sandbox for at most turns, emitting tool calls or a submission and receiving observations. The production bundle of the submitted state is scored by graders the policy cannot observe, giving a terminal reward for episode .
Figure 1. Overview. (1) Agentic generation of base pages with best-of- selection. (2) The five verification layers and the admission protocol that determines . (3) Environments that emit instances at a controlled level of a measured difficulty ladder. (4) Multi-turn agentic RL with hidden graders and group-relative advantages. (5) Oversight calibrates admission against experts, quarantines environments whose judge reward diverges from the canary, and converts failures into new rubric items and operators.
Following RLVE and SCALER [1, 20], an environment is a parameterised problem space with an explicit difficulty ladder,
where is the instruction template and tool set, a generator that maps a seed and level to an instance, the verifier and the maximum level. Training maximises
where the active environment set and level distribution are adapted during training (Section 5.1). We do not assume a single verifier of design quality. We use signals of increasing fidelity and decreasing reliability, and only candidates in the admitted set , validated against the expert layer, contribute to .
Figure 1 shows the system. The substrate is the webapp-synth repository, which provides a seed taxonomy, deterministic checks, a REDGREEN admission gate and a sandboxed agent harness [6]. Component A decides which signals measure improvement, Component B turns them into environments with difficulty ladders, and Component C trains the policy. Policy samples and failures return to Component A as expert labels, rubric items and perturbation operators.
Component A: Layered Verification
Some parts of design quality follow explicit principles and can be computed from a rendered page, such as text contrast, adherence to a spacing scale and the absence of horizontal overflow on narrow viewports. Others need judgement but can be split into checkable questions, and some appear only under comparison. We therefore verify in five layers (Figure 2). hard gates (build, render without errors, anticheat, page volume, functional contracts) are binary and multiply the reward. render metrics, rubric judges and relative preference contribute graded terms. , a blinded expert panel, never enters the reward but decides which lower-layer signals may. Rendering uses headless Chromium on the production bundle, since and above need computed layout.
Figure 2. Layered verification for one rollout of a brief-to-page (F5) instance. The gates pass and 13 of 14 contract tests succeed (); six render metrics give ; admitted rubric items give ; an in-group tournament gives . The expert layer decides admission and scores the canary but is not a reward term. With F5 weights, Eq. (4) gives , above the 0.4 floor.
Render metrics (). Each metric maps the render capture at viewport (DOM, computed styles, box model and accessibility tree) to and is averaged over viewports px and interaction states. The library covers text contrast (WCAG AA [27]), spacing-scale conformity, proximity, type-scale coherence, hierarchy integrity, primary-action salience, line length, alignment, palette discipline, component consistency, responsive integrity, accessibility, performance and token hygiene. Every metric reads computed styles rather than class names, and each has a recorded exploit and counterweight; for example, collapsing all spacing to one value raises scale conformity but lowers the proximity metric. Two further signals target generic output: a default-tell detector for common defaults, and the slop distance, the Mahalanobis distance of a page’s style signature from the centroid of base-model pages. Neither measures quality, so both are used only in the art-direction family.
Rubric judges and preference (, ). Following RaR [14], each brief receives a weighted rubric over ten dimensions (hierarchy, layout, typography, colour, responsiveness, accessibility, copy, conversion, direction, craft) with negative-weight pitfall items. A bound item is scored by its metric, so judges never re-estimate a measurable quantity; a judged item is scored by the ensemble only if its template is in . The ensemble has a code-and-metrics judge and a screenshot judge from model families other than the policy’s, and a third judge is withheld for evaluation. Comments and unrendered strings are stripped before judging, and pairs are judged in both orders [15]. Within a GRPO group of , a Swiss schedule of 12 comparisons replaces the 28 of a round robin, and is the win rate of rollout over its scheduled comparisons.
Expert panel (). Three to five senior designers give blinded forced-choice judgements, item scores and short rationales, with catch trials and repeats. Reliability is reported as Krippendorff’s [28]. Estimating an agreement rate within at 95% confidence needs pairs per dimension.
Admission protocol
A candidate enters only if it passes four tests. Let be the golden pairs , with expert preference , on which the metric difference exceeds its noise floor. Agreement is
Causal sensitivity requires that on at least 90% of single-principle minimal pairs the score drops after a perturbation and recovers after repair. Hack-trap resistance requires that the metric prefers a page built to game it over an expert-preferred page on at most 20% of trap pairs. Stability requires at least 0.90 self-agreement across render seeds, viewport jitter and repeated judge calls. Failing items are kept for evaluation only, and admitted items are re-tested whenever the golden set or the judges change.
Composite reward
For a finished episode,
with weights summing to one. The gate is the product of build, render, anticheat, volume and no-regression checks; the last gives zero reward if a repair degrades any other admitted metric of the base page, so deleting a defective section is never the cheapest fix. is the vitest pass fraction [6], the weighted mean of admitted metrics (restricted in repair families to the metrics affected by the planted defects), the score over admitted judged items, and , penalise page inflation and extra turns. Two rules from webapp-synth apply to every family: integrity is guarded by gates and never by the graded score, and the unmodified starting state must score at most 0.4 (the reward floor). Instructions describe symptoms, never the fix.
Component B: Synthetic Design Environments
SCALER treats an environment, a program that yields unlimited verifiable instances of one problem type at a chosen difficulty, as the unit of training data [1]. In webapp-synth, by contrast, each instance needs a 20 to 30 minute agentic generation session and carries an unmeasured difficulty label [6]. We keep the expensive generation step but run it once per base page, and generate instances with inexpensive programs.
Two-tier generation. Tier 1 runs offline: agentic coding sessions write long-form pages from sampled briefs, the validated judge selects one of several candidates per brief, and selected pages that pass the build, the page audit, a baseline and a human spot check enter a base-page pool. Tier 2 runs online: given a base page and a seed, a program samples perturbation operators and severities, applies them as syntax-tree and stylesheet edits, normalises formatting so the diff does not reveal edit locations, writes a symptom-only instruction and emits the hidden grader configuration in seconds (Figure 3). Generation families use a constraint sampler instead. This split matches the cost asymmetry: a pool of several hundred base pages is affordable, tens of thousands of agentic sessions are not.
Figure 3. Two-tier environment synthesis. Tier 1 (offline) produces judge-selected base pages from briefs. Tier 2 (online) samples a base page and instance parameters (seed , level , operators), applies the transforms and emits an instance . The inset gives the measured pass rate of the base policy at each level; the worked instance is at level 3.
Families and operators. Seven families target complementary aspects of design (Table 1). Page-editing families (F1 to F4, F7) have a well-defined do-nothing floor and mostly deterministic rewards; open-ended families (F5, F6) rely more on judges and relative preference. Fourteen perturbation operators, such as spacing drift, contrast degradation, hierarchy scramble, action dilution and breakpoint removal, each break one principle and are each linked to the metric that detects it. Hidden graders test the violated invariant rather than restoration of the original source, so any correct repair receives full credit. F7 scores improvement over the starting page and accepts a round’s edits only if the hidden score rises [7].
| Family | Task | Difficulty parameters | ||||
| F1 Interaction repair | Repair a broken interaction rule on a long-form page | Coupled seeds, localisation depth | 1.0 | |||
| F2 Design polish | Locate and repair planted design defects | Number, subtlety and coupling of defects | 0.8 | 0.2 | ||
| F3 System adherence | Add a component using only given tokens and specifications | Token-set size, component novelty | 0.2 | 0.6 | 0.2 | |
| F4 Responsive adaptation | Make a desktop-only page correct at every viewport | Affected sections, layout complexity | 0.2 | 0.8 | ||
| F5 Brief to page | Build a page from a brief with explicit constraints | Constraint count, conflicting requirements | 0.1 | 0.3 | 0.3 | 0.3 |
| F6 Art direction | Build a page that commits to a named aesthetic direction | Specificity of direction | 0.2 | 0.4 | 0.4 | |
| F7 Critique revision | Improve a page over several rounds from a critique | Critique specificity, rounds | 0.4 | 0.6 |
Table 1. Environment families, their difficulty parameters and initial reward weights (Eq. (4)). The weights keep every family’s reward floor at or below 0.4.
Generator validation and difficulty. Because Tier 2 generators are partly written by an agent, each must pass SCALER-style checks [1, 29] before entering the pool: the RED→GREEN gate (the instance fails a hidden check, the base page passes all), breadth (two independent correct repairs both pass), depth (instances at one level differ in target elements), the reward floor, volume bounds, absence of leakage, and severity monotonicity. Design has no single scale parameter, so levels are defined empirically: the base policy is run on a grid of operator subsets and severities, and a monotone fit maps grid cells to levels with roughly uniform steps in pass rate.
Adaptive curriculum
An episode is a success if it passes every gate and its reward exceeds a family threshold . With success rate over the rollouts of environment at step , the difficulty is updated as
and the instances of a step are split between the two nearest integer levels so that their mean approximates [1]. A target maximises the chance that a group contains both successes and failures, and hence a non-zero advantage. The outer loop keeps an active set of fixed size and, as in SCALER, retires an environment temporarily when its difficulty slope over 10 steps is not positive, its success rate is zero for 5 steps, or its difficulty stays at for 5 steps. We add one rule for learned rewards: an environment is quarantined for human review when its judge-derived reward rises while its canary score (expert or held-out judge) stays flat over steps,
since rising proxy reward with flat true quality is the typical sign of over-optimisation [16]. Figure 4 shows both loops.
Figure 4. Adaptive curriculum. (a) Outer loop: each line is the difficulty of one environment over training. F2 and F3 are still rising and stay active; F4 stagnates and is retired temporarily; F5’s judge reward rises while its canary score stays flat, so it is quarantined. (b) Inner loop for one F2 environment (, , ): the success rate of each group moves the difficulty by Eq. (5).
Held-out environments. One complete family (F4), two operator types, one industry and a set of designer-authored briefs are excluded from training. A policy that has learned to invert specific operators, rather than to perceive design problems, will show a large gap on them.
Component C: Agentic Reinforcement Learning
A single-turn policy must emit a finished page without seeing it, whereas design practice is iterative, and models benefit from observing and revising their output [7, 10]. Multi-turn episodes also make the repair families possible, since they are operations on a stateful repository. The costs are sandbox time, harder credit assignment and malformed tool calls from a small model, so we keep a single-turn control and a prompt-optimised agent as baselines.
Episodes and tools. The observation holds the instruction, the repository, any design tokens and a budget of 20 turns (Figure 5). The policy can read, search and edit files, build, call render_report(v) for a textual outline of the render at viewport (headings with computed sizes, per-section layout, overflowing or clipped elements), run public functional tests, and call dev_lint. The render report describes what is present, not whether it is good. The key design decision is how much of the verifier to expose. Following the convention of public and hidden tests, dev_lint covers a rotating subset of metrics at relaxed thresholds, while hidden graders cover all admitted metrics at brief-specific thresholds together with the judges. We monitor the visible–hidden gap on shared metrics; a rising shows that the policy is fitting the visible checks.

Figure 5. An agentic episode. (1) The policy sees a symptom-only instruction, the repository, design tokens and its budget, but not the grader. (2) It locates the component, inspects the render report at 375 px, edits spacing and colour classes, rebuilds, runs the linter and submits. (3) Hidden graders score the bundle; with F2 weights, . (4) Rewards in the group become advantages (Eq. (7)) for the policy update.
Objective. For rollouts of an instance, advantages are group-normalised [21],
and the policy maximises the clipped surrogate with a token mask that excludes tool outputs from the loss, plus a small KL penalty to a reference policy, kept because the reward includes learned judges. We use a single terminal reward per episode: it is the hardest to exploit, and potential-based shaping [30] telescopes into an end-of-episode term under trajectory-level GRPO. The policy is a 4B code model (Qwen3.5-4B) with a short warm-up on tool-use traces replayed from the 48 existing brief–page pairs.
System. Rendering and judging run outside the agent sandbox. The policy builds inside a prebaked sandbox image (up to 256 concurrent), which also runs the functional tests. The production bundle is sent to a separate render and reward service with a pool of headless browsers, the metric library and the judge ensemble, which caches results by bundle hash. This keeps the sandbox small, lets renders be batched and cached across rollouts, and makes the hidden graders unreachable from the policy. Monitors read the metric stream and can quarantine an environment through the curriculum controller. Cost is accounted in sandbox-minutes per solved instance.
Experimental Setup
Hypotheses. H1: a measurable share of expert preferences is predictable from render metrics, with the share varying by dimension. H2: adaptive environments keep the learning signal more informative than a static task set. H3: at matched compute, multi-turn agentic RL achieves a higher expert win rate than single-turn training, and part of the gain persists without tools. H4: the layered reward keeps the divergence between reward gain and expert gain small. H5: a code-and-metrics judge agrees with experts nearly as well as a screenshot judge, except on composition.
Experiments. E1 applies the admission protocol to every candidate signal (H1). E2 measures the share of informative GRPO groups (H2). E3 compares training methods at matched compute (H3). E4 measures robustness diagnostics (H4). E5 compares the two judge types per dimension (H5). E6 removes tools at inference (H3). E7 measures distinctiveness across briefs.
Benchmark. TD-Bench extends MiniLandingBench [5] with four splits: held-out instances of training families (S1, in-family); held-out families, operators and industry (S2, held-out); tool-free single-shot generation from designer briefs; and an external slice from OpenDesign [9] and ArtifactsBench [11].
Systems. All systems use the same 4B policy, harness, briefs and render service, are matched in sandbox-minutes and run with three seeds. BASE is the untrained model in the agentic harness. SFT-C is the single-turn control: SFT on the 48 long-form pages, DPO on golden pairs [31] and a short single-turn GRPO stage. GEPA is the base agent with a system prompt evolved by GEPA [23], testing whether gains need training. ARL is our method.
Metrics. The primary metrics are gate pass rate (P1), functional contract rate (P2), four composites of admitted and evaluation-only render metrics (legibility P3, system coherence P4, hierarchy and composition P5, render robustness P6), rubric score (P7), brief-constraint retention (P8), blinded expert win rate (P9) and distinctiveness (P10). The Taste Index (TI) is the unweighted mean of P3 to P8; expert win rate is kept separate as ground truth. Win rates carry Wilson 95% intervals. A two-sided test at with power 0.8 needs about 194 comparisons to detect a win rate of 0.60 and 783 for 0.55, so we allocate 400 expert comparisons for ARL against BASE on S2, 800 against SFT-C, and 200 each against GEPA and in-family.
Success criteria. Fixed before the experiments, on held-out splits: an expert win rate of at least 0.60 against BASE and 0.55 against SFT-C; no loss in gate pass rate or cross-brief diversity relative to BASE; reward–expert divergence at most 0.05; held-out gains at least half of in-family gains; and no unresolved quarantine events.
Results
What is verifiable (E1, E5)
The admission protocol admits 7 of the 16 render signals and 6 of the 10 rubric dimensions (Figure 6). The admitted signals (contrast, spacing scale, type scale, component consistency, hierarchy, responsiveness and accessibility) are tied to explicit standards or to defects that designers name consistently. Line length, palette discipline, action salience, proximity and alignment fall short on agreement or exceed the trap ceiling, and token hygiene fails because experts judge pixels rather than class names. Copy, conversion and craft remain evaluation-only, and direction could not be tested with only 240 golden pairs. Composition is the largest verification gap: of its four signals, only hierarchy integrity enters the reward. The judge comparison supports H5. The screenshot judge agrees with experts clearly better only on layout () and direction (), the code-and-metrics judge is better on accessibility (), where roles and focus order are invisible in pixels, and on the other seven dimensions the two differ by at most .
Figure 6. Admission outcomes (E1). Each row is a candidate signal, with its agreement with expert pairs (left) and hack-trap preference rate (right). Shaded regions satisfy the thresholds (agreement at least 0.85, trap rate at most 0.20). Blue points are admitted, grey points are evaluation-only, and the open point was not tested.
Agentic training against the controls (E3)
In-family, ARL raises the Taste Index from 0.630 to 0.795, against 0.689 for SFT-C and 0.690 for GEPA (Table 2). The largest gains are on metrics carried by admitted signals and heavy family weights: gate pass rate (0.70 to 0.93), functional contracts (0.52 to 0.81) and legibility (0.74 to 0.88). The smallest are on hierarchy and composition (0.61 to 0.77) and rubric score (0.52 to 0.67), which depend on evaluation-only signals and noisy judged items. On the held-out split, ARL wins 0.600 of expert comparisons against BASE (Wilson interval 0.551 to 0.647, ), exactly at the threshold, and 0.569 against SFT-C (0.534 to 0.603, ), which meets the 0.55 criterion (Figure 7). The contrast with GEPA (0.55, ) is reported descriptively.
Figure 7. Expert win rates of ARL with 95% Wilson intervals. Vertical ticks mark the success thresholds and the dashed line marks chance.
| Metric | In-family (S1) BASE | In-family (S1) SFT-C | In-family (S1) GEPA | In-family (S1) ARL | Held-out (S2) BASE | Held-out (S2) SFT-C | Held-out (S2) GEPA | Held-out (S2) ARL |
| Gate pass rate (P1) | 0.70 | 0.79 | 0.78 | 0.93 | 0.68 | 0.73 | 0.74 | 0.82 |
| Functional contracts (P2) | 0.52 | 0.62 | 0.58 | 0.81 | 0.48 | 0.53 | 0.53 | 0.66 |
| Legibility (P3) | 0.74 | 0.80 | 0.79 | 0.88 | 0.73 | 0.76 | 0.77 | 0.82 |
| System coherence (P4) | 0.66 | 0.73 | 0.69 | 0.83 | 0.64 | 0.68 | 0.67 | 0.75 |
| Hierarchy & composition (P5) | 0.61 | 0.67 | 0.64 | 0.77 | 0.59 | 0.62 | 0.62 | 0.69 |
| Render robustness (P6) | 0.70 | 0.74 | 0.74 | 0.86 | 0.60 | 0.62 | 0.63 | 0.70 |
| Rubric score (P7) | 0.52 | 0.58 | 0.60 | 0.67 | 0.50 | 0.53 | 0.57 | 0.60 |
| Constraint retention (P8) | 0.55 | 0.61 | 0.69 | 0.75 | 0.53 | 0.56 | 0.65 | 0.65 |
| Distinctiveness (P10) | 0.48 | 0.44 | 0.55 | 0.45 | 0.47 | 0.45 | 0.53 | 0.45 |
| Taste Index (TI) | 0.630 | 0.689 | 0.690 | 0.795 | 0.598 | 0.631 | 0.650 | 0.700 |
| Win rate vs BASE (P9) | n/a | 0.559 | 0.560 | 0.659 | n/a | 0.532 | 0.551 | 0.601 |
Table 2. Primary metrics on the in-family (S1) and held-out (S2) splits. The best value in each split is in bold. P9 is the expert win rate against BASE.
Generalisation and the role of the loop (E6)
ARL keeps 62% of its in-family gain on held-out families, above the required half (Figure 8a). The advantage depends on the loop: with tools removed, ARL keeps only 35% of its gain (TI 0.654) and matches SFT-C (0.653), which was trained single-turn. Part of the gain therefore persists, as H3 states, but agentic training does not give a better tool-free policy. On the external slice ARL remains ahead (0.647 against 0.610 to 0.618). By family (Figure 8b), ARL gains most where the reward is deterministic, design polish (0.55 to 0.85) and system adherence (0.64 to 0.84), and the held-out responsive family reaches 0.75.
Figure 8. (a) Taste Index on the four TD-Bench splits. (b) Taste Index by environment family; F4 is held out.
Robustness, curriculum and cost (E2, E4)
The divergence between reward gain and expert gain stays within the 0.05 bound in-family and on held-out families (0.040 and 0.045) and exceeds it only on the external slice (0.06) (Figure 9a). The visible–hidden gap reaches 0.06 in-family because dev_lint is visible throughout training, and no-regression violations reach 0.06 under held-out operators. A separate check measures the reward a policy earns without solving the task (Figure 9b). Three families exceed the 0.4 floor: in design polish (0.41) perturbed metrics remain partly high, in responsive adaptation (0.48) an untouched desktop page keeps its functional contracts, and in critique revision (0.46) an unchanged page can outrank worse revisions. Scoring as the repaired fraction, restricting in F4 to responsive contracts, and scoring F7 by improvement remove these violations.
Figure 9. (a) ARL diagnostics divided by their bounds; points in the shaded region exceed the bound. (b) Reward earned without solving the task, per family; red bars exceed the 0.4 floor.
The adaptive curriculum keeps 0.86 of GRPO groups informative, with both successes and failures, against 0.55 for the static-data GRPO stage of SFT-C, which supports H2 for the learning signal; its effect on expert win rate was not isolated. ARL is also the least costly system per solved instance, 6.9 sandbox-minutes against 14.7 for BASE, because its higher success rate outweighs its longer episodes. Brief-constraint retention rises from 0.55 to 0.75, but constraints that must survive an edit are lost twice as often as explicit ones, and conflicting requirements are flagged in only about half of briefs.
Distinctiveness and external comparison (E7)
Distinctiveness is the only success criterion ARL does not meet (Table 3). Slop distance and default tells measure departure from generic output rather than quality, so they cannot pass the agreement test, and direction has too few golden pairs. Without these counterweights, 80% of the F6 reward comes from judge terms that favour the judges’ central taste. On the held-out split, ARL’s distinctiveness falls from 0.47 to 0.454, while GEPA, whose prompt asks for an explicit direction, rises to 0.532. On OpenDesign, ARL scores 78.4 against 75.0 for BASE, seventh of sixteen published systems and 3.5 points below AesCoder-4B, which is trained with a screenshot reward on similar data [9].
| Criterion | Target | Result | Outcome |
| Expert win rate against BASE | 0.600 (0.551–0.647) | Pass, no margin | |
| Expert win rate against SFT-C | 0.569 (0.534–0.603) | Pass | |
| Gate pass rate | BASE (0.68) | 0.82 | Pass |
| Cross-brief diversity | BASE (0.47) | 0.454 | Fail |
| Reward–expert divergence | 0.045 | Pass | |
| Held-out gain in-family gain | 0.62 | Pass | |
| Unresolved quarantine events | 0 | 0 | Pass |
| Criteria met | 6 of 7 |
Table 3. Success criteria applied to the held-out (S2) results.
Discussion and Limitations
Discussion and Limitations
The verifiability map agrees with recent work on designer preference. TASTE reports higher designer agreement for typography and spatial accuracy than for colour harmony [25], and in our results the signals tied to explicit rules pass admission while composition and direction do not. Taste-Bench finds that judgement errors concentrate where the deciding evidence appears late in a trajectory [26]; the dependence of our gains on the render loop is consistent with this, since the loop lets the policy see the effect of its edits before submitting. The results point to four changes: admit distinctiveness terms through a direction-adherence test rather than expert agreement, which addresses the unmet criterion; rescore F2, F4 and F7 so every family respects the reward floor; rotate the dev_lint subset per epoch to reduce ; and add a screenshot-judged layout item to close part of the composition gap.
Limitations. The rubric and metric library may encode a single notion of taste; we keep label distributions where expert agreement is low and condition items on the requested direction. Base pages come from one agentic coder, so judges are drawn from other model families and one judge is held out. The scope is long-form landing pages in React and Tailwind; transfer to dashboards, multi-screen flows or native interfaces is not established. The primary win rate against BASE sits exactly at its threshold and should be read with its interval. The policy is text-only, only one size (4B) was trained, and neither the curriculum’s effect on expert win rate nor scaling with the number of environments was measured.
Conclusion
We presented a method in which a small code model learns design taste by reinforcement learning in environments that make taste checkable in layers. Render metrics and rubric judges enter the reward only through an admission protocol validated against experts, environments are generators with measured difficulty scheduled by a curriculum that also guards against over-optimisation, and the policy learns through multi-turn episodes scored by graders it cannot observe. About half of the candidate signals can be verified, with composition the largest gap. Agentic training improves every primary metric except distinctiveness, transfers to held-out families, and meets the expert win-rate criteria against the base model and the single-turn control, although most of the gain depends on the inference-time loop. Admitting distinctiveness through direction adherence and closing the composition gap are the main directions for future work.
References
- [1]C. Xu, C. Xiao, Z. Peng, X. Wang, and Y. Cao. SCALER: Synthetic scalable adaptive learning environment for reasoning. arXiv preprint arXiv:2601.04809v5, 2026.arxiv.org/abs/2601.04809
- [2]C. Si, Y. Zhang, R. Li, Z. Yang, R. Liu, and D. Yang. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), pages 3956–3974, 2025.arxiv.org/abs/2403.03163
- [3]J. Xiao, M. Wang, M. H. Lam, Y. Wan, J. Liu, Y. Huo, and M. R. Lyu. DesignBench: A comprehensive benchmark for MLLM-based front-end code generation. arXiv preprint arXiv:2506.06251, 2025.arxiv.org/abs/2506.06251
- [4]Tasteful Designs Project. Teaching LLMs UI design taste: Post-training, RL alignment, agentic RL, synthetic environments, design-taste evaluation and meta-harnesses. Internal research report, Tasteful Designs Project, September 2026.
- [5]Tasteful Designs Project. Developing taste for design in models: Rubric-grounded golden preferences and reinforcement post-training of a small LLM for generating better design. Internal pilot design, Tasteful Designs Project, June 2026.
- [6]Tasteful Designs Project. webapp-synth: Architecture and operations guide, and README. Repository documentation, https://github.com/mr-siddy/Tasteful-Designs, 2026.
- [7]ReLook authors. ReLook: Vision-grounded RL with a multimodal LLM critic for agentic web coding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2026. arXiv:2510.11498.
- [8]Z. Yang, W. Hong, et al. UI2Code^N: UI-to-code generation as interactive visual optimization. arXiv preprint arXiv:2511.08195, 2025.
- [9]B. Xiao, L. Jiang, S. Huang, et al. Code aesthetics with agentic reward feedback. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.23272.arxiv.org/abs/2510.23272
- [10]Vision-guided refinement authors. Vision-guided iterative refinement for frontend code generation. arXiv preprint arXiv:2604.05839, 2026.
- [11]Tencent Hunyuan. ArtifactsBench: Bridging the visual-interactive gap in LLM code generation evaluation. arXiv preprint arXiv:2507.04952, 2025.
- [12]S. Mateega et al. UI-Bench: A benchmark for evaluating design capabilities of AI text-to-app tools. arXiv preprint arXiv:2508.20410, 2025.arxiv.org/abs/2508.20410
- [13]J. Wu et al. UIClip: A data-driven model for assessing user interface design. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), 2024.arxiv.org/abs/2404.12500
- [14]A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.arxiv.org/abs/2507.17746
- [15]L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.05685
- [16]L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning (ICML), 2023. arXiv:2210.10760.arxiv.org/abs/2210.10760
- [17]Y. Wang, Z. Li, Y. Zang, Y. Zhou, J. Bu, C. Wang, Q. Lu, C. Jin, and J. Wang. Pref-GRPO: Pairwise preference reward-based GRPO for stable text-to-image reinforcement learning. arXiv preprint arXiv:2508.20751, 2025.arxiv.org/abs/2508.20751
- [18]1D-Bench authors. 1D-Bench. arXiv preprint arXiv:2602.18548, 2026.
- [19]Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760, 2025.arxiv.org/abs/2505.24760
- [20]Z. Zeng, H. Ivison, Y. Wang, L. Yuan, S. S. Li, Z. Ye, S. Li, J. He, R. Zhou, T. Chen, et al. RIVE: Scaling up reinforcement learning for language models with adaptive verifiable environments. arXiv preprint arXiv:2511.07317, 2025.arxiv.org/abs/2511.07317
- [21]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [22]SkyRL team. SkyRL-Agent: Efficient RL training for multi-turn LLM agents. arXiv preprint arXiv:2511.16108, 2025.
- [23]L. A. Agrawal et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (ICLR), 2026. arXiv:2507.19457.arxiv.org/abs/2507.19457
- [24]Jingqi Tong, Mingzhe Li, Hangcheng Li, Yongzhuo Yang, Yurong Mou, et al. AI can learn scientific taste. arXiv preprint arXiv:2603.14473, 2026.arxiv.org/abs/2603.14473
- [25]Haonan Zhu, Elad Hirsch, Alexandria Minetti, Allison Nulty, Ben Huffman, and Purvanshi Mehta. TASTE: A designer-annotated multi-dimensional preference dataset for AI-generated graphic design. arXiv preprint arXiv:2605.20731, 2026.arxiv.org/abs/2605.20731
- [26]Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, and Xiaohua Jia. The tasteful agent: Measuring and improving taste in long-horizon tasks. arXiv preprint arXiv:2609.25804, 2026.arxiv.org/abs/2609.25804
- [27]World Wide Web Consortium. Web content accessibility guidelines (WCAG) 2.2. W3C Recommendation, 2023.
- [28]K. Krippendorff. Content Analysis: An Introduction to Its Methodology. SAGE Publications, 4th edition, 2018.
- [29]J. Fu, X. Yang, H. Zhang, Y. Liu, J. Zhang, Q. Wang, F. Zhang, and G. Zhou. Klear-CodeTest: Scalable test case generation for code reinforcement learning. arXiv preprint arXiv:2508.05710, 2025.arxiv.org/abs/2508.05710
- [30]A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), 1999.
- [31]R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.18290







