RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

Introduction

Language models can supply rewards for their own iterative preference training (Yuan et al., [44]). They can also improve their judgment skills by assessing their own judgments (Wu et al., [37]). The corresponding question for semantic training rubrics is how revising their behavioral requirements affects learned behavior, including when a revision yields gains beyond continuing training with the existing rubric.

Written rubrics specify desired response properties and can supply reward signals (Gunjal et al., [12]) or guide the construction of synthetic preference data (Gallego, [11]). EvoRubrics jointly trains a policy and a rubric generator to adapt rubric-based rewards as learning proceeds (Ding et al., [9]). SERPO co-evolves response evidence, query-specific rubrics, and policy parameters, using rubric scores as rewards for test-time reinforcement learning (Wang et al., [32]). We study rubric revision in Direct Preference Optimization (DPO) (Rafailov et al., [25]), holding the pairs, labels, and objective fixed while changing the rubric context. A matched continuation with the existing rubric measures the edit’s incremental contribution.

We introduce RubricOpt to study how a model revises its own training rubric across preference-training rounds (Figure 1). Fine-grained evaluation describes response quality through explicit skills (Ye et al., [43]). We use a fixed eight-factor ontology to connect the model’s task failures, the contrasts in its preference data, and the rubric’s Prefer/Avoid clauses. Diagnosis-guided self-evolution uses failure histories to direct subsequent learning (Wei et al., [36]). RubricOpt applies this diagnostic idea to rubric revision, with the current policy reading task outcomes, self-judged factor scores, and judge feedback on its own outputs to attribute recurring failures to the eight factors. It retains explicit requirements for further learning and proposes clause edits where the wording leaves recurring failures uncovered. A deterministic gate checks evidence references, factor anchors, and edit size before carrying the rubric into the next round. Training-only Semantic Rubric Steering uses the resulting rubric to condition both responses of a preference pair during DPO [25], with the pairs, labels, and objective held fixed. Its exposure decreases during training, and evaluation uses the resulting policy without the rubric. This use of training-only information has a precedent in privileged-information learning [31]. Our analysis relates the extra update induced by a rubric to rubric-free success through its alignment with task demand (§ 3.3).

The experiments evaluate how changes to the training specification affect comprehensive capability and open-ended generation. Fixed-state probes and criterion comparisons show that rubric text changes the learning update and can alter rubric-free outcomes (§ 4.1). Across model families and preference corpora, rubric conditioning improves some capability and generation evaluations over ordinary DPO, with gains and regressions varying by setting (Table 1 and Table 2). The dataset comparison relates these effect profiles to the behavioral contrasts available in each corpus (§ 4.2). A matched continuation separates revision from further optimization and finds mixed incremental results from the tested edit (Table 3).

Method	GSM8K	MATH-Hard	BBH	MuSR	ARC	Truthful QA	MMLU-Pro	Avg.	MT-Bench	Arena-Hard (%)	WildBench	Avg. $\Delta$
Qwen3-4B
Instruct	85.90	62.24	12.82	43.39	62.80	52.27	41.98	51.63	6.320	49.0	3.730	−2.96
Ordinary DPO	84.53	39.15	15.43	42.59	61.09	52.71	41.88	48.20	6.890	50.9	3.617	0.00
RUBRICOPT-R1	86.81	43.35	19.64	43.12	61.35	53.74	42.10	50.02	6.930	53.0	3.727	+2.58
RUBRICOPT-R2	80.74	37.12	22.73	42.06	59.47	55.16	40.43	48.24	—
Qwen3-8B
Instruct	91.90	70.13	37.43	54.90	94.10	78.20	67.60	70.61	7.410	64.5	4.074	−0.05
Ordinary DPO	91.80	—	38.66	55.20	94.65	76.80	67.35	—	7.219	64.2	4.211	0.00
RUBRICOPT-R1	91.40	72.77	48.26	55.15	95.00	77.85	68.30	72.68	7.416	65.7	4.329	+2.62

Method	GSM8K	MATH-Hard	BBH	MuSR	ARC	Truthful QA	MMLU-Pro	Avg.	MT-Bench	Arena-Hard (%)	WildBench	Avg. $\Delta$
Instruct	45.75	19.65	36.95	52.45	79.45	58.75	30.05	46.15	5.090	18.1	2.336	−8.44
Ordinary DPO	44.45	20.90	40.40	50.95	76.50	53.90	30.00	45.30	4.850	24.0	2.477	0.00
RUBRICOPT-R1	47.90	21.50	32.45	51.85	75.60	55.40	32.15	45.26	5.140	25.8	2.480	+4.53

Method	BBH	MATH-Hard	MT-Bench	Arena-Hard	WildBench	Avg. Δ (%)
RubricOpt-R1	14.96	64.18	6.93	53.0	3.727	+2.58
RubricOpt-R2 (KEEP)	18.05	63.82	6.419	54.6	3.781	+1.66
RubricOpt-R2 (REFINE)	17.80	63.72	6.825	54.5	3.836	+4.06

RubricOpt makes the behavioral specification itself an object of policy-driven revision within preference learning. Shared semantic coordinates connect failure diagnosis and preference-data contrasts to editable rubric clauses, while training-only steering connects those clauses to subsequent parameter updates. This closes the loop from learned behavior to the specification of further learning, with the effect of revision evaluated independently of additional training.

Related Work

Preference Optimization and Credit Assignment. The family descends from RLHF, in which a reward model is fit to pairwise human judgements [28] and a policy is then optimised against it at assistant scale [23, 4]. DPO instead fits chosen–rejected pairs with a classification objective [25], whose implicit reward is the log density ratio between the policy and the reference [29, 5]. Later objectives change one component of that recipe: IPO replaces it with a squared loss on the pairwise margin [3], KTO drops the pairing requirement by scoring each response against a reference point [10], and SimPO removes the reference model and normalises the margin by length [22]. Two analyses place the family in a wider frame: one connects these objectives to imitation learning [39], and one disentangles how their update suppresses the rejected response and preserves the chosen one [6]. A complementary line assigns credit inside a response rather than to the response as a whole: TDPO reweights tokens by a forward-KL penalty [45], TIS-DPO reweights them by importance sampling against an estimated weight [20], SEPO selects tokens using a token-level reward estimate [41], T-REG uses a token-level reward as a regulariser [49], and TIDPO guides the update by token importance [42]. RubricOpt operates at a different level: a semantic factor names a behavioral distinction that can span several tokens or reasoning steps, while the pairwise loss itself is left unchanged.

Criteria and Behavioral Diagnosis. Written criteria already enter learning through several interfaces. As reward signals, Rubrics as Rewards scores on-policy responses against a rubric [12]. As data, Configurable Preference Tuning builds rubric-conditioned synthetic preference pairs [11]. And as weights, CoRT contrasts rubric-conditioned with rubric-free replays to form token-level weights [47]. Criterion-conditioned training also has close relatives outside preference optimisation: learning using privileged information trains on a signal that is absent at test time [31]. Context distillation trains on a context and then evaluates without it, introduced as an alignment technique [2] and developed as a standalone method [26]. And classifier-free guidance interpolates between conditioned and unconditioned predictions [14]. On the diagnostic side, fine-grained evaluation describes response quality through interpretable skill sets and error taxonomies—FLASK by grading along skill dimensions [43], EvalTree by profiling weaknesses as a hierarchical capability tree [46], and ErrorMap and ErrorAtlas by charting the failure landscape into error categories [1]—and diagnosis-driven post-training then acts on what those tools find, with DiagEvo feeding diagnosed errors back into self-evolution through a hierarchical error memory [36] and APTER turning expert-grounded rubrics into post-training supervision [33]. RubricOpt uses shared semantic coordinates to connect task diagnosis, preference-data contrasts, and edits to the training rubric.

Self-Improvement and Recursive Feedback. Self-improvement methods differ in the component they revise and how that revision enters subsequent behavior. Self-Refine edits outputs from self-feedback [21], and automatic prompt optimisation edits inference-time text from feedback [24]. In parameter learning, Self-Rewarding uses the model to generate its own reward signals [44], Meta-Rewarding trains judgment skills through a meta-judge [37], and self-play preference optimisation regenerates preferences [38]. Rubric evolution also enters the reward interface through joint policy–generator learning [9] and co-evolving query-specific rubrics during test-time reinforcement learning [32]. RubricOpt places self-revision in the semantic rubric that conditions preference training. The current policy proposes clause edits from feedback about its own behavior, then learns under the resulting rubric in the next round. The initial clause bank, factor ontology, task outcomes, and fixed scoring prompt provide the reference points for revision.

RubricOpt: Method and Analysis

RubricOpt alternates preference learning under a written rubric with revision of that rubric by the resulting policy.

Problem Setup. We study preference optimization on a fixed dataset $\mathcal{D} = \{(x, y^{+}, y^{-})\}$ of chosen–rejected pairs, with a trainable policy $\pi_{\theta}$ and a frozen reference $\pi_{\mathrm{ref}}$ trained by DPO [25],

with $\bar{\ell}(y \mid x) = L^{-1}\log\pi(y \mid x)$ averaged over the $L$ response tokens. The pairs, labels, and objective are held fixed; only the text that conditions a pair may change, which is the degree of freedom RubricOpt introduces.

Behavior is described by a fixed eight-factor ontology $\mathcal{F} = \{f_{1}, \ldots, f_{8}\}$ (appendix § 8.§ 2), on which a blind judge returns one ordinal score per factor, $z_{k}(x,y) \in\{0,\ldots,4\}$ (appendix § 8.§ 3). Every behavioral quantity here is a contrast between two groups of these scores, and only two contrasts are used. Responses to a benchmark $B$ are grouped by the task outcome $o \in\{0,1\}$, and semantic demand is the mean score gap between the groups, measured for one model on one task,

Inside a preference pair the two responses are contrasted by $\Delta z_k = z_k(x, y^+) - z_k(x, y^-)$, and semantic supply summarizes its distribution,

as the mean signed contrast, the contrast coverage, and the share of pairs whose contrast agrees with the label. Demand is a property of the model–task pair and supply a property of the dataset, and both live on the same eight axes.

The artifact RubricOpt edits is a rubric $R_r$, a short natural-language specification whose clauses are keyed to factors, with $r$ indexing rounds and a frozen clause bank supplying the initial $R_0$. For factor group $G \subseteq\mathcal{F}$ the training context is

where the renderer concatenates the current Prefer/Avoid clauses in the fixed factor order and $\varnothing$ denotes the absent criterion. Here $G = \mathcal{F}$ except in the factor-set ablation of § 4.2. A conditioning context is therefore $u \in\{c_r, \varnothing\}$, $\pi_\theta(y \mid x, u)$ is the policy under that context, the main evaluations use $\pi_\theta(y \mid x)$, and § 4.2 tests rubric reinsertion at inference time. Appendix § 8.§ 1 collects the symbols used throughout.

Diagnosis and Conditioning

Shared Behavioral Coordinates. With the coordinates of § 3 fixed, the diagnosis is a measurement on one model and one dataset: we score responses from Qwen3-4B on each benchmark and each HelpSteer3 pair, then read out demand $D_{m,B,k}$ (Eq. (2)) and supply $\mu_k$, $C_k$, $P_k$ (Eq. (3)). We mark demand gaps of at least 0.15 on the 0–4 scale as meaningful and contrast coverage below 0.25 as sparse. Table 7 reports the complete Qwen3-4B demand matrix. Judge checks and length controls appear in appendices A.3 and D.1.

Mean factor-score gap
Factor	GSM8K	MATH-Hard	BBH	MuSR	ARC	TruthfulQA	MMLU-Pro
$f_1$	+0.080	+0.039	+0.023	+0.005	+0.110	+0.100	+0.069
$f_2$	+0.170	+0.246	+0.066	+0.047	+0.248	+0.157	+0.235
$f_3$	−0.051	+0.090	+0.018	+0.025	+0.105	+0.145	+0.142
$f_4$	+0.036	+0.054	+0.028	+0.006	+0.093	+0.150	+0.091
$f_5$	+0.158	+0.486	+0.232	+0.038	+0.148	+0.034	+0.399
$f_6$	+0.086	+0.065	+0.075	+0.010	−0.007	+0.035	+0.038
$f_7$	−0.048	+0.164	+0.235	+0.069	+0.040	−0.020	+0.132
$f_8$	+0.081	+0.162	+0.095	+0.069	+0.179	+0.110	+0.219
Cohen’s $d$
Factor	GSM8K	MATH-Hard	BBH	MuSR	ARC	TruthfulQA	MMLU-Pro
$f_1$	+0.578	+0.326	+0.103	+0.021	+0.680	+0.280	+0.275
$f_2$	+0.756	+0.709	+0.160	+0.123	+0.988	+0.362	+0.485
$f_3$	−0.112	+0.167	+0.035	+0.050	+0.210	+0.288	+0.283
$f_4$	+0.319	+0.313	+0.124	+0.049	+0.570	+0.327	+0.256
$f_5$	+0.588	+1.105	+0.408	+0.069	+0.297	+0.065	+0.700
$f_6$	+0.270	+0.267	+0.200	+0.038	−0.023	+0.081	+0.122
$f_7$	−0.093	+0.315	+0.447	+0.147	+0.090	−0.043	+0.278
$f_8$	+0.222	+0.366	+0.196	+0.136	+0.317	+0.232	+0.387

The measurement separates the factors into Common ($f_2$ reasoning validity, $f_5$ completeness), Specific ($f_4$ factual grounding, $f_7$ clarity, $f_8$ efficiency), and Low-demand ($f_1$ comprehension, $f_3$ verification, $f_6$ instruction adherence). The shared factor ontology links task diagnosis, preference-data contrasts, and clause-level feedback. All eight factors enter the initial rubric from the frozen clause bank. The demand and supply measurements characterize the learning setting; recurring failures whose requirements are incompletely expressed determine subsequent wording edits.

Training-Only Conditioning. Each round fixes $R_r$ and its full-factor context $c_r = \operatorname{render}(R_r, \mathcal{F})$. At step $t$ of $N$, let $\rho_t = t/N$ and independently sample $u_t = c_r$ with probability $q(\rho_t)$ and $u_t = \varnothing$ otherwise for each pair, sharing it across both responses, the policy, and the frozen reference. Exposure is $q(\rho) = 1$ for $\rho\le0.1$, $\{1 + \cos[\pi(\rho- 0.1)/0.8]\}/2$ for $0.1 < \rho\le0.9$, and 0 thereafter. With response-token mean log-probability $\bar{\ell}(y \mid x,u) = \mathcal{L}(y)^{-1}\log\pi(y \mid x,u)$, training uses the unchanged DPO objective

Rubric tokens are masked from the loss and excluded from response-length normalization. Evaluation uses $\pi_\theta(y \mid x)$ without a rubric.

Evidence-Grounded Recursive Refinement

Given rubric $R_r$, the policy trained with its rendered context $c_r$, and item-level benchmark feedback $\mathcal{B}_{r+1}$, one iteration is

Five prompt stages extract benchmark failures, synthesize recurring patterns, attribute them to the eight factors, propose conservative clause updates, and audit the candidate rubric. The policy $\pi_{r+1}$ performs this analysis and proposes the edits itself, using task outcomes, self-judged factor scores, and fixed-judge feedback on its own rubric-free outputs. Attribution compares each recurring failure with the current clauses. An explicit requirement is retained for further learning, while an incompletely expressed requirement is eligible for refinement. A separate deterministic gate checks evidence references, factor anchors, and edit size before accepting a change. A covered requirement or a failed check returns KEEP. The accepted rubric is rendered as $c_{r+1} = \operatorname{render}(R_{r+1}, \mathcal{G})$ for the next training round; KEEP retains the corresponding clauses from $R_r$. Full prompts, stage contracts, and the deterministic acceptance checks are in appendix § 10.§ 2.

Analysis of the Steering Effect

The Update and Its Rubric-Free Effect. Fix one round, with rubric context $c = c_r$ and absent context $0 = \varnothing$. For $s_u(y) = L^{-1}\nabla_\theta\log\pi_\theta(y \mid x,u)$, the per-pair loss gradient is

Let $d_u = s_u(y^{+}) - s_u(y^{-})$ and $v_u = \mathbb{E}_{D}[\kappa_u d_u]$ be the mean descent direction. Independent pair-level context sampling gives $v_q = v_0 + q\Delta v$, where $\Delta v = v_c - v_0$ is the rubric’s extra update direction.

Fix the evaluation prompt law $\nu$, binary success rule $o_B$, and factor scores $\boldsymbol{z} = (z_1,\ldots,z_8)^\top$. Assume finite prompt and response spaces and a positive, continuously differentiable policy. Rubric-free deployment uses $P_\theta(x,y) = \nu(x)\pi_\theta(y \mid x,0)$, with success probability $p = p_B(\theta) = \mathbb{E}_{P_\theta}[o_B] \in(0,1)$. The extra direction changes deployment log-probabilities at rate

Let $D = \mathbb{E}[z \mid o_B = 1] - \mathbb{E}[z \mid o_B = 0]$ be population demand under $P_\theta$, and $a$ the least-squares coefficients of $h$ on centered factor scores. Write $\epsilon_h,\epsilon_o$ for the root-mean-square projection residuals of $h$ and $o_B - p$, respectively; the full construction is in appendix § 9.§ 2.

Demand $D$ describes the behaviors associated with success; $a$ expresses the rubric-induced change in deployment log-probabilities in those semantic coordinates. Their alignment sets the sign of the local success effect when it dominates the residual contribution outside the measured factors. Changing the preference corpus can change the update directions and hence this alignment, even under the same rubric.

For two rendered rubrics $c_0,c_1$ at a shared policy and reference, replacing $\Delta v$ by $v_{c_1} - v_{c_0}$ gives the local edit comparison at a common exposure rate. This comparison motivates the matched KEEP/REFINE continuation in § 4.3, which measures the edit’s incremental effect over further training with the existing rubric.

Experiments

The analysis in § 3.3 organizes three empirical questions: how rubric text changes the training update, how conditioning affects rubric-free outcomes across datasets, and what an edit adds beyond continuing with the existing rubric. We examine them through fixed-state probes and a criterion-identity pilot, dataset-matched ordinary-DPO controls, and matched KEEP/REFINE continuations, respectively.

Models and Datasets. We cover two families and three sizes: Qwen3-4B-Instruct and Qwen3-8B-Instruct from the Qwen3 family ([40]), and Mistral-7B-Instruct-v0.3 from the Mistral family ([15]). Formal runs train a rank-64 LoRA adapter ($\alpha= 128$, dropout 0.05) at learning rate $10^{-6}$, global batch 64, DPO $\beta= 1$. We use two public preference datasets, HelpSteer3 ([35]) and Tulu3 ([16]).

Benchmark Classes. Evaluation covers two benchmark classes, summarized separately. The comprehensive-capability suite is seven accuracy benchmarks—GSM8K ([8]), MATH-Hard ([13]), BBH ([30]), MuSR ([27]), ARC ([7]), TruthfulQA ([19]), and MMLU-Pro ([34])—each with a task check defining the binary success outcome $o_B$ in Prop. 1. The open-ended generative suite is three protocols—MT-Bench ([48]), Arena-Hard ([17]), and WildBench ([18])—on which a judge scores free-form responses and there is no task check. Generative protocols are scored on matched item sets under one judge.

Feedback and Evaluation. The refinement engine reads item-level outputs from first-round rubric-free benchmark evaluations: judge rationales on Arena-Hard, MT-Bench, and WildBench, and correctness feedback on GSM8K, MATH-Hard, and MMLU-Pro. The KEEP and REFINE continuations are then compared on matched evaluation items under the same benchmark protocols. This evaluates a rubric update informed by benchmark feedback, with the unchanged-rubric branch controlling for additional training. Feedback items overlap the final evaluation set, so the continuation comparison measures adaptation to the observed benchmark feedback. The diagnostic/held-out split in § 3.1 assesses measurement stability. Generation, judging, extraction, paired tests, and all absolute scores are in appendix § 11.§ 1.

How Rubric Text Changes Learning

Context-Dependent Updates. In Eq. (7), context enters the update through both the margin-dependent weight $\kappa_u$ and the pair direction $d_u$. A fixed-state probe over 7,253 preference pairs gives conditioned–unconditioned gradient cosines of 0.74–0.79 and relative gradient changes of 0.65–0.72. The likelihood probe measures the semantic sensitivity of the pairwise margin: the $f_5$ completeness criterion’s margin-change slope is 0.0226 nats per response token per unit of $\Delta z_5$, compared with 0.0116 when read against reasoning contrast $\Delta z_2$.

Criterion Identity at Fixed Exposure. Changing the criterion text can change $v_c$ in § 3.3 while preserving the conditioning interface. A 150-step pilot compares reasoning ($f_2$) and clarity ($f_7$) criteria of nearly equal length, using the same pairs and conditioned rows at $q = 0.2$. MuSR pass@4 is 70.8% under the reasoning criterion, 68.6% under the clarity criterion, and 70.8% under ordinary DPO. The paired $f_2$–$f_7$ comparison has 17 prompts passing only under $f_2$ and 6 only under $f_7$ (appendix § 10.§ 1). The two criterion texts produce different rubric-free outcomes on MuSR pass@4, with the reasoning criterion matching ordinary DPO there.

Dataset Contrasts and Rubric-Free Outcomes

The Two Datasets Contrast Different Factors. The shared coordinates of § 3 are what make the two datasets comparable. Figure 3a reports contrast coverage $C_k$ on both, and Table 8 gives the HelpSteer3 statistics behind it. HelpSteer3 leaves three factors sparse against the 0.25 sparsity rule: $f_2$ reasoning validity (0.146), $f_4$ factual and evidential grounding (0.149), and $f_3$ verification and calibration (0.220). Reasoning validity ($f_2$) is meaningful on five of the seven benchmarks but is contrasted in only 14.6% of HelpSteer3 pairs, while completeness ($f_5$) is meaningful on four and contrasted in 47.4%. Tulu3 contrasts seven of the eight factors more often than HelpSteer3 does, and no factor of its eight falls below that rule; its lowest coverage is 0.294. Its largest advantages over HelpSteer3 are on $f_2$ (+0.268), $f_8$ (+0.199), $f_4$ (+0.161), and $f_7$ (+0.135). These profiles describe the frequency of behavioral contrasts on the same axes used for task demand in § 3.1. The full demand matrix is Table 7, and across diagnostic and held-out subsets its largest change is 0.045, with every gap keeping its sign. Figure 3b groups the same gaps by task family: verification and calibration separates success from failure about six times more strongly on the knowledge benchmarks than on the reasoning benchmarks (0.131 against 0.020), and clarity and organization runs the other way (0.051 against 0.105).

Factor	$\mu_k$	$C_k$	$P_k$	Tie rate	Reduction
$f_1$ task comprehension	0.1042	0.3379	0.2118	0.6621	—
$f_2$ reasoning validity	0.0908	0.1463	0.1032	0.8537	13%
$f_3$ verification and calibration	0.0818	0.2202	0.1506	0.7798	—
$f_4$ factual and evidential grounding	0.0866	0.1493	0.0992	0.8507	—
$f_5$ completeness	0.1556	0.4743	0.3037	0.5257	38%
$f_6$ instruction adherence	0.0988	0.3405	0.2046	0.6595	—
$f_7$ clarity and organization	0.1428	0.3950	0.2611	0.6050	47%
$f_8$ efficiency and conciseness	0.0657	0.3248	0.1850	0.6752	—

Demand profiles differ between the two Qwen3 models in a task-dependent way (Figure 4). On BBH, the eight-factor profiles of Qwen3-4B and Qwen3-8B are negatively correlated ($r = -0.751$): completeness and clarity change from positive to negative gaps, while verification and calibration becomes more strongly associated with success. TruthfulQA retains a similar relative profile across the two models ($r = 0.953$), although the gap magnitudes change. These measurements support diagnosing semantic priorities for each model and task.

Conditioning Effects Across Datasets. Table 1 reports the Tulu3 capability results and the HelpSteer3 open-ended results for both Qwen3 models. BBH improves in both Qwen3 models, with effects measured against their own ordinary-DPO controls. On Qwen3-4B, conditioning raises BBH from 15.43 to 19.64 (+4.20) and MATH-Hard from 39.15 to 43.35, and the seven-task average from 48.20 to 50.02. The other five benchmarks move by at most 2.28 points. Qwen3-8B gains 9.61 BBH points. The Mistral-7B Tulu3 run decreases BBH by 7.95 points relative to ordinary DPO (Table 2; appendix § 11.§ 2).

On Qwen3-4B, the four rubric compositions raise Arena-Hard by 1.4–2.2 percentage points relative to ordinary DPO (Table 10). Common+Specific has the highest WildBench point estimate, while the full rubric has the highest MT-Bench score and average relative change across the three protocols.

Method	GSM8K	MATH-Hard	BBH	MuSR	ARC	Truthful QA	MMLU-Pro	Avg.	MT-Bench	Arena-Hard	WildBench	Avg. $\Delta$
Qwen3-4B
Instruct	85.90	62.24	12.82	43.39	62.80	52.27	41.98	51.63	6.320	49.0%	3.730	−2.96
Ordinary DPO	87.57	63.64	14.73	43.12	62.63	53.91	42.30	52.56	6.890	50.9%	3.617	0.00
Common	87.04	63.48	15.64	43.12	63.23	53.72	42.23	52.64	6.720	52.3%	3.711	+0.96
Common+Specific	87.04	63.74	15.21	42.99	63.40	53.71	42.25	52.62	6.610	53.1%	3.840	+2.14
Common+Specific+Low	87.49	64.18	14.96	43.25	63.31	53.74	42.25	52.74	6.930	53.0%	3.727	+2.58
Common+Low	86.88	64.02	15.14	43.12	63.14	53.69	42.26	52.61	6.900	52.6%	3.602	+1.02

Table 1: Rubric-free Qwen3 results on both benchmark suites, each from its own dataset. Capability scores and their unweighted seven-task mean are percentages from the Tulu3 runs. The open-ended protocols come from the HelpSteer3 runs: MT-Bench and WildBench use native scores and Arena-Hard is the win rate against GPT-4-0314 (%). Open-ended Avg. $\Delta= \frac{100}{3}\sum_b(s_b/s_b^{\mathrm{DPO}}-1)$ averages relative changes across all three benchmarks against the same block’s DPO control. Bold marks maxima within each model block (including ties). Shading marks rubric-trained rows.

Variation Across Datasets and Models. In Prop. 1, the local effect depends on task demand and the corpus-dependent update direction. The ordinary-DPO controls measure the full rubric’s finite-training effect separately within each dataset. On Qwen3-4B, the capability effect is larger on Tulu3: BBH improves by 4.20 points, compared with 0.23 on HelpSteer3, and the seven-task mean by 1.82 points, compared with 0.18 (Table 1 and Table 10). Both methods train for one epoch within each dataset. On HelpSteer3, Qwen3-8B gains 0.20 MT-Bench points and 1.5 Arena-Hard points (Table 1). Mistral-7B gains 0.29 MT-Bench points and 1.8 Arena-Hard points (Table 2).

Table 2: Mistral-7B results on both benchmark suites, each from its own dataset. Capability scores and their unweighted seven-task mean are percentages. Open-ended Avg. $\Delta$ follows the definition in Table 1. Bold marks maxima within each dataset block; shading marks rubric-trained rows.

Adding Factors to a Common Rubric. Four HelpSteer3 compositions each contain the Common factors $f_2$ and $f_5$, with the Specific or Low-demand factors added. Their seven-task means differ by at most 0.13 points and exceed ordinary DPO’s 52.56 by at most 0.18. Common raises BBH by 0.90 points relative to ordinary DPO, adding Specific yields 0.47, adding Low yields 0.40, and including both yields 0.23. All four compositions improve Arena-Hard by 1.4–2.2 points, with a different ordering from BBH. The small average differences among compositions do not identify a superior factor set. The full ontology supplies a fixed set of clauses for subsequent editing. Per-benchmark results are in Table 10.

Rubric Dependence at Inference Time. Reinserting a rubric at inference time has task-dependent effects on the same trained checkpoint (appendix § 11.§ 1). On matched sampled BBH items, the trained $R_1$ wording changes accuracy by −1.25 points and the candidate $R_2$ by +3.70 points. The separate greedy MATH-Hard evaluation changes by +0.68 and +0.76 points respectively.

What the Rubric Edit Contributes

The edit comparison in § 3.3 fixes the starting policy and varies only the rubric-induced update direction; it is carried through a matched continuation, with KEEP controlling for additional training. The trained Qwen3-4B policy reviews its first-round feedback and finds a recurring $f_4$ gap in its training rubric. The frozen clause prohibited unsupported assertions but left the evidence requirement for “common knowledge” ambiguous. The revised Prefer/Avoid pair reads:

Prefer: factual claims are grounded in the given information or explicitly sourced common knowledge. Avoid: invented facts, unsupported assertions, or inferences from vague common knowledge.

Round 1 refines only $f_4$; the other seven factors return KEEP because no recurring uncovered requirement is identified. The edit adds seven words across the Prefer/Avoid pair in a 234-token rubric. Table 3 evaluates it through two HelpSteer3 continuations from the same Qwen3-4B policy, using the same pairs, frozen reference, optimizer settings, seed, and pair-level exposure mask. The KEEP branch retains $R_0$; the REFINE branch uses $R_1$. Unchanged-rubric continuation raises BBH from 14.96 to 18.05, a gain of 3.09 points. Relative to that continuation, the edit changes BBH by −0.25 points, MT-Bench by +0.406, and Arena-Hard by −0.1 percentage points. The edit’s incremental effects differ across tasks: BBH decreases slightly, while the MT-Bench point estimate recovers part of the decline under unchanged-rubric continuation, from 6.93 in round 1 to 6.419 under KEEP and 6.825 under REFINE. Evaluation protocols and paired comparisons are in appendix § 11.§ 2.

The revision trace records how failure attribution and edit constraints determine the next rubric. Factor $f_3$ receives the strongest failure signal, but its existing Avoid clause already names the concrete requirement. The engine nevertheless proposes an $f_3$ rewrite. The gate rejects it because it drops part of the frozen semantic anchor; cross-factor vocabulary also triggers a warning. Factor $f_3$ therefore returns to KEEP. This rejection preserves the original $f_3$ clause for further learning. On Qwen3-8B, attribution marks every requirement in the feedback set as already covered and the operator returns KEEP for all factors. The subsequent 4B feedback pass produces a three-factor $R_2$ candidate for a future training round.

Conclusion

RubricOpt connects policy self-evaluation to revision of the behavioral specification guiding preference learning. Its semantic rubric translates failure feedback into training-time criteria while preference pairs, labels, and the optimization objective remain fixed. Experiments across comprehensive capability and open-ended generation show that rubric semantics can shape rubric-free behavior, with benefits depending on the model and preference corpus. The tested revision yields mixed incremental results over unchanged-rubric continuation. This makes the behavioral specification itself adaptable through the policy’s feedback loop.

Scope and Future Work. The study uses an associative demand measure and a manually specified ontology. Evaluating successive revisions on feedback-disjoint tasks and comparing proposals from initial and updated policies under a shared training budget would test the scope for sustained self-improvement.

Reproducibility Statement

Every trained method in this paper is a rank-64 LoRA adapter over a released checkpoint, and the appendices specify the dataset, optimizer, schedule, seed, conditioning rate, reference convention, evaluation protocols, and judge configuration for every reported run. The two preference datasets are public, and the seven accuracy benchmarks and three open-ended protocols are used through their released forms. The initial formal rubrics can be reconstructed from the canonical bank in appendix § 8.§ 2 and the template in appendix § 10.§ 1. The accepted HelpSteer3 $R_1$ edit is quoted in § 4.3. The refinement operator is specified as five stages with their inputs, outputs, and the deterministic acceptance checks (appendix § 10.§ 2).

Broader Impact Statement

RubricOpt exposes behavioral criteria as short, inspectable training artifacts, with an evidence trail for each accepted edit. The policy under diagnosis proposes the edits, so its interpretation of failures can reinforce its own biases. Deterministic checks enforce evidence references and textual constraints; rubric-free evaluation measures the resulting behavior. Deployments should review proposed rubric changes against their intended behaviors and evaluate each edit with a matched continuation control.

LLM Usage Disclosure. Large language models (LLMs) were used throughout this work to assist with several aspects of the research and writing process. Specifically, we used LLMs to improve the clarity and presentation of the manuscript, to support literature retrieval and the discovery of relevant prior work, and to assist in drafting portions of the paper. The authors reviewed, revised, and verified all LLM-assisted content and remain fully responsible for the final manuscript.

Preliminaries and Measurement

Notation

Models and data $m$, $\pi_\theta$, $\pi_{\mathrm{ref}}$: Current model, its trainable policy, and the frozen reference. $B$, $o$: Diagnostic task and its binary outcome. $\mathcal{D}$, $x$, $y$, $y^{+}$, $y^{-}$: Preference data, prompt, response, and chosen/rejected responses. $L$: Number of response tokens in a response; $\bar{\ell}$ averages a log-probability over them.

Behavioral measurements $\mathcal{F} = \{f_1, \ldots, f_8\}$: Fixed factor ontology of eight factors. $z_k(x,y) \in\{0,\ldots,4\}$: Ordinal score a blind judge assigns to factor $f_k$. $D_{m,B,k}$: Model- and task-specific semantic demand (Eq. (2)). $\Delta z_k$, $\mu_k$, $C_k$, $P_k$: Pair contrast, its mean, contrast coverage, and label-agreement rate (Eq. (3)).

Conditioning and updates $\mathcal{G} \subseteq\mathcal{F}$, $c_r$, $\varnothing$: Selected factors (all eight in the main method), the context rendered from rubric $R_r$ by Eq. (4), and the absent criterion. $u$, $u_t$, $q(\cdot)$, $p_t$: Conditioning context $u \in\{c_r,\varnothing\}$, its draw at training step $t$ within round $r$, and the conditioning rate at progress $p_t$. $\beta$, $\sigma$, $\mathcal{L}$: DPO scale, logistic function, and Semantic Rubric Steering loss. $M_u$, $s_u$, $g_u$, $\kappa_u$: Length-normalized chosen–rejected margin, mean per-token response score gradient, loss gradient, and scalar multiplier, all in context $u$. $\Lambda$, $A_k$, $H_k$: Evaluation prompt distribution, rubric-free expected factor score $\mathbb{E}_{\nu}[\bar{z}_k(x)]$, and its gradient Lipschitz constant. $G_c$, $G_0$, $G_q$: Mean loss gradients with and without the criterion, and their mixture at rate $q$. $\eta$, $\theta_q$, $\theta_0$, $\gamma_k$: Step size, two local updated parameter states, and the transfer remainder. $\bar{z}_k(x)$, $\delta_c$, $\delta_0$: Prompt-specific expected factor score and two actual optimizer increments from a shared state. Shift$_k$: Semantic shift, the change in mean factor score after an update.

Rounds and probabilistic analysis $R_r$, $\pi_{r+1}$, $\mathcal{B}_{r+1}$: Round-$r$ rubric, the policy trained under its context $c_r$, and the resulting behavioral feedback (Eq. (6)). $\psi_0$, $F$: Full-response deployment score (Eq. (10)) and its Fisher matrix (Eq. (26)). $D$, $\Sigma$, $\Gamma_u$, $P_{\tau:T}$: Deployment demand vector and factor covariance (Eq. (19)), training–deployment gradient alignment (Eq. (11)), and remaining Fisher path length. RUBRICOPT-R2 (KEEP): Matched re-optimization branch that keeps the criterion $R_0$. RUBRICOPT-R2 (REFINE): Matched re-optimization branch that refines the criterion to $R_1$. The two branches differ only in the criterion.

Measurement and the Factor Ontology

The measurements use the definitions of § 3 and the blind judge of appendix § 8.§ 3.

Worked Demand Example. A GSM8K problem asks the cost of six pounds of butter at $4 per pound when every second pound is half price. A successful response answers $18, while a failed response answers $16. Their judge vectors are [2, 3, 1, 3, 2, 3, 3, 3] and [2, 1, 1, 3, 2, 3, 3, 3], differing only on reasoning validity. This illustrates scoring, not estimation of demand, which aggregates the success–failure gap over the benchmark population.

Worked Supply Example. In one HelpSteer3 joke-request pair, the chosen–rejected score difference is [0, 0, 0, 0, +1, 0, 0, 0]: it contrasts completeness but not reasoning validity. A pair about female US presidents contrasts several factors at once, with difference [+2, +2, 0, +2, +1, +2, 0, +1]. Supply therefore summarizes a distribution of within-pair contrasts rather than assigning each pair to one factor.

The Factor Ontology. Every quantity above is indexed by the fixed eight-factor ontology, whose demand profile over the seven benchmarks is Table 7. The frozen bank supplies one Prefer and one matched Avoid clause per factor, as rendered for training:

f1 task comprehension
    Prefer:   show a correct understanding of what was asked
    Avoid:    answering a different question or ignoring stated conditions
f2 reasoning validity
    Prefer:   reasoning steps that are logically grounded and internally
              consistent
    Avoid:    unsupported intermediate inferences or invalid deductions
f3 verification and calibration
    Prefer:   verify key intermediate results and express uncertainty in a
              calibrated way
    Avoid:    unchecked steps or unsupported overconfidence
f4 factual and evidential grounding
    Prefer:   factual claims are grounded in the given information or common
              knowledge
    Avoid:    invented facts or unsupported assertions
f5 completeness
    Prefer:   responses that include the reasoning and information needed to
              fully address the task
    Avoid:    missing necessary reasoning, information, or important parts of
              the task
f6 instruction adherence
    Prefer:   follow the stated instructions and stay on task
    Avoid:    drifting off-topic or violating requested formats
f7 clarity and organization
    Prefer:   clearly expressed and logically organized
    Avoid:    jumbled steps or answers that are hard to locate
f8 efficiency and conciseness
    Prefer:   keep only the necessary content
    Avoid:    repeating reasoning or padding with filler

Judge Configuration and Reliability

Production Configuration. Semantic factor scores reported in this paper come from a self-judge. The policy under diagnosis scores its own rubric-free outputs with one fixed, version-tagged judge prompt. Decoding is greedy (temperature 0, top-$p$ 1, top-$k$ 1) with at most 1100 new tokens, at batch size 64. The self-judge is blind to the task outcome, the gold answer, the preference role, and the identity of the generating model. Each call returns one raw integer score per factor together with a supporting sentence, and the scores are not normalized across factors, responses, or runs.

Score Anchors. The shared scale is 0 = clearly poor or absent, 1 = weak, 2 = mixed or partial, 3 = good, and 4 = strong. The production prompt defines the top of the scale as an absolute standard rather than a comparative one: a 4 is earned by concrete extra rigor, such as re-checking a computation, stating and handling an assumption or edge case, noticing an ambiguity, or flagging its own uncertainty, and it is explicitly not awarded for being correct and fluent. A 3 is competent, complete, routine work with no extra rigor; a 2 does the minimum with visible gaps; a 1 is missing or merely asserts an important behavior. The prompt also asks the judge to list two strengths and two weaknesses before scoring, and gives per-factor rules that cap a score at 1. For reasoning validity, the cap applies when any step is a non-sequitur or an equation does not follow. The prompt permits NA only when a behavior cannot occur for that response kind, and states that a required behavior which is missing still scores 0 or 1 rather than NA. In practice that rule never fired: across the 108,685 scored responses, every factor was assessable on every response.

Agreement Audits. A different judge model rescored 1,200 responses. On items where the audit judge gave a decisive score, mean agreement with the production self-judge was 0.741. Per factor, agreement was highest for task comprehension (0.909), factual and evidential grounding (0.857), and instruction adherence (0.800), and lowest for clarity and organization (0.633), verification and calibration (0.616), and efficiency and conciseness (0.559). Table 4 gives the per-factor agreement. The three generative benchmarks are scored by MiniMax-M3.

Factor	Decisive	Agree	Decisive-only rate
$f_{1}$ task comprehension	11	10	0.9091
$f_{2}$ reasoning validity	52	41	0.7885
$f_{3}$ verification and calibration	232	143	0.6164
$f_{4}$ factual and evidential grounding	7	6	0.8571
$f_{5}$ completeness	105	80	0.7619
$f_{6}$ instruction adherence	15	12	0.8000
$f_{7}$ clarity and organization	199	126	0.6332
$f_{8}$ efficiency and conciseness	247	138	0.5587
Mean			0.7406

Table 4: Blind pairwise agreement across four benchmarks. Pairwise verdicts are compared with the direction of independent per-response scores. Decisive excludes ties in either judgment; agreement rates use this subset.

Factor Separability. Across 108,685 complete responses, the production self-judge yields mean absolute inter-factor correlation 0.195 and maximum 0.362. The older judge prompt, used as a collapse control, reaches maximum correlation 1.000 and mean 0.378, and produces only four distinct score vectors across 288 responses, whereas the production prompt produces 822. The eight factors are therefore not collapsed onto one quality axis in this analysis.

Proofs

Local Behavioral Transfer

The Deployment Space. The analysis uses finite prompt and response spaces and a strictly positive, continuously differentiable policy on a neighborhood of every path considered. The fullresponse deployment score is

For a training triple $\xi=(x',y^+,y^-)\sim\mathcal{D}$, write $d_u(\xi)$ for its pair direction. The incremental rate of change of deployment log-probability along the rubric-induced direction is

The score identity gives $\mathbb{E}_{P_\theta}[\psi_0\mid x]=0$ and hence $\mathbb{E}_{P_\theta}[h\mid x]=0$, so each fixed preference pair contributes through the alignment of its training gradient with the deployment score.

The Behavioral Readout. What remains is to characterize that direction. For a rubric-free evaluation distribution $\nu$, define the behavioural readout

and let $G_c$ and $G_0$ be mean preference-loss gradients with and without the criterion, with $G_q=(1-q)G_0+qG_c$.

Proposition 2 Suppose $A_k$ has an $H_k$-Lipschitz gradient on a neighborhood containing the two update segments. For $\theta_q=\theta-\eta G_q$ and $\theta_0=\theta-\eta G_0$,

The Score-Function Identity. Assume differentiation and expectation can be interchanged and define $\bar{z}_k(x)=\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[z_k(x,y)]$ at the current parameter state. The score-function identity gives

The baseline term vanishes because the conditional mean score gradient is zero. A finite response space with a smooth, positive policy provides a sufficient setting for this identity, and the factor score itself need not be differentiable in the response: a randomized judge enters through its fixed conditional mean score. Eq. (15) is taken on generated responses under the rubric-free policy, which is a different quantity from the training-pair contrast $\Delta z_k$.

Actual Optimizer Steps. For two parameter increments $\delta_c$ and $\delta_0$ from the same parameter state, the same Taylor argument yields

Here the same smoothness assumption holds along both update segments, and the form accommodates adaptive updates and momentum, which is the regime of the experiments.

Eq. (16) compares updates from shared parameters, optimizer state, and minibatch. Their effect on a behavior depends on the alignment of the increment difference with that behavior’s gradient. The context probes in § 4.1 measure training-pair gradient and margin changes at a fixed state. The benchmark comparisons measure rubric-free outcomes after training.

Proof Of Prop. 2. For either $G=G_q$ or $G=G_0$, the Lipschitz-gradient assumption gives the Taylor expansion

Subtract the two expansions and use $G_q-G_0=q(G_c-G_0)$. The remainder is $r_k=e(G_q)-e(G_0)$, whose absolute value is bounded by Eq. (14) by the triangle inequality. In particular, a sufficient condition for strict improvement over the ordinary step is

A nonzero first-order term dominates the quadratic remainder for sufficiently small positive $\eta$, proving the sign statements. A zero inner product leaves the sign undetermined. Improvement over the initial policy is a different comparison: its first-order term is $-\eta\langle\nabla_\theta A_k,G_q\rangle$. □

Proof Of Prop. 1

Semantic Projections and the Log-Odds Form. At a fixed state, write $p=p_B(\theta)$, $Z=z-\mathbb{E}[z]$, and

Here $\Sigma^{\dagger}$ is the Moore–Penrose pseudoinverse, $a$ is the coefficient vector for the log-probability rate $h$, and $\epsilon_h,\epsilon_o$ are the corresponding root-mean-square residuals. The residual contribution is

If either projection residual vanishes, then $\mathcal{R}=0$. For $\theta_q(\eta)=\theta+\eta v_q$ and $\theta_0(\eta)=\theta+\eta v_0$, with directions fixed at $\theta$ and $\operatorname{logit}(p)=\log[p/(1-p)]$, the equivalent log-odds form is

Proof. The deployment score satisfies $\mathbb{E}[\psi_0\mid x]=\sum_y\nabla_\theta\pi_\theta(y\mid x,0)=0$. For any fixed direction $w$, differentiating the deployment expectation gives

Taking $w=\Delta v$ and dividing by $p(1-p)$ expresses the incremental log-odds derivative as

Likewise, conditioning on the binary outcome gives $\mathbb{E}[Z(o_B-p)]=p(1-p)D$.

The normal equations for Eq. (20) hold even when $\Sigma$ is singular. Indeed, $w\in\ker\Sigma$ implies $\mathbb{E}[(w^{\mathsf{T}}Z)^2]=0$, hence $w^{\mathsf{T}}Z=0$ almost surely. Both covariance vectors being projected are therefore in the range of $\Sigma$. Thus $\mathbb{E}[Zr_h]=\mathbb{E}[Zr_o]=0$, and

Cauchy–Schwarz bounds the last expectation by $\epsilon_o\epsilon_h$. The least-squares residual variance for $o_B-p$ is $p(1-p)-\mathbb{E}[Z(o_B-p)]^{\mathsf{T}}\Sigma^{\dagger}\mathbb{E}[Z(o_B-p)]$, which gives Eq. (21). Finally, $v_q-v_0=q\Delta v$ proves Eq. (22); multiplying by $p(1-p)$ gives Eq. (9). A strictly positive derivative of the log-odds difference, which is zero at $\eta=0$, makes that difference positive for sufficiently small positive $\eta$. The logit is strictly increasing. The negative case follows identically. □

Retention After Rubric Withdrawal

Let $\tau$ mark the end of rubric exposure and $T$ the final checkpoint, with success probability $p_t=p_B(\theta(t))$ and its monotone transform $\alpha_t=\arcsin\sqrt{p_t}$. The Fisher path length $\mathcal{P}_{\tau:T}$ measures accumulated movement of the rubric-free response distribution over the remaining training.

Fisher Path Length and Probability Bounds. For the piecewise continuously differentiable path $\theta(t)$, define

Under the regularity assumptions of Appendix § 9.§ 1, Eq. (27) gives

For a fixed comparison probability $p_\star\in(0,1)$ under the same prompt law and success rule, a sufficient condition for $p_B(\theta(T)) > p_\star$ is

Proof. Write $p_t=p_B(\theta(t))$ and $h_t=\langle\psi_0,\dot{\theta}(t)\rangle$, with $\psi_0$ evaluated at $\theta(t)$. The score identity gives $\mathbb{E}[h_t]=0$ and $\dot{p}_t=\mathbb{E}[(o_B-p_t)h_t]$. Hence

Since $\dot{\alpha}_t=\dot{p}_t/[2\sqrt{p_t(1-p_t)}]$, integration on each smooth segment and the triangle inequality prove Eq. (27). Clipping its lower bound at zero and using monotonicity of $\sin^2$ on $[0,\pi/2]$ gives Eq. (28) and Eq. (29). For sharpness, consider one prompt and two responses, with success probabilities $(\sin^2\alpha,\cos^2\alpha)$ and $\alpha\in(0,\pi/2)$. The Fisher information for parameter $\alpha$ is $4$. Every monotone path then has $\mathcal{P}_{\tau:T}=2|\alpha_T-\alpha_\tau|$, achieving equality. □

For actual optimizer increments $\delta_j=\theta_{j+1}-\theta_j$, linear interpolation gives the exact path length

This form applies to a realized AdamW trajectory, including momentum and weight decay.

For the idealized mean-gradient path $\dot{\theta}=v_0+q\Delta v$ the Fisher seminorm satisfies $\sqrt{\dot{\theta}^{\top}F\dot{\theta}}\le\sqrt{v_0^{\top}Fv_0}+q\sqrt{\Delta v^{\top}F\Delta v}$, with every term evaluated at the evolving policy.

Semantic Rubric Steering and the RubricOpt Loop

The Criterion Bank and Training-Only Conditioning

Appendix A.2 gives the frozen canonical names and Prefer/Avoid clauses used in formal training. For each selected factor, the renderer takes its current clauses in the order $f_1$–$f_8$. The main method selects all eight factors, while the ablation selects the groups in Table 10. The renderer stays fixed across rounds:

Capability priorities: <factor A>, <factor B>
Prefer:
- <Prefer clause of factor A>;
- <Prefer clause of factor B>.
Avoid:
- <Avoid clause of factor A>;
- <Avoid clause of factor B>.

The Common rubric uses the $f_2$ and $f_5$ entries of appendix A.2; larger sets extend the same lists. With the Qwen3-4B tokenizer, Common, Common+Specific, Common+Low, and the full rubric cost 70, 151, 153, and 234 suffix tokens respectively. Gate-accepted edits change clauses, not the factor order or renderer.

The Development Pilot. The pilot compares ordinary DPO, a single-factor $f_2$ criterion, and a single-factor $f_7$ criterion while holding the pairs, labels, DPO objective, data order, and LoRA configuration fixed. It trains Qwen3-4B for 150 steps at a fixed conditioning rate $q=0.2$, so the $f_2$ and $f_7$ conditions condition exactly the same training rows and the comparison varies the criterion text alone. Six Ascend 910B1 NPUs with per-device batch 2 and accumulation 4 give 48 pairs per step and about 7,200 pairs over those 150 steps (0.20 epoch). Maximum prompt and sequence lengths are 1,024 and 2,048 tokens, in bf16 with FlashAttention-2, and prompt truncation retains the final 1,024 tokens, including the criterion suffix. Each run takes about 47 minutes.

Pilot evaluation uses its own sampling schedule. It draws 500 prompts per benchmark with 4 samples per prompt at temperature 0.7, top-$p$ 0.8, and top-$k$ 20. The unit of analysis is the prompt, and the paired tests are exact McNemar tests on the pass@4 indicator together with a two-sided sign-flip permutation test of 10,000 resamples (seed 20260904) on the paired per-prompt mean-correctness differences. The two criteria differ only in their text and are within one token of each other in length.

Under the paired protocol, the $f_2$ criterion is favored over the $f_7$ criterion on all four pass@4 comparisons (Table 5). The $f_2$ criterion is ahead on GSM8K and on MuSR pass@4 (17 passes unique to $f_2$ against 6 unique to $f_7$), and MATH-Hard and BBH show the same direction. The $f_7$ criterion shifts negatively relative to ordinary DPO on MuSR pass@4 (3 passes unique to $f_7$ against 14 unique to ordinary DPO) and does not separate from ordinary DPO on the other three benchmarks. The $f_2$ criterion and ordinary DPO do not differ on any of the four benchmarks, and ordinary DPO does not differ from the Instruct model on any of them. Changing the criterion text alone therefore moved the two criteria apart on MuSR pass@4, and the $f_7$ criterion is the direction the diagnosis does not support. Across the four benchmarks, counts of prompts passing only under the first versus only under the second condition are [4/1, 4/2, 17/6, 23/17] for $f_2$ versus $f_7$ and [4/5, 2/3, 3/14, 15/13] for $f_7$ versus ordinary DPO.

Benchmark	Instruct	Ordinary DPO	$f_2$ Criterion	$f_7$ Criterion
GSM8K accuracy	0.8945	0.8955	0.9015	0.8910
GSM8K pass@4	0.928	0.928	0.932	0.926
MATH-Hard accuracy	0.9070	0.9105	0.9125	0.9095
MATH-Hard pass@4	0.964	0.964	0.966	0.962
MuSR accuracy	0.5870	0.5825	0.5795	0.5840
MuSR pass@4	0.706	0.708	0.708	0.686
BBH accuracy	0.0520	0.0535	0.0560	0.0495
BBH pass@4	0.142	0.126	0.142	0.130

Stages, Contracts and the Deterministic Gate

Evidence Set and Engine. Figure 2 in the main text gives the six-step workflow. The diagnostic stage reads item-level judge rationales for the all-eight-factor method against ordinary DPO and the Instruct model on Arena-Hard, MT-Bench (including a dedicated second-turn batch), and WildBench, together with generated correctness items carrying a gold answer on GSM8K, MATH-Hard, and MMLU-Pro. That evidence is the first-round rubric-free benchmark evaluation output, read item by item, as described in § 4. The analysis and proposal engine is the current policy itself, the all-eight-factor merged checkpoint, so the model that reads the benchmark feedback is the model the feedback is about. The external anchors are the gold answers and benchmark outcomes, the fixed judge’s rationales and checklists, and the frozen $R_0$ text.

Five prompt stages process feedback from policy $\pi_{r+1}$, trained with the frozen round-$r$ rubric $R_r$, in the order 1 (benchmark diagnostic), 2 (cross-benchmark synthesis), 3 (factor attribution), 4 (conservative update), and 5 (rubric audit), followed by step 6 (deterministic gate). Each stage is a separate call with its own schema, and no stage sees the next stage’s output.

Stage 1 sees the factor schema, the criterion currently in force, and one benchmark’s evidence, and returns a four-key YAML document:

preserve:
  - behavior:  / factor:  / evidence:
improve:

- failure_mode:   / factor:   / evidence:
criterion_gaps:
  - factor:   / current_problem:   / suggested_direction:
overall_diagnosis:

Stage 1 uses only $f_1$–$f_8$ and separates behaviors to preserve from those to improve. An isolated example requires strong evidence, and attributing a benchmark change to the criterion requires behavioral evidence. The output must be concrete and concise.

Stage 2 aggregates every stage-1 report for one policy. It may not rewrite the criterion or propose wording. It organises evidence and records disagreement, adding the benchmarks that support each pattern, an evidence-strength level (strong, moderate, weak), and an explicit record of contradictions. A benchmark where the policy did not change still counts as evidence about what is stable.

Stage 3 attributes each recurring behaviour onto the ontology and decides whether the current wording already covers it. The coverage decision follows this rule:

If the current Prefer/Avoid wording states this failure mode in full, the model simply did not learn it: the decision is KEEP. If the wording does not state the failure mode, or states only part of it, that is the only case in which REFINE may be considered.

Attribution disambiguates reasoning validity, verification and calibration, completeness, and instruction adherence, naming the primary factor when several are involved. It attributes only what the evidence supports.

Stage 4 proposes the smallest clause edits that address high-confidence recurring gaps while preserving factor identity and behaviors marked for preservation. Edits may clarify wording, narrow an overly broad reading, make an implicit requirement explicit, or refine a matched Avoid clause. Stage 4 prohibits examples, task-specific tricks, benchmark or dataset names, scores, numerical weights, and intensifiers. Unsupported changes return KEEP. The output is a candidate $R_{r+1}$ and a per-factor ledger with the action, reason, and old and new clauses.

Stage 5 audits semantic identity, benchmark and dataset leakage, evidential support, specificity, Prefer/Avoid matching, preserved behaviors, and edit magnitude. It emits a justification per axis and one PASS or FAIL. The audit is advisory rather than decisive: in one executed round it returned a clean eight-of-eight PASS on a candidate that the deterministic gate then rejected.

Step 6: The Deterministic Gate. The binding checks constrain eligibility, semantic identity, and procedural consistency.

Acceptance is decided by eight checks in round 1 and ten in round 2. Round 2 adds that only gate-accepted factors may differ, and that the recorded decisions are consistent with the rendered rubric.

1. All eight factors must occur exactly once, each with a KEEP/REFINE decision.

2. The input clauses of every refinement must match the frozen canonical wording byte-for-byte.

3. Each new clause must retain at least 0.60 of the old clause’s content tokens, with its length change inside [−5,+10] words.

4. Every factor must retain one matched Prefer/Avoid pair.

5. Clauses must contain no benchmark or dataset names, digits, intensifiers, or example markers.

6. Every refinement must be traceable to an attribution judging the failure not fully covered.

7. Edit magnitude must remain within the specified budget.

8. Cross-factor vocabulary imports raise a warning rather than a rejection.

9. In round 2, only gate-accepted factors may differ.

10. In round 2, the recorded decisions must agree with the rendered rubric.

Failures of the binding checks reject the affected edit, so a factor that violates a check is reverted to KEEP.

Matched Re-Optimization and the Dataset Control

The shared recipe is in § 4: rank-64 LoRA ($\alpha= 128$, dropout 0.05), mean-per-token DPO with $\beta= 1$ at learning rate $10^{-6}$, global batch 64, and seed 42. The formal runs and the HelpSteer3 continuations take 566 steps. The HelpSteer3 continuations start from the frozen RUBRICOPT-R1 checkpoint and use fresh AdamW and FP32 LoRA adapters, cosine decay with 5% warmup, and one shared exposure mask on identical ordered pairs. Their reference is the frozen RUBRICOPT-R1 checkpoint with the new adapter disabled, and the starting policy of every other block is the released Instruct checkpoint with its own adapter disabled. Only the rubric text differs: KEEP uses $R_0$ and REFINE uses $R_1$. The primary comparison RUBRICOPT-R2 (REFINE) minus RUBRICOPT-R2 (KEEP) measures editing; comparison with RUBRICOPT-R1 includes further training.

Tulu3 uses the same initial rubric and training recipe on an independently assembled dataset, over 1,818 steps at global batch 64. The budgets match ordinary DPO within each dataset by epoch count, not across datasets by step count. The Tulu3 second-round change combines editing and additional training.

Experimental Settings and Results

This appendix gives the measurement and protocol detail behind the main-text results: evaluation protocols, training-protocol audits, and the per-cell measurements that the main tables summarize.

Evaluation and Measurement Protocols

Splits and Evaluation Protocols. The factor measurements use diagnostic and held-out subsets assigned by an md5 hash of the prompt identifier, and comparing these subsets tests the stability of the measured demand gaps. Evaluation prompts come from benchmark sources, while the preference-training pairs come from HelpSteer3. Rubric refinement uses first-round benchmark evaluation outputs as feedback and compares the continuation branches on matched evaluation items (§ 4). The formal study uses the released benchmarks through the standard evaluation harness, with adapter evaluation enabled and the adapter merged, and the reported baselines use this protocol (Table 1 and Table 10). The demand values in Table 7 are for Qwen3-4B.

Open-Ended Protocols. Arena-Hard and WildBench use greedy decoding with a 4,096-token generation limit and one response per prompt. WildBench uses 256 prompts and MiniMax-M3 scores on a 1–10 scale, averaged over usable verdicts (255 for Qwen3-8B RUBRICOPT-R1, 256 for the other reported conditions). MT-Bench uses a 1,024-token generation limit. Every Arena-Hard condition is judged against the fixed answers of the standard Arena-Hard anchor (GPT-4-0314), with the candidate always in slot B, so Arena-Hard here is a fixed-anchor win rate: every method faces the same anchor on the same items, rather than a head-to-head between methods. Per-item outcomes are wins, ties, or losses, with ties counted as half-wins, and all conditions use the same tie rule, anchor, and items.

Accuracy Evaluation Protocols. Absolute scores are reported in Table 1 to Table 3 and Table 10. BBH covers all 27 subtasks in the project’s sampled evaluation. The paired differences in Table 9 use the same (prompt_id, sample_idx) keys within each model–dataset cell. MATH-Hard uses the lm-eval task template with 7 subtasks. Scores are the unweighted mean exact match over subtasks under vLLM/bf16 with merged adapters.

Cell	Comparison	$\Delta$	Reading
4B × HS3	Common − Ordinary DPO	+0.0090	positive
4B × HS3	Common+Specific − Ordinary DPO	+0.0047	positive
4B × HS3	Common+Low − Ordinary DPO	+0.0040	positive
4B × HS3	RUBRICOPT-R1 − Ordinary DPO	+0.0023	null
4B × HS3	RUBRICOPT-R2 (KEEP) − RUBRICOPT-R1	+0.0309	positive
4B × HS3	RUBRICOPT-R2 (REFINE) − RUBRICOPT-R2 (KEEP)	−0.0025	null
4B × Tulu3	RUBRICOPT-R1 − Ordinary DPO	+0.0420	positive
8B × Tulu3	RUBRICOPT-R1 − Ordinary DPO	+0.0961	positive

Training-Protocol Details. The trainer computes Eq. (5) with each response’s own unpadded token count. Policy and frozen reference receive identical batch tensors and context; reference log-probabilities are recomputed with the adapter disabled and no cache. Rubric tokens remain in the masked prompt region and enter neither response-token numerator nor denominator, and the chosen and rejected response text is unchanged by conditioning.

Matched-Length Criterion Comparison. At formal scale, Common+Specific and Common+Low vary the criterion text, and different additions to Common. Common+Specific exceeds Common+Low by 0.07 BBH points, 0.01 points on the seven-task average, and 0.5 Arena-Hard points.

Formal Runs. Formal runs use eight Ascend NPUs under distributed data parallel. The realized conditioning exposure is 0.5009 on HelpSteer3 and 0.4971 on the Tulu3 conditioned runs, and the criterion is withdrawn for the final 10% of training and for main evaluation.

Inference-Time Rubric Reinsertion. The same Qwen3-4B RUBRICOPT-R2 checkpoint is evaluated with no rubric, the trained wording $R_1$, and the untrained candidate $R_2$. The rubric is inserted at the last message before templating, adding 243 and 256 prompt tokens respectively. Table 6 reports six benchmarks generated in one matched batch window, with identical (prompt_id, sample_idx) sets of 2,000 rows per benchmark. On MATH-Hard, OFF scores 63.17%, $R_1$ ON 63.85%, and $R_2$ ON 63.93%.

Condition	GSM8K	MMLU-Pro	TruthfulQA	ARC	BBH	MuSR
OFF	0.8995	0.6390	0.6955	0.9070	0.4310	0.5755
$R_1$ ON	0.8960	0.6205	0.6895	0.8945	0.4185	0.5640
$R_2$ ON	0.9055	0.6285	0.6905	0.9000	0.4680	0.5775

The Demand Matrices. Table 7 reports the full Qwen3-4B demand matrix with effect sizes. The largest diagnostic–held-out gap difference is 0.045, with no sign reversal. The 8B matrix is in Table 11.

Benchmark	$f_1$	$f_2$	$f_3$	$f_4$	$f_5$	$f_6$	$f_7$	$f_8$
GSM8K	+0.113	+0.130	−0.096	+0.014	+0.189	+0.058	−0.065	+0.171
MATH-Hard	+0.011	+0.079	+0.182	+0.013	+0.144	+0.058	−0.163	+0.080
BBH	−0.038	−0.007	+0.142	+0.001	−0.100	−0.028	−0.171	+0.073
MuSR	+0.022	+0.042	+0.039	+0.012	+0.042	+0.022	+0.038	+0.024
ARC	+0.072	−0.167	+0.109	+0.089	−0.141	−0.033	−0.006	−0.035
TruthfulQA	+0.016	+0.053	+0.078	+0.100	−0.017	−0.031	−0.131	+0.039
MMLU-Pro	+0.057	+0.136	+0.190	+0.059	+0.521	+0.092	+0.183	+0.137

Factual and evidential grounding separates success from failure more strongly on the knowledge benchmarks than on the reasoning benchmarks (0.111 against 0.031, Figure 3b). Within the knowledge group, ARC is driven by reasoning validity, TruthfulQA by verification and factual grounding, and MMLU-Pro by completeness. MuSR is close to flat on every factor. A deployment target usually contains several benchmarks with different semantic needs, and how they are combined changes which factor appears most important. On the knowledge group the equal-per-benchmark average puts reasoning validity first (0.213) and completeness second (0.193), whereas pooling by response count reverses the order, because MMLU-Pro supplies about 86% of the responses in the group, raising completeness to 0.356. The per-benchmark profiles are kept. The equal-per-benchmark average is a descriptive summary.

Model-Dependent Capability Signatures. Demand changes with the evaluated model as well as with the task. The Qwen3-4B and Qwen3-8B profiles correlate strongly on TruthfulQA ($r = 0.953$), GSM8K ($r = 0.906$), and MMLU-Pro ($r = 0.868$), moderately on MuSR ($r = 0.522$), MATH-Hard ($r = 0.461$), and ARC ($r = 0.457$), and negatively on BBH ($r = -0.751$). The disagreement is not confined to one factor (Figure 4a): on BBH, $f_5$, $f_7$, $f_6$, and $f_2$ all change sign. The $f_5$ gap changes from +0.232 on Qwen3-4B to −0.100 on Qwen3-8B, and the $f_7$ gap from +0.235 to −0.171. The demand scope changes as well, and that matters more for a diagnosis than any single gap: reasoning validity is meaningful on five of the seven benchmarks for Qwen3-4B but on only one for Qwen3-8B, verification and calibration rises from none to two, and clarity and organization remains meaningful on two (Table 11). A priority read off the weaker model would not describe the stronger one, which is consistent with a fixed ontology and a changing capability signature rather than one universal factor ranking, and it makes the grouping of § 3.1 specific to Qwen3-4B.

Demand and Supply. Reasoning validity ($f_2$) is meaningful on five benchmarks but has the lowest coverage (0.146), while completeness ($f_5$) is meaningful on four and has the highest coverage (0.474). The length controls below show that their supply contrasts have different sensitivity to response length.

Chosen responses are longer, so we also report a length-matched subset: pairs whose chosen and rejected responses differ by at most 164 characters, the lower-quartile bound of the length-difference distribution, giving $n = 9{,}551$. For each factor we additionally fit $\Delta z_k = \alpha+ \gamma z(\Delta L)$ over the full set and report $\gamma$ per standard deviation of the length difference. Across the factors we examined, $\gamma$ is 0.040 for reasoning validity, 0.021 for factual grounding, 0.175 for completeness, 0.105 for clarity, and −0.177 for efficiency.

Supply Detail and Length Matching. Table 8 reports mean contrasts, coverage, positive fractions, and tie rates. Negative contrasts are a minority for every factor (at most 0.171), the contrast SDs lie between 0.476 and 0.835, and Cohen’s $d_z$ between 0.089 and 0.193.

The 38,173 assessable pairs include 1,917 held out from training, and the same coverage values are reused in the 8B diagnosis. All factors have more positive than negative contrast mass, but $f_2$ and $f_4$ tie on about 85% of pairs. 59.2% of pairs have a longer chosen response, with mean +180 and median +106 characters. On the length-matched subset ($|\Delta L| \le164$, $n = 9{,}551$, Figure 5), mean contrasts change from 0.156 to 0.097 for $f_5$, 0.143 to 0.076 for $f_7$, 0.091 to 0.079 for $f_2$, and 0.087 to 0.094 for $f_4$. Among these four factors, reasoning validity and factual grounding change less than completeness and clarity. The length slope is $\gamma= +0.175$ for $f_5$, $+0.105$ for $f_7$, $-0.177$ for $f_8$, 0.040 for $f_2$, and 0.021 for $f_4$. This control assesses the length sensitivity of the supply diagnostic; the effect of rubric self-revision is evaluated by the matched continuations in Table 3.

Paired Effects and Results Across Datasets and Models

Table 9 reports BBH contrasts from the full-subtask evaluation. MATH-Hard changes use the separate greedy protocol of appendix § 11.§ 1.

Per-Benchmark Differences and the Matched Continuation. Five comparisons are fixed in advance, each holding pairs, labels, objective, budget, and schedule fixed: A (Common minus ordinary DPO) asks whether the broadly demanded factors help, B (Common + Specific minus Common) measures the additional specific factors, C (Common + Low minus Common) measures the additional low-demand factors, D (full rubric minus Common + Specific) measures adding Low once Specific is present, and E (Common + Specific minus Common + Low) compares the two additions. The absolute scores for these comparisons are in Table 10. Reported per-benchmark differences range from −0.53 to +0.90 percentage points. The corresponding changes in the unweighted seven-benchmark mean are +0.08, −0.02, −0.03, +0.12, and +0.01 points for A–E, respectively.

Table 3 reports the starting policy and both HelpSteer3 continuations. The shared configuration in appendix § 10.§ 3 makes RUBRICOPT-R2 (REFINE) minus RUBRICOPT-R2 (KEEP) the edit comparison. Besides the main-text BBH and MT-Bench changes, the edit changes MATH-Hard by −0.10 points and WildBench by +0.055 native-score units.

The Qwen3 and Mistral benchmark suites are in Table 1 and Table 2; factor-set results are in Table 10. INSTRUCT is the released checkpoint every method starts from, DPO is rubric-free preference optimisation on the same pairs, and RUBRICOPT-R1 is the policy after the first RubricOpt round, trained under the initial rubric $R_0$ over all eight factors. RUBRICOPT-R2 (KEEP) and RUBRICOPT-R2 (REFINE) are the two matched second-round branches: both start from the same frozen RUBRICOPT-R1, run one epoch over the same pairs with the same exposure mask and optimiser, and differ only in whether the rubric is the unchanged $R_0$ or the refined $R_1$.

The Second Dataset. The Tulu3 refined continuation starts from the merged RUBRICOPT-R1 policy and changes $f_3$, $f_4$, and $f_8$. Its absolute scores are in Table 1, and the continuation changes MATH-Hard accuracy from 43.35% to 37.12%, a −6.23-point joint change from additional training and editing.

The Second Model. Table 7 and Table 11 report the demand gaps for the two scales. The 8B block trains an ordinary-DPO method and a RUBRICOPT-R1 method on the first dataset, 566 steps each from the released checkpoint at global batch 64 with the same LoRA and seed, differing only in the rubric block. Both methods are evaluated on all seven benchmarks, and the conditioned method has 14,000 factor-scored responses.

Mistral Evaluation. Mistral-7B uses the same recipe, with Tulu3 MATH-Hard and BBH read through the extraction ladder rather than the Qwen3 protocols. Table 2 reports both suites. BBH falls by 7.95 points against its own ordinary-DPO control, with the seven-task average unchanged.