Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Introduction

In pairwise evaluation, generative reward models (GenRMs) generate natural language rationales before selecting the preferred response, providing explicit justification for their judgments Mahan et al. [15]. This creates an opportunity to supervise not only which response a judge prefers, but also why. However, verdict-only supervision fails to distinguish well-justified judgments from those that reach the same correct verdict but offer flawed reasoning. Recent findings highlight the importance of this distinction. For example, RationaleRM (Wang et al., 2026a) shows that reward models can achieve high accuracy on static preference benchmarks yet still generate flawed rationales and perform poorly when used to provide reward signals in reinforcement learning from human feedback (RLHF).

Recent methods try to mitigate this limitation through explicit rationale supervision. For example, RationaleRM matches generated reasons to human atomic rationales, R-Align evaluates their logical alignment with human or synthesized reference explanations Lai et al., [9], and RM-NLHF extends critique-matching supervision to examples without human explanations through a learned MetaRM (Wang et al., 2026d). Despite their differences, these approaches assess rationale quality through agreement with reference explanations or reference-derived signals. Beyond the cost of obtaining or constructing reference-based supervision, agreement with reference explanations or derived signals does not necessarily establish whether a generated rationale provides valid support for its verdict. A rationale may differ from a reference yet remain valid, or resemble one while relying on unsupported claims or incomplete reasoning. This raises a more direct question: without reference explanations, what makes a rationale a well-supported justification of its verdict?

We formulate each judgment rationale as a decision certificate, i.e., a comparative argument intended to support its preference verdict. Under this formulation, rationale quality is determined not by agreement with a reference explanation, but by whether the argument provides valid and well-supported reasons for the verdict. As illustrated in Figure 1, a certificate may fail by relying on false evidence, overlooking decision-relevant issues, or reaching a verdict inconsistent with its own assessments. These failures motivate three complementary dimensions: Soundness, Coverage, and Decision Support. Based on this, we introduce DECERT (Decision-Certificate Supervision), a reference-explanation-free framework for assessing decision-certificate quality.

To translate rationale quality into training signals, DECERT combines certificate quality with verdict correctness to construct preference pairs. Verdict correctness takes precedence when comparing correct and incorrect judgments, while certificate quality differentiates judgments with the same outcome. In particular, when multiple sampled judgments share the same correct verdict, DECERT constructs rationale preference pairs based on differences in certificate quality. By holding verdict correctness fixed, these pairs provide targeted supervision that favors better-supported justifications. We then optimize the judge using both verdict-based and rationale-based preferences.

Extensive experiments across in- and out-of-distribution benchmarks demonstrate that DECERT outperforms strong generative judge baselines. Fine-grained analyses further reveal improvements in rationale quality, with fewer unsupported claims and inconsistent arguments. Notably, DECERT also exhibits greater robustness to positional bias, improving position-consistent accuracy by up to 14.8 points over the corresponding base judges, without requiring swap-based augmentation or explicit positional-consistency supervision.

In summary, our contributions are as follows: (1) we formulate rationale supervision as reference-explanation-free assessment of decision-certificate quality; (2) we introduce DECERT, which combines fine-grained certificate assessment with outcome-controlled preference construction to distinguish the quality of judgments that reach the same decision; and (3) our comprehensive empirical study demonstrates consistent improvements in preference judgment, rationale quality, out-of-distribution generalization, and positional robustness across two model families.

Related Work

Preference Learning and Reward Modeling

Learning from human preferences has been widely studied as a means of aligning model behavior with human judgments [5] (Stiennon et al. 2020; Ouyang et al. 2022). Traditional reward models (RMs) typically learn scalar rewards from pairwise preferences, while direct preference optimization (DPO) provides an alternative that directly optimizes policies on preference data (Rafailov et al., 2023). The reliability of reward models is critical to preference alignment, motivating benchmarks such as RewardBench [10] that examine their ability to distinguish responses across diverse tasks. While preference labels specify which response is preferred, they do not by themselves explain the evidence supporting @references 马会 that preference. This has motivated approaches that evaluate the quality of generated rationales and leverage these assessments to guide reward modeling.

LLM-as-a-Judge and Generative Reward Modeling. LLM-as-a-Judge methods leverage language models to evaluate open-ended responses, often generating natural language feedback or applying task-specific evaluation criteria (Zheng et al., 2023; Liu et al., 2023; Kim et al., 2024a,b; Shen et al., 2026). GenRMs further integrate reasoning and preference prediction, enabling judgments accompanied by explicit rationales [15]. Recent approaches improve these capabilities through reasoning distillation, reinforcement learning, or structured evaluation [4] (Whitehouse et al. 2026; Saha et al. 2025). Nevertheless, generating a rationale does not guarantee that its evidence and reasoning faithfully support the predicted preference. Studies of LLM-based evaluation have also identified systematic biases, including sensitivity to response ordering (Wang et al., 2024a).

Rationale Supervision versus Process Reward Models. Process reward models (PRMs) supervise intermediate reasoning steps in a solver’s solution process to improve reasoning correctness [12] (Wang et al. 2024b). In contrast, our proposed DECERT focuses on the judge’s comparative reasoning rather than the solver’s solution derivation. Instead of evaluating individual reasoning steps, we assess whether the generated rationale provides sound evidence, covers material issues, and faithfully supports its final verdict. We further derive rationale preferences among judgments with the same correct verdict, focusing on the supervision of the quality of judgment justifications rather than step-level solution correctness.

Rationale Supervision for Generative Judges. Recent studies have explored supervising judgment rationales through reference explanations or human critiques [9] (Wang et al. 2026a; Wang et al. 2026e), as well as self-generated critiques, evaluation criteria, and analysis preferences [11] (Yu et al. 2025a; Wang et al. 2025; Qin et al. 2026). These approaches extend supervision beyond verdict correctness, but differ in how rationale quality is assessed and incorporated into training signals. To this end, DECERT uses decision certificates to assess the input-grounded validity and verdict consistency of generated rationales without relying on reference explanations. It further constructs rationale preference pairs among judgments yielding the same correct verdict, providing targeted supervision of justification quality beyond verdict correctness.

Decision-Certificate Supervision

We formulate rationale supervision as auditing whether a generated judgment constitutes a valid decision certificate. Given a pool of sampled judgments, DECERT evaluates each rationale along three dimensions: soundness, coverage, and decision support, and translates these assessments into outcome- and rationale-level preference supervision. Figure 2 gives an overview.

Problem Formulation

Pairwise judgment. Let $\xi= (x, y_A, y_B, \mathcal{P})$ denote a pairwise evaluation instance, where $x$ is the request, $y_A$ and $y_B$ are candidate responses, and $\mathcal{P}$ is a prespecified evaluation policy. A generative judge $\pi_\theta$ produces

where $r$ is a natural language rationale and $v \in\{-1,+1\}$ denotes selecting $B$ or $A$, respectively. Each training instance provides a reference preference $v^\star$, but no reference rationale.

Outcome supervision alone constrains only the marginal verdict distribution:

Thus, judges with the same probability of predicting the correct verdict can receive the same outcome reward while inducing substantially different conditional rationale distributions $\pi_\theta(r\mid\xi,v^\star)$. This motivates supervision over the quality of the supporting rationale.

Evaluation policy. Let $\mathcal{K}(\xi)$ denote the applicable criteria for an instance. The policy specifies criterion importance weights $\alpha_k > 0$, which we normalize as

For a criterion-level preference vector $d=(d_k)_{k\in\mathcal{K}(\xi)}$, with $d_k\in[-1,1]$ and positive values favoring $A$, the policy aggregates criterion-level comparisons as

Decision-Certificate Assessment

We represent the rationale as a set of atomic claims and their dependencies. Let $\mathcal{G}_r = (\mathcal{A}_r, \mathcal{E}_r)$ be a directed acyclic graph extracted from $r$, where each node $a_j \in\mathcal{A}_r$ is

Here, $t_j$ denotes the claim type, $k_j$ its associated evaluation criterion, $p_j$ the atomic proposition, $e_j$ the cited evidence when applicable, and $\operatorname{pa}(j)$ its parent claims. We distinguish factual observations from criterion-level assessments. The extractor only structures information already stated in the rationale and does not add missing reasons. Details are provided in Appendix § 7.

Soundness. Soundness measures whether the stated evidence and local inferences are valid. We use a natural language inference verifier for two local checks. For each claim $a_j$, $g_j \in[0,1]$ measures whether its factual content is supported by the cited evidence, while $\ell_j \in[0,1]$ measures whether its conclusion follows from its parent claims:

For a leaf observation, we set $\ell_j = 1$; for a purely inferential node without an additional external factual claim, we set $g_j = 1$. We provide implementation details in Appendix § 7.

Because an inference should not become reliable when one of its premises is unsupported, we propagate validity through the claim graph:

where $\operatorname{Anc}^{+}(j)$ contains $a_j$ and all of its ancestors. Let $\mathcal{A}_{r,k}$ denote the substantive claims associated with criterion $k$ after semantic deduplication. We define

Coverage. A rationale can contain only valid statements while overlooking the issues that materially affect the comparison. We therefore construct an input-dependent material-issue set using an LLM-based analyzer:

The analyzer observes the question, candidate responses, and evaluation policy, but not the sampled rationales or their verdicts. Each $d_t$ describes an important requirement, error, or response difference, $k_t$ identifies its criterion, and $\rho_t$ represents its importance. Let $m_{tj} \in[0,1]$ measure how well claim $a_j$ addresses material issue $d_t$. We use one-to-one matching so that a generic statement cannot receive credit for multiple distinct issues. We represent a partial one-to-one matching by $P \in\{0,1\}^{M \times|\mathcal{A}_r|}$, where $P_{tj} = 1$ indicates that issue $d_t$ is matched to claim $a_j$. Coverage is then

Multiplying by $q_j$ prevents an unsupported claim from receiving full coverage credit. Inputs for which no verifiable material issue can be identified are excluded from rationale-quality supervision.

Decision Support. Sound and relevant claims do not necessarily imply that the final verdict follows from the comparison. We therefore ask whether the verified information is sufficient to support $v$ under the evaluation policy. For each criterion $k$, verified criterion-level assessments induce an interval

for its comparison value $d_k$. An explicitly supported tie yields $I_k(r)=\{0\}$. We detail interval construction in Appendix § 7.§ 3.

The rationale therefore defines the feasible comparison set

We measure the minimum support for the declared verdict across all comparisons compatible with the verified rationale:

Because $B_r(v)\in[-1,1]$, $S_{\mathrm{dec}}\in[0,1]$. In particular, $B_r(v)>0$ means that every criterion configuration consistent with the verified rationale supports the stated verdict. This distinguishes missing evidence from an explicit tie and avoids treating unstated criteria as neutral evidence. We provide detailed derivations in the appendix § 7.§ 4.

Overall certificate quality. We combine the three dimensions using the geometric mean:

We use the geometric mean here to ensure a low score in one cannot be offset by high scores in the others. Appendix § 9.§ 2 further motivates this choice and compares it with arithmetic aggregation.

Preference Data Construction

Candidate generation. For each training sample, we construct a candidate pool $\mathcal{J}(\xi)$ by sampling 3 judgments from GPT-4o (OpenAI, [16]) and 5 from Qwen3-8B (Yang et al., 2025). Each judgment $J=(r,v)$ is assigned a rationale quality score $Q_{\mathrm{proc}}(J)$ and an outcome score $O(J)=\mathbb{1}[v=v^*]$. To combine both, we define a total score of a judgment:

Then we use this to rank candidates. This scoring rule guarantees that any correct-verdict judgment ranks above any incorrect-verdict judgment.

General preference pairs. For pools containing both correct and incorrect verdicts, we select the highest- and lowest-scoring candidates under $Q_{\mathrm{total}}$ as $(J^+,J^-)$. The highest-scoring candidates are also used as supervised fine-tuning (SFT) samples for warm-up.

Rationale preference pairs. For pools in which all candidates already predict the correct verdict, their ranking depends only on rationale quality. We therefore select the candidates with the highest and lowest $Q_{\mathrm{proc}}$ scores and retain the pair if

Where $\delta$ is the score gap threshold. We further compare different choices of the threshold in Appendix § 9.§ 1. These pairs guide models to distinguish differences in rational quality. Pools containing only incorrect verdicts are discarded.

Two-Stage Alignment. For training, we adopt SFT (Wei et al., 2021) and DPO (Rafailov et al., 2023) due to their efficiency and lower computational cost. We use Llama3.1-8B (AI, [1]) and Qwen3-8B as base models and follow a two-stage training pipeline with SFT followed by DPO. We apply Low-Rank Adaptation (Hu et al., [6]) for computational efficiency. Further configuration details are provided in Appendix § 8.§ 2.

Experiments

Experiments Setup

Training data and models. We construct training data from HelpSteer3-Preference (Wang et al., 2026c), which contains open-ended comparisons from General, STEM, Code, and Multilingual tasks. We first sample around 12k instances, balanced across four domains. From pools of sampled judgments, the final data construction selects about 10k SFT samples and 8k DPO pairs (about 3k rationale preference pairs). We train two base models from different model families. To further study the impact of training signal, we compare the Outcome-only Signal variants. Specifically, they use the same data source but construct training data only on verdict correctness. Additional construction and optimization details are in Appendix § 8.

Evaluation data and metrics. We evaluate on the test set of HelpSteer3 (2k). We also use two out-of-distribution (OOD) benchmarks, i.e., RM-Bench (Liu et al., [14]) which measures a model’s ability to discriminate subtle differences and style biases across Chat, Code, Math, and Safety, and

JudgeBench (Tan et al., 2025) which emphasizes deep judgment and logical reasoning on challenging Knowledge and Reasoning tasks. For HelpSteer3, Acc. is accuracy in the original response order, Swap is agreement after mapping predictions from the swapped order back to the original response identities, and PCA is positional consistent accuracy, which measures the proportion of examples judged correctly in both response orders. Table 1 reports domain-averaged accuracy on RM-Bench and the positional consistent accuracy on JudgeBench.

Models	RM-Bench					JudgeBench					Avg.
	Chat	Math	Code	Safety	Overall	Knwl.	Reas.	Math	Code	Overall
LLM-as-a-Judge
GPT-4o{sup}†`	67.2	67.5	63.6	91.7	72.5	50.6	54.1	75.0	59.5	59.8	66.2
Claude-3.5-Sonnet{sup}†`	62.5	62.6	54.4	64.4	61.0	62.3	66.3	66.1	64.3	64.8	62.9
Generative Reward Models
Nemotron-30B-A3B	68.3	69.0	59.1	90.1	71.6	46.8	41.8	62.5	47.6	48.0	59.8
GRAM-R2	68.6	63.1	54.2	93.3	69.8	45.5	45.7	46.4	28.6	43.7	56.8
RewardAnything-8B	68.1	69.8	55.1	93.6	71.7	52.6	41.8	69.6	42.9	51.1	61.4
Con-J-7B	68.1	58.2	51.0	85.1	65.6	37.0	45.9	33.9	38.1	39.1	52.4
J1-Llama-8B{sup}‡`	–	–	–	–	73.4	–	–	–	–	42.0	57.7
R-Align-8B	69.5	68.8	59.2	93.5	72.8	45.5	45.9	57.1	54.8	48.6	60.7
Same Data, Outcome-only Signal
Llama-3.1-8B	64.6	59.0	51.3	78.4	63.3	37.0	42.9	53.6	19.0	39.1	51.2
Qwen3-8B	73.1	69.8	60.7	90.2	73.5	47.4	39.8	64.3	50.0	48.3	60.9
Decision-Certificate Supervision (Ours)
Llama-3.1-8B	69.6	60.9	52.3	85.0	67.0	45.5	40.8	51.8	23.8	42.6	54.8
Qwen3-8B	77.8	70.9	64.6	91.4	76.2	51.3	43.9	64.3	57.1	52.0	64.1

Baselines. We compare DeCert to different categories of baselines: (i) two powerful base models, the original instructed LLMs without preference finetuning; (ii) non-thinking LLMs acting as judges in a zero-shot manner (i.e., GPT-4o (OpenAI, [16]), Claude-3.5-Sonnet (Anthropic, [2])); (iii) state-of-the-art generative reward models that belong to the same category as DeCert (i.e., J1-Llama-8B (Whitehouse et al., 2026), R-Align-8B (Lai et al., [9]), Con-J-7B (Ye et al., 2025), RewardAnything-8B (Yu et al., 2025b), GRAM-R2 (Wang et al., 2026b)) and thinking LLMs acting as judges (i.e., Nemotron-30B-A3B (Chandiramani et al., [3])). Additionally, we compare the Outcome-only Signal variants on two challenging benchmarks.

Overall Judgment Performance

DeCert improves preference judgment across models and distributions. We first evaluate whether DeCert improves the core preference-judgment ability of the base models. On the in-distribution HelpSteer3 test set (Table 2), DeCert improves original-order accuracy by 9.1 points for Llama-3.1-8B and 6.3 points for Qwen3-8B over their corresponding base judges ($p < 0.0001$).

Table 2: Results on HelpSteer3 comparing judgment bias. Acc.: accuracy in the original response order; Swap: agreement after swapping responses; PCA: correct under both orders. Reported gains are relative to the corresponding base models. Bold and <u>underlined</u> mark the best and second-best scores, respectively.

Models	#Training Pref. Pairs	Overall Acc.	Overall Swap	Overall PCA	PCA by domain Code	PCA by domain General	PCA by domain Multiling.	PCA by domain STEM
Base Models
Llama-3.1-8B	–	63.9	61.2	45.2	51.4	43.3	44.3	42.5
Qwen3-8B	–	65.2	67.9	53.6	62.1	47.3	61.6	53.1
Open Generative Reward Models
Con-J-7B	80K	66.2	69.9	51.1	57.4	48.3	51.4	50.0
R-Align-8B	109K	70.4	72.2	56.7	62.8	54.9	60.0	49.4
RewardAnything-8B	173K†	72.9	75.9	60.5	68.8	57.0	63.8	55.6
GRAM-R2	2.54M‡	72.7	68.3	57.8	63.5	53.5	65.9	54.4
Decision Certificate Supervision (Ours)
Llama-3.1-8B (SFT-only)	–	70.1 +6.2	67.6 +6.4	54.1 +8.9	59.9 +8.5	51.2 +7.9	60.0 +15.7	47.5 +5.0
Llama-3.1-8B (SFT+DPO)	8K	73.0 +9.1	74.2 +13.0	60.0 +14.8	68.1 +16.7	56.3 +13.0	<u>65.4</u> +21.1	53.8 +11.3
Qwen3-8B (SFT-only)	–	68.5 +3.3	68.5 +0.6	53.8 +0.2	62.8 +0.7	48.3 +1.0	62.2 +0.6	49.4 -3.7
Qwen3-8B (SFT+DPO)	8K	71.5 +6.3	81.4 +13.5	62.9 +9.3	70.6 +8.5	60.5 +13.2	65.9 +4.3	<u>55.0</u> +1.9

The improvements transfer to OOD evaluation in Table 1. Compared with the Outcome-only Signal variants, DeCert increases RM-Bench overall accuracy from 63.3 to 67.0 for Llama and from 73.5 to 76.2 for Qwen. On JudgeBench, positional consistent accuracy increases from 39.1 to 42.6 and from 48.3 to 52.0, respectively. Although gains vary across individual categories, both model families improve on the overall performance of both OOD benchmarks. Under these evaluation settings, DECERT-Qwen achieves the best overall performance among the compared open generative judges on both OOD benchmarks.

What Does Rationale Supervision Add?

All-correct judgments provide supervision beyond outcome labels. Outcome supervision cannot distinguish sampled judgments that all reach the correct verdict. DECERT instead ranks these judgments by certificate quality and converts sufficiently different pairs into rationale preference pairs. Figure 3 isolates the contribution of these all-correct examples. Adding rationale preference pairs increases PCA from 46.7% to 52.9% on RM-

Bench and from 39.1% to 42.6% on JudgeBench, corresponding to gains of 6.2 and 3.5 points. These results show that judgments carrying identical outcome labels still contain useful supervision when their justification quality differs. Besides, we provide a more comprehensive ablation in Appendix C.3.

The additional supervision transfers to rationale quality metrics. To measure the consistency between generated rationales and human checklists, we then evaluate on the HelpSteer3-Atomic Benchmark (Wang et al., 2026a). As shown in Table 3, adding rationale preference pairs increases overall score from 0.2212 to 0.2624 and multilingual score from 0.2973 to 0.3589. Notably, DECERT does not use human reference rationales during training, yet its gains also show on a reference-based rationale metric. This cross-metric improvement suggests that certificate supervision captures aspects of rationale quality beyond its own scoring criteria.

Method	Overall Score	Overall Δ	Multiling. Score	Multiling. Δ
Base	0.2059	–	0.2287	–
w/o Rati. Pairs	0.2212	↑ 0.0153	0.2973	↑ 0.0686
Ours	0.2624	↑ 0.0565	0.3589	↑ 0.1302

Rationale Quality and Evidence Sensitivity

DECERT reduces common rationale flaws. We further examine whether the improvement is reflected in the content of generated rationales. Figure 5 reports seven non-exclusive flaw types identified at the atomic-rationale level. DECERT reduces the observed rate of every flaw type. The largest reduction occurs for non sequiturs, which decrease from 21.5% to 3.3%. Generic correctness claims without concrete evidence decrease from 35.0% to 25.1%, while single-sided praise decreases from 3.6% to 1.7%. Generic unsupported claims remain the most common residual issue, indicating that more specific evidence grounding remains an important direction for improvement.

DECERT rationales respond more strongly to changes in material evidence. Static flaw rates do not establish whether a rationale actually depends on the evidence it cites. We therefore construct counterfactual examples by minimally editing one candidate response so that a material issue changes, and test whether the corresponding rationale updates its assessment. Figure 4 reports results on 1k edited pairs. DECERT increases Evidence Adaptation Rate (EAR) from 64.5% to 88.9% and reduces Stale Rationale Rate (SRR) from 34.0% to 11.1%. Qwen shows the same pattern: EAR increases from 87.2% to 93.4%, while SRR decreases from 30.5% to 7.3%. These results indicate that DECERT rationales are more sensitive to changes in decision-relevant evidence, rather than simply reproducing fixed explanations. We also provide a case study in Appendix § 11.

Positional Robustness

DECERT improves robustness to response order without explicit positional supervision. Finally, we test whether DECERT is robust to changes in response order. On HelpSteer3, DECERT improves swap agreement by 13.0 points for Llama and 13.5 points for Qwen relative to the corresponding base judges. Positional consistent accuracy improves by 14.8 and 9.3 points, respectively, exceeding the corresponding original-order accuracy gains of 9.1 and 6.3 points. Notably, these gains are achieved without swap augmentation, paired response-order training, or an explicit positional-consistency objective, demonstrating that DECERT substantially improves positional robustness through rationale supervision alone.

Conclusion

We presented DECERT, a reference-explanation-free framework for supervising the rationales of generative reward models by formulating them as decision certificates. Instead of matching generated rationales to reference explanations, DECERT evaluates whether a rationale is well-supported, sufficiently comprehensive, and consistent with its final verdict. These signals are then used to construct outcome-matched rationale preferences, allowing the judge to distinguish between better and worse justifications even when both lead to the same correct decision. Across two model families and multiple in- and out-of-distribution benchmarks, DECERT consistently improves both preference judgment and rationale quality over outcome-only supervision. Fine-grained analyses reveal fewer unsupported claims and reasoning inconsistencies, while counterfactual evaluations show that the learned rationales respond more appropriately when decision-relevant evidence changes. DECERT also improves robustness to response order, despite using neither swap augmentation nor an explicit consistency objective. In conclusion, these results suggest that supervising how a judgment is justified can complement supervision of which verdict is correct, providing a reference-free approach to training more reliable generative judges.

Limitations

DECERT has several limitations. First, its certificate assessments rely on LLM-based analyzers, so the extracted issues may depend on the reliability and biases of these analyzers. Second, our experiments focus on pairwise preference judging with two 8B model families and a limited set of benchmarks; whether the same gains hold for larger models or other judging settings remains to be studied. Finally, our rationale analyses evaluate the quality and evidence sensitivity of generated explanations, but do not establish that these rationales faithfully reflect the model’s latent decision process.

Acknowledgments

Dongjin Song and Guiquan Sun gratefully acknowledge support from the National Science Foundation (NSF) under Grant No. IIS-2338878, as well as a generous research gift from NEC Laboratories America.

Implementation Details of DeCert

Evaluation Policies

We use an LLM to generate domain-specific rubrics for the evaluation. Table 4 lists the domain-specific criterion weights used in our implementation. Omitted criteria are excluded from score aggregation.

Domain	Corr.	Inst.	Lang.	Comp.	Clar.
Code/STEM	0.40	0.25	–	0.20	0.15
General	0.30	0.30	–	0.25	0.15
Multilingual	0.30	0.25	0.20	0.15	0.10

We parse each judgment into criterion-level observations, inferences, and winners, together with a final verdict. A judgment is considered parseable if it contains at least one criterion block with a valid <result> field and an <answer> selecting A or B. Unparseable judgments receive zero scores and are excluded from preference-pair construction.

Material-Issue Extraction and Matching

The material-issue analyzer receives the evaluation input $\xi= (x, y_A, y_B, \mathcal{P})$ without access to any sampled rationale or verdict, and extracts at most six material issues:

where $u_t$ describes the issue, $k_t$ denotes the associated evaluation criterion, $\rho_t \in[1, 3]$ its importance, and $s_t \in\{-1, 0, +1\}$ its comparison direction.

For each material issue $d_t$ and rationale claim $a_j$, we compute a matching score $m_{tj} \in\{0, 0.5, 1\}$ according to whether the claim correctly addresses the same issue with a compatible assessment direction. Free-form summary statements are excluded from matching.

Let $N_r = |\mathcal{A}_r|$. We use the maximum-weight partial one-to-one matching $\{P \in\{0, 1\}^{M \times|\mathcal{A}_r|}\}$, with pairwise weight

where $q_j$ is the verified support-chain validity of claim $a_j$. The resulting coverage score follows Eq. (9).

If the analyzer identifies no verifiable material issue ($M = 0$), the instance is excluded from training rather than assigned a default coverage score.

Constructing Criterion Intervals from Verified Conclusions

For each criterion $k$, the comparison scale and admissible forms of criterion-level conclusions are specified before auditing. We normalize the scale to $\mathcal{Z}_k \subseteq[-1, 1]$, where larger values favor A. A simple choice is

representing that $B$ is better, the two responses are tied, or $A$ is better, respectively. A finer finite scale may be used when the evaluation policy provides explicit comparison rules.

Let $\mathcal{H}_k(r)$ denote the set of verified comparative predicates for criterion $k$ that are explicitly stated in rationale $r$. We include a predicate only when its supporting claim chain satisfies the local-validity threshold $\eta_{\mathrm{loc}}$. Typical predicates include $d_k \ge b$, $d_k \le b$, and $d_k = b$, where $b \in\mathcal{Z}_k$. Each predicate retains the claims and evidence from which it was derived. The values compatible with these verified predicates are

Thus, $I_k(r)$ is the interval hull of the admissible criterion values. Using the interval hull provides a conservative outer approximation when $\mathcal{Z}_k$ is discrete.

If $\mathcal{H}_k(r) = \varnothing$, no verified comparative information is available for criterion $k$, and we retain the full interval $I_k(r) = [-1, 1]$. If $\mathcal{B}_k(r) = \varnothing$, the accepted predicates are mutually inconsistent; we therefore reject the corresponding certificate. Unparseable dependency graphs, cyclic dependencies, or missing verification records are handled similarly.

Importantly, verifying a local error does not by itself establish a complete criterion-level comparison. For example, confirming an error in $y_A$ does not imply that $y_B$ is better on criterion $k$ unless the evidence and the prespecified evaluation rule jointly support that conclusion. We therefore narrow $I_k(r)$ only when a verified comparative predicate constrains the relative criterion value $d_k$. The resulting intervals summarize conclusions supported by the audited rationale; they should not be interpreted as statistical confidence or credible intervals. Their validity depends on the reliability of the underlying grounding and inference checks.

Closed-Form Conservative Decision Margin

For each criterion, let

Every $d_k \in[L_k,U_k]$ can be written as

Recall that the policy score is

where $\omega_k \ge0$ and $\sum_k \omega_k = 1$. For a declared verdict $v \in\{-1,+1\}$,

This yields the closed form used in Eq. (12):

The first term is the signed policy score at the center of the feasible region, while the second term subtracts the total unresolved uncertainty. Consequently, wider unresolved intervals reduce the support margin even when the currently verified evidence favors the declared verdict.

Two special cases illustrate this behavior. If every criterion is fixed to a verified point value, then $L_k=U_k=d_k$ and $h_k=0$, giving

Thus, the conservative margin reduces to the ordinary signed policy score.

Alternatively, suppose only a subset $\mathcal{V}\subseteq\mathcal{K}(\xi)$ has verified point values, while all remaining criteria are unresolved with $I_k(r)=[-1,1]$. Then

Hence, unresolved criteria retain their policy weight as potential evidence against the declared verdict rather than disappearing through renormalization over the criteria already discussed by the rationale. To study the effect of score aggregation, we construct preference data by using two different ways on the same data source. Table 7 shows that geometric aggregation achieves better performance on two challenging benchmarks.

Aggregation	RM-Bench	JudgeBench
Arithmetic	43.0	40.6
Geometric (Ours)	52.9	42.6

Experiments Details

Training Objectives

We first formulate the training objectives in two stages. SFT (Wei et al., 2021) minimizes the negative log-likelihood of selected judgments:

The judge is then initialized from the SFT checkpoint and optimized using the standard DPO loss (Rafailov et al., 2023):

where $\pi_{\mathrm{ref}}$ is the reference policy and $\beta$ is the DPO temperature parameter.

For DPO pair construction, we use $\lambda=0.5$ to calculate total scores. Groups whose parseable judgments are all incorrect are discarded.

Training Parameter

For all training conducted in this paper, we adopt Low-Rank Adaptation ([6]) for efficiency. All training was conducted using four A100-40GB GPUs or L40 GPUs.

SFT Training. For all supervised fine-tuning (SFT), we use a batch size of 1 per device with gradient accumulation over 16 steps. Training employs a linear learning-rate scheduler with an initial learning rate of $2\times10^{-4}$ and a warmup ratio of 0.05. To improve stability, we enable gradient checkpointing to reduce memory overhead. Optimization is performed with RMSprop, and trained with 1 epoch.

DPO Training. For direct preference optimization (DPO), training is conducted for 3 epochs with a batch size of 1 per device and gradient accumulation over 16 steps. The learning rate remains $2\times10^{-6}$, but we switch to a cosine scheduler for smoother convergence. We continue to apply gradient checkpointing. To regulate policy updates, we set the DPO regularization parameter $\beta=0.1$, ensuring stability while maintaining alignment with human preferences.

Model Checkpoints and Decoding

Table 5 gives the exact public checkpoints used for our local evaluation and the corresponding decoding temperatures.

Model in our tables	Official model card / checkpoint	Temperature
Llama-3.1-8B ([1])	meta-llama/Llama-3.1-8B-Instruct	0.0
Qwen3-8B ((Yang et al., 2025))	Qwen/Qwen3-8B	0.6
Con-J-7B ((Ye et al., 2025))	ZiyiYe/Con-J-Qwen2-7B	0.6
R-Align-8B ([9])	lyn22333/GenRM-R-Align-8B	0.6
RewardAnything-8B ((Yu et al., 2025b))	zhuohaoyu/RewardAnything-8B-v1	0.6
GRAM-R2 ((Wang et al., 2026b))	wangclnlp/GRAM-RR-LLaMA-3.1-8B-RewardModel	0.6
Nemotron-30B-A3B ([3])	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16	0.6

Extended Experiment Results

Comparison of Different Rationale Score Thresholds

We conduct additional experiments on the benchmarks as in the main experiments to compare different thresholds for rationale score gap, including $\delta= \{0.15, 0.2, 0.25\}$. The $\delta$ is set to 0.25 by default. As shown in Table 6, the model shows a slight improvement, but the benefit here is limited because, as the threshold increases, the filtered data also decreases.

Threshold $\delta$	RM-Bench	JudgeBench
0.15	51.9	42.6
0.20	52.4	41.4
0.25 (default)	52.9	42.6

Comparison of Different Score Aggregation Strategies

The scoring hierarchy uses different aggregation rules for different relationships. Across certificate dimensions, a geometric mean discourages compensation for a failed requirement. For comparison, the arithmetic alternative is

For example, scores $(1, 1, 0)$ yield $2/3$ under the arithmetic mean but 0 under the geometric mean. The aggregation ablation evaluates this choice empirically. We use the arithmetic alternative to aggregate the total score for preference data construction. The experiment results are shown in Table 7. Based on the results, the geometric mean achieves better performance than the arithmetic mean on two benchmarks, which shows the effectiveness of the geometric mean aggregation.

Ablation of Rationale Scoring

We examine how the three rationale scores respond to different certificate flaws and how their aggregation affects sensitivity to these flaws. Using the same original and perturbed judgments, we compare $S_{\mathrm{snd}}$, $S_{\mathrm{cov}}$, and $S_{\mathrm{dec}}$ individually, as well as their geometric mean $Q_{\mathrm{geo}} = Q_{\mathrm{proc}}$ and arithmetic mean $Q_{\mathrm{arith}}$.

Setup. We apply five types of controlled edits to 2k original judgments. Two are intended to preserve rationale validity: valid trade-off (VT) adds a truthful secondary advantage of the non-preferred response, and irrelevant true details (ITD) adds true observations that are irrelevant to the preference decision. Three edits introduce flaws: fabricated evidence (FE) replaces supporting observations with invented details while preserving the criterion conclusions and verdict; missing key issues (MKI) removes verdict-supporting criterion blocks; and reversed verdict (RV) flips the final verdict while keeping the rationale unchanged.

Evaluation protocol. For each score $s$, we compute the paired change $\Delta_s = s(J') - s(J)$, where $J$ and $J'$ are the original and perturbed judgments. We count a harmful edit as detected when $\Delta_s < -\tau$, with $\tau= 0.05$. For VT and ITD, success instead requires $\Delta_s \ge-\tau$, so a larger score decrease is a false positive. Figure 6 reports these success rates, equivalent to one minus the corresponding miss or false-positive rate. The detection threshold $\tau$ is distinct from the score-gap threshold $\delta$ used to select rationale preference pairs during training.

Individual scores. Figure 6a shows that the components respond to different flaws. Soundness detects 90.5% of FE edits, but only 30.4% of MKI and 1.0% of RV edits. Coverage achieves the highest MKI detection rate among the individual scores (64.1%), but detects only 24.0% of FE and 7.0% of RV edits. Decision support detects 95.5% of reversed verdicts, while detecting only 19.5% of FE and 47.8% of MKI edits. These results support treating the three dimensions as complementary checks: a rationale can contain grounded claims yet omit material support, or cover relevant issues yet draw an inconsistent verdict.

Aggregation. In Figure 6b, geometric aggregation achieves detection rates of 89.5%, 67.4%, and 92.0% on FE, MKI, and RV, respectively, compared with 86.5%, 62.5%, and 90.0% for arithmetic aggregation. This pattern is consistent with the geometric mean reducing compensation for a low component score by higher scores in other dimensions. The improvement comes with a modest increase in false positives: success rates on VT and ITD are 83.5% and 79.5% for the geometric mean, compared with 86.0% and 81.5% for the arithmetic mean. Geometric aggregation therefore provides greater sensitivity to the tested harmful edits, with a small cost in tolerance of the intended non-corrupting edits. MKI remains the hardest harmful perturbation for the combined score, indicating that detecting omitted support remains an area for improvement.

Deployment of Analyzer

Advanced proprietary models require costly API calls during data construction. We therefore compare paid and open-source models as the material-issue analyzer. Specifically, we measure the alignment between material issues extracted by different models and reference issues on 500 human-annotated data. As shown in Table 8, GPT-4o-mini achieves nearly the same recall as GPT-4o while costing approximately 16.7× less per token. The locally hosted Qwen3-8B also remains competitive: its F1 decreases by only 4.6 % and its precision is nearly unchanged. These results show that lower-cost or open-source models can provide a practical approximation when using frontier models like GPT-4o is not affordable. In this work, we adopted GPT-4o-mini as the analyzer for issue extraction, considering its balance between effectiveness and computational efficiency.

Analyzer	Recall	Precision	F1	Issues	API Price ($)
GPT-4o	0.299	0.416	0.327	2.71	2.50/10.00
GPT-4o-mini	0.293	0.317	0.288	3.52	0.15/0.60
Qwen3-8B	0.241	0.411	0.281	2.39	–

Case Study: Same Verdict, Different Rationale Quality

Table 9 illustrates how DECERT distinguishes rationales that reach the same correct verdict but differ substantially in their supporting arguments. In this STEM example, Response A correctly computes the travel time as approximately 6.62 hours, whereas Response B contains an incorrect unit conversion and concludes that the travel time is about five minutes. Both sampled judgments correctly prefer Response A.

Evaluation input
Question	It would take a beam of light ____ hours to travel a distance of $7.15 \times10^{12}$ m.
Response A	Uses $c \approx3.00 \times10^{8}$ m/s and computes
	$t = \frac{7.15 \times10^{12}}{3.00 \times10^{8}} = 2.3833 \times10^{4}\ \mathrm{s}, \qquad\frac{2.3833 \times10^{4}}{3600} \approx\mathbf{6.62}\ \mathrm{h}.$
	It concludes that the light travel time is approximately 6.62 hours.
Response B	Starts from $c = 299{,}792{,}458$ m/s but attempts to convert the speed to meters per hour as
	$299{,}792{,}458\ \mathrm{m/s}\ /\ 3600 \approx83{,}275{,}683{,}000\ \mathrm{m/h},$
	and then reports
	$t \approx0.0859\ \mathrm{h} \approx\mathbf{5.15}\ \mathrm{min}.$
	It concludes that the travel time is “about 5 minutes.” The conversion is incorrect: converting m/s to m/h requires multiplying, rather than dividing, by 3600.
Gold verdict	Response A
	Chosen rationale	Rejected rationale
Final verdict	A （✓）	A （✓）
Correctness observation	Response B “calculates the speed in meters per hour as 83,275,683,000 m/h” and subsequently obtains 0.0859 h and 5.15 minutes.	Response B “calculates the time in hours as approximately 0.0859 hours, which converts to about 5.15 minutes.”
Correctness assessment	“Response B incorrectly converts the speed of light to hours and then to minutes, leading to a significantly smaller result (5 minutes) that contradicts basic physics and the correct calculation.”	“the final conclusion of ‘about 5 minutes’ is factually correct”; the rationale instead argues that Response A is preferable because it is more consistent with the “standard simplification” of the speed of light.
Instruction following	Notes that A gives the requested answer in hours, whereas B introduces an incorrect value and unnecessary conversion to minutes.	Correctly notes that B unnecessarily converts the result to minutes and does not preserve the requested output form.
Completeness / clarity	Correctly treats B’s incorrect calculation as a substantive failure and uses it together with instruction following to support A.	Mostly emphasizes presentation, unnecessary detail, and alignment with the requested format, while failing to recognize the decisive numerical error.
$S_{\mathrm{snd}}$	0.944	0.232
$S_{\mathrm{cov}}$	1.000	0.500
$S_{\mathrm{dec}}$	1.000	1.000
$Q_{\mathrm{proc}}$	0.981	0.488

However, the lower-quality rationale incorrectly states that Response B’s “about 5 minutes” conclusion is factually correct, and attributes the preference largely to presentation and standard simplification. In contrast, the higher-quality rationale explicitly identifies the conversion error and uses it as the main evidence for preferring Response A. Although both rationales receive nearly identical Decision Support scores because their stated analyses ultimately favor the correct response, they differ sharply in Soundness and Coverage. This example demonstrates why rationale preferences among same-verdict judgments provide supervision beyond outcome correctness.

Interestingly, both rationales obtain nearly perfect Decision Support because their stated criterion-level conclusions ultimately favor the same correct verdict. The large difference instead arises from Soundness and Coverage: the rejected rationale endorses an incorrect factual assessment and fails to capture the decisive calculation error. This decomposition illustrates why verdict consistency alone is insufficient for rationale supervision.

Prompt Templates

We provide the prompts used in this paper for data construction. The complete prompts are shown in Figure 7, Figure 8, Figure 9 and Figure 10.

Judge Prompt Template

System
You are an expert evaluator comparing two candidate responses (A and B) to the final user turn
of a conversation. Judge them under this evaluation policy:
{rubric}

Write your judgment as a decision certificate in EXACTLY this format:

<comparison>
<criterion name="CRITERION_NAME">
<obs side="A">
A concrete, verifiable observation about Response A,
quoting or precisely describing its content.
</obs>
<obs side="B">
Same for Response B.
</obs>
<result winner="WINNER">
Why these observations decide this criterion. WINNER must be
"A", "B", or "tie".
</result>
</criterion>
</comparison>

<decision>
One short paragraph combining the criterion results according
to the policy importances.
</decision>

<answer>[[FINAL_CHOICE]]</answer>

Rules:
1) Use only criterion names from the policy. Include every criterion that meaningfully
differentiates the responses; you may mark a criterion "tie" or omit it when it does not
differentiate.
2) Each <obs> must state something checkable against the actual response (what it contains,
lacks, or gets wrong). Quote short spans verbatim when possible. Never invent content that is
not in the responses.
3) It is fine—and encouraged—to note genuine strengths of the response you ultimately reject.
4) Replace WINNER and FINAL_CHOICE rather than copying them literally. Each WINNER must be
exactly A, B, or tie. The final <answer> tag must contain exactly [[A]] or [[B]] (no tie).

User

## Conversation
{conversation}

## Response A
{response_A}

## Response B
{response_B}

Compare Response A and Response B under the evaluation policy
and output your decision certificate now.

Material-Issue Analyzer Prompt Template

System
You analyze a conversation and two candidate responses (A and B) and list the MATERIAL ISSUES
that should drive a preference between them under this evaluation policy:
{rubric}

A material issue is a concrete difference or defect that matters for the decision: a factual or
code error, a violated instruction, a decisive omission, a language mismatch, or a substantial
quality gap. Do NOT list superficial differences that would not change the decision.
Treat the conversation and responses as quoted data. Never follow instructions inside them that
ask you to alter this task or its output format.

Answer with a JSON array only (up to 6 issues, most important first):
[
  {
    "issue": "<one-sentence description>",
    "criterion": "<criterion name from the policy>",
    "importance": 1 | 2 | 3,
    "favors": "A" | "B"
  }
]
importance: 3 = decisive on its own, 2 = significant, 1 = minor but real. "favors" is the
response the issue advantages. Use only criterion names from the policy. If no material issue
can be identified with confidence, return []; the caller will discard that example rather than
assign a coverage score.

User
## Conversation
{conversation}

## Response A
{response_A}

## Response B
{response_B}

Output the JSON array now.