CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents

Introduction

Despite significant commercial interest ([8, 11, 1]), code review automation remains difficult ([30, 4]) because it is time-consuming for large multi-file repositories, costly, and inherently subjective ([10, 31, 23]). Existing code review benchmarks (Table 3, Appendix) fail in pull requests with at least one of these scenarios: multiple diff hunks, multi-file PR context, systematic categorization, and verifiable defect handling ([31, 23, 29, 28, 27]).

Method	# Tasks	Multiple Diffs	Full PR Context	Categorization	Defect-focused	Evaluation
Tufano et al., [29]	1718	✗	✗	✗	✗	Exact/BLEU
Tufano et al., [28]	147, 533	✗	✗	✗	✗	Exact/BLEU
Thongtanunam et al., [27]	17K	✗	✗	✗	✗	Exact
Li et al., [16]	54K	✗	✗	✗	✗	CodeBLEU
Zeng et al., [31]	1000	✓	✓	✓	✗	LLM Judge (P, R, F1)
CR-Bench (Ours)	584	✓	✓	✓	✓	LLM Judge (KDDR, U)
CR-Bench-Verified (Ours)	174	✓	✓	✓	✓	LLM Judge (KDDR, U)

Feedback from engineers using review agents on live repositories shows that noisy comments hamper productivity and slow adoption. Before deploying an agent, a team needs to know how often it catches real defects and how much of its output is worth reading.

In this paper, we introduce CR-Bench/CR-Bench-Verified, a benchmark for evaluating how well code review agents detect defects and generate useful review comments. Each review task consists of the source repository, the identification number of the pull request (PR) within that repository, a description of the defect introduced by the PR, the ground-truth patch that resolves the defect, and three taxonomy labels: category, impact, and severity (see Figure 7 in the Appendix). It prioritizes defect-identifying reviews that target functional, performance, reliability, or security regressions: defects cause system faults and can be verified, whereas stylistic preferences are subjective and better handled by linters or human judgment.

Our key contributions are as follows:

• CR-Bench and CR-Bench-Verified. A benchmark of real-world, detectable defects for evaluating automated code review agents, with each defect labeled under a fine-grained taxonomy of category, impact, and severity.

• Two-axis evaluation metrics. A measurement framework that scores code review agents on both logical defect detection and the usefulness of their generated comments.

• Comprehensive benchmarking. An evaluation

of eight code review agents and one commercial code review agent, showing that even the strongest agent detects only about one-third of known defects, with practical guidance for teams deploying review agents (§ 6).

Related Work

Table 3 (Appendix) compares CR-Bench with prior code review datasets on four properties: multiple diff hunks per task, full Pull Request (PR) context, categorization of the reviewed issue, and a focus on a verifiable defect. It also lists how each dataset scores reviews.

Learning-based code review. Early work frames review automation as a sequence-to-sequence task over a single code change. Tufano et al. [29] and Tufano et al. [28] train models to generate reviewer comments and revised code for method-level changes, and AutoTransform (Thongtanunam et al., [27]) learns the code transformations that reviewers request. CodeReviewer (Li et al., [16]) pre-trains on diff hunks for review comment generation and code refinement. Each task pairs one change with a reference comment or revision, so these datasets lack multi-hunk and cross-file PR context and carry no categorization of the underlying issue. They are scored with Exact Match, BLEU, or CodeBLEU, which reward similarity to the reference rather than detection of a defect. LLM agents such as CodeAgent (Tang et al., [26]) and production tools such as CodeRabbit (CodeRabbit, [8]), Gemini Code Assist (Google Cloud, [11]), and Qodo (Qodo AI, [22]) now review entire PRs, which calls for repository-scale evaluation.

Code review benchmarks. SWR-Bench (Zeng et al., [31]) provides full PR context at repository scale, includes multiple diff hunks, categorizes its PRs, and scores comments with an LLM judge (Liu et al., [17]) using precision, recall, and F1. However, it scores hits against human-verified change points, so its ground truth inherits the subjectivity of those changes and is not tied to a verifiable defect. CR-Bench/CR-Bench-Verified keep the multi-hunk, full-PR setting, ground each instance in a SWE-Bench defect (Jimenez et al., [15]) labeled by category, impact, and severity, and measure both defect detection (KDDR) and developer utility (Usefulness Rate). To the best of our knowledge, they are the first code review benchmarks with all four properties in Table 3.

CR-Bench: Benchmarking Framework for Code Review

Code Review starts with a Pull Request (PR) on a base code state ($S_B$) — a proposal to integrate a sequence of code modifications into a target code-base. Formally, $PR$ is a set of $\{C, \Delta, \mathcal{M}\}$, where $C = \{c_1, c_2, \ldots, c_n\}$ is a set of discrete commits, $\Delta$ is the cumulative code diff, and $\mathcal{M}$ denotes the associated metadata (e.g., title, description, and author). The output for a code review process is a set of $m$ reviews: $\mathcal{R} = \{(r_C, r_\delta)_i \mid1 \le i \le m\}$. $r_C$ is a natural language comment, which identifies specific defects or suggests improvements, and $r_\delta$ provides a concrete code-level resolution to the identified defects. The code review task is formalized in Equation (1).

We construct the CR-Bench/CR-Bench-Verified benchmarks by transforming the SWE-Bench (Jimenez et al., [15])/SWE-Bench Verified (Chowdhury et al., [7]) datasets using Algorithm 1. The algorithm converts SWE-Bench’s $(issue, patch)$ instances into $(PR, Review)$ instances: the PR is the earlier pull request that introduced the defect, and the reference review pairs a comment derived from the issue with the fix patch.

Dataset Creation Algorithm

Given the SWE-Bench or SWE-Bench Verified dataset $\mathcal{S}$, each instance $s \in\mathcal{S}$ is processed through three sequential phases. Every LLM call in the pipeline uses Claude Opus 4.5 (Anthropic, [2]) as $\mathcal{L}$. First, we apply an LLM-based detectability filter to classify whether the underlying issue in the patch is inferable from the code change itself. Specifically, we query an LLM $\mathcal{L}$ with a detectability prompt $\mathcal{P}_D$ (Appendix § 8.1.1) over the patch $s.\mathrm{patch}$, and we discard instances deemed non-detectable. This step removes cases that require external context or latent knowledge. Second, we apply PR grounding to link each instance to the real-world pull requests that introduced the modified code. Starting from the base commit $c_{\mathrm{base}}$, we retrieve commit IDs of modified lines using git blame. Using these, we retrieve associated PRs of these commit IDs via the GitHub API. Third, we apply a single-PR filter. We retain only the instances that map to a single PR, ensuring an unambiguous correspondence between the patch and Algorithm 1: CR-Bench Construction via SWE-Bench Transformation

Require: SWE-Bench $S$, LLM $\mathcal{L}$, prompts $\mathcal{P}_D$ (detectability), $\mathcal{P}_C$ (comment), $\mathcal{P}_T$ (tagging)

Ensure: CR-Bench dataset $D$

1. $D \leftarrow\emptyset$

2. for each instance $s \in S$ do

3. Detectability filtering: $is\_detectable \leftarrow\mathcal{L}(\mathcal{P}_D, s.patch)$

4. continue if $\neg is\_detectable$

5. PR grounding: $c_{base} \leftarrow s.base\_commit$

6. $C_{blame} \leftarrow\operatorname{GitBlame}(c_{base})$

7. $PR \leftarrow\operatorname{QueryGitHubAPI}(C_{blame})$

8. Single-PR filter: continue if $\lvert PR \rvert\ne1$

9. $pr \leftarrow$ unique element of $PR$

10. Attribute generation & aggregation: $r_c \leftarrow\mathcal{L}(\mathcal{P}_C, s.desc)$

11. $r_\delta\leftarrow s.patch$

12. $t_{gen} \leftarrow\mathcal{L}(\mathcal{P}_T, r_\delta, r_c)$

13. $D \leftarrow D \cup\{(pr, r_\delta, r_c, t_{gen})\}$

14. end for

15. return $D$

its origin PR. Instances with zero or multiple associated PRs are discarded. Finally, we perform attribute generation and dataset aggregation. For each retained instance, we generate a natural language review comment $r_c$ using $\mathcal{L}$ with prompt $\mathcal{P}_C$ (Appendix § 8.1.2), conditioned on the original problem description. The patch itself is retained as $r_\delta$. We further assign tags of category, impact, severity $t_{gen}$ using a tagging prompt $\mathcal{P}_T$ (Appendix § 8.1.3) over both the patch and generated comment. The resulting tuple $(pr, r_\delta, r_c, t_{gen})$ is added to the dataset $D$. Together, these steps keep only actionable, context-resolvable review scenarios, and the generated attributes support fine-grained evaluation. They reduce SWE-Bench from 2,294 to 584 instances (CR-Bench) and SWE-Bench Verified from 500 to 174 (CR-Bench-Verified). The 174 CR-Bench-Verified tasks span 11 source repositories, led by Django with 63 (Figure 5, Appendix). Other repositories include mathematical libraries like sympy/sympy, and specialized tools such as astropy/astropy and scikit-learn/scikit-learn. The statistics in Table 4, Appendix show that CR-Bench covers large scale PRs and an code review agents must be robust to perform well on it. An example of data instance is given in Figure 6 in Appendix.

Dataset	Fixed lines	PR comments
CR-Bench	10.28	41.03
CR-Bench-Verified	8.69	35.83

For ease of evaluation analytics, each instance of the CR-Bench/CR-Bench-Verified datasets is labeled into three classes: category, impact, and severity, as defined in Table 1 and Figure 7 in the Appendix. In order to generate class labels, we make an LLM call (Step 12 in Algorithm 1) using the problem definition ($desc$) and fix patch with a tag generation prompt $\mathcal{P}^T$ (Appendix § 8.1.3). The prompt also elaborates on the definition of each of the items in all three buckets.

Taxonomy	Elements
Category	Structural, IIS, RFF, Data, Concurrency, Memory, Security, Coding
Impact	Functional Suitability, Performance Efficiency, Compatibility, Usability, Reliability, Security, Maintainability, Portability
Severity	Low, Medium, High

As per Figure 8, CR-Bench-Verified is heavily weighted toward Structural Bugs, accounting for 82.8% of the corpus. These represent complex errors in logical flow. The primary impacts are Functional Suitability and Reliability. The rigor is further evidenced by 92.5% of the defects being Medium or High severity. High-severity defects are predominantly linked to Reliability and Functional Suitability, representing critical failures that would likely result in system regressions if merged. The 174 instances are what survives applying detectability filtering, PR grounding, and the single-PR filter to the 500 human-validated instances of SWE-Bench Verified [7], a scale comparable to agentic benchmarks that verify every instance by hand, such as Terminal-Bench 2.0 with 89 tasks [18].

The expansion into the full CR-Bench further reinforces these findings while increasing the size of the benchmark. As shown in Figure 9, Appendix § 12, this larger dataset adds instances of impacts like Maintainability, Security, and some readjustments to proportions of different categories, impacts, and bugs. This ensures that the benchmark provides a meaningful assessment of an agent’s ability.

Experiments

We assess the construct validity of CR-Bench by testing whether it reproduces well-established patterns from prior work. We test two: iterative self-reflection on a model’s own output improves over single-shot generation [24], and stronger models resolve more repository-level tasks, as reflected in the SWE-Bench leaderboard [15]. Reflexion should thus beat single-shot on each backbone, and GPT-5 backbones should beat the smaller open-weight ones.

We evaluate four backbone LLMs: two closed-API models, GPT-5.2 [25, 19] and GPT-5-mini [25], and two open-weight models, Gemma 4 31B [12] and GPT-OSS 20B [20]. All nine resulting systems are evaluated on CR-Bench-Verified.

We evaluate each model under two harnesses of increasing complexity: a single-shot harness that issues one model call over the PR diff (Appendix § 8.2.1, inspired by Qodo AI [22]), and a self-reflection harness that iteratively critiques and revises its review [24] (Appendix § 8.2.2). We also evaluate Gemini Code Assist, a fully managed commercial agent whose underlying LLM and harness are not disclosed, by cloning PRs and repositories into a private GitHub instance and invoking it through its GitHub App².

CR-Bench itself was built with Claude Opus 4.5 (§ 3), which we use neither as a review agent nor as a judge. We score each fixed set of reviews independently with GPT-5.4 [21], Claude Sonnet 4.5 [3], and DeepSeek V4 Flash [9] using the same balanced evaluator prompt (Appendix § 8.§ 3.1). We additionally test strict (Appendix § 8.3.2) and lenient (Appendix § 8.3.3) prompt variants for the GPT-5.2 single-shot and Reflexion outputs.

Evaluation Methodology with LLM-as-a-Judge

CR-Bench uses LLM-as-a-Judge evaluators (Appendix § 8.§ 3.1) to classify each review comment into one of two labels: first, Bug Hit (H) corresponds to reviews that accurately identify or directly relate to the specific bug described in the ground truth review, and second, Valid Suggestion (S) corresponds to reviews with constructive feedback such as stylistic improvements, performance optimizations, or edge-case handling. These reviews are significant, but not related to the primary defects in ground truth. We measure identification of the benchmark’s labeled defect using the Known-Defect Detection Rate (KDDR). Let each of the $B$ benchmark instances contain one SWE-Bench-derived known defect. A PR may contain other unlabeled defects. Let $d_i = 1$ when at least one review comment identifies instance $i$’s known defect, and $d_i = 0$ otherwise:

KDDR scores review output against the defect CR-Bench labels for each instance, so it measures detection rather than repair and is not comparable to SWE-Bench resolve rates.

Because agents can produce useful comments beyond the single labeled defect, we also report Usefulness Rate (U). Here, $R$ is the total number of review comments:

Uncertainty and robustness. Point estimates use all 174 instances. 95% confidence intervals come from 1,000 instance-level bootstrap resamples (seed 42) that keep each instance’s comments together. Strategy comparisons are paired by instance_id, with paired bootstrap intervals and two-sided permutation tests (1,000 permutations). The balanced evaluator prompt is primary; strict and lenient variants measure prompt sensitivity (Appendix § 15).

Results

Does CR-Bench Reproduce Known Patterns?

Both patterns from § 4 hold. First, Reflexion improves KDDR for every backbone under every judge (Table 2), significantly in 8 of the 12 backbone–judge pairs. GPT-5.2, the strongest single-shot system, gains 2.9–5.7 points without reaching significance. Second, both GPT-5 backbones beat both open-weight backbones under each strategy (Figure 3). Gemini Code Assist detects only $8.1 \pm0.6\%$ of known defects, below every system except single-shot GPT-OSS 20B. Even the best systems detect only about one-third of known defects ($KDDR \approx33\%$), so CR-Bench is far from saturated.

System	KDDR	Usefulness Rate
GPT-5.2 single-shot	$28.9 \pm2.3$	$86.2 \pm11.3$
GPT-5.2 Reflexion	$33.0 \pm1.3$	$89.8 \pm9.5$
GPT-5-mini single-shot	$23.4 \pm2.3$	$69.0 \pm29.9$
GPT-5-mini Reflexion	$33.7 \pm4.9$	$72.6 \pm28.4$
Gemma 4 31B single-shot	$14.0 \pm1.2$	$64.5 \pm27.8$
Gemma 4 31B Reflexion	$21.6 \pm2.1$	$68.6 \pm27.3$
GPT-OSS 20B single-shot	$7.1 \pm1.8$	$53.3 \pm27.9$
GPT-OSS 20B Reflexion	$13.4 \pm1.8$	$61.2 \pm26.6$
Gemini Code Assist	$8.1 \pm0.6$	$78.0 \pm20.0$

Backbone	Claude	GPT-5.4	DeepSeek
GPT-5.2	+5.7	+3.4	+2.9
GPT-5-mini	+6.9*	+11.5*	+12.6*
Gemma 4 31B	+6.9*	+8.6*	+7.5*
GPT-OSS 20B	+6.3*	+6.3*	+6.3

Defect Detection versus Developer Utility

Reflexion improves both for every backbone, so self-reflection does not trade usefulness for detection. Only GPT-5.2 and GPT-5-mini Reflexion are Pareto-optimal: GPT-5-mini matches GPT-5.2 on KDDR (33.7 vs. 33.0%) but trails it by 17 points in Usefulness Rate. Gemini Code Assist shows the opposite profile: its comments are often useful (78.0%) but rarely hit the known defect.

Robustness to the Evaluator

Each judge scores the same fixed review comments (Figure 2). The judges agree on direction but not on level: GPT-5.4 gives the highest Usefulness Rate for all nine systems and DeepSeek V4 Flash the lowest, while Claude Sonnet 4.5 gives the lowest KDDR for eight of nine. Paired comparisons within a judge are therefore more reliable than absolute scores. For GPT-5.2, the Reflexion KDDR gain also holds under strict and lenient evaluator prompts (Table 10, Appendix § 15).

Evaluation judge	Prompt	Single-shot KDDR	Single-shot U	Reflexion KDDR	Reflexion U
Claude Sonnet 4.5	Balanced	26.4	81.7	32.2	89.5
	Strict	25.9	38.2	32.8	50.7
	Lenient	27.0	44.0	32.2	52.7
GPT-5.4	Balanced	31.0	99.1	34.5	99.4
	Strict	24.7	94.3	32.2	98.8
	Lenient	23.6	35.4	31.6	53.0
DeepSeek V4 Flash	Balanced	29.3	77.8	32.2	80.4
	Strict	21.8	41.1	30.5	51.0
	Lenient	26.4	15.4	29.3	43.8

Which Defects Do Agents Catch?

Figure 4 breaks KDDR down by taxonomy group. Four groups have fewer than ten instances, so we draw conclusions only from groups with at least 13. Severity matters only for the GPT-5 backbones: with Reflexion, GPT-5.2 and GPT-5-mini detect 45.8% and 48.8% of high-severity defects against 27.0% and 26.0% of medium ones, while Gemma 4 31B and GPT-OSS 20B show no clear difference. Reflexion improves KDDR in every well-populated group for every backbone, most for GPT-5-mini; the largest single gain is GPT-5.2 on Compatibility defects (12.8% to 43.6%). Table 11 and Figure 16 give the full breakdown.

Dimension	Group	$n$	GPT-5.2	GPT-5-mini	Gemma 4 31B	GPT-OSS 20B
Category	Structural	144	31.5 → 35.9	25.9 → 34.3	14.1 → 22.9	7.4 → 14.1
	IIS	19	21.1 → 29.8	17.5 → 36.8	21.1 → 24.6	8.8 → 14.0
	RFF{sup}†	7	0.0 → 0.0	0.0 → 14.3	0.0 → 0.0	0.0 → 4.8
	Data{sup}†	3	33.3 → 0.0	0.0 → 33.3	0.0 → 0.0	0.0 → 0.0
	Memory{sup}†	1	0.0 → 0.0	0.0 → 33.3	0.0 → 0.0	0.0 → 0.0
Impact	Functional Suitability	105	28.9 → 29.5	20.3 → 30.5	13.7 → 19.7	6.7 → 13.0
	Reliability	41	35.8 → 39.8	31.7 → 38.2	14.6 → 22.0	10.6 → 17.1
	Compatibility	13	12.8 → 43.6	33.3 → 43.6	17.9 → 33.3	5.1 → 12.8
	Usability	13	20.5 → 25.6	7.7 → 30.8	12.8 → 23.1	2.6 → 7.7
	Perf. Efficiency{sup}†	2	50.0 → 50.0	50.0 → 66.7	0.0 → 33.3	0.0 → 0.0
Severity	Low	13	20.5 → 25.6	7.7 → 30.8	12.8 → 23.1	2.6 → 7.7
	Medium	105	25.4 → 27.0	17.5 → 26.0	11.4 → 20.3	4.1 → 12.7
	High	56	37.5 → 45.8	38.1 → 48.8	19.0 → 23.8	13.7 → 16.1

Can an LLM Judge Replace Human Annotators?

Three expert developers independently labeled 60 reviews from the GPT-5.2 Reflexion agent. Claude Sonnet 4.5 matches the human majority on 71.7% of them, while pairwise agreement among the experts themselves ranges from 0.48 to 0.75. Under the Alternative Annotator Test [6], its winning rate is 67% at $\epsilon= 0.3$ and 100% at $\epsilon= 0.5$, so once the cost of expert annotation is counted ($\epsilon\ge0.3$), it is a statistically justified substitute for an individual expert annotator. Appendix § 14 reports the full study.

Conclusion

We introduced CR-Bench, a defect-centric (PR, Review) benchmark of 584 instances (174 verified) labeled by root cause, impact, and severity, which scores review agents on known-defect detection and usefulness.

Implications for practice. (1) The best systems catch about one-third of known defects, so agents should assist human reviewers, not gate merges. (2) Self-reflection raised both KDDR and Usefulness Rate for every backbone. (3) Useful comments alone misleads: Gemini Code Assist’s comments are useful 78.0% of the time, yet it detects 8.1% of known defects, so test agents against known defects before deployment. (4) LLM judges differ in calibration, so compare agents under the same judge.

Future work will add more languages and agent designs and use CR-Bench for post-training.

Limitations

Scope of the data. CR-Bench inherits the scope of SWE-Bench: all instances come from open-source Python repositories, and the 174 CR-Bench-Verified tasks span only 11 of them, with Django alone contributing 63. Results may not transfer to other languages, to proprietary or monorepo codebases, or to review practices outside these projects. The reported agent evaluation uses only the verified split; broader conclusions require evaluating the full corpus and additional ecosystems.

One known defect per instance. Each instance supplies a single known defect, so KDDR measures whether an agent finds that defect, not whether it finds every defect in a PR. Other correct defect reports are counted as Valid Suggestions rather than Bug Hits and contribute to the Usefulness Rate instead.

LLM-generated references and labels. Claude Opus 4.5 filters instances for detectability, writes the reference review comments, and assigns the taxonomy labels. Errors in these steps propagate to KDDR and to the per-group results, and the detectability filter may exclude defects that a human reviewer could catch.

Evaluator dependence. Absolute scores depend on judge calibration: the three judges agree on the direction of differences but not on their magnitude. Our primary human-validated judge, Claude Sonnet 4.5, belongs to the same model family as the model that generated the references, which may favor phrasing close to those references. The human study covers 60 comments from a single system (GPT-5.2 Reflexion) and validates only that judge. We mitigate these effects by reporting three judges, prompt variants, and paired within-judge comparisons.

Contamination. The repositories, issues, and fixes behind SWE-Bench are public and may appear in the training data of the evaluated models, which could inflate KDDR.

Agents evaluated. We evaluate two reproducible harnesses, neither of which executes code or uses tools, and one commercial agent whose model and harness are undisclosed and may change over time. We do not report cost or latency, so the extra model calls that Reflexion requires are not weighed against its gains.

Ethical Considerations

Data. CR-Bench is derived from SWE-Bench and SWE-Bench Verified, which are built from public open-source repositories. It adds pull request identifiers, LLM-generated reference comments, and taxonomy labels, and it contains no private or proprietary code.

Human annotation. Two in-house software engineers and one technical director labeled 60 review comments as Bug Hit, Valid Suggestion, or Noise as part of their regular duties, and they agreed to the use of their labels. The task involved only public code and model-generated review text.

Responsible use. The best evaluated systems detect about one-third of known defects. Teams that rely on review agents as the only safeguard may therefore merge defective code. We recommend using such agents to assist human reviewers rather than to replace them, and testing them against known defects before deployment.

Prompts

Dataset Creation Prompts

Detectability Filter Prompt

System Prompt: You are a software engineering expert analyzing code changes. Your task is to determine if a change is of the kind that might have been prevented by reviewing the PR/commit that made that change more carefully or NOT. Name these PREVENTABLE and UNPREVENTABLE. PREVENTABLE: Code changes that fix incorrect behavior, errors, crashes, or unexpected results introduced by a previous code change (PR/commit). UNPREVENTABLE: Everything else including but not limited to: New features, refactoring, documentation, performance improvements, or enhancements.

Respond with JSON only: { "category": "PREVENTABLE" or "UNPREVENTABLE", "confidence": 0.0 to 1.0, "reasoning": "brief explanation" }

User Prompt: Problem Statement: {problem_statement}

Patch: {patch}

Is this a bug fix that might have been introduced by a previous code change (PR/commit)?

Bug Description Generation Prompt

System Prompt: You are a software engineering expert writing technical bug descriptions.

Given a problem statement and the code fix patch, write a concise technical description of the bug.

Format Requirements:

• Start with a clear statement of what's wrong

• Include specific code snippets showing the problematic pattern

• Explain why it's a bug (runtime impact, incorrect behavior)

• Keep it under 100 words

• Use technical language

• Focus on the root cause, not symptoms

Respond with ONLY the bug description (no JSON, no extra formatting).

User Prompt: Bug Description Task

Problem Statement

{problem_statement}

Code Fix

File: {file_path} Lines: {line_start}-{line_end}

{patch}

Write a concise technical description of this bug.

Bug Taxonomy Tagging Prompt

System Prompt: You are a software engineering expert analyzing bug reports to classify them according to a predefined bug taxonomy used in empirical software engineering research.

Analyze the bug from the issue description (problem statement) and the code fix patch, and provide a classification using the following fields.

---

1. category

Choose EXACTLY ONE of the following bug types:

1. Requirements, Features, and Functionality Bugs

Source: Ambiguous, incomplete, misunderstood, or changing requirements/specifications.

Types:

• Requirements bugs: Missing, ambiguous, or inconsistent requirements.

• Feature bugs: Wrong, missing, or extra features, including undesired enhancements.

Detection/Remedy: Formal specification languages, functional testing (transaction flow, domain, logic, state-based testing).

2. Structural Bugs

Source: Code control flow, sequence, and logical structure.

Types:

• Control & sequence bugs: Missing paths, unreachable code, improper loop nesting, misused GOTO/labels.

• Logic bugs: Incorrect logical expressions and Boolean errors.

• Processing bugs: Arithmetic, algorithm, type conversion, overflow errors.

• Initialization bugs: Uninitialized variables or improper first-loop values.

• Data-flow anomalies: Using uninitialized or stale data or failing to store or modify data correctly.

Detection/Remedy: Structural/path testing, unit testing, domain testing.

3. Data Bugs

Source: Specification, format, initial values, and usage of data objects. Types:

• Dynamic data bugs: Temporary data issues, shared memory residues.

• Static data bugs: Fixed content, parameters, and preprocessing errors.

• Role-based bugs: Errors in data used as control, parameter, or information.

• Attributes, structure, content bugs: Incorrect semantics, memory allocation, or representation.

Detection/Remedy: Validate all data declarations, centralize shared resources, defensive coding, compile-time checks.

Coding Bugs

Source: Programming errors, including wild-card or arbitrary mistakes.

Types:

• Syntax errors: Usually caught by compilers.

• Documentation bugs: Misleading comments or manuals.

Detection/Remedy: Proper code review, accurate documentation, automated syntax checking.

Interface, Integration, and System Bugs

Source: Interactions between components, systems, or hardware.

Types:

• External interface bugs: Wrong device or protocol handling.

• Internal interface bugs: Improper module communication.

• Hardware bugs: Misunderstood device behavior, I/O errors.

• OS bugs: Misuse or assumption of OS services.

• Software architecture bugs: Emergent behavior due to module interactions or stress.

• Integration bugs: Incompatibilities between integrated modules.

• System bugs: Complex interaction of multiple components, rare but expensive.

Detection/Remedy: Design for modularity, stress testing, integration testing, specialist interfaces.

Test and Test Design Bugs

Source: Errors in testing processes or test code.

Types:

• Test execution bugs: Incorrect test implementation.

• Test design bugs: Flawed test scenarios or criteria.

Detection/Remedy: Test debugging, automation of test execution and design, test quality assurance.

Memory-related Bugs

Source: Improper memory handling.

Types:

• Buffer overflow: Access beyond allocated memory.

• Stack smashing: Overwriting function return addresses.

• Memory leak: Lost pointers to allocated memory.

• Uninitialized read: Accessing memory before initialization.

• Double free: Freeing memory twice.

Detection/Remedy: Careful memory management, automated memory checking tools.

Concurrent Bugs

Source: Multi-threading or multi-process synchronization issues.

Types:

• Data race bugs: Conflicting access to shared memory.

• Atomicity bugs: Interrupted sequences of operations.

• Deadlock: Processes waiting indefinitely for shared resources.

Detection/Remedy: Proper synchronization, lock ordering, deadlock detection mechanisms.

---

Severity

Classify the runtime impact of the bug:

• Critical: System crash, severe malfunction, data loss, or serious security risk.

• High: Core functionality broken or system unusable in common scenarios.

• Medium: Partial malfunction, degraded behavior, or non-critical failures.

• Low: Minor issues, cosmetic problems, edge cases, or test-only failures.

---

Impact

Choose EXACTLY ONE aspect of the software that the bug directly compromises:

1. Functional Suitability

Description: Bugs that cause incorrect, missing, or inappropriate behavior relative to requirements.

• Examples: Wrong outputs, missing features, incorrect calculations.

2. Performance Efficiency

Description: Bugs that degrade system performance or resource utilization.

• Examples: Slow response, high CPU/memory usage, poor throughput.

3. Compatibility

Description: Bugs that interfere with interoperability or co-existence with other systems, devices, or software.

• Examples: Integration failures, protocol mismatches, API incompatibilities.

#### 4. Usability

* Description: Bugs that make the system confusing, hard to use, or increase user errors. * Examples: Misleading error messages, poor workflow, inaccessible UI elements.

#### 5. Reliability

* Description: Bugs that cause system instability, crashes, incorrect state, or inability to recover from failures. * Examples: Crashes, deadlocks, memory leaks, data corruption.

#### 6. Security

* Description: Bugs that expose the system to threats, unauthorized access, or data compromise. * Examples: Buffer overflows, improper access control, injection vulnerabilities.

#### 7. Maintainability

* Description: Bugs that make code harder to understand, modify, or extend. * Examples: Poorly structured code, missing documentation, complex interdependencies.

#### 8. Portability

* Description: Bugs that prevent software from being installed, executed, or transferred across different environments. * Examples: OS-specific assumptions, hardware incompatibilities, environment-specific failures.

---

#### 4. reasoning

Concisely explain the reasoning behind the category, impact, and severity of the bug.

---

### Output format

Respond ONLY with valid JSON in the following format:

{
  "category": "...",
  "severity": "...",
  "impact": "...",
  "reasoning": "..."
}

Do not include explanations, markdown, or any additional text outside the JSON.

User Prompt: # Bug Classification Task

## Problem Statement

{problem_statement}

## Code Fix File: {file_path} Lines: {line_start}-{line_end}

{correct_fix}

Analyze this bug and provide classification as JSON.

Agent Prompts

Single-Shot Agent Prompt

System Prompt:

You are an expert code reviewer analyzing pull requests for potential bugs and issues.

Your task is to review the provided code changes and identify:

1. Potential bugs or errors

2. Logic issues

3. Edge cases not handled

4. Security vulnerabilities

5. Performance problems

For each issue found, provide:

• file: The file path

• line: The line number (approximate)

• comment: Clear description of the issue

• severity: "high", "medium", or "low"

Output your review as a JSON array of comments. Example: [ { "file": "src/module.py", "line": 42, "comment": "Potential null pointer dereference. Variable 'user' may be None.", "severity": "high" } ]

If no issues found, return an empty array: []

Focus on actual bugs, not style or formatting issues. Do not be verbose, just provide the review comments in concise manner that describes the issue completely.

User Prompt: # Pull Request Review

Repository: {{ repo }} PR Number: {{ pr_number }} Title: {{ title }} Description: {{ description }}

## Diff

\{\{ diff \}\}

Please review the above changes and identify any potential bugs or issues.

Reflexion Agent Prompt

System Prompt: You are an expert code reviewer specializing in bug detection and security analysis.

Review the provided code changes and identify ONLY actual bugs and issues, NOT style or formatting concerns.

Focus Areas:

1. Bugs: Logic errors, incorrect implementations, type mismatches

2. Edge Cases: Unhandled null/undefined, empty arrays, boundary conditions

3. Security: SQL injection, XSS, authentication bypasses, data leaks

4. Concurrency: Race conditions, deadlocks, thread safety issues

5. Performance: Memory leaks, infinite loops, inefficient algorithms

Output Format:

Return a JSON array. Each issue must include:

• file: Exact file path from the diff

• line: Line number where issue occurs

• comment: Specific, actionable description of the bug (not generic warnings)

• severity: "high" (crashes/security), "medium" (data corruption/incorrect behavior), "low" (minor issues)

Quality Standards:

• Be SPECIFIC: Point to exact problematic code, not general concerns

• Be CERTAIN: Only report issues you're confident are real bugs

• Explain WHY: Include the consequence of the bug

• Suggest FIX: Briefly mention how to resolve it

Return [] if no bugs found.

Example:

[ { "file": "src/auth.py", "line": 23, "comment": "Missing null check before accessing 'user.email'. Will throw AttributeError if user is None. Add: if user is None: return error", "severity": "high" } ]

User Prompt:

Pull Request to Review

Repository: {{ repo }}

PR #{{ pr_number }}: {{ title }}

{{ description }}

Code Changes

{{ diff }}

Analyze the diff above and identify concrete bugs. Focus on what could actually break or cause incorrect behavior.

Reflection System Prompt: You are performing iterative self-improvement using the Reflexion framework.

Your goal: Produce a MORE COMPLETE and ACCURATE review by finding bugs you missed.

Reflexion Strategy:

PRIMARY FOCUS: Find Missed Bugs (False Negatives)

Your previous review likely MISSED important bugs. Search for:

• Null/undefined dereferences you overlooked

• Logic errors in conditionals or loops

• Off-by-one errors, incorrect operators

• Unhandled exceptions or error cases

• Security vulnerabilities (injection, auth bypass)

• Resource leaks, race conditions, memory issues

SECONDARY: Remove Only OBVIOUS False Positives

When in doubt, KEEP the issue. Err on the side of caution.

TERTIARY: Improve Existing Issues

• Make vague comments more specific

• Adjust severity if clearly wrong

• Add fix suggestions

Output Requirements:

• Return the FULL JSON array with ALL valid issues

• Prioritize ADDING missed bugs over removing uncertain ones

• Each comment must point to specific problematic code

• Be thorough - it's better to report a potential bug than miss a real one

Return [] only if you're absolutely certain no bugs exist.

Reflection User Prompt:

Reflection - Iteration {{ iteration }}: Find What You Missed

Repository: {{ repo }} | PR: {{ pr_number }}

Code to Review

{{ diff }}

---

Your Previous Review (Iteration {{ prev_iteration }})

You found {{ num_comments }} issue(s): ````markdown

{{ previous_comments }}

---

Reflexion Task: FIND MISSED BUGS

PRIMARY: What Bugs Did You Miss?

Carefully re-examine the diff. Look for bugs you may have overlooked in the previous review, including failures that emerge from interactions across changed lines, invalid assumptions, missing validation, incorrect state transitions, and changed call sites. Do not duplicate issues already covered by the previous review.

Output Your Complete Refined Review

Return a JSON array with:

1. All valid issues from previous review (unless clearly false positive)

2. NEW bugs you discovered in this reflection

3. Improved comments and severity levels

Format:

[
  {
    "file": "path/to/file.py",
    "line": 42,
    "comment": "Specific bug: what breaks, why, and how to fix",
    "severity": "high|medium|low"
  }
]

Return [] only if absolutely no bugs exist. ````

Evaluation Prompts

LLM-as-a-Judge Prompt – Balanced

System Prompt: You are a careful code review evaluation assistant. You classify review comments against a known bug. Respond with valid JSON only and avoid extra prose.

User Prompt: You are evaluating code review comments against a known bug.

Known Bug Description:

{bug_description}

Files Changed in Fix:

{patch_files}

Review Comment to Evaluate:

File: {file} Line: {line} Comment: {comment}

Task:

Classify this review comment into ONE of these categories:

1. BUG_HIT: The comment identifies or relates to the same issue described in the bug description

2. VALID_SUGGESTION: The comment makes a valid point (style, performance, maintainability, edge case, etc.) but is NOT about the known bug

3. NOISE: The comment is incorrect, irrelevant, or not actionable, only select this if really feel annoying or not helpful else classify as valid.

Respond with ONLY a JSON object:

{"classification": "BUG_HIT|VALID_SUGGESTION|NOISE", "reason": "brief explanation"}

LLM-as-a-Judge Prompt – Strict

System Prompt: You are a code review evaluation assistant. You classify review comments against a known bug. Return valid JSON only.

User Prompt: You are a code review classifier.

Known bug:

{bug_description}

Files changed in fix:

{patch_files}

Review comment:

File: {file} Line: {line} Comment: {comment}

Decision rubric:

BUG_HIT:

• Mentions the same defect

• Identifies the same edge case

• Describes the same incorrect behavior

• Points to the same logical flaw

VALID_SUGGESTION:

• Correct and actionable

• Improves maintainability, style, readability, safety, or performance

• Discusses a different issue than the known bug

NOISE:

• Incorrect analysis

• Non-actionable

• Irrelevant

• Pure speculation without evidence

Output ONLY:

{"classification":"BUG_HIT|VALID_SUGGESTION|NOISE","reason":"brief explanation"}

LLM-as-a-Judge Prompt – Lenient

System Prompt:

You are a generous but honest code review evaluation assistant.

Give credit to comments that are meaningfully related or clearly useful, but still distinguish true bug hits from general suggestions.

Return valid JSON only.

User Prompt:

You are evaluating whether a code review comment matches a known bug.

Known bug:

{bug_description}

Files changed:

{patch_files}

Review comment:

File: {file}

Line: {line}

Comment: {comment}

First determine:

1. What issue the review comment is describing

2. Whether that issue matches the known bug

Then classify as:

• BUG_HIT

• VALID_SUGGESTION

• NOISE

Return ONLY:

{"classification":"BUG_HIT|VALID_SUGGESTION|NOISE","reason":"brief explanation"}

Novelty Positioning Matrix

Table 3 positions CR-Bench against prior code review datasets on the four properties discussed in § 2: multiple diff hunks, full PR context, categorization, and a focus on a verifiable defect.

Dataset

This appendix reports dataset statistics and an example instance.

Statistics

Table 4 reports per-instance means for both splits, and Figure 5 shows how the CR-Bench-Verified tasks are distributed across source repositories.

Example Instance

Figure 6 shows one CR-Bench instance: the pull request, the known fix patch, and the reference review comment.

Taxonomy

Figure 7 lists the taxonomy labels assigned to each defect. Figure 8 and Figure 9 show how the labels co-occur in CR-Bench-Verified and CR-Bench.

Alternative Annotator Test

We use Alt-Test [6] to evaluate the suitability of an LLM as a replacement for a human code reviewer.

Annotation Setup

To verify the quality of CR-Bench’s LLM-as-a-judge, Claude Sonnet 4.5, we study its alignment with humans. We employ $M = 3$ expert developers to independently annotate a subsample of 60 reviews from Reflexion agent with GPT-5.2 (Table 5 shows one annotated instance).

instance_id	django__django-12708
Repository	django/django
Bug Origin PR	8432
PR Link	https://github.com/django/django/pull/8432
Known Bug Description	Deleting an index_together could crash when the same columns are also in unique_together because the schema editor searched for any constraint matching the columns and index=True, which also matches unique constraints. In alter_index_together() it called: self._delete_composed_index(model, fields, {'index': True}, self.sql_delete_index) This can return both the unique and non-unique index constraints, causing _delete_composed_index() to raise ValueError due to multiple matches. The fix constrains the lookup to non-unique indexes ({'index': True, 'unique': False}), ensuring only the intended index is deleted.
Known Fix Patch	diff --git a/django/db/backends/base/schema.py b/django/db/backends/base/schema.py --- a/django/db/backends/base/schema.py +++ b/django/db/backends/base/schema.py @@ -393,7 +393,12 @@ def alter_index_together(self, model, old_index_together, new_index_together): news = {tuple(fields) for fields in new_index_together} # Deleted indexes for fields in olds.difference(news): - self._delete_composed_index(model, fields, {'index': True}, self.sql_delete_index) + self._delete_composed_index( + model, + fields, + {'index': True, 'unique': False}, + self.sql_delete_index, + )
Files Changed in Fix	django/db/backends/base/schema.py
Review File	django/contrib/auth/backends.py
Review Line	57
Review Comment	Potential compatibility risk if this branch is intended to support Python <2.7: set literals and dict comprehensions used throughout this PR (e.g., "%s.%s" % (ct, name) for ct, name in perms) require Python 2.7+/3.x. If any supported runtime is older, this will raise SyntaxError at import time. Fix by keeping set(... for ...) / dict(...).
Review Type Label	Noise

Formalism

We formalize code review evaluation as a classification task over a set of annotators, comprising both LLMs and human reviewers. Let $x \in\mathcal{X}$ denote a code review instance and $y \in\mathcal{Y}$ denote a review label. Each annotator $a \sim\mathcal{A}$ induces a conditional distribution $P_a(y \mid x)$.

Given the inherently subjective nature of code review, label assignment is influenced by annotator-specific biases and contextual factors. To capture this variability, we define the aggregated label distribution as:

We quantify the degree of subjectivity for an instance $x$ as the entropy of the aggregated distribution:

Individual Annotator Evaluation. Given an annotator $a$ producing prediction $y_a(x)$, this approach compares this prediction against the judgments $\{R_i\}_{i=1}^{M}$ of $M$ independent annotators using parameterized majority voting. The agreement score is defined as ($k$ is a parameter):

This formulation implicitly treats consensus labels as ground truth, thereby collapsing the underlying distribution of valid annotations into a single point estimate and introducing majority bias in subjective settings. However, annotator disagreement is intrinsic to such tasks and reflects a latent distribution rather than noise. To account for this, we adopt the Alternative Annotator Test (Alt-Test) [6], which evaluates annotator quality relative to the inter-annotator agreement baseline instead of a fixed consensus label.

The Alt-Test excludes each human annotator and compares LLM alignment with the remaining. It needs two parameters: the cost-benefit penalty ($\epsilon$) and the False Discovery Rate threshold ($q$). $\epsilon$ quantifies the cost-benefit trade-off of using an LLM. A higher $\epsilon$ means a larger associated penalty to using human annotators, maybe because they are more expensive, slow, etc. We evaluate the system across a range of penalties ($\epsilon\in[0.0, 1.0]$) and report winning rate $\omega$, the fraction of annotators an LLM can be better than. We use $q = 0.05$, meaning we strictly limit the probability of falsely concluding the LLM as a valid replacement to 5%.

Results

As per IAA score in Table 6, model can’t make a strict agreement with all the annotators ($k = 3$), but is able to maintain a consistency at lesser thresholds. With the $\omega$ analysis using Alt-Test, we further determine that LLM cannot statistically justify replacing humans at low $\epsilon$ ($\omega= 0\%$). However, as $\epsilon$ increases to simulate the operational benefits of automation, the model’s $\omega$ climbs to 67% at $\epsilon= 0.3$ and reaches 100% at $\epsilon= 0.5$.

Metric	Setting	Value(%)
IAA score	$k = 1$	71.66%
IAA score	$k = 2$	71.66%
IAA score	$k = 3$	28.33%
Winning rate	$\epsilon= 0.1$	0%
Winning rate	$\epsilon= 0.3$	67%
Winning rate	$\epsilon= 0.5$	100%

As per experienced software engineer annotators’ feedback, this is a complex and time-taking labeling. So, consistent with the Alt-Test paper, this is $\epsilon\ge0.2$. Further, $\omega\ge50\%$ translates to LLMs being able to replace humans. We sweep $\omega$ as a function of $\epsilon$ in Figure 10a and show that our judge achieves 67% $\omega$ at $\epsilon= 0.3$ and 100% at $\epsilon= 0.5$. $\omega= 0$ for $\epsilon\le0.2$ shows that code-review of this standard is a difficult task for even skilled humans to annotate.

Further, as discussed earlier, we only consider $N = 60$ out of approx. 1600 reviews generated from 174 PRs for this study. In order to understand its sufficiency, we plot a graph of $\omega$ v/s $N$. For each $n \in N$, we randomly sample $n$ instances and calculate $\omega$ for a given $\epsilon$. We do this 20 times and plot the average in Figure 10b. Saturation of $\omega$ after $N \ge50 \ \forall\ \epsilon\in\{0.1, 0.3, 0.5\}$ shows that $N = 60$ is sufficient for statistical significance.

Agreement Analyses

Further analyses also show that classifying review comments as Bug Hit, Valid Suggestion, or Noise is a difficult task even for expert annotators. Pairwise agreement between the expert annotators themselves spans 0.48 to 0.75, and the judge aligns most closely with the most central annotator (Figure 11), which suggests it approximates consensus behavior rather than any one reviewer. The confusion matrix against the human majority (Figure 12) shows strong agreement on Bug Hits, with most residual error being Bug Hits softened into Valid Suggestions. Figure 13, Figure 14 show inter and intra-annotator entropy of AI falls in similar range with majority humans.

Pairwise agreement among the human annotators ranges from 0.48 to 0.75. Human-2 has the highest agreement with the other annotators. Claude Sonnet 4.5 has its highest human agreement with Human-2 (0.72). This pattern is consistent with consensus-oriented labeling.

Results

Evaluator Robustness and Uncertainty

All results below use clean judge runs with zero fallback classifications. To separate review-system performance from evaluator choice, each judge independently classifies the same fixed review comments. Table 8 reports every review-system–judge combination, while Table 9 directly compares Reflexion with single-shot for matched backbones. Confidence intervals resample benchmark instances. Paired tests align strategies by instance_id.

Review system	Evaluation judge	KDDR (%) [95% CI]	Usefulness Rate (%) [95% CI]
GPT-5.2 single-shot	Claude Sonnet 4.5	26.4 [20.1,33.9]	81.7 [77.9,85.5]
	GPT-5.4	31.0 [24.7,37.9]	99.1 [98.6,99.5]
	DeepSeek V4 Flash	29.3 [23.0,37.4]	77.8 [75.5,80.4]
GPT-5.2 Reflexion	Claude Sonnet 4.5	32.2 [25.8,39.1]	89.5 [86.3,92.7]
	GPT-5.4	34.5 [27.6,42.0]	99.4 [98.9,99.8]
	DeepSeek V4 Flash	32.2 [25.9,39.1]	80.4 [78.3,82.7]
GPT-5-mini single-shot	Claude Sonnet 4.5	21.3 [14.9,28.2]	74.8 [70.6,78.9]
	GPT-5.4	25.9 [19.5,32.8]	95.5 [93.7,97.0]
	DeepSeek V4 Flash	23.0 [16.7,29.3]	36.6 [33.0,40.5]
GPT-5-mini Reflexion	Claude Sonnet 4.5	28.2 [21.8,35.1]	78.5 [74.7,82.4]
	GPT-5.4	37.4 [29.9,44.8]	97.5 [96.3,98.4]
	DeepSeek V4 Flash	35.6 [28.2,43.1]	41.7 [38.2,45.9]
Gemma 4 31B single-shot	Claude Sonnet 4.5	12.6 [8.0,17.8]	67.9 [61.9,73.4]
	GPT-5.4	14.9 [9.8,20.7]	90.5 [87.2,93.6]
	DeepSeek V4 Flash	14.4 [9.8,19.5]	35.2 [29.9,40.1]
Gemma 4 31B Reflexion	Claude Sonnet 4.5	19.5 [13.8,25.3]	75.2 [70.8,79.4]
	GPT-5.4	23.6 [17.8,29.9]	92.0 [89.9,93.9]
	DeepSeek V4 Flash	21.8 [16.1,28.2]	38.7 [34.3,43.6]
GPT-OSS 20B single-shot	Claude Sonnet 4.5	5.8 [2.9,9.2]	52.2 [46.0,59.0]
	GPT-5.4	6.3 [3.4,9.8]	81.7 [76.5,86.7]
	DeepSeek V4 Flash	9.2 [5.2,13.2]	25.9 [20.9,31.4]
GPT-OSS 20B Reflexion	Claude Sonnet 4.5	12.1 [7.5,17.2]	63.0 [57.0,69.5]
	GPT-5.4	12.6 [8.0,18.4]	86.8 [83.8,90.1]
	DeepSeek V4 Flash	15.5 [10.3,21.3]	33.7 [29.2,39.0]
Gemini Code Assist	Claude Sonnet 4.5	7.5 [3.4,11.5]	77.7 [69.4,86.5]
	GPT-5.4	8.1 [4.0,12.6]	98.1 [96.1,99.7]
	DeepSeek V4 Flash	8.6 [4.6,13.2]	58.1 [50.1,66.8]

Backbone	Evaluation judge	$\Delta$KDDR [CI], $p$	$\Delta$Usefulness Rate [CI], $p$
GPT-5.2	Claude Sonnet 4.5	+5.7 [-0.0,11.5], 0.083	+7.8 [5.2,10.7], 0.001
	GPT-5.4	+3.4 [-2.3,9.2], 0.336	+0.3 [-0.2,0.9], 0.284
	DeepSeek V4 Flash	+2.9 [-3.4,9.2], 0.504	+2.6 [-0.2,5.3], 0.066
GPT-5-mini	Claude Sonnet 4.5	+6.9 [1.1,13.2], 0.036	+3.7 [0.2,7.4], 0.101
	GPT-5.4	+11.5 [4.6,18.4], 0.002	+2.1 [0.4,3.9], 0.012
	DeepSeek V4 Flash	+12.6 [5.7,19.5], 0.004	+5.1 [1.2,9.0], 0.032
Gemma 4 31B	Claude Sonnet 4.5	+6.9 [1.7,12.1], 0.027	+7.4 [2.5,12.1], 0.007
	GPT-5.4	+8.6 [2.3,14.9], 0.006	+1.5 [-1.8,4.9], 0.318
	DeepSeek V4 Flash	+7.5 [1.7,13.8], 0.016	+3.5 [-1.8,9.2], 0.251
GPT-OSS 20B	Claude Sonnet 4.5	+6.3 [1.7,11.5], 0.023	+10.8 [3.9,17.2], 0.001
	GPT-5.4	+6.3 [1.7,11.5], 0.030	+5.2 [0.3,10.2], 0.032
	DeepSeek V4 Flash	+6.3 [0.6,12.6], 0.060	+7.8 [2.3,13.5], 0.004

Across judges, Reflexion has higher KDDR than single-shot for every backbone. The magnitude of Usefulness Rate varies more because judges differ in how often they classify comments as Noise.

KDDR by Taxonomy Group

Table 11 reports KDDR for every taxonomy group, and Figure 16 plots the severity groups with confidence intervals.