WorksComputer Science
AI Code Review Requires Revision-Aware Evidence of Concern Resolution
Abstract
AI-assisted code review increasingly judges whether reported concerns have been resolved. Yet closing a thread or modifying a commented line does not establish repair, and later revisions can invalidate earlier fixes. This position paper argues that resolution requires inspectable, revision-specific evidence that is revalidated when its dependencies change. We model review concerns as persistent verification obligations, derive a sufficient condition under which dependency-gated revalidation matches exhaustive reassessment, and introduce REVAL, a lightweight procedure adapting principles from regression-test selection and build systems. We evaluate REVAL against oracle ground truth in synthetic revision histories covering 441 concerns, 1,284 assessments, and 12 defect families. A sticky outdated-line proxy falsely marked 72.9% of unresolved states as resolved, while stopping checks after initial success missed every recurrence. With three oracle-sampled witnesses, REVAL achieved a false-resolution rate of 0.11 (95% CI: 0.06-0.16) and detected 80.4% of recurrences. With witnesses derived solely from review comments and assessed using independent held-out checks, the corresponding results were 0.13 and 88.3%. Across 6,420 paired comparisons, REVAL matched exhaustive reassessment while reducing post-comment test executions by 34%. Maintenance overhead increased runtime for microsecond-scale tests; the estimated break-even test cost was 0.42 ms. These findings support revision-aware evidence maintenance in a controlled setting, while distinguishing revalidation correctness from witness adequacy. A prespecified evaluation protocol outlines how to assess generalisability in real review histories.
Introduction
Modern code review is an established practice for improving software quality in industrial and open-source development [1, 17, 24]. Its benefits extend beyond defect detection to include improvements in maintainability and evolvability [2, 21]. Research on review automation has progressed from pre-trained models for generating comments and refining code [15, 26] to deployed systems that propose repairs, check development practices, and participate in pull-request workflows [5, 9, 29]. More recently, automated support has extended to determining whether previously reported concerns have been addressed [11]. This development introduces a distinct verification problem: what evidence justifies declaring a review concern resolved, and under what conditions does that judgement remain valid?
The distinction between administrative resolution and substantive repair is central to this problem. Review platforms record workflow events, including thread closure and the displacement of comments from their original code locations. These events describe the progression of a review, but do not, by themselves, establish that the underlying concern has been corrected. Nevertheless, resolution status is also used to evaluate automated review systems. Industrial reports characterise assistant impact through the proportion of automated comments that were resolved [5], while empirical studies operationalise resolution using platform thread state [7]. Interpreting such measures as evidence of successful repair requires an assumption that administrative state corresponds to the status of the underlying concern. The validity of this assumption is therefore consequential for both automated decision-making and empirical evaluation.
A further difficulty arises from the temporal nature of software development. A repair that is correct at one revision may be invalidated by subsequent changes. These changes need not affect the originally commented lines: modifications to a helper function, configuration setting, or other dependency may reintroduce the concern while leaving the earlier resolution decision unchanged. Resolution should therefore be understood as a claim about a particular code revision. Maintaining that claim requires determining whether the evidence supporting it remains applicable as the code evolves.
Consider a reviewer who observes at revision that parse_quantity('-3') returns , although all non-positive quantities should raise ValueError. At , reformatting makes the comment outdated without changing the behaviour. At , a guard rejects negative values but continues to accept zero, producing a partial repair. At , validation is moved to a helper with the correct boundary condition, fully addressing the concern. At , a change to that helper causes zero to be accepted again. An outdated-line proxy registers a workflow transition at , whereas a check restricted to the reviewer’s example passes from onwards. Neither establishes that the complete concern is absent only at (Figure 1). The example distinguishes three sources of error: substituting workflow state for repair evidence, using evidence that captures only part of a concern, and retaining a resolution judgement after its supporting evidence has become invalid.

Figure 1. The running example, built from the quantity family’s variants. Each cell is the decision computed by running the method online. The witness set is the reviewer’s example plus the cases ’0’ and ’7’. The outdated-line proxy resolves at the cosmetic edit. Evidence from the reviewer’s example alone accepts the partial fix. Only a maintained, stronger witness set tracks the true sequence.
This paper advances the position that a resolution decision should be tied to a specific code revision and supported by inspectable evidence; when subsequent changes invalidate that evidence, the concern must be reassessed. Under this position, review concerns are persistent verification obligations whose status is maintained throughout a revision history. Two requirements follow. First, the evidence must adequately represent the concern being assessed. Second, its reuse at later revisions must be justified by the stability of the dependencies on which it relies. These requirements distinguish the strength of the evidence from the correctness of the procedure used to maintain it.
The position comprises three claims:
(P1) Measurement validity. Administrative and syntactic proxies, including outdated-line status and changed-line heuristics, can diverge from the substantive repair of a review concern.
(P2) Temporal validity. A resolution judgement that is correct at one revision can become incorrect at a later revision. Methods that cease checking after the first successful assessment cannot detect subsequent recurrence.
(P3) Evidence maintenance. Under a sufficient dependency model, revalidating evidence when its dependencies change can preserve the decisions obtained by exhaustive reassessment while reducing executions. Agreement with exhaustive reassessment does not, however, establish that the evidence fully captures the concern.
The computational principles underlying evidence maintenance are established. Safe regression-test selection identifies tests that require re-execution after a change [12, 23]; build systems reuse results while their dependency fingerprints remain valid [18]; and truth-maintenance systems revise beliefs when their justifications no longer hold [8]. The contribution of this paper lies in applying and evaluating these principles at the level of persistent review concerns. A concern expresses an obligation that may be only partially represented by its executable witnesses. Evaluating its resolution consequently requires separating witness adequacy, dependency validity, and the policy governing reassessment.
We instantiate this perspective as REVAL, a lightweight procedure for maintaining revision-specific resolution evidence. The procedure records the dependencies of an assessment, reuses its result while those dependencies remain unchanged, and triggers reassessment when relevant changes occur. Its evaluation follows individual concerns across revision histories, explicitly accounting for partial repairs and recurrence. The study addresses three research questions:
RQ1 In controlled revision histories, how frequently, and under which conditions, do an outdated-line proxy and a surface-pattern heuristic disagree with oracle-verified concern status?
RQ2 To what extent does maintaining revision-specific evidence reduce false resolution and improve recurrence detection relative to stopping after first success, and what is its cost relative to reassessment at every revision?
RQ3 How do witness strength and dependency granularity affect the remaining errors, including when witnesses are derived solely from review comments and evaluated against independently written checks?
The paper makes five contributions:
(C1) A formal model of review concerns as revision-indexed verification obligations, including a sufficient condition for dependency-gated revalidation to match exhaustive reassessment (Proposition 1), an explicit account of the guarantee’s limits (Remark 1), and a decomposition of false resolution into three mechanisms (Corollary 1).
(C2) A taxonomy characterising divergence between administrative closure and substantive repair (Table 2).
| Divergence | What happens | Mechanism |
| Cosmetic edit | Commented lines change, so the comment becomes outdated, but behaviour is unchanged | (a) no evidence |
| Partial fix | Reviewer’s example passes; other in-scope cases still fail | (b) witness insufficiency |
| Relocated fix | Repair moves the check to a helper; line heuristics mis-score a correct fix | (a) for line proxies |
| Inline regression | A later edit of the same function drops the repair | (c) stale (policy) |
| Remote regression | A helper in another file is weakened; the commented function is untouched | (c) stale; needs footprint |
| Configuration regression | A module-level constant changes and re-opens the defect | (c) incomplete footprint |
| Contract change | The subject is renamed and the evidence cannot run; it may later be restored | should yield UNDETERMINED |
| Invalid concern | The original comment was wrong | human disposition |
| Obsolete concern | Requirements changed and the concern no longer applies | human disposition |
Table 2. Taxonomy of divergence between administrative closure and actual repair, with the mechanism of Corollary 1. Rows above the rule are instantiated in the controlled study.
(C3) REVAL, an evidence-maintenance procedure with an explicit dependency model, conservative fallbacks, and regression tests for edge cases (§ 4).
(C4) A controlled, executable evaluation using oracle-sampled and blind comment-derived witnesses, with the latter assessed against independently written held-out checks. The evaluation includes shared-draw bootstrap uncertainty, separate witness-sampling uncertainty, and runtime measurements that distinguish execution costs from maintenance overhead (§ 5 and § 6).
(C5) A pre-specified protocol for evaluating naturally occurring pull-request histories (§ 7.3), supported by a replication package containing verification checks.
The empirical scope is deliberately bounded. All quantitative findings are obtained from synthetic revision histories constructed from small, deterministic Python programs. The baselines isolate mechanisms through which resolution judgements can fail; they do not constitute measurements of GitHub, Copilot, or developer behaviour. The study therefore evaluates these mechanisms and their relative effects within the constructed setting, without estimating their prevalence in operational review workflows. The protocol for naturally occurring histories specifies the subsequent evaluation needed to assess external validity. Accordingly, the paper retains a position-paper framing: it establishes the conceptual and formal basis for revision-aware concern resolution and provides controlled evidence supporting further empirical investigation.
Background and Related Work
Modern Code Review and Review Outcomes
Empirical research has established that code review serves several complementary purposes. Although defect detection is a prominent motivation, field studies also identify maintainability improvements, knowledge transfer, and shared awareness as important outcomes [1, 24]. Repository-mining studies have characterised the problems corrected during review [2], examined associations between review practices and post-release defects [17], and identified review-related changes that developers consider suitable for automation [21]. Related work has modelled comment usefulness using developer judgements and observable comment characteristics [3, 28].
These studies establish the value and diversity of review outcomes, but evaluating whether a comment is useful differs from determining whether its underlying concern has been repaired. A useful comment may lead to an incomplete repair, and an initially successful repair may subsequently be reversed. A particularly relevant temporal perspective is provided by Oliveira et al. [20], who trace understandability-improvement comments through accepted changes and later reversions. Their analysis examines the persistence of review-induced changes. The present paper addresses the related but distinct question of whether a behavioural concern is absent at each subsequent revision, including cases in which the implementation changes without an explicit reversion of the original repair.
Automation of Code Review and Comment Resolution
Early approaches to review automation used pre-trained models to generate comments and refine code in response to review feedback [15, 26]. Subsequent qualitative analysis identified recurring limitations underlying their apparent strengths [27]. Industrial deployments have extended this work to concrete review activities. At Google, authors addressed 7.5% of all reviewer comments by applying an ML-suggested edit [9]. Other systems support LLM-based best-practice checking [29] and automated participation in pull-request review; a deployment at Beko reported that 73.8% of automated comments were resolved [5].
Agent-generated development activity has also enabled datasets such as AIDev [14]. Using AIDev-derived repositories, Cynthia et al. [7] examine developer responses to agent-generated review comments and operationalise resolution through thread state. Automatic comment-resolution functionality extends automation to the resolution decision itself [11].
These developments make the interpretation of resolution increasingly important. Applying a suggested edit, closing a thread, and establishing that a concern has been repaired are distinct outcomes. Reported adoption or resolution rates describe the corresponding workflow events, but their interpretation as evidence of substantive repair requires additional validation. Our study examines this distinction through controlled mechanism baselines. It makes no assumption that the cited systems implement those baselines or share their failure rates.
Evaluation of Automated Review
Evaluation methods for automated review address several different constructs. A mapping study documents heterogeneity in datasets and metrics [30], while a recent roadmap advocates evaluation that accounts for context and practical value [34]. EvaCRC models attributes of review-comment quality [33]; CodeReviewQA evaluates comprehension of review intent [16]; and HalluJudge identifies generated comments that are insufficiently grounded in the code [25]. These approaches assess properties of a comment or a model’s interpretation of it.
Other evaluations incorporate executable evidence or iterative development. The c-CRAB benchmark derives tests from human reviews to assess agent-generated reviews [35], while SWE-Review evaluates agentic reviewers through the resolve rate of generate–review–revise loops [31]. Repository-level issue-resolution benchmarks provide a broader precedent for evaluating software changes against executable criteria [13]. LLM-as-judge research offers a further methodological basis for assessments that rely on model-generated judgements [36].
The present paper complements these approaches by changing the unit and temporal scope of evaluation. Its unit is an individual review concern, and its outcome is the status of that concern at each revision. This formulation distinguishes initial repair from partial repair and subsequent recurrence. It also separates two questions that an aggregate success measure may combine: whether the available evidence is strong enough to represent the concern, and whether that evidence remains valid after later changes. Table 1 summarises these distinctions for selected related studies.
| Work | Unit of evaluation | Per concern | Across revisions | Partial repairs | Recurrence | Executed evidence |
| EvaCRC [33] | Comment quality attributes | – | – | – | – | – |
| CodeReviewQA [16] | Comprehension of review intent | ✓ | – | – | – | – |
| HalluJudge [25] | Grounding of generated comments | ✓ | – | – | – | – |
| c-CRAB [35] | Agent reviews assessed against human-derived tests | ✓ | – | – | – | ✓ |
| SWE-Review [31] | Resolution in review–revise loops | – | ◦ | – | – | ✓ |
| Oliveira et al. [20] | Persistence of understandability changes | ✓ | ✓ | – | ◦ | – |
| Cynthia et al. [7] | Developer responses and thread state | ✓ | – | – | – | – |
| This paper | Concern status at each revision | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1. Positioning of selected related studies by evaluation focus. ✓ denotes explicit treatment, ◦ denotes partial treatment, and – denotes a dimension outside the stated evaluation focus.
Artefact Obsolescence and Evidence Validity
Research on code–comment inconsistency examines how documentation that was initially accurate becomes inconsistent with evolving implementations [32]. Just-in-time inconsistency detection addresses this problem at the point of change [22]. This literature provides a useful conceptual parallel for review resolution: an assessment may be justified when made but become unsupported as the code changes.
The object maintained in this paper is the evidential basis of a resolution judgement. A review thread may retain its resolved status even after a change invalidates the checks that justified closure. Conversely, a substantial code change need not invalidate a judgement if the relevant evidence remains applicable. The resulting problem is therefore one of maintaining the relationship among a concern, its supporting evidence, and the revision to which that evidence applies.
Dependency-Based Revalidation and Checkable Evidence
Several established approaches provide the computational foundations for maintaining evidence under change. Safe regression-test selection determines which tests require re-execution following a program modification [23]. Dynamic dependency tracking supports selective execution using dependencies observed during earlier runs, including file-level tracking [12]. Build systems similarly reuse derived results when their inputs and dependency fingerprints remain unchanged [18]. Truth-maintenance systems provide a related account of how derived beliefs should be revised when their justifications cease to hold [8].
REVAL adapts these principles to persistent review concerns. The concern remains the object of assessment, executable witnesses provide evidence about its status, and dependency information determines when that evidence requires revalidation. The formal guarantee concerns agreement with exhaustive execution of the same witnesses under the stated dependency assumptions. It does not establish that those witnesses constitute a complete specification of the concern. This distinction is essential because a sound revalidation policy can consistently preserve the result of an inadequate check.
Two additional research traditions inform the evaluation. Proof-carrying code motivates associating a decision with evidence that can be independently checked [19]. Selective prediction motivates comparing error rates at matched decision coverage when methods may abstain from issuing a judgement [10]. Together, these foundations support the paper’s emphasis on inspectable evidence, explicit conditions for reuse, and evaluation that distinguishes decision reliability from the frequency with which decisions are made.
Concerns as Revision-Indexed Verification Obligations
Programs, revisions and concerns
Let be the revision history of a change under review. Here is the commented revision and each is a complete program snapshot. A revision consists of modules. is its set of units: the top-level functions and module-level bindings of every module, plus one namespace unit per module , whose text is the set of names bound at top level in . We write for the text of unit in , with if . Namespace units make the addition or removal of a binding observable even when no existing unit’s text changes.
Definition 1 (Concern). A concern is , with intent and anchor (the commented lines of ). The intent is made operational as an oracle of behavioural checks, . Here means the check is inapplicable because the subject of the concern cannot be located in , and is the reviewer’s own example.
Definition 2 (Concern status).
Humans assign two further dispositions, which are never counted as repairs. A concern is if was wrong at , and if a later requirement change makes it inapplicable.
A concern recurs at if and . Status is a function of the revision, not a property of the thread.
Resolution claims and their evidence
Definition 3 (Evidence-carrying claim). A claim is . It consists of an asserted status , an assessed revision , a witness set of executed checks with outcomes , a footprint of units the execution depended on, and a fingerprint for . The witness set need not be drawn from . In practice it is written from the review comment, and is unknown.
Definition 4 (Evidence validity). Claim is valid at if and ; otherwise it is stale at .
Proposition 1 (Soundness of dependency-gated revalidation). Assume that (i) each check in is deterministic, and (ii) is complete for : every revision that agrees with on the text of all units in yields the same outcomes for $W. If is valid at , then re-executing on reproduces , and hence .
Proof. Validity gives agreement on . Completeness then gives identical outcomes, and is a function of the outcomes.
Remark 1 (Three different levels of guarantee). (1) Mathematical. Proposition 1 states equivalence to re-running the same witnesses. It says nothing about whether passing implies , which depends on how well covers (Corollary 1b). (2) Implemented. Completeness is an assumption. Our implementation guarantees it only for the supported dependency model of Section 4: top-level functions, module-level bindings read by executed code, and module namespaces of executed modules, all for deterministic code. Dependencies outside this model, such as I/O, environment, time, randomness or reflection, require a conservative fallback. (3) Empirical. The agreement between REVAL and exhaustive re-checking reported in Section 6 is an observation on this benchmark. It is consistent with completeness there but does not prove it.
Corollary 1 (Decomposition of false resolution). If a method asserts at while , at least one of the following holds: (a) no evidence: the assertion is not backed by executing any check; (b) witness insufficiency: passes at but some check of fails; (c) stale evidence: the claim was produced at and was not revalidated, either because the policy stopped checking or because was incomplete.
Proof. Suppose the assertion is backed by evidence. If that evidence was produced at , or is valid at with complete (Proposition 1), then passes at , which is case (b). Otherwise it is case (c).
Metrics
Let be the assessed (concern, revision) pairs with and determinate truth. The primary metric is
We also report fixed-label precision , macro- over , and recurrence recall (recurring states predicted ). We report the contract-change flag rate ( states not asserted ), decision coverage (determinate states without abstention), and post-comment test-case executions per concern, which exclude the initial assessment at . Wall-clock runtime is reported separately (Section 6.5).
Reval: Revision-Aware Evidence Maintenance
REVAL keeps one concern record per atomic concern. A record holds the intent and anchor, the witness set , the latest claim and the history of past claims. A reviewer can therefore see which revision supports the current status and why it changed. Algorithm 1 gives the online procedure.
Supported dependency model and fallbacks. The footprint is dynamic, collected while the witnesses run. Repairs that move logic into helpers therefore extend automatically. Three edge cases are handled explicitly. First, while a concern is there is no valid evidence, so every new revision is reassessed (line 3). This covers restoration of a missing target by adding a definition, which leaves all existing units unchanged. Second, namespace units detect new bindings, for example a module-level name that shadows a builtin read by the target, and removed bindings. Third, a dependency outside the model (I/O, environment, time, randomness, reflection, dynamic import) cannot be traced. In that case the record must fall back to file-level or every-revision reassessment. In this benchmark every witness is deterministic and in-model, so no fallback was triggered. The implementation ships with regression tests for target removal, restoration by addition, builtin shadowing, helper and constant changes, and unrelated edits.
Assessor-agnostic. Here consists of executable checks. The same record and staleness test can also gate re-invocation of an LLM assessor, with the footprint taken from the code the assessor was shown. That variant belongs to the protocol in § 7.3.
Algorithm 1: REVAL: online assessment of concern over revisions
Input: history (revealed one at a time); witness set ; granularity
Output: status and supporting claim for each
;
for to do
if or then
; // no valid evidence: reassess
else
keep ; // evidence valid: decision carries over
emit ;
Function Assess:
run each check of on a fresh instance of under a call tracer; collect outcomes , executed functions (and their modules ), and names read by executed code;
if any check is inapplicable; else if all pass; else ;
if then
;
else if then
;
else the modules , compared by whole-file text;
return ;
Controlled Study Design
Benchmark construction
We constructed 12 concern families (Table 8, Section A). Each is a two-file Python program, with core.py holding the commented function and constants and util.py holding helpers. Each comes with a review comment and a hidden oracle of 6–7 behavioural checks, including positive cases that a correct fix must keep passing. The families cover input validation (3), security (path traversal, SQL injection, output escaping), exception handling (2), resource management, boundary conditions, null handling and shared mutable state. Each family provides a defective commented version, a partial fix, a complete inline fix, a complete fix delegating to a helper and a weakened helper. Three families also have a configuration constant whose regression re-opens the defect. By design, partial fixes and weakened helpers make the reviewer’s example pass while other in-scope checks fail.
| Family | Category | Review comment | Oracle |
| quantity{sup}†` | input validation | parse_quantity(’-3’) returns ; non-positive must raise | 7 |
| mean | exception handling | mean([]) raises ZeroDivisionError; return None | 7 |
| paginate | boundary condition | paginate(items,1,3) skips the first page; page invalid | 7 |
| safe_join | security | ’../etc/passwd’ escapes the base directory | 7 |
| user_query | security | name interpolated into SQL; use a bound parameter | 6 |
| greeting | security | ’<script>’ emitted raw; escape user text | 6 |
| first_line | resource management | handle leaks when readline raises OSError | 6 |
| with_retry{sup}†` | exception handling | all errors swallowed, returns None; surface the error | 6 |
| display_name | null handling | crashes when profile is None; fall back to name | 6 |
| set_volume{sup}†` | input validation | set_volume(150) returns 150; clamp to | 7 |
| add_tag | state management | mutable default list leaks tags across calls | 6 |
| normalize_email | input validation | ’bob@’ accepted; need local part and dotted domain | 7 |
Table 8. Table 8: The twelve concern families. “Oracle” gives the number of hidden checks; † marks families with a configuration constant.
Histories come from 13 scenario scripts (Table 3), each instantiated with three seeds that insert zero to two unrelated edits at random positions. This yields 441 concerns and 1284 post-comment assessments, with 2.91 revisions per concern on average. Ground truth is obtained by executing the full oracle at every revision. It contains 631 PRESENT, 561 FIXED and 92 UNDETERMINED states, including 155 recurring states. The PRESENT states comprise cosmetic edits (194), partial fixes (145), untouched (137), remote helper regressions (93), inline regressions (49) and configuration regressions (13).
| ID | Developer actions after the comment | Probes |
| S01 | fix | baseline repair |
| S02 | cosmetic → fix | premature closure |
| S03 | partial | witness insufficiency |
| S04 | partial → fix | delayed true repair |
| S05 | fix → inline regression | local recurrence |
| S06 | fixref → helper regression | remote recurrence |
| S07 | fix → constant regression (3 families) | configuration recurrence |
| S08 | cosmetic → cosmetic | never repaired |
| S09 | fix → behaviour-preserving refactor | control (no status change) |
| S10 | fixref → helper regression → restore helper | recurrence and re-repair |
| S11 | cosmetic → partial | compound divergence |
| S12 | fix → rename target | contract change (UNDETERMINED) |
| S13 | fix → rename target → restore by addition | re-anchoring after UNDETERMINED |
Table 3. Scenario scripts (unrelated edits are inserted at random). “fixref” is a complete fix delegating to a helper; “restore” in S13 re-adds the original function name by a new definition.
Methods compared
All methods run online and see only after deciding on . The first two are controlled mechanism baselines. Each isolates one failure mechanism and is not a model of any product’s behaviour.
Outdated-line proxy (OUT). It predicts FIXED from the first revision at which the commented lines no longer appear verbatim, and the decision is sticky by construction. It represents using “the comment became outdated” (or a thread resolved on that basis) as a repair label. It is not the decision procedure of GitHub’s isOutdated handling or of Copilot’s auto-resolution, whose assessment method is not public [11].
Surface-pattern heuristic (PAT). A stateless AST heuristic: FIXED if the commented function contains more defensive constructs (conditionals, raises, try/with, sanitising or clamping calls, helper delegation) than at . It represents judgements that reward fixes that merely look defensive. It is not an LLM judge.
Evidence policies. ONCE assesses at , then only when the commented function’s text changes while the status is not yet FIXED. Once FIXED is asserted, it stops checking by design, so its zero recurrence recall follows from the policy. ALWAYS reassesses at every revision. FILE, FUNC and REVAL (FUNC+GLOBALS) reassess when their footprint is stale (Algorithm 1). ALWAYS and ONCE run without the tracer.
Witness conditions
Experiment A: oracle-sampled witnesses. With , the witness is the reviewer’s example (). With , it is plus two checks sampled from without replacement, independently for each concern. Each concern is evaluated under 5 independent draws, and every oracle has at least six checks, so no budget exhausts it; the code asserts this. This condition is a controlled ablation. It gives the method well-chosen checks from the same source as the ground truth, so it shows what maintenance achieves given strong witnesses.
Experiment B: blind comment-derived witnesses and held-out ground truth. To evaluate the approach on concerns where no oracle is available, two further authors wrote checks under blinding. Each author was a separate AI agent instance with no conversation history. Each saw only the review comment and the commented revision of core.py for every family, was instructed not to open any other file, and never saw the oracle, the variants or any future revision. The witness author wrote exactly three checks per family, as a review assistant would, with the first reproducing the reviewer’s example. The held-out author independently wrote 6–7 acceptance checks per family that define “fully fixed without breaking legitimate behaviour”. Both files were frozen (SHA-256 recorded) before any evaluation and were not edited afterwards. In Experiment B, ground truth is the held-out suite. The witnesses are scored against it and, for comparison, against the original oracle. Blinding was enforced by instruction and by giving each agent only the input file, not by sandboxing. Both authors are instances of the same model family as the one that assisted in preparing this study. They are therefore independent of the oracle’s content but not independent human judges (§ 7.4).
Statistical analysis and runtime
Assessments within a family share code and checks, so all confidence intervals are 95% percentile intervals from a family-clustered bootstrap [4] with 2,000 resamples of the 12 families. One matrix of resampled family indices is shared by all methods and all paired differences. Methods with identical predictions therefore receive identical intervals, and paired differences are directly comparable. Two sources of uncertainty are kept separate. Family selection is captured by the bootstrap of the draw-averaged metric. Witness sampling is captured by the range of the point estimate across the 5 draws. With twelve clusters, intervals indicate stability across concern types, not population inference.
Runtime is measured as wall-clock time per concern, median of 7 repeats on Python 3.13.16, with the witnesses of draw 0. It is split into execution (compiling, loading and running checks, tracer included where active), fingerprinting (parsing units), staleness comparison and remaining policy logic.
Results
RQ1: Mechanism baselines diverge from repair in structured ways
In the synthetic histories, the sticky outdated-line proxy falsely marked 460 of 631 still-present concern states as resolved (FRR = 0.73, CI 0.66–0.78; Table 4). It disagreed with oracle status at least once in 66.7% of histories, and its fixed-label precision was 0.53. The errors follow the mechanisms (Figure 3). Every cosmetic edit, inline regression and configuration regression produced a false resolution. Errors also ran the other way: on 41 fixed states the commented line survived unchanged because the fix added lines around it. These magnitudes follow from the proxy’s construction and the scenario mix, not from measured platform behaviour. What generalises is the pattern: a line-change signal cannot separate cosmetic from behavioural edits, nor detect regressions after it fires.
| Method | FRR | PrecF | Macro-F1 | Rec | FlagU | Exec. |
| Mechanism baselines (no execution) | ||||||
| Outdated-line proxy | 0.73 [0.66, 0.78] | 0.53 | 0.54 | 0.05 [0.00, 0.16] | 0.12 | 0.00 |
| Surface-pattern judge | 0.37 [0.33, 0.41] | 0.69 | 0.76 | 0.36 [0.29, 0.48] | 1.00 | 0.00 |
| Evidence, reviewer’s example only () | ||||||
| Verify-once | 0.48 [0.46, 0.49] | 0.65 | 0.74 | 0.00 [0.00, 0.00] | 0.00 | 1.11 |
| Always recheck | 0.37 [0.33, 0.40] | 0.71 | 0.80 | 0.45 [0.33, 0.59] | 1.00 | 2.91 |
| File-level footprint | 0.37 [0.33, 0.40] | 0.71 | 0.80 | 0.45 [0.33, 0.59] | 1.00 | 2.53 |
| Function footprint | 0.37 [0.33, 0.40] | 0.70 | 0.80 | 0.42 [0.31, 0.56] | 1.00 | 1.89 |
| REVAL (functions + globals) | 0.37 [0.33, 0.40] | 0.71 | 0.80 | 0.45 [0.33, 0.59] | 1.00 | 1.91 |
| Evidence, oracle-sampled witness set (; mean of 5 draws) | ||||||
| Verify-once | 0.30 [0.27, 0.34] | 0.75 | 0.84 | 0.00 [0.00, 0.00] | 0.00 | 3.51 |
| Always recheck | 0.11 [0.06, 0.16] | 0.89 | 0.94 | 0.80 [0.73, 0.88] | 1.00 | 8.73 |
| File-level footprint | 0.11 [0.06, 0.16] | 0.89 | 0.94 | 0.80 [0.73, 0.88] | 1.00 | 7.60 |
| Function footprint | 0.12 [0.07, 0.18] | 0.88 | 0.93 | 0.73 [0.63, 0.85] | 1.00 | 5.67 |
| REVAL (functions + globals) | 0.11 [0.06, 0.16] | 0.89 | 0.94 | 0.80 [0.73, 0.88] | 1.00 | 5.73 |
Table 4. Experiment A (oracle-sampled witnesses) over 1284 post-comment assessments of 441 concerns. PrecF: fixed-label precision; Rec: recurrence recall; FlagU: contract-change states not asserted FIXED; Exec.: post-comment test-case executions per concern. Brackets: 95% family-clustered bootstrap CIs (shared draws). rows average 5 witness draws. Decision coverage on determinate states is 1.00 for all methods.
Figure 3. False-resolution rate by divergence type (columns) and method (rows), Experiment A. Each method fails on a characteristic subset, as Corollary 1 predicts.
The surface-pattern heuristic reached FRR = 0.37 and failed in complementary places. It was immune to cosmetic edits and inline regressions, which remove defensive constructs, but accepted 0.93 of partial fixes and 0.92 of remote regressions, because these look like fixes or leave the commented function unchanged.
RQ2: Maintaining evidence detects recurrence at fewer executions
ONCE reduced false resolution relative to the proxy but, by design, had zero recurrence recall. With the same witnesses, REVAL reduced the false-resolution rate from 0.30 to 0.11 (paired difference -0.20, CI -0.22 to -0.18). It raised recurrence recall to 0.80 (CI 0.73–0.88). Across witness draws, its FRR ranged from 0.10 to 0.12.
REVAL’s decisions were identical to ALWAYS in all 6,420 paired assessments (100.0%), as Proposition 1 predicts when the footprint is complete. It needed 5.73 instead of 8.73 post-comment test-case executions per concern, 34% fewer (Figure 4). Unrelated edits rarely triggered reassessment (5.0% of unrelated-only revisions, all of them while a concern was UNDETERMINED), against 58.5% for file-level footprints. Contract changes were flagged UNDETERMINED by every re-checking policy. In S13, re-adding the target by a new definition was always detected and reassessed. The sticky policies carried their earlier FIXED over (FlagU column).
Figure 4. Accuracy versus post-comment test-case executions (Experiment A). Squares: baselines; circles: ; diamonds: . REVAL (orange) matches ALWAYS with fewer executions. Wall-clock runtime is a different matter (§ 6.5).
RQ3: Dependency granularity and witness strength govern residual error
Granularity. Function-only footprints missed every configuration regression (Figure 3: 1.00 vs. 0.17). This lowered recurrence recall by 0.07 (CI 0.00–0.14) and agreement with ALWAYS to 99.2%. Adding globals read by executed code, together with namespace units, restored full agreement at 0.06 extra executions per concern. File-level footprints also agreed fully but reassessed on most unrelated edits.
Witness strength. With only the reviewer’s example, every evidence policy accepted all partial fixes. This is a witness-insufficiency floor (FRR ≈ 0.37) that no revalidation removes, and its size partly reflects our construction of partial fixes. Figure 5 varies using the same sampler and draws as Table 4, so its point (0.106) equals the table entry. Under REVAL, FRR falls from 0.366 () through 0.205 and 0.106 to 0.058 and 0.026 (), and recurrence recall rises from 0.45 to 0.97. No budget exhausts an oracle, so still leaves at least one hidden check. At every , REVAL matched ALWAYS, with 34–34% fewer post-comment executions. This stability is an observation, not a guarantee. Extra witnesses can enlarge the dynamic footprint and change which revisions require reassessment. Here they did not, because all witnesses of a family exercise the same functions.
Figure 5. Witness-set size ablation under REVAL, oracle-sampled witnesses (same sampler as Table 4; band: range over 5 draws). Left: false-resolution rate. Right: post-comment executions for REVAL and ALWAYS. Decisions of the two policies were identical at every .
Figure 2. False-resolution rate in Experiment A with 95% family-clustered bootstrap CIs (shared draws). Grey: mechanism baselines; light blue: reviewer’s example only; dark blue: oracle-sampled (mean of 5 draws); orange: REVAL.
Witnesses written without oracle access (Experiment B)
Table 5 reports Experiment B, separately from Experiment A. The two blind ground truths largely agree: the held-out suite and the original oracle gave the same status on 92.8% of assessments. The 92 disagreements are concentrated in three families where the comment underdetermines intent. Is mutating a caller-supplied list acceptable in add_tag? Must non-transient errors skip retries in with_retry? Does a “dotted domain” exclude a trailing dot in normalize_email? Specification ambiguity is thus a real source of disagreement even before any assessor is involved.
Against the held-out truth, REVAL with all three comment-derived witnesses reached FRR (CI 0.00–0.26). Its recurrence recall was 0.88, and it never rejected a held-out-fixed state (missed-repair rate 0.00). That compares with 0.36 using only the first comment-derived witness (difference -0.23, CI -0.31 to -0.14) and 0.32 for ONCE with the same witnesses (difference -0.19, CI -0.24 to -0.15). Again
REVAL matched ALWAYS on every decision (100.0%). Scored against the original oracle instead, the same method reached FRR = 0.11. The remaining false resolutions came from three families. In mean and display_name, the witness author did not test a sibling input (an empty tuple, an empty profile), which is witness insufficiency. In normalize_email, the two blind authors read the comment differently, which is specification disagreement rather than an assessment error. In this setting, then, comment-derived witnesses performed comparably to oracle-sampled ones of the same size (0.17 against the held-out truth). The confidence intervals are wide.
Runtime and overhead
Execution counts are not runtime. Table 6 separates the components. In this benchmark a single check runs in about 44 s untraced and 122 s under Python’s call tracer, a factor of 2.8. Fingerprinting dominates REVAL’s cost. As a result, REVAL was 3.3 slower in wall-clock time than untraced exhaustive re-checking, despite 34% fewer executions. Its overhead per concern (1.27 ms) is repaid only when one test execution costs more than about 0.42 ms. Real test suites usually exceed that by orders of magnitude, and LLM assessments exceed it far more, but we did not measure such settings. The efficiency claim of this paper is therefore limited to fewer post-comment test-case executions. Total computational savings depend on the cost of an assessment relative to tracing and fingerprinting.
| Policy | Exec./concern (incl. ) | Execution ms | Fingerprint ms | Staleness ms | Other ms | Total ms |
| Always recheck (untraced) | 11.73 | 0.40 | 0.00 | 0.00 | 0.10 | 0.50 |
| Verify-once (untraced) | 6.49 | 0.25 | 0.00 | 0.00 | 0.25 | 0.50 |
| File-level footprint | 10.60 | 0.85 | 0.46 | 0.01 | 0.15 | 1.46 |
| REVAL | 8.73 | 0.75 | 0.73 | 0.01 | 0.16 | 1.64 |
Table 6. Wall-clock runtime per concern (median of 7 repeats, , draw 0), including the initial assessment at . Execution includes the tracer where active.
| Method | FRR | Prec, | Macro- | Rec, | Flag, | Exec. |
| Mechanism baselines | ||||||
| Outdated-line proxy | 0.73 [0.65, 0.80] | 0.52 | 0.53 | 0.06 [0.00, 0.21] | 0.12 | 0.00 |
| Surface-pattern judge | 0.35 [0.27, 0.41] | 0.70 | 0.79 | 0.36 [0.30, 0.47] | 1.00 | 0.00 |
| Blind comment-derived witnesses, first check () | ||||||
| Verify-once | 0.46 [0.36, 0.56] | 0.65 | 0.74 | 0.00 [0.00, 0.00] | 0.00 | 1.12 |
| Always recheck | 0.36 [0.22, 0.49] | 0.71 | 0.81 | 0.52 [0.37, 0.70] | 1.00 | 2.91 |
| REVAL (functions + globals) | 0.36 [0.22, 0.49] | 0.71 | 0.81 | 0.52 [0.37, 0.70] | 1.00 | 1.91 |
| Blind comment-derived witnesses, all three () | ||||||
| Verify-once | 0.32 [0.23, 0.43] | 0.73 | 0.83 | 0.00 [0.00, 0.00] | 0.00 | 3.51 |
| Always recheck | 0.13 [0.00, 0.26] | 0.87 | 0.93 | 0.88 [0.72, 1.00] | 1.00 | 8.73 |
| REVAL (functions + globals) | 0.13 [0.00, 0.26] | 0.87 | 0.93 | 0.88 [0.72, 1.00] | 1.00 | 5.73 |
| Reference: oracle-sampled witnesses () | ||||||
| Verify-once | 0.35 [0.26, 0.44] | 0.71 | 0.80 | 0.00 [0.00, 0.00] | 0.00 | 3.51 |
| REVAL (functions + globals) | 0.17 [0.06, 0.29] | 0.83 | 0.88 | 0.82 [0.73, 0.91] | 1.00 | 5.73 |
Table 5. Experiment B: witnesses written blind from the review comment and initial code, scored against an independently, blind-written held-out suite (: 639 PRESENT, 553 FIXED, 128 recurring states). Columns and CIs as in Table 4. Oracle-sampled rows are re-scored against for reference.
Summary: support within the constructed setting
Within the synthetic histories, the mechanism baselines diverged from repair in structured ways (P1). Policies that stop after first success detected no recurrence, while maintained evidence did (P2). Dependency-gated revalidation reproduced exhaustive re-checking with fewer executions, provided the footprint covered globals and namespaces, and its residual error was governed by witness strength and, in Experiment B, by specification ambiguity (P3). These are mechanism-level results. Their prevalence and their payoff on real repositories remain to be measured.
Discussion
Implications for tool builders
Resolve with a receipt. An automated resolution should record the revision assessed, the checks or reasoning used, and what that judgement depended on. This makes “resolved” auditable, in the spirit of proof-carrying code [19].
Resolution is not terminal. A resolved concern should re-open when its evidence goes stale, as a build system invalidates a cached result [18]. The footprint must include module-level state and namespaces, not only functions.
Abstain when the subject moves, and reassess until re-anchored. Renames should yield an explicit “needs re-anchoring” state, which is reassessed at every revision until valid evidence exists again.
Spend effort on witnesses; a proposed workflow. Residual error was dominated by witness insufficiency and specification ambiguity. We therefore propose, without having evaluated it, a workflow in which an assistant drafts boundary and sibling cases from the comment and the reviewer confirms or edits them, so that ambiguities such as those in § 6.4 are resolved by the person who raised the concern. Experiment B evaluated only the first half, assistant-drafted witnesses without reviewer confirmation. The confirmation step, its cost to reviewers and its effect on accuracy are untested.
Mind the overhead. Where assessments are cheap, re-checking everything may be faster than maintaining footprints (§ 6.5). Evidence maintenance pays off where assessments are expensive, such as integration tests or LLM-based judgements, and where the receipt itself has value.
Implications for researchers
Thread state (isResolved, isOutdated) is an observed workflow variable, not a correctness label. Reported “resolution rates” of AI-generated comments [5, 7] measure tool and developer behaviour, not the repair rate of the underlying concerns. Benchmarks for review agents [31, 35] would gain from scoring concerns longitudinally and at matched decision coverage [10]. They should also report agreement between independently specified ground truths, since our two blind specifications disagreed on 92 of 1284 states.
Protocol for naturally occurring histories
We defer a natural-history study and pre-specify its protocol (Table 7).
| Experiment | Design | Establishes |
| E1 Proxy audit | Observed thread closure and outdated status vs. adjudicated per-revision status | Prevalence of each divergence (P1) |
| E2 Assessor comparison | Per-revision vs. history-aware vs. evidence-maintaining LLM judges, matched budgets | Whether evidence helps beyond context (P3) |
| E3 Stress sequences | Cosmetic, partial, complete, recurrence and control edits built from real concerns | Tracking of the underlying concern (P2) |
| E4 Executable subset | Independent tests on 30–50 runnable cases | Label agreement with behaviour |
| E5 Witness workflow | Assistant-drafted witnesses with and without reviewer confirmation | Value and cost of confirmation |
Table 7. Pre-specified protocol for the natural-history study.
Corpus. Start from AIDev [14], whose review tables cover pull requests in repositories with more than 100 stars. Reconstruct histories from the GitHub review-comment and review-thread APIs. Record PR authorship and comment authorship separately. Begin with Python and behaviourally checkable concerns. Audit 50 histories for reconstruction reliability, then target about 300 atomic concerns across at least 15 repositories.
Ground truth. Two human annotators independently label each concern’s validity and its per-revision status. Agreement is reported before adjudication [6]. A 30–50-case runnable subset receives independently written tests [35].
Assessors and controls. Compare thread state and outdated status as observed (not simulated), a per-revision LLM judge, the same judge with full history, and the judge gated by a REVAL-style record. Use at least two model families with matched information and budgets, an ablation without the record, hidden future revisions and repository hold-outs. Measure wall-clock and monetary cost alongside assessment counts.
Falsification. The position is weakened in three cases. First, if observed thread state agrees closely with verified repair (P1). Second, if recurrence after correct resolution is too rare for post-closure checking to matter (P2). Third, if evidence records do not reduce FRR over an equally informed per-revision judge at matched coverage and cost (P3).
Threats to validity
Construct validity. Each oracle encodes one reading of a comment. Experiment B shows that a second, blind reading agrees on 92.8% of states, and that the disagreements are interpretable ambiguities. The outdated-line proxy is a simplification of GitHub’s diff-position logic, and it is sticky by design. It should be read as a mechanism baseline, not as an estimate of platform or developer error.
Internal validity. We designed the code variants, so magnitudes partly reflect design choices. In particular, partial fixes pass the reviewer’s example by construction. Ground truth is always computed by execution. All methods are online and deterministic. The blind authors in Experiment B are AI agent instances, blinded by instruction rather than sandboxing, and from the same model family that assisted with the study. Their independence is therefore from the oracle’s content, not from shared model biases. A human-authored replication is part of the protocol.
External validity. Programs are small, single-language and deterministic. Many real concerns, such as readability or design, have no executable witness. Real dependencies include I/O, configuration files and non-determinism outside our dependency model. Runtime results depend on the cost of an assessment relative to tracing, and we measured only microsecond-scale checks.
Conclusion validity. With twelve clusters, bootstrap intervals are approximate. Family-selection and witness-sampling uncertainty are reported separately. Key contrasts are large relative to both, but small differences should not be over-interpreted.
Conclusion
As AI assistants begin to decide when review concerns are resolved, the field needs a definition of resolution that survives code evolution. We proposed one: a resolution is a revision-specific, evidence-carrying claim that must be revalidated when what it depends on changes.
Key findings (synthetic histories). (1) A sticky outdated-line proxy falsely marked 72.9% of still-present concern states as resolved, failing on cosmetic edits, partial fixes and all regressions. (2) A policy that stops after first success detected no recurrence. (3) REVAL reduced false resolution to 0.11 with oracle-sampled witnesses and to 0.13 with blind comment-derived witnesses scored against an independent held-out suite. It detected most recurrences and matched exhaustive re-checking on every decision, with 34% fewer post-comment test-case executions. (4) Residual error came from witness insufficiency and specification ambiguity. Function-only footprints additionally missed configuration regressions.
Limitations. The evidence is synthetic and covers behavioural concerns in small deterministic programs. Blind authors were AI agents, not humans. No LLM assessor was evaluated. With microsecond-scale checks, the approach was slower in wall-clock time than re-checking everything.
Implications. Tools should attach receipts to automated resolutions, re-open concerns when receipts go stale, abstain when the subject moves, and invest in witnesses confirmed by reviewers. Researchers should not treat thread state as a repair label, and should report agreement between independent specifications.
Future directions. Future work should (i) run the natural-history protocol, with human annotators; (ii) gate LLM assessors with evidence records and measure monetary cost; (iii) evaluate reviewer-confirmed witness drafting; and (iv) extend the dependency model to configuration files and I/O, with conservative fallbacks, inside continuous integration.
Data and Code Availability
The replication package (experiment/) contains the following:
the concern families and oracles;
the frozen blind-authored witness and held-out files, with their input file and SHA-256 hashes;
every generated history, with the source code of each revision, ground truth and sampled witnesses (
results/histories.jsonl);raw per-assessment predictions (
results/rows.csv);pinned dependency versions and a single command (
reproduce.sh) that regenerates all tables, figures and numbers;regression tests (
tests.py);verify.py, which checks the reported counts (441 histories, 1284 assessments, class counts), REVAL–ALWAYS agreement, sweep/table consistency and file hashes againstexpected.json.
Declarations
Funding: none. Competing interests: none. Use of AI tools: AI assistance was used for drafting, code development and reference checking. Separate AI agent instances authored the blind witness and held-out checks of Experiment B, as described in § 5.3. The author reviewed and takes responsibility for all content.
References
Concern Families
Blind Authoring Protocol (Experiment B)
Each author received one JSON file listing, per family, the family identifier, target function name, review comment and the commented revision of core.py. The witness author was asked to act as a review assistant and write exactly three checks per family, the first reproducing the reviewer’s example, without over-specifying incidental details such as error messages. The held-out author was asked to write 6–8 acceptance checks per family covering the full scope of the concern, including positive cases. Both were told not to open any other file and could execute their checks only against the defective initial code. Both reported their resolutions of ambiguity. For example, both accepted any placeholder style for SQL parameters and supplied several candidate keys for the unspecified “account name”. The held-out author read “dotted domain” strictly and did not require non-transient errors to skip retries. The instructions, the outputs and their hashes are part of the replication package.
References
- [1]Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In Proceedings of the 35th International Conference on Software Engineering (ICSE), pages 712–721. IEEE, 2013. doi: 10.1109/ICSE.2013.6606617.DOI
- [2]Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. Modern code reviews in open-source projects: Which problems do they fix? In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR), pages 202–211. ACM, 2014. doi: 10.1145/2597073.2597082.DOI
- [3]Amíangshu Bosu, Michaela Greiler, and Christian Bird. Characteristics of useful code reviews: An empirical study at Microsoft. In Proceedings of the 12th IEEE/ACM Working Conference on Mining Software Repositories (MSR), pages 146–156. IEEE, 2015. doi: 10.1109/MSR.2015.21.
- [4]A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3):414–427, 2008. doi: 10.1162/rest.90.3.414.DOI
- [5]Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. Automated code review in practice. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2025. URL https://arxiv.org/abs/2412.18531. arXiv:2412.18531.
- [6]Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. doi: 10.1177/001316446002000104.DOI
- [7]Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy, Ting Zhang, and David Lo. “go home Copilot, you’re drunk”: Understanding developer responses to agent-generated code review comments. arXiv preprint arXiv:2607.21997, 2026. URL https://arxiv.org/abs/2607.21997.
- [8]Jon Doyle. A truth maintenance system. Artificial Intelligence, 12(3):231–272, 1979. doi: 10.1016/0004-3702(79)90008-0.DOI
- [9]Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. Resolving code review comments with machine learning. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 204–215. ACM, 2024. doi: 10.1145/3639477.3639746.DOI
- [10]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html.arxiv.org/abs/1705.08500
- [11]GitHub. Copilot code review: An improved review experience. GitHub Changelog, September 2026. URL https://github.blog/changelog/2026-09-18-copilot-code-review-an-improved-review-experience. Accessed 2026-10-20.
- [12]Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. Practical regression test selection with dynamic file dependencies. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), pages 211–222. ACM, 2015. doi: 10.1145/2771783.2771784.DOI
- [13]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=VTf8yNQM66.arxiv.org/abs/2310.06770
- [14]Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003. Dataset: https://huggingface.co/datasets/hao-li/AIDev, 2025. URL https://arxiv.org/abs/2507.15003. Accessed 2026-10-20.
- [15]Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 1035–1047, 2022. doi: 10.1145/3540250.3549081.
- [16]Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, and Christoph Treude. CodeReviewQA: The code review comprehension assessment for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9138–9166, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.476.DOI
- [17]Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21 (5):2146–2189, 2016. doi: 10.1007/s10664-015-9381-9.DOI
- [18]Andrey Mokhov, Neil Mitchell, and Simon Peyton Jones. Build systems à la carte. Proceedings of the ACM on Programming Languages, 2(ICFP):1–29, 2018. doi: 10.1145/3236774.DOI
- [19]George C. Necula. Proof-carrying code. In Proceedings of the 24th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), pages 106–119. ACM, 1997. doi: 10.1145/263699.263712.DOI
- [20]Delano Oliveira, Reydne Santos, Benedito de Oliveira, Martin Monperrus, Fernando Castor, and Fernanda Madeira. Understanding code understandability improvements in code reviews. IEEE Transactions on Software Engineering, 51(1):14–37, 2025. doi: 10.1109/TSE.2024.3453783.DOI
- [21]Sebastiano Panichella and Nik Zaugg. An empirical investigation of relevant changes and automation needs in modern code review. Empirical Software Engineering, 25(6):4833–4872, 2020. doi: 10.1007/s10664-020-09870-3.DOI
- [22]Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J. Mooney. Deep just-in-time inconsistency detection between comments and source code. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 427–435, 2021. doi: 10.1609/aaai.v35i1.16119.DOI
- [23]Gregg Rothermel and Mary Jean Harrold. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(2):173–210, 1997. doi: 10.1145/248233.248262.DOI
- [24]Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern code review: A case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 181–190. ACM, 2018. doi: 10.1145/3183519.3183525.DOI
- [25]Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, and Ming Wu. Halludoge: A reference-free hallucination detection for context misalignment in code review automation. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (FSE Companion ’26), Industry Track, pages 632–643. ACM, 2026. doi: 10.1145/3803437.3805236. arXiv:2601.19072.DOI
- [26]Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards automating code review activities. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE), pages 163–174, 2021. doi: 10.1109/ICSE43902.2021.00027.DOI
- [27]Rosalia Tufano, Ozren Dabić, Antonio Mastroppoalo, Matteo Ciniselli, and Gabriele Bavota. Code review automation: Strengths and weaknesses of the state of the art. IEEE Transactions on Software Engineering, 50(2):338–353, 2024. doi: 10.1109/TSE.2023.3348172.DOI
- [28]Asif Kamal Turzo and Amiangshu Bosu. What makes a code review useful to OpenDev developers? An empirical investigation. Empirical Software Engineering, 29(1):6, 2024. doi: 10.1007/s10664-023-10411-x.DOI
- [29]Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pascal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, Daniel Tarlow, Petros Maniatis, and René Just. AI-assisted assessment of coding practices in modern code review. In Proceedings of the 1st ACM International Conference on AI-Powered Software (AIware), pages 85–93. ACM, 2024. doi: 10.1145/3664646.3665664.DOI
- [30]Dong Wang, Yuki Ueda, Raula Gaikovina Kula, Takashi Ishio, and Kenichi Matsumoto. Can we benchmark code review studies? A systematic mapping study of methodology, dataset, and metric. Journal of Systems and Software, 180:111009, 2021. doi: 10.1016/j.jss.2021.111009.
- [31]Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, and Haoli Bai. SWE-Review: Closing the loop on issue resolution with agentic code review. arXiv preprint arXiv:2607.06065, 2026. URL https://arxiv.org/abs/2607.06065.
- [32]Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In Proceedings of the 27th IEEE/ACM International Conference on Program Comprehension (ICPC), pages 53–64, 2019. doi: 10.1109/ICPC.2019.00019.DOI
- [33]Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. EvaCRC: Evaluating code review comments. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 275–287, 2023. doi: 10.1145/3611643.3616245.DOI
- [34]Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. A roadmap for modern code review: Challenges and opportunities. ACM Transactions on Software Engineering and Methodology, 35(9):1–42, 2026. doi: 10.1145/3800963. Preprint: arXiv:2405.18216.DOI
- [35]Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. Code review agent benchmark. arXiv preprint arXiv:2603.23448, 2026. URL https://arxiv.org/abs/2603.23448.
- [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, pages 46595–46623, 2023.arxiv.org/abs/2306.05685



