betaFeedback

WorksComputer Science

AI Code Review Requires Revision-Aware Evidence of Concern Resolution

Abstract

AI-assisted code review increasingly judges whether reported concerns have been resolved. Yet closing a thread or modifying a commented line does not establish repair, and later revisions can invalidate earlier fixes. This position paper argues that resolution requires inspectable, revision-specific evidence that is revalidated when its dependencies change. We model review concerns as persistent verification obligations, derive a sufficient condition under which dependency-gated revalidation matches exhaustive reassessment, and introduce REVAL, a lightweight procedure adapting principles from regression-test selection and build systems. We evaluate REVAL against oracle ground truth in synthetic revision histories covering 441 concerns, 1,284 assessments, and 12 defect families. A sticky outdated-line proxy falsely marked 72.9% of unresolved states as resolved, while stopping checks after initial success missed every recurrence. With three oracle-sampled witnesses, REVAL achieved a false-resolution rate of 0.11 (95% CI: 0.06-0.16) and detected 80.4% of recurrences. With witnesses derived solely from review comments and assessed using independent held-out checks, the corresponding results were 0.13 and 88.3%. Across 6,420 paired comparisons, REVAL matched exhaustive reassessment while reducing post-comment test executions by 34%. Maintenance overhead increased runtime for microsecond-scale tests; the estimated break-even test cost was 0.42 ms. These findings support revision-aware evidence maintenance in a controlled setting, while distinguishing revalidation correctness from witness adequacy. A prespecified evaluation protocol outlines how to assess generalisability in real review histories.

Introduction

Modern code review is an established practice for improving software quality in industrial and open-source development [1, 17, 24]. Its benefits extend beyond defect detection to include improvements in maintainability and evolvability [2, 21]. Research on review automation has progressed from pre-trained models for generating comments and refining code [15, 26] to deployed systems that propose repairs, check development practices, and participate in pull-request workflows [5, 9, 29]. More recently, automated support has extended to determining whether previously reported concerns have been addressed [11]. This development introduces a distinct verification problem: what evidence justifies declaring a review concern resolved, and under what conditions does that judgement remain valid?

The distinction between administrative resolution and substantive repair is central to this problem. Review platforms record workflow events, including thread closure and the displacement of comments from their original code locations. These events describe the progression of a review, but do not, by themselves, establish that the underlying concern has been corrected. Nevertheless, resolution status is also used to evaluate automated review systems. Industrial reports characterise assistant impact through the proportion of automated comments that were resolved [5], while empirical studies operationalise resolution using platform thread state [7]. Interpreting such measures as evidence of successful repair requires an assumption that administrative state corresponds to the status of the underlying concern. The validity of this assumption is therefore consequential for both automated decision-making and empirical evaluation.

A further difficulty arises from the temporal nature of software development. A repair that is correct at one revision may be invalidated by subsequent changes. These changes need not affect the originally commented lines: modifications to a helper function, configuration setting, or other dependency may reintroduce the concern while leaving the earlier resolution decision unchanged. Resolution should therefore be understood as a claim about a particular code revision. Maintaining that claim requires determining whether the evidence supporting it remains applicable as the code evolves.

Consider a reviewer who observes at revision r0r_0 that parse_quantity('-3') returns −3-3, although all non-positive quantities should raise ValueError. At r1r_1, reformatting makes the comment outdated without changing the behaviour. At r2r_2, a guard rejects negative values but continues to accept zero, producing a partial repair. At r3r_3, validation is moved to a helper with the correct boundary condition, fully addressing the concern. At r4r_4, a change to that helper causes zero to be accepted again. An outdated-line proxy registers a workflow transition at r1r_1, whereas a check restricted to the reviewer’s example passes from r2r_2 onwards. Neither establishes that the complete concern is absent only at r3r_3 (Figure 1). The example distinguishes three sources of error: substituting workflow state for repair evidence, using evidence that captures only part of a concern, and retaining a resolution judgement after its supporting evidence has become invalid.

Decision matrix for the running example

Figure 1. The running example, built from the quantity family’s variants. Each cell is the decision computed by running the method online. The k=3k=3 witness set is the reviewer’s example plus the cases ’0’ and ’7’. The outdated-line proxy resolves at the cosmetic edit. Evidence from the reviewer’s example alone accepts the partial fix. Only a maintained, stronger witness set tracks the true sequence.

This paper advances the position that a resolution decision should be tied to a specific code revision and supported by inspectable evidence; when subsequent changes invalidate that evidence, the concern must be reassessed. Under this position, review concerns are persistent verification obligations whose status is maintained throughout a revision history. Two requirements follow. First, the evidence must adequately represent the concern being assessed. Second, its reuse at later revisions must be justified by the stability of the dependencies on which it relies. These requirements distinguish the strength of the evidence from the correctness of the procedure used to maintain it.

The position comprises three claims:

  • (P1) Measurement validity. Administrative and syntactic proxies, including outdated-line status and changed-line heuristics, can diverge from the substantive repair of a review concern.

  • (P2) Temporal validity. A resolution judgement that is correct at one revision can become incorrect at a later revision. Methods that cease checking after the first successful assessment cannot detect subsequent recurrence.

  • (P3) Evidence maintenance. Under a sufficient dependency model, revalidating evidence when its dependencies change can preserve the decisions obtained by exhaustive reassessment while reducing executions. Agreement with exhaustive reassessment does not, however, establish that the evidence fully captures the concern.

The computational principles underlying evidence maintenance are established. Safe regression-test selection identifies tests that require re-execution after a change [12, 23]; build systems reuse results while their dependency fingerprints remain valid [18]; and truth-maintenance systems revise beliefs when their justifications no longer hold [8]. The contribution of this paper lies in applying and evaluating these principles at the level of persistent review concerns. A concern expresses an obligation that may be only partially represented by its executable witnesses. Evaluating its resolution consequently requires separating witness adequacy, dependency validity, and the policy governing reassessment.

We instantiate this perspective as REVAL, a lightweight procedure for maintaining revision-specific resolution evidence. The procedure records the dependencies of an assessment, reuses its result while those dependencies remain unchanged, and triggers reassessment when relevant changes occur. Its evaluation follows individual concerns across revision histories, explicitly accounting for partial repairs and recurrence. The study addresses three research questions:

RQ1 In controlled revision histories, how frequently, and under which conditions, do an outdated-line proxy and a surface-pattern heuristic disagree with oracle-verified concern status?

RQ2 To what extent does maintaining revision-specific evidence reduce false resolution and improve recurrence detection relative to stopping after first success, and what is its cost relative to reassessment at every revision?

RQ3 How do witness strength and dependency granularity affect the remaining errors, including when witnesses are derived solely from review comments and evaluated against independently written checks?

The paper makes five contributions:

(C1) A formal model of review concerns as revision-indexed verification obligations, including a sufficient condition for dependency-gated revalidation to match exhaustive reassessment (Proposition 1), an explicit account of the guarantee’s limits (Remark 1), and a decomposition of false resolution into three mechanisms (Corollary 1).

(C2) A taxonomy characterising divergence between administrative closure and substantive repair (Table 2).

DivergenceWhat happensMechanism
Cosmetic editCommented lines change, so the comment becomes outdated, but behaviour is unchanged(a) no evidence
Partial fixReviewer’s example passes; other in-scope cases still fail(b) witness insufficiency
Relocated fixRepair moves the check to a helper; line heuristics mis-score a correct fix(a) for line proxies
Inline regressionA later edit of the same function drops the repair(c) stale (policy)
Remote regressionA helper in another file is weakened; the commented function is untouched(c) stale; needs footprint
Configuration regressionA module-level constant changes and re-opens the defect(c) incomplete footprint
Contract changeThe subject is renamed and the evidence cannot run; it may later be restoredshould yield UNDETERMINED
Invalid concernThe original comment was wronghuman disposition
Obsolete concernRequirements changed and the concern no longer applieshuman disposition

Table 2. Taxonomy of divergence between administrative closure and actual repair, with the mechanism of Corollary 1. Rows above the rule are instantiated in the controlled study.

(C3) REVAL, an evidence-maintenance procedure with an explicit dependency model, conservative fallbacks, and regression tests for edge cases (§ 4).

(C4) A controlled, executable evaluation using oracle-sampled and blind comment-derived witnesses, with the latter assessed against independently written held-out checks. The evaluation includes shared-draw bootstrap uncertainty, separate witness-sampling uncertainty, and runtime measurements that distinguish execution costs from maintenance overhead (§ 5 and § 6).

(C5) A pre-specified protocol for evaluating naturally occurring pull-request histories (§ 7.3), supported by a replication package containing verification checks.

The empirical scope is deliberately bounded. All quantitative findings are obtained from synthetic revision histories constructed from small, deterministic Python programs. The baselines isolate mechanisms through which resolution judgements can fail; they do not constitute measurements of GitHub, Copilot, or developer behaviour. The study therefore evaluates these mechanisms and their relative effects within the constructed setting, without estimating their prevalence in operational review workflows. The protocol for naturally occurring histories specifies the subsequent evaluation needed to assess external validity. Accordingly, the paper retains a position-paper framing: it establishes the conceptual and formal basis for revision-aware concern resolution and provides controlled evidence supporting further empirical investigation.

Background and Related Work

Modern Code Review and Review Outcomes

Empirical research has established that code review serves several complementary purposes. Although defect detection is a prominent motivation, field studies also identify maintainability improvements, knowledge transfer, and shared awareness as important outcomes [1, 24]. Repository-mining studies have characterised the problems corrected during review [2], examined associations between review practices and post-release defects [17], and identified review-related changes that developers consider suitable for automation [21]. Related work has modelled comment usefulness using developer judgements and observable comment characteristics [3, 28].

These studies establish the value and diversity of review outcomes, but evaluating whether a comment is useful differs from determining whether its underlying concern has been repaired. A useful comment may lead to an incomplete repair, and an initially successful repair may subsequently be reversed. A particularly relevant temporal perspective is provided by Oliveira et al. [20], who trace understandability-improvement comments through accepted changes and later reversions. Their analysis examines the persistence of review-induced changes. The present paper addresses the related but distinct question of whether a behavioural concern is absent at each subsequent revision, including cases in which the implementation changes without an explicit reversion of the original repair.

Automation of Code Review and Comment Resolution

Early approaches to review automation used pre-trained models to generate comments and refine code in response to review feedback [15, 26]. Subsequent qualitative analysis identified recurring limitations underlying their apparent strengths [27]. Industrial deployments have extended this work to concrete review activities. At Google, authors addressed 7.5% of all reviewer comments by applying an ML-suggested edit [9]. Other systems support LLM-based best-practice checking [29] and automated participation in pull-request review; a deployment at Beko reported that 73.8% of automated comments were resolved [5].

Agent-generated development activity has also enabled datasets such as AIDev [14]. Using AIDev-derived repositories, Cynthia et al. [7] examine developer responses to agent-generated review comments and operationalise resolution through thread state. Automatic comment-resolution functionality extends automation to the resolution decision itself [11].

These developments make the interpretation of resolution increasingly important. Applying a suggested edit, closing a thread, and establishing that a concern has been repaired are distinct outcomes. Reported adoption or resolution rates describe the corresponding workflow events, but their interpretation as evidence of substantive repair requires additional validation. Our study examines this distinction through controlled mechanism baselines. It makes no assumption that the cited systems implement those baselines or share their failure rates.

Evaluation of Automated Review

Evaluation methods for automated review address several different constructs. A mapping study documents heterogeneity in datasets and metrics [30], while a recent roadmap advocates evaluation that accounts for context and practical value [34]. EvaCRC models attributes of review-comment quality [33]; CodeReviewQA evaluates comprehension of review intent [16]; and HalluJudge identifies generated comments that are insufficiently grounded in the code [25]. These approaches assess properties of a comment or a model’s interpretation of it.

Other evaluations incorporate executable evidence or iterative development. The c-CRAB benchmark derives tests from human reviews to assess agent-generated reviews [35], while SWE-Review evaluates agentic reviewers through the resolve rate of generate–review–revise loops [31]. Repository-level issue-resolution benchmarks provide a broader precedent for evaluating software changes against executable criteria [13]. LLM-as-judge research offers a further methodological basis for assessments that rely on model-generated judgements [36].

The present paper complements these approaches by changing the unit and temporal scope of evaluation. Its unit is an individual review concern, and its outcome is the status of that concern at each revision. This formulation distinguishes initial repair from partial repair and subsequent recurrence. It also separates two questions that an aggregate success measure may combine: whether the available evidence is strong enough to represent the concern, and whether that evidence remains valid after later changes. Table 1 summarises these distinctions for selected related studies.

WorkUnit of evaluationPer concernAcross revisionsPartial repairsRecurrenceExecuted evidence
EvaCRC [33]Comment quality attributes–––––
CodeReviewQA [16]Comprehension of review intent✓––––
HalluJudge [25]Grounding of generated comments✓––––
c-CRAB [35]Agent reviews assessed against human-derived tests✓–––✓
SWE-Review [31]Resolution in review–revise loops–◦––✓
Oliveira et al. [20]Persistence of understandability changes✓✓–◦–
Cynthia et al. [7]Developer responses and thread state✓––––
This paperConcern status at each revision✓✓✓✓✓

Table 1. Positioning of selected related studies by evaluation focus. ✓ denotes explicit treatment, ◦ denotes partial treatment, and – denotes a dimension outside the stated evaluation focus.

Artefact Obsolescence and Evidence Validity

Research on code–comment inconsistency examines how documentation that was initially accurate becomes inconsistent with evolving implementations [32]. Just-in-time inconsistency detection addresses this problem at the point of change [22]. This literature provides a useful conceptual parallel for review resolution: an assessment may be justified when made but become unsupported as the code changes.

The object maintained in this paper is the evidential basis of a resolution judgement. A review thread may retain its resolved status even after a change invalidates the checks that justified closure. Conversely, a substantial code change need not invalidate a judgement if the relevant evidence remains applicable. The resulting problem is therefore one of maintaining the relationship among a concern, its supporting evidence, and the revision to which that evidence applies.

Dependency-Based Revalidation and Checkable Evidence

Several established approaches provide the computational foundations for maintaining evidence under change. Safe regression-test selection determines which tests require re-execution following a program modification [23]. Dynamic dependency tracking supports selective execution using dependencies observed during earlier runs, including file-level tracking [12]. Build systems similarly reuse derived results when their inputs and dependency fingerprints remain unchanged [18]. Truth-maintenance systems provide a related account of how derived beliefs should be revised when their justifications cease to hold [8].

REVAL adapts these principles to persistent review concerns. The concern remains the object of assessment, executable witnesses provide evidence about its status, and dependency information determines when that evidence requires revalidation. The formal guarantee concerns agreement with exhaustive execution of the same witnesses under the stated dependency assumptions. It does not establish that those witnesses constitute a complete specification of the concern. This distinction is essential because a sound revalidation policy can consistently preserve the result of an inadequate check.

Two additional research traditions inform the evaluation. Proof-carrying code motivates associating a decision with evidence that can be independently checked [19]. Selective prediction motivates comparing error rates at matched decision coverage when methods may abstain from issuing a judgement [10]. Together, these foundations support the paper’s emphasis on inspectable evidence, explicit conditions for reuse, and evaluation that distinguishes decision reliability from the frequency with which decisions are made.

Concerns as Revision-Indexed Verification Obligations

Programs, revisions and concerns

Let H=⟨r0,…,rn⟩\mathcal{H} = \langle r_0,\ldots,r_n\rangle be the revision history of a change under review. Here r0r_0 is the commented revision and each rtr_t is a complete program snapshot. A revision consists of modules. U(r)\mathcal{U}(r) is its set of units: the top-level functions and module-level bindings of every module, plus one namespace unit νM\nu_M per module MM, whose text is the set of names bound at top level in MM. We write txt⁡r(u)\operatorname{txt}_r(u) for the text of unit uu in rr, with txt⁡r(u)=⊥\operatorname{txt}_r(u) = \bot if u∉U(r)u \notin\mathcal{U}(r). Namespace units make the addition or removal of a binding observable even when no existing unit’s text changes.

Definition 1 (Concern). A concern is c=(ιc,Ac,r0)c = (\iota_c,A_c,r_0), with intent ιc\iota_c and anchor AcA_c (the commented lines of r0r_0). The intent is made operational as an oracle Ωc={o1,…,om}\Omega_c = \{o_1,\ldots,o_m\} of behavioural checks, oi(r)∈{⊤,⊥,⋆}o_i(r) \in\{\top,\bot,\star\}. Here ⋆\star means the check is inapplicable because the subject of the concern cannot be located in rr, and o1o_1 is the reviewer’s own example.

Definition 2 (Concern status).

σc(r)={UNDETERMINEDif ∃i:oi(r)=⋆,FIXEDif ∀i:oi(r)=⊤,PRESENTotherwise.(1)\sigma_c(r) = \begin{cases} \mathrm{UNDETERMINED} & \text{if } \exists i : o_i(r) = \star,\\ \mathrm{FIXED} & \text{if } \forall i : o_i(r) = \top,\\ \mathrm{PRESENT} & \text{otherwise.} \end{cases} \tag*{(1)}

Humans assign two further dispositions, which are never counted as repairs. A concern is INVALID\mathrm{INVALID} if ιc\iota_c was wrong at r0r_0, and OBSOLETE\mathrm{OBSOLETE} if a later requirement change makes it inapplicable.

A concern recurs at rtr_t if σc(rt)=PRESENT\sigma_c(r_t) = \mathrm{PRESENT} and ∃s<t:σc(rs)=FIXED\exists s < t : \sigma_c(r_s) = \mathrm{FIXED}. Status is a function of the revision, not a property of the thread.

Resolution claims and their evidence

Definition 3 (Evidence-carrying claim). A claim is κ=(s,ra,W,ρ,F,ϕ)\kappa= (s,r_a,W,\rho,F,\phi). It consists of an asserted status ss, an assessed revision rar_a, a witness set WW of executed checks with outcomes ρ\rho, a footprint F⊆U(ra)F \subseteq\mathcal{U}(r_a) of units the execution depended on, and a fingerprint ϕ(u)=txt⁡ra(u)\phi(u) = \operatorname{txt}_{r_a}(u) for u∈Fu \in F. The witness set need not be drawn from Ωc\Omega_c. In practice it is written from the review comment, and Ωc\Omega_c is unknown.

Definition 4 (Evidence validity). Claim κ\kappa is valid at rr if s≠UNDETERMINEDs \ne\mathrm{UNDETERMINED} and ∀u∈F:txt⁡r(u)=ϕ(u)\forall u \in F : \operatorname{txt}_r(u) = \phi(u); otherwise it is stale at rr.

Proposition 1 (Soundness of dependency-gated revalidation). Assume that (i) each check in WW is deterministic, and (ii) FF is complete for WW: every revision that agrees with rar_a on the text of all units in FF yields the same outcomes for $W. If κ\kappa is valid at rr, then re-executing WW on rr reproduces ρ\rho, and hence ss.

Proof. Validity gives agreement on FF. Completeness then gives identical outcomes, and ss is a function of the outcomes. □\square

Remark 1 (Three different levels of guarantee). (1) Mathematical. Proposition 1 states equivalence to re-running the same witnesses. It says nothing about whether passing WW implies σc(r)=FIXED\sigma_c(r) = \mathrm{FIXED}, which depends on how well WW covers Ωc\Omega_c (Corollary 1b). (2) Implemented. Completeness is an assumption. Our implementation guarantees it only for the supported dependency model of Section 4: top-level functions, module-level bindings read by executed code, and module namespaces of executed modules, all for deterministic code. Dependencies outside this model, such as I/O, environment, time, randomness or reflection, require a conservative fallback. (3) Empirical. The agreement between REVAL and exhaustive re-checking reported in Section 6 is an observation on this benchmark. It is consistent with completeness there but does not prove it.

Corollary 1 (Decomposition of false resolution). If a method asserts FIXED\mathrm{FIXED} at rtr_t while σc(rt)=PRESENT\sigma_c(r_t)=\mathrm{PRESENT}, at least one of the following holds: (a) no evidence: the assertion is not backed by executing any check; (b) witness insufficiency: WW passes at rtr_t but some check of Ωc\Omega_c fails; (c) stale evidence: the claim was produced at ra<rtr_a < r_t and was not revalidated, either because the policy stopped checking or because FF was incomplete.

Proof. Suppose the assertion is backed by evidence. If that evidence was produced at rtr_t, or is valid at rtr_t with FF complete (Proposition 1), then WW passes at rtr_t, which is case (b). Otherwise it is case (c). □\square

Metrics

Let R\mathcal{R} be the assessed (concern, revision) pairs with t≥1t \ge1 and determinate truth. The primary metric is

FRR=∣{(c,r)∈R:σc(r)=PRESENT∧σ^c(r)=FIXED}∣∣{(c,r)∈R:σc(r)=PRESENT}∣.(2)\mathrm{FRR} = \frac{\left|\{(c,r) \in\mathcal{R} : \sigma_c(r) = \mathrm{PRESENT} \land\hat{\sigma}_c(r) = \mathrm{FIXED}\}\right|}{\left|\{(c,r) \in\mathcal{R} : \sigma_c(r) = \mathrm{PRESENT}\}\right|}. \tag*{(2)}

We also report fixed-label precision P(σ=FIXED∣σ^=FIXED)P(\sigma= \mathrm{FIXED} \mid\hat{\sigma} = \mathrm{FIXED}), macro-F1F_1 over {FIXED,PRESENT}\{\mathrm{FIXED}, \mathrm{PRESENT}\}, and recurrence recall (recurring states predicted PRESENT\mathrm{PRESENT}). We report the contract-change flag rate (UNDETERMINED\mathrm{UNDETERMINED} states not asserted FIXED\mathrm{FIXED}), decision coverage (determinate states without abstention), and post-comment test-case executions per concern, which exclude the initial assessment at r0r_0. Wall-clock runtime is reported separately (Section 6.5).

Reval: Revision-Aware Evidence Maintenance

REVAL keeps one concern record per atomic concern. A record holds the intent and anchor, the witness set WW, the latest claim kk and the history of past claims. A reviewer can therefore see which revision supports the current status and why it changed. Algorithm 1 gives the online procedure.

Supported dependency model and fallbacks. The footprint is dynamic, collected while the witnesses run. Repairs that move logic into helpers therefore extend FF automatically. Three edge cases are handled explicitly. First, while a concern is UNDETERMINED\mathrm{UNDETERMINED} there is no valid evidence, so every new revision is reassessed (line 3). This covers restoration of a missing target by adding a definition, which leaves all existing units unchanged. Second, namespace units detect new bindings, for example a module-level name that shadows a builtin read by the target, and removed bindings. Third, a dependency outside the model (I/O, environment, time, randomness, reflection, dynamic import) cannot be traced. In that case the record must fall back to file-level or every-revision reassessment. In this benchmark every witness is deterministic and in-model, so no fallback was triggered. The implementation ships with regression tests for target removal, restoration by addition, builtin shadowing, helper and constant changes, and unrelated edits.

Assessor-agnostic. Here WW consists of executable checks. The same record and staleness test can also gate re-invocation of an LLM assessor, with the footprint taken from the code the assessor was shown. That variant belongs to the protocol in § 7.3.

Algorithm 1: REVAL: online assessment of concern cc over revisions

Input: history r0,…,rnr_0,\ldots,r_n (revealed one at a time); witness set WW; granularity g∈{file,func,func+globals}g \in\{\mathrm{file},\mathrm{func},\mathrm{func+globals}\}

Output: status σ^c(rt)\hat{\sigma}_c(r_t) and supporting claim for each tt

  1. κ←ASSESS⁡(r0,W,g)\kappa\leftarrow\operatorname{ASSESS}(r_0,W,g);

  2. for t←1t \leftarrow1 to nn do

  3. &nbsp;&nbsp;&nbsp;&nbsp;if κ.s=UNDETERMINED\kappa.s = \mathrm{UNDETERMINED} or ∃u∈κ.F:txt⁡rt(u)≠κ.ϕ(u)\exists u \in\kappa.F : \operatorname{txt}_{r_t}(u) \ne\kappa.\phi(u) then

  4. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;κ←ASSESS⁡(rt,W,g)\kappa\leftarrow\operatorname{ASSESS}(r_t,W,g); // no valid evidence: reassess

  5. &nbsp;&nbsp;&nbsp;&nbsp;else

  6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;keep κ\kappa; // evidence valid: decision carries over

  7. &nbsp;&nbsp;&nbsp;&nbsp;emit (κ.s,κ.ra)(\kappa.s,\kappa.ra);

  8. Function Assess(r,W,g)(r,W,g):

  9. &nbsp;&nbsp;&nbsp;&nbsp;run each check of WW on a fresh instance of rr under a call tracer; collect outcomes ρ\rho, executed functions EE (and their modules MEM_E), and names NN read by executed code;

  10. &nbsp;&nbsp;&nbsp;&nbsp;s←UNDETERMINEDs \leftarrow\mathrm{UNDETERMINED} if any check is inapplicable; else FIXED\mathrm{FIXED} if all pass; else PRESENT\mathrm{PRESENT};

  11. &nbsp;&nbsp;&nbsp;&nbsp;if g=func+globalsg = \mathrm{func+globals} then

  12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;F←E∪{module-level bindings named in N}∪{νM:M∈ME}F \leftarrow E \cup\{\text{module-level bindings named in }N\} \cup\{\nu_M : M \in M_E\};

  13. &nbsp;&nbsp;&nbsp;&nbsp;else if g=funcg = \mathrm{func} then

  14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;F←EF \leftarrow E;

  15. &nbsp;&nbsp;&nbsp;&nbsp;else F←F \leftarrow the modules MEM_E, compared by whole-file text;

  16. &nbsp;&nbsp;&nbsp;&nbsp;return (s,r,ρ,F,ϕr∣F)(s,r,\rho,F,\phi_{r\mid F});

Controlled Study Design

Benchmark construction

We constructed 12 concern families (Table 8, Section A). Each is a two-file Python program, with core.py holding the commented function and constants and util.py holding helpers. Each comes with a review comment and a hidden oracle of 6–7 behavioural checks, including positive cases that a correct fix must keep passing. The families cover input validation (3), security (path traversal, SQL injection, output escaping), exception handling (2), resource management, boundary conditions, null handling and shared mutable state. Each family provides a defective commented version, a partial fix, a complete inline fix, a complete fix delegating to a helper and a weakened helper. Three families also have a configuration constant whose regression re-opens the defect. By design, partial fixes and weakened helpers make the reviewer’s example pass while other in-scope checks fail.

FamilyCategoryReview commentOracle
quantity{sup}†`input validationparse_quantity(’-3’) returns −3-3; non-positive must raise7
meanexception handlingmean([]) raises ZeroDivisionError; return None7
paginateboundary conditionpaginate(items,1,3) skips the first page; page <1< 1 invalid7
safe_joinsecurity’../etc/passwd’ escapes the base directory7
user_querysecurityname interpolated into SQL; use a bound parameter6
greetingsecurity’<script>’ emitted raw; escape user text6
first_lineresource managementhandle leaks when readline raises OSError6
with_retry{sup}†`exception handlingall errors swallowed, returns None; surface the error6
display_namenull handlingcrashes when profile is None; fall back to name6
set_volume{sup}†`input validationset_volume(150) returns 150; clamp to [0,MAX][0,\mathrm{MAX}]7
add_tagstate managementmutable default list leaks tags across calls6
normalize_emailinput validation’bob@’ accepted; need local part and dotted domain7

Table 8. Table 8: The twelve concern families. “Oracle” gives the number of hidden checks; † marks families with a configuration constant.

Histories come from 13 scenario scripts (Table 3), each instantiated with three seeds that insert zero to two unrelated edits at random positions. This yields 441 concerns and 1284 post-comment assessments, with 2.91 revisions per concern on average. Ground truth is obtained by executing the full oracle at every revision. It contains 631 PRESENT, 561 FIXED and 92 UNDETERMINED states, including 155 recurring states. The PRESENT states comprise cosmetic edits (194), partial fixes (145), untouched (137), remote helper regressions (93), inline regressions (49) and configuration regressions (13).

IDDeveloper actions after the commentProbes
S01fixbaseline repair
S02cosmetic → fixpremature closure
S03partialwitness insufficiency
S04partial → fixdelayed true repair
S05fix → inline regressionlocal recurrence
S06fixref → helper regressionremote recurrence
S07fix → constant regression (3 families)configuration recurrence
S08cosmetic → cosmeticnever repaired
S09fix → behaviour-preserving refactorcontrol (no status change)
S10fixref → helper regression → restore helperrecurrence and re-repair
S11cosmetic → partialcompound divergence
S12fix → rename targetcontract change (UNDETERMINED)
S13fix → rename target → restore by additionre-anchoring after UNDETERMINED

Table 3. Scenario scripts (unrelated edits are inserted at random). “fixref” is a complete fix delegating to a helper; “restore” in S13 re-adds the original function name by a new definition.

Methods compared

All methods run online and see rtr_t only after deciding on rt−1r_{t-1}. The first two are controlled mechanism baselines. Each isolates one failure mechanism and is not a model of any product’s behaviour.

  • Outdated-line proxy (OUT). It predicts FIXED from the first revision at which the commented lines no longer appear verbatim, and the decision is sticky by construction. It represents using “the comment became outdated” (or a thread resolved on that basis) as a repair label. It is not the decision procedure of GitHub’s isOutdated handling or of Copilot’s auto-resolution, whose assessment method is not public [11].

  • Surface-pattern heuristic (PAT). A stateless AST heuristic: FIXED if the commented function contains more defensive constructs (conditionals, raises, try/with, sanitising or clamping calls, helper delegation) than at r0r_0. It represents judgements that reward fixes that merely look defensive. It is not an LLM judge.

  • Evidence policies. ONCE assesses at r0r_0, then only when the commented function’s text changes while the status is not yet FIXED. Once FIXED is asserted, it stops checking by design, so its zero recurrence recall follows from the policy. ALWAYS reassesses at every revision. FILE, FUNC and REVAL (FUNC+GLOBALS) reassess when their footprint is stale (Algorithm 1). ALWAYS and ONCE run without the tracer.

Witness conditions

Experiment A: oracle-sampled witnesses. With k=1k = 1, the witness is the reviewer’s example (o1o_1). With k=3k = 3, it is o1o_1 plus two checks sampled from Ωc\Omega_c without replacement, independently for each concern. Each concern is evaluated under 5 independent draws, and every oracle has at least six checks, so no budget k≤5k \le5 exhausts it; the code asserts this. This condition is a controlled ablation. It gives the method well-chosen checks from the same source as the ground truth, so it shows what maintenance achieves given strong witnesses.

Experiment B: blind comment-derived witnesses and held-out ground truth. To evaluate the approach on concerns where no oracle is available, two further authors wrote checks under blinding. Each author was a separate AI agent instance with no conversation history. Each saw only the review comment and the commented revision of core.py for every family, was instructed not to open any other file, and never saw the oracle, the variants or any future revision. The witness author wrote exactly three checks per family, as a review assistant would, with the first reproducing the reviewer’s example. The held-out author independently wrote 6–7 acceptance checks per family that define “fully fixed without breaking legitimate behaviour”. Both files were frozen (SHA-256 recorded) before any evaluation and were not edited afterwards. In Experiment B, ground truth σH\sigma^{H} is the held-out suite. The witnesses are scored against it and, for comparison, against the original oracle. Blinding was enforced by instruction and by giving each agent only the input file, not by sandboxing. Both authors are instances of the same model family as the one that assisted in preparing this study. They are therefore independent of the oracle’s content but not independent human judges (§ 7.4).

Statistical analysis and runtime

Assessments within a family share code and checks, so all confidence intervals are 95% percentile intervals from a family-clustered bootstrap [4] with 2,000 resamples of the 12 families. One matrix of resampled family indices is shared by all methods and all paired differences. Methods with identical predictions therefore receive identical intervals, and paired differences are directly comparable. Two sources of uncertainty are kept separate. Family selection is captured by the bootstrap of the draw-averaged metric. Witness sampling is captured by the range of the point estimate across the 5 draws. With twelve clusters, intervals indicate stability across concern types, not population inference.

Runtime is measured as wall-clock time per concern, median of 7 repeats on Python 3.13.16, with the k=3k=3 witnesses of draw 0. It is split into execution (compiling, loading and running checks, tracer included where active), fingerprinting (parsing units), staleness comparison and remaining policy logic.

Results

RQ1: Mechanism baselines diverge from repair in structured ways

In the synthetic histories, the sticky outdated-line proxy falsely marked 460 of 631 still-present concern states as resolved (FRR = 0.73, CI 0.66–0.78; Table 4). It disagreed with oracle status at least once in 66.7% of histories, and its fixed-label precision was 0.53. The errors follow the mechanisms (Figure 3). Every cosmetic edit, inline regression and configuration regression produced a false resolution. Errors also ran the other way: on 41 fixed states the commented line survived unchanged because the fix added lines around it. These magnitudes follow from the proxy’s construction and the scenario mix, not from measured platform behaviour. What generalises is the pattern: a line-change signal cannot separate cosmetic from behavioural edits, nor detect regressions after it fires.

MethodFRR ↓\downarrowPrecF↑\uparrowMacro-F1 ↑\uparrowRecrec↑_{\mathrm{rec}}\uparrowFlagU↑\uparrowExec.↓\downarrow
Mechanism baselines (no execution)
Outdated-line proxy0.73 [0.66, 0.78]0.530.540.05 [0.00, 0.16]0.120.00
Surface-pattern judge0.37 [0.33, 0.41]0.690.760.36 [0.29, 0.48]1.000.00
Evidence, reviewer’s example only (k=1k=1)
Verify-once0.48 [0.46, 0.49]0.650.740.00 [0.00, 0.00]0.001.11
Always recheck0.37 [0.33, 0.40]0.710.800.45 [0.33, 0.59]1.002.91
File-level footprint0.37 [0.33, 0.40]0.710.800.45 [0.33, 0.59]1.002.53
Function footprint0.37 [0.33, 0.40]0.700.800.42 [0.31, 0.56]1.001.89
REVAL (functions + globals)0.37 [0.33, 0.40]0.710.800.45 [0.33, 0.59]1.001.91
Evidence, oracle-sampled witness set (k=3k=3; mean of 5 draws)
Verify-once0.30 [0.27, 0.34]0.750.840.00 [0.00, 0.00]0.003.51
Always recheck0.11 [0.06, 0.16]0.890.940.80 [0.73, 0.88]1.008.73
File-level footprint0.11 [0.06, 0.16]0.890.940.80 [0.73, 0.88]1.007.60
Function footprint0.12 [0.07, 0.18]0.880.930.73 [0.63, 0.85]1.005.67
REVAL (functions + globals)0.11 [0.06, 0.16]0.890.940.80 [0.73, 0.88]1.005.73

Table 4. Experiment A (oracle-sampled witnesses) over 1284 post-comment assessments of 441 concerns. PrecF: fixed-label precision; Recrec_{\mathrm{rec}}: recurrence recall; FlagU: contract-change states not asserted FIXED; Exec.: post-comment test-case executions per concern. Brackets: 95% family-clustered bootstrap CIs (shared draws). k=3k=3 rows average 5 witness draws. Decision coverage on determinate states is 1.00 for all methods.

Figure 3. False-resolution rate by divergence type (columns) and method (rows), Experiment A. Each method fails on a characteristic subset, as Corollary 1 predicts.

The surface-pattern heuristic reached FRR = 0.37 and failed in complementary places. It was immune to cosmetic edits and inline regressions, which remove defensive constructs, but accepted 0.93 of partial fixes and 0.92 of remote regressions, because these look like fixes or leave the commented function unchanged.

RQ2: Maintaining evidence detects recurrence at fewer executions

ONCE reduced false resolution relative to the proxy but, by design, had zero recurrence recall. With the same k=3k=3 witnesses, REVAL reduced the false-resolution rate from 0.30 to 0.11 (paired difference -0.20, CI -0.22 to -0.18). It raised recurrence recall to 0.80 (CI 0.73–0.88). Across witness draws, its FRR ranged from 0.10 to 0.12.

REVAL’s decisions were identical to ALWAYS in all 6,420 paired assessments (100.0%), as Proposition 1 predicts when the footprint is complete. It needed 5.73 instead of 8.73 post-comment test-case executions per concern, 34% fewer (Figure 4). Unrelated edits rarely triggered reassessment (5.0% of unrelated-only revisions, all of them while a concern was UNDETERMINED), against 58.5% for file-level footprints. Contract changes were flagged UNDETERMINED by every re-checking policy. In S13, re-adding the target by a new definition was always detected and reassessed. The sticky policies carried their earlier FIXED over (FlagU column).

Figure 4. Accuracy versus post-comment test-case executions (Experiment A). Squares: baselines; circles: k=1k=1; diamonds: k=3k=3. REVAL (orange) matches ALWAYS with fewer executions. Wall-clock runtime is a different matter (§ 6.5).

RQ3: Dependency granularity and witness strength govern residual error

Granularity. Function-only footprints missed every configuration regression (Figure 3: 1.00 vs. 0.17). This lowered recurrence recall by 0.07 (CI 0.00–0.14) and agreement with ALWAYS to 99.2%. Adding globals read by executed code, together with namespace units, restored full agreement at 0.06 extra executions per concern. File-level footprints also agreed fully but reassessed on most unrelated edits.

Witness strength. With only the reviewer’s example, every evidence policy accepted all partial fixes. This is a witness-insufficiency floor (FRR ≈ 0.37) that no revalidation removes, and its size partly reflects our construction of partial fixes. Figure 5 varies kk using the same sampler and draws as Table 4, so its k=3k=3 point (0.106) equals the table entry. Under REVAL, FRR falls from 0.366 (k=1k=1) through 0.205 and 0.106 to 0.058 and 0.026 (k=4,5k=4,5), and recurrence recall rises from 0.45 to 0.97. No budget exhausts an oracle, so k=5k=5 still leaves at least one hidden check. At every kk, REVAL matched ALWAYS, with 34–34% fewer post-comment executions. This stability is an observation, not a guarantee. Extra witnesses can enlarge the dynamic footprint and change which revisions require reassessment. Here they did not, because all witnesses of a family exercise the same functions.

Figure 5. Witness-set size ablation under REVAL, oracle-sampled witnesses (same sampler as Table 4; band: range over 5 draws). Left: false-resolution rate. Right: post-comment executions for REVAL and ALWAYS. Decisions of the two policies were identical at every kk.

Figure 2. False-resolution rate in Experiment A with 95% family-clustered bootstrap CIs (shared draws). Grey: mechanism baselines; light blue: reviewer’s example only; dark blue: oracle-sampled k=3k=3 (mean of 5 draws); orange: REVAL.

Witnesses written without oracle access (Experiment B)

Table 5 reports Experiment B, separately from Experiment A. The two blind ground truths largely agree: the held-out suite and the original oracle gave the same status on 92.8% of assessments. The 92 disagreements are concentrated in three families where the comment underdetermines intent. Is mutating a caller-supplied list acceptable in add_tag? Must non-transient errors skip retries in with_retry? Does a “dotted domain” exclude a trailing dot in normalize_email? Specification ambiguity is thus a real source of disagreement even before any assessor is involved.

Against the held-out truth, REVAL with all three comment-derived witnesses reached FRR =0.13= 0.13 (CI 0.00–0.26). Its recurrence recall was 0.88, and it never rejected a held-out-fixed state (missed-repair rate 0.00). That compares with 0.36 using only the first comment-derived witness (difference -0.23, CI -0.31 to -0.14) and 0.32 for ONCE with the same witnesses (difference -0.19, CI -0.24 to -0.15). Again

REVAL matched ALWAYS on every decision (100.0%). Scored against the original oracle instead, the same method reached FRR = 0.11. The remaining false resolutions came from three families. In mean and display_name, the witness author did not test a sibling input (an empty tuple, an empty profile), which is witness insufficiency. In normalize_email, the two blind authors read the comment differently, which is specification disagreement rather than an assessment error. In this setting, then, comment-derived witnesses performed comparably to oracle-sampled ones of the same size (0.17 against the held-out truth). The confidence intervals are wide.

Runtime and overhead

Execution counts are not runtime. Table 6 separates the components. In this benchmark a single check runs in about 44 μ\mus untraced and 122 μ\mus under Python’s call tracer, a factor of 2.8. Fingerprinting dominates REVAL’s cost. As a result, REVAL was 3.3×\times slower in wall-clock time than untraced exhaustive re-checking, despite 34% fewer executions. Its overhead per concern (1.27 ms) is repaid only when one test execution costs more than about 0.42 ms. Real test suites usually exceed that by orders of magnitude, and LLM assessments exceed it far more, but we did not measure such settings. The efficiency claim of this paper is therefore limited to fewer post-comment test-case executions. Total computational savings depend on the cost of an assessment relative to tracing and fingerprinting.

PolicyExec./concern (incl. r0r_0)Execution msFingerprint msStaleness msOther msTotal ms
Always recheck (untraced)11.730.400.000.000.100.50
Verify-once (untraced)6.490.250.000.000.250.50
File-level footprint10.600.850.460.010.151.46
REVAL8.730.750.730.010.161.64

Table 6. Wall-clock runtime per concern (median of 7 repeats, k=3k=3, draw 0), including the initial assessment at r0r_0. Execution includes the tracer where active.

MethodFRR ↓\downarrowPrecF_F,↑\uparrowMacro-F1F_1 ↑\uparrowRecrec_{\mathrm{rec}},↑\uparrowFlagU_U,↑\uparrowExec.↓\downarrow
Mechanism baselines
Outdated-line proxy0.73 [0.65, 0.80]0.520.530.06 [0.00, 0.21]0.120.00
Surface-pattern judge0.35 [0.27, 0.41]0.700.790.36 [0.30, 0.47]1.000.00
Blind comment-derived witnesses, first check (k=1k=1)
Verify-once0.46 [0.36, 0.56]0.650.740.00 [0.00, 0.00]0.001.12
Always recheck0.36 [0.22, 0.49]0.710.810.52 [0.37, 0.70]1.002.91
REVAL (functions + globals)0.36 [0.22, 0.49]0.710.810.52 [0.37, 0.70]1.001.91
Blind comment-derived witnesses, all three (k=3k=3)
Verify-once0.32 [0.23, 0.43]0.730.830.00 [0.00, 0.00]0.003.51
Always recheck0.13 [0.00, 0.26]0.870.930.88 [0.72, 1.00]1.008.73
REVAL (functions + globals)0.13 [0.00, 0.26]0.870.930.88 [0.72, 1.00]1.005.73
Reference: oracle-sampled witnesses (k=3k=3)
Verify-once0.35 [0.26, 0.44]0.710.800.00 [0.00, 0.00]0.003.51
REVAL (functions + globals)0.17 [0.06, 0.29]0.830.880.82 [0.73, 0.91]1.005.73

Table 5. Experiment B: witnesses written blind from the review comment and initial code, scored against an independently, blind-written held-out suite (σH\sigma^{H}: 639 PRESENT, 553 FIXED, 128 recurring states). Columns and CIs as in Table 4. Oracle-sampled rows are re-scored against σH\sigma^{H} for reference.

Summary: support within the constructed setting

Within the synthetic histories, the mechanism baselines diverged from repair in structured ways (P1). Policies that stop after first success detected no recurrence, while maintained evidence did (P2). Dependency-gated revalidation reproduced exhaustive re-checking with fewer executions, provided the footprint covered globals and namespaces, and its residual error was governed by witness strength and, in Experiment B, by specification ambiguity (P3). These are mechanism-level results. Their prevalence and their payoff on real repositories remain to be measured.

Discussion

Implications for tool builders

Resolve with a receipt. An automated resolution should record the revision assessed, the checks or reasoning used, and what that judgement depended on. This makes “resolved” auditable, in the spirit of proof-carrying code [19].

Resolution is not terminal. A resolved concern should re-open when its evidence goes stale, as a build system invalidates a cached result [18]. The footprint must include module-level state and namespaces, not only functions.

Abstain when the subject moves, and reassess until re-anchored. Renames should yield an explicit “needs re-anchoring” state, which is reassessed at every revision until valid evidence exists again.

Spend effort on witnesses; a proposed workflow. Residual error was dominated by witness insufficiency and specification ambiguity. We therefore propose, without having evaluated it, a workflow in which an assistant drafts boundary and sibling cases from the comment and the reviewer confirms or edits them, so that ambiguities such as those in § 6.4 are resolved by the person who raised the concern. Experiment B evaluated only the first half, assistant-drafted witnesses without reviewer confirmation. The confirmation step, its cost to reviewers and its effect on accuracy are untested.

Mind the overhead. Where assessments are cheap, re-checking everything may be faster than maintaining footprints (§ 6.5). Evidence maintenance pays off where assessments are expensive, such as integration tests or LLM-based judgements, and where the receipt itself has value.

Implications for researchers

Thread state (isResolved, isOutdated) is an observed workflow variable, not a correctness label. Reported “resolution rates” of AI-generated comments [5, 7] measure tool and developer behaviour, not the repair rate of the underlying concerns. Benchmarks for review agents [31, 35] would gain from scoring concerns longitudinally and at matched decision coverage [10]. They should also report agreement between independently specified ground truths, since our two blind specifications disagreed on 92 of 1284 states.

Protocol for naturally occurring histories

We defer a natural-history study and pre-specify its protocol (Table 7).

ExperimentDesignEstablishes
E1 Proxy auditObserved thread closure and outdated status vs. adjudicated per-revision statusPrevalence of each divergence (P1)
E2 Assessor comparisonPer-revision vs. history-aware vs. evidence-maintaining LLM judges, matched budgetsWhether evidence helps beyond context (P3)
E3 Stress sequencesCosmetic, partial, complete, recurrence and control edits built from real concernsTracking of the underlying concern (P2)
E4 Executable subsetIndependent tests on 30–50 runnable casesLabel agreement with behaviour
E5 Witness workflowAssistant-drafted witnesses with and without reviewer confirmationValue and cost of confirmation

Table 7. Pre-specified protocol for the natural-history study.

Corpus. Start from AIDev [14], whose review tables cover pull requests in repositories with more than 100 stars. Reconstruct histories from the GitHub review-comment and review-thread APIs. Record PR authorship and comment authorship separately. Begin with Python and behaviourally checkable concerns. Audit 50 histories for reconstruction reliability, then target about 300 atomic concerns across at least 15 repositories.

Ground truth. Two human annotators independently label each concern’s validity and its per-revision status. Agreement is reported before adjudication [6]. A 30–50-case runnable subset receives independently written tests [35].

Assessors and controls. Compare thread state and outdated status as observed (not simulated), a per-revision LLM judge, the same judge with full history, and the judge gated by a REVAL-style record. Use at least two model families with matched information and budgets, an ablation without the record, hidden future revisions and repository hold-outs. Measure wall-clock and monetary cost alongside assessment counts.

Falsification. The position is weakened in three cases. First, if observed thread state agrees closely with verified repair (P1). Second, if recurrence after correct resolution is too rare for post-closure checking to matter (P2). Third, if evidence records do not reduce FRR over an equally informed per-revision judge at matched coverage and cost (P3).

Threats to validity

Construct validity. Each oracle encodes one reading of a comment. Experiment B shows that a second, blind reading agrees on 92.8% of states, and that the disagreements are interpretable ambiguities. The outdated-line proxy is a simplification of GitHub’s diff-position logic, and it is sticky by design. It should be read as a mechanism baseline, not as an estimate of platform or developer error.

Internal validity. We designed the code variants, so magnitudes partly reflect design choices. In particular, partial fixes pass the reviewer’s example by construction. Ground truth is always computed by execution. All methods are online and deterministic. The blind authors in Experiment B are AI agent instances, blinded by instruction rather than sandboxing, and from the same model family that assisted with the study. Their independence is therefore from the oracle’s content, not from shared model biases. A human-authored replication is part of the protocol.

External validity. Programs are small, single-language and deterministic. Many real concerns, such as readability or design, have no executable witness. Real dependencies include I/O, configuration files and non-determinism outside our dependency model. Runtime results depend on the cost of an assessment relative to tracing, and we measured only microsecond-scale checks.

Conclusion validity. With twelve clusters, bootstrap intervals are approximate. Family-selection and witness-sampling uncertainty are reported separately. Key contrasts are large relative to both, but small differences should not be over-interpreted.

Conclusion

As AI assistants begin to decide when review concerns are resolved, the field needs a definition of resolution that survives code evolution. We proposed one: a resolution is a revision-specific, evidence-carrying claim that must be revalidated when what it depends on changes.

Key findings (synthetic histories). (1) A sticky outdated-line proxy falsely marked 72.9% of still-present concern states as resolved, failing on cosmetic edits, partial fixes and all regressions. (2) A policy that stops after first success detected no recurrence. (3) REVAL reduced false resolution to 0.11 with oracle-sampled witnesses and to 0.13 with blind comment-derived witnesses scored against an independent held-out suite. It detected most recurrences and matched exhaustive re-checking on every decision, with 34% fewer post-comment test-case executions. (4) Residual error came from witness insufficiency and specification ambiguity. Function-only footprints additionally missed configuration regressions.

Limitations. The evidence is synthetic and covers behavioural concerns in small deterministic programs. Blind authors were AI agents, not humans. No LLM assessor was evaluated. With microsecond-scale checks, the approach was slower in wall-clock time than re-checking everything.

Implications. Tools should attach receipts to automated resolutions, re-open concerns when receipts go stale, abstain when the subject moves, and invest in witnesses confirmed by reviewers. Researchers should not treat thread state as a repair label, and should report agreement between independent specifications.

Future directions. Future work should (i) run the natural-history protocol, with human annotators; (ii) gate LLM assessors with evidence records and measure monetary cost; (iii) evaluate reviewer-confirmed witness drafting; and (iv) extend the dependency model to configuration files and I/O, with conservative fallbacks, inside continuous integration.

Data and Code Availability

The replication package (experiment/) contains the following:

  • the concern families and oracles;

  • the frozen blind-authored witness and held-out files, with their input file and SHA-256 hashes;

  • every generated history, with the source code of each revision, ground truth and sampled witnesses (results/histories.jsonl);

  • raw per-assessment predictions (results/rows.csv);

  • pinned dependency versions and a single command (reproduce.sh) that regenerates all tables, figures and numbers;

  • regression tests (tests.py);

  • verify.py, which checks the reported counts (441 histories, 1284 assessments, class counts), REVAL–ALWAYS agreement, sweep/table consistency and file hashes against expected.json.

Declarations

Funding: none. Competing interests: none. Use of AI tools: AI assistance was used for drafting, code development and reference checking. Separate AI agent instances authored the blind witness and held-out checks of Experiment B, as described in § 5.3. The author reviewed and takes responsibility for all content.

References

Concern Families

Blind Authoring Protocol (Experiment B)

Each author received one JSON file listing, per family, the family identifier, target function name, review comment and the commented revision of core.py. The witness author was asked to act as a review assistant and write exactly three checks per family, the first reproducing the reviewer’s example, without over-specifying incidental details such as error messages. The held-out author was asked to write 6–8 acceptance checks per family covering the full scope of the concern, including positive cases. Both were told not to open any other file and could execute their checks only against the defective initial code. Both reported their resolutions of ambiguity. For example, both accepted any placeholder style for SQL parameters and supplied several candidate keys for the unspecified “account name”. The held-out author read “dotted domain” strictly and did not require non-transient errors to skip retries. The instructions, the outputs and their hashes are part of the replication package.

References

  1. [1]Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In Proceedings of the 35th International Conference on Software Engineering (ICSE), pages 712–721. IEEE, 2013. doi: 10.1109/ICSE.2013.6606617.DOI
  2. [2]Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. Modern code reviews in open-source projects: Which problems do they fix? In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR), pages 202–211. ACM, 2014. doi: 10.1145/2597073.2597082.DOI
  3. [3]Amíangshu Bosu, Michaela Greiler, and Christian Bird. Characteristics of useful code reviews: An empirical study at Microsoft. In Proceedings of the 12th IEEE/ACM Working Conference on Mining Software Repositories (MSR), pages 146–156. IEEE, 2015. doi: 10.1109/MSR.2015.21.
  4. [4]A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3):414–427, 2008. doi: 10.1162/rest.90.3.414.DOI
  5. [5]Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. Automated code review in practice. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2025. URL https://arxiv.org/abs/2412.18531. arXiv:2412.18531.
  6. [6]Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. doi: 10.1177/001316446002000104.DOI
  7. [7]Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy, Ting Zhang, and David Lo. “go home Copilot, you’re drunk”: Understanding developer responses to agent-generated code review comments. arXiv preprint arXiv:2607.21997, 2026. URL https://arxiv.org/abs/2607.21997.
  8. [8]Jon Doyle. A truth maintenance system. Artificial Intelligence, 12(3):231–272, 1979. doi: 10.1016/0004-3702(79)90008-0.DOI
  9. [9]Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. Resolving code review comments with machine learning. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 204–215. ACM, 2024. doi: 10.1145/3639477.3639746.DOI
  10. [10]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html.arxiv.org/abs/1705.08500
  11. [11]GitHub. Copilot code review: An improved review experience. GitHub Changelog, September 2026. URL https://github.blog/changelog/2026-09-18-copilot-code-review-an-improved-review-experience. Accessed 2026-10-20.
  12. [12]Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. Practical regression test selection with dynamic file dependencies. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), pages 211–222. ACM, 2015. doi: 10.1145/2771783.2771784.DOI
  13. [13]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=VTf8yNQM66.arxiv.org/abs/2310.06770
  14. [14]Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003. Dataset: https://huggingface.co/datasets/hao-li/AIDev, 2025. URL https://arxiv.org/abs/2507.15003. Accessed 2026-10-20.
  15. [15]Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 1035–1047, 2022. doi: 10.1145/3540250.3549081.
  16. [16]Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, and Christoph Treude. CodeReviewQA: The code review comprehension assessment for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9138–9166, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.476.DOI
  17. [17]Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21 (5):2146–2189, 2016. doi: 10.1007/s10664-015-9381-9.DOI
  18. [18]Andrey Mokhov, Neil Mitchell, and Simon Peyton Jones. Build systems à la carte. Proceedings of the ACM on Programming Languages, 2(ICFP):1–29, 2018. doi: 10.1145/3236774.DOI
  19. [19]George C. Necula. Proof-carrying code. In Proceedings of the 24th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), pages 106–119. ACM, 1997. doi: 10.1145/263699.263712.DOI
  20. [20]Delano Oliveira, Reydne Santos, Benedito de Oliveira, Martin Monperrus, Fernando Castor, and Fernanda Madeira. Understanding code understandability improvements in code reviews. IEEE Transactions on Software Engineering, 51(1):14–37, 2025. doi: 10.1109/TSE.2024.3453783.DOI
  21. [21]Sebastiano Panichella and Nik Zaugg. An empirical investigation of relevant changes and automation needs in modern code review. Empirical Software Engineering, 25(6):4833–4872, 2020. doi: 10.1007/s10664-020-09870-3.DOI
  22. [22]Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J. Mooney. Deep just-in-time inconsistency detection between comments and source code. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 427–435, 2021. doi: 10.1609/aaai.v35i1.16119.DOI
  23. [23]Gregg Rothermel and Mary Jean Harrold. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(2):173–210, 1997. doi: 10.1145/248233.248262.DOI
  24. [24]Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern code review: A case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 181–190. ACM, 2018. doi: 10.1145/3183519.3183525.DOI
  25. [25]Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, and Ming Wu. Halludoge: A reference-free hallucination detection for context misalignment in code review automation. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (FSE Companion ’26), Industry Track, pages 632–643. ACM, 2026. doi: 10.1145/3803437.3805236. arXiv:2601.19072.DOI
  26. [26]Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards automating code review activities. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE), pages 163–174, 2021. doi: 10.1109/ICSE43902.2021.00027.DOI
  27. [27]Rosalia Tufano, Ozren Dabić, Antonio Mastroppoalo, Matteo Ciniselli, and Gabriele Bavota. Code review automation: Strengths and weaknesses of the state of the art. IEEE Transactions on Software Engineering, 50(2):338–353, 2024. doi: 10.1109/TSE.2023.3348172.DOI
  28. [28]Asif Kamal Turzo and Amiangshu Bosu. What makes a code review useful to OpenDev developers? An empirical investigation. Empirical Software Engineering, 29(1):6, 2024. doi: 10.1007/s10664-023-10411-x.DOI
  29. [29]Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pascal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, Daniel Tarlow, Petros Maniatis, and René Just. AI-assisted assessment of coding practices in modern code review. In Proceedings of the 1st ACM International Conference on AI-Powered Software (AIware), pages 85–93. ACM, 2024. doi: 10.1145/3664646.3665664.DOI
  30. [30]Dong Wang, Yuki Ueda, Raula Gaikovina Kula, Takashi Ishio, and Kenichi Matsumoto. Can we benchmark code review studies? A systematic mapping study of methodology, dataset, and metric. Journal of Systems and Software, 180:111009, 2021. doi: 10.1016/j.jss.2021.111009.
  31. [31]Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, and Haoli Bai. SWE-Review: Closing the loop on issue resolution with agentic code review. arXiv preprint arXiv:2607.06065, 2026. URL https://arxiv.org/abs/2607.06065.
  32. [32]Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In Proceedings of the 27th IEEE/ACM International Conference on Program Comprehension (ICPC), pages 53–64, 2019. doi: 10.1109/ICPC.2019.00019.DOI
  33. [33]Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. EvaCRC: Evaluating code review comments. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 275–287, 2023. doi: 10.1145/3611643.3616245.DOI
  34. [34]Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. A roadmap for modern code review: Challenges and opportunities. ACM Transactions on Software Engineering and Methodology, 35(9):1–42, 2026. doi: 10.1145/3800963. Preprint: arXiv:2405.18216.DOI
  35. [35]Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. Code review agent benchmark. arXiv preprint arXiv:2603.23448, 2026. URL https://arxiv.org/abs/2603.23448.
  36. [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, pages 46595–46623, 2023.arxiv.org/abs/2306.05685

Paper details

vlnc:W_6xhjwz7u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI assistance, AI agent instances

View statement
Looking for related papers…
Contents