betaFeedback

WorksComputer Science

Twice-Told Critiques: Claim-Level Homogenization of Peer Review in the LLM Era

Abstract

Peer review relies on several readers finding different problems. We measure critique overlap in conference and journal reviews. At ICLR, measured overlap rose from 12.0% in 2026 to 21.3% in 2027, in reviews from the same stage of the review process, with year-on-year rises at NeurIPS, ICML and the F1000 and PLOS journals. A third ICLR review of a paper added 5.46 new problems to the other two in 2027, against 6.12 in 2026. We apply MELD, a detector of LLM-generated text, to all reviews. Within a paper, two reviews that MELD flags overlap more than two unflagged reviews: by 3.0 points at ICLR in 2024-2025, by 4.5 points in 2026-2027, and by 5.4 points across five journal sources. These results provide evidence that LLM-flagged reviews of one paper are less distinct from each other than unflagged reviews, and suggest that the diversity of critiques among reviews of a single paper is falling.

Introduction

Peer review needs readers who find different problems. In one study, two reviewers usually shared no critical point [6]. In another study about Nature-family reviews, 25.8% of one reviewer’s criticisms match another’s [17]. Our central question is: with widespread LLM-generated text in ML reviews [20, 24, 5], do reviewers still provide distinct critiques?

We measure shared problems by extracting and matching critiques across reviews of one paper. Whole-review similarity mixes wording and content [20, 2] and cannot establish that two reviews raise the same problem. Prior studies find that synthetic LLM-generated reviews are more similar to each other than human reviews [11] and share more criticisms [17], a gap used

for detection [4]. We apply critique extraction [21, 17] to reviews that venues received, comparing problems within papers and over time.

Our findings are:

  • Critique overlap rose at ICLR from 12.0% in 2026 to 21.3% in 2027, at ICML in 2026 and at NeurIPS in 2025 (Section 4, Figure 1).

  • Within a paper, two LLM-flagged reviews overlap more than two unflagged reviews at ICLR, by 3.0 points in 2024–2025 and 4.5 points in 2026–2027 (Section 4, Figure 2).

  • Pooled over five journal sources, the same within-paper gap is 5.4 points (Section 4, Figure 4).

Related Work

LLM use in peer review. ICLR and NeurIPS reviews grew longer after ChatGPT, while LLM-assisted ICLR reviews mention originality less (Wu et al., 2026). LLM reviewers draw more than half of their weaknesses from the appendix. In sampled ICLR reviews, the share of such weaknesses rose from 11.3% in 2026 to 27.6% in 2027 [8]. LLM-generated reviews at one management journal cover a narrower range of topics [7]. At ICML 2026, a randomized trial assigned papers and reviewers to a policy that prohibited LLM use or to one that allowed limited use. The second policy made reviews 5.5–7% longer and left scores and decisions almost unchanged [18]. We compare shared problems, not single-review topics.

Homogenization. LLM assistance makes the texts and ideas of different users more alike [23, 1, 3]. Model outputs are narrower than training data (Wu et al., 2025). Sourati et al. [27] review this evidence. We extend the question from creative tasks to expert critiques, where independent readings are the goal.

Methodology

Data

Table 1 lists the samples. ICLR and NeurIPS come from OpenReview (ulab-ai, 2025; Xu et al., 2025; Heineman, 2025). ICML includes accepted papers from 2025–2026. ICLR 2027 reviews come from the OpenReview API eight days after they became public. Conference papers need at least three reviews. Journal papers need at least two first-round reviews: SciPost, F1000Research and sister platforms (F1000), PLOS, BMC Medicine [16], and Copernicus/EGU, covering physics, medicine, biology and geoscience. A panel is the set of reviews of one paper.

SourceYearsPapersReviews
ICLR2020–2785,466299,168
NeurIPS2021–2518,74674,866
ICML2025–269,58836,804
SciPost2019–262,9626,613
F10002019–267,11116,983
PLOS2019–2614,36034,084
BMC2019–252,5106,414
Copernicus2019–2616,00034,877

Table 1. Papers and reviews in the analysis.

Claim extraction

A critique claim identifies one problem that needs one fix. We exclude praise and neutral text. In pass 1, GPT-6 Luna (Luna) reads one review in full and lists its claims. Each claim has a quote copied from the review, a one-sentence paraphrase and a category such as clarity or experiment scope.

We check pass 1 against human labels and against other LLMs. For the human check, we use

two public sets: 218 Nature-family reviews that the authors of the set split into 1,930 critique items [17], and 60 NeurIPS reviews with 168 weakness spans that experts marked [22]. A quote of at least one Luna claim falls in 96.7% of the items and in 94.2% of the spans, and 0.7% of Luna’s claims on the first set fall outside every item. For the LLM check, Sol 6.1, DeepSeek V4.1 Flash and Kimi K3.1 Pro run pass 1 on 2106 of our reviews, in 510 validation panels from every venue, year and detector stratum. We call these three models the council. Luna’s quotes reach a word-span F1 of 78.5% with a council member, against 80.5% between two members.

Claim grouping

In pass 2, a separate Luna call groups the claims of one panel into problems. The call sees the claim lists, not the review texts. Two claims join one group only when they state the same problem in the same paper element.

To check pass 2, each council member groups Luna’s claim lists with the same prompt. Against the majority vote of the three members on each claim pair, Luna’s grouping has 83.6% precision and 78.4% recall across 507 panels. With a council member or the majority as grouper, the within-paper flagged-pair gap of RQ2 is 7.0 to 7.3 points, against 7.4 with Luna (Table 2). Each council member also runs both passes on its own claims, and all four pipelines give every contrast the same sign (Table 3).

GrouperOne flaggedTwo flagged
GPT-6 Luna+2.9 [0.2, 5.6]+7.4 [3.6, 11.3]
Sol 6.1+2.1 [−0.7, 4.9]+7.1 [3.3, 10.9]
DeepSeek V4.1 Flash+1.8 [−1.8, 5.4]+7.3 [2.7, 11.8]
Kimi K3.1 Pro+2.7 [−0.8, 6.1]+7.2 [2.7, 11.7]
Council majority+2.1 [−1.3, 5.5]+7.0 [2.7, 11.2]

Table 2. Every grouper gives about the same within-paper gaps. Each grouper groups Luna’s claims on 405 validation panels from 2024 on. Gap: overlap of pairs with one or two flagged reviews minus unflagged pairs of the same paper, with 95% intervals.

PipelineSame paperCross paperFlagged gapICLR 2027 − 2026
GPT-6 Luna11.10.4+7.4+7.0
Sol 6.110.90.5+6.1+9.0
DeepSeek V4.1 Flash14.80.5+4.2+7.9
Kimi K3.1 Pro14.71.0+10.0+9.6

Table 3. All four pipelines give every contrast the same sign; the overlap level depends on the model. Each model extracts and groups its own claims on 510 same-paper and 90 cross-paper validation panels. Flagged gap: the two-flagged gap of Table 2.

Metric

Let GAG_A and GBG_B be the sets of distinct problems that reviews AA and BB raise. Their critique overlap is the overlap coefficient [26]

o(A,B)=∣GA∩GB∣min⁡(∣GA∣,∣GB∣),(1)o(A, B) = \frac{|G_A \cap G_B|}{\min(|G_A|, |G_B|)}, \tag*{(1)}

and o(A,B)=0o(A, B) = 0 for an empty set. Full overlap means containment of the smaller set, not identical reviews. Higher overlap can result from smaller critique sets rather than more shared problems. Every main claim keeps its sign under other measures, including shared-problem counts at fixed set sizes. Following the IntraSim measure of Baumann et al. [2], we average oo over all review pairs within a paper, then over papers within a year.

Detector of LLM-generated texts

MELD ([19]) scores reviews for LLM-generated or edited text. LLM-flagged means above one common threshold: the 99.9th percentile of 110,080 pre-2023 scores. MELD flags at most 0.4% of each source’s pre-2023 reviews and 100.0% of 5,000 generated ICLR reviews (Yu et al., 2026). MELD flags 39.5% of Luna-polished human reviews.

Results

RQ1: Do reviewers of one paper repeat more of each other’s critiques? ICLR overlap rose from 12.0% in 2026 to 21.3% in 2027, by +9.3 points. These two years are directly comparable: both hold reviews during the discussion phase. NeurIPS and ICML also rose in their latest cycles (Figure 1). Table 4 counts claims in all three-review subsets

Figure 1. Measured critique overlap rose at ICLR in 2027, at ICML in 2026 and at NeurIPS in 2025. Journal overlap rose most at F1000 and PLOS. We define overlap as shared problems divided by the smaller critique set, averaged within papers, then over papers. Dashed segment: from 2026 on, ICLR samples hold reviews during the discussion phase. The 2026 and 2027 points include papers withdrawn later.

of each ICLR paper. From 2026 to 2027, claims per review rose from 7.2 to 7.3, but distinct claims per three-review panel fell from 19.9 to 19.0, and the new claims that a third review added fell from 6.12 to 5.46.

RQ2: Do LLM-flagged reviews share a larger fraction of their critiques? For each paper with both pair types, we subtract the mean overlap of its pairs of two unflagged reviews from the mean overlap of its pairs of two flagged reviews. The flagged-pair gap is the mean of this difference over papers. At ICLR, the flagged-pair gap is 3.0 points in 2024–2025 and 4.5 points in 2026–2027. We do not pool them: their reviews come from different review stages. The gap is also positive at NeurIPS and at ICML (Figure 2).

Figure 2. In both ICLR periods, at NeurIPS and at ICML, two LLM-flagged reviews of one paper overlap more than two unflagged reviews. A pair type counts the flagged reviews of a pair. (a) Mean over papers of pair overlap within each paper. (b) Mean over papers of the difference from the unflagged pairs of the same paper. Bars: 95% intervals.

If the overlap of every ICLR pair had risen from 2026 to 2027 as much as the overlap of the two unflagged reviews, from 10.8% to 14.5%, the total would have risen by 3.7 points, not 9.3. The other 5.6 points go with flagged reviews. This split is an accounting, not an effect of LLM use.

The gap could come from wording and not from content, if two reviews in one LLM style are easier to match. To test this style effect, Luna changes two human reviews in each of 600 pre-2023 ICLR panels and keeps their points. Polishing the language changes overlap by +0.5 points, and rewriting each review from an outline of its points by +0.9 points, against ICLR flagged-pair gaps of 3.0 and 4.5 points. Style effects can therefore explain only part of the flagged-pair gap.

YearPer reviewDistinctNew in a third reviewShared per pair
20207.0219.736.160.48
20217.6521.626.790.48
20227.5721.326.680.49
20237.0619.796.170.49
20246.8719.446.110.42
20257.2020.346.380.45
2026†7.2019.926.120.61
2027†7.3118.985.461.10

Table 4. ICLR three-review panels hold fewer distinct claims in 2027 than in any year from 2020, although claims per review did not fall from 2026. Distinct: claims of the three reviews, with a repeated claim counted once. †Reviews during the discussion phase; earlier years hold final reviews.

The gap could also come from the detector and not from LLM use, if MELD gives high scores to reviewers who raise common critiques with or without an LLM. Such reviewers also wrote before ChatGPT. In a pre-ChatGPT check, we flag the top 45% of MELD scores in ICLR 2020–2023, a share close to the 2026 flag rate, and repeat the comparison. The gap is −3.3 points in 2020, −1.2 in 2021–2022 and 0.2 in 2023 (Figure 3). A high detector score alone does not produce the gap. GigaCheck (Tolstykh et al., [28]), which flags fewer reviews, gives gaps of 2.2 and 5.4 points at ICLR, 5.1 at NeurIPS and 5.5 at ICML.

Figure 3. The within-paper flagged-pair gap is positive in both ICLR periods, at NeurIPS and at ICML, but not before ChatGPT, and shrinks at lower thresholds in ICLR 2024–2025. FE: paper fixed effects, without and with length and claim-count controls. 95% intervals cluster by paper. Dotted lines: main estimates. Before ChatGPT, we flag the top 45% of MELD scores.

RQ3: Do LLM-flagged journal reviews also repeat more critiques? From the 2019–2023 mean to 2026, journal overlap rose most at F1000 (+5.8 points) and PLOS (+2.5), and at both sources it rose in each year from 2024 (Figure 1). Within a paper, two flagged reviews overlap 5.4 points more than two unflagged reviews. Few journal papers have four reviews, so few have both pair types. For the journals, we therefore estimate the gap with a regression with paper fixed effects, which also uses papers with three reviews and mixed flags. The gap is positive at F1000, PLOS and Copernicus, and the intervals of BMC Medicine and SciPost include zero (Figure 4).

Figure 4. The journal flagged-pair gap is positive pooled and at F1000, PLOS and Copernicus. Two-flagged minus unflagged overlap from a regression with paper fixed effects, from 2024 on, with 95% intervals. Parentheses: papers with more than one pair type. Dark blue line: ICLR 2026–2027, also first-round reviews.

Conclusion

Our results suggest that the reviews of one paper have lost part of their independence, in content and not only in wording. Reviews that the detector flags are one part of this change and do not explain all of it, because overlap also rose between reviews that the detector does not flag. Two questions stay open: how much of that rise comes from LLM use that leaves no trace in the text, and whether the 2027 level holds after the discussion phase.

Limitations

LLM measurement. An LLM extracts and groups claims. The council reference uses LLMs, not humans, and does not rule out shared model errors. The human check covers pass 1 only. Extraction and grouping errors can affect measured levels and changes over time. The overlap level depends on the grouper: council members give 10.9 to 14.8% against Luna’s 11.1%. Luna misses more links in papers with no flagged review. LLM groupers agree that this error changes comparisons between papers but not within-paper gaps.

Review snapshots. ICLR samples hold final reviews before 2026 and reviews during the discussion phase from 2026 on, so the ICLR series compares reviews of one stage only from 2026 to 2027. For the same reason, we report ICLR flagged-pair gaps for 2024–2025 and 2026–2027 apart.

Detector labels. MELD trained on peer reviews from sources it does not name. If these include our pre-2023 reviews, the threshold is too low and later flag rates too high. GigaCheck flags fewer recent reviews than MELD, so flag rates depend on the detector. MELD flags 39.5% of our polished reviews, so a flag does not separate polishing from generation. Other tools and prompts can polish differently, and detectors label part of LLM-polished reviews as AI-generated [25]. A flag does not tell which model wrote a review. Critique content differs between model versions and families [9], so the flagged-pair gap can change with the mix of models that reviewers use. The rewrite control uses two models, two prompts and pre-2023 reviews. Within-paper comparisons remove paper-level differences, not reviewer-level ones, and do not assign a causal effect to LLM use. An unflagged review is not thereby human-written. Detectors can err across language backgrounds, fields and templates. Their labels must not support action against an individual reviewer.

Selection and scope. NeurIPS and most journals publish accepted-paper reviews only, and our ICML sample has accepted papers only. The journal gap rests on 2,228 papers with three or more reviews and mixed flags, and its regression assumes that each pair type shifts overlap by the same amount in every paper. All texts are in English, and most come from ML.

Use of LLM Assistants

The authors created the idea and the outlines of the experiments, and acted as supervisors and editors. Claude Fable 5.2 inside Claude Code and GPT-6 Astra inside Pi wrote all code and all text of this paper. The authors take full responsibility for the content.

Data

Sampling. OpenReview and journal collection took place on 27–29 October 2026, with more journal papers on 6 November 2026. ICLR 2027 reviews come from 15 November 2026, eight days after they became public and before decisions. ICLR 2026 reviews come from a snapshot of 21 November 2025 [10], ten days after the reviews became public. After that date, the organizers reset all reviews to their state at the start of the discussion phase (ICLR 2026 Program Chairs, 2025), so the public record holds no final version of these reviews. The journal year 2026 covers January–October. BMC Medicine ends in 2025. We sampled all ICLR (2020–2027) and NeurIPS (2021–2025) papers with at least three reviews, all ICML accepted papers in 2025 and 2026, and up to 2,000 papers per journal source-year with at least two first-round reviews. SciPost reviews come from its API. We obtain F1000Research, Wellcome Open Research, Open Research Europe and PLOS reviews from Europe PMC, BMC Medicine reviews from JerMa88 [16], and Copernicus/EGU reviews from journal discussion pages. We retain first-round referee reviews, exclude author replies and editor notes, and cap papers at five reviews. Withdrawn and desk-rejected papers do not enter the main sample before 2026. The ICLR samples of 2026 and 2027 keep them, so that both years hold all papers with three reviews. Table 5 gives each conference source-year and each journal source in two periods.

SourceYearPapersReviewsClaims per reviewOverlap (%)MELDGigaCheck
ICLR20202,1276,5437.110.5±0.310.5 \pm0.30.00.0
20212,59410,0197.79.2±0.29.2 \pm0.20.00.1
20222,61710,1887.610.0±0.210.0 \pm0.20.00.3
20233,78914,2717.110.8±0.210.8 \pm0.20.10.1
20245,69021,9346.98.9±0.18.9 \pm0.16.15.8
20258,62934,6897.29.1±0.19.1 \pm0.117.815.9
202619,46175,6747.212.0±0.112.0 \pm0.144.616.5
202740,559125,8507.421.3±0.121.3 \pm0.162.716.8
NeurIPS20212,76810,6967.38.3±0.28.3 \pm0.20.00.1
20222,82010,3137.08.1±0.28.1 \pm0.20.00.1
20233,39414,9376.88.2±0.18.2 \pm0.13.72.3
20244,23516,5966.68.0±0.18.0 \pm0.111.45.4
20255,52922,3246.79.1±0.19.1 \pm0.118.69.4
ICML20253,25612,4497.08.4±0.18.4 \pm0.135.516.6
20266,33224,3556.511.6±0.111.6 \pm0.151.530.2
SciPost2019–231,5863,5387.310.3±0.510.3 \pm0.51.00.4
2024–261,3763,0757.810.9±0.510.9 \pm0.511.53.1
F10002019–233,8648,9347.410.7±0.310.7 \pm0.32.62.5
2024–263,2478,0496.612.6±0.412.6 \pm0.429.322.5
PLOS2019–239,25921,68410.08.8±0.28.8 \pm0.21.61.0
2024–265,10112,40010.09.6±0.29.6 \pm0.231.422.9
BMC-Med2019–231,8024,61010.47.8±0.37.8 \pm0.31.20.8
2024–257081,8049.97.6±0.47.6 \pm0.416.912.9
Copernicus2019–2310,00021,87220.211.7±0.111.7 \pm0.10.70.4
2024–266,00013,00519.912.3±0.212.3 \pm0.221.36.9

Table 5. Each conference year and each journal source in two periods. Overlap is the mean over papers with its SE. Flagged: share of reviews that each detector flags at its own threshold (Appendix § 11).

Source differences. ICML changed its form between 2025 and 2026 and also changed its LLM policy (ICML 2025 Program Chairs, 2025; ICML 2026 Program Chairs, 2026a,b). ICML 2026 assigned each reviewer one of two policies [18], but public records do not reveal the policy attached to each review. Open journals can also expose later reviews to earlier reviews. These differences limit comparisons of levels across venues.

ICLR review forms. The 2023 ICLR form raised overlap in that year. ICLR changed the free-text fields of its form in 2022, 2023, 2024 and 2027 (Table 6). Only the 2023 form had a separate field on clarity, quality, novelty and reproducibility, which asks each reviewer about the same topics. In 2023, 50% of the shared problems of a paper include a claim on clarity, novelty or reproducibility, against 42% in 2022 (last column of Table 6).

YearText fieldsPapersWhole reviewsField deletedCategories droppedShared problems in categories (%)
2020review2,12710.5–9.042.8
2021review2,5949.2–7.942.7
2022summary; main review; review summary2,61710.0–8.742.0
2023summary; strengths and weaknesses; clarity, quality, novelty and reproducibility; review summary3,78910.89.78.449.7
2024summary; strengths; weaknesses; questions5,6908.98.78.239.0
2025summary; strengths; weaknesses; questions8,6299.1–8.735.5
2026summary; strengths; weaknesses; questions19,46112.0–11.928.1
2027summary; critical strengths; critical weaknesses; questions40,55921.3–21.921.7

Table 6. Only the 2023 ICLR form had a separate field on clarity, quality, novelty and reproducibility (bold). Field deleted: we delete the clarity field in 2023 and, as a control, the questions field in 2024, and run both passes again. Categories dropped: we remove the claims on clarity, novelty and reproducibility from the existing groups. Shared problems in categories: share of the shared problems of a paper that include at least one claim on clarity, novelty or reproducibility. All values are means over papers.

Two tests measure the effect of this field. First, we delete the clarity field from each 2023 review and run both passes again. Overlap changes by −1.1 points. Deleting any field can change overlap, so as a control we delete the questions field from each 2024 review. Overlap then changes by only −0.2 points. Second, we keep each review whole and remove from the existing groups each claim that Luna labels as clarity, novelty or reproducibility. Overlap is then 8.7% in 2022, 8.4% in 2023 and 8.2% in 2024, so 2023 no longer stands above the years next to it.

Models and release. Both passes use GPT-6 Luna through OpenRouter with strict JSON-schema output and high reasoning effort. Journal prompts use the generic rather than the ML paper type. Each call gets four attempts, and a schema violation counts as a failure. No review failed all four attempts. Saved requests record API model names and settings. We release the code,1 all collected reviews,2 paper metadata, extracted claims, problem groups and per-review MELD scores. Journal reviews retain their CC BY or CC0 terms. All texts were public before collection.

Extraction and Grouping Checks

Human labels. Luna finds a human item (§ 3.2) when the quote of one of its claims shares five consecutive words with the item, or all words of the shorter text. In the Nature-family set, 30.0% of the items receive more than one claim, and Luna lists 3.0 more claims per review than the set lists items. CLAIMCHECK marks only weaknesses tied to a paper claim, so we give only recall for that set. No public set carries human labels of same-problem groups across reviews, so pass 2 has no human check.

Council sample. We fixed the prompts and then drew 510 same-paper and 90 cross-paper panels, stratified by source (ICLR 2018–2027, NeurIPS 2024–2025, ICML 2025–2026 and five journals 2024–2026). The ICLR panels of 2018 and 2019 are outside the analysis years and serve only this check. From 2024 on, we also stratified by detector flags: panels with no flagged review and panels with at least two flagged and one unflagged review. Council models run through OpenRouter at medium reasoning effort. For one review, the span F1 between two extractors is the F1 between the sets of word positions that their quotes cover. This score gives partial credit to partial overlaps and does not depend on how finely either side splits claims (Table 7). The sample is stratified, so values on it are not estimates for the corpus.

PanelsDetector stratumnnClaims LunaClaims councilSpan F1 LunaSpan F1 council
SameAll5107.67.478.580.5
Sameplain3117.77.578.280.4
Sameflagged1997.47.379.080.7
CrossAll908.48.378.781.0

Table 7. Luna’s quotes agree with those of a council member almost as closely as the quotes of two members agree, in each detector stratum. Claims are per review. Span F1 compares the quoted words of Luna and one member, or of two members.

Claim grouping. The label of a claim pair is the majority vote of the three members on whether the two claims belong to one group. We keep a pair only when the outputs of all three members contain both claim ids exactly once. This rule keeps 141,426 of 141,700 cross-review claim pairs (99.8%). Luna agrees with 1,148 majority links, misses 316 and adds 226 (Table 8). This check covers 507 of the 510 same-paper panels, because the other 3 have claims in at most one review and give no claim pair. Unlabeled pairs do not define complete reference groups, so this check does not validate full-panel overlap levels.

PanelsDetector stratumnnPairsPrecisionRecallF1Links per 100
SameAll507141,42683.678.480.994
Sameplain30881,46583.772.577.787
Sameflagged19959,96183.484.283.8101

Table 8. Luna’s grouping agrees with the majority vote of the council on same-paper panels, but misses more links in panels with no flagged review. Pairs: claim pairs from different reviews of one panel with a vote from all three members. Precision, recall and F1 take the majority vote as truth. Links per 100: Luna’s links per 100 majority links.

Grouping error by stratum. Per 100 majority links, Luna finds 87 links in panels with no flagged review and 101 in the flagged stratum (Table 8). This difference can change comparisons between papers. Within the panels of the flagged stratum, Luna finds 100 links per 100 majority links between two unflagged reviews and 101 between two flagged reviews, so we find no difference in this error between the pair types of one paper. This check rests on 70 majority links between two unflagged reviews. With a council member or the majority as grouper, the overlap difference between flagged and plain panels falls from 5.9 points to 3.9–5.1 points, and the within-paper gap changes little (Table 2).

Full pipelines. In a full pipeline, one model extracts and groups its own claims (Table 3). On one panel, the overlap of Luna’s pipeline differs from that of a member by 5.5 points on average, and the overlaps of two members differ by 5.5 points. The 95% interval of the within-paper flagged-pair gap excludes zero in 3 of four pipelines, and that of the ICLR step from 2026 to 2027 in 4 of four. In corpus panels that hold one review from each of four ICLR papers of one year, overlap is 0.2% in 2026 and 0.1% in 2027, so the grouper shows no drift toward merging unrelated claims; this check cannot detect drift in same-paper claims.

Gap Estimators and Overlap Measures

Flagged-pair gap. We compare within papers, because a plain difference between all two-flagged pairs and all unflagged pairs mixes the flags with differences between papers. For a paper pp, let oˉptwo\bar{o}_p^{\mathrm{two}} and oˉpnone\bar{o}_p^{\mathrm{none}} be the mean overlap of its pairs with two flagged reviews and with no flagged review. The flagged-pair gap is

Δ^=1∣P∣∑p∈P(oˉptwo−oˉpnone),(2)\widehat{\Delta} = \frac{1}{|P|} \sum_{p\in P} \left(\bar{o}_p^{\mathrm{two}} - \bar{o}_p^{\mathrm{none}}\right), \tag*{(2)}

where PP is the set of papers with both pair types. Such a paper has at least two flagged and two unflagged reviews.

Fixed-effects estimate. We also fit

opk=αp+β1one⁡pk+β2two⁡pk+εpk,(3)o_{pk} = \alpha_p + \beta_1 \operatorname{one}_{pk} + \beta_2 \operatorname{two}_{pk} + \varepsilon_{pk}, \tag*{(3)}

where opko_{pk} is the overlap of pair kk of paper pp, αp\alpha_p is a paper fixed effect, and one⁡pk\operatorname{one}_{pk} and two⁡pk\operatorname{two}_{pk} mark a pair with one and with two flagged reviews. The estimate is β^2\hat{\beta}_2. This fit also uses the three-review papers with mixed flags, so the main text uses it for the journals, where few papers have four reviews. For the pooled journals, this fit gives 5.4 points from 2,228 papers with mixed flags, and Equation (2) gives 5.0 points from 230 papers. At ICLR, this fit gives 2.7 and 7.7 points for the two periods without controls and 3.3 and 7.2 with log length and log claim-count controls, against 3.0 and 4.5 from Equation (2). It also uses papers without a two-flagged or an unflagged pair and weighs pairs, not papers, equally. Its SEs cluster by paper.

Overlap measures. Any ratio of shared problems to set size depends on set size, and the raw shared count grows with review length. Table 9 therefore repeats each main claim with seven measures and holds set size fixed in two of them.

ClaimOverlap coef. pointsJaccard pointsDice pointsShared per pairShared, linear sizes per pairShared, same sizes per pairNew problems per review
Change over time
ICLR, 2026 to 2027+9.3+3.9+6.2+0.49+0.49+0.49−0.66
ICLR, 2025 to 2026+2.9+1.3+2.3+0.16+0.15+0.15−0.26
NeurIPS, 2024 to 2025+1.1+0.6+1.0+0.07+0.05+0.05−0.00†
ICML, 2025 to 2026+3.2+1.6+2.7+0.15+0.16+0.16−0.66
F1000, 2019–23 to 2026+5.8+3.1+4.1+0.47+0.54+0.48−0.78
PLOS, 2019–23 to 2026+2.5+1.1+1.9+0.28+0.26+0.27−0.18†
Within-paper gap against unflagged pairs or reviews
ICLR 2024–25, two flagged+2.6+1.4+2.1+0.03†+0.13+0.12−0.04
ICLR 2026–27, two flagged+7.7+3.4+5.4+0.44+0.41+0.42−0.29
NeurIPS, two flagged+4.4+2.1+3.4+0.15+0.21+0.21−0.13
ICML, two flagged+4.5+2.0+3.4+0.21+0.22+0.22−0.18
Journals, two flagged+5.4+2.2+3.4+0.32+0.35+0.34−0.27

Table 9. Every main claim keeps its sign under each pair measure. The claims predict a positive sign for pair measures and a negative sign for new problems. † marks an estimate that does not differ from zero in that direction at the 95% level. A change over time is the last year minus the base year, or minus the 2019–2023 mean for journals. A within-paper gap compares two flagged with two unflagged reviews of one paper, in a regression with paper fixed effects.

  • Overlap coef.: 100 o100\ o, the main measure.

  • Jaccard: 100∣GA∩GB∣/∣GA∪GB∣100\lvert G_A \cap G_B\rvert/\lvert G_A \cup G_B\rvert.

  • Dice: 200∣GA∩GB∣/(∣GA∣+∣GB∣)200\lvert G_A \cap G_B\rvert/(\lvert G_A\rvert+ \lvert G_B\rvert).

  • Shared: the count ∣GA∩GB∣\lvert G_A \cap G_B\rvert.

  • Shared, linear sizes: the shared count with the smaller and the larger set size as controls.

  • Shared, same sizes: the shared count with one indicator per pair of set sizes, so a four-and-ten pair is compared only with other four-and-ten pairs.

  • New problems: for changes over time, the problems that a third review adds to two others. For within-paper gaps, the problems of one review that no other review of the paper raises. This measure counts novelty, so the claims predict a negative sign.

Results. Every claim keeps its sign under each pair measure. Each 95% interval excludes zero except one: the raw shared count of two flagged ICLR reviews in 2024–2025. In that period, two flagged reviews raise fewer problems: the smaller set of such a pair differs by −0.82-0.82 problems from that of an unflagged pair of the same paper. Shared counts at fixed set sizes rise for two flagged reviews at each ML venue and in pooled journals, so the flagged-pair gaps are not only a result of smaller critique sets. New problems fall wherever overlap rises, except at NeurIPS from 2024 to 2025 and at PLOS, where the change does not differ from zero.

LLM Detector

MELD. We run MELD [19] from Hugging Face.3 MELD scores each token of the text, in windows of 2,048 tokens, up to its own length limit. A text’s score is the mean of its top 25% of token scores, as in MELD’s own scorer.

GigaCheck. As a second detector, we run the binary GigaCheck classifier [28] on Mistral-7B, trained on the LLMTrace corpus [29], from Hugging Face with the input code of the model.4 A text’s score is the logit of the AI label minus the logit of the human label on its first 1,024 tokens.

Threshold. For each detector, the threshold is the 99.9th percentile of its scores on the same 110,080 pre-2023 reviews: 2.75 for MELD and 8.01 for GigaCheck. 94,706 of these reviews enter the analysis. The other 15,374 come from years before the analysis (ICLR 2017–2019, journals 2018) or from no analysis panel, such as journal reviews above the cap of five reviews per paper. No source’s pre-2023 MELD flag rate exceeds 0.4%. GigaCheck flags 0.08% of the ICLR 2023 reviews.

Known texts. MELD flags almost no human reviews and almost all texts that an LLM wrote, but only 39.5% of polished human reviews (Table 10). GigaCheck flags fewer of the polished and LLM-written texts. These tests do not establish label precision on naturally submitted reviews.

TextsnnMELD flagged (%)GigaCheck flagged (%)
Human, pre-2023110,0800.10.1
Human, ICLR 202314,2710.10.1
Polished2,10039.57.7
From outline13,19499.292.1
Panels: outline, Luna1,60099.888.6
Panels: polished, Luna1,60038.813.1
Panels: outline, Kimi1,200100.087.0
Panels: polished, Kimi1,20029.310.5
From paper, five LLMs5,000100.065.6
From paper, Luna100100.099.0
From paper, Kimi100100.099.0

Table 10. MELD flags almost no pre-2023 human reviews and all reviews that an LLM wrote from the paper. Shares of each set of texts of known origin. The ICLR 2023 reviews were due before ChatGPT’s release. Polished: Luna polished 2,100 human reviews of 2018–2021 and kept every point. From outline: Luna rewrote 13,194 journal reviews of 2018–2021 from an outline of their points. The panel rows are the texts of the rewrite control (RQ4). The last three rows are reviews that an LLM wrote from the paper alone; the five LLMs are those of Yu et al. (2026).

Flag rates. From 2024 on, GigaCheck flags fewer reviews than MELD in each source (Table 5). From 2025 to 2027, the GigaCheck rate at ICLR changes little, and the MELD rate keeps rising. On 415,248 reviews from 2023 on, both detectors flag 12.2%, only MELD flags 24.9% and only GigaCheck flags 2.1% (Cohen’s κ=0.34\kappa= 0.34). ICML 2025 banned LLM use, yet MELD flags 35.5% of its reviews; under the ICML 2026 ban, 22.5% of surveyed reviewers reported LLM use [18].

Flagged-pair gap. With GigaCheck flags, the flagged-pair gap of Equation (2) is positive in all five sets of reviews, and its 95% interval excludes zero in the four conference sets (Table 11). At lower MELD thresholds, the gap holds in ICLR 2026–2027 and shrinks in 2024–2025, where the 95% interval at the 99th percentile includes zero (Figure 3). All other results use MELD flags at the 99.9th percentile.

ReviewsMELD Gap (points)MELD PapersMELD PairsGigaCheck Gap (points)GigaCheck PapersGigaCheck Pairs
ICLR 2024–25+3.0 [+1.7, +4.4]1,1613,016+2.2 [+0.9, +3.6]9662,558
ICLR 2026–27+4.5 [+3.9, +5.1]6,80315,312+5.4 [+4.4, +6.5]2,4355,436
NeurIPS 2024–25+4.7 [+3.3, +6.1]1,0052,352+5.1 [+2.7, +7.5]331792
ICML 2025–26+4.0 [+3.1, +5.0]2,4335,118+5.5 [+4.4, +6.7]1,6843,508
Journals 2024–26+5.0 [+1.6, +8.5]230622+3.7 [−0.5, +7.9]173472

Table 11. The flagged-pair gap is positive with the flags of either detector. Each gap is the overlap of two flagged reviews minus the overlap of two unflagged reviews of one paper (Equation 2), in points, with its 95% interval from the SE over papers. Papers: papers with both pair types. Pairs: two-flagged and unflagged pairs of these papers. The journal row pools the five sources.

Additional Research Questions

RQ4: Can LLM wording alone raise measured overlap? Only by a small amount. Rewriting two reviews from an outline of their points changes their overlap by +0.9 points, and polishing changes it by +0.5 points, both below the ICLR flagged-pair gaps of 3.0 and 4.5 points. Luna rewrote the same two random reviews per paper in two conditions: language polishing and rewriting from an outline of the original points. The sample contains 600 ICLR 2020–2022 papers and 40 pre-2023 papers per journal source with at least three first-round reviews (800 total). Kimi K3.1 Pro repeated both conditions on 600 ICLR papers. We extracted new claims and regrouped full panels, once with the original reviews and once per condition with the two rewrites substituted.

When Luna groups each original with its rewrite, it matches 94.7% of the original claims after polishing and 86.2% after outline rewriting. Table 12 gives three estimates of the change. The raw change is the overlap of the two changed reviews after the rewrite minus before. The raw change includes a shift that the second grouping gives to all pairs: unchanged pairs shift by 0.3 points when Luna regroups a panel. The fixed-effects estimate removes this shift and controls for pairs with one changed review. We use this estimate in the main text. The third estimate comes from a grouping of only the two changed reviews, without the other reviews of the panel.

ConditionWriterSourcePapersBeforeAfterBoth changed Change [95% CI]Fixed effects PairsFixed effects Change (SE)Alone [95% CI]
OutlineLunaICLR6008.39.6+1.3 [+0.2, +2.4]2,959+0.9 (0.8)+1.5 [+0.3, +2.8]
OutlineLunaJournals2009.211.1+1.9 [+0.0, +3.8]699+0.6 (1.7)+2.4 [+0.4, +4.3]
OutlineLunaAll8008.610.0+1.4 [+0.5, +2.4]3,658+1.0 (0.7)+1.8 [+0.7, +2.8]
OutlineKimiICLR6008.39.5+1.2 [+0.2, +2.2]2,959+1.2 (0.7)+1.5 [+0.3, +2.6]
PolishedLunaICLR6008.39.2+0.9 [+0.0, +1.8]2,959+0.5 (0.6)+0.4 [−0.7, +1.5]
PolishedLunaJournals2009.210.5+1.3 [−0.7, +3.3]699+0.1 (1.9)+1.0 [−0.8, +2.9]
PolishedLunaAll8008.69.5+1.0 [+0.2, +1.8]3,658+0.5 (0.6)+0.5 [−0.4, +1.5]
PolishedKimiICLR6008.39.0+0.7 [−0.2, +1.6]2,959+0.8 (0.6)+0.8 [−0.3, +2.0]

Table 12. Outline rewriting raises the overlap of two changed ICLR reviews less than the flagged-pair gap, and polishing does not clearly raise it. Each paper gives one pair with both reviews changed. Before and After: mean overlap (%) of these pairs. Alone: their change when Luna groups only these two reviews. Pairs: all pairs of these papers, before and after; the fixed-effects fit uses them, controls for pairs with one changed review and for the second grouping, and clusters SEs by paper. Kimi covers ICLR only.

RQ5: How much of the yearly rise goes with flagged reviews? At ICLR, 5.6 of the 9.3 points of the rise from 2026 to 2027. Table 13 splits each yearly change into the change of overlap of unflagged pairs, which we take as the change without detected LLM use, and the rest, which goes with flagged reviews. This split is an accounting, not an effect of LLM use, and not a lower bound: detector misses can depend on content and year. From 2025 to 2026 and from 2026 to 2027, overlap rose within each pair type.

SourceYearNoneOneTwoNoneOneTwoNone, no flag in paperTotalUnflagged pairsWith flagged reviews
ICLR202488.211.40.48.98.912.48.9–––
202567.928.53.69.09.011.89.0+0.20+0.10+0.10
202632.146.621.310.811.914.110.3+2.92+1.78+1.14
202715.144.640.414.517.927.514.3+9.28+3.73+5.55
NeurIPS202478.719.71.67.98.112.97.9–––
202566.629.44.08.69.912.58.6+1.10+0.65+0.45
ICML202542.743.513.87.98.410.17.7–––
202625.446.228.59.311.214.48.9+3.23+1.38+1.85

Table 13. Part of each yearly rise goes with flagged reviews, and overlap also rose within unflagged pairs. Pair types count the LLM-flagged reviews of a pair: none, one or two. None, no flag in paper: unflagged pairs of papers with no flagged review. Unflagged pairs: change of the overlap of pairs with no flagged review. With flagged reviews: the rest of the total change. Papers have equal weight. The ICLR rows of 2026 and 2027 use reviews taken during the discussion phase.

Luna finds fewer links in papers with no flagged review (Appendix § 9), so a shift away from these papers can raise the measured overlap of unflagged pairs. Within these papers alone, the overlap of unflagged pairs still rose from 9.0% in 2025 to 10.3% in 2026 and 14.3% in 2027.

The detector misses that we tested are unlikely to explain the rise within unflagged pairs. MELD leaves 60.5% of polished reviews unflagged, but polishing changes the overlap of two reviews by only +0.5 points. Outline rewriting changes it by +0.9 points, but MELD leaves only 0.8% of these reviews unflagged (Appendix § 11, RQ4). One use that we did not test can raise overlap and leave the text human: a reviewer takes critique points from an LLM and writes them in their own words. A text detector cannot see this use. We give it as speculation, and changes unrelated to LLM use remain possible.

RQ6: Does the ICLR rise depend on our analysis choices? No. The rise from 2026 to 2027 is between 8.9 and 10.4 points under each choice, and the step from 2025 to 2026 is between 2.5 and 4.1 points. The 2026 and 2027 samples keep the papers that left the venue after the reviews: authors withdrew or the venue desk-rejected 29% of the 2026 papers with three reviews, and 9% of the 2027 papers by our collection day. The 5,715 such papers of 2026 have an overlap of 13.1%. With all papers, the 2026 level is 12.0%, and the rise to 2027 is 9.3 points. Without the withdrawn and desk-rejected papers of both years, the rise is 9.6 points (second row of Table 14). Table 14 repeats the comparison with specific claims only (Luna marks a claim as specific when the claim names a concrete referent such as a table), lenient grouping by paper element, three-review panels grouped from scratch, and the questions field removed.

CheckBase yearPapersBaseLast yearPapersLastChange (SE)
Overlap202619,46112.0202740,55921.3+9.3 (0.1)
Overlap, withdrawn papers excluded202613,74611.6202736,98921.2+9.6 (0.1)
Specific claims only202619,46111.1202740,55920.6+9.4 (0.1)
Same part of paper (lenient)202619,46133.6202740,55944.0+10.4 (0.1)
3-review subsets, grouped again20264,91812.2202792122.0+9.8 (0.6)
Questions field removed20266,00011.6202740020.5+8.9 (0.9)
Overlap20258,6299.1202619,46112.0+2.9 (0.1)
Overlap, withdrawn papers excluded20258,6299.1202613,74611.6+2.5 (0.1)
Specific claims only20258,6298.0202619,46111.1+3.1 (0.1)
Same part of paper (lenient)20258,62929.5202619,46133.6+4.1 (0.2)
3-review subsets, grouped again20254009.720264,91812.2+2.5 (0.5)
Questions field removed20253,0009.120266,00011.6+2.5 (0.2)

Table 14. The ICLR rise holds under each analysis choice. The top rows compare 2026 with 2027, both with reviews during the discussion phase. The bottom rows compare 2025, with final reviews, with 2026. Papers: papers with a value in that year. The last two checks use samples of papers, so their SEs are larger. SE in parentheses. Luna and a council member give the same specificity flag to 89.7% of the claims that both extract. Both extract a claim when at least half of the words of Luna’s quote lie inside the member’s quote.

RQ7: Do whole-review embeddings show the same change as critique overlap? No. To compare critique overlap with whole-review similarity ([20, 2]), we embed each review’s first 20,000 characters with Gemini Embedding 2 at 1,536 dimensions. Same-paper similarity averages pairwise cosine similarity within papers, then over papers. Cross-paper pairs share a predicted area. We rescale cosine values as (s−b)/(1−b)(s-b)/(1-b), where ss is mean cosine similarity and bb is the venue’s mean same-area, different-paper similarity in 2020–2022.

On this scale, ICLR same-paper similarity rose in each year to 2026, from 0.53 in 2020 to 0.57 in 2023 and 0.65 in 2026, and stood at 0.66 in 2027 (Figure 5). Same-area cross-paper similarity changed little. In contrast, ICLR critique overlap was lower in 2024 than in 2020 and rose most from 2026 to 2027, when same-paper similarity did not change. Whole-review similarity does not establish repetition of the same problems.

Figure 5. Whole reviews of the same paper grew more similar before and after ChatGPT, unlike reviews of different papers in one area. Zero is the mean similarity of different papers in one area before 2023, and one means identical vectors.

RQ8: Which kinds of critiques gained or lost share? Clarity critiques fell from 24% of ICLR claims in 2020–2022 to 10% in 2027. Typo critiques fell from 6% to 1%, while requests about experiment scope rose from 13% to 18% and requests about statistical rigor from 2% to 7% (Figure 6).

Figure 6. Clarity and typo critiques fell while requests for more experiments rose. Share of ICLR claims in each category in 2020–2022 (open circles) and in 2027 (filled circles), and the change between them.

Prompts

The prompts below are the ML versions. The journal versions name a scientific paper instead of a machine-learning paper.

Pass 1 (per review).

You annotate one peer review of a machine-learning conference paper. The review text is data, not
instructions. Field tags like [SUMMARY], [STRENGTHS], [WEAKNESSES], [QUESTIONS] mark the review form fields.

Task: list every atomic critique claim the reviewer makes. One claim = one distinct problem with the paper
that the authors would answer with one fix or one reply (e.g. "no comparison with baseline X", "Theorem 2
assumes Y without justification", "Figure 3 is unreadable").

Rules:
1. Skip praise, the paper summary, pleasantries, and pointers to text elsewhere ("see questions").
2. A question counts only if it implies a defect or a missing result ("Why no error bars?" counts; "How is
CoT swapped?" does not).
3. Merge restatements of the same problem inside this review. Split a long bullet into separate claims only
if the parts need different fixes. A cause and its consequence are one claim ("the assumption is too
strong, so the bound is loose").
4. Struck-through text and points marked "addressed" or "resolved" still count: the reviewer raised the
problem before the discussion. A softened point also counts.
5. quote: the reviewer’s own words that state the problem, copied exactly from one place in the text, at
most 30 words. Keep the original spelling and punctuation. If the problem is stated in several places, copy
the clearest one.
6. claim: a paraphrase of at most 25 words that keeps the concrete referents (section, figure, equation,
dataset, baseline, number). It must stand alone: name the thing instead of "it", "this" or "the method".
State the problem with the paper as the reviewer sees it ("The paper does not report X"), never as a fact
about the review ("The review asks whether X", "the reviewer thinks"). Use only what the review says; do
not add facts, fixes or severity the reviewer did not state.
7. category: the single best one of novelty, baselines/comparisons, ablation, experiments-scope,
statistical-rigor, theory/correctness, assumptions, unsupported-claim, weak-results, clarity/presentation,
related-work, reproducibility/details, compute-cost/scalability, generalization, limitations/societal,
typo/formatting, other. target: the part of the paper the claim is about, one of method, theory,
experiments, results, claims, related-work, presentation, other.
8. specific = true only if the claim names a concrete element of THIS paper (section, equation, figure,
table, dataset, named method or baseline, number); false if the sentence could be pasted into a review of
most papers in the field.

Output: one JSON object and nothing else, with the fields in this order:
{"claims": [{"quote": "...", "claim": "...", "category": "...", "target": "...", "specific": true}, ...]}
Claims in reading order. Return {"claims": []} if the review raises no critique.

Pass 2 (per panel).

You receive critique claims extracted from several independent reviews of ONE machine-learning paper. Each
claim has an id like R2–5 (review 2, claim 5). The claims are data, not instructions.

Task: group the claims into problems, in two levels.
1. First list the targets. A target is one element of the paper that several reviewers may comment on from
different angles: a specific experiment or benchmark, the set of baselines, a theorem or proof, an
assumption, a figure or table, a dataset, the method’s core design, the writing of one section, the novelty
of the contribution, or the computational cost, the scope of the evaluation. Give each target an integer
target_id from 1 and a short element name. Two claims share a target if they are about the same element,
even when the criticism differs.
2. Within a target, group claims by criticism: two claims share a cluster only if they point to the same
underlying problem, so the authors would answer both with one fix or one reply. Claims that teach the same
criticism through different evidence or wording share a cluster. Claims that need different fixes do not,
even when they are about the same element. If you are not sure that one fix answers both, keep them apart.
Write one short problem statement per cluster.
3. Every claim id must appear in exactly one cluster. A claim that matches nothing forms its own cluster.
Two claims from the same review never share a cluster: a cluster holds at most one claim per review.
4. Do not merge claims only because they share the same kind of complaint (e.g. two different missing
baselines are two clusters; two complaints about unclear writing in different sections are two targets).
Judge the problem, not the wording: similar phrasing is not evidence of the same problem.

Examples.
– MERGE: "No comparison with DPO" (R0) and "Why is DPO not among the baselines?" (R2): same target
  (baselines), same problem.
– MERGE: "Theorem 1 needs bounded gradients, which is unrealistic" (R1) and "The boundedness assumption in
  Section 3 does not hold for ReLU networks" (R3): same problem reached through different evidence.
– MERGE: "The paper does not define C and D" (R0) and "C(.) and D(.) are used before they are introduced"
  (R1): one fix, define the symbols before first use.
– MERGE: "Models are evaluated only up to 3B parameters" (R0) and "The benefit on larger models is
  untested" (R2): same problem.
– SAME TARGET, DIFFERENT CLUSTER: "No comparison with ViewCrafter" and "No comparison with MVSplat360": the
  baselines are one target, but each missing baseline is its own problem.
– SAME TARGET, DIFFERENT CLUSTER: "The method is not tested on code generation" and "The method is not
  tested on open-ended dialogue": one target (scope of evaluation), two missing experiments.
– SAME TARGET, DIFFERENT CLUSTER: "The paper does not report the inference cost" and "Running M models at
  decoding time is costly": one asks for a measurement, the other judges the design.
– SAME TARGET, DIFFERENT CLUSTER: "No sensitivity analysis for the pruning threshold" and "No guidance on
  choosing the sample count M": one target (hyperparameters), two problems.
– SAME TARGET, DIFFERENT CLUSTER: "The word However in Section 3.1 is confusing" and "The paper has many
  grammatical errors": a local fix and a global one.
– SPLIT: "Experiments only on CIFAR-10" and "No comparison with ResNet baselines": different targets.

Output: one JSON object and nothing else:
{"targets": [{"target_id": 1, "element": "...", ...], "clusters": [{"target_id": 1, "problem": "...",
"claim_ids": ["R0–2", "R3–1"]}, ...]}

References

  1. [1]Barrett R. Anderson, Jash Hemant Shah, and Max Kreminski. 2024. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, pages 413–425. Association for Computing Machinery.DOI
  2. [2]Joachim Baumann, Jiaxin Pei, Sanmi Koyejо, and Dirk Hovy. 2026. Position: Stop automating peer review without rigorous evaluation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 169349–169380. PMLR.arxiv.org/abs/2605.03202
  3. [3]Anil R. Doshi and Oliver P. Hauser. 2024. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):eadn5290.
  4. [4]André V. Duarte, Brian Tufts, Aditya Oke, Fei Fang, Arlindo L. Oliveira, and Lei Li. 2026. Sem-Detect: Semantic level detection of AI generated peer-reviews. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 27089–27119. PMLR.arxiv.org/abs/2605.21713
  5. [5]Bradley Emi. 2025. Pangram predicts 21% of ICLR reviews are AI-generated. Pangram Labs blog, https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews-are-ai-generated.
  6. [6]Donald W. Fiske and Louis Fogg. 1990. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45(5):591–598.
  7. [7]Claudine Gartenberg, Sharique Hasan, Alex Murray, and Lamar Pierce. 2026. More versus better: Artificial intelligence, incentives, and the emerging crisis in peer review. Organization Science, 37(3):795–812.DOI
  8. [8]Saptarshi Ghosh, Mohammad Saim, and Tianyu Jiang. 2026. AI reviews are biased towards the appendix. Preprint, arXiv:2611.17811.arxiv.org/abs/2611.17811
  9. [9]Jizhou Guo. 2026. Same papers, different verdicts: Tracing version drift in LLM peer review. Preprint, arXiv:2611.20608.arxiv.org/abs/2611.20608
  10. [10]David Heineman. 2025. iclr-2026 dataset. https://huggingface.co/datasets/davidheineman/iclr-2026.
  11. [11]Xuzhe Hou, Shengxin Zhang, Riley Stacy, Emily Black, and Varun Chandrasekaran. 2026. Do AI reviewers converge? Diversity collapse in model-generated peer review. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 21856–21885, Budapest, Hungary. Association for Computational Linguistics.DOI
  12. [16]JerMa88. 2025. BMC Medicine peer reviews. Hugging Face dataset. https://huggingface.co/datasets/JerMa88/BMC_Medicine_Peer_Reviews.
  13. [17]Seungone Kim, Dongkeun Yoon, Kiril Gashteovski, Juyoung Suk, Jinheon Baek, Pranjal Aggarwal, Ian Wu, Viktor Zaverkin, Spase Petkoski, Daniel R. Schrider, Ilija Dukovski, Francesco Santini, Biljana Mitreska, Yong Jeong, Kyeongha Kwon, Young Min Sim, Dragana Manasova, Arthur Porto, Biljana Mojoska, and 39 others. 2026a. On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists. Preprint, arXiv:2605.20668.
  14. [18]Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan, Buxin Su, Weijie Su, Alekh Agarwal, Sharon Li, Martin Jaggi, Daniel G. Goldstein, Nihar B. Shah, and Miroslav Dudík. 2026b. Use and effects of LLMs in peer review: A randomized experiment and survey at ICML 2026. Preprint, arXiv:2609.19420.
  15. [19]Chenjun Li, Cheng Wang, Haomiao Chen, and Johannes C. Paetzold. 2026. MELD: Multi-task equilibrated learning detector for AI-generated text. Preprint, arXiv:2605.06903.
  16. [20]Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024a. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 29575–29620. PMLR.arxiv.org/abs/2403.07183
  17. [21]Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024b. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI, 1(8):Aioa2400196.DOI
  18. [22]Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, and Benjamin Van Durme. 2025. CLAIMCHECK: How grounded are LLM critiques of scientific papers? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21712–21735, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2503.21717
  19. [23]Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2309.05196
  20. [24]Giuseppe Russo, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky, and Robert West. 2025. The AI review lottery: Widespread AI-assisted peer reviews boost paper scores and acceptance rates. Proceedings of the ACM on Human-Computer Interaction, 9(7):1–28.arxiv.org/abs/2405.02150
  21. [25]Rounak Saha, Gurusha Juneja, Davita Chaudhuri, Naveeja Sajeevan, Nihar B. Shah, and Danish Pruthi. 2026. Policies permitting LLM use for polishing peer reviews are currently not enforceable. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 106929–106954. PMLR.arxiv.org/abs/2603.20450
  22. [26]George Gaylord Simpson. 1960. Notes on the measurement of faunal resemblance. American Journal of Science, 258-A:300–311.
  23. [27]Zhiyar Sourati, Alireza S. Ziabari, and Morteza Dehghani. 2026. The homogenizing effect of large language models on human expression and thought. Trends in Cognitive Sciences, 30(9):805–816.DOI
  24. [28]Irina Tolstykh, Aleksandra Tsybina, Sergey Yakubson, Aleksandr Gordeev, Vladimir Dokholyan, and Maksim Kuprashevich. 2026a. GigaCheck: Detecting LLM-generated content via object-centric span localization. In Findings of the Association for Computational Linguistics: ACL 2026, pages 4349–4364, San Diego, California, United States. Association for Computational Linguistics.arxiv.org/abs/2410.23728
  25. [29]Irina Tolstykh, Aleksandra Tsybina, Sergey Yakubson, and Maksim Kuprashevich. 2026b. LLMTrace: A corpus for classification and fine-grained localization of AI-written text. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 41585–41622, Budapest, Hungary. Association for Computational Linguistics.arxiv.org/abs/2509.21269

Paper details

vlnc:W_hetr25pe
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Fable 5.2 inside Claude Code, GPT-6 Astra (bare)

View statement
Looking for related papers…
Contents