WorksComputer Science
Level Up! AI-Driven Challenges and Adaptations to Code Review Practices
Abstract
AI can write code faster than humans can review, causing a bottleneck and necessitating changes to code review practices. This bottleneck is hotly debated, giving rise to two extreme and orthogonal imagined futures: one where humans continue to be firmly entrenched in the review process, and one where AI agents merge pull requests entirely without human involvement. Yet, these futures remain rooted in folk wisdom. Through a mixed-methods study of interviews () and a survey () at a large, multinational software company with widespread access to AI, we identify AI-driven code review challenges and strategies reported as effective to meet them. We found that AI enables, necessitates, and complicates a shift in the human reviewer's role toward high-level concerns (e.g., design and intent). We found that AI-assisted authoring results in increased burden for reviewers due to challenges such as a lack of author stewardship and PR volume and size, potentially leading to downstream debt. Our survey showed AI review usage was not correlated with less difficulty keeping up with PR volume, indicating using AI for review alone is not sufficient for solving AI-driven review challenges. Developers found the most useful strategies to be organizations supporting and resourcing review, pushing review burden upstream, and prioritizing reviewer attention on the concerns that matter most.
CCS Concepts: • Software and its engineering → Software verification and validation; Collaboration in software development.
Additional Key Words and Phrases: software engineering, code review, AI-assisted software development, AI coding assistants, pull requests, developer productivity, human-AI collaboration, mixed-methods study
Introduction
Across organizations, AI-assisted authoring has spurred increases in pull request (PR) volume that outpace human reviewing capacity, necessitating changes to code review practices [6, 11, 24, 30, 40]. For example, at one mid-sized company, a top-down mandate to double throughput with AI catalyzed a symmetric increase in review load, pushing AI review to largely replace human review [30]. As PR volume outpaces review capacity, review scrutiny may decline and latency may increase [1, 63, 77]. Stakeholders await the return on investment from AI adoption, and code review appears to be a culprit: one study found that
Authors’ Contact Information: Kaia Newman, Carnegie Mellon University, Pittsburgh, USA, [email protected]; Christian Bird, Microsoft, Redmond, USA, [email protected]; Jenna Butler, Microsoft, Redmond, USA, [email protected]; Emerson Murphy-Hill, Microsoft, Redmond, USA, [email protected]. AI adoption caused a systematic redistribution of developer effort rather than increasing productivity, offsetting throughput with review burden [2].
It is highly contested how code review should be adapted to meet these pressures. Reductively, a debate has emerged with two sides: one that views human review deliberation processes as the bottleneck preventing software development from realizing promised productivity gains from AI, and one that believes humans must remain accountable and embedded in the review process at every step. Some researchers claim agents supersede human inspection, declaring “the end of code review” [51], and companies (e.g., OpenAI, StrongDM) have gone further, building agent-driven factories which develop software with little to no human involvement [44, 66]. One edict in StrongDM’s agent instructions makes the position unmistakable: “Code must not be reviewed by humans.” Online discussions are mixed, critiquing the quality of public factory code, asserting humans “own the merge button”, or arguing for human review of select concerns [19, 53, 61]. On the other side, guidelines from OWASP and Debian require human review for all PRs, citing security concerns and a need for human responsibility [18, 54]. Some researchers claim AI should be used to augment human abilities [29] or recommend mechanisms to force reviewers to engage critically, as “oversight degrades the overseer” [50]. Ultimately, no one solution is universally accepted.
Much of this debate is litigated by vendors, standards bodies, researchers, and anecdotal evidence from social media accounts that may not extend to real software development, general opinion, or multiple contexts. Less is known about challenges AI-assisted authoring brings to reviewers themselves, or which strategies may help. In this work, we study the reviewer’s perspective, as they witness review challenges and organically iterate on promising strategies. We ask:
RQ1: How is AI shifting the role of the human reviewer?
RQ2: What new challenges do reviewers face from AI-assisted authoring?
RQ3: How, and how effectively, is AI used for review?
RQ4: Which strategies involving organizational practices, review workflows, and reviewer attention are reported to effectively mitigate review challenges?
To answer these questions, we did a two-phase, mixed-methods study at a large, multinational company with widespread access to AI coding tools. First, we did semi-structured interviews with developers with high review load () to gather contextual experiences of challenges and strategies. We then conducted an anonymous survey of randomly-sampled developers () to see how the severity of review challenges and efficacy of review strategies generalize. Our study is grounded in the freely-spoken experiences of diverse practitioners amid this sea change, investigates why challenges may occur or strategies may work, and demonstrates the magnitude of challenges and strategies at scale. To our knowledge, we contribute the first account of how developers are experiencing code review challenges from AI-assisted authoring and adapting their practices and roles as reviewers in response. Key findings include:
Most surveyed developers (71%, vs. 14% who disagreed, ) reported that their code review has shifted to focus on higher-level concerns, such as design and intent. Interviews reveal that
AI adoption enables this shift by clearing up low-level issues, but also necessitates and complicates it with increased review burden and shallow review.
(2) Developers reported that AI-assisted authoring increases review burden via PR quality issues, productivity pressures, poor author stewardship, and PR volume and size. This can lead to fatigue and shallow or skipped review, potentially accumulating debt.
(3) Developers reported that AI review aids quality and speed and used it for different purposes, such as generating review comments, understanding PRs, and prioritizing attention. However, in our survey, AI review adoption was not correlated with less difficulty keeping up with review volume and pace, and developers had concerns about overreliance.
(4) The strategies reported as most helpful involve organizations promoting review and improving its conditions (e.g., by documenting implicit knowledge for AI tooling) and shifting review work upstream to authors and tooling (e.g., through a pre-review step). Interviewees thought these strategies, along with prioritizing reviewer attention, could free reviewers up to review for important, high-level concerns.
Based on our findings, we provide specific recommendations for organizations, review workflows, and tooling (§ 6), such as requiring authors to iterate with AI review prior to PR submission.
Background and Related Work
We present (1) background on code review and (2) related work on AI-assisted authoring, automating review with AI, and reviewers’ perspectives on AI-authored PRs.
Code Review
Code reviews are peer reviews of authored changes to a codebase and related artifacts (e.g., tests, documentation, etc.), with goals such as defect detection, code improvements, knowledge transfer, increased team awareness, and shared code ownership [9]. Modern code review evolved as a lightweight alternative to code inspection, an intensive, synchronous process involving structured meetings and defined roles (e.g., moderator, reviewer) [23]. Code review is often integrated with version control tooling and spans from when a PR is opened by an author until it is merged or closed. An author submits a PR, and one or more reviewers are selected by heuristic-based tooling [13] or by the author. Reviewers inspect the changes (the changeset) and leave review comments on relevant lines, to which authors respond and make revisions accordingly. Reviewers then decide to reject, accept, or wait for further iteration, until the PR is ultimately accepted, rejected, or closed. Common challenges of code review include review latency and changeset size, and best practices include giving timely feedback and breaking up PRs [45]. In § 6.1, we compare established challenges, best practices, and outcomes against those found in our work.
AI-Driven Changes to Code Review
AI-assisted authoring has been linked to changes in PR volume, code churn, review latency, review depth, and code quality [6, 11, 24, 30, 31, 40, 77]. Research has also identified how humans review and interact with AI-authored PRs (e.g., the degree of required effort) [21, 37]. Researchers have built or ideated on LLM-based systems that generate and incorporate review comments [36, 41, 42, 43, 55, 58, 68, 70, 75] or support social aspects of review such as tone and knowledge transfer [32, 56]. Research mining software repositories has found human and AI comments complementary in value offered [27, 78] and characterized merge rates of AI-reviewed PRs and incorporation rates of AI comments [14, 67]. Evidence on the effects of AI on review quality and speed is mixed [15, 71].
Closest to our work, grey literature studies have synthesized a causal theory of AI’s team-mediated effects on review [3] and practitioners’ challenges with code review and expectations for AI review [17], finding potential effects on code plausibility, review workload, and knowledge transfer. One survey captured how developers believe AI will change code review (e.g., hoping for continued human oversight) [20], and Alami et al. found AI review tools weaken peer review’s collective accountability due to aspects such as absent reciprocity and low human interaction [4]. A recent literature synthesis surfaced potential sociotechnical risks of AI adoption in code review and identified human–AI collaboration in review as an understudied topic [76].
Prior work establishes that AI-assisted authoring and reviewing are changing code review, but practitioner perspectives come largely from potentially biased or acontextual grey literature [35, 52], and no work investigates strategies practitioners use beyond using AI for review. We address both gaps with a study on reviewer perspectives, identifying the challenges they face from AI-assisted authoring and the full range of strategies they use in response.
Methods
To understand code review challenges from AI-assisted authoring and how review may be effectively adapted in response, we used a two-phase exploratory sequential design [16]: (1) interviews to qualitatively surface themes and understand the relationships and reasoning behind them, followed by (2) a survey to quantitatively generalize these findings across a large, diverse population of developers. The study was conducted at a large, multinational company that makes varied software products, where developers have widespread access to AI and tokens and encouragement to experiment. Both phases were approved by our company’s internal ethics review.
Phase 1: Interviews
The first author conducted 50-minute semi-structured interviews with 24 developers (referred to as P1–24) on Microsoft Teams. To develop the interview protocol, the first author had informal discussions (pilots, excluded from results) with seven stakeholders (e.g., developers, product managers) and one group about AI-driven code review challenges, strategies, and changes to reviewing practices. Interviews progressed through four topics related to participant contextualization and our RQs: (1) Background and Review Goals, (2) Code Review Challenges, (3) Adaptations and AI Review Usage, and (4) Wishlist (e.g., “Are there ways review processes or tools could be changed to better support code review?”). Interviewees were asked to compare their experiences to those before AI-assisted authoring throughout. Our interview protocol is in our supplementary materials.2
Recruitment and Population. Our goal was to interview developers who experience AI-accelerated development burdens more acutely and use AI for review in diverse ways. To reach developers with the heaviest review loads, we recruited P1–16 and P18 from two large, multi-project repositories on a company DevOps platform, messaging developers in waves of five per repository in descending order of PRs commented on over the prior six weeks. After 16 interviews, themes had saturated, but some AI usage patterns seen in pilots had not surfaced. To fill this gap, we recruited P17 and P19–24: one via snowball sampling, one pilot participant, and five whose recent review comments indicated AI assistance. Developers who agreed to participate filled out a consent form and screener on their review load, AI review usage, demographics, and eligibility. Eligible participants had done code review professionally before AI-authored code entered PRs they review, so they could compare before and after. Participants were men (), women (), or unknown or preferred not to say (). They were from the United States (), India (), and Norway, Australia, Croatia, Brazil, Ireland, or unknown (one each). They were aged 26 – 57 years (median of 36.5) and had 3 – 30+ years of professional software development experience (median of 14).
Analysis. The first author qualitatively analyzed interviews using inductive, reflexive thematic analysis [12] in parallel with data collection. They created analytic memos during all interviews and open coded the first two interviews to develop an initial codebook, which they iteratively refined and applied to all 24 interviews using ATLAS.ti [7]. The codebook was hierarchical, with top-level codes organizing material by topic for easy retrieval and lower-level codes identifying specific concepts (e.g., code review for ai challenges: PR quality). Codes were split, reinterpreted, and combined during analysis. All authors met for an hour weekly to challenge and augment the first author’s interpretations. We did not calculate inter-rater reliability (IRR), consistent with best practices in HCI when the codebook is part of the analysis process, rather than its product [49]. We avoid counting qualitative items due to our pursuit of unexpected findings during our inductive analysis, in line with recommended guidance [28]; our survey gives quantitative support for themes.
Phase 2: Survey
We designed an anonymous, branching Qualtrics survey to generalize qualitative findings, with items derived from major themes and codes in our codebook. Both branches shared consent, eligibility, basic AI review usage, and demographics questions, but were otherwise self-contained modules about (1) Challenges With Code Review and Strategies (the “challenges branch”, designed to take 15 minutes) or (2) Effectively Using AI for Code Review (the “AI usage branch”, 10 minutes). Participants with no AI review experience were assigned to the challenges branch, while others were randomly assigned to either. Figure 1 shows survey logic and per-branch participant counts.
Figure 1. Survey logic flowchart, including participant numbers.
Questions were 5-point Likert-type, multi- or single-select, or free response. The challenges branch asked about current code review challenges, how challenges and practices have changed since AI-assisted authoring (“comparison questions”), and strategies for reviewer attention, review workflows, and organizations. Comparison questions were shown only to participants whose professional experience predated AI-assisted authoring. All such participants answered how their ability to keep up with review volume had changed, and then were randomly shown two of four thematic blocks (e.g., PR quality) of comparison questions. The AI usage branch asked about how AI was used for review, its effects on review quality and speed, its challenges, and strategies to improve its precision and use. Branching and randomized block assignment reduced participant fatigue while preserving statistical power and RQ coverage. Questions were optional except consent, eligibility, basic AI review usage, and comparison questions.
We iteratively refined both survey branches for clarity, display and branching logic, and duration, piloting with seven developers, project managers, and researchers recruited via word-of-mouth and online channels, including two think-aloud sessions. Pilot data are excluded from results.
Recruitment and Sample. We randomly sampled 50,000 users who had posted a comment on any repository on a company DevOps platform in the prior six weeks, then removed known bots and interview participants, to avoid double-counting perspectives. We recruited via email in waves (first 1% of the sample, 4%, and then six of 10%) until response numbers to all questions were sufficiently powerful for planned statistical analyses. Ultimately, emails were sent to 28,130 users. Survey data collection took place from July 29 through August 21, 2026.
We excluded responses with implausible demographics: those older than three standard deviations above the mean, or with years of experience exceeding age minus 14. The remaining 2,590 participants who filled out the consent, eligibility, and AI usage questions form the analytic pool of responses represented in this work. 1,806 finished the survey (i.e., clicked to the end), resulting in a response rate of 9% for all responses used and 6% without partial responses. Of the 2,557 participants who were assigned a branch, 40% were assigned the challenges branch; the rest were assigned the other. Because of branching, optional questions, and the use of partial responses, all statistics are computed over respondents that answered the relevant questions.
Population. Demographic questions were optional, so counts do not sum to 2,590. Participants were men (, 81%), women (, 16%), nonbinary or self-described (, 1%), or preferred not to say (, 3%). They had 0 – 45 years of professional experience (median of 10), were aged 20 – 69 (median of 34), and hailed from 45 countries across six continents: North America (, 64%), Asia (, 18%), Europe (, 14%), South America (, 2%), Oceania (, 2%), and Africa (, 1%). Because we sampled by code review activity, developers spanned many job disciplines, most commonly “Software Engineer” (, 90%) and “Software Architect” (, 2%). The median participant reported reviewing 6 – 10 hours per week.
Analysis. We used one-sample Wilcoxon signed-rank tests against the neutral response to detect directional skew in comparison questions [34]. To rank strategies by helpfulness when respondents chose from a subset of those they had used, we fit extended Plackett–Luce models that accommodate sub-rankings and ties [72]. All -values shown are Benjamini–Hochberg-adjusted for multiple comparisons within hypotheses and related tests [10]. AI was used to help write analysis scripts.
Results
We present our results, organized by research question.
★ Survey notation: Survey results on review challenge frequency, and changes since AI-assisted authoring, appear inline for Sections 4.1 and 4.2. We use the notation (ID, [,] , direction): ID labels the survey item, (where available) is the percent of respondents who experience a challenge at least half the time, gives the split of non-neutral responses (more/less or agree/disagree), and direction is the significant direction from the Wilcoxon test (), or inconclusive otherwise. Trends (i.e., ) are reported as weakly in a direction.
RQ1 - The Role of Human Review
Our results demonstrate a shift in the role of the human reviewer toward higher-level concerns, a shift that AI both enables and necessitates, while also complicating reviewers’ ability to make it. Table 1 presents survey data on this shift. Below, we describe how AI enables high-level review. In Section 4.3.3, we demonstrate how AI-assisted authoring necessitates higher-level review, and how it complicates quality review by creating increased burden for the human reviewer.
| ID | Change | Distribution | |
| RC1 | My review has shifted toward higher-level concerns (e.g., design and intent). | 14% — 71% | |
| RC2 | I spend more effort verifying that PRs address the right problem. | 15% — 56% | |
| RC3 | Code review is now less valuable for learning new coding approaches. | 28% — 50% | |
| RC4 | PR authors discuss design with me less in advance than they used to. | 20% — 44% | |
| RC5 | PR authors take my comments about their code less personally. | 12% — 31% |
Table 1. Perceived review changes after AI-assisted authoring. Percentages are respondents who agreed (purple) vs. disagreed (yellow). . Survey items are cut for space.
Surveyed developers strongly indicated that their review is transitioning to focus more on higher-level concerns, such as intent and design (RC1, 71%/14%, agree). Interviews revealed how this shift was or will be enabled by the capabilities of AI to eliminate low-level issues in code, freeing the human reviewer to look at concerns AI is less capable of reviewing, e.g., “As the lower-level review gets automated and out of the way, it enables for much richer higher-level review, which can bring even more value to the table than it can today” (P9). Interviewees who could trust AI review to clear up many defects or other quality issues (e.g., P1) reported reviewing at a high level with less concern.
Design review may also now be deferred to code review: surveyed developers thought authors discuss the design of PRs less in advance of PR submission than before AI-assisted authoring (RC4, 56%/15%, agree). Interviews revealed design was not discussed in advance because AI can be used to rapidly prototype and try out designs. Some interviewees described how their team would review design in tandem with code: “Design and code writing can be all done in one single PR. . .I’m trying to change the perspective of my team that your baby is the [design] document, not the code itself. . . that’s one way we are trying to reduce the friction and make this cycle shorter” (P22).
Interviewees (e.g., P5) noted that they felt freer to nitpick or skip tone calibration when reviewing, since authors often interact with PRs via agents. Surveyed developers reported that authors take review feedback less personally (RC5, 31%/12%, agree). Interviewees pointed to two possible explanations: a diminished sense of ownership over AI-authored code (seen in prior work [47, 59]) and, when the reviewer was also using AI, the sense that critique was not coming from a human: “You can say, ‘yes, it’s the AI that’s complaining about your code, it’s not me’. . . the [author] can also say, ‘I don’t know why the AI did that”’ (P21). Developers may also no longer see review as a useful channel for learning low-level programming techniques (RC3, 50%/28%, agree).
However, this shift in the reviewer’s role is necessitated and complicated by AI-assisted authoring. For example, developers spend more effort verifying PRs solve the right problem (RC2, 56%/15%, agree) than before, but this may in part be due to uncertain intent in PRs (see § 4.3).
Reviewers indicated their role has shifted toward higher-level concerns (e.g., intent and design). This shift is enabled, necessitated, and complicated by AI adoption.
RQ2 - Challenges From AI-Assisted Authoring
Our results show that AI authoring brings many challenges that necessitate shallower review or the focus on high-level concerns. Factors such as a lack of author stewardship and PR volume can lead to an increased burden on the reviewer, leading to fatigue, shallow or missed review, and downstream debt. Table 2 shows most survey results for Sections 4.2.1–4.2.3.
| ID | Challenge | Distribution | More vs. Less () |
| C1 | A PR I reviewed had a changeset that was too large | 53% — 47% | 60% vs. 13% () |
| C2 | I felt pressure to review faster | 59% — 41% | 42% vs. 13% () |
| C3 | PR lacked evidence that it was adequately tested | 60% — 40% | 34% vs. 34% () |
| C4 | I didn’t do as thorough of a review as I wanted | 65% — 35% | 47% vs. 19% () |
| C5 | I spent extra time understanding poorly-explained PRs | 66% — 34% | 42% vs. 25% () |
| C6 | Code did not follow architectural or stylistic conventions | 67% — 33% | 33% vs. 34% () |
| C7 | PR description did not give needed context | 67% — 33% | 20% vs. 47% () |
| C8 | I found problems the author should have caught | 69% — 31% | 35% vs. 28% () |
| C9 | Review affected my other job responsibilities | 72% — 28% | 32% vs. 21% () |
| C10 | Code that was subtly incorrect | 73% — 27% | 40% vs. 26% () |
| C11 | Code reimplementing existing functionality | 73% — 27% | 39% vs. 26% () |
| C12 | A PR addressed something unimportant or unnecessary | 74% — 26% | 38% vs. 25% () |
| C13 | Lack of author reasoning needed to review effectively | 75% — 25% | 31% vs. 33% () |
| C14 | The author could not explain or justify a change | 76% — 24% | 49% vs. 26% () |
| C15 | The same kinds of mistakes appeared across PRs | 77% — 23% | 24% vs. 32% () |
| C16 | Cultural friction because of differing review expectations | 78% — 22% | 25% vs. 23% () |
| C17 | A PR I wanted to review did not need it when I looked | 79% — 21% | 13% vs. 14% () |
| C18 | I needed to repeat feedback I’d given on other PRs | 82% — 18% | 23% vs. 21% () |
| C19 | A PR I wanted to review was merged without my review | 85% — 15% | 19% vs. 14% () |
| C20 | The author’s changes did not address my comments | 86% — 14% | 24% vs. 31% () |
| C21 | I felt that the author was not being respectful of my time | 87% — 13% | 22% vs. 31% () |
Table 2. Code review challenges. Numbers on the right of the distribution indicate % “About half the time” or more; those to the left indicate % “Sometimes” or “Never”. Per-item . More vs. Less () reports changes since AI-assisted authoring (per-item ). Survey items are cut for space. Never Sometimes About half the time Most of the time Always
Factors: PR Quality and Productivity Pressures
We detail upstream factors contributing to increased burden on reviewers, including PR quality issues and productivity pressures.
PR Quality Issues. Developers noted that AI was imprecise and traced this back to its lack of situational or codebase awareness (e.g., lack of knowledge contained in teams). Interviewees noted AI authoring or reviewing misses important context more often in legacy codebases that may have uneven architecture or use old language patterns or API calls, e.g., “[In] codebases that are brownfield, where there’s some new stuff, some old stuff, you want to match the new patterns, not the old patterns. That’s not always easy for AI to tell” (P1). Similarly, this problem can arise in large codebases: “Because I work in humongous repos, we’re talking about 22,000 to 25,000 projects in a repository, AI… may not be able to go and find, hey, in this project we’ve done it this way” (P23). AI may also exhibit more precision issues when rare patterns or conventions are common and desired in a codebase (e.g., Hungarian variable naming, for P5).
Developers reported a rise in PRs that are subtly incorrect (C10, 27%, 40%/26%, more), which could be amplified with authors who lack relevant expertise with the codebase (e.g., junior engineers or cross-team partners, according to P18). Interviewees noted these PRs may have signals of correctness (e.g., passing tests or builds) or fix a local issue, but require domain expertise to understand negative downstream impacts. P12 described one such “insidious” PR: “[AI] fixed the problem… it actually would have introduced another regression how everyone views mail like 99% of the time… We only realized that after it got checked in.” Developers also reported AI-driven challenges with missed code reuse opportunities (C11, 27%, 39%/26%, weakly more).
Developers also reported code non-conformant with desired or conventional style and architecture, e.g., one PR where P5 could *“clearly tell that… it was entirely vibe-coded… this just isn’t how we do things in [our product]...” However, survey results did not show a significant change in unconventional code (C6, 33%, 33%/34%, inconclusive). Interviewees (e.g., P10) also noted that AI-generated artifacts lag with iteration, which can lead to extra work.
Productivity Pressures. Surveyed developers felt pressure to review faster than before (C2, 41%, 42%/13%, more). Interviews revealed this can be motivated by review volume, wanting to unblock others (e.g., given increased review latency), team culture, or pressure from authors, e.g., “People are in a rush—they want an approval, they don’t want a review” (P9). Productivity pressures also stem from perceived organizational pressures to increase throughput based on AI capabilities: “Leaders just assume that AI can do something in 40 minutes. It’s just not true. Yeah, I might be able to pump out code, but code reviewing…that’s not factored into anything” (P24). Interviewees also mentioned that quantitative metrics, such as the need “to merge 3 PRs a week” (P16), may be more valued on performance reviews. Interviewees indicated these pressures could lead to shallower or missed review and downstream debt (see Section 4.2.3).
Factors: Lack of Stewardship and Uncertain Intent
We found that AI authoring can lead to a lack of author stewardship and uncertain intent, creating burden for the reviewer.
Lack of Author Stewardship. Developers reported that AI allows authors to submit PRs they do not understand, resulting in authors being unable to answer questions about their PRs (C14, 24%, 49%/26%, more) and reviewers needing to spend additional effort understanding changes due to inadequate explanation from the author (C5, 34%, 42%/25%, more). “My biggest challenge in reviewing…people these days, they don’t look at the code they’re producing...you ask [the author], oh, ‘why didn’t you do this?’ Oh, ‘AI just did this and I left it like that.’ That makes me a little sad” (P15). Interviewees (e.g., P4) noted this could result from overtrust in, or cognitive surrender [60] to, AI.
Interviewees also experienced difficulty with authors incorporating their comments improperly or being intermediaries for their authoring agents, e.g., “I was trying to coach the AI into doing something and then at the end of the day, it wasn’t able to do it properly…It’s more responsibility on the people who are reviewing and zero responsibility on the authors. And that shift in responsibility has led to a lot more of my time being in these PRs…” (P6). Interviewees (e.g., P14) noted seeing issues (e.g., large changesets) that authors could have caught, expressing frustration. Interviewees noted that they see repetitive mistakes across PRs more often due to the difficulty of getting authors and their agents to learn from reviewer feedback, e.g., “I block a PR and then a month later I get another very similar PR …[AI] doesn’t have the context…” (P2).
By contrast, surveyed developers reported experiencing authors failing to incorporate comments, a felt lack of respect from authors, and seeing repetitive mistakes less often than before AI-assisted authoring, and infrequently (e.g., C20, 14%, 24%/31%, less, as well as C21, C15, and C18). Surveyed developers also did not report significant changes in preventable PR quality issues left by authors (C8) or cultural friction based on the expected rigor of code review (C16). These divergences may relate to differing attributes between interview and survey samples (discussed in § 5).
Uncertain Intent. Developers reported that AI-generated PRs often have misaligned or uncertain intent (i.e., they lack justification or seem useless), e.g., “LLMs love to do like, ‘you won’t need it’, kinds of things…when you review, you don’t really know whether this was the intent of the author. I find myself commenting, ‘do we mean to do that?’ And often the answer is no…this is [an] introduced maintenance cost” (P9). Interviewees (e.g., P15) suggested they spend more time now reconstructing intent, and survey results agree, suggesting that developers more often encounter seemingly-unimportant PRs (C12, 26%, 38%/25% weakly more). However, they may not be less able to review effectively due to lost intent (C13, 25%, 31%/33%, inconclusive).
Interviewees reported that lacking knowledge about the author’s AI usage can lead to difficulty calibrating review scope and depth. Interviewees noted this calibration is often based on social factors related to risk, such as the author’s historical PR quality, seniority, and expertise. However, these heuristics can be useless when the degree of author engagement is unknown: “In the past, if you knew…that [somebody] typically write[s] code that works…you don’t review it as rigorously…now it’s like every PR needs to have the exact same scrutiny as if an intern has written the code” (P8).
4.2.3 Factors: Within- and Between-PR Attention Strain. Developers reported that AI-driven PR size and volume can lead to attention strain, leading to shallow or missed review.
Within-PR Attention Strain. Developers noted that AI-assisted authoring has increasingly led to large PRs (C1, 47%, 60%/13%, more), making each PR harder or impossible to parse, e.g., “One of the problems I have is the sheer size of some of the reviews…you have to build all this context around what’s changing, and it can be really high friction…” (P11). Given PR volume and how review scope has increased, some interviewees noted “context switching is higher” (P10), though others (e.g., P2) noted it was an expected part of their role or a personal strength. Surveyed developers indicated they worry their review is shallow (C4, 35%, 47%/19%, more), and interviewees commented that “lots of people just rubber stamp PRs” (P13) (i.e., approve PRs as a formality).
Between-PR Attention Strain. All interviewees mentioned challenges keeping up with the volume and pace of code reviews, a difficulty strongly corroborated by our survey: more respondents agreed than disagreed that keeping up had been difficult. Surveyed developers reported this difficulty has worsened (56%/22%, more difficult).
Interviewees mentioned that PR volume and engineering system friction have increased review latency, e.g., “I could produce five PRs a day…but now I’m going to be paying…ten times the cost of…handling code review feedback, re-queuing my builds, babysitting the pull requests…it holds me back” (P14). Interviewees mentioned frustration with review routing exacerbating this latency (C17, 21%, 13%/14%, inconclusive). Interviewees (e.g., P17) indicated this is an old challenge, but that PR volume has made it more pressing. Authors may also send PRs out before they are ready, causing notification noise: “People are more likely…to push out what the agent said, and then ping people for code reviews…and then [expect] other people to be the checks and balances” (P7). This noise was noted to be increased by review latency (e.g., by P9) and productivity pressures (e.g., by P24).
Developers noted attention strain and productivity pressures can lead to PRs slipping through without desired human review (C19, 15%, 19%/14%, inconclusive), e.g., “I do a lot more reviews now because they come in very, very quickly and people are expected to ramp up on code almost instantly, so things are getting pushed that aren’t ready” (P6). Interviewees (e.g., P2) noted this is something they usually recognize in hindsight.
Consequences: Fatigue and Debt
Due to increased review burden, developers reported fatigue and effects on technical debt, but also challenges related to cognitive or intent debt [65] (undeveloped team codebase understanding or rationale) because of uncertain intent, unobserved merges, and an erosion of nontechnical review benefits. Table 3 shows relevant survey results for this section.
| ID | Change | Distribution | |
| DC1 | My review interactions with authors feel less like mentoring opportunities. | 19% agree, 62% disagree | |
| DC2 | I feel more mentally drained by code review than before. | 24% agree, 56% disagree | |
| DC3 | I have less patience and focus for code review than before. | 19% agree, 53% disagree | |
| DC4 | Code review is less valuable for maintaining understanding of the codebase. | 38% agree, 43% disagree | |
| Strongly disagree | Somewhat disagree | Neither | |
| Somewhat agree | Strongly agree |
Table 3. Changes to review outcomes and goals after AI-assisted authoring. Percentages are respondents who agreed (purple) vs. disagreed (yellow). . Survey items are cut for space.
Fatigue. Developers reported increased review burden impacted their other job responsibilities (C9, 28%, 32%/21%, weakly more): “So many PRs need to be reviewed. ‘When do I do my own work?’ is a question that I keep getting from the developers” (P20). Some also reported impacts on their fatigue (DC2, 56%/24%, agree) or patience for review (DC3, 53%/19%, agree), e.g., “The biggest challenge has been wanting, being asked, to stay totally in control. . . a switch flipped and I no longer had patience to sit and read line by line” (P19).
Debt. AI authoring may influence technical debt: some interviewees perceived an increase in “little regressions” (P3) or “constant patches” (P11). AI can also proliferate technical debt: “LLMs view that code [and are] like, ‘oh, there’s an existing pattern here. Let me do that pattern again’” (P6).
Rapid churn can also lead to difficulty maintaining codebase understanding: “There are many more PRs. . .And then this learning phase, where you actually read the code and understand, people don’t have time for it” (P13). However, this is still a desired benefit of review for many, as surveyed developers did not significantly report a change in how much code review was an opportunity to maintain understanding of the codebase (DC4, 43%/38%, inconclusive). Interviewees were reluctant to fully automate review due to a loss of this benefit, e.g., “[Code review is] my main mechanism for understanding what’s happening in the product. So that’s one reason I’m a little reticent to just turn [my review agent] into an agent that runs every day and does my work for me” (P21). This lack of system understanding from churn can lead to more difficulty understanding merge implications: “Because of the rate of change… it’s very difficult to know the interdependencies and… how we make sure that this is not going to cause any further downstream [impacts]…” (P20). Rapid churn can even lead to multiple codependent PRs existing in parallel, making merge implications impossible to know (a challenge for P24’s team). Developers reported that code review is less of a mentorship opportunity than before (DC1, 62%/19%, agree), partly because review interactions sometimes happen via agents: “*I’m not mentoring the engineers and building great, best-in-class engineers by doing code reviews and sharing knowledge now. I’m just a pass through back to these LLMs” (P2).
Factors from AI authoring, including PR quality issues, productivity pressures, lack of author stewardship, uncertain intent, and PR volume and size, lead to increased burden on reviewers. This burden leads to fatigue and shallow and missed review, which leads to downstream debt.
RQ3 - AI Review Tooling: Impacts and Effective Use
We detail the effects of AI review tooling on review challenges, use cases, and challenges and strategies for effective use. Participants often used general-purpose AI or “skills” (i.e., reusable prompts) and platforms that they or a team maintained, in manual or autonomous ways.
Effects on Review Volume
Effects on Review Volume. Difficulty keeping up with the volume and pace of review made using AI for review necessary for many participants: “That’s actually why I wrote the agent. . . all of a sudden it was like, oh, wow, I could be spending 8 hours a day on nothing but code reviews because there’s so many coming in so fast and they’re so big” (P21). To investigate whether using AI for review mitigates review volume difficulty, we tested whether four measures of AI use and exposure—whether respondents had used AI for review or seen AI-generated comments on PRs they review, and how frequently they use AI for review or encounter AI-generated comments (among those who had done so in the past six weeks)—were correlated with current review volume difficulty or changes in that difficulty after AI-assisted authoring. No adoption measure was correlated with a decrease in current difficulty or with a perceived change in difficulty after AI-assisted authoring (Spearman’s , Mann-Whitney ). Personally using AI for review at all was instead correlated with increased current difficulty (, ) with a small effect size, as was more frequently seeing AI-generated comments (a trend at , ). Although these correlations cannot establish causation, interview evidence suggests that increased PR volume may motivate adoption of AI for review, rather than adoption causing difficulty with volume. These results do not support a narrative that using AI for review eliminates AI-driven review volume difficulties. These results do not reflect low adoption: of all respondents (), 93% have seen an AI review bot comment on PRs they review, 69% have seen other AI-generated comments, and 82% have personally used AI for review, 80% of whom () use it at least half the time. Only 1% had no AI review experience. However, survey respondents thought AI had improved the quality and speed of their review (78% better vs. 4% worse quality; 68% faster vs. 11% slower). Some interviewees’ experiences with AI review were unhelpful (e.g., P2) or mixed (e.g., P6).
AI Review Use Cases
AI Review Use Cases. Table 4 shows data on how survey respondents used AI for review and their interactions with other AI-generated comments on PRs they review.
| ID | Current AI use | Used% | ID | Response to AI comments | Used% |
| U1 | Understand the PR better and guide review | 79% | CI1 | Checked whether valid or relevant | 89% |
| U2 | Discuss to refine and understand comments | 67% | CI2 | Used to understand PR or guide attention | 57% |
| U3 | Generate comments and selectively post | 46% | CI3 | Dismissed or flagged unhelpful ones | 56% |
| U4 | Generate comments and edit before posting | 39% | CI4 | Endorsed or elaborated on useful ones | 51% |
| U5 | Check that previous comments were addressed | 34% | CI5 | Followed up with authors | 47% |
| U6 | Prioritize changes within a PR | 25% | CI6 | Other | 4% |
| U7 | Generate comments and auto-post them as-is | 23% | CI7 | Mostly ignored them | 4% |
| U8 | Prioritize generated comments | 15% | |||
| U9 | Orchestrate other review agents with an agent | 14% | |||
| U10 | Prioritize PRs | 13% | |||
| U11 | Generate comments and manually post as-is | 13% | |||
| U12 | Other | 7% |
Table 4. Current AI use for review ( = 996) and responses to AI-generated comments ( = 1202), by % of respondents. Survey items shortened for space.
Understand PRs. Developers used AI to understand PRs or guide their review (U1, used by 79%). Interviewees used AI to ask “pointed questions” (P18) about common or potential issues, get PR summaries or line-by-line walkthroughs, or interrogate and contrast information sources. Interviewees (e.g., P22) used AI to understand PRs more often when in an unfamiliar part of the codebase or when PRs were large or complex. Developers also check whether AI comments posted on PRs they review are valid (CI1, 89%) or use them to understand PRs (CI2, 57%).
Generate Comments. Developers used AI to generate review comments for PRs (used by 39 – 46% respondents (U3, U4) in a human-in-the-loop process, and 13 – 23% (U7, U11) autonomously). Developers edit, augment, omit, verify, and discuss (U2, 67%) review comments to improve comment precision or presentation and improve issue understanding. For example, P19 would sometimes use the code location “anchored” to an AI-generated comment to verify it: “I can either have the pull request open. . . and look at that anchor position and come to my own conclusion, or I can ask follow-up questions. . .it [asks] me to approve, reject, skip, or discuss.”
Prioritize Attention. Developers used AI to prioritize their attention between changes (U6, 25%), between comments (U8, 15%), and between PRs (U10, 13%). Interviewees reported that AI prioritization often relied on issue priority or severity, supporting the attentional strategies described in § 4.5.2. Interviewees (e.g., P18) also used AI to show how changes fit together (e.g., based on their control flow or dependencies) to guide their attention and understanding.
Orchestrate Review Agents. Developers used agent orchestration (U9, 14%) to have AI agents delegate review agents to certain PRs or multiple to one (in the case of multiagent systems). For example, P21 used agent orchestration to aggregate and rank generated comments across PRs into a report, which he mediates individually. P19 had orchestration agents dynamically dispatch resources used (e.g., LLM models, tokens, numbers of agents, etc.) based on review priority.
Track Iterations. Developers used AI to check whether comments had been integrated (U5, 34%) after iteration, e.g., “If…the developer comes back and says, ‘oh, I’ve fixed all the comments’…[AI] looks at all the responses…and check[s]…if the fix is actually accurate” (P20). Developers also tracked iterations by checking author responses to AI-generated comments (CI5, 47%).
AI Review Tooling Challenges and Effective Strategies
We detail six categories of AI review tooling challenges, with indicative challenges and strategies shown in Table 5.
Table 5. Selected AI review challenges and corresponding strategies. Top: challenges. Bottom: indicative strategies for each challenge category (some had no strategy), ranked by a Plackett-Luce model fit to respondents’ top two most helpful strategies among those they had used or seen used. Worth is the estimated probability that a strategy is ranked first. ST = social tension, UE = usage effort, OR = overreliance, PM = perception management, PR = precision, PS = presentation.
| ID | AI Review Challenge | Cat. | Distribution |
| AC1 | Not all important issues were caught by AI comments | PR | 46% Sometimes; 54% Most of the time or Always |
| AC2 | AI comments were overly verbose or complex | PS | 47% Sometimes; 53% Most of the time or Always |
| AC3 | I worried my own review was insufficient after relying on AI | OR | 54% Sometimes; 46% Most of the time or Always |
| AC4 | AI-generated comments were not engaged with by the author | PM | 55% Sometimes; 45% Most of the time or Always |
| AC11 | AI review feedback was difficult or time-consuming to verify | UE | 72% Sometimes; 28% Most of the time or Always |
| AC14 | PR authors expressed frustration with AI comments | ST | 81% Sometimes; 19% Most of the time or Always |
| Never | |||
| Sometimes | |||
| About half the time | |||
| Most of the time | |||
| Always |
Table 5. Per-item –1214. Numbers on the right of the distribution indicate % “About half the time” or more; those to the left indicate % “Sometimes” or “Never”.
| ID | AI Review Strategy | Cat. | Used % | Worth | CI |
| AIS2 | Have AI review do multiple passes or put multiple agents on one PR | PR | 55% | 0.20 | [0.14, 0.25] |
| AIS3 | Add repository- or team-specific review expectations to reusable instructions (e.g., a skill) | PR | 55% | 0.11 | [0.08, 0.15] |
| AIS10 | Encourage developers to engage with AI review comments | PM | 48% | 0.02 | [0.01, 0.02] |
| AIS11 | Scale back AI review based on negative developer feedback | ST | 11% | 0.02 | [0.00, 0.03] |
| AIS13 | Instruct AI review to adjust the length, tone, or format of comments | PS | 26% | 0.01 | [0.00, 0.01] |
Table 5 (PDF p. 15).
respondents indicated whether they used each strategy; 3% selected Other, 3% selected “None, despite challenges,” and 2% selected “None, works off-the-shelf.” respondents (those whose top favorites were not Other) were used to fit the Plackett-Luce model.
Precision. The most frequent surveyed AI review challenge was incompleteness of AI comments (AC1, 54% reported occurring half the time or more). Survey results also indicate nitpicking comments (AC5, 42%), comment inaccuracy (AC7, 35%), repetitive mistakes (AC10, 28%), and repetitive mistakes despite prior instruction (AC15, 17%). Unsound comments may also be subtly incorrect: “[AI] makes a comment that actually makes no sense. . . but superficially it might make sense and people go ahead and change the code to do it that way and introduce problems” (P10). Developers reported that incorrect AI review feedback can result from a lack of implicit project knowledge (AC8, 32%) or code or documentation beyond the PR (AC9, 28%).
To improve AI review precision, developers reported using skills with repository- or team-specific instructions (AIS3, used by 55%) and would customize instructions (AIS9, 39%) for different repositories and areas of the code. Some interviewees (e.g., P7) used reusable prompts maintained by developer tooling teams and changed instructions for their use cases. These instructions were noted to benefit both AI review and AI authoring, e.g., “We have instructions that are specific to an app… we also want the people to use [the] same instructions when they build the code” (P13). Developers also gave AI review access to code or documentation (AIS4, 46%). AI review precision was improved in iterative processes where developers would incorporate human feedback from review sessions into AI review instructions or context (AIS12, 25%), e.g., “I don’t have 100% trust that [AI] has got everything… We’ve always understood that the human in the loop is the thing that is helping us make it better over time. The efficacy of review is improving because the human in the loop is actually tweaking the instructions” (P20).
Developers also adjusted prompt tone (AIS8, 34%) to improve precision. Multi-agent or -pass systems were also perceived as highly useful for AI review precision (AIS2, 55%), though interviewees (e.g., P22) noted that multiple agents or passes would sometimes catch the same issues. To raise the potential for diversified AI review findings and or catching critical issues, developers also specified concerns for AI review (AIS6, 47%), e.g., “I have very specific focus area[s]. . . security is important, or performance is important. . .I know each individual person’s behavior over the last some years of time with them. . . so [I ask] AI to pull that out” (P22). Developers also mediated AI-generated comments during review to improve precision and reduce social tension.
Presentation. Developers reported overly verbose or complex comments (AC2, 53%), overly-numerous comments (AC6, 39%), duplicate comments (AC16, 16%), and ill-formatted comments (AC17, 12%).
Developers adjusted how AI review findings were displayed to reduce social tension with authors and the review burden on themselves. Developers instructed AI review to adjust tone and reduce verbosity of comments (AIS13, 26%) and configured the display of review findings in custom applications or command line output for themselves (AIS14, 10%) to reduce cognitive load, e.g., “It will present [comments] to me one at a time. . .I cannot tolerate having to scroll back through 15 different things and read them all at the same time and make my choices” (P19).
Social Tension. Developers reported that authors could get frustrated with poor AI-generated comments (AC14, 19%), e.g., “an agent would go rogue. . . and shotgun blast somebody’s pull request with a bunch of comments [that] may or may not be legitimate. . .I was embarrassed when it happened on my behalf, and I have a tendency to take it personally when it happens to me” (P19).
If automated AI review was not precise enough or poorly presented, developers sometimes scaled back AI review (AIS11, 11%) to reduce ire from developers involved in the review process, e.g., “It was autonomous for a little bit, and it went nuts. . .I got feedback from people like, ‘hey, this thing is way too verbose.’. . . at this point it is a bot that I run locally. . . [and] sift through the feedback. . .” (P6).
Perception Management. Interviewees had trouble getting authors to attend to, trust, and properly evaluate AI-generated comments, e.g., “When we run these agentic reviews, it’s important for them to recognize that this is a first pass. . . people will see agentic reviews and they’ll be like, ‘oh, okay, no human reviewed it’” (P1). Previous experiences with AI review can also influence perception: “The first iterations of the code review bot were super noisy and so people are going to ignore it” (P2). Surveyed developers frequently saw AI-generated comments without author interaction (AC4, 45%), and sometimes saw unanswered comments when they felt a response was necessary (AC13, 23%).
To manage author perception of AI comments, developers reported dismissing or flagging inaccurate comments (CI3, 56%, see Table 4), endorsing accurate ones (CI4, 51%), and following up with authors about comments (CI5, 47%). Developers also reported encouraging developers to engage with AI-generated review comments (AIS10, 48%). Interviewees (e.g., P18) also reported strategically avoiding letting review agents post autonomously or leaving a note that they personally reviewed the code to assure authors there was a human reviewer involved.
Usage Effort. AI review can be difficult to verify (AC11, 28%) and effortful to configure (AC12, 26%). Interestingly, some interviewees did not personally use AI for review because it goes beyond their responsibility: “Invoking review on someone else’s PR…I am taking on the responsibility of verifying all of the things that the AI says also, versus if the [bot] comments. . . then that is the reviewer’s responsibility to figure out. . . (P12).
Overreliance. Surveyed developers were often worried about overreliance on AI leading to shallower review (AC3, 46%). Interviewees were actively determining what level of engagement is necessary, especially for nontechnical review value: “A majority of code changes become more black box, because I am asking AI to translate it… I’m trying to figure out what’s the best way I still can… understand the overall picture and also details of the changes” (P22). Some interviewees (e.g., P11) avoided use of agentic review due to concerns about cognitive debt.
AI-driven PR volume motivates reviewers to use AI for review, but adoption does not correlate with decreased difficulty keeping up with review volume. Developers primarily used AI to understand PRs, generate comments, and prioritize attention during review. Developers reported the most severe challenges when using AI for review are precision, presentation, overreliance, and perception management, and offered helpful strategies for all but overreliance.
RQ4 - Strategies Mitigating Review Challenges
Developers reported strategies based in organizational practices, review workflows, and reviewer attention that are helpful to resolve increased review burden from AI authoring and facilitate an effective shift toward high-level review for human reviewers. Surveyed strategies developers used are ranked by perceived helpfulness in Table 6.
| ID | Strategy | Used % | Helpful % |
| S1 | Create expectation for self-review step before submitting a PR (e.g., with a review skill or manually) | 41% | 27% |
| S2 | Give reviewers more context on the author’s intent for the PR or decisions made in it | 36% | 26% |
| S3 | Improve documentation of implicit knowledge about the codebase | 40% | 19% |
| S4 | Have organization or management incentivize or encourage code review | 27% | 16% |
| S5 | Require PRs to be ready before requesting review (e.g., passing build, won’t be iterated on, etc.) | 41% | 15% |
| S6 | Improve or create a standardized pipeline for routing PRs to reviewers | 26% | 12% |
| S7 | Block PRs over a certain size or complexity (e.g., touching too many files) | 18% | 11% |
| S8 | Invest more resources (e.g., money, formal teams) in code review tooling or processes | 19% | 11% |
| S9 | Assign more people or hours for code review | 31% | 11% |
| S10 | Give reviewers more context on how the PR was made (e.g., if/how AI was used) and the author’s understanding | 18% | 10% |
| S11 | Have organization or management measure and prioritize code quality | 24% | 10% |
| S12 | Provide code review training or dedicated sessions for sharing knowledge and tools | 12% | 10% |
| S13 | Improve or create tools or processes for prioritizing which PRs need review first | 17% | 8% |
| S14 | Meetings or messaging about community norms to improve code review culture | 22% | 3% |
Table 6. Non-AI strategies for code review, by perceived helpfulness. Helpful % is the % who picked each strategy among their top two most helpful. Used % is the % who have used strategies or seen them used. Used % ; Helpful % . 5% used Other (3% favored), 9% selected “None, we haven’t done anything to help with code review challenges,” and 4% selected “None, we have no significant code review challenges”.
Organizational Practices
To mitigate increased review burden from PR quality issues and productivity pressures, developers thought the most helpful strategies involved organizations promoting review and improving its conditions. Document Implicit Knowledge. To align developers and AI tooling and better scale their capabilities, developers found documentation of implicit knowledge about the codebase helpful (AIS5, 48%; S3, used or seen used by 40%). “That’s [what] I’m most looking forward to, to have well-defined what used to be tribal knowledge… so the agents can not only review that, but also [it] can be available when the agent is producing the code to start with” (P9). Documentation and curation can be difficult, as AI can pick up on unintentional patterns, e.g., “Sometimes [AI agents] are trying to adhere to these decisions that were never intended to be that strict. And then [documentation] ends up creating more friction rather than less” (P19).
Resource and Promote Review. Developers thought having organizations prioritize and incentivize code review (S4, 27%), devote resources to code review (S8, 19%), or assign more reviewers or hours (S9, 31%) would be particularly helpful for decreasing review latency, e.g., “If there was an actual organizational push for people to be doing reviews, [review] would be going faster. Apparently everyone wants things to go super fast here, so I don’t understand why that’s still a problem” (P15). It may also aid author stewardship and reduce cognitive debt: “The issues that we’ve been seeing with people putting up all these changes that they don’t really understand…wouldn’t be as large of an issue if people were regularly doing code review” (P12). Developers also reported attending meetings to improve code review culture (S14, 22%) or dedicated knowledge and tool sharing sessions (S12, 12%).
Measure Review Quality. Developers thought organizations could measure code quality (S11, 24%) or review health to better assess engineering productivity and mitigate productivity pressures, e.g., “Speed is what’s being measured…it doesn’t seem like we’re measuring how many iterations on these PRs are we going through…If we [measure] the right things…things would get better” (P6).
4.4.2 Review Workflows. To mitigate increased review burden from PR quality issues, uncertain intent, and a lack of author stewardship, developers thought the most helpful strategies push review burden upstream to authors and tooling.
Pre-Review. Requiring a step where authors or AI tools review PRs before human review was the most helpful strategy across our survey questions on organization and workflow as well as AI review strategies (S1, 41%; AIS1, 69%). Interviewees noted that pre-review could clear up AI-driven PR quality issues or consequences of diminished author stewardship: “I feel like it should be mandatory…you must carefully review your own code before you’re asking my time to review the code the robot wrote” (P16). Interviewees also described their pipelines which had automated kick-off (AIS7, 38%) or expectations for running centrally- or personally-maintained AI review skills prior to submission, e.g., “We wrote our own code review agent…I ask everyone to run it before they create the PR…I repeat the same question through AI on the [DevOps platform] and see whether they actually did the code review before or not” (P22, an engineering manager). However, publicly posted AI-generated comments may clutter PRs for human reviewers: “The agents really love to comment on the entire file, which makes it impossible to review…” (P8).
Require Merge Readiness. Developers thought requiring PR readiness before requesting human review (S5, 41%) could reduce review burden. Developers (e.g., P23) also found it helpful to block (S7, 18%) or require authors to chunk PRs with large changesets, a common best practice [45]. This gating of PRs, along with pre-review steps by authors and AI review, were seen as strategies that could free reviewers up to focus on the high-level concerns that now define the human reviewer’s role (see § 4.1): “I think we’re missing a step. . .if we spend time reviewing something that. . .will be iterated on 10 times based on agent review comments. . .It’s a problem to be solved to have a clear way to signal that. . . this is ready to review for humans. . .All the common mistakes that you have encoded as review agents are out of the way, so now you can focus 100% on the big picture” (P9). Interviewees (e.g., P3) also marked PRs as drafts to signal they were not stable.
Document Intent and Provenance. Giving reviewers context on author intent for the PR (S2, 36%) could help reviewers evaluate whether PRs follow their desired purpose and design. For example, P22 asked his team to “create a plan doc. . . explain what’s the design, why did I do it, what pattern I’m taking. . . and then keep the plan along with your change, so people can easily approach your code.”
Context on how PRs are made and author understanding (S10, 18%) could also help reviewers better calibrate review scope and depth. If reviewers know the provenance of a change, they can selectively look for human-typical errors: “If I know that I’m reviewing human-authored code, I’m going to check every conditional and make sure that you got the logic right” (P14). They can also selectively look for AI-typical errors (e.g., loss of intent): “Leaders in the org wanted people to disclose when they had used AI. . . [and] if AI did the whole thing. . .It kind of totally changes how you look at the PR in terms of, ‘Am I also trying to figure out if this change is the right change at all?’” (P12) However, some interviewees noted it is difficult to label the degree to which a PR is “AI-authored” (e.g., P5) or that in practice, they do not find AI disclosures useful (e.g., P19).
respondents indicated whether they used each strategy; 13% selected Other, and 8% selected “I don’t do any of these more than I used to before.”
respondents (those who chose top favorites were not Other) were used to fit a Plackett-Luce model to rank strategies.
Reviewer Attention
To mitigate attention strain from PR volume and size, developers reported prioritizing their attention during review (see Table 7), often using AI to help. Developers reported that pressure from PR volume necessitates a shift toward reviewing for higher-level concerns.
| ID | Option | Used % | worth | CI |
| AS1 | Being selective or prioritizing things I pay attention to within a PR when reviewing | 57% | 0.16 | [0.12, 0.20] |
| AS2 | Allocating more time for code review | 34% | 0.16 | [0.11, 0.21] |
| AS3 | Scoping down the domain of PRs I review (e.g., to a smaller part of the codebase) | 30% | 0.15 | [0.10, 0.20] |
| AS4 | Assessing risk to calibrate how much attention to pay to a PR | 36% | 0.13 | [0.09, 0.18] |
| AS5 | Triaging my review queue based on importance or where I can add the most value | 40% | 0.11 | [0.08, 0.14] |
| AS6 | Assessing a PR up front to decide whether to review it in depth, wait on author, or skip it | 41% | 0.10 | [0.07, 0.13] |
| AS7 | Being selective about comments I leave (i.e., avoiding commenting on less major issues) | 38% | 0.10 | [0.07, 0.13] |
| AS8 | Doing related code reviews together to decrease unnecessary context switching | 21% | 0.09 | [0.05, 0.13] |
Table 7. Attention allocation strategies, ranked by perceived helpfulness. Worth is the estimated probability that a strategy is ranked first.
Within-PR Attention Strategies. Developers scope their review to reduce within-PR attention strain burden by prioritizing concerns within PRs (AS1, 57%) and being selective about comments made (AS7, 38%), e.g., “[I review] more PRs on the top level. . .I need to pick what I’ll focus on” (P13). Developers indicated that they assess PR risk or importance to determine the depth of review needed (AS4, 36%). To aid context switching, some developers noted that they complete related reviews together to reduce onboarding effort (AS8, 21%).
Between-PR Attention Strategies. Reviewers adapted to AI-driven PR volume by allocating more time for code review (AS2, 34%). To improve review routing, developers (e.g., P2) adapted their attention by scoping the domain of PRs they review (AS3, 30%). Developers also reported assessing PRs up front (AS6, 41%) based on factors such as the depth and quantity of other reviews or PR quality to determine if they needed their review yet, if at all: “If folks have build errors, I’ll wait. . .My threshold for reviewing a PR now is higher because the volume has increased. I want to use my time to review PRs that are ready to be merged” (P3). Similar to how developers prioritize between changes within a PR, developers also reported triaging review queues (AS5, 40%) based on factors like risk, importance, deadline proximity, and the relative value they had to add, e.g., “I can often tell. . .whether it’s important or not. . .I know this guy works on these critical areas, so if he’s got a pull request out, then it’s probably important to look at it” (P21). To prioritize attention between PRs, developers also thought tooling and workflows could be adapted to improve or standardize review routing (S6, 26%) or prioritize between PRs (S13, 17%).
The strategies reported as most helpful for mitigating review burden and potentially freeing up reviewers to do higher-level review involve organizations promoting review and improving its conditions (e.g., by documenting implicit knowledge for AI tooling) and shifting review work upstream to authors and tooling (e.g., through a pre-review step).
Limitations and Divergent Findings
Limitations. Reported helpfulness of strategies is not demonstrated effectiveness. Future work could establish whether strategies participants found helpful measurably improve review outcomes. Our study also captures an uncertain transition from manual review toward AI-supported workflows; practices reported are provisional, rather than settled accounts of the future. Few survey respondents (, 8%) had only reviewed post-AI, so the experiences of “AI-native” reviewers, including whether they build the same codebase knowledge, remain largely unknown.
Threats to Validity. Survey items may not measure themes expressed by interview participants, threatening construct validity [57]. We mitigate this by providing quotes and survey item wording so readers may judge their agreement. Comparison questions also had lower sample sizes, so some inconclusive results may not signal non-generality of phenomena.
Interview and Survey Divergences. Interview and survey responses sometimes diverged. Divergence may reflect sample sizes or construct invalidity, but it also reveals tensions worth exploring.
On author stewardship, interviewees reported more recurring mistakes in AI-authored PRs, less respect from authors, and more revisions ignoring reviewer concerns, while surveyed results significantly show the opposite (C15, C18, C21, C20). Participant attributes may partially explain this gap: interviewees were senior developers selected for high review load, whereas survey respondents were randomly sampled. Developers with high review load may encounter challenges with author stewardship more often, increasing review burden. One repository we recruited interviewees from also contains many legacy codebases, where AI was reported to be imprecise (see § 4.2.1), which may shift more burden onto reviewers. Future work could investigate which reviewer attributes and sociotechnical conditions influence AI-driven review challenges.
On test and description quality, interviewees also reported that AI-generated PR descriptions often omitted why changes were made and that AI-generated tests could be vacuous or overfit (commonly-surfaced issues [39]), whereas surveyed developers reported lessened challenges with description or testing quality (C7, C3). Interviewees (e.g., P18) suggested AI may have increased the number of PRs that have these artifacts at all.
Discussion and Conclusion
We (1) situate review challenges, strategies, and outcomes with respect to prior work and (2) provide concrete recommendations for relevant stakeholders based on our results and concluding remarks.
Comparison to Pre-AI Work on Code Review
Our results demonstrate that many code review challenges identified in prior work before AI adoption, including understanding a change and its motivation, review latency, review size, understanding change provenance, and communication overhead [9, 45], all persist, but AI-assisted authoring has made some more severe (e.g., PR volume and size) and others less so (e.g., untested changes). We also identified challenges without pre-AI analogues, including a lack of author stewardship, uncertain intent, subtly incorrect code, missed reuse opportunities, and productivity pressures. Similarly, our results demonstrate that established strategies (e.g., small changes, clear descriptions, pre-review checks, and decision traceability [45]) have become more salient as ways to address AI-specific challenges. Respondents also proposed new ones, such as documenting implicit knowledge for AI and spending more resources on code review. Review goals and outcomes are also changing. Knowledge transfer is a central outcome of review [9], but respondents suggested learning programming techniques through review matters less, while maintaining a mental model of the codebase through review still matters, though AI-specific review challenges make it harder.
Recommendations for Stakeholders and Conclusion
Organizations: resource and incentivize code review. Reflecting a broader shift in developer effort toward validation of AI-generated outputs [2], participants reported spending more time on code review (C9, AS2) and wanted organizations to incentivize and allocate more resources to review (S4, S8, S9, S12). Organizations could make review contributions visible to teams and cite them in performance evaluations. Interviewees (e.g., P14) lacked time to maintain AI review skills themselves and reported waiting for off-the-shelf tooling, suggesting organizations could fund dedicated “context management” teams to maintain review skills tailored to code areas (S3).
Organizations: focus on multidimensional measures of code and review quality. Participants indicated that productivity pressures could exacerbate shallow review and debt (see § 4.2.1) and thought quality metrics were helpful (S11). Targeting metrics such as lines of code, review latency, and PR count could create perverse incentives [38, 48], and AI makes these metrics trivial to game. Measurements of developer productivity may be more effective if they are multidimensional, transparent, and target value beyond individual developer activity [25, 73]. Measurements can also gauge the distribution of familiarity for regions of a codebase (i.e., the “bus factor”) [8, 69] to determine which are particularly under threat of cognitive debt.
Review workflows: require authors to review PRs with AI before human review. The highest-ranked strategy in our study was a pre-review step (AIS1, S1) that may clear up quality issues and help authors understand and steward their PRs. Verifying AI review findings took effort, adding to reviewer burden (AC11, AC12). Requiring authors to do rounds of review with AI could reduce this effort for reviewers, improve author understanding, reduce author misevaluation of AI comments (AC4, AC13), create documented feedback to improve AI tooling (AIS12), and clear up low-level issues before human review. However, authors and reviewers have different perspectives, so author-run AI review may not by itself free reviewers up to look for higher-level concerns. Since developers saw duplicate or previously addressed AI comments (AC10, AC15), tools could be iteration-aware (e.g., via fingerprinting [64] or semantic similarity detection [5]). Assigning reviewers only after PRs are ready to merge and AI comments are addressed could also ease review routing and notification fatigue (S5, S7).
Tools: prioritize reviewer attention between and within PRs. Developers prioritized their attention between and within PRs (AS1–8) by importance or ease of understanding. Current review tools were not built for today’s PR volume, scope, and size, leading some interviewees (e.g., P1) to build their own. Tools could direct attention to risky or high-priority PRs and issues (e.g., RADAR [1]), and route reviews by author request, by the concerns humans are best positioned to address, or for developer awareness. However, directing attention may bias reviewers and reduce ability to independently identify defects [26, 71], so tools could mitigate this (e.g., by signaling uncertainty [74]).
Tools: surface author intent, author understanding, and PR provenance. Developers now review for high-level concerns (RC1), but review tools center the diff as the main artifact, a mismatch between the unit of analysis tools surface and the unit of attention review demands [62]. Developers found strategies surfacing intent, understanding, and provenance helpful (S2, S10). Tools could show “heatmaps” of AI and human authoring activity (e.g., via fine-grained interaction data [79] and its visualization [22, 33]), addressing both the desire for provenance information and the difficulty of labeling PRs as AI- or human-authored. Tools could also make the author’s intent or design a primary review artifact (as practiced by P22 and in [46]), while keeping code accessible, since interviewees still inspected it (e.g., to verify AI comments).
While our study surfaces challenges from AI-assisted authoring, the same advances that are disrupting code review now may ultimately help resolve them in the future. We believe our results demonstrate that AI could strengthen value from human review, rather than undermining it.
Data Availability
Because this study was conducted within a proprietary organization, data cannot be publicly released. In particular, we cannot share individual survey responses or interview transcripts due to confidentiality and privacy constraints. To support reproducibility, we provide the survey instrument, interview protocol, and aggregated survey results as supplemental material. This approach is consistent with prior software engineering research conducted in industrial settings.
Tables
| ID | Challenge | Cat | Count | Distribution |
| V1 | To what extent do you agree or disagree with the following statement: Over the past 6 weeks, keeping up with the volume and pace of code I was expected or wanted to review was difficult. | X | 911 | 18% — 62% |
| Strongly disagree; Somewhat disagree; Neither agree nor disagree; Somewhat agree; Strongly agree |
Table 8. Difficulty keeping up with code review volume and pace over the past 6 weeks.
Table 9. Attention challenge comparisons before vs after AI involvement, ordered by higher ’more often’ responses. Respondents per item ranged from to .
| ID | Challenge | Cat | Count | Distribution | More vs Less | p | p adj. |
| C1 | A PR I reviewed had a changeset that was too large or touched too many files | X | 316 | 13% — 60% | 60% vs 13% | ||
| C4 | I didn’t do as thorough of a review as I would have liked to | X | 315 | 19% — 47% | 47% vs 19% | ||
| C2 | I felt pressure to review faster | X | 317 | 13% — 42% | 42% vs 13% | ||
| C9 | My reviewing responsibilities affected my ability to complete other job responsibilities | X | 317 | 21% — 32% | 32% vs 21% | ||
| C18 | I needed to repeat feedback I’d already given on other PRs | X | 316 | 21% — 23% | 23% vs 21% | ||
| C19 | A PR I wanted to review was merged without my review | X | 315 | 14% — 19% | 19% vs 14% | ||
| C17 | A PR I wanted to review did not need my review when I looked at it (e.g., it was already approved, wasn’t ready, did not fit my expertise) | X | 316 | 14% — 13% | 13% vs 14% | ||
| Much less often | |||||||
| Less often | |||||||
| About the same | |||||||
| More often | |||||||
| Much more often |
Table 9.
Table 10. Volume and pace comparison before vs after AI involvement (). Respondents were asked: “Compared to before PR authors used AI for code generation, keeping up with the volume and pace of code I’m expected or want to review is ____.”
| Count | Distribution | More difficult vs Less | p | p adj. |
| 825 | 22% — 56% | 56% vs 22% | ||
| Much easier | ||||
| Easier | ||||
| About the same | ||||
| More difficult | ||||
| Much more difficult |
Table 10.
Table 11. PR quality challenge comparisons before vs after AI involvement, ordered by higher ’more often’ responses. Respondents per item ranged from to .
| ID | Challenge | Cat | Count | Distribution | More vs Less | p | p adj. |
| C10 | Code that appeared correct on the surface, but was wrong in a way that was difficult to notice | X | 308 | 26% — 40% | 40% vs 26% | ||
| C11 | Code that reimplemented something that exists somewhere else | X | 308 | 26% — 39% | 39% vs 26% | ||
| C12 | A PR addressed something that did not seem important or necessary | X | 308 | 25% — 38% | 38% vs 25% | ||
| C3 | PR lacked evidence that it was adequately tested or verified | X | 308 | 34% — 34% | 34% vs 34% | ||
| C6 | Code that didn’t follow architectural or stylistic conventions in the codebase | X | 308 | 34% — 33% | 33% vs 34% | ||
| C15 | The same kinds of mistakes appeared across many different PRs | X | 308 | 32% — 24% | 24% vs 32% | ||
| C7 | PR description or title did not give me context I needed to review effectively | X | 308 | 47% — 20% | 20% vs 47% | ||
| Much less often | |||||||
| Less often | |||||||
| About the same | |||||||
| More often | |||||||
| Much more often |
Table 11.
| ID | Challenge | Cat | Count | Distribution | More vs Less | p | p adj. |
| C14 | When I asked about a change, the author could not explain or justify it | X | 282 | 26% Much less often; 49% Much more often | 49% vs 26% | ||
| C5 | I had to spend additional time determining how or why changes worked because the author had not adequately explained them | X | 292 | 25% Much less often; 42% Much more often | 42% vs 25% | ||
| C8 | I found problems that the author should reasonably have caught before review | X | 291 | 28% Much less often; 35% Much more often | 35% vs 28% | ||
| C13 | I lacked information about the author’s intent or reasoning that I needed to review effectively | X | 285 | 33% Much less often; 31% Much more often | 31% vs 33% | ||
| C16 | I experienced friction with authors or other developers because of differing expectations about code review (e.g., its thoroughness, scope, or speed) | X | 262 | 23% Much less often; 25% Much more often | 25% vs 23% | ||
| C20 | My comments were met with responses or changes that did not address my concerns | X | 275 | 31% Much less often; 24% Much more often | 24% vs 31% | ||
| C21 | I felt that the author was not being respectful of my time | X | 242 | 31% Much less often; 22% Much more often | 22% vs 31% |
Table 12. Social interaction challenge comparisons before vs after AI involvement, ordered by higher ’more often’ responses. Respondents per item ranged from to . Much less often; Less often; About the same; More often; Much more often.
Table 13. AI review challenges, ordered by the percentage answering about half the time or more. Respondents per item ranged from . Categories: ST = social tension, UE = usage effort, OR = overreliance, PM = perception management, PR = precision, PS = presentation.
| ID | Challenge | Cat. | Distribution |
| AC1 | Not all important issues were caught by AI comments | PR | 46% — 54% |
| AC2 | AI comments were overly verbose or complex | PS | 47% — 53% |
| AC3 | After relying on AI review, I worried that I had not personally reviewed the PR closely enough | OR | 54% — 46% |
| AC4 | AI-generated comments were not engaged with by the author | PM | 55% — 45% |
| AC5 | An AI-generated comment was not worth leaving (e.g., was a nitpick) | PR | 58% — 42% |
| AC6 | AI comments were too numerous | PS | 61% — 39% |
| AC7 | An AI-generated comment was inaccurate or uncomplete | PR | 65% — 35% |
| AC8 | AI review gave incorrect feedback because it lacked unwritten project knowledge | PR | 68% — 32% |
| AC9 | AI review gave incorrect feedback because it did not account for relevant code or documentation beyond the PR | PR | 72% — 28% |
| AC10 | An AI-generated comment made the same kind of mistake I saw AI make in the past | PR | 72% — 28% |
| AC11 | AI review feedback was difficult or time-consuming to verify | UE | 72% — 28% |
| AC12 | Providing necessary context or configuration for AI review took too much effort | UE | 74% — 26% |
| AC13 | Some AI-generated comments that I thought should be responded to did not have author engagement | PM | 77% — 23% |
| AC14 | PR authors expressed frustration with AI comments | ST | 81% — 19% |
| AC15 | AI review repeated a mistake despite receiving prior instructions not to | PR | 83% — 17% |
| AC16 | AI comments repeated points from existing comments | PS | 84% — 16% |
| AC17 | AI comments were left in a broken or hard-to-read format | PS | 88% — 12% |
| Never; Sometimes; About half the time; Most of the time; Always |
Table 13. AI review challenges, ordered by the percentage answering about half the time or more. Respondents per item ranged from . Categories: ST = social tension, UE = usage effort, OR = overreliance, PM = perception management, PR = precision, PS = presentation.
Table 14. AI review tool strategies, ranked by perceived helpfulness. respondents indicated whether they used each strategy; 3% selected Other, 3% selected “None, despite challenges,” and 2% selected “None, works off-the-shelf.” respondents (those whose top favorites were not Other) were used to fit a Plackett-Luce model to rank strategies, which estimates each strategy’s relative worth from respondents’ top-choice preferences among used strategies. Categories: ST = social tension, UE = usage effort, OR = overreliance, PM = perception management, PR = precision, PS = presentation. Empty categories help with challenges from AI-assisted authoring.
| ID | Option | Cat | Used % | worth | CI |
| AIS1 | Have authors use AI review on their PRs before submitting for human review | – | 69% | 0.29 | [0.22, 0.35] |
| AIS2 | Have AI review do multiple passes or put multiple agents on one PR | PR | 55% | 0.20 | [0.14, 0.25] |
| AIS3 | Add repository- or team-specific review expectations to reusable instructions (e.g., a skill) | PR | 55% | 0.11 | [0.08, 0.15] |
| AIS4 | Give AI review access to more relevant code or supporting information | PR | 46% | 0.11 | [0.07, 0.15] |
| AIS5 | Improve documentation of the codebase or implicit knowledge to aid AI review | PR | 48% | 0.08 | [0.05, 0.10] |
| AIS6 | Instruct AI review to address a few specific concerns rather than do a generic pass | PR | 47% | 0.06 | [0.04, 0.08] |
| AIS7 | Automate AI review to run on a schedule or based on triggers (e.g., when a PR is submitted) | – | 38% | 0.03 | [0.02, 0.05] |
| AIS8 | Tune the language, tone, or emphasis of instructions (e.g., "do this at max effort," "go step by step") | PR | 34% | 0.03 | [0.02, 0.05] |
| AIS9 | Customize AI review for different repos or areas (e.g., pointing it to different context, using a different model) | PR | 39% | 0.03 | [0.02, 0.05] |
| AIS10 | Encourage developers to engage with AI review comments | PM | 48% | 0.02 | [0.01, 0.02] |
| AIS11 | Scale back AI review based on negative developer feedback | ST | 11% | 0.02 | [0.00, 0.03] |
| AIS12 | Refine AI review based on human feedback from review sessions | PR | 25% | 0.01 | [0.01, 0.02] |
| AIS13 | Instruct AI review to adjust the length, tone, or format of comments | PS | 26% | 0.01 | [0.00, 0.01] |
| AIS14 | Change how AI review findings are displayed to the user (e.g., in a dashboard or aggregated report, inline comments) | PS | 10% | 0.00 | [0.00, 0.01] |
Table 15. Interview participant table. Rev. hrs/week = the number of hours doing code review per week, PRs/week = number of reviewed PRs per week, AI review use = how frequently they use AI for review. A “monorepo” is a large repository containing multiple codebases. Most data for P24 is missing from nonresponse.
| ID | Job Position | YOE | Rev. hrs/week | PRs/week | AI review use | Country | Age | Gender | Recruitment source |
| P1 | Engineering Manager | 15 | 6-10 | 40 | Always | United States | 36 | M | Monorepo A |
| P2 | Software Architect | 16 | 20+ | – | Rarely | – | – | M | Monorepo A |
| P3 | Software Engineer | 12 | 6-10 | 20-50 | Always | United States | 38 | W | Monorepo B |
| P4 | Software Engineer | 16 | 6-10 | 100 | Never | United States | 38 | M | Monorepo A |
| P5 | Software Engineer | 14 | 1-2 | 30 | Never | United States | 36 | M | Monorepo A |
| P6 | Software Engineer | 18 | 11-20 | 22 | Sometimes | United States | 40 | M | Monorepo A |
| P7 | Software Engineer | 11 | 3-5 | 40 | Sometimes | United States | 32 | W | Monorepo B |
| P8 | Software Engineer | 5 | 6-10 | 15+ | Never | United States | 27 | W | Monorepo B |
| P9 | Software Engineer | 13 | 3-5 | 12 | Sometimes | Norway | 37 | M | Monorepo B |
| P10 | Software Engineer | 15 | 6-10 | 10 | Sometimes | United States | 36 | W | Monorepo B |
| P11 | Software Engineer | 8 | 11-20 | 10+ | Sometimes | United States | 32 | M | Monorepo A |
| P12 | Software Engineer | 7 | 3-5 | 10 | Never | United States | 26 | M | Monorepo A |
| P13 | Software Architect | 14 | 11-20 | 50 | Always | Croatia | 37 | M | Monorepo B |
| P14 | Engineering Manager | 14 | 3-5 | 10 | Never | United States | 36 | M | Monorepo A |
| P15 | Software Engineer | 10 | 6-10 | 10 | Never | Brazil | 30 | M | Monorepo B |
| P16 | Software Engineer | 15+ | 11-20 | 15-30 | N/A | United States | 47 | M | Monorepo A |
| P17 | Software Engineer | 3 | 3-5 | 3 | Always | United States | 27 | W | Word of mouth |
| P18 | Software Engineer | 12 | 6-10 | 8 | Always | India | 35 | M | Monorepo B |
| P19 | Software Engineer | 26 | 6-10 | 10+ | Always | United States | 46 | M | Pilot |
| P20 | Engineering Manager | 26 | 3-5 | – | Always | India | 48 | M | Repo Collection C |
| P21 | Software Architect | 30 | 1-2 | 5 | Always | United States | 57 | M | Repo Collection C |
| P22 | Engineering Manager | 30+ | 6-10 | 15+ | Always | United States | 54 | M | Repo Collection C |
| P23 | Software Engineer | 30 | 20+ | 50+ | Sometimes | Australia | 46 | M | Repo Collection C |
| P24 | Software Engineer | – | – | – | – | Ireland | – | – | Repo Collection C |
References
- [1]Chris Adams, Arjun Singh Banga, Parveen Bansal, Souvik Bhattacharya, Payal Bhuptani, Rujin Cao, Pedro Canahuati, Nate Cook, Brian Ellis, Prabhakar Goyal, et al. 2026. Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency. arXiv preprint arXiv:2605.30208 (2026).arxiv.org/abs/2605.30208
- [2]Sadia Afroz, Zixuan Feng, Tyler Menezes, Katie Kimura, Bianca Trinkენreich, Igor Steinmacher, and Anita Sarma. 2025. The Fast and Spurious: Developer Productivity with GenAI. arXiv preprint arXiv:2510.24265 (2025).arxiv.org/abs/2510.24265
- [3]Shyam Agarwal, Courtney Miller, Christian Kästner, and Bogdan Vasilescu. 2026. 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse. arXiv preprint arXiv:2607.07980 (2026).arxiv.org/abs/2607.07980
- [4]Adam Alami, Victor Jensen, and Neil Ernst. 2025. Accountability in code review: The role of intrinsic drivers and the impact of llms. ACM Transactions on Software Engineering and Methodology 34, 8 (2025), 1–44.DOI
- [5]Sai Venkata Akhil Ammu, Sukhjit Singh Sehra, Sumeet Kaur Sehra, and Jaiteg Singh. 2025. Amalgamation of classical and large language models for duplicate bug detection: A comparative study. Computers, Materials and Continua 83, 1 (2025), 435–453.DOI
- [6]Shan Appajodu and Ravi Boyapati. 2026. Scaling Code Reviews: Adapting to a Surge in AI-Generated Code. Salesforce Engineering Blog. Accessed: 2026-10-01. https://engineering.salesforce.com/scaling-code-reviews-adapting-to-a-surge-in-ai-generated-code/
- [7]ATLAS.ti Scientific Software Development GmbH. [n. d.]. ATLAS.ti. Berlin, Germany. https://atlasti.com/
- [8]Guilherme Avelino, Leonardo Passos, Andre Hora, and Marco Tulio Valente. 2016. A novel approach for estimating truck factors. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC). IEEE, 1–10.DOI
- [9]Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International conference on software engineering (ICSE). IEEE, 712–721.
- [10]Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300.DOI
- [11]Lee Boonstra. 2026. When AI Writes the Code, Who Reviews It? Google Cloud Transform. https://cloud.google.com/transform/when-ai-writes-the-code-who-reviews-it-to-google-cloud
- [12]Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
- [13]H Alperen Çetin, Emre Doğan, and Eray Tüzün. 2021. A review of code reviewer recommendation studies: Challenges and future directions. Science of Computer Programming 208 (2021), 102652.DOI
- [14]Kowshik Chowdhury, Dipayan Banik, KM Ferdous, and Shazibul Islam Shamim. 2026. From industry claims to empirical reality: An empirical study of code review agents in pull requests. In Proceedings of the 23rd International Conference on Mining Software Repositories. 984–988.
- [15]Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Automated code review in practice. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 425–436.arxiv.org/abs/2412.18531
- [16]John W Creswell and Vickil Plano Clark. 2007. Mixed methods research. Thousand Oaks, CA 143 (2007).
- [17]Nicole Davila, Jorge Melegati, and Igor Wiese. 2024. Tales from the trenches: Expectations and challenges from practice for code review in the generative ai era. IEEE Software 41, 6 (2024), 38–45.DOI
- [18]Debian Project. 2026. General Resolution: LLM Usage in Debian. Debian Vote Page. https://www.debian.org/vote/2026/vote_002
- [19]dhorthy. 2026. Why Software Factories Fail (or: harness engineering is not enough). Hacker News. https://news.ycombinator.com/item?id=49023019
- [20]Michael Dorner, Andreas Bauer, Darja Šmite, Lukas Thode, Daniel Mendez, Ricardo Britto, Stephan Lukasczyk, Ehsan Zabardast, and Michael Kormann. 2025. Quo vadis, code review? exploring the future of code review. arXiv preprint arXiv:2508.06879 (2025).DOI
- [21]Kacper Duma, Patryk Wróblewski, Jagoda Bobińska, Julia Winiarska, and Piotr Przymus. 2026. These Aren’t the Reviews You’re Looking For: How Humans Review AI-Generated Pull Requests. arXiv preprint arXiv:2605.02273 (2026).arxiv.org/abs/2605.02273
- [22]SC Eick, Joseph L Steffen, and Eric E Sumner Jr. 1992. SeeSoft-a tool for visualizing line oriented software statistics. IEEE Transactions on Software Engineering 18, 11 (1992), 957.DOI
- [23]Michael E Fagan. 1999. Design and code inspections to reduce errors in program development. IBM Systems Journal 38, 2.3 (1999), 258–287.
- [24]Faros AI. 2025. The AI Productivity Paradox Report. Faros AI Blog. https://www.faros.ai/blog/ai-software-engineering
- [25]Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There’s more to it than you think. Queue 19, 1 (2021), 20–48.DOI
- [26]Enrico Fregnan, Larissa Braz, Marco D’Ambros, Gül Çalıklı, and Alberto Bacchelli. 2022. First come first served: The impact of file position on code review. In Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering. 483–494.DOI
- [27]Saul Goldman, Hong Yi Lin, Jirat Paksuksmit, Patamamon Thongtanum, Kla Tanti thamthavorn, Zhe Wang, Ray Zhang, Ali Behnaz, Fan Jiang, Michael Siers, et al. 2025. What types of code review comments do developers most frequently resolve?. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3760–3765.arxiv.org/abs/2510.05450
- [28]David R Hannah and Brenda A Lautsch. 2011. Counting in qualitative research: Why to conduct it, when to avoid it, and when to closet it. Journal of management inquiry 20, 1 (2011), 14–22.DOI
- [29]Andreas Haupt and Erik Brynjolfsson. 2025. Position: AI should not be an imitation game: Centaur evaluations. In Forty-second International Conference on Machine Learning Position Paper Track.
- [30]Hao He, Shyam Agarwal, Yeergen Denisov-Blanch, Pavel Azaletskiy, Sammi Koyejo, and Bogdan Vasilescu. 2026. AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate. arXiv preprint arXiv:2607.01904 (2026).arxiv.org/abs/2607.01904
- [31]Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, and Bogdan Vasilescu. 2026. Speed at the cost of quality: How Cursor AI increases short-term velocity and long-term complexity in open-source projects. In Proceedings of the 23rd international conference on mining software repositories. 181–193.DOI
- [32]Lo Heander, Småne Söderberg, and Christoffer Rydénfält. 2025. Support, not automation: towards AI-supported code review for code quality and beyond. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 591–595.DOI
- [33]William C Hill, James D Hollan, Dave Wroblewski, and Tim McCandless. 1992. Edit wear and read wear. In Proceedings of the SIGCHI conference on Human factors in computing systems. 3–9.DOI
- [34]Myles Hollander, Douglas A Wolfe, and Eric Chicken. 2013. Nonparametric statistical methods. John Wiley & Sons.
- [35]Nan Hu, Jie Zhang, and Paul A Pavlou. 2009. Overcoming the J-shaped distribution of product reviews. Commun. ACM 52, 10 (2009), 144–147.DOI
- [36]Imen Jaoua, Oussama Ben Sghaier, and Houari Sahraoui. 2025. Combining large language models with static analyzers for code review generation. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 174–186.
- [37]Syrine Khelifi, Ali Ouni, and Maha Khemja. 2026. Behind agentic pull requests: An empirical study on developer interventions in AI agent-authored pull requests. In Proceedings of the 23rd International Conference on Mining Software Repositories, 842–846.DOI
- [38]Amy J Ko. 2019. Why we should not measure productivity. In Rethinking Productivity in Software Engineering. Springer, 21–26.DOI
- [39]Michael Konstantinou, Renzo Devogivanni, and Mike Papadakis. 2024. Do llms generate test oracles that capture the actual or the expected program behaviour? arXiv preprint arXiv:2410.21136 (2024).
- [40]Hao Li, Haoxiang Zhang, and Ahmed E Hassan. 2025. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003 (2025).arxiv.org/abs/2507.15003
- [41]Yingling Li, Yuhan Wu, Zi’ao Wang, Lei Huang, Junjie Wang, Jianping Li, and Mingyun Huang. 2025. CodeDoctor: multi-category code review comment generation. Automated Software Engineering 32, 1 (2025), 25.DOI
- [42]Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, and Christoph Treude. 2025. CodeReviewQA: the code review comprehension assessment for large language models. In Findings of the Association for Computational Linguistics: ACL 2025. 9138–9166.arxiv.org/abs/2503.16167
- [43]Hong Yi Lin, Patanamon Thongtanunam, Christoph Treude, Michael W Godfrey, Chunhua Liu, and Wachiraphan Charoenwet. 2026. Leveraging reviewer experience in code review comment generation. ACM Transactions on Software Engineering and Methodology 35, 5 (2026), 1–34.arxiv.org/abs/2409.10959
- [44]Ryan Lopopolo. 2026. Harness Engineering: Leveraging Codex in an Agent-First World. OpenAI Blog. https://openai.com/index/harness-engineering/
- [45]Laura MacLeod, Michaela Greiler, Margaret-Anne Storey, Christian Bird, and Jacek Czerwonka. 2017. Code reviewing in the trenches: Challenges and best practices. IEEE Software 35, 4 (2017), 34–42.DOI
- [46]Chandra Maddila, Mashrur Rashik, Euna Mehnaz Khan, Smriti Jha, James Saindon, Nachi Nagappan, and Peter C Rigby. 2026. From Code Review to Code Critique: Intent, Drift, and Spotlight for AI-Generated Diffs at Scale. arXiv preprint arXiv:2607.29516 (2026).arxiv.org/abs/2607.29516
- [47]Alberto Martin-Lopez, Rosalia Tufano, Emanuele Guglielmi, Ana B Sánchez, Ana Lourdes Sanz, Simone Scalabrino, Rocco Oliveto, Sergio Segura, and Gabriele Bavota. 2026. More Code, Less Understanding? On the Impact of AI Assistants on Developers’ Productivity and Code Ownership. IEEE Transactions on Software Engineering (2026).
- [48]Christopher Mattson, Reamer L Bushardt, and Anthony R Artino Jr. 2021. When a measure becomes a target, it ceases to be a good measure. Journal of Graduate Medical Education 13, 1 (2021), 2–5.DOI
- [49]Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on human-computer interaction 3, CSCW (2019), 1–23.
- [50]Margaret Mitchell, Avijit Ghosh, and Samir Passi. 2026. AI agents push humans out of the loop. arXiv preprint arXiv:2608.23642 (2026).arxiv.org/abs/2608.23642
- [51]Martin Monperrus. 2026. The End of Code Review: Coding Agents Supersede Human Inspection. arXiv preprint arXiv:2606.13175 (2026).arxiv.org/abs/2606.13175
- [52]Kaia Newman, Sarah Snay, Madeline Endres, Manasvi Parikh, and Andrew Begel. 2025. Disclosure of Neurodivergence in Software Workplaces: a Mixed Methods Study of Forum and Survey Perspectives. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility, 1–17.DOI
- [53]Addy Osmani. 2026. Code Review in the Age of AI. Elevate (Substack). https://addyo.substack.com/p/code-review-in-the-age-of-ai
- [54]OWASP Cheat Sheet Series Team. 2026. Secure Coding with AI Cheat Sheet. OWASP Cheat Sheet Series. https://cheatsheetseries.owasp.org/cheatsheets/Secure_Coding_with_AI_Cheat_Sheet.html
- [55]Shirin Pirouzkhah, Pooja Rani, Francesco Sovrano, Vincent Hellendoorn, and Alberto Bacchelli. 2026. The price of precision: the cost of preprocessing for automated code revision in code review. Empirical Software Engineering 31, 2 (2026), 47.DOI
- [56]Md Shamimur Rahman, Zadia Codabux, and Chanchal K Roy. 2024. Do words have power? understanding and fostering civility in code review discussion. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1632–1655.
- [57]Paul Ralph and Ewan Tempero. 2018. Construct validity in software engineering research and software metrics. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. 13–23.DOI
- [58]Xiaoxue Ren, Chaoqun Dai, Qiao Huang, Ye Wang, Chao Liu, and Bo Jiang. 2025. Hydra-Reviewer: A holistic multi-agent system for automatic code review comment generation. IEEE Transactions on Software Engineering (2025).DOI
- [59]Jwawon Seo, Elmira Deldari, and Helena M Mentis. 2026. Whose Code Is It? How AI Autonomy Reshapes Ownership, Responsibility, and Disclosure in AI-Assisted Programming. In Proceedings of the 31st International Conference on Intelligent User Interfaces. 393–425.DOI
- [60]Steven D. Shaw and Gideon Nave. 2026. Thinking—fast, slow, and artificial: How AI is reshaping human reasoning and the rise of cognitive surrender. Available at SSRN 6097646 (2026).
- [61]Elle Shwer. 2025. Code Review in the Age of AI: Why Developers Will Always Own the Merge Button. The GitHub Blog. https://github.blog/ai-and-ml/generative-ai/code-review-in-the-age-of-ai-why-developers-will-always-own-the-merge-button/
- [62]Emma Söderberg, Luke Church, Jürgen Börstler, Diederick Niehorster, and Christofer Ryndfält. 2022. Understanding the experience of code review: misalignments, attention, and units of analysis. In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering. 170–179.DOI
- [63]Fangchen Song, Ashish Agarwal, and Wen Wen. 2025. The impact of generative AI on collaborative open-source software development: Evidence from GitHub Copilot. Information Systems Research ja (2025).arxiv.org/abs/2410.02091
- [64]Jaime Spacco, David Hovemeyer, and William Pugh. 2006. Tracking defect warnings across versions. In Proceedings of the 2006 international workshop on Mining software repositories. 133–136.DOI
- [65]Margaret-Anne Storey. 2026. From technical debt to cognitive and intent debt: Rethinking software health in the age of AI. arXiv preprint arXiv:2603.22106 (2026).
- [66]StrongDM AI. 2026. Software Factory. StrongDM. https://factory.strongdm.ai/
- [67]Kexin Sun, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, and Christoph Treude. 2026. Does ai code review lead to code changes? a case study of github actions. IEEE Transactions on Software Engineering (2026).
- [68]Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F Bissyandé. 2024. Codeagent: Autonomous communicative agents for code review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11279–11313.
- [69]Patanamon Thongtanunam, Shane McIntosh, Ahmed E Hassan, and Hajimu Iida. 2016. Revisiting code ownership and its relationship with software quality in the scope of modern code review. In Proceedings of the 38th international conference on software engineering. 1039–1050.DOI
- [70]Rosalia Tufano, Ozren Dabić, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. 2024. Code review automation: strengths and weaknesses of the state of the art. IEEE Transactions on Software Engineering 50, 2 (2024), 338–353.arxiv.org/abs/2401.05136
- [71]Rosalia Tufano, Alberto Martin-Lopez, Ahmad Tayeb, Ozren Dabić, Sonia Haiduc, and Gabriele Bavota. 2025. Deep learning-based code reviews: A paradigm shift or a double-edged sword?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1640–1652.arxiv.org/abs/2411.11401
- [72]Heather L. Turner, Jacob van Etten, David Firth, and Ioannis Kosmidis. 2020. Modelling Rankings in R: The PlackettLuce Package. Computational Statistics 35 (2020), 1027–1057. doi:10.1007/s00180-020-00959-3DOI
- [73]Medha Umarji and Forrest Shull. 2009. Measuring developers: Aligning perspectives and other best practices. IEEE software 26, 6 (2009), 92–94.DOI
- [74]Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q Vera Liao, and Jennifer Wortman Vaughan. 2025. Generation probabilities are not enough: Uncertainty highlighting in ai code completions. ACM Transactions on Computer-Human Interaction 32, 1 (2025), 1–30.DOI
- [75]Miku Watanabe, Yutaro Kashiwa, Bin Lin, Toshiki Hirao, Ken’Ichi Yamaguchi, and Hajimu Iida. 2024. On the use of chatgpt for code review: Do developers like reviews by chatgpt?. In Proceedings of the 28th international conference on evaluation and assessment in software engineering. 375–380.DOI
- [76]Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. 2026. A Roadmap for Modern Code Review: Challenges and Opportunities. ACM Transactions on Software Engineering and Methodology (2026).DOI
- [77]Haoran Yu, Lifei Liu, Xiaochong Jiang, Yuwen Jia, Su Wang, Pin Qian, and Yihang Chen. 2026. Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code. arXiv preprint arXiv:2606.22721 (2026).arxiv.org/abs/2606.22721
- [78]Suzhen Zhong, Shayan Noei, Ying Zou, and Bram Adams. 2026. Human- ai synergy in agentic code review. arXiv preprint arXiv:2603.15911 (2026).arxiv.org/abs/2603.15911
- [79]Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN international symposium on machine programming. 21–29.arxiv.org/abs/2205.06537
