We Ran a Research Sprint With AI. The AI Worked. The Research Didn't

Introduction

Studies document individual productivity gains alongside task-dependent limitations [6, 11, 46]. Farach et al. [14], in a field experiment with 388 employees at a Fortune 500 retailer, argue that the structural conditions surrounding AI use matter alongside access. Prior work examines individual coding and writing assistance [47, 25], agentic execution [50, 5], and automated machine learning drafts [29, 51]. AI-generated research ideas have been rated more novel, but slightly less feasible, than expert ideas [42]; controlled writing and coding studies report productivity gains [33, 36]. These results motivate testing, rather than assuming, improvements in team research.

We focus on what changes when the unit of work is a team. Human-AI interaction guidelines [1] and empirical design-space surveys [22] provide foundations for studying people working with AI. Our question concerns shared context and collective evaluation when researchers work in parallel, hand off context, and make consequential decisions together. The CSCW and team-cognition literatures offer relevant concepts: common ground [9], workspace awareness [15], distributed cognition [18], transactive memory [49], and observability and directability when automation joins a team [21, 39]. In parallel, AI access has been folded into expectations of speed without renegotiating verification costs. AI-generated phrasing has diffused measurably across scientific papers [28], although this does not establish declining quality. We use research slop for the broader risk of generation outpacing evaluation, not an outcome measured here. Table 1 distinguishes three interaction topologies. Our pilot motivates investigating the third: AI supporting shared context and collective evaluation rather than only individual production.

Mode	Unit of analysis	Status	Evaluation concern
Human-AI	One person, one model	Default	Private contexts may fragment team evaluation.
AI-AI	One pipeline, no human in loop	Trajectory [29, 51]	Human verification deferred downstream.
Human-AI-Human	The team; AI mediates between researchers	Design agenda	Collective evaluation remains to be tested.

We conducted a 100-minute instrumented pilot with twelve researchers in three independently operating teams, each paired with a silent observer, within a seventeen-person session. Teams were asked to produce an empirical paper with access to Claude, Claude Code, ChatGPT, and Copilot. Observers logged AI interactions and dispositions; stage-transition micro-surveys and a closing adversarial debrief captured participant reflections. A parallel autonomous pipeline produced a draft without human intervention, serving as a limit case rather than a quality comparison. Across teams, reflections, logs, and artifacts suggested two recurring patterns: a consumption bottleneck, where generation outpaced collective evaluation, and AI-induced research silos, where private sessions impeded shared understanding. One participant wrote: “AI is a multiplier for personal work, not team work.” These convergent observations motivate hypotheses, not causal estimates.

Proposed Position: AI tools designed as personal multipliers may amplify consumption bottlenecks and private research silos under deadline pressure. We argue for Human-AI-Human systems that connect researchers’ contexts, expose disagreements, and support collective evaluation. Better workflows and existing collaboration platforms may realize this category; stronger models alone do not guarantee it. Collective evaluation is a testable pathway toward research quality, not an improvement demonstrated by this pilot.

Contributions. We characterize two candidate team-level mechanisms, a consumption bottleneck and AI-induced silos (§ 4); distinguish autonomous production from human evaluation (§ 4.3); organize interaction topologies (Table 1); and propose seven falsifiable design hypotheses for team-centered AI (§ 5).

Related Work

We map five clusters relevant to AI-assisted research, distinguishing evidence about individual assistance, autonomous execution, and team collaboration.

AI as research worker. Tool-using and agentic systems [52, 50, 19] have produced scientific instantiations including Coscientist [5], FunSearch [37], end-to-end paper pipelines [29, 51], and Agent Laboratory [38]. Galactica targets scientific knowledge generation [45]. These systems establish production capabilities, not necessarily shared team evaluation. Human participation is not categorically absent: AutoGen supports human inputs, and Agent Laboratory enables feedback at each research stage.

Generative AI and knowledge workers. Productivity gains are documented in several settings [33, 6, 36, 10], but the qualifications are substantive. Dell’Acqua et al. [11] report poorer performance outside the model’s frontier. Vaccaro et al. [46] find that human+AI underperforms the better of either alone on average, with important task differences. In short-story writing, Doshi and Hauser [13] find higher individual creativity but reduced diversity across outputs; this is an analogy, not evidence about research teams. Metacognitive demands [44, 43] and suggestion-verification costs [32] motivate our consumption-bottleneck hypothesis. Whether shared research artifacts can sustain collective evaluation across private AI sessions remains a question.

Scaffolding AI use at the team level. The most directly relevant recent evidence is Farach et al. [14], a field experiment with 388 employees at Gap Inc. that varied how AI use was structured. Their synchronous Create-Out-Loud protocol (pairs discussed verbally, then prompted Copilot to draft from the transcript) was associated with documents scoring 4.96 points lower on a 22-point rubric ($p < .001$) and lower odds of producing a document ($OR = 0.12$). Session timing, differential attrition, and length-sensitive grading complicate causal interpretation. Reframing AI as a thought partner showed gains concentrated in the upper quality distribution. At Procter & Gamble, individuals with AI matched unaided teams on innovation tasks [12]. A medical field study links “engaged augmentation” to professionals actively interrogating AI outputs [23]. These findings motivate testing how AI supports collective evaluation, not prescribing who speaks first. Human-AI-Human concerns shared context and decision authority, rather than a particular ordering of conversation and drafting.

Human-AI interaction and co-creation. Interaction guidelines [1] and decision-making surveys [22] provide design foundations. Appropriate reliance depends on explanation design, cognitive effort, and automation conditions [3, 48, 7, 34, 35]. CoAuthor documents individual writer-model interaction [25]; a broader design space spans writing-assistant tasks and interactions [26]; VISAR supports argument planning and draft prototyping [53]. Opinionated models can shift users’ views [20], motivating our hypothesis that private sessions could alter team alignment.

Team cognition, CSCW, AI in groups. This cluster motivates questions about common ground [9], workspace awareness [15], distributed cognition [18], transactive memory [49], observable and directable automation [21, 39], and AI-mediated communication [16]. Automation can leave humans with demanding residual roles [2]. Group-AI work is suggestive: LLM devil’s advocates can promote appropriate group reliance on AI [8]; LLM-augmented brainstorming explores group ideation and evaluation [40]; selectively requesting peer second opinions can mitigate AI overreliance [30]; and AI suggestions improved empathy in text-based peer support [41]. Messeri and Crockett [31] discuss risks of illusions of understanding in science; our position examines candidate organizational mechanisms.

Nearest neighbors. Table 2 distinguishes seven related works from our proposed focus on shared context and team-visible evaluation.

Prior work	Claim	What we add
Farach et al. [14]	A synchronous AI drafting protocol was associated with lower document quality and production.	A motivation to test shared context and collective evaluation, not a fixed sequence of human and AI turns.
Tankelevitch et al. [44]	A conceptual framework for individual metacognitive demands of generative AI.	A proposed team-level bottleneck when shared evaluation competes with production.
Messeri and Crockett [31]	AI may foster illusions of understanding in science.	Candidate team-level organizational mechanisms.
Vaccaro et al. [46]	Human+AI is not automatically better than the best of either.	Shared context and collective evaluation as specific team-level design variables.
Liang et al. [27]	Developers report difficulty controlling outputs and meeting code requirements.	Cross-person context and collective evaluation as additional design concerns.
Chiang et al. [8]	LLM devil’s advocates can promote appropriate group reliance on AI.	A broader topology encompassing this intervention.
Wu et al. [50]	Multi-agent conversations can combine models, tools, and human inputs.	Explicit support for shared researcher contexts and collective decisions.

The Session as Instrument

This mechanism-generating pilot involved seventeen people: twelve applied AI researchers in three four-person teams, three non-participating AI-researcher observers, an orchestrator, and a critic. Recruitment was voluntary from an internal organizational cohort; task participants were already familiar with the available AI tools. Participants provided written informed consent covering possible publication, and the study underwent internal organizational research review. The shared task was to produce an empirical paper on how do we capture the evolution of a research idea, and find the gaps in it, in a human-agent setup? The 100-minute session imposed hard cutoffs across ideation, design, execution, analysis, and writing. Teams chose among prompt styles (A Direct, B Collaborative, C Roleplay, D Adversarial) and freely used Claude, Claude Code, ChatGPT, or Copilot, without facilitation on prompting or division of labor. A silent observer per team logged interactions and dispositions (accepted / modified / rejected / unresolved). Reliance research informed this focus [35, 48]; the four categories were our operationalization. Short stage-transition surveys and a closing adversarial debrief supplied complementary evidence. Participants described AI as useful for literature review, refining ideas, evaluation criteria, and executing specified plans, but difficult to consume, verify, and consolidate under deadline.

Why 100 minutes, and the autonomous parallel run. The compressed deadline made end-to-end observation feasible; it cannot establish whether pressure exposed or created the patterns. Teams chose B/C/D for ideation, describing adversarial prompts as “too aggressive” and roleplay as “surface level.” Once problem statements were locked, an autonomous paper pipeline received the same problem statement and ran without human intervention during the remaining session time. It serves only as a production-capacity limit case (§ 4.3), not a controlled comparison.

Findings

Participants described faster generation without sufficient time for collective evaluation. Two recurring patterns appeared across all three independently operating teams; we interpret them as candidate mechanisms rather than identified causal effects.

The consumption bottleneck

We define the consumption bottleneck as the gap between the rate at which AI produces outputs and the rate at which the team can evaluate them. “Evaluate” is the operative term: not read, but read closely enough to catch a substantively wrong claim. Under deadline pressure, this form of reading is the first activity to be deferred. The phenomenon was named by all three teams in survey or debrief, was followed by an observable behavioral correction in two of three teams (AI was reduced or unplugged at consequential moments), and registered in the instrument as a logging collapse: of 33 logged interactions, only one was accepted and twenty-seven received no disposition (Figure 2). Two of the three teams independently produced the same verbatim self-diagnosis, “plowed ahead instead of vetting,” and reported tool-boundary friction in which copy/paste between Claude Code and the shared pipeline absorbed the time that would have been spent evaluating outputs.

What participants said. The adversarial debrief asked each team for the most friction-filled moment. Team 1: “Content consumption and assimilation.” Team 2: “A lot of information to consume; combining outputs generated individually.” Team 3, mid-session in a Stage 2 micro-survey: “No time to consume AI generated results and make knowledgeable decisions on it.” Three teams, three instruments, three nearly interchangeable formulations. These responses emphasize difficulty consuming outputs, rather than generation speed alone.

What teams did when they noticed. Two of three teams produced an observable behavioral correction. Team 1, approximately thirty minutes in, made an explicit decision: “No AI tool used to design experiment.” The team disabled the tools for the one stage in which incorrect outputs would be unrecoverable. Team 2 abandoned the adversarial prompt protocol mid-task and switched to a direct prompt because, in the participant’s own debrief language, the adversarial style “didn’t work well” under the available time. These adaptations suggest that participants responded to friction by reducing AI involvement or simplifying prompting; they do not establish that better prompts could not help. Both teams named the same compromise in the closeout debrief: “due to time constraints, [we tried] to plow ahead instead of ensuring the quality of the work.”

Instrument behavior. Across all three teams, 33 interactions were logged: one accepted, five modified, twenty-seven with no disposition recorded (Figure 2). Observer logging depended on a team-visible decision moment. Missing entries are consistent with private or deferred decisions, but may also reflect observer overload. They establish incomplete disposition records, not that every output was used without evaluation. Participant reflections and artifacts provide complementary evidence for the proposed bottleneck.

Mechanism. Using and evaluating generative AI imposes cognitive and metacognitive demands [44, 43]; fluent text need not be grounded in meaning [4]. Automation research documents attention-related complacency and demanding residual human roles [34, 2]. Explanations can increase acceptance regardless of correctness [3], but can reduce overreliance when they lower verification costs [48]. Suggestion-verification time can be underestimated [32]. Our proposed extension is team-level: private contexts may make verification harder to distribute. Participants reported prioritizing progress over vetting. Whether this mechanism contributes to unevaluated research output remains untested.

Falsifiable form. Teams using private AI assistants under time pressure will evaluate a smaller share of AI outputs collectively than teams using shared-context assistants under equivalent pressure, or private assistants with more time.

AI-induced research silos

The second failure is geometry, not volume. Unlike AI-mediated communication between people [16], private AI sessions need not expose their context to teammates. Under deadline, we hypothesize that attention may shift toward these sessions at the expense of shared workspace awareness [15]. Private sessions can leave integration to individual judgment. Reduced diversity across AI-assisted stories [13] offers a possible analogy, not direct team-level evidence. Team 3 named it mid-session (“AI is multiplier for personal work not team work”); Team 1 theorized it in real time (“AI utility theorized to increase when working in silo”); Team 2 produced the most direct artifact, a Claude-drafted hypothesis summary “with acronyms without any context, making it difficult to understand for other team members.” Integration was performed, in Team 1’s words, “subjectively by a team member”; Team 3 logged idle time, “waiting for the team picking the dataset.” Asked what they would change, no team asked for stronger models or longer context. They asked to take the parallelism out.

Mechanism. The CSCW and team-cognition literatures offer interpretive foundations: grounding contributions in shared understanding [9], awareness of others’ workspace activity [15], cognition distributed across people and artifacts [18], and knowledge of who knows what [49]. Team-automation research motivates observability and directability [21, 39], while trust research emphasizes appropriate reliance [24]. We hypothesize that private Human-AI sessions can weaken these conditions when teammates cannot inspect the context; these sources do not establish that causal link.

Falsifiable form. Teams using AI under time pressure will show lower cross-person communication density and higher within-person AI interaction density than comparable teams without AI under equivalent pressure.

Autonomous production: AI-AI limit case

The autonomous pipeline produced a complete draft on the same problem statement during the remaining session time, without human intervention. This demonstrates end-to-end production capacity, not superior research quality: neither its artifact nor the team artifacts received independent quality scores. The run is a mechanism probe, not a control condition or a statistical comparison. It illustrates the generation–evaluation asymmetry with in-session human evaluation removed, rather than its cost reduced. Verification is deferred to downstream readers unless an explicit review process is provided. Autonomous execution combined with inspectable team review remains compatible with Human-AI-Human (§5.3); this run did not test that configuration.

Convergence

Three independent teams, same clock, same tools, no shared communication channel, surfaced the same failures (Figure 3). We treat this as within-session convergence, not independent experimental replication or statistical generalization. Shared organizational context and session conditions limit the inference.

From the closed room to the enterprise floor. Participants described the session as resembling their normal week “with the time budget compressed.” This motivates, but does not establish, relevance to enterprise research. Time pressure, role clarity, workflow maturity, and missing shared infrastructure may jointly explain the patterns. Faster generation could increase evaluation demands, while more reliable models or practiced workflows could reduce them. Controlled field studies must distinguish these possibilities. Section 5 turns the observations into a testable design agenda.

The Human-AI-Human design category

Participants named the missing tool before we did: “a better coordination tool so that context from different team members can be used to make better decisions from Claude”; “a way to share sessions to have a final consolidation”; two teams asked for Claude Code to be “integrated to the tool we provided” to remove copy/paste; every team asked to take parallelism out and run stages sequentially.

The category, defined

Here, Human-AI denotes individual-centered assistance [47, 25, 17]. Human-first drafting, AI-first proposals, and repeated refinement all fit this topology. AI-AI denotes autonomous runs without in-loop human evaluation [29, 51]; it defers verification rather than eliminating its cost. This labels an execution mode, not every use of a framework: AutoGen and Agent Laboratory also support human input [50, 38].

Human-AI-Human makes the team the unit of interaction and decision-making. AI selectively connects researchers’ working contexts, preserves provenance, exposes conflicts, and routes claims for collective evaluation before consequential decisions. This is an interaction topology, not a fixed sequence: either humans or AI may initiate, and interaction may be single-pass or iterative. Unlike sharing only a final document, the category makes assumptions, intermediate evidence, and acceptance decisions inspectable across people. Existing collaboration platforms can implement it through shared memory, expertise-aware routing, and team-visible decision gates. Key challenges include permission-respecting context sharing, multi-user reasoning, and avoiding information overload.

Design requirements as a falsifiable agenda

Table 3 derives seven modular hypotheses from observed gaps (§ 4), not validated prescriptions. They can span assistants, workspaces, and workflows; no single tool must implement all seven. Their benefits and burdens require testing separately and in combination.

ID	Requirement	Failure addressed	Falsifiable prediction
R1	Shared context substrate across sessions and tools	Idle Claude Code session; cross-tool copy/paste	Tools sharing a substrate show higher cross-person reuse of AI outputs
R2	Conflict and redundancy detection across workstreams	Parallel-session fragmentation in all teams	Tools that flag converging or contradictory sessions reduce late integration failures
R3	Forced shared evaluation checkpoints	27/33 dispositions unrecorded (Figure 2)	Team-level forcing functions raise the share of AI outputs passing team-visible evaluation
R4	Disposition logging team-visible by default	82% blank rate on external observer log under load	Built-in logging produces a more complete record than external annotation
R5	Role-aware routing of AI output to relevant expertise	Team 2 acronym-laden hypothesis summary	Routing to expertise-matched teammates with a translation pass reduces incomprehensibility reports
R6	Decision-ownership tracking	First 10 min: research question silently delegated to Claude	Tools recording decision ownership produce more contested mid-session course corrections
R7	Epistemic load management	Consumption bottleneck across all teams	Throttling unreviewed AI output (batching, ack gates) reduces unevaluated content in the artifact

The minimum initial intervention combines shared cross-person context (R1) and collective evaluation checkpoints (R3). Compare it against existing tools with matched facilitation, measuring cross-person reuse, reviewed-output share, integration failures, and unreviewed content. This tests the combination, not each component’s effect.

Alternative positions and boundary cases

Existing platforms may suffice. AutoGen [50] and MetaGPT [19] provide agent coordination. Combined with shared editing, such frameworks could support Human-AI-Human if they connect working contexts and collective decisions. The distinction is functional, not a claim that a separate product is necessary.

Why not autonomous execution and shared review? Automated discovery and experimentation [37, 5] can feed one shared synthesis, reducing private silos and immediate attention costs. A summary alone may hide consequential assumptions. This becomes Human-AI-Human when provenance, uncertainty, alternatives, and evidence remain inspectable and the team can challenge or revise decisions at consequential checkpoints. Autonomous execution is compatible with collective agency.

Better workflows may suffice. Clearer roles, facilitation, practice, improved prompts, or more time could mitigate the observed problems; our pilot does not exclude this explanation. Follow-up studies should compare these improvements with shared-context AI, rather than assume tooling is the sole cause. Individual productivity findings [36] do not settle which approach best supports collective evaluation.

Limitations and what would change our minds

This single-organization, opportunistic pilot involved twelve task-performing researchers in a seventeen-person, 100-minute session. It generates hypotheses, not causal, prevalence, or effect-size estimates. Without a no-AI control, it cannot separate AI-specific amplification from pressure, role clarity, or workflow maturity. Neither team artifacts nor the single autonomous draft received independent quality scores. Missing dispositions may reflect both private decisions and logging limitations. The tool snapshot is time-specific; fundamental research, longer projects, and non-research tasks remain untested settings.

Longitudinal, multi-organization studies should vary time budgets, workflow training, repeated sessions, and individual/shared/no-AI conditions, recording baseline AI proficiency. Outcomes should include communication, integration and verification time, collective review, and independently assessed research quality. Recovery through practice or facilitation alone would weaken the tooling explanation; no added benefit from shared-context systems would challenge the proposed intervention. The present diagnostic study neither implemented nor validated the agenda in § 5.2.

Conclusion

Our pilot surfaced recurring consumption and integration difficulties when researchers used personal AI tools under deadline pressure. It does not show that AI caused every difficulty or reduced research quality. It motivates a shift in design and evaluation: from individual throughput alone toward inspectable collective reasoning. Human-AI-Human names this team-centered topology, compatible with existing platforms and autonomous execution. Shared context and collective evaluation checkpoints offer a concrete starting point for testing whether better team processes translate into better research.