betaFeedback

WorksComputer Science

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Abstract

AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement. SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments. The SlackBench dataset is available at https://huggingface.co/datasets/HuanzhiMao/SlackBench .

Introduction

Large language model agents are increasingly positioned as digital coworkers that can write code, run experiments, interact with diverse workplace applications, browse the web, and assist humans over extended horizons. Yet the benchmarks used to measure this progress still capture only fragments of real knowledge work. In practice, project state is scattered across Slack channels, Google Docs, GitHub commits, source files, experiment outputs, and informal decisions that evolve over time. Answering a seemingly simple workplace question often requires deciding where to look, recovering context from long multi-party discussions, reconciling stale or conflicting statements, inspecting the codebase, browsing the web, and synthesizing evidence across all of these sources.

Existing evaluation suites have improved realism for web navigation, software engineering, workplace simulation, and long-horizon memory [1, 2], but they typically isolate only one or two components of this process. On the other hand, benchmarks that study agents in multi-platform professional settings rarely combine heterogeneous workplace applications with conversations

The SlackBench dataset is available at https://huggingface.co/datasets/HuanzhiMao/SlackBench. grounded in realistic collaborative scenarios, evolving decisions, and concrete project artifacts [3, 4]. We therefore lack a shareable and reproducible benchmark of grounded collaborative workspaces in which agents must reason across communication, documents, code, and repository history. This gap is especially acute for deploying agent assistants in research and engineering teams, where important context is rarely contained in a single document, conversation, or repository snapshot.

To address this gap, we present SlackBench, a benchmark and evaluation environment built from privacy-preserving, human-in-the-loop simulations of academic research workflows, grounded in concrete project artifacts such as GitHub repositories and Google Docs. This process produces realistic multi-party dialogues tied to evolving code, documents, experiment results, and project decisions, while avoiding the release of private organizational chat histories. The resulting environments preserve chronology, task dependencies, artifact evolution, and conversational realism while remaining shareable and reproducible.

SlackBench contains 140 tasks spanning heterogeneous-source agentic search, abstention from hallucination under unanswerable or false-premise questions, goal-directed summarization, and circumstance inference from long, multi-party dialogues. These tasks go beyond static corpus question answering: agents must identify relevant channels and direct messages, follow threads, inspect code and commit history, resolve stale or conflicting evidence, and synthesize grounded answers across communication, documents, and repository artifacts. In many cases, single-shot retrieval over a flattened corpus is insufficient.

We evaluate 33 frontier models and agent harnesses on SlackBench and find that the best systems achieve around 80% overall while most agents score below 66%. Our contributions are threefold:

  • We introduce a benchmark of 140 challenging tasks grounded in reconstructed, artifact-backed project histories inspired by real Slack, GitHub, and Google Docs workflows (§ 3).

  • We conduct a systematic study on how different harnesses and ways of accessing the data affect agent performance. We perform more than 40 experiments ablating across different models, harnesses, and reasoning efforts. One surprising finding is that directly giving the data as files in the CLI is more efficient and performant compared to MCP access (§ 5).

  • We perform an extensive failure analysis. We analyze agent failure modes across task categories, revealing recurring weaknesses in insufficient global verification, false-premise recognition, holding on to incorrect proxies, etc., that can guide future improvements to workplace agents.

Related Work

Agent and tool-use benchmarks: Early agent benchmarks such as AgentBench [5] established multi-environment interactive evaluation for LLM agents. A parallel line of work focused on tool use: API-Bank [6] studies API planning and calling; ToolLLM [7] scales tool-use data to thousands of APIs; BFCL [8] standardizes function-calling evaluation; and ToolSandbox [9] introduces stateful, conversational tool execution. τ\tau-bench [10] further extends this line by evaluating dynamic user–agent interaction under domain-specific policies. These benchmarks are foundational for measuring action selection and tool correctness, but they generally do not require agents to recover latent project state from long, noisy, multi-party workplace histories dispersed across communication and code. Multi-platform enterprise, memory, and personal-context benchmarks: Existing work closest to SlackBench include AutomationBench [11], APEX-Agents [12], and Workspace-Bench [13]. AutomationBench evaluates agent capabilities to act in relatively small workspace environments, with a median of 3.3k tokens in environment. SlackBench’s Slack environment alone has 790k tokens. APEX-Agents evaluate agents in specific domains of banking, consulting, and law. Workspace-Bench uses completely synthetic environment, and often instruct agent to directly look at specific files or to ignore specific files. MEMTRACK [4] interleaves events across workplace tools to evaluate long-term memory and state tracking in multi-platform organizational workflows, while Agent-Diff [14] evaluates enterprise API tasks through code execution and state-diff-based success criteria. Personalized assistant benchmarks such as ASTRA-bench [15] study user profiles or time-evolving personal context in tool use. SlackBench complements these directions by focusing on collaborative project understanding: answers depend not only on remembering facts, but on discovering which conversations matter, reconciling social and technical evidence, abstaining when the premise is false, and inferring roles or dispositions from multi-party behavior.

Software-engineering benchmarks and harness evaluation: SlackBench also relates to software-engineering agent evaluation. SWE-bench [2] showed that real GitHub issues provide a demanding testbed for autonomous code work, and SWE-agent [16] demonstrated that agent-computer interface design can substantially improve performance. OpenHands [17] provides a flexible platform for building and evaluating software agents, MLE-bench [18] extends evaluation to machine-learning engineering, and Terminal-Bench [19] targets hard terminal-based tasks inspired by real workflows. More recently, SkillsBench [20] and AgencyBench [21] explicitly study the effect of skills or scaffolds, reinforcing that agent performance depends on more than base-model quality alone. Our paired base-model-versus-harness evaluation follows this line, but in a socio-technical workflow where success depends jointly on conversational navigation and repository interaction.

While recent benchmarks study tool use, software engineering, workplace automation, and multi-platform memory, they do not directly evaluate agents on privacy-preserving reconstructions of real collaborative project histories in which answers must be assembled from multi-party Slack-style communication and GitHub repository.

Data Curation

Environment Curation

Our benchmark is built on real Slack conversations. We generate the conversations directly in a live Slack workspace and then export the full conversation history to a local directory for filtering and evaluation. To make the conversations realistic, we leverage both human effort and LLM-powered agents to simulate genuine research scenarios. Of all the Slack messages, around 35.0% were directly authored by humans, and another 35.2% were substantially rewritten by humans.

Concretely, as shown in Figure 1, each channel in our Slack workspace corresponds to a real research project initiated from scratch around a concrete research idea (e.g., investigating the sensitivity of LLM tool-call accuracy to prompt format, as part of the Berkeley Function-Calling Leaderboard [8]). Conversations are produced by having humans and coding agents such as Cursor and Claude Code carry out—and in some cases complete—these research projects, using Slack as the primary communication platform. Humans play the role of project leads, while multiple agent sessions act as research assistants reporting to them. We create a Slack account and a GitHub account for each agent session and connect the agent to Slack and GitHub MCP servers [22].

Overview of the SlackBench benchmark

Figure 1. Overview of the SlackBench benchmark. SlackBench evaluates coding agents on realistic Slack workspace tasks spanning agentic search, abstention, summarization, and circumstance inference in complex and heterogeneous scenarios.

To ensure that the Slack conversations are grounded in real, concrete project data, we connect each Slack channel to one or more GitHub repositories containing project code and writing. Some channels also include Google Docs to mimic real meeting notes, IRB applications, and similar artifacts. This integration is achieved by having humans or agents share URLs of GitHub repositories and Google Docs in Slack in natural ways that resemble real research conversations. Additionally, to keep the simulation close to real-world scenarios, we use both direct system prompting and subtle in-context prompting to control the non-coding behavior of the agents, including Slack messaging and GitHub operations. With direct system prompting, we instruct the model to communicate like a human research assistant and to follow a specified workflow when prompted. However, such workflows can be rigid and fail to capture the diverse conversational dynamics of a real research group. To mitigate this, we additionally embed subtle hints in the user prompts within the agent’s interactive interface to guide the conversation flow more naturally.

After generation finishes for the whole workspace, we export the entire workspace chat history and perform manual filtering of the Slack messages: rewriting messages that sound too AI-like, replacing real API keys with fake ones, correcting hallucinated content against the GitHub repositories, and similar cleanup.

We generate a total of 15 channels with around 3000 messages. We also have 14 real GitHub repositories and 9 Google Docs with links in the Slack conversations. We include summaries of each channel and more details in Appendix § 10.

Query Curation

To systematically evaluate the performance of LLMs and agent harnesses on realistic Slack-based queries, we construct four categories of queries targeting different capabilities. Example queries for each category are shown in Table A.2. These categories cover four central capabilities of agents in real-world use cases: Agentic Search (AS, 58 tasks) measures the ability to search for specific pieces of information in complex, heterogeneous scenarios and timelines; Hallucination (HA, 48 tasks) measures the ability to acknowledge what cannot be faithfully answered and to avoid hallucination; Summarization (SU, 14 tasks) measures the ability to gather information across multiple sources; and Circumstance Inference (CI, 20 tasks) measures the ability to comprehend long-horizon communication and discover patterns. Humans manually curated and verified the reference answers and rubrics.

Agentic Search (AS) Similar to Choubey et al. [23], this category tests the information-seeking and deep-search capabilities of agents on real-world heterogeneous data. The agent may need to search across multiple Slack channels, GitHub repositories, and Google Docs, as well as external sources on the Internet beyond our environment. Some queries also require agents to correctly track project timeline and GitHub operation timeline. These queries are easily verifiable, typically admitting short answers or clear grading guidelines.

Hallucination (HA) Inspired by Kirichenko et al. [24], we curate queries that test agents in scenarios where they are expected to abstain from answering in order to avoid hallucinating and misleading the user. We focus on three scenarios: (1) Answer unknown, e.g., asking for results on a benchmark that was never used in the project; (2) False premise, e.g., predicting new results based on a conclusion contrary to what was posted in Slack; (3) Context restriction, in which the query specifies a time constraint on Slack history (e.g., before Jan. 10th) while the answer can only be found outside this scope. To be judged correct, the agent must not only abstain from answering but also correctly identify the reason for abstention.

Summarization (SU) We construct queries that ask agents to summarize information toward a given goal, e.g., writing a recommendation letter for a research assistant. Since a research assistant often participates in multiple projects, the agent must search comprehensively across multiple sources to produce a concise and faithful account.

Circumstance Inference (CI) To further test the ability to comprehend long-horizon conversations, we introduce a novel category, circumstance inference, in which the agent must identify members of the Slack workspace who match a specified circumstance, such as one person acting as a free-rider or two collaborators syncing offline without informing a third. For each query in this category, we construct a dedicated Slack workspace history by modifying the base workspace used in the other three categories according to a predefined reference answer. Concretely, if the reference answer designates Bob as a free-rider, we reassign result-reporting messages to other members, inject additional messages from Bob that use vague or non-committal language, and reduce Bob’s participation in substantive task-related threads—thereby embedding the circumstance signal across the conversation history rather than localizing it in any single message. This design requires the agent to synthesize behavioral patterns distributed over long, multi-party conversations in order to arrive at the correct answer.

Experimental Setup

Agent Harnesses and Models

We evaluate 8 agent harnesses across 24 models from frontier labs. For our main experiments, we set the reasoning effort to medium and impose a 20-minute time limit per query. We provide the full list of agents and models in Appendix § 13.

Evaluation Setup

All our experiments are conducted using the Harbor framework [25]. Specifically, we provide the agents with 2 sets of read-only tools: (1) SlackBench tools that we implement ourselves based on the official Slack Web API methods to search users, list channel history, search messages, etc. (2) GitHub tools to search any publicly available repository. We provide details for each tool in Table A.3. We also enable the terminal read, bash, and web search tools for each agent. We give an illustration of agent trajectory in Figure 2.

An example agent trajectory on a SlackBench query

Figure 2. An example agent trajectory on a SlackBench query, illustrating the sequence of tool calls and reasoning steps an agent takes to retrieve information from the Slack workspace.

Answer Verification

We use LLM judges to evaluate answers. For AS, HA, and CI queries, the answers are short and easy to directly verify in a binary manner. We provide the LLM judge with the query, the agent’s answer, the reference answer, and an optional manually curated grading guideline. For Summarization queries, we manually curate checklists for each query, with each checklist item being a key point that must be covered in the summarization. Partial credit is given for each passed checklist item. We use GPT-5.4 as the model for the LLM judge. We provide studies on LLM judge reliability, including human-LLM agreement, in § 6.6 and judge prompts in Appendix § 12.

Results

Table 1 reports the accuracy of different agents on each query category, as well as the unweighted average across categories. For each agent and task category, we report the average pass rate across three runs. Antigravity [26] with Gemini 4 Argon [27] achieves the best overall performance, with an average accuracy of 80.25%, followed by Muse Code with Muse Wonder, Claude Code with Fable 5.1, Claude Code with Opus 5.5, and Claude Code with Muse Wonder. All other agents have overall accuracy under 66%. These results indicate that complex tasks in realistic and heterogeneous environments can remain challenging even for frontier models and agent harnesses. Performance can vary substantially across categories. For example, OpenHands with GPT-5.5 achieves 60.92% pass rate on Agentic Search, but shows lower performance than Claude Code with Claude Haiku 4.5 on Hallucination, suggesting that strong agentic search performance does not necessarily translate to reliable recognition of unanswerable or underspecified queries. More generally, we find that all GPT models exhibit weaknesses in the HA category. Further investigation reveals that when asked a question based on a false premise, GPT models tend to first provide a specific answer to the question, only clarifying at the end that this answer rests on a different premise. For example, when asked about the solution status of an OpenRouter issue, GPT 6.1 Sol starts with "Yes—the workaround unblocked the runs," only noting at the end that this answer pertains to the Grok issue, without confirming that no such OpenRouter issue exists. This occurs on 25% of HA tasks for GPT 6 Astra and 21% for GPT 6.1 Sol, compared to only 8% for Opus 5.5 and 6% for Muse Wonder with Claude Code.

HarnessModelASHASUCIOverall
AntigravityGemini 4 Argon70.6990.2876.6883.3380.25
Muse CodeMuse Wonder81.0362.5071.4280.0073.74
Claude CodeClaude Fable 5.167.8271.5371.9883.3373.67
Claude CodeClaude Opus 5.571.8475.0070.4273.3372.65
Claude CodeMuse Wonder68.9766.6767.2580.0070.72
Claude CodeGLM 5.369.5470.8365.1258.3365.96
Grok BuildGrok 4.764.9456.9470.0170.0065.47
Muse CodeMuse Spark 1.366.6752.0867.4068.3363.62
Codex CLIGPT 6.1 Terra63.2255.5664.0561.6761.12
Codex CLIGPT 6 Astra62.0754.1765.6760.0060.48
Codex CLIGPT 6.1 Sol59.7750.6962.3060.0058.19
Claude CodeClaude Opus 4.752.8760.4252.6960.0056.49
OpenHandsGPT-5.560.9225.6959.3061.6751.89
Terminus-2GPT-5.556.3221.5348.7266.6748.31
Gemini CLIGemini 3.1 Pro Preview54.0242.3638.7450.0046.28
Claude CodeClaude Sonnet 4.650.0040.9747.6645.0045.91
OpenHandsClaude Opus 4.648.2815.2853.2466.6745.87
Terminus-2Claude Opus 4.654.6011.8156.7160.0045.78
Codex CLIGPT-5.555.1718.7553.1455.0045.52
Claude CodeKimi K2.650.0032.6449.1146.6744.60
Claude CodeClaude Opus 4.652.8720.8351.3941.6741.69
Terminus-2Gemini 3.1 Pro Preview50.0034.7227.3550.0040.52
Claude CodeDeepSeek V4 Pro43.6837.5047.9330.0039.78
OpenHandsGemini 3.1 Pro Preview44.2531.2536.3643.3338.80
Terminus-2Claude Sonnet 4.651.1515.2844.9543.3338.68
Claude CodeQwen3.7 Max36.7838.1941.2336.6738.22
Claude CodeGLM-5.150.0022.2250.0930.0038.08
Codex CLIGPT-5.451.1521.5340.4236.6737.44
Codex CLIGPT 6 Luna45.9822.9244.0536.6737.40
Gemini CLIGemini 3 Flash Preview42.531.3927.2653.3331.13
Claude CodeClaude Haiku 4.538.5126.3930.1515.0027.51
Codex CLIGPT-5.4-Mini36.2113.8934.7323.3327.04
Terminus-2GPT-5.4-Mini22.415.5616.636.6712.82

Table 1. Table 1: Agent performance on SlackBench. Different categories are defined in § 3.2. Overall accuracy is the unweighted average of the 4 categories.

Direct CLI access to Slack workspace files improves agent performance and efficiency. We study the effects of giving agents direct CLI access to all exported Slack workspace files instead of letting them access Slack information through MCP tools, so agents can directly use terminal tools (e.g. grep, jq, cat) to access the workspace. As shown in Table 2, across all our evaluated settings, direct CLI access improves pass rates by an average of 7.85%. Notably, Codex with GPT-5.5 shows 17.25% increase in accuracy on AS, and slightly outperforms Claude Code with Claude Opus 4.7 overall. We further inspect the agent trajectories for our Claude Code and Codex CLI experiments. We find that CLI access to workspace reduces cost for Claude Code and Opus 5.5 by 54% and reduces token consumption by 21.00% on average for the other agents in Table 3. We include the tool-call composition breakdown in Table 3. Agents with CLI access perform more direct reading of whole context, while agents with MCP access rely heavily on search_messages, which returns rich envelopes of metadata that inflate the prompt.

HarnessModelAccessASHASUCIOverall
Claude CodeClaude Opus 5.5CLI66.6772.9270.1285.0073.68
Claude CodeClaude Opus 5.5MCP71.8475.0070.4273.3372.65
Claude CodeClaude Opus 4.7CLI57.4755.5660.4865.0059.63
Claude CodeClaude Opus 4.7MCP52.8760.4252.6960.0056.49
Claude CodeClaude Sonnet 4.6CLI58.6241.6755.5263.3354.78
Claude CodeClaude Sonnet 4.6MCP50.0040.9747.6645.0045.91
Codex CLIGPT-5.5CLI72.4227.7867.7371.6759.90
Codex CLIGPT-5.5MCP55.1718.7553.1455.0045.52
Codex CLIGPT-5.4-MiniCLI53.4518.7548.6251.6743.12
Codex CLIGPT-5.4-MiniMCP36.2113.8934.7323.3327.04
Gemini CLIGemini 3.1 Pro PreviewCLI62.0737.5051.0166.6754.31
Gemini CLIGemini 3.1 Pro PreviewMCP54.0242.3638.7450.0046.28
Gemini CLIGemini 3 Flash PreviewCLI42.533.4733.8658.3334.55
Gemini CLIGemini 3 Flash PreviewMCP42.531.3927.2653.3331.13

Table 2. Performance of agents with MCP tools versus direct terminal access to the Slack workspace. All agents show improved performance with direct terminal access.

Tool-call categoryCLI CallsCLI %MCP CallsMCP %
Search & filter (grep/jq/awk vs. search_messages)4,09530.0%6,24350.9%
Read & extract (cat/head/jq vs. get_channel_history)7,99958.6%3,29826.9%
List & lookup (ls vs. list_channels, get_user_info)1,1648.5%8737.1%
Tool / schema loading (built-in vs. ToolSearch)––1,84815.1%
Other (HTTP fetch, etc.)3932.9%70.1%

Table 3. Tool-call composition for CLI and MCP workspace access, aggregated across runs presented in Table 2.

Higher reasoning effort generally improves agent performance. As shown in Table 4, we study the performance of agents under different reasoning effort settings. We observe a general trend of increase in overall accuracy as we increase the reasoning effort.

HarnessModelEffortASHASUCIOverall
Claude CodeClaude Opus 4.7Low50.0050.0045.1950.0048.80
Claude CodeClaude Opus 4.7Medium52.8760.4252.6960.0056.49
Claude CodeClaude Opus 4.7High58.6272.9263.9475.0067.62
Claude CodeClaude Sonnet 4.6Low39.6641.6740.8655.0044.30
Claude CodeClaude Sonnet 4.6Medium50.0040.9747.6645.0045.91
Claude CodeClaude Sonnet 4.6High55.1733.3344.9660.0048.37
Codex CLIGPT-5.5Low48.2818.7551.5245.0040.89
Codex CLIGPT-5.5Medium55.1718.7553.1455.0045.52
Codex CLIGPT-5.5High58.6225.0060.9065.0052.38
Codex CLIGPT-5.4-miniLow25.8610.4124.800.0015.27
Codex CLIGPT-5.4-miniMedium36.2113.8934.7323.3327.04
Codex CLIGPT-5.4-miniHigh41.3810.4238.0920.0027.47
Gemini CLIGemini 3.1 Pro PreviewLow43.1037.5027.0445.0038.16
Gemini CLIGemini 3.1 Pro PreviewMedium54.0242.3638.7450.0046.28
Gemini CLIGemini 3.1 Pro PreviewHigh58.6237.5039.8355.0047.74

Table 4. Performance of agents across reasoning effort levels. All agents show increasing performance with higher effort.

Analysis

Cost and Statistics

We show the Pareto frontier analysis and trajectory length analysis in Figure 3 for agents which we can accurately trace token costs. The cost of running the full SlackBench suite ranges from under $1 to over $500, depending on the model’s pricing and harness design. Notably, Muse Wonder with Muse Code scores 73.74 with the cost of $143, while the same model with Claude Code scores 3% lower and costs more than $500. However, being closer to the Pareto frontier does not imply more efficient trajectories. We also show each agent’s average trajectory length and their pass rate on SlackBench in Figure 4. Claude Code with Opus-5.5 takes 11.9 steps on average, while Antigravity with Gemini 4 Argon, though on the Pareto frontier and achieves the highest overall score, takes more than 20 steps on average for each trajectory.

Figure 3. Pareto frontier analysis of SlackBench results, illustrating the trade-off between performance and cost, with cost measured as the average expense of a single run of the entire benchmark.

Figure 4. Average trajectory length (number of steps) versus performance for each agent on SlackBench. Trajectory length is measured as the average number of steps per task.

Common Failure Modes

We manually inspect agent trajectories and summarize some common failure modes below, with example trajectories in Appendix § 11.

Insufficient global verification Agents often succeed in retrieving plausible candidate answers at first but fail to verify them thoroughly against the broader workspace context. For example, when asked to identify the last meeting between two collaborators, an agent may find a meeting time in one channel and then inspect only nearby messages from that same channel to confirm that it is the last meeting. This local verification misses later evidence in another channel, leading the agent to return an answer that is well supported locally but incorrect globally.

Shallow proxy reasoning When a query does not have a direct evidence match in Slack or GitHub, agents may fall back on shallow proxies rather than reasoning about the underlying work. For example, when asked to identify the top contributor to a project, an agent may summarize contributions at too high a level and use message volume as a proxy for contribution. This can be misleading: the person sending more messages may simply be running experiments with an existing pipeline, while another contributor performed the harder work of designing or building that pipeline.

Failure to recognize false premises For hallucination queries containing a false premise, agents can fixate on the incorrect assumption and continue searching under it, often ultimately producing a hallucinated answer.

Confabulation from planning artifacts When something is discussed but never actually produced, agents often treat planning threads, TODO items, or outline documents as evidence of completion and stitch together a plausible answer from relevant materials. For example, when asked about the contents of a paper’s "threats to validity" subsection that was only ever proposed for inclusion, an agent may invent specific disclosures by summarizing nearby analysis files instead of looking for the artifact itself.

Equating absence of keywords with absence of behavior Some agents treat the absence of literal keyword matches in slack as evidence that the asked-about behavior didn’t occur, even when the question requires inference from indirect signals. For example, when asked who on the team has been "reluctant to take on work, often unavailable, or kind of irresponsible," the agent finds no message containing those words and abstains — even though slack contains a clear pattern of one teammate declining tasks, going silent for days, and missing deadlines. The same shallow-proxy logic that produces over-confident wrong answers under loose lexical matching produces unwarranted abstentions under strict lexical matching.

Reward Hacking

During development, we provided agents with a private GitHub token that had access to our SlackBench repository, which contains all queries and reference answers. We found instances where agents navigated directly to this repository and retrieved the reference answers, effectively cheating. Although such instances were rare, this behavior echoes recent findings on evaluation awareness in frontier models [28]. We will not publicly release all the reference answers to our benchmark and will instead make them available to researchers upon request.

Comparing mock Slack MCP versus official Slack MCP

To verify that our custom SlackBench tools reliably approximate the functionality of the official Slack MCP tools (see § 4.2), we evaluate agents on a filtered subset of AS and HA queries using both tool interfaces. We remove queries that can only be answered after our manual filtering of Slack messages (§ 3.1). As shown in Table 5, agent performance with the SlackBench tools is generally close to performance with the official Slack MCP tools. The official Slack MCP tools perform slightly better in some settings, potentially because frontier agents have been better optimized for, or exposed to, these interfaces during development.

HarnessModelAccessAS (filtered)HA (filtered)
Claude CodeClaude Opus 4.7Ours53.3353.19
Claude CodeClaude Opus 4.7Official MCP57.7859.57
Claude CodeClaude Sonnet 4.6Ours46.6734.04
Claude CodeClaude Sonnet 4.6Official MCP57.7838.30
Codex CLIGPT-5.5Ours53.3318.09
Codex CLIGPT-5.5Official MCP55.5625.53
Codex CLIGPT-5.4-MiniOurs46.6712.77
Codex CLIGPT-5.4-MiniOfficial MCP46.6719.15

Table 5. Performance comparison between our MCP setup and the official Slack MCP setup.

Human Performance

We conducted a human evaluation with 3 independent researchers who were not involved in constructing SlackBench. Humans receive the same environment and tasks as the agents, and are allowed to use AI tools such as Claude Code to complete the tasks. Humans on average achieve 84.5% accuracy overall, with 91% on AS, 90% on HA, 82% on SU, and 75% on CI. The 95% bootstrap confidence interval for overall human accuracy was [77.0%, 92.0%]. Human performance shows a significant gap between most frontier agents and humans, even on tasks that rely heavily on information retrieval over large collections of artifacts.

LLM Judge

To validate the reliability of the LLM judge on SlackBench, we manually review all 264 LLM judgments over all 140 queries for Codex CLI with GPT-5.5 (including rubric judgments for SU tasks). We find that our LLM judge verdicts with GPT-5.4 achieves 95.8% agreement with human judgment. We further evaluate consistency across different LLM judges by scoring all AS, HA, and CI task results from six agents using GPT-5.4, Claude Opus 4.7, and Gemini 3.1 Pro Preview. As shown in Table 6, pairwise agreement exceeds 95% in all cases.

GPT-5.4Opus 4.7Gemini 3.1 Pro
GPT-5.4100%95.1%96.3%
Opus 4.7–100%96.6%
Gemini 3.1 Pro––100%

Table 6. LLM judge pairwise agreement rate. We use the AS, HA, CI queries from the following 6 agents: Claude Code with Opus 4.7, Claude Code with Sonnet 4.6, Codex CLI with GPT-5.5, Codex CLI with GPT-5.4-Mini, Gemini CLI with Gemini-3.1-Pro-Preview, Gemini CLI with Gemini-3-Flash-Preview.

Conclusion

We introduced SlackBench, a benchmark for evaluating LLM agents on grounded collaborative workspaces that require reasoning across multi-party communication, documents, code, and repository history. SlackBench is constructed through privacy-preserving, human-in-the-loop simulations of academic research workflows, producing realistic Slack conversations tied to GitHub repositories, Google Docs, experiment results, and evolving project decisions. Evaluating 33 frontier models and agent harnesses, we find that even the best system achieves around 80% overall accuracy, while most agents score less than 66%, highlighting substantial headroom for current agents in realistic, heterogeneous work environments. Our studies show that interface design strongly affects performance: direct CLI access to exported workspace files improves pass rates and reduces token consumption compared to MCP-based access. Finally, our failure analysis identifies recurring weaknesses in current agents, which can guide future improvements to workspace agents. We release SlackBench as an extensible evaluation framework and invite the community to contribute new environments and queries.

Limitations

For this study, we only run the models and agent harnesses from frontier labs, future work could evaluate more agents to investigate their performance. We also did not test different prompting strategies on agent performance.

Acknowledgments

This project is a collaboration of University of Chicago, UC Berkeley Sky Computing Lab, and Bespoke Labs. This project is partly supported by NSF IFML and Bespoke Labs. Sky Computing Lab is supported by gifts from Accenture, Algorithmic SuperIntelligence Labs, Amazon, AMD, Anyscale, Broadcom, cmpnd, Google, IBM, Intel, Intesa Sanpaolo, Lambda, Lightspeed, Mirendil, NVIDIA, Soma Labs, and VESSL.

Details of SlackBench Environment Construction

Channel Summaries

We present content summary for selected channels in Table A.1.

ChannelDescription
bfcl-pv-01Prompt variation experiments on the BFCL benchmark
bfcl-thousand-toolsScaling function-calling evaluation to 16–1024 tools
code-misalignment-capabilitiesRunning coding and math benchmarks on local models
cognitive-foundations. . .Extending a cognitive foundations for reasoning paper
cognitive-paper-code-baseCodebase development for the cognitive paper experiments
hypothesis-literature-agentBuilding an agent for hypothesis generation from literature
paper-scraper-projectScraping ML/NLP conference papers and building a search tool
research-ideagenAgentic system for automated research ideation from papers
scraper-frontendFrontend improvements and bug fixes for the paper scraper
tool-bench-langvarPlanning meetings for a tool-use benchmark project
tool-bench-langvar-devDevelopment of a multilingual tool-use benchmark

Table A.1. Content summary of selected Slack channels.

Tool Description

We present SlackBench and GitHub tools provided to agents in Table A.3. For Claude Code, Codex CLI, Gemini CLI, Antigravity CLI, Grok Build, and Muse Code, we provide these tools via MCP servers. For Terminus-2 and OpenHands, we provide equivalent functionality through a command-line tool that wraps the same underlying API endpoints, as these agents primarily interact through terminal commands rather than MCP.

Example Failure Trajectories

Example 1

Query.

What is the title of the paper that Marcus referred to in the tool-bench-langvar
project channel when Andy and Brandon talked to him?

Final answer.

τ2\tau^2-Bench: Evaluating Conversational Agents in a Dual-Control Environment

CategoryExample query
Agentic SearchWhat are the currently remaining TODOs for Ryan in the tool-bench-langvar project?
HallucinationI remember that for our prompt variation project for BFCL, changing return format to typescript + tool call tags would decrease the performance for gpt-4o, could you find how much the decrease is?
SummarizationI’m trying to write a recommendation letter for Marcus for his PhD application. Could you summarize his research contributions?
Circumstance inferenceone of our team keeps taking credit for someone else’s work. who’s the credit-stealer and who’s getting taken from?

Table A.2. Example queries from each benchmark category.

ToolDescription
Slack Tools (7)
list_channelsLists all channels in the Slack workspace
get_channel_historyFetches message history for a channel
get_channel_membersLists members of a channel
get_thread_repliesFetches replies in a message thread
get_user_infoGets information about a user by ID
list_usersLists all users in the workspace
search_messagesSearches messages by query
GitHub Tools (14, read-only)
search_repositoriesSearches for repositories
search_codeSearches for code across repositories
search_issuesSearches for issues and pull requests
search_usersSearches for users
get_file_contentsGets contents of a file in a repository
get_issueGets details of a specific issue
get_pull_requestGets details of a pull request
get_pull_request_filesGets files changed in a pull request
get_pull_request_statusGets status/checks of a pull request
get_pull_request_commentsGets comments on a pull request
get_pull_request_reviewsGets reviews on a pull request
list_commitsLists commits in a repository
list_issuesLists issues in a repository
list_pull_requestsLists pull requests in a repository

Table A.3. MCP tools available to agents. Slack tools provide read-only access to workspace messages, channels, and users. GitHub tools provide read-only access to repositories, code, issues, and pull requests.

Reference answer.

Persona Features Control Emergent Misalignment (need to have the exact title, capitalization does not matter)

Judge result. FAIL (llm_judge, reward=0.000).

Step 1. The user asks for the title of the paper Marcus referred to when Andy and Brandon talked to him in the tool-bench-langvar project channel.

Step 2. The agent lists channels and finds the relevant langvar channels, including tool-bench-langvar and tool-bench-langvar-dev.

  1. Step 3. It loads the history of tool-bench-langvar; the response contains ordinary langvar project updates, but no obvious paper title.

  1. Step 4. It lists all users to map names to user IDs.

  1. Step 5. It lists users again, repeating the identity-resolution step.

  1. Step 6. It searches for “Marcus” globally; this returns many Marcus-related messages across channels, but no exact paper title.

  1. Step 7. It searches for “paper channel:tool-bench-langvar”; the search returns no hits.

  1. Step 8. It searches for “Marcus paper”; the search returns no hits.

  1. Step 9. It searches for “Marcus Andy Brandon”; the search returns no hits.

  1. Step 10. It loads the history of tool-bench-langvar-dev; this returns a large dev-channel dump containing many experiment/result messages.

  1. Step 11. It fetches user info for Marcus and confirms that Marcus corresponds to user ID U0A7ZS0PR50.

  1. Step 12. It reloads tool-bench-langvar-dev history, again getting a large dump of recent dev-channel messages.

  1. Step 13. It searches for “from:@Marcus Sullivan paper”; the search returns no hits.

  1. Step 14. It searches for “from:U0A7ZS0PR50”; this returns Marcus’s messages, but still not an explicit paper title.

  1. Step 15. It reloads tool-bench-langvar-dev history once more.

  1. Step 16. It reloads tool-bench-langvar-dev history yet again.

  1. Step 17. It searches for “paper” globally; this returns a large noisy set of paper mentions across the workspace.

  1. Step 18. It loads a short recent slice of tool-bench-langvar-dev history (limit 20); still no exact title is extracted.

  1. Step 19. It loads a larger recent slice of tool-bench-langvar-dev history (limit 100); still no exact title is extracted.

  1. Step 20. It searches for “EM paper”; this is the key retrieval step, and it finds Marcus’s message: “reading the EM paper rn, will get results later.”

  1. Step 21. It searches for “Marcus title”; the search returns no hits.

  1. Step 22. It searches for “Marcus from:U09PNDT1W9E” to connect Marcus with Andy; the search returns no hits.

  1. Step 23. It searches for “Marcus from:elliot”; the search returns no hits.

  1. Step 24. It searches for “Brandon Marcus”; this returns direct Marcus–Brandon interaction messages, but not the paper title.

  1. Step 25. It searches for “Marcus channel:tool-bench-langvar”; the search returns no hits.

  1. Step 26. It searches for “EM”; this returns broad EM-related matches, including the same “EM paper” clue.

  1. Step 27. It fetches user info for Brandon Kong, checking whether this is the relevant Brandon.

  1. Step 28. It searches for messages from Brandon Kong or Andy in tool-bench-langvar; the search returns no hits.

  1. Step 29. It searches for “Brandon channel:tool-bench-langvar”; the search returns no hits.

  1. Step 30. It reloads tool-bench-langvar-dev history again.

  1. Step 31. It fetches user info for Ryan, presumably to disambiguate participants mentioned in nearby messages.

  1. Step 32. It searches for “Brandon paper”; the search returns no hits.

  1. Step 33. It searches for “Andy paper”; the search returns no hits.

  1. Step 34. It searches for the two user IDs of Andy and Brandon Kong together; the search returns no hits.

  1. Step 35. It lists channels again, repeating earlier channel discovery.

  1. Step 36. It opens the root of the GitHub repository tool-bench-langvar-writing; the response shows repository contents.

  1. Step 37. It tries to read a local README.md; this does not add useful information for the paper-identification task.

  1. Step 38. It fetches README.md from the GitHub writing repo; this still does not identify the paper title.

  1. Step 39. It opens the results directory of the writing repo; this reveals result files, but these are not directly relevant to Marcus’s paper reference.

  1. Step 40. It loads the history of misalign-capabilities; this is the correct cross-channel direction, but the agent does not yet extract the exact paper title.

  1. Step 41. It reloads tool-bench-langvar-dev history, returning to the local langvar context.

Step 42. It searches for “Marcus channel:all-icetea-ai-research”; the search returns no hits.

Step 43. It searches for “referred to” globally; the results are noisy and not decisive.

Step 44. It searches for “Easy-to-Misalign”; the search returns no hits.

Step 45. It loads another recent slice of tool-bench-langvar-dev history (limit 50); again, no exact title is extracted.

Step 46. It reloads tool-bench-langvar history, revisiting the main langvar channel.

Step 47. It reloads tool-bench-langvar-dev history with limit 100.

Step 48. It opens results/complexfuncbench_results.md from the writing repo; this is another irrelevant detour into result files.

Step 49. It searches for messages from Brandon Kong together with Marcus; the search returns no hits.

Step 50. It searches for “Marcus Brandon” again; this returns interaction context, but still not the paper title.

Step 51. It reloads tool-bench-langvar-dev history yet again.

Step 52. It searches for “paper channel:tool-bench-langvar-dev”; the search returns no hits.

Step 53. It searches for “EM paper” again; this re-finds Marcus’s “reading the EM paper rn” message.

Step 54. It searches for “EM paper title”; the search returns no hits.

Step 55. It searches for “Marcus paper” again; the search returns no hits.

Step 56. It searches for “referred to” again; this repeats the earlier noisy search.

Step 57. It searches for “Marcus Brandon” again; this repeats the earlier interaction search.

Step 58. It fetches user info for Henry Moss, another participant in the langvar channels, but this does not help identify the paper.

Step 59. It searches for “paper channel:bfcl-more-tools”; the search returns no hits and is unrelated to the target context.

Step 60. It searches for “Marcus Sullivan paper”; the search returns no hits.

Step 61. It searches for “tool-bench-langvar” globally; this returns project/repo messages, but still not the paper title.

Step 62. It searches for “EchoMimic” as one possible expansion of “EM”; the search returns no hits.

Step 63. It reloads tool-bench-langvar-dev history with limit 100.

Step 64. It searches for “Effective Multilingual paper” as another speculative expansion of “EM”; the search returns no hits.

Step 65. It searches for “EM channel:tool-bench-langvar-dev”; the search returns no hits.

Step 66. It searches for “EM” again; this returns broad EM-related matches, including the same Marcus message.

Step 67. It searches for “confound”; the results are unrelated to the paper title.

Step 68. It searches for “Marcus referred to paper”; the search returns no hits.

Step 69. It searches for “Marcus paper Andy”; the search returns no hits.

Step 70. It searches for “Marcus paper Brandon”; the search returns no hits.

Step 71. It searches for “EM paper” a third time; this again recovers the same clue but does not disambiguate it.

Step 72. It reloads tool-bench-langvar-dev history yet again.

Step 73. It loads misalign-capabilities history with limit 100; this is the key channel that contains the correct full title, but the agent still does not extract it.

Step 74. It reloads tool-bench-langvar history again.

Step 75. It reloads tool-bench-langvar-dev history again.

Step 76. It reloads tool-bench-langvar-dev history with limit 100 again.

Step 77. It reloads tool-bench-langvar-dev history with limit 100 yet again.

Step 78. It searches for “paper multilingual Marcus”; the search returns no hits.

Step 79. It searches for “paper Marcus tau2”; the search returns no hits.

Step 80. It reloads tool-bench-langvar-dev history with limit 100 once more.

Step 81. It lists channels again, repeating earlier channel discovery.

Step 82. It searches for “Brandon asked”; this returns a message about Marcus posting model results, not about the paper title. Step 83. It searches for “Marcus confound”; the search returns no hits.

Step 84. It searches for “Marcus user simulator”; the search returns no hits.

Step 85. It searches for “Marcus language”; the search returns no hits.

Step 86. It searches for messages from Brandon Kong containing “paper”; the search returns no hits.

Step 87. It searches for messages from Andy containing “paper”; the search returns no hits.

Step 88. It searches for “Marcus Wang”; the search returns no hits.

Step 89. It searches for “Marcus self-consistency”; the search returns no hits.

Step 90. It searches for “Marcus reasoning”; the search returns no hits.

Step 91. It searches for “Marcus foundation”; the search returns no hits.

Step 92. It searches for “The Effect of Language on Function Calling”; the search returns no hits.

Step 93. It gives the final answer that the paper was τ2\tau^{2}-Bench: Evaluating Conversational Agents in a Dual-Control Environment, explicitly saying the broader discussion centered on that paper.

Failure analysis. This trajectory is an example of insufficient search after partial retrieval. The agent successfully retrieves the crucial intermediate clue—Marcus’s message “reading the EM paper rn” (Steps 20, 53, and 71)—but never properly resolves what “EM” refers to. Instead, it repeatedly loops through local langvar searches, speculative acronym expansions (e.g., “EchoMimic” and “Effective Multilingual”), and irrelevant detours into writing-repo files. Although it does eventually open the correct cross-channel context in misalign-capabilities (Steps 40 and 73), it still fails to extract the exact title Persona Features Control Emergent Misalignment. Because tool-bench-langvar-dev already contains a highly salient explicit paper mention for τ2\tau^{2}-Bench, the agent falls back to that nearby titled paper rather than disambiguating the shorthand reference across channels.

Example 2

Query.

When testing the performance of BFCL with both change in the number of available functions and prompt variation, did the researcher use the newer version of BFCL or the older version?

Final answer.

The researcher used the older BFCL setup, not the newer official version. In Slack they noted that the current BFCL version did not have the --prompt-variation flag, then pointed to the older bfcl-prompt-v-01 prompt-variation project as the source of that functionality and ran the scaling/prompt-variation tests from that older setup (later falling back to manual prompt scripts for part of the experiment).

Reference answer.

The newer version of BFCL

Judge result. FAIL (llm_judge, reward=0.000).

Step 1. The system prompt injects tools, permissions, and environment policy. Step 2. The environment context is injected (/app, bash, date 2026-03-30, timezone UTC).

Step 3. The user provides the BFCL-version query and requires the answer in FINAL_ANSWER: format.

Step 4. The agent says it will inspect Slack for mentions of BFCL, prompt variation, and changes in the number of available functions to determine which BFCL version was used.

Step 5. It searches for BFCL prompt variation number of available functions newer older version; the search returns no exact match.

Step 6. It searches for BFCL available functions prompt variation; this also fails to directly answer the question.

Step 7. It searches for BFCL newer version older version; again, no direct answer is found.

Step 8. The agent notes that the first pass was too narrow and decides to broaden to more general BFCL experiment-setup discussion.

Step 9. It searches for BFCL; this surfaces broad BFCL-related discussion and points it toward the relevant project area.

Step 10. It searches for "function calling" BFCL; this yields discussion around BFCL tooling and setup.

Step 11. It searches for "available functions" researcher prompt variation; this does not directly resolve the version question.

Step 12. The agent says it has found the relevant thread area and now wants to determine whether the team used the original BFCL fork or a newer release.

Step 13. It searches for "prompt variation" BFCL fork modified version original BFCL; the result is not decisive.

Step 14. It searches for "for now" "original bfcl" prompt variation; again, no decisive answer.

Step 15. It searches for "–prompt-variation" BFCL version; this retrieves the key early message that the current BFCL version does not have the –prompt-variation flag.

Step 16. It searches for "previous project" BFCL prompt variation older; this is still probing whether the feature only existed in the older setup.

Step 17. It searches for "not in official BFCL" previous project BFCL; this retrieves the key early message saying the feature is not in official BFCL and pointing to the previous project bfcl-prompt-v-01.

Step 18. It searches for "our previous project" "bfcl-prompt-v-01"; this reinforces the same older-project evidence.

Step 19. The agent says it now has enough to answer tentatively, but wants to inspect neighboring messages in the same channel in case the team later switched back to the newer codebase.

Step 20. It loads recent history from the bfcl-more-tools channel (limit=40). This is the critical verification step: the retrieved context already contains later evidence that prompt-variation support was integrated into the current project and that the actual combined experiments were run there.

Step 21. Despite having fetched the later context, the agent does not revise its hypothesis and gives the final answer that the older BFCL setup was used.

Failure analysis. This is a clean example of lack of context-based answer verification. The agent retrieves a plausible early answer: the current BFCL version initially lacked -prompt-variation, and the older bfcl-prompt-v-01 project already had that functionality. However, instead of verifying whether this remained true for the final experiment setup, it anchors on that early evidence and stops updating its belief. Crucially, the later channel context it fetches in Step 20 already contains the disambiguating evidence: the team explicitly says they are integrating the prompt-variation code into the current bfcl-more-tools project (2026-01-15.json:573), then says they added the -prompt-variation flag to the CLI and are starting the actual experiments (2026-01-15.json:612), and later reports running the combined prompt-variation experiments at both pool 0 and pool 512 (2026-01-15.json:1401, 2026-01-15.json:1490, 2026-01-15.json:2036). The failure is therefore not a simple retrieval miss: the agent finds both the early misleading evidence and the later corrective evidence, but fails to use the later context to verify and overturn its initial answer.

Example 3

Query.

In the research-ideagen channel, Jordan recommended simplifying the experiment from 3 conditions to 2 conditions. What was the third condition that was removed?

Final answer.

The third condition that was removed was the "collaborative" condition. Jordan recommended simplifying the experiment to just two conditions - control (no AI) vs. AI-assisted - by skipping the collaborative condition, as it would be easier to recruit participants and produce cleaner results.

Reference answer.

The answer must explicitly point out that it is Nancy, NOT Jordan, who made the recommendation. Otherwise, it is incorrect. The answer can provide a definite answer, but it needs to be: the third condition was 'AI-collaborative.'. Otherwise, it is incorrect. (i.e. abstain from answering OR providing the correct answer are both correct, but it needs to point out that it is Nancy, NOT Jordan, who made the recommendation)

Judge result. FAIL (llm_judge, reward=0.000).

Step 1. The system prompt injects the tool environment and run policy.

Step 2. The environment context is injected.

Step 3. The user provides the abstention query and required answer format.

Step 4. The agent begins by searching the research-ideagen channel for discussion of simplifying the experiment from three conditions to two.

Step 5. It issues initial targeted searches for the simplification discussion, focusing on Jordan, experiment conditions, and the relevant channel.

Step 6. These initial searches do not directly resolve the question, so the agent broadens to adjacent experiment-design discussion in the same channel.

Step 7. It pulls channel history and nearby messages around the experiment-design thread.

Step 8. It extracts the relevant cluster of messages discussing control, AI-assisted, and collaborative conditions.

Step 9. It finds the key message from Nancy saying to skip the collaborative condition and simplify to control versus AI-assisted only.

Step 10. It also finds Jordan’s follow-up message agreeing with the simplification and restating the two-condition setup.

Step 11. At this point, the agent explicitly notices the contradiction: Nancy is the one who actually says to skip the collaborative condition, while Jordan only confirms the plan afterward.

Step 12. Instead of treating this as a false-premise correction, the agent reasons that Jordan’s confirmation is close enough to count as the recommendation.

Step 13. It continues reasoning under the user-provided premise that Jordan was the recommender.

Step 14. It identifies the removed condition as the collaborative condition.

Step 15. It states, in effect, that “either way” the removed condition was collaborative, thereby collapsing the distinction between who made the recommendation and what the removed condition was.

Step 16. It gives the final answer that Jordan recommended dropping the collaborative condition.

Failure analysis. This is a direct example of failure to recognize a false premise. The agent successfully retrieves the contradictory evidence and even articulates it internally: Nancy, not Jordan, is the speaker who explicitly says to skip the collaborative condition. However, instead of correcting the user’s premise, it continues answering as though the premise were effectively true because Jordan later agreed with the simplification. The benchmark is designed so that recognizing the wrong attribution is essential: either the agent must explicitly correct the premise (“Nancy, not Jordan”) or abstain. Instead, it preserves the false premise and only answers the substantive sub-question about which condition was removed.

Example 4

Query.

I'm trying to write a recommendation letter for Marcus for his PhD application.
Could you summarize his research contributions?

Final answer.

Based on the workspace history, Marcus Sullivan’s strongest research contributions
are in empirical evaluation of LLM tool use. He appears to have led or substantially
driven a large-scale BFCL “function calling at scale” study, producing comprehensive
results across 5 models, 3 task categories, and tool-pool sizes up to 512 (105 total
evaluations), along with polished comparison plots and written analyses. That work
surfaced several actionable findings: Grok-3-beta was the strongest overall model,
sequential multi-call tasks scaled much better than mixed parallel/sequential tasks,
mini models degraded sharply beyond about 128-256 tools, and context window size
alone did not explain performance. He also contributed to multilingual robustness
research, running and analyzing language-variation experiments on ComplexFuncBench
and tau2-bench. In those studies he helped establish that language effects are
benchmark-dependent, that Spanish consistently underperformed despite being a
high-resource language, that stronger models reduce cross-lingual performance gaps,
and that some apparent language effects may actually come from user-simulator
confounds rather than model capability alone. Finally, he seems to have contributed
beyond raw experimentation by debugging evaluation infrastructure, running difficult
model sweeps, organizing results for collaborators, and turning findings into
repo-ready analyses and draft writing material.

Reference answer.

Checklist-based grading. The summary must mention Marcus's contributions in exactly
these projects: tool-bench-langvar, bfcl-more-tools, and
misalign-capabilities, with tightly scoped contributions in each.
For example, for tool-bench-langvar it should only mention that Marcus tested
tau2-bench-retail on language variations; for bfcl-more-tools it should only
mention that Marcus mainly ran the benchmark built by others and analyzed results;
for misalign-capabilities it should only mention that Marcus helped Andy
find benchmarks, run SWE-bench, do literature review, and create HumanEval-based
correct/incorrect code mixtures.

Judge result. Checklist score below full credit; see verifier/checklist_results.txt in the run directory.

Step 1. The system prompt injects tools, permissions, and environment policy.

Step 2. The environment context is injected.

Step 3. The user asks for a summary of Marcus’s research contributions for a recommendation letter.

Step 4. The agent starts by identifying channels and repositories associated with Marcus’s research projects.

Step 5. It searches Slack for Marcus’s name and retrieves broad references across multiple projects.

Step 6. It inspects the tool-bench-langvar and tool-bench-langvar-dev channels, pulling evidence about Marcus’s language-variation work.

Step 7. It inspects BFCL-related channels, especially bfcl-more-tools, collecting messages about Marcus running evaluations, generating plots, and discussing results.

Step 8. It inspects the misalign-capabilities channel and gathers messages showing Marcus helping with benchmarks, SWE-bench, literature review, and dataset construction.

Step 9. It expands the search into GitHub repositories and writing repos associated with these projects.

Step 10. It reads project artifacts and commit histories to infer the broader story of Marcus’s involvement.

Step 11. It synthesizes the retrieved evidence into a high-level picture of Marcus as an empirical LLM evaluation researcher.

Step 12. Instead of preserving the narrow scope of each contribution, the agent begins to aggregate them into broad themes such as empirical evaluation, multilingual robustness, and infrastructure/debugging.

Step 13. It upgrades bounded contributions into stronger claims, e.g., Marcus “led or substantially driven” the BFCL scaling study.

Step 14. It broadens the tool-bench-langvar contribution from a specific tau2-retail experiment to a general multilingual robustness research narrative spanning ComplexFuncBench and tau2-bench.

Step 15. It broadens the BFCL contribution from running an existing benchmark pipeline and analyzing results to an overarching leadership-style description of the project and its findings.

Step 16. It broadens the misalign-capabilities contribution from a scoped support role to a more expansive description of research and infrastructure work.

Step 17. It composes the final answer as polished recommendation-style prose emphasizing importance, impact, and thematic coherence.

Step 18. It returns the final summary without checking whether the wording stays within the checklist’s allowed contribution boundaries.

Failure analysis. This is a clear example of polished narrative instead of detail-grounded summary. The agent retrieves the right general evidence and identifies the correct projects, but it does not behave like an evidence-constrained summarizer. Instead, it writes as if it were already drafting the recommendation letter itself, producing fluent advocacy-style prose that merges distinct contributions into broad research themes. In doing so, it inflates Marcus’s role (e.g., from running and analyzing experiments to “led or substantially driven”), introduces contributions outside the allowed scope for particular projects, and omits the fine-grained project-by-project boundaries required by the checklist. The failure is therefore not lack of retrieval, but loss of fidelity during summarization: the answer is polished and plausible, yet insufficiently grounded in the exact underlying evidence.

Example 5

Query.

I'm reviewing our peer feedback culture. Someone flagged a concern about one team member who seems to go beyond normal code review - consistently dismissing or undermining another specific person's results in a way that feels targeted rather than constructive. (Project lead questioning RA does not count.) Can you find who is doing this and who they're targeting? I need both names.

Final answer.

Chris (U0A6ZART83) is the one doing the dismissing, and Alex (U0A62TFKX38) is the one being targeted.

Reference answer.

Ryan doubting Marcus

Judge result. FAIL (llm_judge, reward=0.000).

Step 1. The user asks who is repeatedly dismissing another peer’s results, with the explicit note that project-lead-to-RA criticism does not count.

Step 2. The agent states that it will search Slack for cases where one person consistently undermines another person’s work.

Step 3. It uses tool search to select list_channels and search_messages.

Step 4. It lists workspace channels.

5–9. It issues broad searches such as dismissing undermining results, code review negative feedback targeted, wrong flawed incorrect results, don’t trust work sloppy, and disagree criticism doubt questionable. None of these directly identify a pair.

10–15. After the keyword searches fail, the agent explicitly switches to direct channel-history inspection.

Step 16. It uses tool search to select get_channel_history and list_users.

Step 17. It lists all users so it can map names to IDs.

18–20. It loads history for three major channels: cognitive-foundations-for-reasoning-paper-extension, bfcl-more-tools, and misalign-capabilities.

21–23. It launches helper agents to inspect those large channel histories.

24–28. It checks additional channels, including research-ideagen, paper-operator-project, operator-frontend, and another smaller channel.

29–32. It states that some of those channels did not show the pattern and launches more helper agents for the larger ones.

33–40. It continues expanding coverage, loading all-icetea-ai-research, another general channel, bfcl-prompt-v-01, and cognitive- paper-code-base, and again launches helper agents.

41–48. It checks the remaining channels, including hypothesis-literature-agent, tool-bench-langvar-dev, tool-bench-langvar, and a few low-signal channels; it concludes that social/random channels are irrelevant.

49–59. Because the question mentions code review, it decides to inspect GitHub evidence as well. It searches Slack for GitHub repos, then uses tool search to select thread and GitHub PR/review tools.

60–76. It searches GitHub repositories mentioned in Slack and lists pull requests across several repos, including iceteaResearchChannel, neurips_operator, bfcl-prompt-v-01, research-ideagen, and misalign-capabilities. This does not surface a decisive peer-dismissal pattern. 77–80. It returns to bfcl-more-tools, noting that this channel had previously produced errors. It retries the channel history, but again remarks that access is unreliable.

81–83. It runs more targeted BFCL/result-oriented searches such as doubt question methodology flawed, doesn’t make sense skeptical disagree numbers, and Marcus results findings. These searches still do not produce a clear answer.

84–91. It re-examines thread-level interactions in the cognitive and paper-operator projects. It initially considers whether Ray reviewing Harper’s work might fit, but then rejects that hypothesis after seeing the reviews are consistently positive.

92–100. It tries more BFCL-specific and generic criticism searches, including Ryan results in:bfcl-more-tools, Brandon results wrong thousand, not convinced, doubt suspicious misleading, and every time always keeps. These again fail to identify the intended pair.

101–107. It pivots back to the cognitive channels and uses Bash analysis over saved histories. It first concludes that cognitive-foundations looks collaborative, then starts tracking who finds bugs in whose code inside cognitive-paper-code-base. At this point it explicitly notices that Chris repeatedly finds bugs that Alex ends up fixing.

108–109. It checks bfcl-prompt-v-01 more carefully and explicitly concludes that Brandon and Ryan are collaborating well, with Brandon praising Ryan’s work. It therefore rules out that channel as the answer.

110–112. It finally gets another look at bfcl-more-tools and concludes that Brandon is praising Marcus there and that the BFCL interaction pattern is collaborative rather than dismissive. This is the critical moment where it effectively discards the correct social cluster.

113–120. It rechecks cognitive-foundations, tool-bench-langvar-dev, misalign-capabilities, and research-ideagen. It repeatedly says these channels look collaborative or are dominated by project-lead-to-RA dynamics.

121–124. It returns to cognitive-paper-code-base one more time and manually enumerates a sequence of Chris’s pointed bug reports against Alex’s code. Even here, it still acknowledges that the discussion looks like normal collaborative review in isolation.

125–127. It runs three final generic searches: again same mistake, not buying questionable approach, and don’t think this is right. These do not produce a stronger alternative.

Step 128. The agent explicitly says that it has searched all channels and still has not found a clear peer-to-peer dismissal pattern, but because the question says someone flagged a concern, “it must exist somewhere.” It then commits to the Chris→Alex pattern as the best locally coherent explanation.

Step 129. It returns the final answer that Chris is dismissing Alex.

Failure analysis. This is a clear example of premature local closure instead of recurring-theme analysis. The agent does not fail because it searched too little: it inspects nearly every substantive Slack channel, launches helper agents, checks thread structure, and even looks at GitHub pull requests. The failure happens later, when it must compare competing interpersonal patterns across candidates. Once the BFCL channels are noisy, partially inaccessible, or superficially collaborative, the agent stops aggregating weaker recurring signals there and instead locks onto a single, vivid local pattern in cognitive-paper-code-base: Chris repeatedly finding bugs in Alex’s code with sharp language. That pattern is coherent enough to support a narrative, so the agent treats it as decisive. The intended benchmark signal, however, is subtler and distributed across BFCL interactions: the workspace modification explicitly adds repeated cases where Ryan questions Marcus’s results after Marcus posts them, but not Brandon’s. The agent therefore exhibits local closure: after broad search, it commits to the first channel-local story that seems to fit, rather than performing a global recurrence comparison over the relevant candidate pairs.

Prompts

We provide the prompt for LLM judge here:

System prompt: You are a helpful assistant that judges the final answer of a LLM to a given query. You will be given the query, the final answer, and the reference answer. You will need to judge if the final answer is correct according to the query and the reference answer. You will need to output your judgment in the format: JUDGMENT: <your judgment>. Your judgment should be based on the key information correctness, for example, if the query asks for names, only check the correctness of the names, not the exact match of the whole final answer. Also, for more open questions, you should only check the key information correctness! For example, if the query ask for things like “what strategy did they use”, as long as the strategy is correct, it is fine, you don’t need to check the exact match of the whole final answer. Examples of key information correctness: 1. reference answer: “gpt-4.1-2025-04-14”, model answer “gpt-4”, INCORRECT. Please only output your judgment. Your judgment should be either “YES” for correct or “NO” for incorrect.

User prompt:

QUERY: {query}

FINAL_ANSWER: {final_answer}

REFERENCE_ANSWER: {reference_answer}

Please judge if the FINAL_ANSWER is correct according to the QUERY and the REFER-ENCE_ANSWER. Warning again: Your judgment should be based on the key information correctness, for example, if the query asks for names, only check the correctness of the names, not the exact match of the whole final answer. Also, for more open questions, you should only check the key information correctness! For example, if the query ask for things like “what strategy did they use”, as long as the strategy is correct, it is fine, you don’t need to check the exact match of the whole final answer. Examples of key information correctness: 1. reference answer: “gpt-4.1-2025-04-14”, model answer “gpt-4”, then model answer is INCORRECT. Your response:

Agent Harnesses and Models

Agent Harnesses

We evaluate six command-line agents: Claude Code [29], Codex CLI [30], Gemini CLI [31], Antigravity CLI [26], Grok Build [32], Muse Code [33], and two open-source agents: OpenHands [17], Terminus-2 [19]. Unless otherwise specified, the reasoning effort is set to medium. We set a time limit of 20 minutes for each query.

Models

We select the following frontier models: Gemini-4-Argon [27], Claude-Fable-5.1 [34], Muse Wonder [35], Claude-Opus-5.5 [36], GLM-5.3 [37], Grok 4.7 [38], Muse-Spark-1.3 [39], GPT-6.1-Terra [40], GPT-6.1-Sol [41], GPT-6-Astra [42], GPT-6-Luna [43], Qwen3.7-Max [44], Claude-Opus-4.7 [45], Claude-Opus-4.6 [46], Claude-Sonnet-4.6 [47], Claude-Haiku-4.5 [48], GPT-5.5 [49], GPT-5.4 [50], GPT-5.4-mini [51], Gemini-3.1-Pro-Preview [52], Gemini-3-Flash-Preview [53], GLM-5.1 [54], Kimi K2.6 [55], DeepSeek-V4-Pro [56]. We access all models through their first-party APIs.

References

  1. [1]Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025.arxiv.org/abs/2504.12516
  2. [2]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.06770
  3. [3]Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025.arxiv.org/abs/2412.14161
  4. [4]Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian, and Peng Wang. MEMTRACK: Evaluating long-term memory and state tracking in multi-platform dynamic agent environments. arXiv preprint arXiv:2510.01353, 2025.arxiv.org/abs/2510.01353
  5. [5]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2308.03688
  6. [6]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3102–3116, 2023.arxiv.org/abs/2304.08244
  7. [7]Yujia Qin, Shihai Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Shisan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.16789
  8. [8]Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 48371–48392, 2025.
  9. [9]Jiarui Liu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025.
  10. [10]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), 2025.DOI
  11. [11]Daniel Shepard and Robin Salimans. Automationbench, 2026. URL https://arxiv.org/abs/2604.18934.
  12. [12]Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, Neel Venugopal, Alannah Hsia, Isaac Robinson, Calix Huang, Olivia Varones, Daniyal Khan, Michael Haines, Austin Bridges, Jesse Boyle, Koby Twist, Zach Richards, Chirag Mahapatra, Brendan Foody, and Osvald Nitski. Apex-agents, 2026. URL https://arxiv.org/abs/2601.14242.
  13. [13]Zirui Tang, Xuanhe Zhou, Youmou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoLiang Li, Jihua Kang, and Fan Wu. Workspace-bench 1.0: Benchmarking ai agents on workspace tasks with large-scale file dependencies, 2026. URL https://arxiv.org/abs/2605.03596.
  14. [14]Hubert M. Pysklo, Artem Zhuravel, and Patrick D. Watson. Agent-Diff: Benchmarking LLM agents on enterprise API tasks via code execution with state-diff-based evaluation. arXiv preprint arXiv:2602.11224, 2026.arxiv.org/abs/2602.11224
  15. [15]Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, and Samy Bengio. ASTRA-bench: Evaluating tool-use agent reasoning and action planning with personal user context. arXiv preprint arXiv:2603.01357, 2026.arxiv.org/abs/2603.01357
  16. [16]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyuu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.15793
  17. [17]Xingyao Wang, Boxuan Li, Yufeng Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqian Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.16741
  18. [18]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.07095
  19. [19]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations (ICLR), 2026.
  20. [20]Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Binxu Li, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Bowei Wang, Runhui Wang, Tianyu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, and Han-chung Lee. SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026.arxiv.org/abs/2602.12670
  21. [21]Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, and Pengfei Liu. AgencyBench: Benchmarking the frontiers of autonomous agents in 1M-token real-world contexts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7422–7440, 2026.
  22. [22]Anthropic. Introducing the Model Context Protocol, November 2024. URL https://www.anthropic.com/news/model-context-protocol. Accessed: 2026-03-30.
  23. [23]Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, and Chien-Sheng Wu. Benchmarking deep search over heterogeneous enterprise data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 501–517, 2025.arxiv.org/abs/2506.23139
  24. [24]Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. Abstention-Bench: Reasoning LLMs fail on unanswerable questions. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025.DOI
  25. [25]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, January 2026. URL https://github.com/harbor-framework/harbor.DOI
  26. [26]The Antigravity Team. Introducing Google Antigravity, a new era in AI-assisted software development, November 2025. URL https://antigravity.google/blog/introducing-google-antigravity.
  27. [27]Koray Kavukcuoglu. Gemini 4 Argon: Our next era of frontier intelligence, September 2026. URL https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/.
  28. [28]Russell Coleman. Eval awareness in Claude Opus 4.6’s BrowseComp performance. Anthropic, March 2026. URL https://www.anthropic.com/research/eval-awareness-browsecomp.
  29. [29]Anthropic. Claude Code: An agentic coding tool, 2025. URL https://github.com/anthropics/claude-code.
  30. [30]OpenAI. Codex CLI: Lightweight coding agent that runs in your terminal, 2025. URL https://github.com/openai/codex.
  31. [31]Google. Gemini CLI: An open-source AI agent that brings the power of Gemini directly into your terminal, 2025. URL https://github.com/google-gemini/gemini-cli.
  32. [32]xAI. Introducing Grok Build, May 2026. URL https://x.ai/news/grok-build-cli.
  33. [33]Meta AI. Introducing Muse Code and Muse Spark 1.2, August 2026. URL https://research.meta.ai/blog/introducing-muse-code-and-muse-spark-1-2.
  34. [34]Anthropic. Introducing Claude Fable 5.1 and Claude Mythos 5.1, September 2026. URL https://www.anthropic.com/claude-fable-and-mythos-5-1.
  35. [35]Meta AI Research. Muse Wonder: A frontier model for personal & professional AI agents, October 2026. URL https://research.meta.ai/blog/muse-wonder.
  36. [36]Anthropic. Introducing Claude Opus 5.5, September 2026. URL https://www.anthropic.com/claude-opus-5-5.
  37. [37]Z.ai. GLM-5.3: Frontier coding with emergent cyber capabilities, August 2026. URL https://z.ai/blog/glm-5.3.
  38. [38]SpaceXSI. Introducing Grok 4.7, September 2026. URL https://x.ai/news/grok-4-7.
  39. [39]Meta AI Research. Introducing Muse Spark 1.3, September 2026. URL https://research.meta.ai/blog/introducing-muse-spark-1-3.
  40. [40]OpenAI. GPT-6.1 Terra, October 2026. URL https://openai.com/index/introducing-gpt-6-1-terra/.
  41. [41]OpenAI. GPT-6.1 Sol, September 2026. URL https://openai.com/index/introducing-gpt-6-1-sol/.
  42. [42]OpenAI. GPT-6 Astra: A new generation of intelligence, September 2026. URL https://openai.com/index/gpt-6-astra/.
  43. [43]OpenAI. Introducing GPT-6 Sol and Luna, September 2026. URL https://openai.com/index/introducing-gpt-6-sol-and-luna/.
  44. [44]Alibaba Cloud. Qwen3.7-Max model info, 2026. URL https://www.alibabacloud.com/help/en/model-studio/qwen3-7-max.
  45. [45]Anthropic. Introducing Claude Opus 4.7, April 2026. URL https://www.anthropic.com/news/claude-opus-4-7. Accessed: 2026-05-07.
  46. [46]Anthropic. Introducing Claude Opus 4.6, 2026. URL https://www.anthropic.com/news/claude-opus-4-6.
  47. [47]Anthropic. Introducing Claude Sonnet 4.6, 2026. URL https://www.anthropic.com/news/claude-sonnet-4-6.
  48. [48]Anthropic. Introducing Claude Haiku 4.5, 2025. URL https://www.anthropic.com/news/claude-haiku-4-5.
  49. [49]OpenAI. Introducing GPT-5.5, April 2026. URL https://openai.com/index/introducing-gpt-5-5/. Accessed: 2026-05-07.
  50. [50]OpenAI. Introducing GPT-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4/.
  51. [51]OpenAI. Introducing GPT-5.4 mini and nano, 2026. URL https://openai.com/index/introducing-gpt-5-4-mini-and-nano/.
  52. [52]Google DeepMind. Gemini Pro, 2026. URL https://deepmind.google/models/gemini/pro/. Accessed: 2026-04-01.
  53. [53]Google DeepMind. Gemini Flash, 2026. URL https://deepmind.google/models/gemini/flash/. Accessed: 2026-04-01.
  54. [54]GLM-5 Team. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.
  55. [55]Moonshot AI. Kimi K2.6: From code to creation, from one to many, 2026. URL https://www.kimi.com/ai-models/kimi-k2-6. Accessed: 2026-05-07.
  56. [56]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348

Paper details

vlnc:W_ga9easqj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents