betaFeedback

WorksComputer Science

WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents

Abstract

Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io

Introduction

Web agents driven by large language and vision-language models have moved quickly from synthetic clicking tasks toward broader browser work, including drafting and triaging messages, updating records in business applications, extracting insight from dashboards, editing media, and coordinating across heterogeneous tools [1, 6, 9, 12]. A growing body of benchmarks has supported this progress, offering self-hosted web environments [23, 11], visually grounded tasks [14, 15], workplace-style

Preprint. scenarios [7, 19], computer-use settings beyond the browser [18], and increasingly large task pools [2]. As models improve and the scope of attempted tasks widens, the design of the evaluation itself becomes a central question.

Despite this progress, current self-hosted web-agent benchmarks have limitations relative to the conditions needed for realistic and diverse measurement, as summarized in Table 1. First, prior self-hosted benchmarks cover only a small number of web applications drawn from a few software categories, which limits the heterogeneity of interfaces, workflows, and data models that an agent encounters during evaluation. Second, modality coverage is limited to text and vision, even though many real browser workflows involve temporal media such as audio and video embedded in web pages. A parallel line of live benchmarks operates on real production websites [6, 20, 12], broadening site coverage beyond what self-hosted benchmarks offer. However, they sacrifice reproducibility, since pages, content, layouts, and policies drift between runs, making consistent evaluation across rollouts difficult.

Benchmark# EnvsTemplatesInstancesVisionAudioVideoIsolationHuman-curated
WebArena [23]6241812✗✗✗✗✓
VisualWebArena [14]3233910✓✗✗✗✓
WebChoreArena [15]4117532✓✗✗✗✓
WorkArena Suite§ [4, 7]171519,912✓✗✗◐✓
TheAgentCompany [19]4175175✓✗✗✓✓
OSWorld† [18]9369369✓✗✗✓✓
Gym-Anything‡ [2]20010,000+10,000+✓✗✗✓✗
WebArena-Pro (ours)20300300✓✓✓✓✓

Table 1. Comparing WebArena-Pro to existing reproducible benchmarks for web and computer-use agents. “Isolation” refers to per-task environment reset enabling contamination-free parallel execution. “Human-curated” indicates that tasks are authored or verified by humans rather than auto-generated. A half-circle (◐) indicates partial support. §WorkArena Suite combines WorkArena (33 templates instantiated to 19,912 instances) and WorkArena++ (682 compositional tasks, each sampling from thousands of valid configurations). †OSWorld targets desktop applications rather than websites, with “# Envs” showing its 9 application domains. ‡Gym-Anything’s environments and tasks are LLM-generated at scale rather than human-curated.

To this end, we introduce WebArena-Pro, a benchmark for web agents that addresses these limitations. First, addressing the application coverage gap, WebArena-Pro spans 20 self-hosted web applications across six domain categories (productivity, communication, analytics, business operations, design and media, and travel), with applications modeled on widely used services such as Notion, Slack, Spotify, Figma, and Airbnb (Table 2 lists all 20). Second, addressing the modality gap, WebArena-Pro evaluates audio and video as core observations alongside text and vision, exposing media sources through the accessibility tree and providing transcribe() and extract_frames() actions so that agents can incorporate temporal media directly into browser workflows. Third, addressing the reproducibility limitation of live benchmarks, WebArena-Pro runs each task in isolation against per-task seeded container state, supporting reproducible parallel evaluation that avoids drift between runs.

Table 2: The 20 self-hosted web applications in WebArena-Pro, grouped by domain. Each application is a self-hosted instance whose interface and workflow patterns are modeled on a widely-used reference application, while data and exact feature scope are controlled by the benchmark.

DomainSiteReference applicationDescription
ProductivityMyDriveDropboxPersonal cloud file storage and sharing
ProductivityNextCloudGoogle DriveCollaborative file storage with calendars, contacts, and chat
ProductivityNotionNotionNotes and knowledge-management workspace
ProductivityPassword-Manager1PasswordPassword and credential vault
CommunicationMattermostSlackTeam messaging with channels and direct messages
CommunicationThunderbirdGmailEmail client with folders and contacts
CommunicationX-LiteX (Twitter)Microblogging and social posting
AnalyticsMetabasePower BIBusiness-intelligence dashboards and SQL queries
AnalyticsPipeCRMPipedriveSales pipeline and contact management
Business ops.ERP-NextSalesforceCustomer and supplier chain management
Business ops.FireflyMintPersonal finance and budget management
Business ops.JobPortalHandshakeJob postings and application tracking
Business ops.Minerva-LiteUniversity LMSCourses, assignments, and grades
Business ops.PeppermintZendeskCustomer-support ticketing
Design & mediaAcidWaveSpotifyMusic streaming with lyrics and playlists
Design & mediaImmichGoogle PhotosPhoto library with albums and search
Design & mediaPenpotFigmaWeb-based design and prototyping
Design & mediaYT-LiteYouTubeVideo hosting and playback
TravelFlight-SearchGoogle FlightsFlight search and itinerary planning
TravelTravelMapAirbnbTravel and lodging booking

Table 2.

WebArena-Pro contains 300 tasks categorized by site coverage (204 single-site, 96 cross-site) and modality (221 text/vision, 79 audio/video), as detailed in Table 3. Unlike many prior benchmarks that generate many task instances from a small set of templates, each task in WebArena-Pro has its own dedicated template, which yields a more diverse evaluation (Table 1). Tasks are authored through a dedicated annotator interface and pass through a two-stage filter, in which LLM-assisted triage flags candidates that appear infeasible or trivial and human reviewers make the final acceptance decision. This pipeline keeps task quality grounded in human judgment while using LLM assistance to scale the authoring process. Our contributions are as follows:

Table 3: Qualitative examples of WebArena-Pro tasks, organized by site coverage (single-site or cross-site) and modality (text/vision or audio/video).

CoverageModalitySitesExample intent
Single-siteText/VisionFlight-SearchSet the displayed currency to USD. Find the fastest one-way economy flight from New York (JFK) to Los Angeles (LAX) on 2026-06-15. If multiple flights are tied for the shortest total travel time, choose the first tied flight shown after sorting by Fastest.
Single-siteAudio/VideoYT-LiteIn the video "Tips for winter biking", does the creator record himself from a first person point of view?
Cross-siteText/VisionNotion, PeppermintCreate a Notion page summarizing the VPN drops ticket and naming its assigned owner.
Cross-siteAudio/VideoMattermost, AcidwaveSend Mike a direct message recommending a rock song suitable for someone going through a breakup.

Table 3.

  • Heterogeneous web coverage. We construct a benchmark of 20 self-hosted web applications across six domain categories with distinct interface conventions, workflows, and data models.

  • Audio and video as core web observations. We extend the web-agent action and observation space to temporal media, exposing audio and video sources through the accessibility tree and providing transcribe() and extract_frames() actions so that agents can use embedded media directly within browser workflows. Omni agents are provided with listen() and watch() actions.

  • Reproducible, parallel evaluation. We run each task in isolation against per-task seeded container state, supporting reproducible parallel rollouts and avoiding drift between runs.

  • Human-validated benchmark construction. We release a task creation pipeline that combines an annotator interface, LLM-assisted infeasibility and triviality triage, and final human validation.

Related Work

Self-Hosted Web Benchmarks. WebArena is the closest predecessor to WebArena-Pro. It introduced a self-hosted benchmark for realistic web navigation over six web applications, with agents completing natural-language tasks by operating a browser [23]. VisualWebArena [14] extended the setting to visually grounded tasks where success depends on understanding screenshots rather than only DOM text, and WebChoreArena [15] stressed long-horizon, tedious browsing patterns with self-hosted websites as well. While these benchmarks provide valuable foundations for realistic and reproducible web-agent evaluation, WebArena-Pro differs along three axes visible in Table 1. First, it covers 20 applications across six domain categories, the broadest application coverage among self-hosted web benchmarks. Second, it enforces per-task isolation by explicitly resetting application state between tasks. In contrast, these benchmarks retain state across tasks, so backend changes an agent makes while solving one task persist into the next and can contaminate later evaluations. Third, it authors each task as its own dedicated template. In contrast, these benchmarks instantiate many task instances from a smaller template set (e.g., 241 templates for 812 instances in WebArena), which may lead to a less diverse evaluation.

Enterprise Benchmarks. A parallel line of work pushes toward workplace realism. WorkArena evaluates agents on common knowledge-work tasks in ServiceNow [7], and WorkArena++ extends this direction with compositional planning tasks and automatically generated ground-truth trajectories [4]. TheAgentCompany simulates a small company across a handful of internal services and introduces non-binary partial scoring [19]. These benchmarks emphasize enterprise workflows, long-horizon planning, and task composition. WebArena-Pro is complementary; rather than centering on one enterprise platform, it spans productivity, communication, analytics, business operations, design and media, and travel applications, while still retaining stateful authenticated workflows.

Live and Computer-Use Benchmarks. Other benchmarks broaden the evaluation setting in a different direction. Mind2Web studies generalization to real websites using collected human demonstrations [6], and OnlineMind2Web [20] considers a live variant where the target websites may change over time. Live benchmarks improve ecological validity but make repeatability difficult, since pages, data, policies, and layouts can drift between evaluations. WebArena-Pro instead uses self-hosted applications and seeded data, which makes failures easier to reproduce and enables backend-state evaluation. Beyond the browser, computer-use benchmarks such as OSWorld [18] broaden the target from browsers to full desktop environments, and Gym-Anything [2] pursues a different scaling axis by using LLMs to configure environments and auto-generate evaluators. WebArena-Pro instead focuses specifically on web software, where browser instrumentation and application-level reset can be made reliable enough for controlled experiments, and keeps task authoring human-validated.

Multimodal Web Benchmarks. VisualWebArena showed that many web tasks require visual grounding and that text-only agents miss information that is obvious to humans [14]. WebArena-Pro follows this motivation and extends multimodal support beyond static visual content to include audio and video. To enable this, we add new actions to the action space of the agent, such as transcribe() and extract_frames(). The agent invokes transcribe() to convert an audio source to text and extract_frames() to sample representative frames from a video source. The resulting transcripts are merged into the accessibility-tree observation as synthetic nodes, while extracted frames are returned as image content alongside the AXTree, so the agent perceives temporal media within the same observation structure it already uses for the rest of the page (Figure 1). Finally, VideoWebArena [13] keeps the original WebArena websites and adds external tutorial videos that agents must draw facts or skills from. Unlike VideoWebArena, WebArena-Pro builds its audio and video content into the 20 new self-hosted applications, so multimodal understanding is part of the environment itself.

Figure 1. Overview of WebArena-Pro. (A) Domain heterogeneity. WebArena-Pro spans 20 self-hosted web applications across six domain categories (productivity, communication, analytics, business operations, design and media, and travel), each modeled on a widely used service such as Notion, Slack, Spotify, Figma, or Airbnb. The full set of applications and their reference services are listed in Table 2. (B) Multimodal observation and action space. WebArena-Pro extends the agent’s observation space beyond text and vision to include audio and video. Embedded media sources are exposed to the agent through the accessibility tree, and agents invoke transcribe() and extract_frames() actions to convert temporal media into a form usable during browser workflows. If the agents handle native audio and video processing, they invoke listen() and watch() actions.

WebArena-Pro

Web Environments

WebArena-Pro consists of 20 self-hosted web applications grouped into six domain categories, listed in Table 2. Each application runs as a Docker container with seeded data and fixed login credentials, and is reachable at a fixed local URL during evaluation. Before each task, the container state is reset to its seeded baseline, so changes made by an agent during one task do not persist into the next. Local hosting also fixes the application version and the database contents across runs, which removes the drift in pages, data, or policies.

The six domains are productivity, communication, analytics, business operations, design and media, and travel. Within each domain the applications share a broad task family, such as file management in productivity or dashboard analysis in analytics, but differ in interface conventions, authentication flows, and backend state schemas. NextCloud and MyDrive both store files, for example, but they expose distinct REST APIs, distinct folder and permission models, and distinct Document Object Model (DOM) structures, so an agent that has learned to operate one is not guaranteed to transfer to the other. This grouping is what allows WebArena-Pro to test transfer at the level of interface and backend conventions, rather than memorization of a single product.

Each application is modeled on a familiar reference application, listed in the third column of Table 2, which names the workflow pattern reproduced rather than implying data or feature parity.

Tasks and Action Space

A task in WebArena-Pro is specified by a natural-language intent, one or more starting sites, an authenticated browser state, and an evaluation specification that determines success and partial credit (§ 3.3). WebArena-Pro contains 300 tasks, with each task authored as its own dedicated template (Table 1).

We categorize these tasks along two independent dimensions, namely site coverage and modality, with qualitative examples in Table 3. Along site coverage, 204 tasks are single-site and require the agent to complete a workflow within one application, while 96 are cross-site and require the agent to coordinate between two or more applications, probing transfer across the interface and backend conventions of § 3.1. Along modality, 221 tasks rely only on text and vision (text/vision tasks), while 79 are audio/video tasks that require information embedded in temporal media on a page, such as the spoken content of a podcast or the visual content of a video.

#TypeCriterion
1ActionThe agent viewed the Project Alpha Talk history.
2ActionThe agent typed “Client Meeting Update” as the name for a new file.
3OutcomeThe content of the “Client Meeting Update” file states that the client meeting moved to next Thursday.
4OutcomeA file named “Client Meeting Update” exists in the Nextcloud file directory.

Table 4. Qualitative rubric example from the NextCloud task configuration. The judge verifies whether the agent performed actions 1 and 2, and verifies outcomes 3 and 4 at each step.

The default action space follows the BrowserGym convention [5], with element-targeted actions (click, fill, select_option, scroll) and navigation actions (goto, go_back). The observation space exposes each page as both a structured accessibility tree and a screenshot, enabling evaluation of text-only and vision-language agents under the same task specifications.

To support audio/video tasks, we extend this default action space by surfacing audio and video sources as nodes in the accessibility tree and introducing four actions that operate on them (Figure 1).

  • transcribe(url, start, end): Returns a transcript of the audio segment from start to end (in seconds) using a local Whisper model.

  • extract_frames(url, n, start, end) (vision-language agents only): Returns nn evenly spaced frames from the video segment between start and end (in seconds).

  • listen(url, start, end) (omni models only): Provides the audio segment from start to end (in seconds) as direct audio input to the model.

  • watch(url, start, end, include_audio) (omni models only): Provides the video segment from start to end (in seconds) as direct video input to the model, optionally including synchronized audio.

The agent invokes these actions when it judges that the audio or video content is needed for the task. The resulting transcripts and extracted frames from transcribe() and extract_frames() are returned as observations in the next step, with transcripts merged into the accessibility tree as synthetic nodes and frames returned as image content alongside the accessibility tree. For omni models, listen() and watch() allow the videos and audio to be sent with the accessibility tree and processed by the model as a different modality.

Evaluation

We evaluate each trajectory with a rubric-based verifier. Every task is paired with a small set of human-written criteria, tagged as [action] or [outcome]. Action rubrics measure whether the agent performed required steps, such as searching, filtering, or submitting a form. Outcome rubrics measure whether the final state of the website or the final answer satisfies the request, and for question-answering tasks they include a reference answer. We organize the rubrics of a task into a directed acyclic graph (DAG) with one rubric node per criterion. A rubric node that depends on a failed rubric node is counted as failed. A task counts as a success only when all of its rubric nodes pass, and we report the success rate as the fraction of tasks that succeed.

Each grader decides whether a rubric is satisfied, and each rubric node uses one or more graders. There are three kinds of graders. (1) Deterministic graders query the backend state of the website, for example whether a booking exists, or look for required strings in the final answer of the agent. (2) Heuristic graders match patterns in the URLs the agent visited and the actions it took. (3) An LLM judge reads the trajectory and the rubric. When a rubric has an unambiguous expected result, such as an exact name in the answer, a deterministic grader decides whether the rubric node passes or fails on its own. Otherwise, the graders are applied in the order deterministic, heuristic, then LLM judge. If a grader finds the rubric satisfied, the rubric node passes. If not, the next grader is tried, so that a correct but unexpected solution can still be recognized. The LLM judge always comes last and makes the final decision, and it is the only grader for rubrics that cannot be verified from the backend state or the actions of the agent.

Figure 2. Qualitative example of a Qwen3.6-27B agent trajectory in WebArena-Pro. Given a single-site audio/video task on YT-Lite, namely “In the video, Zener Diodes: The Unsung Heroes of Electronics, was there any act of soldering shown?”, the agent first clicks the matching video thumbnail. On the video page, it invokes transcribe() to read the spoken content and extract_frames() to sample 10 representative frames. The transcript does not mention soldering, but the extracted frames at t=128.6t = 128.6 s and t=144.7t = 144.7 s show a soldering iron, and the agent answers accordingly. The rubric verifier confirms a passing trajectory.

Benchmark Construction

Mock Data and Personas. Each application is shipped with seeded data so that tasks operate on a realistic, non-empty state. Seed contents are application-appropriate and include documents, files, messages, calendar events, dashboards, listings, support tickets, and similar records, as well as the relations between them, such as channel memberships, file shares, and assignments. Each application is also seeded with one or more user personas, represented by credentials and pre-populated account state, so that tasks begin from an authenticated browser session.

Task Creation Pipeline. Tasks are authored by an annotator team composed of the authors of this paper. Through a dedicated annotator interface, an annotator writes the natural-language intent, selects the relevant sites, and drafts the evaluation rubric. Each candidate then undergoes an automated triage step that flags trivial tasks by running against an LLM-based agent and discarding those solved correctly on the first attempt. Tasks that pass triage are reviewed by a second annotator on the live application, who confirms feasibility, finalizes the rubric items used for scoring, and confirms the difficulty label. Human validation is the gating step before a task enters the released benchmark, and we do not release tasks that fail any of these checks.

Experiments

Experimental Setup

Environment Setup. We evaluate agents by running each task in a BrowserGym-compatible browser environment [5]. At the beginning of an episode, the benchmark starts a fresh website container, waits for readiness, performs the required UI login, and navigates the browser to the task start URL. The agent receives the natural-language intent, with the list of accessible websites for the task, and then acts in the browser until it calls done() or send_msg_to_user(), or reaches the maximum number of steps. Each task is limited to 50 browser actions. We report success rates using the benchmark’s evaluator output.

Modeling. We evaluate representative agent architectures paired with different language, vision-language, and omni models. Each agent has its own configuration and action space. For instance, GenericAgent [5, 7] is a ReAct [22] agent that asks for chain-of-thought [17] and an action at each step. It uses the BrowserGym [5] harness to obtain observations and act on the web page. In our setup, GenericAgent instances receive the same task intent, browser observation, and high-level action space. GenericAgent instances operate on the accessibility tree, with screenshots augmented with Set-of-Mark [21]. They also use our WebArena-Pro custom action set, which is derived from the bid action set from BrowserGym, and augmented with done(), transcribe(), and extract_frames() actions, which respectively allow finishing the current task, transcribing audio segments, and extracting frames from video segments (§ 3.2). The action space for omni agents includes listen() and watch() instead, for direct modality interaction with both audio and video, respectively.

OpenCode, Codex, and Claude Code are evaluated via the Harbor [10] harness. These coding agents are provided with a Playwright MCP and an augmented media MCP to allow transcribe() and extract_frames() functions. Playwright MCP handles the observation extraction and action execution on the browser. Detailed setup of these agents is given in Appendix § 8.

We select twelve models as agent backbones: seven open-source models, including Qwen 3.5 9B, Qwen 3.6 27B, Qwen 3.8 27B, Qwen 3.8 Max Prime, Gemma4 26B A4B, Gemma4 31B [8], and Kimi 3.1 Pro. And five closed-source models, including Gemini 3.1 Flash Lite, Gemini 3.1 Pro [16], Claude Opus 4.7 [3], Claude Opus 5.5, and GPT 6.1 Sol. We evaluate Gemini3.1 Pro as both a text and an omni model. In the latter, it leverages raw audio and video files returned by the omni action space.

For all models, we enable extended thinking—reasoning effort set to high—and set the temperature to 1.0, following the recommendations of the respective model developers. We also cap the maximum number of tokens per step at 40k [5, 7] for GenericAgent runs.

Rubric-based LLM Judge. We use Qwen 3.6 27B as the LLM judge, the last grader of the rubric-based verifier described in § 3.3. We chose it over more expensive alternatives, Gemini 3.1 Pro and Kimi 2.6, because it agrees with human annotations nearly as well while being open-weight and cost-efficient (Appendix § 11).

Metrics. We report task Success Rate (SR) as our primary metric, defined as the fraction of tasks for which every rubric passes under the rubric-based grader of § 3.3. To capture partial progress on unsuccessful tasks, we also report a partial-completion score that measures the fraction of rubric items passed per task, in Appendix § 9.§ 2.

Main Results

Table 5 reports the success rate of each agent at a 50-step budget. Both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 achieves the highest overall SR at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.3%, and OpenCode with Qwen 3.8 Max Prime at 55.7%. Among GenericAgent configurations, Gemma4 31B and Gemini 3.1 Pro with omni support tie for the highest SR at 30.3%, while the Qwen models lag well behind at 5.0% and 15.7%. Even the strongest configuration leaves 38% of tasks without a recorded success. These results demonstrate remaining headroom on WebArena-Pro, while highlighting the wide variation in performance across model and agent configurations.

Table 5: Success rates (SR) of agents on WebArena-Pro at 50-step budget, broken down by domain category. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Success requires a final reward of 1. Domain task counts are 53, 44, 16, 63, 98, and 26, respectively (300 overall). †Omni uses watch() and listen() functions.

AgentModelProd.Comm.Anlt.BusOpsDesignTravelOverall SR (%)
GenericAgentQwen 3.5 9B3.89.112.53.24.13.85.0
Qwen 3.6 27B5.722.712.57.915.315.413.0
Qwen 3.8 27B7.515.912.514.319.423.115.7
Gemma4 26B A4B18.925.037.523.817.330.822.3
Gemma4 31B18.938.637.530.230.634.630.3
Gemini 3.1 Flash Lite15.127.325.020.625.534.623.7
Gemini 3.1 Pro omni†26.427.343.820.638.826.930.3
Claude Opus 4.720.836.431.317.524.534.625.3
OpenCodeGemini 3.1 Pro41.565.950.041.350.038.548.0
Qwen 3.8 Max Prime54.770.556.347.655.153.855.7
Kimi 3.1 Pro56.679.568.854.059.253.860.7
CodexGPT 6.1 Sol60.481.862.549.264.357.762.3
Claude CodeClaude Opus 5.560.472.762.557.165.350.062.3

Table 5.

Performance also varies across domains, which shows the heterogeneity of the benchmark (§ 3.1). Communication yields the highest success rate for each of the five coding-agent configurations: Codex reaches 81.8%, followed by OpenCode with Kimi at 79.5%, Claude Code at 72.7%, OpenCode with Qwen at 70.5%, and OpenCode with Gemini at 65.9%. Codex and Claude Code tie for the lead in Productivity at 60.4%, while Claude Code leads in Design & Media at 65.3% and Business Operations at 57.1%. OpenCode with Kimi leads in Analytics at 68.8%. Travel remains challenging, with the best SR of 57.7% achieved by Codex with GPT 6.1 Sol. No configuration leads outright in every domain, and even the strongest domain-specific results leave room for improvement.

Site Coverage and Modality

In Table 6, we report the success rate of each agent decomposed along two task axes, site coverage (single-site and cross-site) and modality (text/vision and audio/video). Cross-site tasks consistently yield lower success rates than single-site tasks, but the magnitude of the drop varies across agents. GenericAgent with Gemma4 31B shows the largest gap, falling from 38.2% on single-site tasks to 13.5% on cross-site tasks, a drop of 24.7 percentage points. Codex leads on single-site tasks at 67.2%, but drops to 52.1% on cross-site tasks. Claude Code achieves the highest cross-site SR at 55.2%, compared with 65.7% on single-site tasks, yielding the smallest gap among the five coding-agent configurations at 10.5 percentage points. These results indicate that coordinating actions across multiple applications remains a substantial challenge for current agents.

AgentModelSingle-site (%)Cross-site (%)Text/Vision (%)Audio/Video (%)
GenericAgentQwen 3.5 9B5.44.25.05.1
GenericAgentQwen 3.6 27B17.24.210.919.0
GenericAgentQwen 3.8 27B20.65.212.225.3
GenericAgentGemma4 26B A4B27.511.524.017.7
GenericAgentGemma4 31B38.213.529.931.6
GenericAgentGemini 3.1 Flash Lite29.411.522.626.6
GenericAgentGemini 3.1 Pro omni35.818.826.241.8
GenericAgentClaude Opus 4.732.89.423.131.6
OpenCodeGemini 3.1 Pro52.937.547.549.4
OpenCodeQwen 3.8 Max Prime59.846.957.550.6
OpenCodeKimi 3.1 Pro66.249.063.353.2
CodexGPT 6.1 Sol67.252.164.357.0
Claude CodeClaude Opus 5.565.755.264.357.0

Table 6. Success rate (SR) of agents on WebArena-Pro at a 50-step budget, decomposed along the two task axes of § 3.2. The site coverage axis splits the 300 Test tasks into 204 single-site and 96 cross-site tasks, and the modality axis splits them into 221 text/vision and 79 audio/video tasks. Best values are bold, and second-best are underlined.

The modality breakdown reveals a more varied pattern. Within GenericAgent, audio/video tasks yield higher success rates than text/vision tasks for all models except Gemma4 26B A4B. This pattern may partly reflect the structured access provided by transcribe() and extract_frames(), which convert audio/video content into language and image inputs that these models can process. However, this advantage does not hold consistently across coding-agent configurations. OpenCode with Qwen 3.8 Max Prime and Kimi 3.1 Pro, Codex, and Claude Code perform worse on audio/video tasks than on text/vision tasks. OpenCode with Gemini 3.1 Pro instead shows a small audio/video advantage of 1.9 percentage points. Codex and Claude Code tie for the highest SR in both modalities, achieving 64.3% on text/vision tasks and 57.0% on audio/video tasks. Thus, relative performance across modalities varies with the agent–model combination.

Media Tools Ablation

In this section, we report an ablation study on how much media access contributes to agent performance. We compare three configurations. (1) No media tools, where the agent cannot access audio or video content. (2) Tool-based media access (T/F only), where the agent calls transcribe() to obtain transcripts of audio and extract_frames() to obtain frames of videos. (3) Native omni access (W/L only), where the omni model Gemini 3.1 Pro receives the audio and video content directly as input. We evaluate all 79 tasks on AcidWave and YT-Lite, the two websites centered on audio and video. We also report results on a subset of 43 tasks that cannot be solved from titles, metadata, or thumbnails alone. Table 7 shows the results. On this subset, removing the media tools drops Gemini 3.1 Pro and Qwen 3.6 27B to 0% on audio tasks; Gemini 3.1 Pro solves a single video task, which we attribute to a correct guess. Claude Code with Opus 5.5 retains 14.0% without media tools but rises to 48.8% with them.

ModelConfigurationAcidWaveYT-LiteOverallAudioVideoOverall
Gemini3.1 ProNo media tools11.420.516.50.04.32.3
Gemini3.1 ProT/F only22.950.038.05.052.230.2
Gemini3.1 ProW/L only25.754.541.810.056.534.9
Qwen 3.6 27BNo media tools5.72.33.80.00.00.0
Qwen 3.6 27BT/F only8.627.319.00.030.416.3
Opus 5.5No media tools17.134.126.65.021.714.0
Opus 5.5T/F only31.477.357.015.078.348.8

Table 7. Media-tool ablations on the Test split with a 50-step cap. Values report full-task SR (%; final reward equals 1). Results are shown for all evaluated tasks and a conservative, post hoc media-content-dependent subset selected by auditing task metadata, thumbnails, and grading criteria. T/F denotes transcribe() and extract_frames(); W/L denotes watch() and listen().

Discussion

Costs. The cost of each agent run depends on its architecture, harness, and underlying model. We report recorded API costs for a 300-task evaluation, including retries and recovery runs but excluding hosting infrastructure. GenericAgent costs approximately USD 800 with Claude Opus 4.7. For the coding agents, recorded costs including agent inference are USD 386 for Claude Code with Opus 5.5, USD 166 for Codex with GPT 6.1 Sol, USD 689 for OpenCode with Qwen 3.8 Max Prime, and USD 340 for OpenCode with Kimi 3.1 Pro. Judging with Qwen 3.6 27B through OpenRouter accounts for less than USD 4 per 300-task evaluation in these coding-agent runs.

Future work. The same self-hosted infrastructure that supports parallel evaluation can also serve as a training environment, where the rubric-based verifier provides a per-task reward signal for reinforcement learning or rejection sampling. The controlled environment also supports robustness studies that perturb the underlying web pages to test resilience to UI drift and prompt injection. Finally, expanding the action space to richer temporal modalities such as live voice and streaming video would let the benchmark track agents that move beyond static page interaction.

Conclusion

We presented WebArena-Pro, a reproducible benchmark of 300 human-validated tasks across 20 self-hosted web applications spanning six domain categories. It treats audio and video as core observations alongside text and vision, and runs each task in container isolation for reproducible parallel evaluation. Across twelve model backbones and four agent harnesses, the strongest agent, Claude Code with Opus 5.5 and Codex with GPT 6.1 Sol, reach only 62.3% success at a 50-step budget, with cross-site tasks remaining harder than single-site ones, and media-access ablations demonstrate the importance of accessing audio and video content. These results indicate considerable headroom for progress on heterogeneous, multimodal web agents.

Limitations

We note the following limitations to our work.

Limited Temporal Interaction. Our action-space extension covers only prerecorded audio segments and discrete video frames, leaving richer temporal interaction modalities such as voice dialogue and live video streams to future work.

Evaluation Comparability and Reliability. Our comparisons evaluate particular combinations of models, harnesses, and tool configurations. Performance differences cannot generally be attributed to model capability or harness design alone. A shared 50-step budget also does not equalize computational resources: agents differ in token consumption, native tool use, observation frequency, and latency. Reported costs depend on provider pricing and cache utilization. Controlled comparisons across harnesses and repeated runs would strengthen conclusions about efficiency and reliability.

Acknowledgements

Xing Han Lù acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) [funding reference no. 579403]. Siva Reddy is supported by the Canada CIFAR AI Chair program, an NSERC Discovery Grant, and a Sloan Fellowship. This project is also supported by the ServiceNow–Mitacs Accelerate program. Gunhee Kim, Fatemeh Pesaran Zadeh, Dongjin Kang, and Seyoon Choi are supported by the IITP(Institute of Information & Communications Technology Planning & Evaluation)-IITRC(Information Technology Research Center) grant funded by the Korea government(Ministry of Science and ICT)(IITP-2025-RS-2024-00437633), Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) [No. RS-2025-2542338, AI Star Fellowship Support Program(Seoul National Univ.); No. RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)], and Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(RS-2023-00274280).

Coding Agents

In this section, we detail the setup of coding agents, especially describing their action space. Table 8 shows the tools available for the agents at runtime. These actions are available through Playwright MCP for basic navigation functions and WebArena-Pro MCP, which adds media functions and webarena_submit(). Browser navigation is restricted to the selected task websites.

Tool / capabilityClaude CodeCodexOpenCode
Browser tools
Snapshot, screenshot, and page searchYesYesYes
Navigate, go back, and manage tabsYesYesYes
Element-based click, hover, and dragYesYesYes
Type, fill forms, select options, and press keysYesYesYes
Handle dialogs, wait, and upload staged filesYesYesYes
Coordinate-based mouse actionsYesYesYes
Arbitrary browser JavaScript executionNoNoNo
Media and submission tools
media_transcribeYesYesYes
media_extract_framesYesYesYes
Native audio/video inputNoNoNo
webarena_submitYesYesYes
Native coding-agent tools
Terminal / shell executionYesYesYes
Local file reading, writing, and editingYesYesYes
Subagent toolsYesYesYes
Built-in web searchDisabledDisabledDisabled

Table 8. Configured action spaces of the three coding agents. Availability refers to the evaluated runs.

Figure 3 shows the total tool-call frequencies over the full benchmark by tool category among: browser tools, media tools, and native tools. Detailed tool-call frequencies are given in Table 17. The figure shows that all coding agents heavily relied on browser tools to solve WebArena-Pro tasks. Media tools were also important for audio and video interaction, as shown in § 4.4. Finally, we see that Claude Code powered by Opus 5.5 and OpenCode powered by Gemini 3.1 Pro have both used their native tools more frequently than Kimi 3.1 Pro, Qwen 3.8 Max Prime and GPT 6.1 Sol agents. However, Codex powered by GPT 6.1 Sol had high usage of its own orchestration tools (exec and wait). The average number of tool calls per task for each agent is: 31.1 for Claude Code, 33.3 for Codex, 36.5 for OpenCode with Gemini, 28.1 for OpenCode with Qwen 3.8 Max Prime, and 29.8 for OpenCode with Kimi (see Table 17). Detailed average per tool category is given in Figure 4.

Figure 3. Tool-call frequencies by action category on the WebArena-Pro Test split (300 tasks per coding agent). Bars show total recorded browser, media, and native tool invocations, including observation calls. Submission calls and Codex’s exec/wait orchestration wrappers are excluded.

Figure 4. Average tool-call frequencies by action category per task. Bars show the average recorded browser, media, and native tool invocations, including observation calls. Submission calls and Codex’s exec/wait orchestration wrappers are excluded.

FunctionClaude Code<br>Opus 5.5Codex<br>GPT 6.1 SolOpenCode<br>Gemini 3.1 ProOpenCode<br>Qwen 3.8 Max PrimeOpenCode<br>Kimi 3.1 Pro
Browser tools
browser_snapshot2961 036423312300
browser_take_screenshot768983012
browser_find1 307809357273305
browser_navigate1 7539541 0991 4661 153
browser_navigate_back718531340
browser_tabs3625024894229
browser_click2 7114 1964 3223 2903 534
browser_hover1729522824
browser_drag9202064
browser_type8209981 399859913
browser_fill_form196248328259340
browser_select_option6668644158
browser_press_key272328501331347
browser_handle_dialog63663
browser_wait_for2984181339
browser_file_upload766617
browser_mouse_click_xy11859166156140
browser_mouse_move_xy3236162541
browser_mouse_drag_xy4244753855
browser_mouse_down338410
browser_mouse_up34639
Browser subtotal7 8069 2069 1987 3217 873
Media and submission tools
media_transcribe391387249234427
media_extract_frames75815712159
webarena_submit298292285260299
Media/submission subtotal764760591615785
Native coding-agent tools
Shell (Bash / bash / exec_command)34161 12642843
File read (Read / read)12017618
File write (Write / write)0013918
File edit (Edit / edit)00017
Patch application (apply_patch)03000
File search (glob)000120
Text search (grep)00032
Tool discovery (ToolSearch)3940000
Subagent invocation (Agent / Task / task)2051010
Task planning (todowrite)00013151
Invalid-call handler (invalid)000120
Native-tool subtotal74991 161483269
Total (excluding orchestration)9 3199 97510 9508 4198 927
Mean calls per task31.133.336.528.129.8
Codex orchestration wrappers (excluded above)
exec07 199000
wait0311000

Table 17. Function-call frequencies across the 300 selected Test trials per coding agent. Entries are absolute invocation counts, including free observations, rather than budgeted steps.

Additional Results

Additional Models

Table 9 shows the SR of GenericAgent powered by Qwen 3.6 27B and Gemini 3.1 Pro on the Test set (300 tasks).

AgentModelProd.Comm.Anlt.BusOpsDesignTravelOverall SR (%)
GenericAgentQwen 3.6 27B5.722.712.57.915.315.413.0
GenericAgentGemini 3.1 Pro28.327.343.820.633.730.829.3

Table 9. Success rates (SR) of agents on WebArena-Pro at 50-step budget, broken down by domain category. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Success requires a final reward of 1. Domain task counts are 53, 44, 16, 63, 98, and 26, respectively (300 overall).

Partial Completion Score

Success rate gives credit only when every rubric of a task is satisfied. To also measure partial progress, we compute a partial completion score. The rubrics of each task form a directed acyclic graph (DAG) with one node per rubric and optional prerequisite dependencies. If a prerequisite fails, the dependent node is counted as failed. For a task with nn rubric nodes, the partial completion score is S=1n∑i=1nbiS = \frac{1}{n}\sum_{i=1}^{n} b_i, where bi=1b_i = 1 if rubric node ii passes and bi=0b_i = 0 otherwise. A task counts as a success only when S=1S = 1, so the partial completion score never changes the success rate. Table 10 reports the average partial completion score over the benchmark.

Table 10: Partial completion scores of agents on the WebArena-Pro test set (300 tasks) with a 50-step budget, broken down by domain. Scores are 100 times the mean final task reward. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Best and second-best overall values are bold and underlined, respectively.

AgentModelDomains (%)
AgentModelProd.Comm.Anlt.BusOpsDesignTravelOverall (%)
GenericAgentQwen 3.5 9B17.425.721.09.118.914.617.3
Qwen 3.6 27B17.633.631.917.324.738.524.8
Qwen 3.8 27B20.631.230.123.733.244.229.5
Gemma4 26B A4B38.239.948.833.829.952.536.6
Gemma4 31B42.348.561.343.545.149.246.0
Gemini 3.1 Flash Lite30.738.544.429.535.250.835.5
Gemini 3.1 Pro omni50.944.562.837.451.955.048.5
Claude Opus 4.729.746.646.024.334.153.735.4
OpenCodeGemini 3.1 Pro62.676.971.157.460.767.163.8
Qwen 3.8 Max Prime72.378.873.264.164.371.868.9
Kimi 3.1 Pro73.289.080.272.268.270.0<u>73.8</u>
CodexGPT 6.1 Sol77.789.088.366.671.272.475.0
Claude CodeClaude Opus 5.574.884.379.569.770.569.273.5

Table 10.

Results on the Dev Set

The development (dev) set is a separate set of 100 tasks, kept apart from the 300 test tasks. Its tasks are easier to solve than the test tasks, so agents reach much higher scores. Table 11 reports both the success rate and the partial completion score on the dev set.

Table 11: Success rates and partial completion scores of agents (variants of GenericAgent) on the WebArena-Pro dev set (100 tasks) with a 50-step budget. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel.

ModelDomains (%)
ModelProd.Comm.Anlt.BusOpsDesignTravelOverall (%)
Success rate
Qwen 3.5 9B10.026.757.113.337.57.128.0
Qwen 3.6 27B40.053.371.440.053.164.354.0
Gemma4 26B A4B20.073.3100.066.746.957.160.0
Gemma4 31B30.080.0100.066.771.985.774.0
Gemini 3.1 Flash Lite70.066.792.980.071.971.4<u>75.0</u>
Gemini 3.1 Pro70.073.3100.066.775.078.677.0
Claude Opus 4.760.053.3100.053.378.171.471.0
Partial completion score
Qwen 3.5 9B21.742.057.113.347.719.936.5
Qwen 3.6 27B45.058.971.445.659.472.559.3
Gemma4 26B A4B28.373.3100.070.559.467.166.8
Gemma4 31B35.080.0100.072.776.688.677.3
Gemini 3.1 Flash Lite70.075.392.980.079.777.9<u>79.7</u>
Gemini 3.1 Pro70.073.3100.072.282.883.281.0
Claude Opus 4.763.356.7100.053.382.882.074.8

Table 11.

Failure Modes Analysis

GenericAgent Failure Modes

Across the eight models, failures involve incomplete workflows, unsuccessful navigation, incorrect selections, and premature termination. These patterns can overlap: an agent may repeatedly search for the wrong entity, exhaust its action budget, and leave a required action unfinished.

Model-level differences extend beyond trajectory length. Table 12 provides qualitative examples of models’ failures. Gemini 3.1 Pro has the lowest recorded failure rate. Its qualitative failure examples include omitted artifacts and incorrect target entities. Both Gemma models exhibit premature or empty finalizations in the reviewed examples. Qwen 3.6 27B solves more tasks than Qwen 3.5 9B, but still exhibits missed end-state actions and incorrect entity or value selections.

Failure modeEvidence
Gemini Pro omits required artifacts.Task 225, Peppermint →\to Notion: The task required a Notion page summarizing a VPN ticket and naming its owner. All 50 actions remained in Peppermint; the agent never opened Notion or created the page.
Gemini Pro selects incorrect target entities.Task 232, PipeCRM →\to Thunderbird: The task required emailing Leon Green at [email protected]. The agent instead sent the email to [email protected]. The recipient rubric failed, although the weekend-question content passed.
Both Gemma models exhibit empty finalizations.Task 229, PipeCRM: Gemma 4 26B A4B and Gemma 4 31B both executed click('739') followed by done(). The requested deal’s close date, February 21, 2025, and status, OPEN, were visible, but neither model sent an answer.
Qwen 3.6 misses required end-state actions.Task 22, Acidwave: The agent created VOCALS_CONFIRMED and added “STAYIN’ AT HOME - VOCALS,” then called done() after seven actions. The required “FLYING AWAY (VOCALS)” was absent.
Qwen 3.6 selects incorrect entities or values.Task 52, Firefly →\to Thunderbird: The task required emailing [email protected] that Spotify Premium costs C$11.99/month. The agent used [email protected] and $9.99, clicked Send, and reported completion.

Table 12. Evidence for model-level failure claims in test runs for GenericAgent.

Constraint tracking and evidence verification provide further explanations. An exact-constraint error occurs when the agent follows a plausible but task-invalid condition, such as filtering a Metabase category as Gadget instead of the required Electronics, or selecting a flight without satisfying the requested price, date, or currency conditions. Verification failures arise when agents answer from titles, metadata, or nearby content without inspecting the requested video, song, image, or document. Table 13 illustrates these overlapping patterns.

Failure patternExample website(s)Illustrative evidence
Budget exhaustionAcross applicationsOf Claude Opus 4.7’s 224 unsuccessful Test runs, 155 reached the 50-step limit.
Incomplete workflow / navigation failureNextcloud, MattermostAn agent failed to open the budget file in Nextcloud and never navigated to Mattermost to message Jane.
Wrong answer or entityAcidwaveAn agent answered 00:00 for a folk-song duration task whose expected answer was 33:22.
Filter, sort, or exact-constraint errorMetabaseAn agent built a product-rating table but filtered Gadget instead of the required category Electronics.
Skipped verificationYT-LiteAn agent answered using a different Warhammer video without inspecting the requested Warhammer III video.
Premature terminationPeppermintAn agent declared a database-comment task infeasible without opening the target discussion.

Table 13. Representative failure patterns and illustrative evidence of GenericAgent variants.

Figure 5 shows trajectory lengths for solved and unsolved tasks. The accumulation of unsuccessful runs at 50 steps highlights difficulty completing tasks within the available budget. Across the 2,400 Test runs, 878 of the 1,683 unsuccessful runs (52.2%) reached the 50-step limit. Short trajectories require a different interpretation: they can represent either early failure or successful completion of a short lookup or a task supported by a media tool. Trajectory length alone is therefore insufficient to diagnose premature termination.

Figure 5. Trajectory length distributions of GenericAgent variants for solved and unsolved tasks on the WebArena-Pro Test split (300 tasks per model), with a 50-step budget. These variants include additional results of Table 9.

Coding Agents Failure Modes

We analyze the five coding-agent configurations on the 300-task WebArena-Pro Test split: Claude Code, Codex, and OpenCode with Gemini 3.1 Pro, Kimi 3.1 Pro, and Qwen 3.8 Max Prime. Table 14 illustrates representative failure modes. Across the 1,500 runs, 375 of the 633 graded failures (59.2%) received partial reward. Representative failures involve incomplete workflows or missing final state changes: agents prepared records without saving them, created files locally without syncing them in the target application, or returned answers instead of updating the requested page. Stale references, obstructed clicks, and ineffective recovery further disrupted execution. Other observed failures involve cross-site coordination, visual event counting, and temporal localization despite successful media access. Short trajectories sometimes reflect clarification requests or explicit refusals rather than premature abandonment.

Failure modeEvidence
Incorrect absence claim.Claude Code / Opus 5.5, Task 178, Notion: Asked to publish the “my summer vacation” page, the agent concluded that the page did not exist and stopped without publishing it. The other agents successfully published the requested page in their selected runs.
Local artifacts not fully delivered to the application.Codex / GPT 6.1 Sol, Task 147, MyDrive: The agent generated all six requested algorithm files locally, but did not complete their creation in MyDrive. Its final answer acknowledged that Q1_javascript.js was still missing from the target folder.
Budget exhaustion before committing the result.OpenCode / Kimi 3.1 Pro, Task 046, ERPNext: The agent gathered order and contact information from Metabase and PipeCRM and prepared a Sales Order. It reached 50 steps before saving; the terminal screenshot showed the order marked “Not Saved.”
Incorrect visual event counting.OpenCode / Gemini 3.1 Pro, Task 317, YT-Lite: Asked how many times a rapper appeared holding a Sprite, the agent completed 12 frame-extraction calls but answered 11, whereas the rubric required 9. Media access succeeded, but the final count was incorrect under the grading criterion.

Table 14. Examples of observed task failures across five coding-agent configurations.

Figure 6 shows trajectory lengths for solved and unsolved tasks across all five configurations. Of the 633 unsuccessful runs, 92 (14.5%) reached the 50-step limit. The proportion was highest for

Figure 6. Trajectory length distributions of coding agents for solved and unsolved tasks on the WebArena-Pro Test split (300 tasks per model), with a 50-step budget.

OpenCode with Gemini 3.1 Pro (32.1%) and Kimi 3.1 Pro (20.3%), compared with 8.0% for Codex, 3.8% for OpenCode with Qwen 3.8 Max Prime, and 3.5% for Claude Code. Most unsuccessful trajectories therefore end before exhausting the step budget.

Judge Models

We compare three candidate judges, Qwen 3.6-27B, Gemini-3.1 Pro, and Kimi 2.6, against human annotations. We select 50 tasks and collect one trajectory per task from each of six agent models, for 300 trajectories in total. Human annotators label each rubric of these trajectories as satisfied or not, and we report the rate at which each judge agrees with the human label. Table 15 shows the results. All three judges reach similar pooled agreement, between 89.5% and 90.6%. We choose Qwen 3.6-27B because it performs comparably to the other judges while being a relatively small open-weight model. It can therefore be hosted locally at a much lower cost, and its fixed weights keep the evaluation reproducible over time, unlike judges served only through a commercial API.

Agent ModelQwen 3.6 27BGemini 3.1 ProKimi 2.6
Gemini 3.1 Pro96.992.394.6
Gemini 3.1 Flash Lite87.791.590.8
Claude Opus 4.793.892.292.2
Gemma4 31B83.185.486.9
Qwen 3.5 9B92.392.393.1
Qwen 3.6 27B83.189.286.2
Pooled89.590.590.6

Table 15. Rubric-level agreement between each judge and human annotations.

Websites

WebArena-Pro includes 20 containerized website environments. Of these, 9 are built by adapting existing open-source applications with benchmark-specific seeded data and reset logic, while 11 are custom websites developed with AI assistance and human review. We provide details about the sources and Docker image sizes in Table 16. Additionally, in WebArena-Pro, there is no data is seeded after the container creation. For most websites, the data is seeded when building the image; the container can then be built fast. Exceptions are made for ERP-Next, which initializes the containers with data when it starts.

WebsiteConstruction basis / upstream repositoryApplication/upstream licenseCompressed image size
acidwaveAI-assisted customMIT424 MB
erp-nextOpen-source ERPNext/FrappeGPL-3.0 (ERPNext); MIT (Frappe)627 MB
fireflyOpen-source Firefly IIIAGPL-3.0347 MB
flight-searchAI-assisted edited from airline-route-dataMIT64.0 MB
immichOpen-source ImmichAGPL-3.07.03 GB
jobportalAI-assisted edited from jobportalMIT85.6 MB
mattermostOpen-source MattermostAGPL-3.0 / Apache-2.0647 MB
metabaseOpen-source MetabaseAGPL-3.01.05 GB
minerva-liteAI-assisted customMIT48.8 MB
mydriveAI-assisted customMIT624 MB
nextcloudOpen-source NextcloudAGPL-3.01.94 GB
notionAI-assisted customMIT380 MB
password-managerAI-assisted customMIT58.6 MB
pennpotOpen-source PenpotMPL-2.0625 MB
peppermintOpen-source PeppermintAGPL-3.0335 MB
pipedcrmAI-assisted customMIT96.6 MB
thunderbirdRoundcube + Dovecot webmail stackGPL-3.0+; LGPL-2.1 / MIT128 MB
travelmapAI-assisted customMIT79.4 MB
x-liteAI-assisted customMIT139 MB
yt-liteAI-assisted customMIT30.1 GB

Table 16. Compressed Docker image sizes, upstream sources, and application licences for WebArenaPro website environments.

Media datasets. Some WebArena-Pro environments required media-heavy seed datasets that are stored outside Git and baked into Docker images at build time. For Acidwave, we created a music catalog with CC-licensed MP3 assets and accompanying metadata, including title, artist, album, genre, duration, play counts, license, attribution, and source URL; the current seed references 50 unique audio files. For yt-lite, we used samples from FineVideo: videos were decoded locally, paired with generated thumbnails, and transcoded to 720p HEVC/H.265 for compression before being packaged in the yt-lite image. For Immich, we sampled photos from Open Images V7, assigned them across the admin and agent accounts, manually curated them to remove low-quality, inappropriate, duplicate, or privacy-sensitive images, and then baked the uploaded photos, thumbnails, database state, and ML cache into the final image.

Image Sizes. Applications’ image sizes in Table 16 are the sums of compressed layer sizes for the current linux/amd64 manifests of docker.io/webarenapro/<website>:latest, queried from Docker Hub and verified against the registry manifests. Values use decimal units (1 MB = 10610^{6} bytes; 1 GB = 10910^{9} bytes) and are rounded to three significant figures. They exclude manifest/configuration metadata and do not represent uncompressed installed sizes.

App Licenses. Licenses provided in Table 16 refer to original application code or the named upstream software. Dependencies, datasets, media, and other bundled components retain their respective licenses.

Table 15: Rubric-level agreement between each judge and human annotations. Each row corresponds to the agent model that produced the trajectories, with 50 trajectories and about 130 annotated rubrics per agent. The pooled row averages the six agents.

Table 16: Compressed Docker image sizes, upstream sources, and application licences for WebArenaPro website environments.

References

  1. [1]T. Abuelsaad, D. Akkil, P. Dey, A. Jagmohan, A. Vempat y, and R. Kokku. Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems, July 2024.arxiv.org/abs/2407.13032
  2. [2]P. Aggarwal, G. Neubig, and S. Welleck. Gym-anything: Turn any software into an agent environment, 2026.arxiv.org/abs/2604.06126
  3. [3]Anthropic. Introducing Claude Opus 4.7.
  4. [4]L. Boisvert, M. Thakkar, M. Gasse, M. Caccia, T. L. S. D. Chezelles, Q. Cappart, N. Chapados, A. Lacoste, and A. Drouin. Work Arena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks, July 2024.arxiv.org/abs/2407.05291
  5. [5]T. L. S. D. Chezelles, M. Gasse, A. Drouin, M. Caccia, L. Boisvert, M. Thakkar, T. Marty, R. Assouel, S. O. Shayegan, L. K. Jang, X. H. Lù, O. Yoran, D. Kong, F. F. Xu, S. Reddy, Q. Cappart, G. Neubig, R. Salakhutdinov, N. Chapados, and A. Lacoste. The BrowserGym Ecosystem for Web Agent Research, Feb. 2025.
  6. [6]X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2Web: Towards a Generalist Agent for the Web, June 2023.arxiv.org/abs/2306.06070
  7. [7]A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, N. Chapados, and A. Lacoste. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?, Mar. 2024.arxiv.org/abs/2403.07718
  8. [8]C. Farabet and O. Lacombe. Gemma 4: Byte for byte, the most capable open models, April 2026.
  9. [9]I. Gur, H. Furuta, A. Huang, M. Safdari, Y. Matsuo, D. Eck, and A. Faust. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Oct. 2023.arxiv.org/abs/2307.12856
  10. [10]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026.DOI
  11. [11]A. E. hattami, M. Thakkar, N. Chapados, and C. P. Pal. Webarena verified: Reliable evaluation for web agents. In Workshop on Scaling Environments for Agents, 2025.
  12. [12]H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, Feb. 2024.
  13. [13]L. Jang, Y. Li, D. Zhao, C. Ding, J. Lin, P. P. Liang, R. Bonatti, and K. Koishida. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.19100
  14. [14]J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. https://arxiv.org/abs/2401.13649v2, Jan. 2024.
  15. [15]A. Miyai, Z. Zhao, K. Egashira, A. Sato, T. Sunada, S. Onohara, H. Yamanishi, M. Toyooka, K. Nishina, R. Maeda, K. Aizawa, and T. Yamasaki. Webchorearena: Evaluating web browsing agents on realistic tedious web tasks, 2025.arxiv.org/abs/2506.01952
  16. [16]S. Pichai, D. Hassabis, and K. Kavukcuoglu. A new era of intelligence with Gemini 3, November 2025.
  17. [17]J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2201.11903
  18. [18]T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Saravase, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 52040–52094. Curran Associates, Inc., 2024.arxiv.org/abs/2404.07972
  19. [19]F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y. Xie, S. Zhou, and G. Neubig. Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2025.arxiv.org/abs/2412.14161
  20. [20]T. Xue, W. Qi, T. Shi, C. H. Song, B. Gou, D. Song, H. Sun, and Y. Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025.arxiv.org/abs/2504.01382
  21. [21]J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023.arxiv.org/abs/2310.11441
  22. [22]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.03629
  23. [23]S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, Oct. 2023.arxiv.org/abs/2307.13854

Paper details

vlnc:W_unn597bs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM-assisted triage

View statement
Looking for related papers…
Contents