PLE: Physical-Engineering's Last Exam

github.com/google-research/ple | ple-bench.com

Introduction

The physical systems and infrastructure that modern society runs on (hospitals, schools and university campuses, office and laboratory buildings, power distribution, water and transportation networks) are designed by professional physical engineers. Demand for this engineering work is rising rapidly: the American Society of Civil Engineers estimates a $3.7 trillion gap over 2024–2033 between what U.S. physical infrastructure needs to reach a state of good repair and forecasted investment [2], and worldwide infrastructure investment needs are forecast to reach $94 trillion by 2040 [12]. Delivering even a single campus building, such as a new university laboratory, takes electrical, mechanical, structural and civil engineers to size its power distribution and feeders, its heating, cooling and ventilation plant, its frame and foundations, and its site grading and stormwater systems. They work to legally binding codes and physical laws (Figure 1), and their designs are signed off by licensed professionals, because errors in the design are costly and potentially catastrophic. Engineering design services represent hundreds of billions of dollars in annual economic activity (U.S. engineering-services firms alone report over $300 billion in yearly revenue [34]), making professional physical engineering a high-value frontier for AI agents.

As Figure 1 illustrates across the four domains of PLE, a real-world physical engineering job requires far more than answering a textbook prompt: (A) Structural Engineering (Roof deck design) requires sizing steel roof decking, open-web joists and lateral diaphragm framing under gravity, snow-drift and shear loads so member demand stays within code capacity; (B) Civil Engineering (Site design) requires laying out a phased campus, access roads and utility corridors around delineated wetland buffers as a 3D IFC model; (C) Electrical Engineering (LV cable sizing) requires sizing low-voltage switchboards, overhead cable trays and 32 feeder and branch circuits under ampacity and voltage-drop limits; and (D) Mechanical Engineering (Cooling-tower LCC) requires selecting heat-rejection equipment, pumps and piping to meet thermal load within hydraulic flow limits while minimising 25-year life-cycle cost. Each task is long-horizon, spread across many documents in native formats, and combines retrieval, spatial and physical modelling, code compliance and cost optimisation, yet it is unusually measurable. Its deliverables are real engineering work products that can be checked against the governing documents and physics, so both the eligibility and the economic quality of a design can be scored objectively.

No existing benchmark measures this economically valuable work comprehensively, and the benchmarks that come closest are already too easy for leading agents. Existing engineering benchmarks for language models are mostly question answering over textbook problems or rule documents ([9] (Zhou et al. 2024; Zhou et al. 2025)), or self-contained toy design and optimisation problems whose full specification fits in the prompt ([8, 15]). Others target an isolated tool or modality, such as finite-element software ([26]), CAD generation ([10, 17]), circuit design ([6, 19]) or drawing revision ((Li et al., 2025)). Solving those benchmarks does not get real engineering design work done, i.e., none requires an agent to navigate a realistic, messy, multi-format project corpus and issue a complete, billable engineering deliverable. The general professional-work benchmarks GDPval ([31]) and APEX-Agents ([36]) are closest in spirit. They grade with expert pairwise preferences or expert rubrics. However, in physical engineering, a rubric of binary checks alone cannot tell a merely code-compliant design from an economically competitive one, e.g. two wire schedules can both satisfy every code check while one costs over 20% more to install.

We introduce PLE, a benchmark of 85 expert-grade professional physical engineering tasks for AI agents, spanning electrical, civil, structural and mechanical engineering. Unlike prior academic or single-tool benchmarks, PLE is built for realism and direct economic value: solving a task in PLE means completing a real-world engineering assignment. Each task places an agent in a sandboxed environment that holds a realistic project file system of PDFs, spreadsheets, CAD/BIM drawings (DXF/DWG/IFC), correspondence and logs. The agent has tool access for documents, spreadsheets and code execution, and must produce cited, production-format deliverable files such as engineering memos, calculation workbooks, equipment schedules or 2D/3D drawings. We build the benchmark through two complementary routes:

• Expert-Authored tasks. Practising engineers from the expert pool of our data vendor (Mercor) create tasks end to end inside realistic project “environments”, written as the real engineering requests a colleague or client would send on an active job. Each task is graded by a weighted rubric of binary, evidence-bound criteria (Vidgen et al., [36]) (§ 3.1).

• Co-Authored tasks. Practising engineers write the task briefs, and an adversarial Creator–Solver agent pipeline builds each task’s environment and grader from real, licensed engineering documents (Xu et al., 2026). Every task is iteratively redesigned until a frontier solver agent no longer saturates it, and passes an independent ground-truth audit. Mercor’s experts then validate or intensify the difficulty of every task (§ 3.2). These tasks are scored by both rubric-style constraints and an economic or performance objective, which complements the rubric-only scoring of the expert-authored tasks.

We evaluate 14 frontier models under seven publicly available agent harnesses (Figure 2). The best configuration, Claude Opus 5.5 in Claude Code, reaches a mean score of only 50.6%, which demonstrates the difficulty of real-world physical engineering work.

In summary, our contributions are:

1. PLE, the first agentic benchmark for realistic, economically valuable professional physical engineering work. It comprises 85 tasks across electrical, civil, structural and mechanical engineering, each grounded in a multi-format project corpus with a mean of 61 files.

2. A two-route curation methodology pairing expert authoring with human-agent co-authoring to scale up task creation and create objective-oriented, challenging tasks.

3. Comprehensive baseline results for 14 frontier models across seven agent harnesses, analysing per-discipline difficulty, run-to-run variance, performance-cost trade-offs, etc.

4. An open competition and leaderboard for AI physical-engineering agents. We evaluate submitted agent implementations on PLE, posting results to a public leaderboard.

Related Work

Agentic benchmarks for software and ML engineering. The closest agentic benchmarks evaluate digital engineering domains. SWE-bench and its variants [16, 25, 29] (Yang et al. 2024) measure software engineering on real repositories, and MLE-bench [7] and RE-Bench [39] measure machine-learning engineering; related suites cover terminals [23], operating systems and the web [41] (Milanon et al. 2023; Zhou et al. 2023), tool-user interaction (Yao et al., 2024), security (Zhang et al., 2024), scientific code [33] and paper replication [32]. These benchmarks established the sandboxed, tool-calling protocol that we also adopt, following code-asaction agents (Wang et al., 2024a,b; Yao et al., 2022). Their artifacts, however, are purely digital: code, models and their test suites, where success is a unit-test pass or a held-out metric. Professional physical engineering instead produces designs for physical systems, governed by laws of physics and legally binding building codes, and documented in production drawings, schedules and calculations that a licensed reviewer signs off on.

Benchmarks of professional work. A recent line of work measures economically valuable knowledge work. TheAgentCompany (Xu et al., 2024b) simulates a software company. GDPval [31] collects deliverable-producing tasks from experienced professionals in 44 occupations and grades them by blinded expert pairwise comparison. APEX [35] and APEX-Agents [36] have banking, consulting, legal and medical experts author tasks and rubrics. APEX-Agents runs them in long-horizon, cross-application “environments” through the Archipelago harness [22], which our environment builds on. HealthBench [5] and PaperBench [32] show that large expert rubrics, graded by a validated LLM judge [18] (Gu et al. 2024; Zheng et al. 2023), scale open-ended evaluation. PLE brings this focus on workplace realism and direct economic value to professional physical engineering: it adopts expert rubrics for its expert-authored tasks, while complementing half of the benchmark with an objective, programmatic measure of engineering design quality and cost efficiency.

Professional physical engineering. Few benchmarks touch physical engineering, and only in small slices (Table 1): 143 of EngiBench’s 1,760 textbook-style problems (Zhou et al., 2025), 13 generic member-sizing and topology tasks in EngDesign [15], and 3 of Frontier-Eng’s 47 simulator-based optimisation tasks [8]. ElecBench (Zhou et al., 2024) and DrafterBench (Li et al., 2025) cover narrow sub-tasks (power-system question answering and single-sheet drawing revisions), and building services and structures have LLM systems [21] and reviews [27] but no agentic benchmark. These benchmarks abstract away the multi-document project corpus and production deliverables of real practice, and they are close to saturation: the best agent beats Frontier-Eng’s baseline on 97.9% of its tasks, and leading models exceed 80% at EngiBench Level 2. PLE instead tests whether an agent can carry a professional job from raw project documents to a deliverable package checked against codes, physics and economic optimality (§ 3.2). Testing co-authored tasks against a frontier solver and keeping part of the benchmark private, like unpublished problem sets [11], guard PLE against saturation and training-data contamination [1] (Xu et al. 2024a).

Benchmark	#Physical eng. tasks	Real work tasks	Expert curation	Large input corpus	Large output artifacts	Objective function	Rubric
Software & ML engineering
SWE-bench (Jimenez et al., [16])	—	✓	—	—	✓	✓	—
MLE-bench (Chan et al., [7])	—	—	—	—	✓	✓	—
Professional work
TheAgentCompany (Xu et al., 2024b)	—	✓	—	✓	✓	✓	—
GDPval (Patwardhan et al., [31])	—	✓	✓	✓	✓	—	✓
APEX-Agents (Vidgen et al., [36])	—	✓	✓	✓	✓	—	✓
Professional physical engineering
EngiBench (Zhou et al., 2025)	143	—	—	—	—	✓	✓
EngDesign (Guo et al., [15])	13	—	✓	—	—	✓	—
Frontier-Eng (Chi et al., [8])	3	—	—	—	✓	✓	—
DrafterBench (Li et al., 2025)	1,920	—	—	—	✓	✓	—
ElecBench (Zhou et al., 2024)	288	—	—	—	—	✓	—
PLE (ours)	85	✓	✓	✓	✓	✓	✓

Data Curation

A task in PLE is designed to mirror a real, economically valuable engineering assignment rather than an academic exercise, and consists of four parts. The instruction is the brief a professional physical engineer would receive on an active project. The environment is a realistic project file system that holds the multi-format engineering corpus the agent must work from. The deliverable specification names the production work-product files the agent must issue so that real engineering design work gets done. The grader is either a weighted expert rubric or an executable verifier with withheld reference deliverables. All tasks are packaged in the APEX format [36] and run in Archipelago-style environment containers [22]. These expose the environment through MCP applications (filesystem, documents, spreadsheets, PDFs, mail, calendar, chat) and a code-execution sandbox.

Detailed example. Figure 3 shows a concrete task from PLE (MEP coordination, hall A with 8 input files): an MEP coordinator is handed the drawing set, schedules, schematics, design codes and raw 3D model of an equipment hall with 128 equipment cabinets and asked to deliver a fully coordinated 3D model. The engineer must read positions off the plans, sizes off the schedules and elevations off the section, and reconcile conflicts across the sheets. They must then route the cable tray, power busway, chilled-water mains and fire sprinklers through the ceiling space so that nothing clashes with the cabinets, columns, containment or sprinkler heads, every cabinet row and cooling unit is served, the sprinkler and electrical code clearances hold, and the routes are as short and economical as practical. The deliverable is a production-ready 3D model that a licensed reviewer and our executable grader can clash-check and measure against the governing documents, physical clearances and routing-length objective.

The tasks in PLE come from two curation approaches. Expert-Authored tasks are written end to end by practising engineers (§ 3.1). Co-Authored tasks start from briefs written by practising engineers, are built by our agent pipeline, and are then validated and have their difficulty intensified by practising engineers (§ 3.2).

Expert-Authored Tasks

Our first route relies entirely on human expertise to capture authentic workplace realism and billable engineering workflows. We collaborated with Mercor, which maintains a large pool of vetted domain professionals and has used it to build expert benchmarks of professional work [35, 36]. For PLE, Mercor recruited practising civil, electrical and mechanical engineers. Each expert built a task in three stages.

Task authoring. Experts first write the task as a realistic work request from a project colleague or client. The request describes the engineering situation with its constraints and specifies the exact deliverables needed to advance the project. For example:

“Garrett wants the Phase 3 estimate out to the surety broker and I don’t want it going anywhere until Section 2 has been checked against what’s actually on the plans. […] Give me the corrected Section 2 subtotal, a line-by-line table of every quantity or unit-price change with the sheet it comes off, and the revised rounded bond figure. [...] Put it in a memo to Callaway for my signature, dated 10/31/2026, saved as Correspondence/Client/Memo_to_Callaway_2026-10-31.docx.”

Answering this request requires cross-referencing a cost estimate (XLSX), three drainage sheets of a nine-sheet plan set (PDF) and a hydraulic-grade-line report. The solver must then apply the stated pricing rules and write a signed memo. The expert-authored tasks cover plan-review responses, submittal reviews, capacity and energization decisions, geotechnical and foundation checks, tank shell-thickness review, close-out letters and cost positions.

Environment building. The documents a request depends on live in its environment: a realistic project file system of the kind an engineer inherits on joining a job. Experts build the environment around the task. It contains drawing sets, calculation reports, equipment and motor schedules, test reports, cost estimates, meeting minutes, e-mail threads and superseded revisions, organised in folders such as Design-Drafting/, Cost Estimates/, Reports-Studies/ and Correspondence/. Each task comes with 63 environment files on average (range 50–112) in six native formats, mostly PDF and XLSX (Table 2). The 44 tasks share 12 environments, so a shared environment counts once for each task that uses it.

Metric	Expert	Co-authored	All
Tasks	44	41	85
Environments	12	41	53
Corpus per task
Env files, mean	62.8	58.7	60.8
Env files, max	112	207	207
Grading
Grader	rubric	rubric + exec.	hybrid
Checks per task, mean	21.4	29.9	25.3

Grading. Each task is graded by a weighted rubric of binary verifiers. Every verifier takes as input the deliverable file it reads, a single checkable criterion (e.g. “the memorandum states that the corrected Section 2 subtotal is $1,303,717.70”), a severity and a numerical weight. Load-bearing numbers carry high weights (10 in the example above) and presentation items carry low ones (1). The 44 expert-authored tasks carry 941 rubric items, 21.4 per task on average. A task’s score is the weight-normalised fraction of verifiers passed, as judged by an LLM that reads the deliverable file itself. Each task went through Mercor’s quality control before delivery.

Co-Authored Tasks

Expert authoring yields realistic tasks, but it scales with expert hours. Our second route co-authors each task between practising engineers and an agent pipeline. Engineers supply the task brief at the start and validate the finished task at the end; in between, frontier LLM agents operate under a written method and are checked by programs and independent agents at every stage. A Task Creator agent builds each task in the same three stages as the experts: task authoring, environment building and grading (Figure 4). A Task Solver agent then calibrates its difficulty, and an independent audit checks its ground truth. Mercor’s experts finally validate and intensify the difficulty of the output. Both agents use Claude Opus 5: the Task Creator runs in Claude Code, and the Task Solver runs in Hermes Agent ([28]), one of the harnesses we use for the baselines (section A).

Task authoring. Each co-authored task starts from an expert seed prompt submitted by a practising engineer through our web-based task nomination interface (Figure 9), which captures the engineering situation, workspace inputs, optimisation goal, physical/code constraints, verification method, and expected deliverable format. Based on this seed prompt, the generated task instruction states the engineering concerns and names the governing documents, but leaves the solver to find the applicable requirements in the environment. Four ground rules govern creation. (i) Ground in reality: every task is built on real, licensable artifacts (federal criteria, published standards, public drawing sets, vendor data, site surveys). (ii) Authentic difficulty: difficulty must come from the inherent complexity of the artifacts, never from vague instructions or hidden grading traps. (iii) *Engineered tasks, not retrieved answers: the agent is given ingredients such as geometry, properties and limits, and it must model, integrate or optimise its way to an answer that cannot be looked up. (iv) Verifiable deliverables: all deliverables can be verified with deterministic simulators or objective rubrics, against a reference deliverable whose ground truth is itself audited.

Environment building: deep research. The environment is built by a deep-research stage. It first maps the practising engineer’s workflow to the document classes consulted at each step, including a public example of the deliverable’s own industry format. It then harvests a corpus in native formats. Each source’s licence is confirmed from the publisher’s own page, and an unresolved licence excludes the source. An accepted environment must have at least three distinct formats, distractors such as superseded revisions and unrelated sheets, a realistic directory tree (criteria/, specs/, drawings/, vendor/, …), and information spread across documents so that the solver must cross-reference them.

Verification and grading. Deliverables (CSV or XLSX schedules, DXF drawings, JSON design records and calculation books) are checked against independent constraints per task, drawn from distinct sources or distinct physical or regulatory concerns (target 20). The constraints mix code limits, physical feasibility, interfaces between documents, operational and procurement limits, and deliverable format. Each constraint is tagged hard (safety and feasibility) or soft (documentation and conventions), traced to its source row, and checked by the grader. The solver must also explicitly list the requirements it applied, each with clause and value from the associated grounding document. The grader checks the listed requirements against the references, so successful navigation of the corpus becomes a metric in the final grading.

Rubric-only scoring, as used for the expert-authored tasks and by Mercor’s rubric graders, awards the weighted fraction of binary criteria met. For design work this has a blind spot. When both designs fulfil the same set of rubrics, it cannot distinguish a better design from a merely feasible one. Our co-authored tasks are instead scored on both constraint compliance and an objective score as

where $p \in[0,1]$ is the fraction of all constraints passed (as in rubric scoring), $G \in\{0,1\}$ is the hard-constraint (safety, feasibility) gate, and $O' \in[0,1]$ is the objective score. $G$ is necessary because without it, a design can hack the objective score by delivering something unsafe or infeasible, which is detrimental in a real-world task. A task’s score is the mean of $s$ over the items of its deliverable. $O'$ measures how good the design is, and the larger 0.6 weight prioritises design quality and hard-constraint satisfaction. For a task that asks for the cheapest compliant design, the raw objective is $O = c_{\mathrm{bound}}/c$, where $c$ is the cost of the agent’s design and $c_{\mathrm{bound}}$ is a certified lower bound on the cost of any compliant design. $O$ lies between 0 and 1 and reaches 1 only when the design is as cheap as the bound. We normalise it against the task’s naive baseline,

where $O_{\mathrm{naive}}$ is the objective of the un-optimised but compliant design from the task’s calibration ($O_{\mathrm{naive}} = 0$ where a task does not come with a naive design). The intuition of this objective function is that a naive design that fulfils all constraints earns no objective credit, and $O'$ measures the quality lift of a good design from a naive one.

The Creator–Solver loop. Figure 4 (stages 2 and 3) shows this loop. The Creator drafts all three components together: the task statement, the environment, and an executable grader with its reference deliverable. An independent Task Solver then attacks the task by trying to solve it. It makes $N$ parallel attempts (typically $N=3$) in sandboxed containers, with network access restricted to the model endpoint and no access to the grader, rubrics or references. A task meets the difficulty target only if the mean of the attempts stays below the difficulty bar. The creator reads the solver’s trajectories (which file it opened, which shortcut or exploit it found). The creator checks a solved task and identifies the component that made the task easy. If a component’s difficulty can be intensified, the task undergoes another round of redesign by modifying the objective function or adding constraints to the grader. If the environment itself is too thin, the task returns to deep research, which finds more diverse context documents to make the task more challenging.

Mercor validation and difficulty intensification. Every co-authored task is delivered to Mercor for independent expert validation. An expert reviews the prompt, environment files, grader and reference deliverables, and confirms that they are realistic and consistent with each other. A task that can be made harder has its difficulty intensified: the expert adds their own items and modifies the prompt, environment, grader or reference deliverables accordingly.

Data Analysis

Table 2 summarises PLE. The two curation approaches contribute 44 expert-authored and 41 co-authored tasks set in 53 distinct environments: the expert-authored tasks share 12 project environments, while every co-authored task has its own. Each environment is a corpus, not a prompt attachment. In total the agent faces 5,170 environment files, 61 per task on average and up to 207. Expert-authored tasks are graded against 941 rubric items (21.4 per task), and the 37 constraint-graded co-authored tasks against 29.9 constraints per task on average, on top of their objective score (25.3 checks per task overall). Every task asks for one or more deliverable files, and none can be answered with a simple LLM call.

The two approaches complement each other in discipline (Figure 5). Mechanical (30 tasks) and electrical (27) engineering are the largest disciplines, followed by civil (18) and structural (10). The expert-authored tasks are concentrated in mechanical, electrical and civil practice, while all ten structural tasks, and the building-systems design work in the other disciplines, come from the co-authored route.

The two approaches also stress different skills on the input and output sides. Expert-authored environments are document-centric (Figure 6): 51.6% PDFs, 25.2% spreadsheets and data tables, and 22.4% text, Word documents and e-mail. Co-authored environments balance PDFs (31.2%), spreadsheets and data (27.5%) and text (16.3%) with CAD and BIM models (12.9%), schematics and imagery (7.1%), web portals (2.5%) and code, and are an order of magnitude larger on disk (38.1 MB per task against 3.0 MB). Both include distractors such as superseded revisions and unrelated sheets, so the agent must decide which document governs before it computes anything. On the output side (Figure 7), the 85 tasks require 249 deliverable files. Expert-authored tasks mostly issue a single formal report (.docx, 77.6%), workbook (12.2%) or CAD model (10.2%), 1.1 files per task, whereas co-authored tasks issue multi-file engineering packages of 4.9 files on average: schedules, calculation workbooks and data files (62.5%), design narratives and specifications (21.0%), 2D CAD drawings and 3D BIM models (13.0%), executable scripts (2.0%) and plotted sheets (1.5%).

Results

Experimental Setup

Agent harnesses. We evaluate 14 models, Claude Opus 5.5, Claude Opus 5, Claude Sonnet 5.5, Claude Fable 5, GPT 6 Astra, Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.1 Pro, Grok 4.7, GLM 5.2, Qwen3 Coder 480B, MiniMax M2, Kimi K2 Thinking and Llama 4 Maverick, each under seven publicly available agent harnesses (GPT 6 Astra only in Codex and Grok 4.7 only in its own harness, Grok Build (xAI, 2026)): Antigravity (Google, [13]), Claude Code (Anthropic, [4]), Codex (OpenAI, [30]), Hermes (Nous Research, [28]), OpenCode (Anomaly, 2025), OpenHands (Wang et al., [38]) and Pi (Zechner, 2025). Crossing models with harnesses separates how much of the score depends on the model and how much on the scaffolding around it. Each harness runs inside the task’s container with the environment mounted. Docker-environment tasks expose their applications (filesystem, documents, spreadsheets, PDFs, mail, calendar, chat) as MCP tool servers; Pi has no MCP support and reaches them through the shell. During the agent phase the container can reach only a model proxy, which pins one reasoning effort for every harness; every run has a 6-hour agent budget. When the agent exits, its deliverable files are collected and graded outside the agent’s container, either host-side or in a separate verifier container that receives only the task’s declared artifacts, so the solver can never reach the grader or the reference deliverables. Harness versions and settings are listed in section A.

Grading. The agent sees only the instruction and the environment, with no rubric, verifier or score feedback. Rubric items of the expert-authored tasks are judged by Claude Opus 5.5, which reads the deliverable file itself. Co-authored tasks are graded by their frozen executable graders and reported with eq. (1); twelve of these graders also ask Claude Opus 5.5 to judge three to six narrative items, which count as soft constraints in $p$. The two site-design and two MEP-coordination tasks keep their graders’ own scores, which give 0% to an MEP routing with a hard clash or an unserved row and at most 50% to an infeasible site design. Task scores lie in $[0,1]$ and we report them in percent. Every model runs every task three times in each of its harnesses, and we report the mean. A run that fails for infrastructure reasons (e.g. a provider error before the first model call) is re-run; a run that ends without a gradable deliverable scores 0%. All results are over the 85 tasks of PLE (Table 3).

Harness	All (85)	Expert (44)	Co-authored (41)	Electrical (27)	Civil (18)	Mechanical (30)	Structural (10)
Pi	47.3	49.2	45.3	46.7	60.7	40.5	45.0
OpenCode	47.4	50.6	44.1	47.9	62.1	39.2	44.4
Antigravity	48.4	50.8	45.8	49.5	62.1	41.4	41.6
Hermes	48.6	49.2	48.0	51.9	65.1	38.4	40.8
Codex	49.3	50.6	48.0	53.1	67.8	36.6	43.8
OpenHands	49.9	50.7	49.1	50.6	67.0	40.2	46.6
Claude Code	50.6	53.1	48.0	50.7	69.9	40.5	46.6

Main Results

Figure 2 places all 86 model (harness) pairs by score and cost, Table 3 breaks down the strongest model by harness, curation approach and discipline, Figure 8 shows its per-task distribution, and Table 5 lists every pair.

Model	Agy	CC	Cdx	Hrm	OC	OH	Pi
Claude Opus 5.5	48.4	50.6	49.3	48.6	47.4	49.9	47.3
GPT 6 Astra	–	–	48.3	–	–	–	–
Claude Sonnet 5.5	45.2	43.6	42.7	43.7	42.3	42.2	43.5
Gemini 4 Argon	41.7	43.2	40.3	41.1	38.4	31.1	40.2
Claude Opus 5	42.6	40.8	38.8	39.6	39.8	41.9	40.4
Claude Fable 5	33.8	34.9	34.7	32.8	34.5	33.4	33.0
Grok 4.7†			32.0 in Grok Build
Gemini 3.8 Flash	26.4	28.2	27.6	17.8	26.0	27.6	26.9
GLM 5.2	23.0	22.3	25.3	23.9	24.4	22.9	23.8
Gemini 3.1 Pro	13.7	14.1	14.1	14.2	13.3	14.0	13.6
Qwen3 Coder 480B	4.5	5.6	4.4	6.2	4.1	5.7	4.3
Kimi K2 Thinking	3.5	4.4	3.0	5.2	5.9	4.7	4.2
MiniMax M2	3.7	5.3	4.4	5.3	3.9	4.5	4.7
Llama 4 Maverick	0.1	0.4	0.0	0.1	0.0	0.7	0.7

Performance and cost. Three proprietary model families hold the upper performance-cost frontier (above $2 per task, as shown in Figure 2). At the top, Claude Opus 5.5 scores 50.6% at $17.18 per task in Claude Code and 49.9% at $14.82 in OpenHands, while GPT 6 Astra in Codex comes within 2.3 points of the top score at about half the cost (48.3%, $9.15). Gemini 4 Argon covers the $6–9 band across four harnesses (40.2%–43.2%), and Gemini 3.8 Flash in Claude Code retains 56% of the top score for 15% of its cost (28.2%, $2.55). Below $0.20 per task, the open-weight models on the frontier score under 7%. Higher cost does not always yield a higher score: the eight most expensive configurations, all Claude models in Antigravity, Claude Code or Codex, cost $27–54 per task yet score only 33.8%–49.3%, all below Claude Opus 5.5 in Claude Code ($17.18 per task).

PLE is far from saturated. The strongest configuration, Claude Opus 5.5 in Claude Code, averages 50.6% (Table 5), ahead of the best configurations of the next two models: GPT 6 Astra in Codex (48.3%) and Claude Sonnet 5.5 in Antigravity (45.2%). In Claude Code (Table 3), Claude Opus 5.5 has a median task score of 43.5% and scores below 50% on 48 of the 85 tasks; even taking the best harness separately for each task leaves 35 tasks below 50%. Two tasks score 0% in every run, and seven score above 90%. By discipline, mechanical (40.5%) and structural (46.6%) work are the hardest, while civil is the easiest (69.9%). Co-authored tasks are harder than expert-authored ones in every harness (48.0% versus 53.1% in Claude Code).

The gap between models. The model matters far more than the harness (Table 5). Taking each model’s best harness, scores fall from 50.6% (Claude Opus 5.5) and 48.3% (GPT 6 Astra) to 45.2% (Claude Sonnet 5.5), 43.2% (Gemini 4 Argon) and 42.6% (Claude Opus 5). They then drop to 34.9% for Claude Fable 5, 32.0% for Grok 4.7, 28.2% for Gemini 3.8 Flash, 25.3% for GLM 5.2 and 14.2% for Gemini 3.1 Pro, while Qwen3 Coder 480B, MiniMax M2, Kimi K2 Thinking and Llama 4 Maverick stay below 7%. This 50-point spread across models far exceeds the 0.7–12.1 point spread across harnesses for a fixed model (median 3.0). Costs vary just as widely (Figure 2): the best configurations of Claude Opus 5.5 and GPT 6 Astra cost $17.18 and $9.15 per task, while Gemini 4 Argon reaches 43.2% for $8.21 and Gemini 3.8 Flash reaches 28.2% for $2.55.

The harness’s effect. For Claude Opus 5.5, the seven harnesses span only 47.3%–50.6%, much less than the spread across tasks (Figure 8). Harness rankings also vary across models (Table 5). Claude Code is the best harness for Claude Opus 5.5, Gemini 4 Argon, Claude Fable 5 and Gemini 3.8 Flash; Antigravity for Claude Sonnet 5.5 and Claude Opus 5; Hermes for Gemini 3.1 Pro and Qwen3 Coder 480B; Codex for GLM 5.2; and OpenCode for Kimi K2 Thinking. MiniMax M2 (Claude Code and Hermes) and Llama 4 Maverick (OpenHands and Pi) each tie between two harnesses. Runs that score 0% contribute to these differences; they end without a gradable deliverable or fail all-or-nothing gates such as those of site design and MEP coordination. Out of 255 runs per harness, Claude Opus 5.5 has 16 such runs in Pi, 15 in Codex, 12 in OpenCode, 11 in Claude Code, 8 in Hermes, 7 in Antigravity and 6 in OpenHands.

Analysis

Run-to-run variance is substantial for single tasks. Set-level means are stable: for Claude Opus 5.5, each harness’s mean has a standard deviation of 0.3–1.8 points across its three runs (e.g. Claude Code 52.0% / 49.3% / 50.6%). Individual tasks, by contrast, vary widely. The mean range of a task-harness cell over three runs is 15.4 points, and 105 of the 595 cells span 30 points or more. Reported per-task numbers should therefore always be averages over repeated runs.

Co-authored tasks demand far more work. For Claude Opus 5.5 in Claude Code, an attempt at a co-authored task consumes 39.3M tokens and 97 minutes of agent time, compared with 15.6M tokens and 31 minutes for an expert-authored task (27.0M tokens and 63 minutes overall). Harnesses also differ widely in inference cost at similar scores (Figure 2). For Claude Opus 5.5, Claude Code ($17.18/task, 50.6%) and OpenHands ($14.82/task, 49.9%) sit on the performance-cost frontier, while OpenCode ($12.53/task, 47.4%) and Pi ($13.33/task, 47.3%) come close at lower cost; Codex, by contrast, averages $35.15/task (49.3%).

Effect of the rubric judge. We re-graded all 1,159 LLM-judged runs of Claude Opus 5.5 across the seven harnesses using a second pass of Claude Opus 5.5, Claude Opus 5, and Gemini 4 Argon. The second Claude Opus 5.5 pass reproduces 98.7% of item verdicts (Cohen’s $\kappa= 0.97$) with an unchanged mean. Against it, Claude Opus 5 and Gemini 4 Argon agree on 95.8% and 95.5% of items ($\kappa= 0.90$), raising re-graded scores by 3.3 and 1.3 points (2.1 and 0.9 overall), almost entirely on rubric-graded tasks. Claude Code ranks first under every judge: Claude Opus 5 preserves the harness order (Kendall’s $\tau= 1.0$), and Gemini 4 Argon swaps only Codex and Hermes ($\tau= 0.90$). On the 44 rubric-graded tasks in Claude Code, switching to Gemini 4 Argon raises its score by 3.6 points versus 1.3 for Claude Opus 5.5, a 2.3-point combined self-preference.

Where agents fail. Across all runs of the 86 model (harness) pairs, 23.2% score 0%. Of these, 25.3% end without a gradable deliverable, 4.3% hit the agent’s time budget or an agent-side error (a crashed harness, an exhausted context window or an output limit), and 70.4% deliver files that the grader scores 0%. This last group comprises 2,020 runs on tasks scored with eq. (1), 817 on rubric-graded tasks and 726 at the all-or-nothing gates of the site-design and MEP-coordination graders. The strongest configuration, Claude Opus 5.5 in Claude Code, rarely fails outright, scoring 0% in only 11 of its 255 runs. Its losses stem instead from hard constraints: on co-authored tasks, its deliverable items satisfy 91% of their constraints on average, yet only 41.3% of the items satisfy every hard constraint and so earn objective credit under eq. (1). Producing a design that is fully safe and feasible while pushing its cost toward the bound remains the core difficulty of PLE.

Conclusion

We introduced PLE, a benchmark of 85 agentic tasks that evaluates whether frontier models can perform the realistic, economically valuable work of professional physical engineers. Solving a task requires an agent to navigate a multi-format project corpus, reconcile the governing physical and code requirements, and issue production-ready engineering deliverables. The tasks come from two complementary curation approaches: expert-authored tasks written end to end by practising engineers and graded with weighted rubrics, and co-authored tasks built from engineers’ briefs by an adversarial Creator–Solver agent pipeline and then validated and intensified by experts. For design tasks, a hybrid score rewards a cost-effective, fully compliant design rather than a merely admissible one.

Across 14 frontier models and seven agent harnesses, only one configuration exceeds 50%: Claude Opus 5.5 in Claude Code, at 50.6%. The model matters far more than the harness, yet the harness still shifts scores and costs substantially; results on PLE should therefore always be reported with both the model and the harness. The main obstacle remains producing a design that satisfies every hard constraint while still optimising its cost, leaving substantial headroom for future agents.

Ethical discussion. No task contains proprietary or client-confidential information: expert-authored environments are written from scratch by the participating engineers, and every file in a co-authored environment comes from a public-domain or permissively licensed source, or is generated for the task, under a licence that permits redistribution. The engineers were paid through Mercor at professional rates: on average $4,663 per expert-authored task and $3,157 per co-authored task for validation and difficulty intensification.

Acknowledgements

We are grateful to Michael Colesanti, Dan Roush, Jeff Thiele, Amanda Gunckle, Julie Sellers, Daniela Zellers, Xia Fang, Patricia Perez, and Shilpa Seroha, who helped review and contributed to this work.

References

Evaluation Configuration

Table 4 lists the settings behind every result in § 5. All 14 models are served through one model proxy, the only endpoint the agent container can reach, which pins each model’s highest reasoning setting in every harness. Each model and harness pair runs every task three times with a 6-hour agent budget per run. After the agent exits, Claude Opus 5.5 judges the rubric items and co-authored tasks are scored by their frozen executable graders outside the agent container (§ 5.1).

Setting	Value
Agent models	14 models: claude-opus-5-5, claude-opus-5, claude-sonnet-5-5, claude-fable-5, gpt-6-astra, gemini-4-argon, gemini-3.8-flash, gemini-3.1-pro-preview, grok-4.7, glm-5.2, qwen3-coder-480b, minimax-m2, kimi-k2-thinking, llama-4-maverick; all served through one model proxy
Harnesses	Antigravity 1.3.3, Claude Code 2.1.296, Codex 0.162.1, Hermes 0.22.2, OpenCode 2.0.26, OpenHands 1.54.0, Pi 1.1.0, Grok Build 1.0.52
Reasoning effort	each model’s highest reasoning setting (Qwen3 Coder 480B and Llama 4 Maverick have none)
Network during agent phase	no-internet
Agent budget	6 hours (21,600 s) of agent time per run
Runs per task, model & harness	3
Rubric judge	claude-opus-5-5 for every model, reads the deliverable file
Co-authored grading	frozen executable grader, eq. (1), outside the agent container

Results for Every Model and Harness

Table 5 gives the mean score of every model and harness pair on all 85 tasks; the model and harness comparisons in § 5 are drawn from it.

Agy = Antigravity; CC = Claude Code; Cdx = Codex; Hrm = Hermes; OC = OpenCode; OH = OpenHands. Dashes: not run. †Grok 4.7 runs only in its own harness, Grok Build.

Expert Seed Nomination Interface

Expert Seed Nomination Interface

To ensure that every co-authored task in PLE originates from an authentic, economically valuable engineering problem rather than a synthetic prompt, practising physical engineers nominate seed tasks through our web-based Task Nomination Interface (Figure 9). Rather than writing a free-form prompt, the nominating engineer fills out six structured specification fields:

1. Situation: Realistic engineering background, project context, and operational persona for the assignment.

2. Inputs: High-level description of the starting project assets the engineer would expect in the workspace (e.g. CAD/BIM models, schedules, geotechnical or weather data).

3. Goal: Quantifiable engineering target or economic optimisation objective (e.g. minimising installed material cost, peak load, or routing length).

4. Constraints: Governing building codes, physical feasibility boundaries, safety limits, and hardware compatibility rules.

5. Verification: High-level specification of how an automated deterministic script, simulator, or rubric should evaluate the submitted deliverable.

6. Expected Output: Exact production deliverable formats required from the solver agent (e.g. XLSX schedules, DXF drawings, IFC4 BIM models).

Once submitted, this structured nomination serves as the Expert Seed Prompt (Stage 1 in Figure 4) that guides the Task Creator agent’s deep research, multi-format environment construction, and executable verifier synthesis.