betaFeedback

WorksComputer Science

Embodied First Exam

Abstract

Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.

Website: embodied-first-exam.ai Code: github.com/mll-lab-nu/embodied-first-exam Trajectory Data: huggingface.co/datasets/MLL-Lab/embodied-exam-data

Introduction

Robots acquire new skills either by manual programming or by learning from data, as in recent vision-language-action models [10, 12, 36, 67, 110, 122] and world action models [133, 159, 182]. Coding agents powered by large language models suggest a third way. They already complete challenging software engineering tasks in the digital world [64, 73, 102, 104]. What makes these agents promising for robotic engineering is their capabilities to turn instructions into executable code and revise their plans through automated experimentation [84, 141, 151]. Recent early success in agentic robots suggests such a potential paradigm shift, where agents directly control robots [47, 130, 191, 199], write robot programs [32, 48, 62, 101, 145, 167], or train policies [83, 166]. This motivates a systematic assessment of the capability boundary of coding agents:

What robotic tasks can general-purpose coding agents solve, under what physical-world constraints, and where do they fail?

Existing benchmarks do not yet fully address this question. The capability boundary cannot be established from a focused set of robotics tasks [17, 59, 76, 87, 106, 184], nor from automated programming tasks without imposing physical-world constraints [14, 64, 102, 117, 169, 198]. This requires defining which robotic capabilities to evaluate, measuring how physical-world constraints affect success, and supporting different models and harnesses through task interfaces.

To answer this question, we introduce the Embodied First Exam (EFE), a unified evaluation framework that connects coding agents to simulated robots (Figure 1), following three principles.

Figure 1. Overview of the Embodied First Exam, a comprehensive evaluation framework for coding agents on robotics tasks. It is large and diverse, spanning 1,156 tasks, 35 suites and 47 robots, and easy to run and evaluate: one command runs any coding agent, and each task can be graded in up to three evaluation settings.

First, include diverse tasks that go beyond pick-and-place. Spatial puzzles test reasoning about spatial constraints, while sim-to-sim engineering tests how agents revise controllers from execution feedback under unfamiliar dynamics. Dynamic tasks test responses to a changing world. Multi-robot interaction and whole-body control test coordination. EFE therefore contains 1,156 tasks in 35 suites on 47 robots. It combines 26 suites adapted from existing robotics benchmarks with 9 new suites covering these capabilities and safety.

Second, measure how success depends on physical-world constraints. Its tasks should follow the constraints of a real robot as closely as possible. Success with full simulator state and resets does not establish what an agent can achieve through partial observations, with no resets, or while the dynamic world keeps running. We therefore define up to three evaluation settings per task. Standard Setting follows real-robot conditions: the agent observes the environment only through the robot’s sensors and cannot reset it by default. The simulation waits while the agent reasons, and its Realtime variant keeps the environment running while the agent makes decisions. Privileged Setting provides an approximate upper bound and can be used to generate training data, where the agent has access to the full simulator state and unlimited resets, allowing it to search for an oracle solution by trial and error. We call each <task–setting> pair a test. Tests of the same task share the initial scene and success rule, allowing us to compare success as state access, reset limits, and timing change.

Third, provide unified infrastructure for different models, harnesses, and control approaches. With a single command, EFE runs different combinations of models and harnesses across diverse robotics tasks. Given a natural-language task instruction, the agent chooses its control approach (Figure 2). In Direct control, it directly outputs actions after receiving observations [47, 191, 199]. In Code to control, it outputs a planner, controller, or other program that generates the robot’s actions [84]. In Train a policy, it trains a policy in simulation and uses it to control the robot [83, 95]. The framework enforces the access permitted by each evaluation setting, and a verifier outside the agent’s control grades every attempt [196, 203].

Figure 2. Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs mark the approaches used (model is direct control), and the framed line is a policy.

We evaluate eight coding agents on a core set of 456 hard tasks. The strongest agent, GPT-6 Astra [116] in Codex, solves 77% of the Privileged tests but only 30% of the Standard tests of the same 391 tasks. Apart from the three strongest agents, GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon, the best result comes from Kimi 3.1 Pro, which solves 9% of the Standard tests of the same 391 tasks (34 tests, Figure 11) and 7% of the Realtime tests (Figure 8). The Realtime Setting is harder still: 63% of its tests remain unsolved by any agent, compared with 50% of the Standard tests of the same 54 tasks. To identify where failures occur within a task, we divide each task into stages. Each stage corresponds to a change in the world required to complete the task. For example, the task of putting a tomato into a closed fridge has three stages: the fridge is open, the tomato is inside, and the fridge is closed again. For each failed attempt of the eight agents on 427 tasks, a separate coding agent labels the stage at which the failure occurred. Our main findings are as follows.

Figure 8. Success rate of each agent on the core set, with solved tests in parentheses. The harness dsh is DeepSeek Harness, DeepSeek’s own harness.

Figure 11. The 391 core tasks that have both a Privileged and a Standard test, split for each agent by the settings in which it solves them.

  • Privileged versus Standard (Section 3.2). Resets help GPT-6 Astra more than the exact object state does, and with both, its success rate closes most of the gap to the Privileged Setting. Stages that need precision become much harder in the Standard Setting. At these stages, resets help, but the exact state alone does not.

  • Where agents fail (Section 3.3). Failures on a task tend to happen at the same stage, usually the first stage of the task. This stage also tends to be the same in the Privileged and Standard settings. Stages that change a material or move the robot’s body have the highest failure rates.

  • Differences between agents (§ 3.5). Weaker agents are much more likely than stronger agents to fail before making the required contact. In the Standard Setting, the three strongest agents solve partly different sets of tests. Switching harnesses can change a model’s success rate by up to 12 percentage points.

  • How agents control the robot (§ 3.7). In the Standard and Realtime settings of a soccer task, agents trained a policy when given a GPU and a training library, and only trained policies won. Agents mostly control the robot directly in the Standard Setting and write programs to control it in the Privileged and Realtime settings.

The Embodied First Exam: A Unified Evaluation Framework for Coding Agents in Robotics

In this section, we describe how we evaluate coding agents on robotics tasks (§ 2.1), the resource constraints of each task (§ 2.2), and the tasks of the exam (§ 2.3).

Framework and Evaluation Protocol

Existing robotics benchmarks focus on visuomotor policy learning with a relatively fixed input-output interface, whereas we evaluate coding agents that can choose their own control approach. Our design principle is therefore to give the agent broad freedom over how it acts and to constrain only the result: we fix where it starts, what resources it gets, and how success is checked. To achieve this, we package every robotics task in the format of Harbor [44], the evaluation framework of Terminal-Bench [102], which runs the task in containers, as shown in Figure 3a. The agent has its own workspace, where it can install packages, write and run programs, and train models. Each task starts from a fixed initial state; a verifier outside the agent’s container reports whether the agent’s answer meets the task’s success rule.

The EFE framework and evaluation protocol

Figure 3. The EFE framework and evaluation protocol. (a) Our unified framework supports different models, harnesses, control approaches, and simulators. (b) In Privileged, the agent and simulator share a container, and the verifier replays the submitted answer separately. In Standard and Realtime, the simulator and verifier are separated from the agent and grade the live episode.

On top of this framework, we define a unified evaluation protocol with up to three settings per task (Figure 3b). We design Privileged as an approximate upper bound on performance, with full access to the simulator for trial and error, while Standard and Realtime make the evaluation as close to a real robot as possible. The bound holds over a set of tasks, not for every single task, because an agent can solve a task in Standard that fails in Privileged (Section D). For a given task, all settings share the same initial scene, success rule, and time limit, so their results are paired. They differ in what the agent observes, whether it can reset, whether the world waits, and where the simulator and the verifier run, never in the control approach the agent may choose. We ablate the first two differences in Section 3.2.

Privileged. In Privileged, the agent runs in the same container as the simulator, so it can read the full simulation state and reset without limit. The agent must submit a low-level robot trajectory (or a controller program for some tasks). The verifier replays the trajectory (or runs the controller) in a separate container to determine task success.

Standard. In Standard, we run the simulator as a service in a separate container. Through predefined APIs, the agent has limited access to the simulator: it can query proprioceptive state (e.g., joint positions, velocities, and torques), and sensor data (e.g., calibrated RGB-D images), and send action chunks to control the robot. The simulation is paused between API requests, so the agent faces no real-time pressure. Resets are disallowed unless the task provides an explicit reset budget. The verifier grades task progress online as the agent controls the robot, and in most suites the interface reports a success to the agent and ends the episode.

Realtime. Realtime is the same as Standard, except that the simulation does not pause between agent actions but keeps running in real time; for computationally expensive tasks, it runs at a fixed fraction of real time (e.g., 0.5×0.5 \times). Due to compute constraints, we currently evaluate Realtime only on dynamic tasks. Section C details each setting.

Resource Constraints for Coding Agents

As shown in Figure 4, we set resource constraints for each task: a time limit and a token budget for the agent, a reset budget and, in some tasks, a step limit for the robot (RoboDojo’s own limit in Figure 4, or a cap on the episode length that the instruction states), and caps on CPU, memory, GPU memory, and disk. The token budget is high enough that no attempt reaches it. The time limit is the same in every setting of a task, and when it runs out we stop the agent and grade the result it has at that point. Resets are unlimited in Privileged. In Standard and Realtime, there are no resets unless the task grants a budget, which nine suites do, with 3 to 50 resets. We set the compute caps per task from its measured needs and monitor usage throughout the run (Section C).

Figure 4. Resource constraints across the exam. The large number and the tick give the median over tests (for GPU memory, over the tests that reserve a GPU), the line spans the range, and the dark box spans the middle half of the tests.

We observe that coding agents may choose different control approaches under different resource constraints and in different settings (Figure 2 and § 3.7). For example, if we provide a GPU, agents can train a policy to control the robot (Figure 19).

Figure 19. GPT-6 Astra in Codex on the two soccer versions in the Standard Setting. Each row gives the approach and result, two replay frames and the goals over match time.

Task Suites

The exam contains 1,156 tasks in 35 suites. A suite is a set of related tasks on one simulator. Each task states a goal, usually in natural language, such as untying a knot or turning the page of a book with a dexterous hand, and the agent must control the robot to reach that goal. We built 9 suites with 411 tasks for the exam and adapted the other 26 suites with 745 tasks from existing robotics benchmarks. Together, the tasks use 47 robots of nine types (Table 2 in Section A, which lists the simulator, the robots and the number of tasks of every suite).

Table 2 | The suites of the exam. The 9 suites that we built come first, followed by the 26 suites adapted from existing benchmarks, which Section A.2 cites. The Kind column gives the kinds of robot body in the suite, nine over the exam (mobile manip. = mobile manipulator; car = wheeled car). The Robots column counts distinct robot models, the Tests column counts tests over all tasks, and the Core column counts tasks in the core set.

SuiteSimulation backendKindRobotsTasksTestsCore
DexHandExamMuJoCohand3306023
Embodied Olympics - FunMuJoCohumanoid27917956
Embodied Olympics - MatchMuJoCoarm, humanoid, mobile manip.5144113
Embodied Safety ExamMuJoCoarm, two arms, mobile manip.4727650
iSpatialGymOmniGibson (Isaac Sim)mobile manip.112248
RoboARCMuJoCoarm, two arms310621258
RoboDynamicExamManiSkill / SAPIENarm, two arms2256910
RoboEngineerMuJoCo + learned dynamicsaerial, car, humanoid, quadruped5303015
RoboPaintManiSkill / SAPIENarm1438622
ABC SimMuJoCotwo arms129600
AM-BenchIsaac Lab / Isaac Simaerial19172
AutoBioMuJoCoarm, two arms49180
BEHAVIOR-1KOmniGibson (Isaac Sim)mobile manip.110020071
BiGym 2.0MuJoCohumanoid1133910
DexJoCoMuJoCohand211222
DexToolBenchMuJoCohand1214220
EmbodiedSWEIsaac Lab / Isaac Simarm, two arms, humanoid3244812
HumanoidBenchMuJoCohumanoid210308
KinDERMuJoComobile manip.15104
LabUtopiaIsaac Simarm19180
LIBERO-MAXrobosuite / MuJoCoarm1641282
ManiGuardOmniGibson (Isaac Sim)arm130607
MetaWorld+MuJoCoarm1501002
MolmoSpacesMuJoCoarm, mobile manip.28162
MuJoCo PlaygroundMuJoCoarm, two arms, quadruped513277
MyoSuiteMuJoComusculoskeletal6224513
RoboCasarobosuite / MuJoComobile manip.129582
RoboCasa-GR1robosuite / MuJoCohumanoid124481
RoboCasa365robosuite / MuJoComobile manip.1501015
RoboDojoIsaac Lab / Isaac Simtwo arms15410212
RoboLabIsaac Lab / Isaac Simarm1531066
RoboTwin 2.0SAPIENtwo arms147943
RoboWitsGenesistwo arms115306
VLABenchMuJoCo (dm_control)arm136722
YAMLabIsaac Lab / Isaac Simtwo arms110202
Total471,1562,288456

Table 2. The suites of the exam. The 9 suites that we built come first, followed by the 26 suites adapted from existing benchmarks, which Section A.2 cites. The Kind column gives the kinds of robot body in the suite, nine over the exam (mobile manip. = mobile manipulator; car = wheeled car). The Robots column counts distinct robot models, the Tests column counts tests over all tasks, and the Core column counts tasks in the core set.

New suites. We built nine new suites because the adapted suites contain few tasks for several embodiments and capabilities. Most of their original benchmarks focus on visuomotor policy learning (§ 4), and 618 of their 745 tasks use single-arm, bimanual or mobile manipulators. The new suites form three groups, which are the rows of Figure 5, and Section A.1 describes each suite. RoboARC, DexHandExam and RoboPaint require contact-rich manipulation of deformable objects, multi-part assemblies and tools. RoboDynamicExam and the two Embodied Olympics suites involve moving objects and opponents, so the agent must act at the right time, and they contain most of the humanoid tasks of the exam. iSpatialGym, Embodied Safety Exam and RoboEngineer test abilities beyond executing a stated goal, since they give the goal only as an image, mix hazardous with benign requests, or require adapting a controller to hidden dynamics.

Figure 5. The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks, in the three groups of § 2.3, and a frame from each of the 26 adapted suites, ordered by their main robot embodiment, from single arms to aerial robots.

Adapted suites. The other 26 suites come from existing benchmarks, and their tasks range from language-conditioned tabletop manipulation and long-horizon household activities to humanoid whole-body control, dexterous tool use, musculoskeletal control, aerial manipulation and laboratory automation (Section A.2). For each suite, we keep the scenes, robots and success rules of the original benchmark, in some suites with a stricter success rule. We save the initial state of each task as a frozen scene, write an instruction that states every convention that the agent cannot infer and connect the suite to the interfaces of the three settings. We also remove the expert policies and demonstration data of the original benchmark, so that the agent cannot copy a known solution from its container.

Core set. We report results on a core set of 456 hard tasks (901 tests), because tasks within a suite often require similar skills and current agents already solve many of the tasks. The core set excludes tasks that we judged easy and tasks that an initial screening agent, GPT-6 Luna, solved, and it keeps every task that the authors of a suite asked to keep. Because GPT-6 Luna belongs to the same model family as GPT-6 Astra, the screening may bias the core set against tasks that this family solves. Table 2 gives the number of core tasks in each suite. Each core task also has a default setting, and the controlled runs of Section 3.2 use the default test of each core task. The default is Standard for 403 tasks and Realtime for 53 tasks in which timing is essential; one more core task also has a Realtime test. The remaining 700 tasks stay in the exam and can be run in the same way. We split 427 of the core tasks into stages, such as opening a fridge (§ 3), and Figure 6 groups the core tasks by the categories of these stages (Table 7). Relation stages, such as placing or assembling objects, are the most common, and most tasks with Dynamic, Body or Competition stages come from the new suites (86 of 99).

Figure 6. The core tasks by the stage categories of their task graphs, with an example task of each category. Each seat is one task in one category, and the angle of a category is proportional to its number of tasks. A task counts once in each category its task graph contains. The centre counts all 456 core tasks, and the seats cover the 427 with a task graph; no task graph has a Safety or Information stage, since the graphs of Embodied Safety Exam give its safety rules as conditions of their stages (Section H.1).

CategoryStage types
Relationplace, put in, take out, gather, stack, insert, assemble, disassemble, screw/unscrew, thread/hang, connect, hand over
Posepick up, regrasp, push, pull, flip over, adjust pose, follow path, stabilize
Dynamicthrow, strike, catch, dribble
Mechanismopen, close, rotate, press, switch on/off
Materialfold, flatten, untangle, pour, scoop, cut, clean, stir, draw/write, change temperature, ignite
Bodynavigate, traverse, jump, perform acrobatics, hang and swing, maintain balance, maintain pose
Safetyrefuse, warn, mitigate
Informationsearch, inspect, communicate
Competitionscore, defend, win

Table 7. Stage types, by category. Competition stages (score, defend, win) are counted by category only.

Analysis

We evaluate eight coding agents on the core set (Figure 8). § 3.2 gives their success rates, and Sections 3.2 to § 3.7 report four findings. An agent is a model in a harness, such as Codex [115] or Claude Code [5], at high reasoning effort, and the harness is part of what we measure (§ 3.5). Each agent makes one attempt per test, and a test is solved when the verifier passes it (Section B). Success rates are the results of the public leaderboard (Section D).

Overall Results

Stage-level failure labels. The verifier decides whether an attempt passed, but not where it went wrong. We therefore split 427 of the 456 core tasks into stages in a task graph (Figure 7a). Its edges give the order in which the stages must be met. Each stage has a type, such as open, in one of nine categories (Table 7), and the contact it needs. Figure 7b shows how many stages hold during three attempts. A failed attempt has at most one failure stage: the stage of an irreversible event, such as a fall, or else of the earliest mistake that the agent did not correct. This mistake can come before the stage’s prerequisites are met, as when an agent drops the tomato before it has opened the fridge. It also has up to two failure phenomena. Claude Opus 5.5, for example, falls while carrying the milk, so it fails at stage 4 with a loss of balance. An attempt reaches a stage if it meets the stage’s condition or fails there. We analyze the failed attempts of the eight agents on these 427 tasks. GPT-6.1 Sol in Codex wrote the labels of most attempts and Claude Sonnet 5.5 in Claude Code those of the remaining attempts; Claude Sonnet 5.5 reviewed every label file (Section H.1).

Figure 7. Task graph and progress maps of carrying in groceries (BEHAVIOR-1K). Panel (a) gives the stages, the order in which they must be met, and the type and contact of each stage. Panel (b) gives the share of the stages whose conditions hold at each moment of the graded episode, for three attempts. Red lines mark failure stages and their failure phenomena (Section H.6).

Overall Results

Three agents each solve more than twice as many Standard tests as any of the other five. GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon solve 24.1% to 33.6% of the 456 Standard tests (Figure 8). We call them the three strongest agents and the other five the weaker agents. The Realtime Setting is harder than the Standard Setting. On the 54 tasks with both tests, GPT-6 Astra solves 23 Standard and 17 Realtime tests, Claude Opus 5.5 solves 10 and 4, and Gemini 4 Argon solves 15 and 5. 63% of these Realtime tests remain unsolved by any agent, compared with 50% of the Standard tests of the same tasks.

Together, the three strongest agents solve 48% of the Standard tests. Figure 9 shows the outcome of every attempt. In the Standard Setting, Claude Opus 5.5 and Gemini 4 Argon solve many tests that GPT-6 Astra fails, whereas in the Privileged Setting GPT-6 Astra alone solves

Figure 9. Every attempt on the core Privileged and Standard tests, with the agents in the order of Figure 8.

nearly every test that any agent solves. The five weaker agents add only a few tests in either setting. § 3.5 examines which tests each agent solves.

In most suites with at least five Standard tests, at least half of these tests are solved by none of the three strongest agents. This holds in 14 of these 22 suites. Of the 20 suites with at least five Privileged tests, it holds only in HumanoidBench (Figure 10). Averaged over the three strongest agents, the success rate is at least 70% in 10 Privileged suites and in no Standard suite.

Figure 10. Success rates by suite, for suites with at least five core tests in the setting. Dots give the success rate of each of the three strongest agents, and bars the share of tests that none of them solves. The IKEA assembly tasks of RoboARC count as their own suite.

What Causes the Drop from the Privileged to the Standard Setting?

Key Takeaways: Privileged versus Standard

  • Resets help GPT-6 Astra more than the exact object state does. Resets raise its success rate from 28% to 53%, the state to 45%.

  • Stages that need precision become much harder in the Standard Setting. Their failure rate rises by 24 to 46 percentage points, that of coarse stages by 1 to 20.

  • At stages that need precision, resets help GPT-6 Astra, but the exact state alone does not. Of its failures there, a run without the success signal solves 19%, a run with the state 21% and a run with resets 36%.

Figure 11 pairs the Privileged and Standard tests of the 391 core tasks that have both. Of the 303 tasks that GPT-6 Astra solves in the Privileged Setting, it fails 191 in the Standard Setting. It solves only 4 tasks in the Standard Setting alone, and 116 (30%) in all. Its success rate drops in each of the 20 suites with at least five such tasks. Two of the features that the Standard Setting lacks are the exact state of every object and unlimited resets.

Adding resets closes more of this gap than adding the exact state. In controlled runs of GPT-6

Astra in Codex, we add one or both of two features to the Standard interface: a query for the exact state of every object, and unlimited resets with saved states (Figure 12a, Section G). On the 391 core tasks that have both settings, the success rate rises from 28% to 45% with the state, to 53% with resets and to 68% with both. On their Privileged tests, it is 77%. With resets, 66% of the successes come in an episode that starts from a saved state, so the agent retries a hard stage without repeating the whole task (Section E.1).

Figure 12. (a) Success rate of GPT-6 Astra in Codex on the 391 core tasks that have both settings, as features of the Privileged Setting are added to the Standard interface. (b) Failure rate of each stage type compared in at least two suites, eight agents. (c) Share of GPT-6 Astra’s failures at precise and coarse stages that each run solves.

Stages that need precision fail much more often in the Standard Setting. From the stage labels of the eight agents, we compute the failure rate of each stage type in each setting: the number of stages of the type at which attempts fail, divided by the number that attempts reach. We compare the settings within each suite (Figure 12b). A stage type is precise when it needs a tight fit, a target pose or path, or contact with a moving object, and coarse when it needs only a region, a container, a grasp or the state of a mechanism (Section H.5). Among the types compared in at least two suites, the failure rate rises by 24 to 46 percentage points at precise types and by 1 to 20 points at coarse types. Assembling, for example, rises from 34% to 80%, and placing from 23% to 35%. If we count only stages that the same agent reaches on the same task in both settings, the difference between the two groups remains significant: precise stages rise by 25 points and coarse stages by 15 (Section E.1). Figure 13 shows a precise stage that fails in the Standard Setting. There, Claude Opus 5.5 holds the front guard of a fan 36 to 39 degrees off the fan’s axis, where seating allows 5 degrees, and pushing it further moves the whole fan. Section H.6 gives the full attempt and two more paired examples (Figure 33).

Figure 13. Claude Opus 5.5 installing the front guard of a fan (DexHandExam) in the Privileged and the Standard Setting. Red marks where the Standard attempt goes wrong. Times are seconds on the replay of the Standard episode.

Figure 33. Privileged and Standard settings. (a) RoboTwin 2.0, hanging a mug: progress maps of GPT-6 Astra. (b) Embodied Olympics - Fun, heading a ball ten times: the headers of each agent in the Privileged (P) and Standard (S) settings.

At precise stages, resets help GPT-6 Astra, but the exact state alone does not. We take the Standard tests that GPT-6 Astra failed in the baseline run and group them by the type of the stage at which they failed (Figure 12c). Some of these failures would succeed in any second attempt. To estimate how many, we use a run that adds neither feature and only hides the success signal. Of the 91 failures at precise stages, this run solves 19%, the run with the state 21% and the run with resets 36%, with 31 of its 33 successes coming after a reset or a restored state. Compared with the run without the success signal, the state thus solves about two more of the 91 failures, and resets about 16 more. At coarse stages, by contrast, the state alone solves 34% of the 97 failures, against 14% for the run without the success signal (Section E.1).

At Which Stage of a Task Do Agents Fail?

Key Takeaways: Where Agents Fail

  • Failures on a task tend to happen at the same stage. For the three strongest agents, 56% of the failure pairs of a test share a stage on average, against 31% by chance.

  • This stage also tends to be the same in the Privileged and Standard settings. Leaving out tasks where it is the first stage in both settings, 57% of tasks match, against 9% by chance.

  • Stages that change a material or move the robot’s body have the highest failure rates. Once reached, they fail at 66% and 45%.

Failures on a task tend to happen at the same stage. The concentration of failure stages measures this (Figure 14a). For each test, it is the share of pairs of failed attempts, by different agents, that failed at the same stage. We use the attempts with a located failure stage and average this share over tests. The baseline draws each failure stage at random, with the probability of each relative position in the task taken from the failures on the other tasks of the setting

Figure 14. (a) Concentration of failure stages, mean over tests, and share of tasks with the same most common failure stage in both settings, without the tasks where it is the first stage in both. Gray bars: random failure stages; black lines: 95% intervals. (b) Failure rate of a stage once reached, by category, both settings pooled (Table 18).

(Section H.5). Over the eight agents, the concentration is 61% in the Standard Setting, against 37% for the baseline. The five weaker agents often fail before they reach any stage of a task (§ 3.5), so much of this agreement could come from failures at the first stage. We therefore check the three strongest agents separately. Over the 217 Standard tests with at least two located failures among them, the concentration is 56%, against 31% for the baseline built from their failures on the other Standard tasks. The most common failure stage of a test is usually the first stage of its task: of the 264 Standard tests whose failures have a single most common stage, it is the first stage in 65% (Section E.2). Figure 15 shows a shared stage that is not the first. In the Standard Setting, all eight agents fail this test, and seven of them catch a thrown ball in a cup and then fail, in different ways, at the recatch. In Figure 32 in Section H.6, the three strongest agents each put the first of three batteries into a bin and then tip the robot over on the way to the second, whereas four of the five weaker agents fail at the first battery.

Figure 15. Catching a ball twice (RoboDynamicExam): the stages that the attempt of each agent reached and the stage at which it failed, in each setting, and Standard replay frames at three failures at stage 2.

Figure 32. BEHAVIOR-1K, disposing of batteries, Standard Setting: the task graph and the stages of each agent, where the wastebasket on the floor holds from the start, and head-camera frames as the robot tips over.

The stage at which agents fail also tends to be the same in the two settings. We take the 188 tasks whose failures have a single most common stage in both the Privileged and the Standard Setting. In 105 of them, this stage is the first stage in both settings, and we leave these tasks out. Of the other 83 tasks, 47 (57%) have the same common stage in both settings (Figure 14a). If the failure stages of each setting were drawn at random as above, this share would be 9%, again leaving out draws in which both common stages are the first stage (Section H.5).

Stages that change a material or move the robot’s body have the highest failure rates (Figure 14b). The failure rate of a stage category is the number of stages at which attempts fail divided by the number that they reach, with the Privileged and Standard settings pooled. Material stages, such as wiping, fail at 66%, body stages, such as walking, at 45%, and mechanism stages, such as opening a door, at 20%. Each category occurs mostly in particular suites and with particular robots, so these rates do not show that the category itself causes the failures (Section E.2).

How Do Agents Differ

Key Takeaways: Differences between Agents

  • Weaker agents are much more likely than stronger agents to fail before making the required contact. The five weaker agents do so at 9.1% to 18.5% of contact stages, the three strongest at 2.7% to 3.7%.

  • In the Standard Setting, the three strongest agents solve partly different sets of tests. Between them, Claude Opus 5.5 and Gemini 4 Argon solve 64 tests that GPT-6 Astra fails.

  • Switching harnesses can change a model’s success rate by up to 12 percentage points. On the 456 core tests of the Standard Setting, GPT-6 Astra solves 34% in its own Codex and 22% in Claude Code.

Which Control Approach Do Agents Use?

The five weaker agents more often fail without making the required contact. In the failure profiles of Figure 16a, this happens in 17% to 24% of their failed Standard and Realtime attempts, against 7% to 10% for the three strongest agents. Of their failed Standard attempts on tasks with at least two stages, they reach no stage at all in 44% to 64%, against 27% to 36% (Table 19). Counted per contact stage that they reach, the weaker agents fail without contact at 9.1% to 18.5% of these stages, against 2.7% to 3.7% (Table 19). Figure 17 shows a BEHAVIOR-1K test, preparing a lunch box, that all eight agents failed in the Standard Setting. The three strongest agents each packed three or four foods, Kimi 3.1 Pro packed two, and the other four weaker agents none. Section H.6 describes the attempts and two more tests (Figure 31). GPT-6.1 Sol, which wrote most of the labels, is from the same family as GPT-6 Astra, which could bias these comparisons (Section H.5).

Figure 16. (a) Failure profile of each agent; zero progress means that the attempt reached no stage. (b) Share of the row agent’s solved Standard tests (first column) that the column agent also solves; the last column uses any agent above it in Figure 8. (c) Success rate by model and harness on the 456 core tests of the Standard Setting; hatched bars mark other harnesses.

Figure 17. Preparing a lunch box (BEHAVIOR-1K, Standard Setting): a head-camera frame of four failed attempts, with the failure circled.

AstraOpusArgonKimiG-3.8 FlashQwenDeepSeekGLM
Failed attempts316372328432449433467472
Zero progress, Standard2736324447645959
Failed to make contact9.87.310.116.718.720.321.224.2
Lost hold17.412.610.112.510.57.29.410.8
Inaccurate pose20.920.423.525.221.211.112.814.2
Obstruction/collision17.115.18.26.510.011.518.812.9
Constraint violation6.64.34.33.01.84.63.63.4
Loss of balance12.717.29.813.012.719.920.118.9
Not attempted0.32.710.70.53.18.84.53.8
Failed while exploring3.27.34.31.63.88.113.77.6
False completion belief0.60.30.90.97.61.61.33.2
Feedback control or trajectory/IK4.714.03.710.24.79.712.08.1
No contact, per contact stage reached2.82.73.79.111.917.216.018.5
Lost hold, per grasp stage reached5.66.54.48.18.18.48.99.8
GPT-6 Astra reached more stages–787092951009998
Tests compared–8761959279120111

Table 19. Failure profile of each agent, given as the share in percent of its failed attempts in the Standard and Realtime settings with each label. Zero progress, an attempt that met no stage of the task, counts only Standard attempts on tasks with at least two stages whose progress shows whether they met a stage. Of each agent’s failed Standard attempts on such tasks, 3 to 33% have no readable progress and are left out, the most for Qwen3.8 Max. The highest value of each row is in bold. The rows from “Failed to make contact” to “Not attempted” are phenomena, that row gives the position of the failure stage, and the two rows after it are root causes. The two rows “per stage reached” divide by stages instead of failed attempts: among the reached stages that need contact (or a grasp), the share at which the attempt failed without contact (or with a lost hold). The last two rows compare each agent with GPT-6 Astra on the tests with at least three stages that both failed, excluding ties: the share of these tests on which GPT-6 Astra reached more stages, and their number.

Figure 31. Standard Setting. (a) RoboARC, threading a rope. (b) RoboCasa365, fetching cereal and a bowl; the column of stage 4 shows the end of the episode.

Panel (a): 1 Grommet G1; 2 Ring A; 3 Grommet G2; 4 Ring B; 5 Grommet G3. Standard Setting. GPT-6 Astra: stage 1 reached, stage 2 reached, stage 3 reached, failure stage at stage 4, stage 5 not reached; rope slipped out at ring B. Claude Opus 5.5: stages 1–5 reached; passed. Gemini 4 Argon: stages 1–3 reached, failure stage at stage 4, stage 5 not reached; rope stopped short of ring B. Kimi 3.1 Pro: stages 1–2 reached, failure stage at stage 3, stages 4–5 not reached; handover missed at grommet G2. Gemini 3.8 Flash: failure stage at stage 1, stages 2–5 not reached; dropped the rope. Qwen3.8 Max: failure stage at stage 1, stages 2–5 not reached; handover missed the rope. DeepSeek V4.1 Flash: failure stage at stage 1, stages 2–5 not reached; pushed it the wrong way. GLM 5.3 Flash: failure stage at stage 1, stages 2–5 not reached; never held the rope. Stage reached; failure stage; not reached. GPT-6 Astra, stage 4: the fingers catch on the rim of ring B. GPT-6 Astra, stage 4: the rope has slipped out of the gripper. GLM 5.3 Flash, stage 1: the gripper closes on the rope. GLM 5.3 Flash, stage 1: it lifts empty; the rope stays.

Panel (b): 1 Open the cabinet; 2 Cereal on the counter; 3 Bowl on the counter; 4 Cabinet closed, met from the start. Standard Setting. GPT-6 Astra: stages 1–2 reached, failure stage at stage 3, stage 4 not reached; bowl slipped from the pinch. Claude Opus 5.5: stages 1–2 reached, failure stage at stage 3, stage 4 not reached; bowl slipped while lowered. Gemini 4 Argon: stages 1–3 reached, failure stage at stage 4; left the right door open. Kimi 3.1 Pro: stage 1 reached, failure stage at stage 2, stage 3 not reached, stage 4 reached; never grasped the cereal. Gemini 3.8 Flash: stage 1 reached, stage 2 not reached, failure stage at stage 3, stage 4 reached; came away empty. Qwen3.8 Max: failure stage at stage 1, stages 2–3 not reached, stage 4 reached; never opened the cabinet. DeepSeek V4.1 Flash: stage 1 reached, failure stage at stage 2, stage 3 not reached, stage 4 reached; never reached the cereal. GLM 5.3 Flash: failure stage at stage 1, stages 2–3 not reached, stage 4 reached; never held a handle. Stage reached; failure stage; not reached. GPT-6 Astra, stage 3: pinches the bowl's rim. GPT-6 Astra, stage 3: the bowl tips out of the pinch. GLM 5.3 Flash, stage 1: at the handles; the doors stay shut.

The three strongest agents succeed on partly different tests. In the Standard Setting, no agent’s solved tests are a subset of another agent’s (Figure 16b). GPT-6 Astra solves 59% of Claude Opus 5.5’s solved Standard tests and 66% of Gemini 4 Argon’s. In the Privileged Setting, these shares are 92% and 95% (Figure 23). Between them, Claude Opus 5.5 and Gemini 4 Argon solve 64 Standard tests that GPT-6 Astra fails. Each agent made one attempt per test, so part of this difference may come from the variation between runs. Of the 69 Standard tests that at least one weaker agent solves, the three strongest agents solve 65. For each weaker agent, 93% to 100% of its solved tests are also solved by at least one agent above it in Figure 8, as the last column of Figure 16b shows (Section E.3).

Figure 23. Share of the Privileged tests solved by the row agent that the column agent also solves, as in Figure 16b, with the agents in the order of Figure 8.

The harness also changes the result (Figure 16c). We ran GPT-6 Astra in Claude Code, Claude Opus 5.5 in Codex, and Gemini 4 Argon in both. On the 456 core tests of the Standard Setting, GPT-6 Astra solves 34% in Codex and 22% in Claude Code. Gemini 4 Argon solves 29% in Codex, 25% in Claude Code and 24% in its own Antigravity CLI, and Claude Opus 5.5 solves 26% in Codex and 25% in Claude Code. Cross-provider runs go through an adapter (Section G). Antigravity CLI ends the session when Gemini’s content filter blocks a reply, which may explain part of Gemini 4 Argon’s gap (Section E.3).

Which Control Approach Do Agents Use?

  • In the Standard and Realtime settings of a soccer task, agents trained a policy when given a GPU and a training library, and only trained policies won.

  • In the Standard Setting, agents mostly control the robot directly. They do so in 61% of the sampled Standard attempts.

  • In the Privileged and Realtime settings, agents mostly write programs to control the robot. They do so in 82% of the sampled Privileged attempts and 97% of the Realtime attempts.

In the Standard and Realtime settings of a soccer task, agents trained a policy when given a GPU and a training library, and only trained policies won. The soccer match of Embodied Olympics - Match comes in two versions with the same frozen scene and opponent (Figure 18b). In both, the agent controls two humanoid robots joint by joint, so it needs a walking gait. The GPU version adds a GPU, a reinforcement-learning library, GPU copies of the match for training and an example of training in its instruction. On this version, 10 of 12 attempts trained a policy with reinforcement learning in the Standard Setting, and 7 of 8 did in the Realtime Setting. Three attempts won in the Standard Setting and one in the Realtime Setting, all with a trained policy. On the CPU version, agents wrote gaits by hand, and none of the 12 Standard and 8 Realtime attempts won. The Standard counts include the runs of the harness comparison, so some models appear more than once (Table 4). In the Privileged Setting, agents also won on the CPU version, but every win there reused the opponent’s walking network. The two versions differ in several ways, and this one task cannot show which difference matters. GPT-6 Astra, for example, trained policies and won 10–8 with the GPU, and tuned a hand-written gait and lost 3–13 without it (Figure 19).

Share of each approach by setting and soccer outcomes

Figure 18. (a) Share of each approach by setting in 1,489 sampled attempts. Mixed attempts switched approach midway or used two approaches equally. (b) Soccer attempts in the Standard and Realtime settings that trained a policy, scored or won.

SettingVersionAttemptsTrained with RLPlayed with itScored a goalWon
StandardGPU1210973
StandardCPUs only120030
RealtimeGPU87521
RealtimeCPUs only80000
PrivilegedGPU850–2
PrivilegedCPUs only800–3

Table 4. Attempts of the agents on the two versions of the soccer task, in their own harness and in the harness comparison. “Played with it” counts the attempts that played the match with the policy they trained. The Privileged Setting does not record goals.

In the Standard Setting, agents mostly control the robot directly, and in the Privileged and Realtime settings they mostly write programs (Figure 18a). A coding agent labeled the control approach of 1,489 sampled attempts (Section E.4). Where the simulation pauses between requests, 61% of the sampled Standard attempts send one action chunk at a time and look at the camera images or sensor readings before choosing the next. With the full state, 82% of the Privileged attempts compute the motion in a program. When the clock keeps running, 97% of the Realtime attempts use a program, and in 82% the program runs the control loop. Six of the 1,489 attempts were labeled as training a policy.

Related Work

This section reviews the closest lines of work. Section K gives an extended review.

Benchmarks for coding agents. Coding agents and other language-model agents are evaluated mostly on digital tasks, such as resolving issues in real repositories [64, 104], completing tasks in a terminal [102], machine learning engineering and research [14, 143], and operating websites and desktop applications [169, 198]. The interface between the model and the computer affects how often an agent succeeds [174], and other work measures how often agents exploit the tests that grade them [196]. The Embodied First Exam extends these evaluations to robotics tasks. In its Standard and Realtime settings, the agent must interpret sensor observations and often cannot undo an action. In every setting, a program that the agent cannot change grades each attempt.

Language models for robots. Language models have been used to sequence skills into plans [1, 52], to write robot programs that call perception and control primitives [84], to specify value maps or keypoint constraints that a planner turns into motions [53, 54], and to write reward functions [95]. Most of these systems pair a model with an interface designed for it. The Embodied First Exam evaluates unchanged general-purpose coding agents on the whole loop of reading an interface, choosing a control approach, carrying it out and revising it. In embodied benchmarks for language models [78, 139, 176], the model chooses from an action space that the benchmark defines. The exam gives the agent the robot’s control interface, so the agent also decides how its actions are produced.

Agentic robots and their harnesses. Recent work runs general-purpose agents on robots [125] with the three control approaches of Figure 2. In direct control, GPT-6 Astra serves as the policy on manipulation benchmarks [61, 191, 199]. In code to control, coding agents write programs for simulated and real robots [50, 56, 62, 150], often with skill libraries or other added tools [48, 93]. In train a policy, agents train policies in the real world or in simulation [83, 166]. Most of these studies fix the control approach in advance. The exam supports all three approaches and leaves the choice to the agent. Harnesses for these agents differ in the action interface they provide [18, 23, 34], in whether the agent can call learned policies as tools [33, 80], and in the support they add for tracking state and checking outcomes [30, 157]. The exam runs each harness unmodified with the models of its own provider, and through an adapter with other providers’ models (Section G), so different harnesses can be compared on the same tasks.

Benchmarks for robots. Most robotics benchmarks focus on visuomotor policy learning and evaluate learned policies, such as vision-language-action models [10, 12, 36, 67], on manipulation, household and humanoid tasks [16, 76, 87, 106, 132]. A learned policy receives only a fixed set of observations. A coding agent can run any program on its computer, so the exam adds rules about what the agent may access (§ 2.1) and adapts 26 existing benchmarks (Section A.2). Benchmarks for agentic robots have also appeared. CaP-X [32] varies the level of abstraction of the primitives that coding agents compose, RobotWorld [178] spans manipulation, locomotion, driving and aerial control, and EmbodiedSWE [134] and BiGym 2.0 [194] evaluate coding agents on dexterous and humanoid tasks. The Embodied First Exam covers 47 robots of nine types. It also separates what an agent can do with full access to the simulator from what it can do through the robot’s sensors alone.

Conclusion

We introduced the Embodied First Exam, a unified evaluation framework that connects any coding agent to simulated robots and lets the agent execute actions in three ways. The exam contains 1,156 tasks on 47 robots. On the 391 hard core tasks that have both settings, the strongest agent reaches a success rate of 77% in the Privileged Setting but only 30% in the Standard Setting. In controlled runs with this agent, adding unlimited resets closes more of this gap than adding the exact object state, and at stages that need precision, resets help but the exact state alone does not. Failures on a task tend to happen at the same stage, weaker agents are much more likely than stronger agents to fail before making the required contact, and in the Standard and Realtime settings of a soccer task, agents trained a policy when given a GPU and a training library, and only trained policies won. The strongest agent already solves most tasks when it has full access to the simulator, but in the Standard and Realtime settings, which are designed to mirror how a real robot is deployed and tested, it still has much room for improvement.

Future work. Running each core test several times will separate failures that an agent repeats from failures of a single run, and it will give confidence intervals for the comparisons between agents and between settings. We will also add more Realtime tasks, since the Realtime Setting has the largest share of tests that no agent solves. Finally, because the Standard Setting gives the agent only the robot’s sensor readings and control interface, the same interface can be used on physical hardware once there is a way to grade the outcome. In a first demonstration, two agents played a buzz wire game with a physical robot arm (§ 8.2), and we plan to evaluate agents on real robots systematically through this interface.

Limitations. Each agent makes a single attempt per test, the stage-level and control-approach labels come from agents, and all results are in simulation. Section J discusses these and further limitations.

Acknowledgements

We acknowledge and disclose the use of AI tools in coding and paper writing. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.

Suites and Tasks

What a task contains. We package every task in the Harbor format with a fixed directory structure. The file instruction.md gives the agent its prompt: the goal, the submission requirements, the limits, and every convention the agent cannot infer, including coordinate frames, units, the action layout, how to use the robot client, and the reset budget. Any goal specified as an image, video, or other file is placed inside the agent’s container and named in instruction.md. We store the task configuration in task.toml, which specifies the time limit, the CPU, memory, and disk caps of the agent’s container, the verifier settings, and task metadata (the simulator, the robot, and whether a reference solution exists). The environment/ directory defines the runtime via a Dockerfile for the agent’s workspace container and a docker-compose.yaml that adds the remaining containers (the simulator service in Standard and Realtime, and a resource monitor) along with their resource caps. Within it, environment/instance.json defines the frozen scene (the objects, the robot, and their initial state), which the simulator checks against a stored hash at every build and reset so every episode starts from the same state. The tests/ directory contains the verifier: its own Dockerfile and docker-compose.yaml, the entry script test.sh, and the grading code with the success rule, all running in an isolated container that the agent never sees. When a task has a reference solution, we include solution/solve.sh, which validates the task and is never shown to the agent, and we provide README.md to describe the task for human readers. Finally, we pin every base image to a fixed tag or digest so every run of a task uses the exact same software.

Table 1 gives the size of the exam, and Table 2 lists its 35 suites, of which 9 were built for the exam and 26 are adapted from existing benchmarks, whose scenes, robots and success rules we keep. The Robots column counts distinct robot models in the suite. The same robot can appear in several suites, so the total counts each of the 47 robots once. A test is one task in one setting, and the Core column counts the tasks in the core set. Time limits range from 30 minutes to four hours by suite. Of the 1,156 tasks, 590 allow one hour and 351 allow two hours or more.

ItemCount
Suites (new)35 (9)
Robots47
Tasks1,156
Tests2,288
Tasks with a full / partial reference solution686 / 78
Core tasks456
Core tests901

Table 1. Size of the exam. Rows list the number of suites (with the number built for the exam in parentheses), robots, tasks, tests, tasks with a full or partial reference solution, core tasks and core tests.

New Suites

RoboARC, DexHandExam and RoboPaint require contact-rich manipulation of deformable objects, multi-part assemblies and tools. In RoboARC, single- and dual-arm robots untie knots, thread ropes through holes, disentangle rings from wire puzzles, guide a loop along a wire without touching it, and assemble IKEA furniture. Earlier work studied such tasks with dedicated systems for untangling cables [152] and with datasets of furniture assembly manuals [158]. In DexHandExam, dexterous hands turn book pages, open jars and insert a USB stick, and objects move only through contact and gravity. In RoboPaint, a robot arm reproduces a painting or a calligraphy character with a brush or a pen, so the task tests fine tool control. Dedicated robot painters such as FRIDA [129] plan their strokes in a differentiable simulation of the painting process.

RoboDynamicExam and the two Embodied Olympics suites involve dynamic objects and opponents, so the agent has to act under tight timing constraints. These three suites contain 53 of the 86 tasks that have a Realtime test. RoboDynamicExam asks single- and dual-arm robots to catch, strike and hand over moving objects such as tossed balls and sliding pucks. In Embodied Olympics - Fun, a humanoid performs athletic skills in 79 task families, such as dribbling a ball through a slalom, and in Embodied Olympics - Match, robots play full matches of table tennis, badminton, soccer, foosball or fencing against scripted or learned opponents. Robots have learned to play table tennis [24], badminton [97] and soccer [43] with reinforcement learning and task-specific training.

The remaining three suites test abilities beyond executing a stated goal. In iSpatialGym, a mobile bimanual robot receives the goal only as an image of what its head camera should show when the task is done, and it must rearrange the scene or reposition itself until its camera view matches this goal image. Multimodal models have been evaluated on spatial reasoning by answering questions about videos of indoor scenes [173], and iSpatialGym requires the agent to act on such reasoning. Embodied Safety Exam pairs hazardous requests, whose literal execution would cause harm, with benign requests of the same kind, so the agent must refuse the hazardous requests and complete the benign ones, even when the hazard is hidden or emerges during the task. Its verifier decides success from what physically happens in the simulation. RoboEngineer asks the agent to tune a controller on a nominal simulator that it can access, deploy the controller to an evaluation simulator with undisclosed dynamics and refine it from the deployment feedback. This loop resembles sim-to-real transfer, which prior work addresses with domain and dynamics randomization [118, 148] and by aligning simulated with real physics [46]. RoboEngineer’s drone tasks build on the learned dynamics models of Neural Lander and Neural-Fly [112, 135], and its racing-car tasks port the MuJoCo vehicle model of AnyCar [165], a wheeled car, without its learned dynamics model.

Adapted Suites

The other 26 suites come from existing benchmarks, and we group them by their main embodiment, as Figure 5 (right) also does. The single-arm suites are LIBERO-MAX [193], RoboLab [172], MetaWorld+ [99], VLABench [190], ManiGuard [119], EmbodiedSWE [134], LabUtopia [79], MolmoSpaces [71] and AutoBio [74], and they range from language-conditioned tabletop manipulation to laboratory automation. The bimanual suites are RoboDojo [17], RoboTwin 2.0 [16], ABC Sim [4], RoboWits [85] and YAMLab [8], and the mobile manipulation suites are BEHAVIOR-1K [76], RoboCasa365 [107], RoboCasa [106] and KinDER [55]. The first three include long-horizon household activities in house and kitchen scenes, and KinDER tests physical reasoning. RoboCasa-GR1 [110], BiGym 2.0 [194] and HumanoidBench [132] cover humanoid manipulation and whole-body control. DexToolBench [65] and DexXoJoco [155] cover dexterous manipulation and tool use. MyoSuite [13] covers musculoskeletal control, MuJoCo Playground [187] covers quadruped locomotion together with a few arm tasks, and AM-Bench [162] covers aerial manipulation.

Protocol Details

Budgets. Each attempt runs in a container with 2 to 16 CPU cores and 4 to 39 GiB of memory, with a GPU where the simulator or the task needs one, with a wall-clock limit of 30 minutes to four hours, depending on the suite, and with a token budget that no attempt reaches. The harness can reach only the API of its model and the package registries, except in Embodied Olympics - Fun, which runs with open network access. The harness’s web search is off in every task. A proxy between the harness and the model provider forwards only the requests to the model’s API.

Reference solutions and integrity checks. Every task must fail when the robot does nothing and pass with its reference solution wherever one exists (686 tasks have a full reference solution, and 78 have a partial one). Tasks without a reference solution rely on the original success rule and a human review. Reference solutions never enter the agent’s container, and the solutions of the original benchmarks are removed from the container images. Before a suite is added, an agent is instructed to pass its tasks without solving them. If the agent succeeds on any task, the suite is not added until its verifier is fixed.

Leaderboard rules. Each agent makes one attempt per test. An attempt that ends with an infrastructure or API error is rerun, and the latest finished attempt counts. A test counts as solved when the verifier reports success for its attempt. We report each setting over all of its core tests, including those of Embodied Olympics - Fun, whose tasks run with open network access, and of the suites with their own runtime, Embodied Olympics - Match and RoboEngineer, and Figure 10 gives the results by suite. The harness comparison uses the 456 core tests of the Standard Setting. When we pool the three strongest agents, as in § 3.2 and D, each test counts once per agent, and rates are reported per attempt.

Approach labels. For § 3.7, an agent read the logs of a random sample of 1,489 attempts and assigned each attempt the approach that produced its graded answer. Code to control covers three kinds of program. An open-loop program computes the whole motion in advance, as waypoints, key frames, inverse-kinematics solutions or a recorded sequence, and runs it without feedback. A closed-loop program reads the state or the cameras at every control step or action chunk and adjusts its commands. A parameter search finds a passing motion, or its parameters, by trying many episodes and resets or by running an optimizer such as the cross-entropy method (CEM), CMA-ES, or a grid or random search. In Direct control, the agent sends one action chunk at a time and decides on the next one after looking at images or readings. In Train a policy, the agent first trains a control policy, such as a neural network or a reinforcement-learning policy, which then controls the robot. Attempts that switched approach midway or used two approaches equally are labeled mixed, and attempts that only explored or never ran anything are labeled as having no control plan. Figure 2 shows an example of each approach.

Interface, Grading and Physics

This section expands § 2.1 with the protocol rules and each suite’s values and examples.

Interface

Privileged Setting: a simulator library in the agent’s container. The agent’s container holds the simulator of the suite and a small Python library that the exam provides, which we call the tooling. The tooling rebuilds the frozen scene, resets it, advances it by applying actions, reads the state of every object and records each applied action as one row of an array. The agent may also call the planning primitives of the original benchmark, such as the grasp and place primitives of RoboTwin 2.0 with its CuRobo planner, or write its own planner, optimizer or learned controller. Whatever the agent calls, the tooling records the low-level action row applied at every control step, and the answer is this array of actions, saved as a trajectory file. For RoboTwin 2.0, for example, a row has 34 columns (16 joint position targets, 16 joint velocity targets and two gripper commands) and one row is applied per 4 ms physics step. In the Privileged tests of Embodied Olympics - Match and EmbodiedSWE the answer is instead a program (one that plays the match, or one that controls the robot through the task), and the verifier runs it in a fresh simulator.

Standard Setting: the simulator as a service. In the Standard and Realtime settings, the simulator runs as the simulator service in a separate container that the agent cannot access. The agent reaches it over HTTP through a Python client library that the exam provides. One call returns the specification of the robot: the action types with their layouts, units and limits; every joint with its position, velocity and effort limits and drive gains; the grippers and tool frames; the cameras with their intrinsics and mounting; the layout of the joint readings; the conventions of the world frame; and the budgets. A second call returns observations: either the current camera images, without advancing the simulation, or the images after the simulation has run for a requested number of seconds with the last command held. A third call executes an action chunk of TT rows, each of dimension dd, and returns the video of the execution. Where the service offers it, a fourth computes forward and inverse kinematics on the model of the robot without advancing physics. A reset call exists only in tasks that grant resets. Every response contains the camera images (RGB, and depth on request), one row of joint readings per image (joint positions and velocities, gripper opening, tool pose and an estimate of the drive effort), the camera calibration, the simulated time and the time the agent has left. Object poses, segmentation, scene files, the seed and the success rule are never sent. Contact forces on objects are not sent either, except as safety telemetry that a suite documents field by field, as in RoboARC, and that never gives an object’s pose. Any other request, such as a query for object states or for success, returns an error. The images come back as binary data that the agent’s code can store and process, so they do not have to pass through the model’s context.

From a chunk to motion. Joint position rows are dtdt seconds apart, where dtdt is a multiple of the control period and at most 0.5 s. The service rejects a value of dtdt that breaks this rule and does not round it. The service streams the rows to the drives of the joints in the way a position controller on a real robot follows a trajectory, either interpolating between rows or holding each row until the next. It clips any value outside the joint limits and reports each clipped entry in the response. End-effector rows go through the inverse kinematics or the motion planner of the suite. Where the original benchmark controls the robot through a higher-level controller, the agent sends the commands of that controller, as in RoboCasa and in the whole-body controller of BiGym 2.0. Each suite keeps its own control rate and physics step. Control rates range from 5 Hz (the mobile robot in MolmoSpaces) to 500 Hz (AutoBio, where one row is one 2 ms physics step), and one control step runs up to 100 physics steps (VLABench). The success rule is checked after every action chunk and every observation request that advances physics, and in most suites the first success ends the episode. In the Standard tests of RoboDojo, a success does not end the episode, so that the end of the episode does not tell the agent that it has succeeded, which would give it a success check to try against until it passes. The episode ends at the task’s step limit, when the agent ends it, at its time limit, or when a check of the task fails, and it passes if the success rule held at any point, which matches RoboDojo’s own success within its step limit.

Realtime Setting: a simulation that keeps running. The Realtime Setting uses the same calls. From the first request that advances physics, the simulation advances one control step every SS control periods of wall-clock time, whatever the agent does, where S≥1S \ge1 is a slowdown factor that each task states. A submitted chunk is answered at once and replaces the queued actions starting at a control step that the agent specifies. When the queue is empty, the robot holds its last target. The agent can therefore run a program in its own container that observes the robot and sends new chunks in a loop. By default, SS is 1 when the simulator runs at least three times faster than real time, and larger otherwise, so that the service can run each step on time. The service records the steps it could not run on time, and the time the robot stood still because of them.

Suites with their own runtime. RoboEngineer and Embodied Olympics - Match use their own runtime instead of the simulator service, in their Standard tests and in the Realtime tests of Embodied Olympics - Match. In RoboEngineer, the agent tunes a controller in a simulator that the agent can access and deploys the controller to a second simulator, which the agent cannot access and whose dynamics differ from those of the first in ways not disclosed to the agent. The verifier then scores the controller that the agent selects on episodes that the agent has not seen.

Step limits and time budgets. The protocol adds no step limit to the Standard and Realtime settings. RoboDojo keeps its own step limit (Figure 4), and many other suites cap the episode length, which the instruction states. A Privileged trajectory may have at most a stated number of rows. The main limit is the agent’s wall-clock time, from 30 minutes to four hours depending on the suite, and it is the same in every setting of a task. Each response reports the time left, counted from the first request of the agent. One call may return only a stated number of images and may run for at most 15 minutes by default, so that one call cannot exhaust the memory of the service or occupy it indefinitely.

Limits. The limits of a test are the agent’s wall-clock time and token budget, the resets that the task grants, the slowdown SS in the Realtime Setting, the per-call limits above, and the CPU, memory, process, disk and GPU memory caps of each container, which a resource monitor records and enforces where the container runtime does not. Privileged tests allow unlimited resets, and Standard and Realtime tests allow no resets unless the task grants a budget, which its instruction states (§ 2.2). These values are part of the configuration of a result, and results are compared only between runs with the same configuration.

Grading

Success rules. Each task is graded by one success rule, a condition on the simulator state. Adapted suites keep the success rule of the original benchmark and tighten it in some tasks. Examples are the success function of a RoboTwin 2.0 task, the BDDL goal of a BEHAVIOR-1K activity evaluated on OmniGibson object states, and the solved flag of MyoSuite. For the suites that we built, we wrote our own rules on the simulator state. The success rule of Embodied Olympics - Fun checks the suite’s scoring checkpoints, legal contacts and fouls after every 1 ms physics step, RoboDynamicExam checks ordered phases (a catch, a hold, a crossing, a release) after every 1 ms physics step, and DexHandExam computes its checkpoints from contact pairs and contact forces. The instruction states each condition of the success rule with the same thresholds that the rule uses.

Settle and hold windows. Some rules are checked over a time window. A hold window requires the goal to hold for a stated time, so that an object in flight or a grasp that is about to slip does not count. In BEHAVIOR-1K the goal must hold for 1 s with the object released and at rest. MyoSuite requires its solved flag for 1 s (5 s for the Baoding balls), BiGym 2.0 requires its goal for 1 s and no fall during the episode, and ManiGuard requires its goal for 10 consecutive control steps and no safety violation during the episode. A settle window keeps the simulation running with the last command held while the rule is checked, as in RoboLab, which holds the last command for 2 s. Rules without a window are checked on a single state. A window is part of the rule of the task and applies in every setting. It is checked on the live simulator in the Standard and Realtime settings and after the last row of the replay in the Privileged Setting.

Privileged Setting: a fresh replay. The verifier rebuilds the frozen scene in a fresh process, checks that its initial state matches the stored hash, replays the submitted actions open loop and applies the success rule to the replay. By default it replays once. In BEHAVIOR-1K, AM-Bench, RoboDojo and RoboLab, whose simulators do not replay bit for bit, it replays up to three times, each in a fresh process, and the attempt passes if any replay succeeds. Where the answer is a program, as in Embodied Olympics - Match and EmbodiedSWE, the verifier runs the program in a fresh simulator instead. Because only the actions are replayed, a change that the agent made directly to the state of its own simulator, such as moving an object directly, has no effect on the result. A file that is not a valid trajectory fails.

Standard and Realtime settings: the live episode. The simulator service checks the success rule on the episode that it ran, and the verifier reports the result from the service’s record of the episode. The service checks the hash of the start state when the episode begins and after every reset. When the agent’s time is up, the program that runs the evaluation stops every process in the agent’s container before it signals the service. From then on the service refuses every request that would advance the simulator, runs any settle window, checks the success rule on the live simulator and writes the result with the recorded trajectory. The verifier reports that result, and nothing is replayed to decide it. The result depends only on the state of the simulator, never on what the agent reports. An episode in which the agent’s actions cause a simulator error fails. A failure of the infrastructure, such as a scene that does not rebuild to its stored start state or a service that crashed, gives no result, and the whole attempt can be rerun. With resets, only the last episode is graded. We also record whether the first episode, before any reset, succeeded, and report this as first-episode success.

Isolation. The verifier runs in its own container, built from the task’s grading files, without network, and receives only the files that the task declares for grading: the trajectory file (or the submitted program) in the Privileged Setting and the record of the simulator service, which checks the success rule, in the Standard and Realtime settings. It first removes from the agent’s output directory every entry that is not a regular file, such as a link. No writable volume is shared between the agent’s container and the containers of the simulator service and the verifier, and the simulator service opens no port on the host. The agent can never read the reference solution or the files of the verifier, and in the Standard and Realtime settings it cannot read the code of the success rule either. A program submitted as the answer runs under a user account without administrator rights and without network, cannot read the files of the simulator or the verifier, and every process it starts ends before the result is written. The result is written before the video of the attempt is rendered, and the video never changes it.

Physics

Simulators. Each suite keeps the simulator of its original benchmark, and new suites use the simulator that suits their physics. MuJoCo [149] runs most suites, directly or through robosuite, dm_control and MyoSuite, in versions 3.2 to 3.14 and mostly on the CPU. NVIDIA Isaac Sim and Isaac Lab with PhysX run RoboLab, RoboDojo, YAMLab, AM-Bench, LabUtopia and EmbodiedSWE, and OmniGibson on Isaac Sim runs BEHAVIOR-1K, ManiGuard and iSpatialGym. SAPIEN 3 with PhysX runs RoboTwin 2.0, and through ManiSkill 3 it also runs RoboPaint and RoboDynamicExam. Genesis runs RoboWits. RoboEngineer runs MuJoCo, and its drone tasks add learned dynamics models, a learned ground-effect model of a quadrotor and a wind-force model fitted to published flight logs.

What is simulated. All suites simulate rigid bodies with frictional contact, joint limits and actuator limits. Several also simulate deformable and particle matter: cloth in the blanket tasks of RoboARC (a deterministic GPU solver) and the folding tasks of RoboDojo (GPU PhysX), ropes and cords as soft bodies in RoboARC, liquids as GPU PhysX particles in RoboDojo, and the object states of OmniGibson in BEHAVIOR-1K, where objects can be sliced, cooked or filled with particles. MyoSuite moves its limbs with simulated muscles instead of motors, and RoboPaint models brushes and pens on paper. Physics steps range from 1 ms (Embodied Olympics - Fun and RoboDynamicExam) to 1/60 s (LabUtopia), with 2 ms in DexHandExam, DexJoCo and AutoBio and 1/120 s in iSpatialGym, ManiGuard, AM-Bench and YAMLab.

Determinism. Privileged grading replays the submitted actions, so when we build a task we check whether fresh replays of a trajectory end in the same final state on the hardware used for evaluation. Where they do not, the verifier replays up to three times (Section C.2). The frozen scene is also rebuilt and checked against its stored start-state hash before every episode. Where a task does not replay reliably at all, as with the GPU cloth and liquids of RoboDojo, the task is offered only in the Standard Setting, where the live episode is graded, and its video is a recording of the live episode rather than a replay.

Realism rules. The robot moves only through its own actuators and the controllers stated in the task. In the Standard and Realtime settings the agent has no way to set the state of the simulator, and in the Privileged Setting only actions are replayed, so moving an object directly has no effect on the grade in any setting. The suites we built for the exam add stricter rules. In Embodied Olympics - Fun, the simulator never attaches a held object rigidly to the hand, never pins a body or moves it with a motion-capture constraint, and never sets a velocity directly after the start. Balls come from ball machines and goalkeepers are force-limited mechanisms. Each robot uses its own hands and actuators, modeled on measured actuator data from the manufacturer (torque-speed curves, rotor inertia and friction). Where we increased a parameter above the manufacturer’s value, the increase is at most a factor of 1.5 to 2 and is stated in the task; the one stated exception is the grip on the high bar. DexHandExam has no rigid attachments or scripted object motion, so objects move only through contact and gravity, and in the furniture assembly tasks of RoboARC, the robot holds parts only by finger friction. Where a suite keeps a grasp aid, its documentation states it. In the knot, ring and rope tasks of RoboARC, a rigid attachment holds a grasped object and slips when the pull exceeds 60 to 65 N for 20 ms. Adapted suites keep the grasp model of the original benchmark, such as the assisted grasping of OmniGibson and the hand of MetaWorld+, which a motion-capture constraint moves.

Additional Results

This section describes the agents, the counting rules and the data behind § 3. It then shows the success rates of Figure 8 as bars and gives further results by suite, by cost and by task type (§ 3.2).

Agents. We evaluate eight agents, each a model running in a harness at high reasoning effort. The three strongest are GPT-6 Astra [116] in Codex [115], Claude Opus 5.5 [7] in Claude Code [5] and Gemini 4 Argon in Antigravity CLI. The other five are Kimi 3.1 Pro and Qwen3.8 Max in Claude Code and three Flash-class models: Gemini 3.8 Flash in Antigravity CLI, GLM 5.3 Flash in Claude Code and DeepSeek V4.1 Flash in DeepSeek Harness (dsh), DeepSeek’s own harness. We report each result for a model in a particular harness, because the way the harness exposes tools, manages the context and ends a session is part of what we measure. Switching harnesses can change a model’s success rate by up to 12 percentage points (§ 3.5).

Counting rules. Section B gives the leaderboard rules. Each column of Figure 8 covers all core tests of one setting (456 Standard, 391 Privileged and 54 Realtime tests), and every core task has a Standard test. When we pool the three strongest agents, each test counts once per agent. For results by suite, we treat the IKEA assembly tasks of RoboARC as a separate suite.

Data. Success rates are the results of the public leaderboard. The controlled runs add features of the Privileged Setting to the Standard Setting or run a model in a different harness.

Success rate of each agent. Figure 20 shows the success rates of Figure 8 as bars, for each setting. GPT-6 Astra leads in every column and solves a third of the Standard tests. The three strongest agents keep the same order in every column except the Realtime column, where Gemini 4 Argon is ahead of Claude Opus 5.5. Outside the core set, success rates are higher.

Figure 20. Success rate of each agent on the core set. Each bar gives the share of the core tests of one setting that one agent solves.

Paired outcomes. Figure 11 in Section 3.2 splits the 391 core tasks that have both a Privileged and a Standard test by the settings in which each agent solves them. GPT-6 Astra solves 4 tasks in the Standard Setting but not in the Privileged Setting, and Gemini 4 Argon solves 18 such tasks. The Privileged result is therefore an approximate upper bound on the Standard result over a set of tasks, not on each single task (§ 2.1).

Results by suite. GPT-6 Astra leads or ties in most suites, while Claude Opus 5.5 and Gemini 4 Argon each lead in some (Figure 10). We count only suites with at least five tests in the setting. In the Standard Setting, at least half of the tests are solved by none of the three strongest agents in 14 of 22 suites, among them DexToolBench, the IKEA assembly tasks, HumanoidBench and BEHAVIOR-1K. The three strongest agents solve at least 60% of their attempts only in ManiGuard, whose safety rules are stated in the instruction. In the Privileged Setting, at least half of the tests are unsolved only in HumanoidBench, and the three strongest agents solve at least 70% of their attempts in 10 of 20 suites: Embodied Olympics - Match, EmbodiedSWE, RoboDynamicExam, RoboCasa365, RoboWits, MuJoCo Playground, Embodied Olympics - Fun, BiGym 2.0, RoboPaint and ManiGuard.

Cost of an attempt. Figure 21 prices each attempt from the tokens that its harness logged, at the providers’ list prices of November 2026. These are list-price costs, not billed costs. Each graded attempt is priced from the run that the leaderboard counts for it. In the Standard Setting, an attempt of GPT-6 Astra costs $27.1 on average, about four times as much as one of Claude Opus 5.5 or Gemini 4 Argon. Kimi 3.1 Pro and Gemini 3.8 Flash spend more per attempt than these two agents and solve less than half as many Standard tests. For every agent, in the Privileged and in the Standard Setting, the median failed attempt costs more than the median solved attempt.

Figure 21. Success rate of each agent against the mean list-price cost of its attempts.

Results by task type. A task type describes a core task as a whole, whereas the stage-level failure labels of § 3 assign a type to each stage of a task. By task type (Figure 22), GPT-6 Astra’s lead on the Standard tests is widest in the Tool Use and Precision & Contact types, and Gemini 4 Argon is ahead of it in Object Transport, at 39%. For the three strongest agents together, Object Transport has the highest success rate of the nine task types, at 38% of their attempts, and Tool Use has the lowest, at 11%.

Figure 22. Share of the core tasks of each type whose Standard test each agent solves, with a gray line joining the three strongest agents. The number of core tasks of the type is in parentheses, and a task can have several types.

Further Analysis

This section gives the full analyses behind the key takeaways of Sections 3.2 to § 3.7, in the same order. The definitions and statistics that these analyses use are in Sections G and H.

Causes of the Drop from the Privileged to the Standard Setting

To measure how much two features that the Standard Setting lacks contribute to the drop, we ran GPT-6 Astra in Codex on the default tests of the core tasks with one or both of them added to the Standard interface (Figure 12a, Section G). The first feature is a query that returns the state of every object, including its pose, size and velocity. The second is unlimited resets together with calls that save the simulator state and restore it later. On the 391 core tasks that have both settings, GPT-6 Astra solves 28% with the unmodified interface. The state query raises this rate to 45%, and unlimited resets raise it to 53%. With both features, the agent solves 68%, against 77% on their Privileged tests. With unlimited resets, GPT-6 Astra uses saved states to retry a hard stage without repeating the whole task. Of the 206 successes of the reset run on these tasks, 14% came in the first episode, and 66% came in an episode that started from a saved state.

The stage labels show at which kinds of stages the Standard Setting raises the failure rate (Figure 12b). The failure rate of a stage type is the number of its stages at which attempts fail, divided by the number of its stages that attempts reach. We compare it between the two settings within each suite and average over the suites (Table 12). Section H.5 defines the precise and coarse groups of stage types and gives the statistical tests. Among the stage types compared in at least two suites, the failure rate rises by 24 to 46 percentage points for precise types and by 1 to 20 points for coarse types. The failure rate of cleaning, which is in neither group, also rises by 51 points. Pooled within suites, the failure rate rises from 22% to 56% at precise stages and from 22% to 38% at coarse stages. Figure 13 shows an example, and Section H.6 gives it in full. The difference between the two groups is smaller, but still significant, when the same agent reaches the same stage of a task in both settings. There, the rise is 25 points at precise stages and 15 points at coarse stages. At follow-path stages, most of them waypoints of DexToolBench, the rate rises by 11 points in this comparison. Without these stages, the rise at precise stages is 33 points.

Stage typeSuitesWeightPSS − P
Clean2683384+51
Fold1353889+51
Assemble2563480+46
Strike22211155+44
Perform acrobatics1292871+43
Flatten1364284+42
Adjust pose5712364+41
Dribble1421455+41
Catch21001857+39
Throw15661100+39
Insert2462562+36
Draw/write11655994+35
Jump21343666+30
Traverse2952754+27
Stack2232854+26
Disassemble2474470+26
Thread/hang32082449+25
Follow path44972347+24
Put in127972848+20
Open42652947+18
Pick up5842947+18
Switch on/off2241025+15
Place104842335+12
Close726459+4
Press29612+1

Table 12. Failure rate of each stage type in percent, compared within suites: the types compared in at least two suites, as in Figure 12b, and the types of one suite with a weight of at least 25. P is the Privileged and S the Standard Setting. The Suites column gives the number of suites compared, and the Weight column the sum of the smaller counts. Types compared in only one suite are less reliable. Close includes stages that already hold at the start, such as a closed fridge, which count as reached even when the attempt never acts on them.

Combining the controlled runs with the stage labels shows which feature helps at which kind of stage (Figure 12c, Table 14). We take the Standard tests that GPT-6 Astra failed in the baseline run of the controlled runs and group them by the type of the failure stage. Of these failures, 91 are at precise stages and 97 are at coarse stages. Some of these failures would turn into successes if the test were simply run again. We therefore compare each run with the run without the success signal, a further controlled run in which the interface no longer reports when an episode succeeds. This run repeats the same tests and changes only the success signal, so we use it as a rough approximation of a second attempt. Its success rate is within 4 points of the baseline’s (Table 5), but this run does not separate the variation between attempts from the effect of hiding the success signal. At precise stages, the state query solves 21% of the failures, against 19% for the run without the success signal. Unlimited resets solve 36%, and 31 of these 33 successes came after a reset or a restored state. The agent did call the state query in the runs that offer it. At precise stages, adding the state query to the resets raises the share solved from 36% to 47%. At coarse stages, the state query alone solves 34% of the failures, against 14% for the run without the success signal, and the two features together solve 67%, against 60% in the Privileged Setting. With both features, the agent solves 47% of the precise failures, compared with 73% in the Privileged Setting, and most of this remaining gap lies at follow-path, thread and assemble stages. Neither added feature lets the agent advance the physics one step at a time or call planning primitives, as the Privileged Setting does, and these abilities may account for part of the remaining gap.

RunStandardRealtime
Standard (baseline)2830
+ object-state query4538
+ unlimited resets and save/load states5364
+ both6880
Privileged7784
No success signal3126

Table 5. Success rate (%) of GPT-6 Astra in Codex in the controlled runs, on the 391 core tasks that have both a Privileged and a Standard test, and in the Realtime column on the 50 Realtime tests of these tasks. The Standard column includes the Realtime tests. The Privileged row uses the Privileged tests of the same tasks.

RunPrecise (91)Coarse (97)Material (36)Other (32)
No success signal19 (11–26)14 (8–22)28 (14–42)16 (3–28)
+ object-state query21 (13–30)34 (25–43)42 (25–58)28 (13–44)
+ unlimited resets and save/load states36 (26–46)44 (35–54)47 (31–64)31 (16–47)
+ both47 (37–58)67 (58–76)69 (56–83)41 (22–56)
Privileged73 (64–81)60 (51–69)83 (69–94)72 (56–88)

Table 14. Share (%) of GPT-6 Astra’s labeled failures in the Standard Setting that each controlled run solves, by type of failure stage, with 95% intervals over failures. Material stages include drawing, pouring and cleaning. The Privileged row uses the Privileged test of the same task.

The comparisons in this subsection have limits. The controlled runs use one agent. The labels cannot separate errors of perception from errors of control (Section H.5).

Where Agents Fail

Failures on a task tend to happen at the same stage (Figure 14a). We measure this by the concentration of failure stages, which is the probability that two of the failed attempts on a test, chosen at random, failed at the same stage. Each agent makes one attempt per test, so the two attempts always come from different agents, and the measure pools the agents that failed the test. On the Standard tests that at least two agents failed at located stages, on tasks with at least two stages, the concentration averages 61%, and we compare it with two baselines. In the first baseline, each agent fails at a stage chosen at random, with the probability of each stage position set by how often failures occur at that position on the other tasks of the same setting, so early stages are more likely. This baseline gives 37%. In the second baseline, an agent fails at each stage with the failure rate of the stage’s type on the other tasks, and this baseline gives 48% (Table 16). For the three strongest agents alone, the concentration is 56%, against 31% for the first baseline built from their failures on the other tasks (Section H.5). Failures therefore concentrate more than the stage types predict. The most common failure stage is usually the first stage of the task. Of the 264 Standard tests whose failures have a single most common stage, this stage is the first stage in 65%, and it is the first stage in 56% of those whose tasks have at least two types of stages. In a RoboDynamicExam task, for example, seven of the eight Standard attempts caught a thrown ball in a cup and then failed at the recatch, in three different ways (Figure 15, with all eight attempts in Section H.6).

Naming ruleAllTwo or more stage typesConcentrationPrivilegedStandard
Stage-type rates, other tasks7367Observed5061
Stage-type rates, same suite7571Stage-type rates, other tasks3648
Always the first stage6556Stage-type rates, same suite4051
Random failure stages3237

Table 16. Stage types and the common failure stage, in percent: how often a rule names the most common failure stage of a Standard test, over the 264 Standard tests that have one and over those among them whose tasks have at least two types of stages, and the concentration of failure stages, observed and expected if each agent failed independently at the given rates.

Failure rates differ between stage categories (Figure 14b). Pooled over the Privileged and Standard settings, attempts fail at 66% of the material stages (such as wiping or pouring) that they reach, at 45% of the body stages (such as walking or keeping balance) and at 20% of the mechanism stages. Within a category, the rate also depends on how the robot must act. Opening a door or a lid fails in 38% of cases, whereas closing one fails in 19% when it does not already hold at the start (Section H.5), and pressing fails in 3% (Table 18). Relation stages, which place or fit one object relative to another (Table 7), fail more often when two arms act together or the legs carry the robot’s weight than when one hand acts (Figure 28 in Section H.5). However, which stage types fail most differs from suite to suite. When we rank each pair of stage types by their failure rates on all suites but one, the ranking holds in the held-out suite for 55% of the 313 pairs, close to what a random ranking would give. Whether an attempt fails at a stage that it reaches is also predicted better by the identity of the agent than by the stage type (Section H.5). Each category occurs mostly in particular suites and with particular robots, so these rates do not show that the category itself causes the failures.

Figure 28. Failure rate of a stage once reached, by stage category and by the robot part that makes contact, with the Privileged and Standard settings pooled (Table 18). Each cell gives the rate, the stages failed over the stages reached, and a bar on a scale from 0 to 100%. The rows are those of Figure 14b, at the same heights.

CategoryStage typeSuitesPrivilegedStandardBoth
Mechanismall1814 (124/856)27 (220/825)20 (344/1,681)
Press62 (3/179)6 (7/114)3 (10/293)
Close95 (15/287)8 (25/305)7 (40/592)
Switch on/off411 (5/44)24 (8/33)17 (13/77)
Open1229 (92/321)47 (164/350)38 (256/671)
Poseall1616 (253/1,592)51 (389/763)27 (642/2,355)
Follow path614 (191/1,334)50 (267/534)25 (458/1,868)
Pick up1024 (28/116)42 (51/121)33 (79/237)
Adjust pose924 (29/123)67 (65/97)43 (94/220)
Dynamicall317 (121/699)62 (266/430)34 (387/1,129)
Strike312 (50/418)56 (128/229)28 (178/647)
Relationall2626 (615/2,356)45 (1,051/2,350)35 (1,666/4,706)
Place1923 (150/651)32 (289/895)28 (439/1,546)
Thread/hang622 (69/311)49 (113/229)34 (182/540)
Put in2028 (274/994)47 (426/905)37 (700/1,899)
Screw/unscrew322 (14/64)81 (26/32)42 (40/96)
Insert523 (16/69)64 (47/74)44 (63/143)
Stack425 (10/40)66 (25/38)45 (35/78)
Competitionall218 (26/141)50 (67/133)34 (93/274)
Bodyall728 (143/504)67 (267/401)45 (410/905)
Traverse422 (43/194)58 (62/106)35 (105/300)
Navigate533 (8/24)68 (52/77)59 (60/101)
Materialall744 (189/433)84 (447/535)66 (636/968)
Clean333 (28/84)82 (62/76)56 (90/160)
Pour325 (3/12)72 (113/156)69 (116/168)

Table 18. Failure rate (%) of each stage category and of the stage types found in at least three suites, in the Privileged Setting, in the Standard Setting and in both together, with the stages at which attempts failed over the stages that attempts reached. The Suites column gives the number of suites with the stage type.

The most common failure stage of a task is also usually the same in both settings. Of the tasks whose failures have a single most common stage in both settings, 81% have the same such stage in the two settings, compared with 49% if each setting’s failure stages were sampled independently. Excluding the tasks whose most common failure stage is the first stage in both settings, the share is still 57%, compared with 9% for random failure stages (Section H.5). Of the attempts that get past the stage at which the other agents on the same test most often failed, 64% pass the test in the Privileged Setting and 33% in the Standard Setting. Part of this link between getting past the stage and passing follows from the order of the stages, because an attempt that gets past the common stage is already closer to the end of the task.

Differences between Agents

Tests solved by one agent and by another. For two agents A and B, the cell in row A and column B of Figure 16b gives the share of the tests that A solves that B also solves. The denominator of row A is the number of tests that A solves. It is given in parentheses after the name of A, and it equals A’s count of solved tests in the Standard column of Figure 8. The diagonal is left empty, because it would always be 100%. The last column gives the share of A’s solved tests that at least one agent above A in Figure 8 also solves. A test breaks the order when some agent solves it and GPT-6 Astra fails it. We count such a test under the highest agent in Figure 8 that solves it, so every agent above this agent fails the test. We use the Standard Setting in the main text because every core task has a Standard test and all agents reach the robot through the same interface there. Each of the five weaker agents solves 11 to 46 Standard tests. A single test therefore changes a share in its row by 2 to 9 percentage points. Among these five agents, the order of Figure 8 does not hold. Gemini 3.8 Flash, for example, solves 2 of the 11 Standard tests that GLM 5.3 Flash solves.

In the Privileged Setting, the order of Figure 8 holds more closely. GPT-6 Astra solves more than 90% of the Privileged tests that Claude Opus 5.5 or Gemini 4 Argon solves (Figure 23), and only 24 of the 327 Privileged tests that some agent solves break the order (Table 3), against 68 of 221 in the Standard Setting. Every agent except GPT-6 Astra solves at most 5 Realtime tests, so we do not draw a Realtime matrix. Of the 20 Realtime tests that some agent solves, 3 break the order: Claude Opus 5.5 solves 2 of them and Gemini 4 Argon 1.

SuiteStandard TestsStandard Claude Opus 5.5Standard Gemini 4 ArgonStandard WeakerPrivileged TestsPrivileged Claude Opus 5.5Privileged Gemini 4 ArgonPrivileged Weaker
Embodied Safety Exam50850––––
RoboARC5082150510
Embodied Olympics - Fun5651056100
ManiGuard74007000
RoboDojo1202112000
Embodied Olympics - Match1311113200
BEHAVIOR-1K7111071611
Other suites1972051182412
All456471743911833

Table 3. Tests that break the order of Figure 8, by suite and setting. “Tests” gives the number of tests of the suite in the setting. Each test that breaks the order is counted in the column of the highest agent in Figure 8 that solves it, and “Weaker” sums the five weaker agents. The six suites with the most such Standard tests and BEHAVIOR-1K, which has the most such Privileged tests, are listed, and “Other suites” sums the rest. RoboARC excludes its IKEA assembly tasks, which are counted under Other suites.

Where the order breaks. Table 3 counts the tests that break the order by suite. In the Standard Setting, Embodied Safety Exam has the most, 13 of its 50 tests, followed by RoboARC with 11 of 50. In the Privileged Setting, BEHAVIOR-1K has the most, 8 of 71. We checked three of these tests, two of which Figure 24 shows. In the RoboARC blanket test, which all seven other agents failed, Gemini 3.8 Flash computed the corner positions from camera images and depth. It grasped the two far corners with both grippers and folded the blanket in half. It then ran four adjustment programs before the episode ended with success. In an iSpatialGym test, whose goal is given as a picture of the finished scene, DeepSeek V4.1 Flash placed a wrapped food package on a plate. The simulator service ended the episode with success at control step 407 of the 1,900 that the task allows. Gemini 4 Argon opened a jam jar with a dexterous hand in a DexHandExam test, which every other agent failed. These checks cannot show that the agents that failed would fail again. The leaderboard counts one attempt per agent and test. A test that one agent solves and another fails may therefore reflect the variation between runs as well as a difference between the agents.

Figure 24. First and last frames of the graded episodes of two Standard tests that break the order of Figure 8. Gemini 3.8 Flash folds a blanket in RoboARC, and Gemini 4 Argon opens a jam jar in DexHandExam.

How the agents fail. When GPT-6 Astra and another agent both fail a test after meeting different numbers of stages, GPT-6 Astra has usually met more of the task’s stages (Table 19). The kind of failure differs as well (Figure 16a). The five weaker agents fail without contact at 9.1% to 18.5% of the contact stages they reach, against 2.7% to 3.7% for the three strongest agents, and the weaker agents’ rate is higher in all 19 suites with at least 20 such stages per group (Section H.5). A lost hold makes up a larger share of GPT-6 Astra’s failures than of any other agent’s, but GPT-6 Astra also reaches more stages that need a hold. Per stage reached that needs a grasp, it loses the hold in 5.6% of cases, against 7.5% for the other seven agents, and the 95% interval of the difference, −0.2 to 3.6 points, includes zero (Section H.5). In a BEHAVIOR-1K task that all eight agents failed, for example, the three strongest agents each packed three or four foods into a lunch box and failed in three different ways, Kimi 3.1 Pro packed two, and the other four weaker agents none (Figure 30 in Section H.6).

Figure 30. Progress maps of the eight agents on preparing a lunch box (BEHAVIOR-1K, Standard Setting). Crosses mark the failure stages.

Harness. The harness also changes the result (Figure 16c). We ran GPT-6 Astra in Claude Code, Claude Opus 5.5 in Codex, and Gemini 4 Argon in Codex and in Claude Code, with the same tasks, time limits and reasoning effort as in the main evaluation (Section G). On the 456 core tests of the Standard Setting, Gemini 4 Argon solves 29% in Codex and 25% in Claude Code, compared with 24% in its own Antigravity CLI. The difference to Codex is significant, and the difference to Claude Code is not. GPT-6 Astra solves 34% in Codex and 22% in Claude Code, and this difference is significant also when only the first episode counts. Claude Opus 5.5 solves 26% in Codex and 25% in Claude Code. Part of Gemini 4 Argon’s gap may come from how each harness handles a reply that Gemini’s content filter blocks: Antigravity CLI ends the session, whereas Codex sends the request again and Claude Code asks the model to continue. Attempts with a blocked reply pass at 5% in Antigravity CLI and at about the rate of unblocked attempts in Claude Code.

Overlap among the three strongest agents. Figure 25 groups the core tests of each setting by which of the three strongest agents solve them. § 3.2 and § 3.5 give the totals. Here, an agent solves a test alone when the other two of the three fail it. In the Standard Setting, all three solve 43 tests and none of them solves 239. GPT-6 Astra alone solves 56, Claude Opus 5.5 alone solves 27 and Gemini 4 Argon alone solves 17. In the Privileged Setting, all three solve 139 tests and none of them solves 67. GPT-6 Astra alone solves 69, Claude Opus 5.5 alone solves 13 and Gemini 4 Argon alone solves 3. The Privileged tests that only GPT-6 Astra solves include tasks of precise contact and of arranging several objects. The Standard tests that only Claude Opus 5.5 solves include object transport tasks. In a ManiGuard task, for example, Claude Opus 5.5 closed the hinged lid of a glass jar and carried the jar to the goal, whereas GPT-6 Astra knocked the jar over (Section F).

Figure 25. Core tests of each setting, grouped by which of the three strongest agents solve them.

Control Approaches

A separate coding agent read the logs of 1,489 attempts, sampled at random, and labeled each attempt with the approach that produced its graded answer (Section B). Many Privileged tests give the agent a GPU to run the simulator, and their instructions name no training library. Apart from these, only eight tests give the agent a GPU: the GPU versions of the soccer match below and of a table tennis match, in each setting, and two RoboEngineer tests that ask the agent to train a walking policy.

The labels split Code to control into three kinds. Open-loop programs compute the whole motion in advance. Closed-loop programs read the robot’s state or cameras as they run. Parameter searches try many episodes to find a passing motion. In the Privileged Setting, agents use the simulator to compute or search for a motion before they submit it. Most of their programs in this setting are closed-loop, and 114 of the 549 programs search over many episodes. The same agent can use different control approaches for the same task in different settings. On a RoboTwin 2.0 task, GPT-6 Astra scripted a hand-over of the bottles with the simulator’s motion primitives in the Privileged Setting. In the Standard Setting, it sent end-effector commands one at a time and checked the camera images after each command (Section F).

In the Standard Setting, Direct control places the model inside the control loop, so the robot waits for the model’s reasoning before each chunk. In the Realtime Setting, the environment keeps running, and most attempts run the control loop in a program. In a badminton match of Embodied Olympics - Match, GPT-6 Astra let a program track the shuttlecock and swing the racket. At each pause, it revised the program from the program’s logs and won 11–4 (Section F).

The soccer match has a GPU version and a CPU version. In both soccer versions, each team has two Booster T1 humanoids controlled at the joint level, and the agent’s team plays a team with a learned policy on the same frozen scene, with a limit of four hours. The GPU version gives the agent a GPU with 12 GiB of memory, the mjlab simulator with the rsl_rl reinforcement-learning library and PyTorch, and training copies of the match that run on the GPU. Its instruction lets the agent train a controller or write one, and it gives an example of training a walking policy with rsl_rl. The CPU version has no GPU and no machine-learning framework, and its instruction allows learning code in NumPy or SciPy.

Table 4 counts the attempts of the eight agents on both versions in each setting, and of the four runs of the harness comparison (Section G) in the Standard Setting.

In the Standard and Realtime settings, 17 of the 20 attempts with the GPU trained a policy with reinforcement learning, and none of the 20 with only CPUs did. Of these 17, 14 played the match with the trained policy. DeepSeek V4.1 Flash, in the Standard Setting, wrote a joint controller by hand instead. With only CPUs, agents wrote walking gaits by hand, for example with inverse kinematics or a capture-point model. Of the 15 attempts whose final reports we read, 10 tuned the gait with CMA-ES, the cross-entropy method, or a grid or random search. One attempt, by GPT-6 Astra in Codex in the Realtime Setting, distilled its hand-written controller into a small network by supervised learning. We do not count it as training with reinforcement learning.

With the GPU, agents scored in 9 of 20 matches and won 4, all with a trained policy. With only CPUs, they scored in 3 of 20 matches and won none. Counting only the attempts in each agent’s own harness, the GPU attempts trained in 13 of 16 cases and won 3 times, and the 16 CPU attempts neither trained nor won. The same models appear in several settings and harnesses, so the attempts are not independent, and we report these counts without a significance test.

Figure 19 shows GPT-6 Astra in Codex on both versions of the soccer match in the Standard Setting. With the GPU, it trained walking and ball-control policies and combined them with hand-written tactics. Its robots walked 690 m, against 693 m for the opponent’s, and it won 10–8. With only CPUs, it wrote a gait by hand and tuned it with differential evolution and CMA-ES. Its robots walked less than half as far as the opponent’s and fell 59 times. The mercy rule ended the match early at 3–13. In this example, the two robots of GPT-6 Astra touched the ball 150 times with the GPU and 54 times with only CPUs. The opponent’s robots touched it 142 and 97 times, respectively. The goal times in the figure come from the referee state that the agent queried after each goal, and the frames come from the replay of each attempt.

In the Privileged Setting, every win reused the opponent’s walking network. With the GPU, 5 of 8 attempts ran reinforcement-learning code and 2 won. With only CPUs, 3 of 8 attempts won. In this setting, the agent can replay the same match in its own simulator. According to their code or final reports, all five winning attempts reused the opponent’s walking network that comes with the simulator, and none used a policy that it had trained.

We decided whether an attempt trained a policy from three signals: calls to a reinforcement-learning library or gradient updates in the agent’s code, the GPU memory that the resource monitor recorded, and the attempt’s final report, which another agent read. The three signals agree on every Standard and Realtime attempt with the GPU. The comparison covers one task and one frozen scene. The two versions differ at the same time in the GPU, the library, the training copies of the match and the instruction, so the comparison cannot separate their effects.

Example Attempts

This section describes further attempts that illustrate the findings. Every attempt is public, and most can be replayed step by step on the exam’s website.

RoboLab, the white mug in the center of the table (Standard, solved by Gemini 4 Argon). Gemini 4 Argon located the table and the mug in the point clouds of three calibrated depth cameras and found that the mug, 9 cm across, was wider than the gripper’s largest opening. It therefore grasped the mug’s wall at the rim, with one finger inside the mug, and moved the mug to the center of the table with inverse-kinematics targets sent as joint positions. To let go, it set the mug down and moved the inner finger to the middle of the mug before lifting the gripper. The episode, its only one, succeeded after about half an hour. GPT-6 Astra failed this test.

DexToolBench, red brush (Standard, solved by none of the three strongest agents). A DexToolBench task asks the agent to move a tool with a dexterous hand through goal poses, 38 in this task. Claude Opus 5.5 pushed the brush until it stuck out over the table edge, grasped it and reached goals 0 to 4. Goal 5 needed a 90-degree turn about the brush’s long axis. The palm-down grasp could not make it, and the brush flew out of the hand as the arm changed configuration.

RoboARC, IKEA Stegesund (Privileged, failed). With the full state, GPT-6 Astra pressed the halves together, slid in the shelf and drove the first bolt. Its hex key turned only 120 degrees per insertion, so one bolt took 40 minutes, and at the three-hour limit the second bolt was still 3.7 mm short. None of the three strongest agents solved this task.

BEHAVIOR-1K, setting a table (Privileged, failed). GPT-6 Astra spent 62 minutes placing two plates, and the robot then tipped over against the kitchen island on the way to the fridge.

RoboARC, threading a rope through six holes (Standard, failed). Claude Opus 5.5 mapped the wall, its holes and the coiled rope from depth point clouds. The rope slipped from the gripper whenever the robot turned its wrist, and the agent threaded none of the six holes.

RoboDojo, inserting a key (Standard, failed). Claude Opus 5.5 located the key and the lock from depth images but dropped the key at least nine times during hand-overs and large joint moves. Its most accurate insertion, at minute 36, was about 9 mm off and hit the face of the lock.

RoboDynamicExam, toss, recatch and deliver (Standard, failed). In this task, unlike that of Figure 15, the robot catches a thrown ball in a cup, tosses it up out of the cup itself, catches it again and releases it into a moving box. GPT-6 Astra caught the ball using depth images, but its first episodes ended early (one ended when the cup grazed a moving box) and used up its three allowed resets. In the last episode it caught and tossed the ball, but on the second catch the ball’s contact with the tilted cup was flagged as a limit violation.

RoboPaint, a character in running script (Standard, failed). With no resets available, GPT-6 Astra calibrated the stroke width on a practice sheet, wrote five strokes and then retouched the character in one long stroke for 38 minutes. The overlap and edge scores met their thresholds, but the long stroke raised the total length of the inked path to 2,048 mm, above the limit of 1,450 mm. In addition, one of the nine regions was only 65.7% covered, below the required 80%.

HumanoidBench, opening a door (Standard, failed). Claude Opus 5.5 kept the humanoid standing with ankle feedback and located the handle with stereo cameras. It tried more than ten ways of opening the door (pressing, hooking, pushing up and using both hands) and pulled the handle back by up to 26 cm, but the door never opened. The agent used 45 resets in 27 minutes.

Embodied Olympics - Fun, dribble slalom (Privileged solved, Standard failed). In the Privileged Setting, Gemini 4 Argon saved and restored exact simulator states to try candidate kicks in eight parallel processes and passed all 13 checkpoints of the slalom. In the Standard Setting, it estimated the positions of the robot and the ball from the pixels of a fixed camera, about 9 cm off to the side, and its best episode passed four poles before the 120 minutes ran out.

BiGym 2.0, closing the dishwasher (Privileged, solved). Gemini 4 Argon read the dishwasher’s code and collision shapes and pushed both racks in. The robot then crouched, hooked the door’s front edge and swung the door shut along an arc. Its answer, a 693-step trajectory, passed on replay.

RoboCasa365, defrosting meat in the microwave (Privileged, solved). Claude Opus 5.5 used Jacobian inverse kinematics and closed-loop control of the base to take the box out of the freezer, put it in the microwave, close the door and press start. The agent submitted its answer at minute 46, and the answer passed on replay.

ManiGuard, dusty transfer (Gemini 4 Argon, Privileged, solved). Working in a copy of the simulator that it kept running, Gemini 4 Argon wiped 18 dust particles from the box and then tilted the tray so that the potato rolled in without being touched. It finished at minute 26, and its answer passed on replay.

RoboPaint, cursive character (Privileged, solved). GPT-6 Astra read how the brush model draws a stroke, wrote a C++ program that reproduces this model and fitted three strokes with it. The first strokes that the agent executed passed every check, and the agent submitted its answer after four minutes.

BEHAVIOR-1K, bringing in plywood (Privileged solved, Standard failed). A mobile robot with two arms must carry three plywood sheets from the garden onto the corridor floor. In the Privileged Setting, Claude Opus 5.5 kept a copy of the simulator running, wrote separate scripts for grasping, opening the door and carrying, and tested each stage in this copy, where a fall of the robot required only a reset. Its answer passed after 72 minutes. In the Standard Setting, the same agent picked up two of the sheets, one in each gripper, but then drove the base backwards and sideways with a single command and did not check the camera images while the base moved, until the robot hit a curb and tipped over.

ManiGuard, dusty transfer (GPT-6 Astra, Privileged solved, Standard failed). The robot must wipe a box clean and then tip a tray so that a potato rolls into the box without being touched. GPT-6 Astra touched or flung the potato in its first three Privileged episodes, then reset and passed with a new wrist angle. In the Standard Setting, the test failed because a finger touched the potato once while the robot regrasped the tray, although the potato later reached the box.

DexToolBench, screwdriver (Standard, failed). Claude Opus 5.5 estimated the axis of a screwdriver from the point cloud but could not determine how far the handle had rolled about that axis, and it used all 20 resets that this task allows without reaching any of its 76 goal poses.

ManiGuard, closing a jar (Standard, solved by Claude Opus 5.5). The robot must close the hinged lid of a glass jar and carry the jar to a goal without breaking the task’s safety rules. Claude Opus 5.5 estimated the jar’s shape and the position of the hinge from the depth point cloud, slid the robot’s closed fingers under the lid and swung the lid shut about the hinge. It then lifted the jar by 1 cm to check the grasp before carrying the jar to the goal. GPT-6 Astra knocked the jar over while pushing the lid.

RoboDynamicExam, toss and recatch (Privileged). Claude Opus 5.5 used the arm’s Jacobian to compute the largest downward acceleration of the point from which the ball is tossed, and it chose its toss and recatch to stay within this limit.

RoboWits, tipping a cube out of a container (Privileged). Claude Opus 5.5 tried six pour positions to tip the cube out of the container and kept the steadiest.

RoboTwin 2.0, three bottles into a dustbin (Privileged and Standard). In the Privileged Setting, GPT-6 Astra read the task’s source code, scripted the grasps with the simulator’s motion primitives, handed the bottles from one arm to the other in mid-air and submitted its answer after five minutes. In the Standard Setting, it sent end-effector commands one at a time and checked the camera images after each command. It knocked one bottle over, picked the bottle up again with the other arm and finished about 30 seconds before the time limit.

Embodied Olympics - Match, badminton (Realtime, won). A program written by GPT-6 Astra ran in the background, located the shuttlecock with two cameras, fitted a drag model to its flight and swung the racket. Whenever play paused, the agent read the program’s logs and revised the program. It won 11–4 against an opponent that picks its shots with a learned policy.

Controlled Runs

This section defines the controlled runs of Sections 3.2 and § 3.5 (Figure 12a and Figure 16c).

Baseline. Every controlled run of Figure 12a uses GPT-6 Astra in Codex with the same reasoning effort, time limit and task versions as the leaderboard run of Figure 8, and the baseline is that leaderboard run. Each run covers the default tests of the 456 core tasks, and Table 5 and Figure 12a compare the runs on the 391 core tasks that have both a Privileged and a Standard test. Each run changes one or two features of the Standard interface and nothing else. Apart from these features, every task behaves exactly as in the leaderboard run.

Object-state query. We add two read-only calls to the Standard interface: one lists the objects in the scene, and one returns their state. The state consists of position, orientation, size and bounding box, linear and angular velocity, and the joint positions of the robot and of articulated objects such as doors and drawers. In Embodied Olympics - Match, the state of the opponent and of the ball counts as object state. The calls do not advance the simulation or use up simulated time, and they never return success flags, scores, thresholds or goal regions. The only progress value they return is the paint coverage of the canvas in RoboPaint. We remove the fields that would reveal what the agent is meant to find out itself: goal markers and zones, objects that appear only later in a BEHAVIOR-1K task, hint notes in RoboCasa365, and hazard labels in Embodied Safety Exam.

Unlimited resets with saved states. The reset budget of every task is raised to 1,000, and the interface gains calls that save the current simulator state under a handle and restore it later. The simulator service still grades the last episode as it runs. Restoring a saved state does not erase a rule violation that has already been recorded. In Embodied Safety Exam, for example, recorded violations remain after a restore.

Both. This run combines the two features above.

No success signal. The interface no longer reports that an episode has succeeded. The episode does not end at success, and no success flag, progress or score is returned. The agent ends the episode itself with a finish call, or the episode ends when the time runs out or the task’s own failure rule is triggered. The simulator service checks the success rule on the state at the end of the episode, so a success that was reached and then undone counts as a failure.

Harness comparison. For Figure 16c, we ran GPT-6 Astra in Claude Code, Claude Opus 5.5 in Codex, and Gemini 4 Argon in Codex and in Claude Code, with the same tasks, time limits and reasoning effort as the leaderboard runs. Codex and Claude Code support only their own provider’s API format, so an adapter passes their requests to the other provider and translates the answers back. The adapter also adapts the system prompt and model metadata, so these runs only approximate a native setup. We score every run on the 456 core tests of the Standard Setting, the tests of the Standard leaderboard. Each group of bars in Figure 16c compares a model’s runs on these 456 tests, and we test differences with a paired exact McNemar test.

Further results of the controlled runs. Table 5 gives the success rates of each run. Without the success signal, the success rate stays within a few points of the baseline.

The agent uses the saved states to retry. With unlimited resets, only 14% of its successes came in the first episode, and 66% came in an episode that started from a saved state. The saved states let the agent return to the moment before a hard stage instead of repeating the whole task. The state query helps most where the scene is hard to measure from cameras, as in iSpatialGym, RoboDojo and BEHAVIOR-1K, and it changes little in Embodied Olympics - Fun and DexHandExam.

Further results of the harness comparison. Table 6 gives the success rate of each model in each harness, counted by the final result and by first-episode success, that is, success in the first episode before any reset (Section C.2). Gemini 4 Argon does better in Codex than in its own harness (paired exact test, p=0.02p = 0.02 for the final result and p<0.05p < 0.05 for first-episode success). In Claude Code, the difference is not significant under either count (p=0.61p = 0.61 and p=0.20p = 0.20). In both harnesses, BEHAVIOR-1K contributes the largest part of its gain. GPT-6 Astra does better in Codex than in Claude Code under both counts (p<0.001p < 0.001), and most of this difference is already present in the first episode. For Claude Opus 5.5, the difference is not significant under either count. These runs are reported separately from the leaderboard of Figure 8.

Model (tests)CodexClaude CodeAntigravity CLI
GPT-6 Astra (456)34 / 26*22 / 18–
Claude Opus 5.5 (456)26 / 2125 / 21*–
Gemini 4 Argon (456)29 / 2525 / 2424 / 21*

Table 6. Success rate (%) of each model in each harness, on the 456 core tests of the Standard Setting. The first number counts the final result, and the second counts first-episode success. The model’s own harness is marked with an asterisk.

Failure Labels

This section describes the stage-level failure labels of § 3. Section H.1 explains how they were made, Section H.2 gives the taxonomy, and Section H.3 lists the excluded attempts. Section H.4 breaks the failures down by suite, robot body, agent and setting. Section H.5 gives further statistics, in the order of Sections 3.2 to § 3.5, and the limitations of the labels. Section H.6 describes the case studies of the figures in full, with further examples. The labels are released with the attempts at huggingface.co/datasets/MLL-Lab/efe-failure-analysis.

Labeling Procedure

Task graph. For each of 427 core tasks, the labeling agent, GPT-6.1 Sol in Codex, wrote one graph of stages, such as the graph of Figure 7a in § 3, from the task instruction, the verifier’s code and a successful attempt when one exists. A stage is a change in the world that a person would list when describing how to do the task, such as “open the fridge” or “put the first mango in the fridge”. Conditions on how well a stage must be done, such as thresholds, alignment or “without touching X”, belong to that stage’s condition and are not stages of their own. A condition that is already true at the start and must still be true at the end, such as a closed fridge, becomes a stage of its own that is met from the start, so that the progress map shows when an attempt breaks it. Each stage has a stage type, its prerequisite stages, a condition that can be checked on the replay video, and the contact it needs. The contact is a list of pairs of the robot part that touches and how it touches (who ×\times how), one list for each way of doing the stage. The edges of the graph give the order in which the stages must be met for the task to pass; a stage that holds from the start can hold before its prerequisites. The stage types fall into nine categories (Table 7). For example, material stages change the state of a material, such as pouring or cleaning, and body stages move the robot’s own body, such as walking or keeping balance. The 427 graphs have 2,411 stages, 5.6 per task on average.

Progress map. For each attempt, the labels give the time intervals during which each stage is met, on the timeline of the graded episode’s replay video (Figure 7b). The condition of a stage can stop holding after it has been met, so the progress map can go down. Progress is read from the replay video, or from the logs where there is no video; attempts without a submitted trajectory usually have no readable progress. An attempt reaches a stage if it meets the stage’s condition at some moment or if the stage is its failure stage.

Failure stage, phenomena and root causes. Each failed attempt is assigned at most one failure stage, so that failures can be counted by stage and compared across attempts and agents. A failure stage is located when the labels give one. If an irreversible event occurred, such as a fall or a violation of a task rule that makes the attempt fail, the failure stage is the stage during which the event occurred, unless an earlier stage had already failed permanently. Otherwise, it is the stage with the earliest uncorrected mistake among the stages whose condition was never met. Mistakes that the agent later corrected do not count. The labels also record where the attempt was when it failed: during the stage, between stages, while exploring, before the first stage or at the final success check. Up to two failure phenomena describe what visibly went wrong at the failure stage, and up to two root causes describe why it happened, with a video time or log entry supporting each cause. A time limit, a used-up budget or the agent stopping is never a phenomenon or a cause: the label names what was going wrong when the attempt ended. A root cause is given only when the evidence supports it, and it may stay at the category level “Control” when the subtype cannot be determined, as in 47% of the analyzed failures.

Writer and reviewer. GPT-6.1 Sol in Codex, at high reasoning effort, wrote every graph and the label files of most attempts from the inputs above, and Claude Sonnet 5.5 in Claude Code wrote those of the remaining attempts from the same inputs with the same prompts. Claude Sonnet 5.5 in Claude Code reviewed each file, with up to two rounds of fixes and a final confirmation round. The prompts were frozen before the final labeling. § 3.7 uses a second set of labels, which give the control approach of a sample of attempts (Section B). Agents wrote both sets of labels.

Human oversight. We defined the taxonomy (Section H.2) and settled it in team meetings before the final labeling, and the agents assign its values to each task and attempt. Earlier versions of the labels were reviewed by hand and in team meetings. Each error we found was written into the prompts as a worked example: the task or attempt, the wrong label, the right label and the rule. Most of the worked examples in the final prompts come from this review, and the rest come from rechecks by agents. The labels used here were then written again with the final prompts.

Core set covered. The labels cover 6,779 attempts on 857 tests of 427 tasks, with at most one labeled attempt per agent and test.

Taxonomy

Tables 7 to 10 list the values of the taxonomy. A root cause can also stay at the category level, such as Control (Section H.1). Stage types let us compare stages across tasks. A stage type names the robot action that directly brings about the stage’s change in the world, and the types fall into the nine categories of Table 7. In the task of Figure 7, for example, opening the fridge is a mechanism stage of type open, and putting the tomato in is a relation stage of type put in. The contact of a stage says who touches, such as a gripper, both arms or the mobile base, and how, such as grasping, pushing or supporting the robot’s weight. Failure phenomena describe what visibly went wrong at the failure stage. In the same example, Claude Opus 5.5 fails with a loss of balance, and DeepSeek V4.1 Flash fails with a lost hold. Root causes, such as a wrong perception or an unsuitable grasp, describe why a failure happened.

Excluded Attempts

The eight agents failed 5,272 of their 6,779 labeled attempts. The analyses in Sections 3.2 to § 3.5 exclude 500 of these failures, in two groups that overlap in 83 attempts. Table 11 counts them by agent and setting. Both groups are failures under the rules of the exam. They are excluded only from the statistics computed from the failure labels.

AgentSettingFailedFilter cutoffswith failure stageMissing trajectoryExcluded
Gemini 4 ArgonP2211089134163
S3263622036
R479309
Kimi 3.1 ProP238004949
Gemini 3.8 FlashP291419798
S4003203
Qwen3.8 MaxP311109494
GLM 5.3 FlashP340004141
All eight agents5,27216137422500

Table 11. Failed attempts excluded from the failure analysis, by agent and setting. P, S and R stand for the Privileged, Standard and Realtime settings. The column “with failure stage” counts filter cutoffs whose labels also give a failure stage with a physical phenomenon. The column “Excluded” counts filter cutoffs and missing trajectories, and an attempt in both groups counts once. GPT-6 Astra, Claude Opus 5.5 and DeepSeek V4.1 Flash are not listed: they have no filter cutoffs and 1, 2 and 4 missing trajectories in the Privileged Setting.

Filter cutoffs, 161 attempts. In those attempts the model provider’s content filter stopped the agent’s session, and the attempt ended or submitted nothing. Gemini 4 Argon accounts for 153 of them, Gemini 3.8 Flash for 7, and Qwen3.8 Max for one. Antigravity CLI, the harness of both

Who
Fingers/gripperone hand or gripper touches the object directly, also when the pose of a held tool is graded
Bimanualtwo hands or arms act on the stage together
Toolthe robot acts through a held or mounted tool
Legs/torsofeet, legs, head, torso or landing gear on a support surface
Mobile basewheels or a wheeled base, while moving
No contactno contact to label, as when flying, observing or reporting
How
Graspthe hand closes on the object
Push/sweeppush, nudge or sweep without closing the hand
Press/wipepress or wipe with pressure on the contact surface
Cradle/scoopsupport, catch or scoop the object from below
Hookhook with a finger or a tool
In-hand adjustmentreorient or reposition the object with the fingers without letting go
Supportthe body or base bears weight, as in standing or landing
Avoid contactpass without touching a named object

Table 8. Contact of a stage: who touches and how.

Gemini models, ends the session after a blocked reply, whereas Codex sends the request again and Claude Code asks the model to continue (Section E.3). In 37 of the 161 attempts, the labels also give a failure stage with a physical phenomenon, and the column “with failure stage” of Table 11 counts them. Of these 37, 34 are Gemini 4 Argon’s, 22 of them in the Standard Setting. Because almost all of them are Gemini 4 Argon’s, its failure profile in Figure 16a and Table 19 is the statistic that this choice affects most.

Missing trajectories, 422 attempts. In the Privileged Setting the agent’s answer is a recorded trajectory that the verifier replays or, in Embodied Olympics - Match, a program that the verifier runs. In these attempts the agent never submitted an answer, and the verifier recorded neither a trajectory nor a replay. We exclude these attempts because they leave no submitted trajectory or graded replay to label. Apart from the 83 that are also filter cutoffs, a missing trajectory is the agent’s own failure. Gemini 4 Argon has 134 such attempts, of which 79 are also filter cutoffs. Gemini 3.8 Flash has 97, Qwen3.8 Max 94, Kimi 3.1 Pro 49 and GLM 5.3 Flash 41. For Gemini 3.8 Flash and Qwen3.8 Max they are about a third of all Privileged failures, 97 of 291 and 94 of 311.

Failures without readable progress, 101 attempts. Besides the 500 failures above, the stage-level statistics, namely the failure rates of stages and the predictions of the failure stage, exclude 101 failed attempts whose labels give no stage progress, because neither the video nor the log shows when the stages held. These statistics therefore use 6,178 attempts, passed and failed: the 6,779 labeled attempts without the 500 excluded failures and these 101. The 101 attempts remain in the statistics that do not need stage progress, such as the failure profiles of Table 19. The 72 of them with a located failure stage, that is, a failure stage that the labels give, also remain in the concentration of failure stages and in the common failure stage.

Failures by Suite, Robot Body, Agent and Setting

Figure 26 and Figure 27 break the 4,772 analyzed failures down by suite, robot body, agent and setting. Each row splits the group’s failures among the labels in four blocks, so its values are shares of the failures and should not be read as failure rates. The first block says how far the failed attempt got. Failures on tasks with a single stage form their own column. For longer tasks, the block uses the attempt’s peak progress, the largest share of the task’s stages met at one moment, counting stages already met at the start. Its columns are no stage, more than none but under half, at least half but not all, all stages, and not readable. Peak progress is not readable when the status of at least one stage could not be read, and this column holds 448 failures, among them 98 of the 101 failures without any readable progress (Section H.3). The second block gives the category of the failure stage. The third and fourth blocks give the failure phenomena and the root causes. A failure can have up to two of each, so these two blocks can add up to more than 100%. The rows pool the three settings, except the rows of the settings themselves. The robot body is the kind of body of the task’s robot, as in Table 2.

Figure 26. Failures by suite. Each cell gives the share (%) of the suite’s analyzed failures in the column, with the three settings pooled, and nn is the number of failures. A failure can have up to two phenomena and up to two root causes, so these two blocks can add up to more than 100%.

Figure 27. Failures by robot body, agent and setting, in the same format as Figure 26. Here too, the phenomena and the root causes can add up to more than 100%.

Shares are rounded, <1 marks a share above zero but below 1%, and empty cells are zero. Gray labels mark groups with fewer than 20 failures. The phenomena and root causes are those of Table 9 and Table 10, and None means that a failure has no label in the block. Other contact pools the four rarer contact phenomena, and Balance or gait pools loss of balance and gait failure. Other covers out of reach, reporting and defects that the taxonomy does not name, such as gaps in a painting or folds left in a cloth. No analyzed failure lies at a safety or information stage or has a task or environment root cause, so these columns are omitted.

CategoryPhenomenonMeaning
ContactFailed to make contactthe contact the stage needs never existed, as in a missed grasp or fingers that never closed on the object
ContactLost holdheld and then lost: slipped or dropped while held or carried
ContactAbnormal contact forcevisible or sensed wrong force, such as a part not fully inserted or a latch not engaged
ContactFailed to releasethe object stays in the hand or tool when the stage needs it to leave
ContactUnstable after releaseafter a planned release the object falls over, rolls away or bounces
ContactDrags the basemanipulating a part moves the body that should stay still
MotionInaccurate posethe hand, tool or object misses the required position or orientation
MotionObstruction/collisionblocked by contact, or knocking over or displacing objects
MotionOut of reachevidence of a reachability or joint limit
MotionDynamic timing errorcontact with or release of a moving object too early or too late
Whole bodyLoss of balanceany robot loses support or tips over
Whole bodyGait failurewalking, running or jumping falls short without a fall
ConstraintConstraint violationa task rule broken, with no more specific phenomenon
ReportingMissing report, incorrect reporta graded information act, such as a warning or a required answer, missing or wrong
Not attemptedNot attempteda stage the graded attempt needed was never started

Table 9. Failure phenomena: what visibly went wrong at the failure stage, up to two per attempt.

CategoryRoot causeMeaning
CognitionTask understandingthe behavior shows the task was misread
Reasoning/planningwrong goal, stage order or overall strategy
Perceptionthe robot acted on a wrong estimate of where or what something is
Safety judgmentrecords show a hazard was not recognized or misjudged
False completion beliefa false belief that the task was done affected stopping or decisions
ControlGrasp and contact designthe contact geometry or load layout does not suit the stage
Trajectory or IKgoal and contact plan are feasible, but no trajectory or IK solution is found or followed
Feedback controltracking or stabilization under feedback fails
Parameter tuningparameters shown to be wrong
DeliveryDelivery operation errorrecording, saving or checking the submitted answer went wrong, such as recording without a reset
Task or environmentTask or environmentevery legal solution is ruled out, or an event the agent cannot react to

Table 10. Root causes: why the failure happened, up to two per attempt, each with its evidence.

By suite. Figure 26 gives the rows of the suites. The categories of the failure stages differ across suites, and in some suites all failures lie at one kind of stage. Every RoboPaint failure is at a material stage, every Embodied Olympics - Match failure is at a competition stage, and every DexToolBench failure with a located failure stage is at a pose stage. In Embodied Olympics - Match and DexToolBench, every stage of every task is of that category. How far the failed attempts get also differs. In DexToolBench and KinDER, most failed attempts never meet a stage of the task, while in RoboPaint almost every failed attempt meets at least half of the stages. In Embodied Safety Exam and ManiGuard, some failed attempts meet all stages at one moment. In Embodied Safety Exam, most of these attempts have a constraint violation. Failures at the same kind of stage can still differ between suites. In both RoboLab and RoboCasa365, most failures lie at relation stages, but the most common phenomenon of RoboLab failures is a lost hold, and in RoboCasa365 it is a failure to make the contact the stage needs. Among the suites with at least 20 failures, a cognitive root cause is most frequent in Embodied Safety Exam, RoboPaint and iSpatialGym, at 28 to 38% of their failures, compared with under 8% in ten others, such as HumanoidBench, MyoSuite and DexToolBench.

By robot body. The upper rows of Figure 27 give the robot bodies. Each kind of body comes from a few suites: three quarters of failures of mobile manipulators are BEHAVIOR-1K’s, and all those of quadrupeds come from MuJoCo Playground. The bodies also differ in their mix of settings. These rows therefore do not isolate the effect of the body. Loss of balance is the most common phenomenon for humanoids, quadrupeds and mobile manipulators. Failure to make contact is the most common for dexterous hands and for the musculoskeletal models of MyoSuite, and most failed attempts with a dexterous hand never meet a stage of the task. For

robots with one or two arms, collisions and inaccurate poses are the most common phenomena apart from task-specific defects.

By agent and setting. The lower rows of Figure 27 give the agents and the settings. For every agent, relation stages hold the largest share of the failures, from 33 to 43%, and the shares of the other categories differ by about five points or less between agents. The agents differ in how far they get and in what goes wrong, as in § 3.5. The failures of the five weaker agents more often meet no stage of the task, and more often lack the contact that the stage needs. Gemini 4 Argon’s failures more often have the not-attempted label. Between the settings, Realtime failures more often lie at competition and dynamic stages and more often involve a timing error. Privileged failures more often have the not-attempted label. Collisions are more common in Standard failures, and losses of balance are more common in Standard and Realtime failures than in Privileged ones. The settings cover different mixes of tests, so these comparisons do not isolate the effect of the setting.

Further Statistics

Failure rate of a stage. The failure rate of a stage is the number of attempts that failed at the stage divided by the number of attempts that reached it. Attempts that never reached the stage are not counted, so the rate measures whether an attempt that gets to the stage fails there. For the rate of a stage type, each stage of that type counts separately, so an attempt that reaches three put-in stages adds three stages. A stage counts as failed when it is the attempt’s failure stage, and as met when its condition held at some moment, even if it already held at the start or stopped holding later. To compare the two settings without mixing suites, we compute the rate within each suite and average over the suites in which attempts reach the stage at least eight times in each setting, weighted by the smaller of the two counts. Table 12 and Table 13 give the results.

ValueSuitesWeightPSS − P
WhoLegs/torso68442764+37
WhoBimanual107413971+32
WhoTool129102455+31
WhoFingers/gripper263,4202446+22
WhoMobile base61,1692643+18
WhoNo contact31731124+13
HowIn-hand adjustment4962970+41
HowCradle/scoop82682357+34
HowSupport111,9562752+25
HowAvoid contact32302449+25
HowPress/wipe63893255+23
HowGrasp253,3912446+22
HowHook41863150+19
HowPush/sweep166401731+14

Table 13. Failure rate by contact in percent, compared within suites as in Table 12. Bimanual stages are the hardest in both settings, and among the values of how, the failure rates of in-hand adjustment and of cradling or scooping rise the most.

The Privileged and Standard settings. The two settings differ in two conditions that matter for Section 3.2. In the Privileged Setting the agent reads the exact pose of every object and can reset the simulator without limit, so it can compute a precise motion from the full state, test it and try again within the time limit before it submits an answer. In the Standard Setting it must estimate poses from camera images and depth, the episode is graded as it runs, and the task limits the number of resets, so a precise stage is attempted on estimates and an error can remain in the graded episode. The comparisons of Section 3.2 concern the stages of a task. They do not concern how the agent controls the robot, which changes with the setting (§ 3.7). We split the stage types into two groups by their definitions in the taxonomy (Table 7). A stage type is precise if it needs a tight fit, a mated or threaded relation, a target pose or path, or contact with a moving object. The 13 precise types are insert, assemble, screw/unscrew, thread/hang, connect, adjust pose, follow path, stabilize, regrasp, throw, strike, catch and dribble. A stage type is coarse if it needs only to reach a region, a container, a grasp or the open or closed state of a mechanism.

The 12 coarse types are place, put in, take out, gather, pick up, push, pull, open, close, press, switch on/off and navigate. All other types, such as the material types and the body types other than navigate, are in neither group. Section 3.2, Section E.1 and Table 14 use these groups.

Compared within suites as in Table 12, the failure rate rises from 22% to 56% at precise stages, in 9 suites, and from 22% to 38% at coarse stages, in 25 suites. With suite fixed effects and intervals clustered by task, the precise stages rise by 19 points more than the coarse stages (95% interval 13 to 25). These comparisons use different attempts in the two settings. When the same agent reaches the same stage of a task in both settings, the failure rate rises by 25 points at precise stages and by 15 points at coarse stages, and the difference between the two groups, 10 points, is smaller but significant (95% interval 5 to 16 points). Follow-path stages, most of them waypoints of DexToolBench, make up 38% of the precise stages in this comparison, and their rate rises by 11 points. Without them, the rate at precise stages rises by 33 points, 18 points more than at coarse stages (95% interval 12 to 25). Among the types compared in at least two suites in Table 12, the rates of striking, assembling, adjusting a pose and catching rise by 35 to 41 points in this paired comparison, and those of closing, placing and picking up rise by 3 to 17 points.

The labels cannot show how much of the loss comes from perception and how much from control (see the limitations at the end of this section). The labels also cannot show how much comes from the missing state and how much from the missing resets. The controlled runs of Section G separate the two at the level of success rates: on these tests, adding either one alone closes a third to a half of the gap. Joined with the labels, they also separate the two by stage type.

State and resets at precise stages. We join the controlled runs of GPT-6 Astra to its labeled failures in the Standard Setting, using the results of Section G. Of these failures, 256 have a located failure stage. Of them, 91 are at precise stages and 97 at coarse stages, split as above. Of the rest, 36 are at material stages and 32 at other stages. Table 14 gives the share of them that each run solves.

Each run makes one attempt per test, so some failures would turn into successes simply by trying again. The run without the success signal approximates such a second attempt. It is not an exact repeat, since the agent must end the episode itself, and its success rate is within 4 points of the baseline’s Table 5. It solves 19% of the failures at precise stages and 14% of those at coarse stages.

At precise stages (Table 14), 22 failures are solved by exactly one of the state-query run and the reset run, and the reset run solves 18 of them (exact McNemar test, p=0.004p = 0.004).

The logs show how the agent used each feature. Of the 33 precise failures that the resets solve, 31 succeed only after a reset or a restored state. The agent did call the state query in the runs that have it. The run with both solves more of these failures than the run with resets alone (16 failures that only one of the two runs solves, 13 of them by the run with both, p=0.021p = 0.021).

Let the recovery of a run be its share of solved failures minus that of the run without the success signal, both on the same failures. On the precise failures that have a Privileged test, the recovery of the run with both features is 53% of the recovery of the Privileged Setting, with a 95% interval from 35% to 70% over failures. The Privileged Setting solves 23 more precise failures than the run with both. Stages that follow a path, thread or assemble account for 15 of these 23 failures. At striking and catching stages, the run with both solves exactly the failures that the Privileged Setting solves.

These comparisons have limits. Precise and coarse failures come from different suites. Most precise failures are in Embodied Olympics - Fun, DexToolBench and RoboARC, and 38 of the 97 coarse failures are in BEHAVIOR-1K. Leaving out any one suite reverses none of the comparisons between runs or between the two groups of stages. The intervals and tests treat the failed tests as independent.

Concentration of failure stages. The concentration of a test measures how much the agents that failed it agree on the stage at which they failed. We measure it with the Simpson index: for a test that n≥2n \ge2 agents failed at located stages, with nkn_k of them at stage kk, it is D=∑knk(nk−1)/(n(n−1))D = \sum_k n_k(n_k - 1)/(n(n - 1)), the share of pairs of these agents that failed at the same stage. We average DD over the tests of a setting whose tasks have at least two counted stages, and resample tests 2,000 times for the 95% interval. If each agent’s failure stage is sampled independently with probabilities qkq_k, the expected value of DD is ∑kqk2\sum_k q_k^2. The random-stage value, the gray bars of Figure 14a, assumes that each agent fails at a random stage but keeps one property of the observed failures: where in a task they usually fall. First, we order the KK stages of each task graph topologically, breaking ties by their order in the task file, so that the ii-th stage covers the interval [(i−1)/K,i/K][(i - 1)/K, i/K]. Second, over all failures in the other tasks of the same setting that have at least two stages, we build a ten-bin histogram of the relative position (i−1/2)/K(i - 1/2)/K of the failure stage. Third, each stage receives the mass of this histogram over its interval, and these masses are the probabilities qkq_k. A simpler uniform baseline, qk=1/Kq_k = 1/K for a task with KK stages, gives 28% in the Privileged Setting, 28% in the Standard Setting and 30% in the Realtime Setting. This baseline ignores that early stages fail more often, so the main text does not use it. The value of DD for the actual failure stages exceeds the random-stage value in 74% of Standard and 64% of Privileged tests, compared with about 39% in both if the failure stages were sampled from the random-stage probabilities qkq_k. The 95% interval of the mean difference is 21 to 28 points in the Standard Setting and 14 to 22 points in the Privileged Setting. In the Realtime Setting, which Section 3.3 excludes because only 27 tests qualify, the concentration is 53%, compared with 43% for random failure stages, and the interval of the difference, 1 to 20 points, lies just above zero. On tasks with at least three stages, the concentration in the Standard Setting is 50% over 194 tests, compared with 26% for random failure stages and 18% for the uniform baseline. The comparison with random failure stages assumes that, apart from where in a task failures usually fall, the failure stages of different agents are independent. Because each agent makes one attempt per test, every pair counted in DD compares two different agents. For the three strongest agents alone, we use the Standard tests that at least two of them failed at located stages, and we build the histogram from their failures on the other tasks. Over these 217 tests, the concentration is 56% (95% interval 50% to 62%), compared with 31% for random failure stages, and the 95% interval of the difference is 20 to 31 points. With the histogram of the eight agents, the random-stage value is 35%. Pairs that agree at a stage after the first contribute 28 points of the 56%, compared with 13 for random failure stages. In 34% of these tests, exactly two of the three agents have located failures, so the concentration on each such test is either 0 or 1. Among the located Standard failures, 62% of those of the five weaker agents are at the first stage, compared with 41% for the three strongest. In the Privileged Setting, 69 tests qualify, and 78% of them have exactly two located failures among these agents. The concentration is 40%, compared with 25% for random failure stages, and the 95% interval of the difference is 5 to 25 points.

Shared failure stages. We count the Standard tests with at least three stages that four or more agents failed. In 64 of these 190 tests, at least 80% of the failures fall on one stage. Table 15 lists one example per suite among the 64. The Realtime test of the same RoboDynamicExam task as Figure 15 shows the same shared stage: seven agents failed at the recatch.

SuiteTaskStagesShared stageTypeFailed here
BEHAVIOR-1Kclean up branches and twigs5first branch into the binput in8/8
Embodied Safety Examthree spice jars onto a tray3second jar onto the trayplace8/8
RoboARCthread a fiber bundle through rings around a post18first fiber through its ringthread/hang7/8
RoboCasa365defrost meat in the microwave4container into the microwaveput in7/7
DexHandExaminstall the front guard of a fan5seat the front guardassemble6/7
RoboDynamicExamcatch, return through a slot, deliver to a moving cart3recatch after the reboundcatch7/8
RoboDojoput two slices of bread into a toaster and press the lever3first slice into a toaster slotput in6/7

Table 15. Examples of shared failure stages in the Standard Setting, one per suite. The Stages column gives the number of stages of the task. The column “Failed here” gives the failed attempts that failed at this stage over all failed attempts with a located failure stage, without the attempts excluded in Section H.3.

The common failure stage. A test has a single most common failure stage when one stage holds more of its located failures than any other. In all, 203 Privileged and 264 Standard tests have one. For the statements of Section 3.3 built on it, the stages of a task are ordered by their depth in the task graph, with ties broken by the order in the task file. To ask whether stage types predict this stage, we estimate, for each setting, how often each stage type fails once an attempt reaches it, using the tasks other than the test’s own. Each rate is shrunk toward the setting’s mean rate with the weight of ten attempts. A second variant uses the rate of each type within the test’s own suite, shrunk toward the pooled rate of the type in the same way. In a sequential model with failure rates h1,…,hkh_1,\ldots,h_k in stage order, the chance that the failure stage is stage kk is proportional to hk∏j<k(1−hj)h_k\prod_{j<k}(1-h_j). The predicted stage is the one with the highest chance, and the concentration expected under these rates is ∑kqk2\sum_k q_k^2 for the normalized chances qkq_k. To measure how strongly getting past the common stage predicts passing the test, we take for each attempt the most common failure stage of the other agents’ failed attempts on the same test, when at least two of them failed there and no other stage ties with it. We count the attempt as past this stage when its progress map shows the stage met at some moment. Two agents share a failure phenomenon when their labels name at least one phenomenon in common, and pairs in which either attempt has no phenomenon are excluded. Agents that fail at the same stage need not fail there in the same way: in the Standard Setting, two agents that failed at the same stage share a failure phenomenon in 40% of pairs, as in the RoboDynamicExam task of Figure 15 (Section H.6).

Table 16 compares the common failure stage with what the stage types predict, with rates from the other tasks and with rates from the same suite, and Section 3.3 reports the rates from the other tasks. A prediction from the stage types finds the common stage in 73% of the tests, and always naming the first stage finds it in 65%. On tasks with at least two types of stages, the two rules find it in 67% and 56%. The concentration it predicts is 36% in the Privileged and 48% in the Standard Setting, against 50% and 61% observed. Table 17 shows how strongly getting past the common stage predicts passing the test. Part of this link follows from the order of the stages, since an attempt that reached more stages is closer to the end. The settings still differ: in the Privileged Setting most attempts that pass the common stage go on to pass the test, while in the Standard Setting fewer than half do, and most of the others fail at a later stage.

The common failure stage across settings. In all, 188 tasks have a single most common failure stage in both the Privileged and the Standard Setting, and it is the same stage in 152 of them, or 81%. For a baseline, we sample, for each of these tasks and each setting, as many failure stages as the task has located failures in that setting. The samples are independent, with the stage probabilities qkPq_k^{P} and qkSq_k^{S} of the random-failure-stage value above and the stages in depth order. We keep the samples in which both settings have a single most common stage. Averaged over the tasks, the two most common stages then agree with probability 49%. In 105 of the 188 tasks the common stage is the first stage in both settings, so much of the agreement comes from tasks that fail at their first stage. On the other 83 tasks the common stage is still the same in 57% (95% interval 46% to 67% over tasks). Under the same baseline, leaving out the samples in which both common stages are the first stage, the average over these 83 tasks is 9%. Where the two common stages differ, the one of the Standard Setting lies earlier in the task in 25 tasks and later in 11. These counts describe where failures fall. They do not show that an agent follows the same plan in the two settings.

PrivilegedStandard
Passed the test, all attempts covered3010
Passed the test, attempts past the common stage6433
Passed the test, past a common stage that is not the first7641
Failed exactly at the common stage, attempts that never met it7783

Table 17. Getting past the common failure stage, in percent. For each attempt, the common stage is the most common failure stage of the other agents on the same test. The table covers the 1,235 Privileged and 1,874 Standard attempts that have such a stage and a readable status at it.

Pooled failure rates. Table 18 gives the rates behind Figure 14b, for each setting and for both together: every attempt that reached a stage counts, whatever its suite. The stage categories are pooled over all their stage types, and the rows list the stage types found in at least three suites and reached at least 30 times in the two settings together.

Interpreting the stage rates. The rates of Figure 14b and Figure 28 show an association between kinds of stages and failure. They cannot be read as causal effects, for three reasons. First, each kind of stage occurs mostly in particular suites and with particular robots: the part that makes contact depends on the robot, so legs and torso belong to legged robots, and most material stages come from RoboPaint and Embodied Safety Exam. Second, stages differ in how much the robot must do in them. A relation stage usually covers grasping, carrying and releasing its object, because the task graphs list picking up as a stage of its own in only 11 of the 229 tasks with relation stages. A rate per stage therefore compares units of different size. Third, the taxonomy records what a stage changes but not how much contact it needs. Explaining the rates by contact therefore requires a judgment of what each kind of stage asks of the robot. Opening a door or a lid fails at 38% of the stages that attempts reach. Closing one fails at 19% when it does not already hold at the start, and at 7% if the doors and fridges that are closed from the start count too. Relation stages fail at 36% of the stages that attempts reach with one hand or gripper, at 52% with two arms and at 65% with the legs or torso. Rates with tool contact can be lower or higher than with direct contact: stages on moving objects fail less often when the robot acts through a cup or a racket, while pose stages fail more often when it moves an object through a tool. The same mode of contact can be easy or hard: stages done by pressing or wiping fail at 8% of the stages that attempts reach when they press on a mechanism, such as a button, and at 69% when they wipe or press on a material. The pose, relation and dynamic categories lie between these extremes, and their order differs between the two settings, so Figure 14b, which counts both settings together, does not rank these three reliably.

What predicts the failure stage. We predict whether an attempt that reaches a stage fails there from the setting plus one other attribute, such as the stage type or the contact. We use five-fold cross-validation over tasks, so that the tasks of the held-out fold are not seen in training, and report the relative drop in log loss compared with the setting alone. The stage type gives 7.4%, contact as who ×\times how 6.9%, the stage category 5.6%, the suite 3.9%, how alone 3.8%, the robot’s body, grouped into eight kinds, 2.8% and who alone 1.8%. The agent gives 13%. With the setting and the suite as the baseline, the stage type adds 8.2%, who ×\times how 7.4%, the category 6.6%, how 5.6% and who 1.6%. To test whether difficulty rankings hold across suites, we hold out one suite at a time, rank each pair of values by their failure rates on the other suites, where each value must be reached at least 20 times, and check the ranking inside the held-out suite, where it must be reached at least 10 times. The rankings agree in 55% of 313 pairs for stage types, 53% of 150 pairs for how, and 66% of 76 pairs for stage categories.

Failure profiles of the agents. Table 19 gives the share of each agent’s failed attempts in the Standard and Realtime settings with each label. A failed attempt can have two phenomena, so the columns do not sum to 100%. The last two rows compare each agent with GPT-6 Astra on the tests with at least three stages that both failed, with ties excluded. Ties are common. Of the 145 tests that GPT-6 Astra and Claude Opus 5.5 both failed with readable progress, GPT-6 Astra reached more stages on 68 tests, Claude Opus 5.5 on 19, and the two tie on 58. Between the other two strongest agents, Claude Opus 5.5 reached more stages than Gemini 4 Argon in 51% of the 68 such tests. In the Standard Setting, two of the three strongest agents that failed the same test failed at the same stage in 55% of pairs pooled over tests, and two of the five weaker agents did so in 72%. When they did, that stage came after the first stage in 51% of the 277 such pairs of the strongest agents and in 27% of the 1,700 such pairs of the weaker agents.

The shares of failed attempts depend on how far an agent gets: an agent that reaches more stages that need a hold has more chances to lose one. The two rows “per stage reached” of Table 19 therefore divide by the stages that the attempts reached, using the attempts with readable stage progress. Of the reached stages that need contact, the weaker agents fail without contact at 9.1 to 18.5%, and the strongest agents do so at 2.7 to 3.7%. The pooled difference, 11 percentage points, has a 95% interval from 9 to 13 percentage points, and the weaker agents’ rate is the higher one in all 19 suites with at least 20 such stages per group. Over the five pairs of GPT-6 Astra with a weaker agent, on the tests that both failed, 9 to 10% of GPT-6 Astra’s failures lack contact, compared with 19 to 25% of the weaker agent’s. Lost holds show a different pattern. They make up more of GPT-6 Astra’s failed attempts than of any other agent’s, but per stage reached that needs a grasp, GPT-6 Astra loses the hold in 5.6% of cases, compared with 7.5% for the other seven agents pooled, and the 95% interval of the difference over tasks is −0.2 to 3.6 points. Per stage reached, GPT-6 Astra’s observed rate is therefore lower than that of the other agents pooled, although the interval includes zero. Among single agents, only Gemini 4 Argon has a lower rate. Constraint violations, too, make up more of GPT-6 Astra’s failures than of any other agent’s. Per 100 attempts, Qwen3.8 Max has 4.5 and GPT-6 Astra 4.4, so per attempt the two are about equal.

Limitations of the labels. The labels used here were written and reviewed by agents (Section H.1), and GPT-6.1 Sol, which wrote most of them, is from the same family as GPT-6 Astra. Labels come from the replay video and the logs, without the simulator state, so a wrong estimate and a wrong motion often cannot be distinguished. Perception is named in 4% of Standard failures, and 47% of the analyzed failures have a root cause given at the category level, “Control”. The root-cause layer therefore supports few conclusions, and the analyses of Sections 3.2 to § 3.5 rest mainly on failure stages, stage types and phenomena. Claude Sonnet 5.5 labeled all Kimi 3.1 Pro attempts and 629 of the analyzed attempts of the other seven agents. The two label writers leave the root cause empty at different rates, so the root-cause shares in Figure 27 and Table 19 should not be used for comparisons between agents. Each agent makes one attempt per test, so per-agent shares on a few dozen tests have wide uncertainty.

Case Studies

This section gives in full the attempts behind the examples of § 3, each with its figure. Times are seconds on the replay video of the graded episode.

Carrying in groceries (Figure 7). In this BEHAVIOR-1K task a mobile two-armed robot must carry a tomato and a carton of milk from a car into the fridge and then close the car and the fridge. The robot must open the garage door and the fridge before either food can go in, and close the fridge only after both are in, while closing the car does not depend on the other stages. Stage 5, the closed fridge, is met from the start and stops holding when the robot opens the fridge. GPT-6 Astra in the Privileged Setting reaches all six stages and passes. Claude Opus 5.5 in the Standard Setting opens the garage door and the fridge, closes and reopens the fridge twice, puts the tomato in and falls while carrying the milk through the door, so its failure stage is stage 4. DeepSeek V4.1 Flash in the Standard Setting lifts the tomato but drops it, so it fails at stage 3, and only the closed fridge is still met.

Installing a fan guard (Figure 29). In this DexHandExam task a dexterous hand must seat the front guard of a fan on its rear guard and close four clips. The two upper clips need the seated guard, and the two lower clips do not. In the Privileged Setting, Claude Opus 5.5 closed the lower clips, seated the guard at 20 s and closed the upper clips. In the Standard Setting it closed the lower clips in the other order, but it held the guard 36 to 39 degrees off the fan’s axis, whereas seating allows at most 5 degrees. When it pushed the guard on, the fan moved, and the simulator service ended the episode at 24 s. The labels attribute the failure to a wrong estimate of the guard’s pose in the hand. In the same Standard test, GPT-6 Astra seated the guard twice, each time for less than 2 s. As the hand withdrew its fingers, the guard came off with it and reopened a lower clip, and the attempt ended when the guard dropped.

Figure 29. Task graph of a DexHandExam task, installing the front guard of a fan, and the stages that three attempts met over time. A bar shows when a stage’s condition held, and a cross marks the failure stage.

Catching a ball twice (Figure 15). In this RoboDynamicExam task the robot must catch a thrown ball in a cup, pour it into a slot, catch it again after it rolls down and rebounds off a ramp, then pour it into a moving cart. In the Standard Setting, all eight agents failed, and seven of them caught the thrown ball and failed at the recatch stage. In four attempts, the episode ended because the cup touched the slot or the rebound ramp, which the task forbids. GPT-6 Astra, Gemini 4 Argon, Gemini 3.8 Flash and Qwen3.8 Max hit them while pouring the ball out or moving the empty cup downstream. The cup of Claude Opus 5.5 was still too high when the ball came off the ramp. Claude Opus 5.5 reported that the ball can be caught for only about 1.3 to 1.6 s after it leaves the cup, while straightening the cup after the pour alone takes about 1.4 s. In the runs of DeepSeek V4.1 Flash and GLM 5.3 Flash the ball never left the tilted cup. GLM 5.3 Flash then carried the ball to the cart, where it stayed in the cup again. Kimi 3.1 Pro did not move the arm, and its episode ended after about 1 s, so it failed at the first catch. In the Privileged Setting, GPT-6 Astra, Claude Opus 5.5 and Kimi 3.1 Pro completed all three stages, and Gemini 4 Argon, Gemini 3.8 Flash and Qwen3.8 Max submitted no answer.

Preparing a lunch box (Figure 30). In this BEHAVIOR-1K task a mobile two-armed robot must put two apple halves, a sandwich and a cookie from a cutting board, and a tea bottle from the closed fridge, into a lunch box, and leave the fridge closed. The closed fridge is met from the start. All eight agents failed the Standard test. GPT-6 Astra put both apple halves, the cookie and the sandwich into the box by 250 s. Driving to the fridge with empty hands, the robot tipped over at 284 s, and the episode ended with the fall. Gemini 4 Argon packed the same four foods by 85 s, then pulled open a tall dark cabinet that it had mistaken for the fridge, and it had not opened the fridge when the episode ended. The labels name a perception error. Claude Opus 5.5 packed an apple half and the cookie. At 210 s, the sandwich slipped from the gripper and landed behind the box, and placing the sandwich is its failure stage. After packing the second apple half at 242 s, the robot tipped over on its way to the fridge. Kimi 3.1 Pro packed the two apple halves by 292 s, and the robot tipped over at about 380 s as it went for the sandwich. Gemini 3.8 Flash opened and closed the fridge but never lifted an apple half off the board, and Qwen3.8 Max never grasped one in its 845 s episode. DeepSeek V4.1 Flash and GLM 5.3 Flash tipped the robot over within 8 s while exploring the kitchen and packed nothing. No agent passed this task in the Privileged Setting either, where the three strongest agents reached five or six of the seven stages.

Threading a rope through five passages (§ 3.5, Figure 31a). In this RoboARC task two arms with parallel grippers must thread a rope, in order, through a grommet in a wall, a ring, a second grommet, a second ring and a third grommet. Each passage is a stage. In the Standard Setting, Claude Opus 5.5 passed. GPT-6 Astra threaded the first three passages by 100 s. At the second ring, the fingers caught on the rim. At 106 s, the rope slipped out of the gripper, and the time limit ended the episode soon after. Gemini 4 Argon also threaded the first three passages, by about 55 s, but the rope never went through the second ring. Four of the five weaker agents did not get the rope through the first grommet, and two of them never got the hold they needed. GLM 5.3 Flash closed its gripper on the rope at 116 s and again later, and each time lifted the gripper without the rope. Qwen3.8 Max pushed the rope end into the grommet, but at 115 s the gripper on the other side closed without catching it, and the rope slid back. Gemini 3.8 Flash lifted the rope and dropped it at 16 s, and DeepSeek V4.1 Flash tried to thread the first grommet

in the wrong direction. Kimi 3.1 Pro threaded the first grommet and the first ring by 395 s, and at the second grommet the gripper closed without catching the rope.

Fetching cereal and a bowl (§ 3.5, Figure 31b). In this RoboCasa365 task a one-armed robot on a mobile base must open a cabinet, put a cereal box and a bowl from it on the counter, and close the cabinet. The closed cabinet is met from the start. In the Standard Setting, GPT-6 Astra and Claude Opus 5.5 opened the cabinet, put the cereal on the counter and failed at the bowl after holding it. GPT-6 Astra tried to pick up the bowl by its rim from 16 s. At 23 s, the bowl slipped out of the pinch and stayed on the shelf, and later pinches tipped it onto its side. Claude Opus 5.5 lifted the bowl out of the cabinet, and at 30 s, as the arm lowered it, the bowl slipped from the fingers and fell out of reach. Gemini 4 Argon put the cereal on the counter and carried the bowl out of the cabinet but did not put it on the counter, and it left the right door open. Four of the five weaker agents failed without the contact that their failure stage needed. Kimi 3.1 Pro opened the cabinet but never got hold of the cereal box. Gemini 3.8 Flash closed its gripper at the bowl at 2 s and came away empty. GLM 5.3 Flash never got hold of a door handle, and DeepSeek V4.1 Flash opened a door but never reached the cereal box. Qwen3.8 Max never opened the cabinet.

Disposing of batteries (Section 3.3, Figure 32). In this BEHAVIOR-1K task a mobile two-armed robot must put three batteries into the wastebasket of a copy room, one from a cabinet in the copy room and two from a desk in an office along the corridor, and leave the wastebasket on the floor. The task graph has four stages, one for each battery and one for the wastebasket on the floor, which holds from the start. In the Standard Setting, the three strongest agents each put the cabinet battery into the wastebasket and then failed at the second battery, when the robot tipped over. GPT-6 Astra dropped the first battery into the wastebasket at 90 s. At 359 s, as it drove into the office, the base caught the door jamb and the robot tipped onto its side. It righted the robot by 542 s and picked up a second battery at 592 s, and it still held the battery when the episode ended. Claude Opus 5.5 dropped the first battery in at 102 s, and the robot tipped over at 486 s as it turned at the office door. Gemini 4 Argon dropped the first battery in at 23 s and tipped the robot over at 90 s while turning in the office. In both of these runs the simulator

service ended the episode because the robot had fallen. Four of the five weaker agents failed at the first battery. Gemini 3.8 Flash never lifted it from the cabinet, DeepSeek V4.1 Flash tipped the robot over while searching for it, and GLM 5.3 Flash lost the robot’s balance while searching. Qwen3.8 Max’s robot also fell. Kimi 3.1 Pro put the cabinet battery into the wastebasket and then searched the office without finding another battery before the episode ended.

Hanging a mug (Section 3.2, Figure 33a). In this RoboTwin 2.0 task a two-armed robot must pick up a mug with the left arm, turn it and put it down in the middle of the table, and then hang it on a rack with the right arm. In the Privileged Setting, GPT-6 Astra put the mug down at 9 s and hung it at 23 s. In the Standard Setting it put the mug down at 4 s and carried it to the rack with the right arm. At 9 s, having reported that the peg was through the handle, it opened the gripper, and the mug slipped off as the gripper withdrew and fell to the table. Two more tries also failed, and the episode ended with the mug not hung. Claude Opus 5.5 passed both the Privileged and the Standard test.

Heading a ball ten times (Section 3.2, Figure 33b). In this Embodied Olympics - Fun task a humanoid robot must head a ball from a ball machine ten times in a row, and each header is a stage. In the Privileged Setting, GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon and Gemini 3.8 Flash all made the ten headers. In the Standard Setting the agent sees the ball only in camera images, the task grants 5 resets, and only the last episode counts. There, the graded episodes of the same four agents reached five, four, two and one headers. Claude Opus 5.5 first watched the lob twice while standing still. It then tracked the ball in the venue camera, fitted its flight, and timed a crouch and a push up of the legs to the ball’s arrival. From its third episode on, this approach made the first header every time, and it spent the remaining resets aiming the headers with the waist and ankles. In its graded episode the fifth ball fell past the head and hit the ground at 6.2 s. GPT-6 Astra made its first header in its fourth episode, and its graded episode had reached five headers when the agent’s time ran out. Gemini 4 Argon’s third touch, at 3.6 s, sent the ball sideways. Gemini 3.8 Flash made one header, and in its graded episode the second ball passed the head and hit the ground at 3.1 s.

Real-Robot Demonstration

Because the Standard Setting gives the agent only the robot’s sensor readings and its control interface (§ 2.1), an interface of the same kind can connect a coding agent to a physical robot. As a first test, we connected two coding agents to a physical robot arm, and each agent played a buzz wire game for eight trials in a row (Figure 34). This demonstration is not part of the exam, and its results are not included in any success rate in this paper.

Figure 34. Buzz wire game on a physical robot arm. (a) Four frames in time order of one trial of GPT-6 Astra and one of Claude Opus 5.5, with a close-up of the light and the buzzer and the command being executed. In the last frame of Claude Opus 5.5’s trial, the loop has touched the wire and the light is on. (b) The plot gives the progress score of each agent in each of its eight trials.

Task. In a buzz wire game, a player guides a metal loop along a bent wire from one end to the other without touching the wire. The loop and the wire belong to one electric circuit, so a contact closes the circuit and turns on a light and a buzzer. In our setup, both ends of the bent wire are fixed to a wooden board, and the robot holds the handle of the loop in its gripper (Figure 34a). A contact counts as a failure, as in the buzz wire tasks of RoboARC (Section A.1), in which a simulated Franka Panda arm guides a metal loop along a wire.

Commands and video. The agent controls the robot with discrete commands. Each command moves the gripper up, down, left or right or rotates it in either direction. In each panel of

Figure 34a, the pad at the lower left highlights the command being executed, and the inset shows a close-up of the light and the buzzer, so a contact is visible in the video.

Limitations

Agents and trials. The two agents use GPT-6 Astra and Claude Opus 5.5, the two models with the highest Standard success rates in Figure 8. Each agent ran one sequence of eight trials on the same wire.

Progress score. Each trial receives a progress score, the percentage of the wire that the loop covers before the trial ends. A score of 100% means that the loop passed along the whole wire without a contact.

Results. Both agents score higher in the last four trials than in the first four, and GPT-6 Astra scores higher than Claude Opus 5.5 in every trial (Figure 34b). Averaged over the first four trials, the progress is 65.0% for GPT-6 Astra and 35.0% for Claude Opus 5.5. Averaged over the last four trials, the progress is 93.8% and 62.5%. GPT-6 Astra scores at least 80% in five of its eight trials, and Claude Opus 5.5 does so in two. GPT-6 Astra starts at 60% and passes along the whole wire in trials 7 and 8. Its progress falls once, from 80% in trial 3 to 60% in trial 4. Claude Opus 5.5 scores 0% in trial 1 and between 40% and 80% in trials 2 to 8, and it does not pass along the whole wire in any trial. Its progress varies more from trial to trial. It rises to 80% in trial 6, falls to 40% in trial 7 and returns to 80% in trial 8. In the trial shown in Figure 34a, Claude Opus 5.5 touches the wire, and the light in its close-up turns on. The two agents rank in the same order as in the Standard Setting in simulation, where GPT-6 Astra solves 33.6% of the Standard tests and Claude Opus 5.5 solves 25.0% (Figure 8).

Scope. The demonstration covers one task, one robot and two agents, with one sequence of eight trials per agent. It shows that a coding agent can control a physical robot through an interface of this kind, and that the progress of both agents rose over the eight trials. With one sequence per agent, the demonstration cannot separate this rise from variation between runs. It also does not measure how well success rates in simulation predict results on hardware (Section J). A systematic evaluation on physical robots, with a grading program and several runs per agent, remains future work.

Limitations

Each result holds for one model in one harness. Each agent also makes a single attempt per test, so the results cannot separate an agent’s ability from the variation between runs. The core set excludes tasks that we judged easy and tasks that GPT-6 Luna solved in a screening run, so its success rates are lower than those of the full exam by an amount that differs between suites. Because GPT-6 Luna belongs to the same family as GPT-6 Astra, the screening may also bias the core set against tasks that this family solves. Many per-suite rates are based on few tests. In a suite with five or six tests, a single test changes the rate by 17 to 20 percentage points, and a single Realtime test changes an agent’s success rate over all Realtime tests by about two points. The stage-level and control-approach labels come from agents. The model that wrote the stage-level labels, GPT-6.1 Sol, belongs to the same family as GPT-6 Astra. This shared family could bias the label-based comparisons between agents. Of the 1,156 tasks, 392 have no reference solution, so we have not shown that each of them can be solved in every setting. Time limits differ between suites, the tasks of Embodied Olympics - Fun run with open network access, and results on tasks adapted from public benchmarks may partly reflect memorization of published solutions (Section B). Finally, all results outside § 8.2 are in simulation. The demonstration in § 8.2 covers one task on a physical robot, so how well the simulation results predict results on hardware remains open.

Extended Related Work

This appendix extends the review in § 4. We first review benchmarks for coding agents and benchmarks for robot learning, and then work that uses language models for robots. We then review recent work that runs general-purpose agents on robots, together with the harnesses and benchmarks built for these agents.

Benchmarks for coding agents. Language models and the agents built on them are evaluated mostly on digital tasks. These tasks include writing programs from a problem description [15, 58], resolving issues in real repositories [64, 104, 175], completing long software tasks and tasks in a terminal [73, 102], engineering algorithms for hard optimization problems [57], machine learning engineering and research [14, 143, 164], writing efficient GPU kernels [117], cybersecurity [189], and operating websites, desktop applications and other tools [90, 103, 169, 180, 198]. Most of these agents interleave reasoning with tool calls [179] and revise their work from feedback [137]. SWE-agent [174] shows that the interface between the model and the computer affects how often an agent succeeds, and OpenHands [160] provides an open platform for building and evaluating such agents. Other work measures how often agents exploit the tests that grade them [196] and proposes practices for building rigorous agentic benchmarks [203]. The Embodied First Exam keeps each coding agent’s harness unchanged and extends these evaluations to robotics tasks. In its Standard and Realtime settings, the agent must interpret sensor observations and often cannot undo an action, and in the Realtime Setting the environment keeps running while the agent reasons. In every setting, a program that the agent cannot change grades each attempt.

Benchmarks for robot learning. Most robotics benchmarks focus on visuomotor policy learning and evaluate learned policies, from imitation-learning policies [22, 195] to generalist policies and vision-language-action models [10, 11, 12, 36, 67, 68, 110, 113, 120, 121, 122] trained on large robot datasets [66, 114]. Recent world action models build policies on pretrained video generation models such as Cosmos [109] and predict future observations together with actions [69, 94, 182], extending earlier work that extracts actions from generated videos [26] or learns a joint representation of video and actions [81]. These benchmarks are built on physics simulators, control suites and robot learning frameworks [39, 41, 98, 105, 111, 123, 144, 146, 147, 149, 202]. They cover tabletop and bimanual manipulation [8, 16, 17, 59, 63, 87, 99, 100, 110, 124, 172, 184, 190, 193], household activities [71, 76, 106, 107, 138, 142], humanoid and legged locomotion and whole-body control [3, 132, 187, 194], dexterous and tool manipulation [20, 65, 155], musculoskeletal control [13], physical reasoning and creative problem solving [55, 85], aerial manipulation [162], laboratory automation [74, 79] and safety [119]. Policies trained on real data are also evaluated in simulated counterparts of real setups [82], on real robots at scale [9, 201] and in learned world models [38, 42, 126], which can also generate training data for policies [60, 177]. Most of these benchmarks do not need to restrict what the evaluated system can access, because a learned policy receives only a fixed set of observations. A coding agent can run any program on its computer and could read the simulator’s files, so the exam adds rules about what the agent may access (§ 2.1). The exam adapts 24 of these benchmarks and two others, EmbodiedSWE [134] and ABC Sim [4], and keeps their scenes and success rules, so that coding agents can attempt their tasks under the exam’s rules (Section A.2). Its Standard Setting gives the agent observations of the kind that a learned policy receives, so coding agents and learned policies can be compared with the same observations in future work.

Language models for robot planning and control. Language models have been used to plan tasks for robots, write robot programs and design training signals. SayCan [1], Inner Monologue [52], ProgPrompt [141], Text2Motion [86] and SayPlan [127] sequence learned or scripted skills into plans, PaLM-E [25] and Gemini Robotics-ER [36, 37] train multimodal models for embodied reasoning, and embodied chain-of-thought reasoning [188] trains a vision-language-action model to reason about plans, sub-tasks, motions and object locations before it acts. AutoRT [2] uses foundation models to propose and assign tasks to a fleet of robots, and SOAR [200] uses them to propose tasks and practice them autonomously. Code as Policies [84], ChatGPT for Robotics [151], Instruct2Act [51] and RoboTool [171] turn instructions into programs that call perception and control primitives, and Code-as-Monitor [197] applies the same idea to failure detection, with a model that writes code to check spatio-temporal constraints during execution. VoxPoser [53], ReKep [54] and MOKA [29] have a model specify value maps, keypoint constraints or keypoints selected from candidates marked on the image, which a planner or an optimizer then turns into motions. Language to Rewards [185], Eureka [95], Text2Reward [168], DrEureka [96] and RDA [75] have a model write and refine reward functions, which a controller or a reinforcement learning algorithm then optimizes. RoboGen [161] and GenSim [156] generate simulated tasks and training data, and Voyager [154] builds a library of code skills in an open-ended game. Most of these systems pair a model with an interface designed for it, such as a set of primitives, a planner or a training pipeline. Primitives and planners contribute the kind of classical engineering that Goldberg [40] argues robotics still needs, because robot data are far scarcer than the internet data used to train vision-language models. The Embodied First Exam builds on this line of work and evaluates unchanged general-purpose coding agents on the whole loop of reading an interface, choosing a control approach, carrying it out and revising it.

Embodied benchmarks for language models. ALFWorld [139] pairs text-based household tasks with their embodied versions and has become a common testbed for language-model agents [137, 179]. Embodied Agent Interface [78], EmbodiedBench [176] and VisualAgent-Bench [91] evaluate language models as decision makers in simulated households and other embodied environments, and MineDojo [28] provides open-ended Minecraft tasks for embodied agents. Embodied Agent Interface divides embodied decision making into goal interpretation, subgoal decomposition, action sequencing and transition modeling, and EmbodiedBench evaluates both high-level tasks and low-level navigation and manipulation. In these benchmarks the model chooses from an action space that the benchmark defines, which ranges from high-level skills such as picking up an object to discretized end-effector poses. The exam gives the agent the robot’s control interface, so the agent also decides how its actions are produced.

Agentic robots. A growing body of recent work runs general-purpose agents on robots, many with GPT-6 Astra [116] or Claude Fable [6] as the model. Recent surveys organize this work by model, data, environment and harness [125] and compare agents that use robots with agents that use computers [19]. This work differs in two respects: the control approach that the agent uses, and the harness that connects the model to the robot. This paragraph groups the work by the three control approaches of Figure 2, and the next paragraph reviews harnesses.

Direct control. In this approach, the agent observes the scene and outputs each action itself. Several studies use GPT-6 Astra as the policy in this way. They evaluate it on RoboDojo, RoboTwin 2.0 and other benchmarks [130, 191, 199], on memory-dependent manipulation tasks [61], on contact-rich tasks with tactile feedback [163], and with demonstrations and its own interaction history in context [21, 88].

Code to control. In this approach, the agent writes programs that produce the robot’s actions. Several studies give a general-purpose coding agent the robot’s programming interface and add few tools for the task. FAEA [150] runs the unmodified Claude Agent SDK on simulated manipulation benchmarks with access to the simulator state, and Agent as Policy [62] has a general-purpose agent interpret camera images, write programs and send motion commands on real robots. Claude Code (Opus 5) and Codex (GPT-5.6 Sol and GPT-6 Astra) write task and motion planning programs that generalize to unseen problem instances [101]. With Claude Fable, a coding agent writes a controller that solves Push-T without demonstrations [167] and writes force-feedback programs for contact-rich tasks [145]. In Anthropic’s Project Fetch, human teams programmed a quadruped robot with and without Claude [31], and Claude Opus 4.7, working without human help, was later about 20 times faster than the fastest human team at all the tasks that the teams had completed [56]. Other systems add procedures that help the agent write better programs. They search over multi-file policy repositories that compose perception, planning and control primitives [27], build libraries of programs from execution feedback [72, 93] or of verified skills from a few demonstrations [108], reuse body descriptions, past experience and skills derived from the agent’s own routines [45], write a shared code library that makes every decision without a model at test time [49], infer a program from a single human demonstration [92], let the agent check and retry primitives within Python cells [140], let the agent develop its tools in simulation before it controls a real robot [48], or have one agent revise, on a real robot, the tools that another agent calls [35]. RAX [50] has a coding agent explore online without hand-designed primitives, given at most a few goal images or demonstrations, and turn the successful action sequences into programs for single-arm, bimanual, dexterous, mobile and humanoid systems, both in simulation and on real robots.

Train a policy. In this approach, the agent trains a policy and then uses it to control the robot. ENPIRE [166] gives coding agents a real-world loop of resetting the scene, running a policy, verifying the outcome and refining the policy. With this loop, frontier coding agents train policies that reach a 99% success rate on dexterous tasks such as organizing a pin box. HARBOR [83] has a team of agents set up simulated environments, design rewards and tune reinforcement learning. The reward design methods above automate one step of this approach.

Most of these studies fix the control approach in advance, and Agent as Policy and ENPIRE let the agent choose between two of the approaches. The exam supports all three approaches and leaves the choice to the agent, so it also measures which approach agents choose (§ 3.7).

Harnesses for agentic robots. The harness is the program between the model and the robot. Harnesses for robots differ in the action interface they provide, in whether the agent can call learned policies, and in the support they add for tracking state and checking outcomes.

Action interfaces. The success rates of coding agents depend on the level of abstraction of the interface [32]. Show-Harness [18] gives vision-language models a small set of semantic action units, which robot-specific interpreters convert into robot actions, and VIA [47] gives off-the-shelf agents a visual interface with tools to probe pixels and set end-effector targets. HexaShell [34] shares 14 general tools for motion, perception and state across all tasks and robots, because code refined for one benchmark tends to overfit it. OpenRUA [23] gives an unmodified coding agent only terminal access to the robot’s ROS 2 interface, its documentation and basic tools. The uni-robot-agent harness [186] supports further training of open-weight models.

Learned policies as tools. Hi Robot [136] trains a high-level vision-language model that sends language commands to a vision-language-action policy. RoboClaw [80] and Harness VLA [192] let the agent call vision-language-action policies as tools, Recursive Harness Distillation [70] distills a strong agent’s interventions in a vision-language-action policy into a playbook for a lighter agent, and Gemini Robotics-ER 1.5 orchestrates the Gemini Robotics 1.5 vision-language-action model [37]. The Physical Agency orchestrator [33] plans subgoals, calls vision-language-action policies and parametrized skills, checks the outcomes and recovers from failures. Yao et al. [181] evaluate GPT-6 Astra on EBench both as the policy and as an agent that can also call a vision-language-action or world action model policy, and compare both with the policies alone.

State tracking and verification. Other harnesses add a persistent representation of the scene and a check of each action’s outcome [157], calibrated 3D measurements and a preview of each motion [153], sub-agents for perception, verification and memory [30], or a vision-language model that monitors a policy and corrects it after a confirmed failure [77].

The Embodied First Exam runs each harness unmodified with the models of its own provider, and through an adapter with other providers’ models (Section G), so different harnesses can be compared on the same tasks.

Benchmarks for agentic robots. Several benchmarks evaluate agents that control robots. Cap-X [32] evaluates coding agents that compose perception and control primitives at several levels of abstraction and finds that their success rates depend on the abstractions that designers provide. RobotWorld [178] evaluates agents on 84 tasks in manipulation, mobile manipulation, locomotion, driving and aerial control with executable success checks. EmbodiedSWE [134] evaluates coding agents on long-horizon dexterous tasks and turns successful agent solutions into demonstrations for training vision-language-action policies. RoboQuest [89] evaluates whether agents explore to find the information a task requires, BiGym 2.0 [194] compares learned policies with programs that coding agents develop on humanoid household tasks, and Xu et al. [170] evaluate whether coding agents avoid an obstacle they are told not to touch and propose obstacle-aware harnesses for route planning and contact. Earlier safety studies jailbreak robots that language models control into harmful actions [128], test whether models recognize undesirable situations in images and text [131], and test whether agents that plan with high-level actions reject hazardous household tasks [183]. In the exam’s Embodied Safety Exam, the agent controls the robot itself, and the verifier decides success from what physically happens in a physics simulation. Of the agentic-robot studies above, many report results on RoboDojo, several on RoboTwin 2.0 and one on KinDER [101]. The exam adapts all three suites. The Embodied First Exam covers 47 robots of nine types, which include humanoids, quadrupeds, dexterous hands, aerial robots and a wheeled car. It also separates what an agent can do with full access to the simulator from what it can do through the robot’s sensors alone.

References

References

  1. [1]Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691, 2022. URL https://arxiv.org/abs/2204.01691. CoRL 2022.
  2. [2]Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, and Zhuo Xu. AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents. arXiv preprint arXiv:2401.12963, 2024. URL https://arxiv.org/abs/2401.12963.
  3. [3]Firas Al-Hafez, Guoping Zhao, Jan Peters, and Davide Tateo. LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion. arXiv preprint arXiv:2311.02496, 2023. URL https://arxiv.org/abs/2311.02496.
  4. [4]Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, and Angjoo Kanazawa. Scalable Behavior Cloning with Open Data, Training, and Evaluation. arXiv preprint arXiv:2606.27375, 2026. URL https://arxiv.org/abs/2606.27375.
  5. [5]Anthropic. Claude Code. https://github.com/anthropics/claude-code, 2025.
  6. [6]Anthropic. Claude Fable 5 and Claude Mythos 5. https://www.anthropic.com/news/claude-fable-5-mythos-5, 2026.
  7. [7]Anthropic. Claude Opus 5.5. https://www.anthropic.com/claude-opus-5-5, 2026.
  8. [8]ARISE Initiative. YAMLab. https://github.com/ARISE-Initiative/yamlab, 2026.
  9. [9]Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, Jonathan Tremblay, Kanav Arora, et al. RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. arXiv preprint arXiv:2506.18123, 2025. URL https://arxiv.org/abs/2506.18123.
  10. [10]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liviming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164, 2024. URL https://arxiv.org/abs/2410.24164.
  11. [11]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, et al. RT-1: Robotics Transformer for Real-World Control at Scale. arXiv preprint arXiv:2212.06817, 2022. URL https://arxiv.org/abs/2212.06817. RSS 2023.
  12. [12]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818, 2023. URL https://arxiv.org/abs/2307.15818. CoRL 2023.
  13. [13]Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. MyoSuite – A contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600, 2022. URL https://arxiv.org/abs/2205.13600.
  14. [14]Jun Sh ern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv preprint arXiv:2410.07095, 2024. URL https://arxiv.org/abs/2410.07095. ICLR 2025.
  15. [15]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
  16. [16]Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. arXiv preprint arXiv:2506.18088, 2025. URL https://arxiv.org/abs/2506.18088. ICML 2026.
  17. [17]Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, and Masayoshi Tomizuka. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. arXiv preprint arXiv:2607.04434, 2026. URL https://arxiv.org/abs/2607.04434.
  18. [18]Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, and Mike Zheng Shou. Show-Harness: Just a VLM Agent Can Play Robots. arXiv preprint arXiv:2609.10522, 2026. URL https://arxiv.org/abs/2609.10522.
  19. [19]Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, et al. Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use. arXiv preprint arXiv:2610.30108, 2026. URL https://arxiv.org/abs/2610.30108.
  20. [20]Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, and Yaodong Yang. Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning. arXiv preprint arXiv:2206.08686, 2022. URL https://arxiv.org/abs/2206.08686.
  21. [21]Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu. In-Context Robot Learning with VLM Agents. arXiv preprint arXiv:2609.19138, 2026. URL https://arxiv.org/abs/2609.19138.
  22. [22]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv preprint arXiv:2303.04137, 2023. URL https://arxiv.org/abs/2303.04137. RSS 2023.
  23. [23]Zhaoyang Chu, Earl T. Barr, Claire Le Goues, Peter O’Hearn, Mark Harman, Federica Sarro, and He Ye. OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies. arXiv preprint arXiv:2610.02459, 2026. URL https://arxiv.org/abs/2610.02459.
  24. [24]David B. D’Ambrosio, Saminda Abeyruwan, Laura Graesser, Atil Iscen, Heni Ben Amor, Alex Bewley, Barney J. Reed, Krista Reymann, Leila Takayama, Yuval Tassa, et al. Achieving Human Level Competitive Robot Table Tennis. arXiv preprint arXiv:2408.03906, 2024. URL https://arxiv.org/abs/2408.03906.
  25. [25]Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An Embodied Multimodal Language Model. arXiv preprint arXiv:2303.03378, 2023. URL https://arxiv.org/abs/2303.03378. ICML 2023.
  26. [26]Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. arXiv preprint arXiv:2302.00111, 2023. URL https://arxiv.org/abs/2302.00111. NeurIPS 2023.
  27. [27]Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, and Matei Zaharia. RHO: Your Coding Agent is Secretly a Roboticist. arXiv preprint arXiv:2606.16458, 2026. URL https://arxiv.org/abs/2606.16458.
  28. [28]Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. arXiv preprint arXiv:2206.08853, 2022. URL https://arxiv.org/abs/2206.08853. NeurIPS 2022 Datasets and Benchmarks Track.
  29. [29]Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. arXiv preprint arXiv:2403.03174, 2024. URL https://arxiv.org/abs/2403.03174. RSS 2024.
  30. [30]Zhirui Fang, Qingchhi Yu, Ziyang Chen, Longfei Li, Haoran Ma, Keru Zhou, Xinrun Xu, Samith Va, Yuxuan Hu, Peixuan Song, Qiang Du, Bin Qian, Yongkang Deng, Xin Li, Yezhen Wang, Zhe Li, Hao Luo, Shuyan Li, Ziwei Wang, Weijian Deng, and Xiu Li. EMERGE-Policy: A Robot Mind Emerges Beyond a Single Policy. arXiv preprint arXiv:2608.29896, 2026. URL https://arxiv.org/abs/2608.29896.
  31. [31]Frontier Red Team. Project Fetch: Can Claude train a robot dog? https://www.anthropic.com/research/project-fetch-robot-dog, 2025.
  32. [32]Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi “Jim” Fan. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of PMLR, 2026. URL https://arxiv.org/abs/2603.22435.
  33. [33]Liane Galanti, Dhruv Shah, and Tri Dao. Addressing the Orchestration Gap in Generalist Robots via Physical Agency. arXiv preprint arXiv:2607.21725, 2026. URL https://arxiv.org/abs/2607.21725.
  34. [34]Hongcheng Gao, Jianshu Zeng, Alex Zhou, Shijia Ge, Xuan Shangguan, Yazhe Wang, Yexing Wan, Zelin Zheng, Jay Zhu, Xiping Li, Di Wu, Yongli Wang, et al. HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World. arXiv preprint arXiv:2610.23479, 2026. URL https://arxiv.org/abs/2610.23479.
  35. [35]Hongcheng Gao, Jingjing Zhou, Zelin Zheng, Shijia Ge, Jay Zhu, Yazhe Wang, Jianshu Zeng, Xuan Shangguan, Di Wu, Lingyu He, Zhiqi Jia, Sihang Wu, and Xiao He. Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence. arXiv preprint arXiv:2609.35432, 2026. URL https://arxiv.org/abs/2609.35432.
  36. [36]Gemini Robotics Team. Gemini Robotics: Bringing AI into the Physical World. arXiv preprint arXiv:2503.20020, 2025. URL https://arxiv.org/abs/2503.20020.
  37. [37]Gemini Robotics Team. Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer. arXiv preprint arXiv:2510.03342, 2025. URL https://arxiv.org/abs/2510.03342.
  38. [38]Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, et al. Evaluating Gemini Robotics Policies in a V eo World Simulator. arXiv preprint arXiv:2512.10675, 2025. URL https://arxiv.org/abs/2512.10675.
  39. [39]Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peih ao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, et al. RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning. arXiv preprint arXiv:2504.18904, 2025. URL https://arxiv.org/abs/2504.18904.
  40. [40]Ken Goldberg. Good old-fashioned engineering can close the 100,000-year “data gap” in robotics. Science Robotics, 10(105):eaea7390, 2025. doi: 10.1126/scirobotics.aea7390.DOI
  41. [41]Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. arXiv preprint arXiv:2302.04659, 2023. URL https://arxiv.org/abs/2302.04659. ICLR 2023.
  42. [42]Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-World: A Controllable Generative World Model for Robot Manipulation. arXiv preprint arXiv:2510.10125, 2025. URL https://arxiv.org/abs/2510.10125. ICLR 2026.
  43. [43]Tuomas Haarn oja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyasuvunakool, Noah Y. Siegel, Roland Hafner, et al. Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning. arXiv preprint arXiv:2304.13653, 2023. URL https://arxiv.org/abs/2304.13653.
  44. [44]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. https://doi.org/10.5281/zenodo.20953922, 2026.
  45. [45]Sida He, Lingxi Xie, Yunning Cao, Pengfei Chen, Kaiwen Duan, Jiannan Ge, Xinyue Huo, Jiacheng Shao, and Qi Tian. Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer. arXiv preprint arXiv:2609.31770, 2026. URL https://arxiv.org/abs/2609.31770.
  46. [46]Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbabu, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi Fan, Yuke Zhu, Changliu Liu, and Guanya Shi. ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills. In Robotics: Science and Systems (RSS), 2025. URL https://arxiv.org/abs/2502.01143.
  47. [47]Hengyuan Hu, Jensen Gao, Priya Sundaresan, Satvik Sharma, Jeannette Bohg, and Dorsa Sadigh. VIA: Visual Interface Agent for Robot Control. arXiv preprint arXiv:2607.11119, 2026. URL https://arxiv.org/abs/2607.11119.
  48. [48]Jiaheng Hu, Roberto Martin-Martin, Peter Stone, Rocky Duan, Zhenyu Jiang, and Guanya Shi. SimEX: Simulation-Integrated Robotics AutoResearch. arXiv preprint arXiv:2609.38982, 2026. URL https://arxiv.org/abs/2609.38982.
  49. [49]Kairui Hu, Siyuan Hu, Fangzhou Hong, Zhaoxi Chen, and Ziwei Liu. Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement. arXiv preprint arXiv:2610.12369, 2026. URL https://arxiv.org/abs/2610.12369.
  50. [50]Runhan Huang, Xiangcheng Zhang, Haoyu Chen, Yitong Li, Yen-Jen Wang, Bohan Lyu, Haotian Ye, Chuye Hong, Yixuan Wang, Chloe Su, Haoyu Xiong, Yangzhen Wu, et al. Scaling Online Exploration for Robot Agents. arXiv preprint arXiv:2610.22496, 2026. URL https://arxiv.org/abs/2610.22496.
  51. [51]Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model. arXiv preprint arXiv:2305.11176, 2023. URL https://arxiv.org/abs/2305.11176.
  52. [52]Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. arXiv preprint arXiv:2207.05608, 2022. URL https://arxiv.org/abs/2207.05608. CoRL 2022.
  53. [53]Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2307.05973
  54. [54]Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. arXiv preprint arXiv:2409.01652, 2024. URL https://arxiv.org/abs/2409.01652. CoRL 2024.
  55. [55]Yixuan Huang, Bowen Li, Vaibhav Saxena, Yichao Liang, Utkarsh A. Mishra, Liang Ji, Lihan Zha, Jimmy Wu, Nishanth Kumar, Sebastian Scherer, Danfei Xu, and Tom Silver. KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning. arXiv preprint arXiv:2604.25788, 2026. URL https://arxiv.org/abs/2604.25788. RSS 2026.
  56. [56]Michael Ilie, C. Daniel Freeman, and Kevin K. Troy. Project Fetch: Phase two. https://www.anthropic.com/research/project-fetch-phase-two, 2026.
  57. [57]Yuki Imaju ku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering. arXiv preprint arXiv:2506.09050, 2025. URL https://arxiv.org/abs/2506.09050. NeurIPS 2025 Datasets and Benchmarks Track.
  58. [58]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974, 2024. URL https://arxiv.org/abs/2403.07974. ICLR 2025.
  59. [59]Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The Robot Learning Benchmark & Learning Environment. arXiv preprint arXiv:1909.12271, 2019. URL https://arxiv.org/abs/1909.12271. IEEE RA-L 2020.
  60. [60]Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, et al. DreamGen: Unlocking Generalization in Robot Learning through Video World Models. arXiv preprint arXiv:2505.12705, 2025. URL https://arxiv.org/abs/2505.12705.
  61. [61]Jack B. Jedlicki, Amish Sethi, and Heng Yang. Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation? arXiv preprint arXiv:2610.17350, 2026. URL https://arxiv.org/abs/2610.17350.
  62. [62]Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, and Meng Jiang. Agent as Policy for Robotic Manipulation. arXiv preprint arXiv:2609.12541, 2026. URL https://arxiv.org/abs/2609.12541.
  63. [63]Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: Robot Manipulation with Multimodal Prompts. arXiv preprint arXiv:2210.03094, 2022. URL https://arxiv.org/abs/2210.03094. ICML 2023.
  64. [64]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv preprint arXiv:2310.06770, 2023. URL https://arxiv.org/abs/2310.06770.
  65. [65]Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg, and C. Karen Liu. SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation. arXiv preprint arXiv:2602.16863, 2026. URL https://arxiv.org/abs/2602.16863.
  66. [66]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, et al. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv preprint arXiv:2403.12945, 2024. URL https://arxiv.org/abs/2403.12945. RSS 2024.
  67. [67]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint arXiv:2406.09246, 2024. URL https://arxiv.org/abs/2406.09246. CoRL 2024.
  68. [68]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. arXiv preprint arXiv:2502.19645, 2025. URL https://arxiv.org/abs/2502.19645. RSS 2025.
  69. [69]Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. arXiv preprint arXiv:2601.16163, 2026. URL https://arxiv.org/abs/2601.16163.
  70. [70]Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, and Nojun Kwak. Recursive Harness Distillation across Agents for Robot Manipulation. arXiv preprint arXiv:2609.33378, 2026. URL https://arxiv.org/abs/2609.33378.
  71. [71]Yejin Kim, Wilbert Pumacay, Omar Rayyan, Max Argus, Winson Han, Eli VanderBilt, Jordi Salvador, Abhay Deshpande, Rose Hendrix, Snehal Jauhri, Shuo Liu, Nur Muhammad Mahi Shafiullah, Maya Guru, Ainaz Eftekhar, Karen Farley, Donovan Clay, Jiafei Duan, Arjun Guru, Piper Wolters, Alvaro Herrasti, Ying-Chun Lee, Georgia Chlvatzaki, Yuchen Cui, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation. arXiv preprint arXiv:2602.11337, 2026. URL https://arxiv.org/abs/2602.11337.
  72. [72]So Kuroki, Kento Kawaharazuka, and Yujin Tang. Learning and Transferring Closed-Loop Robot Software. arXiv preprint arXiv:2609.19906, 2026. URL https://arxiv.org/abs/2609.19906.
  73. [73]Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neve Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring AI Ability to Complete Long Software Tasks. arXiv preprint arXiv:2503.14499, 2025. URL https://arxiv.org/abs/2503.14499.
  74. [74]Zhiqian Lan, Yuxuan Jiang, Ruiqi Wang, Xuanbing Xie, Rongkui Zhang, Yicheng Zhu, Peihang Li, Tianshuo Yang, Tianxing Chen, Haoyu Gao, Xiaokang Yang, Xuelong Li, Hongyuan Zhang, Yao Mu, and Ping Luo. AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory. arXiv preprint arXiv:2505.14030, 2025. URL https://arxiv.org/abs/2505.14030.
  75. [75]Hojoon Lee, Ajay Subramanian, Ben AbbateMatteo, Vijay Veerabadran, Pedro Matias, Karl Ridgeway, and Nitin Kamra. Reward Design Agent for Reinforcement Learning. arXiv preprint arXiv:2606.01672, 2026. URL https://arxiv.org/abs/2606.01672. RLC 2026.
  76. [76]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabriel Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation. arXiv preprint arXiv:2403.09227, 2024. URL https://arxiv.org/abs/2403.09227.
  77. [77]Long Li, Qichao Zhao, Yue Yang, Fan Xu, Zhe Wang, Alan Wee-Chung Liew, Chao Qiu, Heng Tao Shen, and Shirui Pan. Spotter: Let the Embodied Model Lead, and the VLM Reflect for It. arXiv preprint arXiv:2609.36808, 2026. URL https://arxiv.org/abs/2609.36808.
  78. [78]Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. arXiv preprint arXiv:2410.07166, 2024. URL https://arxiv.org/abs/2410.07166. NeurIPS 2024 Datasets and Benchmarks Track.
  79. [79]Rui Li, Zixuan Hu, Wenxi Qu, Jinouwen Zhang, Zhenfei Yin, Sha Zhang, Xuantuo Huang, Hanqing Wang, Tai Wang, Jiangmiao Pang, Wanli Ouyang, Lei Bai, Wangmeng Zuo, Ling-Yu Duan, Dongzhan Zhou, and Shixiang Tang. LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents. arXiv preprint arXiv:2505.22634, 2025. URL https://arxiv.org/abs/2505.22634. NeurIPS 2025 Datasets and Benchmarks Track.
  80. [80]Ruiviying Li, Yunlang Zhou, YuYao Zhu, Kylin Chen, Jingyuan Wang, Sukai Wang, Kongtao Hu, Minhui Yu, Bowen Jiang, Zhan Su, Jiayao Ma, Xin He, Yongjian Shen, Yang Yang, Guanghui Ren, Maoqing Yao, Wenhao Wang, and Yao Mu. RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks. arXiv preprint arXiv:2603.11558, 2026. URL https://arxiv.org/abs/2603.11558.
  81. [81]Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified Video Action Model. arXiv preprint arXiv:2503.00200, 2025. URL https://arxiv.org/abs/2503.00200. RSS 2025.
  82. [82]Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating Real-World Robot Manipulation Policies in Simulation. arXiv preprint arXiv:2405.05941, 2024. URL https://arxiv.org/abs/2405.05941. CoRL 2024.
  83. [83]Zechu Li, Yufeng Jin, Xiaoyang Liu, Puze Liu, Vignesh Prasad, Carlo D’Eramo, and Georgia Chalvatzaki. HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning. arXiv preprint arXiv:2606.08610, 2026. URL https://arxiv.org/abs/2606.08610.
  84. [84]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. arXiv preprint arXiv:2209.07753, 2022. URL https://arxiv.org/abs/2209.07753. ICRA 2023.
  85. [85]Chunru Lin, Hongxin Zhang, Fenghao Yu, Zehuan Chen, Thomas L. Griffiths, Yejin Choi, David Held, and Chuang Gan. RoboWits: Unexpected Challenges for Robotic Creative Problem Solving. arXiv preprint arXiv:2605.30326, 2026. URL https://arxiv.org/abs/2605.30326.
  86. [86]Kevin Lin, Christopher Agia, Toki Migitmatsu, Marco Pavone, and Jeannette Bohg. Text2Motion: From Natural Language Instructions to Feasible Plans. arXiv preprint arXiv:2303.12153, 2023. URL https://arxiv.org/abs/2303.12153. Autonomous Robots 2023.
  87. [87]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. arXiv preprint arXiv:2306.03310, 2023. URL https://arxiv.org/abs/2306.03310. NeurIPS 2023 Datasets and Benchmarks Track.
  88. [88]Fangcheng Liu, Yeqing Shen, Anda Cheng, Weishi Mi, Chao Tang, Chenyuan Liu, Yushun Xiang, Tingguang Li, Yong-Lu Li, and Yehui Tang. RoboLCL: Embodied In-Context Learning with GPT-6 Astra. arXiv preprint arXiv:2609.34261, 2026. URL https://arxiv.org/abs/2609.34261.
  89. [89]Renhang Liu, Navonil Majumder, Tej Deep Pala, and Soujanya Poria. RoboQuest: Generalist Physical Agents that Search, Inspect and Test. arXiv preprint arXiv:2610.10388, 2026. URL https://arxiv.org/abs/2610.10388.
  90. [90]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyui Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents. arXiv preprint arXiv:2308.03688, 2023. URL https://arxiv.org/abs/2308.03688. ICLR 2024.
  91. [91]Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyu Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents. arXiv preprint arXiv:2408.06327, 2024. URL https://arxiv.org/abs/2408.06327.
  92. [92]Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. RAPID: Robot Agentic Programming from Demonstrations. arXiv preprint arXiv:2609.30249, 2026. URL https://arxiv.org/abs/2609.30249.
  93. [93]Runyu Lu, Yubo Wu, Ethan Kou, Haotian Lin, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Jan Kautz, Yuke Zhu, Linxi “Jim” Fan, and Guanzhi Wang. ASPIRE: Agentic / Skills Discovery for Robotics. arXiv preprint arXiv:2607.00272, 2026. URL https://arxiv.org/abs/2607.00272.
  94. [94]Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, et al. World-Action Models for Robot Learning and Control: A Survey. arXiv preprint arXiv:2609.16074, 2026. URL https://arxiv.org/abs/2609.16074.
  95. [95]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi “Jim” Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.12931
  96. [96]Yecheng Jason Ma, William Liang, Hung-Ju Wang, Yuke Zhu, Linxi “Jim” Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: Language Model Guided Sim-To-Real Transfer. arXiv preprint arXiv:2406.01967, 2024. URL https://arxiv.org/abs/2406.01967. RSS 2024.
  97. [97]Yuntao Ma, Andrei Cramariuc, Farbod Farshidian, and Marco Hutter. Learning coordinated badminton skills for legged manipulators. Science Robotics, 10(102):eadu3922, 2025. URL https://arxiv.org/abs/2505.22974.
  98. [98]Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. arXiv preprint arXiv:2108.10470, 2021. URL https://arxiv.org/abs/2108.10470. NeurIPS 2021 Datasets and Benchmarks Track.
  99. [99]Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K.R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Meta-World+: An Improved, Standardized, RL Benchmark. arXiv preprint arXiv:2505.11289, 2025. URL https://arxiv.org/abs/2505.11289. NeurIPS 2025 Datasets and Benchmarks Track.
  100. [100]Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. arXiv preprint arXiv:2112.03227, 2021. URL https://arxiv.org/abs/2112.03227. IEEE RA-L 2022.
  101. [101]Matteo Merler, Bowen Li, Josh Roy, Yichao Liang, Qianwei Wang, Yixuan Huang, and Tom Silver. Coding Agents for Generalized Task and Motion Planning Problems. arXiv preprint arXiv:2609.30233, 2026. URL https://arxiv.org/abs/2609.30233.
  102. [102]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guangchao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastro michalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpoutakhoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Darius h Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv preprint arXiv:2601.11868, 2026. URL https://arxiv.org/abs/2601.11868. ICLR 2026.
  103. [103]Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants. arXiv preprint arXiv:2311.12983, 2023. URL https://arxiv.org/abs/2311.12983. ICLR 2024.
  104. [104]Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? arXiv preprint arXiv:2502.12115, 2025. URL https://arxiv.org/abs/2502.12115.
  105. [105]Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. arXiv preprint arXiv:2301.04195, 2023. URL https://arxiv.org/abs/2301.04195. IEEE RA-L 2023.
  106. [106]Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. arXiv preprint arXiv:2406.02523, 2024. URL https://arxiv.org/abs/2406.02523. RSS 2024.
  107. [107]Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. arXiv preprint arXiv:2603.04356, 2026. URL https://arxiv.org/abs/2603.04356. ICLR 2026.
  108. [108]Chang Nie, Zhe Liu, and Hesheng Wang. Teach and Grow: An Agent-Centered Architecture for General Robot Learning. arXiv preprint arXiv:2608.17209, 2026. URL https://arxiv.org/abs/2608.17209.
  109. [109]NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, et al. Cosmos World Foundation Model Platform for Physical AI. arXiv preprint arXiv:2501.03575, 2025. URL https://arxiv.org/abs/2501.03575.
  110. [110]NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi “Jim” Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv preprint arXiv:2503.14734, 2025. URL https://arxiv.org/abs/2503.14734.
  111. [111]NVIDIA, Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzyniak, et al. Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. arXiv preprint arXiv:2511.04831, 2025. URL https://arxiv.org/abs/2511.04831.
  112. [112]Michael O’Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds. Science Robotics, 7(66):eabm6597, 2022. URL https://arxiv.org/abs/2205.06908.
  113. [113]Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An Open-Source Generalist Robot Policy. arXiv preprint arXiv:2405.12213, 2024. URL https://arxiv.org/abs/2405.12213. RSS 2024.
  114. [114]Open X-Embodiment Collaboration et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv preprint arXiv:2310.08864, 2023. URL https://arxiv.org/abs/2310.08864. ICRA 2024.
  115. [115]OpenAI. Codex CLI. https://github.com/openai/codex, 2025.
  116. [116]OpenAI. GPT-6 Astra: A new generation of intelligence. https://openai.com/index/gpt-6-astra/, 2026.
  117. [117]Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs Write Efficient GPU Kernels? arXiv preprint arXiv:2502.10517, 2025. URL https://arxiv.org/abs/2502.10517. ICML 2025.
  118. [118]Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv preprint arXiv:1710.06537, 2017. URL https://arxiv.org/abs/1710.06537.
  119. [119]Yiyan Peng, Philip Wang, Simon Sinong Zhan, Yiqi Lyu, Zhengyang Ni, Jixin Yan, Fiorelli Wong, Ruocheng Jiao, Hang Yin, Xinyu Cao, Huajie Shao, Manling Li, Ruohan Zhang, and Qi Zhu. MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation. arXiv preprint arXiv:2608.17386, 2026. URL https://arxiv.org/abs/2608.17386.
  120. [120]Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, et al. FAST: Efficient Action Tokenization for Vision-Language-Action Models. arXiv preprint arXiv:2501.09747, 2025. URL https://arxiv.org/abs/2501.09747.
  121. [121]Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, et al. π₀.₆: a VLA That Learns From Experience. arXiv preprint arXiv:2511.14759, 2025. URL https://arxiv.org/abs/2511.14759.
  122. [122]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054, 2025. URL https://arxiv.org/abs/2504.16054.
  123. [123]Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimir Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandar Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. arXiv preprint arXiv:2310.13724, 2023. URL https://arxiv.org/abs/2310.13724. ICLR 2024.
  124. [124]Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation. arXiv preprint arXiv:2402.08191, 2024. URL https://arxiv.org/abs/2402.08191. RSS 2024.
  125. [125]Chonghan Qin, Dongming Wu, Kaituo Feng, Jiaxu Wang, Xinlei Yu, Peiwen Sun, Yijia Fan, Xilin He, Bingao Chen, Weitong Lian, Ryan Ji, Wenqi Ren, et al. Towards Agentic Robots: Model, Data, Environment, and Harness. arXiv preprint arXiv:2610.20442, 2026. URL https://arxiv.org/abs/2610.20442.
  126. [126]Julian Quevedo, Ansh Kumar Sharma, Yixiang Sun, Varad Suryavanshi, Percy Liang, and Sherry Yang. WorldGym: World Model as An Environment for Policy Evaluation. arXiv preprint arXiv:2506.00613, 2025. URL https://arxiv.org/abs/2506.00613. ICLR 2026.
  127. [127]Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Sünderhauf. SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. arXiv preprint arXiv:2307.06135, 2023. URL https://arxiv.org/abs/2307.06135. CoRL 2023.
  128. [128]Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, and George J. Pappas. Jailbreaking LLM-Controlled Robots. arXiv preprint arXiv:2410.13691, 2024. URL https://arxiv.org/abs/2410.13691.
  129. [129]Peter Schaldenbrand, James McCann, and Jean Oh. FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environment. arXiv preprint arXiv:2210.00664, 2022. URL https://arxiv.org/abs/2210.00664.
  130. [130]SenseNova Joint Research Team, Zhongang Cai, Ziang Cao, Tailai Chen, Han Chong Chua, Linjun Dai, Hanming Deng, Hokin Deng, Haiwen Diao, Chenjing Ding, Weichen Fan, Xiao Fu, et al. Has GPT-6-Astra Achieved Embodied Intelligence? arXiv preprint arXiv:2610.22471, 2026. URL https://arxiv.org/abs/2610.22471.
  131. [131]Pierre Sermanet, Anirudha Majumdar, Alex Irpan, Dmitry Kalashnikov, and Vikas Sindhwani. Generating Robot Constitutions & Benchmarks for Semantic Safety. arXiv preprint arXiv:2503.08663, 2025. URL https://arxiv.org/abs/2503.08663. CoRL 2025.
  132. [132]Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid-Bench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation. arXiv preprint arXiv:2403.10506, 2024. URL https://arxiv.org/abs/2403.10506.
  133. [133]Qiuhong Shen, Shihua Zhang, Yue Liao, Qi Li, Zhenxiong Tan, Shizun Wang, Shuicheng Yan, and Xinchao Wang. World Action Models: A Survey. arXiv preprint arXiv:2606.20781, 2026. URL https://arxiv.org/abs/2606.20781.
  134. [134]Zeyu Shen, Haoxiang You, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Mingtong Zhang, Suning Huang, Jiankai Sun, Qianzhong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Henderson, Ian Abraham, and Canwen Xu. EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics. arXiv preprint arXiv:2609.27308, 2026. URL https://arxiv.org/abs/2609.27308.
  135. [135]Guanya Shi, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandakumar, Yisong Yue, and Soon-Jo Chung. Neural Lander: Stable Drone Landing Control Using Learned Dynamics. In IEEE International Conference on Robotics and Automation (ICRA), pages 9784–9790, 2019. URL https://arxiv.org/abs/1811.08027.
  136. [136]Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. arXiv preprint arXiv:2502.19417, 2025. URL https://arxiv.org/abs/2502.19417. ICML 2025.
  137. [137]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366, 2023. URL https://arxiv.org/abs/2303.11366. NeurIPS 2023.
  138. [138]Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. arXiv preprint arXiv:1912.01734, 2019. URL https://arxiv.org/abs/1912.01734. CVPR 2020.
  139. [139]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv preprint arXiv:2010.03768, 2020. URL https://arxiv.org/abs/2010.03768. ICLR 2021.
  140. [140]Ruiyang Si, Jianxin Bi, Shunyu Yang, Rui Ni, Wenbo Huang, Qiang Wang, Shulong Jiang, Yintianrun Zhang, Feixiang Ruan, Kewei Zhang, Duomin Wang, Xiuyu Li, Haiwen Feng, Zhen Dong, and Daquan Zhou. Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens. arXiv preprint arXiv:2610.01939, 2026. URL https://arxiv.org/abs/2610.01939.
  141. [141]Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In IEEE International Conference on Robotics and Automation (ICRA), 2023.arxiv.org/abs/2209.11302
  142. [142]Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. arXiv preprint arXiv:2108.03332, 2021. URL https://arxiv.org/abs/2108.03332. CoRL 2021.
  143. [143]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s Ability to Replicate AI Research. arXiv preprint arXiv:2504.01848, 2025. URL https://arxiv.org/abs/2504.01848.
  144. [144]Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. arXiv preprint arXiv:2106.14405, 2021. URL https://arxiv.org/abs/2106.14405. NeurIPS 2021.
  145. [145]Xiao Tan, Yixuan Wang, Zhexi Wang, and Wenzhen Yuan. FORCA: A Force-Aware Coding Agent for Contact-Rich Robot Manipulation. arXiv preprint arXiv:2610.31841, 2026. URL https://arxiv.org/abs/2610.31841.
  146. [146]Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhaio Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI. arXiv preprint arXiv:2410.00425, 2024. URL https://arxiv.org/abs/2410.00425.
  147. [147]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite. arXiv preprint arXiv:1801.00690, 2018. URL https://arxiv.org/abs/1801.00690.
  148. [148]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv preprint arXiv:1703.06907, 2017. URL https://arxiv.org/abs/1703.06907.
  149. [149]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026–5033, 2012.DOI
  150. [150]Brian Y. Tsui, Alan Y. Fang, and Tiffany J. Hwu. Demonstration-Free Robotic Control via LLM Agents. arXiv preprint arXiv:2601.20334, 2026. URL https://arxiv.org/abs/2601.20334. IROS 2026.
  151. [151]Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. ChatGPT for Robotics: Design Principles and Model Abilities. arXiv preprint arXiv:2306.17582, 2023. URL https://arxiv.org/abs/2306.17582.
  152. [152]Vainavi Viswanath, Kaushik Shivakumar, Justin Kerr, Brijen Thanan jeyan, Ellen Novoseller, Jeffrey Ich nowski, Alejandro Escontrela, Michael Laskey, Joseph E. Gonzalez, and Ken Goldberg. Autonomously Untangling Long Cables. In Robotics: Science and Systems (RSS), 2022. URL https://arxiv.org/abs/2207.07813.
  153. [153]Chengcheng Wang, Yuheng Shi, Siyu Xu, Yutong Wang, Jin Jing Zhao, Chenghao Xia, Jinqi Liao, Jiankun Peng, Ligong Bi, Yuechuan Ma, Jiaqi Liu, Wenbo Wang, Zijian Wang, Yuemin Wu, Minjing Dong, Xiu Su, Tao Huang, Jianyuan Guo, Chengbin Du, Yunke Wang, and Chang Xu. Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation. arXiv preprint arXiv:2611.01112, 2026. URL https://arxiv.org/abs/2611.01112.
  154. [154]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv preprint arXiv:2305.16291, 2023.arxiv.org/abs/2305.16291
  155. [155]Hanwen Wang, Weizhi Zhao, Xiangyu Wang, Siyuan Huang, He Lin, Boyuan Zheng, Rongtao Xu, Gang Wang, Yao Mu, He Wang, Lue Fan, Hongsheng Li, Zhaoxiang Zhang, and Tieniu Tan. DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo. arXiv preprint arXiv:2605.16257, 2026. URL https://arxiv.org/abs/2605.16257.
  156. [156]Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. GenSim: Generating Robotic Simulation Tasks via Large Language Models. arXiv preprint arXiv:2310.01361, 2023. URL https://arxiv.org/abs/2310.01361. ICLR 2024.
  157. [157]Qi Wang, Tianyi Wang, Chengyang Li, Shikun Ban, Yurun Chen, Yizhong Ge, Jason Qin, Chengtai Li, and Wentao Zhu. Towards the Harness of Embodied Agents. arXiv preprint arXiv:2608.11246, 2026. URL https://arxiv.org/abs/2608.11246.
  158. [158]Ruo cheng Wang, Yunzhi Zhang, Jiayuan Mao, Ran Zhang, Chin-Yi Cheng, and Jiajun Wu. IKEA-Manual: Seeing Shape Assembly Step by Step. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2022. URL https://arxiv.org/abs/2302.01881.
  159. [159]Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He, Feihong Liu, Chenchen Yang, Yikang Zhou, Zhaoye Fei, Jingjing Gong, Jinlan Fu, Mike Zheng Shou, Xuanjing Huang, Xipeng Qiu, and Yu-Gang Jiang. World Action Models: The Next Frontier in Embodied AI. arXiv preprint arXiv:2605.12090, 2026. URL https://arxiv.org/abs/2605.12090.
  160. [160]Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jas ikrat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Tianjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint arXiv:2407.16741, 2024. URL https://arxiv.org/abs/2407.16741. ICLR 2025.
  161. [161]Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation. arXiv preprint arXiv:2311.01455, 2023. URL https://arxiv.org/abs/2311.01455. ICML 2024.
  162. [162]Yutong Wang, Dongjae Lee, Xiaofeng Guo, Yuanzhu Zhan, Yufei Jiang, Bavin Saravanan, Muqing Cao, Jia Xie, Chenyang Mao, Sebastian Scherer, Junyi Geng, and Guanya Shi. AM-Bench: A Modular Simulation Suite and Benchmark for Aerial Manipulation Policy Learning. arXiv preprint arXiv:2609.00641, 2026. URL https://arxiv.org/abs/2609.00641.
  163. [163]Ziyong Wang, Yupei Lin, Runxuan Huang, Liang Lin, Dongyu Zhang, and Guangrun Wang. GPT-6-Astra Understands Touch and Outperform Vision–Tactile–Language–Action Models. arXiv preprint arXiv:2610.28457, 2026. URL https://arxiv.org/abs/2610.28457.
  164. [164]Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Er icheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024. URL https://arxiv.org/abs/2411.15114.
  165. [165]Wenli Xiao, Haoru Xue, Tony Tao, Dvij Kalaria, John M. Dolan, and Guanya Shi. AnyCar to Anywhere: Learning Universal Dynamics Model for Agile and Adaptive Mobility. arXiv preprint arXiv:2409.15783, 2024. URL https://arxiv.org/abs/2409.15783.
  166. [166]Wenli Xiao, Jia Xie, Tonghe Zhang, Haotian Lin, Letian “Max” Fu, Haoru Xue, Jalen Lu, Yi Yang, Cunxi Dai, Zi Wang, Jimmy Wu, Guanzhi Wang, S. Shankar Sastry, Ken Goldberg, Linxi “Jim” Fan, Yuke Zhu, and Guanya Shi. ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. arXiv preprint arXiv:2606.19980, 2026. URL https://arxiv.org/abs/2606.19980. CoRL 2026.
  167. [167]Shuangyu Xie, Kaiyuan Chen, and Ken Goldberg. Revisiting the “Push-T” Robot Manipulation Task with Agentic Robotics. arXiv preprint arXiv:2608.18227, 2026. URL https://arxiv.org/abs/2608.18227.
  168. [168]Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. arXiv preprint arXiv:2309.11489, 2023. URL https://arxiv.org/abs/2309.11489. ICLR 2024.
  169. [169]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024.arxiv.org/abs/2404.07972
  170. [170]Bingxin Xu, Yuzhang Shang, Zhen Dong, and Emilio Ferrara. Coding Agents with Harness for Safe Robot Control. arXiv preprint arXiv:2609.20822, 2026. URL https://arxiv.org/abs/2609.20822.
  171. [171]Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, and Ding Zhao. Creative Robot Tool Use with Large Language Models. arXiv preprint arXiv:2310.13065, 2023. URL https://arxiv.org/abs/2310.13065.
  172. [172]Jenia Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies. arXiv preprint arXiv:2604.09860, 2026. URL https://arxiv.org/abs/2604.09860.
  173. [173]Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. URL https://arxiv.org/abs/2412.14171.
  174. [174]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv preprint arXiv:2405.15793, 2024. URL https://arxiv.org/abs/2405.15793. NeurIPS 2024.
  175. [175]John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? arXiv preprint arXiv:2410.03859, 2024. URL https://arxiv.org/abs/2410.03859. ICLR 2025.
  176. [176]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. arXiv preprint arXiv:2502.09560, 2025. URL https://arxiv.org/abs/2502.09560. ICML 2025.
  177. [177]Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning Interactive Real-World Simulators. arXiv preprint arXiv:2310.06114, 2023. URL https://arxiv.org/abs/2310.06114. ICLR 2024.
  178. [178]Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, et al. RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments. arXiv preprint arXiv:2610.10409, 2026. URL https://arxiv.org/abs/2610.10409.
  179. [179]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629, 2022. URL https://arxiv.org/abs/2210.03629. ICLR 2023.
  180. [180]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024. URL https://arxiv.org/abs/2406.12045.
  181. [181]Yifei Yao, Yude Zou, Guanbo Jiao, Xing Gao, Ning Gao, Yukai Wang, Jiantong Chen, Zihou Zhu, Xinyu Li, Ningyu Yan, Shuai Wang, Jianming Ma, et al. Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy. arXiv preprint arXiv:2610.35832, 2026. URL https://arxiv.org/abs/2610.35832.
  182. [182]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi “Jim” Fan, and Joel Jang. World Action Models are Zero-shot Policies. arXiv preprint arXiv:2602.15922, 2026. URL https://arxiv.org/abs/2602.15922.
  183. [183]Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, et al. SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents. arXiv preprint arXiv:2412.13178, 2024. URL https://arxiv.org/abs/2412.13178.
  184. [184]Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. arXiv preprint arXiv:1910.10897, 2019. URL https://arxiv.org/abs/1910.10897. CoRL 2019.
  185. [185]Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to Rewards for Robotic Skill Synthesis. arXiv preprint arXiv:2306.08647, 2023. URL https://arxiv.org/abs/2306.08647. CoRL 2023.
  186. [186]Jiarui Yuan, Yuxin Zuo, Xiaoyu Huang, Wenbo Zhang, Kwanyoung Park, Junlin Yang, Tianxiang Zhang, Ruoqu Chen, Yutao Ouyang, Bingxiang He, Xingtaï Lv, Hongjin Wu, et al. Democratizing Robot-Use Agents: A Unified Harness with Open Models. arXiv preprint arXiv:2611.01560, 2026. URL https://arxiv.org/abs/2611.01560.
  187. [187]Kevin Zakka, Baruch Tabannpour, Qiyuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. K. Kahrs, Carmelo Sferrazza, Yuval Tassa, and Pieter Abbeel. MuJoCo Playground. arXiv preprint arXiv:2502.08844, 2025. URL https://arxiv.org/abs/2502.08844.
  188. [188]Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic Control via Embodied Chain-of-Thought Reasoning. arXiv preprint arXiv:2407.08693, 2024. URL https://arxiv.org/abs/2407.08693.
  189. [189]Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, and Percy Liang. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. arXiv preprint arXiv:2408.08926, 2024. URL https://arxiv.org/abs/2408.08926. ICLR 2025.
  190. [190]Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. arXiv preprint arXiv:2412.18194, 2024. URL https://arxiv.org/abs/2412.18194.
  191. [191]Wenbo Zhang, Kaixuan Wang, Yutao Ouyang, Xiaoyu Huang, Liyang Li, Kailun Su, Weiyang Jin, Wenhao Chai, Haotian Liang, Zhiyang Dou, Yue Chen, and Tianxing Chen. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond. arXiv preprint arXiv:2609.24170, 2026. URL https://arxiv.org/abs/2609.24170.
  192. [192]Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Yi Nie, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, Jiajı Rao, Zhengrui Fang, Changxu Wei, Yu Wang, Wenbo Ding, and Chao Yu. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents. arXiv preprint arXiv:2607.08448, 2026. URL https://arxiv.org/abs/2607.08448.
  193. [193]Yunbei Zhang, Zijian Jin, Yuanzhe Liu, Janet Wang, Xilun Zhang, Youyou Zhang, Zhenyu Zhang, Daoan Zhang, Shuaicheng Niu, Gen Li, Jianfei Yang, Jihun Hamm, Ismini Lourentzou, Weirui Ye, Bo Liu, Peter Stone, and Marco Pavone. LIBERO-MAX: Do Robot Policies Adapt When the World Changes? arXiv preprint arXiv:2609.36518, 2026. URL https://arxiv.org/abs/2609.36518.
  194. [194]Zexi Zhang, Zecheng Zhu, Zidong Chen, Zulkhhuu Tuya, and Stephen James. BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation. arXiv preprint arXiv:2610.07594, 2026. URL https://arxiv.org/abs/2610.07594.
  195. [195]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv preprint arXiv:2304.13705, 2023. URL https://arxiv.org/abs/2304.13705. RSS 2023.
  196. [196]Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases. arXiv preprint arXiv:2510.20270, 2025. URL https://arxiv.org/abs/2510.20270.
  197. [197]Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and He Wang. Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection. arXiv preprint arXiv:2412.04455, 2024. URL https://arxiv.org/abs/2412.04455. CVPR 2025.
  198. [198]Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854, 2023. URL https://arxiv.org/abs/2307.13854. ICLR 2024.
  199. [199]Youwei Zhou, Bohan Hou, Yankai Fu, Yaoxu Lyu, Yifan Ye, Shilin Shan, Chen Guo, Jiaming Liu, Ran He, Caifeng Shan, Shanghang Zhang, Jianfei Yang, and Zhihe Lu. Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning. arXiv preprint arXiv:2610.28259, 2026. URL https://arxiv.org/abs/2610.28259.
  200. [200]Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Autonomous Improvement of Instruction Following Skills via Foundation Models. arXiv preprint arXiv:2407.20635, 2024. URL https://arxiv.org/abs/2407.20635. CoRL 2024.
  201. [201]Zhiyuan Zhou, Pranav Atreya, You Liang Tan, Karl Pertsch, and Sergey Levine. AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World. arXiv preprint arXiv:2503.24278, 2025. URL https://arxiv.org/abs/2503.24278.
  202. [202]Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddakuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv preprint arXiv:2009.12293, 2020. URL https://arxiv.org/abs/2009.12293.
  203. [203]Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv preprint arXiv:2507.02825, 2025. URL https://arxiv.org/abs/2507.02825.

Paper details

vlnc:W_3tbb2myc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI tools

View statement
Looking for related papers…
Contents