Embodied Playground: A Long-Horizon Agentic Robotics Benchmark for Physical, Strategic, and Spatial Reasoning

Introduction

Foundation models have shown promising capabilities in robotic tasks, supported by their ability to interpret visual observations, reason about spatial relationships and physical interactions, and formulate strategies. [1, 2, 3, 4] Existing robotics benchmarks largely emphasize the performance and generalization of control policies. [5, 6, 7, 8] This leaves a complementary question: can a foundation-model agent sustain these capabilities over long-horizon robotic tasks, where actions must establish the conditions for later stages and execution errors can affect subsequent decisions? Addressing this question requires evaluating how an agent tracks task progress, satisfies intermediate constraints, and adapts its plan throughout physical execution.

We introduce Embodied Playground, a benchmark for studying long-horizon robotic problem solving by foundation-model agents. Tasks couple physical, strategic, and spatial reasoning across dependent stages: an agent must determine what to do next, execute it through the robot interface, and use the resulting observations to update its understanding of the remaining task. Earlier actions can change object configurations, consume resources, or constrain later choices, making sustained coordination between planning and execution central to successful completion. Given a task description, permitted observations, and a specified set of tools, the agent must ground its decisions in the observed scene and revise its plan using execution feedback. Success requires achieving the final goal while respecting task rules and physical constraints throughout the episode.

Embodied Playground comprises 410 robotic task environments, ranging from basic object manipulation to long-horizon planning and dexterous coordination. These tasks vary in planning depth, geometric constraints, and physical execution demands. Tower of Hanoi requires executing legal move sequences while preserving valid intermediate configurations; cube puzzles require keeping track of puzzle states and object orientations through successive physical turns; and Kongming locks couple assembly or disassembly order with geometric feasibility and contact-rich manipulation. Multi-stage household tasks combine ordered interactions with object-placement and final-scene requirements. Launcher tasks connect trajectory prediction and causal effects with finite ammunition and, in selected scenes, target-order constraints. Shorter manipulation, precision, and dynamics tasks complement these extended sequences by probing constituent capabilities across multiple robot embodiments.

To facilitate evaluation, we curate EP-100, a representative selection of 100 tasks organized into five subsets: Launch, Tower, Lock, Cube, and Home. Together, these subsets cover 16 task types and provide focused evaluations alongside the full playground. EP-100 is divided into EP-Easy and EP-Hard by ranking the tasks of each type according to their workload, and EP-20 offers a compact selection that spans both splits. All tasks follow a common interaction interface and evaluation protocol, with specified observation and tool access, execution budgets, repeated trials, and task-specific physical success criteria.

Our contributions are threefold:

• An agent-centered perspective on long-horizon robotic problem solving. We evaluate a foundation-model agent’s ability to sustain coordinated perception, reasoning, tool use, and action across interdependent task stages, using execution feedback to track progress, revise plans, and satisfy task constraints throughout an episode.

• A diverse playground spanning task horizons and execution demands. We introduce 410 robotic task environments spanning basic manipulation and multi-stage problem solving, with variations in planning depth, geometry, and physical execution requirements. EP-100 provides a representative selection across five suites and 16 task types, complemented by two difficulty splits and the compact EP-20 selection.

• An episode-level protocol for evaluating completion and validity. We specify observation and tool-access levels, resource budgets, and repeated-trial conditions, with independent checks of final goal completion, task-rule compliance, and physical validity over the executed trajectory.

Related Work

Foundation-model agents and harnesses for Robotics.

Early systems such as SayCan, Code as Policies, and ProgPrompt connected language-model reasoning to robot action through skill selection or executable programs [9, 10, 11]. Inner Monologue and subsequent systems such as REFLECT and Statler used execution feedback, failure explanations, or maintained world-state estimates to inform later actions [12, 13, 14]. Spatial grounding developed through representations such as scene graphs, value maps, and keypoint constraints [15, 16, 17]. Systems developed in 2024–2025, such as RoboScript, ROS-LLM, and RAI, connected model-generated decisions to robot software and execution services [18, 19, 20]. Hi Robot, Agentic Robot, and Code-as-Monitor illustrate related directions in hierarchical action selection, execution coordination, and failure monitoring [21, 22, 23]. The interface between reasoning and execution has become an explicit design focus, exemplified by Robot Context Protocol and later visual, semantic, and native-software interfaces in VIA, Show-Harness, and OpenRUA [24, 25, 26, 27]. Recent tool-based harnesses also examine perception support, context management, and outcome verification [28, 29, 30, 31, 32, 33]. Alongside these interface designs, systems such as RoboHarness, RATs, ASPIRE, and CarryOn use memory or reusable skills to adapt across interactions [34, 35, 36, 37]. We offer a common basis for assessing these diverse agent designs through their physical outcomes. Its protocol makes differences in assistance and resource use explicit, supporting fair comparisons of task completion, consistency, and efficiency.

Simulation infrastructure and robotic policy benchmarks.

Simulation platforms and task benchmarks serve complementary roles in robotic policy evaluation. Early simulators such as Gazebo and MuJoCo, followed by platforms such as PyBullet, SAPIEN, and Isaac Gym, established infrastructure for robotic experiments [38, 39, 40, 41, 42]. Building on simulator backends, frameworks such as robosuite and ORBIT package environments and robot-learning workflows, with more recent examples including Isaac Lab, MuJoCo Playground, and ManiSkill3 [43, 44, 45, 46, 47]. Task benchmarks specify the learning problems and evaluation protocols. Early benchmarks such as Meta-World examine multi-task and meta-reinforcement learning, while RLBench provides manipulation tasks with demonstrations [48, 49]. Subsequent suites study generalizable manipulation, language-conditioned action sequences, and lifelong transfer, as illustrated by ManiSkill, CALVIN, ManiSkill2, and LIBERO [50, 51, 52, 53]. Household activities and longer manipulation tasks feature in benchmarks such as BEHAVIOR-1K, RoboCasa, VLABench, and RoboCasa365 [54, 55, 56, 57]. Recent evaluation also examines robustness to variation, for example in Colosseum, RoboTwin 2.0, LIBEROPlus, RoboLab, and Colosseum V2 [58, 59, 60, 61, 62]. SIMPLER and RoboDojo connect policy evaluation in simulation with real-robot performance [63, 64]. Embodied Playground adopts this foundation and organizes its evaluation around a foundation-model agent’s problem-solving episode. Its combination of abstract challenges and household tasks makes reasoning demands central to evaluation while preserving the need for successful physical execution.

Embodied-agent benchmarks.

Embodied-agent benchmarks examine reasoning through task performance under specified observation and action interfaces. Earlier examples examined physical puzzles in PHYRE and language-guided robot planning in LoHoRavens and RoboEval [65, 66, 67]. Benchmarks such as LoTa-Bench and Embodied Agent Interface subsequently evaluated task planning and its component capabilities [68, 69]. In 2025, EmbodiedEval and EmbodiedBench evaluated multimodal agents in simulated tasks, while PARTNR examined collaborative household planning [70, 71, 72]. Physical reasoning benchmarks such as PhyBlock and DeepPHY, followed by COIN and KinDER, probe assembly planning, physical prediction, causal interaction, and kinematic or dynamic constraints [73, 74, 75, 76]. Recent benchmarks examine the interaction interface itself: CaP-X and Embodied Skill Bench vary control abstractions and supplied capabilities, while VABench studies active perception and metric control [77, 78, 79]. CodeActionBench and LIBERO-Agent evaluate direct robot interaction, while Embodied Agent Arena connects component capabilities to task completion [80, 81, 82]. Longer execution and broader task coverage feature in benchmarks such as EmbodiedSWE and RobotWorld [83, 84]. LADDER varies the number of subgoals within the same environment [85]. SCOPER studies recovery under controlled skill failures, while SEAR isolates the contribution of retained experience under controlled interaction budgets [86, 87]. Physical-robot evaluations also appear in Words to World and the AA-Robotics research preview [88, 89]. Our structured task coverage supports interpretable comparisons across reasoning demands and robot embodiments. It allows performance to be examined as task demands increase, helping identify strengths and limitations that a single aggregate score may obscure.

Embodied Playground

Embodied Playground places foundation-model agents in simulated robotic environments where task completion requires both reasoning and physical execution. Each environment specifies a goal, an initial scene, a robot embodiment, and rules governing valid interaction. The collection combines familiar puzzles and everyday manipulation with controlled variations in geometry, dynamics, and planning depth.

Agent–Environment Interaction

An agent receives a task description, the robot’s control specification, and observations allowed by the evaluation setting. Public observations include camera images, robot proprioception, and declared task metadata; additional state information is governed by the selected access level. The agent uses these inputs to interpret the scene, construct a plan, and produce robot commands, optionally invoking permitted perception or planning tools. Subsequent observations provide feedback for tracking progress, correcting execution errors, and revising the plan. Figure 2 summarizes this interaction.

All environments share four core operations: describe exposes task and interface specifications; observe returns permitted observations; step applies an action and advances simulation; and finish ends the episode. The common interface preserves embodiment-specific action spaces, such as arm joint targets, gripper commands, and finger or floating-hand controls. Observation reads leave physical time unchanged, and initialization and reset are managed by the environment. Actions take effect through the declared robot controllers and task mechanisms. An independent evaluator checks goal completion, task-rule compliance, and physical validity, including any required release and stability conditions. This interaction loop connects high-level decisions to their physical consequences; access levels and scoring are detailed in § 6.

Task Environments and Robot Embodiments

Table 1 summarizes the 410 task environments in the playground. Each entry pairs a task configuration with a robot embodiment. For example, 40 Hanoi configurations are instantiated with three robot embodiments, yielding 120 entries. Seeds and initial operating modes specify trial conditions within an entry. The environments use MuJoCo [39] for launcher tasks, Isaac Lab [45] for the puzzle and bookshelf families, and ManiSkill [50] for the additional household, geometry, and dynamics tasks.

Task family	Count	Task focus	Embodiments
Launcher	100	Aiming, indirect impacts, fracture, target constraints	Dual Panda
Tower of Hanoi	120	Legal rearrangement, grasping, peg insertion	Panda; FR3–Sharpa; floating Sharpa
Pocket cube	20	Prescribed layer turns and regrasping	ALOHA; dual floating Sharpa
Rubik’s cube	60	Cube solving and physical face turns	Panda; floating Sharpa
Kongming lock	60	Partial and full assembly and disassembly	Dual Panda; dual floating Sharpa
Bookshelf	20	Sorting, arrangement, and clutter removal	Dual Panda
ManiSkill tasks	30	Household sequences, precision, and dynamics	Fetch; Panda; Panda–Sharpa; xArm7–Allegro; UR10e–Shadow
Total	410

Task variations couple discrete goals with continuous control. Hanoi spans shortest legal plans of 1, 3, 9, and 31 moves, with variations in support height, insertion clearance, mass, friction, and obstacles. Pocket-cube tasks require state transitions under a prescribed face-order interlock, whereas Rubik’s-cube tasks require restoring uniform face colors using declared auxiliary fixtures for support, grasp attachment, and layer control. Kongming locks range from inserting or removing a key to assembling or disassembling a six-piece structure. Bookshelf tasks combine category-dependent placement, orientation, and packing requirements. The ManiSkill tasks extend this range to activities such as preparing a tea station, inserting a keyed connector, catching a moving ball, and guiding a spring-closing door. Shared task definitions across selected embodiments support comparisons between parallel-gripper manipulation and dexterous-hand control.

Spatial, Physical, and Strategic Reasoning

Spatial reasoning connects visual observations to object configurations and feasible motions. Agents must relate task-relevant positions and orientations to the robot’s workspace, account for occlusion, and identify geometric compatibility. Examples include aligning a ring with a peg, orienting a notched lock piece for insertion, tracking cube faces through reorientation, and arranging upright books within a narrow shelf. These tasks require geometric understanding to guide grasp selection, motion, and placement.

Physical reasoning concerns how actions change the scene through forces, contact, and motion. Launcher tasks combine trajectory prediction with indirect effects such as rolling objects, falling structures, and glass fracture. Manipulation tasks introduce mass and friction variations, stability requirements, and dynamic interactions such as stopping a rolling object or controlling a spring-loaded articulation. Agents must connect intended outcomes to executable controls and use observed motion to adjust subsequent actions.

Strategic reasoning concerns action dependencies and decisions over extended horizons. Hanoi requires maintaining legal intermediate states; lock assembly requires an order compatible with the interlocking geometry; and household tasks combine multiple milestones into a final arrangement. Launcher scenes additionally impose finite ammunition, target-order requirements, and protected objects, making later decisions depend on earlier impacts and the remaining scene. These demands interact: a viable strategy must be geometrically feasible and physically executable, while execution feedback can change which strategy is appropriate.

EP-100: A Representative Evaluation Set

Task Selection

Task Selection

All EP subsets evaluate launcher tasks only in the loaded mode, with the first projectile preloaded. Ground-mode initialization is outside their evaluation scope. Other task families use their default initialization mode. Each task therefore defines one evaluation group with four prescribed conditions; a separate attempt budget determines whether Pass@1 alone or both Pass@1 and Pass@2 are measured. EP-100 selects 100 of the 410 environments so that a complete evaluation remains affordable and still reflects the breadth of the playground. The selection favors tasks that most agents can attempt through the shared runtime. Parallel-gripper embodiments therefore receive most of the slots, and each dexterous-hand family contributes a smaller sample drawn from its easier native configurations. Within every family the selected tasks follow the family’s own difficulty axis, such as the launcher tier, the Hanoi planning depth, or the subtask and pose of a lock, so that the native difficulty range of each family remains visible in the selection. Launcher scenes are drawn from the 80 scenes with a recorded full-task success in the task atlas, which establishes that each selected scene can be completed within its shot budget. Several case identifiers recur across embodiments, and this shared structure allows the same task definition to be compared between parallel grippers and dexterous hands.

The selected tasks form five suites, and each suite divides into task types that share a single dominant demand (Table 2). Launch progresses from open-field aiming to scenes whose outcome depends on indirect causal chains, on the fracture of glass panes, or on rules that protect or order the targets. Tower varies the planning depth and the physical conditions of the Hanoi board and then repeats representative boards with dexterous hands. Lock covers assembly and disassembly of the Kongming lock with two Panda arms and with two floating Sharpa hands, and Cube combines Rubik’s-cube solving with prescribed pocket-cube turns. Home extends the evaluation to household settings, where its bookshelf and long-horizon types test how an agent structures a multi-step task and its precision and dynamics types test how accurately it controls contact and motion. Because each type isolates one dominant demand, results aggregated by type indicate which capability limits an agent.

Suite	Type	Tasks	E / H	Principal demand
Launch	Aim	7	4 / 3	Direct shots at single or paired targets, including azimuth aiming
	Causal	13	6 / 7	Indirect hits through rolling bodies, dominoes, and collapsing structures
	Glass	5	3 / 2	Breakable glass panes with fracture thresholds
	Rules	9	4 / 5	Ordered or protected targets and recovery after a miss
Tower	Plan	5	3 / 2	Planning depth from 1 to 31 moves on a standard board
	Physics	8	4 / 4	Physical board variations under the same legal-move rule
	Dex	8	4 / 4	Representative boards with an FR3–Sharpa or floating Sharpa hand
Lock	Assemble	6	3 / 3	Inserting the key, the key and bar, or the full lock with two Panda arms
	Disassemble	6	3 / 3	Removing the key, the key and bar, or all six pieces with two Panda arms
	Dex	4	2 / 2	Assembly and disassembly with two floating Sharpa hands
Cube	Rubik	8	4 / 4	Solving scrambles of 5, 10, or 15 tokens through the auxiliary fixture
	Pocket	4	2 / 2	Prescribed layer turns with ALOHA grippers or two floating Sharpa hands
Home	Tidy	5	2 / 3	Organizing a bookshelf and correcting misplaced objects
	Long	4	2 / 2	Multi-step household sequences with a mobile Fetch robot
	Precise	4	2 / 2	Insertion, hooking, and narrow-gap placement with a Panda arm
	Dynamic	4	2 / 2	Intercepting, pushing, and closing objects in motion with a Panda arm
Total	16 types	100	50 / 50

Evaluation Subsets

EP-100 is divided into two disjoint splits of 50 tasks, EP-Easy and EP-Hard. The assignment is derived from the task definitions, which keeps each split fixed as agents improve and makes it independent of any particular evaluation outcome. Each task receives a workload score from the quantities that make its family demanding, such as the move count of a Hanoi board, the scramble length of a cube, or the subtask and pose of a lock, and the score rises for dexterous-hand embodiments. For Launch, only the loaded-mode difficulty contributes to the workload score; ground-mode pickup requirements are excluded. The tasks of each type are ranked by this score, and the lower half of every type forms EP-Easy. A type with an odd number of tasks places its median task in the half whose neighboring score is closer, and the remaining ties are resolved so that each split holds exactly 50 tasks and the two splits differ by at most one task within any suite. Both splits therefore contain all 16 task types in nearly the same proportions, so a gap between EP-Easy and EP-Hard results reflects difficulty within a comparable task mix. Appendix § 9.8 defines the score for each suite.

Evaluation Protocol

EP-20 is a compact selection for rapid and representative evaluation. It takes ten tasks from EP-Easy and ten from EP-Hard, which together cover every task type and robot embodiment of EP-100 on all three simulators, and its six launcher scenes span the five native tiers. Half of its Hard tasks have been completed at least once in pilot agent runs on the same or closely related scenes, so EP-20 can register progress on difficult tasks before an agent masters the full set. Every subset is scored with the suite-balanced aggregation of § 6.5, which gives each of the five suites one fifth of the subset score. Table 3 summarizes the four subsets, and Appendices § 9.8–§ 9.11 list their members.

Subset	Tasks	Launch	Tower	Lock	Cube	Home	Conditions
EP-100	100	34	21	16	12	17	400
EP-Easy	50	17	11	8	6	8	200
EP-Hard	50	17	10	8	6	9	200
EP-20	20	6	4	3	3	4	80

Evaluation Protocol

Evaluation uses four prescribed trial conditions per task and four levels of observation and tool access. The paper adopts loaded-only launcher initialization.¹ The protocol fixes the information and tools available to an agent, the conditions of repeated interaction, and the physical criteria for success. All trials use the shared environment runtime introduced in § 3.1.

Access Levels

Capability levels specify available assistance independently of task difficulty. The four levels are cumulative:

• L0: code-enabled control. The agent may use Python’s standard library, NumPy, Pillow, and self-written algorithms, including its own inverse kinematics, dynamics prediction, and planning. Inputs comprise RGB images, proprioception, camera calibration, and public task and robot information. Prebuilt domain tools and auxiliary pretrained perception or grasp models are unavailable.

• L1: perception and planning tools. Explicit grants add installed tools for visual reconstruction, geometry, inverse kinematics, motion planning, symbolic solving, and observation-derived proxy dynamics or optimization. Geometry and dynamics estimates are derived from permitted observations and declared task parameters.

• L2: grasp and contact models. Explicit grants additionally permit pretrained models that generate or refine grasp and contact proposals, which the agent incorporates into its planning and control.

• L3: privileged observations. A declared read-only profile additionally exposes selected simulator fields, such as object poses, velocities, geometry, or contacts, according to the environment’s observation contract.

Each run records the observation profile, capability level, exact tool grants, and available tool versions. RGB-D observations require an authorized profile and a supporting environment. A higher level makes tools eligible for use; the actual tool set is fixed explicitly. Privileged read access preserves the same action interface and task rules. The evaluated model retains its declared frozen weights; additional pretrained action controllers and reference trajectories are excluded. Agents receive public task parameters, including launcher spring stiffness, while private initialization records and undeclared evaluator diagnostics remain with the evaluation host.

Trial Design

An evaluation group fixes the task, base seed, initial mode, operation and observation profiles, capability level, tool grants, budgets, and protocol and implementation versions. The agent version, code, and instructions are also fixed before execution. Every group contains exactly four independently initialized episodes. The standard schedule uses base seed zero; additional base seeds define separate groups.

Launcher tasks have one evaluated group: loaded, with the first projectile preloaded. Subsequent reloading remains part of any task that requires further shots. Each other task has one default group. Thus, evaluation of the full playground contains 410 groups and 1,640 trial conditions per fixed setting. EP-100 contains 100 groups and 400 conditions. EP-Easy and EP-Hard each contain 50 groups and 200 conditions, and EP-20 contains 20 groups and 80 conditions. These counts describe initial conditions, not the number of attempts under a Pass@2 budget. The runtime also supports ground initialization, but that mode is excluded from these benchmark scores.

The four conditions preserve each task’s goal and rules while varying family-specific factors. Launcher trials use spring stiffnesses of 22, 24, 26, and 28 N/m, together with prescribed launcher and ammunition offsets and visual backgrounds. Hanoi and pocket-cube trials use consecutive reset seeds with bounded pose, mass, friction, and lighting perturbations; bookshelf trials use seeded native layout sampling. Rubik’s cube, Kongming lock, and ManiSkill tasks retain their authored physical configurations and use lighting multipliers of 0.85, 0.95, 1.05, and 1.15. Formal ManiSkill trials use physics variant zero. Exact conditions are specified in the per-task schedule.

Each trial starts with a fresh environment and agent session. Within an episode, the agent may retain observations and adapt its plan, while physical changes, damage, and consumed resources persist. Between episodes, agent memory, workspaces, and caches are cleared; trajectories and private outcomes are not reused for adaptation. Initialization and reset belong to the evaluation host. All four registered conditions remain in the record. In the base single-attempt protocol, completed failures or invalid attempts cannot be replaced by additional retries. A declared multi-attempt experiment must retain each attempt and specify its opportunity budget and infrastructure-exclusion policy separately; § 7.1 defines the Codex Pass@1/Pass@2 extension.

Resource Budgets

Table 4 gives the registered execution limits. Actions advance simulation at the native control rate, with physics evaluated at the corresponding substep frequency. Observation and tool reads leave simulated time unchanged. A separate supervisor enforces elapsed wall time, including startup and initialization. Dynamics tasks can additionally impose event deadlines of 5–20 s; the required successful dwell must finish within the applicable deadline.

Task family	Physics/control (Hz)	Maximum actions	Simulated time	Wall time
Launcher	1,000/50	7,500	150	600
Hanoi	120/30	18,000	600	3,600
Pocket cube	240/60	18,000	300	3,600
Rubik’s cube	120/30	109,800	3,660	19,800
Kongming lock	120/30	36,000	1,200	3,600
Bookshelf	120/30	1,500–12,300	50–410	3,600
ManiSkill: long horizon	240/20	6,000	300	3,600
ManiSkill: geometry/dynamics	240/20	2,400	120	3,600

Model-side budgets, including token consumption, image inputs, and auxiliary computation, are specified by the experiment host separately from the environment limits. Comparisons use the same declared allowances and report realized usage. The environment gateway meters physical execution; model-resource accounting is performed by the agent evaluation system.

Episode Scoring

Success is determined by the task’s independent evaluator from the executed physical trajectory. The final state must satisfy the goal, the interaction history must satisfy task rules, and all required physical, release, and stability conditions must hold before the applicable budget expires. Recorded violations remain binding throughout the episode. Calling finish evaluates existing evidence and grants no additional simulation or settling time.

The success predicates reflect each task’s structure. Launcher tasks require the specified targets, impact order, protection rules, and any required fracture, toppling, or damage paths; protected-target scenes include a minimum 3 s observation period after the last required-target hit. Hanoi requires the exact ring arrangement, legal moves, valid seating, and 1 s stability. Pocket cubes require the target state and prescribed number of committed quarter turns, aligned detents, no active layer, and 1 s stability. Rubik’s cubes require a solved physical configuration, legal fixture use, return and release, and a fresh 0.5 s stable window. Kongming locks require the case-specific arrangement and release, followed by 2 s stability for assembly or 1 s for disassembly. Bookshelf tasks require all placement rules to hold continuously for 0.5 s. ManiSkill tasks require the ordered milestones, final predicates, protected-object conditions, and 1 s successful dwell. Numerical tolerances remain fixed in each task contract.

Each registered trial is classified as success, failure, invalid, or missing. A valid failure includes an unsuccessful simulated-time timeout, object loss, physical-rule violation, or voluntary termination without success. Initialization or evaluator errors, rejected actions, agent exceptions, and wall-clock expiration produce invalid attempts. A slot without a terminal record is missing. These categories preserve the distinction between an unsuccessful physical attempt and an incomplete evaluation record.

Evaluation Metrics

A group is complete when all four slots contain valid terminal outcomes. For a complete group $g$, let $z_{g,i} \in\{0,1\}$ denote success in trial $i$. We report its success rate and all-four-success indicator:

The fixed denominator includes every successful or failed trial. If any slot is invalid or missing, the group is marked partial and both final metrics are undefined. Its observed success count $n_g^{+}$, invalid and missing trial indices, and lower bound $n_g^{+}/4$ are retained explicitly.

Group metrics are combined in three stages so that each suite carries equal weight. A task $\tau$ with registered groups $\mathcal{G}_{\tau}$ contributes to its suite $k$ and to the evaluated subset through

where $\mathcal{T}_k$ is the set of subset tasks in suite $k$. At a fixed evaluation setting, each task has one group, so its task score equals $S_g$, including for loaded-mode launcher tasks. The suite score averages the tasks of one suite, and the subset score assigns one fifth of its weight to each of the five suites, so the reported score of EP-100, EP-Easy, EP-Hard, or EP-20 reflects every suite equally regardless of its size. The all-four indicator is aggregated in the same way and yields $A_{\tau}$, $A_k$, and $A$. We also organize results by task type and embodiment, keeping different access settings and attempt budgets distinct. Partial evaluations retain the planned denominator, observed-success lower bound, completed-group coverage, and invalid or missing counts, with final aggregate metrics left undefined.

Success rates are accompanied by efficiency and failure statistics. Time@S is the mean wall-clock time from episode start to the success event over successful trials, and Tok@S is the mean number of model tokens that a successful trial consumes, counting cached and uncached input together with output. Each failed trial receives one failure category from its first recorded violation or from its termination reason, and the share of each category describes how agents fail. Table 19 in Appendix § 9.8 lists the categories, including the separate category for invalid attempts.

Code	Meaning
COLL	Forbidden or excessive contact (robot-object, arm-arm, force limit)
PEN	Penetration beyond the audited limit
LIMIT	Joint, speed or physics limit exceeded
DROP	Object dropped, unseated or pushed out of the workspace
RULE	Task rule broken (illegal move, wrong order, protected item hit, launcher interlock)
MECH	Assist-mechanism constraint broken (cube fixture relation, mechanism command)
AMMO	All shots used without clearing the required targets
TIME	Environment step, simulated-time or wall-clock budget exhausted
QUIT	Agent finished without a qualified success
BUDGET	Agent token or image budget exhausted (orchestrator stop)
ERR	Invalid attempt caused by a runtime, evidence, or model-provider fault; it is recorded as invalid and leaves its group partial

Each episode record includes the task and group identity, trial index, agent and environment revisions, observation and tool settings, budgets, realized initialization, action count, elapsed simulated and wall time, terminal outcome, and first recorded violation. Detailed state and scorer logs are retained by the evaluator; agents receive only authorized observations and filtered results. These records support reproducible comparisons and trajectory-level analysis of perception, planning, and execution.

Experimental Evaluation

We evaluate GPT-6-Astra and Opus-5.5-extra on EP-100.

Experimental Setup

Evaluation configuration. We evaluate Codex using gpt-6-astra with high reasoning effort and L3 privileged observations. Opus-5.5-extra uses extra reasoning effort. The agents generate and execute control programs through the public environment SDK, using Python’s standard library, NumPy, and Pillow. No auxiliary tool grants, reference solutions, or pretrained action controllers are provided. Each attempt follows the physical execution budgets in Table 4, with a declared allowance of 30 million tokens, including cached input, and 100 image inputs.

The evaluation covers all 100 EP-100 tasks under four prescribed conditions at base seed zero, yielding 400 task–condition pairs. Launcher tasks use loaded initialization only: the first projectile is preloaded, while subsequent loading remains part of multi-shot tasks. EP-Easy, EP-Hard, and EP-20 are evaluated by selecting the corresponding records under the loaded-only definitions in § 4.3.

Attempt eligibility. Each condition is scored using its first two valid attempts in chronological order, stopping at the first success. For GPT-6-Astra, the final audit retains 846 historical records: 736 scored attempts, comprising 91 successes and 645 failures; 69 confirmed infrastructure incidents excluded from the opportunity budget; and 41 extra attempts outside the scoring window. Those extra attempts include two native successes, which remain unscored. Diagnostic replays are used only for auditing. All 400 conditions have complete outcomes, with no outstanding valid opportunities. Ordinary agent failures and budget exhaustion, including wall-clock timeouts, consume an opportunity; confirmed infrastructure incidents do not. These rules extend the base single-attempt protocol.

Recorded process launches establish separate agent threads and public, scratch, and runtime directories for all 736 scored attempts. Trial identity is supported by 548 standalone manifests and 188 delegated native trace manifests. This supports isolation at the session and directory level, but does not certify cleared host or service caches or complete simulator-state independence. Native verdicts are retained, including failures involving incompletely disclosed rules, with disclosure defects reported separately below.

Metrics. Pass@1 measures success on the first valid attempt; Pass@2 measures success within the first two valid attempts on the same condition. A first-attempt success contributes to both metrics. Let $p_{\tau,i}^{(k)}$ indicate success within $k$ valid attempts for task $\tau$ and condition $i$. For subset $\mathcal{D}$, we compute

where $\mathcal{D}_s$ contains the tasks in suite $s$. Conditions are averaged within tasks, tasks within suites, and the five suites equally. The four conditions are separate evaluation instances, not additional attempts at one instance. Pass@2 is computed from observed outcomes rather than inferred from Pass@1.

Main Results

Performance across attempts. GPT-6-Astra achieves 13.80% Pass@1 and 19.94% Pass@2 on EP-100 (Table 5), a gain of 6.15 percentage points before rounding. Opus-5.5-extra achieves 29.22% Pass@1 and 35.85% Pass@2 using 683 valid attempts. For GPT-6-Astra, the first valid attempt solves 64 of 400 conditions; second attempts add 27 successes, recovering 8.04% of the 336 initially unsuccessful conditions. The pooled condition rates are 16.00% and 22.75%, respectively, whereas the headline scores weight each suite equally. Thus, a second opportunity improves coverage, but 309 GPT-6-Astra conditions remain unsuccessful.

Subset	Model	Tasks	Conditions	Attempts	Pass@1	Pass@2
EP-100	GPT-6-Astra	100	400	736	13.80	19.94
	Opus-5.5-extra	100	400	683	29.22	35.85
EP-Easy	GPT-6-Astra	50	200	358	18.42	27.41
	Opus-5.5-extra	50	200	332	33.99	42.88
EP-Hard	GPT-6-Astra	50	200	378	8.81	12.01
	Opus-5.5-extra	50	200	351	24.21	28.48
EP-20	GPT-6-Astra	20	80	147	15.42	18.75
	Opus-5.5-extra	20	80	137	30.58	39.13

Subset performance. For GPT-6-Astra, EP-Easy reaches 18.42% Pass@1 and 27.41% Pass@2, compared with 8.81% and 12.01% on EP-Hard. The second-attempt gains are 8.99 and 3.20 percentage points, respectively. The workload-based split therefore separates aggregate performance in this evaluation, although the relationship is not uniform: Home scores higher on EP-Hard than on EP-Easy (Table 6). EP-20 obtains 15.42% Pass@1 and 18.75% Pass@2 over 80 conditions. Its suite profile differs from EP-100, especially for Tower and Home. As a curated selection partly informed by pilot success evidence, EP-20 provides a compact diagnostic evaluation rather than an unbiased estimate of full-set performance.

Suite	Model	EP-100	EP-Easy	EP-Hard	EP-20
Launch	GPT-6-Astra	19.12 / 25.74	20.59 / 30.88	17.65 / 20.59	12.50 / 16.67
	Opus-5.5-extra	15.51 / 27.70	15.21 / 32.60	15.81 / 22.81	5.65 / 19.78
Tower	GPT-6-Astra	29.76 / 41.67	45.45 / 61.36	12.50 / 20.00	12.50 / 25.00
	Opus-5.5-extra	60.59 / 66.14	71.51 / 79.82	48.58 / 51.09	50.00 / 56.25
Lock	GPT-6-Astra	6.25 / 14.06	12.50 / 28.13	0.00 / 0.00	0.00 / 0.00
	Opus-5.5-extra	39.81 / 45.63	52.03 / 60.56	27.60 / 30.70	37.63 / 37.63
Cube	GPT-6-Astra	2.08 / 2.08	4.17 / 4.17	0.00 / 0.00	8.33 / 8.33
	Opus-5.5-extra	6.04 / 10.72	8.18 / 15.03	3.90 / 6.41	14.82 / 27.41
Home	GPT-6-Astra	11.76 / 16.18	9.38 / 12.50	13.89 / 19.44	43.75 / 43.75
	Opus-5.5-extra	24.17 / 29.06	23.03 / 26.41	25.17 / 31.41	44.80 / 54.58

Performance across tasks. For GPT-6-Astra, Tower has the highest suite-level Pass@2 at 41.67%. Its Physics and Plan types reach 68.75% and 50.00%, while Dex reaches 9.38%. Launch reaches 25.74%, with Aim at 46.43% and Rules at 5.56%. Lock reaches 14.06%, with all nine successful conditions in EP-Easy and no success in Lock-Dex. Cube records one successful Pocket condition and no Rubik successes, yielding 2.08% overall. Home reaches 16.18%, with successes confined to Precise and Dynamic; Tidy and Long record none. These differences describe the evaluated task configurations and do not constitute controlled comparisons of reasoning or embodiment.

For GPT-6-Astra, at least one condition succeeds for 31 tasks on the first valid attempt and 40 tasks within two attempts. Only four and nine tasks, respectively, succeed across all four conditions. Success therefore remains uneven across the prescribed variations, even with a second opportunity. The latter count permits two attempts per condition and is distinct from the base all-four single-attempt metric. Table 7 reports condition outcomes for both models and Opus-5.5-extra successful-attempt timing.

Suite	Cond.	GPT-6-Astra Success	GPT-6-Astra Failure	Opus-5.5-extra Success	Opus-5.5-extra Failure	Time@S (min)
Launch	136	35	101	36	100	6.2
Tower	84	35	49	55	29	38.5
Lock	64	9	55	20	44	37.0
Cube	48	1	47	5	43	32.5
Home	68	11	57	17	51	26.6

Evaluation Diagnostics

Observed failure events. The audit resolves the 399 previously unclassified failure records and assigns diagnostic labels to all 645 currently scored failures. Table 8 summarizes the most frequent labels by suite. Launch failures frequently involve limit and interlock checks, including 86 elevation-limit events and 50 launches before completion of an aiming cycle. Tower failures include penetration, ring displacement, and native target-validation aborts. Lock is dominated by contact checks, while 64 of 94 Cube failures carry mechanism-constraint labels. Home failures include time limits, penetration, and object loss. These labels identify native termination events; they do not, by themselves, establish whether the agent, controller, or simulator caused the underlying physical behavior.

Suite	Failed attempts	Frequent labels (count)
Launch	211	Limit (129), rule (58)
Tower	108	Penetration (35), displacement/loss (33), limit (27)
Lock	115	Contact (97), penetration (19)
Cube	94	Mechanism (64), limit (16)
Home	117	Time (40), penetration (27), displacement/loss (23)

Failure attribution. The audit identifies incomplete rule or threshold disclosure affecting 574 of the 645 scored failures (88.99%). Examples include launcher joint-speed margins, forbidden-contact thresholds in Lock, and fixed-joint integrity tolerances in Cube. A missing numerical tolerance is distinguished from an undisclosed task obligation. These verdicts remain in the reported scores under the audit policy; disclosure defects are not treated as successes or automatically excluded. Consequently, the scores measure completion under the implemented native contracts, while the failure distribution cannot be interpreted solely as a measure of agent reasoning or control deficiencies. Evaluations with fully disclosed contracts are needed to isolate those limitations.

Measurement scope. Mean supervisor-recorded time over the 91 GPT-6-Astra successful attempts is approximately 6.25 minutes, measured at roughly 0.25-second polling resolution; it excludes prior unsuccessful attempts. Complete provider token usage is unavailable for this run, so Tok@S is not reported. Controlled access-level, controller, and feedback ablations are needed to separate agent capabilities from evaluation artifacts.

Conclusion and Future Work

We introduced Embodied Playground, a benchmark comprising 410 robotic task environments for evaluating spatial, physical, and strategic reasoning through embodied interaction. EP-100 and its complementary subsets support focused evaluation under a shared protocol with independent physical success checks. The audited Codex results show limited and uneven task completion under the implemented native contracts. The audit also identifies rule-disclosure gaps that constrain attribution of failures to agent capabilities.

We will maintain and expand the benchmark with new tasks and improved evaluation mechanisms, including clearer rule disclosure and stronger validation, to strengthen coverage, reliability, and reproducibility. We will also evaluate a broader range of foundation models and agent systems to characterize their capabilities and track progress in embodied problem solving.

Task Catalog and Evaluation Subset Indices

Catalog Organization

The full Embodied Playground contains 410 registered task environments. Each identifier has the form family/embodiment/case; an environment is counted once for each registered task–embodiment combination. Repeated trials and launcher initialization modes belong to the evaluation protocol and do not increase the catalog size. The catalog below follows the task manifest at Agent-bench revision a58a288. Compact ranges enumerate all cases, while variant tables describe their objectives and principal sources of difficulty.

The launcher, Hanoi, lock, cube, and household environments contribute 100, 120, 60, 80, and 50 tasks, respectively. Household environments comprise 20 bookshelf tasks and 30 ManiSkill tasks. Native difficulty labels describe the design of an individual task family; evaluation subset assignments are recorded separately.

Launcher: Aiming and Physical Causality

The launcher catalog contains marble/dual-panda/A001–A100, implemented in MuJoCo. Two Panda arms operate the launcher through physical controls: aiming handwheels, a brake, a spring mechanism, and ammunition loading. Finite ammunition couples trajectory selection with decisions about projectile type, draw, and firing order. The runtime supports loaded and ground initialization of the same authored scene with different loading requirements. All EP benchmark subsets use loaded initialization only, with the first projectile preloaded; subsequent reloading remains part of multi-shot tasks.

The scenes range from unobstructed targets to interactions mediated by rolling objects, dominoes, bridges, roofs, and collapsing structures. Glass introduces fracture thresholds; protected targets and required impact sequences constrain otherwise successful shots. For example, A007 introduces draw control with a nearby target, A002 restricts the firing corridor with a bunker, A036 requires clearing an upper target while preserving one below, and A099 requires contact with an inclined return bank before either target can be cleared. These tasks combine geometric aiming with prediction of delayed physical consequences and recovery within a finite shot budget.

Tier	Tasks	Case IDs
1	4	A007, A008, A009, A010
2	9	A001, A011, A012, A013, A017, A018, A023, A037, A075
3	25	A002, A003, A004, A014, A019, A020, A024, A025, A026, A038, A039, A040, A044, A049, A050, A053, A054, A056, A059, A061, A064, A071, A076, A077, A080
4	40	A005, A006, A015, A016, A021, A022, A027, A028, A029, A030, A041, A042, A043, A045, A048, A051, A052, A055, A057, A058, A060, A062, A063, A065, A066, A069, A070, A072, A073, A078, A079, A081, A082, A083, A084, A085, A086, A090, A091, A094

Hanoi: Planning with Geometric and Physical Constraints

Forty board configurations are instantiated with three embodiments: hanoi_panda/panda, hanoi_sharpa/fr3-sharpa, and hanoi_floating/floating-sharpa. Each prefix contains cases H01–H40, yielding 120 Isaac Lab environments. The board contains five rings and three pegs. An agent must reach the specified arrangement by moving only an exposed ring and preserving the size ordering on every peg, while grasping, transporting, and seating rings physically.

The configurations form ten groups of four cases. Within each group, the cases have native levels 1–4 and shortest legal solution lengths of 1, 3, 9, and 31 moves, respectively. The groups vary spatial layout, contact conditions, or motion constraints. This structure provides related planning problems across parallel grippers and dexterous hands, while exposing the interaction between a legal symbolic plan and its physical execution.

Cases	Variation	Principal demand
H01–H04	State decomposition	Identify the required subproblem and maintain legal ring ordering.
H05–H08	Planar layout	Adapt transfers to different peg arrangements.
H09–H12	Support height	Transfer rings between supports at different elevations.
H13–H16	Insertion clearance	Align ring holes and pegs under varying clearance.
H17–H20	Mass distribution	Adapt lifting and transport to ring masses.
H21–H24	Friction	Maintain stable grasps under different contact conditions.
H25–H28	Handle orientation	Approach and grasp handles with varying yaw.
H29–H32	Path obstacles	Route ring transfers around intervening obstacles.
H33–H36	Peg length	Clear and seat rings on pegs of different lengths.
H37–H40	Combined constraints	Coordinate planning with multiple geometric and physical variations.

Cubes: Layer Manipulation and Sequential Solving

Pocket cube.. Ten prescribed transitions on a $2 \times2$ cube are instantiated under pocket_cube/aloha and pocket_cube/dual-sharpa, giving 20 Isaac Lab environments. The latter uses two floating Sharpa hands. Each case specifies an initial cube state and a target transition. A face-order interlock constrains the sequence, and success requires the target state, the prescribed number of committed quarter turns, and aligned, stable layers. These tasks emphasize support, torque application, regrasping, and coordination between the manipulating and stabilizing hands. Table 11 lists all ten transitions; the native level is shared across the two embodiments.

Case	Level	Transition	Manipulation demand
C01	1	U	Support the lower layer during a top-layer turn.
C03	1	L	Create clearance for a left-layer turn.
C07	2	D	Free the bottom layer from its support.
C08	2	U2	Complete a half turn through intermediate detents.
C09	3	L	Control a side-layer turn with increased mass.
C10	3	R’	Maintain contact during a low-friction reverse turn.
C14	4	U F R	Coordinate successive turns about three axes.
C15	4	F’ U’ R’	Execute a reverse sequence to restore the target state.
C17	5	R U R’ U’	Maintain state tracking through a commutator sequence.
C20	5	B’ U’ R’ F’	Recover through a multi-axis sequence with combined perturbations.

Rubik’s cube. Thirty scrambled $3 \times3$ configurations are instantiated under rubiks_panda/panda and rubiks_sharpa/floating-sharpa, giving 60 Isaac Lab environments. Each case fixes a scramble, pose, and logical orientation. Scramble length is measured in construction tokens: a half turn counts as one token but two quarter turns. The token count therefore describes the construction procedure rather than the shortest solution length.

An agent must infer the cube state, plan a solving sequence, and execute layer turns through the declared grasp attachment and detent mechanics of hybrid_fixture_v1. Completion requires a solved cube, return to the designated home configuration, release, and stability. Pose and orientation variations require agents to maintain a consistent mapping between observed faces and planned moves.

Cases	Native level	Scramble tokens	Environments
C01–C10	Easy	5	20
C11–C20	Medium	10	20
C21–C30	Hard	15	20

Kongming Lock: Assembly and Disassembly

The lock catalog comprises a six-piece burr puzzle with five notched bars and one solid key. Thirty cases are instantiated under kongming_lock/dual-panda and kongming_lock/dual-floating-sharpa, giving 60 Isaac Lab environments. Case prefixes A and D denote assembly and disassembly, respectively. Each direction contains complete-lock, key-only, and key-plus-bar objectives, allowing the catalog to vary the extent of a manipulation sequence independently of the initial pose.

Each five-case block spans poses P01–P05 in order. Their native pose difficulty labels are Easy, Easy, Medium, Hard, and Hard. For partial tasks, the key-plus-bar objective concerns the key and frame_y1; complete tasks concern all six pieces. Assembly requires physically consistent insertion and a stable released structure. Disassembly requires extracting the designated pieces and placing them in their prescribed final arrangement. Both directions test ordering dependencies, spatial clearance, and coordinated support during contact-rich motion.

Assembly	Disassembly	Objective	Environments
A01–A05	D01–D05	Complete six-piece lock	20
A06–A10	D06–D10	Key only	20
A11–A15	D11–D15	Key and one bar	20

Bookshelf: Organization and Scene Correction

Twenty Isaac Lab tasks use bookshelf/dual-panda. Their case IDs combine one of five setting names with a native level D1–D4. The agent organizes a desk hutch according to object-specific placement rules: books stand upright and form a packed row on the middle shelf; display objects stand on the top shelf with the required spacing and orientation; crates or boxes occupy the bottom region; and trash belongs in the waste bin. Completion requires released, stationary objects in valid placements.

The settings vary the starting container, object composition, and need to correct an existing arrangement. Higher native levels introduce additional objects, flat or awkward initial poses, named-wall packing, clutter, or misplaced objects. In correction tasks, the agent must identify which existing placements violate the rules from the scene. The task names below summarize all twenty cases.

Case	Task configuration
BooksTableD1	Three upright books onto the middle shelf
BooksTableD2	Five books, some lying flat
BooksTableD3	Books onto the middle shelf
BooksTableD4	Ten books against a named wall
ShelfTableD1	Books, a trophy and one box
ShelfTableD2	Books, three display objects and two boxes
ShelfTableD3	Organize a full desk hutch from the desk
ShelfTableD4	A crowded desk with flat books and awkward display objects
ShelfCrateD1	Unpack a neatly packed crate
ShelfCrateD2	Unpack a small jumbled crate
ShelfCrateD3	Unpack a jumbled moving crate onto the hutch
ShelfCrateD4	Unpack a full crate against a named wall
ShelfCrateFixD1	One book on the wrong layer of a tidy shelf
ShelfCrateFixD2	Unpack a packed crate and correct a misplaced globe
ShelfCrateFixD3	Unpack the crate and correct a misplaced globe
Case	Task configuration
ShelfCrateFixD4	Unpack the crate and correct two misplaced objects
TableCleanD1	Three books and two pieces of trash
TableCleanD2	Books, display objects and four pieces of trash
TableCleanD3	Organize the hutch and throw the trash away
TableCleanD4	Organize the hutch with trash already hidden on it

ManiSkill: Household Sequences, Precision, and Dynamics

Thirty environments run in ManiSkill/SAPIEN: maniskill_long_horizon contains L1–L10, maniskill_geometry contains P1–P10, and maniskill_dynamics contains D1–D10. Each case has one registered embodiment, listed below using its exact runtime identifier. The robot collection includes Fetch, Panda, and the three arm–hand combinations registered as everyday_robot_01, everyday_robot_03, and everyday_robot_09. Together, the family prefix, robot identifier, and case ID recover the complete environment identifier.

Long-horizon tasks.. These tasks require object selection, ordered interaction with storage mechanisms, and restoration of the workspace. Goals specify both what to change and which objects to preserve. Dependencies include retracting a rack before closing a door, inserting dividers before packing, and clearing obstructions before closing a case. Ordered milestones and final-state predicates capture these intermediate and terminal requirements.

Case	Robot identifier	Objective
L1	fetch	Clear a writing area, store books and stationery, close the drawer, and preserve a colleague’s objects.
L2	everyday_robot_09	Store ingredients and utensils, retract the oven rack, switch off both knobs, and close storage without disturbing the loaf.
L3	fetch	Arrange two mug–spoon settings, clear the serving area, close storage, and preserve the third setting.
L4	everyday_robot_03	Select the specified screwdriver, handle, and screws; pack, close, and latch the repair case.
L5	fetch	Unload specified mugs and cutlery, preserve the bowl, retract the rack, and close the dishwasher and storage.
L6	fetch	Pack bottles below a divider and food above it; preserve the separate lunch and close storage.
L7	fetch	Select exactly three colored markers and a drawing book, store blocks, and preserve excluded items.
L8	fetch	Select a suitable planter and saucer, arrange supplies, clear the work area, and preserve existing plants.
L9	everyday_robot_01	Insert dividers, pack camera equipment by compartment, clear the lid rim, and latch the case.
Case	Robot identifier	Objective
L10	everyday_robot_09	Move a box and mug to access a book, deliver the book, and restore the displaced objects.

Geometric precision.. These tasks emphasize the orientation and clearance required for insertion, extraction, hooking, and rotation. Neighboring objects and constrained approach directions make the path to the goal part of the task. The washer task additionally requires acquiring a thin object from a flat surface and maintaining the requested orientation.

Case	Robot identifier	Objective
P1	panda	Seat round, square, and long pegs in their matching sockets without moving the board.
P2	everyday_robot_09	Insert a key, turn it clockwise by a quarter turn, and withdraw it onto the mat.
P3	panda	Align and seat a keyed connector while preserving neighboring connectors.
P4	everyday_robot_03	Align an asymmetric lid with its corner tab and seat the entire rim.
P5	panda	Guide a hook through a mug handle and release it in stable suspension.
P6	everyday_robot_09	Extract a spatula through a cluttered holder without disturbing adjacent utensils.
P7	panda	Insert an upright book into a narrow gap under a low shelf.
P8	everyday_robot_01	Align, lift, and rotate a portafilter into its locked position.
P9	everyday_robot_09	Align a dial with a narrow target tick while preserving adjacent controls.
P10	panda	Acquire a thin washer, orient its face upright, and hold it steadily above the mat.

Dynamic interaction.. These tasks require reasoning about motion, momentum, balance, or spring forces. Some impose a single intervention, an exact transferred quantity, or a completion deadline. The agent must anticipate the evolution of the scene after contact and use feedback to regulate motion while preserving protected objects or carried contents.

Case	Robot identifier	Objective
D1	panda	Stop a rolling orange on a safe mat before it leaves the counter.
D2	panda	Position a catcher beneath a ball leaving a ramp and retain it after impact.
D3	panda	Push a puck once so that it coasts to rest inside the scoring region.
D4	everyday_robot_01	Transport a loaded tray while keeping cups upright and contents contained.
D5	panda	Push a tall bottle around a guide block without tipping it.
D6	panda	Close a loaded drawer within five seconds without disturbing its contents.
D7	everyday_robot_03	Pour exactly eight of twelve beads and retain the remaining four in the source cup.
D8	panda	Guide a spring-closing door shut without slamming or rebound.
D9	everyday_robot_01	Tilt a maze to guide a ball through two gates while avoiding traps and direct contact.
D10	fetch	Dock a wheeled cart accurately while keeping its load upright and contained.

EP-100 Index

All launcher entries use loaded initialization. Each EP-100 task carries a suite index such as Launch-01, a task type, and a split. Table 18 defines the workload scores that rank tasks within each type, and Table 19 lists the failure categories used in reporting. Table 20 gives the complete index, in which a check mark denotes membership in EP-20.

Index	Type	Task	Split	EP-20	Environment identifier
Launch-01	Aim	Two-range ammunition planning	Hard		marble/dual-panda/A001
Launch-02	Aim	Single-target warm-up	Easy	✓	marble/dual-panda/A007
Launch-03	Aim	Long-range energy budget	Easy		marble/dual-panda/A008
Launch-04	Aim	Elevated-platform shot	Easy		marble/dual-panda/A009
Launch-05	Aim	Toppling timber gate	Easy		marble/dual-panda/A010
Launch-06	Aim	Three-target fan	Hard		marble/dual-panda/A075
Launch-07	Aim	Left-low and right-high fork	Hard		marble/dual-panda/A031
Launch-08	Causal	Orchard rolling bodies	Easy	✓	marble/dual-panda/A017
Launch-09	Causal	One beam, two targets	Easy		marble/dual-panda/A004
Index	Type	Task	Split	EP-20	Environment identifier
Launch-10	Causal	Light roof on three supports	Easy		marble/dual-panda/A039
Launch-11	Causal	Gravity-driven stone at a low fort	Easy		marble/dual-panda/A044
Launch-12	Causal	Whole arch and separate crown stone	Easy		marble/dual-panda/A071
Launch-13	Causal	Branching dominoes	Hard		marble/dual-panda/A006
Launch-14	Causal	Indirect clearance with two bridges	Hard		marble/dual-panda/A015
Launch-15	Causal	Breakable window in a fixed frame	Easy		marble/dual-panda/A029
Launch-16	Causal	Staggered twin towers	Hard		marble/dual-panda/A034
Launch-17	Causal	Two-ball relay	Hard		marble/dual-panda/A055
Launch-18	Causal	Upper-to-lower beam transfer	Hard		marble/dual-panda/A063
Launch-19	Causal	Movable flap at a narrow throat	Hard	✓	marble/dual-panda/A072
Launch-20	Causal	Near target before collapse	Hard		marble/dual-panda/A073
Launch-21	Glass	Rigid glass village	Easy		marble/dual-panda/A018
Launch-22	Glass	Two glass layers	Easy	✓	marble/dual-panda/A025
Launch-23	Glass	Low lintel, high target	Easy		marble/dual-panda/A064
Launch-24	Glass	One-shot glass breach	Hard		marble/dual-panda/A027
Launch-25	Glass	Glass-supported lower story	Hard		marble/dual-panda/A035
Launch-26	Rules	Far target before near target	Easy		marble/dual-panda/A026
Launch-27	Rules	Protect the garden neighbor	Easy		marble/dual-panda/A028
Launch-28	Rules	Fire again with damage retained	Easy		marble/dual-panda/A030
Launch-29	Rules	Attic target above a protected floor	Hard		marble/dual-panda/A036
Launch-30	Rules	Keep the lower chamber safe	Hard		marble/dual-panda/A047
Launch-31	Rules	Fragment-risk well	Hard	✓	marble/dual-panda/A068
Launch-32	Rules	Numbered fan courtyard	Easy		marble/dual-panda/A091
Launch-33	Rules	Upper floor before lower floor	Hard		marble/dual-panda/A092
Launch-34	Rules	Material-ordered courtyard	Hard	✓	marble/dual-panda/A093
Tower-01	Plan	Top-ring transfer	Easy		hanoi_panda/panda/H01
Tower-02	Plan	Local two-ring transfer	Easy		hanoi_panda/panda/H02
Tower-03	Plan	Reassemble a distributed state	Easy		hanoi_panda/panda/H03
Tower-04	Plan	Long five-ring rearrangement	Hard	✓	hanoi_panda/panda/H04
Tower-05	Plan	Long combined challenge	Hard		hanoi_panda/panda/H40
Tower-06	Physics	Triangular transfer	Easy	✓	hanoi_panda/panda/H06
Tower-07	Physics	Low-to-high platform	Easy		hanoi_panda/panda/H09
Tower-08	Physics	Medium-hole replacement	Hard		hanoi_panda/panda/H14
Tower-09	Physics	Light-ring lift	Easy		hanoi_panda/panda/H17
Tower-10	Physics	Standard-friction transfer	Hard		hanoi_panda/panda/H22
Tower-11	Physics	Single tilted handle	Easy		hanoi_panda/panda/H25
Tower-12	Physics	Long and short peg pair	Hard		hanoi_panda/panda/H34
Tower-13	Physics	Offset vertical barrier	Hard		hanoi_panda/panda/H31
Tower-14	Dex	Top-ring transfer	Easy		hanoi_floating/floating-sharp/H01
Tower-15	Dex	Local two-ring transfer	Hard	✓	hanoi_floating/floating-sharp/H02
Tower-16	Dex	Low-to-high platform	Hard		hanoi_floating/floating-sharp/H09

Suite	Workload score
Launch	Ten times the native tier, plus the loaded-mode difficulty level D1–D5 recorded in the task atlas, plus the number of required targets
Tower	Twice the depth index, which is 0 to 3 for 1, 3, 9, and 31 moves, plus 1 for a physical theme, plus 2 for FR3–Sharpa or 3 for floating Sharpa
Lock	Subtask weight of 0 for the key, 2 for the key and bar, and 4 for the full lock, plus 1 for a hard pose, plus 0.5 for assembly, plus 2 for two floating Sharpa hands
Cube	Scramble or transition length in quarter turns, plus 3 for a Sharpa embodiment
Home	Number of objects for bookshelf tasks, and number of ordered milestones and final predicates for ManiSkill tasks

Index	Type	Task	Split	EP-20	Environment identifier
Tower-17	Dex	Single tilted handle	Hard		hanoi_floating/floating-sharpa/H25
Tower-18	Dex	Top-ring transfer	Easy	✓	hanoi_sharpa/fr3-sharpa/H01
Tower-19	Dex	Local two-ring transfer	Hard		hanoi_sharpa/fr3-sharpa/H02
Tower-20	Dex	Low-to-high platform	Easy		hanoi_sharpa/fr3-sharpa/H09
Tower-21	Dex	Single tilted handle	Easy		hanoi_sharpa/fr3-sharpa/H25
Lock-01	Assemble	Assemble full lock	Hard		kongming_lock/dual-panda/A01
Lock-02	Assemble	Assemble key	Easy		kongming_lock/dual-panda/A06
Lock-03	Assemble	Assemble key + bar	Easy	✓	kongming_lock/dual-panda/A11
Lock-04	Assemble	Assemble full lock	Hard		kongming_lock/dual-panda/A04
Lock-05	Assemble	Assemble key	Easy		kongming_lock/dual-panda/A09
Lock-06	Assemble	Assemble key + bar	Hard		kongming_lock/dual-panda/A14
Lock-07	Disassemble	Disassemble full lock	Hard	✓	kongming_lock/dual-panda/D01
Lock-08	Disassemble	Disassemble key	Easy		kongming_lock/dual-panda/D06
Lock-09	Disassemble	Disassemble key + bar	Easy		kongming_lock/dual-panda/D11
Lock-10	Disassemble	Disassemble full lock	Hard		kongming_lock/dual-panda/D04
Lock-11	Disassemble	Disassemble key	Easy		kongming_lock/dual-panda/D09
Lock-12	Disassemble	Disassemble key + bar	Hard		kongming_lock/dual-panda/D14
Lock-13	Dex	Assemble full lock	Hard		kongming_lock/dual-floating-sharpa/A01
Lock-14	Dex	Assemble key	Easy	✓	kongming_lock/dual-floating-sharpa/A06
Lock-15	Dex	Disassemble full lock	Hard		kongming_lock/dual-floating-sharpa/D01
Lock-16	Dex	Disassemble key	Easy		kongming_lock/dual-floating-sharpa/D06
Cube-01	Rubik	Solve scramble C01	Easy		rubiks_panda/panda/C01
Cube-02	Rubik	Solve scramble C02	Easy		rubiks_panda/panda/C02
Cube-03	Rubik	Solve scramble C03	Easy		rubiks_panda/panda/C03
Cube-04	Rubik	Solve scramble C01	Easy		rubiks_sharpa/floating-sharpa/C01
Cube-05	Rubik	Solve scramble C02	Hard		rubiks_sharpa/floating-sharpa/C02
Cube-06	Rubik	Solve scramble C11	Hard	✓	rubiks_panda/panda/C11
Cube-07	Rubik	Solve scramble C12	Hard		rubiks_panda/panda/C12
Cube-08	Rubik	Solve scramble C21	Hard		rubiks_panda/panda/C21
Cube-09	Pocket	Clockwise top turn	Easy	✓	pocket_cube/aloha/C01
Cube-10	Pocket	Clockwise top turn	Hard		pocket_cube/dual-sharpa/C01
Cube-11	Pocket	Half-turn in two stages	Easy		pocket_cube/aloha/C08
Cube-12	Pocket	Half-turn in two stages	Hard	✓	pocket_cube/dual-sharpa/C08
Home-01	Tidy	Three upright books onto the middle shelf	Easy		bookshelf/dual-panda/BooksTableD1
Home-02	Tidy	Unpack a neatly packed crate	Easy		bookshelf/dual-panda/ShelfCrated1
Home-03	Tidy	One book on the wrong layer of a tidy shelf	Hard	✓	bookshelf/dual-panda/ShelfCrateFixD1

Index	Type	Task	Split	EP-20	Environment identifier
Home-04	Tidy	Books, a trophy and one box	Hard		bookshelf/dual-panda/ShelfTableD1
Home-05	Tidy	Three books and two pieces of trash	Hard		bookshelf/dual-panda/TableCleanD1
Home-06	Long	Prepare a shared writing desk	Easy	✓	maniskill_long_horizon/fetch/L1
Home-07	Long	Prepare a tea station for two	Easy		maniskill_long_horizon/fetch/L3
Home-08	Long	Unload cups and cutlery from the dishwasher	Hard		maniskill_long_horizon/fetch/L5
Home-09	Long	Pack a picnic basket	Hard		maniskill_long_horizon/fetch/L6
Home-10	Precise	Insert three matching pegs	Hard		maniskill_geometry/panda/P1
Home-11	Precise	Seat a keyed electronics connector	Easy		maniskill_geometry/panda/P3
Home-12	Precise	Hang a mug by its handle	Hard	✓	maniskill_geometry/panda/P5
Home-13	Precise	Insert a book into a narrow shelf gap	Easy		maniskill_geometry/panda/P7
Home-14	Dynamic	Stop a rolling orange	Easy	✓	maniskill_dynamics/panda/D1
Home-15	Dynamic	Push a tall bottle without tipping	Hard		maniskill_dynamics/panda/D5
Home-16	Dynamic	Close a loaded drawer gently	Hard		maniskill_dynamics/panda/D6
Home-17	Dynamic	Slide a puck into a scoring zone	Easy		maniskill_dynamics/panda/D3

EP-Easy Index

EP-Easy contains the lower-scoring half of every task type. Table 21 lists its 50 tasks by type, using the suite index numbers of Table 20.

Type	Tasks	Index numbers
Launch-Aim	4	02, 03, 04, 05
Launch-Causal	6	08, 09, 10, 11, 12, 15
Launch-Glass	3	21, 22, 23
Launch-Rules	4	26, 27, 28, 32
Tower-Plan	3	01, 02, 03
Tower-Physics	4	06, 07, 09, 11
Tower-Dex	4	14, 18, 20, 21
Lock-Assemble	3	02, 03, 05
Lock-Disassemble	3	08, 09, 11
Lock-Dex	2	14, 16
Cube-Rubik	4	01, 02, 03, 04
Cube-Pocket	2	09, 11
Home-Tidy	2	01, 02
Home-Long	2	06, 07
Home-Precise	2	11, 13
Home-Dynamic	2	14, 17

EP-Hard Index

EP-Hard contains the higher-scoring half of every task type. Table 22 lists its 50 tasks by type, using the suite index numbers of Table 20.

Type	Tasks	Index numbers
Launch-Aim	3	01, 06, 07
Launch-Causal	7	13, 14, 16, 17, 18, 19, 20
Launch-Glass	2	24, 25
Launch-Rules	5	29, 30, 31, 33, 34
Tower-Plan	2	04, 05
Tower-Physics	4	08, 10, 12, 13
Type	Tasks	Index numbers
Tower-Dex	4	15, 16, 17, 19
Lock-Assemble	3	01, 04, 06
Lock-Disassemble	3	07, 10, 12
Lock-Dex	2	13, 15
Cube-Rubik	4	05, 06, 07, 08
Cube-Pocket	2	10, 12
Home-Tidy	3	03, 04, 05
Home-Long	2	08, 09
Home-Precise	2	10, 12
Home-Dynamic	2	15, 16

EP-20 Index

Launcher entries use loaded initialization only. EP-20 takes ten tasks from EP-Easy and ten from EP-Hard. Table 23 lists them with their splits, embodiments, and native levels.

Index	Task	Type	Split	Embodiment	Native level
Launch-02	Single-target warm-up	Aim	Easy	Dual Panda	tier 1
Launch-08	Orchard rolling bodies	Causal	Easy	Dual Panda	tier 2
Launch-22	Two glass layers	Glass	Easy	Dual Panda	tier 3
Launch-19	Movable flap at a narrow throat	Causal	Hard	Dual Panda	tier 4
Launch-31	Fragment-risk well	Rules	Hard	Dual Panda	tier 5
Launch-34	Material-ordered courtyard	Rules	Hard	Dual Panda	tier 5
Tower-06	Triangular transfer	Physics	Easy	Panda	L2, 3 moves
Tower-18	Top-ring transfer	Dex	Easy	FR3–Sharpa	L1, 1 move
Tower-04	Long five-ring rearrangement	Plan	Hard	Panda	L4, 31 moves
Tower-15	Local two-ring transfer	Dex	Hard	Floating Sharpa	L2, 3 moves
Lock-03	Assemble key + bar	Assemble	Easy	Dual Panda	key + bar, easy pose
Lock-14	Assemble key	Dex	Easy	Dual floating Sharpa	key, easy pose
Lock-07	Disassemble full lock	Disassemble	Hard	Dual Panda	full lock, easy pose
Cube-09	Clockwise top turn	Pocket	Easy	ALOHA	1 quarter turn
Cube-06	Solve scramble C11	Rubik	Hard	Panda	14 quarter turns
Cube-12	Half-turn in two stages	Pocket	Hard	Dual floating Sharpa	2 quarter turns
Home-06	Prepare a shared writing desk	Long	Easy	Fetch	13 conditions
Home-14	Stop a rolling orange	Dynamic	Easy	Panda	3 conditions
Home-03	One book on the wrong layer of a tidy shelf	Tidy	Hard	Dual Panda	D1
Home-12	Hang a mug by its handle	Precise	Hard	Panda	4 conditions