WorksComputer Science
A Controlled Study of Models, Skills, Feedback, and Curriculum in Agentic Robotics
Abstract
Agentic robotics asks coding agents to implement and iteratively improve robot policies from simulated and real-world physical interactions. We study the self-learning loop of agentic robotics through a controlled simulation study spanning nine manipulation tasks, 631 different experiment configurations, and 28K coding agent rollouts, systematically varying model capability, skill priors, feedback structure, and curriculum. The experiment data suggests that the bottleneck is often not the access to sufficient information, but interpreting it into the right policy update. Across these factors, a successful self-learning loop depends on how effectively agents attribute failures, reason task-relevant spatial and geometric structure, and selectively restructure code to improve policies.
Introduction
Recent coding agents, such as GPT Astra [30] and Claude Fable [2], have become increasingly capable of generating and iteratively repairing executable programs, motivating a growing class of Agentic Robotics (AR) systems in which LLM- and VLM-based agents write code that composes perception, planning, and control skills into robot policies [21, 36, 11, 8]. However, self-improvement in robotics differs fundamentally from software debugging. In software, failures, such as compiler errors, are often exposed through symbolic feedback that is already closely aligned with the program; but in robotics, the evidence to improve a policy is instead embedded in physical interactions: object geometry, spatial relationships, contact, motion, and changes in the scene over time. As illustrated in Figure 1, a failed placement can be attributed to an earlier incompatible grasp and a collision requires reasoning about the geometry of an object after it becomes part of the moving robot. In this paper, we ask: what determines whether an agent can interpret physical interaction and convert it into effective robot-policy improvement?

Figure 1. Representative failure modes in Agentic Robotics. Left: Manipulation planning fails to select a grasp and approach strategy compatible with the downstream placement. Center: Motion planning fails to jointly reason about held-object geometry, obstacles, and clearance to produce a collision-free trajectory. Right: Deformable manipulation requires reasoning over object configuration and contact that is difficult to capture through local motion decisions.
We design a controlled simulation study of agentic robot self-learning across five contact-rich VABAR manipulation tasks Xie et al. [40] and four longer-horizon MimicGen tasks Mandlekar et al. [26]. Holding the overall self-learning harness fixed, we systematically vary four components of the learning loop: model capability, skill prior, feedback structure, and curriculum, as illustrated in Figure 2. Across these studies, we find that self-learning is limited less by the availability of useful information than by the agent’s ability to turn the information into effective policy updates. More execution feedback is not necessarily more useful: feedback is most effective when aligned with semantically meaningful policy decisions rather than reported at the finest temporal resolution. Similarly, world-frame states contain all the information needed for manipulation, yet agents often fail to recover task-relevant object-relative and functional relationships unless these are made explicit. Even when agents identify local geometric errors, they often struggle to translate them into coherent collision-aware motion plans. We find that this limitation can also affect curriculum design: agents tend to preserve and patch policy structure inherited from earlier stages rather than fundamentally reorganize it, so privileged learning can either provide a useful initialization or constrain later learning when the inherited policy depends on privileged precision or brittle manipulation strategies. Our results further suggest that progress in agentic robot self-learning comes from the better structures of skills, feedback, and state representations and the learned physical reasoning ability from the underlying models.

Figure 2. Overview of our controlled study of agentic robot self-learning of model, skill, feedback, and curriculum design. A coding agent initializes a policy , executes the current policy in simulation, receives feedback , and iteratively edits its policy as . We evaluate zero-shot and self-learning performance, and final task success across VABAR and MimicGen manipulation tasks.
This paper makes three contributions: (1) A controlled study design for Agentic Robotics that isolates four important components of simulation-guided self-learning, (2) Experiment data from 631 implemented experiments with 28 thousand simulation rollouts on Agentic robotics policies, (3) Six findings on model capability, skill priors, feedback structure, and curriculum.
Related Work
Agentic Robotics. Foundation models have been used to compose robot skills and generate executable policies through language-model planning and code generation [20, 1, 36]. More recent work integrates foundation models with explicit spatial representations, geometric constraints, and task-and-motion planning to support open-vocabulary and multi-step manipulation [15, 17, 34, 29]. Other approaches close the loop by using simulation or real-world execution feedback to revise policies, acquire skills, and improve task performance over repeated interactions [11, 8, 38, 24]. Rather than introducing a new agent architecture, we study how individual components of this self-learning loop affect performance.
Controlled Evaluation of Agentic Robotics. Robot-learning benchmarks increasingly use standardized task suites to evaluate generalization, long-horizon reasoning, and manipulation under controlled variation [13, 19, 28, 22, 42]. More recent benchmarks extend controlled evaluation to foundation-model agents by measuring interactive decision making, embodied reasoning, perception, and action across diverse environments [23, 27, 33]. Robot-specific evaluations further study robot-centric perception and decision making, coding-agent behavior under different abstraction and interaction settings, and repeated manipulation under controlled task variation ([39, 11, 40]). Unlike prior agentic robotics benchmarks that primarily vary tasks, environments, or complete agent systems, we hold the self-learning harness fixed and independently vary model capability, skill priors, feedback structure, and curriculum to isolate how each factor affects initial policy generation and subsequent self-improvement.
Skills, Feedback, and Curriculum for Self-Learning. Prior work has shown that reusable skills and execution feedback can improve embodied agent learning ([1, 37, 15, 11]). Skill libraries provide reusable action abstractions that constrain generation and support compositional behavior, while systems such as Voyager jointly combine an executable skill library, environment feedback, and an automatic curriculum for continual improvement ([37]). More broadly, language agents can use environment observations, linguistic critiques, or tool-derived feedback to revise plans and outputs across interaction or repeated attempts ([41, 35, 25, 12]). However, prior work also shows that self-correction can be unreliable without sufficiently informative external feedback ([14]). Curriculum learning instead structures the progression of experience ([6]), while privileged-information methods use state available during training to simplify learning before transferring to image-based or otherwise non-privileged observations ([32, 7]). These factors are often introduced jointly within a single system. We instead study them separately by varying the skill prior, temporal and spatial feedback representation, and privileged-to-vision curriculum.
Problem Formulation
Agentic robotics for manipulation. A robot manipulation task class can be defined as a tuple : a natural-language instruction , a fixed workcell with a known robot and sensor configuration , a bounded set of objects with known models, a state space of robot configurations and object poses, a belief space giving the distribution over initial object poses, and a reward , here the per-object success indicator . A task instance is drawn from and . The agent produces a policy represented as executable source code composed from reusable robot skills. Following the graph-based policy representation of PaG ([8]), we view as a computation graph whose nodes invoke skills and whose edges define their execution and control flow. The policy is constructed over a skill library , where contains basic robot and sensor primitives; contains reusable general-purpose skills; contains task-specific skills provided by a human expert; and contains task-specific skills synthesized by the agent. Given , an LLM agent constructs by selecting, parameterizing, and composing available skills, while implementing new skills when the existing library is insufficient.
On a task instance , the resulting policy maps observations to robot actions in closed loop, and the resulting episode is evaluated by . Given a task class and skill library , the agent seeks to construct a policy that succeeds across the belief space: , using only , , and the outcomes of its own policies. Policy improvement may therefore modify both the composition of the policy graph and the implementation of its constituent skills.
Self-learning loop. Unlike reinforcement learning, feedback in the self-learning loop is not restricted to a scalar reward. At iteration , the current policy runs in simulation on the training instances . A feedback function turns these runs into a textual (and optionally visual) report . The agent then edits its own policy, conditioned on the report and on its edit history . The loop runs for iterations. Success is reported on a held-out set of instances for the initial policy (zero-shot, ) and for the iterate with the best validation score, with . Held-out success at every iteration gives a learning curve for each setting and task. Each study below varies one component of this loop: the agent , the skills , the feedback , or the curriculum over training instances.
Assumptions and Scope. We assume access to a reliable physics simulator. Training and evaluation remain within the same manipulation task class, with a fixed robot and sensor configuration and bounded scene variation. We evaluate generalization to held-out instances of these task classes and do not study sim-to-real transfer or generalization to new embodiments.
Study Design
Task Setup. We evaluate five simulated manipulation tasks from the Variational Automation Benchmark for Agentic Robotics (VABAR) Xie et al. [40]: (1) Tool Hanging, (2) Syringe Packing, (3) Gear Meshing, (4) USB-C Insertion, and (5) Cable Routing. For the curriculum study, we evaluate four multi-step tasks from MimicGen Mandlekar et al. [26] using the Medium (D1) layouts: (1) Kitchen, (2) Coffee, (3) Threading, and (4) Mug Cleanup (Figure 3).

Figure 3. Five VABAR tasks (Left) and four MimicGen tasks (Right) used for this study.
Skill Setup. We use open-robot-skills provided by Chen et al. [8] as task-agnostic skill library . The library contains a set of 30 skills in the Agent Skills format, each with a natural-language SKILL.md describing when the skill applies and conditions. The library covers perception, grasping and motion planning primitives. The full list of skills can be found in Appendix § 10.
Metrics and Evaluation scale. In evaluation, every agent setting runs the same closed-loop episodes on the same training, validation and testing task instances with repeated runs. We report the success rate of the hold-out testing task instance that the scene configuration is different from the training and testing task instances. We run each of the ten agent settings on the five tasks with five independent runs per task with two rounds of feedback. In each round the current program runs on 4 training, 5 validation and 10 held-out instances. We use full spatial and temporal feedback and no curriculum for the skill composition study and spatial temporal skill expansion study. We use Opus 5 xhigh as default for the feedback and skill-library studies. Including pilot studies, we use 17,900 GPU-hours on NVIDIA RTX PRO 6000 Blackwell GPUs and 2,700 GPU-hours on NVIDIA A100 with in total 27,600 coding agent rollouts with an estimation of $70,000 computing cost. The study takes 2,500 coding agents editing sessions with $36,500 for LLM inference cost.
Experiments
Agent Model
We study how the choice of coding model affects both initial policy generation and subsequent self-learning. Each agent is given the shared general skill library, and the task-specific expert skills provided with VABAR (Xie et al. [40]). The skill library is sufficiently expressive for successful task execution. Each agent generates a zero-shot policy (@0) and then revises its code from execution feedback for two rounds (@1 and @2). We evaluate Claude Opus 5 [3], Claude Opus 5.5 [4], Claude Sonnet 5 [5], Claude Fable 5.1 [2], GPT-6 Astra [30], GPT-6 Sol [31], DeepSeek V4 [10], and Kimi K3 [18]. Table 1 reports held-out success rate after each round, and Figure 5 analyzes how models construct their initial policies from provided and generated skills.
| Model | Overall @0 | Overall @1 | Overall @2 | Overall 1 | Overall 2 | Tool Hanging @0 | Tool Hanging @1 | Tool Hanging @2 | Syringe Packing @0 | Syringe Packing @1 | Syringe Packing @2 | Gear Meshing @0 | Gear Meshing @1 | Gear Meshing @2 | USB-C Insertion @0 | USB-C Insertion @1 | USB-C Insertion @2 | Cable Routing @0 | Cable Routing @1 | Cable Routing @2 |
| Human | 82 | – | 87 | 90 | 100 | 87 | 47 | |||||||||||||
| Opus 5, xhigh | 45 | 64 | 74 | +19 | +10 | 22 | 62 | 60 | 28 | 38 | 35 | 100 | 100 | 100 | 0 | 50 | 100 | 73 | 72 | 73 |
| Opus 5, high | 41 | 62 | 81 | +21 | +19 | 25 | 72 | 97 | 40 | 30 | 45 | 67 | 100 | 100 | 0 | 33 | 87 | 75 | 76 | 76 |
| Opus 5, low | 57 | 67 | 74 | +10 | +7 | 0 | 30 | 57 | 51 | 50 | 60 | 100 | 100 | 100 | 60 | 80 | 80 | 73 | 76 | 76 |
| DeepSeek V4 | 38 | 43 | 44 | +5 | +1 | 0 | 0 | 0 | 20 | 29 | 30 | 100 | 100 | 100 | 0 | 10 | 12 | 70 | 77 | 77 |
| Sonnet 5 | 45 | 46 | 46 | +1 | 0 | 0 | 0 | 0 | 32 | 34 | 34 | 100 | 100 | 100 | 25 | 25 | 25 | 67 | 72 | 72 |
| Kimi K3 | 39 | 43 | 46 | +4 | +3 | 0 | 0 | 2 | 30 | 47 | 55 | 97 | 97 | 97 | 0 | 0 | 0 | 69 | 71 | 73 |
| Fable 5.1 | 44 | 57 | 64 | +13 | +7 | 24 | 32 | 32 | 49 | 49 | 56 | 67 | 100 | 100 | 0 | 25 | 50 | 79 | 79 | 79 |
| GPT-6 Astra | 43 | 47 | 48 | +4 | +1 | 0 | 0 | 0 | 20 | 25 | 30 | 100 | 100 | 100 | 15 | 30 | 30 | 80 | 80 | 80 |
| Opus 5.5 | 67 | 74 | 77 | +7 | +3 | 25 | 40 | 55 | 46 | 50 | 50 | 100 | 100 | 100 | 85 | 100 | 100 | 79 | 79 | 79 |
| GPT-6 Sol | 33 | 41 | 53 | +8 | +12 | 0 | 0 | 0 | 30 | 30 | 30 | 50 | 50 | 50 | 0 | 40 | 100 | 83 | 83 | 83 |
Table 1. Self-learning on VABAR by model. Held-out per-object success (%) reached within 0, 1, and 2 rounds of feedback (@) reported by mean over the runs scored at each round. denotes the gain in overall success from the -th round of feedback, i.e. @ − @(−1).
Figure 5. Zero-shot policy composition and attribution by code source.
Table 1 shows zero-shot performance is not sufficient to evaluate if the model can self-improve. Several models begin from nearly identical initial success rates but diverge after receiving the same execution feedback. For example, Opus 5 xhigh, Fable 5.1, GPT-6 Astra, and Sonnet 5 start at 43–45% overall success, yet after two rounds Opus 5 xhigh and Fable 5.1 improve to 74% and 64%, respectively, while GPT-6 Astra and Sonnet 5 remain at 48% and 46%. The same distinction appears across individual tasks: models that initially fail Tool Hanging or USB-C Insertion can sometimes repair these policies substantially, whereas others remain near their initial performance despite observing the same failures. Opus 5.5 is strong at both stages, increasing from 67% zero-shot to 77% after feedback. These results suggest that generating a plausible initial manipulation policy and using execution evidence to diagnose and repair that policy are distinct model capabilities (Figure 4).
Figure 4. Outcomes of edits targeting a reported failing skill.
Finding 1. Differences in model capability are less apparent at policy initialization but become pronounced during self-learning. Models with similar zero-shot performance can diverge substantially after receiving the same execution feedback, suggesting that policy generation and policy repair are distinct capabilities.
Skill
We next study how agents use the skills they are given. The skill library provides an important prior: the expert skills encode task-specific manipulation strategies, while the general library provides reusable perception, grasping, and motion-planning routines. Such a library could help the agent by providing useful building blocks, or could instead constrain it to behaviors already represented in the library. To distinguish these cases, we trace the source of each line in the zero-shot policy (@0). After normalizing each line, we match it verbatim against the expert task-specific skills, the general-library skills also used by the human policy, and the remaining general-library skills; unmatched lines are attributed to new code written by the agent.
Figure 5 reports the mean number of lines from each source, averaged over runs per task and then over tasks, and relates it to success after feedback in Table 1. As shown in Figure 5, models draw on the same skills in different ways. Models that improve most with feedback also write a larger share of their policy as new task-specific code: new code makes up 24% of the lines for Opus 5 xhigh, 18% for Opus 5.5, and 12% for Fable 5.1. They use the expert skills as a basis but add or modify task-specific logic where the provided implementations are insufficient. In contrast, Sonnet 5, DeepSeek V4, Kimi K3, and GPT-6 Astra write only 4–7% new code and rely more heavily on behavior already represented in the expert and general skills. These models also improve the least, ending at 39–46% overall success after two rounds. Using more of the library is not the same as adapting beyond it: Kimi K3 uses the largest share of general skills not used by the human policy (13% of its lines), yet still writes little new code and gains only 7 points. Since several of these models start from similar zero-shot success, the difference is not simply in the quality of the first policy, but in whether the agent can modify and extend the provided skills when the existing behavior is insufficient.
Finding 2. Expert skills are a boundary for weaker models but a scaffold for stronger ones. Weaker models largely compose behaviors already represented in the skill library, while stronger models selectively reuse and depart from human-provided skills to create new task-specific approaches.
Feedback
Temporal Failure Attribution. We next study whether self-learning agents can attribute an observed robot failure to the earlier policy decision that caused it. A manipulation episode consists of multiple perception, planning, and execution steps, so the point at which a failure becomes observable need not be the point at which the underlying error was introduced. As shown in Figure 6, the policy skill selected for repair can occur throughout the preceding trajectory, sometimes substantially before the point at which the episode finally halts. We therefore ask whether exposing intermediate execution outcomes helps the agent identify which earlier policy step should be revised.
Figure 6. Number of skills traced back from the observed failing skill before each post-failure policy edit. Finer-grained and semantic feedback leads agents to trace failures further upstream.
Table 2 shows that the localized temporal feedback depends on where the underlying error occurs relative to where the failure becomes observable. Reporting every motion primitive can expose an earlier error once it manifests as incorrect motion, reaching 45% on Tool Hanging and 80% on Gear Meshing. However, this finer-grained feedback does not expose non-motion operations such as perception. The useful unit of feedback is therefore the intermediate outcome associated with a meaningful policy decision, rather than the finest available time step. In Tool Hanging, the per-episode agent repeatedly attributed failure to the final reaching stage and modified that part of the policy, reaching only 7% held-out success. The observed miss, however, originated from an earlier incorrect estimate of the wrench ring. Skill-level feedback records the state changes and return values around each semantic step, exposing this earlier error and increasing held-out success to 90%. Gear Meshing shows the same pattern, improving from 10% to 70%. Conversely, localized temporal feedback is less useful when the cause of failure is already apparent in the final state. In Syringe Packing, a syringe left outside or across the container opening directly exposes the relevant geometric failure, and per-episode feedback reaches 90% held-out success.
| Feedback | Overall | Tool Hanging | Syringe Packing | Gear Meshing | USB-C Insertion | Cable Routing |
| Initial policy | 3 | 0 | 0 | 0 | 0 | 17 |
| Per episode | 25 | 7 | 90 | 10 | 0 | 17 |
| Per motion | 36 | 45 | 55 | 80 | 0 | 0 |
| Per skill | 52 | 90 | 50 | 70 | 20 | 27 |
Table 2. Effect of temporal feedback structure on held-out success (%). Per motion reports the outcome of each robot motion that is not aligned with the policy’s semantic skills, Per skill reports the outcome of each semantic skills.
Finding 3. Feedback can be more useful when reported at the level of semantically meaningful policy skills, not simply at the finest temporal resolution. In multi-stage manipulation, the observed failure can occur well after the decision that caused it. Skill-level feedback exposes the intermediate outcomes of meaningful policy decisions and improves failure attribution, while finer motion-level feedback misses non-motion decisions such as perception.
Spatial and Geometric Representation We next study whether the representation of execution state affects an agent’s ability to reason about spatial failures. A conventional world-frame representation describes where objects are, but leaves the agent to reconstruct the object-relative quantities needed for manipulation. As suggests in Table 3, the world-frame state is sufficient in principle to recover the manipulation-relevant quantities, the agent often fails to reliably compose the required coordinate transformations and connect them to a policy decision. Object-frame feedback makes these same relationships explicit, such as grasp offsets and relative orientations, and substantially improves performance on Tool Hanging and, to a lesser extent, Gear Meshing. This suggests that information completeness alone is insufficient, and agents can possess the necessary coordinates yet still fail to derive the spatial relationships needed for manipulation. The same limitation extends beyond coordinate frames to object geometry itself. Providing accurate object state and geometry does not mean the agent understands how that geometry constrains action, because inferring the affordance features of the wrench and hanger from whole-object poses is challenging for agents. Part annotations further improve feedback by exposing the functional structures that matter for interaction. Part-annotated feedback reaches 52% overall success, compared with 40% for object frames and 30% for world-frame state and relations.
| Spatial representation | Overall | Tool Hanging | Syringe Packing | Gear Meshing | USB-C Insertion | Cable Routing |
| Outcome only | 19 | 15 | 45 | 0 | 0 | 33 |
| State + relations, world frame | 30 | 25 | 60 | 50 | 0 | 13 |
| State + relations, object frames | 40 | 63 | 50 | 60 | 0 | 27 |
| + part annotations | 52 | 90 | 50 | 70 | 20 | 27 |
Table 3. Effect of spatial representation on held-out success (%). World-frame and object-frame reports describe the same underlying scene state at different levels of relational organization, while part annotations add task-relevant functional geometry.
Finding 4. Recovering task-relevant spatial and geometric relationships from raw state information remains a bottleneck for agentic robotics. Although world-frame and object-frame feedback describe the same underlying scene, agents perform substantially better when object-relative geometry and functional spatial relationships are explicitly available, rather than requiring them to derive them.
Feedback Reasoning. We next study how self-learning agents translate spatial understanding into executable robot motion. We compare how human-written and agent-learnt policies structure perception, motion planning, and recovery. Figure 8 shows that human policies frequently make geometric structure explicit through repeated perception and re-registration, intermediate waypoints, and dedicated motion-planning steps. These intermediate poses encode geometric priors such as approaching from a safer direction, maintaining clearance, or moving through configurations from which a feasible motion is easier to find. In contrast, self-learning agents more often use verification, retry, and recovery around coarser manipulation steps. They can reuse motions containing useful geometric structure, but are less consistent at reconstructing why that structure is necessary when generating or modifying the policy themselves.
Figure 8. Policy structure by human(H), Astra(A), and Fable(F).
Syringe Packing provides a concrete example. The robot must carry a 150 mm syringe through a 55 mm opening while reasoning about the syringe as part of the moving system, surrounding obstacles, reorientation clearance, and insertion orientation (Fig-Figure 7 ure 11). The human policy represents these dependencies explicitly with a collision world, held-object registration, collision-aware waypoints, and orientation-constrained insertion. In contrast, none of the agent runs without expert skills independently constructed an equivalent collision representation, and 17 of 19 initial self-written policies carried or released the syringe flat over the opening. Feedback often allowed agents to recognize the resulting errors and eventually adopt heuristics such as reorienting above the lid and descending vertically, but they still tended to encode these fixes as clearance constants, staging heights, and retries rather than as a general collision-aware model.
Figure 7. Spatial representations of the feedback report on one failed. The same execution state expressed at different levels of spatial organization.

Figure 9. Human motion plan (blue) considers a collision free spot (B) to reorientate gripper, while agent plan (red) has limited room to reorientate.
Finding 5. Reasoning over collision geometry and spatial relationships remains a key bottleneck for agentic robot motion planning. Agents can recognize local geometric errors, but struggle to integrate object pose, held-object geometry, obstacles, and clearance into a coherent collision-aware motion plan.
Curriculum Design
We study a representative privileged-to-perception curriculum, a common design in reinforcement learning and used in prior works such as Pinto et al. [32]; Chen et al. [7]. As a focused case study of curriculum transfer in agentic robotics, the agent first learns with privileged simulation states with exact simulator geometry and is then transferred to non-privileged simulation states such as RGB-D perception. We evaluate this curriculum on four MimicGen tasks using their Medium (D1) layouts with 8 privileged and 8 perception iterations and Claude Fable 5.1. We also use three VABAR tasks: Tool Hanging (Level 2), Gear Meshing (Level 2), and USB-C Insertion (Level 1) with 5 privileged iterations and 5 perception iterations with Claude Opus 5. Figure 10 reports results on four MimicGen and three VABAR tasks. Exact task configurations, training and held-out layouts, iteration budgets, probe settings, and other protocol details are provided in Appendix J.
Figure 10. Held-out success (top) and stage reward (bottom) for the privileged-to-perception curriculum and perception-only control on MimicGen (left) and VABAR tasks (right), evaluated on 10 fixed held-out layouts per task. Lines show means over three seed replicates and bands show ±1 standard error. The dotted line marks transfer to perception; shading indicates the privileged phase, during which only the curriculum experiment has access to privileged geometry.
Figure 10 suggests that privileged-to-perception learning does not uniformly improve over learning directly from perception. Privileged information removes perception uncertainty during early learning, but, as suggested by Findings 4 and 5, accurate geometry alone does not guarantee that the agent discovers the right spatial representation or manipulation strategy. The privileged phase can learn useful task structure, as in Threading, Coffee, and Mug Cleanup, but it can also sometimes commit early to brittle or non-generalizable choices. For example, Kitchen develops a grasp strategy that depends on geometric precision unavailable after transfer, while Tool Hanging remains below 10% held-out success during privileged learning despite access to exact wrench geometry. Thus, privileged information can simplify the learning problem without necessarily producing a policy that remains appropriate under perception. We further visualize and analyze representative failure cases in Appendix J.2.
In addition, Table 4 further suggests that once a policy structure is established under privileged state, subsequent perception-based learning tends to preserve and patch that structure rather than fundamentally reorganize it (which corresponds to Finding 1). Across tasks, the privileged-to-perception policies change little in size after transfer, from 12.0 skills at the end of the privileged phase to 12.7 after perception learning on average, while the final perception-only policies contain 16.1 skills. The resulting path dependence can be beneficial when the early structure captures a manipulation strategy that remains valid under vision, but this can be harmful when the structure assumes geometric precision unavailable from perception. In such cases, the curriculum does not merely provide an initialization; it can constrain subsequent self-learning.
| Task | w/ curr. | w/o curr. | w/ curr. | w/o curr. |
| Tool Hanging | 19.3 | 28.0 | 16 | 63 |
| Gear Meshing | 18.3 | 21.7 | 33 | 40 |
| USB-C Insertion | 18.0 | 10.7 | 31 | 16 |
| Kitchen | 13.0 | 13.3 | 33 | 62 |
| Coffee | 5.0 | 11.7 | 76 | 64 |
| Threading | 8.7 | 18.3 | 86 | 2 |
| Mug Cleanup | 6.3 | 9.3 | 93 | 77 |
| Overall | 12.7 | 16.1 | 52 | 46 |
Table 4. Policy size (# Skills, final policy) and held-out success (mean over the last three iterations) with (w/ curr.) and without (w/o curr.) the privileged-to-perception curriculum; mean over three runs.
Finding 6. Privileged-to-perception curriculum learning can either accelerate or constrain subsequent perception-based learning. Privileged agents learn policies that rely on exact state, usually with less verification and recovery, and often retain this structure after transfer. This helps when the learned manipulation strategy remains valid, allowing perception learning to focus on perception errors, but hurts when the policy depends on state precision unavailable under perception or lacks mechanisms for handling perceptual uncertainty.
Discussion and Limitations
Our results return to the central question of this study: what determines whether an agent can interpret physical interaction and convert it into effective robot-policy improvement? Figure 11 provides a complementary view across the experiments. Only 7.3% of the human-solvable Level-1 test cases are solved by zero-shot policy generation alone, but the results suggest that improving agentic robotics requires the combination of iterative self-learning with richer simulation feedback, human task skills, and more powerful models. At the same time, this decomposition shows that 4.1% of test cases solved by a human-written policy under the same robot and simulation setup remain unsolved by the agents. Moreover, this study primarily considers Level-1 VABAR tasks with uncluttered environments and relatively simple motions. The remaining gap is concentrated in agents’ ability to interpret physical interaction into task-relevant reasoning and policy changes, particularly for spatial reasoning, manipulation planning, and collision-aware motion planning.
Figure 11. What closes the gap to human-written policies? For VABAR Level-1 test cases solved by the human policy with Opus 5, improvements come progressively with the earlier findings, while a small fraction unsolved.
Limitations. This study does not cover the full range of conditions that may affect agentic robot self-learning, such as prompting strategies and harness configurations. Given the large number of agent rollouts required for controlled evaluation, we leave a broader study of these factors to future work. Our model comparison is also necessarily a snapshot in time, as new agent models are released frequently; we plan to update the evaluation as new models become available. We focus on simulation to enable repeatable evaluation, leaving sim-to-real transfer and adaptation to real-world feedback as important next steps; real-to-sim toolkits such as RoboSimStudio [9] may make it easier to construct matching simulations of real tasks for such studies. Finally, we do not consider human-in-the-loop self-learning, where human feedback may complement autonomous simulation feedback during policy improvement.
Appendix
The appendix documents what is needed to read, check and reproduce the study, in three parts. Experimental setup (Appendices § 9–§ 15.4) specifies the tasks, how they are scored and how their instances are drawn; the skill library and what each experiment ships; the self-learning harness, its complete prompts and feedback reports, and its revert, stopping and selection rules; and the models, their inference settings and the compute and cost of the study. Additional results (Appendix § 17) locates where the programs fail. Curriculum study (Appendix § 19) details the privileged-to-perception experiments of § 5.4.
Experimental Setup
Tasks, Scoring and Instances
This appendix specifies the five VABAR tasks (Figure 3) at the level needed to reproduce a score: what each task demands (Appendix § 9.§ 1), how an episode becomes per-object success, the metric of every table (Appendix § 9.§ 2), and how the 4 training, 5 validation and 10 held-out instances of each task are drawn (Appendix § 9.§ 3). Appendix § 9.§ 4 then relates held-out success on each instance to the quantities it was drawn with.
Task Specifications
The five tasks are the Easy (Level-1) tier of VABAR Xie et al. [40]. Every task instance is one RoboSimStudio task document that, with its suite, fixes the robot, the objects, the start layout, a step cap, a goal predicate and a reward (stated, or the share of goal clauses met). Table 5 summarizes the five tasks.
| Tool Hanging | Syringe Packing | Gear Meshing | USB-C Insertion | Cable Routing | |
| Instruction | hang the wrench on the hook | pick up each syringe from the instrument tray, one at a time, and drop it through the hole in the top of the yellow sharps disposal box, until both are in the box | put the gear on its peg so it meshes with the pinion, then press the red button at the near end of the board to start it | insert the USB-C plug into the receptacle | weave the cable through the two pegs and route its free end into the port |
| Robot | bimanual I2RT YAM; prompted with one arm | Franka Panda | Franka Panda with Robotiq 2F-85 | bimanual I2RT YAM; prompted with one arm | bimanual I2RT YAM; prompted with both arms |
| Objects | wrench (260 mm); hook on a pegboard | two identical syringes 90 mm apart in an instrument tray; sharps box with a top aperture | spur gear (16, 20 or 24 teeth) on a mat; board with a peg, a driven pinion and a start button | USB-C plug (2× scale) on an attached 240 mm cable; receptacle bolted to the bench, 0.15 mm clearance per side | 600 mm cable, one end welded in an anchor bracket; two pegs; a port |
| Success trials per episode | 1: the goal | 2: each syringe in the box | 1: the goal | 1: the goal | 3: seated at peg 1, at peg 2, in the port with its free end |
| Partial-credit reward (floor) | share of the 3 goal clauses (1/3: a wrench at rest meets the speed clause) | share of the 2 syringes in the box (0) | turning fraction of its one gear, so 0 or 1 (0) | insertion depth / 10.4 mm, capped at 1; 0 while the lateral offset exceeds 8.8 mm (0) | lower of the shares of the 4 containment and the 2 release clauses (0) |
| Step cap | 8,500 | 22,800 | 6,000 | 9,000 | 13,000 |
| Motion stack | cuRobo | cuRobo | OSC | OSC | OSC |
| Segmentation | SAM3 | SAM3 | SAM3 | color palette | color palette |
Table 5. The five tasks. The instruction is the task document’s, verbatim, and is the task description the agent receives. Success is scored per trial (object); the reward is partial credit, logged but never reported as success. Step caps are in control steps.
What each task demands. Each task stresses a different part of a policy. Tool Hanging moves only the wrench, so its difficulty lies in a grasp that keeps the ring free and a collision-free carry onto the hook. Syringe Packing must turn each 150 mm syringe upright to pass the 55 mm aperture, twice, among obstacles; it is the longest task (a median episode of 11,345 control steps, and 22% of episodes reach the step cap against 1–5% elsewhere) and the collision-geometry example of Finding 5. Gear Meshing is the only task with a temporal goal: the gear must turn with the pinion for 1 s of simulated time after the button press, unheld. USB-C Insertion is the precision task, with 0.15 mm of clearance per side and a cable dragging on the plug. Cable Routing manipulates a deformable object directly, with three separately scored sub-goals.
Instruction, embodiment and perception. The agent receives the document’s instruction verbatim; it is identical across a task’s 19 documents, so instances differ only in the drawn scene. Only Cable Routing is prompted as bimanual (Table 13). The motion stack and segmentation backend are fixed per task and identical in every run, so no comparison in the main text mixes back ends.
Goal Predicates and Scoring
An episode ends when the policy returns (after a 60-control-step settle, 120 for Gear Meshing), reaches the step cap or diverges, and a Syringe Packing or Gear Meshing episode also as soon as its goal holds. The simulator then reads the document’s goal, a predicate over object features, contacts and speeds; once and hold-for clauses track the whole episode. Listing 1 gives the goal clauses.
Listing 1: Goal clauses of the five tasks’ documents, in Table 5’s order.
; Goal, reward and scored regions of each task's first training document, from the
; frozen RoboSimStudio task documents (comments dropped, closing parentheses joined,
; unscored regions elided as ...). All 19 documents of a task state the same goal and
; reward. Regions are (x0 y0 x1 y1) in metres; the syringe box region's ranges shift
; per document with the box.
; Tool Hanging -- tool_hanging/toolhanging-easy-hangwrench-001.task
(:goal
(And
(encloses wrench_1.ring wrench_hook_1.shank)
(>= (contact-force wrench_1 wrench_hook_1) 0.2)
(<= (speed wrench_1) 0.15)))
; Syringe Packing -- sharps_disposal/tray_pair_both_t00.task
(:regions
...
(syringe_box_region
(:target syringe_box)
(:ranges ((-0.037638 -0.111298 0.051362 0.054702)))))
(:goal
(and
(In syringe_0 syringe_box_region)
(In syringe_pack_0 syringe_box_region)))
; Gear Meshing -- gear_mesh/gearmesh-easy-single-01.task
(:goal
(once
(after (>= (gears-seated board) 1)
(after (started board)
(hold-for 1.0
(worst_of
(And (gear0-on-peg board) (gear0-turning board)
(not (gripped gear_a.body)))))))))
(:reward (progress (turning-fraction board) (:gate 1.0)))
; USB-C Insertion -- usbc_insert/usbcinsertion-easy-insertplug-01.task
(:goal
(And
(>= (insertion plug port.bore) 0.0104)
(<= (insertion:lat plug port.bore) 0.0087931792)
(<= (speed plug) 0.05)
(not (gripped plug.body))
(> (tcp-distance plug) 0.04)))
(:reward
(progress (insertion plug port.bore)
(:gate 0.0104)
(:gated-by (<= (insertion:lat plug port.bore) 0.0087931792))))
; Cable Routing -- cable_route/cablerouting-easy-routecable-01.task
(:regions
(seat0 (:target peg0) (:ranges ((-0.045 -0.035 -0.003 0.035))))
(seat1 (:target peg1) (:ranges (( 0.003 -0.035 0.045 0.035))))
(portbox (:target port) (:ranges ((-0.045 -0.045 0.045 0.045))))
(:goal
(worst_of
(And
(>= (containment cable seat0) 0.039999999999)
(>= (containment cable seat1) 0.039999999999)
(>= (containment cable portbox) 0.059999999999)))
Per-object success. Success is scored per object with VABAR’s scorer: derive_trial_keys strips the modal wrappers and makes one trial per single-object goal, or one per (object, region) pair when a goal has several containment clauses (Table 5); release clauses belong to no trial, and a diverged episode fails every trial. Scoring objects rather than episodes keeps the partial progress of the two multi-object tasks: at the kept iterate, 43% of held-out Syringe Packing episodes place exactly one of the two syringes, and no Cable Routing program meets every trial on all ten layouts. Some readings were scored before this rule was final; they count one Syringe Packing trial per episode (part of the reference runs of the feedback and skill-library studies) or a seated gear as a met Gear Meshing goal (some Gear Meshing reference runs). The instance analysis (Appendix § 9.§ 4) and the failure analysis (Appendix § 17) rescore every batch they use with the final rule.
Partial credit. Each document also defines a partial-credit reward in (Table 5), which the loop logs but no reported number uses, because it adds little to per-object success: over every training and held-out episode of the tables’ finished runs, 80% of Tool Hanging episodes sit exactly at the floor, 3% of USB-C Insertion rewards lie strictly between 0 and 1, Gear Meshing’s is binary and Syringe Packing’s equals the share of syringes placed; only Cable Routing’s quarters add resolution.
Scene Generation and Splits
A Tool Hanging, USB-C Insertion or Cable Routing instance is its base document drawn once by the base’s sampler at a fixed seed, from NumPy’s default_rng(seed + episode) at episode 0. A Gear Meshing or Syringe Packing instance records its draw as start poses: each gear board is drawn at its own seed, its tooth count from one balanced draw over the 40 boards, and the 30 syringe layouts are drawn from one default_rng(0) stream (tray shifts at least 10 mm apart). Either way a document reproduces its scene exactly, so every setting runs the same episodes. Table 6 lists the samplers, the ranges of every randomized quantity, and the documents of each split; Figure 12 shows each instance’s draw. The agent reads feedback from the training episodes only; validation decides when a run stops and which iterate is kept; held-out success is only reported (Appendix § 11.§ 1): Table 1’s @k is the best held-out reading within rounds, and the other tables read the zero-shot program and the kept iterate.
Figure 12. Main randomized quantities of every instance by split, recomputed from the samplers; the segment through each Syringe Packing point shows the tray’s yaw, and dashed boxes are the sampling ranges (Table 6).
| Task | Base document, sampler and randomized quantities | Documents: 4 train, 5 val., 10 held-out | Seed |
| Tool Hanging | wrenches_level1, layout: LEVEL1: wrench each mm; hook fixed | train: 001, 012–014<br>val.: 015–019<br>held-out: 002–011 | doc : seed |
| Syringe Packing | tray_pair_both, 30-trial layout sweep: tray arrangement yaw about the first syringe, shift each mm, shifts mm apart; sharps box each mm | train: t00, t11–t13<br>val.: t14–t18<br>held-out: t01–t10 | trial of one sweep at seed 0 |
| Gear Meshing | 40 drawn single-station boards, draw_layout: tooth count 16/20/24 (balanced); plate center each mm; chain yaw ; gear dropped uniformly on the mat ( mm from the plate), clocked | train: 01, 12–14<br>val.: 15–19<br>held-out: 02–11 | doc : 31-bit SHA-256 of ; teeth: one balanced draw |
| USB-C Insertion | insert_easy, staging: USBC_EASY: plug yaw , each mm; receptacle mm across, mm along, yaw , redrawn until it clears the bench | train: 01, 12–14<br>val.: 15–19<br>held-out: 02–11 | doc : seed |
| Cable Routing | port_easy, staging: PORT_BOARD: cable bow mm, shift mm, yaw , wave mm (fractions ), anchor follows the cable; port each mm, yaw ; peg gap the 240 mm pitch within the anchor–port corridor, lateral wobble mm, board yaw , board position uniform in the corridor, rejected unless turning, wrap and route length stay near the declared board’s values | train: 01, 12–14<br>val.: 15–19<br>held-out: 02–11 | doc : seed |
Table 6. Scene generation and splits. Each instance is one task document, its base document drawn once as the last column states.
Coverage gaps. The splits are fixed document ids, not stratified, and nine training and validation draws leave parts of each sampling range to the held-out split alone. No Gear Meshing training board has 24 teeth and no validation board has 16. Training wrenches are offset at most +2.9 mm in (validation +7.4 mm), while 7 of the 10 held-out wrenches are offset 8–18 mm. Training and validation trays are shifted at most 27.6 mm toward , while 4 held-out trays are shifted 39–60 mm. Training and validation plugs are offset at most 7.2 mm toward , while 2 held-out plugs are offset 10.8–11.0 mm. Training cables bow by a fraction of at most +0.03 and validation cables by at most +0.20, while 6 held-out cables bow +0.15 to +0.68.
Held-Out Instance Difficulty
To see whether some held-out instances are systematically harder, we pool the runs of all three studies’ tables and compute, per held-out object, the share of programs that succeed on it, at (each distinct zero-shot program once; seeded feedback arms share their reference’s) and at the kept iterate, scoring every program with the final per-object rule (Figure 13).
Figure 13. Held-out success per instance: the share of programs that succeed on each held-out object at (hollow) and at the kept iterate (filled), pooled over the runs of all three tables, against one randomized quantity. Diamonds are the second syringe and the mean of the two peg seats; the band spans the training draws.
Success is mostly a property of the program. On Gear Meshing and USB-C Insertion a program usually solves every held-out instance or none: 68% of zero-shot Gear Meshing programs solve all ten boards, and even at the kept iterate 84% of USB-C programs are all-or-nothing, with per-layout shares of 34–42% and no trend in the plug’s offset, including the two plugs beyond every training draw. Zero-shot Tool Hanging succeeds on 7–11% of programs on every layout. On these tasks a held-out score mostly counts the programs that work.
Layout effects present from the start. Where success does vary across layouts, the variation coincides with coverage gaps. On the four Syringe Packing layouts whose tray lies beyond every training and validation tray in (two of them also in ), the first syringe is placed by 26% of zero-shot and 40% of kept programs, against 58% on the other six, and grasp misses (G in Table 23) make up 21% of its kept-iterate failures there against 6% elsewhere; the second syringe is harder on every layout (23% against 51% at the kept iterate). On Cable Routing the peg seats stay at 56–93% at the kept iterate, but the port trial varies with the cable’s bow (12–68%): on the six layouts bowing +0.15 or more, 28% of zero-shot and 24% of kept programs reach the port, against 58% and 55% on the other four. These six layouts also have the most negative port yaws, which ten layouts cannot separate from the bow.
| Category | Rule | Example message |
| P Perception | A perception node raises: the target, part or region is not found or not localized. | SAM3 could not localize 'ring' |
| G Grasp miss/loss | The jaws close on nothing, or grasp verification finds the object gone from the hand. | the jaws closed on nothing (gap 8.00 mm, lag 0.00 mm) |
| K Planning/IK refusal | The planner or IK returns no feasible grasp, orientation, path or approach. | plan-grasp: 0/1 candidate grasps had a feasible CuRobo plan ... |
| T Reach/tracking/transport | The hand stops short of, or tracks off, a commanded pose. | the hand stopped 23 mm from the grasp pose |
| A Placement/insertion misalignment | The graph finishes with the object’s placement clause unmet, or a placement or insertion check raises. | encloses/wrench_1.ring/wrench-hook_1.shank: False |
| R Release/goal-clause | The graph finishes with placement met but a rest, release or turning clause unmet. | speed/wrench_1: False |
| S Self-check abort | The program routes itself to a failure end node; no node raises. | workflow terminated at end node 'unfinished' (failure) |
| B Program bug/tool misuse | The program’s own code raises, or it calls a tool outside the tool’s contract. | TypeError: run() missing 1 required positional argument: 'fixture.feature' |
| U Step budget | The episode reaches its step limit. | close.kind: truncated |
| I Infrastructure/simulator | The physics diverges or the machine fails. | OSError: [Errno 28] No space left on device |
Table 23. Failure categories of held-out objects. A failed object takes its episode’s category, assigned by outcome first, then by the error text of the node that raised, then by the goal clause left unmet; examples are verbatim from held-out episode records: the error message, or for A and R an unmet goal predicate and for U the close kind.
Gaps that open with learning. On Gear Meshing and Tool Hanging the instances start out equal and learning separates them. Zero-shot Gear Meshing success is flat across tooth counts (70–72%), but the kept iterate fixes 26% of the zero-shot failures on 16-tooth boards against 46–52% on the others, leaving them at 69% against 78–79%: the 24-tooth boards, which no training board has, gain the most, and the 16-tooth boards, which no validation board has, the least. At the kept iterate the seven Tool Hanging wrenches offset +8 mm or more succeed for 28% of programs against 34% for the other three, with reach failures (T) twice as common on them (22% against 11% of failures), though single layouts of the two groups overlap (21–33% against 33–38%).
Implications. Because every setting meets the same instances, the comparisons of Table 1, Table 2 and Table 3 stay paired, but every task’s absolute held-out level includes instances beyond the training draws. These cost success from the start only on Syringe Packing and Cable Routing, and learning does not close those gaps: it narrows Syringe Packing’s from 31 to 18 points and leaves Cable Routing’s unchanged. Of the two gaps that open with learning, the larger (Gear Meshing, 10 points) lies where validation, which selects the kept iterate, has no instance, not where training has none.
Skill Library
The agent composes its policy over (§ 3), where is the code it writes. We first describe how a program uses a skill, what the three parts the harness provides, and what each experiment ships; the skill-library study qualifies Finding 2.
Library Format and Use
The general skills and the tool bundles ship in the Agent Skills format: a directory with a SKILL.md (when the skill applies, its state flow, declared tools, required inputs and named exits) and scripts. Each subgraph of a program carries a skill label, the skill it runs or generated for a stage the agent wrote. With the source, the agent copies the skill’s scripts into its workflow; in the docscallable arm a node names a script and the runtime resolves it from the named skill. Every run enforces its arm: the runtime loads only the shipped skills, the guard refuses other labels, and the library checkout is closed to the agent. All runs read open-robot-skills ed6ab27, except three early Table 1 runs, which read 113261b (the same skills, three scripts differ); the earliest reference runs read a 113261b working tree whose skills equal ed6ab27’s.
Primitive Tools (S0)
holds 70 to 78 tools per task: robot primitives, the motion backend’s planner, six tool bundles (cuRobo planning, curve fitting, point-cloud geometry, Grounding DINO detection, SAM3 segmentation and tracking, a VLM) and, on USB-C Insertion and Cable Routing, a few audited tools of the task’s own (Table 7). The agent gets SKILLS.md, which documents every tool, and the six bundle manuals; sim.check_success and the privileged sim.* state readers are withheld from every run, since the goal is what the agent has to learn. is thus not a set of bare primitives: detection, segmentation and grasp proposals are single calls.
| Namespace | Tools |
| robot (23) | close_gripper, describe_arm, describe_gripper, describe_workspace, execute_trajectory, forward_kinematics, get_camera_pose, get_ee_pose, get_gripper, get_gripper_pose, get_observation, go_home, go_to_pose, go_to_pose_cartesian, grasp_frame, instruction, move_cartesian_until_contact, move_to_joints, open_gripper, set_grip, solve_ik, stream_dual, wait_steps |
| motion (5/8) | describe_planner, get_robot_collision_spheres, plan_joint, plan_linear, plan_support_disengagement; with cuRobo also build_world_tsdf, plan_to_pose, solve_ik_batch |
| curobo (9) | cloud_to_attachment, plan_directed_linear, plan_grasp_motion, plan_linear, plan_to_grasp_poses, plan_to_pose, plan_with_grasped_object, validate_joint_trajectory_grasped, validate_joint_trajectory_robot |
| curve (2) | fit_centerline, track_centerline |
| geometry (22) | build_world_config, cloud_to_attachment, compute_drop_position, compute_feature_mate, compute_obb, compute_xy_distance, depth_to_point_cloud, exclude_robot_points, filter_and_compute_obb, filter_noise, fit_linear_feature, fit_planar_feature, front_grasp_from_obb, iou, mask_to_world_points, pixel_to_world_point, pose_distance, rotate_quat_z90, select_top_down_grasp, top_down_grasp_candidates, top_down_grasp_from_obb, transform_points |
| grounding-dino (1) | detect |
| sam3 (6) | segment_box, segment_point, segment_text, tracker_close, tracker_init, tracker_update |
| vlm (2) | query, query_yes_no |
| usbc (5) | grip_state, instruction, next_pair, receptacle_shape, set_gripper |
| cable (8) | carry_with_feed, curve_memory, next_station, plan_station, plan_support, set_gripper, support_memory, wait_steps |
Table 7. The tool surface (the none privilege tier every run uses), by namespace. Each task gets the 23 robot tools, the motion tools of its backend (8 with cuRobo on Tool Hanging and Syringe Packing, 5 with OSC on the other tasks) and the 42 tools of the six bundles that ship with open-robot-skills; USB-C Insertion and Cable Routing add 5 and 8 of their own tools, admitted after an audit found they read no privileged state. Totals from Tool Hanging to Cable Routing: 73, 73, 70, 75 and 78 tools.
General Skills (Sg)
open-robot-skills holds 30 skills with 72 scripts (Table 8), most of them for perception (11) and grasping (8); five skills compute, plan and execute the motion of a held object. A run admits only skills whose scripts call nothing outside its , so no program reaches simulator state through a skill: three skills call withheld simulator tools and never ship, and the two executing skills need cuRobo’s motion.plan_to_pose. This leaves 27 callable skills on the cuRobo tasks and 25 on the OSC tasks.
| Skill | Scripts | cuRobo | OSC | What it does |
| Perception (11) | ||||
perceiving-deformable-linear-objects | 1 | ✓ | ✓ | Ordered 3D cable centerline from RGB-D, kept current across a task |
perceiving-functional-features | 6 | ✓ | ✓ | Fits a described part (loop, shaft, tip, aperture) from RGB-D |
perceiving-next-item | 3 | Container and next remaining item from one view, or done | ||
perceiving-object-parts | 1 | ✓ | ✓ | Detects the parent object, then segments a named subpart in its crop |
perceiving-objects | 1 | ✓ | ✓ | DINO detection, VLM crop tournament, SAM3 mask, world-frame box |
perceiving-objects-oneshot | 1 | ✓ | ✓ | DINO plus one VLM set-of-marks pick, with a clean not-found exit |
perceiving-relational-correspondences | 2 | ✓ | ✓ | Language-conditioned matching of source and destination instances |
perceiving-routing-fixtures | 2 | ✓ | ✓ | Surveys a cable bench: stations, cable centerline, owed crossing sides |
perceiving-sorting-pairs | 3 | ✓ | ✓ | Labeled four-compartment destination and the next source with its region |
reconstructing-collision-worlds | 2 | ✓ | ✓ | Planner collision world from RGB-D, with a carved approach corridor |
registering-held-objects | 1 | ✓ | ✓ | Re-observes a held object from the wrist; feature in TCP frame |
| Tracking (1) | ||||
tracking-objects | 0 | ✓ | ✓ | Long-running SAM3 tracker publishing an object’s mask and box |
| Grasping (8) | ||||
grasping-concave-contour | 1 | ✓ | ✓ | Grasps an awkward part across inward waists of its silhouette |
grasping-direct-ik | 4 | ✓ | ✓ | IK grasp that aligns above the target, then descends straight down |
grasping-linear-feature | 2 | ✓ | ✓ | Fits an elongated part’s 3D axis; perpendicular, inclination-aware grasp |
grasping-short-axis | 7 | ✓ | ✓ | Deterministic top-down cuRobo grasp across the box’s short axis |
grasping-with-planner | 6 | ✓ | ✓ | Axis-locked top-down linear descent with a cuRobo planner fallback |
proposing-short-axis-grasps | 1 | ✓ | ✓ | Proposes top-down grasps closing across an elongated object’s short axis |
proposing-side-grasps | 1 | ✓ | ✓ | Proposes side-entry grasps at twelve azimuths, ranked by jaw margin |
selecting-reachable-grasp | 1 | Takes the first ranked grasp candidate the arm can reach | ||
| Planning (2) | ||||
computing-feature-mating-poses | 1 | ✓ | ✓ | Approach, engaged and mate TCP poses from held and fixture features |
planning-held-object-motion | 3 | ✓ | ✓ | Plans clearance lift, reorientation, transit and engagement for a held object |
| Motion (3) | ||||
executing-feature-mating | 2 | ✓ | Executes a mating plan: tracked legs, servo, contact seating, release | |
executing-held-object-motion | 2 | ✓ | Collision-aware pose sequence for a held object, without releasing it | |
transporting-objects | 11 | ✓ | ✓ | Carries the held object to a destination and releases it |
| Manipulation (2) | ||||
placing-held-feature | 1 | Moves a held object so its declared feature lands on a target | ||
seating-a-cable-crossing | 1 | ✓ | ✓ | Streams one cable crossing leg, then measures its seat by vision |
| Verification (3) | ||||
verifying-a-cable-route | 1 | ✓ | ✓ | Judges a finished cable route from a camera, with margins |
verifying-grasps | 3 | ✓ | ✓ | Checks arrival, jaw gap and lifted visibility of one grasp |
verifying-placement | 1 | ✓ | ✓ | Confirms from one RGB-D view that an object passed an aperture |
| Total: 30 skills | 72 | 27 | 25 |
Table 8. The general skills (open-robot-skills) by the category their SKILL.md declares, with their Python scripts (tracking-objects runs from a tool module) and whether each is callable on the cuRobo tasks (Tool Hanging, Syringe Packing, and USB-C Insertion, Cable Routing), i.e. whether that task’s holds every tool its scripts call.
Expert Skills (Sτ)
For each task, holds the scripts of the human-written VABAR vision policy (Table 9): 11 bundles with 43 entry scripts, the script nodes of each human graph without its wiring. Expert scripts carry no exit contract, so the agent copies what it uses into its own scripts and labels those subgraphs generated. ranges from 17 entry scripts on Syringe Packing to a single glue script on Tool Hanging, whose human policy is built from 15 scripts of 10 skills in .
| Task, human graph | Bundle | Scripts |
| Tool Hanging | glue (1) | choose_placement_mode |
gap_perception | from (15) | compute_mate, execute_placement_plan, release_and_retract, execute_approach_only, execute_reorientation, compute_align_pose, execute_grasp_align, perceive_fixture_feature, perceive_object_feature, select_next_source, plan_clearance_motion, plan_feature_engagement, build_collision_world, register_held, descend_release_linear (of 10 skills) |
| Syringe Packing | perception (8) | build_collision_world, count_remaining, map_syringe_to_hand, perceive_fixture, perceive_target, verify_grasp, verify_placement, verify_seated |
gap_perception_v2 | motion (9) | compute_grasp, execute_insertion, execute_reorientation, grasp_checked, lift_held, plan_clearance_motion, plan_visual_insert, release_and_retract, unjam_aperture |
| helpers | motion_checks, perception_checks | |
| from (1) | compute_mate (of computing-feature-mating-poses) | |
| Gear Meshing | perception (1) | watch |
gap_perception | motion (6) | carry, clear, next_station, pick, press, seat |
| helpers | gearboard, sight | |
| USB-C Insertion | perception (4) | check_grasp, ground_vision, inspect_vision, relook_vision |
mate | motion (2) | grip-step, move_stage |
| glue (2) | next_pair, route_perception | |
| helpers | bore_cv, held_frame | |
| omitted (5) | ground_oracle, inspect_oracle, relook_oracle (call sim.perceive_mate); hold_plan (calls usbc.plan_hold); plan_stroke (calls usbc.plan_mate) | |
| Cable Routing | perception (3) | perceive_rod, perceive_weave_vision, verify_weave_vision |
weave_terminated | motion (6) | anchor_far_end, plan_crossing, plug_port, restore_seats, run_legs, seat_station |
| glue (1) | next_station | |
| helpers | fixtures_cv, grasp_vision, terminal_vision |
Table 9. Expert skills : the entry scripts of each task’s human VABAR vision policy, bundled by the tools they call (count in parentheses), without the graph that wires them. Helpers are modules the entry scripts import; scripts taken from ship there, and scripts that call a tool outside are omitted.
Availability per Experiment
Table 10 lists what each experiment ships and how many runs it has; every setting of Table 1 gets the whole library. The feedback study’s reference omits because early zero-shot programs copied the human scripts, and a reference built on human code would measure that code more than the feedback. The skill-library study removes one part of the reference at a time: docs callable withholds the source but keeps every skill documented and callable, separating a skill’s interface from its code, and tools only removes .
| Table 1, every setting | Feedback study; full library | Tools only | Docs callable | |
| Library | docs of | |||
| skills callable | 27 / 25 | 27 / 25 | 0 | 27 / 25 |
| source readable | ✓ | ✓ | – | – |
| expert scripts | ✓ | – | – | – |
SKILLS.md | ✓ | ✓ | ✓ | ✓ |
SKILL_LIBRARY.md | notice only | |||
skills/ | – | – | ||
WORKFLOW_FORMAT.md | ✓ | ✓ | ✓ | ✓ |
| Runs (all tasks) | 250 | 200 | 25 | 25 |
Table 10. Skill availability per experiment. Callable counts are for the cuRobo / OSC tasks; the middle rows are the workspace files that carry each part (SKILLS.md documents , SKILL_LIBRARY.md lists the callable skills, skills/ holds their source).
Agent Harness
Every run of Table 1 of the feedback study (Tables 2 and 3) executes the loop of Section 3 in one harness; the runs differ only in the agent and its reasoning budget, the shipped skills, the feedback renderer, the evaluation schedule and the time budgets. Each instance set has one role: the agent sees only the training episodes, validation decides when a run stops and which iterate it keeps, and the held-out set only measures. We follow one iteration of the loop (Appendix C.1), then give each study’s evaluation schedule (Appendix C.2) and the workspace and the checks a proposal passes (Appendix C.3). Appendices D and E reproduce what the agent reads, and Appendix F gives the revert, stopping and selection rules and relates the @ of Table 1 to the validation-selected iterate.
Self-Learning Loop

Figure 14. The self-learning loop. Blue states and the preflight’s fix sessions run an agent session; the harness runs everything else. No validation or held-out reading reaches the agent.
At the agent writes from the task brief; every iteration then runs one unscored preflight episode on the first training instance, the four training episodes and, when scheduled, the held-out batch, and builds the feedback report from the training episodes alone, after the revert rule has chosen the policy the next session edits (Figure 14). The five validation episodes run after the report is written, for the stopping and selection rules (Appendix F); unless the run stops, a new agent session edits the policy into . The session keeps no conversation from earlier sessions, but its report summarizes every earlier iterate (training result, edit, behavior and the previous session’s own account, Appendix E) and history/ holds the exact diffs. A rate limit that outlasts the retries of Table 18 defers the round without recording an iteration; any other provider error ends the session without an edit and counts as an unchanged iteration (1% of iterations).
Evaluation Schedule
Table 11 gives the schedule of each study and Appendix A the instance splits. Table 1 reports a learning curve, so it scores the held-out set every round; the 6-iteration studies report only and the kept iterate, and score only those two.
| Table 1 (models) | Feedback and skill-library studies | |
| Iterations | 3 () | 6 () |
| Training | 4 instances per iteration | 4 instances per iteration |
| Validation | 5 instances per iteration | 5 instances per iteration |
| Held-out | 10 instances per iteration | 10 instances at and on the kept iterate |
| Initial policy | written by the agent | written by the agent; in the feedback study every arm reuses the reference run’s with its training batch and held-out reading, and runs its own validation |
| Kept iterate | first iterate with the highest validation success | |
| Reported | @: best held-out success within rounds | initial policy: held-out success of ; other rows: held-out success of the kept iterate |
Table 11. Evaluation schedule. Every instance is one episode, scored per object; an unscored preflight episode precedes each training batch. The Opus 5, xhigh row of Table 1 runs the right column’s schedule and reports @ (Appendix C.2).
Two first-pass runs of the Opus 5, xhigh row of Table 1 (Tool Hanging and Gear Meshing) depart from the splits: they train on the first training instance repeated four times and have no validation set (Appendix F.4). That row reuses the first three iterates of 6-iteration runs with the general and expert skills (); its @ averages the runs with a held-out reading at round : the reading, the kept iterate’s, one carried from an earlier stop, or a re-score of the or iterate on the same ten held-out instances after the run.
Workspace, Guard and Preflight
Workspace and sandbox. Each session runs in a sealed workspace whose files Listing 6 lists; workflow/ holds the policy and is the only directory the harness takes back. Claude Code runs with a deny list that blocks reading, listing and searching the simulator, the harness, the skill sources and all run outputs (the whole home directory), and with shell access limited to ./validate.sh (plus echo and mkdir). The Codex CLI cannot start its own sandbox on our machines, so its
AGENTS.md states the same rules and off-limits paths, and the harness audits every command and file change of each session for those paths afterwards; no command or file change named one.
Guard. A static guard accepts an improve session’s proposal only if every tool it names (tool nodes, recovery steps, literal ctx.tool("...") calls) is on the tier’s allowlist, no edited script computes a tool name at run time, every subgraph label names a shipped skill or marks the subgraph as the agent’s own, and ./validate.sh passes. and preflight repairs skip the guard and are checked only by the ./validate.sh their session is asked to run. A rejected proposal returns to the agent with the violations for up to two fix rounds; one still rejected, or unchanged, carries the edited policy (, or the incumbent after a revert) forward. The guard rejects 9% of first proposals, never for a withheld tool or an unshipped skill: every rejection is for a tool name computed at run time, which no static check can clear, and every rejected proposal passes after one fix round.
Preflight. The preflight separates a policy that cannot run from one that runs and fails. Before anything is scored, an episode goes back to a fix session (up to two) if its graph fails to load or an error message carries a Python error no correct program raises (AttributeError, ImportError, IndexError, KeyError, NameError, SyntaxError, TypeError, UnboundLocalError, ZeroDivisionError), a tool-argument error or tool refusal, or one of a list of interpreter, tool-refusal and executor-binding messages. Any other error, such as a RuntimeError a policy raises to report a failure, counts as a failed episode. Capped at 1,500 steps, the preflight costs a broken program one short episode instead of a scored batch of identical failures. 3% of preflights crash (5% in the 3-iteration runs, 2% in the 6-iteration runs), mostly on TypeError, graphs that fail to load and tool-argument errors; fix sessions repair 88% of them, and the rest, all in 3-iteration runs, ruin their batch unrepaired. Table 12 counts these interventions per 100 agent sessions; preflights crash most often in the restricted skill-library arms (7.4 per 100 sessions), whose programs call raw tools or skills with hidden source.
| Event (per 100 agent sessions) | Table 1 | Feedback | Skill library | All |
| Guard rejections | 5.5 | 6.3 | 4.3 | 5.7 |
| non-literal tool name (%) | 100 | 100 | 100 | 100 |
| Crashed preflights | 3.6 | 0.5 | 7.4 | 2.7 |
| repaired by fix sessions (%) | 82 | 100 | 100 | 88 |
| run unrepaired (%) | 18 | 0 | 0 | 12 |
| Reverts | 8.7 | 16.7 | 3.2 | 11.6 |
| training success rate fell (%) | 37 | 43 | 67 | 42 |
| closest approach grew (%) | 34 | 36 | 0 | 35 |
| every episode aborted (%) | 27 | 17 | 33 | 21 |
| fewer goal reaches or carries (%) | 2 | 3 | 0 | 3 |
| Deferred rounds (rate limit) | 1.9 | 3.4 | 0.0 | 2.4 |
| API errors recorded as unchanged | 0.2 | 0.0 | 3.2 | 0.4 |
| Updates left unchanged by the agent | 1.3 | 0.7 | 0.0 | 0.9 |
Table 12. Harness events per 100 agent sessions (zero-shot generation, improvement and preflight-fix sessions), with the studies split as in Table 20; indented rows give the share (%) of the event above. The Opus 5, xhigh runs of Table 1 count all six rounds.
Prompts
Every coding-agent session starts in a sealed workspace, and the whole prompt is its instruction file INSTRUCTIONS.md: the session’s command-line prompt (Table 18) only points to that file, and we add no system prompt. INSTRUCTIONS.md is a task template with its {{slots}} filled, followed by a workspace block. We give the session templates, the workspace block, the slots whose values depend on the task, and what differs between the arms of a study. The templates are byte-identical in every harness snapshot the runs used.
| Claude Code call | claude -p "Follow INSTRUCTIONS.md in this directory." --output-format json --max-turns 80 --allowedTools "Read,Edit,Write,Grep,Glob,Bash(./validate.sh:*), Bash(echo:*), Bash(mkdir:*)" --model <id>; Kimi K3 and DeepSeek V4 set ANTHROPIC_BASE_URL to OpenRouter’s Anthropic-compatible endpoint. |
| Codex call | codex exec --json --skip-git-repo-check -s danger-full-access -C <workspace> -o <transcript>/last_message.txt -m <id> -c model_reasoning_effort="xhigh" "Follow INSTRUCTIONS.md in this directory. AGENTS.md states the rules." |
| Reasoning | Opus 5, high and low add MAX_THINKING_TOKENS=32000/4000 to the environment; Codex gets model reasoning effort in the call. Claude Code’s effort level is not set: the local machine’s CLI settings pin effortLevel to xhigh, and the remote machines did not record it. |
| Session limits | 2 h wall clock per session; turn cap --max-turns 80 for Claude Code, none for Codex. |
| Sampling | temperature and top-p not set (provider default). |
| Context and output | as the CLI reports them. Context: 1M tokens for the Claude models, 200k for Kimi K3, DeepSeek V4 and the sessions rerun through OpenRouter (the CLI’s default for a model outside its catalog). Maximum output per response: 64k or 128k (Opus 5, Sonnet 5), 128k (Opus 5.5), 64k (Fable 5.1, the reruns), 32k (Kimi K3, DeepSeek V4). Codex reports neither. |
| Rate limits | from 2026-09-22, a Claude session refused with HTTP 429 on the subscription login is rerun at once with the same model through OpenRouter; before that, and for Kimi K3 and DeepSeek V4, it is retried after the limit resets when that is within 1 h (at most 4 attempts). A later reset, or a Codex usage limit, defers the round, and the run resumes it. |
Table 18. Inference parameters of every agent session of the VABAR studies (the curriculum study’s are in Appendix § 19). <id> is the model id of Table 17; the working directory is the session’s workspace.
Session Templates
A generate session () reads Listing 2; every improve session reads Listing 3, with the feedback report in FEEDBACK.md (Appendix § 15.1). When the guard or the validator rejects a proposal, the next round edits the rejected proposal with Listing 4 appended. When the unscored preflight episode raises a program error, a fix session runs the improve template with Listing 5 appended.
The templates hold only task-independent advice. The generate brief asks for stages with a routable failure exit each, so that a failure ends the episode at a named step the report can name instead of exhausting the step budget. The improve brief names every channel a report may carry, including the stills that only the stills arm renders (Table 14), so that it is the same text under every feedback condition.
| Condition | Run arm | Form | Spatial | Granularity |
| Reference (Table 1; Per skill, + part annotations) | reference | state, relations | parts | semantic |
| Spatial representation (Table 3) | ||||
| Outcome only | success_only | -- | (parts) | (semantic) |
| State + relations, world frame | positions | state, relations | positions | semantic |
| State + relations, object frames | frames | state, relations | frames | semantic |
| Temporal resolution (Table 2) | ||||
| Per episode | episode | state, relations | parts | episode |
| Per motion | motion | state, relations | parts | motion |
| Single-form reports | ||||
| Outcome + stills | visual | visual | (parts) | semantic |
| Outcome + object state | state | state | parts | semantic |
| Outcome + relations | relations | relations | parts | semantic |
Table 14. The feedback conditions as configuration (feedback.design), grouped by the table that reports them. Every non-reference condition changes one axis of the reference and continues from the reference run’s initial policy. Parenthesized values select nothing: the spatial representation applies to state and relations, the granularity to any form beyond the outcome. Run arm: the condition’s name in the released configurations.
Listing 2: generate_brief.md, the template of the generate session.
# Author a robot policy
Write a GaP workflow that accomplishes this task:
> {{instruction}}
Read `WORKFLOW_FORMAT.md` for the workflow format and `SKILLS.md` for the **complete** tool surface you may call | nothing outside it exists at run time. Run conditions for this policy:
- Perception body: **{{perception}}** (a graph may ship both an oracle and a vision body behind a router reading the workflow input `perception`; if you ship one body, make it this one).
- Privilege tier: **{{privileged}}** | SKILLS.md marks which `sim.*` tools this tier admits. Do not call tools outside the tier.
- Update mode for later iterations: **{{update_mode}}**.
{{bimanual_note}}
Design guidance:
- Decompose into subgraphs along natural stages (perceive → grasp → transport → place is the canonical shape for pick-and-place).
- Route failures: give every subgraph an `on_error` label and map every exit at the top level; a recovery `end` node that opens the gripper and goes home beats a hang.
- Verify what you can (did the gripper actually close on something?) before committing to the next stage.
{{layout_note}}
{{extra_guidance}}
Listing 3: improve_freeform.md, the template of every improve session.
# Improve a robot policy from execution feedback
The task:
> {{instruction}}
The current workflow just ran in the simulator; its results are in the feedback (per-episode outcomes and, depending on this experiment's selected channels, visual observations, structured state, state deltas, motion spans, evaluator facts, or execution traces). Improve the policy so more episodes succeed.
Run conditions: perception body **{{perception}}**, privilege tier **{{privileged}}**, update mode **freeform** | you may change `workflow.json` and any script, add scripts, or restructure the graph. Every tool you call must be in `SKILLS.md`'s tier table, with a **literal** name string.
{{bimanual_note}}
How to use the feedback:
- If FEEDBACK.md opens with a **LOOP NOTICE**, the harness has already reset `workflow/` to an earlier, better-scoring iteration: the feedback describes the policy that regressed, not the one you are editing. Read it for what the undone edits (in `history/kNNN.reverted.diff`) broke, then improve the restored policy; do not re-apply the undone edits as they were.
- If the feedback lists captured frames, **Read the middle frame of the first failed episode before diagnosing** | one image often shows what no number in
- Diagnose before editing: find where each failed episode diverged (a wrong grasp height reads differently from a drop mid-transport or a never-found object).
- Prefer the smallest change that addresses the diagnosed failure; wholesale rewrites discard what already works.
{{layout_note}}
- If every episode fails the same way at the same node, that node is the problem; if failures are mixed, look for a fragile assumption upstream.
{{extra_guidance}}
Listing 4: fix_validation.md, appended after a rejected proposal.
````text
Your previous proposal was rejected
The guards/validator refused the last update for these reasons:
{{guard_feedback}}
Produce a corrected update that resolves every violation while keeping the intended improvement. Do not work around a violation by renaming or aliasing | fix the underlying edit. ````
Listing 5: fix_preflight.md, appended after a preflight crash.
````text
Your policy could not run
One episode was run before scoring this iteration, and the policy raised before it could finish:
{{preflight_feedback}}
These are program errors | a field read off a return value that does not carry it, a tool argument the tool does not declare, a name that was never bound. They are not failed grasps, so no episodes have been scored yet and none will be until the policy runs.
Read the exact signatures in SKILLS.md before editing: the parameter names, and the keys each tool actually returns, are listed there. Fix the calls themselves rather than wrapping them in try/except | a swallowed error becomes a silent wrong answer, which costs a whole iteration to find. ````
Workspace Instructions
Listing 6 is appended to every brief. It names the workspace files and the one shell command the agent needs, and asks for a closing summary that the loop records as the iteration’s rationale; the next report quotes it as unverified (Listing 14), next to the harness’s own account of the edit. In the workspace of a Fable 5.1 Tool Hanging run of Table 1, the files the block names take 126 KB: the workflow-format primer WORKFLOW_FORMAT.md (6.7 KB), the tool contract SKILLS.md (31 KB), the skill catalog SKILL_LIBRARY.md (21 KB), six tool-bundle manuals (30 KB) and the validator’s surface skills.lock.json (37 KB), next to the 28 shipped skill bundles. Codex sessions also find AGENTS.md (Listing 7), which states the rules Claude Code enforces with its deny list (Appendix C.3).
Listing 6: The workspace block (claude_code.mechanics); slots as in Table 13.
| Slot | In | Value in the study |
| {{instruction}} | G, I | the task’s instruction, verbatim (Table 5) |
| {{perception}} | G, I | vision |
| {{privileged}} | G, I | none |
| {{update_mode}} | G | freeform |
| {{layout_note}} | G, I | every task: “- Each training episode is a DIFFERENT layout of this task, and the policy is scored afterwards on ten further layouts it never sees, where the objects (and possibly the fixtures) sit elsewhere. Locate everything that can move with perception at run time: a coordinate read off this feedback fits the one layout it was measured in, and a policy built on such coordinates fails the rest.” The Opus 5, xhigh runs of Table 1 that train on one layout (Appendix C.2): “- Every training episode replays ONE fixed layout, but the policy is scored afterwards on layouts it never sees, where the objects (and possibly the fixtures) sit elsewhere. Locate everything that can move with perception at run time: a coordinate read off this feedback fits the training layout only, and a policy built on one fails every held-out layout.” |
| {{bimanual_note}} | G, I | Cable Routing: “This robot has 2 arms. Every robot./motion. verb takes arm_id (0 is the work arm, 1 the support arm); an omitted arm id drives arm 0. Sequence the arms deliberately — SKILLS.md states each tool’s arm semantics.” Other tasks: empty. |
| {{extra_guidance}} | G, I | USB-C Insertion and Cable Routing: which queries the color-palette matcher behind sam3.segment accepts, and the task’s goal clause in words, including that an episode ending with the object still held scores zero (Listings 8 and 9). Other tasks: empty. |
| {{guard_feedback}} | V | the guard’s and validator’s violations, e.g. “non-literal ctx.tool(...) call in scripts/grasp/locked_reach.py:268” |
| {{preflight_feedback}} | P | what the unscored preflight episode raised, e.g. “- Node ’clear_sg.clear’ failed: TypeError: int() argument must be a string, a bytes-like object or a real number, not ’builtin_function_or_method’” |
| {{action}} | M | generate: “Author the policy into workflow/ (create workflow/workflow.json and workflow/scripts/...).” Improve: “Edit the policy in workflow/ in place. Read FEEDBACK.md first.” |
| {{tool_bundle_manuals}} | M | every task: curobo.md, curve.md, geometry.md, grounding-dino.md, sam3.md, vlm.md |
| {{shipped_skills}} | M | full library: the shipped skills’ names, i.e. the callable general skills (Table 8) and, in Table 1, the task’s 1–3 expert skills (Appendix B). Tools only and docs callable: this line and the SKILL_LIBRARY.md line above it become “SKILL_LIBRARY.md — what the skill library holds for this run (it may say that it holds nothing).” |
WORKFLOW_FORMAT.md, Skills | full library: as written (Listing 10); docs callable and tools only: replaced (Listings 11 and 12) |
Table 13. Table 13: Template slots and their values. In: in the template that uses the slot (G generate, I improve, V fix-validation, P fix-preflight, M workspace block). Quoted values are verbatim; the workspace block and the primer are those of every snapshot after the first, whose earlier wording the text quotes.
````text
Workspace
Everything you need is in this directory, and nothing outside it is relevant. {{action}}
WORKFLOW_FORMAT.md| the workflow format.SKILLS.md| the complete tool surface; nothing else exists at run time.FEEDBACK.md| what the last run did (absent on the first iteration).skills.lock.json| the machine-readable surface the validator checks against (the appendix in SKILLS.md is generated from it).tool_bundles/| the manual for each tool bundle behind the surface ({{tool_bundle_manuals}}). SKILLS.md says what these tools are FOR; the appendix states no signature for them, because they run out of process and cannot be introspected. Their argument names, units, return shapes and gotchas are here and nowhere else | read the one for any bundle you call before you call it.SKILL_LIBRARY.md| the skills this run ships, what each is for, and (where it has them) its tools and exits.skills/<name>/| each shipped skill's bundle ({{shipped_skills}}): itsSKILL.md(state flow, hard rules) and the scripts to copy intoworkflow/scripts/.history/kNNN.diff| the exact edit iteration N applied, harness-computed (absent when no edit has been accepted yet). When the feedback shows a score drop after an iteration, read that iteration's diff: reverting it inworkflow/is a legitimate change, and the diff | not any summary of it | is what to trust about what changed. ````
- `history/kNNN.reverted.diff` | present only when the loop itself reset
`workflow/` to an earlier iteration before this session (FEEDBACK.md opens
with a LOOP NOTICE saying so): the combined edits it undid, i.e. what the
policy in `workflow/` does NOT contain any more.
Work efficiently | you have a limited number of turns, so spend them on
reading these files and editing the policy, not on exploring the filesystem.
Use Read/Glob/Grep for files; **the only shell command you need is**:
```bash
./validate.sh
Run it exactly as written, from this directory | an absolute path to the same script is a different command and is refused, which costs a turn. Run it before finishing and fix every error it reports. Do not run the simulator, install anything, explore outside this directory, or edit anything but workflow/. You cannot delete files and do not need to: a script no node references is never loaded, so to retire one, stop referencing it from workflow.json and leave it alone.
When the validator is clean, stop and summarize what you changed and why in a few sentences | that summary is recorded as the iteration's rationale.
Listing 7: AGENTS.md of a Codex session (GPT-6 Sol, Gear Meshing); absolute paths replaced by their role.
Rules for this workspace
Follow INSTRUCTIONS.md. Everything you need is in this directory.
Edit only
workflow/. Do not edit, create or delete anything else here.The only command you need is
./validate.sh, run from this directory. Do not run the simulator, install packages, or start background processes.Do not read, list or search anything outside this directory. In particular these paths hold the task's reference solution and its source and are OFF LIMITS; reading any of them invalidates the run:
<simulator snapshot><home directory><temporary directory><runs' output root><this run's directory><harness snapshot><skill-library snapshot><the task's expert-skill bundles>When
./validate.shis clean, finish with a few sentences on what you changed and why.
### Task-Specific Slots {#sec:D.3}
{xref}`Table 13` lists every slot and its value. Across tasks, only the instruction, the shipped skills, the Cable Routing bimanual note and the USB-C Insertion and Cable Routing guidance ({xref}`Listings 8 and 9`) differ. The layout note is in every brief because the reports print the training layouts’ world coordinates, which fit no held-out layout if copied into a policy. The bimanual note is set only for Cable Routing, whose expert policy anchors the cable’s far end with the support arm, while every motion verb drives arm 0 unless told otherwise.
The extra guidance states two facts no report shows. On these two tasks `sam3.segment_text` runs a color matcher that `sam3.md` does not describe, and its error for other queries blames missing SAM3 weights, which no edit can fix. And reports describe the scene but never the goal clause: in an early USB-C Insertion run, a report showed the plug 12.8 mm past the bore’s mouth, 0.2 mm off its axis and “seated in `port.bore: yes`” in an episode that scored zero, because the fingers still held it; once the brief stated the goal clause, the next training batch succeeded in all four episodes.
Listing 8: {{extra_guidance}}, USB-C Insertion.
- Perception in this cell: `sam3.segment_text` is served by the suite's colour-palette matcher
(ROBOSIMSTUDIO_SEGMENTATION_BACKEND=palette), not by SAM3, and `tool_bundles/sam3.md` does
not describe it. It accepts only a query that names one of the scene's palette colours:
`orange` finds the plug, `blue` the receptacle, `white` achromatic white regions. Any other
query raises "the RGB palette fallback does not support ...", whatever that message says
about SAM3 weights or HF_TOKEN. It returns the documented masks, scores and boxes, ranked
best first. - How this cell is scored, from the document's own goal clause: the plug counts
as inserted when its depth into `port.bore` is at least 10.4 mm, its lateral offset at most
8.8 mm, the plug has settled (speed at most 0.05 m/s), the gripper is NOT holding it, and
the hand has moved at least 40 mm clear of it. An episode that ends with the plug still held
at the port scores zero however well it is aligned.
Listing 9: {{extra_guidance}}, Cable Routing.
- Perception in this cell: `sam3.segment_text` is served by the suite's colour-palette matcher
(ROBOSIMSTUDIO_SEGMENTATION_BACKEND=palette), not by SAM3, and `tool_bundles/sam3.md` does
not describe it. It accepts only a query that names one of the scene's palette colours:
`white` finds the cable and the white rods (add `cable`, `rod` or `wire` to the query to rank
thin, elongated regions first), `orange` the orange blocks and the spool, `blue` blue
regions. Any other query raises "the RGB palette fallback does not support ...", whatever
that message says about SAM3 weights or HF_TOKEN. It returns the documented masks, scores
and boxes, ranked best first. - How this cell is scored, from the document's own goal
clause: the cable must be seated in seat0 and seat1 (containment at least 0.04 each) and in
the port box (at least 0.06, with its end_b fully contained), AND at the end the hand must
be at least 50 mm clear of the cable with the cable settled (speed at most 0.05 m/s). An
episode that ends holding the cable scores zero.
Differences Between Study Arms
Within a task, every feedback condition receives the same improve brief, byte for byte up to the earlier wording below, so the report is the only input that differs between the rows of Tables 2 and 3. Between the skill-library arms, the brief differs in the shipped-skill lines of the workspace block and in the Skills section of the workflow primer (Listings 10–12); the skills themselves are in Appendix B.
First harness snapshot. Runs launched on the first harness snapshot read an earlier wording of two passages; these are most runs of the reference configuration (the feedback reference and the full-library arm) and the Opus 5, xigh runs of Table 1 that train on one layout. Their workspace block lacks the sentence “Run it exactly as written, from this directory—an absolute path to the same script is a different command and is refused, which costs a turn.”, and their primer ends the second paragraph of Listing 10 with “… and copy its canonical_scripts into your scripts/ directory.” The rest of both texts is identical.
Listing 10: Skills section of WORKFLOW_FORMAT.md, full library.
## Skills | do not write every stage from scratch
"skill": "generated" means *you* author this subgraph's nodes. It is the
right answer when nothing in the library fits, and the wrong one by default.
`SKILL_LIBRARY.md` lists the strategies available in THIS run | already
filtered to the tools you may call, so everything listed is runnable. Name one
("skill": "proposing-short-axis-grasps"), read that bundle's `SKILL.md` under
`skills/<name>/` for its state flow and hard rules, and reproduce its
`canonical_scripts` into your `scripts/` directory with Read and Write | there
is no shell copy here, and a `cp` costs you a refused turn.
The reason this matters more than convenience: a stage usually has **more than
one strategy**, and the library's skills say so. A propose skill that declines
("the short axis is wider than this hand opens") hands you a routable exit to a
*different* rung. A subgraph you wrote yourself has no such exit | when its one
approach does not fit the object, the only move left is to re-use numbers that
cannot help, and the iteration is spent.
Listing 11: Skills section of WORKFLOW_FORMAT.md, docs callable (source withheld).
## Skills | do not write every stage from scratch
`SKILL_LIBRARY.md` lists every strategy available in THIS run, already filtered to the tools
you may call, and naming one on a subgraph ("skill": "proposing-short-axis-grasps") runs
it.
**This run ships no skill source.** There is no `skills/<name>/` directory to read and the
library checkout is off limits, so program against what each skill's entry states it does,
what it needs, and which exits it can take. A skill that declines hands you a routable exit
to a different strategy, which a hand-written subgraph does not.
Listing 12: Skills section of WORKFLOW_FORMAT.md, tools only.
## Skills
This run ships no skill library. Every subgraph is your own ("skill": "generated"), written
from the tools in `SKILLS.md` and the manuals in `tool_bundles/`. Stage your graph along the
task's natural stages all the same, and give each stage an exit you can route on.
Feedback Reports
One renderer turns the recorded training batch into FEEDBACK.md, the report of Section 3 and the only feedback an improve session reads. We define the feedback conditions as settings of that renderer, list what each renders, walk through one report, compare one batch under every condition, and measure report length. The reference condition is the feedback of Table 1 and of the Per skill and + part annotations rows of Tables 2 and 3, which are the same runs.
Conditions
Each condition sets three axes of the renderer: the form of the report, its spatial representation and its temporal unit (Table 14). Every non-reference condition changes one axis of the reference, starts from the reference run’s initial policy, and first re-renders the reference’s own training batch, so that differences between rows reflect the report rather than the initial policy.
What Each Condition Renders
Every condition renders the outcome block: per-episode and per-object success, steps, how each episode ended, and the errors the policy raised. Table 15 lists what each condition adds. Three consequences matter for reading the main tables. Outcome only is not a scalar reward: it names the node where each aborted episode raised and quotes the error, but not why a finished episode missed. A per-motion report has no unit for a step that moves nothing, so what a perception step returns never appears, the gap Finding 3 attributes to motion-level feedback. Syringe Packing declares no parts, so there the reference and object-frame reports differ only in wording, and the two rows end at the same 50% in Table 3. On Cable Routing the contact record omits the deformable cable, so no condition reports contact with the cable or a grasp or carry of it, and no close is reported as meeting the cable.
| Rendered | Reference | Outcome only | Outcome + stills | Outcome + state | Outcome + relations | World frame | Object frames | Per episode | Per motion |
| Outcome: episode table, per-object goals, abort, budget and stage banners, raised errors | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Node names (one unit per node, with status and time) | ✓ | ◦a | ◦b | ✓ | ✓ | ✓ | ✓ | ◦a | ◦a |
| Up to 16 stills (two cameras) around the nodes nearest the first failure | – | – | ✓ | – | – | – | – | – | – |
| Tool names, commanded vs. achieved targets | ✓ | – | – | ✓ | – | ✓ | ✓ | – | ✓ |
Node outputs (returned key=value) | ✓ | – | – | ✓ | – | ◦c | ✓ | – | – |
| World positions of objects and hands, moves | ✓ | – | – | ✓ | – | ✓ | ✓ | ✓ | ✓ |
| Orientations, own-axes offsets, poses in the target’s frame | ✓ | – | – | ✓ | – | – | ✓ | ✓ | ✓ |
| Declared parts and part geometry | ✓ | – | – | ✓ | ◦d | – | – | ✓ | ✓ |
| Relation predicates (rests on, lies flat, encloses: yes/no, goal met) | ✓ | – | – | – | ✓ | ✓ | ✓ | ✓ | ✓ |
| Contact and grasp events (came into contact, let go, what the close met) | ✓ | – | – | – | ✓ | ✓ | ◦e | ✓ | ✓ |
| History (): goal distances (median at the end, closest) | ✓ | – | – | ✓ | – | ◦f | ◦f | ◦g | ✓ |
| History (): reached-then-lost, held at the end, grasp counts | ✓ | – | – | – | ✓ | ✓ | ✓ | ◦h | ✓ |
Table 15. What each feedback condition renders (✓ whenever the batch has the item, ◦ partly, – never): the reference, then the conditions that change its form, its spatial representation or its temporal granularity (Table 14). aonly the node the graph raised at, in the outcome block; bonly the nodes that carry stills; corientation-valued outputs dropped; dpart names and kinds only, no geometry; eevents between start and end only, not what the close met; fobject-level (containment) goals only; gmedian distance at the end only; hheld at the end only. The stills arm’s first report has none: the reference batch it re-renders was recorded without video. Syringe Packing declares no parts, so there the reference differs from object frames only in its legend and goal lines.
Report Anatomy
Listing 13 is the opening of the reference report for the four training episodes that the first Tool Hanging reference run scored at : the task line, the legend naming the condition, the training result, the space conventions and declared parts, the episode table and the banners. One representative episode follows per failure signature (the first failing node, how the episode ended, the goals met), at most six, each followed by the other episodes that failed the same way, so a batch that fails one way costs one episode’s text.
Listing 13: Opening of the reference report, up to the first representative episode.
# Execution feedback | iteration 0
Task: `tool_hanging/toolhanging-easy-hangwrench-001` | hang the wrench on the hook
What this feedback contains: outcome + simulation feedback: state, relations; per named graph
step (each node of your workflow); space written as positions, orientations and the assets'
declared parts.
**Train result: 0/4 episodes succeeded (0%); per object 0/4 goals met.**
Positions are world coordinates in metres [x, y, z]. Orientations are given as axis directions
in world coordinates; an offset marked 'its own axes' is along the object's own box axes
(long/mid/short). Parts are declared by each asset in its own frame (below).
Declared parts | the vocabulary the goal is written and scored in:
- `wrench_1.ring` | a loop at [0.17, 0, 0], enclosure_tolerance 0.008, inner_r 0.01414 (the
entity's own frame, metres)
- `wrench_hook_1.shank` | a rod from [0.0611, 0.0042, 0.00249] to [0, 0.0042, 0.00249], radius
0.00249 (the entity's own frame, metres)
| episode | run | success | objects | steps | ended |
|---|---|---|---|---|---|
| 0 | toolhanging-easy-hangwrench-001 · repeat 0 | no | 0/1 (wrench_1 ✗) | 2534 |
policy_request (graph raised at `grasp_sg.reach`) |
| 1 | toolhanging-easy-hangwrench-012 · repeat 0 | no | 0/1 (wrench_1 ✗) | 2503 |
policy_request (graph raised at `grasp_sg.reach`) |
| 2 | toolhanging-easy-hangwrench-013 · repeat 0 | no | 0/1 (wrench_1 ✗) | 1076 |
policy_request (graph raised at `plan_carry_sg.plan`) |
| 3 | toolhanging-easy-hangwrench-014 · repeat 0 | no | 0/1 (wrench_1 ✗) | 5416 |
policy_request |
**3 of 4 episodes aborted: your graph raised instead of finishing.** Where: `grasp_sg.reach`
×2 · `plan_carry_sg.plan` ×1. Recorded error: `Node 'grasp_sg.reach' failed: RuntimeError:
the hand stopped 109.0 mm from the commanded grasp pose; closing here would grip whatever is
beside the tool`
An aborted episode is scored as a failure. Fix the raising node first.
Where the failed episodes' graphs raised: `grasp_sg.reach` ×2 · `plan_carry_sg.plan` ×1.
**1 failed episode(s) raised nothing | your graph finished and the goal was not met.**
## Episode 0 | FAILURE, 2534 steps, ended: policy_request
Per-object: 0/1 (wrench_1 ✗)
**Your graph raised at `grasp_sg.reach`: `Node 'grasp_sg.reach' failed: RuntimeError: the hand
stopped 109.0 mm from the commanded grasp pose; closing here would grip whatever is beside
the tool`**
Loop notice and history. From the report summarizes every earlier iterate: its training result, its edit, what its policy did, derived from its own recorded run, and the previous session’s own account of its change, marked as unverified. When the training batch regresses under the revert rule (Appendix § 15.2.§ 1), the loop restores the earlier iterate’s policy and the report opens with a notice, because the policy the agent now edits is no longer the one the report describes (Listing 14). Here the reverted iterate brought the wrench closer to the goal (median 0.067 m), but every episode aborted when the planner refused the waypoint engaging the hook (Finding 5), and the rule always counts such a batch as a regression.
Listing 14: Loop notice and history block of the reference report at ; the quoted account is cut after its first line.
**LOOP NOTICE | the policy in `workflow/` is iteration 0's, not the one this feedback
measured.** This batch (iteration 1: train 0%, reached the goal geometry in 0, median
closest approach 0.067 m, 4 carries) regressed against iteration 0 (train 0%, reached the
goal geometry and in 1, median closest approach 0.398 m, 1 carries), so the loop restored
iteration 0's workflow. The edits accepted at k=0 are no longer applied; their combined diff
is `history/k001.reverted.diff`.
Everything below describes the regressed policy.
## What has been tried so far
| iteration | train | edit | outcome |
|---|---|---|---|
| k=0 | 0% of 4 | bindings changed +4, loc delta +148, scripts changed +2 | applied |
What each past policy did (derived from its own recorded run):
- k=0: graphs raised at `grasp_sg.reach` ×2 · `plan_carry_sg.plan` ×1; 1 finished without
raising and missed; first recorded error: "Node 'grasp_sg.reach' failed: RuntimeError: the
hand stopped 109.0 mm from the commanded grasp pose; closing here would grip whatever is
beside the tool"; goal wrench_1.ring→wrench_hook_1.shank: median distance at the end 0.398
m, closest 0.014 m, reached the goal geometry and then lost it in 1/4, held at the end 0/4;
grasp attempts 2: met the object 2, carried 1.
The previous iteration described its own change as | its account, unverified:
> `./validate.sh` is clean (`OK`). Summary of the diagnosis and the edits:
> [...]
One Batch Under Every Condition
Since every condition re-renders the same batch, the reports compare line by line. Listing 15 shows the node grasp_sg.close of Episode 2 under each condition that renders it, Listing 16 the end of Episode 0 under the three spatial representations, and Listing 17 the per-episode report’s single unit for that episode.
Listing 15: Node grasp_sg.close, Episode 2, by condition.
(a) Reference (state + relations, parts, per node)
- **`grasp_sg.close`** (tool, ok, 19.1 s) | returned position=0.26320087909698486, ˜steps=144
- jaws closed at [0.717, 0.003, 0.746]; 0.123 m from wrench_1's centre (world ∆ [0.123,
0.000, 0.008]); offset +122 mm long, +8 mm short (its own axes); nearest declared part:
ring 48 mm; aperture 12.4 mm.
- arm 0's fingers and wrench_1 came into contact.
- arm 0's fingers and table stopped touching.
- arm 0 closed around wrench_1, but it never moved with the hand.
- the close met wrench_1.
- the jaws closed nearest wrench_1's `ring`.
- now arm 0's fingers are closed on wrench_1.
(b) Outcome + object state
- **`grasp_sg.close`** (tool, ok, 19.1 s) | returned position=0.26320087909698486, ˜steps=144
- jaws closed at [0.717, 0.003, 0.746]; 0.123 m from wrench_1's centre (world ∆ [0.123,
0.000, 0.008]); offset +122 mm long, +8 mm short (its own axes); nearest declared part:
ring 48 mm; aperture 12.4 mm.
(c) Outcome + relations
- **`grasp_sg.close`** (tool, ok, 19.1 s)
- arm 0's fingers and wrench_1 came into contact.
- arm 0's fingers and table stopped touching.
- arm 0 closed around wrench_1, but it never moved with the hand.
- the close met wrench_1.
- the jaws closed nearest wrench_1's `ring`.
- now arm 0's fingers are closed on wrench_1.
(d) State + relations, world frame
- **grasp_sg.close** (tool, ok, 19.1 s) | returned position=0.26320087909698486, ~steps=144
- jaws closed at [0.717, 0.003, 0.746]; 0.123 m from wrench_1's centre (world Δ [0.123,
0.000, 0.008]); aperture 12.4 mm.
- arm 0's fingers and wrench_1 came into contact.
- arm 0's fingers and table stopped touching.
- arm 0 closed around wrench_1, but it never moved with the hand.
- the close met wrench_1.
- now arm 0's fingers are closed on wrench_1.
(e) State + relations, object frames
- **grasp_sg.close** (tool, ok, 19.1 s) | returned position=0.26320087909698486, ~steps=144
- jaws closed at [0.717, 0.003, 0.746]; 0.123 m from wrench_1's centre (world Δ [0.123,
0.000, 0.008]); offset +122 mm long, +8 mm short (its own axes); aperture 12.4 mm.
- arm 0's fingers and wrench_1 came into contact.
- arm 0's fingers and table stopped touching.
- arm 0 closed around wrench_1, but it never moved with the hand.
- the close met wrench_1.
- now arm 0's fingers are closed on wrench_1.
(f) Per motion: the same close as a motion unit
### Motion 4 · `robot.close_gripper` · arm 0 · steps 503{625
- jaws closed at [0.717, 0.003, 0.746]; 0.123 m from wrench_1's centre (world Δ [0.123, 0.000,
0.008]); offset +122 mm long, +8 mm short (its own axes); nearest declared part: ring 48 mm;
aperture 12.4 mm.
- arm 0's fingers and wrench_1 came into contact.
- arm 0's fingers and table stopped touching.
- arm 0 closed around wrench_1, but it never moved with the hand.
- the close met wrench_1.
- the jaws closed nearest wrench_1's `ring`.
- now arm 0's fingers are closed on wrench_1.
Listing 16: End of Episode 0 by spatial representation.
(a) World frame
### At the end of the episode
- wrench_1 at [0.600, 0.018, 0.738].
- wrench_1's centre is 0.481 m from wrench_hook_1's origin (world Δ [-0.280, 0.021, -0.391]).
- wrench_1 → wrench_hook_1: goal not met.
(b) Object frames
### At the end of the episode
- wrench_1 at [0.600, 0.018, 0.738]; long axis → [+1.00, +0.00, +0.00] (world), 90° from
vertical.
- wrench_1's centre is 0.481 m from wrench_hook_1's origin (world Δ [-0.280, 0.021, -0.391]);
in wrench_hook_1's frame [0.280, −0.391, 0.021].
- wrench_1 rests on table.
- wrench_1 lies flat.
- wrench_1 → wrench_hook_1: goal not met.
(c) Part annotations (reference)
### At the end of the episode
- wrench_1 at [0.600, 0.018, 0.738]; long axis → [+1.00, +0.00, +0.00] (world), 90° from
vertical.
- wrench_1.ring to wrench_hook_1.shank: 0.398 m at the end.
- wrench_1.ring's centre relative to wrench_hook_1.shank's tip: +48 mm along it, −395 mm
vertical, +18 mm aside.
- wrench_1.ring's plane is 90° off wrench_hook_1.shank's axis.
- wrench_1 rests on table.
- wrench_1 lies flat.
- wrench_1.ring encloses wrench_hook_1.shank: no.
Listing 17: Episode 0 in the per-episode report.
### Start → end of the episode
- wrench_1.ring to wrench_hook_1.shank: 0.398 m at the end.
- wrench_1.ring's centre relative to wrench_hook_1.shank's tip: +48 mm along it, −395 mm
vertical, +18 mm aside.
- wrench_1.ring's plane is 90° off wrench_hook_1.shank's axis.
- wrench_1.ring encloses wrench_hook_1.shank: no.
- at the start: wrench_1 rests on table; wrench_1 lies flat.
- at the end: wrench_1 rests on table; wrench_1 lies flat.
Reading the listings side by side. Every report with relations says that the wrench never moved with the hand after the close; the state-only report gives the numbers without that verdict. The world-frame report gives the jaws’ offset from the wrench’s center as a world vector (0.123 m), which is an offset along the handle only because this wrench lies along world , a fact that report never states; object frames give the offset directly (+122 mm along the long axis), and part annotations add that the jaws closed 48 mm from the ring, the part the goal is written in. At the end of Episode 0, likewise, only part annotations relate the ring to the shank. The world-frame report holds the information but leaves the transform to the agent, the step Finding 4 finds unreliable. The per-episode report has no unit for the close, so a failure caused there is attributed to what the end state shows, and the per-motion report files the close under Motion 4, robot.close_gripper, which the agent must map back to its own graph step (Finding 3).
Report Size and Truncation
A report shows at most 12 outputs per node, each value cut to 60 characters, and at most 40 units per episode (the first and last four and those around the failing unit). Node errors are cut to 300 characters, a graph traceback to its last 1,500 characters, and the previous session’s account to 800 characters. The unit cap almost never binds (fewer than 1% of the reports behind Figure 15, all of them per-motion reports). Figure 15 shows how long the reports are: outcome-only reports stay near 3k characters, while state numbers and part geometry bring the reference to 21k, and Syringe Packing reports of Table 1 run to a median of 70k.
Figure 15. Length of the feedback report per condition, in characters: median (bar) and interquartile range (whiskers) over the distinct reports the improve sessions read. The gray bar pools every run of Table 1, all of which use the reference condition.
More text is not better. Length does not predict learning. World-frame and object-frame reports have the same median length (16.9k characters), yet reach 30% and 40% overall held-out success (25% and 63% on Tool Hanging). The state-only report is nearly as long as the reference (18.2k against 20.8k) yet ends below outcome only at 3.0k, 11% against 19% overall. The per-motion report is a third shorter than the reference yet reaches 90% against 45% on Tool Hanging. Across the eight conditions, the rank correlation between median length and overall held-out success is 0.24: what helps is which quantities a report makes explicit and at which unit (Findings 3 and 4), not how much it says.
Revert, Stopping and Selection Rules
Three rules decide which policy the loop carries and which one a run reports: the revert rule chooses the policy the next session edits, the stopping conditions end the run, and the selection rule picks the kept iterate. We give each rule and what it did in the runs, then relate the of Table 1 to the kept iterate. Table 16 holds the thresholds and the per-task rates.
| Tool Hanging | Syringe Packing | Gear Meshing | USB-C Insertion | Cable Routing | |
| Validation target (success) | 1 | 0.83 | 1 | 1 | 0.67 |
| in objects | 5/5 | 9/10 | 5/5 | 5/5 | 11/15 |
| Revert margin (train success) | 0.5 | 0.25 | 0.5 | 0.5 | 0.25 |
| in objects | 2/4 | 2/8 | 2/4 | 2/4 | 3/12 |
| Episode wall budget (s) | 1500 + 4200 | 1500 + 12000 | 1500 + 7200 | 1500 + 7200 | 1500 + 14000 |
| Unchanged streak | 3 iterations without an accepted edit | 3 iterations without an accepted edit | 3 iterations without an accepted edit | 3 iterations without an accepted edit | 3 iterations without an accepted edit |
| Spend per run | $60 | $60 | $60 | $60 | $60 | ||
| Wall time per run | 120 h (most runs: 30 h) | 120 h (most runs: 30 h) | 120 h (most runs: 30 h) | 120 h (most runs: 30 h) | 120 h (most runs: 30 h) |
| Fix rounds | 2 after a guard rejection, 2 after a preflight crash | 2 after a guard rejection, 2 after a preflight crash | 2 after a guard rejection, 2 after a preflight crash | 2 after a guard rejection, 2 after a preflight crash | 2 after a guard rejection, 2 after a preflight crash |
| Preflight | 1 unscored episode, 1,500 steps | 1 unscored episode, 1,500 steps | 1 unscored episode, 1,500 steps | 1 unscored episode, 1,500 steps | 1 unscored episode, 1,500 steps |
| Stop reason (% of runs) | |||||
| , iteration cap | 84 | 100 | 8 | 69 | 4 |
| , validation target | 16 | 0 | 92 | 31 | 96 |
| , iteration cap | 81 | 100 | 40 | 81 | 79 |
| , validation target | 15 | 0 | 55 | 19 | 21 |
| , perfect training streak | 4 | 0 | 5 | 0 | 0 |
| Kept iterate is (% of runs) | |||||
| 68 | 54 | 88 | 62 | 81 | |
| 26 | 29 | 30 | 62 | 21 | |
| Reverts (% of edited iterates) | 15 | 37 | 9 | 5 | 19 |
Table 16. Stopping conditions per task and common to all tasks, and what they did: how the runs stopped, how often they kept the zero-shot program , and how often they reverted the run under an edit. Success rates are per object: the validation target over the five validation instances, the revert margin over the four training instances.
Revert Rule
The revert rule compares each training batch with the incumbent, the best iterate so far, lexicographically on per-object success rate, failed episodes that reached the goal geometry, median closest approach to the goal, and grasps that carried the object, each with a hysteresis margin (Table 16; 3 episodes, 20 mm, 3 carries); a lower key never overrules the direction of the success rate, and a batch in which every episode aborted is always a regression. On a regression the next session edits the incumbent instead of the latest iterate, and its report opens with a notice naming the undone edits (Listing 14). We add the rule because in pilot runs sessions told that their last edit had regressed still built on it. The rule reads only training episodes. It fires after 18% of edited iterates (42% of these on success rate, 35% on closest approach, 21% on all-abort batches), most often where the margin is smallest relative to the batch: after 37% of Syringe Packing and 19% of Cable Routing edits (margins of 2/8 and 3/12 objects), against 5–15% on the one-object tasks.
Stopping Conditions
After each validation batch the harness checks, in order, the iteration cap, the validation target, the perfect-training streak (only in the runs without a validation set, Appendix § 11.§ 2), the unchanged streak, the spend and the wall time, and stops at the first condition that holds (Table 16); a target met in the last iteration is recorded under the cap. The validation target replaces the first passes’ stop on two perfect training batches (Appendix § 15.2.§ 4), since the agent edits against the training instances. The spend and wall time are runaway ceilings, not budgets; the unchanged streak and the spend never end a run. Most 6-iteration runs ran with a 30 h wall-time limit, and part of them with shorter episode budgets (1,400, 6,000, 2,400, 1,200–2,400 and 9,000 s in task order).
How the runs stop. Of the 3-iteration runs, 54% stop at the cap and 46% on the target, 36% of all runs already at . Nearly every Gear Meshing and Cable Routing run meets its target (92% and 96%), mostly with , and carries that reading forward, so these columns of Table 1 largely measure the zero-shot program. Syringe Packing never meets its target, so every Syringe run uses all its rounds. The 6-iteration runs, most without the expert skills, meet the target less often and later: 76% stop at the cap, 22% on the target (5% at ) and 2% after two perfect training batches.
Kept Iterate
The kept iterate is the first iterate with the highest validation success, so an edit is kept only if it raises validation success. It is in 70% of the 3-iteration runs, for three reasons that split by task (Table 16). Half of these runs meet the target with , mostly on Gear Meshing and Cable Routing. A third never succeed on a validation instance, so every iterate ties at zero: on Tool Hanging no run of DeepSeek V4, Kimi K3 or either GPT-6 setting ever validates, nor any Kimi K3 run on USB-C Insertion, the repair failure of Finding 1 at its sharpest. In the remaining 15%, almost all on Syringe Packing, validates at least as well as both edits. The 6-iteration runs keep in 34% of runs, as they rarely stop at and have five edits to improve on it. The runs without a validation set keep the revert rule’s incumbent, and one skill-library run, resumed to its iteration cap after a wall-time stop, keeps the iterate chosen at that stop.
Superseded First Passes
Three passes preceded the protocol of Appendix § 11. The first stopped after two perfect training batches and kept the iterate that trained best, and its first launch shipped the expert scripts with the reference library, so the agents copied the human solution; two of its runs enter the Opus 5, xhigh row of Table 1 (Appendix § 11.§ 2), and its Tool Hanging run stopped with 70% held-out success. The second trained on four distinct instances but still stopped and selected on them; a brief third validated on three. Since the report describes exactly the training episodes, selecting on them rewards edits fitted to those layouts, so we moved stopping and selection to five validation instances the agent never sees and, in the feedback and skill-library studies, ship only to the arm.
Models and Inference
This appendix documents the ten agent settings of Table 1: models and access (Appendix § 15.3.§ 1), call parameters (Appendix § 15.3.§ 2), reasoning controls and realized thinking (Appendix § 15.3.§ 3), and session statistics (Appendix § 15.3.§ 4), the last two bearing on Finding 1. Appendix § 15.4 gives the compute and cost of all three studies.
Models and Access
Table 17 identifies each setting by its model id, the date the id was first listed, and the dates its sessions ran. The Claude models run in the Claude Code CLI on a subscription login. Kimi K3 and DeepSeek V4 run in the same CLI against OpenRouter’s Anthropic-compatible endpoint, which keeps the prompt, tools, allowed commands and turn cap of the Claude settings and changes only the model. GPT-6 Astra and Sol run in OpenAI’s Codex CLI 0.155.1, which reads the same instruction file plus an AGENTS.md stating the workspace rules (Appendix § 11.4). The feedback and skill-library studies use the command and model of Opus 5, xhigh.
| Setting | Model id | Listed | Sessions (UTC) | Harness control | Effort xhigh | Thinking tokens Generate | Improve |
| Opus 5, xhigh | claude-opus-5 | 07-24 | 09-18–22 | – | 64% | 30.8k | 16.5k |
| Opus 5, high | claude-opus-5 | 07-24 | 09-22–24 | budget 32k | 13% | 17.8k | 21.2k |
| Opus 5, low | claude-opus-5 | 07-24 | 09-22–25 | budget 4k | 12% | 18.6k | 15.4k |
| DeepSeek V4 | deepseek/deepseek-v4-pro-0813 | 08-12 | 09-22–25 | – | 22% | 75.1k | 69.2k |
| Sonnet 5 | claude-sonnet-5 | 06-30 | 09-22–25 | – | 21% | 46.2k | 20.2k |
| Kimi K3 | moonshotai/kimi-k3 | 07-16 | 09-22–25 | – | 12% | 30.4k | 18.8k |
| Fable 5.1 | claude-fable-5-1 | 09-01 | 09-22–25 | – | 7% | 22.9k | 27.1k |
| GPT-6 Astra | gpt-6-astra | 09-04 | 09-23–25 | effort xhigh | 100% | 3.4k | 1.6k |
| Opus 5.5 | claude-opus-5-5 | 09-22 | 09-22–25 | – | 21% | 10.3k | 6.0k |
| GPT-6 Sol | gpt-6-sol | 09-22 | 09-23–25 | effort xhigh | 100% | 5.1k | 3.2k |
Table 17. Agent settings of Table 1 (Opus 5, xhigh: rounds ). Listed: first OpenRouter listing of the model id in 2026 (the DeepSeek V4 family: 04-24). Sessions: first and last session date. Harness control: the reasoning setting the harness passes (Table 18). Effort xhigh: share of sessions whose CLI logged effort level xhigh. Thinking tokens: median per session as the CLI reports them.
Inference Parameters
Table 18 gives the exact call of each CLI and every parameter the harness controls. We pass only what the CLIs expose: sampling stays at the provider defaults, every session has a 2-hour wall clock, and Claude Code caps sessions at 80 turns. From 2026-09-22, a rate-limited Claude session is rerun at once with the same model through OpenRouter, so a rate limit can change a session’s route but never its model or prompt.
Reasoning Budget
Controls and logged effort. The harness sets a reasoning control only for Opus 5, high and low (thinking budgets of 32k and 4k tokens per request) and the two GPT-6 settings (Codex effort xhigh; Table 18). Claude Code’s effort level is left to each machine’s CLI settings. Where the logs record it, it is xhigh, except for 13% of the Opus 5, xhigh sessions (high, all on 2026-09-18) and most locally run Fable 5.1 sessions (medium); the remote machines record no level. Table 17 reports the sessions logged at xhigh per setting. In the feedback and skill-library studies these are the 26% of sessions run on the local machine; the rest ran on the remote machines.
Realized thinking. The 32k and 4k budgets yield indistinguishable sessions (Table 17): median thinking of 17.8k and 18.6k tokens for initial programs and 21.2k and 15.4k for feedback rounds. The budget caps a single request, which rarely approaches it: Opus 5 thinks a median of 420 tokens per turn under the 32k budget and 387 under the 4k one. We therefore read the three Opus 5 rows as three samples of one model, not a sweep of reasoning effort (the xhigh row sets no budget and comes from 6-iteration runs, Appendix § 11.§ 2). Read this way, they support Finding 1: the three rows start further apart (41–57%) than the six settings of DeepSeek V4, Sonnet 5, Kimi K3, Fable 5.1 and GPT-6 (33–45%), yet all three reach 74–81% at @2, far above the 44–48% at which DeepSeek V4, Sonnet 5, Kimi K3 and GPT-6 Astra end.
Agent Sessions
Table 19 summarizes the agent sessions. The median session lasts 3.0 to 11.8 minutes (DeepSeek V4: 29.0), far below the 2-hour limit, and takes 31–47 turns in Claude Code (Codex: 22 and 28 tool actions).
| Setting | Fix sessions | Per session Turns | Per session Minutes | Lines per edit | Gain @0→@2 |
| Opus 5, xhigh | 7% | 37 | 8.1 | 113 | +29 |
| Opus 5, high | 10% | 42 | 8.1 | 156 | +40 |
| Opus 5, low | 14% | 38 | 8.5 | 150 | +17 |
| DeepSeek V4 | 4% | 47 | 29.0 | 30 | +6 |
| Sonnet 5 | 6% | 31 | 8.7 | 27 | +1 |
| Kimi K3 | 21% | 36 | 11.8 | 88 | +7 |
| Fable 5.1 | 6% | 44 | 11.8 | 196 | +20 |
| GPT-6 Astra | 0% | 22 | 4.7 | 63 | +5 |
| Opus 5.5 | 14% | 37 | 6.4 | 54 | +10 |
| GPT-6 Sol | 32% | 28 | 3.0 | 35 | +20 |
Table 19. Agent sessions of the settings of Table 1 (Opus 5, xhigh: rounds ): the share of fix sessions (repairing a rejected proposal or a crashing preflight), median turns (Codex: commands and file edits), minutes and lines added plus removed per accepted edit, and the change in Overall success from @0 to @2 in Table 1.
Turns and thinking do not track repair. DeepSeek V4 takes the most turns, the longest sessions and the most thinking (69.2k tokens per feedback round), yet gains 6 points, while GPT-6 Sol gains 20 in 3-minute sessions that think 3.2k. Across the ten settings, the gain’s rank correlation with median turns (; 0.35 over the eight Claude Code settings alone), minutes () and thinking per feedback round () is weak; edit size tracks it (). The Opus 5 rows and Fable 5.1 change 113–196 lines per accepted edit and gain 17–40 points; Sonnet 5 and DeepSeek V4 change 27–30 lines and gain 1–6. Repair is thus not bought with more turns or thinking (Finding 1), and the models that repair little adjust their programs only slightly (Finding 2). Opus 5.5 edits little but starts highest (67% at @0). The exception is Sol, whose whole gain is USB-C Insertion (0% to 100%), reached with edit of 6–29 lines.
Fix sessions. Most settings spend at most 14% of their sessions repairing a proposal the guard or validator rejected or a program that crashed in preflight (Appendix § 11.§ 3). Kimi K3 (21%) and GPT-6 Sol (32%) spend the most; in both, the largest single source is one Gear Meshing run whose graph aborts before its first node runs. Kimi’s is repaired in its zero-shot round and scores 90%, Sol’s never passes the preflight and scores 0% in every round. Producing a program that loads is thus a hurdle separate from the behavioral repair that the gain measures.
Compute and Cost
This appendix accounts for the study’s compute: the coding-agent sessions that write and edit the policies (Appendix H.1), the simulation that executes them (Appendix H.2), and the total LLM spend with the pilot and curriculum studies (Appendix H.3). Table 20 counts the runs, agent sessions, tokens, cost and simulation of every setting of Table 1 and of each study, with superseded passes and dropped arms in a row of their own. Neither Table 20 nor the study and pilot rows of Table 22 include the API calls that perception tools make inside the policies.
| Setting | Runs | Agent sessions | Tokens (M) Input | Tokens (M) Output | Cost (USD) | Simulator hours | Policy episodes |
| Opus 5, xhigh | 25 | 62 | 194 | 3.4 | 292 | 669 | 1,000 |
| Opus 5, high | 25 | 47 | 138 | 2.3 | 193 | 520 | 718 |
| Opus 5, low | 25 | 49 | 148 | 2.3 | 199 | 641 | 714 |
| DeepSeek V4 | 25 | 56 | 165 | 6.0 | 33 | 599 | 996 |
| Sonnet 5 | 25 | 56 | 199 | 3.7 | 112 | 806 | 978 |
| Kimi K3 | 25 | 78 | 197 | 3.4 | 198 | 828 | 1,083 |
| Fable 5.1 | 25 | 59 | 97.9 | 3.6 | 417 | 1,045 | 1,049 |
| GPT-6 Astra | 25 | 55 | 53.1 | 0.4 | 115 | 550 | 998 |
| Opus 5.5 | 25 | 49 | 107 | 1.8 | 108 | 932 | 772 |
| GPT-6 Sol | 25 | 88 | 71.1 | 0.7 | 33 | 562 | 1,140 |
| Table 1 | 250 | 600 | 1,369 | 27.8 | 1,700 | 7,152 | 9,447 |
| Feedback study | 200 | 957 | 2,519 | 36.0 | 3,187 | 7,166 | 10,098 |
| Skill library | 50 | 325 | 428 | 10.2 | 747 | 1,563 | 3,212 |
| Superseded passes and dropped arms | 131 | 576 | 1,459 | 25.7 | 1,923 | 2,030 | 4,862 |
| Total | 631 | 2,459 | 5,775 | 99.7 | 7,557 | 17,910 | 27,620 |
Table 20. Compute and cost per setting of Table 1 and per study. Agent sessions are the coding-agent calls (initial program, feedback rounds, preflight repairs); input tokens include cache reads and writes; cost is repriced at the rates of Table 21; simulator hours sum the wall time of every train, validation and held-out episode. Opus 5, xhigh counts rounds of its six-round runs and the row of superseded passes and dropped arms its later rounds; the skill library’s full-library arm is the feedback study’s reference.
Table 22. Total LLM spend of the paper’s coding agents. The first four rows are the study rows of Table 20. The pilot study covers the other Claude Code and Codex sessions on our development workstation from 2026-08-18 to 2026-09-26 (Appendix § 15.4.§ 3), priced by ccusage from the session logs (models it does not price at the rates of Table 21) or, for harness sessions whose logs had been deleted, at the cost their transcripts record. The curriculum study’s spend is reported by its authors.
Agent Cost
Pricing. Cost is repriced from each session’s tokens at the rates of Table 21, since Claude Code records Kimi K3 and DeepSeek V4 at Claude rates and the Codex CLI records no cost.
| Model | Input | Cache read | Cache write | Output |
| Opus 5 | 5 | 0.5 | 10 | 25 |
| Opus 5.5 | 4 | 0.2 | 8 | 20 |
| Sonnet 5 | 2 | 0.2 | 4 | 10 |
| Fable 5.1 | 10 | 0.25 | 20 | 50 |
| Haiku 4.5 (CLI background model) | 1 | 0.1 | 1.25 | 5 |
| Kimi K3 | 3 | 0.3 | – | 15 |
| DeepSeek V4 | 0.462 | 0.0154 | – | 1.386 |
| GPT-6 Astra | 10 | 1 | 12.5 | 50 |
| GPT-6 Sol | 2 | 0.2 | 2.5 | 10 |
Table 21. Token prices (USD per million tokens) used to reprice every agent session: the list rates at which Claude Code prices the Claude models, and OpenRouter’s list rates of 2026-09-24, during the runs, for the others (DeepSeek V4’s fell to 0.264/0.0088/0.792 by 2026-09-26). Claude Code writes a one-hour prompt cache at twice the input rate; the background model and the sessions rerun through OpenRouter write a five-minute cache at 1.25; Kimi K3 and DeepSeek V4 wrote no cache.
What drives cost. Input tokens, mostly cache reads of the session’s growing context, outnumber output tokens more than twenty-fold in every setting, and cost follows the rates more than the token counts: Fable 5.1 reads the fewest input tokens of the Claude Code settings but costs the most, at twice Opus 5’s input, cache-write and output rates, while DeepSeek V4 writes the most output tokens but is the cheapest Claude Code setting, at under a tenth of Opus 5’s rates. Repairs of rejected proposals and crashing preflights add sessions but no feedback rounds; Kimi K3 and GPT-6 Sol spend 21% and 32% of their sessions on them (Table 19). Spend does not track success: Opus 5, high and Opus 5.5 each cost less than half as much as Fable 5.1 and end above it at @2 in Table 1 (81% and 77% against 64%).
Simulation
By task. In each of the three studies, Syringe Packing and Cable Routing take most of the simulator time although they contribute fewer than half of the episodes: their episodes run the most control steps (their step caps are the two largest, Table 5), and Cable Routing is also the slowest task to simulate per step.
By split. Across splits, the evaluation schedule (Table 11) decides where the time goes. A setting of Table 1 evaluates ten held-out instances every round, so held-out episodes take the largest share of its simulator time; the 6-iteration runs of the other two studies evaluate them only at and on the kept iterate, so validation takes the largest share there.
Total LLM Spend
Table 22 totals the LLM spend of the paper: the four study rows of Table 20, the pilot study and the curriculum study. The pilot study covers the other Claude Code and Codex sessions on our development workstation, which also ran part of the study: those in its session logs, measured with ccusage, and older harness sessions, at the cost their transcripts record; older interactive sessions left no record, and the other study machines kept no logs of their sessions outside Table 20. The curriculum study’s spend is the figure its authors report. Nothing is counted twice: the study-run sessions of Table 20 are removed from the pilot study by session id, and the curriculum study ran on its authors’ machines.
Additional Results
Failure Analysis
This appendix asks where the programs behind the paper’s tables fail. We define ten failure categories (Appendix § 17.§ 1) and compare them across tasks (Appendix § 17.§ 2) and across models and feedback rounds (Appendix § 17.§ 3). A category marks where a failure becomes observable, possibly several skills after the decision that caused it (Finding 3). The harness’s own interventions (guard rejections, preflight crashes, reverts) are reported with the harness and its rules (Appendices § 11 and § 15.2).
Taxonomy and Assignment
Unit. The unit is the failed held-out object (two per Syringe Packing episode, three per Cable Routing episode, otherwise one; Table 5) of the zero-shot and @2 programs of every Table 1 run and of the kept programs of the feedback and restricted skill-library runs, scored with the final per-object rule of Appendix § 9.§ 2. Pooled columns count each held-out batch once, so an initial program kept by several seeded arms counts once.
Precedence. An episode’s failed objects take its category from the first rule that applies: (1) the outcome, step budget (U) or a diverged simulation or failed machine (I); (2) the message of the first node that raised, matched by keyword rules to P, G, K, T, B, or A for a placement check that raises; (3) a self-check abort (S) when the program routes itself to a failure end; (4) for a finished program the evaluator fails, A when the placement clause is unmet and R when only a rest, release or turning clause is. An error outranks the goal clause because a node that raises stops the program and leaves the placement unmet as a consequence. Of the final programs’ failed objects, 47% are assigned by the goal clause, 36% by an error message and 18% by the outcome or end node. Table 23 gives each rule with a verbatim example; the keyword rules were checked by eye on each category’s commonest messages. Held-out episodes store no trace, so a self-check abort cannot be traced to the check that fired.
Failures by Task
Table 24 gives the shares per task (left) and per study and reading (right). Placement is 49% of all failures and the largest category everywhere except Tool Hanging; 93% of it comes from programs that finish without an error while the object is misplaced.
| Category | Tool Hanging | Syringe Packing | Gear Meshing | USB-C Insertion | Cable Routing | All | Table 1 @0 | Table 1 @2 | Feedback | Skill library |
| P Perception | 1 | 15 | 17 | 1 | 7 | 8 | 11 | 7 | 6 | 25 |
| G Grasp | <1 | 14 | 0 | 17 | 0 | 6 | 13 | 11 | 0 | 1 |
| K Planning/IK | 45 | 1 | 0 | 18 | 0 | 13 | 16 | 16 | 10 | 0 |
| T Reach/tracking | 13 | 6 | 0 | 2 | 0 | 5 | 9 | 5 | 5 | 4 |
| A Placement | 36 | 43 | 67 | 36 | 69 | 49 | 31 | 39 | 63 | 69 |
| R Release/clause | 1 | 0 | 4 | 0 | 0 | <1 | 0 | 0 | 1 | 0 |
| S Self-check abort | 0 | 1 | 9 | 9 | 20 | 7 | 9 | 8 | 9 | 0 |
| B Program bug | 0 | 0 | 0 | 5 | 0 | 1 | 3 | 0 | 2 | 0 |
| U Step budget | 4 | 20 | 4 | 9 | 4 | 10 | 8 | 14 | 4 | 1 |
| I Infra./sim. | <1 | 0 | 0 | 4 | 1 | 1 | <1 | 1 | 0 | 0 |
Table 24. Share (%) of the failed held-out objects in each category: per task and in All over the final programs of the three studies (Table 1 @2, and the kept programs of every feedback arm, including the Outcome + object state and Outcome + relations rows of Table 14, and of the two restricted skill-library arms), each held-out batch once; then per study, split as in Table 20, with Table 1 at its zero-shot (@0) and @2 programs. Bold marks each column’s largest category.
Per task. The categories beyond placement set the tasks apart.
Tool Hanging fails mostly before the hook: 45% are planning refusals, three in five of them the planner rejecting every candidate grasp (
plan_grasp: 0/1 candidate grasps had a feasible CuRobo plan ...).
Syringe Packing adds step budget (20%), perception (15%, mostly
could not perceive 'medical syringe') and grasp failures (14%, mostly the program’s own in-hand check seeing no syringe follow the lifted hand) to placement (43%).Gear Meshing, which most programs solve, fails mostly with no gear seated (67%); most of its perception failures (17%) are programs that find no red start button (
nothing red enough to be the button is visible).USB-C Insertion spreads over placement (36%, nearly all a plug short of the insertion depth), planning refusals (18%, e.g.
no reachable transit above the plug) and grasp misses (17%); its program bugs (5%) all come from the seeded arms’ inherited initial policy, which callsusbc.receptacle_shapeon a port the tool does not describe.Cable Routing adds self-check aborts (20%, more than half to an end node named
unfinished) to placement (69%).
Failures by Model and Feedback Round
Figure 16 splits the @0 and @2 columns of Table 24 by model.
Figure 16. Category shares (%) of the failed held-out objects per Table 1 setting, for the zero-shot program (left) and the program @2 reads (right). Letters follow Table 23; rows follow Table 1.
Pooled. From @0 to @2, perception (11→7%), grasp (13→11%), reach (9→5%) and program-bug failures (3→0%) shrink while placement (31→39%) and step budget (8→14%) grow: improved programs fail later in the task. The program bugs that still reach held-out episodes escape the single preflight episode, and none survives to @2. Both growing categories mostly carry no error message, so their cause must be traced back from intermediate outcomes, the attribution problem of Finding 3.
By model. Planning refusals, flat at 16% pooled, separate the models. The three Opus 5 rows, Fable 5.1 and Opus 5.5 cut them to at most 6% at @2, leaving placement as their largest category (43–60%); on Tool Hanging, Opus 5, low, Fable 5.1 and Opus 5.5 move from failing mainly at the planner to failing mainly at the hook. GPT-6 Astra cuts them as well (to 6%) but trades them for grasp failures (13→32%). DeepSeek V4, Sonnet 5, Kimi K3 and GPT-6 Sol have 16–35% at @2; on Tool Hanging, which they leave at 0–2%, planning refusals are each model’s largest category. For DeepSeek V4, feedback repaired the perception behind 70% of its @0 Tool Hanging failures only to expose planning refusals behind all of its @2 ones. Models that start alike thus stall at different stages (Finding 1), and four of the five models that stall on Tool Hanging stall where a collision-aware planner refuses their grasp or path (Finding 5); GPT-6 Astra stalls at the hook (85% of its @2 Tool Hanging failures are placement).
Standard Deviation
Other studies. The feedback study’s kept programs fail at placement more often (63%) than Table 1’s @2 programs (39%) and never at the grasp. The restricted skill-library arms’ perception share (25%) comes entirely from the tools-only arm on Cable Routing, which segments no cable in any held-out episode.
| Model | Overall @0 | Overall @1 | Overall @2 | Tool Hanging @0 | Tool Hanging @1 | Tool Hanging @2 | Syringe Packing @0 | Syringe Packing @1 | Syringe Packing @2 |
| Human | 82 | 87 | 90 | ||||||
| Opus 5, xhigh | |||||||||
| Opus 5, high | |||||||||
| Opus 5, low | |||||||||
| DeepSeek V4 | |||||||||
| Sonnet 5 | |||||||||
| Kimi K3 | |||||||||
| Fable 5.1 | |||||||||
| GPT-6 Astra | |||||||||
| Opus 5.5 | |||||||||
| GPT-6 Sol |
Table 25.
| Model | Gear Meshing @0 | Gear Meshing @1 | Gear Meshing @2 | USB-C Insertion @0 | USB-C Insertion @1 | USB-C Insertion @2 | Cable Routing @0 | Cable Routing @1 | Cable Routing @2 |
| Human | 100 | 87 | 47 | ||||||
| Opus 5, xhigh | |||||||||
| Opus 5, high | |||||||||
| Opus 5, low | |||||||||
| DeepSeek V4 | |||||||||
| Sonnet 5 | |||||||||
| Kimi K3 | |||||||||
| Fable 5.1 | |||||||||
| GPT-6 Astra | |||||||||
| Opus 5.5 | |||||||||
| GPT-6 Sol |
Table 25 (PDF p. 45). Table 1 with run-to-run variation: mean standard deviation (percentage points) over the five runs of each entry; Human is VABAR’s Level-1 result, without runs.
Curriculum Study
Privileged-to-Perception Curriculum
This appendix provides the full experimental details for the curriculum study in § 5.4 (Figure 10 and Table 4). Section J.1 describes the two learning settings, tool access, tasks, training and evaluation protocol, and the conversion from privileged geometry to perception. Section J.2 then analyzes representative cases in which transfer is unsuccessful.
Experimental Setup
Settings and Agent
Learning settings. We compare two settings that use the same self-learning loop (§ 3), model, instructions, skill library, feedback, training layouts, and total iteration budget. They differ only in how the policy observes the scene.
Privileged perception (w/ curr.). For the first iterations, the policy observes exact simulator geometry and has no camera access. We then convert the best privileged checkpoint to a perception policy and continue learning for iterations using perception only.
Perception only (w/o curr.). The policy uses cameras and perception tools for all iterations.
MimicGen and VABAR runs use different configurations, described in Section J.1.4. Within each suite, the privileged perception and perception-only settings use the same model, probe alts_mid=1 lowance, feedback format, training and held-out layouts, iteration budget, and evaluation protocol; they differ only in the scene information available to the policy.
On MimicGen, we use Claude Fable 5.1 with (16 iterations per run). On VABAR, we use Claude Opus 5 with (10 iterations per run). We run each setting three times per task (runs 000, 001, and 002), giving curriculum runs and 21 perception-only runs.
Initial policies. Iteration 0 is a zero-shot policy written by the agent from the task instruction and skill library. In the perception-only setting, each run starts from a fixed zero-shot perception policy shared with the other perception-only studies in the paper. In the privileged setting, the agent writes a separate zero-shot policy using the privileged tool surface, because a perception policy cannot execute with the privileged tool list. On VABAR, run 000 in each setting uses the first workflow graph produced for that run, while runs 001 and 002 are generated in new sessions from the same frozen prompt and model without additional training.
Agent harness. The agent is Claude Code with temperature 0.2. It can read, write, and search files in the policy workspace, run ./validate.sh, and copy library skills into the workspace. It cannot invoke the simulator directly, except on MimicGen, where it may call ./probe.sh for probe episodes (Appendix § 19.1.5). Each improvement step starts a new session that receives the current policy, the latest feedback report, and a summary of previous iterations and code changes. On MimicGen, a session is limited to 35 turns, 10 minutes of active time, and 45 minutes total. On VABAR, a session is limited to 80 turns with no wall-clock limit, as in the other VABAR experiments. The conversion session uses the same turn limit and no wall-clock limit.
Tool Access
Privileged tools. During the privileged phase, the policy observes the scene only through simulator geometry queries: sim.list_objects, sim.get_object_pose, and sim.get_object_obb provide exact object poses and oriented bounding boxes, while sim.list_features and sim.get_feature provide world-frame geometry for named functional parts. These parts are defined by the task assets; examples include the needle handle and shaft in Threading, the stove burner in Kitchen, the machine holder and lid in Coffee, the drawer handle and floor in Mug Cleanup, and each wrench ring and hook shank in Tool Hanging. VABAR additionally provides sim.list_regions and sim.get_region_obb for regions, and sim.get_fixture_layout for multi-part fixtures without a single object pose, such as the peg centers, pinion center, and button on the Gear Meshing board.
All camera and perception tools are disabled during this phase, including RGB-D observations, Grounding DINO detection, SAM3 segmentation and tracking, and geometry helpers that recover 3D structure from masks and depth. Robot control, motion, and grasp skills are unchanged. The agent is explicitly instructed to learn task decomposition and manipulation logic that can later be executed from RGB-D observations, and not to hard-code poses observed in feedback.
Perception tools. The perception policy observes the scene only through calibrated cameras that provide RGB, depth, intrinsics, and camera pose. MimicGen uses the fixed agentview camera and a wrist camera; VABAR uses each task’s own camera set: fixed overhead, front, side, or top-down views, plus wrist cameras in Tool Hanging and USB-C Insertion. The policy can use Grounding DINO for text-prompted bounding boxes, SAM3 for masks from text, boxes, or points, and geometry utilities that back-project masks and depth into world coordinates and fit boxes, lines, and planes. All sim.* geometry queries are disabled.
In both settings, the policy never receives task flags, goal predicates, success checks, or task-specific planners. These signals are used only for scoring and feedback.
Tool enforcement. Each setting uses a fixed tool list. Before an edited policy can run, a validator checks every policy script and rejects any call outside the allowed tool surface, including calls constructed dynamically at run time. The agent must pass this check with ./validate.sh before the session ends. If the check fails, the violation is returned to the agent for up to two repair rounds.
Tasks
Table 26 lists the seven tasks used in the study. We select tasks that require both accurate state estimation and non-trivial manipulation, so that separating perception from manipulation can meaningfully affect learning.
| Suite | Task | Instruction | Stages | Episode |
| MimicGen | Coffee (D1) | Insert the coffee pod into the coffee machine and close the lid. | 7 | 1500 steps |
| MimicGen | Kitchen (D1) | Cook the bread in the pot on the stove, serve the pot with the bread retained, and finish with the stove off. | 7 | 1500 steps |
| MimicGen | Mug Cleanup (D1) | Open the target drawer, pick up the mug, place it upright inside the drawer, release it, close the drawer. | 7 | 1500 steps |
| MimicGen | Threading (D1) | Pick up the needle by its handle, align its shaft with the tripod ring, and thread it through the ring. | 5 | 1500 steps |
| VABAR | Tool Hanging (Level 2) | Hang both wrenches on the two hooks. | 7 | task budget |
| VABAR | Gear Meshing (Level 2) | Put each gear on its own peg so they mesh, then press the red button at the near end of the board. | 5 | task budget |
| VABAR | USB-C Insertion (Level 1) | Insert the USB-C plug into the receptacle. | 4 | task budget |
Table 26. Tasks used in the curriculum study. “Stages” denotes the number of ordered reference stages used for stage feedback and stage reward, including the final success stage.
Threading and Coffee require precise insertion. A failure can therefore arise from target localization, the object’s pose in the gripper, or the insertion motion itself.
Kitchen and Mug Cleanup are long, multi-stage tasks with articulated objects, so errors early in the trajectory can affect later stages.
Tool Hanging (Level 2) contains two wrenches, two hooks, and distractor tools. Each wrench must be grasped so that its ring remains available for hanging and the object remains stable during transport.
Gear Meshing (Level 2) requires placing three gears on thin pegs so that they mesh with a fixed pinion, followed by a button press.
USB-C Insertion (Level 1) requires aligning a small plug with a receptacle to within a few millimeters and inserting it to sufficient depth.
In the privileged phase, task-relevant parts such as rings, pegs, and ports are available through exact geometry, which largely removes localization error. After conversion, the same structures must be recovered from RGB-D observations, including small or low-contrast parts.
Differences Between MimicGen and VABAR Runs
For each benchmark, the two curriculum settings use the same experimental configuration. The VABAR experiments use the same setup as the other VABAR experiments in this paper: Claude Opus 5, text-only feedback, a fixed set of four training layouts, no probe episodes, and sessions of up to 80 turns with no time limit.
For MimicGen, we adapt this setup to make self-learning more tractable because its tasks present a harder learning problem. MimicGen tasks are long-horizon, multi-stage, and involve multiple objects. For example, Kitchen requires the agent to interact with the stove, pot, bread, and serving area in the correct sequence over a 1500-step episode. Moreover, the skill library contains no task-specific skills for most MimicGen tasks, so the agent must develop the task-specific skills needed to solve them during self-learning. Object configurations are also randomized over a wide range, making it easy for a policy trained on a fixed set of layouts to overfit.
We therefore use a stronger model for MimicGen (Claude Fable 5.1), provide failure-image feedback, allow more learning iterations and training episodes, resample training seeds at every iteration to reduce overfitting, and include probe episodes so that the agent can test a hypothesis before modifying its policy. Because MimicGen learning runs are longer (16 iterations), we additionally limit each session to 35 turns and 10 minutes of agent update time. This encourages each iteration to make a more localized policy update. The differences between the VABAR and MimicGen configurations are summarized in Table 27.
| MimicGen | VABAR | |
| Tasks | Coffee, Kitchen, Mug Cleanup, Threading (D1) | Tool Hanging, Gear Meshing (Level 2); USB-C Insertion (Level 1) |
| Model | Claude Fable 5.1 | Claude Opus 5 |
| Iterations (privileged + perception) | 8 + 8 | 5 + 5 |
| Iterations (perception only) | 16 | 10 |
| Training episodes per iteration | 10 | 4 |
| Training layouts | 10 new every iteration (seeds 0–159 over 16 iterations) | same 4 every iteration |
| Probe episodes | up to 2 per iteration (seeds 160–449) | none |
| Held-out | 10 layouts (seeds 450–459) | 10 layouts (layouts 012–021) |
| Episode length | 1500 steps | task time limit |
| Feedback | text+2 end-of-episode agentview frames | text only |
| Success measure | native task success | per-object success |
| Turns per session | 35 | 80 |
| Time per session | 10 min active, 45 min total | no limit |
| Cameras | agentview, wrist | fixed views per task; wrist cameras in Tool Hanging and USB-C |
Table 27. Configuration differences between MimicGen and VABAR curriculum runs. Both settings (privileged perception and perception only) use the same configuration within a suite. The conversion session uses the suite’s turn limit with no time limit.
Training and Evaluation
Training, held-out, and probe layouts. On MimicGen, each iteration trains on 10 newly sampled layouts, with training layouts not reused across iterations. The agent may additionally run up to two unscored probe episodes per iteration to test a hypothesis. Because VABAR contains fewer layouts, each iteration instead uses the same four fixed training layouts, one episode per layout, and probe episodes are disabled. In both cases, the agent is told that evaluation uses unseen layouts and that movable objects must therefore be localized at run time.
After every iteration, the policy is evaluated on 10 fixed held-out layouts that are disjoint from training. Held-out results are never shown to the agent; they are used only to construct the learning curves and to select the privileged checkpoint for conversion. Within each benchmark, the curriculum and perception-only settings use the same training sequence and the same held-out layouts.
One learning iteration. Each iteration has three stages. (1) Preflight: the current policy runs on two unscored episodes of at most 300 steps to detect program errors. If it crashes, the agent receives the error and may perform up to two repair sessions. (2) Training: the policy runs on the training layouts and a feedback report is generated. (3) Improvement: a new agent session reads the report and edits the policy; the resulting policy must pass the tool validator. Held-out evaluation of the new policy runs in parallel with the next learning iteration. We do not revert edits or stop runs early, so every run uses its full iteration budget.
Feedback. Both settings receive the same feedback structure. The report describes object states and inter-object relations using named object parts and object-centered frames, organized by the named steps of the policy (§ 3). For each episode, it also reports which ordered reference stages were reached (Table 26) and the corresponding stage reward: the number of stages completed in order before the first missed stage, divided by the total number of stages and averaged across episodes. Up to two episodes are shown in full detail, and the previous six iterations are summarized as history. MimicGen reports additionally include two end-of-episode agentview frames to provide richer feedback and further facilitate learning, whereas VABAR reports are text only. The privileged and perception-only settings therefore differ in policy observations, not in the feedback used for self-improvement.
Metrics. We report held-out success after every iteration: native task success on MimicGen and per-object success on VABAR (for example, each wrench or gear is scored separately). We also report held-out stage reward. In Figure 10, each line is the mean across the three runs and the shaded band shows standard error. In Table 4, # Skills is the number of nodes in the final policy workflow graph, where each node is one code-executing step, and Success is the mean held-out success over the last three iterations; both are averaged across the three runs.
Conversion to Perception
At the end of the privileged phase, we select the privileged checkpoint with the highest held-out success, breaking ties by stage reward and then by the latest iteration. A dedicated conversion session receives the selected policy, its most recent feedback report, and the perception tool list, together with the following instruction:
The privileged perception phase is over. This checkpoint must now run on the vision-only tool surface. Preserve its task decomposition, manipulation logic, recovery branches, and bounded diagnostics. Replace every privileged sim geometry read and every privileged perception skill with calibrated RGB-D perception using the shipped vision skills and tools. Do not hard-code object poses observed in the privileged phase. Finish only when ./validate.sh reports OK on the vision surface.
The conversion session uses the same turn limit as a normal improvement session and has no wall-clock limit. It does not execute the policy or receive new environment feedback, so the first perception iteration evaluates the converted policy exactly as produced by the conversion session. All 21 conversions passed the validator on the first attempt, with conversion costs of 12–39 minutes and $5–15 in API usage. The converted checkpoint then becomes the starting policy for the perception phase.
Failure Analysis of Unsuccessful Transfer
We further analyze the failure modes observed in the curriculum study from § 5.4. Figure 17 illustrates representative examples and reveals two distinct limitations of privileged-to-perception learning. First, policies learned with exact geometry may rely on a level of geometric precision that perception cannot reliably recover after transfer. Second, exact geometry alone may be insufficient for learning the object affordances and contact strategies required for robust manipulation.
Figure 17. Representative failures of privileged-to-perception learning. (a) Policies learned with exact geometry can rely on geometric precision that perception cannot reliably recover after transfer. In Kitchen, the narrow handle tab is difficult to localize precisely; in Gear Meshing, the short pegs are difficult to detect. (b) Exact geometry alone may be insufficient for learning robust manipulation strategies. In Tool Hanging, the policy learns an unstable wrench grasp; in USB-C Insertion, it aligns the plug but fails during contact. Purple borders indicate privileged policies, and blue borders indicate perception policies.
Dependence on privileged geometric precision. Policies learned with exact geometry can inherit manipulation strategies that are highly sensitive to localization errors after transfer to perception, as illustrated in Figure 17(a). In Kitchen, the privileged policy in one run achieves 90–100% held-out success by pinching a narrow tab on the pot handle using exact geometry. After transfer to perception, success drops to 0–30% and eventually reaches 0%. The agent repeatedly identifies the tab grasp as the source of failure, but modifies its reaching and retry behavior rather than adopting a more robust grasp. Because the usable grasp region is only a few millimeters wide, the inherited strategy is highly sensitive to perceptual error.
Gear Meshing shows a related failure in one run. Held-out success drops from 70–90% before transfer to at most 7% afterward. The inherited policy can still grasp and carry the gears, but after transfer it must recover the peg locations from vision. The policy uses the front camera, where each peg occupies only 2–3 raised depth pixels, rather than the overhead camera, where each occupies about 18 pixels (Figure 17(a)). As a result, the short pegs are often merged with the plate or missed entirely, leaving the policy without a valid placement target. In contrast, a perception-only run that uses the overhead camera reaches 97–100% held-out success. Subsequent iterations mainly try to repair peg detection, rather than changing the inherited camera choice or manipulation strategy. Together with Kitchen, this shows that privileged learning can produce effective strategies that become brittle when they depend on geometric precision that perception cannot reliably recover.
Manipulation failures despite exact geometry. Exact geometry alone may also be insufficient for understanding object affordances and selecting robust manipulation strategies, as illustrated in Figure 17(b). In Tool Hanging, the privileged state provides the exact wrench pose but does not specify where or how the wrench should be grasped for reliable transport. Consequently, the agent learns to grasp the wrench near its far end, resulting in an unstable grasp during the subsequent carry motion. Because the resulting failure becomes apparent only during transport, the agent struggles to attribute it to the earlier grasp decision and fails to learn an effective strategy. Privileged training never exceeds 10% held-out success in any of the three runs, whereas perception-only learning reaches 60–100%.
USB-C Insertion reveals a related limitation in contact reasoning. In one privileged training run, the policy consistently aligns the plug with the bore but fails during insertion because contact with the socket causes the plug to tilt and become wedged. The controller fails to recover from this contact-induced error, resulting in 0% success. These examples show that exact geometry can reduce localization error without providing the affordance understanding and contact reasoning needed for robust manipulation.
References
- [1]Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., & al., . et . (2023). Do as i can, not as i say: Grounding language in robotic affordances. Conference on Robot Learning, 287–318.
- [2]Anthropic. (n.d.). Claude Fable 5.1 and Claude Mythos 5.1 System Card. Retrieved https://www.anthropic.com/claude-fable-5-1-mythos-5-1-system-cardanthropic.com/claude-fable-5-1-mythos-5-1-system-card
- [3]Anthropic. (n.d.). Introducing claude opus 5. Retrieved https://www.anthropic.com/news/claude-opus-5anthropic.com/news/claude-opus-5
- [4]Anthropic. (n.d.). Claude Opus 5.5 System Card. Retrieved https://www.anthropic.com/claude-opus-5-5-system-cardanthropic.com/claude-opus-5-5-system-card
- [5]Anthropic. (n.d.). Claude Sonnet 5 System Card. Retrieved https://www.anthropic.com/claude-sonnet-5-system-cardanthropic.com/claude-sonnet-5-system-card
- [6]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48, 2009. doi: 10.1145/1553374.1553380.DOI
- [7]Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on robot learning, pp. 66–75. PMLR, 2020.arxiv.org/abs/1912.12294
- [8]Kaiyuan Chen, Shuangyu Xie, Letian Fu, Justin Yu, William Pacini, Sandeep Bajamahal, Hudson Kim, Jaimyn Drake, Daehwa Kim, Haoru Xue, Jonathan Francis, Christian Juette, Peter Schaldenbrand, Muhammet Yunus Seker, Ruwan Wickramarachchi, Uksang Yoo, Guanzhi Wang, Adithyavairavan Murali, Balakumar Sundaralingam, S. Shankar Sastry, Spencer Huang, Yuke Zhu, Linxi Fan, and Ken Goldberg. GaP: A graph-as-policy multi-agent self-learning harness for variational automation tasks. Conference on Robot Learning (CoRL), 2026a.arxiv.org/abs/2607.05369
- [9]Chen, K., Xie, S., Yoo, U., Zhu, X., Magner, C., Zhu, A., Kim, D., Heiden, E., Coumans, E., Macklin, M., Huang, S., & Goldberg, K. (n.d.). RoboSimStudio: An interactive toolkit for rapid Real2Sim of robot manipulation tasks. Retrieved https://robosimstudio.github.io/robosimstudio.github.io
- [10]DeepSeek-AI. (2026). DeepSeek-V4 Preview: Entering the Era of Affordable Million-Token Context. https://deepseek.com/en/news/v4-preview/deepseek.com/en/news/v4-preview
- [11]Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi Fan. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation. In Proceedings of the International Conference on Machine Learning, 2026.arxiv.org/abs/2603.22435
- [12]Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.11738
- [13]Gu, J., Xiang, F., Li, X., Ling, Z., Liu, X., Mu, T., Tang, Y., Tao, S., Wei, X., Yei, Y., Yao, X., Xie, P., Huang, Z., Chen, R., & Su, H. (2023). Manskill2: A unified benchmark for generalizable manipulation skills. International Conference on Learning Representations.
- [14]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International conference on learning representations, volume 2024, pp. 32808–32824, 2024a.arxiv.org/abs/2310.01798
- [15]Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., & Fei-Fei, L. (n.d.). Voxposer: Composable 3d value maps for robotic manipulation with language models. Conference on Robot Learning, 540–562.
- [17]Huang, W., Wang, C., Li, Y., Zhang, R., & Fei-Fei, L. (n.d.). Rekep: Spatiotemporal reasoning of relational keypoint constraints for robotic manipulation. In arXiv preprint.arXiv
- [18]Team, K. (2026). Kimi K3: Open Frontier Intelligence. In arXiv preprint. https://arxiv.org/abs/2607.24653
- [19]Li, C., Gokmen, C., Levine, G., Martín-Martín, R., Srivastava, S., Wang, C., Wong, J., Zhang, R., Lingelbach, M., Sun, J., & al., . et . (2022). BEHAVIOR-1K: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. Conference on Robot Learning.
- [20]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, pp. 9493–9500, 2023a.arxiv.org/abs/2209.07753
- [21]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pp. 9493–9500. IEEE, 2023b.arxiv.org/abs/2209.07753
- [22]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023.arxiv.org/abs/2306.03310
- [23]Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., & al., . et . (2024). Agentbench: Evaluating llms as agents. International Conference on Learning Representations, 2024, 52989–53046.
- [24]Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi “Jim” Fan, and Guanzhi Wang. ASPIRE: Agentic /skills discovery for robotics, 2026.arxiv.org/abs/2607.00272
- [25]Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Yang, S. G., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.
- [26]Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2310.17596
- [27]Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L. E., Zhang, R., Liu, W., Liang, P., Fei-Fei, L., Mao, J., & Wu, J. (2024). Embodied agent interface: Benchmarking LLMs for embodied decision making. NeurIPS 2024 Track on Datasets and Benchmarks.
- [28]Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022.arxiv.org/abs/2112.03227
- [29]Murray, M., Gupta, A., & Cakmak, M. (2025). Teaching robots with show and tell: Using foundation models to synthesize robot policies from language and visual demonstration. Proceedings of The 8th Conference on Robot Learning, 270, 4033–4050.
- [30]OpenAI. (2026). GPT-6 Astra: A New Generation of Intelligence. <a href="https://doi.org/https://openai.com/index/gpt-6-astra/" rel="noopener noreferrer" target="_blank">https://doi.org/https://openai.com/index/gpt-6-astra/</a>
- [31]OpenAI. (2026). GPT-6 Sol. OpenAI API Model Documentation. https://developers.openai.com/api/docs/models/gpt-6-soldevelopers.openai.com/api/docs/models/gpt-6-sol
- [32]Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. Robot: Science and Systemsn, 2017.arxiv.org/abs/1710.06542
- [33]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EMBODIEDBENCH: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025.arXiv
- [34]Shen, W., Kumar, N., Chintalapudi, S., Wang, J., Watson, C., Hu, E. S., Cao, J., Jayaraman, D., Kaelbling, L. P., & Lozano-Pérez, T. (2026). TiPToP: A modular open-vocabulary planning system for robotic manipulation. 2603.09971
- [35]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2303.11366
- [36]Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523–11530. IEEE, 2023.arxiv.org/abs/2209.11302
- [37]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2305.16291
- [38]Wenli Xiao, Jia Xie, Tonghe Zhang, Haotian Lin, Letian ”Max” Fu, Haoru Xue, Jalen Lu, Yi Yang, Cunxi Dai, Zi Wang, Jimmy Wu, Guanzhi Wang, S. Shankar Sastry, Ken Goldberg, Linxi ”Jim” Fan, Yuke Zhu, and Guanya Shi. ENPIRE: Agentic robot policy self-improvement in the real world, 2026.arxiv.org/abs/2606.19980
- [39]Shuangyu Xie, Kaiyuan Chen, Ziyang Chen, Simeon Adebola, Yixuan Huang, Zehan Ma, Tianshuang Qiu, Wentao Yuan, Dhruv Shah, Pannag R. Sanketi, and Ken Goldberg. Robovista: Evaluating vision-language models for diverse robot applications. In Robotics: Science and Systems (RSS), 2026a.arxiv.org/abs/2607.04610
- [40]Xie, S., Chen, K., Yoo, U., Zhu, X., Shen, M., Drake, J., Solowjow, E., Gulich, L., Millea, A., Cathomen, R., Stuyck, T., Subramanian, S., Pan, Y., McGowan, B., Benedict, M., Fan, L., Zhu, Y., Huang, S., & Goldberg, K. (2026). VABAR: A variational automation benchmark for agentic robotics. 2611.06881
- [41]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.03629
- [42]Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. VLMBench: A compositional benchmark for vision-and-language manipulation. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2206.08522











