betaFeedback

WorksComputer Science

VABAR: A Variational Automation Benchmark for Agentic Robotics

Abstract

For robots to operate reliably in commercial and industrial applications, they must persistently perform specific tasks with known variations in object shape and pose. We call this Variational Automation (VA) to differentiate these settings from fixed automation. Inspired by recent papers on Agentic Robotics, where pretrained LLM and VLM coding agents combine interpretable robot programming with the open-world adaptability of model free policies, we propose a general set of agentic robotics benchmark design criteria that includes accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency. We instantiate these principles in six manipulation task classes spanning industrial, medical, and infrastructure applications: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. We introduce the Variational Automation Benchmark for Agentic Robotics (VABAR), an open, easily replicable, simulation-based manipulation benchmark to systematically evaluating agentic robotics systems. Built fully on open-source tools, VABAR features detailed implementations using the Newton physics engine, standardized task variations, difficulty tiers, success metrics, and a library of 33 generalizable skills with human-authored programmatic reference solutions. We provide initial baseline evaluations using an open agentic manipulation harness, Graph-as-Policy (GaP), and Isaac Lab Arena.

Overview of the six manipulation task classes in VABAR

Figure 1. Overview. Inspired by industrial applications, VABAR includes six manipulation task classes: (1) Tool Sorting, (2) Tool Hanging, (3) Gear Meshing, (4) Syringe Packing, (5) Cable Insertion, and (6) Cable Routing. The benchmark includes both single-arm and bimanual robot embodiments and evaluates manipulation under specific tiers of geometric and physical variations. VABAR is open access and runs on GPU workstations and on Isaac Lab Arena.

Introduction

For robots to operate reliably in commercial and industrial applications, they must solve Variational Automation (VA) tasks, where operational objectives and workspace persist with structured variations in objects, layouts, task parameters, and execution conditions across instances. In fixed automation, robots repeatedly execute carefully engineered programs in highly structured environments, often achieving high reliability and throughput but requiring substantial human effort to design, deploy, and retune when conditions change [1]. In comparison, generalist robotics aims to perform a broad range of open-ended tasks, but current model-free end-to-end approaches struggle to achieve the speed and reliability required for commercial and industrial adoption. VA tasks lie between fixed automation and fully generalist robotics. Recent advances in agentic robotics provide a promising approach to implementing VA by combining the reliability and interpretability of classical robot programming with the adaptability of learned policies. Rather than directly mapping observations to low-level actions through a monolithic model-free policy, agentic robotic systems use multiple LLM agents to reason about task instances, generate and modify executable robot programs, and invoke perception, planning, and control modules named skills.

Existing robot manipulation benchmarks (e.g., LIBERO [2], ManiSkill [3], and SimplerEnv [4]) evaluate the robustness and generalization of language-conditioned deep visuomotor policies by offering simulation environments, human-teleoperated training episodes, and success metrics across new scene configurations. Many of these have recently been saturated by systems that achieve very high success rates and they primarily focus on household pick-and-place tasks.

We introduce VABAR, a Variational Automation Benchmark for Agentic Robotics. VABAR provides six application-oriented manipulation task classes spanning industrial, medical, and infrastructure settings: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. Each task is defined with a set of instances with explicit variation axes, difficulty levels, and success conditions. For each task, we provide a physics and photorealistic simulation environment, a set of generalizable skills to facilitate execution, and a human-authored reference solution expressed as a computation graph over the same skill library. The reference solution is executed with the same perception, planning, and execution stack available to coding agents.

We evaluate VABAR using GaP, an open agentic manipulation baseline [5], and with Isaac Lab Arena to establish baselines and to study how performance scales with task difficulty and structured variation.

This paper makes 5 contributions:

  1. Variational Automation task formulation. We introduce and formalize Variational Automation, a manipulation class between fixed automation and unconstrained general-purpose manipulation, in which persistent operational objectives must be achieved across a structured distribution of task instances.

  1. Design principles for agentic robotics benchmarks. We define practical criteria for evaluating agentic systems, emphasizing accessibility, repeatability, controlled diversity, application relevance, physical realizability, resistance to saturation, and computational efficiency.

  1. VABAR task definition We instantiate these principles in six application-oriented manipulation task classes—(1) tool sorting, (2) tool hanging, (3) gear meshing, (4) syringe packing, (5) cable insertion, and (6) cable routing—with standardized task variations, difficulty levels, success metrics, and support for single-arm Franka and bimanual YAM embodiments.

  1. VABAR simulation, Agentic Skills and Human-authored solutions For each task, we provide a photorealistic simulation environment, 33 generalizable skills that facilitate solving the tasks, and a human-authored code solution.

  1. Initial agentic manipulation baselines. We provide initial baseline evaluations using an open agentic robotic system and characterize performance as task complexity and instance variation increase, establishing reference results and identifying remaining challenges for agentic manipulation.

Robot Manipulation Benchmarks

Robot manipulation benchmarking has roots in early efforts to evaluate robotic assembly. In its discussion of benchmark tests for comparing manipulators, the U.S. National Institute of Standards and Technology defines benchmark for robotics such as the Robot Characterization Testing report [6] references Nof and Paul’s work on planning robotic assembly [7]. The Cranfield Assembly Benchmark [8] provided a standardized physical assembly test, while YCB [9] later broadened evaluation through shared object sets and protocols. NIST assembly task boards [10] further standardized tests of insertion, fastening, and gear meshing. Alongside these physical benchmarks, reusable simulation suites emerged to evaluate manipulation algorithms and learned policies. Meta-World [11] and RLBench [12] provide diverse manipulation task suites, while ManiSkill [3, 13] emphasizes generalization across manipulation skills and object geometries. CALVIN [14] studies long-horizon language-conditioned manipulation, and LIBERO [2] focuses on knowledge transfer and lifelong robot learning. VIMA-Bench [15], ARNOLD [16], and VLABench [17] further evaluate multimodal prompting, language grounding, and generalization. Beyond tabletop manipulation, BEHAVIOR [18, 19], RoboCasa [20], and ManiSkill-HAB [21] consider household and mobile manipulation, while FurnitureBench [22] and RoboTwin [23, 24] emphasize contact-rich, assembly, and bimanual tasks. RoboLab [25] provides language-guided scene and variation generation and can perform sensitivity analysis of robot policies, and RoboDojo [26] provides a comprehensive simulation and real evaluation to robot Vision-Language-Action policies. These benchmarks enable systematic evaluation across increasingly diverse tasks, environments, and embodiments, but are primarily organized around general-purpose manipulation, household activities, individual skills, or generalization across distinct tasks. VABAR focuses on application-oriented automation, where a persistent operational objective must be achieved across structured variations of the same task class over time.

Robustness and Variation in Manipulation Evaluation

A growing class of benchmarks explicitly evaluates robustness under controlled variation and distribution shift. LIBERO-Plus [27] introduces perturbations across object layouts, camera viewpoints, robot initial states, language instructions, lighting, textures, and sensor noise; LIBERO-PRO [28] varies manipulated objects, initial states, task instructions, and environments; and LIBERO-X [29] evaluates increasingly challenging spatial, object, and instruction-level shifts. Rather than treating variation as a perturbation around a nominal task, VABAR defines structured variation as an intrinsic part of a Variational Automation task: a persistent operational objective is paired with an expected distribution of variation in objects, geometry, layout, task parameters, and execution conditions. Success therefore requires reliable performance across the intended operating range of an automation application, where variation is expected rather than anomalous.

Agentic Robotics and Executable Robot Policies

Recent work has explored robotic systems in which foundation models reason over task instances, invoke external tools, and generate executable robot behaviors. Code-as-Policies [30] demonstrated that language models can compose perception and control APIs into executable programs, while subsequent agentic approaches extend this paradigm toward task decomposition, planning, tool use, execution feedback, and iterative refinement. CaP-X [31] extends this paradigm by systematically evaluating and improving coding agents for robot manipulation, emphasizing executable policy generation across diverse manipulation tasks. Recent approaches iteratively improve robot programs through feasibility checking and execution feedback. GaP [5] operates offline; it explicitly represents policies as computation graphs and refines their structure and parameters through parallel simulation rehearsal and generates ROS2-like graph for repeated online execution. ENPIRE [32] extends agentic policy improvement to repeated real-world trials, and ASPIRE [33] accumulates reusable skills through iterative program repair and exploration.

Agentic robotics is particularly relevant to Variational Automation because agentic systems can retain the interpretability and modularity of classical robot programming while adapting behavior across task instances. However, comparing such systems is difficult because they may differ substantially in internal representations, tools, perception interfaces, planning methods, and controllers. VABAR evaluates externally measurable task objectives and structured variation distributions rather than prescribing a particular agent architecture, enabling generated-code, computation-graph, tool-use, learned-policy, and hybrid systems to be compared under common conditions.

Criteria for Agentic Robotics Benchmarks

Agentic Robotics Benchmark Design Criteria

1. Relevance to agentic robotics. The benchmark should support evaluating agents that construct, test, and revise robot programs from task specifications. It should provide documented robot and tool interfaces for composing perception, planning, and manipulation capabilities, together with execution feedback for program and skill refinement during development. Evaluation should assess reliable task completion across variations without prescribing a particular program representation or agent architecture.

2. Application relevance. Tasks and variations should reflect realistic operating requirements, with benchmark results predictive of real-world success, cycle time, and intervention needs.

3. Accessibility and clarity. Benchmark environments, task specifications, assets, evaluation metrics, and variation parameters should be publicly available and clearly documented so that independent researchers can reproduce and extend the evaluation.

4. Controlled diversity. Each task should specify explicit variation axes and difficulty levels rather than rely solely on unconstrained randomization.

5. Repeatability. Task generation and evaluation should use controlled random seeds and standardized success criteria to support repeatable experiments and meaningful comparisons between methods.

6. Computational efficiency. Evaluation should be computationally efficient enough to support repeated experimentation, sensitivity analysis, and large-scale benchmarking.

7. Durability. The benchmark should support systematic extensions of variation ranges and task complexity to remain informative as robotic systems improve. Extensions should be versioned to preserve comparability with previously reported results.

Task Definition

We define a Variational Automation (VA) task class as T=(E,R,A,O,pO,S)\mathcal{T} = (\mathcal{E}, \mathcal{R}, \mathcal{A}, \mathcal{O}, p_{\mathcal{O}}, \mathcal{S}), where E\mathcal{E} defines environmental coordinate frames, R\mathcal{R} denotes the robot and gripper specification, A\mathcal{A} denotes the set of admissible initial assets, each with physical properties and collision mesh, and O\mathcal{O} denotes the space of admissible object configurations. pOp_{\mathcal{O}} defines a distribution over admissible object configurations, and S\mathcal{S} denotes the measurable success condition. Each task instance configuration τi={(aj,xj)}j=1Ni∈O\tau_i = \{(a_j, x_j)\}_{j=1}^{N_i} \in\mathcal{O}, specifies each instantiated aj∈Aa_j \in\mathcal{A} and its initial pose xj∈SE(3)x_j \in SE(3) for each of the NiN_i objects and fixtures in the scene.

Importantly, O\mathcal{O} is not defined by unconstrained scene randomization. Instead, the variation space and its sampling distribution are designed to represent meaningful changes in the operating conditions of an automation task.

We use the following principles to define admissible variations: a. Task invariance. All generated instances remain instances of the same VA task. b. Physical feasibility. Generated configurations satisfy physical constraints on object support, collision, and geometric consistency, preventing inadmissible initial scene configurations. c. Robot feasibility. Variations are bounded by the operational workspace of the specified robot R\mathcal{R}. Object and fixture configurations are restricted to reachable regions, to avoid failures caused solely by physically inaccessible configurations. d. Controlled difficulty. Each task has levels of difficulty variation regimes. Increasing difficulty progressively broadens difficulty for the robot to solve. e. Skill preservation. Variations are designed such that instances across the same difficulty levels can be solved using the same underlying set of manipulation skills or extensions.

Overview of the VABAR evaluation framework

Figure 2. Overview of the VABAR evaluation framework. Left: Tasks are specified by initial and goal configurations, an asset library, and robot definitions. Top right: Admissible task variations span objects, arrangements, and positions, with examples at two levels. Bottom right: Robot policies are evaluated across diverse manipulation tasks, including object sorting, tool hanging, gear assembly, insertion, and cable routing.

These principles define the admissible configuration space O\mathcal{O} and constrain the support of pOp_{\mathcal{O}}. An individual task instance is τi∼pO\tau_i \sim p_{\mathcal{O}}.

Task Execution Records

Task-level success indicates whether an execution achieves its objective, but does not explain where or why it fails. This is particularly important for agentic robotics, where execution feedback may be used to diagnose failures, revise programs, and introduce recovery behaviors. VABAR therefore records the execution evidence needed to relate task outcomes to both the program’s decisions and the resulting physical interaction. For agentic robotics, we define the task execution record as decomposition mapping between components.

Here, we define st\mathbf{s}_t as the true physical state of the robot and scene, available only to a “privileged” observer; ot\mathbf{o}_t is the sensor observation; s^t\hat{\mathbf{s}}_t is the system’s estimated task state; at\mathbf{a}_t is the action produced by the policy, program, or execution graph; CtC_t records physical contacts generated during execution; and et+1e_{t+1} denotes the resulting verification or execution event. We define the following mapping that forms the diagnostic flow, as shown in Figure 3:

Diagnostic execution flow

Figure 3. Diagnostic execution flow. VABAR records observations and estimated state, system actions, resulting physics states and contacts, and verification outcomes. Dashed boxes denote privileged simulator state and dotted box denote the physics object contact information used for diagnosis.

  • ot→s^t\mathbf{o}_t \rightarrow\hat{\mathbf{s}}_t: perception or state-estimation failures;

  • s^t→at\hat{\mathbf{s}}_t \rightarrow\mathbf{a}_t: program-generation, planning, skill-selection, or decision failures;

  • at→(st+1,Ct)\mathbf{a}_t \rightarrow(\mathbf{s}_{t+1}, C_t): grasping, motion-execution, collision, contact, or other physical-interaction failures;

  • (st+1,Ct)→et+1(\mathbf{s}_{t+1}, C_t) \rightarrow e_{t+1}: task verification, failure detection, and recovery triggering.

The distinction between physical state and contact is important. Object poses and velocities describe the resulting scene configuration, whereas contact records describe how that configuration arose through physical interaction. For example, a failed placement may leave an object in an incorrect pose, while its contact history can reveal whether it collided with a fixture, was supported by another object, or became jammed. VABAR records both forms of evidence so that failure attribution can be grounded in the resulting physical execution rather than only in task-level success or software error messages.

VABAR

VABAR spans six tasks across rigid-object manipulation, contact-rich assembly, and deformable-object manipulation, covering challenges in semantic perception, geometric reasoning, precision contact, and deformable-object interaction. To support these tasks with a consistent and reproducible benchmark, VABAR integrates four key components: a standardized asset library, representative single- and dual-arm robot embodiments, a unified task specification, and multi-physics simulation support.

Asset Library. VABAR provides a library of over 55 assets, spanning industrial tools, mechanical components, medical supplies, fixtures, connectors, and deformable objects. Assets are normalized in format, scale, appearance, and collision geometry to support controlled object and geometric variation across task instances.

Robot Embodiment. VABAR supports the single-arm Franka and bimanual YAM platforms, covering two representative manipulation settings. Franka provides a widely adopted platform for reproducible tabletop manipulation, while YAM enables tasks requiring coordinated bimanual interaction, particularly deformable-object manipulation.

TaskLevel 1Level 2
1 TSSingulated tools; no distractorsCluttered tools
2 THOne tool; known fixture poseMultiple tools and fixture types; randomized poses
3 GMAssemble two gearsAssemble three gears
4 SPSingulated targetsCluttered scenes
5 CIFixed, visible port; nearby connector with small misalignmentRandomized port pose; moderate connector misalignment
6 CR2–3 widely spaced pegs; short route; low turning angle4–6 moderately spaced pegs; multiple turns; moderate initial cable variation

Table 1. Task difficulty levels. 1. TS: Tool Sorting; 2. TH: Tool Hanging; 3. GM: Gear Meshing; 4. SP: Syringe Packing; 5. CI: Cable Insertion; 6. CR: Cable Routing.

Task Specification. VABAR represents tasks declaratively to specify task objects, initial configurations, variations, and success conditions.

Unified Multi-physics Support. VABAR provides unified support for rigid-body and deformable-body simulation and their interactions, with NVIDIA Newton enabling multi-physics dynamics. This allows the benchmark to span rigid manipulation, contact-rich assembly and insertion, and deformable-object tasks such as cable routing.

VABAR Tasks

(1) Tool Sorting The robot must place industrial tools into corresponding bins. Instance variation is introduced by changing the tools, their poses, and tool density.

(2) Tool Hanging The robot must pick up one or more industrial tools and hang or insert them into designated hooks, clips, or slots on a pegboard. Tools may include wrenches, screwdrivers, hammers, scissors. Task instance variation is based on tool type, the number of tools, initial tool pose, and holder pose.

(3) Gear Meshing The task input is gears and axles. The single-arm robot must pick up multiple gears, assemble them onto a axles, such that they properly mesh, and press a button to activate the mechanism. The task varies the number of gears, gear size, and initial gear poses. To ensure physically valid instances, scenes can be initialized from a valid assembled configuration before selected gears are removed and repositioned.

(4) Syringe Packing The robot must place syringes into a disposal container while avoiding distractor objects. Instance variation includes the number object and disposal box type.

(5) Cable Insertion The YAM robot must pick up a connector, align it with a USB-C port, and fully insert it. Instance variation includes the position and orientation of the connector and the port.

(6) Cable Routing The bimanual YAM robot must route a flexible cable around a specified ordered sequence of cylindrical fixtures mounted on a planar routing board. The task varies the number, position, and spacing of pegs; the ordered target sequence; cable length; initial cable configuration; cable slack and curvature; endpoint position; and the 3D pose of the routing board.

VABAR Skills

To support the evaluation of agentic robotics systems in the VABAR task sandbox, we extend the Modular Open Robot Skill Library(MORSL) [5] with reusable procedures for functional-feature manipulation, constrained placement, and deformable cable handling (Figure 4). Building on existing perception, grasp generation, and collision-aware planning primitives, the extensions provide metric feature localization, source–destination correspondence, in-hand registration, and staged feature mating. Cable-related skills estimate and track ordered 3D centerlines and support anchoring, routing, and terminal placement.

VABAR skill library showing perception and state estimation; grasp proposal and execution; motion planning and execution; and verification.

Figure 4. VABAR skill library. The 33 skills span perception, grasping, motion planning and execution, and verification. Colored cards marked “+” identify 25 added skill bundles relative to the May Graph-as-Policy library; white cards denote eight retained counterparts. Additions support functional-feature manipulation, constrained placement, cable handling, and execution checks.

Baseline Experiments

We focus on three questions: (1) how does agentic robotics success rate and cycle time change as the magnitude and dimensionality of task variation increase? (2) what are failure modes: perception, grasping, and motion? (3) can agentic systems reuse skills as task difficulty changes?

Robot Simulation Implementation

We provide two implementations of VABAR: on Newton and on Isaac Lab-Arena (ILA). The implementations have different texture, lighting, asset and robot implementations.

Newton: We develop VABAR with Newton Physics engine. To determine the success conditions, we extend Behavior Domain Definition Language (BDDL) [18] to represent both rigid and deformable objects as a common specification across the full task suite.

Isaac Lab-Arena (ILA): We provide an implementation of VABAR on top of NVIDIA Isaac Lab-Arena, a framework for composing, executing and analyzing policy evaluation benchmarks with GPU-acceleration. Isaac Lab-Arena provides tools for environment composition, including object placement based on semantic spatial relationships, multiple physics backends including Newton. It also enables running parallel evaluation on GPUs and analysis of policy robustness across environment and task variations. We execute VABAR on a cluster of NVIDIA L40 GPU nodes.

Evaluation Protocol

We report the success rate, cycle time, and failure mode rates. Success is determined by the dependency structure of each task. For tasks composed of repeated manipulation attempts that are independent by construction, such as sorting or hanging independently sampled objects, each object-level attempt is treated as a trial and success rate is the fraction of objects successfully manipulated. This avoids assigning zero success to an otherwise largely successful multi-object execution.

In tasks such as Gear Meshing, later insertions are conditioned on the states produced by earlier insertions: inserting one gear changes the geometry and contact configuration encountered by subsequent gears. Similarly, multiple cable insertions are treated as separate trials only when each insertion is independently initialized or reset; otherwise, their outcomes remain coupled through the evolving task state. We retain the complete task instance as the evaluation unit for sequential, state-dependent tasks, while optionally reporting subgoal-level success (e.g., success for each gear position) as a diagnostic measure. Cycle time is the simulated time elapsed from policy start until success, failure, or timeout.

For each unsuccessful execution, we record the observed failure outcome and assign a primary failure source when supported by execution evidence. For cascading failures, we attribute the execution to the earliest supported failure that explains the subsequent outcome. Failure-source percentages are conditioned on unsuccessful executions.

Baselines

We integrate VABAR with GaP baselines. GaP is a multi-agent harness that represents a robot policy as a computation graph whose nodes are skills from MORSL [5]. We evaluate the following systems, labeled as in Table II: (a) GaP (Human), a manually specified computation graph connecting MORSL skills, executed on both the Newton and ILA implementations (Human–Newton and Human–ILA); (b) Claude Code, a general-purpose coding agent (Claude Code with Claude Opus 5 [34]) without a robotics-specific harness or skill library; (c) GaP, where the computation graph is autonomously generated by the same model and connected to MORSL. Systems (a) and (c) share the same perception, skill, planning, and execution modules, so their gap reflects graph composition and parameterization alone. Each task–difficulty combination contains N=30N = 30 independently generated instances per evaluation condition.

Performance

Table 2 summarizes performance. Across all evaluated tasks, system performance exhibits a consistent inverse relationship with task difficulty: transitioning from low-variation regimes to multi-object, cluttered environments induces notable performance degradation alongside increased execution cycle times. This drop is particular in precision alignment and deformable object tasks, such as USB-C insertion and cable routing, where introducing geometric variation and tight physical constraints significantly decreases overall task completion. In contrast, discrete item-level manipulation tasks like tool hanging maintain relatively stable per-object success rates across difficulty tiers, though total episode times scale with object count. Overall, execution duration closely reflects task geometry and physical complexity, with single-object hanging tasks completing rapidly while multi-stage routing and cluttered tray operations require substantially longer execution windows. In addition, the same graph

TaskDiff.a-i. GaP (Human w/ Newton) Succ. (%)a-i. GaP (Human w/ Newton) Cyc. (s)a-ii. GaP (Human w/ ILA) Succ. (%)a-ii. GaP (Human w/ ILA) Cyc. (s)b. Claude w/ Newton Succ. (%)b. Claude w/ Newton Cyc. (s)c. GaP w/ Newton Succ. (%)c. GaP w/ Newton Cyc. (s)
Tool SortingLevel 147.1421.620.0555.00.0N/A26.7134.2
Level 233.7913.8––0.0N/A13.3232.3
Tool HangingLevel 186.786.167.598.316.796.233.381.5
Level 281.7158.450.0138.310.0190.810.0270.9
Gear MeshingLevel 1100.0165.987.5198.90.0N/A70.0156.6
Level 290.0235.580.0381.30.0N/A60.0284.1
Syringe PackingLevel 187.0228.852.5939.00.0N/A33.3371.1
Level 263.0472.340.01810.00.0N/A12.1652.2
Cable InsertionLevel 186.778.897.5104.90.0N/A36.779.0
Level 230.074.217.5118.80.0N/A23.380.5
Cable RoutingLevel 146.7116.050.0145.93.3120.736.750.8
Level 246.7189.85.0151.50.0N/A13.3120.3

Table 2. Task performance comparison across VABAR tasks and difficulty levels.

Metrics: Succ.: success rate (%); Cyc.: median cycle time (s) among successful trials.

running in two different framework (Newton and ILA) has different object texture, scene configuration and lighting.

Discussion

(a) Agentic harness and skill library enabling better performance. Using the same underlying language model, GaP achieves higher success rates than Claude Code in 11 of 12 task–difficulty conditions, supporting the value of a robotics-specific harness coupled with reusable manipulation skills. For example, GaP achieves 70.0% and 60.0% success on Level 1 and Level 2 Gear Meshing, respectively, whereas Claude Code achieves no successful trials. However, GaP remains below the human-authored graphs on Newton across all conditions, despite sharing the same perception, skill, planning, and execution stack. This gap suggests that reliable skill composition and parameterization remain important challenges for autonomous policy generation. These comparisons evaluate the combined system; they do not isolate the contributions of the harness and skill library.

(b) Most errors are due to motion planning. We decompose unsuccessful GaP executions into perception, grasp, and motion-planning failures. Table III reports failure sources across tasks and difficulty levels. In low-variation settings, perception and grasp generation operate with high reliability across most tasks, leaving motion planning and trajectory execution as the primary source of failure. However, as task variation increases, the dominant failure modes branch based on physical task characteristics. In cluttered scenes and deformable-object manipulation—such as syringe packing and cable routing—perception and grasping degrade significantly due to visual occlusions and unmodeled cable mechanics. Conversely, in precision contact tasks like USB-C insertion, state estimation remains robust under variation, but motion planning and fine insertion alignment overwhelmingly account for nearly all observed failures.

TaskDifficultyPerception (%)Grasp (%)Motion (%)
Tool SortingLevel 137.851.410.8
Level 256.428.714.9
Tool HangingLevel 10.00.0100.0
Level 221.428.650.0
Gear MeshingLevel 1N/AN/AN/A
Level 27.121.471.4
Syringe PackingLevel 10.00.0100.0
Level 257.90.042.1
Cable InsertionLevel 150.00.050.0
Level 20.04.096.0
Cable RoutingLevel 127.345.527.3
Level 25.331.663.2

Table III. Failure modes across VABAR difficulty levels. Failure modes: Perc.: perception; Grasp: grasp generation or execution; Motion: motion planning or trajectory execution, including insertion, placement, and routing. Percentages are conditioned on failures.

(c) Detailed physics model supports failure diagnosis. By pairing task-level verification with simulator-ground-truth inspection, VABAR helps identify the physical conditions associated with execution failures. Figure 5 illustrates two jammed syringe-disposal episodes, t01 and t29, which terminate with final verification reporting blocked and 7/8 and 6/8 syringes recorded as posted, respectively. In both examples, a syringe straddles the lid plane, with its needle end inside the cavity and its opposite end above the opening, consistent with incomplete passage into the container. The surrounding syringes occupy different configurations, highlighting execution-induced variation: earlier placements alter the geometry encountered by later insertions.

Example of an agent inspecting a Newton physics model to diagnose and fix failure due to syringes jamming inside a disposal box

Figure 5. Example: Agent inspects Newton physics model to diagnose and fix failure due to syringes jamming inside disposal box. Left: simulated syringe-disposal scene with a top-down inset. Right: Newton ground-truth front and side internal views of jammed episodes t01 and t29. Red segments highlight syringes straddling the lid plane; blue segments show other syringes.

(d) Skill reuse is extensive across diverse task classes. Despite spanning rigid-object manipulation, contact-rich assembly, precision insertion, and deformable-object manipulation, VABAR policies exhibit substantial reuse of underlying skills and tools. Core robot and perception capabilities, including end-effector pose retrieval and SAM3 [35]-based segmentation, are shared across all six task classes. Higher-level semantic skills are also reused across tasks with different objects and applications but similar manipulation structure. For example, Tool Hanging and Syringe Packing both rely on skills for held-object motion, functional-feature perception, object registration, and feature mating. This reuse suggests that increasing task diversity does not necessarily require an entirely new set of skills. Instead, VABAR exposes a central challenge for agentic robotics on selecting appropriate skills, parameterizing them for the current geometry and task state, and composing them into effective task-specific behaviors. We hope VABAR will be used by others to facilitate and compare research progress worldwide.

References

  1. [1]K. Goldberg, Should robot generalists get off their high horse? Available online, 2026.
  2. [2]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “LIBERO: Benchmarking knowledge transfer for lifelong robot learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 44776–44791.arxiv.org/abs/2306.03310
  3. [3]J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, X. Yuan, P. Xie, et al., “ManiSkill2: A unified benchmark for generalizable manipulation skills,” in International Conference on Learning Representations, 2023.arxiv.org/abs/2302.04659
  4. [4]X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Nawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, et al., “Evaluating real-world robot manipulation policies in simulation,” in Conference on Robot Learning, 2024.arxiv.org/abs/2405.05941
  5. [5]K. Chen, S. Xie, L. Fu, J. Yu, W. Pacini, S. Bajamahal, H. Kim, J. Drake, D. Kim, H. Xue, J. Francis, C. Juette, et al., “GaP: A graph-as-policy multi-agent self-learning harness for variational automation tasks,” Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2607.05369
  6. [6]N. G. Dagalakis, “Robot characterization testing,” National Institute of Standards and Technology, Gaithersburg, MD, USA, Tech. Rep. NISTIR 4510, Jan. 1991.DOI
  7. [7]S. Y. Nof and R. L. Paul, “A method for advanced planning of assembly by robots,” in Proceedings of the Society of Manufacturing Engineers Autofact-West Conference, Anaheim, CA, USA, Nov. 1980, pp. 425–435.
  8. [8]K. Collins, A. J. Palmer, and K. Rathmill, “The development of a European benchmark for the comparison of assembly robot programming systems,” in Robot Technology and Applications, Berlin, Heidelberg: Springer, 1985, pp. 187–199.
  9. [9]B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: Using the Yale-CMU-Berkeley object and model set,” IEEE Robotics & Automation Magazine, vol. 22, no. 3, pp. 36–52, 2015.DOI
  10. [10]K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokokohji, “Benchmarking protocols for evaluating small parts robotic assembly systems,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 883–889, 2020.pubmed.ncbi.nlm.nih.gov/33029555
  11. [11]T. Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on Robot Learning, 2020.arxiv.org/abs/1910.10897
  12. [12]S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “RLBench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019–3026, 2020.arxiv.org/abs/1909.12271
  13. [13]T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, “ManiSkill: Generalizable manipulation skill benchmark with large-scale demonstrations,” in Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021.arxiv.org/abs/2107.14483
  14. [14]O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7327–7334, 2022.arxiv.org/abs/2112.03227
  15. [15]Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “VIMA: General robot manipulation with multimodal prompts,” in International Conference on Machine Learning, 2023, pp. 14975–15022.arxiv.org/abs/2210.03094
  16. [16]R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.-C. Zhu, B. Jia, and S. Huang, “ARNOLD: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2304.04321
  17. [17]S. Zhang, Z. Xu, P. Liu, X. Yu, Y. Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y.-G. Jiang, and X. Qiu, “VLABench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks,” arXiv preprint arXiv:2412.18194, 2024.arxiv.org/abs/2412.18194
  18. [18]S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, et al., “BEHAVIOR: Benchmark for everyday household activities in virtual, interactive, and ecological environments,” in Conference on Robot Learning, 2021.arxiv.org/abs/2108.03332
  19. [19]C. Li, C. Gokmen, G. Levine, R. Martín-Martín, S. Srivastava, C. Wang, J. Wong, R. Zhang, M. Lingelbach, J. Sun, et al., “BEHAVIOR-1K: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,” in Conference on Robot Learning, 2022.
  20. [20]S. Nasiriany, A. Maddakuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu, “RoboCasa: Large-scale simulation of everyday tasks for generalist robots,” in Robotics: Science and Systems, 2024.arxiv.org/abs/2406.02523
  21. [21]A. Shukla, S. Tao, and H. Su, “ManiSkill-HAB: A benchmark for low-level manipulation in home rearrangement tasks,” in International Conference on Learning Representations, 2025.arxiv.org/abs/2412.13211
  22. [22]M. Heo, Y. Lee, D. Lee, and J. J. Lim, “FurnitureBench: Reproducible real-world benchmark for long-horizon complex manipulation,” in Robotics: Science and Systems, 2023.
  23. [23]Y. Mu, T. Chen, S. Peng, Z. Chen, Y. Gao, Y. Zou, L. Lin, Z. Xie, and P. Luo, “RoboTwin: Dual-arm robot benchmark with generative digital twins,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2504.13059
  24. [24]T. Chen et al., “RoboTwin 2.0: A scalable data generator and benchmark for bimanual robotic manipulation,” arXiv preprint arXiv:2506.18088, 2025.arxiv.org/abs/2506.18088
  25. [25]J. X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay, “Robolab: A high-fidelity simulation benchmark for analysis of task generalist policies,” Robotics: Science and Systems, 2026.arxiv.org/abs/2604.09860
  26. [26]T. Chen, Y. Chen, Z. Li, J. Tang, K. Su, H. Lu, W. Wan, B. Chen, S. Liu, H. Yan, et al., “RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies,” arXiv preprint arXiv:2607.04434, 2026.arxiv.org/abs/2607.04434
  27. [27]S. Fei et al., “LIBERO-Plus: In-depth robustness analysis of vision-language-action models,” arXiv preprint arXiv:2510.13626, 2025.arxiv.org/abs/2510.13626
  28. [28]X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun, “LIBERO-PRO: Towards robust and fair evaluation of vision-language-action models beyond memorization,” arXiv preprint arXiv:2510.03827, 2025.arxiv.org/abs/2510.03827
  29. [29]G. Wang, C. Zhang, Q. Liu, J. Zhang, J. Cai, J. Liu, and X. Liu, “LIBERO-X: Robustness limits for vision-language-action models,” arXiv preprint arXiv:2602.06556, 2026.arxiv.org/abs/2602.06556
  30. [30]J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in IEEE International Conference on Robotics and Automation, 2023, pp. 9493–9500.arxiv.org/abs/2209.07753
  31. [31]L. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, F.-F. Li, G. Shi, J. Wu, S. Sastry, et al., “Cap-X: A framework for benchmarking and improving coding agents for robot manipulation,” in Proceedings of the International Conference on Machine Learning, 2026.arxiv.org/abs/2603.22435
  32. [32]W. Xiao, J. Xie, T. Zhang, H. Lin, L. Fu, H. Xue, J. Lu, Y. Yang, C. Dai, Z. Wang, J. Wu, G. Wang, et al., “ENDPIRE: Agentic robot policy self-improvement in the real world, 2026. arXiv: 2606.19980.arxiv.org/abs/2606.19980
  33. [33]R. Lu, Y. Wu, E. Kou, L. Fu, W. Xiao, A. Mandlekar, Y. Xue, G. Shi, K. Goldberg, A. Chen, M. Chowdhury, Y. Zhu, et al., “ASPIRE: Agentic /skills discovery for robotics, 2026. arXiv: 2607.00272.arxiv.org/abs/2607.00272
  34. [34]Anthropic, Introducing Claude opus 5, https://www.anthropic.com/news/claude-opus-5, Accessed: 2026-09-06, Jul. 2026.
  35. [35]N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al., “Sam 3: Segment anything with concepts, 2025,” URL https://arxiv.org/abs/2511.16719, vol. 2, 2025.

Paper details

vlnc:W_fjaxrnrj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 17

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents