survey_scene Prompts with an Annotated Episode TracePlanner System Promptsurvey_scene PromptEffect of the Shared Object VocabularyLong-Term MemoryWorked Example of Long-Term Memory-Guided RecoveryBaseline Reproduction FidelityIntegration DetailsFair Evaluation ProtocolPlanner VLM Swap in BaselinesMemory-Content DecompositionMemory ChannelsResultsDiscussionFailure-Mode TaxonomyReal World ExperimentsPlanner-Level Failure AnalysisCost and Latency BreakdownModel Calls and CostLatency BreakdownBenchmark and Object-Category detailsTask Instruction FormatObject Annotation PipelineObject-Category DistributionOcclusion Threshold JustificationEpisode Generation and Seed SystemLighting ModesTool-Call Flow, Push–Grasp Selection, and Human AssessmentGrasp vs. push: contextual selectionHuman Assessment of Action DecisionsSupplementary VideoReferencesWorksComputer Science
CLARO: Clutter Aware Robot Orchestrator for Open-Vocabulary Grasping using VLM Agents
Abstract
Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware Robot Orchestrator), a VLM agent that orchestrates a set of perception and manipulation skills: open-vocabulary object detection, segmentation, 6-DoF grasping, and non-prehensile pushing to locate and retrieve a language-specified target in clutter. CLARO maintains a memory of its past actions to support long-horizon execution, and emits manipulation targets (grasp poses/3D waypoints) allowing transfer across robot arms without task-specific training. We evaluate CLARO against a range of methods on a dynamic simulation environment and standard image-based benchmarks, demonstrating strong generalization across object classes, lighting, and clutter levels. CLARO attains an average success rate of 89.1%, a relative improvement of over 16% on the strongest competing method in high cluttered scenes. We further analyze performance along grounding and grasping axes to characterize where each method succeeds and fails. Finally, we deploy CLARO on a robot arm across diverse cluttered scenes, achieving a success rate of 88.8%, and demonstrating strong sim2real generalizability.
main.pdf
Introduction
Grasping objects specified by natural-language is a longstanding goal of robotics and embodied AI [8]. An instruction like “hand me the small screwdriver behind the mug” requires grounding an open-vocabulary referred object and reasoning about its spatial position in the scene. Vision Language Models (VLMs) have brought this goal within reach: by jointly reasoning over images and language,
they form a perception backbone that grounds free-form text instructions to the robot’s visual observations [7, 28]. VLMs have been adapted for Language-Driven Robot Manipulation (LDRM) in tasks such as pick-and-place and articulated-object manipulation [2, 22].

Figure 1. Approaches for Language-Driven Robot Manipulation (LDRM) are distinguished by whether and how they close the loop with the environment. (a) Referring grasp synthesis (RGS) predicts a grasp pose in a single open-loop pass. (b) Vision language action policies close the loop at the action level, mapping observations and language to low-level motor commands. (c) Tool-calling VLM agents such as CLARO close it at the tool level with a VLM planner invoking perception and manipulation tools conditioned on a past memory of tool calls and outcomes.
Recent works use two frameworks for LDRM. Referring Grasp Synthesis (RGS) methods finetune VLMs to predict grasp poses directly from an image and a task instruction, which are then executed by a controller [2, 23, 29]. These methods are trained on large datasets comprising static RGB-D scenes in which the target is assumed to be visible and graspable from the camera view. This assumption breaks down in dynamic environments when objects are occluded and the robot must intelligently search through multiple manipulation steps. The second framework uses end-to-end Vision Language Action (VLA) models that learn a mapping from pixels and language to motor actions, and are deployed closed-loop in both simulation and real hardware [9, 14, 22]. VLAs enable fine-grained robot control and adaptability to a wide range of embodiments but are often evaluated in-distribution: on objects, scenes, and instructions close to the training data. VLAs report poor performance on unseen and open-vocabulary objects, making them unsuitable for deployment in novel environments [1, 32]. Neither framework suffices: RGS cannot search through clutter, and VLAs do not generalize beyond their training distribution, yet retrieving a target in practice requires both. Real environments such as homes, labs, and warehouses are noisy and cluttered, with targets frequently occluded, so a robust system must actively declutter the scene through a multi-step, closed-loop process of planning, grounding, and execution, all while remaining open-vocabulary and deployable without retraining.
| Method | Grasp Modality | Open-Vocab. | Closed-Loop Planning | Long-Term Memory | Non-Prehensile Manip. | Cross-Arm Deploy. |
| Referring Grasp Synthesis methods | ||||||
| CROG [30] | 4-DoF | ✗ | ✗ | ✗ | ✗ | ✓ |
| MapleGrasp [2] | 6-DoF | ✗ | ✗ | ✗ | ✗ | ✓ |
| UnoGrasp [13] | 6-DoF | ✗ | ✓ | ✗ | ✗ | ✓ |
| Language-agnostic grasp generation | ||||||
| GraspGen [19] | 6-DoF | ✗ | ✗ | ✗ | ✗ | ✓ |
| Vision–Language–Action policies | ||||||
| π0.5 [22] | 6-DoF | ✗ | ✓ | ✗ | ✓ | ✗ |
| MolmoBot [9] | 6-DoF | ✗ | ✓ | ✓ | ✓ | ✗ |
| Training-free methods | ||||||
| OWG [29] | 4-DoF | ✓ | ✓ | ✗ | ✗ | ✓ |
| ThinkGrasp [23] | 6-DoF | ✓ | ✓ | ✗ | ✗ | ✓ |
| GraspMAS [20] | 4-DoF | ✓ | ✗ | ✗ | ✗ | ✓ |
| RoboDexVLM [17] | 6-DoF | ✓ | ✓ | ✓ | ✗ | ✗ |
| CLARO (ours) | 6-DoF | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1. CLARO combines all six capabilities for general-purpose open-vocabulary grasping in clutter. Referring grasp synthesis methods are finetuned on a closed set of objects and operate single-shot (except UnoGrasp). VLA policies are closed-loop but tied to their training motion distribution and embodiment; existing training-free methods add open-vocabulary grounding and closed-loop planning but lack long-horizon memory or non-prehensile manipulation. ✓/✗ denote presence/absence of each capability. Non-Prehensile Manip. is manipulation beyond grasp-and-lift (e.g., pushing); Cross-Arm Deploy. denotes methods that deploy across robot arms without re-training.
Grounding novel objects and decluttering over multiple steps requires a system that perceives, plans, and acts in a closed loop. Tool-calling VLM agents, which show strong spatial reasoning beyond conventional VLMs, are naturally suited to this: rather than learning a monolithic policy, they invoke skills, observe outcomes, and re-plan [5, 26]. A
VLM agent in robotics can use a set of skill APIs within a closed loop safety-bounded rulebook, to invoke perception and manipulation functions, observe the environment, and re-plan [11]. Such methods are suitable for clutter manipulation precisely because robot environments are dynamic, the scene changes after every action, demanding iterative grounding, occlusion reasoning, and re-planning (Fig. 1).
We present CLARO (CLutter-Aware Robot Orchestrator), a VLM agent that turns a set of perception and manipulation skills into a general-purpose, open-vocabulary object grasper. Given a cluttered workspace and a language-specified target, CLARO generates planning traces, reasons about occlusions, and executes motion that actively declutters the scene while searching for the target. Once the object is sufficiently exposed, it executes a feasible grasp to retrieve it. CLARO requires no environment-specific finetuning and is the only system to combine open-vocabulary grounding, closed-loop planning, long-term memory, 6-DoF grasping, and non-prehensile decluttering (Tab. 1).
The main contributions of this work are:
We develop CLARO (Fig. 2), a VLM agent that plans, declutters, and grasps open-vocabulary objects in tightly cluttered workspaces using a robotic arm, and transfers across robot arms and environments without retraining.
We benchmark CLARO against open-vocabulary and trained VLM baselines, across a simulation environ-
ment spanning diverse objects, lighting conditions, occlusion levels, and standard image-based RGS benchmarks. CLARO consistently outperforms all methods, with a relative improvement of 16% in high-clutter success rate over the strongest baseline.
We deploy CLARO on a robotic arm across 80 trials covering 40 objects in two clutter levels, achieving an average success rate of 88.8%.
Related Work
Referring Grasp Synthesis (RGS). These methods leverage the joint vision-language embedding space of VLMs to predict a grasp pose directly from an RGB-D image and referring instruction by training on large-scale RGB-D object pose and grasping datasets. CROG used CLIP encoders and a lightweight decoder to train on the OCID-VLG dataset for 4-DoF grasp pose estimation [30]. MapleGrasp introduced mask-guided feature pooling to focus grasp prediction on the referred region while contributing the Ref-GraspNet 6-DoF grasping dataset [2]. UnoGrasp finetuned a VLM for multi-step obstruction reasoning over cluttered scenes through an occlusion pointing task [13]. Some methods incorporate chain-of-thought reasoning or attribute and spatial grounding to predict grasps from flexible language instructions [6, 33, 34]. While trained RGS models report strong results on in-domain objects, their performance degrades on out-of-domain objects as models finetuned on a fixed closed-vocabulary datasets lose priors about the open-world objects [2]. RGS models are evaluated on static images (where the target is assumed visible and graspable in the scene) rather than in dynamic environments.
Vision Language Action (VLA) models. VLAs map RGB observations and language to motor actions, trained on large-scale simulation and tele-operation corpora to provide fine-grained, closed-loop control [15, 22]. They excel at contact-rich, articulated and pick-and-place tasks [9], but are typically evaluated in-distribution (objects, scenes, and task variations close to their training data) and in low-clutter settings [1]. Current VLAs struggle with limited generalization to unseen tasks [32] making them unsuitable for the open-vocabulary grasping problem studied in this paper.
VLM agents for open-vocabulary grasping. These methods target the limitations of the previous two by pairing a strong open-vocabulary perception module (usually untrained to avoid dataset-specific bias) that grounds the language in 2D bounding boxes or masks with a separate extensively trained grasping module able to handle diverse geometries. OWG combines a VLM with segmentation and grasp synthesis, grounding the target via visual prompting and ranking candidate grasps, while removing obstacles when the target is blocked [29]. ThinkGrasp prompts a VLM to identify the goal object or the next object likely to reveal it and to suggest grasp locations, which it executes with a segmentation and grasping network [23]. Grasp-MAS uses compositional python program synthesis, generating executable code that chains vision tools to ground the referring expression followed by grasping [20]. RoboDexVLM [17] plans a sequence of atomic skills with a VLM, verifies each execution through depth and fingertip feedback, and re-invokes the planner, conditioned on earlier episodes, when a skill fails; it executes with a dexterous hand, and always acts on the target rather than clearing occluders first. We adapt it to our parallel-jaw setting for comparison (Supp. Sec. 3.1). Other methods add occlusion-aware 3D detection with active viewpoint planning [25] or retrieve past episodes to guide planning [16]. CLARO falls in this family: it is a training-free, closed-loop VLM agent, that incorporates long-horizon memory and non-prehensile push primitive allowing the robot to operate in cluttered scenes to find and grasp the target object.
VLM agents for general manipulation. Concurrent work extends the agentic framework beyond grasping, such as articulated-object manipulation that requires a broader skill vocabulary. VoLo [4] treats a VLA model as an interruptible tool, monitoring its rollout and halting or redirecting it when a subgoal stalls, while Maestro [24] composes perception, geometry, and control modules by writing and revising executable code, repositioning a wrist camera to obtain the views needed. In both, the target is always visible and reachable. VLMgineer [12] applies the same priors offline, placing the model inside an evolutionary search that co-designs an end-effector and the open-loop trajectory wielding it; its tools are hardware synthesised for a target action rather than modules invoked over a long horizon. Articulated manipulation is outside the scope of this work, but the toolshed is the natural place to extend it: skills such as rotate_handle or grasp_and_pull would be exposed to CLARO exactly as grasp and push are, leaving the memory and replanning loop unchanged.
Methodology
CLARO retrieves a referred target by calling perception and manipulation tools in a closed loop (Fig. 2). Open-vocabulary grasping in clutter poses three challenges: (i) An occluded target requires multi-step search through a scene that changes after every action, which CLARO handles by closed-loop replanning, re-perceiving the scene and acting on the observed outcome rather than a fixed plan; (ii) Long-horizon planning introduces drifts, where the agent can ground a rejected object or retry a failed grasp. CLARO prevents this through long-term memory over past images, tool calls, and reasoning traces; (iii) Manipulation in clutter involves removing obstructions, yet grasping each can be slow or infeasible. CLARO adds a non-prehensile push tool that can expose a target without grasping.

Figure 2. CLARO is a training-free VLM agent that retrieves a language-specified object from clutter by calling perception and manipulation tools. Given an RGB-D observation and an instruction, a VLM planner conditions on the current scene and a long-term memory of past images, tool calls, and plan/belief traces to choose the next tool. The perception tools ground the scene while the manipulation tools act on it. After each action, the scene changes and the updated observation is appended to memory.
Planner
CLARO uses Gemini 3.1 Pro as a VLM planner with access to the current RGB-D observation and its tools. At each step it selects a tool and emits a short plan and belief statement recording its intent and current understanding of the scene, such as which objects remain occluded or have already been cleared. Appended to the interaction history, these traces form a part of CLARO’s long-term memory and condition every subsequent decision, so the planner avoids the cyclic re-grounding and repeated failed grasps that arise when steps are treated independently.
Toolshed
CLARO’s planner has access to four tools necessary to ground and manipulate objects in each scene.
survey_scene. Given an RGB image, this tool returns the 2D points of all visible objects and their labels, a top-down scene graph with directional predicates (e.g. left-of, behind), and a description of the scene. It is the planner’s primary visual grounding tool, supplying context needed to identify the referred target or spot occluders whose removal would reveal hidden objects in the scene. We instantiate this tool with Gemini 3.1 Pro (same as the planner) to maintain a consistent vocabulary through RGB observations. By doing so, survey_scene’s object labels and the planner’s own reasoning are drawn from a single concept space (that the VLM learns due to its internet-scale pre-training), which reduces ambiguity in mapping the referred target in text to scene objects (Supp. Sec. 2.3). Planner is encouraged to use this tool after manipulation steps to update visual memory.
locate_object. This tool takes the RGB image and a text describing the object and returns a 2D bounding box and segmentation mask of the referred object. Referring text is matched to the corresponding entry in the latest survey_scene scene graph, and the 2D point is used to prompt SAM3 [3] to produce a mask and a tightest-fitting bounding box. This tool is a precursor to manipulation: 2D mask is lifted to 3D through the depth map which provides a point cloud with one metric point per pixel, . The mask is applied to index the object’s 3D points, which are then transformed into the robot base frame by the extrinsics .
grasp. Given an RGB-D image and an object mask, this tool uses GraspGen [19] to produce a 6-DoF grasp for the object. The planner calls it in two scenarios: when the target is visible and the robot should retrieve it, or when an occluder can be grasped and removed.
push. This tool produces a non-prehensile motion by using an RGB-D image, object mask, and a direction signal from the planner. The planner issues push with an object mask, a direction in the image frame, and a sweep distance cm (default 10); the direction is mapped to the table plane and expressed in the robot base frame through . The start point is obtained by ray-tracing from the object’s centroid opposite to the push direction, across a 2D occupancy mask of all detected objects (using 2D points from survey_scene’s scene graph passed through SAM3 [3]), until a gap large enough to lower the closed gripper is found; if no such gap exists, the tool returns failure, indicating that the clutter is too dense for a precise push. The end point is the start point displaced by the sweep distance along the push direction, and the gripper, closed into a flat paddle, follows the straight segment between them at a fixed height derived from the object’s own point cloud, repositioning obstructors that are better moved aside than grasped due to geometric constraints.
Closed-Loop Deployment
CLARO runs as a closed loop. Each tool call is executed, the environment updates, and the settled RGB-D observation returns to the planner, which replans based on the outcome. This makes the search robust to the scene changes, since failed grasps, shifted obstructors, and newly revealed objects are observed and grounded in the next step rather than ignored. The resulting tool sequence does not follow a fixed schedule: the planner decides each call from the current scene and prior outcomes (Supp. Fig. 1). We analyse the tool-call sequences CLARO produces in Supp. § 6.1.
Experiments
We evaluate CLARO across three complementary settings. First, a simulation benchmark of densely cluttered tabletop scenes measures end-to-end closed-loop performance under reproducible conditions. Second, established RGS benchmarks situate CLARO’s grounding and grasping against methods evaluated on the standard single-image grasp-detection protocol. Third, real-robot experiments test generalization across a distinct embodiment, environment, and camera viewpoint: a broader transfer setting than conventional evaluations that reuse the same robot-asset configuration across simulation and real-world experiments. We omit VLA baselines: they generalize poorly out of distribution (79.2% / 39.2% for MolmoBot / on unseen objects [9]) and are embodiment-locked, requiring retraining that conflicts with our training-free, cross-arm setup. We will release the CLARO code and simulation benchmark.
Simulation Environment and Benchmark
Environment. Our benchmark is built on the PyBullet physics engine with photorealistic Blender-based rendering for RGB images as inputs to the perception modules across our baselines. A UR5 arm equipped with a Robotiq 2F-140 parallel-jaw gripper operates over a tabletop observed by a single static overhead RGB-D camera (). The camera is fixed, and every method receives the identical observation at scene initialization. To probe robustness to appearance, each scene is rendered under two lighting modes: easy, a flat ambient source with default shading, and hard, randomized indoor HDRI environment maps that introduce directional shadows, color casts, and varying intensity. Tabletop textures are randomized per episode. Two simulator-level guards are applied to all methods. (i) A safety check runs an inverse-kinematics feasibility and collision test on every manipulation waypoint; infeasible motion is reported to the method as feedback. (ii) A perception-loop guard terminates an episode if a method issues five consecutive perception calls without an intervening manipulation action, preventing cyclic loops.
Asset library. Scenes are populated from a collection of 395 household objects sourced from BOP [21] and Digital Twin Catalogue [10] model collections, manually filtered to retain objects whose category and identity are recognizable from an overhead viewpoint without relying on a privileged angle. Each object is annotated with a name, color, and physical properties (collision and mass), and spans categories, including bottles, toys, electronics, food items, kitchenware, figurines, tools, and office supplies.
Benchmark. The benchmark comprises 600 episodes: 3 random seeds 2 lighting modes 100 episodes each (3 2 100 = 600). Supp. Fig. 5 shows representative scenes spanning the object, clutter, and lighting diversity.
Baselines. We compare against six methods that support language driven grasping: 4 VLM agents from open-vocabulary grasping (OWG [29], ThinkGrasp [23], Grasp-MAS [20] and RoboDexVLM [17], using GPT 5.4 as the planning VLM), 1 from closed-loop RGS (UnoGrasp [13]), and a naïve lower bound implemented using GraspGen [19] that executes the highest confidence grasp in the scene, sequentially removing objects without semantic reasoning (denoted by GraspGen*). Baseline implementation details are provided in Supp. Sec. 3.1. We adapt RoboDexVLM by dropping dexterous-hand retargeting and its hand-mounted camera for our fixed overhead view. All methods run with a budget of 10 manipulation actions per episode.
Evaluation Metrics
Simulation. We categorize each episode into two clutter modes, measuring performance under varying complexity. An episode is classified as high clutter if the target’s visibility is below 40% and low clutter otherwise. Visibility is the ratio of the target’s visible pixel area in the scene to its area in isolation. The 600 benchmark episodes comprise 250 low-clutter and 350 high-clutter scenes. For each clutter mode, we report four metrics; SR denotes success rate.
Grounding SR – the fraction of episodes in which the referred target is localized with a segmentation mask. This measures perception accuracy (whether the method can identify the referred object) independent of whether it is subsequently grasped or removed.
Grasping SR – the ratio of successful to attempted grasps, averaged over all episodes. This metric isolates grasping accuracy, independent of target grounding.
Task SR – fraction of episodes in which the target is grasped and placed in a receptacle basket. This is the primary metric used to evaluate open-vocabulary grasping.
Action Steps – average manipulation actions (grasp+push) per episode; lower value indicates faster solution.
Static RGS. We use the standard grasp detection metric: a predicted grasp is correct if the 2D grasp rectangle overlaps the ground-truth grasp with IOU and its orientation error is below , reported as a percentage [20].
Results
Our evaluation studies three questions. (RQ1) How does CLARO perform compared to existing open-vocabulary methods and how does its performance vary with scene complexity and clutter? (RQ2) How do individual components (grounding/grasping) contribute to the overall task performance under varying clutter? and (RQ3) How does CLARO compare in the standard RGS evaluation criteria on static image benchmarks and annotated grasp poses? We answer RQ1 and RQ2 in simulation (Sec. 5.1), RQ3 on RGS benchmarks (Sec. 5.2), and isolate the contribution of CLARO’s components through ablations (§ 2.2.1).
Results on the Simulation Benchmark
Table 2 reports the overall results. CLARO attains the highest task success rate in both clutter regimes (89.5% low, 88.6% high) and, unlike every baseline, does not degrade significantly as clutter increases (RQ1). The strongest baseline, OWG, falls from 83.3% to 76.3%, a 7-point drop, while CLARO yields a relative improvement of roughly 16% in high clutter compared to baselines. ThinkGrasp reports stable performance across clutter but with lower task success rates (), and the finetuned and multi-agent baselines (UnoGrasp, GraspMAS) trail substantially.
| Method | Grounding SR. (%) ↑ Low | Grounding SR. (%) ↑ High | Grasping SR. (%) ↑ Low | Grasping SR. (%) ↑ High | Task SR. (%) ↑ Low | Task SR. (%) ↑ High | Action Steps ↓ Low | Action Steps ↓ High |
| GraspGen* [19] | 36.2 | 35.4 | 48.8 | 42.2 | 19.2 | 9.4 | 8.8 | 9.2 |
| GraspMAS [20] | 62.8 | 55.0 | 29.2 | 25.8 | 27.2 | 16.7 | 1.3 | 1.1 |
| UnoGrasp [13] | 52.7 | 46.1 | 32.5 | 18.8 | 34.2 | 23.8 | 8.7 | 9.2 |
| ThinkGrasp [23] | 73.8 | 68.8 | 34.5 | 19.2 | 76.2 | 75.3 | 5.4 | 6.4 |
| OWG [29] | 76.1 | 71.4 | 52.5 | 26.9 | 83.3 | 76.3 | 2.7 | 3.9 |
| RoboDexVLM [17] | 70.2 | 65.6 | 38.0 | 28.5 | 60.4 | 49.2 | 6.5 | 8.6 |
| CLARO (ours) | 88.2 | 86.3 | 60.4 | 45.3 | 89.5 | 88.6 | 3.0 | 4.9 |
Table 2. CLARO achieves the highest task success rate in both low and high clutter, and unlike the baselines does not degrade as clutter increases. GraspMAS reports fewest action steps because its planner VLM rejects most grasp proposals before execution (≈ 70% of episodes terminate without a single grasp attempt). Bold: best; _underline_: second best. GraspGen* is a naive grasp-only baseline that calibrates the task difficulty of our benchmark: simulating a random-pick grasping model only achieves a 19.2% task SR even in the easier low clutter regime, highlighting the need for occlusion planning and grounding as an important component of open-vocabulary grasping.
Grounding. In high clutter scenes, CLARO leads Grounding SR (86.3% vs. OWG’s 71.4% and ThinkGrasp’s 68.8%), and the gap widens relative to low clutter: every method grounds less reliably as occlusion increases, but CLARO degrades the least (88.2% 86.3%, a 1.9-point drop, vs. OWG’s 4.7-point drop 76.1%71.4%). OWG labels SAM3-segmented objects and prompts a VLM to select the target, but over-segmentation under clutter misleads this choice. ThinkGrasp prompts LangSAM [3, 18] with VLM-generated text, where naming mismatches often yield no detection. CLARO improves grounding performance with a shared object vocabulary across planning and grounding. Replacing the pointing tool in survey_scene with Molmo 2 [7] (separate object vocabulary between planner and grounder) lowers Grounding SR from 88.2/86.3 to 77.2/71.6; the steeper drop in high clutter indicating that preserving object identity across module boundaries matters more as occlusion increases (Supp. § 4.3).
Grasping. CLARO leads in both clutter regimes and the separation is largest in high clutter. Grasp reliability also degrades less steeply for CLARO (15-point drop) than for OWG (26-point drop) as clutter increases. CLARO’s 6-DOF grasping pipeline, combined with the option to push objects into graspable configurations, lets it sustain reliable grasps where other baselines collapse.
Action efficiency. CLARO uses more steps than OWG (4.9 vs. 3.9 in high clutter), which reflects the additional pushing primitive. We report this metric to show that CLARO’s gains come not from manipulating more, but from more focused manipulation through decluttering the target before grasping (Supp. Fig. 4). In successful episodes, CLARO displaces 3.8 occluders in low clutter and 6.6 in high clutter. These exceed the corresponding Action Steps because a single push often shifts several neighbouring objects at once. Pushes are excluded from Grasping SR and reported here: CLARO issues 1.6 and 2.6 pushes per episode in low and high clutter. CLARO’s closed-loop planning averages 34.8 () model calls per high clutter episode. We provide cost/latency analysis in the Supp. Sec. 6.
Training-free vs. finetuned models. UnoGrasp uses a finetuned VLM that performs obstruction reasoning by pointing to objects to remove. We reproduce this setup by pairing it with SAM3 segmentation and a GraspNet grasp generator (similar to authors’ robot experiments). Its Task SR is poor even in low clutter (34.2%), driven by the lowest Grounding SR among all methods: finetuning on static RGB-D datasets encodes its object and environment biases, which do not transfer to novel scenes in our benchmark. Training-free methods (OWG, ThinkGrasp, CLARO) ground with open-world VLM backbones, generalizing better to unseen objects. The same effect appears on the grasping side: GraspMAS grounds well via a strong VLM (GPT 5.4), but its grasp model, trained on RGS datasets, yields a low Grasping SR and hence a low Task SR. Our results show that pairing open-set detectors with open-set grasp generators is a robust recipe for open-vocabulary grasping.
Results on Static RGS Benchmarks
The simulation benchmark highlights CLARO’s interactive abilities, which are not measured in static single-image benchmarks. To verify that our VLM agentic design does not compromise basic grounding competence, we evaluate on two RGS benchmarks (Tab. 3) under the standard grasp-detection metric. CLARO attains the highest success rate on both OCID-VLG (67.2%) and GraspAnything++ (76.5%).
Ablations
We ablate the capabilities that distinguish CLARO from existing open-vocabulary grasping methods: non-prehensile pushing, long-term memory, closed-loop replanning, and VLM based planning, which we replace with a hard-coded survey→locate→grasp/push sequence (VLM decides grasp or push). We also ablate over CLARO’s system design choices, GraspGen for GraspNet as the grasp tool and Gemini 3.1 Pro or the open-weight Qwen3.5-397B-A17B for GPT 5.4 as the planner/grounder in survey_scene, over the same 600 episodes (Tab. 4). Without push, high-clutter Task SR falls 28.6 points (88.6 → 60.0) and Action Steps rise as the agent falls back on repeated grasps to clear obstructor. Without long-term memory, the planner oscillates, frequently calling perception tools rather than manipulating the scene until our perception-loop guard terminates the episode (1.9/3.8 Action Steps); removing closed-loop replanning reduces performance (56.6), as the agent retries grasps whose failure it no longer observes. A hard-coded fixed planner, instead of a VLM, is the largest drop of all (62.5/54.8) showing that CLARO’s tool orchestration rather than the manipulation primitives strengthens the performance. Varying system design choices affects performance marginally, with every variant still exceeding all baselines (Tab. 2): GraspNet gives 82.1/80.9, still outperforming OWG and ThinkGrasp, while GPT 5.4 and the open-weight Qwen3.5-397B-A17B based planner reports 83.8/78.2 and 86.0/85.2 success rates indicating that CLARO’s performance is not tied to a particular tool choice.
Table 3. On standard benchmarks, CLARO exceeds baselines despite being a training-free agent. Success rate (%) measured under the RGS protocol (Sec. 4.2). CLARO attains the best score on both OCID-VLG and GraspAnything++ benchmarks. OWG, ThinkGrasp, GraspMAS are evaluated with our implementation; ViperGPT is quoted from the original paper.
| Variant | Task SR (%) ↑ Low | Task SR (%) ↑ High | Action Steps ↓ Low | Action Steps ↓ High |
| CLARO (full) | 89.5 | 88.6 | 3.0 | 4.9 |
| w/o push tool | 77.4 | 60.0 | 6.6 | 7.5 |
| w/o long-term memory | 71.2 | 52.9 | 1.9 | 3.8 |
| w/o closed-loop replanning | 74.9 | 56.6 | 3.1 | 4.1 |
| w/o VLM based planning | 62.5 | 54.8 | 5.4 | 6.2 |
| GraspNet as grasper | 82.1 | 80.9 | 4.5 | 5.0 |
| GPT 5.4 as planner | 83.8 | 78.2 | 3.3 | 5.2 |
| Qwen3.5-397B-A17B as planner | 86.0 | 85.2 | 4.3 | 7.0 |
Table 4. CLARO’s core capabilities drive its clutter robustness: removing long-term memory, closed-loop replanning, or pushing drops high-clutter Task SR upto 36 pp. Replacing the grasp generator (GraspGen → GraspNet) or the VLM planner (Gemini 3.1 Pro → GPT 5.4 or the open-weight Qwen3.5-397B-A17B) causes modest drops. _w/o VLM based planning_ runs with a hard-coded loop where the VLM only decides when to grasp/push.
Real-Robot Experiments
Setup. We used a Franka arm, mounted on a tabletop and equipped with a parallel gripper. A fixed ZED2 RGB-D camera captures images of the tabletop in front-view. We use 40 objects to create cluttered scenes for our experiments (Fig. 3). We compare CLARO with OWG and ThinkGrasp.
Protocol. Each method is evaluated on 16 tasks, with 5 independent trials. Out of the 16 tasks, 8 correspond to low clutter scenes (≤7 objects well spaced) and 8 correspond to high clutter (≥8 objects and densely placed). Thus, we run 80 experiments per method and report the overall task success rate, average number of action steps with range of deviation. In each task, the robot is given a natural-language target object and an initial RGB-D observation of the cluttered tabletop. Each method operates in closed loop until the target is successfully retrieved, the method declares failure, or its grasp budget is exhausted. All methods are allowed up to five grasp attempts per episode.

Figure 3. A robot rollout of CLARO retrieving an occluded target. The task is to grasp the can of cherries, initially hidden behind other objects. CLARO first calls survey_scene to map the pile (T=1), then alternates between locate_object and grasp to remove the occluders that block the target (T=2–5), before localizing and grasping the can of cherries once exposed (T=6–7).
Table 5. Results on a Franka arm ( per cell: 8 low/high clutter scenes × 5 trials). Task SR is reported with 95% confidence intervals from a scene-clustered bootstrap. CLARO outperforms both baselines in high clutter (paired Wilcoxon signed-rank over 8 scenes: vs OWG, vs ThinkGrasp).
Results. Table 5 shows that CLARO achieves the highest task success at both clutter levels, improving over the strongest baseline by 27.5 points in low clutter (97.5 vs. 70.0) and 30.0 points in high clutter (80.0 vs. 50.0). Fig. 3 shows a sample task from our experiments. The low-to-high clutter gap is wider on the real robot than in simulation, for two reasons. First, the camera observes the table from an oblique front view: this aids language grounding but makes manipulation harder, as feasible grasps require accurate 3D alignment whereas top-down grasps tolerate slight misalignment. Second, our manually designed scenes induce stronger occlusion than the simulator, including cases where the target is not visible from the initial view.
The baseline ranking reverses relative to simulation: ThinkGrasp outperforms OWG in real-world experiments. This is a consequence of the oblique viewpoint. OWG’s grasp generator predicts 4-DoF grasps, which need to be orthographically reconstructed to 6-DoF grasps introducing additional noise. ThinkGrasp’s 6-DoF grasps remain feasible from the same view. All methods deteriorate in high clutter, where heavy occlusion and long-horizon planning make the task difficult. Stronger planners and richer manipulation tools are promising directions to close this gap.
Conclusion
We presented CLARO, a training-free, closed-loop VLM agent for open-vocabulary grasping in clutter. By equipping a VLM planner with a compact set of perception and manipulation tools while preserving the full interaction history as long-horizon memory, CLARO searches, declutters, and retrieves a language-specified target without any environment or embodiment specific finetuning. On our 600-episode simulation benchmark, CLARO achieves the highest task success in both low and high clutter and, unlike all baselines, does not degrade as clutter increases, improving over the strongest competing method by in high clutter. Real-robot experiments on a Franka arm establish strong sim2real generalizability, demonstrating CLARO can transfer to different robot arms, camera, and object sets.
Practical takeaways for VLM agents in robotics. Long-horizon memory is effective when provided to VLMs as structured outcome state rather than visual replay. Removing memory costs 35.7 pp of high-clutter Task SR, and decomposing the retained context into its textual and visual channels (Supp. Sec. 4) locates that benefit: discarding every past frame while keeping the tool-result trace costs 8.2 pp, whereas keeping the frames and discarding the text costs 26.9. Second, a cheap non-prehensile primitive is worth more than a stronger grasp model, and its value grows with occlusion: removing push costs 12.1 pp in low clutter but 28.6 in high. Since a grasp-only toolset appears adequate in light clutter, VLM-agent evaluations should be stratified by target visibility rather than reported in aggregate. Third, reliable grounding does not imply reliable manipulation: from low to high clutter CLARO’s Grounding SR falls only 1.9 pp (88.2→86.3) while per-attempt Grasping SR falls 15.1 (60.4→45.3), and on the real robot grasp and push execution account for 80.4% of failure events against 19.6% for segmentation (Supp. Tab. 4).
Limitations and future work. CLARO operates from a single camera, adopted for fair comparison with our single-view baselines. This limits point-cloud density for grasp estimation and prevents the agent from choosing where to observe. Active view selection could further improve grounding and grasping under occlusion. CLARO’s planner may select any tool within its safety and budget constraints, and it can perform redundant actions, such as running survey_scene after a grasp that leaves the scene unchanged. Re-grounding only the objects near the contact point can reduce such redundancies. We leave metrics that reduce this planning overhead to future work. Retained frame history also dominates prompt-token growth; since the structured outcome trace carries most of the memory benefit (Supp. Sec. 4), pruning frames under a bounded window or summarization policy is a natural extension to longer horizons. CLARO manipulates objects without object specific physical constraints, such as avoiding fragile items. Extending to object-aware manipulation preferences is a promising direction for safety-critical settings.
CLARO: Clutter-Aware Robot Orchestrator for Open-Vocabulary Grasping using VLM Agents — Supplementary Material
Sungsu Kim; Vineet Bhat; Prashanth Krishnamurthy; Ramesh Karri; Farshad Khorrami
supp.pdf
Overview of Supplementary Material
This supplementary provides the implementation detail, additional experiments, and analyses that support the main paper’s claims. Sec. 2 gives the full planner and survey_scene prompts, a precise specification of CLARO’s long-term memory, and an annotated episode trace. Sec. 3 documents how each baseline was reproduced in our unified environment. Sec. 4 decomposes the memory ablation to identify which component drives the 36 pp (main paper, table 4) effect. Sec. 5 presents a failure-mode taxonomy for the real-robot runs, Sec. 6 reports the per-episode cost and latency referenced in the main text, and Sec. 7 details the benchmark instructions, object pool, occlusion threshold, and seeding. § 6.1 analyzes how CLARO uses its tools: when the planner pushes rather than grasps, and how CLARO’s decisions compare with human judgment. A supplementary video (§ 6.2) shows real-robot successes and failures.
Planner and survey_scene Prompts with an Annotated Episode Trace
Planner System Prompt
CLARO uses a VLM planner to choose the next high-level action from the current RGB observation and the accumulated interaction history. The planner is given access to four tools: survey_scene, locate_object, grasp, and push. We provide the full system prompt used in our simulation experiments in Fig. 1.
survey_scene Prompt
The survey_scene tool uses a VLM to convert the current top-down RGB observation into a structured scene graph. This scene graph gives the planner a shared object vocabulary for later grounding calls: visible objects are assigned names, descriptions, normalized image locations, and spatial relations such as on_top_of, partially_occluded_by, and next_to. The full survey_scene system prompt is shown in Fig. 2.
This prompt is designed to make perception and planning share the same object-level representation. The planner reasons over the object names, descriptions, and spatial relations produced by survey_scene, and then passes the same natural-language object references to locate_object for precise mask grounding. This shared vocabulary reduces mismatches between high-level planning and low-level grounding, especially in cluttered scenes where the target may be partially occluded and must be identified through its relations to surrounding objects.
Effect of the Shared Object Vocabulary
To isolate the contribution of the shared object vocabulary, we decouple naming from grounding while holding the remainder of the pipeline fixed: the Gemini 3.1 Pro component supplying object identity to grounding is replaced by the open-weight model Molmo 2-8B, with SAM3, the planner, and all 600 benchmark episodes unchanged. Grounding SR falls from 88.2/86.3 to 77.2/71.6 (low/high clutter). Failure inspection shows cross-model naming drift: Molmo can reduce a specific reference such as “coffee can with MASTER written on it” to the coarser “blue can,” making nearby objects harder to disambiguate. The drop is larger in high clutter, where less of the target is visible: with its distinguishing features occluded, a coarse label such as “blue can” is more readily matched to a co-present object than to the target itself.

Figure 1 (PDF p. 12). Full planner system prompt used by CLARO in simulation experiments.
Long-Term Memory
At each step, CLARO maintains two buffers: the VLM-visible conversation memory conversation and the structured execution log action_history. The memory field passed back to the VLM is conversation. Before the VLM call at step , conversation contains the previous turns , including prior RGB observations rgb_1, , rgb_{t-1}, user prompts, and VLM JSON responses. The current call additionally receives the current RGB observation rgb_t and prompt user_prompt_t.
After the VLM is called at step , CLARO appends the current user turn and the VLM response to conversation:
{"role": "user",
"text": user_prompt_t,
"images": [rgb_t]}
{"role": "assistant",
"text": json(vlm_response_t),
"images": []}
Therefore, after step , the VLM-visible memory contains the sequence of past observations, prompts, and responses up to that step. Let and . Then
where denotes conversation after step .
The VLM response is a JSON tool-call record. It stores the selected tool, the belief string, the plan string when emitted, and the tool-specific arguments:
vlm_response_t =
{
"tool": tool_t,
"belief": belief_t,
"plan": plan_t,
"...tool_specific_args...": "..."
}
For example, locate_object includes object_name, grasp includes object_id, and push includes object_id, direction, and optionally distance_cm. Thus, the tool call, tool arguments, belief trace, plan trace, current image, and prompt are stored in conversation and remain visible in later VLM calls.
After the selected tool is executed, CLARO appends one tool-specific record to action_history. This record stores the executed tool, belief string, success/failure outcome, and tool-specific execution details:
{"step_number": t,
"tool": tool_t,
"belief": belief_t,
"success": success_t,
"...execution_details...": "..."}
At step , CLARO reads the latest entry action_history[-1] and serializes selected fields into a TOOL RESULT block. This block contains the tool name, the succeeded/failed outcome, and selected tool fields such as object name, push direction, and push distance. For perception tools, the prompt may also include the current scene graph or the latest locate_object result when available.
The TOOL RESULT block is inserted into user_prompt_{t+1}, and that prompt is then appended to conversation. Consequently, older tool outcomes persist in long-term memory because previous prompts containing TOOL RESULT blocks remain in conversation, together with the corresponding past images and VLM responses.

Figure 2 (PDF p. 13). Full survey_scene system prompt used by CLARO to generate a structured scene graph from the current RGB observation with the survey_scene tool.
Worked Example of Long-Term Memory-Guided Recovery
Fig. 3 shows a worked high-clutter success episode for the instruction “grasp the can of alphabet soup.” The episode illustrates how CLARO uses the accumulated interaction history to recover from a failed target grasp. After the first target grasp fails, the next turn does not repeat the same grasp. Instead, the planner uses the previous failure outcome to identify the nearby sugar box as the remaining constraint, pushes it away, re-localizes the target after the scene change, and then grasps the target successfully.
Baseline Reproduction Fidelity
To ensure a fair comparison, we integrate all baselines into a unified simulation environment with identical evaluation protocols. Table 1 summarizes the pipeline components and integration approach for each method.
| Method | VLM | Grounding | Grasping |
| CLARO (Ours) | Gemini 3.1 Pro | SAM3 | GraspGen 6-DOF |
| OWG | GPT-5.4 | SAM3 | GR-ConvNet 4-DOF |
| ThinkGrasp | GPT-5.4 | LangSAM | GraspNet 6-DOF |
| GraspMAS | GPT-5.4 | VLPart + SAM3 | RAGT 4-DOF |
| UnoGrasp | UnoGrasp (fine-tuned) | SAM3 | GraspNet 6-DOF |
| RoboDexVLM | GPT-5.4 | LangSAM | AnyGrasp 6-DOF |
| GraspGen* | None (random) | SAM3 | GraspGen 6-DOF |
Table 1 (PDF p. 14). Baseline reproduction fidelity. All methods are evaluated in the same PyBullet environment (UR5 + Robotiq 2F-140) with identical seeds, episode sequences, manipulation budgets (10 actions), and success criteria.

Figure 3 (PDF p. 14). Worked high-clutter success episode for grasping the can of alphabet soup. The dashed red region highlights the memory-guided recovery: after the target grasp fails, CLARO does not repeat the same failed grasp. Instead, it pushes the sugar box away, re-localizes and grasps the target successfully.
Integration Details
OWG. The original OWG pipeline uses an open-vocabulary detector for object grounding followed by GR-ConvNet for 4-DOF planar grasp generation. In our integration, we replace the open-vocabulary detector with SAM3 since it is a more recent and better segmentation model (box-prompted from VLM-predicted regions) while preserving the same GR-ConvNet grasp generator and VLM planning logic. The 4-DOF grasp formulation is maintained as in the original work. We use the same SAM3 model in CLARO too.
ThinkGrasp. ThinkGrasp employs VLM-based scene reasoning followed by LangSAM grounding and GraspNet 6-DOF grasp generation. Our integration preserves this full pipeline, adapting only the execution layer for the PyBullet UR5 workspace. LangSAM and GraspNet operate with their released weights.
GraspMAS. GraspMAS uses a multi-agent VLM architecture (observer, planner, coder) with VLPart for part-level detection and RAGT for 4-DOF grasp generation. We reproduce the full pipeline using the released VLPart and RAGT weights, adapting only the scene interface for our environment. For consistency, we use SAM3 to obtain segmentation masks.
UnoGrasp. UnoGrasp uses a fine-tuned Qwen2.5-VL model that predicts 2D coordinates indicating which object to grasp. The original pipeline pairs this with GroundedSAM (text-prompted using the predicted object name) and GraspNet on the full scene point cloud, filtering grasps by the segmentation mask. In our integration, we use the released checkpoint and pair the VLM output with SAM3 (point-prompted using the predicted coordinates) and GraspNet on the masked point cloud. This re-pairing is more direct: it uses the model’s coordinate output without an intermediate text-to-detection step and we verified that the checkpoint reproduces the published benchmark numbers by running the authors’ evaluation code on the Un-OBench test set separately.
RoboDexVLM. RoboDexVLM sequences atomic skills with a VLM planner over LangSAM grounding and executes them with a five-finger dexterous hand. Its grasps originate as parallel-gripper proposals from AnyGrasp, which a kinematic retargeting stage converts into hand configurations. Our adaptation removes that stage rather than replacing it: the Robotiq 2F-140 consumes the proposals in their original form. We further replace its hand-mounted RGB-D camera with the fixed overhead RGB-D camera shared by all other methods, equalizing the observation across baselines. Skill sequencing, memory retrieval, grounding, and replanning are preserved.
| GPT 5.4 | Gemini 3.1 Pro | |||
| Task SR (%) | Low | High | Low | High |
| OWG | 83.3 | 76.3 | 55.2 | 49.8 |
| ThinkGrasp | 76.2 | 75.3 | 80.8 | 78.2 |
Table 2 (PDF p. 15). Swapping the planner VLM in baselines. Replacing GPT 5.4 with Gemini 3.1 Pro breaks OWG’s set-of-mark prompting and yields only a marginal gain for ThinkGrasp; neither approaches CLARO (89.5/88.6).
Reproduction verification. Where published numbers exist on the benchmarks we use, our reproductions match them: OWG and GraspMAS fall within 1 pp of their reported OCID-VLG and GraspAnything++ scores (main paper, Tab. 3), and the UnoGrasp checkpoint reproduces its published UnoBench results under the authors’ own evaluation code. ThinkGrasp and RoboDexVLM report on neither benchmark, so their numbers here are our measurements and are not externally anchored.
Fair Evaluation Protocol
All baselines are evaluated under identical conditions:
Same PyBullet environment: UR5 robot, Robotiq 2F-140 gripper, fixed table geometry
Same object assets: unified object set with realistic meshes and physics
Same seed: deterministic episode sequences across all methods
Same budget: 10 manipulation actions per episode
Same camera: fixed overhead view
Same success criterion: target object detected inside the receptacle basket
This protocol ensures that performance differences reflect algorithmic capability rather than environmental advantages.
Planner VLM Swap in Baselines
To separate the contribution of CLARO’s planner model from that of its architecture, we replaced the GPT 5.4 planner with Gemini 3.1 Pro in the two strongest baselines and re-evaluated on the full benchmark (Table 2). The two baselines react in opposite directions. OWG’s Task SR falls from 83.3/76.3 to 55.2/49.8 (low/high clutter): OWG uses set-of-mark visual prompting, in which the planner sees the image overlaid with colored masks and must select the target by index, and Gemini 3.1 Pro associates the referred object with its overlay less reliably than GPT 5.4, so misselections propagate to grounding and grasping. ThinkGrasp, which prompts the planner with the raw RGB image (as CLARO does), improves marginally from 76.2/75.3 to 80.8/78.2. Both variants remain well below CLARO (89.5/88.6), so the choice of planner VLM alone does not explain CLARO’s advantage.
| CLARO Memory Variant | Task SR (%) Low | Task SR (%) High | Action Steps Low | Action Steps High |
| Full (default) | 89.5 | 88.6 | 3.0 | 4.9 |
| Text-Only | 86.8 | 80.4 | 3.1 | 5.2 |
| Images-Only | 76.3 | 61.7 | 2.4 | 4.0 |
| Actions-Only | 73.1 | 55.8 | 2.1 | 3.9 |
| No-Memory | 71.2 | 52.9 | 1.9 | 3.8 |
Table 3 (PDF p. 15). Memory decomposition. The textual belief trace (tool results, scene graphs) accounts for the majority of the memory benefit. Images-Only and Actions-Only perform close to No-Memory, confirming that structured outcome signals, not raw visual replay, drive multi-step planning. Low action steps in degraded variants reflect stalling (repeated failures) rather than efficiency.
Memory-Content Decomposition
A key result from our ablation study is that removing long-term memory drops high-clutter Task SR by 36 pp (88.6%→52.9%). This raises a natural follow-up: which component of the accumulated context drives this effect? We decompose the agent’s memory into three orthogonal channels and evaluate each in isolation.
Memory Channels
At each reasoning step, CLARO’s conversation history contains interleaved turns:
User turns: an RGB observation image and a structured textual block comprising tool results (scene graphs, grounding outcomes, grasp success/failure), task reminders, and budget/warning signals.
Assistant turns: the VLM’s own prior tool-call decisions (JSON-structured action selections with reasoning traces).
We isolate three subsets:
Text-Only: All textual history (tool results, scene graphs, budget state) but no prior images. The VLM reasons from its accumulated belief state without visual grounding of earlier steps.
Images-Only: All prior RGB observations but text replaced with a minimal placeholder. The VLM sees how the scene changed but receives no explicit outcomes.
Actions-Only: Only the VLM’s own prior tool-call JSON responses. It recalls what it decided but lacks both observations and outcomes.
Results
Table 3 reveals clear stratification: Text-Only retains most of the benefit. Removing images from history costs only 2.7 pp in low clutter and 8.2 pp in high clutter, retaining of the memory advantage. The structured belief trace: scene graphs, grasp outcomes, budget warnings provides sufficient context for coherent multi-step planning. The agent correctly avoids re-attempting failed grasps, tracks which occluders have been removed, and adapts strategy under budget pressure, all without reobserving prior frames.
Images-Only recovers little over No-Memory. Retaining only prior RGB frames yields marginal improvement over having no memory at all (+5.1 pp low, +8.8 pp high). Visual memory helps the VLM notice scene changes (objects disappearing, new surfaces exposed) but without explicit tool-result text, it cannot reliably distinguish successful grasps from failed attempts that left the scene unchanged. The reduced action steps (2.4/4.0 vs. 3.0/4.9 for Full) indicate that the agent stalls, cycling through perception calls without productive manipulation, rather than acting efficiently.
Actions-Only is near-equivalent to No-Memory. Knowing what was attempted without knowing outcomes or current visual state provides almost no planning benefit (+1.9 pp low, +2.9 pp high over No-Memory). The only observable effect is mild repetition-avoidance, the agent occasionally picks a different object than its last attempt.
Discussion
This decomposition reveals that CLARO’s primary memory advantage is through the structured belief trace (scene graphs, tool success/failure signals, budget state) that enables efficient multi-step planning under occlusion. Visual memory provides a complementary change-detection signal but is not the dominant driver.
This finding connects to two other results. First, it explains why the W/O CLOSED-LOOP REPLANNING ablation (main paper, Table 4) also causes a large drop: replanning depends on tool-result text to know what changed, so removing feedback is functionally similar to removing textual memory. Second, it has practical implications for deployment under context-window constraints: preserving the textual plan trace should be prioritized over retaining all prior frames, and investing in high-quality tool-result formatting yields outsized returns.
Failure-Mode Taxonomy
Real World Experiments
We analyze failed CLARO episodes by counting failure-inducing steps during the episode rather than assigning a single root cause to each episode.
| Failure source | Count | Fraction |
| Perception / SAM-mask quality | 10 | 19.61% |
| Grasp generation or execution | 22 | 43.14% |
| Push planning or execution | 19 | 37.25% |
| Total failure events | 51 | 100.00% |
Table 4 (PDF p. 16). Real-robot failures are dominated by manipulation-level budget consumption rather than a single semantic planning error. Grasp generation/execution accounts for the largest share of failure events, followed by push-related failures and SAM-mask quality issues.
Perception / SAM-mask quality: This category captures cases where the selected point was associated with an incomplete or noisy SAM mask. These mask errors can affect subsequent manipulation: for grasping, a partial mask can produce a grasp on a part of the object and lead to collision; for pushing, mask noise in the oblique-view in robot trials can shift the perspective of an object causing the push motion to start from a bad location.
Grasp generation or execution: This category covers cases where the intended object was localized, but the grasp step consumed manipulation budget without success. Examples include low-confidence or unavailable grasp generation, collision during execution, or a grasp that fails to lift the object. We did not observe inverse kinematics based infeasibility as a dominant failure in the real-robot runs, since the workspace was constrained to a reachable region.
Push planning or execution: This category covers push actions that did not de-clutter the scene or made the scene setup more complex for subsequent manipulation. Examples include pushes that increased occlusion, failed pushes during execution, or consumed action budget without creating useful clearance for the target grasp.
Overall, the largest fraction of failure-inducing events comes from grasp generation or execution, followed by push-related failures and SAM-mask quality. This suggests that the highest impacting factor to improving real-robot performance is to reduce compounding grasping failures. For videos showing failure in robot experiments, please refer to the supplementary material.
Planner-Level Failure Analysis
The taxonomy above concentrates on perception- and manipulation-level events; here we ask how often the planner’s reasoning itself causes failure. We manually reviewed every failed CLARO episode in one 100-episode pass of the benchmark (one seed, easy lighting), excluded failures attributable to grounding or grasp execution, and classified the remainder into three modes.
Semantic grounding hallucination (5 episodes): the
planner never resolves the target during the episode.
High-level reasoning error (3 episodes): the target is resolved but budget is committed to an ineffective strategy, such as pushing non-blocking distractors while a visible occluder remains.
Planning loop (2 episodes): the planner repeats non-productive actions on the correct target without changing approach.
These account for 10 episodes, distributed as 5, 3, and 2 respectively. No single reasoning pathology dominates, indicating that planner errors are diffuse rather than traceable to one systematic weakness. Isolating them at scale would require a planner-level metric and a large-scale human evaluation of planning trajectories, which we leave for future work.
Cost and Latency Breakdown
Model Calls and Cost
CLARO’s closed-loop planning uses multiple VLM calls per episode: one planner call per reasoning step (deciding which tool to invoke), plus one additional VLM call for each survey_scene invocation (the scene-graph perception prompt). Local models (SAM3, GraspGen) incur no API cost. Table 5 summarises the per-episode breakdown measured over 100 high-clutter episodes (easy lighting).
| Call Type | Mean | Median | Cost Share |
| Planner | 21.0 | 17 | 74% |
survey_scene | 13.8 | 6 | 26% |
| Total model calls | 34.8 | 23 | — |
| API cost / episode | $0.13 | $0.09 | — |
Table 5 (PDF p. 17). Per-episode model-call and cost breakdown (based on Gemini 3.1 Pro Preview, June 2026 pricing).
Baseline inference budgets. For a like-for-like comparison we measured per-episode VLM usage for the two strongest closed-loop baselines under the same Gemini 3.1 Pro backend CLARO uses (Table 6). CLARO’s retained RGB history dominates token consumption, and the TEXT-ONLY variant (Sec. 4) discards it entirely, reducing cost to per episode while attaining 86.8/80.4 low/high clutter Task SR, above both OWG (83.3/76.3) and ThinkGrasp (76.2/75.3) evaluated at their full budgets. CLARO TEXT-ONLY variant reports slightly higher VLM calls per episode than the full version, but since it uses no historical RGB frames its overall token utilization and thus cost is lower. TEXT-ONLY variant still retains a x inference budget over the baselines.
| Method | VLM calls / ep. | API cost / ep. |
| OWG | 6.2 | $0.023 |
| ThinkGrasp | 5.6 | $0.021 |
| CLARO (full) | 34.8 | $0.13 |
| CLARO (TEXT-ONLY memory) | 35.2 | $0.052 |
Table 6. Per-episode inference budget, all methods under the Gemini 3.1 Pro planner. All methods are priced at the same Gemini 3.1 Pro Preview rates (June 2026), so cost differences reflect call count and tokens per call rather than model pricing; CLARO’s retained RGB history makes its calls more image-heavy than the baselines’. Discarding those frames (TEXT-ONLY) cuts CLARO’s cost by more than half at an unchanged call count.
Latency Breakdown
VLM inference accounts for only 22.5% of total wall-clock time; the remaining 77.5% is simulation infrastructure overhead: primarily Blender’s EEVEE keyframe rendering (used solely for realistic high quality images as input to baselines) and PyBullet physics stepping with conservative settle-time checks. The average wall-clock time per simulation episode in CLARO, 142.5s, is therefore a property of the simulation harness (dominated by rendering and physics settling that every method pays for its own calls), not an execution latency of the architecture: the VLM round-trip averages 32 s per episode, and the rendering/settling component is absent in real-robot deployment.
Accuracy–latency trade-off. CLARO’s sequential, closed-loop decision-making (median 17 planner calls, mean 21) is a deliberate design choice: each step conditions on the updated scene state after the previous action, enabling reactive re-planning when a grasp fails, an object shifts unexpectedly, or a push reveals previously-occluded geometry. This stands in contrast to open-loop planners that commit to a full action sequence upfront.
The additional calls are not themselves the source of the performance advantage: despite costing less than half as much per episode, the TEXT-ONLY variant still outperforms every baseline (§ 5.1).
Reducing the per-episode call count (e.g. via batched multi-step planning, plan caching across similar scenes, or distillation of the VLM policy into a lightweight reactive controller) is a promising avenue for future work.
Benchmark and Object-Category details
Task Instruction Format
Task instructions in the benchmark follow a fixed template:
Grasp the "{object_name}" where object_name is the human-verified common name assigned to the target object (e.g., “pink computer mouse”, “canned peaches”, “small ceramic vase”). This templated format was chosen deliberately to isolate the physical challenge of cluttered grasping from the linguistic challenge of diverse instruction parsing:

Figure 4. CLARO plans focused actions to grasp the target, while the best baseline (OWG) removes the wrong objects and fails to recover. The task is to “grasp the black shampoo bottle”, which is marked by the green bounding box. Each panel shows the top-view RGB observation immediately before a manipulation, with the chosen primitive overlaid in red and the simulator’s per-step outcome below.

Figure 5. Our simulation benchmark spans diverse object categories, materials, table textures, and clutter levels, with targets ranging from prominent to heavily occluded. Each panel shows a scene with its referred target marked in a green bounding box.
Names are appearance-grounded: Each object name references visually discriminative attributes (color, size, material, or function) that distinguish it from co-present distractors. Examples: “pink dumbbell” (vs. green dumbbell), “navy boat shoe” (vs. white boat shoe), “rectangular supplement box” (vs. supplement bottle).
Prompt-template generalization study. To test sensitivity to instruction phrasing beyond the fixed template, we replaced “Grasp the {object}” with instructions sampled from a 50-prompt set: 25 direct queries that name the target with varied phrasing (“Pick up the {object}”, “I need the {object}, fetch it for me”) and 25 indirect queries that specify it only by color, shape, or spatial relation (“Retrieve the small red object on the left”, “Grab the cylindrical item next to the blue box”). Each generated instruction was manually verified to identify a unique target in its scene. CLARO along with the two strongest baselines are re-evaluated over the same 600 benchmark episodes (Table 7, averaged over both clutter regimes).
| Method | Grounding SR (%) Direct | Grounding SR (%) Indirect | Task SR (%) Direct | Task SR (%) Indirect |
| ThinkGrasp | 71.2 | 64.0 | 75.6 | 66.6 |
| OWG | 73.6 | 61.8 | 79.4 | 70.2 |
| CLARO | 87.2 | 83.2 | 89.0 | 85.4 |
Table 7. Prompt-template generalization over the 600 benchmark episodes with 25 direct and 25 indirect instruction templates, averaged over both clutter regimes. CLARO’s indirect-query Task SR exceeds both baselines’ direct-query scores.
On direct queries every method stays within half a point of its original-template score, so phrasing variation alone is not a bottleneck. Indirect queries separate the methods. Grounding SR falls by 11.8 pp for OWG, 7.2 pp for ThinkGrasp, and 4.0 pp for CLARO, and the losses propagate downstream: Task SR falls by 9.2 pp for OWG and 9.0 pp for ThinkGrasp, against 3.6 pp for CLARO. CLARO’s indirect-query Task SR (85.4) remains above both baselines’ direct-query scores.
The difference reflects how each method resolves the referring expression. OWG’s planner selects targets from an image overlaid with colored masks, which alters apparent object color and obscures the neighbouring context that referring expressions depend on. ThinkGrasp’s grounding backend (LangSAM) struggles with complex referring queries. CLARO instead uses survey_scene to recover scene-level attributes and spatial predicates, then passes the selected object reference to locate_object for precise grounding.
Object Annotation Pipeline
The benchmark object pool comprises 395 objects drawn from two mesh datasets:
BOP Objects (220 meshes): house-hold and industrial items.
DTC Objects (776 meshes from DTC scans): smaller items including toy food, figurines, fruits, dumbbells, calculators.
From the combined pool of 996 meshes, we selected 395 based on graspability (fits within the Robotiq 2F-140 gripper’s 14 cm aperture) and visual distinctiveness.
VLM-assisted labelling. Object metadata was generated using Gemini 3.1 Flash with an 8-view rendered turntable of each mesh (for objects with rendered thumbnails) or UV texture maps (for texture-only assets). The VLM was prompted to produce: (1) a concise common name, (2) dominant color(s), (3) shape descriptor, (4) graspability assessment, and (5) confidence score. Each annotation was then manually verified by two annotators who accepted, corrected, or rejected names. The final fields per object are:
{name, color, category, graspable, size_cm}
Targets were selected to span diverse sizes (3–20 cm), textures (glossy, matte, transparent), and geometries (cylindrical bottles, flat mice, irregular figurines).
Object-Category Distribution
Table 8 shows the object-category breakdown. Figure 6 presents representative rendered thumbnails from each category.
| Category | Total |
| Toys & Figurines | 86 |
| Bottles & Cans | 37 |
| Boxes & Packaging | 56 |
| Electronics | 28 |
| Footwear | 49 |
| Kitchenware | 27 |
| Home & Decor | 22 |
| Office & Tools | 16 |
| Fruits | 11 |
| Sports | 4 |
| Other | 59 |
| Total | 395 |
Table 8. Object pool by super-category.
Figure 6. Representative objects from each category. The pool spans diverse geometries, materials (glossy, matte, textured), sizes, and visual complexity.
Occlusion Threshold Justification
The benchmark reports results split by a 40% visibility threshold on low-high clutter scenes: episodes where the target’s visibility is less than 40% are categorized as high-clutter, and low-clutter if otherwise.
Empirical calibration. This threshold was set by running a naive GraspGen-only baseline (no planning, no pushing: simply attempt a 6-DOF grasp on the target’s visible mask) across 200 scenes with varying occlusion levels. At 60% (40% visibility) occlusion, the baseline’s direct-grasp success rate drops below 5%, confirming that objects above this threshold are effectively unreachable without deliberate decluttering.
Episode Generation and Seed System
Each experiment uses a single base seed that deterministically controls:
Object selection: 15 objects sampled from the 395-object pool (one designated target, 14 distractors).
Spawn positions: random poses on the table surface with physics settling.
Target assignment: selected to ensure the target is reachable (not fallen off table) after settling.
The episode seed is computed as: . All methods share identical initial conditions per episode (same objects, same poses, same target).
Lighting Modes
Easy: A fixed directional light matching PyBullet’s default configuration—consistent, shadow-free, neutral color temperature.
Hard: A randomly-sampled indoor HDRI environment map per episode, producing variable shadow directions, color casts (warm/cool), and specular reflections on glossy objects (Figure 7).
Figure 7. Same scenes rendered under easy (directional light) and hard (random HDRI environment map) lighting. Hard lighting introduces strong directional shadows, color casts, and specular highlights that challenge perception modules.
Tool-Call Flow, Push–Grasp Selection, and Human Assessment
Grasp vs. push: contextual selection
The near-balanced 49%/47% push–vs–grasp split after a successful localization is not a fixed alternation: it reflects context-dependent selection based on the state of the scene. To characterize these contexts, we tagged the planner’s free-form reasoning field in every push and grasp call issued immediately after a locate_object success, across 100 simulation episodes (one seed, easy lighting; pushes, grasps).
| Contextual cue in planner’s reasoning | push | grasp |
| Target/occluder hidden | 29.0% | 30.8% |
| Insufficient clearance around target | 30.6% | 17.4% |
| Object needs reorientation for stable grasp | 7.9% | 4.1% |
| Object obstructs the target directly | 1.2% | 5.7% |
| Target object visible and reachable | 5.5% | 60.1% |
Table 9. Fraction of push and grasp calls whose reasoning text mentions each context cue (keyword-tagged; categories are not mutually exclusive). Pushes are dominated by approach clearance and revealing hidden targets; grasps are dominated by removing a now-exposed obstructor or target.
Three patterns emerge. First, pushes are preferred when the target is not directly reachable: 30.6% of pushes cite clearing an approach path or making room for the gripper, versus 17.4% of grasps. Second, grasps dominate when an obstructor is already exposed and lightweight enough to remove: 60.1% of grasps explicitly cite completing the task or removing a known obstructor. Third, both primitives are used to reveal hidden objects at similar rates (29–31%), which is what pushes the two totals so close together in the aggregate: revealing an occluded region can be done either by shoving a large item aside (push) or by lifting a small item out (grasp), and the planner picks based on the item’s manipulability. Representative examples from the logs are given below.
Push: “pushing the black boot to the left to reveal any objects hidden underneath it.”
Push: “the towel is still partially covering the area. I will push it to the left to completely uncover any hidden objects.”
Grasp: “removing the white bunny figurine to reveal any objects hidden underneath it, as the target toy football is currently occluded.”
Grasp: “the target fresh nectarine is now fully exposed. . . grasping it will complete the task.”
When locate_object returns not_found, it retries in 87% of cases rather than abandoning the target — the object is typically occluded rather than absent, so a retry after further decluttering is the productive response. After a push, it calls locate_object in 78% of cases: the displaced object is still present and its identity unchanged, so only its position needs updating. After grasping an occluder, the pattern inverts and the planner calls survey_scene in 72% of cases, since removing an object can expose previously hidden ones that no existing scene-graph entry describes.
Human Assessment of Action Decisions
Protocol. Each participant was assigned one randomly drawn trial per episode. They were shown CLARO’s earlier actions as context and the current action it had taken together with the reasoning it recorded, and rated that action’s effect on retrieving the target as positive (helps), neutral (does not meaningfully help or hurt), or negative (makes retrieval harder). Whenever the rating was not positive, they also selected a reason: wrong object, wrong primitive, wrong direction, repeated failed action, or unnecessary action because the target was already reachable. Because annotators see the action and its recorded reasoning but not its
outcome, the rating evaluates the decision rather than its execution. We collected 300 completed judgments from eight annotators, none of whom judged consecutive steps of the same rollout. These judgments cover every executed action in the real-robot episodes, so the grasp share (68.7%) is higher than the post-localization split reported in § 6.1.1: a failed grasp is retried and counted again, and real scenes are sparser than the simulated ones, leaving the target directly reachable more often.
Ratings. Table 10 summarizes the outcome. Across all judgments 85.3% of CLARO’s decisions were rated positive, 10.0% neutral and 4.7% negative. Grasps were rated positive in 88.8% of judgments and pushes in 77.7%, but pushes drew the lower negative rate, 3.2% against 5.3%: what separates the two primitives is the neutral share, not the negative one. Alongside the ambiguity over which direction to push, a second case falls into the neutral category. CLARO often clears an object adjacent to the target whose removal eases access without being strictly required: such an action neither fails to help nor makes retrieval harder, and with only three options available an annotator records it as neutral. The reason counts support this reading over the alternative that neutral marks a wasted action: while 19.1% of pushes were rated neutral, only 8.5% were flagged as unnecessary because the target was already reachable, so fewer than half of the neutral pushes were judged pointless.
| action | % () | pos | neu | neg |
| grasp | 68.7 (206) | 88.8 | 5.9 | 5.3 |
| push | 31.3 (94) | 77.7 | 19.1 | 3.2 |
| overall | 100.0 (300) | 85.3 | 10.0 | 4.7 |
Table 10. Human assessment of CLARO’s action decisions. 300 judgments by eight independent annotators, each rating the effect of CLARO’s action on retrieving the target as positive, neutral, or negative.
Supplementary Video
The supplementary video shows some success and failure cases of the robot experiments with CLARO, compared to the baselines. In our robot setup, the robot arm might collide with an object which triggers a safety-terminate step on the robot controller. When this happens during an experiment, we manually reset the arm to a free space and resume the experiment from that step for all the baselines. We never manipulate the intermediate scene clutter.
References
- [1]Vineet Bhat, Yu-Hsiang Lan, Prashanth Krishnamurthy, Ramesh Karri, and Farshad Khorrami. 3D CAVLA: Leveraging depth and 3d context to generalize vision language action models for unseen tasks. arXiv preprint arXiv:2505.05800, 2025. 2, 3arxiv.org/abs/2505.05800
- [2]Vineet Bhat, Naman Patel, Prashanth Krishnamurthy, Ramesh Karri, and Farshad Khorrami. MapleGrasp: Mask-guided feature pooling for language-driven efficient robotic grasping. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7554–7564, 2026. 1, 2, 3arxiv.org/abs/2506.06535
- [3]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. 4, 6
- [4]Siyi Chen, Hugo Hadfield, Alex Zook, Mikaela Angelina Uy, Chan Hee Song, Erwin Coumans, Xuning Yang, Faisal Ladhak, Qing Qu, Stan Birchfield, Jonathan Tremblay, and Valts Blukis. VoLo: A physical orchestrator for open-vocabulary long-horizon manipulation. arXiv preprint arXiv:2606.07723, 2026. 3arxiv.org/abs/2606.07723
- [5]Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birchfield, Valts Blukis, and Jonathan Tremblay. SpaceTools: Tool-augmented spatial reasoning via double interactive RL. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. 2arxiv.org/abs/2512.04069
- [6]Xiaomeng Chu, Jiajun Deng, Guoliang You, Wei Liu, Xingchen Li, Jianmin Ji, and Yanyong Zhang. GraspCoT: Integrating physical property reasoning for 6-dof grasping under flexible language instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10130–10140, 2025. 3arxiv.org/abs/2503.16013
- [7]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Rohan Tripathi, Sangho Lee, Mohammedraza Salehi, Jason Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Ali Farhadi, and Ranjay Krishna. Molmo2: Open weights and data for vision-language models with video understanding and grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28652–28668, 2026. 1, 6arxiv.org/abs/2601.10611
- [8]Michael Danielczuk, Andrey Kurenkov, Ashwin Balakrishna, Matthew Matl, David Wang, Roberto Martín-Martín, Animesh Garg, Silvio Savarese, and Ken Goldberg. Mechanical Search: Multi-step retrieval of a target object occluded by clutter. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019. 1DOI
- [9]Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhri, Ainaz Eftehkar, Rohan Tripathi, Max Argus, Jordi Salvador, Haoquan Fang, Matthew Wallingford, et al. MolmoBOT: Large-scale simulation enables short-horizon manipulation. arXiv preprint arXiv:2603.16861, 2026. 2, 3, 5
- [10]Zhao Dong, Ka Chen, Zhaoyang Lv, Hong-Xing Yu, Yunzhi Zhang, Cheng Zhang, Yufeng Zhu, Stephen Tian, Zhengqin Li, Geordie Moffatt, Sean Christofferson, James Fort, Xiqing Pan, Mingfei Yan, Jiajun Wu, Carl Yuheng Ren, and Richard Newcombe. Digital Twin Catalog: A large-scale photorealistic 3d object digital twin dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 5arxiv.org/abs/2504.08541
- [11]Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Li Fei-Fei, Guanya Shi, Jiajun Wu, S. Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi Fan. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation. In Forty-third International Conference on Machine Learning, 2026. 2arxiv.org/abs/2603.22435
- [12]George Gao, Tianyu Li, Junyao Shi, Yihan Li, Zizhe Zhang, Nadia Figueroa, and Dinesh Jayaraman. VLMgineer: Vision-language models as robotic toolsmiths. In International Conference on Learning Representations, pages 91732–91763, 2026. 3arxiv.org/abs/2507.12644
- [13]Runyu Jiao, Matteo Bortolon, Francesco Giuliari, Alice Fasoli, Sergio Povoli, Guofeng Mei, Yiming Wang, and Fabio Poiesi. Obstruction reasoning for robotic grasping. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2026. 2, 3, 5, 7arxiv.org/abs/2511.23186
- [14]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. In Proceedings of Robotics: Science and Systems, 2025. 2
- [15]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, pages 2679–2713. PMLR, 2025. 3arxiv.org/abs/2406.09246
- [16]Guowei Lan, Kaixian Qu, René Zurbrügg, Changan Chen, Christopher E Mower, Haitham Boul-Ammar, and Marco Hutter. Experience is the best teacher: Grounding vlms for robotics through self-generated memory. arXiv preprint arXiv:2507.16713, 2025. 3
- [17]Haichao Liu, Sikai Guo, Pengfei Mai, Jiahang Cao, Haoang Li, and Jun Ma. RoboDexVLM: Visual language model-enabled task planning and motion control for dexterous robot manipulation. In Proceedings of the IEEE/RSJ 2025 International Conference on Intelligent Robots and Systems (IROS), pages 1381–1388. IEEE, 2025. 2, 3, 5, 7arxiv.org/abs/2503.01616
- [18]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qiang Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marry- Adithyavaravan Murali, Balakumar Sundaralingam, Yu-Wei Chao, Jun Yamada, Wentao Yuan, Mark Carlson, Fabio Ramos, Stan Birchfield, Dieter Fox, and Clemens Eppner. GraspGen: A diffusion-based framework for 6-dof grasping with on-generator training. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026. 2, 4, 5, 7
- [20]Quang Nguyen, Tri Le, Huy Nguyen, Thieu Vo, Tung D Ta, Baoru Huang, Minh N Vu, and Anh Nguyen. GraspMAS: Zero-shot language-driven grasp detection with multi-agent system. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. 2, 3, 5, 6, 7arxiv.org/abs/2506.18448
- [21]Van Nguyen Nguyen, Stephen Tyree, Andrew Guo, Mederic Fourmy, Anas Gouda, Taeyeop Lee, Sunghpill Moon, Hyeontae Son, Lukas Ranftl, Jonathan Tremblay, Eric Braeckmann, Bertram Drost, Vincent Lepetit, Carsten Rother, Stan Birchfield, Jiri Matas, Yann Labbe, Martin Sunder-meyer, and Tomas Hodan. BOP challenge 2024 on model-based and model-free 6d object pose estimation. arXiv preprint arXiv:2504.02812, 2025. 5arxiv.org/abs/2504.02812
- [22]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. 1, 2, 3
- [23]Yaoyao Qian, Xupeng Zhu, Ondrej Biza, Shuo Jiang, Linfeng Zhao, Haojie Huang, Yu Qi, and Robert Platt. ThinkGrasp: A vision-language system for strategic part grasping in clutter. In Conference on Robot Learning (CoRL), 2024. 1, 2, 3, 5, 7, 8arxiv.org/abs/2407.11298
- [24]Junyao Shi, Ruijia Yang, Kaitian Chao, Selina Bingqing Wan, Yifei Shao, Jiahui Lei, Jianing Qian, Long Le, Pratik Chaudhari, Kostas Daniilidis, Chuan Wen, and Dinesh Jayaraman. Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots. In NeurIPS 2025 Workshop on Space in Vision, Language, and Embodied AI, 2025. 3
- [25]Yitian Shi, Di Wen, Guangqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, and Rania Rayyes. VISO-Grasp: Vision-language informed spatial object-centric 6-dof active view planning and grasping in clutter and invisibility. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. 3arxiv.org/abs/2503.12609
- [26]Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, and Tomas Pfister. Watch and Learn: Learning to use computers from online videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5421–5431, 2026. 2arxiv.org/abs/2510.04673
- [27]Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via Python execution for reasoning. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023. 7arxiv.org/abs/2303.08128
- [28]Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Ashwin Balakrishna, Nathan Batchelor, Alex Bewley, Jeff Bingham, et al. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer. arXiv preprint arXiv:2510.03342, 2025. 1arxiv.org/abs/2510.03342
- [29]Georgios Tziafas and Hamidreza Kasaei. Towards open-world grasping with large vision-language models. In 8th Annual Conference on Robot Learning, 2024. 1, 2, 3, 5, 7, 8arxiv.org/abs/2406.18722
- [30]Georgios Tziafas, Yucheng XU, Arushi Goel, Mohammadreza Kasaei, Zhibin Li, and Hamidreza Kasaei. Language-guided robot grasping: CLIP-based referring grasp synthesis in clutter. In 7th Annual Conference on Robot Learning, 2023. 2, 3, 7arxiv.org/abs/2311.05779
- [31]An Dinh Vuong, Minh Nhat Vu, Baoru Huang, Nghia Nguyen, Hieu Le, Thieu Vo, and Anh Nguyen. Language-driven grasp detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17902–17912, 2024. 7arxiv.org/abs/2406.09489
- [32]Xuning Yang, Rishit Dagar, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A high-fidelity simulation benchmark for analysis of task generalist policies. In Proceedings of Robotics: Science and Systems, Sydney, Australia, 2026. 2, 3arxiv.org/abs/2604.09860
- [33]Houjian Yu, Zheming Zhou, Min Sun, Omid Ghasemalizadeh, Yuyin Sun, Cheng-Hao Kuo, Arnie Sen, and Changhyun Choi. Attribute-based object grounding and robot grasp detection with spatial reasoning. In 2025 IEEE-RAS 24th International Conference on Humanoid Robots (Humanoids), pages 1123–1130. IEEE, 2025. 3arxiv.org/abs/2509.08126
- [34]Haoran Zhang, Shuanghao Bai, Wanqi Zhou, Yuedi Zhang, Qi Zhang, Pengxiang Ding, Cheng Chi, Donglin Wang, and Badong Chen. VCoT-Grasp: Grasp foundation models with visual chain-of-thought reasoning for language-driven grasp generation. arXiv preprint arXiv:2510.05827, 2025. 3arxiv.org/abs/2510.05827

