betaFeedback

WorksComputer Science

A2^2-VLA: Adapting Vision-Language-Action Policies to New Tools through Agentic Affordance Discovery

Abstract

Tools with different shapes can have similar functions, but transferring manipulation skills across tools requires understanding how tool geometry relates to the specific task and adjusting the manipulation strategy accordingly. The key challenge is to discover the common affordances that contribute to task completion across different tools and verify whether they support new tools in performing similar work. We present A2^2-VLA, a tool-adaptation framework that combines an embodied agent with a VLA policy. It treats tool affordances as hypotheses the agent can construct, test, and revise. Given teleoperated demonstrations with a few source tools, the agent reconstructs the scenes in simulation and autonomously determines task-relevant affordance features: functional regions and their key geometric properties. It develops affordance-extraction skills and refines both the features and skills through autonomous experiments in the reconstructed scenes. Validated extraction skills accumulate in a reusable skill library for unseen tools. Finally, A2^2-VLA uses encoded functional-region features as conditioning inputs to a π0.5π_{0.5} backbone during training and inference. Evaluation examines affordance reliability in reconstructed scenes and real-robot task success on unseen tools. Our method achieves 86.1% success in simulated cross-tool trajectory retargeting based on affordances. On three real-robot tool-use tasks, A2^2-VLA improves task success by 30.7 percentage points over π0.5π_{0.5}.

Overview of A²-VLA for agentic affordance discovery and affordance-guided VLA

Figure 1. Given demonstrations of a task performed with a few source tools, A2A^2-VLA analyzes functional-region features, reconstructs scenes in simulation, and validates the inferred affordances. Refined extraction skills annotate the demonstrations with affordance features for VLA training. For an unseen tool, A2A^2-VLA uses validated skills to extract its affordance, verifies it in simulation, and supplies the encoded functional-region features to guide VLA execution.

Introduction

People can often reuse a familiar manipulation skill with an unfamiliar tool after inspecting it and attempting to interact with the environment using it. A differently shaped implement may still provide functional regions similar to those of a familiar tool, such as an edge for cutting or a tip for hooking, even though its appearance and required wrist motion differ substantially from those of the original tool. What transfers is therefore neither the tool’s identity nor a fixed end-effector trajectory, but an understanding of the properties that make the intended interaction possible. Enabling robots to acquire and reuse this understanding from limited tool experience would reduce the need to collect demonstrations for every new tool–grasp combination.

Toward this goal, we study how to enable a robot system to adapt to new tools that can perform the same task, starting from teleoperated demonstrations using a small set of source tools. Vision-language-action (VLA) policies already possess some tool-manipulation capabilities [3, 6, 14], but adapting them to entirely new tools still requires repeated rollout collection and fine-tuning to maintain performance. Predefined functional representations of tools have shown useful structure for transfer: MimicFunc establishes functional correspondences through keypoints and function frames [18], and FuncBridge learns functional trajectories separately from their grounding into actions [23]. Meanwhile, improvements in the embodied manipulation capabilities of general-purpose large vision-language models (VLMs) make autonomous analysis of tool function possible. Our focus is how an agent system can autonomously identify task-relevant functional regions and determine their key geometric features, capturing functional commonality across tools to guide a VLA policy in adapting to new tools.

We treat task-relevant affordance features as experimentally testable hypotheses. Although hierarchical robot systems already connect reasoning by general-purpose VLMs to low-level policy execution through language or spatial guidance [4, 7, 16], language or keypoints alone cannot fully represent the geometric details of tools. Cross-tool adaptation additionally requires deciding which tool properties should be represented for a specific task and testing whether these properties can guide the policy to complete it correctly. We introduce A2^{2}-VLA, in which the agent starts from a few real-robot teleoperated demonstrations, reconstructs the demonstrated scenes and tools in simulation, and conducts experiments within them. The agent analyzes how different source tools accomplish the task, determines task-relevant affordance features, develops affordance-extraction skills, and applies them to other tools to attempt task execution. Based on experimental feedback, it analyzes unsuccessful interactions and refines the selected functional regions, geometric features, or extraction skills. Successful skills and their applicability conditions accumulate during this process in a reusable skill library. When faced with an unseen tool, the agent can therefore reuse previously validated skills to identify corresponding functional regions and extract the same features.

We condition the low-level VLA execution policy on agent-encoded functional-region features through a low-dimensional affordance interface. While analyzing the source tools, the agent determines task-relevant affordance features, which are encoded under a unified protocol as conditioning inputs for VLA training. Unseen tools’ affordances are described through the same protocol: their specific affordance parameters are extracted while the coordinate representation and feature semantics remain unchanged. This gives the agent freedom to select functional regions and geometric features while maintaining a consistent policy input format. Encoded features serve as conditioning inputs when training a π0.5\pi_{0.5} policy with an auxiliary affordance prediction head. During deployment, the agent reconstructs the unseen tool in simulation, extracts and tests candidate affordances, and refines their parameters from simulation feedback before encoding them under the same protocol for the VLA policy. At execution time, the VLA conditions each action chunk on the updated affordance features and current observations.

Our evaluation examines the reliability and reusability of autonomously discovered affordances and their effectiveness in cross-tool transfer. Simulation tests compare our affordance discovery and validation process with other tool-adaptation methods, including task success when using affordances to retarget trajectories in reconstructed scenes. Three real-robot tool-use tasks test the robustness of cross-tool transfer. Human Oracle uses human-provided affordances for both VLA policy training and inference to distinguish failures caused by poor affordance features from those caused by the VLA’s execution capability.

Our main contributions are:

  • An agentic tool-adaptation framework that starts from teleoperated demonstrations using a small set of source tools and autonomously determines task-relevant affordance features. The agent reconstructs the demonstrated scenes in simulation, tests and refines these features through autonomous experiments, and retains refined affordance-extraction procedures as reusable skills across tools.

  • An affordance-guided VLA model that integrates the resulting low-dimensional feature vectors into action generation. Affordance conditioning and auxiliary affordance-trajectory supervision train the policy to faithfully follow the supplied affordance information while using current observations and robot state to complete tool-use tasks.

Related Work

Tool adaptation. Tool adaptation requires understanding how a tool itself functions in a specific task. KETO [15] and GIFT [19] learn sparse grasp and functional keypoints, while FuncBridge [23] predicts 2D functional keypoint trajectories to guide action generation. However, functional points alone may not capture more complex manipulation characteristics. Other work, such as MimicFunc [18], automatically extracts keypoints and constructs function frames for cross-tool transfer. Its affordance schema and frame-construction rules are predefined, however, so adaptation remains organized around interactions expressible within that abstraction rather than autonomously determining which functional features a task requires. MAGIC [9] matches DINOv2 visual features and local curvature to transfer a demonstrated contact strategy, with physical simulation checking candidate motions. Its candidate generation relies on visual–geometric analogies and struggles with substantially different tools. Other work uses simulation environments to discover and validate reliable tool geometries, including GeT-USE [20] and ToolAnalogy [1], thereby improving the ability to select suitable tools.

Vision-language-action models. VLAs learn observation-conditioned manipulation from multimodal pretraining and robot demonstrations. Representative models include OpenVLA [6], the flow-matching policy π0\pi_0 [3], and π0.5\pi_{0.5} [14]. However, their manipulation knowledge remains dependent on the behaviors and interactions covered by training data. Unfamiliar tool geometries and contact patterns can expose gaps in this coverage, limiting cross-tool generalization. Hierarchical architectures decouple high-level task-related information from low-level policies, allowing task reasoning and action control to be modeled separately. Hi Robot uses language subgoals and HAMSTER uses 2D paths; these intermediate interfaces guide low-level execution with task goals or spatial plans [4, 7, 16]. Existing work also adds functional guidance to VLAs and related visuomotor policies, grounding task intent in spatial interaction structure. RT-Affordance [12] uses key-stage end-effector poses, FuncBridge [23] predicts functional keypoint trajectories, and AFI [21] uses 3D affordance fields to guide waypoints and select trajectories. These representations provide explicit cues about where and how to interact, connecting semantic goals to geometrically grounded action generation.

Embodied agents. Embodied agents use general knowledge to analyze tool functions and formulate execution strategies. ReKep [5] generates relational keypoint constraints, and ToolAnalogy [1] proposes functional features. Meanwhile, agentic real-to-sim systems, such as WorldMirror and SceneRig, reconstruct interactive environments for trajectory replay and policy testing [8, 11, 22], allowing agents to test their understanding of manipulation through interaction. For reusable skill development, ASPIRE [10] uses execution feedback to repair control programs and distill reusable skills. A2^2-VLA applies real-to-sim interaction testing and reusable skill development to tool adaptation. The agent autonomously identifies task-relevant functional regions and determines their key geometric features, then refines these features and extraction skills through simulation feedback to guide VLA execution with unseen tools.

Problem Formulation

Given a task instruction ℓ\ell, we are provided a small set of source tools, each with real-robot teleoperated demonstrations. At time tt, the observation ot=(It,qt)o_t = (I_t, q_t) contains camera images ItI_t and the robot configuration qtq_t. The first command predicted from oto_t is ata_t. An action chunk At=(at,…,at+H−1)A_t = (a_t, \ldots, a_{t+H-1}) contains HH consecutive robot commands. The demonstrations provide the observations and trajectories needed to reconstruct tool interactions in simulation.

We seek transfer to unseen tools, disjoint from source tools, without additional task demonstrations. The agent autonomously determines task-relevant affordance features by analyzing source-tool demonstrations, identifying functional regions and key geometric properties shared across tools. It develops skills to extract corresponding features from unseen tools. We denote the encoded functional-region feature vector at time tt by ftf_t. Feature values follow a unified task-level encoding protocol. For an unseen tool, the skills use inspection observations and its attachment to the robot. Different tools supply different values under the same feature definitions and encoding. The policy πθ\pi_\theta uses ftf_t together with oto_t and ℓ\ell to complete the task. We evaluate affordance reliability through simulated cross-tool retargeting and transfer performance through real-robot task success.

A2-VLA

A2-VLA connects an agent that analyzes tool function with a VLA that executes the task (Figure 2). The agent reconstructs demonstrated interactions, develops affordance-extraction skills, and tests them through simulation. Validated affordances condition VLA training and deployment.

Figure 2. A2^{2}-VLA overview. (a) The agent reconstructs demonstrated scenes, autonomously determines task-relevant affordance features, develops affordance-extraction skills, and tests cross-tool retargeting in simulation. Diagnostic feedback guides revisions; refined extraction skills enter the reusable library. (b) Encoded functional-region features join image, language, and robot-state tokens to condition the π0.5\pi_{0.5} action expert. It iteratively denoises action tokens, decoded into robot actions and an auxiliary affordance trajectory.

Reconstructing Demonstrated Interactions

The agent performs real-to-sim reconstruction using observations and recorded robot trajectories from source-tool demonstrations, together with calibrated camera parameters and a robot model, to reconstruct the tools, target objects, and their spatial arrangement in simulation. Recorded motions anchor the reconstruction to the demonstrated interaction. The resulting environment enables the agent to replace a tool, change its functional parameters, adjust object positions, and execute candidate action sequences while observing contact and task outcomes. The reconstructed environment enables the agent to rapidly test candidate functional regions through simulated tool interactions.

Autonomous Affordance Discovery

From examples to extraction skills. As shown in Figure 2(a), the agent compares source-tool demonstrations to identify the functional region and properties that enable the task, such as position, local orientation, and task-specific direction. It develops executable affordance-extraction skills for these task-relevant features. Each skill extracts and encodes features as ftf_t at each demonstrated time step, with spatial features expressed in the robot-base frame through tool attachment and kinematics. Ordering these features in time yields the demonstrated affordance trajectory.

Affordance validation in simulation. The agent validates each tool’s extracted affordance features in simulation. Specifically, it tests whether feature-based trajectory retargeting enables different tools to complete the task. Let Q=(q0,…,qT−1)\mathbf{Q} = (q_0, \ldots, q_{T-1}) be a feasible TT-step robot trajectory that completes the task with the source tool in simulation, yielding reference features ftsrc\mathbf{f}^{\mathrm{src}}_t. For the target-tool trajectory Q′=(q0′,…,qT−1′)\mathbf{Q}' = (q'_0, \ldots, q'_{T-1}), features ftgt(qt′)\mathbf{f}^{\mathrm{tgt}}(q'_t) express local functional-region geometry in the robot-base frame, combining robot forward kinematics at qt′q'_t with the tool’s pose relative to the wrist. Subject to reachability and collision constraints, it minimizes

Q∗=arg⁡min⁡Q′∑t=0T−1Lalign(ftgt(qt′),ftsrc)+Lsmooth(Q′).(1)\mathbf{Q}^{*} = \underset{\mathbf{Q}'}{\arg\min}\sum_{t=0}^{T-1}\mathcal{L}_{\mathrm{align}}\left(\mathbf{f}^{\mathrm{tgt}}(q'_t), \mathbf{f}^{\mathrm{src}}_t\right) + \mathcal{L}_{\mathrm{smooth}}(\mathbf{Q}'). \tag*{(1)}

Here Lalign\mathcal{L}_{\mathrm{align}} aligns functional-region poses across source and target tools, matching position and task-relevant orientation. Lsmooth(Q′)\mathcal{L}_{\mathrm{smooth}}(\mathbf{Q}') is a trajectory-smoothing function.

Refining and retaining skills. Simulation returns task outcomes, contact information, and diagnostic views. The agent uses this feedback to refine functional regions, geometric features, or affordance-extraction skills, then repeats the test. Source-tool analysis determines the affordance features before VLA training; their values follow a unified task-level encoding protocol. Successful extraction skills and their applicability conditions enter a skill library. The retained skills annotate all source-tool teleoperated demonstrations with tool affordance features for VLA training. For unseen tools, the agent retrieves and applies these skills to identify corresponding functional regions and extract the same features. Figure 3 illustrates how simulation tests guide revisions to candidate functional regions and orientations.

Figure 3. Simulation-guided affordance refinement. In the source-tool demonstration, the nearly horizontal hook region at the base of the hook supports the ring (orange). (a) The initial region on the target tool misses the ring. (b) The agent tests alternative contact locations and orientations. In Test 1, the selected region fails to enter the ring; in Test 2, the forward-tilted hook region allows the ring to slide off during lifting; Test 3 retains the ring. (c) The refined skill selects a flat or gently concave hook region near the tool tip and keeps it nearly horizontal throughout the lift.

Affordance-Conditioned VLA

We condition a π0.5\pi_{0.5} backbone [14] on the encoded affordance features ft\mathbf{f}_t, current observation ot=(It,qt)\mathbf{o}_t = (I_t, \mathbf{q}_t), and task instruction ℓ\ell. An action prediction head and an auxiliary affordance prediction head output the action chunk and future affordance trajectory:

(A^t,F^t)=πθ(ot,ℓ,ft).(2)(\widehat{\mathbf{A}}_t, \widehat{\mathbf{F}}_t) = \pi_\theta(\mathbf{o}_t, \ell, \mathbf{f}_t). \tag*{(2)}

The training target is Ft=(ft+1,…,ft+H)\mathbf{F}_t = (\mathbf{f}_{t+1}, \ldots, \mathbf{f}_{t+H}), extracted from demonstrated action-target configurations by the agent’s validated affordance-extraction skill. Each ft+h+1\mathbf{f}_{t+h+1} describes the functional region at the target of at+h\mathbf{a}_{t+h}, for h=0,…,H−1h = 0, \ldots, H - 1. Thus ft\mathbf{f}_t is the current affordance feature vector, while Ft\mathbf{F}_t contains the HH subsequent target-step features, paired one-to-one with the HH commands in At\mathbf{A}_t. Hats denote predictions. The two heads are trained jointly with L=Lact+Laff\mathcal{L} = \mathcal{L}_{\mathrm{act}} + \mathcal{L}_{\mathrm{aff}}. The action term is the backbone’s flow-matching objective. The auxiliary term supervises the predicted affordance trajectory, encouraging similar affordance trajectories across tools performing the same task. See Figure 2(b) and Appendix § 10 for details.

To strengthen the VLA’s ability to follow affordance guidance, we additionally include 10% counterfactual trajectory-retargeting data. We randomly select chunks from source-tool demonstrations, modify the pose of the functional-region affordance, and replan the subsequent action sequence so that the modified affordance pose smoothly returns to the affordance-pose trajectory of the original task demonstration. This process aims to train the VLA to execute the task by strictly following the current affordance pose and observations of the target object. More formally, for a sampled rigid pose perturbation ΔT∈SE(3)\Delta T \in\mathrm{SE}(3), we construct

f~t=ΔT⋅ft,A~t=Retarget⁡(f~t,Ft).(3)\begin{aligned} \widetilde{\mathbf{f}}_t &= \Delta T \cdot\mathbf{f}_t,\\ \widetilde{\mathbf{A}}_t &= \operatorname{Retarget}(\widetilde{\mathbf{f}}_t,\mathbf{F}_t). \tag*{(3)} \end{aligned}

Here ΔT⋅ft\Delta T \cdot\mathbf{f}_t rigidly transforms the feature’s spatial components, and Retarget replans from the perturbed affordance using the original Ft\mathbf{F}_t as the reference, with the alignment and smoothness objective in Eq. (1).

When using an unseen tool, the policy can thus use the tool affordances analyzed by the agent to execute the task with logic similar to the source-tool demonstrations.

Unseen-Tool Adaptation

At deployment, the agent first reconstructs the unseen tool in simulation, then extracts candidate affordance parameters with its affordance-extraction skills. Using these parameters, it retargets demonstrations for simulation tests and iteratively refines the parameters from the feedback. The refined parameters are then encoded as ftf_t using the unified task-level encoding protocol and supplied to the VLA. Following the backbone’s inference procedure, the VLA conditions on ftf_t and current observation oto_t, predicts an action chunk, executes part of it, and receives a new observation before predicting the next chunk. Spatial components of ftf_t track the robot configuration and tool attachment pose. The agent handles tool analysis and simulation tests before real-robot execution; the VLA controls the subsequent real-robot actions.

Experiments

We assess the reliability and reusability of affordances discovered through autonomous simulation experiments, as well as their effectiveness in supporting real-robot task transfer to unseen tools. We evaluate our method on three common tool-use tasks. In Pnp Ring, a tool moves a keyring from the left peg to the rightmost peg; Cut Sand requires slicing through the sand with the tool’s functional edge and placing the cut portion in the adjacent bowl; and Reset Plate requires retrieving a plate and placing it on the top level.

Evaluation Setup

For each task, we provide real-robot teleoperated demonstrations using a set of source tools. A disjoint set of unseen tools is supplied through RGB-D inspection observations, with no manipulation demonstrations. CAD-assisted simulation baselines additionally receive the geometric inputs specified below. Our agent starts from the skill library built using source tools. Real-robot evaluation tests both source tools used for demonstrations (Table 2) and unseen tools (Table 3). For each task and tool set, VLA-based methods use 50 evaluation episodes and Agent-Only direct control uses 20. Each episode samples a tool uniformly from the corresponding set. Task-specific counts and success criteria are listed in Appendix § 8 (Table A1 and Table A2).

MethodPnp RingCut SandReset PlateMean
π0.5\pi_{0.5}34122022.0
Agent-Only20352526.7
w/o Sim. Iteration40343235.3
Human Oracle64485054.0
Full A2^{2}-VLA52424446.0

Table 2. Real-robot task success (%) on source tools used for demonstrations. VLA-based variants share the pretrained π0.5\pi_{0.5} backbone and training budget. Each task uses 50 episodes for VLA methods and 20 for GPT-6 Astra direct control (Agent-Only), sampling tools uniformly. Human Oracle uses human-provided affordances in training and inference. Mean averages the three task rates.

MethodPnp RingCut SandReset PlateMean
π0.5\pi_{0.5}10044.7
Agent-Only25302025.0
w/o Sim. Iteration28201822.0
Human Oracle54404847.3
Full A2^{2}-VLA46263435.3

Table 3. Real-robot task success (%) on unseen tools, using the evaluation protocol of Table 2. Mean averages the three task rates.

TaskSource toolsUnseen toolsDemonstrations per source toolEvaluation episodes per task VLA-basedEvaluation episodes per task Agent-Only
Pnp Ring1010105020
Cut Sand1210105020
Reset Plate55105020

Table A1. Tool and demonstration counts and real-robot evaluation protocol. Source-tool and unseen-tool sets are evaluated separately. For each task and tool set, VLA-based methods use 50 episodes and Agent-Only (GPT-6 Astra direct control) uses 20. Each episode samples a tool uniformly from the corresponding set.

TaskSuccess criterion
Pnp RingMove the keyring to the rightmost target position.
Cut SandSlice through the sand with the tool’s functional edge and place the cut portion in the adjacent bowl.
Reset PlateRetrieve the plate from the rack and place it on the top-layer target.

Table A2. Task-specific success criteria used in evaluation.

Simulation comparisons. All simulation scenes and tools are reconstructed by GPT-6 Astra from real demonstrations and checked using trajectory replay. Based on the affordances inferred by each method, we retarget the source-tool demonstration trajectory to unseen tools and execute the resulting trajectory in the reconstructed simulation scene. Task success measures the reliability of the inferred affordances. Comparisons include direct GPT-6 Astra affordance prediction without an agentic harness (Raw VLM), MAGIC [9], Interaction Warping (IW) [2], MimicFunc [18], A2^{2}-VLA, R-NDF [17], and TAX-Pose [13]. Where the task and output interfaces permit, methods retain their native affordance representations and corresponding transfer procedures. Evaluation episodes use the same simulation initial conditions and trajectory-planning program for all methods. The IW and R-NDF rows additionally use original tool CAD, including mounting geometry, with model matching and rigid registration assisted by the provided reconstructed proxies. Table 1 summarizes this comparison.

MethodPnp RingCut SandReset PlateMean
MAGIC8.842.50.017.1
IW w/ CAD3.821.30.08.3
MimicFunc25.015.00.013.3
R-NDF w/ CAD13.80.030.014.6
TAX-Pose22.525.033.326.9
Raw VLM31.343.838.337.8
Full A2^{2}-VLA81.393.883.386.1

Table 1. Table 1. Simulation retargeting success (%) with shared retargeting and robot control. We evaluate 80, 60, and 80 episodes for Cut Sand, Reset Plate, and Pnp Ring, respectively, sampling unseen tools uniformly in each episode. IW and R-NDF additionally receive original CAD with proxy-assisted registration (w/ CAD). Missing mappings and unreachable plans count as failures. Mean averages the three task rates. Raw VLM directly predicts affordances with GPT-6 Astra.

Real-robot comparisons. We construct the task environments using a Dobot Nova 2 bimanual manipulation platform and three RealSense D405 cameras (Figure 4). For each unseen tool, our agent is allowed to build a simulation model and run simulation tests to determine and refine its affordance before real-robot execution.

Tool-use tasks and real-robot setup

Figure 4. Tool-use tasks and real-robot setup. The three tasks evaluate cross-tool transfer in hooking, cutting, and object lifting. The leftmost column shows their everyday inspirations, and the corresponding functional regions are highlighted in orange. Real-robot experiments use a Dobot Nova 2 bimanual platform with RGB-D observations.

Table 2 and Table 3 compare π0.5\pi_{0.5} without affordance guidance, Agent-Only control, w/o Sim. Iteration (our method without simulation-guided refinement), Human Oracle, and Full A2^{2}-VLA. VLA-based methods share the pretrained π0.5\pi_{0.5} checkpoint and training budget. Affordance-guided variants randomly initialize the affordance encoder and auxiliary affordance prediction head. Human Oracle uses human-provided affordances during both training and inference to distinguish poor affordance representations from limitations of VLA execution. Agent-Only does not use the VLA; the agent directly controls the robot’s wrist end-effector from observations. All agents use GPT-6 Astra.

Results and Analysis

Affordance reliability. Table 1 evaluates affordance reliability through simulated cross-tool retargeting. Full A2^{2}-VLA achieves 86.1% mean success, exceeding 37.8% for Raw VLM, which uses GPT-6 Astra for direct affordance prediction without an agentic harness. This improvement demonstrates the value of simulation-guided testing and refinement of affordance-extraction skills. Compared with other tool-adaptation methods, our agent more effectively identifies functionally similar regions for precise trajectory retargeting when switching tools.

Real-robot transfer experiments. Table 2 and Table 3 report real-robot success on source and unseen tools, respectively. Full A2^{2}-VLA achieves 46.0% mean success on source tools, compared with 22.0% for π0.5\pi_{0.5}.

When transferred from source tools to unseen tools, Full A2^{2}-VLA retains 35.3% mean success, outperforming both π0.5\pi_{0.5} and direct Agent-Only control. Comparing w/o Sim. Iteration with Full A2^{2}-VLA further shows that simulation-guided refinement improves affordance-guided VLA execution on the real robot.

ABLATIVE & QUALITATIVE STUDIES To assess the effects of simulation-guided refinement and source-tool diversity on our method’s performance, we vary the refinement iterations and the number of source tools used for affordance discovery. Figure 5(a) shows that six refinement iterations improve retargeting success by 23.8, 18.3, and 30.0 percentage points for Cut Sand, Reset Plate, and Pnp Ring, respectively, demonstrating the benefit of revising affordances using simulated task outcomes.

Simulation retargeting success with unseen tools

Figure 5. Simulation retargeting success with unseen tools. (a) Simulation-guided refinement progressively improves task success for Cut Sand, Reset Plate, and Pnp Ring by testing and refining tool affordances. (b) Increasing the number of source tools for affordance discovery generally raises the mean success rate for Pnp Ring and reduces success-rate variation across unseen tools. The shaded band spans their minimum and maximum success rates.

Figure 5(b) varies source-tool set size for Pnp Ring affordance discovery. From one to five source tools, mean success across unseen tools rises from 56.2% to 81.3%, and their minimum–maximum success-rate range narrows from 0.0–100.0% to 78.7–86.4%, showing higher mean success and less variation across unseen tools. These trends suggest that with few source tools, the agent may identify affordance features and extraction skills suited only to a small subset of tools. Increasing source-tool diversity helps the agent discover shared functional structure and develop more broadly applicable affordance features and extraction skills. Iteration-wise rates and source-tool-count results are listed in Appendix § 8 (Table A3 and Table A4).

IterationCut SandReset PlatePnp Ring
070.065.051.3
184.870.058.8
289.781.758.8
393.081.780.2
493.381.780.2
593.683.381.3
693.883.381.3

Table A3. Simulation retargeting success (%) at each refinement iteration.

Source toolsMean (%)Minimum (%)Maximum (%)
156.20.0100.0
257.10.097.2
378.752.5100.0
471.654.087.9
581.378.786.4

Table A4. Mean and minimum–maximum range of simulation retargeting success (%) across unseen tools for Pnp Ring versus the number of source tools used for affordance discovery.

Figure 6 shows keyframes of real-robot task execution with different tools. A2^{2}-VLA adjusts its execution strategy according to each tool’s geometry and functional region.

Keyframes of real-robot task execution with different tools

Figure 6. Keyframes of real-robot task execution with different tools. (a) Pnp Ring: the robot uses a flat or gently concave hook region near the tool tip to engage and lift the ring. (b) Cut Sand: the robot uses a working edge or terminal region to cut through the sand and transfer the cut portion into the bowl. (c) Reset Plate: the robot uses a suitable part of the tool to form a support plane, lifting the plate from the rack and placing it on the upper level. A2^{2}-VLA adapts its execution strategy to each tool’s geometry.

Limitations

Our method is currently limited to tool-transfer problems that can be addressed effectively through retargeting. Although task-relevant affordance features can be determined autonomously, some forms of tool-skill transfer in real deployment cannot be solved simply through retargeting. In addition, the VLA may fail to follow correct affordance guidance, leading to task failure.

Conclusion

A2^{2}-VLA uses agent reasoning and planning to guide VLA task execution by autonomously determining task-relevant affordance features. Starting from demonstrations with a small set of source tools, the agent reconstructs scenes, identifies functional regions and their key geometric features, and continually improves affordance-extraction skills through simulated retargeting tests. At deployment, the agent uses the validated skills to extract the affordances of an unseen tool and supplies them as inputs that guide the VLA policy in completing the task. This enables reliable skill transfer and ultimately improves task success on the real robot.

Task-Specific Evaluation Details

Final successful episodes: 75/80 for Cut Sand, 50/60 for Reset Plate, and 65/80 for Pnp Ring.

Gains from simulation-guided refinement. From iterations 0 to 6, Table A3 reports gains of 23.8, 18.3, and 30.0 percentage points for Cut Sand, Reset Plate, and Pnp Ring, respectively. Pnp Ring improves most at iteration 3, and Cut Sand at iteration 1. Continued affordance testing and refinement stabilize success at higher levels.

Mean success and across-tool range. In Table A4, using one to five source tools for affordance discovery raises mean success across unseen tools by 25.1 percentage points. Their minimum–maximum success-rate range shrinks from a width of 100.0 to 7.7 percentage points, with the minimum rising from 0.0% to 78.7%, showing higher mean success and less cross-tool variation.

Task-Specific Affordance Discoveries

We record the agent’s final discoveries for each task in two parts: the affordance features identified through simulation-guided refinement and their encoding. The vectors below are task-specific instances of ft\mathbf{f}_t; the time index is omitted. Positions and directions are expressed in the robot-base frame. We use p\mathbf{p} for a functional-region core, t\mathbf{t} for a unit tangent, n\mathbf{n} for a unit normal, and d\mathbf{d} for a task-specific direction, with semantic subscripts for each task. For a symmetric 3×33 \times3 matrix M\mathbf{M}, the common packing operation is

vec⁡6(M)=[Mxx,Myy,Mzz,Mxy,Mxz,Myz]T.\operatorname{vec}_6(\mathbf{M}) = [M_{xx}, M_{yy}, M_{zz}, M_{xy}, M_{xz}, M_{yz}]^{\mathsf{T}}.

Pnp Ring

Discovered features. The agent identifies a short, locally linear terminal segment that hooks and supports the ring. It records the region’s core position phook\mathbf{p}_{\mathrm{hook}}, terminal tangent thook\mathbf{t}_{\mathrm{hook}}, and concavity-opening direction dhook\mathbf{d}_{\mathrm{hook}} that allows the ring to be supported during lifting against gravity. The directional feature is absent when the segment has no distinguishable curvature.

Encoding. We encode the terminal axis through its tangent outer product and retain the opening direction separately:

fhook=[phookvec⁡6(thookthookT)dhook]∈R12.(4)\mathbf{f}_{\mathrm{hook}} = \begin{bmatrix} \mathbf{p}_{\mathrm{hook}} \\ \operatorname{vec}_6\left(\mathbf{t}_{\mathrm{hook}}\mathbf{t}_{\mathrm{hook}}^{\mathsf{T}}\right) \\ \mathbf{d}_{\mathrm{hook}} \end{bmatrix} \in\mathbb{R}^{12}. \tag*{(4)}

An absent opening direction is stored as dhook=0\mathbf{d}_{\mathrm{hook}} = \mathbf{0}.

Cut Sand

Discovered features. The agent identifies a short, locally linear terminal working segment. Its features are the working-edge core pedge\mathbf{p}_{\mathrm{edge}}, unit edge tangent tedge\mathbf{t}_{\mathrm{edge}}, and optional cutting-face direction dcut\mathbf{d}_{\mathrm{cut}} specifying how the segment cuts into the material. This directional feature is absent for rod-like tools without a distinct sharp edge or cutting-face direction.

Encoding. We encode the working-edge axis through the same tangent outer product and retain the cutting-face direction:

fcut=[pedgevec⁡6(tedgetedgeT)dcut]∈R12.(5)\mathbf{f}_{\mathrm{cut}} = \begin{bmatrix} \mathbf{p}_{\mathrm{edge}} \\ \operatorname{vec}_6\left(\mathbf{t}_{\mathrm{edge}}\mathbf{t}_{\mathrm{edge}}^{\mathsf{T}}\right) \\ \mathbf{d}_{\mathrm{cut}} \end{bmatrix} \in\mathbb{R}^{12}. \tag*{(5)}

An absent cutting-face direction is stored as dcut=0\mathbf{d}_{\mathrm{cut}} = \mathbf{0}.

Reset Plate

Discovered features. The agent selects suitable parts of the tool that together form a support plane for lifting the plate. It records the core position psup\mathbf{p}_{\mathrm{sup}} of the selected support region, the unit normal nsup\mathbf{n}_{\mathrm{sup}} of the resulting support plane, and the lifting direction dlift\mathbf{d}_{\mathrm{lift}}.

Encoding. We encode the constructed support plane’s normal axis through its outer product, using the same six-entry packing as the line cases, and retain the signed lifting direction:

fplate=[psupvec⁡6(nsupnsupT)dlift]∈R12.(6)\mathbf{f}_{\mathrm{plate}} = \begin{bmatrix} \mathbf{p}_{\mathrm{sup}} \\ \operatorname{vec}_6\left(\mathbf{n}_{\mathrm{sup}}\mathbf{n}_{\mathrm{sup}}^{\mathsf{T}}\right) \\ \mathbf{d}_{\mathrm{lift}} \end{bmatrix} \in\mathbb{R}^{12}. \tag*{(6)}

The six normal-axis entries are invariant to the sign of nsup\mathbf{n}_{\mathrm{sup}}; dlift\mathbf{d}_{\mathrm{lift}} retains the signed lifting direction.

Model Architecture and Training Details

Single-Token Affordance Conditioning

We retain the π0.5\pi_{0.5} policy architecture [14], with its Gemma-based transformer backbone, visual encoder, language encoder, and robot-state projection. The agent supplies the current tool’s encoded functional-region feature vector ft∈R12\mathbf{f}_t \in\mathbb{R}^{12}, constructed as described in Appendix § 9. A learned affordance encoder EaffE_{\mathrm{aff}} projects this vector into a single affordance feature token zt=Eaff(ft)\mathbf{z}_t = E_{\mathrm{aff}}(\mathbf{f}_t) at the backbone’s context-embedding width. The complete conditioning prefix is

Xt=[Ximg,Xlang,Xstate,zt],(7)\mathbf{X}_t = [\mathbf{X}_{\mathrm{img}}, \mathbf{X}_{\mathrm{lang}}, \mathbf{X}_{\mathrm{state}}, \mathbf{z}_t], \tag*{(7)}

where the first three groups encode ItI_t, ℓ\ell, and qt\mathbf{q}_t, respectively. Thus each observation contributes one affordance feature token alongside the standard multimodal context. Its value is determined by the tool’s current functional-region features. All twelve feature coordinates are provided to the encoder; an absent task-specific direction is represented by the zero vector specified in Appendix § 9.

Action Expert and Prediction Heads

The action horizon is represented by HH noisy action tokens in the action-expert suffix. During each flow-matching step, the action expert processes these tokens while attending to the conditioning prefix Xt\mathbf{X}_t. Let hτ,h\mathbf{h}_{\tau,h} denote its hidden state for action step t+ht+h, where h∈{0,…,H−1}h \in\{0,\ldots,H-1\}, at flow time τ\tau. Two heads read these states:

v^h=Dact(hτ,h),\widehat{\mathbf{v}}_h = D_{\mathrm{act}}(\mathbf{h}_{\tau,h}),
F^t[h]=Daff([Qaff(hτ,h),zt]).(8)\widehat{\mathbf{F}}_t[h] = D_{\mathrm{aff}}([Q_{\mathrm{aff}}(\mathbf{h}_{\tau,h}), \mathbf{z}_t]). \tag*{(8)}

The action prediction head DactD_{\mathrm{act}} directly maps each action-step state to a 14-dimensional flow velocity. In the auxiliary affordance prediction head, a query MLP QaffQ_{\mathrm{aff}} maps the same state to an action-step query. An output MLP DaffD_{\mathrm{aff}} fuses this query with the single affordance feature token zt\mathbf{z}_t by concatenation and predicts a 12-dimensional affordance feature vector. The query varies across action steps, while zt\mathbf{z}_t supplies the current tool’s functional-region context throughout the chunk. Stacking these vectors over the horizon gives F^t∈RH×12\widehat{\mathbf{F}}_t \in\mathbb{R}^{H \times12}. Both outputs are obtained from the same action-expert forward pass.

Sequence Supervision and Training

For each source tool, the agent identifies the functional region and develops the task-relevant extraction skill. We compose the tool’s pose relative to the wrist with the wrist pose from robot forward kinematics, expressing the functional region in the robot-base frame. The current input ft\mathbf{f}_t uses the observed configuration qt\mathbf{q}_t; each subsequent target ft+h+1\mathbf{f}_{t+h+1} uses the configuration commanded by at+h\mathbf{a}_{t+h}. Stacking these targets gives Ft∈RH×12\mathbf{F}_t \in\mathbb{R}^{H \times12}, paired step by step with At∈RH×14\mathbf{A}_t \in\mathbb{R}^{H \times14}.

We use the backbone’s action flow-matching objective. Given Gaussian noise ϵ\epsilon and flow time τ\tau, the noisy action chunk is xτ=τϵ+(1−τ)At\mathbf{x}_{\tau} = \tau\epsilon+ (1-\tau)\mathbf{A}_t, with target velocity ϵ−At\epsilon- \mathbf{A}_t. With zero-based row indexing, At[h]=at+h\mathbf{A}_t[h] = \mathbf{a}_{t+h} and Ft[h]=ft+h+1\mathbf{F}_t[h] = \mathbf{f}_{t+h+1} for h=0,…,H−1h = 0,\ldots,H-1. The two loss terms are

Lact=114H∑h=0H−1∥v^h−(ϵ[h]−At[h])∥22,Laff=112H∑h=0H−1∥F^t[h]−Ft[h]∥22.(9)\begin{aligned} \mathcal{L}_{\mathrm{act}} &= \frac{1}{14H}\sum_{h=0}^{H-1}\left\lVert\widehat{\mathbf{v}}_h - \left(\epsilon[h] - \mathbf{A}_t[h]\right)\right\rVert_2^2, \\ \mathcal{L}_{\mathrm{aff}} &= \frac{1}{12H}\sum_{h=0}^{H-1}\left\lVert\widehat{\mathbf{F}}_t[h] - \mathbf{F}_t[h]\right\rVert_2^2. \tag*{(9)} \end{aligned}

The auxiliary objective supervises all twelve coordinates, including zero-valued direction entries. The affordance encoder and auxiliary prediction head are randomly initialized, and the VLA is initialized from the pretrained π0.5\pi_{0.5} checkpoint. The training set includes the 10% counterfactual trajectory-retargeting data described in § 4.3.

Training budget. All VLA-based methods in the real-robot comparisons—π0.5\pi_{0.5}, w/o Sim. Iteration, Human Oracle, and Full A2\mathrm{A}^{2}-VLA—are trained for 35,000 steps with a batch size of 32.

Policy Inference

At each policy update, the current observation, instruction, and affordance feature token form the conditioning prefix. Starting from Gaussian action noise, the action expert iteratively predicts flow velocities and denoises the action chunk from τ=1\tau= 1 to τ=0\tau= 0. The robot executes part of this chunk before receiving a new observation and replanning. The auxiliary head produces the corresponding affordance trajectory. Spatial components of ftf_t are updated from robot kinematics and the tool’s attachment pose, and the encoder produces the updated feature token for the next control cycle.

References

  1. [1]M. Tuluhan Akbulut, Varun Satheesh, Ahmed Jaafar, Alper Ahmetoglu, Shane Parr, Aditya Ganeshan, Shivam Vats, and George Konidaris. Creative robot tool use by counterfactual reasoning. arXiv preprint arXiv:2605.05411, 2026.arxiv.org/abs/2605.05411
  2. [2]Ondrej Biza, Skye Thompson, Kishore Reddy Pagidi, Abhinav Kumar, Elise van der Pol, Robin Walters, Thomas Kipf, Jan-Willem van de Meent, Lawson L. S. Wong, and Robert Platt. One-shot imitation learning via interaction warping. In Proceedings of the 7th Conference on Robot Learning, pages 2519–2536, 2023.arxiv.org/abs/2306.12392
  3. [3]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, et al. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems, 2025.DOI
  4. [4]William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Ria Doshi, Andy Tang, Danny Driess, Karl Pertsch, and Sergey Levine. Steerable vision-language-action policies for embodied reasoning and hierarchical control. arXiv preprint arXiv:2602.13193, 2026.arxiv.org/abs/2602.13193
  5. [5]Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. ReKep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024.arxiv.org/abs/2409.01652
  6. [6]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, et al. OpenVLA: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
  7. [7]Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, et al. HAMSTER: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.05485, 2025.arxiv.org/abs/2502.05485
  8. [8]Yihan Li, Yating Feng, Shengjiu Sun, Jianing Chen, Hao Ren, Bowen Yang, Weisheng Xu, Qiwei Wu, Hui Cheng, and Renjing Xu. Agentic RSR: Real-to-sim-to-real through scene reconstruction and execution-grounded robot policies. arXiv preprint arXiv:2610.10479, 2026.arxiv.org/abs/2610.10479
  9. [9]Yuyao Liu, Jiayuan Mao, Joshua B. Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. One-shot manipulation strategy learning by making contact analogies. In IEEE International Conference on Robotics and Automation, 2025.arxiv.org/abs/2411.09627
  10. [10]Runyu Lu, Yubo Wu, Ethan Kou, Haotian Lin, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Jan Kautz, Yuke Zhu, Linxi Fan, and Guanzhi Wang. ASPIRE: Agentic /skills discovery for robotics. arXiv preprint arXiv:2607.00272, 2026.arxiv.org/abs/2607.00272
  11. [11]Rundong Luo, Yunong Liu, Di Cao, Matthew Tancik, Shyamal Buch, Colton Stearns, and Wenqi Xian. SceneRig: An agentic system for simulation-ready 3D scene reconstruction from single images. arXiv preprint arXiv:2610.28585, 2026.arxiv.org/abs/2610.28585
  12. [12]Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. RT-Affordance: Affordances are versatile intermediate representations for robot manipulation. arXiv preprint arXiv:2411.02704, 2024.arxiv.org/abs/2411.02704
  13. [13]Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. TAX-Pose: Task-specific cross-pose estimation for robot manipulation. In Proceedings of the 6th Conference on Robot Learning, pages 1783–1792, 2023.arxiv.org/abs/2211.09325
  14. [14]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, et al. π₀.₅: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.arxiv.org/abs/2504.16054
  15. [15]Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. KETO: Learning keypoint representations for tool manipulation. In IEEE International Conference on Robotics and Automation, 2020.arxiv.org/abs/1910.11977
  16. [16]Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. arXiv preprint arXiv:2502.19417, 2025.arxiv.org/abs/2502.19417
  17. [17]Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Pulkit Agrawal. SE(3)-equivariant relational rearrangement with neural descriptor fields. In Proceedings of the 6th Conference on Robot Learning, pages 835–846, 2023.arxiv.org/abs/2211.09786
  18. [18]Chao Tang, Anxia Xiao, Yuhong Deng, Tianrun Hu, Wenlong Dong, Hanbo Zhang, David Hsu, and Hong Zhang. MimicFunc: Imitating tool manipulation from a single human video via functional correspondence. In Proceedings of the 9th Conference on Robot Learning, pages 4473–4492, 2025.arxiv.org/abs/2508.13534
  19. [19]Dylan Turpin, Liquan Wang, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. GIFT: Generalizable interaction-aware functional tool affordances without labels. arXiv preprint arXiv:2106.14973, 2021.arxiv.org/abs/2106.14973
  20. [20]Bohan Wu, Paul de La Sayette, Li Fei-Fei, and Roberto Martín-Martín. GeT-USE: Learning generalized tool usage for bimanual mobile manipulation via simulated embodiment extensions. arXiv preprint arXiv:2510.25754, 2025.arxiv.org/abs/2510.25754
  21. [21]Siyu Xu, Zijian Wang, Yunke Wang, Chenghao Xia, Tao Huang, and Chang Xu. Affordance field intervention: Enabling VLAs to escape memory traps in robotic manipulation. arXiv preprint arXiv:2512.07472, 2025.arxiv.org/abs/2512.07472
  22. [22]Tianling Xu, Weijie Wan, Honghao Su, Kailun Su, Junyuan Tang, Huichi Zhou, Haozheng Zhang, Qi Xiong, Ping Luo, Wenbo Ding, Tianxing Chen, and Yue Chen. WorldMirror: Agentic real-to-sim generation of interactive worlds from in-the-wild robot demonstrations. arXiv preprint arXiv:2610.31834, 2026.arxiv.org/abs/2610.31834
  23. [23]Chuhao Zhou, Liquan Wang, Shuxin Cao, Xiangyu Chen, Yuxuan Hu, Boyu Ma, Animesh Garg, and Jianfei Yang. FuncBridge: Towards functional tool-use generalization via keypoint trajectory reasoning. arXiv preprint arXiv:2607.05780, 2026.arxiv.org/abs/2607.05780

Paper details

vlnc:W_rw97vqw7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents