BEHAVIOR-Spatial Benchmark: Evaluating Spatial Understanding for Robotics

Introduction

Instructions such as “reach the second tool from the left” require a robot to resolve a spatial relation, identify the intended target, and execute an action. Navigation similarly requires interpreting destinations relative to current and previously observed surroundings. Vision-language-action models (VLAs) are initialized from vision-language models (VLMs) and assume that the backbone’s spatial understanding supports robot behavior, yet this connection is rarely studied directly. Task success alone provides ambiguous evidence: a policy may exploit familiar layouts without following spatial language, or select the correct target but fail during execution. Relating VLM spatial capabilities to VLA behavior requires jointly evaluating spatial perception, target grounding, and action.

We introduce BEHAVIOR-Spatial, a benchmark built on BEHAVIOR-1K (Li et al., 2024a) that connects these evaluations across household manipulation and navigation. It comprises three components: (1) BEHAVIOR-Spatial-VQA, a spatial VQA suite containing 3,256 questions across 49 task activities, including 1,572 general spatial questions and 1,684 task-specific questions, with both single- and multi-frame evaluations; (2) BEHAVIOR-Spatial-IR, a point-based intermediate representation (IR) generation suite containing 1,240 target-grounding queries across 14 manipulation tasks; and (3) BEHAVIOR-Spatial-Action, a robot action suite covering the same 14 manipulation tasks and 10 navigation tasks. Shared scenes and relations across suites enable studying how VLM question-answering and grounding accuracy relate to VLA performance.

Our evaluation distinguishes generalization to new configurations from responsiveness to spatial instructions. Held-out manipulation and navigation configurations test generalization, while a manipulation prompt probe changes the requested target within a training layout to test whether behavior follows the instruction. We measure VQA answer accuracy and whether generated points fall within the instructed target region. For robot action, we report soft success as a behavioral proxy for target-selection intention, alongside wrong-target commitments and strict success.

Our experiments reveal persistent spatial instruction-following failures, especially when the target changes within a familiar layout. The tested LoRA adaptation and DROID (Khazatsky et al., 2025) initialization strategies provide limited improvements on this probe, whereas supplying IR visual target cues substantially improves target-consistent behavior. These cues isolate the benefit of providing the grounding decision, rather than demonstrating learned grounding from language. For two of our best manipulation variants, the dot-IR and attention-map policies, grounding accuracy correlates more strongly with soft success than VQA accuracy does. However, the perception–action relationship varies across policy variants, and direction-question accuracy shows little correlation with navigation success across the ten evaluated tasks.

Our contributions are threefold: (1) a spatial benchmark connecting question answering, point-based target grounding, and household manipulation and navigation; (2) a diagnostic evaluation protocol that tests changed instructions in training layouts and separates target-consistent behavior from task completion; and (3) an empirical study of policy adaptation and visual grounding interfaces relating VLM spatial understanding to VLA behavior.

Related Work

Spatial understanding in VLMs and its benchmarks. A growing body of work studies spatial reasoning in vision-language models, including 3D-grounded spatial QA ([4, 6]), relative position and multi-view correspondence (What’sUp (Kamath et al., 2023), BLINK ([12])), egocentric video (VSI-Bench (Yang et al., 2025)), spatial mental models (MindCube (Wang et al., 2026a)), and atomic embodied capabilities such as pointing (BEAR (Qi et al., 2026)). Robo2VLM ([5]) is closest to our VQA setting, generating questions from real robot trajectories. These benchmarks largely assess spatial question answering or pointing in isolation. BEHAVIOR-Spatial instead aligns VQA, pointing, and execution on the same scenes, objects, and relations, testing whether spatial understanding transfers to grounded action.

VLMs as the foundation of VLA models. Vision-language-action models augment pretrained VLM backbones with action prediction or control modules ([3, 2, 16, 13, 13] (Kim et al. 2024; Team et al. 2024; NVIDIA et al. 2025)), implicitly relying on the backbone’s semantic and spatial knowledge to transfer into the policy. Recent work has begun to examine this assumption: VLM4VLA (Zhang et al., 2026a) studies backbone choice; Hancock et al. (2025) identify catastrophic forgetting of vision-language abilities during action finetuning; and Qwen-VLA (Wang et al., 2026b) uses joint vision-language and action training to mitigate this issue. OpenHelix ([8]) and Mind-to-Hand (Tang et al., 2025) further incorporate explicit embodied reasoning. Our benchmark complements these efforts by separately measuring VQA, referent grounding, and execution.

Benchmarks and evaluation for robot policies. CALVIN (Mees et al., 2022), LIBERO (Liu et al., 2023), VLABench (Zhang et al., 2024), and SimplerEnv (Li et al., 2024b) evaluate language-conditioned policies primarily through end-task success, while BEHAVIOR-1K (Li et al., 2024a) provides the long-horizon household activities and simulator we build on. Federated evaluations such as The Colosseum (Pumacay et al., 2024) and DOMINO ([10]) expose specific sources of policy brittleness. BEHAVIOR-Spatial similarly adopts a diagnostic evaluation, but targets spatial language grounding: prompt-probe pairs hold the training scene fixed while varying the instruction, and outcome buckets and strict/soft metrics separate grounding from execution.

Intermediate representations and explicit grounding for policies. Points, marks, traces, and sketches have been used as interfaces between perception and control (Lee et al., 2025; Li et al., 2026; Zhang et al., 2026b). We use such an intermediate representation for a different purpose: as a controlled intervention on an otherwise fixed policy. Providing an object-center mark tests whether supplying the correct referent resolves the grounding failure and isolates the remaining execution gap. We then ask whether the grounding signal can instead be supplied by a VLM at rollout time.

The Behavior-Spatial Benchmark

Design Overview

BEHAVIOR-Spatial comprises three components, overviewed in Figure 2.

BEHAVIOR-Spatial-VQA evaluates spatial understanding through general and task-grounded questions. The general subset covers spatial relations within individual views and reasoning across multiple views, while the task-grounded subset queries objects and relations relevant to the robot tasks. BEHAVIOR-Spatial-IR evaluates whether a model can ground a spatial instruction by predicting a point within the specified target object or placement region. Finally, BEHAVIOR-Spatial-Action evaluates spatially grounded action through manipulation and navigation tasks in which spatial instructions determine the intended target or destination.

Task-grounded VQA and BEHAVIOR-Spatial-IR (together, the task-grounded perception suites) share task contexts, objects, and spatial relations with the action suite, enabling task-level comparison between VLM perception and VLA behavior.

Spatial Taxonomy

BEHAVIOR-Spatial covers direction, distance, and size. Direction includes relative-position relations in object-, and robot-relative frames, as well as region identification (locate) and spatial ordering (ordinal). Distance and size require comparisons of physical separation and object size, respectively. Spatial VQA further distinguishes single-frame understanding from multi-frame reasoning across robot rotation or navigation. Appendix § 11.§ 1 details the concepts and settings.

Dataset Construction

We construct BEHAVIOR-Spatial from BEHAVIOR-1K demonstrations, using recorded observations and actions together with object geometry and camera metadata recovered through OmniGibson replay as needed. Figure 3 summarizes the pipeline. We curate 14 manipulation tasks and 10 navigation tasks across five navigation scenes. Manipulation tasks require identifying a target among spatially distinct candidates; navigation tasks require grounding an instructed destination. Instruction variants specify different target objects or goal regions, and curated scene-context traversals support navigation. Appendix § 12 describes task curation and lists the tasks and variants.

A metadata-to-template pipeline generates spatial questions and their answers, with visibility checks, relation-margin filters, and sampling controls to reduce ambiguity and imbalance. General VQA spans 49 activities and contains 1,078 single-frame and 494 multi-frame questions. Task-grounded VQA derives from the curated robot task instances and contains 1,082 manipulation and 602 navigation questions. For point grounding, we compute target-object references and placement positions or regions from segmentation and simulator geometry. VQA and grounding use distinct visibility criteria. See Appendix § 11 for curation details.

VLA Model

Model Architecture

BEHAVIOR-Spatial-VLA builds on the $\pi_{0.5}$ architecture (Intelligence et al., [16]), replacing its VLM backbone with Qwen3.5-4B (Team, 2026). The VLM encodes visual observations and the language instruction, with discretized robot proprioception included in the text prefix. A flow-matching action expert attends to the resulting prefix features and predicts 32-step chunks of 23-dimensional actions controlling the torso, dual arms, grippers, and mobile base. Unless otherwise stated, the VLM is frozen and the action expert is trained from scratch.

For manipulation, policies are trained on BEHAVIOR-Spatial-Action’s manipulation demonstrations and receive head- and two wrist-camera views. IR-conditioned variants additionally receive oracle grounding images through the same visual encoding interface. For navigation, policies receive the current head-camera view and a variable-length sequence of curated scene-context frames, together with the instruction and robot state.

Policy Variants

We compare six manipulation policies, all conditioned on camera observations, the instruction, and robot state: (1) Baseline (frozen VLM); (2) +LoRA-VLM, which adapts the vision and language components through LoRA; (3) +DROID pretrain init, which initializes from DROID pretraining before task finetuning; (4) +dot IR, which adds a target-marked head-camera RGB image; (5) +dot IR + first-frame anchor, which also supplies a fixed, target-marked initial head view; and (6) +attn-map IR, which uses a standalone Gaussian heatmap in place of the dot-marked image.

These variants test backbone adaptation, robot-data transfer, and alternative grounding interfaces. The dot and attention-map comparison changes both spatial spread and image background. For navigation, we train a single frozen-VLM policy jointly on the 10 tasks. Appendices C.1 and C.2 detail adaptation, initialization, and training. Appendix § 13.§ 3 specifies IR construction.

Policy Evaluation Protocol

Held-out & Prompt-probe evaluation. The held-out set ($n=469$) uses scene configurations from unseen demonstrations, paired with unseen instruction variants that reverse the spatial relation used during training. This setting tests generalization under changes to both scene configuration and instruction. The prompt-probe set ($n=700$) retains scene configurations used during training but changes the instruction to specify a different valid target. This setting tests whether policies follow the instruction when it conflicts with the demonstrated target.

Manipulation evaluation. All manipulation policies are evaluated on the same instances. Episodes use per-task step budgets calibrated to the corresponding demonstration horizons and are scored using the metrics defined in § 5.1. For IR-conditioned policies, we provide ground-truth grounding references computed from simulator state. This oracle setting evaluates action performance given correct grounding, independently of errors in VLM grounding predictions.

Navigation evaluation. We evaluate the jointly trained 10-task navigation policy on spawn-validated held-out instances. Appendix § 13.§ 4 describes instance construction and visualizes initial poses and scene layouts. Training settings and evaluated checkpoints are provided in Appendix § 13.§ 2.

Code-as-policy agents

Inspired by Fu et al. [11], we additionally evaluate code-as-policy agents, in which a writer language model generates a Python program that controls the robot through a perception and control API including base and end-effector motion primitives, gripper control, RGB-D observations, and shared perception tools (an open-vocabulary pointing model and a VLM). Programs never receive privileged object poses, identities, or segmentation. For each task, the writer iteratively develops a program over 10 training episodes, disjoint from the test set, using execution traces, final observations, and simulator-derived outcome feedback. A validation-and-fallback procedure selects a program, which is then frozen and executed without writer revisions on all test episodes. Programs are evaluated on the same held-out episodes and with the same success metrics as the VLA policies. Appendix § 16.§ 5 describes the program-development details, execution settings, and per-task results.

Experiments and Results

Metrics

VQA and IR Accuracy. VQA accuracy measures the fraction of questions answered correctly under the four-option evaluation interface; unparseable responses count as incorrect. For IR evaluation, we ask model to predict a target point, measuring strict IR accuracy uses task-specific target-region criteria and tolerant IR accuracy that additionally allow small localization errors for reach tasks. We report task-grounded manipulation VQA and IR accuracy as averages of per-task accuracies. Detailed scoring rules, tolerances, and aggregation conventions are provided in Appendix § 14.

Robot Task Success. For manipulation, we primarily use soft success as a proxy for target-selection intention. It credits completed correct interactions as well as non-committed episodes that exhibit correct intention for target approaching under task-specific rules and timeout budgets. We also report strict success for task completion rate, and the wrong-target rate measuring wrong commitments to non-instructed candidates or locations. All three rates report percentage over all evaluated episodes. Task-family rules and exceptions are detailed in Table S4. Navigation uses the recorded binary goal result: the robot must satisfy the instructed region-entry or object-proximity criterion within the episode timeout budget. We report per-task and pooled success, without the strict/soft distinction.

Family	Strict success (correct)	Additional soft credit (for none)	Tasks
Reach (contact)	The end-effector touches the instructed candidate. Touching a non-instructed candidate is wrong.	The minimum end-effector distance to the instructed target over the episode is smaller than the minimum distance to every distractor.	food, item, plate, toys, dice, paperback, pizza, storage, tool, hardback
Reach (base proximity)	The end-effector touches the instructed can, or the base ends within 0.70 m of it and satisfies the nearest-candidate rule.	Same minimum-distance rule as contact reach.	soda
Directional placement	The object is released onto the countertop, at least 0.17 m from the reference, on the instructed side along the robot-right axis. The opposite side is wrong; releases within 0.17 m or off the countertop are on-ref failures.	At episode end, the still-held object lies on the instructed side along the robot-right axis.	carving, saucer
Container placement	The object rests inside the instructed compartment’s height band. A different compartment is wrong; outside the container is none.	The object’s peak height within the container footprint falls within the permitted band around the instructed compartment. Passing through the band before overshooting does not qualify.	tupperware, bowl
Navigation	The robot base satisfies the instructed goal criterion (binary subtask_success); no strict/soft split applies.	—	all move.to

VLM Evaluation on BEHAVIOR-Spatial-VQA and BEHAVIOR-Spatial-IR

How well do VLMs answer spatial questions? Table 1 evaluates 14 VLMs, comprising five proprietary – GPT-6 Astra (OpenAI, 2026), Gemini 3 (Google DeepMind, [14]), Fable5.1 (Anthropic, [1]), GPT-5 (Singh et al., 2026), and Gemini 2.5 (Comanci et al., [7]), and nine open-weight models spanning Qwen (Qwen Team, 2026), Molmo (Deitke et al., [9]), Gemma (Gemma Team et al. [13]), and DeepSeek (Wu et al. (2024)) families. VQA evaluation prompts, including spatial conventions and multi-frame context instructions, are provided in Appendix § 15.§ 1. The highest single-frame accuracy is 82.1%, compared with 51.8% for the strongest open-weight model. Multi-frame performance varies substantially across models, with several open-weight models at or below the random-guess baseline on direction questions. Spatial VQA thus remains challenging, especially across multiple views. Extracted VLA backbones also score poorly (Appendix § 16.§ 1).

Model	Single-frame				Multi-frame
	All	Dir.	Dist.	Size	All	Dir.
Proprietary
GPT-6 Astra	82.1	85.8	77.3	77.7	74.4	74.6
Gemini 3	77.8	80.5	75.2	72.7	71.1	69.6
Fable5.1	70.5	69.3	70.2	76.3	68.4	66.8
GPT-5	64.7	61.7	68.1	69.1	63.2	59.6
Gemini 2.5	59.9	55.0	65.2	68.3	60.1	57.4
Open-weight
Molmo2-8B	51.8	45.0	57.5	66.9	36.8	30.7
Qwen3.5-9B	50.6	42.0	58.1	69.1	54.5	51.4
Qwen3.5-4B	46.8	44.8	48.1	51.8	46.6	44.1
Molmo2-4B	45.5	34.2	58.7	62.6	32.4	23.7
PaliGemma	36.2	32.5	40.7	41.0	37.9	34.4
PaliGemma 2	34.8	29.8	39.2	45.3	27.3	23.4
Gemma 3-4B	34.4	35.0	33.6	33.8	29.2	21.3
Qwen2.5-VL	33.4	31.2	37.8	32.4	28.9	22.9
DeepSeek-VL2-Tiny	31.4	30.5	34.8	26.6	25.0	17.6
VLM backbones extracted from released VLA models
OpenHelix	9.6	5.8	14.2	15.1	—	—
$\pi_{0.5}$	0.9	0.8	0.3	2.9	—	—
Chance	25.0	25.0	25.0	25.0	25.0	25.0

Does general VQA performance transfer to task-grounded questions? We evaluate the best open-source models from the general VQA comparison on task-grounded questions from the manipulation suite (Table 2); only open-source models are evaluated, as only these can serve as our VLA backbones. Qwen3.5-9B achieves the highest accuracy at 57.2%, followed by Molmo2-4B at 53.0% and Molmo2-8B at 50.0%. Even Qwen3.5-9B attains 56.3% on direction and 61.4% on distance, but only 47.3% on locate sub-types of direction questions. Strong general VQA thus does not guarantee task-grounded perception, motivating task-grounded evaluation of candidate backbones.

Model	Task-grounded VQA				IR
	All	Dir.	Dist.	Size	Strict	Tolerant
Qwen3.5-9B	57.2	56.3	61.4	52.5	44.8	49.4
Molmo2-4B	53.0	54.1	56.4	46.8	26.8	39.5
Molmo2-8B	50.0	49.9	53.4	45.3	44.6	52.2
Qwen3.5-4B	49.4	50.9	48.6	48.6	38.6	46.6
Gemma 3-4B	33.5	23.0	42.4	36.0	34.1	44.1

How accurately can VLMs ground the instructed target? The IR evaluation asks models to predict a target point directly from the instruction and head-camera view (Table 2), using the prompts provided in Appendix § 15.§ 2. Qwen3.5-9B achieves the highest strict accuracy at 44.8%, followed by Molmo2-8B at 44.6% and Qwen3.5-4B at 38.6%. Under tolerant scoring, these accuracies increase to 49.4%, 52.2%, and 46.6%, respectively. Molmo2-4B and Molmo2-8B reverse order between VQA and pointing, so answer selection and localization probe different aspects of perception. We relate these task scores to policy behavior in §5.3.3; qualitative examples appear in Appendix § 16.§ 2.

VLA evaluation on robot tasks

We evaluate the policy variants of § 4.2 on the manipulation and navigation tasks of BEHAVIOR-Spatial-Action (Table 4 and Table S11). Checkpoints are selected around similar training steps for fair comparisons.

Policy	Held-out +Soft↑	Held-out S↑	Prompt-probe +Soft↑	Prompt-probe S↑	Prompt-probe W↓
$\pi_{0.5}$-Qwen (frozen VLM)	41.6	27.9	16.1	10.0	70.0
+ LoRA-VLM	43.5	30.9	17.9	9.6	68.1
+ DROID pretrain init	40.1	24.7	16.6	8.9	66.0
IR-conditioned
+ dot IR	60.3	44.8	38.0	24.9	43.9
+ dot IR + first-frame anchor	59.1	41.8	36.3	23.7	42.4
+ attn-map IR	59.3	39.0	35.0	17.0	46.0
Data scaling (§F.9)
+ dot IR @ 50% data	53.7	33.7	30.6	19.4	45.7
+ dot IR @ 30% data	56.1	34.5	32.6	19.9	46.6

Task	Scene / region	VQA↑	Success↑
move_to_bookcase	single-floor house, toy room	30.0	92.5
move_to_bathroom	single-floor house, bedroom	40.0	90.0
move_to_corner_booth	restaurant diner	23.3	87.5
move_to_table_in_kitchen	two-floor house, main floor	38.2	75.0
move_to_table_near_sofa	single-floor house, open living area	23.9	73.2
move_to_bar	restaurant diner	25.0	50.0
move_to_kitchen_sink	restaurant diner	16.7	45.0
move_to_fridge	two-floor house, main floor	33.8	35.0
move_to_oven	restaurant diner	36.7	27.5
move_to_garage	two-floor house, main floor	34.2	22.5
Pooled		30.2	59.9

The policy predicts 32-step action chunks using 10 flow-matching denoising steps per inference. Execution uses asynchronous receding-horizon control with a 10-step replanning threshold measured from the committed plan’s input-observation timestamp. Actions are temporally ensembled over up to three committed plans, with gripper commands taken from the newest plan. Details are in Appendix § 16.§ 3.

VLA performance on manipulation tasks

Results are task-dependent, with particularly limited instruction following when the requested target changes in training layouts. VLM adaptation and robot-data pretraining offer limited gains, while oracle target grounding yields larger improvements.

How well do policies act toward spatially specified targets? Pooled across manipulation tasks, on held-out instances the baseline achieves 41.6% soft and 27.9% strict success, with 38.2% wrong targets.

The baseline’s per-task results (Appendix § 16.§ 4) reveal distinct bottlenecks. For place_carving_knife, 70.8% soft success versus 29.2% strict success suggests execution difficulty despite frequent target-directed progress. Bowl placement achieves only 15.0% on both measures, with a 77.5% wrong-target rate, indicating that target selection itself is difficult. Both target selection and execution thus limit completion.

Do policies follow changed instructions in training layouts? The baseline often fails to redirect its behavior toward a newly requested target, even in training layouts. On prompt-probe episodes, it achieves only 16.1% soft and 10.0% strict success, while 70.0% of episodes commit to a non-instructed target. The policy relies on scene–action associations rather than the instruction. In the failed probe example in Figure S11, the instruction requests the largest food item, but the policy reaches the chip bag associated with the original training instruction rather than the indicated oatmeal box. Held-out target-directed behavior thus does not imply reliable instruction following.

Does VLM adaptation or robot-data pretraining improve instruction following? LoRA adaptation increases held-out soft success from 41.6% to 43.5%, but yields only a small change on prompt-probe episodes (16.1% to 17.9%), with 68.1% wrong-target commitments remaining. DROID initialization achieves 40.1% held-out and 16.6% prompt-probe soft success. Neither recipe improves strict probe success over the baseline (9.6% and 8.9%, respectively, versus 10.0%).

Does explicit target grounding improve policy performance? Oracle IR produces substantially larger gains: dot-IR conditioning raises soft success from 41.6% to 60.3% on held-out set and from 16.1% to 38.0% on the prompt probe. Across the three IR variants, probe wrong-target commitments decrease from 70.0% to 42.4–46.0%. Strict success also improves, reaching 44.8% held-out and 24.9% on the probe for the strongest variant with dot IR. Explicit grounding thus redirects policy behavior and improves completion.

Code-as-policy agents on manipulation tasks

Do code-as-policy agents act toward spatially specified targets? With the same perception tools and motion API, the proprietary writers achieve approximately 50–66% soft success, compared with 37–39% for the two open-weight writers. Thus, target-consistent behavior varies substantially with the model composing perception and control into executable programs. However, soft success does not imply reliable completion: Qwen3.5-4B and Qwen3.5-9B achieve only 5.1% and 5.3% strict success, respectively, whereas Gemini 3, GPT-6, and Fable 5.1 achieve 33.3%, 47.3%, and 51.4%. See Table S6 for per-task results.

Table S6: Per-task code-as-policy performance. Each cell reports soft / strict success (%). $N$ is the number of test episodes. Bold indicates the highest soft success within each task, including ties. Per-task values are rounded to whole percentages; the pooled row is computed from aggregate episode counts.

Task	$N$	Fable 5.1	GPT-6	Gemini 3	Qwen3.5-9B	Qwen3.5-4B
place_bowl_in_cabinet	40	72 / 72	48 / 32	28 / 28	20 / 20	18 / 3
place_carving_knife	24	100 / 58	75 / 58	38 / 21	50 / 8	71 / 0
place_upperware_in_fridge	22	18 / 18	59 / 9	5 / 5	18 / 9	5 / 5
reach_dice	32	97 / 94	91 / 88	78 / 56	66 / 0	56 / 0
reach_food_item	32	69 / 59	59 / 56	59 / 53	47 / 6	44 / 6
reach_hardback_from_bookcase	48	83 / 48	42 / 19	27 / 19	21 / 0	19 / 0
reach_item	32	56 / 50	53 / 50	34 / 28	3 / 0	9 / 0
reach_paperback_book_aug	22	64 / 45	73 / 64	64 / 41	50 / 0	45 / 0
reach_pizza_plate	40	55 / 22	72 / 28	58 / 12	50 / 0	62 / 0
reach_plate	31	45 / 35	65 / 61	45 / 32	32 / 0	32 / 0
reach_soda_can	42	64 / 57	76 / 76	64 / 45	57 / 7	52 / 26
reach_storage_box	32	72 / 59	78 / 72	88 / 66	56 / 25	47 / 28
reach_tool	32	56 / 41	47 / 38	56 / 3	62 / 0	31 / 0
reach_toys	40	60 / 50	38 / 28	58 / 52	22 / 0	32 / 0
Pooled	469	66.1 / 51.4	61.2 / 47.3	50.3 / 33.3	39.0 / 5.3	37.1 / 5.1

Writer	+Soft↑	S↑	W↓
Proprietary
Fable 5.1	66.1	51.4	23.5
GPT-6	61.2	47.3	23.0
Gemini 3	50.3	33.3	28.8
Open-weight
Qwen3.5-9B	39.0	5.3	8.7
Qwen3.5-4B	37.1	5.1	7.2

What failures remain? The open-weight writers frequently produce partial target-directed behavior without completing an interaction, with 83–87% of episodes ending without a decision. The proprietary writers complete interactions more frequently, but wrong-target commitments remain substantial.

Perception–Action correlation for manipulation

Do perception scores track spatially directed behavior? We compare Qwen3.5-4B’s task-level VQA and tolerant IR accuracies with each policy’s soft held-out success across 14 manipulation tasks, using Spearman’s rank correlation with average ranks for ties (Table 3). For two of the strongest-performing variants, dot IR and attention-map IR, grounding accuracy correlates more strongly with success than VQA accuracy ($\rho= 0.701$ versus 0.560, and 0.753 versus 0.608, respectively). This suggests that the metric that tracks the policy is the one the VLM has to generate, not the one it has to answer. Correlation plots are provided in Appendix § 16.§ 7.

Policy	VQA ($\rho$)	IR ($\rho$)
Baseline	0.713	0.658
LoRA VLM	0.616	0.581
IR	0.560	0.701
IR + first-frame IR	0.612	0.618
Attention map	0.608	0.753

Ablation Studies for manipulation tasks

We examine how target cues and demonstration design affect manipulation performance.

Does the IR rendering affect target-consistent behavior? We compare two ways of presenting the same oracle target center: a dot on the head-camera RGB image and a standalone Gaussian attention map. Both yield similar soft success on held-out and probe sets, but dot IR achieves higher strict success, possibly because it preserves the surrounding RGB context.

With VLM-predicted instead of oracle grounding, the dot-IR policy retains part of this benefit: Qwen3.5-4B predictions yield 48.8% held-out soft success, compared with 41.6% for the baseline. Performance remains below oracle grounding (60.3%). Appendix § 16.7 provides the rollout protocol and full results, including held-out comparisons across three grounding VLMs.

Does a first-frame reference improve performance? The target can leave the head-camera view as the robot moves toward it, motivating a fixed reference that retains the initial scene and target mark. We add the target-marked first frame as a fifth image input to the dot IR policy, held constant throughout the rollout. Both soft and strict success decrease slightly, so a persistent initial view adds no benefit over the current-view cue.

Does changing the demonstration distribution affect target selection? To examine the influence of demonstration geometry, we study the reach_paperback_book task in which the referred book is associated with a fixed side in the training data. We prefix each demonstration with motion from the two candidate books’ perpendicular bisector to the original start, changing the approach geometry while retaining the original interaction. The main 14-task evaluation uses the augmented paperback demonstrations. On the shared prompt-probe set, this augmentation reduces wrong-target commitments from 79.7% to 43.5% and raises soft probe success from 11.6% to 36.2%, while strict success remains 10.1%. This intervention changes the failure distribution without improving completion. Details are in Appendix § 16.8.

How much does performance depend on demonstration volume? We retrain our best-performing variant, the dot-IR policy, on nested 30% and 50% subsets of the demonstrations, preserving the task mixture and training for the same number of updates. Increasing the data from 30% to 100% raises held-out soft/strict success from 56.1%/34.5% to 60.3%/44.8%, and prompt-probe soft/strict success from 32.6%/19.9% to 38.0%/24.9%. The full-data policy performs best overall, although gains are not monotonic: the 50% subset yields similar or lower success than the 30% subset. Even with 30% of the demonstrations, dot IR substantially outperforms the full-data frozen-VLM baseline on both evaluation sets. See Appendix § 16.9 for details.

VLA performance on navigation tasks

How does navigation success vary across tasks? Success ranges from 22.5% for the garage to 92.5% for the bookcase (Table S11). The bookcase and bathroom tasks achieve high success on short-range navigations, while the garage task remains challenging. The diner scene and two-floor house also exhibit substantial variation across destinations, with success from 25–35% to 75–85%. Scene identity alone therefore does not account for the task ordering. Reaching a nearby object or region and navigating across a larger environment impose different demands on context use and motion. Route length, obstacles, and doorway crossings are plausible contributors to these differences. See Appendix § 16.12. Does navigation VQA accuracy track policy success? We evaluate Qwen3.5-4B on multi-frame direction-question for navigation, accuracy ranges from 16.7% to 40.0%. Across the ten tasks, the weak correlation with navigation success indicates that VQA accuracy alone does not explain variation in task success. Navigation-VQA examples are provided in Figure S13. The complete results for all ten navigation tasks are reported in Table S11 (Appendix § 16.11). The table pairs VQA accuracy and policy success by task and scene to show variation across environments.

Limitations and Conclusion

Limitations

Our evaluation is conducted entirely in simulation; transfer to physical robots remains to be established. In the tupperware task, correlations between the acting hand and the target layer can confound spatial instruction following with hand-specific manipulation behavior. Success rates also depend on the episode timeout budget, so cross-run comparisons should account for differences in the available execution time. Finally, the IR-conditioned policies require simulator-derived IR target centers during training; applying this training approach to datasets without such annotations would require an alternative source of grounding supervision.

Conclusion

We introduced BEHAVIOR-Spatial to jointly evaluate spatial question answering, target grounding, and robot action in household tasks. The evaluated policies often fail to act on the instructed target, particularly when prompts change in familiar layouts. Explicit IR target cues improve target-directed behavior, with gains persisting under VLM-predicted grounding, although reliable completion remains challenging. Perception scores offer policy-dependent diagnostics but do not consistently explain action success. By distinguishing target selection, partial progress, and task completion, BEHAVIOR-Spatial provides a framework to measure when spatial understanding supports successful action and when it fails.

AI Use Statement

Generative AI tools were used to assist with drafting and editing manuscript text, writing figure-generation scripts, and aggregating evaluation results from logged metrics. The authors reviewed the AI-assisted text and code and verified the reported results against the experimental logs. The authors take responsibility for the final manuscript, including all claims and artifacts.

Ethics Statement

Our benchmark uses simulated environments and programmatically generated labels. The grounding failures identified in this work highlight potential risks of deploying robots that do not reliably follow instructions. Simulation performance should not be interpreted as evidence of safe real-world deployment.

Reproducibility Statement

To support reproducibility, we describe benchmark construction and label generation in § 3.3, evaluation splits and protocols in § 4.3, and experimental settings in § 5. We will release the benchmark and code upon publication.

Additional Data Curation Details

This section supplements the pipeline in § 3.3 with spatial concepts, question curation and frame selection, and oracle IR construction.

Spatial Concepts and Observation Settings

We detail the three spatial concepts covered by BEHAVIOR-Spatial and the observation settings used for spatial VQA.

Direction. Direction concerns the relative position of an entity with respect to a reference, including left/right, front/back, and above/below relations. These relations are expressed in object- or robot-relative reference frames. We also include two subtypes that require finer-grained spatial identification: locate identifies the region of a container or piece of furniture occupied by an object, such as a shelf layer, fridge level, or bookcase grid cell; ordinal identifies an object by its rank in a spatial ordering, such as “the second leftmost tool.” Ordinal questions appear only in the task-grounded VQA subset, where they reflect the referring expressions used in the robot tasks.

Distance. Distance concerns comparisons of physical separation between entities. Questions and task instructions identify the candidate closest to or farthest from a reference object, the camera, or the robot arm. Resolving these relations requires comparing distances across candidates.

Size. Size concerns comparisons of physical size among objects of the same category, such as identifying the largest or smallest candidate. These comparisons require accounting for perspective, since an object’s apparent size in the image depends on both its physical size and its distance from the camera.

Observation setting. Single-frame questions evaluate spatial understanding from one rendered view. Multi-frame questions require integrating spatial information across viewpoints and comprise two settings. In rotation, the robot rotates in place, and questions require relating entities observed in different views within a common egocentric frame. In navigation, the robot moves through a scene, and questions require tracking spatial relations across observations while accounting for the robot’s displacement.

Notes. Braces indicate placeholders; slash-separated entries indicate alternative reference entities, not literal question wording. Each question has four answer options; $n$ denotes the number of evaluation questions. Single-frame direction comprises 150 object-to-object and 150 robot-arm-relative questions; including the locate subtype, it totals 600 questions. Single-frame distance comprises 143 object-relative, 143 camera-relative, and 53 robot-arm-relative questions.

VQA Generation, Filtering, and Frame Selection

BEHAVIOR-Spatial-VQA comprises general and task-grounded spatial questions, both generated through a metadata-to-template pipeline. For each sampled frame, we extract 3D object states, projected 2D bounding boxes, and robot reference frames. A spatial calculator computes direction relations in object- and robot-relative frames; metric distances, including perpendicular distances to support surfaces; size comparisons; and containment relations for locate questions. Per-category templates convert these relations into four-option multiple-choice questions, with distractors drawn from the corresponding objects or spatial relations.

General VQA filtering and sampling. Generation is followed by three filtering and sampling steps. First, relation-margin checks require at least a 15% relative separation for distance comparisons and a 10% margin for size comparisons. Second, visibility checks filter queried and candidate objects by their image pixel counts, with a stricter threshold for the ground-truth object where applicable. Third, a sublinear cap on questions per frame limits the contribution of cluttered scenes. We then subsample the resulting pool to control category coverage, yielding 1,078 single-frame and 494 multi-frame questions. These checks reduce ambiguity and sampling imbalance without guaranteeing that every geometric distinction is visually recoverable. Table S1 summarizes the question templates, answer spaces, and evaluation counts for the general VQA subset.

Setting	Category	Template	Answer space	$n$
single-frame	direction	“From the viewpoint of \{frame_idx\}, where is \{objA\} relative to \{objB / robot arm\}?”	left / right / front / back / above / below	300
single-frame	locate	“From the viewpoint of \{frame_idx\}, where is \{objA\} located on \{objB\}?”	container region (layer/level/grid)	300
single-frame	distance	“Measuring from the center of each object, which object is the closest to \{objA / camera / robot arm\}?”	candidate objects	339
single-frame	size	“Measuring from the length of the longest dimension (length, width, or height), which object is the longest?”	candidate objects	139
multi-frame	direction	“From the viewpoint of the last frame, measuring from room centers, where is \{objA\} relative to \{objB\}?”	front-left / front-right / back-left / back-right	401
multi-frame	distance	“Based on the given frames, measuring from room centers, which room is the closest to the robot’s position at the last frame?”	candidate rooms	77
multi-frame	size	“Based on the given frames, which room has the largest area?”	candidate rooms	16

Multi-frame inputs. General rotation questions supply four frames captured as the robot rotates in place, jointly covering the full $360^\circ$ surround. General navigation questions use observations captured during movement through a scene. Multi-frame direction questions use four diagonal relations (front-left, front-right, back-left, and back-right), whereas single-frame direction questions use axis-aligned relations.

Task-grounded VQA. BEHAVIOR-Spatial-TaskVQA uses task-conditioned templates to query objects and spatial relations associated with policy-evaluation instances. Frames are selected from these instances and filtered for visibility of the entities required by each question. Question counts are balanced across instruction variants within each task, yielding 1,082 manipulation questions and 602 navigation questions. The perception and action suites are aligned at the task level, while their valid evaluation samples may differ because of suite-specific visibility requirements.

Navigation context. Task-grounded navigation VQA receives curated per-environment context frames together with the current observation. Navigation policies receive the same ordered context frames as variable-length visual input (§ 4.1). Question answers are computed from simulator state. Appendix § 12 describes context traversal construction and illustrates the resulting frame sequences.

Point-Grounding Annotation and Validation

Evaluation frame selection. Evaluation frame counts are balanced across instruction variants within each task. VQA and point grounding apply distinct visibility criteria to the same source frames because they require different visual evidence. A VQA question such as “Where is the knife relative to the frying pan?” requires the knife and pan to be visible. A grounding instruction such as “Mark where the knife should go, right of the pan” additionally requires the corresponding placement region to be visible. If that region lies entirely outside the image, there is no valid image-space answer. Grounding evaluation therefore retains only frames with a visible answer region, and its eligible frame set may differ from that of VQA.

Evaluator validation. We use IR predictions and uniformly sampled image points as complementary checks of the grounding evaluator against its geometric acceptance regions. Appendix § 15.§ 2 presents the grounding query template.

Task Suite Overview

The benchmark task suite comprises 14 manipulation tasks and 10 navigation tasks. Figure S1 and Figure S2 illustrate the manipulation task settings and navigation scene layouts, respectively. Table S2 lists all tasks and their instruction variants.

Task	Category	Example instruction	Instruction variants	$n$
Manipulation
reach_food_item	size	“Reach the largest food item on the chopping board.”	{largest, 2nd largest, smallest, 2nd smallest}	32
reach_item	size	“Reach the smallest food item on the countertop.”	{largest, 2nd largest, smallest, 2nd smallest}	32
reach_plate	size	“Reach the largest plate from the countertop.”	{largest, 2nd largest, smallest, 2nd smallest}	31
reach_toys	size	“Reach the second largest toy on the desk.”	{largest, 2nd largest, 3rd largest, smallest, 2nd smallest}	40
reach_dice	distance	“Reach the dice closest to the board game on the bed.”	{closest, farthest} $\times$ {board game, wall}	32
reach_paperback.book	distance	“Reach the paperback book farthest from the folder on the desk.”	{closest, farthest} $\times$ {folder, pencil case}	22
reach_pizza.plate	distance	“Reach the pizza plate closest to the fridge on the breakfast table.”	{closest, farthest} $\times$ {fridge, black chair, white chair}	40
reach_soda.can	distance	“Reach the can of soda farthest from the sofa on the floor.”	{closest, farthest} $\times$ {sofa, coffee table, bottom cabinet}	42
reach_storage.box	distance	“Reach the storage box closest to the coffee table on the floor.”	{closest, farthest} $\times$ {coffee table, bottom cabinet}	32
reach_tool	ordinal	“Reach the tool on the second leftmost of the countertop.”	{leftmost, 2nd leftmost, rightmost, 2nd rightmost}	32
reach.hardback	locate	“Reach the hardback book at the 2x2 grid of the bookcase.”	grid cells {1x1, 1x2, 2x1, 2x2, 3x1, 3x2}	48
place_carving.knife	direction	“Place the carving knife on the countertop and to the right of the cutting board.”	{left, right} $\times$ {cutting board, microwave, burner, frying pan}	24
place_saucer	direction	“Place the saucer on the countertop and to the left of the electric kettle.”	{left, right} $\times$ {coffee maker, kettle, coffee bottle}	train-only
place_tupperware	locate	“Place the tupperware in the layer 5 of the fridge.”	layers {4, 5, 6}	22
place_bowl	locate	“Place the bowl in the layer 3 of the bottom cabinet.”	layers {1, 2, 3, 4, 5}	40
Navigation (two-floor house)
move_to_garage	navigation	“Move to the garage.”	—	40
move_to_fridge	navigation	“Move to the fridge.”	—	40
move_to_table.in.kitchen	navigation	“Move to the table in the kitchen.”	—	40
Navigation (single-floor house)
move_to_bookcase	navigation	“Move to the bookcase.”	—	40
move_to_bathroom	navigation	“Move to the bathroom.”	—	40
move_to_table.near.sofa	navigation	“Move to the table near the sofa.”	—	41
Navigation (restaurant diner)
move_to_corner.booth	navigation	“Move to the corner booth.”	—	40
move_to_bar	navigation	“Move to the bar.”	—	40
move_to_kitchen_sink	navigation	“Move to the kitchen sink.”	—	40
move_to_oven	navigation	“Move to the oven.”	—	40

Manipulation task curation. We select manipulation subtasks in which a spatial relation determines the intended target among multiple plausible candidates. The tasks cover direction, including locate and ordinal relations, as well as distance and size. Demonstrations are trimmed to the corresponding subtask windows.

Instruction variants. For each task, we curate instruction variants that change the spatial relation or reference object and assign the corresponding target object or goal region to each variant. These variants support the held-out and prompt-probe evaluations described in § 4.3.

Navigation tasks and context traversals. The navigation suite spans five scenes and requires navigating to an instructed furniture target or region. For each environment, we manually curate a traversal covering relevant navigable regions and doorways and collect a dense, ordered sequence of context frames along the route. Figure S3 and Figure S4 show the complete context-frame sequences for all five scenes in input order. Question generation using these frames is described in Appendix § 11.§ 2.

Additional Model Training and Evaluation Details

Policy Adaptation and Initialization

Baseline. The Qwen3.5-4B backbone (4.54B parameters) is frozen; the action expert and the action, state, and time projections (0.30B) are trained from scratch.

LoRA adaptation. We add LoRA adapters (rank 16, $\alpha= 16$, no dropout, $B$ initialized to zero) to every linear layer of the language model and vision encoder. This adds 39M trainable parameters (0.9% of the backbone), trained jointly with the baseline’s trainable components.

DROID initialization. We initialize from step 160k of DROID pretraining (frozen VLM, joint-position actions padded to 23 dimensions, horizon 32, batch size 256, peak LR $3 \times10^{-4}$), so the transferred weights reside in the action expert and projections. Finetuning is otherwise identical to the baseline.

Training Setup

Table S3 lists the training hyperparameters. All runs use AdamW ($\beta= (0.9, 0.95)$), no weight decay, gradient clipping at 1.0, bf16 without EMA, 4k steps of linear warmup, an action horizon of 32, and four GPUs. Manipulation tasks are sampled uniformly from the 14 benchmark tasks plus one training-only saucer task; navigation samples uniformly over its 10 tasks.

Policy	Batch	Peak LR	LR schedule	Checkpoint
Manipulation
$\pi_{0.5}$-Qwen (frozen VLM)	256	$4 \times10^{-4}$	cosine	100k
+ LoRA-VLM	128	$2.8 \times10^{-4}$	cosine	110k
+ DROID pretrain init	256	$4 \times10^{-4}$	cosine	90k
+ IR variants (all)	256	$4 \times10^{-4}$	cosine	100k
Navigation
$\pi_{0.5}$-Qwen (frozen VLM)	512	$4.2 \times10^{-4}$	cosine	60k

Oracle Intermediate Representation Construction

Target localization. The oracle IR policy variants receive simulator-derived grounding images during both training and evaluation. For reaching tasks, the reference tracks the target object’s current 3D bounding box. For placement tasks, we fix the box at the object’s destination in the final demonstration frame and reproject this fixed world-space reference through the current head camera. Thus, the placement cue marks the goal location throughout the episode. We project the box corners using the camera intrinsics and extrinsics, clip the resulting 2D bounding box to the image boundaries, and take its integer midpoint. We scale this coordinate to the 224 $\times$ 224 policy image and round to the nearest pixel. The reference is geometric and does not use an occlusion mask; an occluded target can therefore retain a cue.

Dot IR. We resize the head-camera RGB image to $224 \times224$ before drawing a filled red marker (RGB $(255, 51, 51)$) of radius 4 pixels with a 1-pixel white outline at the target coordinate. When no valid target projection is available, the image contains no marker. The annotated image is supplied as an additional input alongside the original head and two wrist-camera images.

Gaussian attention-map IR. For a target coordinate $(c_x, c_y)$, we construct

on the same $224 \times224$ grid. The Gaussian has unit peak and is not normalized to sum to one. We quantize its values as $q(x, y) = \operatorname{round}(255H(x, y))$ and map them to RGB using a 256-entry analytic jet lookup table sampled over $[0, 0.875]$. This produces a dark-blue background (RGB $(0, 0, 128)$) and a red peak (RGB $(255, 0, 0)$). An invalid projection produces a uniform dark-blue image. The map is a standalone image without an RGB underlay and occupies the same additional image-input slot as dot IR. See Figure S5 for the difference.

Temporal conditioning. The current-view cue is recomputed for each policy observation using the current camera pose. Dot and Gaussian rendering use the same respective parameters during training and evaluation. The first-frame anchor variant additionally retains the initial dot-marked head image and supplies this fixed image throughout the episode as a fifth input.

Policy Evaluation Instance Construction

Navigation test instances. We construct held-out navigation instances by varying the robot’s initial position and orientation within each environment. Candidate starting states undergo simulator-based spawn validation, including checks for excessive displacement, vertical motion, and tilt during physics settling. Figure S6 provides spatial context for these examples by showing evaluation starting poses, destinations, and surrounding furniture. For the six house tasks, the maps additionally show cached training-trajectory coverage.

Evaluation Scoring Details

VQA Aggregation. We report VQA accuracy within each spatial category and task. Task-grounded manipulation VQA uses an unweighted mean of per-task accuracies; navigation VQA is reported per task and pooled over questions.

Reach IR Scoring. Predicted target points are evaluated against annotated target regions derived from instance segmentation. Predictions assigned to an accepted target region are labeled correct, those assigned to a non-target object are labeled wrong, and unassigned or unparseable predictions are labeled none. Tolerant scoring additionally accepts points within approximately 2% of the image diagonal of the target region (approximately 20 pixels for a 720 $\times$ 720 image), while preserving all strictly correct predictions. Tolerance expansion applies only to reach tasks. See Figure S7 for illustration.

Placement IR Scoring. For directional placement, the predicted point defines a camera ray whose intersection with the countertop determines the placement location. Its projection onto the robot-right axis determines whether it lies on the instructed side of the reference. Locations within 0.17 m of the reference center receive an on-ref outcome and no correct credit; off-surface predictions are incorrect. For container placement, the ray is intersected with the container volume, and its entry height is compared with the instructed compartment’s height band ($\pm0.05$ m). The reported manipulation IR average weights each of the 14 evaluated tasks equally. See Figure S7 for illustration.

Manipulation Outcome Rules. Table S4 specifies task-family completion criteria, soft-credit conditions, and exceptions. For non-committed reach episodes, soft credit requires the end-effector’s minimum distance to the instructed target over the episode to be smaller than its minimum distance to every distractor. For directional placement, the still-held object must lie on the instructed side at episode end. For container placement, the object’s peak height while within the container footprint must fall within the permitted compartment band; passing through the band before overshooting does not qualify. Floor-drop failures and released on-ref outcomes receive no additional soft credit.

The soda-can task additionally accepts base proximity within 0.70 m under a nearest-candidate rule. Commitments to non-instructed candidates or locations count as wrong-target outcomes. Released on-ref outcomes count as failures. Soft credit captures observable target-consistent progress; it does not directly establish a policy’s internal grounding decision.

VQA and Point-Grounding Evaluation Prompts

This section documents the prompt components for BEHAVIOR-Spatial-VQA and BEHAVIOR-Spatial-IR, including spatial conventions, multi-frame context instructions, and output formats. Text inside boxes reproduces the evaluation prompt wording, with line wrapping adjusted for presentation.

VQA Evaluation Prompts

VQA prompts concatenate an observation-setting preamble, an answer-format instruction, applicable spatial rules, and the question with its answer choices. Table S1 lists the general VQA question templates.

Single-frame preamble. Single-frame questions begin with the following instruction.

Answer the question based on the given image(s).

Answer format. Multiple-choice questions receive the following response-format instruction.

Answer with the option’s letter and text from the given choices directly.
Example: if you think the correct answer is ’C.front’ from ’A.above B.under C. front
D.behind.’, your response should only be ’C.front’.

Direction rules. Non-locate direction questions receive the following spatial conventions, except for task-grounded navigation, which uses the robot-relative instructions below.

Spatial Understanding Rules:
- Use the camera’s perspective as the reference frame for all spatial relationships
(unless explicitly stated otherwise).
- Front/Back: The object that appears farther from the camera (deeper into the scene)
is considered "in front of" the object that appears closer to the camera. The object
closer to the camera is at the "back" of the farther object.
- Left/Right: Defined from the camera’s viewpoint. An object located on the left side
of the image relative to another object is considered "left of" that object, and
similarly for right.
- Above/Below: Determined by relative vertical position in the scene.

Locate rules. Locate questions receive conventions for interpreting container regions.

Locate Question Rules:
- Grid Positions (Row x Column): Rows are counted from bottom to top. Columns are
counted from left to right. For example, "1x1" is the bottom-left grid, "2x3" is the
second row from the bottom and third column from the left.
- Layers: Counted from bottom to top of the container. Layer 1 is the bottommost level
(closest to the floor/base), and higher layers are numbered sequentially upward.
- Levels: Counted from bottom to top, described by their relative height within the
container. (e.g., "low level," "middle level," "high level," or specific numerical
levels)
- Please ONLY consider the internal space and organizational structure of the specific
container mentioned in the question.

Camera-distance rules. Distance questions referencing the camera receive the following clarification.

Camera Distance Question Rules:
- When questions refer to "the camera", note that the robot hand visible in the image is
NOT the camera. The camera is the viewpoint from which the image was captured, which is
separate from any robot hand or manipulator visible in the scene.
- Distance is defined along the view trace from the camera to the 3Dcenter of the
object.

General multi-frame questions. For general multi-frame questions, the single-frame preamble is replaced by the following instruction.

These are consecutive observations from the head camera of a robot. Answer the question
based on the given images.

Rotation questions additionally receive:

The robot is rotating in place. The images show different viewpoints as the robot
turns.

Navigation questions additionally receive:

The robot is navigating through the environment. The images show the robot’s
observations as it moves.

Task-grounded navigation. The current task-grounded navigation prompt uses a separate preamble that distinguishes context frames from the current observation. It replaces the general multi-frame preamble and direction-rule block.

You are looking through the head camera of a mobile robot.
All images except the last one are a CONTEXT TOUR: earlier views recorded while moving
through this environment. Use them to learn where things are. They were NOT taken from
the robot’s current position.
The LAST image is the robot’s CURRENT camera view. All answers are relative to the
robot’s current position and heading in that last image.

Definitions (robot-centric, top-down):
- FRONT = the direction the camera is facing; BACK = directly behind the robot.
- LEFT / RIGHT = the robot’s left and right when facing forward.
- The target may be OUT OF VIEW (behind the robot, or behind a wall). It still has a
direction. Locate the target from the tour, locate the robot from the current view, and
answer with the quadrant the target lies in.

Question and choices. The question and answer choices are appended after the applicable instructions in the following format.

Question: {question}
Options: {options}

Point-Grounding Evaluation Prompts

For BEHAVIOR-Spatial-IR, the model receives the task instruction and an image, and predicts one target point. The prompt specifies integer coordinates normalized to a 0–1000 scale and requires a JSON response.

Point-grounding template.

The robot’s task is: {instruction}
{layer hint}
Predict the center of the target in the frame: {target description}.
Output the spatial coordinate of the center as x, y (integer coordinates normalized to a
0–1000 scale).
You must provide a valid prediction for the target --- do not return null or empty
values.
Predict exactly ONE point: x and y must each be a single integer, never a list. If
multiple objects are plausible, commit to the single most likely one.
Respond ONLY with valid JSON in the exact format shown below. Do not include any other
text.
Example: {"results": {"target": {"x": <int>, "y": <int>}}}

Target description. For object targets, the target description is:

the object that the robot must reach or manipulate to accomplish this task

For placement targets, it is:

the target location where the robot must place the object it is holding to accomplish
this task

Layer-counting hint. For placement instructions containing the word “layer”, the following hint is inserted immediately after the task instruction. Otherwise, the hint is omitted.

Layers are counted from bottom to top: for example, "layer 5" is the fifth layer from
the bottom of the fridge.

Additional Experimental Analyses

Extracted VLA Backbones on Spatial VQA

The extracted OpenHelix and $\pi_{0.5}$ backbones attain 9.6% and 0.9% single-frame accuracy under the tested VQA interface. Because invalid answers are scored as incorrect, these scores combine answer correctness with the ability to produce a valid option.

Qualitative Perception Examples

Figure S8 shows some examples for VQA and IR predictions on the same set of evaluated frames.

Policy Execution and Checkpoint Selection

Timing and action alignment. Each predicted chunk is indexed from the simulation step of its input observation, so a chunk that arrives late is entered at a latency-dependent offset. The controller keeps the committed chunk while this index is below ten and within the prediction horizon; otherwise, it switches to the newest chunk covering the current step. Because the threshold counts from the observation rather than from commitment, plan switches do not occur at a fixed interval.

Ensembling and fallback. Non-gripper dimensions, including the mobile base, are averaged over the entries of a three-entry commitment history whose horizons cover the current step. The weights are $w_r \propto\exp(0.01r)$, with $r = 0,\ldots,m-1$ ordered oldest to newest and $m \le3$; for $m = 3$, they are approximately $(0.330, 0.333, 0.337)$. Gripper commands instead come from the newest chunk covering the current step, which need not be the committed one. If no committed entry covers the current step, the controller uses the most recently committed chunk with its index clamped to the horizon. Before any chunk is committed, it repeats the last action.

Checkpoint selection. The main manipulation comparison uses checkpoints at nearby, rather than identical, training steps: 100k updates for the baseline and the three core IR variants, 110k for LoRA-VLM, and 90k for the DROID-initialized policy.

Per-Task Baseline Results

Table S5 provides the per-task held-out results supporting the baseline analysis in § 5. Strict success, soft success, and wrong-target rates distinguish completed interactions, target-consistent partial progress, and incorrect target commitments. Task statistics are included for context.

Task	$n$	Strict↑	Soft↑	Wrong↓	Cand.	Size (px)	Frames (k)	Len.
reach_food_item	32	53.1	56.2	34.4	4.0	52	319	318
reach_storage_box	32	46.9	68.8	25.0	2.0	85	145	1365
reach_soda_can	42	38.1	52.4	42.9	3.0	35	97	696
reach_plate	31	35.5	35.5	61.3	3.7	124	55	212
reach_hardback	48	31.2	37.5	37.5	4.3	66	237	421
reach_item	32	31.2	34.4	53.1	7.0	98	307	311
place_carving_knife	24	29.2	70.8	4.2	—	62	58	184
reach_paperback_book	22	27.3	54.5	18.2	2.0	80	193	602
reach_toys	40	22.5	30.0	37.5	5.8	70	126	441
reach_pizza_plate	40	20.0	50.0	25.0	2.0	94	288	1212
reach_dice	32	18.8	56.2	6.2	2.0	22	145	793
place_bowl_in_cabinet	40	15.0	15.0	77.5	—	110	202	450
place_tupperware_in_fridge	22	13.6	13.6	45.5	—	132	200	501
reach_tool	32	6.2	15.6	46.9	4.6	53	87	461

Code-as-Policy Agent Evaluation

This appendix supplements § 4.4 and § 6.0.2 with program-development details, execution settings, and per-task results. The agent interface and evaluation metrics are described in the main paper.

Program Development and Selection

Writer context. For each task, all writers receive an identical system prompt containing the API reference, restrictions on privileged information, development instructions, task-specific scoring notes, and spatial-relation conventions. Each request includes the last three text exchanges. The initial request additionally includes head- and wrist-camera images from an example scene. Later requests include the previous episode’s verdict, printed output, API call log, final camera views, and a history of development outcomes. Each response contains a short plan and a complete Python module defining run(api).

Development episodes. Each writer develops its program over 10 training episodes per task. Feedback includes the strict outcome and whether the hand ended nearer the correct or a wrong candidate. This simulator-derived feedback is available during development, not during test execution.

Before simulation, candidate programs are screened using a mock API constructed from a recorded scene. Each development step allows up to three writing attempts. Programs are rejected if they raise an error, never command a motion, issue 100 consecutive perception calls without a motion, or make no API call for 20 s; programs still running at the 60 s or 500-call cap are accepted. If all three attempts fail, the previous version (if it passed the check) is run instead. If the previous version also failed, the step is skipped and the writer is told no episode was run. At the first step, a failing program is still run. These checks are applied during development only.

Validation and freezing. After the 10 development episodes, the final program is run once on a held-aside episode from the training distribution. This episode is distinct from the development episodes, test episodes, and example scene. Validation requires strict success; target-consistent partial progress alone does not pass.

If the final program passes validation, it is frozen. Otherwise, the most recent version that achieved strict success during development is frozen. If no development version succeeded, the final program is frozen despite failing validation.

Writer Implementation

Qwen3.5-4B and Qwen3.5-9B use Qwen/Qwen3.5-4B and Qwen/Qwen3.5-9B, served with vLLM 0.19.1 in bf16. Both use thinking mode with the qwen3 reasoning parser, temperature 1.0, top-p 0.95, top-k 20, min-p 0, and presence penalty 1.5.

Gemini 3 uses gemini-3.1-pro-preview through Vertex AI with high thinking level. GPT-6 uses gpt-6-astra through the OpenAI Responses API with high reasoning effort. Fable 5.1 uses claude-fable-5-1 through the Anthropic Messages API with adaptive thinking, high effort. Other generation settings use the respective API defaults.

Agent API

Programs define run(api) and interact with the robot only through the api object below. Positions are in metres in the robot base frame ($x$ forward, $y$ left, $z$ up, origin on the floor under the robot); the frame moves with the robot, so positions must be re-expressed or re-observed after base motion. Cameras are head, left_wrist, and right_wrist. Perception and state calls do not advance simulated time; motion, gripper, and wait calls block until completion. Programs may import numpy, math, re, cv2, PIL, scipy, and the Python standard library.

Observation and state.

• instruction: the natural-language instruction for the episode.

• get_state(): end-effector poses of both arms, gripper opening, grasp state, joint positions, and the current step.

• get_image(camera): the camera’s RGB image ($H \times W \times3$).

• get_depth(camera): the camera’s depth image (metres along the optical axis, pixel-aligned with the RGB image).

Geometry.

• pixel_to_point(u, v, camera): the 3-D base-frame point seen at pixel $(u, v)$, from the camera’s depth (median of a $5 \times5$ patch).

• point_to_pixel(p, camera): the pixel at which base-frame point $p$ appears, or none if behind the camera.

• pixel_to_ray(u, v, camera): the ray (origin, unit direction) through pixel $(u, v)$.

Perception tools.

• point_at(text, camera): image points for every instance of text in the current image, from an open-vocabulary pointing model (Molmo2-8B; prompt “Point to every {text} in the image.”, greedy decoding). text may be any description, including the full instruction.

• ask_vlm(question, cameras): a free-form answer from a VLM (GPT-5, low reasoning effort) about the current images of the given cameras.

Motion and gripper.

• approach(p, standoff, arm): turns the base to face point $p$ and drives until $p$ is standoff metres ahead (default 0.55), offset toward the reaching arm; returns $p$ re-expressed in the new base frame.

• move_base(dx, dy, dyaw): relative base motion; stops short if blocked.

• move_ee_to(p, arm, speed, use_torso): straight-line end-effector motion to point $p$ (orientation not controlled); use_torso lets the torso lean for extra reach.

• move_ee_by(d, arm, speed, use_torso): relative end-effector motion.

• open_gripper(arm), close_gripper(arm): gripper control.

• wait(seconds): hold still.

Every motion call returns its success and, on failure, a reason (stalled: contact or reach limit; blocked: base obstructed; pushing: excessive force, after which the arm backs off; timeout). With arm="auto", the arm already holding an object is used; otherwise the left arm for targets with $y \ge0$ and the right arm otherwise.

Utilities.

• log(msg, **data): write to the episode log. Printed output is returned to the writer during development.

Execution and Evaluation

Closed-loop execution. Frozen programs are executed without writer revisions on all test episodes of their task. They can repeatedly observe, query perception tools, inspect motion results, branch, and retry actions. Thus, freezing the program does not prevent closed-loop behavior.

The simulator is paused during computation, perception calls, and state queries. Only motion and gripper commands and wait advance simulated time; these calls block until completion. Evaluation imposes no explicit tool-call limit. The perception tools retry network errors up to four times. If a program issues no robot command for 900 seconds of wall-clock time, it is stopped and the robot holds still until the episode ends.

Simulation budgets and termination. The nominal simulation budgets match those used for the VLA policies. Writers may declare completion, but termination and success are determined by the simulator under the same step budgets as the VLA policies.

Metrics. We evaluate all writers on the same held-out set across 14 manipulation tasks, using the task-specific strict-success, soft-success, and wrong-target criteria defined in § 5.1. Table S6 reports per-task soft and strict success.

Task-Level Correlation Plots

Each point in Figure S9 represents one manipulation task for a policy variant. The perception scores are from Qwen3.5-4B, and policy outcomes are soft held-out success rates. Lines show descriptive linear fits; the main Table 3 reports Spearman rank correlations, a distinct statistic.

Ground-truth vs. VLM-Predicted IR at Rollout

To test whether explicit target conditioning remains beneficial with predicted grounding, we replace the simulator-derived dot at inference with a VLM-predicted point while keeping the oracle-trained dot-IR policy fixed. The grounding VLM receives the current head-camera image at $512 \times512$ resolution and the task instruction, and predicts the target object location for reaching or the destination for placement. The predicted point is rescaled to the $224 \times224$ policy image and rendered using the same dot interface. A failed prediction produces an unmarked image; simulator coordinates are not used as a fallback. We evaluate Qwen3.5-4B, Qwen3.5-9B, and Molmo2-8B as grounding VLMs, with the same Qwen3.5-4B policy in all three cases.

On the 469 held-out episodes, predicted dots improve soft success from 41.6% to 47.1–48.8% and strict success from 27.9% to 33.5–34.5% (Table S7). These improvements recover approximately 29–39% of the gain from oracle dots, which achieve 60.3% soft and 44.8% strict success. Thus, explicit target conditioning remains beneficial with VLM-predicted grounding, although a substantial gap to oracle conditioning remains.

Rollout with VLM-predicted dot IR. Success (%) on the same held-out set. All rows use the Qwen3.5-4B policy backbone; predicted rows replace the ground-truth dot IR at inference with a dot predicted by the listed grounding VLM.

IR source	Soft↑	Strict↑
Baseline	41.6	27.9
Predicted dot
Qwen3.5-4B	48.8	34.1
Qwen3.5-9B	47.3	34.5
Molmo2-8B	47.1	33.5
Oracle dot	60.3	44.8

The three grounding VLMs yield similar rollout success rates: their results span 1.7 percentage points on soft success and 1.0 point on strict success, with no model leading on both metrics. These end-to-end results complement the IR grounding accuracy evaluated in the main paper: rollout success measures whether the predicted cues support successful control, rather than grounding accuracy alone.

Training-Distribution Case Study

We study a separate reach_paperback_book training setup in which the referred book is associated with a fixed side in the demonstrations. To vary the approach geometry, we prefix each demonstration with motion from an initial robot position on the perpendicular bisector of the segment connecting the two candidate books to the original starting pose (Figure S10). The original interaction is retained.

We compare policies trained with and without this augmentation on the same 69 prompt-probe instances (Table S8). Augmentation reduces wrong-target commitments from 79.7% to 43.5% and increases soft success from 11.6% to 36.2%, while strict success remains 10.1%. These results show that changing the demonstration distribution can improve target selection and partial progress without increasing task completion.

Training data	Soft↑	Strict↑	Wrong↓
Original	11.6	10.1	79.7
+ prefix augmentation	36.2	10.1	43.5

The main 14-task evaluation uses the augmented demonstrations for reach_paperback_book. Here, we isolate this task to compare training with and without augmentation on a shared prompt-probe set. The augmentation changes motion trajectories and data exposure along with the spatial regularities of the demonstrations, so the observed effect cannot be attributed solely to reduced side bias.

Data Scaling

We study how demonstration volume affects the dot-IR policy by retraining it on nested 30% and 50% subsets of BEHAVIOR-Spatial-Action and comparing against the full-data model. Subsets are sampled per task to preserve the task mixture. All models are trained for the same number of updates and evaluated under the same protocol. Table 4 reports aggregate results, and Table S9 and Table S10 provide per-task breakdowns.

Task	Base	Dot IR 100%	Dot IR 50%	Dot IR 30%
reach_food_item	53.1/56.2	65.6/71.9	65.6/68.8	59.4/68.8
reach_plate	35.5/35.5	71.0/74.2	48.4/64.5	51.6/67.7
reach_soda_can	38.1/52.4	71.4/78.6	69.0/81.0	73.8/81.0
reach_storage_box	46.9/68.8	68.8/78.1	65.6/78.1	65.6/78.1
reach_item	31.2/34.4	50.0/50.0	50.0/50.0	43.8/43.8
reach_paperback_book	27.3/54.5	54.5/86.4	27.3/68.2	36.4/59.1
reach_pizza_plate	20.0/50.0	32.5/65.0	17.5/52.5	20.0/62.5
reach_tool	6.2/15.6	37.5/46.9	15.6/34.4	15.6/37.5
reach_toys	22.5/30.0	42.5/60.0	25.0/37.5	25.0/55.0
reach_hardback	31.2/37.5	29.2/43.8	14.6/35.4	16.7/43.8
reach_dice	18.8/56.2	37.5/68.8	15.6/59.4	25.0/62.5
place_carving_knife	29.2/70.8	33.3/100.0	25.0/100.0	12.5/79.2
place_tupperware	13.6/13.6	31.8/31.8	22.7/27.3	18.2/22.7
place_bowl	15.0/15.0	10.0/12.5	12.5/17.5	17.5/25.0
Pooled	27.9/41.6	44.8/60.3	33.7/53.7	34.5/56.1

Task	Base	Dot IR 100%	Dot IR 50%	Dot IR 30%
reach_food_item	16.0/16.0	40.0/40.0	28.0/30.0	22.0/24.0
reach_soda_can	12.0/16.0	62.0/62.0	76.0/76.0	66.0/68.0
reach_storage_box	4.0/4.0	52.0/54.0	26.0/28.0	30.0/34.0
reach_plate	6.0/6.0	20.0/24.0	8.0/12.0	26.0/28.0
reach_tool	8.0/8.0	28.0/32.0	22.0/36.0	26.0/32.0
reach_paperback_book	12.0/40.0	32.0/66.0	30.0/46.0	22.0/46.0
reach_dice	18.0/28.0	26.0/66.0	14.0/52.0	18.0/66.0
reach_hardback	24.0/24.0	20.0/36.0	4.0/18.0	4.0/26.0
reach_item	4.0/4.0	14.0/14.0	8.0/10.0	12.0/12.0
reach_pizza_plate	10.0/26.0	2.0/36.0	0.0/20.0	4.0/16.0
reach_toys	0.0/0.0	6.0/16.0	2.0/6.0	2.0/12.0
place_carving_knife	8.0/26.0	20.0/56.0	22.0/58.0	12.0/50.0
place_tupperware	10.0/20.0	22.0/24.0	14.0/16.0	18.0/22.0
place_bowl	8.0/8.0	4.0/6.0	18.0/20.0	16.0/20.0
Pooled	10.0/16.1	24.9/38.0	19.4/30.6	19.9/32.6

The full-data model achieves the highest pooled soft and strict success on both evaluation sets. From 30% to 100% data, held-out soft success increases from 56.1% to 60.3% (+4.2 percentage points), while strict success increases from 34.5% to 44.8% (+10.3 points). Prompt-probe soft and strict success also improve, from 32.6% to 38.0% (+5.4 points) and from 19.9% to 24.9% (+5.0 points), respectively. The prompt-probe wrong-object commitment rate decreases from 46.6% to 43.9%. Thus, additional demonstrations improve both success and object selection in the aggregate; these results do not isolate the benefit to execution alone.

Scaling is not monotonic at the intermediate data fraction. At 50% data, held-out soft/strict success is 53.7%/33.7%, compared with 56.1%/34.5% at 30%; prompt-probe success is also slightly lower at 50% (30.6%/19.4%) than at 30% (32.6%/19.9%). Nevertheless, the 30% dot-IR policy exceeds the full-data frozen-VLM baseline by 14.5 and 6.6 points in held-out soft and strict success, and by 16.5 and 9.9 points on prompt-probe.

Per-task results reveal substantial variation. Held-out strict success on reach_tool increases from 15.6% at 30% data to 37.5% at 100%, and place_carving_knife improves from 12.5% to 33.3%. In contrast, held-out reach_storage_box changes little (65.6% to 68.8% strict), although its prompt-probe strict success rises from 30.0% to 52.0%. More data does not help every task: place_bowl performs best at reduced data fractions on both evaluation sets. These differences indicate that the benefit of additional demonstrations depends on both the task and the evaluation setting.

Per-Task Policy Comparisons

Table S9 and Table S10 provide per-task held-out and prompt-probe results for the frozen-VLM baseline and dot IR policies used in §F.9. The 100%, 50%, and 30% columns show the effect of training-data fraction for that configuration.

Navigation Results

Table S11 reports VQA accuracy and navigation success for all ten tasks, with the scene or region specified for each task.

Qualitative Policy and Navigation-VQA Examples

The qualitative navigation examples illustrate potential sources of difficulty beyond the aggregate results in § 6.1. Longer routes expose the policy to more obstacles, while narrow doorway crossings require precise base motion. Observed failures include collisions with scene objects and unsuccessful doorway traversal, suggesting that route geometry contributes to the variation in task success.

Tasks requiring travel opposite to the order of the context frames (fridge, kitchen sink, and oven) also have relatively low success. This pattern is consistent with reliance on the demonstrated traversal direction, although differences in route geometry prevent us from isolating the effect of context order. A related probe in the two-floor house found success in 19/20 episodes starting in the trained region and 0/20 episodes starting from novel kitchen locations. These results suggest limited generalization across starting locations, but do not establish whether the limitation arises from context interpretation, route familiarity, or motion execution.

Figure S11, Figure S12, and Figure S13 show selected manipulation-policy, navigation-policy, and navigation-VQA examples, respectively.