betaFeedback

WorksComputer Science

GeoMind: A Benchmark for Spatial Understanding in Robotic Manipulation

Abstract

Robotic manipulation has advanced rapidly in mapping visual observations and language instructions to actions. However, reliable manipulation in open-world environments requires robust spatial understanding: the ability to identify targets in complex layouts (spatial reasoning) and adapt to scenes that continuously evolve through interaction (spatial memory). Yet existing embodied benchmarks fall short in evaluating spatial understanding: they often introduce strong visual cues that bypass spatial reasoning, and focus on static scenes that offer little insight into an agent's capacity for spatial memory. To address this gap, we introduce GeoMind, a systematic benchmark for structured spatial understanding through tabletop manipulation. GeoMind comprises 54 evaluated task variants organized around spatial reasoning and spatial memory. It also incorporates a scalable pipeline that automates the generation and annotation of task instances with verified targets and executable interactions. Extensive evaluations across diverse embodied agents reveal their consistent weaknesses in spatial understanding. Our findings highlight that these agents struggle to track identities after spatial changes and convert relational or geometric constraints into executable actions. We release the GeoMind benchmark at https://github.com/Notfound-JH/GEOMIND-benchmark.git.

Introduction

Advances in embodied intelligence have enabled robots to translate visual observations and language instructions into manipulation actions [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. However, reliable manipulation often requires more than recognizing individual objects: robots must reason about spatial relationships to identify targets and maintain task-relevant spatial information as scenes change [15, 16, 17]. For example, a robot may be asked to pick up a block based on its position relative to other objects, or to pick up the same block after observing several visually similar blocks being rearranged. While the former requires reasoning about the current spatial configuration, the latter additionally requires tracking object identity across scene changes. Such tasks highlight the importance of spatial understanding for reliable manipulation.

Existing embodied benchmarks [18, 19, 20, 21, 22, 23, 24] have made substantial progress in evaluating language-guided manipulation and generalization. However, overall task success provides limited insight into the spatial capabilities underlying an agent’s behavior. Distinctive visual or semantic cues may allow agents to identify targets without reasoning about spatial relationships. Similarly, tasks involving scene changes may be solved from the current observation without tracking previous spatial states. Conversely, manipulation failures may arise from execution errors even when the intended spatial target has been correctly identified. These limitations motivate a diagnostic evaluation that makes spatial requirements explicit and examines agents’ spatial capabilities beyond overall task success.

To enable a more targeted assessment of spatial understanding, we focus on two key dimensions, spatial reasoning and spatial memory, drawing on the cognitive distinction between reasoning about space and maintaining spatial information over time [25]. Spatial reasoning concerns identifying target objects and inferring goal locations or regions from spatial relationships and structured layouts, while spatial memory concerns tracking object identities and updating spatial states as scenes change. These dimensions specify the spatial capabilities that each task is designed to assess.

We introduce GEOMIND, a benchmark for systematically evaluating spatial understanding through tabletop robotic manipulation. It comprises six task families: (1) target identification, (2) relational grounding, (3) constraint composition, and (4) geometric inference for spatial reasoning, and (5) scene updating and (6) identity tracking for spatial memory. Together, these families span 54 evaluated task variants: 20 Base, 7 Semantic, 7 Visual, and 20 Execution variants. Tasks are designed around structured object layouts and controlled scene changes, with explicit spatial rules for determining target objects or goal locations. Many base tasks feature visually identical or near-identical objects to reduce reliance on appearance cues. To enable reliable evaluation of these diverse spatial objectives, we develop an agent-assisted pipeline that translates spatial constraints into executable grounding code, computes ground-truth spatial targets, and verifies the resulting task instances through simulation.

We evaluate generalist and spatially enhanced visuomotor policies alongside modular planning methods on GEOMIND under a unified evaluation protocol. GEOMIND combines a base task suite (GEOMIND-BASE) with controlled semantic, visual, and execution augmentations (GEOMIND-AUG) to probe robustness. Beyond task success rates, we use diagnostic spatial metrics and comparisons with an oracle policy to examine spatial grounding and execution behavior. Our results show that current methods can solve some basic grounding tasks but struggle to reason about structured spatial relationships and track spatial states across scene changes. Performance is sensitive to semantic augmentations, while visual and execution perturbations have more model-dependent effects.

The main contributions of this work are threefold: (1) A systematic benchmark for structured spatial understanding. We introduce GEOMIND, which organizes spatial understanding into six task families and provides controlled base and augmented evaluations. (2) An agent-assisted spatial grounding and verification pipeline. We develop a pipeline that translates spatial constraints into executable grounding programs, derives ground-truth targets, and verifies task instances through simulation. (3) A comprehensive diagnostic evaluation. We evaluate generalist and spatially enhanced policies alongside modular planning methods using diagnostic spatial metrics and oracle comparisons, revealing limitations in spatial reasoning, memory, and robustness.

Related Work

General Embodied Manipulation Benchmarks. Prior embodied manipulation benchmarks have progressively advanced standardized evaluation of language-conditioned manipulation, long-horizon control, and generalization across diverse tasks and environments [18, 19, 20, 22, 24, 26, 27, 28, 29]. VIMA includes history-dependent rearrangement and visual constraint satisfaction [23], while GENMANIP uses task-oriented scene graphs for constrained scene construction [30]. These benchmarks already cover several spatial requirements considered here; aggregate task success alone, however, does not distinguish spatial goal grounding errors from execution failures. More specialized benchmarks investigate memory-dependent manipulation. RMBench [31] evaluates tasks with varying memory complexity, while RoboMME [32] examines temporal, spatial, object, and procedural memory. MEMOBench [33] further introduces process-level evaluations of memory storage, updating, and compression. Together, these benchmarks examine both memory demands and the processes underlying memory-dependent manipulation. GEOMIND complements them by systematically assessing spatial reasoning and spatial memory through capability-specific manipulation tasks defined by explicit spatial constraints and controlled scene changes. Table 1 compares selected task requirements and evaluation designs. A checkmark denotes an explicitly targeted task or evaluation subset with reported results; a cross denotes that no explicit evaluation was identified in the reviewed protocol, not that the capability is absent.

BenchmarkDesign Structure GenerationSpatial Memory Scene UpdatingSpatial Memory Identity TrackingSpatial Reasoning Target IdentificationSpatial Reasoning Relational GroundingSpatial Reasoning Constraint CompositionSpatial Reasoning Geometric InferenceEvaluation Diagnostic Metrics
VIMA [23]✓✓✓✓✓✓✗✗
GENMANIP [30]✓✗✗✗✓✗✗✗
RoboMME [32]✓✓✓✗✗✓✗✗
MEMOBench [33]✓✓✗✗✓✗✗✓
RoboSPA [34]✓✗✓✓✓✓✗✗
BEHAVIOR-Spatial [35]✗✗✗✓✓✓✗✓
VABench [36]✓✓✗✗✓✗✗✓
GEOMIND✓✓✓✓✓✓✓✓

Table 1. Comparison of spatial capabilities and benchmark design. ✓: explicitly evaluated; ✗: no explicit evaluation identified in the reviewed protocol.

Benchmarks for Spatial Understanding. Spatial understanding benchmarks span static image-based evaluations and interactive embodied settings [25, 37, 38, 39, 40, 41]. VSR [42] evaluates spatial relations in image–text pairs, while other benchmarks examine orientation, navigation, mental rotation, localization, and video recall [43, 44, 45, 46, 47, 48]. ESI-Bench [49] emphasizes active information acquisition for spatial reasoning, while SpaMME [50] probes spatial state updating and long-horizon belief maintenance through action-conditioned observation sequences. RoboSPA [34] evaluates spatial reference resolution, compositional spatial selection, and procedural-memory tasks, and reports object-normalized success and progress measures. BEHAVIOR-Spatial [35] evaluates spatial instruction following through question answering, target grounding, and robot execution, while VABench [36] combines visual demonstrations, active perception, metric control, and trajectory-level diagnostics. GEOMIND complements these benchmarks by organizing explicit geometric goal construction, current-state re-grounding, and historical identity binding within a shared, controllable tabletop protocol. Structure-preserving sampling, deterministic spatial targets, controlled scene changes, and matched task variants support targeted comparisons across these requirements; scene updating need not imply dependence on unobserved history.

Spatial Capabilities of Embodied Policies. Current embodied agents increasingly rely on vision-language models for language-guided manipulation [3, 4, 51, 52, 53], yet complex spatial instructions remain challenging. Recent approaches therefore incorporate more explicit spatial representations and grounding mechanisms. RoboSpatial [54] provides large-scale spatial annotations for training vision-language models, while InternVLA-M [16] combines spatial grounding pretraining with spatially guided action learning to connect where to act with how to act. Other approaches introduce 3D spatial representations to improve the alignment of visual, linguistic, and geometric information for action generation [15, 37, 55, 56]. While these methods aim to improve spatial capabilities, GEOMIND provides a benchmark for systematically examining their strengths and limitations.

GeoMind

We present the overall design of GeoMind by turning structured spatial understanding into a controllable benchmark construction problem. Sec. 3.1 introduces the GeoMind taxonomy, and Sec. 3.2 describes spatial structure modeling and the benchmark construction pipeline. Sec. 3.3 presents the task suites, showing how executable benchmark episodes are organized into GeoMind-BASE and GeoMind-AUG.

Taxonomy of Embodied Spatial Understanding Tasks

Rather than categorizing tasks solely by manipulation actions, we organize them according to the spatial capabilities required to determine task-relevant targets and goals. Following the dimensions introduced above, the taxonomy organizes structured spatial understanding around spatial reasoning and spatial memory. The former concerns how agents infer targets or goals from spatial relations and scene structure, while the latter concerns how agents maintain and update task-relevant spatial information as scenes change.

  • Target Identification focuses on static object selection from structured layouts. The agent resolves a spatial description to the correct target object in a fixed scene.

  • Relational Grounding requires the agent to realize an explicit spatial relation through action, such as placing one object beside, in front of, or inside another reference object.

  • Constraint Composition tests whether the agent can jointly satisfy multiple spatial constraints that define a target object or a valid goal region.

  • Geometric Inference requires the agent to infer implicit target locations or regions from scene geometry, such as alignment, symmetry, or pattern continuation across objects.

  • Scene Updating evaluates whether the agent can update task-relevant spatial relations and re-ground its target after objects have moved or been manipulated.

  • Identity Tracking requires the agent to maintain object identity over time by linking a previously identified object to its current state across scene changes, such as object rearrangement or position exchanges.

Figure 1 qualitatively illustrates the spatial reasoning and memory demands of the six task families and presents representative examples with initial scenes, robot trajectories, and ground-truth spatial targets. The high/low positions in Figure 1 indicate intended capability demands. These tasks remain close to common robotic behaviors, such as selecting, placing, and retrieving objects, while exposing the spatial reasoning and memory demands behind seemingly simple actions.

Figure 1. Overview of GEOMIND. GEOMIND organizes structured spatial understanding along two diagnostic dimensions: spatial memory and spatial reasoning. The six task families cover different combinations of these dimensions. Surrounding examples illustrate how GEOMIND couples spatial instructions with structured object layouts and scene changes. The instructions are illustrative; evaluated actions and success criteria are specified in Appendix § 11.

Spatial Structure Modeling and Benchmark Construction

GeoMind constructs benchmark episodes through spatial structure modeling and an agent-assisted, rule-verified pipeline. Spatial structures allow object configurations to vary while preserving task-relevant constraints. The pipeline combines task template generation, executable spatial grounding, and simulation-based verification to produce structured manipulation tasks with deterministic ground-truth targets (Fig. 3).

Spatial Structure Modeling. To evaluate spatial understanding across different scene configurations, GeoMind represents object layouts using spatial structures that are not tied to a fixed set of coordinates. For example, objects arranged in a line can preserve their left-to-right order while varying in position and spacing.

Visualization of the six fundamental structural layouts and their derived variants

Figure 2. Visualization of the six fundamental structural layouts and their derived variants.

Overview of the benchmark construction pipeline

Figure 3. Overview of the benchmark construction pipeline. The pipeline first proposes task descriptions and structure templates, then resolves spatial queries over instantiated scenes to derive ground-truth annotations, and finally validates candidate episodes through interaction-graph-based simulation verification.

A task such as selecting the second object from the left therefore follows the same spatial rule across different layouts. We formalize a spatial structure SS as a set of relational, topological, and geometric constraints over objects. Given SS, GEOMIND generates diverse scene configurations by sampling object poses XX from a structure-preserving distribution:

X∼P(X∣S,ϵ),X \sim P(X \mid S, \epsilon),

where ϵ\epsilon controls allowable geometric variations, such as positional jitter and variable spacing, without violating the constraints encoded in SS. The structure library S={Sline,Sgrid,…}S = \{S_{\mathrm{line}}, S_{\mathrm{grid}}, \ldots\} includes linear, grid, and other canonical layouts (Fig. 2). Sampled configurations must satisfy structural validity and collision-free workspace constraints. This representation supports controlled evaluation across variations in object positions and spacing. Formal definitions are provided in Appendix § 10.

LLM-Assisted Task and Structure Proposal. Given a task category and a candidate set of spatial structure types, the LLM proposes matching instructions and structure templates within a predefined schema. Each template specifies the structure type, object roles, layout constraints, sampling ranges, and optional scene-update rules. Candidate templates are screened to remove ambiguous, logically conflicting, or task-incompatible specifications, with human review during template selection. The retained templates form a fixed library for subsequent programmatic scene instantiation. The benchmark uses a fixed set of tasks selected through human review, with task instances generated programmatically from the corresponding templates.

Spatial Target Grounding. An LLM-based coding agent translates spatial rules into programs that deterministically resolve target objects or goal regions from simulator states. Directional predicates use the canonical front-camera frame, with object coordinates transformed from the robot base frame. Reasoning tasks ground targets in the current configuration. In memory tasks, instructions precede scripted object motion, and evaluated actions begin after it ends: Scene Updating re-grounds targets in the updated scene, while Identity Tracking preserves their initial identities. Each agent can store earlier observations or maintain an internal memory to identify the target after the objects move.

Closed-Loop Verification. Each candidate episode is executed in a physics simulator to check physical feasibility and target consistency. For tasks involving scene changes, objects move through scripted motion or simulated robot manipulation. Throughout execution, we record simulator object states and construct a dynamic interaction graph Gt=(V,Et)G_t = (V, E_t), where nodes represent objects and edges encode discretized pairwise directions and distances. The graph sequence is used to check whether scene transitions and resulting spatial states are consistent with the task specifications and ground-truth targets. Episodes failing state-validity or target-consistency checks are discarded.

Task Suites and Dataset

GEOMIND is organized into two complementary evaluation suites: GEOMIND-BASE and GEOMIND-AUG. Together, they provide 54 evaluated task variants for structured spatial understanding. A task variant is a specific test setting that pairs a spatial objective with a scene structure, governed by specific grounding rules and evaluation conditions. GEOMIND-BASE evaluates the six spatial task families under canonical tabletop settings, while GEOMIND-AUG introduces controlled perturbations to assess robustness under additional semantic, visual, and execution challenges.

Detailed information about GeoMIND task suites

Figure 4. Detailed information about GeoMIND task suites. (a) shows how GEOMIND-AUG is constructed from GEOMIND-BASE. (b) shows the distribution of evaluated variants across task families and augmentation types (20 Base, 7 Semantic, 7 Visual, and 20 Execution).

GEOMIND-BASE consists of 20 evaluated base variants with diverse language instructions and multiple spatial structure types. To reduce visual shortcuts, many GEOMIND-BASE variants contain multiple visually identical or near-identical objects. In these settings, target selection or placement depends on spatial cues such as relative position, order, geometric relations, or scene updates, rather than object appearance alone. GEOMIND-BASE serves as the core benchmark suite and evaluates the spatial reasoning and memory capabilities defined in our taxonomy. Fig. 4b presents the distribution of evaluated task variants across task families and augmentation types.

GEOMIND-AUG contains 34 augmented variants: 7 Semantic, 7 Visual, and 20 Execution variants. Semantic augmentation modifies language grounding, visual augmentation modifies visual conditions, and execution augmentation tightens action precision requirements. Together with the 20 Base variants, these form 54 evaluated variants. These counts refer to evaluated task variants rather than unique spatial objectives or evaluation episodes. Examples and task details appear in Fig. 4a and Appendix § 11.

Experiments

In this section, we systematically evaluate representative embodied agents on the GEOMIND benchmark under a controlled setting. We first describe the experimental setup and the selected embodied agents. Then, we present the evaluation results, analyzing their implications for spatial understanding.

Experimental Setup

Benchmarked Embodied Agents. We evaluate representative embodied agents across GEOMIND-BASE and GEOMIND-AUG, covering three paradigms: general policies, spatially enhanced policies, and modular agents. These methods differ in whether spatial grounding, planning, and execution are handled implicitly by an end-to-end policy or explicitly by separate modules. Specifically, we evaluate OpenVLA [3], π0\pi_0 [4], and π0.5\pi_{0.5} [5] as general policies; SpatialVLA [15], InternVLA-M1 [16], and Spatial Forcing [56] as spatially enhanced policies; and COPA [57] and MOKA [58] as modular agents. We additionally include an oracle baseline that uses simulator states and deterministic spatial grounding to compute ground-truth targets, which guide a scripted controller. The oracle provides a reference for assessing task feasibility under correct target information.

Evaluation Protocol. All experiments are conducted in the IsaacLab [59] simulator using a Franka Research 3 (FR3) robotic arm. All methods are evaluated on the same task instances, random seeds, episode horizons, and success criteria. The evaluated VLA policies use fixed checkpoints without GEOMIND-specific fine-tuning, and their outputs are adapted to a common action interface. Privileged scene information, including simulator-resolved object identities and ground-truth target coordinates, is used for evaluation and the oracle baseline but is not provided to the evaluated VLA policies or modular agents during inference. Each episode is limited to 1000 control steps and terminates early upon task success or workspace violation. We evaluate Nep=48N_{\mathrm{ep}} = 48 episodes per task variant, with evaluation seeds saved for reproducibility. The physics simulation runs at 480 Hz, and the policy control frequency is 24 Hz. Detailed observation and action interfaces, deployment configurations, and evaluation protocols are provided in Appendix § 9.

Evaluation Metrics. We evaluate task completion using Task Success Rate (SR), the proportion of episodes satisfying task-specific success conditions, reported overall, by task family, and by augmentation type. Robustness is measured by ΔAug=SRAug−SRref\Delta_{\mathrm{Aug}} = \mathrm{SR}_{\mathrm{Aug}} - \mathrm{SR}_{\mathrm{ref}} in percentage points; negative values indicate degradation. The reference is the matched base task for semantic and visual variants, and the corresponding non-execution variant for execution variants. Spatial Goal Error (SGE) measures final spatial deviation, while Target Tracking Error (TTE) measures average end-effector distance to the target:

SGEi=di(sifinal,sigoal),TTEi=1Ti∑t=1Ti∥pi,teef−pi,ttar∥2.\mathrm{SGE}_i = d_i\left(s_i^{\mathrm{final}}, s_i^{\mathrm{goal}}\right), \qquad\mathrm{TTE}_i = \frac{1}{T_i}\sum_{t=1}^{T_i}\left\lVert\mathbf{p}_{i,t}^{\mathrm{eef}} - \mathbf{p}_{i,t}^{\mathrm{tar}}\right\rVert_2.

Here, ii indexes episodes, did_i is the evaluator’s goal-distance function, TiT_i is the episode length, and peef\mathbf{p}^{\mathrm{eef}} and ptar\mathbf{p}^{\mathrm{tar}} denote end-effector and target positions. SGE is aggregated only over valid goal-distance values. TTE uses recorded target trajectories when available, otherwise the initial target position as a marked static-layout proxy. Lower values indicate smaller errors. Applicability and aggregation details are provided in Appendix § 8.

Main Results

We organize the results around five questions concerning overall performance, robustness, failure modes, spatial memory, and spatial reasoning.

Q1. How well do embodied agents perform in structured spatial understanding?

We begin with GeoMind-Base, which evaluates six spatial capabilities under standard task conditions without the additional perturbations introduced in GeoMind-Aug. Table 2 reports the success rates of different agents across the six task families. The best-performing learned policy, InternVLA-M1, achieves 26.4% success, followed by Spatial-Forcing at 24.0%, substantially below the oracle’s 99.3%. Spatially enhanced policies achieve the highest overall scores, although their improvements over general policies are uneven across models and task families. Modular agents achieve 4.0–16.9% success, indicating that explicit VLM-based grounding and scripted execution also struggle with structured spatial tasks.

Embodied AgentsTarget Iden.Relational GroundingConstraint Comp.Geometric InferenceScene UpdatingIdentity TrackingAVG
Oracle100.098.0100.0100.098.0100.099.3
General PoliciesOpenVLA37.50.06.25.041.737.520.7
General Policiesπ0\pi_04.20.00.025.06.320.911.3
General Policiesπ0.5\pi_{0.5}4.20.00.027.56.325.012.5
Spatially Enhanced PoliciesSpatialVLA12.50.00.017.545.88.318.3
Spatially Enhanced PoliciesInternVLA-M142.550.06.212.016.720.826.4
Spatially Enhanced PoliciesSpatial-Forcing12.50.050.05.033.337.524.0
Modular MethodsCOPA GPT-4o22.911.116.74.29.713.911.1
Modular MethodsCOPA Gemini-2.5-Flash10.42.86.31.76.90.84.6
Modular MethodsCOPA Claude-Sonnet-4.620.813.916.713.316.019.415.9
Modular MethodsMOKA GPT-4o18.812.518.81.78.313.910.1
Modular MethodsMOKA Gemini-2.5-Flash8.31.44.20.87.61.44.0
Modular MethodsMOKA Claude-Sonnet-4.618.813.916.713.318.620.816.9

Table 2. Success rates (%) of different paradigms of embodied agents on GeoMind-Base. Excluding the oracle, best and second-best overall results are highlighted with green shades, while best and second-best family-level results are highlighted with purple shades.

To better understand these performance gaps, we examine the contributions of target grounding and execution. The oracle achieves near-perfect success using ground-truth targets and scripted control, demonstrating the feasibility of the benchmark tasks under correct spatial grounding. More directly, in a paired diagnostic evaluation of π0.5\pi_{0.5}, explicitly specifying the target through its name and a visual cue improves success from 11.3% to 39.2%, while keeping the policy, task instances, action interface, and success criterion fixed. This result provides evidence that target identification contributes substantially to the observed failures, although execution and deployment limitations may also play a role.

Finding: Current embodied agents remain far from reliable on structured spatial manipulation tasks. The controlled target-cue intervention suggests that inaccurate spatial target grounding is an important contributor to these failures, beyond execution limitations alone.

Q2. How robust is structured spatial understanding under semantic, visual, and execution perturbations?

Table 3 shows that success in GEOMIND-BASE is fragile under augmented variants of the same spatial goals. The clearest drop comes from semantic augmentation: averaged over the six embodied policies, success decreases from about 22.6% on the corresponding Base tasks to about 14.2%, and every policy degrades. This indicates that many Base successes still rely on simple instruction grounding rather than robust spatial-language understanding. The effects of visual and execution augmentation are more uneven. OpenVLA falls to 12.5% under both settings, whereas π0\pi_0 remains unchanged under visual perturbation and decreases by 6.5 percentage points under execution. These results suggest that structured spatial understanding degrades under complex instruction semantics and increased visual noise.

PoliciesCorresponding BaseSemanticVisualExecution
OpenVLA25.016.6 (−8.4-8.4)12.5 (−12.5-12.5)12.5 (−12.5-12.5)
π0\pi_016.712.5 (−4.2-4.2)16.7 (+0.0+0.0)10.2 (−6.5-6.5)
π0.5\pi_{0.5}18.512.5 (−6.0-6.0)16.7 (−1.8-1.8)10.2 (−8.3-8.3)
SpatialVLA18.88.3 (−10.5-10.5)12.5 (−6.3-6.3)12.5 (−6.3-6.3)
InternVLA-M137.527.1 (−10.4-10.4)37.5 (+0.0+0.0)35.4 (−2.1-2.1)
Spatial-Forcing18.88.3 (−10.5-10.5)12.5 (−6.3-6.3)16.7 (−2.1-2.1)

Table 3. Success rates (%) on GEOMIND-AUG and the reported base references. Parentheses show changes in percentage points.

Finding: Spatial success in embodied agents is condition-dependent rather than goal-invariant: the same spatial objective often fails once its linguistic, visual, or execution support is perturbed.

Q3. Why do embodied agents fail on GEOMIND?

We examine representative failure cases in Fig. 5 to connect the quantitative trends with concrete trajectories. For embodied policies in Fig. 5a, the illustrated failures fall into three categories: spatial relation grounding errors, where policies bind the correct anchor but execute the wrong relation; spatial relation reasoning errors, where policies fail to infer structure-defined targets; and scene-updating errors, where policies fail to ground the requested target in the updated configuration. These examples suggest that some failures involve converting spatial observations into appropriate action targets, although the trajectories alone do not isolate grounding errors from control errors. For modular embodied agents in Fig. 5b, the illustrated failures include object localization errors that can propagate to planning and control. This suggests a different bottleneck: modular agents may expose explicit reasoning steps, but remain vulnerable when intermediate spatial grounding is inaccurate.

Finding: The illustrated failures suggest difficulties in translating spatial observations into correct action targets. These examples identify possible failure mechanisms, but do not establish their prevalence across the benchmark.

Q4. What makes spatial memory difficult for embodied agents?

Table 2 and Table 4 show that agents struggle with tasks involving scene changes. Scene Updating requires selecting a target from the updated configuration, whereas Identity Tracking requires linking an object identified before the change to its new position. An agent may therefore fail an identity-tracking task by selecting an object that matches the current spatial relation but is not the original target. This provides a possible explanation for failures beyond imprecise execution. However, aggregate SGE and TTE cannot distinguish these causes: both are affected by scene geometry, initial target distance, and rollout duration. We use them as auxiliary diagnostics rather than direct measures of identity preservation or evidence that one task family is intrinsically harder.

Capability DimensionsTask FamilyFailure Rate ↓SGE (m) ↓TTE (m) ↓
Spatial MemoryScene Updating68.89%0.1890.330
Spatial MemoryIdentity Tracking85.00%0.1920.486
Spatial ReasoningTarget Identification72.08%0.1160.811
Spatial ReasoningRelational Grounding98.33%0.1800.672
Spatial ReasoningConstraint Composition87.50%0.1420.627
Spatial ReasoningGeometric Inference86.67%0.1580.727

Table 4. Family-level failure rates and auxiliary spatial errors. Lower values are better.

Finding: Identity Tracking requires preserving object correspondence across scene changes; the current aggregate results do not establish how much failure is attributable to incorrect target selection versus execution.

Q5. What makes spatial reasoning difficult for embodied agents?

Table 4 reports the highest failure rate for relational grounding among the reasoning families in that diagnostic summary, at 98.33%, compared with 72.08% for target identification. Constraint composition and geometric inference are also difficult, with failure rates of 87.50% and 86.67%, respectively. This indicates that current agents do not only struggle with recognizing spatial targets; they also fail to convert explicit spatial relations into correct embodied actions. Table 2 shows that geometric inference has the lowest best-agent success rate, with the strongest method reaching only 27.5%, suggesting that implicit structure-based reasoning remains hard even for the best agents. Together, these results suggest that agents remain weak when they must jointly satisfy multiple spatial constraints or infer targets from implicit geometric structure.

Finding: Spatial reasoning is difficult not only because agents struggle to identify the correct spatial target, but because they fail to transform spatial constraints into executable manipulation behavior.

Visualization of typical failure cases in GeoMIND

Figure 5. (a) Embodied policies. (b) Modular agents. Visualization of typical failure cases in GeoMIND.

Conclusion

We present GeoMIND, a systematic benchmark for evaluating spatial understanding in embodied agents. By organizing tasks around spatial memory and spatial reasoning, GeoMIND enables a more fine-grained analysis than overall task success alone. We further emphasize how spatial structure underpins spatial understanding, and present an automated generation pipeline that allows scalable benchmark construction. Our experiments show that current embodied agents remain far from achieving consistently high success rates on GeoMIND, with large performance gaps and highly uneven abilities across task families. These findings suggest that spatial understanding remains a core limitation of embodied agents.

Limitations. GEOMIND is limited to simulated, single-arm tabletop manipulation, without real-robot validation. Its controlled layouts do not capture the full diversity of real-world environments. Heavy occlusion, viewpoint changes, sensor noise, and complex contact dynamics are not systematically evaluated, although incidental occlusion may occur during execution. Semantic and visual augmentations cover selected transformations rather than exhaustive robustness conditions. Our experiments evaluate frozen pretrained policies, and their performance may also depend on observation and action-interface compatibility. We do not report benchmark-specific fine-tuning results; learning spatial skills and generalizing to held-out structures or variants remain untested. The results therefore characterize frozen-policy performance in the evaluated setting and do not establish transfer to real robots.

Code and Data Availability

The benchmark code and data have not yet been publicly released. We plan to release the benchmark assets, evaluation code, configuration files, and reproduction instructions in the project repository: https://github.com/Notfound-JH/GEOMIND-benchmark.git.

Acknowledgments and Disclosure of Funding

Funding Statement. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Competing Interests. The authors declare no competing interests.

Metrics Formulation

Success Rate

We compute the overall and family success rates as

SRAll=1∣T∣Nep∑τ∈T∑i=1NepI[sτ,i=1],SRf=1∣Tf∣Nep∑τ∈Tf∑i=1NepI[sτ,i=1],\mathrm{SR}_{\mathrm{All}} = \frac{1}{|\mathcal{T}|N_{\mathrm{ep}}} \sum_{\tau\in\mathcal{T}} \sum_{i=1}^{N_{\mathrm{ep}}} \mathbb{I}[s_{\tau,i}=1], \qquad\mathrm{SR}_{f} = \frac{1}{|\mathcal{T}_{f}|N_{\mathrm{ep}}} \sum_{\tau\in\mathcal{T}_{f}} \sum_{i=1}^{N_{\mathrm{ep}}} \mathbb{I}[s_{\tau,i}=1],

where sτ,is_{\tau,i} indicates whether episode ii of task variant τ\tau succeeds, T\mathcal{T} denotes the set of evaluated task variants, and Tf\mathcal{T}_{f} denotes the subset in family ff. Each variant has Nep=48N_{\mathrm{ep}} = 48 episodes. With equal episode counts per variant, this aggregation weights task variants equally rather than weighting task families equally.

Spatial Goal Error

Spatial Goal Error (SGE) is an auxiliary diagnostic metric that measures the final distance between the achieved spatial state and the target spatial goal. It is designed to capture final spatial precision, rather than full task completion.

For the ii-th episode, we define

SGEi=di(sifinal,sigoal),(1)\mathrm{SGE}_{i} = d_{i}\left(s_{i}^{\mathrm{final}}, s_{i}^{\mathrm{goal}}\right), \tag*{(1)}

where sifinals_{i}^{\mathrm{final}} denotes the final achieved state and sigoals_{i}^{\mathrm{goal}} denotes the target spatial state, and did_{i} is the evaluator’s goal-distance function for episode ii. For tasks with an explicit target position, this reduces to the Euclidean distance

SGEi=∥pifinal−pigoal∥2.(2)\mathrm{SGE}_{i} = \left\|p_{i}^{\mathrm{final}} - p_{i}^{\mathrm{goal}}\right\|_{2}. \tag*{(2)}

In our implementation, SGE is obtained from the evaluator-provided goal-distance proxy, such as evaluation.goal_distance or distance_m. If an episode does not provide a valid goal-distance value, its SGE is treated as missing and excluded from SGE aggregation. Therefore, whenever SGE is reported, we also record the number of valid SGE episodes.

For a model mm and task family ff, we aggregate SGE over valid episodes:

SGE‾m,f=1∣Im,fSGE∣∑i∈Im,fSGESGEi,(3)\overline{\mathrm{SGE}}_{m,f} = \frac{1}{|\mathcal{I}_{m,f}^{\mathrm{SGE}}|} \sum_{i \in\mathcal{I}_{m,f}^{\mathrm{SGE}}} \mathrm{SGE}_{i}, \tag*{(3)}

where Im,fSGE\mathcal{I}_{m,f}^{\mathrm{SGE}} denotes the set of episodes with valid SGE values for model mm and family ff.

For a specified set M\mathcal{M} of evaluated models, we first compute the model-family average and then average over models:

SGE‾f=1∣M∣∑m∈MSGE‾m,f.(4)\overline{\mathrm{SGE}}_{f} = \frac{1}{|\mathcal{M}|} \sum_{m \in\mathcal{M}} \overline{\mathrm{SGE}}_{m,f}. \tag*{(4)}

These averages are defined only for nonempty valid-episode sets. A model-family pair with no valid values has an undefined SGE and must not be treated as zero; the cross-model formula applies when each model in the specified set has a defined value. This weighting avoids giving models with more valid episodes more weight.

Interpretation. Lower SGE indicates better final spatial precision. However, SGE does not measure whether the agent followed the correct trajectory, selected the correct object throughout the rollout, or satisfied all spatial relations. It is therefore used only as an auxiliary diagnostic, not as a replacement for official task success.

Target Tracking Error

Target Tracking Error (TTE) is an auxiliary trajectory-level diagnostic metric. While SGE measures final spatial error, TTE measures whether the end-effector remains close to the intended target throughout the rollout. It measures target proximity along the trajectory, but does not directly measure spatial memory or identity preservation.

For the ii-th episode with trajectory length TiT_i, we define

TTEi=1Ti∑t=1Ti∥pi,teef−pi,ttar∥2,(5)\mathrm{TTE}_i = \frac{1}{T_i}\sum_{t=1}^{T_i}\left\lVert\mathbf{p}^{\mathrm{eef}}_{i,t} - \mathbf{p}^{\mathrm{tar}}_{i,t}\right\rVert_2, \tag*{(5)}

where pi,teef\mathbf{p}^{\mathrm{eef}}_{i,t} is the end-effector position at timestep tt, and pi,ttar\mathbf{p}^{\mathrm{tar}}_{i,t} is the ground-truth target position at the same timestep.

When dynamic object poses are available in the trajectory trace, the target position is taken from the target object’s dynamic pose:

pi,ttar=pi,tobj.(6)\mathbf{p}^{\mathrm{tar}}_{i,t} = \mathbf{p}^{\mathrm{obj}}_{i,t}. \tag*{(6)}

When dynamic object poses are unavailable, we use the static target position from the sampled layout as a proxy:

pi,ttar=pi,0tar.(7)\mathbf{p}^{\mathrm{tar}}_{i,t} = \mathbf{p}^{\mathrm{tar}}_{i,0}. \tag*{(7)}

In this case, the metric is marked as a static-layout proxy.

For a model mm and task family ff, TTE is first computed at the episode level and then averaged over valid episodes:

TTE‾m,f=1∣Im,fTTE∣∑i∈Im,fTTETTEi,(8)\overline{\mathrm{TTE}}_{m,f} = \frac{1}{\left\lvert\mathcal{I}^{\mathrm{TTE}}_{m,f}\right\rvert}\sum_{i\in\mathcal{I}^{\mathrm{TTE}}_{m,f}}\mathrm{TTE}_i, \tag*{(8)}

where Im,fTTE\mathcal{I}^{\mathrm{TTE}}_{m,f} denotes the set of episodes with valid trajectory traces for model mm and family ff.

When reporting a family-level average across models, we compute

TTE‾f=1∣M∣∑m∈MTTE‾m,f.(9)\overline{\mathrm{TTE}}_f = \frac{1}{\lvert\mathcal{M}\rvert}\sum_{m\in\mathcal{M}}\overline{\mathrm{TTE}}_{m,f}. \tag*{(9)}

The same nonempty-set requirement applies to TTE: a model-family pair without a valid trajectory has an undefined value, and the cross-model formula requires a defined value for each model in the specified set. This model-wise aggregation prevents a run with more episodes from dominating the reported value. We avoid pooling all timesteps across episodes directly, because such pooling would give longer trajectories larger weight.

Interpretation. Lower TTE means that the end-effector stays closer to the intended target over the trajectory. Unlike SGE, which only evaluates the final state, TTE captures trajectory-level target following. However, TTE still does not directly verify successful grasping, placement, or spatial-relation satisfaction, so it is used as an auxiliary diagnostic rather than an official success metric.

Detailed Experimental Setup

This section provides the implementation details needed to reproduce the GEOMIND evaluation. We focus on the experimental variables that directly affect validity and reproducibility: simulator and robot setup, control frequency, policy inputs, action conversion, rollout protocol, baseline deployment, and information access. Unless otherwise stated, all methods are evaluated on the same task instances, random seeds, workspace constraints, episode horizon, and success criteria.

Simulation, Robot, and Control

All experiments are conducted in GEOMIND LAB, an IsaacLab-based benchmark package built on Isaac Sim. We use a single Franka Research 3 (FR3) arm equipped with the FR3 parallel-jaw gripper. The robot receives relative end-effector commands and is controlled by a differential inverse-kinematics controller. The gripper is controlled through finger joint-position commands and is exposed to policies as a binary open/close command.

The physics simulation runs at 480 Hz, while policies are queried at 24 Hz. Each policy action is therefore executed through the low-level controller over 20 physics substeps. An episode terminates when the task succeeds, the robot violates the workspace guard, or the maximum horizon is reached. Table 5 summarizes the main simulator and control settings.

ItemSetting
SimulatorIsaac Sim / IsaacLab
Benchmark packageGEOMIND LAB
RobotFranka Research 3 (FR3)
GripperFR3 parallel-jaw gripper
Low-level controllerDifferential inverse kinematics
Physics frequency480 Hz
Policy control frequency24 Hz
Episode horizon1000 control steps
TerminationSuccess / workspace violation / horizon
Workspace guard (x,y,z)(x, y, z)[0.15,0.75]×[−0.45,0.45]×[0.70,1.20][0.15, 0.75] \times[-0.45, 0.45] \times[0.70, 1.20] m

Table 5. Simulation and control settings used for all evaluations.

Observation and Action Interface

All end-to-end VLA policies are evaluated through a common observation and action interface. The policy input contains the language instruction, two RGB camera observations, and robot proprioception. The two cameras consist of a static table-view camera and a wrist-mounted camera, both rendered at 224×224224 \times224 resolution.

Importantly, privileged simulator information is not exposed to VLA policies. Object poses, object identities, segmentation masks, ground-truth task variables, oracle target coordinates, and task metadata are used only for evaluation. Table 6 summarizes the policy interface and the evaluation-only states.

TypeFieldDescription
Policy inputInstructionNatural-language task instruction
Policy inputTable-view RGBStatic external camera observation, 224×224224 \times224
Policy inputWrist RGBWrist-mounted camera observation, 224×224224 \times224
Policy inputJoint stateRobot joint positions and velocities
Policy inputEnd-effector / gripper stateCurrent end-effector pose and gripper state
Policy actionTranslationRelative Cartesian displacement
Policy actionRotationRelative rotation-vector command
Policy actionGripperBinary open/close command
Evaluation onlyObject posesUsed for success checking and diagnostics
Evaluation onlyObject identitiesUsed for ground-truth target resolution
Evaluation onlySegmentation masksNot exposed to VLA policies
Evaluation onlyTask metadataUsed only for evaluation and diagnostics
Evaluation onlyTarget regions / coordinatesNot exposed to VLA policies

Table 6. Observation and action interface. Privileged simulator states are excluded from VLA policy inputs.

The common action interface is a 7-DoF relative end-effector command,

at=[Δx,Δy,Δz,Δrx,Δry,Δrz,g],\mathbf{a}_t = [\Delta x, \Delta y, \Delta z, \Delta r_x, \Delta r_y, \Delta r_z, g],

where the first three dimensions represent Cartesian displacement, the next three dimensions represent a relative rotation-vector command, and the final dimension controls the gripper. For policies whose native output format differs from this interface, we convert the output to the same 7-DoF command before execution.

We use a unified normalized action protocol. Each policy output is first converted to a 7-DoF action vector and clipped to [−1,1][-1, 1]. The translational and rotational components are then scaled by a model-specific multiplier sms_m. In the following equations, aa denotes the converted, already clipped action vector. Slices use zero-based indexing with an exclusive upper bound: a0:3a_{0:3} contains the three translation components and a3:6a_{3:6} the three rotation components. The clip operator acts componentwise:

ItemSetting
GEOMIND-BASE20 variants, 48 episodes per variant, 960 episodes per method
GEOMIND-AUG34 augmented variants, 48 episodes per variant, 1632 episodes per method
Task instancesShared across methods
Random seedsShared across methods
Episode horizon1000 control steps
Policy control frequency24 Hz
Physics frequency480 Hz
Camera resolution224×224224 \times224
Primary metricSuccess rate
VLA adaptationNo GEOMIND-specific fine-tuning

Table 7. Evaluation protocol for GEOMIND-BASE and GEOMIND-AUG.

uxyz=clip⁡(sma0:3,−1,1),u_{xyz} = \operatorname{clip}(s_m a_{0:3}, -1, 1),
urot={0,if rotation is disabled for the model,clip⁡(2sma3:6,−1,1),otherwise.u_{rot} = \begin{cases} 0, & \text{if rotation is disabled for the model,}\\ \operatorname{clip}(2s_m a_{3:6}, -1, 1), & \text{otherwise.} \end{cases}

The controller converts these normalized components to physical increments using scales of 0.5 m for translation and 0.5 rad for rotation, yielding

Δp=0.5uxyz m,Δr=0.5urot rad.\Delta p = 0.5u_{xyz}\ \mathrm{m}, \qquad\Delta r = 0.5u_{rot}\ \mathrm{rad}.

The gripper command is thresholded at zero: positive values are mapped to open and negative values are mapped to close. For baselines without reliable gripper outputs, we keep the gripper fixed according to their default deployment protocol and report this setting explicitly.

Evaluation Protocol

We evaluate each method on the same task instances and random seeds. Each evaluated task variant contains 48 evaluation episodes. GEOMIND-BASE contains 20 variants, resulting in 960 episodes per method. GEOMIND-AUG contains 34 augmented variants (7 Semantic, 7 Visual, and 20 Execution), resulting in 1632 episodes per method, for 2592 episodes across both suites. Success rate is the primary metric and is computed over all episodes in the corresponding task suite or task family.

VLA Baseline Deployment

All VLA baselines are evaluated as frozen policies without GEOMIND-specific fine-tuning. Each model is deployed as a remote policy server and queried online during rollout. For each model, we use a fixed checkpoint and server configuration across all evaluated episodes.

The policy server receives only the allowed policy inputs described in Table 6. No VLA baseline receives privileged simulator object poses, segmentation masks, ground-truth task variables, oracle target coordinates, or task metadata through the policy interface.

Some VLA models return action chunks rather than a single action. In our deployment, the client executes only the first valid action from the returned chunk and then queries the policy again at the next control step. This keeps all policies closed-loop at the shared 24 Hz control frequency while preserving each model’s native action-generation interface.

Different VLA models use different native action heads and deployment conventions. We therefore report the model-specific action multiplier, checkpoint setting, rotation setting, and gripper setting in Table 8. These settings are fixed for a given model across the full evaluation. Simulations were conducted on a workstation equipped with an NVIDIA RTX 3090 GPU, while model inference was performed on an NVIDIA RTX 4090 (48 GB).

ModelFine-tuningCheckpointsms_mRotation / gripper
OpenVLANoFixed2.0Rotation enabled; binary gripper
π0\pi_0NoFixed0.5Rotation disabled; fixed gripper
π0.5\pi_{0.5}NoFixed0.5Rotation disabled; fixed gripper
SpatialVLANoFixed5.0Rotation enabled; binary gripper
InternVLA-M1NoFixed0.5Rotation enabled; binary gripper
Spatial-ForcingNoFixed0.2Rotation enabled; binary gripper

Table 8. Deployment settings of VLA baselines. sms_m denotes the normalized action gain used before controller-level command limiting. All models are evaluated with fixed checkpoints and without GEOMIND-specific fine-tuning.

Modular Baselines and Diagnostic Use

We additionally evaluate modular agents based on COPA [57] and MOKA [58]. These methods represent a different embodied-agent paradigm from end-to-end VLA policies: instead of directly predicting low-level actions from visual observations and language instructions, they decompose manipulation into explicit visual-language grounding, target selection, and scripted execution. We include them as modular baselines to compare different design paradigms for structured spatial manipulation, and also use their intermediate predictions to diagnose where failures arise.

Each modular agent receives the task instruction and RGB observations, and uses a VLM backbone to identify the task-relevant object or spatial target from the scene. We instantiate the modular agents with three VLM backbones: GPT-4o, Gemini-2.5-Flash, and Claude-Sonnet-4.6. The predicted target is then converted into Cartesian waypoints or a target region and executed with the same Franka differential-IK controller used in the VLA evaluation. For pick-and-place tasks, the scripted routine follows a top-down sequence of approach, grasp, lift, place, release, and retreat. For reaching tasks, the controller moves the end effector toward the predicted target region and evaluates success using the same task evaluator as other methods.

All modular agents are evaluated on the same task instances, random seeds, workspace constraints, episode horizon, and success criteria as the end-to-end policies. They do not receive privileged simulator information during inference: oracle target coordinates, simulator-resolved object identities, ground-truth task variables, segmentation masks, and task metadata are not exposed to the VLM or the planner. These privileged states are used only by the evaluator for success checking and diagnostic logging.

Because modular agents use a different inference and execution structure from VLA policies, their results should be interpreted as a comparison across embodied-agent paradigms rather than as a fully controlled comparison of identical policy interfaces. This distinction is important: end-to-end policies test whether a learned visuomotor model can directly map observations and language to actions, whereas modular agents test whether explicit VLM-based grounding and scripted execution can solve the same spatial objectives. Their low success rates therefore provide complementary evidence that structured spatial understanding is difficult not only for end-to-end action generation, but also for explicit target grounding and target-to-action conversion pipelines.

Fairness and Information Access

The main comparison in GEOMIND covers multiple embodied-agent paradigms under the same task instances, random seeds, workspace constraints, episode horizon, and success criteria. For end-to-end VLA policies, all models receive the same observation modalities and are evaluated as frozen policies without GEOMIND-specific fine-tuning. For modular baselines, the VLM grounding stage receives the same task instruction and RGB observations, while execution is performed by a scripted controller after target selection.

No evaluated method receives privileged simulator information during inference. Object poses, simulator-resolved object identities, oracle target coordinates, ground-truth task variables, segmentation masks, and task metadata are excluded from both VLA policies and modular baselines. These states are used only for evaluation, success checking, and diagnostic logging.

InformationVLA policiesModular baselines
Language instructionYesYes
RGB observationsYesYes
Robot proprioceptionYesYes
Object posesNoNo
Object identitiesNoNo
Segmentation masksNoNo
Task metadataNoNo
Oracle target coordinatesNoNo
Ground-truth task variablesNoNo
GEOMIND-specific fine-tuningNoNo
Role in evaluationEnd-to-end policyModular baseline
Inference structureDirect action predictionVLM grounding + scripted execution
Diagnostic signalTrajectory/action behaviorGrounding and target-selection errors

Table 9. Information access for end-to-end VLA policies and modular baselines. Both method families are evaluated without privileged simulator information during inference. Modular baselines use explicit VLM-based grounding and scripted execution, so their results are interpreted as cross-paradigm comparisons with additional diagnostic value.

Since VLA policies and modular baselines differ in inference structure, their results should be read as a cross-paradigm comparison rather than a comparison of identical policy interfaces. This is intentional: GEOMIND aims to evaluate whether different embodied-agent designs can solve the same structured spatial objectives under the same task conditions. The modular results complement the VLA results by revealing whether explicit visual grounding and scripted execution can overcome the spatial failures observed in end-to-end policies.

Spatial Structure Formalization

This appendix gives a compact description of the spatial structures used in GEOMIND. Figure 6 shows representative structure layouts. Our goal is not to bind each task to one fixed object coordinate template. Instead, each task is associated with a spatial structure, and GEOMIND samples multiple physical realizations that preserve the same task-relevant relation.

Example layouts generated by the spatial structure modeling and construction pipeline

Figure 6. Example layouts generated by the spatial structure modeling and construction pipeline.

Structure-Preserving Sampling

Let a scene contain NobjN_{\mathrm{obj}} objects O={oi}i=1Nobj\mathcal{O} = \{o_i\}_{i=1}^{N_{\mathrm{obj}}}. The pose of object oio_i is

xi=(pi,qi),(10)\mathbf{x}_i = (\mathbf{p}_i, \mathbf{q}_i), \tag*{(10)}

where pi\mathbf{p}_i is the object position and qi\mathbf{q}_i is the object orientation. For tabletop relations, we use the planar position pˉi=(xi,yi)\bar{\mathbf{p}}_i = (x_i, y_i). The full scene state is

X={xi}i=1Nobj.(11)\mathbf{X} = \{\mathbf{x}_i\}_{i=1}^{N_{\mathrm{obj}}}. \tag*{(11)}

We denote a spatial structure as SS. It specifies what should remain invariant during sampling, such as an object order, a grid layout, an anchor-based relation, or a geometric target rule. Given SS, GEOMIND samples object poses as

X∼P(X∣S,ϵ),(12)\mathbf{X} \sim\mathcal{P}(\mathbf{X} \mid S, \epsilon), \tag*{(12)}

where ϵ\epsilon controls small layout variations such as jitter or spacing changes. A sampled scene is accepted only if

X∈Ωstruct(S,ϵ)∩Ωphys.(13)\mathbf{X} \in\Omega_{\mathrm{struct}}(S, \epsilon) \cap\Omega_{\mathrm{phys}}. \tag*{(13)}

Here, Ωstruct(S,ϵ)\Omega_{\mathrm{struct}}(S, \epsilon) is the set of configurations satisfying the structural constraints, and Ωphys\Omega_{\mathrm{phys}} is the set of physically valid configurations.

In short, SS describes the stable spatial meaning of the task, while X\mathbf{X} is one sampled scene that realizes this meaning.

Representative Spatial Structures

As illustrated in Figure 6, GEOMIND uses several common spatial structures. We summarize them below with minimal constraints.

Ordered structure. This structure is used for tasks such as “pick the leftmost object” or “pick the second object from the right.” Let u\mathbf{u} be a unit vector in the ordering direction and define the projected coordinate

si=uTpˉi.(14)s_i = \mathbf{u}^{\mathsf{T}}\bar{\mathbf{p}}_i. \tag*{(14)}

Let π\pi be the permutation specifying the intended object order oπ(1)≺⋯≺oπ(Nobj)o_{\pi(1)} \prec\cdots\prec o_{\pi(N_{\mathrm{obj}})}. For a minimum projected separation δmin⁡>0\delta_{\min} > 0, a valid sample must satisfy

sπ(k+1)−sπ(k)≥δmin⁡,k=1,…,Nobj−1.(15)s_{\pi(k+1)} - s_{\pi(k)} \ge\delta_{\min}, \qquad k = 1, \ldots, N_{\mathrm{obj}} - 1. \tag*{(15)}

This keeps the object order unchanged while allowing the exact coordinates to vary.

Grid structure. This structure is used for row-column layouts, such as “the object in the top-left cell” or “the object in the same row as the bowl.” Each object has a grid index gi=(ri,ci)\mathbf{g}_i = (r_i, c_i). A sampled position is generated around the corresponding grid cell:

pˉi=c+(ci−1)dxu+(ri−1)dyv+ηi,∥ηi∥∞≤ϵ.(16)\bar{\mathbf{p}}_i = \mathbf{c} + (c_i - 1)d_x\mathbf{u} + (r_i - 1)d_y\mathbf{v} + \boldsymbol{\eta}_i, \qquad\lVert\boldsymbol{\eta}_i\rVert_{\infty} \le\epsilon. \tag*{(16)}

Here, c\mathbf{c} is the grid origin, dx,dy>0d_x, d_y > 0 are the grid spacings, u\mathbf{u} and v\mathbf{v} are unit vectors along the column and row directions, and ηi\boldsymbol{\eta}_i is the positional perturbation. The perturbation bound alone does not guarantee that row-column relations are preserved; accepted configurations must also satisfy the structural-validity condition above.

Anchor-based relation. This structure is used for tasks such as “the object left of the cup”, “the object near the plate”, or “the object between two bowls.” For an object oio_i and an anchor oao_a, define

Δia=pˉi−pˉa.(17)\Delta_{ia} = \bar{\mathbf{p}}_i - \bar{\mathbf{p}}_a. \tag*{(17)}

A directional relation is valid when

urTΔia≥τr,(18)\mathbf{u}_r^{\mathsf{T}}\Delta_{ia} \ge\tau_r, \tag*{(18)}

where ur\mathbf{u}_r is a unit vector in the relation direction and τr>0\tau_r > 0 is a directional separation threshold that excludes ambiguous cases. Let R(oi,oa)∈{0,1}R(o_i,o_a) \in\{0,1\} be the corresponding task predicate over candidate objects and the anchor; ineligible objects have predicate value zero. The target is accepted only when the relation selects one unique object:

{oi∈O:R(oi,oa)=1}={o∗}.(19)\{o_i \in\mathcal{O}: R(o_i,o_a) = 1\} = \{o^*\}. \tag*{(19)}

Geometric target. This structure is used when the target is an implicit position rather than a visible object. For example, the target may be the midpoint between two anchors:

gmid=12(pˉa+pˉb),(20)\mathbf{g}_{\mathrm{mid}} = \frac{1}{2}(\bar{\mathbf{p}}_a + \bar{\mathbf{p}}_b), \tag*{(20)}

or the centroid of the positions of a nonempty set of anchors indexed by A\mathcal{A}:

gcenter=1∣A∣∑a∈Apˉa.(21)\mathbf{g}_{\mathrm{center}} = \frac{1}{|\mathcal{A}|} \sum_{a \in\mathcal{A}} \bar{\mathbf{p}}_a. \tag*{(21)}

These tasks require the agent to infer a target from scene geometry, instead of directly selecting an existing object.

Dynamic Relations

For tasks with scene changes, GEOMIND distinguishes two cases. In scene-updating tasks, the target is determined by the current relation:

{oi∈O:Rt(oi,oa)=1}={ot∗}.(22)\{o_i \in\mathcal{O}: R_t(o_i,o_a) = 1\} = \{o_t^*\}. \tag*{(22)}

For example, the correct object is the one currently left of the cup.

In identity-tracking tasks, the target is determined by the initial relation and then kept fixed:

{oi∈O:R0(oi,oa)=1}={o∗},target⁡(t)=o∗.(23)\{o_i \in\mathcal{O}: R_0(o_i,o_a) = 1\} = \{o^*\}, \qquad\operatorname{target}(t) = o^*. \tag*{(23)}

For example, the correct object is the one that was originally left of the cup, even if it later moves away.

Physical Validation

Finally, each sampled scene is checked in the simulator. We retain instances that satisfy the simulator’s collision and workspace checks and have a reachable target. For distinct objects whose footprints must remain separated in the tabletop plane, a circular-footprint separation condition can be written as

∥pˉi−pˉj∥2≥ρi+ρj+dsafe,pˉi∈Ωtable,g∈Ωreach.(24)\lVert\bar{\mathbf{p}}_i - \bar{\mathbf{p}}_j \rVert_2 \ge\rho_i + \rho_j + d_{\mathrm{safe}}, \qquad\bar{\mathbf{p}}_i \in\Omega_{\mathrm{table}}, \qquad\mathbf{g} \in\Omega_{\mathrm{reach}}. \tag*{(24)}

Here, ρi\rho_i is the radius of a bounding disk for object ii’s planar footprint, dsafe≥0d_{\mathrm{safe}} \ge0 is a separation margin, Ωtable\Omega_{\mathrm{table}} is the admissible tabletop region, and g∈Ωreach\mathbf{g} \in\Omega_{\mathrm{reach}} specifies a reachable goal position. The pairwise inequality applies only to distinct object pairs requiring planar separation; it is not a universal constraint for containment or stacking. Task-specific contact feasibility is checked in simulation.

Summary

The structure gallery in Figure 6 provides visual examples of the layouts used in GEOMIND. The formulas above specify only the minimal constraints needed to preserve these structures during sampling. This allows GEOMIND to generate diverse scenes with the same task semantics, so evaluation focuses on spatial understanding rather than memorization.

Task Details

We present 26 illustrative task cards covering a subset of the benchmark. The full evaluation set contains 54 task variants: 20 Base, 7 Semantic, 7 Visual, and 20 Execution variants. The latter three categories constitute the 34 augmented variants. The number of task cards is not the total number of evaluated variants.

Each card gives a compact specification of one task variant, including the task instruction, the corresponding suite and task family, the spatial structure used for scene generation, the ground-truth target, the numerical ground-truth pose, and the evaluation criterion. The scene image on the left illustrates one sampled realization of the task, while the fields on the right define the task in a reproducible form. All target poses are reported in the simulator world frame, and distances are measured in meters unless otherwise stated.

Sampled realization of BindingBase_AnchorDirection_v0

Figure 7 (PDF p. 23).

BindingBase_AnchorDirection_v0

Task suite and family. GEOMIND-BASE / Target Identification

Instruction. Move the end effector near the cube to the right of the blue bowl.

Structure. anchor_direction_injected structure.

GT Target. Protocol target object block_right proximity region.

GT Pose. p∗=(0.40,−0.15,0.03) mp^{*} = (0.40, -0.15, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Sampled realization of ConstraintBase_SelectExclusionUnique_v0

Figure 8 (PDF p. 23).

ConstraintBase_SelectExclusionUnique_v0

Task suite and family. GEOMIND-BASE / Constraint Composition

Instruction. Move the end effector near the red cube that is closest to the blue plate and is not the leftmost cube.

Structure. exclusion_distance_constraint_injected structure.

GT Target. Protocol target object red_block_target proximity region.

GT Pose. p∗=(0.56,0.02,0.03) mp^{*} = (0.56, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Sampled realization of ConstraintBase_SelectVisualSpatial_v0

Figure 9 (PDF p. 23).

ConstraintBase_SelectVisualSpatial_v0

Task suite and family. GEOMIND-BASE / Constraint Composition

Instruction. Move the end effector near the wood-grain cylinder to the right of the red block.

Structure. visual_spatial_constraint_injected structure.

GT Target. Protocol target object wood_cylinder_target proximity region.

GT Pose. p∗=(0.40,−0.08,0.03) mp^{*} = (0.40, -0.08, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Sampled realization of GeometricBase_IsometricExtension_v0

Figure 10 (PDF p. 23).

GeometricBase_IsometricExtension_v0

Task suite and family. GEOMIND-BASE / Geometric Inference

Instruction. You already hold the cube. Place it on the red–blue line extension at equal spacing.

Structure. 2-anchor_line_blocks_runtime_injected structure.

GT Target. Implicit equal-spacing extension point beyond the second anchor.

GT Pose. p∗=(0.66,0.02,0.03) mp^{*} = (0.66, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.05< 0.05 m and the final state to remain stable for 1.0 s. GeometricBase_PerfectMidpoint_v0

GeometricBase_PerfectMidpoint_v0 task scene

Figure 11 (PDF p. 24).

Task suite and family. GEOMIND-BASE / Geometric Inference

Instruction. You already hold the cube. Place it midway between the red and blue reference cubes.

Structure. 2-anchor line_blocks_runtime_injected structure.

GT Target. Implicit midpoint region between the two colored anchor blocks.

GT Pose. p∗=(0.42,0.02,0.03) m\mathbf{p}^{*} = (0.42, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.05 m< 0.05\ \mathrm{m} and the final state to remain stable for 1.0 s.

GeometricBase_PointSymmetry_v0 task scene

Figure 12 (PDF p. 24).

GeometricBase_PointSymmetry_v0

Task suite and family. GEOMIND-BASE / Geometric Inference

Instruction. You already hold the cube. Place it symmetrically to the blue cube about the red cube.

Structure. 2-anchor diagonal_blocks_runtime_injected structure.

GT Target. Implicit point-symmetric reflection of the reference anchor across the center anchor.

GT Pose. p∗=(0.30,−0.08,0.03) m\mathbf{p}^{*} = (0.30, -0.08, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.05 m< 0.05\ \mathrm{m} and the final state to remain stable for 1.0 s.

GeometricBase_SquareCompletion_v0 task scene

Figure 13 (PDF p. 24).

GeometricBase_SquareCompletion_v0

Task suite and family. GEOMIND-BASE / Geometric Inference

Instruction. You already hold the fourth cube. Place it to complete a square with the three anchor cubes.

Structure. 3-anchor square_corner_runtime_injected structure.

GT Target. Implicit missing square corner inferred from three anchors.

GT Pose. p∗=(0.48,0.12,0.03) m\mathbf{p}^{*} = (0.48, 0.12, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.05 m< 0.05\ \mathrm{m} and the final state to remain stable for 1.0 s.

GeometricBase_TrendContinuation_v0 task scene

Figure 14 (PDF p. 24).

GeometricBase_TrendContinuation_v0

Task suite and family. GEOMIND-BASE / Geometric Inference

Instruction. You already hold the next cube. Place it to continue the visible row/column pattern.

Structure. 3-anchor diagonal_trend_blocks_runtime_injected structure.

GT Target. Implicit next point in the anchor trend.

GT Pose. p∗=(0.61,0.15,0.03) m\mathbf{p}^{*} = (0.61, 0.15, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.05 m< 0.05\ \mathrm{m} and the final state to remain stable for 1.0 s.

BindingBase_GridRowcolGrounding_v0 task scene

Figure 15 (PDF p. 24).

BindingBase_GridRowcolGrounding_v0

Task suite and family. GEOMIND-BASE / Target Identification

Instruction. Move the end effector near the cube at row 1, column 2.

Structure. runtime_grid_2x3_static structure.

GT Target. Protocol target object grid_cell_1 proximity region.

GT Pose. p∗=(0.37,−0.03,0.03) m\mathbf{p}^{*} = (0.37, -0.03, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06 m< 0.06\ \mathrm{m} and the final state to remain stable for 1.0 s.

Robot arm above five cubes

Figure 1 (PDF p. 25).

BindingBase_OrdinalInsertStatic_v0 Task suite and family. GEOMIND-BASE / Target Identification Instruction. Move the end effector near the second cube from the left. Structure. runtime_block_line_static_ordinal structure. GT Target. Protocol target object ordinal_static_1 proximity region. GT Pose. p∗=(0.43,−0.06,0.03)p^{*} = (0.43, -0.06, 0.03) m; θ∗=0.00\theta^{*} = 0.00 rad. Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above four colored cubes

Figure 2 (PDF p. 25).

ReindexBase_ReindexExtreme_v0 Task suite and family. GEOMIND-BASE / Scene Updating Instruction. Move the end effector near the leftmost cube after reordering. Structure. runtime_block_line structure. GT Target. Protocol target object reindex_block_1 proximity region. GT Pose. p∗=(0.42,−0.05,0.03)p^{*} = (0.42, -0.05, 0.03) m; θ∗=0.00\theta^{*} = 0.00 rad. Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above four cubes

Figure 3 (PDF p. 25).

ReindexBase_ReindexInsert_v0 Task suite and family. GEOMIND-BASE / Scene Updating Instruction. Move the end effector near the second-from-left cube. Structure. runtime_block_line_insert structure. GT Target. Protocol target object insert_block_3 proximity region. GT Pose. p∗=(0.43,0.18,0.03)p^{*} = (0.43, 0.18, 0.03) m; θ∗=0.00\theta^{*} = 0.00 rad. Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above cubes including a red cube

Figure 4 (PDF p. 25).

ReindexBase_ReindexNeighbor_v0 Task suite and family. GEOMIND-BASE / Scene Updating Instruction. Move the end effector near the cube immediately left of the red cube. Structure. runtime_red_anchor_neighbor structure. GT Target. Protocol target object right_neighbor_block proximity region. GT Pose. p∗=(0.40,0.06,0.03)p^{*} = (0.40, 0.06, 0.03) m; θ∗=0.00\theta^{*} = 0.00 rad. Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above red and other colored cubes

Figure 5 (PDF p. 25).

ReindexBase_ReindexSubset_v0 Task suite and family. GEOMIND-BASE / Scene Updating Instruction. Move the end effector near the middle cube among the red cubes only. Structure. runtime_red_subset_with_distractors structure. GT Target. Protocol target object red_subset_block_2 proximity region. GT Pose. p∗=(0.40,0.11,0.03)p^{*} = (0.40, 0.11, 0.03) m; θ∗=0.00\theta^{*} = 0.00 rad. Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

ReindexAugSem_ReindexExtreme_v0 task scene

Figure 1 (PDF p. 26).

ReindexAugSem_ReindexExtreme_v0

Task suite and family. GEOMIND-AUG / Scene Updating

Instruction. Move the end effector near the leftmost cube after reordering.

Structure. runtime_block_line structure.

GT Target. Protocol target object reindex_block_1 proximity region.

GT Pose. p∗=(0.42,−0.05,0.03) mp^{*} = (0.42, -0.05, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

ReindexAugVisual_ReindexExtreme_v0 task scene

Figure 2 (PDF p. 26).

ReindexAugVisual_ReindexExtreme_v0

Task suite and family. GEOMIND-AUG / Scene Updating

Instruction. Move the end effector near the leftmost cube after reordering.

Structure. runtime_block_line structure.

GT Target. Protocol target object reindex_block_1 proximity region.

GT Pose. p∗=(0.42,−0.05,0.03) mp^{*} = (0.42, -0.05, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

RelationAugSem_PlaceDirectional_v0 task scene

Figure 3 (PDF p. 26).

RelationAugSem_PlaceDirectional_v0

Task suite and family. GEOMIND-AUG / Relational Grounding

Instruction. You already hold the wooden cube. Place it in the valid target region in front of the blue plate.

Structure. directional_region_semantic_distractors structure.

GT Target. Directional target zone relative to blue_plate.

GT Pose. p∗=(0.57,0.02,0.03) mp^{*} = (0.57, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.09< 0.09 m and the final state to remain stable for 1.0 s.

RelationAugVisual_PlaceDirectional_v0 task scene

Figure 4 (PDF p. 26).

RelationAugVisual_PlaceDirectional_v0

Task suite and family. GEOMIND-AUG / Relational Grounding

Instruction. You already hold the wooden cube. Place it in the valid target region in front of the blue plate.

Structure. directional_region_visual_clutter structure.

GT Target. Directional target zone relative to blue_plate.

GT Pose. p∗=(0.57,0.02,0.03) mp^{*} = (0.57, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.09< 0.09 m and the final state to remain stable for 1.0 s.

RelationBase_PlaceDirectional_v0 task scene

Figure 5 (PDF p. 26).

RelationBase_PlaceDirectional_v0

Task suite and family. GEOMIND-BASE / Relational Grounding

Instruction. You already hold the wooden cube. Place it in the valid target region in front of the blue plate.

Structure. directional_region_injected structure.

GT Target. Directional target zone relative to blue_plate.

GT Pose. p∗=(0.65,0.02,0.03) mp^{*} = (0.65, 0.02, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the distance from the object center to the target position to be <0.09< 0.09 m and the final state to remain stable for 1.0 s.

Task image

Figure 1 (PDF p. 27).

RelationBase_PlaceInside_v0

Task suite and family. GEOMIND-BASE / Relational Grounding

Instruction. You already hold the cube. Put it fully inside the blue box.

Structure. containment_standard_injected structure.

GT Target. Stable containment inside blue_box_floor.

GT Pose. p∗=(0.43,0.04,0.03) mp^{*} = (0.43, 0.04, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires stable containment in the box, a center error <0.06< 0.06 m relative to the target position, and stability for 1.0 s.

Task image

Figure 2 (PDF p. 27).

RelationBase_PlaceOnTop_v0

Task suite and family. GEOMIND-BASE / Relational Grounding

Instruction. You already hold the cube. Release it onto the top surface of the red wooden block so it rests stably.

Structure. support_top_injected structure.

GT Target. Stable top placement on red_support_block.

GT Pose. p∗=(0.41,0.03,0.07) mp^{*} = (0.41, 0.03, 0.07)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires stable placement on the support surface, a placement error <0.01< 0.01 m, and stability for 1.0 s.

Task image

Figure 3 (PDF p. 27).

TrackBase_TrackClutter_v0

Task suite and family. GEOMIND-BASE / Identity Tracking

Instruction. After the clutter settles, move the end effector near the cube that was initially in the middle.

Structure. clutter_robust_runtime_identity_line structure.

GT Target. Protocol target object identity_block_1 proximity region.

GT Pose. p∗=(0.41,−0.01,0.03) mp^{*} = (0.41, -0.01, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Task image

Figure 4 (PDF p. 27).

TrackBase_TrackMulti_v0

Task suite and family. GEOMIND-BASE / Identity Tracking

Instruction. Move the end effector near the cube that started at the far left.

Structure. runtime_identity_multistep_line structure.

GT Target. Protocol target object identity_block_0 proximity region.

GT Pose. p∗=(0.40,−0.20,0.03) mp^{*} = (0.40, -0.20, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Task image

Figure 5 (PDF p. 27).

TrackBase_TrackSingle_v0

Task suite and family. GEOMIND-BASE / Identity Tracking

Instruction. Move the end effector near the cube that started in the middle.

Structure. runtime_identity_line structure.

GT Target. Protocol target object identity_block_1 proximity region.

GT Pose. p∗=(0.41,−0.01,0.03) mp^{*} = (0.41, -0.01, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above three cubes

Figure 1 (PDF p. 28).

TrackAugSem_TrackSingle_v0

Task suite and family. GEOMIND-AUG / Identity Tracking

Instruction. Move the end effector near the cube that started in the middle.

Structure. runtime_identity_line structure.

GT Target. Protocol target object identity_block_1 proximity region.

GT Pose. p∗=(0.41,−0.01,0.03) mp^{*} = (0.41, -0.01, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

Robot arm above three cubes

Figure 2 (PDF p. 28).

TrackAugVisual_TrackSingle_v0

Task suite and family. GEOMIND-AUG / Identity Tracking

Instruction. Move the end effector near the cube that started in the middle.

Structure. runtime_identity_line structure.

GT Target. Protocol target object identity_block_1 proximity region.

GT Pose. p∗=(0.41,−0.01,0.03) mp^{*} = (0.41, -0.01, 0.03)\ \mathrm{m}; θ∗=0.00 rad\theta^{*} = 0.00\ \mathrm{rad}.

Eval. Success requires the end-effector distance to the target object’s center to be <0.06< 0.06 m and the final state to remain stable for 1.0 s.

References

  1. [1]Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.DOI
  2. [2]Generalist AI Team. GEN-0: Embodied foundation models that scale with physical interaction. Generalist AI Blog, 2025. https://generalistai.com/blog/preview-uqlxvb-bb.html.
  3. [3]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-Source Vision-Language-Action model, 2024.arxiv.org/abs/2406.09246
  4. [4]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Lilyming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. Pi-0: A Vision-Language-Action flow model for general robot control, 2024.arxiv.org/abs/2410.24164
  5. [5]Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π0.5: a Vision-Language-Action model with Open-World Generalization. In 9th Annual Conference on Robot Learning, 2025.arxiv.org/abs/2504.16054
  6. [6]Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. UniVLA: Learning to act anywhere with task-centric latent actions, 2025.arxiv.org/abs/2505.06111
  7. [7]Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1B: a diffusion foundation model for bimanual manipulation, 2024.arxiv.org/abs/2410.07864
  8. [8]Brianna Zitkovich, Tianhe Yu, Sichun Xu, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR, 2023.arxiv.org/abs/2307.15818
  9. [9]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.arxiv.org/abs/2602.15922
  10. [10]Jiaming Zhou, Qihang Zhang, Gangwei Xu, et al. Zero-WAM: In-context world-action modeling from human videos for open-ended task generalization. arXiv preprint arXiv:2608.26103, 2026.arxiv.org/abs/2608.26103
  11. [11]Yuran Wang, Siqiao Huang, Mingleyang Li, et al. OpenWAM: An open, modular exploration towards systematic world-action model pretraining. arXiv preprint arXiv:2609.07398, 2026.arxiv.org/abs/2609.07398
  12. [12]Heng Yu, David D. Yuan, Juze Zhang, et al. OpenWAM: An open framework for composable world-action models. arXiv preprint arXiv:2610.07922, 2026.arxiv.org/abs/2610.07922
  13. [13]Runyu Lu, Yubo Wu, Ethan Kou, et al. ASPIRE: Agentic /skills discovery for robotics. arXiv preprint arXiv:2607.00272, 2026.arxiv.org/abs/2607.00272
  14. [14]Wenli Xiao, Jia Xie, Tonghe Zhang, et al. ENPIRE: Agentic robot policy self-improvement in the real world. arXiv preprint arXiv:2606.19980, 2026.arxiv.org/abs/2606.19980
  15. [15]Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring spatial representations for Visual-Language-Action model, 2025.arxiv.org/abs/2501.15830
  16. [16]Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang, Yu Qiao, et al. InternVLA-M1: A spatially guided vision-language-action framework for generalist robot policy. arXiv preprint arXiv:2510.13778, 2025.arxiv.org/abs/2510.13778
  17. [17]Zhengshen Zhang, Hao Li, Yalun Dai, Zhengbang Zhu, Lei Zhou, Chenchen Liu, Dong Wang, Francis E. H. Tay, Sijin Chen, Ziwei Liu, Yuxiao Liu, Xinghang Li, and Pan Zhou. From spatial to actions: Grounding Vision-Language-Action model in spatial foundation priors, 2025.arxiv.org/abs/2510.17439
  18. [18]Bo Liu, Yifeng Zhu, Chonkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning, 2023.arxiv.org/abs/2306.03310
  19. [19]Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhaio Huang, Rui Chen, and Hao Su. ManiSkill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023.arxiv.org/abs/2302.04659
  20. [20]Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. RLBench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020.arxiv.org/abs/1909.12271
  21. [21]Shiduo Zhang, Zhe Xu, Peijiu Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VLABench: A Large-Scale benchmark for Language-Conditioned robotics manipulation with Long-Horizon reasoning tasks, 2024.arxiv.org/abs/2412.18194
  22. [22]Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020.arxiv.org/abs/1910.10897
  23. [23]Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning, 2023.arxiv.org/abs/2210.03094
  24. [24]Tianxiang Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025.arxiv.org/abs/2506.18088
  25. [25]Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412.14171, 2024.arxiv.org/abs/2412.14171
  26. [26]Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks, July 2022. arXiv:2112.03227 [cs].arxiv.org/abs/2112.03227
  27. [27]Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024.arxiv.org/abs/2405.05941
  28. [28]Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-Scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024.arxiv.org/abs/2406.02523
  29. [29]Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI. arXiv preprint arXiv:2410.00425, 2024.arxiv.org/abs/2410.00425
  30. [30]Ning Gao, Yilun Chen, Shuai Yang, Xinyi Chen, Yang Tian, Hao Li, Haifeng Huang, Hanqing Wang, Tai Wang, and Jiangmiao Pang. GENMANIP: LLM-driven simulation for generalizable Instruction-Following manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12187–12198, June 2025.arxiv.org/abs/2506.10966
  31. [31]Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, et al. RMBench: Memory-Dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026.arxiv.org/abs/2603.01229
  32. [32]Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, and Joyce Chai. RoboMME: Benchmarking and understanding memory for robotic generalist policies. arXiv preprint arXiv:2603.04639, 2026.arxiv.org/abs/2603.04639
  33. [33]Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, and Zhou Zhao. MEMOBench: A process level memory benchmark for robotic manipulation. arXiv preprint arXiv:2609.07047, 2026.arxiv.org/abs/2609.07047
  34. [34]Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, and Yueting Zhuang. RoboSPA: Can VLA models go beyond simple scenes and Short-Horizon tasks? In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 23457–23492. Association for Computational Linguistics, 2026.arxiv.org/abs/2609.05324
  35. [35]Sirui Chen, Huang Huang, Qineng Wang, Wenlong Huang, Manling Li, Ruohan Zhang, Li Fei-Fei, and Jiajun Wu. BEHAVIOR-Spatial benchmark: Evaluating spatial understanding for robotics. arXiv preprint https://arxiv.org/abs/2610.34051, 2026.
  36. [36]Zhongbo Zhang, Jiayi Jin, Yifan Wang, Zaibin Zhang, Haiwen Diao, Lijun Wang, and Huchuan Lu. VABench: Measuring embodied spatial intelligence through visual demonstrations, active perception, and metric control. arXiv preprint https://arxiv.org/abs/2609.19554, 2026.
  37. [37]Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. RoboRefer: Towards spatial referring with reasoning in Vision-Language models for robotics. arXiv preprint arXiv:2506.04308, 2025.arxiv.org/abs/2506.04308
  38. [38]Yanpeng Zhao, Wentao Ding, Hongtao Li, Baoxiong Jia, and Zilong Zheng. ESPIRE: A diagnostic benchmark for embodied spatial reasoning of Vision-Language models. arXiv preprint arXiv:2603.13033, 2026.arxiv.org/abs/2603.13033
  39. [39]Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, and Elia Bruni. iVISPAR—an interactive Visual-Spatial reasoning benchmark for VLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26745–26769, 2025.arxiv.org/abs/2502.03214
  40. [40]Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. SoFar: Language-Grounded orientation bridges spatial reasoning and object manipulation, 2025.arxiv.org/abs/2502.13143
  41. [41]Kairun Wen, Yuzhi Huang, Runyu Chen, Hui Zheng, Yunlong Lin, Panwang Pan, Chenxin Li, Wenyan Cong, Jian Zhang, Junbin Lu, et al. DynamicVerse: A Physically-Aware multimodal framework for 4d world modeling. arXiv preprint arXiv:2512.03000, 2025.arxiv.org/abs/2512.03000
  42. [42]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 06 2023.
  43. [43]Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jie Wu, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, et al. Thinking in dynamics: How multimodal large language models perceive, track, and reason dynamics in physical 4d world. arXiv preprint arXiv:2603.12746, 2026.arxiv.org/abs/2603.12746
  44. [44]Ilias Stogiannidis, Steven McDonagh, and Sotirios A. Tsaftaris. Mind the gap: Benchmarking spatial reasoning in Vision-Language models, 2025.arxiv.org/abs/2503.19707
  45. [45]Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. ViewSpatial-Bench: Evaluating multi-perspective spatial localization in Vision-Language models, 2025.arxiv.org/abs/2505.21500
  46. [46]Songsong Yu, Yuxin Chen, Hao Ju, Lianjie Jia, Fuxi Zhang, Shaofei Huang, Yuhan Wu, Rundi Cui, Binghao Ran, Zaibin Zhang, Zhedong Zheng, Zhipeng Zhang, Yifan Wang, Lin Song, Lijun Wang, Yanwei Li, Ying Shan, and Huchuan Lu. How far are VLMs from visual spatial intelligence? A Benchmark-Driven perspective. arXiv preprint arXiv:2509.18905, 2025.arxiv.org/abs/2509.18905
  47. [47]Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. STI-Bench: Are mllms ready for precise spatial-temporal world understanding? arXiv preprint arXiv:2503.23765, 2025.arxiv.org/abs/2503.23765
  48. [48]Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniSpatial: Towards comprehensive spatial reasoning benchmark for vision language models. arXiv preprint arXiv:2506.03135, 2025.arxiv.org/abs/2506.03135
  49. [49]Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, and Yejin Choi. ESI-Bench: Towards embodied spatial intelligence that closes the Perception-Action loop. arXiv preprint https://arxiv.org/abs/2605.18746, 2026.
  50. [50]Chih-Ting Liao, Xi Xiao, Chunlun Meng, Zhangquan Chen, Yitong Qiao, Weilin Zhou, Tianyang Wang, Xu Zheng, and Xin Cao. SpaMEM: Benchmarking dynamic spatial reasoning via Perception-Memory integration in embodied environments. arXiv preprint https://arxiv.org/abs/2604.22409, 2026.
  51. [51]Johan Bjorck, Fernando Castañeda, Nikita Chernia­dev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GROOT n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.
  52. [52]Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing AI into the physical world. arXiv preprint arXiv:2503.20020, 2025.arxiv.org/abs/2503.20020
  53. [53]Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, et al. IGen: Scalable data generation for robot learning from Open-World images. arXiv preprint arXiv:2512.01773, 2025.arxiv.org/abs/2512.01773
  54. [54]Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. RoboSpatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15768–15780, 2025.arxiv.org/abs/2411.16537
  55. [55]Ji Zhang, Shihan Wu, Xu Luo, Hao Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. InSpire: Vision-Language-Action models with intrinsic spatial reasoning, 2025.arxiv.org/abs/2505.13888
  56. [56]Fuhao Li, Wenxuan Song, Han Zhao, Jingbo Wang, Pengxiang Ding, Donglin Wang, Long Zeng, and Haoang Li. Spatial forcing: Implicit spatial representation alignment for vision-language-action model. arXiv preprint arXiv:2510.12276, 2025.arxiv.org/abs/2510.12276
  57. [57]Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. COPA: General robotic manipulation through spatial constraints of parts with foundation models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9488–9495. IEEE, 2024.arxiv.org/abs/2403.08248
  58. [58]Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. MOKA: Open-world robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174, 2024.arxiv.org/abs/2403.03174
  59. [59]Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831

Paper details

vlnc:W_ubbaeudg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents