ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models

Please see Contributions and Author List for more author details.

Introduction

General-purpose reasoning models are beginning to generalize across diverse embodied settings. GPT-6 Astra [1], in particular, demonstrates strong performance in embodied manipulation. A closer examination of its behavior shows that this embodied competence is accompanied by strong spatial perception. It can localize task-relevant interaction targets in cluttered scenes despite the presence of visually similar distractors.

This observation highlights the role of spatial perception in translating high-level reasoning into physical action. Grounding provides this connection by linking natural-language descriptions to their visual referents in an image [2, 3]. Reliable grounding requires precise localization of all image regions that satisfy a description, with explicit recognition when no match exists [4]. In embodied environments, clutter and occlusion make this correspondence particularly difficult to establish [5]. A grounding error can therefore misdirect an agent’s actions even when its high-level reasoning is sound.

We therefore examine Astra’s spatial grounding capabilities in greater detail. Our analysis in § 3 illustrates its ability to handle diverse grounding queries, while also revealing localization errors in tasks with precise spatial requirements. Using Astra as a reference, we ask whether an open 9B-parameter vision-language model can match or surpass Astra’s grounding performance by expanding the scale and coverage of grounding supervision, with the model architecture and training recipe held fixed.

To investigate this question, we develop a unified pipeline for constructing grounding supervision from existing open datasets. The pipeline combines robot manipulation recordings from more than thirty dataset families with complementary visual data to broaden task coverage. We automatically annotate the robot recordings to supply the object-level labels they generally lack. All sources are then converted into question–answer supervision that is verifiable against the supporting annotations, with controls to reduce shortcuts between question wording and answer format. This pipeline yields approximately 1.5 million training samples, which we use to fine-tune Qwen3.5-9B [6] for a single epoch, producing ACE-Brain-0.6. Figure 1 provides an overview of our approach.

We evaluate ACE-Brain-0.6 on seven public grounding benchmarks. The model outperforms the Qwen3.5-9B baseline on six benchmarks and remains behind on PixMo-Count. It exceeds GPT-6 Astra on four benchmarks and achieves a slightly higher seven-benchmark average (75.41 versus 73.68), while remaining behind on PixMo-Count, RefSpatial, and PointArena. These results suggest that a 9B open model trained with diverse, verifiable grounding supervision can match Astra’s aggregate performance on the evaluated benchmarks.

We further examine data scaling by fine-tuning from the same base checkpoint with a fixed training recipe for each data release. Across releases that expand both sample count and supervision coverage, the seven-benchmark average increases approximately linearly with the logarithm of sample count over the evaluated range, slightly exceeding Astra’s average at about 1.5 million samples. In a control experiment, adding samples only from existing sources improves this average by less than one point, suggesting that broader supervision coverage contributes to the observed gains.

Our contributions are threefold:

• A data pipeline for grounding. We collect large-scale open-source robot, egocentric, object-centric, and spatial data, automatically annotate robot recordings, and convert these sources into verifiable question–answer supervision for target identification, complete instance coverage, and precise localization.

• An empirical study of grounding-data scaling. Through a controlled series of training runs, we show that grounding performance improves steadily with data scale and that expanding the coverage of supervision contributes substantially more than increasing the volume of existing sources.

• ACE-Brain-0.6. By scaling verifiable grounding data to approximately 1.5 million samples, we train a 9B model that exceeds GPT-6 Astra on four of seven grounding benchmarks and achieves a higher overall average.

Related Work

Visual Grounding and Embodied Spatial Understanding

Visual grounding links language descriptions to image locations. Detector-based methods such as GLIP and Grounding DINO align visual and language features for phrase grounding and open-set localization [7, 8]. Generative approaches represent spatial outputs as token sequences, from coordinate and class prediction in Pix2Seq [9] to multi-task visual prediction in Florence-2 [10] and detection, referring, and pointing in Rex-Omni [11]. Grounding also supports broader embodied understanding: PaLM-E integrates sensor inputs into language modeling [12], Gemini Robotics-ER develops spatial understanding and embodied reasoning [13], and RynnBrain and Embodied-R1.5 combine localization with embodied understanding and reasoning [14, 15]. These works connect spatial prediction with increasingly broad model capabilities. We focus on grounding itself, keeping the base vision-language model and training recipe fixed to examine the effect of training data.

Grounding Data and Supervision

Grounding supervision comes from human annotation, automatic labeling, and geometric derivation. Molmo-and-PixMo collects language-conditioned point annotations, including complete instance sets and absent-target examples [16]. GrIT pairs caption phrases with predicted image regions [17]. SpatialVLM constructs spatial question–answer data using segmentation, depth, and object descriptions [18], while RoboPoint derives instruction-point supervision from simulated scene geometry [19]. We build on these approaches by combining robot recordings, egocentric interactions, dense object and part annotations, and 3D scenes. Our pipeline automatically annotates robot recordings and converts these heterogeneous sources into a common question–answer format. Answers remain linked to supporting annotations, with checks for instance completeness, spatial constraints, and target absence. This training collection allows us to study how supervision scale and composition affect grounding performance.

Analysis of GPT-6 Astra

Our experiments with GPT-6 Astra (hereafter Astra) as a robotic manipulation policy highlight how visual grounding supports analytic geometry in translating task objectives into physical actions. Grounding identifies relevant objects, interaction regions, and spatial references in the observed scene, and analytic geometry uses spatial quantities derived from these references to determine positions, relative displacements, and motion parameters. In this section, we first illustrate this dependency through two manipulation cases, and then examine Astra’s grounding capabilities in greater detail.

Case Studies: From Grounding to Geometric Robot Manipulation

The two cases illustrate complementary forms of grounding used in geometric manipulation (Figure 2). Fruit basket placement requires selecting points on the manipulated object and its destination, whereas T-shirt folding requires extracting the garment’s spatial extent. In both cases, these visual references determine the geometric quantities used to construct the action.

Fruit basket placement. To place a loaded fruit basket onto a black sponge, Astra selected points along the basket rim and the corners of the destination surface. Using calibrated RGB-D observations, it fitted the basket rim and support plane to estimate their relative configuration. The resulting planar centers of the basket and destination, $c_b$ and $c_s$, defined an initial displacement $\Delta_{xy}=c_s-c_b$. Astra used these estimates to construct coordinated arm motions and reassessed the basket’s pose during placement. Here, grounding specifies the object geometry and support surface to which the placement calculation applies.

T-shirt folding. To fold an already side-tucked T-shirt into a square, Astra constructed a color-based mask within a selected image region and extracted the garment contour. Projecting this contour onto an assumed cloth plane gave an estimated length $L \approx363$ mm and width $W \approx300$ mm. These dimensions informed a bottom fold of depth $d=L-W$, with a corresponding hem displacement $\Delta_{\mathrm{hem}}=2d \approx125$ mm. Both arms executed the staged fold, and Astra extracted the contour again after release to assess the resulting shape. Grounding the garment’s extent thus supplies the dimensions needed to select a fold that satisfies the task’s shape requirement.

Together, these cases illustrate how task-relevant grounding supplies the spatial inputs needed for geometric action construction and subsequent visual verification. These observations motivate studying grounding as a measurable component of spatial perception and using Astra’s capabilities as a reference for improving our model.

A Closer Look at Astra’s Grounding Capabilities

Figure 3 illustrates Astra’s grounding through eight examples of its strengths and two examples of its limitations. The queries require selecting objects, locating all members of a set, identifying spatial regions, or associating functions and actions with visual locations. These examples make concrete the range of visual references that Astra can supply for subsequent geometric reasoning and the localization errors that can arise.

Strengths. Astra can resolve descriptions that distinguish a particular instance from other objects of the same category. In the object grounding example (a), it identifies the chair farthest from the viewer. In the ordinal referring example (b), it selects the picture that is third from the top in the second column from the left.

Grounding also extends to sets of instances. In the counting example (c), Astra returns ten points corresponding to ten books, with the number of predictions matching the reference count. In the multi-instance example (f), it returns a point on each of the 15 bottles. Both queries ask for all matching objects, and both responses represent the target set through individual locations.

The examples of region grounding and spatial relations require locating unoccupied space under relational descriptions. In (d), Astra places points in the free space between the green and white blocks at the back of the scene. In (g), it locates vacant space to the right of the toilet. The predictions lie within the corresponding reference regions.

Functional descriptions and interaction goals provide further ways to specify a visual target. In the affordance example (e), Astra locates the window from a description of a structure that can be opened or closed to admit light and air. For the interaction task in (h), it predicts a point within the annotated end-effector interaction region for cutting with a knife. The former identifies an object through its function; the latter localizes a region relevant to a specified action.

Limitations. The fine-grained localization example (i) asks which part of a mug should be gripped to lift it. Astra’s point lies close to the handle but just outside the valid reference mask. The prediction is near the relevant object part, yet it fails the geometric localization requirement. This case highlights the sensitivity of fine-grained grounding to local boundaries.

The spatially constrained query in (j) asks for free space in front of the steamer, but Astra places its points on the cutting board to the right, outside the annotated region. Here, the error concerns localization of the region specified by a spatial relation. Together, the two failures show why grounding must satisfy both local geometric boundaries and relations to the intended reference objects.

Implications for ACE-Brain. These examples illustrate a range of grounded references that can support geometric reasoning, while the failures expose remaining difficulties in precise localization and spatial constraint satisfaction. They motivate using Astra’s grounding performance as a reference for developing these capabilities in our model. We investigate whether scaling diverse, verifiable grounding supervision can bring our model toward or beyond this reference. § 4 describes the data pipeline, and § 5 evaluates the resulting gains and remaining gaps across grounding tasks.

Method

This section describes the three components of ACE-Brain-0.6: the model and the grounding tasks it is trained to solve (§ 4.1), the data we collect and the pipeline that turns it into grounding supervision (§ 4.2), and the training recipe (§ 4.3).

Model and Problem Formulation

Grounding as text generation. We cast every grounding task as conditional text generation: given one or more images and a natural-language query, the model produces a short answer in one of five forms, namely a point $(x, y)$, a list of points, a box $(x_1, y_1, x_2, y_2)$, an integer count, or a categorical token (yes or no for spatial judgments, none for absent targets). Coordinates are integers normalized to $[0, 1000]$ along each image axis, and point lists are written as $[[x_1, y_1], \ldots]$ without per-point labels. This single output interface covers the task families behind the capabilities of § 3: referring by category, attribute, position, relation, or function; pointing to and counting every instance of a set; spatial relation judgments and free-space localization; grounding of parts, contact regions, placement destinations, and end-effector trajectories; and, in robot scenes, locating a target across two camera views.

Base model. ACE-Brain-0.6 is initialized from Qwen3.5-9B [6]. We fine-tune all language-model parameters, keep the vision encoder frozen, and disable the multi-token-prediction head. The model is trained and evaluated in non-thinking mode: the chat template emits an empty reasoning block before each answer, and we exclude these template tokens from the training loss so that the model never learns to emit them inside an answer.

Data Pipeline and Data Scale

Data collection. We draw training data from three kinds of sources. Robot manipulation data covers the target domain: manipulation episodes spanning single-arm, dual-arm, mobile, and humanoid embodiments. Since these recordings rarely carry object labels, we annotate them automatically as described below. Egocentric human–object interaction data adds first-person views close to a wrist or head camera, with hand–object masks and contact regions. General pointing and grounding data supplies what robot data lacks: exhaustive instance sets for counting, part masks for functional grounding, and 3D annotations for spatial relations and free space; a small amount of general visual question answering data preserves broad multimodal ability.

Automatic annotation of robot data. Robot recordings provide observations, actions, and instructions but rarely object labels. We convert them into grounding supervision in three steps (Figure 4). First, we select a few keyframes per episode from the action signal, favoring gripper-state transitions and large action changes while keeping the start, middle, and end of the trajectory. Second, a VLM lists the objects visible in each episode and camera with short names and appearance descriptions [20], giving an open vocabulary tied to the scene and viewpoint. Third, Grounding DINO [8] detects these objects and SAM 3 [21] refines their boxes; a separate SAM 3 recognition branch then relabels each box with its best-overlapping concept detection, correcting names without moving or adding boxes. The resulting annotations yield robot-scene supervision for referring, pointing, counting, free-space and destination grounding, and rejection of absent objects.

Question generation. All sources are turned into question–answer pairs from templates, under rules that keep every answer verifiable against its annotation. Counting and every-instance questions are emitted only when the instance set is complete; negative questions are checked against the vocabulary of the image so that none is correct; free-space targets mark unoccupied 2D regions and do not assert 3D feasibility. Answers are written in the output interface of § 4.1, and shortcuts between question wording and answer form are removed: no answer form exceeds 70% within a template, yes/no answers are balanced, and the number of points matches what the question asks for.

Quality control and data scale. Every detector box in robot scenes is verified by a VLM and dropped if its label does not match the region. Images and episodes from the evaluation benchmarks are excluded from training by matching. One training sample is one or more images with a conversation of one or more question–answer turns. The final collection contains about 1.54 million samples, comprising about 3.41 million question–answer pairs across about 1.39 million distinct image paths.

Training

Objective. We minimize the token-averaged cross-entropy over answer tokens only. Question and image tokens receive no loss, and, as noted in § 4.1, the four tokens of the empty reasoning block are masked as well. Conversations are packed into fixed-length sequences of 16,384 tokens with variable-length attention, so that tokens from different conversations never attend to each other; each global step consists of 16 such sequences, or roughly 140 samples.

Optimization. We use AdamW with $\beta= (0.9, 0.95)$, $\epsilon= 10^{-8}$, and weight decay 0.1. The learning rate warms up linearly over 50 steps to a peak of $1 \times10^{-5}$ and then follows a cosine schedule down to $1 \times10^{-7}$. The peak learning rate was selected once by a four-point sweep on an early data release and then held fixed for every subsequent run, so that differences between runs reflect differences in data alone. Each run trains for a single epoch over its data. Master weights are kept in fp32, and training uses fully sharded data parallelism with activation checkpointing and FlashAttention-2.

Experiments

We compare ACE-Brain-0.6 with open embodied models of the same size, and with three closed models, GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon on seven public grounding benchmarks and on a robot-scene benchmark of our own (§ 5.1 and § 5.2). We then ask where the gains come from (§ 5.3), and how they grow with the training data (§ 5.4).

Setup

Benchmarks. We evaluate on seven public benchmarks that together cover the capabilities analyzed in § 3: referring under spatial constraints and reasoning (RefSpatial [22], PointArena [23]), multi-instance perception (PixMo-Count [16] and the counting subset of PointArena), spatial relations and free-space localization (RoboSpatial [24], Where2Place [19]), and functional and interaction grounding (RoboAfford [25] and the affordance split of ShareRobot [26]). Every answer is a point, a count, or a yes/no token. We report the official metric of each benchmark, the mean over all seven (Avg-7). We also evaluate on RobotGrounding, our own benchmark of 2,591 questions in 14 types on held-out robot manipulation frames with human-annotated boxes, ranging from referring and counting to absent targets and cross-view correspondence between two cameras; its score is the macro average over question types.

Protocol. All models are evaluated with the same harness and the same prompt per benchmark, at temperature 0.2 with a 128-token output budget; PixMo-Count, whose answers can list many points, uses a 1,024-token budget. Closed models are queried through their APIs with the same questions and scorer.

Main Results

Compared models. We compare against Qwen3.5-9B [6]; a general-purpose open VLM of the same size, Qwen3-VL-8B [31]; open embodied models of the same scale, RynnBrain-8B [14], RoboBrain 2.5-8B [32], ACE-Brain-0.5-8B [33], Embodied-R1.5-8B [15], SenseNova-SI-1.1-8B [34], and PhysBrain 1.5-8B [35]; and five closed models, Claude Opus 5.5 [27], Gemini 4 Argon [28], GPT-5.5 [29], GPT-5.6-sol [30] and GPT-6 Astra [1], the reference analyzed in § 3.

Table 1 gives the full comparison, and Figure 5 details the results on RobotGrounding. Three results stand out: ACE-Brain-0.6 is the strongest open model of its size, it matches GPT-6 Astra on the seven-benchmark average while still trailing clearly on RefSpatial, and it leads every compared model in robot scenes.

Model	Where2Place	PixMo-Count	RoboSpatial	RefSpatial	PointArena	RoboAfford	ShareRobot	Avg-7	RobotGrounding
Closed models
Claude Opus 5.5 [27]	66.44	83.74	73.30	71.10	61.20	75.10	40.20	67.30	90.89
Gemini 4 Argon [28]	71.20	87.71	80.94	51.56	66.40	71.48	29.53	65.55	78.51
GPT-5.5 [29]	45.25	82.61	59.19	60.29	68.02	68.04	38.57	60.28	83.96
GPT-5.6-sol [30]	46.23	83.55	64.34	54.51	69.76	71.45	36.66	60.93	85.18
GPT-6 Astra [1]	71.10	88.10	69.55	80.14	75.36	84.94	46.60	73.68	91.10
General-purpose open models
Qwen3-VL-8B [31]	62.60	86.40	62.66	54.15	61.81	74.83	35.80	62.61	92.02
Qwen3.5-9B [6]	72.00	90.35	53.46	57.76	68.33	78.65	39.47	65.72	86.70
Open embodied models
RynnBrain-8B [14]	68.00	11.70	70.00	56.68	65.58	79.29	15.50	52.39	80.14
RoboBrain 2.5-8B [32]	64.00	78.80	63.71	59.21	59.27	76.33	56.24	65.37	81.35
ACE-Brain-0.5-8B [33]	70.70	85.40	58.30	53.43	68.02	75.37	34.38	63.66	66.91
Embodied-R1.5-8B [15]	77.00	81.70	68.96	57.04	71.38	83.43	37.21	68.10	75.24
SenseNova-SI-1.1-8B [34]	47.00	59.20	59.65	29.78	51.83	50.71	25.98	46.31	66.83
PhysBrain 1.5-8B [35]	76.30	85.63	69.74	55.60	69.86	81.24	33.47	67.41	87.05
ACE-Brain-0.6 (ours)	81.44	86.96	76.56	71.84	74.13	88.14	48.82	75.41	93.42

Fine-tuning improves the base model on six of the seven benchmarks and raises Avg-7 by almost ten points. The largest gains appear where grounding requires a decision rather than a glance, on the spatial relations of RoboSpatial and the constrained references of RefSpatial, whereas on counting, which the base model already handles well, fine-tuning brings no gain and costs a few points.

Among open embodied models of the same scale, ACE-Brain-0.6 is the best on every benchmark except ShareRobot and leads the next model on Avg-7 by more than seven points. Against GPT-6 Astra the picture is mixed: our model is ahead on four benchmarks, slightly behind on PixMo-Count and PointArena, and well behind on RefSpatial (71.8 vs. 80.1), so the two averages end up close (Avg-7 75.4 vs. 73.7). The other four closed models trail ACE-Brain-0.6 on at least five of the seven benchmarks; only on RoboSpatial is Gemini 4 Argon ahead of every model. On RobotGrounding, ACE-Brain-0.6 is ahead of all compared models, including the five closed ones. § 5.3 relates these differences to the capabilities and limitations of Astra identified in § 3, using the labeled subsets of the public benchmarks (Table 2) and the question types of RobotGrounding.

Benchmark	Subset	Ours	Astra
RoboSpatial	Configuration (relation, yes/no)	83.74	86.2
	Compatibility (fits?, yes/no)	84.76	76.2
	Context (free point)	62.27	53.4
RefSpatial	Location	73.00	77.0
	Placement	75.00	84.0
PointArena	Spatial	85.64	88.2
	Reasoning	75.13	56.3
	Counting (every instance)	63.27	60.0
	Affordable	92.42	86.7
RoboAfford	Object affordance (part)	91.94	84.7
	Object reference	92.98	92.8
	Spatial affordance (free region)	77.92	70.8

Where the Gains Come From

Placement under spatial constraints. Both models are accurate when a target is named by category, function, or attribute, and Astra is ahead on RefSpatial location, where the description adds a spatial constraint. The RefSpatial gap is larger in placement (Table 2), which tests the feasibility under spatial constraints discussed in § 3: the answer must satisfy a relation to one or more reference objects and lie in free space. A valid placement point also needs a supporting surface and enough clearance next to the reference. Our free-space targets mark unoccupied 2D regions and do not assert 3D feasibility (§ 4.2), so we expect the gap to require targets verified in 3D rather than more 2D free-space data. Consistent with this, adding several thousand placement questions with targets computed exactly from 2D object masks, a small share of the training data, left placement unchanged.

Parts and free space. Our model leads Astra on all three RoboAfford subsets, on ShareRobot affordance, on RoboSpatial, and on free-space pointing in Where2Place. On RoboSpatial the lead comes from the compatibility and context questions, while Astra remains slightly ahead on relation judgments (Table 2). Part-level targets such as “the handle” or “where to hold” belong to the fine-grained geometric localization that § 3 identified as a limitation of Astra. Across our data releases, the largest ShareRobot gain coincides with the addition of part masks selected by the action verb of the query, which suggests that this supervision closed the gap to Astra.

Robot scenes. ACE-Brain-0.6 leads every compared model on RobotGrounding Benchmark (Figure 5(a)). Most of the gain over the base model comes from three abilities it largely lacks: recognizing that a named object is absent, recognizing that no object has the requested attribute, and matching an object marked in one camera to the same object in another (Figure 5(b)). Our model is far ahead of Astra on the first two and on par with it on the third, while Astra stays ahead when every instance must be pointed to or the target is defined by a relation. The two models also err in opposite directions: Astra often reports objects that are absent, whereas ours more often misses objects that are present.

Data Scaling

The training collection of § 4.2 was assembled in a sequence of releases, from 0.26M to 1.55M samples, each adding new task families or image domains to the previous one. Every release was trained from the base model with the same recipe (§ 4.3), so the resulting checkpoints form a controlled series in which only the data differs. Figure 6 plots them against the number of training samples.

Performance grows steadily with data and passes Astra’s average. Over a roughly sixfold range of training data, from 0.26M to 1.55M samples, Avg-7 rises from 69.3 to 75.4 along a log-linear trend of about +2.4 points per doubling ($r = 0.99$); counting the base model, the full span is 65.7 to 75.4. The 1.55M release continues the trend, with no sign of saturation within the range we tested.

What scales is coverage, not repetition. Each release added new kinds of supervision rather than more of the same, so in the main series the number of samples and the coverage of supervision grow together and cannot be fully separated. A single control run separates them: it doubles only the sources of an early release, reaching the size of the next release without any new task or domain, and gains less than one point over its starting point, landing within one point of the release that added new sources instead. The same pattern holds inside the sequence. Adding more samples whose answers list many points did not improve PointArena questions with many instances. For a fixed backbone and recipe, then, what moves grounding is the number of distinct kinds of supervision and how their answers are represented, not the number of examples.

Conclusions

We asked how far training data alone can carry an open 9B vision-language model toward the grounding capabilities of GPT-6 Astra. By collecting open robot, egocentric, object-centric, and spatial data, annotating the robot recordings automatically, and converting every source into question–answer supervision that can be verified against its evidence, we trained ACE-Brain-0.6 from Qwen3.5-9B with a standard single-epoch recipe. The model is ahead of Astra on four of seven public grounding benchmarks, on par on two, and above it on their average, and reaches 93.4 on our robot-scene benchmark against Astra’s 91.1. A controlled series of training runs shows that grounding performance rises log-linearly with the number of training samples and that the rise is driven by the coverage of supervision.

Two limitations remain, and they mark the direction of the next release. ACE-Brain-0.6 still trails Astra when every instance of a set must be enumerated, and on placement under spatial constraints, where our two-dimensional free-space supervision does not capture three-dimensional feasibility. Both are properties of the data, and the scaling results suggest that the remedy is new supervision, complete multi-instance targets and feasibility-aware placement data, rather than more of what the collection already contains. These are the next additions to the pipeline.

Appendix

Contributions and Author List

Contributors: Yinhe Wang$^{1,*}$, Tao Tao$^{1,2,*}$, Zexin He$^{1,3}$, Shuang Luo$^{1,3}$, Wei Mao$^{1}$, Yao Mu$^{5}$, Feng Pan$^{1}$, Hanming Deng$^{4}$, Lewei Lu$^{4}$.

Advisors: Zhi Hou$^{1}$, Xingang Pan$^{3}$, Dahua Lin$^{4}$, Dacheng Tao$^{3}$, Xiaogang Wang$^{1}$.

Affiliations: $^{1}$ACE Robotics $^{2}$Sichuan University $^{3}$Nanyang Technological University $^{4}$SenseTime Research $^{5}$Shanghai Jiao Tong University

• Equal contribution.

The RobotGrounding Benchmark

RobotGrounding is our own evaluation set for grounding in the scenes a robot policy observes. It contains 2,591 questions over held-out frames from robot manipulation recordings, with object boxes annotated by hand and checked question by question; the episodes these frames come from are excluded from training, together with every other evaluation benchmark (§ 4.2). The questions follow the task families of § 4.1: referring to an object by name, attribute, ordinal position, relation, or task role, answered with a box or a point; pointing to small objects that occupy well under one percent of the image; pointing to every instance of a described set; counting with an integer answer; existence questions whose correct answer is none when the named object, or an object with the named attribute, is absent, paired with positive questions on objects that are present; spatial yes/no judgments; and cross-view correspondence, in which an object is marked by a box in one camera and must be located in a simultaneous view from another camera, without being named. Negative and positive questions are balanced within each type so that always answering none does not score. Figure 7 shows one example of each question type and Figure 8 the composition of the benchmark.

A point answer is correct when it contains at least one point and every point lies inside or on the boundary of the target’s annotated bounding box; an every-instance answer must contain exactly as many points as there are instances, each inside or on the boundary of a different annotated instance box; a box is correct at an intersection-over-union of at least 0.5; a count or a categorical answer must match exactly; and an absent-target question is answered correctly by any reply that contains no point. Because the reference regions are boxes rather than masks, the point criterion is lenient for thin or hollow objects. The overall score is the unweighted mean over 14 scoring groups, so that the many referring questions do not dominate the fewer, harder ones: referring by name with a point; referring with a box at IoU $\ge0.5$ and, as a separate group, at IoU $\ge0.75$; referring by attribute; by ordinal position or relation, merged into one group; by task role; placement locations; spatial yes/no judgments; pointing to every instance; existence; attribute existence; integer counting; small objects; and cross-view correspondence.