VoxProg: 3D Spatial Reasoning through Voxel Programming

Project homepage: voxprog.github.io

Introduction

Natural-language spatial instructions often specify goals whose geometric requirements must be inferred from the scene [37, 3, 7]. Placing a bottle inside an open display holder requires identifying the holder’s receiving structure and determining how the bottle should be positioned and oriented to fit within it. Similarly, extending a row of cups requires inferring the direction and spacing of the existing arrangement to derive the next placement target. Here, the geometric references that define the goal are themselves outcomes of spatial computation over the 3D scene.

Existing approaches are limited by both the task requirements their interfaces can express and the geometric computations their representations support. Predefined task interfaces couple task coverage to task-specific implementations, limiting generalization to new spatial requirements [13, 20, 22]. Geometric representations present a different difficulty: sparse keypoints can leave extended contact regions and local surface shapes under-specified [21, 17, 30, 39, 9], while raw meshes require programs to organize irregular surface elements into the regions and references a task needs. Open-ended spatial reasoning therefore requires both programmatic computation that adapts to free-form instructions and a structured 3D surface representation that exposes scene geometry directly to code.

To this end, we formulate voxel programming, in which a language model expresses its understanding of a spatial task as a program that constructs and operates on local voxel views (Figure 1). By discretizing object surfaces onto regular grids at task-appropriate resolutions, local voxel views expose spatial coordinates and neighborhood relations directly as array operations. Generated programs can index these grids to isolate contact regions, extract boundaries, and measure local extent and orientation, turning a uniform grid representation into task-specific geometric references for 6-DoF pose computation.

VoxProg realizes this formulation through a two-stage programming pipeline grounded by visual anchors and execution feedback. Starting from the instruction and raycast visual anchors, the model first writes an evidence program to construct the surface regions required by the task, such as a dish rack’s dividers and bottom support rails, and to specify the spatial relations they must satisfy. Conditioned on this constructed evidence, a pose program then queries the selected geometry to compute the moved object’s target pose. At both stages, numerical diagnostics and rendered views from program execution allow the model to verify intermediate geometry, revise faulty selections, and iteratively refine its generated code.

We evaluate VOXPROG on VoxPROG-Bench, a new benchmark of 71 open-ended placement tasks where target poses depend on part-level affordances, 3D shape compatibility, and surrounding layouts. Because valid placements must satisfy both semantic intent and physical plausibility, our evaluation suite pairs a perceptual ranking metric, IA Rank Score, with 3D Physical Validity checks for static support and object interpenetration. IA Rank Score measures how consistently a method’s rendered result outranks competitors in similarity to the ground-truth arrangement. Across VoxPROG-Bench, VOXPROG substantially outperforms relation-based, keypoint-based, and mesh-based baselines, achieving its largest margin on geometric tasks where object contours govern fit and clearance. Ablation studies further show that constructing explicit geometric evidence before pose synthesis improves both physical validity and intent alignment over direct mesh programming, while local voxel views further improve intent alignment across all 71 tasks.

Contributions. ❶ We formulate voxel programming and develop VOXPROG, a two-stage system that constructs task-relevant surface geometry via local voxel views to compute 3D object placements. ❷ We introduce VoxPROG-Bench, a benchmark of 71 language-guided 3D placement tasks spanning functional, geometric, and contextual arrangements. ❸ We propose a complementary evaluation protocol combining perceptual intent ranking (IA Rank Score) with 3D geometric verification (Physical Validity).

Related Work

Spatial Task Specification. Language-guided placement has been studied through layout prediction, rearrangement search, and geometric optimization. Scene graphs encode inter-object relations for scene synthesis [4, 46, 26, 16], while scene-layout and object-placement methods map language to spatial configurations [10, 45, 33, 38, 18, 1, 47, 23]. Rearrangement methods coordinate multi-object structures using learned spatial priors [28, 29, 41, 43, 44] or pattern-guided search [2]. At a finer scale, object manipulation methods formulate 6-DoF pose optimization over sparse semantic keypoints, directional axes, and part-level affordances ([30, 17, 21, 35, 34, 48]).

While layout, surface-constraint, and keypoint formulations parameterize placement over bounding boxes, planar support patches, or sparse point and axis primitives, open-ended tasks often hinge on extended surface structures (such as receiving slots, angled ledges, curved rims, or recessed compartments) whose relevant regions depend on the instruction and paired objects. VOXPROG therefore treats geometric reference construction as an explicit, programmable stage of task specification: generated code composes spatial selections and neighborhood operations over local voxel views, retaining both the selected voxel regions and their clipped source-mesh patches for downstream pose measurement so that the same 3D object geometry can be segmented and parameterized differently across placement tasks.

Language Models for Spatial Computation. Executable programs let language models delegate quantitative calculation and visual queries to external interpreters ([12, 5, 14, 40, 15]). In robotics, generated code sequences task plans ([36]), composes perception calls with control primitives ([25, 31, 24]), and synthesizes 3D value maps over workspace grids for motion planning ([19]). Recent methods further incorporate geometric processing via part-mask refinement ([42]), primitive fitting ([27]), and frame-grounded tool calls ([6]).

In most programmatic frameworks, however, 3D perception remains encapsulated behind fixed API calls that return object centroids, bounding boxes, or pre-segmented masks ([25, 31]), or operates on coarse workspace grids to score trajectories ([19]). We develop voxel programming so that generated code can inspect and dissect 3D object surfaces directly at the part level. By discretizing candidate objects into local voxel views at task-chosen resolutions, VOXPROG turns 3D surface inspection into regular-grid array programming while retaining exact correspondence with the underlying mesh triangles. Generated code can thus construct custom surface witnesses on demand, verify them through visual and numerical execution feedback, and measure both discrete grid bounds and clipped source-mesh patches to solve for precise, physically valid 6-DoF object placements.

Method

Given a natural-language instruction $\mathcal{I}$ (e.g., “Place the single plate in an open slot of the empty dish rack with its rim supported”) and a 3D scene $\mathcal{S} = \{o_n\}_{n=1}^{N}$ with known object meshes and initial poses, VOXPROG synthesizes and executes Python programs to compute a target placement pose $T \in SE(3)$. Rather than mapping language directly to 6-DoF coordinates in a single step, VOXPROG decouples geometric reference construction from pose calculation: an evidence program $P_e$ isolates task-relevant surface regions on the moved and reference objects, and a pose program $P_p$ measures these constructed surface regions and their relational constraints to solve for the moved object’s final rigid transformation.

Figure 2 illustrates this pipeline on an upright plate-placement task. The model first organizes the required spatial references into a directed evidence graph and grounds visible cues into 3D surface anchors (§ 3.1). Operating on local voxel views of scene meshes (§ 3.2), the evidence program constructs the rack’s adjacent dividers, lower support rails, and slot region while refining its selections through visual and geometric selection checks and program revision (§ 3.3). The verified evidence record and relational claims are then passed to the pose program, which queries both discrete voxel bounds and clipped source-mesh patches to align, center, and seat the plate upright within the target receiving slot (§ 3.4).

Task Interpretation and Visual Grounding

An LLM formalizes its understanding of the instruction as a directed evidence graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$. Each node $v \in\mathcal{V}$ specifies either a visible semantic anchor or a derived geometric region to construct, along with its functional role in the task; each directed edge $(u,v) \in\mathcal{E}$ indicates that constructing $v$ depends on the location or geometry of $u$. In Figure 2, visual cues on the plate and dish rack initiate a dependency chain that extracts the plate’s outer rim alongside the rack’s adjacent dividers, support rails, and slot region.

Each node in $\mathcal{G}$ is routed according to how its geometry can be acquired. For a directly visible semantic cue, VOXPROG queries a VLM for a 2D pixel coordinate in the rendered scene image and casts a camera ray against the scene mesh to recover a 3D surface point as a spatial anchor in $\mathcal{A}$. References that cannot be reduced to a single clickable pixel, such as an interior slot region, a recessed support ledge, or an occluded underside, are passed to the evidence program together with their descriptions and upstream 3D anchors, which localize where on the object mesh the program should inspect and slice the surface.

Voxel Programming

Generated programs operate on scene geometry through local voxel views. Given a source mesh $s$ and voxel size $h > 0$, $U = \operatorname{VoxelView}(s,h)$ discretizes the surface of $s$ onto a regular 3D grid at resolution $h$. On this grid, the view exposes occupied surface cells as integer coordinate arrays $(N, 3)$ alongside directional exposure masks and connected-component queries, while a rigid grid-to-world transform maps selected voxel indices and local bounds back to continuous 3D coordinates for downstream geometric computation.

Each constructed geometric reference forms a witness in the evidence record $\mathcal{R}$. To construct a witness $v$, the evidence program instantiates a local view $U_v$ on source geometry $s_v$ at resolution $h_v$ and evaluates a programmatic binary mask $m_v$ over its surface cells to obtain the selected surface cells $W_v$:

Because $U_v$ is a regular grid, the program can compose $m_v$ through standard array operations (axis-aligned slicing, proximity filters around visual anchors or upstream witnesses, directional exposure queries, and connected-component labeling) to isolate witnesses such as the rack’s adjacent dividers $D$, $E$, lower support rails $S$, and combined slot region $B$ in Figure 2. For each witness $v$, the entry $\mathcal{R}.v$ in $\mathcal{R}$ stores these selected cells $W_v$, their parent view $U_v$, the source object $s_v$, and the provenance edges from $\mathcal{G}$.

Parameterizing the voxel size $h_v$ per view allows the evidence program to match discretization scale to local geometric structure without voxelizing the entire 3D scene at a uniform fine resolution. A program can use coarse grids to localize broad object surfaces and switch to a finer resolution $h_v$ when resolving thin wire dividers, narrow receiving slots, or shallow rims that would merge on a coarse grid, while retaining exact geometric correspondence with the underlying triangles of the source object mesh.

Evidence Program Generation and Refinement

Conditioned on the instruction, evidence graph $\mathcal{G}$, scene $\mathcal{S}$, and grounded anchors $\mathcal{A}$, the LLM generates an executable evidence program $P_e$ following the topological order of $\mathcal{G}$:

where $\mathcal{R}$ stores the constructed witnesses and $\mathcal{C}$ specifies the target spatial relations among them (e.g., placing the plate’s rim in slot $B$ between dividers $D$, $E$ while resting on support rails $S$ in Figure 2). Following the dependencies in $\mathcal{G}$ allows downstream witnesses to be defined relative to earlier selections.

To tie each witness to its functional role before writing code, the model first states explicit geometric inclusion and exclusion criteria for each target witness (such as excluding adjacent rack loops or vertical side walls). When a structural question remains unresolved, the model can execute lightweight probes via the same voxel API to print numerical statistics (such as voxel counts, component counts, and coordinate spans) and inspect rendered diagnostic views of intermediate voxel selections.

Once a complete candidate $P_{\mathrm{e}}$ is executed, geometric checks and a VLM critic inspect each constructed witness for part identity, region completeness, and the absence of unwanted geometry (Figure 2, center). When a candidate selection leaks into adjacent structure or remains incomplete, this feedback guides the model to revise its programmatic selection rules or adjust the voxel resolution $h_{\mathrm{v}}$ and re-run $P_{\mathrm{e}}$.

Pose Program Generation and Refinement

Given the verified evidence record $\mathcal{R}$ and relational specification $\mathcal{C}$, the LLM generates an executable pose program $P_{\mathrm{p}}$ to compute the moved object’s target rigid transformation:

where $T_{\mathrm{cur}}$ is the moved object’s current world-frame pose (initialized at $T_0$) and $\mathcal{S}$ provides the scene meshes and initial poses. Every witness enters $P_{\mathrm{p}}$ as concrete geometry rather than pre-fitted point or vector constants, so $P_{\mathrm{p}}$ derives all required positions, axes, and offsets directly from $\mathcal{R}$ under $\mathcal{C}$.

Beyond discrete grid coordinates, each witness entry $\mathcal{R}.v$ in $\mathcal{R}$ also exposes the continuous source-mesh patch clipped to its selected voxels $W_v$, preserving exact face correspondence and outward surface normals. As shown in Figure 2 (step 3), $P_{\mathrm{p}}$ combines discrete grid bounds with continuous mesh surfaces to solve accurately for the target pose $T$ on the original object geometry, computing the divider plane normal $\mathbf{n}$ from $\mathcal{R}.D$, the slot mid-span $x$ from $\mathcal{R}.B$, and the support depth and height $y$, $z$ from $\mathcal{R}.S$.

Because all witnesses in $\mathcal{R}$ remain fixed in world coordinates at the initial scene state, $P_{\mathrm{p}}$ maps any initial world-space point $\mathbf{x}$ on the moved object (at initial pose $T_0$) to its position under a candidate pose $T$ as

where $\tilde{\mathbf{x}}$ denotes homogeneous coordinates. After each execution of $P_{\mathrm{p}}$, the candidate pose $T$ updates $T_{\mathrm{cur}}$ in the 3D scene while $\mathcal{R}$ stays recorded in the initial scene state without recomputing evidence geometry, and the model receives a rendered observation of the resulting candidate placement together with numerical contact and clearance diagnostics to guide iterative revision of the pose program $P_{\mathrm{p}}$.

Experiments

We evaluate VoxProg on VoxProg-Bench against relation-, keypoint-, and mesh-based baselines (§ 4.1–§ 4.2), ablate staged reference construction and local voxel views alongside external tasks from RoboCasa and Open6DOR v2 (§ 4.3), and analyze physical and perceptual failure modes (§ 4.4).

Experimental Setup

Benchmark. We evaluate on VoxProg-Bench, a benchmark of 71 individually curated language-guided 3D placement tasks. Each case presents an initial tabletop scene and a free-form instruction specifying the desired placement of an object. We emphasize diverse instructions that call for fine-grained reasoning about functional object parts, geometric fit, and contextual arrangements. Although an instruction can involve multiple spatial cues, we assign each case to one category according to its primary spatial requirement:

• Functional (28 cases): The placement must support a specific use, such as placing a cup beneath a grinder chute to catch grounds, returning headphones to a stand, or orienting a funnel for pouring.

• Geometric (14 cases): The placement depends directly on object contours, such as nesting an oval bowl inside a matching container or fitting a baking tray into a narrow box at an angle.

• Contextual (29 cases): The placement takes its spatial cues from the surrounding multi-object arrangement, such as completing a missing place setting or restoring a symmetric tabletop layout.

Scenes are assembled from generated 3D assets through randomized placement and manual refinement. Detailed instructions, object geometry, and scene context make the intended arrangement well defined, with one manually annotated ground-truth 6-DoF pose per task. All methods receive the scene meshes, initial poses, instruction, and target-object identity; ground-truth poses and renderings serve only for evaluation.

Evaluation Suite. We design an evaluation suite around two complementary requirements of object placement: realizing the intended spatial arrangement and maintaining physical plausibility. We evaluate Intent Alignment via visual comparison with ground truth and Physical Validity via 3D geometric checks.

Intent Alignment measures perceptual similarity using DreamSim (Fu et al., [11]) between full-scene renderings of the predicted and ground-truth placements under matched cameras and lighting. Within each case, we rank the $M$ compared methods by increasing DreamSim distance and assign rank $r$ the score $100(M-r)/(M-1)$, using average ranks for ties. Averaging over cases gives the $IA Rank Score$ (0–100; higher is better), which measures how consistently a method outranks its competitors across tasks. Each task contributes equally; scores are compared within the same method pool.

Physical Validity assesses whether the final placement establishes plausible static support and avoids interpenetration with surrounding scene objects. We report the percentage of evaluated task placements whose final 3D meshes satisfy both static support-contact and collision-clearance checks.

Baselines. We compare VoxProg against three language-guided placement approaches. GenManip (Gao et al., [13]) constructs placements using predefined spatial relations between target and anchor objects. GenSim2-KPAM (Hua et al., [17]; Manuelli et al., [30]) specifies placement targets through predicted semantic keypoints together with geometric point and axis constraints. Direct Mesh generates Blender Python code for placement and executes it through Blender-MCP².

Model Configuration. All compared methods use GPT-5.6-Sol under a shared inference configuration for both language-guided code generation and visual grounding across all evaluated tasks.

Results on the Task Suite

Quantitative Results. VoxProg achieves the highest IA Rank Score overall (83.10) and across all three task groups, alongside the highest Physical Validity at 95.8% (Table 1), outperforming Direct Mesh (48.59 / 87.3%), GenSim2-KPAM (42.72 / 77.5%), and GenManip (25.59 / 73.2%). Although Direct Mesh also writes task-specific Python code over explicit scene meshes, VoxProg improves its IA Rank Score by 34.51 points and Physical Validity by 8.5 points: code generation alone remains brittle without an intermediate representation that isolates the surfaces relevant to the instruction.

Method	IA Rank Score (%) ↑ Overall	IA Rank Score (%) ↑ Func.	IA Rank Score (%) ↑ Geom.	IA Rank Score (%) ↑ Context.	Physical Validity (%) ↑ Overall	Physical Validity (%) ↑ Func.	Physical Validity (%) ↑ Geom.	Physical Validity (%) ↑ Context.
GenManip	25.59	25.00	32.14	22.99	73.2	67.9	92.9	69.0
GenSim2-KPAM	42.72	34.52	35.71	54.02	77.5	78.6	85.7	72.4
Direct Mesh	48.59	53.57	44.05	45.98	87.3	75.0	100.0	93.1
VoxProg	83.10	86.90	88.10	77.01	95.8	92.9	100.0	96.6

Breaking down performance by category reveals how different representations behave under distinct spatial demands. On Functional tasks, VoxProg reaches 86.90 versus 53.57 for Direct Mesh and 34.52 for GenSim2-KPAM (with Physical Validity rising from 75.0% and 78.6% to 92.9%), as functional use requires isolating mating parts, such as a chute outlet or headphone arch, rather than whole-object bounding boxes or sparse keypoints. On Contextual tasks, the strongest baseline shifts from Direct Mesh (45.98) to GenSim2-KPAM (54.02), while VoxProg retains a clear lead at 77.01. The largest margin occurs on Geometric tasks: VoxProg reaches an IA Rank Score of 88.10 versus 44.05 for Direct Mesh, although both attain 100.0% Physical Validity. The difference lies in resolving the local contours and clearances that distinguish the intended fit from other physically valid configurations.

Qualitative Results. Figure 3 exposes the gap between locating an object and identifying its functional contact region. In the headphone case (b), all three baselines bring the earcups against the stand or wrap the headphones around its post, missing the inner headband support specified by the instruction. VoxProg matches the inner arch to the curved crown, establishing the intended part-level contact.

On the sloped stand (a), GENMANIP leaves the sneaker above the support and GENSIM2-KPAM turns it across the ramp. VOXPROG aligns the sole with the incline and keeps the toe facing forward. For the VR headset (d), GENMANIP and GENSIM2-KPAM leave the headset above the cradle or in front of the stand, while VOXPROG positions it over the stand with the headband behind the front shell. Explicit surface references connect these local contact and orientation requirements to pose computation.

Ablation and External Evaluation

Effect of Voxel Programming. To separate the effect of staged reference construction from that of local voxel views, we compare DIRECT MESH, MESH EVIDENCE, and VOXPROG across all 71 tasks in Table 2. DIRECT MESH writes a single placement program directly over the scene meshes; MESH EVIDENCE shares VOXPROG’s two-stage pipeline and execution feedback but constructs witness regions through direct mesh operations on vertices and faces; VOXPROG constructs witnesses on local voxel views while retaining access to the clipped source-mesh surfaces during pose computation.

Method	IA Rank Score (%) ↑	Physical Validity (%) ↑
DIRECT MESH	26.76	87.3
MESH EVIDENCE	56.34	95.8
VOXPROG	66.90	95.8

Within this three-method comparison, moving from single-stage DIRECT MESH to the staged MESH EVIDENCE pipeline raises Physical Validity from 87.3% to 95.8% and more than doubles the IA Rank Score from 26.76 to 56.34. When a single program must simultaneously locate task-relevant surface geometry and compute 6-DoF coordinates over raw scene meshes, imprecise region selection directly degrades the resulting placement pose; introducing an explicit evidence stage allows candidate surface regions to be constructed, verified, and revisited through execution feedback before downstream pose synthesis.

Replacing direct mesh selection with local voxel views raises the IA Rank Score from 56.34 to 66.90 while maintaining 95.8% Physical Validity. Regular grid indexing and neighborhood operations help resolve the intended receiving side of a structure. In Figure 4, MESH EVIDENCE places the tablet behind the stand’s supports, which obstruct its screen, whereas VOXPROG seats it in front with the display unobstructed and facing the user.

External Placement Tasks. We evaluate generalization on 19 tasks adapted from RoboCasa [32] and 13 from Open6DOR v2 [8, 35], formulated as final-state 3D placement problems under matched inputs Figure 5. On RoboCasa, where appliances and cabinets introduce semi-enclosed boundaries, VOXPROG achieves an IA Rank Score of 87.72 against 26.32–45.61 for the baselines, while maintaining 94.7% Physical Validity compared with 26.3%–42.1% for the baselines. On Open6DOR v2, VOXPROG leads in IA Rank Score (61.54 vs. 57.69 for DIRECT MESH) and raises Physical Validity to 92.3% versus 46.2% for DIRECT MESH: while broad directional instructions admit many 2D-plausible overhead positions, the baselines often leave objects floating or interpenetrating in 3D.

In the mug task (Figure 6a), GENMANIP leaves the mug beside the dispenser, while GENSIM2-KPAM and DIRECT MESH leave it interpenetrating the drip tray. VOXPROG centers the opening beneath the nozzle and rests the base on the tray, jointly satisfying the alignment and support required to receive coffee. The SD-card task (c) exposes a complementary contact failure: GENMANIP and DIRECT MESH leave the card floating above the USB drive, whereas VOXPROG seats it on the top surface.

Failure Analysis

Physical Failure Profiles. Figure 7a decomposes physical rejections on VoxProg-Bench into support and interpenetration failures. GENMANIP incurs 19 rejections (11 support, 8 intersection), versus 16 for GENSIM2-KPAM (13 intersection, 3 support). These patterns are consistent with coarse relation offsets leaving support contact under-specified and sparse point-and-axis constraints leaving surface clearance under-constrained. DIRECT MESH has 9 rejections (6 intersection, 3 support), while both staged variants cut rejections to 3 (3 intersection for MESH EVIDENCE; 2 intersection, 1 support for VOXPROG).

Visually Ambiguous Contact. The pot-lid case in Figure 7b illustrates why 2D visual inspection alone cannot establish physical validity in 3D object placement. From the primary camera view, the lid appears properly seated: it stands upright with its handle facing outward, its circular contour centered between the side supports, and its lower rim aligned over the receiving rack. A side view of the same final pose reveals, however, that the lid remains suspended above the rack, separated by a clear vertical gap from both the base tray and the inclined backrest. Here, the rack’s opaque front frame provides a strong visual cue to the destination while occluding the recessed region where support contact should occur. Because 2D projection overlap along the viewing ray can conceal 3D separation under self-occlusion, a single-view visual comparison can accept a physically unsupported placement even when the target object and its apparent relation to the reference structure look aligned in image space. Pairing perceptual evaluation with 3D geometric verification addresses this ambiguity by checking local 3D support contact and interpenetration conditions between the placed object and surrounding scene meshes, making the distinction between apparent 2D seating in the rendered camera view and actual physical support in the 3D scene explicit.

Conclusion

We presented VOXPROG, a framework for 3D spatial reasoning that formulates language-guided object placement through voxel programming. Rather than parameterizing tasks over fixed spatial relations or sparse keypoint primitives, VOXPROG treats the geometric references required by an instruction as intermediate surface structures constructed by executable code. By coupling regular-grid selection over local voxel views with exact surface normals and patches from the underlying meshes, our two-stage pipeline enables a language model to isolate functional contact regions, verify them through visual and geometric feedback, and solve for physically valid $SE(3)$ poses. Across the 71-task VoxPROG-Bench benchmark and external tasks from RoboCasa and Open6DOR v2, VoxPROG consistently outperforms relation-based, keypoint-based, and mesh-based baselines in both intent alignment and physical validity. Coupling such programmable surface witnesses with downstream motion generation provides a natural direction for extending fine-grained geometric reasoning from target placement specification to full manipulation execution.