betaFeedback

WorksComputer Science

Isolating Perception for Multi-View Planning under Occlusion-Induced Partial Observability

Abstract

Vision-language models (VLMs) are beginning to saturate visual planning benchmarks, yet they fail to resolve depth ambiguity and degrade sharply with problem size under occlusion-induced partial observability (PO). We introduce BW-PO based on the Blocksworld domain, a controlled benchmark that reduces existing VLM planners and formalizers to near-zero performance, and propose VLM-as-perceiver, a paradigm that narrows the VLM's role to local perception while delegating cross-view evidence reconciliation and plan generation to external solvers. Across model-specific challenging subsets of BW-PO, this raises plan success from 1% to 97% for Gemma-4-31B, 0% to 95% for Qwen3.5-122B-A10B, and 3% to 84% for Gemini-3.1-Pro. Isolating perception in this way separates two bottlenecks that are otherwise entangled: reconciliation under depth ambiguity, and completeness of visual extraction under scene density.

Introduction

In visual planning, vision-language models (VLMs) must jointly perform perception, state estimation, and action generation. This entanglement makes errors difficult to localize and the resulting plans are often plausible but inconsistent with environment dynamics [7]. An alternative is to use these models as formalizers, translating observations into formal representations, such as the Planning Domain Definition Language (PDDL), which classical planners can then solve deterministically [14].

The central bottleneck in this approach is therefore not action generation, but the accurate extraction of the objects, states, and relations required to formalize a planning problem. This challenge is particularly acute under multi-view partial observability (PO), where each observation provides incomplete evidence about the environment. Recent work has only begun to study planning under such conditions, particularly when partial observability arises from restricted local fields of view and an unexplored environment [11].

Perception research has followed a complementary path, isolating several challenges to accurate state extraction. These include reasoning about occluded objects [12, 19], resolving the depth ambiguity of which objects lie in front of others [17], and characterizing how perceptual accuracy degrades as the number of objects increases [13]. Because these studies consider one image at a time, they do not address the additional demands of multi-view state recovery: cross-view correspondences, viewpoint alignment, and the fusion of conflicting evidence. Current models remain weak on all three [18, 3].

This gap motivates our study of multi-view planning under occlusion-induced partial observability, where accurate state extraction requires both cross-view integration and robust perceptual reasoning. We introduce BW-PO, built on the widely used Blocksworld domain [8], the first benchmark to evaluate this setting along two orthogonal dimensions of difficulty: depth ambiguity and increasing scene complexity as the number of objects grows. Both existing approaches, VLM-as-planner and VLM-as-formalizer, degrade to near zero on its most challenging subsets. We therefore propose VLM-as-perceiver, which narrows the VLM’s role to perception while delegating cross-view evidence reconciliation to a constraint optimizer and plan generation to a classical planner, raising performance from 1% to 97% for Gemma-4-31B, 0% to 95% for Qwen3.5-122B-A10B, and 3% to 84% for Gemini-3.1-Pro, on model-specific challenging subsets. Because reconciliation becomes deterministic, we can also attribute failure to a specific stage along each axis: reconciliation under depth ambiguity, and completeness of visual extraction under scene density.

Benchmarks

Recent benchmarks such as ProDG (Shirai et al., [14]), BLOCKSWORLD-REAL (He et al., [4]), and ViPlan-BW (Merler et al., [10]) extend the text-based Blocksworld domain, which VLMs have since largely saturated. BW-PO instead comprises six datasets of 110 problems each, spanning two orthogonal difficulty dimensions: depth ambiguity in BW-PO-DEPTH and increasing object count NN in BW-PO-N.

Each BW-PO state ss is a front- and a side-view image; occlusion leaves each view revealing only part of the scene, so recovering ss requires correspondences across viewpoints and reconciliation of both images. A planning problem p=(sinit,sgoal)p = (s_{\mathrm{init}}, s_{\mathrm{goal}}) pairs an initial and a goal state, each given by its own front- and side-view images, and the task is to generate a valid sequence of actions transforming sinits_{\mathrm{init}} into sgoals_{\mathrm{goal}}. Both states are thus subject to partial observability.

BW-PO-DEPTH introduces depth ambiguity within each view: it is not apparent from a single image whether a block xx rests directly on a visible block yy or on another block occluded by yy. The alternative viewpoint supplies further evidence, but the two observations induce competing interpretations that must be resolved to recover the true state, as shown in Figure 1. This dataset therefore evaluates whether a method can integrate multi-view evidence while reasoning robustly about occlusion and depth.

Figure 1. A BW-PO-DEPTH problem where each view cannot individually determine whether the blue block rests on the red block or on the purple block that the red block occludes. In our proposed method (c), the conflicting local claims are reconciled by anchoring on the purple block that has no conflicting relations. Across (a)–(c), progressively more of the pipeline moves from stochastic to deterministic components.

BW-PO-NN, where N∈{6,12,20,40,100}N \in\{6, 12, 20, 40, 100\}, instead evaluates how performance scales with scene density, well beyond the six blocks of existing Blocksworld visual-planning benchmarks.

Methods

VLM-as-planner (Figure 1a). All three methods are given the object set. VLM-as-planner additionally receives the available actions and both states at once, each observed from two views, and handles the entire pipeline: it must perceive objects and relations, reconcile the views, and search for a valid action sequence. An error at any stage invalidates the plan, and the direct output gives little indication of whether the failure originated in perception, state estimation, or search.

VLM-as-formalizer (Figure 1b). The VLM is instead given the predicate vocabulary and invoked once per state, translating the two views into a grounded symbolic state to be solved by Fast Downward (Helmert, [5]). Cross-view reconciliation remains the model’s responsibility, so errors committed to the symbolic state before the planner is invoked cannot be corrected downstream.

VLM-as-perceiver (Figure 1c). The VLM is given neither actions nor predicates and is restricted to local perception within a single view; reconciliation, formalization, and planning are delegated to the deterministic Parser, Reconciler, and Planner stages. It is invoked once per state: for each view v∈V={front,side}v \in V = \{\text{front},\text{side}\} the VLM independently reports a set of vertical stacks Σv\Sigma_v, where each σ=(b1,…,bk)∈Σv\sigma= (b_1,\ldots,b_k) \in\Sigma_v lists blocks bottom to top. We write OO for the set of blocks reported in any view; unmentioned blocks are absent from the resulting state.

Parser. Each stack is decomposed into k−1k-1 relational claims, one per adjacent pair, and two empty-slot claims, for the cells below b1b_1 and above bkb_k. Claims are tallied rather than deduplicated, so a claim made by both views counts twice:

m(x,y)=∑v∣{σ∈Σv:y≺x in σ}∣,m(x,y)=\sum_v\left|\{\sigma\in\Sigma_v:y\prec x\text{ in }\sigma\}\right|,
e(x,below)=∑v∣{σ∈Σv:b1=x}∣,e(x,\text{below})=\sum_v\left|\{\sigma\in\Sigma_v:b_1=x\}\right|,
e(x,above)=∑v∣{σ∈Σv:bk=x}∣.e(x,\text{above})=\sum_v\left|\{\sigma\in\Sigma_v:b_k=x\}\right|.

where y≺xy \prec x means yy immediately precedes xx bottom-to-top.

Reconciler. The tallies define a weighted MaxSMT problem solved with Z3 Optimize de Moura and Bjørner [2]; ); Bjørner et al. [1]. Each x∈Ox \in O receives an integer stack index s(x)s(x) and height h(x)h(x), so that On⁡(x,y)≡s(x)=s(y)∧h(x)=h(y)+1\operatorname{On}(x,y) \equiv s(x)=s(y)\land h(x)=h(y)+1, subject to the single hard constraint that no cell is occupied twice,

∀x≠y:s(x)≠s(y)∨h(x)≠h(y).\forall x\ne y:\quad s(x)\ne s(y)\lor h(x)\ne h(y).

All claims are soft, with base weights w(x,y)=2m(x,y)w(x,y)=2m(x,y) and w0(x,d)=e(x,d)w_0(x,d)=e(x,d): an observed contact outweighs a conflicting claim that one of its cells is empty, so empty-slot evidence breaks ties without overriding an observed relation.

To resolve competing claims that base weights alone cannot separate, we privilege uncontested local structure. Writing U(x)={y:m(y,x)>0}U(x)=\{y:m(y,x)>0\} and D(x)={y:m(x,y)>0}D(x)=\{y:m(x,y)>0\} for the candidate blocks observed immediately above and below xx, and O+⊆OO^+\subseteq O for the blocks appearing in at least one relational claim, the reliable blocks are those with an unambiguous neighborhood,

R={x∈O+∣∣U(x)∣≤1∧∣D(x)∣≤1},R=\{x\in O^+\mid|U(x)|\le1\land|D(x)|\le1\},

and each reliable endpoint earns a bonus B=1+∑(x,y)w(x,y)B=1+\sum_{(x,y)}w(x,y), giving:

w^(x,y)=w(x,y)+B∣{x,y}∩R∣.\widehat{w}(x,y)=w(x,y)+B\left|\{x,y\}\cap R\right|.

A single bonus outweighs every relational claim combined, so a relation anchored by an undisputed block wins its cell regardless of the opposing evidence, while base weights still discriminate among relations with equal bonuses. Letting Empty⁡(x,d)\operatorname{Empty}(x,d) denote that the cell adjacent to xx in direction dd is unoccupied, the solver returns the placement (s∗,h∗)(s^*,h^*) maximizing

∑(x,y)∈Cw^(x,y)1[On⁡(x,y)]+∑(x,d)∈Ew0(x,d)1[Empty⁡(x,d)].\sum_{(x,y)\in C}\widehat{w}(x,y)\mathbf{1}[\operatorname{On}(x,y)] + \sum_{(x,d)\in E}w_0(x,d)\mathbf{1}[\operatorname{Empty}(x,d)].

The gripper is reconciled separately by a precedence rule qq over per-view votes, detailed in Appendix B. Because no gripper is visible in any of our scenes, this always yields handempty.

Planner. The placement maps deterministically to the target predicate vocabulary: on for a block with a selected support and ontable otherwise, clear for a block with no selected neighbor above, and handempty with holding according to qq. The state is then passed to the same classical planner used by VLM-as-formalizer.

Experiments

Setup. We evaluate each method with three models: Gemma-4-31B Team et al., [15], Qwen3.5-122B-A10B Team, [16], and Gemini-3.1-Pro. Our metric is plan success rate, the percentage of generated plans that are valid for the corresponding planning problem. Plans are validated with VAL ([6]) against the ground-truth domain and problem files. This avoids comparison against a single reference plan, since multiple action sequences may correctly transform the initial state into the goal state, and avoids direct comparison against the ground-truth problem file, since different symbolic formalizations can represent equivalent states and support valid plans.

BW-PO-6 is saturated. Qwen3.5-122B-A10B reaches 100% plan success with all three methods and Gemini-3.1-Pro with both VLM-as-formalizer and VLM-as-perceiver (Table 1). Six-block scenes therefore discriminate little among current models, even after introducing occlusion-induced partial observability, motivating the harder conditions below.

DatasetGemma-4-31B PlannerGemma-4-31B FormalizerGemma-4-31B PerceiverQwen3.5-122B-A10B PlannerQwen3.5-122B-A10B FormalizerQwen3.5-122B-A10B PerceiverGemini-3.1-Pro PlannerGemini-3.1-Pro FormalizerGemini-3.1-Pro Perceiver
BW-PO-DEPTH1%0%97%0%0%95%73%71%100%
BW-PO-682%62%99%100%100%100%95%100%100%
BW-PO-1256%20%86%45%50%51%81%95%100%
BW-PO-202%2%22%0%0%6%29%93%100%
BW-PO-400%0%1%0%0%0%28%89%90%
BW-PO-1000%0%0%0%0%0%1%3%84%

Table 1. Plan success rates across all datasets, methods, and models. VLM-as-perceiver achieves or ties for the highest performance across all model–dataset pairs.

Depth ambiguity exposes failures in cross-view reconciliation. BW-PO-DEPTH produces competing interpretations across views, and only VLM-as-perceiver reconciles them reliably, solving 97% of instances with Gemma-4-31B, 95% with Qwen3.5-122B-A10B, and 100% with Gemini-3.1-Pro. Both existing approaches collapse on the open-weight models, with VLM-as-planner and VLM-as-formalizer reaching 1% and 0% with Gemma-4-31B and 0% with Qwen3.5-122B-A10B. The gap narrows with Gemini-3.1-Pro (73% and 71%) but VLM-as-perceiver stays ahead by nearly 30 points.

To locate the source of these failures, we label each block placement by whether the two views describe it consistently, using the per-view descriptions recovered from a VLM-as-perceiver run (implementation in Appendix § 10). Across these descriptions, the two views disagree on 31% of placements, and in every case the truth is one of the two competing claims, making each disagreement a binary choice at 50% chance. Because a valid plan requires both states to be recovered in full, a single incorrect support is enough to invalidate a problem, so per-block selection accuracy compounds sharply into plan success. Excluding blocks a method leaves unplaced, the two open-weight models select the correct view at or barely above chance under VLM-as-formalizer and VLM-as-planner (49.9%–61.4%). Gemini-3.1-Pro reconciles far better, at 92.9% and 91.4%. VLM-as-perceiver eliminates the gap entirely, reaching 99.8%–100% for all three models: an open-weight model under VLM-as-perceiver reconciles views more reliably than Gemini-3.1-Pro does under either existing method, despite the substantial difference in underlying capability. Since all three methods receive per-view descriptions of the same quality, the performance gap under depth ambiguity is attributable to cross-view reconciliation rather than perception.

Scaling exposes a recall bottleneck in visual state reconstruction. As BW-PO scales from 6 to 100 blocks, performance degrades because the VLM omits an increasing share of the scene. At 100 blocks, the open-weight models under VLM-as-perceiver retain high average precision but very low recall across the initial and goal states: 92.3% against 9.7% for Gemma-4-31B, and 77.1% against 5.1% for Qwen3.5-122B-A10B (complete metrics in Appendix Table 4). The relations that are represented are therefore usually correct, but most blocks are absent altogether, and since a valid plan requires both states in full, these omissions drive plan success to zero. The limiting factor in this regime is thus completeness of scene reconstruction rather than local spatial reasoning. Gemini-3.1-Pro sustains 98.9% recall at the same scale, indicating that what limits performance at this scale is the model’s visual extraction capacity.

Output token cost. Measured as generated tokens per successful plan, VLM-as-perceiver is the least expensive method in 11 of the 12 model–dataset cells where the comparison is defined, with a median reduction of 1.8×1.8 \times over the stronger baseline that widens to 44×44 \times on BW-PO-100. Full results in Appendix Table 2.

Conclusion

We introduced BW-PO and VLM-as-perceiver. VLM-as-perceiver matches or exceeds VLM-as-planner and VLM-as-formalizer on every model–dataset pair, at lower token cost in 11 of 12 cells where the comparison is defined. Under depth ambiguity it selects the correct view with nearly perfect accuracy, showing that the bottleneck there is reconciliation rather than perception. When reconciliation is deterministic at 100 blocks, the open-weight models recover only about 10% of placements while Gemini-3.1-Pro sustains 98.9% recall.

VLM-as-perceiver does not eliminate the visual-planning bottleneck; it shifts it. Solver-based reconciliation is effectively solved, but recall under dense scenes remains the key constraint separating frontier models from open-weight models on our hardest datasets.

Limitations

Our evaluation assumes access to the planning domain and the complete object inventory. This design isolates the capability targeted by BW-PO: recovering initial- and goal-state relations from partially observable visual inputs and using them to produce a valid plan. The benchmark therefore evaluates relational state extraction rather than open-world object discovery, domain induction, or the recovery of missing object identities.

Both BW-PO and VLM-as-perceiver are tied to the vertical-stack structure of Blocksworld: the benchmark induces partial observability through a specific mechanism, one block occluding another along the camera axis, and our reconciler encodes matching assumptions of discrete stack indices, direct vertical support, and mutually exclusive cell occupancy. The paradigm itself is not domain-specific, requiring only that local perceptual claims be expressible as weighted constraints, a condition met by any domain with a discrete relational state. We therefore expect it to transfer, though instantiating it elsewhere requires a richer constraint model whose cost we do not measure. Which bottleneck dominates in such domains is a separate question: our reconciliation result comes from the depth-ambiguity setting, while at high object counts the limiting factor is instead the completeness of the VLM’s scene reconstruction.

BW-PO problems are constructed by pairing photographed states, so each state serves as both an initial and a goal state across the dataset. The 110 problems within each dataset are consequently not statistically independent, and reported success rates should be read as performance over the benchmark instances rather than as estimates derived from 110 independent scene samples.

Scaling introduces a further confound. Increasing the number of blocks reduces the pixels available per object and increases the difficulty of reading their numerical identifiers. Degradation at larger scales may therefore reflect not only the greater number of objects and relations to be recovered, but also reduced visual resolution and increased recognition difficulty. Future variants could control object size and image resolution independently of scene complexity.

Dataset Construction

Problem generation. For each dataset we capture 11 distinct physical states, each photographed from a front and a side viewpoint, and form problems from all ordered pairs of distinct states. This yields P(11,2)=11×10=110P(11, 2) = 11 \times10 = 110 planning problems per dataset. Ordering matters, since (si,sj)(s_i, s_j) and (sj,si)(s_j, s_i) pose different problems, and every state serves as both an initial and a goal state across the dataset.

Difficulty filtering. In the six-block datasets the reachable configuration space is small enough that randomly paired states frequently admit trivial or near-trivial solutions. For these datasets we therefore capture additional states, discard every pair whose optimal ground-truth plan contains fewer than five actions, and sample 110 problems from those remaining, preserving a consistent dataset size across all of BW-PO.

Block identity. We retain the same six block colors at every value of NN, so colors are necessarily reused as the number of blocks grows. Each physical block is therefore marked with a unique numerical identifier, and blocks are referred to by the color–identifier pair throughout. Color alone is sufficient to name a block only in the six-block datasets.

Gripper Reconciliation

The gripper state is reconciled independently of the block placement. Each view contributes at most one vote

gv∈O∪{∅,⊥},g_v \in O \cup\{\emptyset, \perp\},

where ∅\emptyset denotes an empty gripper and ⊥\perp denotes abstention. Within a view, a report naming a held block takes precedence over a conflicting empty-hand statement, and a view that states only that the gripper is full, without identifying the held block, abstains. A named block is accepted when the other view agrees or abstains; conflicting named votes and all remaining cases default to an empty gripper:

g={x,gfront=gside=x∈O,x,gv=x∈O∧gv′=⊥,∅,otherwise.g = \begin{cases} x, & g_{\mathrm{front}} = g_{\mathrm{side}} = x \in O, \\ x, & g_v = x \in O \land g_{v'} = \perp, \\ \emptyset, & \text{otherwise.} \end{cases}

A held block is detached from the reconstructed arrangement and receives no spatial predicates of its own. If another block was reported as resting on the held block, that block is instead placed on the table; any block reported beneath the held block is treated as clear. The rule is defined for generality: it is required in embodied settings where the gripper is observable, and on our data it acts only to reject unsupported holding claims.

Inference Details

The open-weight models, Gemma-4-31B and Qwen3.5-122B-A10B, are served locally with vLLM [9] on eight NVIDIA H100 GPUs. Gemini-3.1-Pro refers to gemini-3.1-pro-preview, accessed through the Gemini Enterprise Agent Platform. We use the default inference hyperparameters for each model, except that temperature is set to zero to improve reproducibility.

Error Analysis on BW-PO-DEPTH

We reduce each method’s output to a support map: for every block, it reports the single object it rests on, whether another block, the table, or the gripper. This is the only claim all three methods make in common, and it is scored against ground truth over 110 problems ×\times 2 states ×\times 6 blocks =1320= 1320 placements. VLM-as-formalizer and VLM-as-perceiver are read directly from their emitted PDDL. VLM-as-planner emits no state at all, so its support map must be recovered from its plan. Each action commits the plan to a belief about the block in the initial state: (unstack x y) entails that xx was on yy, and (pick-up x) that xx was on the table. Taking the first action to touch each block and simulating forward from there yields the weakest initial state under which the plan is executable. Blocks the plan never touches carry no such commitment and are excluded from its score. We report coverage, the share of the 1320 placements that remain, which is 39.5% for Gemma-4-31B, 36.2% for Qwen3.5-122B-A10B, and 63.6% for Gemini-3.1-Pro. Because the excluded blocks are precisely those the plan avoided, VLM-as-planner’s accuracy is measured on a self-selected subset and should be read as an upper bound.

DatasetGemma-4-31B Plan success rate ReasoningGemma-4-31B Plan success rate No reasoningGemma-4-31B Mean tokens per problem ReasoningGemma-4-31B Mean tokens per problem No reasoningQwen3.5-122B-A10B Plan success rate ReasoningQwen3.5-122B-A10B Plan success rate No reasoningQwen3.5-122B-A10B Mean tokens per problem ReasoningQwen3.5-122B-A10B Mean tokens per problem No reasoning
BW-PO-DEPTH97%100%7.70.495%82%9.01.8
BW-PO-699%96%2.70.3100%44%3.51.3
BW-PO-1286%38%4.70.551%7%16.34.8
BW-PO-2022%15%15.31.06%1%39.113.3
BW-PO-401%0%35.816.80%0%49.424.9
BW-PO-1000%0%58.012.20%0%58.934.3

Table 3. VLM-as-perceiver with each open-weight model’s reasoning mode enabled and disabled. Unlike Table 2, token counts here are averaged over all problems rather than over successful plans, since the no-reasoning configurations succeed too rarely for the per-success ratio to be meaningful.

DatasetGemma-4-31B PlannerGemma-4-31B FormalizerGemma-4-31B PerceiverQwen3.5-122B-A10B PlannerQwen3.5-122B-A10B FormalizerQwen3.5-122B-A10B PerceiverGemini-3.1-Pro PlannerGemini-3.1-Pro FormalizerGemini-3.1-Pro Perceiver
BW-PO-DEPTH1290–7.9––9.623.040.121.1
BW-PO-69.710.12.76.97.23.53.93.64.0
BW-PO-1220.567.05.448.047.332.018.010.98.4
BW-PO-20932.91244.170.3––613.980.818.511.7
BW-PO-40––3943.5–––91.525.115.0
BW-PO-100––––––3564.42068.947.5

Table 2. Mean generated tokens per successful plan, in thousands. Dashes indicate that no plan was validated successfully, leaving the ratio undefined.

Ambiguity labels come from a single VLM-as-perceiver run with Gemini-3.1-Pro that records each view’s description before reconciliation. A placement is ambiguous when both views place the block and disagree, and unambiguous otherwise. Re-deriving the labels from a Gemma-4-31B run agrees with the Gemini-3.1-Pro-derived labels on 97.9% of the 1320 placements, so the split reflects the images rather than the model that produced it. Because a block that both views describe identically but wrongly is labelled unambiguous, the ambiguous set is a lower bound. Under these labels the 1320 placements divide into 389 that both views place identically, 519 that exactly one view places, and 412 ambiguous ones; none are placed by neither view. Every uncontradicted claim is correct, and in all 412 ambiguous cases the true support is one of the two competing claims, so each disagreement is a binary choice with the answer present in the evidence. Selection on an ambiguous placement is therefore one of four outcomes, in which a method names the support one view claims, the support the other claims, a third support neither claims, or declines to place the block; the rates reported in § 4 exclude the last case.

Effect of Reasoning Mode

The restricted perception task used by VLM-as-perceiver asks the VLM only to report local spatial observations within a single view, which raises the question of whether the open-weight models require their reasoning modes in this setting. We therefore re-evaluate Gemma-4-31B and Qwen3.5-122B-A10B under VLM-as-perceiver with reasoning disabled, holding all other inference settings fixed.

The effect depends on scene size rather than on perceptual difficulty. On BW-PO-DEPTH, Gemma-4-31B is marginally better without reasoning, reaching 100% against 97% while generating roughly 20×20 \times fewer tokens; Qwen3.5-122B-A10B loses 13 points on the same dataset. As object count grows, reasoning becomes necessary: Gemma-4-31B falls from 86% to 38% on BW-PO-12, and Qwen3.5-122B-A10B from 100% to 44% on BW-PO-6 and from 51% to 7% on BW-PO-12. We therefore find no setting in which reasoning can be disabled uniformly, and report all main-text results with each model’s default reasoning mode enabled.

DatasetMethodGemma-4-31B PrecisionGemma-4-31B RecallQwen3.5-122B-A10B PrecisionQwen3.5-122B-A10B RecallGemini-3.1-Pro PrecisionGemini-3.1-Pro Recall
BW-PO-6Planner84.462.3100.075.595.571.1
BW-PO-6Formalizer79.579.4100.0100.0100.0100.0
BW-PO-6Perceiver100.099.2100.099.6100.0100.0
BW-PO-12Planner81.370.682.372.097.684.6
BW-PO-12Formalizer87.375.993.092.299.599.5
BW-PO-12Perceiver99.798.398.084.0100.0100.0
BW-PO-20Planner41.338.345.639.089.383.4
BW-PO-20Formalizer83.565.081.065.799.899.8
BW-PO-20Perceiver94.086.793.856.3100.0100.0
BW-PO-40Planner35.630.99.06.677.676.2
BW-PO-40Formalizer70.146.152.536.795.095.0
BW-PO-40Perceiver92.652.076.435.199.799.7
BW-PO-100Planner23.88.36.32.08.74.1
BW-PO-100Formalizer34.719.324.512.059.940.7
BW-PO-100Perceiver92.39.777.15.199.898.9

Table 4. Planner follows the same recovery as described in Appendix § 10 in order to compute precision and recall. Each value is the mean of the initial state and goal state score, reported in percent.

References

  1. [1]Nikolaj Bjørner, Anh-Dung Phan, and Lars Fleckenstein. 2015. vZ - an optimizing smt solver. In Tools and Algorithms for the Construction and Analysis of Systems, pages 194–199, Berlin, Heidelberg. Springer Berlin Heidelberg.
  2. [2]Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340, Berlin, Heidelberg. Springer Berlin Heidelberg.DOI
  3. [3]Zhiyuan Feng, Zhaolu Kang, Qijie Wang, Zhiying Du, Jiongru Yan, Shubin Shi, Chengyo Yuan, Huizhi Liang, Yu Deng, Qixiu Li, Rushuai Yang, Arctanx An, Leqi Zheng, Weijie Wang, Shawn Chen, Sicheng Xu, Yaobo Liang, Jiaolong Yang, and Baining Guo. 2026. Seeing across views: Benchmarking spatial reasoning of vision-language models in robotic scenes. Preprint, arXiv:2510.19400.arxiv.org/abs/2510.19400
  4. [4]Muyu He, Yuxi Zheng, Yuchen Liu, Zijian An, Bill Cai, Jiani Huang, Lifeng Zhou, Feng Liu, Ziyang Li, and Li Zhang. 2025. Vision language models cannot plan, but can they formalize? Preprint, arXiv:2509.21576.arxiv.org/abs/2509.21576
  5. [5]M. Helmert. 2006. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191–246.
  6. [6]R. Howey, D. Long, and M. Fox. 2004. Val: automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294–301.
  7. [7]Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. 2023. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842.arxiv.org/abs/2311.17842
  8. [8]IPC. 1998. International planning competition.
  9. [9]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
  10. [10]Matteo Merler, Nicola Dainese, Minttu Alakuijala, Giovanni Bonetta, Pietro Ferrazzi, Yu Tian, Bernardo Magnini, and Pekka Marttinen. 2026. Viplan: A benchmark for visual planning with symbolic predicates and vision-language models. Preprint, arXiv:2505.13180.arxiv.org/abs/2505.13180
  11. [11]Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Jackson Zhang, Vittal Thirumalai, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Harrison, Anuj Mahajan, brian ichter, and Hamsa Balakrishnan. 2024. Long-horizon planning for multi-agent robots in partially observable environments. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  12. [12]Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. 2025. Capture: Evaluating spatial reasoning in vision language models via occluded object counting. Preprint, arXiv:2504.15485.
  13. [13]Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu, and Donald E. Brown. 2026. Can vision-language models count? a synthetic benchmark and analysis of attention-based interventions. Preprint, arXiv:2511.17722.arxiv.org/abs/2511.17722
  14. [14]Keisuke Shirai, Cristian C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. 2024. Vision-language interpreter for robot task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2051–2058.arxiv.org/abs/2311.00967
  15. [15]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juveong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Biao Zhang, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamdel, Jakub Adamek, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chryso\́valantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Daniele Calandriello, Glenn Cameron, Charlotte Caucheteux, Rahma Chaabouni, Garima Chadha, Jetha Chan, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu hsiang Chien, Nikolai Chiniay, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Sebastian Fennerhag, Takumi Fujimoto, João Gabriel Oliveira, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Tamara von Glehn, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Mayank Gupta, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Ivan Korotkov, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Vivek Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhkin, Alon Levkovitch, Dongdong Li, Qiuqia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Ivan Lobov, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Maciej Mikuła, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O’Donnell, Brendan O’Donoghue, Kate Olszewska, Bernett Orlando, Wangqiong Pan, Rina Panigrahy, Unnati Parekh, Nicolas Perez-Nieves, Chunlong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, George Scrivener, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Bobak Shahriari, Yuan Shanguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Jean Tarbouriech, Chetan Tekur, Shantanu Thakoor, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Çağlar Ünlü, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wasserberg, John Wieting, Cindy Wu, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Morteza Zadimoghaddam, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, and Chen Zou. 2026. Gemma 4 technical report. Preprint, arXiv:2607.02770.
  16. [16]Qwen Team. 2026. Qwen3.5: Accelerating productivity with native multimodal agents.
  17. [17]Quynh Vo, Phuc Dao, Cong-Duy Nguyen, and Thong Nguyen. 2026. When depth is better told than shown: Depth-ordinal prompting for vision-language spatial reasoning. Preprint, arXiv:2607.11173.arxiv.org/abs/2607.11173
  18. [18]Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. Muirbench: A comprehensive benchmark for robust multi-image understanding. Preprint, arXiv:2406.09411.arxiv.org/abs/2406.09411
  19. [19]Heecheol Yun and Eunho Yang. 2026. Integrating multimodal large language model knowledge into amodal completion. Preprint, arXiv:2603.28333.arxiv.org/abs/2603.28333

Paper details

vlnc:W_pjbwsvpg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents