betaFeedback

WorksComputer Science

Decide Before You Move: Diffusion-Generated Search Reference for Zero-Shot Open-Vocabulary Object Navigation in Semi-Static Environments

Abstract

Open-world navigation requires a robot to locate a user-specified target in an indoor scene under open-vocabulary queries. Realistic indoor environments are semi-static: walls and furniture largely persist while smaller objects move, relocate, or vanish between visits. Prior-map systems answer queries from object poses recorded at reconstruction time, which expire as soon as objects move, whereas online methods re-derive where to look from current observations every few steps, often through expensive calls to a vision–language model. Assuming that walls and large furniture do not move, we keep their map to plan paths, but use a target–furniture co-occurrence prior to decide where to search. We present PreNav, an ObjectNav framework on semi-static environments. Rather than invoking a vision-language model at each step, PreNav generates a single global search reference before the robot moves: an LLM converts target–furniture co-occurrence into a spatial prior, a conditional diffusion model turns that prior and the persistent structure into an ordered centerline with a band around it, and the robot approaches the target once it is detected online. Experiments on a Gibson ObjectNav benchmark show that PreNav outperforms a strong online baseline under matched structural priors, using a generator trained only on room-level ScanNet++ scenes and applied to floor-level Gibson without fine-tuning. Real-world experiments under object relocation and open-vocabulary queries further validate the approach.

Introduction

Object navigation requires a robot to move through an indoor environment and find a target specified only by a user query [1, 2]. Existing methods keep either too much of the building or too little. Prior-map systems answer queries from dense object-level maps [3, 4], whose object poses expire as soon as objects move. Map-free methods use no prior knowledge of the building and decide where to look online, re-deriving candidate regions from current observations every few steps, often through expensive calls to a vision–language model [5, 6]. Real buildings are semi-static: walls and large furniture such as wardrobes and tables barely move, whereas objects such as keys and cups change position between visits. Between these extremes, two lines of work handle the semi-static building differently. One follows what changes: systems for changing scenes keep a semantic map of objects and update it online as they move, adding and removing voxels [7] or tracking whether each object instance is still in place [8], yet their search still runs over the whole maintained map, movable objects included. The other builds on what persists: osmAG [9], a lightweight, text-based hierarchical map of areas such as rooms and corridors, supports localization [10] and LLM-based map understanding and navigation [11, 12] even when incomplete, and osmAG-LLM [13] uses it to reliably identify the room that holds a moved or never-mapped target. Within that room, however, the robot can only check candidate locations recorded during mapping, namely viewpoints along the mapping trajectory and past object positions, which need not cover where the target now lies. With the search already narrowed to a room, and tracking every object as it moves requiring continual map maintenance, we ask whether the part of the room that barely changes, its walls and large furniture, suffices to plan where to look, and in what order, before the robot moves.

We argue that it does. Real buildings are semi-static: walls and large furniture persist across missions, while movable objects, including the target, may change location. Therefore, instead of continually updating object locations, or treating the environment as a completely unknown scene, we treat only the objects as unknown and rely on the layout of walls and furniture to infer where the target may be. Thus, even if the target has moved, we do not rely on previously recorded target locations. Instead, we use semantic scores between the target and furniture to guide the search: a cup may be near a dining table, whereas a television is unlikely to be near a toilet.

We propose PreNav, which uses this prior knowledge of a scene to pre-plan an informed object search for a given query whenever a query arrives. An LLM scores target–furniture co-occurrence once, offline, and the scores are projected onto the mapped furniture as a spatial prior. A conditional diffusion model then turns this prior, the wall raster, and the start pose into what we call the search reference: a globally planned reference path that optimizes the order of where to search in a floor or room. The reference is thus both an order to follow and a region to sweep. Trained on references from a solver for the traveling salesman problem (TSP), the generator produces one in about a second rather than tens of seconds. The robot then follows the reference, detects the target online, and approaches it, without repeated foundation-model calls or query-conditioned global replanning.

We evaluate PreNav on an expanded Gibson [14] ObjectNav benchmark, asking three questions: whether it outperforms

Yuxuan Feng, Fujing Xie, and Sören Schwertfeger are with the School of Information Science and Technology, ShanghaiTech University, Shanghai, China. Email: fengyx12025, xiefj, [email protected].

Hermann Blum is with the University of Bonn and the Lamarr Institute, Bonn, Germany. Email: [email protected].

We provide our code and dataset at: https://anonymous.4open.science/r/PreNav forms online baselines under matched structural priors; how generation efficiency trades against navigation quality when compared with its TSP teacher; and whether it transfers to real-world robots under open-vocabulary queries.

As a summary, our main contributions are:

  • We propose PreNav, an ObjectNav framework for semi-static environments: it guides online search through a search reference that is globally optimized with respect to the known room layout and semantic areas.

  • We introduce an open-vocabulary interface that projects any user query onto the persistent furniture through LLM-generated co-occurrence scores between the query and each furniture category. As the generator’s only query-dependent input, these scores let a conditional diffusion model plan a search reference for any query without retraining.

  • We demonstrate zero-shot transfer from room-level ScanNet++ training to floor-level Gibson scenes, without fine-tuning the generator on Gibson. On the six-category benchmark, our method outperforms SGM under matched structural priors and achieves navigation performance comparable to its TSP teacher with an order of magnitude less generation time. We further demonstrate deployment on a real mobile robot with open-vocabulary queries.

Related Work

Prior-map systems organize a scanned building into persistent semantic representations and answer queries directly from this memory, without re-exploring. HOV-SG [3] and ConceptGraphs [4] structure open-vocabulary 3D features into hierarchical scene graphs that support language queries over a pre-scanned building, and osmAG-LLM [13] tolerates such changes by searching online, but still draws its candidates from objects and views recorded during mapping. Their answers, however, depend on object poses recorded at reconstruction time—once objects move, the memory returns expired answers. PreNav likewise assumes a familiar building, but deliberately retains only walls and large-furniture regions; the current pose of the queried object is never provided.

Knowledge about the scene has to come from somewhere when no prior is given. Existing methods fall into three families. The first family decides on an online semantic map: SemExp [15] predicts exploration goals on the map it builds, PONI [16] learns potential functions that prioritize promising regions, and SGM [17] extends this map into unseen regions with a self-supervised generative model, selecting goals on the completed map. The second family explores through frontier scoring: CoWs [18] scores frontiers with CLIP, VLFM [5] builds a vision-language value map, and OpenFrontier [6] treats visually grounded frontiers as sparse semantic anchors. The third family follows route instructions end-to-end: recent VLN methods such as NaVILA [19] fine-tune a vision–language model to map natural-language route descriptions and RGB observations to navigation actions.

The first two families require no persistent map, since the scene is entirely unknown when the task starts. They build a temporary map from observations and use the goal’s semantics to decide the next action, and every few steps the robot re-decides through semantic reasoning, sometimes through expensive calls to a vision–language model. The third family builds no map; instead, each instruction already implies the scene it describes. The instruction itself is the prior—scene-specific, human-given. Of these, only the first family decides on a top-down semantic map into which known walls and furniture can be written before the episode starts; we therefore compare against its most recent member, SGM, under the same priors.

Semi-Static Environments

DynaMem [7] adds and removes voxels as objects move, treating stable and movable parts of the scene alike. OpenIN [20] and Where Did I Leave My Glasses? [8] separate the two, through static carriers or per-instance stationarity, and let an LLM rank where to search. All three, however, maintain a map that includes the movable objects and choose their next goal during navigation. PreNav keeps only walls and furniture, which need no maintenance, and plans the whole search before the robot moves.

Diffusion Models for Planning

Diffusion models learn to generate outputs from a training distribution by iterative denoising [21]. In robot planning they serve at three levels: at the trajectory level, Diffuser [22] generates long-horizon state–action sequences; at the policy level, NoMaD [23] navigates on topological memory with a single diffusion policy; and at the level of combinatorial solving, TSPDiffuser [24] casts TSP as a denoising process over solution space, showing that diffusion can act as a learned sampler for combinatorial path planning. The last is closest to ours: the training labels likewise come from a combinatorial solver—TSP, the traveling salesman problem with neighborhoods, supplies the ground-truth path points—and we additionally learn a width at every path point, forming a band that marks where the target is likely to appear. This output is trained on room-level scenes, infers on floor-level scenes, and is validated on a real robot, with zero-shot and open-vocabulary queries supported by construction.

Methodology

PreNav separates search-reference generation from online perception. When a query comes, the search reference is generated in two steps. First, the furniture-conditioned search prior (§ 3-A) computes a spatial prior that encodes where the target is likely to be found relative to known furniture. Second, the diffusion-based global search generator (§ 3-B) takes this prior, the persistent wall raster, and the start pose as inputs and generates a single search reference. The robot follows this reference, detects the target online, and then approaches it; subsequent execution requires neither repeated high-level foundation-model calls nor query-conditioned global replanning. Figure 1 summarizes the complete data flow.

Overview of PreNav

Figure 1. Overview of PreNav. (a) For a query qq, an LLM scores how useful each furniture category cc is for finding the target; the scores rc(q)r_c(q) are computed once and cached. (b) The normalized weights αc(q)\alpha_c(q), projected onto the mapped furniture, form the spatial prior DqD_q, which is stacked with the wall raster WW and the start map S(s0)S(s_0). (c) Conditioned on these three channels, a diffusion model denoises random points into the search reference. (d) The reference on the real lab map: a centerline that fixes the visiting order, and a band (teal) that widens over high-scoring furniture (outlined). (e) Real-robot execution: viewpoints are sampled near the centerline to cover where the band overlaps furniture whose weight αc(q)\alpha_c(q) clears a threshold—the regions most worth searching, shown in violet. The robot visits them in order, detects the target with an open-vocabulary detector, and approaches it. Steps (a)–(d) run once, before the robot moves.

Furniture-Conditioned Search Prior

In a semi-static environment, the target’s current location is unknown, but its co-occurrence with persistent furniture provides a stable search prior. For example, a book is likely near a bookshelf, whereas an earphone is unlikely to appear near a refrigerator. We capture this through a two-step process: an LLM scores target–furniture co-occurrence, and the resulting weights are projected onto the spatial layout of the mapped furniture.

LLM co-occurrence scoring. Let C\mathcal{C} denote the vocabulary of the supplied furniture prior. A query qq is free-form text that describes the target: an object category such as vase, a description such as tray with fruits, or a broader concept such as computer parts. For every persistent furniture category c∈Cc \in\mathcal{C}, an LLM is asked how useful cc is for finding the target under each of three spatial relations—the target resting on cc, contained in cc, or merely near cc—and returns the three relation scores together with a single search-usefulness score rc(q)∈[0,100]r_c(q) \in[0,100] that summarizes them. Only rc(q)r_c(q) enters the prior. We normalize it per query to obtain weights,

αc(q)=exp⁡(rc(q)/τq)∑c′∈Cexp⁡(rc′(q)/τq),τq=0.65 σq,(1)\alpha_c(q) = \frac{\exp(r_c(q)/\tau_q)}{\sum_{c' \in\mathcal{C}} \exp(r_{c'}(q)/\tau_q)}, \qquad\tau_q = 0.65\,\sigma_q, \tag*{(1)}

where σq\sigma_q is the standard deviation of {rc(q)}c∈C\{r_c(q)\}_{c \in\mathcal{C}}. The temperature is set per query rather than globally because the LLM does not use a consistent range across queries: on the Gibson queries the top score is 95 for chair but 50 for oven, and a single temperature would leave the conservatively scored queries with a near-uniform prior. Scaling by the spread makes αc(q)\alpha_c(q) depend on how sharply the LLM separated the supports, not on the absolute numbers it happened to emit. We omit furniture categories with

αc(q)<0.01\alpha_c(q) < 0.01 when constructing the condition map. These scores are computed once per query and cached; a new query requires one run of the same scoring utility before the robot moves, rather than LLM calls during navigation.

Spatial prior construction. Each furniture instance fif_i with category c(i)c(i) is reduced to its extent on a 2D 64×6464 \times64 raster plane. It induces an axis-aligned anisotropic Gaussian Ki(y,x)K_i(y,x) centered at the box centroid, whose per-axis standard deviations are half the corresponding box extents, each lower-bounded at 1.5 cells so that small supports remain visible to the planning grid. The query-conditioned spatial prior is

Dq(y,x)=∑iαc(i)(q)Ki(y,x)∑y′,x′∑iαc(i)(q)Ki(y′,x′).(2)D_q(y,x) = \frac{\sum_i \alpha_{c(i)}(q) K_i(y,x)}{\sum_{y',x'} \sum_i \alpha_{c(i)}(q) K_i(y',x')}. \tag*{(2)}

DqD_q is a spatial weighting over furniture supports rather than a target-occupancy prediction: the label generator seeks to cover the retained supports, while DqD_q conditions the learned route and sets its supervised search-band width through Eq. (4)—for instance, a query for vase gives wider width targets near dining tables. Because the weights are assigned to furniture categories rather than instances, they remain applicable when an object relocates between compatible supports.

Diffusion-Based Global Search Generation

The search reference must capture long-range dependencies, such as which supports to visit and in what order, while remaining flexible to local geometric constraints. Diffusion models are well-suited for this task because they learn multimodal distributions over structured outputs through iterative denoising, avoiding the combinatorial complexity of explicit tour optimization at inference.

Problem formulation. The model receives a three-channel condition

c=[W, S(s0), Dq],(3)\mathbf{c} = [W,\ S(s_0),\ D_q], \tag*{(3)}

where W\mathcal{W} is the wall/free-space raster, S(s0)S(s_0) is a Gaussian centered at the start pose with a standard deviation of 1.25 cells, and DqD_q is the spatial prior from Eq. (2). It denoises a sequence G∈RL×3\mathbf{G} \in\mathbb{R}^{L \times3} whose rows are (yj,xj,wj)(y_j,x_j,w_j) targets: (yj,xj)(y_j,x_j) is a centerline point on the planning grid and wjw_j specifies the desired search half-width around that point. We refer to G\mathbf{G} as the search reference: its centerline gives the visiting order, and its half-widths define the band swept around that centerline.

Training data and reference supervision. We generate structured reference labels offline from ScanNet++ [25] rather than training from embodied rollouts. Each training example contains a wall raster, persistent furniture instances, a start location, a target category, and an ordered search reference. The dataset contains 83 eligible scenes, split 75/8 into training and validation; Gibson [14] is reserved for evaluation and never used during training. For each empirical support-frequency vector, the data generator adds up to five randomly sampled support-weight variants, exposing the denoiser to diverse spatial priors rather than only the fixed co-occurrence table used at evaluation. During training, horizontal and vertical flips and rotations by multiples of 90∘90^\circ are applied consistently to all condition channels and reference coordinates.

To produce the labels, we follow TSPDiffuser [24] and solve a TSP with neighborhoods [26] offline: around the start and each furniture center we take a free-space neighborhood, and CP-SAT [27] selects one point per neighborhood and a closed visit order under A* path costs [28]. Joining the selected points by their A* paths gives a loop from the start, which we cubically interpolate to L=100L = 100 samples clamped to the planning grid. At each sample, the width target is derived from the same spatial prior:

wj=wmin⁡+(wmax⁡−wmin⁡)Dq(yj,xj),gj=(yj,xj,wj).(4)\begin{aligned} w_j &= w_{\min} + (w_{\max} - w_{\min})D_q(y_j,x_j),\\ \mathbf{g}_j &= (y_j,x_j,w_j). \tag*{(4)} \end{aligned}

Here wmin⁡=1w_{\min}=1 and wmax⁡=12w_{\max}=12 grid cells (half-width). All three output channels and all condition channels are normalized to [−1,1][-1,1] during training. TSP is used only to create these offline labels and is absent at inference.

Training objective. Let G0\mathbf{G}_0 be such a TSP label, and Gk\mathbf{G}_k its noised version at diffusion timestep kk, drawn uniformly from 0≤k<10000 \le k < 1000 for each sample. The standard noise-prediction loss is

Ldiff=EG0,k,ϵ[∥ϵ−ϵθ(Gk,k,c)∥22].(5)\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{\mathbf{G}_0,k,\epsilon}\left[\left\|\epsilon-\epsilon_\theta(\mathbf{G}_k,k,\mathbf{c})\right\|_2^2\right]. \tag*{(5)}

We additionally encourage the predicted centerline to avoid walls. For low-noise steps (k<200k < 200), we reconstruct the denoised estimate G^0=(Gk−1−αˉkϵθ)/αˉk\widehat{\mathbf{G}}_0=(\mathbf{G}_k-\sqrt{1-\bar{\alpha}_k}\epsilon_\theta)/\sqrt{\bar{\alpha}_k} and bilinearly sample the wall channel at the predicted centerline points. For a minibatch of NN samples, the implementation averages this penalty only over samples whose timestep satisfies ki<200k_i < 200:

Lwall=∑i=1N1[ki<200](1L∑j=1LBW(i)(y^ij,x^ij))max⁡(1,∑i=1N1[ki<200]),(6)\mathcal{L}_{\mathrm{wall}} = \frac{\sum_{i=1}^{N}\mathbf{1}[k_i<200]\left(\frac{1}{L}\sum_{j=1}^{L}B_{\mathcal{W}}^{(i)}(\widehat{y}_{ij},\widehat{x}_{ij})\right)} {\max\left(1,\sum_{i=1}^{N}\mathbf{1}[k_i<200]\right)}, \tag*{(6)}
L=Ldiff+0.2 Lwall,\mathcal{L}=\mathcal{L}_{\mathrm{diff}}+0.2\,\mathcal{L}_{\mathrm{wall}},

where BW(i)(y,x)∈[0,1]B_{\mathcal{W}}^{(i)}(y,x)\in[0,1] is the sampled wall value for sample ii. Thus, the auxiliary term is applied only to low-noise training timesteps (the late portion of the reverse process), where reconstruction of G0\mathbf{G}_0 from the predicted noise is numerically better conditioned.

Architecture. A convolutional encoder maps the 64×6464\times64 condition to a 768-channel 16×1616\times16 feature grid; fixed 2D sinusoidal positions are added before the 256 map tokens are projected to d=256d=256. Each three-dimensional reference token is projected to the same width and receives a fixed 1D sinusoidal position and a learned timestep embedding. A four-layer pre-norm Transformer decoder (eight heads, 1,024-dimensional feed-forward layers) applies self-attention over the reference tokens and cross-attention to the map tokens, then predicts one three-dimensional noise vector per reference token. The network has 5.24M trainable parameters and uses 1,000 squared-cosine diffusion steps.

Inference. At deployment, the diffusion model uses the same channel order and normalization as during training. Starting from Gaussian noise, it produces one 100-point search reference in 50-step DDPM sampling. This is the only query-conditioned global decision made for the entire deployment.

Experiments

We evaluate whether persistent structural priors improve ObjectNav, how diffusion trades navigation performance against the generation cost of its TSP teacher, and whether the framework transfers to real-world object queries. The main comparison and subsequent analyses use a six-category subset; we also report the complete ten-category evaluation.

Experimental Setup

Datasets and benchmark

We evaluate on the five validation scenes of Gibson [14] v1.1: COLLIERVILLE, COROZAL, DARDEN, MARKLEEVILLE and WONICONISCO. Of the 15 annotated semantic categories, five serve as persistent furniture and the other ten as query targets, with 100 episodes each, 1,000 in total. Four of these targets, book, bottle, clock and cup, are limited by detection rather than by search; the main comparison and subsequent analyses therefore use the remaining six—chair, oven, potted plant, refrigerator, tv and vase—comprising 600 episodes. Sec. IV-C details the detector analysis behind this choice and reports the full ten-category results.

Comparison protocol. Within each evaluation set, all methods use the same episodes, a 500-action budget and the ObjectNav success criterion used by SGM [17] and PONI [16]. We report success rate (SR) and success weighted by path length (SPL) [15, 16], both as percentages. All ablation experiments use Mask R-CNN as the object detector, the same detector used by the baselines. To separate search from recognition, we further evaluate PreNav with stronger detectors and with a 3DSG oracle. Gibson’s object labels in Habitat come from the 3D Scene Graph (3DSG) annotation [29], which Habitat converts into a semantic mesh and renders through its semantic sensor; the 3DSG oracle replaces the detector with these rendered labels, which cover every object pixel in the current frame. SGM’s released code provides the same oracle. Success, in contrast, is judged on the top-down GT semantic map of the ObjectNav dataset [15], a separate annotation that does not fully coincide with 3DSG (Sec. IV-C).

SGM’s pretrained semantic model supports the dataset’s original 15 semantic categories, including five that we treat as furniture: dining table, couch, bed, toilet and sink. For a fair comparison, we supply both SGM and PreNav with the same wall prior and restrict their furniture priors to these five categories (✓ ⁣\\checkmark\!\backslash in Table I and Table II). PreNav also accepts open-vocabulary furniture categories through its spatial-prior interface, without retraining the generator. We therefore additionally evaluate it with the full-scene furniture prior (✓\checkmark), which adds the furniture that the scenes leave unlabeled, annotated by us as 2D boxes; this is a comparison within PreNav to measure the benefit of additional furniture information.

MethodWallFurnitureSR ↑SPL ↑
SGM××55.029.6
×✓ ⁣\\checkmark\!\backslash60.533.1
✓\checkmark×61.336.2
✓\checkmark✓ ⁣\\checkmark\!\backslash62.036.2
Ours✓\checkmark✓ ⁣\\checkmark\!\backslash64.339.2
✓\checkmark✓\checkmark70.339.8

Table I. ObjectNav on the six-category Gibson subset (600 episodes). ×: no prior; ✓\checkmark: wall or full-scene furniture prior; ✓ ⁣\\checkmark\!\backslash: furniture categories supported by SGM.

MethodWallFurnitureSR ↑SPL ↑
SGM××36.422.0
×✓ ⁣\\checkmark\!\backslash40.625.5
✓\checkmark×39.924.1
✓\checkmark✓ ⁣\\checkmark\!\backslash42.826.9
Ours✓\checkmark✓ ⁣\\checkmark\!\backslash44.226.9
✓\checkmark✓\checkmark44.526.1

Table II. ObjectNav on the full ten-category Gibson set (1,000 episodes). ×: no prior; ✓\checkmark: wall or full-scene furniture prior; ✓ ⁣\\checkmark\!\backslash: furniture categories supported by SGM.

Implementation details. The denoiser is trained for 800 epochs using AdamW at 10−410^{-4}, a cosine learning-rate schedule with 100 warm-up steps, gradient clipping at 1.0, and train/validation batch sizes of 32/8. We evaluate in Habitat [30] under the common action-budget protocol. The agent carries a 640×480 RGB-D sensor (79∘79^\circ HFOV, 0.88 m height) with discrete actions (0.25 m forward and 30∘30^\circ turns) and a 5 cm occupancy grid. Category-specific perception uses Mask R-CNN R-50-FPN-3x [31] trained on COCO [32]; score thresholds are 0.3 (ten-category) and 0.7 (six-category), applied identically to all methods. The co-occurrence scores are produced once by DeepSeek V4 Pro and normalized with the per-query softmax of Eq. (1). The diffusion model samples one 100-point search reference in 50 denoising steps, with the half-width channel clipped to [1, 12] grid cells.

Diffusion and TSP receive identical semantic inputs. All experiments run on an NVIDIA RTX 4090 24GB; generator timing is measured on an AMD Ryzen 9 9950X with 16 single-thread workers (Table IV).

MethodRef. gen. (s) Mean ↓Ref. gen. (s) P95 ↓Mask R-CNN SR ↑Mask R-CNN SPL ↑3DSG oracle SR ↑3DSG oracle SPL ↑
TSP25.705169.89371.039.682.744.0
Diffusion0.9951.02970.339.882.546.6

Table IV. Search-reference generation cost and navigation performance comparison

Baseline Comparison

SGM benefits from structural priors. The walls and furniture we supply to SGM are taken from the top-down GT semantic map. SGM is itself trained to complete such maps from partially masked ones, so giving it part of the GT map is exactly the input its training prepares it for. Furniture alone raises its SR by 5.5% and SPL by 3.5%, and walls alone by 6.3% and 6.6% (Table I); with both, its SR and SPL rises by 7.0% and 6.6% over the original SGM. These priors are therefore informative, and SGM exploits them as its training intends.

PreNav exploits the same priors better, and gains from more furniture. SGM receives furniture as a pixel-level semantic map, whereas PreNav needs only a 2D bounding box per piece (§ 3-A), a coarser description. Even so, with the same walls and five furniture categories, PreNav outperforms SGM by 2.3% in SR and 3.0% in SPL. To show that furniture drives this search, we additionally annotate, as 2D boxes, the furniture that the scenes leave unlabeled; it lies outside the 15 annotated categories, and hence outside what SGM’s semantic model supports. This full-scene prior raises the SR of PreNav by another 6.0% and its SPL by 0.6%. PreNav thus exploits both the layout and the semantics of walls and furniture, and since each additional piece costs only a 2D box, supplying more furniture is a cheap way to make the target more likely to be found.

The Full Ten-Category Benchmark

Why the main analysis uses six categories. The Gibson ObjectNav data annotate 15 semantic categories, but the validation set queries only six of them, and three of these—couch, bed and toilet—are furniture in our setting, leaving only three targets. We therefore extend the query set: of the 15 categories, five serve as furniture and the other ten as targets, with 100 episodes each, 1,000 in total. Detector ablation (Table III) shows that four of the added targets are limited by perception rather than by search. On book, bottle, clock and cup, PreNav succeeds in only 21.2% of episodes with Mask R-CNN; the stronger SAM 3, which improves the other six categories by 3.8%, lowers them by 9.4%, and even the 3DSG oracle raises them only to 47.0%. Part of this gap comes from the annotations themselves: the oracle and the success criterion read different annotations, which disagree even at the category level. In the validation scene COLLIERVILLE, the 3DSG annotation contains two bottles that the top-down GT semantic map lacks, although both annotations mark the cup on the shelf below (Figure 2). Conversely, in WICONISCO the map contains a clock while the scene’s 3DSG annotation has no clock class, so the oracle can never report the target in the 57 clock episodes set there. This is why Mask R-CNN exceeds the 3DSG oracle on clock in Table III (34% vs. 28%): in WICONISCO it succeeds in 32 of these 57 episodes, the oracle in none. To measure navigation rather than recognition while maintaining comparability with the Object-Nav benchmark, we report the remaining six categories as the main benchmark and the full ten categories here.

CategoryMask R-CNNSAM 33DSG oracle
chair707398
oven799196
potted plant805481
refrigerator417175
tv545774
vase333466
book291853
bottle201667
clock341228
cup2140
Six-category subset59.563.381.7
Other four21.211.847.0
All 1,000 ep.44.242.767.8

Table III. Comparison of per-category success rates (%) across different detectors. Bold compares Mask R-CNN and SAM 3. 3DSG oracle: per-pixel labels rendered from the 3DSG annotation.

RGB view of a shelf, 3DSG oracle labels, and top-down map

Figure 2. The 3DSG oracle and the success criterion read different annotations (COLLIERVILLE). (a) RGB view of a shelf, with each 3DSG object boxed. (b) The 3DSG oracle labels two bottles and a cup. (c) The top-down GT semantic map used to judge success marks the cup (right) but has no bottle anywhere in the scene; 3DSG objects are projected onto it for comparison, and the wedge marks the camera’s field of view.

Results on all ten categories. The same priors again help SGM (Table II): furniture alone raises its SR by 4.2% and SPL by 3.5%, walls alone by 3.5% and 2.1%, and both together by 6.4% and 4.9%. Under these priors, PreNav outperforms SGM by 1.4% in SR with equal SPL, and the full-scene furniture prior adds a further 0.3% in SR, at 0.8% lower SPL. This slightly lower SPL indicates that the additional prior information leads to a few more found objects, but at the expense of generally slightly longer routes.

Room-to-Floor Transfer and Efficiency

We evaluate whether a search-reference generator trained on room-level data transfers to floor-level scenes while retaining navigation performance close to its teacher. Diffusion is trained only on ScanNet++ and applied directly to Gibson, without training or fine-tuning on Gibson. We compare it with the TSP teacher planner solving the Gibson instances again. Both use the same full-scene furniture prior, and six-category protocol with Mask R-CNN. TSP uses K=3K = 3 candidates per neighborhood and a 300 s solver time limit.

Approaching teacher performance at lower generation cost. With 25.8× less generation time than TSP on average (165× less at P95), diffusion achieves only 0.7% lower SR and even gains 0.2% in SPL with Mask R-CNN (Table IV); with the 3DSG oracle, it drops 0.2% in SR and gains 2.6% in SPL. When transferring our diffusion model from the room level to the floor level, navigation performance drops by less than 1% compared to our TSP teacher, yet SPL actually improves. More importantly, we avoid the high computational costs associated with solvers, and our model’s execution time does not increase with the number of furniture items, making it far more feasible for real-world deployment.

Planning: the reference covers the target. We first examine the references themselves, on the raw generator output. Let TqT_q denote the region of the queried category in the top-down GT semantic map, projected onto the planning grid, shown as the GT target in Figure 4(a,Figure 4b), where it comprises all potted plants in the scene; it enters only for evaluation. Sweeping each band along its centerline, we record the reference length at which the band first contains a wall-free cell of TqT_q (Figure 4(c)). By the end of the reference, the band contains the target in 76.7% of episodes for diffusion and 78.8% for TSP, and the two curves stay close over the first 20 m, within which most of this coverage is reached. The learned references are also shorter, 23.1 m against 30.4 m on average and 52.4 m against 61.1 m at most, where each curve ends.

Search references and navigation performance of diffusion and TSP

Figure 4. Search references of diffusion and its TSP teacher on the 600 six-category episodes. (a,b) Search references of diffusion and TSP for one episode in the Gibson validation scene DARDEN, with the query potted plant. The native band is the region swept by each generator’s own half-widths wjw_j along its centerline; the purple GT targets are the potted plants in the scene, overlaid for evaluation only. (c) Planning: share of episodes whose band, swept along the reference, contains the target, against reference length; each curve ends with that generator’s longest reference. (d) Execution: share of episodes that succeed within a distance traveled, under Mask R-CNN.

Execution: success per distance traveled. For a travel budget dd, we count the episodes that succeed within dd meters of travel under Mask R-CNN (Figure 4(d)). The two curves are almost identical: both reach 50% success before 10 m, and about 95% of successful episodes finish within 20 m, matching the planning curves in Figure 4(c). A reference that covers the target thus also lets the agent find it in execution.

Failure Analysis

This section addresses the following question: What limits navigation success when using the generated search reference? We analyze the 600 episodes of PreNav on the six-category subset with the full-scene furniture prior and Mask R-CNN (70.3% SR, Table I), grouping each episode by target detection and execution outcome (Fig. 5). For target detection, we back-project each detection with depth onto the 2D map and take the agent’s last estimate of the target position, computing its minimum distance to TqT_q, the queried category’s region in the top-down GT semantic map (Sec. IV-D). A distance of at most 1 m counts as a true positive. The detected wrong target group contains the remaining detections: either the estimated position lies more than 1 m from TqT_q, or no position could be estimated, for example because the detection mask contained no valid depth. For execution, an episode succeeds, stops at an incorrect position, exhausts its 500 action budget, or exhausts the reference without a detection. Success follows almost entirely from a correct target estimate: 421 of the 422 successful episodes are true positives.

Failures, in turn, come mainly from target detection. The detected-wrong-target group (115 episodes, 19.2%) is the largest source, and its main flow, 72 episodes, ends at an incorrect stop position, where the agent stops at an object mistaken for the target. The few true positives that fail do so in the final stop. Failures traceable to the search reference lie within the never-detected group, only 30 episodes (5.0%), 23 of which (3.8%) complete the reference without a detection. Target detection, rather than the search reference, therefore limits navigation success.

Real-World and Open-Vocabulary Evaluation

We deploy PreNav on a Fetch mobile robot with a head-mounted Intel RealSense D435i RGB-D camera, using SAM 3 [33] as the open-vocabulary detector. The test scene is a cluttered laboratory of over 200 m2^2, half office area and half leisure area (Figure 3). Its walls come from an osmAG map [9] aligned to the robot’s occupancy grid, and its 21 furniture pieces are annotated once as boxes; the positions of the queried objects are never given.

Figure 3. Real-world test scene, a laboratory of over 200 m2^2. (a) The Fetch robot that used for real-world experiments, photographed in the test scene. (b) The room’s osmAG map [9] with the 21 annotated furniture boxes, the only semantic input, colored by category. (c) A 3D scan of the same room with the osmAG walls overlaid.

For each query, the LLM scores over the annotated furniture give DqD_q, and the generator produces the search reference as in simulation. Execution differs in one aspect: instead of following waypoints along the reference, the robot samples viewpoints near the centerline to cover where the band overlaps furniture whose weight αc(q)\alpha_c(q) clears a threshold, and navigates to them one by one until the target is detected and approached. The prior and the generator are unchanged.

The queries include tray with fruits, robot arm and quadruped robot, which lie outside the COCO vocabulary of the simulation detector. PreNav succeeds in 13 of 15 trials. One failure mode is an object placed far from any furniture: a quadruped robot left on the floor lay outside the covered region, so none of the viewpoints brought it into view.

Conclusion

We presented PreNav, which decides where to search before the robot moves: a conditional diffusion model generates one global search reference from persistent structure and an LLM-derived furniture prior. Under matched structural priors it outperforms SGM on Gibson; trained only on room-level ScanNet++, it transfers zero-shot to floor-level scenes close to its TSP teacher at a fraction of the generation cost, and it demonstrates performance on real-robot with open-vocabulary queries.

PreNav assumes that the target lies on or near furniture. This holds for most everyday objects, but an object left far from any furniture falls outside the band and is not found, as happened with a quadruped robot lying on the open floor of our lab. A natural extension is to search the free space that the reference leaves uncovered when the reference is exhausted.

References

  1. [1]D. Batra et al., “ObjectNav revisited: On evaluation of embodied agents navigating to objects,” 2020, arXiv:2006.13171.arxiv.org/abs/2006.13171
  2. [2]P. Anderson et al., “On evaluation of embodied navigation agents,” 2018, arXiv:1807.06757.arxiv.org/abs/1807.06757
  3. [3]A. Werby, C. Huang, M. Büchner, A. Valada, and W. Burgard, “Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation,” in Proc. Robot.: Sci. Syst., 2024.DOI
  4. [4]Q. Gu et al., “ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 5021–5028.arxiv.org/abs/2309.16650
  5. [5]N. H. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “VLFM: Vision-language frontier maps for zero-shot semantic navigation,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 42–48.arxiv.org/abs/2312.03275
  6. [6]E. Padilla Cerdio, B. Sun, M. Pollefeys, and H. Blum, “OpenFrontier: General navigation with visual-language grounded frontiers,” in Proc. Robot.: Sci. Syst., 2026.
  7. [7]P. L. Liu et al., “DynaMem: Online dynamic spatio-semantic memory for open world mobile manipulation,” in Proc. IEEE Int. Conf. Robot. Automat., 2025, pp. 13 346–13 355.arxiv.org/abs/2411.04999
  8. [8]B. Bogenberger et al., “Where did I leave my glasses? Open-vocabulary semantic exploration in real-world semi-static environments,” IEEE Robot. Automat. Lett., vol. 11, no. 3, pp. 3342–3349, Mar. 2026.
  9. [9]D. Feng, C. Li, Y. Zhang, C. Yu, and S. Schwertfeger, “osmAG: Hierarchical semantic topometric area graph maps in the OSM format for mobile robotics,” in Proc. IEEE Int. Conf. Robot. Biomimetics, 2023, pp. 1–7.arxiv.org/abs/2309.04791
  10. [10]F. Xie and S. Schwertfeger, “Robust lifelong indoor LiDAR localization using the area graph,” IEEE Robot. Automat. Lett., vol. 9, no. 1, pp. 531–538, Jan. 2024.DOI
  11. [11]F. Xie and S. Schwertfeger, “Empowering robot path planning with large language models: osmAG map topology & hierarchy comprehension with LLMs,” in Proc. IEEE Int. Conf. Robot. Biomimetics, 2024, pp. 707–712.arxiv.org/abs/2403.08228
  12. [12]F. Xie, J. Zhang, and S. Schwertfeger, “Intelligent LiDAR navigation: Leveraging external information and semantic maps with LLM as copilot,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2025, pp. 20 402–20 409.arxiv.org/abs/2409.08493
  13. [13]F. Xie, S. Schwertfeger, and H. Blum, “osmAG-LLM: Zero-shot open-vocabulary object navigation via semantic maps and large language models reasoning,” IEEE Robot. Automat. Lett., vol. 11, no. 3, pp. 2426–2433, Mar. 2026.
  14. [14]F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson Env: Real-world perception for embodied agents,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 9068–9079.arxiv.org/abs/1808.10654
  15. [15]D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 4247–4258.arxiv.org/abs/2007.00643
  16. [16]S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, “PONI: Potential functions for objectgoal navigation with interaction-free learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 18 868–18 878.arxiv.org/abs/2201.10029
  17. [17]S. Zhang, X. Yu, X. Song, X. Wang, and S. Jiang, “Imagine before go: Self-supervised generative map for object goal navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 16 414–16 425.DOI
  18. [18]S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “CoWs on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 23 171–23 181.arxiv.org/abs/2203.10421
  19. [19]A.-C. Cheng et al., “NaVILA: Legged robot vision-language-action model for navigation,” in Proc. Robot.: Sci. Syst., 2025.arxiv.org/abs/2412.04453
  20. [20]Y. Tang et al., “OpenIN: Open-vocabulary instance-oriented navigation in dynamic domestic environments,” IEEE Robot. Automat. Lett., vol. 10, no. 9, pp. 9256–9263, Sep. 2025.arxiv.org/abs/2501.04279
  21. [21]J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 6840–6851.arxiv.org/abs/2006.11239
  22. [22]M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine, “Planning with diffusion for flexible behavior synthesis,” in Proc. Int. Conf. Mach. Learn., 2022, pp. 9902–9915.arxiv.org/abs/2205.09991
  23. [23]A. Sridhar, D. Shah, C. Glossop, and S. Levine, “NoMaD: Goal masked diffusion policies for navigation and exploration,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 63–70.arxiv.org/abs/2310.07896
  24. [24]R. Yonetani, “TSPDiffuser: Diffusion models as learned samplers for traveling salesperson path planning problems,” in Proc. IEEE Int. Conf. Robot. Automat., 2025, pp. 10 824–10 830.
  25. [25]C. Yeshwanth, Y.-C. Liu, M. Nießner, and A. Dai, “ScanNet++: A high-fidelity dataset of 3D indoor scenes,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2023, pp. 12–22.arxiv.org/abs/2308.11417
  26. [26]E. M. Arkin and R. Hassin, “Approximation algorithms for the geometric covering salesman problem,” Discrete Appl. Math., vol. 55, no. 3, pp. 197–218, 1994.DOI
  27. [27]L. Perron, F. Didier, and S. Gay, “The CP-SAT-LP solver (invited talk),” in Proc. Int. Conf. Princ. Pract. Constraint Program., 2023, pp. 3:1–3:2.DOI
  28. [28]P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” IEEE Trans. Syst. Sci. Cyber., vol. 4, no. 2, pp. 100–107, 1968.
  29. [29]I. Armeni et al., “3D scene graph: A structure for unified semantics, 3D space, and camera,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 5664–5673.arxiv.org/abs/1910.02527
  30. [30]M. Savva et al., “Habitat: A platform for embodied AI research,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 9338–9346.arxiv.org/abs/1904.01201
  31. [31]K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2980–2988.
  32. [32]T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 740–755.arxiv.org/abs/1405.0312
  33. [33]N. Carion et al., “SAM 3: Segment anything with concepts,” in Proc. Int. Conf. Learn. Representations, 2026.arxiv.org/abs/2511.16719

Paper details

vlnc:W_3nza48e2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents