WorksComputer Science
Decide Before You Move: Diffusion-Generated Search Reference for Zero-Shot Open-Vocabulary Object Navigation in Semi-Static Environments
Abstract
Open-world navigation requires a robot to locate a user-specified target in an indoor scene under open-vocabulary queries. Realistic indoor environments are semi-static: walls and furniture largely persist while smaller objects move, relocate, or vanish between visits. Prior-map systems answer queries from object poses recorded at reconstruction time, which expire as soon as objects move, whereas online methods re-derive where to look from current observations every few steps, often through expensive calls to a vision–language model. Assuming that walls and large furniture do not move, we keep their map to plan paths, but use a target–furniture co-occurrence prior to decide where to search. We present PreNav, an ObjectNav framework on semi-static environments. Rather than invoking a vision-language model at each step, PreNav generates a single global search reference before the robot moves: an LLM converts target–furniture co-occurrence into a spatial prior, a conditional diffusion model turns that prior and the persistent structure into an ordered centerline with a band around it, and the robot approaches the target once it is detected online. Experiments on a Gibson ObjectNav benchmark show that PreNav outperforms a strong online baseline under matched structural priors, using a generator trained only on room-level ScanNet++ scenes and applied to floor-level Gibson without fine-tuning. Real-world experiments under object relocation and open-vocabulary queries further validate the approach.
Introduction
Object navigation requires a robot to move through an indoor environment and find a target specified only by a user query [1, 2]. Existing methods keep either too much of the building or too little. Prior-map systems answer queries from dense object-level maps [3, 4], whose object poses expire as soon as objects move. Map-free methods use no prior knowledge of the building and decide where to look online, re-deriving candidate regions from current observations every few steps, often through expensive calls to a vision–language model [5, 6]. Real buildings are semi-static: walls and large furniture such as wardrobes and tables barely move, whereas objects such as keys and cups change position between visits. Between these extremes, two lines of work handle the semi-static building differently. One follows what changes: systems for changing scenes keep a semantic map of objects and update it online as they move, adding and removing voxels [7] or tracking whether each object instance is still in place [8], yet their search still runs over the whole maintained map, movable objects included. The other builds on what persists: osmAG [9], a lightweight, text-based hierarchical map of areas such as rooms and corridors, supports localization [10] and LLM-based map understanding and navigation [11, 12] even when incomplete, and osmAG-LLM [13] uses it to reliably identify the room that holds a moved or never-mapped target. Within that room, however, the robot can only check candidate locations recorded during mapping, namely viewpoints along the mapping trajectory and past object positions, which need not cover where the target now lies. With the search already narrowed to a room, and tracking every object as it moves requiring continual map maintenance, we ask whether the part of the room that barely changes, its walls and large furniture, suffices to plan where to look, and in what order, before the robot moves.
We argue that it does. Real buildings are semi-static: walls and large furniture persist across missions, while movable objects, including the target, may change location. Therefore, instead of continually updating object locations, or treating the environment as a completely unknown scene, we treat only the objects as unknown and rely on the layout of walls and furniture to infer where the target may be. Thus, even if the target has moved, we do not rely on previously recorded target locations. Instead, we use semantic scores between the target and furniture to guide the search: a cup may be near a dining table, whereas a television is unlikely to be near a toilet.
We propose PreNav, which uses this prior knowledge of a scene to pre-plan an informed object search for a given query whenever a query arrives. An LLM scores target–furniture co-occurrence once, offline, and the scores are projected onto the mapped furniture as a spatial prior. A conditional diffusion model then turns this prior, the wall raster, and the start pose into what we call the search reference: a globally planned reference path that optimizes the order of where to search in a floor or room. The reference is thus both an order to follow and a region to sweep. Trained on references from a solver for the traveling salesman problem (TSP), the generator produces one in about a second rather than tens of seconds. The robot then follows the reference, detects the target online, and approaches it, without repeated foundation-model calls or query-conditioned global replanning.
We evaluate PreNav on an expanded Gibson [14] ObjectNav benchmark, asking three questions: whether it outperforms
Yuxuan Feng, Fujing Xie, and Sören Schwertfeger are with the School of Information Science and Technology, ShanghaiTech University, Shanghai, China. Email: fengyx12025, xiefj, [email protected].
Hermann Blum is with the University of Bonn and the Lamarr Institute, Bonn, Germany. Email: [email protected].
We provide our code and dataset at: https://anonymous.4open.science/r/PreNav forms online baselines under matched structural priors; how generation efficiency trades against navigation quality when compared with its TSP teacher; and whether it transfers to real-world robots under open-vocabulary queries.
As a summary, our main contributions are:
We propose PreNav, an ObjectNav framework for semi-static environments: it guides online search through a search reference that is globally optimized with respect to the known room layout and semantic areas.
We introduce an open-vocabulary interface that projects any user query onto the persistent furniture through LLM-generated co-occurrence scores between the query and each furniture category. As the generator’s only query-dependent input, these scores let a conditional diffusion model plan a search reference for any query without retraining.
We demonstrate zero-shot transfer from room-level ScanNet++ training to floor-level Gibson scenes, without fine-tuning the generator on Gibson. On the six-category benchmark, our method outperforms SGM under matched structural priors and achieves navigation performance comparable to its TSP teacher with an order of magnitude less generation time. We further demonstrate deployment on a real mobile robot with open-vocabulary queries.
Related Work
Navigation with Prior Maps
Prior-map systems organize a scanned building into persistent semantic representations and answer queries directly from this memory, without re-exploring. HOV-SG [3] and ConceptGraphs [4] structure open-vocabulary 3D features into hierarchical scene graphs that support language queries over a pre-scanned building, and osmAG-LLM [13] tolerates such changes by searching online, but still draws its candidates from objects and views recorded during mapping. Their answers, however, depend on object poses recorded at reconstruction time—once objects move, the memory returns expired answers. PreNav likewise assumes a familiar building, but deliberately retains only walls and large-furniture regions; the current pose of the queried object is never provided.
Navigation without Prior Maps
Knowledge about the scene has to come from somewhere when no prior is given. Existing methods fall into three families. The first family decides on an online semantic map: SemExp [15] predicts exploration goals on the map it builds, PONI [16] learns potential functions that prioritize promising regions, and SGM [17] extends this map into unseen regions with a self-supervised generative model, selecting goals on the completed map. The second family explores through frontier scoring: CoWs [18] scores frontiers with CLIP, VLFM [5] builds a vision-language value map, and OpenFrontier [6] treats visually grounded frontiers as sparse semantic anchors. The third family follows route instructions end-to-end: recent VLN methods such as NaVILA [19] fine-tune a vision–language model to map natural-language route descriptions and RGB observations to navigation actions.
The first two families require no persistent map, since the scene is entirely unknown when the task starts. They build a temporary map from observations and use the goal’s semantics to decide the next action, and every few steps the robot re-decides through semantic reasoning, sometimes through expensive calls to a vision–language model. The third family builds no map; instead, each instruction already implies the scene it describes. The instruction itself is the prior—scene-specific, human-given. Of these, only the first family decides on a top-down semantic map into which known walls and furniture can be written before the episode starts; we therefore compare against its most recent member, SGM, under the same priors.
Semi-Static Environments
DynaMem [7] adds and removes voxels as objects move, treating stable and movable parts of the scene alike. OpenIN [20] and Where Did I Leave My Glasses? [8] separate the two, through static carriers or per-instance stationarity, and let an LLM rank where to search. All three, however, maintain a map that includes the movable objects and choose their next goal during navigation. PreNav keeps only walls and furniture, which need no maintenance, and plans the whole search before the robot moves.
Diffusion Models for Planning
Diffusion models learn to generate outputs from a training distribution by iterative denoising [21]. In robot planning they serve at three levels: at the trajectory level, Diffuser [22] generates long-horizon state–action sequences; at the policy level, NoMaD [23] navigates on topological memory with a single diffusion policy; and at the level of combinatorial solving, TSPDiffuser [24] casts TSP as a denoising process over solution space, showing that diffusion can act as a learned sampler for combinatorial path planning. The last is closest to ours: the training labels likewise come from a combinatorial solver—TSP, the traveling salesman problem with neighborhoods, supplies the ground-truth path points—and we additionally learn a width at every path point, forming a band that marks where the target is likely to appear. This output is trained on room-level scenes, infers on floor-level scenes, and is validated on a real robot, with zero-shot and open-vocabulary queries supported by construction.
Methodology
PreNav separates search-reference generation from online perception. When a query comes, the search reference is generated in two steps. First, the furniture-conditioned search prior (§ 3-A) computes a spatial prior that encodes where the target is likely to be found relative to known furniture. Second, the diffusion-based global search generator (§ 3-B) takes this prior, the persistent wall raster, and the start pose as inputs and generates a single search reference. The robot follows this reference, detects the target online, and then approaches it; subsequent execution requires neither repeated high-level foundation-model calls nor query-conditioned global replanning. Figure 1 summarizes the complete data flow.

Figure 1. Overview of PreNav. (a) For a query , an LLM scores how useful each furniture category is for finding the target; the scores are computed once and cached. (b) The normalized weights , projected onto the mapped furniture, form the spatial prior , which is stacked with the wall raster and the start map . (c) Conditioned on these three channels, a diffusion model denoises random points into the search reference. (d) The reference on the real lab map: a centerline that fixes the visiting order, and a band (teal) that widens over high-scoring furniture (outlined). (e) Real-robot execution: viewpoints are sampled near the centerline to cover where the band overlaps furniture whose weight clears a threshold—the regions most worth searching, shown in violet. The robot visits them in order, detects the target with an open-vocabulary detector, and approaches it. Steps (a)–(d) run once, before the robot moves.
Furniture-Conditioned Search Prior
In a semi-static environment, the target’s current location is unknown, but its co-occurrence with persistent furniture provides a stable search prior. For example, a book is likely near a bookshelf, whereas an earphone is unlikely to appear near a refrigerator. We capture this through a two-step process: an LLM scores target–furniture co-occurrence, and the resulting weights are projected onto the spatial layout of the mapped furniture.
LLM co-occurrence scoring. Let denote the vocabulary of the supplied furniture prior. A query is free-form text that describes the target: an object category such as vase, a description such as tray with fruits, or a broader concept such as computer parts. For every persistent furniture category , an LLM is asked how useful is for finding the target under each of three spatial relations—the target resting on , contained in , or merely near —and returns the three relation scores together with a single search-usefulness score that summarizes them. Only enters the prior. We normalize it per query to obtain weights,
where is the standard deviation of . The temperature is set per query rather than globally because the LLM does not use a consistent range across queries: on the Gibson queries the top score is 95 for chair but 50 for oven, and a single temperature would leave the conservatively scored queries with a near-uniform prior. Scaling by the spread makes depend on how sharply the LLM separated the supports, not on the absolute numbers it happened to emit. We omit furniture categories with
when constructing the condition map. These scores are computed once per query and cached; a new query requires one run of the same scoring utility before the robot moves, rather than LLM calls during navigation.
Spatial prior construction. Each furniture instance with category is reduced to its extent on a 2D raster plane. It induces an axis-aligned anisotropic Gaussian centered at the box centroid, whose per-axis standard deviations are half the corresponding box extents, each lower-bounded at 1.5 cells so that small supports remain visible to the planning grid. The query-conditioned spatial prior is
is a spatial weighting over furniture supports rather than a target-occupancy prediction: the label generator seeks to cover the retained supports, while conditions the learned route and sets its supervised search-band width through Eq. (4)—for instance, a query for vase gives wider width targets near dining tables. Because the weights are assigned to furniture categories rather than instances, they remain applicable when an object relocates between compatible supports.
Diffusion-Based Global Search Generation
The search reference must capture long-range dependencies, such as which supports to visit and in what order, while remaining flexible to local geometric constraints. Diffusion models are well-suited for this task because they learn multimodal distributions over structured outputs through iterative denoising, avoiding the combinatorial complexity of explicit tour optimization at inference.
Problem formulation. The model receives a three-channel condition
where is the wall/free-space raster, is a Gaussian centered at the start pose with a standard deviation of 1.25 cells, and is the spatial prior from Eq. (2). It denoises a sequence whose rows are targets: is a centerline point on the planning grid and specifies the desired search half-width around that point. We refer to as the search reference: its centerline gives the visiting order, and its half-widths define the band swept around that centerline.
Training data and reference supervision. We generate structured reference labels offline from ScanNet++ [25] rather than training from embodied rollouts. Each training example contains a wall raster, persistent furniture instances, a start location, a target category, and an ordered search reference. The dataset contains 83 eligible scenes, split 75/8 into training and validation; Gibson [14] is reserved for evaluation and never used during training. For each empirical support-frequency vector, the data generator adds up to five randomly sampled support-weight variants, exposing the denoiser to diverse spatial priors rather than only the fixed co-occurrence table used at evaluation. During training, horizontal and vertical flips and rotations by multiples of are applied consistently to all condition channels and reference coordinates.
To produce the labels, we follow TSPDiffuser [24] and solve a TSP with neighborhoods [26] offline: around the start and each furniture center we take a free-space neighborhood, and CP-SAT [27] selects one point per neighborhood and a closed visit order under A* path costs [28]. Joining the selected points by their A* paths gives a loop from the start, which we cubically interpolate to samples clamped to the planning grid. At each sample, the width target is derived from the same spatial prior:
Here and grid cells (half-width). All three output channels and all condition channels are normalized to during training. TSP is used only to create these offline labels and is absent at inference.
Training objective. Let be such a TSP label, and its noised version at diffusion timestep , drawn uniformly from for each sample. The standard noise-prediction loss is
We additionally encourage the predicted centerline to avoid walls. For low-noise steps (), we reconstruct the denoised estimate and bilinearly sample the wall channel at the predicted centerline points. For a minibatch of samples, the implementation averages this penalty only over samples whose timestep satisfies :
where is the sampled wall value for sample . Thus, the auxiliary term is applied only to low-noise training timesteps (the late portion of the reverse process), where reconstruction of from the predicted noise is numerically better conditioned.
Architecture. A convolutional encoder maps the condition to a 768-channel feature grid; fixed 2D sinusoidal positions are added before the 256 map tokens are projected to . Each three-dimensional reference token is projected to the same width and receives a fixed 1D sinusoidal position and a learned timestep embedding. A four-layer pre-norm Transformer decoder (eight heads, 1,024-dimensional feed-forward layers) applies self-attention over the reference tokens and cross-attention to the map tokens, then predicts one three-dimensional noise vector per reference token. The network has 5.24M trainable parameters and uses 1,000 squared-cosine diffusion steps.
Inference. At deployment, the diffusion model uses the same channel order and normalization as during training. Starting from Gaussian noise, it produces one 100-point search reference in 50-step DDPM sampling. This is the only query-conditioned global decision made for the entire deployment.
Experiments
We evaluate whether persistent structural priors improve ObjectNav, how diffusion trades navigation performance against the generation cost of its TSP teacher, and whether the framework transfers to real-world object queries. The main comparison and subsequent analyses use a six-category subset; we also report the complete ten-category evaluation.
Experimental Setup
Datasets and benchmark
We evaluate on the five validation scenes of Gibson [14] v1.1: COLLIERVILLE, COROZAL, DARDEN, MARKLEEVILLE and WONICONISCO. Of the 15 annotated semantic categories, five serve as persistent furniture and the other ten as query targets, with 100 episodes each, 1,000 in total. Four of these targets, book, bottle, clock and cup, are limited by detection rather than by search; the main comparison and subsequent analyses therefore use the remaining six—chair, oven, potted plant, refrigerator, tv and vase—comprising 600 episodes. Sec. IV-C details the detector analysis behind this choice and reports the full ten-category results.
Comparison protocol. Within each evaluation set, all methods use the same episodes, a 500-action budget and the ObjectNav success criterion used by SGM [17] and PONI [16]. We report success rate (SR) and success weighted by path length (SPL) [15, 16], both as percentages. All ablation experiments use Mask R-CNN as the object detector, the same detector used by the baselines. To separate search from recognition, we further evaluate PreNav with stronger detectors and with a 3DSG oracle. Gibson’s object labels in Habitat come from the 3D Scene Graph (3DSG) annotation [29], which Habitat converts into a semantic mesh and renders through its semantic sensor; the 3DSG oracle replaces the detector with these rendered labels, which cover every object pixel in the current frame. SGM’s released code provides the same oracle. Success, in contrast, is judged on the top-down GT semantic map of the ObjectNav dataset [15], a separate annotation that does not fully coincide with 3DSG (Sec. IV-C).
SGM’s pretrained semantic model supports the dataset’s original 15 semantic categories, including five that we treat as furniture: dining table, couch, bed, toilet and sink. For a fair comparison, we supply both SGM and PreNav with the same wall prior and restrict their furniture priors to these five categories ( in Table I and Table II). PreNav also accepts open-vocabulary furniture categories through its spatial-prior interface, without retraining the generator. We therefore additionally evaluate it with the full-scene furniture prior (), which adds the furniture that the scenes leave unlabeled, annotated by us as 2D boxes; this is a comparison within PreNav to measure the benefit of additional furniture information.
| Method | Wall | Furniture | SR ↑ | SPL ↑ |
| SGM | × | × | 55.0 | 29.6 |
| × | 60.5 | 33.1 | ||
| × | 61.3 | 36.2 | ||
| 62.0 | 36.2 | |||
| Ours | 64.3 | 39.2 | ||
| 70.3 | 39.8 |
Table I. ObjectNav on the six-category Gibson subset (600 episodes). ×: no prior; : wall or full-scene furniture prior; : furniture categories supported by SGM.
| Method | Wall | Furniture | SR ↑ | SPL ↑ |
| SGM | × | × | 36.4 | 22.0 |
| × | 40.6 | 25.5 | ||
| × | 39.9 | 24.1 | ||
| 42.8 | 26.9 | |||
| Ours | 44.2 | 26.9 | ||
| 44.5 | 26.1 |
Table II. ObjectNav on the full ten-category Gibson set (1,000 episodes). ×: no prior; : wall or full-scene furniture prior; : furniture categories supported by SGM.
Implementation details. The denoiser is trained for 800 epochs using AdamW at , a cosine learning-rate schedule with 100 warm-up steps, gradient clipping at 1.0, and train/validation batch sizes of 32/8. We evaluate in Habitat [30] under the common action-budget protocol. The agent carries a 640×480 RGB-D sensor ( HFOV, 0.88 m height) with discrete actions (0.25 m forward and turns) and a 5 cm occupancy grid. Category-specific perception uses Mask R-CNN R-50-FPN-3x [31] trained on COCO [32]; score thresholds are 0.3 (ten-category) and 0.7 (six-category), applied identically to all methods. The co-occurrence scores are produced once by DeepSeek V4 Pro and normalized with the per-query softmax of Eq. (1). The diffusion model samples one 100-point search reference in 50 denoising steps, with the half-width channel clipped to [1, 12] grid cells.
Diffusion and TSP receive identical semantic inputs. All experiments run on an NVIDIA RTX 4090 24GB; generator timing is measured on an AMD Ryzen 9 9950X with 16 single-thread workers (Table IV).
| Method | Ref. gen. (s) Mean ↓ | Ref. gen. (s) P95 ↓ | Mask R-CNN SR ↑ | Mask R-CNN SPL ↑ | 3DSG oracle SR ↑ | 3DSG oracle SPL ↑ |
| TSP | 25.705 | 169.893 | 71.0 | 39.6 | 82.7 | 44.0 |
| Diffusion | 0.995 | 1.029 | 70.3 | 39.8 | 82.5 | 46.6 |
Table IV. Search-reference generation cost and navigation performance comparison
Baseline Comparison
SGM benefits from structural priors. The walls and furniture we supply to SGM are taken from the top-down GT semantic map. SGM is itself trained to complete such maps from partially masked ones, so giving it part of the GT map is exactly the input its training prepares it for. Furniture alone raises its SR by 5.5% and SPL by 3.5%, and walls alone by 6.3% and 6.6% (Table I); with both, its SR and SPL rises by 7.0% and 6.6% over the original SGM. These priors are therefore informative, and SGM exploits them as its training intends.
PreNav exploits the same priors better, and gains from more furniture. SGM receives furniture as a pixel-level semantic map, whereas PreNav needs only a 2D bounding box per piece (§ 3-A), a coarser description. Even so, with the same walls and five furniture categories, PreNav outperforms SGM by 2.3% in SR and 3.0% in SPL. To show that furniture drives this search, we additionally annotate, as 2D boxes, the furniture that the scenes leave unlabeled; it lies outside the 15 annotated categories, and hence outside what SGM’s semantic model supports. This full-scene prior raises the SR of PreNav by another 6.0% and its SPL by 0.6%. PreNav thus exploits both the layout and the semantics of walls and furniture, and since each additional piece costs only a 2D box, supplying more furniture is a cheap way to make the target more likely to be found.
The Full Ten-Category Benchmark
Why the main analysis uses six categories. The Gibson ObjectNav data annotate 15 semantic categories, but the validation set queries only six of them, and three of these—couch, bed and toilet—are furniture in our setting, leaving only three targets. We therefore extend the query set: of the 15 categories, five serve as furniture and the other ten as targets, with 100 episodes each, 1,000 in total. Detector ablation (Table III) shows that four of the added targets are limited by perception rather than by search. On book, bottle, clock and cup, PreNav succeeds in only 21.2% of episodes with Mask R-CNN; the stronger SAM 3, which improves the other six categories by 3.8%, lowers them by 9.4%, and even the 3DSG oracle raises them only to 47.0%. Part of this gap comes from the annotations themselves: the oracle and the success criterion read different annotations, which disagree even at the category level. In the validation scene COLLIERVILLE, the 3DSG annotation contains two bottles that the top-down GT semantic map lacks, although both annotations mark the cup on the shelf below (Figure 2). Conversely, in WICONISCO the map contains a clock while the scene’s 3DSG annotation has no clock class, so the oracle can never report the target in the 57 clock episodes set there. This is why Mask R-CNN exceeds the 3DSG oracle on clock in Table III (34% vs. 28%): in WICONISCO it succeeds in 32 of these 57 episodes, the oracle in none. To measure navigation rather than recognition while maintaining comparability with the Object-Nav benchmark, we report the remaining six categories as the main benchmark and the full ten categories here.
| Category | Mask R-CNN | SAM 3 | 3DSG oracle |
| chair | 70 | 73 | 98 |
| oven | 79 | 91 | 96 |
| potted plant | 80 | 54 | 81 |
| refrigerator | 41 | 71 | 75 |
| tv | 54 | 57 | 74 |
| vase | 33 | 34 | 66 |
| book | 29 | 18 | 53 |
| bottle | 20 | 16 | 67 |
| clock | 34 | 12 | 28 |
| cup | 2 | 1 | 40 |
| Six-category subset | 59.5 | 63.3 | 81.7 |
| Other four | 21.2 | 11.8 | 47.0 |
| All 1,000 ep. | 44.2 | 42.7 | 67.8 |
Table III. Comparison of per-category success rates (%) across different detectors. Bold compares Mask R-CNN and SAM 3. 3DSG oracle: per-pixel labels rendered from the 3DSG annotation.

Figure 2. The 3DSG oracle and the success criterion read different annotations (COLLIERVILLE). (a) RGB view of a shelf, with each 3DSG object boxed. (b) The 3DSG oracle labels two bottles and a cup. (c) The top-down GT semantic map used to judge success marks the cup (right) but has no bottle anywhere in the scene; 3DSG objects are projected onto it for comparison, and the wedge marks the camera’s field of view.
Results on all ten categories. The same priors again help SGM (Table II): furniture alone raises its SR by 4.2% and SPL by 3.5%, walls alone by 3.5% and 2.1%, and both together by 6.4% and 4.9%. Under these priors, PreNav outperforms SGM by 1.4% in SR with equal SPL, and the full-scene furniture prior adds a further 0.3% in SR, at 0.8% lower SPL. This slightly lower SPL indicates that the additional prior information leads to a few more found objects, but at the expense of generally slightly longer routes.
Room-to-Floor Transfer and Efficiency
We evaluate whether a search-reference generator trained on room-level data transfers to floor-level scenes while retaining navigation performance close to its teacher. Diffusion is trained only on ScanNet++ and applied directly to Gibson, without training or fine-tuning on Gibson. We compare it with the TSP teacher planner solving the Gibson instances again. Both use the same full-scene furniture prior, and six-category protocol with Mask R-CNN. TSP uses candidates per neighborhood and a 300 s solver time limit.
Approaching teacher performance at lower generation cost. With 25.8× less generation time than TSP on average (165× less at P95), diffusion achieves only 0.7% lower SR and even gains 0.2% in SPL with Mask R-CNN (Table IV); with the 3DSG oracle, it drops 0.2% in SR and gains 2.6% in SPL. When transferring our diffusion model from the room level to the floor level, navigation performance drops by less than 1% compared to our TSP teacher, yet SPL actually improves. More importantly, we avoid the high computational costs associated with solvers, and our model’s execution time does not increase with the number of furniture items, making it far more feasible for real-world deployment.
Planning: the reference covers the target. We first examine the references themselves, on the raw generator output. Let denote the region of the queried category in the top-down GT semantic map, projected onto the planning grid, shown as the GT target in Figure 4(a,Figure 4b), where it comprises all potted plants in the scene; it enters only for evaluation. Sweeping each band along its centerline, we record the reference length at which the band first contains a wall-free cell of (Figure 4(c)). By the end of the reference, the band contains the target in 76.7% of episodes for diffusion and 78.8% for TSP, and the two curves stay close over the first 20 m, within which most of this coverage is reached. The learned references are also shorter, 23.1 m against 30.4 m on average and 52.4 m against 61.1 m at most, where each curve ends.

Figure 4. Search references of diffusion and its TSP teacher on the 600 six-category episodes. (a,b) Search references of diffusion and TSP for one episode in the Gibson validation scene DARDEN, with the query potted plant. The native band is the region swept by each generator’s own half-widths along its centerline; the purple GT targets are the potted plants in the scene, overlaid for evaluation only. (c) Planning: share of episodes whose band, swept along the reference, contains the target, against reference length; each curve ends with that generator’s longest reference. (d) Execution: share of episodes that succeed within a distance traveled, under Mask R-CNN.
Execution: success per distance traveled. For a travel budget , we count the episodes that succeed within meters of travel under Mask R-CNN (Figure 4(d)). The two curves are almost identical: both reach 50% success before 10 m, and about 95% of successful episodes finish within 20 m, matching the planning curves in Figure 4(c). A reference that covers the target thus also lets the agent find it in execution.
Failure Analysis
This section addresses the following question: What limits navigation success when using the generated search reference? We analyze the 600 episodes of PreNav on the six-category subset with the full-scene furniture prior and Mask R-CNN (70.3% SR, Table I), grouping each episode by target detection and execution outcome (Fig. 5). For target detection, we back-project each detection with depth onto the 2D map and take the agent’s last estimate of the target position, computing its minimum distance to , the queried category’s region in the top-down GT semantic map (Sec. IV-D). A distance of at most 1 m counts as a true positive. The detected wrong target group contains the remaining detections: either the estimated position lies more than 1 m from , or no position could be estimated, for example because the detection mask contained no valid depth. For execution, an episode succeeds, stops at an incorrect position, exhausts its 500 action budget, or exhausts the reference without a detection. Success follows almost entirely from a correct target estimate: 421 of the 422 successful episodes are true positives.
Failures, in turn, come mainly from target detection. The detected-wrong-target group (115 episodes, 19.2%) is the largest source, and its main flow, 72 episodes, ends at an incorrect stop position, where the agent stops at an object mistaken for the target. The few true positives that fail do so in the final stop. Failures traceable to the search reference lie within the never-detected group, only 30 episodes (5.0%), 23 of which (3.8%) complete the reference without a detection. Target detection, rather than the search reference, therefore limits navigation success.
Real-World and Open-Vocabulary Evaluation
We deploy PreNav on a Fetch mobile robot with a head-mounted Intel RealSense D435i RGB-D camera, using SAM 3 [33] as the open-vocabulary detector. The test scene is a cluttered laboratory of over 200 m, half office area and half leisure area (Figure 3). Its walls come from an osmAG map [9] aligned to the robot’s occupancy grid, and its 21 furniture pieces are annotated once as boxes; the positions of the queried objects are never given.
Figure 3. Real-world test scene, a laboratory of over 200 m. (a) The Fetch robot that used for real-world experiments, photographed in the test scene. (b) The room’s osmAG map [9] with the 21 annotated furniture boxes, the only semantic input, colored by category. (c) A 3D scan of the same room with the osmAG walls overlaid.
For each query, the LLM scores over the annotated furniture give , and the generator produces the search reference as in simulation. Execution differs in one aspect: instead of following waypoints along the reference, the robot samples viewpoints near the centerline to cover where the band overlaps furniture whose weight clears a threshold, and navigates to them one by one until the target is detected and approached. The prior and the generator are unchanged.
The queries include tray with fruits, robot arm and quadruped robot, which lie outside the COCO vocabulary of the simulation detector. PreNav succeeds in 13 of 15 trials. One failure mode is an object placed far from any furniture: a quadruped robot left on the floor lay outside the covered region, so none of the viewpoints brought it into view.
Conclusion
We presented PreNav, which decides where to search before the robot moves: a conditional diffusion model generates one global search reference from persistent structure and an LLM-derived furniture prior. Under matched structural priors it outperforms SGM on Gibson; trained only on room-level ScanNet++, it transfers zero-shot to floor-level scenes close to its TSP teacher at a fraction of the generation cost, and it demonstrates performance on real-robot with open-vocabulary queries.
PreNav assumes that the target lies on or near furniture. This holds for most everyday objects, but an object left far from any furniture falls outside the band and is not found, as happened with a quadruped robot lying on the open floor of our lab. A natural extension is to search the free space that the reference leaves uncovered when the reference is exhausted.
References
- [1]D. Batra et al., “ObjectNav revisited: On evaluation of embodied agents navigating to objects,” 2020, arXiv:2006.13171.arxiv.org/abs/2006.13171
- [2]P. Anderson et al., “On evaluation of embodied navigation agents,” 2018, arXiv:1807.06757.arxiv.org/abs/1807.06757
- [3]A. Werby, C. Huang, M. Büchner, A. Valada, and W. Burgard, “Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation,” in Proc. Robot.: Sci. Syst., 2024.DOI
- [4]Q. Gu et al., “ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 5021–5028.arxiv.org/abs/2309.16650
- [5]N. H. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “VLFM: Vision-language frontier maps for zero-shot semantic navigation,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 42–48.arxiv.org/abs/2312.03275
- [6]E. Padilla Cerdio, B. Sun, M. Pollefeys, and H. Blum, “OpenFrontier: General navigation with visual-language grounded frontiers,” in Proc. Robot.: Sci. Syst., 2026.
- [7]P. L. Liu et al., “DynaMem: Online dynamic spatio-semantic memory for open world mobile manipulation,” in Proc. IEEE Int. Conf. Robot. Automat., 2025, pp. 13 346–13 355.arxiv.org/abs/2411.04999
- [8]B. Bogenberger et al., “Where did I leave my glasses? Open-vocabulary semantic exploration in real-world semi-static environments,” IEEE Robot. Automat. Lett., vol. 11, no. 3, pp. 3342–3349, Mar. 2026.
- [9]D. Feng, C. Li, Y. Zhang, C. Yu, and S. Schwertfeger, “osmAG: Hierarchical semantic topometric area graph maps in the OSM format for mobile robotics,” in Proc. IEEE Int. Conf. Robot. Biomimetics, 2023, pp. 1–7.arxiv.org/abs/2309.04791
- [10]F. Xie and S. Schwertfeger, “Robust lifelong indoor LiDAR localization using the area graph,” IEEE Robot. Automat. Lett., vol. 9, no. 1, pp. 531–538, Jan. 2024.DOI
- [11]F. Xie and S. Schwertfeger, “Empowering robot path planning with large language models: osmAG map topology & hierarchy comprehension with LLMs,” in Proc. IEEE Int. Conf. Robot. Biomimetics, 2024, pp. 707–712.arxiv.org/abs/2403.08228
- [12]F. Xie, J. Zhang, and S. Schwertfeger, “Intelligent LiDAR navigation: Leveraging external information and semantic maps with LLM as copilot,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2025, pp. 20 402–20 409.arxiv.org/abs/2409.08493
- [13]F. Xie, S. Schwertfeger, and H. Blum, “osmAG-LLM: Zero-shot open-vocabulary object navigation via semantic maps and large language models reasoning,” IEEE Robot. Automat. Lett., vol. 11, no. 3, pp. 2426–2433, Mar. 2026.
- [14]F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson Env: Real-world perception for embodied agents,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 9068–9079.arxiv.org/abs/1808.10654
- [15]D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 4247–4258.arxiv.org/abs/2007.00643
- [16]S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, “PONI: Potential functions for objectgoal navigation with interaction-free learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 18 868–18 878.arxiv.org/abs/2201.10029
- [17]S. Zhang, X. Yu, X. Song, X. Wang, and S. Jiang, “Imagine before go: Self-supervised generative map for object goal navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 16 414–16 425.DOI
- [18]S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “CoWs on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 23 171–23 181.arxiv.org/abs/2203.10421
- [19]A.-C. Cheng et al., “NaVILA: Legged robot vision-language-action model for navigation,” in Proc. Robot.: Sci. Syst., 2025.arxiv.org/abs/2412.04453
- [20]Y. Tang et al., “OpenIN: Open-vocabulary instance-oriented navigation in dynamic domestic environments,” IEEE Robot. Automat. Lett., vol. 10, no. 9, pp. 9256–9263, Sep. 2025.arxiv.org/abs/2501.04279
- [21]J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 6840–6851.arxiv.org/abs/2006.11239
- [22]M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine, “Planning with diffusion for flexible behavior synthesis,” in Proc. Int. Conf. Mach. Learn., 2022, pp. 9902–9915.arxiv.org/abs/2205.09991
- [23]A. Sridhar, D. Shah, C. Glossop, and S. Levine, “NoMaD: Goal masked diffusion policies for navigation and exploration,” in Proc. IEEE Int. Conf. Robot. Automat., 2024, pp. 63–70.arxiv.org/abs/2310.07896
- [24]R. Yonetani, “TSPDiffuser: Diffusion models as learned samplers for traveling salesperson path planning problems,” in Proc. IEEE Int. Conf. Robot. Automat., 2025, pp. 10 824–10 830.
- [25]C. Yeshwanth, Y.-C. Liu, M. Nießner, and A. Dai, “ScanNet++: A high-fidelity dataset of 3D indoor scenes,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2023, pp. 12–22.arxiv.org/abs/2308.11417
- [26]E. M. Arkin and R. Hassin, “Approximation algorithms for the geometric covering salesman problem,” Discrete Appl. Math., vol. 55, no. 3, pp. 197–218, 1994.DOI
- [27]L. Perron, F. Didier, and S. Gay, “The CP-SAT-LP solver (invited talk),” in Proc. Int. Conf. Princ. Pract. Constraint Program., 2023, pp. 3:1–3:2.DOI
- [28]P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” IEEE Trans. Syst. Sci. Cyber., vol. 4, no. 2, pp. 100–107, 1968.
- [29]I. Armeni et al., “3D scene graph: A structure for unified semantics, 3D space, and camera,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 5664–5673.arxiv.org/abs/1910.02527
- [30]M. Savva et al., “Habitat: A platform for embodied AI research,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 9338–9346.arxiv.org/abs/1904.01201
- [31]K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2980–2988.
- [32]T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 740–755.arxiv.org/abs/1405.0312
- [33]N. Carion et al., “SAM 3: Segment anything with concepts,” in Proc. Int. Conf. Learn. Representations, 2026.arxiv.org/abs/2511.16719
