WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Introduction

HUMANS navigate unfamiliar cities with little effort. Given a destination, a person consults a web map and follows its turn-by-turn directions, such as continuing along a sidewalk, turning onto a crosswalk or taking the stairs. Neither a metric goal nor precise position is required. The map indicates the next manoeuvre and its approximate location, while vision resolves where the crosswalk begins and when a turn has been completed. Since web maps provide such directions for nearly every place where people walk, a robot that follows them in the same manner could be deployed in any city given only a destination and without a prior mapping run, enabling applications such as last-mile delivery, urban inspection and guidance of visually impaired people.

Existing learned navigation methods fall short of this goal over long routes. Point-goal methods [1, 2, 3, 4, 5] receive the route as a sequence of GNSS waypoints, and their performance is therefore bounded by the accuracy of the position estimate. In cities, GNSS errors of several metres are common [6], enough to place a waypoint on the road or on the opposite side of the street, and over routes of several hundred metres such errors inevitably lead to failures. Waypoints also carry no notion of the social norms, such as staying on the sidewalk and crossing at crosswalks, so a policy that reaches every waypoint may still violate them and pose a safety risk in urban traffic.

Vision-language navigation (VLN) replaces coordinates with natural-language instructions, but it has been studied predominantly in simulated indoor environments [7, 8] and on street-view panoramas [9]. Recent models bring vision-language navigation to real robots [10, 11, 12, 13], but they are typically demonstrated on single instructions over short distances. We find that these models are brittle even at the level of a single atomic instruction (§ 10-D), which makes it infeasible to follow a kilometre-long route by chaining such instructions.

In this work, we simplify the problem. Web-map directions already decompose a route into a sequence of atomic instructions, each describing a single manoeuvre such as “cross the road” that can be grounded in the camera image. Long-range navigation thus separates into deciding which atomic instruction applies at the current moment and executing that instruction reliably. The first problem requires the map and contextual reasoning but only a coarse position, since GNSS is needed only to identify the current step of the route rather than the exact position of the robot. The second problem requires only the camera and a short observation history. Language is a suitable interface between the two levels, since it is what web maps produce and what vision-language models (VLMs) reason in. Based on this observation, we propose WayTrace, a map-to-language-to-trace hierarchy (Figure 1). An off-the-shelf orchestrator VLM reads the web-map directions, a satellite map of the surroundings and the recent camera images, and issues the next atomic command. An onboard trace predictor draws the path for this command in image space, and a reinforcement learning (RL) policy tracks the path lifted to 3D.

We evaluate WayTrace on ten long-range missions in a simulated Zurich old town and in real-world deployments. With perfect GNSS, WayTrace completes 83% of the missions. It achieves the highest success rate at every level of GNSS error, and its success rate decreases gradually to 63% with a drifting GNSS error of 7 m RMS, whereas that of point-goal baselines drops below 25% and they leave the crosswalk or the sidewalk about ten times as often. Vision-language baselines complete at most 7% of the missions, even with language commands from a tuned oracle. In the real world, WayTrace completed three missions of up to 426 m through a busy city centre fully autonomously, and a 1.37 km mission along rural trails and through a university campus with two short operator interventions. The main contributions of this work are as follows.

• WayTrace, a hierarchical navigation framework that translates web-map directions into atomic language commands and executes them as image-space traces.

• A language-conditioned trace predictor together with an automatic labelling pipeline that produces most of its training data from unlabelled street-level image sequences.

• An extensive evaluation against point-goal and vision-language navigation baselines in simulation, an analysis of each component and real-world deployments on a wheeled-legged robot.

We will release the simulation environment, the long-range navigation benchmark, the trace prediction benchmarks, the dataset, the models and the code.

Related Work

Long-Range and Map-Guided Outdoor Navigation

Navigation foundation models learn goal-conditioned policies from data pooled across robots and environments [14, 15, 16], and recent work scales point-goal navigation with crowdsourced robot data [17, 18] or with walking videos from the web [2, 19]. Vision-language-action models for navigation additionally accept GNSS poses, images or language as goals [4]. VAMOS [3] plans candidate paths in the image with a VLM and re-ranks them with an embedding-specific affordance model. Over long distances, these policies are guided by GNSS goals or waypoints, so the route they follow is only as accurate as the position estimate.

Several systems follow routes from map services. ViKiNG [20] reached kilometre-scale goals by using overhead images as heuristics for a learned local policy, and Sorokin et al. [21] followed public map routes with a legged robot while staying on sidewalks. UrbanVLA [22] aligns noisy route waypoints from a map service with the camera observations in an end-to-end policy. Walk With Me [23] lets a VLM ground instructions in map data and GNSS as waypoints for a low-level policy and defers difficult situations, such as crossings, to the VLM. DA-Nav [24] reduces web-map directions to four discrete commands that a VLM grounds as cells in the image, and NaViRратор [25] verbalises a map route once into an instruction for a VLN policy. In WayTrace, the route is converted into atomic language commands in closed loop. Each command is grounded in the current observation rather than in metric coordinates, and hence GNSS errors do not displace the path that the robot follows.

Vision-Language Navigation

Vision-language navigation was established on simulated indoor benchmarks [7], [8], [26] and on street-view graphs of real cities [9], [27, 28, 29], where an agent follows a paragraph of instructions written for the whole route. Video-based VLMs transfer such models to real robots [30, 31, 32], and recent navigation foundation models scale this approach to millions of samples across tasks and embodiments, unifying instruction following, object search and point goals in a single model [33, 34, 35, 36, 37, 38]. NaVILA [12] lets a VLM issue metric language actions that an RL locomotion policy executes on a legged robot, and UrbanNav [39] learns language-guided urban navigation from city-walking videos. TIC-VLA [10] couples language reasoning with control and accounts for the delay of its reasoning in dynamic environments. Foresight [11] lets a VLM propose motion plans in the image and iteratively critique and refine them. VENTURA [13] adapts an image diffusion model to generate path masks that a lightweight policy executes. These models execute one instruction or goal at a time, which a user or a higher-level agent has to provide [36]. WayTrace keeps each instruction atomic and lets an orchestrator decide which instruction applies along the route.

Hierarchical Systems with Language as the Interface

Decomposing long-horizon tasks into language subtasks for a low-level policy is well established in manipulation. In SayCan [40], a language model proposes skills that are scored by affordance functions, and RT-H [41] predicts language motions as an intermediate step between the task and the actions. Hi Robot [42] uses a high-level VLM to issue atomic commands to a vision-language-action model, while $\pi_{0.5}$ [43] uses a single model that first predicts the next subtask in language and then generates the actions for it. In MEM [44], a high-level policy predicts the next subtask together with a language memory of task progress, and a low-level policy executes it with a short-term visual memory. In navigation, LM-Nav [45] parses instructions into landmarks on a previously built topological graph, and Mobility VLA [46] selects a goal frame from a recorded tour. NavMCP [47] lets a VLM agent invoke a navigation foundation model for long-horizon exploration. VbotNav-0 [48] and AsyncVLA [49] split navigation between a remote and an onboard policy, which exchange waypoints and latent action features, respectively, rather than language. WayTrace uses language between an off-the-shelf orchestrator and an onboard trace predictor, which requires no prior visit of the environment and keeps every decision interpretable.

Image-Space Trace Prediction and Automatic Labelling

Predicting a sequence of 2D waypoints in image space, referred to as a trace, to accomplish a navigation task has recently gained popularity with VLMs. NaviTrace [50] benchmarks VLMs on predicting such traces in the image for a given instruction and embodiment. Recent navigation models predict either a trace [3], [11], [13], [51], [52] or only the next waypoint as a point in the image [37], [38]. Following this line of work, WayTrace uses image-space traces as the interface between the language command and the low-level controller, since they can be learned at scale without robot data. Training a trace predictor, however, requires traces that match a specific instruction. Prior work derives navigation labels from web videos [2], [39], [53] or from monocular reconstruction of egocentric video [54]. We, instead, build an automatic labelling pipeline that derives such command-specific traces from unlabeled street-level image sequences [55] with 3D reconstruction [56] and VLM labelling.

Method

Problem Formulation and Overview

A robot has to reach a destination $d$ in a city, given as text, while it moves on sidewalks and crosses roads at crosswalks. At time $t$, it observes an RGB image $I_t$ and a depth image $D_t$ from a front camera, its velocity $v_t$ and pose from odometry and a GNSS position that may be several metres wrong, and it outputs base velocity commands $a_t$. A web map provides a walking route with turn-by-turn directions, and no metric map of the environment is built beforehand.

Neither the route nor the GNSS position is accurate enough to be tracked geometrically. WayTrace therefore expresses the route as a sequence of atomic commands $c \in\mathcal{C}$. An atomic command describes a single manoeuvre whose extent is visible from the current viewpoint, such as “Turn left at the intersection” or “Walk to the bench on the right”, without referring to positions or distances. Navigation then separates into deciding which command applies at the current moment and executing this command. WayTrace solves the first problem with an orchestrator VLM $\mathcal{O}$ and the second with a trace predictor $f_{\theta}$ and an RL navigation policy $\pi$ (Figure 2),

where $x^{\mathrm{o}}$ and $x$ are the inputs of the orchestrator and the trace predictor, $\tau_t$ is a trace of points in the current image, $\mathbf{q}_t$ is a goal point on the trace lifted to 3D, and $\Delta_{\mathcal{O}} \approx1.3\,\mathrm{s}$ and $\Delta_{\tau} \approx1.5\,\mathrm{s}$ are the latencies of the two models. The levels run asynchronously, and the policy follows the latest trace at 5 Hz, so that the robot keeps moving while new commands and traces are computed.

Route and Orchestrator VLM

The route comes from the Google Routes API and is replanned on the pedestrian network of OpenStreetMap, so that it uses sidewalks and mapped crossings. It is divided into directions in the style of the web map, such as “Turn left onto the crosswalk and cross the road”, each attached to a position along the route. The GNSS position, projected onto the route, only selects the next direction and its approximate distance (App. Sec. A).

Every 2 s, an off-the-shelf VLM decides whether the current command still applies or the next manoeuvre is due. Its input $x^{\mathrm{o}}_t$ contains the destination, the nearby directions with their distances, a heading-up satellite image of 50 m width with the route and the driven path, the camera images of the last 15 m and the current command. It answers with a short reasoning followed by the command. The prompt (App. Sec. F) instructs it to determine which manoeuvre comes next and on which side from the directions and the map, and when it is due from the camera images, since distances derived from GNSS may be wrong by 10 to 20 m. A few rules on the robot reject commands that are issued far too early (App. Sec. A).

Language-Conditioned Trace Predictor

The trace predictor executes the current command as a trace in the camera image, which can be learned without robot data and lifted to 3D with depth. Its input $x_t$ consists of five past images taken 1.8 m of travel apart, the current image $I_t$ and a top-down map $B_t$ of the path walked over the last 16 m. The history lets the model recognise which part of the command has already been executed, for instance that a commanded turn is complete. The output is a goal point $\mathbf{g}$ and a trace $\boldsymbol{\tau} = (u_1, \ldots, u_N)$ of $N = 10$ points in normalised image coordinates that ends at the goal, where the goal is the point at which the commanded path leaves the view or becomes occluded.

The model is initialised from RynnBrain-9B [57], writes the coordinates as text and predicts the goal before the trace,

Predicting the goal first commits the model to a single branch at a junction, whereas a model that predicts the trace directly tends to hedge between branches. Both stages are trained by supervised fine-tuning with the autoregressive cross-entropy loss, the second stage with the reference goal in the prompt.

For deployment, a single call replaces the two stages. The goal of about 15 tokens is decoded autoregressively, and an MLP head $h_{\phi}$ regresses the trace $\hat{\boldsymbol{\tau}} = (\hat{\mathbf{u}}_1,\ldots,\hat{\mathbf{u}}_N) = \sigma(h_{\phi}(\mathbf{e}))$ from the hidden state $\mathbf{e}$ at the end of the goal, which avoids decoding about 140 trace tokens. The head is trained with

where $\mathbf{u}_i$ are the points of the reference trace, the second term adds weight to the end point, which is the goal, $\mathcal{L}_{\mathrm{CE}}$ is the cross-entropy on the goal tokens, and $\lambda_g$ and $\lambda_{\mathrm{CE}}$ are weights (App. Sec. A). The single-call model performs on par with the two-stage model and runs onboard the robot (Section IV-E).

Training Data with Intent-Conditioned Traces

Training requires samples $(x,c,\boldsymbol{\tau})$ whose trace follows the command. Projecting the future path of walking sequences into earlier frames yields traces at scale [2], [39], [53], [54], but the intent of the walker is unknown, many crowd-sourced sequences have no clear destination, and every image comes with a single trace, namely the path that was walked. A model trained only on such data can associate an image with a path while ignoring the command, and it never sees the branch that was not taken or a path towards an object. We therefore combine reconstructed walks, which provide real temporal context, with intent-conditioned traces, which pair the same image with several commands, for example “Take the left fork” and “Take the right fork” (Figure 3). Both are built from crowd-sourced street-level sequences [55] (51,907 sequences from 182 cities), and the test sequences are excluded (App. Sec. B).

Reconstructed walks. We select episodes at junctions, turns and straight segments, reconstruct each sequence with VGGT-$\Omega$ [56], scale it with the GNSS arc length and project the ground below the future camera positions into every frame. The trace is cut where a building occludes it, where it leaves the image and where the walker turns back. Since Mapillary images are not taken at uniform intervals, consecutive frames are 1 to 2 m apart, which also varies the spacing of the history. A VLM writes the atomic command that the drawn path depicts together with four paraphrases, and a second VLM pass keeps only the episodes whose command and path agree. This yields 288k samples from 26,740 episodes.

Intent-conditioned traces. We first train a single-image teacher $f^{T}$ with the two-stage formulation (2) on the training split of NaviTrace [50] and on 1.9k traces that we drew by hand for several commands on each of 605 Mapillary images, at about 7 s of labelling effort per trace. We then mine anchor images at junctions, crossings and trails, and a VLM drafts 3.4 commands per kept image on average, one for each visible branch or crossing direction and up to two towards unique landmarks. The teacher predicts a trace for every command, and the reconstruction propagates it into the subsequent frames of the sequence, which provides the history that the teacher does not see. This yields 863k samples from 106k images. Together with the hand-labelled traces, the dataset contains 1.19M samples. Adding the intent-conditioned samples improves the open-loop accuracy monotonically (§ 10-D).

Execution with an RL Navigation Policy

A new trace becomes available only every 1.5 to 3 s, and it describes the scene at the time its image was captured. In the meantime, the robot moves on its path and pedestrians or vehicles may enter its path. The trace predictor also reasons about the route rather than about the clearance of the robot, so a trace can pass close to a pole or through a parked bicycle. The trace therefore cannot be executed open loop. It requires a low-latency controller that knows the embodiment, tracks the trace and avoids obstacles on its own.

For this, we lift the trace to 3D with the depth image, anchor it in the odometry frame and keep only its prefix on observed walkable ground (App. Sec. A). A carrot $g_t$ moves along the lifted path 4.5 m ahead of the robot and serves as the goal of an RL navigation policy [5]. The policy is trained in simulation to reach point goals from depth images while avoiding obstacles, and it runs at 5 Hz on the latest depth image and the velocity of the robot, so that the robot keeps following the previous path while a new trace is computed. A new trace is requested on a new command, after 3 m of travel, after 40 % of the trace has been executed or after 5 s. Under the final command, the robot stops once the goal is within 1.5 m.

Experimental Setup

We evaluate the full system in closed loop in a simulation benchmark, where all methods run on the same missions and GNSS errors, and on a real robot, and the trace predictor in open loop against reference traces.

Closed-Loop Simulation Benchmark

We reconstruct the Zurich old town in Isaac Sim 5.0 from Swiss federal geodata, OpenStreetMap and street-level imagery (Figure 4). The static scene covers 0.91 km$^{2}$ with 123 zebra crossings and 880 shops and restaurants, so that missions can end at named destinations. The robot is a simulated Unitree B2W wheeled-legged robot. The benchmark consists of ten missions between 152 and 864 m, 4.68 km in total, with 14 junction turns, 2 forks, 14 road crossings and one staircase (App. Sec. C). GNSS errors are emulated by a slowly drifting bias with a radial RMS of 2, 5 or 7 m, and each method runs three seeds per mission, giving 30 runs per condition.

Metrics. A run succeeds when the robot stops within 5 m of the destination and crosses every road at the zebra crossing, that is, at least half of its path over the road at each crossing of the route lies on the zebra. We additionally report the crosswalk violation, the share of crossings taken outside the zebra crossing, and the road violation, the share of the driven path on which the robot is on the road instead of the sidewalk.

Closed-Loop Baselines

We compare against several state-of-the-art methods with publicly available models. The point-goal navigation methods CityWalker [2], VAMOS [3] and OmniVLA [4] follow the route as GNSS waypoints with their released control loops. To compare path prediction independently of the controller, we also pair CityWalker and VAMOS with our lifting and RL policy. The language-conditioned methods TIC-VLA [10], Foresight [11] and VENTURA [13] completed no mission with our orchestrator VLM, so they receive their commands from an oracle orchestrator that issues each command at a predefined position along the true route, tuned for each method. To separate the errors of the orchestrator from those of the rest of the hierarchy, we also run WayTrace with such an oracle (App. Sec. C).

Open-Loop Trace Prediction

Each sample consists of an atomic command, the current image with its history and walked-path map, and a reference trace. We use two test sets of Mapillary images from sequences excluded from training (App. Sec. D). Human-Test contains 953 hand-drawn traces on 332 images with several commands per image, including landmarks. Walks-Test contains 31,012 samples, all frames of 2,869 held-out episodes of reconstructed walks, labelled by the same automatic pipeline as the training data.

Metrics. We report the NaviTrace (NT) score [50], a metric for image-space traces that is aligned with human preferences. It combines the dynamic time warping distance to the reference trace, the final displacement error and a semantic penalty for surfaces that should be avoided. The score is normalised so that a perfect trace scores 100 and a straight vertical trace through the image centre scores 0 on average. We further report the average displacement error (ADE) in percent of the image size and the side accuracy, the share of turns whose predicted endpoint lies on the correct side of the image. VAMOS and CityWalker cannot follow language commands, so they receive the goal predicted by our model, which isolates the shape of their paths from the choice of the goal. Some methods plan only a few metres ahead, Foresight for example about 5 m, whereas the references extend to where the path leaves the view. Scoring such a short plan against the full reference penalises it for the part it never claimed to cover. We therefore report every metric against the full reference (Full) and against the reference cut at the horizon of Foresight, where every prediction is cut after the same distance along the reference (Hor.).

Real-World Setup

We deploy WayTrace on a Unitree B2W with a Stereolabs ZED X stereo camera, lidar-inertial odometry [58] and a GNSS receiver. Holistic Fusion [59] fuses the lidar-inertial odometry with the GNSS measurements for state estimation. The trace predictor runs onboard on an NVIDIA Jetson AGX Thor, the orchestrator VLM on a remote server, and the RL navigation policy [5] runs on an NVIDIA Jetson AGX Orin.

Implementation Details

The orchestrator is Qwen3.8-Flash-Next [60] with FP8 weights, used without fine-tuning. It runs on a remote server with four H100 GPUs and takes about 1.3 s per decision. The trace predictor is trained with LoRA adapters on RynnBrain-9B for one epoch on the 1.19M samples. On Human-Test, the single-call model reaches a NaviTrace score of 83.2, on par with the two-stage model in Table III, and with FP8 weights it takes about 1.5 s per request on the Jetson AGX Thor. The open-loop results use the two-stage model, and all closed-loop experiments use the single-call model (App. Sec. A).

Experiments

Our experiments examine whether expressing a web-map route as a sequence of atomic language commands, each executed as an image-space trace, enables a robot to reach destinations in a city reliably, and what each level of the hierarchy contributes. We structure them around five questions.

• Q1 Does the hierarchy follow web-map directions over long routes, and how do existing navigation models perform on the same task? (§ 10-A)

• Q2 How robust is it to GNSS errors, and where do its failures come from? (§ 10-B)

• Q3 Does it transfer to the real world? (§ 10-C)

• Q4 What does each component contribute? (§ 10-D)

• Q5 Which new capabilities does a language interface between map and motion enable? (§ 10-E)

Q1: Long-Range Navigation from Web-Map Directions

We first evaluate whether the full system follows web-map directions over complete missions. Each method runs the ten missions of the simulation benchmark (Section IV-A), three times each with perfect GNSS. WayTrace completes 83.3 % of the missions, whereas no existing method exceeds 20 % (Table I). The point-goal methods follow GNSS waypoints along the route with their released control loops (App. Sec. C). They fail mostly by stalling at obstacles along the sidewalk or by falling, most often at a river embankment that is the most frequent failure location across all baselines (App. Sec. D). OmniVLA, which receives the active waypoint as its goal pose, does not complete any mission and collides with facades, advertising boards and street furniture.

Method	Guidance	Low level	SR [%] ↑
CityWalker [2]	GNSS waypoints	released policy	20.0
VAMOS [3]	GNSS waypoints	released policy	13.3
OmniVLA [4]	GNSS waypoints	released policy	0.0
TIC-VLA [10]	tuned oracle	released policy	6.7
Foresight [11]	tuned oracle	path → RL policy	3.3
VENTURA [13]	tuned oracle	path → RL policy	6.7
WayTrace	orchestrator VLM	trace → RL policy	83.3

The language-conditioned methods require a language instruction for every route segment. Driven by our orchestrator VLM, they did not complete any mission, so we provide them with the strongest available instruction source instead. An oracle orchestrator issues each route command from the true pose without latency, and its command timing and wording are tuned separately for each method (Section IV). Even under these conditions, the language-conditioned methods rarely succeed. All of their successes occur on the shortest mission (152 m), and on the remaining nine missions they leave the route within the first 10 to 130 m. TIC-VLA over-rotates on turn commands and drifts off straight sidewalks, while Foresight and VENTURA turn into walls at junctions or follow side paths. These results indicate that executing individual atomic instructions reliably is the limiting factor for these models, which we examine further in § 10-D.

Q2: Robustness to GNSS Errors and Failure Attribution

CityWalker predicts waypoints learned from city walking videos, and VAMOS re-ranks candidate paths with an embodiment-aware affordance model. On our robot, however, several of their runs fail at the low level, by stalling at obstacles or by falling (§ 10-A). Paths predicted from images do not reliably keep the clearance that the robot needs, which also holds for the traces of WayTrace (Section III-E). A low-latency policy that avoids obstacles is therefore an integral part of the hierarchy of WayTrace, and for a comparison on the same low level, we replace the controllers of CityWalker and VAMOS with this RL navigation policy, which tracks their predicted paths through the same lifting and carrot. With perfect GNSS and our RL policy, these variants rarely stall or fall, but they cross the road outside the zebra crossing at 55 % and 32 % of the crossings, since a GNSS waypoint does not indicate where the crosswalk lies. They therefore succeed in only 36.7 % and 63.3 % of the runs, compared with 83.3 % for WayTrace. We then corrupt the position given to every method with a drifting GNSS error of 2, 5 and 7 m RMS (Figure 5). As the GNSS error grows, their success rate drops to 23.3 % and 10.0 % at 7 m. The additional failures are falls at kerbs and embankments onto which displaced waypoints lead the robot, stops short of or beyond the destination, and stalls at obstacles (App. Sec. D). In contrast, the success rate of WayTrace decreases from 83.3 % to 63.3 %, and its crosswalk and road violations remain below 6 % and 1 %, respectively, at every noise level. Across all noise levels, WayTrace leaves the crosswalk or the sidewalk about ten times less often than the baselines. This is because WayTrace predicts the path in the camera image rather than from the GNSS position, so GNSS errors do not shift the robot off the sidewalk or the crosswalk. Instead, they affect the timing of the orchestrator’s decisions, which relies on the distance to the next manoeuvre, and most failures of WayTrace occur when a manoeuvre is commanded too early, too late or at the wrong junction, so that the robot leaves the planned route.

To measure how much the orchestrator limits WayTrace, we replace the orchestrator VLM with the oracle orchestrator while keeping the trace predictor and the RL policy unchanged. The success rate rises from 83.3 % to 90.0 %.

Q3: Real-World Deployment

We deploy WayTrace on a Unitree B2W wheeled-legged robot in Zurich on four missions between 248 m and 1.37 km (Table II, Figure 1 and Figure 6). The routes come from the web map, and the same models, prompts and parameters as in simulation are used, apart from the RL policy, which is retrained for the real robot.

The Starbucks mission leads from a university building to the nearest Starbucks in the city centre over 426 m (Figure 1). The robot crosses two streets with tram tracks, passes a roadside construction, descends two sets of stairs and crosses the wide roads on the zebra crossing before it stops next to the A-board of the Starbucks. At one tram crossing, the operator briefly paused the robot to check for an approaching tram, since WayTrace cannot yet react to trams (§ 12). We do not count this pause as an intervention, because the commands and the path were not changed. The Snipes and Dieci missions lead through the old town, where the robot shares crowded pedestrian streets with pedestrians and restaurant tables, passes construction sites and, in the Dieci mission, crosses a footbridge and a crossing over tram tracks (Figure 6). In such places, the walkable space is often only a few metres wide, for example the passage between the construction barriers or the footbridge, so a waypoint displaced by a few metres of GNSS error would already lie in the construction site or off the bridge. WayTrace instead draws the path in the camera image along the visible passage. In the Snipes mission, the position estimate drifts by up to 12.7 m from the route while the robot approaches a crosswalk, yet the robot crosses on the zebra, because the crossing is executed using the context from the robot camera images (Figure 6, top, moment 2).

Mission	Type	Characteristics	Length [m]	Speed [m/s]	Commands	Interventions	Autonomy [%]
Starbucks	city centre	stairs, crosswalks, tram tracks, construction sites	426	1.12	26	0	100
Snipes	city centre	crowded streets, construction site, crosswalk	248	1.01	13	0	100
Dieci	city centre	footbridge, tram crossing, sidewalks, crowded streets	312	1.11	19	0	100
Mendokoro	rural trail, campus road, field paths, forest trails, plaza, ramp	1368	1.19	21	2	97.2

The Mendokoro mission covers 1.37 km on a road, field paths, forest trails and a university campus. On this mission, the operator intervened twice by replacing the current command with an alternative language command, without teleoperating the robot, after turns at the campus entrance and on an open plaza (Figure 6, moment 2). In both cases, the orchestrator did not switch back to a straight command after the turn had been taken, and it kept or reissued a turn command instead. On an open plaza, the camera image does not show the intersection that the turn refers to, so the trace predictor continues to draw a turning path, and the robot would turn around if the orchestrator did not issue the straight command in time. The operator therefore sent “continue straight along the path”, after which the orchestrator resumed. This failure shows a limitation of commands that refer to features that are no longer visible, which we discuss in § 12.

Q4: Component Analysis

Trace prediction. We first evaluate the trace predictor in open loop on the two test sets of Section IV-C (Table III). The trace predictor of WayTrace reaches a NaviTrace score of 83.2 on the hand-labelled set, compared with at most 57.0 of the baselines, and it chooses the correct side of a turn in 97.9 % of the cases. Figure 7 shows qualitative examples, including one of the failures. The vision-language navigation models TIC-VLA, Foresight and VENTURA choose the wrong side in a quarter to a third of the turns on the same set. This confirms that these models are brittle already at the level of a single atomic instruction, which explains their failures over long routes in § 10-A.

Temporal history. We compare the temporal trace predictor with a model that observes only the current frame on 50 closed-loop episodes in the simulated old town, each driven by a single command with three seeds. With history, the robot succeeds in 94.7 % of the runs, compared with 74.7 % without. The gap arises mainly from the turns: without history, the robot keeps executing a turn after it is complete (App. Sec. E). The history thus lets the trace predictor recognise the progress of the requested action, which the current frame alone cannot provide.

Intent-conditioned data. Adding the intent-conditioned samples (Section III-D) improves trace prediction monotonically, from a NaviTrace score of 79.3 with the hand-labelled samples and reconstructed walks alone to 83.3 with 863k additional samples (Figure 8). Reconstructed walks pair each image with the single path that was walked, whereas the intent-conditioned samples pair the same image with several commands, including the branches that were not taken and paths to landmarks. We attribute the improvement to this diversity of commands per scene, which encourages the model to follow the command rather than the most likely path.

Orchestrator VLM. Finally, we compare orchestrator models on single decisions replayed from the logged requests of the long-range runs, with the same prompt, map, camera images and request text that the live system sent (Table IV). The label of each decision follows from the route and the true position of the robot. A decision is critical when it would end the mission, for example a manoeuvre issued too early, a manoeuvre of the wrong kind or side, a missed manoeuvre or leaving the zebra early. The self-hosted Qwen3.8-Flash-Next model used by WayTrace makes a critical error in 5.1 % of the decisions, mostly at crosswalk entries and turns. The strongest commercial models reduce this rate to 3.2 %, but they take three to five times longer per call and require a commercial API. This trade-off motivates our choice of a fast. self-hosted model, and it suggests that the orchestrator will improve directly with faster and stronger VLMs.

Method	Human-Test (hand-labelled, 953 samples)						Walks-Test (auto-labelled, 31,012 samples)
	NT ↑		ADE ↓		Side ↑		NT ↑		ADE ↓		Side ↑
	Full	Hor.	Full	Hor.	Full	Hor.	Full	Hor.	Full	Hor.	Full	Hor.
Ours	83.2	85.8	5.0	4.0	97.9	98.3	70.4	72.5	8.6	7.5	92.6	94.8
CityWalker{sup}†`	57.0	66.2	13.9	10.0	90.4	91.2	50.9	60.7	15.8	11.4	87.3	87.2
VAMOS†	56.6	42.6	16.4	19.8	97.7	95.5	48.8	33.7	18.5	22.3	92.5	91.6
Foresight	50.5	68.3	15.0	8.9	77.4	90.5	41.8	60.1	17.3	10.9	75.1	89.2
VENTURA	45.2	60.4	17.1	11.9	65.6	67.8	33.2	48.5	20.2	14.7	70.2	75.2
TIC-VLA	22.6	44.6	26.6	19.3	66.3	72.0	10.5	32.8	29.8	22.1	80.3	85.6

Orchestrator VLM	all	0 m	5 m	s/call	$/1k
Qwen3.8-Flash-Next (ours)	5.1	4.3	6.1	1.4	—
Gemini 3.5 Flash-Lite	5.9	4.0	8.2	5.1	2.9
Gemini 3.8 Flash	3.2	3.2	3.3	6.5	7.4
GPT-6 Sol	3.2	1.5	5.4	4.5	7.1

Applications of the Language Interface

Beyond following a given route, the hierarchy also handles situations in which the planned route cannot be followed, such as a closed street or a missed turn, and missions for which no route is available at all. Since the route enters the hierarchy only through language commands, these cases are handled at the level of the directions, without changes to the trace predictor or the RL policy. We demonstrate three such cases in simulation (Figure 9).

In the first, a construction fence closes a street on the planned route. When the orchestrator commands the turn into this street, the trace predictor does not draw a path into the fence, and the robot continues along the sidewalk (Figure 9, frame A1). Having passed the closed street, the robot requests a new route from the web map, which returns an alternative path around the block, and it reaches the destination along this path. In the second, we mimic an orchestrator failure by dropping a turn command, so that the robot continues straight past the turn. As in a web map that recomputes the route after a missed turn, the robot requests a new route once it is 10 m off the planned route, and it reaches the destination along the new route without turning back.

Finally, a person asked for directions typically describes the way as a sequence of steps grounded in landmarks, such as “Turn right into the narrow lane just before the black ONLY clothes shop”. We give six such directions to the orchestrator and run a short mission of 127 m without any map, route or GNSS, using only the camera images and odometry. The robot successfully navigates to the entrance of the destination using the provided directions. Further details are given in App. Sec. A.

Discussion and Limitations

Orchestrator. The off-the-shelf orchestrator accounts for several of the failures (§ 10-B). It issues turns too early or too late, takes the wrong junction when the target street is narrow or not yet in view, enters crosswalks too early and sometimes does not switch back to a following command after a manoeuvre (§ 10-C). In the Starbucks mission, for example, it issued one of the crossing commands several metres before the crosswalk, so that the robot stepped onto the road beside the crosswalk and joined it only partway across. Moreover, we observe that it tends to weigh the text-based distance to the next manoeuvre more than the camera images. Since this distance inherits the GNSS error, this might explain the decrease of the success rate with the GNSS error, while the rule violations remain low. Stronger commercial models reduce critical decisions by about a third, but at the cost of a three to five times higher latency (Table IV). Fine-tuning the orchestrator

VLM on decisions that are verified by executing them with the trace predictor is a promising direction. The orchestrator also relies on a few heuristic rules on the robot and on a network connection to a remote server.

Dependence on GNSS and on the web map. GNSS only identifies the current step of the route, which makes WayTrace robust to drifting errors of several metres, but long dropouts leave the distance to the next manoeuvre stale. The method also inherits the limitations of the web map, such as routes that omit a crossing.

Scaling to longer missions. The hierarchy places no limit on the mission length, since any route decomposes into atomic manoeuvres. With a critical-error rate of about 5 % per decision (Table IV), however, a route of several kilometres with tens of turns and crossings would have a high failure rate, so the reliability of the components, in particular of the orchestrator, has to improve before such missions become practical.

Trace predictor. The image history covers only about 9 m of travel, whereas crossing a wide road takes 20 to 30 s, so long manoeuvres depend on the orchestrator issuing entry and exit commands at the right time. Since the walked paths in the training data pass around obstacles that the projected traces cross, the trace predictor also draws traces through obstacles and relies on the lifting and the RL policy to avoid them. Finally, it sees only the front camera, so in open areas without visible context, such as the plaza in the Mendokoro mission (§ 10-C), it can keep turning after a turn is complete.

Latency and the RL policy. A new command takes effect in the executed path about 3 s after its image (1.3 s for the orchestrator VLM and 1.5 s for the trace predictor), about 3 m at 1 m/s, which can be enough to miss a narrow opening. This is tolerable only because the RL policy runs at 5 Hz on the latest depth image, keeps following the previous path and avoids obstacles and pedestrians. Moreover, because of this latency, safety-critical reactions such as stopping at a red light or for an approaching tram cannot currently be handled. Lower latency would also allow speeds above the current maximum. speed cap of 1.3 m/s.

Evaluation and other embodiments. Our metrics measure compliance with pedestrian rules, such as crossing at crosswalks, but not the interaction with traffic, and the simulation lacks traffic and pedestrians. Moreover, the real-world missions are qualitative demonstrations without baselines. In principle, the hierarchy transfers to other platforms, such as humanoid robots, by replacing only the RL policy, possibly together with an embodiment-aware trace predictor [50], which remains future work.

Conclusion

We present WayTrace, a hierarchy that follows web-map directions through a city by translating them into atomic language commands and executing each command as a trace in the camera image, which an embodiment-aware RL policy tracks. Separating the decision of which command applies from its execution makes the system robust to GNSS errors and keeps the robot on crosswalks and sidewalks, where point-goal methods fail. A temporal trace predictor trained on automatically labelled street-level data executes atomic commands far more reliably than existing vision-language navigation models. Many of the remaining failures stem from the orchestrator, which makes improving its decisions the main direction for future work. Real-world missions of up to 1.37 km show that the approach transfers from simulation to city streets. We will release the simulation, the benchmark, the

models, the dataset and the code to support further research on long-range urban navigation.

Acknowledgment

The authors would like to thank Turcan Tuna for help with the state estimation and Claudio Bonaglia, Tim Tschanz and Cyril Püntener for help with the real-world deployments. This work was supported by the NVIDIA Academic Grant Program through the project Hierarchical Vision-Language Navigation Foundation Model. It was also supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID abs19 on Alps, by the Luxembourg National Research Fund (Ref. 18990533) and by the Swiss National Science Foundation (SNSF) through projects No. 200021E_229503 and No. 227617.

Author Contributions

MP led the project, developed the method, built the data pipeline and the simulation benchmark, carried out the experiments and the real-world deployments and wrote the manuscript. JF contributed to the brainstorming and the development of ideas throughout the project. PR trained the RL navigation policy used in the real-world deployments and helped with the deployments. MH and CC supervised the project. All authors reviewed and edited the manuscript.

Use of generative AI. A large language model assistant was used to edit the text of all sections and to write the scripts that generate the figures and tables from the logged experiments. The authors reviewed all content and take full responsibility for it.

Implementation Details

Route

The destination is resolved with the Google Places API, and the Google Routes API provides a walking route from the current position. The route is replanned on the pedestrian network of OpenStreetMap and divided into steps at manoeuvres, crossings, stairs and changes of the street name.

Orchestrator Request

Each request contains the destination, the directions within 20 m of the current route position, the next direction with its approximate distance, the distance of the GNSS position from the planned route, the current command, a satellite image of 50 m width rotated heading-up and up to five camera images. The directions are listed with one of three states, namely “in about $N$ m”, “done, $N$ m behind” or “in progress, far kerb in about $N$ m”. An example request is given in App. Sec. F. The camera images are selected by the distance travelled along the pose history, 3, 6, 9, 12 and 15 m before the current position. Images are sent as JPEG with at most 640 pixels on the long side. The model answers in JSON with the reasoning first and the command second. The orchestrator is Qwen3.8-Flash-Next with FP8 weights, served with vLLM on four H100 GPUs with tensor parallelism, temperature 0 and thinking disabled. A new request is sent every 2 s with at most one request in flight. In simulation, the answer is applied 1.2 s after the capture of its images, which matches the median latency of the server (1.21 to 1.29 s across the campaigns).

The prompt prescribes a fixed sequence of commands at a road crossing, namely an entry command (“Go left onto the crosswalk to the far end kerb”), the crossing itself (“Cross the road”) and an exit command that already contains the next turn (“Continue rightwards on the sidewalk”). The final command “Go to the <destination>” is issued once the destination is visible, and no further orchestrator calls are made after it has been accepted.

Rules on the Robot

Table V lists the rules that guard the orchestrator’s decisions.

Rule	Behaviour
Crossing rejection	Rejects a crossing command while the next crossing entry is more than 10 m ahead or on the other side.
Arrival rejection	Rejects “Go to the <destination>” while the destination is more than 15 m ahead.
Arrival lock	No further calls once the final command is accepted.
Turn display	A passed turn stays the next direction until the heading has turned by up to 45° or the robot is 15 m past it.
Command expiry	A command expires 12 s after its image capture, and the robot holds after 20 s without a valid command.
Off-route	More than 20 m from the route ends the run in simulation and pauses the robot.

Trace Predictor

Both stages of the two-stage model are LoRA adapters of rank 64, trained with a learning rate of $4 \cdot10^{-5}$ and effective batch sizes of 256 and 192, which took about 700 GPU hours on 16 GH200 GPUs. The single-call model is trained on 250k samples drawn from the dataset in proportion to its sources, with $\lambda_g = 2$ and $\lambda_{\mathrm{CE}} = 0.5$ in (3). In 30% of these samples, the hidden state is computed with the model’s own predicted goal instead of the reference goal, so that the head learns to complete a slightly wrong goal into a consistent trace. FP8 quantisation changes the NaviTrace score on Human-Test by 0.02. Both models are trained by supervised fine-tuning only (App. Sec. B). The trace predictor receives five past images with 448 pixels on the long side, the current image with 640 pixels on the long side and a 448 × 448 top-down map of the walked path within ±16 m. The images precede the command in the prompt. The MLP head consists of a LayerNorm and two linear layers (4096 → 1024 → 20) with a GELU activation and reads the hidden state after the end of the goal. In the 30% of the samples that use the model’s own goal, the cross-entropy on the goal tokens is masked. The head model is trained with learning rates of $3 \cdot10^{-3}$ for the head and $2 \cdot10^{-4}$ for the adapter, which took about 53 GPU hours. Quantisation uses dynamic FP8 with per-channel weights and per-token activations. The vision encoder and the output layer stay in bf16. On the Jetson AGX Thor, the median request takes 1.48 s, and the field logs show medians of 1.21 to 1.44 s.

Execution

Depth is taken from the simulator or from the stereo camera of the robot. Each trace point is lifted with the median of the valid depth (0.3 to 25 m) in a 5 × 5 window, or by intersecting its ray with the ground plane when no depth is available. The ground filter fits a plane to the depth points in a 15 × 15 pixel window around each trace point and rejects it when the residual exceeds 0.12 m, the normal deviates by more than 40° from the vertical, or the height differs from the running ground plane by more than 0.4 m plus 0.03 m per metre of distance. Up to two consecutive rejected points, for example at a pole or a kerb, are bridged if the trace continues on the ground behind them. If the trace is cut at an obstacle, its last 0.5 m is removed so that the robot stops short of the obstacle. The carrot is placed 4.5 m ahead along the path and published at 5 Hz. A new trace is requested on a new command, after 3 m of travel, after 40% of the trace has been executed or after 5 s. In simulation, the RL policy is the point-goal RL navigation policy of [5] with a velocity scale of 0.7, which yields a mean speed of about 0.95 m/s. On the robot, a retrained version of this policy is used.

Directions from a Person

In the mission with directions from a person (§ 10-E), the orchestrator receives the six directions, the current command, the camera images of the last 15 m and a line from odometry that reports the turns completed so far and the distance walked since the last turn, but no map, route, GNSS position or distances. A few rules on the robot use odometry only. A turn command stays active until the heading has changed by at least $50^\circ$ in that direction or the robot has walked 18 m, no new turn is accepted within 10 m of the last completed one, and the speed is limited to 0.6 m/s. The prompt asks the orchestrator to issue a turn when its landmark is 5 to 8 m ahead.

Training Data and Teacher

Source pool and selection. The Mapillary pool contains 51,907 sequences with 15.7M images from 182 cities and regions, of which 39,931 sequences are recorded at walking pace. Episodes span eleven frames and are selected around junction turns verified against OpenStreetMap, geometric turns away from junctions, passages straight through junctions, which serve as negatives for turning, and straight segments. We keep perspective cameras with a horizontal field of view of at most $80^\circ$ and do not undistort the images. Junction events require an OpenStreetMap node with at least three walkable ways within 10 m. Geometric turns require a heading change of 30 to $140^\circ$ over 5 m chords that agrees with the camera orientation of the reconstruction. Sequences in which the walker steps onto a major road, pauses or jumps are removed, and every city contributes at most 8% of the events.

Reconstruction and projection. Each sequence is thinned to frames at least 1 m apart along the GNSS track (median 1.4 m) and reconstructed with VGGT-$\Omega$ in windows of 30 frames with a stride of 10. This yields the camera centre $C_j$, pose $T_j$ and intrinsics $K_j$ of every frame $j$, and the metric scale of each window is the ratio of the GNSS arc length to the reconstructed arc length. The ground point below each camera is $X_j = C_j - h\,n$, with $h = 1.5$ m and the mean up direction $n$ of the window, and the trace of frame $t$ consists of the projections $\Pi(K_t,T_t,X_j)$ of the future ground points $j > t$. The trace is cut at the first occlusion, where it leaves the field of view, in which case it follows the image border in the direction of the turn, where the walker turns back or at the end of the window, and it is resampled to ten points. The history of a sample consists of the five preceding frames of the episode. Overlapping windows are aligned by their camera centres, and episodes whose windows disagree in turning direction or by more than 3 m are removed, as are episodes with GNSS freezes, out-and-back paths or a first trace point more than 10 m ahead. Only buildings and walls cut the trace. Poles, vehicles and pedestrians do not, since the trace predictor is expected to draw the path through them and the lifting removes the part behind an obstacle (Section III-E). The traces reach a median of 28.8 m ahead of the camera.

Command labelling and judging. The labeller receives the eleven frames with the future path, a top-down view of the walked path over OpenStreetMap, the lateral displacement of the path from the reconstruction and the GNSS manoeuvres as weak evidence, with the instruction that the drawn path takes precedence. Commands have 3 to 12 words, and words for curvature such as “bend” or “veer” are not allowed, since a turn requires a visible choice. During training, each sample uses the command or one of its paraphrases, and in 15% of the samples the landmark variant if one exists. The judge sees the same frames and classifies each episode as good, wrong command, ambiguous, erratic path or bad image. Only good episodes are kept, which retains 94.5% of them. The result is 26,740 episodes from 14,609 sequences in 161 regions, and every frame of an episode with a valid trace becomes a sample, 288k in total.

Hand labelling. One annotator drew all hand-labelled traces with a web tool that shows the image, the command and, in later rounds, a proposed trace that can be accepted, edited or redrawn. The tool records the time from showing a command to saving the trace. Counting traces that took more than 15 s as breaks and excluding them, the 3,311 traces of all rounds took 5.9 h, with a median of about 7 s per trace. The training set of the teacher contains 1,917 traces on 605 images. The test set Human-Test was labelled in the same way and is excluded from all training data.

Anchor images. The anchor images face a decision point on OpenStreetMap, namely a junction, a crossing or a trail. All anchors at trail forks, junctions between streets and trails, trail segments and crossings are kept, while street junctions are capped at 90k and plain street segments at 15k. Each sequence contributes at most five anchors and each city at most 8%. Sequences of the test sets, the hand-labelled sequences and duplicate routes are excluded.

Intent-conditioned commands. The commands are written by a VLM (App. Sec. F-E), which receives the anchor image and a hint about the type of place from OpenStreetMap (junction, crossing, trail or street). It rejects images in which no command can be followed and writes one command for each visible branch from a fixed set of phrasings, such as “Take the left fork” or “Cross the road and go right”, and up to two commands towards landmarks that pass a uniqueness test. It keeps about 80% of the images and writes three to five commands per kept image, 366k in total for 106k kept images, of which about half refer to a landmark. When a teacher trace is propagated, a subsequent frame is kept while at least three trace points that lie more than 2.5 m ahead remain in view. The 863k samples consist of 366k anchor samples and 497k samples in subsequent frames. The hand-labelled traces are propagated in the same way.

Dataset. The final dataset contains 1,190,594 samples, namely 3,950 hand-labelled and NaviTrace samples, which are repeated ten times, 288k samples from reconstructed walks and 863k intent-conditioned samples. The test split of the reconstructed walks consists of 10% of the sequences, stratified by manoeuvre and region, and it is excluded from all sources together with the sequences of Human-Test.

Single-image teacher. The teacher uses the two-stage formulation with the current image only, at 1280 pixels on the long side. It is RynnBrain-9B trained on the training split of NaviTrace and the hand-labelled traces with supervised fine-tuning followed by GRPO.

Reinforcement learning. We experimented with GRPO, using the NaviTrace score as reward, at several stages of development. It improved the supervised model when the training data was small, but brought no improvement with the large datasets, so the final trace predictor is trained with supervised fine-tuning only.

Simulation Environment and Missions

The scene covers 0.91 km$^{2}$ around the river Limmat. Terrain and buildings come from swissALTI3D and swissBUILD-INGS3D, streets, sidewalks, crossings, shops and street furniture from OpenStreetMap, and the appearance of streets and shop fronts from 380 Mapillary images and street-level imagery of 181 shops. Materials, props such as bicycles, e-scooters, trams, fountains and advertising boards, and the river and bridges were added for the missions, and crossings were repaired where the reconstructed geometry did not match reality. Each mission runs in a sub-scene that keeps all buildings and the terrain and disables distant props more than 120 m from the route. The routes were planned on OpenStreetMap and checked for traversability with a collision box of 1.0 × 0.8 m. The time limit of each mission is 300 s plus 2.5 s per metre of route, and Table VI lists the ten missions. The robot is driven by a learned locomotion policy and has a front camera of 1024 × 576 pixels. The scene contains no pedestrians or vehicles. Since the simulation cannot query the web map during a mission, the directions are generated from each route in the style of the web map. In the real world and in the rerouting experiment (§ 10-E), the route comes from the Google Routes API, and the directions are generated from it in the same way. A run fails when the robot leaves the route by more than 20 m, stalls, falls or exceeds the time limit. A crossing counts as taken on the zebra when at least half of the robot’s path over the road at that crossing lies on the zebra, and crossing the road elsewhere counts as a violation. A run that reaches the destination with a violated crossing fails. The road violation excludes stretches where the route itself follows the road, as in alleys without a sidewalk. The routes are pedestrian routes on OpenStreetMap that were compared with the corresponding web-map walking routes, and the positions of the generated directions were checked against the scene. The robot camera has a horizontal field of view of 104.8°. The episodes run on a cluster of RTX 3090 GPUs at about a quarter of real time. The orchestrator and the trace predictor are served on remote H100 GPUs, and their answers are applied 1.2 and 1.5 s of simulation time after the capture of their images. The crosswalk and road violations are computed from the scene labels of the surface under the robot, sampled every 0.5 m along the driven path. Samples at the kerb edge, with walkable ground within 0.5 m, are not counted as road violations.

The GNSS error is a first-order Gauss-Markov process per axis with a time constant of 45 s and a standard deviation of $N/\sqrt{2}$, so that the radial RMS is $N$ m, plus white noise of 0.2$N$. It affects the route position, the distances and the map of the orchestrator. The trace predictor, the lifting and the RL policy use odometry only. Success and the off-route criterion are evaluated with the true pose.

Baseline protocols. The released CityWalker and VAMOS followed waypoints on the validated route with perfect GNSS, and the benchmark ended the run once the robot came within 3 m of the destination. The variants with our RL policy and OmniVLA followed the web-map walking line with their own stop at 2 m by their GNSS position and were scored like WayTrace. All methods were aborted after 30 s without motion. The baselines were also aborted after 180 s without 2 m of route progress or when they were more than 40 m from the route for 60 s, and WayTrace at 20 m from the route. CityWalker receives five frames 1 s apart with their positions and the active waypoint and predicts five waypoints at 1 Hz. VAMOS samples 50 paths in the image towards the waypoint and re-ranks them with its affordance model. OmniVLA receives the active waypoint as a pose goal. TIC-VLA runs its released asynchronous loop with its measured latency and its own pure-pursuit controller at up to 0.7 m/s. Foresight (with one critique and refinement round) and VENTURA predict a path from the current frame (Foresight with three history frames), which is executed in a stop-and-go manner through our lifting and RL policy at up to 0.7 m/s. Under the orchestrator VLM, TIC-VLA, Foresight and VENTURA completed none of ten runs each. The point-goal baselines receive waypoints every 20 m (CityWalker) or 10 m (VAMOS, OmniVLA). Concurrent navigation foundation models [34, 36, 38] have not released their weights, and LightNav-0 [37] was released very recently, so we do not compare with this concurrent work.

Oracle orchestrator for WayTrace. Each command is issued when the true route position passes its trigger point, which is placed relative to the corresponding direction at the median position where the orchestrator VLM issued it in successful runs plus 1.2 m for its latency, for example 9.3 m before a junction turn and 10.5 m before a crosswalk entry.

Additional Results

Table VII adds the released CityWalker and VAMOS to the comparison with perfect GNSS, and Table VIII lists the failure causes of all methods under each GNSS condition. Table IX breaks the orchestrator decision benchmark down by decision type, and Table X lists the data-scale ablation of the trace predictor.

Method	GNSS [m]	Runs	Off zebra	Water	No ground	Fell	Short	Overshot	Stuck	Lost	Other	Failures
CityWalker	0	30	0	0	2	4	0	0	18	0	0	24
VAMOS	0	30	0	0	0	1	0	0	25	0	0	26
CityWalker + RL policy	0	30	16	0	0	3	0	0	0	0	0	19
CityWalker + RL policy	2	30	10	2	0	4	0	0	2	0	0	18
CityWalker + RL policy	5	30	8	1	0	3	6	2	3	1	0	24
CityWalker + RL policy	7	30	4	0	1	4	9	0	5	0	0	23
VAMOS + RL policy	0	30	9	2	0	0	0	0	0	0	0	11
VAMOS + RL policy	2	30	11	1	0	0	0	0	0	0	0	12
VAMOS + RL policy	5	30	5	6	1	5	3	1	2	2	0	25
VAMOS + RL policy	7	30	2	3	2	7	6	1	2	4	0	27
WayTrace	0	30	0	0	1	0	0	1	0	3	0	5
WayTrace	2	30	0	0	0	0	0	0	1	5	0	6
WayTrace	5	30	1	0	0	1	0	0	1	5	0	8
WayTrace	7	30	1	0	0	0	0	1	1	8	0	11
WayTrace, oracle orchestrator	0	30	2	0	0	0	0	0	0	1	0	3

Model	Critical [%] $\downarrow$ all	0 m	5 m	follow	turn	zebra entry	on zebra	zebra exit	arrival	s/call	$/1k
Qwen3.8-Flash-Next (ours, self-hosted)	5.1	4.3	6.1	0 / 200	18 / 338	20 / 244	2 / 66	9 / 87	0 / 22	1.38	local
Gemini 3.5 Flash-Lite	5.9	4.0	8.2	4 / 200	19 / 338	5 / 244	0 / 66	28 / 87	0 / 22	5.12	$2.9
Gemini 3.8 Flash	3.2	3.2	3.3	0 / 200	10 / 338	4 / 244	12 / 66	5 / 87	0 / 22	6.53	$7.4
GPT-6 Sol	3.2	1.5	5.4	2 / 200	15 / 338	6 / 244	1 / 66	7 / 87	0 / 22	4.52	$7.1

Training data	Samples	NT $\uparrow$	ADE $\downarrow$	Side [%] $\uparrow$
Hand-labelled + reconstructed walks	327.8k	79.33	6.07	95.9
+ 10k intent-conditioned	337.8k	80.66	5.67	97.0
+ 100k intent-conditioned	427.8k	82.57	5.14	97.5
+ 863k intent-conditioned	1,290.6k	83.32	4.94	98.1

Open-loop protocol. Human-Test contains 953 hand-drawn reference traces on 332 images, labelled like the training traces of the teacher, of which 409 refer to a landmark. For 96 % of its samples, the five preceding frames and the walked path are reconstructed from the sequence with VGGT-Ω. Walks-Test contains all frames ($t = 0,\ldots,10$) of 2,869 judge-approved episodes from 1,524 held-out sequences in 116 regions, the held-out 10 % of the reconstructed-walk sequences, with 17,069 turns, 7,996 passages straight through a junction and 5,947 straight segments. The NaviTrace score combines dynamic time warping, the endpoint error and a semantic penalty from a Mapillary Vistas segmentation for the human embodiment [50]. ADE is the mean distance after resampling both traces to ten points. The side accuracy is computed on samples whose reference ends more than 8 % of the image width from the image centre. The NaviTrace score is computed in pixels of the original image, with the prediction resampled to the length of the reference, and an unparseable output scores 0. VAMOS receives our predicted goal as a pixel goal, and CityWalker receives it lifted to a metric goal on the reconstructed ground plane, with its four preceding frames and their positions. Foresight receives the last four frames cropped to its input format, VENTURA the current frame and TIC-VLA the current frame with four history frames, whose metric waypoints are projected through a flat-ground camera model. For the horizon-matched columns, the reference is cut where Foresight’s prediction ends (on average at 69 and 66 % of the reference on the two sets), and every prediction is cut after the same distance along the reference.

Start	Destination	[m]	Manoeuvres	Zebras	Limit
M1	Neumarkt	Rheinfelder Bierhaus	152	1 turn	0	669
M2	Bellevue	Karl der Grosse	223	1 turn	0	858
M3	Müh lesteg	dieci pizza takeaway	328	4 turns, 2 crossings	2	1119
M4	Müh lesteg	Burger King	361	2 turns, 3 crossings	3	1195
M5	Seilergraben	Burger King	393	1 fork, 2 crossings, stairs down	3	1275
M6	Seilergraben	Café Zähringer	406	2 crossings	2	1308
M7	Neumarkt	Starbucks	518	2 turns, 1 crossing	1	1596
M8	Bellevue	Café Henrici	633	1 turn	0	1881
M9	Zunfthaus zur Waag	dieci pizza takeaway	799	1 turn, 1 fork, 2 crossings	2	2289
M10	Limmatquai, Central	Karl der Grosse	864	2 turns, 2 crossings	3	2461
Total		4676	14 turns, 2 forks, 14 crossings, 1 staircase	16

Method	SR [$\uparrow$] [%]	Crosswalk viol. [$\downarrow$] [%]	Road viol. [$\downarrow$] [%]
CityWalker	20.0	20.8	1.63
VAMOS	13.3	12.5	0.47
CityWalker + RL policy	36.7	54.8	7.58
VAMOS + RL policy	63.3	31.8	1.84
WayTrace	83.3	2.2	0.82

Temporal History

Each of the 50 episodes is driven by a single command, such as “Turn left at the intersection and continue along the street”, which the trace predictor executes for the whole episode. Half of the episodes are turns, in which the command no longer describes the view once the manoeuvre is complete, and the other half are controls, namely crossings, forks and straight segments. A run succeeds when the robot comes within 2 m of an end zone on the commanded branch. Requiring the robot to enter the end zone exactly yields 86.7 against 66.0 %, which preserves the gap. Table XI lists the results and Figure 10 shows typical failures of the model without history.

Trace predictor	SR [%] $\uparrow$ all	turns	controls	Repeated turn [%] $\downarrow$ turns
Current frame only	74.7	56.0	93.3	56.0
Temporal	94.7	94.7	94.7	9.3

Prompts

This section lists the prompts of all models, verbatim. Text in angle brackets marks images, and text in braces marks fields that are filled in for each request.

Orchestrator VLM

The same prompt is used in simulation and on the robot. The model answers in JSON with a schema that enforces the fields reason and command.

System prompt.

You are the navigator of a sidewalk robot. Each request gives you:
- the mission goal, its legs, and the last command the robot received (still active);
- a top-down satellite map (width given in the text), HEADING UP: rotated so the robot always points up, so left and right on
  the map are the robot's left and right (a small arrow marks north); centred on the robot: red arrow = robot position and heading,
  blue line = planned route (blue dot = where the next Google Maps manoeuvre starts), teal line = path driven so far;
- the next Google Maps instruction and the distance to it along the route;
- the camera image now, and the camera images 3, 6, 9, 12 and 15 m ago.
Give the ONE command the robot should follow now: either keep the last command or replace it.
The robot understands short commands like these (examples from its training vocabulary):
  Following: "Continue along the sidewalk", "Continue along the pedestrian street", "Continue along the path",
    "Continue straight along the sidewalk", "Follow the paved path straight ahead", "Follow the dirt path curving to the left".
  Straight through a junction: "Go straight through the intersection and continue along the road",
    "Continue straight past the green trash can".
  Turns (no "and continue along ..." part): "Turn left at the intersection", "Turn right at the intersection",
    "Turn left at the junction", "Turn right at the junction", "Turn left onto the side street", "Turn right onto the side street",
    "Turn left into the pedestrian street", "Turn right at the corner", "Go left at the junction", "Go right at the junction",
    "Turn right at the fork", "Turn left at the end of the path", "Take the right path at the junction", "Go left at the fork".
  Slight bearing (NOT a turn: a fork, a wall or a building edge that splits the way): "Take the left fork", "Take the right fork",
    "Keep left of the wall and continue along the path", "Keep right of the building and continue along the sidewalk",
    "Bear left along the building". Use these instead of "turn left/right" when the route only deviates slightly.
  Turns anchored on something visible: "Turn right at the lamp post and continue along the sidewalk",
    "Turn left at the red building and continue along the sidewalk", "Turn left at the stone mausoleum and continue along the path".
  Crossings: always these commands, in this order:
    1. Close to the crosswalk: "Go left onto the crosswalk to the far end kerb" / "Go right onto the crosswalk to the far end kerb" /
       "Go straight onto the crosswalk to the far end kerb" (always say the side).
    2. Once the robot is on the crosswalk and already faces the far kerb, while the far kerb is still more than about 5 m
       away: "Cross the road". Do not keep "Go left/right onto the crosswalk" once the robot faces across the road: the side
       word keeps turning it on the crosswalk.
    3. While the robot is on the crosswalk, the route directions show the crossing as "in progress" and the next route
       instruction is the exit, "After the crossing, ...". Its blue dot is the far kerb, where the exit happens.
       Give the exit when that instruction is about 5 m away, picked by this instruction:
       - "After the crossing, turn left onto the <surface>": "Continue leftwards on the <surface>"
       - "After the crossing, turn right onto the <surface>": "Continue rightwards on the <surface>"
       - "After the crossing, continue straight onto the <surface>": "Continue along the <surface>"
       Until then keep the crossing command ("Cross the road") once the robot faces the far kerb. Once you have given the
       exit, never go back to a crossing command for the same crosswalk.
       This exit command IS the turn after the crossing; do not give a separate "Turn left/right at the intersection" for it.
  Going to a visible object: "Head to the traffic light pole", "Walk to the black trash bin", "Head to the metal bike rack",
    "Go to the parked van on the left".
  Arriving at the destination: "Go to the <Location>" -- exactly this, with the destination's name and nothing else (no
    entrance, A-board, side or "stop"). The robot finds the shop's door or board and stops there by itself.
SURFACE WORDS: the route directions name the surface the robot walks on after each turn and crossing ("onto the sidewalk",
"onto the pedestrian street", "onto the bridge"). Use exactly that word in your command ("Continue along the pedestrian
street", "Turn right at the intersection and continue along the pedestrian street"). Say "sidewalk" only when the directions
say sidewalk: a pedestrian street has no sidewalk, and the robot will look for one.
How to decide:
- The GPS position and every distance in the text ("in about 16 m", "destination in about 12 m") are noisy and can be off
  by 10-20 m. Never time a manoeuvre from the distance countdown alone. The one exception is the crossing exit (rule 3),
  which you time from the distance to the far kerb.
- Use the global context (mission legs, map, planned route, driven path so far) to know WHICH manoeuvre comes next and on
  which side, and what has already been completed.
- Use the camera images to decide WHEN.
- TURNS at a junction, intersection, side street or opening (not crosswalks): give the turn command as soon as the opening,
  side street or intersection for that turn is visible in the current image, even far ahead, only partly visible, or at
  the edge of the image. Do not wait until it is close or beside the robot: the robot can only take the turn while the
  opening is still ahead of it, and your turn command reaches it a few seconds late. Only keep the follow command when no
  such opening is in view yet.
- CROSSWALKS: give the crossing command when the robot is close to the crosswalk (rule 1 above), "Cross the road" once it
  is on the crosswalk facing the far kerb (rule 2), and the exit when the far kerb is about 5 m away (rule 3).
- SHOPS: give the arrival command "Go to the <Location>" as soon as the destination shop (its front, sign, logo or A-board)
  is in view.
Answer as JSON: {"reason": "one sentence", "command": "..."} . Write the reason FIRST: where the robot is now (e.g. on the
sidewalk, on the crosswalk, on the traffic island, at the far kerb) and which route instruction applies with its side; then
give the command that follows from that reason.

The following request is taken verbatim from a simulation run.

Request.

Mission goal: Take me to the dieci pizza takeaway. Destination: dieci pizza takeaway.
Route directions (Google Maps style; the words are for you, not for the robot):
  Now: continue along the sidewalk for about 0 m (destination dieci pizza takeaway in about 307 m)

1. Cross the road straight ahead [in about 0 m <- NEXT]
2. After the crossing, turn right onto the sidewalk [in about 15 m]
Next route instruction: Cross the road straight ahead in about 0 m (the blue dot on the map marks where it happens).
Distance from the planned route: 0 m.
Last command given to the robot (still active): Cross the road
Map width about 50 m. First image: the map. Then the camera images in the order listed.
Top-down map
<image: heading-up satellite map, 50 m wide>
Camera now
<image: current camera image>
Camera 3 m ago
<image>
Camera 6 m ago
<image>
Camera 9 m ago
<image>
Camera 12 m ago
<image>
Camera 15 m ago
<image>

Trace Predictor

The two-stage model and the single-call model use the same prompts. The single-call model receives the goal prompt and decodes the goal, after which the MLP head reads the trace from the hidden state. The images precede the user prompt.

Goal stage, system prompt.

You are a navigation expert controlling a walking robot. You are given, in order: five past first-person frames (about one to two meters apart, oldest first), the CURRENT first-person frame, and a top-down map showing the path walked so far (orange line; the blue dot at the centre is the current position, and the current orientation points up).

Predict the GOAL POINT of the future path in the CURRENT first-person frame as a single 2D point in normalized image coordinates (ranging from 0 to 1, where [0,0] is the top-left and [1,1] is the bottom-right).

- The goal point is where the commanded route ends in this view: extend it along the route until it leaves the view or becomes fully obscured; if the route leaves the image to the side (a sharp turn), place it on the image border in the turn direction
- Follow the navigation instruction; at junctions take the branch it commands
- Use the past frames and the top-down map to understand what part of the instruction has already been executed
- Output **only** the single goal point in the format: `[[x, y]]`
- Do not include any explanation or additional output

Goal stage, user prompt.

**Instruction**: {instruction}

The first five images are past views, the sixth is the current view, and the last is the top-down map of the walk so far. Predict the SINGLE goal point of the next path segment as one 2D point `[[x, y]]` in normalized image coordinates of the current view.

Trace stage, system prompt.

You are a navigation expert controlling a walking robot. You are given, in order: five past first-person frames (about one to two meters apart, oldest first), the CURRENT first-person frame, and a top-down map showing the path walked so far (orange line; the blue dot at the centre is the current position, and the current orientation points up).

Predict the future path in the CURRENT first-person frame as exactly 10 2D points in normalized image coordinates (ranging from 0 to 1, where [0,0] is the top-left and [1,1] is the bottom-right of the image).

- The GOAL POINT is provided to you: your trajectory MUST end at that goal point, tracing a feasible route to it
- Start your trajectory near the bottom of the image where the walked path enters the view (approximately y = 0.95)
- Follow the navigation instruction; at junctions take the branch it commands
- Use the past frames and the top-down map to stay consistent with the walked direction
- Output **only** the list of 2D points in the following format: `[[x1, y1], [x2, y2], ..., [x10, y10]]`
- Do not include any explanation or additional output

Trace stage, user prompt.

**Instruction**: {instruction}
**Goal point**: {goal}

The first five images are past views, the sixth is the current view, and the last is the top-down map of the walk so far. Predict the next path segment as 10 points in normalized image coordinates of the current view, ending exactly at the given goal point.

Single-Image Teacher

The teacher uses prompts derived from the NaviTrace benchmark [50] with the embodiment set to Human, and it receives only the current image.

Goal stage, system prompt.

You are a navigation expert for various embodiments including robots and humans. Given an image of the current scenario, a specified embodiment (e.g., legged robot, wheeled robot, human, or bike), and a navigation task (e.g., "Go down the road"), you will predict the SINGLE GOAL POINT: the final destination where a feasible trajectory for this embodiment would end, as one 2D point in normalized image coordinates (ranging from 0 to 1, where [0,0] is the top-left and [1, 1] is the bottom-right).

• The image shows a first-person view of the navigation scenario

• The goal point is where the path ends: extend it all the way to the destination if the path is visible; if the path is occluded, place it where the path becomes fully obscured (unless it can be reasonably inferred); if a red traffic light or unsafe crossing makes it unsafe to proceed, place it at the appropriate waiting position (e.g., just before the intersection or curb)

• The goal must be adapted to the embodiment's abilities and limitations

• All tasks that you are given have a solution

• Output only the single goal point in normalized image coordinates in the format: [[x, y]]

• Do not include any explanation or additional output

Embodiment Movement Characteristics

• Human: A standard pedestrian. Can navigate stairs and ramps but cannot climb tall obstacles.

• Legged Robot: A quadruped like ANYmal. Behaves similarly to a human, but it is shorter. It can handle stairs and escalators.

• Wheeled Robot: A wheeled delivery robot. Behaves like a wheelchair, preferring smooth surfaces such as walkways and ramps. It cannot use stairs or escalators.

• Bicycle: A standard cyclist. Follows traffic regulations and prefers bike lanes or streets. Cannot navigate stairs.

Goal stage, user prompt.

Embodiment: {embodiment}

Task: {task}

The image shows a first-person view from the embodiment's current position. Predict the SINGLE goal point (the final destination of the feasible trajectory) as one 2D point [[x, y]] in normalized coordinates (values from 0 to 1), adapted to the embodiment and the scenario shown.

Trace stage, system prompt.

You are a navigation expert for various embodiments including robots and humans. Given an image of the current scenario, a specified embodiment (e.g., legged robot, wheeled robot, human, or bike), a navigation task (e.g., "Go down the road"), and the GOAL POINT (the final destination), you will predict a feasible future trajectory as a sequence of 2D points in normalized image coordinates (ranging from 0 to 1, where [0,0] is the top-left and [1, 1] is the bottom-right).

• The image shows a first-person view of the navigation scenario

• Start your trajectory near the bottom center of the image, which corresponds approximately to normalized coordinate [0.5, 0.95] (representing the current position of the embodiment)

• The GOAL POINT is provided to you: your trajectory MUST end at that goal point, tracing a feasible route to it

• The trajectory should be adapted to the embodiment's abilities and limitations, following surfaces the embodiment can traverse and avoiding obstacles and costly terrain

• All tasks that you are given have a solution

• Output only a list of exactly 10 2D points in normalized image coordinates (values between 0 and 1, 3 decimals), evenly spaced along the path, ending at the goal, in the following format: [[x1, y1], [x2, y2], ..., [x10, y10]]

• Do not include any explanation or additional output

Embodiment Movement Characteristics

• Human: A standard pedestrian. Can navigate stairs and ramps but cannot climb tall obstacles.

• Legged Robot: A quadruped like ANYmal. Behaves similarly to a human, but it is shorter. It can handle stairs and escalators.

• Wheeled Robot: A wheeled delivery robot. Behaves like a wheelchair, preferring smooth surfaces such as walkways and ramps. It cannot use stairs or escalators.

• Bicycle: A standard cyclist. Follows traffic regulations and prefers bike lanes or streets. Cannot navigate stairs.

Trace stage, user prompt.

Embodiment: {embodiment}

Task: {task}

Goal point: {goal}

The image shows a first-person view from the embodiment's current position. Begin your trajectory near the bottom center of the image (around normalized coordinate [0.5, 0.95]) and predict the path forward as a list of exactly 10 2D points in normalized image coordinates, tracing a feasible route for the embodiment that ends exactly at the given goal point.

Command Labelling of Reconstructed Walks

The labeller and the judge are Qwen3.8-Flash-Next, the model of the orchestrator. The frames show the future path as a red line with a yellow end dot. The GNSS facts are listed for every turn that the GNSS track shows in the episode.

Labeller, system prompt.

You label short walking episodes for a robot navigation dataset. You see the frames of one episode in temporal order (about one metre apart). On most frames a RED line with a YELLOW end dot is drawn: it is the path the walker ACTUALLY took from that frame onward (about 10 m ahead, projected into the image). After the frames you see a top-down view: the walked trajectory (blue, starting at the first frame = triangle, initial motion up) over the OpenStreetMap ways of the area (dashed green = footpaths, grey = roads).

Describe what the walker does over this episode as ONE short imperative navigation command, then give 4 alternative phrasings of the SAME command. The command teaches a robot ATOMIC, GENERIC motions. Rules:

• THE RED LINE IS THE PRIMARY EVIDENCE. It shows exactly where the walker went: which branch it takes at a split, which side of the road it ends on, whether it leaves the current way. Your command must describe THE ROUTE THE RED LINE TAKES — read it on several frames, especially the earlier ones where the whole manoeuvre is visible. The frames tell you WHAT is there (roads, paths, corners, crossings, objects); the red line tells you what the walker DID there. The map and the listed GPS facts are weak background evidence only — when they contradict the red line, the red line wins.

• After the top-down view you get IMAGE-TRACE FACTS: the sideways displacement of the red line on selected frames, measured from reliable visual odometry (NOT GPS). Trust them over the GPS facts. If they report a clear LEFT/RIGHT displacement, the walker really moved that way — command the manoeuvre at the branch visible in the frames (a plain "continue" command is then WRONG, unless the frames show a single path with no alternative that itself turns). If they report no clear side on every frame, do NOT command a turn, whatever the GPS facts list.

• Only what happens FROM the first frame ON counts. Anything the walker did before frame 1 is not part of the command.

• Structure: atomic motions (turn / cross / continue / go up-down), optionally followed by "and continue along the <surface>". 3–12 words

• NEVER describe path curvature. The words "bend", "bends", "curve", "curving", "veer", "follow the path as it" are FORBIDDEN.

• A turn command requires a CHOICE. If the red line leaves the current road/path onto another one at a visible junction, fork, side street, driveway or crossing — command that turn, anchored on what you see. If the red line stays on the one and only path while the path itself changes direction (typical on hiking trails: no visible alternative), the command is "Continue along the <trail/path>", NOT a turn.

• If the red line shows SEVERAL manoeuvres in this episode, the command includes all of them, in order, still concise: "Cross the road straight ahead, turn right, then cross to the left". Never invent a manoeuvre the red line does not show.

• TERMINOLOGY — use these words precisely, and classify the anchor of the (first) manoeuvre in the "anchor_type" field:

  • "crossing" / crosswalk: ONLY for walking ACROSS a road to its other side, with the verb "cross" ("Cross the road ..."). NEVER say "turn at the crossing" for a junction turn. Every crossing command MUST state its direction ("straight ahead" / "to the left" / "to the right"). — "Cross the road straight ahead", "Cross the road to the left", "Cross the road to the right" — decided by the red line and the IMAGE-TRACE FACTS. A bare "Cross the road" is incomplete.

  • "intersection": two or more roads/streets meeting or crossing (X or +), typically with corners and buildings.

  • "fork": ONE path or trail splitting into two branches (Y shaped), both of the same kind.

  • "the end of the path": the current way terminates on another one (T shape) and the walker must go left or right.

  • "side street" / "driveway": turning off the current road into a clearly smaller one.

  • "junction": generic fallback ONLY when none of the above fits. The goal wording MUST use the word or an anchor-type (or a close synonym of it) — never "intersection" in the goal with anchor_type "fork". For a plain continue with no manoeuvre, anchor_type is "none".

• Surfaces: path, footpath, sidewalk, trail, road, street, stairs.

• Style to imitate (real examples): "Turn left at the intersection and continue along the path" "Turn left at the fork" "Cross the road straight ahead and go right along the sidewalk" "Cross the road to the left and continue along the sidewalk" "Go left at the end of the path and continue straight" "Take the next left and then follow the trail" "Continue along the hiking trail" "Go up the stairs and take a right"

• The 4 variants must describe the same manoeuvre(s) with different wording.

• EXTRA, only when a distinctive PERMANENT object is clearly visible in the frames (a trash can, a lamp post, a gate, a bench, a sign, a post box, a hydrant, a traffic mirror, a distinctive building): one additional variant that anchors the manoeuvre on it, e.g. "Take a right at the trash can". NEVER anchor on vehicles, people, animals or anything else that will have moved by tomorrow — a robot may execute this command on a different day. If nothing distinctive and permanent is visible, use null. Answer STRICT JSON only, with "checks" FIRST — one entry per listed GPS manoeuvre, "confirmed" or "rejected: <what the red line / frames show instead>": {"checks": ["confirmed" | "rejected: ...", ...], "anchor_type": "intersection" | "fork" | "crossing" | "end_of_path" | "side_street" | "driveway" | "junction" | "none", "goal": "...", "variants": ["...", "...", "...", "..."], "landmark_variant": "..." | null}

Labeller, user message.

Episode frames in temporal order, ~1 m apart (red line = actual walked path from that frame onward):
<image: frame 1 with the walked path drawn> ... <image: frame 11>
Top-down view of the walked trajectory:
<image: top-down view of the walked trajectory over OpenStreetMap>
Facts (measured from the trajectory; the map hint is weak evidence — the frames decide):
- at ~{k} m in the GPS trajectory turns ~{angle}° {left|right}; the map also shows a junction
  there I the map shows no junction there, but the path often misses small streets and paths — check the frames.
IMAGE-TRACE FACTS (visual odometry, trust over GPS): frame {k}: red path ends {dx} image-widths {LEFT|RIGHT} ({clear|slight|no clear side
  }); ...
The navigation command and its 4 variants:

The judge receives no GNSS facts.

Judge, system prompt.

You are a quality judge for robot-navigation training labels. You see the frames of one short walking episode in temporal order (~1 m apart). On most frames a RED line with a YELLOW end dot is drawn: the path the walker ACTUALLY took from that frame onward (~10 m, projected from reliable visual odometry). The first frame is repeated LARGER at the end. You also get IMAGE-TRACE FACTS: measured sideways displacement of the red line on selected frames — these numbers come from the same projection and are AUTHORITATIVE for the side (left/right); reading a perspective photo plainly by eye is error-prone, so when your visual reading of the red line's side contradicts the numbers, THE NUMBERS WIN. Finally you get the INSTRUCTION that will be paired with this episode as a training label — it describes what the walker does FROM THE FIRST FRAME ON.
THE ONLY QUESTION: does the instruction match THE ROUTE THE RED LINE TAKES — the manoeuvre(s), their order, their side, and the branch chosen where the frames show several options?
- A manoeuvre already completed DISAPPEARS from later frames' red lines: judge the route from the FIRST frames and the numbers, never reject a turn instruction because late frames show a straight red line.
- A turn instruction while the red line (per the numbers) holds one way with no clear side change is a wrong_command; a plain "continue" instruction while the red line clearly leaves the current road/path at a visible branch is a wrong_command.
- If the numbers show a clear side and the instruction commands that same side, the side is CORRECT.
- A crossing command should state its direction ("straight ahead" / "to the left" / "to the right"). A stated direction contradicting the red line is a wrong_command; a MISSING direction while the red line crosses clearly to one side is "ambiguous"; a missing direction on a straight-ahead crossing is acceptable.

• verdict "erratic_path" only if the red line runs through walls, vehicles, water or fences; "bad_image" only if the images are unusable (blurry, blocked, inside a vehicle); "ambiguous" only if the scene offers several branches on the commanded side and the instruction does not single one out.

• TERMINOLOGY is reported but NEVER a reason to reject: set "terminology_ok" false when the anchor word mismatches the scene ("cross"/"crossing" = walking ACROSS a road; "intersection" = roads crossing; "fork" = one path splitting in two; "the end of the path" = a T), but the verdict stays "good" if the route matches. Answer with ONE JSON object only: {"instruction_matches_route": true|false, "side_correct": true|false|null, "terminology_ok": true|false, "image_ok": true|false, "verdict": "good" | "wrong_command" | "ambiguous" | "erratic_path" | "bad_image", "confidence": <0-1>, "reason": "<= 25 words"} Judge, user message. Episode frames in temporal order, ~1 m apart (red line = actual walked path from that frame onward): <image: frame 1 with the walked path drawn> ... <image: frame 11> FIRST frame again, larger (the command's viewpoint): <image: frame 1, larger> IMAGE-TRACE FACTS (visual odometry, trust over GPS): frame {k}: red path ends {dx} image-widths {LEFT|RIGHT} ({clear|slight|no clear side}); ... INSTRUCTION: {instruction} Return the JSON now.

Command Generation for Intent-Conditioned Traces

The VLM that writes the commands is Qwen3.8-Flash-Next at temperature 0.3. It receives the anchor image and one hint that depends on the type of place according to OpenStreetMap.

System prompt.

You write navigation commands for a walking robot. The photo is the robot's current view — every command must be executable starting from exactly there, and a human will later draw the exact walking path for each command, so every command must be unambiguous and checkable against the photo.

You also get an OSM HINT about what should be ahead. THE PHOTO ALWAYS WINS. The hint is often wrong or refers to something out of view; never write a command the photo cannot justify.

First VET the photo: reject it if the view is blocked, heavily blurred, ground/sky only, from inside a vehicle, crowd-occluded, or shows no walkable path.

Then write commands of these two types. Write EVERY structural command the scene genuinely supports (never pad with redundant ones), AND — this is a requirement, not an option — TWO LANDMARK commands whenever the photo contains two objects that pass the uniqueness test below, otherwise zero. Only when the photo contains no identifiable object at all do you return zero landmark commands and set reason "no_landmark". Landmark commands are as valuable as structural ones; a photo with only structural commands is an incomplete answer.

1. STRUCTURAL — route the robot through what is ACTUALLY VISIBLE.
*** Decide first, from the PHOTO ONLY, whether the way ahead splits.
- It does NOT visibly split: the only structural command is a corridor command — "Continue along the sidewalk", "Continue along the path", "Continue along the trail". Do not invent a junction because the hint mentions one.
- It DOES split: you must write ONE command per feasible branch. Naming only one of two visible branches is just as wrong as inventing a branch that is not there — the robot needs every option it could be asked to take. A branch counts as visible if you can see where the robot would walk to take it, even if it continues out of frame. ***
When a branch is visible, write ONE command per feasible branch:
- STREET junction (T or X, branch leaves at roughly a right angle): "Go left at the intersection" / "Go right at the intersection" / "Continue straight through the intersection". You may name what the branch is when it is clear: "Turn left onto the side street".
- PATH / TRAIL split (a Y, both branches still lead roughly forward): "Take the left fork" / "Take the right fork" / "Take the right fork of the trail". The word "fork" is ONLY for path and trail splits — never for a street intersection.
- STAIRS / ramp / opening, when visible: "Go down the stairs", "Go right at the opening ahead".
CROSSINGS — decide from the PHOTO whether a road can be crossed here (marked, or a sensible unmarked spot). If so, write one command per crossable direction, in EXACTLY this phrasing:
"Cross the road and go left" / "Cross the road and go right" / "Cross the road straight ahead"
(you may name the far side when it is clear: "Cross the road and turn right onto the sidewalk"). Do NOT write "cross ... to the left". Do not describe the crossing type (no "zebra", "marked", "signalised"). If no road actually needs crossing to continue, write a plain directional command instead ("Go left on the sidewalk").

2. LANDMARK — send the robot to a specific STATIC object or structure.
*** THE UNIQUENESS TEST — apply it to every landmark command you write. Read your own command back while looking at the photo: does EXACTLY ONE thing in the photo match it? If two or more things match, the command is useless, because the human drawing the path cannot know which one you meant. Either qualify it until exactly one matches, or name a different

object. ***
Street furniture that REPEATS along a street — poles, signposts, lamp
posts, bollards, bike racks, parked cars in a row — fails the test on its
own ("the utility pole" when four are in view). It passes as soon as you
pin it down by POSITION or by what it stands NEXT TO:
    "Go to the signpost at the corner of the fence" ·
    "Walk to the bicycle in front of the house" ·
    "Walk to the green bench next to the trash can on the left"
Qualify by RELATION or position, not by a guessed colour.
Objects that usually pass on their own, and are the best landmarks when
present: an individually recognisable vehicle ("the black scooter", "the
grey toyota"), a trash can, a bench, a gate, a doorway or building
entrance, a traffic light, a bus stop, stairs, a flag, a kiosk, a
container, a fence with something on it, a building with a distinguishing
feature ("the building with the red roof").
Name only what you can INDIVIDUALLY IDENTIFY: if you are not sure what an
object is, do not name it, and do not name text, brands or signs you
cannot read. Keep it as SHORT as it can be while still unique: "Go to the
trash can" beats "Approach the green wheelie bin with the yellow lid".
AVOID trees/vegetation unless a single clearly isolated one. NEVER a
person, animal, or anything that moves. Reachable ON FOOT along visible
walkable ground (no fences/walls/water in the way).

Style for every command: imperative, ONE action, CONCISE, at most 12 words,
no trailing period. Start with "Go", "Walk to", "Take", "Continue", "Cross"
or "Turn" — never "Approach", "Head to" or "Walk towards". Do NOT name the
surface material: write "path" or "trail", never "dirt track", "gravel path"
or "paved path". Do not invent objects or turns. Vary the verbs.

Commands in exactly the register we want (real accepted examples):
    "Go left on the sidewalk" · "Cross the road and go left" ·
    "Cross the road and turn right onto the sidewalk" · "Continue along the
    sidewalk" · "Continue along the trail" · "Go left at the intersection" ·
    "Turn left onto the side street" · "Take the right fork of the trail" ·
    "Take the right path past the parked bicycles" · "Go down the stairs" ·
    "Go right at the opening ahead" · "Go to the trash can" · "Go to the black
    scooter" · "Walk to the building with the red roof" · "Walk to the green
    bench next to the trash can on the left" · "Go to the building entrance
    with the guard" · "Go to the traffic light on the diagonally opposite end" ·
    "Walk to the bicycle in front of the house" · "Go to the fence with the
    posters" · "Go to the gate of the house on the left" · "Go to the bus stop
    sign" · "Walk to the Swedish flag" · "Go to the white container" ·
    "Walk to the grey toyota" · "Go to the signpost at the corner of the fence"

Return STRICT JSON:
{"reject": {"is_rejected": bool, "reason": "ok|blocked_view|blur|no_walkable_path|no_visible_junction|no_visible_crossing|no_landmark|
        other"},
 "goals": [{"task": "...", "task_type": "structural|landmark",
            "direction": "left|right|ahead", "target": "object or way, or empty"}]}
JSON only.

User message.
<image: anchor image>
One hint, depending on the type of place:
junction:        OSM HINT: a walkable junction/fork should be ~{m} m ahead. Cover EVERY feasible branch with a structural command, AND
                 add TWO landmark commands (one only if the photo holds just one object that passes the uniqueness test). If no junction is actually
                 visible/reachable, reject (no_visible_junction).
crossing:        OSM HINT: a pedestrian crossing should be ~{m} m ahead. Include one crossing command per crossable direction, plus every
                 structural command the scene supports, AND TWO landmark commands (one only if the photo holds just one object that passes the
                 uniqueness test) — an object on the FAR side of the road is an excellent target. If no crossing is visible, reject (
                 no_visible_crossing).
trail corridor:  OSM HINT: this is a junction-free hiking/trail corridor. Write EXACTLY ONE command: continue along the trail/path (e.g.
                 'Continue along the trail', 'Follow the trail uphill'). Never name the surface material. No landmark commands.
street corridor: OSM HINT: this is a junction-free street. Write 1–3 commands, at least one landmark command to a distinctive object; '
                 Continue along the sidewalk/path' style is allowed. If nothing distinctive is visible and the walk is trivial, reject (no_landmark)
.